This is an image from apple's documentation. They show a transform from a cube to sphere and also to some random geometry.
Only a few lines lower they state:
A morpher and its target geometries may be loaded from a scene file or
created programmatically. The base geometry and all target geometries
must be topologically identical—that is, they must contain the same
number and structural arrangement of vertices.
Could someone explain this paragraph because apparently I don't understand it.
Since a sphere will never have the same structural arrangement of vertices as cube(at least I think so), it is impossible to make transformation. But hey, we all see it in the picture. I also tried do to the transformation and I don't get the expected results. So how do you go from sphere to cube or vice versa?
"Topologically identical" means that the relationships between vertices in a mesh must be preserved, but their locations in space can change. Here's an example of that in 2D:
These two meshes have the same eight vertices, connected to each other in the same ways, but their positions (and thus the shape they form) differ.
To do the same in 3D with SceneKit, you need custom vertex data — the primitive shapes that SceneKit can generate for you (like SCNSphere, SCNBox, and whatnot) all have different topologies, so they can't be used as morpher targets.
If you want to morph a box into a sphere, you'll need to generate your own box and sphere with identical topology. The "some random shape" in Apple's illustration is a hint at how you might do that — it appears to be one of the variants of a superellipsoid. If you use the equations in that Wikipedia page you can generate a set of points that can be either on a sphere or on a cube depending on other parameters. Vary those parameters to generate a couple of meshes, create SCNGeometry from those meshes, and you've got valid SCNMorpher targets.
You can see a simpler example of morphing in Apple's SceneKit WWDC 2014 Slides sample app.
You can't presume the locations of each vertex in the given images; the cube doesn't neccesarily have eight and the left-most doesn't guaruntee to have 6.
Admittedly, I've not played with SCNMorpher but from that description I imagine it will interpolate on a per-vertex basis (so they will have to match up).
If it helps, picture the sphere as having a lot of 'dots' spread equally along its surface and those are pushed or squeezed to make the other surfaces
Related
I'm struggling with a texture-baking process with 3DSmax software. I have a white 3D mesh with 2 image textures. I'm trying to get a diffusemap (see target_diffuse_map.jpg). To do this, I exectue the following steps:
1) Affect image-texture1 and image-texture2 to face1 and face2 of the objet.
2) Clone the object to get the white colors when baking texture.
3) unwrap UVM.
4) Rendering Texture to obtain the diffuse map.
5) Projection of the texture + white colors on the cloned object.
Please, find these steps on this small video I made: https://drive.google.com/file/d/1h4v2CrL8OCLwdeVtLmpQwD250cawgJpi/view
I obtain a bad sampled and weird diffuse map (please see obtained_diffuse_map.jpg). What I want is target_diffuse_map.jpg.
I'm I forgetting some steps?
Thank you for your help.
You need to either:
Add a small amount of "Push" in the Projection Modifier
Uncheck "Use Cage" in the Projection Options dialog, while setting a very small value for the offset
Projection Mapping works by casting rays from points on the cage towards corresponding model points on your mesh. You did not push the cage out at all, therefore rays are not well defined; rays are cast from a point toward a direction which is the exact same point. This causes numerical errors and z-fighting. The there needs to be some time amount of push so the "from" and "to" points of each ray are different giving them a well-defined direction to travel.
The second option, instead of using the cage defined in the projection modifier, is to use the offset method (you probably still need to apply projection modifier though). This method defines each rays as starting from a point defined by taking the model point of the mesh and moving outward by a fixed offset amount in the direction of the normal. The advantage is that for curved objects with large polygons, it produces less distortion because the system uses the smoothed shading normal at each point. The disadvantage you can't have different cage distances at different points of the model, for better control. Use this method for round wooden barrels and other simplistic objects with large, smooth curves.
Also, your situation is made difficult by having different parts of the model very close to each other (touching) and embedded within each other - namely how the mouth of the bottle is inside the cap and the cap it touching the base. For this case, it might make sense to break the objects apart after you have the overall UV mapping, run projection mapping separately on each one separately, and then combine the maps back together in an image editor.
I want to create an Animoji in my APP. But when I contact with some designers they didn't know how to design an Animoji 3D model. Where can I find a solution for reference?
Solution I can thought is create many bones on face of 3D model, And when I get blendShapes of ARFaceAnchor, which contain the detail information of face expression, then I use it to update bone animations of partial face.
Thank you for reading. Any advises is appreciated.
First, to clear the air a bit: Animoji is a product built on top of ARKit, not in any way a feature of ARKit itself. There's no simple path to "build a model in this format and it 'just works' in (or like) Animoji".
That said, there are multiple ways to use the face expression data vended by ARKit to perform 3D animation, so how you do it depends more on what you and your artist are comfortable with. And remember, for any of these you can use as many or as few of the blend shapes as you like, depending on how realistic you want the animation to be.
Skeletal animation
As you suggested, create bones corresponding to each of the blend shapes you're interested in, along with a mapping of blend shape values to bone positions. For example, you'll want to define two positions for the bone for the browOuterUpLeft parameter such that one of them corresponds to a value of 0.0 and another to a value of 1.0 and you can modulate its transform anywhere between those states. (And set up the bone influences in the mesh such that moving it between those two positions creates an effect similar to the reference design when applied to your model.)
Morph target animation
Define multiple, topologically equivalent meshes, one for each blend shape parameter you're interested in. Each one should represent the target state of your character for when that blend shape's weight is 1.0 and all other blend shapes are at 0.0.
Then, at render time, set each vertex position to the weighted average of the same vertex's position in all blend shape targets. Pseudocode:
for vertex in i..<vertexCount {
outPosition = float4(0)
for shape in 0..<blendShapeCount {
outPosition += targetMeshes[shape][vertex] * blendShapeWeights[shape]
}
}
An actual implementation of the above algorithm is more likely to be done in a vertex shader on the GPU, so the for vertex part would be implicit there — you'd just need to feed all your blend shape targets in as vertex attributes. (Or use a compute shader?)
If you're using SceneKit, you can let Apple implement the algorithm for you by feeding your blend shape target meshes to SCNMorpher.
This is where the name "blend shape" comes from, by the way. And rumor has it the built-in ARFaceGeometry is built this way, too.
Simpler and Hybrid approaches
As you can see in Apple's sample code, you can go even simpler — breaking a face into separate pieces (nodes in SceneKit) and setting their positions or transforms based on the blend shape parameters.
You can also combine some of these approaches. For example, a cartoon character could use morph targets for skin deformation around the mouth, but have floating 2D eyebrows that animate simply through setting node positions.
Check-out the 'weboji' javascript library on gitHub. The CG artists we hired to create the 3D models get used with the workflow in minutes. Also, it could be an interesting approach to avoid proprietary formats and closed ecosystem issues.
Screenshots of a 3D Fox (THREE.JS based demo) and a 2D Cartman (SVG based demo).
Demo on youtube featuring a 2D 'Cartman'.
I've been trying without success to extract face features, for instance the mouth, from ARSCNFaceGeometry in order to change their color or add a different material.
I understand I need to create an SCNGeometry for which I have the SCNGeometrySource but haven't been able to create the SCNGeometryElement.
Have tried creating it from ARFaceAnchor in update(from faceGeometry: ARFaceGeometry) but so far have been unable.
Would really appreciate someone help
ARSCNFaceGeometry is a single mesh. If you want different areas of it to be different colors, your best bet is to apply a texture map (which you do in SceneKit by providing images for material property contents).
There’s no semantic information associated with the vertices in the mesh — that is, there’s nothing that says “this point is the tip of the nose, these points are the edge of the upper lip, etc”. But the mesh is topologically stable, so if you create a texture image that adds a bit of color around the lips or a lightning bolt over the eye or whatever, it’ll stay there as the face moves around.
If you need help getting started on painting a texture, there are a couple of things you could try:
Create a dummy texture first
Make a square image and fill it with a double gradient, such that the red and blue component for each pixel is based on the x and y coordinate of that pixel. Or some other distinctive pattern. Apply that texture to the model, and see how it looks — the landmarks in the texture will guide you where to paint.
Export the model
Create a dummy ARSCNFaceGeometry using the init(blendShapes:) initializer and an empty blendShapes dictionary (you don’t need an active ARFaceTracking session for this, but you do need an iPhone X). Use SceneKit’s scene export APIs (or Model I/O) to write that model out to a 3D file of some sort (.scn, which you can process further on the Mac, or something like .obj).
Import that file into your favorite 3D modeling tool (Blender, Maya, etc) and use that tool to paint a texture. Then use that texture in your app with real faces.
Actually, the above is sort of an oversimplification, even though it’s the simple answer for common cases. ARSCNFaceGeometry can actually contain up to four submeshes if you create it with the init(device:fillMesh:) initializer. But even then, those parts aren’t semantically labeled areas of the face — they’re the holes in the regular face model, flat fill-ins for the places where eyes and mouth show through.
I have implemented realtime ray tracer with MetalFramework for iOS and it is implemented for following optical prisms like dodecahedron, icosahedron, octahedron, cube, etc. All my figures are composed from triangles, for example cube - 12 triangles, octahedron - 4 triangles. I trace rays and search intersection with figure, then I search how ray moves in prism. Then ray leaves figure and I search intersection with skybox. The problem is in complicated figures. When I test cube fps is 60, but when I test dodecahedron fps is 6. In my algorithm intersection with figure is the same as intersection with any triangle. It means that when I check intersection with ray and figure I have to check intersection with all triangles. I need some idea how to do not check intersections for all triangles. Thanks.
let say you have world bounded by some bounding box
create grid (dividing this box to cubes or whatever)
each voxel/cell
Is a list of triangles that intersects or are in it so before rendering for each cell process all triangles and store index of all triangles inside or crossing
rewrite ray-tracer to trace through this voxel map
So just increment the ray through neighboring voxels it is the same as line rasterization on pixels. This way you have partially Z-sort done. So take first voxel hit by ray and test only triangles contained in it. If any hit on voxel was found then stop (no need to test other voxels because they are farer).
further optimizations
You can add flag if triangle has been tested so test only those which where not already tested because many triangles will be multiple times tested otherwise
[notes]
Number of voxels per axis greatly affect performance so you need to play with it a bit to achieve best performance. If you have dynamic objects then the voxel map lists computations must be done once in a while or even per each frame. For static scene there is sufficient to do this just once.
To trace efficiently you'll need to use an acceleration structure, for example a KD-tree or a bounding volume hierarchy(BVH). This is similar to using a binary search tree to find a matching element.
I would suggest using a BVH because it is easier to construct and traverse than a KD-tree. And I would suggest against using a uniform voxel grid structure. A voxel grid can easily have very poor performance when triangles are unevenly distributed through the scene or heavily concentrated in a few voxels.
The BVH is just a tree of bounding volumes, such as an axis aligned bounding box (AABB) which encompass the primitives within it. This way if your a ray misses the bounding volume you know that it does not hit any primitives contained with it.
To construct a BVH:
Put all the triangle in one bounding volume. This will be the root of the tree.
Divide the triangles into two sets where the bounding volume of each set of triangles is minimized. More properly you'd want to follow the surface area heuristic (SAH), where you want to create set of triangles where you minimize the sum of the (surface area of the BVH)/(# triangles) for both sets of triangles.
Repeat step 2 for each node recursively until you the number of triangles you have left hits some threshold (4 is a good number to try).
To traverse
Check if the ray hits the root bounding box, if it does then proceed to step 2 otherwise no hit.
Check if it hits the child bounding boxes. If it does then repeat this step for its children bounding boxes. Otherwise no hit.
When you get the a bounding box which only contains triangles you'll need to test each triangle to see if it is hit just like normal.
This is a basic idea of a BVH. There much more detail that I haven't gone into about the BVH that you'll have to search for, since there are so many variations in the details.
In Short Implement a bounding volume hierarchy to trace.
I'm developing an image warping iOS app with OpenGL ES 2.0.
I have a good grasp on the setup, the pipeline, etc., and am now moving along to the math.
Since my experience with image warping is nil, I'm reaching out for some algorithm suggestions.
Currently, I'm setting the initial vertices at points in a grid type fashion, which equally divide the image into squares. Then, I place an additional vertex in the middle of each of those squares. When I draw the indices, each square contains four triangles in the shape of an X. See the image below:
After playing with photoshop a little, I noticed adobe uses a slightly more complicated algorithm for their puppet warp, but a much more simplified algorithm for their standard warp. What do you think is best for me to apply here / personal preference?
Secondly, when I move a vertex, I'd like to apply a weighted transformation to all the other vertices to smooth out the edges (instead of what I have below, where only the selected vertex is transformed). What sort of algorithm should I apply here?
As each vertex is processed independently by the vertex shader, it is not easy to have vertexes influence each other's positions. However, because there are not that many vertexes it should be fine to do the work on the CPU and dynamically update your vertex attributes per frame.
Since what you are looking for is for your surface to act like a rubber sheet as parts of it are pulled, how about going ahead and implementing a dynamic simulation of a rubber sheet? There are plenty of good articles on cloth simulation in full 3D such as Jeff Lander's. Your application could be a simplification of these techniques. I have previously implemented a simulation like this in 3D. I required a force attracting my generated vertexes to their original grid locations. You could have a similar force attracting vertexes to the pixels at which they are generated before the simulation is begun. This would make them spring back to their default state when left alone and would progressively reduce the influence of your dragging at more distant vertexes.