What I am looking to do is to be able to push and pop transform matrices into SpriteBatch. I have 2 sprites, parent and child, and the child should be scaled, rotated, translated relative to the parent.
I currently have an implementation using textured quads but I think this should be possible using the built in SpriteBatch class and using Matrix.Decompose(). I'm not sure how to pass the decomposed values to SpriteBatch once I've decomposed them though.
I understand how to keep the matrix stack, I'm just looking for an example of using values coming from Matrix.Decompose() in conjunction with SpriteBatch.
Figured this one out myself finally. Most of the credit belongs to this blog post though.
You can use this method to decompose your matrix:
private void DecomposeMatrix(ref Matrix matrix, out Vector2 position, out float rotation, out Vector2 scale)
{
Vector3 position3, scale3;
Quaternion rotationQ;
matrix.Decompose(out scale3, out rotationQ, out position3);
Vector2 direction = Vector2.Transform(Vector2.UnitX, rotationQ);
rotation = (float)Math.Atan2((double)(direction.Y), (double)(direction.X));
position = new Vector2(position3.X, position3.Y);
scale = new Vector2(scale3.X, scale3.Y);
}
You can then build a transform matrix your leaf sprites like so:
Matrix transform =
Matrix.CreateScale(new Vector3(this.Scale, 1.0f)) *
Matrix.CreateRotationZ(this.Rotation) *
Matrix.CreateTranslation(new Vector3(this.Position, 0));
For your parent sprites, include the origin:
Matrix transform =
Matrix.CreateTranslation(new Vector3(-this.Origin, 0)) *
Matrix.CreateScale(new Vector3(this.Scale, 1.0f)) *
Matrix.CreateRotationZ(this.Rotation) *
Matrix.CreateTranslation(new Vector3(this.position, 0));
Multiply by all the matrices in the stack in reverse order.
Related
I wanted to de skew the image(snapshot of ARSCNView) using the values we get in projectionMatrix, or eulerAngles or transform properties of ARCamera. Or is there any other way to de skew image using any values we get from ARKit? My assumption is that projectionMatrix gives us perspective distortion values.
Can we de skew it without selection four points? Any pointers in right direction will be of great help.
Here is one of my try using values of projectionMatrix and eulerAngles:
func deskewImage(camera: ARCamera) {
let rotationZ = CGFloat(camera.eulerAngles.z)
var transform = CATransform3DIdentity
transform.m34 = CGFloat(camera.projectionMatrix.columns.2.z)
transform = CATransform3DRotate(transform, rotationZ, 0, 0, 1)
self.imageView.layer.transform = transform
}
I was able to de-skew image using CIPerspeciveCorrection using rectangle detection (just like CamScanner apps) but it de-skews only detected rectangle and crops it. I want complete image to be de skewed just like shown in this example without selecting four points.
I am looking for the approach using ARKit as values are already threre.
I'm trying to achieve Augmented Reality with SceneKit.
I got a intrinsic camera matrix and a extrinsic matrix by estimating pose of a marker, using ARuco (OpenCV augmented reality library).
And I set up the SCNCamera's projectionTransform with parameters of the intrinsic matrix (fovy, aspect, zNear, zFar).
Normally in OpenGL, world coordinate relative to camera coordinate is calculated with ModelView but in SceneKit, there is no things such as modelView.
So I calculated inverse matrix of the extrinsic matrix to get the camera coordinate relative to the world coordinate(the marker coordinate).
And I think I've got correct camera's position by the inverse matrix which contains rotation and translate matrix.
However I cannot get camera's rotation from that.
Do you have any ideas?
SceneKit has the same view matrixes that you've come across in OpenGL, they're just a little hidden until you start toying with shaders. A little too hidden IMO.
You seem to have most of this figured out. The projection matrix comes from your camera projectionTransform, and the view matrix comes from the inverse of your camera matrix SCNMatrix4Invert(cameraNode.transform). In my case everything was in world coordinates making my model matrix a simple identity matrix.
The code I ended up using to get the classic model-view-projection matrix was something like...
let projection = camera.projectionTransform()
let view = SCNMatrix4Invert(cameraNode.transform)
let model = SCNMatrix4Identity
let viewProjection = SCNMatrix4Mult(view, projection)
let modelViewProjection = SCNMatrix4Mult(model, viewProjection)
For some reason I found SCNMatrix4Mult(...) took arguments in a different order than I was expecting (eg; opposite to GLKMatrix4Multiply(...)).
I'm still not 100% on this, so would welcome edits/tips. Using this method I was unable to get the SceneKit MVP matrix (as passed to shader) to match up with that calculated by the code above... but it was close enough for what I needed.
#lock's answer looks good with a couple additions:
(1) access SCNNode worldTransform instead of transform in case the cameraNode is animated or parented:
let view = SCNMatrix4Invert(cameraNode.presentationNode.worldTransform)
(2) the code doesn't account for the view's aspect ratio. e.g., assuming a perspective projection, you'll want to do:
perspMatrix.m11 /= viewportAR; //if using Yfov -> adjust Y`
/* or, */
perspMatrix.m22 *= viewportAR; //if using Xfov -> adjust X`
Where, viewportAR = viewport.width / viewport.height
Another way to do it is to have one node with a rendered delegate in the scene, and retrieve SceneKit’s matrices from that delegate (they are passed as options):
FOUNDATION_EXTERN NSString * const SCNModelTransform;
FOUNDATION_EXTERN NSString * const SCNViewTransform;
FOUNDATION_EXTERN NSString * const SCNProjectionTransform;
FOUNDATION_EXTERN NSString * const SCNNormalTransform;
FOUNDATION_EXTERN NSString * const SCNModelViewTransform;
FOUNDATION_EXTERN NSString * const SCNModelViewProjectionTransform;
I am currently using a Project Tango tablet for robotic obstacle avoidance. I want to create a matrix of z-values as they would appear on the Tango screen, so that I can use OpenCV to process the matrix. When I say z-values, I mean the distance each point is from the Tango. However, I don't know how to extract the z-values from the TangoXyzIjData and organize the values into a matrix. This is the code I have so far:
public void action(TangoPoseData poseData, TangoXyzIjData depthData) {
byte[] buffer = new byte[depthData.xyzCount * 3 * 4];
FileInputStream fileStream = new FileInputStream(
depthData.xyzParcelFileDescriptor.getFileDescriptor());
try {
fileStream.read(buffer, depthData.xyzParcelFileDescriptorOffset, buffer.length);
fileStream.close();
} catch (IOException e) {
e.printStackTrace();
}
Mat m = new Mat(depthData.ijRows, depthData.ijCols, CvType.CV_8UC1);
m.put(0, 0, buffer);
}
Does anyone know how to do this? I would really appreciate help.
The short answer is it can't be done, at least not simply. The XYZij struct in the Tango API does not work completely yet. There is no "ij" data. Your retrieval of buffer will work as you have it coded. The contents are a set of X, Y, Z values for measured depth points, roughly 10000+ each callback. Each X, Y, and Z value is of type float, so not CV_8UC1. The problem is that the points are not ordered in any way, so they do not correspond to an "image" or xy raster. They are a random list of depth points. There are ways to get them into some xy order, but it is not straightforward. I have done both of these:
render them to an image, with the depth encoded as color, and pull out the image as pixels
use the model/view/perspective from OpenGL and multiply out the locations of each point and then figure out their screen space location (like OpenGL would during rendering). Sort the points by their xy screen space. Instead of the calculated screen-space depth just keep the Z value from the original buffer.
or
wait until (if) the XYZij struct is fixed so that it returns ij values.
I too wish to use Tango for object avoidance for robotics. I've had some success by simplifying the use case to be only interested in the distance of any object located at the center view of the Tango device.
In Java:
private Double centerCoordinateMax = 0.020;
private TangoXyzIjData xyzIjData;
final FloatBuffer xyz = xyzIjData.xyz;
double cumulativeZ = 0.0;
int numberOfPoints = 0;
for (int i = 0; i < xyzIjData.xyzCount; i += 3) {
float x = xyz.get(i);
float y = xyz.get(i + 1);
if (Math.abs(x) < centerCoordinateMax &&
Math.abs(y) < centerCoordinateMax) {
float z = xyz.get(i + 2);
cumulativeZ += z;
numberOfPoints++;
}
}
Double distanceInMeters;
if (numberOfPoints > 0) {
distanceInMeters = cumulativeZ / numberOfPoints;
} else {
distanceInMeters = null;
}
Said simply this code is taking the average distance of a small square located at the origin of x and y axes.
centerCoordinateMax = 0.020 was determined to work based on observation and testing. The square typically contains 50 points in ideal conditions and fewer when held close to the floor.
I've tested this using version 2 of my tango-caminada application and the depth measuring seems quite accurate. Standing 1/2 meter from a doorway I slid towards the open door and the distance changed form 0.5 meters to 2.5 meters which is the wall at the end of the hallway.
Simulating a robot being navigated I moved the device towards a trash can in the path until 0.5 meters separation and then rotated left until the distance was more than 0.5 meters and proceeded forward. An oversimplified simulation, but the basis for object avoidance using Tango depth perception.
You can do this by using camera intrinsics to convert XY coordinates to normalized values -- see this post - Google Tango: Aligning Depth and Color Frames - it's talking about texture coordinates but it's exactly the same problem
Once normalized, move to screen space x[1280,720] and then the Z coordinate can be used to generate a pixel value for openCV to chew on. You'll need to decide how to color pixels that don't correspond to depth points on your own, and advisedly, before you use the depth information to further colorize pixels.
The main thing is to remember that the raw coordinates returned are already using the basis vectors you want, i.e. you do not want the pose attitude or location
I am trying to applyTorque to a node in my scene. The documentation states:
Each component of the torque vector relates to rotation about the
corresponding axis in the local coordinate system of the SCNNode
object containing the physics body. For example, applying a torque of
{0.0, 0.0, 1.0} causes a node to spin counterclockwise around its
z-axis.
However in my tests it seems that Physics animations do not affect actual position of the object. Therefore, the axis remain static (even though the actual node obviously moves). This results in the torque always being applied from the same direction (wherever the z axes was when the scene was initiated).
I would like to be able to apply torque so that it is always constant in relation to the object (e.g. to cause node to spin counterclockwise around z-axis of the node's presentationNode not the position node had(has?) when the scene was initiated)
SceneKit uses two versions of each node: the model node defines static behavior and the presentation node is what's actually involved in dynamic behavior and used on screen. This division mirrors that used in Core Animation, and enables features like implicit animation (where you can do things like set node.position and have it animate to the new value, without other parts of your code that query node.position having to working about intermediate values during the animation).
Physics operates on the presentation node, but in some cases--like this one--takes input in scene space.
However, the only difference between the presentation node and the scene is in terms of coordinate spaces, so all you need to do is convert your vector from presentation space to scene space. (The root node of the scene shouldn't be getting transformed by physics, actions, or inflight animations, so there's no practical difference between model-scene space and presentation-scene space.) To do that, use one of the coordinate conversion methods SceneKit provides, such as convertPosition:fromNode:.
Here's a Swift playground that illustrates your dilemma:
import Cocoa
import SceneKit
import XCPlayground
// Set up a scene for our tests
let scene = SCNScene()
let view = SCNView(frame: NSRect(x: 0, y: 0, width: 500, height: 500))
view.autoenablesDefaultLighting = true
view.scene = scene
let cameraNode = SCNNode()
cameraNode.camera = SCNCamera()
cameraNode.position = SCNVector3(x: 0, y: 0, z: 5)
scene.rootNode.addChildNode(cameraNode)
XCPShowView("view", view)
// Make a pyramid to test on
let node = SCNNode(geometry: SCNPyramid(width: 1, height: 1, length: 1))
scene.rootNode.addChildNode(node)
node.physicsBody = SCNPhysicsBody.dynamicBody()
scene.physicsWorld.gravity = SCNVector3Zero // Don't fall off screen
// Rotate around the axis that looks into the screen
node.physicsBody?.applyTorque(SCNVector4(x: 0, y: 0, z: 1, w: 0.1), impulse: true)
// Wait a bit, then try to rotate around the y-axis
node.runAction(SCNAction.waitForDuration(10), completionHandler: {
var axis = SCNVector3(x: 0, y: 1, z: 0)
node.physicsBody?.applyTorque(SCNVector4(x: axis.x, y: axis.y, z: axis.z, w: 1), impulse: true)
})
The second rotation effectively spins the pyramid around the screen's y-axis, not the pyramid's y-axis -- the one that goes through the apex of the pyramid. As you noted, it's spinning around what was the pyramid's y-axis as of before the first rotation; i.e. the y-axis of the scene (which is unaffected by physics), not that of the presentation node (that was rotated through physics).
To fix it, insert the following line (after the one that starts with var axis):
axis = scene.rootNode.convertPosition(axis, fromNode: node.presentationNode())
The call to convertPosition:fromNode: says "give me a vector in scene coordinate space that's equivalent to this one in presentation-node space". When you apply a torque around the converted axis, it effectively converts back to the presentation node's space to simulate physics, so you see it spin around the axis you want.
Update: Had some coordinate spaces wrong, but the end result is pretty much the same.
Unfortunately the solution provided by rickster does not work for me :(
Trying to solve this conundrum I have created (what i believe to be) a very sub-standard solution (more a proof of concept). It involves creating (null) objects on the axis i am trying to find, then I use their position to find the vector aligned to the axes.
As I have a fairly complex scene, I am loading it from a COLLADA file. Within that file i have modelled a simple coordinate tripod: three orthogonal cylinders with cones on top (makes it easer to visualise what is going on).
I then constrain this tripod object to the object I am trying to apply torque to. This way I have objects that allow me to retrieve two points on the axes of the presentationNode of the object I am trying to apply torque to. I can then use these two points to determine the vector to apply the torque from.
// calculate orientation vector in the most unimaginative way possible
// retrieve axis tripod objects. We will be using these as guide objects.
// The tripod is constructed as a cylinder called "Xaxis" with a cone at the top.
// All loaded from an external COLLADA file.
SCNNode *XaxisRoot = [scene.rootNode childNodeWithName:#"XAxis" recursively:YES];
SCNNode *XaxisTip = [XaxisRoot childNodeWithName:#"Cone" recursively:NO];
// To devise the vector we will need two points. One is the root of our tripod,
// the other is at the tip. First, we get their positions. As they are constrained
// to the _rotatingNode, presentationNode.position is always the same .position
// because presentationNode returns position in relation to the parent node.
SCNVector3 XaxisRootPos = XaxisRoot.position;
SCNVector3 XaxisTipPos = XaxisTip.position;
// We then convert these two points into _rotatingNode coordinate space. This is
// the coordinate space applyTorque seems to be using.
XaxisRootPos = [_rotatingNode convertPosition:XaxisRootPos fromNode:_rotatingNode.presentationNode];
XaxisTipPos = [_rotatingNode convertPosition:XaxisTipPos fromNode:_rotatingNode.presentationNode];
// Now, we have two *points* in _rotatingNode coordinate space. One is at the center
// of our _rotatingNode, the other is somewhere along it's Xaxis. Subtracting them
// will give us the *vector* aligned to the x axis of our _rotatingNode
GLKVector3 rawXRotationAxes = GLKVector3Subtract(SCNVector3ToGLKVector3(XaxisRootPos), SCNVector3ToGLKVector3(XaxisTipPos));
// we now normalise this vector
GLKVector3 normalisedXRotationAxes = GLKVector3Normalize(rawXRotationAxes);
//finally we are able to apply toque reliably
[_rotatingNode.physicsBody applyTorque:SCNVector4Make(normalisedXRotationAxis.x,normalisedXRotationAxis.y,normalisedXRotationAxis.z, 500) impulse:YES];
As you can probably see, I am quite inexperienced in SceneKit, but even I can see that much easier/optimised solution does exits, but I am unable to find it :(
I recently had this same problem, of how to convert a torque from the local space of the object to the world space required by the applyTorque method. The problem with using the node's convertPosition:toNode and fromNodes methods, is that they are also applying the node's translation to the torque, so this will only work when the node is at 0,0,0. What these methods do is treat the SCNVector3 as if it's a vec4 with a w component of 1.0. We just want to apply the rotation, in other words, we want the w component of the vec4 to be 0. Unlike SceneKit, GLKit gives us 2 options for how we want our vec3s to be multiplied:
GLKMatrix4MultiplyVector3 where
The input vector is treated as it were a 4-component vector with a w-component of 0.0.
and GLKMatrix4MultiplyVector3WithTranslation where
The input vector is treated as it were a 4-component vector with a w-component of 1.0.
What we want here is the former, just the rotation, not the translation.
So, we could roundtrip to GLKit. To convert for instance the local x axis (1,0,0), eg a pitch rotation, to the global axis needed for apply torque, would look like this:
let local = GLKMatrix4MultiplyVector3(SCNMatrix4ToGLKMatrix4(node.presentationNode.worldTransform), GLKVector3(v: (1,0,0)))
node.physicsBody?.applyTorque(SCNVector4(local.x, local.y, local.z, 10), impulse: false)
However, a more Swiftian approach would be to add a * operator for mat4 * vec3 which treats the vec3 like a vec4 with a 0.0 w component. Like this:
func * (left: SCNMatrix4, right: SCNVector3) -> SCNVector3 { //multiply mat4 by vec3 as if w is 0.0
return SCNVector3(
left.m11 * right.x + left.m21 * right.y + left.m31 * right.z,
left.m12 * right.x + left.m22 * right.y + left.m32 * right.z,
left.m13 * right.x + left.m23 * right.y + left.m33 * right.z
)
}
Although this operator makes an assumption about how we want our vec3s to be multiplied, my reasoning here is that as the convertPosition methods already treat w as 1, it would be redundant to have a * operator that also did this.
You could also add a mat4 * SCNVector4 operator that would let the user explicity choose whether or not they want w to be 0 or 1.
So, instead of having to roundtrip from SceneKit to GLKit, we can just write:
let local = node.presentationNode.worldTransform * SCNVector3(1,0,0)
node.physicsBody?.applyTorque(SCNVector4(local.x, local.y, local.z, 10), impulse: false)
You can use this method to apply rotation on multiple axes with one applyTorque call. So say if you have stick input where you want x on the stick to be yaw (local yUp-axis) and y on the stick to be pitch (local x-axis), but with flight-sim style "down to pull back/ up", then you could set it to SCNVector3(input.y, -input.x, 0)
Summary
I'm trying to apply a displacement map (Height map) to a rather simple object (Hexagonal plane) and I'm having some unexpected results. I am using grayscale and as such, I was under the impression my height map should only be affecting the Z values of my mesh. However, the displacement map I've created stretches the mesh across the X and Y planes. Furthermore, it doesn't seem to use the UV mapping I've created that all other textures are successfully applied to.
Model and UV Map
Here are reference images of my hexagonal mesh and its corresponding UV map in Blender.
Diffuse and Displacement Textures
These are the diffuse and displacement map textures I am applying to my mesh through Three.JS.
Renders
When I render the plane without a displacement map, you can see that the hexagonal plane stays within the lines. However, when I add the displacement map it clearly affects the X and Y positions of the vertices rather than affecting only the Z, expanding the plane well over the lines.
Code
Here's the relevant Three.js code:
// Textures
var diffuseTexture = THREE.ImageUtils.loadTexture('diffuse.png', null, loaded);
var displacementTexture = THREE.ImageUtils.loadTexture('displacement.png', null, loaded);
// Terrain Uniform
var terrainShader = THREE.ShaderTerrain["terrain"];
var uniformsTerrain = THREE.UniformsUtils.clone(terrainShader.uniforms);
//uniformsTerrain["tNormal"].value = null;
//uniformsTerrain["uNormalScale"].value = 1;
uniformsTerrain["tDisplacement"].value = displacementTexture;
uniformsTerrain["uDisplacementScale"].value = 1;
uniformsTerrain[ "tDiffuse1" ].value = diffuseTexture;
//uniformsTerrain[ "tDetail" ].value = null;
uniformsTerrain[ "enableDiffuse1" ].value = true;
//uniformsTerrain[ "enableDiffuse2" ].value = true;
//uniformsTerrain[ "enableSpecular" ].value = true;
//uniformsTerrain[ "uDiffuseColor" ].value.setHex(0xcccccc);
//uniformsTerrain[ "uSpecularColor" ].value.setHex(0xff0000);
//uniformsTerrain[ "uAmbientColor" ].value.setHex(0x0000cc);
//uniformsTerrain[ "uShininess" ].value = 3;
//uniformsTerrain[ "uRepeatOverlay" ].value.set(6, 6);
// Terrain Material
var material = new THREE.ShaderMaterial({
uniforms:uniformsTerrain,
vertexShader:terrainShader.vertexShader,
fragmentShader:terrainShader.fragmentShader,
lights:true,
fog:true
});
// Load Tile
var loader = new THREE.JSONLoader();
loader.load('models/hextile.js', function(g) {
//g.computeFaceNormals();
//g.computeVertexNormals();
g.computeTangents();
g.materials[0] = material;
tile = new THREE.Mesh(g, new THREE.MeshFaceMaterial());
scene.add(tile);
});
Hypothesis
I'm currently juggling three possibilities as to why this could be going wrong:
The UV map is not applying to my displacement map.
I've made the displacement map incorrectly.
I've missed a crucial step in the process that would lock the displacement to Z-only.
And of course, secret option #4 which is none of the above and I just really have no idea what I'm doing. Or any mixture of the aforementioned.
Live Example
You can view a live example here.
If anybody with more knowledge on the subject could guide me I'd be very grateful!
Edit 1: As per suggestion, I've commented out computeFaceNormals() and computeVertexNormals(). While it did make a slight improvement, the mesh is still being warped.
In your terrain material, set wireframe = true, and you will be able to see what is happening.
Your code and textures are basically fine. The problem occurs when you compute vertex normals in the loader callback function.
The computed vertex normals for the outer ring of your geometry point somewhat outward. This is most likely because in computeVertexNormals() they are computed by averaging the face normals of each neighboring face, and the face normals of the "sides" of your model (the black part) are averaged into the vertex normal calculation for those vertices that make up the outer ring of the "cap".
As a result, the outer ring of the "cap" expands outward under the displacement map.
EDIT: Sure enough, straight from your model, the vertex normals of the outer ring point outward. The vertex normals for the inner rings are all parallel. Perhaps Blender is using the same logic to generate vertex normals as computeVertexNormals() does.
The problem is how your object is constructed becuase the displacement happens along the normal vector.
the code is here.
https://github.com/mrdoob/three.js/blob/master/examples/js/ShaderTerrain.js#L348-350
"vec3 dv = texture2D( tDisplacement, uvBase ).xyz;",
This takes a the rgb vector of the displacement texture.
"float df = uDisplacementScale * dv.x + uDisplacementBias;",
this takes only red value of the vector becuase uDisplacementScale is normally 1.0 and uDisplacementBias is 0.0.
"vec3 displacedPosition = normal * df + position;",
This displaces the postion along the normal vector.
so to solve you either update the normals or the shader.