I currently work on a little Project, where I render a CubeMap with WebGL and then apply some sounds with the "web audio API" Web Audio API
Since the project is very large, I just try to explain what I am looking for. When I load an audio file, the sounds gets visualized (looks like a cube). The audio listener position is ALWAYS at position 0,0,0. What I have done so far is that I have created "Camera" (gl math library) with lookAt and perspective and when I rotate the camera away from the audio emitting cube, the audio played should sound different.
How am I doing this?
Every Frame I set the the orientation of the PannerNode (panner node set orientation) to the upVector of the camera. Here is the every frame update-Method (for the sound):
update(camera) {
this._upVec = vec3.copy(this._upVec, camera.upVector);
//vec3.transformMat4(this._upVec, this._upVec, camera.vpMatrix);
//vec3.normalize(this._upVec, this._upVec);
this._sound.panner.setOrientation(this._upVec[0], this._upVec[1], this._upVec[2]);
}
And here is the updateViewProjectionMethod-Methof from my Camera class, where I update the Orientation of the listener:
updateViewProjMatrix() {
let gl = Core.mGetGL();
this._frontVector[0] = Math.cos(this._yaw) * Math.cos(this._pitch);
this._frontVector[1] = Math.sin(this._pitch);
this._frontVector[2] = Math.sin(this._yaw) * Math.cos(this._pitch);
vec3.normalize(this._lookAtVector, this._frontVector);
vec3.add(this._lookAtVector, this._lookAtVector, this._positionVector);
mat4.lookAt(this._viewMatrix, this._positionVector, this._lookAtVector, this._upVector);
mat4.perspective(this._projMatrix, this._fov * Math.PI / 180, gl.canvas.clientWidth / gl.canvas.clientHeight, this._nearPlane, this._farPlane);
mat4.multiply(this._vpMatrix, this._projMatrix, this._viewMatrix);
Core.getAudioContext().listener.setOrientation(this._lookAtVector[0], this._lookAtVector[1], this._lookAtVector[2], 0, 1, 0);
}
Is this the right way? I can hear that the sound is different if I rotate the camera, but I am not sure. And do I have to multiply the resulting upVector with the current viewProjectionMatrix?
Related
I'm trying to draw a billboarded quad using SceneKit and ARKit. I have basic billboarding working, however when I roll the camera the billboard also rotates in place. This video shows this in action as I roll the camera to the left (the smily face is the billboard):
Instead I'd like the billboard to still face the camera but keep oriented vertically in the scene, no matter what the camera is doing
Here's how I compute billboarding:
// inside frame update function
struct Vertex {
var position: SIMD3<Float>
var texCoord: SIMD2<Float>
}
let halfSize = Float(0.25)
let cameraNode = sceneView.scene.rootNode.childNodes.first!
let modelTransform = self.scnNode.simdWorldTransform
let viewTransform = cameraNode.simdWorldTransform.inverse
let modelViewTransform = viewTransform * modelTransform
let right = SIMD3<Float>(modelViewTransform[0][0], modelViewTransform[1][0], modelViewTransform[2][0]);
let up = SIMD3<Float>(modelViewTransform[0][1], modelViewTransform[1][1], modelViewTransform[2][1]);
// drawBuffer is a MTL buffer of vertex data
let data = drawBuffer.contents().bindMemory(to: ParticleVertex.self, capacity: 4)
data[0].position = (right + up) * halfSize
data[0].texCoord = SIMD2<Float>(0, 0)
data[1].position = -(right - up) * halfSize
data[1].texCoord = SIMD2<Float>(1, 0)
data[2].position = (right - up) * halfSize
data[2].texCoord = SIMD2<Float>(0, 1)
data[3].position = -(right + up) * halfSize
data[3].texCoord = SIMD2<Float>(1, 1)
Again this gets the billboard facing the camera correctly, however when I roll the camera, the billboard rotates along with it.
What I'd like instead is for the billboard to point towards the camera but keep its orientation in the world. Any suggestions on how to fix this?
Note that my code example is simplified so I can't use SCNBillboardConstraint or anything like that; I need to be able to compute the billboarding myself
Here's the solution I came up with: create a new node that matches the camera's position and rotation, but without any roll:
let tempNode = SCNNode()
tempNode.simdWorldPosition = cameraNode.simdWorldPosition
// This changes the node's pitch and yaw, but not roll
tempNode.simdLook(at: cameraNode.simdConvertPosition(SIMD3<Float>(0, 0, 1), to: nil))
let view = tempNode.simdWorldTransform.inverse
let modelViewTransform = view * node.simdWorldTransform
This keeps the billboard pointing upwards in world space, even as the camera rolls.
I had actually tried doing this earlier by setting tempNode.eulerAngles.z = 0, however that seems to effect the rest of the transform matrix in unexpected ways
There's probably a way to do this without creating a temporary node too but this works well enough for me
I have two play two videos simultaneously on a view .Both videos would be same.
Now, my concern is the video on right is actually to be flipped horizontally along x-axis and then saved in photo library.I have tried googling a lot and found that CGAFFineRotateTransform can help but I am not able to use that in my code.Kindly help me to flip the video on right horizontally while keeping the scale and move same.
Any help or guidance in this direction would be appreciable .Thanks in advance!
Check the difference between video on left and right,video on left is complete but video on right is showing half video only
To just flip (mirror) the video horizontally, use a negative x-value for scaling:
CGAffineTransform scale = CGAffineTransformMakeScale( -1.0, 1.0);
Edit: Regarding your more general question on how to position tracks: For each video track involved:
build a bounding rect based on the original size of the track - this is the source rect
ask yourself where the track should end up, in terms of origin and size in the resulting video - this gives you the destination rect
You can then derive the corresponding affine transform with a function like:
CGAffineTransform NHB_CGAffineTransformMakeRectToRect( CGRect srcRect, CGRect dstRect)
{
CGAffineTransform t = CGAffineTransformIdentity;
t = CGAffineTransformTranslate( t, dstRect.origin.x - fmin( 0., dstRect.size.width), dstRect.origin.y - fmin( 0., dstRect.size.height));
t = CGAffineTransformScale( t, dstRect.size.width / srcRect.size.width, dstRect.size.height / srcRect.size.height);
return t;
}
To mirror, provide a negative size for the corresponding axis (the - fmin(,) part compensates the offset).
Given a video track and assuming, for example, the track should go mirrored to the right half of a 640x480 video, you can get the corresponding transform with:
CGSize srcSize = videoTrack.naturalSize;
CGRect srcRect = CGRectMake( 0, 0, srcSize.width, srcSize.height);
CGRect dstRect = CGRectMake( 320, 0, -320, 480);
CGAffineTransform t = NHB_CGAffineTransformMakeRectToRect(srcRect, dstRect);
Of course this may stretch the video track; to keep the aspect ratio, you'll have to take the source size into account when calculating the destination rect.
Some remarks:
note that in NHB_CGAffineTransformMakeRectToRect I deliberately chose to start with the identity matrix, and then add the required transforms one by one. This way much more complex transforms can be build, including rotation. As I said, try to get a grasp on affine transforms, they're really powerful
AVAssetTracks naturalSize sometimes returns confusing results for some videos with complex SAR/PAR definitions. To make this bullet proof, you'll have to derive the size from the dimensions in the corresponding format descriptions, but that's a whole new topic...
I'm making a "geolocational AR app" in which I use Paint() to draw bitmap on my screen with the use of sensors to do some translation so that my image will only be shown at a specific point.
I've done everything
However, the image shakes while I present it.
I really want the effect as the following videos presents
https://www.youtube.com/watch?v=8U3vWETmk2U
I've implemented low pass filter to ease the situation but it is still not as steady as the images from the video.
This is how I achieve AR movement
float dx = (float) ( (canvas.getWidth()/ horizontalFOV) * (Math.toDegrees(orientation[0])-curBearingTo));
float dy = (float) ( (canvas.getHeight()/ verticalFOV) * Math.toDegrees(orientation[1]));
canvas.translate(0.0f, 0.0f-dy);
canvas.translate(0.0f-dx, 0.0f);
The curBearingTois the result of android location API: BearingTo(location, destination)
And this is how I get Orientation matrix
if (lastAccelerometer != null && lastCompass != null) {
boolean gotRotation = SensorManager.getRotationMatrix(rotation,
identity, lastAccelerometer, lastCompass);
if (gotRotation) {
// remap such that the camera is pointing straight down the Y
// axis
SensorManager.remapCoordinateSystem(rotation,
SensorManager.AXIS_X, SensorManager.AXIS_Z,
cameraRotation);
// orientation vector
SensorManager.getOrientation(cameraRotation, orientation);
}
Any suggestion?
Im doing some practices on XNA, and i created a class that represents a Camera.
My objective is that when the user press some keys make a translation of the camera (not the target) 90 degrees in the X axys (to see an object that i placed in the scene from different angles). By the moment i move the camera in X, Y, and Z without problems.
Actually to set up my camera i use the following lines of code:
public void SetUpCamera()
{
#region ## SET DEFAULTS ##
this.FieldOfViewAngle = 45.0f;
this.AspectRatio =1f;
this.NearPlane = 1.0f;
this.FarPlane = 10000.0f;
#endregion
this.ProjectionMatrix = Matrix.CreatePerspectiveFieldOfView(MathHelper.ToRadians(this.FieldOfViewAngle), 16 / 9, this.NearPlane, this.FarPlane);
this.ViewMatrix = Matrix.CreateLookAt(new Vector3(this.PositionX, this.PositionY, this.PositionZ), new Vector3(this.TargetX, this.TargetY, this.TargetZ), Vector3.Up);
}
I have this method to move the camera:
public void UpdateView()
{
this.ViewMatrix = Matrix.CreateLookAt(new Vector3(this.PositionX, this.PositionY, this.PositionZ), new Vector3(this.TargetX, this.TargetY, this.TargetZ), Vector3.Up);
}
Then in the game (update event handler i have the following code)
if (keyboardstate.IsKeyDown(Keys.NumPad9))
{
this.GameCamera.PositionZ -= 1.0f;
}
if (keyboardstate.IsKeyDown(Keys.NumPad3))
{
this.GameCamera.PositionZ += 1.0f;
}
this.GameCamera.UpdateView();
I would like to know how to make this camera translation of 90 degrees to surround one object that i placed in the screen.
To explain my self better about the camera movement here is a video on youtube that uses the exact movement that im trying to describe (see from 14 second)
http://www.youtube.com/watch?v=19mbKZ0I5u4
Assuming the camera in the video is orbiting the car, here is how you would accomplish that in XNA.
For the sake of readability, we'll just use vectors instead of their individual components. So 'target' means it's a Vector3 that includes TargetX, TargetY, & TargetZ. Same with the camera’s position. You can break X, Y, Z values out into fields and make Vector3s out of them to plug into this code if you want to later, but really it would be best for you to work at vector level instead of component level.
//To orbit the car (target)
cameraPosition = Vector3.Transform(cameraPosition – target, Matrix.CreateRotationY(0.01f)) + target;
this.ViewMatrix = Matrix.CreateLookAt(cameraPosition, target, Vector3.Up);
Since all matrix rotations act about an axis that intersects the world origin, to use a rotation matrix to rotate the camera around the car, the object of rotation has to be shifted such that the target (and thus the rotation axis) is located at the world origin. CameraPosition - target accomplishes that. Now cameraPosition can be rotated a little bit. Once cameraPosition is rotated a little bit, it needs to be sent back to the scene at hand, that's what the '+ target' at the end of the line is for.
The 0.01f can be adjusted to whatever rotation rate suits you.
So I have a XNA application set up. The camera is in first person mode, and the user can move around using the keyboard and reposition the camera target with the mouse. I have been able to load 3D models fine, and they appear on screen no problem. Whenever I try to draw any primitive (textured or not), it does not show up anywhere on the screen, no matter how I position the camera.
In Initialize(), I have:
quad = new Quad(Vector3.Zero, Vector3.UnitZ, Vector3.Up, 2, 2);
quadVertexDecl = new VertexDeclaration(this.GraphicsDevice, VertexPositionNormalTexture.VertexElements);
In LoadContent(), I have:
quadTexture = Content.Load<Texture2D>(#"Textures\brickWall");
quadEffect = new BasicEffect(this.GraphicsDevice, null);
quadEffect.AmbientLightColor = new Vector3(0.8f, 0.8f, 0.8f);
quadEffect.LightingEnabled = true;
quadEffect.World = Matrix.Identity;
quadEffect.View = Matrix.CreateLookAt(cameraPosition, cameraTarget, Vector3.Up);
quadEffect.Projection = this.Projection;
quadEffect.TextureEnabled = true;
quadEffect.Texture = quadTexture;
And in Draw() I have:
this.GraphicsDevice.VertexDeclaration = quadVertexDecl;
quadEffect.Begin();
foreach (EffectPass pass in quadEffect.CurrentTechnique.Passes)
{
pass.Begin();
GraphicsDevice.DrawUserIndexedPrimitives<VertexPositionNormalTexture>(
PrimitiveType.TriangleList,
quad.Vertices, 0, 4,
quad.Indexes, 0, 2);
pass.End();
}
quadEffect.End();
I think I'm doing something wrong in the quadEffect properties, but I'm not quite sure what.
I can't run this code on the computer here at work as I don't have game studio installed. But for reference, check out the 3D audio sample on the creator's club website. They have a "QuadDrawer" in that project which demonstrates how to draw a textured quad in any position in the world. It's a pretty nice solution for what it seems you want to do :-)
http://creators.xna.com/en-US/sample/3daudio