I'm taking images with AVCapturePhotoOutput and then using their JPEG representation as the texture on a SceneKit SCNPlane that is the same aspect ratio as the image:
let image = UIImage(data: dataImage!)
let rectangle = SCNPlane(width:9, height:12)
let rectmaterial = SCNMaterial()
rectmaterial.diffuse.contents = image
rectmaterial.isDoubleSided = true
rectangle.materials = [rectmaterial]
let rectnode = SCNNode(geometry: rectangle)
let pos = sceneSpacePosition(inFrontOf: self.pictCamera, atDistance: 16.5) // 16.5 is arbitrary, but makes the rectangle the same size as the camera
rectnode.position = pos
rectnode.orientation = self.pictCamera.orientation
pictView.scene?.rootNode.addChildNode(rectnode)
sceneSpacePosition is a bit of code that can be found here on SO that maps CoreMotion into SceneKit orientation. It is used to place the rectangle, which does indeed appear at the right location with the right size. All very cool.
The problem is that the image is rotated 90 degrees to the rectangle. So I did the obvious:
rectmaterial.diffuse.contentsTransform = SCNMatrix4MakeRotation(Float.pi / 2, 0, 0, 1)
This does not work property; the resulting image is unrecognizable. It appears that one small part of the image has been stretched to a huge size. I thought it might be the axis, but I tried all three with the same result.
Any ideas?
You are rotating on the upper left corner as suggested by Alain T.
If you move your image down, you may get the rotation you were expecting.
Try this:
let translation = SCNMatrix4MakeTranslation(0, -1, 0)
let rotation = SCNMatrix4MakeRotation(Float.pi / 2, 0, 0, 1)
let transform = SCNMatrix4Mult(translation, rotation)
rectmaterial.diffuse.contentsTransform = transform
Related
I'm trying to draw a billboarded quad using SceneKit and ARKit. I have basic billboarding working, however when I roll the camera the billboard also rotates in place. This video shows this in action as I roll the camera to the left (the smily face is the billboard):
Instead I'd like the billboard to still face the camera but keep oriented vertically in the scene, no matter what the camera is doing
Here's how I compute billboarding:
// inside frame update function
struct Vertex {
var position: SIMD3<Float>
var texCoord: SIMD2<Float>
}
let halfSize = Float(0.25)
let cameraNode = sceneView.scene.rootNode.childNodes.first!
let modelTransform = self.scnNode.simdWorldTransform
let viewTransform = cameraNode.simdWorldTransform.inverse
let modelViewTransform = viewTransform * modelTransform
let right = SIMD3<Float>(modelViewTransform[0][0], modelViewTransform[1][0], modelViewTransform[2][0]);
let up = SIMD3<Float>(modelViewTransform[0][1], modelViewTransform[1][1], modelViewTransform[2][1]);
// drawBuffer is a MTL buffer of vertex data
let data = drawBuffer.contents().bindMemory(to: ParticleVertex.self, capacity: 4)
data[0].position = (right + up) * halfSize
data[0].texCoord = SIMD2<Float>(0, 0)
data[1].position = -(right - up) * halfSize
data[1].texCoord = SIMD2<Float>(1, 0)
data[2].position = (right - up) * halfSize
data[2].texCoord = SIMD2<Float>(0, 1)
data[3].position = -(right + up) * halfSize
data[3].texCoord = SIMD2<Float>(1, 1)
Again this gets the billboard facing the camera correctly, however when I roll the camera, the billboard rotates along with it.
What I'd like instead is for the billboard to point towards the camera but keep its orientation in the world. Any suggestions on how to fix this?
Note that my code example is simplified so I can't use SCNBillboardConstraint or anything like that; I need to be able to compute the billboarding myself
Here's the solution I came up with: create a new node that matches the camera's position and rotation, but without any roll:
let tempNode = SCNNode()
tempNode.simdWorldPosition = cameraNode.simdWorldPosition
// This changes the node's pitch and yaw, but not roll
tempNode.simdLook(at: cameraNode.simdConvertPosition(SIMD3<Float>(0, 0, 1), to: nil))
let view = tempNode.simdWorldTransform.inverse
let modelViewTransform = view * node.simdWorldTransform
This keeps the billboard pointing upwards in world space, even as the camera rolls.
I had actually tried doing this earlier by setting tempNode.eulerAngles.z = 0, however that seems to effect the rest of the transform matrix in unexpected ways
There's probably a way to do this without creating a temporary node too but this works well enough for me
I'm trying to create something like canvas in SceneKit using an SCNBox, with a UIImage "wrapped" around from one surface and onto the four others adjacent to it.
The only way I can currently think to do this would be to chop up the UIImage into five separate images and put those onto the sides as materials, but I'm sure there must be an easier way.
Can anyone steer me in the right direction here? The box will have a separate texture/material on the side opposite the "front".
The easiest way would probably be to create a custom geometry with matching texture coordinates using +geometryWithSources:elements:
You can use contentsTransform property from SCNMaterialProperty, for adjust needed texture coordinates from your image to SCNBox
Some explanations with simplified example:
Lets suppose that you are using cube and you have a texture like this
By dividing it into rectangles, you will have
You want to skip rectangles 1, 3, 7, 9 and cover your cube with this texture.
For this just normalize the size of side from your SCNBox between 0 and 1, and use it to set the scale and transform in contentsTransform matrix.
I have a cube with equal sides in my example - so it will be the third part of the whole texture. For taking the 5 rectangle from the texture
let normalizedWidth = 1/3
let normilizedHeight = 1/3
let xOffset = 1 //skip 1,4,7 line
let yOffset = 1 //skip 1,2,3 line
let sideMaterial = SCNMaterial()
sideMaterial.diffuse.contents = textureImage
let scaleMatrix = SCNMatrix4MakeScale(normalizedWidth, normilizedHeight, 0.0)
sideMaterial.diffuse.contentsTransform = SCNMatrix4Translate(scaleMatrix,
normalizedWidth * xOffset, yOffset * yOffset, 0.0)
You can fill 5 sides with configured materials, and the last on (on the back) just with the color and set them to materials property of your SCNBox.
In the result you will have
I am using SpriteKit to draw a graph (with the ability to zoom in and pan around).
When I use an SKCropNode to crop the grid of my graph it doesn't crop the desired area. It crops less, no matter if I use a rectangular SKShapeNode or a SKSpriteNode (with image) as .maskNode.
Here is my code:
//GRID
let grid = SKCropNode()
graphViewModel.graphScene.addChild(grid)
let ratio:CGFloat = 1000 / 500
let width = (graphViewModel.sceneSize.width*0.95)
let newSize = CGSize(width: width, height: width/ratio)
let origin = CGPoint(x: -newSize.width/2.0, y: 0.0)
let rectangularMask = SKShapeNode(rect: CGRect(origin: origin, size: newSize))
rectangularMask.fillColor = UIColor.lightGray
rectangularMask.zPosition = -10.0 //So it appears behind the grid, doesn't affect the cropping
grid.maskNode = rectangularMask
graphViewModel.graphScene.addChild(rectangularMask)
Here are two screenshots to illustrate what I mean:
This is the graph with its grid not being cropped.
This is the graph with the maskNode set.
The lightGray Area is the actual rectangularNode and the grid is being cut off a lot less than it ought to be.
My scene is scaled so I can zoom in without pixelating.
When I disable zooming (setting the scene's size to the view's size) then the bug disappears. Unfortunately I need zooming without any pixel artefacts.
Maybe someone has an idea how to fix this issue. It might also be a SpriteKit Bug.
Given a CGRect, I want to use GPUImage to crop a video. For example, if the rect is (0, 0, 50, 50), the video would be cropped at (0,0) with a length of 50 on each side.
What's throwing me is that GPUImageCropFilter doesn't take a rectangle, rather a normalized crop region with values ranging from 0 to 1. My intuition was to to this:
let assetSize = CGSizeApplyAffineTransform(videoTrack.naturalSize, videoTrack.preferredTransform)
let cropRect = CGRect(x: frame.minX/assetSize.width,
y: frame.minY/assetSize.height,
width: frame.width/assetSize.width,
height: frame.height/assetSize.height)
to calculate the crop region based on the size of the incoming asset. Then:
// Filter
let cropFilter = GPUImageCropFilter(cropRegion: cropRect)
let url = NSURL(fileURLWithPath: "\(NSTemporaryDirectory())\(String.random()).mp4")
let movieWriter = GPUImageMovieWriter(movieURL: url, size: assetSize)
movieWriter.encodingLiveVideo = false
movieWriter.shouldPassthroughAudio = false
// add targets
movieFile.addTarget(cropFilter)
cropFilter.addTarget(movieWriter)
cropFilter.forceProcessingAtSize(frame.size)
cropFilter.setInputRotation(kGPUImageRotateRight, atIndex: 0)
What should the movie writer size be? Shouldn't it be the size of the frame I want to crop with? And should I be using forceProcessingAtSize with the size value of my crop frame?
A complete code example would be great; I've been trying for hours and I can't seem to get the section of the video that I want.
FINAL:
if let videoTrack = self.asset.tracks.first {
let movieFile = GPUImageMovie(asset: self.asset)
let transformedRegion = CGRectApplyAffineTransform(region, videoTrack.preferredTransform)
// Filters
let cropFilter = GPUImageCropFilter(cropRegion: transformedRegion)
let url = NSURL(fileURLWithPath: "\(NSTemporaryDirectory())\(String.random()).mp4")
let renderSize = CGSizeApplyAffineTransform(videoTrack.naturalSize, CGAffineTransformMakeScale(transformedRegion.width, transformedRegion.height))
let movieWriter = GPUImageMovieWriter(movieURL: url, size: renderSize)
movieWriter.transform = videoTrack.preferredTransform
movieWriter.encodingLiveVideo = false
movieWriter.shouldPassthroughAudio = false
// add targets
// http://stackoverflow.com/questions/37041231/gpuimage-crop-to-cgrect-and-rotate
movieFile.addTarget(cropFilter)
cropFilter.addTarget(movieWriter)
movieWriter.completionBlock = {
observer.sendNext(url)
observer.sendCompleted()
}
movieWriter.failureBlock = { _ in
observer.sendFailed(.VideoCropFailed)
}
disposable.addDisposable {
cropFilter.removeTarget(movieWriter)
movieWriter.finishRecording()
}
movieWriter.startRecording()
movieFile.startProcessing()
}
As you note, the GPUImageCropFilter takes in a rectangle in normalized coordinates. You're on the right track, in that you just need to convert your CGRect in pixels to normalized coordinates by dividing the X components (origin.x and size.width) by the width of the image and the Y components by the height.
You don't need to use forceProcessingAtSize(), because the crop will automatically output an image of the appropriate cropped size. The movie writer's size should be matched to this cropped size, which you should know from your original CGRect.
The one complication you introduce is the rotation. If you need to apply a rotation in addition to your crop, you might want to check and make sure that you don't need to swap your X and Y for your crop region. This should be apparent in the output if the two need to be swapped.
There were some bugs with applying rotation at the same time as a crop a while ago, and I can't remember if I fixed all those. If I didn't, you could insert a dummy filter (gamma or brightness set to default values) before or after the crop and apply the rotation at that stage.
I'm trying to make Jigsaw puzzle game in SpriteKit. To make things easier I using 9x9 squared tiles board. On each tile is one childNode with piece of image from it area.
But here's starts my problem. Piece of jigsaw puzzle isn't perfect square, and when I apply SKTexture to node it just place from anchorPoint = {0,0}. And result isn't pretty, actually its terrible.
https://www.dropbox.com/s/2di30hk5evdd5fr/IMG_0086.jpg?dl=0
I managed to fix those tiles with right and top "hooks", but left and bottom side doesn't care about anything.
var sprite = SKSpriteNode()
let originSize = frame.size
let textureSize = texture.size()
sprite.size = originSize
sprite.texture = texture
sprite.size = texture.size()
let x = (textureSize.width - originSize.width)
let widthRate = x / textureSize.width
let y = (textureSize.height - originSize.height)
let heightRate = y / textureSize.height
sprite.anchorPoint = CGPoint(x: 0.5 - (widthRate * 0.5), y: 0.5 - (heightRate * 0.5))
sprite.position = CGPoint(x: frame.width * 0.5, y: frame.height * 0.5)
addChild(sprite)
Can you give me some advice?
I don't see a way you can get placement right without knowing more about the piece texture you are using because they will all be different. Like if the piece has a nob on any of the sides and the width width/height the nob will add to the texture. Hard to tell in the pic but even if it doesn't have a nob and instead has an inset it might add varying sizes.
Without knowing anything about how the texture is created I am not able to offer help on that. But I do believe the issue starts with that. If it were me I would create a square texture with additional alpha to center the piece correctly. So the center of that texture would always be placed in the center of a square on the grid.
With all that being said I do know that adding that texture to a node and then adding that node to a SKNode will make your placement go smoother with the way you currently have it. The trick will then only be placing that textured piece correctly within the empty SKNode.
For example...
let piece = SKSpriteNode()
let texturedPiece = SKSpriteNode(texture: texture)
//positioning
//offset x needs to be calculated with additional info about the texture
//for example it has just a nob on the right
let offsetX : CGFloat = -nobWidth/2
//offset y needs to be calculated with additional info about the texture
//for example it has a nob on the top and bottom
let offsetY : CGFloat = 0.0
texturedPiece.position = CGPointMake(offsetX, offsetY)
piece.addChild(texturedPiece)
let squareWidth = size.width/2
//Now that the textured piece is placed correctly within a parent
//placing the parent is super easy and consistent without messing
//with anchor points. This will also make rotations nice.
piece.position = CGPoint(x: squareWidth/2, y: squareWidth/2)
addChild(piece)
Hopefully that makes sense and didn't confuse things further.