Looking through different web sites and analyzing different resources I found out that for playing 360 videos on the iPhone you should use 3-d party lib (Panorama). But I'm really interested in how it is possible to do it by your own. Because standard iOS elements does not support such functionality.
Please give some advices about approaches that should i use to create own player for 360 videos.
You can do it using scenekit, without any external libraries or dependencies.
Create a SCNScene and map your camera to the device movements. The cameras are a bit offsetted as so to map one per eye and create a 3D stereoscopic effect.
override func viewDidLoad() {
super.viewDidLoad()
leftSceneView?.backgroundColor = UIColor.blackColor()
rightSceneView?.backgroundColor = UIColor.whiteColor()
// Create Scene
scene = SCNScene()
leftSceneView?.scene = scene
rightSceneView?.scene = scene
// Create cameras
let camX = 0.0 as Float
let camY = 0.0 as Float
let camZ = 0.0 as Float
let zFar = 50.0
let leftCamera = SCNCamera()
let rightCamera = SCNCamera()
leftCamera.zFar = zFar
rightCamera.zFar = zFar
let leftCameraNode = SCNNode()
leftCameraNode.camera = leftCamera
leftCameraNode.position = SCNVector3(x: camX - 0.5, y: camY, z: camZ)
let rightCameraNode = SCNNode()
rightCameraNode.camera = rightCamera
rightCameraNode.position = SCNVector3(x: camX + 0.5, y: camY, z: camZ)
camerasNode = SCNNode()
camerasNode!.position = SCNVector3(x: camX, y:camY, z:camZ)
camerasNode!.addChildNode(leftCameraNode)
camerasNode!.addChildNode(rightCameraNode)
camerasNode!.eulerAngles = SCNVector3Make(degreesToRadians(-90.0), 0, 0)
cameraRollNode = SCNNode()
cameraRollNode!.addChildNode(camerasNode!)
cameraPitchNode = SCNNode()
cameraPitchNode!.addChildNode(cameraRollNode!)
cameraYawNode = SCNNode()
cameraYawNode!.addChildNode(cameraPitchNode!)
scene!.rootNode.addChildNode(cameraYawNode!)
leftSceneView?.pointOfView = leftCameraNode
rightSceneView?.pointOfView = rightCameraNode
// Respond to user head movement. Refreshes the position of the camera 60 times per second.
motionManager = CMMotionManager()
motionManager?.deviceMotionUpdateInterval = 1.0 / 60.0
motionManager?.startDeviceMotionUpdatesUsingReferenceFrame(CMAttitudeReferenceFrame.XArbitraryZVertical)
leftSceneView?.delegate = self
leftSceneView?.playing = true
rightSceneView?.playing = true
}
Update the camera position in the sceneRenderer:
func renderer(aRenderer: SCNSceneRenderer, updateAtTime time: NSTimeInterval){
// Render the scene
dispatch_async(dispatch_get_main_queue()) { () -> Void in
if let mm = self.motionManager, let motion = mm.deviceMotion {
let currentAttitude = motion.attitude
var orientationMultiplier = 1.0
if(UIApplication.sharedApplication().statusBarOrientation == UIInterfaceOrientation.LandscapeRight){ orientationMultiplier = -1.0}
self.cameraRollNode!.eulerAngles.x = Float(currentAttitude.roll * orientationMultiplier)
self.cameraPitchNode!.eulerAngles.z = Float(currentAttitude.pitch)
self.cameraYawNode!.eulerAngles.y = Float(currentAttitude.yaw)
}
}
}
Here is some code to add a SCNSphere displaying an AVPlayer.
func play(){
//let fileURL: NSURL? = NSURL(string: "http://www.kolor.com/360-videos-files/noa-neal-graffiti-360-music-video-full-hd.mp4")
let fileURL: NSURL? = NSURL.fileURLWithPath(NSBundle.mainBundle().pathForResource("vr", ofType: "mp4")!)
if (fileURL != nil){
videoSpriteKitNode = SKVideoNode(AVPlayer: AVPlayer(URL: fileURL!))
videoNode = SCNNode()
videoNode!.geometry = SCNSphere(radius: 30)
let spriteKitScene = SKScene(size: CGSize(width: 2500, height: 2500))
spriteKitScene.scaleMode = .AspectFit
videoSpriteKitNode!.position = CGPoint(x: spriteKitScene.size.width / 2.0, y: spriteKitScene.size.height / 2.0)
videoSpriteKitNode!.size = spriteKitScene.size
spriteKitScene.addChild(videoSpriteKitNode!)
videoNode!.geometry?.firstMaterial?.diffuse.contents = spriteKitScene
videoNode!.geometry?.firstMaterial?.doubleSided = true
// Flip video upside down, so that it's shown in the right position
var transform = SCNMatrix4MakeRotation(Float(M_PI), 0.0, 0.0, 1.0)
transform = SCNMatrix4Translate(transform, 1.0, 1.0, 0.0)
videoNode!.pivot = SCNMatrix4MakeRotation(Float(M_PI_2), 0.0, -1.0, 0.0)
videoNode!.geometry?.firstMaterial?.diffuse.contentsTransform = transform
videoNode!.position = SCNVector3(x: 0, y: 0, z: 0)
scene!.rootNode.addChildNode(videoNode!)
videoSpriteKitNode!.play()
playingVideo = true
}
}
I've put together a project on github to show how, with instructions that should be clear !
Works in VR too with a google cardboard.
https://github.com/Aralekk/simple360player_iOS
Apart from Aralekk's repo, I've found hanton's repo useful when creating my own video 360° player.
Here is the link to my repo and here is related blogpost.
You can use this fame work for 360 degree player pod 'NYT360Video' in Objective C and Swift their is a
For Objective default Example is provided
For Swift use the same Frame work
/// For 360 degree view
// just import the frame work
import NYT360Video
// declare the nyt360VC global fro the view controller
var nyt360VC: NYT360ViewController!
/// Player implementation
let videoURL = Bundle.main.url(forResource: "360Video", withExtension: "mp4")!
let player = AVPlayer(url: videoURL)
let playerLayer = AVPlayerLayer(player: player)
playerLayer.frame = self.view.bounds
self.view.layer.addSublayer(playerLayer)
player.play()
/// 360 Degree VC implementation along with manager
let manager: NYT360MotionManagement = NYT360MotionManager.shared()
self.nyt360VC = NYT360ViewController.init(avPlayer: player2,
motionManager: manager)
// Adding the 360 degree view controller to view
self.addChildViewController(nyt360VC)
self.view.addSubview(self.nyt360VC.view)
self.nyt360VC.didMove(toParentViewController: self)
I implemented a similar solution using SceneKit for iOS here: ThreeSixtyPlayer.
In my opinion this code is a bit simpler and more scalable than other solutions out there. It is however just the basics (no stereoscopic playback, only supports sphere geometry, doesn't yet support cardboard, etc.).
Related
I'm trying to show a video in augmented reality using Vuforia - but for the sake of this question, just showing the scene and video would be fine.
What's expected:
Show the video (playing) at the correct speed for video and audio and have them both in sync.
What's happening:
Audio plays at correct speed. Video plays at a seriously fast speed - like 10x.
Tried:
I've tried changing the rate - it's ignored completely.
I've tried using different ways (AVPlayer, AVPlayerLayer,
SKVideoNode(withURL)) of putting the video into the scene - all
suffer from hyperactive-video-syndrome
I've tried other file formats - nope
I've tried local files and URL - no dice
I've tried throwing my laptop at a wall - it made the video go away
Code to return a the scene with the video:
private func createVideoScene(with view: VuforiaEAGLView) -> SCNScene {
// create the asset & player and grab the dimensions
let asset = AVAsset(URL: NSURL(string: "https://inm-baobab-prod-eu-west-1.s3.amazonaws.com/public/inm/media/video/2016/09/02/61537094SansSouciGirlsSchool.mp4")!)
let size = asset.tracksWithMediaType(AVMediaTypeVideo)[0].naturalSize
let player = AVPlayer(playerItem: AVPlayerItem(asset: asset))
let videoNode = SKVideoNode(AVPlayer: player)
videoNode.size = size
videoNode.position = CGPoint(x: size.width * 0.5, y: size.height * 0.5)
let videoScene = SKScene(size: size)
videoScene.addChild(videoNode)
let videoWrapperNode = SCNNode(geometry: SCNPlane(width: 10, height: 8))
videoWrapperNode.position = SCNVector3(x: 0, y: 0, z: 0)
videoWrapperNode.geometry?.firstMaterial?.diffuse.contents = videoScene
videoWrapperNode.geometry?.firstMaterial?.doubleSided = true
videoWrapperNode.scale.y = -1
videoWrapperNode.name = "video"
let scene = SCNScene()
scene.rootNode.addChildNode(videoWrapperNode)
return scene
}
Thank you
PS. Help in Objective-C is also welcome :)
I need to focus to certain node, for example a pyramid. Then apply distance to the camera, then move the camera based on user click. My approach is like this :
import SceneKit
class GameViewController: UIViewController {
let scene = SCNScene()
override func viewDidLoad() {
super.viewDidLoad()
let camera = SCNCamera()
camera.usesOrthographicProjection = true
camera.orthographicScale = 4
camera.zNear = 1
camera.zFar = 100
let cameraNode = SCNNode()
cameraNode.position = SCNVector3(x: 0, y: 0, z: 6)
cameraNode.camera = camera
let cameraOrbit = SCNNode()
cameraOrbit.name = "orbit"
cameraOrbit.addChildNode(cameraNode)
scene.rootNode.addChildNode(cameraOrbit)
let Py = SCNPyramid(width: 2, height: 3, length: 2)
Py.firstMaterial?.diffuse.contents = UIColor.purple()
let P = SCNNode(geometry: Py)
P.position = SCNVector3(x:0,y:0,z:2) //see the note
scene.rootNode.addChildNode(P)
/* N O T E :
the position of the pyramid must not be changed
as my intention is to rotate the camera
not the pyramid node
I repeat, I don't want to rotate the pyramid
*/
let scnView = self.view as! SCNView
scnView.scene = scene
scnView.allowsCameraControl = false
scnView.backgroundColor = UIColor.black()
// user rotates the camera by tapping
let tapGesture = UITapGestureRecognizer(target: self, action: #selector(handleTap(_:)))
scnView.addGestureRecognizer(tapGesture)
}
//the function which does camera rotation :
func handleTap(_ gestureRecognize: UIGestureRecognizer) {
//I guess the solution is around here
//like modify the cameraOrbit.position ?
//or cameraNode.position ?
//tried both but doesn't work
//or I used them wrong
let cameraOrbit = scene.rootNode.childNode(withName: "orbit", recursively: true)!
SCNTransaction.begin()
SCNTransaction.animationDuration = 2
cameraOrbit.eulerAngles.z += Float(M_PI_2) //see below
SCNTransaction.commit()
/*
I realise that by performing rotation on the camera
(the camera position is unchanged) works for z rotation,
but this is not what I want. What I want is a solution,
which also works for eulerAngles.x and eulerAngles.y.
I used eulerAngles.z as example since it's easier to observe.
I guess the true solution involves moving the camera
with specific trajectory, not only rotation of "anchored" camera.
*/
}
//...
}
The result is :
What I want to achieve is to make the rotation relative to its centre :
My question is, how to adjust the pivot so I can achieve the rotation relative to the pyramid's centre?
Note : I don't want to rotate the pyramid.
I found the solution, but I can't explain why. Please comment if you can explain.
The camera orbit position must be set to match the object's location (in this case, the pyramid). So
cameraOrbit.position = P.position
Then here comes the mystery solution, add cameraOrbit.position.y by half of pi :
cameraOrbit.position.y += Float(M_PI_2)
//therefore we have the final cameraOrbit.position as (0, pi/2, 2)
Tested and works perfectly for all cameraOrbit.eulerAngles.
But I don't have a clue why it works. If pi/2 is coming from some projection thingy, then why it's tucked on y only? I mean, when I do any of the cameraOrbit.eulerAngles, I don't need to assign this pi/2 to either x or z.
Here it is the complete code
import SceneKit
class GameViewController: UIViewController {
let scene = SCNScene()
override func viewDidLoad() {
super.viewDidLoad()
let camera = SCNCamera()
camera.usesOrthographicProjection = true
camera.orthographicScale = 4
camera.zNear = 1
camera.zFar = 100
let cameraNode = SCNNode()
cameraNode.position = SCNVector3(x: 0, y: 0, z: 6)
cameraNode.camera = camera
let cameraOrbit = SCNNode()
cameraOrbit.name = "orbit"
cameraOrbit.addChildNode(cameraNode)
scene.rootNode.addChildNode(cameraOrbit)
let Py = SCNPyramid(width: 2, height: 3, length: 2)
Py.firstMaterial?.diffuse.contents = UIColor.purple()
let P = SCNNode(geometry: Py)
P.position = SCNVector3(x:0,y:0,z:2)
scene.rootNode.addChildNode(P)
// S O L U T I O N :
cameraOrbit.position = P.position
cameraOrbit.position.y += Float(M_PI_2)
//therefore we have the final cameraOrbit.position as (0, pi/2, 2)
let scnView = self.view as! SCNView
scnView.scene = scene
scnView.allowsCameraControl = false
scnView.backgroundColor = UIColor.black()
// user rotates the camera by tapping
let tapGesture = UITapGestureRecognizer(target: self, action: #selector(handleTap(_:)))
scnView.addGestureRecognizer(tapGesture)
}
//the function which does camera rotation :
func handleTap(_ gestureRecognize: UIGestureRecognizer) {
//I was wrong, the solution is not here
let cameraOrbit = scene.rootNode.childNode(withName: "orbit", recursively: true)!
SCNTransaction.begin()
SCNTransaction.animationDuration = 2
cameraOrbit.eulerAngles.z += Float(M_PI_2) //works also for x and y
SCNTransaction.commit()
}
...
}
I'm trying to make a 360 degree video player, by projecting a video on a sphere & then turning the normals inside out & placing the camera inside the sphere, however I'm getting a black screen when I try to play the video on the sphere, but I can hear the audio play.
override func viewDidLoad() {
super.viewDidLoad()
//create sphere
makeSphere();
//create user-interface
resizeScreen();
}
ViewDidLoad ^ I call resizeScreen(); which makes the video player 'adapt' to the screen.
//set variables
var player : AVPlayer? = nil;
var playerLayer : AVPlayerLayer? = nil;
var asset : AVAsset? = nil;
var playerItem : AVPlayerItem? = nil;
var timer = NSTimer();
//start functions
func resizeScreen() {
//Get device screen width & height
var screenSize: CGRect = UIScreen.mainScreen().bounds;
//set screen width and height in variables
var screenWidth = screenSize.width;
var screenHeight = screenSize.height;
//change size of video player
playerLayer?.frame.size.height = screenHeight * 0.75;
playerLayer?.frame.size.width = screenWidth * 1.0;
//change size of button
PlayVid.frame.size.height = screenHeight * 0.75;
PlayVid.frame.size.width = screenWidth * 1;
//set position of text label
TemporaryURL.center.x = screenWidth * 0.5;
TemporaryURL.center.y = screenHeight * 0.9
}
resizeScreen();, as mentioned above makes the video player and button adapt to the screen size.
func makeSphere() {
let sceneView = SCNView(frame: self.view.frame);
self.view.addSubview(sceneView);
//Get device screen width & height
var screenSize: CGRect = UIScreen.mainScreen().bounds;
//set screen width and height in variables
var screenWidth = screenSize.width;
var screenHeight = screenSize.height;
//set scene's height, width and x-position
sceneView.frame.size.height = screenHeight * 0.75;
sceneView.frame.size.width = screenWidth * 1;
sceneView.center.x = screenWidth * 0.5;
//create scene
let scene = SCNScene();
sceneView.scene = scene;
//camera positioning
let camera = SCNCamera();
let cameraNode = SCNNode();
cameraNode.camera = camera;
cameraNode.position = SCNVector3(x: -3.0, y: 3.0, z: 3.0);
//lighting
let light = SCNLight();
light.type = SCNLightTypeOmni;
let lightNode = SCNNode();
lightNode.light = light;
lightNode.position = SCNVector3(x: 0, y: 0, z: 0);
//create a sphere
let sphereGeometry = SCNSphere(radius: 2.5);
//create sphere node
let sphereNode = SCNNode(geometry: sphereGeometry);
//create constraint for sphere
let constraint = SCNLookAtConstraint(target: sphereNode);
//constraint.gimbalLockEnabled = true;
cameraNode.constraints = [constraint];
//set nodes
//scene.rootNode.addChildNode(lightNode);
scene.rootNode.addChildNode(cameraNode);
scene.rootNode.addChildNode(sphereNode);
inlineVideo();
player!.play();
//make variable
let videoMaterial = SCNMaterial();
var imgURL = NSURL(string: "http://imgurl");
var videoURLWithPath = "http://videourl";
var videoURL = NSURL(string: videoURLWithPath);
//apply texture to variable
videoMaterial.diffuse.contents = AVPlayerLayer(player: self.player);
videoMaterial.specular.contents = UIColor.redColor();
videoMaterial.shininess = 1.0;
//set texture on object of name sphereGeomtetry
sphereGeometry.materials = [videoMaterial];
}
makeSphere(); this creates the sphere & initialises the video for said Sphere.
func inlineVideo(){
//play inline video
//insert url of video into textfield
var PasteBoard = UIPasteboard.generalPasteboard().string;
TemporaryURL.text = PasteBoard;
//get path of video
//var videoURLWithPath = PasteBoard;
var videoURLWithPath = "http://videourl";
var videoURL = NSURL(string: videoURLWithPath);
if (videoURL == nil){
videoURL = NSURL(string: "http://videourl");
}else{
//do nothing
}
//get video url & set it
asset = AVAsset(URL: videoURL!) as AVAsset;
playerItem = AVPlayerItem(asset:asset!);
//set target for video
player = AVPlayer(playerItem: self.playerItem!);
playerLayer = AVPlayerLayer(player: self.player);
}
This is the video player code I use to initialise the regular video, I tried to bind PlayerLayer to the texture of the sphere, but all I'm getting is a black screen, even though im hearing sound, why isn't the video projecting on the sphere? I think that something like this should work?
Check the following links:
https://github.com/nomtek/spherical_video_player_ios
https://github.com/Aralekk/simple360player_iOS
https://github.com/hanton/HTY360Player
There are 3 different approaches (e.g. OpenGL/SceneKit, shader/fixed-function pipeline, swift/obj-c, RGB/YCbCr) so everyone should find any handy piece of code.
You need to use SKVideoNode to display a video on a sphere, see SKVideoNode only on a small part of SCNSphere - still minor issues, will update when it's fixed.
Set up your player and start playing before you assign it as material to make it work.
var player = AVPlayer(url: yourURL)
player.play()
node.geo.material.emissive.contents = player
I created a floor in a scene and loaded a collada file(.dae) into my scene and tried to create a plane below that model. But I have problems like below.
This problem generates only when custom camera is used. Scene renders properly when system camera is used i.e if no custom camera is added
My code is as below:
import UIKit import SceneKit
class TestVC: UIViewController, SCNSceneRendererDelegate {
var cameraNode: SCNNode!
var modelNode: SCNNode!
var scnView: SCNView!
var camNode: SCNNode!
var cameraPosition: SCNVector3!
override func viewDidLoad() {
super.viewDidLoad()
scnView = SCNView(frame: self.view.frame)
scnView.delegate = self
view.addSubview(scnView)
scnView.delegate = self
let scene = SCNScene()
scnView.scene = scene
//camera
let camera = SCNCamera()
cameraNode = SCNNode()
cameraNode.camera = camera
camera.zFar = 1000
cameraPosition = SCNVector3(0, 100, 150)
cameraNode.position = cameraPosition
scene.rootNode.addChildNode(cameraNode)
//floor
let floor = SCNFloor()
floor.reflectivity = 0
let floorNode = SCNNode(geometry: floor)
let firstmaterial = SCNMaterial()
firstmaterial.diffuse.contents = UIImage(named: "art.scnassets/grass.jpg")
firstmaterial.diffuse.wrapS = SCNWrapMode.Repeat
firstmaterial.diffuse.wrapT = SCNWrapMode.Repeat
floor.materials = [firstmaterial]
scene.rootNode.addChildNode(floorNode)
let light = SCNLight()
let lightNode = SCNNode()
lightNode.light = light
light.type = SCNLightTypeAmbient
light.color = UIColor.darkGrayColor()
scene.rootNode.addChildNode(lightNode)
let light1 = SCNLight()
let light1Node = SCNNode()
light1Node.light = light1
light1Node.position = SCNVector3(0, 200, -100)
light1.type = SCNLightTypeOmni
scene.rootNode.addChildNode(light1Node)
let modelScene = SCNScene(named: "Barcelona Chair.dae")
let solNode = modelScene?.rootNode
solNode?.eulerAngles = SCNVector3(GLKMathDegreesToRadians(-90), 0, 0)
scene.rootNode.addChildNode(solNode!)
scnView.allowsCameraControl = true
let plane = SCNPlane(width: 100, height: 100)
let planeNode = SCNNode(geometry: plane)
planeNode.pivot = SCNMatrix4MakeRotation(Float(M_PI_2), 1, 0, 0)//(CGFloat(M_PI_2), 1, 0, 0)
planeNode.position = SCNVector3(0, 1, 0)
plane.firstMaterial?.diffuse.contents = UIColor.redColor()
scene.rootNode.addChildNode(planeNode)
}
What you're probably seeing is known as "z-fighting". Basically, your floor and your plane are coexisting - they are coplanar - so the rendering of them is ambiguous. Try lifting the plane up a little by setting its position higher than the floor plane, and you should see this problem go away.
Don't know why the above problem exists but I got a different way of doing the thing. You can use SCNShape to draw an overlay over a floor. I drew a rectangular path using UIBezierPath and set the path property of the SCNShape class. :)
I'm attempting to map a video as texture to a primitive cylinder for a VR project by using Scenekit: an SKVideoNode embedded in an SKScene as a texture for a SceneKit SCNTube object, and I just can't get video to display as a still image would. PLayground code below should generate moving video mapped to cylinder, but the mapping does not work:
EDIT: ADDED SINGLE LINE AT END OF LISTING TO FIX. CODE BELOW SHOULD WORK
import UIKit
import SceneKit // for 3D mapping
import SpriteKit // for SKVideoNode
import QuartzCore // for basic animation
import XCPlayground // for live preview
import AVFoundation // for video playback engine
// create a scene view with an empty scene
var sceneView = SCNView(frame: CGRect(x: 0, y: 0, width: 300, height: 300))
var scene = SCNScene()
sceneView.scene = scene
// start a live preview of that view
XCPShowView("The Scene View", view: sceneView)
// default lighting
sceneView.autoenablesDefaultLighting = true
// a geometry object
var tube = SCNTube(innerRadius: 1.99, outerRadius: 2, height: 3)
var tubeNode = SCNNode(geometry: tube)
scene.rootNode.addChildNode(tubeNode)
// video scene
let urlStr = NSBundle.mainBundle().pathForResource("sample", ofType: "mp4")
let url = NSURL(fileURLWithPath: urlStr!)
let asset = AVURLAsset(URL: url, options: nil)
let playerItem = AVPlayerItem(asset: asset)
let player = AVPlayer(playerItem: playerItem)
let videoNode = SKVideoNode(AVPlayer: player)
let spritescene = SKScene(size: CGSize(width: 1211, height: 431))
videoNode.size.width=spritescene.size.width
videoNode.size.height=spritescene.size.height
spritescene.addChild(videoNode)
// configure the geometry object
var myImage = UIImage.init(named: "BandImage.jpeg")
tube.firstMaterial?.diffuse.contents = spritescene
// set a rotation axis (no angle) to be able to
// use a nicer keypath below and avoid needing
// to wrap it in an NSValue
tubeNode.rotation = SCNVector4(x: 0.0, y: 1.0, z: 0.0, w: 0.0)
// animate the rotation of the torus
var spin = CABasicAnimation(keyPath: "rotation.w") // only animate the angle
spin.toValue = 2.0*M_PI
spin.duration = 3
spin.repeatCount = HUGE // for infinity
tubeNode.addAnimation(spin, forKey: "spin around")
// starts the video, solving the issue
sceneView.playing = true
I've posted my code (and some sample panoramic content) on github for anyone who wants to have a working sample code or is interested in collaborating on an opensource panoramic video player:
https://github.com/jglasse/OSVR
As it turns out, it looks like simulator (and playgrounds) doesn't support this feature. Moving the above code to a project and running on a device, I finally have it working.
So the moral of the story is - if you're using SKVideoNodes as textures for Scenekit, use an actual device for testing.