How to record IOS screen programmatically - ios

Is there any way to record IOS screen programmatically. Means whatever activity you are doing like clicking buttons, Scrolling tableviews.
Even if a video is playing that will be captured again along with some other activity?
Have tried these
https://www.raywenderlich.com/30200/avfoundation-tutorial-adding-overlays-and-animations-to-videos
https://github.com/alskipp/ASScreenRecorder
but with these libraries won't provide quality video. I need quality video.
The issue is that with video playing in the background when i capture screen it does not show smooth video. It shows like one frame of video and then after 3-4 secs 2nd frame and so on. Also quality of video is not good its blurred

As of iOS 9, it looks like ReplayKit is available to greatly simplify this.
https://developer.apple.com/reference/replaykit
https://code.tutsplus.com/tutorials/ios-9-an-introduction-to-replaykit--cms-25458
Update: This may be less relevant now that iOS 11 has a built-in screen recorder, but the following Swift 3 code worked for me:
import ReplayKit
#IBAction func toggleRecording(_ sender: UIBarButtonItem) {
let r = RPScreenRecorder.shared()
guard r.isAvailable else {
print("ReplayKit unavailable")
return
}
if r.isRecording {
self.stopRecording(sender, r)
}
else {
self.startRecording(sender, r)
}
}
func startRecording(_ sender: UIBarButtonItem, _ r: RPScreenRecorder) {
r.startRecording(handler: { (error: Error?) -> Void in
if error == nil { // Recording has started
sender.title = "Stop"
} else {
// Handle error
print(error?.localizedDescription ?? "Unknown error")
}
})
}
func stopRecording(_ sender: UIBarButtonItem, _ r: RPScreenRecorder) {
r.stopRecording( handler: { previewViewController, error in
sender.title = "Record"
if let pvc = previewViewController {
if UI_USER_INTERFACE_IDIOM() == UIUserInterfaceIdiom.pad {
pvc.modalPresentationStyle = UIModalPresentationStyle.popover
pvc.popoverPresentationController?.sourceRect = CGRect.zero
pvc.popoverPresentationController?.sourceView = self.view
}
pvc.previewControllerDelegate = self
self.present(pvc, animated: true, completion: nil)
}
else if let error = error {
print(error.localizedDescription)
}
})
}
// MARK: RPPreviewViewControllerDelegate
func previewControllerDidFinish(_ previewController: RPPreviewViewController) {
previewController.dismiss(animated: true, completion: nil)
}

ReplayKit is available, although you are not allowed to access the result video, the only way I've found so far is to make a number of screenshots (store them in array of images) and then convert these images to the video, not very efficient from performance standpoint though, but might work when you don't really need a 30/60 fps screen recording and might be ok w/ 6-20 pfs. Here's the full example.

Check out ScreenCaptureView, this has video-recording support built-in (see link).
What this does is it saves the contents of a UIView to a UIImage. The author suggests you can save a video of the app in use by passing the frames through AVCaptureSession.
I believe it hasn't been tested with an OpenGL subview, but assuming that it works you might be able to modify it slightly to include audio and then you'd be set.
AVCaptureSession Sample
AVCaptureSession Reference
import UIKit
import AVFoundation
class ViewController: UIViewController {
let captureSession = AVCaptureSession()
let stillImageOutput = AVCaptureStillImageOutput()
var error: NSError?
override func viewDidLoad() {
super.viewDidLoad()
let devices = AVCaptureDevice.devices().filter{ $0.hasMediaType(AVMediaTypeVideo) && $0.position == AVCaptureDevicePosition.Back }
if let captureDevice = devices.first as? AVCaptureDevice {
captureSession.addInput(AVCaptureDeviceInput(device: captureDevice, error: &error))
captureSession.sessionPreset = AVCaptureSessionPresetPhoto
captureSession.startRunning()
stillImageOutput.outputSettings = [AVVideoCodecKey:AVVideoCodecJPEG]
if captureSession.canAddOutput(stillImageOutput) {
captureSession.addOutput(stillImageOutput)
}
if let previewLayer = AVCaptureVideoPreviewLayer(session: captureSession) {
previewLayer.bounds = view.bounds
previewLayer.position = CGPointMake(view.bounds.midX, view.bounds.midY)
previewLayer.videoGravity = AVLayerVideoGravityResizeAspectFill
let cameraPreview = UIView(frame: CGRectMake(0.0, 0.0, view.bounds.size.width, view.bounds.size.height))
cameraPreview.layer.addSublayer(previewLayer)
cameraPreview.addGestureRecognizer(UITapGestureRecognizer(target: self, action:"saveToCamera:"))
view.addSubview(cameraPreview)
}
}
}
func saveToCamera(sender: UITapGestureRecognizer) {
if let videoConnection = stillImageOutput.connectionWithMediaType(AVMediaTypeVideo) {
stillImageOutput.captureStillImageAsynchronouslyFromConnection(videoConnection) {
(imageDataSampleBuffer, error) -> Void in
let imageData = AVCaptureStillImageOutput.jpegStillImageNSDataRepresentation(imageDataSampleBuffer)
UIImageWriteToSavedPhotosAlbum(UIImage(data: imageData), nil, nil, nil)
}
}
}
override func didReceiveMemoryWarning() {
super.didReceiveMemoryWarning()
}
}

You can use this library to record a view : screen-cap-view available on GitHub written in Objective C.
**And to use it in swift:**
--> Drag and drop the .m and .h files in your xcode project.
--> Make a header file and import the this file in that : *#import "IAScreenCaptureView.h"*
--> Then give a View this class from the PropertyInspector and then make a IBOutlet for that view . Something like this:
*#IBOutlet weak var contentView: IAScreenCaptureView!*
--> Then Finally just simply start and stop the recording of the view where ever and when ever you want and for that the code will be like this :
For Starting the Recording : *contentView.startRecording()*
For Stoping the Recording : *contentView.stopRecording()*
//Hope this helps.Happy coding. \o/ , ¯\_(ツ)_/¯ ,(╯°□°)╯︵ ┻━┻

Related

Swift - taking still frames from live camera and feeding them into Vision framework

Note: I'm almost completely new to Swift, so apologies if this seems like an obvious answer
I'm trying to code an app that can detect people doing exercises (think jumping jacks) and use that to control a simple game. I have a camera feed set up and showing on the screen, but I can't figure out how to get from that to still images/frames that it could analyze. My second problem is that, even if I knew how to do that, I don't know how to feed those images into the Vision framework to analyze them. Here's my code:
class ViewController2: UIViewController{
override func viewDidLoad() {
super.viewDidLoad()
labelchange()
//If it says to put it in the viewDidLoad function put it under here
self.addCameraInput()
self.showCameraFeed()
self.captureSession.startRunning()
} //Do any additional setup after loading the view
#IBOutlet var thingtext : UILabel!
func labelchange() {
if typeChosen == "ice" {
thingtext.text = "You're doing " + exercise + " for an " + typeChosen + " attack!"
}
else {
thingtext.text = "You're doing " + exercise + " for a " + typeChosen + " attack!"
}
}
private let captureSession = AVCaptureSession()
private func addCameraInput() {
guard let device = AVCaptureDevice.DiscoverySession(
deviceTypes: [.builtInWideAngleCamera, .builtInDualCamera, .builtInTrueDepthCamera],
mediaType: .video,
position: .front).devices.first else {
fatalError("No back camera device found, please make sure to run SimpleLaneDetection in an iOS device and not a simulator")
}
let cameraInput = try! AVCaptureDeviceInput(device: device)
self.captureSession.addInput(cameraInput)
}
private lazy var previewLayer = AVCaptureVideoPreviewLayer(session: self.captureSession)
private func showCameraFeed() {
self.previewLayer.videoGravity = .resizeAspectFill
self.view.layer.addSublayer(self.previewLayer)
self.previewLayer.frame = self.view.frame
}
override func viewDidLayoutSubviews() {
super.viewDidLayoutSubviews()
self.previewLayer.frame = self.view.frame
}
}
If anyone can help me with this, it would be greatly appreciated.

Camera preview layer does not show up Swift 4

I am creating a ViewController in which I want to have a somewhat small UIView in the corner of the ViewController to display the camera preview. I am using a function to do this. However when I pass in the small UIView into the function the camera preview is not showing up. The weird thing is if I tell the function to display the preview on self.view everything works fine and I can see the camera preview. For this reason I think the problem is with the way I insert the layer or something similar.
Here is the function I am using to display the preview...
func displayPreview(on view: UIView) throws {
guard let captureSession = self.captureSession, captureSession.isRunning else { throw CameraControllerError.captureSessionIsMissing }
self.previewLayer = AVCaptureVideoPreviewLayer(session: captureSession)
self.previewLayer?.videoGravity = AVLayerVideoGravity.resizeAspectFill
self.previewLayer?.connection?.videoOrientation = .portrait
view.layer.insertSublayer(self.previewLayer!, at: 0)
self.previewLayer?.frame = view.frame
}
I call this function from inside another function which handles setting up the capture session and other similar things.
func configureCameraController() {
cameraController.prepare {(error) in
if let error = error {
print("ERROR")
print(error)
}else{
}
print("hello")
try! self.cameraController.displayPreview(on: self.mirrorView)
}
}
configureCameraController()
How can I get the camera preview layer to show up on the smaller UIView?
Can you try adding the following
let rootLayer: CALayer = self.yourSmallerView.layer
rootLayer.masksToBounds = true
self.previewLayer.frame = rootLayer.bounds
rootLayer.addSublayer(self.previewLayer)
in place of
view.layer.insertSublayer(self.previewLayer!, at: 0)
Also ensure, yourSmallerView.contentMode = UIViewContentMode.scaleToFill

UIButton to capture image not responding

I am new to iOS development and I am trying to follow the following tutorial to learn how to capture images and then save them on a server for my purposes:
https://medium.com/#rizwanm/swift-camera-part-2-c6de440a9404
The tutorial is great and I am only initializing the camera and trying to snap the image (not doing the QR part).
Unfortunately, when I hit the 'capture' button, the image is not being taken or saved in my gallery and I am not able to understand why. I use Xcode 8.3 and I am running iOS 10.3 on my iPhone.
I have connected the button with the view controller and called it in the function onTapTakePhoto as shown in the code below. Please advise as to why this might be happening.
import UIKit
import AVFoundation
class ViewController: UIViewController {
#IBOutlet weak var previewView: UIView!
#IBOutlet weak var CaptureButton: UIButton!
//helps transfer data between one or more input device like camera
var captureSession: AVCaptureSession?
//instance variable
var capturePhotoOutput: AVCapturePhotoOutput? //capturePhotoOutput will help us snap a live photo
//helps render the camera view finder in ViewController
var videoPreviewLayer: AVCaptureVideoPreviewLayer?
override func viewDidLoad() {
super.viewDidLoad()
CaptureButton.layer.cornerRadius = CaptureButton.frame.size.width / 2
CaptureButton.clipsToBounds = true
//set up camera here
let captureDevice = AVCaptureDevice.defaultDevice(withMediaType: AVMediaTypeVideo)
// get an instance of the AVCAptureDevicenput
// serves as a middle man to attach input device to capture device
// chance the input device unavailable so wrap in do catch
do {
// Get an instance of the AVCaptureDeviceInput class using the previous deivce object
let input = try AVCaptureDeviceInput(device: captureDevice)
// Initialize the captureSession object
captureSession = AVCaptureSession()
// Set the input devcie on the capture session
captureSession?.addInput(input)
//set up preview view to see live feed
videoPreviewLayer = AVCaptureVideoPreviewLayer(session: captureSession)
videoPreviewLayer?.videoGravity = AVLayerVideoGravityResizeAspectFill
videoPreviewLayer?.frame = view.layer.bounds
previewView.layer.addSublayer(videoPreviewLayer!)
// finally start the capture session
captureSession?.startRunning()
// get an instance of AVCapturePhotoOutput class
capturePhotoOutput = AVCapturePhotoOutput()
capturePhotoOutput?.isHighResolutionCaptureEnabled = true
//set the outut on the capture session
captureSession?.addOutput(capturePhotoOutput)
} catch {
print (error)
return
}
}
#IBAction func onTapTakePhoto(_ sender: UIButton) {
// Make sure capturePhotoOutput is valid
guard let capturePhotoOutput = self.capturePhotoOutput else { return }
// Get an instance of AVCapturePhotoSettings class
let photoSettings = AVCapturePhotoSettings()
// Set photo settings for our need
photoSettings.isAutoStillImageStabilizationEnabled = true
photoSettings.isHighResolutionPhotoEnabled = true
photoSettings.flashMode = .auto
// Call capturePhoto method by passing our photo settings and a delegate implementing AVCapturePhotoCaptureDelegate
capturePhotoOutput.capturePhoto(with: photoSettings, delegate: self)
}
}
// to get the captured image
extension ViewController : AVCapturePhotoCaptureDelegate {
func capture(_ captureOutput: AVCapturePhotoOutput,
didFinishProcessingPhotoSampleBuffer photoSampleBuffer: CMSampleBuffer?,
previewPhotoSampleBuffer: CMSampleBuffer?,
resolvedSettings: AVCaptureResolvedPhotoSettings,
bracketSettings: AVCaptureBracketedStillImageSettings?,
error: Error?) {
// Make sure we get some photo sample buffer
guard error == nil,
let photoSampleBuffer = photoSampleBuffer else {
print("Error capturing photo: \(String(describing: error))")
return
}
// Convert photo same buffer to a jpeg image data by using AVCapturePhotoOutput
guard let imageData = AVCapturePhotoOutput.jpegPhotoDataRepresentation(forJPEGSampleBuffer: photoSampleBuffer, previewPhotoSampleBuffer: previewPhotoSampleBuffer) else {
return
}
// Initialise an UIImage with our image data
let capturedImage = UIImage.init(data: imageData , scale: 1.0)
if let image = capturedImage {
// Save our captured image to photos album
UIImageWriteToSavedPhotosAlbum(image, nil, nil, nil)
}
}
}
One thing that you should check is that, if you have added the permission in the info.plist file to access the camera.
You have to make a key-value entry for the key
Privacy - Photo Library Usage Description
(if you are using the gallery)
and
Privacy - Camera Usage Description
for using the camera itself.
Refer to this for more details.
If it doesn't work, put a debugger on the click action of the button and check the code flow. If the debugger is not hit after pressing the button, there could be an issue with the button action outlet, try to remake the outlet action.
First of all with the help of break points, check which section of your code is not being executed.There might be some problem related to your button connection.
OR
Try this one
https://www.youtube.com/watch?v=994Hsi1zs6Q
OR
this one in Objective-C
https://github.com/omergul/LLSimpleCamera
Thank you for the answers :)
The problem was a missing connection to my action button code from the story board. Feels like such a silly mistake in retrospect
func imagePickerController(_ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [String : Any]) {
if (info[UIImagePickerControllerOriginalImage] as? UIImage) != nil {
}
dismiss(animated: true, completion:
{
// write your set image code for UIButton is here.
})
}

Capturing image with avfoundation

I am using AvFoundation for camera.
This is my live preview:
It looks good. When user presses to "Button" I am creating a snapshot on same screen. (Like snapchat)
I am using following code for capturing image and showing it on the screen:
self.stillOutput.captureStillImageAsynchronouslyFromConnection(videoConnection){
(imageSampleBuffer : CMSampleBuffer!, _) in
let imageDataJpeg = AVCaptureStillImageOutput.jpegStillImageNSDataRepresentation(imageSampleBuffer)
let pickedImage: UIImage = UIImage(data: imageDataJpeg)!
self.captureSession.stopRunning()
self.previewImageView.frame = CGRect(x:0, y:0, width:UIScreen.mainScreen().bounds.width, height:UIScreen.mainScreen().bounds.height)
self.previewImageView.image = pickedImage
self.previewImageView.layer.zPosition = 100
}
After user captures an image screen looks like this:
Please look at the marked area. It wasn't looking on the live preview screen(Screenshot 1).
I mean live preview is not showing everything. But I am sure my live preview works well because I compared with other camera apps and everything was same as my live preview screen. I guess I have a problem with captured image.
I am creating live preview with following code:
override func viewWillAppear(animated: Bool) {
super.viewWillAppear(animated)
captureSession.sessionPreset = AVCaptureSessionPresetPhoto
let devices = AVCaptureDevice.devices()
for device in devices {
// Make sure this particular device supports video
if (device.hasMediaType(AVMediaTypeVideo)) {
// Finally check the position and confirm we've got the back camera
if(device.position == AVCaptureDevicePosition.Back) {
captureDevice = device as? AVCaptureDevice
}
}
}
if captureDevice != nil {
beginSession()
}
}
func beginSession() {
let err : NSError? = nil
do {
try captureSession.addInput(AVCaptureDeviceInput(device: captureDevice))
} catch{
}
captureSession.addOutput(stillOutput)
if err != nil {
print("error: \(err?.localizedDescription)")
}
previewLayer = AVCaptureVideoPreviewLayer(session: captureSession)
previewLayer?.videoGravity=AVLayerVideoGravityResizeAspectFill
self.cameraLayer.layer.addSublayer(previewLayer!)
previewLayer?.frame = self.cameraLayer.frame
captureSession.startRunning()
}
My cameraLayer :
How can I resolve this problem?
Presumably you are using an AVCaptureVideoPreviewLayer. So it sounds like this layer is incorrectly placed or incorrectly sized, or it has the wrong AVLayerVideoGravity setting. Part of the image is off the screen or cropped; that's why you don't see that part of what the camera sees while you are previewing.
Ok, I found the solution.
I used the
captureSession.sessionPreset = AVCaptureSessionPresetHigh
Instead of
captureSession.sessionPreset = AVCaptureSessionPresetPhoto
Then problem fixed.

How BarCode,QR Code are recognized without capturing the image?

I wonder how Bar Code and QR Code (even character) are recognized without capturing it.I have seen in many app, when we keep our device above any of these (QR/Bar Code), the app automatically recognize it and starts processing. Is there any scanning mechanism used for this? How this can be achieved? What are mechanism involving in this?
Thanks in advance.
1) The phone camera will be launched by the library it will autofocus and scans until it finds the decoded info from the image displayed by camera
2) The info will be parsed by the library and it will give you the result.
the decoded info is "the bar code which has a information decoded in it"
Example for QRCode: The data is present as square
for barcode: the data is present as vertical lines
The library has all the logics for detecting the type of code and decoding as per the format.
Please read more the QrCode/Bar code libraries docs or implement it and learn
You can use AVCaptureSession, e.g.:
let session = AVCaptureSession()
var qrPayload: String?
func startSession() {
guard !started else { return }
let output = AVCaptureMetadataOutput()
output.setMetadataObjectsDelegate(self, queue: .main)
let device: AVCaptureDevice?
if #available(iOS 10.0, *) {
device = AVCaptureDevice
.DiscoverySession(deviceTypes: [.builtInWideAngleCamera], mediaType: .metadataObject, position: .back)
.devices
.first
} else {
device = AVCaptureDevice.devices().first { $0.position == .back }
}
guard
let camera = device,
let input = try? AVCaptureDeviceInput(device: camera),
session.canAddInput(input),
session.canAddOutput(output)
else {
// handle failures here
return
}
session.addInput(input)
session.addOutput(output)
output.metadataObjectTypes = [.qr]
let videoLayer = AVCaptureVideoPreviewLayer(session: session)
videoLayer.frame = view.bounds
videoLayer.videoGravity = .resizeAspectFill
view.layer.addSublayer(videoLayer)
session.startRunning()
}
And extend your view controller to conform to AVCaptureMetadataOutputObjectsDelegate:
extension QRViewController: AVCaptureMetadataOutputObjectsDelegate {
func metadataOutput(_ output: AVCaptureMetadataOutput, didOutput metadataObjects: [AVMetadataObject], from connection: AVCaptureConnection) {
guard
qrPayload == nil,
let object = metadataObjects.first as? AVMetadataMachineReadableCodeObject,
let string = object.stringValue
else { return }
qrPayload = string
print(qrPayload)
// perhaps dismiss this view controller now that you’ve succeeded
}
}
Note, I’m testing to make sure that the qrPayload is nil because I find that you can see metadataOutput(_:didOutput:from:) get called a few times before the view controller is dismissed.

Resources