I'm applying list of filters to detect shape during camera capturing process. Once shape is detected - want to save it to photos for review. Googled for imageFromCurrentFramebuffer, but is always saves black picture.
// camera init
var videoCamera = GPUImageVideoCamera(sessionPreset: AVCaptureSessionPreset1920x1080, cameraPosition: .Back)
videoCamera!.outputImageOrientation = .Portrait;
videoCamera?.startCameraCapture()
// filter init
var houghTransformFilter = GPUImageHoughTransformLineDetector()
houghTransformFilter!.lineDetectionThreshold = 0.3
houghTransformFilter!.useNextFrameForImageCapture() //without this crashes
houghTransformFilter!.linesDetectedBlock = {
// my custom shape detection logic
if (found) {
self.videoCamera?.pauseCameraCapture()
var capturedImage:UIImage? = self.houghTransformFilter!.imageFromCurrentFramebuffer()
UIImageWriteToSavedPhotosAlbum(capturedImage, nil, nil, nil);
}
}
I had a similar problem. You probably need to add this line before the call to grab the frame:
self.houghTransformFilter?.useNextFrameForImageCapture()
Related
This is a long question so I wanted to put a TL;DR on top:
I want to track QR codes via on of two methods: image tracking by cropping them upon detection, or placing anchors with raycasting. Both of these methods fail when the phone is in portrait mode. Camera source is an ARSession, SceneKit and RealityKit not used. There's only ARKit. What to do?
I am currently working on an application with Swift in which I try to render some stuff on a server, transmit the video to iPhone and display it on screen using a MTKView. I only needed a custom Meal shader to apply some complex calculations to received frames, so I did not use SceneKit or RealityKit. I only have ARSession from ARKit and a Metal view here, and up to this point everything works fine.
I am able to do image tracking at this point. However, I want to apply this behaviour to QR codes. What I want is to detect a QR code (multiple if possible) and then track it just like images. Since I don't have the QR code as ARReferenceImages beforehand like normal image tracking, I was left with two options:
Option 1: Using raycast(_:) on ARSession
This is probably the right way to do it. However, for this I need to activate both plane tracking options on ARSession, which then creates many anchors and managing them with image tracking becomes harder. This is not the actual problem though. Actual problem is that when the phone is in landscape mode, raycasting works as intended. When phone goes into portrait mode, even if I pass the frame in correct orientation it misses everything and hit test results return empty. I am not using hitTest(_:) because it is deprecated.
I want to explain the "correct orientation" thing here before going into second option. ARSession is capturing frames and I am able to check each frame through didUpdate delegate function of the session. When I read the pixel buffer out of the frame using frame.capturedImage and turn it into a CIImage, the image is always in landscape mode (width > height). Doesn't matter if the phone is in portrait mode or not. So whenever I want to pass this image, I am using oriented(.right) for portrait and oriented(.up) for landscape. I got that idea from another question asked about QR bounding box, and so far it is the best option (but not good enough). Just want to note that when I tried raycasting, I tried it with the image size, not screen size (screen size = my Metal view size because it is fullscreen) since the image is larger than the screen in reality. I am able to see this if I put a breakpoint and quicklook my CIImage created from current camera frame.
Option 2: Cropping the QR and treating it as image tracking
This is another approach which I am currently working on. Algorithm is simple: check every frame with Vision. If there are detected QR codes, read their data first. If that data matches with an existing QR, then re-read it if the cropped QR size is larger than existing one. If not, do nothing. Then use this cropped QR image for tracking QR as an image. At this point we would have the data already so no problems here.
However, I tried many times to do the proper transformation explained here in the answer. Again, I think I am able to transform normalized bounding box into a real rect which can correctly crop the image. Yet, as it is in raycasting, works perfectly only if the phone is in landscape position. When in portrait it works good enough ONLY IF the phone is really close to QR code and it is centered on the screen.
For related code, I have this in my View controller:
private var ciContext: CIContext = CIContext.init(options: nil)
private var sequenceHandler: VNImageRequestHandler?
And then I have this code to extract QR codes from CIImage:
func extractQrCode(image: CIImage) -> [VNBarcodeObservation]? {
self.sequenceHandler = VNImageRequestHandler(ciImage: image)
let barcodeRequest = VNDetectBarcodesRequest()
barcodeRequest.symbologies = [.QR]
try? self.sequenceHandler?.perform([barcodeRequest])
guard let results = barcodeRequest.results else {
return nil
}
return results
}
An this is the delegate that checks and operates on every frame (code currently for Option 2):
func session(_ session: ARSession, didUpdate frame: ARFrame) {
let rotImg = self.renderer?.getInterfaceOrientation() == .portrait ? CIImage(cvPixelBuffer: frame.capturedImage).oriented(.right) : CIImage(cvPixelBuffer: frame.capturedImage)
if let barcodes = self.extractQrCode(image: rotImg) {
for barcode in barcodes {
guard let payload = barcode.payloadStringValue else { continue }
var rect = CGRect()
rect = VNImageRectForNormalizedRect(barcode.boundingBox.botToTop(), Int(rotImg.extent.width), Int(rotImg.extent.height))
let existingQR = TrackedImagesManager.imagesToTrack.filter{ $0.isQR && $0.QRData == payload}.first
if ((rect.size.width < 800 || rect.size.height < 800 || abs(rect.size.height - rect.size.width) > 32) && existingQR == nil) {
DispatchQueue.main.async {
self.showToastMessage(message: "Please get closer to the QR code and try centering it on your screen.", font: UIFont.systemFont(ofSize: 18), duration: 3)
}
continue
} else if (existingQR != nil) {
if (rect.width > existingQR?.originalImage?.size.width ?? 999) {
let croppedImg = rotImg.cropped(to: rect)
let croppedCgImage = self.ciContext.createCGImage(croppedImg, from: croppedImg.extent)!
let trackImg = UIImage(cgImage: croppedCgImage)
existingQR?.originalImage = trackImg
existingQR?.image = ARReferenceImage(croppedCgImage, orientation: .up, physicalWidth: 0.1)
} else {
continue
}
} else if rect.width != 0 {
let croppedImg = rotImg.cropped(to: rect)
let croppedCgImage = self.ciContext.createCGImage(croppedImg, from: croppedImg.extent)!
let trackImg = UIImage(cgImage: croppedCgImage)
TrackedImagesManager.imagesToTrack.append(TrackedImage(id: 9, type: 1, image: ARReferenceImage(croppedCgImage, orientation: .up, physicalWidth: 0.1), originalImage: trackImg, isQR: true, QRData: payload))
print("qr norm rect: \(barcode.boundingBox) \n qr rect: \(rect) \nqr data: \(payload) \nqr hittestres: ")
}
}
}
}
Finally, for the transformation, I have this extension (tried various ways, this is the best so far):
extension CGRect {
func botToTop() -> CGRect {
let transform = CGAffineTransform(scaleX: 1, y: -1).translatedBy(x: 0, y: -1)
return self.applying(transform)
}
}
So for both options I need some advice to make things right. Android side of the same thing is implemented as in Option 2, but Android returns a nicely cropped QR code upon detection. We don't have that. What do I do now?
I am stuck on a problem. I need to apply transformation (scale, rotation, position) right after i add model to my rootNode. Right after when i apply transformation on child model added to rootNode it shows fine on screen but when i apply transformation on rootNode it doesn't refresh. i experimented that as soon i touch screen UI updates. I also tried putting delay of 2,3 secs.
expected
UIView should update as soon i apply transformation to rootNode.
let res = SCNAction.repeatForever(SCNAction.rotateBy(x: 0, y: 0.5, z: 0, duration: 1))
// let res = SCNAction.sequence([SCNAction.wait(duration: 2000), SCNAction.rotateTo(x: CGFloat(180), y: CGFloat(90), z: CGFloat(0), duration: 1.0)])
self.rootNode.runAction(res)
i tried putting code in
RunLoop.main.perform {}
i tried using
scnView.preferredFramesPerSecond = 30
scnView.rendersContinuously = true
But none works. i am using sdk IOS 13.2. Any help please.
Edit:
var rootNode = SCNNode()
viewDidload(){
scnScene.rootNode.addChildNode(rootNode)
....
}
func initSceneWithModel(modelURL: URL) {
do {
try personModel = addModel(url: modelURL)
menuButton.setImage(UIImage.fontAwesomeIcon(name: .bars, style: .solid, textColor: .white, size: XConstants.FONT_AWSOME_SIZE), for: .normal)
selectedModel = personModel
centerPivot(for: personModel!)
moveNodeToCenter(node: personModel!)
setupEyeBlocker()
// selectedModel = eyeBlocker
updateFieldUI()
DispatchQueue.main.asyncAfter(deadline: .now() + 3) {
self.applyInitTransformations()
}
} catch let error {
Utilities.xalert(inView: self.view, desc: error.localizedDescription)
}
}
func applyInitTransformations() {
if let info = vm.physicialFile.extraInfo {
// personModel?.position = info.person.position
// personModel?.scale = info.person.scale
// personModel?.eulerAngles = info.person.rotation
var valueRotPos = SCNMatrix4Mult(SCNMatrix4MakeRotation(0,0,0,0), SCNMatrix4MakeTranslation(0,0,0))
var valueScale = SCNMatrix4MakeScale(7.0,7.0,7.0) // scales to 0.1 of original size
rootNode.transform = SCNMatrix4Mult(valueRotPos, valueScale)
// rootNode.position = info.root.position
// rootNode.scale = info.root.scale
// rootNode.eulerAngles = info.root.rotation
}
else {
applyEyeBlockerDefaultPosition()
}
}
Apple clearly says:
...
You should not modify the transform property of the root node.
...
(https://developer.apple.com/documentation/scenekit/scnscene/1524029-rootnode)
This might be causing the issues you have with your scene. Avoid SCNActions to be run on the rootNode. They are designed to run on the content of the rootNode (any SCNNode added to the rootNode).
You could probably take a common SCNNode, call it like myRootNode, add it to the real rootNode and add all your other content to myRootNode. Transformations should then apply correctly to all your sub-content, if this is your goal.
BTW: scnView.preferredFramesPerSecond = 30 never gave me more performence or any benefits. Leave it default. Scenekit switches automatically to lower framerates if required.
EDIT:
apply transformation like so:
// Precalculate the Rotation the Position and the Scale
var valueRotPos = SCNMatrix4Mult(SCNMatrix4MakeRotation(0,0,0,0), SCNMatrix4MakeTranslation(0,0,0))
var valueScale = SCNMatrix4MakeScale(0.1,0.1,0.1) // scales to 0.1 of original size
then you do:
myRootNode.transform = SCNMatrix4Mult(valueRotPos, valueScale)
(you could also try to use the worldTransform of the node or the other transform properties of the nodes presentation node-object)
There could be several things wrong with my implementation, but I feel like it’s close.
I'm trying to record the camera feed using GPUImage, as well as set a dynamic overlay that updates 30 (or 60) times per second onto the video while it's recording. I don't want this to be done after the video has been recorded.
I have a pixel buffer that is updated 30 times a second in this case, and I'm creating a GPUImageRawDataInput object from the base address (UnsafeMutablePointer<GLubyte>). With the GPUImageRawDataInput object, I'm setting it's target to the 'filter' variable, which is just a GPUImageFilter(). I'm not sure if this is the correct way to set it up.
Currently the video it’s recording is just the camera feed, there’s no overlay.
func setupRecording() {
movieWriter = GPUImageMovieWriter(movieURL: fileURL(), size: self.view.frame.size)
movieWriter?.encodingLiveVideo = true
videoCamera = GPUImageVideoCamera(sessionPreset: AVCaptureSession.Preset.hd1920x1080.rawValue, cameraPosition: .back)
videoCamera?.outputImageOrientation = .portrait
videoCamera?.horizontallyMirrorFrontFacingCamera = true
videoCamera?.horizontallyMirrorRearFacingCamera = false
let userCameraView = gpuImageView
userCameraView?.fillMode = kGPUImageFillModePreserveAspectRatioAndFill;
//filter's declaration up top - let filter = GPUImageFilter()
videoCamera?.addTarget(filter)
videoCamera?.audioEncodingTarget = movieWriter;
filter.addTarget(userCameraView)
filter.addTarget(movieWriter)
videoCamera?.startCapture()
}
func shouldUpdateRawInput(_ data: UnsafeMutablePointer<GLubyte>!) {//updated 30x per second
if let rawDataInput = rawDataInput {
rawDataInput.updateData(fromBytes: data, size: self.view.frame.size)
rawDataInput.processData()
} else {
//first time creating it
rawDataInput = GPUImageRawDataInput(bytes: data, size: self.view.frame.size, pixelFormat: GPUPixelFormatBGRA)
rawDataInput?.processData()
rawDataInput?.addTarget(filter)
}
}
//----------------------------------------
//this is my conversion of the pixel buffer to the GLubyte in another file
CVPixelBufferLockBaseAddress(pixelBuf, 0);
GLubyte* rawDataBytes=(GLubyte*)CVPixelBufferGetBaseAddress(pixelBuf);
[_delegate shouldUpdateRawInput:rawDataBytes];
So I am using AVCaptureSession to take pictures with front camera. I am also creating previewLayer from this session to display current image on screen.
previewLayer = AVCaptureVideoPreviewLayer(session: session)
previewLayer.videoGravity = AVLayerVideoGravityResizeAspectFill
It all works like it should.
But now I have a problem because I need to implement a button which will flip / mirror (transform) this preview layer - so users have a choice to take normal selfie picture or take mirrored one.
I have already tried transforming previewLayer and it KINDA works. The problem is that if you rotate device, preview picture rotates in the other way since it is transformed. (in the default or any other camera app picture rotates with camera). Anyone has any idea how to achieve that?
Mirroring preview layer: (I tried transforming layer and even view later, same result).
#IBAction func mirrorCamera(_ sender: AnyObject) {
cameraMirrored = !cameraMirrored
if cameraMirrored {
// TRANSFORMING VIEW
self.videoPreviewView.transform = CGAffineTransform(scaleX: -1, y: 1);
// OR LAYER
self.previewLayer.transform = CATransform3DMakeScale(-1, 1, 1);
} else {
self.videoPreviewView.transform = CGAffineTransform(scaleX: 1, y: 1);
self.videoPreviewView.transform = CATransform3DMakeScale(1, 1, 1);
}
}
Nowadays, if you use the mirrored property of the preview layer directly, you will get a deprecation warning at runtime. The current way to do it is using directly the connection from the camera. You must do something like this (code below is not real code, property names probably will differ, but you get the idea)
if (cameraPreviewLayer.connection.SupportsVideoMirroring) {
cameraPreviewLayer.connection.automaticallyAdjustsVideoMirroring = false
cameraPreviewLayer.connection.videoMirrored = true
}
AVCaptureVideoPreviewLayer has a property mirrored. Set this true or false, as required.
I am trying to change local video resolution in webRTC. I used following method to create local video tracker:
-(RTCVideoTrack *)createLocalVideoTrack {
RTCVideoTrack *localVideoTrack = nil;
RTCMediaConstraints *mediaConstraints = [[RTCMediaConstraints alloc] initWithMandatoryConstraints:nil optionalConstraints:nil];
RTCAVFoundationVideoSource *source =
[self.factory avFoundationVideoSourceWithConstraints:mediaConstraints];
localVideoTrack =
[self.factory videoTrackWithSource:source
trackId:#"ARDAMSv0"];
return localVideoTrack;
}
I set the mandatory constraint as follow, but it doesn't work:
#{#"minFrameRate":#"20",#"maxFrameRate":#"30",#"maxWidth":#"320",#"minWidth":#"240",#"maxHeight":#"320",#"minHeight":#"240"};
Could anyone help me?
Latest SDK builds don't provide factory method to build capturer with constraints any more. Solution should be based on AVCaptureSession instead and WebRTC will take care about CPU and bandwidth utilization.
For this you need to keep reference to your RTCVideoSource that was passed to capturer. It has method:
- (void)adaptOutputFormatToWidth:(int)width height:(int)height fps:(int)fps;
Calling this function will cause frames to be scaled down to the requested resolution. Also, frames will be cropped to match the requested aspect ratio, and frames will be dropped to match the requested fps. The requested aspect ratio is orientation agnostic and will be adjusted to maintain the input orientation, so it doesn't matter if e.g. 1280x720 or 720x1280 is requested.
var localVideoSource: RTCVideoSource?
You may create your video track this way:
func createVideoTrack() -> RTCVideoTrack? {
var source: RTCVideoSource
if let localSource = self.localVideoSource {
source = localSource
} else {
source = self.factory.videoSource()
self.localVideoSource = source
}
let devices = RTCCameraVideoCapturer.captureDevices()
if let camera = devices.first,
// here you can decide to use front or back camera
let format = RTCCameraVideoCapturer.supportedFormats(for: camera).last,
// here you have a bunch of formats from tiny to up to 4k, find 1st that conforms your needs, i.e. if you usemax 1280x720, then no need to pick 4k
let fps = format.videoSupportedFrameRateRanges.first?.maxFrameRate
// or take smth in between min..max, i.e. 24 fps and not 30, to reduce gpu/cpu use {
let intFps = Int(fps)
let capturer = RTCCameraVideoCapturer(delegate: source)
capturer.startCapture(with: camera, format: format, fps: intFps)
let videoTrack = self.factory.videoTrack(with: source, trackId: WebRTCClient.trackIdVideo)
return videoTrack
}
retun nil
}
And when you need to change resolution, you can tell this video source to do "scaling".
func changeResolution(w: Int32, h: Int32) -> Bool {
guard let videoSource = self.localVideoSource else {
return false
}
// TODO: decide fps
videoSource.adaptOutputFormat(toWidth: w, height: h, fps: 30)
return true
}
Camera will still capture frames with resolution providd in format to startCapture. And if you care about resource utilization, then you can also use next methods prior to adaptOutputFormat.
// Stops the capture session asynchronously and notifies callback on completion.
- (void)stopCaptureWithCompletionHandler:(nullable void (^)(void))completionHandler;
// Starts the capture session asynchronously.
- (void)startCaptureWithDevice:(AVCaptureDevice *)device format:(AVCaptureDeviceFormat *)format fps:(NSInteger)fps;