i have implemented in Xamarin IOS simple app to take picture using AVCaptureSession and AVCapturePhotoCaptureDelegate.
All work fine, but my problem is the image size.
Using in my code AVCaptureSession.PresetPhoto like session preset in my AVCaptureSession i obtain very big photo size.
After some researches i write my method to obtain the correct format scale.
public UIImage RescaleImage(UIImage img) {
var format = new UIGraphicsImageRendererFormat();
format.Scale = 1;
var renderer = new UIGraphicsImageRenderer(img.Size, format);
return renderer.CreateImage(c => img.Draw(CGPoint.Empty) );
}
After that the size of my photo has the same resolution of IOS camera (3024 x 4032).
Now i should compress image like jpg.
//Image is UIImage
var jpgImag = Image.AsJPEG(0f);
// Converting UIImage to byte[] array
Byte[] myByteArray = new Byte[jpgImag.Length];
System.Runtime.InteropServices.Marshal.Copy(jpgImag.Bytes,
myByteArray, 0, Convert.ToInt32(jpgImag.Length));
Here my capture session code:
CaptureSession = new AVCaptureSession();
CaptureSession.SessionPreset = AVCaptureSession.PresetPhoto;
CaptureOutput = new AVCapturePhotoOutput();
CaptureSession.AddOutput(CaptureOutput);
CaptureDevice = AVCaptureDevice.GetDefaultDevice(AVMediaType.Video);
NSError Error;
// Prepare device for configuration
if (!CaptureDevice.LockForConfiguration(out Error)) {
// There has been an issue, abort
Console.WriteLine("Error: {0}", Error.LocalizedDescription);
CaptureDevice.UnlockForConfiguration();
return;
}
CaptureDevice.FocusMode = AVCaptureFocusMode.ContinuousAutoFocus;
// Unlock configuration
CaptureDevice.UnlockForConfiguration();
CaptureDeviceInput = AVCaptureDeviceInput.FromDevice(CaptureDevice);
CaptureSession.AddInput(CaptureDeviceInput);
My settings:
var settings = AVCapturePhotoSettings.Create();
var previewPixelType = settings.AvailablePreviewPhotoPixelFormatTypes.First();
settings.PreviewPhotoFormat = new NSDictionary<NSString, NSObject>(CVPixelBuffer.PixelFormatTypeKey, settings.AvailablePreviewPhotoPixelFormatTypes[0]);
settings.FlashMode = AVCaptureFlashMode.Auto;
return settings;
If i save myByteArray i obtain a 20MB file for example. If i shoot same photo with ios camera i obtain 3MB image (for example).
The only difference between 2 images is the Bit depth.
My app is 32. IOS camera 24
And dpi
My app is 96. IOS camera 72
My question is:
how can set my AVCaptureSession to obtain more similar result like IOS camera?
(swift code is fine too)
Thanks
Related
Im instanciating the AVPlayerItemVideoOutput like so:
let videoOutput = AVPlayerItemVideoOutput(pixelBufferAttributes: [String(kCVPixelBufferPixelFormatTypeKey): NSNumber(value: kCVPixelFormatType_32BGRA)])
And retrieving the pixelBuffers like this:
#objc func displayLinkDidRefresh(link: CADisplayLink) {
let itemTime = videoOutput.itemTime(forHostTime: CACurrentMediaTime())
if videoOutput.hasNewPixelBuffer(forItemTime: itemTime) {
if let pixelBuffer = videoOutput.copyPixelBuffer(forItemTime: itemTime, itemTimeForDisplay: nil) {
}
}
}
But for some reason CVPixelBufferGetHeight(pixelBuffer) or Width. always return 1280x720 when the video was taken when the iPhone's camera (landscape or portrait) always height=1280 width=720. EVEN if the video is 4k. If I load a square video from instagram or any other video downloaded from the internet (not created directly with the camera app) the width and height are printed correctly when the resolution is less than 720p. But a different resolution, for ex. a 1008x1792 will throw CVPixelBufferGetHeight(pixelBuffer) = 1280
Videos taken with the camera... it always throws a lower res. I tried 4k and 1080 settings (you can change that in iOS Settings > Camera). still.. even in 1080, I get 1280x720 pixel buffers.
I figured out that th UIPickerController I was using was set to default transcode the selected video from library to a Medium setting. in this case it was 1280x720
I ended up changing this properties of the picker
picker.videoQuality = .typeHigh
picker.videoExportPreset = AVAssetExportPresetHighestQuality
Altho the property that actually makes the change is the videoExportPreset the other one I dont know what it does, even tho the Documentation specifies it is for when you record a video OR you pick a video.
A few things I want to establish first:
This works properly on multiple iPhones (iOS 10.3 & 11.x)
This works properly on any iPad simulator (iOS 11.x)
What I am left with is a situation where when I run the following code (condensed from my application to remove unrelated code), I am getting an image that is upside down (landscape) or rotated 90 degrees (portrait). Viewing the video that is processed just prior to this step shows that it is properly oriented. All testing has been done on iOS 11.2.5.
* UPDATED *
I did some further testing and found a few more interesting items:
If the video was imported from a phone, or an external source, it is properly processed
If the video was recorded on the iPad in portrait orientation, then the reader extracts it rotated 90 degrees left
If the video was recorded on the iPad in landscape orientation, then the reader extracts it upside down
In the two scenarios above, UIImage reports an orientation of portrait
A condensed version of the code involved:
import UIKit
import AVFoundation
let asset = ...
let assetReader = try? AVAssetReader(asset: asset)
if let assetTrack = asset.tracks(withMediaType: .video).first,
let assetReader = assetReader {
let assetReaderOutputSettings = [
kCVPixelBufferPixelFormatTypeKey as String : NSNumber(value: kCVPixelFormatType_32BGRA)
]
let assetReaderOutput = AVAssetReaderTrackOutput(track: assetTrack,
outputSettings: assetReaderOutputSettings)
assetReaderOutput.alwaysCopiesSampleData = false
assetReader.add(assetReaderOutput)
var images = [UIImage]()
assetReader.startReading()
var sample = assetReaderOutput.copyNextSampleBuffer()
while (sample != nil) {
if let image = sample?.uiImage { // The image is inverted here
images.append(image)
sample = assetReaderOutput.copyNextSampleBuffer()
}
}
// Continue here with array of images...
}
After some exploration, I came across the following that allowed me to obtain the video's orientation from the AVAssetTrack:
let transform = assetTrack.preferredTransform
radians = atan2(transform.b, transform.a)
Once I had that, I was able to convert it to degrees:
let degrees = (radians * 180.0) / .pi
Then, using a switch statement I could determine how to rotate the image:
Switch Int(degrees) {
case -90, 90:
// Rotate accordingly
case 180:
// Flip
default:
// Do nothing
}
I would like to use ARKit to obtain a light estimate from an image. I was able to retrieve a light estimate from a frames in a video.
var SceneView = new ARSCNView();
var arConfig = new ARKit.ARWorldTrackingConfiguration { PlaneDetection = ARPlaneDetection.Horizontal };
SceneView.Session.Run(arConfig, ARSessionRunOptions.ResetTracking);
var frame = SceneView.Session.CurrentFrame;
float light = frame.LightEstimate.AmbientIntensity;
However is it possible to instantiate an ARFrame using a CGImage?
Like
CGImage img = new CGImage("my file.jpg");
ARFrame frame = new ARFrame(img);
float light = frame.LightEstimate.AmbientIntensity;
Solutions using swift or Xamarin are welcome
Sorry, but ARFrame wraps CVPixelBuffer, which represents a video frame and depending on the device is likely in a different format than CGImage. Also ARFrame has no public initializer and the var capturedImage: CVPixelBuffer property is read only. However if you are getting the CGImage from the camera then why no get the light estimate at the time of capture and save it along with the image?
There could be several things wrong with my implementation, but I feel like it’s close.
I'm trying to record the camera feed using GPUImage, as well as set a dynamic overlay that updates 30 (or 60) times per second onto the video while it's recording. I don't want this to be done after the video has been recorded.
I have a pixel buffer that is updated 30 times a second in this case, and I'm creating a GPUImageRawDataInput object from the base address (UnsafeMutablePointer<GLubyte>). With the GPUImageRawDataInput object, I'm setting it's target to the 'filter' variable, which is just a GPUImageFilter(). I'm not sure if this is the correct way to set it up.
Currently the video it’s recording is just the camera feed, there’s no overlay.
func setupRecording() {
movieWriter = GPUImageMovieWriter(movieURL: fileURL(), size: self.view.frame.size)
movieWriter?.encodingLiveVideo = true
videoCamera = GPUImageVideoCamera(sessionPreset: AVCaptureSession.Preset.hd1920x1080.rawValue, cameraPosition: .back)
videoCamera?.outputImageOrientation = .portrait
videoCamera?.horizontallyMirrorFrontFacingCamera = true
videoCamera?.horizontallyMirrorRearFacingCamera = false
let userCameraView = gpuImageView
userCameraView?.fillMode = kGPUImageFillModePreserveAspectRatioAndFill;
//filter's declaration up top - let filter = GPUImageFilter()
videoCamera?.addTarget(filter)
videoCamera?.audioEncodingTarget = movieWriter;
filter.addTarget(userCameraView)
filter.addTarget(movieWriter)
videoCamera?.startCapture()
}
func shouldUpdateRawInput(_ data: UnsafeMutablePointer<GLubyte>!) {//updated 30x per second
if let rawDataInput = rawDataInput {
rawDataInput.updateData(fromBytes: data, size: self.view.frame.size)
rawDataInput.processData()
} else {
//first time creating it
rawDataInput = GPUImageRawDataInput(bytes: data, size: self.view.frame.size, pixelFormat: GPUPixelFormatBGRA)
rawDataInput?.processData()
rawDataInput?.addTarget(filter)
}
}
//----------------------------------------
//this is my conversion of the pixel buffer to the GLubyte in another file
CVPixelBufferLockBaseAddress(pixelBuf, 0);
GLubyte* rawDataBytes=(GLubyte*)CVPixelBufferGetBaseAddress(pixelBuf);
[_delegate shouldUpdateRawInput:rawDataBytes];
I am trying to change local video resolution in webRTC. I used following method to create local video tracker:
-(RTCVideoTrack *)createLocalVideoTrack {
RTCVideoTrack *localVideoTrack = nil;
RTCMediaConstraints *mediaConstraints = [[RTCMediaConstraints alloc] initWithMandatoryConstraints:nil optionalConstraints:nil];
RTCAVFoundationVideoSource *source =
[self.factory avFoundationVideoSourceWithConstraints:mediaConstraints];
localVideoTrack =
[self.factory videoTrackWithSource:source
trackId:#"ARDAMSv0"];
return localVideoTrack;
}
I set the mandatory constraint as follow, but it doesn't work:
#{#"minFrameRate":#"20",#"maxFrameRate":#"30",#"maxWidth":#"320",#"minWidth":#"240",#"maxHeight":#"320",#"minHeight":#"240"};
Could anyone help me?
Latest SDK builds don't provide factory method to build capturer with constraints any more. Solution should be based on AVCaptureSession instead and WebRTC will take care about CPU and bandwidth utilization.
For this you need to keep reference to your RTCVideoSource that was passed to capturer. It has method:
- (void)adaptOutputFormatToWidth:(int)width height:(int)height fps:(int)fps;
Calling this function will cause frames to be scaled down to the requested resolution. Also, frames will be cropped to match the requested aspect ratio, and frames will be dropped to match the requested fps. The requested aspect ratio is orientation agnostic and will be adjusted to maintain the input orientation, so it doesn't matter if e.g. 1280x720 or 720x1280 is requested.
var localVideoSource: RTCVideoSource?
You may create your video track this way:
func createVideoTrack() -> RTCVideoTrack? {
var source: RTCVideoSource
if let localSource = self.localVideoSource {
source = localSource
} else {
source = self.factory.videoSource()
self.localVideoSource = source
}
let devices = RTCCameraVideoCapturer.captureDevices()
if let camera = devices.first,
// here you can decide to use front or back camera
let format = RTCCameraVideoCapturer.supportedFormats(for: camera).last,
// here you have a bunch of formats from tiny to up to 4k, find 1st that conforms your needs, i.e. if you usemax 1280x720, then no need to pick 4k
let fps = format.videoSupportedFrameRateRanges.first?.maxFrameRate
// or take smth in between min..max, i.e. 24 fps and not 30, to reduce gpu/cpu use {
let intFps = Int(fps)
let capturer = RTCCameraVideoCapturer(delegate: source)
capturer.startCapture(with: camera, format: format, fps: intFps)
let videoTrack = self.factory.videoTrack(with: source, trackId: WebRTCClient.trackIdVideo)
return videoTrack
}
retun nil
}
And when you need to change resolution, you can tell this video source to do "scaling".
func changeResolution(w: Int32, h: Int32) -> Bool {
guard let videoSource = self.localVideoSource else {
return false
}
// TODO: decide fps
videoSource.adaptOutputFormat(toWidth: w, height: h, fps: 30)
return true
}
Camera will still capture frames with resolution providd in format to startCapture. And if you care about resource utilization, then you can also use next methods prior to adaptOutputFormat.
// Stops the capture session asynchronously and notifies callback on completion.
- (void)stopCaptureWithCompletionHandler:(nullable void (^)(void))completionHandler;
// Starts the capture session asynchronously.
- (void)startCaptureWithDevice:(AVCaptureDevice *)device format:(AVCaptureDeviceFormat *)format fps:(NSInteger)fps;