avcapture video repeated part on changing orientation - ios

I have got an issue regarding video capture orientation. first of all its about voip app which uses pjsip which can transmit video. the video is being captured using AVCapture frame. so the issue arises while device orientation changes then i have to set the avcapture orientation as well.
for example:
capConnection.videoOrientation = AVCaptureVideoOrientationLandscapeLeft;
it works fine but i have repeat image part.
so the question is how to get rid of this repeated image part. i have tried this solution but it keeps crashing on vImageRotate90_ARGB8888 any idea how resolve this issue?
in order to try this yourself out, you can get PJSIP version 2.3 video with sample project compile and run it against a test SIP server.
edit: the preview layer is rotated and scaled fine. the particular issue happens on receiving RTP (video) stream when that device rotates and sends images with repeated edges. for instance, if iPadA(horizontal) starts video call with iPadB(horizontal) the image is fine and not repeated edges. but if the iPadA rotates to vertical then iPadB gets this repeated edge images. notice on rotation the capture connection orientation is set to current device orientation.
note the the preview layer has AVLayerVideoGravityResize but that does not affect the outgoing video stream.

The two key pieces are setting the autoresizingMask in viewDidLoad and adjusting your captureVideoPreviewLayer.frame to self.view.layer.bounds; in willAnimateRotationToInterfaceOrientation.
- (void)viewDidLoad
{
[super viewDidLoad];
self.view.autoresizingMask = UIViewAutoresizingFlexibleWidth | UIViewAutoresizingFlexibleHeight;
//
// react to device orientation notifications
//
[[NSNotificationCenter defaultCenter] addObserver : self
selector : #selector(deviceOrientationDidChange:)
name : UIDeviceOrientationDidChangeNotification
object : nil];
[[UIDevice currentDevice] beginGeneratingDeviceOrientationNotifications];
}
- (void) willAnimateRotationToInterfaceOrientation : (UIInterfaceOrientation)toInterfaceOrientation
duration : (NSTimeInterval)duration
{
captureVideoPreviewLayer.frame = self.view.layer.bounds;
[[captureVideoPreviewLayer connection] setVideoOrientation:toInterfaceOrientation];
}
- (void)deviceOrientationDidChange: (NSNotification*)notification
{
UIDeviceOrientation orientation = [UIDevice currentDevice].orientation;
switch (orientation)
{
case UIDeviceOrientationPortrait:
case UIDeviceOrientationPortraitUpsideDown:
case UIDeviceOrientationLandscapeLeft:
case UIDeviceOrientationLandscapeRight:
currentDeviceOrientation = orientation;
break;
// unsupported?
case UIDeviceOrientationFaceUp:
case UIDeviceOrientationFaceDown:
default:
break;
}
}

Related

Why a default webrtc video capturer orientation is 90° on iOS?

Upon looking into RTCCameraVideoCapturer.m I found this:
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection {
#if TARGET_OS_IPHONE
switch (_orientation) {
case UIDeviceOrientationPortrait:
_rotation = RTCVideoRotation_90;
break;
case UIDeviceOrientationPortraitUpsideDown:
_rotation = RTCVideoRotation_270;
break;
case UIDeviceOrientationLandscapeLeft:
_rotation = usingFrontCamera ? RTCVideoRotation_180 : RTCVideoRotation_0;
break;
case UIDeviceOrientationLandscapeRight:
_rotation = usingFrontCamera ? RTCVideoRotation_0 : RTCVideoRotation_180;
break;
}
#else
// No rotation on Mac.
_rotation = RTCVideoRotation_0;
#endif
}
Why does a default camera capturer's orientation is 90°. I made a simple voip client and the videofrom my iPhone looks rotated even when the device is in Portrait. What am I missing here? Is there any way to rotate capturer's video orientation.
Any thoughts appreciated
Check AVCaptureConnection.videoOrientation.
https://developer.apple.com/documentation/avfoundation/avcaptureconnection/1389415-videoorientation
I guess the raw orientation depends of the camera manufacturer. To offer maximum speed to client, image data are transferred straight from sensor buffer. You can't rotate capturer's video orientation. You can only mix video and interface orientation properties to properly present it to the user.

AVCaptureVideoPreviewLayer wrong orientation when UI rotation is disabled

I have a AVCaptureVideoPreviewLayer instance added to a view controller view hierarchy.
- (void) loadView {
...
self.previewLayer = [AVCaptureVideoPreviewLayer layerWithSession:nil];
self.previewLayer.frame = self.view.bounds;
self.previewLayer.videoGravity = AVLayerVideoGravityResizeAspectFill;
[self.view.layer addSublayer: _previewLayer];
// adding other UI elements
...
}
...
- (void) _setupPreviewLayerWithSession: (AVCaptureSession*) captureSession
{
self.previewLayer.session = self.captureManager.captureSession;
self.previewLayer.connection.videoOrientation = AVCaptureVideoOrientationLandscapeRight;
}
The layer frame is updated in -viewDidLayoutSubviews method. View controller orientation is locked to UIInterfaceOrientationMaskLandscapeRight.
The issue is as following:
the device is held in landscape orientation
the view controller is presented modally - video layer displays correctly.
the device is then locked and while it's locked the device is rotated to portrait orientation.
the device is then unlocked while still being in portrait orientation and for several seconds the video layer is displayed rotated 90 degrees. However, the frame for the video layer is correct. All other UI elements display correctly. After several seconds the layer snaps to correct orientation.
Please find the bounds for the layer and UI elements below
I've tried updating the video layer orientation as following (with no results):
subscribing to AVCaptureSessionDidStartRunningNotification and UIApplicationDidBecomeActiveNotification notifications
calling the update when -viewWillTransitionToSize:withTransitionCoordinator: method is called
on -viewWillAppear:
The issue doesn't seem to be connected to video layer orientation itself, but more to view hierarchy layout.
UPDATE:
As suggested, I also tried updating the video layer orientation on device orientation change which didn't help.
I also noticed that the issue mostly happens after the application is launched and the screen is presented for the first time. On subsequent screen presentations during the same session, the reproduce rate for the issue is really low (something like 1/20).
Try this code:
[[NSNotificationCenter defaultCenter] addObserver:self
selector:#selector(orientationChanged:)
name:UIDeviceOrientationDidChangeNotification
object:nil];
-(void)orientationChanged:(NSNotification *)notif {
[_videoPreviewLayer setFrame:_viewPreview.layer.bounds];
if (_videoPreviewLayer.connection.supportsVideoOrientation) {
_videoPreviewLayer.connection.videoOrientation = [self interfaceOrientationToVideoOrientation:[UIApplication sharedApplication].statusBarOrientation];
}
}
- (AVCaptureVideoOrientation)interfaceOrientationToVideoOrientation:(UIInterfaceOrientation)orientation {
switch (orientation) {
case UIInterfaceOrientationPortrait:
return AVCaptureVideoOrientationPortrait;
case UIInterfaceOrientationPortraitUpsideDown:
return AVCaptureVideoOrientationPortraitUpsideDown;
case UIInterfaceOrientationLandscapeLeft:
return AVCaptureVideoOrientationLandscapeLeft;
case UIInterfaceOrientationLandscapeRight:
return AVCaptureVideoOrientationLandscapeRight;
default:
break;
}
// NSLog(#"Warning - Didn't recognise interface orientation (%d)",orientation);
return AVCaptureVideoOrientationPortrait;
}

AVCaptureVideoPreviewLayer, video orientation stuck

I have an iPhone camera app in which I use the AVCaptureVideoPreviewLayer. When someone takes a picture/stillImage this is shown in a new ViewController. The magic happens once this view controller is dismissed.
The whole app rotates properly, both viewcontroller; but for one exception. If I take a picture in one orientation, rotate the device in the new ViewController and then go back to the AVCaptureVideoPreviewLayer the whole layer rotates along just fine, except for the image, so suddenly the input is presented sideways through the previewLayer.
I have checked, and tried setting the frame for the PreviewLayer, but this all seems fine, the values I see when debugging are all correct. It is just the image that is displayed that is skewed. Rotating the device back and forth fixes this issue during use.
Has anyone seen this before, and does anyone have a clue how to fix this?
You could change the previewLayer's connection's videoOrientation when the interface rotates
- (void)didRotateFromInterfaceOrientation:(UIInterfaceOrientation)fromInterfaceOrientation {
self.previewLayer.connection.videoOrientation = self.interfaceOrientation;
}
Since didRotateFromInterfaceOrientation: is deprecated in iOS 8, I change the video orientation of the connection in viewDidLayoutSubviews
switch ([UIApplication sharedApplication].statusBarOrientation) {
case UIInterfaceOrientationPortrait:
self.captureVideoPreviewLayer.connection.videoOrientation = AVCaptureVideoOrientationPortrait;
break;
case UIInterfaceOrientationPortraitUpsideDown:
self.captureVideoPreviewLayer.connection.videoOrientation = AVCaptureVideoOrientationPortraitUpsideDown;
break;
case UIInterfaceOrientationLandscapeLeft:
self.captureVideoPreviewLayer.connection.videoOrientation = AVCaptureVideoOrientationLandscapeLeft;
break;
case UIInterfaceOrientationLandscapeRight:
self.captureVideoPreviewLayer.connection.videoOrientation = AVCaptureVideoOrientationLandscapeRight;
break;
default:
break;
}
I had a similar problem in the past.
Once dismissed camera controller you could refresh view rotation with this:
UIDeviceOrientation orientation = [[UIDevice currentDevice] orientation];
UIWindow *window = [[UIApplication sharedApplication] keyWindow];
UIView *topView = [window.subviews objectAtIndex:0];
[topView removeFromSuperview];
[window addSubview:topView];
So, removing top most view on shown window and resetting it on window, it should refresh view rotation properly.
Hope it helps
Maybe, instead of changing AVCaptureVideoPreviewLayer frame, you could try to put it in a container UIView, and set this container's frame. (I fixed a 'sizing' bug on ÀVCaptureVideoPreviewLayerthis way : sizing directly PreviewLayer had no effect, but sizing its containingUIView` worked).
Or,
maybe you can force orientation in your viewController viewWillAppear (or viewDidAppear ?)
- (void)viewWillAppear:(BOOL)animated
{
[super viewWillAppear:animated];
// try and force current orientation
UIApplication *application = [UIApplication sharedApplication];
UIInterfaceOrientation currentOrientation = application.statusBarOrientation;
[application setStatusBarOrientation:currentOrientation animated:animated];
}
Good luck !
Since the interfaceOrientation is deprecated, the up to date swift 2.0 solution shall be:
let previewLayerConnection = self.previewLayer.connection
let orientation = AVCaptureVideoOrientation(rawValue: UIApplication.sharedApplication().statusBarOrientation.rawValue)
previewLayerConnection.videoOrientation = orientation!

How to determine if iOS device's orientation *actually* changes?

I am writing this code to make some changes to how my app is laid out depending on if it is in landscape or portrait mode. I have set up the app to get a notification on the orientation changing like this:
[[NSNotificationCenter defaultCenter] addObserver:self selector:#selector(receivedRotate:) name:UIDeviceOrientationDidChangeNotification object:NULL];
And then defining receivedRotate:
-(void) receivedRotate: (NSNotification*) notification {
UIDeviceOrientation interfaceOrientation = [[UIDevice currentDevice] orientation];
// ... do something
}
However, UIDeviceOrientationDidChangeNotification is called whenever the device is moved slightly. E.g., the UIDeviceOrientation can be any of the following:
UIDeviceOrientationPortrait,
UIDeviceOrientationPortraitUpsideDown,
UIDeviceOrientationLandscapeLeft,
UIDeviceOrientationLandscapeRight,
UIDeviceOrientationFaceUp,
UIDeviceOrientationFaceDown
This is great but I just want to know if the device orientation is in portrait or landscape, and I also only want to run this code when the orientation actually changes - from portrait to landscape or vice-versa. If they turn the phone all the way around, it is still in landscape mode - just a different form of landscape mode. What's the ideal way to make sure that my rotation code only runs when it is necessary?
DeviceOrientation vs. ScreenSize vs StatusBar.isLandscape?
iOS 11, Swift 4 and Xcode 9.X
Regardless of using AutoLayout or not, there are several ways to get the right device orientation, and they could be used to detect rotation changes while using the app, as well as getting the right orientation at app launch or after resuming from background.
This solutions work fine in iOS 11 and Xcode 9.X
1. UIScreen.main.bounds.size:
If you only want to know if the app is in landscape or portrait mode, the best point to start is in viewDidLoad in the rootViewController at launch time and in viewWillTransition(toSize:) in the rootViewController if you want to detect rotation changes while the app is in background, and should resume the UI in the right orientation.
let size = UIScreen.main.bounds.size
if size.width < size.height {
print("Portrait: \(size.width) X \(size.height)")
} else {
print("Landscape: \(size.width) X \(size.height)")
}
This also happens early during the app/viewController life cycles.
2. NotificationCenter
If you need to get the actual device orientation (including faceDown, faceUp, etc). you want to add an observer as follows (even if you do it in the application:didFinishLaunchingWithOptions method in the AppDelegate, the first notifications will likely be triggered after the viewDidLoad is executed
device = UIDevice.current
device?.beginGeneratingDeviceOrientationNotifications()
notificationCenter = Notifi`enter code here`cationCenter.default
notificationCenter?.addObserver(self, selector: #selector(deviceOrientationChanged),
name: Notification.Name("UIDeviceOrientationDidChangeNotification"),
object: nil)
And add the selector as follows. I split it in 2 parts to be able to run inspectDeviceOrientation() in viewWillTransition(toSize:)
#objc func deviceOrientationChanged() {
print("Orientation changed")
inspectDeviceOrientation()
}
func inspectDeviceOrientation() {
let orientation = UIDevice.current.orientation
switch UIDevice.current.orientation {
case .portrait:
print("portrait")
case .landscapeLeft:
print("landscapeLeft")
case .landscapeRight:
print("landscapeRight")
case .portraitUpsideDown:
print("portraitUpsideDown")
case .faceUp:
print("faceUp")
case .faceDown:
print("faceDown")
default: // .unknown
print("unknown")
}
if orientation.isPortrait { print("isPortrait") }
if orientation.isLandscape { print("isLandscape") }
if orientation.isFlat { print("isFlat") }
}
Note that the UIDeviceOrientationDidChangeNotification may be posted several times during launch, and in some cases it may be .unknown. What I have seen is that the first correct orientation notification is received after the viewDidLoad and viewWillAppear methods, and right before viewDidAppear, or even applicationDidBecomeActive
The orientation object will give you all 7 possible scenarios(from the enum UIDeviceOrientation definition):
public enum UIDeviceOrientation : Int {
case unknown
case portrait // Device oriented vertically, home button on the bottom
case portraitUpsideDown // Device oriented vertically, home button on the top
case landscapeLeft // Device oriented horizontally, home button on the right
case landscapeRight // Device oriented horizontally, home button on the left
case faceUp // Device oriented flat, face up
case faceDown // Device oriented flat, face down
}
Interestingly, the isPortrait read-only Bool variable is defined in an extension to UIDeviceOrientation as follows:
extension UIDeviceOrientation {
public var isLandscape: Bool { get }
public var isPortrait: Bool { get }
public var isFlat: Bool { get }
public var isValidInterfaceOrientation: Bool { get }
}
3. StatusBarOrientation
UIApplication.shared.statusBarOrientation.isLandscape
This also works fine to determine if orientation is portrait or landscape orientation and gives the same results as point 1. You can evaluate it in viewDidLoad (for App launch) and in viewWillTransition(toSize:) if coming from Background. But it won't give you the details of top/bottom, left/right, up/down you get with the notifications (Point 2)
I believe what you're looking for is UIInterfaceOrientation, not UIDeviceOrientation. To use this, just implement the following function in your view controller.
- (void)didRotateFromInterfaceOrientation:(UIInterfaceOrientation)fromInterfaceOrientation
{
UIInterfaceOrientation currentOrientation = self.interfaceOrientation;
//
}
This enum is declared as the following
typedef enum : NSInteger {
UIInterfaceOrientationPortrait = UIDeviceOrientationPortrait,
UIInterfaceOrientationPortraitUpsideDown = UIDeviceOrientationPortraitUpsideDown,
UIInterfaceOrientationLandscapeLeft = UIDeviceOrientationLandscapeRight,
UIInterfaceOrientationLandscapeRight = UIDeviceOrientationLandscapeLeft
} UIInterfaceOrientation;
Save the previous orientation in a property and compare it with current one.
#0x7ffffff's and #Kunal Balani's answers are down the right path, but I'd recommend you take a look at UIViewController's documentation for rotation handling on iOS 6+ applications vs how it's handled in iOS 5 and older, since you don't mention whether or not you're supporting iOS 5 (which at the time of writing of this answer is still supported). This will help resolve any issues you could run into down the road. I'd recommend specifically taking a look at willRotateToInterfaceOrientation:duration: / willAnimateRotationToInterfaceOrientation:duration: / didRotateFromInterfaceOrientation: and when they are called during the lifecycle of a rotation.
You will definitely want to store the previous orientation as mentioned in the other answers so you don't run your code more than once.
As for your specific question on more "agnostic" as to what version of landscape vs portrait, UIKit's function reference contains a number of pertinent macros, specifically UIInterfaceOrientationIsPortrait() and UIInterfaceOrientationIsLandscape().

When does it changes orientation to default orientation?

I'm developing an iOS application with latest SDK.
My app only supports Landscape right orientation and it only has one view controller. I set the only available orientation modifying myProject-info.plist.
I'm trying to understand why this code outputs a log for a view in Portrait mode when I only supports landscape right orientation.
- (void) viewWillAppear:(BOOL)animated
{
[super viewWillAppear:animated];
UIInterfaceOrientation orientation = [UIApplication sharedApplication].statusBarOrientation;
UIView *videoView = self.videoPreviewView;
NSLog(#"Video frame: %#", NSStringFromCGRect(videoView.frame));
}
Console output:
Video frame: {{0, 0}, {320, 568}}
I'm working with AVCaptureVideoPreviewLayer and I need to set its frame, but when I do it on viewWillAppear: I get portrait values but orientation value is Landscape right.
Is there an event or anything similar that i gets triggered when orientation changes to landscape?
When does the app changes orientation to default orientation?
Go into xCode, and click the project name at the top of the screen (with an xcode LOGO underneath it).
Then reveal the iPhone/Ipad development info and select the orientations you want to be available for users :)
If you add the line:
[[NSNotificationCenter defaultCenter] addObserver:self selector:#selector(orientationChanged) name:UIDeviceOrientationDidChangeNotification object:nil];
then it will call the -(void)orientationChanged; method every time the orientation is changed. You can then do something like:
-(void)orientationChanged {
if (UIDeviceOrientationIsPortrait([UIApplication sharedApplication].statusBarOrientation) {
//device is portrait;
}
else {
//device is landscape;
}
}
Then you can use this to determine the layout of your views and setup based on the orientation.
EDIT:
You can also check if the orientation returned by the statusBar's orientation is equal to UIDeviceOrientationLandscapeLeft/LandscapeRight/Portrait/PortraitUpsideDown.

Resources