I a trying to transcode an H264 video to HEVC using AVAssetWriter and it fails on iPhone 6s. Supposedly, the iPhone 6s supports HEVC for transcoding, not real-time video encoding. The same code works on iPhone 7 and above. If the iPhone 6s doesn't support the HEVC codec, how do we programmatically determine supported codecs at runtime?
let bitrate = trackBitrate / 5
let trackDimensions = trackSize
let compressionSettings: [String: Any] = [
AVVideoAverageBitRateKey: bitrate,
AVVideoMaxKeyFrameIntervalKey: 30,
AVVideoProfileLevelKey: kVTProfileLevel_HEVC_Main_AutoLevel
]
var videoSettings: [String : Any] = [
AVVideoWidthKey: trackDimensions.width,
AVVideoHeightKey: trackDimensions.height,
AVVideoCompressionPropertiesKey: compressionSettings
]
videoSettings[AVVideoCodecKey] = AVVideoCodecType.hevc
I ended up doing it this way
if #available(iOS 11.0, *), AVCaptureVideoDataOutput().availableVideoCodecTypes.contains(.hevc) {
// use .hevc settings here
} else {
// use .h264 settings here
}
The #available check is needed to make the compiler happy if your app is targeting < iOS 11
You can get the iPhone model by the following code:
+ (NSString *) deviceModel {
struct utsname systemInfo;
uname(&systemInfo);
return [NSString stringWithCString: systemInfo.machine encoding: NSUTF8StringEncoding];
}
and determine if iPhone 6S disable H265 encode and iPhone7 above enable H265 encode.
Related
I have a sample project for resizing videos that works well for most videos. However, AVAssetWriter fails to write the audio from specific videos with the error:
Error Domain=AVFoundationErrorDomain
Code=-11800 "The operation could not be completed"
UserInfo={
NSLocalizedFailureReason=An unknown error occurred (-12780),
NSLocalizedDescription=The operation could not be completed,
NSUnderlyingError=0x282e956e0 {
Error Domain=NSOSStatusErrorDomain Code=-12780 "(null)"
}
}
What is even more problematic is that the same code works fine if I run it on macOS, but it breaks in iOS. I think it isn't a hardware problem because it also breaks in the iOS simulator.
These are the settings I use for (de)compressing the asset tracks:
func audioDecompressionSettings() -> [String: Any] {
return [
AVFormatIDKey: kAudioFormatLinearPCM
]
}
func audioCompressionSettings() -> [String: Any] {
var audioChannelLayout = AudioChannelLayout()
memset(&audioChannelLayout, 0, MemoryLayout<AudioChannelLayout>.size)
audioChannelLayout.mChannelLayoutTag = kAudioChannelLayoutTag_Stereo
return [
AVFormatIDKey: kAudioFormatMPEG4AAC,
AVSampleRateKey: 44100,
AVEncoderBitRateKey: 128000,
AVNumberOfChannelsKey: 2,
AVChannelLayoutKey: NSData(bytes: &audioChannelLayout, length: MemoryLayout<AudioChannelLayout>.size)
]
}
func videoDecompressionSettings() -> [String: Any] {
return [
kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_420YpCbCr8BiPlanarFullRange,
kCVPixelBufferMetalCompatibilityKey as String: true
]
}
func videoCompressionSettings(size: CGSize) -> [String: Any] {
return [
AVVideoCodecKey: AVVideoCodecType.h264,
AVVideoWidthKey: size.width,
AVVideoHeightKey: size.height
]
}
The complete source code can be found here.
In that project there are two targets, one for Mac and other for iOS, both of them using the same code for resizing the video. I also included two sample video files: fruit.mp4 and rain.mp4. The first one works well in both targets, but the second one breaks in iOS.
Am I missing something here or this is likely to be an Apple bug?
The audio settings for the problematic video are:
Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, 5.1, fltp, 386 kb/s (default)
and for the other one are:
Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 137 kb/s (default)
The important difference between the two is the number of audio channels: 5.1 (5 full bandwidth channels + one low-frequency effects channel) in the first one and stereo (2) in the second one.
When reading the video file, we specify the decompression settings:
[AVFormatIDKey: kAudioFormatLinearPCM]
Which means that the decompressed audio will have the same number of channels as the source file. In our case, we have a 5.1 (actually 6) channels asset and we want to write it to a 2 channels file. It seems that AVAssetWriterInput doesn't handle that case properly in iOS and we get an error.
The solution to the problem is to specify the number of audio channels we want when decompressing the audio from the asset, like this:
[
AVFormatIDKey: kAudioFormatLinearPCM
AVNumberOfChannelsKey: 2
]
I am using AVAssetWriter to save screen recoding to mp4 (Withd H.264 video and aac audio encoding). Everything works perfectly as expected but some users complained about the 0KB video size issue, that is no data is being written to the output file.
By trying over and over again I was able to reproduce this issue. It happened after a few successful writes (sometimes after 2 sometimes after 5 and on some machines it never happens). I compared the successful flow and failure looked at the logs for it. The only thing which I found out was this
CMIO_Unit_Converter_Audio.cpp:588:RebuildAudioConverter AudioConverterSetProperty() failed (1886547824)
Where
kAudioFormatUnsupportedPropertyError = 1886547824
Which leads me to check the audio format in case of failure but the Audio format for AVAssetWriter was perfectly fine.
I am setting up audio writer input as
AVAssetWriter Inputs
[<AVAssetWriterInput: 0x600002047950, mediaType = vide, outputSettings = {
AVVideoCodecKey = avc1;
AVVideoHeightKey = 840;
AVVideoWidthKey = 1360;
}>, <AVAssetWriterInput: 0x60000205e740, mediaType = soun, outputSettings = {
AVFormatIDKey = 1633772320;
AVNumberOfChannelsKey = 2;
AVSampleRateKey = 44100;
}>]
My code to create AVAssetWriterInput for video and audio is as follow
Audio AVAssetWriterInput
settings = [
AVFormatIDKey : kAudioFormatMPEG4AAC,
AVNumberOfChannelsKey : ch,
AVSampleRateKey : rate,
]
audioInput = AVAssetWriterInput(mediaType: .audio, outputSettings: settings)
Video AVAssetWriterInput
var settings: [String : Any] = [
AVVideoWidthKey : cx,
AVVideoHeightKey : cy,
AVVideoCodecKey : AVVideoCodecType.h264,
]
videoInput = AVAssetWriterInput(mediaType: .video, outputSettings: settings)
I noticed error in the log appears after AVCaptureSession is being started and before AVCaptureVideoDataOutputSampleBufferDelegate's captureOutput. I logged AVCaptureSession inputs and output as well.
AVCaptureSession Inputs
[<AVCaptureScreenInput: 0x6000022e18e0>, <AVCaptureDeviceInput: 0x6000022e0a00 [Built-in Microphone]>]
AVCaptureSession Outputs
[<AVCaptureAudioDataOutput: 0x6000022e1f40>, <AVCaptureVideoDataOutput: 0x60000228b7a0>]
But again I don't see any problem here. I have been trying to figure this out for a good few days but failed to do so. I don't have any idea what triggers it and even when everything is properly set, it fails to write any output. This is only happening with video+audio if I write video only it works properly.
Apple announced HEVC encode support for A10 devices running iOS 11, and HEVC decode support for A9 devices running iOS 11.
Before create those hardware codec, how to check if current device is support the feature?
What is the chip? A8, A9, or A10 without hard code the model.
Don't check for the specific SOC, check for the feature you actually want. You'll need the VideoToolbox call VTIsHardwareDecodeSupported, passing the kCMVideoCodeType_HEVC key:
VTIsHardwareDecodeSupported(kCMVideoCodeType_HEVC)
However, iOS has software decoder fallbacks for HEVC if you need them.
Edit: Ah, sorry - I misread and thought you were talking about decoding. For encoding, you may be able to get what you want with VTCopySupportedPropertyDictionaryForEncoder, using the kCMVideoCodeType_HEVC and specifying the parameters you want to encode. I don't know if iOS has a fallback software encoder for HEVC, so this may give false positives.
For encoder, I could not find an official way but this seems to work in my tests:
#import <AVFoundation/AVFoundation.h>
#import <VideoToolbox/VideoToolbox.h>
- (BOOL)videoCodecTypeHevcIsSuppored {
if (#available(iOS 11, *)) {
CFMutableDictionaryRef encoderSpecification = CFDictionaryCreateMutable( NULL,0, &kCFTypeDictionaryKeyCallBacks, &kCFTypeDictionaryValueCallBacks);
CFDictionarySetValue(encoderSpecification, kVTCompressionPropertyKey_ProfileLevel, kVTProfileLevel_HEVC_Main_AutoLevel);
CFDictionarySetValue(encoderSpecification, kVTCompressionPropertyKey_RealTime, kCFBooleanTrue);
OSStatus status = VTCopySupportedPropertyDictionaryForEncoder(3840, 2160, kCMVideoCodecType_HEVC, encoderSpecification, nil, nil);
if (status == kVTCouldNotFindVideoEncoderErr) {
return NO;
}
return YES;
}
return NO;
}
kanso's great answer in Swift 4.
Assumes that we target only iOS11 or higher and adds an extra check:
import AVFoundation
import VideoToolbox
#available (iOS 11, *)
func isHEVCHardwareEncodingSupported() -> Bool {
let encoderSpecDict : [String : Any] =
[kVTCompressionPropertyKey_ProfileLevel as String : kVTProfileLevel_HEVC_Main_AutoLevel,
kVTCompressionPropertyKey_RealTime as String : true]
let status = VTCopySupportedPropertyDictionaryForEncoder(3840, 2160,
kCMVideoCodecType_HEVC,
encoderSpecDict as CFDictionary,
nil, nil)
if status == kVTCouldNotFindVideoEncoderErr {
return false
}
if status != noErr {
return false
}
return true
}
I am working on camera app. i am using AVCapturePhotoOutput for ios 10.x device and AVCaptureStillImageOutput for below 10.x devices.
I am using below capture settings while capturing Photo
let settings = AVCapturePhotoSettings()
let previewPixelType = settings.availablePreviewPhotoPixelFormatTypes.first!
let previewFormat = [kCVPixelBufferPixelFormatTypeKey as String: previewPixelType,
kCVPixelBufferWidthKey as String: 1080,
kCVPixelBufferHeightKey as String: 1080,
]
settings.previewPhotoFormat = previewFormat
settings.isHighResolutionPhotoEnabled = true
settings.flashMode = .on
settings.isAutoStillImageStabilizationEnabled = true
self.captureOutputPhoto?.capturePhoto(with: settings, delegate: self)
when i am try to capture photo using above setting
captureOutput:didFinishProcessingPhotoSampleBuffer:previewPhotoSampleBuffer:resolvedSettings:bracketSettings:error
above delegate throws error first time. I am beginner for AVCapturePhotoSettings. the problem is occurs after every successful photo capture with flash mode.
From Apple documentation:
You may not enable image stabilization if the flash mode is
on
. (Enabling the flash takes priority over the
isAutoStillImageStabilizationEnabled
setting.)
Not sure, if it should throw error, but you can try to remove this string
settings.isAutoStillImageStabilizationEnabled = true
captureOutput:didFinishProcessingPhotoSampleBuffer:previewPhotoSampleBuffer:resolvedSettings:bracketSettings:error:, an Objective-C delegate method, whose Swift version is photoOutput(_:didFinishProcessingPhoto:previewPhoto:resolvedSettings:bracketSettings:error:), is deprecated.
Instead implement the Swift method photoOutput(_:didFinishProcessingPhoto:error:).
I'm using this method for handling the flash settings. AVCaptureDevice is basically the camera that you are using and the AVCaptureFlashMode is the flash mode you want to use.
func changeFlashSettings(device: AVCaptureDevice, mode: AVCaptureFlashMode) {
do {
try device.lockForConfiguration()
device.flashMode = mode
device.unlockForConfiguration()
} catch {
print("Change Flash Configuration Error: \(error)")
}
}
With this you can set the flash setting to on, off or auto. Hope this helps.
I have a project for Android reading a short[] array with PCM data from microphone Buffer for live analysis. I need to convert this functionality to iOS Swift. In Android it is very simple and looks like this..
import android.media.AudioFormat;
import android.media.AudioRecord;
...
AudioRecord recorder = new AudioRecord(MediaRecorder.AudioSource.DEFAULT, someSampleRate, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT, AudioRecord.getMinBufferSize(...));
recorder.startRecording();
later I read the buffer with
recorder.read(data, offset, length); //data is short[]
(That's what i'm looking for)
Documentation: https://developer.android.com/reference/android/media/AudioRecord.html
I'm very new to Swift and iOS. I've read a lot of documentation about AudioToolkit, ...Core and whatever. All I found is C++/Obj-C and Bridging Swift Header solutions. Thats much to advanced and outdated for me.
For now I can read PCM-Data to a CAF-File with AVFoundation
settings = [
AVLinearPCMBitDepthKey: 16 as NSNumber,
AVFormatIDKey: Int(kAudioFormatLinearPCM),
AVLinearPCMIsBigEndianKey: 0 as NSNumber,
AVLinearPCMIsFloatKey: 0 as NSNumber,
AVSampleRateKey: 12000.0,
AVNumberOfChannelsKey: 1 as NSNumber,
]
...
recorder = try AVAudioRecorder(URL: someURL, settings: settings)
recorder.delegate = self
recorder.record()
But that's not what I'm looking for (or?). Is there an elegant way to achieve the android read functionality described above? I need to get a sample-array from the microphone buffer. Or do i need to do the reading on the recorded CAF file?
Thanks a lot! Please help me with easy explanations or code examples. iOS terminology is not mine yet ;-)
If you don't mind floating point samples and 48kHz, you can quickly get audio data from the microphone like so:
let engine = AVAudioEngine() // instance variable
func setup() {
let input = engine.inputNode!
let bus = 0
input.installTapOnBus(bus, bufferSize: 512, format: input.inputFormatForBus(bus)) { (buffer, time) -> Void in
let samples = buffer.floatChannelData[0]
// audio callback, samples in samples[0]...samples[buffer.frameLength-1]
}
try! engine.start()
}