How to loop AVPlayer from 4 second to 8 second in swift 3? - ios

I have an AVPlayer in swift 3 that plays video - the problem is that I want to use loop from A to B seconds (for example from 4 to 8 second)here is my codes for loop but didn't work
NotificationCenter.default.addObserver(forName: .AVPlayerItemDidPlayToEndTime, object: self.Player.currentItem, queue: nil, using: { (_) in
DispatchQueue.main.async {
self.Player.seek(to: kCMTimeZero)
self.Player.play()
DispatchQueue.main.asyncAfter(deadline: DispatchTime.now() + 4.0) {
// check if player is still playing
if self.Player.rate != 0 {
print("OK")
print("Player reached 4.0 seconds")
let timeScale = self.Player.currentItem?.asset.duration.timescale;
// let seconds = kCMTimeZero
let time = CMTimeMakeWithSeconds( 8.0 , timeScale!)
self.Player.seek(to: time, toleranceBefore: kCMTimeZero, toleranceAfter: kCMTimeZero)
self.Player.play()
}
}
}
})
the problem is that this loop doesn't work and because of AVPlayerItemDidPlayToEndTime the print("OK") won't work until the player has finished the movie

There are a few options:
If you want gapless playback, you can start off by using:
Pre iOS 10: https://developer.apple.com/library/content/samplecode/avloopplayer/Introduction/Intro.html
iOS 10+:
https://developer.apple.com/documentation/avfoundation/avplayerlooper
The pre-ios10 "solution" from apple does work, and is the only way I have gotten gapless looping since I target ios9.
If you are using that solution, you also need to either feed it an avplayeritem the right length or add to the solution to cut it up as you send it to the player.
For that, you can do something like how I changed apples code (sorry if its a bit sparse - just trying to show the main changes) - Basically adding in sending the track and the chunk of time to use, then make that an AVMutableCompositionTrack (I got rid of all the stuff for video - you will want to keep that in) :
class myClass: someClass {
var loopPlayer:QueuePlayerLooper!
var avAssetLength:Int64!
var avAssetTimescale:CMTimeScale!
var avAssetTimeRange:CMTimeRange!
let composition = AVMutableComposition()
var playerItem:AVPlayerItem!
var avAssetrack:AVAssetTrack!
var compAudioTrack:AVMutableCompositionTrack!
var uurl:URL!
var avAsset:AVURLAsset!
func createCMTimeRange(start:TimeInterval, end:TimeInterval) -> CMTimeRange {
avAssetTimescale = avAssetTrack.naturalTimeScale
let a:CMTime = CMTime(seconds: start, preferredTimescale: avAssetTimescale)
let b:CMTime = CMTime(seconds: end, preferredTimescale: avAssetTimescale)
return CMTimeRange(start: a, end: b)
}
func startLoopingSection() {
loopPlayer = QueuePlayerLooper(audioURL: uurl, loopCount: -1, timeRange: createCMTimeRange(start: a_playbackPosition, end: b_playbackPosition))
loopPlayer.start()
}
}
//--==--==--==--==--==--==--==--==--
/*
Copyright (C) 2016 Apple Inc. All Rights Reserved.
See LICENSE.txt for this sample’s licensing information
Abstract:
An object that uses AVQueuePlayer to loop a video.
*/
// Marked changed code with ++
class QueuePlayerLooper : NSObject, Looper {
// MARK: Types
private struct ObserverContexts {
static var playerStatus = 0
static var playerStatusKey = "status"
static var currentItem = 0
static var currentItemKey = "currentItem"
static var currentItemStatus = 0
static var currentItemStatusKey = "currentItem.status"
static var urlAssetDurationKey = "duration"
static var urlAssetPlayableKey = "playable"
}
// MARK: Properties
private var player: AVQueuePlayer?
private var playerLayer: AVPlayerLayer?
private var isObserving = false
private var numberOfTimesPlayed = 0
private let numberOfTimesToPlay: Int
private let videoURL: URL
++var assetTimeRange:CMTimeRange!
++let composition = AVMutableComposition()
++var currentTrack:AVAssetTrack!
++var assetTimeRange:CMTimeRange!
// MARK: Looper
required init(videoURL: URL, loopCount: Int, ++timeRange:CMTimeRange) {
self.videoURL = videoURL
self.numberOfTimesToPlay = loopCount
++self.assetTimeRange = timeRange
super.init()
super.init()
}
func start(in parentLayer: CALayer) {
stop()
player = AVQueuePlayer()
playerLayer = AVPlayerLayer(player: player)
guard let playerLayer = playerLayer else { fatalError("Error creating player layer") }
playerLayer.frame = parentLayer.bounds
parentLayer.addSublayer(playerLayer)
let videoAsset = AVURLAsset(url: videoURL)
++currentTrack = composition.addMutableTrack(withMediaType: AVMediaTypeVideo, preferredTrackID: CMPersistentTrackID())
++currentTrack = videoAsset.tracks(withMediaType: AVMediaTypeVideo)
++try! compositionTrack.insertTimeRange(assetTimeRange, of: currentTrack, at: CMTimeMake(0, 1))
videoAsset.loadValuesAsynchronously(forKeys: [ObserverContexts.urlAssetDurationKey, ObserverContexts.urlAssetPlayableKey]) {
/*
The asset invokes its completion handler on an arbitrary queue
when loading is complete. Because we want to access our AVQueuePlayer
in our ensuing set-up, we must dispatch our handler to the main
queue.
*/
DispatchQueue.main.async(execute: {
var durationError: NSError?
let durationStatus = videoAsset.statusOfValue(forKey: ObserverContexts.urlAssetDurationKey, error: &durationError)
guard durationStatus == .loaded else { fatalError("Failed to load duration property with error: \(durationError)") }
var playableError: NSError?
let playableStatus = videoAsset.statusOfValue(forKey: ObserverContexts.urlAssetPlayableKey, error: &playableError)
guard playableStatus == .loaded else { fatalError("Failed to read playable duration property with error: \(playableError)") }
guard videoAsset.isPlayable else {
print("Can't loop since asset is not playable")
return
}
guard CMTimeCompare(videoAsset.duration, CMTime(value:1, timescale:100)) >= 0 else {
print("Can't loop since asset duration too short. Duration is(\(CMTimeGetSeconds(videoAsset.duration)) seconds")
return
}
/*
Based on the duration of the asset, we decide the number of player
items to add to demonstrate gapless playback of the same asset.
*/
let numberOfPlayerItems = (Int)(1.0 / CMTimeGetSeconds(videoAsset.duration)) + 2
for _ in 1...numberOfPlayerItems {
let loopItem = AVPlayerItem(asset: ++self.composition)
self.player?.insert(loopItem, after: nil)
}
self.startObserving()
self.numberOfTimesPlayed = 0
self.player?.play()
})
}
}
}}

You can add periodic time observer to monitor current time
let timeObserverToken = player.addPeriodicTimeObserver(forInterval: someInterval, queue: DispatchQueue.main) { [unowned self] time in
let seconds = CMTimeGetSeconds(cmTime)
if seconds >= 8.0 {
// jump back to 4 seconds
// do stuff
}
}

Related

MTAudioProcessingTap EXC_BAD_ACCESS , doesnt always fire the finalize callback. how to Release it?

Im trying to implement MTAudioProcessingTap and it works great. The problem is when Im done using the Tap and I reinstaniate my class and create a new Tap.
How Im supposely releasing the tap
1- I retain the tap as a property when created, hoping I can access it and release it later
2- In deinit() method of the class, I set the audiomix to nil and try to do a self.tap?.release()
The thing is.. sometimes it works and calls the FINALIZE callback and everything is great, and sometimes it doesn't and just crashes at the tapProcess Callback line:
let selfMediaInput = Unmanaged<VideoMediaInput>.fromOpaque(MTAudioProcessingTapGetStorage(tap)).takeUnretainedValue()
Here's the full code: https://gist.github.com/omarojo/03d08165a1a7962cb30c17ec01f809a3
import Foundation
import UIKit
import AVFoundation;
import MediaToolbox
protocol VideoMediaInputDelegate: class {
func videoFrameRefresh(sampleBuffer: CMSampleBuffer) //could be audio or video
}
class VideoMediaInput: NSObject {
private let queue = DispatchQueue(label: "com.GenerateMetal.VideoMediaInput")
var videoURL: URL!
weak var delegate: VideoMediaInputDelegate?
private var playerItemObserver: NSKeyValueObservation?
var displayLink: CADisplayLink!
var player = AVPlayer()
var playerItem: AVPlayerItem!
let videoOutput = AVPlayerItemVideoOutput(pixelBufferAttributes: [String(kCVPixelBufferPixelFormatTypeKey): NSNumber(value: kCVPixelFormatType_32BGRA)])
var audioProcessingFormat: AudioStreamBasicDescription?//UnsafePointer<AudioStreamBasicDescription>?
var tap: Unmanaged<MTAudioProcessingTap>?
override init(){
}
convenience init(url: URL){
self.init()
self.videoURL = url
self.playerItem = AVPlayerItem(url: url)
playerItemObserver = playerItem.observe(\.status) { [weak self] item, _ in
guard item.status == .readyToPlay else { return }
self?.playerItemObserver = nil
self?.player.play()
}
setupProcessingTap()
player.replaceCurrentItem(with: playerItem)
player.currentItem!.add(videoOutput)
NotificationCenter.default.removeObserver(self)
NotificationCenter.default.addObserver(forName: NSNotification.Name.AVPlayerItemDidPlayToEndTime, object: nil, queue: nil) {[weak self] notification in
if let weakSelf = self {
/*
Setting actionAtItemEnd to None prevents the movie from getting paused at item end. A very simplistic, and not gapless, looped playback.
*/
weakSelf.player.actionAtItemEnd = .none
weakSelf.player.seek(to: CMTime.zero)
weakSelf.player.play()
}
}
NotificationCenter.default.addObserver(
self,
selector: #selector(applicationDidBecomeActive(_:)),
name: UIApplication.didBecomeActiveNotification,
object: nil)
}
func stopAllProcesses(){
self.queue.sync {
self.player.pause()
self.player.isMuted = true
self.player.currentItem?.audioMix = nil
self.playerItem.audioMix = nil
self.playerItem = nil
self.tap?.release()
}
}
deinit{
print(">> VideoInput deinited !!!! 📌📌")
if let link = self.displayLink {
link.invalidate()
}
NotificationCenter.default.removeObserver(self)
stopAllProcesses()
}
public func playVideo(){
if (player.currentItem != nil) {
print("Starting playback!")
player.play()
}
}
public func pauseVideo(){
if (player.currentItem != nil) {
print("Pausing playback!")
player.pause()
}
}
#objc func applicationDidBecomeActive(_ notification: NSNotification) {
playVideo()
}
//MARK: GET AUDIO BUFFERS
func setupProcessingTap(){
var callbacks = MTAudioProcessingTapCallbacks(
version: kMTAudioProcessingTapCallbacksVersion_0,
clientInfo: UnsafeMutableRawPointer(Unmanaged.passUnretained(self).toOpaque()),
init: tapInit,
finalize: tapFinalize,
prepare: tapPrepare,
unprepare: tapUnprepare,
process: tapProcess)
var tap: Unmanaged<MTAudioProcessingTap>?
let err = MTAudioProcessingTapCreate(kCFAllocatorDefault, &callbacks, kMTAudioProcessingTapCreationFlag_PostEffects, &tap)
self.tap = tap
print("err: \(err)\n")
if err == noErr {
}
print("tracks? \(playerItem.asset.tracks)\n")
let audioTrack = playerItem.asset.tracks(withMediaType: AVMediaType.audio).first!
let inputParams = AVMutableAudioMixInputParameters(track: audioTrack)
inputParams.audioTapProcessor = tap?.takeRetainedValue()//tap?.takeUnretainedValue()
// tap?.release()
// print("inputParms: \(inputParams), \(inputParams.audioTapProcessor)\n")
let audioMix = AVMutableAudioMix()
audioMix.inputParameters = [inputParams]
playerItem.audioMix = audioMix
}
//MARK: TAP CALLBACKS
let tapInit: MTAudioProcessingTapInitCallback = {
(tap, clientInfo, tapStorageOut) in
tapStorageOut.pointee = clientInfo
print("init \(tap, clientInfo, tapStorageOut)\n")
}
let tapFinalize: MTAudioProcessingTapFinalizeCallback = {
(tap) in
print("finalize \(tap)\n")
}
let tapPrepare: MTAudioProcessingTapPrepareCallback = {
(tap, itemCount, basicDescription) in
print("prepare: \(tap, itemCount, basicDescription)\n")
let selfMediaInput = Unmanaged<VideoMediaInput>.fromOpaque(MTAudioProcessingTapGetStorage(tap)).takeUnretainedValue()
selfMediaInput.audioProcessingFormat = AudioStreamBasicDescription(mSampleRate: basicDescription.pointee.mSampleRate,
mFormatID: basicDescription.pointee.mFormatID, mFormatFlags: basicDescription.pointee.mFormatFlags, mBytesPerPacket: basicDescription.pointee.mBytesPerPacket, mFramesPerPacket: basicDescription.pointee.mFramesPerPacket, mBytesPerFrame: basicDescription.pointee.mBytesPerFrame, mChannelsPerFrame: basicDescription.pointee.mChannelsPerFrame, mBitsPerChannel: basicDescription.pointee.mBitsPerChannel, mReserved: basicDescription.pointee.mReserved)
}
let tapUnprepare: MTAudioProcessingTapUnprepareCallback = {
(tap) in
print("unprepare \(tap)\n")
}
let tapProcess: MTAudioProcessingTapProcessCallback = {
(tap, numberFrames, flags, bufferListInOut, numberFramesOut, flagsOut) in
print("callback \(bufferListInOut)\n")
let selfMediaInput = Unmanaged<VideoMediaInput>.fromOpaque(MTAudioProcessingTapGetStorage(tap)).takeUnretainedValue()
let status = MTAudioProcessingTapGetSourceAudio(tap, numberFrames, bufferListInOut, flagsOut, nil, numberFramesOut)
//print("get audio: \(status)\n")
if status != noErr {
print("Error TAPGetSourceAudio :\(String(describing: status.description))")
return
}
selfMediaInput.processAudioData(audioData: bufferListInOut, framesNumber: UInt32(numberFrames))
}
func processAudioData(audioData: UnsafeMutablePointer<AudioBufferList>, framesNumber: UInt32) {
var sbuf: CMSampleBuffer?
var status : OSStatus?
var format: CMFormatDescription?
//FORMAT
// var audioFormat = self.audioProcessingFormat//self.audioProcessingFormat?.pointee
guard var audioFormat = self.audioProcessingFormat else {
return
}
status = CMAudioFormatDescriptionCreate(allocator: kCFAllocatorDefault, asbd: &audioFormat, layoutSize: 0, layout: nil, magicCookieSize: 0, magicCookie: nil, extensions: nil, formatDescriptionOut: &format)
if status != noErr {
print("Error CMAudioFormatDescriptionCreater :\(String(describing: status?.description))")
return
}
print(">> Audio Buffer mSampleRate:\(Int32(audioFormat.mSampleRate))")
var timing = CMSampleTimingInfo(duration: CMTimeMake(value: 1, timescale: Int32(audioFormat.mSampleRate)), presentationTimeStamp: self.player.currentTime(), decodeTimeStamp: CMTime.invalid)
status = CMSampleBufferCreate(allocator: kCFAllocatorDefault,
dataBuffer: nil,
dataReady: Bool(truncating: 0),
makeDataReadyCallback: nil,
refcon: nil,
formatDescription: format,
sampleCount: CMItemCount(framesNumber),
sampleTimingEntryCount: 1,
sampleTimingArray: &timing,
sampleSizeEntryCount: 0, sampleSizeArray: nil,
sampleBufferOut: &sbuf);
if status != noErr {
print("Error CMSampleBufferCreate :\(String(describing: status?.description))")
return
}
status = CMSampleBufferSetDataBufferFromAudioBufferList(sbuf!,
blockBufferAllocator: kCFAllocatorDefault ,
blockBufferMemoryAllocator: kCFAllocatorDefault,
flags: 0,
bufferList: audioData)
if status != noErr {
print("Error cCMSampleBufferSetDataBufferFromAudioBufferList :\(String(describing: status?.description))")
return
}
let currentSampleTime = CMSampleBufferGetOutputPresentationTimeStamp(sbuf!);
print(" audio buffer at time: \(currentSampleTime)")
self.delegate?.videoFrameRefresh(sampleBuffer: sbuf!)
}
}
How I use my class
self.inputVideoMedia = nil
self.inputVideoMedia = VideoMediaInput(url: videoURL)
self.inputVideoMedia!.delegate = self
the second time I do that.. it crashes (but not always). The times it doesnt crash I can see printed in the console the FINALIZE print.
If VideoMediaInput is deallocated before the tap is deallocated (which can happen as there seems to be no way to synchronously stop a tap), then the tap callback can choke on a reference to your deallocated class.
You can fix this by passing (a wrapped, I guess) weak reference to your class. You can do it like this:
First delete your tap instance variable, and any references to it - it's not needed. Then make these changes:
class VideoMediaInput: NSObject {
class TapCookie {
weak var input: VideoMediaInput?
deinit {
print("TapCookie deinit")
}
}
...
func setupProcessingTap(){
let cookie = TapCookie()
cookie.input = self
var callbacks = MTAudioProcessingTapCallbacks(
version: kMTAudioProcessingTapCallbacksVersion_0,
clientInfo: UnsafeMutableRawPointer(Unmanaged.passRetained(cookie).toOpaque()),
init: tapInit,
finalize: tapFinalize,
prepare: tapPrepare,
unprepare: tapUnprepare,
process: tapProcess)
...
let tapFinalize: MTAudioProcessingTapFinalizeCallback = {
(tap) in
print("finalize \(tap)\n")
// release cookie
Unmanaged<TapCookie>.fromOpaque(MTAudioProcessingTapGetStorage(tap)).release()
}
let tapPrepare: MTAudioProcessingTapPrepareCallback = {
(tap, itemCount, basicDescription) in
print("prepare: \(tap, itemCount, basicDescription)\n")
let cookie = Unmanaged<TapCookie>.fromOpaque(MTAudioProcessingTapGetStorage(tap)).takeUnretainedValue()
let selfMediaInput = cookie.input!
...
let tapProcess: MTAudioProcessingTapProcessCallback = {
(tap, numberFrames, flags, bufferListInOut, numberFramesOut, flagsOut) in
print("callback \(bufferListInOut)\n")
let cookie = Unmanaged<TapCookie>.fromOpaque(MTAudioProcessingTapGetStorage(tap)).takeUnretainedValue()
guard let selfMediaInput = cookie.input else {
print("Tap callback: VideoMediaInput was deallocated!")
return
}
...
I'm not sure if the cookie class is necessary, it exists only to wrap the weak reference. Cutting edge Swift experts may know how to mash the weakness through all the teenage mutant ninja raw pointers, but I don't.
The audio context runs in its own real-time thread. So audio processes don't stop synchronously with a stop or cancel function call, but some unknown time later (on the order of the duration of some number of audio samples in some internal audio buffers), after the real-time thread drains.
Thus, audio buffers, objects, and callbacks should not be released (or reallocated) until some (unknown, but less than a couple seconds) time after stopping any real-time audio stream.
Depending on deallocation object messages or instance variable states (including weak references) betweens real-time threads is reported to be currently unsafe in Swift (see WWDC 2018 session on audio). Thus, I recommend using semaphores (outside of a real-time context, such as audio), or posix memory barriers (inside a bridged call to a C function). (...until some future version of Swift figures out a real-time concurrency mechanism.) (...especially on iOS or Apple Silicon (M1) devices which can re-order memory writes).

Allowing background audio with Swift not working

I want to allow background audio while the app is not in focus. I currently have this code, which should allow that:
do {
try AKSettings.setSession(category: .playback, with: .mixWithOthers)
} catch {
print("error")
}
AKSettings.playbackWhileMuted = true
I also have the setting 'Audio, Airplay and Picture in Picture' enabled in capabilities settings. However, when I press the home button on my device the audio doesn't keep playing. What am I doing wrong? I am using AudioKit to produce sounds if that matters.
I am using a singleton to house all of the AudioKit components which I named AudioPlayer.swift. Here is what I have in my AudioPlayer.swift singleton file:
class AudioPlayer: NSObject {
var currentFrequency = String()
var soundIsPlaying = false
var leftOscillator = AKOscillator()
var rightOscillator = AKOscillator()
var rain = try! AKAudioFile()
var rainPlayer: AKAudioPlayer!
var envelope = AKAmplitudeEnvelope()
override init() {
super.init()
do {
try AKSettings.setSession(category: .playback, with: .mixWithOthers)
} catch {
print("error")
}
AKSettings.playbackWhileMuted = true
AudioKit.output = envelope
AudioKit.start()
}
func setupFrequency(left: AKOscillator, right: AKOscillator, frequency: String) {
currentFrequency = frequency
leftOscillator = left
rightOscillator = right
let leftPanner = AKPanner(leftOscillator)
leftPanner.pan = -1
let rightPanner = AKPanner(rightOscillator)
rightPanner.pan = 1
//Set up rain and rainPlayer
do {
rain = try AKAudioFile(readFileName: "rain.wav")
rainPlayer = try AKAudioPlayer(file: rain, looping: true, deferBuffering: false, completionHandler: nil)
} catch { print(error) }
let mixer = AKMixer(leftPanner, rightPanner, rainPlayer)
//Put mixer in sound envelope
envelope = AKAmplitudeEnvelope(mixer)
envelope.attackDuration = 2.0
envelope.decayDuration = 0
envelope.sustainLevel = 1
envelope.releaseDuration = 0.2
//Start AudioKit stuff
AudioKit.output = envelope
AudioKit.start()
leftOscillator.start()
rightOscillator.start()
rainPlayer.start()
envelope.start()
soundIsPlaying = true
}
}
And here is an example of one of my sound effect view controllers, which reference the AudioKit singleton to send it a certain frequency (I have about a dozen of these view controllers, each with its own frequency settings):
class CalmView: UIViewController {
let leftOscillator = AKOscillator()
let rightOscillator = AKOscillator()
override func viewDidLoad() {
super.viewDidLoad()
leftOscillator.amplitude = 0.3
leftOscillator.frequency = 220
rightOscillator.amplitude = 0.3
rightOscillator.frequency = 230
}
#IBAction func playSound(_ sender: Any) {
if shared.soundIsPlaying == false {
AudioKit.stop()
shared.setupFrequency(left: leftOscillator, right: rightOscillator, frequency: "Calm")
} else if shared.soundIsPlaying == true && shared.currentFrequency != "Calm" {
AudioKit.stop()
shared.leftOscillator.stop()
shared.rightOscillator.stop()
shared.rainPlayer.stop()
shared.envelope.stop()
shared.setupFrequency(left: leftOscillator, right: rightOscillator, frequency: "Calm")
} else {
shared.soundIsPlaying = false
shared.envelope.stop()
}
}
}
I instantiated the AudioPlayer singleton in my ViewController.swift file.
It depends on when you are doing your configuration in relation to when AudioKit is started. If you're using AudioKit you should be using its AKSettings to manage your session category. Basically not only the playback category but also mixWithOthers. By default, does this:
/// Set the audio session type
#objc open static func setSession(category: SessionCategory,
with options: AVAudioSessionCategoryOptions = [.mixWithOthers]) throws {
So you'd do something like this in your ViewController:
do {
if #available(iOS 10.0, *) {
try AKSettings.setSession(category: .playAndRecord, with: [.defaultToSpeaker, .allowBluetooth, .allowBluetoothA2DP])
} else {
// Fallback on earlier versions
}
} catch {
print("Errored setting category.")
}
So I think its a matter of getting that straight. It might also help to have inter-app audio set up. If you still have trouble and provide more information, I can help more, but this is as good an answer as I can muster based on the info you've given so far.

MTAudioProcessingTap with kMTAudioProcessingTapCreationFlag_PostEffects not reflecting AVAudioMix volume

I am trying to build level metering for AVPlayer. I am doing this with an MTAudioProcessingTap that gets passed to an AVAudioMix, which in turns gets passed to the AVPlayerItem. The MTAudioProcessingTap gets created with the kMTAudioProcessingTapCreationFlag_PostEffects flag.
Technical Q&A QA1783 has the following to say about the PreEffects and PostEffects flags:
When you create a "pre-effects" audio tap using the kMTAudioProcessingTapCreationFlag_PreEffects flag, the tap will be called before any effects specified by AVAudioMixInputParameters are applied; when you create a "post-effects" tap by using the kMTAudioProcessingTapCreationFlag_PostEffects flag, the tap will be called after those effects are applied. Currently the only "effect" that AVAudioMixInputParameters supports is a linear volume ramp.
The problem:
When created with the kMTAudioProcessingTapCreationFlag_PostEffects, I would expect the that samples received by the MTAudioProcessingTap would reflect the the volume or audio ramps set on the AVAudioMixInputParameters. For example, if I set the volume to 0, I would expect to get all 0 samples. However the samples I receive seem to be totally unaffected by the volume or volume ramps.
Am I doing something wrong?
Here is a quick an dirty playground that illustrates the problem. The example sets the volume directly, but I observed the same problem when using audio ramps. Tested on both macOS and iOS:
import Foundation
import XCPlayground
import PlaygroundSupport
import AVFoundation
import Accelerate
PlaygroundPage.current.needsIndefiniteExecution = true;
let assetURL = Bundle.main.url(forResource: "sample", withExtension: "mp3")!
let asset = AVAsset(url: assetURL)
let playerItem = AVPlayerItem(asset: asset)
var audioMix = AVMutableAudioMix()
// The volume. Set to > 0 to hear something.
let kVolume: Float = 0.0
var parameterArray: [AVAudioMixInputParameters] = []
for assetTrack in asset.tracks(withMediaType: .audio) {
let parameters = AVMutableAudioMixInputParameters(track: assetTrack);
parameters.setVolume(kVolume, at: kCMTimeZero)
parameterArray.append(parameters)
// Omitting most callbacks to keep sample short:
var callbacks = MTAudioProcessingTapCallbacks(
version: kMTAudioProcessingTapCallbacksVersion_0,
clientInfo: nil,
init: nil,
finalize: nil,
prepare: nil,
unprepare: nil,
process: { (tap, numberFrames, flags, bufferListInOut, numberFramesOut, flagsOut) in
guard MTAudioProcessingTapGetSourceAudio(tap, numberFrames, bufferListInOut, flagsOut, nil, numberFramesOut) == noErr else {
preconditionFailure()
}
// Assume 32bit float format, native endian:
for i in 0..<bufferListInOut.pointee.mNumberBuffers {
let buffer = bufferListInOut.pointee.mBuffers
let stride: vDSP_Stride = vDSP_Stride(buffer.mNumberChannels)
let numElements: vDSP_Length = vDSP_Length(buffer.mDataByteSize / UInt32(MemoryLayout<Float>.stride))
for j in 0..<Int(buffer.mNumberChannels) {
// Use vDSP_maxmgv tof ind the maximum amplitude
var start = buffer.mData!.bindMemory(to: Float.self, capacity: Int(numElements))
start += Int(j * MemoryLayout<Float>.stride)
var magnitude: Float = 0
vDSP_maxmgv(start, stride, &magnitude, numElements - vDSP_Length(j))
DispatchQueue.main.async {
print("buff: \(i), chan: \(j), max: \(magnitude)")
}
}
}
}
)
var tap: Unmanaged<MTAudioProcessingTap>?
guard MTAudioProcessingTapCreate(kCFAllocatorDefault, &callbacks, kMTAudioProcessingTapCreationFlag_PostEffects, &tap) == noErr else {
preconditionFailure()
}
parameters.audioTapProcessor = tap?.takeUnretainedValue()
}
audioMix.inputParameters = parameterArray
playerItem.audioMix = audioMix
let player = AVPlayer(playerItem: playerItem)
player.rate = 1.0

How to disable audio in webrtc mobile app(ios) without changing in framework

I am working with webrtc mobile(ios). I can't disable audio in webrtc(ios). I have got no flag to disable audio.By changing in framwork/library it can done easily. My purpose is that I have to disable audio without changing in framework/library. Can anyone help me?.
Update your question with code snippet, how you are creating mediaStrem or tracks(audio/video).
Generally with default Native WebRTC Framework,
RTCMediaStream localStream = [_factory mediaStreamWithStreamId:kARDMediaStreamId];
if(audioRequired) {
RTCAudioTrack *aTrack = [_lmStream createLocalAudioTrack];
[localStream addAudioTrack:aTrack];
}
RTCVideoTrack *vTrack = [_lmStream createLocalVideoTrack];
[localStream addVideoTrack:vTrack];
[_peerConnection addStream:localStream];
If you want to mute the Audio during the call, use below function.
- (void)enableAudio:(NSString *)id isAudioEnabled:(BOOL) isAudioEnabled {
NSLog(#"Auido enabled: %d streams count:%d ", id, isAudioEnabled, _peerConnection.localStreams.count);
if(_peerConnection.localStreams.count > 0) {
RTCMediaStream *lStream = _peerConnection.localStreams[0];
if(lStream.audioTracks.count > 0) { // Usually we will have only one track. If you have more than one, need to traverse all.
// isAudioEnabled == 1 -> Unmute
// isAudioEnabled == 0 -> Mute
[lStream.audioTracks[0] setIsEnabled:isAudioEnabled];
}
}
}
in my case I didnt use streams and directly add audio track to peerconnection.
private func createMediaSenders() {
let streamId = "stream"
// Audio
let audioTrack = self.createAudioTrack()
self.pc.add(audioTrack, streamIds: [streamId])
// Video
/* let videoTrack = self.createVideoTrack()
self.localVideoTrack = videoTrack
self.peerConnection.add(videoTrack, streamIds: [streamId])
self.remoteVideoTrack = self.peerConnection.transceivers.first { $0.mediaType == .video }?.receiver.track as? RTCVideoTrack
// Data
if let dataChannel = createDataChannel() {
dataChannel.delegate = self
self.localDataChannel = dataChannel
}*/
}
private func createAudioTrack() -> RTCAudioTrack {
let audioConstrains = RTCMediaConstraints(mandatoryConstraints: nil, optionalConstraints: nil)
let audioSource = sessionFactory.audioSource(with: audioConstrains)
let audioTrack = sessionFactory.audioTrack(with: audioSource, trackId: "audio0")
return audioTrack
}
to mute and unmute microphone I use this function
public func muteMicrophone(_ mute:Bool){
for sender in pc.senders{
if (sender.track?.kind == "audio") {
sender.track?.isEnabled = mute
}
}
}

iOS Adjust Pitch Whilst Playing via AVAudioUnitTimePitch

I’m trying to get some audio to be able to have the pitch adjusted whilst playing. I’m very new to Swift and iOS, but my initial attempt was to just change timePitchNode.pitch whilst it was playing; however, it wouldn’t update whilst playing. My current attempt is to reset audioEngine, and have it just resume from where it was playing (below). How do I determine where the audio currently is, and how do I get it to resume from there?
var audioFile: AVAudioFile?
var audioEngine: AVAudioEngine?
var audioPlayerNode: AVAudioPlayerNode?
var pitch: Int = 1 {
didSet {
playResumeAudio()
}
}
…
func playResumeAudio() {
var currentTime: AVAudioTime? = nil
if audioPlayerNode != nil {
let nodeTime = audioPlayerNode!.lastRenderTime!
currentTime = audioPlayerNode!.playerTimeForNodeTime(nodeTime)
}
if audioEngine != nil {
audioEngine!.stop()
audioEngine!.reset()
}
audioEngine = AVAudioEngine()
audioPlayerNode = AVAudioPlayerNode()
audioEngine!.attachNode(audioPlayerNode!)
let timePitchNode = AVAudioUnitTimePitch()
timePitchNode.pitch = Float(pitch * 100)
timePitchNode.rate = rate
audioEngine!.attachNode(timePitchNode)
audioEngine!.connect(audioPlayerNode!, to: timePitchNode, format: nil)
audioEngine!.connect(timePitchNode, to: audioEngine!.outputNode, format: nil)
audioPlayerNode!.scheduleFile(audioFile!, atTime: nil, completionHandler: nil)
let _ = try? audioEngine?.start()
audioPlayerNode!.playAtTime(currentTime)
}
I was being dumb apparently. You can modify the pitch during playback, and it does update. No need to reset any audio, just mutate the node as it’s playing, and it’ll work.

Resources