I have an AVAudioCompressedBuffer instance that gets correctly decoded and played by my AVAudioEngine.
The problem is that after converting it to Data and then back to AVAudioCompressedBuffer it is no longer playable and throws a kAudioCodecBadDataError.
This is how I'm currently managing the conversion to and from Data:
// Convert AVAudioCompressedBuffer to Data
let capacity = Int(compressedBuffer.byteLength)
let compressedBufferPointer = compressedBuffer.data.bindMemory(to: UInt8.self, capacity: capacity)
var compressedBytes: [UInt8] = [UInt8].init(repeating: 0, count: capacity)
compressedBufferPointer.withMemoryRebound(to: UInt8.self, capacity: capacity) { sourceBytes in
compressedBytes.withUnsafeMutableBufferPointer {
$0.baseAddress!.initialize(from: sourceBytes, count: capacity)
}
}
let data = Data(compressedBytes)
// Convert Data to AVAudioCompressedBuffer
let compressedBuffer: AVAudioCompressedBuffer = AVAudioCompressedBuffer.init(format: format, packetCapacity: packetCapacity, maximumPacketSize: maximumPacketSize)
compressedBuffer.byteLength = byteLength
compressedBuffer.packetCount = packetCount
data.withUnsafeBytes {
compressedBuffer.data.copyMemory(from: $0.baseAddress!, byteCount: byteLength)
}
let buffer = compressedBuffer
The values for all of the buffer attributes (format, packetCapacity, maximumPacketSize, byteLength, packetCount, byteLength) are the same on both ends of the conversion.
It turns out that, for some reason, converting AVAudioCompressedBuffer that way fails to include the buffer's packetDescriptions. These are stored as a C-Style array of AudioStreamPacketDescriptions structs in the buffer. By creating a Codable struct (PacketDescription) and mapping the descriptions objects separately the reassembled buffer worked as expected.
var packetDescriptions = [PacketDescription]()
for index in 0..<compressedBuffer.packetCount {
if let packetDescription = compressedBuffer.packetDescriptions?[Int(index)] {
packetDescriptions.append(
.init(mStartOffset: packetDescription.mStartOffset,
mVariableFramesInPacket: packetDescription.mVariableFramesInPacket,
mDataByteSize: packetDescription.mDataByteSize))
}
}
packetDescriptions?.enumerated().forEach { index, element in
compressedBuffer.packetDescriptions?[index] = AudioStreamPacketDescription(mStartOffset: element.mStartOffset!,
mVariableFramesInPacket: element.mVariableFramesInPacket!,
mDataByteSize: element.mDataByteSize!)
}
Related
I'm working with a custom network protocol that requires a precise bytes sequence to do a sort of handshake, as example the first call should send a body like:
0003joy that, translated in a [UInt8] should be [0x00,0x00,0x00,0x03,0x6a,0x6f,0x79]
(please note that the first 4 numbers should not be converted to char... I'm sending the numeric value, as per protocol request)
I'm trying to send this sequence to an outputstream but I'm wondering if the steps I'm following are correct, here is my code... do you see anything strange that might prevent this sequence to reach the server?
// Create Bytes sequence
let bytes:[UInt8] = [0x00,0x00,0x00,0x03,0x6a,0x6f,0x79]
// Convert Bytes array do Data
let dt = Data(bytes)
// Send Data to stream
_ = dt.withUnsafeBytes {
guard let pointer = $0.baseAddress?.assumingMemoryBound(to: UInt8.self) else {
return
}
outputStream.write(pointer, maxLength: dt.count)
}
Also, do I need to convert the bytes sequence to Data? is there a way to send bytes sequence directly into the socket without converting it into Data?
I don't see anything wrong with your code but the conversion to Data is not needed:
bytes.withUnsafeBytes {
guard let pointer = $0.baseAddress?.assumingMemoryBound(to: UInt8.self) else {
return
}
outputStream.write(pointer, maxLength: bytes.count)
}
since [UInt8] conforms to ContiguousBytes.
Actually, the following is also possible:
bytes.withUnsafeBufferPointer {
guard let baseAddress = $0.baseAddress else { return }
outputStream.write(baseAddress, maxLength: bytes.count)
}
I am reading data from an MFi external device into a buffer using a 3rd party SDK "sessionController". See below:
let handle: UInt64 = self.sessionController.openFile(file.path, mode: openMode)
if handle == 0 {
//Error
return
}
let c: UInt64 = file.size
var bytesArray: [UInt8] = [UInt8](fileData)
let bufferPointer: UnsafeMutablePointer<UInt8> = UnsafeMutablePointer<UInt8>.allocate(capacity: Int(c))
bufferPointer.initialize(repeating: 0, count: Int(c))
defer {
bufferPointer.deinitialize(count: Int(c))
bufferPointer.deallocate()
}
var sum: UInt32 = 0
let singleSize: UInt32 = 8 << 20
while sum < c {
let read = self.sessionController.readFile(handle, data: bufferPointer, len: singleSize)
if read == 0 {
//There was an error
return
}
sum += read
}
let newPointer : UnsafeRawPointer = UnsafeRawPointer(bufferPointer)
fileURL = try! FileManager.default.url(for: .documentDirectory, in: .userDomainMask, appropriateFor: nil, create: false).appendingPathComponent("test.MOV")
fileData = Data(bytes: newPointer, count: Int(c))
try! fileData.write(to: fileURL)
//Now use this fileURL to watch video in an AVPlayer...
//AVPlayer(init: fileURL)
For some reason the data stored at the fileURL becomes corrupted (I think) and I am unable to play the video file. I think I am not doing something correctly with Unsafe Swift but I am not sure what. How can I make sure that I have properly read the data from the device into memory, and then taken that data from memory and stored it on the hard drive at the fileURL? What am I doing wrong here? The video will not play in AVPlayer given the fileURL.
The main error is here:
let read = self.sessionController.readFile(handle, data: bufferPointer, len: singleSize)
If you read in multiple chunks then the second and all subsequent reads will overwrite the data read previously. So that should probably be
let read = self.sessionController.readFile(handle, data: bufferPointer + sum, len: singleSize)
Note also that the file size is defined as UInt64, but the variable sum (which holds the total number of bytes read so far) is an UInt32. This will lead to problems if there is more than 4GB data.
But generally I would avoid to read the complete data into a memory buffer. You already read in chunks, so you can write the data immediately to the destination file. Here is how that could look like:
// Output file:
let fileURL = ...
let fileHandle = try FileHandle(forWritingTo: fileURL)
defer { fileHandle.closeFile() }
// Buffer:
let bufferSize = 1024 * 1024 // Choose some buffer size
var buffer = Data(count: bufferSize)
// Read/write loop:
let fileSize: UInt64 = file.size
var remainingToRead = fileSize
while remainingToRead > 0 {
let read = buffer.withUnsafeMutableBytes { bufferPointer in
self.sessionController.readFile(handle, data: bufferPointer, len: UInt32(min(remainingToRead, UInt64(bufferSize))))
}
if read == 0 {
return // Read error
}
remainingToRead -= UInt64(read)
fileHandle.write(buffer)
}
Note also that the data is read directly into a Data value, instead of reading it into allocated memory and then copying it to another Data.
I am trying to process audio data in real-time so that I can display an on-screen spectrum analyzer/visualization based on sound input from the microphone. I am using AVFoundation's AVCaptureAudioDataOutputSampleBufferDelegate to capture the audio data, which is triggering the delgate function captureOutput. Function below:
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
autoreleasepool {
guard captureOutput != nil,
sampleBuffer != nil,
connection != nil,
CMSampleBufferDataIsReady(sampleBuffer) else { return }
//Check this is AUDIO (and not VIDEO) being received
if (connection.audioChannels.count > 0)
{
//Determine number of frames in buffer
var numFrames = CMSampleBufferGetNumSamples(sampleBuffer)
//Get AudioBufferList
var audioBufferList = AudioBufferList(mNumberBuffers: 1, mBuffers: AudioBuffer(mNumberChannels: 0, mDataByteSize: 0, mData: nil))
var blockBuffer: CMBlockBuffer?
CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(sampleBuffer, nil, &audioBufferList, MemoryLayout<AudioBufferList>.size, nil, nil, UInt32(kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment), &blockBuffer)
let audioBuffers = UnsafeBufferPointer<AudioBuffer>(start: &audioBufferList.mBuffers, count: Int(audioBufferList.mNumberBuffers))
for audioBuffer in audioBuffers {
let data = Data(bytes: audioBuffer.mData!, count: Int(audioBuffer.mDataByteSize))
let i16array = data.withUnsafeBytes {
UnsafeBufferPointer<Int16>(start: $0, count: data.count/2).map(Int16.init(bigEndian:))
}
for dataItem in i16array
{
print(dataItem)
}
}
}
}
}
The code above prints positive and negative numbers of type Int16 as expected, but need help in converting these raw numbers into meaningful data such as power and decibels for my visualizer.
I was on the right track... Thanks to RobertHarvey's comment on my question - Use of the Accelerate Framework's FFT calculation functions is required to achieve a spectrum analyzer. But even before I could use these functions, you need to convert your raw data into an Array of type Float as many of the functions require a Float array.
Firstly, we load the raw data into a Data object:
//Read data from AudioBuffer into a variable
let data = Data(bytes: audioBuffer.mData!, count: Int(audioBuffer.mDataByteSize))
I like to think of a Data object as a "list" of 1-byte sized chunks of info (8 bits each), but if I check the number of frames I have in my sample and the total size of my Data object in bytes, they don't match:
//Get number of frames in sample and total size of Data
var numFrames = CMSampleBufferGetNumSamples(sampleBuffer) //= 1024 frames in my case
var dataSize = audioBuffer.mDataByteSize //= 2048 bytes in my case
The total size (in bytes) of my data is twice the number of frames I have in my CMSampleBuffer. This means that each frame of audio is 2 bytes in length. In order to read the data meaningfully, I need to convert my Data object which is a "list" of 1-byte chunks into an array of 2-byte chunks. Int16 contains 16 bits (or 2 bytes - exactly what we need), so lets create an Array of Int16:
//Convert to Int16 array
let samples = data.withUnsafeBytes {
UnsafeBufferPointer<Int16>(start: $0, count: data.count / MemoryLayout<Int16>.size)
}
Now that we have an Array of Int16, we can convert it to an Array of Float:
//Convert to Float Array
let factor = Float(Int16.max)
var floats: [Float] = Array(repeating: 0.0, count: samples.count)
for i in 0..<samples.count {
floats[i] = Float(samples[i]) / factor
}
Now that we have our Float array, we can now use the Accelerate Framework's complex math to convert the raw Float values into meaningful ones like magnitude, decibels etc. Link to documentation:
Apple's Accelerate Framework
Fast Fourier Transform (FFT)
I found Apple's documentation rather overwhelming. Luckily, I found a really good example online which I was able to re-purpose for my needs, called TempiFFT. Implementation as follows:
//Initiate FFT
let fft = TempiFFT(withSize: numFrames, sampleRate: 44100.0)
fft.windowType = TempiFFTWindowType.hanning
//Pass array of Floats
fft.fftForward(floats)
//I only want to display 20 bands on my analyzer
fft.calculateLinearBands(minFrequency: 0, maxFrequency: fft.nyquistFrequency, numberOfBands: 20)
//Then use a loop to iterate through the bands in your spectrum analyzer
var magnitudeArr = [Float](repeating: Float(0), count: 20)
var magnitudeDBArr = [Float](repeating: Float(0), count: 20)
for i in 0..<20
{
var magnitudeArr[i] = fft.magnitudeAtBand(i)
var magnitudeDB = TempiFFT.toDB(fft.magnitudeAtBand(i))
//..I didn't, but you could perform drawing functions here...
}
Other useful references:
Converting Data into Array of Int16
Converting Array of Int16 to Array of Float
I have a question about how I could do this instruction in swift?
NSData * data = characteristic.value;
Byte *resultByte = (Byte *)[data bytes];
I understand that the first line is like this, but how can I get the bytes
let data = characteristic.value! as NSData
You can create an array of bytes from the data simply with
if let data = characteristic.value {
let bytes = Array(data) // [UInt8]
}
But often you don't need to create an extra array because Data
is a collection and you can directly access the individual bytes via
subscripting:
if let data = characteristic.value {
let byte0 = data[0]
let byte1 = data[1]
// ...
}
or get a pointer to the raw bytes with
if let data = characteristic.value {
data.withUnsafeBytes { (bytePtr: UnsafePointer<UInt8>) in
// ...
}
}
I am struggling to initialize the MIDIMetaEvent structure found in MusicPlayer.h with swift The header file defines the structure as follows:
struct MIDIMetaEvent {
var metaEventType: UInt8
var unused1: UInt8
var unused2: UInt8
var unused3: UInt8
var dataLength: UInt32
var data: (UInt8)
}
Which seems fairly straightforward up until that 'data' member. Is that a 1 element tuple definition? I can easily initialize all other struct elements but have tried in vain to set 'data' to anything else than a single value. In my code I used an UInt8 array called myData and attempted to init the structure like so:
var msg = MIDIMetaEvent(
metaEventType : UInt8(0x7F),
unused1 : UInt8(0),
unused2 : UInt8(0),
unused3 : UInt8(0),
dataLength : UInt32(myData.count),
data : UnsafeBufferPointer<UInt8>(start: UnsafePointer<UInt8>(myData), count:myData.count) )
But the compiler is not happy with this and complains about "UnsafeBufferPointer no convertible to UInt8". If I simply set data to a single value but set dataLength to a value more than 1, the resulting MIDIEventData shows that the first value in the event is what I stuck in 'data' followed by gibberish data bytes in accordance with 'dataLength' bytes. So clearly 'data' is seen as some sort of continuous memory.
So how do I set that 'data' element to UInt8 elements from an array?
The AudioToolbox framework defines MIDIMetaEvent as
typedef struct MIDIMetaEvent
{
UInt8 metaEventType;
UInt8 unused1;
UInt8 unused2;
UInt8 unused3;
UInt32 dataLength;
UInt8 data[1];
} MIDIMetaEvent;
where data[1] is actually used as a "variable length array".
In (Objective-)C one can just allocate a pointer to a memory block of the
actually needed size:
MIDIMetaEvent *mep = malloc(sizeof(MIDIMetaEvent) + data.count);
Swift is more strict with pointer casts and fixed size arrays are mapped to
Swift tuples (which can be cumbersome to handle with).
The following utility class shows how this could be solved:
class MyMetaEvent {
private let size: Int
private let mem : UnsafeMutablePointer<UInt8>
let metaEventPtr : UnsafeMutablePointer<MIDIMetaEvent>
init(type: UInt8, data: [UInt8]) {
// Allocate memory of the required size:
size = sizeof(MIDIMetaEvent) + data.count
mem = UnsafeMutablePointer<UInt8>.alloc(size)
// Convert pointer:
metaEventPtr = UnsafeMutablePointer(mem)
// Fill data:
metaEventPtr.memory.metaEventType = type
metaEventPtr.memory.dataLength = UInt32(data.count)
memcpy(mem + 8, data, UInt(data.count))
}
deinit {
// Release the allocated memory:
mem.dealloc(size)
}
}
Then you can create an instance with
let me = MyMetaEvent(type: 0x7F, data: myData)
and pass me.metaEventPtr to the Swift functions taking a UnsafePointer<MIDIMetaEvent>
argument.
Update for Swift 3/4:
Simply converting a pointer to a different type is no longer possible,
it must be "rebound":
class MyMetaEvent {
private let size: Int
private let mem: UnsafeMutablePointer<UInt8>
init(type: UInt8, data: [UInt8]) {
// Allocate memory of the required size:
size = MemoryLayout<MIDIMetaEvent>.size + data.count
mem = UnsafeMutablePointer<UInt8>.allocate(capacity: size)
mem.initialize(to: 0, count: size)
// Fill data:
mem.withMemoryRebound(to: MIDIMetaEvent.self, capacity: 1) { metaEventPtr in
metaEventPtr.pointee.metaEventType = type
metaEventPtr.pointee.dataLength = UInt32(data.count)
memcpy(&metaEventPtr.pointee.data, data, data.count)
}
}
deinit {
// Release the allocated memory:
mem.deallocate(capacity: size)
}
func withMIDIMetaEventPtr(body: (UnsafePointer<MIDIMetaEvent>) -> Void) {
mem.withMemoryRebound(to: MIDIMetaEvent.self, capacity: 1) { metaEventPtr in
body(metaEventPtr)
}
}
}
Create an instance with custom data:
let me = MyMetaEvent(type: 0x7F, data: ...)
Pass to a function taking a UnsafePointer<MIDIMetaEvent> argument:
me.withMIDIMetaEventPtr { metaEventPtr in
let status = MusicTrackNewMetaEvent(track, 0, metaEventPtr)
}