Im stuck on an issue on my objective C App.
I'm reading a byte array from a serveur (Socket c#) who send me an PCM encoded sound, and i'm currently looking for a sample code that decode for me this byte array (NSData), and play it.
Does anyone know a solution ? Or how can I read a u-Law audio?
Thanks a lot ! :D
This link has information about mu-law encoding and decoding:
http://dystopiancode.blogspot.com.es/2012/02/pcm-law-and-u-law-companding-algorithms.html
#define MULAW_BIAS 33
/*
* Description:
* Decodes an 8-bit unsigned integer using the mu-Law.
* Parameters:
* number - the number who will be decoded
* Returns:
* The decoded number
*/
int16_t MuLaw_Decode(int8_t number)
{
uint8_t sign = 0, position = 0;
int16_t decoded = 0;
number=~number;
if(number&0x80)
{
number&=~(1<<7);
sign = -1;
}
position = ((number & 0xF0) >>4) + 5;
decoded = ((1<<position)|((number&0x0F)<<(position-4))|(1<<(position-5)))
- MULAW_BIAS;
return (sign==0)?(decoded):(-(decoded));
}
When you have the uncompressed audio you should be able to play it using the Audio Queue API.
Good luck!
Related
I had a need to transmit sound over the network and for this I chose libraries "PortAudio" and "Opus". I am new to working with sound and therefore i don’t know many thing.I am new to working with sound and therefore i don’t know many things, but i read the documentation and looked at some examples, but i still have some problems with encoding/decoding with Opus. I do not understand how to correctly restore the original encoded PСM.I have some sequence of actions:
Some consts
const int FRAMES_PER_BUFFER = 960;
const int SAMPLE_RATE = 48000;
int NUM_CHANNELS = 2;
int totalFrames = 2 * SAMPLE_RATE; /* Record for a few seconds. */
int numSamples = totalFrames * 2;
int numBytes = numSamples * sizeof(float);
float *sampleBlock = nullptr;
int bytesOfPacket = 0;
unsigned char *packet = nullptr;
I get PСM to sampleBlock
paError = Pa_ReadStream(**&stream, sampleBlock, totalFrames);
if (paError != paNoError) {
cout << "PortAudio error : " << Pa_GetErrorText(paError) << endl;
std::system("pause");
}
Encoding sampleBlock
OpusEncoder *encoder;
int error;
int size;
encoder = opus_encoder_create(SAMPLE_RATE, NUM_CHANNELS, OPUS_APPLICATION_VOIP, &error);
size = opus_encoder_get_size(NUM_CHANNELS);
encoder = (OpusEncoder *)malloc(size);
packet = new unsigned char[480];
error = opus_encoder_init(encoder, SAMPLE_RATE, NUM_CHANNELS, OPUS_APPLICATION_VOIP);
if (error == -1) {
return -1;
}
bytesOfPacket = opus_encode_float(encoder, sampleBlock, FRAMES_PER_BUFFER, packet, 480);
opus_encoder_destroy(encoder);
Ok, i received a encoded packet to Opus
Decoding
OpusDecoder *decoder;
int error;
int size;
decoder = opus_decoder_create(SAMPLE_RATE, NUM_CHANNELS, &error);
size = opus_decoder_get_size(NUM_CHANNELS);
decoder = (OpusDecoder *)malloc(size);
error = opus_decoder_init(decoder, SAMPLE_RATE, NUM_CHANNELS);
opus_decode_float(decoder, packet, bytesOfPacket, sampleBlock, 480, 0);
opus_decoder_destroy(decoder);
Here i am trying to decode the Opus back to the PCM and save the result to the sampleBlock
Playing the sound
paError = Pa_WriteStream(**&stream, sampleBlock, totalFrames);
if (paError != paNoError) {
cout << "PortAudio error : " << Pa_GetErrorText(paError) << endl;
std::system("pause");
}
I get silence. I don't really understand the subtleties in working with sound since i am new to this business. Help please understand what is wrong.
As for your settings you're encoding 20ms of audio per opus_encode_float call. I don't see any iteration over this call so I suppose you don't hear anything because you encode only 20ms of audio. You should pass to opus_encode_float 20ms worth of samples with your sampleBlock pointer incrementing it through the whole buffer x times.
Try to encode more audio and remember that you have to add some sort of framing to decode it. You cannot just feed the whole buffer to the decoder. You should feed the decoder one time for each encoder call with the same data that each encoder call outputs.
Damiano
Here is an example of turning 4 bytes into a 32bit integer in objective-c. The function readInt grabs 4 bytes from the read function and then converts it into a single 32 bit int. Does anyone know how I would convert 4 bytes to a float? I believe it is big endian. Basically I need a readFloat function. I can never grasp these bitwise operations.
EDIT:
I forgot to mention that the original data comes from Java's DataOutputStream class. The writeFloat function accordign to java doc is
Converts the float argument to an int using the floatToIntBits method
in class Float, and then writes that int value to the underlying
output stream as a 4-byte quantity, high byte first.
This is Objective c trying to extract the data written by java.
- (int32_t)read{
int8_t v;
[data getBytes:&v range:NSMakeRange(length,1)];
length++;
return ((int32_t)v & 0x0ff);
}
- (int32_t)readInt {
int32_t ch1 = [self read];
int32_t ch2 = [self read];
int32_t ch3 = [self read];
int32_t ch4 = [self read];
if ((ch1 | ch2 | ch3 | ch4) < 0){
#throw [NSException exceptionWithName:#"Exception" reason:#"EOFException" userInfo:nil];
}
return ((ch1 << 24) + (ch2 << 16) + (ch3 << 8) + (ch4 << 0));
}
OSByteOrder.h contains functions for reading, writing, and converting integer data.
You can use OSSwapBigToHostInt32() to convert a big-endian integer to the native representation, then copy the bits into a float:
NSData* data = [NSData dataWithContentsOfFile:#"/tmp/java/test.dat"];
int32_t bytes;
[data getBytes:&bytes length:sizeof(bytes)];
bytes = OSSwapBigToHostInt32(bytes);
float number;
memcpy(&number, &bytes, sizeof(bytes));
NSLog(#"Float %f", number);
[data getBytes:&myFloat range:NSMakeRange(locationWhereFloatStarts, sizeof(float)] ought to do the trick.
Given that the data comes from DataOutputStream's writeFloat() method, then that is documented to use Float.floatToIntBits() to create the integer representation. intBitsToFloat() further documents how to interpret that representation.
I'm not sure if it's the same thing, but the xdr API seems like it might handle that representation. The credits on the man page refer to Sun Microsystems standards/specifications, so it seems likely it's related to Java.
So, it may work to do something like:
// At top of file:
#include <rpc/types.h>
#include <rpc/xdr.h>
// In some function or method:
XDR xdr;
xdrmem_create(&xdr, (char*)data.bytes + offset, data.length - offset, XDR_DECODE);
float f;
if (!xdr_float(&xdr, &f))
/* handle error */;
xdr_destroy(&xdr);
If the data consists of a whole stream in eXternal Data Representation, then you would create one XDR stream for the whole task of extracting items from it, and use many xdr_...() calls between creating and destroying it to extract all of the items.
I'm using Apple Video Toolbox framework to compress raw frames captured by the device camera.
My callback is being called with a CMSampleBufferRef object that contains CMBlockBuffer.
The CMBlockBuffer object contain the H264 elementary stream but I didn't find any way to get a pointer to the elementary stream.
When I printed into the console the CMSampleBufferRef object I got:
(lldb) po blockBufferRef
CMBlockBuffer 0x1701193e0 totalDataLength: 4264 retainCount: 1 allocator: 0x1957c2c80 subBlockCapacity: 2
[0] 4264 bytes # offset 128 Buffer Reference:
CMBlockBuffer 0x170119350 totalDataLength: 4632 retainCount: 1 allocator: 0x1957c2c80 subBlockCapacity: 2
[0] 4632 bytes # offset 0 Memory Block 0x10295c000, 4632 bytes (custom V=0 A=0x0 F=0x18498bb44 R=0x0)
It seems that the CMBlockBuffer object that I managed to get pointer to is contaning another CMBlockBuferRef (4632 bytes) which is not accessible.
Can anyone post how to access the H264 elemantry stream?
Thank you!
I've been struggling with this myself for quite some time now, and have finally figured everything out.
The function CMBlockBufferGetDataPointer gives you access to all the data you need, but there are a few not very obvious things you need to do to convert it to an elementary stream.
AVCC vs Annex B format
The data in the CMBlockBuffer is stored in AVCC format, while elementary streams are typically following the Annex B specification (here is an excellent overview of the two formats). In the AVCC format, the 4 first bytes contains the length of the NAL unit (another word for H264 packet). You need to replace this header with the 4 byte start code: 0x00 0x00 0x00 0x01, which functions as a separator between NAL units in an Annex B elementary stream (the 3 byte version 0x00 0x00 0x01 works fine too).
Multiple NAL units in a single CMBlockBuffer
The next not very obvious thing is that a single CMBlockBuffer will sometimes contain multiple NAL units. Apple seems to add an additional NAL unit (SEI) containing metadata to every I-Frame NAL unit (also called IDR). This is probably why you are seeing multiple buffers in a single CMBlockBuffer object. However, the CMBlockBufferGetDataPointer function gives you a single pointer with access to all the data. That being said, the presence of multiple NAL units complicates the conversion of the AVCC headers. Now you actually have to read the length value contained in the AVCC header to find the next NAL unit, and continue converting headers until you have reached the end of the buffer.
Big-Endian vs Little-Endian
The next not very obvious thing is that the AVCC header is stored in Big-Endian format, and iOS is Little-Endian natively. So when you are reading the length value contained in an AVCC header pass it to the CFSwapInt32BigToHost function first.
SPS and PPS NAL units
The final not very obvious thing is that the data inside the CMBlockBuffer does not contain the parameter NAL units SPS and PPS, which contains configuration parameters for the decoder such as profile, level, resolution, frame rate. These are stored as metadata in the sample buffer's format description and can be accessed via the function CMVideoFormatDescriptionGetH264ParameterSetAtIndex. Note that you have to add the start codes to these NAL units before sending. The SPS and PPS NAL units does not have to be sent with every new frame. A decoder only needs to read them once, but it is common to resend them periodically, for example before every new I-frame NAL unit.
Code Example
Below is a code example taking all of these things into account.
static void videoFrameFinishedEncoding(void *outputCallbackRefCon,
void *sourceFrameRefCon,
OSStatus status,
VTEncodeInfoFlags infoFlags,
CMSampleBufferRef sampleBuffer) {
// Check if there were any errors encoding
if (status != noErr) {
NSLog(#"Error encoding video, err=%lld", (int64_t)status);
return;
}
// In this example we will use a NSMutableData object to store the
// elementary stream.
NSMutableData *elementaryStream = [NSMutableData data];
// Find out if the sample buffer contains an I-Frame.
// If so we will write the SPS and PPS NAL units to the elementary stream.
BOOL isIFrame = NO;
CFArrayRef attachmentsArray = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, 0);
if (CFArrayGetCount(attachmentsArray)) {
CFBooleanRef notSync;
CFDictionaryRef dict = CFArrayGetValueAtIndex(attachmentsArray, 0);
BOOL keyExists = CFDictionaryGetValueIfPresent(dict,
kCMSampleAttachmentKey_NotSync,
(const void **)¬Sync);
// An I-Frame is a sync frame
isIFrame = !keyExists || !CFBooleanGetValue(notSync);
}
// This is the start code that we will write to
// the elementary stream before every NAL unit
static const size_t startCodeLength = 4;
static const uint8_t startCode[] = {0x00, 0x00, 0x00, 0x01};
// Write the SPS and PPS NAL units to the elementary stream before every I-Frame
if (isIFrame) {
CMFormatDescriptionRef description = CMSampleBufferGetFormatDescription(sampleBuffer);
// Find out how many parameter sets there are
size_t numberOfParameterSets;
CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description,
0, NULL, NULL,
&numberOfParameterSets,
NULL);
// Write each parameter set to the elementary stream
for (int i = 0; i < numberOfParameterSets; i++) {
const uint8_t *parameterSetPointer;
size_t parameterSetLength;
CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description,
i,
¶meterSetPointer,
¶meterSetLength,
NULL, NULL);
// Write the parameter set to the elementary stream
[elementaryStream appendBytes:startCode length:startCodeLength];
[elementaryStream appendBytes:parameterSetPointer length:parameterSetLength];
}
}
// Get a pointer to the raw AVCC NAL unit data in the sample buffer
size_t blockBufferLength;
uint8_t *bufferDataPointer = NULL;
CMBlockBufferGetDataPointer(CMSampleBufferGetDataBuffer(sampleBuffer),
0,
NULL,
&blockBufferLength,
(char **)&bufferDataPointer);
// Loop through all the NAL units in the block buffer
// and write them to the elementary stream with
// start codes instead of AVCC length headers
size_t bufferOffset = 0;
static const int AVCCHeaderLength = 4;
while (bufferOffset < blockBufferLength - AVCCHeaderLength) {
// Read the NAL unit length
uint32_t NALUnitLength = 0;
memcpy(&NALUnitLength, bufferDataPointer + bufferOffset, AVCCHeaderLength);
// Convert the length value from Big-endian to Little-endian
NALUnitLength = CFSwapInt32BigToHost(NALUnitLength);
// Write start code to the elementary stream
[elementaryStream appendBytes:startCode length:startCodeLength];
// Write the NAL unit without the AVCC length header to the elementary stream
[elementaryStream appendBytes:bufferDataPointer + bufferOffset + AVCCHeaderLength
length:NALUnitLength];
// Move to the next NAL unit in the block buffer
bufferOffset += AVCCHeaderLength + NALUnitLength;
}
}
Thanks Anton for an excellent answer! Am putting a naive Swift-port of your solution for people interested in using the concepts discussed here straight in their Swift-based projects.
public func didEncodeFrame(frame: CMSampleBuffer)
{
print ("Received encoded frame in delegate...")
//----AVCC to Elem stream-----//
var elementaryStream = NSMutableData()
//1. check if CMBuffer had I-frame
var isIFrame:Bool = false
let attachmentsArray:CFArray = CMSampleBufferGetSampleAttachmentsArray(frame, false)!
//check how many attachments
if ( CFArrayGetCount(attachmentsArray) > 0 ) {
let dict = CFArrayGetValueAtIndex(attachmentsArray, 0)
let dictRef:CFDictionaryRef = unsafeBitCast(dict, CFDictionaryRef.self)
//get value
let value = CFDictionaryGetValue(dictRef, unsafeBitCast(kCMSampleAttachmentKey_NotSync, UnsafePointer<Void>.self))
if ( value != nil ){
print ("IFrame found...")
isIFrame = true
}
}
//2. define the start code
let nStartCodeLength:size_t = 4
let nStartCode:[UInt8] = [0x00, 0x00, 0x00, 0x01]
//3. write the SPS and PPS before I-frame
if ( isIFrame == true ){
let description:CMFormatDescriptionRef = CMSampleBufferGetFormatDescription(frame)!
//how many params
var numParams:size_t = 0
CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description, 0, nil, nil, &numParams, nil)
//write each param-set to elementary stream
print("Write param to elementaryStream ", numParams)
for i in 0..<numParams {
var parameterSetPointer:UnsafePointer<UInt8> = nil
var parameterSetLength:size_t = 0
CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description, i, ¶meterSetPointer, ¶meterSetLength, nil, nil)
elementaryStream.appendBytes(nStartCode, length: nStartCodeLength)
elementaryStream.appendBytes(parameterSetPointer, length: unsafeBitCast(parameterSetLength, Int.self))
}
}
//4. Get a pointer to the raw AVCC NAL unit data in the sample buffer
var blockBufferLength:size_t = 0
var bufferDataPointer: UnsafeMutablePointer<Int8> = nil
CMBlockBufferGetDataPointer(CMSampleBufferGetDataBuffer(frame)!, 0, nil, &blockBufferLength, &bufferDataPointer)
print ("Block length = ", blockBufferLength)
//5. Loop through all the NAL units in the block buffer
var bufferOffset:size_t = 0
let AVCCHeaderLength:Int = 4
while (bufferOffset < (blockBufferLength - AVCCHeaderLength) ) {
// Read the NAL unit length
var NALUnitLength:UInt32 = 0
memcpy(&NALUnitLength, bufferDataPointer + bufferOffset, AVCCHeaderLength)
//Big-Endian to Little-Endian
NALUnitLength = CFSwapInt32(NALUnitLength)
if ( NALUnitLength > 0 ){
print ( "NALUnitLen = ", NALUnitLength)
// Write start code to the elementary stream
elementaryStream.appendBytes(nStartCode, length: nStartCodeLength)
// Write the NAL unit without the AVCC length header to the elementary stream
elementaryStream.appendBytes(bufferDataPointer + bufferOffset + AVCCHeaderLength, length: Int(NALUnitLength))
// Move to the next NAL unit in the block buffer
bufferOffset += AVCCHeaderLength + size_t(NALUnitLength);
print("Moving to next NALU...")
}
}
print("Read completed...")
}
The SimpleControls example of the Red Bear Labs BLE Mini module (https://github.com/RedBearLab/iOS/tree/master/Examples/SimpleControls_OSX) enables to send analog readings (e.g. temperature sensor) from an Arduino to iOS / OSX with following Arduino code:
uint16_t value = analogRead(ANALOG_IN_PIN)
BLEMini_write(0x0B);
BLEMini_write(value >> 8);
BLEMini_write(value);
However, I tried to convert the raw analog readings (e.g. 162) into actual temperature reading (e.g. degree celsius / 27.15) and transmit the conversion to iOS / OSX, but on OSX I just read strange values (e.g. 13414). The Arduino code I used is following:
int reading = analogRead(ANALOG_IN_PIN);
float voltage = reading * 5.0;
float temp = (voltage - 0.5) * 100;
int tempINT = temp;
uint16_t value = tempINT;
BLEMini_write(0x0B);
BLEMini_write(value >> 8);
BLEMini_write(value);
The code-part of the OSX-app is following:
-(void) bleDidReceiveData:(unsigned char *)data length:(int)length
{
NSLog(#"Length: %d", length);
// parse data, all commands are in 3-byte
for (int i = 0; i < length; i+=3)
{
NSLog(#"0x%02X, 0x%02X, 0x%02X", data[i], data[i+1], data[i+2]);
if (data[i] == 0x0A) // Digital In data
{
if (data[i+1] == 0x01)
lblDigitalIn.stringValue = #"HIGH";
else
lblDigitalIn.stringValue = #"LOW";
}
else if (data[i] == 0x0B) // Analog In data
{
UInt16 Value;
Value = data[i+2] | data[i+1] << 8;
lblAnalogIn.stringValue = [NSString stringWithFormat:#"%d", Value];
}
}
}
It seems that the problem are "float" or converted "int" values and if someone could help me to solve this problem I would be really happy!
All characteristic data is just bytes. Once a characteristic's data has been read it is up to the central app to convert the data to an appropriate format (as described by the peripheral's manufacturer or some characteristics also contain a format descriptor which will describe out to format its data.)
I've got a ~1s mono .WAV on disk. I would like my OSX (and later iOS) app to read it into a float buffer.
What's the simplest way to achieve this?
Solution is to use ExtAudioFile()
I found it reading the most excellent Core-Audio bible
The libsndfile way :)
SF_INFO sfinfo;
SNDFILE *sf;
float *buf;
int err_code;
sfinfo.format = 0;
sf = sf_open("/meow.wav", SFM_READ, &sfinfo);
err_code = sf_error(sf);
if (err_code == SF_ERR_NO_ERROR) {
buf = malloc(sfinfo.frames * sfinfo.channels * sizeof(float));
sf_read(sf, buf, sfinfo.frames * sfinfo.channels);
printf("Done!\n");
} else {
printf("%s\n", sf_error_number(err_code));
}