iOS - Sending and receiving data simultaneously - ios

thanks for taking the time to read my question.
Basically I'm connecting to another device using MultipeerConnectivity and I want to stream data to the other device while receiving data.
My issue is that one of the devices ends up only sending data, and the other device ends up only receiving it.
Here is my sending method. This method is a delegate method from AVCaptureAudioDataOutputSampleBufferDelegate so it is automatically called.
func captureOutput(_ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, from connection: AVCaptureConnection!) {
//-------------------------------------------------------
// Writing to output stream
var blockBuffer: CMBlockBuffer?
var audioBufferList: AudioBufferList = AudioBufferList.init()
CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(sampleBuffer, nil, &audioBufferList, MemoryLayout<AudioBufferList>.size, nil, nil, kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment, &blockBuffer)
let buffers = UnsafeMutableAudioBufferListPointer(&audioBufferList)
for buffer in buffers {
let u8ptr = buffer.mData!.assumingMemoryBound(to: UInt8.self)
let output = outputStream!.write(u8ptr, maxLength: Int(buffer.mDataByteSize))
if (output == -1) {
let error = outputStream?.streamError
print("\(#file) > \(#function) > Error on outputStream: \(error!.localizedDescription)")
}
else {
print("\(#file) > \(#function) > Data sent")
}
}
}
And here is my receiving method (I do handle other stream events they are just irrelevant here). You can ignore all of the audio related stuff, it doesn't work right now, but it's not my main issue. This method is a delegate of OutputStream, so it is also automatically called when data is available from the stream.
func stream(_ aStream: Stream, handle eventCode: Stream.Event) {
case Stream.Event.hasBytesAvailable:
print("\(#file) > \(#function) > New data has arrived")
readFromStream()
}
func readFromStream() {
while (inputStream!.hasBytesAvailable) {
var buffer = [UInt8](repeating: 0, count: 4096)
let length = inputStream!.read(&buffer, maxLength: buffer.count)
if (length > 0) {
if (audioEngine!.isRunning) {
print("\(#file) > \(#function) > audioEngine is running")
audioEngine!.stop()
audioEngine!.reset()
}
else {
print("\(#file) > \(#function) > audioEngine is NOT running")
}
print("\(#file) > \(#function) > \(length) bytes read")
let audioBuffer = bytesToAudioBuffer(buffer)
let mainMixer = audioEngine!.mainMixerNode
audioEngine!.connect(audioPlayer!, to: mainMixer, format: audioBuffer.format)
audioPlayer!.scheduleBuffer(audioBuffer, completionHandler: nil)
do {
try audioEngine!.start()
}
catch let error as NSError {
print("\(#file) > \(#function) > error: \(error.localizedDescription)")
}
audioPlayer!.play()
}
}
}
From these two methods you can see I have print statements for when data comes in and when data is being sent out. On one of the devices I end up consistently reading data and never sending any, and on the other device I end up constantly sending data and never receiving any.
I imagine this is because I'm not using threads.. But I've never worked with threads on iOS. If this is the case, can anyone guide me on how to do so?
If you need any more information please let me know.

Eventually I did figure out a feasible way to use threads:
let streamReceiverQueue = DispatchQueue(label: "newQueue", qos: DispatchQoS.userInteractive)
streamReceiverQueue.sync {
// code here
}
streamReceiverQueue.async {
// code here
}

Related

CFStreamCreatePairWithSocketToHost - How to handle the network disconnect?

We have created socket for streaming using the API "CFStreamCreatePairWithSocketToHost".
func openStream() {
if let aUrl = url {
var readStream: Unmanaged<CFReadStream>?
var writeStream: Unmanaged<CFWriteStream>?
CFStreamCreatePairWithSocketToHost(
kCFAllocatorDefault,
aUrl as CFString,
port!,
&readStream,
&writeStream)
iStream = readStream!.takeRetainedValue()
oStream = writeStream!.takeRetainedValue()
if ssl == true {
let dict = [
kCFStreamSSLValidatesCertificateChain: kCFBooleanFalse, // allow self-signed certificate
kCFStreamSSLLevel: kCFStreamSSLValidatesCertificateChain // don't understand, why there isn't a constant for version 1.2
] as CFDictionary
let sslSetRead = CFReadStreamSetProperty(iStream, CFStreamPropertyKey(kCFStreamPropertySSLSettings), dict)
let sslSetWrite = CFWriteStreamSetProperty(oStream, CFStreamPropertyKey(kCFStreamPropertySSLSettings), dict)
if sslSetRead == false || sslSetWrite == false {
//print("SSL Configuration Failed")
}
}
commonStr = ""
open(iStream)
open(oStream)
print("Open Stream")
}
}
Used the below delegate for getting events.
func stream(_ aStream: Stream, handle eventCode: Stream.Event) {
// print("Stream : \(aStream.debugDescription) Event : \(eventCode.rawValue)")
switch eventCode {
case .endEncountered, .errorOccurred:
streamError()
break
case .hasBytesAvailable:
if let iStream = iStream {
if aStream == iStream {
//read data
let bufferSize = 1024
let buffer = UnsafeMutablePointer<UInt8>.allocate(capacity: bufferSize)
var len: Int
while iStream.hasBytesAvailable {
len = iStream.read(buffer, maxLength: bufferSize)
if len > 0 {
let output = Data.init(bytes: buffer, count: len)
bufferTokenize(output: output)
}else {
break
}
}
buffer.deallocate()
}
}
break
case .hasSpaceAvailable:
if aStream == oStream! {
streamReady()
}
break
case .openCompleted:
if aStream == oStream! {
if let del = delegate {
del.connectionCompleted()
}
}
break
default:
break
}
}
Other Supporting functions are
func streamError() {
stopStream()
NSObject.cancelPreviousPerformRequests(withTarget:self)
perform(#selector(callReconnect), with: nil, afterDelay: 2)
}
func open(_ stream: Stream?) {
if let aStream = stream {
aStream.delegate = self
aStream.schedule(in: RunLoop.main, forMode: .common)
aStream.open()
}
}
func stopStream() {
print("Stop Stream")
close(iStream)
close(oStream)
}
func close(_ stream: Stream?) {
if nil != stream {
stream?.close()
stream?.remove(from: RunLoop.main, forMode: .common)
}
}
func resetStream() {
stopStream()
openStream()
}
It is working fine for below steps
Create connection - Open stream
get "openCompleted" event - then Create request
get "hasSpaceAvailable" event - then Send Request
get "hasBytesAvailable" event - then Handle Data
Disconnect connection from server for testing purpose
get "endEncountered" event - then Close the connection and Recreate the connection from step 1
But the below scenario not working properly
Follow the steps 1 - 4 then
Disconnect Internet connection for testing purpose - the event "hasBytesAvailable" stopped
Again reconnect the Internet connection within 30 seconds - now we get "hasBytesAvailable" event
But the reconnect time extent to 5 minute - We don't have any event, and don't get "endEncountered" or "errorOccurred" also. Then how to move to next stop.
Please Help, Thanks
Regards,
Elamvazhuthi K

How to convert UnsafeMutablePointer<UInt8> to UIImage in Swift

I have been trying to send image data from android and receive at iOS using Socket,But the problem is that I am not able to show Image from data which is received from stream.
Here is the complete code:-
Setting Up the network communication
var readStream: Unmanaged<CFReadStream>?
var writeStream: Unmanaged<CFWriteStream>?
CFStreamCreatePairWithSocketToHost(kCFAllocatorDefault,
ip as CFString,
3003,
&readStream,
&writeStream)
inputStream = readStream!.takeRetainedValue()
outputStream = writeStream!.takeRetainedValue()
inputStream.delegate = self
// outputStream.delegate = self
inputStream.schedule(in: .current, forMode: RunLoop.Mode.common)
outputStream.schedule(in: .current, forMode: RunLoop.Mode.common)
inputStream.open()
outputStream.open()
Receive data through the delegates as shown below:-
extension ChatRoom: StreamDelegate {
func stream(_ aStream: Stream, handle eventCode: Stream.Event) {
switch eventCode {
case Stream.Event.hasBytesAvailable:
print("new message received")
readAvailableBytes(stream: aStream as! InputStream)
case Stream.Event.endEncountered:
print("end received")
stopStreamSession()
case Stream.Event.errorOccurred:
print("error occurred")
case Stream.Event.hasSpaceAvailable:
print("has space available")
case Stream.Event.openCompleted:
print("Stream opened")
default:
print("some other event...")
break
}
}
private func readAvailableBytes(stream: InputStream) {
let buffer = UnsafeMutablePointer<UInt8>.allocate(capacity: maxReadLength)
while stream.hasBytesAvailable {
let numberOfBytesRead = inputStream.read(buffer, maxLength: maxReadLength)
if numberOfBytesRead < 0 {
if let _ = stream.streamError {
break
}
}
NSLog("number of bytes read = %i", numberOfBytesRead)
if numberOfBytesRead > 0 {
dataOfImage.append(buffer, count: numberOfBytesRead)
}
}
}
}
I have been trying to show image from data received using UIImage(data:dataOfImage), But it is showing nil , when I inspect the dataOfImage it consists of all the data received from stream.
Can anyone suggest a way to get the image out of UnsafeMutablePointer<UInt8> Data.
Also if there is any better approach for data transfer through sockets, Please do Suggest..
Rather than allocate memory declare a buffer variable with the capacity of maxReadLength.
And stream(_:handle:) is called multiple times. You have to check the result of the read function
private var dataOfImage = Data()
private func readAvailableBytes(stream: InputStream) {
var buffer = [UInt8](repeating: 0, count: maxReadLength)
let bytesRead = inputStream.read(&buffer, maxLength: maxReadLength)
switch bytesRead {
case 0: // success, end of file
case -1: // error
default: // append data
dataOfImage.append(contentsOf: buffer)
}
NSLog("number of bytes read = %i", bytesRead)
}

Can the timestamp be null in a core audio render callback?

I have an iOS workout app that needs to stay running (and playing audio) while in the background. Sadly, the only way to accomplish this, that I know of, is to play an inaudible sound through Core Audio.
One user is experiencing a repeating crash that I can't reproduce in my simulators. (They have an iPhone 5, iOS 10.) The crash report in Xcode points to a line that I've never seen crash before. Can timestamp be null in the render callback?
private func render(ctx: UnsafeMutableRawPointer,
ioActionFlags: UnsafeMutablePointer<AudioUnitRenderActionFlags>,
timeStamp: UnsafePointer<AudioTimeStamp>,
bus: UInt32,
nFrames: UInt32,
ioData: UnsafeMutablePointer<AudioBufferList>?) -> OSStatus
{
guard let ioData = ioData else { return noErr }
let abl = UnsafeMutableAudioBufferListPointer(ioData)
guard let bufferData = abl[0].mData else { return noErr }
let impl = Unmanaged<MyClass>.fromOpaque(ctx).takeUnretainedValue()
let buf: UnsafeMutablePointer<Float32> = bufferData.assumingMemoryBound(to: Float32.self)
// **** Crash appears to pointing to this next line.
var t: Int = Int(timeStamp.pointee.mSampleTime) % impl.samples.count
for i in 0..<Int(nFrames) {
buf[i] = impl.samples[t]
t += 1
if t >= impl.samples.count { t = 0 }
}
...

Voice over bluetooth in iOS

I am doing research over four days, But I am not found any solution for calling over Bluetooth between two iOS devices within a distance.
I found that audio streaming is possible between two iOS devices using multipeer connectivity framework but this is not helpful for me. I want real time voice chat between two devices over Bluetooth.
Is there any CO-DAC for voice over Bluetooth?
My code is:
var engine = AVAudioEngine()
var file: AVAudioFile?
var player = AVAudioPlayerNode()
var input:AVAudioInputNode?
var mixer:AVAudioMixerNode?
override func viewDidLoad() {
super.viewDidLoad()
mixer = engine.mainMixerNode
input = engine.inputNode
engine.connect(input!, to: mixer!, format: input!.inputFormat(forBus: 0))
}
#IBAction func btnStremeDidClicked(_ sender: UIButton) {
mixer?.installTap(onBus: 0, bufferSize: 2048, format: mixer?.outputFormat(forBus: 0), block: { (buffer: AVAudioPCMBuffer, AVAudioTime) in
let byteWritten = self.audioBufferToData(audioBuffer: buffer).withUnsafeBytes {
self.appDelegate.mcManager.outputStream?.write($0, maxLength: self.audioBufferToData(audioBuffer: buffer).count)
}
print(byteWritten ?? 0)
print("Write")
})
do {
try engine.start()
}catch {
print(error.localizedDescription)
}
}
func audioBufferToData(audioBuffer: AVAudioPCMBuffer) -> Data {
let channelCount = 1
let bufferLength = (audioBuffer.frameCapacity * audioBuffer.format.streamDescription.pointee.mBytesPerFrame)
let channels = UnsafeBufferPointer(start: audioBuffer.floatChannelData, count: channelCount)
let data = Data(bytes: channels[0], count: Int(bufferLength))
return data
}
Thanks in Advance :)
Why is MultipeerConnectivity not helpful for you? It is a great way to stream audio over bluetooth or even wifi.
When you call this:
audioEngine.installTap(onBus: 0, bufferSize: 17640, format: localInputFormat) {
(buffer, when) -> Void in
You need to use the buffer, which has type AVAudioPCMBuffer. You then need to convert that to NSData and write to the outputStream that you would've opened with the peer:
data = someConverstionMethod(buffer)
_ = stream!.write(data.bytes.assumingMemoryBound(to: UInt8.self), maxLength: data.length)
Then on the other device you need to read from the stream and convert from NSData back to an AVAudioPCMBuffer, and then you can use an AVAudioPlayer to playback the buffer.
I have done this before with a very minimal delay.

Decrypt Media Files in chunks and play via AVPlayer

I have a mp4 video file which i am encrypting to save and decrypting to play via AVPlayer. Using CRYPTOSWIFT Library for encrypting/decrypting
Its working fine when i am decrypting whole file at once but my file is quite big and taking 100% CPU usage and lot of memory. So, I need to decrypt encrypted file in chunks.
I tried to decrypt file in chunks but its not playing video as AVPlayer is not recognizing decrypted chunk data maybe data is not stored sequentially while encrypting file. I have tried chacha20, AES, AES.CTR & AES.CBC protocols to encrypt and decrypt files but to no avail.
extension PlayerController: AVAssetResourceLoaderDelegate {
func resourceLoader(resourceLoader: AVAssetResourceLoader, shouldWaitForLoadingOfRequestedResource loadingRequest: AVAssetResourceLoadingRequest) -> Bool {
let request = loadingRequest.request
guard let path = request.URL?.path where request.URL?.scheme == Constants.customVideoScheme else { return true }
if let contentRequest = loadingRequest.contentInformationRequest {
do {
let fileAttributes = try NSFileManager.defaultManager().attributesOfItemAtPath(path)
if let fileSizeNumber = fileAttributes[NSFileSize] {
contentRequest.contentLength = fileSizeNumber.longLongValue
}
} catch { }
if fileHandle == nil {
fileHandle = NSFileHandle(forReadingAtPath: (request.URL?.path)!)!
}
contentRequest.contentType = "video/mp4"
contentRequest.byteRangeAccessSupported = true
}
if let data = decryptData(loadingRequest, path: path), dataRequest = loadingRequest.dataRequest {
dataRequest.respondWithData(data)
loadingRequest.finishLoading()
return true
}
return true
}
func decryptData(loadingRequest: AVAssetResourceLoadingRequest, path: String) -> NSData? {
print("Current OFFSET: \(loadingRequest.dataRequest?.currentOffset)")
print("requested OFFSET: \(loadingRequest.dataRequest?.requestedOffset)")
print("Current Length: \(loadingRequest.dataRequest?.requestedLength)")
if loadingRequest.contentInformationRequest != nil {
var data = fileHandle!.readDataOfLength((loadingRequest.dataRequest?.requestedLength)!)
fileHandle!.seekToFileOffset(0)
data = decodeVideoData(data)!
return data
} else {
fileHandle?.seekToFileOffset(UInt64((loadingRequest.dataRequest?.currentOffset)!))
let data = fileHandle!.readDataOfLength((loadingRequest.dataRequest?.requestedLength)!)
// let data = fileHandle!.readDataOfLength(length!) ** When I use this its not playing video but play fine when try with requestedLength **
return decodeVideoData(data)
}
}
}
Decode code to decode nsdata :
func decodeVideoData(data: NSData) -> NSData? {
if let cha = ChaCha20(key: Constants.Encryption.SecretKey, iv: Constants.Encryption.IvKey) {
let decrypted: NSData = try! data.decrypt(cha)
return decrypted
}
return nil
}
I need help regarding this issue, Kindly guide me to the right way to achieve this.
For in depth and a more complete CommonCrypto wrapper, check out my CommonCrypto wrapper. I've extracted bits and pieces for this answer.
First of all, we need to define some functions that will do the encryption/decryption. I'm assuming, for now, you use AES(256) CBC with PKCS#7 padding. Summarising the snippet below: we have an update function, that can be called repeatedly to consume the chunks. There's also a final function that will wrap up any left overs (usually deals with padding).
import CommonCrypto
import Foundation
enum CryptoError: Error {
case generic(CCCryptorStatus)
}
func getOutputLength(_ reference: CCCryptorRef?, inputLength: Int, final: Bool) -> Int {
CCCryptorGetOutputLength(reference, inputLength, final)
}
func update(_ reference: CCCryptorRef?, data: Data) throws -> Data {
var output = [UInt8](repeating: 0, count: getOutputLength(reference, inputLength: data.count, final: false))
let status = data.withUnsafeBytes { dataPointer -> CCCryptorStatus in
CCCryptorUpdate(reference, dataPointer.baseAddress, data.count, &output, output.count, nil)
}
guard status == kCCSuccess else {
throw CryptoError.generic(status)
}
return Data(output)
}
func final(_ reference: CCCryptorRef?) throws -> Data {
var output = [UInt8](repeating: 0, count: getOutputLength(reference, inputLength: 0, final: true))
var moved = 0
let status = CCCryptorFinal(reference, &output, output.count, &moved)
guard status == kCCSuccess else {
throw CryptoError.generic(status)
}
output.removeSubrange(moved...)
return Data(output)
}
Next up, for the purpose of demonstration, the encryption.
let key = Data(repeating: 0x0a, count: kCCKeySizeAES256)
let iv = Data(repeating: 0, count: kCCBlockSizeAES128)
let bigFile = (0 ..< 0xffff).map { _ in
return Data(repeating: UInt8.random(in: 0 ... UInt8.max), count: kCCBlockSizeAES128)
}.reduce(Data(), +)
var encryptor: CCCryptorRef?
CCCryptorCreate(CCOperation(kCCEncrypt), CCAlgorithm(kCCAlgorithmAES), CCOptions(kCCOptionPKCS7Padding), Array(key), key.count, Array(iv), &encryptor)
do {
let ciphertext = try update(encryptor, data: bigFile) + final(encryptor)
print(ciphertext) // 1048576 bytes
} catch {
print(error)
}
That appears to me as quite a large file. Now decrypting, would be done in a similar fashion.
var decryptor: CCCryptorRef?
CCCryptorCreate(CCOperation(kCCDecrypt), CCAlgorithm(kCCAlgorithmAES), CCOptions(kCCOptionPKCS7Padding), Array(key), key.count, Array(iv), &decryptor)
do {
var plaintext = Data()
for i in 0 ..< 0xffff {
plaintext += try update(decryptor, data: ciphertext[i * kCCBlockSizeAES128 ..< i * kCCBlockSizeAES128 + kCCBlockSizeAES128])
}
plaintext += try final(decryptor)
print(plaintext == bigFile, plaintext) // true 1048560 bytes
} catch {
print(error)
}
The encryptor can be altered for different modes and should also be released once it's done, and I'm not too sure how arbitrary output on the update function will behave, but this should be enough to give you an idea of how it can be done using CommonCrypto.

Resources