So I have some code I've been using to upload files in my app, along the lines of this:
var mutableURLRequest = URLRequest(url: url)
var uploadData = try! Data(contentsOf: dataUrl)
session.uploadTask(with: mutableURLRequest, from: uploadData).resume()
There's a little more to it than that, but those are the relevant parts. However I've noticed for some large video files Data(contentsOf: dataUrl) fails since the file is to big to load into memory. I want to restructure this so that I'm able to stream piece by piece to the server without ever having to load the whole file into memory.
I already have this figured out from my server, the only piece I haven't figured out is how to get a chunkSize piece from the data in a URL, without putting it into a data object. I essentially want this construct:
let chunkSize = 1024 * 1024
let offset = 0
let chunk = //get data from dataUrl of size chunkSize offset by offset
//Upload each chunk and increment offset
NSInputStream seemed promising in being able to do this, but I wasn't able to figure out the minimum set up in order to pull bytes from a file on disk in this fashion. What code can I use above to fill in the let chunk = line to do such a task?
I have a working solution, might need a little tweaking, but seems to work for big files I've tried:
public func getNextChunk() -> Data?{
if _inputStream == nil {
_inputStream = InputStream(url: _url)
_inputStream?.open()
}
var buffer = [UInt8](repeating: 0, count: CHUNK_SIZE)
var len = _inputStream?.read(&buffer, maxLength: CHUNK_SIZE)
if len == 0 {
return nil
}
return Data(buffer)
}
I also call _inputStream?.close() on deinit of my class that manages the chunking of a file on disk.
Related
I am migrating my AudioKit-Code to AudioKit 5.2 and I am having problems with the AK5 Sampler() which I can not solve by myself. In AK previously it was AKSampler() and it had a function called:
loadAKAudioFile(from: AKSampleDescriptor, file: AKAudioFile)
Since Sampler() does not have that function, I took a piece of code from the source Sampler.swift which does the work, but it is extremely slow, it takes ages to load and map the 12 different small samples that I map on the note numbers from 60 to 71:
internal func loadAudioFile(from sampleDescriptor: SampleDescriptor, file: AVAudioFile) {
guard let floatChannelData = file.toFloatChannelData() else { return }
let sampleRate = Float(file.fileFormat.sampleRate)
let sampleCount = Int32(file.length)
let channelCount = Int32(file.fileFormat.channelCount)
var flattened = Array(floatChannelData.joined())
flattened.withUnsafeMutableBufferPointer { data in
var descriptor = SampleDataDescriptor(sampleDescriptor: sampleDescriptor,
sampleRate: sampleRate,
isInterleaved: false,
channelCount: channelCount,
sampleCount: sampleCount,
data: data.baseAddress)
akSamplerLoadData(au.dsp, &descriptor)
}
}
The AKSampler() did not have any noticable latency when doing the work but with Sampler() it takes more than a second to load a sample. Obviously AKSampler() worked asynchronously. I am new to swift and audio so I have no idea how to make Sampler() work asychronously.
Would be great to get some bits of code that could help it, thank's
In parsing a file, I need to locate a certain valued byte. In this case, I am searching for the value 13. I know that this value will exist at the start of a 32-byte block. With all of that information, I have come up with a few solutions similar to the following:
let mut file = File::open("file_to_open").unwrap();
let mut read_array = [0u8;32];
while read_array[0] != 13 {
file.read_exact(&mut read_array).unwrap();
}
[...]
The problem with this is it takes a very long time. Running this code 100,000 times, for instance, takes around 9 seconds. On the other hand, if I load a large amount of data into a Vec and then search through memory, I can read it much faster. But, the file is large and if I load too big of a chunk at a time, it will take longer than reading byte-by-byte in cases where the data would have been found early in the file. I tried using SeekFrom::Current, but seeking forward 32 bytes was slower than just reading all 32 bytes at once.
Is my only option to come up with an acceptable chunk size and iterate those chunks?
Edit:
The following is currently my quickest implementation:
fn get_raw(file: &mut File) -> usize {
let start_bit= 32;
let mut iter = 0;
file.seek(SeekFrom::Start(start_bit)).unwrap();
let read_size = 32;
let block_size = read_size * 100;
let mut my_vec = vec![0u8;block_size];
while let Ok(n) = file.read(&mut my_vec) {
if n != block_size {
break;
}
for i in (0..block_size).step_by(read_size) {
if my_vec[i] == 13 {
return iter;
}
iter += 1;
}
}
panic!("The database is corrupt; this process cannot continue.");
}
I am reading data from an MFi external device into a buffer using a 3rd party SDK "sessionController". See below:
let handle: UInt64 = self.sessionController.openFile(file.path, mode: openMode)
if handle == 0 {
//Error
return
}
let c: UInt64 = file.size
var bytesArray: [UInt8] = [UInt8](fileData)
let bufferPointer: UnsafeMutablePointer<UInt8> = UnsafeMutablePointer<UInt8>.allocate(capacity: Int(c))
bufferPointer.initialize(repeating: 0, count: Int(c))
defer {
bufferPointer.deinitialize(count: Int(c))
bufferPointer.deallocate()
}
var sum: UInt32 = 0
let singleSize: UInt32 = 8 << 20
while sum < c {
let read = self.sessionController.readFile(handle, data: bufferPointer, len: singleSize)
if read == 0 {
//There was an error
return
}
sum += read
}
let newPointer : UnsafeRawPointer = UnsafeRawPointer(bufferPointer)
fileURL = try! FileManager.default.url(for: .documentDirectory, in: .userDomainMask, appropriateFor: nil, create: false).appendingPathComponent("test.MOV")
fileData = Data(bytes: newPointer, count: Int(c))
try! fileData.write(to: fileURL)
//Now use this fileURL to watch video in an AVPlayer...
//AVPlayer(init: fileURL)
For some reason the data stored at the fileURL becomes corrupted (I think) and I am unable to play the video file. I think I am not doing something correctly with Unsafe Swift but I am not sure what. How can I make sure that I have properly read the data from the device into memory, and then taken that data from memory and stored it on the hard drive at the fileURL? What am I doing wrong here? The video will not play in AVPlayer given the fileURL.
The main error is here:
let read = self.sessionController.readFile(handle, data: bufferPointer, len: singleSize)
If you read in multiple chunks then the second and all subsequent reads will overwrite the data read previously. So that should probably be
let read = self.sessionController.readFile(handle, data: bufferPointer + sum, len: singleSize)
Note also that the file size is defined as UInt64, but the variable sum (which holds the total number of bytes read so far) is an UInt32. This will lead to problems if there is more than 4GB data.
But generally I would avoid to read the complete data into a memory buffer. You already read in chunks, so you can write the data immediately to the destination file. Here is how that could look like:
// Output file:
let fileURL = ...
let fileHandle = try FileHandle(forWritingTo: fileURL)
defer { fileHandle.closeFile() }
// Buffer:
let bufferSize = 1024 * 1024 // Choose some buffer size
var buffer = Data(count: bufferSize)
// Read/write loop:
let fileSize: UInt64 = file.size
var remainingToRead = fileSize
while remainingToRead > 0 {
let read = buffer.withUnsafeMutableBytes { bufferPointer in
self.sessionController.readFile(handle, data: bufferPointer, len: UInt32(min(remainingToRead, UInt64(bufferSize))))
}
if read == 0 {
return // Read error
}
remainingToRead -= UInt64(read)
fileHandle.write(buffer)
}
Note also that the data is read directly into a Data value, instead of reading it into allocated memory and then copying it to another Data.
I try to add bytes with bytes in order to get space that will be occupied by photos which i retrieve from internet.
I have following code, it gets sizes in bytes for each id in array of id
var diskSpace:Int64 = 0
for var i = 0; i < array.count; i++ {
let id = array[i]
let urlString = "urlToFetchData"
if let url = NSURL(string: urlString) {
if let data = try? NSData(contentsOfURL: url, options: []) {
let json = JSON(data: data)
let size = Int64(json["size"].stringValue)
diskSpace = diskSpace + size!
}
}
}
var diskSpaceInMb = diskSpace / 1024 / 1024
print("diskSpaceInMb is \(diskSpaceInMb)")
for example, I try to get size of three elements, which have following size in bytes (these sizes in bytes I receive in json)
3223653
5855382
8948976
when the code above is executed i receive result of
diskSpaceInMb is 8
which is obviously not try
How to convert bytes to megabytes correctly ?
let fileSizeWithUnit = ByteCountFormatter.string(fromByteCount: diskSpace, countStyle: .file)
print("File Size: \(fileSizeWithUnit)")
The problem is obviously in the for loop. Maybe the JSON is not as you expect.
Another reason why this might fail is the you use the try? keyword, which in this context it means that it gives you a value if it succeeds, but otherwise it returns nil. In your case, it may silently fail. If you want to check if it fails, you could add an else branch.
I am starting a project to create an iOS app to communicate with a device over BLE. Being a new effort, I am trying to do this is Swift if possible. The interface uses GATT and an existing set of custom message structures. I get to a point where I have the data from BLE in an NSData object. I'd like to cast it or directly convert it to my message structure in a fairly generic way.
I know that I can extract the data by hand either directly from the byte array from the NSData object or using an NSInputStream. While that works, it could be a maintenance issue and the interface has a number of different messages in it.
Is there an easier ways to do this?
I'd be willing to create the message structures in Objective-C and do the casting there, but my knowledge of Objective-C is not much better than my knowledge of Swift.
Some sample code of what I've been playing in my playground is shown below. It all works as expected.
func getBytesFromNSData(data: NSData, start: Int) -> [UInt8] {
let count = data.length / sizeof(UInt8)
let remaining = count - start
let range = NSMakeRange(start, remaining )
var dataArray = [UInt8](count: remaining, repeatedValue: 0)
data.getBytes(&dataArray, range: range)
return dataArray
}
class TestObject {
var a: Byte
var b: Byte
init() {
a = 0x01
b = 0x02
}
init(data: NSData) {
let dataBytes = getBytesFromNSData(data, 0)
a = Byte(dataBytes[0])
b = Byte(dataBytes[1])
}
func populateFromStream(data: NSData) {
var stream = NSInputStream(data: data)
stream.open()
var bytesRead = stream.read(&a, maxLength: 1)
println("\(bytesRead)")
bytesRead = stream.read(&b, maxLength: 1)
println("\(bytesRead)")
}
func toArray() -> [Byte] {
var result = [Byte](count: 2, repeatedValue: 0)
result[0] = a
result[1] = b
return result
}
}
let test = TestObject()
let testArray = test.toArray()
let length = testArray.count
let testData = NSData(bytes: testArray, length: length)
println("\(testData)")
let testIn = [ Byte(0x0d), Byte(0x0e) ]
let testDataIn = NSData(bytes: testIn, length: testIn.count)
println("\(testDataIn)")
let testConstructor = TestObject(data: testDataIn)
var testObject = TestObject()
testObject.populateFromStream(testDataIn)
I found a method that is fairly generic that may work is some cases.
Create an Objective-C riding header
Create the data structure as an Objective-C structure
Import the header with the data structure into the bridging header
Assuming that you have a struct called Foo and an NSData object called rawData:
Use the following code to get an cast a pointer.
let structureSize = sizeof(Foo)
var myObject = UnsafeMutablePointer<Foo>.alloc(1).memory
rawData.getbytes(&myObject, length: structureSize)
This will not work in all instances and unfortunately does work in my particular case. The specific problems I have found are:
The Objective-C structure is word aligned. If your structure is not properly aligned to work boundaries, you may have a size that is incorrect. (something I ran into in my particular interface)
If you and dealing with a system that doesn't send the data in the same order you are expecting, this will not handle any byte order conversion, that would still need to be done and the structure would possibly need to be reordered to compensate. That work might negate any saving from this method.
This is the most concise method I have found if it happens to work with your particular message formats.