In parsing a file, I need to locate a certain valued byte. In this case, I am searching for the value 13. I know that this value will exist at the start of a 32-byte block. With all of that information, I have come up with a few solutions similar to the following:
let mut file = File::open("file_to_open").unwrap();
let mut read_array = [0u8;32];
while read_array[0] != 13 {
file.read_exact(&mut read_array).unwrap();
}
[...]
The problem with this is it takes a very long time. Running this code 100,000 times, for instance, takes around 9 seconds. On the other hand, if I load a large amount of data into a Vec and then search through memory, I can read it much faster. But, the file is large and if I load too big of a chunk at a time, it will take longer than reading byte-by-byte in cases where the data would have been found early in the file. I tried using SeekFrom::Current, but seeking forward 32 bytes was slower than just reading all 32 bytes at once.
Is my only option to come up with an acceptable chunk size and iterate those chunks?
Edit:
The following is currently my quickest implementation:
fn get_raw(file: &mut File) -> usize {
let start_bit= 32;
let mut iter = 0;
file.seek(SeekFrom::Start(start_bit)).unwrap();
let read_size = 32;
let block_size = read_size * 100;
let mut my_vec = vec![0u8;block_size];
while let Ok(n) = file.read(&mut my_vec) {
if n != block_size {
break;
}
for i in (0..block_size).step_by(read_size) {
if my_vec[i] == 13 {
return iter;
}
iter += 1;
}
}
panic!("The database is corrupt; this process cannot continue.");
}
Related
I've got an array of audio files that I want to normalize so they all have similar perceived loudness. For testing purposes, I decided to adapt the AVAudioPCMBuffer.normalize method from AudioKit to suit my purposes. See here for implementation: https://github.com/AudioKit/AudioKit/blob/main/Sources/AudioKit/Audio%20Files/AVAudioPCMBuffer%2BProcessing.swift
I am converting each file into an AVAudioPCMBuffer, and then performing a reduce on that array of buffers to get the highest peak across all of the buffers. Then I created a new version of normalize called normalize(with peakAmplitude: Float) -> AVAudioPCMBuffer takes that peak amplitude, calculates a gainFactor and then iterates through the floatData for each channel and multiplies the floatData by the gainFactor. I then call my new flavor of normalize with the peak.amplitude that I get from the reduce operation on all the audio buffers.
This produces useful results, sometimes.
Here's the actual code in question:
extension AVAudioPCMBuffer {
public func normalize(with peakAmplitude: Float) -> AVAudioPCMBuffer {
guard let floatData = floatChannelData else { return self }
let gainFactor: Float = 1 / peakAmplitude
let length: AVAudioFrameCount = frameLength
let channelCount = Int(format.channelCount)
// i is the index in the buffer
for i in 0 ..< Int(length) {
// n is the channel
for n in 0 ..< channelCount {
let sample = floatData[n][i] * gainFactor
self.floatChannelData?[n][i] = sample
}
}
self.frameLength = length
return self
}
}
extension Array where Element == AVAudioPCMBuffer {
public func normalized() -> [AVAudioPCMBuffer] {
var minPeak = AVAudioPCMBuffer.Peak()
minPeak.amplitude = AVAudioPCMBuffer.Peak.min
let maxPeakForAllBuffers: AVAudioPCMBuffer.Peak = reduce(minPeak) { result, buffer in
guard
let currentBufferPeak = buffer.peak(),
currentBufferPeak.amplitude > result.amplitude
else {
return result
}
return currentBufferPeak
}
return map { $0.normalize(with: maxPeakForAllBuffers.amplitude) }
}
}
Three questions:
Is my approach reasonable for multiple files?
This appears to be using "peak normalization" vs RMS or EBU R128 normalization. Is that why when I give it a batch of 3 audio files and 2 of them are correctly made louder that 1 of them is made louder even though ffmpeg-normalize on the same batch of files makes that 1 file significantly quieter?
Any other suggestions on ways to alter the floatData across multiple AVAudioAudioPCMBuffers in order to make them have similar perceived loudness?
I have this function:
use std::io;
pub fn recv(mut buf: &mut [u8]) -> io::Result<usize> {
let size_to_extend = 50; // I want to increase the size of "buf" by 50
unsafe {
/* ??? */
}
}
How can I adjust the size of the array buf, even though it's a parameter? It is necessary that this method needs to happen.
What you are attempting to do is all-but-guaranteed to cause undefined behavior. Find a better API.
If you want the changes to be reflected outside of the function, you are out of luck. There's no way that this function signature will allow for that to happen, for the same reason that fn foo(x: i32) won't allow you to change the value passed in as observed by the caller.
If you just need this inside of the function, use slice::from_raw_parts. I've marked the entire function as unsafe because certain inputs will cause undefined behavior and this code cannot possibly guard against it:
use std::slice;
pub unsafe fn recv(buf: &mut [u8]) {
let size_to_extend = 50;
let ptr = buf.as_mut_ptr();
let len = buf.len();
let bad_idea = slice::from_raw_parts_mut(ptr, len + size_to_extend);
for b in bad_idea.iter_mut() {
*b = 10;
}
}
If you can change the API, something like this works to expose the change outside the function:
pub unsafe fn recv(buf: &mut &mut [u8]) {
let size_to_extend = 50;
let ptr = buf.as_mut_ptr();
let len = buf.len();
let bad_idea = slice::from_raw_parts_mut(ptr, len + size_to_extend);
for b in bad_idea.iter_mut() {
*b = 10;
}
*buf = bad_idea;
}
See also:
How can I get an array or a slice from a raw pointer?
So I have some code I've been using to upload files in my app, along the lines of this:
var mutableURLRequest = URLRequest(url: url)
var uploadData = try! Data(contentsOf: dataUrl)
session.uploadTask(with: mutableURLRequest, from: uploadData).resume()
There's a little more to it than that, but those are the relevant parts. However I've noticed for some large video files Data(contentsOf: dataUrl) fails since the file is to big to load into memory. I want to restructure this so that I'm able to stream piece by piece to the server without ever having to load the whole file into memory.
I already have this figured out from my server, the only piece I haven't figured out is how to get a chunkSize piece from the data in a URL, without putting it into a data object. I essentially want this construct:
let chunkSize = 1024 * 1024
let offset = 0
let chunk = //get data from dataUrl of size chunkSize offset by offset
//Upload each chunk and increment offset
NSInputStream seemed promising in being able to do this, but I wasn't able to figure out the minimum set up in order to pull bytes from a file on disk in this fashion. What code can I use above to fill in the let chunk = line to do such a task?
I have a working solution, might need a little tweaking, but seems to work for big files I've tried:
public func getNextChunk() -> Data?{
if _inputStream == nil {
_inputStream = InputStream(url: _url)
_inputStream?.open()
}
var buffer = [UInt8](repeating: 0, count: CHUNK_SIZE)
var len = _inputStream?.read(&buffer, maxLength: CHUNK_SIZE)
if len == 0 {
return nil
}
return Data(buffer)
}
I also call _inputStream?.close() on deinit of my class that manages the chunking of a file on disk.
I'm not sure if there is an issue or not, so i'm just gonna write it down.
I'm developing using swift, xcode 7.2 , on iphone 5s.
And calculating execution time using
NSDate.timeIntervalSinceReferenceDate()
I created 2 arrays, one with 200,000 elements and one with 20.
and try to have random access to their elements. accessing elements on big one is almost 55 times slower! i know its bigger but isn't this O(1) ?
I also tried the same on java and the accessing speed is the same for big and small array.
From CFArrayheader in apple documentation, i found this:
Accessing any value at a particular index in an array is at worst O(log n), but should usually be O(1).
but it think this cant be true based on the numbers i've tested.
I know i didn't make a big test or anything special, but the fact that its not working is really messing with my head!
i kinda need this for what i'm working on. and the algorithm is not working on swift and iOS and its working on java and android.
let bigSize:Int = 200000
var bigArray = [Int](count:bigSize,repeatedValue:0)
let smallSize:Int = 20
var smallArray = [Int](count:smallSize,repeatedValue:0)
for i in 0..<bigSize
{
bigArray[i] = i + 8 * i
}
for i in 0..<smallSize
{
smallArray[i] = i + 9 * i
}
let indexBig = Int(arc4random_uniform(UInt32(bigSize)) % UInt32(bigSize))
let indexSmall = Int(arc4random_uniform(UInt32(smallSize)) % UInt32(smallSize))
var a = NSDate.timeIntervalSinceReferenceDate()
print(bigArray[indexBig])
var b = NSDate.timeIntervalSinceReferenceDate()
print(b-a) \\prints 0.000888049602508545
a = NSDate.timeIntervalSinceReferenceDate()
print(smallArray[indexSmall])
b = NSDate.timeIntervalSinceReferenceDate()
print(b-a) \\prints 6.90221786499023e-05
java :
(accessing one element is so fast on java and its on pc, so i access more elements, but same number on both arrays)
int bigSize = 200000;
int[] bigArray = new int[bigSize];
Random rand = new Random();
int smallSize = 20;
int[] smallArray = new int[smallSize];
for(int i = 0;i < bigSize;i++)
bigArray[i] = i + i * 8;
for(int i = 0;i < smallSize;i++)
smallArray[i] = i + i * 8;
int smallIndex = rand.nextInt(smallSize);
int bigIndex = rand.nextInt(bigSize);
int sum = 0;
long a = System.currentTimeMillis();
for(int i = 0;i < 10000;i++)
{
sum += bigArray[rand.nextInt(bigSize)];
}
System.out.println(sum);
long b = System.currentTimeMillis();
System.out.println(b-a); //prints 2
a = System.currentTimeMillis();
sum = 0;
for(int i = 0; i < 10000;i++)
{
sum += smallArray[rand.nextInt(smallSize)];
}
System.out.println(sum);
b = System.currentTimeMillis();
System.out.println(b - a); //prints 1
If you change the order of your two tests, you'll find that the performance is flipped. In short, the first test runs more slowly than the second one, regardless of whether it's the small array or the big one. This is a result of some dynamics of print. If you do a print before you perform the tests, the delay resulting from the first print is eliminated.
A better way to test this would be to create a unit test, which (a) repeats the subscript operator many times; and (b) uses measureBlock to repeat the test a few times to check for standard deviation and the like.
When I do that, I find the access time is indistinguishable, consistent with O(1). This were my unit tests:
let bigSize: Int = 200_000
let smallSize: Int = 20
func testBigArrayPerformance() {
let size = bigSize
let array = Array(0 ..< size).map { $0 + 8 * $0 }
var value = 0
measureBlock {
let baseIndex = Int(arc4random_uniform(UInt32(size)))
for index in 0 ..< 1_000_000 {
value += array[(baseIndex + index) % size]
}
}
print(value)
print(array.count)
}
func testSmallArrayPerformance() {
let size = smallSize
let array = Array(0 ..< size).map { $0 + 8 * $0 }
var value = 0
measureBlock {
let baseIndex = Int(arc4random_uniform(UInt32(size)))
for index in 0 ..< 1_000_000 {
value += array[(baseIndex + index) % size]
}
}
print(value)
print(array.count)
}
Admittedly, I've added some mathematical operations that change the index (my intent was to make sure the compiler didn't do some radical optimization that removed my attempt to repeat the subscript operation), and the overhead of that mathematical operation will dilute the subscript operator performance difference. But, even when I simplified the index operator, the performance between the two renditions was indistinguishable.
I have a long string (sometimes over 1000 characters) that I want to convert to an array of boolean values. And it needs to do this many times, very quickly.
let input: String = "001"
let output: [Bool] = [false, false, true]
My naive attempt was this:
input.characters.map { $0 == "1" }
But this is a lot slower than I'd like. My profiling has shown me that the map is where the slowdown is, but I'm not sure how much simpler I can make that.
I feel like this would be wicked fast without Swift's/ObjC's overhead. In C, I think this is a simple for loop where a byte of memory is compared to a constant, but I'm not sure what the functions or syntax is that I should be looking at.
Is there a way to do this much faster?
UPDATE:
I also tried a
output = []
for char in input.characters {
output.append(char == "1")
}
And it's about 15% faster. I'm hoping for a lot more than that.
This is faster:
// Algorithm 'A'
let input = "0101010110010101010"
var output = Array<Bool>(count: input.characters.count, repeatedValue: false)
for (index, char) in input.characters.enumerate() where char == "1" {
output[index] = true
}
Update: under input = "010101011010101001000100000011010101010101010101"
0.0741 / 0.0087, where this approach is faster that author's in 8.46 times. With bigger data correlation more positive.
Also, with using nulTerminatedUTF8 speed a little increased, but not always speed higher than algorithm A:
// Algorithm 'B'
let input = "10101010101011111110101000010100101001010101"
var output = Array<Bool>(count: input.nulTerminatedUTF8.count, repeatedValue: false)
for (index, code) in input.nulTerminatedUTF8.enumerate() where code == 49 {
output[index] = true
}
In result graph appears, with input length 2196, where first and last 0..1, A – second, B – third point.
A: 0.311sec, B: 0.304sec
import Foundation
let input:String = "010101011001010101001010101100101010100101010110010101010101011001010101001010101100101010100101010101011001010101001010101100101010100101010"
var start = clock()
var output = Array<Bool>(count: input.nulTerminatedUTF8.count, repeatedValue: false)
var index = 0
for val in input.nulTerminatedUTF8 {
if val != 49 {
output[index] = true
}
index+=1
}
var diff = clock() - start;
var msec = diff * 1000 / UInt(CLOCKS_PER_SEC);
print("Time taken \(Double(msec)/1000.0) seconds \(msec%1000) milliseconds");
This should be really fast. Try it out. For 010101011010101001000100000011010101010101010101 it takes 0.039 secs.
I would guess that this is as fast as possible:
let targ = Character("1")
let input: String = "001" // your real string goes here
let inputchars = Array(input.characters)
var output:[Bool] = Array.init(count: inputchars.count, repeatedValue: false)
inputchars.withUnsafeBufferPointer {
inputbuf in
output.withUnsafeMutableBufferPointer {
outputbuf in
var ptr1 = inputbuf.baseAddress
var ptr2 = outputbuf.baseAddress
for _ in 0..<inputbuf.count {
ptr2.memory = ptr1.memory == targ
ptr1 = ptr1.successor()
ptr2 = ptr2.successor()
}
}
}
// output now contains the result
The reason is that, thanks to the use of buffer pointers, we are simply cycling through contiguous memory, just like the way you cycle through a C array by incrementing its pointer. Thus, once we get past the initial setup, this should be as fast as it would be in C.
EDIT In an actual test, the time difference between the OP's original method and this one is the difference between
13.3660290241241
and
0.219357967376709
which is a pretty dramatic speed-up. I hasten to add, however, that I have excluded the initial set-up from the timing test. This line:
let inputchars = Array(input.characters)
...is particularly expensive.
This should be a little faster than the enumerate() where char == "1" version (0.557s for 500_000 alternating ones and zeros vs. 1.159s algorithm 'A' from diampiax)
let input = inputStr.utf8
let n = input.count
var output = [Bool](count: n, repeatedValue: false)
let one = UInt8(49) // 1
for (idx, char) in input.enumerate() {
if char == one { output[idx] = true }
}
but it's also a lot less readable ;-p
edit: both versions are slower than the map variant, maybe you forgot to compile with optimizations?
One more step should speed that up even more. Using reserveCapacity will resize the array once before the loops starts instead of trying to do it as the loop runs.
var output = [Bool]()
output.reserveCapacity(input.characters.count)
for char in input.characters {
output.append(char == "1")
}
Use withCString(_:) to retrieve a raw UnsafePointer<Int8>. Iterate over that and compare to 49 (ascii value of "1").
What about a more functional style? It's not fastest (47 ms), today, for sure...
import Cocoa
let start = clock()
let bools = [Bool](([Character] ("010101011001010101001010101100101010100101010110010101010101011001010101001010101100101010100101010101011001010101001010101100101010100101010".characters)).map({$0 == "1"}))
let msec = (clock() - start) * 1000 / UInt(CLOCKS_PER_SEC);
print("Time taken \(Double(msec)/1000.0) seconds \(msec%1000) milliseconds");
I need to some testing to be sure but I think one issue with many approaches given including the original map is that they need to iterate over the string to count the characters and then a second time to actually process the characters.
Have you tried:
let output = [Bool](input.characters.lazy.map { $0 == "1" })
This might only do a single iteration.
The other thing that could speed things up is if you can avoid using strings but instead use arrays of characters of an appropriate encoding (particularly if is more fixed size units (e.g. UTF16 or ASCII). Then then length lookup will be O(1) rather than O(n) and the iteration may be faster too
BTW always test performance with the optimiser enabled and never in the Playground because the performance characteristics are completely different, sometimes by a factor of 100.