Now facing some challenges using CoreBlueTooth L2CAP channel. In order to better understand how things work. I have taken the L2CapDemo (master) (https://github.com/paulw11/L2CapDemo) from GitHub and tried to experiment with it. Here is what I have done, along with one question.
In have replaced the sendTextTapped function, with this one :
#IBAction func sendTextTapped(_ sender: UIButton) {
guard let ostream = self.channel?.outputStream else {
return
}
var lngStr = "1234567890"
for _ in 1...10 {lngStr = lngStr + lngStr}
let data = lngStr.data(using: .utf8)!
let bytesWritten = data.withUnsafeBytes { ostream.write($0, maxLength: data.count) }
print("bytesWritten = \(bytesWritten)")
print("WR = \(bytesWritten) / \(data.count)")
}
And the execution result is:
bytesWritten = 8192
WR = 8192 / 10240
That allows me to see what happens in the case where bytesWritten < data.count.
In other words, all the bytes cannot be sent over in one chunk.
Now comes the question. The problem is I see nothing, the bytes left over seems to be just ignored.
I want to know what to do if I do not want to ignore those bytes. What is the way to care about the rest of the bytes? There will be cases where we will need to transfer tens of thousands or even hundreds of thousands of bytes.
You simply need to make a note of how many characters were sent, remove those from the data instance and then when you get a delegate callback indicating space is available in the output stream, send some more.
For example, you could add a couple of properties to hold the queued data and a serial dispatch queue to ensure thread-safe access to that queue:
private var queueQueue = DispatchQueue(label: "queue queue", qos: .userInitiated, attributes: [], autoreleaseFrequency: .workItem, target: nil)
private var outputData = Data()
Now, in the sendTextTapped function you can just add the new data to the output queue:
#IBAction func sendTextTapped(_ sender: UIButton) {
var lngStr = "1234567890"
for _ in 1...10 {lngStr = lngStr + lngStr}
let data = lngStr.data(using: .utf8)!
self.queue(data:data)
}
the queue(data:) function adds the data to the outputData object in a thread-safe manner and calls send()
private func queue(data: Data) {
queueQueue.sync {
self.outputData.append(data)
}
self.send()
}
send() ensures that the stream is connected, there is data to send and there is space available in the output stream. If all is OK then it sends as many bytes as it can. The sent bytes are then removed from output data (again in a thread safe manner).
private func send() {
guard let ostream = self.channel?.outputStream, !self.outputData.isEmpty, ostream.hasSpaceAvailable else{
return
}
let bytesWritten = outputData.withUnsafeBytes { ostream.write($0, maxLength: self.outputData.count) }
print("bytesWritten = \(bytesWritten)")
queueQueue.sync {
if bytesWritten < outputData.count {
outputData = outputData.advanced(by: bytesWritten)
} else {
outputData.removeAll()
}
}
}
The final change is to call send() in response to a .hasSpaceAvailable stream event:
func stream(_ aStream: Stream, handle eventCode: Stream.Event) {
switch eventCode {
case Stream.Event.openCompleted:
print("Stream is open")
case Stream.Event.endEncountered:
print("End Encountered")
case Stream.Event.hasBytesAvailable:
print("Bytes are available")
case Stream.Event.hasSpaceAvailable:
print("Space is available")
self.send()
case Stream.Event.errorOccurred:
print("Stream error")
default:
print("Unknown stream event")
}
}
You can see the modified code in the largedata branch of the example
iOS 12.x Swift 5
I am using sockets to talk to another box and I wrote this code which I think makes sense. but I wanted to check with an expert. I ask cause I think I have some sort of memory leak with this app. And I suspect pointers, as you do. I am watching the memory it uses slowly increase as I am running it.
func sendMessage(message: String) {
let data = message.data(using: String.Encoding.utf8, allowLossyConversion: false)!
let dataMutablePointer = UnsafeMutablePointer<UInt8>.allocate(capacity: data.count)
//Copies the bytes to the Mutable Pointer
dataMutablePointer.initialize(to: 0)
data.copyBytes(to: dataMutablePointer, count: data.count)
//Cast to regular UnsafePointer
let dataPointer = UnsafePointer<UInt8>(dataMutablePointer)
//Your stream
outputStream.write(dataPointer, maxLength: data.count)
defer {
dataMutablePointer.deinitialize(count: data.count)
}
}
Does this code look sane? Could it be the source of my memory leak? Do I need to deinitalize this pointer?
Could it be the source of my memory leak?
Yes, surely.
Do I need to deinitalize this pointer?
As commented by bscothern, you need to deinitalize the pointer if the Pointee is non-trivial type. But in case of UInt8, it is not mandatory.
But you need to deallocate the pointer eventually.
Your code would be something like this:
func sendMessage(message: String) {
let data = message.data(using: String.Encoding.utf8, allowLossyConversion: false)!
let dataMutablePointer = UnsafeMutablePointer<UInt8>.allocate(capacity: data.count)
defer {
dataMutablePointer.deallocate()
}
//Copies the bytes to the Mutable Pointer
dataMutablePointer.initialize(to: 0)
defer {
//This is not required, but you can put `deinitialize` here if you prefer
dataMutablePointer.deinitialize(count: data.count)
}
data.copyBytes(to: dataMutablePointer, count: data.count)
//Cast to regular UnsafePointer
let dataPointer = UnsafePointer<UInt8>(dataMutablePointer)
//Your stream
outputStream.write(dataPointer, maxLength: data.count)
}
defer defers the execution of its block until the end of the enclosing block, so dataMutablePointer.deallocate() will be executed after outputStream.write(dataPointer, maxLength: data.count) is finished.
But I would write something equivalent without copying data:
func sendMessage(message: String) {
let data = message.data(using: String.Encoding.utf8, allowLossyConversion: false)!
data.withUnsafeBytes {bytes in
let dataPointer = bytes.baseAddress!.assumingMemoryBound(to: UInt8.self)
//Your stream
outputStream.write(dataPointer, maxLength: data.count)
}
}
Or, more simply:
func sendMessage(message: String) {
//Your stream
outputStream.write(message, maxLength: message.utf8.count)
}
I'm doing a multiplayer game based on the MPC framework for the data communication between the devices of the players (and spritekit for the game).
The logic is that one of the device is the master, does all the game logic and calculations, and sends the positions of the various sprites to the slave devices that just have to update the sprites positions.
The master opens streams between him and all the slaves and sends the various data through these streams. Data are stored in dictionaries of type (String, Any).
The issue is twofold :
little by little I see a difference between the number of time the master send data and the number of time a slave received data
the more the master increase the number of data sent the more the slave receive empty/partial data
I have put various checks and controls, I don't have errors thrown by the send or receive functions, just lost data or empty/partial data.
Here is below some relevant parts of the code. Let me know if more is needed
Thx
J.
Stream initialization between master and slaves
Executed by the master
func InitMPCStream()
{
for Peer in Session.connectedPeers
{
if Peer.displayName != gMasterPeerName
{
let IndexPeer = gtListPlayerMulti.index(where: { $0.Name == Peer.displayName } )
try! gtListPlayerMulti[IndexPeer!].Stream = Session.startStream(withName: "Stream " + Peer.displayName, toPeer: Peer)
gtListPlayerMulti[IndexPeer!].Stream!.delegate = self
gtListPlayerMulti[IndexPeer!].Stream!.schedule(in: RunLoop.main, forMode:RunLoopMode.defaultRunLoopMode)
gtListPlayerMulti[IndexPeer!].Stream!.open()
}
}
}
Executed by the slaves
func session(_ session: MCSession, didReceive InStream: InputStream, withName StreamName: String, fromPeer PeerID: MCPeerID)
{
InputStream = InStream
InputStream.delegate = self
InputStream.schedule(in: RunLoop.main, forMode: RunLoopMode.defaultRunLoopMode)
InputStream.open()
}
Function that sends the data to the slaves
This function is executed on the master side in the update loop of the SKScene, several times per update, so very often.
The logic is that for each update cycle several events occur (between 10 to 100). Each event generates a call to this function. The quantity of data sent is not very important : a dictionary of (String, Any) of 3 to 10 lines where Any can be a string, number (Int, CGFloat...) or bool.
func SendStreamData(IDPeer: Int = 0, DataToSend: Dictionary<String, Any>)
{
let DataConverted = NSKeyedArchiver.archivedData(withRootObject: DataToSend)
if IDPeer == 0
{
for Peer in Session.connectedPeers
{
let IndexPeer = gtListPlayerMulti.index(where: { $0.Name == Peer.displayName } )
if gtListPlayerMulti[IndexPeer!].Stream!.hasSpaceAvailable
{
let bytesWritten = DataConverted.withUnsafeBytes { gtListPlayerMulti[IndexPeer!].Stream!.write($0, maxLength: DataConverted.count) }
if bytesWritten == -1 { print("Erreur send stream") } else { gSendingNumber = gSendingNumber + 1 }
} else { print("No space in stream") }
}
}
else
{
let IndexPeer = gtListPlayerMulti.index(where: { $0.ID == IDPeer } )
let bytesWritten = DataConverted.withUnsafeBytes { gtListPlayerMulti[IndexPeer!].Stream!.write($0, maxLength: DataConverted.count) }
}
}
Function called when data is received on the slave side
func stream(_ aStream: Stream, handle eventCode: Stream.Event)
{
switch(eventCode)
{
case Stream.Event.hasBytesAvailable:
gSendingNumber = gSendingNumber + 1
let InputStream = aStream as! InputStream
var Buffer = [UInt8](repeating: 0, count: 1024)
let NumberBytes = InputStream.read(&Buffer, maxLength:1024)
let DataString = NSData(bytes: &Buffer, length: NumberBytes)
if let Message = NSKeyedUnarchiver.unarchiveObject(with: DataString as Data) as? [String:Any] //deserializing the NSData
{ ProcessMPCDataReceived(VCMain: self, PeerID: PeerID, RawData: DataString as Data) }
else { print("Empty Data") }
case Stream.Event.hasSpaceAvailable:
break
case Stream.Event.errorOccurred:
print("ErrorOccurred: \(aStream.streamError?.localizedDescription)")
default:
break
}
}
The issue here is on the NSKeyedUnarchiver.unarchiveObject line which return nil because DataString does not conform to the expected dictionary
With Swift 3 leaning towards Data instead of [UInt8], I'm trying to ferret out what the most efficient/idiomatic way to encode/decode swifts various number types (UInt8, Double, Float, Int64, etc) as Data objects.
There's this answer for using [UInt8], but it seems to be using various pointer APIs that I can't find on Data.
I'd like to basically some custom extensions that look something like:
let input = 42.13 // implicit Double
let bytes = input.data
let roundtrip = bytes.to(Double) // --> 42.13
The part that really eludes me, I've looked through a bunch of the docs, is how I can get some sort of pointer thing (OpaquePointer or BufferPointer or UnsafePointer?) from any basic struct (which all of the numbers are). In C, I would just slap an ampersand in front of it, and there ya go.
Note: The code has been updated for Swift 5 (Xcode 10.2) now. (Swift 3 and Swift 4.2 versions can be found in the edit history.) Also possibly unaligned data is now correctly handled.
How to create Data from a value
As of Swift 4.2, data can be created from a value simply with
let value = 42.13
let data = withUnsafeBytes(of: value) { Data($0) }
print(data as NSData) // <713d0ad7 a3104540>
Explanation:
withUnsafeBytes(of: value)
invokes the closure with a buffer pointer covering the raw bytes of the value.
A raw buffer pointer is a sequence of bytes, therefore Data($0) can be used to create the data.
How to retrieve a value from Data
As of Swift 5, the withUnsafeBytes(_:) of Data invokes the closure with an “untyped” UnsafeMutableRawBufferPointer to the bytes. The load(fromByteOffset:as:) method the reads the value from the memory:
let data = Data([0x71, 0x3d, 0x0a, 0xd7, 0xa3, 0x10, 0x45, 0x40])
let value = data.withUnsafeBytes {
$0.load(as: Double.self)
}
print(value) // 42.13
There is one problem with this approach: It requires that the memory is property aligned for the type (here: aligned to a 8-byte address). But that is not guaranteed, e.g. if the data was obtained as a slice of another Data value.
It is therefore safer to copy the bytes to the value:
let data = Data([0x71, 0x3d, 0x0a, 0xd7, 0xa3, 0x10, 0x45, 0x40])
var value = 0.0
let bytesCopied = withUnsafeMutableBytes(of: &value, { data.copyBytes(to: $0)} )
assert(bytesCopied == MemoryLayout.size(ofValue: value))
print(value) // 42.13
Explanation:
withUnsafeMutableBytes(of:_:) invokes the closure with a mutable buffer pointer covering the raw bytes of the value.
The copyBytes(to:) method of DataProtocol (to which Data conforms) copies bytes from the data to that buffer.
The return value of copyBytes() is the number of bytes copied. It is equal to the size of the destination buffer, or less if the data does not contain enough bytes.
Generic solution #1
The above conversions can now easily be implemented as generic methods of struct Data:
extension Data {
init<T>(from value: T) {
self = Swift.withUnsafeBytes(of: value) { Data($0) }
}
func to<T>(type: T.Type) -> T? where T: ExpressibleByIntegerLiteral {
var value: T = 0
guard count >= MemoryLayout.size(ofValue: value) else { return nil }
_ = Swift.withUnsafeMutableBytes(of: &value, { copyBytes(to: $0)} )
return value
}
}
The constraint T: ExpressibleByIntegerLiteral is added here so that we can easily initialize the value to “zero” – that is not really a restriction because this method can be used with “trival” (integer and floating point) types anyway, see below.
Example:
let value = 42.13 // implicit Double
let data = Data(from: value)
print(data as NSData) // <713d0ad7 a3104540>
if let roundtrip = data.to(type: Double.self) {
print(roundtrip) // 42.13
} else {
print("not enough data")
}
Similarly, you can convert arrays to Data and back:
extension Data {
init<T>(fromArray values: [T]) {
self = values.withUnsafeBytes { Data($0) }
}
func toArray<T>(type: T.Type) -> [T] where T: ExpressibleByIntegerLiteral {
var array = Array<T>(repeating: 0, count: self.count/MemoryLayout<T>.stride)
_ = array.withUnsafeMutableBytes { copyBytes(to: $0) }
return array
}
}
Example:
let value: [Int16] = [1, Int16.max, Int16.min]
let data = Data(fromArray: value)
print(data as NSData) // <0100ff7f 0080>
let roundtrip = data.toArray(type: Int16.self)
print(roundtrip) // [1, 32767, -32768]
Generic solution #2
The above approach has one disadvantage: It actually works only with "trivial"
types like integers and floating point types. "Complex" types like Array
and String have (hidden) pointers to the underlying storage and cannot be
passed around by just copying the struct itself. It also would not work with
reference types which are just pointers to the real object storage.
So solve that problem, one can
Define a protocol which defines the methods for converting to Data and back:
protocol DataConvertible {
init?(data: Data)
var data: Data { get }
}
Implement the conversions as default methods in a protocol extension:
extension DataConvertible where Self: ExpressibleByIntegerLiteral{
init?(data: Data) {
var value: Self = 0
guard data.count == MemoryLayout.size(ofValue: value) else { return nil }
_ = withUnsafeMutableBytes(of: &value, { data.copyBytes(to: $0)} )
self = value
}
var data: Data {
return withUnsafeBytes(of: self) { Data($0) }
}
}
I have chosen a failable initializer here which checks that the number of bytes provided
matches the size of the type.
And finally declare conformance to all types which can safely be converted to Data and back:
extension Int : DataConvertible { }
extension Float : DataConvertible { }
extension Double : DataConvertible { }
// add more types here ...
This makes the conversion even more elegant:
let value = 42.13
let data = value.data
print(data as NSData) // <713d0ad7 a3104540>
if let roundtrip = Double(data: data) {
print(roundtrip) // 42.13
}
The advantage of the second approach is that you cannot inadvertently do unsafe conversions. The disadvantage is that you have to list all "safe" types explicitly.
You could also implement the protocol for other types which require a non-trivial conversion, such as:
extension String: DataConvertible {
init?(data: Data) {
self.init(data: data, encoding: .utf8)
}
var data: Data {
// Note: a conversion to UTF-8 cannot fail.
return Data(self.utf8)
}
}
or implement the conversion methods in your own types to do whatever is
necessary so serialize and deserialize a value.
Byte order
No byte order conversion is done in the above methods, the data is always in
the host byte order. For a platform independent representation (e.g.
“big endian” aka “network” byte order), use the corresponding integer
properties resp. initializers. For example:
let value = 1000
let data = value.bigEndian.data
print(data as NSData) // <00000000 000003e8>
if let roundtrip = Int(data: data) {
print(Int(bigEndian: roundtrip)) // 1000
}
Of course this conversion can also be done generally, in the generic
conversion method.
You can get an unsafe pointer to mutable objects by using withUnsafePointer:
withUnsafePointer(&input) { /* $0 is your pointer */ }
I don't know of a way to get one for immutable objects, because the inout operator only works on mutable objects.
This is demonstrated in the answer that you've linked to.
In my case, Martin R's answer helped but the result was inverted. So I did a small change in his code:
extension UInt16 : DataConvertible {
init?(data: Data) {
guard data.count == MemoryLayout<UInt16>.size else {
return nil
}
self = data.withUnsafeBytes { $0.pointee }
}
var data: Data {
var value = CFSwapInt16HostToBig(self)//Acho que o padrao do IOS 'e LittleEndian, pois os bytes estavao ao contrario
return Data(buffer: UnsafeBufferPointer(start: &value, count: 1))
}
}
The problem is related with LittleEndian and BigEndian.
I am trying to read a file given in an NSURL and load it into an array, with items separated by a newline character \n.
Here is the way I've done it so far:
var possList: NSString? = NSString.stringWithContentsOfURL(filePath.URL) as? NSString
if var list = possList {
list = list.componentsSeparatedByString("\n") as NSString[]
return list
}
else {
//return empty list
}
I'm not very happy with this for a couple of reasons. One, I'm working with files that range from a few kilobytes to hundreds of MB in size. As you can imagine, working with strings this large is slow and unwieldy. Secondly, this freezes up the UI when it's executing--again, not good.
I've looked into running this code in a separate thread, but I've been having trouble with that, and besides, it still doesn't solve the problem of dealing with huge strings.
What I'd like to do is something along the lines of the following pseudocode:
var aStreamReader = new StreamReader(from_file_or_url)
while aStreamReader.hasNextLine == true {
currentline = aStreamReader.nextLine()
list.addItem(currentline)
}
How would I accomplish this in Swift?
A few notes about the files I'm reading from: All files consist of short (<255 chars) strings separated by either \n or \r\n. The length of the files range from ~100 lines to over 50 million lines. They may contain European characters, and/or characters with accents.
(The code is for Swift 2.2/Xcode 7.3 now. Older versions can be found in the edit history if somebody needs it. An updated version for Swift 3 is provided at the end.)
The following Swift code is heavily inspired by the various answers to
How to read data from NSFileHandle line by line?. It reads from the file in chunks, and converts complete lines to strings.
The default line delimiter (\n), string encoding (UTF-8) and chunk size (4096)
can be set with optional parameters.
class StreamReader {
let encoding : UInt
let chunkSize : Int
var fileHandle : NSFileHandle!
let buffer : NSMutableData!
let delimData : NSData!
var atEof : Bool = false
init?(path: String, delimiter: String = "\n", encoding : UInt = NSUTF8StringEncoding, chunkSize : Int = 4096) {
self.chunkSize = chunkSize
self.encoding = encoding
if let fileHandle = NSFileHandle(forReadingAtPath: path),
delimData = delimiter.dataUsingEncoding(encoding),
buffer = NSMutableData(capacity: chunkSize)
{
self.fileHandle = fileHandle
self.delimData = delimData
self.buffer = buffer
} else {
self.fileHandle = nil
self.delimData = nil
self.buffer = nil
return nil
}
}
deinit {
self.close()
}
/// Return next line, or nil on EOF.
func nextLine() -> String? {
precondition(fileHandle != nil, "Attempt to read from closed file")
if atEof {
return nil
}
// Read data chunks from file until a line delimiter is found:
var range = buffer.rangeOfData(delimData, options: [], range: NSMakeRange(0, buffer.length))
while range.location == NSNotFound {
let tmpData = fileHandle.readDataOfLength(chunkSize)
if tmpData.length == 0 {
// EOF or read error.
atEof = true
if buffer.length > 0 {
// Buffer contains last line in file (not terminated by delimiter).
let line = NSString(data: buffer, encoding: encoding)
buffer.length = 0
return line as String?
}
// No more lines.
return nil
}
buffer.appendData(tmpData)
range = buffer.rangeOfData(delimData, options: [], range: NSMakeRange(0, buffer.length))
}
// Convert complete line (excluding the delimiter) to a string:
let line = NSString(data: buffer.subdataWithRange(NSMakeRange(0, range.location)),
encoding: encoding)
// Remove line (and the delimiter) from the buffer:
buffer.replaceBytesInRange(NSMakeRange(0, range.location + range.length), withBytes: nil, length: 0)
return line as String?
}
/// Start reading from the beginning of file.
func rewind() -> Void {
fileHandle.seekToFileOffset(0)
buffer.length = 0
atEof = false
}
/// Close the underlying file. No reading must be done after calling this method.
func close() -> Void {
fileHandle?.closeFile()
fileHandle = nil
}
}
Usage:
if let aStreamReader = StreamReader(path: "/path/to/file") {
defer {
aStreamReader.close()
}
while let line = aStreamReader.nextLine() {
print(line)
}
}
You can even use the reader with a for-in loop
for line in aStreamReader {
print(line)
}
by implementing the SequenceType protocol (compare http://robots.thoughtbot.com/swift-sequences):
extension StreamReader : SequenceType {
func generate() -> AnyGenerator<String> {
return AnyGenerator {
return self.nextLine()
}
}
}
Update for Swift 3/Xcode 8 beta 6: Also "modernized" to
use guard and the new Data value type:
class StreamReader {
let encoding : String.Encoding
let chunkSize : Int
var fileHandle : FileHandle!
let delimData : Data
var buffer : Data
var atEof : Bool
init?(path: String, delimiter: String = "\n", encoding: String.Encoding = .utf8,
chunkSize: Int = 4096) {
guard let fileHandle = FileHandle(forReadingAtPath: path),
let delimData = delimiter.data(using: encoding) else {
return nil
}
self.encoding = encoding
self.chunkSize = chunkSize
self.fileHandle = fileHandle
self.delimData = delimData
self.buffer = Data(capacity: chunkSize)
self.atEof = false
}
deinit {
self.close()
}
/// Return next line, or nil on EOF.
func nextLine() -> String? {
precondition(fileHandle != nil, "Attempt to read from closed file")
// Read data chunks from file until a line delimiter is found:
while !atEof {
if let range = buffer.range(of: delimData) {
// Convert complete line (excluding the delimiter) to a string:
let line = String(data: buffer.subdata(in: 0..<range.lowerBound), encoding: encoding)
// Remove line (and the delimiter) from the buffer:
buffer.removeSubrange(0..<range.upperBound)
return line
}
let tmpData = fileHandle.readData(ofLength: chunkSize)
if tmpData.count > 0 {
buffer.append(tmpData)
} else {
// EOF or read error.
atEof = true
if buffer.count > 0 {
// Buffer contains last line in file (not terminated by delimiter).
let line = String(data: buffer as Data, encoding: encoding)
buffer.count = 0
return line
}
}
}
return nil
}
/// Start reading from the beginning of file.
func rewind() -> Void {
fileHandle.seek(toFileOffset: 0)
buffer.count = 0
atEof = false
}
/// Close the underlying file. No reading must be done after calling this method.
func close() -> Void {
fileHandle?.closeFile()
fileHandle = nil
}
}
extension StreamReader : Sequence {
func makeIterator() -> AnyIterator<String> {
return AnyIterator {
return self.nextLine()
}
}
}
Efficient and convenient class for reading text file line by line (Swift 4, Swift 5)
Note: This code is platform independent (macOS, iOS, ubuntu)
import Foundation
/// Read text file line by line in efficient way
public class LineReader {
public let path: String
fileprivate let file: UnsafeMutablePointer<FILE>!
init?(path: String) {
self.path = path
file = fopen(path, "r")
guard file != nil else { return nil }
}
public var nextLine: String? {
var line:UnsafeMutablePointer<CChar>? = nil
var linecap:Int = 0
defer { free(line) }
return getline(&line, &linecap, file) > 0 ? String(cString: line!) : nil
}
deinit {
fclose(file)
}
}
extension LineReader: Sequence {
public func makeIterator() -> AnyIterator<String> {
return AnyIterator<String> {
return self.nextLine
}
}
}
Usage:
guard let reader = LineReader(path: "/Path/to/file.txt") else {
return; // cannot open file
}
for line in reader {
print(">" + line.trimmingCharacters(in: .whitespacesAndNewlines))
}
Repository on github
Swift 4.2 Safe syntax
class LineReader {
let path: String
init?(path: String) {
self.path = path
guard let file = fopen(path, "r") else {
return nil
}
self.file = file
}
deinit {
fclose(file)
}
var nextLine: String? {
var line: UnsafeMutablePointer<CChar>?
var linecap = 0
defer {
free(line)
}
let status = getline(&line, &linecap, file)
guard status > 0, let unwrappedLine = line else {
return nil
}
return String(cString: unwrappedLine)
}
private let file: UnsafeMutablePointer<FILE>
}
extension LineReader: Sequence {
func makeIterator() -> AnyIterator<String> {
return AnyIterator<String> {
return self.nextLine
}
}
}
Usage:
guard let reader = LineReader(path: "/Path/to/file.txt") else {
return
}
reader.forEach { line in
print(line.trimmingCharacters(in: .whitespacesAndNewlines))
}
I'm late to the game, but here's small class I wrote for that purpose. After some different attempts (try to subclass NSInputStream) I found this to be a reasonable and simple approach.
Remember to #import <stdio.h> in your bridging header.
// Use is like this:
let readLine = ReadLine(somePath)
while let line = readLine.readLine() {
// do something...
}
class ReadLine {
private var buf = UnsafeMutablePointer<Int8>.alloc(1024)
private var n: Int = 1024
let path: String
let mode: String = "r"
private lazy var filepointer: UnsafeMutablePointer<FILE> = {
let csmode = self.mode.withCString { cs in return cs }
let cspath = self.path.withCString { cs in return cs }
return fopen(cspath, csmode)
}()
init(path: String) {
self.path = path
}
func readline() -> String? {
// unsafe for unknown input
if getline(&buf, &n, filepointer) > 0 {
return String.fromCString(UnsafePointer<CChar>(buf))
}
return nil
}
deinit {
buf.dealloc(n)
fclose(filepointer)
}
}
This function takes a file URL and returns a sequence which will return every line of the file, reading them lazily. It works with Swift 5. It relies on the underlying getline:
typealias LineState = (
// pointer to a C string representing a line
linePtr:UnsafeMutablePointer<CChar>?,
linecap:Int,
filePtr:UnsafeMutablePointer<FILE>?
)
/// Returns a sequence which iterates through all lines of the the file at the URL.
///
/// - Parameter url: file URL of a file to read
/// - Returns: a Sequence which lazily iterates through lines of the file
///
/// - warning: the caller of this function **must** iterate through all lines of the file, since aborting iteration midway will leak memory and a file pointer
/// - precondition: the file must be UTF8-encoded (which includes, ASCII-encoded)
func lines(ofFile url:URL) -> UnfoldSequence<String,LineState>
{
let initialState:LineState = (linePtr:nil, linecap:0, filePtr:fopen(url.path,"r"))
return sequence(state: initialState, next: { (state) -> String? in
if getline(&state.linePtr, &state.linecap, state.filePtr) > 0,
let theLine = state.linePtr {
return String.init(cString:theLine)
}
else {
if let actualLine = state.linePtr { free(actualLine) }
fclose(state.filePtr)
return nil
}
})
}
So for instance, here's how you would use it to print every line of a file named "foo" in your app bundle:
let url = NSBundle.mainBundle().urlForResource("foo", ofType: nil)!
for line in lines(ofFile:url) {
// suppress print's automatically inserted line ending, since
// lineGenerator captures each line's own new line character.
print(line, separator: "", terminator: "")
}
I developed this answer by modifying Alex Brown's answer to remove a memory leak mentioned by Martin R's comment, and by updating it to for Swift 5.
Try this answer, or read the Mac OS Stream Programming Guide.
You may find that performance will actually be better using the stringWithContentsOfURL, though, as it will be quicker to work with memory-based (or memory-mapped) data than disc-based data.
Executing it on another thread is well documented, also, for example here.
Update
If you don't want to read it all at once, and you don't want to use NSStreams, then you'll probably have to use C-level file I/O. There are many reasons not to do this - blocking, character encoding, handling I/O errors, speed to name but a few - this is what the Foundation libraries are for. I've sketched a simple answer below that just deals with ACSII data:
class StreamReader {
var eofReached = false
let fileHandle: UnsafePointer<FILE>
init (path: String) {
self.fileHandle = fopen(path.bridgeToObjectiveC().UTF8String, "rb".bridgeToObjectiveC().UTF8String)
}
deinit {
fclose(self.fileHandle)
}
func nextLine() -> String {
var nextChar: UInt8 = 0
var stringSoFar = ""
var eolReached = false
while (self.eofReached == false) && (eolReached == false) {
if fread(&nextChar, 1, 1, self.fileHandle) == 1 {
switch nextChar & 0xFF {
case 13, 10 : // CR, LF
eolReached = true
case 0...127 : // Keep it in ASCII
stringSoFar += NSString(bytes:&nextChar, length:1, encoding: NSASCIIStringEncoding)
default :
stringSoFar += "<\(nextChar)>"
}
} else { // EOF or error
self.eofReached = true
}
}
return stringSoFar
}
}
// OP's original request follows:
var aStreamReader = StreamReader(path: "~/Desktop/Test.text".stringByStandardizingPath)
while aStreamReader.eofReached == false { // Changed property name for more accurate meaning
let currentline = aStreamReader.nextLine()
//list.addItem(currentline)
println(currentline)
}
Or you could simply use a Generator:
let stdinByLine = GeneratorOf({ () -> String? in
var input = UnsafeMutablePointer<Int8>(), lim = 0
return getline(&input, &lim, stdin) > 0 ? String.fromCString(input) : nil
})
Let's try it out
for line in stdinByLine {
println(">>> \(line)")
}
It's simple, lazy, and easy to chain with other swift things like enumerators and functors such as map, reduce, filter; using the lazy() wrapper.
It generalises to all FILE as:
let byLine = { (file:UnsafeMutablePointer<FILE>) in
GeneratorOf({ () -> String? in
var input = UnsafeMutablePointer<Int8>(), lim = 0
return getline(&input, &lim, file) > 0 ? String.fromCString(input) : nil
})
}
called like
for line in byLine(stdin) { ... }
Following up on #dankogai's answer, I made a few modifications for Swift 4+,
let bufsize = 4096
let fp = fopen(jsonURL.path, "r");
var buf = UnsafeMutablePointer<Int8>.allocate(capacity: bufsize)
while (fgets(buf, Int32(bufsize-1), fp) != nil) {
print( String(cString: buf) )
}
buf.deallocate()
This worked for me.
Thanks
Swift 5.5: use url.lines
ADC Docs are here
Example usage:
guard let url = URL(string: "https://www.example.com") else {
return
}
// Manipulating an `Array` in memory seems to be a requirement.
// This will balloon in size as lines of data get added.
var myHugeArray = [String]()
do {
// This should keep the inbound data memory usage low
for try await line in url.lines {
myHugeArray.append(line)
}
} catch {
debugPrint(error)
}
You can use this in a SwiftUI .task { } modifier or wrap this in a Task return type to get its work off the main thread.
It turns out good old-fasioned C API is pretty comfortable in Swift once you grok UnsafePointer. Here is a simple cat that reads from stdin and prints to stdout line-by-line. You don't even need Foundation. Darwin suffices:
import Darwin
let bufsize = 4096
// let stdin = fdopen(STDIN_FILENO, "r") it is now predefined in Darwin
var buf = UnsafePointer<Int8>.alloc(bufsize)
while fgets(buf, Int32(bufsize-1), stdin) {
print(String.fromCString(CString(buf)))
}
buf.destroy()
(Note: I'm using Swift 3.0.1 on Xcode 8.2.1 with macOS Sierra 10.12.3)
All of the answers I've seen here missed that he could be looking for LF or CRLF. If everything goes well, s/he could just match on LF and check the returned string for an extra CR at the end. But the general query involves multiple search strings. In other words, the delimiter needs to be a Set<String>, where the set is neither empty nor contains the empty string, instead of a single string.
On my first try at this last year, I tried to do the "right thing" and search for a general set of strings. It was too hard; you need a full blown parser and state machines and such. I gave up on it and the project it was part of.
Now I'm doing the project again, and facing the same challenge again. Now I'm going to hard-code searching on CR and LF. I don't think anyone would need to search on two semi-independent and semi-dependent characters like this outside of CR/LF parsing.
I'm using the search methods provided by Data, so I'm not doing string encodings and stuff here. Just raw binary processing. Just assume I got an ASCII superset, like ISO Latin-1 or UTF-8, here. You can handle string encoding at the next-higher layer, and you punt on whether a CR/LF with secondary code-points attached still counts as a CR or LF.
The algorithm: just keep searching for the next CR and the next LF from your current byte offset.
If neither is found, then consider the next data string to be from the current offset to the end-of-data. Note that the terminator length is 0. Mark this as the end of your reading loop.
If a LF is found first, or only a LF is found, consider the next data string to be from the current offset to the LF. Note that the terminator length is 1. Move the offset to after the LF.
If only a CR is found, do like the LF case (just with a different byte value).
Otherwise, we got a CR followed by a LF.
If the two are adjacent, then handle like the LF case, except the terminator length will be 2.
If there is one byte between them, and said byte is also CR, then we got the "Windows developer wrote a binary \r\n while in text mode, giving a \r\r\n" problem. Also handle it like the LF case, except the terminator length will be 3.
Otherwise the CR and LF aren't connected, and handle like the just-CR case.
Here's some code for that:
struct DataInternetLineIterator: IteratorProtocol {
/// Descriptor of the location of a line
typealias LineLocation = (offset: Int, length: Int, terminatorLength: Int)
/// Carriage return.
static let cr: UInt8 = 13
/// Carriage return as data.
static let crData = Data(repeating: cr, count: 1)
/// Line feed.
static let lf: UInt8 = 10
/// Line feed as data.
static let lfData = Data(repeating: lf, count: 1)
/// The data to traverse.
let data: Data
/// The byte offset to search from for the next line.
private var lineStartOffset: Int = 0
/// Initialize with the data to read over.
init(data: Data) {
self.data = data
}
mutating func next() -> LineLocation? {
guard self.data.count - self.lineStartOffset > 0 else { return nil }
let nextCR = self.data.range(of: DataInternetLineIterator.crData, options: [], in: lineStartOffset..<self.data.count)?.lowerBound
let nextLF = self.data.range(of: DataInternetLineIterator.lfData, options: [], in: lineStartOffset..<self.data.count)?.lowerBound
var location: LineLocation = (self.lineStartOffset, -self.lineStartOffset, 0)
let lineEndOffset: Int
switch (nextCR, nextLF) {
case (nil, nil):
lineEndOffset = self.data.count
case (nil, let offsetLf):
lineEndOffset = offsetLf!
location.terminatorLength = 1
case (let offsetCr, nil):
lineEndOffset = offsetCr!
location.terminatorLength = 1
default:
lineEndOffset = min(nextLF!, nextCR!)
if nextLF! < nextCR! {
location.terminatorLength = 1
} else {
switch nextLF! - nextCR! {
case 2 where self.data[nextCR! + 1] == DataInternetLineIterator.cr:
location.terminatorLength += 1 // CR-CRLF
fallthrough
case 1:
location.terminatorLength += 1 // CRLF
fallthrough
default:
location.terminatorLength += 1 // CR-only
}
}
}
self.lineStartOffset = lineEndOffset + location.terminatorLength
location.length += self.lineStartOffset
return location
}
}
Of course, if you have a Data block of a length that's at least a significant fraction of a gigabyte, you'll take a hit whenever no more CR or LF exist from the current byte offset; always fruitlessly searching until the end during every iteration. Reading the data in chunks would help:
struct DataBlockIterator: IteratorProtocol {
/// The data to traverse.
let data: Data
/// The offset into the data to read the next block from.
private(set) var blockOffset = 0
/// The number of bytes remaining. Kept so the last block is the right size if it's short.
private(set) var bytesRemaining: Int
/// The size of each block (except possibly the last).
let blockSize: Int
/// Initialize with the data to read over and the chunk size.
init(data: Data, blockSize: Int) {
precondition(blockSize > 0)
self.data = data
self.bytesRemaining = data.count
self.blockSize = blockSize
}
mutating func next() -> Data? {
guard bytesRemaining > 0 else { return nil }
defer { blockOffset += blockSize ; bytesRemaining -= blockSize }
return data.subdata(in: blockOffset..<(blockOffset + min(bytesRemaining, blockSize)))
}
}
You have to mix these ideas together yourself, since I haven't done it yet. Consider:
Of course, you have to consider lines completely contained in a chunk.
But you have to handle when the ends of a line are in adjacent chunks.
Or when the endpoints have at least one chunk between them
The big complication is when the line ends with a multi-byte sequence, but said sequence straddles two chunks! (A line ending in just CR that's also the last byte in the chunk is an equivalent case, since you need to read the next chunk to see if your just-CR is actually a CRLF or CR-CRLF. There are similar shenanigans when the chunk ends with CR-CR.)
And you need to handle when there are no more terminators from your current offset, but the end-of-data is in a later chunk.
Good luck!
I wanted a version that did not continually modify the buffer or duplicate code, as both are inefficient, and would allow for any size buffer (including 1 byte) and any delimiter. It has one public method: readline(). Calling this method will return the String value of the next line or nil at EOF.
import Foundation
// LineStream(): path: String, [buffSize: Int], [delim: String] -> nil | String
// ============= --------------------------------------------------------------
// path: the path to a text file to be parsed
// buffSize: an optional buffer size, (1...); default is 4096
// delim: an optional delimiter String; default is "\n"
// ***************************************************************************
class LineStream {
let path: String
let handle: NSFileHandle!
let delim: NSData!
let encoding: NSStringEncoding
var buffer = NSData()
var buffSize: Int
var buffIndex = 0
var buffEndIndex = 0
init?(path: String,
buffSize: Int = 4096,
delim: String = "\n",
encoding: NSStringEncoding = NSUTF8StringEncoding)
{
self.handle = NSFileHandle(forReadingAtPath: path)
self.path = path
self.buffSize = buffSize < 1 ? 1 : buffSize
self.encoding = encoding
self.delim = delim.dataUsingEncoding(encoding)
if handle == nil || self.delim == nil {
print("ERROR initializing LineStream") /* TODO use STDERR */
return nil
}
}
// PRIVATE
// fillBuffer(): _ -> Int [0...buffSize]
// ============= -------- ..............
// Fill the buffer with new data; return with the buffer size, or zero
// upon reaching end-of-file
// *********************************************************************
private func fillBuffer() -> Int {
buffer = handle.readDataOfLength(buffSize)
buffIndex = 0
buffEndIndex = buffer.length
return buffEndIndex
}
// PRIVATE
// delimLocation(): _ -> Int? nil | [1...buffSize]
// ================ --------- ....................
// Search the remaining buffer for a delimiter; return with the location
// of a delimiter in the buffer, or nil if one is not found.
// ***********************************************************************
private func delimLocation() -> Int? {
let searchRange = NSMakeRange(buffIndex, buffEndIndex - buffIndex)
let rangeToDelim = buffer.rangeOfData(delim,
options: [], range: searchRange)
return rangeToDelim.location == NSNotFound
? nil
: rangeToDelim.location
}
// PRIVATE
// dataStrValue(): NSData -> String ("" | String)
// =============== ---------------- .............
// Attempt to convert data into a String value using the supplied encoding;
// return the String value or empty string if the conversion fails.
// ***********************************************************************
private func dataStrValue(data: NSData) -> String? {
if let strVal = NSString(data: data, encoding: encoding) as? String {
return strVal
} else { return "" }
}
// PUBLIC
// readLine(): _ -> String? nil | String
// =========== ____________ ............
// Read the next line of the file, i.e., up to the next delimiter or end-of-
// file, whichever occurs first; return the String value of the data found,
// or nil upon reaching end-of-file.
// *************************************************************************
func readLine() -> String? {
guard let line = NSMutableData(capacity: buffSize) else {
print("ERROR setting line")
exit(EXIT_FAILURE)
}
// Loop until a delimiter is found, or end-of-file is reached
var delimFound = false
while !delimFound {
// buffIndex will equal buffEndIndex in three situations, resulting
// in a (re)filling of the buffer:
// 1. Upon the initial call;
// 2. If a search for a delimiter has failed
// 3. If a delimiter is found at the end of the buffer
if buffIndex == buffEndIndex {
if fillBuffer() == 0 {
return nil
}
}
var lengthToDelim: Int
let startIndex = buffIndex
// Find a length of data to place into the line buffer to be
// returned; reset buffIndex
if let delim = delimLocation() {
// SOME VALUE when a delimiter is found; append that amount of
// data onto the line buffer,and then return the line buffer
delimFound = true
lengthToDelim = delim - buffIndex
buffIndex = delim + 1 // will trigger a refill if at the end
// of the buffer on the next call, but
// first the line will be returned
} else {
// NIL if no delimiter left in the buffer; append the rest of
// the buffer onto the line buffer, refill the buffer, and
// continue looking
lengthToDelim = buffEndIndex - buffIndex
buffIndex = buffEndIndex // will trigger a refill of buffer
// on the next loop
}
line.appendData(buffer.subdataWithRange(
NSMakeRange(startIndex, lengthToDelim)))
}
return dataStrValue(line)
}
}
It is called as follows:
guard let myStream = LineStream(path: "/path/to/file.txt")
else { exit(EXIT_FAILURE) }
while let s = myStream.readLine() {
print(s)
}