twisted buffer full in tcp connection - ios

Question

I´m having problems with receiving long data (>1024bytes) in a simple twisted server implementation.
From the beginning, I´m developing an ios App that has to synchronize with a twisted server. I prepare the information to send in JSON format. Then I start to send that data in chuncks (right now in chunck of 256bytes + 4 bytes for the command - Yes, I´m implementing my own protocol). The connection is ok, and I receive those packet´s in my server (in the dataReceived function of my own Protocol subclass).
The ios method: NSInteger writtenBytes =[self.outputStream write:[data bytes] maxLength:[data length]] return the written bytes into the stream. For the first 4 packets the value returned is the expected (260 bytes). If I have more available bytes to send, the next time I call that method it returns 0 (which apple documentation says: "If the receiver is a fixed-length stream and has reached its capacity, 0 is returned.").
So I deduce that the input buffer is full. I don´t know how to free that buffer (I don´t know how to reach that buffer). I don't know where is the limit of that buffer (it seems to me almost ridiculous).
This is a basic test of the server (Just the important things for this question with a basic based in strings protocol)
from twisted.internet.protocol import Protocol, Factory
from twisted.internet import reactor
class IphoneSync(Protocol):
def __init__(self):
self.__buffer = ""
def connectionMade(self):
self.transport.write("0:")
self.factory.clients.append(self)
print "clients are ", self.factory.clients
def connectionLost(self, reason):
self.factory.clients.remove(self)
def dataReceived(self, data):
#print "data is ", data
a = data.split(':')
if len(a) > 1:
command = a[0]
content = a[1]
msg = ""
if command == "iam":
#user&Pass checking
msg = "1"
elif command == "msg":
self.__buffer += data
msg = "1: continue"
elif command == "fin":
#procesaremos todo
#Convertir datos en json
#insertar/actualizar data en sqlite
#devolver respuesta
print "buffer is", self.__buffer
msg = "2: procesing"
print msg
self.transport.write(msg)
#for c in self.factory.clients:
#c.message(msg)
def message(self, message):
self.transport.write(message)
#self.transport.write(message + '\n')
factory = Factory()
factory.protocol = IphoneSync
factory.clients = []
dir(factory)
reactor.listenTCP(8000, factory)
print "Iphone Chat server started"
reactor.run()
I saw the LineReceiver class but i´m not sending lines. The transfered data could be very big (10Mb-50Mb). I´m thinking about the Consumer/Producer model, or RPC Protocols like (AMP, or PB) as a solution but i wanted to work with my own protocol.
If someone knows how to help me, i´ll appreciate very much. Thanks anyway.

Answer 1

The connection is ok, and I receive those packet´s in my server (in the dataReceived function of my own Protocol subclass).
Probably not. TCP is a "stream oriented" protocol. Your application's use of it is not in terms of packets but in terms of a sequence of bytes. There is no guarantee whatsoever that dataReceived will be called with the same string that you passed to outputStream write. If you write "hello, world", dataReceived may be called with "hello, world" - or it may be called twice, first with "hello," and then with " world". Or it may be called 12 times: first "h", then "e", then "l", etc.
And if you call outputStream write twice, once with "hello," and once with " world", then it's entirely possible dataReceived will be called just once with "hello, world". Or perhaps twice, but with "h" and then "ello, world".
So this brand new protocol you're inventing (which I see you mentioned you recognized you were doing, but you didn't explain why this is a good idea or an important part of your application, instead of just a large source of potential bugs and a poor use of time :) has to do something called "framing" in order to let you actually interpret the byte sequence being passed around. This is why there are protocols like AMP.
To actually answer your question, outputStream write returns the number of bytes it was actually able to buffer for sending. You must always check its return value and re-try writing any bytes it wasn't able to send, preferably after waiting for notification that there is more buffer space. Buffer space becomes available after bytes using that space are sent across the network and acknowledged by the receiver. This takes time, as networks are not instantaneous. Notification about buffer space being available comes in many forms, the oldest and most widespread of which (but not necessarily the best in your environment), the select(2) system call.

Answer 2

In addition to Jean-Paul Calderone's answer (ensuring that data are being sent completely from the obj-c side by using select or thread), for protocol part I would suggest using length-prefixed string (AKA Netstring) for simple use case.
Here's an implementation. Whenever something is received, you need to call NSBuffer.write then NSBuffer.extract to get available strings.