Reading bytes from stream, and convert to string - delphi

Im having a problem while reading a stream from a Client. Im using Delphi XE3 with Indy 10.
This Client, send me a tagged text data stream, that is in XML format. My problem start when i read data with a TIdTCPServer. With this code:
if Acontext.Connection.IoHandler.InputbufferisEmpty then
begin
Acontext.Connection.IoHandler.CheckforDataonSource(1000);
Acontext.Connection.IoHandler.CheckForDisconnect;
if Acontext.Connection.IoHandler.InputbufferisEmpty then Exit;
end;
Acontext.Connection.Iohandler.ReadBytes(Abuffer,
Acontext.connection.ioHandler.Inputbuffer.size);
data := BytesToString(Abuffer, Tencoding.bigendianUnicode);
Data is encoded in BigEndianUnicode. I noticed that when im recieving data, the server read and show me the data in 2 'blocks' diferents. If the server recieve the same data all the time, when i parse with BytesToString the data get "cut" at same char everytime.
Here is a result of what im getting:
<rootNode>
<Node1></Node1>
<Node2></Node2>
<
Node3></Node3>
<Node4></Node4>
The problem get bigger, when i assign that WideString to an XMLDocument. I get an Invalid at the top level of the document error.
Sometimes, i run the server and get all the data correct, without this error, but when i close the aplication and run it again, the server recieve the data corrupted.
Sorry, for my english. I need some help with this.

You are only reading whatever raw bytes happen to be in the IOHandler.InputBuffer at the moment you access it, so you are likely not reading the entire XML, or worse you are reading more data that follows the XML. If the client is sensible, it should be sending either the XML length before the XML data, or a unique terminator after the XML data. Either of which would allow you to read the entire XML by itself in full without having to resort to calling CheckforDataOnSource() at all (which you should be avoiding in most situations), eg:
len := AContext.Connection.IOHandler.ReadLongWord; // or however the XML length is sent
data := AContext.Connection.IOHandler.ReadString(len, IndyUTF16BigEndianEncoding);
Or:
data := AContext.Connection.IOHandler.ReadLn('the terminator here', IndyUTF16BigEndianEncoding);

Related

Default TEncoding.UTF8 discarding invalid blocks of data in TStreamReader input

I'm using a TStreamReader to read data from a file that purports to be utf-8. I have no problem reading the file until it comes to a section containing what appears to me to be a UTF-8 "£" symbol with the preceding xC2 missing - the file only contains the xA3 part of the character. I've traced this through the run-time library until it calls
Result := UnicodeFromLocaleChars(FCodePage, FMBToWCharFlags,
PAnsiChar(Bytes), ByteCount, nil, 0);
which returns 0 indicating that it doesn't like the input. Unfortunately the TStreamReader simply ends up discarding this buffer of input and then continues with the rest of the file without raising an error. This is extremely misleading about what the problem but that is just a side issue.
The issue appears to be a "defect" in the UTF-8 TEncoding class in that it simply discards the results of a failed conversion whilst the TStreamReader assumes that this isn't the behaviour of TEncoding.
I can work around this by using
Reader := TStreamReader.Create(FileStream, TMBCSEncoding.Create(CP_UTF8, 0, 0));
instead of
Reader := TStreamReader.Create(FileStream, TEncoding.UTF8);
as this makes it ignore the corrupt UTF-8 and simply include something (I haven't checked what) in my output. However, I would like to combine allowing the data through with reporting it and there doesn't seem to be any obvious way of doing this as the behaviour is hidden deep within the library.
Does anyone know of any standard Delphi library tools for doing this or do I need to resort to a lot of custom code?

Imap4 client command LSUB

I have a problem with function TIdIMAP4.ListSubscribedMailBoxes(AMailBoxList: TStrings): Boolean; with this implementation :
function TIdIMAP4.ListSubscribedMailBoxes(AMailBoxList: TStrings): Boolean;
begin
{CC2: This is one of the few cases where the server can return only "OK completed"
meaning that the user has no subscribed mailboxes.}
Result := False;
CheckConnectionState([csAuthenticated, csSelected]);
SendCmd(NewCmdCounter, IMAP4Commands[cmdLSub] + ' "" *',
[IMAP4Commands[cmdList], IMAP4Commands[cmdLSub]]); {Do not Localize}
if LastCmdResult.Code = IMAP_OK then begin
// ds - fixed bug # 506026
ParseLSubResult(AMailBoxList, LastCmdResult.Text);
Result := True;
end;
end;
When I debug I see that the LastCmdResult.Text stringlist is empty, but the LastCmdResult.FormattedReply stringlist has all folders on my email server (Inbox, Sent, Trash, ...). When I tried to use LastCmdResult.FormattedReply count or text, it had immediately lost its data and gave LastCmdResult.FormattedReply.Count=0 and LastCmdResult.FormattedReply.Text=''. So I'd like to know if there is a way to enter the data inside LastCmdResult.FormattedReply and get my email server folders or there is another way to solve my problem ?
I have a problem with function TIdIMAP4.ListSubscribedMailBoxes(AMailBoxList: TStrings): Boolean; with this implementation :
Works fine for me when I try it using the latest SVN version of Indy.
When I debug I see that the LastCmdResult.Text stringlist is empty, but the LastCmdResult.FormattedReply stringlist has all folders on my email server (Inbox, Sent, Trash, ...).
When I run it, the opposite happens. LastCmdResult.Text contains the expected text, and LastCmdResult.FFormattedReply is empty (notice I mention the FFormattedReply data member directly, see below).
When I tried to use LastCmdResult.FormattedReply count or text, it had immediately lost its data and gave LastCmdResult.FormattedReply.Count=0 and LastCmdResult.FormattedReply.Text=''.
That is by design. The FormattedReply property is intended to be used by a client to parse a server reply so it can populate TIdReply's property values, and to be used by a server to generate a new reply using TIdReply's property values. So, you cannot read from the FormattedReply property on the client side.
So I'd like to know if there is a way to enter the data inside LastCmdResult.FormattedReply and get my email server folders or there is another way to solve my problem ?
The whole purpose of ListSubscribedMailBoxes() is to return the folder names in the AMailBoxList parameter. If that is not working for you, then either
you are using a older/buggy version of Indy.
your server is sending the data in a format that TIdIMAP4 is not able to parse.
Without knowing which version of Indy you are actually using, or what the server's reply data actually looks like, there is no way to diagnose your issue one way or the other.

Testing Somesite CRLF Bugs Hole Using Indy IdMappedPortTCP

I want to create a small tool like CRLF Injection or HTTP header respons splitting. I was successful created thousands NetData pattern (data payload) lists. The NetData pattern like this example:
GET http://somebug.com/ HTTP/1.1[CRLF]Host : somehost.com[CRLF]GET somesite.com HTTP/1.1[CRLF][CRLF]
GET http://somebug.com/ HTTP/1.1[CRLF]Host : somehost.com[CRLF][CRLF]GET somesitesite.com HTTP/1.1[CRLF][CRLF]
HEAD http://somebug.com/ HTTP/1.1[CRLF]Host : somehost.com[CRLF]CONNECT somesitesite.com HTTP/1.0[CRLF][CRLF][CRLF][CRLF]
...
If just one data pattern/data payload, I can write example code like:
procedure T_CRLFTest.IdMappedPortTCP1Execute(AContext: TIdContext);
begin
if(Pos('CONNECT',TIdMappedPortContext(AContext).NetData)<>0) then
TIdMappedPortContext(AContext).NetData := 'GET http://somebug.com/ HTTP/1.1'#13#10'Host : somehost.com'#13#10+TIdMappedPortContext(AContext).NetData+#13#10#13#10
end;
The problem is, how to test all data pattern let say over 20,000 lists using IdMappedPortTCP with multi threaded technique?
I'm using Delphi 2007 and Indy 10.
NetData contains whatever raw data was available on the socket at the moment the OnExecute event was fired. There is no guarantee of the content of NetData on any given triggering of the event. So every time the event is triggered, you need to store that data to your own per-connection buffer somewhere, then you can parse that buffer looking for complete lines and tweaking them as needed, then update the NetData with new data as needed. Whatever data is in NetData when the event handler exits is the data that gets passed along to the target server.
BTW, HEAD http://somebug.com/ HTTP/1.1[CRLF]Host : somehost.com[CRLF]CONNECT somesitesite.com HTTP/1.0[CRLF][CRLF][CRLF][CRLF] is two HTTP commands overlapping each other. That should never happen in a real scenario. If it is, then the client that is sending those commands is faulty.

TIdMessageParts.CountParts Returns 0

I'm trying to pull multipart emails in MIME format from an IMAP server using Indy 10.5.5 in Delphi 2010. These are the lines of code that I'm having trouble with are below, where I instatiate the curMessage object, retrieve a message into it, and then call CountParts:
var
curMessage: TIdMessage;
IMAP4: TIdIMAP4;
msgIndex: Integer;
begin
...
curMessage := TIdMessage.Create(nil);
IMAP4.Retrieve(msgIndex, curMessage);
curMessage.MessageParts.CountParts;
//code that checks counts
//and
end;
I then have some code that checks the various count properties of curMessage.MessageParts (i.e. TextPartCount). However, the CountPart procedure isn't returning anything, because the Count property referenced in the procedure block is 0, even though I've verified that the message is retrieved and placed into the curMessage.
One thing I've noticed, and haven't gotten to the bottom of yet, is that IsMsgSinglePartMime is coming back as true, even though all the messages on the server have Content-Type: multipart/mixed;.
Any help would be really appreciated.
What am I missing here? I can provide more code if needed,
Without seeing the actual email data, it is difficult to say for sure exactly why the data is not where you expect it to be. But if the TIdMessage.IsMsgSinglePartMime is getting set to True then that means that either:
TIdMessage.Encoding is meMIME but TIdMessage.MIMEBoundary.Count is 0, meaning there was no MIME boundary value detected in the top-level Content-Type header. If the Content-Type is a 'multipart/...' type, a boundary is required. If it is present, it is likely malformed in a way that prevented Indy from parsing it.
TIdMessage.Encoding is mePlainText but TIdMessage.ContentTransferEncoding is either 'base64' or 'quoted-printable'.
In either case, if there is body content present then it would end up in the TIdMessage.Body property if it is textual data, otherwise it would end up in the TIdMessage.MessageParts as an attachment instead. Since TIdMessage.MessageParts.Count is 0 in your case, the data is either in TIdMessage.Body, or is got discarded.
You may want to consider upgrading to a newer Indy version. The version shipped with D2010 is pretty old, and there have been fixes/changes made to TIdIMAP4 and TIdMessage (and its internal parsers) in recent years.

Overbyte ICS HTTPS POST

I'm wanting to create a CloudFlare client in the Firemonkey framework. For those who don't know, CloudFlare serves as a CDN of sorts for anyone with a website. They have an API available, and as with many web API's, they are using JSON with a token-based system. It requires both the account email address and the account token to access the API. It runs on HTTPS, and as you can imagine, attempting to access the API via HTTP/non-SSL simply produces null results.
The application i wish to create would serve as an all-in-one management tool, intending to eliminate the need for me to use a web browser to manage my CloudFlare settings. I'm having the most basic of issues; SSL POST. See, i can submit an API request via a web browser and get a list of results (e.g. https://www.cloudflare.com/api_json.html?a=stats&z=DOMAIN&u=EMAIL&tkn=TOKEN - Personal details removed for obvious reasons), but i'm unsure how i would go about getting these same results (or any results from the API for that matter) in Firemonkey.
I've got Overbyte ICS with SSL installed, as well as the basic bundled Indy components, but i'm struggling to get started with this. I need to post a list of parameters to https://www.cloudflare.com/api_json.html via HTTPS/SSL, but i've very little idea on where to start. I've seen a few various example around SO, mostly using ICS, but i've been unable to find any specific to posting with multiple parameters, how i should format it, etc.
One example i tried was using ICS TSSLHttpCli, writing my parameters as a single string (i.e. a=stats&z=DOMAIN&u=EMAIL&tkn=TOKEN), writing that to the SendStream of TSSLHttpCli, seeking to 0,0, setting the URL (i.e. https://www.cloudflare.com/api_json.html?), and then calling the Post method. However, this gives me Connection aborted on request. This is the code i've tried (though i've replaced personal details with generic values);
var
Data : AnsiString;
RcvStrm, SndStrm : TMemoryStream;
begin
SndStrm := TMemoryStream.Create;
RcvStrm := TMemoryStream.Create;
Data := '?a=stats&z=MYDOMAIN&u=MYEMAIL&tkn=MYTOKEN';
SslHttpCli.SendStream := SndStrm;
SslHttpCli.SendStream.Write(Data[1],Length(Data));
SslHttpCli.SendStream.Seek(0,0);
Memo1.Lines.LoadFromStream(SndStrm);
ShowMessage('Waiting!');
SslHttpCli.RcvdStream := RcvStrm;
SslHttpCli.URL := 'https://www.cloudflare.com/api_json.html';
SslHttpCli.Post;
Memo1.Lines.Clear;
Memo1.Lines.LoadFromStream(RcvStrm);
Memo1.Lines.Add('.....');
RcvStrm.Free;
SndStrm.Free;
ShowMessage('Complete!');
end;
The ShowMessage procedures are simply there to provide a visual break so i can see what data is in the stream at each time. When Memo1.Lines.LoadFromStream(SndStrm); is called, i get a single question mark the contents of the Data in the memo as expected.
When i call Memo1.Lines.LoadFromStream(RcvStrm);, i expect it to add the return result from the API, and then the 5 dots underneath it. However, this does not happen, and it's apparent that the message i'm receiving is related to the issue. I'm assuming i've not set up the data correctly, but i'm simply unsure exactly how i should format it prior to attempting to post it. I've even commented out everything below Memo1.Lines.LoadFromStream(RcvStrm); to the end to see whether the Clear procedure is called on the memo, but the contents of the memo remain the same as they were when i called LoadFromStream(SndStrm). The final ShowMessage is also not called.
I initially tried using String instead of AnsiString, but this simply output the first character of Data rather than the whole string.
There could be numerous reasons why it's not working (all details for API access are correct, so it's an issue with the code), but i need someone with more experience and knowledge to point me in the right direction.
My network coding knowledge is limited, and i've only dealt with basic SQL and FTP in Delphi so far. I've still got to work with the parsed JSON once i do get past this step, but for now, can anyone assist me in this endeavor so i can get started?
I noticed you seemed to solve this with a GET request, but I noticed two immediate problems with your POST request:
as Runner Suggested, drop the '?' in your data. The '?' is only used when appending parameters to the URL in a GET request.
You never set the content type of the HTTP Request (should be application/x-www-form-urlencoded). You can do this with the following code:
SSLHttpCli.ContentTypePost := 'application/x-www-form-urlencoded';
Just a helpful thought. I checked https://www.cloudflare.com/docs/client-api.html and they mention that POST requests are accepted. It's possible the server rejects requests that have any other content type.
Just some food for thought if you ever need to contact another API via POST requests and want to use the Overbyte Components.
Hope the info is useful!
Try this;
SndStrm := TMemoryStream.Create;
RcvStrm := TMemoryStream.Create;
Data := 'a=stats&z=MYDOMAIN&u=MYEMAIL&tkn=MYTOKEN';
SndStrm.Write(Data[1], Length(Data));
SndStrm.Seek(0, 0);
SslHttpCli.SendStream := SndStrm;

Resources