I want to load a url directly into a string without any data stream,what is the best way, internet open url works but it seems not clear.
I don't want to use any component for reading some short messages
Delphi 6 and later ship with Indy, which has a TIdHTTP client component, eg:
uses
..., IdHTTP;
var
Reply: String;
begin
Reply := IdHTTP1.Get('http://test.com/postaccepter?=msg1=3444&msg2=test');
...
end;
Or:
uses
..., IdHTTP;
var
Reply: TStream;
begin
Reply := TMemoryStream.Create;
try
IdHTTP1.Get('http://test.com/postaccepter?=msg1=3444&msg2=test', Reply);
Reply.Position := 0;
...
finally
Reply.Free;
end;
end;
Depending on your needs.
You can use Synapse, a very light weight library that has a simple function call to get just what your asking for:
uses
classes, httpsend;
var
Response : TStringlist;
begin
if HttpGetText(URL,Response) then
DoSomethingWithResponse(Response.Text);
end;
I would suggest getting the latest copy from SVN, which is more current and contains support for the latest versions of Delphi. There are also simple functions for posting form data, or retrieving binary resources. These are implemented as simple functions and are a great template if you want to do something extra, or that is not directly supported.
You can use our SynCrtSock unit, which is even lighter than Synapse.
See http://synopse.info/fossil/finfo?name=SynCrtSock.pas
It is a self-contained unit (only one dependency with the WinSock unit), and it works from Delphi 6 up to Delphi XE.
You have these two functions available to get your data in one line of code:
/// retrieve the content of a web page, using the HTTP/1.1 protocol and GET method
function HttpGet(const server, port: AnsiString; const url: TSockData): TSockData;
/// send some data to a remote web server, using the HTTP/1.1 protocol and POST method
function HttpPost(const server, port: AnsiString; const url, Data, DataType: TSockData): boolean;
Or you can use textfile-based commands (like readln or writeln) to receive or save data.
TSockData is just a wrapper of RawByteString (under Delphi 2009/2010/XE) or AnsiString (up to Delphi 2007).
If you need also to write a server, you have dedicates classes at hand, resulting in fast processing and low resource consummation (it uses a Thread pool, and is implemented over I/O Completion Ports).
If I'm already using XML in an application (and the MSXML2_TLB), I generally use IXmlHttpRequest to perform http operations. If you open and send the request, you can either use the response data as XML DOM using the ResponseXML, as text using ResponseText or as a data-stream using ResponseStream, see here for an example how to use this in Delphi: http://yoy.be/item.asp?i142
Related
I have a JSON-RPC service which for one of the requests returns a continuous stream of JSON objects.
I.e. :
{id:'1'}
{id:'2'}
//30 minutes of no data
{id:'3'}
//...
Of course, there's no Content-Length because the stream is endless.
I'm using custom TStream descendant to receive and parse the data. But internally TIdHttp buffers the data and does not pass it to me until RecvBufferSize bytes are received.
This results in:
{id:'1'} //received
{id:'2'} //buffered by Indy but not received
//30 minutes of no data
{id:'3'} //this is where Indy commits {id:'2'} to me
Obviously this won't do because the message which mattered 30 minutes ago should have been delivered 30 minutes ago.
I'd like Indy to do just what sockets do: read up to RecvBufferSize or less if there's data available and return immediately.
I've found this discussion from 2005 where some poor soul tried to explain the problem to Indy developers but they didn't understand him. (Read it; it's a sad sight)
Anyway, he worked around this by writing custom IOHandler descendant, but that was back in 2005, maybe there are some ready solutions today?
Sounds to me like a WebSocket task, since your connection is not plain HTTP question/answer oriented any more, but a stream of content.
See WebSocket server implementations for Delphi for some code.
There is at least one based on Indy, from the author of AsmProfiler.
AFAIK there are two kind of stream in websockets: binary and text. I suspect your JSON stream is some text content, from the websocket point of view.
Another option is to use long-pooling or some older protocols, which are more rooter-friendly - when the connection switch to websockets mode, it is no standard HTTP any more, so some "sensible" packet-inspection tools (on a corporate network) may identify it as a security attack (e.g. DoS), so may stop the connection.
You do not need to write a IOHandler descendant, it is already possible with the TIdTCPClient class. It exposes a TIdIOHandler object, which has methods to read from the socket. These ReadXXX methods block until the requested data has been read or a timeout occurs. As long as the connection exists, ReadXXX can be executed in a loop and whenever it receives a new JSON object, pass it to the application logic.
Your example looks like all JSON objects only have one line. JSON objects however could be multi-lined, in this case the client code needs to know how they are separated.
Update: in a similar Stackoverflow question (for .Net) for a 'streaming' HTTP JSON web service, the most upvoted solution used a lower-level TCP client instead of a HTTP client: Reading data from an open HTTP stream
While using TCP stream was an option, in the end I went with original solution of writing custom TIdIOHandlerStack descendant.
The motivation was that with TIdHTTP I know what doesn't work and only need to fix that, while switching to lower level TCP means new problems can arise.
Here's the code that I'm using, and I'm going to discuss the key points here.
New TIdStreamIoHandler has to inherit from TIdIOHandlerStack.
Two functions need to be rewritten: ReadBytes and ReadStream:
function TryReadBytes(var VBuffer: TIdBytes; AByteCount: Integer;
AAppend: Boolean = True): integer; virtual;
procedure ReadStream(AStream: TStream; AByteCount: TIdStreamSize = -1;
AReadUntilDisconnect: Boolean = False); override;
Both are modified Indy functions which can be found in IdIOHandler.TIdIOHandler. In ReadBytes the while clause has to be replaced with a singe ReadFromSource() request, so that TryReadBytes returns after reading up to AByteCount bytes in one go.
Based on this, ReadStream has to handle all combinations of AByteCount (>0, <0) and ReadUntilDisconnect (true, false) to cyclically read and then write to stream chunks of data arriving from the socket.
Note that ReadStream need not terminate prematurely even in this stream version if only part of the requested data is available in the socket. It just has to write that part to the stream instantly instead of caching it in FInputBuffer, then block and wait for the next part of data.
There is actually a length data right before the content of packet which transferred in chunked encoding transfer mode. Using this length data, IOhandler of idhttp read one packet by one packet to stream. The minimum meaningful unit is a packet, So there should be no need to read characters one by one from a packet and then no need to change the functions of IOHandler. The only problem is idhttp wouldn't stop an turn the stream data to next step because of the endless of stream data: there is no ending packet. So the solution is using idhttp onwork event to trigger a reading from stream and setting the stream position to zero in order to avoid overflow .like this:
//add a event handler to idhttp
IdHTTP.OnWork := IdHTTPWork;
procedure TRatesStreamWorker.IdHTTPWork(ASender: TObject; AWorkMode: TWorkMode; AWorkCount: Int64);
begin
.....
ResponseStringStream.Position :=0;
s:=ResponseStringStream.ReadString(ResponseStringStream.Size) ;//this is the packet conten
ResponseStringStream.Clear;
...
end;
procedure TForm1.ButtonGetStreamPricesClick(Sender: TObject);
var
begin
.....
source := RatesWorker.RatesURL+'EUR_USD';
RatesWorker.IdHTTP.Get(source,RatesWorker.ResponseStringStream);
end;
Yet use a custom write() function of Tstream may be a better solution for this kind of requirement.
We're building a Delphi REST server that serves up rather large chunks of data (1.5MB per request, of which there are many) to a native Android application. All works fine, except the data sizes in this case will be problematic, causing long transfer times in our environment (limited mobile data rates). I've tried adding the ZLibCompression filter on the DSHTTPWebDispatcher, but the response only comes back again as uncompressed text/html.
Is there any way to force the server to use the filter added as an event before the dispatch?
The server is built using Delphi XE3.
I've managed to figure out where to add the compression and relevant header changes in the DataSnap project.
The key here is the TWebModule class. If one uses the wizard to create a new project, a default implementation of the TWebModule class is constructed, with event properties for BeforeDispatch, AfterDispatch etc.
The naming here refers to the action of dispatching the incoming request to where it will be handled. So, BeforeDispatch happens when the request arrives, some processing happens on the server and AfterDispatch triggers just before the response is sent back to the caller.
AfterDispatch is therefore the correct event to use if one wants to modify the constructed response after the fact. This can include changes to both the content and headers.
On the AfterDispatch event:
procedure TWebModule1.WebModuleAfterDispatch(
Sender: TObject;
Request: TWebRequest;
Response: TWebResponse;
var Handled: Boolean);
var
srcbuf, destbuf : TBytes;
str : string;
begin
str := Response.Content;
//prepare byte array
srcbuf := BytesOf(str);
//compress to buff (System.ZLib)
ZCompress(srcbuf, destbuf, zcMax);
//prepare responsestream and set content encoding and type
Response.Content := '';
Response.ContentStream := TMemoryStream.Create;
Response.ContentEncoding := 'deflate';
Response.ContentType := 'application/json';
//current browser implementations incorrectly handles the first 2 bytes
//of a ZLib compressed stream, remove them
Response.ContentStream.Write(#(destbuf[2]),length(destbuf)-2);
Response.ContentLength := (length(destbuf))-2;
end;
Not very fancy, one could enable/disable compression depending on the content sent back, but for our implementation we kept it simple.
This works 100% with Fiddler and browsers that can handle deflate.
I am implementing a REST server API in Delphi XE3 (first time using Delphi in about a decade so am a bit rusty). Currently it is using Indy server for debug purposes, but eventually it will be an ISAPI dll.
Now I have implemented a number of TDSServerClass classes and want to access the request header within the class methods. So for example when the user requests mysite.com/datasnap/rest/foo/bar I want to be able to read the header within the foo class method called bar. Is this possible?
If not, is it possible to create a global filter of incoming requests before they get to the REST class method? I need to check the API key and user authentication on incoming requests and not sure the best way to implement. Thanks.
I don't know if anything changed in XE3, but in XE2 you can do the following:
uses
Web.HTTPApp,
Datasnap.DSHTTPWebBroker;
function TServerMethods1.EchoString(Value: string): string;
var
Module: TWebModule;
begin
Module := GetDataSnapWebModule;
Result := Module.Request.RemoteIP + ': ' + Value;
end;
I've written a DataSnap server method that returns a TStream object to transfer a file. The client application calls the method and reads the stream to download the file. The server method is very simple :
function TServerMethods.DownloadFile(sFilePath: string): TStream;
var
strFileStream: TFileStream;
begin
strFileStream := TFileStream.Create(sFilePath, fmOpenRead);
Result := strFileStream;
end;
It works fine downloading many file types (PDF, GIF, BMP, ZIP, EXE) but it doesn't work when downloading JPG files. On the client side the stream object returned from the method call is always 0 in size with JPGs. I can successfully stream JPG files locally on my PC, so it must be something to do with DataSnap. I've done some research which suggests DataSnap converts the stream to JSON behind the scenes and there could be a problem with this when it comes to JPG files - can anybody confirm this? On the client side I'm using the TDSRESTConnection to call the server method. I realise I could ZIP the JPG files before streaming, but would rather not have to do this.
Thought I'd update the thread on my attempts to resolve this. I never found a way to transfer a JPEG file over DataSnap using TStream, but have done it by converting the stream to a TJSONArray and passing this back instead. So my server method now looks as follows:
function TServerMethods.DownloadJPEGFile(sFilePath: string): TJSONArray;
var
strFileStream: TFileStream;
begin
strFileStream := TFileStream.Create(sFilePath, fmOpenRead);
Result := TDBXJSONTools.StreamToJSON(strFileStream, 0, strFileStream.Size);
end;
then at the client end I convert back to a TStream with:
strFileStream := TDBXJSONTools.JSONToStream(JSONArray);
I have created this as a new server method call purely for downloading JPEGs, as I've found transferring the files using TJSONArray instead of TStream is as much as 4 times slower, so I use my original method for all other file types.
Just as an update - after further research I've found this is related to the system locale in use on the PC. I'm using 'English (United Kingdom)' but if I change this to for example 'Japan (Japanese)' then the errors disappear and the file transfer works fine. I've logged this as a QC report with Embarcadero.
Embarcadero have now come back with a fix to this problem (which also affects .DOC files) :
1.Copy '...\RAD Studio\9.0\source\data\datasnap\Datasnap.DSClientRest.pas' to your DataSnap Client project folder
2.Add the .pas file to the project
3.Modify Line#1288 as below
// LResponseJSON := TJSONObject.ParseJSONValue(BytesOf(LResponseText.StringValue), 0);
LResponseJSON := TJSONObject.ParseJSONValue(BytesOf(UTF8String(LResponseText.StringValue)), 0);
4.Rebuild DataSnap REST Client project
5.Run it with REST Server
This fixes the problem.
Add this line to your DownloadFile method:
GetInvocationMetadata.ResponseContentType := 'image/jpeg';
I tried to employ Indy 10.5.5 (shipped with Delphi 2010) for:
connecting to telnet server
performing username/password authentication (gaining access to the command shell)
executing a command with returning resulting data back to application
and had no success, additionally i'm completely lost in spaghetti logic of Indy's internals and now have no idea why it didnt work or how i supposed to send strings to the server and grab the results. Need some sample code to study.
Formal form of the question: Where can i get 3-rd party contributed demo covering TIdTelnet component? (indyproject.org demos webpage do not have one)
The main problem with Telnet is that it DOES NOT utilize a command/response model like most other Internet protocols do. Either party can send data at any time, and each direction of data is independant from the other direction. This is reflected in TIdTelnet by the fact that it runs an internal reading thread to receive data. Because of this, you cannot simply connect, send a command, and wait for a response in a single block of code like you can with other Indy components. You have to write the command, then wait for the OnDataAvailable event to fire, and then parse the data to determine what it actually is (and be prepared to handle situations where partial data may be received, since that is just how TCP/IP works).
If you are connecting to a server that actually implements a command/response model, then you are better off using TIdTCPClient directly instead of TIdTelnet (and then implement any Telnet sequence decoding manually if the server really is using Telnet, which is rare nowadays but not impossible). For Indy 11, we might refactor TIdTelnet's logic to support a non-threaded version, but that is undecided yet.
done with indy.
no comments.. just som old code :-)
telnet don't like the send string kommand.. use sendch.
telnetdude.Host := 1.1.1.1;
try
telnetdude.connect;
except
on E: Exception do begin
E.CleanupInstance;
end; {except}
if telnetdude.Connected then begin
for i := 1 to length(StringToSend) do telnetdude.sendch(StringToSend[i]);
telnetdude.sendch(#13);
end;
end; {while}
end; {if}
if telnetdude.Connected then telnetdude.Disconnect;
end;
I hope this helps anyone looking for answers to a similar question.
Firstly, It would seem the typical command/response model (as mentioned above, does indeed NOT apply).
So I just got it working for some very simple application (rebooting my router).
Specific additions to above code from Johnny Lanewood (and perhaps some clarification)
a) You have to send #13 to confirm the command
b) I got "hangs" on every command I sent / response I requested UNTIL I enabled ThreadedEvent. (this was my big issue)
c) the OnDataAvailable event tells you when new data is available from the Telnet Server - however there are no guarantees as to what this data is - i.e. it's pretty what you get in the command line / what ever is appended to the previous responses. But is is NOT a specific response line to your command - it's whatever the telnet server returns (could be welcome info, ASCII drawings etc etc.)
Given (c) above, one would rather check the OnDataAvailable event and parse the data (knowing what you'd expect). When the output stops (i.e. you need build a mechanism for this), you can parse the data and determine whether the server is ready for something new from the client. For the purpose of my code below, I set a read timemout and I just used Sleep(2000) - ignorantly expecting no errors and that the server would be ready after the sleep for the next command.
My biggest stumbling block was ThreadedEvent := True (see above in b)
Thus, my working solution (for specific application, and possibly horrible to some).
lIDTelnet := TIdTelnet.Create(nil);
try
lIdTelnet.ReadTimeout := 30000;
lIDTelnet.OnDataAvailable := TDummy.Response;
lIDTelnet.OnStatus := TDummy.Status;
lIdTelnet.ThreadedEvent := True;
try
lIDTelnet.Connect('192.168.0.1', 23);
if not lIDTelnet.Connected then
Raise Exception.Create('192.168.0.1 TELNET Connection Failed');
Sleep(2000);
lIdtelnet.SendString(cst_user + #13);
Sleep(2000);
lIdtelnet.SendString(cst_pass + #13);
Sleep(2000);
lIdtelnet.SendString(cst_reboot + #13);
Sleep(2000);
if lIDTelnet.Connected then
lIDTelnet.Disconnect;
except
//Do some handling
end;
finally
FreeAndNil(lIdTelnet);
end;
and then
class procedure TDummy.Response(Sender: TIdTelnet; const Buffer: TIdBytes);
begin
Write(TDummy.ByteToString(Buffer));
end;
class function TDummy.ByteToString(
const aBytes: TIdBytes): String;
var
i : integer;
begin
result := '';
for i := 0 to Length(aBytes) -1 do
begin
result := result + Char(aBytes[i]);
end;
end;