We're building a Delphi REST server that serves up rather large chunks of data (1.5MB per request, of which there are many) to a native Android application. All works fine, except the data sizes in this case will be problematic, causing long transfer times in our environment (limited mobile data rates). I've tried adding the ZLibCompression filter on the DSHTTPWebDispatcher, but the response only comes back again as uncompressed text/html.
Is there any way to force the server to use the filter added as an event before the dispatch?
The server is built using Delphi XE3.
I've managed to figure out where to add the compression and relevant header changes in the DataSnap project.
The key here is the TWebModule class. If one uses the wizard to create a new project, a default implementation of the TWebModule class is constructed, with event properties for BeforeDispatch, AfterDispatch etc.
The naming here refers to the action of dispatching the incoming request to where it will be handled. So, BeforeDispatch happens when the request arrives, some processing happens on the server and AfterDispatch triggers just before the response is sent back to the caller.
AfterDispatch is therefore the correct event to use if one wants to modify the constructed response after the fact. This can include changes to both the content and headers.
On the AfterDispatch event:
procedure TWebModule1.WebModuleAfterDispatch(
Sender: TObject;
Request: TWebRequest;
Response: TWebResponse;
var Handled: Boolean);
var
srcbuf, destbuf : TBytes;
str : string;
begin
str := Response.Content;
//prepare byte array
srcbuf := BytesOf(str);
//compress to buff (System.ZLib)
ZCompress(srcbuf, destbuf, zcMax);
//prepare responsestream and set content encoding and type
Response.Content := '';
Response.ContentStream := TMemoryStream.Create;
Response.ContentEncoding := 'deflate';
Response.ContentType := 'application/json';
//current browser implementations incorrectly handles the first 2 bytes
//of a ZLib compressed stream, remove them
Response.ContentStream.Write(#(destbuf[2]),length(destbuf)-2);
Response.ContentLength := (length(destbuf))-2;
end;
Not very fancy, one could enable/disable compression depending on the content sent back, but for our implementation we kept it simple.
This works 100% with Fiddler and browsers that can handle deflate.
Related
I am using datasnap technology where I got a client a server application.
The Server gets the data from database and puts into stream which is then fetched at client side.
I keep getting a "Missing Data Provider or Data Packet" error on the client side code.
Server side code:
function GetFiles(ClientID: integer): TStream;
var
CDS: TClientDataSet;
begin
try
Result := TMemoryStream.Create;
CDS := TClientDataSet.Create(nil);
with adsFiles do // ads is adodataset and dspFiles is the dataset provider which has its dataset property set to adsFiles
begin
Close;
Parameters.ParamByName('ClientID').Value := ClientID;
Open;
CDS.Data := dspFiles.Data;
CDS.Open;
CDS.SaveToStream(Result);
Result.Position := 0;
end;
finally
CDS.Free;
end;
Client Side Code:
procedure ExportData;
var
StreamData: TStream;
begin
StreamData := SvrMethodClass.GetFiles(AClientID);
StreamData.Seek(0,soFromBeginning);
StreamData.Position:= 0;
cdsClientFiles.LoadFromStream(StreamData); // getting the error message "Missing Data Provider or Data Packet"
cdsClientFiles.Open;
end;
Client side I have dropped a clientdataset component and trying to load the data into this dataset where the issue is raised any help pointing me where I am going wrong would be really appreciated. Thanks
similar question: Streaming TClientDataSet using Datasnap in Delphi XE6
checked the solution but it did not work out
I spent a lot of time investigating the problems with returning data as a stream from a datasnap server - see e.g. Can't retrieve TStreams bigger than around 260.000 bytes from a Datasnap Server, but was unable to come up with a satisfactory solution for stream data.
However, experiments revealed that a similar problem does not occur with string data (I tested strings up to several hundred megabytes). So, I would suggest you try sending your binary/blob data from the Datasnap server as a Base64-encoded string.
Of course, if a string representation of your ADO or CDS data is adequate for what you want, one way to achieve that would be to do an adsFile.SaveToFile(..., pfXML) or CDS.SaveToFile(... dfXML) in the server method, then return the resulting file as a string.
When you return an image using a file stream object in a Delphi rest server, it will not display in a browser. Here is an example method that returns an image:
function TServerClass.Image: TFileStream;
begin
Result := TFileStream.Create('pathtofile\image.png', fmOpenRead or fmShareDenyNone);
end;
The problem is that Delphi REST server always sets the content type to text/html. This confuses the browser when you send other types of content. It is a bug, since most responses are json, which means the most sensible default content type should be application/json.
Fortunately there is a way to override the content type from within the server method.
You need to add Data.DBXPlatform to the uses list of your implementation.
This unit contains the function GetInvocationMetadata, which gives access to the response that is being build. It returns a TDSInvocationMetadata object which among varius other useful properties has the ResponseContentType property.
Setting this property overrides the Content-Type header that the method returns in the http response.
The given example becomes:
function TServerClass.Image: TFileStream;
begin
Result := TFileStream.Create('pathtofile\image.png', fmOpenRead or fmShareDenyNone);
GetInvocationMetadata.ResponseContentType := 'image/png';
end;
Now the result image will be displayed properly in the browser.
I've found this problem too trying to download different file types (png, pdf, xlsx, docx, etc...) from DataSnap REST server (Delphi XE3) to a JavaScript web client.
Some browsers (es.: FireFox) will take correct action anyway, but not all. Internet Explorer doesn't recognize the proper action for the downloaded file without correct content-type.
The #Anders solution initially seems to work for me because I was working with PDF and Firefox.
But when I've tested on IE (and others) and with different extensions, the files where not recognised. Using FireBug I've seen that Content-Type was always "text/html" and not the assigned one using
GetInvocationMetadata.ResponseContentType := '...my assigned content type ...';
The workaround found working for me is:
In ServerMethodsUnit
var
ContentTypeHeaderToUse: string; // Global variable
TServerMethods1.GetFile(params: JSON):TStream;
begin
.... processing ....
ContentTypeHeaderToUse := '...' (assign correct content type).
end;
In WebModuleUnit
procedure TWebModule1.WebModuleAfterDispatch(Sender: TObject;
Request: TWebRequest; Response: TWebResponse; var Handled: Boolean);
begin
if ContentTypeHeaderToUse<>'' then begin
Response.ContentType := ContentTypeHeaderToUse;
ContentTypeHeaderToUse := ''; // Reset global variable
end;
end;
I used a similar solution for assigning Content-Disposition too. This is a useful header key in order to set file name to download and attachment/inline mode.
With this one the code is:
procedure TWebModule1.WebModuleAfterDispatch(Sender: TObject;
Request: TWebRequest; Response: TWebResponse; var Handled: Boolean);
begin
if ContentDispositionHeaderToUse<>'' then begin
Response.SetCustomHeader('content-disposition',ContentDispositionHeaderToUse);
ContentDispositionHeaderToUse := '';
end;
if ContentTypeHeaderToUse<>'' then begin
Response.ContentType := ContentTypeHeaderToUse;
ContentTypeHeaderToUse := '';
end;
end;
Assign ContentDispositionHeaderToUse into the server methods implementation.
EDIT
This workaround doesn't work in ISAPI DLL on IIS with data compression enabled!
With no data compressione (local debuggin IIS) the response header is:
Connection close
Content-Disposition inline; filename="Privacy-0.rtf.pdf"
Content-Length 150205
Content-Type application/pdf; charset=ISO-8859-1
Pragma dssession=28177.371935.39223,dssessionexpires=1200000
but with production enabled IIS the response comes with:
Content-Encoding gzip
Content-Length 11663
Content-Type text/html
Date Thu, 11 Sep 2014 21:56:43 GMT
Pragma dssession=682384.52215.879906,dssessionexpires=1200000
Server Microsoft-IIS/7.5
Vary Accept-Encoding
X-Powered-By ASP.NET
Content-disposition and content-type assigned in DataSnap code are not surfaced.
I want to load a url directly into a string without any data stream,what is the best way, internet open url works but it seems not clear.
I don't want to use any component for reading some short messages
Delphi 6 and later ship with Indy, which has a TIdHTTP client component, eg:
uses
..., IdHTTP;
var
Reply: String;
begin
Reply := IdHTTP1.Get('http://test.com/postaccepter?=msg1=3444&msg2=test');
...
end;
Or:
uses
..., IdHTTP;
var
Reply: TStream;
begin
Reply := TMemoryStream.Create;
try
IdHTTP1.Get('http://test.com/postaccepter?=msg1=3444&msg2=test', Reply);
Reply.Position := 0;
...
finally
Reply.Free;
end;
end;
Depending on your needs.
You can use Synapse, a very light weight library that has a simple function call to get just what your asking for:
uses
classes, httpsend;
var
Response : TStringlist;
begin
if HttpGetText(URL,Response) then
DoSomethingWithResponse(Response.Text);
end;
I would suggest getting the latest copy from SVN, which is more current and contains support for the latest versions of Delphi. There are also simple functions for posting form data, or retrieving binary resources. These are implemented as simple functions and are a great template if you want to do something extra, or that is not directly supported.
You can use our SynCrtSock unit, which is even lighter than Synapse.
See http://synopse.info/fossil/finfo?name=SynCrtSock.pas
It is a self-contained unit (only one dependency with the WinSock unit), and it works from Delphi 6 up to Delphi XE.
You have these two functions available to get your data in one line of code:
/// retrieve the content of a web page, using the HTTP/1.1 protocol and GET method
function HttpGet(const server, port: AnsiString; const url: TSockData): TSockData;
/// send some data to a remote web server, using the HTTP/1.1 protocol and POST method
function HttpPost(const server, port: AnsiString; const url, Data, DataType: TSockData): boolean;
Or you can use textfile-based commands (like readln or writeln) to receive or save data.
TSockData is just a wrapper of RawByteString (under Delphi 2009/2010/XE) or AnsiString (up to Delphi 2007).
If you need also to write a server, you have dedicates classes at hand, resulting in fast processing and low resource consummation (it uses a Thread pool, and is implemented over I/O Completion Ports).
If I'm already using XML in an application (and the MSXML2_TLB), I generally use IXmlHttpRequest to perform http operations. If you open and send the request, you can either use the response data as XML DOM using the ResponseXML, as text using ResponseText or as a data-stream using ResponseStream, see here for an example how to use this in Delphi: http://yoy.be/item.asp?i142
i am trying to send a picture from 'C:\picture.bmp' to 'c:\temp\picture.bmp' using server and client socket
clients onconnect event handler is as follow:
procedure TForm2.ClientSocket1Connect(Sender: TObject;
Socket: TCustomWinSocket);
var
fs : tfilestream;
begin
fs := TFileStream.create('C:\picture.bmp', fmOpenRead);//picture allready exists
socket.SendStream(fs);
fs.free;
end;
and servers onclientread as :
procedure TForm2.ServerSocket1ClientRead(Sender: TObject;
Socket: TCustomWinSocket);
var
fmm : tfilestream;
iLen: Integer;
Bfr: Pointer;
begin
iLen := Socket.ReceiveLength;
GetMem(Bfr, iLen);
fmm := TFileStream.Create('c:\temp\picture.bmp', fmCreate or
fmShareDenyWrite);
try
Socket.ReceiveBuf(Bfr^, iLen);
fmm.Write(Bfr^, iLen);
finally
FreeMem(Bfr);
fmm.Free;
end;
end;
picture is recieved/created but is either corrupt on was never recieved i.e created because of tfilestream.create method?
please help!what am i doing wrong?
Despite its name, SendStream() is NOT guaranteed to send the entire stream (especially if you are using a non-blocking socket). Its return value returns how many bytes are actually sent. If less than the full size of the stream are sent in one call, you have to call SendStream() again, potentially many times, to finish sending the entire stream (the same problems exists with SendText() as well).
On the other side, ReceiveLength() only reports how many bytes are available on the socket AT THAT MOMENT. That is likely to be less than the full stream being sent (likewise, ReceiveText() may not receive a full sent string either because it uses ReceiveLength() internally).
The best way to send a stream (or any arbitrary data in general) is to send the data's size first, then send the actual data afterwards. Keep calling SendBuf/Stream/Text() until that size is reached (if -1 is returned by a non-blocking socket without raising an exception, you have to wait for the socket's OnWrite event to trigger before the socket can accept more data again). On the receiving end, read the size first, then keep reading until the specified size is reached. You may have to read in multiple triggering of the OnRead event before you get all of the data.
Go to http://www.deja.com and http://forums.embarcadero.com to search the Borland/CodeGear/Embarcadero newsgroup/forum archives. I have posted example code many times before.
I don't know what's wrong, but I'd try troubleshooting a simpler problem. i.e. can you even transfer somethign simple? See if you can transfer c:\hello.txt containing just "Hello" and have it arrive in the right order. It should be easier to examine the stream and resulting file, to see if/where things are getting garbled. If you don't receive "Hello" on the server, then you know it's got nothing to do with the size or complexity of the data.
I am using HttpCli component form ICS to POST a request. I use an example that comes with the component. It says:
procedure TForm4.Button2Click(Sender: TObject);
var
Data : String;
begin
Data:='status=no';
HttpCli1.SendStream := TMemoryStream.Create;
HttpCli1.SendStream.Write(Data[1], Length(Data));
HttpCli1.SendStream.Seek(0, 0);
HttpCli1.RcvdStream := TMemoryStream.Create;
HttpCli1.URL := Trim('http://server/something');
HttpCli1.PostAsync;
end;
But it fact, it sends not
status=no
but
s.t.a.t.u
I can't understand, where is the problem. Maybe someone can show an example, how to send POST request with the help of HttpCli component?
PS I can't use Indy =)
I suppose you're using Delphi 2009 or later, where the string type holds two-byte-per-character Unicode data. The Length function gives the number of characters, not the number of bytes, so when you put your string into the memory stream, you only copy half the bytes from the string. Even if you'd copied all of them, though, you'd still have a bunch of extra data in the stream since each character has two bytes and the server probably only expects to get one.
Use a different string type, such as AnsiString or UTF8String.