I have created a FMX Windows app that connects to a web server to obtain REST data. I have been using the REST Client, Response, Request and ResponseDataAdapter and have connected that to a Client Data Set. I have then connected the Datasets to a string grid through live bindings. I have done this for 2 different string grids with no problems at all, And then I come to the very last request I want to make and I am getting some very strange behaviour. I set everything up in a data module and did an execute of the RestRequest in the IDE and got the content I expected in the RESTResponse. I then activated the RESTResponseAdapter and ClientDataset. The clientdata set was populated and I was able to add the fielddefs through the ide by just going to add fields.
I have a timer setup on the app to update the string grids etc,,, Works fine for two string grids. However on the last one all I ever get on the StringGrid is the data that I originally fetched while in the IDE. I assumed this could be due to some caching on the clientdataset so I put a memo on the form and after each request execute I posted the response content to the memo.... The bizarre thing is that I occasionally get the response the server is currently sending back (Verified by going to the webserver through Chrome) but sometimes the Response Content is the data that I originally requested when I set it up in the IDE. So I went back to the IDE and cleared the response data from the Rest Response. Tried again and get the same... I get the expected result sometimes and other times I get the response that I originally got in the IDE yesterday. So then I thought perhaps the webserver was sending it back. So have run the same REST request through the webserver and never get back the data that the restresponse is showing...
The code below fires on my timer. The top two sets of code are working fine the last one is the buggy one.
restDataModule.adapterOperators.ClearDataSet;
restDataModule.cdsOperators.Close;
restDataModule.responseOperators.Content.Empty;
restDataModule.reqOnlineOperators.ClearBody;
restDataModule.reqOnlineOperators.Execute;
restDataModule.cdsOperators.Open;
restDataModule.adapterStats.ClearDataSet;
restDataModule.cdsStats.Close;
restDataModule.responseOperatorStats.Content.Empty;
restDataModule.reqOperatorStats.ClearBody;
restDataModule.reqOperatorStats.Execute;
restDataModule.cdsStats.Open;
try
restDataModule.adapterChats.ClearDataSet;
restDataModule.cdsChats.Close;
restDataModule.responseChats.Content.Empty;
restDataModule.reqChats.ClearBody;
restDataModule.reqChats.Execute;
restDataModule.cdsChats.Open;
except on E: Exception do
// ignore
memo1.Lines.Add('Failed!')
end;
memo1.Lines.Add(restDataModule.responseChats.Content);
Any suggestions welcome.
Ok the solution was to add a Parameter to the RestClient with the following settings:-
Kind = pkHTTPHEADER
Name = Cache-Control
Value = no-cache
Simple but elusive
Related
I'm using WinHttpSendRequest/WinHttpWriteData to upload a large (54Mb) file to our server, sending it in 4Kb lumps to give user feedback. This has been working well, as far as I know, until recently. Now, when I try it, the upload goes very quickly and then the WinHttpReceiveResponse() call times-out and incomplete data is received by the server.
I'm using Win 8.1 64bit, IE11 11.0.15 (I think that WinHttp is updated with IE) but on my colleague's PC - same version of Windows, IE, the upload is much slower and the response doesn't time-out. When I try testing on various virtual machines, the problem isn't apparent. Other colleagues, however ... oh Windows!!
Just to be clear
As far as I am aware, this code used to work!
WinHttpOpen is called without the ASYNC flag set.
The HTTP verb is POST
The code in Delphi XE2
Result:= WinHttpSendRequest(RequestHandle,PWideChar(Headers),Length(Headers),WINHTTP_NO_REQUEST_DATA,0,FormBuffer.Size,Cardinal(Self));
If Result
then begin
BytesToWrite:= FormBuffer.Size;
while BytesToWrite > 0
do begin
If BytesToWrite > SizeOf(WriteBuffer)
then BufFill:= SizeOf(WriteBuffer)
else BufFill:= BytesToWrite;
FormBuffer.ReadBytes(WriteBuffer,BufFill); // FormBuffer is my object to supply data and headers
If WinHttpWriteData(RequestHandle,#WriteBuffer[0],BufFill,Written)
then Dec(BytesToWrite,Written)
else Error('WinHttpWriteData'); // Error() method calls GetLastError, assembles error message and logs it
If Assigned(OnDataWrite)
then OnDataWrite(Self,Written); // Event that notifies user
end;
FetchResponse(RequestHandle); // Calls WinHttpReceiveResponse() and then fetches data
Result:= True;
end
else GLE:= Error('WinHttpSendRequest');
This code was largely an adaptation of this code:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa384120(v=vs.85).aspx
The "WinHttp Sample code to do a PUT." at the bottom.
It's AVG ...!
Disabling AVG gives normal performance for the upload ... now it's just a matter of finding out which part(s) are getting in the way.
I'm using Delphi XE4 and i usually use Indy with IdHttp.POST to POST request to websites,
This time, whenever i try to POST the request i get Error: Your browser is not sending the correct data.
I'm very sure that I'm POSTing the right data, and i'm using the IOHandler and CookieManager.
Been dealing with this for days(literally)
Here is the code(the site in the code):
procedure TForm1.Button1Click(Sender: TObject);
var s, lge, Kf1, Kf2, Kf3, Kf4 : String;
lParam : TStringList;
begin
S := http.Get('https://www.neobux.com/m/l/');
Memo1.Lines.Add(S);
getParamLge(s,lge,'lge');
GetInput(s,Kf1,'id="Kf1"');
GetInput(s,Kf2,'id="Kf2"');
GetInput(s,Kf3,'id="Kf3"');
GetInput(s,Kf4,'id="Kf4"');
lParam := TStringList.Create;
lParam.Add('lge='+lge);
lParam.Add(Kf1+'=USERNAME');
lParam.Add(Kf2+'=PASSWORD');
lParam.Add(Kf3+'=');
lParam.Add(Kf4+'=');
lParam.Add('login=1');
memo1.Lines.Add(http.Post('https://www.neobux.com/m/l/', lParam));
end;
(the getParamLge and GetInput function, are just simple copy and pos functions to extract value from the GET respone).
I thought maybe it needed cookies so i've added this in the beginning:
Cookie.CookieCollection.Clear;
Cookie.CookieCollection.AddClientCookies('CFID=21531887; CFTOKEN=20369251; dh=20130709111845,1920x1080,{ts ''2013-07-09 06:18:58''}; __utma=90161412.436822896.1373368451.1373368451.1373368451.1; __utmb=90161412.11.10.1373368451; __utmc=90161412; __utmz=90161412.1373368451.1.1.'+'utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __asc=06ff77ad13fc32381fd1f5d6405; __auc=06ff77ad13fc32381fd1f5d6405; __atuvc=4%7C28; MS=flat');
But all in vain.
I'm very sure that I'm POSTing the right data
Since it does not work - obviously you do not (or Delphi does not - that makes no difference for server).
You should start usual debugging loop:
Observe reference working behaviour.
Observe your program behavior
Spot the difference
Eliminate the difference
Check if the program works now
If not - go to step 2.
Reference implementation would be some WWW browser working with site: Opera, Chrome, Firefox, MS IE, etc.
Observing tool would be some HTTP Sniffer like WireShark or OmniPacket or Microsoft Net Monitor or else, however this tinkers with OS work on rather deep level.
Or it can be local proxy with GUI, like Proxomitron or Membrane Monitor - but that would require special setup for both the program and the browser, to route their traffic through that local proxy.
Then you should read about HTTP, starting with shallow observation at Wikipedia and then opening related RFC documents (specifications of different part of HTTP protocol) so that you would understand what do the observed differences mean and how to fix them. For example many people use POST request when they actually should use GET request or such.
You want to debug HTTP program but for this HTTP logs, workign and borken, are required and your question lacks them. More so, most probably you can fix it your self, just bring your program's HTTP log to accordance with both RFCs theory and working browsers practice.
I have some files uploaded at a filehoster which I want to download programmatically, using Delphi. They don't require any captchas or the like, normally you simply press a button and you get the file. Let's take this as an example.
Now I thought I could simply take the URL the Download Now - Button is pointing at, use an TIdHTTP.Get request and save it with a MemoryStream / Filestream / whatever. Copying the link address leads to this site, which, when entered into my browser pops up the download prompt.
var
MemStream: TMemoryStream;
code: string; // added for solution
number: integer; // added for solution
begin
with TIdHTTP.Create(nil) do
try
HandleRedirects := true;
System.Delete(code,1,AnsiPos('var n =',code)+7); // added
number := StrToInt(AnsiLeftStr(code,AnsiPos(' ',code)-1)) + 1; // added
MemStream := TMemoryStream.Create;
try
// Get('http://www56.zippyshare.com/d/5862319/604061/bgAvgTable.png', MemStream);
Get(TIdURI.URLEncode('http://www56.zippyshare.com/d/5862319/' + IntToStr(number)
+ '/bgAvgTable.png'), MemStream); // added for solution
MemStream.SaveToFile('test.png');
finally
MemStream.Free;
end;
finally
Free;
end;
end;
However, using a checking tool I found that it contains a 302 redirect to the original site, thus when performing the GET-request I have to set HandleRedirects to avoid error messages and I get the HTML code of the original site rather than the file I had suspected.
So, I am kind of confused about how
1) I somehow get the file from my browser though the URL only contains a 302 redirect to the previous page and
2) I can achieve the same from within my code. Any chance someone of you might educate me a little there ? ;)
EDIT
Thanks to your input I could find the issue, turns out that the address I have to use gets generated using a random number, which is to be found in the original source. So posting a request to get the number first does the trick. I have edited the code accordingly.
File hosting sites make different tricks to ensure you was not hotlinking and show you advertisement and perhaps counter. There can be
simple analysis of HTTP Referrer field in the request
setting and checking session-unique cookies
having HTTP Forms with hidden one-time values, and Download button would be not the link but the form's Submit action.
generating one-time hashed URL, and encoding different parameters like your IP and your browser name into it
maybe more
Tools like USDownloader and JDownloader makes a lot of attempts to circumvent it.
While zippyshare seems to be more liberal, it still cannot afford hotlinking and should implement at least some measures of self-defense.
When analysing traffic - start with absolutely fresh browser loading zippyshare page for the 1st time in its life and check it all.
As i re-load the page few times i see that the number "604061" is different and link keep changing time and again after each reload. You probably have to load the page, parse the link, set the HTTP referer and only then download the file.
You do not show the HTTP traffic logs so it is hard to tell for sure.
The server may be checking for some trace to avoid the file to be downloaded programmatically.
It may be anything the hostmaster wants to check, from a wide range of possibilities, but the most typical check is the referrer.
When you navigate in a web browser from one page to another using an link, the browser adds the first page as a referrer to the second page in the request header.
Indy have support for you to add a referrer:
IdHTTP1.Request.Referer := 'http://www.any.other.page';
If the check fails, the server script just redirects the input to the donwload page. This is done to show advertising or to filfull other goals of the file hosting service.
I'm wanting to create a CloudFlare client in the Firemonkey framework. For those who don't know, CloudFlare serves as a CDN of sorts for anyone with a website. They have an API available, and as with many web API's, they are using JSON with a token-based system. It requires both the account email address and the account token to access the API. It runs on HTTPS, and as you can imagine, attempting to access the API via HTTP/non-SSL simply produces null results.
The application i wish to create would serve as an all-in-one management tool, intending to eliminate the need for me to use a web browser to manage my CloudFlare settings. I'm having the most basic of issues; SSL POST. See, i can submit an API request via a web browser and get a list of results (e.g. https://www.cloudflare.com/api_json.html?a=stats&z=DOMAIN&u=EMAIL&tkn=TOKEN - Personal details removed for obvious reasons), but i'm unsure how i would go about getting these same results (or any results from the API for that matter) in Firemonkey.
I've got Overbyte ICS with SSL installed, as well as the basic bundled Indy components, but i'm struggling to get started with this. I need to post a list of parameters to https://www.cloudflare.com/api_json.html via HTTPS/SSL, but i've very little idea on where to start. I've seen a few various example around SO, mostly using ICS, but i've been unable to find any specific to posting with multiple parameters, how i should format it, etc.
One example i tried was using ICS TSSLHttpCli, writing my parameters as a single string (i.e. a=stats&z=DOMAIN&u=EMAIL&tkn=TOKEN), writing that to the SendStream of TSSLHttpCli, seeking to 0,0, setting the URL (i.e. https://www.cloudflare.com/api_json.html?), and then calling the Post method. However, this gives me Connection aborted on request. This is the code i've tried (though i've replaced personal details with generic values);
var
Data : AnsiString;
RcvStrm, SndStrm : TMemoryStream;
begin
SndStrm := TMemoryStream.Create;
RcvStrm := TMemoryStream.Create;
Data := '?a=stats&z=MYDOMAIN&u=MYEMAIL&tkn=MYTOKEN';
SslHttpCli.SendStream := SndStrm;
SslHttpCli.SendStream.Write(Data[1],Length(Data));
SslHttpCli.SendStream.Seek(0,0);
Memo1.Lines.LoadFromStream(SndStrm);
ShowMessage('Waiting!');
SslHttpCli.RcvdStream := RcvStrm;
SslHttpCli.URL := 'https://www.cloudflare.com/api_json.html';
SslHttpCli.Post;
Memo1.Lines.Clear;
Memo1.Lines.LoadFromStream(RcvStrm);
Memo1.Lines.Add('.....');
RcvStrm.Free;
SndStrm.Free;
ShowMessage('Complete!');
end;
The ShowMessage procedures are simply there to provide a visual break so i can see what data is in the stream at each time. When Memo1.Lines.LoadFromStream(SndStrm); is called, i get a single question mark the contents of the Data in the memo as expected.
When i call Memo1.Lines.LoadFromStream(RcvStrm);, i expect it to add the return result from the API, and then the 5 dots underneath it. However, this does not happen, and it's apparent that the message i'm receiving is related to the issue. I'm assuming i've not set up the data correctly, but i'm simply unsure exactly how i should format it prior to attempting to post it. I've even commented out everything below Memo1.Lines.LoadFromStream(RcvStrm); to the end to see whether the Clear procedure is called on the memo, but the contents of the memo remain the same as they were when i called LoadFromStream(SndStrm). The final ShowMessage is also not called.
I initially tried using String instead of AnsiString, but this simply output the first character of Data rather than the whole string.
There could be numerous reasons why it's not working (all details for API access are correct, so it's an issue with the code), but i need someone with more experience and knowledge to point me in the right direction.
My network coding knowledge is limited, and i've only dealt with basic SQL and FTP in Delphi so far. I've still got to work with the parsed JSON once i do get past this step, but for now, can anyone assist me in this endeavor so i can get started?
I noticed you seemed to solve this with a GET request, but I noticed two immediate problems with your POST request:
as Runner Suggested, drop the '?' in your data. The '?' is only used when appending parameters to the URL in a GET request.
You never set the content type of the HTTP Request (should be application/x-www-form-urlencoded). You can do this with the following code:
SSLHttpCli.ContentTypePost := 'application/x-www-form-urlencoded';
Just a helpful thought. I checked https://www.cloudflare.com/docs/client-api.html and they mention that POST requests are accepted. It's possible the server rejects requests that have any other content type.
Just some food for thought if you ever need to contact another API via POST requests and want to use the Overbyte Components.
Hope the info is useful!
Try this;
SndStrm := TMemoryStream.Create;
RcvStrm := TMemoryStream.Create;
Data := 'a=stats&z=MYDOMAIN&u=MYEMAIL&tkn=MYTOKEN';
SndStrm.Write(Data[1], Length(Data));
SndStrm.Seek(0, 0);
SslHttpCli.SendStream := SndStrm;
I get an error whenever I try do a request to a SOAP service:
Unmarshalling Error: unexpected element (uri:"http://www.domain.com/ws/servicename/", local:"dummyArg"). Expected elements are <{}dummyArg>
The method that I'm calling has is defined as:
function GetTxServer(UseWSDL: Boolean; Addr: string; HTTPRIO: THTTPRIO): TxServer;
I have little experience with SOAP, and I couldn't find any useful information on this. Feel free to ask any question that might speed up the process in finding the issue.
I believe that the way that I am calling the function is not the correct way!
I'm using Delphi 2010, and I've called the method like so:
Response := GetTxServer.requestIVULoto(cm);
Use SoapUI (the free version is fine) to consume the WSDL and make sure that you can properly send a request to the server and get a response that makes sense. Then make a "mock" service in SoapUI, to act as the server. Send your Delphi requests to the mockservice (typically done by setting your endpoint to http://localhost:8089 or some such) so that you can inspect the XML that you're sending out. Now you can experiment and determine whether the problem is due to sending out bad requests, the server returning bad/unexpected results, trouble interpreting good results, etc..
Aside from that, I'd guess that you're failing to allocate or populate "cm" correctly. I assume that's your request object.
Also... big tip here....
Use the RIO_BeforeExecute event to debug this. At that point, the SOAPRequest is a string that you can inspect or dump to a file. So you can see what you're sending, without having to use SoapUI, Fiddler2, Wireshark, etc..