I'm using WinHttpSendRequest/WinHttpWriteData to upload a large (54Mb) file to our server, sending it in 4Kb lumps to give user feedback. This has been working well, as far as I know, until recently. Now, when I try it, the upload goes very quickly and then the WinHttpReceiveResponse() call times-out and incomplete data is received by the server.
I'm using Win 8.1 64bit, IE11 11.0.15 (I think that WinHttp is updated with IE) but on my colleague's PC - same version of Windows, IE, the upload is much slower and the response doesn't time-out. When I try testing on various virtual machines, the problem isn't apparent. Other colleagues, however ... oh Windows!!
Just to be clear
As far as I am aware, this code used to work!
WinHttpOpen is called without the ASYNC flag set.
The HTTP verb is POST
The code in Delphi XE2
Result:= WinHttpSendRequest(RequestHandle,PWideChar(Headers),Length(Headers),WINHTTP_NO_REQUEST_DATA,0,FormBuffer.Size,Cardinal(Self));
If Result
then begin
BytesToWrite:= FormBuffer.Size;
while BytesToWrite > 0
do begin
If BytesToWrite > SizeOf(WriteBuffer)
then BufFill:= SizeOf(WriteBuffer)
else BufFill:= BytesToWrite;
FormBuffer.ReadBytes(WriteBuffer,BufFill); // FormBuffer is my object to supply data and headers
If WinHttpWriteData(RequestHandle,#WriteBuffer[0],BufFill,Written)
then Dec(BytesToWrite,Written)
else Error('WinHttpWriteData'); // Error() method calls GetLastError, assembles error message and logs it
If Assigned(OnDataWrite)
then OnDataWrite(Self,Written); // Event that notifies user
end;
FetchResponse(RequestHandle); // Calls WinHttpReceiveResponse() and then fetches data
Result:= True;
end
else GLE:= Error('WinHttpSendRequest');
This code was largely an adaptation of this code:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa384120(v=vs.85).aspx
The "WinHttp Sample code to do a PUT." at the bottom.
It's AVG ...!
Disabling AVG gives normal performance for the upload ... now it's just a matter of finding out which part(s) are getting in the way.
Related
I have about 30 unique sites which I login to and download a few files from each one of them. Sometimes I have about 50 sites, which the 20 extra are the same as previous ones but with different login credentials.
If I run the download process for any of them with all the other sites disabled, all of them work great. But if I try to download them one after another, I usually get about 5-10 errors from 40 sites!
Exceptions that occur are usually socket error, connection closed gracefully or unknown error occured!
Right now, I create a class for each site, and in each class I create a TIdHTTP or TIdFTP object (very few are FTP and I don't have a problem with any of them). When the download from one class is finished, I destroy the class and destroy the TIdHTTP, and start the download process for the next class (or site), so the downloads are not parallel. But I do create about 40 TIdHTTP and a few TIdFTP objects at the very beginning, and I start the download process one after another.
Is my approach correct? Or should I use only one TIdHTTP object in all classes? But if I have to do so, how can I refresh or reset it? Sometimes I have to login to a single site multiple times with different credentials.
I should also mention that one approach I came up with that did help a little bit (maybe solved one or two errors!) was this:
// mMaxTryCount is 4 and mSleepInterval is 1000
for I := 1 to mMaxTryCount do
begin
if not(isSuccess) then
begin
if (i = mMaxTryCount) then
begin
responseCode.Clear;
idHttp.Post(URL, requestList, responseCode);
end
else
try
responseCode.Clear;
idHttp.Post(URL, requestList, responseCode);
isSuccess := true;
except
Sleep(mSleepInterval * (i + (i - 1)));
end;
end;
end;
I'm using Delphi XE4 and i usually use Indy with IdHttp.POST to POST request to websites,
This time, whenever i try to POST the request i get Error: Your browser is not sending the correct data.
I'm very sure that I'm POSTing the right data, and i'm using the IOHandler and CookieManager.
Been dealing with this for days(literally)
Here is the code(the site in the code):
procedure TForm1.Button1Click(Sender: TObject);
var s, lge, Kf1, Kf2, Kf3, Kf4 : String;
lParam : TStringList;
begin
S := http.Get('https://www.neobux.com/m/l/');
Memo1.Lines.Add(S);
getParamLge(s,lge,'lge');
GetInput(s,Kf1,'id="Kf1"');
GetInput(s,Kf2,'id="Kf2"');
GetInput(s,Kf3,'id="Kf3"');
GetInput(s,Kf4,'id="Kf4"');
lParam := TStringList.Create;
lParam.Add('lge='+lge);
lParam.Add(Kf1+'=USERNAME');
lParam.Add(Kf2+'=PASSWORD');
lParam.Add(Kf3+'=');
lParam.Add(Kf4+'=');
lParam.Add('login=1');
memo1.Lines.Add(http.Post('https://www.neobux.com/m/l/', lParam));
end;
(the getParamLge and GetInput function, are just simple copy and pos functions to extract value from the GET respone).
I thought maybe it needed cookies so i've added this in the beginning:
Cookie.CookieCollection.Clear;
Cookie.CookieCollection.AddClientCookies('CFID=21531887; CFTOKEN=20369251; dh=20130709111845,1920x1080,{ts ''2013-07-09 06:18:58''}; __utma=90161412.436822896.1373368451.1373368451.1373368451.1; __utmb=90161412.11.10.1373368451; __utmc=90161412; __utmz=90161412.1373368451.1.1.'+'utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __asc=06ff77ad13fc32381fd1f5d6405; __auc=06ff77ad13fc32381fd1f5d6405; __atuvc=4%7C28; MS=flat');
But all in vain.
I'm very sure that I'm POSTing the right data
Since it does not work - obviously you do not (or Delphi does not - that makes no difference for server).
You should start usual debugging loop:
Observe reference working behaviour.
Observe your program behavior
Spot the difference
Eliminate the difference
Check if the program works now
If not - go to step 2.
Reference implementation would be some WWW browser working with site: Opera, Chrome, Firefox, MS IE, etc.
Observing tool would be some HTTP Sniffer like WireShark or OmniPacket or Microsoft Net Monitor or else, however this tinkers with OS work on rather deep level.
Or it can be local proxy with GUI, like Proxomitron or Membrane Monitor - but that would require special setup for both the program and the browser, to route their traffic through that local proxy.
Then you should read about HTTP, starting with shallow observation at Wikipedia and then opening related RFC documents (specifications of different part of HTTP protocol) so that you would understand what do the observed differences mean and how to fix them. For example many people use POST request when they actually should use GET request or such.
You want to debug HTTP program but for this HTTP logs, workign and borken, are required and your question lacks them. More so, most probably you can fix it your self, just bring your program's HTTP log to accordance with both RFCs theory and working browsers practice.
I'm wanting to create a CloudFlare client in the Firemonkey framework. For those who don't know, CloudFlare serves as a CDN of sorts for anyone with a website. They have an API available, and as with many web API's, they are using JSON with a token-based system. It requires both the account email address and the account token to access the API. It runs on HTTPS, and as you can imagine, attempting to access the API via HTTP/non-SSL simply produces null results.
The application i wish to create would serve as an all-in-one management tool, intending to eliminate the need for me to use a web browser to manage my CloudFlare settings. I'm having the most basic of issues; SSL POST. See, i can submit an API request via a web browser and get a list of results (e.g. https://www.cloudflare.com/api_json.html?a=stats&z=DOMAIN&u=EMAIL&tkn=TOKEN - Personal details removed for obvious reasons), but i'm unsure how i would go about getting these same results (or any results from the API for that matter) in Firemonkey.
I've got Overbyte ICS with SSL installed, as well as the basic bundled Indy components, but i'm struggling to get started with this. I need to post a list of parameters to https://www.cloudflare.com/api_json.html via HTTPS/SSL, but i've very little idea on where to start. I've seen a few various example around SO, mostly using ICS, but i've been unable to find any specific to posting with multiple parameters, how i should format it, etc.
One example i tried was using ICS TSSLHttpCli, writing my parameters as a single string (i.e. a=stats&z=DOMAIN&u=EMAIL&tkn=TOKEN), writing that to the SendStream of TSSLHttpCli, seeking to 0,0, setting the URL (i.e. https://www.cloudflare.com/api_json.html?), and then calling the Post method. However, this gives me Connection aborted on request. This is the code i've tried (though i've replaced personal details with generic values);
var
Data : AnsiString;
RcvStrm, SndStrm : TMemoryStream;
begin
SndStrm := TMemoryStream.Create;
RcvStrm := TMemoryStream.Create;
Data := '?a=stats&z=MYDOMAIN&u=MYEMAIL&tkn=MYTOKEN';
SslHttpCli.SendStream := SndStrm;
SslHttpCli.SendStream.Write(Data[1],Length(Data));
SslHttpCli.SendStream.Seek(0,0);
Memo1.Lines.LoadFromStream(SndStrm);
ShowMessage('Waiting!');
SslHttpCli.RcvdStream := RcvStrm;
SslHttpCli.URL := 'https://www.cloudflare.com/api_json.html';
SslHttpCli.Post;
Memo1.Lines.Clear;
Memo1.Lines.LoadFromStream(RcvStrm);
Memo1.Lines.Add('.....');
RcvStrm.Free;
SndStrm.Free;
ShowMessage('Complete!');
end;
The ShowMessage procedures are simply there to provide a visual break so i can see what data is in the stream at each time. When Memo1.Lines.LoadFromStream(SndStrm); is called, i get a single question mark the contents of the Data in the memo as expected.
When i call Memo1.Lines.LoadFromStream(RcvStrm);, i expect it to add the return result from the API, and then the 5 dots underneath it. However, this does not happen, and it's apparent that the message i'm receiving is related to the issue. I'm assuming i've not set up the data correctly, but i'm simply unsure exactly how i should format it prior to attempting to post it. I've even commented out everything below Memo1.Lines.LoadFromStream(RcvStrm); to the end to see whether the Clear procedure is called on the memo, but the contents of the memo remain the same as they were when i called LoadFromStream(SndStrm). The final ShowMessage is also not called.
I initially tried using String instead of AnsiString, but this simply output the first character of Data rather than the whole string.
There could be numerous reasons why it's not working (all details for API access are correct, so it's an issue with the code), but i need someone with more experience and knowledge to point me in the right direction.
My network coding knowledge is limited, and i've only dealt with basic SQL and FTP in Delphi so far. I've still got to work with the parsed JSON once i do get past this step, but for now, can anyone assist me in this endeavor so i can get started?
I noticed you seemed to solve this with a GET request, but I noticed two immediate problems with your POST request:
as Runner Suggested, drop the '?' in your data. The '?' is only used when appending parameters to the URL in a GET request.
You never set the content type of the HTTP Request (should be application/x-www-form-urlencoded). You can do this with the following code:
SSLHttpCli.ContentTypePost := 'application/x-www-form-urlencoded';
Just a helpful thought. I checked https://www.cloudflare.com/docs/client-api.html and they mention that POST requests are accepted. It's possible the server rejects requests that have any other content type.
Just some food for thought if you ever need to contact another API via POST requests and want to use the Overbyte Components.
Hope the info is useful!
Try this;
SndStrm := TMemoryStream.Create;
RcvStrm := TMemoryStream.Create;
Data := 'a=stats&z=MYDOMAIN&u=MYEMAIL&tkn=MYTOKEN';
SndStrm.Write(Data[1], Length(Data));
SndStrm.Seek(0, 0);
SslHttpCli.SendStream := SndStrm;
Using Delphi 7 I am reading from a serial port.
The read is always preceded by a write which triggers the h/w to measure from a sensor and write something for me to read (and there is always something to read).
I have two possibilities: manually enter a command and click a button to write that to the serial port (read model or f/w version, etc) or click a button to loop reading measurements until a stop button is pressed. These both use the same internal functions, so the code looks something like this:
WriteSerial('?model');
SerialData := ReadSerial(); // returns string
WriteSerial('?fw');
SerialData := ReadSerial();
and
while stopButtonNotPressed do
begin
WriteSerial('?data');
SerialData := ReadSerial();
Memo1.Lines.Add(SerialData );
end;
The first variant (manually entering a command & pressing a button) is always successful, no matter how quickly or slowly I enter commands (hold down button for repeat), where are the second goes
pass
fail
pass
pass
fail
pass
pass
fail
... add infinitum
adding calls to sleep produces nothing, but trying to debug, I found that if I add a modal dialog box MsgDialog, 'Please close this dialog...', mtInfo, [mrOK]); to the loop, then it no longer fails.
Now, it doesn't look like timing (else surely adding Sleep(2000); to the loop would make it pass & does not, so why does pressing a button on the main form or the modal dialog cause it to succeed?
Btw, the h/w user guide says nothing of CTS / RTS, and the sole code example provide also does not.
Note: if I manually enter ?data repeatedly it never fails ...
Any ideas?
Your serial devices need time to react, so obviously you need a break for the device to catch up. When you use the keyboard to push the button you're providing the brake it needs because the keyboard repeat isn't all that fast.
As you say Sleep(2000) should provide plenty of "break", but there are two other potential problems you'll need to take care of:
Serial communication isn't necessary buffered: Sleep(2000) might be too long!
The serial library you're using might be using windows messages to process incoming bytes. Sleep() inhibits the message pump, so no more messages flow towords your application
Try "sleeping" using something like this:
procedure BusyWait(ms: Cardinal);
var StopAt: TDateTime;
begin
StopAt := Now + EncodeTime(0, 0, ms div 1000, ms mod 1000);
while StopAt > Now do
begin
Application.ProcessMessages;
Sleep(50); // per Michał Niklas's suggestion, to keep the CPU from reaching 100%
end;
end;
This routine will wait, but it'll keep the message pump going, allowing your serial library to receive messages. If that's the problem...
Maybe adding Application.ProcessMessages() before Sleep() will help.
I'm using TNMHTTP in Delphi to retrieve the code from a webpage. The code is relatively simple:
NMHTTP1 := TNMHTTP.Create(Self);
NMHTTP1.InputFileMode := FALSE;
NMHTTP1.OutputFileMode := FALSE;
NMHTTP1.ReportLevel := Status_Basic;
NMHTTP1.TimeOut := 3000;
URL := 'http://www....';
NMHTTP1.Get(URL);
S := NMHTTP1.Body;
I am catching exceptions in a try/except block, but that is not the problem.
The problem is that on executing the NMHTTP1.Get method when the URL is a redirect, that method does not return and the program hangs. This is despite the fact that I've put a timeout of 3000 seconds in.
So I see three possible ways of solving this (in order of easiest to hardest for me to modify my program):
Do whatever is necessary to get the NMHTTP1.Get method to respond.
Do some sort of check in advance of the NMHTTP1.Get statement to see if the URL is a redirect and get the URL it is redirecting to.
Use another method to get a webpage using Delphi. When I wrote this, I used Delphi 4 and did not have Indy. I now have Delphi 2009, so I would be willing to use something that works in it (maybe INDY) if a simple #1 or #2 answer is not available.
I would love to get an answer from someone that will work for me. Thanks in advance.
I would avoid the NetMasters controls, period.
Instead, you can use Indy's IdHTTP component, which has a RedirectMaximum property (defaults to 15) and an OnRedirect event in case you want to track the details.
I can advice you to switch to Indy. They are great for a lot of network protocols (with the exception of the IRC protocol). There are nice examples included so you can examine the working examples yourself.
Also have a look at http://www.indyproject.org/index.en.aspx for more information on indy.