Using TidHttp to download Jpeg images from URL (only those that exist)? - delphi

I am trying to retrieve a large number of images from the web using a TidHttp component.
The problem is that there is a number of images that are missing (Example: 7403, 7412, etc)
How do i test for only those that exist and save those to file?
procedure TForm.Button1Click(Sender: TObject);
var
MS : TMemoryStream;
JPEGImage: TJPEGImage;
Url, numString: String;
I, Code: Integer;
begin
for I := 7400 to 7500 do
begin
{
Url :='http://www.mywebpage.com/images/DSC' + numString+ '.jpg';
try
idhttp1.Head(URL);
code := idhttp1.ResponseCode;
except on E: EIdHTTPProtocolException do
code := idhttp1.ResponseCode;
end;//try except
if code = 200 then
begin
MS := TMemoryStream.Create;
JPEGImage := TJPEGImage.Create;
try
try
idhttp1.Get(Url, MS); //Send the request and get the image
code := idhttp1.ResponseCode;
MS.Seek(0,soFromBeginning);
JPEGImage.LoadFromStream(MS);//load the image in a Stream
Image1.Picture.Assign(JPEGImage);//Load the image in a Timage component
Image1.Picture.SaveToFile('C:\Museum_Data\DSC' + numString + '.jpg');
Application.ProcessMessages;
except
on E: EIdHTTPProtocolException do
code := idhttp1.ResponseCode; // or: code := E.ErrorCode;
end; //try except
finally
MS.free;
JPEGImage.Free;
end; //try finally
end; //if
end;
end;

You don't have to do anything extra for that. If you try to access a non-existant URL, the HTTP server will report an error that TIdHTTP than wraps into an EIdHTTPProtocolException exception. You do not have to bother with calling TIdHTTP.Head() first, since you are downloading the images to a TMemoryStream before saving them. You can catch the exception when calling TIdHTTP.Get() by itself, no need to check the ResponseCode at all.
Try this:
procedure TForm.Button1Click(Sender: TObject);
var
MS: TMemoryStream;
JPEG: TJPEGImage;
Url: String;
I: Integer;
begin
MS := TMemoryStream.Create;
try
JPEG := TJPEGImage.Create;
try
for I := 7400 to 7500 do
begin
Url := 'http://www.mywebpage.com/images/DSC' + IntToStr(I) + '.jpg';
MS.Clear;
try
IdHTTP1.Get(Url, MS);
except
on E: EIdHTTPProtocolException do
Continue;
end;
MS.Position := 0;
JPEG.LoadFromStream(MS);
Image1.Picture.Assign(JPEG);
JPEG.SaveToFile('C:\Museum_Data\DSC' + IntToStr(I) + '.jpg');
Application.ProcessMessages;
end;
finally
JPEG.Free;
end;
finally
MS.Free;
end;
end;
You do not actually need the TImage in order to save the data to file. If you can omit the TImage.Picture.Assign() stage, then the code a bit simpler by eliminating the TJPEGImage altogether (unless you are trying to validate the download files are valid), eg:
procedure TForm.Button1Click(Sender: TObject);
var
MS: TMemoryStream;
Url: String;
I: Integer;
begin
MS := TMemoryStream.Create;
try
for I := 7400 to 7500 do
begin
Url := 'http://www.mywebpage.com/images/DSC' + IntToStr(I) + '.jpg';
MS.Clear;
try
IdHTTP1.Get(Url, MS);
except
on E: EIdHTTPProtocolException do
Continue;
end;
MS.Position := 0;
MS.SaveToFile('C:\Museum_Data\DSC' + IntToStr(I) + '.jpg');
Application.ProcessMessages;
end;
finally
MS.Free;
end;
end;
Or:
procedure TForm.Button1Click(Sender: TObject);
var
FS: TFileStream;
Url, FileName: String;
I: Integer;
begin
for I := 7400 to 7500 do
begin
Url := 'http://www.mywebpage.com/images/DSC' + IntToStr(I) + '.jpg';
FileName := 'C:\Museum_Data\DSC' + IntToStr(I) + '.jpg';
FS := TFileStream.Create(FileName, fmCreate);
try
try
try
IdHTTP1.Get(Url, FS);
except
on E: EIdHTTPProtocolException do
Continue;
end;
Application.ProcessMessages;
finally
Fs.Free;
end;
except
DeleteFile(FileName);
end;
end;
end;

Related

send and recive TStringStream with indyTcp server and client

I am trying to send a TStringStream from client to server, then send it back from server to client, using Indy TCP components.
Here is my client code:
var
Jpg: TJPEGImage;
StringStream: TStringStream;
strcams, StringImageData: String;
byt, i: integer;
procedure SendCommandWithParams(Command, Params: String);
begin
Lock;
try
if not FTCP.Connected then
begin
exit;
end;
FTCP.Socket.WriteLn('1' + Command, IndyTextEncoding_UTF8);
FTCP.Socket.WriteLn(Params, IndyTextEncoding_UTF8);
finally
Unlock;
end;
end;
begin
Jpg := TJPEGImage.Create;
StringStream := TStringStream.Create('');
try
try
Jpg.Performance := jpBestSpeed;
Jpg.ProgressiveEncoding := True;
Jpg.ProgressiveDisplay := True;
Jpg.Assign(Image2.Picture.Bitmap);
Jpg.CompressionQuality := 25;
Jpg.Compress;
Jpg.SaveToStream(StringStream);
StringImageData := StringStream.DataString;
strcams := '<[S:' + IntToStr(Length(StringImageData)) + 'B]>' +
StringImageData;
if Length(strcams) < byt then
begin
SendCommandWithParams('SIMGSEND', strcams + sep + 'IMGID5423' + sep);
end;
except
on e: exception do
//
end;
finally
StringImageData := '';
FreeAndNil(Jpg);
FreeAndNil(StringStream);
end;
end;
I can receive the TStringStream data, but the data received is corrupted, and some times it gets replaced with the second parameter that I send which is 'IMGID5423' + sep. I am not sure if this is because of some limit of packet sending through TCP so the data does not arrive complete, or is this a parser issue?
My current parser should separate each text that ended with #13#10. Here is how it looks:
var
ReceiveParams, ReceiveStream: Boolean;
S: string;
Command: String;
begin
Command := Fholdcommand;
ReceiveParams := false;
ReceiveStream := false;
if Command[1] = '1' then // command with params
begin
Command := Copy(Command, 2, MaxInt);
ReceiveParams := True;
end;
if ReceiveParams then // params incomming
begin
S := FTCP.Socket.ReadLn(IndyTextEncoding_UTF8);
FCMD := Command;
FPRMS := S;
FSTREAM := false;
if Assigned(FOnCallbackProc) then
begin
Synchronize(DoCallbackProc);
end;
end;
I am still confused about the real issue. I try to send the TStringStream in a local procedure, and it is received normally without any corruption.
Am I sending the data wrong altogether through Indy?
This is how I am receiving the data:
procedure CreateJpg(Data:string);
var
StringStream : TStringStream;
JpegImage : TJPEGImage;
Bitmap : TBitmap;
tmpPos:integer;
pp:string;
label check;
begin
GData := Data;
if LeftStr(GData,4) = '<[S:' then
begin
tmpPos := Pos(WideString('B]>'),GData);
pp := Copy(GData,5,tmpPos-5);
CDataELen := StrToInt(pp); //MidStr(st,5,tmppos - 5);
CData := RightStr(GData,length(GData)-(tmppos+2));
goto check;
end;
CData := CData + GData;
check:
//if CDataELen = length(CData) then
begin
StringStream := TStringStream.Create('');
JpegImage := TJpegImage.Create;
StringStream.WriteString(CData);
CData := '';
try
try
StringStream.Seek(0, soFromBeginning);
JpegImage.LoadFromStream(StringStream);
Bitmap := TBitmap.Create;
with Bitmap do
begin
Canvas.Lock;
try
Width := JpegImage.Width;
Height := JpegImage.Height;
Canvas.Draw(0, 0, JpegImage);
finally
Canvas.Unlock;
end;
end;
img.Picture.Bitmap.Width := Bitmap.Width;
img.Picture.Bitmap.Height := Bitmap.Height;
img.Picture.Bitmap.Canvas.Draw(0, 0, Bitmap);
except
on E: Exception do
//
end;
finally
FreeAndNil(StringStream);
FreeAndNil(JpegImage);
FreeAndNil(Bitmap);
end;
end;
end;
The problem is that you are saving the JPG binary data to a TStringStream and then letting it reinterpret the binary data as if it were string data. You can't do that. You need to save the JPG data to a binary stream instead, like TMemoryStream, and then encode the binary data using a string-safe encoding, like Base64.
Try something more like this instead:
uses
..., IdCoder, IdCoderMIME;
...
var
Jpg: TJPEGImage;
JpegStream: TMemoryStream;
strcams, StringImageData: String;
begin
try
JpegStream := TMemoryStream.Create;
try
Jpg := TJPEGImage.Create;
try
Jpg.Performance := jpBestSpeed;
Jpg.ProgressiveEncoding := True;
Jpg.ProgressiveDisplay := True;
Jpg.Assign(Image2.Picture.Bitmap);
Jpg.CompressionQuality := 25;
Jpg.Compress;
Jpg.SaveToStream(JpegStream);
finally
Jpg.Free;
end;
JpegStream.Position := 0;
StringImageData := TIdEncoderMIME.EncodeStream(JpegStream);
finally
JpegStream.Free;
end;
strcams := '<[S:' + IntToStr(Length(StringImageData)) + 'B]>' + StringImageData;
SendCommandWithParams('SIMGSEND', strcams + sep + 'IMGID5423' + sep);
except
on e: exception do
//
end;
end;
And then on the receiving end:
procedure CreateJpg(Data: string);
var
JpegStream: TMemoryStream;
JpegImage: TJPEGImage;
Bitmap: TBitmap;
tmpPos, tmpLen: integer;
pp: string;
begin
try
if not TextStartsWith(Data, '<[S:') then
begin
// bad data, do something else...
Exit;
end;
tmpPos := Pos('B]>', Data);
pp := Copy(Data, 5, tmpPos-5);
tmpLen := StrToInt(pp);
Data := Copy(Data, tmpPos+3, tmpLen);
Bitmap := TBitmap.Create;
try
JpegImage := TJpegImage.Create;
try
JpegStream := TMemoryStream.Create;
try
TIdDecoderMIME.DecodeStream(Data, JpegStream);
JpegStream.Position := 0;
JpegImage.LoadFromStream(JpegStream);
finally
JpegStream.Free;
end;
with Bitmap do
begin
Canvas.Lock;
try
Width := JpegImage.Width;
Height := JpegImage.Height;
Canvas.Draw(0, 0, JpegImage);
finally
Canvas.Unlock;
end;
end;
finally
JpegImage.Free;
end;
img.Picture.Assign(Bitmap);
finally
Bitmap.Free;
end;
except
on E: Exception do
//
end;
end;
Your problem appears to be that you are treating binary data as though it is text. Binary data can contain anything, for instance #13#10 line breaks or indeed anything whatsoever.
If you wish to send that data as text, then you need to use a text encoding. For example, encode it as base64.
Or transmit the content as binary rather than text.

Get filesize from IdTCPClient1

I want to get filesize, but after receiving file lose 4 bytes. Because of what is happening and how to fix it?
procedure TForm1.IdTCPClient1Connected(Sender: TObject);
var
FS: TFileStream;
FPath, FName: String;
begin
FPath := ExtractFilePath(Application.ExeName) + 'Downloads';
FName := IdTCPClient1.IOHandler.ReadLn;
if not DirectoryExists(FPath) then ForceDirectories(FPath);
FS := TFileStream.Create(FPath + '\' + ExtractFileName(FName), fmCreate);
try
CProgressBar.Max := IdTCPClient1.IOHandler.ReadLongInt(); // Proble here. Without thats ok
IdTCPClient1.IOHandler.ReadStream(FS, -1, True);
finally
FreeAndNil(FS);
end;
IdTCPClient1.Disconnect;
CLabel.Caption := 'File received';
end;
File size should be sent separately

How to get image binary data using XMLHTTPRequest in Delphi

I need to access binary image data using XMLHttpRequest in Delphi. I am using the following code but its not working, could someone please tell me what's wrong with this code, thanks in advance.
//I am using this function to get Image Binary data into Memory Stream.
procedure SendGETRequest(p_Url: string; p_resStream: TMemoryStream);
begin
FXmlHttpReq.open(METHOD_GET, p_Url, false, FUsername, FPassword);
FXmlHttpReq.setRequestHeader(HTTP_AUTHENTICATION, HTTP_BASIC + EncodeBase64(
FUsername + ':'+FPassword));
FXmlHttpReq.setRequestHeader(HTTP_CACHE_CONTROL, HTTP_NO_CACHE);
//FXmlHttpReq.setRequestHeader('Content-type','application/octet-stream');
FXmlHttpReq.send('');
if not VarIsEmpty(FXmlHttpReq.responseBody) then
begin
p_resStream:= OleVariantToMemoryStream(FXmlHttpReq.responseStream);
end;//if...
end;
function OleVariantToMemoryStream(OV: OleVariant): TMemoryStream;
var
Data: PByteArray;
Size: integer;
begin
Result := TMemoryStream.Create;
try
Size := VarArrayHighBound (OV, 1) - VarArrayLowBound(OV, 1) + 1;
Data := VarArrayLock(OV);
try
Result.Position := 0;
Result.WriteBuffer(Data^, Size);
finally
VarArrayUnlock(OV);
end;
except
Result.Free;
Result := nil;
end;
end;
responseStream is IStream. You need to convert it using TOleStream (AxCtrls):
uses AxCtrls, ComObj, ActiveX;
procedure TForm1.Button1Click(Sender: TObject);
var
oXMLHTTP: OleVariant;
MemoryStream: TMemoryStream;
Stream: IStream;
OleStream: TOleStream;
begin
oXMLHTTP := CreateOleObject('MSXML2.XMLHTTP.3.0');
oXMLHTTP.open('GET', 'https://www.google.com/images/srpr/logo11w.png', False);
oXMLHTTP.send(EmptyParam);
Stream := IUnknown(oXMLHTTP.ResponseStream) as IStream;
OleStream := TOleStream.Create(Stream);
try
OleStream.Position := 0;
MemoryStream := TMemoryStream.Create;
try
MemoryStream.CopyFrom(OleStream, OleStream.Size);
MemoryStream.SaveToFile('logo11w.png');
finally
MemoryStream.Free;
end;
finally
OleStream.Free;
end;
end;

Delphi idhttp long URL parameter is splittet automatically

I am using indys idhttp to submit an URL (post)
Procedure submit_post(url_string,EncodedStr:string;amemo:TMemo);
var
aStream: TMemoryStream;
Params: TStringStream;
begin
aStream := TMemoryStream.create;
Params := TStringStream.create('');
try
with Fmain.IdHTTP1 do
begin
Params.WriteString(EncodedStr);
Request.ContentType := 'application/x-www-form-urlencoded';
Request.Charset := 'utf-8';
try
Response.KeepAlive := False;
Post(url_string, params, aStream);
except
on E: Exception do
begin
Screen.Cursor := crDefault;
exit;
end;
end;
end;
aStream.WriteBuffer(#0' ', 1);
aStream.Position := 0;
amemo.Lines.LoadFromStream(aStream);
Screen.Cursor := crDefault;
finally
aStream.Free;
Params.Free;
end;
end;
It works like a charm for me. I am trying to submit a URL (post) with a parameter containing 300 chars, but will be splittet automatically by adding an "&" every 90 chars. So the server only receives 90 chars instead of 300.
How can I submit an URL with a 300 character parameter without this automatic separation ?
function SubmitPost(Params:String): string;
const
URL= 'http://xxxx.com/register.php?';
var
lHTTP: TIdHTTP;
Source,
ResponseContent: TStringStream;
I:Integer;
begin
lHTTP := TIdHTTP.Create(nil);
lHTTP.Request.ContentType := 'text/xml';
lHTTP.Request.Accept := '*/*';
lHTTP.Request.Connection := 'Keep-Alive';
lHTTP.Request.Method := 'POST';
lHTTP.Request.UserAgent := 'OS Test User Agent';
Source := TStringStream.Create(nil);
ResponseContent:= TStringStream.Create;
try
try
lHTTP.Post(URL+Params, Source, ResponseContent);
Result := ResponseContent.DataString;
except
//your exception here
end;
finally
lHTTP.Free;
Source.Free;
ResponseContent.Free;
end;
end;
Usage
mmo1.Text := SubmitPost('Username=xxxx&Password=xxxx');
I found the mistake. My Post function works perfectly, but the URL is built by params coming from a memo line. With "WantReturns = FALSE", I can build a URL with the maximum line length of the memo. I guess 1024 characters per line which is okay for me.

Please help how to send images from client to server?

This is my server's coding
procedure TForm1.IdTCPServerExecute(AThread: TIdPeerThread);
var
InputString: string;
ACommand: string[1];
AFileName: string;
ATempFileName: string;
AFileStream: TFileStream;
begin
InputString := UpperCase(AThread.Connection.ReadLn);
ACommand := Copy(InputString, 1, 1);
AFileName := FPicFilePath + Copy(InputString, 2, 5) + '.jpg';
if ACommand = 'R' then begin
AFileStream := TFileStream.Create(AFileName, fmOpenRead + fmShareDenyNone);
try
AThread.Connection.WriteStream(AFileStream, true, true);
finally
AFileStream.Free;
end;
end else if ACommand = 'S' then begin
ATempFileName := FPicFilePath + 'TEMP.jpg';
if FileExists(ATempFileName) then
DeleteFile(ATempFileName);
AFileStream := TFileStream.Create(ATempFileName, fmCreate);
try
AThread.Connection.ReadStream(AFileStream, -1, false);
//RenameFile(ATempFileName, AFileName);
finally
AFileStream.Free;
end;
end;
AThread.Connection.Disconnect;
end;
And this is my client's coding
procedure TForm1.SendImageToServer(ASendCmd: string);
var
AFileStream: TFileStream;
begin
MessageDlg('Sending ' + ASendCmd + ' :' + FSendFileName, mtInformation, [mbOK], 0);
Screen.Cursor := crHourGlass;
with IdTCPClient do begin
if Connected then Disconnect;
Host := '127.0.0.1';
Port := 2108;
AFileStream := TFileStream.Create(FSendFileName, fmOpenRead);
try
try
Connect;
try
WriteLn(ASendCmd);
WriteStream(AFileStream, true, false);
finally
Disconnect;
end;
finally
AFileStream.Free;
end;
except
end;
end;
Screen.Cursor := crDefault;
end;
I can successfully get images from server, but when I had to send a new image back to server, I just had an empty TEMP.jpg.
Please help.
Thanks.
Delphi 5, Indy 9
When sending a file from the client to the server, the client is not telling WriteStream() to send the stream size, but the server is telling ReadStream() to expect the stream size to arrive, so you have a mismatch.
When sending a file from the server to the client, the server is telling WriteStream() to send the stream size, and the client is telling ReadStream() to expect the stream size to arrive, so there is no mismatch.

Resources