How do I extract the target URL from a Google search result? - delphi

Question

I am trying to extract URLs from Google search results. I use Indy IdHTTP to get HTML results from Google, and I use Achmad Z's code for getting the link hrefs from the page. How can I get the real link target for each URL instead of the one that goes through Google's redirector?
I tried that but I get an "Operand no applicable" error in this part of the code:
function ToUTF8Encode(str: string): string;
var
b: Byte;
begin
for b in BytesOf(UTF8Encode(str)) do
begin
Result := Format('%s%s%.2x', [Result, '%', b]);
end;
end;
I use Delphi 7 with Indy 9.00.10. Maybe indy update will help ?

Answer 1

If I get it right you are trying to fetch the Google search results using TIdHTTP.Get method. If so, thenyou should definitely focus on some Google Search API implementation because
it's impossible to fetch the results this way because you don't have any access to the document inside the iframe in which the search results are, so you won't get any search results by using HTTP GET in this case (or at least I haven't heard about the request which can do that)
it's against Google policies and you should use proper Google Search API instead, for instance Google SOAP Search API, there are available also several types of Google Search API's for various purposes
You can find e.g. here the Delphi wrapper with the demo code for Google Search API. I've tested it with Delphi 2009 on Windows 7/64 and it works fine for me.

Answer 2

In the previous post here I've tried to explain why you should use Google Search API, in this one I'll try to provide you an example with a hope it will work in your Delphi 7.
You need to have the SuperObject (JSON parser for Delphi), I've used this version (latest at this time). Then you need Indy; the best would be to upgrade to the latest version if possible. I've used the one shipped with Delphi 2009, but I think the TIdHTTP.Get method is so important that it must work fine also in your 9.00.10 version.
Now you need a list box and a button on your form, the following piece of code and a bit of luck (for compatibility :)
The URL request building you can see for instance in the DxGoogleSearchApi.pas mentioned before but the best is to follow the Google Web Search API reference. In DxGoogleSearchApi.pas you can take the inspiration e.g. how to fetch several pages.
So take this as an inspiration
uses
IdHTTP, IdURI, SuperObject;
const
GSA_Version = '1.0';
GSA_BaseURL = 'http://ajax.googleapis.com/ajax/services/search/';
procedure TForm1.GoogleSearch(const Text: string);
var
I: Integer;
RequestURL: string;
HTTPObject: TIdHTTP;
HTTPStream: TMemoryStream;
JSONResult: ISuperObject;
JSONResponse: ISuperObject;
begin
RequestURL := TIdURI.URLEncode(GSA_BaseURL + 'web?v=' + GSA_Version + '&q=' + Text);
HTTPObject := TIdHTTP.Create(nil);
HTTPStream := TMemoryStream.Create;
try
HTTPObject.Get(RequestURL, HTTPStream);
JSONResponse := TSuperObject.ParseStream(HTTPStream, True);
if JSONResponse.I['responseStatus'] = 200 then
begin
ListBox1.Items.Add('Search time: ' + JSONResponse.S['responseData.cursor.searchResultTime']);
ListBox1.Items.Add('Fetched count: ' + IntToStr(JSONResponse['responseData.results'].AsArray.Length));
ListBox1.Items.Add('Total count: ' + JSONResponse.S['responseData.cursor.resultCount']);
ListBox1.Items.Add('');
for I := 0 to JSONResponse['responseData.results'].AsArray.Length - 1 do
begin
JSONResult := JSONResponse['responseData.results'].AsArray[I];
ListBox1.Items.Add(JSONResult.S['unescapedUrl']);
end;
end;
finally
HTTPObject.Free;
HTTPStream.Free;
end;
end;
procedure TForm1.Button1Click(Sender: TObject);
begin
GoogleSearch('Delphi');
end;

Answer 3

Answer to my question , maybe it can help someone:
Fetching web page :
memo1.Lines.Text := idhttp1.Get('http://ajax.googleapis.com/aja...tart=1&rsz=large&q=max');
extracting URL's :
function ExtractText(const Str, Delim1, Delim2: string; PosStart: integer; var PosEnd: integer): string;
var
pos1, pos2: integer;
begin
Result := '';
pos1 := PosEx(Delim1, Str, PosStart);
if pos1 > 0 then
begin
pos2 := PosEx(Delim2, Str, pos1 + Length(Delim1));
if pos2 > 0 then
begin
PosEnd := pos2 + Length(Delim2);
Result := Copy(Str, pos1 + Length(Delim1), pos2 - (pos1 + Length(Delim1)));
end;
end;
end;
And on Button1 just put :
procedure TForm1.Button1Click(Sender: TObject);
var Pos: integer;
sText: string;
begin
sText := ExtractText(Memo1.Lines.Text, '"url":"', '","visibleUrl"', 1, Pos);
while sText <> '' do
begin
Memo2.Lines.Add(sText);
sText := ExtractText(Memo1.Lines.Text, '"url":"', '","visibleUrl"', Pos, Pos);
end;
end;
www.delphi.about.com has nice documentation on string manipulation , Zarko Gajic does great job on that site :)
NOTE: if google changes it's source this will be useless.