How to parse article page links in rails with pismo and mechanic? - ruby-on-rails

Question

I'm trying to parse links on multiple-page articles to automatically click through them to extract the whole article content. I'm using mechanize, regarding to my last question and the helpful answer.
How can I search for pagination links? Each articles may have different link architectures, like:
ZEITONLINE:
<a id="hp.article.bottom.paginierung.2" class="pn-forward pn-button" title="Vor" href="http://www.zeit.de/politik/ausland/2013-01/Syrien-Fotografie-Reportage/seite-2">Vorwärts</a>
ARSTECHNICA:
<span class="next">Next <span class="arrow">→</span></span>
IGN:
Next »
For the IGN Link it's relatively simple to parse the link, because it contains the link text Next. But what about the other links? I know it should be doable, because pocket, readability and instapaper are extracting multiple page content.
Hope you can help me a bit.

Answer 1

I Write One Function For This
function proccessURL($ParentURL,$URL){
$parse_url=parse_url($URL);
if(#$parse_url['host']==""){
$Parent_URL=parse_url($ParentURL);
$path=explode("/",#$parse_url['path']);
$redirect=0;
$lkey=0;
$flag=false;
while(list($key,$val)=each($path)){
if($val==".." or $val=="." or $val=="..."){
$redirect++;
$lkey=$key;
$flag=true;
}else{
break;
}
}
if($flag){
$matches=explode("/",$Parent_URL['path']);
end($matches);
$b=each($matches);
$n=$b['key'];
$url='';
for($i=0;$i<$n-$redirect;$i++){
$url.=$matches[$i]."/";
}
for($i=$redirect+1;next($path);$i++){
$url.=$path[$i]."/";
}
rtrim($url,"/");
$parse_url['path']=$url;
}else{
$parse_url['path']="/".#$parse_url['path'];
}
}else{
$Parent_URL['scheme']=$parse_url['scheme'];
$Parent_URL['host']=$parse_url['host'];
}
//print_r($parse_url);
if(#$parse_url['query']!=""){
$parse_url['query']="?".#$parse_url['query'];
}
if(#$parse_url['fragment']!=""){
$parse_url['fragment']="#".#$parse_url['fragment'];
}
return $Parent_URL['scheme']."://".#$Parent_URL['host'].#$parse_url['path'].#$parse_url['query'].#$parse_url['fragment'];
}
This Function solve link Address
sample:
$CorrectLink=proccessURL("http://www.sepidarcms.ir/kernel/","../plugin/1.php");
The Output is "http://www.sepidarcms.ir/plugin/1.php"
Now You Can Parse Url By preg_match_all
$html="Your HTML Str";
$URL="Your HTML Page Link";
preg_match_all("/href=\"([^\"]*)\"/is", $html, $matches);
while(list($key,$val)=each($matches[1])){
$val=proccessURL($URL,$val);
echo $val;
}
This code List All href Url For You Correctly