I have 1000+ URLs that I want to scrape to retrieve the title value from the HTML. After trying different things, I ultimately used iMacros scripts, which I don't know anything about. Nonetheless, I managed to make a script after reading guides.
My script is working perfectly but has a problem: When leeching URLs titles, if it encounters an HTTP error (e.g. dead link, forbidden page, etc), it crashes with an error message like this one:
Error -1350: Error loading page. Http status 403. Line 4: URL GOTO=http://url.com
Instead of crashing when the script encounters these errors, I would like it to simply skip the URL and continue running. How can I modify my script to do this? Here is my script:
VERSION BUILD=9002379
TAB T=1
TAB CLOSEALLOTHERS
URL GOTO=http://google.com/
ADD !EXTRACT {{!URLCURRENT}}
TAG POS=1 TYPE=TITLE ATTR=* EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=d:/ FILE=links.txt
SET !EXTRACT_TEST_POPUP NO
Output:
http://google.com/,Google
I would also like to replace the comma after the URL in the output with a semicolon.
At critical points where you don't want to exit at fails:
SET !ERRORIGNORE YES
If you want to revert back at some point to do stop on an error:
SET !ERRORIGNORE NO
You can use these two as many times as you like, even every second row turn on and off.
Related
I have a problem passing special parameters from a bash script to a browser URL bar.
I would like to open a PDF file on a certain page, and with a certain zoom factor, and at a certain position (relative to a page corner, I think that's called "focus") in Brave, from the command line.
I thought that should work like this:
$ brave-browser file:/home/user/foo.pdf#page=5&zoom=150,50,50
(the ",50,50" would be the "focus") but it doesn't. It just provokes an error message.
Copy/pasting the entire line with all the special characters manually to the URL bar works fine, but I would like to do that without the mouse or keyboard, from the CL. In other words, copying the string
file:/home/user/foo.pdf#page=5&zoom=150,50,50
and pasting that straight into the URL bar will open the file the way I want in Brave.
This shortened command works from the CL:
$ brave-browser file:/home/user/foo.pdf#page=5
but adding the &zoom=150,50,50 provokes an error message from Brave.
I just can't get my head around how to format the rest of the line so that the file opens in the browser as intended (zoomed and focussed), when called from the CL.
I begin to wonder if that is possible at all.
I have tried everything that was suggested here:
Shell script to open a URL
and a couple of other ideas I had (mainly, creative use of quotation marks)
I also read this, but I am not sure if it refers to the same question - and the post is 11 years old. Was hoping that things changed:
Can a website pass focus to the browsers url field?
Thanks a lot for any help!
Just found out that all it takes is a backslash in front of the ampersand...
Strange: no matter how I do write that here, with the *** before and after the code, or without the three *** , I can never see the backslash in the post's preview!
browser file:/home/user/foo.pdf#page=5 \ &zoom=150,50,50
I have to type blanks before and after the hash to make it visible in this post as above. Looks wrong but this is the only way I can show what I mean. Also strange: the hash # works without the backslash \
Thanks, should someone have looked into this in the meantime!
I am using drafts 5 and would like to convert the markdown into a powerpoint using a callback URL action. It appears like it should be quite straightforwards but Im not getting any success using the following:
makeslides://x-callback-url?input=[[draft]]&type=md&name=mydoc
I just keep getting an error message saying invalid URL generated. Any ideas on how to get this to work?
I manage to get it working when I do two steps, first copy to clipboard and second use the callback URL to makeslides
makeslides://x-callback-url?input=[[clipboard]]&type=md&name=[[title]]
I am making an JSF 2 application. Sometimes I want to do simple things that are easy to do using plain HTML but I get error messages that I don't understand.
For example I sometimes want to write:
Test
But this makes the page break because of "?test1=ee&test2=oo". Having only one parameter seems to be fine but if I try to add more parameters using "&" it complains and says I need to use a ';'-delimiter.
Any explanations why this happens and maybe how to get around it?
I have a url like this: /hello/world#/lol/backbone
I'd like to get access, in ruby, to the full, entire path. I'd normally do something like this: request.fullpath but that returns only /hello/world and leaves off the #/lol/backbone/
Been Googling for awhile. Can't seem to find it.
You can't get the fragment in your server code, the URL fragment (#/lol/backbone) is a client-side issue. The browser will interpret the fragment but it will not send it to your server.
If you need the fragment then you'll have to convert it to a parameter by intercepting your link with a bit of JavaScript, then the server could see something like
/hello/world?frag=/lol/backbone
and you could pull the fragment out of params[:frag]. This does of course assume that your JavaScript link interceptor will get run, that's not guaranteed so your server-side code should be prepared for a missing params[:frag].
I've got following URL in symfony (specifics not important):
/frontend_dev.php/something/25/apple
... and a routing rule:
/something/:id/:word
The URL works fine when clicked through to on the site, but not when I type in the URL. Instead, symfony says:
Unable to find a matching route to generate url for params "NULL".
The weird thing is that I can navigate to this page and it works, but when hitting Enter in the browser address bar, it no longer finds it.
Any thoughts on what might be the cause of something like this generally?
I should also add that the URL was working fine when typed in the address bar earlier, but doesn't anymore, and I'm not sure what's there that might be interfering with it.
Thanks in advance.
UPDATE:
The exact routing rule:
profile:
url: /profile/:id/:un
param: {module: profile, action: profile}
UPDATE:
Debug toolbar log (the higlighted part):
Unable to find a matching route to generate url for params "NULL". Toggle debug stack
#4 » in sfException::outputStackTrace() from SF_ROOT_DIR\lib\vendor\symfony\lib\exception\sfException.class.php line 110
#3 » in sfException->printStackTrace() from SF_ROOT_DIR\lib\vendor\symfony\lib\controller\sfFrontWebController.class.php line 52
#2 » in sfFrontWebController->dispatch() from SF_ROOT_DIR\lib\vendor\symfony\lib\util\sfContext.class.php line 170
#1 » in sfContext->dispatch() from SF_ROOT_DIR\web\frontend_dev.php line 13
RESOLVED:
Found it....
I had this in my code on the failing page:
<?php echo link_to(__('Back to previous page').' ›', $sf_request->getReferer()) ?>
The referer fails when typing address into bar.
Unless you're neglecting to mention that you navigate to this page usually from forms and it's the lack of POST data that is upsetting the routing; which I'm thinking this is not- this surely must be an issue with your browser.
Have you tried this in a different browser on your system?
If you're still having issues can you copy in the relevant section of your routing.yml?
I was getting this error message on a live site and found the issue to be a link_to command which was being dynamically populated by a field from the database.
However the field was null, so the link_to received no link info.