How to write a script to download all videos from the links in a webpage
Hey Guys,
I want to write a script to download all rails screen casts from this location http://railscasts.com/episodes/archive
Any ideas on how this can be automated?
I'd personally go with wget -l inf -r -np http://railscasts.com/episodes.
Related
I have a specific task to accomplish which involves downloading a file from Google sheets. I need to always have just one file downloaded so the new file will overwrite any previous one (if it exists)
I have tried the following command but I can't quite get it to work. Not sure what's missing.
/usr/local/bin/php -q https://docs.google.com/spreadsheets/d/11rFK_fQPgIcMdOTj6KNLrl7pNrwAnYhjp3nIrctPosg/ -o /usr/local/bin/php /home/username/public_html/wp-content/uploads/wpallimport/files.csv
Managed to solve with the following:
curl --user-agent cPanel-Cron https://docs.google.com/spreadsheets/d/[...]/edit?usp=sharing --output /home/username/public_html/wp-content/uploads/wpallimport/files/file.csv
There is a web page www.somepage.com/images/
I know some of the images there (e.g. www.somepage.com/images/cat_523.jpg, www.somepage.com/images/dog_179.jpg)
I know there are some more but I don't know the names of those photos. How can I scan whole /images/ folder?
you can use wget to download all the files
--no-parent to grab all the files below in the directory hierachy
--recursive to look into subfolders
wget --recursive --no-parent -A jpeg,jpg,bmp,gif,png http://example.com/
If they are on the webpage as an img tag you could try just searching the page source for an img tag. If you are using terminal you could also try using a tool such as wget to download the web page and then try using grep on the file for the img tag.
I'm trying out MGT Development Environment 7.0 and installed a fresh copy of Magento 2.
every time after php bin/magento setup:upgrade, and reload the page, generated files in var, pub, generated have different user and group clp:clp.
Instead of running chmod -R 777 . every time. Can anyone suggest a better solution?
Thank in advance.
After view phpinfo(), found out that php in running by user clp
Simply chown the webroot clp:clp and everytime run php command by sudo -u clp php yourCommand, which solve the problem.
I tried lots of suggestion but i can't find a solution (I don't know if it's possible) I use terminal of Ubuntu 15.04
I'd need to download in a text file all of internal and external links from mywebsite.com/links_ (all links start with links_) For example http://www.mywebsite.com/links_sony.aspx I don't need all other links ex. mywebsite.com/index.aspx or conditions.asp etc. I use
wget --spider --recursive --no-verbose --output-file="links.csv" http://www.mywebsite.com
Can you help me please? Thanks in advance
If you don't mind using a couple of other tools to coax wget, then you can try this bash script that employs awk, grep, wget and lynx:
#! /bin/bash
lynx --dump $1 | awk '/http/{print $2}' | grep $2 > /tmp/urls.txt
for i in $( cat /tmp/urls.txt ); do wget $i; done
Save the above script as getlinks and then run it as
./getlinks 'http://www.mywebsite.com' 'links_' > mycollection.txt
This approach does not load/need too many other tools; instead reuses commonly available tools.
You may have to play with quoting depending what shell you are using. The above works in standard bash and is not dependent on specific versions of these tools.
You could customize the part
do wget $1
with appropriate switches to meet your specific needs, such as recursive, spider, verbosity, etc. Insert those switches between wget and $1.
I'm a Junior Programmer where I work. Our website was written using PHP 4. We're migrating from PHP 4 to PHP 5.3. There are roughly 5000 PHP files in around 595 directories. So, as you can imagine, the scope of this project is pretty huge.
We use Subversion for version control. I have two separate checkouts. I have two VMs that act as separate webhosts - one stack emulates our actual webserver (CentOS 4, PHP4, etc) and the other is a PHP 5.3 stack (Ubuntu 12.04 LTS).
I took the time to check the files for basic syntax errors using the following commands:
Edit: I ran the following recursive searches from the root of the website.
find ./ -type f -name \*.php -exec php -l {} \; < ~/php5_basic_syntax_assessment.txt
find ./ -type f -name \*.inc -exec php -l {} \; < ~/php5_basic_syntax_inc_assessment.txt
I realize that using php -l to check basic syntax doesn't reveal deprecated code structures/functions and doesn't provide warnings (IE: use preg_slice() instead of slice()). Therefore, I decided to install PHP CodeSniffer.
First, I installed PEAR: [I accepted all the default parameters]
cd ~/
mkdir pear
cd pear
wget http://pear.php.net/go-pear.phar
php go-pear.phar
Next, I installed git:
cd ~/
sudo apt-get update
sudo apt-get install git
Next, I installed PHP Code Sniffer
pear install PHP_CodeSniffer
Finally, I installed the following PHP 5.3 Compatibility standards for the PHP Code Sniffer:
git clone git://github.com/wimg/PHP53Compat_CodeSniffer.git PHP53Compatibility
I did all of the above so that I could assess the 5K PHP files in an automated kind of way. It would be extremely tedious and time consuming to go through each file to make sure they manually follow the PHP 5.3 coding standards.
Finally, here's the command I used to run the PHP Code Sniffer:
phpcs --standard=/home/my_user_name/PHP53Compatibility -p --report-file=/home/my_user_name/php53_assessment.txt /path/to/web/root
To make sure that the specific standards aren't the problem, I also ran the PHP Code Sniffer using the default standards:
phpcs -p --report-file=/home/my_user_name/php53_assessment.txt /path/to/web/root
Either way, the reports freeze in the same place. I've been awake for over 24 hours. I waited for 18 hours before stopping the first run by using CTRL+C. The second is still running and has been running for about an hour and a half.
So, what is causing my PHP Code Sniffer to freeze?
All help is very much appreciated.
Bit late, but I ran into the same issue. Limit the files to just PHP files should do the trick: phpcs -p -- ./**/*.php