I just need a hint. I am trying to run the following command from the GNU parallel tutorial (GNU Parallel tutorial):
parallel -S $SERVER1,$SERVER2 echo ::: running on more hosts
I replaced $SERVERX with known hosts in my network. If I execute the command I'm getting asked for my password for each server and after that nothing happens anymore. The curser blinks all day long and I do not get any error message.
I tried different servers with the same result.
The verbose mode shows:
ssh $SERVER1 -- exec perl -e #GNU_Parallel\\=split/_/,\\"use_IPC::Open3\\;_use_MIME::Base64\\"\\;eval\\"#GNU_Parallel\\"\\;\\$SIG\{CHLD\}\\=\\"IGNORE\\"\\;my\\$zip\\=\(grep\{-x\\$_\}\\"/usr/local/bin/bzip2\\"\)\[0\]\\|\\|\\"bzip2\\"\\;open3\(\\$in,\\$out,\\"\>\\&STDERR\\",\\$zip,\\"-dc\\"\)\\;if\(my\\$perlpid\\=fork\)\{close\\$in\\;\\$eval\\=join\\"\\",\\<\\$out\>\\;close\\$out\\;\}else\{close\\$out\\;print\\$in\(decode_base64\(join\\"\\",#ARGV\)\)\\;close\\$in\\;exit\\;\}wait\\;eval\\$eval\\;
and Followed by random characters
Something similar appears four times. I guess for the four jobs I started. I'd be very happy for help.
I think you are expected to set up passwordless ssh logins to all the remotes so GNU Parallel can get into them. – Mark Setchell
This was the right suggestion. Setting up key authentication using ssh-keygen and ssh-copy-id did the job! Thank you very much now it works. A short hint in the tutorial would have been great.
Good day. I am attempting to create/run a script that will allow me to send an updated IOS from a server to my network devices. The following code works when I put in a manual IP address right before the ":flash" command.
#!/user/bin/expect
set IOSroot "/xxxxx/xxx/c3750e-universalk9-mz.150-2.SE10a.bin"
set pw xxxxxxxxxxxxxxxxxxx
spawn scp $IOSroot 1.1.1.1:flash:/c3750e-universalk9-mz.150-2.SE10a.bin
expect "TACACS Password:"
send "$pw\r"
interact
The code there works great and as expected. The issue arises when I try to use a file called "ioshost" with a list of IP's and use that within this script to get some automation. I have tried various things to get this to work. Some of them are as follows:
Settings Variables
IPHosts=$(cat ioshost)
set IPHost 'cat ioshost'
Along with trying to use the read/do command...
while read line; do
spawn scp $IOSroot $line:flash:/c3750e-universalk9-mz.150-2.SE10a.bin
done < ioshost
None of these seem to work and I am looking for guidance. Please note I understand that setting a password is not best practice but setting RSA keys as mentioned in other articles is not allowed so I am forced to do it this way.
Thank you for your time.
You can use one Expect script and one Bash script.
First update your Expect script a bit:
#!/user/bin/expect
set IOSroot "/xxxxx/xxx/c3750e-universalk9-mz.150-2.SE10a.bin"
set pw xxxxxxxxxxxxxxxxxxx
spawn scp $IOSroot [lindex $argv 0]:flash:/c3750e-universalk9-mz.150-2.SE10a.bin
# ^^^^^^^^^^^^^^^^
expect "TACACS Password:"
send "$pw\r"
interact
Then write a simple Bash for loop:
for host in $(<ioshost); do
expect /your/script.exp $host
done
How can I resume pull when disconnected? The pull process always start from the beginning every time I run docker pull some-image again after disconnected. My connection is so unstable that even downloading just a 100MB image take so long and almost fails every time. So, it is almost impossible for me to pull a bigger image. So, how can I resume the pull process?
Update:
The pull process will now automatically resume based on which layers have already been downloaded. This was implemented with https://github.com/moby/moby/pull/18353.
Old:
There is no resume feature yet. However there are discussions around this feature being implemented with docker's download manager.
Docker's code isn't as updated as the moby in development repository on github. People have been having issues for several years relating to this. I had tried to manually use several patches which aren't in the upstream yet, and none worked decent.
The github repository for moby (docker's development repo) has a script called download-frozen-image-v2.sh. This script uses bash, curl, and other things like JSON interpreters via command line. It will retrieve a docker token, and then download all of the layers to a local directory. You can then use 'docker load' to insert into your local docker installation.
It does not do well with resume though. It had some comment in the script relating to 'curl -C' isn't working. I had tracked down, and fixed this problem. I made a modification which uses a ".headers" file to retrieve initially, which has always returned a 302 while I've been monitoring, and then retrieves the final using curl (+ resume support) to the layer tar file. It also has to loop on the calling function which retrieves a valid token which unfortunately only lasts about 30 minutes.
It will loop this process until it receives a 416 stating that there is no resume possible since it's ranges have been fulfilled. It also verifies the size against a curl header retrieval. I have been able to retrieve all images necessary using this modified script. Docker has many more layers relating to retrieval, and has remote control processes (Docker client) which make it more difficult to control, and they viewed this issue as only affecting some people on bad connections.
I hope this script can help you as much as it has helped me:
Changes:
fetch_blob function uses a temporary file for its first connection. It then retrieves 30x HTTP redirect from this. It attempts a header retrieval on the final url and checks whether the local copy has the full file. Otherwise, it will begin a resume curl operation. The calling function which passes it a valid token has a loop surrounding retrieving a token, and fetch_blob which ensures the full file is obtained.
The only other variation is a bandwidth limit variable which can be set at the top, or via "BW:10" command line parameter. I needed this to allow my connection to be viable for other operations.
Click here for the modified script.
In the future it would be nice if docker's internal client performed resuming properly. Increasing the amount of time for the token's validation would help tremendously..
Brief views of change code:
#loop until FULL_FILE is set in fetch_blob.. this is for bad/slow connections
while [ "$FULL_FILE" != "1" ];do
local token="$(curl -fsSL "$authBase/token?service=$authService&scope=repository:$image:pull" | jq --raw-output '.token')"
fetch_blob "$token" "$image" "$layerDigest" "$dir/$layerTar" --progress
sleep 1
done
Another section from fetch_blob:
while :; do
#if the file already exists.. we will be resuming..
if [ -f "$targetFile" ];then
#getting current size of file we are resuming
CUR=`stat --printf="%s" $targetFile`
#use curl to get headers to find content-length of the full file
LEN=`curl -I -fL "${curlArgs[#]}" "$blobRedirect"|grep content-length|cut -d" " -f2`
#if we already have the entire file... lets stop curl from erroring with 416
if [ "$CUR" == "${LEN//[!0-9]/}" ]; then
FULL_FILE=1
break
fi
fi
HTTP_CODE=`curl -w %{http_code} -C - --tr-encoding --compressed --progress-bar -fL "${curlArgs[#]}" "$blobRedirect" -o "$targetFile"`
if [ "$HTTP_CODE" == "403" ]; then
#token expired so the server stopped allowing us to resume, lets return without setting FULL_FILE and itll restart this func w new token
FULL_FILE=0
break
fi
if [ "$HTTP_CODE" == "416" ]; then
FULL_FILE=1
break
fi
sleep 1
done
Try this
ps -ef | grep docker
Get PID of all the docker pull command and do a kill -9 on them. Once killed, re-issue the docker pull <image>:<tag> command.
This worked for me!
I am working on a ruby on rails application.I want to ping the URL with a given timeout and interval.I was able to get status code using curl and even with net/http without specifying the timeout.
This works fine without timeout :
%x{curl -sL -w "%{http_code}" "URl" -o /dev/null}
Can I know if there is any other way or modification to this
One can request only the headers using HTTP HEAD, as option -I in curl(1).
$ curl -I /
Lengthy HTML response bodies are a pain to get in command-line, so I'd like to get only the header as feedback for my POST requests. However, HEAD and POST are two different methods.
How do I get cURL to display only response headers to a POST request?
-D, --dump-header <file>
Write the protocol headers to the specified file.
This option is handy to use when you want to store the headers
that a HTTP site sends to you. Cookies from the headers could
then be read in a second curl invocation by using the -b,
--cookie option! The -c, --cookie-jar option is however a better
way to store cookies.
and
-S, --show-error
When used with -s, --silent, it makes curl show an error message if it fails.
from the man page. so
curl -sS -D - www.acooke.org -o /dev/null
follows redirects, dumps the headers to stdout and sends the data to /dev/null (that's a GET, not a POST, but you can do the same thing with a POST - just add whatever option you're already using for POSTing data)
note the - after the -D which indicates that the output "file" is stdout.
The other answers require the response body to be downloaded. But there's a way to make a POST request that will only fetch the header:
curl -s -I -X POST http://www.google.com
An -I by itself performs a HEAD request which can be overridden by -X POST to perform a POST (or any other) request and still only get the header data.
The Following command displays extra informations
curl -X POST http://httpbin.org/post -v > /dev/null
You can ask server to send just HEAD, instead of full response
curl -X HEAD -I http://httpbin.org/
Note: In some cases, server may send different headers for POST and HEAD. But in almost all cases headers are same.
For long response bodies (and various other similar situations), the solution I use is always to pipe to less, so
curl -i https://api.github.com/users | less
or
curl -s -D - https://api.github.com/users | less
will do the job.
Maybe it is little bit of an extreme, but I am using this super short version:
curl -svo. <URL>
Explanation:
-v print debug information (which does include headers)
-o. send web page data (which we want to ignore) to a certain file, . in this case, which is a directory and is an invalid destination and makes the output to be ignored.
-s no progress bar, no error information (otherwise you would see Warning: Failed to create the file .: Is a directory)
warning: result always fails (in terms of error code, if reachable or not). Do not use in, say, conditional statements in shell scripting...
Much easier – this also follows links.
curl -IL http://example.com/in-the-shadows
While the other answers have not worked for me in all situations, the best solution I could find (working with POST as well), taken from here:
curl -vs 'https://some-site.com' 1> /dev/null
headcurl.cmd (windows version)
curl -sSkv -o NUL %* 2>&1
I don't want a progress bar -s,
but I do want errors -S,
not bothering about valid https certificates -k,
getting high verbosity -v (this is about troubleshooting, is it?),
no output (in a clean way).
oh, and I want to forward stderr to stdout, so I can grep against the whole thing (since most or all output comes in stderr)
%* means [pass on all parameters to this script] (well(https://stackoverflow.com/a/980372/444255), well usually that's just one parameter: the url you are testing
real-world example (on troubleshooting proxy issues):
C:\depot>headcurl google.ch | grep -i -e http -e cache
Hostname was NOT found in DNS cache
GET HTTP://google.ch/ HTTP/1.1
HTTP/1.1 301 Moved Permanently
Location: http://www.google.ch/
Cache-Control: public, max-age=2592000
X-Cache: HIT from company.somewhere.ch
X-Cache-Lookup: HIT from company.somewhere.ch:1234
Linux version
for your .bash_aliases / .bash_rc:
alias headcurl='curl -sSkv -o /dev/null $# 2>&1'
-D, --dump-header
Write the protocol headers to the specified file.
This option is handy to use when you want to store the headers
that a HTTP site sends to you. Cookies from the headers could
then be read in a second curl invocation by using the -b,
--cookie option! The -c, --cookie-jar option is however a better
way to store cookies.