Got a Curl question (and yeah, i've searched Google, this site, as well as the docs!!)
Somewhat new to curl. Trying to use Curl from the cmdline (linux) to fetch pages from a college website (sjsu). I can somewhat get the 1st page, but the 2nd/2rd pages use referers/cookies.. I think this is where things are screwed up..
I'm also dealing with fetching data from within frames on the given page. If I analyze the page, in combination with the livehttpheader data, this shouldn't be an issue.
I've tried using cookie-jar, as well as cookie, in various combinations with the "-e" for the referer, and the "-d" for the posting data...
I'm posting the data from the livehttpheader process for the pages, as well as the test shell script i'm using.. Any thoughts/pointers would be greatly appreciated...
Thanks
-bruce
** this is the data that is displayed, when i select the "submit" btn on the location/term page to get to the "class select" page" as i understand the process, the Curl function should more or less replicate this process, with the referer, the postdata, and the target http/url
livehttpheaders data:
https://cmshr.sjsu.edu/psc/HSJPRDF/EMPLOYEE/HSJPRD/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL
POST /psc/HSJPRDF/EMPLOYEE/HSJPRD/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL HTTP/1.1
Host: cmshr.sjsu.edu
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.11) Gecko/2009061118 Fedora/3.0.11-1.fc9 Firefox/3.0.11
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: https://cmshr.sjsu.edu/psc/HSJPRDF/EMPLOYEE/HSJPRD/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL?FolderPath=PORTAL_ROOT_OBJECT.PA_HC_CLASS_SEARCH&PortalActualURL=https%3a%2f%2fcmshr.sjsu.edu%2fpsc%2fHSJPRDF%2fEMPLOYEE%2fHSJPRD%2fc%2fCOMMUNITY_ACCESS.CLASS_SEARCH.GBL&PortalRegistryName=EMPLOYEE&PortalServletURI=https%3a%2f%2fcmshr.sjsu.edu%2fpsp%2fHSJPRDF%2f&PortalURI=https%3a%2f%2fcmshr.sjsu.edu%2fpsc%2fHSJPRDF%2f&PortalHostNode=HRMS&NoCrumbs=yes
Cookie: I4Web_uuid=ece1faf2b59ace0ba40a4849c80e1d1b#e18f35cc4*0339838ad3db6ba9a32396e947e5d0d6#1418f55cc4*8bc80f3fe499835a27c78d5593811a3a#3818fc5cc4*166fa2936446e617426d945230e89e49#1e19045cc4*88210f324b634276bce6758d896eca38#8195a5cc4*d1da10f4207bca1bd889fea53873c471#3c1baf5cc4*d56e90391bc1ab51115f5c448ac2a124#71be65cc4*87ccd281d6940c53a49366c366796911#3e1bee5cc4*bb23f7cd74e303c96a11b4fdec5b7646#812a35cc6*84f727d9a8f7a11871756dad0c00d545#1a3b445cb4*; cssln164HSJ1-84915=B1h9LG2L1yydzpj3TvqQrHBHKdbWpnl8!695030755; ExpirePage=https://cmshr.sjsu.edu/psp/HSJPRDF/; PS_LOGINLIST=https://cmshr.sjsu.edu/HSJPRDF; PS_TOKENEXPIRE=9_Jan_2010_18:06:54_GMT; SignOnDefault=CMSPUBLIC; cssln169HSJ1-84915=JpgTLL9RJVQPWjphl3Vybx1xnvW8zbMl!-1963651055; cssln162HSJ1-84915=1LrQLH7b20GmB0kx50cf54GPGjj97WWM!813942549; cssln118HSJ1-84915=xbDyLLFZZCTbpLFhJQpCNwNF2ppgjH4y!-923260117; PS_TOKEN=AAAApgECAwQAAQAAAAACvAAAAAAAAAAsAARTaGRyAgBOcQgAOAAuADEAMBQLu6wR/++AI1n7RObpon/kagLJUAAAAGYABVNkYXRhWnicHclNDkBADIbhd4ZYWbkHYYKMrSF+gghxEPdzOJ3plz5tWuBVOopRSOnPm+HYuTl56NlYcAkjB1PKLPdVPhdDjaGkks7D9FqxoqEQjdiEvQtpJZYfnXoL+Q==
Content-Type: application/x-www-form-urlencoded
Content-Length: 339
ICType=Panel&ICElementNum=0&ICStateNum=2&ICAction=CLASS_SRCH_WRK2_SSR_PB_SRCH%2457%24&ICXPos=0&ICYPos=0&ICFocus=&ICSaveWarningFilter=0&ICChanged=-1&ICResubmit=0&ICSID=HpLTZLhQFp4p&CLASS_SRCH_WRK2_INSTITUTION%2445%24=SJ000&CLASS_SRCH_STRM1=2102&CLASS_SRCH_WRK2_SSR_CLS_SRCH_TYPE%2458%24=06&CLASS_SRCH_WRK2_SSR_CLS_SRCH_TYPE%2458%24%24rad=06
HTTP/1.x 200 OK
Cache-Control: no-cache
Date: Sat, 09 Jan 2010 18:07:47 GMT
Content-Length: 21137
Content-Type: text/html; CHARSET=UTF-8
IgnorePortalRegisteredURL: 1
PortalRegisteredURL: https://cmshr.sjsu.edu/psc/HSJPRDF/EMPLOYEE/HSJPRD/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL
UsesPortalRelativeURL: true
X-Powered-By: Servlet/2.4 JSP/2.0
Set-Cookie: PS_TOKENEXPIRE=9_Jan_2010_18:07:47_GMT; domain=.sjsu.edu; path=/; secure
test Curl shell script:
=============================================
#!/bin/sh -v
#
# test shell for curl..
#
#curl --cookie lcookie.lwp --cookie-jar lcookie.lwp --output "ctest.dat" -L "http://my.sjsu.edu/"
#foo="http://my.sjsu.edu/"
#curl --cookie lcookie.lwp --cookie-jar lcookie.lwp --output "ctest.dat" -L "$foo"
#exi
curl -v --cookie-jar lcookie.lwp --output "ctest2.dat" https://cmshr.sjsu.edu/psp/HSJPRDF/EMPLOYEE/HSJPRD/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL?FolderPath=PORTAL_ROOT_OBJECT.PA_HC_CLASS_SEARCH
#exit
curl -v -A "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" --cookie-jar lcookie.lwp --output "ctest3.dat" -e "https://cmshr.sjsu.edu/psp/HSJPRDF/EMPLOYEE/HSJPRD/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL?FolderPath=PORTAL_ROOT_OBJECT.PA_HC_CLASS_SEARCH" -L "https://cmshr.sjsu.edu/psc/HSJPRDF/EMPLOYEE/HRMS/s/WEBLIB_PT_NAV.ISCRIPT1.FieldFormula.IScript_UniHeader_Frame?c=uA%2buCaKuiBh5DTZEFHMBvNKbD7XLjINl&FolderPath=PORTAL_ROOT_OBJECT.PA_HC_CLASS_SEARCH&PortalActualURL=https%3a%2f%2fcmshr.sjsu.edu%2fpsc%2fHSJPRDF%2fEMPLOYEE%2fHSJPRD%2fc%2fCOMMUNITY_ACCESS.CLASS_SEARCH.GBL&PortalRegistryName=EMPLOYEE&PortalServletURI=https%3a%2f%2fcmshr.sjsu.edu%2fpsp%2fHSJPRDF%2f&PortalURI=https%3a%2f%2fcmshr.sjsu.edu%2fpsc%2fHSJPRDF%2f&PortalHostNode=HRMS&PortalIsPagelet=true&NoCrumbs=yes"
#get the page with the term/location --- uses the get
curl -v -A "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.11) Gecko/2009061118 Fedora/3.0.11-1.fc9 Firefox/3.0.11" --cookie-jar lcookie.lwp --output "ctest4.dat" -e "https://cmshr.sjsu.edu/psp/HSJPRDF/EMPLOYEE/HSJPRD/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL?FolderPath=PORTAL_ROOT_OBJECT.PA_HC_CLASS_SEARCH" -L "https://cmshr.sjsu.edu/psc/HSJPRDF/EMPLOYEE/HSJPRD/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL?FolderPath=PORTAL_ROOT_OBJECT.PA_HC_CLASS_SEARCH&PortalActualURL=https%3a%2f%2fcmshr.sjsu.edu%2fpsc%2fHSJPRDF%2fEMPLOYEE%2fHSJPRD%2fc%2fCOMMUNITY_ACCESS.CLASS_SEARCH.GBL&PortalRegistryName=EMPLOYEE&PortalServletURI=https%3a%2f%2fcmshr.sjsu.edu%2fpsp%2fHSJPRDF%2f&PortalURI=https%3a%2f%2fcmshr.sjsu.edu%2fpsc%2fHSJPRDF%2f&PortalHostNode=HRMS&NoCrumbs=yes"
#
# the following two lines are attempts to get the page with the class display... neither one works.
# -instead, the output files are simply the same as the above page, with the location/term menu..
#
# the page should be a pae that lists a class schedule select menu..
#
#get the page with the search class menu... it's a post
curl -v -A "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.11) Gecko/2009061118 Fedora/3.0.11-1.fc9 Firefox/3.0.11" --cookie-jar lcookie.lwp --output "ctest5.dat" -e "Referer: https://cmshr.sjsu.edu/psc/HSJPRDF/EMPLOYEE/HSJPRD/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL?FolderPath=PORTAL_ROOT_OBJECT.PA_HC_CLASS_SEARCH&PortalActualURL=https%3a%2f%2fcmshr.sjsu.edu%2fpsc%2fHSJPRDF%2fEMPLOYEE%2fHSJPRD%2fc%2fCOMMUNITY_ACCESS.CLASS_SEARCH.GBL&PortalRegistryName=EMPLOYEE&PortalServletURI=https%3a%2f%2fcmshr.sjsu.edu%2fpsp%2fHSJPRDF%2f&PortalURI=https%3a%2f%2fcmshr.sjsu.edu%2fpsc%2fHSJPRDF%2f&PortalHostNode=HRMS&NoCrumbs=yes" -d "ICType=Panel&ICElementNum=0&ICStateNum=2&ICAction=CLASS_SRCH_WRK2_SSR_PB_SRCH%2457%24&ICXPos=0&ICYPos=0&ICFocus=&ICSaveWarningFilter=0&ICChanged=-1&ICResubmit=0&ICSID=HpLTZLhQFp4p&CLASS_SRCH_WRK2_INSTITUTION%2445%24=SJ000&CLASS_SRCH_STRM1=2102&CLASS_SRCH_WRK2_SSR_CLS_SRCH_TYPE%2458%24=06&CLASS_SRCH_WRK2_SSR_CLS_SRCH_TYPE%2458%24%24rad=06" -L "https://cmshr.sjsu.edu/psc/HSJPRDF/EMPLOYEE/HSJPRD/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL"
#get the page with the search class menu... it's a post
curl -v -A "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.11) Gecko/2009061118 Fedora/3.0.11-1.fc9 Firefox/3.0.11" --cookie lcookie.lwp --output "ctest6.dat" -e "Referer: https://cmshr.sjsu.edu/psc/HSJPRDF/EMPLOYEE/HSJPRD/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL" -d "ICType=Panel&ICElementNum=0&ICStateNum=2&ICAction=CLASS_SRCH_WRK2_SSR_PB_SRCH%2457%24&ICXPos=0&ICYPos=0&ICFocus=&ICSaveWarningFilter=0&ICChanged=-1&ICResubmit=0&ICSID=HpLTZLhQFp4p&CLASS_SRCH_WRK2_INSTITUTION%2445%24=SJ000&CLASS_SRCH_STRM1=2102&CLASS_SRCH_WRK2_SSR_CLS_SRCH_TYPE%2458%24=06&CLASS_SRCH_WRK2_SSR_CLS_SRCH_TYPE%2458%24%24rad=06" -L "https://cmshr.sjsu.edu/psc/HSJPRDF/EMPLOYEE/HSJPRD/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL"
Note that --cookie-jar only is for saving cookies. You have to use --cookie with the same file as argument if you want to read in the cookie file (as in your first two commented curl lines).
Related
I'm trying to learn about the Docker Engine restful API, v1.41, but that's a problem. I managed to use cURL successfully to send all HTTP requests concerning containers, images, volumes and network, but it's impossible to me to use them for exec instances.
I read in the official documentation that when I use the docker exec command, he behaves the same way as calling POST /containers/:id/exec and then POST /exec/:id/start.
Now, in order to execute docker exec -it my_container bash that's what I did and got:
[claudio#gulliver ~]$ curl -X POST --unix-socket /run/docker.sock\
http/containers/18d7dee7470d/exec\
-H "Content-Type: application/json"\
-d '{
"AttachStdin":true,
"AttachStdout": true,
"AttachStderr": true,
"Tty": true,
"Cmd":["bash"]
}'
{"Id": "853938b2621606f85c04409ec7e345b884d46b95985a4fca0e8ddf28e20e1f79"}
[claudio#gulliver ~]$ curl -X POST --unix-socket /run/docker.sock\
http/exec/853938b2621606f85c04409ec7e345b884d46b95985a4fca0e8ddf28e20e1f79/start\
-H "Content-Type: application/json"\
-d '{
"Detach":false,
"Tty":true
}' --verbose
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying /run/docker.sock:0...
* Connected to http () port 80 (#0)
> POST /exec/853938b2621606f85c04409ec7e345b884d46b95985a4fca0e8ddf28e20e1f79/start HTTP/1.1
> Host: http
> User-Agent: curl/7.85.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 28
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: application/vnd.docker.raw-stream
< Api-Version: 1.41
< Docker-Experimental: false
< Ostype: linux
< Server: Docker/20.10.21 (linux)
* no chunk, no close, no size. Assume close to signal end
<
And from now I cannot do anything, just interrupt the process.
Is there a way I don't know to interact with this?
Normally, e.g., for the alpine image, we obtain an auth token via:
curl -i "https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/alpine:pull"
Then we can use it to obtain the manifest from the registry:
curl -i -H "Authorization: Bearer $TOKEN" -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" https://registry-1.docker.io/v2/library/alpine/manifests/latest
When we replace library/alpine with a private repository of ours (ourcompany/ourrepo) obtaining a token still works, however, downloading the manifest results in:
HTTP/1.1 401 Unauthorized
Content-Type: application/json
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:ourcompany/ourrepo:pull",error="insufficient_scope"
Date: Tue, 26 May 2020 10:32:56 GMT
Content-Length: 168
Strict-Transport-Security: max-age=31536000
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"ourcompany/ourrepo","Action":"pull"}]}]}
How to circumvent this 401 error?
Do we need to obtain additional tokens? Send authentication credentials in addition? Do something completely differently?
You need to authenticate the call to /token using Basic Authentication
https://docs.docker.com/registry/spec/auth/jwt/#getting-a-bearer-token
Then from that you receive a Bearer token which you use as you did with a public repository. The /token endpoint only supports basic authentication.
curl -i -H "Authorization: Basic $BASIC_AUTH" "https://auth.docker.io/token?service=registry.docker.io&scope=repository:myprivaterepo/myprivateimage:pull"
curl -i -H "Authorization: Bearer $TOKEN" -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" https://registry-1.docker.io/v2/myprivaterepo/myprivateimage/manifests/latest
I have several containers that are running on my Centos7 VM and I would like to retrieve their CPU and Memory usage using the following command:
echo -e "GET /containers/(container_name)/stats HTTP/1.0\r\n" | \
nc -U /var/run/docker.sock
However, I just receive the following message without any statistics:
HTTP/1.0 200 OK
Server: Docker/1.10.3 (linux)
Date: Sun, 22 Jan 2017 15:53:49 GMT
Content-Type: text/plain; charset=utf-8
The "containers/(container_name)/top" command works fine.
Can you please help me to understand why I don't receive this container's statistics?
Command to use get the stats of the container:
curl -X GET http://127.0.0.1:6000/containers/<container_id>/stats
The stats will be displayed for every second.
Stats can be fetched only for running containers.
Refer this :
how to configure docker daemon port.
docker version: 1.11.2
curl version: 7.50.3 (x86_64-pc-linux-gnu) libcurl/7.50.3 OpenSSL/1.0.1e zlib/1.2.7
/usr/local/sbin/bin/curl --unix-socket /var/run/docker.sock http://images/json -v
* Trying /var/run/docker.sock...
* Connected to images (/var/run/docker.sock) port 80 (#0)
> GET /json HTTP/1.1
> Host: images
> User-Agent: curl/7.50.3
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Thu, 22 Sep 2016 06:11:52 GMT
< Content-Length: 19
<
404 page not found
* Curl_http_done: called premature == 0
* Connection #0 to host images left intact
Is there anything wrong with my docker daemon? How can I get the containers info from the docker unix-socket?
docker deamon is absolutely started.
I followed this page:https://docs.docker.com/engine/reference/api/docker_remote_api/#/v1-23-api-changes, its suggestion us to use curl 7.40 or later, command curl --unix-socket /var/run/docker.sock http:/containers/json. You can found that there is a unavild URL http:/containers/json in this command.
Then I download the newest curl 7.50.3, the key of this problem is the curl's version, we should exec like below:
curl --unix-socket /var/run/docker.sock http://localhost/images/json
More detail watch this page.https://superuser.com/questions/834307/can-curl-send-requests-to-sockets. Hope it help some other people who confused.
I am trying to upload Docker image (tarball) into private Docker registry using following API.
I am following the documentation here: http://docs.docker.com/registry/spec/api/
Step 1: Initiate the upload and get location URL
$ curl -v -X POST http://localhost:5000/v2/hello-world/blobs/uploads/ * About to connect() to localhost port 5000 (#0)
* Trying ::1...
* Connected to localhost (::1) port 5000 (#0)
> POST /v2/hello-world/blobs/uploads/ HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:5000
> Accept: */*
>
< HTTP/1.1 202 Accepted
< Content-Length: 0
< Docker-Distribution-Api-Version: registry/2.0
< Docker-Upload-Uuid: dd319793-f017-45b3-afe4-c8102363a8df
< Location: http://localhost:5000/v2/hello-world/blobs/uploads/dd319793-f017-45b3-afe4-c8102363a8df?_state=SB2605fFM_7KNkYTHjVrVQVT62dufwXNTw9QzO2_aRR7Ik5hbWUiOiJoZWxsby13b3JsZCIsIlVVSUQiOiJkZDMxOTc5My1mMDE3LTQ1YjMtYWZlNC1jODEwMjM2M2E4ZGYiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTUtMTAtMjdUMjI6MjU6MjEuMTM0MTI5MDNaIn0%3D
< Range: 0-0
< Date: Tue, 27 Oct 2015 22:25:21 GMT
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host localhost left intact
Step 2: Use location URL to upload the actual Docker image
$ curl -v -H "Content-type: application/octet-stream" -H "Content-Length: 13824" --data-binary #/tmp/hello-world.tar -X PUT -L "http://localhost:5000/v2/hello-world/blobs/uploads/dd319793-f017-45b3-afe4-c8102363a8df?_state=SB2605fFM_7KNkYTHjVrVQVT62dufwXNTw9QzO2_aRR7Ik5hbWUiOiJoZWxsby13b3JsZCIsIlVVSUQiOiJkZDMxOTc5My1mMDE3LTQ1YjMtYWZlNC1jODEwMjM2M2E4ZGYiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTUtMTAtMjdUMjI6MjU6MjEuMTM0MTI5MDNaIn0%3D&digest=tarsum.v2+sha256:97bbb955c700a6414fd48ae147986e9b42c0508c8a766cea61e7e3badf0f7dde"
* About to connect() to localhost port 5000 (#0)
* Trying ::1...
* Connected to localhost (::1) port 5000 (#0)
> PUT /v2/hello-world/blobs/uploads/dd319793-f017-45b3-afe4-c8102363a8df?_state=SB2605fFM_7KNkYTHjVrVQVT62dufwXNTw9QzO2_aRR7Ik5hbWUiOiJoZWxsby13b3JsZCIsIlVVSUQiOiJkZDMxOTc5My1mMDE3LTQ1YjMtYWZlNC1jODEwMjM2M2E4ZGYiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTUtMTAtMjdUMjI6MjU6MjEuMTM0MTI5MDNaIn0%3D&digest=tarsum.v2+sha256:97bbb955c700a6414fd48ae147986e9b42c0508c8a766cea61e7e3badf0f7dde HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:5000
> Accept: */*
> Content-type: application/octet-stream
> Content-Length: 13824
> Expect: 100-continue
>
* Done waiting for 100-continue
< HTTP/1.1 400 Bad Request
< Content-Type: application/json; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< Date: Tue, 27 Oct 2015 22:29:02 GMT
< Content-Length: 131
* HTTP error before end of send, stop sending
<
{"errors":[{"code":"DIGEST_INVALID","message":"provided digest did not match uploaded content","detail":"digest parsing failed"}]}
* Closing connection 0
Also, I am using sha256sum as follows:
$ sha256sum /tmp/hello-world.tar
97bbb955c700a6414fd48ae147986e9b42c0508c8a766cea61e7e3badf0f7dde /tmp/hello-world.tar
What I am possibly doing wrong here? How to get around DIGEST_INVALID error?
Here is a simple example that should get you on track:
local reponame=foo/bar
local uploadURL
local numBytes=10000000 # 10 Megabytes
uploadURL=$(curl -siL -X POST "https://registrydomain/v2/$reponame/blobs/uploads/" | grep 'Location:' | cut -d ' ' -f 2 | tr -d '[:space:]')
blobDigest="sha256:$(head -c $numBytes /dev/urandom | tee upload.tmp | shasum -a 256 | cut -d ' ' -f 1)"
echo "Uploading Blob of 10 Random Megabytes"
time curl -T upload.tmp --progress-bar "$uploadURL&digest=$blobDigest" > /dev/null