Related
basically i'm doing a curl and grepping some stuff.
But, i want set the output of this curl to a variable, to then use it on another curl.
e.g:
curl -u asd:asd http://zzz:123/aa/aa.aaa?cmd=ls | grep -B1 -E '<bbb>[4-7]\d{8,}' | grep yyy | tail -n 1 | sed -n -e 's/.*<xxx>\(.*\)<\/xxx>.*/\1/p')
but then I want set the output to a var and use it:
RUN aaa=$(previous curl) && curl -u asd:asd http://$aaa.com
tried with ${aaa}, with "$aaa", etc... didn't work. any solutions?
UPDATE:
something wrong is happening in previous curl 'cause doesn't return the value. probably for not doing the curl
I fear you will not be able to acheive this, because from my understanding RUN statement is to execute a command. To store value you'll have use SET.
For me the following workaround helped
RUN export aaa=$(curl -u asd:asd http://$aaa.com);echo aaa;
You can add the downsteam commands that will use the variable aaa towards right of the semicolon
I am trying to create a docker image to run a shell script as aws lambda function. As far as I understood, I need a runtime and a handler.
The default runtime (bootstrap) looks like this:
#!/bin/sh
set -euo pipefail
# Handler format: <script_name>.<bash_function_name>
#
# The script file <script_name>.sh must be located at the root of your
# function's deployment package, alongside this bootstrap executable.
source $(dirname "$0")/"$(echo $_HANDLER | cut -d. -f1).sh"
while true
do
# Request the next event from the Lambda runtime
HEADERS="$(mktemp)"
EVENT_DATA=$(curl -v -sS -LD "$HEADERS" -X GET "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/next")
INVOCATION_ID=$(grep -Fi Lambda-Runtime-Aws-Request-Id "$HEADERS" | tr -d '[:space:]' | cut -d: -f2)
# Execute the handler function from the script
RESPONSE=$($(echo "$_HANDLER" | cut -d. -f2) "$EVENT_DATA")
# Send the response to Lambda runtime
curl -v -sS -X POST "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/$INVOCATION_ID/response" -d "$RESPONSE"
done
The Dockerfile:
FROM public.ecr.aws/lambda/provided:al2
COPY bootstrap ${LAMBDA_RUNTIME_DIR}
COPY function.sh ${LAMBDA_TASK_ROOT}
RUN … (install some dependencies)
CMD ["function.handler"]
From what I've read, your supposed to set CMD to the handler. However this does not seem to work as the $_HANDLER variable is not known inside of the container.
I can't figure out what makes the difference, but this combination of Dockerfile and bootstrap works.
#!/bin/sh
set -euo pipefail
source "$LAMBDA_TASK_ROOT"/"$(echo $_HANDLER | cut -d. -f1).sh"
while true
do
# Request the next event from the Lambda runtime
HEADERS="$(mktemp)"
EVENT_DATA=$(curl -sS -LD "$HEADERS" -X GET "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/next")
INVOCATION_ID=$(grep -Fi Lambda-Runtime-Aws-Request-Id "$HEADERS" | tr -d '[:space:]' | cut -d: -f2)
# Execute the handler function from the script
RESPONSE=$($(echo "$_HANDLER" | cut -d. -f2) "$EVENT_DATA")
# Send the response to Lambda runtime
curl -sS -X POST "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/$INVOCATION_ID/response" -d "$RESPONSE"
done
FROM public.ecr.aws/lambda/provided:al2
COPY ./bootstrap ${LAMBDA_RUNTIME_DIR}
COPY ./function.sh ${LAMBDA_TASK_ROOT}
RUN chmod 755 ${LAMBDA_RUNTIME_DIR}/bootstrap ${LAMBDA_TASK_ROOT}/function.sh
RUN …
WORKDIR ${LAMBDA_TASK_ROOT}
CMD ["function.handler"]
I have a script that works fine in sh on a linux host as well as inside an alpine container. But when I try executing that using docker exec <containerID> sh -c "<script>" it misbehaves. The script's function is to output stuff similar to ps.
systick=$(getconf CLK_TCK); for c in /proc/*/cmdline; do d=$(dirname $c); name=$(grep Name: $d/status); pid=$(basename $d); uid=$(grep Uid: $d/status); uid=$(echo ${uid#Uid:} | xargs); uid=${uid%% *}; user=$(grep :$uid:[0-9] /etc/passwd); user=${user%%:*}; cmdline=$(cat $c|xargs -0 echo); starttime=$(($(awk '{print $22}' $d/stat) / systick)); uptime=$(awk '{print int($1)}' /proc/uptime); elapsed=$(($uptime-$starttime)); echo $pid $user $elapsed $cmdline; done
EDIT: sh -c "<script>" has the same behavior.
You are not able to run this script from docker exec because the variables will be interpolated before they sent to the container (i.e., you are going to get values from your local machine, not from within the container).
In order to run it as you wish, you need to replace $ with \$ for every occurrence of $ in your script.
What might work better is to put your script into a file, then map the file to a location within the container, using -v (i.e., -v script.sh:/path/to/script.sh), and call the script via docker exec /path/to/script.sh
Part 1: A Working Answer
A Working One-Liner (Quoted For Use By Docker)
getProcessDataDef='shellQuoteWordsDef='"'"'shellQuoteWords() { sq="'"'"'"'"'"'"'"'"'"; dq='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'; for arg; do printf "'"'"'"'"'"'"'"'"'%s'"'"'"'"'"'"'"'"' " "$(printf '"'"'"'"'"'"'"'"'%s\n'"'"'"'"'"'"'"'"' "$arg" | sed -e "s#${sq}#${sq}${dq}${sq}${dq}${sq}#g")"; done; printf '"'"'"'"'"'"'"'"'\n'"'"'"'"'"'"'"'"'; }'"'"'; shellQuoteNullSeparatedStream() { xargs -0 sh -c "${shellQuoteWordsDef};"'"'"' shellQuoteWords "$#"'"'"' _; }; getProcessData() { systick=$(getconf CLK_TCK); for c in /proc/*/cmdline; do d=${c%/*}; pid=${d##*/}; name=$(awk '"'"'/^Name:/ { print $2 }'"'"' <"$d"/status); uid=$(awk '"'"'/^Uid:/ { print $2 }'"'"' <"$d"/status); pwent=$(getent passwd "$uid"); user=${pwent%%:*}; cmdline=$(shellQuoteNullSeparatedStream <"$c"); starttime=$(awk -v systick="$systick" '"'"'{print int($22 / systick)}'"'"' "$d"/stat); uptime=$(awk '"'"'{print int($1)}'"'"' /proc/uptime); elapsed=$((uptime-starttime)); echo "$pid $user $elapsed $cmdline"; done; }; getProcessData'
sh -c "$getProcessDataDef" # or docker exec <container> sh -c "$getProcessDataDef"
A Working One-Liner (Before Quoting/Escaping)
shellQuoteWordsDef='shellQuoteWords() { sq="'"'"'"; dq='"'"'"'"'"'; for arg; do printf "'"'"'%s'"'"' " "$(printf '"'"'%s\n'"'"' "$arg" | sed -e "s#${sq}#${sq}${dq}${sq}${dq}${sq}#g")"; done; printf '"'"'\n'"'"'; }'; shellQuoteNullSeparatedStream() { xargs -0 sh -c "${shellQuoteWordsDef};"' shellQuoteWords "$#"' _; }; getProcessData() { systick=$(getconf CLK_TCK); for c in /proc/*/cmdline; do d=${c%/*}; pid=${d##*/}; name=$(awk '/^Name:/ { print $2 }' <"$d"/status); uid=$(awk '/^Uid:/ { print $2 }' <"$d"/status); pwent=$(getent passwd "$uid"); user=${pwent%%:*}; cmdline=$(shellQuoteNullSeparatedStream <"$c"); starttime=$(awk -v systick="$systick" '{print int($22 / systick)}' "$d"/stat); uptime=$(awk '{print int($1)}' /proc/uptime); elapsed=$((uptime-starttime)); echo "$pid $user $elapsed $cmdline"; done; }; getProcessData "$#"
What Went Into That One-Liner
shellQuoteWordsDef='shellQuoteWords() { sq="'"'"'"; dq='"'"'"'"'"'; for arg; do printf "'"'"'%s'"'"' " "$(printf '"'"'%s\n'"'"' "$arg" | sed -e "s#${sq}#${sq}${dq}${sq}${dq}${sq}#g")"; done; printf '"'"'\n'"'"'; }'
shellQuoteNullSeparatedStream() {
xargs -0 sh -c "${shellQuoteWordsDef};"' shellQuoteWords "$#"' _
}
getProcessData() {
systick=$(getconf CLK_TCK)
for c in /proc/*/cmdline; do
d=${c%/*}; pid=${d##*/}
name=$(awk '/^Name:/ { print $2 }' <"$d"/status)
uid=$(awk '/^Uid:/ { print $2 }' <"$d"/status)
pwent=$(getent passwd "$uid")
user=${pwent%%:*}
cmdline=$(shellQuoteNullSeparatedStream <"$c")
starttime=$(awk -v systick="$systick" '{print int($22 / systick)}' "$d"/stat)
uptime=$(awk '{print int($1)}' /proc/uptime)
elapsed=$((uptime-starttime))
echo "$pid $user $elapsed $cmdline"
done
}
What Went Into The Shell-Quoting Helper Used By That One-Liner
To allow easier reading and editing, the function stringified above looks like:
# This is the function we're including in our code passed to xargs in-band above:
shellQuoteWords() {
sq="'"; dq='"'
for arg; do
printf "'%s' " "$(printf '%s\n' "$arg" | sed -e "s#${sq}#${sq}${dq}${sq}${dq}${sq}#g")"
done
printf '\n'
}
Part 2: How That Answer Was Created
Python has an excellent shlex.quote() function (or pipes.quote() in Python 2) that can be used to generate a shell-quoted version of a string. In this context, that can be used as follows:
Python 3.7.6 (default, Feb 27 2020, 15:15:00)
[Clang 7.1.0 (tags/RELEASE_710/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> s = r'''
... shellQuoteWords() {
... sq="'"; dq='"'
... for arg; do
... printf "'%s' " "$(printf '%s\n' "$arg" | sed -e "s#${sq}#${sq}${dq}${sq}${dq}${sq}#g")"
... done
... printf '\n'
... }
... '''
>>> import shlex
>>> print(shlex.quote(s))
'
shellQuoteWords() {
sq="'"'"'"; dq='"'"'"'"'"'
for arg; do
printf "'"'"'%s'"'"' " "$(printf '"'"'%s\n'"'"' "$arg" | sed -e "s#${sq}#${sq}${dq}${sq}${dq}${sq}#g")"
done
printf '"'"'\n'"'"'
}
'
That result is itself a perfectly valid string in shell. That is to say, one can run:
s='
shellQuoteWords() {
sq="'"'"'"; dq='"'"'"'"'"'
for arg; do
printf "'"'"'%s'"'"' " "$(printf '"'"'%s\n'"'"' "$arg" | sed -e "s#${sq}#${sq}${dq}${sq}${dq}${sq}#g")"
done
printf '"'"'\n'"'"'
}
'
eval "$s"
shellQuoteWords "hello world" 'hello world' "hello 'world'" 'hello "world"'
...and get completely valid output.
The same process was followed to generate a string that evaluated to the definition of getProcessData.
I'm trying to automate a JMX command using a tool called JMXterm.
The script looks like so:
#!/bin/bash
beanstemp="/tmp/jmx_beans"
read -r -p "Enter server name in FQDN " scrapername
bean1="BrokerName=localhost,Connection=Scraper_$scrapername,ConnectorName=openwire,Type=Connection"
echo beans -d $domain | $cmd > $beanstemp
idnum=$(grep "ID_$scrapername" $beanstemp | grep openwire | awk -F- '{print $2"-"$3}')
idname="$scrapername-$idnum"
bean2="BrokerName=localhost,Connection=ID_"$idname"2_0,ConnectorName=openwire,Type=Connection"
domain="org.apache.activemq"
server="scrapermq.sj.peer39.com:1099"
cmd="java -jar Downloads/jmxterm-1.0-alpha-4-uber.jar -l $server"
echo run stop -b $bean1 -d $domain | $cmd -l $server
echo run stop -b $bean2 -d $domain | $cmd -l $server
When I run the script I get the proper response for the first command but the second one fails.
Here's the output of sh -x:
itaig#iganot-lt:~$ sh -x jmx.sh
+ beanstemp=/tmp/jmx_beans
+ read -r -p Enter server name in FQDN scrapername
Enter server name in FQDN selvmnj27.domain.company.com
+ bean1=BrokerName=localhost,Connection=Scraper_selvmnj27.domain.company.com,ConnectorName=openwire,Type=Connection
+ echo beans -d
+
+ + grep openwire
grep ID_selvmnj27.domain.company.com+ awk -F- {print $2"-"$3}
/tmp/jmx_beans
+ idnum=
+ idname=selvmnj27.domain.company.com-
+ bean2=BrokerName=localhost,Connection=ID_selvmnj27.domain.company.com-2_0,ConnectorName=openwire,Type=Connection
+ domain=org.apache.activemq
+ server=scrapermq.sj.peer39.com:1099
+ cmd=java -jar Downloads/jmxterm-1.0-alpha-4-uber.jar -l scrapermq.sj.peer39.com:1099
+ echo+ run stop -b BrokerName=localhost,Connection=Scraper_selvmnj27.domain.company.com,ConnectorName=openwire,Type=Connectionjava -jar -d Downloads/jmxterm-1.0-alpha-4-uber.jar -l org.apache.activemq scrapermq.sj.peer39.com:1099 -l
scrapermq.sj.peer39.com:1099
Welcome to JMX terminal. Type "help" for available commands.
$>run stop -b BrokerName=localhost,Connection=Scraper_selvmnj27.domain.company.com,ConnectorName=openwire,Type=Connection -d org.apache.activemq
#calling operation stop of mbean org.apache.activemq:BrokerName=localhost,Connection=Scraper_selvmnj27.domain.company.com,ConnectorName=openwire,Type=Connection
#operation returns:
null
$>+ + echo runjava stop -b -jar Downloads/jmxterm-1.0-alpha-4-uber.jar BrokerName=localhost,Connection=ID_selvmnj27.domain.company.com-2_0,ConnectorName=openwire,Type=Connection -l scrapermq.sj.peer39.com:1099 -l -d scrapermq.sj.peer39.com:1099
org.apache.activemq
Welcome to JMX terminal. Type "help" for available commands.
$>run stop -b BrokerName=localhost,Connection=ID_selvmnj27.domain.company.com-2_0,ConnectorName=openwire,Type=Connection -d org.apache.activemq
#InstanceNotFoundException: org.apache.activemq:BrokerName=localhost,Connection=ID_selvmnj27.domain.company.com-2_0,ConnectorName=openwire,Type=Connection
$>itaig#iganot-lt:~$
"Operation returns: null" is the response I'm looking for.
The problem begins with this line:
echo beans -d $domain | $cmd > $beanstemp
When it runs the output of the command should be written to the $beanstemp file but the file stays empty.
When I run the command manually from terminal it works:
itaig#iganot-lt:~$ echo beans -d org.apache.activemq | java -jar Downloads/jmxterm-1.0-alpha-4-uber.jar -l scrapermq.domain.company.com:1099 > /tmp/jmx_beans
Welcome to JMX terminal. Type "help" for available commands.
$>beans -d org.apache.activemq
#domain = org.apache.activemq:
$>itaig#iganot-lt:~$ tail -5 /tmp/jmx_beans
org.apache.activemq:BrokerName=localhost,Type=Subscription,clientId=Scraper_selvmnj28.domain.company.com,consumerId=ID_selvmnj28.domain.company.com-45117-1431866382308-0_416_1_1,destinationName=SCRAPER_VIDEO_INPUT,destinationType=Queue,persistentMode=Non-Durable
org.apache.activemq:BrokerName=localhost,Type=Subscription,clientId=Scraper_selvmnj29.domain.company.com,consumerId=ID_selvmnj29.domain.company.com-52961-1431866382264-0_416_-1_1,destinationName=topic_//ActiveMQ.Advisory.TempQueue_topic_//ActiveMQ.Advisory.TempTopic,destinationType=Topic,persistentMode=Non-Durable
org.apache.activemq:BrokerName=localhost,Type=Subscription,clientId=Scraper_selvmnj29.domain.company.com,consumerId=ID_selvmnj29.domain.company.com-52961-1431866382264-0_416_1_1,destinationName=SCRAPER_VIDEO_INPUT,destinationType=Queue,persistentMode=Non-Durable
org.apache.activemq:BrokerName=localhost,Type=Subscription,clientId=Scraper_selvmnj30.domain.company.com,consumerId=ID_selvmnj30.domain.company.com-33036-1431866396456-0_416_-1_1,destinationName=topic_//ActiveMQ.Advisory.TempQueue_topic_//ActiveMQ.Advisory.TempTopic,destinationType=Topic,persistentMode=Non-Durable
org.apache.activemq:BrokerName=localhost,Type=Subscription,clientId=Scraper_selvmnj30.domain.company.com,consumerId=ID_selvmnj30.domain.company.com-33036-1431866396456-0_416_1_1,destinationName=SCRAPER_VIDEO_INPUT,destinationType=Queue,persistentMode=Non-Durable
itaig#iganot-lt:~$
So my question is: What am I doing wrong and why does the command work when running manually from terminal but not from the script?
Thanks in advance
Ok I found the problem.
Certain lines included variables which have not been declared in time.
After re-organizing the order of variable declaration the problem has been solved.
I try to locate one specific tag for a Docker image. How can I do it on the command line? I want to avoid downloading all the images and then removing the unneeded ones.
In the official Ubuntu release, https://registry.hub.docker.com/_/ubuntu/, there are several tags (release for it), while when I search it on the command line,
user#ubuntu:~$ docker search ubuntu | grep ^ubuntu
ubuntu Official Ubuntu base image 354
ubuntu-upstart Upstart is an event-based replacement for ... 7
ubuntufan/ping 0
ubuntu-debootstrap 0
Also in the help of command line search https://docs.docker.com/engine/reference/commandline/search/, no clue how it can work?
Is it possible in the docker search command?
If I use a raw command to search via the Docker registry API, then the information can be fetched:
$ curl https://registry.hub.docker.com//v1/repositories/ubuntu/tags | python -mjson.tool
[
{
"layer": "ef83896b",
"name": "latest"
},
.....
{
"layer": "463ff6be",
"name": "raring"
},
{
"layer": "195eb90b",
"name": "saucy"
},
{
"layer": "ef83896b",
"name": "trusty"
}
]
When using CoreOS, jq is available to parse JSON data.
So like you were doing before, looking at library/centos:
$ curl -s -S 'https://registry.hub.docker.com/v2/repositories/library/centos/tags/' | jq '."results"[]["name"]' |sort
"6"
"6.7"
"centos5"
"centos5.11"
"centos6"
"centos6.6"
"centos6.7"
"centos7.0.1406"
"centos7.1.1503"
"latest"
The cleaner v2 API is available now, and that's what I'm using in the example. I will build a simple script docker_remote_tags:
#!/usr/bin/bash
curl -s -S "https://registry.hub.docker.com/v2/repositories/library/$#/tags/" | jq '."results"[]["name"]' |sort
Enables:
$ ./docker_remote_tags library/centos
"6"
"6.7"
"centos5"
"centos5.11"
"centos6"
"centos6.6"
"centos6.7"
"centos7.0.1406"
"centos7.1.1503"
"latest"
Reference:
jq: https://stedolan.github.io/jq/ | apt-get install jq
I didn't like any of the solutions above because A) they required external libraries that I didn't have and didn't want to install. B) I didn't get all the pages.
The Docker API limits you to 100 items per request. This will loop over each "next" item and get them all (for Python it's seven pages; other may be more or less... It depends)
If you really want to spam yourself, remove | cut -d '-' -f 1 from the last line, and you will see absolutely everything.
url=https://registry.hub.docker.com/v2/repositories/library/redis/tags/?page_size=100 `# Initial url` ; \
( \
while [ ! -z $url ]; do `# Keep looping until the variable url is empty` \
>&2 echo -n "." `# Every iteration of the loop prints out a single dot to show progress as it got through all the pages (this is inline dot)` ; \
content=$(curl -s $url | python -c 'import sys, json; data = json.load(sys.stdin); print(data.get("next", "") or ""); print("\n".join([x["name"] for x in data["results"]]))') `# Curl the URL and pipe the output to Python. Python will parse the JSON and print the very first line as the next URL (it will leave it blank if there are no more pages) then continue to loop over the results extracting only the name; all will be stored in a variable called content` ; \
url=$(echo "$content" | head -n 1) `# Let's get the first line of content which contains the next URL for the loop to continue` ; \
echo "$content" | tail -n +2 `# Print the content without the first line (yes +2 is counter intuitive)` ; \
done; \
>&2 echo `# Finally break the line of dots` ; \
) | cut -d '-' -f 1 | sort --version-sort | uniq;
Sample output:
$ url=https://registry.hub.docker.com/v2/repositories/library/redis/tags/?page_size=100 `#initial url` ; \
> ( \
> while [ ! -z $url ]; do `#Keep looping until the variable url is empty` \
> >&2 echo -n "." `#Every iteration of the loop prints out a single dot to show progress as it got through all the pages (this is inline dot)` ; \
> content=$(curl -s $url | python -c 'import sys, json; data = json.load(sys.stdin); print(data.get("next", "") or ""); print("\n".join([x["name"] for x in data["results"]]))') `# Curl the URL and pipe the JSON to Python. Python will parse the JSON and print the very first line as the next URL (it will leave it blank if there are no more pages) then continue to loop over the results extracting only the name; all will be store in a variable called content` ; \
> url=$(echo "$content" | head -n 1) `#Let's get the first line of content which contains the next URL for the loop to continue` ; \
> echo "$content" | tail -n +2 `#Print the content with out the first line (yes +2 is counter intuitive)` ; \
> done; \
> >&2 echo `#Finally break the line of dots` ; \
> ) | cut -d '-' -f 1 | sort --version-sort | uniq;
...
2
2.6
2.6.17
2.8
2.8.6
2.8.7
2.8.8
2.8.9
2.8.10
2.8.11
2.8.12
2.8.13
2.8.14
2.8.15
2.8.16
2.8.17
2.8.18
2.8.19
2.8.20
2.8.21
2.8.22
2.8.23
3
3.0
3.0.0
3.0.1
3.0.2
3.0.3
3.0.4
3.0.5
3.0.6
3.0.7
3.0.504
3.2
3.2.0
3.2.1
3.2.2
3.2.3
3.2.4
3.2.5
3.2.6
3.2.7
3.2.8
3.2.9
3.2.10
3.2.11
3.2.100
4
4.0
4.0.0
4.0.1
4.0.2
4.0.4
4.0.5
4.0.6
4.0.7
4.0.8
32bit
alpine
latest
nanoserver
windowsservercore
If you want the bash_profile version:
function docker-tags () {
name=$1
# Initial URL
url=https://registry.hub.docker.com/v2/repositories/library/$name/tags/?page_size=100
(
# Keep looping until the variable URL is empty
while [ ! -z $url ]; do
# Every iteration of the loop prints out a single dot to show progress as it got through all the pages (this is inline dot)
>&2 echo -n "."
# Curl the URL and pipe the output to Python. Python will parse the JSON and print the very first line as the next URL (it will leave it blank if there are no more pages)
# then continue to loop over the results extracting only the name; all will be stored in a variable called content
content=$(curl -s $url | python -c 'import sys, json; data = json.load(sys.stdin); print(data.get("next", "") or ""); print("\n".join([x["name"] for x in data["results"]]))')
# Let's get the first line of content which contains the next URL for the loop to continue
url=$(echo "$content" | head -n 1)
# Print the content without the first line (yes +2 is counter intuitive)
echo "$content" | tail -n +2
done;
# Finally break the line of dots
>&2 echo
) | cut -d '-' -f 1 | sort --version-sort | uniq;
}
And simply call it: docker-tags redis
Sample output:
$ docker-tags redis
...
2
2.6
2.6.17
2.8
--trunc----
32bit
alpine
latest
nanoserver
windowsservercore
As far as I know, the CLI does not allow searching/listing tags in a repository.
But if you know which tag you want, you can pull that explicitly by adding a colon and the image name: docker pull ubuntu:saucy
This script (docker-show-repo-tags.sh) should work for any Docker enabled host that has curl, sed, grep, and sort. This was updated to reflect the fact the repository tag URLs changed.
This version correctly parses the "name": field without a JSON parser.
#!/bin/sh
# 2022-07-20
# Simple script that will display Docker repository tags
# using basic tools: curl, awk, sed, grep, and sort.
# Usage:
# $ docker-show-repo-tags.sh ubuntu centos
# $ docker-show-repo-tags.sh centos | cat -n
for Repo in "$#" ; do
URL="https://registry.hub.docker.com/v2/repositories/library/$Repo/tags/"
curl -sS "$URL" | \
/usr/bin/sed -Ee 's/("name":)"([^"]*)"/\n\1\2\n/g' | \
grep '"name":' | \
awk -F: '{printf("'$Repo':%s\n",$2)}'
done
This older version no longer works. Many thanks to #d9k for pointing this out!
#!/bin/sh
# WARNING: This no long works!
# Simple script that will display Docker repository tags
# using basic tools: curl, sed, grep, and sort.
#
# Usage:
# $ docker-show-repo-tags.sh ubuntu centos
for Repo in $* ; do
curl -sS "https://hub.docker.com/r/library/$Repo/tags/" | \
sed -e $'s/"tags":/\\\n"tags":/g' -e $'s/\]/\\\n\]/g' | \
grep '^"tags"' | \
grep '"library"' | \
sed -e $'s/,/,\\\n/g' -e 's/,//g' -e 's/"//g' | \
grep -v 'library:' | \
sort -fu | \
sed -e "s/^/${Repo}:/"
done
This older version no longer works. Many thanks to #viky for pointing this out!
#!/bin/sh
# WARNING: This no long works!
# Simple script that will display Docker repository tags.
#
# Usage:
# $ docker-show-repo-tags.sh ubuntu centos
for Repo in $* ; do
curl -s -S "https://registry.hub.docker.com/v2/repositories/library/$Repo/tags/" | \
sed -e $'s/,/,\\\n/g' -e $'s/\[/\\\[\n/g' | \
grep '"name"' | \
awk -F\" '{print $4;}' | \
sort -fu | \
sed -e "s/^/${Repo}:/"
done
This is the output for a simple example:
$ docker-show-repo-tags.sh centos | cat -n
1 centos:5
2 centos:5.11
3 centos:6
4 centos:6.10
5 centos:6.6
6 centos:6.7
7 centos:6.8
8 centos:6.9
9 centos:7.0.1406
10 centos:7.1.1503
11 centos:7.2.1511
12 centos:7.3.1611
13 centos:7.4.1708
14 centos:7.5.1804
15 centos:centos5
16 centos:centos5.11
17 centos:centos6
18 centos:centos6.10
19 centos:centos6.6
20 centos:centos6.7
21 centos:centos6.8
22 centos:centos6.9
23 centos:centos7
24 centos:centos7.0.1406
25 centos:centos7.1.1503
26 centos:centos7.2.1511
27 centos:centos7.3.1611
28 centos:centos7.4.1708
29 centos:centos7.5.1804
30 centos:latest
I wrote a command line tool to simplify searching Docker Hub repository tags, available in my PyTools GitHub repository. It's simple to use with various command line switches, but most basically:
./dockerhub_show_tags.py repo1 repo2
It's even available as a Docker image and can take multiple repositories:
docker run harisekhon/pytools dockerhub_show_tags.py centos ubuntu
DockerHub
repo: centos
tags: 5.11
6.6
6.7
7.0.1406
7.1.1503
centos5.11
centos6.6
centos6.7
centos7.0.1406
centos7.1.1503
repo: ubuntu
tags: latest
14.04
15.10
16.04
trusty
trusty-20160503.1
wily
wily-20160503
xenial
xenial-20160503
If you want to embed it in scripts, use -q / --quiet to get just the tags, like normal Docker commands:
./dockerhub_show_tags.py centos -q
5.11
6.6
6.7
7.0.1406
7.1.1503
centos5.11
centos6.6
centos6.7
centos7.0.1406
centos7.1.1503
The v2 API seems to use some kind of pagination, so that it does not return all the available tags. This is clearly visible in projects such as python (or library/python). Even after quickly reading the documentation, I could not manage to work with the API correctly (maybe it is the wrong documentation).
Then I rewrote the script using the v1 API, and it is still using jq:
#!/bin/bash
repo="$1"
if [[ "${repo}" != */* ]]; then
repo="library/${repo}"
fi
url="https://registry.hub.docker.com/v1/repositories/${repo}/tags"
curl -s -S "${url}" | jq '.[]["name"]' | sed 's/^"\(.*\)"$/\1/' | sort
The full script is available at: https://github.com/denilsonsa/small_scripts/blob/master/docker_remote_tags.sh
I've also written an improved version (in Python) that aggregates tags that point to the same version: https://github.com/denilsonsa/small_scripts/blob/master/docker_remote_tags.py
Add this function to your .zshrc file or run the command manually:
#usage list-dh-tags <repo>
#example: list-dh-tags node
function list-dh-tags(){
wget -q https://registry.hub.docker.com/v1/repositories/$1/tags -O - | sed -e 's/[][]//g' -e 's/"//g' -e 's/ //g' | tr '}' '\n' | awk -F: '{print $3}'
}
Thanks to this -> How can I list all tags for a Docker image on a remote registry?
For anyone stumbling across this in modern times, you can use Skopeo to retrieve an image's tags from the Docker registry:
$ skopeo list-tags docker://jenkins/jenkins \
| jq -r '.Tags[] | select(. | contains("lts-alpine"))' \
| sort --version-sort --reverse
lts-alpine
2.277.3-lts-alpine
2.277.2-lts-alpine
2.277.1-lts-alpine
2.263.4-lts-alpine
2.263.3-lts-alpine
2.263.2-lts-alpine
2.263.1-lts-alpine
2.249.3-lts-alpine
2.249.2-lts-alpine
2.249.1-lts-alpine
2.235.5-lts-alpine
2.235.4-lts-alpine
2.235.3-lts-alpine
2.235.2-lts-alpine
2.235.1-lts-alpine
2.222.4-lts-alpine
Reimplementation of the previous post, using Python over sed/AWK:
for Repo in $* ; do
tags=$(curl -s -S "https://registry.hub.docker.com/v2/repositories/library/$Repo/tags/")
python - <<EOF
import json
tags = [t['name'] for t in json.loads('''$tags''')['results']]
tags.sort()
for tag in tags:
print "{}:{}".format('$Repo', tag)
EOF
done
For a script that works with OAuth bearer tokens on Docker Hub, try this:
Listing the tags of a Docker image on a Docker hub through the HTTP API
You can use Visual Studio Code to provide autocomplete for available Docker images and tags. However, this requires that you type the first letter of a tag in order to see autocomplete suggestions.
For example, when writing FROM ubuntu it offers autocomplete suggestions like ubuntu, ubuntu-debootstrap and ubuntu-upstart. When writing FROM ubuntu:a it offers autocomplete suggestions, like ubuntu:artful and ubuntu:artful-20170511.1