Hashicorp Vault inject directory - devops

I want to inject a whole directory using the agent injector.
I would, firstly, like to know if this is even possible.
I will explain myself:
I have this secrets directory: /secret/dev/app/ and under app, I have aws/some_secrets, db/some_secrets, etc...
Is it possible to inject the app directory without having the full secret name?


I would say take a look at Agent Templates.
If you take a look at step 7 of the tutorial:
{{ with secret "secret/data/customers/acme" }}
Organization: {{ .Data.data.organization }}
ID: {{ .Data.data.customer_id }}
Contact: {{ .Data.data.contact_email }}
{{ end }}
You could simply template this template file with a script then run the agent. But your script that generates the dynamic template file would have to do some heavy lifting...
List all secrets under a KV v2 basepath (if the engine mount path has no / characters in it):
#!/usr/bin/env bash
listall() {
kv2opt="/metadata"
if [ "${1}" = "-kv2" ]; then
kv2opt="/metadata"
shift
elif [ "${1}" = "-kv1" ]; then
kv2opt=""
shift
fi
sarg=$(printf '%s' "${1}" | sed -E 's~/*$~~g' | sed -E 's~^/*~~g')
engine=$(printf '%s' "${sarg}" | cut -d/ -f1 )
if [ "$(printf '%s' "${sarg}" | cut -d/ -f2)" = "metadata" ]; then
vpath=$(printf '%s' "${sarg}" | sed -E "s~^${engine}/metadata/?~~g" )
else
vpath=$(printf '%s' "${sarg}" | sed -E "s~^${engine}/?~~g" )
fi
curl -s -H "X-Vault-Request: true" -H "X-Vault-Token: ${VAULT_TOKEN}" --request LIST \
"${VAULT_ADDR}/v1/${engine}${kv2opt}/${vpath}" | jq -rc '.data.keys[]' | while IFS= read -r li; do
if [ "${li: -1}" != "/" ]; then
printf "%s/%s\n" "${sarg}" "${li}"
else
listall "${sarg}/${li}"
fi
done
}
listall -kv2 "secret/dev/app" | while IFS= read -r path; do
cat << EOF >> template.tpl
{{ with secret "${path}" }}
${path}: {{ .Data.data }}
{{ end }}
EOF
done
...and then maybe run the resultant template.tpl file through the Vault Agent using the template process. But that's pretty useless if things have to be read by a machine after the template finishes, so you may need to have a new loop read each secret to figure out what the keys are on each secret. And then do some advanced formatting. However, the way you structured your question, this technically answers it, and you can figure out how to do the rest (or reframe your question, or ask a new question).

Related

Jenkins Pipeline: I cannot add a variable from a concatenated command in bash script

I have created several bash scripts that work perfect in the Linux shell, but when I try to incorporate them in a Jenkins Pipeline I get multiple errors, I attach an example of my Pipeline where I just want to show the value of my variables, the pipeline works fine except when I added in line 5 the environment, you can see that there are special characters that are not interpreted by Groovy as the Bash does.
pipeline {
agent {
label params.LABS == "any" ? "" : params.LABS
}
environment{
PORT_INSTANCE="${docker ps --format 'table {{ .Names }} \{{ .Ports }}' --filter expose=7000-8999/tcp | (read -r; printf "%s\n"; sort -k 3) | grep web | tail -1 | sed 's/.*0.0.0.0.0://g'|sed 's/->.*//g'}"
}
stages {
stage('Setup parameters') {
steps {
script {
properties([
parameters([
choice(
choices: ['LAB-2', 'LAB-3'],
name: 'LABS'
),
string(
defaultValue: 'cliente-1',
name: 'INSTANCE_NAME',
trim: true
),
string(
defaultValue: '8888',
name: 'PORT_NUMBER',
trim: true
),
string(
defaultValue: 'lab.domain.com',
name: 'DOMAIN_NAME',
trim: true
)
])
])
}
sh """
echo '${params.INSTANCE_NAME}'
echo '${params.PORT_NUMBER}'
echo '${params.DOMAIN_NAME}'
echo '${PORT_INSTANCE}
"""
}
}
}
}
I already tried the same thing from the sh section """ command """ and they throw the same errors.
Can someone help me to know how to run advanced commands that work in the linux shell (bash), that is, is there any way to migrate scripts from bash to Jenkins?
Thank you very much for your help ;)
I want to be able to create a variable from a bash script command from the Pipeline in Jenkins
PORT_INSTANCE="${docker ps --format 'table {{ .Names }} {{ .Ports }}' --filter expose=7000-8999/tcp | (read -r; printf "%s\n"; sort -k 3) | grep web | tail -1 | sed 's/.0.0.0.0.0://g'|sed 's/->.//g'}"

I believe that you can't execute a bash script in the environment step based on the documentation.
You can create a variable from a bash script using the sh step with returnStdout set to true. Declarative pipeline doesn't allow you to assign the retrun value to a variable, so you will need to call sh inside a script like this:
stage('Calculate port') {
steps {
script {
// When you don't use `def` in front of a variable, you implicitly create a global variable
// This means that the variable will exist with a value, and can be used in any following line in your scipt
PORT_INSTANCE = sh returnStdout: true, script: "docker ps --format 'table {{ .Names }} \{{ .Ports }}' --filter expose=7000-8999/tcp | (read -r; printf \"%s\n\"; sort -k 3) | grep web | tail -1 | sed 's/.*0.0.0.0.0://g'|sed 's/->.*//g'"
// Shell output will contain a new line character at the end, remove it
PORT_INSTANCE = PORT_INSTANCE.trim()
}
}
}
I would add a stage like this, as the first stage in my pipeline.
Note that I didn't run the same shell command as you when I was testing this, so my command may have issues like un-escaped quotes.

How to make sh_test depend on building docker image?

I have a sh_test invoking docker run my_image where my_image is produced by a container_bundle rule. I need the container_bundle rule to be run as dependency to sh_test. How to achieve that? Adding container_bundle to sh_test's data only invokes container_bundle build, but I need run which pushes an image to a docker registry.

What we do is pass the rootpath of the container image rule to the script (as $IMAGE_LOADER) and do:
$IMAGE_LOADER --norun | tee image-loader.log
IMAGE_ID=$(cat ./image-loader.log | grep "Loaded image ID" | cut -d":" -f2-)

The easiest way I know is wrapping a docker_push rule around your bundle. Then your test rule can run the docker_push's output file, which is a binary that will do the docker load. Use runfiles.bash to get its full path.
Something like this:
# --- begin runfiles.bash initialization v2 ---
# Copy-pasted from the Bazel Bash runfiles library v2.
set -uo pipefail; f=bazel_tools/tools/bash/runfiles/runfiles.bash
source "${RUNFILES_DIR:-/dev/null}/$f" 2>/dev/null || \
source "$(grep -sm1 "^$f " "${RUNFILES_MANIFEST_FILE:-/dev/null}" | cut -f2- -d' ')" 2>/dev/null || \
source "$0.runfiles/$f" 2>/dev/null || \
source "$(grep -sm1 "^$f " "$0.runfiles_manifest" | cut -f2- -d' ')" 2>/dev/null || \
source "$(grep -sm1 "^$f " "$0.exe.runfiles_manifest" | cut -f2- -d' ')" 2>/dev/null || \
{ echo>&2 "ERROR: cannot find $f"; exit 1; }; f=; set -e
# --- end runfiles.bash initialization v2 ---
$(rlocation "my_workspace/some/package/my_container_push")
With some/package/BUILD having this:
load("#io_bazel_rules_docker//contrib:push-all.bzl", "docker_push")
load("#io_bazel_rules_docker//container:container.bzl", "container_bundle")
container_bundle(
name = "my_container_bundle",
# All your existing attrs here, etc etc.
)
docker_push(
name = "my_container_push",
bundle = ":my_container_bundle",
)
sh_test(
name = "my_test",
data = [
":my_container_push",
],
deps = [
"#bazel_tools//tools/bash/runfiles",
],
)

No such file found in jenkis pipeline

Here is my groovy file
timestamps{
node('cf_slave'){
checkout scm
stage('Read the file') {
def PWD = pwd()
withEnv(["prj_option=${params.project}"]) {
def response =sh(returnStdout: true, script: 'sh \'jenkins/security/get_values.sh\'')
}
}
This is my get_values.sh file
echo "The project option is:" $prj_option
prj_name=$(echo "$prj_option" | tr '[:upper:]' '[:lower:]')
file_name="va_input_file_$prj_name.txt"
echo "The project option is:" $file_name
ls -la
chmod 775 jenkins/security/$file_name
ls -la
get_input_values() {
file=$1
IFS=''
while read line
do
if [ `echo ${line} | grep -E -c -w "NAME_SPACE" ` -gt 0 ]; then
NAME_SPACE=$(echo " ${line}" | cut -d'=' -f2)
echo "The name space value is $NAME_SPACE"
elif [ `echo ${line} | grep -E -c -w "IMAGE_NAMES" ` -gt 0 ]; then
values=$(echo " ${line}" | cut -d'=' -f2)
echo "THE DOCKERIMAGES are $DOCKER_IMAGES_NAMES"
else
echo "Please provide input for namespace and docker images to be scanned by VA_TOOl"
fi
done < ${file}
}
images=$(get_input_values ${file_name})
so here my text file is under jenkins/security folder of gitrepo but unfortunately I am getting this error:
16:05:28 + sh jenkins/security/get_values.sh
16:05:28 jenkins/security/get_values.sh: 16: jenkins/security/get_values.sh: cannot open va_input_file_icp.txt: No such file```

Unfortunately, there is a ticket for this in Jenkins (https://issues.jenkins-ci.org/browse/JENKINS-51245) which was closed as a duplicate of this ticket: (https://issues.jenkins-ci.org/browse/JENKINS-27413)
JENKINS-27413 was raised in 2015, and is still open. The File Parameter appears not to work in Jenkins Pipeline. It does however work when used in a Freestyle project. While not ideal, I would recommend changing your job to be a Freestyle project if that's feasible.

How can I list all tags for a Docker image on a remote registry?

How can I list all tags of a Docker image on a remote Docker registry using the CLI (preferred) or curl?
Preferably without pulling all versions from the remote registry. I just want to list the tags.

Update: sadly this solution will no longer work because Docker has deprecated the v1 API.
I got the answer from here . Thanks a lot! :)
Just one-line-script:(find all the tags of debian)
wget -q https://registry.hub.docker.com/v1/repositories/debian/tags -O - | sed -e 's/[][]//g' -e 's/"//g' -e 's/ //g' | tr '}' '\n' | awk -F: '{print $3}'
UPDATE
Thanks for #degelf's advice.
Here is the shell script.
#!/bin/bash
if [ $# -lt 1 ]
then
cat << HELP
dockertags -- list all tags for a Docker image on a remote registry.
EXAMPLE:
- list all tags for ubuntu:
dockertags ubuntu
- list all php tags containing apache:
dockertags php apache
HELP
fi
image="$1"
tags=`wget -q https://registry.hub.docker.com/v1/repositories/${image}/tags -O - | sed -e 's/[][]//g' -e 's/"//g' -e 's/ //g' | tr '}' '\n' | awk -F: '{print $3}'`
if [ -n "$2" ]
then
tags=` echo "${tags}" | grep "$2" `
fi
echo "${tags}"
You can just create a new file name, dockertags, under /usr/local/bin (or add a PATH env to your .bashrc/.zshrc), and put that code in it.
Then add the executable permissions(chmod +x dockertags).
Usage:
dockertags ubuntu ---> list all tags of ubuntu
dockertags php apache ---> list all php tags php containing 'apache'

As of Docker Registry V2, a simple GET suffice:
GET /v2/<name>/tags/list
See docs for more.

If you want to use the docker registry v2 API, it lists tags by pages. To list all the tags of an image, you may would like to add a large page_size parameter to the url, e.g.
curl -L -s 'https://registry.hub.docker.com/v2/repositories/library/centos/tags?page_size=1024'|jq '."results"[]["name"]'

The Docker V2 API requires an OAuth bearer token with the appropriate claims. In my opinion, the official documentation is rather vague on the topic. So that others don't go through the same pain I did, I offer the below docker-tags function.
The most recent version of docker-tags can be found in my GitHubGist : "List Docker Image Tags using bash".
The docker-tags function has a dependency on jq. If you're playing with JSON, you likely already have it.
#!/usr/bin/env bash
docker-tags() {
arr=("$#")
for item in "${arr[#]}";
do
tokenUri="https://auth.docker.io/token"
data=("service=registry.docker.io" "scope=repository:$item:pull")
token="$(curl --silent --get --data-urlencode ${data[0]} --data-urlencode ${data[1]} $tokenUri | jq --raw-output '.token')"
listUri="https://registry-1.docker.io/v2/$item/tags/list"
authz="Authorization: Bearer $token"
result="$(curl --silent --get -H "Accept: application/json" -H "Authorization: Bearer $token" $listUri | jq --raw-output '.')"
echo $result
done
}
Example
docker-tags "microsoft/nanoserver" "microsoft/dotnet" "library/mongo" "library/redis"
Admittedly, docker-tags makes several assumptions. Specifically, the OAuth request parameters are mostly hard coded. A more ambitious implementation would make an unauthenticated request to the registry and derive the OAuth parameters from the unauthenticated response.

You can list all the tags with skopeo and jq for json parsing through cli.
skopeo --override-os linux inspect docker://httpd | jq '.RepoTags'
[
"2-alpine",
"2.2-alpine",
"2.2.29",
"2.2.31-alpine",
"2.2.31",
"2.2.32-alpine",
"2.2.32",
"2.2.34-alpine",
"2.2.34",
"2.2",
"2.4-alpine",
"2.4.10",
"2.4.12",
"2.4.16",
"2.4.17",
"2.4.18",
"2.4.20",
"2.4.23-alpine",
"2.4.23",
"2.4.25-alpine",
"2.4.25",
"2.4.27-alpine",
"2.4.27",
"2.4.28-alpine",
"2.4.28",
"2.4.29-alpine",
"2.4.29",
"2.4.32-alpine",
"2.4.32",
"2.4.33-alpine",
"2.4.33",
"2.4.34-alpine",
"2.4.34",
"2.4.35-alpine",
"2.4.35",
"2.4.37-alpine",
"2.4.37",
"2.4.38-alpine",
"2.4.38",
"2.4.39-alpine",
"2.4.39",
"2.4.41-alpine",
"2.4.41",
"2.4.43-alpine",
"2.4.43",
"2.4",
"2",
"alpine",
"latest"
]
For external registries:
skopeo --override-os linux inspect --creds username:password docker://<registry-url>/<repo>/<image> | jq '.RepoTags'
Note: --override-os linux is only needed if you are not running on a linux host. For example, you'll have better results with it if you are on MacOS.

If the JSON parsing tool, jq is available
wget -q https://registry.hub.docker.com/v1/repositories/debian/tags -O - | \
jq -r '.[].name'

I've managed to get it working using curl:
curl -u <username>:<password> https://myrepo.example/v1/repositories/<username>/<image_name>/tags
Note that image_name should not contain user details etc. For example if you're pushing image named myrepo.example/username/x then image_name should be x.

Building on Yan Foto's answer (the v2 api), I created a simple Python script to list the tags for a given image.
Usage:
./docker-registry-list.py alpine
Output:
{
"name": "library/alpine",
"tags": [
"2.6",
"2.7",
"3.1",
"3.2",
"3.3",
"3.4",
"3.5",
"3.6",
"3.7",
"edge",
"latest"
]
}

You can achieve by running on terminal this:
curl -L -s 'https://registry.hub.docker.com/v2/repositories/library/mysql/tags/' | jq . | grep name
Also, if you don't have jq you have to install it by
sudo apt-get install jq

See CLI utility: https://www.npmjs.com/package/docker-browse
Allows enumeration of tags and images.
docker-browse tags <image> will list all tags for the image. e.g. docker-browse tags library/alpine
docker-browse images will list all images in the registry. Not currently available for index.docker.io.
You may connect it to any registry, including your private one, so long as it supports Docker Registry HTTP API V2

Here's a Powershell script I wrote for Windows. Handles v1 and v2 repos:
Get-DockerImageVersions.ps1:
param (
[Parameter (Mandatory=$true)]$ImageName,
[Parameter (Mandatory=$false)]$RegistryURL
)
if (!$RegistryURL)
{
$RegistryURL = "https://registry.hub.docker.com/v1/repositories"
}
$list = ""
if ($RegistryURL -like "*v2*")
{
$list = "/list"
}
$URL = "$RegistryURL/$ImageName/tags$list"
write-debug $URL
$resp = Invoke-WebRequest -UseBasicParsing $URL | ConvertFrom-Json
if ($RegistryURL -like "*v2*")
{
$tags = $resp | select tags
$tags.tags
} else {
$tags = $resp | select name
$tags.name
}

Get all tags from Docker Hub: this command uses the command-line JSON processor jq to select the tag names from the JSON returned by the Docker Hub Registry (the quotes are removed with tr). Replace library with the Docker Hub user name, debian with the image name:
curl -s 'https://registry.hub.docker.com/v2/repositories/library/debian/tags/' | jq -r '."results"[]["name"]'

To view all available tags in a browser:
https://registry.hub.docker.com/v1/repositories/<username>/<image_name>/tags
i.e. https://hub.docker.com/r/localstack/localstack/tags
Or, you can get a json response using this endpoint:
https://registry.hub.docker.com/v1/repositories/localstack/localstack/tags

My contribution:
Shell script
As short and simple as possible
Requires curl and jq
Uses Docker v2 REST API
Returns all tags using REST API pagination
Example:
$ docker-tags prantlf/chromedriver-headless
latest
102
93
86
Script contents:
#!/bin/sh
image=$1
if [ "$image" == "" ]; then
echo "Usage:
docker-tags <image>
Example:
docker-tags library/ubuntu"
exit 0
fi
page_size=100
page_index=0
while true; do
page_index=$((page_index+1))
results=`curl -L -s "https://registry.hub.docker.com/v2/repositories/$image/tags?page=$page_index&page_size=$page_size" | jq -r 'select(.results != null) | .results[]["name"]'`
if [ $? != 0 ] || [ "$results" == "" ]; then
break
fi
echo "$results"
done

As of 2023, there are a number of tools to do this
regclient
skopeo
crane
reg - hasn't been updates for years, unfortunately. Doesn't work with quay.io.
docker run --rm ghcr.io/regclient/regctl:v0.4.5 tag ls ghcr.io/regclient/regctl
docker run --rm quay.io/skopeo/stable:v1.9.2 list-tags docker://quay.io/skopeo/stable \
| jq -r '.Tags[]'
docker run --rm gcr.io/go-containerregistry/crane ls gcr.io/go-containerregistry/crane
docker run --rm r.j3ss.co/reg:v0.16.1 tags r.j3ss.co/reg
BTW - there are even more tools. This list looks comprehensive:
iximiuz/awesome-container-tinkering.

curl -u <username>:<password> https://$your_registry/v2/$image_name/tags/list -s -o - | \
tr -d '{' | tr -d '}' | sed -e 's/[][]//g' -e 's/"//g' -e 's/ //g' | \
awk -F: '{print $3}' | sed -e 's/,/\n/g'
You can use it if your env has no 'jq', = )

You can use:
skopeo inspect docker://<REMOTE_REGISTRY> --authfile <PULL_SECRET> | jq .RepoTags

Here is a script that lists all tags either with 2 or 3 digits.
You can get the code directly on github
https://github.com/youssefalaoui/dockerhub-tools/blob/main/dockerhub-list-tags.sh
dockerhub_list_tags()
{
#local LOCAL_IMAGE LOCAL_GET_TWO_DIGITS_VERSIONS
LOCAL_IMAGE=${1:-null}
LOCAL_GET_TWO_DIGITS_VERSIONS=${2:-true}
if [[ $LOCAL_IMAGE == "" || $LOCAL_IMAGE == null ]]
then
printf "Image name is required: %s" ${FUNCNAME[0]};
exit 1;
fi
#[[ $LOCAL_IMAGE == "" || $LOCAL_IMAGE == null ]] && printf "Image name is required: %s" ${FUNCNAME[0]}; exit 1;
echo "Listing tags from docker hub for your image '$LOCAL_IMAGE'"
# Check if 2 digits format is requested, otherwise, show it in normal format
if [[ "$LOCAL_GET_TWO_DIGITS_VERSIONS" == true ]]; then
DOCKERHUB_LIST_TAGS=($(curl -L -s "https://registry.hub.docker.com/v2/repositories/$LOCAL_IMAGE/tags?page_size=1024"|jq '."results"[]["name"]' | sed 's/"//g' | sed 's/\.[^.]*$//'))
else
DOCKERHUB_LIST_TAGS=($(curl -L -s "https://registry.hub.docker.com/v2/repositories/$LOCAL_IMAGE/tags?page_size=1024"|jq '."results"[]["name"]' | sed 's/"//g'))
fi
for TAG in ${DOCKERHUB_LIST_TAGS[#]}
do
echo $TAG
done
}
# Test example
dockerhub_list_tags "library/nginx" false

You can also use this scrap :
# vim /usr/sbin/docker-tags
& Append Following (as it is):
#!/bin/bash
im="$1"
[[ -z "$im" ]] && { echo -e '\e[31m[-]\e[39m Where is the image name ??' ; exit ; }
[[ -z "$(echo "$im"| grep -o '/')" ]] && { link="https://hub.docker.com/r/library/$im/tags/" ; } || { link="https://hub.docker.com/r/$im/tags/" ; }
resp="$(curl -sL "$link")"
err="$(echo "$resp" | grep -o 'Page Not Found')"
if [[ ! -z "$err" ]] ; then
echo -e "\e[31m[-]\e[39m No Image Found with name => [ \e[32m$im\e[39m ]"
exit
else
tags="$(echo "$resp"|sed -e 's|}|\n|g' -e 's|{|\n|g'|grep '"result"'|sed -e 's|,|\n|g'|cut -d '[' -f2|cut -d ']' -f1|sed '/"tags":/d'|sed -e 's|"||g')"
echo -e "\e[32m$tags\e[39m"
fi
Make it Executable :
# chmod 755 /usr/sbin/docker-tags
Then Finally Try By :
$ docker-tags testexampleidontexist
[-] No Image Found with name => [ testexampleidontexist ]
$ docker search ubuntu
$ docker-tags teamrock/ubuntu
latest
[ Hope you are aware of $ & # before running any command ]

If folks want to read tags from the RedHat registry at https://registry.redhat.io/v2 then the steps are:
# example nodejs-12 image
IMAGE_STREAM=nodejs-12
REDHAT_REGISTRY_API="https://registry.redhat.io/v2/rhel8/$IMAGE_STREAM"
# Get an oAuth token based on a service account username and password https://access.redhat.com/articles/3560571
TOKEN=$(curl --silent -u "$REGISTRY_USER":"$REGISTRY_PASSWORD" "https://sso.redhat.com/auth/realms/rhcc/protocol/redhat-docker-v2/auth?service=docker-registry&client_id=curl&scope=repository:rhel:pull" | jq --raw-output '.token')
# Grab the tags
wget -q --header="Accept: application/json" --header="Authorization: Bearer $TOKEN" -O - "$REDHAT_REGISTRY_API/tags/list" | jq -r '."tags"[]'
If you want to compare what you have in your local openshift registry against what is in the upstream registry.redhat.com then here is a complete script.

The Docker Registry API has an endpoint to list all tags.
Looks like Tutum has a similar endpoint, as well as a way to access via tutum-cli.
With the tutum-cli, try the following:
tutum tag list <uuid>

In powershell 5.1, I have a simple list_docker_image_tags.ps1 script like this:
[CmdletBinding()]
param (
[Parameter(Mandatory = $true)]
[string]
$image
)
$url = "https://registry.hub.docker.com/v1/repositories/{0}/tags" -f $image
Invoke-WebRequest $url | ConvertFrom-Json | Write-Output
Then I can grep for 4.7 tags like this:
./list_docker_image_tags.ps1 microsoft/dotnet-framework | ?{ $_.name -match "4.7" }

Here's an answer that's applicable for v2 of the registry.
If you have jq and curl installed on your machine:
curl https://registry.hub.docker.com/v2/repositories/$REPOSITORY/tags?page_size=10000 | jq '.results[] | { name: .name, architectures: ([ (.images[] | if .variant? then .os + "/" + .architecture + .variant? else .os + "/" + .architecture end) ] | join(", ")) }'
For instance, running this command for the curlimages/curl repository yields:
{
"name": "latest",
"architectures": "linux/ppc64le, linux/s390x, linux/arm64, linux/386, linux/armv7, linux/amd64"
}
{
"name": "7.78.0",
"architectures": "linux/armv7, linux/arm64, linux/386, linux/s390x, linux/ppc64le, linux/amd64"
}
{
"name": "7.77.0",
"architectures": "linux/ppc64le, linux/arm64, linux/s390x, linux/armv7, linux/386, linux/amd64"
}
{
"name": "7.76.1",
"architectures": "linux/386, linux/arm64, linux/armv7, linux/ppc64le, linux/s390x, linux/amd64"
}
{
"name": "7.76.0",
"architectures": "linux/armv7, linux/386, linux/s390x, linux/amd64, linux/ppc64le, linux/arm64"
}
{
"name": "7.75.0",
"architectures": "linux/armv7, linux/ppc64le, linux/386, linux/amd64, linux/arm64, linux/s390x"
}
{
"name": "7.74.0",
"architectures": "linux/armv7, linux/386, linux/amd64, linux/ppc64le, linux/s390x, linux/arm64"
}
{
"name": "7.73.0",
"architectures": "linux/arm64, linux/armv7, linux/s390x, linux/ppc64le, linux/amd64, linux/386"
}
{
"name": "7.72.0",
"architectures": "linux/s390x, linux/amd64, linux/arm64, linux/386, linux/ppc64le, linux/armv7"
}
{
"name": "7.71.1",
"architectures": "linux/s390x, linux/arm64, linux/ppc64le, linux/amd64, linux/386, linux/armv7"
}
{
"name": "7.71.0",
"architectures": "linux/arm64, linux/ppc64le, linux/386, linux/s390x, linux/amd64, linux/armv7"
}
{
"name": "7.70.0",
"architectures": "linux/386, linux/arm64, linux/s390x, linux/amd64, linux/ppc64le, linux/armv7"
}
{
"name": "7.69.1",
"architectures": "linux/amd64"
}
{
"name": "7.69.0",
"architectures": "linux/amd64"
}
{
"name": "7.68.0",
"architectures": "linux/amd64"
}
{
"name": "7.67.0",
"architectures": "linux/amd64"
}
{
"name": "7.66.0",
"architectures": "linux/amd64"
}
{
"name": "7.65.3",
"architectures": "linux/amd64"
}

Edit:
In answer to the question:
How can I list all tags of a Docker image on a remote Docker registry using the CLI (preferred) or curl?
Preferably without pulling all versions from the remote registry. I just want to list the tags.
To get all the tags for an image you can use "curl" to get the specific image you want and pipe the output into "jq" to extract the information.
curl -L -s 'https://registry.hub.docker.com/v2/repositories/library/python/tags?page_size=1024'|jq '.results[]["name"]'
Output (truncated not the full list):
"3.9-windowsservercore"
"alpine3.14"
"alpine3.13"
"alpine"
"3.9.8-alpine3.14"
"3.9.8-alpine3.13"
"3.9.8-alpine"
Further should you need additional information from the registry you can access additional field information like this.
This command will give you both the tags and the size of the image which might be useful to have too.
curl -L -s 'https://registry.hub.docker.com/v2/repositories/library/python/tags?page_size=1024'|jq '.results[] as $results | ($results["name"] + " - " + ($results["full_size"] | tostring))'
Output (truncated not the full list):
"3.9-windowsservercore - 2241040278"
"alpine3.14 - 17565702"
"alpine3.13 - 17556181"
"alpine - 17565702"
"3.9.8-alpine3.14 -17362557"
"3.9.8-alpine3.13 - 17353629"
"3.9.8-alpine - 17362557"

There is a lot of duplication among the answers given thus far.
Most of them fail to take into account that the GitHub API (at least v2) will never return more than 100 results at a time, even if you ask for more. I noticed this when requesting the tags for php.
The following script works around this.
It only works for public repositories.
#!/bin/sh
# list the tags on Docker Hub for the given image(s)
# thank you, https://stackoverflow.com/questions/28320134/how-can-i-list-all-tags-for-a-docker-image-on-a-remote-registry
TagsFor()
{
curl -L -s 'https://registry.hub.docker.com/v2/repositories/library/'$1'/tags?page='$2'&page_size'=$3
}
for i in "$#"
do
TagsFor "$i" 1 10 |
jq -r .count |
while read nr_of_tags
do
nr_of_pages=`expr $nr_of_tags / 100`
seq 1 $nr_of_pages |
while read p
do
TagsFor "$i" "$p" 100 |
jq -r '.results[] | .name'
done
done
done
I just ran the script; it retrieved 7200 php tags. For all I know, it may be running into yet another API limit, but 7200 >> 100.

I have done this thing when I have to implement a task in which if user somehow type the wrong tag then we have to give the list of all the tag present in the repo(Docker repo) present in the register.
So I have code in batch Script.
<html>
<pre style="background-color:#bcbbbb;">
#echo off
docker login --username=xxxx --password=xxxx
docker pull %1:%2
IF NOT %ERRORLEVEL%==0 (
echo "Specified Version is Not Found "
echo "Available Version for this image is :"
for /f %%i in (' curl -s -H "Content-Type:application/json" -X POST -d "{\"username\":\"user\",\"password\":\"password\"}" https://hub.docker.com/v2/users/login ^|jq -r .token ') do set TOKEN=%%i
curl -sH "Authorization: JWT %TOKEN%" "https://hub.docker.com/v2/repositories/%1/tags/" | jq .results[].name
)
</pre>
</html>
So in this we can give arguments to out batch file like: Dockerfile java version7

Building on #AlexForbes's answer I've improved the api v2 docker-registry-list.py to support:
  – slashes in the repository name (eg curlimages/curl) and
  – private repos (authentication by username and password)
https://github.com/axil/docker-registry-list
Usage:
./docker-registry-list.py -u dockerid -p password dockerid/myrepo
Output:
{
"name": "dockerid/myrepo",
"tags": [
"1.0"
]
}

Was looking for an sdk in java that I could use to hit the Docker V2 API but couldn't find one. Repo here for anyone that might find it useful: https://github.com/fern-api/docker-registry-api.
Should be possible to generate in other languages too, feel free to open an issue on the repo!

Parse URL in shell script

I have url like:
sftp://user#host.net/some/random/path
I want to extract user, host and path from this string. Any part can be random length.

[EDIT 2019]
This answer is not meant to be a catch-all, works for everything solution it was intended to provide a simple alternative to the python based version and it ended up having more features than the original.
It answered the basic question in a bash-only way and then was modified multiple times by myself to include a hand full of demands by commenters. I think at this point however adding even more complexity would make it unmaintainable. I know not all things are straight forward (checking for a valid port for example requires comparing hostport and host) but I would rather not add even more complexity.
[Original answer]
Assuming your URL is passed as first parameter to the script:
#!/bin/bash
# extract the protocol
proto="$(echo $1 | grep :// | sed -e's,^\(.*://\).*,\1,g')"
# remove the protocol
url="$(echo ${1/$proto/})"
# extract the user (if any)
user="$(echo $url | grep # | cut -d# -f1)"
# extract the host and port
hostport="$(echo ${url/$user#/} | cut -d/ -f1)"
# by request host without port
host="$(echo $hostport | sed -e 's,:.*,,g')"
# by request - try to extract the port
port="$(echo $hostport | sed -e 's,^.*:,:,g' -e 's,.*:\([0-9]*\).*,\1,g' -e 's,[^0-9],,g')"
# extract the path (if any)
path="$(echo $url | grep / | cut -d/ -f2-)"
echo "url: $url"
echo " proto: $proto"
echo " user: $user"
echo " host: $host"
echo " port: $port"
echo " path: $path"
I must admit this is not the cleanest solution but it doesn't rely on another scripting
language like perl or python.
(Providing a solution using one of them would produce cleaner results ;) )
Using your example the results are:
url: user#host.net/some/random/path
proto: sftp://
user: user
host: host.net
port:
path: some/random/path
This will also work for URLs without a protocol/username or path.
In this case the respective variable will contain an empty string.
[EDIT]
If your bash version won't cope with the substitutions (${1/$proto/}) try this:
#!/bin/bash
# extract the protocol
proto="$(echo $1 | grep :// | sed -e's,^\(.*://\).*,\1,g')"
# remove the protocol -- updated
url=$(echo $1 | sed -e s,$proto,,g)
# extract the user (if any)
user="$(echo $url | grep # | cut -d# -f1)"
# extract the host and port -- updated
hostport=$(echo $url | sed -e s,$user#,,g | cut -d/ -f1)
# by request host without port
host="$(echo $hostport | sed -e 's,:.*,,g')"
# by request - try to extract the port
port="$(echo $hostport | sed -e 's,^.*:,:,g' -e 's,.*:\([0-9]*\).*,\1,g' -e 's,[^0-9],,g')"
# extract the path (if any)
path="$(echo $url | grep / | cut -d/ -f2-)"

The above, refined (added password and port parsing), and working in /bin/sh:
# extract the protocol
proto="`echo $DATABASE_URL | grep '://' | sed -e's,^\(.*://\).*,\1,g'`"
# remove the protocol
url=`echo $DATABASE_URL | sed -e s,$proto,,g`
# extract the user and password (if any)
userpass="`echo $url | grep # | cut -d# -f1`"
pass=`echo $userpass | grep : | cut -d: -f2`
if [ -n "$pass" ]; then
user=`echo $userpass | grep : | cut -d: -f1`
else
user=$userpass
fi
# extract the host -- updated
hostport=`echo $url | sed -e s,$userpass#,,g | cut -d/ -f1`
port=`echo $hostport | grep : | cut -d: -f2`
if [ -n "$port" ]; then
host=`echo $hostport | grep : | cut -d: -f1`
else
host=$hostport
fi
# extract the path (if any)
path="`echo $url | grep / | cut -d/ -f2-`"
Posted b/c I needed it, so I wrote it (based on #Shirkin's answer, obviously), and I figured someone else might appreciate it.

This solution in principle works the same as Adam Ryczkowski's, in this thread - but has improved regular expression based on RFC3986, (with some changes) and fixes some errors (e.g. userinfo can contain '_' character). This can also understand relative URIs (e.g. to extract query or fragment).
# !/bin/bash
# Following regex is based on https://www.rfc-editor.org/rfc/rfc3986#appendix-B with
# additional sub-expressions to split authority into userinfo, host and port
#
readonly URI_REGEX='^(([^:/?#]+):)?(//((([^:/?#]+)#)?([^:/?#]+)(:([0-9]+))?))?(/([^?#]*))(\?([^#]*))?(#(.*))?'
# ↑↑ ↑ ↑↑↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑
# |2 scheme | ||6 userinfo 7 host | 9 port | 11 rpath | 13 query | 15 fragment
# 1 scheme: | |5 userinfo# 8 :… 10 path 12 ?… 14 #…
# | 4 authority
# 3 //…
parse_scheme () {
[[ "$#" =~ $URI_REGEX ]] && echo "${BASH_REMATCH[2]}"
}
parse_authority () {
[[ "$#" =~ $URI_REGEX ]] && echo "${BASH_REMATCH[4]}"
}
parse_user () {
[[ "$#" =~ $URI_REGEX ]] && echo "${BASH_REMATCH[6]}"
}
parse_host () {
[[ "$#" =~ $URI_REGEX ]] && echo "${BASH_REMATCH[7]}"
}
parse_port () {
[[ "$#" =~ $URI_REGEX ]] && echo "${BASH_REMATCH[9]}"
}
parse_path () {
[[ "$#" =~ $URI_REGEX ]] && echo "${BASH_REMATCH[10]}"
}
parse_rpath () {
[[ "$#" =~ $URI_REGEX ]] && echo "${BASH_REMATCH[11]}"
}
parse_query () {
[[ "$#" =~ $URI_REGEX ]] && echo "${BASH_REMATCH[13]}"
}
parse_fragment () {
[[ "$#" =~ $URI_REGEX ]] && echo "${BASH_REMATCH[15]}"
}

Using Python (best tool for this job, IMHO):
#!/usr/bin/env python
import os
from urlparse import urlparse
uri = os.environ['NAUTILUS_SCRIPT_CURRENT_URI']
result = urlparse(uri)
user, host = result.netloc.split('#')
path = result.path
print('user=', user)
print('host=', host)
print('path=', path)
Further reading:
os.environ
urlparse.urlparse()

If you really want to do it in shell, you can do something as simple as the following by using awk. This requires knowing how many fields you will actually be passed (e.g. no password sometimes and not others).
#!/bin/bash
FIELDS=($(echo "sftp://user#host.net/some/random/path" \
| awk '{split($0, arr, /[\/\#:]*/); for (x in arr) { print arr[x] }}'))
proto=${FIELDS[1]}
user=${FIELDS[2]}
host=${FIELDS[3]}
path=$(echo ${FIELDS[#]:3} | sed 's/ /\//g')
If you don't have awk and you do have grep, and you can require that each field have at least two characters and be reasonably predictable in format, then you can do:
#!/bin/bash
FIELDS=($(echo "sftp://user#host.net/some/random/path" \
| grep -o "[a-z0-9.-][a-z0-9.-]*" | tr '\n' ' '))
proto=${FIELDS[1]}
user=${FIELDS[2]}
host=${FIELDS[3]}
path=$(echo ${FIELDS[#]:3} | sed 's/ /\//g')

Just needed to do the same, so was curious if it's possible to do it in single line, and this is what i've got:
#!/bin/bash
parse_url() {
eval $(echo "$1" | sed -e "s#^\(\(.*\)://\)\?\(\([^:#]*\)\(:\(.*\)\)\?#\)\?\([^/?]*\)\(/\(.*\)\)\?#${PREFIX:-URL_}SCHEME='\2' ${PREFIX:-URL_}USER='\4' ${PREFIX:-URL_}PASSWORD='\6' ${PREFIX:-URL_}HOST='\7' ${PREFIX:-URL_}PATH='\9'#")
}
URL=${1:-"http://user:pass#example.com/path/somewhere"}
PREFIX="URL_" parse_url "$URL"
echo "$URL_SCHEME://$URL_USER:$URL_PASSWORD#$URL_HOST/$URL_PATH"
How it works:
There is that crazy sed regex that captures all the parts of url, when all of them are optional (except for the host name)
Using those capture groups sed outputs env variables names with their values for relevant parts (like URL_SCHEME or URL_USER)
eval executes that output, causing those variables to be exported and available in the script
Optionally PREFIX could be passed to control output env variables names
PS: be careful when using this for arbitrary input since this code is vulnerable to script injections.

Here's my take, loosely based on some of the existing answers, but it can also cope with GitHub SSH clone URLs:
#!/bin/bash
PROJECT_URL="git#github.com:heremaps/here-aaa-java-sdk.git"
# Extract the protocol (includes trailing "://").
PARSED_PROTO="$(echo $PROJECT_URL | sed -nr 's,^(.*://).*,\1,p')"
# Remove the protocol from the URL.
PARSED_URL="$(echo ${PROJECT_URL/$PARSED_PROTO/})"
# Extract the user (includes trailing "#").
PARSED_USER="$(echo $PARSED_URL | sed -nr 's,^(.*#).*,\1,p')"
# Remove the user from the URL.
PARSED_URL="$(echo ${PARSED_URL/$PARSED_USER/})"
# Extract the port (includes leading ":").
PARSED_PORT="$(echo $PARSED_URL | sed -nr 's,.*(:[0-9]+).*,\1,p')"
# Remove the port from the URL.
PARSED_URL="$(echo ${PARSED_URL/$PARSED_PORT/})"
# Extract the path (includes leading "/" or ":").
PARSED_PATH="$(echo $PARSED_URL | sed -nr 's,[^/:]*([/:].*),\1,p')"
# Remove the path from the URL.
PARSED_HOST="$(echo ${PARSED_URL/$PARSED_PATH/})"
echo "proto: $PARSED_PROTO"
echo "user: $PARSED_USER"
echo "host: $PARSED_HOST"
echo "port: $PARSED_PORT"
echo "path: $PARSED_PATH"
which gives
proto:
user: git#
host: github.com
port:
path: :heremaps/here-aaa-java-sdk.git
And for PROJECT_URL="ssh://sschuberth#git.eclipse.org:29418/jgit/jgit" you get
proto: ssh://
user: sschuberth#
host: git.eclipse.org
port: :29418
path: /jgit/jgit

You can use bash string manipulation. It is easy to learn. In case you feel difficulties with regex, try it. As it is from NAUTILUS_SCRIPT_CURRENT_URI, i guess there may have port in that URI. So I also kept that optional.
#!/bin/bash
#You can also use environment variable $NAUTILUS_SCRIPT_CURRENT_URI
X="sftp://user#host.net/some/random/path"
tmp=${X#*//};usr=${tmp%#*}
tmp=${X#*#};host=${tmp%%/*};[[ ${X#*://} == *":"* ]] && host=${host%:*}
tmp=${X#*//};path=${tmp#*/}
proto=${X%:*}
[[ ${X#*://} == *":"* ]] && tmp=${X##*:} && port=${tmp%%/*}
echo "Potocol:"$proto" User:"$usr" Host:"$host" Port:"$port" Path:"$path

I don't have enough reputation to comment, but I made a small modification to #patryk-obara's answer.
RFC3986 § 6.2.3. Scheme-Based Normalization
treats
http://example.com
http://example.com/
as equivalent. But I found that his regex did not match a URL like http://example.com. http://example.com/ (with the trailing slash) does match.
I inserted 11, which changed / to (/|$). This matches either / or the end of the string. Now http://example.com does match.
readonly URI_REGEX='^(([^:/?#]+):)?(//((([^:/?#]+)#)?([^:/?#]+)(:([0-9]+))?))?((/|$)([^?#]*))(\?([^#]*))?(#(.*))?$'
# ↑↑ ↑ ↑↑↑ ↑ ↑ ↑ ↑↑ ↑ ↑ ↑ ↑ ↑
# || | ||| | | | || | | | | |
# |2 scheme | ||6 userinfo 7 host | 9 port || 12 rpath | 14 query | 16 fragment
# 1 scheme: | |5 userinfo# 8 :... || 13 ?... 15 #...
# | 4 authority |11 / or end-of-string
# 3 //... 10 path

If you have access to Bash >= 3.0 you can do this in pure bash as well, thanks to the re-match operator =~:
pattern='^(([[:alnum:]]+)://)?(([[:alnum:]]+)#)?([^:^#]+)(:([[:digit:]]+))?$'
if [[ "http://us#cos.com:3142" =~ $pattern ]]; then
proto=${BASH_REMATCH[2]}
user=${BASH_REMATCH[4]}
host=${BASH_REMATCH[5]}
port=${BASH_REMATCH[7]}
fi
It should be faster and less resource-hungry then all the previous examples, because no external process is be spawned.

A simplistic approach to get just the domain from the full URL:
echo https://stackoverflow.com/questions/6174220/parse-url-in-shell-script | cut -d/ -f1-3
# OUTPUT>>> https://stackoverflow.com
Get only the path:
echo https://stackoverflow.com/questions/6174220/parse-url-in-shell-script | cut -d/ -f4-
# OUTPUT>>> questions/6174220/parse-url-in-shell-script
Not perfect, as the second command strips the preceding slash so you'll need to prepend it by hand.
An awk-based approach for getting just the path without the domain:
echo https://stackoverflow.com/questions/6174220/parse-url-in-shell-script/59971653 | awk -F"/" '{ for (i=4; i<=NF; i++) printf"/%s", $i }'
# OUTPUT>>> /questions/6174220/parse-url-in-shell-script/59971653

I did further parsing, expanding the solution given by #Shirkrin:
#!/bin/bash
parse_url() {
local query1 query2 path1 path2
# extract the protocol
proto="$(echo $1 | grep :// | sed -e's,^\(.*://\).*,\1,g')"
if [[ ! -z $proto ]] ; then
# remove the protocol
url="$(echo ${1/$proto/})"
# extract the user (if any)
login="$(echo $url | grep # | cut -d# -f1)"
# extract the host
host="$(echo ${url/$login#/} | cut -d/ -f1)"
# by request - try to extract the port
port="$(echo $host | sed -e 's,^.*:,:,g' -e 's,.*:\([0-9]*\).*,\1,g' -e 's,[^0-9],,g')"
# extract the uri (if any)
resource="/$(echo $url | grep / | cut -d/ -f2-)"
else
url=""
login=""
host=""
port=""
resource=$1
fi
# extract the path (if any)
path1="$(echo $resource | grep ? | cut -d? -f1 )"
path2="$(echo $resource | grep \# | cut -d# -f1 )"
path=$path1
if [[ -z $path ]] ; then path=$path2 ; fi
if [[ -z $path ]] ; then path=$resource ; fi
# extract the query (if any)
query1="$(echo $resource | grep ? | cut -d? -f2-)"
query2="$(echo $query1 | grep \# | cut -d\# -f1 )"
query=$query2
if [[ -z $query ]] ; then query=$query1 ; fi
# extract the fragment (if any)
fragment="$(echo $resource | grep \# | cut -d\# -f2 )"
echo "url: $url"
echo " proto: $proto"
echo " login: $login"
echo " host: $host"
echo " port: $port"
echo "resource: $resource"
echo " path: $path"
echo " query: $query"
echo "fragment: $fragment"
echo ""
}
parse_url "http://login:password#example.com:8080/one/more/dir/file.exe?a=sth&b=sth#anchor_fragment"
parse_url "https://example.com/one/more/dir/file.exe#anchor_fragment"
parse_url "http://login:password#example.com:8080/one/more/dir/file.exe#anchor_fragment"
parse_url "ftp://user#example.com:8080/one/more/dir/file.exe?a=sth&b=sth"
parse_url "/one/more/dir/file.exe"
parse_url "file.exe"
parse_url "file.exe#anchor"

I did not like above methods and wrote my own. It is for ftp link, just replace ftp with http if your need it.
First line is a small validation of link, link should look like ftp://user:pass#host.com/path/to/something.
if ! echo "$url" | grep -q '^[[:blank:]]*ftp://[[:alnum:]]\+:[[:alnum:]]\+#[[:alnum:]\.]\+/.*[[:blank:]]*$'; then return 1; fi
login=$( echo "$url" | sed 's|[[:blank:]]*ftp://\([^:]\+\):\([^#]\+\)#\([^/]\+\)\(/.*\)[[:blank:]]*|\1|' )
pass=$( echo "$url" | sed 's|[[:blank:]]*ftp://\([^:]\+\):\([^#]\+\)#\([^/]\+\)\(/.*\)[[:blank:]]*|\2|' )
host=$( echo "$url" | sed 's|[[:blank:]]*ftp://\([^:]\+\):\([^#]\+\)#\([^/]\+\)\(/.*\)[[:blank:]]*|\3|' )
dir=$( echo "$url" | sed 's|[[:blank:]]*ftp://\([^:]\+\):\([^#]\+\)#\([^/]\+\)\(/.*\)[[:blank:]]*|\4|' )
My actual goal was to check ftp access by url. Here is the full result:
#!/bin/bash
test_ftp_url() # lftp may hang on some ftp problems, like no connection
{
local url="$1"
if ! echo "$url" | grep -q '^[[:blank:]]*ftp://[[:alnum:]]\+:[[:alnum:]]\+#[[:alnum:]\.]\+/.*[[:blank:]]*$'; then return 1; fi
local login=$( echo "$url" | sed 's|[[:blank:]]*ftp://\([^:]\+\):\([^#]\+\)#\([^/]\+\)\(/.*\)[[:blank:]]*|\1|' )
local pass=$( echo "$url" | sed 's|[[:blank:]]*ftp://\([^:]\+\):\([^#]\+\)#\([^/]\+\)\(/.*\)[[:blank:]]*|\2|' )
local host=$( echo "$url" | sed 's|[[:blank:]]*ftp://\([^:]\+\):\([^#]\+\)#\([^/]\+\)\(/.*\)[[:blank:]]*|\3|' )
local dir=$( echo "$url" | sed 's|[[:blank:]]*ftp://\([^:]\+\):\([^#]\+\)#\([^/]\+\)\(/.*\)[[:blank:]]*|\4|' )
exec 3>&2 2>/dev/null
exec 6<>"/dev/tcp/$host/21" || { exec 2>&3 3>&-; echo 'Bash network support is disabled. Skipping ftp check.'; return 0; }
read <&6
if ! echo "${REPLY//$'\r'}" | grep -q '^220'; then exec 2>&3 3>&- 6>&-; return 3; fi # 220 vsFTPd 3.0.2+ (ext.1) ready...
echo -e "USER $login\r" >&6; read <&6
if ! echo "${REPLY//$'\r'}" | grep -q '^331'; then exec 2>&3 3>&- 6>&-; return 4; fi # 331 Please specify the password.
echo -e "PASS $pass\r" >&6; read <&6
if ! echo "${REPLY//$'\r'}" | grep -q '^230'; then exec 2>&3 3>&- 6>&-; return 5; fi # 230 Login successful.
echo -e "CWD $dir\r" >&6; read <&6
if ! echo "${REPLY//$'\r'}" | grep -q '^250'; then exec 2>&3 3>&- 6>&-; return 6; fi # 250 Directory successfully changed.
echo -e "QUIT\r" >&6
exec 2>&3 3>&- 6>&-
return 0
}
test_ftp_url 'ftp://fz223free:fz223free#ftp.zakupki.gov.ru/out/nsi/nsiProtocol/daily'
echo "$?"

I found Adam Ryczkowski's answers helpful. The original solution did not handle /path in URL, so I enhanced it a little bit.
pattern='^(([[:alnum:]]+):\/\/)?(([[:alnum:]]+)#)?([^:^#\/]+)(:([[:digit:]]+))?(\/?[^:^#]?)$'
url="http://us#cos.com:3142/path"
if [[ "$url" =~ $pattern ]]; then
proto=${BASH_REMATCH[2]}
user=${BASH_REMATCH[4]}
host=${BASH_REMATCH[5]}
port=${BASH_REMATCH[7]}
path=${BASH_REMATCH[8]}
echo "proto: $proto"
echo "user: $user"
echo "host: $host"
echo "port: $port"
echo "path= $path"
else
echo "URL did not match pattern: $url"
fi
The pattern is complex, so please use this site to understand it better: https://regex101.com/
I tested it with a bunch of URLs. However, if there are any issues, please let me know.

If you have access to Node.js:
export MY_URI=sftp://user#host.net/some/random/path
node -e "console.log(url.parse(process.env.MY_URI).user)"
node -e "console.log(url.parse(process.env.MY_URI).host)"
node -e "console.log(url.parse(process.env.MY_URI).path)"
This will output:
user
host.net
/some/random/path

Here's a pure bash url parser. It supports git ssh clone style URLs as well as standard proto:// ones. The example ignores protocol, auths, and port but you can modify to collect as needed... I used regex101 for handy testing: https://regex101.com/r/5QyNI5/1
TEST_URLS=(
https://github.com/briceburg/tools.git
https://foo:12333#github.com:8080/briceburg/tools.git
git#github.com:briceburg/tools.git
https://me#gmail.com:12345#my.site.com:443/p/a/t/h
)
for url in "${TEST_URLS[#]}"; do
without_proto="${url#*:\/\/}"
without_auth="${without_proto##*#}"
[[ $without_auth =~ ^([^:\/]+)(:[[:digit:]]+\/|:|\/)?(.*) ]]
PROJECT_HOST="${BASH_REMATCH[1]}"
PROJECT_PATH="${BASH_REMATCH[3]}"
echo "given: $url"
echo " -> host: $PROJECT_HOST path: $PROJECT_PATH"
done
results in:
given: https://github.com/briceburg/tools.git
-> host: github.com path: briceburg/tools.git
given: https://foo:12333#github.com:8080/briceburg/tools.git
-> host: github.com path: briceburg/tools.git
given: git#github.com:briceburg/tools.git
-> host: github.com path: briceburg/tools.git
given: https://me#gmail.com:12345#my.site.com:443/p/a/t/h
-> host: my.site.com path: p/a/t/h

Resources