Rails app fails to deploy when upgrading elastic beanstalk stack - ruby-on-rails
I am upgrading my elastic beanstalk version to Puma with Ruby 2.6 running on 64bit Amazon Linux/2.11.8 via elb's ui. When I do so I get this error. It works if I revert back to platform version 2.11.4
Initialization failed at 2020-08-07T04:41:35Z with exit status 1 and error: Hook /opt/elasticbeanstalk/hooks/preinit/22_gems.sh failed.
++ /opt/elasticbeanstalk/bin/get-config container -k script_dir
+ EB_SCRIPT_DIR=/opt/elasticbeanstalk/support/scripts
++ /opt/elasticbeanstalk/bin/get-config container -k gem_dir
+ EB_GEM_DIR=/opt/elasticbeanstalk/support/gems/puma
+ . /opt/elasticbeanstalk/support/scripts/use-app-ruby.sh
++ . /usr/local/share/chruby/chruby.sh
+++ CHRUBY_VERSION=0.3.9
+++ RUBIES=()
+++ for dir in '"$PREFIX/opt/rubies"' '"$HOME/.rubies"'
+++ [[ -d /opt/rubies ]]
++++ ls -A /opt/rubies
+++ [[ -n ruby-1.9.3-p551
ruby-2.0.0-p648
ruby-2.1.10
ruby-2.2.10
ruby-2.3.8
ruby-2.4.9
ruby-2.5.7
ruby-2.6.5
ruby-current ]]
+++ RUBIES+=("$dir"/*)
+++ for dir in '"$PREFIX/opt/rubies"' '"$HOME/.rubies"'
+++ [[ -d /.rubies ]]
+++ unset dir
+++ cat /etc/elasticbeanstalk/.ruby_version
++ chruby 2.6.6
++ case "$1" in
++ local dir match
++ for dir in '"${RUBIES[#]}"'
++ dir=/opt/rubies/ruby-1.9.3-p551
++ case "${dir##*/}" in
++ for dir in '"${RUBIES[#]}"'
++ dir=/opt/rubies/ruby-2.0.0-p648
++ case "${dir##*/}" in
++ for dir in '"${RUBIES[#]}"'
++ dir=/opt/rubies/ruby-2.1.10
++ case "${dir##*/}" in
++ for dir in '"${RUBIES[#]}"'
++ dir=/opt/rubies/ruby-2.2.10
++ case "${dir##*/}" in
++ for dir in '"${RUBIES[#]}"'
++ dir=/opt/rubies/ruby-2.3.8
++ case "${dir##*/}" in
++ for dir in '"${RUBIES[#]}"'
++ dir=/opt/rubies/ruby-2.4.9
++ case "${dir##*/}" in
++ for dir in '"${RUBIES[#]}"'
++ dir=/opt/rubies/ruby-2.5.7
++ case "${dir##*/}" in
++ for dir in '"${RUBIES[#]}"'
++ dir=/opt/rubies/ruby-2.6.5
++ case "${dir##*/}" in
++ for dir in '"${RUBIES[#]}"'
++ dir=/opt/rubies/ruby-current
++ case "${dir##*/}" in
++ [[ -z '' ]]
++ echo 'chruby: unknown Ruby: 2.6.6'
chruby: unknown Ruby: 2.6.6
++ return 1.
Process default has been unhealthy for 34 minutes (Target.FailedHealthChecks).
How do I go about debugging this? The rails app is running on ruby 2.6.6.
You probably need to use a ruby in the available rubies (like 2.6.5) or figure out how to make other rubies available.
Related
docker run apache spark using Git Bash terminal
I am running native Windows, and primarily using Git Bash as my terminal for programming.
I am trying to run the command
winpty docker run -it apache/spark-py /opt/spark/bin/pyspark
This works fine in the regular Windows Command Prompt terminal (without winpty), but it does not work in Git Bash. The error it throws back is
$ winpty docker run -it apache/spark-py /opt/spark/bin/pyspark
++ id -u
+ myuid=185
++ id -g
+ mygid=0
+ set +e
++ getent passwd 185
+ uidentry=
+ set -e
+ '[' -z '' ']'
+ '[' -w /etc/passwd ']'
+ echo '185:x:185:0:anonymous uid:/opt/spark:/bin/false'
+ '[' -z /usr/local/openjdk-11 ']'
+ SPARK_CLASSPATH=':/opt/spark/jars/*'
+ env
+ grep SPARK_JAVA_OPT_
+ sort -t_ -k4 -n
+ sed 's/[^=]*=\(.*\)/\1/g'
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS
+ '[' -n '' ']'
+ '[' -z ']'
+ '[' -z ']'
+ '[' -n '' ']'
+ '[' -z ']'
+ '[' -z ']'
+ '[' -z x ']'
+ SPARK_CLASSPATH='/opt/spark/conf::/opt/spark/jars/*'
+ case "$1" in
+ echo 'Non-spark-on-k8s command provided, proceeding in pass-through mode...'
Non-spark-on-k8s command provided, proceeding in pass-through mode...
+ CMD=("$#")
+ exec /usr/bin/tini -s -- C:/Users/xxxxxxx/AppData/Local/Programs/Git/opt/spark/bin/pyspark
[FATAL tini (14)] exec C:/Users/xxxxxxx/AppData/Local/Programs/Git/opt/spark/bin/pyspark failed: No such file or directory
It's like it somehow uses the Windows Git /opt/spark/bin/pyspark that I feed in to the docker run command. Any ideas?
Cloning environment in elastic beanstalk
I am wanting to update my Rails EB Linux to 2.12.2 from 1.11.8, so I cloned the environment and committed to it but I am getting this error:
PG::ConnectionBad (could not connect to server: Connection timed out
Is the server running on host "example.ccexample.us-east-1.rds.amazonaws.com" (111.11.21.22) and accepting
TCP/IP connections on port 5432?
Another error -- likely the same issue?:
/opt/elastticbeanstalk/hooks/appdeploy/pre/12_db_migration.sh failed.
My env variables are all correct so shouldn't the database just simply connect?
This is the error log:
[2020-12-31T22:05:28.834Z] INFO [5012] - [Application update app-example/AppDeployStage0/AppDeployPreHook/12_db_migration.sh] : Starting activity...
[2020-12-31T22:07:45.564Z] INFO [5012] - [Application update example/AppDeployStage0/AppDeployPreHook/12_db_migration.sh] : Activity execution failed, because: ++ /opt/elasticbeanstalk/bin/get-config container -k script_dir
+ EB_SCRIPT_DIR=/opt/elasticbeanstalk/support/scripts
++ /opt/elasticbeanstalk/bin/get-config container -k app_staging_dir
+ EB_APP_STAGING_DIR=/var/app/ondeck
++ /opt/elasticbeanstalk/bin/get-config container -k app_user
+ EB_APP_USER=webapp
++ /opt/elasticbeanstalk/bin/get-config container -k support_dir
+ EB_SUPPORT_DIR=/opt/elasticbeanstalk/support
+ . /opt/elasticbeanstalk/support/envvars-wrapper.sh
+++ /opt/elasticbeanstalk/bin/get-config container -k support_dir
++ EB_SUPPORT_DIR=/opt/elasticbeanstalk/support
++ set +x
+ RAKE_TASK=db:migrate
+ . /opt/elasticbeanstalk/support/scripts/use-app-ruby.sh
++ . /usr/local/share/chruby/chruby.sh
+++ CHRUBY_VERSION=0.3.9
+++ RUBIES=()
+++ for dir in '"$PREFIX/opt/rubies"' '"$HOME/.rubies"'
+++ [[ -d /opt/rubies ]]
++++ ls -A /opt/rubies
+++ [[ -n ruby-2.4.10
ruby-2.5.8
ruby-2.6.6
ruby-current ]]
+++ RUBIES+=("$dir"/*)
+++ for dir in '"$PREFIX/opt/rubies"' '"$HOME/.rubies"'
+++ [[ -d /.rubies ]]
+++ unset dir
+++ cat /etc/elasticbeanstalk/.ruby_version
++ chruby 2.5.8
++ case "$1" in
++ local dir match
++ for dir in '"${RUBIES[#]}"'
++ dir=/opt/rubies/ruby-2.4.10
++ case "${dir##*/}" in
++ for dir in '"${RUBIES[#]}"'
++ dir=/opt/rubies/ruby-2.5.8
++ case "${dir##*/}" in
++ match=/opt/rubies/ruby-2.5.8
++ for dir in '"${RUBIES[#]}"'
++ dir=/opt/rubies/ruby-2.6.6
++ case "${dir##*/}" in
++ for dir in '"${RUBIES[#]}"'
++ dir=/opt/rubies/ruby-current
++ case "${dir##*/}" in
++ [[ -z /opt/rubies/ruby-2.5.8 ]]
++ shift
++ chruby_use /opt/rubies/ruby-2.5.8 ''
++ [[ ! -x /opt/rubies/ruby-2.5.8/bin/ruby ]]
++ [[ -n '' ]]
++ export RUBY_ROOT=/opt/rubies/ruby-2.5.8
++ RUBY_ROOT=/opt/rubies/ruby-2.5.8
++ export RUBYOPT=
++ RUBYOPT=
++ export PATH=/opt/rubies/ruby-2.5.8/bin:/opt/elasticbeanstalk/lib/ruby/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
++ PATH=/opt/rubies/ruby-2.5.8/bin:/opt/elasticbeanstalk/lib/ruby/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
+++ /opt/rubies/ruby-2.5.8/bin/ruby -
++ eval 'export RUBY_ENGINE=ruby;
export RUBY_VERSION=2.5.8;
export GEM_ROOT="/opt/rubies/ruby-2.5.8/lib/ruby/gems/2.5.0";'
+++ export RUBY_ENGINE=ruby
+++ RUBY_ENGINE=ruby
+++ export RUBY_VERSION=2.5.8
+++ RUBY_VERSION=2.5.8
+++ export GEM_ROOT=/opt/rubies/ruby-2.5.8/lib/ruby/gems/2.5.0
+++ GEM_ROOT=/opt/rubies/ruby-2.5.8/lib/ruby/gems/2.5.0
++ (( 0 != 0 ))
+ cd /var/app/ondeck
+ su -s /bin/bash -c 'bundle exec /opt/elasticbeanstalk/support/scripts/check-for-rake-task.rb db:migrate' webapp
I also updated the config.yml with the new environment names:
branch-defaults:
master:
environment: NewName
environment-defaults:
NewName:
branch: null
repository: null
RevoltVendor-env:
branch: null
repository: null
global:
application_name: App Name
default_ec2_keyname: null
default_platform: Puma with Ruby 2.5 running on 64bit Amazon Linux
default_region: us-east-1
include_git_submodules: true
instance_profile: null
platform_name: null
platform_version: null
profile: eb-cli
sc: git
workspace_type: Application
Any help would be appreciated
Based on the comments. The issue was caused by wrong security group (SG) inbound rules in RDS. The EB cloning operation creating new SG, which was not reflected in the SG of the RDS. The solution was to update the SG of the RDS and add the SG associated with the cloned EB environment.
Spark on Kubernetes : /opt/entrypoint.sh: /sbin/tini: No such file or directory error
I'm trying to run a Spark jar on Kubernetes. I have built my own docker image using the template spark-2.4.4-bin-hadoop2.7 and tried to run my yaml file with this docker image.
I have got below error from driver log /opt/entrypoint.sh:
line 133: /sbin/tini: No such file or directory.
I understand that /sbin/tini is not present in docker image. Anybody could help me solve this issue? How to get this tini in docker image path?
command entered:
kubectl create -f spark.yaml
driver log:
kubectl logs spark-wordcount-7-driver
++ id -u
+ myuid=0
++ id -g
+ mygid=0
+ set +e
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/bash
+ set -e
+ '[' -z root:x:0:0:root:/root:/bin/bash ']'
+ SPARK_K8S_CMD=driver
+ case "$SPARK_K8S_CMD" in
+ shift 1
+ SPARK_CLASSPATH=':/opt/spark/jars/*'
+ grep SPARK_JAVA_OPT_
+ env
+ sort -t_ -k4 -n
+ sed 's/[^=]*=\(.*\)/\1/g'
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS
+ '[' -n '' ']'
+ '[' -n '' ']'
+ PYSPARK_ARGS=
+ '[' -n '' ']'
+ R_ARGS=
+ '[' -n '' ']'
+ '[' '' == 2 ']'
+ '[' '' == 3 ']'
+ case "$SPARK_K8S_CMD" in
+ CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$#")
+ exec /sbin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=172.17.0.6 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class com.walmart.WordCount spark-internal
/opt/entrypoint.sh: line 133: /sbin/tini: No such file or directory
Since, you are using alpine image, you have to use /sbin/tini NOTE: alpine has moved tini to /sbin/tini In file: entrypoint.sh: Do the following change exec /usr/bin/tini -s -- /usr/bin/spark-operator "$#" to exec /sbin/tini -s -- /usr/bin/spark-operator "$#"
Configure Jenkins with bitbucket for running Test
I am planning to run build and test cases and deploy using jenkins. I have installed Jenkins and creating job.
I have bitbucket repository with mercurial, So configured mercurial and build clone repository and do nothing else. Now I am writing commands in shell for given purpose:
source ~/.profile # load profile and working fine
mkvirtualenv test_build # create virtual environment using virtualenv wrapper. this fails with trace provided.
cd my_project # move to project directory
pip install -r requirements.txt # install packages using pip
Here is an trace for build console on jenkins.
[workspace] $ /usr/local/bin/bash -xe /tmp/hudson3781010986042746968.sh
+ source /usr/local/jenkins/.profile
++ PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/usr/local/jenkins/bin
++ export PATH
++ BLOCKSIZE=K
++ export BLOCKSIZE
++ EDITOR=vi
++ export EDITOR
++ PAGER=more
++ export PAGER
++ ENV=/usr/local/jenkins/.shrc
++ export ENV
++ '[' -x /usr/games/fortune ']'
++ '[' -e /usr/local/bin/virtualenvwrapper.sh ']'
++ export WORKON_HOME=/usr/local/jenkins/virtualenvs
++ WORKON_HOME=/usr/local/jenkins/virtualenvs
++ source /usr/local/bin/virtualenvwrapper.sh
+++ '[' '' = '' ']'
++++ command which python
++++ which python
+++ VIRTUALENVWRAPPER_PYTHON=/usr/local/bin/python
+++ '[' '' = '' ']'
+++ VIRTUALENVWRAPPER_VIRTUALENV=virtualenv
+++ '[' '' = '' ']'
+++ VIRTUALENVWRAPPER_VIRTUALENV_CLONE=virtualenv-clone
+++ VIRTUALENVWRAPPER_ENV_BIN_DIR=bin
+++ '[' '' = Windows_NT ']'
+++ '[' .project = '' ']'
+++ virtualenvwrapper_initialize
++++ virtualenvwrapper_derive_workon_home
++++ typeset workon_home_dir=/usr/local/jenkins/virtualenvs
++++ '[' /usr/local/jenkins/virtualenvs = '' ']'
++++ echo /usr/local/jenkins/virtualenvs
++++ unset GREP_OPTIONS
++++ command grep '^[^/~]'
++++ grep '^[^/~]'
++++ echo /usr/local/jenkins/virtualenvs
++++ unset GREP_OPTIONS
++++ command egrep '([\$~]|//)'
++++ egrep '([\$~]|//)'
++++ echo /usr/local/jenkins/virtualenvs
++++ return 0
+++ export WORKON_HOME=/usr/local/jenkins/virtualenvs
+++ WORKON_HOME=/usr/local/jenkins/virtualenvs
+++ virtualenvwrapper_verify_workon_home -q
+++ RC=0
+++ '[' '!' -d /usr/local/jenkins/virtualenvs/ ']'
+++ return 0
+++ '[' /usr/local/jenkins/virtualenvs = '' ']'
+++ '[' /usr/local/jenkins/virtualenvs = '' ']'
+++ virtualenvwrapper_run_hook initialize
+++ typeset hook_script
+++ typeset result
++++ virtualenvwrapper_tempfile initialize-hook
++++ typeset suffix=initialize-hook
++++ typeset file
+++++ virtualenvwrapper_mktemp -t virtualenvwrapper-initialize-hook-XXXXXXXXXX
+++++ command mktemp -t virtualenvwrapper-initialize-hook-XXXXXXXXXX
+++++ mktemp -t virtualenvwrapper-initialize-hook-XXXXXXXXXX
++++ file=/tmp/virtualenvwrapper-initialize-hook-XXXXXXXXXX.jKEvY7Y1
++++ '[' 0 -ne 0 ']'
++++ '[' -z /tmp/virtualenvwrapper-initialize-hook-XXXXXXXXXX.jKEvY7Y1 ']'
++++ '[' '!' -f /tmp/virtualenvwrapper-initialize-hook-XXXXXXXXXX.jKEvY7Y1 ']'
++++ echo /tmp/virtualenvwrapper-initialize-hook-XXXXXXXXXX.jKEvY7Y1
++++ return 0
+++ hook_script=/tmp/virtualenvwrapper-initialize-hook-XXXXXXXXXX.jKEvY7Y1
+++ '[' -z /usr/local/jenkins/virtualenvs ']'
+++ /usr/local/bin/python -c 'from virtualenvwrapper.hook_loader import main; main()' --script /tmp/virtualenvwrapper-initialize-hook-XXXXXXXXXX.jKEvY7Y1 initialize
+++ result=0
+++ '[' 0 -eq 0 ']'
+++ '[' '!' -f /tmp/virtualenvwrapper-initialize-hook-XXXXXXXXXX.jKEvY7Y1 ']'
+++ source /tmp/virtualenvwrapper-initialize-hook-XXXXXXXXXX.jKEvY7Y1
++++ '[' -f /usr/local/jenkins/virtualenvs/initialize ']'
++++ source /usr/local/jenkins/virtualenvs/initialize
+++ command rm -f /tmp/virtualenvwrapper-initialize-hook-XXXXXXXXXX.jKEvY7Y1
+++ rm -f /tmp/virtualenvwrapper-initialize-hook-XXXXXXXXXX.jKEvY7Y1
+++ return 0
+++ virtualenvwrapper_setup_tab_completion
+++ '[' -n /usr/local/bin/bash ']'
+++ complete -o nospace -F _cdvirtualenv_complete -S/ cdvirtualenv
+++ complete -o nospace -F _cdsitepackages_complete -S/ cdsitepackages
+++ complete -o default -o nospace -F _virtualenvs workon
+++ complete -o default -o nospace -F _virtualenvs rmvirtualenv
+++ complete -o default -o nospace -F _virtualenvs cpvirtualenv
+++ complete -o default -o nospace -F _virtualenvs showvirtualenv
+++ return 0
+ mkvirtualenv test_build
+ typeset -a in_args
+ typeset -a out_args
+ typeset -i i
+ typeset tst
+ typeset a
+ typeset envname
+ typeset requirements
+ typeset packages
+ in_args=("$#")
+ '[' -n '' ']'
+ i=0
+ tst=-lt
+ '[' 0 -lt 1 ']'
+ a=test_build
+ case "$a" in
+ '[' 0 -gt 0 ']'
+ out_args=("$a")
+ i=1
+ '[' 1 -lt 1 ']'
+ set -- test_build
+ eval 'envname=$1'
++ envname=test_build
+ virtualenvwrapper_verify_workon_home
+ RC=0
+ '[' '!' -d /usr/local/jenkins/virtualenvs/ ']'
+ return 0
+ virtualenvwrapper_verify_virtualenv
+ virtualenvwrapper_verify_resource virtualenv
++ command which virtualenv
++ which virtualenv
++ unset GREP_OPTIONS
++ command grep -v 'not found'
++ grep -v 'not found'
+ typeset exe_path=/usr/local/bin/virtualenv
+ '[' /usr/local/bin/virtualenv = '' ']'
+ '[' '!' -e /usr/local/bin/virtualenv ']'
+ return 0
+ '[' -n '' ']'
+ virtualenvwrapper_cd /usr/local/jenkins/virtualenvs
+ '[' -n /usr/local/bin/bash ']'
+ builtin cd /usr/local/jenkins/virtualenvs
+ virtualenv test_build
New python executable in test_build/bin/python2.7
Also creating executable in test_build/bin/python
Installing Setuptools..............................................................................................................................................................................................................................done.
Installing Pip.....................................................................................................................................................................................................................................................................................................................................done.
+ '[' -d /usr/local/jenkins/virtualenvs/test_build ']'
+ virtualenvwrapper_run_hook pre_mkvirtualenv test_build
+ typeset hook_script
+ typeset result
++ virtualenvwrapper_tempfile pre_mkvirtualenv-hook
++ typeset suffix=pre_mkvirtualenv-hook
++ typeset file
+++ virtualenvwrapper_mktemp -t virtualenvwrapper-pre_mkvirtualenv-hook-XXXXXXXXXX
+++ command mktemp -t virtualenvwrapper-pre_mkvirtualenv-hook-XXXXXXXXXX
+++ mktemp -t virtualenvwrapper-pre_mkvirtualenv-hook-XXXXXXXXXX
++ file=/tmp/virtualenvwrapper-pre_mkvirtualenv-hook-XXXXXXXXXX.rPgLzOe4
++ '[' 0 -ne 0 ']'
++ '[' -z /tmp/virtualenvwrapper-pre_mkvirtualenv-hook-XXXXXXXXXX.rPgLzOe4 ']'
++ '[' '!' -f /tmp/virtualenvwrapper-pre_mkvirtualenv-hook-XXXXXXXXXX.rPgLzOe4 ']'
++ echo /tmp/virtualenvwrapper-pre_mkvirtualenv-hook-XXXXXXXXXX.rPgLzOe4
++ return 0
+ hook_script=/tmp/virtualenvwrapper-pre_mkvirtualenv-hook-XXXXXXXXXX.rPgLzOe4
+ '[' -z /usr/local/jenkins/virtualenvs ']'
+ /usr/local/bin/python -c 'from virtualenvwrapper.hook_loader import main; main()' --script /tmp/virtualenvwrapper-pre_mkvirtualenv-hook-XXXXXXXXXX.rPgLzOe4 pre_mkvirtualenv test_build
virtualenvwrapper.user_scripts creating /usr/local/jenkins/virtualenvs/test_build/bin/predeactivate
virtualenvwrapper.user_scripts creating /usr/local/jenkins/virtualenvs/test_build/bin/postdeactivate
virtualenvwrapper.user_scripts creating /usr/local/jenkins/virtualenvs/test_build/bin/preactivate
virtualenvwrapper.user_scripts creating /usr/local/jenkins/virtualenvs/test_build/bin/postactivate
virtualenvwrapper.user_scripts creating /usr/local/jenkins/virtualenvs/test_build/bin/get_env_details
+ result=0
+ '[' 0 -eq 0 ']'
+ '[' '!' -f /tmp/virtualenvwrapper-pre_mkvirtualenv-hook-XXXXXXXXXX.rPgLzOe4 ']'
+ source /tmp/virtualenvwrapper-pre_mkvirtualenv-hook-XXXXXXXXXX.rPgLzOe4
+ command rm -f /tmp/virtualenvwrapper-pre_mkvirtualenv-hook-XXXXXXXXXX.rPgLzOe4
+ rm -f /tmp/virtualenvwrapper-pre_mkvirtualenv-hook-XXXXXXXXXX.rPgLzOe4
+ return 0
+ typeset RC=0
+ '[' 0 -ne 0 ']'
+ '[' '!' -d /usr/local/jenkins/virtualenvs/test_build ']'
+ '[' '!' -z '' ']'
+ workon test_build
+ typeset env_name=test_build
+ '[' test_build = '' ']'
+ virtualenvwrapper_verify_workon_home
+ RC=0
+ '[' '!' -d /usr/local/jenkins/virtualenvs/ ']'
+ return 0
+ virtualenvwrapper_verify_workon_environment test_build
+ typeset env_name=test_build
+ '[' '!' -d /usr/local/jenkins/virtualenvs/test_build ']'
+ return 0
+ activate=/usr/local/jenkins/virtualenvs/test_build/bin/activate
+ '[' '!' -f /usr/local/jenkins/virtualenvs/test_build/bin/activate ']'
+ type deactivate
Build step 'Execute shell' marked build as failure
Finished: FAILURE
Parse URL in shell script
I have url like: sftp://user#host.net/some/random/path I want to extract user, host and path from this string. Any part can be random length.
[EDIT 2019]
This answer is not meant to be a catch-all, works for everything solution it was intended to provide a simple alternative to the python based version and it ended up having more features than the original.
It answered the basic question in a bash-only way and then was modified multiple times by myself to include a hand full of demands by commenters. I think at this point however adding even more complexity would make it unmaintainable. I know not all things are straight forward (checking for a valid port for example requires comparing hostport and host) but I would rather not add even more complexity.
[Original answer]
Assuming your URL is passed as first parameter to the script:
#!/bin/bash
# extract the protocol
proto="$(echo $1 | grep :// | sed -e's,^\(.*://\).*,\1,g')"
# remove the protocol
url="$(echo ${1/$proto/})"
# extract the user (if any)
user="$(echo $url | grep # | cut -d# -f1)"
# extract the host and port
hostport="$(echo ${url/$user#/} | cut -d/ -f1)"
# by request host without port
host="$(echo $hostport | sed -e 's,:.*,,g')"
# by request - try to extract the port
port="$(echo $hostport | sed -e 's,^.*:,:,g' -e 's,.*:\([0-9]*\).*,\1,g' -e 's,[^0-9],,g')"
# extract the path (if any)
path="$(echo $url | grep / | cut -d/ -f2-)"
echo "url: $url"
echo " proto: $proto"
echo " user: $user"
echo " host: $host"
echo " port: $port"
echo " path: $path"
I must admit this is not the cleanest solution but it doesn't rely on another scripting
language like perl or python.
(Providing a solution using one of them would produce cleaner results ;) )
Using your example the results are:
url: user#host.net/some/random/path
proto: sftp://
user: user
host: host.net
port:
path: some/random/path
This will also work for URLs without a protocol/username or path.
In this case the respective variable will contain an empty string.
[EDIT]
If your bash version won't cope with the substitutions (${1/$proto/}) try this:
#!/bin/bash
# extract the protocol
proto="$(echo $1 | grep :// | sed -e's,^\(.*://\).*,\1,g')"
# remove the protocol -- updated
url=$(echo $1 | sed -e s,$proto,,g)
# extract the user (if any)
user="$(echo $url | grep # | cut -d# -f1)"
# extract the host and port -- updated
hostport=$(echo $url | sed -e s,$user#,,g | cut -d/ -f1)
# by request host without port
host="$(echo $hostport | sed -e 's,:.*,,g')"
# by request - try to extract the port
port="$(echo $hostport | sed -e 's,^.*:,:,g' -e 's,.*:\([0-9]*\).*,\1,g' -e 's,[^0-9],,g')"
# extract the path (if any)
path="$(echo $url | grep / | cut -d/ -f2-)"
The above, refined (added password and port parsing), and working in /bin/sh: # extract the protocol proto="`echo $DATABASE_URL | grep '://' | sed -e's,^\(.*://\).*,\1,g'`" # remove the protocol url=`echo $DATABASE_URL | sed -e s,$proto,,g` # extract the user and password (if any) userpass="`echo $url | grep # | cut -d# -f1`" pass=`echo $userpass | grep : | cut -d: -f2` if [ -n "$pass" ]; then user=`echo $userpass | grep : | cut -d: -f1` else user=$userpass fi # extract the host -- updated hostport=`echo $url | sed -e s,$userpass#,,g | cut -d/ -f1` port=`echo $hostport | grep : | cut -d: -f2` if [ -n "$port" ]; then host=`echo $hostport | grep : | cut -d: -f1` else host=$hostport fi # extract the path (if any) path="`echo $url | grep / | cut -d/ -f2-`" Posted b/c I needed it, so I wrote it (based on #Shirkin's answer, obviously), and I figured someone else might appreciate it.
This solution in principle works the same as Adam Ryczkowski's, in this thread - but has improved regular expression based on RFC3986, (with some changes) and fixes some errors (e.g. userinfo can contain '_' character). This can also understand relative URIs (e.g. to extract query or fragment).
# !/bin/bash
# Following regex is based on https://www.rfc-editor.org/rfc/rfc3986#appendix-B with
# additional sub-expressions to split authority into userinfo, host and port
#
readonly URI_REGEX='^(([^:/?#]+):)?(//((([^:/?#]+)#)?([^:/?#]+)(:([0-9]+))?))?(/([^?#]*))(\?([^#]*))?(#(.*))?'
# ↑↑ ↑ ↑↑↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑
# |2 scheme | ||6 userinfo 7 host | 9 port | 11 rpath | 13 query | 15 fragment
# 1 scheme: | |5 userinfo# 8 :… 10 path 12 ?… 14 #…
# | 4 authority
# 3 //…
parse_scheme () {
[[ "$#" =~ $URI_REGEX ]] && echo "${BASH_REMATCH[2]}"
}
parse_authority () {
[[ "$#" =~ $URI_REGEX ]] && echo "${BASH_REMATCH[4]}"
}
parse_user () {
[[ "$#" =~ $URI_REGEX ]] && echo "${BASH_REMATCH[6]}"
}
parse_host () {
[[ "$#" =~ $URI_REGEX ]] && echo "${BASH_REMATCH[7]}"
}
parse_port () {
[[ "$#" =~ $URI_REGEX ]] && echo "${BASH_REMATCH[9]}"
}
parse_path () {
[[ "$#" =~ $URI_REGEX ]] && echo "${BASH_REMATCH[10]}"
}
parse_rpath () {
[[ "$#" =~ $URI_REGEX ]] && echo "${BASH_REMATCH[11]}"
}
parse_query () {
[[ "$#" =~ $URI_REGEX ]] && echo "${BASH_REMATCH[13]}"
}
parse_fragment () {
[[ "$#" =~ $URI_REGEX ]] && echo "${BASH_REMATCH[15]}"
}
Using Python (best tool for this job, IMHO):
#!/usr/bin/env python
import os
from urlparse import urlparse
uri = os.environ['NAUTILUS_SCRIPT_CURRENT_URI']
result = urlparse(uri)
user, host = result.netloc.split('#')
path = result.path
print('user=', user)
print('host=', host)
print('path=', path)
Further reading:
os.environ
urlparse.urlparse()
If you really want to do it in shell, you can do something as simple as the following by using awk. This requires knowing how many fields you will actually be passed (e.g. no password sometimes and not others).
#!/bin/bash
FIELDS=($(echo "sftp://user#host.net/some/random/path" \
| awk '{split($0, arr, /[\/\#:]*/); for (x in arr) { print arr[x] }}'))
proto=${FIELDS[1]}
user=${FIELDS[2]}
host=${FIELDS[3]}
path=$(echo ${FIELDS[#]:3} | sed 's/ /\//g')
If you don't have awk and you do have grep, and you can require that each field have at least two characters and be reasonably predictable in format, then you can do:
#!/bin/bash
FIELDS=($(echo "sftp://user#host.net/some/random/path" \
| grep -o "[a-z0-9.-][a-z0-9.-]*" | tr '\n' ' '))
proto=${FIELDS[1]}
user=${FIELDS[2]}
host=${FIELDS[3]}
path=$(echo ${FIELDS[#]:3} | sed 's/ /\//g')
Just needed to do the same, so was curious if it's possible to do it in single line, and this is what i've got:
#!/bin/bash
parse_url() {
eval $(echo "$1" | sed -e "s#^\(\(.*\)://\)\?\(\([^:#]*\)\(:\(.*\)\)\?#\)\?\([^/?]*\)\(/\(.*\)\)\?#${PREFIX:-URL_}SCHEME='\2' ${PREFIX:-URL_}USER='\4' ${PREFIX:-URL_}PASSWORD='\6' ${PREFIX:-URL_}HOST='\7' ${PREFIX:-URL_}PATH='\9'#")
}
URL=${1:-"http://user:pass#example.com/path/somewhere"}
PREFIX="URL_" parse_url "$URL"
echo "$URL_SCHEME://$URL_USER:$URL_PASSWORD#$URL_HOST/$URL_PATH"
How it works:
There is that crazy sed regex that captures all the parts of url, when all of them are optional (except for the host name)
Using those capture groups sed outputs env variables names with their values for relevant parts (like URL_SCHEME or URL_USER)
eval executes that output, causing those variables to be exported and available in the script
Optionally PREFIX could be passed to control output env variables names
PS: be careful when using this for arbitrary input since this code is vulnerable to script injections.
Here's my take, loosely based on some of the existing answers, but it can also cope with GitHub SSH clone URLs:
#!/bin/bash
PROJECT_URL="git#github.com:heremaps/here-aaa-java-sdk.git"
# Extract the protocol (includes trailing "://").
PARSED_PROTO="$(echo $PROJECT_URL | sed -nr 's,^(.*://).*,\1,p')"
# Remove the protocol from the URL.
PARSED_URL="$(echo ${PROJECT_URL/$PARSED_PROTO/})"
# Extract the user (includes trailing "#").
PARSED_USER="$(echo $PARSED_URL | sed -nr 's,^(.*#).*,\1,p')"
# Remove the user from the URL.
PARSED_URL="$(echo ${PARSED_URL/$PARSED_USER/})"
# Extract the port (includes leading ":").
PARSED_PORT="$(echo $PARSED_URL | sed -nr 's,.*(:[0-9]+).*,\1,p')"
# Remove the port from the URL.
PARSED_URL="$(echo ${PARSED_URL/$PARSED_PORT/})"
# Extract the path (includes leading "/" or ":").
PARSED_PATH="$(echo $PARSED_URL | sed -nr 's,[^/:]*([/:].*),\1,p')"
# Remove the path from the URL.
PARSED_HOST="$(echo ${PARSED_URL/$PARSED_PATH/})"
echo "proto: $PARSED_PROTO"
echo "user: $PARSED_USER"
echo "host: $PARSED_HOST"
echo "port: $PARSED_PORT"
echo "path: $PARSED_PATH"
which gives
proto:
user: git#
host: github.com
port:
path: :heremaps/here-aaa-java-sdk.git
And for PROJECT_URL="ssh://sschuberth#git.eclipse.org:29418/jgit/jgit" you get
proto: ssh://
user: sschuberth#
host: git.eclipse.org
port: :29418
path: /jgit/jgit
You can use bash string manipulation. It is easy to learn. In case you feel difficulties with regex, try it. As it is from NAUTILUS_SCRIPT_CURRENT_URI, i guess there may have port in that URI. So I also kept that optional.
#!/bin/bash
#You can also use environment variable $NAUTILUS_SCRIPT_CURRENT_URI
X="sftp://user#host.net/some/random/path"
tmp=${X#*//};usr=${tmp%#*}
tmp=${X#*#};host=${tmp%%/*};[[ ${X#*://} == *":"* ]] && host=${host%:*}
tmp=${X#*//};path=${tmp#*/}
proto=${X%:*}
[[ ${X#*://} == *":"* ]] && tmp=${X##*:} && port=${tmp%%/*}
echo "Potocol:"$proto" User:"$usr" Host:"$host" Port:"$port" Path:"$path
I don't have enough reputation to comment, but I made a small modification to #patryk-obara's answer. RFC3986 § 6.2.3. Scheme-Based Normalization treats http://example.com http://example.com/ as equivalent. But I found that his regex did not match a URL like http://example.com. http://example.com/ (with the trailing slash) does match. I inserted 11, which changed / to (/|$). This matches either / or the end of the string. Now http://example.com does match. readonly URI_REGEX='^(([^:/?#]+):)?(//((([^:/?#]+)#)?([^:/?#]+)(:([0-9]+))?))?((/|$)([^?#]*))(\?([^#]*))?(#(.*))?$' # ↑↑ ↑ ↑↑↑ ↑ ↑ ↑ ↑↑ ↑ ↑ ↑ ↑ ↑ # || | ||| | | | || | | | | | # |2 scheme | ||6 userinfo 7 host | 9 port || 12 rpath | 14 query | 16 fragment # 1 scheme: | |5 userinfo# 8 :... || 13 ?... 15 #... # | 4 authority |11 / or end-of-string # 3 //... 10 path
If you have access to Bash >= 3.0 you can do this in pure bash as well, thanks to the re-match operator =~:
pattern='^(([[:alnum:]]+)://)?(([[:alnum:]]+)#)?([^:^#]+)(:([[:digit:]]+))?$'
if [[ "http://us#cos.com:3142" =~ $pattern ]]; then
proto=${BASH_REMATCH[2]}
user=${BASH_REMATCH[4]}
host=${BASH_REMATCH[5]}
port=${BASH_REMATCH[7]}
fi
It should be faster and less resource-hungry then all the previous examples, because no external process is be spawned.
A simplistic approach to get just the domain from the full URL:
echo https://stackoverflow.com/questions/6174220/parse-url-in-shell-script | cut -d/ -f1-3
# OUTPUT>>> https://stackoverflow.com
Get only the path:
echo https://stackoverflow.com/questions/6174220/parse-url-in-shell-script | cut -d/ -f4-
# OUTPUT>>> questions/6174220/parse-url-in-shell-script
Not perfect, as the second command strips the preceding slash so you'll need to prepend it by hand.
An awk-based approach for getting just the path without the domain:
echo https://stackoverflow.com/questions/6174220/parse-url-in-shell-script/59971653 | awk -F"/" '{ for (i=4; i<=NF; i++) printf"/%s", $i }'
# OUTPUT>>> /questions/6174220/parse-url-in-shell-script/59971653
I did further parsing, expanding the solution given by #Shirkrin:
#!/bin/bash
parse_url() {
local query1 query2 path1 path2
# extract the protocol
proto="$(echo $1 | grep :// | sed -e's,^\(.*://\).*,\1,g')"
if [[ ! -z $proto ]] ; then
# remove the protocol
url="$(echo ${1/$proto/})"
# extract the user (if any)
login="$(echo $url | grep # | cut -d# -f1)"
# extract the host
host="$(echo ${url/$login#/} | cut -d/ -f1)"
# by request - try to extract the port
port="$(echo $host | sed -e 's,^.*:,:,g' -e 's,.*:\([0-9]*\).*,\1,g' -e 's,[^0-9],,g')"
# extract the uri (if any)
resource="/$(echo $url | grep / | cut -d/ -f2-)"
else
url=""
login=""
host=""
port=""
resource=$1
fi
# extract the path (if any)
path1="$(echo $resource | grep ? | cut -d? -f1 )"
path2="$(echo $resource | grep \# | cut -d# -f1 )"
path=$path1
if [[ -z $path ]] ; then path=$path2 ; fi
if [[ -z $path ]] ; then path=$resource ; fi
# extract the query (if any)
query1="$(echo $resource | grep ? | cut -d? -f2-)"
query2="$(echo $query1 | grep \# | cut -d\# -f1 )"
query=$query2
if [[ -z $query ]] ; then query=$query1 ; fi
# extract the fragment (if any)
fragment="$(echo $resource | grep \# | cut -d\# -f2 )"
echo "url: $url"
echo " proto: $proto"
echo " login: $login"
echo " host: $host"
echo " port: $port"
echo "resource: $resource"
echo " path: $path"
echo " query: $query"
echo "fragment: $fragment"
echo ""
}
parse_url "http://login:password#example.com:8080/one/more/dir/file.exe?a=sth&b=sth#anchor_fragment"
parse_url "https://example.com/one/more/dir/file.exe#anchor_fragment"
parse_url "http://login:password#example.com:8080/one/more/dir/file.exe#anchor_fragment"
parse_url "ftp://user#example.com:8080/one/more/dir/file.exe?a=sth&b=sth"
parse_url "/one/more/dir/file.exe"
parse_url "file.exe"
parse_url "file.exe#anchor"
I did not like above methods and wrote my own. It is for ftp link, just replace ftp with http if your need it.
First line is a small validation of link, link should look like ftp://user:pass#host.com/path/to/something.
if ! echo "$url" | grep -q '^[[:blank:]]*ftp://[[:alnum:]]\+:[[:alnum:]]\+#[[:alnum:]\.]\+/.*[[:blank:]]*$'; then return 1; fi
login=$( echo "$url" | sed 's|[[:blank:]]*ftp://\([^:]\+\):\([^#]\+\)#\([^/]\+\)\(/.*\)[[:blank:]]*|\1|' )
pass=$( echo "$url" | sed 's|[[:blank:]]*ftp://\([^:]\+\):\([^#]\+\)#\([^/]\+\)\(/.*\)[[:blank:]]*|\2|' )
host=$( echo "$url" | sed 's|[[:blank:]]*ftp://\([^:]\+\):\([^#]\+\)#\([^/]\+\)\(/.*\)[[:blank:]]*|\3|' )
dir=$( echo "$url" | sed 's|[[:blank:]]*ftp://\([^:]\+\):\([^#]\+\)#\([^/]\+\)\(/.*\)[[:blank:]]*|\4|' )
My actual goal was to check ftp access by url. Here is the full result:
#!/bin/bash
test_ftp_url() # lftp may hang on some ftp problems, like no connection
{
local url="$1"
if ! echo "$url" | grep -q '^[[:blank:]]*ftp://[[:alnum:]]\+:[[:alnum:]]\+#[[:alnum:]\.]\+/.*[[:blank:]]*$'; then return 1; fi
local login=$( echo "$url" | sed 's|[[:blank:]]*ftp://\([^:]\+\):\([^#]\+\)#\([^/]\+\)\(/.*\)[[:blank:]]*|\1|' )
local pass=$( echo "$url" | sed 's|[[:blank:]]*ftp://\([^:]\+\):\([^#]\+\)#\([^/]\+\)\(/.*\)[[:blank:]]*|\2|' )
local host=$( echo "$url" | sed 's|[[:blank:]]*ftp://\([^:]\+\):\([^#]\+\)#\([^/]\+\)\(/.*\)[[:blank:]]*|\3|' )
local dir=$( echo "$url" | sed 's|[[:blank:]]*ftp://\([^:]\+\):\([^#]\+\)#\([^/]\+\)\(/.*\)[[:blank:]]*|\4|' )
exec 3>&2 2>/dev/null
exec 6<>"/dev/tcp/$host/21" || { exec 2>&3 3>&-; echo 'Bash network support is disabled. Skipping ftp check.'; return 0; }
read <&6
if ! echo "${REPLY//$'\r'}" | grep -q '^220'; then exec 2>&3 3>&- 6>&-; return 3; fi # 220 vsFTPd 3.0.2+ (ext.1) ready...
echo -e "USER $login\r" >&6; read <&6
if ! echo "${REPLY//$'\r'}" | grep -q '^331'; then exec 2>&3 3>&- 6>&-; return 4; fi # 331 Please specify the password.
echo -e "PASS $pass\r" >&6; read <&6
if ! echo "${REPLY//$'\r'}" | grep -q '^230'; then exec 2>&3 3>&- 6>&-; return 5; fi # 230 Login successful.
echo -e "CWD $dir\r" >&6; read <&6
if ! echo "${REPLY//$'\r'}" | grep -q '^250'; then exec 2>&3 3>&- 6>&-; return 6; fi # 250 Directory successfully changed.
echo -e "QUIT\r" >&6
exec 2>&3 3>&- 6>&-
return 0
}
test_ftp_url 'ftp://fz223free:fz223free#ftp.zakupki.gov.ru/out/nsi/nsiProtocol/daily'
echo "$?"
I found Adam Ryczkowski's answers helpful. The original solution did not handle /path in URL, so I enhanced it a little bit.
pattern='^(([[:alnum:]]+):\/\/)?(([[:alnum:]]+)#)?([^:^#\/]+)(:([[:digit:]]+))?(\/?[^:^#]?)$'
url="http://us#cos.com:3142/path"
if [[ "$url" =~ $pattern ]]; then
proto=${BASH_REMATCH[2]}
user=${BASH_REMATCH[4]}
host=${BASH_REMATCH[5]}
port=${BASH_REMATCH[7]}
path=${BASH_REMATCH[8]}
echo "proto: $proto"
echo "user: $user"
echo "host: $host"
echo "port: $port"
echo "path= $path"
else
echo "URL did not match pattern: $url"
fi
The pattern is complex, so please use this site to understand it better: https://regex101.com/
I tested it with a bunch of URLs. However, if there are any issues, please let me know.
If you have access to Node.js: export MY_URI=sftp://user#host.net/some/random/path node -e "console.log(url.parse(process.env.MY_URI).user)" node -e "console.log(url.parse(process.env.MY_URI).host)" node -e "console.log(url.parse(process.env.MY_URI).path)" This will output: user host.net /some/random/path
Here's a pure bash url parser. It supports git ssh clone style URLs as well as standard proto:// ones. The example ignores protocol, auths, and port but you can modify to collect as needed... I used regex101 for handy testing: https://regex101.com/r/5QyNI5/1
TEST_URLS=(
https://github.com/briceburg/tools.git
https://foo:12333#github.com:8080/briceburg/tools.git
git#github.com:briceburg/tools.git
https://me#gmail.com:12345#my.site.com:443/p/a/t/h
)
for url in "${TEST_URLS[#]}"; do
without_proto="${url#*:\/\/}"
without_auth="${without_proto##*#}"
[[ $without_auth =~ ^([^:\/]+)(:[[:digit:]]+\/|:|\/)?(.*) ]]
PROJECT_HOST="${BASH_REMATCH[1]}"
PROJECT_PATH="${BASH_REMATCH[3]}"
echo "given: $url"
echo " -> host: $PROJECT_HOST path: $PROJECT_PATH"
done
results in:
given: https://github.com/briceburg/tools.git
-> host: github.com path: briceburg/tools.git
given: https://foo:12333#github.com:8080/briceburg/tools.git
-> host: github.com path: briceburg/tools.git
given: git#github.com:briceburg/tools.git
-> host: github.com path: briceburg/tools.git
given: https://me#gmail.com:12345#my.site.com:443/p/a/t/h
-> host: my.site.com path: p/a/t/h