Bash - Weird white space error when printing variable - ruby-on-rails

I am trying to create a private access token using the rails-terminal for GitLab using a bash script. I got the following script:
#!/bin/bash
password=$(microk8s kubectl exec -it gitlab-6f6ff575cb-zvcph -n project-utility -- /bin/sh -c 'gitlab-rails runner "puts SecureRandom.hex(4)"')
user_token=$(microk8s kubectl exec -it gitlab-6f6ff575cb-zvcph -n project-utility -- /bin/sh -c 'gitlab-rails runner "user = User.new(name: \"Project\", username: \"System\", email: \"system#project.com\", password: \"'"$password"'\"); user.save!; token = user.personal_access_tokens.create(name: \"System\", scopes: [:api]); puts token.token"')
echo "Generated password: $password"
echo "Generated user token: $user_token"
Now this script does everyting correctly except that there is a warning in the response of the user_token. The response is as folows:
Generated password: 9f9857cc
Generated user token: /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.6.1/lib/rails/commands/runner/runn in middle of line, treated as a mere space
glpat-VuHcqsoK8XkAMq1fhVTS
Now I have tried stripping the white-spaces using this:
user_token="${user_token##* }"
user_token="${user_token%% *}"
echo "Generated user token: $user_token"
This removes the warning, but prints a random space word in the output:
Generated password: 8465390c
Generated user token: space
glpat-Rzoz_Zn8xJ_9kVt1dVek
I also tried the following line to strip whitespaces:
user_token="$(echo "$user_token" | tr -d '[:space:]')"
echo "Generated user token: $user_token"
But this returns the warning again, but then without any spaces in the sentence...
The output I am trying to get is this ofcourse:
Generated password: 8465390c
Generated user token: glpat-Rzoz_Zn8xJ_9kVt1dVek
What am I doing wrong?

Your issue is the "${user_token##* }". This ignores the fact that a newline can be included in the output (which it is). So, you are only gobbling everything up until space<newline>glpat-.... To fix this, one option is to use cut:
user_token=$(cut -d $'\n' -f 2 <<< "$user_token")
user_token="${user_token%% *}"
echo "Generated user token: $user_token"
Of course, this assumes the filepath that's printed doesn't have any newlines, but I don't think that's a practical issue.
A better solution is to modify what you are already doing:
user_token="${user_token##*$'\n'}"
user_token="${user_token##* }"
user_token="${user_token%% *}"
echo "Generated user token: $user_token"

Related

Retrieving RSA key from AWS Secrets Manager in CodeBuild corrupts key "invalid format"

During a CodeBuild run I am retrieving a rsa key from SecretsManager, which is the private key to use to access private sources in BitBucket. To do this I have copied the private key into a secret, then in my buildspec file I have the following snippet:
"env": {
"secrets-manager": {
"LOCAL_RSA_VAR": "name-of-secret"
}
},
In the install portion of the buildspec:
"install": {
"commands": [
"echo $LOCAL_RSA_VAR" > ~/.ssh/id_rsa,
"chmod 600 ~/.ssh/id_rsa",
"yarn install"
]
},
HOWEVER, this always ends up with an error:
Load key "/root/.ssh/id_rsa": invalid format
git#bitbucket.org: Permission denied (publickey).
fatal: Could not read from remote repository.
To determine if the key was wrong I tried uploading the rsa_id file into S3 and then download it from there and used it that way using these commands instead:
"install": {
"commands": [
"aws s3 cp s3://the-bucket-name/id_rsa ~/.ssh/id_rsa",
"chmod 600 ~/.ssh/id_rsa",
"yarn install"
]
},
This works fine.
So I guess the question is... Has anyone tried this and had better success? Is there something that I am not doing correctly that you can think of?
I have encountered the same issue.
Copying the id_rsa generated from the the command echo $LOCAL_RSA_VAR > ~/.ssh/id_rsa in S3 I have noticed that the new lines have not been preseved.
I have resolved putting the var env between double quote "":
echo "$LOCAL_RSA_VAR" > ~/.ssh/id_rsa
I was able to get an answer by diff'ing the output of the Env Var vs the File contents from the S3 file. ('cat' will not print out the content of a secret mgr env variable) It ends up content of the env var was altered by the 'echo' command.
The solution that ended up working for me was:
printenv LOCAL_RSA_VAR > ~/.ssh/id_rsa
this command didn't alter the content of the rsa and I was able to successfully use the certificate.
As a recap this is what I was successful doing:
Generate the new key
Used command "pbcopy < id_rsa" to get local key into clipboard
Pasted that into a new secret in Secret Manager
Used the first set of code above to have the buildspec file retrieve the content into a env variable and then the 'printenv' command above, in the install command portion of the buildspec file, to save that to the default ssh location.
Hope this helps anyone that runs into the same issue.
UPDATE: I found that this works if the RSA is stored as its own secret as one big block of text. If you try and add this as part of a json object, ie:
{
"some": "thing",
"rsa_id": "<the rsa key here>"
}
this does not seem to work. I found that the content is altered with spaces in place of the newline. This is what i found when running an 'od -ax' on each and comparing them:
own secret:
R I V A T E sp K E Y - - - - - nl
json secret:
R I V A T E sp K E Y - - - - - sp
I has the same issue, fixed it my NOT Copy-Paste my private key to SecretManager, but use AWS CLI to upload my private key to SecretManager:
aws secretsmanager put-secret-value --secret-id AWS_CODECOMMIT_SSH_PRIVATE --secret-string file://myprivatekey.pem
And then CloudBuild worked fine:
version: 0.2
env:
secrets-manager:
AWS_CODECOMMIT_SSH_ID : AWS_CODECOMMIT_SSH_ID
AWS_CODECOMMIT_SSH_PRIVATE: AWS_CODECOMMIT_SSH_PRIVATE
phases:
install:
commands:
- echo "Setup CodeCommit SSH Key"
- mkdir ~/.ssh/
- echo "$AWS_CODECOMMIT_SSH_PRIVATE" > ~/.ssh/id_rsa
- echo "Host git-codecommit.*.amazonaws.com" > ~/.ssh/config
- echo " User $AWS_CODECOMMIT_SSH_ID" >> ~/.ssh/config
- echo " IdentityFile ~/.ssh/id_rsa" >> ~/.ssh/config
- echo " StrictHostKeyChecking no" >> ~/.ssh/config
- chmod 600 ~/.ssh/id_rsa
- chmod 600 ~/.ssh/config

fatal: could not read Username for 'https://github.com ': No such device or address - rubyonrails - aws

I have a rubyonrails website which function in such a way that when a user signup with his username it create a repo in that username in my github account. It is working flawlessly in heroku. When I switched to amazon web service I initially get
intializing git
sh: git: command not found
sh: line 0: cd: /home/webapp: No such file or directory
I overcome this error by adding a config file in .ebextensions like
commands:
01_mkdir_webapp_dir:
# use the test directive to create the directory
# if the mkdir command fails the rest of this directive is ignored
test: 'mkdir /home/webapp'
command: 'ls -la /home/webapp'
02_chown_webapp_dir:
command: 'chown webapp:webapp /home/webapp'
03_chmod_webapp_dir:
command: 'chmod 700 /home/webapp'
packages:
yum:
git: []
Then I have a new error log like
fatal: could not read Username for 'https://github.com ': No such device or address
As a side note when I run this script locally, and I signed up the site at localhost:3000 terminal prompt me to submit github username and password. Is that normal. Is this is the cause of the error fatal: could not read Username for 'https://github.com ': No such device or address.
But this code works flawlessly in heroku.
Full log is below.
https://drive.google.com/file/d/1yPYsS1ETHhrEoYFWHJxt4y52jHYRkooj/view?usp=sharing
I have these environment set in aws.
https://photos.app.goo.gl/GeQHXdUWUMuixTgNA
Try to create git config file directly
#/home/webapp/.gitconfig
[user]
name = soumjoyel
email = soumjoyel#gmail.com
using this script
files:
"/home/webapp/.gitconfig" :
mode: "000644"
owner: webapp
group: webapp
content: |
[user]
name = soumjoyel
email = soumjoyel#gmail.com

Jenkins: Echo command adding a space

I have the following script in Jenkins:
echo "" >> ${file}
echo "T|${comment}" >> ${file}
But for some odd reason, Jenkins is adding a space after the comment. You can see it after the word 'comment'. Here is the output:
echo ''
echo 'T|awesomecomment '
Jenkins: Echo command adding a space
Does anyone know how to fix this?
The space you're getting comes from your own commandline, after the " and before the redirection >>. Put the redirection to the file at the beginning of the command line like this
>> ${file} echo "T|${comment}"
See Raymond Chen's post on IO redirection "The redirection can come anywhere on the line, and you can use that to get rid of the spaces" for why that space appears and why putting the redirection first fixes it.

How can I remove duplicates (deduplicate) a mbox format email mailbox?

I've got a mbox mailbox containing duplicate copies of messages, which differ only in their "X-Evolution:" header.
I want to remove the duplicate ones, in as quick and simple a way as possible. It seems like this would have been written already, but I haven't found it, although I've looked at the Python mailbox module, the various perl mbox parsers, formail, and so forth.
Does anyone have any suggestions?
This a small script, which I used for it:
#!/bin/bash
IDCACHE=$(mktemp -p /tmp)
formail -D $((1024*1024*10)) ${IDCACHE} -s
rm ${IDCACHE}
The mailbox needs to be piped through it, and in the meantime it will be deduplicated.
-D $((1024*1024*10)) sets a 10 Mebibyte cache, which is more than 10x the amount needed to deduplicate an entire year of my mail. YMMV, so adjust it accordingly. Setting it too high will cause some performance loss, setting it to low will let it slip duplicates.
formail is part of the procmail utility bundle, mktemp is part of coreutils.
I didn't look at formail (part of procmail) in enough detail. It does have such such an option, as mentioned in places like: http://hints.macworld.com/comment.php?mode=view&cid=115683 and http://us.generation-nt.com/answer/deleting-duplicate-mail-messages-help-172481881.html
'formail -D' and 'reformail -D' can only process one email per execution. Each mail needs to be separated from mbox first before being processed. I use reformail from maildrop instead since it's still in active development.
remove old idcache, tmpmail, nmbox
run dedup.sh .
nmbox is the output with duplicate messages removed.
dedup.sh
#! /bin/sh
# $1 = mbox, thunderbird mailbox
# wmbox.sh is called for each mail.
cat $1 | reformail -s ./wmbox.sh
wmbox.sh
#! /bin/sh
# stdin: a email
# called by dedup.sh
TM=tmpmail
if [ -f $TM ] ; then
echo error!
exit 1
fi
cat > $TM
# mbox format, each mail end with a blank line
echo "" >> $TM
cat $TM | reformail -D 99999999 idcache
# if this mail isn't a dup (reformail return 1 if message-id is not found)
if [ $? != 0 ]; then
# each mail shall have a message-id
if grep -q -i '^message-id:' $TM; then
cat tmpmail >> nmbox
fi
fi
rm $TM

How can I tell from a within a shell script if the shell that invoked it is an interactive shell?

I'm trying to set up a shell script that will start a screen session (or rejoin an existing one) only if it is invoked from an interactive shell. The solution I have seen is to check if $- contains the letter "i":
#!/bin/sh -e
echo "Testing interactivity..."
echo 'Current value of $- = '"$-"
if [ `echo \$- | grep -qs i` ]; then
echo interactive;
else
echo noninteractive;
fi
However, this fails, because the script is run by a new noninteractive shell, invoked as a result of the #!/bin/sh at the top. If I source the script instead of running it, it works as desired, but that's an ugly hack. I'd rather have it work when I run it.
So how can I test for interactivity within a script?
Give this a try and see if it does what you're looking for:
#!/bin/sh
if [ $_ != $0 ]
then
echo interactive;
else
echo noninteractive;
fi
The underscore ($_) expands to the absolute pathname used to invoke the script. The zero ($0) expands to the name of the script. If they're different then the script was invoked from an interactive shell. In Bash, subsequent expansion of $_ gives the expanded argument to the previous command (it might be a good idea to save the value of $_ in another variable in order to preserve it).
From man bash:
0 Expands to the name of the shell or shell script. This is set
at shell initialization. If bash is invoked with a file of com‐
mands, $0 is set to the name of that file. If bash is started
with the -c option, then $0 is set to the first argument after
the string to be executed, if one is present. Otherwise, it is
set to the file name used to invoke bash, as given by argument
zero.
_ At shell startup, set to the absolute pathname used to invoke
the shell or shell script being executed as passed in the envi‐
ronment or argument list. Subsequently, expands to the last
argument to the previous command, after expansion. Also set to
the full pathname used to invoke each command executed and
placed in the environment exported to that command. When check‐
ing mail, this parameter holds the name of the mail file cur‐
rently being checked.
$_ may not work in every POSIX compatible sh, although it probably works in must.
$PS1 will only be set if the shell is interactive. So this should work:
if [ -z "$PS1" ]; then
echo noninteractive
else
echo interactive
fi
try tty
if tty 2>&1 |grep not ; then echo "Not a tty"; else echo "a tty"; fi
man tty :
The tty utility writes the name of the terminal attached to standard
input to standard output. The name that is written is the string
returned by ttyname(3). If the standard input is not a terminal, the
message ``not a tty'' is written.
You could try using something like...
if [[ -t 0 ]]
then
echo "Interactive...say something!"
read line
echo $line
else
echo "Not Interactive"
fi
The "-t" switch in the test field checks if the file descriptor given matches a terminal (you could also do this to stop the program if the output was going to be printed to a terminal, for example). Here it checks if the standard in of the program matches a terminal.
Simple answer: don't run those commands inside ` ` or [ ].
There is no need for either of those constructs here.
Obviously I can't be sure what you expected
[ `echo \$- | grep -qs i` ]
to be testing, but I don't think it's testing what you think it's testing.
That code will do the following:
Run echo \$- | grep -qs i inside a subshell (due to the ` `).
Capture the subshell's standard output.
Replace the original ` ` expression with a string containing that output.
Pass that string as an argument to the [ command or built-in (depending on your shell).
Produce a successful return code from [ only if that string was nonempty (assuming the string didn't look like an option to [).
Some possible problems:
The -qs options to grep should cause it to produce no output, so I'd expect [ to be testing an empty string regardless of what $- looks like.
It's also possible that the backslash is escaping the dollar sign and causing a literal 'dollar minus' (rather than the contents of a variable) to be sent to grep.
On the other hand, if you removed the [ and backticks and instead said
if echo "$-" | grep -qs i ; then
then:
your current shell would expand "$-" with the value you want to test,
echo ... | would send that to grep on its standard input,
grep would return a successful return code when that input contained the letter i,
grep would print no output, due to the -qs flags, and
the if statement would use grep's return code to decide which branch to take.
Also:
no backticks would replace any commands with the output produced when they were run, and
no [ command would try to replace the return code of grep with some return code that it had tried to reconstruct by itself from the output produced by grep.
For more on how to use the if command, see this section of the excellent BashGuide.
If you want to test the value of $- without forking an external process (e.g. grep) then you can use the following technique:
if [ "${-%i*}" != "$-" ]
then
echo Interactive shell
else
echo Not an interactive shell
fi
This deletes any match for i* from the value of $- then checks to see if this made any difference.
(The ${parameter/from/to} construct (e.g. [ "${-//[!i]/}" = "i" ] is true iff interactive) can be used in Bash scripts but is not present in Dash, which is /bin/sh on Debian and Ubuntu systems.)

Resources