Migrating from one service to IBM Cloud for Redis.
I cannot find the correct configuration to connect using TLS. Everything I find on this is related to Heroku. and it ignores verifying the TLS/SSL connection.
I cannot find how to configure our Sidekiq/Redis to connect.
I do have a certificate from the IBM Cloud dashboard and I suspect I have to pass that along somehow.
Configure the Sidekiq.yml like this
:redis:
:url: "rediss://:< PWD >#< DB Name >:< PORT >/0"
:namespace: "app"
:ssl_params:
ca_file: 'path/to/cert'
I keep getting back the error Redis::CommandError - WRONGPASS invalid username-password pair or user is disabled.: however using these same credentials in the migration script I am able to connect to the DB, so the credentials are ok, I think it is not including the certificate correctly and I cannot find the correct way to do this
The sidekiq.yml configuration looks good to me, just make sure this has correct complete path
ca_file: 'path/to/cert'
and change the redis url to
:url: "rediss://< PWD >#< DB Name >:< PORT >/0"
further info you can read from here for TLS secured connection.
I'm not familiar with sidekiq.yml. But I've configured redlin with redis using a python script you can find here: https://github.com/IBM-Cloud/vpc-transit/blob/master/py/test_transit.py. Maybe the configuration is similar.
The relevant code is:
def vpe_redis_test(fip, resource):
"""execute a command in fip to verify postgresql is accessible"""
redis = resource["key"]
credentials = redis["credentials"]
cert_data = credentials["connection.rediss.certificate.certificate_base64"]
cli_arguments = credentials["connection.cli.arguments.0.1"]
command = f"""
#!/bin/bash
set -ex
if [ -x ./redli ]; then
echo redli already installed
else
curl -LO https://github.com/IBM-Cloud/redli/releases/download/v0.5.2/redli_0.5.2_linux_amd64.tar.gz
tar zxvf redli_*_linux_amd64.tar.gz
fi
./redli \
--long \
-u {cli_arguments} \
--certb64={cert_data} << TEST > redis.out
set foo working
I have asked the same question on github but no response. So, I think here may be a better place to ask this kind of coding questions because this is the problem when I tried to do some coding work.
I would like to set up airflow with docker on MacBook Pro with macOS 10.14.x.
I have got zpencerq/docker-airflow:1.10.2 and followed the instructions at https://docs.docker.com/compose/install/#install-compose
Also, I have set up the ssh agent forward with the instructions at https://github.com/uber-common/docker-ssh-agent-forward
When I run:
docker-compose up -d
I got:
Collecting my-data-store from git+ssh://git#github.com/my-data-store.git#v.xx.xx.x#egg=my-data-store
Cloning ssh://git#github.com/my-data-store.git to /my/local/path
Permission denied (publickey).
fatal: Could not read from remote repository.
I have created ssh public key and added the ssh public key in my git profile. I have also added my private key.
I have tried some answers for this kind of question, but none of them work.
If I run the following command in Mac terminal, it works well.
git clone -q ssh://git#github.com/my_path/my-data-store.git /tmp/my_folder
Why docker-compose cannot do the same thing to access github ?
Do I miss something ?
Sounds like you haven't added your keys. Try this from inside your OSX terminal...
ssh-add
Then try again.
Why docker-compose cannot do the same thing to access github ?
Because it (docker) will look for your keys in $HOME/.ssh/id_rsa(.pub).
When you execute git clone -q ssh://git#github.com/... from your Mac, $HOME is your regular home directory.
But from a container, it is /root by default, or the one of the user.
You should mount those keys in your docker-compose.yml (or use secrets).
You can add to a container environment
-e GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=no -o User=<github user> -i <private key>"
When is full key text making by ssh-keygen
Trying to get started with a Rails Amazon EC2 deployment using https://github.com/rubber/rubber, and I keep ending up here after attempting to create a staging server with cap rubber:create_staging:
** [out :: production.foo.com] curl: (7) couldn't connect to host command finished in 2022ms
failed: "/bin/bash -l -c 'sudo -p '\\''sudo password: '\\'' bash -l /tmp/create_inputs'" on production.foo.com
I've been sticking with Rubber's quickstart guide, but can't solve this. I'm using rvm, if that makes a difference to anyone.
Any ideas?
It looks like you are trying to connect to production.foo.com. Change your configuration to connect to the right remote server or if you are running locally in the EC2 instance you can make it localhost.
Make sure you setup your public ssh key in the ~/.ssh/authorized_keys for the user that you are trying to deploy as. This is to allow capistrano/rubber to do passwordless ssh authentication.
Reviving an old post here but there's another possible cause of this issue that I've just encountered so figured this might help someone else.
I had to re-create and download a new keypair for my ec2 instances. When I moved it to ~/.ec2/gsg-keypair I forgot to alter the permissions.
When SSH'ing directly into the instance you get the full warning, which makes debugging it easy:
UNPROTECTED PRIVATE KEY FILE! permissions 0644 for 'xxxxx.pem' are
too open. It is recommended that your private key files are NOT
accessible by others. This private key will be ignored. bad
permissions: ignore key: xxxxx.pem Permission denied (publickey).
But when running a rubber task you simply get a generic CURL error. If this is the case for you too just update the permissions like this:
chmod 600 ~/.ec2/gsg-keypair
I have a problem with jenkins, setting "git", shows the following error:
Failed to connect to repository : Command "git ls-remote -h https://person#bitbucket.org/person/projectmarket.git HEAD" returned status code 128:
stdout:
stderr: fatal: Authentication failed
I have tested with ssh:
git#bitbucket.org:person/projectmarket.git
This is error:
Failed to connect to repository : Command "git ls-remote -h git#bitbucket.org:person/projectmarket.git HEAD" returned status code 128:
stdout:
stderr: Host key verification failed.
fatal: The remote end hung up unexpectedly
I've also done these steps with "SSH key".
Login under Jenkins
sudo su jenkins
Copy your github key to Jenkins .ssh folder
cp ~/.ssh/id_rsa_github* /var/lib/jenkins/.ssh/
Rename the keys
mv id_rsa_github id_rsa
mv id_rsa_github.pub id_rsa.pub
but still not working git repository in jenkins.
thanks by help!.
Change to the jenkins user and run the command manually:
git ls-remote -h git#bitbucket.org:person/projectmarket.git HEAD
You will get the standard SSH warning when first connecting to a new host via SSH:
The authenticity of host 'bitbucket.org (207.223.240.181)' can't be established.
RSA key fingerprint is 97:8c:1b:f2:6f:14:6b:5c:3b:ec:aa:46:46:74:7c:40.
Are you sure you want to continue connecting (yes/no)?
Type yes and press Enter. The host key for bitbucket.org will now be added to the ~/.ssh/known_hosts file and you won't get this error in Jenkins anymore.
Jenkins is a service account, it doesn't have a shell by design. It is generally accepted that service accounts. shouldn't be able to log in interactively.
To resolve "Jenkins Host key verification failed", do the following steps. I have used mercurial with jenkins.
1)Execute following commands on terminal
$ sudo su -s /bin/bash jenkins
provide password
2)Generate public private key using the following command:
ssh-keygen
you can see output as ::
Generating public/private rsa key pair.
Enter file in which to save the key (/var/lib/jenkins/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
3)Press Enter --> Do not give any passphrase--> press enter
Key has been generated
4) go to --> cat /var/lib/jenkins/.ssh/id_rsa.pub
5) Copy key from id_rsa.pub
6)Exit from bash
7) ssh#yourrepository
8) vi .ssh/authorized_keys
9) Paste the key
10) exit
11)Manually login to mercurial server
Note: Pls do manually login otherwise jenkins will again give error "host verification failed"
12)once manually done, Now go to Jenkins and give build
Enjoy!!!
Good Luck
Or you can use:
ssh -oStrictHostKeyChecking=no host
This will be insecure (man in the middle attacks) but easiest solution.
The better way to do that is to generate correct mappings between host and ip address, so ssh will not complain:
#!/bin/bash
for domain in "github.com" "bitbucket.org"; do
sed -i "/$domain/d" ~/.ssh/known_hosts
line=$(ssh-keyscan $domain,`nslookup $domain | awk '/^Address: / { print $2 ; exit }'`)
echo $line >> ~/.ssh/known_hosts
done
Excerpt from gist.
I think, that many people didnt recognize, at least available since jenkins 2.361:
btw. No Verification is for sure not the best option.
Had same problem, i fix it like that :
reset permission on id_rsa* only for current user no group no other
chmod o-rwx ~/.ssh/id*
chmod G-rwx ~/.ssh/id*
ls -lart ~/.ssh/
-rw------- 1 jenkins nogroup 398 avril 3 09:34 id_rsa.pub
-rw------- 1 jenkins nogroup 1675 avril 3 09:34 id_rsa
And clear ~/.ssh/know_hosts
Now Connect as jenkins
sudo su jenkins
Try the jenkins commands
git ls-remote -h git#bitbucket.org:user/project.git HEAD
If no problem appears, now jenkins will be able to connect the repo (for me ^^ at least)
As for the workaround (e.g. Windows slave), define the following environment variable in global properties:
GIT_SSH_COMMAND="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"
Note: If you don't see the option, you probably need EnvInject plugin for it.
login as jenkins using: "sudo su -s /bin/bash jenkins"
git clone the desired repo which causes the key error
it will ask you to add the key by showing Yes/No (enter yes or y)
that's it!
you can now re-run the jenkins job.
I hope you this will fix your issue.
using https://bitbucket.org/YYYY/XX.git
you shoud delete username#
Make sure we are not editing any of the default sshd_config properties to skip the error
Host Verification Failed - Definitely a missing entry of hostname in known_hosts file
Login to the server where the process is failing and do the following:
Sudo to the user running the process
ssh-copy-id destinationuser#destinationhostname
It will prompt like this for the first time, say yes and it will also ask password for the first time:
The authenticity of host 'sample.org (205.214.640.91)' can't be established.
RSA key fingerprint is 97:8c:1b:f2:6f:14:6b:5c:3b:ec:aa:46:46:74:7c:40.
Are you sure you want to continue connecting (yes/no)? *yes*
Password prompt ? give password
Now from the server where process is running, do ssh destinationuser#destinationhostname. It should login without a password.
Note: Do not change the default permissions of files in the user's .ssh directory, you will end up with different issues
I ran into this issue and it turned out the problem was that the jenkins service wasn't being run as the jenkins user. So running the commands as the jenkins user worked just fine.
Copy host keys from both bitbucket and github:
ssh root#deployserver 'echo "$(ssh-keyscan -t rsa,dsa bitbucket.org)" >> /root/.ssh/known_hosts'
ssh root#deployserver 'echo "$(ssh-keyscan -t rsa,dsa github.com)" >> /root/.ssh/known_hosts'
Best way you can just use your "git url" in 'https" URL format in the Jenkinsfile or wherever you want.
git url: 'https://github.com/jglick/simple-maven-project-with-tests.git'
SSH
If you are trying it with SSH, then the Host key Verification error can come due to several reasons.Follow these steps to overcome all the reasons.
Set the Environment variable as HOME and provide the address as the root directory of .ssh folder. e.g:- If your .ssh is kept inside Name folder.
C:/Users/Name.
Now make sure that the public SSH key is being provided in the repository link also. Either it is github or bitbucket or any other.
Open git bash. And try cloning the project from the repository. This will help in adding your repository URL in the known_host file, which is being auto created in the .ssh folder.
Now open jenkins and create a new job. Then click on configure.
provide the cloning URL in Source code management under Git. The URL should be start with git#github.com/......... or ssh://proje........
Under the Credential you need to add the username and password of your repository form which you are cloning the project. Select that credential.
And now apply and save the configuration.
Bingo! Start building the project. I hope now you will not get any Host Key verification error!
Try
ssh-keygen -R hostname
-R hostname Removes all keys belonging to hostname from a known_hosts file. This option is useful to delete hashed hosts
Use ssh-keyscan should be much more easier:
ssh-keyscan bitbucket.org >> ~/.ssh/known_hosts
This command will put all required hosts to ~/.ssh/known_hosts. You will need to run this command inside your Jenkins machine. You can also create a job and put that command into the "Execute shell" section of the Configure of that job and then execute the job.
issue is with the /var/lib/jenkins/.ssh/known_hosts. It exists in the first case, but not in the second one. This means you are running either on different system or the second case is somehow jailed in chroot or by other means separated from the rest of the filesystem (this is a good idea for running random code from jenkins).
Next steps are finding out how are the chroots for this user created and modify the known hosts inside this chroot. Or just go other ways of ignoring known hosts, such as ssh-keyscan, StrictHostKeyChecking=no or so.
After ssh-keygen probably one only needs to copy the public key to remote host with:
ssh-copy-id -i ~/.ssh/mykey user#host
There is a safe and (relative easy) way to accomplish this, which should also work if you have separate worker nodes/clouds (like docker/kubernetes).
Adding host keys to Jenkins configuration
First go to a console and execute ssh-keyscan your_git_server.url
Copy the output of that command
Then navigate to https://YOUR_JENKINS_URL/manage/configureSecurity/
Scroll down to Git Host Key Verification Configuration
Paste the output of the command into the window. it should look like this:
Both bitbucket and github have pages about their keys and servers. Read them and ensure that you are adding the proper keys and not some random keys
Getting the ssh-keyscan via your Jenkins installation
If you for some reason do not have ssh-keyscan, you can go to the script console ( https://YOUR_JENKINS_URL/manage/script ) and paste in the following script:
def sout = new StringBuilder(), serr = new StringBuilder()
def proc = 'ssh-keyscan bitbucket.org'.execute()
proc.consumeProcessOutput(sout, serr)
proc.waitForOrKill(1000)
println "copy this to jenkins>\n$sout"
//println "err> $serr"
I'm using a variety of the GPGR gem (https://github.com/HHRy/gpgr) to do pgp encryption, and when I run a function like installed_public_keys in a production console I get the email (in an array) that is attached to my PGP key. However if I have a button that logs the output of that function in a production gui it outputs an empty array.
I also have a script runner that I can call with a production env flag that runs fine, but when the exact same line is called from crontab as root it fails saying that there are no keys installed.
I'm assuming that this has more to do with each user having a keyring of keys than environment variables but everything production is using root and it is bugging the crap out of me.
The key to running pgp encryption with crontab (runners) is to specify the home directory with the --homedir switch. I.e. instead of using the keys at ~/.gnupg you can do:
gpg --homedir /root/.gnupg -o file.txt.pgp -r email#email.com -e file.txt