I have problem with SSH connection while deploying in GitLab CI.
Error:
ssh: connect to host 207.154.196.22 port 22: Connection refused
Error: exit status 255
deployment_stage:
stage: deployment_stage
only:
- main
script:
- export SSH_PRIVATE_KEY=~/.ssh/id_rsa
- export SSH_PUBLIC_KEY=~/.ssh/id_rsa.pub
- export DROPLET_IPV4=$(doctl compute droplet get board-api-backend --template={{.PublicIPv4}} --access-token $DIGITALOCEAN_API_TOKEN)
- export DROPLET_NAME=board-api-backend
- export CLONE_LINK=https://oauth2:$GITLAB_ACCESS_TOKEN#gitlab.com/$CI_PROJECT_PATH
- export SSH_GIT_CLONE_COMMAND="git clone $CLONE_LINK ~/app"
- export SSH_VAR_INIT_COMMAND="export DIGITALOCEAN_API_TOKEN=$(echo $DIGITALOCEAN_API_TOKEN)"
- export SSH_COMMAND="./deployment/scripts/production/do-deploy.sh"
- echo "Deployment stage"
- mkdir ~/.ssh
- chmod 700 ~/.ssh
- echo "$PRODUCTION_SSH_PRIVATE_KEY" > $SSH_PRIVATE_KEY
- cat $SSH_PRIVATE_KEY
- chmod 400 $SSH_PRIVATE_KEY
- ssh-keyscan -H $DROPLET_IPV4 >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
- cat ~/.ssh/known_hosts
- eval `ssh-agent -s`
- ssh-add $SSH_PRIVATE_KEY
- echo $SSH_VAR_INIT_COMMAND
- echo $SSH_GIT_CLONE_COMMAND
- echo $SSH_COMMAND
- doctl compute ssh $DROPLET_NAME --ssh-command "$SSH_VAR_INIT_COMMAND && rm -R ~/app || true && $SSH_GIT_CLONE_COMMAND" --ssh-key-path $SSH_PRIVATE_KEY --verbose --access-token $DIGITALOCEAN_API_TOKEN
- doctl compute ssh $DROPLET_NAME --ssh-command "cd ~/app && $SSH_COMMAND" --access-token $DIGITALOCEAN_API_TOKEN
Command
- cat $SSH_PRIVATE_KEY
displays private key in correct format
---header---
key without spaces
---footer---
How can I troubleshoot it?
It was working some days ago (maybe about two weeks), but today something went wrong, nothing was changed about ssh deployment.
What it can be?
Related
My Gilab CI script falls and exists with this error
Permission denied, please try again.
Permission denied, please try again.
$SSH_USER#$IPADDRESS: Permission denied (publickey,password).
This is my CI script:
image: alpine
before_script:
- echo "Before script"
- apk add --no-cache rsync openssh openssh-client
- mkdir -p ~/.ssh
- eval $(ssh-agent -s)
- echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
- echo "${SSH_PRIVATE_KEY}" | tr -d '\r' | ssh-add - > /dev/null
- ssh -o 'StrictHostKeyChecking no' $SSH_USER#$IPADDRESS
- cd /var/www/preview.hidden.nl/test
building:
stage: build
script:
- git reset --hard
- git pull origin develop
- composer install
- cp .env.example .env
- php artisan key:generate
- php artisan migrate --seed
- php artisan cache:clear
- php artisan config:clear
- php artisan storage:link
- sudo chown -R deployer:www-data /var/www/preview.hidden.nl/test/
- find /var/www/preview.hidden.nl/test -type f -exec chmod 664 {} \;
- find /var/www/preview.hidden.nl/test -type d -exec chmod 775 {} \;
- chgrp -R www-data storage bootstrap/cache
- chmod -R ug+rwx storage bootstrap/cache
My setup is as follows.
Gitlab server > Gitlab.com
Gitlab runner > Hetzner server
Laravel Project > Same Hetzner server
I generated a new SSH-key (without password) pair for this runner, named gitlab & gitlab.pub. I added the content of "gitlab.pub" to the $KNOWN_HOST variable. I added the content of "gitlab" to the $SSH_PRIVATE_KEY variable.
The problem is, I don't really know what's going on. What I think is happening is the following. The GitLab Ci job is its own separate container. A container can't just ssh to a remote server. So the private key of my Hetzner server needs to be known to the docker container ( task: - echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts ).
Because the key is then known, the docker container should not ask for a password. Yet it prompts for a password and also returns that the password is incorrect. I also have my own private key pair, besides the GitLab key pair, not sure if that causes the issue; but removing my own key there would block access to my server, so I did not test removing that.
Could someone help me in this manner? I've been working on this for two weeks and I can't even deploy a simple hello-world.txt file yet! :-)
I have created an image where I run some tasks.
I want to be able to push some files to a remote server that runs Windows Server 2022.
The gitlab-runner runs on an Ubuntu machine.
I managed to do that using shell executors. But now I want to do the same inside a docker container.
Using the following guide
https://docs.gitlab.com/ee/ci/ssh_keys/#ssh-keys-when-using-the-docker-executor
I don't understand in which user I will create the keys.
In a shell executor case I used gitlab-runner user in which I created a pair of keys. I added the public key to the server that I want to push files to and it worked.
However, I added the same private key into the gitlab CI/CD variable as the guide suggests.
Then inside the job I added the following:
before_script:
- 'command -v ssh-agent >/dev/null || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
script:
- scp -P <port> myfile.txt username#ip:remote_path
But the job fails with errors
Host key verification failed.
lost connection
Should I use the same private key from gitlab-runner user?
PS: The echo "$SSH_PRIVATE_KEY" works. I can see the key I added in the gitlab CI/CD variable.
I used something similar in my CI process, works like a charm, I recall I've used some base64 formatted runner key due to some formatting errors:
- echo $GITLAB_RUNNER_SSH_KEY | base64 -d > $HOME/.ssh/runner_key
- chmod -R 600 ~/.ssh
- eval $(ssh-agent -s)
- ssh-add $HOME/.ssh/runner_key
I can push images to ECR but I am not even close sure what I should do next (what should be the flow) to make my images run on Kubernetes on EKS
jobs:
create-deployment:
executor: aws-eks/python3
parameters:
cluster-name:
description: |
Name of the EKS cluster
type: string
steps:
- checkout
- aws-eks/update-kubeconfig-with-authenticator:
cluster-name: << parameters.cluster-name >>
install-kubectl: true
- kubernetes/create-or-update-resource:
get-rollout-status: true
resource-file-path: tests/nginx-deployment/deployment.yaml
# resource-file-path: configs/k8s/prod-deployment.yaml
resource-name: deployment/prod-deployment
orbs:
aws-ecr: circleci/aws-ecr#6.15.0
aws-eks: circleci/aws-eks#1.1.0
kubernetes: circleci/kubernetes#0.4.0
version: 2.1
workflows:
deployment:
jobs:
- aws-ecr/build-and-push-image:
repo: bwtc-backend
tag: "${CIRCLE_BRANCH}-v0.1.${CIRCLE_BUILD_NUM}"
dockerfile: configs/Docker/Dockerfile.prod
path: .
filters:
branches:
ignore:
- master
- aws-eks/create-cluster:
cluster-name: eks-demo-deployment
requires:
- aws-ecr/build-and-push-image
- create-deployment:
cluster-name: eks-demo-deployment
requires:
- aws-eks/create-cluster
- aws-eks/update-container-image:
cluster-name: eks-demo-deployment
container-image-updates: 'nginx=nginx:1.9.1'
post-steps:
- kubernetes/delete-resource:
resource-names: nginx-deployment
resource-types: deployment
wait: true
record: true
requires:
- create-deployment
resource-name: deployment/nginx-deployment
- aws-eks/delete-cluster:
cluster-name: eks-demo-deployment
requires:
- aws-eks/update-container-image
That's what I've got in my config for now.
The problem I am facing at the moment is:
gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
Exited with code exit status 2
CircleCI received exit code 2
I am using a snippet from CircleCI Documentation, so I guess it should work.
I passed in all the params as I can see but I can't get what I've missed here.
I need your help guys!
The last update has a bug. It is failing due to an incorrect URL. The issue is still open, see here. The correct url should be
https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz
unlike the current
"https://github.com/weaveworks/eksctl/releases/download/latest_release/eksctl_$(uname -s)_amd64.tar.gz"
While you wait for it to be approved, use the following to install eksctl first.
- run:
name: Install the eksctl tool
command: |
if which eksctl > /dev/null; then
echo "eksctl is already installed"
exit 0
fi
mkdir -p eksctl_download
curl --silent --location --retry 5 "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" \
| tar xz -C eksctl_download
chmod +x eksctl_download/eksctl
SUDO=""
if [ $(id -u) -ne 0 ] && which sudo > /dev/null ; then
SUDO="sudo"
fi
$SUDO mv eksctl_download/eksctl /usr/local/bin/
rmdir eksctl_download
and then run the job
- aws-eks/create-cluster:
cluster-name: eks-demo-deployment
That should solve the issue.
An example
# Creation of Cluster
create-cluster:
executor: aws-eks/python3
parameters:
cluster-name:
description: |
Name of the EKS cluster
type: string
steps:
- run:
name: Install the eksctl tool
command: |
if which eksctl > /dev/null; then
echo "eksctl is already installed"
exit 0
fi
mkdir -p eksctl_download
curl --silent --location --retry 5 "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" \
| tar xz -C eksctl_download
chmod +x eksctl_download/eksctl
SUDO=""
if [ $(id -u) -ne 0 ] && which sudo > /dev/null ; then
SUDO="sudo"
fi
$SUDO mv eksctl_download/eksctl /usr/local/bin/
rmdir eksctl_download
- aws-eks/create-cluster:
cluster-name: eks-demo-deployment
I have the weirdest error in GitHub Actions that I have been trying to resolve for multiple hours now and I am all out of ideas.
I currently use a very simple GitHub Action. The end goal is to run specific bash commands via ssh in other workflows.
Dockerfile:
FROM ubuntu:latest
COPY entrypoint.sh /entrypoint.sh
RUN apt update && apt install openssh-client -y
RUN chmod +x entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
entrypoint.sh:
#!/bin/sh
mkdir -p ~/.ssh/
echo "$1" > ~/.ssh/private.key
chmod 600 ~/.ssh/private.key
echo "$2" > ~/.ssh/known_hosts
echo "ssh-keygen"
ssh-keygen -y -e -f ~/.ssh/private.key
echo "ssh-keyscan"
ssh-keyscan <IP>
ssh -i ~/.ssh/private.key -tt <USER>#<IP> "echo test > testfile1"
echo "known hosts"
cat ~/.ssh/known_hosts
wc -m ~/.ssh/known_hosts
action.yml
name: "SSH Runner"
description: "Runs bash commands in remote server via SSH"
inputs:
ssh_key:
description: 'SSH Key'
known_hosts:
description: 'Known Hosts'
runs:
using: 'docker'
image: 'Dockerfile'
args:
- ${{ inputs.ssh_key }}
- ${{ inputs.known_hosts }}
current workflow file in the same repo:
on: [push]
jobs:
try-ssh-commands:
runs-on: ubuntu-latest
name: SSH MY_TEST
steps:
- name: Checkout
uses: actions/checkout#v2
- name: test_ssh
uses: ./
with:
ssh_key: ${{secrets.SSH_PRIVATE_KEY}}
known_hosts: ${{secrets.SSH_KNOWN_HOSTS}}
In the github action online console I get the following output:
ssh-keygen
---- BEGIN SSH2 PUBLIC KEY ----
Comment: "2048-bit RSA, converted by root#844d5e361d21 from OpenSSH"
AAAAB3NzaC1yc2EAAAADAQABAAABAQDaj/9Guq4M9V/jEdMWFrnUOzArj2AhneV3I97R6y
<...>
9f/7rCMTJwae65z5fTvfecjIaUEzpE3aen7fR5Umk4MS925/1amm0GKKSa2OOEQnWg2Enp
Od9V75pph54v0+cYfJcbab
---- END SSH2 PUBLIC KEY ----
ssh-keyscan
# <IP>:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
# <IP>:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
# <IP>:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
# <IP>:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
# <IP>:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
<IP> ssh-ed25519 AAAAC3NzaC1lZD<...>9r5SNohBUitk
<IP> ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRNWiDWO65SKQnYZafcnkVhWKyxxi5r+/uUS2zgYdXvuZ9UIREw5sumR95kbNY1V90<...>
qWXryZYaMqMiWlTi6ffIC5ZoPcgGHjwJRXVmz+jdOmdx8eg2llYatRQbH7vGDYr4zSztXGM77G4o4pJsaMA/
***
Host key verification failed.
known hosts
***
175 /github/home/.ssh/known_hosts
As far as I understand *** is used to replace GitHub secrets which in my case is the key of the known host. Getting *** as a result for the ssh-keyscan and the cat known_host should mean, that the known_hosts file is correct and a connection should be possible. Because in both cases the console output is successfully censored by GitHub. And since the file contains 175 characters I can assume it contains the actual key. But as one can see the script fails with Host key verification failed.
When I do the same steps manually in another workflow with the exact same input data I succeed. Same goes for ssh from my local computer with the same private_key and known_host files.
This for example works with the exact same secrets
- name: Create SSH key
run: |
mkdir -p ~/.ssh/
echo "$SSH_PRIVATE_KEY" > ../private.key
sudo chmod 600 ../private.key
echo "$SSH_KNOWN_HOSTS_PROD" > ~/.ssh/known_hosts
shell: bash
env:
SSH_PRIVATE_KEY: ${{secrets.SSH_PRIVATE_KEY}}
SSH_KNOWN_HOSTS: ${{secrets.SSH_KNOWN_HOSTS}}
- name: SSH into DO and run
run: >
ssh -i ../private.key -tt ${SSH_USERNAME}#${SERVER_IP}
"
< commands >
"
Using the -o "StrictHostKeyChecking no" flag on the ssh command in the entrypoint.sh also works. But I would like to avoid this for security reasons.
I have been trying to solve this issue for hours, but I seem to miss a critical detail. Has someone encountered a similar issue or knows what I am doing wrong?
So after hours of searching I found out what the issue was.
When force accepting all host keys with the -o "StrictHostKeyChecking no" option no ~/.ssh/known_hosts file is created. Meaning that the openssh-client I installed in the container does not seem to read from that file.
So telling the ssh command where to look for the file solved the issue:
ssh -i ~/.ssh/private.key -o UserKnownHostsFile=/github/home/.ssh/known_hosts -tt <USER>#<IP> "echo test > testfile1"
Apparently one can also change the location of the known_hosts file within the ssh_config permanently (see here).
Hope this helps someone at some point.
First, add a chmod 600 ~/.ssh/known_hosts as well in your entrypoint.
For testing, I would check if options around ssh-keyscan make any difference:
ssh-keyscan -H <IP>
# or
ssh-keyscan -t rsa -H <IP>
Check that your key is generated using the default rsa public-key cryptosystems.
The HostKeyAlgorithms used might be set differently, in which case:
ssh-keyscan -H -t ecdsa-sha2-nistp256 <IP>
I am trying to add some continuous deployment for a typescript API built with node, and mongodb.
I would like to do so via the gitlab instance that I already have :
Runner config (/etc/gitlab-runner/config.toml) :
[[runners]]
name = "runner"
url = "https://git.[DOMAIN].[EXT]"
token = "[ID]"
executor = "docker"
[runners.docker]
tls_verify = false
image = "mhart/alpine-node:6.5"
privileged = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
So my deploy job looks as follow :
Deployment_preprod:
stage: Deploy
before_script:
# https://docs.gitlab.com/ee/ci/ssh_keys/
- 'which ssh-agent || ( apk add --no-cache --virtual openssh-client )'
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
- mkdir -p ~/.ssh
- echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
- chmod 700 ~/.ssh
script:
- scp -r dist user#[IP]:/home/[user]/preprod-back
- ssh -tt user#[IP] cd /home/[user]/preprod-back && yarn run doc && docker-compose restart
environment:
name: preprod
url: https://preprod.api.[DOMAIN].[EXT]
only:
- develop
Question :
this job fail on /bin/sh: eval: line 91: docker-compose: not found which suprise me since running docker-compose [whatever] just works fine server-side when I log in the server via ssh.
The && are tripping you up. You should quote the entire remote command.
script:
- scp -r dist user#[IP]:/home/[user]/preprod-back
- ssh -tt user#[IP] "cd /home/[user]/preprod-back && yarn run doc && docker-compose restart"