I am trying to set up a Satis installation with the official Docker image composer/satis.
The private repository I have put uses an SSH key with a passphrase.
I use a syntax in the SSH config that allows me to use one key by repository.
When I launch docker-compose up with this file :
version: "3.2"
services:
satis:
image: composer/satis
container_name: satis
hostname: satis
volumes:
- /var/www/html/satis:/build
- /usr/local/bin/composer:/composer/composer
- ~/.ssh/id_my_repo:/root/.ssh/id_my_repo
- ~/.ssh/known_hosts:/root/.ssh/known_hosts
- ~/.ssh/satisConfig.txt:/root/.ssh/config
#- /~/.ssh/known_hosts:/etc/ssh/ssh_known_hosts
environment:
SATIS_CONFIG_FILE: /build/satis.json
COMPOSER_NO_INTERACTION: 1
PRIVATE_REPO_DOMAIN_LIST: bitbucket.org gitlab.com github.com
GIT_SSH_COMMAND: "ssh -v"
ports:
- 8181:80
(Obviously, I replace my real repository name).
and the satis.json file :
{
"name": "myuser/my-satis",
"homepage": "http://my.satis.url",
"repositories": [
{
"type": "vcs",
"url": "git#github.com-my-repo:myuser/my-repo.git",
"name:": "myuser/my-repo",
"options": {
"ssh2": {
"username": "myuser",
"password": "mypassword",
"pubkey_file": "/home/myuser/.ssh/id_my-repo.pub",
"privkey_file": "/home/myuser/.ssh/id_my-repo"
}
}
}
]
}
and my SSH config :
Host github.com-my-repo
Hostname github.com
IdentityFile=/root/.ssh/id_my_repo
Host *
UserKnownHostsFile /root/.ssh/known_hosts
GlobalKnownHostsFile /root/.ssh/known_hosts
IdentitiesOnly yes
ServerAliveInterval 30
ServerAliveCountMax 6
the git command git clone --mirror -- 'git#github.com-mon-repo:myuser/mon-repo.git' '/composer/cache/vcs/git-github.com-mon-repo-myuser-mon-repo.git/' fails with the message :
ssh: Could not resolve hostname github.com-my-repo: Name does not resolve
If I launch this command at the outside of the container, it works...so what can be wrong?
Docker version 20.10.17, build 100c701.
docker-compose version 1.29.2, build 5becea4c
EDIT
I have some slashes before my ~ in front of some of my paths.
I fixed that. Now I have :
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: No more authentication methods to try.
git#github.com: Permission denied (publickey).
Related
Part of my CircleCI config is to deploy to a remote server using scp, now I added SSH private key (https://circleci.com/docs/add-ssh-key) and it looks like (the values masked intentionally):
And here is a snapshot of my config:
deploy-web:
working_directory: ~/subdir/web
docker:
- image: cimg/node:16.16
steps:
- add_ssh_keys:
fingerprints:
- "d7:*****fa"
- checkout:
path: ~/subdir
- node/install-packages:
pkg-manager: yarn
- run:
name: Build
command: yarn build
- run:
name: Deploy
command: |
SSH_DEPLOY_PATH=/apps/my-app
scp -r dist/* "$SSH_USER#$SSH_HOST:$SSH_DEPLOY_PATH"
Everything runs fine but the ssh part outputs:
The authenticity of host '************** (**************)' can't be established.
ECDSA key fingerprint is SHA256:6pix3P******M.
Are you sure you want to continue connecting (yes/no/[fingerprint])?
Please not that i copied the fingerprint that is in the config from the web (in the screenshot). Now, is there anything am doing wrong or how do I go about it, because so far, google has not been resourceful.
I managed to resolve this, and this is the hack (I can't believe I didn't think of this sooner), I added this step just before the scp step:
- run:
name: Add SSH host to known
command: ssh-keyscan -H $SSH_HOST >> ~/.ssh/known_hosts
I have been having serious troubles to get ssh-agent forwarded into the docker container (with my docker-compose installation). I have Mac running Catalina, with docker-engine 19.03.8 and Compose # 1.24.
The following is my docker-compose file:
version: '3.7'
services:
platform:
build:
context: .
dockerfile: ./platform/compose/Dockerfile.platform.local
working_dir: /root/platform
ports:
- "3000:3000"
command: ["./compose/scripts/start_rails.sh"]
tty: true
stdin_open: true
volumes:
- type: bind
source: /run/host-services/ssh-auth.sock
target: /run/host-services/ssh-auth.sock
env_file: ./platform/.env
environment:
TERM: xterm-256color
SSH_AUTH_SOCK: /run/host-services/ssh-auth.sock
volumes:
The way I have configured ssh-agent forwarding is as specified in docker-compose documentation
The ./compose/scripts/start_rails.sh script does bundle install && bundle exec rails s. I have few gems that I am pulling from private-repositories and I thought I should be able to install these gems by forwarding ssh-agent.
I have also tried starting the ssh-agent before I spin the docker-compose up, but that doesnt seem to do anything.
{
"debug": true,
"experimental": true,
"features": {
"buildkit": true
}
}
This is what I have added inside my docker configuration file. Any help is appreciated.
**UPDATE: 0 **
The following in my .ssh directory structure and config:
tree ~/.ssh
├── config
├── known_hosts
├── midhun
│ ├── id_rsa
│ └── id_rsa.pub
└── client
├── id_rsa
└── id_rsa.pub
cat ~/.ssh/config
Host github.com
HostName github.com
User git
IdentityFile ~/.ssh/client/id_rsa
Host me.github.com
HostName github.com
User git
IdentityFile ~/.ssh/midhun/id_rsa
UPDATE: 1
Updated my config with ForwardAgent Yes and it didn't work either. I have recorded entire ssh-logs in this gist -> https://gist.github.com/midhunkrishna/8f77ebdc90c7230d2ffae0834dc477cc .
I believe below change to your ~/.ssh/config should fix the issue:
Host github.com
HostName github.com
User git
IdentityFile ~/.ssh/client/id_rsa
ForwardAgent yes
Host me.github.com
HostName github.com
User git
IdentityFile ~/.ssh/midhun/id_rsa
ForwardAgent yes
Update 1: 5th May 2020
In your case, the reason it may not be working is that the agent on the host is key less.
You can confirm that using:
$ ssh-add -L
$ ssh-add -l
The agent will only forward the keys it has in its memory, nothing on your disk. Else you risk exposing every key that is there without any permission. What you need do is make sure you add those keys to your ssh-agent at startup:
$ ssh-add ~/.ssh/client/id_rsa
$ ssh-add ~/.ssh/midhun/id_rsa
Then if you do ssh-add -L on host and inside the docker terminal you should see both keys. And the ssh-agent also will work.
I used docker-compose to run a local gitlab server.
git:
container_name: git-server
image: gitlab/gitlab-ce:latest
hostname: 'gitlab.example.com'
ports:
- '8090:80'
- '22:22'
volumes:
- "$PWD/srv/gitlab/config:/etc/gitlab"
- "$PWD/srv/gitlab/logs:/var/log/gitlab"
- "$PWD/srv/gitlab/data:/var/opt/gitlab"
networks:
- net
I want to setup custom-hooks for a project repo i created in the gitlab webUI so that it triggers a jenkins job. As per gitlab documentation, this is the path for repos in omnibus installations where i will have to create custom-hooks directory
/var/opt/gitlab/git-data/repositories/<group>/<project>.git
But inside of /var/opt/gitlab/git-data/repositories , I don't see a group directory or project directory at all
root#gitlab:~# ls -lt /var/opt/gitlab/git-data/repositories
total 0
drwxr-s---. 3 git root 16 Apr 18 04:05 #hashed
drwxr-sr-x. 3 git root 17 Apr 18 04:00 +gitaly
root#gitlab:~#
I tried searching using find. But it returned nothing. I tried searching by name of files in my project repo, but that didn't return anything as well.
In the gitlab webUI, I am able to see it all. But in the server, none of the file and dir exists.
How is it that I am not able to find any of the file in my repos when i ssh to gitlab-server?
Since I am not able to go this way, I tried by creating a post-receive.d directory under the global hooks directory /opt/gitlab/embedded/service/gitlab-shell/hooks and then adding my post-receive file as below
#!/bin/bash
# Get branch name from ref head
if ! [ -t 0 ]; then
read -a ref
fi
IFS='/' read -ra REF <<< "${ref[2]}"
branch="${REF[2]}"
if [ "$branch" == "master" ]; then
crumb=$(curl -u "jenkins:1234" -s 'http://jenkins:8080/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)')
curl -u "jenkins:1234" -H "$crumb" -X POST http://jenkins:8080/job/maven/build?delay=0sec
if [ $? -eq 0 ] ; then
echo "*** Ok"
else
echo "*** Error"
fi
jenkins is the name of the container which is in the same network as that of gitlab server.
gitlab docs says then I will have to change permission of the file to git and then make it executable. I did so. But it didn't work either. Also, I find all of the git directories is owned by root in my container.
After pushing code, i figured the hook I put in the /opt/gitlab/embedded/service/gitlab-shell/hooks/post-receive.d directory is not working and in the logs, I see below error right after I push code changes to my maven repo
==> /var/log/gitlab/nginx/gitlab_error.log <==
2020/04/18 04:57:31 [crit] 832#0: *256 connect() to unix:/var/opt/gitlab/gitlab-workhorse/socket failed (13: Permission denied) while connecting to upstream, client: <my_public_ip>, server: gitlab.example.com, request: "GET /jenkins/maven.git/info/refs?service=git-receive-pack HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/socket:/jenkins/maven.git/info/refs?service=git-receive-pack", host: "gitlab.example.com:8090"
Here, gitlab.example.com is mapped to my public ip in the /etc/hosts file of my host on which I am running docker.
If you run the following command inside of the container you should see your group repos
gitlab-rake gitlab:storage:rollback_to_legacy
inside of /var/opt/gitlab/git-data/repositories , I don't see a group directory or project directory at all
The documentation "Install GitLab using docker-compose" includes the following volumes:
volumes:
- '$GITLAB_HOME/gitlab/config:/etc/gitlab'
- '$GITLAB_HOME/gitlab/logs:/var/log/gitlab'
- '$GITLAB_HOME/gitlab/data:/var/opt/gitlab'
That means if you see locally some repos in $GITLAB_HOME/gitlab/data/git-data/repositories, you should see the same in /var/opt/gitlab/git-data/repositories/.
Assuming, of course, that you have created at least one projet/repo in your GitLab instance.
I use an ansible script to load & start the https://hub.docker.com/r/rastasheep/ubuntu-sshd/ container.
so it starts well of course :
bash-4.4$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8bedbd3b7d88 rastasheep/ubuntu-sshd "/usr/sbin/sshd -D" 37 minutes ago Up 36 minutes 0.0.0.0:49154->22/tcp test
bash-4.4$
so after ansible failure on ssh access to it I tested manually from shell
this is also ok.
bash-4.4$ ssh root#172.17.0.2
The authenticity of host '172.17.0.2 (172.17.0.2)' can't be established.
ECDSA key fingerprint is SHA256:YtTfuoRRR5qStSVA5UuznGamA/dvf+djbIT6Y48IYD0.
ECDSA key fingerprint is MD5:43:3f:41:e9:89:45:06:6f:f6:42:c4:6a:70:37:f8:1d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.17.0.2' (ECDSA) to the list of known hosts.
root#172.17.0.2's password:
root#8bedbd3b7d88:~# logout
Connection to 172.17.0.2 closed.
bash-4.4$
so the step that failed is trying to get on it from ansible script & make access to ssh-copy-id
ansible error message is :
Fatal: [172.17.0.2]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added '172.17.0.2' (ECDSA) to the list of known hosts.\r\nPermission denied (publickey,password).\r\n", "unreachable": true}
---
- hosts: 127.0.0.1
tasks:
- name: start docker service
service:
name: docker
state: started
- name: load and start the container we wanna use
docker_container:
name: test
image: rastasheep/ubuntu-sshd
state: started
ports:
- "49154:22"
- name: Wait maximum of 300 seconds for ports to be available
wait_for:
host: 0.0.0.0
port: 49154
state: started
- hosts: 172.17.0.2
vars:
passwordadmin: $6$pbE6yznA$AeFIdI.....K0
passwordroot: $6$TMrxQUxT$I8.JIzR.....TV1
ansible_ssh_extra_args: "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"
tasks:
- name: Build test container root user rsa ssh-key
shell: docker exec test ssh-keygen -b 2048 -t rsa -f /root/.ssh/id_rsa -q -N ""
so I cannot even run the needed step to build ssh
how to do then ??
1st step (ansible task) : load docker container
2cd step (ansible task on only 172.17.0.2) : connect to it & setup it
there will be 3rd step to run application on it after that.
the problem occurs only when starting the 2cd step
Ok after many trys on a second container
conclusion is my procedure was bad
what I have done to solve that :
build a diroctory tree separating ./ ./inventory ./includes
build 1 yaml file by host (local, docker, labo)
build 1 main yaml file on ./
build 1 new host file in ./inventory
connect forced by sshpass to docker on default password
changed it
add the host key on authorized key to a login dedicated usage
installed pyhton (needed to answer ansible host else it makes
randomly module errors or refused connections depending on current
action)
setup a ssh login user in sudoers
then I can un the docker.yaml actions
then only at last I can run the labo.yaml actions.
Thanks for help
now I'm able to build the missing tools.
So I am doing everything Dockerized here. Traefik is running in a container, as is my docker Registry instance. I am able to push/pull just fine from the registry if I hit it at mydomain.com:5000/myimage.
The problem comes when I try to hit it through 443 using mydomain.com/myimage. The setup I have here is Traefik reverse proxy listening on 443 at mydomain.com, and forwarding that request internally to :5000 of my Registry instance.
When I go to push/pull from the Traefik url, it hangs and counts down waiting to retry on a loop. When I look at the logs of Registry, each I can see the instance IS in fact in communication with the reverse proxy Traefik, however, I get this error in the log over and over (on each push retry from the client side):
2018/05/31 21:10:43 http: TLS handshake error from proxy_container_ip:port: remote error: tls: bad certificate
Docker Registry is really tight and strict when it comes to the TLS issue. I'm using all self signed certs here, as I'm still in development. Any idea what is causing this error? I'm assuming that either the Traefik proxy detects that the certificate offered from Registry is not to be trusted (self-signed), and therefore does not complete the "push" request, or the other way around - Registry, when sending the response back through to the Traefik proxy detects that it is not to be trusted.
I can provide additional information if needed. Current setup is that both Traefik and Registry have their own set of .crt and .key files. Both (of course) TLS enabled.
Thanks.
Here is a working solution with a self-signed certificate that you can try out on https://labs.play-with-docker.com
Server
Add a new instance node1 in your Docker playground. We configure it as our server. Create a directory for the certificates:
mkdir /root/certs
Create wildcard certificate *.domain.local:
$ openssl req -newkey rsa:2048 -nodes -keyout /root/certs/domain.local.key -x509 -days 365 -out /root/certs/domain.local.crt
Generating a 2048 bit RSA private key
...........+++
...........+++
writing new private key to '/root/certs/domain.local.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) []:
State or Province Name (full name) []:
Locality Name (eg, city) []:
Organization Name (eg, company) []:
Organizational Unit Name (eg, section) []:
Common Name (eg, fully qualified host name) []:*.domain.local
Email Address []:
Create two files docker-compose.yml and traefik.toml in directory /root. You can download them using:
wget https://gist.github.com/maiermic/cc9c9aab939f7ea791cff3d974725e4a/raw/8c5d787998d33c752f2ab369a9393905780d551c/docker-compose.yml
wget https://gist.github.com/maiermic/cc9c9aab939f7ea791cff3d974725e4a/raw/8c5d787998d33c752f2ab369a9393905780d551c/traefik.toml
docker-compose.yml
version: '3'
services:
frontproxy:
image: traefik
command: --api --docker --docker.swarmmode
ports:
- "80:80"
- "443:443"
volumes:
- ./certs:/etc/ssl:ro
- ./traefik.toml:/etc/traefik/traefik.toml:ro
- /var/run/docker.sock:/var/run/docker.sock # So that Traefik can listen to the Docker events
deploy:
labels:
- traefik.port=8080
- traefik.frontend.rule=Host:traefik.domain.local
docker-registry:
image: registry:2
deploy:
labels:
- traefik.port=5000 # default port exposed by the registry
- traefik.frontend.rule=Host:registry.domain.local
- traefik.frontend.auth.basic=user:$$apr1$$9Cv/OMGj$$ZomWQzuQbL.3TRCS81A1g/ # user:password, see https://docs.traefik.io/configuration/backends/docker/#on-containers
traefik.toml
defaultEntryPoints = ["http", "https"]
# Redirect HTTP to HTTPS and use certificate, see https://docs.traefik.io/configuration/entrypoints/
[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
[[entryPoints.https.tls.certificates]]
certFile = "/etc/ssl/domain.local.crt"
keyFile = "/etc/ssl/domain.local.key"
# Docker Swarm Mode Provider, see https://docs.traefik.io/configuration/backends/docker/#docker-swarm-mode
[docker]
endpoint = "tcp://127.0.0.1:2375"
domain = "docker.localhost"
watch = true
swarmMode = true
Initialize Docker Swarm (replace <ip-of-node1> with the IP address of node1, for example 192.168.0.13):
docker swarm init --advertise-addr <ip-of-node1>
Deploy traefik and Docker registry:
docker stack deploy myregistry -c ~/docker-compose.yml
Client
Since we don't have a DNS server, we change /etc/hosts (replace <ip-of-node1> with the IP address of our server node1, for example 192.168.0.13):
echo "<ip-of-node1> registry.domain.local traefik.domain.local" >> /etc/hosts
You should be able now to request the health status from traefik
$ curl -ksS https://traefik.domain.local/health | jq .
{
"pid": 1,
"uptime": "1m37.501499911s",
"uptime_sec": 97.501499911,
"time": "2018-07-19 07:30:35.137546789 +0000 UTC m=+97.600568916",
"unixtime": 1531985435,
"status_code_count": {},
"total_status_code_count": {},
"count": 0,
"total_count": 0,
"total_response_time": "0s",
"total_response_time_sec": 0,
"average_response_time": "0s",
"average_response_time_sec": 0
}
and you should be able to request all images (none) from our registry
$ curl -ksS -u user:password https://registry.domain.local/v2/_catalog | jq .
{
"repositories": []
}
Let's configure docker on our client. Create the directory for the registry certificates:
mkdir -p /etc/docker/certs.d/registry.domain.local/
Get the certificate from our server:
scp root#registry.domain.local:/root/certs/domain.local.crt /etc/docker/certs.d/registry.domain.local/ca.crt # Are you sure you want to continue connecting (yes/no)? yes
Now you should be able to login to our registry and add an image:
docker login -u user -p password https://registry.domain.local
docker pull hello-world:latest
docker tag hello-world:latest registry.domain.local/hello-world:latest
docker push registry.domain.local/hello-world:latest
If you request all images from our registry after that, you should see
$ curl -ksS -u user:password https://registry.domain.local/v2/_catalog | jq .
{
"repositories": [
"hello-world"
]
}