I tried to run mitmproxy via docker to collect API requests we send from the app to the server. Already set up the process locally and start working on putting it into a docker container.
Firstly I tried to use a standard docker image: https://hub.docker.com/r/mitmproxy/mitmproxy
Run the following command:
docker run --rm -it -p 8282:8080 -p 127.0.0.1:8182:8081
mitmproxy/mitmproxy mitmweb --web-host 0.0.0.0 --web-port 8282
And faced the issue with mitmproxy certificate, while tryining to collect the 'https' traffic, it has not been trusted.
When I tried to write a custom image based on standard one through the docker file, I added a corresponding mitmproxy certificate to the container in there, but it doesn't help for some reasons.
Not Truster sertificate example: https://i.stack.imgur.com/nSWb6.png
Browser view after performing some search: https://i.stack.imgur.com/l9RXV.png
Dockerfile:
https://i.stack.imgur.com/P5qOm.png
I run Windows machine and I'm super new to docker, I'm trying to setup LetsEncrypt on my site for HomeAssistant purpose.
I create a folder in C:/Docker/LetsEncrypt in my Windows machine and then I run this command.
PS C:\Users\test> docker run -it --rm -p 80:80 --name certbot -v "C:Docker/LetsEncrypt/etc/letsencrypt:/etc/letsencrypt" -v "C:Docker/LetsEncrypt/var/lib/letsencrypt:/var/lib/letsencrypt" -v "C:Docker/LetsEncrypt/var/log/letsencrypt:/var/log/letsencrypt" quay.io/letsencrypt/letsencrypt:latest certonly --standalone --standalone-supported-challenges http-01 --email myemail#mail.com -d mysite.duckdns.org
This is the result I got
Warning: This Docker image will soon be switching to Alpine Linux.
You can switch now using the certbot/certbot repo on Docker Hub.
The standalone specific supported challenges flag is deprecated. Please use the --preferred-challenges flag instead.
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
/opt/certbot/venv/local/lib/python2.7/site-packages/josepy/jwa.py:107: CryptographyDeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead.
signer = key.signer(self.padding, self.hash)
-------------------------------------------------------------------------------
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf. You must
agree in order to register with the ACME server at
https://acme-v01.api.letsencrypt.org/directory
-------------------------------------------------------------------------------
(A)gree/(C)ancel: a
-------------------------------------------------------------------------------
Would you be willing to share your email address with the Electronic Frontier
Foundation, a founding partner of the Let's Encrypt project and the non-profit
organization that develops Certbot? We'd like to send you email about EFF and
our work to encrypt the web, protect its users and defend digital rights.
-------------------------------------------------------------------------------
(Y)es/(N)o: y
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for mysite.duckdns.org
Waiting for verification...
Cleaning up challenges
IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/mysite.duckdns.org/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/mysite.duckdns.org/privkey.pem
Your cert will expire on 2018-06-22. To obtain a new or tweaked
version of this certificate in the future, simply run certbot
again. To non-interactively renew *all* of your certificates, run
"certbot renew"
- Your account credentials have been saved in your Certbot
configuration directory at /etc/letsencrypt. You should make a
secure backup of this folder now. This configuration directory will
also contain certificates and private keys obtained by Certbot so
making regular backups of this folder is ideal.
- If you like Certbot, please consider supporting our work by:
Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le
Looks like everything is fine except I can't find the file fullchaim.pem and privkey.pem in my Windows machine which is suppose to be inside C:\Docker\LetsEncrypt\etc\letsencrypt.
What am I missing?
Here is the command you executed
PS C:\Users\test> docker run -it --rm -p 80:80 --name certbot
-v "C:Docker/LetsEncrypt/etc/letsencrypt:/etc/letsencrypt"
-v "C:Docker/LetsEncrypt/var/lib/letsencrypt:/var/lib/letsencrypt"
-v "C:Docker/LetsEncrypt/var/log/letsencrypt:/var/log/letsencrypt"
quay.io/letsencrypt/letsencrypt:latest
certonly --standalone --standalone-supported-challenges
http-01 --email myemail#mail.com -d mysite.duckdns.org
docker allows you to mount directories on our local machine such that internal to the launched container those same directories are mapped to new names however the directory contents are identical. For example in above it says
-v "C:Docker/LetsEncrypt/etc/letsencrypt:/etc/letsencrypt"
which is a volume pair where left of : delimiter is a directory local to your machine C:Docker/LetsEncrypt/etc/letsencrypt and on right hand side is what that same directory gets called from perspective inside container as per /etc/letsencrypt ... this mapping frees up the container's internal perspective to be isolated from a given person's local directory structure ... now look closely at this message :
- Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/mysite.duckdns.org/fullchain.pem
that is from perspective of inside the container ... so now your are armed with the knowledge to discover where you missing keys are
SOLUTION when inside of container it says
/etc/letsencrypt/live/mysite.duckdns.org/fullchain.pem
that same file is mapped to your local machine at location
C:Docker/LetsEncrypt/etc/letsencrypt/live/mysite.duckdns.org/fullchain.pem
Background:
To setup a private docker registry server at path c:\dkrreg on localhost on Windows 10 (x64) system, installed with Docker for Windows, have successfully tried following commands:
docker run --detach --publish 1005:5000 --name docker-registry --volume /c/dkrreg:/var/lib/registry registry:2
docker pull hello-world:latest
docker tag hello-world:latest localhost:1005/hello-world:latest
docker push localhost:1005/hello-world:latest
docker pull localhost:1005/hello-world:latest
Push and Pull from localhost:1005/hello-world:latest via command line succeeds too.
Issue:
If i use my IP address via docker pull 192.168.43.239:1005/hello-world:latest it gives following error in command shell:
Error response from daemon: Get https://192.168.43.239:1005/v1/_ping: http: server gave HTTP response to HTTPS client
When using 3rd party Docker UI Manager via docker run --detach portainer:latest it also shows error to connect as:
2017/04/19 14:30:24 http: proxy error: dial tcp [::1]:1005: getsockopt: connection refused
Tried other stuff also. How can I connect my private registry server that is localhost:1005 from LAN using any Docker Management UI tool ?
At last find solution to this which was tricky
Generated CA private key and certificate as ca-cert-mycompany.pem and ca-cert-key-companyname.pem. And configured docker-compose.yml to save both files as :ro in these locations: /usr/local/share/ca-certificates, /etc/ssl/certs/, /etc/docker/certs.d/mysite.com. But I also tried only copying certificate to /usr/local/share/ca-certificates was enough as docker will ignore duplicate CA certificates. This extra copying is because at many placed docker fellow recommended the same. I did not executed command: update-ca-certificates this time in registry container but was doing earlier as against what is suggested by many.
Defined in docker-compose.yml: random number as REGISTRY_HTTP_SECRET, and server's chained certificate (CA certificate appended to end of it) to REGISTRY_HTTP_TLS_CERTIFICATE amd server's public key to REGISTRY_HTTP_TLS_KEY. Had disabled HTTP authentication. Especially used some naming for file names as found with other certificates in container folder as mysite.com_server-chained-certificate.crt instead of just certificate.crt.
V-Imp: pushed certificate to trusted root in windows using command certutil.exe -addstore root .\Keys\ca-certificate.crt followed with restarting Docker for Windows from taskbar icon and then creating container using docker-compose up -d. This is most important step without this nothing worked.
Now can perform docker pull mysite.com:1005/my-repo:my-tag.
You need to specify to your Docker daemon that your registry is insecure: https://docs.docker.com/registry/insecure/
Based on your OS/system, you need to change the configuration of the daemon to specify the registry address (format IP:PORT, use 192.168.43.239:1005 rather than localhost:1005).
Once you have done that, you should be able to execute the following:
docker pull 192.168.43.239:1005/hello-world:latest
You should also be able to access it via Portainer using 192.168.43.239:1005 in the registry field.
If you want to access your registry using localhost:1005 inside Portainer, you can try to run it inside the host network.
docker run --detach --net host portainer:latest
Is there a way I can download a Docker image/container using, for example, Firefox and not using the built-in docker-pull.
I am blocked by the company firewall and proxy, and I can't get a hole through it.
My problem is that I cannot use Docker to get images, that is, Docker save/pull and other Docker supplied functions since it is blocked by a firewall.
Just an alternative - This is what I did in my organization for couchbase image where I was blocked by a proxy.
On my personal laptop (OS X)
~$ $ docker save couchbase > couchbase.tar
~$ ls -lh couchbase.docker
-rw------- 1 vikas devops 556M 12 Dec 21:15 couchbase.tar
~$ xz -9 couchbase.tar
~$ ls -lh couchbase.tar.xz
-rw-r--r-- 1 vikas staff 123M 12 Dec 22:17 couchbase.tar.xz
Then, I uploaded the compressed tar ball to Dropbox and downloaded on my work machine. For some reason Dropbox was open :)
On my work laptop (CentOS 7)
$ docker load < couchbase.tar.xz
References
https://docs.docker.com/engine/reference/commandline/save/
https://docs.docker.com/engine/reference/commandline/load/
I just had to deal with this issue myself - downloading an image from a restricted machine with Internet access, but no Docker client for use on a another restricted machine with the Docker client, but no Internet access. I posted my question to the DevOps Stack Exchange site:
Downloading Docker Images from Docker Hub without using Docker
With help from the Docker Community I was able to find a resolution to my problem. What follows is my solution.
So it turns out that the Moby Project has a shell script on the Moby GitHub account which can download images from Docker Hub in a format that can be imported into Docker:
download-frozen-image-v2.sh
The usage syntax for the script is given by the following:
download-frozen-image-v2.sh target_dir image[:tag][#digest] ...
The image can then be imported with tar and docker load:
tar -cC 'target_dir' . | docker load
To verify that the script works as expected, I downloaded an Ubuntu image from Docker Hub and loaded it into Docker:
user#host:~$ bash download-frozen-image-v2.sh ubuntu ubuntu:latest
user#host:~$ tar -cC 'ubuntu' . | docker load
user#host:~$ docker run --rm -ti ubuntu bash
root#1dd5e62113b9:/#
In practice I would have to first copy the data from the Internet client (which does not have Docker installed) to the target/destination machine (which does have Docker installed):
user#nodocker:~$ bash download-frozen-image-v2.sh ubuntu ubuntu:latest
user#nodocker:~$ tar -C 'ubuntu' -cf 'ubuntu.tar' .
user#nodocker:~$ scp ubuntu.tar user#hasdocker:~
and then load and use the image on the target host:
user#hasdocker:~ docker load -i ubuntu.tar
user#hasdocker:~ docker run --rm -ti ubuntu bash
root#1dd5e62113b9:/#
I adapted a python script for having an OS independant solution:
docker-drag
Use it like that, and it will create a TAR archive that you will be able to import using docker load :
python docker_pull.py hello-world
python docker_pull.py alpine:3.9
python docker_pull.py kalilinux/kali-linux-docker
Use Skopeo. It is a tool specifically made for that (and others) purpose.
After install simply execute:
mkdir ubuntu
skopeo --insecure-policy copy docker://ubuntu ./ubuntu
Copy these files and import as you like.
First, check if your Docker daemon is configured for using the proxy. With boot2docker and docker-machine, for instance, this is done on docker-machine create, with the --engine-env option.
If this is just a certificate issue (i.e., Firefox does access Docker Hub), try and install that certificate:
openssl s_client -connect index.docker.io:443 -showcerts /dev/null | openssl x509 -outform PEM > docker.pem
sudo cp docker.pem /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust
sudo systemctl restart docker
sudo docker run hello-world
The other workaround (not a recommended solution) would be to access Docker Hub without relying on certificate with --insecure-registry:
If the firewall is actively blocking any Docker pull, to the point you can't even access Docker Hub from Firefox, then you would need to docker save/docker load an image archive. Save it from a machine where you did access Docker Hub (and where the docker pull succeeded). Load it on your corporate machine (after approval of your IT system administrators, of course).
Note: you cannot easily "just" download an image, because it is often based on top of other images which you would need to download too. That is what docker pull does for you. And that is what docker save does too (create one archive composed of all the necessary images).
The OP Ephreal adds in the comments:
[I] didn't get my corp image to work either.
But I found that I could download the Docker file and recreate the image my self from scratch.
This is essentially the same as downloading the image.
So, by definition, a Docker pull client command actually needs to talk to a Docker daemon, because the Docker daemon assembles layers one by one for you.
Think of it as a POST request - it's causing a mutation of state, in the Docker daemon itself. You're not 'pulling' anything over HTTP when you do a pull.
You can pull all the individual layers over REST from the Docker registry, but that won't actually be the same semantics as a pull, because pull is an action that specifically tells the daemon to go and get all the layers for an image you care about.
Another possibly might be an option for you if your company firewall (and policy) allows for connecting to a remote SSH server. In that case you can simply set up a SSH tunnel to tunnel any traffic to the Docker registry through it.
The Answer and solution to my original question were that I found that I could download the Docker file and all the necessary support files and recreate the image my self from scratch. This is essentially the same as downloading the image.
This solution has been in the questions and comments above, I just pinned it out here.
This is although no longer an issue for me since my company have changed policy and allowed docker pull commands to work.
thanks #Ham Co for answer,
I adapted a golang tool for having an OS independant solution:
golang http pull docker image
./gopull download redis
get a docker importable archive redis.tar
References:
https://github.com/NotGlop/docker-drag
I upgraded my Mac (OS X) from an older Docker installation to Docker Toolbox, meaning that I'm now working with Docker Machine, and in the process discovered that certs I had working for push/pull with a private registry are not there, and I can't for the life of me figure out how to get them in place. At the moment when I try a test pull I get the dreaded x509: certificate signed by unknown authority error. I've searched around, looked at issues in Github, but nothing has worked for me. I even tried ssh'ing into the machine VM and manually copying them into /etc/ssl/certs, and various other things, with no luck. And I certainly don't want to get into the "insecure-registry" stuff. This used to work with boot2docker prior to moving to docker-machine.
This seems like a very simple question: I have a couple of .crt files that I need put in the right place so that I can do a push/pull. How does one do this? And secondarily, how can this not be documented anywhere? Can we wish for a docker-machine add-cert command someday?
Thanks for any help, and I hope a good answer here can stick around to assist others who run into this.
Okay so let's imagine I have a registry running at the address: 192.168.188.190:5000 and I have a proper certificate for this address.
I would now run the following commands to install the root certificate into my machine:
docker-machine scp ./dockerCA.crt $MACHINE_NAME:dockerCA.crt
docker-machine ssh $MACHINE_NAME sudo mkdir -p /etc/docker/certs.d/192.168.188.190:5000
docker-machine ssh $MACHINE_NAME sudo mv dockerCA.crt /etc/docker/certs.d/192.168.188.190:5000/dockerCA.crt
Set the variable MACHINE_NAME to whatever the name of your machine is. The machine will now trust your root certificate.
Having the same issue I read the Documentation in Docker on how to add a certificate to my computer.
As you mentioned that you are on a updated Mac OS X, proceed by doing the following:
Copy the cert file from your docker registry to your hard drive, e.g.
scp user#docker.reg.ip:/path/to/crt/domain.crt /tmp/domain.crt
Add the certificate to your trusted certificates using the following command
sudo security add-trusted-cert -d -r trustRoot \
-k /Library/Keychains/System.keychain /tmp/domain.crt
Restart your local docker handler and now you should be able to upload your local Docker images to the Docker registry.
If you are running on any other operating systems please check this site on how to add trusted root certificates.