I have a Ubuntu 14 desktop, on which I do some of my development work.
This work mainly revolves around Django & Flask development using PyCharm
I was wandering if it was possible to wrap the entire OS file system in a Docker container, so my whole development environment, including PyCharm and any other tools, would become portable
Yes, this is where Docker shines. Once you install Docker you can run:
docker run --name my-dev -it ubuntu:14.04 /bin/bash
and this will put you, as root, inside a Docker container's bash prompt. It is for all intents and purposes the entire os without anything extra, you will need to install the extras, like pycharm, flask, django, etc. Your entire environment. The environment you start with has nothing, so you will have to add things like pip (apt-get install -y python-pip), and other goodies. Once you have your entire environment you can exit (with exit, or ^D) and you will be back in your host operating system. Then you can commit :
docker commit -m 'this is my development image' my-dev my-dev
This takes the Docker image you just ran (and updated with changes) and saves it on your machine with the tag my-dev:v1, any time in the future you can run this again using the invocation:
docker run -it my-dev /bin/bash
Building a Docker image like this is harder, it is easier once you learn how to make a Dockerfile that describes the base image (ubuntu:14.04) and all of the modifications you want to make to it in a file called Dockerfile. I have an example of a Dockerfile here:
https://github.com/tacodata/pythondev
This builds my python development environment, including git, ssh keys, compilers, etc. It does have my name hardcoded in it, so, it won't help you much doing development (I need to fix that). Anyway, you can download the Dockerfile, change it with your details in it, and create your own image like this:
docker build -t my-dev -< Dockerfile
There are hundreds of examples on the Docker hub which is where I started with mine.
-g
Related
I've been attempting to move an application to the cloud for a while now and have most of the services set up in pods running in a k8s cluster. The last piece has been giving me trouble, I need to set up an image with an older piece of software that cannot be installed silently. I then attempted in my dockerfile to install its .net dependencies (2005.x86, 2010.x86, 2012.x86, 2015.x86, 2015.x64) and manually transfer a local install of the program but that also did not work.
Is there any way to run through a guided install in a remote windows image or be able to determine all of the file changes made by an installer in order to do them manually?
You can track the changes done by the installer following these steps:
start a new container based on your base image
docker run --name test -d <base_image>
open a shell in the new container (I am not familiar with Windows so you might have to adapt the command below)
docker exec -ti test cmd
Run whatever commands you need to run inside the container. When you are done exit from the container
Examine the changes to the container's filesystem:
docker container diff test
You can also use docker container export to export the container's filesystem as a tar archive, and then docker image import to create an image from that archive.
Background
I have a large Python service that runs on a desktop PC, and I need to have it run as part of a K8S deployment. I expect that I will have to make several small changes to make the service run in a deployment/pod before it will work.
Problem
So far, if I encounter an issue in the Python code, it takes a while to update the code, and get it deployed for another round of testing. For example, I have to:
Modify my Python code.
Rebuild the Docker container (which includes my Python service).
scp the Docker container over to the Docker Registry server.
docker load the image, update tags, and push it to the Registry back-end DB.
Manually kill off currently-running pods so the deployment restarts all pods with the new Docker image.
This involves a lot of lead time each time I need to debug a minor issue. Ideally, I've prefer being able to just modify the copy of my Python code already running on a pod, but I can't kill it (since the Python service is the default app that is launched, with PID=1), and K8S doesn't support restarting a pod (to my knowledge). Alternately, if I kill/start another pod, it won't have my local changes from the pod I was previously working on (which is by design, of course; but doesn't help with my debug efforts).
Question
Is there a better/faster way to rapidly deploy (experimental/debug) changes to the container I'm testing, without having to spend several minutes recreating container images, re-deploying/tagging/pushing them, etc? If I could find and mount (read-write) the Docker image, that might help, as I could edit the data within it directly (i.e. new Python changes), and just kill pods so the deployment re-creates them.
There are two main options: one is to use a tool that reduces or automates that flow, the other is to develop locally with something like Minikube.
For the first, there are a million and a half tools but Skaffold is probably the most common one.
For the second, you do something like ( eval $(minikube docker-env) && docker build -t myimagename . ) which will build the image directly in the Minikube docker environment so you skip steps 3 and 4 in your list entirely. You can combine this with a tool which detects the image change and either restarts your pods or updates the deployment (which restarts the pods).
Also FWIW using scp and docker load is very not standard, generally that would be combined into docker push.
I think your pain point is the container relied on the python code. You can find a way to exclude the source code from docker image build phase.
For my experience, I will create a docker image only include python package dependencies, and use volume to map source code dir to the container path, so you don't need to rebuild the image if no dependencies are added or removed.
Example
I have not much experience with k8s, but I believe it must be more or less the same as docker run.
Dockerfile
FROM python:3.7-stretch
COPY ./python/requirements.txt /tmp/requirements.txt
RUN pip install --no-cache-dir -r /tmp/requirements.txt
ENTRYPOINT ["bash"]
Docker container
scp deploy your code to the server, and map your host source path to the container source path like this:
docker run -it -d -v /path/to/your/python/source:/path/to/your/server/source --name python-service your-image-name
With volume mapping, your container no longer depend on the source code, you can easily change your source code without rebuilding your image.
Why?
I'm trying to create a general purpose solution for running docker-compose on Heroku. I want to make a one click deployment solution through the use of Heroku Button deployment. This way, a user does not need any knowledge of git, Heroku cli and docker.
The problem.
Docker and the docker daemon are only available when I set the stack to container. There are buildpacks that give you docker and docker-compose CLI but without the docker daemon you cannot run the docker image. So buildpacks won't work.
With the stack set to container I can use the file heroku.yml (article). In there I define my processes. (It replaces Procfile. If I still add a Procfile to my project it will do nothing.)
I can also define a Dockerfile there to build my docker image.
When I however run the docker image the following error pops up:
2019-02-28T15:32:48.462101+00:00 app[worker.1]: Couldn't connect to Docker daemon at http+docker://localhost - is it running?
2019-02-28T15:32:48.462119+00:00 app[worker.1]:
2019-02-28T15:32:48.462122+00:00 app[worker.1]: If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.
The problem is inside the Docker container the Docker daemon is not running. The solution to this is to mount it:
docker run -v /var/run/docker.sock:/var/run/docker.sock ...
And since you cannot use Procfile I cannot run that command. (See above heroku.yml replaces Procfile.) And if I was using a buildpack I could use Procfile but the docker daemon wouldn't be running.....
I tried defining a VOLUME within the Dockerfile and the problem persists. Furthermore a heroku article says "Volume mounting is not supported. The filesystem of the dyno is ephemeral."
On Heroku it is possible to run a docker image. What I am struggling at is running a docker in docker image.
Running a docker in docker image works fine on my VPS by mounting /var/run/docker.sock but this cannot(?) be done on Heroku.
Last words:
I'm trying to make this work so that other people can easily deploy software solution even though they are not comfortable with git, heroku cli and docker.
Unfortunately the answer to your question is: not yet.
For securiy reasons Heroku does not provide to the users the ability to run priviledged containers because the container could access to host capabilities.
The documentation is pretty clear about your limitations, e.g: No --priviledged container and no root user either, no VOLUMES and disk is ephemeral.
After playing with DinD images for your concern, I came to the conclusion that trying to run Docker inside a Heroku container is not the right choice and design.
I am pretty sure what you are trying to achieve is close to what Heroku is offering to the users. Offering a platform or an application where non-developper can push and deploy applications with just a button can be very interesting in various ways. And it can be done with an application using their Platform API.
In this situation a Web application (running on Heroku) may not (up to my knowledge) be able to do what you want. Instead you need to embed in a Desktop application: git, docker, and your app for parsing, verifying, building and pushing your applications/components to Heroku's container registry.
In the end, if you still think what you need a DinD solution, well, your primary solution to use a VPS is the only solution for the moment. But be aware that it may open security vulnerabilities to your system and you may arrive to offer something similar to Heroku's offer when trying to limit those security doors.
I don't think you can run a service on Heroku that can use the docker command to start some docker container.
I want to make a one click deployment solution through the use of Heroku Button deployment.
I think you can update the reference of the Deploy button to some of your automation servers (ex: an instance Jenkins already deploy on Heroku/another cloud) to trigger the deploy pipeline and don't let people interact with git/docker, etc.
But yes, you have to deal with a lot of problems like security, parameter. when not using popular solutions like Jenkins/CircleCI login and then deploy...
What I did was install it in my dockerfile like this:
RUN curl -fsSLO https://get.docker.com/builds/Linux/x86_64/docker-17.04.0-ce.tgz \
&& tar xzvf docker-17.04.0-ce.tgz \
&& mv docker/docker /usr/local/bin \
&& rm -r docker docker-17.04.0-ce.tgz
Then in the args section for running the docker I added this:
args '--user root -v /var/run/docker.sock:/var/run/docker.sock'
For further explanation on why this works see: stackoverflow.com/q/27879713/354577 This works well for me though.
I have a totally empty debian9 on which I installed docker-ce and nothing else.
My client wants me to run a website (already done locally on my PC) that he can migrate/move rapidly from one server to another moving docker images.
My idea is to install some empty docker image, and then install on it manually all dependencies (ngingrtmp, apache2, nodejs, mysql, phpmyadmin, php, etc...)
I need to install all these dependencies MANUALLY (to keep control) - not using a ready to go docker images from dockerhub, and then to create an IMAGE of ALL things I have done (including these dependencies, but also files I will upload).
Problem is : I have no idea how to start a blank image, connect to it and then save a modified image with components and dependencies I will run.
I am aware that the SIZE may be bigger with a simple dockerfile, but I need to customize lots of things such as using php5.6, apache2.2, edit some php.ini etc etc..
regards
if you don't want to define you're dependencies on the docker file then you can have an approach like this, spin up a linux container with a base image and go inside the docker
sudo docker exec -it <Container ID> /bin/bash
install your dependencies as you install on any other linux server.
sudo apt-get install -y ngingrtmp apache2 nodejs mysql phpmyadmin php
then exit the container by ctrl+p and ctrl+q and now commit the changes you made
sudo docker commit CONTAINER_ID new-image-name
run docker images command and you will see the new image you have created, then you can use/move that image
You can try with a Dockerfile with the following content
FROM SCRATCH
But then you will need to build and add the operating system yourself.
For instance, alpine linux does this in the following way:
FROM scratch
ADD rootfs.tar.xz /
CMD ["/bin/sh"]
Where rootfs.tar.xz is a file of less of 2MB available on alpine's github repository (version 3.7 for x86_64 arch):
https://github.com/gliderlabs/docker-alpine/tree/61c3181ad3127c5bedd098271ac05f49119c9915/versions/library-3.7/x86_64
Or you can begin with alpine itself, but you said that you don't want to depend on ready to go docker images.
A good start point for you (if you decide to use alpnie linux), could look like the one available at https://github.com/docker-library/httpd/blob/eaf4c70fb21f167f77e0c9d4b6f8b8635b1cb4b6/2.4/alpine/Dockerfile
As you can see, A Dockerfile can became very big and complex because within it you provision all the software you need for running your image.
Once you have your Dockerfile, you can build the image with:
docker build .
You can give it a name:
docker build -t mycompany/myimage:1.0
Then you can run your image with:
docker run mycompany/myimage:1.0
Hope this helps.
I'm in front of one dilemma that I'd like to discuss here to open a constructive discussion.
My use case is very simple:
I need to run a bash script which execute several commands like install npm, execute aws-cli and query PostgreSQL. For this last task I use psql. Easy task I'd say however Docker slightly complicate the situation.
The problem would be solved if I would create an image where I'd install all the dependencies. However the result would be a pretty big image and I'd not go with this solution.
What about to run the script with one Docker image and then from the script (inside Docker) run something like
docker run postgres:9.6.3-alpine psql
docker run node:9.8 npm
In other words would be to run docker inside docker. What do you think?
If you want to execute docker run inside a docker, just execute first docker run with -v /var/run/docker.sock:/var/run/docker.sock option.
With this, you can access in containers to docker images defined in host or wherever.