I'm trying to better understand and leverage remote caching.
I'm running bazel build in a docker container, while specifying remote cache directory on a local disk:
docker run --rm -it -v $PWD:/work -w /work bazel:latest \
bazel build --disk_cache=.bazel-disk-cache //...
At the end of this run, the remote cache is populated in the .bazel-disk-cache directory on the local disk.
Then I ran the same command again. This time it is a new container instance with an empty local cache, but the remote cache is present. However, it took almost the same amount of time to complete the task, as the first run.
Is that expected? I was hoping to reduce the build time when using remote cache. What do I miss?
Related
Anyone know if Docker somehow caches files/file systems? And if it does, is there a way to suppress that or force it to regenerate/refresh the cache. I am editing my files outside my docker image, but the directory inside the docker image doesn't seem to include them in it. However, outside the docker image the "same" directory does include the files. The only thing that makes sense to me is that docker has an internal "copy" of the directory and isn't going to the disk, so it sees an outdated copy of the directory before the file was added.
Details:
I keep my "work" files in a directory on a separate partition (/Volumes/emacs) on my MacBook, i.e.:
/Volumes/emacs/datapelago
I do my editing in emacs on the MacBook, but not in the docker container. I have a link to that directory called:
/projects
And I might edit or create a file called:
/projects/nuc-440/qflow/ToAttribute.java
In the meantime I have a docker container I created this way:
docker container create -p 8080:8080 -v /Volumes/emacs/datapelago:/projects -v /Users/christopherfclark:/Users/cfclark --name nuc-440 gradle:7.4.2-jdk17 tail -f /dev/null
I keep a shell running in that container:
docker container exec -it nuc-440 /bin/bash
cd /projects/nuc-440
And after making changes I run the build/test sequence:
gradle clean;gradle build;gradle test
However, recently I have noticed that when I make changes, add files, they don't always get reflected inside the docker container, which of course can cause the build to fail or the test not to pass etc.
Thus, this question.
I'd rather not have to start/stop the container each time and instead just keep it running and tell it to "refetch" the projects/nuc-440 directory and its children.
I restarted the docker container and it is now "tracking" file changes again. That is I can make changes in MacOS and they are reflected inside docker with no additional steps. I don't seem to have to continually restart it. It must have gotten into a "wierd" state. However, I don't have any useful details beyond that. Sorry.
I created a docker image from a Dockerfile that has an environment to to run robot framework ready. I want to run a local file using the docker image and save the output of the test locally.
I know I can mount a file when I run an image and execute the test.
$ docker create -ti -v /path/to/file/testWithRobot/robottest.robot:/test/robottest.robot dc3dccfee60 robot test/robottest.robot
and then I know to save a file from docker is cp
I have 2 problems, I know that as soon as I stop that instance of docker the file gets removed, or better said when I start the mage again is a different instance of it. if I go inside the docker image mounting the file and running the tests I know the output files gets saved in the root
Output: /output.xml
Log: /log.html
Report: /report.html
how do I save those files output.xml, log.html, and report.xml into the host. I am hoping I can use one command to run tests and save. I hope I explained well enough quite new to docker.
Just run a container using:
docker run --name mytests -v /path/to/file/testWithRobot/robottest.robot:/test/robottest.robot dc3dccfee60 robot test/robottest.robot
and grab the contents using the name we set:
docker cp mytests:/output.xml .
docker cp mytests:/log.html .
docker cp mytests:/report.html .
Then you can delete the stopped container with:
docker rm mytest
Also, using this approach you won't face file permission and ownership issues.
We are about to "dockerize" our not-so-big infrastructure. One crucial question here is the whole backup / restore workflow, which is I think crucial for most enterprise but even private users.
I know about the export and save features of docker which will generate a tarball of a running container, which is neat because it can be done without shutting down the container.
So let's say we are running a container X and we have mounted some volumes:
-v /home/user/dockerapp-X/data:/var/www/html
-v /home/user/dockerapp-X/logs:/var/logs/app-x
-v /home/user/dockerapp-X/config:/etc/app-x
The biggest benefit of this is, if we update app-X we just have to pull the new image and restart the container.
But:
This way those directories wouldn't get backupped if we do docker-export or save.
So either we can just backup those directories extra, with rsync, backula or whatever. I guess this would be the "standart" way of backupping. But there is no guarantee and also no connection between the current version of the image and the data.
On a VM we would just make a snapshot to have the data and the app connected.
So the question is:
Is it a best practice to just make a Dockerfile with the current app-x version and copy the volumes in the image and build/push the whole image to our private repo?
so it would look like this:
FROM repo/app-x
COPY /home/user/dockerapp-X/data:/var/www/html
COPY /home/user/dockerapp-X/logs:/var/logs/app-x
COPY /home/user/dockerapp-X/config:/etc/app-x
then
docker build -t repo/infra/app-x:backup-v1-22.10.2016 .
docker push repo/infra/app-x:backup-v1-22.10.2016
This would mean that in our repo there is a snapshot for the current version of the app and the image contains all current data of the volumes.
So restoring would be:
docker run --name=backup-restored repo/infra/app-x:backup-v1-22.10.2016
And we could even mount the data folders locally on the host again:
docker run --name=backup-restored \
-v /home/user/dockerapp-X/data:/var/www/html
-v /home/user/dockerapp-X/logs:/var/logs/app-x
-v /home/user/dockerapp-X/config:/etc/app-x
repo/infra/app-x:backup-v1-22.10.2016
Will my data and my app have the correct data and app version?
How do I copy files from a docker container to the host machine during docker build command?
As a part of building my docker image for my app, I'm running some tests inside it, and I would like to copy the output of the test run into the host (which is a continuous integration server) to do some reporting.
I wouldn't run the tests during build, this will only increase the size of your image. I would recommend you to build the image and then run it mounting a host volume into the container and changing the working directory to the mount point.
docker run -v `pwd`/results:/results -w /results -t IMAGE test_script
There is no easy way to do this. Volumes are only created at run time. You can grab it out of the docker filesystem, (e.g. mine is /var/lib/docker/devicemapper/mnt/CONTAINER_ID/rootfs/PATH_TO_FILE) though there is no good way to figure out when your test process is complete. You could create a file when it's finished and do an inotify, but this is ugly.
I have a dev Docker container on OSX with boot2docker.
I connect to the container and mount my project's source directory in the docker container via...
docker run -it -p 8080:8080 -v /local_src:/container_src foo/bar /bin/bash
When inside of the container_src certain operations are very slow. For example, git status takes about 8 seconds to complete.
However, it all works fine if I use source that's cloned into the container
Any ideas as to why this would be?
I'm wondering if there's some overhead from a combination of mounting the volume and using boot2docker.
(I'm not an expert on this field - just making some conclusion)
Git actually works on collection of small files. When running git status it needs to access files inside .git folder (on one of my repos it's ~150 files) and run lstat() on each files in repo to ensure they're not modified (more info here). Therefore git inside docker must request info about each file (which needs to be transfered to vm). If there is even small overhead on each request (like 100 ms), and you have 8000 files inside git folder and in git - this results in process taking ~8 seconds.