I'm trying to understand how the Docker layerdb works and am confused about how Docker stores information in the layerdb.
I created a very simple image with 3 layers
FROM ubuntu
RUN apt-get update
RUN apt-get install -y python3
When I inspect the image I get
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:6515074984c6f8bb1b8a9962c8fb5f310fc85e70b04c88442a3939c026dbfad3",
"sha256:248dfada031369060fd48c5fa9f8ce888d467651c1a9c01fe97acbc813e4b629",
"sha256:a339a99118749b8cfd59ee816172d0b5da19bae5bd65e020fd5983357d9123fe"
]
},
For the first layer, there is a corresponding directory with the name 6515074984c6f8bb1b8a9962c8fb5f310fc85e70b04c88442a3939c026dbfad3. This makes sense to me, the layer directory has the same name as the hash
For the next layer 248dfada031369060fd48c5fa9f8ce888d467651c1a9c01fe97acbc813e4b629 there is no directory in the layerdb with that name. Instead I find this hash in df3566a26ffeaf9233f0878dde886a70a87fd3ec78b8bd315cb53b2f8aa552fb/diff, why is this directory named completely differently from the hash of the layer? Do these names mean anything? Finally, is the hash found in diff a hash of the raw bytes found in the folder pointed to by the cache id?
Related
If it is a docker file, I want to remove the directory by executing the following command.
RUN rm /usr/bin/wget
How can I do it? any help is appreciated
First thing to emphasize: in Dockerfile, RUN rm /usr/bin/wget doesn't physically remove the file. Files and directories in previous layers will physically stay there forever. So, if you are trying to remove a file with sensitive information using rm, it's not going to work. As an example, recently, this oversight has led to a high-profile security breach in Codecov.
Docker Layer Attacks: Publicly distributed Docker images should be either squashed or multistage such that intermediate layers that contain sensitive information are excluded from the final build.
What happens is, RUN rm /usr/bin/wget creates another layer that contains a "whiteout" file /usr/bin/.wh.wget, and this new layer sits on top of all previous layers. Then at runtime, it's just that container runtimes will hide the file and you will not see it. However, if you download the image and inspect each layer, you will be able to see and extract both /usr/bin/wget and /usr/bin/.wh.wget files. So, yes, doing rm later doesn't reduce the size of the image at all. (BTW, each RUN in Dockerfile creates a new layer at the end. So, for example, if you remove files within the same RUN like RUN touch /foo && rm /foo, you will not leave /foo in the final image.)
Therefore, with Jib, if the file or directory you want to "delete" is coming from a base image, what you can do is to create a new whiteout file for it. Jib has the <extraDirectories> feature to copy arbitrary files into an image. So, for example, since <project root>/src/main/jib is the default extra directory, you can create an empty src/main/jib/usr/bin/.wh.wget, which will be coped into /usr/bin/.wh.wget in an image.
And of course, if you really want to physically remove the file that comes from the base image, the only option is to rebuild your base image so that it doesn't contain /usr/bin/wget.
For completeness: if the file or directory you want to remove is not from your base image but from Jib, you can use the Jib Layer-Filter extension (Maven/Gradle). (This is app-layer filtering and doesn't involve whiteout files.) However, normally there will be no reason to remove files put by Jib.
I read the Docker Image Specification v1.2.0.
It said:
Layers are referenced by cryptographic hashes of their serialized representation. This is a SHA256 digest over the tar archive used to transport the layer, represented as a hexadecimal encoding of 256 bits, e.g., sha256:a9561eb1b190625c9adb5a9513e72c4dedafc1cb2d4c5236c9a6957ec7dfd5a9. Layers must be packed and unpacked reproducibly to avoid changing the layer ID, for example by using tar-split to save the tar headers. Note that the digest used as the layer ID is taken over an uncompressed version of the tar.
I want find out the specific process. So I try the flowing:
chao#manager-02:~/image_lab$ docker image save busybox:1.27-glibc > busybox.tar
chao#manager-02:~/image_lab$ tar -xvf busybox.tar
47f54add1c481ac7754f9d022c2c420099a16e78faf85b4f2926a96ee40277fe/
47f54add1c481ac7754f9d022c2c420099a16e78faf85b4f2926a96ee40277fe/VERSION
47f54add1c481ac7754f9d022c2c420099a16e78faf85b4f2926a96ee40277fe/json
47f54add1c481ac7754f9d022c2c420099a16e78faf85b4f2926a96ee40277fe/layer.tar
fe2d514cd10652d0384abf2b051422722f9cdd7d189e661450cba8cd387a7bb8.json
manifest.json
repositories
chao#manager-02:~/image_lab$ ls
47f54add1c481ac7754f9d022c2c420099a16e78faf85b4f2926a96ee40277fe Dockerfile manifest.json
busybox.tar fe2d514cd10652d0384abf2b051422722f9cdd7d189e661450cba8cd387a7bb8.json repositories
chao#manager-02:~/image_lab$ sha256sum 47f54add1c481ac7754f9d022c2c420099a16e78faf85b4f2926a96ee40277fe/layer.tar
545903a7a569bac2d6b75f18d399251cefb53e12af9f644f4d9e6e0d893095c8 47f54add1c481ac7754f9d022c2c420099a16e78faf85b4f2926a96ee40277fe/layer.tar
Why the sha256sum I generated is not equal to sha256sum of the image layer?
Technically, you did answer your own question. This is what the Docker image spec says (as you quoted):
[DiffID] is a SHA256 digest over the tar archive used to transport the layer (...) Note that the digest used as the layer ID is taken over an uncompressed version of the tar.]"
But later on, when describing the content of the image, the same doc also says:
There is a directory for each layer in the image. Each directory is named with a 64 character hex name that is deterministically generated from the layer information. These names are not necessarily layer DiffIDs or ChainIDs.
If you look at the manifest.json of your image, you'll see that the rootfs.diff_ids array points to same hash you obtained by sha256suming layer.tar. The hash you computed is the DiffID.
The obvious follow up question then is: where did that directory name came from?!
I am not sure, but it seems that it is generated by whatever algorithm was used to generate layer IDs on the older Docker image format v1. Back then, images and layers were conflated into a single concept.
I'd guess they kept the v1 directory names unchanged to simplify the use old layers with newer Docker versions.
Footnote: AFAIU, the Docker image format spec is superseded by the OCI image format specification, but docker image save seems to generate archives in the older Docker format.)
Say there is an image A described by the following Dockerfile:
FROM bash
RUN mkdir "/data" && echo "FOO" > "/data/test"
VOLUME "/data"
I want to specify an image B that inherites from A and modifies /data/test. I don't want to mount the volume, I want it to have some default data I specify in B:
FROM A
RUN echo "BAR" > "/data/test"
The thing is that the test file will maintain the content it had at the moment of VOLUME instruction in A Dockerfile. B image test file will contain FOO instead of BAR as I would expect.
The following Dockerfile demonstrates the behavior:
FROM bash
# overwriting volume file
RUN mkdir "/volume-data" && echo "FOO" > "/volume-data/test"
VOLUME "/volume-data"
RUN echo "BAR" > "/volume-data/test"
RUN cat "/volume-data/test" # prints "FOO"
# overwriting non-volume file
RUN mkdir "/regular-data" && echo "FOO" > "/regular-data/test"
RUN echo "BAR" > "/regular-data/test"
RUN cat "/regular-data/test" # prints "BAR"
Building the Dockerfile will print FOO and BAR.
Is it possible to modify file /data/test in B Dockerfile?
It seems that this is intended behavior.
Changing the volume from within the Dockerfile: If any build steps change the data within the volume after it has been declared, those changes will be discarded.
VOLUMEs are not part of your IMAGE. So what is the use case to seed data into it. When you push the image into another location, it is using an empty volume at start. The dockerfile behaviour does remind you of that.
So basically, if you want to keep the data along with the app code, you should not use VOLUMEs. If the volume declaration did exist in the parent image then you need to remove the volume before starting your own image build. (docker-copyedit).
There are a few non-obvious ways to do this, and all of them have their obvious flaws.
Hijack the parent docker file
Perhaps the simplest, but least reusable way, is to simply use the parent Dockerfile and modify that. A quick Google of docker <image-name:version> source should find the github hosting the parent image Dockerfile. This is good for optimizing the final image, but destroyes the point of using layers.
Use an on start script
While a Dockerfile can't make further modifications to a volume, a running container can. Add a script to the image, and change the Entrypoint to call that (and have that script call the original entry point). This is what you will HAVE to do if you are using a singleton-type container, and need to partially 'reset' a volume on start up. Of course, since volumes are persisted outside the container, just remember that 1) your changes may already be made, and 2) Another container started at the same time may already be making those changes.
Since volumes are (virtually) forever, I just use one time setup scripts after starting the containers for the first time. That way I easily control when the default data is setup/reset. (You can use docker inspect <volume-name> to get the host location of the volume if you need to)
The common middle ground on this one seems to be to have a one-off image whose only purpose is to run once to do the volume configurations, and then clean it up.
Bind to a new volume
Copy the contents of the old volume to a new one, and configure everything to use the new volume instead.
And finally... reconsider if Docker is right for you
You probably already wasted more time on this than it was worth. (In my experience, the maintenance pain of Docker has always far outweighed the benefits. However, Docker is a tool, and with any tool, you need to sometimes take a moment to reflect if you are using it right, and if there are better tools for the job.)
I would like to edit a docker images metadata for the following reasons:
I don't like an image parents EXPOSE, VOLUME etc declaration (see #3465, Docker-Team did not want to provide a solution), so I'd like to "un-volume" or "un-expose" the image.
I dont't like an image ContainerConfig (see docker inspect [image]) cause it was generated from a running container using docker commit [container]
Fix error durring docker build or docker run like:
cannot mount volume over existing file, file exists [path]
Is there any way I can do that?
Its a bit hacky, but works:
Save the image to a tar.gz file:
$ docker save [image] > [targetfile.tar.gz]
Extract the tar file to get access to the raw image data:
tar -xvzf [targetfile.tar.gz]
Lookup the image metadata file in the manifest.json file: There should be a key like .Config which contains a [HEX] number. There should be an exact [HEX].json in the root of the extracted folder.
This is the file containing the image metadata. Edit as you like.
Pack the extracted files back into an new.tar.gz-archive
Use cat [new.tar.gz] | docker load to re-import the modified image
Use docker inspect [image] to verify your metadata changes have been applied
EDIT:
This has been wrapped into a handy script: https://github.com/gdraheim/docker-copyedit
I had come across the same workaround - since I have to edit the metadata of some images quite often (fixing an automated image rebuild from a third party), I have create a little script to help with the steps of save/unpack/edit/load.
Have a look at docker-copyedit. It can remove or overrides volumes as well as set other metadata values like entrypoint and cmd.
I'm using [or, trying to use] the docker cayley from here: https://github.com/saidimu/cayley-docker
I created a data dir at /var/lib/apps/cayley/data and dropped the .cfg file in there that I was instructed to make:
{
"database": "myapp",
"db_path": "/var/lib/apps/cayley/data",
"listen_host": "0.0.0.0"
}
I ran docker cayley with:
docker run -d -p 64210:64210 -v /var/lib/apps/cayley/data/:/data saidimu/cayley:v0.4.0
and it runs fine, I'm looking at it's UI in the browser:
And I add a triple or two, and I get success messages.
Then I go to the query interface and try to list any vertices:
> g.V
and there is nothing to be found (I think):
{
"result": null
}
and there is nothing written in the data directory I created.
Any ideas why data isn't being written?
Edit: just to be sure there wasn't something wrong with my data directory, I ran the local volume mounted docker for neo4j on the same directory and it saved data correctly. So, that eliminates some possibilities.
I can not comment yet but I think to obtain results from your query you need to use the All keyword
g.V().All() // Print All the vertices
OR
g.V().Limit(20) // Limits the results to 20
If that was not your problem I can edit and share my dockerfile which is derived from the same docker-file that you are using.
You may refer to the lib here to learn more about how to use Cayley's APIs and the data format in Cayley and some other stuff like N-Triples N-quads and RDF:
Cayley APIs usages examples (mocha test cases)
Clearly designed entry-level N-quads data for getting you in: in the project test/data directory