Docker image format - docker

Question

I would like to build a Docker image without docker itself. I have looked at [Packer](http://www.packer.io/docs/builders/docker.html, but it requires that Docker be installed on the builder host.
I have looked at the Docker Registry API documentation but this information doesn't appear to be there.
I guess that the image is simply a tarball, but I would like to see a complete specification of the format, i.e. what exact format is required and whether there are any metadata files required. I could attempt downloading an image from the registry and look what's inside, but there is no information on how to fetch the image itself.
The idea of my project is to implement a script that creates an image from artifacts I have compiled, and uploads it to the registry. I would like to use OpenEmbedded for this purpose, essentially this would be an extension to Bitbake.

Answer 1

The Docker image format is specified here: https://github.com/docker/docker/blob/master/image/spec/v1.md
The simplest possible image is a tar file containing the following:
repositories
uniqid/VERSION
uniqid/json
uniqid/layer.tar
Where VERSION contains 1.0, layer.tar contains the chroot contents and json/repositories are JSON files as specified in the spec above.
The resulting tar can be loaded into docker via docker load < image.tar

Answer 2

After reading James Coyle's blog, I figured that docker save and docker load commands are what I need.
> docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
progrium/consul latest e9fe5db22401 11 days ago 25.81 MB
> docker save e9fe5db22401 | tar x
> ls e9fe5db22401*
VERSION json layer.tar
The VERSION file contains only 1.0, and json contains quite a lot of information:
{
"id": "e9fe5db224015ddfa5ee9dbe43b414ecee1f3108fb6ed91add11d2f506beabff",
"parent": "68f9e4929a4152df9b79d0a44eeda042b5555fbd30a36f98ab425780c8d692eb",
"created": "2014-08-20T17:54:30.98176344Z",
"container": "3878e7e9b9935b7a1988cb3ebe9cd45150ea4b09768fc1af54e79b224bf35f26",
"container_config": {
"Hostname": "7f17ad58b5b8",
"Domainname": "",
"User": "",
"Memory": 0,
"MemorySwap": 0,
"CpuShares": 0,
"Cpuset": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"PortSpecs": null,
"ExposedPorts": {
"53/udp": {},
"8300/tcp": {},
"8301/tcp": {},
"8301/udp": {},
"8302/tcp": {},
"8302/udp": {},
"8400/tcp": {},
"8500/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"HOME=/",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"SHELL=/bin/bash"
],
"Cmd": [
"/bin/sh",
"-c",
"#(nop) CMD []"
],
"Image": "68f9e4929a4152df9b79d0a44eeda042b5555fbd30a36f98ab425780c8d692eb",
"Volumes": {
"/data": {}
},
"WorkingDir": "",
"Entrypoint": [
"/bin/start"
],
"NetworkDisabled": false,
"OnBuild": [
"ADD ./config /config/"
]
},
"docker_version": "1.1.2",
"author": "Jeff Lindsay <progrium#gmail.com>",
"config": {
"Hostname": "7f17ad58b5b8",
"Domainname": "",
"User": "",
"Memory": 0,
"MemorySwap": 0,
"CpuShares": 0,
"Cpuset": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"PortSpecs": null,
"ExposedPorts": {
"53/udp": {},
"8300/tcp": {},
"8301/tcp": {},
"8301/udp": {},
"8302/tcp": {},
"8302/udp": {},
"8400/tcp": {},
"8500/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"HOME=/",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"SHELL=/bin/bash"
],
"Cmd": [],
"Image": "68f9e4929a4152df9b79d0a44eeda042b5555fbd30a36f98ab425780c8d692eb",
"Volumes": {
"/data": {}
},
"WorkingDir": "",
"Entrypoint": [
"/bin/start"
],
"NetworkDisabled": false,
"OnBuild": [
"ADD ./config /config/"
]
},
"architecture": "amd64",
"os": "linux",
"Size": 0
}
The layer.tar file appears to be empty. So inspected the parent, and the grandparent, both contained no file in their layer.tar files.
So assuming that 4.0K is the standard size for an empty tarball:
for layer in $(du -hs */layer.tar | grep -v 4.0K | cut -f2)
do (echo $layer:;tar tvf $layer)
done
To see that these contain simple incremental changes to the filesystem.
So one conclusion is that it's probably best to just use Docker to build the image and push it the registry, just as Packer does.
The way to build an image from scratch is described in the docs.
It turns out that docker import - scratch doesn't care about what's in the tarball. I simply assumes that is the rootfs.
> touch foo
> tar c foo | docker import - scratch
02bb6cd70aa2c9fbaba37c8031c7412272d804d50b2ec608e14db054fc0b9fab
> docker save 02bb6cd70aa2c9fbaba37c8031c7412272d804d50b2ec608e14db054fc0b9fab | tar x
> ls 02bb6cd70aa2c9fbaba37c8031c7412272d804d50b2ec608e14db054fc0b9fab/
VERSION json layer.tar
> tar tvf 02bb6cd70aa2c9fbaba37c8031c7412272d804d50b2ec608e14db054fc0b9fab/layer.tar
drwxr-xr-x 0/0 0 2014-09-01 13:46 ./
-rw-r--r-- 500/500 0 2014-09-01 13:46 foo
In terms of OpenEmbedded integration, it's probably best to build the rootfs tarball, which is something Yocto provides out of the box, and use the official Python library to import the rootfs tarball with import_image(src='rootfs.tar', repository='scratch') and then push it private registry method.
This is not the most elegant solution, but that's how it would have to work at the moment. Otherwise one probably can just manage and deploy rootfs revisions in their own way, and just use docker import on the target host, which still won't be a nice fit, but is somewhat simple.