Docker docker-compose not picking up relevant cached image - docker

Question

I ran docker-compose build celery and (after hours of attempts on my poor connection) it succeeded. The first 80% of the app Dockerfile is identical, yet it doesn't reuse the cache. From what I can tell browsing around, Docker compares the base image and the instructions in Dockerfile and reuses if possible.
UPDATE: The problem this question refers to disappeared, no idea why. Notes below.
Yet I'm getting:
docker-compose build celery
Building celery
Step 1 : FROM python:2.7
---> eb867117097c
Step 2 : RUN apt-get update && apt-get install -y vim gdal-bin libgdal-dev postgresql-client
---> Using cache
---> 2966946ca235
. . . identical steps . . .
Step 9 : RUN pip install --no-cache-dir -r requirements/production.txt
---> Running in 02b42f721a34
Collecting git+https://github.com . . .
. . .
---> f70ecc01cada
Removing intermediate container 02b42f721a34
Step 10 : RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
---> Running in 3575383edcef
When I type docker images I can see f70ecc01cada, but it is not found in the following:
docker-compose build app
Building app
Step 1 : FROM python:2.7
---> eb867117097c
Step 2 : RUN apt-get update && apt-get install -y vim gdal-bin libgdal-dev postgresql-client
---> Using cache
---> 03e5040df047
. . . identical steps . . .
Step 9 : RUN pip install --no-cache-dir -r requirements/production.txt
---> Running in bb26fab28548
Collecting git+https://github.com . . .
. . .
Am I right in thinking the latter should reuse the f70ecc01cada cached image? [Edit: I now realize 03e5040df047 is just a container thanks #Ohmen, so no point deleting it]
I have tried editing docker-compose to be identical memory etc but it didn't help. Can I force reuse?
UPDATE: The app build all of a sudden started picking up the cache from the celery build. I don't know what fixed it. Between it failing and working, I did connect to the image on the celery hierarchy just before the Dockerfiles diverged, manually ran the remaining commands on the app image (docker run -it ... --entry=bash, adduser and chown app/), and docker committed the resulting container. I then re-ran docker-compose build app and for some reason it picked up the cache from the celery hierarchy correctly and ran the remaining commands (create user and chown) successfully. I have no idea what made it work all of a sudden.
EDIT: Docker history of the two image trees, showing that the history diverged on the line RUN apt-get update && apt-get install -y vim gdal-bin libgdal-dev postgresql-client. Maybe I ran both builds simultaneously? I was pretty sure I didn't, as on my connection they take an hour or more, but perhaps that is the boring and embarrassing explanation. Perhaps Docker was randomly selecting the tree that subsequently failed, then randomly (after 56 attempts) selected the one that didn't.
Initial successful build - layers sequenced from bottom up:
Chriss-MacBook-Pro:~ technicaltitch$ docker history fdw_celery
IMAGE CREATED CREATED BY SIZE COMMENT
d71bf8f2b902 28 hours ago /bin/sh -c #(nop) ENTRYPOINT ["/bin/sh" "-c" 0 B
336bdb2b9ca9 28 hours ago /bin/sh -c #(nop) USER [celery] 0 B
a1259f89bc1f 28 hours ago |3 PIP_INDEX_URL=https://pypi.python.org/simp 15.39 MB
57dd9a330337 28 hours ago |3 PIP_INDEX_URL=https://pypi.python.org/simp 335.2 kB
3accad8aa55c 28 hours ago |3 PIP_INDEX_URL=https://pypi.python.org/simp 0 B
b4ff0c1d71fb 28 hours ago /bin/sh -c #(nop) COPY dir:a8f922c5264fe2275a 15.39 MB
bc65bc84abbc 28 hours ago |3 PIP_INDEX_URL=https://pypi.python.org/simp 0 B
f70ecc01cada 28 hours ago |3 PIP_INDEX_URL=https://pypi.python.org/simp 214.1 MB
85eb413bd5da 28 hours ago /bin/sh -c #(nop) ARG PIP_INDEX_URL=https:// 0 B
3cee97bf3ce1 28 hours ago /bin/sh -c #(nop) ARG PIP_TRUSTED_HOST=127.0 0 B
cb090c4b1886 28 hours ago /bin/sh -c #(nop) WORKDIR /usr/src/app 0 B
d0a1d3d15f94 28 hours ago /bin/sh -c #(nop) COPY dir:ec1503af2cbef5220d 3.145 kB
e27ecd707562 28 hours ago |1 http_proxy= /bin/sh -c mkdir -p /usr/src/a 0 B
0b6e2857206c 28 hours ago |1 http_proxy= /bin/sh -c VERSION=$(gdal-conf 5.903 MB
2966946ca235 29 hours ago |1 http_proxy= /bin/sh -c apt-get update && a 241.4 MB
eb867117097c 42 hours ago /bin/sh -c #(nop) CMD ["python2"] 0 B
Second, failing build, with same script:
Chriss-MacBook-Pro:~ technicaltitch$ docker history 5ee
IMAGE CREATED CREATED BY SIZE COMMENT
5ee29875a771 29 hours ago /bin/sh -c #(nop) ARG PIP_INDEX_URL=https:// 0 B
daad37fd701b 29 hours ago /bin/sh -c #(nop) ARG PIP_TRUSTED_HOST=127.0 0 B
f891ee9277da 29 hours ago /bin/sh -c #(nop) WORKDIR /usr/src/app 0 B
2cce4acf9f2f 29 hours ago /bin/sh -c #(nop) COPY dir:ec1503af2cbef5220d 3.145 kB
9484e6b7fa51 29 hours ago |1 http_proxy=None /bin/sh -c mkdir -p /usr/s 0 B
0031cdd56926 29 hours ago |1 http_proxy=None /bin/sh -c VERSION=$(gdal- 5.903 MB
03e5040df047 <-- [already diverged] |1 http_proxy=None /bin/sh -c apt-get update 243.1 MB
eb867117097c 42 hours ago /bin/sh -c #(nop) CMD ["python2"] 0 B
Finally, this is the image history after it magically succeeded. I had not deleted the other images when this worked (and still haven't):
Chriss-MacBook-Pro:~ technicaltitch$ docker history fdw_app
IMAGE CREATED CREATED BY SIZE COMMENT
edc615da4d6f 26 hours ago /bin/sh -c #(nop) ENTRYPOINT ["/usr/src/app/ 0 B
747c8744f6a7 26 hours ago /bin/sh -c #(nop) EXPOSE 8000/tcp 0 B
8c543333e10b 26 hours ago /bin/sh -c #(nop) USER [django] 0 B
b08f02d80d29 26 hours ago |3 PIP_INDEX_URL=https://pypi.python.org/simp 15.39 MB
ba42415ad78b 26 hours ago |3 PIP_INDEX_URL=https://pypi.python.org/simp 335.2 kB
027e0c8e39a9 26 hours ago |3 PIP_INDEX_URL=https://pypi.python.org/simp 0 B
d56a78c02d18 26 hours ago /bin/sh -c #(nop) COPY dir:9274abe4540edd1e86 15.39 MB
bc65bc84abbc 28 hours ago |3 PIP_INDEX_URL=https://pypi.python.org/simp 0 B
f70ecc01cada 28 hours ago |3 PIP_INDEX_URL=https://pypi.python.org/simp 214.1 MB
85eb413bd5da 28 hours ago /bin/sh -c #(nop) ARG PIP_INDEX_URL=https:// 0 B
3cee97bf3ce1 28 hours ago /bin/sh -c #(nop) ARG PIP_TRUSTED_HOST=127.0 0 B
cb090c4b1886 28 hours ago /bin/sh -c #(nop) WORKDIR /usr/src/app 0 B
d0a1d3d15f94 28 hours ago /bin/sh -c #(nop) COPY dir:ec1503af2cbef5220d 3.145 kB
e27ecd707562 28 hours ago |1 http_proxy= /bin/sh -c mkdir -p /usr/src/a 0 B
0b6e2857206c 28 hours ago |1 http_proxy= /bin/sh -c VERSION=$(gdal-conf 5.903 MB
2966946ca235 <-- [correct img, suddenly..why?!] ..xy= /bin/sh -c apt-get update && a 241.4 MB
eb867117097c 42 hours ago /bin/sh -c #(nop) CMD ["python2"] 0 B
The following is the start of both Dockerfiles. I have tried making the machine definitions in docker-compose.yml identical but to no avail (possibly after the 03e image was created).
FROM python:2.7
RUN apt-get update && apt-get install -y vim gdal-bin libgdal-dev postgresql-client
# On Jessie, $(gdal-config --version) gives 1.10.1 and Pypi only has 1.10.0
# so we need to strip the last part of the version number
RUN VERSION=$(gdal-config --version); CFLAGS=$(gdal-config --cflags) easy_install GDAL==${VERSION%.*}
RUN mkdir -p /usr/src/app/requirements
COPY requirements /usr/src/app/requirements
WORKDIR /usr/src/app
ARG PIP_TRUSTED_HOST=127.0.0.1
ARG PIP_INDEX_URL=https://pypi.python.org/simple/
RUN pip install --no-cache-dir -r requirements/production.txt
# Clean up APT when done.
RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
COPY . /usr/src/app
RUN mkdir -p /usr/src/app/log

Answer 1

COPY requirements /usr/src/app/requirements
If you change ANY file in this directory it will invalidate the cache from this point onward. This includes tmp files created by your editor, unless you put them in your .dockerignore file.
ARG PIP_TRUSTED_HOST=127.0.0.1
ARG PIP_INDEX_URL=https://pypi.python.org/simple/
If you change either of these ARG it will invalidate the cache from this point onward.
My guess is that one of these changes, causing it to skip the cache.

Answer 2

The ---> 03e5040df047 is just a temporary name for the container this does not indicate diferent images...
Docker uses the cache if the base image is known and the commands are the same. For ADD od COPY also the checksum of the files has to match as well. As if I can tell docker uses the cache in your snippet.
You can only force docker to not use the cache.