Deploying Docker images from Gitlab's private Docker registry to Openshift - docker

Question

I'm working on a Docker-based project. The project code is hosted in a private Gitlab installation, git.example.com. With it, the Docker private registry shipped with Gitlab is deployed, registry.example.com.
The project has a CI setup which ends up building Docker images and pushing to the registry, this part works as expected. As Gitlab+Docker registry does not yet support multiple images related to the same Git repo, I'm using the tags workaround which specifies an image as:
registry.example.com/group/my.project:web
registry.example.com/group/my.project:app
etc.
I've created a user and attached it to the projects, logged in via it locally and tried to pull above images, that works as expected.
I've added the ImageStream block as so:
apiVersion: v1
kind: ImageStream
metadata:
name: web
spec:
tags:
-
from:
kind: DockerImage
name: registry.example.com/group/my.project:web
name: latest
This adds the image in the Images section, but it cannot pull it Openshift doesn't have access to the Docker Registry yet. I add a new Docker secret as described here and am now able to see image metadata in Openshift, everything looks as expected.
But, if I add a deployment config, like so:
apiVersion: v1
kind: DeploymentConfig
metadata:
creationTimestamp: null
labels:
service: web
name: web
spec:
replicas: 1
selector:
service: web
strategy:
resources: { }
template:
metadata:
creationTimestamp: null
labels:
service: web
spec:
containers:
-
name: web
ports:
-
containerPort: 80
resources: { }
restartPolicy: Always
test: false
triggers:
-
type: ConfigChange
-
type: ImageChange
imageChangeParams:
automatic: true
containerNames:
- web
from:
kind: ImageStreamTag
name: 'web:latest'
status: { }
I keep getting error:
Failed to pull image "registry.example.com/group/my.project#sha256:3333022641e571d7e4dcae2953d35be8cdf9416b13967b99537c4e8f150f74e4": manifest unknown: manifest unknown
in the Events tab of the pod created. This basically kills my plan to deploy prebuilt images to Openshift.
I know about Docker 1.9 -> 1.10 incompatibility, but this is Openshift 1.4.1, images were pushed with Docker 1.13 so it shouldn't be a problem.
How do I even start debugging this, is there a way to access any sort of log which would explain what's going on? Why is ImageStream able to find everything it needs (and access my registry), but not the DeploymentConfig?

Answer 1

To answer my own question: it seems Docker's Distribution (registry daemon) has a bug which manifests itself in quite a weird way.
Basically, the problem is:
Registry is behind Apache reverse proxy
the image gets built and pushed from CI runner to Gitlab's Registry, digest SHA:1234 (example, of course)
the image gets imported to Openshift, it queries the metadata and Docker Distribution claims the digest is SHA:ABCD, you can reproduce this by pushing and then pulling right away, the digests are supposed to be identical both times, as explained in the link
when Openshift tries to actually pull the image, it will get the dreaded "Manifest unknown" error above (as it's trying to fetch the image using an invalid digest, not by fault of its own)
all symptoms look exactly like with Docker v1 => Docker v2 API changes, except for totally different reasons
I've since moved my Gitlab instance to another machine (where it's behind Nginx) and it works without a problem.