Repository/image with invalid name in Sonatype Nexus3 Docker registry - docker-registry

Context
I am using the Docker registry feature in Sonatype Nexus3 (version 3.18.1-01 OSS) to store private Docker images.
A couple of days ago, images with invalid names ended up in the registry, causing multiple reoccurring errors with internal cleanup cron jobs within Nexus3. You can see the faulty repositories names in the v2/_catalog request output below.
$ curl https://registry.example.com/v2/_catalog -i
HTTP/1.1 200 OK
Date: Wed, 11 Mar 2020 12:55:29 GMT
Server: Nexus/3.18.1-01 (OSS)
X-Content-Type-Options: nosniff
Content-Security-Policy: sandbox allow-forms allow-modals allow-popups allow-presentation allow-scripts allow-top-navigation
X-XSS-Protection: 1; mode=block
Docker-Distribution-Api-Version: registry/2.0
Content-Type: application/json
Content-Length: 2941
{
"repositories": [
"MiniProfiler.EF6",
"MiniProfiler.Shared",
...
]
}
At this point, I do not know if these images were pushed in the registry by someone on my team or if they are the result of some unexpected blob store problem - I don't have any way to verify this. Interesting fact: there are NuGet packages named identically in another blob store within the same Nexus instance.
Troubleshooting steps
I have been trying to remove the faulty images from the registry by multiple means, all without success.
Nexus3 GUI
The faulty repositories are not displayed in the Docker registry browser in Nexus' GUI. Neither can they be found by using Nexus' search feature.
Nexus3 embedded OrientDB
I looked for the asset and component classes associated with the faulty repositories in OrientDB. Unfortunately, there are no assets or components with such names in Nexus's database.
Docker registry API v2
I also tried to delete them via the Docker registry API V2, but the invalid name format seems to be causing issues when I attempt to do any operation on the faulty repository.
$ curl https://registry.example.com/v2/MiniProfiler.EF6/manifests/latest -i
HTTP/1.1 400 Bad Request
Date: Wed, 11 Mar 2020 12:58:24 GMT
Server: Nexus/3.18.1-01 (OSS)
X-Content-Type-Options: nosniff
Content-Security-Policy: sandbox allow-forms allow-modals allow-popups allow-presentation allow-scripts allow-top-navigation
X-XSS-Protection: 1; mode=block
Docker-Distribution-Api-Version: registry/2.0
Content-Type: application/json
Content-Length: 165
{
"errors": [
{
"code": "NAME_INVALID",
"message": "invalid repository name",
"detail": [
{
"Name": "MiniProfiler.EF6"
},
{
"Reason": "invalid path component: MiniProfiler.EF6"
}
]
}
]
}
Docker CLI
As expected, the Docker CLI doesn't like the invalid name format either.
$ docker pull registry.example.com/MiniProfiler.EF6
invalid reference format: repository name must be lowercase
If I remove the capital letters from the image name, the registry returns a 404 Not Found error.
$ docker pull registry.example.com/miniprofiler.ef6
Using default tag: latest
Error response from daemon: manifest for registry.example.com/miniprofiler.ef6:latest not found: manifest unknown: manifest unknown
Questions
Has anyone ever encountered such problems with Nexus3? Has anyone ever encountered such problems with another Docker registry implementation?
If anyone has any troubleshooting steps to suggest, I'm pretty much open to anything at this point...
I will also be filing a bug with Sonatype to ensure this situation gets good visibility.

Related

Docker Registry v2 authentication using OAuth2 does not return refresh token when `access_type=offline`

By following command snippet in https://docs.docker.com/registry/spec/auth/oauth/ as below and set access_type=offline, refresh_token is not present in returned response.
curl -iX POST https://auth.docker.io/token
-H "Content-Type: application/x-www-form-urlencoded"
-d "grant_type=password&username=${user}&password=${password}&service=hub.docker.io&client_id=dockerengine&access_type=offline"
Command succeeds with response below:
HTTP/1.1 200 OK
content-type: application/json
date: Tue, 04 Jan 2022 03:08:37 GMT
transfer-encoding: chunked
strict-transport-security: max-age=31536000
{
"access_token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsIng1YyI6WyJNSUlDK1RDQ0FwK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakJHTVVRd1FnWURWUVFERXp0U1RVbEdPbEZNUmpRNlEwZFFNenBSTWtWYU9sRklSRUk2VkVkRlZUcFZTRlZNT2taTVZqUTZSMGRXV2pwQk5WUkhPbFJMTkZNNlVVeElTVEFlRncweU1UQXhNalV5TXpFMU1EQmFGdzB5TWpBeE1qVXlNekUxTURCYU1FWXhSREJDQmdOVkJBTVRPMVZQU1ZJNlJFMUpWVHBZVlZKUk9rdFdRVXc2U2twTFZ6cExORkpGT2tWT1RFczZRMWRGVERwRVNrOUlPbEpYTjFjNlRrUktWRHBWV0U1WU1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBbnZBeVpFK09sZHgrY3hRS0RBWUtmTHJJYk5rK2hnaEg3Ti9mTFpMVDhEYXVPMXRoTWdoamxjcGhFVkNuYTFlMEZpOHVsUlZ4WG1HdWpZVDNXbnFsZ2ZpM2ZYTUQvQlBRTmlkWHZkeWprbDFZS3dPTkl3TkFWMnRXbExxaXFsdGhSWkFnTFdvWWZZMXZQMHFKTFZBbWt5bUkrOXRBcEMxNldNZ1ZFcHJGdE1rNnV0NDlMcDlUR1J0aDJQbHVWc3RSQ1hVUGp4bjI0d3NnYlUwVStjWTJSNEpyZmVJdzN0T1ZKbXNESkNaYW5SNmVheFYyVFZFUkxoZnNGVTlsSHAzcldCZ1RuNVRCSHlMRDNRdGVFLzJ3L3MvcUxZcmdIK1hCMmZBazJPd1NIRG5YWDg4WWVJd0EyVGJJMDdYNS8xQnVsaUwrUDduOWVBT1RmbDkxVlZwNER3SURBUUFCbzRHeU1JR3ZNQTRHQTFVZER3RUIvd1FFQXdJSGdEQVBCZ05WSFNVRUNEQUdCZ1JWSFNVQU1FUUdBMVVkRGdROUJEdFZUMGxTT2tSTlNWVTZXRlZTVVRwTFZrRk1Pa3BLUzFjNlN6UlNSVHBGVGt4TE9rTlhSVXc2UkVwUFNEcFNWemRYT2s1RVNsUTZWVmhPV0RCR0JnTlZIU01FUHpBOWdEdFNUVWxHT2xGTVJqUTZRMGRRTXpwUk1rVmFPbEZJUkVJNlZFZEZWVHBWU0ZWTU9rWk1WalE2UjBkV1dqcEJOVlJIT2xSTE5GTTZVVXhJU1RBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlFQTBkN3l1azQrWElabmtQb3RJVkdCeHBRSndpMzQwdExSb3R3Qzl4NkJpdWNDSUhFSmIyWGg0QzhtYVZic1Exd3ZUSCthRGV0VXhBS21lYkdXa3F6Z1J1Z1QiXX0.eyJhY2Nlc3MiOltdLCJhdWQiOiJodWIuZG9ja2VyLmlvIiwiZXhwIjoxNjQxMjY2MDE3LCJpYXQiOjE2NDEyNjU3MTcsImlzcyI6ImF1dGguZG9ja2VyLmlvIiwianRpIjoiV1dUV090ZVhnVWUwM0tWNWUwbEgiLCJuYmYiOjE2NDEyNjU0MTcsInN1YiI6ImM3YWJkMmU3LTJmNDgtNGFmNS1hOTExLTk5ZGM2MWQ2MmQ4OSJ9.D6YL422MrrS6bPv6A_BqEZa-6DhOWlkOvI2y2kq1uaIubSG09G7zodw97EE2RH2_1Wl94l0nVmN4nxSWHQvXT-e7v69XzLuO1gRxlFMZzmupn4JMRQ42UlFPM3VIKWeV3Opx4zLbtLvY1y9fR_ZSa3jcbP3HLKhBWH4dqYyp_oaFd3nVEgngEksyivqZHYu0JYID-EGw-2mZFFlLT030U3DcsFqcTsZWa1jfeDZIsxjdhEkqsxKbfqOpSY6-6p4b6Y0-1FDw1EiX2q4Y6PzbMfNJg9v_lQAftSUuCzMqrhVtrvPn07Su0nN_BpAJ5fDum5jHS1gDmmX7pnGnB0gd0g",
"scope": "",
"expires_in": 300,
"issued_at": "2022-01-04T03:08:37.398945485Z"
}
Document explicitly said :
refresh_token
(Optional) Token which can be used to get additional access tokens for the same subject with different scopes. This token should be kept secure by the client and only sent to the authorization server which issues bearer tokens. This field will only be set when access_type=offline is provided in the request.
The same effect is observed when I tested deployment of a private docker registry:2.7 along with a docker_auth (https://github.com/cesanta/docker_auth, version 1.9) authentication server.
From Docker registry OAuth specification, it seems the feature is already in place but if it does not work on Docker auth server and the other project follows this specification, I can't help to wonder if this is a feature in future or just I missed somethings in my configuration.

Jenkins prometheus plugin not showing data on endpoint

My Jenkins - 2.263.1(LTS) deployed through tomcat and i have installed Prometheus metrics plugin - 2.0.8 and restarted the service.
My jenkins base URL - http://jenkins-server:8080/jenkins
But my prometheus end-point - http://jenkins-server:8080/jenkins/prometheus not showing any metrics data.
I have added below in my prometheus.yml
- job_name: 'jenkins'
metrics_path: '/jenkins/prometheus'
scheme: http
static_configs:
- targets: ['jenkins-server:8080']
Currently LDAP authentication and Project-based Matrix Authorization configured. Also i have tried with domain credential password and token in my prometheus.yml but still it doesn't show the plugin generated data in my end-point. Just shows the blank page on my browsers(IE and Chrome).
basic_auth:
username: domain-user-id
password: 98qw37asdkdsjfeiq1dedsewe
Curl response
$ curl -v jenkins-server:8080/jenkins/prometheus
* Trying 206.25.26.27...
* TCP_NODELAY set
* Connected to jenkins-server (206.25.26.27) port 8080 (#0)
> GET /jenkins/prometheus HTTP/1.1
> Host: jenkins-server:8080
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 302
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< X-Content-Type-Options: nosniff
< Location: /jenkins/prometheus/
< Content-Length: 0
< Date: Wed, 17 Feb 2021 11:42:00 GMT
<
* Connection #0 to host jenkins-server left intact
$ curl -X GET jenkins-server:8080/jenkins/prometheus/
$ curl -X GET http://jenkins-server:8080/jenkins/prometheus/
Empty response for above commands. Please share some pointers to resolve this issue. thanks in advance.
#poshak, Generated access key and tired on my browser with https://jenkins_ipaddres:portnumber/jenkins/metrics/accesskey Now i can able to view the below.
Is these data is enough for promethues?
Try to generate Access Keys in the metrics section and access the url https://jenkins_ipaddres:portnumber/metrics/accesskey you would now be able to view the metrics.
Path to generate the Access Keys:
Jenkins > Manage Jenkins > Configure Systems > Metrics >> Add >> Generate >> Save
Thanks
It was an Jenkins Prometheus plugin issue. After upgrading it to 2.0.9 issue solved.

Docker registry: unable to push image since dns_unresolved_hostname

I'm not able to push an image to my local registry
$ docker image push registry.local:5000/covid-backend:60988b0-dirty
The push refers to repository [registry.local:5000/covid-backend]
eff147c1024b: Preparing
790a9d8e41bb: Preparing
20dd87a4c2ab: Preparing
78075328e0da: Preparing
9f8566ee5135: Preparing
error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<HTML><HEAD>\r\n<TITLE>Network Error</TITLE>\r\n</HEAD>\r\n<BODY>\r\n<FONT face=\"Helvetica\">\r\n<big><strong></strong></big><BR>\r\n</FONT>\r\n<blockquote>\r\n<TABLE border=0 cellPadding=1 width=\"80% \">\r\n<TR><TD>\r\n<FONT face=\"Helvetica\">\r\n<big>Network Error (dns_unresolved_hostname)</big>\r\n<BR>\r\n<BR>\r\n</FONT>\r\n</TD></TR>\r\n<TR><TD>\r\n<FONT face=\"Helvetica\">\r\nYour requested host \"registry.local\" could not be resolved by DNS.\r\n</FONT>\r\n</TD></TR>\r\n<TR><TD>\r\n<FONT face=\"Helvetica\">\r\n\r\n</FONT>\r\n</TD></TR>\r\n<TR><TD>\r\n<FONT face=\"Helvetica\" SIZE=2>\r\n<BR>\r\nFor assistance, contact your network support team.\r\n</FONT>\r\n</TD></TR>\r\n</TABLE>\r\n</blockquote>\r\n</FONT>\r\n</BODY></HTML>\r\n"
HTML content response contains:
Network Error (dns_unresolved_hostname)
Your requested host \\"registry.local\\" could not be resolved by DNS.
I've tried to reach it using curl:
$ curl -s registry.local:5000/v2/_catalog
{"repositories":["covid-backend","skaffold-covid-backend"]}
My /etc/hosts:
127.0.0.1 localhost registry.local
I've also tried to add it into my ~/.docker/config.json as insecure registry:
"insecure-registries" : [
"registry.local:5000"
]
I've also took a look on docker logs:
abr 27 09:30:25 psgd dockerd[15476]: time="2020-04-27T09:30:25.967945384+02:00" level=info msg="Attempting next endpoint for push after error: Get https://registry.local:5000/v2/: Service Unavailable"
abr 27 09:30:29 psgd dockerd[15476]: time="2020-04-27T09:30:29.121878880+02:00" level=error msg="Upload failed: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: \"<HTML><HEAD>\\r\\n<TITLE>Network Error</TITLE>\\r\\n</HEAD>\\r\\n<BODY>\\r\\n<FONT face=\\\"Helvetica\\\">\\r\\n<big><strong></strong></big><BR>\\r\\n</FONT>\\r\\n<blockquote>\\r\\n<TABLE border=0 cellPadding=1 width=\\\"80% \\\">\\r\\n<TR><TD>\\r\\n<FONT face=\\\"Helvetica\\\">\\r\\n<big>Network Error (dns_unresolved_hostname)</big>\\r\\n<BR>\\r\\n<BR>\\r\\n</FONT>\\r\\n</TD></TR>\\r\\n<TR><TD>\\r\\n<FONT face=\\\"Helvetica\\\">\\r\\nYour requested host \\\"registry.local\\\" could not be resolved by DNS.\\r\\n</FONT>\\r\\n</TD></TR>\\r\\n<TR><TD>\\r\\n<FONT face=\\\"Helvetica\\\">\\r\\n\\r\\n</FONT>\\r\\n</TD></TR>\\r\\n<TR><TD>\\r\\n<FONT face=\\\"Helvetica\\\" SIZE=2>\\r\\n<BR>\\r\\nFor assistance, contact your network support team.\\r\\n</FONT>\\r\\n</TD></TR>\\r\\n</TABLE>\\r\\n</blockquote>\\r\\n</FONT>\\r\\n</BODY></HTML>\\r\\n\""
abr 27 09:30:29 psgd dockerd[15476]: time="2020-04-27T09:30:29.122824956+02:00" level=info msg="Attempting next endpoint for push after error: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: \"<HTML><HEAD>\\r\\n<TITLE>Network Error</TITLE>\\r\\n</HEAD>\\r\\n<BODY>\\r\\n<FONT face=\\\"Helvetica\\\">\\r\\n<big><strong></strong></big><BR>\\r\\n</FONT>\\r\\n<blockquote>\\r\\n<TABLE border=0 cellPadding=1 width=\\\"80% \\\">\\r\\n<TR><TD>\\r\\n<FONT face=\\\"Helvetica\\\">\\r\\n<big>Network Error (dns_unresolved_hostname)</big>\\r\\n<BR>\\r\\n<BR>\\r\\n</FONT>\\r\\n</TD></TR>\\r\\n<TR><TD>\\r\\n<FONT face=\\\"Helvetica\\\">\\r\\nYour requested host \\\"registry.local\\\" could not be resolved by DNS.\\r\\n</FONT>\\r\\n</TD></TR>\\r\\n<TR><TD>\\r\\n<FONT face=\\\"Helvetica\\\">\\r\\n\\r\\n</FONT>\\r\\n</TD></TR>\\r\\n<TR><TD>\\r\\n<FONT face=\\\"Helvetica\\\" SIZE=2>\\r\\n<BR>\\r\\nFor assistance, contact your network support team.\\r\\n</FONT>\\r\\n</TD></TR>\\r\\n</TABLE>\\r\\n</blockquote>\\r\\n</FONT>\\r\\n</BODY></HTML>\\r\\n\""
My NO_PROXY environment variable content:
$ echo $NO_PROXY
localhost,127.0.0.1/8,::1,192.168.99.0/8,registry.local
The problem is that I need to configure correctly docker behind proxy.
I though proxy-related environment variables is enough (HTTP_PROXY, HTTPS_PROXY and NO_PROXY).
You can find how to configure docker behind a proxy here.
I've just added NO_PROXY on /etc/systemd/system/docker.service.d/ files like:
[Service]
Environment="HTTP_PROXY=http://<proxy-ip>:<proxy-port>/" "NO_PROXY=localhost, 127.0.0.1/8, ::1, 192.168.99.0/8, registry.local"
Solved.

Trying to configure build agent to work with a proxy server

I'm trying to make my build agent work with an already configured proxy server.
The proxy server address is : http://MY_SERVER_DNS:8081
On my build machine inside the agent directory I created a .proxy file which contains the above proxy address and created the relevant environment variables (VSTS_HTTP_PROXY_USERNAME, VSTS_HTTP_PROXY_PASSWORD).
After a restart to the build agent service the indicator in VSTS goes red (instead of green).
A partial (most relevant I guess) agent log is attached:
{ Request = Method: GET, RequestUri: 'https://siemplify.visualstudio.com/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1', Version: 1.1, Content: <null>, Headers:
{
User-Agent: VSServices/15.255.65000.0
User-Agent: (NetStandard; Microsoft Windows 6.3.9600)
User-Agent: VstsAgentCore-win7-x64/2.120.2
User-Agent: (Microsoft Windows 6.3.9600)
X-VSS-E2EID: 5aadb1b3-6269-4998-b258-4a5fcc1b9345
Accept-Language: en-US
X-TFS-FedAuthRedirect: Suppress
X-TFS-Session: 13f3aaa0-7f5c-40e1-8af0-0b5feb53d4bc
Expect: 100-continue
}, LoggingRequestId = ab31853f-a392-486f-b288-f9ca4bdee28c, Timestamp = 237057153 }
[2018-03-08 14:15:50Z INFO HttpTrace] Trace System.Net.Http.Response event:
{ Response = StatusCode: 400, ReasonPhrase: 'Bad Request', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Connection: close
Date: Thu, 08 Mar 2018 14:15:32 GMT
Server: Microsoft-HTTPAPI/2.0
Content-Length: 324
Content-Type: text/html; charset=us-ascii
}, LoggingRequestId = ab31853f-a392-486f-b288-f9ca4bdee28c, TimeStamp = 237507550 }
[2018-03-08 14:15:51Z ERR VisualStudioServices] GET request to https://siemplify.visualstudio.com/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 failed. System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to read data from the transport connection. The connection was closed before all data could be read. Expected 324 bytes, read 0 bytes.
Any help would be greatly appreciated!
You could first upgrade your private build agent by right-click the agent pool, and then click Update all agents. The latest version of build agent is 2.129.1.
After updating the agent, try to follow the steps in the following link to run a v2 private agent behind a web proxy: VSTS, TFS 2018 RTM and newer:
Pass --proxyurl, --proxyusername and --proxypassword during agent configuration.
For example:
./config.cmd --proxyurl http://127.0.0.1:8888 --proxyusername "1" --proxypassword "1"
Make sure that the proxy file that you created doesn't have spaces around the proxy address which you mention in the proxy file. Make sure you trim it off.
More info here link

Terraform apply access denied error

So I was upgrading Terraform from 0.9.5 to 0.9.6 and I am now getting the following error when I run a jenkins job on a build slave with IAM permissions attached:
terraform096 apply -var db_snap_stamp=171120171217 -var db_snapshot=rds-dev-13102017 -var-file=env.tfvars -no-color
Error loading state: AccessDenied: Access Denied
status code: 403, request id: 288766CE5CCA24A0, host id: FOOBAR
The jenkins job does run terraform init before hand and on my local test server I am not seeing the error. On the local test server I am using an aws credentials file.
I have had a look through the release notes for 0.9.6 but I can't see which of the changes could be causing this ( https://github.com/hashicorp/terraform/issues/14423 maybe?).
Any ideas?
UPDATE
I turned on terraform debug and found that the 403 was happening on a s3 list object. The IAM role in use allows this in 0.9.5 but NOT in 0.9.6 - I tried giving the role admin access but no change:
-----------------------------------------------------
2017/11/17 15:01:47 [DEBUG] [aws-sdk-go] DEBUG: Response s3/ListObjects
Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 403 Forbidden
Connection: close
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 17 Nov 2017 15:01:47 GMT
Server: AmazonS3
X-Amz-Bucket-Region: eu-west-2
The S3 bucket in question does use KMS encryption but all that is set up in the init run prior:
terraform096 init -backend=true -get=true -input=false -backend-config="bucket=${BUCKET}" -backend-config="key=${ENV}.tfstate" -backend-config="region=eu-west-2" backend-config="profile=${AWS_PROFILE}" -backend-config="encrypt=true" -backend-config="kms_key_id=${KMS}"
So I raised this as an issue with Hashicorp and this does look like a bug - if anyone wants to keep an eye on the progress have a look here:
https://github.com/hashicorp/terraform/issues/16710

Resources