I've set up Docker and running a private repository on example.com:5000. I followed the instructions listed here: https://docs.docker.com/registry/deploying/
And uses the docker-compose.yml:
registry:
restart: always
image: registry:2
ports:
- 5000:5000
environment:
REGISTRY_HTTP_TLS_CERTIFICATE: /certs/domain.crt
REGISTRY_HTTP_TLS_KEY: /certs/domain.key
REGISTRY_AUTH: htpasswd
REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
volumes:
- /path/data:/var/lib/registry
- /path/certs:/certs
- /path/auth:/auth
I can push and pull images to the repository, but I can't get docker search example.com:5000/library to run. I get an: Error response from daemon: Unexpected status code 404.
When I point curl to the endpoint I get the following result:
$ curl -v -X GET http://example.com:5000/v2/images
Note: Unnecessary use of -X or --request, GET is already inferred.
* Trying 37.139.20.160...
* Connected to example.com (192.167.201.2) port 5000 (#0)
> GET /v2/images HTTP/1.1
> Host: domain.com:5000
> User-Agent: curl/7.47.0
> Accept: */*
>
* Connection #0 to host example.com left intact
How can I make the search command working so that I can manage the repository? And where can I find the API documentation of the endpoint? Or are there better ways to manage a Docker private repo?
It seems you have to activate the search option, according to Search-engine options
The Docker Registry can optionally index repository information in a database for the GET /v1/search endpoint.
(I don't see a search in the V2 API. You can list tags)
The search_backend setting selects the search backend to use.
If search_backend is empty, no index is built, and the search endpoint always returns empty results.
For instance, using the SQLAlchemy database
common:
search_backend: sqlalchemy
sqlalchemy_index_database: sqlite:////tmp/docker-registry.db
On initialization, the SQLAlchemyIndex class checks the database version. If the database doesn't exist yet (or does exist, but lacks a version table), the SQLAlchemyIndex creates the database and required tables.
Related
I am trying to find a way to import a realm in Keycloak version 17.0.1 that can be done at starting up a docker container (with docker-compose). I want to be able to do this in "start" mode and not "start-dev" mode as in my experience so far "start-dev" in 17 is forcing an H2/in-mem database and not allowing me to point to an external db which I would like to do to more closely resemble dev/prod environments when running locally.
Things I've tried:
1) It appears that according to recent conversations on Github (Issue 10216 and Issue 10754 to name a couple) that the environment variable that used to allow this (KEYCLOAK_IMPORT or KC_IMPORT_REALM in some versions) is no longer a trigger for this. In my attempts it also did not work for version 17.0.1.
2) I've also tried appending the following command in my docker-compose setup for keycloak and had no luck (also tried with just "start") - It appears to just ignore the command (no error or anything):
command: ["start-dev", "-Dkeycloak.import=/tmp/my-realm.json"]
3) I tried running the kc.sh command "import" in the Dockerfile (both before and after Entrypoint/start) but got error: Unmatched arguments from index 1: '/opt/keycloak/bin/kc.sh', 'im port', '--file', '/tmp/my-realm.json'
4) I've shifted gears and have tried to see if it is possible to just do it after the container starts (even with manual intervention) just to get some sanity restored. I attempted to use the admin-cli but after quite a few different attempts at different points/endpoints etc. I just get that localhost refuses to connect.
bin/kcadm.sh config credentials --server http://localhost:8080/auth --realm master --user admin --password adminpassword
Responds when hitting the following ports as shown:
8080: Failed to send request - Connect to localhost:8080 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
8443: Failed to send request - localhost:8443 failed to respond
I am sure there are other ways that I've tried and am forgetting - I've kind of spun my wheels at this point.
My code (largely the same as the latest docs on the Keycloak website):
Dockerfile:
FROM quay.io/keycloak/keycloak:17.0.1 as builder
ENV KC_METRICS_ENABLED=true
ENV KC_FEATURES=token-exchange
ENV KC_DB=postgres
RUN /opt/keycloak/bin/kc.sh build
FROM quay.io/keycloak/keycloak:17.0.1
COPY --from=builder /opt/keycloak/lib/quarkus/ /opt/keycloak/lib/quarkus/
WORKDIR /opt/keycloak
# for demonstration purposes only, please make sure to use proper certificates in production instead
ENV KC_HOSTNAME=localhost
RUN keytool -genkeypair -storepass password -storetype PKCS12 -keyalg RSA -keysize 2048 -dname "CN=server" -alias server -ext "SAN:c=DNS:localhost,IP:127.0.0.1" -keystore conf/server.keystore
ENTRYPOINT ["/opt/keycloak/bin/kc.sh", "start" ]
Docker-compose.yml:
version: "3"
services:
keycloak:
build:
context: .
volumes:
- ./my-realm.json:/tmp/my-realm.json:ro
env_file:
- .env
environment:
KC_DB_URL: ${POSTGRESQL_URL}
KC_DB_USERNAME: ${POSTGRESQL_USER}
KC_DB_PASSWORD: ${POSTGRESQL_PASS}
KEYCLOAK_ADMIN: admin
KEYCLOAK_ADMIN_PASSWORD: adminpassword
ports:
- 8080:8080
- 8443:8443 # <-- I've tried with only 8080 and with only 8443 as well. 8443 appears to be the only that I can get the admin console ui to even work on though.
networks:
- my_net
networks:
my_net:
name: my_net
Any suggestion on how to do this in a programmatic + "dev-opsy" way would be greatly appreciated. I'd really like to get this to work but am confused on how to get past this.
Importing realm upon docker initialization thru configuration is not supported yet. See https://github.com/keycloak/keycloak/issues/10216. They might release this feature in next release v18.
The workarounds people had shared in github thread is create own docker image and import the realm thru json file when building it.
FROM quay.io/keycloak/keycloak:17.0.1
# Make the realm configuration available for import
COPY realm-and-users.json /opt/keycloak_import/
# Import the realm and user
RUN /opt/keycloak/bin/kc.sh import --file /opt/keycloak_import/realm-and-users.json
# The Keycloak server is configured to listen on port 8080
EXPOSE 8080
EXPOSE 8443
# Import the realm on start-up
CMD ["start-dev"]
As #tboom said, it was not supported yet by keycloak 17.x. But it is now supported by keycloak 18.x using the --import-realm option :
bin/kc.[sh|bat] [start|start-dev] --import-realm
This feature does not work as it was before. The JSON file path must not be specified anymore: the JSON file only has to be copied in the <KEYCLOAK_DIR>/data/import directory (multiple JSON files supported). Note that the import operation is skipped if the realm already exists, so incremental updates are not possible anymore (at least for the time being).
This feature is documented on https://www.keycloak.org/server/importExport#_importing_a_realm_during_startup.
I set up a Shopware 6 project with ddev. Now I want to write cypress tests for one of my plugins. The shopware testsuite starts a node express server on port 8005 in the web container. I have configured the port for ddev so that I can open the express endpoint in my browser: http://my.ddev.site:8005/cleanup. That is working.
For cypress I have created a new ddev container with a new docker-compose file:
version: '3.6'
services:
cypress:
container_name: ddev-${DDEV_SITENAME}-cypress
image: cypress/included:4.10.0
tty: true
ipc: host
links:
- web:web
environment:
- CYPRESS_baseUrl=https://web
- DISPLAY
labels:
com.ddev.site-name: ${DDEV_SITENAME}
com.ddev.approot: $DDEV_APPROOT
volumes:
# Project root
- ../shopware:/project
# Storefront and Administration
- ../shopware/vendor/shopware/platform/src/Storefront/Resources/app/storefront/test/e2e:/e2e-Storefront
- ../shopware/vendor/shopware/platform/src/Administration/Resources/app/administration/test/e2e:/e2e-Administration
# Custom plugins
- ../shopware/custom/plugins/MyPlugin/src/Resources/app/administration/test/e2e:/e2e-MyPlugin
# for Cypress to communicate with the X11 server pass this socket file
# in addition to any other mapped volumes
- /tmp/.X11-unix:/tmp/.X11-unix
entrypoint: /bin/bash
I can now successfully open the cypress interface and I see my tests. The problem is now, that always before a cypress test is executed, the express endpoint is called (with the URL from above) and the cypress container seems to has no access to the endpoint. This is the output:
cy.request() failed trying to load:
http://my.ddev.site:8005/cleanup
We attempted to make an http request to this URL but the request failed without a response.
We received this error at the network level:
> Error: connect ECONNREFUSED 127.0.0.1:8005
-----------------------------------------------------------
The request we sent was:
Method: GET
URL: http://my.ddev.site:8005/cleanup
So I can call this endpoint in my browser, but cypress can't. Is there any configuration in the cypress container missing to call the port 8005 from the web container?
You need to add this to the cypress service:
external_links:
- "ddev-router:${DDEV_HOSTNAME}"
and then your http URL will be accessed through the router via ".ddev.site".
If you need a trusted https URL it's a little more complicated, but for http this should work fine.
I have tried a lot, but I can't find a solution to this problem.
I am running a nexus sonatype (3.21.1-01) docker image on a centos7 server behind a vthunder a10 proxy.
The docker login and pull works great but docker push fail with EOF after some retrying.
Here the interested routes:
docker image port 8081 > my.server:8081
docker image port 8443 > my.server:8443
proxy.domain.local:443 > my.server:8081
proxy.domain.local:8443 > my.server:8443
I have created a docker repository in nexus which have the http connector exposed on 8443
The proxy is exposed under ssl with self signed certificate
The client's /etc/docker/daemon.json file contains the insecure registry options:
"insecure-registries": ["proxy.domain.local:8443","proxy.domain.local"]
Here the situation:
If I try to push from the client an image of which all layers already exist on the remote server (but missing on nexus repository), it works.
If I try the same but adding some difference to the same image (such as a new LABEL), it fail in this way:
(9c27e219663c: Layer already exists
Patch https://proxy.domain.local:8443/v2/test4/blobs/uploads/6862fe60-d63b-4942-bbb6-f403307e677a: EOF)
If I push directly from my.server machine, pointing to localhost:8443 it works.
If i push from the client machine an image with new layers it fail in this way after some retrying (the same behavior with smaller images):
docker push proxy.domain.local:8443/ara
The push refers to repository [proxy.domain.local:8443/ara]
edb7a4f74e22: Retrying in 8 seconds
de421654540d: Retrying in 8 seconds
-------------
The push refers to repository [proxy.domain.local:8443/ara]
edb7a4f74e22: Pushing [==================================================>] 172.6MB/172.6MB
de421654540d: Pushing [==================================================>] 200.8MB/200.8MB
EOF
this is a summary of what happen in wireshark
the.client my.server HTTP 316 GET /v2/ HTTP/1.1
...
my.server the.client HTTP 654 HTTP/1.1 401 Unauthorized (application/json)
...
the.client my.server HTTP 442 HEAD /v2/alpine-test/blobs/sha256:95f5ecd24e438e09033c8e69ec136079f8774ab8284f1431f5433a829054b5e7 HTTP/
(asking to nexus if the image is already uploaded)
my.server the.client HTTP 493 HTTP/1.1 404 Not Found
(it isn't)
the.client my.server HTTP 437 POST /v2/alpine-test/blobs/uploads/ HTTP/1.1
(so it start to post the image)
my.server the.client HTTP 584 HTTP/1.1 202 Accepted
...
the.client my.server HTTP 437 POST /v2/alpine-test/blobs/uploads/ HTTP/1.1
...
my.server the.client HTTP 584 HTTP/1.1 202 Accepted
..
and so on with some FIN/ACK in the middle until the client stops to send it...
** on nexus server log there is absolutely no trace about this **
this is the nexus docker compose:
services:
nexus:
build:
context: .
args:
DOCKER_GID: ${DOCKER_GID}
NEXUS_UID: ${NEXUS_UID}
NEXUS_GID: ${NEXUS_GID}
restart: always
environment:
- NEXUS_UID_GID=${NEXUS_UID_GID}
- HOSTNAME_DOCKER_NEXUS=${HOSTNAME_DOCKER_NEXUS}
ports:
- "8081:8081"
- "8443:8443"
user: ${NEXUS_UID_GID}
hostname: ${HOSTNAME_DOCKER_NEXUS}
volumes:
- /var/nexus-data:/nexus-data
- /etc/hosts:/etc/hosts
- /var/run/docker.sock:/var/run/docker.sock
Can you help me?
I was thinking about a possibile nexus-docker-user permission issue on the local machine/docker binary permissions (if i try from localhost it works, yes, but the image is already stored on the system of course) - but I think it is not so probable.
I was thinking also about proxy configuration issue (more probable), but I don't know much about proxy.
[Workaround]
Because I can not figure out the problem, I ended up with make proxy transparent and configuring nexus to serve directly in https throught it's jetty.xml, jetty.https and nexus.properties.
Serving https directly from jetty instead of let the proxy upgrade the connection solved the above problem.
When I install CouchDB and use the GUI and run verify.
I get the error
Error: could not resolve http://any:5984/verifytestdb/
And Replication status get's an X saying I can't replicate. Any suggestion on how to fix this problem.
It's running in a Docker Container and the Ports says
4369/tcp, 9100/tcp, 0.0.0.0:5984->5984/tcp
The GUI should same it works an not show an Error.
Feel like a port might be blocked in 5986 required for replication.
Use the Config setting on on the CouchDB GUI
Go to httpd
Then select bind_address
And and change the value from "Any" to "bind_address"
Run the test again and it should work.
for me what works is adding to couchdb config or change in UI
[httpd]
bind_address = 0.0.0.0
tested with verify and
curl -vX POST http://127.0.0.1:5984/_replicate -d '{"source":"albums","target":"albums-replica","create_target":true}' -H "Content-Type: application/json"
{"ok":true,"session_id":"9ab3e4f1a9cae16df05b32866088510c","source_last_seq":"6-g1AAAAILeJyNkU0OgjAQRqto1IVn0CMA_YGu5CZKOzVIsF2o......
with docker exposing only port
services:
couchdb:
ports:
- "5984:5984"
I'm trying to setup a CI server inside a corporate network with drone (open source edition). Its author describes drone as very simple solution even for programmer (as I am), though some moments are not clear for me (may be official documentation misses them).
First, I've made up an docker image for my rails application: rails-qna.
Next, composing drone images:
docker-compose.yml:
version: '2'
services:
drone-server:
image: drone/drone:0.5
ports:
- 80:8000
volumes:
- ./drone:/var/lib/drone/
restart: always
environment:
- DRONE_OPEN=true
- DRONE_ADMIN=khataev
- DRONE_GITHUB_CLIENT=github-client-string
- DRONE_GITHUB_SECRET=github-secret-string
- DRONE_SECRET=drone-secret-string
drone-agent:
image: drone/drone:0.5
command: agent
restart: always
depends_on: [ drone-server ]
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
- DRONE_SERVER=ws://drone-server:8000/ws/broker
- DRONE_SECRET=drone-secret-string
Application is registered on Github and secret/client strings are provided.
I placed .drone.yml file into my project repository:
pipeline:
build:
image: rails-qna
commands:
- bundle exec rake db:drop
- bundle exec rake db:create
- bundle exec rake db:migrate
- bundle exec rspec
Unclear moments:
1) While registering OAuth application on github, we should specify Homepage URL and authorization callback URL. Where should they point to? Drone server container? Guessing that so, I specified
mycorporatedomain.com:3005
and
mycorporatedomain.com:3005/authorize
and setup port forwarding from 3005 port to 80 port of host, where drone docker is running. May be I'm wrong?
2) What should I specify in key DRONE_GITHUB_URL?
https://github.com or full path to my project repository, i.e.
https://github.com/khataev/qna?
3) What if I want to build some branch other than master? Were should I specify it? For now drone ready branch (with .drone.yml) is not a master branch - would it work?
4) Why DRONE_GITHUB_GIT_USERNAME and DRONE_GITHUB_GIT_PASSWORD are optional? How it is supposed to work if, I don't specify username and password for my github account?
5) When I start drone images with docker up, I get this errors:
→ docker-compose up
Starting drone_drone-server_1
Starting drone_drone-agent_1
Attaching to drone_drone-server_1, drone_drone-agent_1
drone-server_1 | time="2017-03-04T17:00:33Z" level=fatal msg="version control system not configured"
drone-agent_1 | 1:M 04 Mar 17:00:35.208 * connecting to server ws://drone-server:8000/ws/broker
drone-agent_1 | 1:M 04 Mar 17:00:35.229 # connection failed, retry in 15s. websocket.Dial ws://drone-server:8000/ws/broker: dial tcp: lookup drone-server on 127.0.0.11:53: no such host
drone_drone-server_1 exited with code 1
drone-server_1 | time="2017-03-04T16:53:38Z" level=fatal msg="version control system not configured"
UPD
5) this was solved - forgot to specify
DRONE_GITHUB=true
Homepage URL is the address of the server where drone is running on.
E.g. http://155.200.100.0
Authorize URL is the same address appended by /authorize
Eg. http://155.200.100.0/authorize
You dont have to specify that. DRONE_GITHUB=true says drone to use github url.
You can limit a single section to a branch or the whole drone build.
Single Section:
pipeline:
build:
image: node:latest
commands:
- npm install
- npm test
when:
branch: master
Whole build process:
pipeline:
build:
image: node:latest
commands:
- npm install
- npm test
branches: master
You don't need username and password when using OAuth.
Source:
http://readme.drone.io/admin/setup-github/
http://readme.drone.io/usage/skipping-builds/
http://readme.drone.io/usage/skipping-build-steps/
UPDATE:
Documentation is shifted to http://docs.drone.io/ due to version 0.6 of Drone