I created a host using the DigitalOcean driver. However, I had to generate a new access token and am now unable to connect to it. How do I reconfigure docker-machine to use the new access token?
Each host spun up with docker-machine creates a folder holding its configuration under ~/.docker/machine/machines/ The DigitalOcean access token and other information about the droplet are stored in a config.json file. Let's look at a specific one as an example:
$ cat ~/.docker/machine/machines/docker-001/config.json | jq .
{
"DriverName": "digitalocean",
"Driver": {
"AccessToken": "9dasd89ssf6542notarealtoken455b44sdgf4685",
"DropletID": 4906043,
"DropletName": "",
"Image": "ubuntu-14-04-x64",
"MachineName": "docker-001",
"IPAddress": "45.32.128.70",
"Region": "nyc3",
"SSHKeyID": 7697371,
"Size": "512mb",
"CaCertPath": "/home/asb/.docker/machine/certs/ca.pem",
"PrivateKeyPath": "/home/asb/.docker/machine/certs/ca-key.pem",
"DriverKeyPath": "",
"SwarmMaster": false,
"SwarmHost": "tcp://0.0.0.0:3376",
"SwarmDiscovery": ""
},
"CaCertPath": "/home/asb/.docker/machine/certs/ca.pem",
"ServerCertPath": "",
"ServerKeyPath": "",
"PrivateKeyPath": "/home/asb/.docker/machine/certs/ca-key.pem",
"ClientCertPath": "",
"SwarmMaster": false,
"SwarmHost": "tcp://0.0.0.0:3376",
"SwarmDiscovery": ""
}
Related
I am setting up continuous integration using Jenkins and GitHub Enterprise.
So far I have been able to easily create Jenkins projects and run jobs regularly.
Jenkins is using SSH public/private keys to authenticate and clone GitHub repos.
Jenkins runs on a Linux server that is inside my company's VPN network.
My next step is to configure Jenkins/GitHub so that each git push to GitHub would trigger a Jenkins build job. I have set up the post-commit webhook on GitHub, and enabled the GitHub hook trigger for GITScm polling checkbox in my Jenkins projects.
However, GitHub fails to connect to Jenkins.
Here is the log:
We couldn’t deliver this payload: failed to connect
{
"zen": "Design for failure.",
"hook_id": 287451636,
"hook": {
"type": "Repository",
"id": 287451636,
"name": "web",
"active": true,
"events": [
"push"
],
"config": {
"content_type": "json",
"insecure_ssl": "0",
"url": "http://1.2.3.4/jenkins/github-webhook/"
},
"updated_at": "2021-03-19T07:45:12Z",
"created_at": "2021-03-19T07:45:12Z",
"url": "https://api.github.com/repos/mycompany/my_repo/hooks/287451636",
"test_url": "https://api.github.com/repos/mycompany/my_repo/hooks/287451636/test",
"ping_url": "https://api.github.com/repos/mycompany/my_repo/hooks/287451636/pings",
"last_response": {
"code": null,
"status": "unused",
"message": null
}
},
"repository": {
"id": 347030450,
"node_id": "MDEwOlJlcG9zaXRvcnkzNDcwMzA0NTA=",
"name": "my_repo",
"full_name": "mycompany/my_repo",
"private": true,
"owner": {
"login": "mycompany",
"id": 11980925,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjExOTgwOTI1",
"avatar_url": "https://avatars.githubusercontent.com/u/11980925?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/mycompany",
"html_url": "https://github.com/mycompany",
"followers_url": "https://api.github.com/users/mycompany/followers",
"following_url": "https://api.github.com/users/mycompany/following{/other_user}",
"gists_url": "https://api.github.com/users/mycompany/gists{/gist_id}",
"starred_url": "https://api.github.com/users/mycompany/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/mycompany/subscriptions",
"organizations_url": "https://api.github.com/users/mycompany/orgs",
"repos_url": "https://api.github.com/users/mycompany/repos",
"events_url": "https://api.github.com/users/mycompany/events{/privacy}",
"received_events_url": "https://api.github.com/users/mycompany/received_events",
"type": "Organization",
"site_admin": false
},
"html_url": "https://github.com/mycompany/my_repo",
"description": "My test repo.",
"fork": false,
"url": "https://api.github.com/repos/mycompany/my_repo",
"forks_url": "https://api.github.com/repos/mycompany/my_repo/forks",
"keys_url": "https://api.github.com/repos/mycompany/my_repo/keys{/key_id}",
"collaborators_url": "https://api.github.com/repos/mycompany/my_repo/collaborators{/collaborator}",
"teams_url": "https://api.github.com/repos/mycompany/my_repo/teams",
"hooks_url": "https://api.github.com/repos/mycompany/my_repo/hooks",
"issue_events_url": "https://api.github.com/repos/mycompany/my_repo/issues/events{/number}",
"events_url": "https://api.github.com/repos/mycompany/my_repo/events",
"assignees_url": "https://api.github.com/repos/mycompany/my_repo/assignees{/user}",
"branches_url": "https://api.github.com/repos/mycompany/my_repo/branches{/branch}",
"tags_url": "https://api.github.com/repos/mycompany/my_repo/tags",
"blobs_url": "https://api.github.com/repos/mycompany/my_repo/git/blobs{/sha}",
"git_tags_url": "https://api.github.com/repos/mycompany/my_repo/git/tags{/sha}",
"git_refs_url": "https://api.github.com/repos/mycompany/my_repo/git/refs{/sha}",
"trees_url": "https://api.github.com/repos/mycompany/my_repo/git/trees{/sha}",
"statuses_url": "https://api.github.com/repos/mycompany/my_repo/statuses/{sha}",
"languages_url": "https://api.github.com/repos/mycompany/my_repo/languages",
"stargazers_url": "https://api.github.com/repos/mycompany/my_repo/stargazers",
"contributors_url": "https://api.github.com/repos/mycompany/my_repo/contributors",
"subscribers_url": "https://api.github.com/repos/mycompany/my_repo/subscribers",
"subscription_url": "https://api.github.com/repos/mycompany/my_repo/subscription",
"commits_url": "https://api.github.com/repos/mycompany/my_repo/commits{/sha}",
"git_commits_url": "https://api.github.com/repos/mycompany/my_repo/git/commits{/sha}",
"comments_url": "https://api.github.com/repos/mycompany/my_repo/comments{/number}",
"issue_comment_url": "https://api.github.com/repos/mycompany/my_repo/issues/comments{/number}",
"contents_url": "https://api.github.com/repos/mycompany/my_repo/contents/{+path}",
"compare_url": "https://api.github.com/repos/mycompany/my_repo/compare/{base}...{head}",
"merges_url": "https://api.github.com/repos/mycompany/my_repo/merges",
"archive_url": "https://api.github.com/repos/mycompany/my_repo/{archive_format}{/ref}",
"downloads_url": "https://api.github.com/repos/mycompany/my_repo/downloads",
"issues_url": "https://api.github.com/repos/mycompany/my_repo/issues{/number}",
"pulls_url": "https://api.github.com/repos/mycompany/my_repo/pulls{/number}",
"milestones_url": "https://api.github.com/repos/mycompany/my_repo/milestones{/number}",
"notifications_url": "https://api.github.com/repos/mycompany/my_repo/notifications{?since,all,participating}",
"labels_url": "https://api.github.com/repos/mycompany/my_repo/labels{/name}",
"releases_url": "https://api.github.com/repos/mycompany/my_repo/releases{/id}",
"deployments_url": "https://api.github.com/repos/mycompany/my_repo/deployments",
"created_at": "2021-03-12T10:40:44Z",
"updated_at": "2021-03-18T13:59:52Z",
"pushed_at": "2021-03-18T13:59:49Z",
"git_url": "git://github.com/mycompany/my_repo.git",
"ssh_url": "git#github.com:mycompany/my_repo.git",
"clone_url": "https://github.com/mycompany/my_repo.git",
"svn_url": "https://github.com/mycompany/my_repo",
"homepage": null,
"size": 3,
"stargazers_count": 0,
"watchers_count": 0,
"language": "Makefile",
"has_issues": true,
"has_projects": true,
"has_downloads": true,
"has_wiki": true,
"has_pages": false,
"forks_count": 0,
"mirror_url": null,
"archived": false,
"disabled": false,
"open_issues_count": 0,
"license": null,
"forks": 0,
"open_issues": 0,
"watchers": 0,
"default_branch": "main"
},
"sender": {
"login": "MyLogin",
"id": 63738097,
"node_id": "MDQ6VXNlcjYzNzM4MDk3",
"avatar_url": "https://avatars.githubusercontent.com/u/13818097?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/MyLogin",
"html_url": "https://github.com/MyLogin",
"followers_url": "https://api.github.com/users/MyLogin/followers",
"following_url": "https://api.github.com/users/MyLogin/following{/other_user}",
"gists_url": "https://api.github.com/users/MyLogin/gists{/gist_id}",
"starred_url": "https://api.github.com/users/MyLogin/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/MyLogin/subscriptions",
"organizations_url": "https://api.github.com/users/MyLogin/orgs",
"repos_url": "https://api.github.com/users/MyLogin/repos",
"events_url": "https://api.github.com/users/MyLogin/events{/privacy}",
"received_events_url": "https://api.github.com/users/MyLogin/received_events",
"type": "User",
"site_admin": false
}
}
What could be the problem?
Well the problem is exactly because github cannot access your internal network.
There's an article in Jenkins Community Blog where author suggests using smee (released under ISC License) for webhooks forwarding.
If using smee to deliver webhooks from GitHub to your VPNed Jenkins (you set up Github to send events to smee.io, and than you use the Smee client to retrieve the events and use them internally on Jenknins), here is a quick way to set it up.
Go to Smee.io and get yourself a "channel", write down the URL you get (for example https://smee.io/Wua8D46nBvJ63nPP).
Install Smee on your Jenkins: sudo npm install --global smee-client
Configure the internal forwarding: smee -u https://smee.io/Wua8D46nBvJ63nPP --path /github-webhook/ --port 8080
Go to your GitHub repository: Settings->Webhooks->Add webhook, and paste the URL under "Payload URL", choose Content type "application/json".
Setup your Smee to run as a service on Jenkins, to start at boot time. You will need two files:
Init script /etc/init.d/smee
systemctl service description file /etc/systemd/system/smee.service
/etc/init.d/smee:
#! /bin/sh
### BEGIN INIT INFO
# Provides: smee
# Required-Start: $all
# Required-Stop:
# Default-Start: 2 3 4 5
# Default-Stop:
# Short-Description: Start SMEE daemon at boot time
# Description: Forwards GitHub webhook notifications to Jenkins.
### END INIT INFO
PATH=/bin:/usr/bin:/sbin:/usr/sbin
DAEMON=/usr/bin/smee
PIDFILE=/var/run/smee.pid
test -x $DAEMON || exit 0
. /lib/lsb/init-functions
case "$1" in
start)
log_daemon_msg "Starting smee" "smee"
start_daemon -p $PIDFILE $DAEMON
log_end_msg $?
;;
stop)
log_daemon_msg "Stopping smee" "smee"
killproc -p $PIDFILE $DAEMON
log_end_msg $?
;;
force-reload|restart)
$0 stop
$0 start
;;
status)
status_of_proc -p $PIDFILE $DAEMON atd && exit 0 || exit $?
;;
*)
echo "Usage: /etc/init.d/smee {start|stop|restart|force-reload|status}"
exit 1
;;
esac
exit 0
/etc/systemd/system/smee.service:
[Unit]
Description=smee.io webhook delivery from GitHub
After=network.target
StartLimitIntervalSec=0
[Service]
Type=simple
Restart=always
RestartSec=1
User=jenkins
ExecStart=/usr/bin/smee -u https://smee.io/Wua8D46nBvJ63nPP --path /jenkins/github-webhook/ --port 8080
[Install]
WantedBy=multi-user.target
Start the service: sudo /etc/init.d/smee start
Check the status: sudo /etc/init.d/smee status
The notifications from your repository should now be sent to Jenkins and Jenkins should be able to see and handle them. To quickly check if Jenkins handles any Smee comands, open the "Dashboard->System Log" and search for "smee".
I am having difficulties to write slow logs for elasticsearch into docker image.
Elasticsearch docker settings
"HostConfig": { "Binds": [ "/mnt/mydisk/data:/usr/share/elasticsearch/data", "/mnt/mydisk/logs:/usr/share/elasticsearch/logs" ],
I changed elasticsearch index settings like below;
{
"index.search": {
"slowlog": {
"level": "info",
"threshold": {
"fetch": {
"warn": "2s",
"trace": "200ms",
"debug": "500ms",
"info": "800ms"
},
"query": {
"warn": "10s",
"trace": "500ms",
"debug": "2s",
"info": "5s"
}
}}}}
I can only see gc.logs in "/mnt/mydisk/logs" path, and there is no "/usr/share/elasticsearch/logs" folder or path.
How can I save slow logs into /mnt/mydisk/logs ?
BTW I can see slow logs via "docker logs elasticsearch" command, but I cannot find where its saved or change the path.
You are looking for this properties file: log4j2.properties. If you are using the official elastisearch image, the default setting is to log everything on stdout (i.e. docker logs )
Read more here.
I'm developing an app where we pop containers with volumes and custom network.
I need to add the feature where admin will be able to upgrade the running container to latest version. So I was hoping to be able to fetch the various information from it the pop a new container with the old config.
Question
However I'm not sure what I really need to grab on the old container and how to use it. For instance,
Is NetworkSettings.Networks enough or is there network information elsewhere in the data ?
How do I mount a volume with the infos in Mounts?
{
"Type": "volume",
"Name": "841d6a1709b365763c85fb4b7400c87f264d468eb1691a660fe81761da6e374f",
"Source": "/var/lib/docker/volumes/841d6a1709b365763c85fb4b7400c87f264d468eb1691a660fe81761da6e374f/_data",
"Destination": "/home/mast/.ssh",
"Driver": "local",
"Mode": "",
"RW": True,
"Propagation": ""
}
I still need to check the network part, but using the full path to the previous Source as the new source works to mount the volume.
update: volumes is a list of mount points (destination not the source).
import docker
containers = docker_api.containers()
docker_api.create_container(
image='docker.site.fr:5000/coaxis/coaxisopt_daemon:latest',
volumes=['/home/mast/.ssh', '/etc/mast'],
host_config=docker_api.create_host_config(
binds={
"/var/lib/docker/volumes/841d6a1709b365763c85fb4b7400c87f264d468eb1691a660fe81761da6e374f/_data": {
'bind': "/home/mast/.ssh",
'mode': 'rw'
},
"/var/lib/docker/volumes/002730cbb4dd9b37ad808915a60081508885d533fe003b529b8d0ab4fa46e92e/_data": {
'bind': "/etc/mast",
'mode': 'rw'
}
}
))
Hi didn't really knew if my question was more for serverfault or here, I hope devops won't mind me posting here.
I am working on a stack with mesos/marathon/docker/glusterfs, I feel tired with the lake of documentation.
I am looking for a sample marthon deployement file for deploying using glusterfs driver.
The author says that we should create the volume before, but he doesn't say anything about mounting it.
"container": {
"type": "DOCKER",
"docker": {
"image": "kylemanna/openvpn:latest",
"parameters": [
{
"key": "volume-driver",
"value": "glusterfs"
},
{
"key": "cap-add",
"value": "NET_ADMIN"
}
],
"network": "BRIDGE",
"portMappings": [
{
"containerPort": 1194
}
]
},
"volumes": [
{
"containerPath": "/etc/openvpn",
"hostPath": "openvpn-data",
"mode": "RW"
}
]
}
My container keep restarting in marathon and logs says that /usr/local/bin/ovpn_run: line 16: /etc/openvpn/ovpn_env.sh: No such file or directory
On my gluster fileserver, I have these file present in /data/openvpn-data/ovpn_env.sh
I don't see any mount point in /mnt, I guess marathon did the mount itself, but because the container keep restarting, I dont see it.
I did a docker inspect to check where was stored the filesystem and I found that it is stored in /var/lib/docker-volumes/_glusterfs/openvpn-data
So here are my questions :
Is my marathon json file correct ?
Will the container wait for all data to be downloaded and should I configure something for that ?
Are the data erased when deleting a container on marathon?
Should I have my ovpn_env.sh in /data/myvolume/ovpn_env.sh or /data/myvolume/etc/openvpn/ovpn_env.sh
Have a look at the folowing issue
https://github.com/mesosphere/marathon/issues/2493#issuecomment-196743212
and the docs at
https://github.com/mesosphere/marathon/blob/bd076173b662b12d18e5dd568629a286b242ba91/docs/docs/persistent-volumes.md
Quote:
Docker volumes with plugin drivers is not available right now.
You'll have to create the volume/mount before you start the container, and map the host folder when you launch the app via Marathon (you do this already). I guess that's why it's currently called "persistent local volumes"...
Define it in "parameters" part, like this:
"parameters": [
{
"key": "volume-driver",
"value": "glusterfs"
},
{
"key": "volume",
"value": "openvpn-data:/etc/openvpn"
}
]
I can't find how to manage images in a private registry. I can push or pull an image because i know the id but how to get the list of pushed images ?
Take for example a person who wants to see the available images under the private registry of his organization. How can she do ?
Unless I'm mistaken, I can't find API or Web UI to discover the registry content like the index.docker.io do with the public registry.
Are there any open source projects to manage this ?
thanks.
Are there any open source projects to manage this ?
There is a containerized web application that provides administration of one-to-many private registries. Its name is Docker Registry UI and it is FOSS.
The source is on Github and you can run it in a container like so:
docker run -p 8080:8080 -v my_data_dir:/var/lib/h2/ atcol/docker-registry-ui
Disclaimer: I wrote the web-app as I could not find one myself. I believe this answers your question (as quoted).
Thanks Thomas !
To allow the use of the search API, you must start the container by specifying the value of the environment variable SEARCH_BACKEND like this :
docker run -d -e SEARCH_BACKEND=sqlalchemy -p 5000:5000 --name registry samalba/docker-registry
Then i have a result for this query :
GET http://registry_host:5000/v1/search?q=base
Result :
{
"num_results": 1,
"query": "base",
"results": [{"description": "", "name": "test/base-img"}]
}
To list all images, you can do this :
GET http://registry_host:5000/v1/search
Result :
{
"num_results": 2,
"query": "",
"results": [
{"description": "", "name": "test/base-img"},
{"description": "", "name": "test/base-test"}]
}
And to know the available versions of an image :
GET http://localhost:5000/v1/repositories/**test/base-img**/tags
Result :
{
"0.1": "04e073e1efd31f50011dcde9b9f4d3148ecc4da94c0b7ba9abfadef5a8522d13",
"0.2": "04e073e1efd31f50011dcde9b9f4d3148ecc4da94c0b7ba9abfadef5a8522d13",
"0.3": "04e073e1efd31f50011dcde9b9f4d3148ecc4da94c0b7ba9abfadef5a8522d13"
}
I've written a docker-registry-frontend that you can find on github. It allows you to browse your private registry and do almost everything that is available through the Docker registry API v1. Plus, it can be run as a docker container on its own.
Here's a list of basic features with some screenshots: https://github.com/kwk/docker-registry-frontend/wiki/Features. In addition to these features, there's support for SSL encryption and Kerberos authentication.
I want to present for you, my frontend for private registry, you may try it from github or dockerhub.
Also you can find interface screenshots there.
To sum up it has:
- internal db (BoltBD) gives it ability to store info, and as result it responses much more faster then after direct api call like in other projects
- app can pars, store and show info from registry such as:
- image layers info:
- name / tag
- image size and pushes number
- upload and push dates
- image creating commands history
- it is possible to set multiple repositories in case you have more than one registries and observe them in one place
- show statistics pretty, draw curves for uploads number and image sizes for tag with respects to dates
Update 2017-02-15
So far also there was added:
find a parent
show tree-graph of parents
image deletion
Bearer token auth support
As far as I see, the Docker registry has a REST API, very similar to Docker itself. You can find the documentation at http://docs.docker.io/reference/api/registry_api/. But on the first glance I don't see a method to just list all images.
There is also an REST API for the official index (infos at http://docs.docker.io/reference/api/docker-io_api/).
EDIT
I just tested the Docker registry API and it is not so self-explanatory. You can query all images of a certain repository. In my case, my repository is called "thomas/busybox". I can query all images in there by calling:
https://my-private-registry.com/v1/repositories/thomas/busybox/images
Result:
[
{
"id": "2d8e5b282c81244037eb15b2068e1c46319c1a42b80493acb128da24b2090739"
},
{
"id": "6c991eb934609424f761d3d0a7c79f4f72b76db286aa02e617659ac116aa7758"
},
{
"id": "9f4e93171ec525221fa9013d0e21f8690cef68590664eb5249e0b324c5faf31a"
},
{
"id": "511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158"
}
]
Now I know that I have four images in my repository and I can query every image. The query for the first image would be:
https://my-private-registry.com/v1/images/2d8e5b282c81244037eb15b2068e1c46319c1a42b80493acb128da24b2090739/json
Result:
{
"id": "2d8e5b282c81244037eb15b2068e1c46319c1a42b80493acb128da24b2090739",
"parent": "9f4e93171ec525221fa9013d0e21f8690cef68590664eb5249e0b324c5faf31a",
"created": "2014-04-24T15:59:59.47081913Z",
"container": "d15320d6935ca35bc4198e373f29e730f4c53cce32b3809c2fecec22eb30018b",
"container_config": {
"Hostname": "4964db5b599b",
...
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"HOME=\/",
"PATH=\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin"
],
"Cmd": [
"\/bin\/sh",
"-c",
"#(nop) CMD [\/bin\/sh -c \/bin\/sh]"
],
"Image": "9f4e93171ec525221fa9013d0e21f8690cef68590664eb5249e0b324c5faf31a",
...
"OnBuild": [
]
},
"docker_version": "0.10.0",
"author": "J\u00c3\u00a9r\u00c3\u00b4me Petazzoni <jerome#docker.com>",
"config": {
"Hostname": "4964db5b599b",
"Domainname": "",
"User": "",
"Memory": 0,
...
"Env": [
"HOME=\/",
"PATH=\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin"
],
"Cmd": [
"\/bin\/sh",
"-c",
"\/bin\/sh"
],
"Image": "9f4e93171ec525221fa9013d0e21f8690cef68590664eb5249e0b324c5faf31a",
...
"OnBuild": [
]
},
"architecture": "amd64",
"os": "linux",
"Size": 0
}
You can also search for an image, but I do not get any results:
https://my-private-registry.com/v1/search?q=thomas
Result:
{"num_results": 0, "query": "thomas", "results": []}
Sonatype Nexus Repository Manager 3.0 has Private Registry for Docker