Using private docker images in AWS Beanstalk - docker

I have tried following the documentation on this link :
<https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker.container.console.html#docker-images-private>
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker.container.console.html#docker-alami
My .dockercfg file looks something like this:
{
"https://index.docker.io/v1/" :
{
"auth" : "username:pwd [base 64 enc]",
"email" : "email_id"
}
}
I'm trying to pull a private image from Docker hub and my Dockerrun.aws.json looks something like this :
{
"AWSEBDockerrunVersion": "1",
"Authentication": {
"Bucket": "my-bucket",
"Key": ".dockercfg"
},
"Image": {
"Name": "dishvy/imgname:tag",
"Update": "true"
},
"Ports": [
{
"ContainerPort": "80"
}
],
"Volumes": [
{
"HostDirectory": "/var/app/mydb",
"ContainerDirectory": "/etc/mysql"
}
],
"Logging": "/var/log/nginx"
}
I have added the dockercfg file at the root of the bucket. And when i'm trying to deploy this to AWS Beanstalk, I get this error :
Error response from daemon: pull access denied for dishvy/imgname:tag, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
This essentially means my authentication file on S3 isn't correct but i followed the steps given in documentation and can't figure out where i went wrong. Can anybody help me out here?

Related

Elasticsearch: how to make snapshot to minio s3?

when i am trying to make snapshots to my s3 (minio deployed locally), by executing the following command inside my elasticsearch docker container:
curl -X PUT "localhost:9200/_snapshot/s3_repository/snapshot_1?wait_for_completion=true&pretty"
i got the following :
{
"error" : {
"root_cause" : [
{
"type" : "repository_exception",
"reason" : "[s3_repository] Could not determine repository generation from root blobs"
}
],
"type" : "repository_exception",
"reason" : "[s3_repository] Could not determine repository generation from root blobs",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "Exception when listing blobs by prefix [index-]",
"caused_by" : {
"type" : "sdk_client_exception",
"reason" : "Unable to execute HTTP request: my-first-bucket.s3",
"caused_by" : {
"type" : "unknown_host_exception",
"reason" : "my-first-bucket.s3"
}
}
}
},
"status" : 500
}
Any help please ?
Notice, i created s3_repository using the followig :
curl -XPUT 'http://localhost:9200/_snapshot/s3_repository?verify=false&pretty' -H 'Content-Type: application/json' -d'
{
"type": "s3",
"settings": {
"bucket": "my-first-bucket",
"endpoint": "http://s3:9000"
}
}'
Notice s3 in http://s3:9000 is the name of the service minio in my docker-compose.yml
Thank you in advance!
Most likely the problem here is that the bucket is not created at MinIO startup. The environment variable MINIO_DEFAULT_BUCKETS is not officially supported by MinIO at least. After MinIO starts, you can try to confirm if the bucket exists as expected.
As the bucket only needs to be created once, you could do it via a script after MinIO starts.
Please know you can find us at https://slack.min.io/ 24/7/365. If you have commercial questions, please reach out to us on hello#min.io or on our Ask an Expert Chat functionality at https://min.io/pricing?action=talk-to-us.

Elasticsearch docker slow log file location settings

I am having difficulties to write slow logs for elasticsearch into docker image.
Elasticsearch docker settings
"HostConfig": { "Binds": [ "/mnt/mydisk/data:/usr/share/elasticsearch/data", "/mnt/mydisk/logs:/usr/share/elasticsearch/logs" ],
I changed elasticsearch index settings like below;
{
"index.search": {
"slowlog": {
"level": "info",
"threshold": {
"fetch": {
"warn": "2s",
"trace": "200ms",
"debug": "500ms",
"info": "800ms"
},
"query": {
"warn": "10s",
"trace": "500ms",
"debug": "2s",
"info": "5s"
}
}}}}
I can only see gc.logs in "/mnt/mydisk/logs" path, and there is no "/usr/share/elasticsearch/logs" folder or path.
How can I save slow logs into /mnt/mydisk/logs ?
BTW I can see slow logs via "docker logs elasticsearch" command, but I cannot find where its saved or change the path.
You are looking for this properties file: log4j2.properties. If you are using the official elastisearch image, the default setting is to log everything on stdout (i.e. docker logs )
Read more here.

Migrating Elasticsearch logs to a different cluster

I have an Elasticsearch deployment on Kubernetes (AKS). I'm using the official Elastic's docker images for deployments. Logs are being stored in a persistent Azure Disk. How can I migrate some of these logs to another cluster with a similar setup? Only those logs that matches a filter condition based on datetime of the logs needs to be migrated.
Please use Reindex API for achieving the same
POST _reindex
{
"source": {
"remote": {
"host": "http://oldhost:9200",
"username": "user",
"password": "pass"
},
"index": "source",
"query": {
"match": {
"test": "data"
}
}
},
"dest": {
"index": "dest"
}
}
Note:
Run the aforementioned command on your target instance.
Make sure that the source instance is whitelisted in elasticsearch.yml
reindex.remote.whitelist: oldhost:9200
Run the process asynchronously using below query param
POST _reindex?wait_for_completion=false

example marathon json deployment file when using glusterfs volume driver

Hi didn't really knew if my question was more for serverfault or here, I hope devops won't mind me posting here.
I am working on a stack with mesos/marathon/docker/glusterfs, I feel tired with the lake of documentation.
I am looking for a sample marthon deployement file for deploying using glusterfs driver.
The author says that we should create the volume before, but he doesn't say anything about mounting it.
"container": {
"type": "DOCKER",
"docker": {
"image": "kylemanna/openvpn:latest",
"parameters": [
{
"key": "volume-driver",
"value": "glusterfs"
},
{
"key": "cap-add",
"value": "NET_ADMIN"
}
],
"network": "BRIDGE",
"portMappings": [
{
"containerPort": 1194
}
]
},
"volumes": [
{
"containerPath": "/etc/openvpn",
"hostPath": "openvpn-data",
"mode": "RW"
}
]
}
My container keep restarting in marathon and logs says that /usr/local/bin/ovpn_run: line 16: /etc/openvpn/ovpn_env.sh: No such file or directory
On my gluster fileserver, I have these file present in /data/openvpn-data/ovpn_env.sh
I don't see any mount point in /mnt, I guess marathon did the mount itself, but because the container keep restarting, I dont see it.
I did a docker inspect to check where was stored the filesystem and I found that it is stored in /var/lib/docker-volumes/_glusterfs/openvpn-data
So here are my questions :
Is my marathon json file correct ?
Will the container wait for all data to be downloaded and should I configure something for that ?
Are the data erased when deleting a container on marathon?
Should I have my ovpn_env.sh in /data/myvolume/ovpn_env.sh or /data/myvolume/etc/openvpn/ovpn_env.sh
Have a look at the folowing issue
https://github.com/mesosphere/marathon/issues/2493#issuecomment-196743212
and the docs at
https://github.com/mesosphere/marathon/blob/bd076173b662b12d18e5dd568629a286b242ba91/docs/docs/persistent-volumes.md
Quote:
Docker volumes with plugin drivers is not available right now.
You'll have to create the volume/mount before you start the container, and map the host folder when you launch the app via Marathon (you do this already). I guess that's why it's currently called "persistent local volumes"...
Define it in "parameters" part, like this:
"parameters": [
{
"key": "volume-driver",
"value": "glusterfs"
},
{
"key": "volume",
"value": "openvpn-data:/etc/openvpn"
}
]

Any API or Web UI project to manage a Docker private registry?

I can't find how to manage images in a private registry. I can push or pull an image because i know the id but how to get the list of pushed images ?
Take for example a person who wants to see the available images under the private registry of his organization. How can she do ?
Unless I'm mistaken, I can't find API or Web UI to discover the registry content like the index.docker.io do with the public registry.
Are there any open source projects to manage this ?
thanks.
Are there any open source projects to manage this ?
There is a containerized web application that provides administration of one-to-many private registries. Its name is Docker Registry UI and it is FOSS.
The source is on Github and you can run it in a container like so:
docker run -p 8080:8080 -v my_data_dir:/var/lib/h2/ atcol/docker-registry-ui
Disclaimer: I wrote the web-app as I could not find one myself. I believe this answers your question (as quoted).
Thanks Thomas !
To allow the use of the search API, you must start the container by specifying the value of the environment variable SEARCH_BACKEND like this :
docker run -d -e SEARCH_BACKEND=sqlalchemy -p 5000:5000 --name registry samalba/docker-registry
Then i have a result for this query :
GET http://registry_host:5000/v1/search?q=base
Result :
{
"num_results": 1,
"query": "base",
"results": [{"description": "", "name": "test/base-img"}]
}
To list all images, you can do this :
GET http://registry_host:5000/v1/search
Result :
{
"num_results": 2,
"query": "",
"results": [
{"description": "", "name": "test/base-img"},
{"description": "", "name": "test/base-test"}]
}
And to know the available versions of an image :
GET http://localhost:5000/v1/repositories/**test/base-img**/tags
Result :
{
"0.1": "04e073e1efd31f50011dcde9b9f4d3148ecc4da94c0b7ba9abfadef5a8522d13",
"0.2": "04e073e1efd31f50011dcde9b9f4d3148ecc4da94c0b7ba9abfadef5a8522d13",
"0.3": "04e073e1efd31f50011dcde9b9f4d3148ecc4da94c0b7ba9abfadef5a8522d13"
}
I've written a docker-registry-frontend that you can find on github. It allows you to browse your private registry and do almost everything that is available through the Docker registry API v1. Plus, it can be run as a docker container on its own.
Here's a list of basic features with some screenshots: https://github.com/kwk/docker-registry-frontend/wiki/Features. In addition to these features, there's support for SSL encryption and Kerberos authentication.
I want to present for you, my frontend for private registry, you may try it from github or dockerhub.
Also you can find interface screenshots there.
To sum up it has:
- internal db (BoltBD) gives it ability to store info, and as result it responses much more faster then after direct api call like in other projects
- app can pars, store and show info from registry such as:
- image layers info:
- name / tag
- image size and pushes number
- upload and push dates
- image creating commands history
- it is possible to set multiple repositories in case you have more than one registries and observe them in one place
- show statistics pretty, draw curves for uploads number and image sizes for tag with respects to dates
Update 2017-02-15
So far also there was added:
find a parent
show tree-graph of parents
image deletion
Bearer token auth support
As far as I see, the Docker registry has a REST API, very similar to Docker itself. You can find the documentation at http://docs.docker.io/reference/api/registry_api/. But on the first glance I don't see a method to just list all images.
There is also an REST API for the official index (infos at http://docs.docker.io/reference/api/docker-io_api/).
EDIT
I just tested the Docker registry API and it is not so self-explanatory. You can query all images of a certain repository. In my case, my repository is called "thomas/busybox". I can query all images in there by calling:
https://my-private-registry.com/v1/repositories/thomas/busybox/images
Result:
[
{
"id": "2d8e5b282c81244037eb15b2068e1c46319c1a42b80493acb128da24b2090739"
},
{
"id": "6c991eb934609424f761d3d0a7c79f4f72b76db286aa02e617659ac116aa7758"
},
{
"id": "9f4e93171ec525221fa9013d0e21f8690cef68590664eb5249e0b324c5faf31a"
},
{
"id": "511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158"
}
]
Now I know that I have four images in my repository and I can query every image. The query for the first image would be:
https://my-private-registry.com/v1/images/2d8e5b282c81244037eb15b2068e1c46319c1a42b80493acb128da24b2090739/json
Result:
{
"id": "2d8e5b282c81244037eb15b2068e1c46319c1a42b80493acb128da24b2090739",
"parent": "9f4e93171ec525221fa9013d0e21f8690cef68590664eb5249e0b324c5faf31a",
"created": "2014-04-24T15:59:59.47081913Z",
"container": "d15320d6935ca35bc4198e373f29e730f4c53cce32b3809c2fecec22eb30018b",
"container_config": {
"Hostname": "4964db5b599b",
...
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"HOME=\/",
"PATH=\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin"
],
"Cmd": [
"\/bin\/sh",
"-c",
"#(nop) CMD [\/bin\/sh -c \/bin\/sh]"
],
"Image": "9f4e93171ec525221fa9013d0e21f8690cef68590664eb5249e0b324c5faf31a",
...
"OnBuild": [
]
},
"docker_version": "0.10.0",
"author": "J\u00c3\u00a9r\u00c3\u00b4me Petazzoni <jerome#docker.com>",
"config": {
"Hostname": "4964db5b599b",
"Domainname": "",
"User": "",
"Memory": 0,
...
"Env": [
"HOME=\/",
"PATH=\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin"
],
"Cmd": [
"\/bin\/sh",
"-c",
"\/bin\/sh"
],
"Image": "9f4e93171ec525221fa9013d0e21f8690cef68590664eb5249e0b324c5faf31a",
...
"OnBuild": [
]
},
"architecture": "amd64",
"os": "linux",
"Size": 0
}
You can also search for an image, but I do not get any results:
https://my-private-registry.com/v1/search?q=thomas
Result:
{"num_results": 0, "query": "thomas", "results": []}
Sonatype Nexus Repository Manager 3.0 has Private Registry for Docker

Resources