Alongside the Docker containers running my micro-services written in go (one container per micro-service), I need to create a Docker container that runs the Service Registry service provided by a third-party framework (Koding's kontrol).
Having said that, I need to create a separate container that runs this third-party go application by executing the following commands.
Retrieve the application:
go get github.com/koding/kite/kontrol/kontrol
Generate RSA key pair:
openssl genrsa -out key.pem 2048
openssl rsa -in key.pem -pubout > key_pub.pem
Set some environment variables:
KONTROL_PORT=6000
KONTROL_USERNAME="kontrol"
KONTROL_STORAGE="etcd"
KONTROL_KONTROLURL="http://127.0.0.1:6000/kite"
KONTROL_PUBLICKEYFILE="certs/key_pub.pem"
KONTROL_PRIVATEKEYFILE="certs/key.pem"
Initialize the Registry Service:
./bin/kontrol -initial
Start the Registry Service:
./bin/kontrol
How do I include these steps in my project? For instance, I have a docker-composer.yaml file like this:
version: '3.3'
services:
api:
container_name: 'api'
build: './api'
ports:
- '8080:8080'
volumes:
- './api:/go/src/app'
depends_on:
- 'mongo'
etcd0:
container_name: 'etcd0'
image: 'quay.io/coreos/etcd'
command: >
etcd -name etcd0
-advertise-client-urls http://127.0.0.1:2379,http://127.0.0.1:4001
-listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001
-initial-advertise-peer-urls http://127.0.0.1:2380
-listen-peer-urls http://0.0.0.0:2380
-initial-cluster-token etcd-cluster-1
-initial-cluster etcd0=http://127.0.0.1:2380
-initial-cluster-state new
mongo:
container_name: 'mongo'
image: 'mongo:latest'
ports:
- '27017:27017'
volumes:
- '/var/lib/mongodb:/var/lib/mongodb'
Related
I am building an application using two docker containers in the same network:
mcr.microsoft.com/azure-storage/azurite
jupyter/pyspark-notebook
Here is my docker-compose file:
version: "3.9"
services:
azurite:
image: mcr.microsoft.com/azure-storage/azurite:latest
ports:
- "10000:10000"
- "10001:10001"
- "10002:10002"
volumes:
- azurite_volume:/data
pyspark:
image: jupyter/pyspark-notebook:latest
ports:
- 10003:8888
user: root
working_dir: /home/${NB_USER}
environment:
- NB_USER=${NB_USER}
- CHOWN_HOME=yes
- GRANT_SUDO=yes
command: start-notebook.sh --NotebookApp.password="" --NotebookApp.token=""
volumes:
- /my/local/folder:/home/${NB_USER}/work
volumes:
azurite_volume:
driver: local
from the jupyter notebook I am trying to connect to and read data from azurite. Here is my code:
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName('test') \
.config(
'fs.azure.account.key.devstoreaccount1.blob.core.windows.net',
'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==') \
.getOrCreate()
df = spark.read.json('wasb://my-container#devstoreaccount1/path/to/file.json')
However, this code returns an error:
org.apache.hadoop.fs.azure.AzureException: org.apache.hadoop.fs.azure.AzureException: Unable to access container bronze in account devstoreaccount1 using anonymous credentials, and no credentials found for them in the configuration.
The container in azurite has already been set to "public" although it wouldn`t be necessary because I am providing the credential in the spark config. Even though, the error tells me that I am using anonymous credentials...
I am probably setting the credentials wrongly but I couldn't find anywhere how to set them properly.
How can I set up the credentials to be able to read from azurite using pyspark?
My docker-compose.yml:
version: "3"
services:
[...]
portainer:
image: portainer/portainer-ce
ports:
- "10280:9000"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
- "./portainer:/data"
restart: unless-stopped
command: --admin-password $$2b$$05$$XJA5Fr6FGLsptH8mb2/L2uwH2mXGDJkfbTUkpuFEnSkpWY9D2EKCO
[...]
(the "[...]" just is for other services which aren't related to the problem)
I configured the admin password with command: --admin-password [bcryptHash] but how do I configure it to use the local / "volumed" docker instance / socket from docker-compose and not from the web interface?
Try using this command
command: -H unix:///var/run/docker.sock
I found a reference to this call for the -H flag here: https://docs.portainer.io/v/ce-2.6/advanced/reverse-proxy/traefik
This contains a full docker-compose file example that sets up a reverse proxy for portainer using traefik. The relevant section is:
version: "3.3"
services:
portainer:
image: portainer/portainer-ce:2.6.3
command: -H unix:///var/run/docker.sock
restart: always
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- portainer_data:/data
volumes:
portainer_data:
From the official docker documentation site, there is a link to the awesome-compose repo that also has a docker-compose file example for portainer.
So from this document, it would appear that both the volume map for the socket and the command line flag are required.
I'm building 2 docker containers, "app" and "db", via a docker-compose file.
The app server just installs java/tomcat via a Dockerfile which is what docker-compose uses to build.
The db server uses an MS SQL image.
When I run:
docker-compose up
I follow that with a build process of software I need to load which deploys a war to the tomcat directory in the app server and builds the database in the database server.
My problem is: The build process can reference localhost:8080 to install/patch the software to the app server and reference localhost:1433 to install/patch the database portion of the software to the database server. However, when I start Tomcat the system doesn't come online because the app server can't connect to the database server via "localhost:1433" so it requires me to jump in and update the properties file after the build to the docker internal IP address and THEN it works.
My question is: How am I able to get my localhost and my app container to reference the DB in the same manner in a database url?
Dockerfile for app server:
FROM centos:centos7
COPY apache-tomcat-9.0.20.tar.gz /tmp/
WORKDIR /tmp/
RUN yum -y update
RUN yum -y install java-11-openjdk-devel
RUN tar -xf apache-tomcat-9.0.20.tar.gz
RUN mv apache-tomcat-9.0.20 /opt/tomcat/
RUN export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/
RUN export PATH=$PATH:$JAVA_HOME/jre/bin
RUN export CATALINA_HOME=/opt/tomcat/
RUN export PATH=$PATH:$CATALINA_HOME/bin
WORKDIR /opt/tomcat/webapps
RUN mkdir testapp
enter code here
enter code here
Docker-Compose File:
version: '3.3'
services:
db:
image: "mcr.microsoft.com/mssql/server:2017-latest"
restart: always
volumes:
- db_data:/var/lib/mssql
environment:
- ACCEPT_EULA=Y
- SA_PASSWORD=Test123
network_mode: bridge
hostname: db
ports:
- "1433:1433"
app:
build: './testapp'
volumes:
- './system/build:/opt/tomcat/webapps/testapp/'
ports:
- "8080:8080"
- "8009:8009"
network_mode: bridge
tty: true
depends_on:
- db
volumes:
db_data:
Bring your service to the same network and target the service by service name. For that you need to define a docker network like below. For the following example I can access DB with http://mongo:27017.
mongo:
image: mongo:latest
ports:
- "27017:27017"
volumes:
- ./data/db:/data/db
networks:
- my-net
spring:
depends_on:
- mongo
image: docker-spring-http-alpine
ports:
- "8080:8080"
networks:
- my-net
networks:
my-net:
I've been using localstack to develop a service against locally. I've just been running their docker image via docker run --rm -p 4567-4583:4567-4583 -p 8080:8080 localstack/localstack
And then I manually run a small script to set up my S3 buckets, SQS queues, etc.
Now, I'd like to make this easier for others so I thought I'd just add a Dockerfile and docker-compose.yml file. Unfortunately, when I try to get this up and running, using docker-compose up I get an error that the command from my setup script can't connect to the localstack services.
make_bucket failed: s3://localbucket Could not connect to the endpoint URL: "http://localhost:4572/localbucket"
Dockerfile:
FROM localstack/localstack
#since this is just local dev set up, localstack doesn't require
anything specific here.
ENV AWS_DEFAULT_REGION='[useast1]'
ENV AWS_ACCESS_KEY_ID='[lloyd]'
ENV AWS_SECRET_ACCESS_KEY='[christmas]'
COPY bin/localSetup.sh /localSetup.sh
COPY fixtures/notifications.json /notifications.json
RUN ["chmod", "+x", "/localSetup.sh"]
RUN pip install awscli
# expose service & web dashboard ports
EXPOSE 4567-4582 8080
ENTRYPOINT ["/localSetup.sh"]
docker-compose.yml
version: '3'
services:
localstack:
build: .
ports:
- "8080:8080"
- "4567-4582:4567-4582"
localSetup.sh
#!/bin/bash
aws --endpoint-url=http://localhost:4572 s3 mb s3://localbucket
#additional similar calls but left off for brevity
I've tried switching localhost to 127.0.0.1 in my script commands, but I wind up with the same error. I'm probably missing something silly here.
There is another way to create your custom AWS resources when localstack freshly starts up. Since you already have a bash script for your resources, you can simply volume mount your script to /docker-entrypoint-initaws.d/.
So my docker-compose file would be:
localstack:
image: localstack/localstack:latest
container_name: localstack_aws
ports:
- '4566:4566'
volumes:
- './localSetup.sh:/etc/localstack/init/ready.d/init-aws.sh'
Also, I would prefer awslocal over aws --endpoint in the bash script, as it leverages the credentials work and endpoint for you.
try adding hostname to the docker-compose file and editing your entrypoint file to reflect that hostname.
docker-compose.yml
version: '3'
services:
localstack:
build: .
hostname: localstack
ports:
- "8080:8080"
- "4567-4582:4567-4582"
localSetup.sh
#!/bin/bash
aws --endpoint-url=http://localstack:4572 s3 mb s3://localbucket
This was my docker-compose-dev.yaml I used for testing out an app that was using localstack. I used the command docker-compose -f docker-compose-dev.yaml up, I also used the same localSetup.sh you used.
version: '3'
services:
localstack:
image: localstack/localstack
hostname: localstack
ports:
- "4567-4584:4567-4584"
- "${PORT_WEB_UI-8082}:${PORT_WEB_UI-8082}"
environment:
- SERVICES=s3
- DEBUG=1
- DATA_DIR=${DATA_DIR- }
- PORT_WEB_UI=${PORT_WEB_UI- }
- DOCKER_HOST=unix:///var/run/docker.sock
volumes:
- "${TMPDIR:-/tmp/localstack}:/tmp/localstack"
- "/var/run/docker.sock:/var/run/docker.sock"
networks:
- backend
sample-app:
image: "sample-app/sample-app:latest"
networks:
- backend
links:
- localstack
depends_on:
- "localstack"
networks:
backend:
driver: 'bridge'
I'm trying to deploy an instance of Portainer to a docker swarm. I'm not sure how to set the correct flag to enable SSL.
From the docs:
$ docker run -d -p 443:9000 --name portainer --restart always -v ~/local-certs:/certs -v portainer_data:/data portainer/portainer --ssl --sslcert /certs/portainer.crt --sslkey /certs/portainer.key
https://portainer.readthedocs.io/en/stable/deployment.html
But how do you translate that into a docker compose yml file?
Possibly I'm a bit late to the party, but it looks what you have to use Portainer's flags to enable ssl for your Portainer (as said in documentation) and composerize.com lost that part somewhere, so you should add this to your compose:
command:
--sslcert /certs/portainer.crt
--sslkey /certs/portainer.key
or for full compose file:
version: 3
services:
portainer:
image: portainer/portainer
container_name: portainer
restart: always
ports:
- '443:9000'
volumes:
- '~/local-certs:/certs'
- 'portainer_data:/data'
command:
--sslcert /certs/portainer.crt
--sslkey /certs/portainer.key
According to Portainer documentation:
By default, Portainer’s web interface and API is exposed over HTTP.
This is not secured, it’s recommended to enable SSL in a production
environment.
To do so, you can use the following flags --ssl, --sslcert and
--sslkey:
$ docker run -d -p 443:9000 --name portainer --restart always -v
~/local-certs:/certs -v portainer_data:/data portainer/portainer --ssl
--sslcert /certs/portainer.crt --sslkey /certs/portainer.key
You can use the following commands to generate the required files:
$ openssl genrsa -out portainer.key 2048
$ openssl ecparam -genkey -name secp384r1 -out portainer.key
$ openssl req -new -x509 -sha256 -key portainer.key -out portainer.crt -days 3650
Note that Certbot could be used as well to generate a certificate and a key.
As Rubin suggests, you can use https://composerize.com/ to generate a docker-compose.yml from docker command.
So, your docker-compose file should be something like this:
version: '3'
services:
portainer:
image: portainer/portainer
container_name: portainer
restart: always
ports:
- '443:9000'
volumes:
- '~/local-certs:/certs'
- 'portainer_data:/data'
command:
--ssl
--sslcert /certs/portainer.crt
--sslkey /certs/portainer.key
volumes:
portainer_data:
https://composerize.com/ can help to translate your docker command into a docker-compose.yml
The following works for me:
version: '3'
services:
portainer:
image: portainer/portainer-ce
volumes:
- "/local-certs:/certs"
- "portainer_data:/data"
restart: always
ports:
- "9000:9000"
container_name: portainer
command:
- --ssl
- --sslcert
- /certs/wildcard.crt
- --sslkey
- /certs/wildcard.key