I have a Docker Swarm cluster setup as follows:
Setup on node 1
docker swarm init --advertise-addr ${NODE_1_IP} --data-path-port=7789
Setup on node 2
docker swarm join --advertise-addr ${NODE_2_IP} --token XXX ${NODE_1_IP}:2377
I have then installed weave on both nodes as follows.
sudo curl -L git.io/weave -o /usr/local/bin/weave
sudo chmod a+x /usr/local/bin/weave
docker plugin install weaveworks/net-plugin:latest_release
docker plugin disable weaveworks/net-plugin:latest_release
docker plugin set weaveworks/net-plugin:latest_release WEAVE_PASSWORD=XXX
docker plugin enable weaveworks/net-plugin:latest_release
I wanted to set a password because I need the network to be encrypted.
I then set up a network and a service. The constraint makes the service consist of one container running on node 2.
docker network create --driver=weaveworks/net-plugin:latest_release --attachable testnet_weave_encrypted
docker service create --network=testnet_weave_encrypted --name web_encrypted --publish 80 --replicas=1 --constraint 'node.labels.datastore001 == true' nginx:latest
Finally I test it inside another container running on node 1:
docker run --rm --name alpine --net=testnet_weave_encrypted -ti alpine:latest sh
apk add --no-cache curl
curl web_encrypted
This fails with the message:
curl: (7) Failed to connect to web_encrypted port 80: Host is unreachable
I know that web_encrypted is not wrong because when I try a different value I get a different error.
After bashing my head against this wall for hours I have discovered that I can do the following on node 1:
curl web_encrypted.1.lsrdyz8n66jdotaqgdzk9u1uo
And it works!
But of course this is useless to me because the exact container name will change every time the service recreates it.
Is this a bug in the weave plugin or have I missed a step in setting this up?
If you specify endpoint_mode as DNSRR, you should be able to use the service name as well as any aliases you specify. The weavenet network in the snippet below was created using the Weave network plugin.
host#docker service create --network name=weavenet,alias=ngx --endpoint-mode=dnsrr --name nginx nginx
host#docker run -it -d --name alpine --network=weavenet alpine:curl
host#docker exec -it alpine sh
/ # curl nginx
<!DOCTYPE html>
<html>
....
/ # curl ngx
<!DOCTYPE html>
<html>
I would like to know who tried to install and create a Kubernetes cluster inside LXC proxmox.
Which steps should I follow to realize that?
You can use below articles to get the desired result:
Run kubernetes inside LXC container or Run Kubernetes as Proxmox container
To summarize above articles you should perform next steps:
1) Add overlay driver for docker.
echo overlay >> /etc/modules
2) Add more privileges to container by modifying container config
lxc.apparmor.profile: unconfined
lxc.cap.drop:
lxc.cgroup.devices.allow: a
lxc.mount.auto: proc:rw sys:rw
3) Use shared filesystem to /etc/rc.local
echo '#!/bin/sh -e
mount --make-rshared /' > /etc/rc.local
4) Init cluster using kubeadm
Many questions exists about how to run containers in detached mode.
My question though is kinda specific to running Atlassian Bitbucket server in detached mode containers.
I tried the below as the last layer in my dockerfile and when i run the container with -d the process is not started
RUN /opt/atlassian-bitbucket/bin/start-bitbucket.sh
I tried using ENTRYPOINT like below
ENTRYPOINT ["/opt/atlassian-bitbucket/bin/start-bitbucket.sh"]
but container always exits after the start script completes.
Not sure if anyone has setup a Bitbucket Data center in containers but i am curious to see how they would have run multiple containers of the same image and made them join a single cluster.
Full disclosure: I work for Atlassian Premier Support, work closely with our Bitbucket Server team, and have been the primary maintainer of the atlassian/bitbucket-server Docker image for the past couple of years.
Short version
First: Use our official image, there are a host of problems we've solved over the years so rather than trying to start from scratch use ours as a base.
Second: You can indeed run a Data Center cluster in Docker. My personal test environment consists of 3 cluster nodes and a couple of Smart Mirrors, all using the official image, with HAProxy in front acting as a load balancer and an external Elasticsearch instance managing search. Check out the README above for a list of common configuration options - the ones you'll likely need can be set by passing environment variables
Long Version
AKA "How can I spin up a full DC cluster in a test environment?"
Here's a simple tutorial I put together for our own internal support teams a long time ago. It uses a custom HAProxy Docker container to give you an out-of-the-box load balancer. It's intended for testing on a single host, so if you want to do something different or closer to a production deployment, this won't cover that.
There's a lot to cover here, so let's start with the basics.
Networking
There are a few ways to connect up individual Docker containers so they can find each other and communicate (e.g. the --link parameter), but a Docker Network is by far the most flexible. With a dedicated network, we get the following:
Inter-container communication: Containers on the same network can communicate with each other and access services from other containers without the need to publish specific ports to the host.
Automatic DNS: Containers can find each other via their container name (defined by the --name parameter). Unlike real DNS however, when a container is down its DNS resolution ceases to exist. This can cause some issues for services like HAProxy - but we'll get to that later. Also worth noting is that this does not set the machine's hostname, which needs to be set separately if required.
Static IP assignment: For certain use cases it's useful to give Docker containers static IP addresses within their network
Multicast: Docker networks support multicast by default, which is perfect for Data Center nodes communicating over Hazelcast
One thing a Docker network doesn't do is attach the host to its network, so you, the user, can't connect to containers by container name, and you still need to publish ports to the local machine. However, there are situations where doing this is both useful and necessary. The simplest workaround is to add entries to your hosts file that point each container name you wish to access to the loopback address, 127.0.0.1
To create a Docker network, run the following command. In my example we're going to name our network atlasNetwork. If you want to use another name, remember to change the network name in all subsequent docker commands.
docker network create --driver bridge \
--subnet=10.255.0.0/16 \
atlasNetwork
Here, we're creating a network using the bridge driver - this is the simplest type of network. More complex network types allow the network to span multiple hosts. We're also manually specifying the subnet - if we leave this out Docker will choose one at random, and it could conflict with an existing network subnet, so it's safest to choose our own. We're also specifying a /16 mask to allow us to use IP address ranges within the last two octets - this will come up later!
Storage
Persistent data such as $BITBUCKET_HOME, or your database files, need to be stored somewhere outside of the container itself. For our test environment, we can simply store these directly on the host, our local OS. This means we can edit config files using our favourite text editor, which is pretty handy!
In the examples below, we're going to store our data files in the folder ~/dockerdata. There's no need to create this folder or any subfolders, as Docker will do this automatically. If you want to use a different folder, make sure to update the examples below.
You may wonder why we're not using Docker's named volumes instead of mounting folders on the host. Named volumes are an easier to manage abstraction and are generally recommended; however for the purposes of a test environment (particularly on Docker for Mac, where you don't have direct access to the virtualised file system) there's a huge practical benefit to being able to examine each container's persistent data directly. You may want to edit a number of configuration files in Bitbucket, or Postgres, or HAProxy, and this can be difficult when using a named volume, as it requires you to open a shell into the container - and many containers don't contain basic text editor utilities (not even vi!). However, if you prefer to use volumes, you can do so simply by replacing the host folder with the named volume in all of the below examples.
Database
The first service we need on our network is a database. Let's create a Postgres instance:
docker run -d \
--name postgres \
--restart=unless-stopped \
-e POSTGRES_PASSWORD=mysecretpassword \
-e PGDATA=/var/lib/postgresql/data/pgdata \
-v ~/dockerdata/postgres:/var/lib/postgresql/data/pgdata \
--network=atlasNetwork \
-p 5432:5432 \
postgres:latest
Let's examine what we're doing here:
-d
Run the container and detach from it (return to the prompt). Without this option, when the container starts we'll be attached directly to its stdout, and cancelling out would stop the container.
--name postgres
Set the name of the container to postgres, which also acts as its DNS record on our network.
--restart=unless-stopped
Sets the container to automatically start when Docker starts, unless you have explicitly stopped the container. This way, when you restart your computer, Postgres comes back up automatically
-e POSTGRES_PASSWORD=mysecretpassword
Sets password for the default postgres user to mysecretpassword
-e PGDATA=/var/lib/postgresql/data/pgdata
The official Postgres docker image recommends specifying this custom location when mounting the data folder to an external volume
-v ~/dockerdata/postgres:/var/lib/postgresql/data/pgdata
Mounts the folder /var/lib/postgresql/data/pgdata inside the container to an external volume, located on the host at ~/dockerdata/postgres. This folder will be created automatically
--network=atlasNetwork
Joins the container to our custom Docker network
-p 5432:5432
Publishes the Postgres port to the host machine, so we can access Postgres on localhost:5432. This isn't necessary for other containers to access the service, but it is necessary for us to get to it
postgres:latest
The latest version of the official Postgres docker image
Run the command, and hey presto, you can now access a fully functioning Postgres instance. For the sake of consistency, you may want to add your very first hosts entry here:
127.0.0.1 postgres
Now you, and any running containers, can access the instance at postgres:5432
Before you move on, you should connect to your database using your DB admin tool of choice. Connect to the hostname postgres with the username postgres, the default database postgres and the password mysecretpassword, and create a Bitbucket database ready to go:
CREATE USER bitbucket WITH PASSWORD 'bitbucket';
CREATE DATABASE bitbucket WITH OWNER bitbucket ENCODING 'UTF8';
If you don't have a DB admin tool handy, you can create a DB by using docker exec to run psql directly in the container:
# We need to run two commands because psql won't let
# you run CREATE DATABASE from a multi-command string
docker exec -it postgres psql -U postgres -c \
"CREATE USER bitbucket WITH PASSWORD 'bitbucket';"
docker exec -it postgres psql -U postgres -c \
"CREATE DATABASE bitbucket WITH OWNER bitbucket ENCODING 'UTF8';"
Elasticsearch
The next service we'll set up is Elasticsearch. We need a dedicated instance that all of our Data Center nodes can access. We have a great set of instructions on how to install a compatible version, configure it for use with Bitbucket, and install Atlassian's buckler security plugin: Install and configure a remote Elasticsearch instance
So how do we set this up in Docker? Well, it's easy:
docker pull dchevell/bitbucket-elasticsearch:latest
docker run -d \
--name elasticsearch \
-e AUTH_BASIC_USERNAME=bitbucket \
-e AUTH_BASIC_PASSWORD=mysecretpassword \
-v ~/dockerdata/elastic:/usr/share/elasticsearch/data \
--network=atlasNetwork \
-p 9200:9200 \
dchevell/bitbucket-elasticsearch:latest
Simply put, dchevell/bitbucket-elasticsearch is a pre-configured Docker image that is set up according to the instructions on Atlassian's Install and configure a remote Elasticsearch instance KB article. Atlassian's Buckler security plugin is installed for you, and you can configure the username and password with the environment variables seen above. Again, we're mounting a data volume to our host machine, joining it to our Docker network, and publishing a port so we can access it directly. This is solely for troubleshooting purposes, so if you want to poke around in your local Elasticsearch instance without going through Bitbucket, you can.
Now we're done, you can add your second hosts entry:
127.0.0.1 elasticsearch
HAProxy
Next, we'll set up HAProxy. Installing Bitbucket Data Center provides some example configuration, and again, we have a pre-configured Docker image that does all the hard work for us. But first, there's a few things we need to figure out first.
HAProxy doesn't play well with a Docker network's DNS system. In the real world, if a system is down, the DNS record still exists and connections will simply time out. HAProxy handles this scenario just fine. But in a Docker network, when a container is stopped, its DNS record ceases to exist, and connections to it fail with an "Unknown host" error. HAProxy won't start when this happens, which means we can't configure it to proxy connections to our nodes by container name. Instead, we will need to give each node a static IP address, and configure HAProxy to use the IP address instead.
Even though we have yet to create our nodes, we can decide on the IP addresses for them now. Our Docker network's subnet is 10.255.0.0/16, and Docker will dynamically assign containers addresses on the last octet (e.g. 10.255.0.1, 10.255.0.2 and so on). Since we know this, we can safely assign our Bitbucket nodes static IP addresses using the second-last octet:
10.255.1.1
10.255.1.2
10.255.1.3
With that out of the way, there's one more thing. HAProxy is going to be the face of our instance, so its container name is going to represent the URL we use to access the instance. In this example, we'll call it bitbucketdc. We're also going to set the host name of the machine to be the same.
docker run -d \
--name bitbucketdc \
--hostname bitbucketdc \
-v ~/dockerdata/haproxy:/usr/local/etc/haproxy \
--network=atlasNetwork \
-e HTTP_NODES="10.255.1.1:7990,10.255.1.2:7990,10.255.1.3:7990" \
-e SSH_NODES="10.255.1.1:7999,10.255.1.2:7999,10.255.1.3:7999" \
-p 80:80 \
-p 443:443 \
-p 7999:7999 \
-p 8001:8001 \
dchevell/bitbucket-haproxy:latest
In the above example, we're specifying the HTTP endpoints of our future Bitbucket nodes, as well as the SSH endpoints, as a comma separated list. The container will turn this into valid HAProxy configuration. The proxied services will be available on port 80 and port 443, so we're publishing them both. This container is configured to automatically generate a self-signed SSL certificate based on the hostname of the machine, so we have HTTPS access available out of the box.
Since we're proxying SSH as well, we're also publishing port 7999, Bitbucket Server's default SSH port. You'll notice we're also publishing port 8001. This is to access HAProxy's Admin interface, so we can monitor which nodes are detected as up or down at any given time.
Lastly, we're mounting HAProxy's config folder to a data volume. This isn't really necessary, but it will let you directly access haproxy.cfg so you can get a feel for the configuration options there.
Now it's time for our third hosts entry. This one is , since it impacts things like Base URL access, is absolutely required
127.0.0.1 bitbucketdc
Bitbucket nodes
Finally we're ready to create our Bitbucket nodes. Since these are all going to be accessed via the load balancer, we don't have to publish any ports. However, for troubleshooting and testing purposes there are times when you'll want to hit a particular node directly, so we're going to publish each node to a different local port so we can access it directly when needed.
docker run -d \
--name=bitbucket_1 \
-e ELASTICSEARCH_ENABLED=false \
-e HAZELCAST_NETWORK_MULTICAST=true \
-e HAZELCAST_GROUP_NAME=bitbucket-docker \
-e HAZELCAST_GROUP_PASSWORD=bitbucket-docker \
-e SERVER_PROXY_NAME=bitbucketdc \
-e SERVER_PROXY_PORT=443 \
-e SERVER_SCHEME=https \
-e SERVER_SECURE=true \
-v ~/dockerdata/bitbucket-shared:/var/atlassian/application-data/bitbucket/shared \
--network=atlasNetwork \
--ip=10.255.1.1 \
-p 7001:7990 \
-p 7991:7999 \
atlassian/bitbucket-server:latest
docker run -d \
--name=bitbucket_2 \
-e ELASTICSEARCH_ENABLED=false \
-e HAZELCAST_NETWORK_MULTICAST=true \
-e HAZELCAST_GROUP_NAME=bitbucket-docker \
-e HAZELCAST_GROUP_PASSWORD=bitbucket-docker \
-e SERVER_PROXY_NAME=bitbucketdc \
-e SERVER_PROXY_PORT=443 \
-e SERVER_SCHEME=https \
-e SERVER_SECURE=true \
-v ~/dockerdata/bitbucket-shared:/var/atlassian/application-data/bitbucket/shared \
--network=atlasNetwork \
--ip=10.255.1.2 \
-p 7002:7990 \
-p 7992:7999 \
atlassian/bitbucket-server:latest
docker run -d \
--name=bitbucket_3 \
-e ELASTICSEARCH_ENABLED=false \
-e HAZELCAST_NETWORK_MULTICAST=true \
-e HAZELCAST_GROUP_NAME=bitbucket-docker \
-e HAZELCAST_GROUP_PASSWORD=bitbucket-docker \
-e SERVER_PROXY_NAME=bitbucketdc \
-e SERVER_PROXY_PORT=443 \
-e SERVER_SCHEME=https \
-e SERVER_SECURE=true \
-v ~/dockerdata/bitbucket-shared:/var/atlassian/application-data/bitbucket/shared \
--network=atlasNetwork \
--ip=10.255.1.3 \
-p 7003:7990 \
-p 7993:7999 \
atlassian/bitbucket-server:latest
You can see that we're specifying the static IP addresses we decided on when we set up HAProxy. It's up to you whether you add hosts entries for these nodes, or simply access their ports via localhost. Since no other containers need to access our nodes via host name, it's not really necessary, and I personally haven't bothered.
The official Docker image adds the ability to set a Docker-only variable, ELASTICSEARCH_ENABLED=false to prevent Elasticsearch from starting in the container. The remaining Hazelcast properties are natively supported in the official docker image, because Bitbucket 5 is based on Springboot and can automatically translate environment variables to their equivalent dot properties for us.
Turn it all on
Now we're ready to go!
Access your instance on https://bitbucketdc (or whatever name you chose). Add a Data Center evaluation license (You can generate a 30 day one on https://my.atlassian.com) and connect it to your Postgres database. Log in, then go to Server Admin and connect your Elasticsearch instance (remember, it's running on port 9200, so set the Elasticsearch URL to http://elasticsearch:9200 and use the username and password we configured when we created the Elasticsearch container.
Visit the Clustering section in Server Admin, and you should see all of the nodes there, demonstrating that Multicast is working and the nodes have found each other.
That's it! Your Data Center instance is fully operational. You can use it as your daily instance by shutting down all but one node, and simply use it as a single node test instance - then, whenever you need, turn on the additional nodes.
see official docker image: https://hub.docker.com/r/atlassian/bitbucket-server/
just run:
docker run -v /data/bitbucket:/var/atlassian/application-data/bitbucket --name="bitbucket" -d -p 7990:7990 -p 7999:7999 atlassian/bitbucket-server
you can also take a look at the official dockerfile: https://hub.docker.com/r/atlassian/bitbucket-server/dockerfile
If you use the command to spin up the bitbucket container, you'll get the message below after the build:
The path /data/bitbucket is not shared from the host and is not known to Docker. You can configure shared paths from Docker -> Preferences... -> Resources -> File Sharing.
I have the Docker Swarm cluster setup in my preprod servers(3 manager node and 7 worker nodes); however I would like to replicate the same in Production servers but rather than using commands I prefer using a script.
At present I am using "docker swarm init" to initialize the swarm and then adding the workers and managers with the generated key.
I would have 30 servers and planning for 7 manager and 23 worker nodes.
I have searched the net; but could not find any script which can initialize the docker swarm automatically with a script in all the servers.
Any help would really appreciated.
The way I approached this was to have a common build script for all nodes, and to use Consul for sharing the docker swarm manager token with the rest of the cluster.
The first node (at 10.0.0.51) calls docker swarm init and places the token in the key value store, and the remaining nodes (at 10.0.0.52 onwards) read the token back and use it to call docker swarm join.
The bash looked something like this -
# Get the node id of this machine from the local IP address
privateNetworkIP=`hostname -I | grep -o 10.0.0.5.`
nodeId=`(echo $privateNetworkIP | tail -c 2)`
if [ $nodeId -eq 1 ]; then
sudo docker swarm init
MANAGER_KEY_IN=`sudo docker swarm join-token manager -q`
curl --request PUT --data $MANAGER_KEY_IN http://10.0.0.51:8500/v1/kv/docker-manager-key
else
MANAGER_KEY_OUT=`curl -s http://10.0.0.51:8500/v1/kv/docker-manager-key?raw`
sudo docker swarm join --token $MANAGER_KEY_OUT 10.0.0.51:2377
fi
... and works fine provided that node 1 is built first.
There is nothing like in-built utilities for that you can use the command like this :
You can create your custom script like this :
for i in `cat app_server.txt` ; do echo $i ; ssh -i /path/to/your_key.pem $i "sudo docker swarm join --token your-token-here ip-address-of-manager:port" ; done
Here app_server.txt is the ip address of your worker node that you want to add in your swarm.
--token : your token generated by manager on docker swarm init
Hope this may help.
You can also use ansible for the same but that require ansible docker modules installed on all the worker nodes.
Thank you!
I have the following questions:
I am logged into a Kubernetes pod using the following command:
./cluster/kubectl.sh exec my-nginx-0onux -c my-nginx -it bash
The 'ip addr show' command shows its assigned the ip of the pod. Since pod is a logical concept, I am assuming I am logged into a docker container and not a pod, In which case, the pod IP is same as docker container IP. Is that understanding correct?
from a Kubernetes node, I do sudo docker ps and then do the following:-
sudo docker exec 71721cb14283 -it '/bin/bash'
This doesn't work. Does someone know what I am doing wrong?
I want to access the nginx service I created, from within the pod using curl. How can I install curl within this pod or container to access the service from inside. I want to do this to understand the network connectivity.
Here is how you get a curl command line within a kubernetes network to test and explore your internal REST endpoints.
To get a prompt of a busybox running inside the network, execute the following command. (A tip is to use one unique container per developer.)
kubectl run curl-<YOUR NAME> --image=radial/busyboxplus:curl -i --tty --rm
You may omit the --rm and keep the instance running for later re-usage. To reuse it later, type:
kubectl attach <POD ID> -c curl-<YOUR NAME> -i -t
Using the command kubectl get pods you can see all running POD's. The <POD ID> is something similar to curl-yourname-944940652-fvj28.
EDIT: Note that you need to login to google cloud from your terminal (once) before you can do this! Here is an example, make sure to put in your zone, cluster and project:
gcloud container clusters get-credentials example-cluster --zone europe-west1-c --project example-148812
The idea of Kubernetes is that pods are assigned on a host but there is nothing sure or permanent, so you should NOT try to look up the IP of a container or pod from your container, but rather use what Kubernetes calls a Service.
A Kubernetes Service is a path to a pod with a defined set of selectors, through the kube-proxy, which will load balance the request to all pods with the given selectors.
In short:
create a Pod with a label called 'name' for example. let's say name=mypod
create a Service with the selector name=mypod that you call myService for example, to which you assign the port 9000 for example.
then you can curl from a pod to the pods served by this Service using
curl http://myService:9000
This is assuming you have the DNS pod running of course.
If you ask for a LoadBalancer type of Service when creating it, and run on AWS or GKE, this service will also be available from outside your cluster. For internal only service, just set the flag clusterIP: None and it will not be load balanced on the outside.
see reference here:
https://kubernetes.io/docs/concepts/services-networking/service/
https://kubernetes.io/docs/tutorials/services/
Kubernetes uses the IP-per-pod model. All containers in the same pod share the same IP address as if they are running on the same host.
The command should follow docker exec [OPTIONS] CONTAINER COMMAND [ARG...]. In your case, sudo docker exec -it 71721cb14283 '/bin/bash' should work. If not, you should provide the output of your command.
It depends on what image you use. There is nothing special about installing a software in a container. For nginx, try apt-get update && apt-get install curl
There's an official curl team image these days:
https://hub.docker.com/r/curlimages/curl
Run it with:
kubectl run -it --rm --image=curlimages/curl curly -- sh