I have mesos-master (mesosphere/mesos-master) and mesos-slave (mesosphere/mesos-slave) running inside my Kubernetes cluster.
Mesos slave starts the docker containers (docker is accessed by mounting /usb/bin/docker from host) with my data processing application (short lived, 1-5 min) which needs to access other kubernetes services. So shortly speaking I need to access Kubernetes DNS from a container.
Is it possible to do that?
Thanks
I found only one way:
I am resolving "kube-dns.kube-system" host into an IP address. Then I am injecting "metadata.namespace" into environment variable KUBERNETES_NAMESPACE. and finally I am passing --dns RESOLVED_IP and --dns-search ${KUBERNETES_NAMESPACE}.svc.cluster.local, so a mesos's docker container is able to talk to the services.
Related
I want to connect a docker container running locally to a service running on a Kubernetes cluster. To do so I have exposed a service through reserving some static IP addresses.
I have also saved those IP addresses in local DNS, in the /etc/hosts/ file:
123.123.123.12 host1
456.456.456.45 host2
I want to link my container to that such that all the traffic is routed to those addresses so that it can be processed by the cluster. I am using the link feature in the docker container but it isn't working.
I want to connect directly using IP? How should I do this?
There's no difference doing this if the client is or isn't in Docker. However you have the service exposed from Kubernetes, you'd make the same connection to it from a process running on an external host or from a process running in a Docker container on that host.
Say, as in the example in the Kubernetes documentation, you're running a NodePort service that's accessible on port 31496 on every node in the cluster, and you're trying to connect to it from outside the cluster. Maybe as in the question 123.123.123.12 is some node in the cluster. A typical setup would be to get the location of the service from an environment variable (JavaScript process.env.THE_SERVICE_URL; Ruby ENV['THE_SERVICE_URL']; Python os.environ['THE_SERVICE_URL']; ...).
When you're developing, you could set that variable in your local shell:
export THE_SERVICE_URL=http://123.123.123.12:31496
cd here && ./kubernetes_client_script.py
When you go to deploy your application, you can set the same environment variable:
docker run -e THE_SERVICE_URL=http://123.123.123.12:31496 me:k8s-client
I have created a GCP VM instance, with option Deploy as Container pointing to an image in my private GCR(nginx customized).
Also while creating the instance, I had given allow 'https' and 'http' traffic.
Though the application is working fine, on connecting the instance via ssh and inspecting docker containers
(docker ps)
I see the container ports are not exposed. Wondering how the http/https request are handled by the container here via the instance??
When you use the deploying containers option in GCE it runs docker with access to the host network.
From the relevant gcp docs :
Containerized VMs launch containers with the network set to host mode.
A container shares the host network stack, and all interfaces from the
host are available to the container.
More detailed info on the different network modes here.
Other than what #Stefan R has told, you should also use PORT number greater than 1000 as auto deployed container images aren't run as root and hence can't access privileged ports.
https://www.staldal.nu/tech/2007/10/31/why-can-only-root-listen-to-ports-below-1024/
https://www.google.co.in/search?q=privileged+ports+linux&oq=privileged+ports+linux
I've got a Docker swarm cluster where the manager nodes are in constant 'drain' mode, e.g. no container will ever run on them.
Now I'm running Jenkins in a container on a worker node and I'd like Jenkins to be able to deploy images to the swarm cluster.
My reasoning so far:
Mounting /var/run/docker.sock is obviously not an option as the docker manager and Jenkins container are on different hosts and the local docker is not a swarm manager.
Connecting from the Jenkins container to the local docker host using tcp has the same issues
Adding the Jenkins container to the --network host, seems not to be possible: a container cannot also be in an overlay network at the same time
I assume this is quite a common use case, yet I haven't found a solution and maybe someone here has an idea.
Thanks!
I have 6 microservices packed in docker containers. On every swarm node, i have installed consul agent, binded to host ip, and client in 0.0.0.0 mode.
All microservices are in docker-compose file which I am running from Swarm manager.
Microservices are written in Java and in bootstrap.yml I must to specify consul agent endpoint. Possible choices are:
localhost
${HOSTIP} environment variable
Problems:
- localhost is not localhost of host, but container localhost, and I don't have consul agent on container localhost but host.
- ${HOSTIP} in compose file i have to supply this env var. But, I don't know where Swarm MAnager will schedule microservice start so I cannot know which IP address will be used.
I tried to expose on each node host ip address but since i am running compose from manager, it will not read this variable.
Do you have any proposal how to solve this? I have consul cluster, 3 managers and 3 nodes. on each manager and node i have consul agent started (as docker container). No matter what type of networking i am using, i am not able to start up microservice. I started consul as --net=host and --net=bridge, but this is not working.
Is there anyone with some idea?
Thanks ahead.
So you are running consul in containers also, right? Is it possible in your setup to link containers? So you could start the consul containers as "consul" on each host and link your microservices to it. Linked containers get a hosts entry and so the consul service should be reachable at "consul:8500" from within your services.
Edit: If you are using the official Consul Docker image from Hashicorp, you can configure the client address to 0.0.0.0, this should make the consul API available to the other containers running on the host.
Let me answer my own Q: This is not a way we want to do this, I mean, we cannot put some things in Swarm and some thing outside Swarm with expectation that it will work. It will not. Consul as a service discovery cannot be used outside Swarm, too. Simple answer would be to use Docker Orchestration and Service discovery and not to involve Consul. If someone is using Swarm, everything should be in overlay networks (rabbit, redis, elk and so on)...
I have a dockerized dropwizard service deployed on marathon. I am using Hazelcast as a distributed cache which I start has part of my dropwizard service. I have placed a constraint to ensure that each container is started on a unique Host.
"constraints": [
[
"hostname",
"UNIQUE"
]
],
I have exposed 2 ports on my docker container, 10012 for my service and 10013 for Hazelcast. I am using Zookeeper for my Dropwizard service discovery. Thus when I startup my Hazelcast instance I have access to the hostnames of all the machines on which my docker containers are running and I add all of them as below.
TcpIpConfig tcpIpConfig = join.getTcpIpConfig();
// finder is a handle to a service discovery service and the following gets me all the hosts on which my docker containers will run.
List<ServiceNode<ShardInfo>> nodes = finder.getAllNodes();
nodes.stream()
.peek(serviceNode -> log.info("Adding " + serviceNode.representation() + " to hazelcast."))
.map(serviceNode -> serviceNode.getHost())
.forEach(host -> tcpIpConfig.addMember(host));
tcpIpConfig.setRequiredMember(null).setEnabled(true);
Now issues:
If I use network type as BRIDGE while deploying on Marathon, then I don't know the docker container host and thus my 2 docker containers don't know each other. It looks something like this:
ip-10-200-2-219.ap-southeast-1.compute.internal (docker host) - 172.12.1.18 (docker container ip)
ip-10-200-2-220.ap-southeast-1.compute.internal (docker host) - 172.12.1.20 (docker container ip)
From zookeeper I get the docker host IPs but not the docker container IPs.
If I use network type as HOST then everything works but an issue is that I then have to make sure that ports which my docker containers are running always have port 1001 and 10013 available. (With BRIDGE the docker container ports are bound to a random ports).
Analysis:
The two docker containers are inside their own network localized to the slaves. They need to recognise each other using the public IP of the slave and the bridged port to which 5701 (or whatever hazelcast port you are using).
Solution
In the TCP/IP configuration set the Public Address and Port when starting the instance. All instances will do this and they will talk to each other using the marathon slave IP and the randomized port used to it.
Use the HOST and PORT_5701 variables provided by marathon and available inside the container to do this.
Config hzConfig = new Config();
hzConfig.getNetworkConfig().setPublicAddress(
String.format("%s:%s",
System.getenv("HOST"),
System.getenv("PORT_5701")));
Refer to hazelcast network config documentation to understand a bit more about the public address option.
You can use TCP/IP Discovery mechanism and make sure hazelcast nodes bind to public ip of docker container. Although this solution is might help if you know docker container IPs, before your deployment.
<hazelcast>
...
<network>
...
<join>
<multicast enabled="false">
</multicast>
<tcp-ip enabled="true">
<member>docker-host1</member>
<member>docker-host2</member>
<member>172.12.1.20</member>
<member>192.168.1.21</member>
</tcp-ip>
...
</join>
...
</network>
...
</hazelcast>
Getting hazelcast members to discover each other and form a cluster is tricky. I ended up following advice of #bitsofinfo -
https://github.com/hazelcast/hazelcast/issues/9219
#santanu 's answer is correct. Public address needs to be set properly for hazelcast members to be able to discover each other. Here's a parameterized way of doing this : https://github.com/gagangoku/hazelcast-docker