While connecting the node in rancher for Kubernetes it says:-
Add at least one host
Waiting for a host to be active
Creating Kubernetes system stack
Starting services (2 of 12)
Waiting for Kubernetes API
Related
I've been having an issue deploying the Portainer agent across a newly created Docker swarm. One of the nodes starts the agent without any issue [we'll call that HOST#1] but then HOST#2 will just
infinitely try to deploy the agent container (showing preparing container under the services menu in portainer), before eventually showing failed with the below error message and then attempting to create a new container.
Error:
starting container failed: error creating external connectivity network: cannot restrict inter-container communication: open /proc/sys/net/bridge/bridge-nf-call-iptables: permission denied
What i've tested/tried
I have been following the instructions outlined on the portainer wiki and using the agent-stack.yml file for adding an existing agent to a swarm, https://docs.portainer.io/v/ce-2.11/start/install/agent/swarm/linux I did also try delete the agent altogether from the swarm and deyploying it again, with the same results.
No issues deploying the hello world service to the swarm.
Temporarily disabling ufw
setting ufw allow in on docker0
setting ufw allow in on docker_gwbridge
docker node ls reports both nodes are Ready & avalible
Environment details:
Both systems running Ubuntu server 20.04
Both systems running Docker version 20.10.12
Both systems running kernel versions 5.4.0*
Both are running as manager nodes in the swarm
Portainer Agent 2.11.0
The system unable to deploy the Agent is a OpenVZ VPS [HOST#2]
The VPS [HOST#2] is connected to my local network via a OpenVPN (layer 2) tap adapter, therefore the swarm is connecting over the VPN
HOST2 is running ufw for firewall management while HOST1 is not
I'm quite new to docker swarm but i have been using docker for many years. Any help highly appreciated
I have a running k3d Kubernetes cluster:
$ kubectl cluster-info
Kubernetes master is running at https://0.0.0.0:6550
CoreDNS is running at https://0.0.0.0:6550/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://0.0.0.0:6550/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
I have a python script that uses the kubernetes client api and manages namespaces, deployments, pod, etc. This works just fine in my local environment because I have all the necessary python modules installed and have direct access to my local k8s cluster. My goal is to containerize so that this same script is successfully run for my colleagues on their systems.
While running the same python script in a docker container, I receive connection errors:
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='172.17.0.1', port=6550): Max retries exceeded with url: /api/v1/namespaces (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f8b637c5d68>: Failed to establish a new connection: [Errno 113] No route to host',))
172.17.0.1 is my docker0 bridge address so assumed that would resolve or forward traffic to my localhost. I have tried loading k8s configuration from my local .kube/config which references server: https://0.0.0.0:6550 and also creating a separate config file with server: https://172.17.0.1:6550 and both give the same No route to host error (with the respective ip address in the HTTPSConnectionPool(host=...))
One idea I was pursing was running a socat process outside the container and tunnel traffic from inside the container across a bridge socket mounted in from the outside, but looks like the docker image I need to use does not have socat installed. However, I get the feeling like the real solution should be much simplier than all of this.
Certainly there have been other instances of a docker container needing access to a k8s cluster served outside of the docker network. How is this connection typically established?
Use docker network command to create a predefined network
You can pass --network to attach k3d to an existing Docker network and also to docker run to do the same for another container
https://k3d.io/internals/networking/
I have a single node docker Swarm. I deployed a stack with Influxdb (placement: manager), Grafana (placement: manager) and cAdvisor (deploy mode global).
I use Portainer to visualize the stack. And it correctly shows 1 manager node with its services.
Then I added a second (worker) node with docker swarm join etc...
Now the cAdvisor service is also launched on the worker, but it has a problem, it keeps logging:
E1115 14:52:46.772290 1 memory.go:91] failed to write stats to influxDb - Post http://influx:8086/write?consistency=&db=cadvisor&precision=&rp=: dial tcp: lookup influx on 127.0.0.11:53: no such host
Any ideas?
Thanx
I am running into a peculiar problem.
I have kubernetes cluster, I setup no_proxy for the master node of the cluster (in docker systemd environment). In order to be able to run docker build/push to a registry that is running on docker on the master node.
Now I have a problem, as my containers cannot access the outside network (because the communication happens through k8s master node I presume).
Or if I choose not to set no_proxy for the master node in docker then I cannot push images to my registry through the external IP of the master, have to use (localhost) as push destination -> which breaks my app later on.
I use weave as my cni plugin
The network communication of containers running on your nodes has nothing to do with the network communication of your master to the outside world or it through a proxy.
Basically, the network communication for your containers running on a node goes through its own network interface, etc.
Having said that, are you running your workloads on your master? If yes, that could be affecting the communication of your master containers (if you set no_proxy for some hostnames). It could also be affecting the communication of your kube-controller-manager, kube-apiserver, core-dns, kubelet and network overlay on the master.
Are you configuring your docker client proxy correctly as per here?
I have been building a distributed load testing application using Kubernetes and Locust (similar to this).
I currently have a multi-node cluster running on bare-metal (running on an Ubuntu 18.04 server, set up using Kubeadm, and with Flannel as my pod networking addon).
The architecture of my cluster is as follows:
I have a 'master instance' of the Locust application running on my master node.
I have 'slave instances' of the Locust application running on all of my other nodes. These slave instances must be able to bind to a port (5558 by default) of the master instance.
As of now, I don't believe that that is happening. My cluster shows that all of my deployments are healthy and running, however I am unable to access the logs of any of my slave instances which are running on nodes other than my master node. This leads me to believe that my pods are unable to communicate with each other across different nodes.
Is this an issue with my current networking or deployment setups (I followed the linked guides pretty-much verbatim)? Where should I start in debugging this issue?
How slaves instances try to join the master instance. You have to create master service (with labels) to access master pod. Also, make sure your SDN is up and master is reachable to slave instances. You can test using telnet to master pod IP from slave instances.
Based on your description of the problem I can guess that you have a connection problem caused by firewall or network misconfiguration.
From the network perspective, there are requirements mentioned in Kubernetes documentation:
all containers can communicate with all other containers without NAT
all nodes can communicate with all containers (and vice-versa) without NAT
the IP that a container sees itself as is the same IP that others see it as
From the firewall perspective, you need to ensure the cluster traffic can pass the firewall on the nodes.
Here is the list of ports you should have opened on the nodes provided by CoreOS website:
Master node inbound: TCP: 443 from Worker Nodes, API Requests, and End-Users
UDP: 8285,8472 from Master & Worker Nodes
Worker node inbound: TCP: 10250 from Master Nodes
TCP: 10255 from Heapster
TCP: 30000-32767 from External Application Consumers
TCP: 1-32767 from Master & Worker Nodes
TCP: 179 from Worker Nodes
UDP: 8472 from Master & Worker Nodes
UPD: 179 from Worker Nodes
Etcd node inbound: TCP: 2379-2380 from Master & Worker Nodes
see ip forwarding is enabled on all the nodes.
# sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
if not enable it like this and test it.
echo 1 > /proc/sys/net/ipv4/ip_forward