I start, stop and remove containers as part of a continuous build process. Each time the build runs, the containers get new IPs.
Im already at 172.17.0.95 after starting at 172.17.0.2 an hour back.
Since I remove old containers each build, I would also like to reset the IP counter, so that I dont have a timebomb where I run out of IP addresses after say a few hundred builds.
Please let me know how I can let the entity (DHCP Server?) know that IPAddress is free again, and to reset counter.
Thanks in advance SO community!
Docker seems to default to using 172.17.0.0/16 for the docker0 interface. That's 255^2 addresses, and if you use 100 every hour you'll run through them all in just over 27 days. I think Docker is just being conservative in not recycling them faster, but will loop around when it reaches the end.
If you need a bigger or different address space, you can use the --bip and --fixed-cidr flags on the Docker server to choose your own CIDR. See the Docker documentation on networking here.
If you really just want to reset the counter, you would need to restart the docker server. This will have the side-effect of terminating all your running containers.
Related
We have several containers which are connected to the same custom network with ipv6 support. Now we need one of those containers to act as the default gateway (in short, the default gateway of our custom network should be set to the ip of this one container).
This is rather impossible (or at least we did not find an answer to that). There is another option however.
Since we need to forward a specific ip address to another one we can simply use:
ip -6 route add xxxx:yy:: via aaaa:bb::c
in each container. That would be a rather unelegant but satisfaying solution. It does not work like that. I suppose we would need to restart the container network. I could not find a way to do this without restarting the container which would of course nullifies the above command.
Any idea on how to do this properly? I apologise for my possible direct tone in this message, I haven't slept much in the last few days and I am thinking of changing careers. How does rocket scientist sound? gotta be easier that working with docker...
so, no network restart was necessary. I had an error in the command syntax. Since I wanted to redirect all IPs to that specific one I had to write it like that:
ip -6 route add xxxx:yy::/64 via aaaa:bb::c
You can easily put this in a bash script and execute it every time a container starts:
apt-get update && apt-get install iproute2 -y
ip -6 route add xxxx:yy::/64 via aaaa:bb::c
Quick side note, for the IP command to be able to run the container has to run in "privileged" mode.
Its has been 3 months since monitoring. And nature of fail2ban that I observe is that it stop blocking ips after few days of busy schedule. Then I use to restart it and it again start working, blocking IP. For few months things goes this way but after few months, fail2ban not block IP even after restarting. Then I have to do fresh installation of fail2ban and then It again start blocking.
Can anyone tell reason for such nature of fail2ban?
Is hard to know what is happening withou any config file or log. Anyway, that kind of behavior can be because of a clock sync problem.
I don't know if you structure could be: one machine generating logs of the services and other machine with fail2ban reading it and banning. Or a host machine with fail2ban and the services in docker containers. In these case of scenarios, the date should be sync, that includes the timezone. Maybe after restarting your server is synchronizing your clock and changing the date... I don't know. Is a possibility.
Fail2ban removes the banned hosts from your list according to the time set in /etc/fail2ban/jail.local
# "bantime" is the number of seconds that a host is banned.
bantime = 3600
No ban on fail2ban is permanent (unless done by some configuration modification), so if you reboot with
sudo /etc/init.d/fail2ban restart
all banned ips will automatically receive an unban.
For you to restart the service without losing the list of banned ips, type
sudo fail2ban-client reload
I've set up a docker swarm mode cluster, with two managers and one worker. This is on Centos 7. They're on machines dkr1, dkr2, dkr3. dkr3 is the worker.
I was upgrading to v1.13 the other day, and wanted zero downtime. But it didn't work exactly as expected. I'm trying to work out the correct way to do it, since this is one of the main goals, of having a cluster.
The swarm is in 'global' mode. That is, one replica per machine. My method for upgrading was to drain the node, stop the daemon, yum upgrade, start daemon. (Note that this wiped out my daemon config settings for ExecStart=...! Be careful if you upgrade.)
Our client/ESB hits dkr2, which does its load balancing magic over the swarm. dkr2 which is the leader. dkr1 is 'reachable'
I brought down dkr3. No issues. Upgraded docker. Brought it back up. No downtime from bringing down the worker.
Brought down dkr1. No issue at first. Still working when I brought it down. Upgraded docker. Brought it back up.
But during startup, it 404'ed. Once up, it was OK.
Brought down dkr2. I didn't actually record what happened then, sorry.
Anyway, while my app was starting up on dkr1, it 404'ed, since the server hadn't started yet.
Any idea what I might be doing wrong? I would suppose I need a health check of some sort, because the container is obviously ok, but the server isn't responding yet. So that's when I get downtime.
You are correct -- You need to specify a healthcheck to run against your app inside the container in order to make sure it is ready. Your container will not receive traffic until this healtcheck has passed.
A simple curl to an endpoint should suffice. Use the Healthcheck flag in your Dockerfile to specify a healthcheck to perform.
An example of the healthcheck line in a Dockerfile to check if an endpoint returned 200 OK would be:
HEALTHCHECK CMD curl -f 'http://localhost:8443/somepath' || exit 1
If you can't modify your Dockerfile, then you can also specify your healthcheck manually at deployment time using the compose file healthcheck format.
If that's also not possible either and you need to update a running service, you can do a service update and use a combination of the health flags to specify your healthcheck.
I am running docker with kubernetes.
I need to find out when the last time docker container is used by a user.
I am creating one container per user.I have to kill that container if the user has not interacted with the container for a specific amount of time.
Currently, I am running a daemon inside docker container which checks last modified files and sends the info.
Is there any docker/kubernetes API for the same?
I thinks there's no API for that as "usage" is something which is hard to measure. One way would be to check whether systems stopped logging at some point back in time.
The other option would be to use the metrics which are exposed by Kubernetes and bring up monitoring and alerting systems like Prometheus to tell you once a Deployment/Pod is not used anymore. "Usage" could then be determined through the exposed network metrics e.g. like this:
max_over_time(
container_network_receive_bytes_total{kubernetes_pod_name=~"^yourdeployment.*$"}[1h]
)
If that's below a certain threshold you could trigger and alert and perform further actions.
I bundled Qugaa bgpd and nsm binaries into a docker based on Ubuntu 3.13.0-36-generic kernel. Now I run multiple of these dockers on a linux sever. These dockers are instantiated using docker-py.
I used pipework and created an Ethernet interface on each docker and assigned an 172.17.xx.xx kind of address. My BGP configuration is such that all the dockers are completely meshed with respect to BGP. i.e each BGP on a docker is connected to every other BGP running on other dockers.
BGP sessions get established and the BGP routes etc are absolutely fine. Now, when the number of dockers exceeds 30, i could never connect to bgp. "top" command doesn't show much cpu usage, memory is in limits, not much network activity and I also don't expect much processing within BGP process.
when i took tcpdump on the container, this is what it looks like.
9 2014-09-26 18:17:54.997861 0a:60:4a:3b:56:31 ARP 44 Who has 172.17.1.32? Tell 172.17.1.6
when I run 40 dockers, i see 40 such ARP requests as shown above followed by an one ARP reply.
however this continuously happens, and it approximately results in 1600 (40*40) such messages in a short span of time. I believe this is what is not allowing me to connect to a local bgp module using "telnet localhost bgpd" command.
I don't think this is anything specific to either Quagga or BGP. I suspect some thing to do with docker networking. Does anybody got such an issue or any idea how to fix this or what is the root cause for this ?
Finally I could find the root cause behind this and also fixed this. This is due to the combination of number of dockers instances, number of MAC entries that i am creating on each docker and the default ARP cache size specified on my linux server as 1024.
In my case I end up creating MAC entries as (number of dockers * number of ethernet interfaces i am creating through pipework). So the problem occurs when the number of MAC entries on each DOCKER exceeds the default size of 1024.
adding below lines at the end of /etc/sysctl.conf file or modifying as below if it already has these entries in /etc/sysctl.conf solved the issue.
net.ipv4.neigh.default.gc_thresh1 = 8192
net.ipv4.neigh.default.gc_thresh2 = 12288
net.ipv4.neigh.default.gc_thresh3 = 16384
After modifying, execute "sysctl -p" command