Configure HTTP Proxy for Containers in Kubernetes - docker

Question

I have a Kubernetes v1.18.3 cluster and the workers have a Docker v19.03.6 dameon.
I'm looking for a way to automatically inject the HTTP_PROXY and HTTPS_PROXY to every container that Kubernetes creates.
I tried creating a ~/.docker/config.json file, but it didn't work.
What would be the proper way to accomplish it?

Answer 1

Was interested in your case, even reproduced with the same docker and k8s versions...
Used official Configure Docker to use a proxy server documentation to set proxy for docker in ~/.docker/config.json
Configure the Docker client On the Docker client, create or edit the
file ~/.docker/config.json in the home directory of the user which
starts containers. Add JSON such as the following, substituting the
type of proxy with httpsProxy or ftpProxy if necessary, and
substituting the address and port of the proxy server. You can
configure multiple proxy servers at the same time.
{
"proxies":
{
"default":
{
"httpProxy": "http://127.0.0.1:3001",
"httpsProxy": "http://127.0.0.1:3001",
"noProxy": "*.test.example.com,.example2.com"
}
}
}
Save the file.
When you create or start new containers, the environment variables are set automatically within the container.
My config was:
{
"proxies": {
"default": {
"httpProxy": "http://user:pass#my.proxy.domain.com",
"httpsProxy": "http://user:pass#my.proxy.domain.com"
}
}
}
So basically after setting above in ~/.docker/config.json, the proxy-server will be automatically used when starting a brand new containers.
In my case that worked, I can verify that by using cli and creating e.g busybox container.
$ docker container run --rm busybox env
HTTP_PROXY=http://user:pass#my.proxy.domain.com
http_proxy=http://user:pass#my.proxy.domain.com
HTTPS_PROXY=http://user:pass#my.proxy.domain.com
https_proxy=http://user:pass#my.proxy.domain.com
HOME=/root
Please keep in mind that there should be issues with next part:
On the Docker client, create or edit the file ~/.docker/config.json in
the home directory of the user which starts containers.
Be careful with USER you use and make sure your HOME env is set to correct one.
Links to github almost similar issue and ways to resolve:
1) https://github.com/kubernetes/kubernetes/issues/45487#issuecomment-312042754
I dug into this a bit and the issue for me was that the HOME
environment variable was empty when kubelet was launched through a
systemd unit. While it's not documented this way, loading the
configuration from /root/docker/config.json or /root/.dockercfg
requires that HOME=/root
Setting User=root in the [Service] declaration fixed it up for me.
2) https://github.com/kubernetes/kubernetes/issues/45487#issuecomment-378116386
3) https://github.com/kubernetes/kubernetes/issues/45487#issuecomment-464516064 (partial info)
(3) vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
Add User=root
File looks kind of like this
[Service]
User=root
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
(4) Reload and restart kubelet
systemctl daemon-reload
systemctl restart kubelet
Exactly in my case everything worked fine from scratch. So read carefully and check points I highlighted. Most probably you have very tiny problem/typo, cause it works as expected.
I hope my investigation will help you

Answer 2

This sounds right & logical, but does not work for me.
I am running kubernetes version "v1.18.6". Data below. This fails. But setting same http-proxy as Env for dockerd, works.
admin#str-s6000-acs-13:/etc/systemd/system/kubelet.service.d$ sudo cat /proc/$(pidof kubelet)/environ | tr '\0' '\n'
LANG=en_US.UTF-8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOME=/root
LOGNAME=root
USER=root
SHELL=/bin/sh
INVOCATION_ID=fd58e75d7be64758b01e2d8d63fdf7f6
JOURNAL_STREAM=9:11737012
KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf
KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml
KUBELET_KUBEADM_ARGS=--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.2
admin#str-s6000-acs-13:/etc/systemd/system/kubelet.service.d$ sudo cat /root/.docker/config.json
{
"proxies":
{
"default":
{
"httpProxy": "http://20.72.201.152:3128",
"httpsProxy": "http://20.72.201.152:3128"
}
}
}
admin#str-s6000-acs-13:/etc/systemd/system/kubelet.service.d$
Apr 12 01:14:49 str-s6000-acs-13 dockerd[27360]: time="2021-04-12T01:14:49.630282308Z" level=error msg="Handler for POST /images/create returned error: Get https://sonicanalytics.azurecr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"