Docker Container Refuses to NOT use Proxy for Docker Network - docker

Question

I'm having issues trying to get networking to work correctly in my container inside a corp domain/behind a proxy.
I've correctly configured (I think) Docker to get around the proxy for downloading images, but now my container is having trouble talking to another container inside the same docker-compose network.
So far, the only resolution is to manually append the docker-compose network to the no_proxy variable in the docker config, but this seems wrong and would need to be configured for each docker-compose network and requires a restart of docker.
Here is how I configured the docker proxy settings on host:
cat << "EOF" >docker_proxy_setup.sh
#!/bin/bash
#Proxy
#ActiveProxyVar=127.0.0.1:80
#Domain
corpdom=domain.org
httpproxyvar=http://$ActiveProxyVar/
httpsproxyvar=http://$ActiveProxyVar/
mkdir ~/.docker
cat << EOL >~/.docker/config.json
{
"proxies":
{
"default":
{
"httpProxy": "$httpproxyvar",
"httpsProxy": "$httpsproxyvar",
"noProxy": ".$corpdom,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"
}
}
}
EOL
mkdir -p /etc/systemd/system/docker.service.d
cat << EOL >/etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=$httpproxyvar"
Environment="HTTPS_PROXY=$httpsproxyvar"
Environment="NO_PROXY=.$corpdom,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"
EOL
systemctl daemon-reload
systemctl restart docker
#systemctl show --property Environment docker
docker run hello-world
EOF
chmod +x docker_proxy_setup.sh
docker_proxy_setup.sh
and basically if I change to this:
#Domain
corpdom=domain.org,icinga_icinga-net
I am able to use curl to test network and it works correctly, but ONLY when using container_name.icinga_icinga-net
Eg:
This fails curl -k -u root:c54854140704eafc https://icinga2-api:5665/v1/objects/hosts
While this succeeds curl -k -u root:c54854140704eafc https://icinga2-api.icinga_icinga-net:5665/v1/objects/hosts
Note that using curl --noproxy seems to have no effect.
Here is some output from container for reference, any ideas what I can do to have containers NOT use proxy for Docker Networks (private IPv4)?
root#icinga2-web:/# ping icinga2-api
PING icinga2-api (172.30.0.5) 56(84) bytes of data.
64 bytes from icinga2-api.icinga_icinga-net (172.30.0.5): icmp_seq=1 ttl=64 time=0.138 ms
64 bytes from icinga2-api.icinga_icinga-net (172.30.0.5): icmp_seq=2 ttl=64 time=0.077 ms
^C
--- icinga2-api ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1025ms
rtt min/avg/max/mdev = 0.077/0.107/0.138/0.030 ms
root#icinga2-web:/# curl --noproxy -k -u root:c54854140704eafc https://172.30.0.5:5665/v1/objects/hosts
curl: (56) Received HTTP code 503 from proxy after CONNECT
root#icinga2-web:/# curl -k -u root:c54854140704eafc https://172.30.0.5:5665/v1/objects/hosts
curl: (56) Received HTTP code 503 from proxy after CONNECT
root#icinga2-web:/# curl -k -u root:c54854140704eafc https://icinga2-api:5665/v1/objects/hosts
curl: (56) Received HTTP code 503 from proxy after CONNECT
root#icinga2-web:/# curl -k -u root:c54854140704eafc https://icinga2-api.icinga_icinga-net:5665/v1/objects/hosts
{"results":[{"attrs":{"__name":"icinga2-api","acknowledgement":0,"acknowledgement_expiry":0,"acknowledgement_last_change":0,"action_url":"","active":true,"address":"127.0.0.1","address6":"::1","check_attempt":1,"check_command":"hostalive","check_interval":60,"check_period":"","check_timeout":null,"command_endpoint":"","display_name":"icinga2-api","downtime_depth":0,"enable_active_checks":true,"enable_event_handler":true,"enable_flapping":false,"enable_notifications":true,"enable_passive_checks":true,"enable_perfdata":true,"event_command":"","executions":null,"flapping":false,"flapping_current":0,"flapping_ignore_states":null,"flapping_last_change":0,"flapping_threshold":0,"flapping_threshold_high":30,"flapping_threshold_low":25,"force_next_check":false,"force_next_notification":false,"groups":["linux-servers"],"ha_mode":0,"handled":false,"icon_image":"","icon_image_alt":"","last_check":1663091644.161905,"last_check_result":{"active":true,"check_source":"icinga2-api","command":["/usr/lib/nagios/plugins/check_ping","-H","127.0.0.1","-c","5000,100%","-w","3000,80%"],"execution_end":1663091644.161787,"execution_start":1663091640.088944,"exit_status":0,"output":"PING OK - Packet loss = 0%, RTA = 0.05 ms","performance_data":["rta=0.055000ms;3000.000000;5000.000000;0.000000","pl=0%;80;100;0"],"previous_hard_state":99,"schedule_end":1663091644.161905,"schedule_start":1663091640.087908,"scheduling_source":"icinga2-api","state":0,"ttl":0,"type":"CheckResult","vars_after":{"attempt":1,"reachable":true,"state":0,"state_type":1},"vars_before":{"attempt":1,"reachable":true,"state":0,"state_type":1}},"last_hard_state":0,"last_hard_state_change":1663028345.921676,"last_reachable":true,"last_state":0,"last_state_change":1663028345.921676,"last_state_down":0,"last_state_type":1,"last_state_unreachable":0,"last_state_up":1663091644.161787,"max_check_attempts":3,"name":"icinga2-api","next_check":1663091703.191943,"next_update":1663091771.339701,"notes":"","notes_url":"","original_attributes":null,"package":"_etc","paused":false,"previous_state_change":1663028345.921676,"problem":false,"retry_interval":30,"severity":0,"source_location":{"first_column":1,"first_line":18,"last_column":20,"last_line":18,"path":"/etc/icinga2/conf.d/hosts.conf"},"state":0,"state_type":1,"templates":["icinga2-api","generic-host"],"type":"Host","vars":{"disks":{"disk":{},"disk /":{"disk_partitions":"/"}},"http_vhosts":{"http":{"http_uri":"/"}},"notification":{"mail":{"groups":["icingaadmins"]}},"os":"Linux"},"version":0,"volatile":false,"zone":""},"joins":{},"meta":{},"name":"icinga2-api","type":"Host"}]}
root#icinga2-web:/#
PS: I'm fairly certain this is not a specific issue to icinga as I've had some random proxy issues w/ other containers. But, I can say I've tested this icinga compose setup outside corp domain and it worked fine 100%.
Partial Resolution!
I would still prefer to use CIDR to have no_proxy work via container name without having to adjust docker-compose/.env but I got it to work.
A few things I did:
Added lowercase to docker service -->:
cat << EOL >/etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=$httpproxyvar"
Environment="HTTPS_PROXY=$httpsproxyvar"
Environment="NO_PROXY=.$corpdom,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"
Environment="http_proxy=$httpproxyvar"
Environment="https_proy=$httpsproxyvar"
Environment="no_proxy=.$corpdom,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"
EOL
Added no_proxy in caps and lower to docker-compose containers and set in .env
Note: lower and CAPS should be used
environment:
- 'NO_PROXY=${NO_PROXY}'
- 'no_proxy=${NO_PROXY}'
NO_PROXY=.domain.org,127.0.0.0/8,172.16.0.0/12,icinga_icinga-net
I would prefer to append to the existing variable at least, but I tried the following and it made the variable no_proxy = ,icinga_icinga-net
NO_PROXY=$NO_PROXY,icinga_icinga-net
NO_PROXY=${NO_PROXY},icinga_icinga-net
Note: NO_PROXY was set on host via export
I still don't understand why it fails when using:
curl --noproxy -k -u root:c54854140704eafc https://172.30.0.4:5665/v1/objects/hosts
when I have no_proxy 172.16.0.0/12 which should equal 172.16.0.0 – 172.31.255.255 but doesn't work.
Update:
I tried setting no_proxy to the IP explicitly (no CIDR) and that worked, but it still failed w/ just container as host (no .icinga-net).
This is all related to this great post -->
https://about.gitlab.com/blog/2021/01/27/we-need-to-talk-no-proxy/

Answer 1

This is the best I can come up with, happy to reward better answers!
Docker Setup (Global):
#!/bin/bash
#Proxy
ActiveProxyVar=127.0.0.7
#Domain
corpdom=domain.org
#NoProxy
NOT_PROXY=127.0.0.0/8,172.16.0.0/12,192.168.0.0/16,10.0.0.0/8,.$corpdom
httpproxyvar=http://$ActiveProxyVar/
httpsproxyvar=http://$ActiveProxyVar/
mkdir ~/.docker
cat << EOL >~/.docker/config.json
{
"proxies":
{
"default":
{
"httpProxy": "$httpproxyvar",
"httpsProxy": "$httpsproxyvar",
"noProxy": "$NOT_PROXY"
}
}
}
EOL
mkdir -p /etc/systemd/system/docker.service.d
cat << EOL >/etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=$httpproxyvar"
Environment="HTTPS_PROXY=$httpsproxyvar"
Environment="NO_PROXY=$NOT_PROXY"
Environment="http_proxy=$httpproxyvar"
Environment="https_proy=$httpsproxyvar"
Environment="no_proxy=$NOT_PROXY"
EOL
systemctl daemon-reload
systemctl restart docker
#systemctl show --property Environment docker
#docker run hello-world
docker-compose.yaml:
environment:
- 'NO_PROXY=${NO_PROXY}'
- 'no_proxy=${NO_PROXY}'
.env:
--Basically, add docker-compose network then each container name...
NO_PROXY=127.0.0.0/8,172.16.0.0/12,192.168.0.0/16,10.0.0.0/8,.icinga_icinga-net,icinga2-api,icinga2-web,icinga2-db,icinga2-webdb,icinga2-redis,icinga2-directordb,icinga2-icingadb,icinga2-web_director