Connection problems from Docker Swarm services - docker

Question

We have a Docker Swarm with manager and several workers (in OpenStack) and encounter a problem that services cannot send emails using SMTP-SSL. Indeed from bash in-container this gets stuck:
# openssl s_client -debug -connect smtp.gmail.com:465 -crlf
CONNECTED(00000003)
write to 0x15bd2e0 [0x15bd360] (305 bytes => 305 (0x131))
0000 - 16 03 01 01 2c 01 00 01-28 03 03 50 0f 63 7c a4 ....,...(..P.c|.
0010 - 3f 96 d0 e4 e1 be 9a 0c-2b de 29 16 c9 54 b4 9d ?.......+.)..T..
0020 - 56 8d d7 76 f5 18 10 93-96 90 87 00 00 aa c0 30 V..v...........0
0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a5 00 a3 00 a1 .,.(.$..........
0040 - 00 9f 00 6b 00 6a 00 69-00 68 00 39 00 38 00 37 ...k.j.i.h.9.8.7
0050 - 00 36 00 88 00 87 00 86-00 85 c0 32 c0 2e c0 2a .6.........2...*
0060 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 2f .&.......=.5.../
0070 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a4 00 a2 00 a0 .+.'.#..........
0080 - 00 9e 00 67 00 40 00 3f-00 3e 00 33 00 32 00 31 ...g.#.?.>.3.2.1
0090 - 00 30 00 9a 00 99 00 98-00 97 00 45 00 44 00 43 .0.........E.D.C
00a0 - 00 42 c0 31 c0 2d c0 29-c0 25 c0 0e c0 04 00 9c .B.1.-.).%......
00b0 - 00 3c 00 2f 00 96 00 41-c0 11 c0 07 c0 0c c0 02 .<./...A........
00c0 - 00 05 00 04 c0 12 c0 08-00 16 00 13 00 10 00 0d ................
00d0 - c0 0d c0 03 00 0a 00 ff-01 00 00 55 00 0b 00 04 ...........U....
00e0 - 03 00 01 02 00 0a 00 1c-00 1a 00 17 00 19 00 1c ................
00f0 - 00 1b 00 18 00 1a 00 16-00 0e 00 0d 00 0b 00 0c ................
0100 - 00 09 00 0a 00 23 00 00-00 0d 00 20 00 1e 06 01 .....#..... ....
0110 - 06 02 06 03 05 01 05 02-05 03 04 01 04 02 04 03 ................
0120 - 03 01 03 02 03 03 02 01-02 02 02 03 00 0f 00 01 ................
0130 - 01 .
^C
Moreover, apt update/install nor pip install works:
# apt update
Hit:1 http://archive.ubuntu.com/ubuntu xenial InRelease
0% [Waiting for headers] [Waiting for headers]
pip install httpie
^CERROR: Operation cancelled by user
^CTraceback (most recent call last):
File "/usr/local/bin/pip", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/main.py", line 75, in main
return command.main(cmd_args)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 105, in main
return self._main(args)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 224, in _main
self.handle_pip_version_check(options)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 149, in handle_pip_version_check
pip_self_version_check(session, options)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/self_outdated_check.py", line 207, in pip_self_version_check
best_candidate = finder.find_best_candidate("pip").best_candidate
File "/usr/local/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 881, in find_best_candidate
candidates = self.find_all_candidates(project_name)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 825, in find_all_candidates
package_links = self.process_project_url(
File "/usr/local/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 790, in process_project_url
html_page = self._link_collector.fetch_page(project_url)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/index/collector.py", line 497, in fetch_page
return _get_html_page(location, session=self.session)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/index/collector.py", line 337, in _get_html_page
resp = _get_html_response(url, session=session)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/index/collector.py", line 126, in _get_html_response
resp = session.get(
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/requests/sessions.py", line 546, in get
return self.request('GET', url, **kwargs)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/network/session.py", line 405, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/cachecontrol/adapter.py", line 53, in send
resp = super(CacheControlAdapter, self).send(request, **kw)
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/urllib3/connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/urllib3/connectionpool.py", line 376, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/urllib3/connectionpool.py", line 994, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/urllib3/connection.py", line 386, in connect
self.sock = ssl_wrap_socket(
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/urllib3/util/ssl_.py", line 370, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/local/lib/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/usr/local/lib/python3.8/ssl.py", line 1040, in _create
self.do_handshake()
File "/usr/local/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
KeyboardInterrupt
(Terminated after significant wait time.)
Of course, everything works well on hosts (manager as well as workers)... For wget I have strange problem, that works for some and for some not:
# wget seznam.cz
--2020-04-30 06:14:34-- http://seznam.cz/
Resolving seznam.cz (seznam.cz)... 77.75.75.172, 77.75.75.176, 2a02:598:4444:1::1, ...
Connecting to seznam.cz (seznam.cz)|77.75.75.172|:80... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: https://www.seznam.cz/ [following]
--2020-04-30 06:14:34-- https://www.seznam.cz/
Resolving www.seznam.cz (www.seznam.cz)... 77.75.75.172, 77.75.74.172, 77.75.75.176, ...
Connecting to www.seznam.cz (www.seznam.cz)|77.75.75.172|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: 'index.html.1'
# wget google.com
--2020-04-30 06:15:10-- http://google.com/
Resolving google.com (google.com)... 216.58.201.110, 2a00:1450:4014:801::200e
Connecting to google.com (google.com)|216.58.201.110|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://www.google.com/ [following]
--2020-04-30 06:15:10-- http://www.google.com/
Resolving www.google.com (www.google.com)... 172.217.23.196, 2a00:1450:4014:80c::2004
Connecting to www.google.com (www.google.com)|172.217.23.196|:80... connected.
HTTP request sent, awaiting response...
Any ideas how to identify and fix the problem please? I am kinda out of ideas.
UPDATE: Now I suspect that the problem could be MTU (Docker default 1500 but ens3 has 1442 on all nodes) - need to figure out how to change it everywhere... But surprising I still DID NOT SOLVE this - adding --mtu, adjusting daemon.json, using --opt when creating networks, etc. still results in MTU 1500!

Answer 1

The problem was caused by MTU mismatch (1500 vs. 1442 used in OpenStack cloud).
I had to change /etc/docker/daemon.json, "break" my swarm, customize ingress network (docs) and docker_gwbridge (on each node before forming swarm again) (docs), remove the interfaces, restart Docker (it created interfaces again), and finally also change MTU on interfaces:
sudo ifconfig docker0 mtu 1442
sudo ifconfig docker_gwbridge mtu 1442
Because it seems that /etc/docker/daemon.json affects only interfaces in containers and not those on host.
If anyone knows a better solution, please let me know...

Answer 2

You can add a service to modify the mtu value After the docker.service
Create a script setdockermtu.sh
#!/bin/bash
sudo ifconfig docker0 mtu 9000
Create a service to run after docker.service
[Unit]
Description=Change docker0 default mtu
# When systemd stops or restarts the docker.service, the action is propagated to this unit
PartOf=docker.service
# Start this unit after the docker.service start
After=docker.service
[Service]
# The program will exit after running the script
Type=oneshot
# Execute the shell script
ExecStart=/usr/local/bin/setdockermtu.sh
# This service shall be considered active after start
RemainAfterExit=yes
[Install]
# This unit should start when docker.service is starting
WantedBy=docker.service
Copy it to /etc/systemd/system/setdockermtu.service
Make sure to add proper execute permission to both script and service
chmod a+x /usr/local/bin/setdockermtu.sh
chmod a+x /etc/systemd/system/setdockermtu.service
Reload changes and start service
sudo systemctl daemon-reload
sudo systemctl start setdockermtu.service
sudo systemctl enable setdockermtu.service