Linux enc28j60 driver always times out on transmit - network-programming

I've wired up an ENC28J60 module with an SPI interface to a Beaglebone Black development board running Linux 3.8, but I cannot get the device to TX (or RX) anything. Every attempt to TX times out and increments the TX errors.
Originally, the driver that shipped with the Linux kernel would not allow me to bring the link up. After a bunch of debugging, I found that if I waited some amount of time during board initialization, that I could get the link to come up. The patch to the driver was in the enc28j60_check_link_status function and was more or less the following.
unsigned long timeout = jiffies + 1*HZ;
while ((enc28j60_phy_read(priv, PHSTAT2) & PHSTAT2_LSTAT) == 0) {
if (time_after(jiffies, timeout)) {
if (netif_msg_drv(priv))
break;
}
schedule();
}
Now, I can reliably load the driver and configure the interface like so.
ifconfig rename3 down
ethtool -s rename3 msglvl 0xffff
ethtool -s rename3 speed 10 duplex full
ifconfig rename3 up
I've customized the ENC28J60's driver to do a bunch of additional printk while I've been searching for the root of the problem. I get the following after bringing the interface up.
Feb 1 00:22:48 arm kernel: [ 100.001942] enc28j60: HELLO WORLD
Feb 1 00:22:48 arm kernel: [ 100.002092] enc28j60: enc28j60_net_open() enter
Feb 1 00:22:48 arm kernel: [ 100.002119] enc28j60: enc28j60_hw_disable() enter
Feb 1 00:22:48 arm kernel: [ 100.002315] enc28j60: enc28j60_hw_init() enter
Feb 1 00:22:48 arm kernel: [ 100.002342] enc28j60: enc28j60_hw_init() - FullDuplex
Feb 1 00:22:48 arm kernel: [ 100.002366] enc28j60: enc28j60_soft_reset() enter
Feb 1 00:22:48 arm kernel: [ 100.006957] enc28j60: nolock_rxfifo_init() enter
Feb 1 00:22:48 arm kernel: [ 100.007117] enc28j60: erxrdpt_workaround() enter
Feb 1 00:22:48 arm kernel: [ 100.007373] enc28j60: nolock_txfifo_init() enter
Feb 1 00:22:48 arm kernel: [ 100.007807] enc28j60: chip RevID: 0x06
Feb 1 00:22:48 arm kernel: [ 100.009680] enc28j60: enc28j60_phy_write() enter
Feb 1 00:22:48 arm kernel: [ 100.010950] enc28j60: enc28j60_phy_write() enter
Feb 1 00:22:48 arm kernel: [ 100.011335] enc28j60: enc28j60_phy_write() enter
Feb 1 00:22:48 arm kernel: [ 100.014973] enc28j60 Hw initialized.
Feb 1 00:22:48 arm kernel: [ 100.014973] HwRevID: 0x06
Feb 1 00:22:48 arm kernel: [ 100.014973] Cntrl: ECON1 ECON2 ESTAT EIR EIE
Feb 1 00:22:48 arm kernel: [ 100.014973] 0x03 0x80 0x01 0x00 0x00
Feb 1 00:22:48 arm kernel: [ 100.014973] MAC : MACON1 MACON3 MACON4
Feb 1 00:22:48 arm kernel: [ 100.014973] 0x0d 0x33 0x00
Feb 1 00:22:48 arm kernel: [ 100.014973] Rx : ERXST ERXND ERXWRPT ERXRDPT ERXFCON EPKTCNT MAMXFL
Feb 1 00:22:48 arm kernel: [ 100.014973] 0x0000 0x19ff 0x0000 0x19ff 0xa1 0x00 0x05ee
Feb 1 00:22:48 arm kernel: [ 100.014973] Tx : ETXST ETXND MACLCON1 MACLCON2 MAPHSUP
Feb 1 00:22:48 arm kernel: [ 100.014973] 0x1a00 0x1fff 0x0f 0x37 0x10
Feb 1 00:22:48 arm kernel: [ 100.015073] enc28j60: enc28j60_set_hw_macaddr() enter
Feb 1 00:22:48 arm kernel: [ 100.015105] enc28j60: rename3: Setting MAC address to ee:31:66:60:5a:52
Feb 1 00:22:48 arm kernel: [ 100.015530] enc28j60: enc28j60_hw_enable() enter
Feb 1 00:22:48 arm kernel: [ 100.015556] enc28j60: enc28j60_hw_enable() enabling interrupts.
Feb 1 00:22:48 arm kernel: [ 100.015578] enc28j60: enc28j60_phy_write() enter
Feb 1 00:22:48 arm kernel: [ 100.017800] enc28j60: enc28j60_check_link_status() enter
Feb 1 00:22:48 arm kernel: [ 100.020726] enc28j60: enc28j60_check_link_status() PHSTAT1: 1800, PHSTAT2: 0200
Feb 1 00:22:48 arm kernel: [ 100.020760] enc28j60:enc28j60_check_link_status() polling PHSTAT2_LSTAT != 0
Feb 1 00:22:48 arm kernel: [ 100.069286] enc28j60:enc28j60_check_link_status() reg/link now 1536/1024
Feb 1 00:22:48 arm kernel: [ 100.069319] enc28j60:enc28j60_check_link_status() have link! setting carrier on
Feb 1 00:22:48 arm kernel: [ 100.069362] net rename3: link up - Full duplex
Feb 1 00:22:58 arm kernel: [ 109.989281] enc28j60: enc28j60_send_packet() enter
Feb 1 00:22:58 arm kernel: [ 109.989329] enc28j60: enc28j60_send_packet() enter
Feb 1 00:22:58 arm kernel: [ 109.989411] enc28j60: enc28j60_tx_work_handler() enter
Feb 1 00:22:58 arm kernel: [ 109.989437] enc28j60: enc28j60_hw_tx() enter
Feb 1 00:22:58 arm kernel: [ 109.989459] enc28j60: Tx Packet Len:70
Feb 1 00:22:58 arm kernel: [ 109.989481] enc28j60: enc28j60_hw_tx - packet len:70
Feb 1 00:22:58 arm kernel: [ 109.989517] pk data: 00000000: 33 33 00 00 00 02 ee 31 66 60 5a 52 86 dd 60 00 33.....1f`ZR..`.
Feb 1 00:22:58 arm kernel: [ 109.989548] pk data: 00000010: 00 00 00 10 3a ff fe 80 00 00 00 00 00 00 ec 31 ....:..........1
Feb 1 00:22:58 arm kernel: [ 109.989576] pk data: 00000020: 66 ff fe 60 5a 52 ff 02 00 00 00 00 00 00 00 00 f..`ZR..........
Feb 1 00:22:58 arm kernel: [ 109.989605] pk data: 00000030: 00 00 00 00 00 02 85 00 21 65 00 00 00 00 01 01 ........!e......
Feb 1 00:22:58 arm kernel: [ 109.989633] pk data: 00000040: ee 31 66 60 5a 52 .1f`ZR
Feb 1 00:22:58 arm kernel: [ 109.989655] enc28j60: enc28j60_packet_write() enter
Feb 1 00:22:58 arm kernel: [ 109.990379] enc28j60: enc28j60_packet_write() after control byte ERWPT:0x1a01
Feb 1 00:22:58 arm kernel: [ 109.990406] enc28j60: spi_write_buf() enter
Feb 1 00:22:58 arm kernel: [ 109.990571] enc28j60: enc28j60_packet_write() spi_write returns 0
Feb 1 00:22:58 arm kernel: [ 109.990761] enc28j60: enc28j60_packet_write() after write packet ERWPT:0x1a47, len=70
Feb 1 00:23:18 arm kernel: [ 129.979247] enc28j60: enc28j60_tx_timeout() enter
Feb 1 00:23:18 arm kernel: [ 129.979313] net rename3: enc28j60 tx timeout
I've narrowed it down with additional print statements to a call to spi_sync in spi_read_buf. Does anyone with knowledge of Linux network drivers or Microchip's ENC28J60 know what I should be looking into to resolve this problem?

Related

docker compose failing (syslog ERROR:Unknown interface index 335 seen even after reload)

Thank you for checking this.
Ubuntu 18 server on AWS EC2, docker-compose up was running just fine, suddenly it stopped building after a reboot. Not sure what changed.
Here is the docker-compose.yml
version: '2'
services:
web:
build: .
restart: "no"
command: gulp serve --max_new_space_size=8192 --max-old-space-size=8192 -LLLL
env_file:
- .env
volumes:
- .:/app/code
ports:
- "8050:8000"
- "8005:8005"
- "8888:8888"
Here is the Dockerfile
FROM node:6.10.3
RUN mkdir /app
RUN mkdir /app/code
WORKDIR /app
# Install JavaScript requirements
COPY package.json /app/
COPY package-lock.json /app/
RUN npm install -d
RUN npm rebuild node-sass
# Link gulp
RUN ln -s /app/node_modules/.bin/gulp /usr/bin/gulp
COPY . /app/code/
WORKDIR /app/code
RUN export NODE_OPTIONS="--max-old-space-size=8192"
# Build webpack files
RUN gulp build
EXPOSE 8000
CMD gulp serve
I see some errors in the syslog, not sure if it is related.
Jun 2 15:25:24 ip-10-0-1-194 kernel: [52500.188965] docker0: port 1(veth638f141) entered blocking state
Jun 2 15:25:24 ip-10-0-1-194 kernel: [52500.188968] docker0: port 1(veth638f141) entered disabled state
Jun 2 15:25:24 ip-10-0-1-194 kernel: [52500.189101] device veth638f141 entered promiscuous mode
Jun 2 15:25:24 ip-10-0-1-194 systemd-networkd[734]: veth638f141: Link UP
Jun 2 15:25:24 ip-10-0-1-194 networkd-dispatcher[947]: WARNING:Unknown index 338 seen, reloading interface list
Jun 2 15:25:24 ip-10-0-1-194 systemd-udevd[5940]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 2 15:25:24 ip-10-0-1-194 systemd-udevd[5940]: Could not generate persistent MAC address for veth3a08f68: No such file or directory
Jun 2 15:25:24 ip-10-0-1-194 systemd-udevd[5941]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 2 15:25:24 ip-10-0-1-194 systemd-udevd[5941]: Could not generate persistent MAC address for veth638f141: No such file or directory
Jun 2 15:25:24 ip-10-0-1-194 containerd[993]: time="2021-06-02T15:25:24.995557031Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/7712e133ca3de4a4d407341b7a51428e984c4bcbf2311e27ffbd43cbff56ef44 pid=6001
Jun 2 15:25:25 ip-10-0-1-194 kernel: [52500.489989] eth0: renamed from veth3a08f68
Jun 2 15:25:25 ip-10-0-1-194 systemd-networkd[734]: veth638f141: Gained carrier
Jun 2 15:25:25 ip-10-0-1-194 systemd-networkd[734]: docker0: Gained carrier
Jun 2 15:25:25 ip-10-0-1-194 kernel: [52500.509809] IPv6: ADDRCONF(NETDEV_CHANGE): veth638f141: link becomes ready
Jun 2 15:25:25 ip-10-0-1-194 kernel: [52500.509869] docker0: port 1(veth638f141) entered blocking state
Jun 2 15:25:25 ip-10-0-1-194 kernel: [52500.509870] docker0: port 1(veth638f141) entered forwarding state
Jun 2 15:25:26 ip-10-0-1-194 systemd-networkd[734]: veth638f141: Gained IPv6LL
Jun 2 15:25:27 ip-10-0-1-194 containerd[993]: time="2021-06-02T15:25:27.979112078Z" level=info msg="shim disconnected" id=7712e133ca3de4a4d407341b7a51428e984c4bcbf2311e27ffbd43cbff56ef44
Jun 2 15:25:27 ip-10-0-1-194 dockerd[1010]: time="2021-06-02T15:25:27.979239439Z" level=info msg="ignoring event" container=7712e133ca3de4a4d407341b7a51428e984c4bcbf2311e27ffbd43cbff56ef44 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 2 15:25:28 ip-10-0-1-194 kernel: [52503.292104] docker0: port 1(veth638f141) entered disabled state
Jun 2 15:25:28 ip-10-0-1-194 kernel: [52503.292214] veth3a08f68: renamed from eth0
Jun 2 15:25:28 ip-10-0-1-194 systemd-networkd[734]: veth638f141: Lost carrier
Jun 2 15:25:28 ip-10-0-1-194 systemd-udevd[6146]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 2 15:25:28 ip-10-0-1-194 systemd-networkd[734]: veth638f141: Link DOWN
Jun 2 15:25:28 ip-10-0-1-194 kernel: [52503.350623] docker0: port 1(veth638f141) entered disabled state
Jun 2 15:25:28 ip-10-0-1-194 kernel: [52503.353895] device veth638f141 left promiscuous mode
Jun 2 15:25:28 ip-10-0-1-194 kernel: [52503.353912] docker0: port 1(veth638f141) entered disabled state
Jun 2 15:25:28 ip-10-0-1-194 networkd-dispatcher[947]: WARNING:Unknown index 337 seen, reloading interface list
Jun 2 15:25:28 ip-10-0-1-194 networkd-dispatcher[947]: **ERROR:Unknown interface index 337 seen even after reload**
Jun 2 15:25:29 ip-10-0-1-194 systemd-networkd[734]: docker0: Lost carrier

Possible to run multiple squid containers on a single host?

I am trying to run multiple squid containers whose configs are built at container run time. Each container needs to route traffic independently from the other. Aside from where traffic is forwarded on, the configs are the same.
I can get a single squid container running and doing what I need it to with no problems.
docker run -v /var/log/squid:/var/log/squid -p 3133-3138:3133-3138 my_images/squid_test:version1.0
Trying to run a second container with:
docker run -v /var/log/squid:/var/log/squid -p 4133-4138:3133-3138 my_images/squid_test:version1.0
This instantly spits out: Aborted (core dumped)
I have one other container running on port 9000 but thats it.
This is a syslog dump from the host at the time the second container launch is attempted
Jun 18 04:45:17 dockerdevr1 kernel: [84821.356170] docker0: port 3(veth89ab0c1) entered blocking state
Jun 18 04:45:17 dockerdevr1 kernel: [84821.356172] docker0: port 3(veth89ab0c1) entered disabled state
Jun 18 04:45:17 dockerdevr1 kernel: [84821.356209] device veth89ab0c1 entered promiscuous mode
Jun 18 04:45:17 dockerdevr1 kernel: [84821.356252] IPv6: ADDRCONF(NETDEV_UP): veth89ab0c1: link is not ready
Jun 18 04:45:17 dockerdevr1 systemd-networkd[765]: veth89ab0c1: Link UP
Jun 18 04:45:17 dockerdevr1 networkd-dispatcher[1048]: WARNING:Unknown index 421 seen, reloading interface list
Jun 18 04:45:17 dockerdevr1 systemd-udevd[25899]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 18 04:45:17 dockerdevr1 systemd-udevd[25900]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 18 04:45:17 dockerdevr1 systemd-udevd[25899]: Could not generate persistent MAC address for vethb0dffb8: No such file or directory
Jun 18 04:45:17 dockerdevr1 systemd-udevd[25900]: Could not generate persistent MAC address for veth89ab0c1: No such file or directory
Jun 18 04:45:17 dockerdevr1 containerd[1119]: time="2020-06-18T04:45:17.567627817Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/85f0acae4a948ed16b3b29988291b5df3d052b10d1965f1198745966e63c3732/shim.sock" debug=false pid=25920
Jun 18 04:45:17 dockerdevr1 kernel: [84821.841905] eth0: renamed from vethb0dffb8
Jun 18 04:45:17 dockerdevr1 kernel: [84821.858172] IPv6: ADDRCONF(NETDEV_CHANGE): veth89ab0c1: link becomes ready
Jun 18 04:45:17 dockerdevr1 kernel: [84821.858263] docker0: port 3(veth89ab0c1) entered blocking state
Jun 18 04:45:17 dockerdevr1 kernel: [84821.858265] docker0: port 3(veth89ab0c1) entered forwarding state
Jun 18 04:45:17 dockerdevr1 systemd-networkd[765]: veth89ab0c1: Gained carrier
Jun 18 04:45:19 dockerdevr1 systemd-networkd[765]: veth89ab0c1: Gained IPv6LL
Jun 18 04:45:19 dockerdevr1 containerd[1119]: time="2020-06-18T04:45:19.221654620Z" level=info msg="shim reaped" id=85f0acae4a948ed16b3b29988291b5df3d052b10d1965f1198745966e63c3732
Jun 18 04:45:19 dockerdevr1 dockerd[1171]: time="2020-06-18T04:45:19.232623257Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 18 04:45:19 dockerdevr1 systemd-networkd[765]: veth89ab0c1: Lost carrier
Jun 18 04:45:19 dockerdevr1 kernel: [84823.251203] docker0: port 3(veth89ab0c1) entered disabled state
Jun 18 04:45:19 dockerdevr1 kernel: [84823.254402] vethb0dffb8: renamed from eth0
Jun 18 04:45:19 dockerdevr1 systemd-networkd[765]: veth89ab0c1: Link DOWN
Jun 18 04:45:19 dockerdevr1 kernel: [84823.293507] docker0: port 3(veth89ab0c1) entered disabled state
Jun 18 04:45:19 dockerdevr1 kernel: [84823.294577] device veth89ab0c1 left promiscuous mode
Jun 18 04:45:19 dockerdevr1 kernel: [84823.294580] docker0: port 3(veth89ab0c1) entered disabled state
Jun 18 04:45:19 dockerdevr1 networkd-dispatcher[1048]: WARNING:Unknown index 420 seen, reloading interface list
Jun 18 04:45:19 dockerdevr1 networkd-dispatcher[1048]: ERROR:Unknown interface index 420 seen even after reload
Jun 18 04:45:19 dockerdevr1 systemd-udevd[26041]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 18 04:45:19 dockerdevr1 systemd-udevd[26041]: link_config: could not get ethtool features for vethb0dffb8
Jun 18 04:45:19 dockerdevr1 systemd-udevd[26041]: Could not set offload features of vethb0dffb8: No such device
Has anyone tried something similar to this? I know I can get multiple nginx containers running on different ports. Any insight would be greatly appreciated!

Docker on Ubuntu doesn't connect to localhost

I have been going for hours trying to understand why docker just so happen to doesn't work on my machine. I am using Ubuntu 18.01 Xfce. Have installed docker using the official site and tried to test run an image using docker container run -it -p 8000:80 nginx command. The first it ran ok, but only I try another time the localhost goes into an endless loop of loading. It only works after I restart docker and then only the first time. I also tried editing docker.service file into:
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// -H tcp://127.0.0.1:2375
as instructed on the https://docs.docker.com/install/linux/linux-postinstall/ again no change.
Is there a solution to this problem? Is it and OS problem? Some kind of conflict problem? If so, is there a solution?
UPDATE:
This what ip addr show docker0 command is showing when container is running:
5: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:4e:4b:8e:af brd ff:ff:ff:ff:ff:ff
inet6 fe80::42:4eff:fe4b:8eaf/64 scope link
valid_lft forever preferred_lft forever
UPDATE 2:
By entering the commen sudo docker run -t -i nginx /bin/bash and reading the tail -f /var/log/syslog I have the following lines:
Feb 5 11:09:43 unkn0wn27-X550VX dockerd[8188]: time="2020-02-05T11:09:43.010897917+02:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 5 11:09:43 unkn0wn27-X550VX kernel: [ 1992.539911] docker0: port 2(vethb213531) entered disabled state
Feb 5 11:09:43 unkn0wn27-X550VX systemd-networkd[415]: vethb213531: Lost carrier
Feb 5 11:09:43 unkn0wn27-X550VX kernel: [ 1992.541590] veth4234a1c: renamed from eth0
Feb 5 11:09:43 unkn0wn27-X550VX systemd-timesyncd[994]: Network configuration changed, trying to establish connection.
Feb 5 11:09:43 unkn0wn27-X550VX systemd-udevd[9744]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb 5 11:09:43 unkn0wn27-X550VX systemd-timesyncd[994]: Synchronized to time server 194.40.240.12:123 (194.40.240.12).
Feb 5 11:09:43 unkn0wn27-X550VX NetworkManager[1280]: <info> [1580893783.1724] manager: (veth4234a1c): new Veth device (/org/freedesktop/NetworkManager/Devices/43)
Feb 5 11:09:43 unkn0wn27-X550VX kernel: [ 1992.641548] audit: type=1107 audit(1580893783.171:310): pid=1164 uid=103 auid=4294967295 ses=4294967295 msg='apparmor="DENIED" operation="dbus_signal" bus="system" path="/org/freedesktop/NetworkManager" interface="org.freedesktop.NetworkManager" member="DeviceAdded" name=":1.13" mask="receive" pid=3643 label="snap.telegram-desktop.telegram-desktop" peer_pid=1280 peer_label="unconfined"
Feb 5 11:09:43 unkn0wn27-X550VX kernel: [ 1992.641548] exe="/usr/bin/dbus-daemon" sauid=103 hostname=? addr=? terminal=?'
Feb 5 11:09:43 unkn0wn27-X550VX kernel: [ 1992.644507] audit: type=1107 audit(1580893783.171:311): pid=1164 uid=103 auid=4294967295 ses=4294967295 msg='apparmor="DENIED" operation="dbus_signal" bus="system" path="/org/freedesktop/NetworkManager" interface="org.freedesktop.NetworkManager" member="PropertiesChanged" name=":1.13" mask="receive" pid=3643 label="snap.telegram-desktop.telegram-desktop" peer_pid=1280 peer_label="unconfined"
Feb 5 11:09:43 unkn0wn27-X550VX kernel: [ 1992.644507] exe="/usr/bin/dbus-daemon" sauid=103 hostname=? addr=? terminal=?'
Feb 5 11:09:43 unkn0wn27-X550VX NetworkManager[1280]: <info> [1580893783.1829] devices added (path: /sys/devices/virtual/net/veth4234a1c, iface: veth4234a1c)
Feb 5 11:09:43 unkn0wn27-X550VX NetworkManager[1280]: <info> [1580893783.1830] device added (path: /sys/devices/virtual/net/veth4234a1c, iface: veth4234a1c): no ifupdown configuration found.
Feb 5 11:09:43 unkn0wn27-X550VX kernel: [ 1992.653532] IPv6: ADDRCONF(NETDEV_CHANGE): veth4234a1c: link becomes ready
Feb 5 11:09:43 unkn0wn27-X550VX kernel: [ 1992.653779] docker0: port 2(vethb213531) entered blocking state
Feb 5 11:09:43 unkn0wn27-X550VX kernel: [ 1992.653784] docker0: port 2(vethb213531) entered forwarding state
Feb 5 11:09:43 unkn0wn27-X550VX networkd-dispatcher[1265]: WARNING:Unknown index 30 seen, reloading interface list
Feb 5 11:09:43 unkn0wn27-X550VX systemd-timesyncd[994]: Network configuration changed, trying to establish connection.
Feb 5 11:09:43 unkn0wn27-X550VX NetworkManager[1280]: <info> [1580893783.1911] device (veth4234a1c): carrier: link connected
Feb 5 11:09:43 unkn0wn27-X550VX NetworkManager[1280]: <info> [1580893783.1934] device (vethb213531): carrier: link connected
Feb 5 11:09:43 unkn0wn27-X550VX systemd-networkd[415]: veth4234a1c: Gained carrier
Feb 5 11:09:43 unkn0wn27-X550VX systemd-networkd[415]: vethb213531: Gained carrier
Feb 5 11:09:43 unkn0wn27-X550VX systemd-timesyncd[994]: Synchronized to time server 194.40.240.12:123 (194.40.240.12).
Feb 5 11:09:43 unkn0wn27-X550VX avahi-daemon[1159]: Interface vethb213531.IPv6 no longer relevant for mDNS.
Feb 5 11:09:43 unkn0wn27-X550VX avahi-daemon[1159]: Leaving mDNS multicast group on interface vethb213531.IPv6 with address fe80::c0e3:82ff:febc:a5c4.
Feb 5 11:09:43 unkn0wn27-X550VX kernel: [ 1992.746817] docker0: port 2(vethb213531) entered disabled state
Feb 5 11:09:43 unkn0wn27-X550VX kernel: [ 1992.753883] device vethb213531 left promiscuous mode
Feb 5 11:09:43 unkn0wn27-X550VX kernel: [ 1992.753893] docker0: port 2(vethb213531) entered disabled state
Feb 5 11:09:43 unkn0wn27-X550VX avahi-daemon[1159]: Withdrawing address record for fe80::c0e3:82ff:febc:a5c4 on vethb213531.
Feb 5 11:09:43 unkn0wn27-X550VX NetworkManager[1280]: <info> [1580893783.3453] devices removed (path: /sys/devices/virtual/net/veth4234a1c, iface: veth4234a1c)
Feb 5 11:09:43 unkn0wn27-X550VX NetworkManager[1280]: <info> [1580893783.3457] devices removed (path: /sys/devices/virtual/net/vethb213531, iface: vethb213531)
Feb 5 11:09:43 unkn0wn27-X550VX kernel: [ 1992.816316] audit: type=1107 audit(1580893783.343:312): pid=1164 uid=103 auid=4294967295 ses=4294967295 msg='apparmor="DENIED" operation="dbus_signal" bus="system" path="/org/freedesktop/NetworkManager" interface="org.freedesktop.NetworkManager" member="DeviceRemoved" name=":1.13" mask="receive" pid=3643 label="snap.telegram-desktop.telegram-desktop" peer_pid=1280 peer_label="unconfined"
Feb 5 11:09:43 unkn0wn27-X550VX kernel: [ 1992.816316] exe="/usr/bin/dbus-daemon" sauid=103 hostname=? addr=? terminal=?'
Feb 5 11:09:43 unkn0wn27-X550VX NetworkManager[1280]: <info> [1580893783.3528] device (vethb213531): released from master device docker0
Feb 5 11:09:43 unkn0wn27-X550VX systemd-networkd[415]: veth4234a1c: Lost carrier
Feb 5 11:09:43 unkn0wn27-X550VX systemd-timesyncd[994]: Network configuration changed, trying to establish connection.
Feb 5 11:09:43 unkn0wn27-X550VX systemd-networkd[415]: veth4234a1c: Removing non-existent address: fe80::42:acff:fe11:3/64 (valid forever), ignoring
Feb 5 11:09:43 unkn0wn27-X550VX systemd-networkd[415]: vethb213531: Lost carrier
Feb 5 11:09:43 unkn0wn27-X550VX systemd-timesyncd[994]: Synchronized to time server 194.40.240.12:123 (194.40.240.12).
Feb 5 11:09:43 unkn0wn27-X550VX dockerd[8188]: time="2020-02-05T11:09:43.530168634+02:00" level=warning msg="8f0463438568dab68c318d3fb928d800b9ca6ec99a918bb06bf8aea4886efa48 cleanup: failed to unmount IPC: umount /var/lib/docker/containers/8f0463438568dab68c318d3fb928d800b9ca6ec99a918bb06bf8aea4886efa48/mounts/shm, flags: 0x2: no such file or directory"
I only spotted these lines:
Feb 5 11:09:43 unkn0wn27-X550VX kernel: [ 1992.746817] docker0: port 2(vethb213531) entered disabled state
Feb 5 11:09:43 unkn0wn27-X550VX kernel: [ 1992.753883] device vethb213531 left promiscuous mode
Feb 5 11:09:43 unkn0wn27-X550VX kernel: [ 1992.753893] docker0: port 2(vethb213531) entered disabled state
And this line:
Feb 5 11:09:43 unkn0wn27-X550VX NetworkManager[1280]: <info> [1580893783.3528] device (vethb213531): released from master device docker0
Not sure if that helps my case.
UPDATE 3:
A work around solution is to constantly write the command sudo ip addr add 172.17.0.1/16 dev docker0.
The problem was that Docker not keeping it's IPV4 address.
All I had to do was open /etc/systemd/network/mynet.network, and add these 2 lines:
[Match]
Name=docker0
[Link]
Unmanaged=yes
Restart systemctl restart systemd-networkd and systemctl restart docker.
All credits for this solution found here: https://vadosware.io/post/a-reliable-fix-to-docker-not-keeping-its-ipv4-address-on-arch/

Unsuccessful build on gitlab runner

Off recently we are facing the below issue while performing a CI/CD build from gitlab runner.
Below is the log snippet from /var/log/syslog.
pr 22 03:02:04 cirunner dockerd[1103]: time="2019-04-22T03:02:04.136857571Z" level=error msg="Handler for DELETE /v1.18/containers/runner-301e5f4d-project-786-concurrent-0-build-4 returned error: No such container: runner-301e5f4d-project-786-concurrent-0-build-4"
Apr 22 03:02:04 cirunner kernel: [1616845.656927] aufs au_opts_verify:1597:dockerd[1568]: dirperm1 breaks the protection by the permission bits on the lower branch
Apr 22 03:02:04 cirunner kernel: [1616846.186616] aufs au_opts_verify:1597:dockerd[1568]: dirperm1 breaks the protection by the permission bits on the lower branch
Apr 22 03:02:05 cirunner kernel: [1616846.383784] aufs au_opts_verify:1597:dockerd[1568]: dirperm1 breaks the protection by the permission bits on the lower branch
Apr 22 03:02:05 cirunner systemd-udevd[1187]: Could not generate persistent MAC address for veth0675b93: No such file or directory
Apr 22 03:02:05 cirunner kernel: [1616846.385245] device veth8b64bcd entered promiscuous mode
Apr 22 03:02:05 cirunner kernel: [1616846.385299] IPv6: ADDRCONF(NETDEV_UP): veth8b64bcd: link is not ready
Apr 22 03:02:05 cirunner systemd-udevd[1188]: Could not generate persistent MAC address for veth8b64bcd: No such file or directory
Apr 22 03:02:05 cirunner kernel: [1616846.788755] eth0: renamed from veth0675b93
Apr 22 03:02:05 cirunner kernel: [1616846.804716] IPv6: ADDRCONF(NETDEV_CHANGE): veth8b64bcd: link becomes ready
Apr 22 03:02:05 cirunner kernel: [1616846.804739] docker0: port 3(veth8b64bcd) entered forwarding state
Apr 22 03:02:05 cirunner kernel: [1616846.804747] docker0: port 3(veth8b64bcd) entered forwarding state
Apr 22 03:02:20 cirunner kernel: [1616861.819201] docker0: port 3(veth8b64bcd) entered forwarding state
Apr 22 03:37:13 cirunner dockerd[1103]: time="2019-04-22T03:37:13.298195303Z" level=error msg="Handler for GET
/v1.18/containers/6f6b71442b5bbc70f980cd05272c8f05d514735f39e9b73b52a094a0e87db475/json returned error: No such container: 6f6b71442b5bbc70f980cd05272c8f05d514735f39e9b73b52a094a0e87db475"
Could you please help me out what exactly is the issue and how can to trouble shoot.
Let me know if you require additional details from my side.

Docker HAProxy SSL termination with Letsencrypt

I currently have a docker setup working with haproxy as a load balancer directing traffic to containers running my web app. I'm trying to add SSL termination to HAProxy and have run into some trouble. When I add DEFAULT_SSL_CERT as an environment variable to my haproxy container I get these errors:
Mar 20 20:15:03 escapes-artist kernel: [3804709.167813] aufs au_opts_verify:1597:dockerd[1595]: dirperm1 breaks the protection by the permission bits on the lower branch
Mar 20 20:15:03 escapes-artist kernel: [3804709.213993] aufs au_opts_verify:1597:dockerd[1595]: dirperm1 breaks the protection by the permission bits on the lower branch
Mar 20 20:15:04 escapes-artist kernel: [3804709.674840] aufs au_opts_verify:1597:dockerd[1595]: dirperm1 breaks the protection by the permission bits on the lower branch
Mar 20 20:15:04 escapes-artist kernel: [3804709.688631] device vethebd7d1d entered promiscuous mode
Mar 20 20:15:04 escapes-artist kernel: [3804709.688767] IPv6: ADDRCONF(NETDEV_UP): vethebd7d1d: link is not ready
Mar 20 20:15:04 escapes-artist systemd-udevd: Could not generate persistent MAC address for veth5c0585c: No such file or directory
Mar 20 20:15:04 escapes-artist systemd-udevd: Could not generate persistent MAC address for vethebd7d1d: No such file or directory
Mar 20 20:15:04 escapes-artist dockerd: time="2017-03-21T02:15:04.671620998Z" level=warning msg="Your kernel does not support swap memory limit."
Mar 20 20:15:04 escapes-artist dockerd: time="2017-03-21T02:15:04.672345380Z" level=warning msg="Your kernel does not support cgroup rt period"
Mar 20 20:15:04 escapes-artist dockerd: time="2017-03-21T02:15:04.672732724Z" level=warning msg="Your kernel does not support cgroup rt runtime"
Mar 20 20:15:04 escapes-artist dockerd: time="2017-03-21T02:15:04Z" level=info msg="Firewalld running: false"
Mar 20 20:15:05 escapes-artist kernel: [3804710.392546] eth0: renamed from veth5c0585c
Mar 20 20:15:05 escapes-artist kernel: [3804710.395273] IPv6: ADDRCONF(NETDEV_CHANGE): vethebd7d1d: link becomes ready
Mar 20 20:15:05 escapes-artist kernel: [3804710.395303] br-5c6735a37ece: port 3(vethebd7d1d) entered forwarding state
Mar 20 20:15:05 escapes-artist kernel: [3804710.395313] br-5c6735a37ece: port 3(vethebd7d1d) entered forwarding state
Mar 20 20:15:05 escapes-artist kernel: [3804711.072047] br-5c6735a37ece: port 2(vethbaf33bd) entered forwarding state
Mar 20 20:15:08 escapes-artist kernel: [3804713.819317] haproxy[29684]: segfault at 7f560000003b ip 00007f56f6ac74bb sp 00007ffe45011290 error 4 in libcrypto.so.1.0.0[7f56f69ce000+3f3000]
Mar 20 20:15:11 escapes-artist sshd: Received disconnect from 122.194.229.7 port 21903:11: [preauth]
Mar 20 20:15:11 escapes-artist sshd: Disconnected from 122.194.229.7 port 21903 [preauth]
Mar 20 20:15:13 escapes-artist kernel: [3804718.789238] haproxy[29686]: segfault at 7fbb0000003b ip 00007fbb747b74bb sp 00007ffc944fcc10 error 4 in libcrypto.so.1.0.0[7fbb746be000+3f3000]
Mar 20 20:15:17 escapes-artist kernel: [3804722.944073] br-5c6735a37ece: port 1(veth610d1f4) entered forwarding state
Mar 20 20:15:18 escapes-artist kernel: [3804723.790663] haproxy[29688]: segfault at 7ff10000003b ip 00007ff1ad6004bb sp 00007fffa6f03cb0 error 4 in libcrypto.so.1.0.0[7ff1ad507000+3f3000]
Mar 20 20:15:20 escapes-artist kernel: [3804725.408060] br-5c6735a37ece: port 3(vethebd7d1d) entered forwarding state
Mar 20 20:15:23 escapes-artist kernel: [3804728.792134] haproxy[29690]: segfault at 7f130000003b ip 00007f13210c54bb sp 00007ffcbe3f7670 error 4 in libcrypto.so.1.0.0[7f1320fcc000+3f3000]
Mar 20 20:15:28 escapes-artist kernel: [3804733.823940] haproxy[29692]: segfault at 7f500000003b ip 00007f500b9d94bb sp 00007ffe6d044f10 error 4 in libcrypto.so.1.0.0[7f500b8e0000+3f3000]
Mar 20 20:15:33 escapes-artist kernel: [3804738.780797] haproxy[29694]: segfault at 7f000000003b ip 00007f00310124bb sp 00007fffd6e979b0 error 4 in libcrypto.so.1.0.0[7f0030f19000+3f3000]
Does anyone know how to fix this? I've experimented for hours trying different formats for the cert file, environment variables, etc. and can't seem to figure anything out. Here is the docker-compose.yml file I'm using:
version: '2'
services:
db:
image: mysql
restart: always
environment:
MYSQL_ROOT_PASSWORD: password
MYSQL_DATABASE: docker
MYSQL_USER: admin
MYSQL_PASSWORD: password
volumes:
- /storage/docker/mysql-datadir:/var/lib/mysql
ports:
- 3306:3306
web:
image: myimage
restart: always
depends_on:
- db
volumes:
- /home/docker/persistent/media/:/home/docker/code/media/
lb:
image: dockercloud/haproxy
links:
- web
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /etc/haproxy/certs:/certs
environment:
STATS_AUTH: admin:password
RSYSLOG_DESTINATION: logs5.papertrailapp.com:41747
DEFAULT_SSL_CERT: (I've tried both pasting cert here directly and a path to cert)
ports:
- 80:80
- 443:443
- 1936:1936
I have Letsencrypt setup on the host machine to autorenew. The cert that I've been trying to use is a combination of the privkey.pem and fullchian.pem. I've tried concatenating them, using awk 1 ORS='\\n' like the dockercloud/haproxy docs suggest, and just about every other configuration I can think of. Any help would be greatly appreciated.
Also, if I use CERT_FOLDER: /certs/ instead of DEFAULT_SSL_CERT and have my certificate stored in /certs/cert0.pem I get this error instead...
Mar 20 21:19:38 escapes-artist dockerd: time="2017-03-21T03:19:38.840340234Z" level=error msg="containerd: deleting container" error="exit status 1: \"container ce6c0b6df31419691b6593be6744d01c8ccecf5f38851106aa4bb8fac915a63a does not exist\\none or more of the container deletions failed\\n\""
Mar 20 21:19:38 escapes-artist kernel: [3808584.302038] br-5c6735a37ece: port 3(veth8b1ea8e) entered disabled state
Mar 20 21:19:38 escapes-artist kernel: [3808584.302192] veth0bcd06c: renamed from eth0
Mar 20 21:19:38 escapes-artist kernel: [3808584.320863] br-5c6735a37ece: port 3(veth8b1ea8e) entered disabled state
Mar 20 21:19:38 escapes-artist kernel: [3808584.321869] device veth8b1ea8e left promiscuous mode
Mar 20 21:19:38 escapes-artist kernel: [3808584.321874] br-5c6735a37ece: port 3(veth8b1ea8e) entered disabled state
Mar 20 21:19:39 escapes-artist dockerd: time="2017-03-21T03:19:39.055316431Z" level=error msg="Handler for GET /v1.25/exec/c79e3c9b77f0c84d849cc641a425950d55fcbb22bf566922d3fd12e6a0e12e07/json returned error: Container ce6c0b6df31419691b6593be6744d01c8ccecf5f38851106aa4bb8fac915a63a is not running: Exited (0) Less than a second ago"
Mar 20 21:19:39 escapes-artist kernel: [3808584.964578] aufs au_opts_verify:1597:dockerd[23058]: dirperm1 breaks the protection by the permission bits on the lower branch
Mar 20 21:19:39 escapes-artist kernel: [3808585.005699] aufs au_opts_verify:1597:dockerd[23058]: dirperm1 breaks the protection by the permission bits on the lower branch
Mar 20 21:19:40 escapes-artist kernel: [3808585.489799] aufs au_opts_verify:1597:dockerd[1595]: dirperm1 breaks the protection by the permission bits on the lower branch
Mar 20 21:19:40 escapes-artist kernel: [3808585.500609] device veth24d6316 entered promiscuous mode
Mar 20 21:19:40 escapes-artist systemd-udevd: Could not generate persistent MAC address for veth24d6316: No such file or directory
Mar 20 21:19:40 escapes-artist kernel: [3808585.505055] IPv6: ADDRCONF(NETDEV_UP): veth24d6316: link is not ready
Mar 20 21:19:40 escapes-artist systemd-udevd: Could not generate persistent MAC address for vethedaad7c: No such file or directory
Mar 20 21:19:40 escapes-artist dockerd: time="2017-03-21T03:19:40.259076690Z" level=warning msg="Your kernel does not support swap memory limit."
Mar 20 21:19:40 escapes-artist dockerd: time="2017-03-21T03:19:40.260183880Z" level=warning msg="Your kernel does not support cgroup rt period"
Mar 20 21:19:40 escapes-artist dockerd: time="2017-03-21T03:19:40.260663645Z" level=warning msg="Your kernel does not support cgroup rt runtime"
Mar 20 21:19:40 escapes-artist dockerd: time="2017-03-21T03:19:40Z" level=info msg="Firewalld running: false"
Mar 20 21:19:40 escapes-artist kernel: [3808585.904671] eth0: renamed from vethedaad7c
Mar 20 21:19:40 escapes-artist kernel: [3808585.918744] IPv6: ADDRCONF(NETDEV_CHANGE): veth24d6316: link becomes ready
Mar 20 21:19:40 escapes-artist kernel: [3808585.919040] br-5c6735a37ece: port 3(veth24d6316) entered forwarding state
Mar 20 21:19:40 escapes-artist kernel: [3808585.919058] br-5c6735a37ece: port 3(veth24d6316) entered forwarding state
Mar 20 21:19:44 escapes-artist kernel: [3808589.585674] haproxy[32235]: segfault at 341 ip 0000000000000341 sp 00007ffe732fe5b8 error 14 in haproxy[55f6998b1000+d1000]
Mar 20 21:19:49 escapes-artist kernel: [3808594.704226] haproxy[32237]: segfault at 341 ip 0000000000000341 sp 00007ffcb4d1aa08 error 14 in haproxy[563827d10000+d1000]
Mar 20 21:19:54 escapes-artist kernel: [3808599.669540] haproxy[32239]: segfault at 341 ip 0000000000000341 sp 00007ffd1e8bb1b8 error 14 in haproxy[562d926fa000+d1000]
Mar 20 21:19:55 escapes-artist kernel: [3808600.928110] br-5c6735a37ece: port 3(veth24d6316) entered forwarding state
Mar 20 21:19:59 escapes-artist kernel: [3808604.602704] haproxy[32241]: segfault at 341 ip 0000000000000341 sp 00007fff142d0898 error 14 in haproxy[5592e3a63000+d1000]
Ok, figured out what the issue was. the dockercloud/haproxy image creates cert files and puts them in /certs/. I had mounted a volume into /certs/, which was messing things up. I moved my mounted volume to /shared-certs/ and everything works!

Resources