Responses from kubernetes containers getting lost - docker

Question

I have installed kubernetes on openstack. The setup has one master and one node on coreos.
I have a pod hosting an SIP application on UDP port 5060 and I have created service as NODEPORT on 5060.
The spec:
"spec": {
"ports": [
{
"port": 5061,
"protocol": "UDP",
"targetPort": 5060,
"nodeport": 5060,
"name": "sipu"
}
],
"selector": {
"app": "opensips"
},
"type": "NodePort"
}
IPs
Public IP of node: 192.168.144.29.
Private IP of node: 10.0.1.215..
IP of the container:10.244.2.4.
docker0 interface: 10.244.2.1.
Now, the problem.
I send a SIP request to the application and expect a 200 OK response, which I am not getting.
To trace the same, I installed TCPDUMP on the container and the node.
On the container, I can see the request and response packet captured while on the node itself I just see the request packet. Don't understand why the packet is getting lost.
Below is tcpdump of container:
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 1514 bytes
06:12:20.391171 IP (tos 0x0, ttl 64, id 13372, offset 0, flags [DF], proto UDP (17), length 1010)
10.244.2.1.50867 > 10.244.2.4.5060: [bad udp cksum 0x1ddc -> 0xe738!] SIP, length: 982
PUBLISH sip:service-1#opensipstest.org SIP/2.0
Via: SIP/2.0/UDP 192.168.144.10:5060;branch=z9hG4bK-5672-1-0
Max-Forwards: 20
From: service <sip:service-1#opensipstest.org>;tag=1
To: sut <sip:service-1#opensipstest.org>
Call-ID: 1-5672#192.168.144.10
CSeq: 1 PUBLISH
Expires: 3600
Event: presence
Content-Length: 607
User-Agent: Sipp v1.1-TLS, version 20061124
<?xml version="1.0"?>
<deleted presence xml to reduce size>
06:12:20.401126 IP (tos 0x10, ttl 64, id 11888, offset 0, flags [DF], proto UDP (17), length 427)
10.244.2.4.5060 > 10.244.2.1.5060: [bad udp cksum 0x1b95 -> 0xeddc!] SIP, length: 399
SIP/2.0 200 OK
Via: SIP/2.0/UDP 192.168.144.10:5060;received=10.244.2.1;branch=z9hG4bK-5672-1-0
From: service <sip:service-1#opensipstest.org>;tag=1
To: sut <sip:service-1#opensipstest.org>;tag=332ff20b76febdd3c55f313f3efc6bb8-ca08
Call-ID: 1-5672#192.168.144.10
CSeq: 1 PUBLISH
Expires: 3600
SIP-ETag: a.1450478491.39.2.0
Server: OpenSIPS (1.8.4-notls (x86_64/linux))
Content-Length: 0
06:12:20.401364 IP (tos 0x0, ttl 64, id 13374, offset 0, flags [DF], proto UDP (17), length 427)
10.244.2.1.58836 > 10.244.2.4.5060: [bad udp cksum 0x1b95 -> 0x1bcc!] SIP, length: 399
SIP/2.0 200 OK
Via: SIP/2.0/UDP 192.168.144.10:5060;received=10.244.2.1;branch=z9hG4bK-5672-1-0
From: service <sip:service-1#opensipstest.org>;tag=1
To: sut <sip:service-1#opensipstest.org>;tag=332ff20b76febdd3c55f313f3efc6bb8-ca08
Call-ID: 1-5672#192.168.144.10
CSeq: 1 PUBLISH
Expires: 3600
SIP-ETag: a.1450478491.39.2.0
Server: OpenSIPS (1.8.4-notls (x86_64/linux))
Content-Length: 0
And tcpdump of node:
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 1514 bytes
06:12:20.390772 IP (tos 0x0, ttl 127, id 20196, offset 0, flags [none], proto UDP (17), length 1010)
192.168.144.10.5060 > 10.0.1.215.5060: [udp sum ok] SIP, length: 982
PUBLISH sip:service-1#opensipstest.org SIP/2.0
Via: SIP/2.0/UDP 192.168.144.10:5060;branch=z9hG4bK-5672-1-0
Max-Forwards: 20
From: service <sip:service-1#opensipstest.org>;tag=1
To: sut <sip:service-1#opensipstest.org>
Call-ID: 1-5672#192.168.144.10
CSeq: 1 PUBLISH
Expires: 3600
Event: presence
Content-Length: 607
User-Agent: Sipp v1.1-TLS, version 20061124
<?xml version="1.0"?>
<presence xmlns="urn:ietf:params:xml:ns:pidf" />
Nodeport rules from Iptable
Chain KUBE-NODEPORT-CONTAINER (1 references)
pkts bytes target prot opt in out source destination
12 8622 REDIRECT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/opensips:sipu */ udp dpt:5060 redir ports 40482
3 95 REDIRECT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-udp-service: */ udp dpt:6000 redir ports 47497
Chain KUBE-NODEPORT-HOST (1 references)
pkts bytes target prot opt in out source destination
0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/opensips:sipu */ udp dpt:5060 to:10.0.1.215:40482
0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-udp-service: */ udp dpt:6000 to:10.0.1.215:47497
I am happy to share more info if required. I have tried to dig in my capacity, but now I don't know what to look therefore requesting some help here.
EDIT
I did the same test on TCP. On TCP, it works as expected.
Thanks

Answer 1

5060 is outside the default Service Node Port Range:
http://kubernetes.github.io/docs/user-guide/services/#type-nodeport
Creation of the service should have failed.
You can try a different port, or create a cluster using a different range, by specifying --service-node-port-range on kube-apiserver.
https://github.com/kubernetes/kubernetes/blob/43792754d89feb80edd846ba3b40297f2a105e14/cmd/kube-apiserver/app/options/options.go#L232
You should also check whether the response can be seen from other nodes. There are issues with "hairpin mode" when communicating within the same node.
Also, when reporting problems, please specify the Kubernetes release.