How to force rerouting of a packet after return to netfilter from NFQUEUE - network-programming

Question

I have a caching application that runs in userspace. Normally, it uses DPDK to interact with the NICs directly without kernel interaction, grabbing packets off of the wire, and either passing them through, generating a response from cache, or in some occasions blocking them. It's transparent to the endpoints, and is written to work on raw packet data straight from the NIC. Notably, I'm not doing normal socket programming here - everything's handled at the packet level, without interaction with the TCP/IP stack.
For reasons that are long and boring, I wanted to add some NAT functionality. As a proof of concept, I was hoping to put a front end on my application using iptables/netfilter. So I did the NAT parts with commands like these:
sudo iptables --table nat --append PREROUTING --in-interface seg1a -j DNAT --to-destination $SERVERIP
sudo iptables --table nat --append POSTROUTING --out-interface seg1b -j MASQUERADE
sudo route add $SERVERIP seg1b
This works well for my purposes. The clients now attach to an interface on my system, but their traffic gets redirected to the server. I'm guaranteed that their traffic passes through my system, so all's good.
But to be useful as a cache, I need to be able to respond to some requests that I get from clients. I had thought I could use NFQUEUE for this, with a small application that reads from the netfilter queue and passes packets to and from my application via IPC. I used an iptables rule like this:
sudo iptables --table mangle --append FORWARD -j NFQUEUE
This works OK as long as my application doesn't respond to anything. But when my cache attempts to respond to something from one of the endpoints, things go wrong. The cache reverses all of the L2-4 headers, manages sequence and ack numbers, etc. But the packets don't get back to the client.
I think what's happening is that routing decisions for the packet were made before it was sent to NFQUEUE. So even though the cache returns something whose source and destination IP addresses are reversed, that's irrelevant for the routing of the packet.
I've tried a number of things to change the routing of my response packets, but nothing seems to work. How does one go about changing the routing on a packet read from netfilter queues? Failing that, is there a good way to just inject packets onto the wire from netfilter queues? If that were doable, I could block the original request and then send the cache's reply as an entirely new thing.
Thanks in advance for any suggestions.

Answer 1

The best answer I came up with for this problem was to open a raw socket and feed it my pre-built packets. Critically, you must create an iptables rule that matches your generated packets and tells netfilter not to do connection tracking. Otherwise, netfilter sees the injected packets and thinks there's a connection collision. It then does port forwarding, so the packets come out with a different port than you put in there!
I built the socket like this:
sid = socket(AF_INET, SOCK_RAW, IPPROTO_TCP);
Then, I turned on a couple of options (leaving failure handling out for brevity):
int one = 1;
const int *val = &one;
retval = setsockopt(sid, IPPROTO_IP, IP_HDRINCL, val, sizeof(one));
retval = setsockopt(sid, SOL_SOCKET, SO_MARK, val, sizeof(one));
The first of these tells the kernel that I have the headers already, so no need to try and create them. The second tells the kernel to put a mark value of 1 on every packet sent via the socket, which I can match against in routing or iptables rules.
Later, I built a destination address and used sendto() to send the packet:
retval = sendto(sid, pL3, len, 0, (struct sockaddr *)&daddr, sizeof(daddr));
Finally, I installed a rule telling netfilter not to do connection tracking on TCP packets marked with a 1. Since I had added the sockopt above so that all of my generated packets had this mark, this worked well:
sudo iptables -t raw -I OUTPUT -p tcp -m mark --mark 1 -j CT --notrack