Implementing kernel bypass for a network card

Question

My situation:

I would like the data received on a network card to reach my application as fast as possible. I have concluded that the best (as in lowest latency) solution is to implement a network stack in my user space.

The network traffic can be a proprietary protocol (if it makes writing the network stack easier) because it is simply between two local computers.

1) What is the bare minimum list of functions my network stack will need to implement?

2) Would I need to remove/disable whatever network stack is currently in my Linux/how would I do this?

3) How exactly would I write the driver? I presume I would need to find exactly where the driver code gets called and then instead of the driver/network stack being called, I would instead send the data to a piece of memory which I can access from my application?

How did you "conclude" this without implementing it and testing the relative performance and other characteristics? That sentence should begin "I assumed that..." — R.. GitHub STOP HELPING ICE, Aug 21 '13 at 22:19
A [related question](http://stackoverflow.com/questions/18343365/zero-copy-networking-vs-kernel-bypass) — artless noise, Aug 22 '13 at 02:10

score 1 · Answer 1 · answered Aug 21 '13 at 20:27

1

I think the already built-in PF_PACKET socket type does exactly what you want to implement.

Drawback: The application must be started with root rights.

There are some enhancements to the PF_PACKET system that are described on this page: Linux packet mmap

answered Aug 21 '13 at 20:27

Martin Rosenau

17,897
3
19
38

So PF_PACKET will essentially skip the (kernel) network stack? – user997112 Aug 21 '13 at 20:58
PF_PACKET will only check the ethernet frame type (e.g. 0x800 is IPv4). You can define a user-type (e.g. 0x1234) and packets with this type are sent to your application. Your application receives raw ethernet packets and sends raw ethernet packets. The system seems not to have the best performance - this is why the link (Linux packet mmap) is describing a hack. – Martin Rosenau Aug 22 '13 at 06:30

score 0 · Answer 2 · answered Aug 21 '13 at 21:03

The Kernel is in control of the NIC card. Whenever you pass data between kernel and user-space, there is a context-switch between the kernel rings, which is costly. My understanding is that you would use the standard API's while setting the buffers to a larger size allowing larger chunks of data to be copied between user and kernel-space at a time, reducing the number of context switches for a given size of data.

As far as implementing your own stack, it is unlikely a single person can created a faster network stack than the one built into the kernel.

If the linux kernel is not capable of processing packets at a speed you require, you might want to investigate NIC cards with more onboard hardware processing power. These sorts of things are used for network throughput testing etc.

But this is a proprietary protocol which really doest need the whole "weight" of the Linux network stack. The latter is written to accommodate TCP/UDP etc, whereas I just want a specialised protocol between two computers. Plus I also want to do this to learn a lot. — user997112, Aug 21 '13 at 21:43
Have you looked at how they implemented "PF_RING" http://www.ntop.org/products/pf_ring/? Fundamentally, I don't think it's possible unless you just do everything in kernel-space. Can you provide more information about your application? — samsquanch, Aug 22 '13 at 14:25

Implementing kernel bypass for a network card

2 Answers2