Increase TCP/IP Responsiveness

[rsg]

I need to increase the responsiveness for small TCP/IP writes and reads. I don't want the stack to coalesce small writes into one packet, because I need to support many small packets as quickly as possible.

Note that I'm necessarily asking about both on the Netburner side and the client (Windows) side, so suggestions on either side are sought.

Thanks in advance,
-Bob

[bbracken]

Disable the NAGLE algorithm.

Windows uses the TCP_NODELAY option for setsockopt.

NetBurner uses SO_NONAGLE option for setsockopt. NB also has a SO_NOPUSH which you probably want to disable.

bb

[rsg]

Thanks, that (Nagle) is what I was looking for - I just could not remember his name!

By the way, what is the NOPUSH option?

[bbracken]

Not entirely sure what SO_NOPUSH does. It looks as if it is not set that each write would cause the data to immediately be sent. Setting SO_NOPUSH seems to conflict with what SO_NONAGLE does? Seems like you don't need both. Maybe SO_NOPUSH is a different (i.e., different for NAGLE) algorithm for combing smaller writes into larger packets?

bb

[rsg]

I tested with both SO_NOPUSH and SO_NONAGLE, and things worked a lot better, except the final message to the Windows client doesn't get sent - it seems to be queued up, though, so that when another message is sent, it gets pushed out first. So SO_NOPUSH is definitely NOT what I want.

However, SO_NONAGLE is indeed exactly what I want. Now, things work great, provided messages are sent in rapid succession - which they are, in general, for my application. Therefore, this is probably "good enough". However, just in case, I will note something I have observed.

When a bunch of messages (> 100) are sent all at once from the PC to the NB, most of the time they are all sent out and received in a tight burst, generally of a few 10s of milliseconds. However, sometimes the burst is split into two, separated by maybe 100 milliseconds. My clock tick is 20 Hz, which suggests two ticks are allowed to pass, but I'm not sure that is correct. Any ideas on this?

Thanks,
-Bob

[bbracken]

Sounds like the issue is on the PC side of things. Are you looking at all of the ethernet traffic on the wire? The PC will do all sorts of "other" things on the wire, like DNS, ARP, NetBIOS... etc. If you are filtering and only looking at your particular network traffic, you may not be seeing other activity. How does the size of the data vary? Maybe under some circumstances it all "fits". I believe that there is some internal size that is used as a maximum packet size. I think it is called MTU something (not sure).

You need to look at the PC side to see exactly what it is sending. If the PC is sending 2 packets, I'll assume that the NB has to receive 2.

What else do you have going on in your system. I did a project once where the NB was sampling an A/D at a high rate (20,000 samples/second) and transferring the data via sockets. The A/D samples were interrupt based. I actually had to read the A/D samples in a burst mode. In other words I had to wait until the A/D hardware FIFO filled and then read all of the samples in the FIFO within a single interrupt, place in a packet and let a non-interrupt task actually send the data via sockets. If I tried to process each A/D sample in it's own interrupt (in general not a good idea anyway), the sockets performance would be significantly reduced since it was getting enough CPU to process a frame without timing out thus generating a re-transmission.

Do you have Etherreal or some other network monitoring application so that you can actually see what's going on?

[hendrixj]

My understanind of the Nagle algorithm is that it only affects when TCP ACKs are sent. Specifically, the Nagle algorithm tries to reduce network traffic by bundling all ACKs with data packets.

[thomastaranowski]

Not really, the prime directive of the nagle algorithm is to reduce
the number of tiny packets on the wire. Think of a telnet session.
Would you want each keypress to result in a packet on the wire? In
most cases you would not, so several small pieces of the TCP stream
are aggregated together in a single packet. It does bundle the data
together until an ack is received, at which point it disgorges the
accumulated segment.

Wikipedia has a reasonable treatment of it.
http://en.wikipedia.org/wiki/Nagle_algorithm
- Show quoted text -






--
Thomas Taranowski
Certified netburner consultant
baringforge.com

[hendrixj]

Ah, thanks for the clarification. The NODELAY option keeps Nagle from causing a delay by sending the ACK immediately.
Right?