Sockets do not seem to close

[vsabino]

Hi,

I have a list of 10 sockets to possibly open, my application only utilizes 6 out of the 10.
The PC sends commands to the NetBurner to open the sockets, to listen on a port. The sockets are opened, one by one, and assigned to specific port numbers.

The 10 sockets have their file descriptors initialized to -1.

I open the sockets with connect(ipaddr, 0, ipport, 100), and use the returned file descriptor. ipaddr is the IP address I want to connect to, ipport is the remote port number.

The PC software application opens the 6 sockets, and closes them when exiting. The NetBurner closes all 10 sockets (open or not) and resets the 10 file descriptors to -1 when the PC exits the application (PC sends a message to the NB to let it know it is exiting).
I do this to reset all the sockets in the NB. I read somewhere that only 32 sockets are available in the NB (I don't remember where).

The problem I have is that I can open and close the PC application only 28 times before I get the following error: TCP_ERR_NONE_AVAIL (-5), which according to the NB manual, means that there no more sockets available. So, what am I doing wrong? I thought I was closing all sockets.

If I continue to open and close the application, the number of sockets that fail to open increments one by one, until only the first two sockets I attempt to open are able to actually connect.
After this point, I can open and close the application indefinitely with only the first two sockets succesfully connecting, and the remaining 4 give me the (-5) error (I tried 72 times).

Questions:
What am I doing wrong?
Where is the 32 socket limit info located?

Thanks,
Victor

[rnixon]

Hi Victor,

The max of 32 is located in the network programmers guide, section: 16 File Descriptors. More good info there.

Start small and work your way up. Can you create a simple app that does nothing more than open and close one socket and verify it works with the pc program? Once you get that far, modify the program to do it with 10 sockets. Note that any listening sockets do consume a socket, such as the web server.

How do you know for sure your program is properly closing the socket? Have you verified with a wireshark trace?

[vsabino]

Hi rnixon,

Thanks for the reply.

"The max of 32 is located in the network programmers guide" Thanks!!! It also says 38-63 are available for expansion. How do I access those? Is there a way to monitor how many sockets are open?

"Note that any listening sockets do consume a socket, such as the web server. " How do I know how many sockets are used by the RTOS? I do not explicitly have a web server. This is what I'm enabling on startup (NB initialization code):

InitializeStack();
if (EthernetIP==0)GetDHCPAddress();
OSChangePrio(MAIN_PRIO);
EnableAutoUpdate();
EnableTaskMonitor();
EnableSmartTraps();

"How do you know for sure your program is properly closing the socket? Have you verified with a wireshark trace?" Even if the PC is not closing the sockets correctly on their end, the NB is supposed to close all sockets when the PC issues a "goodbye" message, correct? At least that is my intention. I'm making the NB close everything in a loop:

loop ( i = 1 to 10)
{
close(socket i)
}


Regards,
Victor

[rnixon]

Hi Victor,

I do not completely understand your description, but I'll give it a try.

I'm not sure what a "goodby" is, how it is being sent, or what pc program you are using. A wireshark trace would enable you to not only tell if the "goodby" was sent, but also if the sockets are being properly closed.

I'm assuming your code sample is psudocode, since you would need to close the EXACT file descriptors that were open (you need to keep a list) and not just call close on numbers 1 through 10.

From the data you provided it sounds like you are not closing all the open socket fd's, so you have a half open socket condition and are running out.

"How do you know for sure your program is properly closing the socket? Have you verified with a wireshark trace?" Even if the PC is not closing the sockets correctly on their end, the NB is supposed to close all sockets when the PC issues a "goodbye" message, correct? At least that is my intention. I'm making the NB close everything in a loop:

loop ( i = 1 to 10)
{
close(socket i)
}


Regards,
Victor

[vsabino]

Hi rnixon,

We write our own PC software and NB firmware. They comunicate with each other over TCP and serial. When the Software application starts, it sends a "hello" message and when it exits, a "goodbye" message. A handshaking scheme we came up with. We are monitoring the messaging with a software called DebugView. The messages come across fine.

By the way, how can I use wireshark to monitor the socket closure?

The sample code was pseudocode. The actual code is:

for (int n = 1; n < NUM_SERVICES; ++n) // reset all services except serial
{
p_SVC_Set[n]->m_running_state = 0;
close(p_SVC_Set[n]->m_fd_service);
p_SVC_Set[n]->m_fd_service = -1;
}

I have each socket as a separate service. I'm keeping a list of the file descriptors, which a use to close them.

How can I determine if the NB is efectivley closing all open sockets?

Thanks,
Victor

[rnixon]

When a tcp socket is properly closed, you can see the FIN and ACK flags in the wireshark trace.

On any (compliant) tcp stack, the close() just tells the stack you desire to close, it isn't like a serial port. The close() actually tells it to finish up what it needs to do (per the rfc) and then close. So if there is unacknowledged data it will continue to try and handle the situation properly, or eventually time out (may take up to 10 minutes per socket in some cases). One thing you can try is to check each socket for any unread data, then read and discard that data, before calling close(). Essentially a flush. I'm not totally positive about this, but that is the way I think it works.

[tod]

Without more code I have to just guess (BTW, Christoph recently posted a large code sample by using gist.github which I thought was an EXCELLENT idea - hint hint).

At any rate the fact that you can do this 28 times make me think you are leaking one resource each time. It looks like you are closing sockets 1-9 and leaving socket 0 open. Your comment makes it appear you are doing this on purpose. When you open them back up are you making sure you aren't opening up 10?

Just and idea.

Tod

[vsabino]

Thanks to both replies. I could not do any further testing yet since my job responsabilities involve more than coding.

rnixon, I will install and test with wireshark. I'll pay attention to the flags and see what is going on.

tod, I had no idea of the gist website! I'll look to register there so I can post code.

Anyway:
NUM_SERVICES = 11. I'm closing services 1 through 10. 0 is left open on purpose.
service 0 is the serial port connection
services 1 -10 are TCP sockets
I open the sockets one by one in response to commands from the PC, a total of 6 out of the 10. That means 4 sockets are never opened.
I close all 10 possible TCP sockets upon the PC's request.

Briefly, this what happens between the PC and the NB when I open/exit the application:
1_ PC: starts application
2_ PC->NB: Initiate session
3_ PC->NB: Commands to open socket
4_ NB: connect(ipaddr, 0, ipport, 100)
5_ NB->PC: Socket open (or return error)

Repeat 3-5 six times.
Do whatever the application needs to do.

6_ PC->NB: Finalize session
7_ NB: close(p_SVC_Set[n]->m_fd_service); (closes all 10 possible sockets in a loop as described)
8_ NB->PC: Session ended.
9_ PC exits application

Up to 28 times, I can open all 6 sockets
The 29th time, I can only open 5 sockets
The 30th time I can only open 4 sockets
The 31st time I can only open 3 sockets
The 32nd time and beyond (tested up to 72 times) I can only open 2 sockets.


Every time I open/exit the application, I call connect() 6 times, and then call close() 10 times.
I thought as tod said, that I was leaking one resource each time, but it seems to bottom out to leaving 2 sockets available. I'm confused.

I guess you would like to see more code, specifically where I handle the list of file descriptors returned by connect(). Anything else that you consider relevant?

Thanks,
Victor

[vsabino]

I've got news!!

rnixon, you were right on the money by suggesting to use wireshark! Thanks for the pointer to look at the FIN flags.

To make the story short, the PC application is sending a command to open a socket, and apparently does not wait enough for the response from the NB, so it commands it to open again.

The NB then opens 2 connections to the same PC port, overwriting the first file descriptor.
So when it's time to close sockets, the second file descriptor gets closed, leaving the first one open. There is the leak.
The "unpatient" PC does not happen all the time, apparently once in a start/exit cycle.
I still don't know exactly why the number turned out to be 28, or why I was left with 2 sockets in the end. Maybe it's a coincidence, although I repeated the test 3 times.
Still, I found a bug in my code (and on the PC side), which is a starting point.

So, as a conclusion, there is no harm in closing sockets that were never open. Correct?

Thanks again.
Victor