MOD5270 Ethernet communication thread issue

Discussion to talk about software related topics only.
Post Reply
prasad04ie19
Posts: 9
Joined: Tue Jun 16, 2015 4:49 am

MOD5270 Ethernet communication thread issue

Post by prasad04ie19 »

Hi,

As you are aware, we have been using MOD5270 extensively in our products since 9 years to support SNMP. Of late in some networks, we are noticing that the IPAddress of the MOD5270 is disappearing from the network (in couple of hours or days after a power cycle). When this happens we were no longer receiving responses to ping or SNMP requests. The units are requiring a manual power cycle to restore the access. We believe the processor has not hanged/ crashed as we found some other threads which had LCD interfaces (RS232/serial) are responding.

While we try to find the reasons, could you let us know:
1. Under what conditions this can happen?
2. Ways of detecting such cases (cable is plugged-in but Ethernet interface stopped working) and probably restart the service or reboot the MOD5270.

Thank you,
Prasad

P.S: We are using NBEclipse IDE ver 2.7.3
User avatar
pbreed
Posts: 1080
Joined: Thu Apr 24, 2008 3:58 pm

Re: MOD5270 Ethernet communication thread issue

Post by pbreed »

The VERY first thing I would do is log the response from the call..

#include <buffers.h>
WORD GetFreeCount();


With UDP based protocols, if you are somehow loosing track of a udp packet once in a while you loose a buffer....

loose them all and you are dead....
prasad04ie19
Posts: 9
Joined: Tue Jun 16, 2015 4:49 am

Re: MOD5270 Ethernet communication thread issue

Post by prasad04ie19 »

Thanks for your response. We will look into it and provide the feedback.
prasad04ie19
Posts: 9
Joined: Tue Jun 16, 2015 4:49 am

Re: MOD5270 Ethernet communication thread issue

Post by prasad04ie19 »

Hi,

We made an observation of 4 units with GetFreeCount() function for 5 hours. During this duration, the function returned values in the range of 260-262 and 1 of the unit disappeared in this duration. Can you provide other suggestions that we should look for?

We are using all serial ports for serial communications with other peripherals, so we do not have serial debug facility. The values that we want to debug are sent over Ethernet via SNMP.

Best regards,
Prasad
User avatar
pbreed
Posts: 1080
Joined: Thu Apr 24, 2008 3:58 pm

Re: MOD5270 Ethernet communication thread issue

Post by pbreed »

The next thing I'd do...
(Asuming a release build, not debug)

Is add:

#include <taskmon.h>

EnableTaskMonitor();

Then run the utility taskscan... see what the differnt tasks are doing.

When the unit disapers:

Is it only SNMP that dies?
Can you ping it?
Can you find it with ipsetup?

Does it have a link light?

If you check the ARP cache on your pc is there some other device with the same address?
prasad04ie19
Posts: 9
Joined: Tue Jun 16, 2015 4:49 am

Re: MOD5270 Ethernet communication thread issue

Post by prasad04ie19 »

Hi,

Please find below the answers to your queries:

Q: (Asuming a release build, not debug)
A: Yes, we use .s19 file from release folder.

Q: Observe tasks
A: TaskScan monitors the threads/ tasks over Ethernet. Since IP is disappearing in our case, we could not use this debug option. When the unit was working, the expected tasks were shown in the list.

Q. Is it only SNMP that dies?
A. We think IP thread is dying when the issue occurs

Q. Can you ping it?
A. No

Q. Can you find it with ipsetup?
A. IPSetup or AutoUpdate is not showing the device

Q. Does it have a link light?
A. Yes, link lights were On. EtherLink() always returns 1 (even in disappeared case also). We verified that EtherLink() was returning 0 when Ethernet cable is unplugged.

Q. If you check the ARP cache on your pc is there some other device with the same address?
A. We checked the ARP cache on the PC. There are no entries with the IP address of the disappeared unit.

Thank you,
Prasad
User avatar
pbreed
Posts: 1080
Joined: Thu Apr 24, 2008 3:58 pm

Re: MOD5270 Ethernet communication thread issue

Post by pbreed »

Is your code using USER_ENTER_CRITICAL, or OSLock ANYWHERE.....

Are you writing your own interrupts of any kind?

You need to dump info to the console to learn what is happening...


In 2.7.x your best choice is
in nburn\include\predef.h
Enable
#define UCOS_STACKCHECK (1)


Then call

#include <ucos.h>
void OSDumpTasks( void );
Dumps task info to the console...


In 2.8.x there is an additional function:
#include <taskmon.h>

const char * GetUCOS_TaskStateBuffer();
prasad04ie19
Posts: 9
Joined: Tue Jun 16, 2015 4:49 am

Re: MOD5270 Ethernet communication thread issue

Post by prasad04ie19 »

Hi,

We further worked on this issue by slowly removing the functionality of our program and found that even a default MOD5270 template program created by the IDE with project options (DHCP , AutoUpdate, TaskMonitor and SmartTraps) is also having this issue. This project was created with v2.8.5 tools. Since, there is no SNMP to report SystemUpTime, we were just pinging the unit to detect the issue. Later, to avoid manual power cycle, we added a small code to detect the condition and do a ForceReboot. You will notice from the captured ping responses that MOD5270 restarted about 5 times (at lines 3032, 32030, 72216, 81197 and 137373 file ping_nb151.txt) in 42 hour weekend test.

We have also attached the project files for your reference in MOD5270_Ethernet_Issue_Files.zip.

The modifications we made to the default template are described below:
1. ip.cpp: added a variable i50msCountEthernet and cleared it in IpTask, whenever a packet is received.
2. main.cpp: In the continuous while loop, i50msCountEthernet is increment every 50ms. Since we have Ethernet traffic, the i50msCountEthernet always resets to zero. We use a 1 minute timeout on this counter to detect the problem and do a ForceReboot.

Kindly suggest the next steps in understanding, debugging and fixing the issue.

Since no other interfaces are used in above test, we are also working on bringing out the access to serial port0 for debugging and iprintf message purposes. However, it will still take few more days.

Thank you,
Prasad
Attachments
MOD5270_Ethernet_Issue_Files.zip
IDE Project using which we observed the problem and the ping captured are present in this archive
(922.13 KiB) Downloaded 200 times
prasad04ie19
Posts: 9
Joined: Tue Jun 16, 2015 4:49 am

Re: MOD5270 Ethernet communication thread issue

Post by prasad04ie19 »

Hi,

We did not use USER_ENTER_CRITICAL, or OSLock in our code. We have written our own interrupts. However, as mentioned in previous message, even the default NetBurner project template (without any of our code) is also showing this problem. Let us know your suggestions in debugging the problem with that program.

We are planning to get access to serial port 0 of one of the units. Initially, we can only see what NetBurner can transmit. As suggested in your previous message, we will try to the dump the tasks info to console. Also, in the test program attached to my previous message, could you suggest where and what debug messages can we add to monitor internal details time to time or print every second?

Thank you,
Prasad
prasad04ie19
Posts: 9
Joined: Tue Jun 16, 2015 4:49 am

Re: MOD5270 Ethernet communication thread issue

Post by prasad04ie19 »

Hi,

FYI. NetBurner analyzed the crash logs of test program provided above and clarified that the issue could occur if auto-negotiation addressing selects half-duplex. The beta fix provided to us looks promising. Awaiting the release of 2.8.6 official release with this fix.

Thank you for your support.

Best regards,
Prasad
Post Reply