UART7 missing bytes on 54415

Discussion to talk about software related topics only.
Post Reply
lslarry86
Posts: 24
Joined: Tue Jul 18, 2017 12:28 pm

UART7 missing bytes on 54415

Post by lslarry86 »

I have a 54415 application with 5 serial ports:
0 - data and interaction (ucos Console is moved to UART 9)
1 - RFID reader 0
2 - RFID reader 1
5 - RFID reader 2
7 - RFID reader 3

My main loop polls each UART in turn. Things seem to work fine except reader 3 on UART7 misses bytes in the _middle_ of its reader's reply. This is the real data from a spy connection:
02 00 08 00 69 00 b3 57

but this is what the software reads:
02000800B357

It's pretty consistent - I just did it 5 times in a row and saw the same string every time.

I've reade elsewhere that the ucos UART buffers are 1500 bytes, so I am not taxing the system here. Breakpoints are bad in this case since all the UARTs start reading bad when I break on the read of UART 7. Code is below. Can anyone spot a mistake?
Thanks,
Larry

For the 4 reader UARTS, the loop is:

Code: Select all

        for (int rix = 0; rix < MAX_READERS; rix++)
        {
        	RfidReaders[rix].Tick();
        }
The Tick() function ends up here:

Code: Select all

	if (RfidRxBufCount < (int)(sizeof(RfidRxBuf) - 1))
	{
		int rc = dataavail(RfidSerialFd);
		if (rc > 0)
		{   
			rc = ReadWithTimeout(RfidSerialFd, (char*)&RfidRxBuf[RfidRxBufCount], CFG_RXBUFLEN - RfidRxBufCount, 1); // 200 uS for one byte, 500 uS for many, if data is available
		}
		if (rc > 0)
		{
			if (Index == 3)
			{
				char buf[64];
				UtilsByteArrayToHexString(buf, &RfidRxBuf[RfidRxBufCount], rc);
				sprintf(WebErrorMessage, "Reader 3 said: %s", buf); //deleteme
			}
			ret = true;
			RfidRxBufCount += rc; // point to next empty space
		}
	}
That hits once per command. On UART7 only, rc is short. The data is correct on each end but two bytes are missing in the middle.
sulliwk06
Posts: 118
Joined: Tue Sep 17, 2013 7:14 am

Re: UART7 missing bytes on 54415

Post by sulliwk06 »

It sounds like it would be more efficient to have a task with a select() call that is listening to all 4 of your file descriptors at once. That way you don't have to have separate timeouts for each one. I think it would be much safer in case you end up getting a lot of simultaneous data from each one.

If it were me, I would do it like this: (of course you could move the reads into a function from your reader class)

Code: Select all


void Task()
{
	fd_set read_fds;
	fd_set error_fds;
	int timeout = 0;
	while(1)
	{				
		FD_ZERO( &read_fds );
		FD_ZERO( &error_fds );

		for (int rix = 0; rix < MAX_READERS; rix++)
		{
			FD_SET( RfidReaders[rix].RfidSerialFd, &read_fds );
			FD_SET( RfidReaders[rix].RfidSerialFd, &error_fds );
		}
		
		if ( select( FD_SETSIZE, &read_fds, ( fd_set * ) 0, &error_fds, timeout ) )
		{
			for (int rix = 0; rix < MAX_READERS; rix++)
			{
				if(FD_ISSET(RfidReaders[rix].RfidSerialFd, &error_fds))
				{
					//error
				}
				else if (FD_ISSET(RfidReaders[rix].RfidSerialFd, &read_fds))
				{
					//read the new data
					int rc = read( RfidReaders[rix].RfidSerialFd, (char*)&RfidReaders[rix].RfidRxBuf[RfidReaders[rix].RfidRxBufCount], CFG_RXBUFLEN - RfidReaders[rix].RfidRxBufCount );
					RfidReaders[rix].RfidRxBufCount += rc; // point to next empty space
				}
			}
		}
	}
}
lslarry86
Posts: 24
Joined: Tue Jul 18, 2017 12:28 pm

Re: UART7 missing bytes on 54415

Post by lslarry86 »

Thanks for your reply, sulliwk06. I would have to rewrite my software to that structure. That might be worth the trouble and risk if I could be sure it would fix the problem.

The way I see it, the chip has 10 UARTs in silicon, running independently. When any UART receives a complete byte, it interrupts the microprocessor and an ISR pulls the byte out to a ucos buffer. Bytes should sit quietly in buffer until I ask for them. My current code runs as a superloop from UserMain, and does not call ReadWithTimeout unless dataavail() returns true. So I don't see how my code is abusing the micro or ucos any worse than a select call would. Either way, the bytes I want sit in ucos buffer until I poll for them.

Code: Select all

		int rc = dataavail(RfidSerialFd);
		if (rc > 0)
		{
			rc = ReadWithTimeout(RfidSerialFd, (char*)&RfidRxBuf[RfidRxBufCount], CFG_RXBUFLEN - RfidRxBufCount, 1); // 200 uS for one byte, 500 uS for many, if data is available
		}
It seems to me that this symptom can only happen if UART7's interrupt is delayed so long that an entire byte comes and goes before the ISR executes. Again, if that is the case, it would probably still happen after I rewrite the code for select().

This design is triggered. At trigger time, all four RFID readers are commanded at the same time. The replies come in more or less in synch too, all at 115200 baud. I will experiment with a short delay before sending the UART7 command and see if that changes anything.

Thanks,
Larry
User avatar
TomNB
Posts: 538
Joined: Tue May 10, 2016 8:22 am

Re: UART7 missing bytes on 54415

Post by TomNB »

What version of tools are you running?

If you only run uart7 do you get all the data? If you add uarts back in one a a time when does the problem occur?
lslarry86
Posts: 24
Joined: Tue Jul 18, 2017 12:28 pm

Re: UART7 missing bytes on 54415

Post by lslarry86 »

Hi, Tom, thanks for weighing in. My toolchain is 2.8.2. All my hardware was bought this year.

If I disable UART7 and keep using UARTS 1-2-5, UART5 starts missing. I don't have it instrumented so I can't say for sure it's the same problem, but it seems likely.

If I disable UARTs 2-5 and keep using 1-7, both the active UARTs are ok, including UART7.

If I disable UART2 OR UART5, the problem becomes intermittent on UART7. I just saw it fail 2 out of 3 times.

When it fails, the string is always short but not always what I said yesterday. I just saw "02000857" reported when the spy connection shows that the reader sent "02 00 08 00 69 00 b3 57" so the missing bytes are still in the middle of the message.

I just tried it in Run mode (vs. Debug) and it started working on all 4 reader UARTs. If that holds up, it's a good clue, but I need to be able to debug. I was hoping to add two more active serial ports to this setup.

I am trying to enforce a stepped delay before sending each command, but it looks like OSTimeDly is not always returning. That will be another post if I can't figure it out this morning.
lslarry86
Posts: 24
Joined: Tue Jul 18, 2017 12:28 pm

Re: UART7 missing bytes on 54415

Post by lslarry86 »

Partial solution, inside a loop where rix is 1..3:

Code: Select all

#ifdef _DEBUG   // 22Nov2017LM: In Debug mode with >2 readers, ISR service delays cause dropped rx bytes unless you ripple the commands.
				// So far Run mode is ok.
						switch(rix)
						{
						case 2:
							OSTimeDly(1);
							break;
						case 3:
							OSTimeDly(2);
							break;
						}
#endif
						RfidReaders[rix].OpenReadWindow(false);
This slows down Debug mode operations, which are already slow anyway, but it does allow me to run the code in Debug mode without false bads.

I think this whole thing tells us that the extra network traffic of Debug mode blocks the serial ISRs (because the missing bytes are always in the middle of the message), creating a problem if more than two UARTs receive data at the same time. I hate the idea that I am operating this close to a performance limit of the system. Is there anything I can do to punch that up? I'm using 54415 in the first place to get the 10 UARTs (8 pinned).

Note: On my 54415 system, OSTimeDly(0) never returns, in Debug or Run modes. An earlier version used "OSTimeDly(rix-1)" where rix is 1..3. It always stalled out on the first call.
User avatar
TomNB
Posts: 538
Joined: Tue May 10, 2016 8:22 am

Re: UART7 missing bytes on 54415

Post by TomNB »

Debug mode will always be much slower for a few reasons:
- All debuggers must turn optimization off to operate
- Network debuggers work using a level 7 non-maskable interrupt for the debug network stack (separate from the system network stack), so it will always take priority from everything, including your serial ISRs
- The very fast single cycle SRAM on the processor is not used for buffers and task stacks like in release mode

So I don't think you are close to the performance limit if you run in release mode, but you do want to architect your code to be very efficient, such as the suggestions by sulliwk06.

OSTimeDly(0) will always block forever. You can't use 0. Not sure why, I think its some type of historical reason. Note also that a delay of 1 will not guarantee you a full delay of 1 tick. That is because it runs from the system timer and if it is half way through a tick count you will only get the remaining half. If you need better resolution look at the High Res Timer example. Significantly more resolution and you have full control of the timer.
Post Reply