Get Web Data

[cmo]

Good day. I'm trying to get data from another web server, for example a stock quote from yahoo. Are there any examples or simple ways I can download an external page and put the data into a string?

I'd like to demonstrate this on a mod 5270 that I have. Also, I am considering purchasing a SBL2e, can it be done on that device also?

Any help is appreciated. Thanks in advance.

[greengene]

you can take a look at forrest's get/post to twitter application note in another
section of this forum.
that has both get and post building along with the necessary socket stuff.
just disregard the twitter parsing specifics and it should be a good framework
to start.

[cmo]

Thanks. That's exactly what I needed.

[cmo]

I'm trying to apply the twitter app to get webpage data using my SBL2e. My script below compiles fine and I've got it partially working. This script reports that it connects, however, it does not have any data to give back, i.e. tcp.DataAvail()=0

Can anyone see what I'm doing wrong? I've tried going through the SBL2e examples and documents with no luck. Any help is very much appreciated. Thanks.

Regards;

Code: Select all

	char method[] = "GET";
	char host[] = "www.google.com"; //example
	char hosturl[] = "/";
	char useragent[] = "NetBurner";

	// Getting IP Address
	IPADDR DestIP_Addr;
	while ((DestIP_Addr = GetIPAddress(host)) == 0) {
		OSTimeDly(TICKS_PER_SECOND * 5);
	}

	// Building GET Command
	char cmd[1024];
	int sizeOfMethod = 0;
	sizeOfMethod = siprintf(cmd, "%s %s HTTP/1.1\r\n"
						"Host: %s\r\n"
						"User-Agent: %s\r\n"
						"\r\n",
						method, hosturl, host, useragent);

	// Sending GET
	static BYTE buffer[256];
	BufferedTcpObject tcp( buffer, 256, NULL );
	int DestPort = 80;
	iprintf( "Attempting TCP connection to %I : %d\r\n", DestIP_Addr, DestPort );
	if( tcp.Connect( DestIP_Addr, DestPort, TICKS_PER_SECOND * 5) ){
	   iprintf( "Connection Succeeded. \r\n");
	}else{
	   iprintf( "*** Connection Failed. \r\n");
	}
	//writing to command
	tcp.Write(cmd,10,TICKS_PER_SECOND*5);

	// Reading Results
	iprintf("Response Bytes: %d", tcp.DataAvail());  //ALWAYS REPORTS 0 BYTES
	while (tcp.DataAvail())
	{
	   char rxbuffer[32];
	   int rv = tcp.ReadAtLeast( rxbuffer, 31, 5, TICKS_PER_SECOND * 5 );
	   if ( rv > 0 )
	   {
	      rxbuffer[rv] = 0;
	      iprintf( "Received%d:[%s]\r\n", rv, rxbuffer );
	   }
	}
	tcp.Close();

[cmo]

Can anyone help?

I've tried messing around with the GET string and the tcp.Write command in my code. I've found that pretty much any GET command string I use in a telnet window returns some data, just not in this script. Also the tcp.Write says it's sending plenty of bytes. I just never get any response using DataAvail. If you see something I'm doing wrong please let me know. Again, the methods are a bit different since I'm using the SBL2e. Thanks.

[pbreed]

You are not waiting ANY time between the write and the read.
It takes some time for the write to go out and the read to come back.

You need to wait around. The easy approach is to add an OSTimeDly, but that's wasteful when the responder is quick.

The easiest way is aloop that looks like:

Code: Select all

DWORD now=TimeTick;

while((Timetick+TIMEOUT) > Timetick)
{
if(tcp.DataAvail())
{
//Do your read stuff....
}


If you want to be REALLY clean then when you create your BufferedTcp Object pass in an initialized Semaphore 
instead of NULL then instead of the OSTimedly you can add an OSSemPend() and only whak up when something happes on the socket.
The Semaphore will be posted to when data comes in or when the other side closes the socket.

Paul



if(tcp.ErrorState!=0)
{
//We have been closed ... or had some other kind of error
//we should check to see if ther is ny last data to read....
 if(tcp.DataAvail())
 {
 //Do your read stuff....
 }
break; //If we got the connection closed we don't need to wait anymore...
}
//For maximum speed delete the next line....
OSTimeDly(1); //Allow other lower priority tasks to run.... 
}//while

[pbreed]

If you really want to be clean, then initalize a Semaphore and pass it into the BufferedTcp object when you construct it.
Then replace the OSTimeDly with a OSSemPend, the TCP connection will post to the semaphore when data comes in or
when the other side closes the connection...


Paul

[cmo]

That was the problem, I failed to give enough wait time between write and read. The semaphore fixed the problem. Thanks again for the help.