	AG throughput issue & frequent errors at higher bauds - Experimentation
	-----------------------------------------------------------------------

																		Name : R.Jeyasudha
																	   Date : 21-Oct-1999
																  Revision : 1

Brief sketch of AG TCP/IP dial-out 
----------------------------------
	AG ( Asynchronous Gateway ) Server is the async server which has a pool
of modems ( any DCE, for that matter ) shared by all the clients (winMCSI 
clients ) in the same LAN. Need & take policy of modem sharing. The winMCSI 
clients after taking the modem ( if they are free to use ) can use that modem
to dial out to any remote PC/router & can transfer files to or from the remote
side. This dial-out & file transfer is achieved by the winMCSI clients by 
running any windows terminal applications like mew ( meterm for windows ), 
hypertrm etc., Thru these applications, we can set modem init strings, dial 
strings, baud rate, flow control, start, stop bits etc., & can have the control
of the modem. This is the overview of AG. 
	This AG server functionality is supported by ProxyServer. Thus proxy also
acts as an AG server.

Terminology 
-----------
winMCSI client -> Network side
remote side -> Serial side ( Proxy's WAN side )

Internally, how AG works ?
--------------------------
	In simple words, thru TCP/IP redirector.

1. File transfer from Network side to Serial side.
--------------------------------------------------
		The entire file will be split into data packets of varying or equal 
sizes. Each data packet will be prepended with AG MCSI, tcp, ip headers ( src
ip as that of winMCSI client's & destination ip as that of Proxy's LAN addr )
& encapsulated into ethernet frames & will be put into LAN of proxy. Proxy 
after receiving those AG packets ( tcp dest port number 599 ) destined to it,
will decapsulate the data packet from all the headers, encapsulate in ppp 
frames & will send it to the resp serial wan port, which is mapped by winMCSI
client. In this scenario, one TCP/IP session will be formed between the winMCSI
client & Proxy. And one serial session established indirectly between winMCSI 
client & remote side thru Proxy thru the mapped serial wan port. Thus from the
TCP/IP session, proxy redirects all the data packets as sent by the client to 
the serial remote side.

Flow of data from Network to Serial								Flow Control
-----------------------------------								------------
1. winMCSI client's TCP buffer -> Proxy's TCP buffer  :  Proxy advertising window size of zero to client if Proxy's tcp buffer is full.
2. Proxy's TCP buffer -> Proxy's AG application 		:  Not needed. Need arises only if malloc fails to copy the data pkt, for which 
																			chances are very less.
3. Proxy's AG application -> Proxy's serial tx buffer :  Proxy queuing pkts in linked lists in application, if free serial_tx_buffer 
																			is not available.
4. Proxy's serial tx buffer -> remote side's serial rx buffer : rts-cts hardware flow control by remote side to Proxy, if remote's free 
																						serial_rx_buffer is not available.

	This file transfer from Network to Serial works perfectly fine, without any
problems or issues, irrespective of any bauds. Flow control is also happening 
fine, without any loop holes. 

2. File transfer from Serial side to Network side.
--------------------------------------------------
		In this type, exactly the reverse happens. The remote serial side will 
be sending data packets encapsulated in ppp frames. Proxy after receiving 
those packets from wan serial scc, will decapsulate the data packet, frame IP
( src ip as that of Proxy's LAN & dest ip as that of winMCSI client ), TCP & 
AG MCSI headers & encapsulate in ethernet frames & will be put in the LAN. 
winMCSI client will take the packet, destined to that.

Flow of data from Serial to Network										Flow Control
-----------------------------------                            ------------ 
1. remote side's serial tx buffer -> Proxy's serial rx buffer : rts-cts hardware flow control by Proxy to remote side, in case, if free 
																						serial_rx_buffer is not available.
2. Proxy's serial rx buffer -> Proxy's AG application : Application not taking any pkts from serial, in case, if there are retained pkts
																			already in application.
3. Proxy's AG application -> Proxy's TCP buffer : retaining pkts in linked lists, in case if Proxy's TCP buffer is full &
																	& not able to take any more pkts from application.
4. Proxy's TCP buffer -> winMCSI client's TCP buffer : winMCSI client advertising a wnd size of zero to Proxy if its TCP
																			buffer is full & not able to accept any more pkts from Proxy.

Description of feature/Problem 
------------------------------
	Some loop holes in the code ( in flow control ) regarding file transfer 
from Serial to Network side, as a result of which, at higher bauds ( other 
than 19200 ) & at specific conditions ( when wnd size advertised by the 
client is zero, as a result of which leading to congestion in Proxy's tcp
buffer & application & serial_rx_buffer one by one ), we used to get crc 
16/32 chksum error at Network side at mew application level. Meaning of which,
we were very cooly losing few bytes or few packets as a whole !!! Need to 
explore whether we are losing bytes or packets & why ? 

Assumptions
-----------
	From the AG application, as soon as, we found out that, we were unable to 
sent packets to Proxy's tcp buffer, we will try to retain that pkt & we will 
be setting a flag in serial, indicating that, serial should not, send any more 
data packet, starting from the very next packet, to application, till application
is able to send all the retained packets to TCP buffer. During those times, 
serial can send NULL pkts instead of any data packet. Then only the sequence 
order of data packets will be maintained. 

How proxy's tcp buffer can become full ?
	If wnd size advertised by the client is zero, all the data sent from the 
application gets accumulated in Proxy's tcp buffer without being sent to the 
client, until the client advertises a wnd size greater than zero. 

How we can set a flag in serial indicating, not to send data packets but NULL
pkts ?
	To explain this, let me give the complete picture of how data flows from
serial to tcp buffer thru our code.

			check_scc2_rx_packets() ( serial.68k ) -> ISR on pkt reception on serial wan2
	 		|	 |	 	|	|						
			|	 |	 	|	* serial_get_a_free_wan_rx_descriptor() ( serrx.c )
			|	 |	 	|			|
			|	 |	 	|			* get buffer from free_list.
			|	 |		|   		* Add it to scheduled_list.
			|	 |		|			* Disable rts if available free_buffers <= 1/3rd max buffers.
			|	 |		|			* Return buffer pointer.
			|	 |		|
			|	 |		* Preserve obtained buffer pointer.
			|	 |
			|	 * serial_rxed_packet() ( serrx.c )
			|	  			|
			|				* get from scheduled_list on which pkt is rxed - current RX_BD.
			|				* Add it to current_list.
			|
			* Replace current RX_BD's data pointer with preserved buffer pointer.

This ISR is called as & when a pkt is rxed on wan2 serial scc.


			operating_system() (rtrware.c)
			|
			* serial_foreground() (serinit.c)
				 |
				 * pass_received_buffers_to_application() (serinit.c)
						|
## Congestion 		* if (stop_passing_rx_buffer_to_application == TRUE)
flag checking		  		call serial_rx_callback with NULL pkts.
							else
							{
## Wrong assumption		while ( current_rx_buffer )
							  		call serial_rx_callback with valid current_rx_buffer data pkt pointer.
							}
						*	serial_rx_callback() (agrx.c)	
									|
									* retained_pkts ?
									  |		|
									  * no	* yes
									  |		* send_retained_pkts !
									  |		* If send not successful, retain_unsent_packet() ( agsess.c ).
									  |		* still retained_pkts ?
									  |		* If yes, set send_status to 0!
									  |		* Else set send_status to 1!
## Error path		 			  |		* ag.fptr_rx_complete() -> serial_rx_complete() (serrx.c)
									  |				|
## Congestion flag setting	  |				* if (send_status == 0)
									  |			 	     stop_passing_rx_buffer_to_application = TRUE;
									  |		  		  else
									  |					  stop_passing_rx_buffer_to_application = FALSE;
									  |				* get pointer from current_list.
									  |				* add it to free list.
									  |				* Enable rts, if disabled & available free >= 2/3rd of max buffers.  				
									  |
									  * Frame MCSI header.
									  * send_serial_rx_packet_on_network() (agnetif.c)
											|
											* ag_tcp_send_serial_rx_packet_on_network() (agip.c)	 
														|
														* send() (sockoapi.c)
														  |	   |   	|
														  |		|		* socket_send_data() (socksubr.c)
														  |		|		|
														  |		|		* sptr_transport_interface->send() -> socket_tcp_send() (tcpsock.c)
														  |		|		|
														  |		|		* copy the data given by application into tcp buffer.
														  |		|		* Later, from tcp_timer() ( tcptimer.c ), tcp_send_fgnd() ( tcpfgnd.c ) will 
														  |		|			be called, which in turn will call tcp_send() ( tcpsend.c ) function, where
														  |		|			we will be forming TCP header & will give it to IP. Then IP level
														  |		|			will take care to frame IP header & to send the pkt to winMCSI client.
														  |		|
														  |		* send is successful !!
														  |		* ag.fptr_rx_complete() -> serial_rx_complete() (serrx.c)
														  |
														  |
														  * send not successful !!
														  * retain_unsent_packet() (agsess.c)
	
This is the entire data flow path in our code.		

## Congestion flag checking
---------------------------
Based on this flag only, serial will know, whether to send data pkts or NULL
pkts to application.

## Congestion flag setting
--------------------------
Here only, we will be setting the flag, which indicates congestion at tcp buffer
has started.

Will explain ## Error path & ## Wrong assumption, at the last.

We have right now : 
MAX_SERIAL_RX_BD as 8.
Total serial receive buffer descriptors are 32. This 32 includes free,
scheduled & current type of buffers. Initially all the 32 will be in free_rx_buffer
list. Then first 8 will be scheduled in scheduled_rx_buffer list, having one
to one correspondence with 8 RX_BDs. So always 8 buffers will be in scheduled
list. As soon as we receive a pkt in wan serial scc, an interrupt for the pkt
reception in that SCC will occur. In that ISR, we will try to first get a free
buffer from free_rx_list & will try to put it in scheduled_rx_list. Also we 
will save the data pointer associated with that free buffer we have obtained.

/* Note1 begins .... */
  
Note1 :
-------
This is needed as we have to replace one data pointer in RX BD, in place of 
the one, we are going to lose. In the sense, the data pointer on which, we 
have received a pkt & for which interrupt has occured, we will give that 
data pointer to application. Same data pointer will be used, without mallocing
a separate buffer in application, to copy that data. So that data pointer
won't be available to serial, till processing in application is over & will 
be returned by the application to serial again.

rts-cts hardware flow control
-----------------------------
While getting a free buffer, to replace in RX BD, rts-cts flow control can 
be done. After getting a free buffer, we will check, if the available free
buffers ( scheduled + free ) are <= 1/3rd of max buffers. If yes, then, it is
the right time, to turn off rts, as we are running short of free buffers to
receive the pkts sent by the remote serial side.

/* ...Note1 ends */
	
	After saving the data pointer, which we have obtained from free_buffers list,
we will call routine, which will add the current scheduled_rx_buffer (the
one, in which, data has been received), into current_rx_buffer list. Now
this current_rx_buffer is ready to be given to the application for processing.
Also now, we will replace the data pointer in the current RX BD, on which we 
have received, with the one, we have obtained & saved from free_list. 

	This process goes on, whenever we get a pkt from remote serial side. On the 
other hand, in one more path, i.e., in serial_foreground() which is being 
called periodically in while(1) loop of our operating_system function, we will
try to find out, if any serial current_rx_buffer exists, which needs to be given to
application, ( we will check this for all the wan ports ). If yes, we will try 
to pass those current_rx_buffers to applications.

	The application, after receiving the current_rx_buffers, will frame the 
required headers in the space reserved in data pkt & will try to send them to
tcp buffer thru socket send calls. If it is successful in sending those pkts,
it will return the current_rx_buffer back to the serial thru serial receive 
complete routines.

	If not successful in sending, application will malloc a new pointer, to copy
& retain this data pkt & then will return the current_rx_buffer back to the
serial. Here, we can pass a variable to serial receive complete routine, to
indicate tcp buffer full condition or free condition & thereby whether serial
can send data pkts or NULL pkts to application, from there on till congestion
is being overcome.
	
	These current_rx_buffers will now be added into free_buffers list, so that
they can be used again, in a cyclic manner. So at any time, total buffers
( free + scheduled + current ) will be 32 & scheduled will always remain 8.
Free & current may vary from time to time. During congestion, free may tend
to decrease to the lower max level & current may increase accordingly, as they
are not being passed to application.

/* Note2 begins... */

Note2 :
-------
	Now again rts-cts hardware flow control can be done while returning a current
rx buffer back to serial to be added into free list, from application. 
In case, if we find that, rts has been turned off, due to inavailability of
free buffers, we will now check, if the available free buffers ( again 
scheduled + free ) are >= 2/3rd of max buffers. If yes, then we can turn on
the rts, as we have enough free buffers now. 

/* ...Note2 ends */

	The data pkt given to or rather copied into proxy's tcp buffer, will be sent 
to the client thru tcp send calls.

	In this scenario, need to explore where we were actually losing data ? In 
which path or in which concept ?

1. from remote serial to proxy's serial ? 
		If that is the case, rts-cts hardware flow control is not fine ?!?!
2. from proxy's serial to application ? 
		If so, retaining mechanism is not proper ?!?!?!
			As well as retaining retain pkts mechanism may also be not proper ?!?!?
3. from application to proxy's tcp buffer ?
		If so, something wrong while calculating the index of the circular tcp
			buffer, to copy data, especially during boundary index values ?!??!!
4. from proxy's tcp buffer to client ?
		If so, tcp chksum error must have been sent by the client, even before
			chksum error at mew application level ?!?!?!?

Experimentation done for the problem
------------------------------------
* winMCSI client & Proxy in same physical lan.
* winMCSI client logged into proxy & mapped wan1 port ( dial-out type, ag 
	function type ) of Proxy. Opened mew application in that client PC.
* Proxy's wan1 direct connected to meterm of one PC.
* Baud at mew ( Network side ) -> 57600.
* Hardware flow control rts-cts enabled in mew.
* Baud at meterm ( remote serial side ) -> 57600.
* Hardware flow control rts-cts enabled in meterm.


** Tried transfering a zip/txt file of 1 MB size from remote serial side to 
	winMCSI Network client.


1. For the first path,

problem was in : from remote serial to proxy's serial ? 
------------------------------------------------------
	We tried explicitly, turning off & on, rts of proxy, thru break-out box
periodically. We didn't get crc 16/32 chksum error at those times. So this
path was ruled out from the cause for the problem.
	Conclusion : rts-cts flow control is fine.
	
2. problem was in : from proxy's serial to application ?
-------------------------------------------------------
	Yes!!!!!! Eureka !!!!!! We got it!!!! Will explain this path, at the last.

3. problem was in : from application to proxy's tcp buffer ? 
-----------------------------------------------------------
	We tried explicitly, to send half the length, of the original data pkt, 
at frequent intervals, to confirm, whether problem is  actually, when we fail
to send data from application to tcp buffer. As well as in packet retain 
mechanism. Due to failure in sending the entire packet length to tcp, we will 
retain those unsent pkt bytes. Even in those retained pkts,  we tried to retain
half of the length of the retained pkt, to find out, whether problem lies
in retaining retain pkts. With this experiment also, we couldn't reproduce
the problem. So, this path was also ruled out.

4. problem was in : from proxy's tcp buffer to client ?
------------------------------------------------------
	Without any experimentation, this path was ruled out. B'cos, if there was 
problem in this path, ( only error can be, tcp pkt getting corrupted exactly
between, before sending the pkt, after calculating crc & after 
sending the pkt, crc calculation fails to match with the sent one, for which
chances are very very meagre.) tcp chksum error might have been sent by the 
client. So that error has no chance in getting reflected in mew application
at all. Even before error in application, error in tcp level might have 
occured. Mew application won't be aware of that error at all. But what we 
actually got is, error at application & not at tcp level. So definitely, this
path has nothing to do with the problem.

Coming back to the problem area ,
--------------------------------
 yes, the problem was in the path, from proxy's serial to application !!!

* First we wanted to confirm, whether we were losing bytes or packets ?
	For this, we tried transfering a txt file with fixed pattern like :
	a1bcdefghijklmnopqrstuvwxyz1234567890a2bcdefghijklmnopqrstuvwxyz1234567890
	ranging from a1 to a50, repeatedly. 

	With this, we were able to find out in packet view, in proxy's lan, that, 
	we were losing few packets ( packets sent by serial, as a whole ) & not 
	bytes.

* With the above conclusion, it was made sure that, we were either losing packets 
	**	at serial level ( rts-cts flow control being done at wrong time ?
		due to wrong calculation of free buffer marking levels ( low - 1/3rd or
		high - 2/3rd ) ? )
	** or from serial to application ( retaining mecahnism not proper ? Losing
		pkts given by serial, somehow, during congestion ?)

	Losing pkts at serial level - from remote serial to proxy's serial
	------------------------------------------------------------------
		We tried to find out, whether something wrong, in the calculation of
available free buffers ? No, it was fine. Whether the total number of buffers
at any time, was summing upto 32. Yes, exactly !!!! Then, we tried increasing 
the low mark level (1/3rd to even 1/2th ) of available free buffers, by which
we will decide to turn off rts. Even with this, we got the same crc error.
This experimentation, gave us the clue, that, inspite of having free buffers
in serial level, for pkt reception, we were losing pkts. This means, we were
not losing pkts during pkt reception, but in the other path, i.e., from serial
to AG application.

	Losing pkts at AG application level - from serial to AG application
	-------------------------------------------------------------------
	In the above experiments, regarding, retaining mechanism, we were
able to conclude that, retaining mechanism as such was proper.	But some other
logical error ?!?!? Then we finally traced in the program that, as soon as,
we find some retained pkts in application, we were not at all bothering to
check, whether the pkts sent by serial to application was NULL or valid pkt ?
There lies the exact problem. We were assuming that, as soon as, we retain
a pkt in application, we will be setting the flag "send_status" to zero, 
meaning, asking serial to send NULL pkts  ( & not valid data pkts from 
current_rx_buffer ) there on. 

## Wrong assumption
-------------------
Our assumption that, serial will be sending NULL pkts, as soon as, we set the
send_status flag was totally wrong. Because, serial, while sending the data pkts
one by one from current_rx_buffer list, in a while loop,( until it encounters a 
NULL current_rx_buffer), simultaneously keeps receiving pkts from remote serial 
( because ISR's will be given higher preference ). So those pkts will also be 
given to application, though send_status is zero. B'cos send_status sent by the 
application to serial, will be seen by serial, only after coming out of the while
loop.

## Error path :
---------------
In the sense, in case, if we find any retained pkts, before calling 
rx_complete routine,	we were not storing/retaining	the pkt sent by serial.
We were not even checking whether data pkt sent by serial is NULL or valid 
data pkt, when we had retained a pkt already. We were assuming that to 
be NULL pkt. Here only, we were losing pkts!!!!!!

Solution to the above problem
-----------------------------
Now, changed in code, in agrx.c, serial_rx_callback() function, exactly at the
## error path. Here we are checking, whether data pkt, sent by the serial is
NULL or valid data pkt, though we had retained data pkts already. If valid 
data pkt, we try to retain that pkt in the same order, in retain link list. 
Now, pkts are not lost & are kept in tact in their sequence order.

