Hi,  
  
I have been writing a Communications protocol component in C# that will  
  
basically be used in client and server applications. The sole purpose of the  
  
component is to allow applications to be able to RELIABLY send  
  
structures/objects across a network (LAN) to other applications that also use  
  
the component. Objects are encapsulated into a packet which has a header to  
  
identify to the receiving end what type of packet it is.  
  
The way the component has been implemented at this point is explained below:  
  
- On sending, the component serializes the object in wants to send into a  
  
stream of bytes (packet) first and then sends it by calling the Socket  
  
object's synchronous Send() method. Before returning, this send process needs  
  
to make sure that the other end received the packet (this is where the  
  
reliability part comes into the picture). To know that a packet was received  
  
correctly, the send process waits 200ms (RTO - ReTransmissionTimeout) for an  
  
ACK packet to come back (the reception of this ACK packet is explained below  
  
in the receive section**). If it doesn't receive an ACK by then, it resends  
  
the same packet. It continues to resend the packet every 200ms for 30 secs.  
  
If it still hasn't received it by then, it disconnects the connection to the  
  
remote endpoint.  
  
- On receiving, bytes are received asynchronously by calling the Socket  
  
object's BeginReceive() and EndReceive() methods. The receiving process  
  
blocks until bytes become available on the receiving port. So when bytes do  
  
become available on the receiving port, they are read and deserialized back  
  
to the original object. In order not to miss any further incoming bytes, the  
  
currently received object is passed to another thread for processing. Hence  
  
the receiving process can return and call BeginReceive() again and wait for  
  
more bytes to come in.  
  
Now, the new thread that just got spawned will process the packet.  
  
Depending on the header of the packet, this thread will have different ways  
  
in dealing with the packet. Say we have 2 types of packets - DATA and ACK. If  
  
the receiving end received a DATA packet, then it sends back an ACK to the  
  
sender to signify correct reception.** But if the packet was an ACK, then it  
  
sets a synchronize flag to let the sending process know that an ACK was  
  
received for a particular packet.  
  
This component is now being tested with a TestServerApplication and a  
  
TestClientApplication. The server simply waits for clients to connect. When  
  
the client connects, the server accepts it and waits. The system was set up  
  
such that the client app sends a 2KB packet to the server and the server  
  
basically echoes it back. When the client receives the echo, it also echoes  
  
that back and so they play "packet tennis". This is simply to test the  
  
component. The size of an ACK packet is 44 bytes. So this is what happens:  
  
- client sends 2KB to server  
  
- server receives and sends a 44-byte ACK, and then the echo 2KB packet to  
  
the client  
  
- client receives the ACK (which ends the first send), and then also  
  
receives the echoed packet... client then sends an ACK to the server, as well  
  
as the echo packet  
  
- and the process repeats...  
  
Everytime the client app receives an packet, it prints out the contents on a  
  
RichTextBox and then clears it for the next packet. So visually, the client's  
  
RichTextBox blinks with the contents of received packets. When i have this in  
  
the code, it works fine. However, if i comment out the section that prints  
  
the packet contents (so the RichTextBox will always be blank), something  
  
bizarre occurs:  
  
For some reason, at certain points in time, the asychronous Receive calls  
  
(at the client end) receive bytes that are of inconsistent size and it always  
  
seems to be decreasing by 44 bytes - the size of an ACK packet. So at first,  
  
the component may consistently receive 2048 bytes, but then later on it  
  
receives 2004, then 1960, then 1916, and so on .. this will obviously  
  
register as incomplete packets on the component and cause errors. This  
  
behaviour only happens when the code which prints out the packet contents on  
  
the client app is not included. But if it is, the thing works fine and the  
  
packet sizes being received are all 2KB. What is going on here??  
  
Please excuse the lengthy post. I needed to explain how my component worked  
  
for easier analysis on your behalf. Thanks in advance.  
  
Michael--J.

Michael--J wrote:  
  
Instead of implementing your own solution, you could use remoting.  
  
Avoid async-IO unless absolutely forced to do it. It is *very*  
  
complicated to get right -- really!  
  
Timing influences the way multi-threaded and asynchroneus programs work.  
  
Try making a *very* small example program and do your tests on that,  
  
so you can understand what's going on.  
  
You can try insert some Thread.Sleep(0) in order to try and provoke some  
  
timing errors by having the threads rescheduled.  
  
Could easily be a synchronization bug in your async-IO. It's *very* hard  
  
to get async I/O right.  
  
Are you allowed to read and write to the Socket from multiple treads at  
  
once? I suspect not. Perhaps writing the ACK to the socket moves a  
  
pointer in the Socket's buffer -- which would explain the behaviour you  
  
are experiencing.  
  
I would guess that most of the content of the post is unrelated to your  
  
problem.  
  
BTW1: I don't really understand why you are using async IO for reading  
  
the input. you could just as well use sync, with exactly the same result.  
  
BTW2: Are you certain you will always receive 1 full packet per read of  
  
the Socket?  
  
--  
  
Helge Jensen  
  
mailto:helge.jensen@slog.dk  
  
sip:helge.jensen@slog.dk  
  
-=> Sebastian cover-music: http://ungdomshus.nu <=-

Michael--J wrote:  
  
That's understandable.  
  
With async-io there really isn't much of a "program-flow", which makes  
  
it very hard to use the usual ways of understanding/arguing how a  
  
program works.  
  
The "Thread.Sleep(0)" is an old hack to try and provoke synchronization  
  
errors by littering the code with thread-switches. Thread.Sleep(0) (in  
  
win32 atleast, not sure about .NET) yields the rest of a proccess'  
  
timeslice to another thread.  
  
Maybe an instrumenter that puts sleep-calls inbetween the original .NET  
  
code would be nice to have for testing multi-threaded programs..... hmmmm...  
  
The problem will propbably not go away, but you will have an improved  
  
chance of finding out whats going on.  
  
If writing to the textbox/console/... is too slow to have the program  
  
exhibit incorrect behaviour you can make an array of logs, something  
  
like this:  
  
public class CyclicLog {  
  
public object[] Logs;  
  
int current;  
  
public CyclicLog(int count) { this.Logs = new object[count]; }  
  
public void Log(object o) {  
  
lock ( this ) {  
  
Logs[current] = o;  
  
current = (current + 1) % Logs.Length;  
  
}  
  
}  
  
}  
  
So you can have a kind of program-flow trace while still getting the  
  
incorrect behaviour.  
  
It may help, but only indirectly -- by reducing the amount of concurrency.  
  
Instead of the pattern:  
  
ReadHandler(AsyncResult r) {  
  
...  
  
s.BeginRead(..., new AsyncCallBack(ReadHandler));  
  
}  
  
void Main() {  
  
s.BeginRead(..., new AsyncCallBack(ReadHandler));  
  
}  
  
you can do:  
  
ReadHandler() {  
  
while(...) {  
  
int read = s.Read(...);  
  
...  
  
}  
  
}  
  
void Main() {  
  
new Thread(new ThreadStart(ReadHandler));  
  
}  
  
Don't expect it to have any influence on the behaviour (it's essentially  
  
the same code, just by different means), but the code becomes a bit more  
  
understandable, especially in the face of exceptions.  
  
From the docs of {Stream,NetworkStream}.BeginRead:  
  
"Multiple simultaneous asynchronous requests render the request  
  
completion order uncertain."  
  
So it seems you are allowed to do multiple reads.  
  
OK, so that pit isn't waiting for you, good ;)  
  
As a last resort, simplify the program while keeping it doing the  
  
invalid behaviour... it's a timeconsuming but effective technique.  
  
--  
  
Helge Jensen  
  
mailto:helge.jensen@slog.dk  
  
sip:helge.jensen@slog.dk  
  
-=> Sebastian cover-music: http://ungdomshus.nu <=-