Monday 5 December 2011

What is difference between Throughput Test and Frame Loss test ( RFC 2544 and RFC 1242)

Yesterday i came across this question  while some were performing these tests.  We know that for any networking device the "Throughput" is very important. The throughput claimed by vendor to vendor is different so there is one RFC which guides the method to be used to measure the throughput . The RFC is 1242 (Benchmarking Terminology for Network Interconnection Devices) and RFC 2544 (Benchmarking Methodology for Network Interconnect Devices)

The RFC 1242 gives all the definitions  or the terminology for the devices and the 2544 RFC explains the method to be used to measure some of the key parameters. In all it covers below tests:

1) Throughput
2) Latency

3) Frame Loss
4) Back to Back
5) System Recovery

6) Reset

I would recommend everyone who is performing anything related to performance tests should go through these RFC's.

One important point is the trail duration for which the tests is to be run. The trail duration varies and i have seen using very less trial duration and quoting the values. The RFC recommends trail duration of 60 second for tests which uses binary search like throughput tests and frame loss tests. If you use less trail duration obviously you will get more throughput which may not be accurate.

Ok so now back to both the tests . Lets see what is throughput and frame loss of the device as per RFC 1242 and RFC 2544:

=============================================================
Throughput: The maximum rate at which none of the offered frame 
are dropped by the device.

Frame Loss Rate: Percentage of frames that should have been forwarded
                by a network device under steady state (constant)
                load that were not forwarded due to lack of
                resources.       

26.1 Throughput

   Objective:  To determine the DUT throughput as defined in RFC 1242.

   Procedure:  Send a specific number of frames at a specific rate
   through the DUT and then count the frames that are transmitted by the
   DUT. If the count of offered frames is equal to the count of received
   frames, the fewer frames are received than were transmitted, the rate
   of the offered stream is reduced and the test is rerun.

   The throughput is the fastest rate at which the count of test frames
   transmitted by the DUT is equal to the number of test frames sent to
   it by the test equipment.
   

26.3 Frame loss rate

   Objective:  To determine the frame loss rate, as defined in RFC 1242,
   of a DUT throughout the entire range of input data rates and frame
   sizes.

   Procedure:  Send a specific number of frames at a specific rate
   through the DUT to be tested and count the frames that are
   transmitted by the DUT.  The frame loss rate at each point is
   calculated using the following equation:

          ( ( input_count - output_count ) * 100 ) / input_count


   The first trial SHOULD be run for the frame rate that corresponds to
   100% of the maximum rate for the frame size on the input media.
   Repeat the procedure for the rate that corresponds to 90% of the
   maximum rate used and then for 80% of this rate.  This sequence
   SHOULD be continued (at reducing 10% intervals) until there are two
   successive trials in which no frames are lost. The maximum
   granularity of the trials MUST be 10% of the maximum rate, a finer
   granularity is encouraged.
 
 ======================================================================
 
So if you Observe that both are saying almost similar thing, we need to
find a rate where there is no frame loss. 
 
Both uses binary search to find the rate. ( depends on vendors test devices ) 
 
Both gives you result where no frame loss is observed. 

so what is the difference between the two ?

The difference is only in the interpretation of the result and the way the 
tests is performed. I could find following differences :
 
1) The trial duration for throughput tests should be 60 sec(recommended),
  
2) Frame loss test is based on configurable total no of frame transmitted at max rate.
   Throughput test the calculation of frames is based on the rate configured. 

3) Generally Frame loss rate and passed throughput rate will be same with very 
 less difference and it is because on the duration for which the tests are run.
 
4) Frame loss test can give you results, which tells that at that particular rate 
what is the frame loss. 
 
5) Frame Loss test comes when there are routing priorities,sequenced flow is 
 important and congestion is present in network. 
 
In  Normal scenarios The results for both the tests are same.