Bandwidth Management and Traffic Optimization
Part 1:
Traffic Shaping
I don't think that any South African broadband user can dispute the fact that we are still forced to pay high costs on our internet access while receiving limited connectivity in return. Whether you pay on a fixed cost leased line where your throughput is shaped or throttled down, or if you pay for a data bundle, you are still forced to conceed that bandwidth costs you a pretty penny to use. To extend this statement, if you are running a geographically dispersed network, where you have to openly share data among all of your WAN nodes, the high cost of dedicated leased lines, or the limited performance of ADSL upstream creates a situation where your limited capacity data lines easily become bottlenecks. In this situation one or two noisy applications or users can easily choke the traffic flow, slowing communication down to a trickle while latency climbs and network timeouts increase.
Logically the first thing you need to look at is the network performance and traffic queueing on your internet lines. One of the biggest problems on umanaged data lines is latency and lack of queueing discipline and the two go hand in glove. In order to get control over this aspect, you need to prioritize your traffic types and determine how much of your internet resource you want to allocate to them. In addition, you also need to ensure that all connections on your network get an equal slice of the pie, otherwise by default your gateway is going to work on a first come, first serve basis. Believe me, when the early user wants to download and archive a copy of the internet, this is bad news for everybody because he is going to get the whole pie to himself, while the rest of the network experiences high latency and poor connectivity!
Linux administrators have a few tools in their bag of tricks which can be deployed to ensure that you get the most value and performance out of your network. In this first part of the article, we will discuss some traffic shaping tools which are prepackaged and freely available in all popular linux distributions. The standard traffic management toolset which is used on just about every linux system is the Traffic Control suite of utilities, or tc as you would use it on the commandline. The Traffic Control suite can be fairly complex by nature and somewhat difficult to understand, so it is the purpose of this article to provide a few simple recipies and examples that will help you to get back control of your traffic.
The tc utility essentially works by establishing a hierarchy of traffic classes and subclasses for a given interface, each with a different set of rules which define that classes priority and bandwidth allocation.
EXAMPLE:
Based on the class we assign it, we could say that http traffic is allowed to use up to 50% of available bandwidth when the line is busy, but burst when additional bandwidth is available. At all times, treat http with high priority and respond quickly to reduce latency experienced when browsing. Whereas, by assigning mail to a different class, we can say that for all classified mail protocols we want to dedicate 33% of bandwidth to mail but allow it to burst if bandwidth is available. Since mail is not as interactive as web browsing, the user is unlikely to notice slightly higher latency, so we can also give mail a slighlty lower priority.
In order to achieve the above configuration, we need to establish at least 5 classes for traffic queuing:
Root Class 1:
|
Parent Class id 1:1
|___________________________________________________________________________
| | |
Sub Class id 1:10 (http) Sub Class id 1:20 (mail) Sub Class id 1:30 (other)
Normally you would still want to classify other traffic types and prioritisations, but for simplicity and to assist you in understanding how it all works, we will base our recipies on the above example. In conclusion, we will provide you with access to a script which provides more wholistic traffic control.
Root Class:
The root class is essentially where we define the basic form of the interface which is being used for internet access. If you are connecting via a 100Mbit network card, you obviously don't have 100mbit of internet throughput, nevertheless we define the root as 100Mbit so that the traffic control utility is more accurately able to shape, control and queue the traffic flows passing through the interface.
Assuming your internet interface is connected to an upstream router, it is a 100Mbit network card and its linux device name is eth2, the parent class would be set as follows:
tc qdisc add dev eth2 root handle 1: cbq avpkt 1000 bandwidth 100mbit
In plain English, we have just told the kernel to create a root queueing discipline for eth2 with the tag (handle) of 1: and we will use Class Based Queueing (cbq) to determine how traffic is classified, and how we want to handle it. Our average packet size will be 1000 bytes and our maximum throughput potential is 100Mbit. The avpkt and bandwidth arguments are mandatory, since tc will use these values to calculate its transmission time tables and idle times. It is essential that the bandwidth you specify is based on the actual bandwidth capability of the underlying physical interface itself, and not what you expect to achieve on your broadband connection.
Parent Class:
One thing to note for effective traffic control is that in almost all cases, the best results are achieved by shaping your outgoing traffic. You have more control over what you send than what you receive, and in most cases the latency you experience is actually caused by your inabilitity to transmit traffic out of your network to upstream servers due to intensive queueing and congestion on your router.
With the parent class we will define how much peak upstream bandwidth we expect our internet connection to yield to us. This is not an exact figure for every occasion. For instance, we may very well have a network card which is capable of 100mbit, but we are actually connected to an upstream provider which is offering us a 2mbit sychronous leased line. Yet again, while it seems logical to shape the parent class to 2Mbit transmission speed, this could be counter productive since your actual internet throughput given line performance and contention might actually be slightly lower than that. By defining your bandwidth as 2mbit, you could cause traffic to start queueing on the router connected to you network card. As the queuing increases on your router, you will see latency start climbing, and the advantages you are seeking to gain by traffic control will bleed away. In order to maintain control, you need to keep the queuing on your gateway server where you are applying traffic control. In fact, even if your actual measured throughput is 2Mbit, you might still want to set your parent class slightly lower in order to better control how much traffic your router actually queues and to reduce your latency while improving your packet transmission time. For this example the parent class would be defined as follows:
tc class add dev eth2 parent 1: classid 1:1 cbq rate 1800kbit allot 1500 prio 5 bounded isolated
Above we have created a new class on eth2 who will use the root class as its parent. This class has been assigned an id of 1:1 and we will use it as a parent for all other sub classes. We are still using Class Based Queueing (cbq) to match traffic to this class and we have set its threshold to 1.8mbit. It is bounded, so it cannot burst or borrow bandwidth from its parent, and it is isolated so it cannot lend its bandwidth to other classes in its hierarchical peer group (if any exist). It can dequeue traffic in 1500 byte chunks each round robin cycle with a priority classification of 5 (classes with the lowest prio are queued first in a round robin arramgement).
High Priority Subclass 1:10 (http):
In this child class we will define the performance parameters we want to assign to our http traffic. We will set it to 50% of available badwidth, while still allowing it to borrow unused bandwidth from its peers. We will also assign a slightly higher priority to this classes traffic:
tc class add dev eth2 parent 1:1 classid 1:10 cbq rate 900kbit allot 1600 prio 1 avpkt 1000
In the line above, we have assigned class 1:1 as the parent and limited it to a trmsfer rate of +/- 50% (900kbit). We have given this class an id of 1:10 and a very high priority of 1. The average packet size we transmit from this class is 1000 bytes (avpft) and we dequeue about 1600 bytes for this class on each round robin cycle (allot).
Medium Priority Subclass 1:20 (mail):
In this child class we will define the performance parameters we want to assign to our http traffic. We will set it to 33% of available badwidth, while still allowing it to borrow unused bandwidth from its peers. We will also assign a slightly lower priority to this classes traffic:
tc class add dev eth2 parent 1:1 classid 1:20 cbq rate 600kbit allot 1600 prio 2 avpkt 1000
In the line above, we have assigned class 1:1 as the parent and limited it to a transfer rate of +/- 33% (600kbit). We have given this class an id of 1:20 and a slightly lower priority of 2. The average packet size we transmit from this class is 1000 bytes (avpft) and we dequeue about 1600 bytes for this class on each round robin cycle (allot).
Low Priority Subclass 1:30 (assigned to other protocols you choose):
In this child class we will define the performance parameters we want to assign to traffic traffic types we use but don't need high throughput on. We will set it to about 25% of available badwidth, while still allowing it to borrow unused bandwidth from its peers. We are not specifically concerned with how much bandwidth this class has, giving it a much lower priority for queueing than its peers should be suffcient for the results we want:
tc class add dev eth2 parent 1:1 classid 1:30 cbq rate 450kbit allot 1600 prio 3 avpkt 1000
In the line above, we have assigned class 1:1 as the parent and limited it to a transfer rate of +/- 25% (450kbit). We have given this class an id of 1:30 and the lowest peer priority of 3. The average packet size we transmit from this class is 1000 bytes (avpft) and we dequeue about 1600 bytes for this class on each round robin cycle (allot).
Classifying Traffic for Shaping:
We have not yet assigned, or mapped any traffic to these queues. So far, all we have done is establish the queuing disciplines we will apply to our traffic.
Linux supplies a great tool for traffic classifcation in it's iptables packet filter suite in it's mangle table. So in order to start matching http to the group we prepared for it, we can use the following rule:
iptables -t mangle -A OUTPUT -o eth2 -p tcp -m tcp --dport 80 -j CLASSIFY --set-class 1:10
iptables -t mangle -A OUTPUT -o eth2 -p tcp -m tcp --dport 443 -j CLASSIFY --set-class 1:10
iptables -t mangle -A FORWARD -o eth2 -p tcp -m tcp --dport 80 -j CLASSIFY --set-class 1:10
iptables -t mangle -A FORWARD -o eth2 -p tcp -m tcp --dport 443 -j CLASSIFY --set-class 1:10
Rinse and repeat for all other traffic classes you want to specify. Using the above example, mail would be classified as class 1:20 and your low priority traffic will be mapped to class 1:30.
Congratulations! You are now one step closer to taking control of your expensive broadband and beginning to make it pay for you! The traffic shaper rules covered in this article as well as other worthwhile optimisations can be found in a script which has been circulating for a very long time. This script, called Wondershaper, was written by Bert Hubert in 2002 and is still very popular for basic traffic control today. You can download it here: http://lartc.org/wondershaper/wondershaper-1.1a.tar.gz
Read the instructions he provides for setting it up, apply the knowledge we have covered in this article when customising the script for your needs, and you will be on your way to some really low latency and top performance.
Of course, shaping your traffic queues and prioritasation is a great first step in decent traffic management, but there are still additional things you can do to squeeze the most out of your connection. With wide area networking and optimal performance in mind, wouldn't it be great if we can get maximum throughput on essential protocols across our WAN links? In the next artcle we will take a look at protocol compression as a means to get more data out of your data lines.
Stay tuned for part 2: Protocol Compression
Tags: bandwidth management, LARTC, Linux advanced routing and traffic control, South African bandwidth, traffic optimization, traffic shaping