Valid HTML 4.01 Transitional

VoIP Versus QoS

James F. Carter <jimc@jfcarter.net>, 2016-09-05

I am installing VoIP service (Voice over Internet Protocol), and I want to follow best practices in my house router to get the best call quality and reliability. An important factor is QoS (Quality of Service), referring to the Type of Service byte in the IP header (see RFC 2474), and I need to renovate my network traffic control structures to take RFC 2474 into account. So what exactly should I do for a network traffic control strategy?

VoIP and QoS Basics

How VoIP Works

For VoIP the client signals to its server or peer who it wants to talk to, and other connection parameters like picking a codec, using SIP (Session Initiation Protocol). Then each end digitizes the voice signal, i.e. what the speaker is saying, and sends it to the other end, generally using RTP (Realtime Transport Protocol). Both of these are normally handled as UDP (Unreliable Datagram Protocol) with a timeout-based retransmission fallback for SIP and no retransmission for lost RTP packets. It is particularly important that SIP packets not be lost, and that RTP packets both are not lost and are sent promptly.

Latency is the jargon for how long it takes for a packet to get from the sender to the recipient, although the round trip time is sometimes incorrectly referred to as the latency. Per ITU-T G.114, a latency of 150msec or a round trip time of 250msec is about the maximum before users complain. Jitter refers to how much the latency varies, and a uniform latency (low jitter) is also important.

Differentiated Services (QoS)

IPv4 is specified in RFC 791 and IPv6 in RFC 2460, both as amended. Each of them provides a byte in the IP header, historically called ToS (Type of Service). Its current interpretation comes from RFC 2474, plus RFC 3168 for ECN (Explicit Congestion Notification). The 64 code points described in RFC 2474 are referred to as Differentiated Services. Each cues the router to apply a different PHB (Per Hop Behavior) to a packet with that DiffServ code point. RFC 2474 does not define what the PHBs should be; in fact the RFC assumes that different enterprise networks may have different implemented PHBs and different code points representing them. But RFC 2474 says that PHBs should be numbered so higher numbers imply a higher probability that the packet will timely go out. (Dropped packets are not timely.)

QoS Code Points and PHBs

However, RFC 2598 recommends (not requires) a code point 0x2e (0xb8 shifted) that requests Expedited Forwarding, i.e. the packet should go out quickly and reliably as wanted for RTP. RFC 2597 recommends a set of code points for Assured Delivery (try not to drop), of which 0x1a (0x68 shifted) is the one that seems most appropriate for SIP. A code point of 0 is the default, and such packets get no special treatment. RFC 1349 (deprecated) was the previous incarnation of ToS, and some applications may use its code points, which have limited backward compatibility with RFC 2474.

The DiffServ scheme gives a different QoS (Quality of Service) to differently marked packets, and the whole scheme is usually referred to as just QoS.

Hogs and Frogs

Let's step aside for some unofficial terminology. A flow refers to a sequence of packets (one way) between particular host/port pairs: for example a file download or upload, or a VoIP call (one flow in each direction). A flow that can saturate the network, i.e. which always has a packet needing to go through, is called a hog. Typically downloading a file or an advertisement on the web is a hog. A frog sends packets at a modest rate limited by the data source or sink, and almost never has a packet waiting, but usually has an operational requirement that when there is a packet it should go out right away. VoIP is the archetypical frog; also interactive terminal sessions and network games are frogs. Streaming audio or video is also in the frog category in that the effect is spoiled if a saturated network link limits packets to less than real time, but their data rates tend to be higher.

Data Rate Regulation in TCP

TCP (Transmission Control Protocol) regulates the rate of data flow by a sliding window. When a packet arrives, the recipient sends back an ACK (acknowledgement) packet announcing how much of the data stream has been received error-free, and how much more the recipient can accept. When the client process, e.g. a web browser, reads some of the incoming payload, that many more bytes can be added to the window, while when a packet comes in, its size is subtracted from the window. The client's operating system adjusts the window size for its convenience in disposing of the packets, and it automatically adjusts for how many packets are flying through the Internet, sent but not yet received. Packets can possibly arrive out of order, but if the sender sees in the ACKs that the recipient is not making progress, it will retransmit the missing packet(s). Then the recipient knows that the net is congested, and it shrinks the window to reduce the flow. It is common practice for routers to drop packets if the data link's maximum rate (or the SLA (Service Level Agreement)) is exceeded. The ECN bits (Explicit Congestion Notification, RFC 3168) give the same effect without wasting a packet that the net has already done some work on.

A TCP connection is bidirectional, and the packets in one direction contain ACKs for the other direction. But almost invariably, one direction has a much higher data rate. For example in HTTP (web), the desired URL is sent in the forward direction and the much larger text or image comes back. The client needs to continue sending ACKs until the whole text or image is received, after which it will send more data in the forward direction requesting another component of the web page. The incoming web page is a hog flow while the forward stream is froggy, and if the webserver is going to fill the available bandwidth it needs to promptly see the ACKs from the client, so it knows it can send more payload.

What Happens Without Traffic Control

In the shadowy past, 2009 or 2010, my son was playing a networked game and I started downloading the new version of my Linux distro. He justifiably complained, but this motivated me to learn about and implement traffic control. A major issue is that routers between the source and sink, not under my administrative control, often have big buffers, and if there is a hog flow it will fill the buffer, so frog packets have to percolate through the buffer. In the game vs. distro situation the measured round trip time (2x latency) went up to over 2 seconds while it was normally more like 10msec (to a geographically nearby server). With traffic control, in the simultaneous hog and frog situation the frog's round trip time returned to the normal value of 10 to 20 msec despite the hog flow running flat out.

Testing Actual Network Behavior Today

When preparing for this upgrade I did an empirical test to see which QoS code points were in actual use, and particularly, what our partners send us. Actually RFC 2474 allows intermediate routers to translate QoS code points, and in some cases this means setting QoS to zero. These services always had a QoS code point of zero: incoming SMTP (mail), incoming and outgoing HTTP and HTTPS (web), outgoing FTP, DNS, IPv6 tunnel, SSH-slogin. These outgoing services had nonzero QoS but in all cases the replies had a QoS of zero: SIP (0x68), RTP (0xb8), scp (0x10, should be 0).

Conclusion: None of our remote partners was ever seen to use QoS. Although attention to QoS on our router is a best practice, it will have almost no effect on service quality as seen my me and my users. My ATA (Analog Telephone Adapter) for VoIP is setting the normal QoS. SSH-slogin-scp needs to be configured to set the right QoS code points.

I have positive confirmation that my ATA turned on QoS in outgoing packets, but QoS had been cleared to zero when it arrived at the server. (This is legal according to RFC 2474.) I would be surprised if the server left out QoS, but I don't have positive confirmation that the bits are being cleared in the incoming direction. In any case, QoS is useless for picking the PHB to use on a packet.

Then I did a functional test with one hog, testing both directions, and several frog services, using several traffic control structures. The frogs were also tested without the hog. The hog task was for either the remote site or the router machine to download a giant file using http (wget). The frog tests were:

The only condition noticeably different from the others was sending out the hog flow with no traffic control at all: ping round trip time went up to 48.8ms and jitter was 21.0ms, but no other effect was seen on the frogs. We achieved 96% of our SLA speed (Service Level Agreement). If I were using DSL I would have seen a lot higher round trip time. Almost certainly our FIOS ONT is enforcing the SLA speed, and is doing a lot more: it is actively putting the frog packets ahead of the hogs. This is not surprising because a part of the product package is streaming video on demand, and frog abuse would kill the video quality.

So should I implement QoS or an alternative on my own router? Or should I rely on traffic control in the ONT? In security the rule is, don't trust unknown outsiders; see to your own security. This policy is probably a best practice for QoS also. More important, each router individually needs to avoid detaining frog packets among buffered hog flows traversing it. So I should continue to implement traffic control on my router. But I should not try to set up a complicated optimum design when I won't be able to test it because the ONT will compensate for any of my features that are inoperative.

Conventional Wisdom Deflated

Let's go through conventional wisdom about VoIP as seen in a lot of blog postings and product hype for professionally installed VoIP systems.

You've got to have QoS.

I don't see QoS code points being set by partners where they would make a difference. I'm going to ignore QoS, i.e. the DiffServ field, and I'm going to achieve quality service by providing enough capacity that the VoIP flows are frogs (which is easy: 1 kbyte/sec with the G.729 codec) and by helping all the frogs to hop quickly.

You've got to allocate bandwidth.

No you don't; a rigid allocation can choke one category when another is not using all of its allocation. You need to identify the flows that need to go out promptly and give them priority. Hog flows whose bandwidth is limited by network capacity clearly have to be given the lowest priority; otherwise they can and do squeeze out the frog flows. And this priority is independent of the unpredictabe bandwidth of the categories or of the whole net segment.

You've got to have VLANs.

They're really saying that VoIP traffic has to be isolated on its own VLAN. But this is the same fallacy as allocating bandwidth specifically for that VLAN (if your switches support that), with the added problem that a clog-up on the real LAN affects all the VLANs.

The value of VLANs is for administration, so you can manage hosts and apply policies more easily in smaller groups, and for defense in depth (security), so a compromised host has fewer neighbors in a broadcast domain that it can attack.

You've got to isolate VoIP from the hogs.

No you don't: you should not leave bandwidth unused; you should send hog flows on it, just so long as the frog packets get priority to go out.

Issues in Traffic Control

How should I design my traffic control? To achieve good quality of service, clearly I will accomplish very little if I pay attention to the QoS field because remote partners don't set it or it gets cleared in transit. Instead I should distinguish frogs from hogs and help them get through promptly, whatever QoS they have requested (if any).

My actual network is very small, but there aren't too many example traffic control scripts posted, and they seem to get copied from one blog post to the next. When reasonable, I want to make my solution apply to larger nets, and so I will be discussing net situations that will not actually occur (and be tested) on my own net.

Definitely I'm going to use fair queueing on the hogs. This means that they take turns sending packets in a round robin pattern. That means that they will all tend to the same data rate: SLA/N if there are N hogs. (SLA means the network capacity or the service level agreement rate.)

On my net there will almost always be zero or one hog flow, and frog flows will turn up at random intervals. But the natural state of a network segment in the enterprise is to be saturated with multiple hog flows, with multiple frogs trying to squeeze between them. Clearly the first step is to distinguish frogs from hogs. I'm setting a rate per flow of 1/4 of the SLA rate as the upper bound for the frogs. In an enterprise net this upper bound has to be set substantially less than SLA/N, lest the N hogs be treated mistakenly as frogs.

The TCP slow start algorithm starts the connection with a small window (and resulting data rate) and then increases it gradually. Thus a future hog flow will start out slow enough that it won't squeeze out the real frogs. But as the flow rate rises the local rate estimator should recognize quickly enough that it's grown into a true hog, before it gets fat enough to damage the frogs. In an enterprise net you will need to tune the time constant on the estimator, and set the frog upper bound low enough that the growing hog flow gets flipped into the hog category before squeezing out the actual frogs.

Since systemd-217, fq_codel (with default parameters) is the default queue discipline whenever an interface is newly brought up. Formerly it was pfifo_fast. Arch Linux wiki article on advanced traffic control mentioning this new default. It was released on 2014-10-28 though distros would adopt it somewhat later.

I'm going to use ECN on hog flows, which is negotiated with the partner. So ECN has to be turned on by setting /proc/sys/net/ipv4/conf/all/tcp_ecn. Values are 0 = never use ECN, 1 = always ask the partner to use it, or 2 = don't ask, but use it if the partner asks. At least in OpenSuSE Leap 42.1, the default is 2. The standard way to set one of the /proc/sys controls is to put a line in /etc/sysctl.conf like net.ipv4.conf.all.tcp_ecn = 1. This setting controls both IPv4 and IPv6. Then one of the boot scripts executes /bin/sysctl -p.

Detailed Traffic Control Design

A very simple design for the outbound traffic control goes like this:

Generic Egress Queue Discipline

This traffic control set is applied to several network devices.

Code for Generic Egress Queue Discipline

Ingress Filter

The generic queue discipline code is nice but it only works on outgoing packets and I'm not doing traffic control on the fast local LAN, just the slower wild side interface. What about incoming packets? Incoming frogs are going to get stuck in the ONT's queue too, if not protected, and the higher data rates are expected on ingress (downloads). Back in 2009 when I first wrote this script, there was a motif which I copied which attaches a simple policing filter to the ingress stream, and drops packets that exceed the configured data rate (95% of SLA). That's better than nothing, but far from ideal when VoIP is involved.

Since 2009 a lot of advances have been made. In particular, you can create an Intermediate Functional Block (fake net interface), flip the incoming packets onto it, and put the generic egress queue discipline on it. Here is how to make that happen.

IPv6 Traffic Control

I have IPv6 service via a tunnel to Hurricane Electric. As previously noted, traffic control would be a lot simpler with native IPv6, which would be handled by the generic traffic control. But my ISP is firmly rooted in the 1900's, so this is not an option. And my family does quite a lot of traffic with IPv6-capable sites, so I need traffic control on the tunnel.

As it turns out, I can just call the subroutines to put the generic egress and ingress traffic control on the tunnel endpoint and a second IFB for ingress. Problem solved.

Should there be more than one band for the frogs? No, I've decided. In this design the frog packets go out right away, and there is supposed to be plenty of capacity to handle them. Thus if there were multiple frog bands an incoming packet would go in one of them, and go right out; very rarely would multiple frog packets be waiting so a choice between them would be relevant. So I'm using only one frog band. But if you are sending out streaming video, which is a frog flow of higher bandwidth, the probability of multiple packets waiting might be higher, and it might make sense to put in a fair queueing module for the frogs, so flows of a similar rate would make similarly even progress.

I debated whether to post the complete traffic control script, or to make the reader write the script himself. Too often people act like script kiddies, drop a pre-made admin script or security recommendation into their system, get burned, and complain. It's very important to understand what the script is trying to accomplish, and to adjust the parameters and strategies according to what's appropriate on your network. In particular, not everyone will have tunneled IPv6 service (I hope that means most people will have native IPv6). Anyway, here are links to the script and a systemd unit file.

Test Results

I did a final test of hog versus frog performance with this traffic control script. The hog task was to download (In) or upload (Out) a file of about 80Mbytes; the partner is in the same city (Los Angeles). The frog task was to ping the same host 10 times using the same family (IPv4 or IPv6) as the hog. The average round trip time is reported, as well as the jitter, defined here as the difference between the best and worst ping time. There were no lost ping packets. For the hog flow, MiB/sec means 220 bytes/sec, whereas Mbit/sec means 106 bits/sec. The Service Level Agreement is 50 Mbit/sec and the Token Bucket Filter is set for 95% of that, or 47.5 Mbit/sec.

Family MiB/sec Mbit/s Ping ms Jitter ms
IPv4 None 0 0 10.3 3.8
IPv4 In 5.42 45.5 13.5 6.0
IPv4 Out 5.41 45.4 12.3 3.5
IPv6 None 0 0 29.2 6.0
IPv6 In 1.97 16.5 27.7 4.5
IPv6 Out 5.08 42.7 29.1 3.1

The frog flow (ping) was not affected at all by the hog; differences (within a family) are not significant. IPv6 takes longer because the packets run through the tunnel up to Fremont (near San Francisco) and back to the partner in Los Angeles via native IPv6. On IPv4 the hog achieved 91% of SLA. On IPv6 the upload speed was almost as good, but the download speed was only about 1/3 SLA. Almost certainly the issue is that the Hurricane Electric free tunnel service is shared among quite a lot of users, and we're seeing a capacity limit, not a defect in my traffic control. It's notable that the frog flow was unaffected despite cloud clogging. Hurricane Electric must limit their buffer size and may have their own traffic control that identifies frogs and gives them priority.

The Conform-Exceed Parameter in a Police Filter

Unfortunately in my distro (OpenSuSE Leap 42.1) I have iproute2-4.2 but the documentation has not caught up with the functions available in the code. (On Google search for tc police man page.) In particular, in a filter with a police action you can specify precisely what is going to happen if a packet does or doesn't exceed the data rate limit. You specify this with the option conform-exceed EXCEEDACT[/NOTEXCEEDACT]. The conform action is optional, and the exceed action comes first, even though it appears second in the keyword. Remember that one filter could have several actions. I did some empirical tests to see what the outcome choices actually do; they were tested alone, not with multiple actions. From the man page, the available actions are:

Index of Traffic Control Modules

Single Queues

tc-bfifo
FIFO queue, max size in bytes.
tc-pfifo
FIFO queue, max size in (bytes/MTU) i.e. packets.
tc-red
When approaching the queue size limit, drops more and more packets with random selection.
tc-sfb
Improves RED.
tc-choke
Improves RED.
tc-codel
Determines the drop probability from the residence time in the queue. Better than RED. Considered to be currently the best algorithm.

Array of Queues

Classy means the queue discipline can have several classes with different priorities. All of these queue disciplines are classy. They need an inner queue discipline without classes.

tc-tbf
Limits output to a given bandwidth.
tc-sfq
Each flow sends round robin.
tc-drr
Round robin dequeuer, improves SFQ.
tc-fq_codel
Round robin dequeuer with builtin codel classes.
tc-htb
Classy, can guarantee bandwidth to classes. Uses TBF subclasses.
tc-cbq
Classy, prioritizes, shapes. HTB is more modern.
tc-hfsc
Precise bandwidth and delay allocation, shares excess allocations.
tc-prio
Classy, strict priority order, can classify packets by RFC 1349 ToS code points (deprecated) (optional).
tc-mqprio
Classifies packets as needed by certain hardware devices.
tc-pfifo_fast
Similar to tc-prio, uses RFC 1349 ToS code points (deprecated).

Auxiliary Functions

tc-bpf
Berkeley Packet Filter to create a classifier etc.
tc-ematch
Matcher, e.g. cmp u8 at 1 mask 0xfc gt 0x40
tc-stab
Add overhead to the packet size.

Web Resources About QoS

The purpose of QoS in our use case is to send out VoIP packets promptly, delaying competing high volume traffic like Youtube videos and big downloads.

www.voip-info.org about QoS

www.voip-info.org about Linux traffic shaping

Their script uses:

	$TC filter add dev ${DEV} parent 1: prio 1 protocol ip u32 \
	    match ip tos 0x68 0xff \
	    match ip protocol 0x11 0xff \
	    flowid 1:1
	
This is attracting TOS 0x68, and another one for 0xb8. Minimum delay 0x10 is put in the next lower level.

www.tmcnet.com about what to do when deploying VoIP

Before deploying VoIP, assess whether QoS actually does something. Send synthetic VoIP with and without QoS turned on and see if it really helps. Determine your maximum expected load and generate this amount of data. Measure VoIP metrics (latency, jitter, packet loss) and save the result for later comparison. Monitor regularly after deployment. Compare with your baseline. Recognize developing problems and fix them right away.

www.sans.org about issues in VoIP

RFC 2474

Wikipedia: Differentiated Services

List of typically implemented PHBs (per hop behaviors). The code points given here are what you would or into the TOS octet, but in documentation they are frequently given relative to the 6 bit DS field and thus have to be shifted left 2 bits for use in actual code. E.g. documentation specifies 0x2e, traffic shaping rule needs 0xb8, the latter is shown below.