Tips from an engineer at Ookla

My name is Brennen Smith, and as the Lead Systems Engineer at Speedtest by Ookla, I spend my time wrangling servers and internet infrastructure. My daily goals range from designing high performance applications supporting millions of users and testing the fastest internet connections in the world, to squeezing microseconds from our stack — so at home, I strive to make sure that my personal internet performance is running as fast as possible.

I live in an area with a DOCSIS ISP that does not provide symmetrical gigabit internet — my download and upload speeds are not equal. Instead, I have an asymmetrical plan with 200 Mbps download and 10 Mbps upload — this nuance considerably impacted my network design because asymmetrical service can more easily lead to bufferbloat.

We will cover bufferbloat in a later article, but in a nutshell, it’s an issue that arises when an upstream network device’s buffers are saturated during an upload. This causes immense network congestion, latency to rise above 2,000 ms., and overall poor quality of internet. The solution is to shape the outbound traffic to a speed just under the sending maximum of the upstream device, so that its buffers don’t fill up. My ISP is notorious for having bufferbloat issues due to the low upload performance, and it’s an issue prevalent even on their provided routers.

As a result, I needed the ability to shape traffic over 200 Mbps speeds — this prevented me from using MIPS or ARM based routers, as they don’t have the CPU horsepower to route over ~150 Mbps without hardware offload (I was actually using Tomato on an Asus AC68U at the time). Very few routers provide the ability to shape a single direction of traffic in software, thus I had to find a solution that could handle bi-directional shaping over 200Mbps. While many of the Ookla Engineering team use Ubiquiti Edge Routers, their CPU limits their traffic shaping performance to the following:

• ERLite-3 and ERPoe-5: below 60 Mbps most likely will work, above 200 Mbps most likely will not work
• ER-8: below 160 Mbps most likely will work, above 450 Mbps most likely will not work
• ERPro-8: below 200 Mbps most likely will work, above 550 Mbps most likely will not work
• ER-X and ER-X-SFP: below 100 Mbps most likely will work, above 250 Mbps most likely will not work

Editor’s note: Since this article has been published, it is now possible on recent firmwares to perform traffic shaping in a single direction on the EdgeRouter platform.

Requirements

Thus, my router requirements were as follows:

1. x86-64 based hardware with a TDP less than 15w.
2. Strong support for native IPv6—many studies have shown
   IPv6 leads to a faster web browsing experience.
3. Ability to perform Point to Point VPN and Split VPN tunnels.
4. 802.1Q VLAN Tagging — I run three separate logical networks that correspond to respective SSIDs on the APs:
   • LAN network: the normal network we use in daily life. Has access to split VPN tunnels, Sonos devices, FreeNas storage server and Xen hypervisors.
   • GUEST network: network we place guests on, has no access to other networks/resources and is inbound rate limited to 100 Mbps.
   • IOT network: network for IOT devices, has no access to any other networks beyond WAN and is inbound rate limited to 5 Mbps. This is split off for security reasons, and the IOT devices we use (
     TPLink Smart Home, alarm system, security cameras) handle NAT translation without issue. (Note: these are affiliate links.)

I grabbed this small x86–64 server and combined it with 4GB of Kingston DDR3L and a 32GB Adata SSD. The key points to this server are fewer but higher frequency cores and 4xGBE Intel NICs. Intel NICs have some of the best support in the Open Source world, and it’s highly recommended to stay far away from the cheaper companies. This machine doesn’t have AES-NI or Intel Quickassist, but it hasn’t had any issues with encryption/decryption at line rate for the VPNs.

The actual routing

Once assembled, I installed PFSense 2.3 for handling the actual routing. For those who haven’t used PFSense, it’s an incredible routing operating system that is based on FreeBSD. It easily met the requirements above, and vastly surpassed them. I was able to apply CodelQ AQM shaping to outbound traffic to prevent bufferbloat, along with splitting the ISP provided IPv6 /60 into /64’s for my 3 VLANs.

In my research and testing, I also evaluated IPCop, VyOS, OPNSense, Sophos UTM, RouterOS, OpenWRT x86, and Alpine Linux to serve as the base operating system, but none were as well supported and full featured as PFSense. The closest runner up to PFSense was VyOS as I love the declarative CLI interface and read only primary/backup partition system, but there were a few reasons which blocked me from using it:

• VyOS doesn’t support IPv6 Prefix Delegation in the stable branch.
• The stable branch is based on Debian Squeeze, which is quite old. There’s a Debian Jessie version, but it’s considered experimental.
• Sadly, their development team and pace has shrunk considerably since the initial Vyatta fork.

PFSense isn’t without its issues, but it’s perfect for my use case. The biggest issue I had was the default DNS configuration. On PFSense, the DNS server (unbound) is set to function as a recursive resolver rather than a forwarding server. While this might have a security benefit in edge cases, the performance impact on lookups is substantial — web browsing was jerky as domain-sharded assets had slow lookups.

QoS Settings

Some people have asked what QoS settings I use in PFSense. I avoided the default wizard QoS settings because in general, I try to avoid proto/port classification. The majority of traffic on the modern web is TCP 80/443 with a smattering of UDP 53, so HSFC class based QoS isn’t as effective as it used to be. However, every case is different, so I’d love to hear about your rule setups.

I essentially emulated FQ-CODEL by placing a FAIRQ scheduler in front of a CODELQ queue. CODEL is capable of prioritizing streams and dropping packets when backoff is necessary, so it’s been highly effective in high contention scenarios. For the very curious, here’s a representation of the QoS tree I have setup in PFSense:

WAN - Scheduler: FAIRQ | BW: 12531 Kbps  
 └── WAN_main - Options: Codel Active Queue | BW: 12531 Kbps

GUEST_LAN - Scheduler: FAIRQ | BW: 100 Mbps  
 └── GUEST_LAN_main - Options: Codel Active Queue | BW: 100 Mbps

IOT_LAN - Scheduler: FAIRQ | BW: 5 Mbps  
 └── IOT_LAN_main - Options: Codel Active Queue | BW: 5 Mbps

LAN - (No limits/queues)

Why 12,531 Kbps?

For the eagle eyes out there — why was my upload speed shaped to 12,531 Kbps when my connection is 10 Mbps up?

The answer is two-fold. First, DOCSIS connections are often over-provisioned to make sure that, even with loss in the cable/modem, users will probably hit the speeds they pay for. So running a Speedtest on my 10 Mbps connection without shaping actually revealed ~13 Mbps. However, I needed to find the point that maximized the upload speed while not filling the buffers of the upstream device.

To find this point, many tutorials recommend “take a Speedtest, and then subtract 20%” — I argue that this is incorrect, as a flat percentage may be too much or not enough. To find the optimum point — I essentially did the following pseudocode:

Turn on QOS with upload at expected speed

Start a few massive uploads

while (true):  
    if (UDP loss > .5% and ICMP latency change is impactful):  
        reduce QOS upload speed by 1%  
    else:  
        increase QOS upload speed by 1%

This can easily be done by hand, and takes about 5 minutes of tweaking to perform. Thus, I settled at 12,531 Kbps as the highest upload speed possible without any impact on my service.

Distribution to client devices

The router then trunks to a HP Procurve 1810G switch, that then passes tagged VLAN traffic to three Ubiquiti UniFi AC Pro AP’s spread around the house. Untagged traffic then goes to other ethernet based devices.

PFSense has great monitoring tools to measure the health and quality of a connection, but I wanted to track the speed of my connection. I built a little Node and HTML5 app called speedlogger that takes a Speedtest every 8 hours and plots it in a pretty graph.

Was it worth it?

Absolutely.

As with any experiment, any conclusions need to be backed with data. To validate the network was performing smoothly under heavy load, I performed the following experiment:

1. Ran a ping6 against speedtest.net to measure latency.
2. Turned off QoS to simulate a “normal router”.
3. Started multiple simultaneous outbound TCP and UDP streams to saturate my outbound link.
4. Turned on QoS to the above settings and repeated steps 2 and 3.

As you can see from the plot below, without QoS, my connection latency increased by ~1,235%. However with QoS enabled, the connection stayed stable during the upload and I wasn’t able to determine a statistically significant delta.

That’s how I maximized the speed on my non-gigabit internet connection. What have you done with your network?

If you made it to the end of this article, you’re probably pretty nerdy like us. We are looking for a skilled Systems Engineer and Senior Software Engineer — if that’s up your alley, check out the postings on Workable.

The future of the internet is fast. Fourteen times faster than the 70 Mbps the US averaged for download speed in March, gigabit-speed fixed broadband is still rare, but it’s making appearances in locations over the globe. Before you get too attached to the idea of downloading 1 billion bits of data per second, know that getting gigabit service and adjusting your set-up to achieve top speeds is harder than you might think.

We’re here to offer a few tips to help you achieve the Speedtest results you dream of. Some of these will help you maximize your potential internet speeds even if gigabit is not available in your area.

Factors you can’t control

1. Is gigabit-level service available in your area?

While internet service providers (ISPs), municipalities and companies like Google have been making headlines with gigabit (the ability to download 1 billion bits of information in one second), service is still rare (and expensive).

Ask around to see if gigabit is available in your area. Google Fiber is one option in some cities. Also check with phone companies and smaller ISPs to see if they offer gigabit. Some forward-thinking governments in places like Longmont, CO; Grant County, WA and New Westminster, BC have even created their own fiber networks.

2. What kind of infrastructure is your service delivered over?

You’ll get the best speeds with fiber because you won’t have to deal with the noise or interference that occurs over copper lines. However, new coaxial technologies, namely DOCSIS 3.1, have the potential to provide gigabit speeds, but not symmetrically (see the next point). Finally, phone lines, used for DSL, absolutely won’t cut it at all.

Having fiber doesn’t mean you’ll automatically have gigabit; the service still needs to be available in your area and you’ll likely pay more for it.

3. Is the available service plan symmetrical?

That is, are the advertised download and upload speeds the same? This varies by ISP, but asymmetrical service is more likely over coaxial connections — symmetrical gigabit service requires the robustness of a fiber optic connection. Asymmetrical service can lead to bufferbloat.

4. Understand the network located upstream of you.

And the quality of that network matters. If your ISP’s central office doesn’t have the bandwidth to support all the gigabit connections in your area, everyone will see slower speeds during peak usage times.

This also applies to peer-to-peer connections. If you’re downloading games and/or streaming movies, your performance is impacted by both the quality of the network those applications are using and how fast those services allow content to be downloaded. Gigabit is great for ensuring that multiple users are having a consistent internet experience, but don’t expect to be downloading games from Steam at gigabit speeds.

5. Data overhead makes 1 Gbps a theoretical number.

Though perfect circumstances might allow you to send 1 billion bits of information per second, some of those bits are overhead (including preamble, inter-frame gaps and TCP) and your actual data throughput will be a little smaller. If there was no overhead, you might be able to achieve a Speedtest result of 997 Mbps, but you’re more likely to top out at 940 Mbps. For more details on the math, read this.

What you can control

6. Good quality wiring is essential.

To achieve the fastest speeds possible, the most important thing you can do is use Cat 6 ethernet wiring to connect your devices to your modem and/or router. Cat 5e can do it but you’ll get less crosstalk using Cat 6. Plus, if you’re going to spend the money on new cables, it’s worth future-proofing your investment. Cat 5e supports up to up to 1,000 Mbps while Cat 6 supports ten times that. Also don’t run your data cabling parallel to power lines — interference from the power lines can cause interference in the ethernet cabling.

7. Are both the ports and the CPU in your router gigabit-ready?

Read the fine print when choosing a router. Not every consumer-grade router can support gigabit speeds over the ports in the back. And sometimes the ports support gigabit but the router’s CPU can’t keep up. In general, x86 processors are fastest, followed by ARM and then MIPS. You still need to check this even if your router was provided by your ISP.

Typically you’ll find that recently-released and the more expensive consumer grade routers are up to the task. Here are two routers we recommend along with affiliate links to make your shopping easy:

• Ubiquiti Edgerouter. The super advanced user will enjoy the pared-down customizability of this router. Many of the Ooklers use some version of this router. It doesn’t have Wi-Fi built in so be sure to get one or more compatible access points.
• Velop Whole Home Wire Mesh. To set up your entire house at once, try this system. It comes pre-loaded with Speedtest so you can easily test your connection.

8. Use a hardwired connection.

While Wi-Fi technology is catching up, you’ll still likely see better speeds if you plug that Cat 6 ethernet cable directly into your computer.

9. Check your adapter.

Not all laptops have ethernet ports, so you’ll need an adapter for a hardwired connection. Make sure the adapter you’re using is gigabit capable. Thunderbolt and USB 3.0 adapters are usually good, but the performance of other adapters varies widely. And don’t forget, USB based adapters also add data overhead.

10. If you must use Wi-Fi, pick a clear channel and sit close to your router.

All kinds of things can interfere with your Wi-Fi signal and thereby slow down your connection: fluorescent bulbs, baby monitors or even a cheap pair of wireless headphones. This is critical for Wi-Fi performance as only one device can use the channel at a time. In addition, Wi-FI uses CSMA-CA to handle collisions — if it detects a collision on the channel, the Wi-Fi device will halt sending and wait until the channel is clear. Interference counts as collisions, so you will end up with a sporadic and halting connection with interference nearby.

If your connection is clear, attenuation (signal drop over distance) is a very real problem when using Wi-Fi. The 2.4 GHz band handles attenuation better but is more subject to interference. The 5GHz band is less subject to interference but has more issues with attenuation. Either way, you’re still likely only to achieve speeds topping out around 600 Mbps.

If you are on the 2.4 GHz band, make sure to chose from channels 1, 6, or 11 (or 14 if allowed by your country) — those are the only non-colliding channels at 20 MHz. At 40 MHz, you will pretty well consume the entire 2.4 channel spectrum, thus, it will be even more at risk of interference. For an illustration, click here.

11. Make sure your computer is using the latest Wi-Fi standards.

The nonprofit Wi-Fi Alliance keeps a close eye on these standards. In 2016 they announced Wi-Fi CERTIFIED ac standards which include Multi-User Multi-Input Multi-Output (MU-MIMO), 160MHz channels, quad-streams and extended 5GHz channel support. These standards change as technology improves, so check to make sure you’re working with the latest certifications. And just because your router supports these standards doesn’t mean your laptop or wireless device does.

12. Decipher the hype behind the marketing.

For example, a wireless router that says it can support 4 gigs doesn’t necessarily mean it can support one 4 Gbps connection. It’s more likely that the device has four radios with 1 Gbps specified maximums (real world performance is likely to be slower).

13. Stay up to date on router firmware, but don’t update on day zero.

Vendors regularly release software updates for their routers to improve their stability, performance and security. It’s usually always the best option to stay up to date with these firmware patches. With that said, many of us Ooklaers wait anywhere from a week to a month to apply these patches (assuming they are not critical security updates) to make sure there are not any regressions or issues.

14. Use our desktop apps to run your Speedtest.

If you’re sure your setup is perfect but you’re still not seeing the Speedtest result you expect, download our free desktop apps for Windows or MacOS. Many lower performance systems can’t reach 1 Gbps via browser tests due to various limitations. Plus our desktop apps give you data on jitter and packet loss.

Advanced options: For the tech savviest

15. Is your network interface card (NIC) up to the task?

Just being rated for 1000-Base-T may not be enough. NICs that use software offload instead of hardware offload are often found in older, cheaper computers and struggle to support gigabit speeds. Intel offers some of the best driver and hardware support on their NICs.

16. Encryption can be slow if it’s not done right.

Temporal Key Integrity Protocol (TKIP) encryption, often enabled by default on Wi-Fi routers, will slow you doooowwwwn. Use Advanced Encryption Standard (AES) instead as it’s often hardware accelerated. The standard for WPA2 encryption, AES is both more secure and faster than TKIP. Some routers have TKIP options for compatibility reasons, though, even if you’re using WPA2, so check.

17. Turn off QoS shaping.

Quality of Service (QoS) shaping on a router can help you prevent large downloads from eating up all your bandwidth. But on consumer hardware, you’re also bypassing hardware acceleration so all your packets of data have to be inspected by the main CPU. This can cut your performance by 10x on a high bandwidth connection.

If you’ve gone this far and still want more, read how our lead systems engineer set up his non-gigabit connection to achieve super fast speeds

Is gigabit worth the trouble?

By now you’re probably thinking, “Getting the fastest internet speeds sure is a lot of work.” For some people hitting the maximum speed is worth any amount of work to get there. Others will be more than happy with the 300, 400 or 500 Mbps that they see on their gigabit plan with minimal tinkering.

Whether you’re gigabit ready or not, these tips will help you get the best speeds out of your internet connection now and in the future. Use this handy little list to keep track of all the steps:

If you answered “yes” to all of the above questions, congratulations! You’re now ready to unlock that superfast Speedtest result.