Measuring Broadband America Platform

The Measuring Broadband America (MBA) platform collects Internet performance data from many millions of homes simultaneously. There are currently over 500,000 homes safely reporting anonymized data. The test agents, which can be either Whiteboxes or routers, run measurements against popular and important services, including streaming services, emergency services, and e-commerce companies, as well as conventional speed tests and other lower-level performance measurements. The data is then pushed to the secure cloud-based platform and made available in the MBA dashboard for real-time analysis. The test methodology is open, published, and constantly reviewed for improvements. New tests, for example, are frequently added to the program.

Testing from the Edge of the Home Network

The causes of poor performance can appear anywhere on the Internet, and while conventional web-based speed tests can indicate whether your Internet connection is “good” or “bad”, they do not help identify the root cause of issues. With more network segmentation, it is possible to identify the origin (and therefore the cause) of poor performance. The MBA methodology provides for a test agent to be located on the edge of the home network. This can either be a SamKnows-enabled router or dedicated hardware, such as the Whitebox.

The MBA test agents run tests against on-net servers, off-net servers, and cloud-based test servers. Off-net servers are all dedicated to the MBA program and monitored closely. Alongside the data collected from these measurements, a separate and extremely large control panel of SamKnows-enabled routers is maintained. This consists of several hundreds of thousands of homes and is growing fast. This data is used to check the smaller MBA panel to ensure that the results from these homes are not out of the ordinary. This control panel is important to ensure high standards of accuracy.

MBA Tests

The MBA test platform supports a large variety of tests, many of which are used in the Measuring Broadband America program. All tests are written in C, a low-level language that enables us to write high-performance measurements with minimal resource usage. More tests are being developed over time so that the MBA can keep up with popular content and measure the most popular services.

Cross-traffic

The Whitebox can detect 'cross traffic' in the home; this is when other devices inside the home are also using the Internet. Naturally, we do not want to carry out measurements when the user is using their Internet connection because (a) we might disrupt their Internet experience and (b) our measurement results would be impaired.

The threshold for cross traffic is defined as 64kbps downstream and 32kbps upstream. If cross traffic is detected when a scheduled test is due to execute then the test is deferred for 30 seconds and retried. If cross traffic persists for five minutes then the test is aborted.

Speed Tests

The speed tests use eight parallel TCP sessions and run for 10 seconds. An additional warm-up period is used to offset the ramp-up period associated with TCP applications. For more detail on how the speed tests work, please see this page.

Latency and Packet Loss

The latency and packet loss tests run continuously in the background, trickling out a steady stream of approximately 2000 UDP echo packets each hour. This provides a much more detailed and realistic view of latency and packet loss than web-based tests and mobile apps, which just measure latency for a few seconds. Our latency and packet loss test also captures disconnection events - when your broadband connection drops completely. For more detail on how the latency and packet loss tests work, please see this page.

Web Browsing

The web browsing test measures how long it takes for a popular web page to load on the user's broadband connection. A selection of nine real websites are used in this metric. Note that this is an example of a test that runs to a real content provider, rather than a synthetic test server. Please see this page for more information on this test.

DNS

The test platform also runs a variety of other tests for DNS performance and failure rate, CDN performance, voice-over-IP emulation, and more. Please see this page for a more detailed description for each test that is supported and the metrics that each test captures.

Scheduled Tests

The Whitebox runs tests according to a pre-defined test schedule. This schedule has been crafted to ensure that the FCC can carry out all of the measurements it needs to fulfill the reporting goals, while not using too much of the user's bandwidth.

The test schedule generally runs more measurements during peak hours (7-11pm local time), as this is the primary focus of the Measuring Broadband America report. Some tests will run more or less frequently than others, depending on the reporting goals and the amount of data they consume.

The most recent test schedule is visible in the Measuring Broadband America technical appendix, available here.

Whitebox users are also able to go to SamKnows One to initiate an Instant Test from their Whitebox. These real-time tests allow users to have an instant view of their broadband performance. Please note that the results of instant tests are not used in the published Measuring Broadband America reports; only scheduled tests are used.

Measurement Agents

The Whitebox

The Whitebox is a small Linux-based hardware device that volunteers are asked to install in their home.

Using a hardware test agent provides a number of benefits that help ensure we are gathering the most accurate measurements possible:

• The Whitebox is connected via 1Gbps Ethernet cable to the user's router, which means that we eliminate in-home factors in our measurements. In-home Wi-Fi can be notoriously unreliable, and the objective of the program is to measure the Internet connection, not in-home factors.
• Measurements can be run 24 hours a day, 7 days a week - no other computers need to be switched on and no user activity is required. This provides a large dataset for us to draw robust conclusions from.
• The Whitebox can monitor 'cross traffic' inside the home (more on this below) and defer tests if it is discovered. Web based tests and mobile apps cannot do this.
• A consistent test environment everywhere - we do not need to worry about a user's phone or PC being underpowered for the broadband connection that is being measured. The Whitebox is able to measure up to 1Gbps.

All measurements carried out by the Whitebox are 'active' measurements - i.e. it generates its own traffic and measures performance characteristics of that. The alternative is 'passive' measurements, which is when a user's existing Internet traffic is monitored in order to determine performance. The Whitebox does not use passive measurements.

To read more about the Whitebox, please see this page.

Test Servers

The Measuring Broadband America program uses a large number of test servers. These are used as the endpoints for many of the tests (but not all, as some run to real content providers).

Test servers are installed in nine major peering locations around the U.S. These are:

• New York
• Washington DC
• Atlanta
• Miami
• Denver
• Dallas
• Los Angeles
• San Francisco
• Seattle

All test servers are hosted 'off net'. This means that they are not installed inside ISP networks, and instead sit outside of the ISP networks at a neutral location. This is done to ensure that we are measuring the complete service that the ISP is providing, up to and including a common handover point to the wider Internet.

The vast majority of test servers have 10Gbps or even 100Gbps. Those locations that are still 1Gbps are being upgraded to 10Gbps over time.