Test Verification

Test Verification is a critical part of Calibre’s performance platform. It ensures the metrics you are viewing are stable and reliable without delay in delivering test results.

The inevitable variability of networking and the web produces various degrees of spikes in metrics. On top of that, it’s necessary to detect any possible abnormal measurements and prevent them from being injected into user-facing reports as they not only distort the data but also are a source of confusion and lack of trust for performance testing.

Test Verification prevents creating uncertainty around your performance results in Calibre.

How test verification works

When Calibre runs tests, it analyses a set of recent measurements for a given Page and Test Profile combination. We leverage several metrics to flag possible deviations:

  • Total Blocking Time: highlights changes in JavaScript main thread performance.
  • Time to First Byte: shows indications of server issues or back-end changes.
  • Lighthouse Performance Score: showcases a blended summary of both paint and main thread performance.

Calibre assesses the difference in those three metrics by using a power curve. If there’s an absolute change above a certain threshold, test results are flagged.

A graph portraying how Test Verification works through multiple steps of test result confirmations

Once a result is flagged, we will run additional tests on the selected Page and Test Profile combination, within 30 seconds of the original test. The same metrics will be re-assessed. If we observe similar measurements, the test with the least amount of deviation is accepted.

If the second test produces different results, it will be flagged again, and re-run for the final, third time. Once Calibre settles on a test result, the most reliable test is released. At this point, metrics are made available on dashboards, through APIs and webhooks.

Throughout the verification process, Calibre also factors in the possibility of missing data points, such as not having Page history or a Page being unavailable for testing (for example, in a case of an outage). In those scenarios, we still can verify your test results.