Track Performance with Lighthouse 6, Analyse Blocking JavaScript and More

Release Notes is a monthly summary of new features, product improvements and upcoming releases.

Test performance with Lighthouse 6

On May 26, we updated our Test Agents to the latest version of Lighthouse. Lighthouse 6 comes with a completely revamped Performance Score algorithm, new metrics, audits, and more. All tests from May 26 onwards run using Lighthouse 6, which is why you are likely to observe changes to your Performance Score.

With Lighthouse 6, we are also reporting a new metric—Cumulative Layout Shift. Alongside Total Blocking Time and Largest Contentful Paint (present in Calibre since September and October 2019), CLS is a part of a new generation of performance metrics that aim to portray user experience more accurately.

This also means that you can track all Core Web Vitals (Total Blocking Time, Largest Contentful Paint and Cumulative Layout Shift) in Calibre. We are committed to keeping our platform on par with the newest developments in the performance space.

Analyse JavaScript main thread performance

Using the Main Thread Execution Timeline is a great way to identify potential issues with JavaScript performance. We updated the visual representation and categorisation of tasks to make it more clear.

The Main Thread Execution Timeline showcases all long task warnings and blocking times in the lifespan of the main thread activity. This means that you can not only inspect tasks that contribute to Total Blocking Time (defined between First Contentful Paint and Time to Interactive boundaries) but other long tasks on the main thread that don’t fall in this category.

Know when your Snapshots fail and why

Along with the latest improvements to Snapshot page showing Snapshot status, we enhanced the transparency around test failures. When a Snapshot fails, you will see a message stating the reason on the Snapshot page.

Fixes to Performance Score reporting

On June 4, we uncovered an issue in which Performance Scores for all mobile Test Profiles were miscalculated between May 26 — June 4, coinciding with our Lighthouse 6 release. This issue has affected all customers with mobile Test Profiles. All other metrics and scores were not affected.

An unforeseen change in how mobile simulations were handled in Lighthouse 6 resulted in Calibre reporting much lower values for mobile Performance Scores than they should have been. On June 4, after consultation with the Lighthouse Core Team, we re-calculated the Performance Score for all mobile Test Profiles within the affected timeline. This change is now reflected in the Calibre interface. The changes are not populated within any data exported with the API or Webhooks. No action is necessary.

It’s our core focus to provide stable and reliable performance reporting you can trust. We understand how an issue of misreported metrics breaches that trust and affects your teams’ ability to be successful in improving speed. This is not the standard we set for ourselves at Calibre, and we recognise that on this occasion, we have failed to provide accurate reporting for the given period. We are sorry for that. We are working towards an even closer collaboration with the Lighthouse Team to prevent such regressions from happening in the future.

New documentation: Test Verification, Downloading Test Artifacts and more

Test Verification plays a significant role in our performance platform. We documented the way Calibre verifies each test it conducts to make sure metrics we report are stable, reliable and believable.

We clarified what test artifacts are available to download through the CLI and Node.js API. You can obtain both Single Page Test and Snapshot artifacts.

Blocking JavaScript with Test Profiles is now documented too, so you know which script requests are blocked when tracking.

Upcoming releases: Budgets 2.0

A few months ago, we shared an early preview of the new Performance Budgeting. Since then, we have made numerous improvements to the initial concept and scope.

With new Budgets, Calibre will guide you in creating performance thresholds that not only make sense for your context but also are in line with industry recommendations for optimal speed.

You will be able to browse your Budgets depending on their status or devices they affect and drill down into specific Pages. You will be notified in Slack or through email based on selected frequency (more and less reactive).

We are excited to share Budgets 2.0 with you within the coming days.

Karolina Szczur

Karolina is the Product Design Lead at Calibre. With years of design experience under her belt, she’s responsible for the comprehensive user experience spanning over product, visuals and content. Find her on Twitter or LinkedIn.