Lighthouse 4 and more

We have been working closely with the Lighthouse team to ensure that Calibre provides the best experience of the most popular performance audit suite.

As of right now, all tests are being conducted using the latest version of Lighthouse.

Here’s a summary of changes:

• Updated from Lighthouse 2.9.x to 4.0 Alpha
• Tests are faster—we’ve seen a 30% decrease in time waited for results
• Lighthouse Performance, Accessibility, Best Practice, PWA and SEO audits have all seen significant improvements and additions

For some of you, it might mean that some Lighthouse grade score metrics are marginally different than before—that’s because the score algorithms have been tweaked to more accurately portray user experience.

Prior to this update, JS Parse & Compile was calculated using an older version of Chrome Devtools that was built into Lighthouse 2.9.x. This older version of devtools did not calculate all aspects of JavaScript runtime performance, so the update to Lighthouse 4 will more than likely show significant increases to the JS runtime metrics. These increases more accurately describe JavaScript main thread performance.

More importantly, Lighthouse results you see in Calibre are far closer to release parity with Chrome Developer Tools add-on and the Chrome extension.

Less variance between tests

We know that test reliability is crucial to performance monitoring and making confident decisions. Test data is now more consistent thanks a number of developments focused on smoother instrumentation.

Test Verification

We’ve added a test verification process in which Calibre watches the results of each test and compares it to historic data.

If an anomaly occurs (for example, a significant increase for a page’s Speed Index on a mobile device), we will automatically re-test the given page a number of times to confirm the results. Test verification has already marked a reduction on metric variation and will prevent unexplained spikes or falls from happening.

Downloading test artifacts

You can now download the Lighthouse JSON, GIF and video renders of each test against your pages. Head to a Snapshots → choose a relevant test → click the dotted menu in the upper right corner and choose what data you’d like to get!

Request details formatting

We have improved the display of request timings and headers so you find what you’re looking for more easily.

Upgrade rollout post mortem

Last week while releasing the Lighthouse 4 upgrade, we suffered from a much longer than expected data migration that caused a few episodes of downtime. No data has been lost. If you experienced a less-than-smooth experience with Calibre during this time, we are truly sorry.

We've written a detailed account of what went wrong, how we will be handling significant releases in the future and providing transparency into the service status.

If you have any questions or concerns we’d be delighted to speak with you directly.

Coming Soon

In the New Year, expect to see detailed email reports, clearer metrics and new tools that will help you make your pages faster.

References:

• Lighthouse 4 Release Notes
• Lighthouse Scoring Guide
• Data-migration down time post mortem

Ben Schwarz

Ben is the founder of Calibre. He regularly speaks at conferences worldwide and has been involved in far reaching open source projects as well as web standards. Find him on Twitter or LinkedIn.