It’s now easier to prioritise which improvement possibilities to focus on based on Lighthouse audits. You can view the Opportunities and Diagnostics for all Lighthouse audit types (Performance, Accessibility, SEO, Best Practices and Progressive Web Apps) for each test.
For Performance audits, you can filter by Core Web Vitals metrics impacted by the audit and check the potential time and byte size savings you could observe by addressing specific advice given by Lighthouse.
Check out the refreshed audits by heading any selected Snapshot!
We published a new patch version of React Live Chat Loader (2.8.1). In this release, we realigned react-live-chat-loader placeholder styling with Intercom’s current default styling (#215) and bumped all outdated dependencies.
We fixed a problem preventing organisation invitation emails from being delivered in the case where the Resend invitation button was pressed multiple times by the same person to send another invitation email.
Previously Calibre recommended that Time to First Byte (TTFB) should be below 150ms for a good user experience. The recommendation has now been updated to < 300ms based on global data from the Chrome User Experience Report (CrUX).
In Calibre, you will now see a warning if your TTFB is above 300ms and a critical error if it is above 800ms.
We improved the badge element contrast to pass Web Content Accessibility Guidelines (WCAG) AAA (enhanced contrast) accessibility audits. Previously, they only met the standard on AA (minimum contrast) requirement.
On top of that, we also slightly amended the appearance of badges to give a more soft, rounded feeling. You can see them in action on the Snapshots list, Snapshot page, and Budgets.
We’ll continue evaluating and improving contrast and other accessibility features across Calibre.
In synthetic and real user monitoring, it’s critical to understand the conditions in which people browse your sites so that we can correlate it to performance. It’s also not uncommon for metric definitions and collection methods to change or bugs to emerge.
To improve the transparency of test settings and debugging changes, we added the Chrome and Lighthouse versions for each test to the Snapshot and Single Page Test report pages:
You can now see your real user data for Time to First Byte (TTFB) and Interaction to Next Paint (INP) in the Core Web Vitals Checker:
Time to First Byte is critical to track as it can have a cascading, adverse effect on page load and speed. Interaction to Next Paint is a new, experimental metric measuring the longest observed delay between user interaction and the response on your page. While these metrics are not included in the Core Web Vitals set now, we’d recommend adding them to your “to track” list.
When you add a new Site to monitor in Calibre, we send a summary email with crucial page speed metrics and how you rank against our measurements dataset. To provide a more comprehensive picture of web performance, alongside Core Web Vitals, we added Time to First Byte to the email report:
This is especially useful if you’d like to set cookies and headers only for Pull Request Reviews tests (not inherited from Test Profiles settings). For example, you could set a header to signify that the test is for a Pull Request and should be safely routed to the staging infrastructure.
If you’re using cookies and headers (either in Test Profiles or Site Settings → Test Agent area), we now change when they are applied.
Previously, Calibre set cookies and headers post-authentication. Now, they are applied during authentication so that you can control more fine-grained authentication scenarios, such as hiding CAPTCHA, so Calibre can quickly test your Site or application.
We introduced two new currencies (EUR and AUD) and several new payment methods, so your team can choose a way of paying that’s most convenient and least costly. You can change your payment currency at any time:
Because limited payment methods can be a software adoption blocker and a source of accounting issues, we also added several new methods of purchasing a subscription—not all of which rely on credit cards. We now accept:
Credit Card (all plans)
Apple Pay (all plans, only in Safari browser)
Google Pay (all plans, only in Google Chrome browser)
Thanks to SEPA and BECS, customers in Australia and SEPA countries with relevant plans (AUD or EUR) can pay for their subscription directly from a nominated bank account. You can change your payment method in Billing → Payment Method tab.
We made it possible to display the request table (from the corresponding HAR file) even when your tests fail. We’re now surfacing more information about errors that lead to Lighthouse failures directly in the Snapshot view, so it’s easier to find out the exact reason:
If your tests are failing (continually or intermittently), you can also refer to the Why do my tests fail? guide that walks through common scenarios and steps to fix them.
With two minor releases of Calibre CLI, we added new commands and ways to query your data. Site.get and Site.update now allow obtaining and updating given Sites. We added missing fields to Test Profiles, Deploys, Location, Metric budgets, and Page queries so that you can receive and manipulate all test data.
We continually build on our best practices to protect your account and data. From now on, all new registrations and, periodically, existing users will have to verify their email accounts unless they are using Google as a method of authentication.
Staying on top of subscription costs and reconciling payments is essential for any business. That’s why we’ve made improvements to make this process easier. Now, you can see the upcoming invoice amount on the Billing → Invoice History and Settings page:
Making changes to Test Packs and User Seats could quickly generate multiple invoices. To avoid dealing with too many PDFs, we batch your changes as they happen and release an invoice after a short time delay. Nobody needs more accounting work!
Each team has different processes. Sending your monitoring data to your favourite services and creating automations can help with saving time and increasing transparency. We built a Zapier integration, so you can connect to over 4,000 supported apps and create custom workflows without any coding required!
We created a handful of boilerplate workflows (Zaps) to try:
get a direct message in Slack when a performance budget alert fires
send page speed data to Google Sheets or Airtable
run a deploy on a new GitHub release
create a Trello card to investigate exceeded performance budgets
We deprecated the site get-pulse-metrics command and replaced it with site metrics (a breaking change). We also added a metric-list command that lists all available metrics, their category, advised ranges and recommendations for tracking:
If you’d like to (or already are) testing performance in GitHub Pull Requests, there is a new way to configure your test settings. With version 2, you can create configuration options for different branches using positive and negative patterns.
This is useful if you’d like to test differently based on branch names, for example, your preview-public branches are publicly accessible, but preview-* are password-protected:
Or, you could report different metrics based on branch name too! Convert your configuration.yaml to the new, multi-block format to create branch-specific testing conditions.
Using integrations can be a powerful way to connect the services you use. But it can be frustrating when an integration stops working, and we don’t know about it.
Now, Calibre sends you an alert when there’s an issue with any of your connected services (such as Slack, Webhooks, GitHub, Netlify) with steps to fix. No more discovering data or notifications aren’t being sent days later!
We’re continuing to improve our GitHub integration by adding more options to skip tests or run them on-demand—whenever it suits your development cycle. Now, you can ask Calibre to test (or re-test) your work by posting a comment with @calibreapp run. Similarly, you can say @calibrepp stop to halt future tests in that PR. Too easy!
There are other situations when you might want to skip testing too. By default, Calibre won’t test draft Pull Requests anymore—only ones ready for review.
React Live Chat Loader now supports Chatwoot and five other live chat providers. If you’d like to decrease the negative speed impact of live chat, install react-live-chat-loader through npm and implement it in your React application.
Previously, we displayed Performance Score, First Contentful Paint and Time to Interactive as default metrics in Calibre. This meant that you’d see those three metrics across Your Sites, Pulse, Pages Leaderboard, Pull Request Reviews and Insights reports. Now, Core Web Vitals (Largest Contentful Paint, Cumulative Layout Shift and Total Blocking Time) are the default metric set.
Core Web Vitals have been actively improved for several years and provide a holistic view on different aspects of user experience. Google also uses them as ranking signals for page experience. You can see Core Web Vitals across the previously mentioned areas. You can still customise metrics displayed in Pull Request Reviews, Pages Leaderboard and Pulse if you’d like to amend them. All measurements are colour-coded, so you know if they’re good (green), need improvement (orange) or poor (red).
In the future, we’ll be introducing more ways of view and metric customisation, as well as keeping track of KPIs.
We know it’s essential always to control your data and be able to delete your account without needlessly contacting support. Now, you can delete your personal account in two clicks if you don’t want to use Calibre anymore.
Our Test Agents now use the latest Lighthouse 8 and Chrome 91 to test your Sites. Lighthouse 8 comes with a few notable changes to metric collection and Performance Score calculation. Here’s a handy explainer of what you might notice in your metrics.
Before, only organisation administrators could create API access tokens, which created a need to give admin privileges to people who shouldn’t necessarily have access to other admin-only areas such as Billing and People Management. Now, anyone within your organisation can create a Personal Access Token with the level of permissions that mimics their account access level. With the introduction of PATs, everyone gains access to Calibre’s APIs.
Administrators can keep track of and manage all access tokens in the API tab:
Personal Access Tokens will become even more helpful with the introduction of Teams. Not only the newly introduced Viewers will be able to retrieve data, but also Calibre will scope everyone but the admin’s access to the Teams they belong to.
React Live Chat Loader, our open source project that helps mitigate the negative performance impact of live chat tools now supports Userlike. You can install react-live-chat-loader through npm and implement it in your React application.
Following the API Tokens refresh in January 2021, we added more capabilities to make managing your access tokens straightforward.
Now, you can sort the tokens by name, creator and expiry date by clicking the respective headings. You also can easily find the token by searching for its name or creator in the search field in the upper right-hand side corner.
We refreshed the way you see and manage people within your Calibre organisation. You can now quickly find a person by searching for their name, email, role or authentication method. You can also sort the information by clicking the table column’s name to, for example, see who is an administrator.
You can still edit, remove and invite new people to collaborate on performance, just more easily.
It’s critical to test speed from various contexts that reflect your audience. performance will differ drastically depending on devices, latency, and location. That’s why we added three new test location regions:
Cape Town, South Africa
You can change your tests’ location in the Settings → Test Agent → Test location field or when creating a new Site. Find a complete list of Test Agent locations and their static IP addresses here.
We refreshed Snapshots page, listing all of the tests for a given Site. Now, you can clearly see the test status (Scheduled, Testing, Verifying, Completed or Failed) on per-Snapshot and per Page and Test Profile combination basis.
If you want to know how a Snapshot was started (manually through the interface, through previously configured test scheduler or API integrations) or investigate test status, Snapshots page is the best place to find it
Calibre can log into your Sites and applications as if it was one of your users. Until now, when using Form Authentication, we could specify the form, username and password CSS selectors, but not the submit button selector. Our Test Agent would then crawl your Sites to submit the form and successfully log in.
This process wasn’t bulletproof, especially in tracking competition scenarios where you can’t make improvements to the markup. You can now specify the Submit selector, so there are no failed tests due to markup mismatch. You have full control over selecting which elements should be interacted with when authenticating.
Our Test Agents now use the latest Lighthouse release, 7.0. As always, we upgrade to stable Lighthouse releases as they become available so that you can leverage the newest version.
This is the first major version release since Lighthouse 6 in May 2019, including numerous significant changes, including new audits. As outlined by the Lighthouse Team, you might see PWA and Accessibility scores changes.
If you are using Pull Request Reviews to track speed before releasing to production, you can now trigger tests only based on a specific list of environments, so the reports in GitHub always relate to what matters to you most.
This is especially helpful when using one deployment provider to release multiple projects or sites. For example, at Calibre, we deploy our marketing website using Vercel. Each deploy publishes a preview of the website and our design system in Storybook. We specify the deployment environment to match the marketing website, so our Pull Request Reviews analyse the website, not Storybook.
We refreshed how you manage and generate API Tokens used to authenticate your account when building automations with our APIs. You can see all tokens for your organisation in the API tab, including their creator and expiration.
When creating a new API Token, you can easily select the areas it should access. We display the new token only once, so it’s essential to save it in a password manager straight away. You can also manage your tokens via the CLI and APIs.
You can now also add global Headers and Cookies to manage how Calibre’s Test Agent accesses your Sites. Previously, this option was only available through Test Profiles. Now, you can use global Headers and Cookies for Site-wide settings, and Test Profiles for more targeted testing, such as A/B tests for specific devices.
It’s a frustrating but not uncommon for Lighthouse performance tests to fail. We introduced Compatibility Mode to provide you with critical performance metrics even when Lighthouse returns an error, which otherwise would yield no results.
We refreshed the Site Settings area so that you can set up and manage your testing faster. You can now search for Pages, which is especially helpful when working with larger Sites. You can explore available integrations in the Integrations tab, to connect Calibre to your favourite services and make the most out of your monitoring data.
We redesigned the primary navigation to make it faster and more straightforward to get around Calibre. Now, you can quickly switch your viewing context between both organisations and Sites, as well as create new ones.
Image Actions, our Open Source GitHub Action that automatically optimises images in your Pull Requests, got a significant upgrade and can now be used with more complex workflows.
You can now see the difference in images pre and post-compression, run the optimisation on a selected schedule, run the action in compressOnly mode that allows for more complex combinations and more. Learn about all changes to Image Actions 2.0 here.
We improved the structure of Lighthouse audits available in Performance, PWA, Best Practices, Accessibility and SEO tabs for each Snapshot. Now, all audits returned from Lighthouse fall within the following categories:
Items to manually check
With improved categorisation, you can pinpoint which areas to focus on easier.
We launched Pages Leaderboard so you can identify fast and slow areas of your Sites quickly. In the Pages Leaderboard, you can sort Pages within your Sites based on three selected metrics and device type (desktop or mobile). Calibre will show the last 15 measurements for each metric in your Pages dashboard, colour-coded based on recommended value ranges, so you know which areas need improvements most.
We encourage customising metrics based on your organisational or project goals. For example, you might be wanting to focus on Core Web Vitals. You can also use the Pages Leaderboard for competitive benchmarking, by tracking your landing pages against competitors. With sorting and filtering, you can easily rank Pages against each other.
There’s a variety of reasons why tests might not be completed successfully—from networking, firewalls, HTTP responses, incorrect markup to timeouts caused by poor performance. Ideally, we spot and address persistent failures quickly, so there’s no disruption to our monitoring and data.
For that exact reason, we introduced Snapshot Failure Emails that alert you when tests are not completing. We will also let you know when tests are running smoothly again.
Anyone can subscribe to Snapshot Failure alerts in Profile → Email Notifications. In case you didn’t see it, we also documented the most common test error types, so you can debug the reasons behind failures more quickly, too.
If you’re testing Sites that require authentication, you can do so without providing the username and rely solely on the password. We know authentication comes in various shapes and forms, thus providing the username to the Calibre Test Agent is now optional.
You can always adjust your authentication settings in Site → Settings → Agent Settings.
Suppose you are using multiple deployment methods for GitHub repositories connected to Pull Request Reviews. In that case, you can now choose the provider that should be used as a basis for performance comparison.
Selecting a deployment provider when many are present will ensure the most accurate results in your performance reports. You can choose a provider in Site → Settings → Integrations → GitHub.
We completely revamped performance budgets to guide you in setting thresholds based on recommended values and your context. With new Budgets, you’re able to see all of your targets with their current status in one place—the Budgets dashboard.
When creating a Budget, Calibre will now showcase the desirable value ranges alongside your past measurements. That way, you can make an informed decision about what’s achievable in the short term, and where would you like to be in the long term. We take out the laborious process of guessing what a good metric value should be.
You can also view your Budgets and see historical data for each Page and Test Profile combination. By doing so, you will be able to easily spot outliers that might need separate thresholds or identify where more performance work is required to meet your goals.
As always, Calibre will notify you about the status of your budgets through email and Slack, based on a selected frequency. In Budgets 2.0, we also identify at-risk values, so you know when a metric is close to exceeding the budget and can address it before it happens.
If your deployment previews are password protected and you’re using Pull Request Reviews, you can now add authentication settings to your config.yml to access those environments. Those authentication settings will only be used for Pull Request Reviews tests. You can still specify separate authentication methods for your scheduled and API-invoked Snapshots.
On May 26, we updated our Test Agents to the latest version of Lighthouse. Lighthouse 6 comes with a completely revamped Performance Score algorithm, new metrics, audits, and more. All tests from May 26 onwards run using Lighthouse 6, which is why you are likely to observe changes to your Performance Score.
With Lighthouse 6, we are also reporting a new metric—Cumulative Layout Shift. Alongside Total Blocking Time and Largest Contentful Paint (present in Calibre since September and October 2019), CLS is a part of a new generation of performance metrics that aim to portray user experience more accurately.
This also means that you can track all Core Web Vitals (Total Blocking Time, Largest Contentful Paint and Cumulative Layout Shift) in Calibre. We are committed to keeping our platform on par with the newest developments in the performance space.
The Main Thread Execution Timeline showcases all long task warnings and blocking times in the lifespan of the main thread activity. This means that you can not only inspect tasks that contribute to Total Blocking Time (defined between First Contentful Paint and Time to Interactive boundaries) but other long tasks on the main thread that don’t fall in this category.
Previously, when your trial has lapsed, or you have run out of your monthly test allocation, Pull Request Reviews would stop running. Now, Calibre will issue a comment informing you of the necessary steps to be taken to resume testing on your Pull Requests.
This will be especially helpful in contexts where developers rely solely on the GitHub integration versus also logging in to Calibre, where this messaging was always present.
React Live Chat Loader, our open source component that mitigates the negative performance impact of live chat tools, now supports Facebook Messenger.
If you are using Intercom, Help Scout or Facebook Messenger in your React-powered site or application, you will be able to use the component out of the box and improve your performance metrics. To see its benefits, read about how we improved the speed of our live chat by 30%.
We have revamped the Billing area to make it easy to see your test allocation usage, manage Plans, tax information and receipts.
In Billing → Overview you can now see per Site test usage as well as delete your organisation.
You can add or update your default credit card under Billing → Payment method. In Billing → Receipt History and Settings you can ensure your receipts include all necessary information for your tax department and accountants.
At any point, you can adjust your Plan accordingly to your usage.
When using Calibre, you are always in control of your monitoring data. We encourage exporting and building custom integrations with the help of our Node.js API and command-line client.
Exporting directly from the Calibre interface is now even more accessible. You can quickly export metric or Snapshot data from Pulse, Metric and Snapshot views. You can download a CSV, Lighthouse JSON or your page video render (depending on the context), or copy a pre-baked code snippet to use with the CLI.
We redesigned the email you receive after adding a new tracked Site to Calibre. Now, you can see three key metrics—Performance Score, Time to Interactive and Largest Contentful Paint and how they compare to the Calibre test dataset. This change should bring more context into where your Sites are at from the very first minutes of testing.
Additionally, all of our other emails also got a splash of new paint to match.
We launched Test Packs to allow increasing testing capacity without upgrading your Plan. Each Test Pack adds 5,000 tests to your account and can be added or removed at any time, depending on your needs.
Learn how to use Test Packs or add one in Billing → Overview by clicking the Change plan and manage Test Packs button.
Max Potential First Input Delay is calculated based on First Input Delay (FID). It portrays the maximum time it takes the browser to respond to a user action, such as clicking a button or trying to type something into a form. Max Potential FID can be used alongside Time to Interactive to quantify when your sites and apps can be successfully interacted with.
You can add Max Potential First Input Delay to the Pulse page through customising the Metric History view and set budgets against it.
Most performance and user experience issues are discovered when it’s already too late—long after they have already impacted your customers. Pull Request Reviews allows you to identify speed bottlenecks before they reach production, directly in your Pull Requests.
If you’re using GitHub and one of the following deployment methods: Vercel, Heroku, Netlify or GitHub deployment statuses, you will be able to enable Pull Request Reviews and receive detailed performance reports on your work-in-progress. Read the announcement or learn how to get started.