Pull Request Reviews

Pull Request Reviews allow you to see the performance impact of development work before it gets released to your customers. It’s a powerful GitHub integration that works with most popular deployment platforms.

Connect Calibre to your GitHub organisation

To enable Pull Request Reviews, go to Site → Settings → Integrations tab and click Add to a repository in the GitHub section. You will be redirected to GitHub to enable the integration for your organisation.

Calibre requires access to commit statuses, deployments, checks and pull requests. We strongly recommend granting access to any selected repositories you’re interested in tracking.

Select comparison branch

After confirming the link between GitHub and Calibre, you will have to select a primary branch that will be the base for performance comparisons. This branch should be either deploying to production or a staging environment. In most cases, this might be main branch.

For the most accurate results, make sure you compare two very similar environments. Some of the common differentiators producing unreliable results can be using a CDN on production, but not on Pull Request deployments, or the presence of ads or other third parties.

When the selected branch is successfully deployed, Calibre creates a deployment marker on your charts and takes a new snapshot of your site. That way, you always know how releases affect your performance metrics.

Create a Pull Request

Calibre will compare each Pull Request to the median of your site metrics since the last deploy. Each report highlights critical measurements (Lighthouse Performance Score, Time to Interactive and First Contentful Paint) overall and across Test Profiles to highlight top-level changes.

You can also analyse each page to see browser previews for specific Test Profiles and most significant changes, including any Budgets that have changed.

Pull Request Reviews provide a helpful performance overview directly in GitHub, with the ability to drill down into metrics further in the Calibre interface.

Using Deployment statuses

Your repository has to be using GitHub deployment statuses so Calibre knows what Pull Requests should be compared to. Providers such as Vercel (previously ZEIT), Netlify and Heroku provide these automatically.

If you have a custom CI pipeline, you can mark deployments and create deployment statuses using the GitHub API.

Calibre will create a Pull Request Review Site when it receives a deployment status with a state of success. The canonical URL of your review Site should be set in the environment_url field.

See GitHub documentation for more information.

Calibre will create Pull Request Review tests from any GitHub deployment status. You can restrict Calibre to only trigger tests from a known provider (one of Vercel, Netlify, Heroku or a custom domain) during setup, or by editing the GitHub integration settings.

Configuration file

You can customise metrics and authentication settings by including a configuration YAML file in your linked repository at /.calibre/config.yml.

There are two top-level properties you must specify:

• version: The current version is 1
• pullRequestReviews: Options for Pull Request Reviews

Configure metrics

By default, Calibre will compare the following metrics:

• Lighthouse Performance Score
• First Contentful Paint
• Time to Interactive
• First Meaningful Paint
• Largest Contentful Paint
• First Interactive
• Total Blocking Time
• Speed Index
• JS Parse & Compile
• Total page transferred
• Total HTML transferred
• Total Image transferred
• Total Video transferred
• Total Font transferred
• Total JSON transferred
• Total CSS transferred
• Total JavaScript transferred
• Asset count

Under pullRequestReviews in your configuration file you can explicitly define which metrics to use, or you can provide a list of metrics to ignore from all available metrics in Calibre.

When you define metrics, the first three will be shown in the summary table.

Example: Define metrics

The following configuration will only compare the metrics listed under measurements:

1version: 1
2pullRequestReviews:
3  measurements:
4    - lighthousePerformanceScore
5    - lighthouseAccessibilityScore
6    - lighthouseSEOScore

Example: Ignore metrics

The following configuration will include all metrics except the ones listed under ignore:

1version: 1
2pullRequestReviews:
3  ignore:
4    - lighthousePwaScore
5    - pageHTMLDownload
6    - timeToFirstByte
7    - lighthouseSeoScore
8    - pageWaitTiming

Configure authentication

Calibre will copy authentication settings to your Pull Request Review tests. The authentication URL will be automatically updated with your deployment URL.

If you require specific authentication details for Pull Request Reviews, you can set them using config.yml.

Example: Define authentication settings

The following configuration will be used for authentication:

1version: 1
2pullRequestReviews:
3  authentication:
4    url: https://different-domain.com/login
5    username: user@email.com
6    password: mysecretpassword
7    formSelector: form
8    usernameSelector: input[name=email]
9    passwordSelector: input[type=password]

Access & security

In order for Calibre to perform its duties for Pull Request Reviews, the following API access to your GitHub respository is required:

• Checks - Read & Write
• Deployments - Read only
• Metadata - Read only
• Pull Requests - Read & Write
• Access to read files - Read & Write access to .calibre/config.yml
• Commit statuses - Read only

In addition to these API items, Calibre will also read the following events for authorised repositories:

• Check run
• Check suite
• Deployment
• Deployment status
• Pull request
• Status

Access is authorised by your team during installation and any changes in access requirements will be confirmed.

If you have any questions or concerns please write to security@calibreapp.com.