Pull Request Reviews

Pull Request Reviews allow you to see the performance impact of development work before it gets released to your customers. It’s a powerful GitHub integration that works with most popular deployment platforms.

Connect to your GitHub organisation

To enable Pull Request Reviews, go to Site → Settings → Integrations tab and click Add to a repository in the GitHub section. You will be redirected to GitHub to enable the integration for your organisation.

Calibre requires access to commit statuses, deployments, checks, comments and pull requests. We strongly recommend granting access to any selected repositories you’re interested in tracking.

Select comparison branch

After confirming the link between GitHub and Calibre, you will have to select a primary branch that will be the base for performance comparisons. This branch should be either deploying to production or a staging environment. In most cases, this might be main branch.

For the most accurate results, make sure you compare two very similar environments. Some of the common differentiators producing unreliable results can be using a CDN on production, but not on Pull Request deployments, or the presence of ads or other third parties.

Define deployment environments

Calibre will create Pull Request Review tests from any GitHub deployment status. If your repository is deploying to several environments in Pull Requests (e.g. in monorepo scenarios or when both Site and a Storybook preview is created), choose one of the following options:

• a specific provider (Vercel, Netlify or Heroku) when multiple providers are present
• custom deployment environment(s) (for Vercel, specify both preview and production environment in a comma-separated list)
• a custom domain

You can also select or ignore specific branches from your testing.

Create a Pull Request

Calibre will compare each Pull Request to the median of your site metrics since the last deploy. Each report highlights critical measurements (Largest Contentful Paint, Cumulative Layout Shift and Total Blocking Time) overall and across Test Profiles to highlight top-level changes.

You can also analyse each page to see browser previews for specific Test Profiles and most significant changes, including any Budgets that have changed.

Pull Request Reviews provide a helpful performance overview directly in GitHub, with the ability to drill down into metrics further in the Calibre interface.

Calibre does not automatically test draft Pull Requests. Test Draft Pull Requests by marking them as Ready for Review or using a Pull Request command. You can also re-run a Pull Request Review by closing and re-opening your Pull Request.

When your Pull Request is merged and selected branch is successfully deployed, Calibre creates a deployment marker on your charts and takes a new snapshot of your Site. That way, you always know how releases affect your performance metrics.

Pull Request commands

You can control Calibre actions by commenting on a PR, too:

• @calibreapp run: will create a Snapshot and start testing this PR with each new deployment.
• @calibreapp stop: will stop testing and ignore all future deployments.

Deployment statuses

Your repository has to be using GitHub deployment statuses, so Calibre knows what to compare Pull Requests to. Providers such as Vercel, Netlify and Heroku provide these automatically.

If you have a custom CI pipeline, you can mark deployments and create deployment statuses using the GitHub API. Calibre creates a Pull Request Review Site when it receives a deployment status with a state of success. Make sure to set the canonical URL of your review Site in the environment_url field.

Deployment branches

You can include or exclude branches that Calibre runs Pull Request Reviews on by setting the deployment branches using positive and negative patterns.

The order branch patterns are defined matters. The last matching pattern will determine if the branch is included or excluded.

Example: Ignore dependabot PRs

Set the Deployment Branches field to: dependabot/**:

Example: Ignore all branches

Set the Deployment Branches field to: !*. You can trigger Calibre actions by using the Pull Request Command @calibreapp run.

Example: Only compare release branches, ignore alpha releases

Set the Deployment Branches field to: releases/**, releases/**-alpha.

Configuration file

You can customise metrics and authentication settings by including a configuration YAML file in your linked repository at /.calibre/config.yml.

There are two top-level properties you must specify:

• version: The current version is 2
• pullRequestReviews: Options for Pull Request Reviews

Version

There are two versions of the Calibre configuration file.

• Version 1: Single configuration block (Deprecated)
• Version 2: Supports multiples blocks of configuration which can be used for branch specific configuration.

You can set the version by specifying the version number as the value of the version key:

1version: 2
2pullRequestReviews:

1version: 1
2pullRequestReviews:

Configure metrics

By default, Calibre will compare the following metrics:

Under pullRequestReviews in your configuration file you can explicitly define which metrics to use, or you can provide a list of metrics to ignore from all available metrics in Calibre.

When you define metrics, the first three will be shown in the summary table.

Example: Define metrics

The following configuration will only compare the metrics listed under measurements:

1version: 2
2pullRequestReviews:
3  - measurements:
4      - firstRender
5      - assetCount
6      - pageSizeInBytes
7      - lighthousePerformanceScore

1version: 1
2pullRequestReviews:
3  measurements:
4    - lighthousePerformanceScore
5    - lighthouseAccessibilityScore
6    - lighthouseSEOScore

Example: Ignore metrics

The following configuration will include all metrics except the ones listed under ignore:

1version: 2
2pullRequestReviews:
3  - ignore:
4      - lighthousePwaScore
5      - pageHTMLDownload
6      - timeToFirstByte
7      - lighthouseSeoScore
8      - pageWaitTiming

1version: 1
2pullRequestReviews:
3  ignore:
4    - lighthousePwaScore
5    - pageHTMLDownload
6    - timeToFirstByte
7    - lighthouseSeoScore
8    - pageWaitTiming

Configure authentication

Calibre will copy authentication settings to each Pull Request Review test. The authentication URL will be updated to use the same host as the pull request deployment.

If you require specific authentication details for Pull Request Reviews, you can set them using config.yml.

Example: Define authentication settings

The following configuration will be used for authentication:

1version: 2
2pullRequestReviews:
3  - authentication:
4      url: https://different-domain.com/login
5      username: user@email.com
6      password: mysecretpassword
7      formSelector: form
8      usernameSelector: input[name=email]
9      passwordSelector: input[type=password]

1version: 1
2pullRequestReviews:
3  authentication:
4    url: https://different-domain.com/login
5    username: user@email.com
6    password: mysecretpassword
7    formSelector: form
8    usernameSelector: input[name=email]
9    passwordSelector: input[type=password]

Example: Define password only settings

Providers like Netlify and Vercel have the option to password protect preview deployments. By omitting the authentication URL and only setting the password fields, the Calibre Agent will login to the deployment URL with a password before each Snapshot:

1version: 2
2pullRequestReviews:
3  - authentication:
4      password: mysecretpassword
5      passwordSelector: input[type=password]

1version: 1
2pullRequestReviews:
3  authentication:
4    password: mysecretpassword
5    passwordSelector: input[type=password]

Example: Set headers and cookies

You can specify headers and cookies for each Pull Request Review test.

1version: 2
2pullRequestReviews:
3  # Headers
4  - headers:
5      - name: X-Calibre-Test
6        value: Pull Request
7      - name: User-Agent
8        value: Calibre
9
10  # Cookies
11  - cookies:
12      - name: seen-cookie-notice
13        value: true
14      - name: uid
15        value: 1
16        domain: calibreapp.com
17        path: /
18        secure: true
19        httpOnly: true
20
21  # Specify headers and cookies for branches prefixed with `stage-`
22  - headers:
23      - name: X-Environment
24        value: staging
25    cookies:
26      - name: staging-cookie
27        value: true
28    branches:
29      - stage-*

Branch specific configuration

You can specify configuration blocks for different branches using positive and negative patterns.

The order branch patterns are defined matters. The last matching pattern will determine if the branch is included or excluded.

Example: Disable authentication for a preview branch

The following configuration will apply authentication settings for all preview- branches except for preview-public as the last matching pattern for this branch does not require authentication.

1version: 2
2pullRequestReviews:
3  - branches:
4      - preview-*
5    authentication:
6      password: mysecretpassword
7      formSelector: form
8      passwordSelector: input[type=password]
9      required: true
10  - branches:
11      - preview-public
12    authentication:
13      required: false

Example: Report different metrics for feature branches

The following configuration will report on a different set of metrics for devops and ui branches:

1version: 2
2pullRequestReviews:
3  - branches:
4      - devops-*
5    measurements:
6      - timeToFirstByte
7      - pageWaitTiming
8      - pageHTMLDNS
9  - branches:
10      - ui-*
11    measurements:
12      - largestContentfulPaint
13      - totalBlockingTime
14      - lighthousePerformanceScore

Access and security

To use Pull Request Reviews, the following API access to your GitHub respository is required:

• Checks - Read and write
• Deployments - Read only
• Metadata - Read only
• Pull Requests - Read and write
• Issues - Read only
• Access to read files - Read and write access to .calibre/config.yml
• Commit statuses - Read only

In addition to these API items, Calibre will also read the following events for authorised repositories:

• Check run
• Check suite
• Deployment
• Deployment status
• Pull request
• Status
• Issue Comment

Your team authorises access during installation and will be prompted to confirm any changes in access requirements.

If you have any questions or concerns, let us know at security@calibreapp.com.