Skip to content

LLD:Testing

Jump to bottom
@greweb edited this page Aug 29, 2023 · 4 revisions

Table of Contents

Introduction

An open source framework for easily writing UI tests for our Electron app. Playwright sets up and tears down your app and allows it to be test-driven remotely. Built on top of Devtools protocol.

Pre-requisites

We assume your setup is running ledger-live monorepo or at least ledger-live-desktop app. If not, please check the installation guide and the wiki.

Visual Studio Code extensions

It's recommended to install those extensions.

Installation

Clone ledger-live repository, install and build dependencies.

git clone https://github.com/LedgerHQ/ledger-live.git
pnpm i --filter="ledger-live-desktop..." --filter="ledger-live"
pnpm build:lld:deps

Setup

Playwright launches Ledger Live application using Electron executablePath and pass the main bundle and other options as parameters.

ℹ️ Please have a look at apps/ledger-live-desktop/tests/fixtures/common.ts .

Execution

Build

Before executing any test, don’t forget to build the app, do it whenever the source code changed.

pnpm desktop build:testing

It must generate bundles in apps/ledger-live-desktop/.webpack/ directory.

Install the playwright dependencies

pnpm desktop exec playwright install --with-deps chromium

Run tests

Run a single test

pnpm desktop test:playwright apps/ledger-live-desktop/tests/specs/<testname>.spec.ts

Run all tests in a directory

pnpm desktop test:playwright apps/ledger-live-desktop/tests/specs/onboarding/

Run all the tests

pnpm desktop test:playwright

Run smoke tests

pnpm desktop test:playwright:smoke

Run all tests matching a tag

pnpm desktop test:playwright --grep @onboarding

Recorder

To speed up test development, you can generate tests by recording your actions.

pnpm desktop test:playwright:recorder

⚠️ Fully recorded tests don’t generate maintainable tests. Please see Development section.

Development

Workflow

Step 1 - Identify elements

Path: src/...

To interact with web elements (buttons, input fields, images, …), they should be identified using a data-test-id for unique elements or a class for multiple elements.

ℹ️ As a convention, this is the pattern we must use: <context>-<text or purpose>-<element type>.

I want to identify the Get started CTA button in onboarding flow.

Look for your element in src/renderer/... and give it a data-test-id attribute.

<Button data-test-id=”onboarding-getstarted-button”>
    Get started
</Button>

❌ Don’t use a className in this case because the element is unique.

Step 2 - Create a page object

ℹ️ Page objects are JavaScript classes.

Path: tests/models/

A page object regroups all elements you can interact with and all actions you can do in a specific app screen.

I want to declare get started button element in a page object related to Onboarding.

Create tests/models/onboardingPage.js

Locators

Declare your element

constructor(page: Page) {
  this.page = page;
  this.getStartedButton = page.locator("data-test-id=onboarding-getstarted-button");
}

Methods

ℹ️ Please name your methods according to the Keyword-driven testing methodology.

Create an action by interacting with declared elements, you can make multiple actions in one method and wait for a specific state if needed.

async getStarted() {
  await this.getStartedButton.click();
  await this.loader.waitFor({ state: "detached" });
};

⚠️ Don’t write any assertions (with expect library) in a page object, make it re-usable by everyone; as an alternative you can use waitFor method if you need to have a better control of the app state.

Example

At the end, it should look like this:

import { Page, Locator } from "@playwright/test";

export class OnboardingPage {
  readonly page: Page;
  readonly getStartedButton: Locator;

  constructor(page: Page) {
    this.page = page;
    this.getStartedButton = page.locator("data-test-id=onboarding-get-started-button");
  }

  async getStarted() {
    await this.getStartedButton.click();
    await this.loader.waitFor({ state: "detached" });
  }
}

Step 3 - Create a test file

Path: tests/specs/

A test file regroups a series of actions and assertions to build a test scenario.

I want to create a file to test onboarding flow.

Create tests/specs/onboarding/new-device.spec.ts

test("Onboarding with a new device", ({ page }) => {
  const onboardingPage = new OnboardingPage(page);

  await test.step("Get started", async () => {
    await onboardingPage.getStarted();
    await expect(page).toHaveScreenshot('next-page.png')
  });
});

Each test() declaration will launch a unique electron/chromium session (cf. Fixtures and apps/ledger-live-desktop/tests/fixtures/common.ts).

Matchers

Playwright uses expect library for test assertions;

  • on page (browser) e.g: await expect(page).toHaveTitle('Ledger Live');
  • or a locator (web element) e.g: await expect(getStartedButton).toBeEnabled();

Visual comparison

ℹ️ Because of the Ledger Live application architecture (Hardware device, Market variation, Blockchain), we are forced to mock basically everything when automated tests are executed.

To make the most of this test framework, we opted for visual comparison, it permits us to assert the whole layout of the app.

Playwright permits to compare screenshots. This assertion should be used carefully to avoid heavy maintenance, as example:

I want to check a button is redirecting me to another page.

Don’t use a visual comparison to check that, await expect(page).toHaveURL(regexp) is enough. Choose wisely your assertions, sometimes it's too overkill to take a screenshot.

Also, be sure to not create duplicates and compare same screen twice.

I want to compare exactly the page I’m seeing.

await expect(page).toHaveScreenshot(name);

ℹ️ You can take a screenshot of the whole page, including overflowing elements.

expect(page.screenshot({ fullPage: true })).toMatchSnapshot(name);

❌ This is currently not working with our application, please see this issue.

I want to organize my screenshot set.

await expect(page).toHaveScreenshot([directory, name]);

I want to compare a single element on the page.

await expect(onboardingPage.getStartedButton).toHaveScreenshot();

Example

At the end, it should look like this:

import test from "../../fixtures/common";
import { expect } from "@playwright/test";
import { OnboardingPage } from "../../models/OnboardingPage";

test("Onboarding new device", async ({ page }) => {
  const onboardingPage = new OnboardingPage(page);

  await test.step("Get started", async () => {
    await onboardingPage.getStarted();
    await expect(page).toHaveScreenshot('next-page.png');
  });
});

Step 4 - Update dataset

Once tests are written, if you made screenshot comparisons, you will need to generate a new dataset of screenshots.

pnpm desktop test:playwright:update-snapshots  apps/ledger-live-desktop/tests/specs/onboarding/new-device.spec.ts

Step 5 - Run test

⚠️ Test your test.

Run it at least 3 times to be sure it's not flaky.

pnpm desktop test:playwright apps/ledger-live-desktop/tests/specs/onboarding/new-device.spec.ts

ℹ️ If you get into troubles, please read this and this.

Playwright API

:note: Here are some resources that might help during a test development.

I want to interact with the browser.

https://playwright.dev/docs/api/class-page

I want to interact with an element.

https://playwright.dev/docs/api/class-locator

I want to create or update a page object.

https://playwright.dev/docs/test-pom

I want to organize my tests.

https://playwright.dev/docs/api/class-test

I want to perform assertions.

https://playwright.dev/docs/assertions

I want to perform visual comparison.

https://playwright.dev/docs/test-snapshots

Continuous Integration

Workflow

Commands

  • Generate new screenshots for all test suites

    /generate-screenshots

    This command is meant to be posted as a comment in your pull request.

    ⚠️ You might not have the permission to do it.

  • Push an empty commit to trigger CI checks

    git commit -m 'Run CI checks' --allow-empty`
git push

Debugging

I want to pause my script execution and check manually what’s going on.

Place page.pause(); in your test file before the line you want to debug.

I want to enable the debugger.

PWDEBUG=1 pnpm desktop test:playwright

I want Chrome DevTools.

DEV_TOOLS=1 pnpm desktop test:playwright

I want verbose logging.

DEBUG=pw:api pnpm desktop test:playwright

Resources

Releases

Documentation

Code coverage



Clone this wiki locally