chore: adds quality gate for rerunning e2e spec files that are new or have been modified #24556

seaona · 2024-05-16T11:15:53Z

Description

This PR adds a quality gate for new or modified e2e spec files. Whenever there is a PR which modifies or changes a test, this will be run more times, in order to prevent introducing a flakiness accidentally. It is done as follows:

Identifies any new or modified e2e file from inside the test/ folder using git diff and using these 2 filters:
- file.filename.startsWith('test/e2e/') &&
- file.filename.endsWith('.spec.js') || file.filename.endsWith('.spec.ts')
Copies the given specs x5 times in the list of testpaths to execute -> this number is arbitrary, we could modify it to any value we want. The reason for taking this approach instead of changing the retrial number is to benefit of the parallelization, as @HowardBraham pointed out in a comment.
Since we already had a flag which could support the re-running successful tests, --retry-until-failure I just leveraged this into the for loop for each test, and if that testcase was identified as new/modified, the flag is added so the new tests fail fast without retrials

Incremental git fetch depth within shallow clone

We use git fetch with incremental depth as @danjm suggested. The ci environment uses a shallow clone, meaning we won't be able to succeed just by using git diff as it won't find the merge base. For fixing that, we start with a git fetch depth of 5, and keep incrementing the depth it the error is no merge base up until 50. Beyond 50, the job will fail and the PR should Update branch, so we are able to do the git diff successfully. The assumption here is that if the branch is that behind, they would need to update their branch anyway possibly due to conflicts with develop.

Related issues

Fixes: #24009

Manual testing steps

Check ci runs (notice previous runs had failing and changed tests on purpose, in order to try the different scenarios described below)

Screenshots/Recordings

Git diff with incremental git fetch

Example on how this works: https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/85809/workflows/1d7a4658-1dd9-460f-9fd4-518858982329/jobs/3115301

=============================================== [UPDATE with the new code changes]

🟢 Case 1: A test has changed -> it's rerun 1+5 times and it's successful (it will be run in different buckets)

https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/85823/workflows/d5de5e43-48b4-4b98-b2f8-863e7e30e39e/jobs/3116106/parallel-runs/7?filterBy=ALL

🟢 Case 2: A test has changed, but it has a mistake in the code (intentionally to simulate a flaky test) -> it fails immediately and there are no more retries. Also the rest of the tests, are retried if they failed as usual

https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/85823/workflows/d5de5e43-48b4-4b98-b2f8-863e7e30e39e/jobs/3116106/parallel-runs/7?filterBy=ALL

reruns-old-no-reruns-new.mp4

🟢 Case 3: A PR has no test spec files changed -> nothing different happens

Note: since we are using parallelization, this means that the same test new/edit spec can be placed in different buckets, so if it fails, it can fail in different buckets at the same time.

Pre-merge author checklist

I’ve followed MetaMask Coding Standards.
I've completed the PR template to the best of my ability
I’ve included tests if applicable
I’ve documented my code using JSDoc format if applicable
I’ve applied the right labels on the PR (see labeling guidelines). Not required for external contributors.

Pre-merge reviewer checklist

I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

github-actions · 2024-05-16T11:16:06Z

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

test/e2e/run-all.js

seaona · 2024-05-16T14:33:16Z

test/e2e/run-all.js

@@ -212,12 +213,26 @@ async function main() {

  console.log('My test list:', myTestList);

+  const changedOrNewTests = await fetchChangedE2eFiles();
+  console.log('Spec files that will be re-run:', changedOrNewTests);


this is left for debugging purposes

You're going to sacrifice some parallel execution speed by doing it this way. It also only has to be done if it's running on CircleCI, and not locally.

It would be a better approach if at the top of function runningOnCircleCI(), you looked at testPaths and then duplicated 5 times each thing that was also in changedOrNewTests.

This would allow it to distribute the workload evenly across the VMs.

thank you for your suggestion @HowardBraham That's definetly a good point! I was thinking it would be negligible given the small amount of retries, but it's true that we can benefit from parallelization witha small tweak. I'll add the changes 🙇‍♀️

changes added 👍

test/e2e/run-all.js

…logic is working properly in failing tests

test/e2e/run-all.js

seaona · 2024-05-21T07:56:42Z

development/lib/retry.js

+  if (retryUntilFailure) {
+    return null;
+  }
+


before it assumed that if we reach the max retries, it was an error and throw the error with rejection message 'Retry limit reached', but in the case of using retryUntilFailure, reaching the max retry limit it's actually a good thing, as it means the test has not failed. That's why in this case I'm returning just null

Can we rename this parameter stopAfterOneFailure for better clarity?

changed 👍

codecov · 2024-05-21T09:50:02Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 65.70%. Comparing base (0dc77eb) to head (c3dbacf).
Report is 32 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop   #24556   +/-   ##
========================================
  Coverage    65.70%   65.70%           
========================================
  Files         1369     1369           
  Lines        54366    54366           
  Branches     14149    14149           
========================================
  Hits         35718    35718           
  Misses       18648    18648

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

metamaskbot · 2024-05-21T13:05:13Z

Builds ready [71bbf50]

builds: chrome, firefox
builds (beta): chrome
builds (flask): chrome, firefox
builds (MMI): chrome, firefox
builds (test): chrome, firefox
builds (test-flask): chrome, firefox
build viz: Build System
mv3: Background Module Init Stats
mv3: UI Init Stats
mv3: Module Load Stats
mv3: Bundle Size Stats
mv2: E2e Actions Stats
code coverage: Report
storybook: Storybook
typescript migration: Dashboard
all artifacts
bundle viz:
- background: 0, 1, 2, 3, 4, 5, 6
- common: 0, 1, 2, 3, 4, 5, 6, 7, 8
- content-script: 0
- ui: 0, 1, 10, 2, 3, 4, 5, 6, 7, 8, 9

Page Load Metrics (623 ± 464 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	60	133	88	21	10
		domContentLoaded	9	54	17	12	6
		load	48	2385	623	966	464
		domInteractive	9	54	17	12	6

Bundle size diffs

background: 0 Bytes (0.00%)
ui: 0 Bytes (0.00%)
common: 0 Bytes (0.00%)

danjm · 2024-05-22T15:33:54Z

test/e2e/fetch-changed-files.js

@@ -0,0 +1,33 @@
+const axios = require('axios');


I'm not sure, but we might be able to avoid a fetch to github here, and instead just use git. Not sure if we have the repo and git history in the right time and place on CI to do it, but it would be good if we could: more efficient than a network request, and not dependent on network conditions or the github api possibly being down.

thank you Dan! That is a good point. I have an alternative branch where I explored a bit the git ci option, however it was not straight-forward and I believe we might need to tweak some things on the config.yml file in order to accomplish it. I decided to go for this option to not over-engineer it, but for the reason you mention it might be worth to try the ci option a bit further.

https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/80853/workflows/9ac00125-66fc-400d-82ac-3f22b4c928c1/jobs/2859112

I didn't try but something like this might work?

gitDiff = await git.diffSummary(['--name-only', `origin/develop...${process.env.CIRCLE_SHA1}`, 'test/**/*.spec.*s']);

https://github.com/MetaMask/metamask-extension/blob/develop/development/generate-rc-commits.js#L2

https://github.com/steveukx/git-js?tab=readme-ov-file#git-diff

Does the local git command like that work if you're doing a shallow checkout? Also, the way this is written, it will be hitting the GitHub API hundreds of times per workflow (because there are hundreds of parallel machines running the workflow). Perhaps do it in prep-deps, and then persist_to_workspace?

Good point on the shallow checkout. Could prob be addressed by a clever enough git fetch invocation (possibly in prep-deps)..? 🤔

thank you for the suggestions 🙏 ❤️ In the shallow clone we do this:

git clone --depth 1 --no-checkout "$CIRCLE_REPOSITORY_URL" . git fetch --depth 1 origin "$CIRCLE_SHA1"

I was thinking we would need to do git fetch develop explicitly to get available the develop branch commits in the ci environment? 🤔 I'm also not sure if we would need to increase the depth too here 🤔 I can investigate this further

logic for git fetch with depth and git diff has been added

HowardBraham · 2024-05-24T03:00:44Z

test/e2e/run-all.js

+      const retryIndex = args.indexOf('--retries');
+      if (retryIndex !== -1) {
+        args.splice(retryIndex, 2);
+      }
+
+      const extraArgs = isTestChangedOrNew
+        ? ['--retry-until-failure', `--retries=${retriesForChangedOrNewTests}`]
+        : [];


There's no need for this duct-taped workaround complexity. Look at line 191 in this file. Just put retries into extraArgs down here instead of args up there.

(if you go with my other suggestion about runningOnCircleCI though, you can probably remove most of this anyway)

thank you!! if we follow the approach of copying the spec files x5 times on runningOnCircleCI then we can indeed remove all of this logic. Just something I'm wondering, would it be okay then to treat each of those new specs the same as rest (same flags)? or we still want them to fail immediately with no-retries?

Given that @legobeat has a PR for reducing test retries, maybe it's fine to treat all the tests equally; then there's no need to add any extra flag, and just copy the spec file x times in the test list from runningOnCiircleCi, and this would simplify a lot the logic

What do you think?

ℹ️ retries has been removed and instead we add them in the testpaths x5 times, so we benefit from parallelization.
Now only the argument for failing fast is passed here

## **Description**  [![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/MetaMask/metamask-extension/pull/24787?quickstart=1) ## **Related issues** Fixes: ## **Manual testing steps** 1. Go to this page... 2. 3. ## **Screenshots/Recordings**  ### **Before**  ### **After**  ## **Pre-merge author checklist** - [ ] I’ve followed [MetaMask Coding Standards](https://github.com/MetaMask/metamask-extension/blob/develop/.github/guidelines/CODING_GUIDELINES.md). - [ ] I've completed the PR template to the best of my ability - [ ] I’ve included tests if applicable - [ ] I’ve documented my code using [JSDoc](https://jsdoc.app/) format if applicable - [ ] I’ve applied the right labels on the PR (see [labeling guidelines](https://github.com/MetaMask/metamask-extension/blob/develop/.github/guidelines/LABELING_GUIDELINES.md)). Not required for external contributors. ## **Pre-merge reviewer checklist** - [ ] I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed). - [ ] I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

danjm · 2024-05-29T12:36:49Z

.circleci/scripts/get-changed-files.sh

+echo "$DIFF_RESULT"
+
+# Store the output of git diff
+git diff --name-only develop..."$CIRCLE_SHA1" >> changed-files/changed-files.txt


maybe this should be a diff with origin/develop? because above there is the code to git fetch origin develop, but those commits are not pulled locally

thank you for the suggestion, I tried it but it does not seem to make any difference 🤔

ℹ️ the problem was that the branch was way behind the depth. Updating the branch fixed it

seaona · 2024-06-06T10:48:20Z

.circleci/scripts/git-diff-develop.js

+}
+
+async function gitDiffWithRetry() {
+  const depths = [5, 10, 15, 20, 30, 40, 50];


to consume fewer resources, we start with small depth jumps +5 and later on we increment with +10 jumps

seaona · 2024-06-06T10:48:40Z

.circleci/scripts/git-diff-develop.js

+    console.error('An error occurred:', error.message);
+    process.exit(1);
+  }
+}


leaving all the logs so we are able to debug ci if needed

metamaskbot · 2024-06-06T11:12:43Z

Builds ready [c3dbacf]

builds: chrome, firefox
builds (beta): chrome
builds (flask): chrome, firefox
builds (MMI): chrome, firefox
builds (test): chrome, firefox
builds (test-flask): chrome, firefox
build viz: Build System
mv3: Background Module Init Stats
mv3: UI Init Stats
mv3: Module Load Stats
mv3: Bundle Size Stats
mv2: E2e Actions Stats
code coverage: Report
storybook: Storybook
typescript migration: Dashboard
all artifacts
bundle viz:
- background: 0, 1, 2, 3, 4, 5
- common: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9
- content-script: 0
- offscreen: 0
- ui: 0, 1, 10, 2, 3, 4, 5, 6, 7, 8, 9

Page Load Metrics (49 ± 3 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	68	93	79	7	3
		domContentLoaded	9	11	9	1	0
		load	42	65	49	6	3
		domInteractive	9	11	9	1	0

Bundle size diffs

background: 0 Bytes (0.00%)
ui: 0 Bytes (0.00%)
common: 0 Bytes (0.00%)

quality gate mock alt

e66f2fa

seaona commented May 16, 2024

View reviewed changes

test/e2e/run-all.js Outdated Show resolved Hide resolved

seaona added 3 commits May 16, 2024 13:33

fix export

d2298f6

fix PR number match

d89be42

overwrite retries number for retry-until-failure cases

57269c9

seaona changed the title ~~chore: quality gate mock alt~~ chore: adds quality gate for rerunning e2e spec files that are new or have been modified May 16, 2024

seaona added 2 commits May 16, 2024 16:17

change file to ts and add log for debugging

3a8b0fa

revert js

90690cd

seaona commented May 16, 2024

View reviewed changes

test/e2e/run-all.js Outdated Show resolved Hide resolved

add ts spec files to the filter and make another test fail to verify …

65b3467

…logic is working properly in failing tests

DDDDDanica reviewed May 20, 2024

View reviewed changes

test/e2e/run-all.js Outdated Show resolved Hide resolved

seaona added 2 commits May 21, 2024 09:19

address dev review: move retries to variable remove try/catch

df8d21e

leave the specs as they were before (changed for ci testing purposes)

5029d49

seaona commented May 21, 2024

View reviewed changes

Merge branch 'develop' into quality-gate-gh

71bbf50

seaona marked this pull request as ready for review May 21, 2024 15:08

seaona requested review from kumavis and a team as code owners May 21, 2024 15:08

seaona added the team-extension-platform label May 21, 2024

seaona self-assigned this May 21, 2024

DDDDDanica previously approved these changes May 21, 2024

View reviewed changes

vthomas13 previously approved these changes May 21, 2024

View reviewed changes

seaona added the DO-NOT-MERGE Pull requests that should not be merged label May 22, 2024

danjm reviewed May 22, 2024

View reviewed changes

seaona requested review from vthomas13 and DDDDDanica May 23, 2024 11:17

HowardBraham reviewed May 24, 2024

View reviewed changes

seaona dismissed stale reviews from vthomas13 and DDDDDanica via 1c728ac May 27, 2024 14:13

seaona requested a review from a team as a code owner May 27, 2024 14:13

danjm reviewed May 29, 2024

View reviewed changes

address comments

c46b3b8

seaona marked this pull request as draft June 4, 2024 09:38

seaona and others added 15 commits June 4, 2024 11:48

origin develop

b6d1f45

depth update

870f875

50 fetch

81f954a

rename funcs

8c0516c

Merge branch 'develop' into quality-gate-gh

bc6b76c

switch from sh to js file

8179d2d

fix changedfilesUtils

27a4be3

move script to config file

af6e4d6

git diff incremental depth

e781327

fix git fetch incremental

a26f7ea

git fetch pr branch with depth too

cd0027f

console log path

6ef0174

define output path

48482c8

format spec entries with path and new lines

b061ba5

fix back the spec files for testing ci

580eee5

seaona removed the DO-NOT-MERGE Pull requests that should not be merged label Jun 6, 2024

only do for loop if there are changed e2e files

c3dbacf

seaona marked this pull request as ready for review June 6, 2024 10:34

seaona commented Jun 6, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: adds quality gate for rerunning e2e spec files that are new or have been modified #24556

chore: adds quality gate for rerunning e2e spec files that are new or have been modified #24556

seaona commented May 16, 2024 •

edited

github-actions bot commented May 16, 2024

seaona May 16, 2024

HowardBraham May 24, 2024

seaona May 24, 2024

seaona Jun 6, 2024

seaona May 21, 2024

HowardBraham May 24, 2024

seaona Jun 6, 2024

codecov bot commented May 21, 2024 •

edited

metamaskbot commented May 21, 2024

danjm May 22, 2024

seaona May 23, 2024

legobeat May 23, 2024 •

edited

HowardBraham May 24, 2024

legobeat May 24, 2024 •

edited

seaona May 24, 2024

seaona Jun 6, 2024

HowardBraham May 24, 2024

HowardBraham May 24, 2024

seaona May 24, 2024

seaona Jun 6, 2024

danjm May 29, 2024

seaona Jun 4, 2024

seaona Jun 6, 2024

seaona Jun 6, 2024

seaona Jun 6, 2024

metamaskbot commented Jun 6, 2024

chore: adds quality gate for rerunning e2e spec files that are new or have been modified #24556

Are you sure you want to change the base?

chore: adds quality gate for rerunning e2e spec files that are new or have been modified #24556

Conversation

seaona commented May 16, 2024 • edited

Description

Incremental git fetch depth within shallow clone

Related issues

Manual testing steps

Screenshots/Recordings

Git diff with incremental git fetch

Pre-merge author checklist

Pre-merge reviewer checklist

github-actions bot commented May 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented May 21, 2024 • edited

Codecov Report

metamaskbot commented May 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

legobeat May 23, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

legobeat May 24, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

metamaskbot commented Jun 6, 2024

seaona commented May 16, 2024 •

edited

codecov bot commented May 21, 2024 •

edited

legobeat May 23, 2024 •

edited

legobeat May 24, 2024 •

edited