Run tests weekly

[Aerijo]

Prerequisites

NA

Description

Two for two packages I've worked on recently have failing specs on the master branch (errored even when all changes in the PR were reverted).

• add windows core package detection snippets#307: I believe an Electron upgrade changed the behaviour of Object.keys, so a pre-existing bug was uncovered
• Fix default location getter tree-view#1350: A dependency is just refusing to compile

It's difficult enough to write a good PR, and for the specs to fail in an unrelated way is really disheartening.

I would really appreciate scheduling tests of all packages so that things like this can be caught. Travis CI allows this. Successes can be ignored, failures quickly checked for network issues (and restarted), and true failures / flaky tests will be found a lot quicker than whenever someone makes the next PR.

/cc @lkashef @darangi

[DeeDeeG]

I think an Azure DevOps Pipeline .yml file could be made for this.

• Boostrap and build Atom
• run just the package tests
  • Could be on all three OSes? (and 32-bit Atom for Windows as a fourth platform?)

Set this to run every so often.

(As far as I know, package tests are run on every PR and on master, but only on the macOS CI image.)

Edit: Unless you mean running the tests at the respective repos for each package.

[Aerijo]

Edit: Unless you mean running the tests at the respective repos for each package.

Exactly. The two examples above would mean that either the main repo tests are not running package tests, or they should have been failing.

And if the package tests aren't being run (or are failing), I don't know how you can confidently upgrade Electron.

[DeeDeeG]

Yeah, I have noted that the CI hasn't been run in a while and devDependencies for several of the packages are out of date. A lot of them last tested against Electron 4, 1, 0.1, etc. (I just recently posted a PR to fix CI for atom/etch.)