TESTING.md

Rama main · réplica de lectura del repositorio original.

## Unit Tests

Unit tests are executed with `npm run build` and `npm run test`. GitHub Actions won't let PRs merge if the tests fail.

To run on Windows, use:

```
jest --testMatch */**/*.test.ts
```

## Integration Test

We have one set of [integration tests](test/default.integ.ts). It is a CDK app that creates a set of runners. Once deployed, the runners should be tested using the [integ.yml](.github/workflows/integ.yml) workflow. It runs all of these workflows, and each one can also be run on its own:

* [integ-all-providers.yml](.github/workflows/integ-all-providers.yml) checks GitHub integration and that all runner provider types are working properly, including common actions like `actions/cache`, `actions/setup-node` and `actions/setup-python`
  * `integ.yml` runs it twice. The provider selector starts no runners for the first set, as if the webhook was off. Runners started for the second set take those jobs, get detected as stolen and replaced, so both sets should succeed
* [integ-idle-reaper.yml](.github/workflows/integ-idle-reaper.yml) starts a runner for a job that already finished, and checks the idle reaper aborts its step function (and doesn't fail it), the runner is not retried, and it doesn't stay registered on GitHub
* [integ-failed-runner.yml](.github/workflows/integ-failed-runner.yml) lets a Lambda runner time out in the middle of a job, and checks `Clean Up Failed Runner` deleted the runner, `All Attempts Failed` passed on the runner's own error, the runner was retried, the retried runner was aborted by the idle reaper, and no runner stays registered on GitHub
* [integ-ec2-dies.yml](.github/workflows/integ-ec2-dies.yml) powers off an EC2 runner in the middle of a job, like a spot interruption, and checks the step function noticed when its heartbeat timed out, the runner was cleaned up and retried, the retried runner was aborted by the idle reaper, and no runner or instance is left behind
* [integ-steal.yml](.github/workflows/integ-steal.yml) gets the idle reaper to reap a runner before GitHub assigns its job, and checks the runner that ends up with that job is detected as stolen and replaced
* [integ-labels.yml](.github/workflows/integ-labels.yml) checks jobs get runners when their labels match a provider in a different case or only partially, and that a job with labels no provider has stays queued without a runner instead of getting cancelled. It then starts a runner for that job itself
* [integ-redelivery.yml](.github/workflows/integ-redelivery.yml) fails the first webhook delivery of a job on purpose, and checks the webhook redelivery Lambda still gets it a runner. If it doesn't, the run is cancelled
* [integ-cancel.yml](.github/workflows/integ-cancel.yml) lets a step on an EC2 runner time out, so the runner cancels it, and checks the runner's step function still succeeds and no runner or instance is left behind
* [integ-warm.yml](.github/workflows/integ-warm.yml) fills the warm runner pool, which never fills on its own schedule, and checks a job only a warm runner can take runs and the warm runner gets replaced. The pool stops itself 30 minutes later
* [integ-cleanup.yml](.github/workflows/integ-cleanup.yml) checks the runners of every finished job in a workflow run left nothing behind: their step functions succeeded, they are not registered on GitHub, and their EC2 instances are terminated
  * `integ.yml` runs it once all-providers and steal are done. When running it on its own, give it the id of the workflow run to check

The provider selector in [`default.integ.ts`](test/default.integ.ts) gives some of these jobs runners that no other job can take. Running all of them at once, or alongside other jobs, is fine.

Most of the workflows check step function executions and GitHub runners on their own. They use the `Integration Test Role` from the stack, which only workflows running on a branch of this repository can assume. Pull requests, including ones from forks, can't. The role can't read any secrets. To check runners on GitHub, it calls a Lambda function that only answers whether a runner is still registered. To run them on your own account:

1. Change `githubRepository` in [`default.integ.ts`](test/default.integ.ts) to your fork. If your account doesn't have a GitHub OIDC provider yet, turn on `createGitHubOidcProvider`.
2. Deploy the integration test. The workflows get the role from `INTEG_ROLE_ARN` on CodeBuild x64 runners, so there is nothing to set up in GitHub.

The stack is close to CloudFormation's limit of 500 resources. Prefer reusing existing providers over adding new ones for tests.

We keep a snapshot of the CloudFormation template generated by the integration test in [test/default.integ.snapshot](test/default.integ.snapshot). During build time and PR validation, we validate the integration test still results in the same CloudFormation template as the snapshot. GitHub Actions won't let PRs merge if the tests fail.

If the snapshot changes, you should commit it as part of your PR. You should ideally deploy and test it. The PR should mention whether the integration test was actually deployed and tested.

To assert the snapshot hasn't changed, use:

```
npm run integ:default:assert
```

To deploy the integration test, use:

```
npm run integ:default:deploy
```

To update the snapshot, use:

```
npm run integ:default:snapshot
```

## Manual Tests 

Integration tests check most paths, happy and unhappy. This is a list of scenarios we should still manually test before releasing a new version:

* Setup page
  * GitHub app
  * Personal access token
  * GitHub Enterprise Server
* Other tests to be automated in the future
  * Go to [`default.integ.ts`](test/default.integ.ts) and enable the test flags at the top