Improve documentation for testing DevTools.

Restructure the current testing documentation with a single
entry point in `test/README.md` and put the E2E and unit test
documentation into a single spot (as much as possible).

There's still room for improvement, but this should give us
a better starting point.

Fixed: 343896438
Change-Id: I2a7881aa2ca7d32c8d20b356ddb7a776c00a5047
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/5776289
Reviewed-by: Danil Somsikov <dsv@chromium.org>
Auto-Submit: Benedikt Meurer <bmeurer@chromium.org>
Commit-Queue: Danil Somsikov <dsv@chromium.org>

Author: bmeurer

docs/README.md

@@ -23,6 +23,7 @@ below.**
 
 *   [Get the Code](get_the_code.md)
 *   [UX Style Guide](./styleguide/ux/README.md)
+*   [Testing Guide](../test/README.md)
 *   [Contributing Changes](contributing_changes.md)
 *   [Chrome DevTools Design Review Guidelines](design_guidelines.md)
 *   [Release Management](release_management.md)
@@ -35,12 +36,6 @@ below.**
     *   [How to add UMA metrics in DevTools frontend](add_uma_metrics.md)
 *   [How to add experiments in DevTools frontend](add_experiments.md)
 
-### Testing
-
-*   [Testing Chromium DevTools](testing.md)
-*   [E2E test guide](../test/e2e/README.md)
-*   [Unit test guide](unit_testing.md)
-
 ### Architectural Documentation
 
 *   [Architecture of DevTools](architecture_of_devtools.md)

docs/images/debugging-e2e-tests-with-vscode.png

@@ -1,3 +1,89 @@
+# Chromium DevTools Unit Testing Helpers
+
 This folder contains helpers for writing Karma unit tests, most of which exist to help render and test components.
 
-See the [unit testing documentation](../../docs/unit_testing.md) for more information.
+See the [unit testing documentation](../../test/unit/README.md) for more information.
+
+[TOC]
+
+## DOM Helpers
+
+### Rendering a component in a test
+
+When running a Karma unit test, the DOM that you create during a test is automatically cleaned up for you before the next test, ensuring that no tests are impacted by stray DOM left behind from a previous test.
+
+To ensure you render your component into the test DOM, use the `renderElementIntoDOM` helper, which takes any `HTMLElement` and renders it. By using this helper you ensure that it's cleaned up at the end of the test.
+
+### Asserting presence of variables for TypeScript
+
+When trying to read a component's shadow DOM, TypeScript will ask that you check it's not null first:
+
+```
+component.shadowRoot.querySelector('.foo') // TS will error here: shadowRoot may be `null`.
+```
+
+The `assert.isNotNull` method will do this for you and fail the test if the shadow root is not present:
+
+```
+assert.isNotNull(component.shadowRoot);
+component.shadowRoot.querySelector('.foo') // TS is happy!
+```
+
+When you query elements from the DOM, you can use `assert.instanceOf` or `assertElements` to check that an element is the expected type. This will ensure the test fails if the DOM is not as expected, and satisfy TypeScript:
+
+```
+const button = component.shadowRoot.querySelector('button.foo');
+assert.instanceOf(button, HTMLButtonElement);
+
+const allDivs = component.shadowRoot.querySelectorAll('div');
+assertElements(allDivs, HTMLDivElement);
+```
+
+## Mutation helpers
+
+When building components we want to ensure that they are efficient and do not update the DOM more than necessary. We can use the mutation helpers to assert on the amount of DOM changes in a given test. These use a [Mutation Observer](https://developer.mozilla.org/en/docs/Web/API/MutationObserver) to track any mutations.
+
+_Important!_: These tests are async, so you must `await` them.
+
+### Expecting mutations
+
+If you are expecting mutations, you can wrap the part of the test that triggers the mutation in a `withMutation` call. The first argument is an array of mutations you expect (read on for the structure of it), the second is the element to observe (will nearly always be `component.shadowRoot`, and a callback function that takes the observed shadow root. It is this function that will be executed whilst the DOM is being observed for mutations.
+
+```
+it('adds a new button when the data changes', async () => {
+  const component = new MyComponent();
+  component.data = {...}
+  renderElementIntoDOM(component);
+
+  await withMutations([{
+    type: MutationType.ADD, // MutationType is an exported enum from MutationHelpers
+    tagName: 'button',
+    max: 2, // the maximum number of mutations to allow; defaults to 10
+  }], component.shadowRoot, shadowRoot => {
+    // do something that updates the component and causes mutations
+    component.data = {...}
+  })
+})
+```
+
+This test will assert that updating the component causes _at most 2 additions_ to the component's shadow DOM. You can pass `MutationType.ADD` or `MutationType.REMOVE` as the `type` to watch for just additions/removals, or you can omit the `type` key and have the test watch for both additions and removals.
+
+You don't need to wrap every test in a `withMutation` block, but if you are testing some code that should update the DOM, it's a good idea to ensure we aren't unexpectedly doing much more work than expected.
+
+### Expecting no mutations
+
+If you have a test where no DOM should mutate, you can use `withNoMutations`. This is the opposite of `withMutations`; it will fail the test should it detect any DOM mutations. Using this helper when testing the `ElementBreadcrumbs` component discovered that when we scrolled the breadcrumbs we caused _over 70 DOM mutations_ (!), which is what lead to the creation of this helper. It's a great way to verify that we're not doing work that can be avoided.
+
+```
+it('does not mutate the DOM when we scroll the component', async () => {
+  const component = new MyComponent();
+  component.data = {...}
+  renderElementIntoDOM(component);
+
+  await withNoMutations(component.shadowRoot, shadowRoot => {
+    // Imagine there's code here to cause the component to scroll
+  })
+})
+```
+
+The above test will fail if any DOM mutations are detected.

docs/images/debugging-unit-tests-with-devtools.png

@@ -1,56 +1,38 @@
-# Testing Chromium DevTools
+# Unit Testing
 
-Follow the steps outlined in [Get the Code](get_the_code.md) to checkout the DevTools front-end code.
+[goo.gle/devtools-testing-guide](http://goo.gle/devtools-testing-guide)
+
+Follow the steps outlined in [Get the Code](../docs/get_the_code.md) to checkout the DevTools front-end code.
 
 [TOC]
 
-## DevTools frontend
+## DevTools front-end tests
+
+The `devtools-frontend` repository contains a variety of test suites, check
+out the individual guides below:
 
-Run tests with
+* [Unit Testing Guide](./unit/README.md)
+* [Interactions Testing Guide](./interactions/README.md)
+* [E2E Testing Guide](./e2e/README.md)
+* [Performance Testing Guide](./perf/README.md)
+
+You can use
 
 ```
 npm test
 ```
 
-This command runs all tests in the devtools frontend repo, which are the [unit tests](../test/unit/README.md),
-[interactions tests](../test/interactions/README.md), [e2e tests](../test/e2e/README.md), and performance tests.
-
-You can also run just a subset of tests like this:
+to run all tests in the devtools frontend repo. You can also run just a
+subset of tests like this:
 
 ```
 npm test \
    front_end/core/common/Color.test.ts \
    front_end/core/sdk
 ```
 
-
-The current test status can be seen at the [test waterfall](https://ci.chromium.org/p/devtools-frontend/g/main/console).
-
-
-### Debugging
-
-#### Debugging with VSCode
-
-To run tests under the debugger, open the "Run and Debug" sidebar, select "Run unit tests in VS Code debugger" from the
-dropdown, and click the start button or press F5.
-
-Current limitations when using VSCode for e2e and interactions tests:
-
-- VSCode only attaches to the node portion of the code (mostly the test files and the test helpers), not to Chrome.
-- VSCode debugging only works with headless mode.
-
-#### Debugging with DevTools
-
-To run tests under the DevTools debugger use the `--debug` command line option.
-
-For unittests this will bring up Chrome with a Karma launcher page. Wait for "Debug" button to appear and click it. A
-new page will open, here you can open DevTools, set breakpoints in the tests and reload page to rerun tests.
-
-For e2e tests, you can debug the "DevTools under test" with DevTools-on-DevTools. Use the standard DevTools key
-combination to open another DevTools instance while you look at the "DevTools under test". You can set breakpoints and
-inspect the status of the "DevTools under test" this way. You can debug the puppeteer side by inspecting the Node.js
-process that runs the e2e suite. Either open `chrome://inspect` or click the Node.js icon in any open DevTools window to
-connect to the puppeteer process. You can step through the puppeteer test code this way.
+The current test status can be seen at the
+[test waterfall](https://ci.chromium.org/p/devtools-frontend/g/main/console).
 
 ### Obtaining code coverage
 

docs/images/debugging-unit-tests-with-vscode.png

@@ -1,4 +1,4 @@
-# Guide on end-to-end testing
+# E2E Testing
 
 This directory hosts the end-to-end tests we run on DevTools. These tests open a target page and a DevTools frontend page, for which the DevTools frontend connects to the target page over CDP. We use [Puppeteer] to talk over CDP and all functionality of Puppeteer is available to you as well when writing end-to-end tests. We use [Mocha] as testing framework.
 
@@ -9,18 +9,132 @@ The tests therefore have a dual purpose:
 1. Verify that core user stories are working as intended and are not broken by a particular DevTools frontend change.
 1. Serve as documentation and reference point for how DevTools is intended to be used.
 
-## Skipping tests
+[TOC]
 
-You can disable a test for all platforms using `it.skip`. If you are disabling a flaky test, consider disabling it only on the affected platforms. For example,
+## Running E2E tests
+
+The end-to-end tests are implicitly run as part of `npm run test`,
+but that also runs all the other test suites. To run only
+**all end-to-end tests**, use:
+
+```
+npm run test test/e2e
+```
+
+To use `out/Debug` instead of the default `out/Default` target
+directory, use:
+
+```
+npm run test -- -t Debug test/e2e
+```
+
+To run the end-to-end tests in **debug mode**, use:
+
+```
+npm run test -- --debug test/e2e
+```
+
+To run only **specific end-to-end tests** from a single `_test.ts`
+file, say `console-log_test.ts` for example, use:
+
+```
+npm run test test/e2e/console/console-log_test.ts
+```
+
+Check the output of `npm run test -- --help` for an overview of
+all options.
+
+## Debugging E2E tests
+
+You can debug the "DevTools under test" with DevTools-on-DevTools. Use the
+standard DevTools key combination to open another DevTools instance while
+you look at the "DevTools under test". You can set breakpoints and inspect
+the status of the "DevTools under test" this way. You can debug the puppeteer
+side by inspecting the Node.js process that runs the e2e suite. Either open
+`chrome://inspect` or click the Node.js icon in any open DevTools window to
+connect to the puppeteer process. You can step through the puppeteer test
+code this way.
+
+### Debugging E2E tests with VSCode
+
+There's experimental support for running unit tests directly from
+within VSCode. Open the "Run and Debug" sidebar, select "Run end-to-end tests
+in VS Code debugger" from the dropdown, and click the start button or
+press F5.
+
+![Debugging E2E tests with VSCode](../../docs/images/debugging-e2e-tests-with-vscode.png "Debugging E2E tests with VSCode")
+
+Current limitations when using VSCode for e2e and interactions tests:
+
+- VSCode only attaches to the node portion of the code (mostly the test files
+  and the test helpers), not to Chrome.
+- VSCode debugging only works with headless mode.
+
+## Dealing with flaky E2E tests
+
+To skip a flaky E2E test, create a new bug on [crbug.com](https://crbug.com) in the
+`Chromium > Platform > DevTools` component, and modify the `it` or `describe`
+block accordingly by adding `.skip` to it, adding a preceeding comment
+why the test is skipp and adding the `crbug.com` reference to the test
+block string. For example
+
+```js
+describe('Foo', () => {
+  it('can return bar', () => {
+    assert.strictEqual((new Foo()).bar(), 'bar');
+  });
+
+  ...
+});
+```
+
+would be changed to look like this
+
+```js
+describe('Foo', () => {
+  // Flaking on multiple bots on CQ after recent CL xyz.
+  it.skip('[crbug.com/12345678] can return bar', () => {
+    assert.strictEqual((new Foo()).bar(), 'bar');
+  });
+
+  ...
+});
+```
+
+if only the one test case should be skipped, or like this
+
+```js
+// Flaking on multiple bots on CQ after recent CL xyz.
+describe.skip('[crbug.com/12345678] Foo', () => {
+  it('can return bar', () => {
+    assert.strictEqual((new Foo()).bar(), 'bar');
+  });
+
+  ...
+});
+```
+
+if all the tests for `Foo` should be skipped. If you are disabling a flaky test,
+consider disabling it only on the affected platforms. For example:
 
 ```js
+import {it} from './relative/path/to/shared/mocha-extensions.js';
+
+...
+
+// Consistently flakes on Mac and Windows bots.
 it.skipOnPlatforms(['mac', 'win32'], '[crbug.com/xxx] ...', () => {...});
-it.skipOnPlatforms(['linux'], '[crbug.com/xxx] ...', () => {...});
+
+// Skipped on Linux because the world isn't round.
+describe.skipOnPlatforms(['linux'], '[crbug.com/xxx] ...', () => {...});
 ```
 
-To use `skipOnPlatforms`, you need to import `it` from `test/shared/mocha-extensions.ts`.
+*** note
+**Note:** To use `skipOnPlatforms`, you need to import `it` or `describe`
+from `test/shared/mocha-extensions.ts`.
+***
 
-## Debugging flaky tests
+### De-flaking E2E tests
 
 The `it.repeat` helper is useful for reproducing a flaky test failure. e.g.
 
@@ -30,8 +144,6 @@ it.repeat(20, 'find element', async () => {...});
 
 `it.repeat` behaves like `it.only` in that it will cause just that single test to be run.
 
-## Debugging flaky tests
-
 To see if certain tests are flaky you can use the E2E stressor bots. Open a CL with your test changes and run the following command specifying your test file:
 
 ```sh

front_end/testing/README.md

@@ -1,4 +1,4 @@
-# Interaction tests
+# Interaction Testing
 
 Interaction tests are used to test individual pieces of DevTools in isolation - whether that be a single UI component, or an entire panel, or something inbetween. They load up the page in the browser and use Puppeteer to query it.
 

docs/testing.md renamed to test/README.md

@@ -0,0 +1,11 @@
+# Chromium DevTools Testing Guide
+
+[logo]: https://github.com/ChromeDevTools/devtools-logo/raw/master/logos/png/devtools-circle-48.png
+[home]: /test/README.md
+
+* [Home][home]
+* [Unit Testing](/test/unit/README.md)
+* [E2E Testing](/test/e2e/README.md)
+* [Interaction Testing](/test/interactions/README.md)
+* [Performance Testing](/test/perf/README.md)
+* [DevTools Documentation](/docs/README.md)

test/e2e/README.md

@@ -0,0 +1,3 @@
+# Performance Testing
+
+TBD