add section on test suites

Author: lundybernard

docs/pages/principles/testing.md

@@ -99,6 +99,91 @@ minimizing the use of mocks/patches/test-doubles.
 > your test suite(s) grow, taxonomy of test cases, the and the use/need for
 > different kinds of tests will become more clear.
 
+## Test Suites
+
+Not all test cases are the same. In the following sections we will discuss many
+kinds of tests, which serve different purposes and provide different benefits.
+Tests should be divided up into different Suites, which can be run independently
+of one another.
+
+Tests which "Fail Fast" save both developer and compute time. Some tests are by
+necessity very slow. [Unit Tests](#unit-tests) should run extremely quickly in
+just a few seconds at most, while [end-to-end](#end-to-end-tests) require time
+to set up and may depend on slow and unreliable external services. By organizing
+tests into suites based on execution time, you can run fast suites first and
+stop if an error is encountered before running slower suites.
+
+### Advantages of Test Suites
+
+- Developers can run relevant tests quickly and frequently.
+- Tests can 'Fail Fast', while still reporting all failures within that suite.
+- Test cases can be divided up conceptually, making them easier to read and
+  reason about.
+- We can avoid false positives, by not running tests we expect to fail due to
+  external factors.
+
+### Guidelines for Test Suites
+
+- Organize test cases into suites based on the type of test and execution time.
+- Set up test-runners (Make, shell scripts, CI/CD) to run fast suites first, and
+  stop when a suite reports a failure.
+- Use markers to enable developers to run the sub-sets of test which are most
+  relevant to their work, with minimal time spent waiting.
+
+### Creating Test Suites
+
+The simplest way to start, is separating tests into directories inside of the
+`tests/` directory:
+
+```
+tests/
+    |- unit/
+    |- integration/
+    |- e2e/
+```
+
+These suites can be run directly with pytest `pytest tests/unit/`.
+
+[Markers](https://docs.pytest.org/en/stable/example/markers.html) provide an
+additional layer of organization, by labeling individual tests. This lets us
+create specialized suites based on markers, independent of directory structure
+or type of test. For example we can mark extremely slow or flakey tests that are
+conceptually part of a larger suite, and skip them when needed.
+
+First, define a new marker in `pyproject.toml`:
+
+```toml
+[tool.pytest.ini_options]
+markers = [
+    "unit: marks unit tests",
+    "slow: marks tests as slow (deselect with '-m \"not slow\"')",
+    "online: tests which require internet access",
+]
+```
+
+To mark an individual test, decorate the test case:
+
+```python
+import pytest
+
+
+@pytest.mark.slow
+def test_somthing_slow(): ...
+```
+
+To mark every test in a directory, add the following to the `conftest.py` in the
+target folder:
+
+```python
+# tests/unit/conftest.py
+import pytest
+
+
+def pytest_collection_modifyitems(session, config, items):
+    for item in items:
+        item.add_marker(pytest.mark.unit)
+```
+
 ## Project Level Integration Tests
 
 The term "Integration Test" is unfortunately overloaded, and used to describe
@@ -390,11 +475,27 @@ def test_pytest(mocker):
     dangerous_sideffects()
 ```
 
-## A Brief Taxonomy of Test Suites
+## Testing Edgecases
+
+While writing unit tests, you may be tempted to test edgecases. You may have a
+critical private function or algorithm, which is not part of the public API (so
+not a good candidate for External testing) and you are concerned about many
+edgecases that you want to defend against using tests.
+
+It is perfectly valid to write extensive edgecase testing for private code, but
+these tests should be kept separate from the unit test suite. Extensive edgecase
+testing makes tests long, and difficult to read (tests are documentation). They
+can slow down execution, we want unit tests to run first, fast, and often.
+
+- Place them in separate files from unit tests, to improve readability
+- [mark them](https://docs.pytest.org/en/stable/how-to/mark.html) so that they
+  can be run as a separate test suite, after your unit test pass
+
+## Additional Types of Test Suites
 
 A non-exhaustive discussion of some common types of tests.
 
-^_^ Dont Panic ^_^
+Dont Panic!
 
 Depending on your project, you may not need many, or most of these kinds of
 tests.
@@ -453,22 +554,6 @@ production-like system, and run tests against it. Some examples are:
 - Processing data from a pre-loaded test database
 - Manual QA testing
 
-## Testing Edgecases
-
-While writing unit tests, you may be tempted to test edgecases. You may have a
-critical private function or algorithm, which is not part of the public API (so
-not a good candidate for External testing) and you are concerned about many
-edgecases that you want to defend against using tests.
-
-It is perfectly valid to write extensive edgecase testing for private code, but
-these tests should be kept separate from the unit test suite. Extensive edgecase
-testing makes tests long, and difficult to read (tests are documentation). They
-can slow down execution, we want unit tests to run first, fast, and often.
-
-- Place them in separate files from unit tests, to improve readability
-- [mark them](https://docs.pytest.org/en/stable/how-to/mark.html) so that they
-  can be run as a separate test suite, after your unit test pass
-
 ### Fuzz Tests and other slow tests
 
 Testing random input, using tools like Hypothesis, is similar to testing edge