Testing

Describe testing categories and tools.

Python test scripts in the repository tests/ directory are run with pytest:

python -m pytest tests/

We use a few pytest plugins. To install them first, it should be sufficient to use the requirements-testing.txt file in the root of the repository.

pip install -r requirements-testing.txt

Some test scripts or examples rely on data that lives in a separate testdata repository. This repository is noted as a git submodule in the .gitmodules configuration file. GitHub respects and records the repository link, so the testdata repository will be cloned with https://github.com/SCALE-MS/scale-ms if you used the --recursive option to git clone. Otherwise, you will have to manually initialize and update the submodule or otherwise acquire the test data.

Unit tests, system tests, and integration tests

The test suite in tests/ includes a lot of integration tests that probe interactions with full RADICAL Pilot sessions because we have had problems reproducing viable execution environments and because we use RP features and interfaces that may not be mature.

Refer to the docker/README.md in the repository for more about reproducible testing environments.

We install several pytest markers and command line options for the tests in tests/. Refer to tests/conftest.py or use pytest tests --help (look for the custom options: section).

Acceptance tests

The automated GitHub Actions test pipeline includes command line invocations of example scripts. We should continue to run short examples to ensure that the scripting interface behaves as expected.