Testing

Python test scripts in the repository tests/ directory are run with pytest.

Dependencies

RADICAL Pilot installation

RADICAL Pilot has some tricky interactions with Python environment virtualization and manipulation that lead to some caveats.

  • Do not try to use “editable” installs.

  • Explicitly activate the virtual environment before running a script that uses RP.

radical.pilot will probably not find the data files or entry point scripts that it relies on unless you first “source” the activate script for a venv containing a regular radical.pilot installation. Perform testing with the virtual environment active. Do not try to exercise the installation or virtual environment through other techniques, such as invoking a Python venv by explicitly calling the sym-linked .../bin/python.

pytest

We use a few pytest plugins. To install them first, it should be sufficient to use the requirements-testing.txt file in the root of the repository.

pip install -r requirements-testing.txt

Test data

Some test scripts or examples rely on data that lives in a separate testdata repository. This repository is noted as a git submodule in the .gitmodules configuration file. GitHub respects and records the repository link, so the testdata repository will be cloned with https://github.com/SCALE-MS/scale-ms if you used the --recursive option to git clone. Otherwise, you will have to manually initialize and update the submodule or otherwise acquire the test data.

Unit tests, system tests, and integration tests

The test suite in tests/ includes a lot of integration tests that probe interactions with full RADICAL Pilot sessions because we have had problems reproducing viable execution environments and because we use RP features and interfaces that may not be mature.

Refer to the docker/README.md in the repository for more about reproducible testing environments.

We install several pytest markers and command line options for the tests in tests/. Refer to tests/conftest.py or use pytest tests --help (look for the custom options: section).

A typical invocation of the test suites in tests/ (including Python interpreter options, pytest options, and options specific to the scalems test fixtures) frequently looks something like the following:

python -X dev -m pytest -x -rA -l --log-cli-level debug tests/ --rp-venv $VIRTUAL_ENV --rp-resource docker.login --rp-access ssh

Acceptance tests

The automated GitHub Actions test pipeline includes command line invocations of example scripts. We should continue to run short examples to ensure that the scripting interface behaves as expected.

Coverage

We use the Python Coverage package to trace test coverage. (For pytest tests, we use the pytest-cov pytest plugin.) In our GitHub Actions test pipeline, we gather coverage for both pytest suites and command line examples, and upload the results to Codecov.io for visualization and for feedback on pull requests.

Aggregating coverage

The --parallel-mode works pretty well to gather multiple data files, and codecov.io does a good job of automatically merging multiple reports received from a pipeline. We just have to make sure to use --append (or --cov-append) appropriately for the data files, and to create appropriately unique xml report files (for upload).

The default coverage behavior automatically follows threads, too. However, for processes launched by RADICAL Pilot, we need to take extra steps to run coverage and gather results.

Gathering remote coverage

When COVERAGE_RUN or SCALEMS_COVERAGE environment variables are detected, scalems.radical.runtime modifies the Master TaskDescription to include python -m coverage run --data-file=coverage_dir/.coverage --parallel-mode ..., and adds an output staging directive to retrieve task:///coverage_dir to the predictably named directory ./scalems-remote-coverage-dir. The --parallel-mode option makes sure that remotely generated master task coverage data file will be uniquely named.

Note that pytest-cov does not set the COVERAGE_RUN environment variable. When pytest --cov is detected, we use a pytest fixture to set SCALEMS_COVERAGE=TRUE in the testing process environment.

Even though the ScaleMSRaptor.request_cb() and ScaleMSRaptor.result_cb() are called in separate threads spawned by RP, coverage should be correct.

We cannot customize the command line for launching the Worker task, so for coverage of the Worker and its dispatched function calls, we need to use the Coverage API. These details are encapsulated in the scalems.radical.raptor.coverage_file() decorator.