Testing
Python test scripts in the repository tests/
directory are run with
pytest
.
Dependencies
RADICAL Pilot installation
RADICAL Pilot has some tricky interactions with Python environment virtualization and manipulation that lead to some caveats.
Do not try to use “editable” installs.
Explicitly activate the virtual environment before running a script that uses RP.
radical.pilot
will probably not find the data files or entry point scripts that
it relies on unless you first “source” the activate
script for a venv containing
a regular radical.pilot
installation.
Perform testing with the virtual environment active.
Do not try to
exercise the installation or virtual environment through other techniques, such
as invoking a Python venv by explicitly calling the sym-linked .../bin/python
.
pytest
We use a few pytest plugins. To install them first, it should be sufficient to
use the requirements-testing.txt
file in the root of the repository.
pip install -r requirements-testing.txt
Test data
Some test scripts or examples rely on data that lives in a separate
testdata repository.
This repository is noted as a
git submodule
in the .gitmodules
configuration file. GitHub respects and records
the repository link, so the testdata repository will be cloned with
https://github.com/SCALE-MS/scale-ms if you used the --recursive
option
to git clone. Otherwise, you will have to manually initialize
and update the submodule or otherwise acquire the test data.
Unit tests, system tests, and integration tests
The test suite in tests/
includes a lot of integration tests that probe
interactions with full RADICAL Pilot sessions because we have had problems
reproducing viable execution environments and because we use RP features and
interfaces that may not be mature.
Refer to the docker/README.md
in the repository for more about
reproducible testing environments.
We install several pytest markers and command line options for the
tests in tests/
. Refer to tests/conftest.py
or use
pytest tests --help (look for the custom options:
section).
A typical invocation of the test suites in tests/
(including Python interpreter options, pytest options, and options specific to the scalems test fixtures)
frequently looks something like the following:
python -X dev -m pytest -x -rA -l --log-cli-level debug tests/ --rp-venv $VIRTUAL_ENV --rp-resource docker.login --rp-access ssh
Acceptance tests
The automated GitHub Actions test pipeline includes command line invocations of example scripts. We should continue to run short examples to ensure that the scripting interface behaves as expected.
Coverage
We use the Python Coverage package to trace test coverage. (For pytest tests, we use the pytest-cov pytest plugin.) In our GitHub Actions test pipeline, we gather coverage for both pytest suites and command line examples, and upload the results to Codecov.io for visualization and for feedback on pull requests.
Aggregating coverage
The --parallel-mode
works pretty well to gather multiple data files, and
codecov.io does a good job of automatically merging multiple reports received
from a pipeline. We just have to make sure to use --append
(or --cov-append
) appropriately for the data files, and to create appropriately
unique xml report files (for upload).
The default coverage
behavior automatically follows threads, too.
However, for processes launched by RADICAL Pilot, we need to take extra steps
to run coverage and gather results.
Gathering remote coverage
When COVERAGE_RUN
or SCALEMS_COVERAGE
environment variables are detected,
scalems.radical.runtime
modifies the Master TaskDescription to include
python -m coverage run --data-file=coverage_dir/.coverage --parallel-mode ...
,
and adds an output staging directive to retrieve task:///coverage_dir
to the predictably named directory ./scalems-remote-coverage-dir
.
The --parallel-mode
option makes sure that remotely generated master task
coverage data file will be uniquely named.
Note that pytest-cov
does not set the COVERAGE_RUN
environment
variable. When pytest --cov is detected, we use a pytest fixture to
set SCALEMS_COVERAGE=TRUE
in the testing process environment.
Even though the ScaleMSRaptor.request_cb()
and ScaleMSRaptor.result_cb()
are
called in separate threads spawned by RP, coverage should be correct.
We cannot customize the command line for launching the Worker task, so for
coverage of the Worker and its dispatched function calls, we need to use the
Coverage API.
These details are encapsulated in the
scalems.radical.raptor.coverage_file()
decorator.