Lmod Regression Testing Guide

Lmod utilizes a comprehensive regression testing suite to ensure its stability, correctness across various shells, and to prevent regressions when new features or fixes are introduced. This guide provides an overview of the testing framework and instructions on how to run and manage these tests.

The testing framework resides in the rt/ directory (short for “regression tests”) within the Lmod source code.

Overview of the Testing Framework

The rt/ directory contains numerous subdirectories. Each subdirectory typically represents a specific test case or a group of related tests focusing on a particular Lmod feature or command (e.g., rt/load/, rt/avail/, rt/purge/).

Inside each test directory, you’ll commonly find the following types of files:

  • ``*.tdesc``: This is the test description file. It’s a crucial component that defines the test’s purpose, the sequence of Lmod commands to execute, the shells to test against, and other test parameters.

  • Modulefiles (often in subdirectories like ``mf/``, ``mf2/``, etc.): Many tests require their own simple modulefiles to exercise specific Lmod behaviors. These are usually located within the test’s directory or its subdirectories.

  • Reference Output Files:

    • ``err.txt``: Contains the expected standard error output for the test.

    • ``out.txt``: Contains the expected standard output for the test.

  • Helper Scripts (e.g., ``*.sh``, ``*.lua``): Some tests might use additional scripts to set up specific conditions or perform complex validation steps.

The primary tool for running these tests is a shell command typically named t, which is made available by sourcing a specific testing environment script (detailed below). When a test is run, the t command executes the steps defined in the .tdesc file. It captures the actual standard output and standard error, placing them in a temporary execution directory (usually a subdirectory named t1/<test_name> within the test case directory). The framework then compares these actual outputs against the reference std.txt and err.txt files.

Prerequisites and Setup

Before you can run the Lmod regression tests, you need to set up your environment:

  1. Install Required Packages: Ensure you have Lua and several Luarocks packages installed. On a Debian-based system, you can typically do this with:

    sudo apt update
sudo apt install lua5.3 liblua5.3-dev
sudo luarocks install luaposix
sudo luarocks install luafilesystem
sudo luarocks install lua-term # For lua-term, ensure it matches Lmod's version if issues arise
sudo luarocks install busted   # Busted is a Lua unit testing framework

    Note: Adjust package names and versions based on your Lua installation and distribution.

  2. Set up Robert McLay’s Testing Environment: Lmod’s testing framework relies on a set of auxiliary tools and shell functions provided by Robert McLay (Lmod’s author).

    • Clone the Hermes repository and add it to your PATH:

      git clone https://github.com/rtmclay/Hermes.git
export PATH=/path/to/your/Hermes/bin:$PATH

    • Clone the Testing_aux_tools repository and source the shell functions:

      git clone https://github.com/rtmclay/Testing_aux_tools.git
source /path/to/your/Testing_aux_tools/testing_tools_shell_funcs.sh

Running Tests

Once your environment is set up, you can run the tests using the t command.

  1. Running All Tests: To run the entire regression test suite:

    cd /path/to/Lmod/rt  # Navigate to Lmod's rt directory
t .

    This command will iterate through all test subdirectories in rt/ and execute them. This can take a significant amount of time.

  2. Running a Single Test Case: To run a specific test (e.g., a test named my_feature_test located in rt/my_feature_test/):

    cd /path/to/Lmod/rt/my_feature_test
t .

    This will execute only the tests defined in the my_feature_test.tdesc file.

Understanding and Managing Test Output

A key aspect of Lmod’s operation is its use of standard output (stdout) and standard error (stderr):

  • Lmod’s ``stdout`` is its primary “payload”: Lmod is designed to output shell commands to stdout. These commands are then typically evaluated (e.g., via eval "$(lmod ...)") by the user’s shell to modify their current environment (setting environment variables, defining aliases, etc.).

  • Lmod’s ``stderr`` is for communication: All informational messages, warnings, and error messages generated by Lmod are directed to stderr.

This separation is critical for how tests are structured and evaluated:

When you run a test using t .:

  • The test commands from the .tdesc file are executed.

  • A temporary directory, typically t1/ (e.g., rt/my_feature_test/t1/my_feature_test/), is created within the specific test’s directory.

  • The actual standard output of the test run is saved to std.txt within this t1/... directory. This file captures the shell commands Lmod generated, representing the intended changes to the environment.

  • The actual standard error is saved to err.txt within this t1/... directory. This file captures all messages, warnings, or errors Lmod produced during the test.

  • The framework then compares these generated std.txt and err.txt files with the reference std.txt and err.txt files located in the root of the test directory (e.g., rt/my_feature_test/std.txt).

  • If the outputs match the reference files, the test passes. Otherwise, it fails, and differences will be reported.

To inspect the output of a failed test:

cd /path/to/Lmod/rt/my_feature_test
cat t1/my_feature_test/out.txt  # View actual standard output
cat t1/my_feature_test/err.txt  # View actual standard error

Compare these with out.txt and err.txt in the current directory (/path/to/Lmod/rt/my_feature_test/) to understand the discrepancies.

Editing a Test

To modify a test, you’ll typically edit its *.tdesc file:

cd /path/to/Lmod/rt/my_feature_test
vim my_feature_test.tdesc  # Or your preferred editor

Updating Test Baselines (Reference Files)

If a test fails because Lmod’s behavior has legitimately changed (e.g., due to a bug fix or a new feature that alters output), and you’ve verified that the new output in t1/.../std.txt and t1/.../err.txt is correct, you need to update the reference baseline files.

To do this, from within the specific test directory (e.g., rt/my_feature_test/):

cp t1/my_feature_test/*.txt .

This copies the newly generated (and now correct) std.txt and err.txt from the temporary t1/my_feature_test/ directory to become the new reference files in the current test directory.

Debugging Lmod

When troubleshooting or developing new features, you may need to inspect Lmod’s internal state or trace its execution flow. Lmod includes a built-in debugging tool that allows you to print messages from the source code, which can be made visible during testing.

Lmod’s debugging function is dbg.print{}. You can add statements like this to the Lua source code (files in src/) to output variables or trace messages:

-- In some file like src/cmdfuncs.lua
dbg.print{"My debug message: some_variable = ", some_variable, "\n"}

By default, this debug output is suppressed. To see the output when running a regression test, you need to enable debugging for the lmod command. This is done by adding the -D flag to the command in your test’s .tdesc file.

For example, if your .tdesc file has the following command:

runLmod avail

You would change it to:

runLmod -D avail

When you re-run the test with t ., the output from any dbg.print{} statements will now appear in the standard error stream, which is captured in t1/.../err.txt.

For quick, temporary debugging, you can also directly edit the generated test script at rt/<test_case>/t1/<test_name>/t1.script to add the -D flag. This avoids modifying the baseline .tdesc file.

Additionally, setting the LMOD_DEBUG environment variable to 1 will also enable debug output.

Re-testing and Advanced Scenarios

  • Retest to Pass: After updating reference files or fixing a test script, re-run the test to confirm it now passes:

    cd /path/to/Lmod/rt/my_feature_test
t .

  • Cleaning Temporary Files: The user note run rm tm/*, t . (if in rt/[test]) suggests that there might be other temporary files (perhaps in a tm/ directory) that sometimes need cleaning before a re-run. The t command usually handles its own cleanup within the t1/ structure.

  • Re-running the Generated Script: If you are deep inside a test’s temporary execution directory (e.g., rt/my_feature_test/t1/some_specific_shell_variant/), the note run t1.script suggests you might be able to directly execute the shell script (t1.script) that the t command generated for that particular test variant. This can be useful for debugging a specific failing case.

  • Re-running Failed Tests: If you’ve run the full test suite and some tests failed, you can re-run only the tests that previously failed. From the main rt/ directory:

    cd /path/to/Lmod/rt
t -r wrong

This is a significant time-saver compared to re-running the entire suite.

This guide should provide a solid foundation for working with Lmod’s regression testing suite. Always refer to the latest scripts and any specific READMEs within the rt/ or Testing_aux_tools directories for the most up-to-date information.