The spider tool¶
Lmod provides a tool called spider to build the spider cache and other files that can help a site manage their modulefiles. The shell script update_lmod_system_cache_files described in System Spider Cache uses the spider command to build the spiderT.lua file which is the filename of the spider cache. The command spider is different from the module spider command. The first one will be mainly used by system administrators where as the second command is typically for users.
The spiderT.lua file contains information about all the modulefiles on your system that can be found from $MODULEPATH and is built by spider. The way this file is built is be evaluating each module and remembering the help message, the whatis calls and the location of the modulefile. It also remembers any properties set by the modulefile. It doesn’t remember the module commands inside the modulefile.
When evaluating the modulefiles, Lmod looks for changes in $MODULEPATH. Each new directory added to MODULEPATH is also searched for modulefiles. This is why this command is called “spider”.
There are some important points about Lmod walking the module tree. Each modulefile is evaluated separately. This means that any changes in MODULEPATH need to be static. That is new path must be defined in the modulefile. If the change in MODULEPATH is dynamic or depends on some environment variable that is dynamic, it is very likely that Lmod’s module spider won’t know about these dynamic changes to MODULEPATH.
The spiderT.lua file is used by Lmod to know any properties set in a modulefile. This information is used by module avail, module spider and module keyword. By default Lmod doesn’t use the spider cache (aka spiderT.lua) to load modulefiles, however a site can configure Lmod to use cache based loads. This just means that a site must keep the spider cache up-to-date.
Lmod looks for special whatis calls to know what the description for a module is. See Using the module spider command for details.
Dynamic Spider Cache Support¶
Lmod 8.7.4+ supports for spidering user modulefiles that compiler/MPI/cuda modules could support. Suppose your site uses the software hierarchy and your site wants to allow users to be able to use module spider to find their modules as well. To do this something like this to your compiler/MPI/cuda modules. Suppose in your sites gcc/10.3.0 module you have:
prepend_path("MODULEPATH", "...") -- System compiler dependent modules
local home_root = pathJoin(os.getenv("HOME"),"myModules")
local userDir = pathJoin(home_root,"Compiler/gcc10")
if (isDir(userDir)) then
prepend_path("MODULEPATH",userDir)
end
Where users know to define their personal hierarchical for gcc/10.* packages in say $HOME/myModules/Compiler/gcc10/.
If your site uses a spider cache (and you rebuild this cache with Lmod 8.8+ then Lmod will reread modulefiles that modify $MODULEPATH. This rereading can be turned with configure time options.
When building the spider cache, Lmod will not find modulefiles that conditionally add to $MODULEPATH. Suppose your standard module (say StdEnv.lua) wants to allow users to have “Core” modules then you might have:
local home_root = pathJoin(os.getenv("HOME"),"myModules")
local userDir = pathJoin(hroot,"Core")
if (isDir(userDir)) then
prepend_path("MODULEPATH",userDir)
end
haveDynamicMPATH()
The point is that since at the time of building the system spider cache, the user’s home directory won’t be known. In this case, your StdEnv.lua file will need the haveDynamicMPATH() function. Thus telling Lmod that the “StdEnv.lua” module will be reread when performing a “module spider” command and allow any user’s Core modules to be found when spidering.
Software page generation¶
Since the package modulefiles are the gold standard of the packages your sites offers, it should be that information which should be used to generate the list of software packages. Lmod provides two kind of output depending on your sites needs. Suppose you have the simple module tree:
foo
├── .2.0.lua
├── 1.0.lua
├── 1.1.lua
└── default -> 1.0.lua
Then Lmod can produce the following information in json format:
$ MODULEPATH=.
$ spider -o jsonSoftwarePage $MODULEPATH | python -mjson.tool
[
{
"defaultVersionName": "1.0",
"description": "foo description",
"package": "foo",
"versions": [
{
"canonicalVersionString": "000000001.*zfinal",
"full": "foo/.2.0",
"help": "foo v.2.0",
"hidden": true,
"markedDefault": false,
"path": "foo/.2.0.lua",
"versionName": "1.0",
"wV": "000000000.000000002.*zfinal"
},
{
"canonicalVersionString": "000000001.000000001.*zfinal",
"full": "foo/1.1",
"help": "foo v1.1",
"markedDefault": false,
"path": "foo/1.1.lua",
"versionName": "1.1",
"wV": "000000001.000000001.*zfinal"
},
{
"canonicalVersionString": "000000001.*zfinal",
"full": "foo/1.0",
"help": "foo v1.0",
"markedDefault": true,
"path": "foo/1.0.lua",
"versionName": "1.0",
"wV": "^00000001.*zfinal"
}
]
}
]
The versions array block is sorted by the “wV” fields. This a weighted version of the canonicalVersionString, where the only difference is that the first character in the string is modified to know that it is marked default. Also if a module is hidden then the “hidden” field will be set to true.
The last entry in the versions array is used to set the description.
Another json output may be of interest. There is more information but it will be up to the site build the summarization that jsonSoftwarePage provides:
$ MODULEPATH=.
$ spider -o spider-json $MODULEPATH | python -mjson.tool
{
"foo": {
"foo/.2.0.lua": {
"Description": "foo description",
"Version": "1.0",
"fullName": "foo/.2.0",
"help": "foo v.2.0",
"hidden": true,
"pV": "000000000.000000002.*zfinal",
"wV": "000000000.000000002.*zfinal",
"whatis": [
"Description: foo description",
"Version: .2.0",
"Categories: foo"
]
},
"foo/1.0.lua": {
"Description": "foo description",
"Version": "1.0",
"fullName": "foo/1.0",
"help": "foo v1.0",
"hidden": false,
"pV": "000000001.*zfinal",
"wV": "^00000001.*zfinal",
"whatis": [
"Description: foo description",
"Version: 1.0",
"Categories: foo"
]
},
"foo/1.1.lua": {
"Description": "foo description",
"Version": "1.1",
"fullName": "foo/1.1",
"help": "foo v1.1",
"hidden": false,
"pV": "000000001.000000001.*zfinal",
"wV": "000000001.000000001.*zfinal",
"whatis": [
"Description: foo description",
"Version: 1.1",
"Categories: foo"
]
}
}
}
