Modifying rules to use bzlmod

January 25, 2024

Michael Boulton

Lessons learned when converting some Bazel rules to use the new bzlmod dependency management system.

A while ago I wrote some simple (ish) Bazel rules for Lua (along with LuaJIT) and Fennel, a Lisp variant that compiles to Lua. This was mainly to teach myself how to properly use toolchains in Bazel, how to manage and arrange a larger set of rules for multiple languages, and transpiling from one language to another inside Bazel without requiring the user to check-in generated files into their repo.

Fennel

A small example of Fennel if you haven't seen it:

(lambda add_runfiles_to_path [runfiles_dir]
  "Add basic packages from runfiles directory to the package path/cpath"
  (table.insert (or package.searchers package.loaders) search_for_module)
  (add_to_path (.. runfiles_dir :/?.lua))
  (add_to_path (.. runfiles_dir :/?/init.lua))
  (each [_ v (pairs (read_lua_mappings))]
    (let [root (.. runfiles_dir "/" (. v 2) "/" (. v 2))]
      (add_to_path (.. root :/share/lua/ lua_numeric_ver :/?.lua))
      (add_to_path (.. root :/share/lua/ lua_numeric_ver :/?/init.lua))
      (add_to_cpath (.. root :/lib/lua/ lua_numeric_ver :/?.so)))))

(-?> (os.getenv :RUNFILES_DIR)
     (add_runfiles_to_path))

Results in

local function add_runfiles_to_path(runfiles_dir)
  _G.assert((nil ~= runfiles_dir), "Missing argument runfiles_dir on lua/private/binary_wrapper.fnl:74")
  table.insert((package.searchers or package.loaders), search_for_module)
  add_to_path((runfiles_dir .. "/?.lua"))
  add_to_path((runfiles_dir .. "/?/init.lua"))
  for _, v in pairs(read_lua_mappings()) do
    local root = (runfiles_dir .. "/" .. v[2] .. "/" .. v[2])
    add_to_path((root .. "/share/lua/" .. lua_numeric_ver .. "/?.lua"))
    add_to_path((root .. "/share/lua/" .. lua_numeric_ver .. "/?/init.lua"))
    add_to_cpath((root .. "/lib/lua/" .. lua_numeric_ver .. "/?.so"))
  end
  return nil
end
do
  local _12_ = os.getenv("RUNFILES_DIR")
  if (nil ~= _12_) then
    add_runfiles_to_path(_12_)
  else
  end
end

Though you can generate fairly readable Lua output from Fennel by writing it in a certain way, some of the special forms end up with fairly unreadable code, so I wanted this to all be hidden from the user.

These rules also had some other features like working with Busted and LuaUnit unit tests in Fennel or Lua.

Problems

The rules basically worked but with a few issues

  • Because of the way the WORKSPACE file worked in older versions of Bazel, you needed to import different things from the repository depending on what you wanted (eg, specific dependencies for Busted tests)
  • You needed to manually register the toolchains for Lua and Fennel in your WORKSPACE. This meant all internal tooling would have to be written in pure Lua in case a user didn't want to register the Fennel toolchain
  • Some parts of the rules required external dependencies like rules_foreign_cc and some didn't, again resulting in conditional includes in the WORKSPACE file
  • The classic problem with Bazel dependencies where basically every project depends on rules_go at slightly different versions

All of these resulted in a fairly big amount of things being added to the workspace, mixing and matching different imports depending what you needed (or didn't) as well as possible conflicts between dependencies required internally by rules_lua and ones that a user might also define in their dependencies.

The plan

I decided that to fix these problems, and just as a learning experience, I'd convert it to use bzlmod only. I looked around and decided that the way rules_proto_grpc was doing it would be the best way to convert different aspects of the repo into different modules:

  • The root module with lua_binary, fennel_library, dependency management, toolchains, etc. The basics required to get any project working
  • A LuaUnit module with lua_luaunit_test and fennel_luaunit_test
  • A Busted module with lua_busted_test and fennel_busted_test

This meant that anyone who didn't want to use Luaunit or Busted could just add one line (a dependency on rules_lua) to their module dependencies, and immediately be able to use the Lua rules without any extra trouble. I would also add a basic runfi;les library so that files could be loaded at runtime which the rules initially couldn't do.

Runfiles and LUA_PATH

The first big problem I ran into was that I basically had a big horrible bash script to add things to the Lua path like this:

export LUA_PATH="?;?.lua;?/init.lua"
export LUA_PATH="$LUA_PATH;$(realpath ..)/?.lua"
export LUA_PATH="$LUA_PATH;$(realpath ..)/?/?.lua"
export LUA_PATH="$LUA_PATH;$(realpath ..)/?/init.lua"
if ls -d ../lua* 2>/dev/null ; then
    for d in $(ls -d ../lua*/lua*); do
        d=$(realpath $d)
        # FIXME get lua version
        export LUA_PATH="$LUA_PATH;$d/lib/lua/5.1/?.lua"
        export LUA_CPATH="$LUA_CPATH;$d/lib/lua/5.1/?.so"
        export LUA_PATH="$LUA_PATH;$d/share/lua/5.1/?.lua"
        export LUA_PATH="$LUA_PATH;$d/share/lua/5.1/?/init.lua"
    done
fi

Obvious problems with this is that it assumed that all dependencies were in ../dependency_name (which was a roughly correct assumption pre-bzlmod (on Linux!)) and that the version of Lua was hardcoded into the script. This also had to be in every single rule - binaries and tests, Lua and Fennel.

Luckily, Lua has a mechanism where you can say "import and run this Lua module before running the actual user code", so I wrote a small Lua script which reads a file pointed to by the magic RUNFILES_REPO_MAPPING environment variable, containing multiple lines where each line is of the form canonical name of source repo,apparent name of target repo,target repo runfiles directory (from rules_go). This isn't really well documented anywhere as far as I can tell, probably because it's only relevant if you're writing rules for an entirely new language (unfortunately I am.) This script read the repo mapping file, and added all the dependencies to the path or cpath as appropriate:

-- read_lua_mappings is a function to read the repo mapping and return only lua dependencies
local lua_dependency_mappings = read_lua_mappings()
local runfiles_dir = os.getenv("RUNFILES_DIR")
local lua_numeric_ver = string.gsub(_VERSION, "Lua ", "")

package.path = ""
package.cpath = ""

add_to_path("?.lua")
add_to_path("?/init.lua")

for k, v in pairs(lua_dependency_mappings) do
    add_to_path(runfiles_dir .. "/?.lua")
    add_to_path(runfiles_dir .. "/?/init.lua")

    local root = runfiles_dir .. "/" .. v[2] .. "/" .. v[2]

    -- luarocks dependencies go in these folders
    add_to_path(root .. "/share/lua/" .. lua_numeric_ver .. "/?.lua")
    add_to_path(root .. "/share/lua/" .. lua_numeric_ver .. "/?/init.lua")
    add_to_cpath(root .. "/lib/lua/" .. lua_numeric_ver .. "/?.so")
end

This was also used to inject basic runfiles functionality (missing the ability to read from other namespaces currently):

if os.getenv("RUNFILES_DIR") then
    local function rlocation(workspace_relative)
        local runfiles_dir = os.getenv("RUNFILES_DIR")
        local resolved = runfiles_dir .. "/" .. "_main" .. "/" .. workspace_relative

        return resolved
    end

    package.preload["runfiles"] = function()
        return {
            rlocation = rlocation
        }
    end
end

To avoid this having to be inserted into every rule like the earlier Bash script, and to fix other issues with getting the correct Lua path at runtime, I imported the aspect_bazel_lib Bash runfiles script and the bazel_tools runfiles library so that all lua_binary, lua_test etc. Could just import this automatically. Trimmed down:

load("@aspect_bazel_lib//lib:paths.bzl", "to_rlocation_path",  "BASH_RLOCATION_FUNCTION")

def _lua_binary_impl(ctx):
    out_executable = ctx.actions.declare_file(ctx.attr.name + "_exec")

    lua_toolchain = ctx.toolchains["//lua:toolchain_type"].lua_info
    lua_executable = lua_toolchain.target_tool[DefaultInfo].files_to_run.executable

    ctx.actions.write(
        out_executable,
        BASH_RLOCATION_FUNCTION + """
set -e
set +u
export LUA_PATH="$LUA_PATH;$(rlocation $(dirname $(rlocation {wrapper})))/?.lua"
$(rlocation {lua_rloc}) -l binary_wrapper $(rlocation {tool}) $@
        """.format(
            lua_rloc = to_rlocation_path(ctx, lua_executable),
            tool = to_rlocation_path(ctx, ctx.file.tool),
            wrapper = to_rlocation_path(ctx, ctx.file._wrapper),
        ),
        is_executable = True,
    )

    runfiles = ctx.runfiles(files = ctx.files.deps + ctx.files._wrapper + ctx.files._runfiles_lib + ...)

    ...

    return [
        DefaultInfo(...),
    ]

lua_binary = rule(
    implementation = _lua_binary_impl,
    attrs = {
        "_runfiles_lib": attr.label(
            allow_files = True,
            default = "@bazel_tools//tools/bash/runfiles",
        ),
        "_wrapper": attr.label(
            allow_single_file = True,
            default = "@rules_lua//lua/private:binary_wrapper.lua",
        ),
        ...
    },
    doc = "Lua binary target. Will run the given tool with the registered lua toolchain.",
    toolchains = [
        "//lua:toolchain_type",
    ],
    executable = True,
)

Once all the binaries and tests were converted to use this format, the code was a lot simpler, easier to understand, and worked better.

Toolchains

With bzlmod, you can register a toolchain in the MODULE.bazel of the module which is used as the toolchain of the required type for things in the module to run. This allows the author of a module to specify a 'sensible default' version for the toolchain instead of relying on using one of the magic built-in Bazel toolchain types (eg, protoc, c++ toolchains) or having to register one themself. The root module for rules_lua contains:

lua_toolchains = use_extension("@rules_lua//lua:repositories.bzl", "lua_toolchains")
lua_toolchains.luajit(
    name = "luajit",
    version = "v2.1",
)
use_repo(lua_toolchains, "luajit_toolchains")

register_toolchains("@luajit_toolchains//:x86_64-unknown-linux-gnu_toolchain")

Which registers LuaJIT as the default Lua toolchain.

The way this works is that it downloads the Lua (or LuaJIT) source repository, adds some build files to define how to compile it and expose a toolchain repository, and let the user register that. Pre-bzlmod, you had to hope that two dependencies didn't happen to register the same repository name for two different things, and the rules_lua code assumed that all Lua source repositories were called lua_git, and would raise an error if it wasn't as expected or if there was a conflict.

In the bzlmod version of rules_lua, this is implemented using a module extension which will download the source for all requested versions of Lua/LuaJIT in one place - this allows us to ensure that only one repository is ever created for each version of Lua/LuaJIT:

def _lua_toolchains_extension(mctx):
    def _verify_toolchain_name(mod, expected, name):
        if name != expected and not mod.is_root:
            fail("""\
            Only the root module may override the default name for the {} toolchains.
            This prevents conflicting registrations in the global namespace of external repos.
            """.format(expected))

    lua_versions = {}
    for mod in mctx.modules:
        for lua in mod.tags.lua:
            _verify_toolchain_name(mod, "lua", lua.name)
            lua_versions[lua.version] = None

    for lua_version in lua_versions:
        lua_repository_name = "lua_src_{version}".format(version = lua_version)

        http_archive(
            name = lua_repository_name,
            patch_args = ["-p", "1"],
            **LUA_VERSIONS[lua_version]
        )

        _lua_register_toolchains(lua.name, lua.version, lua_repository_name)

    # And the same for LuaJIT

lua_toolchains = module_extension(
    implementation = _lua_toolchains_extension,
    tag_classes = {
        "lua": _lua_tag,
        "luajit": _luajit_tag,
    },
)

This goes through every module that wants to register a new lua toolchain, and registers exactly one source repository and toolchain for each version. This means each module, even transitive dependencies, never have a conflict with toolchain repositories names or source repository names.

Luarocks dependencies

Dependencies are also registered using a module extension, similar to above:

_luarocks_tag_attrs = {
    "name": attr.string(
        doc = "name to use, if not lua_<dependency>",
    ),
    ...
}

_luarocks_tag = tag_class(
    doc = "Fetch a dependency from luarocks",
    attrs = _luarocks_tag_attrs,
)

def _lua_dependency_impl(mctx):
    deps = []

    # TODO: There should be something which parses rockspecs use luarocks, or recursively, instead of defining all of these
    #  https://bazel.build/external/migration#integrate-package-manager
    for mod in mctx.modules:
        for luarocks in mod.tags.luarocks:
            p = {k: getattr(luarocks, k) for k in _luarocks_tag_attrs}
            luarocks_dependency(**p)

    # And the same for github dependencies

lua_dependency = module_extension(
    implementation = _lua_dependency_impl,
    tag_classes = {
        "luarocks": _luarocks_tag,
        "github": _github_tag,
    },
)

The rules_lua_luaunit module uses this to fetch a dependency on LuaUnit automatically without the user having to import this themselves (like they would with the old WORKSPACE model):

lua_deps = use_extension("@rules_lua//lua:defs.bzl", "lua_dependency")
lua_deps.github(
    dependency = "luaunit",
    rockspec_path = "luaunit-3.4-1.rockspec",
    sha256 = "b8aea5826f09749d149efa8ef1b13f81e6a9fc6abfbe4c1cbf87a558a6d4e8d0",
    tag = "LUAUNIT_V3_4",
    user = "bluebird75",
)
use_repo(lua_deps, "lua_luaunit")

Example - LuaUnit wrapper

LuaUnit does some magic with the way it runs and runs all functions beginning with test that it can find. One downside of this is that Fennel functions, by default, are not exported globally so LuaUnit can't find them. To get around this, I wrote a wrapper that exports all functions beginning with test globally then runs LuaUnit. Because we can be sure that the Fennel toolchain is now always registered (because it is registered in the rules_lua module file), this can be written in Fennel:

(let [test_files (os.getenv :TEST_FILES)]
  (if (= nil test_files)
      (error "no test files specified"))
  (each [str (string.gmatch test_files "([^,]+)")]
    (let [trimmed (str:gsub "[.]lua" "")
          replaced (trimmed:gsub "/" ".")
          mod (require replaced)]
      (each [fname f (pairs mod)]
        (if (= (type f) "function")
          (if (fname:match "test.+")
            (if (= nil (. _G fname))
              (tset _G fname f))))))))

(let [luaunit (require :luaunit)]
  (os.exit (luaunit.LuaUnit.run)))

This is wrapped in a fennel_binary, which is then used by lua_luaunit_test as a 'test runner'. The Lua binary wrapper described above is loaded before running this, exposing all dependencies and runfiles to the test:

load("@rules_lua//fennel:defs.bzl", "fennel_binary")

fennel_binary(
    name = "luaunit_runner_fennel",
    tool = "luaunit_runner.fnl",
    visibility = ["//:__subpackages__"],
    deps = ["@lua_luaunit"],
)

TODO

Things that still need doing before it's really "complete":

  • I've only really tested this in a few small examples so more thorough examples and testing is required.
  • Read luarocks .spec files to manage dependencies instead of having to specify all dependencies in the module file, similar to how rules_python can read dependencies from a requirements.txt file.
  • Add another module for LÖVE to allow easier packaging of games made with that framework. It already has its own concept of 'files that are packaged with the binary' (aka, runfiles) so this is a pretty good match.
  • Make sure this works on Windows and Mac OS. Currently it depends on some Unix tools, this could be achieved with https://gitlab.arm.com/bazel/rules_coreutils ?
Recent Posts