Thoughts on Bazel
I've been doing a bit of work on Bazel recently and thought it might be useful to write down a few things I've found good/bad about it.
It's not appropriate for every project
When I read about it online it seems like people have the wrong 'idea' about Bazel. It's not supposed to be some sort of a competitor to CMake, Meson, Gradle, Poetry, etc. If you have a project that is in pure C++, or Python, or Java, then it might just be easier to use CMake, setuptools, or Gradle respectively.
On the other hand, if you have a monorepo which has a Javascript frontend that talks to a Go backend, with something like an admin panel that uses Django, which shared a database backend, you can quickly get into a mess with converting data to and from different representations, different components getting out of sync, updating one part of your repo accidentally breaks another part, etc. You may also start accidentally introducing hard linkages between different bits of your repo which can be very difficult to unpick later on.
It's going to be a lot easier if you use protobuf definitions for all your datatypes and use Bazel to manage generating and building all these things at the same time - this makes sure that for a specific commit, everything built will definitely be in sync. You can also use visibility restrictions to make sure that you don't introduce any unwanted dependencies between different parts of the project by specifying that nothing from a specific subfolder can be imported from anywhere else in the project.
Build rules written in starlark
In comparison to something like please or pants [1], all the build rules for Bazel have to be written in starlark. This might seem like a bit of an artifical restriction (and also being a bit annoying at the same time) but it actually is one of the best features of Bazel in my opinion.
If you have a rule defined in Python or something else, there is not only no guarantee that it's going to be hermetic, but it can also make it more difficult for people to modify/contribute to existing rules.
I know that if I go to a Bazel rules_x repo then everybody is working within the same constraints and using the same tools (even if it can be a bit daunting at first). If I want to find out how rules_docker implements a rule, I know they have to stay within the restrictions of the language and build system.
Compare this to something like pants where rules are written in Python. If we take a look at the goal to run a Python test You need to know not only the details of the engine (which is written in another programming language), but all the implementation details of how those goals were written.
Just by looking at a few examples, I was very quickly able to get a Bazel test rule running that would build a Go binary and use it to run a test against some kustomize files to make sure they could be parsed correctly.
Bazel can act as a missing package manager
One of the annoying things I've run into is trying to link against a C library when building a Go/Python binary. You need to either do everything in Docker so you know the dependencies are all correct, or make sure that everyone installs specific versions of libraries on their machines just so they can build and test things. This can become a real nightmare if you then need to install a specific version into, for example, an alpine image, where they will only keep 1 version of a dependency in the package manager at a time.
When using Bazel, you can use rules_foreign_cc to download and build C libraries in a platform-agnostic way, so you know that all anyone has to do is run bazel test //... and it will use the correct versions of everything, no matter what OS/distro someone is using. Because everything is statically linked and Bazel handles cross compilation as well, you know that somebody on OSX can build a Linux docker image.
Note that there are some caveats with this: because Bazel does still use system compilers, it can be beneficial to build all of your docker images inside a 'standard' docker image. This might seem like a bit of a hassle, but you can just reuse this docker image across every developer's machine (no matter the OS/distro they're using) and CI to ensure that you always get 100% identical reproducible builds anywhere.
Language support varies
Bazel and Go have a lot of 1:1 comparisons that make the way they 'work' very similar (and I wouldn't be surprised if the design of Blaze/Bazel informed some of the way Go is built), for example the way that each directory is considered a separate package, compared to something like rules_python or rules_java where you can pretty much decide how you want your packages to be laid out - either one package for a whole set of directories and sub-directories, or one package per directory. This is mostly a decision you need to make based on how much you want to have to maintain, but there is a semi-official tool that just autogenerates BUILD files for Go projects which saves a whole load of time.
Some of these are a lot more mature than others and work better, for example the Python rules can still 'leak' packages from your system install/active virtualenv into the build which can be very confusing if you deactivate a virtualenv and suddenly your builds stop working.
Building docker images is made a million times easier
Using Bazel made me realise how much I hate using Dockerfiles.
Because dockerfiles 'execute' from top to bottom, if you have one specific dockerfile that you end up building a lot (eg, for one of your main services) you either just accept waiting for it to build or end up spending a lot of time just trying to get it to build as quickly as possible. The 'optimal' way to structure a dockerfile ends up being a load of separate stages and then building it with buildkit.
Say you have a Go program that needs to use cgo for something (these dockerfiles are pseudocode):
# BAD
FROM golang:1.16
RUN apt-get update && apt-get install build-essential --yes --no-install-recommends && apt-get clean
RUN mkdir /app
COPY go.mod go.sum /app/
COPY src/ /app/src
WORKDIR /app/service1/cmd
RUN go build -o /service1 .
CMD ["/service1", "..."]
This one is obviously bad because you include all dependencies even though you don't need them in the image:
# BETTER
FROM golang:1.16 AS build
RUN apt-get update && apt-get install build-essential --yes --no-install-recommends && apt-get clean
RUN mkdir /app
COPY go.mod go.sum /app/
COPY src/ /app/src
WORKDIR /app/service1/cmd
RUN go build -o /service1 .
FROM golang:1.16
COPY --from=build /service1 /service1
CMD ["/service1", "..."]
The above may seem good, but if you take into account that you might have multiple services in this repository, a good strategy is to make sure that all Go dependencies are downloaded before trying to build. If you then have 5 different services which are all built in a similar fashion, you can reuse the first bit of the dockerfile:
# BEST
FROM golang:1.16 AS build
RUN apt-get update && apt-get install build-essential --yes --no-install-recommends && apt-get clean
RUN mkdir /app
COPY go.mod go.sum /app/
COPY src/ /app/src
WORKDIR /app
RUN go mod download
### End of caching step
WORKDIR /app/service1/cmd
RUN go build -o /service1 .
FROM golang:1.16
COPY --from=build /service1 /service1
CMD ["/service1", "..."]
You can copy up until the 'end of caching step' into all your other dockerfiles and when you want to build all of them at once, it speeds it up a bit...as long as they're all kept in sync. To get around this you could define one big dockerfile that builds all your services:
...
### End of caching step
FROM golang:1.16 AS build-service1
WORKDIR /app/service1/cmd
RUN go build -o /service1 .
FROM golang:1.16 as app-service1
COPY --from=build /service1 /service1
CMD ["/service1", "..."]
FROM golang:1.16 AS build-service2
WORKDIR /app/service2/cmd
RUN go build -o /service2 .
FROM golang:1.16 as app-service2
COPY --from=build /service2 /service2
CMD ["/service1", "..."]
You can then build each one with, for example, docker build -f dockerfile . --target app-service2.
This is of course an unrealistic suggestion, but if you end up repeatedly building 10+ dockerfiles for a change in your repo, something like this starts to become appealing, especially if you have extra things you need to do like download or copy in extra dependencies/static files.
With Bazel, building a Go image is as simple as just adding this into a build file (remembering that the go_library rule will be automatically generated, and you almost never have to think about it):
go_library(
name = "service1",
srcs = [...],
)
load("@io_bazel_rules_docker//go:image.bzl", "go_image")
go_image(
name = "go_image",
embed = [":service1],
)
You can then put this same bit in the service2 folder to add a new docker image there as well. All dependencies and build artifacts are cached between building these 2 completely separate docker images without you having to mess around with the order of docker files.
This also does not require docker to be installed to build and push docker images on the machine you're building on! If you build things in CI, you will probably understand how much easier this makes setting it up.
Standardising tool versions
Another thing Bazel really helps with is making sure that everyone is using the same version of command line tools. Say you have a genrule which uses kubectl to parse some kustomize files and package the output into a tar file, you want to know that it's always going to use the same version (and that every developer is also using this same version). Using a http_file repository rule, you can specify exactly which version of kubectl you want to use everywhere in the repository. Developers can also access it by doing something like bazel run //:kubectl -- get pods --namespace my-namespace (assuming you've set up an alias for it).
You can then tie this into the dependencies for your libraries - if you have a variable in a top level 'versions.bzl' file defined like
KUBE_VERSION=1.16.9
You can use this variable when defining library dependencies, for example you want to make sure that you're using the same version of the Kubernetes client libraries you can add this as a go_repository download:
# pseudocode
load("//:versions.bzl", "KUBE_VERSION")
go_repository(
name = "com_github_kubernetes_client_go",
tag = KUBE_VERSION,
importpath = "github.com/kubernetes/client-go",
)
As long as you keep this up to date with the version of Kubernetes you're using, this will make sure all your client libraries and command line tools are up to date.
You could also use this to pass in as a variable into some terraform that deploys your cluster (bazel run //:terraform -- apply ...) so that it will redeploy your cluster when you update to the next version.
This can be thought of as a sort of npx for but any command line tool, from anywhere, in any language.
It's very easy to set up a shared cache
One of the benefits of Bazel is that it can be configured to use shared caching to make sure that not only do you not need to keep rebuilding everything, but if you build something then you can upload the result so other people don't have to build it again either. This can also be used to great effect in CI.
If you're using GCP, this is as simple as just setting up a storage bucket (which only your team/CI has access to) and adding this to your .bazelrc file
--remote_cache=https://storage.googleapis.com/bucket-name
--google_default_credentials
I'm not including Buck here because there is basically no story for adding custom rules to buck: https://buck.build/extending/rules.html ↩