Buildbot build order: how to run certain builds before others?

Introduction

The mechanisms for configuring Buildbot build order is documented fairly well. However, what that documentation lacks is a centralized place where ordering is explained. My personal goal was to run smoke tests first and if those don’t pass then cancel all other build requests. That seemed straightforward, but turned out not to be so. There were many, many ways that could in theory be achieved, but most of those did not work at all, or worked only partially. This article attempts to fill the “how do I order builds in Buildbot” gap, including what can and can’t be achieved with each strategy.

Terminology

When you work with Buildbot it is important to understand the terms correctly. Otherwise you can become confused really quickly. Here are the relevant terms for the purposes of this article:

  • SourceStamp: identifies a Change in version control (e.g. Git commit ID)

  • Change: a change made to code; roughly corresponds to Git commit including the diff and metadata.

  • Scheduler: decides when to start a build. Often, but not necessarily, tracks a version control repository for changes. Responsible for creating a BuildRequest.

  • BuildRequest: request from a Scheduler to start build. A Buildrequest ties together a SourceStamp and a Builder.

  • Builder: creates new Builds from Buildrequests. Determines which BuildSteps to take and which Workers can be used. Can be though

  • BuildFactory: defines the BuildSteps to take when a Builder runs.

  • BuildStep: the steps to take during a Build, i.e. the commands to run.

  • Build: represents a single build created from a BuildRequest by a Builder. You can think of a Build as an instance of a Builder class. After the build has finished the Build goes away, whereas the Builder remains.

  • Buildset: a set of Builds that should be tracked as one unit. As Buildbot creates builds dynamically from BuildRequests a Buildset is essentially a set of Builders that must all pass or the whole set will fail.

  • Worker: an environment where builds are run. For example a physical computer, virtual machine or a container. Typically each Worker runs a different operating system or has a configuration that is otherwise different.

DependentScheduler: do not build if smoketests have failed

In a fairly big Buildbot environment each commit probably triggers a very large number of BuildRequests. You can avoid burning down a tree every time somebody pushes to Git by using the DependentScheduler. Basically you have a normal Scheduler (e.g. SingleBranchScheduler) that monitors a source control repository that is configured to use your “smoke test Builders”. Then your DependentScheduler is configured to track your normal Scheduler and has a wider set of Builders. The effect is that if any of the smoke test Builders fail none of the Builders in your DependentScheduler will not even start.

FailingBuildsetCanceller: stop Builds immediately if any builds in a BuildSet fail

You can use the Buildsets and FailingBuildsetCanceller to fail quickly if any Builds in the BuildSet fail. The downside is that if any Build fail early you’re not guaranteed to get all the information you might need to fix all the build issues. Despite this shortcoming FailingBuildsetCanceller can save a lot of CPU cycles.

It does not seem possible, however, to make any particular Builds (created by Builders) to consistently run first. Therefore BuildSets combined with FailingBuildsetCanceller can’t be used to run smoketests first and then stop of errors are found. Fortunately DependentScheduler (see above) is pretty much tailored for that particular use-case.

Builder canStartBuild callable

The canStartBuild callable is meant for checking if a Worker is capable of starting a Build. For example you can check if a Worker has enough memory available, or that it has a file that is required by the Build.

Builder nextBuild callable

The Build Priority function or “nextBuild callable” is meant to tell Buildbot which BuildRequest in its queue a Builder should handle next as described in documentation. You can’t use nextBuild to order a Builder to be instantiated before some other Builder.

The prioritizeBuilders callable

By default Buildbot starts a Build on the Builder that has the oldest pending BuildRequest as described in the documentation. This behavior can be customized with the Builder Priority function or “prioritizeBuilders callable”. On the surface prioritizeBuilders looks as if it could be used to enforce a certain order for Builders. In practice - in my tests at least - the Build order seemed pretty random even though the prioritized list of Builders looked fine.

Here’s a sample of how you might use prioritizeBuilders in master.cfg:

def getBuilderPriority(name):
    """Determine priority of this Builder based on its name"""

    run_first = [".*-smoketest"]

    pattern = "^(" + "|".join(run_first) + ")$"

    if re.search(pattern, name):
        return 0
    else:
        return 1

def prioritizeBuilders(buildmaster, builders):
    """Prioritize Builders based on their name"""

    sorted_builders = sorted(builders, key=lambda b: getBuilderPriority(b.name))

    #print(sorted_builders

    return sorted_builders

c['prioritizeBuilders'] = prioritizeBuilders

Locks

I have written a blog post about locks in Buildbot. Locks can’t be used to order Builders or Builds per se, but they can be used to prevent resource exhausting (e.g. with latent Docker workers) as well to guarantee exclusive access to shared resources. So, in a way, they can affect the order of Builds in Buildbot.

Buildbot build order: summary

If your use case is same as mine, that is, “run smoke tests first and skip other builds if those fail” then the best and only reasonable option seems to be the DependentScheduler. It allows you to set the Buildbot build order as precisely as needed. If you want to avoid resource exhaustion or protect shared resources then use locks. For other uses check prioritizeBuilders (global setting), canStartBuild (check worker state) and nextBuild (builder settings).