Containers

Benchbuild allows the definition of container images to define the base system all experiment runs run in for a given project.

Usage

If you want to run an experiment inside the project’s container, simply replace the usual run subcommand with the container run subcommand. Project and experiment selection are done in the same way.

Example:

benchbuild container run -E raw linpack

This will run the following stages:

1. Build all necessary base images. All images are initiated from a base image. Benchbuild knows how to construct a few base images. These will be prepared with all dependencies required to run benchbuild inside the container.

2. Build all project images. Each project has to define its’ own image.

3. Build the experiment images. Each experiment can add anything it needs to the project images, if required. Use this to bring tools into the image that do not require any knowledge about the environment to run properly. For anything else, consider using a custom base image.

Replace Images

Benchbuild will reuse any existing images it can find in your image registry. The only relevant information is the image tag, e.g., benchbuild:alpine. If you want to avoid reuse and force to rebuild images unconditionally, you can use the --replace flag when running the containers subcommand.

Example:

benchbuild container run --replace -E raw linpack

This will ignore any required image for the given experiments and projects.

Configuration

You can configure the container environment using the following config variables.

• BB_CONTAINER_EXPORT: Path where benchbuild stores exported container images. By default we store it in ./containers/export. Will be created automatically, if needed.

• BB_CONTAINER_IMPORT: Path where to input images from into the registry. By default we load from ./containers/export.

• BB_CONTAINER_FROM_SOURCE: Determine, if we should use benchbuild from the current source checkout, or from pip.

• BB_CONTAINER_ROOT: Where we store our image layers. This is the image registry. Cannot be stored on filesystems that do not support subuid/-gid mapping, e.g. NFS. The default location is ./containers/lib.

• BB_CONTAINER_RUNROOT: Where we store temporary image layers of running containers. See BB_CONTAINER_ROOT for restrictions. The default location is: ./containers/run.

• BB_CONTAINER_RUNTIME: Podman can use any standard OCI-container runtime to launch containers. We use crun by default. Depending on your system, this one has already been installed with podman. The default runtime is: /usr/bin/crun

• BB_CONTAINER_MOUNTS: A list of mountpoint definitions that should be added to all containers. With this you can add arbitrary tools into all containers. Default: []

Definition

A project that wants to use a container image needs to define it in the CONTAINER attribute it using our declarative API provided by benchbuild.environments.domain.declarative.

from benchbuild.environments.domain.declarative import ContainerImage

class Testproject(Project):
  CONTAINER = ContainerImage().from_('benchbuild:alpine')

The available set of commands follows the structure of a Dockerfile.

Runtime requirements

For containers to work properly, you need a few systems set up beforehand.

Buildah

Image construction requires the Buildah tool. All image construction tasks are formulated as buildah command calls in the backend.

Buildah is supported up to version 1.19.8.

Podman

Container construction and execution is handed off to Podman. Podman provides rootless containers and does not requires the execution of a daemon process. However, you need to setup your user namespace properly to allow mapping of subordinate uids/gids. Otherwise, podman will not be able to map users other than the root user to filesystem permissions inside the container.

Please refer to podman’s documentation on how to setup podman properly on your system.

Podman is supported up to version 2.2.1

Module: benchbuild.container

Container construction tool.

This tool assists in the creation of customized uchroot containers. You can define strategies and apply them on a given container base-image to have a fixed way of creating a user-space environment.

class benchbuild.container.BashStrategy

Bases: ContainerStrategy

The user interface for setting up a bash inside the container.

run(context)

Execute a container strategy.

Parameters:

context – A context object with attributes used for the strategy.

class benchbuild.container.Container(executable=None)

Bases: Application

Manage uchroot containers.

VERSION = '6.8'

builddir(tmpdir)

Set the current builddir of the container.

input_file(_container)

Find the input path of a uchroot container.

main(*args)

Implement me (no need to call super)

mounts(user_mount)

Save the current mount of the container into the settings.

output_file(_container)

Find and writes the output path of a chroot container.

shell(custom_shell)

The command to run inside the container.

verbosity

Sets an attribute

class benchbuild.container.ContainerBootstrap(executable=None)

Bases: Application

Check for the needed files.

install_cmake_and_exit()

Tell the user to install cmake and aborts the current process.

main(*args)

Implement me (no need to call super)

class benchbuild.container.ContainerCreate(executable=None)

Bases: Application

Create a new container with a predefined strategy.

We offer a variety of creation policies for a new container. By default a basic ‘spawn a bash’ policy is used. This just leaves you inside a bash that is started in the extracted container. After customization you can exit the bash and pack up the result.

main(*args)

Implement me (no need to call super)

strategy(strategy)

Select strategy based on key.

Parameters:

strategy (str) – The strategy to select.

Returns:

A strategy object.

class benchbuild.container.ContainerList(executable=None)

Bases: Application

Prints a list of the known containers.

main(*args)

Implement me (no need to call super)

class benchbuild.container.ContainerRun(executable=None)

Bases: Application

Execute commannds inside a prebuilt container.

main(*args)

Implement me (no need to call super)

class benchbuild.container.ContainerStrategy

Bases: object

Interfaces for the different containers chosen by the experiment.

abstract run(context)

Execute a container strategy.

Parameters:

context – A context object with attributes used for the strategy.

class benchbuild.container.MockObj(**kwargs)

Bases: object

Context object to be used in strategies.

This object’s attributes are initialized on construction.

class benchbuild.container.SetupPolyJITGentooStrategy

Bases: ContainerStrategy

Interface of using gentoo as a container for an experiment.

run(context)

Setup a gentoo container suitable for PolyJIT.

benchbuild.container.clean_directories(builddir, in_dir=True, out_dir=True)

Remove the in and out of the container if confirmed by the user.

benchbuild.container.find_hash(container_db, key)

Find the first container in the database with the given key.

benchbuild.container.main(*args)

Main entry point for the container tool.

benchbuild.container.pack_container(in_container, out_file)

Pack a container image into a .tar.bz2 archive.

Parameters:

• in_container (str) – Path string to the container image.

• out_file (str) – Output file name.

benchbuild.container.run_in_container(command, container_dir)

Run a given command inside a container.

Mounts a directory as a container at the given mountpoint and tries to run the given command inside the new container.

benchbuild.container.set_input_container(_container, cfg)

Save the input for the container in the configurations.

benchbuild.container.setup_bash_in_container(builddir, _container, outfile, shell)

Setup a bash environment inside a container.

Creates a new chroot, which the user can use as a bash to run the wanted projects inside the mounted container, that also gets returned afterwards.

benchbuild.container.setup_container(builddir, _container)

Prepare the container and returns the path where it can be found.

benchbuild.container.setup_directories(builddir)

Create the in and out directories of the container.

Module: benchbuild.environments.domain.declarative

BenchBuild supports containerized execution of all experiments. This gives you full control about the environment your [projects](/concepts/projects/) and [experiments](/concepts/experiments/) may run in.

The following example uses the latest alpine:latest:

ContainerImage().from_('alpine:latest')
    .run('apk', 'update')
    .run('apk', 'add', 'python3')

class benchbuild.environments.domain.declarative.ContainerImage(iterable=(), /)

Bases: list

Define a container image declaratively.

Start a new image using the .from_ method and provide a base image. Each method creates a new layer in the container image.

add(sources, tgt)

Add given files from the source to the container image.

Dockerfile syntax: ADD <source> [<source>…] <target>

Parameters:

• sources (tp.Iterable[str]) – Source path to add to the target

• tgt (str) – Absolute target path.

Return type:

ContainerImage

property base: str

command(*args)

Set the default command the container runs.

Dockerfile syntax: CMD <command>

Parameters:

*args (str) – A list of command components.

Return type:

ContainerImage

context(func)

Interact with the build context of the container.

Sometimes you have to interact with the build context of a container image. For example, you need to add artifacts to the build context before you can add the to the container image. BenchBuild uses this to add the sources to the container image automatically.

Parameters:

func (tp.Callable[[], None]) – A callable that is executed in the build-context directory.

Return type:

ContainerImage

copy_(sources, tgt)

Copy given files from the source to the container image.

Dockerfile syntax: COPY <source> [<source>…] <target>

Parameters:

• sources (tp.Iterable[str]) – Source path to add to the target

• tgt (str) – Absolute target path.

Return type:

ContainerImage

entrypoint(*args)

Set the entrypoint of the container.

Dockerfile syntax: ENTRYPOINT <command>

This sets the default binary to run to the given command.

Parameters:

*args (str) – A list of command components.

Return type:

ContainerImage

env(**kwargs)

Create an environment layer in this image.

Dockerfile syntax: ENV

Parameters:

kwargs (str) – a dictionary containing name/value pairings to be set as environment variables.

Return type:

ContainerImage

from_(base_image)

Specify a new base layer for this image.

Dockerfile syntax: FROM <image>

Parameters:

base_image (str) – The base image for our new container image.

Return type:

ContainerImage

run(command, *args, **kwargs)

Run a command in the container image.

Dockerfile syntax: RUN <command>

Parameters:

• command (str) – The binary to execute in the container.

• *args (str) – Arguments that will be passed to the container.

• **kwargs (str) – Additional options that will be passed to the backend run command.

Return type:

ContainerImage

workingdir(directory)

Change the working directory in the container.

Dockerfile syntax: WORKINGDIR <absolute-path>

All layers that follow this layer will be run with their working directory set to directory.

Parameters:

directory (str) – The target directory to set our cwd to.

Return type:

ContainerImage

benchbuild.environments.domain.declarative.add_benchbuild_layers(layers)

Add benchbuild into the given container image.

This assumes all necessary depenencies are available in the image already. The installation is done, either using pip from a remote mirror, or using the source checkout of benchbuild.

A source installation requires your buildah/podman installation to be able to complete a bind-mount as the user that runs benchbuild.

Parameters:

layers (ContainerImage) – a container image we will add our install layers to.

Return type:

ContainerImage

Returns:

the modified container image.

Module: benchbuild.environments.domain.model

class benchbuild.environments.domain.model.AddLayer(sources, destination)

Bases: Layer

destination: str

sources: Tuple[str, ...]

class benchbuild.environments.domain.model.Command

Bases: Message

class benchbuild.environments.domain.model.Container(container_id, image, context, name, events=_Nothing.NOTHING)

Bases: object

container_id: str

context: str

events: List[Message]

image: Image

name: str

class benchbuild.environments.domain.model.ContextLayer(func)

Bases: Layer

func: Callable[[], None]

class benchbuild.environments.domain.model.CopyLayer(sources, destination)

Bases: Layer

destination: str

sources: Tuple[str, ...]

class benchbuild.environments.domain.model.EntryPoint(command)

Bases: Layer

command: Tuple[str, ...]

class benchbuild.environments.domain.model.Event

Bases: Message

class benchbuild.environments.domain.model.FromLayer(base)

Bases: Layer

base: str

class benchbuild.environments.domain.model.Image(name, from_, layers, events=_Nothing.NOTHING, env=_Nothing.NOTHING, mounts=_Nothing.NOTHING, layer_index=_Nothing.NOTHING)

Bases: object

append(*layers)

Return type:

None

env: Dict[str, str]

events: List[Message]

from_: FromLayer

is_complete()

Return type:

bool

is_present(layer)

Return type:

bool

layer_index: Dict[Layer, LayerState]

layers: List[Layer]

mounts: List[Mount]

name: str

prepend(layer)

Return type:

None

present(layer)

Return type:

None

update_env(**kwargs)

Return type:

None

class benchbuild.environments.domain.model.Layer

Bases: ABC

A layer represents a filesystem layer in a container image.

Layers can be ‘virtual’ in the sense that they do not lead to changes in the container image filesystem, e.g. setting up the build context.

This more or less represents commands/statements available in buildah or Dockerfiles.

Examples

buildah add -> AddLayer buildah copy -> CopyLayer buildah from -> FromLayer

class benchbuild.environments.domain.model.LayerState(value)

Bases: Enum

An enumeration.

ABSENT = 2

PRESENT = 1

class benchbuild.environments.domain.model.Message

Bases: object

class benchbuild.environments.domain.model.Mount(source, target)

Bases: object

source: str

target: str

class benchbuild.environments.domain.model.RunLayer(command, args, kwargs)

Bases: Layer

args: Tuple[str, ...]

command: str

kwargs: Tuple[Tuple[str, str], ...]

class benchbuild.environments.domain.model.SetCommand(command)

Bases: Layer

command: Tuple[str, ...]

class benchbuild.environments.domain.model.UpdateEnv(env)

Bases: Layer

env: Tuple[Tuple[str, str], ...]

class benchbuild.environments.domain.model.WorkingDirectory(directory)

Bases: Layer

directory: str

benchbuild.environments.domain.model.immutable_kwargs(kwargs)

Convert str-typed kwargs into a hashable tuple.

Return type:

Tuple[Tuple[str, str], ...]

Module: benchbuild.environments.domain.commands

class benchbuild.environments.domain.commands.CreateBenchbuildBase(name, layers)

Bases: Command

layers: ContainerImage

name: str

class benchbuild.environments.domain.commands.CreateImage(name, layers)

Bases: Command

layers: ContainerImage

name: str

class benchbuild.environments.domain.commands.DeleteImage(name)

Bases: Command

name: str

class benchbuild.environments.domain.commands.ExportImage(image, out_name)

Bases: Command

image: str

out_name: str

class benchbuild.environments.domain.commands.ImportImage(image, in_path)

Bases: Command

image: str

in_path: str

class benchbuild.environments.domain.commands.RunProjectContainer(image, name, build_dir, args=_Nothing.NOTHING)

Bases: Command

args: Sequence[str]

build_dir: str

image: str

name: str

benchbuild.environments.domain.commands.fs_compliant_name(name)

Convert a name to a valid filename.

Return type:

str

benchbuild.environments.domain.commands.oci_compliant_name(name)

Convert a name to an OCI compliant name.

For now, we just make sure it is lower-case. This is depending on the implementation of your container registry. podman/buildah require lower-case repository names for now.

Parameters:

name (str) – the name to convert

Return type:

str

Examples

>>> oci_compliant_name("foo")
'foo'
>>> oci_compliant_name("FoO")
'foo'