Source

Base

Provide a base interface for downloadable sources.

class benchbuild.source.base.BaseSource(*args, **kwargs)

Bases: Expandable, Versioned, ContextAwareSource, Protocol

Composition of source protocols.

class benchbuild.source.base.ContextAwareSource(*args, **kwargs)

Bases: Protocol

is_context_free()

Return, if this source needs context to evaluate it’s own list of available versions.

Return type:

bool

versions_with_context(ctx)

Augment the given revision with new variants associated with this source.

Parameters:

ctx (Revision) – the project revision, containing information about every context-free variant.

Return type:

Sequence[Variant]

Returns:

a sequence of project revisions.

class benchbuild.source.base.ContextEnumeratorFn(*args, **kwargs)

Bases: Protocol

class benchbuild.source.base.ContextFreeMixin

Bases: object

Make a context-free source context-aware.

This will setup default implementations that avoids interaction with any context.

is_context_free()

Return type:

bool

versions_with_context(ctx)

Return type:

Sequence[Variant]

class benchbuild.source.base.EnumeratorFn(*args, **kwargs)

Bases: Protocol

class benchbuild.source.base.Expandable(*args, **kwargs)

Bases: Protocol

property is_expandable: bool

Returns true, if this source should take part in version expansion.

Some sources may only be treated as virtual and would not take part in the version expansion of an associated project.

versions()

List all available versions of this source.

Returns:

The list of all available versions.

Return type:

List[str]

class benchbuild.source.base.Fetchable(*args, **kwargs)

Bases: Protocol

fetch()

Fetch the necessary source files into benchbuild’s cache.

Return type:

LocalPath

property key: str

Return the source’s key property.

This provides you with a key component that identifes a single source. It should (no guarantee) be unique among all sources for this project.

While this make no further assumption, but a good candidate is a file-system name/path.

property local: str

The source location (path-like) after fetching it from its remote.

property remote: str | Dict[str, str]

The source location in the remote location.

class benchbuild.source.base.FetchableSource(local, remote)

Bases: ContextFreeMixin

Base class for fetchable sources.

Subclasses have to provide the following protocols:

• Expandable

• Fetchable

• Versioned

abstract property default: Variant

The default version for this source.

explore()

Explore revisions of this source.

This provides access to all revisions this source can offer. BenchBuild own filters will not block any revision here.

Custom sources or source filters can opt in to block revisions anyways.

Returns:

The list of versions to explore.

Return type:

List[str]

abstract fetch()

Fetch the necessary source files into benchbuild’s cache.

Return type:

LocalPath

property is_expandable: bool

property key: str

property local: str

property remote: str | Dict[str, str]

abstract version(target_dir, version)

Fetch the requested version and place it in the target_dir

Parameters:

• target_dir (str) – The filesystem path where the version should be placed in.

• version (str) – The version that should be fetched from the local cache.

Returns:

[description]

Return type:

str

abstract versions()

List all available versions of this source.

Returns:

The list of all available versions.

Return type:

List[str]

class benchbuild.source.base.NoSource(local, remote)

Bases: FetchableSource

property default: Variant

The default version for this source.

fetch()

Fetch the necessary source files into benchbuild’s cache.

Return type:

LocalPath

version(target_dir, version)

Fetch the requested version and place it in the target_dir

Parameters:

• target_dir (str) – The filesystem path where the version should be placed in.

• version (str) – The version that should be fetched from the local cache.

Returns:

[description]

Return type:

str

versions()

List all available versions of this source.

Returns:

The list of all available versions.

Return type:

List[str]

class benchbuild.source.base.Revision(project_cls, _primary, *variants)

Bases: object

A revision captures all variants that form a single project revision.

A project may have an arbitrary number of input sources that are required for it’s defined workloads, e.g., test input files, optional dependencies, or submodules.

BenchBuild considers each source to have different version numbers, encoded as “Variants”. The complete set of “Variants” for a project then forms a project revision.

extend(*variants)

Return type:

None

has_variant(name)

Check if a variant with the given source name exists.

Parameters:

name (str) – The local name of the source.

Return type:

bool

Returns:

True, should a variant with the given name exists

property primary: Variant

project_cls: Type[Project]

sorted()

Return an ordered list of Variants from this revision.

The order is defined by the order in the SOURCE attribute of the associated project class.

Return type:

Sequence[Variant]

source_by_name(name)

Return the source object that matches the key.

Parameters:

name (str) – The local name of the source.

Return type:

FetchableSource

Returns:

the found source object

Raises:

KeyError, if we cannot find the source with this name. –

update(revision)

Return type:

None

variant_by_name(name)

Return the variant for the given source name.

Parameters:

name (str) – The local name of the source.

Return type:

Variant

Returns:

then version of the found source.

variants: Sequence[Variant]

class benchbuild.source.base.RevisionStr(value)

Bases: object

value: str

class benchbuild.source.base.Variant(owner, version)

Bases: object

Provide a ‘string’-like wrapper around source version information.

Use this, whenever you need a ‘version’ string somewhere in benchbuild. In terms of output/logging or use as program arguments, this should not carry more semantics than a simple version string.

However, this wrapper is linked to its ‘owner’. The owner serves as the back-reference to the source code where it originated from.

This can serve as a ‘hook’ to deal with version information the same way as a program variant like a specific configuraiton.

name()

Return type:

str

owner: FetchableSource

source()

Return type:

FetchableSource

version: str

class benchbuild.source.base.Versioned(*args, **kwargs)

Bases: Protocol

property default: Variant

The default version for this source.

version(target_dir, version)

Fetch the requested version and place it in the target_dir

Parameters:

• target_dir (str) – The filesystem path where the version should be placed in.

• version (str) – The version that should be fetched from the local cache.

Returns:

[description]

Return type:

str

versions()

List all available versions of this source.

Returns:

The list of all available versions.

Return type:

List[str]

benchbuild.source.base.enumerate_revisions(project_cls, context_free_enumerator=<function _default_enumerator>, context_aware_enumerator=<function _default_caw_enumerator>)

Enumerates the given sources.

The enumeration requires two phases. 1. A phase for all sources that do not require a context to evaluate. 2. A phase for all sources that require a static context.

Return type:

Sequence[Revision]

benchbuild.source.base.nosource()

Return type:

NoSource

benchbuild.source.base.primary(*sources)

Return the implicit ‘main’ source of a project.

We define the main source as the first source listed in a project.

If you define a new project and rely on the existence of a ‘main’ source code repository, make sure to define it as the first one.

Return type:

TypeVar(SourceT)

benchbuild.source.base.product(*sources)

Return the cross product of the given sources.

Return type:

Iterable[Tuple[Variant, ...]]

Returns:

An iterable containing the cross product of all source variants.

benchbuild.source.base.revision_from_str(revs, project_cls)

Create a Revision from a sequence of revision strings.

A valid Revision can only be created, if the number of valid revision strings is equivalent to the number of sources. A valid revision string is one that has been found in the a source’s version. It is required that each revision string is found in a different source version.

We assume that the first source is the primary source of the revision.

Parameters:

• revs (Sequence[RevisionStr]) – sequence of revision strings, e.g. a commit-hash.

• *sources – sources of a project.

Return type:

Revision

Returns:

A variant context.

benchbuild.source.base.secondaries(*sources)

Return the complement to the primary source of a project.

Return type:

Sequence[TypeVar(SourceT)]

Returns:

A list of all sources not considered primary.

benchbuild.source.base.sources_as_dict(*sources)

Convert fetchables to a dictionary.

The dictionary will be indexed by the Fetchable’s local attribute.

Parameters:

*sources (Fetchable) – Fetchables stored in the dictionary.

Return type:

Dict[str, Fetchable]

benchbuild.source.base.target_prefix()

Return the prefix directory for all downloads.

Returns:

the prefix where we download everything to.

Return type:

str

benchbuild.source.base.to_str(*variants)

Convert an arbitrary number of variants into their string representation.

Return type:

str

Returns:

string representation of all input variants joined by ‘,’.

Git

Declare a git source.

class benchbuild.source.git.Git(remote, local, clone=True, limit=10, refspec='HEAD', shallow=True, version_filter=<function Git.<lambda>>)

Bases: FetchableSource

Fetch the downloadable source via git.

property default: Variant

Return current HEAD as default version for this Git project.

fetch()

Clone the repository, if needed.

This will create a git clone inside the global cache directory.

Parameters:

version (Optional[str], optional) – [description]. Defaults to None.

Returns:

[description]

Return type:

str

version(target_dir, version='HEAD')

Create a new git worktree pointing to the requested version.

Parameters:

• target_dir (str) – The filesystem path where the new worktree should live.

• version (str) – The desired version the new worktree needs to point to. Defaults to ‘HEAD’.

Returns:

[description]

Return type:

str

versions()

List all available versions of this source.

Returns:

The list of all available versions.

Return type:

List[str]

class benchbuild.source.git.GitSubmodule(remote, local, clone=True, limit=10, refspec='HEAD', shallow=True, version_filter=<function Git.<lambda>>)

Bases: Git

property is_expandable: bool

Submodules will not participate in version expansion.

benchbuild.source.git.clone_needed(repository, repo_loc)

Return type:

bool

benchbuild.source.git.maybe_shallow(cmd, enable)

Conditionally add the shallow clone to the given git command.

Parameters:

• cmd (Any) – A git clone command (shallow doesn’t make sense anywhere else.

• shallow (bool) – Should we add the shallow options?

Returns:

A new git clone command, with shallow clone enabled, if selected.

Return type:

Any

HTTP

Declare a http source.

class benchbuild.source.http.HTTP(local, remote)

Bases: FetchableSource

Fetch the downloadable source via http.

property default: Variant

The default version for this source.

fetch()

Fetch via http using default version string.

Return type:

LocalPath

fetch_version(version)

Fetch via http using given version string.

Parameters:

version (str) – the version string to pull via http.

Return type:

LocalPath

Returns:

local path to fetched version.

version(target_dir, version)

Fetch the requested version and place it in the target_dir

Parameters:

• target_dir (str) – The filesystem path where the version should be placed in.

• version (str) – The version that should be fetched from the local cache.

Returns:

[description]

Return type:

str

versions()

List all available versions of this source.

Returns:

The list of all available versions.

Return type:

List[str]

class benchbuild.source.http.HTTPMultiple(local, remote, files)

Bases: HTTP

Fetch and download multiple files via HTTP.

fetch_version(version)

Fetch via http using given version string.

Parameters:

version (str) – the version string to pull via http.

Return type:

LocalPath

Returns:

local path to fetched version.

class benchbuild.source.http.HTTPUntar(local, remote)

Bases: HTTP

Fetch and download source via http and auto-unpack using GNU tar

version(target_dir, version)

Setup the given version of this HTTPUntar source.

This will fetch the given version from the remote source and unpack the archive into the build directory using tar.

The location matches the behavior of other sources. However, you need to consider that benchbuild will return a directory instead of a file path.

When using workloads, you can refer to a directory with the SourceRootRenderer using benchbuild.command.source_root. :rtype: LocalPath

Example

You specify a remote version 1.0 of an archive compression.tar.gz and a local name of “compression.tar.gz”. The build directory will look as follows:

<builddir>/1.0-compression.dir/ <builddir>/1.0-compression.tar.gz <builddir>/compression.tar.gz -> ./1.0-compression.tar.dir

The content of the archive is found in the directory compression.tar.gz. Your workloads need to make sure to reference this directory (e.g. using tokens), e.g., source_root("compression.tar.gz")

benchbuild.source.http.download_required(target_path)

Return type:

bool

benchbuild.source.http.download_single_version(url, target_path)

Return type:

str

benchbuild.source.http.normalize_remotes(remote)

Return type:

Dict[str, str]

benchbuild.source.http.versioned_target_name(target_name, version)

Return type:

str

RSync

Module: source

Declarative API for downloading sources required by benchbuild.