STASIS
A pipeline delivery generator
|
STASIS consolidates the steps required to build, test, and deploy calibration pipelines and other software suites maintained by engineers at the Space Telescope Science Institute.
Download the source code
Create and enter the build directory
Run cmake
Compile and install stasis
Missions definitions live in STASIS's etc/mission
directory. Let's create a new one specifically geared toward generic data analysis tools. You may override the path to the etc
directory by setting the STASIS_SYSCONFDIR
environment variable to different location.
Now populate the new data analysis mission configuration. Refer to the release formatters section to see a list of what each %
formatter does.
Text files containing STASIS template strings can be stored at the same level as the mission configuration, and rendered to anywhere inside the output directory. This will can you time if you plan to release for multiple platforms and architectures.
STASIS's configuration parser does not distinguish input files by extension, so mydelivery.ini
, mydelivery.cfg
, mydelivery.txt
, and abc123.zyx987
are all perfectly valid file names.
All deliveries require a [meta]
section. Here global metadata such as the delivery's name
, version
, and any programs/dependencies that make up the deliverable.
The [conda]
section instructs STASIS how to obtain the conda installer of your choice, and defines the packages to be installed into the delivery's release environment.
Create some test cases for packages.
Long Option | Short Option | Purpose |
---|---|---|
–help | -h | Display usage statement |
–version | -V | Display program version |
–continue-on-error | -C | Allow tests to fail |
–config ARG | -c ARG | Read STASIS configuration file |
–cpu-limit ARG | -l ARG | Number of processes to spawn concurrently (default: cpus - 1) |
–pool-status-interval ARG | n/a | Report task status every n seconds (default: 30) |
–python ARG | -p ARG | Override version of Python in configuration |
–verbose | -v | Increase output verbosity |
–unbuffered | -U | Disable line buffering |
–update-base | n/a | Update conda installation prior to STATIS environment creation |
–fail-fast | n/a | On test error, terminate all tasks |
–overwrite | n/a | Overwrite an existing release |
–no-docker | n/a | Do not build docker images |
–no-artifactory | n/a | Do not upload artifacts to Artifactory |
–no-artifactory-build-info | n/a | Do not upload build info objects to Artifactory |
–no-artifactory-upload | n/a | Do not upload artifacts to Artifactory (dry-run) |
–no-testing | n/a | Do not execute test scripts |
–no-parallel | n/a | Do not execute tests in parallel |
–no-rewrite | n/a | Do not rewrite paths and URLs in output files |
DELIVERY_FILE | n/a | STASIS delivery file |
Name | Purpose |
---|---|
TMPDIR | Change default path to store temporary data |
STASIS_ROOT | Change default path to write STASIS's data |
STASIS_SYSCONFDIR | Change default path to search for configuration files |
STASIS_CPU_COUNT | Number of available CPUs |
STASIS_GH_TOKEN | GitHub API token (Scope: "Contents" repository permissions (write)) |
STASIS_JF_ARTIFACTORY_URL | Artifactory service URL (ending in /artifactory ) |
STASIS_JF_ACCESS_TOKEN | Artifactory Access Token |
STASIS_JF_USER | Artifactory username |
STASIS_JF_PASSWORD | Artifactory password |
STASIS_JF_SSH_KEY_PATH | Path to SSH public key file |
STASIS_JF_SSH_PASSPHRASE | Password associated with SSH public key file |
STASIS_JF_CLIENT_CERT_CERT_PATH | Path to OpenSSL cert files |
STASIS_JF_CLIENT_CERT_KEY_PATH | OpenSSL key file (in cert path) |
STASIS_JF_REPO | Artifactory "generic" repository to write to |
The default path to the configuration file is [CMAKE_INSTALL_PREFIX]/etc/stasis/stasis.ini
. You may override this by setting the STASIS_SYSCONFDIR
environment variable to a path that points elsewhere.
Name | Type | Purpose |
---|---|---|
continue_on_error | Boolean | Keep going even if a test fails |
always_update_base_environment | Boolean | Update all packages in the base to the latest release |
conda_fresh_start | Boolean | Remove conda installation during initialization |
conda_install_prefix | String | Install conda in a custom prefix path |
conda_packages | List | Conda packages to be installed/overridden in the base environment |
pip_packages | List | Python packages to be installed/overridden in the base environment |
conda_staging_url | String | URL to conda channel |
Name | Type | Purpose |
---|---|---|
url | String | Base URL of JFrog CLI release server |
project | String | Product identifier (i.e. jfrog-cli ) |
version_series | String | Product version series (i.e. v2-jf ) |
version | String | Product version to install. [RELEASE] downloads the latest version. |
filename | String | Product file name (i.e. jf ) |
Name | Type | Purpose |
---|---|---|
url | String | Set artifactory service URL (ending in /artifactory) |
repo | String | Set artifactory repository |
All configuration section names and keys are case-sensitive.
Key | Type | Purpose | Required |
---|---|---|---|
mission | String | Mission rule directory to use | Y |
name | String | Name of the delivery | Y |
version | String | Version of delivery | Y |
codename | String | Release code name (HST-only) | N |
rc | Integer | Release candidate level (i.e. delivery iteration) | Y |
final | Boolean | Toggle final delivery logic (possibly deprecated in the future) | N |
based_on | String | A conda environment YAML file/URL to use as a baseline | N |
python | String | Version of Python to base delivery on | Y |
Key | Type | Purpose | Required |
---|---|---|---|
installer_name | String | Conda distribution name | Y |
installer_version | String | Conda distribution version | Y |
installer_platform | String | Conda target platform | Y |
installer_arch | String | Conda target architecture | Y |
installer_baseurl | String | Conda installer URL | Y |
conda_packages | List | Conda packages to install | N |
pip_packages | List | Pypi packages to install | N |
Environment variables exported are global to all programs executed by stasis. There is no limit to the number of environment variables that can be set.
Key | Type | Purpose | Required |
---|---|---|---|
arbitrary name | String | Export environment variables to build and test environment | N |
artitrary_name ... N | - | - | - |
Sections starting with test:
will be used during the testing phase of the stasis pipeline. Where the value of name
following the colon is an arbitrary value, and only used for reporting which test-run is executing. Section names must be unique.
Key | Type | Purpose | Required |
---|---|---|---|
disable | Boolean | Disable script execution (script_setup always executes) | N |
parallel | Boolean | Execute test block in parallel (default) or sequentially | N |
build_recipe | String | Git repository path to package's conda recipe | N |
repository | String | Git repository path or URL to clone | Y |
version | String | Git commit or tag to check out | Y |
runtime | List | Export environment variables specific to test context | Y |
script_setup | List | Body of a shell script that will install dependencies | N |
script | List | Body of a shell script that will execute the tests | Y |
Sections starting with deploy:artifactory:
will define the upload behavior of build and test artifacts to Artifactory. Where the value of name
is an arbitrary value, and only used for reporting. Section names must be unique.
Key | Type | Purpose | Required |
---|---|---|---|
files | List | jf -compatible wildcard path | Y |
dest | String | Remote artifactory path to store files | Y |
The deploy:docker
section controls how Docker images are created, when a Dockerfile
is present in the build_docker_dir
.
Key | Type | Purpose | Required |
---|---|---|---|
registry | String | Docker registry to use | Y |
image_compression | String | Compression program (with arguments) | N |
build_args | List | Values passed to docker build --build-args | N |
tags | List | Docker image tag(s) | Y |
Template strings can be accessed using the {{ subject.key }}
notation in any STASIS configuration file.
Name | Purpose |
---|---|
meta.name | Delivery name |
meta.version | Delivery version |
meta.codename | Delivery codename |
meta.mission | Delivery mission |
meta.python | Python version (e.g. 3.11) |
meta.python_compact | Python (e.g. 311) |
info.time_str_epoch | UNIX Epoch timestamp |
info.release_name | Rendered delivery release name |
info.build_name | Rendered delivery build name |
info.build_number | Rendered delivery build number |
storage.tmpdir | Ohymcal temp directory |
storage.delivery_dir | STASIS delivery output directory |
storage.results_dir | STASIS test results directory |
storage.conda_artifact_dir | STASIS conda package directory |
storage.wheel_artifact_dir | STASIS wheel package directory |
storage.build_sources_dir | STASIS sources directory |
storage.build_docker_dir | STASIS docker directory |
conda.installer_name | Conda distribution name |
conda.installer_version | Conda distribution version |
conda.installer_platform | Conda target platform |
conda.installer_arch | Conda target architecture |
conda.installer_baseurl | Conda installer URL |
system.arch | System CPU Architecture |
system.platform | System Platform (OS) |
deploy.docker.registry | Docker registry |
deploy.jfrog.repo | Artifactory destination repository |
workaround.conda_reactivate | Reinitialize the conda runtime environment. Use this after calling conda install from within a [test:*].script . |
The template engine also provides an interface to environment variables using the {{ env:VARIABLE_NAME }}
notation.
Template functions can be accessed using the {{ func:NAME(ARG,...) }}
notation.
Name | Arguments | Purpose |
---|---|---|
get_github_release_notes_auto | n/a | Generate release notes for all test contexts |
basetemp_dir | n/a | Generate directory path to test block's temporary data directory |
junitxml_file | n/a | Generate directory path and file name for test result file |
Mission rules are defined in the STASIS_SYCONFDIR/mission
directory. Each mission configuration file shares the same name as the directory. To create a new mission, example
, the directory structure will be as follows:
Key | Type | Purpose | Required |
---|---|---|---|
release_fmt | String | A string containing one or more release format specifiers | Y |
build_name_fmt | String | - | Y |
build_number_fmt | String | - | Y |
Formatter | Purpose |
---|---|
n | Delivery name |
c | Delivery codename |
r | Delivery RC level |
R | Delivery RC level (final-aware) |
v | Delivery version |
m | Mission name |
P | Python version (i.e. 3.11) |
p | Compact Python version (i.e. 311 |
a | System CPU Architecture |
o | System Platform (OS) |
t | UNIX Epoch timestamp |
Sections starting with template:
will make use of the built-in template engine, replacing any strings in filename
, relative to the mission directory, with expanded data values. The output will be written to destination
.