danilo.pianini@unibo.it
Compiled on: 2024-11-21 — printable version
The practice of integrating code with a main development line continuously
Verifying that the build remains intact
Traditionally, protoduction is jargon for a prototype that ends up in production
Software that promotes CI practices should:
Plenty of integrators on the market
Circle CI, Travis CI, Werker, done.io, Codefresh, Codeship, Bitbucket Pipelines, GitHub Actions, GitLab CI/CD Pipelines, JetBrains TeamCity…
Naming and organization is variable across platforms, but in general:
In essence, designing a CI system is designing a software construction, verification, and delivery pipeline with the abstractions provided by the selected provider.
Configuration can grow complex, and is usually stored in a YAML file
(but there are exceptions, JetBrains TeamCity uses a Kotlin DSL).
Workflows are configured in YAML files located in the default branch of the repository in the .github/workflows
folder.
One configuration file
For security reasons, workflows may need to be manually activated in the Actions tab of the GitHub web interface.
Executors of GitHub actions are called runners: virtual machines (hosted by GitHub) with the GitHub Actions runner application installed.
Note: the GitHub Actions application is open source and can be installed locally, creating “self-hosted runners”. Self-hosted and GitHub-hosted runners can work together.
Upon their creation, runners have a default environment, which depends on their operating system
Several CI systems inherit the “convention over configuration principle.
For instance, by default (with an empty configuration file) Travis CI builds a Ruby project using rake
.
GitHub actions does not adhere to the principle: if left unconfigured, the runner does nothing (it does not even clone the repository locally).
Probable reason: Actions is an all-round repository automation system for GitHub, not just a “plain” CI/CD pipeline
Minimal, simplified workflow structure:
# Mandatory workflow name
name: Workflow Name
on: # Events that trigger the workflow
jobs: # Jobs composing the workflow, each one will run on a different runner
Job-Name: # Every job must be named
# The type of runner executing the job, usually the OS
runs-on: runner-name
steps: # A list of commands, or "actions"
- # first step
- # second step
Another-Job: # This one runs in parallel with Job-Name
runs-on: '...'
steps: [ ... ]
We discussed that automation / integration pipelines are part of the software
YAML is often used by CI integrators as preferred configuration language as it enables some form of DRY:
&
/ *
)<<:
)hey: &ref
look: at
me: [ "I'm", 'dancing' ]
merged:
foo: *ref
<<: *ref
look: to
Same as:
hey: { look: at, me: [ "I'm", 'dancing' ] }
merged: { foo: { look: at, me: [ "I'm", 'dancing' ] }, look: to, me: [ "I'm", 'dancing' ] }
GHA’s YAML parser does not support standard YAML anchors and merge keys
(it is a well-known limit with an issue report open since ages)
GHA achieves reuse via:
Many actions are provided by GitHub directly, and many are developed by the community.
# This is a basic workflow to help you get started with Actions
name: Example workflow
# Controls when the workflow will run
on:
push:
tags: '*'
branches-ignore: # Pushes on these branches won't start a build
- 'autodelivery**'
- 'bump-**'
- 'renovate/**'
paths-ignore: # Pushes that change only these file won't start the workflow
- 'README.md'
- 'CHANGELOG.md'
- 'LICENSE'
pull_request:
branches: # Only pull requests based on these branches will start the workflow
- master
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "build"
Default-Example:
# The type of runner that the job will run on
runs-on: macos-latest
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout@cbb722410c2e876e24abbe8de2cc27693e501dcb
# Runs a single command using the runners shell
- name: Run a one-line script
run: echo Hello from a ${{ runner.os }} machine!
# Runs a set of commands using the runners shell
- name: Run a multi-line script
run: |
echo Add other actions to build,
echo test, and deploy your project.
Explore-GitHub-Actions:
runs-on: ubuntu-latest
steps:
- run: echo "🎉 The job was automatically triggered by a ${{ github.event_name }} event."
- run: echo "🐧 This job is now running on a ${{ runner.os }} server hosted by GitHub!"
- run: echo "🔎 The name of your branch is ${{ github.ref }} and your repository is ${{ github.repository }}."
- name: Check out repository code
uses: actions/checkout@v4
- run: echo "💡 The ${{ github.repository }} repository has been cloned to the runner."
- run: echo "🖥️ The workflow is now ready to test your code on the runner."
- name: List files in the repository
run: ls ${{ github.workspace }}
- run: echo "🍏 This job's status is ${{ job.status }}."
# Steps can be executed conditionally
- name: Skipped conditional step
if: runner.os == 'Windows'
run: echo this step won't run, it has been excluded!
- run: |
echo This is
echo a multi-line
echo script.
Conclusion:
runs-on: windows-latest
# Jobs may require other jobs
needs: [ Default-Example, Explore-GitHub-Actions ]
# Typically, steps that follow failed steps won't execute.
# However, this behavior can be changed by using the built-in function "always()"
if: always()
steps:
- name: Run something on powershell
run: echo By default, ${{ runner.os }} runners execute with powershell
- name: Run something on bash
shell: bash
run: echo However, it is allowed to force the shell type and there is a bash available for ${{ runner.os }} too.
GitHub Actions allows expressions to be included in the workflow file
${{ <expression> }}
if:
conditionals are automatically evaluated as expressions, so ${{ }}
is unnecessary
if: <expression>
works just fineThe language is rather limited, and documented at
https://docs.github.com/en/actions/learn-github-actions/expressions
The language performs a loose equality
When a string is required, any type is coerced to string
Type | Literal | Number coercion | String coercion |
---|---|---|---|
Null | null |
0 |
'' |
Boolean | true or false |
true : 1 , false : 0 |
'true' or 'false' |
String | '...' (mandatorily single quoted) |
Javascript’s parseInt , with the exception that '' is 0 |
none |
JSON Array | unavailable | NaN |
error |
JSON Object | unavailable | NaN |
error |
Arrays and objects exist and can be manipulated, but cannot be created
( )
[ ]
.
!
, and &&
, or ||
==
, !=
, <
, <=
, >
, >=
Functions cannot be defined. Some are built-in, their expressivity is limited. They are documented at
https://docs.github.com/en/actions/learn-github-actions/expressions#functions
success()
: true
if none of the previous steps failed
if: success()
conditionalalways()
: always true
, causes the step evaluation even if previous failed, but supports combinations
always() && <expression returning false>
evaluates the expression and does not run the stepcancelled()
: true
if the workflow execution has been canceledfailure()
: true
if a previous step of any previous job has failedThe expression can refer to some objects provided by the context. They are documented at
https://docs.github.com/en/actions/learn-github-actions/contexts
Some of the most useful are the following
github
: information on the workflow context
.event_name
: the event that triggered the workflow.repository
: repository name.ref
: branch or tag that triggered the workflow
refs/heads/<branch>
refs/tags/<tag>
env
: access to the environment variablessteps
: access to previous step information
.<step id>.outputs.<output name>
: information exchange between stepsrunner
:
.os
: the operating systemsecrets
: access to secret variables (in a moment…)matrix
: access to the build matrix variables (in a moment…)By default, GitHub actions’ runners do not check out the repository
It is a common and non-trivial operation (the checked out version must be the version originating the workflow), thus GitHub provides an action:
- name: Check out repository code
uses: actions/checkout@v4
Since actions typically do not need the entire history of the project, by default the action checks out only the commit that originated the workflow (--depth=1
when cloning)
Also, tags don’t get checked out
- name: Checkout with default token
uses: actions/checkout@v4.2.2
if: inputs.token == ''
with:
fetch-depth: 0
submodules: recursive
- name: Fetch tags
shell: bash
run: git fetch --tags -f
(code from a custom action, ignore the if
)
Communication with the runner happens via workflow commands
The simplest way to send commands is to print on standard output a message in the form:
::workflow-command parameter1={data},parameter2={data}::{command value}
In particular, actions can set outputs by printing:
::set-output name={name}::{value}
jobs:
Build:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: danysk/action-checkout@0.2.22
- id: branch-name # Custom id
uses: tj-actions/branch-names@v8
- id: output-from-shell
run: ruby -e 'puts "dice=#{rand(1..6)}"' >> $GITHUB_OUTPUT
- run: |
echo "The dice roll resulted in number ${{ steps.output-from-shell.outputs.dice }}"
if ${{ steps.branch-name.outputs.is_tag }} ; then
echo "This is tag ${{ steps.branch-name.outputs.tag }}"
else
echo "This is branch ${{ steps.branch-name.outputs.current_branch }}"
echo "Is this branch the default one? ${{ steps.branch-name.outputs.is_default }}"
fi
Most software products are meant to be portable
A good continuous integration pipeline should test all the supported combinations
The solution is the adoption of a build matrix
if
conditionalsjobs:
Build:
strategy:
matrix:
os: [windows, macos, ubuntu]
jvm_version: [8, 11, 15, 16] # Arbitrarily-made and arbitrarily-valued variables
ruby_version: [2.7, 3.0]
python_version: [3.7, 3.9.12]
runs-on: ${{ matrix.os }}-latest ## The string is computed interpolating a variable value
steps:
- uses: actions/setup-java@v4
with:
distribution: 'adopt'
java-version: ${{ matrix.jvm_version }} # "${{ }}" contents are interpreted by the github actions runner
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python_version }}
- uses: ruby/setup-ruby@v1
with:
ruby-version: ${{ matrix.ruby_version }}
- shell: bash
run: java -version
- shell: bash
run: ruby --version
- shell: bash
run: python --version
We would like the CI to be able to
Both operations require private information to be shared
Of course, private data can’t be shared
printenv
)How to share a secret with the build environment?
Secrets can be stored in GitHub at the repository or organization level.
GitHub Actions can access these secrets from the context:
secrets.<secret name>
context objectSecrets can be added from the web interface (for mice lovers), or via the GitHub API.
#!/usr/bin/env ruby
require 'rubygems'
require 'bundler/setup'
require 'octokit'
require 'rbnacl'
repo_slug, name, value = ARGV
client = Octokit::Client.new(:access_token => 'access_token_from_github')
pubkey = client.get_public_key(repo_slug)
key = Base64.decode64(pubkey.key)
sodium_box = RbNaCl::Boxes::Sealed.from_public_key(key)
encrypted_value = Base64.strict_encode64(sodium_box.encrypt(value))
payload = { 'key_id' => pubkey.key_id, 'encrypted_value' => encrypted_value }
client.create_or_update_secret(repo_slug, name, payload)
Signing in CI is easier if the key can be stored in memory
the alternative is to install a private key in the runner
To do so we need to:
gpg --armor --export-secret-key <key id>
Exports a base64-encoded version of your binary key, with a header and a footer.
-----BEGIN PGP PRIVATE KEY BLOCK-----
M4ATvaZBpT5QjAvOUm09rKsvouXYQE1AFlmMfJQTUlmOA_R6b-SolgYFOx_cKAAL
Vz1BIv8nvzg9vFkAFhB7N7QGwYfzbKVAKhS0IDQutDISutMTS3ujJlvKuQRdoE2z
...
WjEW1UmgYOXLawcaXE2xaDxoXz1FLVxxqZx-LZg_Y/0tsB==
=IN7o
-----END PGP PRIVATE KEY BLOCK-----
Note: armoring is not encryption
In most CI systems, secrets allow enough space for an armored GPG private keys to fit in
In this case, just export the armored version as a secret
Otherwise:
How to tell if you are in CI or not?
CI
environment variable is automatically set to "true"
on most CI environments
if (System.getenv("CI") == true.toString()) {
signing {
val signingKey: String? by project
val signingPassword: String? by project
useInMemoryPgpKeys(signingKey, signingPassword)
}
}
signingKey
and signingPassword
properties must get passed to Gradle
./gradlew -PsignigngKey=... -PsigningPassword=... <tasks>
ORG_GRADLE_PROJECT_<variableName>
env:
ORG_GRADLE_PROJECT_signingKey: ${{ secrets.SIGNING_KEY }}
ORG_GRADLE_PROJECT_signingPassword: ${{ secrets.SIGNING_PASSWORD }}
Imperative behaviour in GitHub Actions is encapsulated into actions
Actions are executed as a single logical step, with inputs and outputs
Their metadata is written in a actions.yml
file on the repository root
GitHub actions stored on GitHub are usable without further deployment steps
owner/repo@<tree-ish>
as referencename: 'A string with the action name'
description: 'A long description explaining what the action does'
inputs:
input-name: # id of input
description: 'Input description'
required: true # whether it should be mandatorily specified
default: 'default value' # Default value, if not specified by the caller
outputs:
# Outputs will be set by the action when running
output-name: # id of output
description: 'Description of the output'
runs: # Content depends on the action type
Composite actions allow the execution of multiple steps that can be scripts or other actions.
runs:
using: composite
steps: [ <list of steps> ]
The action is contained in its metadata descriptor action.yml
root, e.g.:
name: 'Checkout the whole repository'
description: 'Checkout all commits, all tags, and initialize all submodules.'
inputs:
token:
description: The token to use when checking out the repository
required: false
default: ''
runs:
using: "composite"
steps:
- name: Checkout with custom token
uses: actions/checkout@v4.2.2
if: inputs.token != ''
with:
fetch-depth: 0
submodules: recursive
token: ${{ inputs.token }}
- name: Checkout with default token
uses: actions/checkout@v4.2.2
if: inputs.token == ''
with:
fetch-depth: 0
submodules: recursive
- name: Fetch tags
shell: bash
run: git fetch --tags -f
It can be used with:
steps:
# Checkout the repository
- name: Checkout
uses: danysk/action-checkout@0.2.22
For instance this way:
name: 'Composite action with a secret'
description: 'For teaching purposes'
inputs:
token: # github token
description: 'Github token for deployment. Skips deployment otherwise.'
required: true
runs:
using: "composite"
steps:
- run: '[[ -n "${{ inputs.token }}" ]] || false'
name: Fail if the toke is unset
Dockerfile
of the container you want to useENTRYPOINT
action.yml
runs
section set using: docker
and the arguments order
runs:
using: 'docker'
image: 'Dockerfile' # Alternatively, the name of an existing image
args:
- ${{ inputs.some-input-name }}
The most flexible way of writing actions
runs:
using: 'node12'
main: 'index.js'
npm init -y
npm install @actions/<toolkitname>
const core = require('@actions/core');
try {
const foo = core.getInput('some-input');
console.log(`Hello ${foo}!`);
} catch (error) {
core.setFailed(error.message);
}
GitHub actions also allows to configure reusable workflows
workflow_dispatch
if they have more than 10 parametersenv
context object) of the caller are not propagated to the calleeStill, the mechanism enables to some extent the creation of libraries of reusable workflows
name: ...
on:
workflow_call: # Trigger when someone calls the workflow
inputs:
input-name:
description: optional description
default: optional default (otherwise, it is assigned to a previous)
required: true # Or false, mandatory
type: string # Mandatory: string, boolean, or number
secrets: # Secrets are listed separately
token:
required: true
jobs:
Job-1:
... # It can use a matrix, different OSs, and
Job-2:
... # Multiple jobs are okay!
Similar to using an action!
uses
is applied to the entire jobsteps
cannot be definedname: ...
on:
push:
pull_request:
#any other event
jobs:
Build:
uses: owner/repository/.github/workflows/build-and-deploy-gradle-project.yml@<tree-ish>
with: # Pass values for the expected inputs
deploy-command: ./deploy.sh
secrets: # Pass down the secrets if needed
github-token: ${{ secrets.GITHUB_TOKEN }}
Ever happenend?
The sooner the issue is known, the better
cron
CI jobs if there is no action on the repository, which makes the mechanism less usefulThere exist a number of recommended services that provide additional QA and reports.
Non exhaustive list:
The Linux Foundation Core Infrastructure Initiative created a checklist for high quality FLOSS.
CII Best Practices Badge Program https://bestpractices.coreinfrastructure.org/en
A full-fledged CI system allows reasonably safe automated evolution of software
At least, in terms of dependency updates
Assuming that you can effectively intercept issues, here is a possible workflow for automatic dependency updates:
Bots performing the aforementioned process for a variety of build systems exist.
They are usually integrated with the repository hosting provider
Some tasks do require humans:
However, we may still want these documents to follow a template
Most Git hosting services allow to specify a template.
Templates in GitHub are special files found in the .github
folder,
written in YAML or Markdown, and stored on the default branch.
The descriptor generates a form that users must fill.
They are available for both issues and pull requests, and share most of the syntax.
.github
├── PULL_REQUEST_TEMPLATE
│ ├── config.yml
│ ├── example.yml
│ └── another-example.md
└── ISSUE_TEMPLATE
├── config.yml
├── example.yml
└── another-example.md
Any md
or yml
file located in .github/ISSUE_TEMPLATE
is considered as a template for issues
Any md
or yml
file located in .github/PULL_REQUEST_TEMPLATE
is considered as a template for pull requests
If a single template is necessary,
a single .github/ISSUE_TEMPLATE.md
or .github/PULL_REQUEST_TEMPLATE.md
file
replaces the content of the whole directory
---
name: 🐞 Bug
about: File a bug/issue
title: '[BUG] <title>'
labels: bug, to-triage
assignees: someone, someoneelse
---
### Current Behavior:
<!-- A concise description of what you're experiencing. -->
### Expected Behavior:
<!-- A concise description of what you expected to happen. -->
### Steps to reproduce:
1. first do...
2. and then...