Containerisation and Orchestration

Software Process Engineering

Giovanni Ciatto — giovanni.ciatto@unibo.it

Compiled on: 2026-11-07 — printable version

back

Outline

1. Motivation: deployment, tasks, and their execution context
2. Encapsulation: virtual machines vs. containers
3. Docker basics: architecture, running containers, managing images
4. Images: Dockerfiles, layers, caching, tagging, publishing
5. Volumes: bind mounts, named volumes, tmpfs
6. Networks: network drivers, exposing and publishing ports
7. Orchestration with Docker Compose: stacks, services, configs, secrets, testing
8. Running experiments in parallel: from notebooks to parallel, reproducible, containerised experiments
9. Orchestration with Docker Swarm: clusters, stacks, placement, replication

Preliminaries

[Software] Deployment $:=$ all of the activities that make a software system available for use

• Software Engineering: A Practitioner’s Approach, Roger S. Pressman

General need in Software Engineering

To govern the deployment of software systems onto some infrastructure

• software system: multiple interacting software components, each one having its computational requirements

  • component $\approx$ process

• infrastructure: the available hardware and software substratum supporting the execution of software components

  • hardware $\approx$ CPU, memory, storage, network
  • software $\approx$ operating system, drivers, runtimes, libraries

• deployment: orderly instantiating software components onto the infrastructure, while

  • satisfying the computational requirements of each component
  • optimising the exploitation of the infrastructural facilities

What to deploy? (pt. 1)

• Short-lived tasks (a.k.a. jobs): tasks which eventually terminate

  • e.g. data processing or generation tasks
  • e.g. simulations

• Long-lived tasks (a.k.a. services) tasks which run indefinitely

  • e.g. web servers
  • e.g. databases

What to deploy? (pt. 2)

Short-lived tasks

Short-lived tasks are bare computational activities aimed at processing/generating some data

• Pure algorithms accepting data as input and/or producing data as output.

• They will eventually reach their natural termination

  • without requiring external intervention

• Common workflow:

  1. users provide the code and the input data,
  2. they start the task,
  3. they are interested in getting the output data

• Parallelism may be exploited to speed up the computations

What to deploy? (pt. 3)

Long-lived tasks

Long-lived are computational activities which are not meant to terminate automatically (unless explicitly stopped)

• Infinite loops, just waiting for requests, and serving them as soon as possible

• Interactivity is a key aspect:

  • clients interact with the service,
  • the service reacts to the clients’ requests
  • clients may be humans or other software components

• Common workflow:

  • users provide the code and some deployment-automation script
  • they start the task,
  • they are interested in interacting with the service
    • e.g. via the shell, via some GUI, via the browser, etc.

• Parallelism may be exploited to support replication

  • e.g. for fault tolerance
  • e.g. for load balancing
  • e.g. for scaling

Execution context of a computational task

Regardless of their life-span, computational tasks may require computational resources of various sorts:

• specific hardware capabilities

  • e.g. quick / multi-core CPUs for computation intensive tasks
  • e.g. GPUs for tasks requiring parallelism
  • e.g. primary memory (RAM)
  • e.g. large / quick storage (SSD, HDD, etc.)
  • e.g. fast network connections (e.g. optical fibre)

• specific operating systems (Linux, macOS, Windows, etc.) or specific architectures (x86, ARM, etc.)

• specific runtime platforms (e.g. JVM, Python, .NET etc.)

• specific infrastructural components (e.g. database, reverse proxy, load-balancer, broker, etc.)

• specific libraries (e.g. CUDA, numpy, etc.)

The need for encapsulation (pt. 1)

Why can’t we simply configure bare-metal machines to host our tasks?

• Need for multi-tenancy (i.e. sharing) on computational resources

  1. one may have $N$ machines and $M$ applications…
     • … and be willing to decouple applications from machines
  2. different software components may have incompatible HW / SW requirements

• Need for isolation

  • preventing spurious interactions among different tenants
  • a common requirement in multi-tenant scenarios

• Need to formalize / control / reproduce the computational environment of applications

  • formalisation is a pre-requisite for automation
  • automation enables control and scale

• Need to automate deployment of applications into production / testing environments

  • this is the immediate benefit for engineers

• Need for flexibility (in re-deployment), and scalability

  • effort minimisation for application startup or transfer

The need for encapsulation (pt. 2)

Some abstraction is needed to encapsulate applications, and their computational environment, into a single unit

Encapsulation is the hiding of the implementation details of a component from its clients. Encapsulated components have a clear interface which is the only way to interact with them.

How to achieve encapsulation? (pt. 1)

• Virtual machines (VM) on top of VM hypervisors (VMH)

• Containers on top of container engines (CE)

How to achieve encapsulation? (pt. 2)

Virtual machines and hypervisors (VMH)

• VMH run on/as the OS of a bare-metal machine, abstracting HW and SW peculiarities away

  • VM have virtualised HW and SW resources, different from the host’s

• VMH may instantiate multiple VM on the same physical machine

  • partitioning actual resources ahead of time

• each VM runs its own OS, and may host multiple applications

  • in the eyes of the user, the VM is indistinguishable from a bare-metal machine

• VMs may be paused, snapshotted (into a file), resumed, and possibly migrated

  • snapshots are files containing the whole file-system of the VM

• VM are coarse-grained encapsulation units

  • they are heavy-weight (GBs)
  • they are slow to start, run, migrate, snapshot
  • VM commonly encapsulate multiple application-level components
    • database, server, plus all their runtimes and libraries, etc.

• Many industry-ready technologies:

  • VMware, VirtualBox, KVM, Xen, Hyper-V, QEMU, Proxmox, etc.

How to achieve encapsulation? (pt. 3)

Containers and container engines (CE)

• CE run on the OS of a bare-metal/virtual machine, abstracting the runtime, storage, and network environment away

  • the CPU, memory, OS kernel, drivers, and hardware are not virtualised

• one may instantiate multiple containers on the same machine

  • sharing the actual resources dynamically (overbooking support)

• each container shares the OS kernel of the host, yet having its own runtime(s), storage, and network facilities

  • in the eyes of the user, the container is a process running on top of a minimal OS

• each container is an instance of an image, i.e. a read-only template containing deployment instructions

  • differences w.r.t. that image constitute the state of the container (these can be snapshotted into a file of minimal size)

• containers are fine-grained encapsulation units

  • they are light-weight (MBs)
  • they are fast to start, run, snapshot
  • they are commonly used to encapsulate single application-level components
    • e.g. a database instance, a web-server instance, etc.
    • the final application should consist of several, inter-communicating containers

• Many industry-ready technologies:

  • Docker, LXC, LXD, Podman, etc.

Bare-metal vs. VMs vs. Containers

Containers provide runtime isolation without operating system replication

Lightweight virtual machines?

Closer to confined processes

Scaling up to the cluster level (pt. 1)

Cluster $\approx$ a set of similarly configured machines (a.k.a. nodes) interconnected via a network in order to let users exploit their joint computational power

• Users may want to deploy their applications on the cluster

  • by means of a single access point
    • e.g. a Web dashboard, or a CLI

• Deployment shall allocate tasks on the cluster efficiently

  • meeting the tasks’ computational requirements
  • balancing the load among the nodes
  • matching requirements on actual capabilities of nodes

• Infrastructure-as-a-service (IaaS) cloud technologies support deploying tasks on clusters, as VM

  • e.g. OpenStack, vSphere, etc.

• Container orchestrators support deploying tasks on clusters, as containers

  • e.g. Kubernetes, Docker Swarm, etc.

Scaling up to the cluster level (pt. 2)

• The access point (e.g. a container orchestrator) decides where each task runs, so users do not have to
• Tasks declare their requirements, nodes expose their capabilities (cf. labelling nodes in Swarm)

Why containers? (pt. 1)

Why containers? (pt. 2)

• more fine-grained encapsulation (than VM)
• faster to start, stop, snapshot, migrate (w.r.t. VM)
• more modular, more scalable (than VM)
• useful for dev-ops, and CI/CD
• de-facto standard packaging format for cloud-native deployments (e.g. Kubernetes)

Containerisation vs. Orchestration

• Containerisation $\approx$ the process of encapsulating an application into a container

• Orchestration $\approx$ the process of deploying one or more containers onto one or more machines

• Containerisation is a pre-requisite for orchestration

• Two syntaxes involved:

  • one to containerise (i.e. create container images)
  • one to orchestrate (i.e. deploy containers)

Main abstractions

Encapsulation level

• Container: a sandbox for running a process and its computational environment
• Image: a template for creating containers
• Layer: a single, cacheable, step in the creation of an image
• Host: the machine hosting the containers
• Registry: a repository of images (possibly external w.r.t. the host)
• Network: a virtual network for connecting containers (with each other and with the host)
• Volume: a bridge for letting containers share data with the host
• Engine (a.k.a. daemon): the software running on the host, managing containers, images, volumes, layers, and networks

Main abstractions (overview)

About technologies

Docker: the most famous container technology, actually consisting of several components

• Docker Engine: the container engine managing containers and images, locally
• Docker CLI: the command line interface for Docker Engine (this is what you use)
• Docker Desktop: a GUI for Docker Engine, mostly for inspection purposes (on macOS/Windows it also runs the Engine in a VM)
• Docker Hub: the default registry for Docker images, available online
• Docker Compose: a tool for orchestrating containers on a single machine
• Docker Swarm (mode): the Engine’s built-in support for orchestrating containers on a cluster of machines

Configure Docker locally

1. Install Docker
   • https://docs.docker.com/engine/install/
   • most commonly available on package managers
2. [Linux only] Add your user to the docker group
   • sudo usermod -aG docker $USER
   • log out and log in again
   • beware: members of the docker group are de facto root on the host
3. Enable and start the Docker service
   • on most Linux distributions sudo systemctl enable docker; sudo systemctl start docker
   • on macOS and Windows, start the Docker Desktop application
4. Test your installation
   • docker run hello-world
5. Explore admissible sub-commands with docker --help
   • general command structure docker <resource> <command> <options> <args>
     • sometimes, when it’s obvious, <resource> can be omitted

Subsequent examples are shown on Linux, but they should work on any platform

• provided that a bash or zsh shell is available

Running containers

1. Pull an image: docker pull python:3.14
2. Run a container: docker run python:3.14
   • nothing seems to happen: the container starts and terminates immediately

Every image provides a default command (for python:3.14, it is the python3 REPL); running without options runs that default in a non-interactive terminal

• the REPL finds no input, hence it terminates

Running in interactive mode can be achieved with the -i option

• use the -t option to run in a pseudo-tty (always use it whenever you use -i)
• e.g., docker run -it python:3.14 opens the Python REPL inside the container

Running a custom command inside the container can be achieved by writing the command after the image name

• e.g., docker run -it python:3.14 bash opens a shell inside the container
• parameters for the custom command can follow
  • e.g., docker run python:3.14 python -c "import sys; print(sys.version)"
• use --rm to remove the container after use

Container lifecycle

• docker ps lists running containers, docker ps -a lists all of them (including created and exited ones)
• docker start / docker stop / docker rm accept container names (cf. --name) or IDs

Interaction with the outside world

Managing images

Every image has a unique ID, and may have an associated name (including a tag)

The subcommand images lists the pulled images and their associated information

The subcommand image allows for running maintenance tasks, e.g.

• docker image ls – same as docker images
• docker image prune – removes unused images
• docker image rm – removes images by name
• docker image tag – associates a tag to an image

Creating Docker images

Docker images are written in a Dockerfile

• Every filesystem-changing instruction (RUN, COPY, ADD) generates a new layer

  • other instructions (ENV, EXPOSE, CMD, …) only change the image’s metadata

• The final stack of layers creates the final image

• The docker build command interprets the Dockerfile commands to produce a sequence of layers

• Changes to a layer do not invalidate previous layers, but they do invalidate all subsequent ones

Layers and caching (pt. 1)

Every filesystem-changing instruction in a Dockerfile generates a new layer, the others only set the image configuration:

A layer is a diff w.r.t. the previous one

In other words, Docker keeps track of what information is added to the image at each step

Layers and caching (pt. 2)

When a Dockerfile is built, Docker checks whether each layer is already in the build cache

• i.e. same parent layer, same instruction, and (for COPY/ADD) same file contents

If so, it reuses the cached layer, and skips the execution of the corresponding command

• e.g. rebuilding my-service after editing index.mjs only:

Layers and caching (pt. 3)

What if the Dockerfile copied the whole project before installing dependencies?

• i.e. COPY . . followed by RUN npm install, rather than COPY package.json ./, RUN npm install, COPY . .

Rule of thumb: order instructions from the least to the most frequently changing, and copy dependency declarations (package.json, requirements.txt, build.gradle.kts, …) separately from the source code

Layers and caching (pt. 4)

When the container is run, the image layers are read-only, and the container has a read-write layer on top of them

So, when a container is removed (not merely stopped), the read-write layer is discarded, and the image layers are kept

In this way, the space occupied by the container is minimal

Naming images

Image naming is done via tags

• The easiest way to do so is assigning tags at build time with the -t option of docker build

• The option can be repeated to assign multiple tags (names must be lowercase)

  docker build -t "my-image:latest" -t "my-image:0.1.0" path/to/build/context
  

• latest is usually used to identify the most recent version of some image

• For instance, you may build the image from a few slides ago with:

  • docker build -t my-service path/to/my/nodejs/project
    • the image my-service:latest is automatically created

Publishing Docker images

Images get published in registries

• The most famous, free for publicly available images, is Docker Hub

• By default, Docker uses Docker Hub as registry (both for pull and push operations)

• Docker Hub requires registration and CLI login:

  • docker login docker.io

• Once done, publication is performed via push, provided that the image name includes your namespace:

  • docker tag my-image:0.1.0 <username>/my-image:0.1.0
  • docker push <username>/my-image:0.1.0

Building Docker images in CI

• Of course, like any other software, custom docker images should get built in CI

• Several integrators use containers as build environments:

  • it is possible to build a container using a container

• More generally: there is no inherent limit to nesting containers

• For instance:

  • you may run DIND (Docker-in-Docker) via: docker run --privileged --rm -it docker:dind
  • you may use the Docker CLI in a container: docker run -it --rm docker:cli

Volumes

Volumes are bridges between the host’s file system (or memory) and the container’s one

They support several use cases:

• sharing data between the host and the container
• persisting containers’ data
• sharing data among containers

Sorts of volumes (pt. 1)

• Bind mount: a host directory is mounted into the container
  • the host directory is specified as <host path>:<guest path> (use absolute paths, e.g. via $(pwd))
    • e.g. docker run -v /home/$USER/path/to/file:/data ...
  • data present in <host path> can be read within the container at <guest path>
  • data written within the container at <guest path> can be accessed on <host path>
    • even after the container is stopped / deleted

Sorts of volumes (pt. 2)

• Named volumes: virtual drives managed by Docker
  • they are created with docker volume create <name>
  • they are mounted into containers as <name>:<guest path>
  • their content’s life span is independent of any container
  • they can be attached to containers at run-time, referencing their name:
    • e.g. docker run -v my-volume:/my-project ...
  • technically, they are just directories on the host’s file system, yet handled by Docker
    • /var/lib/docker/volumes/<name>/_data
    • on Docker Desktop, this path is inside the Engine’s VM

Sorts of volumes (pt. 3)

• Tmpfs mount [Linux-only]: a temporary file system, living only as long as the container is alive
  • it is useful for storing temporary data
  • do not use it for sharing or persisting data
  • it is like an anonymous, ephemeral named volume
  • created by means of a --tmpfs option to docker run
    • e.g. docker run --tmpfs <guest path> ...
    • or docker run --mount type=tmpfs,destination=<guest path> ...

Volumes in practice

• Named volumes db_data and wordpress_data persist the data of the db and wordpress containers
  • cf. the WordPress stack in the Docker Compose part, where Compose prefixes their names with the project name
• The same volumes can be shared with other containers, e.g. a one-shot backup container mounting them read-only…
• … and sharing results with the host via a bind mount:

  docker run --rm \
      -v db_data:/from/db:ro \
      -v wordpress_data:/from/wp:ro \
      -v /home/$USER/backups:/to \
      alpine tar czf /to/backup.tar.gz -C / from
  

Example: containers’ data is ephemeral

1. Let’s open a Linux shell in a container: docker run -it --rm alpine:latest sh

2. Let’s create a folder in there: mkdir -p /data

3. Let’s create a file in there: echo "Hello world" > /data/hello.txt

4. Does the file exist? ls -la /data

   / # ls -la /data
   total 12
   drwxr-xr-x    2 root     root          4096 Nov  2 09:28 .
   drwxr-xr-x    1 root     root          4096 Nov  2 09:28 ..
   -rw-r--r--    1 root     root            12 Nov  2 09:28 hello.txt
   

5. Let’s exit the container: exit

6. Let’s open a new shell in a new container: docker run -it --rm alpine:latest sh

7. Does the file exist? ls -la /data

   ls: /data: No such file or directory
   

8. Why?

   • the second docker run ... command created a new container

Example: sharing data with bind mounts (pt. 1)

1. Let’s create a folder on the host: mkdir -p $(pwd)/shared

2. Let’s create 10 containers

   1. each one creating a file in ./shared/
   2. named container-<i>.txt
   3. containing random data from /dev/random, encoded in base64
   4. in detached mode (-d), with a bind mount (-v), and a parametric hostname (--hostname)

   for i in $(seq 1 10); do
       docker run --rm -d \
           -v $(pwd)/shared:/data \
           --hostname container-$i \
           alpine sh -c \
           'cat /dev/random | head -n 1 | base64 > /data/$(hostname).txt'
   done
   

3. Let’s have a look at the shared directory: ls -la ./shared

   ls -la ./shared
   totale 56
   drwxr-xr-x  2 user    user     4096  2 nov 10.51 .
   drwxr-xr-x 79 user    user    12288  2 nov 10.56 ..
   -rw-r--r--  1 root    root      329  2 nov 10.51 container-10.txt
   -rw-r--r--  1 root    root      333  2 nov 10.51 container-1.txt
   -rw-r--r--  1 root    root      381  2 nov 10.51 container-2.txt
   -rw-r--r--  1 root    root      167  2 nov 10.51 container-3.txt
   -rw-r--r--  1 root    root      110  2 nov 10.51 container-4.txt
   -rw-r--r--  1 root    root      102  2 nov 10.51 container-5.txt
   -rw-r--r--  1 root    root        9  2 nov 10.51 container-6.txt
   -rw-r--r--  1 root    root      106  2 nov 10.51 container-7.txt
   -rw-r--r--  1 root    root      766  2 nov 10.51 container-8.txt
   -rw-r--r--  1 root    root      203  2 nov 10.51 container-9.txt
   

Example: sharing data with bind mounts (pt. 2)

4. Things to notice:
   • the files are owned by root (inside the container)
   • the files are still owned by root on the host (outside the container)
     • despite the fact that the host user is user
     • if user has no superuser privileges, they cannot edit the files
       • they can still delete them, as ./shared is owned by user
     • (on Docker Desktop, the VM may map ownership to the host user instead)
   • when creating the container, bind mounts should be absolute paths
     • hence the need for $(pwd) to get the current working directory

Example: sharing data with named volumes (pt. 1)

1. Let’s create a named volume: docker volume create my-volume

2. Same 10 containers as before, but using the named volume (-v my-volume:/data) instead of the bind mount

   for i in $(seq 1 10); do
       docker run --rm -d \
           -v my-volume:/data \
           --hostname container-$i \
           alpine sh -c \
           'cat /dev/random | head -n 1 | base64 > /data/$(hostname).txt'
   done
   

3. How to access the data now?

   1. by means of another container, attached to the same volume
   2. by means of the host, inspecting the volume’s file system

Example: sharing data with named volumes (pt. 2)

Accessing the volume from another container

1. Let’s create yet another interactive container attached to the volume
   • docker run --rm -it -v my-volume:/data alpine sh
2. Let’s have a look at the data: ls -la /data

   total 48
   drwxr-xr-x    2 root     root          4096 Nov  2 10:15 .
   drwxr-xr-x    1 root     root          4096 Nov  2 10:15 ..
   -rw-r--r--    1 root     root           608 Nov  2 10:14 container-1.txt
   -rw-r--r--    1 root     root           349 Nov  2 10:15 container-10.txt
   -rw-r--r--    1 root     root           369 Nov  2 10:14 container-2.txt
   -rw-r--r--    1 root     root           304 Nov  2 10:14 container-3.txt
   -rw-r--r--    1 root     root           240 Nov  2 10:14 container-4.txt
   -rw-r--r--    1 root     root           434 Nov  2 10:14 container-5.txt
   -rw-r--r--    1 root     root           150 Nov  2 10:14 container-6.txt
   -rw-r--r--    1 root     root            41 Nov  2 10:14 container-7.txt
   -rw-r--r--    1 root     root            33 Nov  2 10:15 container-8.txt
   -rw-r--r--    1 root     root          1212 Nov  2 10:15 container-9.txt
   

Example: sharing data with named volumes (pt. 3)

Accessing the volume from the host

3. Alternatively, let’s find where the volume is stored on the host

   • docker volume inspect my-volume

4. Let’s analyse the JSON description of the volume:

   [
       {
           "CreatedAt": "2023-11-02T11:13:56+01:00",
           "Driver": "local",
           "Labels": null,
           "Mountpoint": "/var/lib/docker/volumes/my-volume/_data",
           "Name": "my-volume",
           "Options": null,
           "Scope": "local"
       }
   ]
   

Example: sharing data with named volumes (pt. 4)

Accessing the volume from the host

5. The Mountpoint entry reveals the location of the volume on the host’s file system

6. Let’s look into that position (may require super-user access rights, Linux only):

   • sudo ls -la /var/lib/docker/volumes/my-volume/_data

     totale 48
     drwxr-xr-x 2 root root 4096  2 nov 11.15 .
     drwx-----x 3 root root 4096  2 nov 11.13 ..
     -rw-r--r-- 1 root root  349  2 nov 11.15 container-10.txt
     -rw-r--r-- 1 root root  608  2 nov 11.14 container-1.txt
     -rw-r--r-- 1 root root  369  2 nov 11.14 container-2.txt
     -rw-r--r-- 1 root root  304  2 nov 11.14 container-3.txt
     -rw-r--r-- 1 root root  240  2 nov 11.14 container-4.txt
     -rw-r--r-- 1 root root  434  2 nov 11.14 container-5.txt
     -rw-r--r-- 1 root root  150  2 nov 11.14 container-6.txt
     -rw-r--r-- 1 root root   41  2 nov 11.14 container-7.txt
     -rw-r--r-- 1 root root   33  2 nov 11.15 container-8.txt
     -rw-r--r-- 1 root root 1212  2 nov 11.15 container-9.txt
     

How to reclaim space from a volume?

Volumes are NOT automatically cleaned up when containers are deleted

• Volumes must be explicitly deleted by the user

  • docker volume rm <volume name>

• In our example, remember to delete my-volume:

  • docker volume rm my-volume
    • beware: this will delete all the data in the volume!

Sudo-powered containers (pt. 1)

• Sometimes one may want to let a container access the Docker daemon of its host

  • this is useful for running containers that will: create/delete other containers, handle images, etc.

• To achieve that one may exploit bind mounts

  • to share the Docker daemon’s socket with the container

• Explanation:

  • on the host, the Docker daemon is just a service listening for commands on a Unix socket
  • the socket is usually reified as a file on the host’s file system, at /var/run/docker.sock
  • the docker command is just a CLI program writing/reading to/from the socket to govern the daemon

• One may simply create a Docker container with the Docker CLI tool installed on it

  • and share the host’s Docker socket with that container via a bind mount
    • docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock --name sudo docker:cli sh

• Inside that container one may run the docker command as usual, to govern the host’s Docker daemon

  • e.g. docker ps

  CONTAINER ID   IMAGE        COMMAND                  CREATED          STATUS          PORTS      NAMES
  3d9016cd067e   docker:cli   "docker-entrypoint.s…"   5 seconds ago    Up 3 seconds               sudo
  

Sudo-powered containers (pt. 2)

• Containers created from within sudo are siblings of sudo on the host, not nested inside it
  • e.g. bind mounts are resolved against the host’s file system, not against sudo’s one
• Beware: whoever controls the socket controls the host (same as being in the docker group)

Networks

Docker may virtualise container’s networking facilities

1. network interfaces
2. IP addresses
3. layer-4 ports

This is useful to let containers communicate with any external entity
most commonly, other containers or the host

Sorts of networks (pt. 1)

Different sorts of networks are available in Docker, depending on the driver:

Sorts of networks (pt. 2)

Available drivers:

• none: no networking facilities are provided to the container, which is then isolated

• host: remove network isolation between the container and the Docker host, and use the host’s networking directly

• bridge (default): a virtual network, internal to the host, letting multiple containers communicate with each other

• overlay: a virtual network, spanning multiple hosts, letting multiple containers communicate with each other among different nodes of the same cluster

Other drivers are available as well, cf. https://docs.docker.com/engine/network/drivers/

None network driver

• No networking facilities are provided to the container at all, which is then isolated w.r.t. the external world

• This is useful for containers that do not need to communicate via the network

  • i.e. they expose no service, and they must not access the Internet
    • very rare situation

• Example: docker run --network none -it --rm alpine sh

  / # ping 8.8.8.8
  PING 8.8.8.8 (8.8.8.8): 56 data bytes
  ping: sendto: Network unreachable
  

Host driver

• No isolation between the container and the host

  • the container uses the host’s networking facilities directly

• The main consequence is that the container will be assigned the same IP address as the host

  • the layer-4 ports used by the container are also busy for the host
    • high collision probability

• Example (Linux hosts; Docker Desktop supports it only in recent versions, as an opt-in): docker run --network host my-service

  Service 540c3b7bb7860c6a listening on <your pc's hostname>:8080
  

  • try now to start a service on port 8080 on your host
    • you should get an error message stating that port 8080 is already in use

Bridge network (pt. 1)

• Virtual network spanning one or more containers on the same host

  • containers attached to the same network can communicate with each other via TCP/IP
  • each container may be attached to multiple networks
    • e.g. one for internal communication, one for external communication

• Each container gets assigned a private IP address on the network

  • commonly in the range 172.x.y.z
  • the IP address is contactable from the host (Linux only: on Docker Desktop, containers live in a VM)

Bridge network (pt. 2)

To exemplify the usage of IP:

1. let’s create a container using the my-service image created a few slides ago

   • docker run --network bridge -d --rm --name my-container my-service
   • this should be a service listening on port 8080
   • we call the container my-container
   • specifying --network bridge is useless, because that’s the default

2. let’s inspect the container’s IP address

   • docker inspect my-container | grep IPAddress
     • at the time of writing, the IP address is 172.17.0.2

3. open your browser and browse to http://172.17.0.2:8080 (on Linux)

   • you should see a page of the form: [$SERVER_ID@$HOST_NAME:PORT] Hit $VIEW_COUNT times
   • the page is served by the container, contacted by IP
   • so containers attached to bridge-like networks can be contacted by IP, from the host

4. try to contact the container:

   • by host name, from the host
     • this should not work $\Rightarrow$ no host-specific DNS resolution for containers
   • by IP, from another host (in the same LAN of the host)
     • this should not work $\Rightarrow$ bridge networks are host specific

Bridge network (pt. 3)

Let’s create a non-trivial scenario with 2 containers attached to the same network:

1. one container (host name: dind) will run the Docker daemon (Docker-in-Docker)
2. the other container (host name: cli) will run the Docker CLI
3. let’s call the shared network my-network

Bridge network (pt. 4)

1. let’s create the network: docker network create --driver bridge my-network

   • a random string is returned: that’s the network’s ID

2. let’s create the daemon:

   • docker run --privileged -d --rm --network my-network --hostname dind docker:dind

3. let’s create the client: docker run -it --rm --network my-network --hostname cli docker:cli

   1. inside cli, let’s check if dind is contactable by hostname: ping dind

      • this should work $\Rightarrow$ networks come with their own name resolution support

   2. inside cli, let’s check if the Docker client works: docker ps

      error during connect: Get "http://docker:2375/v1.24/containers/json": dial tcp: lookup docker on 127.0.0.11:53: no such host
      

   3. apparently the docker:cli image is preconfigured to contact a daemon on the docker hostname

4. let’s stop dind:

   • you can use docker ps on the host to get the name of the dind container
   • then docker stop <container name>
   • or docker stop $(docker ps -q --filter ancestor=docker:dind)
     • -q (quiet), only display container IDs
     • --filter ancestor=X selects containers whose image is X
   • next time let’s give an explicit name to the container via --name

Bridge network (pt. 5)

5. let’s restart a Docker-in-Docker daemon, using the docker hostname:

   • docker run --privileged -d --rm --network my-network --name dind --hostname docker docker:dind dockerd --host=tcp://0.0.0.0:2375
   • in general, it would be better to specify both --name X and --hostname X
     • same name and hostname to avoid confusion!
   • dockerd --host=tcp://0.0.0.0:2375 is necessary to force the port the daemon will listen on
     • and to disable TLS security (not recommended in production)

6. let’s restart the client: docker run -it --rm --network my-network --hostname cli docker:cli

7. let’s run a command on the CLI, say docker run hello-world

   • this should work $\Rightarrow$ the client is now able to contact the daemon via the docker hostname

Overlay network

• Virtual network spanning multiple hosts, i.e. the nodes of a Swarm cluster
  • containers attached to it communicate as if they were on the same host, regardless of the node they run on
  • traffic among nodes is encapsulated (VXLAN) over the network the nodes are attached to
• E.g. the agents_network network of the Portainer stack, later in this lecture
  • created via docker network create --driver overlay --attachable my-overlay on a manager, or declared in a stack file

Exposing ports (pt. 1)

• When a container wraps a service listening on layer-4 port…

  • … the port is not accessible from the outside world
  • … unless it is explicitly published when running the container

• When creating a Docker image, the EXPOSE command in the Dockerfile is used to declare which ports the service listens on

  • e.g. EXPOSE 8080
  • this is just documentation (metadata): it does not publish the port

• The port exposed by a container can be inspected via docker ps

  • e.g. docker run -d --rm my-service; docker ps

  CONTAINER ID   IMAGE        COMMAND                  CREATED        STATUS                  PORTS      NAMES
  7a14f3bf95cc   my-service   "/bin/sh -c 'npm run…"   1 second ago   Up Less than a second   8080/tcp   wonderful_goldwasser
  

  (look at the PORTS column)

Exposing ports (pt. 2)

• When running a Docker container, the exposed port can be mapped to the host’s ports via the -P option

  • this would make any EXPOSEd port in the image mapped to some random port of the host
  • e.g. docker run -d --rm -P my-service; docker ps
  • to discover the actual port mapping, run docker ps

  CONTAINER ID   IMAGE        COMMAND                  CREATED         STATUS         PORTS                                         NAMES
  b1470831b62b   my-service   "/bin/sh -c 'npm run…"   6 seconds ago   Up 5 seconds   0.0.0.0:32773->8080/tcp, :::32773->8080/tcp   sad_benz
  

• One may control the mapping of ports via the -p <host port>:<guest port> option

  • e.g. docker run -d --rm -p 8888:8080 my-service; docker ps
  • this would map the container’s port 8080 to the host’s port 8888

  CONTAINER ID   IMAGE        COMMAND                  CREATED         STATUS         PORTS                                       NAMES
  f034da93c476   my-service   "/bin/sh -c 'npm run…"   4 seconds ago   Up 3 seconds   0.0.0.0:8888->8080/tcp, :::8888->8080/tcp   bold_booth
  

Exposing ports (pt. 3)

• EXPOSE only documents the port: it is publishing (-P or -p) that makes it reachable from outside the host
• -p also accepts an IP to bind to, e.g. -p 127.0.0.1:8888:8080 publishes the port on the loopback interface only

Orchestration

With Docker Compose

Do we need orchestration?

Operating containers manually is error prone and low-level

• Docker Compose lets you increase the abstraction level on the single machine

• Starting from a user-provided specification of a stack…

  • i.e. a set of containers, their images, their environment, their dependencies

• … Docker Compose can:

  • orderly start the containers up
  • while creating all required networks and volumes
  • and gracefully shut them down upon need

• The only thing the user must do is create a compose.yaml file

  • a YAML file describing the stack
  • following the official specification
  • compose.yaml is the preferred file name nowadays, docker-compose.yml is still supported
  • the top-level version attribute is obsolete, hence omitted in what follows

Docker Compose syntax by example (pt. 1)

The example application is composed of the following parts:

• 2 services, backed by Docker images: webapp and database
• 1 secret (HTTPS certificate), injected into the frontend
• 1 secret (DB credentials), injected into both services
• 1 configuration (HTTP), injected into the frontend
• 1 persistent volume, attached to the backend
• 2 networks

Docker Compose syntax by example (pt. 2)

services:                       # section defining services composing the stack
  frontend:                     # name of the first service (frontend)
    image: example/webapp       # image to use for the first service
    depends_on:                 # section for dependencies
       - backend                # short-form depends_on waits for start, not for health
    environment:                # section for environment variables
      SERVICE_PORT: 8043        # port the service will listen on
      DB_HOST: backend          # hostname of the backend service (= service name)
      DB_PORT: 3306             # port the backend service will listen on
    ports:                      # section for ports published on the host
      - "443:8043"              # container port 8043 as host port 443
    networks:                   # section for networks to be attached to
      - front-tier              # attach to front-tier network (by name)
      - back-tier               # attach to back-tier network (by name)
    # configs are files copied into the container, with no need to build a new image
    configs:                    # section for config files to be injected
      - source: httpd-config    # reference to the configuration file (by name)
        target: /etc/httpd.conf # where the config is mounted (read-only)
    # secrets are reified as read-only files in /run/secrets/<secret_name>
    secrets:                    # section for secrets to be injected
      - server-certificate      # reference to the secret (by name)
      - db-credentials          # reference to the secret (by name)

      # assumption: image example/webapp is programmed to load secrets
      # from /run/secrets/<secret_name>

  backend:                      # name of the second service (backend)
    image: example/database     # image to use for the second service
    # command run inside the container to check whether it is healthy
    healthcheck:
      test: ["CMD", "command", "which", "should", "fail", "if", "db not healthy"]
      interval: 1m30s
      timeout: 10s
      retries: 3
      start_period: 40s
    volumes:                    # section for volumes to be attached to the container
      - db-data:/etc/data       # the volume db-data (reference by name) will be mounted
                                # as read-write in /etc/data
    networks:
      - back-tier               # attach to back-tier network (by name)
    secrets:
      - db-credentials          # reference to the secret (by name)

volumes:                   # section for volumes to be created in the stack
  db-data:                 # name of the volume, to be referenced by services
    driver_opts:
      type: "none"
      o: "bind"            # this volume is a bind mount
      device: "/path/to/db/data/on/the/host"

configs:                   # section for configuration files to be created in the stack
  httpd-config:            # name of the configuration file, to be referenced by services
    file: path/to/http/configuration/file/on/the/host

secrets:                   # section for secrets to be created in the stack
  server-certificate:      # name of the secret, to be referenced by services
    file: path/to/certificate/file/on/the/host
  db-credentials:          # name of the secret, to be referenced by services
    external: true         # not created by the stack, but expected to exist
    name: my-db-creds      # name of the secret, as created by the user

networks:
  # The presence of these objects is sufficient to define them
  front-tier: {}
  back-tier: {}

Orchestration-level notions (pt. 1)

Stacks

A stack is a set of inter-related services, networks, volumes, secrets, configs

• all of which are defined in a single compose.yaml file
• all of which are created / deleted / updated / started / stopped together

Services

From Docker Compose documentation:

• a service is an abstract definition of a computing resource within an application which can be scaled or replaced independently of other components
• they are backed by a set of containers, run by the orchestrator according to replication requirements and placement constraints
• they are backed by containers, hence they are defined by an image and a set of runtime arguments
  • all containers within a service are identically created with these arguments
• they are declared in the services section of the compose.yaml file
  • for each service, attributes may be specified, overriding the default behaviour of the image
  • further attributes may be specified to declare dependencies on other services, volumes, networks, and so on
    • these must be declared in the same compose.yaml file

Orchestration-level notions (pt. 2)

Configs

From Docker Compose documentation:

• configs are files containing configuration data that can be injected into services upon instantiation
• they allow users to configure their services’ behaviour without the need to (re)build a (new) image
• they are read-only files, mounted into services’ containers, similarly to read-only volumes
• they are declared in the configs section of the compose.yaml file

Orchestration-level notions (pt. 3)

Configs

• one config may either be created stack-wise out of a local file from the host…

  configs:
      config_name:
          file: /path/to/config/on/the/host
  

• … or (Swarm mode only) it can be manually created by means of the docker config create [OPTIONS] NAME PATH command

  • in this case they can be referenced in several compose.yaml files:

    configs:
        config_name:
            external: true
            name: "the name used when creating the config"
            # this may be different from config_name
    

• in any case, the config’s mount point should be specified at the service level:

  services:
      my-service:
          ...
          configs:
              - source: config_name # reference to the configs section (by YAML name)
                target: /path/to/config/on/the/container
  

Orchestration-level notions (pt. 4)

Secrets

From Docker Compose documentation:

• secrets are configs containing sensitive data that should be kept secret

• put simply, Docker makes it harder to inspect the content of secrets, both on the host and inside the container

  • in Swarm mode, secrets are encrypted at rest and in transit
  • with plain Docker Compose, they are just bind-mounted files (no encryption)

• one secret may either be created stack-wise out of a local file or an environment variable from the host…

  secrets:
      secret_name:
          file: "/path/to/secret/on/the/host"
  

  secrets:
      secret_name:
          environment: "NAME_OF_THE_VARIABLE_ON_THE_HOST"
  

• … or (Swarm mode only) it can be manually created by means of the docker secret create [OPTIONS] NAME PATH|- command

  • in this case they can be referenced in several compose.yaml files:

    secrets:
        secret_name:
            external: true
            name: "the name used when creating the secret"
            # this may be different from secret_name
    

• in any case, the secret’s mount point should be specified at the service level:

  services:
      my-service:
          ...
          secrets:
              # short syntax, mounted on /run/secrets/<secret_name>
              - secret_name # reference to the secrets section (by YAML name)
              # long syntax
              - source: secret_name # reference to the secrets section (by YAML name)
                target: /path/to/secret/on/the/container
  

Orchestration-level notions (pt. 5)

About secrets and configs

Important remark:

• in the eyes of the container, secrets and configs are just read-only files mounted on some PATH
• the containerised application is responsible for reading the secrets / configs from the mounted PATH
• when using somebody else’s image, you may expect the image to be programmed to read secrets / configs from customisable paths
• when creating your own image, you should make the application able to read secrets / configs from customisable paths
  • e.g. via environment variables

Orchestration-level notions (pt. 6)

Volumes

From Docker Compose documentation:

• volumes in compose are no different from the ones we have already discussed
• they are declared in the volumes section of the compose.yaml file
• one volume may be created stack-wise out of a local folder from the host…

  volumes:
      volume_name:
          driver: local
          driver_opts:
              type: "none"
              o: "bind"
              device: "/path/to/folder/on/the/host"
  

• … or maybe some remote folder exposed by some NFS service on the network

  volumes:
      volume_name:
          driver: local
          driver_opts:
              type: "nfs"
              # meaning of options: https://wiki.archlinux.org/title/NFS#Mount_using_/etc/fstab
              o: "addr=NFS_SERVER_ADDRESS,nolock,soft,rw" # cf. nfs
              device: ":/path/on/the/nfs/server/"
  

Orchestration-level notions (pt. 7)

Volumes (cont’d)

• … or it can be manually created by means of the docker volume create [OPTIONS] NAME command

  • in this case they can be referenced in several compose.yaml files:

    volumes:
        volume_name:
            external: true
            name: "the name used when creating the volume"
            # this may be different from volume_name
    

• in any case, the volume’s mount point should be specified at the service level:

  services:
      my-service:
          ...
          volumes:
              - type: volume
                source: volume_name # reference to the volumes section (by YAML name)
                target: /path/to/folder/on/the/container
              # alternatively, one may define a service-specific bind mount on the fly
              - type: bind
                source: /path/to/folder/on/the/host
                target: /path/to/folder/on/the/container
  

Orchestration-level notions (pt. 8)

Networks

From Docker Compose documentation:

• networks in compose are no different from the ones we have already discussed

• they are declared in the networks section of the compose.yaml file

• one network may be declared stack-wise in the stack file…

  networks:
      network_name:
          driver: overlay # default choice in swarm
          driver_opts:
              # many options, covering many advanced use cases for network engineering
  

• … or it can be manually created by means of the docker network create [OPTIONS] NAME command

  • in this case they can be referenced in several compose.yaml files:

    networks:
        network_name:
            external: true
            name: "the name used when creating the network"
            # this may be different from network_name
    

• in any case, the networks a service is attached to should be specified at the service level:

  services:
      my-service:
          ...
          networks:
              - network_name # reference to the networks section (by YAML name)
  

Docker Compose commands

• See all possibilities with docker compose --help

• docker compose up starts the stack

  • if the stack is not present, it is created
  • if the stack is already present, it is updated
  • if the stack is already present, and it is up-to-date, nothing happens
  • interesting optional flags (see all with --help)
    • -d (detached mode): run containers in the background
    • --wait waits for services to be running or healthy (implies detached mode)

• docker compose down stops the stack

  • if the stack is not present, nothing happens
  • if the stack is present, its containers are stopped and removed, along with its networks
  • interesting optional flags (see all with --help)
    • --remove-orphans removes containers for services not defined in the Compose file (e.g. residuals from previous versions)
    • -v removes named volumes declared in the “volumes” section of the Compose file and anonymous volumes attached to containers

All commands assume a compose.yaml (or docker-compose.yml) file is present in the current working directory

Docker Compose commands: what is created and removed

• Names are prefixed by the project name, i.e. the name of the directory containing compose.yaml (e.g. your-dir)
  • can be overridden via -p <project name> or the top-level name attribute

Working example (pt. 1)

services:
  db:
    image: mariadb:latest
    volumes:
      - db_data:/var/lib/mysql
    restart: always
    networks:
      - back-tier
    environment:
      MYSQL_ROOT_PASSWORD: "..Password1.."
      MYSQL_DATABASE: wordpress_db
      MYSQL_USER: wordpress_user
      MYSQL_PASSWORD: ",,Password2,,"
    healthcheck:
      test: [ "CMD", "healthcheck.sh", "--su-mysql", "--connect", "--innodb_initialized" ]
      start_period: 1m
      start_interval: 10s
      interval: 1m
      timeout: 5s
      retries: 3

  wordpress:
    depends_on:
      db:
        condition: service_healthy # wait for the healthcheck to pass
    image: wordpress:latest
    volumes:
      - wordpress_data:/var/www/html
    ports:
      - "8000:80"
    restart: always
    networks:
      - back-tier
    environment:
      WORDPRESS_DB_HOST: db:3306
      WORDPRESS_DB_USER: wordpress_user
      WORDPRESS_DB_PASSWORD: ",,Password2,,"
      WORDPRESS_DB_NAME: wordpress_db
volumes:
  db_data: {}
  wordpress_data: {}
networks:
  back-tier: {}

Working example (pt. 2)

• Example of docker compose up logs:

  docker compose up
  [+] Building 0.0s (0/0) docker:default
  [+] Running 5/5
  ✔ Network your-dir_back-tier        Created  0.1s
  ✔ Volume "your-dir_wordpress_data"  Created  0.1s
  ✔ Volume "your-dir_db_data"         Created  0.0s
  ✔ Container your-dir-db-1           Created  0.8s
  ✔ Container your-dir-wordpress-1    Created  0.9s
  Attaching to db-1, wordpress-1
  db-1         | 2023-11-23 14:24:57+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.2.2+maria~ubu2204 started.
  db-1         | 2023-11-23 14:24:57+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
  db-1         | 2023-11-23 14:24:57+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.2.2+maria~ubu2204 started.
  db-1         | 2023-11-23 14:24:57+00:00 [Note] [Entrypoint]: Initializing database files
  wordpress-1  | WordPress not found in /var/www/html - copying now...
  wordpress-1  | Complete! WordPress has been successfully copied to /var/www/html
  wordpress-1  | No 'wp-config.php' found in /var/www/html, but 'WORDPRESS_...' variables supplied; copying 'wp-config-docker.php' (WORDPRESS_DB_HOST WORDPRESS_DB_NAME WORDPRESS_DB_PASSWORD WORDPRESS_DB_USER)
  wordpress-1  | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.22.0.3. Set the 'ServerName' directive globally to suppress this message
  wordpress-1  | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.22.0.3. Set the 'ServerName' directive globally to suppress this message
  wordpress-1  | [Thu Nov 23 14:24:58.289485 2023] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.56 (Debian) PHP/8.0.30 configured -- resuming normal operations
  wordpress-1  | [Thu Nov 23 14:24:58.289516 2023] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'
  ...
  

  • notice that the outputs of the two containers are interleaved

• Example of docker ps logs:

  CONTAINER ID   IMAGE              COMMAND                  CREATED         STATUS                   PORTS                                   NAMES
  4691a9641135   wordpress:latest   "docker-entrypoint.s…"   9 minutes ago   Up 9 minutes             0.0.0.0:8000->80/tcp, :::8000->80/tcp   your-dir-wordpress-1
  314feb9d42ab   mariadb:latest     "docker-entrypoint.s…"   9 minutes ago   Up 9 minutes (healthy)   3306/tcp                                your-dir-db-1
  

• Example of docker network ls logs:

  NETWORK ID     NAME                 DRIVER    SCOPE
  b304d1ae5404   bridge               bridge    local
  1bb39315ff98   host                 host      local
  03c8700dee6a   none                 null      local
  470d81d26296   your-dir_back-tier   bridge    local
  

Working example (pt. 3)

services:
  db:
    ...
    environment:
      ...
      MYSQL_ROOT_PASSWORD_FILE: /run/secrets/db_root_password
      MYSQL_PASSWORD_FILE: /run/secrets/db_password
    ...
    secrets:
      - db_root_password
      - db_password

  wordpress:
    ...
    environment:
      ...
      WORDPRESS_DB_PASSWORD_FILE: /run/secrets/db_password
    ...
    secrets:
      - db_password

volumes:
  db_data: {}
  wordpress_data: {}

networks:
  back-tier: {}

secrets:
  db_password:
    file: db_password.txt
  db_root_password:
    file: db_root_password.txt

Docker Compose for testing

• Docker Compose is also useful for software testing

• E.g. whenever the system under test relies on some infrastructural component

  • e.g. a database, a message broker, a cache, etc.

• Usual workflow for testing:

  1. start the component
  2. run the system under test
  3. stop the component

• Docker Compose may be coupled with some test framework of choice to automate the process

Docker Compose for testing: the workflow

Exercise: Docker Compose for testing + JUnit (pt. 1)

Domain model

interface Customer {
    + id: CustomerID
    + firstName: String
    + lastName: String
    + birthDate: LocalDate
    ...
    + name: String
    + isPerson: Boolean
    + isCompany: Boolean
}
note bottom: Entity

interface CustomerID {
    + value: Any
}
note right: Value Object

interface TaxCode {
    + value: String
}
note left: Value Object

interface VatNumber {
    + value: Long
}
note right: Value Object

VatNumber -d-|> CustomerID
TaxCode -d-|> CustomerID

Customer *-r- CustomerID

interface CustomerRepository {
    + findById(id: CustomerID): Iterable<Customer>
    + findByName(name: String): Iterable<Customer>
    + add(customer: Customer)
    + update(oldId: CustomerID, customer: Customer)
    + remove(id: CustomerID)
}
note left: Repository

CustomerRepository o-r- Customer

Exercise: Docker Compose for testing + JUnit (pt. 2)

Implementation details

interface CustomerRepository {
    + findById(id: CustomerID): Iterable<Customer>
    + findByName(name: String): Iterable<Customer>
    + add(customer: Customer)
    + update(oldId: CustomerID, customer: Customer)
    + remove(id: CustomerID)
}

class SqlCustomerRepository {
    - connections: ConnectionFactory
    - tableName: String
    + createTable(replaceIfPresent: Boolean = false)
}

SqlCustomerRepository --|> CustomerRepository

package "db" {
    interface ConnectionFactory {
        + connect(): Connection
    }
    interface MariaDB {
        + database: String
        + username: String
        + password: String
        + host: String
        + port: Int
    }
    class MariaDBConnection
    MariaDBConnection --|> MariaDB
    MariaDBConnectionFactory --|> ConnectionFactory
    MariaDBConnectionFactory ..> MariaDBConnection: creates
}

SqlCustomerRepository .r.> MariaDBConnectionFactory: uses

package "java sql" {
    interface Connection
    ConnectionFactory ..> Connection: creates
    MariaDBConnection --|> Connection
}

Exercise: Docker Compose for testing + JUnit (pt. 3)

class TestMariaDBCustomerRepository {

    companion object {
        @BeforeClass @JvmStatic fun setUpMariaDb() { TODO("compose up") }
        @AfterClass @JvmStatic fun tearDownMariaDb() { TODO("compose down") }
    }

    private val connectionFactory = MariaDBConnectionFactory(DATABASE, USER, PASSWORD, HOST, PORT)
    private lateinit var repository: SqlCustomerRepository

    @Before
    fun createFreshTable() {
        repository = SqlCustomerRepository(connectionFactory, TABLE)
        repository.createTable(replaceIfPresent = true)
    }

    private val taxCode = TaxCode("CTTGNN92D07D468M")
    private val person = Customer.person(taxCode, "Giovanni", "Ciatto", LocalDate.of(1992, 4, 7))
    private val person2 = person.clone(
        birthDate = LocalDate.of(1992, 4, 8),
        id = TaxCode("CTTGNN92D08D468M"),
    )
    private val vatNumber = VatNumber(12345678987L)
    private val company = Customer.company(vatNumber, "ACME", "Inc.", LocalDate.of(1920, 1, 1))

    @Test
    fun complexTestWhichShouldActuallyBeDecomposedInSmallerTests() {
        assertEquals(
            expected = emptyList(),
            actual = repository.findById(taxCode),
        )
        assertEquals(
            expected = emptyList(),
            actual = repository.findById(vatNumber),
        )
        repository.add(person)
        repository.add(company)
        assertEquals(
            expected = listOf(person),
            actual = repository.findById(taxCode),
        )
        assertEquals(
            expected = listOf(company),
            actual = repository.findById(vatNumber),
        )
        repository.remove(vatNumber)
        repository.update(taxCode, person2)
    }
}

Exercise: Docker Compose for testing + JUnit (expected workflow)

Running experiments in parallel

With Docker Compose

Experiments as jobs

Recall: jobs eventually terminate, and may exploit parallelism to speed up the computation

• Experiments (simulations, data analyses, ML trainings, benchmarks…) are typical jobs

  • input: data + parameters $\rightarrow$ output: results (metrics, models, plots, logs)
  • often run as parameter sweeps (e.g. grid search, multiple seeds): many independent runs
  • independent runs are trivially parallelisable

• Why containerise them?

  • reproducibility: same image $\Rightarrow$ same OS, runtime, and pinned dependencies, everywhere
  • automation: one command (re-)runs all experiments, possibly in CI
  • scalability: the same image runs in parallel on one machine (Compose) or on a cluster (Swarm)

Running example: https://github.com/unibo-spe/containerized-experiment, whose commit history we follow step by step,
from a Jupyter notebook to parallel, containerised experiments

The running example, in plain words

No background in machine learning (ML) is needed: what matters is the shape of the computation

• Dataset: a table of examples, here Iris: 150 flowers, 4 measurements each, plus their species (the label)
• Model: an algorithm that learns from examples to predict the label of new ones
  • e.g. a decision tree, a neural network (MLP), a logistic regression, a support vector machine (SVM)
• Hyper-parameters: the knobs of a model (e.g. the maximum depth of a tree), to be chosen before training
• Grid search: train a model for every combination of knob values, keep the best one
  • quality is measured as accuracy (% of correct predictions) on examples not used for training

For us, an experiment is: input data + parameters $\rightarrow$ many independent trainings $\rightarrow$ results (scores, plots, the best model)

Where experiments are born: Jupyter notebooks

The bad situation: experiments in notebooks

What is great for exploration is bad for reproducibility and automation:

• Hidden state: cells can be run in any order, and re-run, so the results depend on the click history
  • “Restart kernel and run all” may give different results, or fail
• Hard-coded parameters: trying another configuration means editing the code, then re-running by hand
• Implicit environment: which Python? which library versions? whatever was installed on the author’s machine
• Outputs mixed with code: results live inside the .ipynb (with paths like /home/gciatto/...), or in the current directory
• No automation: a notebook needs a human (and a browser) to be run, so no CI, no remote / batch execution
• Poor version control: diffs of a JSON with embedded images are hardly readable
• Sequential: all configurations run one after the other, in the same process

Goal: turn the notebook into an experiment which anybody can (re-)run, in parallel, with one command

The plan: one commit per step

Inspect each step via git checkout <commit> in the repository:

Step 1: from notebook to script

DATASET_ID = int(os.getenv('DATASET_ID', "61"))
TEST_SIZE = float(os.getenv('TEST_SIZE', "0.2"))
RANDOM_STATE = int(os.getenv('RANDOM_STATE', "42"))
K_FOLDS = int(os.getenv('K_FOLDS', "5"))
SCORING = os.getenv('SCORING', 'accuracy')
PICTURE_SIZE = tuple(map(int, os.getenv('PICTURE_SIZE', '10,6').split(',')))
OUTPUT_DIR = Path(os.getenv('OUTPUT_DIR', '.'))
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)

Step 2: dockerise

ARG PYTHON_VERSION=3.12
FROM python:${PYTHON_VERSION}
RUN mkdir -p /experiment
VOLUME "/data"
ENV OUTPUT_DIR=/data
WORKDIR /experiment
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY * /experiment/
CMD python experiment.py | tee ${OUTPUT_DIR}/output.log

services:
  experiment:
    build:
      context: .
      dockerfile: Dockerfile
      args:
        PYTHON_VERSION: 3.12
    volumes:
      - ./data:/data
    environment:
      # DATASET_ID: 61 # openml dataset id
      # TEST_SIZE: 0.2
      # RANDOM_STATE: 42
      # K_FOLDS: 5
      # SCORING: accuracy
      # PICTURE_SIZE: 10, 6

Step 3: who owns the results?

Recall: files written by containers via bind mounts are owned by root on the host

ENV OWNER=1000:1000
CMD python experiment.py | tee ${OUTPUT_DIR}/output.log && \
    chown -R ${OWNER} ${OUTPUT_DIR}

    # run as the host user, so that results in ./data are owned by them (cf. README)
    user: &user "${HOST_UID:-1000}:${HOST_GID:-1000}"

Step 4: do not run the same experiment twice

• Parameter sweeps take long, and get interrupted: re-running docker compose up should only run the missing experiments
• Results are saved before the ID file: a run is considered done only if it completed

Step 4: do not run the same experiment twice (code)

# Compute a unique experiment ID based on the hyper-parameters
FOOTPRINT_KEYS = {'DATASET_ID', 'TEST_SIZE', 'RANDOM_STATE', 'K_FOLDS', 'SCORING', 'PICTURE_SIZE', 'MODELS'}
EXPERIMENT_FOOTPRINT = {k: v for k, v in locals().items() if k in FOOTPRINT_KEYS}
EXPERIMENT_FOOTPRINT_YAML = yaml.dump(EXPERIMENT_FOOTPRINT, sort_keys=True)
EXPERIMENT_ID = sha512(EXPERIMENT_FOOTPRINT_YAML.encode()).hexdigest()

# Ensure the output directory exists
DATA_DIR = Path(os.getenv('DATA_DIR', './data'))
OUTPUT_DIR = Path(os.getenv('OUTPUT_DIR', str(DATA_DIR)))
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)

# Do not waste time if the experiment has already been run
# by looking for the existence of a .yaml file named after the EXPERIMENT_ID
for file in DATA_DIR.glob('**/*.yaml'):
    if file.is_file() and file.stem == EXPERIMENT_ID:
        print(f'Experiment with ID {EXPERIMENT_ID[:8]} already exists in {file.parent}', file=sys.stderr)
        exit(0)

• The footprint is the set of all parameters affecting the results
  • forgetting one (e.g. the dataset) means two different experiments get the same ID
• yaml.dump(..., sort_keys=True) makes the footprint canonical: same parameters $\Rightarrow$ same ID

Step 5: parameters out of the code

      MODEL_DecisionTree: |
          sklearn.tree.DecisionTreeClassifier: 
              max_depth: [null, 10, 20]
              min_samples_split: [2, 10, 20]
      MODEL_MLP: |
          sklearn.neural_network.MLPClassifier: 
              hidden_layer_sizes: 
                - [50]
                - [100,]
                - [50, 50]
              activation: ['tanh', 'relu']
              solver: ['adam', 'sgd']
      MODEL_LogisticRegression: |
          sklearn.linear_model.LogisticRegression: 
              C: [0.1, 1, 10]
              solver: ['liblinear', 'lbfgs']
      MODEL_SVM: |
          sklearn.svm.SVC: 
              C: [0.1, 1, 10]
              kernel: ['linear', 'rbf']

Step 6: one service per configuration (pt. 1)

services:
  experiment-dt:
    build: &experiment_build
      context: .
      dockerfile: Dockerfile
      args:
        PYTHON_VERSION: "3.14"
    volumes: &volumes
      - ./data:/data
    # run as the host user, so that results in ./data are owned by them (cf. README)
    user: &user "${HOST_UID:-1000}:${HOST_GID:-1000}"
    # cap the resources of each experiment, so that parallel ones do not starve each other
    deploy: &deploy
      resources:
        limits:
          cpus: "1"
          memory: 2G
    hostname: dt
    environment:
      MODEL_DecisionTree: |
          sklearn.tree.DecisionTreeClassifier: 
              max_depth: [null, 10, 20]
              min_samples_split: [2, 10, 20]

  experiment-mlp:
    build: *experiment_build
    volumes: *volumes
    user: *user
    deploy: *deploy
    hostname: mlp
    environment:
      MODEL_MLP: |
          sklearn.neural_network.MLPClassifier: 
              hidden_layer_sizes: 
                - [50]
                - [100,]
                - [50, 50]
              activation: ['tanh', 'relu']

Step 6: one service per configuration (pt. 2)

• docker compose up: builds the image (once), then runs all experiments in parallel
• docker compose up experiment-svm: runs just one of them
• docker compose logs -f experiment-mlp: follows the output of one experiment

Step 7: polishing

ARG PYTHON_VERSION=3.14
FROM python:${PYTHON_VERSION}
ENV DATA_DIR=/data
VOLUME "/data"
WORKDIR /experiment
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY experiment.py run.sh ./
# The container is meant to run as the host user (cf. compose.yaml),
# who has no home directory in the image: let libraries (e.g. openml,
# matplotlib) put their caches and configs in a writable place
ENV HOME=/tmp
CMD ["bash", "run.sh"]

#!/usr/bin/env bash
# Runs the experiment, storing its outputs in a fresh sub-directory of $DATA_DIR
set -euo pipefail
export OUTPUT_DIR="$DATA_DIR/$(date +%Y-%m-%d-%H-%M-%S)-$(hostname)"
mkdir -p "$OUTPUT_DIR"
python experiment.py | tee "$OUTPUT_DIR/output.log"
# If the experiment was skipped (as already executed), there is nothing worth keeping
if ! compgen -G "$OUTPUT_DIR/*.yaml" > /dev/null; then
    rm -r "$OUTPUT_DIR"
fi

Step 7: not overloading the machine

    # cap the resources of each experiment, so that parallel ones do not starve each other
    deploy: &deploy
      resources:
        limits:
          cpus: "1"
          memory: 2G

Scaling out: experiments on a Swarm

services:
  experiment-svm:
    image: <username>/containerized-experiment:1.0.0
    hostname: svm
    environment:
      MODEL_SVM: |
        sklearn.svm.SVC:
          C: [0.1, 1, 10]
          kernel: ['linear', 'rbf']
    volumes:
      - results:/data
    deploy:
      mode: replicated-job   # run to completion
      replicas: 1
      restart_policy:
        condition: none
      resources:
        limits: { cpus: "1", memory: 2G }
  # ... other experiments ...
volumes:
  results:
    driver: local
    driver_opts:
      type: "nfs"
      o: "addr=NFS_SERVER_ADDRESS,rw"
      device: ":/path/to/results/"

Exercise: containerise your own experiment

Repository: https://github.com/unibo-spe/containerized-experiment

1. Select some experiment of your choice

   • e.g. from a course you attended, from your thesis, or from a published paper
   • better if it is a Jupyter notebook

2. Containerise it, in such a way that:

   • only scripts are present (no notebooks)
   • the experiments can be run via Docker Compose
   • the results are saved in a folder on the host machine, owned by the host user
   • the parameters of the experiments are set via environment variables
   • experiments already executed for a given assignment of parameters are skipped
   • different assignments of parameters run in parallel, each with capped resources

3. Bonus: build the image and run the experiments in CI (cf. Continuous Integration)

Orchestration

With Docker Swarm

Building the cluster (theory)

• In a Docker Swarm cluster, there are two kinds of nodes:
  • manager nodes, which coordinate the cluster by means of the Raft consensus protocol
    • they may also act as worker nodes
  • worker nodes, which simply run containers

• To set up a cluster, one must:
  1. initialise Swarm mode on one node of choice
     • that will become the first manager node
  2. optionally, let other nodes join the cluster as worker nodes
  3. optionally, elevate some worker nodes to manager nodes
     • this operation can only be done by a client connected to some manager node
     • it is better to have multiple manager nodes for redundancy
     • it is better to have an odd number of manager nodes
       • e.g. 3, 5, 7, etc.

Swarm architecture

• Docker Compose deploys a stack on a single host, Docker Swarm deploys it on a cluster of hosts (nodes)
• Users submit stacks to managers (e.g. via docker stack deploy), which keep the cluster state consistent via Raft
• Managers turn services into tasks (i.e. containers), and dispatch them onto nodes, workers or managers alike

A local Swarm via Docker-in-Docker

Disclaimer: in this lecture, the Swarm cluster is made of Docker-in-Docker (DinD) containers, all running on your own machine

• Each node is a docker:dind container, i.e. a full Docker daemon, with its own images, containers, volumes, and networks

  • nodes see each other via a Compose network, and they are reachable by symbolic hostname (master, worker-1, …)

• Why? DinD is the smoother way to obtain a reproducible and stable didactical environment:

  • same cluster for everybody, regardless of OS (Linux, Mac, Windows) and of Docker flavour (Engine, Desktop)
  • no firewalls, VMs, or port forwarding in the way (cf. troubleshooting for real machines)
  • no need for several machines, nor for a working LAN in the classroom
  • disposable: the cluster is created and destroyed with one command, your own Docker daemon is left untouched

• Caveats:

  • nodes are privileged containers $\Rightarrow$ fine for a playground, not for production
  • all nodes share the same hardware $\Rightarrow$ no real scalability or fault-tolerance (but the semantics is the same)
  • each node has its own image cache $\Rightarrow$ images are pulled once per node

A local Swarm via Docker-in-Docker (pt. 2)

• Your local daemon runs the nodes as containers, each node’s daemon runs the tasks of the Swarm
• Your CLI talks to master (the only manager) via DOCKER_HOST, your browser reaches the Swarm’s services via gateway

A local Swarm via Docker-in-Docker (pt. 3)

x-node: &node
  image: docker:dind
  privileged: true
  environment:
    DOCKER_TLS_CERTDIR: "" # plain TCP on port 2375

services:
  master:
    <<: *node
    hostname: master
    ports:
      - "127.0.0.1:2375:2375" # Docker API
  worker-1:
    <<: *node
    hostname: worker-1
  worker-2: ...                # same as worker-1
  worker-3: ...                # same as worker-1
  gateway:
    image: alpine/socat
    entrypoint: ["sh", "-c"]
    command:
      - |
        for port in 9443 8889 8890; do
          socat TCP-LISTEN:$$port,fork,reuseaddr \
                TCP:master:$$port &
        done
        wait
    ports:
      - "9443:9443"           # Portainer
      - "8889:8889"           # later examples
      - "8890:8890"

A local Swarm via Docker-in-Docker (pt. 4)

1. Save the file of the previous slide as compose.yaml, in an empty directory, and start the nodes:

   docker compose up -d
   

   • wait a few seconds for the daemons to start (docker compose logs master should say API listen on [::]:2375)

2. Initialise Swarm mode on master, and let the workers join the cluster:

   docker compose exec master docker swarm init
   TOKEN=$(docker compose exec master docker swarm join-token -q worker)
   for i in 1 2 3; do
       docker compose exec worker-$i docker swarm join --token $TOKEN master:2377
   done
   

3. Let your Docker CLI connect to master (rather than to your local daemon), in the current shell:

   • export DOCKER_HOST=tcp://localhost:2375 (Bash) or $env:DOCKER_HOST="tcp://localhost:2375" (PowerShell)
   • from now on, all docker commands (e.g. docker node ls, docker stack deploy) are executed by master:

     ID                            HOSTNAME   STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
     h7kxqxjgxqnr5bc4zp4pxfsgc *   master     Ready     Active         Leader           29.9.2
     9ig5gfqmiwslf637gzidl9y6d     worker-1   Ready     Active                          29.9.2
     mlipjs2czcoapauecp1haswlf     worker-2   Ready     Active                          29.9.2
     7or20sft5peegyexrb1m9yuax     worker-3   Ready     Active                          29.9.2
     

4. Whenever done, destroy the cluster: unset DOCKER_HOST (or Remove-Item Env:DOCKER_HOST), then docker compose down -v

Building the cluster on real machines

Not needed for this lecture (cf. DinD Swarm), but this is how a real cluster is built

1. On one node running the Docker daemon (say, the teacher’s machine), initialise Swarm mode as follows:

   • docker swarm init
   • this will output a command to be run on other nodes to join the cluster:

     Swarm initialized: current node (<NODE_ID>) is now a manager.
     
     To add a worker to this swarm, run the following command:
     
         docker swarm join --token <SECRET_TOKEN> <NODE_ADDRESS>:2377
     
     To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
     

2. On other nodes running the Docker daemon (say students’ machines), initialise Swarm mode as follows:

   • docker swarm join --token <SECRET_TOKEN> <NODE_ADDRESS>:2377
     • where SECRET_TOKEN and NODE_ADDRESS are the ones provided by the docker swarm init command above

3. On the manager node, one may inspect the node composition of the cluster via docker node ls (sample output)

   ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
   4q3q3q3q3q3q3q3q3q3q3q3q3q *  node-1              Ready               Active              Leader              24.0.7
   5q4q4q4q4q4q4q4q4q4q4q4q4q    node-2              Ready               Active                                  24.0.7
   6q5q5q5q5q5q5q5q5q5q5q5q5q    node-3              Ready               Active                                  24.0.7
   

4. The manager node may promote other nodes to manager nodes via docker node promote <NODE_ID>…

   • and it may demote manager nodes to worker nodes via docker node demote <NODE_ID>
   • it also may remove nodes from the cluster via docker node rm <NODE_ID>

5. Whenever done, one may leave the Swarm via docker swarm leave --force

   • remember to do this, otherwise you won’t be able to use Swarm elsewhere!

Building the cluster (troubleshooting)

Things that may go wrong

1. There may be a firewall blocking the communication among nodes

   • e.g. Windows Defender, the lab’s firewall, your home router’s firewall, etc.
   • you may notice the presence of a firewall by the fact that docker swarm join hangs or fails with a timeout
     • in the absence of a firewall, the command should succeed or fail almost immediately
   • troubleshooting:
     1. identify which and how many firewalls are in place
     2. ensure that the firewall allows communication for Docker Swarm ports
        • commonly: 2377/tcp, 7946/tcp, 7946/udp, 4789/udp

2. You use Docker on Windows or Mac, hence the Docker daemon runs on a virtual machine

   • in this case the IP of the virtual machine is different from the IP of your machine
   • the NODE_ADDRESS in docker swarm join --token <SECRET_TOKEN> <NODE_ADDRESS>:2377 returned by docker swarm init is the IP of the virtual machine
   • troubleshooting:
     1. find the actual IP of your machine (e.g. ipconfig on Windows, ifconfig on Mac), let’s call it ACTUAL_IP
     2. configure your host to redirect Swarm traffic received by ACTUAL_IP towards NODE_ADDRESS
        • for all ports listed above (2377, 7946, 4789)
     3. let other nodes use ACTUAL_IP instead of NODE_ADDRESS in docker swarm join when joining the cluster

Swarm on Docker Desktop

• The address printed by docker swarm init (NODE_ADDRESS) belongs to the Docker Desktop VM, which is not reachable from the LAN
• Other nodes must join via the PC’s address (ACTUAL_IP), and the PC must forward Swarm traffic to the VM (cf. next slide)
• Forwarding via netsh is partial (TCP only): prefer Linux machines as Swarm nodes, or Docker Engine in WSL2 (cf. “Troubleshooting example on Windows (pt. 2)”)

Troubleshooting example on Windows (pt. 1)

Opening ports on Windows Defender

On an administrator Powershell, run the following commands:

netsh advfirewall firewall add rule name="Docker Swarm Intra-manager" dir=in action=allow protocol=TCP localport=2377
netsh advfirewall firewall add rule name="Docker Swarm Overlay Network Discovery TCP" dir=in action=allow protocol=TCP localport=7946
netsh advfirewall firewall add rule name="Docker Swarm Overlay Network Discovery UDP" dir=in action=allow protocol=UDP localport=7946
netsh advfirewall firewall add rule name="Docker Swarm Overlay Network Traffic" dir=in action=allow protocol=UDP localport=4789

• this operation should be done on all nodes of the cluster running on Windows

Forwarding ports on Windows

On an administrator Powershell, run the following commands:

netsh interface portproxy add v4tov4 listenport=2377 listenaddress=$ACTUAL_IP connectport=2377 connectaddress=$NODE_ADDRESS
netsh interface portproxy add v4tov4 listenport=7946 listenaddress=$ACTUAL_IP connectport=7946 connectaddress=$NODE_ADDRESS

• portproxy only forwards TCP: UDP traffic (7946/udp gossip, 4789/udp overlay data) cannot be forwarded this way (cf. next slide)

• this operation should be done on the very first manager node of the cluster, if it’s running on Windows

Troubleshooting example on Windows (pt. 2)

Forwarding UDP traffic too: Docker Engine in WSL2, mirrored networking

netsh cannot forward UDP: rather than forwarding, let the engine use the PC’s own network interfaces

1. Enable mirrored networking for WSL2 (Windows 11 22H2 or later), in %UserProfile%\.wslconfig:

   [wsl2]
   networkingMode=mirrored
   

   then restart WSL via wsl --shutdown

2. Allow inbound traffic towards WSL2 through the Hyper-V firewall (administrator PowerShell):

   Set-NetFirewallHyperVVMSetting -Name '{40E0AC32-46A5-438A-A0B2-2B479E8F2E90}' -DefaultInboundAction Allow
   

3. Install Docker Engine inside a WSL2 distribution (e.g. Ubuntu), and use it instead of Docker Desktop

   • Docker Desktop runs its engine in its own VM, which does not see the mirrored interfaces

4. From that distribution, use the PC’s address when creating / joining the Swarm, for both TCP and UDP ports:

   • docker swarm init --advertise-addr $ACTUAL_IP, or
   • docker swarm join --advertise-addr $ACTUAL_IP --token <SECRET_TOKEN> <MANAGER_ADDRESS>:2377

On macOS, the analogous solution is a Linux VM with a bridged network adapter, running Docker Engine

Stacks on Swarms

• One may deploy a stack on a Swarm cluster via docker stack deploy -c <COMPOSE_FILE_PATH> <STACK_NAME>

  • where COMPOSE_FILE_PATH is the path to a .yml file following the Docker Compose syntax
  • and STACK_NAME is the name of the stack to be deployed

• The command must be run on a client connected to some manager node

  • try to run it on a worker node and see what happens
  • try to run it on a manager node when half or more of the manager nodes are down (i.e. no quorum) and see what happens

• Other operations one may do on stacks (via a manager node):

  • docker stack ls to list stacks
  • docker stack ps <STACK_NAME> to list containers in a stack
  • docker stack services <STACK_NAME> to list services in a stack
  • docker stack rm <STACK_NAME> to remove a stack

• Semantics of stack deployment on Swarms is different from Docker Compose:

  • different services may be allocated on different nodes
  • services may be replicated on different nodes
  • networks must rely on the overlay driver, to support inter-node communication among containers
  • some Compose attributes are ignored (e.g. depends_on, restart, build), others only work in Swarm (e.g. deploy)

Stack Deploy Example: Portainer

services:
  agent:
    image: portainer/agent:lts
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/docker/volumes:/var/lib/docker/volumes
    networks:
      - agents_network
    deploy:
      mode: global # i.e., one (and only one) replica per node

  webservice:
    image: portainer/portainer-ce:lts
    command: -H tcp://tasks.agent:9001 --tlsskipverify
    ports:
      - "9443:9443"
    volumes:
      - portainer_data:/data
    networks:
      - agents_network
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.role == manager

networks:
  agents_network:
    driver: overlay
    attachable: true

volumes:
  portainer_data:

Portainer’s allocation onto the Swarm

• deploy.mode: global $\rightarrow$ one agent per node, placement.constraints $\rightarrow$ webservice on the only manager (master)
• Volumes are created where the containers mounting them run, overlay networks span all the nodes hosting their containers

Portainer’s Initial Configuration

• let’s use software-process-engineering as the password for the admin user

• the setup token is printed in the logs of the dashboard:

  docker service logs portainer_webservice | grep setup_token
  

• this page must be filled within 5 minutes from the deployment, otherwise Portainer locks itself

  • if so, restart it via docker service update --force portainer_webservice

• upon creation of the user, Portainer proposes to set up Edge Compute: let’s just skip it

Portainer’s Home Page

Upon login, you are presented with the home page, which asks you to select one environment

• environments are clusters (one Portainer instance may manage multiple clusters)

Portainer’s Dashboard

Upon environment selection, one is presented with the dashboard, showing the status of the cluster

Things to notice via Portainer

After selecting the only environment available, one may observe:

• which and how many nodes compose the cluster (Dashboard section)

  • you may use the cluster visualiser to spot master, worker-1, worker-2, and worker-3

• which stacks are running on the cluster (Services section)

  • you may observe that the portainer stack is running

• which services are running on the cluster (Services section)

  • you may observe that the portainer stack is composed of two services: agent and webservice

• which containers are running on the cluster (Containers section)

  • you may observe that the portainer_agent service is composed of one container per node (i.e. 4 containers)
  • you may observe that the portainer_webservice service is composed of one container which is running on master (the only manager)

• which volumes are present on the cluster (Volumes section)

  • you may observe that the portainer_portainer_data volume is present on the same node of portainer_webservice, i.e. master

• which networks are present on the cluster (Networks section)

  • you may observe that the portainer_agents_network network is present, using the overlay driver

Labelling nodes

Nodes may be labelled with key=value pairs

• Labels may be used to mark nodes with respect to their properties

  • e.g. hardware / software capabilities
    • e.g. capabilities.cpu=yes
    • e.g. capabilities.web=yes
    • e.g. capabilities.storage=yes

• This can be done programmatically….

  • e.g. via docker node update --label-add KEY=VALUE NODE (where NODE is the node’s ID or hostname)
  • e.g. let’s mark all workers of the DinD Swarm as CPU-capable, for the next examples:

    for i in 1 2 3; do docker node update --label-add capabilities.cpu=yes worker-$i; done
    

• … or via Portainer:

  

• labels can be used to set placement constraints or preferences for services:

  • placement constraints: one service can only be deployed onto nodes with given labels
  • placement preferences: one service’s replicas will be spread among nodes with given labels

Example of placement constraints

Full stack: wordpress-swarm.yaml (continued in the right column).

version: "3.8"
services:
  db:
    image: mariadb:latest
    volumes:
      - db_data:/var/lib/mysql
    networks:
      - back-tier
    environment:
      MYSQL_ROOT_PASSWORD: "..Password1.."
      MYSQL_DATABASE: wordpress_db
      MYSQL_USER: wordpress_user
      MYSQL_PASSWORD: ",,Password2,,"
    healthcheck:
      test: ["CMD", "healthcheck.sh",
        "--su-mysql", "--connect",
        "--innodb_initialized"]
      start_period: 1m
      interval: 1m
      timeout: 5s
      retries: 3
    deploy:
      restart_policy:
        condition: any
      placement:
        constraints:
          - node.labels.capabilities.storage==yes

  wordpress:
    image: wordpress:latest
    volumes:
      - wordpress_data:/var/www/html
    ports:
      - "8889:80"
    networks:
      - back-tier
    environment:
      WORDPRESS_DB_HOST: db:3306
      WORDPRESS_DB_USER: wordpress_user
      WORDPRESS_DB_PASSWORD: ",,Password2,,"
      WORDPRESS_DB_NAME: wordpress_db
    deploy:
      restart_policy:
        condition: any
      placement:
        constraints:
          - node.labels.capabilities.web==yes
volumes:
  db_data: {}
  wordpress_data: {}
networks:
  back-tier:
    driver: overlay

Example of placement constraints (pt. 2)

Exposing ports on the Swarm

When a service exposes a port, and that’s deployed on a Swarm:

• the service is exposed on all nodes in the Swarm (managers and workers)

  • this is the so-called routing mesh
  • requests reaching any node are forwarded to some node actually hosting a replica

• so, the wordpress service above could be visited via any node of the cluster, e.g.:

  • http://master:8889 (i.e. http://localhost:8889 on your machine, via gateway)
  • http://worker-1:8889
  • http://worker-2:8889 (the node actually hosting the service)

Service replication (pt. 1)

• Services may be replicated for the sake of scalability / load-balancing / fault-tolerance, etc.

• Two types of services:

  • global: replicated exactly once per node
  • replicated: replicated N times (with N=1 by default)

• General case:

  services:
    service-name:
      ...
      deploy:
        mode: replicated
        replicas: N                  # this should be a number!
        placement:
          constraints:
            - node.role == manager
            - node.labels.mylabel==myvalue
          preferences:
            - spread: node.labels.mylabel
          max_replicas_per_node: M      # this should be a number!
  

• How to read it:

  • N replicas of the service will be deployed
  • each one will be deployed on a manager node having the mylabel label set to myvalue
  • replicas will be spread among the admissible nodes, as evenly as possible
  • no more than M replicas will be put on the same node

Service replication (pt. 2)

• global services follow the cluster: when a node joins, it gets a replica; when it leaves, its replica is gone
• replicated services keep exactly N replicas alive, re-scheduling them when a node fails (if constraints allow)

Example of placement preferences

services:
  ws:
    image: pikalab/hitcounter:latest
    ports:
      - "8890:8080"
    deploy:
      replicas: 10
      placement:
        constraints:
          - node.labels.capabilities.cpu==yes
        preferences:
          - spread: node.labels.capabilities.cpu

Docker Swarm and Volumes

services:
  simulation:
    image: pikalab/fakesim:latest
    deploy:
      restart_policy:
        condition: on-failure # restart is ignored by Swarm
      replicas: 3
      placement:
        constraints:
          - node.labels.capabilities.cpu==yes
        max_replicas_per_node: 1
    volumes:
      - type: volume
        source: shared_volume
        target: /data
volumes:
    shared_volume: {}

Sharing data among nodes

Lecture is Over

Compiled on: 2026-11-07 — printable version

back to ToC