Continuous Integration and Continuous Deployment

For general information on GitLab CI/CD consult:

https://docs.gitlab.com/ee/ci/
https://collaborating.tuhh.de/help/ci/quick_start/index.md
Join the TUHH-Mattermost Channel GitLab in Forschung und Lehre (de/en)

Variables

General Variables

A lot of variables are used by default in the runner environment provided by GitLab:

https://docs.gitlab.com/ee/ci/variables/

They usually come with the prefix CI_ and are used in the .gitlab-ci.yml, e.g.

  rules:
    - if: $CI_PROJECT_URL != 'https://collaborating.tuhh.de/cfg0846/research-topics-mat-tuhh'
      when: never
    - if: '$CI_COMMIT_REF_NAME != "master"'

Project CI/CD-Variables

The repository stores different CI/CD variables necessary for authentication during ssh-services: https://docs.gitlab.com/ee/ci/variables/#project-cicd-variables

Depending on their domain, they carry different prefixes:

REVIEW_ for the review app hosted on www3-dev.tuhh.de
SSH_ for the review app hosted on www3.tuhh.de
TUHH_ for the deployment on the institute server ssh.mat.tu-harburg.de

Sensible variables can only be accessed for jobs running on the protected branches master and dev. Depending on their suffix, they are used at certain parts of the corresponding jobs in the pipeline.

`*_{HOST,PORT,PRIVATE_KEY,USER}`

These variables are used e.g. in ssh or rsync commands like

echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
#...
rsync -v -r -a --delete -e  "ssh -p${SSH_PORT}" build-www3/* ${SSH_USER}@${HOST_NAME}:/usr/local/apache2/htdocs/.

`*_{KNOWN_HOSTS}`

This variable stores the output of

ssh-keyscan -p $PORT $NAME_OF_THE_HOST

If, e.g. the hostkey for ssh.mat.tu-harburg.de changes, update this info by copying the output of

ssh-keyscan -p 20 ssh.mat.tu-harburg.de

to the runner variable $TUHH_KNOWN_HOSTS.

Pipeline

Depending on the trigger event, the GitLab pipeline decides which jobs are going to be executed. Some of them perform tests, others build artifacts that can be downloaded or viewed in the review app.

Stages

Each job belongs to a certain stage. Jobs at later stages need the jobs of prior stages to finish successfully. Jobs at the same stage are run in parallel.

Default

default:
  before_script:
    - RESEARCH_ROOT=`pwd`; export RESEARCH_ROOT;

This job includes a before_script for each job that does not have an own before_script. Here, a local environment variable is set.

Overview

Overall the pipeline features 5 Stages:

The first two stages are testing stages:

docker-update: checks, whether the docker images from the repository's container registry are available and rebuilds them if the Dockerfiles have changed
testing: performs various tests of the preprocessor and building routines
build: constructs the various products like standalone webpages, and TUHH-delivery packages
review: starts the review apps for the merge request
deploy: ships the products to their final destination (www3, www3-dev, mat)

`docker-update`

Here, all Docker images are updated.

`testing`

The testing stage consists of the following functionality tests: Each time a bash-script with prefix test_ is executed. All scripts reside in /bin and exit with exit code 1 if something goes wrong. In case of a failing test, subsequent states of the pipeline stall. Check the code documentation provided by the scripts for further information.

`build`

Here, the different products are constructed. Most of them rely on the docker image pandoc-bash from the Dockerfile in /docker/pandoc-bash.Dockerfile. The image is stored as

collaborating.tuhh.de:5005/cfg0846/research-topics-mat-tuhh/pandoc-bash

in the container registry but can also be retrieved from Docker Hub under the name eltenedor/pandoc-bash.

The main artifacts of this stage come from the jobs

build-www3-dev (runs only on side branches and deploys to the review app)
build-www3 (runs only on dev and deploys to https://www3.tuhh.de/cfg0846/research-topics-mat-tuhh/
build-tuhh (runs only on master)

which run the script build_webpage.sh but with different options. See .gitlab-ci.yml for details.

The artifacts of the build-tuhh job are created only for runs of the master branch and can be downloaded from the badge

`review`

Runs the review apps

review_tuhh (environment url on www3-dev)

`deploy`

Runs the jobs that deliver the webpage to the production sites. Currently, the following deployment jobs run in the pipeline:

rsync-tuhh-deploy: sends the outcome of build-tuhh to (just for testing purposes)

$TUHH_USER@$TUHH_HOST:$TUHH_DEPLOY/

dev-deploy: sends outcome of build-www3 to www3.tuhh.de (only for dev)