CLCI Gitlab CI

Gitlab CI is the default CI solution for projects hosted on Gitlab. While in-depth knowledge of how Gitlab CI works is not needed to get very basic pipelines set up using this project, it can be useful when attempting to build more complicated pipelines. See https://docs.gitlab.com/ee/ci/ for more documentation. The repository for this project is located at https://gitlab.common-lisp.net/clci/gitlab-ci.

Gitlab CI ultimately requires one or more YAML files that describe the jobs to perform. This project contains the following files:

Currently, this project only works on Linux runners (and may Mac runners if you configure them correctly). However, first-class Windows and MacOS support is on the roadmap. This support is likely to arrive because this project is an offshoot of another (private) project that does have Windows and MacOS support. We mostly just need to get some runners for testing.

Quickstart

This section describes how to set up Gitlab CI for a CL-based project using these helpers. Gitlab CI has a dizzying amount of flexibility, so this quickstart will quickly get you something that works and will be sufficient for many use cases. But you may want to dive deeper into this project or Gitlab CI at some point to perform more advanced tasks.

Prep

This section describes setup that you must perform other than modifying your .gitlab-ci.yml file.

Testing

In order to use the test pipeline, you must provide a method to run your tests (this project does not attempt to guess how they should be run). You must provide either a Makefile target to run your tests or write a script for the CI helpers to run.

Create a file called scripts/ci-test.lisp (alternatively, set $CLCI_TEST_SCRIPT to point to another file). This file will be loaded (with cl:load) into a CL process that has ASDF available (and your chosen dependency manager, if applicable). This file must either enter the debugger (by signaling an uncaught error) or exit with a non-zero exit code (e.g., (uiop:quit 1)) if your tests fail.

For most projects, the contents of this file should be as simple as:

(asdf:test-system "my-cool-system")

NOTE: If you have a Makefile in your repo, the helpers will instead run make test (after setting some environment variables). If you wish to disable this behavior, set $CLCI_TEST_USE_MAKEFILE to no.

Release

In order to use the release pipeline, you should create a CHANGELOG.md file in your repo. The first level heading (line starting with a single #) should be your project name. The second level headings (starting with ##) should start with the version number and the contents should describe the changes to your project. See the CHANGELOG.md file in this repo for an example.

Additionally, you must protect your release tags. Go to "Settings > Repository > Protected Tags" and protect v*.

Then, when you're ready to make a release, push a tag of the form v$VERSION_NUMBER.

Update CLPM bundle

If you are using CLPM for your dependency management, you can use these helpers to periodically update your lock file for you.

To do this, you should create a bot for your project that can open merge requests. Go to "Settings > Access Tokens" and create a token with write_repository and api permissions. Then create a CI variable ("Settings > CI/CD > Variables") called CLCI_MR_TOKEN, whose value is the token you got from the first step.

Last, you should create a scheduled job to check for updates to your dependencies. Go to "CI/CD > Schedules". Create a new scheduled job for your default branch that sets PIPELINE_TYPE to clpm-dep-update.

.gitlab-ci.yml

Copy the following to .gitlab-ci.yml in your repo. Be sure to read the comments, as they give some hints on customizations that may be appropriate for you.

include:
  project: 'clci/gitlab-ci'
  ref: v2-stable
  file:
    - guarded-linux-test-pipeline.gitlab-ci.yml
    - guarded-release-pipeline.gitlab-ci.yml
    # Uncomment if you are using CLPM and want to automatically update your
    # bundle.
    #
    # - guarded-clpm-dep-update-pipeline.gitlab-ci.yml

variables:
  # Uncomment if you want to use Quicklisp as your dependency manager during
  # tests.
  #
  # CLCI_DEPENDENCY_MANAGER: "quicklisp"
  #
  # Uncomment if you have Git submodules that you want the runner to
  # automatically init and update for you. Submodules are sometimes used by
  # projects to bundle their test dependencies.
  #
  # GIT_SUBMODULE_STRATEGY: recursive
  #
  # Uncomment these lines if you want to test against Allegro, you have read
  # the Allegro express license
  # <https://franz.com/ftp/pub/legal/ACL-Express-20170301.pdf>, *and* your use
  # of Allegro Express does not violate the license. Alternatively, uncomment
  # and provide your own Docker image (or runner) that has Allegro installed
  # with your actual license.
  #
  # CLCI_TEST_ALLEGRO: "yes"
  # I_AGREE_TO_ALLEGRO_EXPRESS_LICENSE: "yes"


# This section is not strictly required, but prevents Gitlab CI from launching
# multiple redundent pipelines when a Merge Request is opened.
workflow:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    - if: '$CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS'
      when: never
    - if: '$CI_COMMIT_BRANCH'
    - if: '$CI_COMMIT_TAG'

FAQs

This project is still very new, so I can't really say that these questions are frequently asked, but I imagine they would be :).

How do I install a non-CL dependency before running tests?

First, make sure your dependency isn't already installed. The default Linux Docker images are based on the buildpack-deps image, which includes a lot of commonly used libraries.

If you really need to install something before your tests run, add the following to your .gitlab-ci.yml:

.common:test:clci:
  before_script:
    - apt-get update
    - apt-get install -y foo bar baz

How do I use CLPM as my dependency manager?

Just commit your clpmfile.lock (in the root of your repo) without setting $CLCI_DEPENDENCY_MANAGER and the helpers will take care of the rest.

How do I test using dependencies from Ultralisp?

Set $CLCI_DEPENDENCY_MANAGER to quicklisp. Then set $CLCI_QL_DIST_NAME to ultralisp.

How do I test against both Quicklisp and Ultralisp?

Add the following to your .gitlab-ci.yml:

.common:test:clci:
  parallel:
    matrix:
      - CLCI_QL_DIST_NAME: [quicklisp, ultralisp]

How do I test against both the ASDF bundled with the implementation and the latest released ASDF?

Add the following to your .gitlab-ci.yml:

.common:test:clci:
  parallel:
    matrix:
      - CLCI_ASDF_VERSION: [REQUIRE, latest]

How do I generate documentation (or any other CL-based task)?

Create a CL file in your repo that does what you want and enters the debugger or exits with a non-zero status if it fails. Then add a job in your .gitlab-ci.yml:

generate docs:
  extends:
    - .clci sbcl
    - .clci DEPENDENCY_MANAGER script
  variables:
    CLCI_SCRIPT: path/to/script.lisp
  artifacts:
    paths:
      - doc/

Where DEPENDENCY_MANAGER is replaced with asdf (no dependency manager), clpm, or quicklisp. In the future, a more flexible job may be added that chooses the correct dependency manager based on the value of $CLCI_DEPENDENCY_MANAGER, but that will require a non-trivial amount of refactoring to support.

Your script will then be loaded into a CL process that has ASDF and your dependency manager (if applicable) available.

Detailed Documentation

More detailed documentation for these helpers can be found at: