Skip to main content

How to build Linux packages with PackageCore and Travis-CI

packagecore logo

We recently put up the GitHub repository PackageCore, which is the core functionality of our BytePackager service. PackageCore is a python library for automating the building of Linux packages as part of your development process. It works by spinning up Docker containers, generating the required .spec, DEBIAN/control, PKGBUILD, etc. files, and compiling and building packages for your software on each distribution. It also automates running a series of tests on the package after its been built on a fresh container.

PackageCore

Below I'll show you how to setup a GitHub project with Travis-CI to use PackageCore to generate Linux packages every time a new tag is pushed.

The easiest way to install PackageCore is to use pip and install its non-python pre-requisites ahead of time (I use Arch Linux for local development).

sudo pacman -Syy python3 python3-yaml python3-setuptools docker

And then pull down the latest PackageCore version from PyPI.

sudo pip install packagecore

MatrixInspector

The project is MatrixInspector, tool I created for viewing an manipulating sparse matrices. It's written in C++, uses wxWidgets for GUI compontents, and CMake as a build system.

I like presenting the standard ./configure && make build steps, so I include a configure script with MatrixInpsector that creates build, runs CMake, and generates a Makefile which essentially just calls make -C build $@.

The Configuration File

The packagecore.yaml file contains all the information required to create Linux packages of your software. The first part of the configuration file contains the basic information about the package. Only the name is required, but if the other fields are omitted, most package manager will generate warnings.

name: matrixinspector
maintainer: Dominique LaSalle <dominique@solidlake.com>
license: MIT
summary: Utility for viewing and modifying sparse matrices.
homepage: https://github.com/dlasalle/matrixinspector

The next set of fields specify the project wide commands need to compile and install MatrixInspector.

commands:
  compile: |
    ./configure --prefix=/usr && make
  install: |
    make install DESTDIR="${BP_DESTDIR}"

Setting the commands to compile are straight forward. For the install commands, we use the environment variable ${BP_DESTDIR} as the target root directory. That is, if we want to place a binary in /usr/bin/matrixinspector on systems that install package, we should place a binary at ${BP_DESTDIR}/usr/bin/matrixinspector during the install step.

The packages directive specifies all of the distributions for which to build packages. For each distribution, we also specify what the build dependencies are and what the runtime dependencies are. Build dependencies are install in the Docker container before the compile and install commands are executed. The runtime dependencies are the dependencies of the package itself, and will be installed on a user's system by their package management system. My packages directive for MatrixInspector are as follows:

packages:
  archlinux:
    builddeps:
      - cmake
      - gcc
      - wxgtk
      - glu
    deps:
      - wxgtk
      - glu
  fedora25:
    builddeps:
      - cmake
      - gcc-c++
      - wxGTK3
      - wxGTK3-devel
      - freeglut
      - freeglut-devel
    deps:
      - wxGTK3
      - freeglut
  ubuntu17.10:
    builddeps:
      - cmake
      - g++
      - libwxgtk3.0-0v5
      - libwxgtk3.0-dev
      - freeglut3
      - freeglut3-dev
    deps:
      - libwxgtk3.0-0v5
      - freeglut3

We specify CMake, g++, and wxgtk as the build time dependencies, and just wxgtk as the runtime dependency.

Travis-CI

My initial .travis.yml file looks like:

dist: trusty
sudo: required
language: cpp
compiler:
  - gcc
before_install:
  - sudo apt-get -qqy update
  - sudo apt-get install -qy libwxgtk3.0-0
  - sudo apt-get install -qy libwxgtk3.0-dev
script:
  - ./configure --test && make && make test

It's pretty simple, just installs the needed dependencies (wxWidgets), configures, builds, and runs unit tests.

We need to to make sure we have Docker on our build machine, so add it as a service in our .travis.yaml file.

services:
  - docker

We then need to modify the before_deploy and deploy stages of Travis-CI. In the before_deploy stage, we want to install PackageCore and its pre-requisites, and generate the packages via PackageCore.

before_deploy:
  - sudo apt-get install -qy python3 python3-pip
  - python3 -m pip install packagecore
  - packagecore -o dist/ "${TRAVIS_TAG#v}"

Then, in the deploy stage, we set the provider as releases, which means the build artifacts (the packages) will be uploaded to GitHub as release assets. I generated an encrypted API key for access to Github using the Travis-CI CLI.

deploy:
  provider: releases
  api_key:
    secure: ${GITHUB_ENCRYPTED_TOKEN}
  file_glob: true
  file:
    - dist/*
  skip_cleanup: true
  on:
    tags: true

The file_glob: true specifies we're using wildcards in our file list. Our file: statement says we want to upload everything in the dist directory. We skip the cleanup stage, as we do not want our artifacts removed before they are uploaded. Finally, the on-tags directive tells Travis-CI to only run the before_deploy and deploy steps when a new tag is merged to the repository.

Kick off a build

You can then kick off a build of the packages either by pushing a new tag via git (e.g., git tag -am v0.1.1 v0.1.1 && git push --tags) or by creating a new release via GitHub's web interface.

travis status

Once the Travis-CI build finishes, you should see your finished packages on your GitHub page.

github releases