Continous Integration (CI) and Continuous Delivery (CD) help developers to automate workflows based on code changes. The benefits of using CI/CD include:
- Reduced failure risk through standardized release processes
- Faster rollout times due to automated processes
- Transparency over failed releases
“A phased approach to continuous delivery is not only preferable, it’s infinitely more manageable.” – Maurice Kherlakian
This article aims to dive into CI/CD workflows with GitHub Actions and Go from the start over common use cases up to building a custom dependency bot.
Potential applications of CI/CD might be:
- Automatic application releases
- Code testing and validation
- Generation of documentation
GitHub Actions are an efficient way of running CI/CD pipelines for code hosted on GitHub. GitHub Actions are free to use with the standard executors in public repositories allowing people to try them out without having to pay money. While GitHub Actions offer a lot of great functionality, it is quite cumbersome to set them up to solve specific use cases.
The approach of CI/CD can also be realized with other tools such as Gitlab CI/CD, CircleCI, Travis CI or Jenkins.
The GitHub Actions page is the gate to getting started with Continuous Integration (CI) and Continuous Delivery (CD) based on GitHub repositories.
Contents
- Setup
- Getting started with GitHub Actions
- Running builds and tests
- Analyzing code quality with a linter
- Building and pushing Docker images
- Creating pull requests for dependency updates
- Integrating private repositories as Go dependencies
- Triggering actions only when specific code changes
- Creating status badges
- Summary
Setup
We first create a new GitHub repository. To get started with the CI, we have to create a folder named .github/workflows at the root of our repository. We can also use the GitHub GUI directly.
There we can add YAML configuration files that each define one workflow. One workflow is a concrete GitHub Action defining the handling of specific events such as pushes, pull requests, or releases.
├── .github/ │ ├── workflows/ │ │ └── build.yml │ │ └── lint.yml │ │ └── publish.yml │ │ └── test.yml
The listed configuration files already give an impression of tasks that may be helpful for efficient development processes. Linting analyzes source code for potential issues. While successful linting does not ensure that the code does not contain any bugs, together with software tests it is an established way to reduce the risk of potentially unexpected behavior.
Software deployments usually require a software build, that for instance is to create executable files or containers. We then will have to publish the generated artifacts in such a way that they can be deployed right directly.
For now these workflows are just meant to be placeholders. We will discover how to define them in the later sections.
Getting started with GitHub Actions
Each workflow is performed on a specific event. For instance, we can trigger automated builds on commits to the main branch, update our Docker images on a new release or run code validation such as unit tests on creating pull requests.
name: Demo
on:
push:
branches:
- master
- main
The Demo workflow as defined above shows the configuration to perform workflows on pushes to either the master or the main branch (GitHub changed the default branch from master to main in the past). Development processes often include a stable branch named master or main where new features get integrated from feature branches. Feature branches define individual features and get integrated to the stable branch once the functionality is validated and tested. Usually the integration process would involve code reviews while automated steps such as linting and testing already reduce the risk of breaking the stable code base.
After the workflows are triggered, we want to perform some specific functionality. GitHub has listed many publicly available actions on their website. Feel free to have a look at https://github.com/actions. As CI/CD is a very widely used approached, the amount of pre-defined actions is continuously growing.
In addition, we can define custom workflows and seek for inspiration in blog posts or developer communities such as StackOverflow.
Running builds and tests
We first start with a workflow to automatically run builds and tests for Go code. Go is an open source programming language developed by Google that gained a lot of popularity during the past years as it is very efficient and easy to use and can be utilized for a large variety of tools. However, we want to emphasize that these workflows can also be defined for most other programming languages.
As any other programming language, Go has multiple currently supported versions. In this case we want to ensure that our code successfully runs on Go versions 1.13, 1.14 and 1.15, but you can basically include any version you need. Together with a target platform (ubuntu-latest in this case), we define a matrix strategy. This strategy says that the workflow is run once for every defined settings, so we once run the tests on a Go 1.13 ubuntu system, once on Go 1.14 and once on Go 1.15. In the job definition we then can refer to these defined settings by accessing ${{ matrix.<parameter> }}.
name: Build & Test
on:
push:
branches:
- master
- main
jobs:
test:
strategy:
matrix:
go-version: [1.13.x, 1.14.x, 1.15.x]
platform: [ubuntu-latest] #[ubuntu-latest, macos-latest, windows-latest]
runs-on: ${{ matrix.platform }}
steps:
- name: Install Go
uses: actions/setup-go@v2
with:
go-version: ${{ matrix.go-version }}
- name: Checkout code
uses: actions/checkout@v2
- name: Build
run: go build src/*.go
- name: Test
run: go test ./...
As shown in the example above, we here leverage the two out-of-the-box provided actions setup-go to setup go on the target platform and checkout to make the entire code of the repository available.
We then create an executable binary file using go build and start running the tests with go test. These are just some of the commands that come with Go. To read more about such commands, have a look at https://pkg.go.dev/cmd/go.
package main
// Add adds two integers and returns the value
func Add(x int, y int) int {
return x + y
}
// Sub subtracts two integers and returns the value
func Sub(x int, y int) int {
return x - y
}
To check if our workflow works correctly, we can use the small script above to add and subtract numbers together with some small tests. The tests are not shown here, please find them in the public GitHub repository together with the other sources of this article.
Go has a recommendable documentation where you can read how to get started with using the language (https://go.dev/learn/) or, as it is also relevant for this setting, how to implement software tests (https://go.dev/doc/tutorial/add-a-test).
In case you are not used to Go’s syntax, what we do here is to define a function Add and a function Sub in a package named main. As both functions start with a capital letter, they will be exported through the package. In programming slang, these are often referred to as public functions. Each of the functions takes two integer values x and y as parameters and return a single integer value.
After pushing the code, we then can directly check in the Web UI if the script builds correctly and the given tests pass. We observe that three jobs were run and the builds and tests completed successfully.
The GitHub Actions interface shows run workflows and directly indicates if they passed.
Analyzing code quality with a linter
Continuing with a linter, we can identify bad coding practices such as unused variables or unhandled errors. This allows the developers to keep the code clean but also to identify potential misbehaviors or bugs. In general, linting is a practice we recommend to use for every software development project independent of the settings and used programming languages.
name: Lint
on:
push:
branches:
- master
- main
pull_request:
jobs:
golangci:
name: lint
runs-on: ubuntu-latest
steps:
- name: Install Go
uses: actions/setup-go@v2
- name: Checkout Code
uses: actions/checkout@v2
- name: golangci-lint
uses: golangci/golangci-lint-action@v2
with:
working-directory: src/
args: --skip-files="(.*?)_test.go"
To do so, we use the predefined action for the golangci linter that provides an out-of-the-box solution to automatically run a linter. All we have to provide manually here are the settings. We can, for instance, specify a working directory or define which files should be excluded. Usually a developer would like to integrate all the code that is being used for the application, however, there may be trusted external code sources that potentially would not match the linting standards.
Also, it generally is tolerable to exclude the test files from the linting. This is not a bad practice, as you want to establish the quality of the production code, not necessarily the code quality of the tests. Still, we would recommend not to exclude anything on purpose unless it is really needed (not including being lazy :)). In order to learn more about the settings of the golangci-lint, read more on their documentation at https://golangci-lint.run/.
Note that in this setting we specify a regular expression that will exclude any file ending on '_test.go'. We can specify files here directly too if it is appropriate for the use case. To read more about using these settings, we again recommend to read the official documentation.
As we now can use our linter in our CI/CD pipeline, we want to have a code example that indicates the benefits of using a linter. While the code example above would not indicate any issues as it is just a minimalistic code example, we can add a bad practice on purpose.
package main
// unused variable
var z int
// Add adds two integers and returns the value
func Add(x int, y int) int {
return x + y
}
// Sub subtracts two integers and returns the value
func Sub(x int, y int) int {
return x - y
}
To do so, we changed the script slightly. Here we introduce a global integer variable z that is not used in any of the functions. While this script builds fine with an added main function, we can see that there may be a bad coding smell. While this is quite obvious in such a minimalistic setting, we often experience such bad smells in larger code bases. So let’s see what our linter tells us.
To check the result, we push the code to our repository and open the Actions tab on our GitHub repository and select the lint workflow.
The result as shown below on the one hand tells us that the linting workflow failed, but on the other hand also tells us which issue led to the failing workflow. We observe that the variable z is unused. In case of more than just one error, all of them will be listed at this page.
The GitHub Actions page shows failed workflows as well as the underlying issues.
Building and pushing Docker images
“Docker is a platform designed to help developers build, share, and run modern applications.” – docker.com.
As a developer, one might want to ensure that the application runs and functions on any target environment. To do so, Docker provides the option to build containers as an abstraction from the target environment where we can specify all the needed requirements.
Defining Docker images
First, we need to specify how to build the Docker images. To do so, we create a Dockerfile that loads the code, builds it, and then runs the main executable. We can test it locally using docker build and docker run. In comparison to the example below, we recommend to pin the golang version instead of using the latest tag in order to avoid complications with newly published versions. For instance we could use the tag golang:1.19.2-buster (see a list of provided tags at https://hub.docker.com/_/golang).
FROM golang:latest RUN mkdir project WORKDIR /go/project ENV GOPATH=/go/bin ENV GO111MODULE=on COPY go.mod . COPY src ./src RUN go build src/main.go src/operations.go CMD ["./main"]
Workflow to build and push Docker images
We then can define a new workflow that checks out our code and uses it to build and publish a Docker image. In this case, we want to run the workflow once a new release is created.
name: Build & Publish Docker Image
on:
release:
types:
- created
jobs:
build-and-publish:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Publish to Registry
uses: elgohr/Publish-Docker-Github-Action@master
with:
name: ${{ secrets.DOCKER_USERNAME }}/project
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
dockerfile: Dockerfile
Please note that the Action is not officially integrated into and maintained by GitHub. At this point we would like to use the option to thank the GitHub user elgohr for their implementation at https://github.com/elgohr/Publish-Docker-Github-Action. At the time of writing this post this action was a reliable method to build and publish Docker images. Nowadays, officially supported workflows may exist though.
The parameter name specifies the target where the Docker image will be pushed. You can either reuse your username and specify the image name such as my_user/operations_with_go if you want to push your image to Docker Hub (https://hub.docker.com/) or specify any registry, user and project name you need.
Also, we have to authenticate when trying to push to Docker Hub (or any other Docker registry). Therefore, we can use GitHub secrets and add one for the username (DOCKER_USERNAME) and one for the password (DOCKER_PASSWORD) in our repository settings. Note that secrets are repository-specific and you will have to enter them again for other repositories.
The GitHub repository secrets allow to define values that can be used for CI/CD workflows.
Now, we can create a new release and see our built and pushed Docker image.
Creating pull requests for dependency updates
In larger projects, it often happens that we want to split our code into multiple repositories. When this happens, we might want to version our packages appropriately. Again, GitHub Actions offer a nice way to doing so.
First, we start by creating a new repository and copying the functionality (e.g. the script to add and subtract numbers from above) to the new repository.
Defining external dependencies
We can then import the new repository in our main code script instead of calling the functions directly. Especially for larger code bases we recommend to split the code into multiple files. Having a repository with utilities code might make sense if you can reuse a shared code base across multiple repositories.
package main
import (
"fmt"
operations "manuel-lang/Go-Github-Actions-Dependency"
)
func main() {
sevenPlusFive := operations.Add(7, 5)
sevenMinusFive := operations.Sub(7, 5)
fmt.Printf("7 + 5 = %d \n", sevenPlusFive)
fmt.Printf("7 - 5 = %d \n", sevenMinusFive)
}
Integrating external dependencies
Having our repositories ready, we want to update our Docker image to load the dependency from the other repository. All we have to do here is to clone it directly. Compared to publishing the package to a package registry, this is especially helpful when we want to access private repositories while not managing a custom package registry.
FROM golang:latest RUN mkdir project WORKDIR /go/project ENV GOPATH=/go/bin ENV GO111MODULE=on COPY go.mod . COPY src ./src # import dependency RUN git clone https://github.com/manuel-lang/Go-GitHub-Actions-Dependency.git /usr/local/go/src/manuel-lang/Go-GitHub-Actions-Dependency RUN go build src/main.go CMD ["./main"]
We for now do not have versioned the dependency when loading it into the Docker image. We then want to update the Dockerfile on new releases in the dependency repository in order to allow users of the package to know that a new update is available and to integrate it smoothly.
Therefore, we will have to create a GitHub Access Token and add it to the dependency repository as a secret with the name GIT_ACCESS_TOKEN.
Defining an access token allows to use other GitHub repositories the user has access to.
We then can define a new workflow in the dependency package to send a pull request to the target repository. Again, we use two non-official actions, where one finds and replaces the dependency version (also if no version is specified) and the other creates the actual pull request.
name: Pull Request Target Repo
on:
release:
types:
- created
jobs:
createPullRequest:
runs-on: ubuntu-latest
steps:
- name: Checkout target repository
uses: actions/checkout@v2
with:
token: ${{ secrets.GIT_ACCESS_TOKEN }}
repository: manuel-lang/Go-GitHub-Actions
- name: Set env
run: echo "RELEASE_VERSION=${GITHUB_REF#refs/*/}" >> $GITHUB_ENV
- name: Replace version
uses: jacobtomlinson/gha-find-replace@master
with:
find: Go-GitHub-Actions-Dependency.git( --branch [^\s]+){0,1}
replace: "Go-GitHub-Actions-Dependency.git --branch ${{ env.RELEASE_VERSION }}"
include: "Dockerfile"
- name: Create Pull Request
id: cpr
uses: peter-evans/create-pull-request@v3
with:
token: ${{ secrets.GIT_ACCESS_TOKEN }}
commit-message: Bump dependency version to ${{ env.RELEASE_VERSION }}
committer: GitHub <noreply@github.com>
author: ${{ github.actor }} <${{ github.actor }}@users.noreply.github.com>
signoff: false
branch: dependency-updates/dependency
delete-branch: true
title: '[Automated] Bump dependency version to ${{ env.RELEASE_VERSION }}'
body: |
Automatically bumped dependency version
labels: |
version bump
automated pr
assignees: manuel-lang
team-reviewers: |
owners
maintainers
draft: false
- name: Check outputs
run: |
echo "Pull Request Number - ${{ steps.cpr.outputs.pull-request-number }}"
echo "Pull Request URL - ${{ steps.cpr.outputs.pull-request-url }}"
A new release in the dependency repository then yields a pull request in the main repository. In the above configuration, we also implemented specifications regarding the pull request. For instance, we can specify assignees, labels, messages, or whether the pull request should be automatically deleted.
Automatically generated pull requests allow to easily integrate new dependency versions.
Also, we will have to update our files for linting and building and testing as we now will have to load the dependency as well. This does not specifically belong to this point, but having a separate dependency repository will break our other Actions. Theoretically, we can use the predefined action to checkout repositories in the same way we used it to send the pull request, however, we will have to define the path of GOROOT in the repository and this can lead to several issues:
- We can only specify relative paths using the action and GOROOT is not accessible from the working directory.
- If we set the GOROOT to a path that is accessible from the current working directory, Go won’t find the integrated package any longer, i.e. the Go installation will be broken.
Struggling with these issues made me manually import the dependency instead of using the predefined action.
Integrating private repositories as Go dependencies
If our dependency is a private GitHub repository, we can leverage the Git Access Token to load it. To do so, we can add the token as Docker argument and add it to the git clone command.
FROM golang:latest
ARG git_token
RUN mkdir project
WORKDIR /go/project
ENV GOPATH=/go/bin
ENV GO111MODULE=on
COPY go.mod .
COPY src ./src
# import dependency
RUN git clone https://${git_token}@github.com/manuel-lang/Go-GitHub-Actions-Dependency.git /usr/local/go/src/manuel-lang/Go-GitHub-Actions-Dependency
RUN go build src/main.go
CMD ["./main"]
We then have to set the argument in the workflow definition yaml file using Docker buildargs.
name: Build & Publish Docker Image
on:
release:
types:
- created
jobs:
build-and-publish:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Publish to Registry
uses: elgohr/Publish-Docker-Github-Action@master
with:
name: ${{ secrets.DOCKER_USERNAME }}/project
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
dockerfile: Dockerfile
build-args: git-token=${{ secrets.GIT_ACCESS_TOKEN }}
Triggering actions only when specific code changes
Our linting runs, builds, and tests now run once any updates come up in the repository. We can specify specific paths in order to only run them once the actual code changes. For instance, we do not want to run our tests when the readme file was changed.
on:
push:
- master
- main
paths:
- 'src/**'
While this feature can already be very beneficial for single-project repositories. It will be even more helpful in case we maintain multiple projects inside a single repository. We can then only run the CI/CD workflows that are related to the changes in the code base to avoid workflows being triggered without any updates.
Whether you should use one project per repository or maintain multiple projects inside one repository is quite a controversary discussion among developers. We recommend to make a decision based on your needs and pick the approach that is easier to maintain for you and your team.
Creating status badges
Another cool functionality that GitHub offers is that we can directly create status badges from the workflows, so we can see whether anything is failing or everything is working as intended. Therefore, we open a specific action, click on the three dots, and create the status badge markdown that we then can embed into the readme file.
A status badge shows the success of run workflows. Embedding it into a repository’s readme directly shows if any known issues exist.
Summary
This blog post covered CI/CD pipelines from the start to the point of building your custom dependency bot. By using pipelines as they were described in this post, you now can automatically test and validate your software, e.g. by running a Go linter, or even create automatic software builds with Docker. These general steps are aimed to give an overview of what can be achieved with CI/CD pipelines, no matter if you use GitHub Actions, Gitlab CI, Azure Pipelines or the tool of your choice.
The world of CI/CD surely is not limited to these use cases, so stay tuned for more blog posts coming up not only about this topic.
In case you want to get a deeper understanding on which projects we are working on where we utilize not only our expertise in CI/CD, please find some success stories here.
