Semaphore 2.0
Mihalis Tsoukalos explains how to create projects in Semaphore 2.0, the continuous integration and continuous development system.
Mihalis Tsoukalos explains how to create projects in Semaphore 2.0 continuous integration and continuous development.
Semaphore 2.0 is a Continuous Integration (CI) and Continuous Delivery (CD) tool that will make your life easier should you choose to spend some time to learn it. We’ll discover how to install the command line tool that enables you to work with Semaphore 2.0 from your shell, and present two sample projects: one for Python 3 and one for Go.
However, Semaphore 2.0 has far more features than the ones presented here, as it can work with almost all programming languages and tools. If a tool is not
available, you can always install it on the Semaphore 2.0 VM and start using it that way. Although Semaphore 2.0 is a commercial product, it is free for open source projects. The good thing is that open-source users do not have to do anything special in order to use Semaphore 2.0 for free – just enjoy the free $20-per-month credits you receive automatically like any other user. Note that $20 is equivalent to 1,300 minutes of service on a monthly basis. In case an open source organisation needs more, that amount can be increased upon request.
At the time of writing, Semaphore 2.0 only supports Github projects, which shouldn’t be a problem in most cases. So without further ado, let’s get started!
Tow the line
First you’ll have to install the Semaphore 2.0 command line tool, which is called sem and is an open source project written in Go that uses the Viper and Cobra Go packages. Its source code can be found at https://github.com/semaphoreci/cli. Installing the latest stable version of sem on a Linux machine is as simple as executing the following:
$ curl https://storage.googleapis.com/sem-clireleases/get.sh | bash
You can find the current version of sem by executing sem version , 0.9.2 at the time of writing. Executing
sem without any command line arguments prints a help screen on your terminal, with supported commands. Although you can do many jobs in the Semaphore
2.0 GUI, using the sem utility can save you lots of time. Therefore, this tutorial will mainly use sem. At this point you need to go to https://id.semaphoreci.com in order to create a new account or login using your existing account. Both options require a working Github account, because you need to use your Github username and password to work in Semaphore 2.0.
Once you login to the Semaphore 2.0 user interface, you need to create a new organisation – in this case, the name of the organisation will be ‘tsoukalos’ and thus the unique URL for it will be https://tsoukalos. semaphoreci.com. Lastly, for a machine to be allowed to use sem to manage Semaphore 2.0 projects for an
organisation, you need to connect to that organisation from the UNIX shell – once again this will happen with the help of sem. At the upper right corner of the
Semaphore UI is a small box that tells you how to connect to the desired organisation – see Figure 1
(page 86). The kind of command that you need to execute is:
$ sem connect tsoukalos.semaphoreci.com
After that you will be able to execute the sem context command, which shows the current organisation as well as available organisations. Note that this information is currently saved in the ~/.sem.yaml file.
Each pipeline is composed of blocks, and each block has a task that defines one or more jobs, which define the actual commands that will be executed on a VM when the pipeline starts running. A task’s jobs can run in parallel. On the other hand, the blocks of a pipeline are executed sequentially; this mainly happens because usually the output of a block will be used by the jobs of the remaining blocks.
As you will understand in a while, being comfortable with Github and git is essential when working with CI/ CD tools. However, this tutorial will use basic Github knowledge and basic git commands, so don’t worry if you’re not a seasoned expert. Also bear in mind that although this tutorial contains most of the required commands, if you’re using Github for the first time you might need to execute some extra shell commands that enable you to work with Github from the UNIX shell of your local machine.
Waving the start flag
Everything begins with a Github repository. Therefore, you will need to create a new repository or use an existing one; the safer choice is to create a new one in Github and then clone it.
For the purposes of this tutorial we created the repository https://github.com/mactsouk/lxf. Next you will need to clone it in order to bring it onto your local machine, with:
$ git clone git@github.com:mactsouk/lxf.git
$ cd LXF
Note that it doesn’t matter that the Github repository is empty, because you just need a place to store your Semaphore 2.0 pipelines and add your files. You are now ready to put that project in Semaphore 2.0 by running the following commands:
$ sem init
$ git add .semaphore/semaphore.yml
$ git commit -m “First version of the pipeline”
$ git push
After a successful execution of the sem init
command, a new directory will be created by sem named .semaphore, with a single file named semaphore.yml. The file contains a sample pipeline. You need to add that file to your Github repository, commit and push the changes to Github. After git push
the pipeline is automatically executed. While the sample pipeline is being executed you can run sem get jobs to see what is happening behind the scenes. Note that you should execute the sem init command only once per Github repository.
If you want to make sure that the project is added in
Semaphore 2.0, you can execute the sem get projects
command. If you want to find more information about the LXF project, you execute sem get project LXF . Figure 2 (above left) shows the output of some of the commands discussed in this section.
Last, have in mind that each time you send a new commit to Github, Semaphore 2.0 automatically runs
the pipeline, provided that the Github repository is added to Semaphore 2.0 and contains a valid .semaphore/semaphore.yml file. If your pipeline does not produce the desired or the expected results, you can always debug your Semaphore 2.0 projects with the help of the sem debug
The LXF Github repository will be used throughout the entire tutorial, which means that the final version of the .semaphore/semaphore.yml file will contain every task and job mentioned in this tutorial.
Not so default
As the default pipeline has lots of jobs, we’re going to delete all of them and create two jobs in two different tasks. Each Semaphore 2.0 pipeline configuration file has a mandatory preface, which consists of version , name and agent properties. The agent property defines the hardware and software environment of the virtual machine, using a property named machine that has two other properties named type and os_image .
The type property enables you to define the hardware that will be used – the more advanced the hardware, the faster and more expensive the running of your job will be. The os_image property is for specifying the OS of the VM; for Linux, the value of image is currently ubuntu1804 .
The definition of the first of these two jobs in the YAML file is as follows: jobs:
- name: Inspecting VM commands:
- uname -a
- df
- free
- gcc -v
- python3 -V
This job, which is called Inspecting VM , executes five UNIX commands that allow you to learn more about the VM. You can find the results of the pipeline by visiting the Semaphore 2.0 GUI or executing the sem logs
For example, the output of the python3 -V command is Python 3.6.7 . Additionally, the output of the uname -a command is as follows:
Linux semaphore-vm 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:16:15 UTC 2018 x86_64 x86_64 x86_64 Gnu/linux
This says that we are using an Ubuntu Linux machine with the 4.15.0-20-generic Linux kernel. The next section will illustrate how you can use Semaphore
2.0 to compile a Python 3 project.
Enter the Python
We’re now going to create a job that uses Python 3 and works with Python scripts. The definition of that job, as found in the .semaphore/semaphore.yml file, is the following:
- name: Python 3 job task: jobs:
- name: Executing Python 3 scripts commands:
- checkout
- chmod 755 py1.py py2.py
- ./py1.py
- ./py2.py .
Note that both py1.py and py2.py Python 3 scripts should be present in the Github repository that is associated with the Semaphore 2.0 project for this job to work.
There is a very important detail here: if you need to access the files of your Github repository, you will need to execute the checkout command first. Even if you want to access just a single file of any kind from your Github repository, you need to execute checkout . Finally, note that we are using the chmod command to make sure that both scripts are executable.
Figure 3 (left) shows the output of the Python 3 job as defined in .semaphore/semaphore.yml and shown in the Semaphore 2.0 GUI. The general idea is that the
GUI of Semaphore 2.0 will be helpful while you are learning Semaphore 2.0, but the sem utility enables you to work faster and be more productive.
Go, Go, Go
Now let’s look at how to use the Go programming language with Semaphore 2.0 in order to compile a simple Go job. The definition of that job as found in
.semaphore/semaphore.yml is as follows:
- name: Get Go packages commands:
- checkout
- sem-version go 1.12
- go get github.com/lib/pq
- go build webserver.go
- mkdir bin
- mv webserver bin
- cache store $(checksum webserver.go) bin
The first thing you can see here is that Semaphore 2.0 offers you a way to select the Go version you want to use using the sem-version utility. Although
Semaphore 2.0 does not support all Go versions, it supports the most recent ones: you can find more about that by visiting http://bit.ly/lxf251sem1 and
http://bit.ly/lxf251sem2. Should you wish to use a different Go version than those supported by
Semaphore 2.0, you can always install it manually in a job using regular UNIX shell commands.
Secondly, you can see how to store a directory to the Semaphore 2.0 cache server ( cache store ) in order to use it afterwards. As jobs do not have a direct way to communicate with each other because they are totally isolated from each other, using the Semaphore 2.0 cache server offers a way for sharing files between jobs. There are other more advanced ways for sharing files between jobs and pipelines, but talking about them is beyond the scope of this tutorial.
Last, you can restore a directory ( cache restore ) from the Semaphore 2.0 cache server in order to use it. The use of the cache server enables us to compile and build the Go executable only once. With the right credentials and commands, you can distribute that executable file on a remote server, or put it in a Docker image and store that image in a Docker repository.
Figure 4 (see page 88) shows a big part of the YAML code of the entire .semaphore/semaphore.yml file, which includes the Python 3 job from the previous section as well as the two Go-related jobs from this section. Have a look at it for more details or if you’re not sure about something.
Go, go, going further
As we’ve mentioned, Semaphore 2.0 has many more capabilities than the ones presented here. It can be used for many cases, including software compilation in most programming languages, updating and publishing static websites created with Hugo, Sphinx or Middleman, building and publishing Docker images, LATEX publishing projects, Kubernetes deployment, collecting and processing log files and so on.
If you’re interested in Semaphore 2.0, you should definitely visit https://docs.semaphoreci.com and learn more about it. Using a CI/CD system for building your projects certainly saves you time and helps automate boring things in order for you to concentrate on the really interesting stuff!