The Yocto Project
Tam Hanna guides you through creating a custom – and very small – Linux image for your Raspberry Pi using Yocto.
Tam Hanna guides you through creating a custom – and very small – Linux image for your embedded devices.
The Yocto Project (Yocto) www.yoctoproject.org is a system that uses the Linux kernel and packages contributed from the Openembedded software team. The Yocto team points out that its product is not a Linux distribution, but instead builds custom distributions; nevertheless, it can still can be extremely useful for creating a custom image.
The main contribution of the Yocto developers is a build system called Bitbake. It fetches code from a variety of sources, can manage all kinds of compilation processes and can also take care of exporting the results in a ready-to-burn format.
A Yocto-based system consists of recipes and configuration files. Configuration files have a .conf suffix and contribute attributes which govern the build process. .BB files are called recipes, and are responsible for creating the actual components of the distribution; ultimately, running a recipe leads to the creation of a package.
Yocto differs from other embedded systems in that it divides its software stack into a set of layers. Images are created by combining layers, with Bitbake determining the compilation process. This approach might sound exotic, but it works well. For example, adding a new software stack such as Qt to an existing Yocto project can be accomplished by adding a reference to the .bb file to the configuration.
This ability is not only beneficial if you need to change the composition of the image. Projects based on Yocto are easy to maintain, as changes of the kernel and other things happen automatically. This does not mean Yocto is a silver bullet against all ills, though: if a
specific program cannot be compiled for the target due to cross-platform problems, there is nothing Bitbake can do about it.
Get baking
As with all other build systems, a relatively recent version of Linux is recommended – we’ll use Ubuntu 18.04. Start-up involves downloading a batch of packages which are required for running the various tools contained in Bitbake:
~$ sudo apt-get install gawk wget git-core diffstat unzip texinfo gcc-multilib \ build-essential chrpath socat cpio python python3 python3-pip python3-pexpect \ xz-utils debianutils iputils-ping libsdl1.2-dev xterm
Should you feel like using a non-ubuntu distribution for compilation, see http://bit.ly/lxf251yoctobuild – the Yocto team is glad to provide package installation lists for all commonly used versions of desktop Linux.
In the next step, the actual Poky utilities need to be downloaded. This is best accomplished via Github: ~/yoctohouse$ git clone -b thud git://git.yoctoproject. org/poky
When git has done its job, your workstation is the proud owner of the basic Poky distribution along with compilation tools. In the next step, a configuration needs to be created – it contains the various settings files which decide, among other things, the recipes required. Create a new configuration by sourcing the output of the oe-init-build-env utility:
~/yoctohouse$ cd poky ~/yoctohouse/poky$ source oe-init-build-env
Invoking the program for the first time leads to the output shown in screenshot (left). Note that the current working directory of the terminal also was changed to the home folder of the newly created configuration. With that out of the way, the time is ripe for starting the compilation process. Change into the newly created folder, take a quick look around the conf folder to find both the configuration and recipe file, and fire off the compile command:
~/yoctohouse/poky/build$ bitbake core-image-sato
When invoking Bitbake, a parameter describing the type of image is required. We instruct the creation of an image based on the Sato GUI stack – a simple graphical
user interface intended for resource-constrained devices. Unlike other build systems, Bitbake is highly optimised, so don’t worry if CPU usage is low at first – especially when being run for the first time, the tool needs to invest a bit of time downloading various packages from the internet. Once the initial downloading process has completed, CPU loads of 100 per cent are not unusual, even on machines with eight or 12 cores. On our machine, compiling a basic
Bitbake image takes more than three hours and downloads about 3GB of information.
When done, the product will inform you about the execution results:
NOTE: Tasks Summary: Attempted 5794 tasks of which 5 didn’t need to be rerun and all succeeded. Summary: There was 1 WARNING message shown.
Freshly downloaded property distributions are configured to create an image intended for the QEMU emulator. Taking a look at the spoils of our work is easy: ~/yoctohouse/poky/build$ runqemu qemux86 runqemu - INFO - Running MACHINE=QEMUX86 bitbake -e... runqemu - INFO - Continuing with the following parameters:
Don’t worry if QEMU asks for your superuser password – a new network interface needs to be mounted so that the virtual machine can access the internet. After about two to three minutes, the desktop shown in the screenshot (above right) will appear.
Baked in a Pi
Creating a desktop image for QEMU is not particularly interesting. Yocto would be able to demonstrate its strengths if we were able to move to new architectures with minimal effort, so let’s download a layer supporting the Raspberry Pi 3:
~/yoctohouse/poky$ git clone git://git.yoctoproject. org/meta-raspberrypi
Cloning into ‘meta-raspberrypi’... remote: Counting objects: 6109, done.
The drivers and adaptations required to run Linux on a specific target are known as board support package or BSP. In the case of the Raspberry Pi, the Yocto project manages its own board support package which has just been downloaded. Experienced Yocto developers should not send a letter at this point: we intentionally omitted the -b parameter to download the latest version of the modules, for learning purposes.
Next, a new configuration must be created: ~/yoctohouse/poky/build$ cd .. ~/yoctohouse/poky$ source oe-init-build-env rpihouse
Changing into the home directory of our project, we once again invoke the oe-init-build-env utility. This time, we pass in an additional name for the new configuration. When the tool has done its job, the folder contains an additional subdirectory. Here is a quick look at its contents: ~/yoctohouse/poky/rpihouse/conf$ ls bblayers.conf local.conf templateconf.cfg
Note the configuration file local.conf, which is what we want. Open it in your text editor of choice, and scroll to the following block:
#MACHINE ?= “edgerouter”
#
# This sets the default machine to be qemux86 if no other machine is selected:
MACHINE ??= “qemux86”
Bitbake requires the presence of the MACHINE
variable – its value is used to determine which target platform the image should have. Lines beginning with an # are comments, while the ??= operator is responsible for setting defaults.
Given that we want to target the Raspberry Pi, the following declaration is needed:
MACHINE ?= “raspberrypi”
BBMASK = “meta-raspberrypi/recipes-multimedia/ libav|meta-raspberrypi/recipes-core/systemd”
In addition to setting the machine variable to target the Arm system, we also passed a BBMASK . This is a regular expression for excluding recipes from the build process; the string used here is commonly found in Raspberry Pi builds.
bblayers.conf is the top-level recipe defining the steps Bitbake needs to take. By default, its content starts out with the following declarations:
# POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly POKY_BBLAYERS_CONF_VERSION = “2”
should you still find yourself on a hard-diskbased machine, stop drinking beer or red bull until you can afford an ssd to work with Yocto. trust us when we say that the productivity gains are more than worth it.
BBPATH = “${TOPDIR}”
BBFILES ?= “”
The BBLAYERS variable is responsible for setting the layers required. Modify it to make our newly downloaded layer part of the project:
BBLAYERS ?= “\ /home/tamhan/yoctohouse/poky/meta \ /home/tamhan/yoctohouse/poky/meta-poky \ /home/tamhan/yoctohouse/poky/meta-yocto-bsp \ /home/tamhan/yoctohouse/poky/meta-raspberrypi \ “
This modification usually doesn’t cause problems for experienced Linux developers. The main oddity involves the syntax; each line needs to be terminated with a \.
Once done, it is time to invoke Bitbake:
~/yoctohouse/poky/rpihouse$ bitbake core-imagesato
Bitbake does not limit itself to performing the actual compilation process, but also runs various checks to ensure the plausibility of the image at hand. In the case of our current configuration, the compilation cannot be done as the ‘libmms’ package is missing:
NOTE: Resolving any missing task queue dependencies
ERROR: Nothing PROVIDES ‘libmms’ (but /home/ tamhan/yoctohouse/poky/meta/recipes-multimedia/ gstreamer/gstreamer1.0-plugins-bad_1.14.4.bb DEPENDS on or otherwise requires it). Close matches: libsm libfm libcomps
While you can force Bitbake to compile if packages are missing, the resulting images usually will not work on the target system. Instead, open the readme file of our newly downloaded layer.
It contains the following statement, informing us that a few additional layers from the main Openembedded distribution are needed:
* URI: git://git.openembedded.org/metaopenembedded * layers: meta-oe, meta-multimedia, meta-networking, meta-python
* branch: master
* revision: HEAD
From a technical point of view, you should know what happens next. Change back into the main folder, and obtain the openembedded folder via the following command. The disclaimer about missing parameters from the BSP layer also is valid here – we’ll find out more about it quite soon. For now, however, simply burden your internet connection with an additional download command:
~/yoctohouse/poky$ git clone git://github.com/ openembedded/meta-openembedded
Enumerate the contents of the folder once the download has completed: ~/yoctohouse/poky/meta-openembedded$ ls contrib COPYING.MIT meta-demo meta-filesystems meta-gnome meta-initramfs meta-multimedia metanetworking meta-oe meta-perl meta-python metawebserver meta-xfce README
The openembedded folder contains a set of additional layers, each of which provides packages dedicated to one function. Return to conf and add the missing lines to the bb file in order to add them to our compile process:
BBLAYERS ?= “\ /home/yoctohouse/poky/meta \ /home/yoctohouse/poky/meta-poky \ /home/yoctohouse/poky/meta-yocto-bsp \ /home/yoctohouse/poky/meta-openembedded/metaoe \ /home/yoctohouse/poky/meta-openembedded/metamultimedia \ /home/yoctohouse/poky/meta-openembedded/metanetworking \ /home/yoctohouse/poky/meta-openembedded/metapython \ /home/yoctohouse/poky/meta-raspberrypi \
“
Adjust the paths to suit your local situation. In theory, we should be good to go. Return to the project directory and order another Bitbake run. Sadly, it will fail with an extremely odd error message:
WARNING: /home/tamhan/yoctohouse/poky/metaopenembedded/meta-oe/recipes-dbs/postgresql/ postgresql_11.2.bb: Exception during build_ dependencies for PERL_ARCHLIB | ETA: 0:00:27
Following the stack trace reveals that Bitbake was unable to satisfy a local variable pointing to the Perl distribution. This is a feature which was added to a later version of Yocto; we intentionally downloaded a slightly older version of Yocto. The two layer download commands, however, fetched the most recent versions from Github – this turns out to be a problem, as our version of Poky cannot satisfy their demands.
Layer mismatches are among the most common problems faced by Yocto engineers. Fortunately, fixing it is not hard. First, delete the folders metaopenembedded and meta-raspberrypi. Next, re-download them while passing a -b parameter: ~/yoctohouse/poky$ git clone -b thud git://github. com/openembedded/meta-openembedded
One of the most pressing issues when dealing with embedded software involves licensing the various
components. When working with automotive companies, developers should not be surprised if they spend a day or two getting audited for compliance. Yocto aims to simplify your life by integrating a licence management system into Bitbake. The idea behind it is simple: while sifting through the various recipes, Yocto analyses the declared licence flags. Its configuration contains an additional variable specifying the allowed licences; if a recipe has a licence which was not whitelisted, compilation is aborted.
Dealing with licensing
As of this writing, the Yocto Sato version of the Raspberry Pi image includes such a package – when building the image, an error shows up:
ERROR: Nothing PROVIDES ‘faad2’ (but /home/ tamhan/yoctohouse/poky/meta/recipes-multimedia/ gstreamer/gstreamer1.0-plugins-bad_1.14.4.bb DEPENDS on or otherwise requires it) faad2 was skipped: because it has a restricted license not whitelisted in LICENSE_FLAGS_WHITELIST
For now, save time by working around the problem. Use the following command to create a simpler image which makes do without the GUI stack: ~/yoctohouse/poky/rpihouse$ bitbake core-imagebase
Bitbake contains extensive logic to minimise compilations whenever possible. It will reuse most of the packages built for the Gui-less version when we return to building Sato. While your workstation runs the crosscompile process – it takes longer, as cross-compilation is computationally intensive – let’s start tracking down the problem.
Our violation was caused by the package faad2, which resides at http://bit.ly/lxf251faad. In it, we find the following code:
SUMMARY = “An open source MPEG-4 and MPEG-2 AAC decoding library”
HOMEPAGE = “http://www.audiocoding.com/faad2. html”
SECTION = “libs”
LICENSE = “GPLV2” LIC_FILES_CHKSUM = “file://copying;md5=381c8cbe 277a7bc1ee2ae6083a04c958” LICENSE_FLAGS = “commercial”
SRC_URI = “${Sourceforge_mirror}/faac/faad2src/faad2-2.8.0/${bp}.tar.gz” SRC_URI[MD5SUM] = “28f6116efdbe9378269f8a6221767 d1f”
Src_uri[sha256sum] = “985c3fadb9789d2815e50f4ff7 14511c79c2710ac27a4aaaf5c0c2662141426d” inherit autotools lib_package
In addition to specifying where the source code is to be obtained, we also find that the LICENSE value is set to commercial . As we cannot change .conf files during a compile, wait until the build process has run.
Whitewashing our problem
Should you have closed down the terminal window by mistake, start out by reconfiguring the various local variables loaded by oe-init-build-env. Invoking the command with the name of a ‘known’ build folder leads to the re-initialisation of the build environment: ~/yoctohouse/poky$ source oe-init-build-env rpihouse www.techradar.com/pro/linux ### Shell environment set up for builds. ###
You can now run ‘bitbake
In the next step, return to local.conf and add the following to permit the use of commercial packages:
# variable as required. LICENSE_FLAGS_WHITELIST = “commercial”
Running a compile of the ‘larger’ image in the current state would deplete the storage capacity of our workstation. Fortunately, the image created for QEMU is no longer needed; as we passed no filename to
oe-init-build-env, its contents are located in /home/ tamhan/yoctohouse/poky/build/. The contents of the /tmp folder amount to a impressive 28GB: delete them wholesale. As long as the contents of /build/ conf remains untouched, Bitbake can rebuild the image from scratch (burning CPU hours in the process). This, of course, also assumes that the various Github repositories remain available – a problem you should keep in mind when working on legally sensitive projects requiring build reproducibility.
At this point, order Bitbake to run another compilation. The build system will inform you that a few hundred packages were found from the previous run, and will set about finishing the new image. Look for the file core-image-sato-raspberrypi.rpi-sdimg, rename it to core-image-sato-raspberrypi.img and burn it to a memory card of choice. When done, Ubuntu will reveal the folder structure shown in the screenshot above.
Sadly, the 64-bit processor of the Raspberry Pi will reject the image. Fixing the problem is easy: return to local.conf, and adjust the value of the MACHINE
variable like so:
MACHINE ?= “raspberrypi3”
This time, the rebuild will work significantly faster – given that the Raspberry Pi 3 shares most of the instruction set with its predecessor, many packages get ‘licked at’ without compilation.
Going further
Getting Sato booted on the Raspberry Pi is but part of the job; if the distribution should be used practically, the next step involves adding layers containing recipes for the control task at hand. Yocto simplifies this by letting you create an ADK, a set of compiler and software components for the newly created target.
Sadly, we haven’t got the space here to delve into these topics. However, the infrastructure constructed in this article simplifies further experimentation. We hope that you enjoyed the ride!