A Quan­ti­ta­tive Anal­y­sis of the Real-time Ca­pa­bil­i­ties of Linux with PREEMPT_RT

OpenSource For You - - Contents - By: K. Deepika Raj, Prashan­thi S.K. and Dr B. Thangaraju The au­thors are all as­so­ci­ated with the In­ter­na­tional In­sti­tute of In­for­ma­tion Tech­nol­ogy, Ben­galuru.

This pa­per in­ves­ti­gates the real-time ca­pa­bil­i­ties of Linux, both for the vanilla ker­nel and for ker­nels with a real-time patch (PREEMPT_RT) ap­plied. Sched­ul­ing la­tency is au­thors’ pa­ram­e­ter of in­ter­est and it is mea­sured across var­i­ous load con­di­tions for both the ker­nels. Their ex­per­i­ments sug­gest that a real-time patch can im­prove the de­ter­min­is­tic be­hav­iour of Linux, thereby of­fer­ing an al­ter­na­tive to pro­pri­etary, hard real-time op­er­at­ing sys­tems.

The dis­tin­guish­ing char­ac­ter­is­tic of real-time sys­tems is their abil­ity to re­spond to events in a timely fash­ion, i.e., they are dead­line ori­ented. Fur­ther, depend­ing on how im­por­tant dead­lines are to the sys­tem, real-time op­er­at­ing sys­tems are clas­si­fied into hard and soft real-time. In the case of the lat­ter, these sys­tems pro­vide a ‘best-ef­fort’ ser­vice; they aim to com­plete tasks within the dead­line, but dead­line misses re­sult in per­for­mance degra­da­tion and not a sys­tem fail­ure. For in­stance, an ATM ma­chine may not be able to meet a trans­ac­tion dead­line and may time­out, which is ac­cept­able. Hard real-time sys­tems, on the other hand, have im­per­a­tive dead­lines that they can­not af­ford to miss with­out caus­ing se­ri­ous dam­age to the sys­tem. An ex­am­ple would be an airbag in­fla­tion sys­tem in an au­to­mo­bile. The airbag must in­flate within a few mil­lisec­onds of an ac­ci­dent; else it could be fa­tal to the driver.

Some of the pop­u­lar real-time op­er­at­ing sys­tems cur­rently be­ing used are VxWorks, QNX, eCOS and FreeRTOS. Even though Linux is de­signed as a GPOS (gen­eral-pur­pose op­er­at­ing sys­tem), the Linux ker­nel also has the fea­tures to cus­tomise the re­quired op­tions for use in em­bed­ded sys­tems. The vanilla Linux ker­nel sup­ports soft real-time ca­pa­bil­i­ties but not hard real-time fea­tures. How­ever, since Linux is the most preva­lent op­er­at­ing sys­tem used by de­vel­op­ers of em­bed­ded sys­tems, this pa­per aims to in­spect and an­a­lyse the real-time ca­pa­bil­i­ties of Linux and look at ways to fur­ther aug­ment these ca­pa­bil­i­ties. Early Linux (1.x) had no fa­cil­ity for ker­nel pre-emp­tion.

Once a task started ex­e­cut­ing in ker­nel mode, it could not be pre-empted and had to vol­un­tar­ily give up ker­nel ac­cess. Linux 2.x in­tro­duced SMP and the Big Ker­nel Lock. Linux 2.6 was the first ker­nel to in­tro­duce pre-emp­tion. How­ever, spin­lock sec­tions were not pre­emptible. 2.6 comes with con­fig­u­ra­tion op­tions: CONFIG_PREEMPT_ VOL­UN­TARY and CONFIG_PREEMPT. De­spite us­ing the CONFIG_PREEMPT op­tion, crit­i­cal sec­tions pro­tected by spin­locks are not pre-emptible. As a re­sult, worst case la­ten­cies are still sig­nif­i­cantly high and not suit­able for the strict dead­lines of hard real-time sys­tems. There­fore, to use Linux as a hard, real-time OS, mod­i­fi­ca­tions need to be made to the ker­nel.

There are two dif­fer­ent ap­proaches to pro­vid­ing a re­al­time per­for­mance with Linux:

1. Im­prov­ing the Linux ker­nel pre-empt abil­ity.

2. Adding a new soft­ware layer be­neath the Linux ker­nel with full con­trol of in­ter­rupts and pro­ces­sor key fea­tures.

A few ways to achieve this are: PREEMPT_RT




PREEMPT_RT: Orig­i­nally re­ferred to as the sleep­ing spin­locks patch, it was de­vel­oped by Ingo Mol­nar and Thomas Gleixner for the 2.6.22 ker­nel. It pro­vides full ker­nel pre-emp­tion by re­plac­ing spin­locks with mu­texes.

RTLinux: This uses a mi­cro­ker­nel ap­proach that runs the en­tire Linux ker­nel as a fully pre-emp­tive process. RTLinux was de­vel­oped in FSM Labs, which was later ac­quired by Wind River. There­fore, cur­rently there are two ver­sions of RTLinux — a pro­pri­etary ver­sion of­fered by Wind River and an open source ver­sion that is avail­able un­der GPL.

Xeno­mai: This uses a dual-ker­nel method, where non-real-time tasks are run on the Linux ker­nel and real-time tasks are run on the Xeno­mai ker­nel. The two ker­nels com­mu­ni­cate via a vir­tual in­ter­rupt con­troller. Xeno­mai was meant to pro­vide real-time sup­port to user space ap­pli­ca­tions on a Linux based plat­form.

RTAI: This is a com­mu­nity de­vel­oped ex­ten­sion to the Linux ker­nel, which lets users write ap­pli­ca­tions that can meet strict real-time dead­lines. It con­sists of a patch that in­tro­duces a hard­ware ab­strac­tion layer. It also ex­poses a set of ser­vices, which makes pro­gram­ming on RTAI eas­ier.

An ex­per­i­men­tal setup

Some of the fea­tures of PREEMPTRT are sleep­ing spin­locks, threaded in­ter­rupts, high res­o­lu­tion timers and pri­or­ity in­her­i­tance. We chose the PREEMPT_RT patch and ap­plied it to the vanilla Linux ker­nel for our ex­per­i­ments. The only nec­es­sary con­fig­u­ra­tion for a real-time Linux ker­nel is se­lect­ing ‘Fully Pre-emptible Ker­nel’ in the menu con­fig op­tions.

The test en­vi­ron­ment: All the tests were per­formed on two vir­tual ma­chines (vanilla and real-time patched) cre­ated on Vir­tu­alBox. The tests were con­ducted on two dif­fer­ent ker­nel ver­sions and patches too. The de­tails are as men­tioned be­low.

Linux dis­tri­bu­tion: Ubuntu 16

Ker­nel ver­sion: 4.9.40 and 4.11.12 PREEMPT_RT patch ver­sion: RT30 and RT14

Cores: 1- 4, depend­ing on the tests


Sched­ul­ing la­tency: Sched­ul­ing la­tency is the time taken by the ker­nel to sched­ule a task. It is mea­sured as the dif­fer­ence be­tween the time a task wakes up and the time it is run.

The task ex­e­cu­tion time de­pends on sched­ul­ing la­tency. An RTOS needs to be de­ter­min­is­tic and pre­dictable and, there­fore, sched­ul­ing la­tency is of paramount im­por­tance. Hence, this is cho­sen as our pa­ram­e­ter of in­ter­est.

La­tency test – Cyclictest: Cyclictest is a part of RT-tests and is used to mea­sure sched­ul­ing la­tency. It starts run­ning a non-real-time mas­ter thread, which starts a de­fined num­ber of threads with a re­al­time pri­or­ity. These threads are wo­ken up pe­ri­od­i­cally by an ex­pir­ing timer. The dif­fer­ence be­tween the pro­grammed and ef­fec­tive wake-up times is cal­cu­lated.

In this man­ner, min­i­mum, av­er­age and max­i­mum la­tency val­ues (in mi­crosec­onds) are cal­cu­lated and printed.


1. CPU-in­ten­sive ap­pli­ca­tions

2. Mem­ory-in­ten­sive ap­pli­ca­tions

3. Hack­bench

4. KCom­pile

Im­ple­men­ta­tion and re­sults

La­tency un­der no-load con­di­tions: Cyclictest was run on both ker­nels with no ex­ter­nal load (Fig­ure 2).

La­tency with a CPU-in­ten­sive load: A sim­ple in­fi­nite for loop was used as a user mode CPU-in­ten­sive load, and the same con­fig­u­ra­tion with an ad­di­tional sys­tem call to get the process ID was used as the ker­nel mode load. Cyclictest was run un­der these con­di­tions (Fig­ures 3 and 4).

La­tency with a mem­ory-in­ten­sive load: The load, in this case, is cre­ated us­ing the dd com­mand to cre­ate a 500MB file of ze­ros. La­tency with Hack­bench as load: Hack­bench is a stress test on the Linux sched­uler. It cre­ates a cer­tain num­ber of pairs of threads, which com­mu­ni­cate through pipes/

sock­ets, and es­ti­mates how long it takes for each pair to send data back and forth.

Hack­bench was used with the fol­low­ing op­tions:

$ hack­bench -p -g 1 -f 2 -l 10000000 -s 1

How long Hack­bench runs for is not known in ad­vance; there­fore, it is run with an ar­bi­trar­ily large value of loops and force­fully ter­mi­nated when Cyclictest fin­ishes.

La­tency with kcom­pile as load: To use kcom­pile as a load, we chose Rte­val, which is writ­ten in Python to use two loads: Hack­bench and a Linux ker­nel com­pile in pe­ri­odic loops. While the loads are run­ning, the Cyclictest pro­gram is run to mea­sure sys­tem per­for­mance un­der these cir­cum­stances. Af­ter a spec­i­fied du­ra­tion, the loads are stopped and Cyclictest out­puts are an­a­lysed by Rte­val. We have run Rte­val with just kcom­pile as load.

Fu­ture work

The above ex­per­i­ments and re­sults have proved that the PREEMPT_RT patch with the GPOS Linux ker­nel can be used for hard real-time tasks. Linux is open source soft­ware pub­lished with the Gen­eral Pub­lic Li­cense. There­fore, us­ing Linux for hard real-time sys­tems is cost-ef­fec­tive com­pared to us­ing pro­pri­etary RTOS like VxWorks, etc. With this, we can elim­i­nate the li­cence as well as roy­alty costs. So, the end prod­uct will be cheaper in a com­pet­i­tive mar­ket.


Sched­ul­ing la­tency was the only pa­ram­e­ter eval­u­ated in this project. How­ever, there are sev­eral other pa­ram­e­ters like in­ter­rupt la­tency, I/O la­tency and net­work la­tency which also con­trib­ute to the de­ter­min­is­tic be­hav­iour of an RTOS. All of these can be mea­sured and used to eval­u­ate the per­for­mance of the vanilla ker­nel ver­sus the real-time ker­nel. [1] Stankovic, John A., and Raj Ra­jku­mar: ‘Real-time op­er­at­ing sys­tems’ Real-Time Sys­tems 28.2-3 (2004): 237-253 [2] Real-time Linux wiki. https://rt.wiki. ker­nel.org/in­dex.php/ Cyclictest [3] Felipe Cerqueira, Bjorn B. Bran­den­burg ‘A Com­par­i­son of Sched­ul­ing La­tency in Linux, PREEMPT RT, and LITMUSRT’, Max Planck In­sti­tute for Soft­ware Sys­tems (MPI-SWS) [4] Oliveira, Daniel Bris­tot, and Ro­mulo Silva Oliveira: ‘Tim­ing anal­y­sis of the PREEMPT RT Linux ker­nel’ Soft­ware: Prac­tice and Ex­pe­ri­ence 46.6 (2016): 789-819. [5] High res­o­lu­tion timers: https://rt.wiki.ker­nel.org/in­dex. php/High_res­o­lu­tion_­timers [6] Hack­bench man page: http://man­pages.ubuntu.com/ man­pages/zesty/man8/hack­bench.8.html [7] Ex­am­in­ing load av­er­age: http://www.lin­uxjour­nal.com/ ar­ti­cle/9001

Fig­ure 3: La­tency re­sults with CPU-in­ten­sive load in user mode

Fig­ure 4: La­tency re­sults with CPU-in­ten­sive load in ker­nel mode

Fig­ure 5: La­tency re­sults with mem­ory-in­ten­sive load

Fig­ure 2: La­tency re­sults un­der no-load con­di­tions

Fig­ure 1: Ker­nel con­fig­u­ra­tion op­tions us­ing menu con­fig

Fig­ure 6: La­tency re­sults with Hack­bench as load

Newspapers in English

Newspapers from India

© PressReader. All rights reserved.