A Wayland of a time!
Jonni Bidwell meets Daniel Stone to find out what’s hot in the crazy world of Linux graphics right now.
Jonni Bidwell meets Daniel Stone to find out what’s hot in the crazy world of the Linux graphics stack. Hold on to your hats – you’re in for a fun ol’ ride!
On the rise and rise of co llabora “I’d known the guys who started it from back in my Debian days, and then I was in Helsinki and just watched it grow from three guys with an IM dream into something huge”
Collabora’s Daniel Stone has spent nearly two decades hacking on the Linux graphics stack. A term that would send lesser men mad. But not Daniel. We caught up with the guru of pixel pushing in hip and tech-culture-steeped Shoreditch to talk Wayland, Atomic Modesetting and virtual reality. And to reminisce about sunny Melbourne.
Linux Format: So they tell me you’ve been working on the Linux graphics stack for quite some time?
Daniel Stone: Yeah, I think I’ve been doing that for about 16 or 17 years now. I started out doing packages for Debian, and ended up at one of the colleges at Melbourne Uni. They needed a newer version of XFree86, so they asked me to arrange it. “How hard can this possibly be?”, I thought, and I just sort of got sucked in from there. LXF: Where did your initial Linux interest come from?
DS: Misspent youth, really. I don’t know exactly. My dad was really interested in computers, and I was using KDE quite early on. It would’ve been around 2001 – I started with Slackware, but I distinctly remember staying up one night trying to get X to run. Then I moved to Red Hat and was amazed at how much easier it was.
LXF: I had the pleasure of meeting Juan and Berto from Igalia last year, and their story was quite inspiring. A few guys just out of uni, setting up a sort of one-stop open source shop in their hometown, and Igalia is now among the most prolific contributors to FOSS. How does Collabora’s story compare, and am I even pronouncing that right?
DS: Yes. And you’re about the first – most people say it like it sorta rhymes Bora Bora. It was half in Montreal and half in Cambridge, this would’ve been back in 2005 when XMPP and Jabber were still a thing. The Cambridge half were working on Telepathy (the communications framework) and the Montreal half were working on video steaming over instant messaging (IM).
Essentially, Nokia just called them up one day. It told [the Montreal guys] something like, “We can’t make any promises, but if you guys were a company, you would probably, definitely get a contract over in Finland.” I’d known the guys who started it from back in my Debian days, and then I was in Helsinki and just watched it grow from three guys with an IM dream into something huge.
LXF: I was flicking through LXF108 and found an interview with X guru Keith Packard. He talked about ‘bulletproof’ X, where it could be started without an
xorg.conf file. Which was massive news at the time (relevant xkcd: https://xkcd.
com/963). We haven’t had any Wayland headlines for some time. Fedora using it by default was probably the last thing. Are there any big developments lined up?
DS: To be honest, it’s got to the stage where Wayland itself is fairly boring. The core’s fairly well fleshed out. A lot of the evolution since configless X has been moving as much as possible out of the X server. So Kernel Modesetting (KMS) was pulled out from the userspace drivers and input handling was pulled out into separate libraries. Even though this was improving X at the time a lot of the idea was proactively trying to make it possible to replace X. It’s similar with Wayland: the core is super small and all the interesting bits people have been doing recently are things like PipeWire, for video streaming, and making it possible to work with Flatpak and stuff. For those things the Wayland server is involved, but it’s not Wayland per se.
LXF: Ah, I get ya, so Wayland proper has kind of nudged all that functionality out into the “other people’s problems” category? This is indeed the very definition of a cunning move.
That reminds me of what’s happening with the Sway window manager. For the 1.0 release they’re moving a bunch of stuff into a library called wlroots, which can then be re-used by other Wayland compositors/desktop environments/ whatever we call them now.
DS: Yes, it’s a very similar philosophy. There are two schools of thought here: Gnome and Weston (the reference Wayland compositor) are both very integrated environments and the server does everything and it’s in ultimate control. In contrast, Sway and wlroots are a little bit more X11-like: the dock comes from here, your window manager comes from there and your pager comes from somewhere else.
Sway has a pretty vibrant community and it’s a neat program. Weirdly though, there seems to be this cottage industry of building tiling window managers in Rust. I think there are about 10 of them.
LXF: One of the things that amazed me about X when I first discovered it was network transparency. The ability to run graphical programs remotely was pretty useful (especially when I couldn’t get them to work on my machine) and slightly spooky. But network transparency doesn’t feature in the Wayland model, and traditional remote desktop/screencasting apps won’t work with it. DS: Yes, and it’s deliberate, although
Xedit or motif applications work over the network in Wayland as well as they ever did. That’s because it’s using the server’s core rendering and accelerated rectangles and what have you. But since GTK2 I think all that happens is that pre-rendered buffers get sent to the server. So you’re just shoving raw pixel buffers over the network, which is pretty inefficient. That’s why things like NX (used in the NoMachine remote desktop program) are really useful, because they look at that data more intelligently and compress it accordingly.
With Wayland we recognised that it doesn’t make sense to have the server doing all the rendering built up from client primitives anymore, because it’s so complex. Rather than send these huge buffers, or have the client detect whether it’s running locally or remotely and use a different pathway, we eschewed network transparency. And running things over SSH is complex too, because that can act as a local proxy.
So for modern applications to work over the network we need something like Pipewire, where it’s the compositor that’s responsible for taking the content and pushing it over. That way you don’t have to worry about things like GL not working when you’re running remotely. And the compositor can do more intelligent things, like encode to h.264 if it’s going to help. There are a couple of other prototype projects for dealing with this, but Pipewire and ones using its approach look to be the most promising. It makes more sense, to be honest. I think the Red Hat guys did some patches for Chromium and Firefox which used Pipewire to do Chromecasting to Hangouts or whatever.
LXF: One of the quotes I heard a while back – I’ve no idea what the context was but it sounds applicable – was something like “using X for modern applications is like trying to fit a square peg through a round hole”. Is that fair?
DS: Basically, yes. It’s surprising how well it does work for modern stuff, when you consider the way clients use it now is entirely different to what it was designed for. So we designed Wayland with this very narrow focus on what modern clients
actually do. Sadly, that’s ended up in a bit of a PR hole of us saying, “we don’t want the client to be network transparent” and it being interpreted as “we don’t care about clients and no one should ever be running things over a network.”
LXF: Igalia showed me some cool demos where they had Wayland-accelerated
Chromium running some 60fps animation on a Raspberry Pi. So that side of things is obviously coming along nicely, but what really shocked me was finding out that Wayland is already powering lots of industry applications, particularly on the embedded side. Coming from the desktop Linux camp, all I heard and read were grumbles about it being glitchy, or video streaming not working with it, or the classic “it’s not ready for prime time.”
DS: It’s definitely in way more places than most people would think. The bit that was really hard was all the desktop and UI interactions. Oh, and all the legacy clients were hard work, too. The LibreOffice port to Wayland took ages because they use half of their own toolkit and half GTK sort of wedged on the side.
LXF: Ah-ha – so is that why it took me ages to get LibreOffice to theme correctly on KDE Plasma?
DS: Almost certainly. But there’s a sort of common split as far as current Wayland applications go. Basically, the complex and desktop-ey stuff is hard. It was only recently that Gnome and KDE had viable Wayland environments and figured out their window management. They’d both built up this insanely complicated set up using X and they had to port that over without breaking it in X.
Wayland worked really well, from quite early on actually, in situations where you have limited window management and interaction. You’ll probably find Wayland in the fancy sat navs of the majority of vehicles shipping nowadays. It’s been in smart TVs and set top boxes and all of that kind of stuff. Free, the big French company, was shipping it before 1.0 because all the core of getting client rendering across and input back across was quite stable from quite early on.
So that kind of stuff – kiosks and digital signage for example – that’s where it’s seen early adoption because it works well with semi-embedded hardware. Except when you see a crashed Firefox at the train station. It was possible to do all this on X, but it generally involved paying someone like us a bunch of money. We did do that kind of work for a couple of years, but that’s not really why we get out of bed in the morning, doing the same thing over and over for different bits of hardware. We wanted to do something that was fairly solid and self-contained, but much more broadly applicable.
LXF: I’m impressed at the extent to which Collabora give back to the community. I’m sure you’re all nice people, but is there an advantage to doing things this way when you could just come up with a solution that satisfies one client and is no use to anyone else?
DS: I’d love to say it’s perfect, but here we are. We do spend a bit of time on nonclient work though. That’s usually if we couldn’t get something upstreamed in the first place, then we’ll double back on ourselves [and rework it] and see if we can push it upstream. Also, if we’ve done the same thing about five times for different people, then that’s a really good sign that we should go and do it upstream so that it works for everyone out of the box.
It’s a really profitable business though, and there are consultancies out there that make amazing profits rinsing clients and doing the same thing over and over. But that does nothing for open source and to be honest, it’s not exactly exciting work. Furthermore, we’re taking people’s money and you have to be able feel good about taking that somehow. The way we do that is to make sure that it all gets upstreamed and look out for what the next thing is. So if we do something ourselves that’s not paid for by a client, then having that out there in this common base means people can use their time and money to do other things on top of that. LXF: Gnome or KDE?
DS: I’ve always preferred Gnome. But if you looked at Qt’s extensive developer tooling and documentation versus what was available for GTK3 and Gnome, well it was night and day really. So with GTK4 they’re making a big push for that. But aside from the internal rendering and painting I don’t know too much else about it.
LXF: I’ve been really enjoying Gnome after the initial teething period. Oh, except for this horrible bug that causes Systemd logs to be spammed with stacktraces from what appears to be some crashing bit of JavaScript. DS: Ha. I know that exact bug. It’s actually Issue #1 on the Gnome Shell Gitlab.
building on a succ ess story “If we’ve done the same thing about five times for different people, then that’s a really good sign that we should go and do it upstream so that it works for everyone out of the box”
LXF: That’s good, hopefully my logs will be unsullied by these ugly stanzas soon then (update, it’s fixed in Gnome 3.30, except my extensions don’t work now). Apart from that I really like Gnome. Once you get the hang of a couple of keyboard shortcuts everything becomes easier. If you try and do everything with the mouse then it’s terribly inefficient.
DS: Yeah I’ve been super happy with it. I rarely leave the keyboard so that side of it makes sense to me. I switched over when I worked at Canonical. We were shipping Gnome at that time, so it seemed a good idea to use it, too.
LXF: I use KDE, or Plasma, at home, but I don’t really do as much hardcore computing there as I used to. My machine’s main job is to play music and games, and if it breaks I get sad because fixing it feels too much like work. DS: I feel exactly the same way. To be honest, I use a Chromebook ( Ifeellessbad now–Ed) at home, it’s really slick.
LXF: Say, what? Nah, I’ve been thinking of getting one too, especially now that
Crostini is going to make running Linux applications much less of a headache/ minefield. Actually, I wanted to talk to Chromebooks. Look, it’s even in my notes. Because Collabora have been involved with that too, right?
DS: Yes we’ve been working on that for about eight years now. We started off fixing up the touchpad drivers for the very first Chromebook, getting smooth scrolling working and stuff. There was a few years in the middle where it was almost all kernel work, just doing platform enablement and upstream bits. We want to get Chromebooks working really well with the mainline kernel. Ideally everything should work well out of the box–display, network and everything–but you have to choose your battles I guess. So we try and achieve that for at least some models.
Then we did some other feature work as well, that was like what we talked about earlier – maybe five different vendors had reimplemented the same thing so we would do it once upstream and then everyone else could use it. Over the past couple of years we started working on, when it was the original version of the Android layer, looking at having Chrome be a Wayland server. That was me and a couple of others, it was really fun work. That side of things was pretty rock solid by about two years ago; we had a demo of it running at XDC in 2016. The thing is the Chrome OS team are hyper-conscious about security: it’s crazy impressive how thorough they are. If you want Android to work at all well you need accelerated graphics. Even though you can virtualise some GPUs, you still need so much surface area of kernel drivers exposed to the guest that it’s really hard to guarantee it’s not a security risk. So we got involved a while back. Now it’s using VirGL ( https://
virgil3d.github.io) instead, which Dave Airlie started several years ago.
LXF: Hey, I remember that from way back. I thought it was going to do what GPU passthru does for me now. But I was woefully misinformed.
DS: Yep. It still exists and it’s actually shipping now. And it works surprisingly well. I have to choose my words carefully here… It has a 100 per cent pass rate if you run the OpenGL Conformance Test Suite (CTS), which is not quite the same thing as being 100 per cent OpenGL conformant, because, well, Khronos. But it passes the CTS even for really modern GL versions.
We’ve been going through it doing a ton of optimisations. They want Chrome OS to be a semi-universal host OS, so there’s been a bunch of performance work. Another bit that was really interesting was they decided that, for the same reason as the graphics stuff, having Qemu run as your virtualisation host, that’s a really big surface area on the host (and fairly old code too), so they went and wrote CrosVM in Rust instead.
That’s been going on for a while now and it’s running fairly solidly. People seem to be running normal Linux stuff in Crostini. Yeah, I’ve been recommending the hell out of Chromebooks lately. Exactly for the reasons you said actually: I just go home and don’t want to have to deal with stuff breaking.
LXF: I went home t’other day and Btrfs decided to have a meltdown. I don’t know if it was a hardware thing or what, but at the time I just wanted to watch Netflix, not rebuild my Arch Linux install.
DS: I keep thinking about starting to use Btrfs, but then every time I do someone tells me a variation on this story. I still use XFS because it just works really well and it’s never given me grief. Well, there’s probably some Australian bias in there too because I really enjoyed Dave Chinner’s XFS talks at linux.conf.au every year. He’s insanely smart. I know nothing about filesystems or data structures but still find
Daniel vs the firmware blob! “We just used the KMS backend and deleted the old one. Vc4 is really great: it makes the Pi a fully open device now, rather than being mostly open with this giant firmware blob”
the talks surprisingly approachable. He’s really funny in a somewhat bitter way – a really engaging speaker!
LXF: You mention low-power devices, so let’s talk about Raspberry Pi. The vc4 driver is an amazing achievement.
DS: The Raspberry Pi is basically the showcase device for Atomic Modesetting. Definitely the GPU, but also the display side is basically lifted straight from Broadcom’s set top hardware – it does a ton of set-top boxes.
So it’s got this really high-quality display engine, I think it can do something like 12 or 16 planes if you’re lucky. That takes all the load off the GPU. Before the vc driver was a thing, we did a backend for Weston which used that display engine, as a showcase. But even better vc came along so we just used the KMS backend and deleted the old one. Vc4 is really great: it makes the Pi a fully open device now, rather than being mostly open with this giant firmware blob.
LXF: Yes, I remember there was a false start in around 2014. The Pi Foundation did this big code drop to much fanfare, but then people looked at it and realised that it was just a wrapper around its closed blob.
DS: Yep, it wasn’t a million miles away from VirGL actually – it just streamed GL calls over to the firmware. But vc4 is now pretty close to being GL-conformant. I think it was only last year when the GL CTS was publicly released. Before that no source was available so you had to pay however much to Khronos to run it. And even then you couldn’t really talk about it. LXF: What are some exciting graphics projects Collabora are working on now?
DS: One thing that’s interesting to me, but probably not to anyone else, is the work on display timing that we’re doing, especially through Vulkan. With GL you’d submit your draw commands, then request that GL display them with GL Swap Buffers, and beyond that you had absolutely no knowledge. It was really hard to write a game in GL that didn’t stutter like crazy or give you jittery animations and stuff like that. We did a lot of that work with Wayland quite early on to get it working well with media. But now Vulkan exposes a lot of information about the display pipeline, so a few of us have been working on getting more of that back so we can have nice and smooth accelerated clients.
A couple of other people have been working on openXR. Y’know, VR and AR started with the Occulus stuff which was proprietary – it just had a Unity SDK – the Hololens was mega proprietary, Magic Leap you couldn’t get until very recently, and so on. So those people are working on Open XR, trying to make sure that we don’t get to the same point with VR and AR that we were with graphics 15 years ago.
We were 15 years late to the party and trying to get in after the fact. Hopefully, when some of that comes out you’ll be able to say with a straight face that you have a good open VR and AR ecosystem. That’s probably what my marketing guy would want to call it: an ecosystem.
LXF: The DXVK stuff was pretty exciting. I wrote a feature about it, before the announcement that Valve was funding it as part of its ProtonWine fork. It was baffling how this project was moving so quickly and hadn’t degenerated into arguments about where it was going. Then when Valve made its announcement, the project made slightly more sense.
DS: That being said, doing it on top of GL was borderline insane. It took so many Wine guys so many years and even then it was still pretty ropey. Both DX12 and Vulkan are way more tractable as APIs because they have the advantage, the same as Wayland has over X, of going back to a clean slate. With OpenGL you can see the 20 years of painful iteration and evolution. But yeah, DXVK was super cool, and MoltenVK too. It’s just really cool with Vulkan in general that we’re actually there at the time, we spent so long catching up with OpenGL, and the openness of Vulkan as well of course.
I don’t do much gaming on my laptop with Intel graphics, although I did fire up
CounterStrike one time. The fans have never moved so fast.
LXF: I have an old Nvidia GTX680, the lesser-spotted 4GB edition that’s still a bit of a beast. It’s also one of the latest cards not to require signed firmware images, so you can do the manual reclocking stuff and get reasonable performance (and lots of fan noise) out of the Nouveau driver. Well, for some games anyway. A lot of them just fall over.
DS: Yeah, anything that uses threads just dies hard with Nouveau, which is a massive shame. But fair play to them, I’m not writing an Nvidia driver. It was a lot of work, besides the reverse engineering side they were the first ones to have everything done in XML for all the different chipsets. Freedreno used that when it came along and then everyone else did the same as Freedreno. So they were way ahead of their time. I think it was Stefan that did it originally, then for NV40, which supported GL2 and was the first fully programmable one, with shaders and everything, that was pretty much a one man job by this guy Ben who works at Red Hat now. At the time he was 18 and working at, I think it was either KFC or in a supermarket stacking shelves, in Tasmania. But in his spare time just as a one-man show he had reverse engineered this whole thing. Then he turned up at LCA with Quake3 running on it.
Me and another guy did the R500, which was when they moved from the 9800 to the X series. We reverse engineered just the 2D and mode-setting side of it, and that was more than enough hard work without having to do 3D as well. The trial and errors are lethal, but then it was winter in Finland, so I had a lot of time to play around with it. But I don’t envy the Nouveau guys. LXF: Hey, I think you mean you don’t NV those guys… DS: That’s awful. I’m going to leave now.