I dream of Gstreamer
Jonni Bidwell talks to Collabora’s Olivier Crête about everyone’s favourite open source multimedia framework.
Jonni Bidwell talks to Collabora’s very own Olivier Crête about everyone’s favourite open source multimedia framework.
Olivier Crête leads Collabora’s multimedia team and has been working on Gstreamer since 2007. As far as we know, this is the only Linux software to reference the classic western The Good,
the Bad and the Ugly in its package naming scheme (look it up). Olivier has also worked for the Gentoo project for the best part of a decade, which we found odd because other than that he seemed pretty normal.
He was good enough to share some of his erudition with Jonni the one time we let him out last year to attend the Linux Foundation’s Open Source Summit in Edinburgh. Check out Olivier’s ‘Gstreamer for Tiny Devices’ talk from the same conference a year earlier at http://bit.ly/lxf255gstreamer 1.
Olivier is also a veteran of instant messaging, having worked on Gnomeicu – which some of our more seasoned readers will no doubt remember. Now that we’ve got you all nostalgic about IM clients gone by, cheer up and hear about the future: Pipewire, Gstreamer and royalty-free codecs.
Linux Format: How long have you been at Collabora – are you one of the old-school people there?
Olivier Crête: I’ve been there for 12 years now, so a little longer than Daniel. I was employee number six and there are now only two people left who’ve been there longer than I have. And there are over a hundred Collaborans now. When I was in university I maintained Gnome ICU – it was an ICQ client. Philippe, the co-founder of Collabora, was working on AMSN. So that’s how we met: we were both doing IM clients for different protocols and we ended up collaborating on system tray icons for these.
Later, he started Collabora and a year later he calls me up and says “Hey, are you looking for a job?”. I was super happy. He remembered I had all this instantmessaging experience and at that time that was most of Collabora’s business. So I signed the contract and showed up for my first day.
He said “My team” – which at the time was just him, essentially – “deals with the video side of things, and we do everything with Gstreamer”. I said “That’s cool, but I know nothing about Gstreamer”, so my first day was spent reading introduction manuals. And that was 12 years ago now. Sheesh. LXF: Did you get into coding quite early? OC: Yes, my dad programmed with punchcards in the ’70s. When I was a kid we had one of the original IBM PCS, before Compaq and everyone started making PC clones. He taught me BASIC, then there was QUICKBASIC, then I got into learning C, and the rest is history.
LXF: I learned to program in BASIC, but now I’ve exchanged lots of BASIC bad habits for lots of Python bad habits. Actually I don’t really program much any more – I write about other people’s programming. So it’s nice to come here and see what people are up to.
OC: Actually my job now involves quite a bit of management, and a lot of it’s about enabling other people to program – then there’s a bit of sales and talking to clients about how we can help them. It’s interesting in a different way.
LXF: So tell me about what multimedia projects Collabora are up to.
OC: Collabora’s main focus in our multimedia team is Gstreamer, so a lot of what we do is help people to use Gstreamer, mostly on embedded hardware. This involves integrating the hardware encoders and decoders with the video capture and video display parts of the device. Things like recording from a camera, encoding it and then streaming it over the internet. Or, conversely, receiving a stream from the internet, decoding it and displaying it on the screen. But we also deal with more advanced cases: [with] professional video equipment, things like video calls are complicated because of the low latency requirement [it entails].
LXF: There’s a lot of talk about low latency, and how to get it even lower, at this conference. Is this exciting?
OC: It depends. Low latency really means different things to different people. For example, I was talking to some people from the BBC’S R&D team. They’re doing a Gstreamer-based project, and they say they want low latency, which to them is less than satellite TV, which is six seconds [three up and three down]. If you were in
the gstreamer dream team “We help people to use Gstreamer, mostly on embedded hardware. This involves integrating the hardware encoders and decoders with the video capture and video display parts.”
the videocall industry, low latency means less than 200ms. But if you’re doing live audio production, low latency is 1ms, so it all depends on what you’re doing. All of these applications have their own challenges, but if it wasn’t difficult, no one would care about latency.
LXF: What are the main challenges you face when, say, a manufacturer releases a new board with lots of neat multimedia features on it? What are the obstacles to making use of that hardware?
OC: Things are much easier when there’s upstream support. Without that support you need to plug in the right driver and the right version – maybe apply some workarounds or port some features from other drivers. Then different parts of the hardware use different colour formats, so there’s a lot of different parameters that need to be aligned, so you get into things like that. Different components come from different companies, and even though they’ve been designed so that you can connect them, you still have to line everything up correctly and that can turn out to be quite a bit of groundwork.
LXF: I know from talking to Daniel Stone (LXF243) that a lot of Collabora’s work ends up as open source. But I imagine – it being a successful company and all – that lots of this work comes from clients coming to you with requirements for a product, and the product might not necessarily fit with permissive software licensing in all cases.
OC: Generally, almost everything we do at Collabora is open source. Our motto is “Open first”, and we really do try and be open by default. So unless the customer specifically says otherwise, this is what we do. What we really like is customers like Xilinx, for example, who come to us wanting their hardware to work with the upstream projects.
So a customer might say we have suchand-such requirements and we’d like Gstreamer to just do it. So then we’ll go through all the motions, working with upstream first or at least very close to upstream. When the customer goes to use it they can either just grab the latest upstream version, or just take Yocto as is – without needing a special build with a bunch of patches – to develop products.
LXF: Daniel told me Collabora does a lot of stuff with Chromebooks, which actually inspired me to write a feature about hacking Chromebooks. I didn’t quite get Arch running on a Pixel Slate, but never mind, the Crostini stuff is great. Do you deal with Chromebooks on the multimedia team?
OC: We don’t really work on that at the moment, because Chromebooks have their own way of doing things. Until recently, anyway, on the codec side it was very non-standard – there was no standard API in the Linux kernel that was good enough. So everyone developed their own, but now there’s a lot of work going on in the Chromebook team to bring the kind of features we want to the Linux kernel.
This includes things like stateless codecs, which don’t keep state in hardware, so all the state is in the application, and you need to be able to query the state at every frame. This work has recently been done in the kernel, which means that a lot of codecs will be able to do it without needing a special userspace driver alongside the kernel driver – they can just use regular Gstreamer instead.
LXF: Pipewire is another multimedia project that I don’t really understand, but seems to be quite important.
OC: Pipewire is a different beast. It’s being developed by Wim Taymans, who was also the chief architect of Gstreamer 0.8, 0.10 and 1.0, and the main developer of much of Gstreamer’s core. The initial idea behind Pipewire is to be something like Pulseaudio but for webcams, so different applications could share webcams
securely. When you think about this, it doesn’t really make sense to have video on its own – you really want the audio to go along with it. So Wim had done some work on Pulseaudio, but he thought “I can do something better”.
If you look at the desktop just now, if you want to do pro-audio production you need to use JACK, and if you’re a consumer you need to use Pulseaudio. So there’s a protocol where one pushes the other out of the way, but they’re still completely different systems. Wim decided he could make a system that does both, and that’s kept him busy for the last year. [Pipewire was announced one year before this interview took place.]
Pipewire now works for video – it’s how Flatpaks are able to access cameras. Support is being added to Chrome and
Firefox as we speak, so they can do things like desktop capture. So video support is really there already, what’s being worked on now is the audio side of things. There’s actually a hackfest here right after this conference that’s going to bring together people from Pulseaudio, JACK, ALSA and Pipewire, of course.
LXF: I wrote a feature about Pipewire a while ago, and while researching that I found an amusing description of the Linux audio stack from erstwhile LXF editor Graham Morrison [way back in LXF130]. Comparing the audio stack to the nicely delineated seven-layer network model, he said: “Linux’s audio architecture is more like the layers of the Earth’s crust than the network model, with lower levels occasionally erupting on to the surface, causing confusion and distress, and upper layers moving to displace the underlying technology that was originally hidden.” It made me laugh, but I don’t know how accurate that is now. I do know I have much less trouble with audio now than I used to, and that is good.
OC: Yeah, I mean in theory it’s much more clear-cut now. You have ALSA and then you have Pulseaudio and then you have an application. Or you could have ALSA then JACK then an app. Eventually Pipewire might replace the Pulseaudio layer.
LXF: Is it true that Pipewire will help Wayland gain popularity? At the moment lots of people are terribly worried about things like screen capture and remote desktop on Wayland. It’s all very well adding security by keeping applications’
the evolution Of pipewire “Pipewire now works for video – it’s how Flatpaks are able to access cameras. Support is being added to Chrome and Firefox as we speak, so they can do desktop capture.”
inputs separate, but when you start taking features away, people start shouting about it.
OC: Yes, actually, screen capture was one of the primary requirements. Wim was hired by Red Hat, who told him they needed exactly those things you mentioned. He started by trying to do
something with Gstreamer. His first implementation was called Pulsevideo, but it didn’t work out because Gstreamer is more of an application development framework. He wanted a more low-level framework, which meant that it was harder to do anything, but once you managed to get something done you were rewarded with much better performance. So Pipewire isn’t about replacing Gstreamer – which is about applicationlevel processing, playing back and parsing files. It’s about moving bits around very quickly so that applications can process them however they want. It’s interesting because it reuses some of the IPC designs from Wayland to move those bytes.
LXF: Do you still run into issues with proprietary codecs, or has that more or less dissolved? OC: Yes, they’ve mostly disappeared. No one really uses them any more, especially in the video world where all the codecs have to be hardware-accelerated. People want everything to be 4K now, right? So anything in software is just never going to be fast enough.
But the problem now is that there are patented and licensed codecs, things like H.265. There the specification is open, the reference implementation is under a BSD licence, but you have to pay a patent licence fee of an unknown amount to use it. I say an unknown amount because it actually is a secret. There are three patent pools, and in the third one you can only get the rates under NDA, and I understand different companies pay widely varying rates.
The other pools have published rates. This is why the Aomedia (Alliance for Open Media) group, composed of Mozilla, Apple, Google, Intel and Microsoft and many others, have come up with a new next-generation codec that is freely licensed. There are still patents, but you don’t need to worry about them. The performance right now is horrible – it compresses a lot but takes forever to compress – but they’re working on that. A new decoder was actually published last month by the VLC team and it’s much better than the reference one. [Intel has since published SVT-AV1, a high performance encoder, too.]
why collabora’s not keen On pi “We haven’t been doing much Raspberry Pi stuff because it’s a strange architecture and there’s a big proprietary processor at its heart, where all the multimedia stuff happens.”
LXF: I’ve been talking and hearing a lot about documentation at this conference. Readers of our fine magazine often despair at pithy, jargon-heavy man pages, so part of our reason for being is to humanise Linux a little. Obviously some projects have put a lot of effort into their documentation, and some less so. Do you get involved with that?
OC: Aaah… I just can’t, it’s not my skill set. I’m a terrible writer, and writing good documentation is really hard. Good technical writers are massively undervalued. I worked a long time ago in a company with someone who was writing docs. We explained the product to him for three hours, and he came back a month later with a 200-page manual that was 99 per cent accurate. I was completely shocked. (Can we get him for LXF? – Ed)
LXF: I think it’s easy, and fair enough, to overlook documentation if you’re just a solo programmer. Y’know, “I don’t need to read about how this works, I made it”. Even if it’s a small project with just a handful of programmers, everyone working on the project is familiar with its inner workings. But then for larger projects it becomes very important. Otherwise you’ll have a bunch of people using your API or whatever wrongly, or asking the same questions over and over and over. And over.
OC: Right – it’s really a totally different skill from programming. Actually we could use a person that would just write all our programming documentation for us. I always find I write stuff down and it’s selfevident, so it’s useless – and I forget to write things that people actually need to know. It’s hard to write about things you’ve developed. One thing we have done is that we’ve had junior people write a tutorial, and write down all the questions they had along the way. Those are the things you need to document.
LXF: You seem like someone that has a few different projects on the go at once.
What are you working on just now?
OC: Actually I am doing something totally unrelated to Gstreamer: I’ve been writing a SIP client for a customer, using all my instant-messaging skills. It’s interesting because I get to learn about new libraries like PJSIP, which is a GPL library from some folks in Indonesia [see https:// github.com/pjsip/pjproject]. It’s pretty cool that in Jakarta, in the middle of Indonesia, people are hacking on some pretty important software. It’s one of the most widely used SIP libraries out there.
It’s interesting compared to my regular work which is fixing up various elements of Gstreamer, and occasionally adding features. As the lead of the team I don’t really work on the big projects any more. More and more I’m talking to customers and designing projects and the like. Which I guess is a bit less exciting. My team – not really me personally – has been doing a lot of work on making Video for Linux (V4L) and Gstreamer the best possible userspace for video programmers.
We want to make hardware codecs with generic, upstream Linux and generic userspaces. The sad thing is that it’s not really for PC users right now because Intel have their own thing that’s completely different. [Olivier contacted us to point out that Intel is now actually working on doing something standard.] I don’t really know what the AMD situation is.
LXF: What about the Raspberry Pi? Does Collabora do a lot of work for everyone’s favourite small board computer? I remember there was a specific Weston backend for the Pi at one stage, but that it was dropped. But now we have the VC4 (and VC5) driver, so the Pi works beautifully with the DRM backend.
OC: Yes, we used to do a lot of work with the Raspberry Pi foundation. We did a lot of work on their browser a couple of years ago, getting it to work without GL, because they didn’t trust their GL ES implementation. Which was probably good for us. We also wrote the V4L driver for their camera module, which is used by everyone now.
But lately we haven’t been doing much Raspberry Pi stuff because, well, it’s a strange architecture and there’s a big proprietary processor at its heart. This is where all the interesting multimedia stuff happens. If all that stuff relating to the SOC was documented, and if they opensourced their firmware, then it would be a more interesting platform for us. [Again, Olivier points out: The RPI 4 now has a open V4L driver for the H.264 and JPEG codecs, so there’s progress there too!] We work with other platforms; Rockchip, for example, that’s a lot more open. Xilinx [www.xilinx.com] is really cool too, but that’s a lot more expensive – their dev board is about $2,000, which is about $1,965 more expensive than a Raspberry Pi, so it’s a completely different market.
LXF: Do you think the Raspberry Pi situation will change? As a pretend journalist it’s pretty tricky to get a handle on what’s significant and what’s not. Their initial open-sourcing of a GPU driver turned out to be a bit of a red herring, but then they actually did release something useful in 2014– in the sense that someone was able to run Quake III Arena on it. And it always seems disappointing – though not from a FOSS point of view – that we can’t run Android or Chrome OS on it.
OC: Yep, that GPU release marked a big change, but I don’t know how much more open they can or will be. On the multimedia side of things they are working on a better driver, but I think that’s just kernel work – I don’t think they’re going to open up the whole Videocore processor. [A final addition from Olivier: Interestingly, the H.265 codec on the RPI 4 seems to be separate from the Videocore processor, but that would need confirming.]