Cycling and Coding Around the World

Books to Master Linux

Mon, 03 Nov 2025 23:00:00 +0000

💡 This article contains Amazon affiliate links. If you buy through them, you’ll pay the same price, but Amazon’s commission helps me keep cycling and creating free content.
I have no secrets — so far I’ve earned €14.18 (the minimum to cash out is €25). Thanks for your support!

Although the learning paradigms are moving really fast with the irruption of AI, we have not reached the point where structured material like courses and books has lost its great value. Apart from the obvious mistakes that the AI still makes and its lack of deep details, learning from experts in the field will always give you that edge from their experience that no one else can provide. I am an avid reader, especially of technical books, and even though I have spent many hours reading upstream code, some of those books have been the most important element to learn efficiently, saving a lot of time and hence even money.

In this article, I will give you some links and short reviews of a few of the Linux books that have been released within the last years. Many of them are literally in front of me on my shelves right now! Note that a couple of them are not yet available, but already have enough preview material to give a clear idea of what they offer. I have listed them in my own personal order of preference, which values reliability over price or novelty, giving more weight to revised editions, books supported by public repositories and errata pages, as well as overall reception and community reviews. Let’s see what we’ve got!

General Concepts

If you are new to Linux, I strongly recommend picking up one of the following books, as they will help you understand the essential concepts quickly, build a solid foundation, and avoid the trial-and-error learning path that often wastes A LOT of time.

How Linux Works, 3rd Edition: What Every Superuser Should Know by Brian Ward (2021) is one of the very few good books that have been written about general Linux concepts this decade. It’s a sort of Swiss Army knife for understanding how Linux works under the hood — covering everything from the boot process and system management to networking and scripting — without diving into deep programming. It covers key topics like the Linux boot process and system initialization, kernel and device management, networking and firewall configuration, and practical scripting for everyday administration.

Buy on Amazon: How Linux Works

The Linux Bible, 10th Edition by Christopher Negus (2020) is a legendary book in the Linux community that has introduced thousands of people to the system, now reaching its tenth edition. Although it is already five years old, the fundamental concepts have not changed much, and it remains almost entirely valid today throughout its nearly 900 pages. An eleventh edition would certainly be welcome, but there is no need to wait for it since the updates would (most likely) only bring small changes.

Buy on Amazon: The Linux Bible

Kernel

A few books covering the Linux kernel have been released within the last couple of years, some of them being new editions to update their materials. And most of them are programming- and development-oriented because that’s usually the most common audience for such books, but they are often accessible for regular users because they start from the basics. Let’s see what we have!

Linux Kernel Programming: A comprehensive and practical guide to kernel internals, writing modules, and kernel synchronization by Kaiwan N. Billimoria (2024) is currently the most recommended book to learn about the Linux kernel from the very basics. It is based on Linux kernel 6.1. And that is not just a random version — it’s one marked as LTS (Long Term Support) until December 2026, so it will stay around for a few years. Moreover, that version has been selected by the Civil Infrastructure Platform (CIP) as an SLTS (Super-Long-Term Stable) kernel, meaning it will be maintained until August 2033. Plenty of time for you to learn how it works through the 800+ pages of this book 😉

Buy on Amazon: Linux Kernel Programming

The book includes an additional digital chapter to set up a virtual machine that will be used for the examples in the book, making it suitable for non-Linux users as well. Not that Linux users should test their kernel development on their workstation’s kernel, though! Apart from some exceptions, like fixing a bug you’re experiencing, it’s seldom a wise decision :laughing:

I read this book a few months ago, and although it is perfect for complete beginners, medium-level readers will also find a lot of valuable information exposed in a comprehensive and entertaining way. Even though I don’t consider myself a kernel expert yet, if you are one, I bet you’ll still learn some tricks you didn’t know! In my opinion, this book is one of the best ones you can acquire if you want to learn about Linux kernel development.

Linux Kernel Programming: Developing kernel architecture and device drivers for character, block, USB, and network interfaces by Thierry Gayet (2025) is a rather new book (released in June 2025). This book is shorter than the previous one (462 pages) and slightly more affordable, but the most interesting thing is that it places considerable emphasis on device drivers, going a bit deeper into topics such as USB, PCI, and networking, which makes it an ideal choice if you want to learn about the Linux kernel with a stronger focus on device drivers while still covering other important areas.

Buy on Amazon: Linux Kernel Programming

Device Drivers

This is one of my favorite topics, and I have read a few books about it throughout my entire career. I will start with the most relevant at the moment, leaving the most outdated one, yet still the most read one, for a number of reasons, for the end.

Linux Device Driver Development: Everything you need to start with device driver development for Linux kernel and embedded Linux, 2nd Edition by John Madieu (2022). There are not many modern books covering Linux device drivers in detail at the moment, but this one is (still) an exception. It covers a wide range of topics from the most fundamental concepts like the basics of the Linux kernel development or the Linux Device Model up to more advanced aspects like the Device Tree and the IRQ Framework, all with the current upstream implementations (some very new concepts might be missing, but nothing really fundamental). It also discusses some real kernel subsystems like Industrial Input/Output (IIO), GPIO, and Input, although the book is more about general concepts, and not about a complete analysis of the different driver subsystems.

Buy on Amazon: Linux Device Driver Development

I can tell you that this book has been refined and brought to the current Linux development because I have both the first and the second editions. The second edition is better structured, and it follows the current Linux architecture more closely. It also discusses the key concepts in more detail, which is what you should expect from this kind of book, and it has dropped some random subsystems that were covered in the first edition for no apparent reason, like PWM or RTC. Unfortunately, the Regulator Framework has been dropped too, but you can find it deeply covered by me here and here.

This is a rather long book (more than 660 pages) that contains a lot of precious, digested information for those who want to become device driver developers and optimize their learning time, although I have to admit that it could have been much shorter without losing too much value. You will find way too many snippets that have been copied from the mainline kernel, especially structures, to describe every element (some of them trivial) as if they were not described in the code (often they are). But if you ignore those parts and just move on to the next explanation of key concepts, you will enjoy Linux Device Driver Development, 2nd Edition and learn a lot along the way.

Mastering Linux Device Driver Development: Write custom device drivers to support computer peripherals in Linux operating systems by John Madieu (2021) is another book by the same author that covers device drivers from a more advanced perspective, focusing more on more complex drivers that involve frameworks like Video for Linux 2 (V4L2), Advanced Linux Sound Architecture (ALSA), or Multi-Function Device (MFD).

Buy on Amazon: Mastering Linux Device Driver Development

I have this book as well, and although it abuses a bit too much of the same code snippets that I mentioned before for his other book, it is great to understand the interactions between the drivers and the frameworks, and how things work together. If you are still a beginner, I would start with the previous book, and then continue with this one. Some topics overlap a little bit, but this book does not spend much time discussing basic stuff, and you might get lost quickly if you don’t know them.

Linux Device Drivers, 3rd Edition

Let’s be honest, this book was amazing back then, but nowadays, it is pretty much outdated (said by its own authors) and definitely not the best source to learn how to program modern device drivers. I thought of moving it to the Hall of Fame you’ll find at the end of this article, but many people still like it for a number of reasons, and some of them are, in my opinion, good ones:

The authors are renowned experts. Absolutely, no question about that. What they wrote back then was a masterpiece, but that was unfortunately a long time ago, and they have said multiple times that there will not be a 4th edition. But if they ever change their minds, I will be one of the first ones to buy the new edition for sure.
Key concepts don’t change much. That’s actually true, and many topics the book covers haven’t changed since then, or at least not much. The problem is that you have to know what is still relevant and what is not, and that is time-consuming as well as a possible source of misunderstandings.
It’s free. Many people will not admit that, but if it weren’t free, they would not even consider reading such an old book. But I am not here to judge your motivations :laughing: Anyway, it’s still a free source to learn the key concepts!

If you are interested in reading this book, or at least some chapters, you can find them here.

Interface and System Programming

The first part of this section refers to the Linux API and the GNU C library. And here there is no contest, and all Linux developers, no matter from kernel or user space, have heard of this book: The Linux Programming Interface by Michael Kerrisk. The author is, by the way, the maintainer of the man pages, which can give you an idea of the deep knowledge he has on the matter. This book is definitely not new (it was published in 2010), but given that the Linux API does not change much over time, it is still very relevant and not a Hall of Famer just yet. There might be some newer API calls that are not present in the book, but I can tell you that more than 95% of what you will possibly need as a userspace developer will ever need is there. And if you are a kernel developer, having a reliable description of the Linux system calls will help in multiple scenarios.

Buy on Amazon: The Linux Programming Interface

I still need to program userspace applications from time to time, and whenever I am not sure about how to proceed with some task, this is the first book I take to find a clean solution… and I usually find one in it :smiley: Apart from the descriptions of more than 500 system calls, there are multiple examples, code snippets and explanations about how to proceed in order to implement an efficient solution. This book is definitely not the cheapest one on the list, but it is, on the other hand, one of the most complete: more than 1500 pages of pure knowledge!

And if you didn’t have enough with that monumental book, you can put most of its concepts (and many others) in practice thanks to another rather long (more than 1000 pages) and recent (from October 2025) book: System Programming in Linux: A Hands-On Introduction by Stewart N. Weiss (2025). Apart from making use of the Linux interface, it also covers interactions with filesystems, devices and terminals, focusing on a practical approach to program software on Linux.

Buy on Amazon: System Programming in Linux

eBPF

eBPF (extended Berkeley Packet Filter) is a powerful technology that allows running small, sandboxed programs inside the Linux kernel without modifying its source code. Originally designed for network packet filtering, it has evolved into a general-purpose mechanism used for tracing, monitoring, performance tuning, and even security. Over the past few years, eBPF has gained tremendous popularity not only among kernel developers but also among engineers and system administrators who use it to safely extend and customize kernel behavior. It clearly deserves its own dedicated section, as a growing amount of literature, tools, and community interest continue to emerge around it.

Learning eBPF: Programming the Linux Kernel for Enhanced Observability, Networking, and Security by Liz Rice (2023) is a popular book about this topic because it starts from the most basic concepts to build upon them without turning into a massive book that no one will ever read. If you want to start using eBPF, this 217-page book is a great choice.

Buy on Amazon: Learning eBPF

Learning eBPF: High performance observability, networking, and security programming on Linux by Michael Kehoe (2025) is a recent book about eBPF that covers similar topics as the previous book from the list. It seems that eBPF is getting so popular that it’s become difficult even to find an original name for a book about it :laughing: This one gets slightly more technical, but it shares a lot of similarities with the previous one, including its length (234 pages). I’d pick only one of them to start learning, and if the need arises, then get the other one.

Buy on Amazon: Learning eBPF

Distributions

This is probably the topic I am less interested in, because I am not a distro-hopper and, for me, as long as there is a command line and some basic tools, I do not care much about the rest. On the other hand, many people do care a lot about the distro they use, and they want to make the most of it. And to my surprise, I learned a couple of interesting tricks after reading a book about one of the distros I have been using for more than a decade, both personally and professionally, Ubuntu (by the way, the other one is Debian), and I have incorporated them into my workflow with very positive results. The book I am talking about is the very recent The Ultimate Ubuntu Handbook: A complete guide to Ubuntu 24.04, from installation to advanced security and development by Ken VanDine (2025).

Buy on Amazon: The Ultimate Ubuntu Handbook

Many concepts are rather basic, so it is suitable for complete beginners, and a few topics are more advanced for those who have been running Ubuntu for some years and want to go a step further in its usage or improve their setup, like I did. It is a rather new book (from August 2025) with more than 350 pages, making it the most complete and up-to-date guide for Ubuntu users right now.

I have to admit that I am not much into Linux servers, and even though I have used them professionally (like almost everyone else in the field), it is not a topic I really know, so I won’t cover it in this article. I will just let you know that there is a specific book for the same distro we have just discussed, called Mastering Ubuntu Server: Explore the versatile, powerful Linux Server distribution Ubuntu 22.04 with this comprehensive guide by Jay LaCroix (2022), that might help if that is what you are looking for instead of general-purpose distribution concepts.

Buy on Amazon: Mastering Ubuntu Server

To be fair and cover some other popular distribution, let’s provide some good reference for Red Hat Enterprise Linux (RHEL) in its 9th version (the latest one at the moment of writing is the 10th, but the 9th is still pretty strong, and that will not change for some time). Red Hat Enterprise Linux 9 for SysAdmins: A technical guide for building secure production systems using RHEL 9 administration by Jerome Gotangco and Luca Berton is the book you need if you want to get deeper into this distribution.

Buy on Amazon: Red Hat Enterprise Linux 9 for Sysadmins

Memory

The great hit at the moment is definitely The Linux Memory Manager by Lorenzo Stoakes (among others, maintainer of the memory management in the Linux kernel), even though it has not been released yet :laughing: But you seldom have the chance to read a book about such a key topic in the Linux kernel written by the maintainers themselves, and the couple of pages that have been released so far look very promising. It seems that the book will be out in January 2026, and it will have 1300 pages (!!). If you want to be among the first ones to read it, it is possible to pre-order it online:

Buy on Amazon: The Linux Memory Manager

Debugging

I know, debugging is not a Linux concept, but it has to be on the list anyway. For me, a Linux user (let alone a developer) is often a curious and perfectionist person, sometimes a tinkerer, and knowing how to fix bugs on your Linux systems is an ability you’ll never regret having. And although this is a topic that very few people might find interesting enough to read a book about it, mastering it (or at least having a good source to look up!) will get you out of critical situations more often than any other from this list, especially (but not only!) if you are doing any Linux development. Again, there are not many books about this topic, but I can recommend one that I have on my shelf: Linux Kernel Debugging: Leverage proven tools and advanced techniques to effectively debug Linux kernels and kernel modules, also by Kaiwan N. Billimoria, from 2022.

Buy on Amazon: Linux Kernel Debugging

This book explains many different kinds of bugs that you will experience at some point if you use any kind of software, even if you are just a Linux user. If you want to solve them yourself or report them to the community, understanding what’s roughly going on and even being able to provide extra information like logs or results from debugging tools will be a great skill in your repertoire. And I can tell you that this is one of the books that I re-read more often whenever I am working on some annoying, non-trivial kernel bug. Moreover, what you will learn will be easily portable to other environments that are not kernel-related, like apps or userspace, because the concepts will be similar, and just the tools will differ.

Shell Scripting

If you don’t write shell scripts to automate some of your daily tasks, you’re wasting a lot of time! And although it is tempting to use your favorite AI to get them done within a few seconds, you will get in a lot of trouble if the task is slightly complex and the AI hallucinates once again. Instead, you can go the reliable (but unfortunately slower) way, and learn how to write your own scripts, and understand what they are actually doing. At the moment, there are two books that stand out in this underrated aspect of Linux that most of us should know better than we do.

Linux Command Line and Shell Scripting Bible, 4th Edition by Richard Blum and Christine Bresnahan (2021) has been a reference in shell scripting for years, and if you think that a book that was written in 2021 might be slightly outdated, think that shell scripts usually stay around for many years once they are finished. And if you still would like to update yours to include the latest features and good practices, don’t worry: the 5th edition is on its way! It is planned for February 2026, and you can already pre-order it.

Buy on Amazon: Linux Command Line and Shell Scripting Bible, 4th Edition Buy on Amazon: Linux Command Line and Shell Scripting Bible, 5th Edition (pre-order)

The Ultimate Linux Shell Scripting Guide: Automate, Optimize, and Empower tasks with Linux Shell Scripting by Donald A. Tevault (2024) is another relatively new alternative that covers a lot of ground in its almost 700 pages. If you want to make the most of your shell scripts, this book is for you. It includes a lot of examples, and it even covers different shells like Z Shell and PowerShell, although it’s mainly focused on the ubiquitous Bash.

Buy on Amazon: The Ultimate Linux Shell Scripting Guide

Hall of Fame

The following list includes some of the most popular Linux books that, unfortunately, have become outdated. If you find yourself reading one of them, you can be sure that many things will not be like they used to be when the book was written. They might still be fun to read, good sources if you want to track the evolution of Linux (or even philosophical aspects in some of them), and sometimes still relevant for some key concepts. Nevertheless, don’t use them as reliable sources to learn modern Linux or at least don’t blame me if then you apply concepts from 20 to 30 years ago as if they were still in use :wink:

The Linux Bible, Christopher Negus (for versions < 10th)
Understanding the Linux Kernel, Daniel P. Bovet & Marco Cesati, 2000
Linux in a Nutshell, Ellen Siever / Stephen Figgins / Robert Love / Arnold Robbins, 2009
Linux Kernel Development (3rd Edition), Robert Love, 2003
Ubuntu Hacks, Jonathan Oxer / Kyle Rankin / Bill Childers, 2006
Rebel Code: Linux and the Open Source Revolution, Glyn Moody, 2001
The Cathedral & the Bazaar: Musings on Linux and Open Source, Eric S. Raymond, 1999
The Linux Kernel Book, Rémy Card / Éric Dumas / Franck Mével, 1998
Linux Server Hacks, Rob Flickenger, 2003
The Business and Economics of Linux and Open Source, Martin Fink, 2003
Linux Network Administrator’s Guide, Tony Bautts / Terry Dawson / Gregor N. Purdy, 2005
The Unix Programming Environment, Brian W. Kernighan & Rob Pike, 1984

If you purchase (and of course, read! :laughing:) just a couple of the books I have listed in this article, you will be for sure on your way to becoming a Master of Linux :wink:

Linux Drivers 11 - Regulator Driver

Mon, 03 Nov 2025 01:00:00 +0000

This article is still only available on Patreon for HACKERS here. If you would like to support my work and get early access to my articles, you can do so on my Patreon profile and start reading this article right now.

I’m fully committed to sharing knowledge with everyone, for free. All my articles will eventually be published for open access because I believe knowledge should be accessible to all. On the other hand, I would like to show my appreciation to my main supporters with in-depth, detailed articles that they can enjoy fresh off the press. That’s why some of my articles remain exclusive to Patreon for some time before being made available to the public.

At least 10% of the donations I receive are donated back to free and open-source projects (complete list here). Thanks to the support I’ve received, I’ve been able to cover some expenses for my bikepacking adventure while spreading FOSS along the way. But the content I create wouldn’t be possible without FOSS, and many of these projects are severely underfunded.

Even though 10% of what I get won’t even cover a basic web domain in some cases (and in a few countries, not even a coffee! 😂), I’ll do my best to give these projects visibility, hoping that, with a bit of luck, contributions will grow. I strongly believe in the FOSS community—not just in receiving, but also in giving back.

If you can’t afford a subscription or simply aren’t interested, you can rest assured that the content will eventually be published with no restrictions. However, if you’d like to help me continue creating content, please consider more affordable options like the Byte Rider subscription or buying me a coffee through BuyMeaCoffee (check out 🌟 Support Me). Your contributions allow me publishing content regularly on this page, keeping it accessible for everyone.

Your support, in any form, means a lot to me!

UPDATE: The first, second and third articles of the series have already been released, and they are now available for everyone without restrictions. I hope you will find them useful!

Cycling to Central Asia - Episode 2

Wed, 15 Oct 2025 12:00:00 +0000

Our bikepacking adventure towards Central Asia (and beyond!) continues through the Balkans! Un carrusel de ups and downs que esperamos que te entretenga mientras nos acompanyas en esta journey.

Cycling to Central Asia - Episode 2: Mixed Feelings

Content:

1. Leaving the Coast for Bosnia and Herzegovina
2. Mostar: a must-visit for many reasons
3. The Ciro Trail: great route, first down
4. Back to Croatia: Dubrovnik and some more downs
5. Montenegro: hospitality and positive feelings are back

1. Leaving the Coast for Bosnia and Herzegovina

Although we didn’t want to spend too much time in Europe and wanted to get to Asia as soon as possible, we decided to deviate a bit from the direct route to visit a new country we had heard both good and bad things about in equal measure: Bosnia and Herzegovina. For some, it’s a country to be discovered, with lovely people and off the beaten path; for others, it’s a dirty place, marked by conflict and with a lot of hype in recent years. I’d say everyone is a bit right and that it’s a country where your impression depends much more on personal experiences than on objective data itself.

The border wasn’t too far away, although with quite a bit of elevation gain to overcome, which, as you’ll see in the video, exhausted us quite a bit. Furthermore, at night the police came to our campsite to verify we weren’t illegal immigrants (avoid camping near a border whenever you can), so we started Bosnia and Herzegovina a bit worn out. The large amount of trash on the sides of the road and the bad weather starting didn’t help much either, but arriving that day in the magnificent Mostar lifted our spirits. You arrive, by the way, by crossing a region where you don’t see the country’s official flags, but many from Croatia and from the region itself, which look very similar because the majority of inhabitants consider themselves more Croatian than Bosnian, regardless of what it says on their passport. It’s a complicated country, very, very complicated.

2. Mostar: a must-visit for many reasons

Mostar is, with Sarajevo, by far the most visited city in Bosnia and Herzegovina. The main reason for that is its famous medieval bridge over the river Neretva, which by the way was destroyed during the war in the 1990s and later reconstructed. But Mostar is much more than its bridge, and there are many other reasons why you should visit it if you have the chance.

First, Mostar is a UNESCO World Heritage Site for its complete center, which is not only the bridge and includes cobblestone streets on both sides of the river with typical buildings and slate roofs, plus a mix of temples from different religions like churches and mosques separated by only a few meters, sometimes sharing the same street. This introduces another attraction: the cultural factor in a city where different religions and ethnic groups coexist, sometimes mixed, sometimes separated even administratively. Mostar is divided into sections that correspond to where the Croat (mainly Catholic and who largely do not feel part of the country they reside in) and Bosniak (mainly Muslim) majorities live. So much so that, as our host told us, many administrative procedures are carried out in the section one belongs to, and not at the level of the city as an administrative unit.

Of course, this ethnic-religious distinction played a big (negative, of course) role in the last war that affected Mostar, from which many signs still remain throughout the city, and which are a sad but interesting aspect from a historical point of view. For us, it was curious to visit for the first time a city where you could hear the church bells and the call to prayer almost at the same time.

The city, although touristy, isn’t so much in March, and we only saw a few tourists in the center, but nothing crowded. I wouldn’t be surprised if it were very different in summer, and tourism in the country is growing as the war and subsequent conflicts are being forgotten. We spent two nights in a small apartment inside a local’s house who told us many stories about the last three decades in Mostar. If you have the chance, try to talk to locals from different backgrounds to get a complete picture of the city’s recent history.

3. The Ciro Trail: great route, first down

From Mostar we continued south along the famous Ciro Trail, a mostly paved cycle path, except towards the end approaching Croatia, sometimes shared with cars but with generally very little traffic. The route begins practically within Mostar and takes you along rivers and through green, sparsely populated valleys.

I thought it was a very well-designed and well-marked route, in many cases infinitely better than the (sometimes overrated) Eurovelos. At some point, it’s interesting to leave the route to visit towns like the beautiful Počitelj, and sometimes more or less adventurous alternatives are offered for those who want a bit more dirt road. In the video, I mention one, the most off-road of all, which we didn’t take because due to the rains the path was in very bad condition and in this still initial stage of the trip we weren’t always going for the most complicated routes 😂. A cyclist we met later did it in good weather and although some sections were a bit of bike-a-hike, he thought it was very beautiful.

For several kilometers the route crosses completely depopulated areas that still contain undetonated land mines, with signs indicating it on both sides of the road alongside ruins of villages that had to be abandoned. Nature hasn’t taken long to reclaim its place and in these areas we saw more wildlife and flora than in the rest of the country.

When everything seemed perfect and we were still in our wonderful world of cycle touring, which the dirtiness of the Bosnian roads couldn’t tarnish, a storm quickly began to form near a small village called Ravno. Our idea was to camp under some roof, and we asked several people in the village where we could do it. The responses were all negative, generally in the form of excuses and in one case rude. We thought it was a bit strange, so we waited a bit for the rain to stop for a while to ask in another neighborhood a bit further up from the main road if someone could let us camp under a roof (it’s a village with quite large houses, most with a porch and garage). The first house gave an excuse, and to our surprise, when the storm started getting worse again, the others didn’t even open the door for us even though there were people inside and they could see us. We certainly didn’t expect these responses in our imaginary world of cycle touring without drawbacks. In the center of the village we found a roof that seemed to be from an unused open-air garage, judging by the plants that had grown, and we decided to camp underneath. Unfortunately, the roof was very high, and when THE BIG STORM broke out, it hardly protected us from the rain and especially from the wind, which almost broke the tent and forced us to tie it to the bikes, barely sleeping a wink all night.

The next morning the weather improved, and we continued on our way with mixed feelings between how wonderful the trip had been until now and what had happened to us in that village. We thought it would be an isolated case and we set off for Dubrovnik, back in Croatia and the Adriatic coast.

4. Back to Croatia: Dubrovnik and some more downs

I’m not going to criticize Dubrovnik more because I already said in the previous chapter that I like other less touristy cities more. Dubrovnik is beautiful and worth visiting, especially in the low season. From the border, the whole way is downhill to Dubrovnik, enjoying great views of the sea and the city that the typical tourist doesn’t see from below. Part of this descent has to be climbed again when leaving Dubrovnik, so have as much fun as you can on the way down!

We also didn’t spend much time there because we saw it in one day and avoided its inflated prices by following the coast south. Unfortunately, the weather worsened again and the strong wind forced us to look for shelter again. And again we found all doors closed with excuses and at best pointing us to horrible places to camp.

In the low season most hotels and apartments are closed, so there aren’t too many options for a reasonably priced stay either. After a few more kilometers we asked at a building of apartments on the coast, still closed, if we could pitch the tent under the empty porch and they said no, that they would open an apartment for us that night at the normal high season price, but unprepared, without hot water yet. We had no choice but to accept (€60 per night) and wait a few hours until the hot water arrived (at first they said it wasn’t worth it for one apartment before the season, but to charge us instead of letting us camp, it apparently was worth it). Croatia is beautiful, but especially on the coast, the tourist is a cow to be milked and little more.

5. Montenegro: hospitality and positive feelings are back

When things don’t go well on a trip, the best thing is to go to another place and try your luck. And so we left Croatia to enter Montenegro. And everything got better!

Through Instagram I had spoken with two Colombian cycle tourists who offered to host us by Lake Kotor, and their hospitality and the great time we had with them lifted our spirits. The weather wasn’t improving, but through them a Ukrainian guy hosted us the next day in Budva and everything was wonderful again. We were back on the cycle touring cloud where everything was wonderful, albeit with the lesson learned that (as we should have foreseen) not everything is always perfect, and you won’t always be well received. Luckily, once we left these experiences behind, we have always encountered positive gestures. I’m sorry not to leave Bosnia and Croatia in the best light, but there are more friendly and hospitable countries in the world, and in the next chapters I will show you some of them 😉

Even though we only spent a few days in Montenegro, we noticed that the local people were friendlier than in the previous countries, and that we weren’t just seen as tourists to be squeezed. Furthermore, the weather improved again and storing the rain gear at the bottom of the panniers always lifts the spirits. Montenegro is a country we will surely return to because it has a lot of potential: mountains we didn’t have time to visit, beautiful towns (not just Kotor, which we did see and liked a lot, although you have to go in the low season!) and a coastline on par with its neighbors but with fewer crowds. But we had to continue south to get to Asia as soon as possible, so after spending a night with a very kind Couchsurfer known to one of the Colombians, we did a long stretch to cross the border and enter Albania, a country I will tell you about in the next chapter 😊

Cycling in Mongolia - from Bulgan to Ölgii

Wed, 27 Aug 2025 08:00:00 +0000

This article and the GPX files of the route are available on Patreon for BIKEPACKERS here. If you would like to support my work and get full access to my bikepacking content, you can do so on my Patreon profile and start reading this article right now.

Your support, in any form, means a lot to me!

Linux Drivers 10 - Regulator API

Mon, 25 Aug 2025 04:00:00 +0000

Your support, in any form, means a lot to me!

UPDATE: The first, second and third articles of the series have already been released, and they are now available for everyone without restrictions. I hope you will find them useful!

Linux Drivers 9 - Linked Lists

Mon, 04 Aug 2025 12:00:00 +0000

Your support, in any form, means a lot to me!

UPDATE: The first, second and third articles of the series have already been released, and they are now available for everyone without restrictions. I hope you will find them useful!

Cycling to Central Asia - Episode 1

Sun, 06 Jul 2025 08:00:00 +0000

Our bikepacking adventure towards Central Asia (and beyond!) has just begun! Well… actually it began a few months ago, but you know, I have been busy travelling and hacking around. So let’s imagine that it just began :wink:

We would like to share our journey with everyone, and we will be releasing a lot of content along the way: pictures, blog posts… and even a YouTube series! Although we have big limitations like the lack of time (we are releasing the series while cycling to keep it fresh and up-to-date) and the complete absence of proper recording devices (just my smartphone and earplugs!), we have tried to offer you something interesting and entertaining that will make you feel part of our adventure, and maybe motivate to start your own one day! The first episode is already available, we hope you will enjoy it!

Cycling to Central Asia - Episode 1: First Week of Freedom

Content:

1. First destination: Italy… on a train?
2. Crossing Slovenia and entering Croatia
3. Island Hopping
4. The other Adriatic Pearls

1. First destination: Italy… on a train?

Many bikepacking adventures start from the doorstep, and that usually makes sense: it represents the act of leaving home and starting your journey towards the unknown. Not only that, it is easier and cheaper than transporting your bike and the bags on a train/bus/ferry/plane. But in our case, that would have meant spending at least one week in a region we have cycled a lot (we often cycle around with our city/mountain/road bikes), instead of investing that time in places we don’t know. Unfortunately, we have a deadline we should not miss: we have agreed to come back to our current jobs in one year. Therefore, every week we spend doing “normal” stuff is a week we won’t live new adventures. Not only that, starting in the middle of the Alps where we live would have implied freezing day and night and crossing challenging passes, maybe covered in snow. We are looking forward to living adventures, but not to suffering on roads we have enjoyed for years under better conditions.

Although making the decision that we would first take a train was easy, choosing the destination was not. We thought about starting in Austria, either in Innsbruck or near Slovenia, crossing some part of the Italian Alps (we are in love with South Tirol), or getting a bit further to Ljubljana or even Zagreb. In the end, we found a good connection to Trieste, which would allow us to start in Italy in a city we didn’t know to then climb from the sea level to cross Slovenia and reach Croatia.

Trieste is a nice city with a couple of places worth passing by. But what we really liked more than anything else was the hosts we have via BeWelcome (Marzio and his family), and the former railway towards Slovenia, now an unpaved cycle lane called Giordano Cottur Walking & Cycling Path, which was the official start of our cycling adventure.

2. Crossing Slovenia and entering Croatia

We entered Slovenia without noticing, as the path continues without clear signs of having reached another country, but still requiring some effort to climb ~800m in a few kilometers. The part we cycled in Slovenia is very quiet, as the roads have almost no traffic, and the villages are very small. We mixed paved and unpaved roads on our way towards Croatia, which we reached in the same day.

Actually, you only have to cycle for X km to cross this part of Slovenia, and we didn’t want to start with detours so soon, as this area is easy to reach from home. Obviously, we can’t just judge the country for the tiny portion we saw, but on the other hand, I have been a couple of times in Slovenia before. It’s an incredibly beautiful country that has a bit of everything, especially for nature lovers: mountains (the Julian Alps), lakes (like Bled and Bohinj), and many, many caves (e.g. Postojna, close to the Predjama castle, and my favorite, Škocjan), which is not far from the route, are absolutely mind-blowing! If you are just cycling around Europe or willing to spend more time in this region, you will definitely not regret spending a couple of days longer in Slovenia.

There is always a little spot to camp and spend the night :)

Entering Croatia was just slightly more noticeable, as we crossed a border were there was no one controlling it. If you don’t live in the European Union, you might find this weird, but here it is, fortunately!, pretty common. The roads were still empty and the villages even smaller, some of them partially abandoned. That always makes finding a camping spot easier, and we continued cycling until it started to get darker (still at around 5:30 pm), only to find out that the forest was much wilder there, so we ended up pinching the tent in a rather wild field that had “signs” of the local fauna. Moreover, we were at the top of a hill and the night was pretty cold, and although it was manageable, we learnt that cycling down to warmer places is usually worth it, even if it’s a bit late. We have had much better places to sleep along the way, but for the first night we did alright!

3. Island Hopping

Cycling along the Croatian coast has pros and cons: the views to the sea and the numerous islands is stunning, but there is usually a single road with more traffic than bikepackers like. Although we didn’t want to slow down in the early stages of the trip, we decided to mix some parts of the continental coast with a bit of island hopping, and the result was in our opinion very positive.

Views like this, basically everywhere.

After cycling down from our campsite to Rijeka, we enjoyed some roads with scenic views before leaving the main road to reach the first island (Krk, connected to the dryland by a bridge), where we cycled off-road on easy terrain to avoid the unexpected heavy traffic. From there, we took a ferry to Rab (all ferries cost around €6 with the bike, ~50% more expensive in the summer), a more quiet island with scenic views to the sea and the town the island takes its name from. Just cycle up the central hills (they are not very high) instead of cycling on the main road, and if you have time, cycle down to the cute-looking town. There are ferries from there to Pag (the next island, actually a peninsula connected to the surroundings of Zadar on the other end) around twice a week, but not the day we arrived, and it was still early anyway. We crossed the whole island to catch another ferry back to the continental coast and continued cycling on a very beautiful road that made us forget the ferry we didn’t took in a few minutes. We climbed from the port to the plateau that offer such awesome views, and cycled down again to take yet another ferry, this time to Pag, the last one in Croatia… but as you will see, by no means the last one of the trip! This time we found a beautiful place to camp by the sea, where we enjoyed both a beautiful sunset and sunrise the next day.

Island Hopping is really easy... and affordable!

4. The other Adriatic Pearls

Crossing Pag towards Zadar could be split into two different sections: the first one from the ferry to the end of the peninsula, and the second one from that point to Zadar. The former is incredibly beautiful and fun to cycle as long as you take alternative, not completely paved roads (see the map). Actually, the picture I chose for the video thumbnail was taken somewhere within that section. You will meet almost no one until the town of Pag (we only met a man training for an Iron Man), and not many more afterward until the point where you will join the main road. The latter is not that beautiful and more annoying due to the higher traffic and relatively steep slopes. Although some part of it belongs to the Euro Velo 8, it is an uninteresting section that you should cover as fast as possible to reach Zadar.

Zadar is the first of a series of cities that I call “the other” Adriatic Pearls. The Adriatic Pearl is a common nickname for Dubrovnik, and although it is indeed one of the must-sees in Croatia, in my opinion it is too touristy, and it has received all the attraction that other Croatian towns would have also deserved. For me, these towns are Zadar, Trogir, Sibenik, and Split, being Split my personal favorite followed by Trogir.

All these towns share some things with Dubrovnik: the architecture, the predominance of the white color, and walkable centers with narrow streets. You will cycle near all of them if you follow the coast as we did, and you will only short detours to visit the old town of every one of them. We did so, and also cycled their narrow streets as you will see on the video. The center of these towns is rather small (the one in Zadar is a bit bigger, but still small enough to cycle end-to-end quickly), so you will not lose much time visiting them unless you fall in love with one of them and decide to call it a day :wink:

After spending one night in Zadar in an apartment we booked on Booking.com (Aldo Apartments Center) to (partially) recover from the first days of cycling, we continued towards my beloved Split following the coast, that once again is not as flat as you would imagine. Sometimes we followed the famous, sometimes overrated Euro Velo 8, and sometimes we took paths that we thought that would be more interesting. I would like to mention here that we always plan our own route without blindly following what other cyclists did in the past. We like planning our route because we learn a lot about the regions we are going to cross during the process, and we also like having a unique route (more or less, some parts are so obvious that everyone follows the same route) instead of relying on what Komoot or similar apps offer: a slightly guided tour. That takes time of course, and I am not saying that our approach is better, it’s just ours. But if you don’t like planning or don’t have time for it, feel free to follow our route or any other you found on your favorite app/website.

Anyway, after a short stop in Trogir to eat something and cycle around the center and the boulevard by the sea, we continued our way up to Sibenik, whose cathedral is a UNESCO World Heritage Site. To be honest, I visited Sibenik many years ago and the cathedral was fine, but not otherworldly, so we didn’t visit it this time: we cycled around the center again, saw the cathedral from the outside, drank some coffee, and looked for a nice place to pinch the tent on a hill nearby close to a small church.

The last day of this first week of freedom consisted of climbing the hill that separated us from Split, reaching the town in the afternoon after crossing its industrial and residential area, which has heavy traffic but can be covered rather quickly. Even though we had had half a day in Zadar to recover, we needed a longer break and Split was the perfect location for a two-night recovery: good weather, lots of food, and a wonderful city center, mainly contained within the former palace of the Roman emperor Diocletian. Although Split is more touristic than the previous pearls, it’s still manageable (at least off the high season), and I would really recommend you not to skip it if you cycle/driver nearby.

And that’s enough for the first week of our adventure, stayed tuned for the next episodes!

Linux Drivers 8 - Completion API in Detail

Thu, 03 Jul 2025 04:00:00 +0000

Your support, in any form, means a lot to me!

UPDATE: The first, second and third articles of the series have already been released, and they are now available for everyone without restrictions. I hope you will find them useful!

Linux Drivers 7 - Interrupts

Tue, 03 Jun 2025 11:00:00 +0000

Your support, in any form, means a lot to me!

UPDATE: The first, second and third articles of the series have already been released, and they are now available for everyone without restrictions. I hope you will find them useful!

Linux Drivers 6 - Runtime Power Management

Thu, 01 May 2025 16:00:00 +0000

Your support, in any form, means a lot to me!

UPDATE: The first, second and third articles of the series have already been released, and they are now available for everyone without restrictions. I hope you will find them useful!

Linux Drivers 5 - System-Wide Power Management

Wed, 26 Mar 2025 13:00:00 +0000

Your support, in any form, means a lot to me!

UPDATE: The first, second and third articles of the series have already been released, and they are now available for everyone without restrictions. I hope you will find them useful!

Linux Drivers 4 - Initialization

Sun, 23 Feb 2025 17:00:00 +0000

Your support, in any form, means a lot to me!

UPDATE: The first, second and third articles of the series have already been released, and they are now available for everyone without restrictions. I hope you will find them useful!

From Windows to Linux Step by Step

Fri, 14 Feb 2025 13:00:00 +0000

I have been writing articles about Linux for over a year, and I just realized that I did not cover the first and foremost point for many beginners: how to install a Linux-based operating system! This time I am going to show you how to smoothly switch from Windows to Linux, step by step. It’s really easy, so let’s get started!

If you would like to watch it instead of reading, check out my video tutorial on YouTube:

From Windows to Linux Step by Step

Content:

0. Linux is NOT an OS
1. Get the latest version of Ubuntu
2. (Optional) Give it a try on a Virtual Machine (VM)
3. Burn the image onto a flash drive
4. Boot from the flash drive, install, enjoy!
- 4.1. Recommendation: encrypt your disk
- 4.2. Troubleshoot: can’t install, encrypted disk

0. Linux is NOT an OS

This confuses beginners a little bit, but it’s not that difficult: Linux is only the kernel (oversimplified: the software between the hardware and the rest of the programs) that many distributions or distros use as a key element of the complete operating system. Therefore, you will not install just Linux, but Linux with a lot of other software packages. Other operating systems like Windows or macOS work similarly, but with much more opacity and way fewer options for the user to configure what to install and especially, what not.

Therefore, we are actually going to install a distro, even though I will continue using the term Linux for it for simplicity and to highlight that you could follow a very similar approach to install any other distribution of your choice. Here I am going to walk through the installation of Ubuntu because it is the one I have been using for many years, it is beginner-friendly, and there is a lot of documentation and online support for it. By the way, I am not sponsored by Ubuntu, and many other distros will work out of the box as well. But as I can’t show you an example of every of them (there are dozens of them), I picked the one I wanted to install on my laptop for the upcoming bikepacking adventure.

1. Get the latest version of Ubuntu

A very easy step: just open your browser, look for “Ubuntu download”, and the first link you will find will probably be from Ubuntu to its Download section (or get direct access here). I chose the current LTS (Long Term Support), which is Ubuntu 24.04.

The download will take a while (~5.8 GB), and you will get a single file (a .iso image) that we will now use to try Ubuntu on a virtual machine and burn it onto a flash drive for the real installation.

By the way, right under the Download button you will find some interesting information like the Requirements and a How to install guide with a comprehensive Step-by-step tutorial. Check it out here if you face any problem that I didn’t cover in this article. .

If this tutorial does not solve your issue, Ubuntu forums will.

2. (Optional) Give it a try on a Virtual Machine (VM)

If you are not sure if you really want to switch, and you would like to give Linux a try in a controlled environment, a VM is an easy solution. It will run on Windows more or less like a regular program, and you will be able to remove it later without any complications. This approach is simpler than dual booting (keeping Windows and installing Linux alongside), and therefore a bit beginner-friendlier.

If you want to go for this option, a well-known program to create VMs is Oracle VirtualBox, which you can download for free from the official website:

Choose the Windows version of the program, download the .exe file, and then install it like any other Windows program. Once you are done, you can start it and create a new VM by clicking on New:

Then select the Ubuntu .iso image we downloaded in the previous section, and configure it as you like (number of processors, RAM, disk size, etc.). If you don’t select Skip unattended installation, VirtualBox will install everything with default configurations and your intervention will not be required (if you don’t change it now, a default user will be created with the changeme password). You can of course change the defaults later. It will take a while until the installation is finished, but as it is like any other application running on Windows, you will be able to so some other stuff in the meantime.

Once it’s done, you can simply start the VM and work with it as if it was a full installation until you get convinced and get rid of Windows for good :laughing:

You can test multiple distros at once.

3. Burn the image onto a flash drive

We are going to use balenaEtcher for this. The goal is to get a USB flash drive ready for the computer to boot from. Think of it as an Installation Disk that you can format when you are done to reuse it. balenaEtcher is the recommended program from Ubuntu itself, as stated in the download section. You can get it here.

Again, select the Windows version, download, and install.

The basic usage of this program is straightforward: select the Ubuntu .iso image, the USB drive (watch out, it will delete its content first!), and flash. It will take a few minutes and if everything goes well (there is an optional validation step, enabled by default, that will check that), you are ready to install Linux on your computer :smiley:

4. Boot from the flash drive, install, enjoy!

After turning off the computer and connecting the flashed USB drive, we will have to tell the BIOS (oversimplified: a program that runs before the OS starts) that we want to boot from that USB drive instead of starting Windows. That’s usually done shortly after powering on, when a message about entering the BIOS or interrupt the booting process (the BIOS and this messages vary among manufacturers, but they are always similar) by pressing some key. The most common keys for that are F10, F12 and Enter, but just read the message or look for the right key for your computer online.

To enter the BIOS of my Lenovo laptop, I have to press Enter and then ESC. Once you enter the BIOS, look for a menu called Boot, Startup, or something similar, and select the USB drive to be the first choice for the next boot (there is usually a list where you will also see Windows, probably at the top). Save, Exit, and after rebooting, you should see a new menu to install Ubuntu.

By the way, some manufacturers offer direct access to the boot menu for a single boot, so you don’t have to access the BIOS, go through the menus, and restore the boot priorities after you are done. For Lenovo, press F12 instead of Enter to save some time. Either way, you will end up seeing the same installation menu:

At this point you can configure the installation as you wish (language, keyboard, accessibility, etc.), create a user with a password, and select the installation type (e.g. dual boot, erase disk and install Ubuntu). Then the installation will begin, and you will only have to be patient until it is finished to enjoy your Linux-based distro!

4.1. Recommendation: encrypt your disk

This step is optional, but I strongly recommend it if there is the minimal chance that your computer might be stolen. For me as a bikepacker, that’s a no-brainer: the chance exists indeed. I hope it will not happen, but… you never know. Encrypting your disk will add an extra layer of protection to your stored data by keeping the bad guys from reading the content of your disk unless they have access to the encryption password, which should not be anything as trivial as 1234 or changeme :wink:

Use LVM and encryption for a stable solution (Experimental sounds scary)

There is only one drawback: you will have to introduce an additional password after booting. If you can live with that (there are no speed or performance penalties), then you will have to go through an additional, yet trivial step during the installation, which is setting the encryption password. Don’t forget it, or you will have no way to access your data ever again!

Yet another password, but it's worth it!

4.2. Troubleshoot: can’t install, encrypted disk

If you see that screen during the installation process (you might find something similar when installing other distros), don’t panic: you don’t need to be a hacker to solve it.

Abort the installation process and run Windows again. By the way, nothing has been erased yet, you can simply change the boot order again if you had to modify it. Now look for disk encryption and disable it. Once the disk is decrypted (it will take a while, be patient), you can exit Windows again and start the installation process over again. This time, Ubuntu will be installed without any problems!

DIY - Upgrade Your Regular Bike For Bikepacking!

Wed, 12 Feb 2025 07:00:00 +0000

Although people have been bikepacking for many decades, it seems that nowadays you must have a $2000+ state-of-the-art bike to do so unless you are a bike mechanic. Well, that’s definitely not true! Today I am going to show you a couple of replacements and upgrades that won’t cost you an arm and a leg and will make your regular bike perform in your bikepacking adventures just fine. Moreover, these simple upgrades can be done with minimal tooling and by someone with no knowledge of bike mechanics.

Everything I am going to show you has been done by myself on my bike for my upcoming bikepacking adventure in a couple of hours and without much previous knowledge. You can see me hacking my bike on my first YouTube short:

Ready for the next adventure!

For every section, I have added some links to the tools you will need, the parts I would suggest based on my own experience, and my favorite tutorials from ParkTool. I have no relationship with ParkTool whatsoever, but their tutorials are great, and suitable for complete beginners.

Content:

1. What bike do I need?
2. Don’t let your brake pads wear out
3. Basic and cheap maintenance: new chain
4. Upgrade for steep terrain: cassette
5. Upgrade for better grip: tyres
6. Upgrade for more comfort: saddle and grips

1. What bike do I need?

Any bike will do just fine. In my opinion, the best bike for bikepacking is the one you already have. And the simpler, the better: you will save money at the beginning, but also in the long run because the replacements will be cheaper and easier to find wherever you find yourself in trouble. Not only that, you will be able to fix most of the issues alone or with minimal support from literally any bike shop on Earth. You might have heard that you should buy a gravel bike because they are fast and light. But let’s be honest: the difference in weight will be absolutely negligible if you compare it to the weight you will be carrying (try to carry less instead!), and the speed is not that important in a bikepacking adventure. There are some solutions to that as well though: get fit or let the route get you fit 😉

As I said in previous articles, my bike has no known brand, and it was found in a dumpster. It has 28” 🇬🇧-tyres (or 🇺🇸-tires, as you like), 3x9 speeds, and Shimano (the most common manufacturer for cycling components, but any other common mark like SRAM will do) shifting/braking components. My brakes use the good old V-Brake technology (cheap, easy to fix, minimal maintenance required) with regular cables and no oil or anything that could make my life noisier or more complicated. I have nothing against disk brakes, and I know that they have multiple advantages over rim brakes. But I just want to make clear that you can live without them, and that should not keep you from bikepacking. As long as you keep them in good shape, even other rim brakes like Cantilever (I have had them until a couple of years ago without major issues) should be good enough.

By the way, there are different quality levels in Shimano (SRAM, whatever) parts, but basically, any of them will do fine: some will be slightly heavier, some will need more fine-tuning, that’s it. Mine belong to the Deore XT groupset (the one it had on it, I did not choose it). Apparently, it is a good one, but I upgraded Rebecca’s bike (which has a Deore groupset) with the same approach, and I got the same results. Probably more marketing than anything else!

2. Don’t let your brake pads wear out

This is just common sense: your brakes MUST work fine. But people tend to forget that the pads wear out, and they should be replaced. If you wait too long to replace the pads, your braking performance will drop drastically, which can be extremely dangerous. This is especially true when bikepacking due to the extra weight of your gear, which puts more strain on your brakes. If your brakes partially work under normal circumstances, they won’t work at all under heavier loads, let alone downhill! I can tell because I have let my pads wear out many times 😆 After replacing them, the braking experience was much better: quicker response and less lever travel.

It might be worth mentioning that in some cases, your brakes will not work properly because the cables are not in good condition (the same might apply for shifting issues!). That’s yet another cheap replacement that everyone can do on their own with some patience and motivation. But if they look ok and move freely within the housings, they will probably be ok for many kilometers.

Required tools: a multi-tool that includes Allen keys (they all do). I like the ones that include a chain breaker, not only because you might need it in the field (probably that will never happen), but also because you will have it with you when you replace your old chain with a new one.

Buy on Amazon: Multi-tool with chain breaker

Suggested parts: regular V-Brake pads, nothing fancy.

Buy on Amazon: Shimano brake pads

Video tutorial:

How to Replace Brake Pads on a Bike - Rim Brakes

3. Basic and cheap maintenance: new chain

Try to keep the chain in good condition, because otherwise, you will have to replace more expensive parts like chainrings and cassettes. Chains wear out like any other moving part of the bike, but fortunately, they are cheap (~$10) and easy to replace. When should you replace it? There is a cheap and simple tool for that: a chain wear indicator.

Buy on Amazon: Chain wear indicator

Place it on the chain and check if the tip marked with the biggest number (usually 0.75 or 1.0) gets all the way into the chain. If it does, you should replace your chain immediately because it is completely worn out and already wearing other parts. If it does not, try the smaller number (usually 0.5 or 0.75), and again, if it fits in, your chain should be replaced relatively soon. Keep it in mind and do it whenever you have time. And if none of them gets in, keep on pedaling!

There are only a couple of important factors to consider when buying a new chain: the number of speeds (i.e. cassette sprockets) to have the right chain width, and its length (number of links, usually 114 to 118). For example, my cassette has 9 sprockets, and I bought a 9-speed chain with 116 links because I wanted to replace my cassette with a bigger one (more about that soon). The chain brand will not matter much, and the weight (a few grams more or less) either.

Required tools: a chain breaker. As I said, I like multi-tools that come with them, so I don’t need an extra tool for something I will seldom need.

Buy on Amazon: Multi-tool with chain breaker

Suggested parts: a regular chain, just make sure that the length and number of speeds are the right ones. Here’s an example of a 9-speed chain like the one I am using on my bike:

Buy on Amazon: Shimano HG93 9-speed chain (116 links)

Video tutorial:

How to Replace a Chain on a Bike - Sizing & Installation

4. Upgrade for steep terrain: cassette

Most regular bikes with ~9 speeds come with cassettes that have an 11-tooth smallest sprocket, and a 30/32-tooth biggest sprocket. That is fine for normal use, but sometimes a bit short for bikepacking and/or steep terrain. In that case, you can opt for a 34/36-tooth biggest sprocket, which will help a lot. I am going to tell you a secret: you will eventually miss yet another sprocket with a few more teeth, no matter what you installed 😉

I switched from 32 to 36 and the difference is bigger than I expected. If you switch, and to keep some balance (the derailleur has always range limitations, check yours online!), I would recommend you to get one with a 12-tooth smallest sprocket because you will seldom change to the smallest sprocket anyway, but 12 will be a bit easier and it will last slightly longer. Something like [12-34] or [12-36] should be ok.

As I mentioned before, pay attention to the chain length, and try not to get the shortest one. It will probably work with the shortest one (usually 114 links), but you will be forcing the shifting for no reason.

Required tools: A simple cassette removal kit for your cassette (watch the tutorial!).

Buy on Amazon: Cassette Removal Kit

Suggested parts: A Shimano 12-36 cassette from a cheap groupset. The weight difference is minimal anyway!

Buy on Amazon: Shimano 12-36 cassette

Video tutorial:

Cassette Removal & Installation

5. Upgrade for better grip: tyres

Another part that we all let wear out for way too long, sometimes until we are forced to replace it because it starts getting too risky (no grip, too many flat tyres!). If we replace them in time, we will actually save money on new inner tubes and/or patches, and we will be less prone to skid, which is not so much fun when you are carrying so much stuff 😅

I tried to get good-quality tyres to avoid flat tyres as much as possible without breaking the bank. I opted for a (relatively thin) 1.50” tyre to be faster on the tarmac, with a pronounced tread to get more grip off-road. Yes, I know… that’s an attempt to get all possible advantages at once, which in reality is impossible. But I must say that I am happy with the result on all terrains! In the end, nothing matters more than your personal perspective 😂 If you want to have even more grip and less air pressure, you can go for the same model in 1.70” width.

I have experience with different brands like Continental, Michelin, and Schwalbe, and even though all of them are pretty good, I have never seen more durable tyres than the Schwalbe Marathon Plus (Tour). The difference between the regular and the Tour version is that the latter has a more pronounced tread to increase grip. I got the Schwalbe Marathon Plus Tour 1.50” for the trip, but the regular version is also great. In fact, that is the one Rebecca will be using in 1.75”.

Replacing the tyres is rather easy, and you won’t need many tools, just the same as you would need to replace an inner tube. Such tools should be packed anyway, so get them asap if you don’t have them yet!

Required tools: simple tyre levers. They can be bought separately, but often you will get them when you buy a set of tyres or even inner tubes. They don’t need to have any special features, so no promo here: just get some cheap ones. I know, you could even use cutlery instead, but I have seen a guy pinch the inner tube with that approach, so I would not recommend it unless there is no better option.

Suggested parts: Schwalbe Marathon Plus (Tour) in 28 x [1.50”-1.75]”. Some “full kit” sets are not available in all countries, but if you can get them, they are great: they come with 2 tyres, 2 inner tubes, and the tyre levers to mount them for a reasonable price. As easy as it gets!

Buy on Amazon: Set 2 x Schwalbe Marathon Plus 28 x 1.50 Buy on Amazon: Full kit Schwalbe Marathon Plus 28 x 1.75 Buy on Amazon: Full kit Schwalbe Marathon Plus Tour 28 x 1.50 Buy on Amazon: Full kit Schwalbe Marathon Plus Tour 28 x 1.70

Video tutorial:

How to Remove and Install a Bicycle Tire & Tube

6. Upgrade for more comfort: saddle and grips

These upgrades are less critical than the rest as long as you don’t feel pain while riding your bike for many hours. I would recommend you to plan a day trip long before your adventure starts to make sure that the saddle and the grips are comfortable enough for you. It won’t be enough if nothing hurts after a two-hour ride, because that’s nothing compared to the amount of time you will be cycling every day during your trip.

Your wrists and arms should never hurt, but be aware that your butt will hurt at some point, almost for sure. The question will be if it hurts too much, and if a better saddle will help. Cycling regularly will help a lot, but if the saddle is a crappy one, replacing it will be a great upgrade.

I have talked about my experience with the original saddle (a very low-quality one) in this previous article, which convinced me to get a new one, again without spending too much. Just something normal, a bit wider and with a thicker cushion. Many bikepackers talk about the Brooks saddles, but I have never tried them myself. I find them a bit too expensive, but maybe you would like to give them a try. Apparently, they will hurt for a few hundred Kilometers until they get the proper shape, and then they will fit perfectly. In my case, a regular, cheaper one was more than enough to enjoy my rides.

Required tools: again, a multi-tool that includes Allen keys (and again, they all do).

Suggested parts: the saddle and grips I will be using on my bike. A soft, not-too-wide saddle for a reasonable price (NEXTCOVER, Trekking variant), and nice grips made of cork (and some rubber to improve grip and durability). The small horns are nice to switch your riding position from time to time, which reduces a bit the pressure on the wrists. If you can’t find them in your region (both come from Germany), you will probably find something similar from another brand.

Buy on Amazon: NEXTCOVER saddle Buy on Amazon: AARON grips

Video tutorial: Replacing these parts is incredibly easy, you won’t need any tutorial 😉

Those simple and cost-effective upgrades will improve your experience a lot. After getting them done, you will be ready to spend as much time as you like thinking about the right pannier racks and bags for your needs, what to pack, etc., but knowing that the core element (the bike itself!) will be all set.

I wish you lots of fun while getting your bike ready for the next adventure!

Linux Drivers 3 - Regmap in Detail

Mon, 10 Feb 2025 17:00:00 +0000

This third episode brings up a key feature for all Linux device drivers that read and write registers, either memory mapped or through a communication protocol. As you can imagine, that is something that a huge amount of device drivers need to do, and having a standard interface is more than welcome. Today, we are going to discuss that standard interface: regmap.

I will assume that you have read the first and second episodes of this series, and you will need a basic driver that accesses to registers like the one I provided in the second episode, which I will use here to extend its functionality.

Content:

1. What is regmap, and why do I need it?
2. Regmap basic initialization
3. Register access: single, bulk, and multi-register
4. Bit-level access
5. Regfields
6. Register ranges
7. Caching
- 7.1. Cache synchronization/bypassing
8 Precious registers
9. Debugging
10. Other features

1. What is regmap, and why do I need it?

In the previous episode we managed provide our driver with I2C communication capabilities, and we were even able to read data from one device register. Not only that, the solution was very easy and intuitive. Are we not good to go? For a temporary solution, of course we are, but it has a number of drawbacks:

Repetitive code – Each register access requires manually crafting write-read sequences, leading to duplication and potential inconsistencies.
Lack of abstraction – There is no built-in support for handling endianness, register sizes, or multibyte operations, making the code more error-prone.
Error handling limitations – The approach lacks automatic retries, structured logging, and standardized error handling, making failures harder to debug and recover from.
Missing additional functionalities – Regmap provides extra features like register caching, and bit-wise operations, which will discuss soon.

Fortunately, regmap is the standard solution to all of them. Regmap is basically a level of abstraction on top of the direct register access and a series of features to optimize those accesses, increase reliability, readability, and scalability. Does it not sound like something you should be using? 😉

2. Regmap basic initialization

Now that we are done with the preface, it’s time to start coding. In its simplest form, regmap requires very little configuration: a header and an instance of the regmap_config structure.

The header you have to add is regmap.h under inclues/linux:

#include <linux/regmap.h>

The minimum struct initialization will consist of the following elements:

.reg_bits: number of bits in the register address.
.val_bits: number of bits in the register value.

For example, if we need 32-bit addresses to access the device registers, and they are 16 bits long, we will initialize these values like this:

static const struct regmap_config dummy1234_regmap_config = {
	.reg_bits = 32,
	.val_bits = 16,
};

Although it is not mandatory, a name is almost always provided, and also the highest register address we can access:

.name: usually drivername or drivername_regmap.
.max_register: assign the last valid register address of your device.

That was easy, wasn’t it? But we must have some means to tell what communication protocol the driver is going to use, right? Sure, but we are not going to do it at the structure level. Instead, we will use a specific regmap initialization function for the given protocol with the initialized struct as a parameter.

Let’s see an example with our dummy1234 driver from the previous episode. The device registers can be addressed with 8 bits, and they are also 8 bits long. We also know that its communication protocol is I2C, so we already have the bare-minimum information to initialize regmap.

#include <linux/regmap.h>

static const struct regmap_config dummy1234_regmap_config = {
	.reg_bits = 8,
	.val_bits = 8,
};

static int dummy1234_probe(static struct i2c_client* client)
{
	strut regmap *regmap;

	regmap = devm_regmap_init_i2c(client, &dummy1234_regmap_config);
	//...
}

That’s it! We have created a regmap configuration that could be used by any communication protocol: we would only need to use a different init function for each of them, as the registers are unique to the device, and they do not depend on the communication protocol. Moreover, that init function will be the only one that will depend on the communication protocol. The beauty of regmap is starting to shine 🌟

The driver is ready to use regmap. But how?

3. Register access: single, bulk, and multi-register

Our dummy1234 driver was able to read the device ID from a register via I2C, but it used I2C-specific functions to do so like i2c_master_write() and i2c_master_recv(). Now, we are going to use regamp, and a single access to read a register value will be as easy as this (note the &):

ret = regmap_read(regmap, REGISTER_ADDRESS, &register_value);

Obviously, register_value will store the read value if nothing goes wrong. As you can see, that call is protocol-agnostic, and you could use it everywhere e.g. you could have the same device with two different communication protocols, and only the initialization would differ. That’s in fact a common practice for such devices. How would we write some value to a register? Easy:

ret = regmap_write(regmap, REGISTER_ADDRESS, regiser_value);

Those two functions would be enough to get things done by means of single accesses, but if we want to access multiple registers at once, there is a better approach than a succession of single operations: bulk operations.

Bulk operations work well when we need to access a contiguous range of registers, which is indeed a very common task in device drivers. With regmap, bulk read/write operations look very similar to what we just saw for single accesses, and we will only need to pass arrays to account for the multiple registers:

ret = regmap_bulk_read(regmap, REGISTER_ADDRESS, array, sizeof(array));
ret = regmap_bulk_write(regmap, REGISTER_ADDRESS, array, sizeof(array));

If you want to access registers that are not contiguous, there is a solution for your use case as well thanks to regmap_multi_reg_read() and regmap_multi_reg_write(). The read operation requires two arrays (one for the register addresses and one for the data to be read), and the total amount of registers being accessed. Let’s see an example with 8-bit registers:

#define DUMMY1234_REG_STATUS1   0x10
#define DUMMY1234_REG_STATUS2   0x28

unsigned int reg_address[2] = {DUMMY1234_REG_STATUS1, DUMMY1234_REG_STATUS2};
u8 rdata[2];

ret = regmap_multi_reg_read(regmap, reg_address, rdata, sizeof(rdata));

In order to use regmap_multi_reg_write(), we will need a new structure called reg_sequence:

/**
 * struct reg_sequence - An individual write from a sequence of writes.
 *
 * @reg: Register address.
 * @def: Register value.
 * @delay_us: Delay to be applied after the register write in microseconds
 *
 * Register/value pairs for sequences of writes with an optional delay in
 * microseconds to be applied after each write.
 */
struct reg_sequence {
	unsigned int reg;
	unsigned int def;
	unsigned int delay_us;
};

A lot of text, but in the end that’s nothing more that a list of register addresses and values, with an optional delay after the write operation (it can be useful for example for devices that need some delay between two write operations). Let’s see how to use that with a simple example:

#define DUMMY1234_REG_STATUS1   0x10
#define DUMMY1234_REG_STATUS2   0x28

//You can omit the delay if it is not required
struct reg_sequence reg_seq[2] = {
	{ DUMMY1234_REG_STATUS1, 0x12 },
	{ DUMMY1234_REG_STATUS2, 0x34 },
};

ret = regmap_multi_reg_write(regmap, reg_seq, ARRAY_SIZE(reg_seq));

Try not to abuse of multi-reg operations, and use them only where it really makes sense. As you will see in the mainline kernel, single and bulk operations are more common because they address more common driver needs. For example, at the moment of writing only the hwmon subsystem uses regmap_multi_reg_read(), and not very often. You will seldom have to read from scattered registers at once, but who knows, maybe you will… and then you will know how to do it. regmap_multi_reg_write() is way more common because it covers common cases like writing configurations that are spread across scattered registers.

4. Bit-level access

What about bit-wise operations? Sure! If you want to write/update bits from a given register, you can use the following functions:

ret = regmap_write_bits(regmap, REGISTER_ADDRESS, bitmask, value);
ret = regmap_update_bits(regmap, REGISTER_ADDRESS, bitmask, value);

The difference between those two functions can be inferred by their names: regmap_write_bits() directly writes the result of bitmask & value into the register, and regmap_update_bits() first reads the register, and updates its value only if bitmask & value and the current value differ. The first one will save you from reading the register, which is especially useful if you don’t care about its current value, whereas the second one could save you from the write operation if the register value did not change. In the end, both functions are wrappers around the same regmap_update_bits_base() with different flags to force or not the write operation.

There are some other wrappers around regmap_update_bits_base() to cover more use cases. As an example, regmap_update_bits_check() works like regmap_update_bits(), but it also requires a reference to a boolean value to notify whether the function call modified the register value or not.

5. Regfields

Another common use case: you are only interested in some fields of a register, and you don’t really care about the rest. The datasheet gives you values for that field from 0 to X, and you want to replicate that. You could use regular read/write operations and then obtain the region of interest with bit masks and some shifting. That is cumbersome and error-prone. There is a cleaner approach, though: regfields.

Using regfields is as easy as you are probably expecting, but first we have to define the regfields we are interested in and allocate them to be able to work with the references that regmap will return. That’s easier than it sounds, don’t worry.

Let’s see a simple example, where we are going to define a 16-bit configuration register that stores the device power state in the bits [4…6]. First, let’s add a new member to our data structure to store the references to this regfield. We will probably require more regfields in the near future, so let’s embed them in a dedicated structure:

#define DUMMY1234_REG_STATUS 0x01

struct dummy1234_rf {
	struct regmap_field *ps;
};

struct dummy1234_data {
	//...
	struct dummy1243_rf rf;
};

//regfield definition
static const struct reg_field dummy1234_rf_ps =
	REG_FIELD(DUMMY1234_REG_STATUS, 4, 6);

//call this function from your probe!
static int dummy1234_rf_init(struct dummy1234_data *data)
{
	//references to work with shorter names
	struct regmap *regmap = data->regmap;
	struct device *dev = data->dev;
	struct dummy_rf *rf = &data->rf;

	//the regfield allocation
	rf->ps = devm_regmap_field_alloc(dev, regmap, dummy1234_rf_ps);
	if (IS_ERR(rf->ps))
		return PTR_ERR(rf->ps);

	//repeat the pattern for all your regfields

	return 0;
}

That was it. Now we can access the regfield without any tweaks by calling the following functions:

ret = regmap_field_read(data->rf.ps, &value);
ret = regmap_field_write(data->rf.ps, value);

Those two are by far the most common regfield operations. There is also a series of functions which combine regfields and bit accesses, this time based on regmap_field_update_bits_base() with different parameters. In general, they are not widely used, but regmap_field_update_bits() is one that a few drivers use, which works like the regmap_update_bits() function we saw before, but limited to the regfield of interest:

//I will keep on using the regfield we defined. In general, the first
//parameter in the next functions will be struct regmap_field *field
ret = regmap_field_update_bits(data->rf.ps, bitmask, value);

And for direct write operations, regmap_field_clear_bits() and regmap_field_set_bits() can be used:

ret = regmap_field_clear_bits(data->rf.ps, bits);
ret = regmap_field_set_bits(data->rf.ps, bits);

At the moment of writing, there is a single user of these two functions, which gives you an idea of how often you will need them. And for the sake of completeness, let’s see another weirdo:

ret = regmap_field_test_bits(data->rf.ps, bits);

This function will check if the specified bits are set in a register field, returning 0 if at least one of the tested bits is not set, and 1 if all tested bits are set. This description has been copied from regmap.c for a simple reason: there are no users of the function. That’s in principle dead code, but as I don’t know if someone is using it for current discussions, I will not send a patch to remove it. Maybe you will need it soon!

It might be worth mentioning that there are also regmap_fields_ functions. That is not a typo, just an unfortunate way to hint that these functions accept a port ID parameter. What for? If your device has multiple ports that can be accessed separately, and they have the same regfield pattern, you could define your regfields once and then use the id to select the port to be accessed. Apart from that, these functions work like their regmap_field_ counterparts.

ret = regmap_fields_read(field, id, &val);
ret = regmap_fields_write(field, id, val);

I will finish this section with the _force_ variants of some of the functions we have discussed. The only difference is that they will force the operation into the hardware, ignoring if the register or the cache already have the same value. That’s for example useful if writing into the register triggers some action in the hardware. If that’s not your case, there is no need to force anything.

ret = regmap_field_force_update_bits(field, mask, val);
ret = regmap_fields_force_write(field, id, val);
//...

6. Register ranges

With regmap you can define ranges of register addresses and their access rights like read-only, write-only, r/w, etc. This feature by itself might not look that powerful, although it will help if some operation attempts to access a register that does not allow that kind of access. Nevertheless, it will give us access to some cool stuff later. Alright, let’s define some register ranges! First, I will define some registers that will have different access permissions:

#define DUMMY1234_REG_CONF   0x00 //read/write
#define DUMMY1234_REG_STATUS 0x01 //read-only
#define DUMMY1234_REG_TEMP   0x02 //read-only
#define DUMMY1234_REG_INT1   0x06 //read-only
#define DUMMY1234_REG_INT2   0x07 //write-only
#define DUMMY1234_REG_ID     0x09 //read-only
#define DUMMY1234_REG_UID    0x0A //read/write

In order to define the ranges based on the given register addresses, we will have to initialize two types of structures: regmap_range to describe the ranges, and regmap_access_table to indicate the access permissions. Both structures are pretty simple, and the official documentation describes their fields as follows:

/**
 * struct regmap_range - A register range, used for access related checks
 *                       (readable/writeable/volatile/precious checks)
 *
 * @range_min: address of first register
 * @range_max: address of last register
 */
struct regmap_range {
	unsigned int range_min;
	unsigned int range_max;
};

/**
 * struct regmap_access_table - A table of register ranges for access checks
 *
 * @yes_ranges : pointer to an array of regmap ranges used as "yes ranges"
 * @n_yes_ranges: size of the above array
 * @no_ranges: pointer to an array of regmap ranges used as "no ranges"
 * @n_no_ranges: size of the above array
 *
 * A table of ranges including some yes ranges and some no ranges.
 * If a register belongs to a no_range, the corresponding check function
 * will return false. If a register belongs to a yes range, the corresponding
 * check function will return true. "no_ranges" are searched first.
 */
struct regmap_access_table {
	const struct regmap_range *yes_ranges;
	unsigned int n_yes_ranges;
	const struct regmap_range *no_ranges;
	unsigned int n_no_ranges;
};

The only thing that might be a bit confusing is that combination of “yes” and “no” ranges. In a nutshell, you are expected to provide access tables for your access operations like read and write, and you can use the “yes” and “no” attributes to provide a straight or an inverted logic. For example, you could have a single write-only register, and in that case you could instantiate a write-only access table with that register as “no”, and then assign the table to the rd_table element within the regmap_config structure. That will mean “this is the only register that can’t be read”. If you still don’t get it, continue reading, and I will show you an example a bit later.

Let’s continue with our example. First, we will define the ranges. For that, we will use the regmap_reg_range macro for every read range. If you take a look at the register definitions, you will realize that there are holes i.e. not all addresses are defined, and those holes will act as range delimiters:

static const struct regmap_range dummy1234_read_ranges[] = {
	regmap_reg_range(DUMMY1234_REG_CONF, DUMMY1234_REG_TEMP),
	regmap_reg_range(DUMMY1234_REG_INT1, DUMMY1234_REG_INT2),
	regmap_reg_range(DUMMY1234_REG_ID, DUMMY1234_REG_UID),
};

static const struct regmap_range dummy1234_write_ranges[] = {
	regmap_reg_range(DUMMY1234_REG_CONF, DUMMY1234_REG_CONF),
	regmap_reg_range(DUMMY1234_REG_INT2, DUMMY1234_REG_INT2),
	regmap_reg_range(DUMMY1234_REG_UID, DUMMY1234_REG_UID),
};

Now we can use our ranges to define the access tables and assign them to regmap_config:

static const struct regmap_access_table dummy1234_read_table = {
	.yes_ranges = dummy1234_read_ranges,
	.n_yes_ranges = ARRAY_SIZE(dummy1234_read_ranges),
};

static const struct regmap_access_table dummy1234_write_table = {
	.yes_ranges = dummy1234_write_ranges,
	.n_yes_ranges = ARRAY_SIZE(dummy1234_write_ranges),
};

static const struct regmap_config dummy1234_regmap = {
	//...
	.rd_table = &dummy1234_read_table,
	.wr_table = &dummy1234_write_table,
	//...
};

Now let’s imagine that we have the following set of registers:

#define DUMMY1234_REG_CONF   0x00 //read/write
#define DUMMY1234_REG_STATUS 0x01 //read/write
#define DUMMY1234_REG_TEMP   0x02 //read/write
#define DUMMY1234_REG_INT1   0x03 //read/write
#define DUMMY1234_REG_INT2   0x04 //write-only
#define DUMMY1234_REG_ID     0x05 //read/write

In this case we could opt for the “no” approach:

static const struct regmap_range dummy1234_write_only_ranges[] = {
	regmap_reg_range(DUMMY1234_REG_INT2, DUMMY1234_REG_INT2),
}`;

static const struct regmap_range dummy1234_write_ranges[] = {
	regmap_reg_range(DUMMY1234_REG_CONF, DUMMY1234_REG_ID),
};

static const struct regmap_access_table dummy1234_read_table = {
	.no_ranges = dummy1234_write_only_ranges,
	.n_no_ranges = ARRAY_SIZE(dummy1234_write_only_ranges),
};

static const struct regmap_access_table dummy1234_write_table = {
	.yes_ranges = dummy1234_write_ranges,
	.n_yes_ranges = ARRAY_SIZE(dummy1234_write_ranges),
};

The assignments to regmap_config don’t change because we are still assigning a read and a write table, just with a modified logic:

static const struct regmap_config dummy1234_regmap = {
	//...
	//DUMMY1234_REG_INT2 is the only "no" read register
	.rd_table = &dummy1234_read_table,
	.wd_table = &dummy1234_write_table,
	//...
};

If you still think that the read range is incomplete because no yes_range was defined, I am glad that you don’t believe everything I say. The regmap_check_range_table() function in drivers/base/regmap/regmap.c is in charge of that behavior:

bool regmap_check_range_table(struct regmap *map, unsigned int reg,
			      const struct regmap_access_table *table)
{
	/* Check "no ranges" first */
	if (regmap_reg_in_ranges(reg, table->no_ranges, table->n_no_ranges))
		return false;

	/* In case zero "yes ranges" are supplied, any reg is OK */ <-----READ THIS!
	if (!table->n_yes_ranges)
		return true;

	return regmap_reg_in_ranges(reg, table->yes_ranges,
				    table->n_yes_ranges);
}

If you don’t define any “yes” range, then all accesses for the given operation will be allowed. But if you define a “yes” range, that will not be true anymore, and you will have to provide all registers that allow that access operation.

Keep in mind that the “yes” ranges are way more common than the “no” ones, partially because a lot of hardware does not need to exclude ranges, but also because it is a bit more confusing, and it is easy to get it wrong. I guess that many people provide the “yes” ranges and move on. Anyway, you should be able to handle both scenarios. Who knows if your first driver will be better off with “no” ranges!

The regmap_range structure is not only used for read/write operations, and we are going to see another use case in a moment.

Before finishing this section, I would like to mention that using the structures we just saw is not the only way to define ranges. It is also possible to provide callbacks to check the access rights of the registers:

	bool (*writeable_reg)(struct device *dev, unsigned int reg);
	bool (*readable_reg)(struct device *dev, unsigned int reg);

For this approach, you will have to declare the callbacks, assign them to writeable_reg or readable_reg (also volatile and precious are possible, we will talk about these access rights later) in regmap_config, and program the logic to decide if a register can be accessed that way (often with a switch() or something similar). This way is less elegant than the approach we discussed, and I am not a big fan of it. I believe it existed before the range and access structs were introduced, which are currently preferred for most of the cases. But as you will find several drivers that use this approach, it is good that you know what it is about.

7. Caching

We are now able to access registers like a pro, but do we always have to access the registers? Many of them will keep their value until the driver updates it, so why would we trigger a time-consuming operation like accessing a slow bus when the value has not been modified? Let’s introduce caching to keep on accessing registers like a pro, but only when we have to. In order to support caching, we will need more regmap ranges to specify what registers are updated by the device itself. We will be able to cache registers that are only modified by the driver, because their values should be the last ones the driver wrote. On the other hand, we should not cache registers that the device can update (volatile) because that would invalidate the cache, and we would not notice. Let’s take a look at this new set of registers:

#define DUMMY1234_REG_CONF   0x00 //read/write, not updated by the device
#define DUMMY1234_REG_STATUS 0x01 //read-only, updated by the device
#define DUMMY1234_REG_TEMP   0x02 //read-only, updated by the device
#define DUMMY1234_REG_INT    0x03 //read-write, updated by the device
#define DUMMY1234_REG_ID     0x04 //read-only, constant value

The descriptions simulate the information you would get from the datasheet about those registers. Thanks to that information, we can define some new ranges for the cacheable, volatile, and precious registers. Can you tell to what category belongs every register I defined? If you still can’t, let’s use some logic:

DUMMY1234_REG_CONF: if we can read and write its value, and the device does not update it, the driver has full control. Therefore, this register should be cacheable.
DUMMY1234_REG_STATUS, DUMMY1234_REG_TEMP: those registers are updated by the device, and therefore we should not cache them. They are volatile, and we will need a fresh value for every read operation.
DUMMY1234_REG_INT:
DUMMY1234_REG_ID: constant value, we definitely want to cache this and avoid additional accesses to the hardware over a potentially slow bus.

We are now ready to describe the new range and access table:

static const struct regmap_range dummy1234_volatile_ranges[] = {
	regmap_reg_range(DUMMY1234_REG_STATUS, DUMMY1234_REG_INT),
};

static const struct regmap_access_table dummy1234_volatile_table = {
	.yes_ranges = dummy1234_volatile_ranges,
	.n_yes_ranges = ARRAY_SIZE(dummy1234_volatile_ranges),
};

We are almost ready to update the regmap_config initialization to support caching. Almost? Yes, we still have to define the cache type. Regmap supports different cache types, enumerated in regmap.h as follows:

/*
 * The supported cache types, the default is no cache.  Any new caches
 * should usually use the maple tree cache unless they specifically
 * require that there are never any allocations at runtime and can't
 * provide defaults in which case they should use the flat cache.  The
 * rbtree cache *may* have some performance advantage for very low end
 * systems that make heavy use of cache syncs but is mainly legacy.
 */
enum regcache_type {
	REGCACHE_NONE,
	REGCACHE_RBTREE,
	REGCACHE_FLAT,
	REGCACHE_MAPLE,
};

REGCACHE_NONE is the default cache type i.e. by default, caching is disabled (you don’t need to add .cache_type = REGCACHE_NONE). Usually, you will just follow the description above and use REGCACH_MAPLE unless you have a good reason to use another type.

All things considered, our regmap_config should look like this:

static const struct regmap_config dummy1234_regmap_config = {
	//...
	.volatile_table = &dummy1234_volatile_table,
	.precious_table = &dummy1234_precious_table,
	.cache_type = REGCACHE_MAPLE,
	//...
};

We are ready to move on, but we could also go the extra mile and discuss cache synchronization. This is not a topic that you will see in most of the drivers that use regmap, yet you might stumble upon it from time to time.

7.1. Cache synchronization/bypassing

There are a number of reasons why the regmap cache might require to be synchronized like after returning from a low-power mode or a device reset if those conditions flush the register values. For example, I have seen some hwmon devices that need to be re-initialized after a reset, which invalidates the cache. You can find an interesting real case in drivers/hwmon/ina2xxx.c.

These considerations usually depend on the hardware and the nature of the registers, and it is not something that most drivers will have to consider. But if yours does, you will probably make use of the following functions:

regcache_sync(regmap): the function to synchronize the cache. Obviously, it will ignore volatile registers as they are not cached.
regcache_mark_dirty(regmap): indicate that the device registers have been reset to their default values. This function is usually combined with regcache_sync() to let it know that it will have to update the non-default values from the cache to the device.
regmap_read_bypassed(regmap, REGISTER_ADDRESS, &register_value): to read directly from the device, bypassing the cache. Although it is a very uncommon operation, sometimes it’s useful to access a specific register that might indicate what to do after a reset or a similar event that might have invalidated the cache. If you can avoid using this function and implement a more common approach, please do so. If there is no way around it, consider adding a comment to justify its need in your driver. Otherwise, its usage will be seen as rather suspicious.
regcache_cache_only(regmap, bool enable): if enabled, no hardware access will be carried out by regmap, restricting itself to the cache. This could be useful for register accesses while the hardware is not accessible. The cache will then have to be synchronized when the hardware is accessible again, also setting regcache_cache_only() back to false.

8 Precious registers

There is yet a third scenario that must be observed when describing the device registers: precious registers whose value is automatically updated after reading it, and need special care to avoid unplanned reads. What are unplanned reads? Those that were not requested by the driver. Still, when could that happen? Are we not programming a driver to be in charge of the device? Yes, more or less… but as we will see later, it is possible to bypass the driver and dump (i.e. read) the register values.

A typical example of such precious registers are event/interrupt notifiers that are cleared upon read (to be ready for the next event/interrupt). If someone else reads that register, it will be cleared and the notification will be lost. That’s definitely not what we want as driver developers! Fortunately, we can fix it with the same approach we have been following: regmap_range + regmap_access_table + regmap_config update. In the following example, DUMMY1234_REG_EVENTS is used by the driver to notify events (e.g. interrupts, overflows), and reading its value clears it:

#define DUMMY1234_REG_EVENTS    0x0B //read-only, clear on read

static const struct regmap_range dummy1234_precious_ranges[] = {
	regmap_reg_range(DUMMY1234_REG_EVENTS, DUMMY1234_REG_EVENTS),
};

static const struct regmap_access_table dummy1234_precious_table = {
	.yes_ranges = dummy1234_precious_ranges,
	.n_yes_ranges = ARRAY_SIZE(dummy1234_precious_ranges),
};

static const struct regmap_config dummy1234_regmap_config = {
	//...
	.precious_table = &dummy1234_precious_table,
	//...
};

DUMMY1234_REG_EVENTS will only be read by the driver, allowing us to control the event management. If you are still missing events after declaring a register precious, you will probably have to solve some other kind of issue: hardware, timing, etc. Sorry, mate!

9. Debugging

Once a driver starts controlling a device, you are usually no longer able to access its registers. That makes sense because accessing the registers at that point is very hacky, and you could break many things by playing around with their values. But knowing what values they store is sometimes useful, especially while programming the driver and later on for debugging purposes. Regmap offers a safe solution for those needs: debugfs attributes.

In order to use this feature, we will have to get access to debugfs. Easy task, though: in many systems /sys/kernel/debug/regmap is already visible and available (if at least one device supports regmap). If not, try to mount debugfs like this: mount -t debugfs none /sys/kernel/debug. Then you will be able to see the devices with regmap, hopefully yours among them. For example, my SBC is connected to two I2C devices, one I wrote a driver for a while ago, and the dummy1234 we have been working on:

ls /sys/kernel/debug/regmap/
1-0010-veml6075  1-0029-dummy1234

What do the names mean? In this case, for I2C devices, the first name is the bus address/number (in this case, I2C-1 for both devices), the second is the device address (0x10 and 0x29), and the rest is the name used to initialize the .name member of regmap_config, if a name was defined (empty otherwise).

What’s in there? Let’s see:

ls /sys/kernel/debug/regmap/1-0029-dummy1234
access  name  range  register

These attributes are all read-only and need root access to be read. Let’s see them one by one:

access:: it displays a rather cryptic table with per-register access rights in the following form:

    reg-addr: read write volatile precious

I have populated some register ranges with different access rights, so you can see it in action:

cat access
y y n n
y y n n
y y n n
y y n n
y y n n
y y n n
y y n n
y y n n
y n n n
y n n n
0a: y n n n
0b: y n n n
0c: y n n n
0d: y n n n
0e: y n n n
0f: y n n n
y n y n
y n y n
y n y n
y n y n
y n n n
y n n n
y n n n
y n n y
y n n n

In this example, all registers [0x00-0x18] can be read, only [0x00-0x07] can be written, [0x10-0x13] are volatile, and only 0x17 is precious.

name:: trivial one, the driver name.

cat name
dummy1234

range:: the range of addresses you can access:

cat range
0-16
18-18

Why are there two ranges, and why is the address 0x17 gone? This debugging feature only allows you to read register values, but NO precious registers, as only the driver should be able to access them. As we saw before, 0x17 is precious, and it’s good that it cannot be accessed via debugfs.

registers:: the register values at a given point:

cat registers
01
80
00
00
ff
ff
00
00
00
00
0a: 00
0b: 00
0c: 00
0d: 00
0e: 00
0f: 00
00
00
00
00
01
00
00
ff

As you can see, 0x17 is missing, which is actually what we want. But the rest of the readable registers are available, and knowing their values is great for debugging and way cleaner than hacking printk() functions all over the place.

10. Other features

I bet you have already noticed that regmap covers many different use cases, and for that it has to offer many different features. Even though we have already seen what you will be using ~99% of the time, there are some other features that are worth mentioning, as they could help you in some specific situations.

10.1 Register defaults

Regmap allows drivers to provide default register values for the cache (not to write default values to the hardware!) during its initialization, and they are not meant to be used without it. This feature is extensively used in some subsystems (e.g. sound codecs) due to the nature of their devices, but it is not something that most of the drivers implement.

There are two different approaches to provide default values, and unfortunately, they are poorly documented. But here I am to help you out! You can either provide a set of pairs [address-value] by means of the reg_defaults and num_reg_defaults attributes, or values for a contiguous range via reg_defaults_raw and num_reg_defaults_raw, all of them within regmap_config.

Let’s see some upstream code using the different approaches, first with reg_defaults, which makes use of the reg_default struct to provide the [address-value] pairs:

// drivers/leds/flash/leds-lm3601x.c

static const struct reg_default lm3601x_regmap_defs[] = {
	{ LM3601X_ENABLE_REG, 0x20 },
	{ LM3601X_CFG_REG, 0x15 },
	{ LM3601X_LED_FLASH_REG, 0x00 },
	{ LM3601X_LED_TORCH_REG, 0x00 },
};
//...
static const struct regmap_config lm3601x_regmap = {
	//...
	.reg_defaults = lm3601x_regmap_defs,
	.num_reg_defaults = ARRAY_SIZE(lm3601x_regmap_defs),
	//...
};

And now with reg_defaults_raw:

// sound/soc/codecs/jz4770.c

static const u8 jz4770_codec_reg_defaults[] = {
	0x00, 0xC3, 0xC3, 0x90, 0x98, 0xFF, 0x90, 0xB1,
	0x11, 0x10, 0x00, 0x03, 0x00, 0x00, 0x40, 0x00,
	0xFF, 0x00, 0x06, 0x06, 0x06, 0x06, 0x00, 0x00,
	0x00, 0x00, 0x00, 0x00, 0xFF, 0x00, 0x00, 0x34,
	0x07, 0x44, 0x1F, 0x00
};

static const struct regmap_config jz4770_codec_regmap_config = {
	//...
	.reg_defaults_raw = jz4770_codec_reg_defaults,
	.num_reg_defaults_raw = ARRAY_SIZE(jz4770_codec_reg_defaults),
	//...
};

Note that the first approach is way, way more common than the second. What you will actually find slightly more often is num_reg_defaults_raw without reg_defaults_raw. That is the way to tell regmap that default values can’t be provided by the driver, and num_reg_defaults_raw registers will be directly read from the hardware to initialize the cache. Usually, the driver will configure the device in a certain way in the probe function, and setting defaults won’t make much sense, but that will always depend on the device needs and what is more optimal in every case.

10.2. Patching

This feature allows updating multiple registers efficiently in a single operation, which is useful when initializing hardware with a predefined set of values. When would you need that? The function documentation gives you an example: Typically this is used to apply corrections to the device defaults on startup, such as the updates some vendors provide to undocumented registers. If you ever find yourself requiring such feature, you will need a reg_sequence again like we did for multi-register accesses (which is what this one is doing too, but always directly to the hardware), and its size:

ret = regmap_register_patch(regmap, reg_sequence, num_regs);

Although you should know by now how to use this function, let’s see an upstream use case:

// sound/soc/codecs/cs35l32.c

/* Current and threshold powerup sequence Pg37 in datasheet */
static const struct reg_sequence cs35l32_monitor_patch[] = {
	{ 0x00, 0x99 },
	{ 0x48, 0x17 },
	{ 0x49, 0x56 },
	{ 0x43, 0x01 },
	{ 0x3B, 0x62 },
	{ 0x3C, 0x80 },
	{ 0x00, 0x00 },
};
//...
static int cs35l32_i2c_probe(struct i2c_client *i2c_client)
{
	//...
	ret = regmap_register_patch(cs35l32->regmap, cs35l32_monitor_patch,
				    ARRAY_SIZE(cs35l32_monitor_patch));
	//...
}

10.3. Polling

Polling in regmap refers to the process of repeatedly reading a register until a specific condition is met. This is useful for situations where a hardware event must be waited on, such as checking if a device is ready, waiting for a status bit to change, or synchronizing with hardware.

Regmap provides regmap_read_poll_timeout() to simplify polling logic, automatically handling retries and timeouts. This function is particularly useful when waiting for a device to complete an operation without requiring an interrupt. It avoids busy loops and ensures that the wait time is limited. The macro definition looks like this:

/**
 * regmap_read_poll_timeout - Poll until a condition is met or a timeout occurs
 *
 * @map: Regmap to read from
 * @addr: Address to poll
 * @val: Unsigned integer variable to read the value into
 * @cond: Break condition (usually involving @val)
 * @sleep_us: Maximum time to sleep between reads in us (0 tight-loops). Please
 *            read usleep_range() function description for details and
 *            limitations.
 * @timeout_us: Timeout in us, 0 means never timeout
 *
 * This is modelled after the readx_poll_timeout macros in linux/iopoll.h.
 *
 * Returns: 0 on success and -ETIMEDOUT upon a timeout or the regmap_read
 * error return value in case of a error read. In the two former cases,
 * the last read value at @addr is stored in @val. Must not be called
 * from atomic context if sleep_us or timeout_us are used.
 */
#define regmap_read_poll_timeout(map, addr, val, cond, sleep_us, timeout_us) \
({ \
	int __ret, __tmp; \
	__tmp = read_poll_timeout(regmap_read, __ret, __ret || (cond), \
			sleep_us, timeout_us, false, (map), (addr), &(val)); \
	__ret ?: __tmp; \
})

// would be used like this:
ret = regmap_read_poll_timeout(regmap, reg_address, read_val, condition, sleep_us, timeout_us);

This macro (and similar ones in include/linux/regmap.h to read a regfield, for atomic operations, etc.) is useful to wait for a condition i.e. usually an expected value in the register before taking some action. Again, some real upstream code to illustrate it:

// drivers/clk/clk-si5341.c

/* wait for device to report input clock present and PLL lock */
err = regmap_read_poll_timeout(data->regmap, SI5341_STATUS, status,
	!(status & (SI5341_STATUS_LOSREF | SI5341_STATUS_LOL)),
       10000, 250000);

10.4. Raw access

While regmap provides a structured API for reading and writing registers, sometimes direct access to raw register values is needed. This is where regmap_raw_read() and regmap_raw_write() come in, allowing bulk transfer of data instead of individual register accesses. This is useful when dealing with block transfers, such as firmware loading or reading sensor data in bulk.

Raw access is particularly helpful when a device exposes a memory-mapped interface or when performance is critical. It reduces the overhead of multiple function calls and allows for efficient data transfers. Once again, you should have good reasons to require such optimizations, and even though there are multiple drivers that use it (oh surprise, a few of them under sound/soc/codecs, the king of uncommon regmap features 😝), it is not as common as the previous access functions we discussed.

In order to use them, you will have to provide a void pointer for the payload and its length:

/**
 * regmap_raw_write() - Write raw values to one or more registers
 *
 * @map: Register map to write to
 * @reg: Initial register to write to
 * @val: Block of data to be written, laid out for direct transmission to the
 *       device
 * @val_len: Length of data pointed to by val.
 *
 * This function is intended to be used for things like firmware
 * download where a large block of data needs to be transferred to the
 * device.  No formatting will be done on the data provided.
 *
 * A value of zero will be returned on success, a negative errno will
 * be returned in error cases.
 */
int regmap_raw_write(struct regmap *map, unsigned int reg,
		     const void *val, size_t val_len);
/**
 * regmap_raw_read() - Read raw data from the device
 *
 * @map: Register map to read from
 * @reg: First register to be read from
 * @val: Pointer to store read value
 * @val_len: Size of data to read
 *
 * A value of zero will be returned on success, a negative errno will
 * be returned in error cases.
 */
int regmap_raw_read(struct regmap *map, unsigned int reg, void *val,
		    size_t val_len);

If you are going to use these functions, take a look at existing drivers: you will notice that they often require uncommon handling of the cache with explicit block synchronization and flushing. I have never had to use anything like that so far, and even though it must be interesting, it is most probably an easy way to mess up and introduce bugs 😆

10.5. Locking strategies

By default, regmap handles register access locking by an internal mutex to prevent race conditions when multiple threads or contexts access the same registers. That’s a great default behavior, and usually what we want to have. However, in some cases, a different locking approach can be required, like using a spinlock instead of a mutex or using a custom lock/unlock callback.

For example: if you need justified fast I/O accesses, the fast_io attribute of regmap_config is the boolean you will have to set (it is false by default, of course) to use spinlocks instead of mutexes. It is as easy as assigning .fast_io = true; within regmap_cofig, but it has all the implications of using spinlocks instead of mutexes, so use it carefully and when it is absolutely required.

If you have to overwrite the internal regmap_lock() and regmap_unlock() functions, you will have to provide callbacks with the following signature:

typedef void (*regmap_lock)(void *);
typedef void (*regmap_unlock)(void *);

Again, these functions will be assigned to .lock and .unlock in regmap_config. And again, you should have good reasons to require your own locking mechanism, like requiring some specific calls like raw_spin_lock_irqsave() and raw_spin_unlock_irqrestore() (see drivers/gpio/gpio-104-dio-48e.c for a real example).

Let’s call it a day. If you have read the whole article until the end, you probably know more about regmap that you will ever need, and you are definitely ready to use basic and advanced features of this amazing API. Nevertheless, if you would like me to discuss yet another feature (believe it or not, regmap has more to offer), please let me know, and I will update the article with that obscure feature you desperately need

I hope you will use regmap from day 1 on in your device drivers!

Linux Drivers 2 - Basic layout and Device Model

Thu, 23 Jan 2025 17:00:00 +0000

In this episode we are going to actually write a device driver and interact with a device. What a milestone! And more importantly, we will clarify the modern Linux device model and the general structure of a device driver. And last, but not least, this first device driver will serve as a good base to build upon in the next episodes. For those who still don’t have any target hardware, I have added an example that you can test on your computer and grasp some key concepts that will stay with us for the whole series.

For those who already have real hardware in their hands, we will first check that it is ready to be used. Then we will discuss the basic layout of a device driver in great detail to understand why things are the way they are, and finally we will take a look at some simple drivers for both I2C and platform devices. I hope I managed to mix theory and praxis for everyone to get something valuable out of this article!

I will assume that you have read the first episode, and it would be great if you have ever compiled Linux, but not mandatory. Ready? Let’s go…

Content:

0. Is the device alive?
1. Driver layout
2. Probe and remove functions? No init and exit?
3. Our first driver template
4. Kconfig and Makefile
5. Building the module
5. Testing the driver
6. Revisiting init/exit vs probe/remove and platform devices

0. Is the device alive?

First, you should make sure that the device is operational, or otherwise you will be looking for non-existent bugs in the code, and that will be highly time-consuming. Depending on the device, there are several approaches to make sure that it is properly powered and connected to the system. For example, your device may provide some indicators like LEDs connected to the power supply. That’s nice to have in a prototype, but seldom the case unless you got a breakout board or evaluation kit, where they are rather common. Let’s assume we have none of that.

For those who have some hardware background, the most reliable way is measuring continuity and voltage levels with proper equipment like a multimeter and/or an oscilloscope. The former is very cheap, and all tinkerers should have one. Just make sure that there are no short circuits, and then check that the connections are alright. Using colored wires will also help, as you will identify wrong connections much faster. If you can power on the device with a dedicated power supply before connecting it to the system, that’s always a safer approach, especially if you limit the amount of current the supply delivers. After the previous checks (no short circuits and everything is connected as it should), you should be ready to switch on the system and measure the power supplies to make sure that they are stable and have the right levels. If they don’t, the device might be demanding too much power, and you will have to reconsider your power subsystem. Having said that, if you start small i.e. with simple devices, that problem will be very rare.

If you don’t know how to measure signals, that’s (often) not the end of the world. An evaluation board is usually ready to be connected without setting fire to the entire system, and low-power devices will be seldom able to damage anything… maybe a GPIO/fuse/regulator if you connected something wrong 😆

Now let’s assume that everything is properly powered and connected. Linux does not know anything about the device, does it? Yet there are tools that can tell you if there is something connected to the system. For example, I showed you how to detect I2C devices in this article, and you will find other tools for different communication protocols. In that article I also mentioned another interesting option for serial buses, a logic analyzer, that is also a very powerful while debugging.

If your device should do something visual upon initialization (e.g. a display with backlight), you can simply connect the reset/enable pins to GPIOs and control them from userspace (very easy to do with most SoCs, trivial with Raspberry Pi). If you see some signs of life, you are on the right way!

And remember, if you see smoke, switch off the system immediately! It will be too late for some pieces of the system, but hopefully you just burnt some cheap regulator that you could replace without major issues. Adding fuses to your designs (many SoCs have them already) is always a wise idea, especially if the current consumption is expected to be high. The devices I am going to use are all low-power, and that won’t be an issue, but keep it in mind for your particular case. And please: always be careful while working with electric systems. I have gotten a few electric shocks in my life, and although they were never severe, I don’t miss getting another one ⚡

Great, now we are all set to focus on the software.

1. Driver layout

Let’see first divide a simple driver into general sections to get the picture:

1.1. License

TL;DR; Boilerplate and one-liner: copy and paste at the top of the driver.

The license at the top of the file specifies the legal terms under which the code can be used, modified, and distributed. The SPDX-License-Identifier is a standardized format that declares the license of the file, making it machine-readable and ensuring consistency across projects. In this case, GPL-2.0+ means the code is licensed under the GNU General Public License version 2 or later, which is commonly used in open-source projects like the Linux kernel.

1.2. Description

This section provides a brief overview of the functionality or purpose of the code. It often includes references to relevant documentation, such as datasheets or application notes, to help developers understand the hardware or software being supported. Additionally, it typically includes a copyright notice and contact information for the author or maintainer. It is not always provided, but it does not hurt, it is informative, and easy to implement 😅

1.3. Headers

TL;DR; The headers you need, like in any C program. Apply the same rules you would apply to any C program.

A list of the header files necessary for the code to interact with specific kernel or library functions. These files may provide access to APIs for hardware communication, memory management, data structures, or other system resources that the code requires. There is a tendency to add too many headers (not too few because the kernel bots will catch that before the patch gets applied), often due to copying and pasting without paying much attention. Even if some required headers are nested in those that have already been added, sometimes you will see them explicitly added to this section. But as guards are used in all headers, that is not so critical, and as far as I can tell, very few people really cares.

As we are attempting to write clean code, I would recommend you to add only what you need and check if you are not including anything twice, because if we want to explicitly add all headers we need, we would have to go all the way down to the mos basic headers, and that would be silly.

1.4. Definitions

This section defines constants, macros, and bit masks used throughout the code. These definitions simplify working with hardware registers and other low-level aspects by providing meaningful names for values and operations. It may also include other macros for manipulating specific bits or fields within a register, making the code more readable and maintainable.

1.5. Global data

Often global structures that define key data models used throughout the code. These structures typically represent devices, configurations, or other entities the code interacts with. They may include fields for hardware-specific parameters, function pointers for initialization or handling tasks, and other necessary information. These structures help organize the code and provide a flexible way to extend functionality for different hardware or use cases.

1.6. Functions

TL;DR; the core of the driver, where we bring the hardware to life.

The functions section contains the code that implements the driver’s behavior. Each function is responsible for a specific task, such as initializing hardware, handling interrupts, reading or writing data, and managing power state. Some of the most typical ones that you will find here are the probe function, that we will discuss soon, and the resume/suspend functions to manage power. What you will not find so often (at least not explicitly) in mainline Linux drivers, even though those are the only functions that 95% of the online sources talk about, are the init and exit functions. Keep on reading to know why.

These functions are typically static (if they are needed somewhere else, they should be moved to a subsystem/driver core and declared in a header file), and named to reflect their purpose, more ore less in the form “driver_name_purpose()”. That pattern is useful for debugging purposes (i.e. unequivocal and easy to find) as well as to avoid conflicts if they are ever exported.

1.7. Driver initialization

In this section, the driver is initialized by assigning key components such as the driver’s name, the probe function, and other necessary operations like power management functions. The initialization section also typically registers the driver with the kernel using structures like platform_driver or i2c_driver. These structures tell the kernel how to interact with the driver and specify which functions to call at various stages of the device lifecycle, like when the device is detected, initialized, or removed. Around this section you will usually find the compatible strings and the device-specific data assignments e.g. if the driver supports multiple devices with slight differences.

1.8. Module macros

The macros section includes essential metadata like the author, description, and license of the driver. Common macros such as MODULE_AUTHOR, MODULE_DESCRIPTION, and MODULE_LICENSE provide important information about the driver, helping maintain consistency and visibility. These macros are also used by the kernel build system to include the module’s metadata and ensure proper licensing and attribution. In some cases you will find macros like MODULE_IMPORT_NS and variants of that, which are used to include required namespaces. We will cover that in this series, but not in this episode to keep things manageable.

This pattern is not carved in stone, but as you’ll see, it tends to repeat quite frequently, especially among new drivers. Sometimes more complex drivers are divided into multiple files (e.g. a core, per-bus-files, and a header file), and some of these sections will be spread across them, but in general, even in those cases the layout will be pretty similar.

2. Probe and remove functions? No init and exit?

Let’s discuss a very common misconception among beginners before we carry on. If you have very basic experience with kernel modules, or you have followed trivial tutorials about your first device driver, you might be wondering why I am renaming the init and exit functions. The init function is called when the module is loaded, and the exit function when it is unloaded. Those remove and probe functions will be the same with some sanity checks on top, right? I have even talked to one person who had written two device drivers, and still believed that init and probe were equivalent. Well… they are not, I am afraid.

In the Linux kernel, traditional drivers often use init and exit functions to handle module loading and unloading, respectively. These functions are called when the driver is inserted into or removed from the kernel, and they are responsible for setting up or tearing down global resources, such as registering devices or subsystems. However, modern device drivers that integrate with the Linux device model primarily rely on probe and remove functions, which are tied to specific devices rather than the driver as a whole.

The probe function is called when the kernel matches a device to the driver, and it handles device-specific initialization, like allocating resources or initializing hardware. The remove function complements this by releasing those resources when the device is removed or the driver is unbound. While init and exit still exist and often handle driver-wide tasks (like registering with a bus subsystem or cleaning up global data), they now commonly delegate per-device initialization and cleanup to probe and remove. This separation allows for dynamic device management, hot-plugging, and better scalability in modern systems.

I will mention that there is some official documentation about this topic, but it is a bit scarce in details for beginners, and it then gets into function descriptions, so please keep on reading.

In summary, you will not find __init and __exit attributes in many modern drivers, only in subsystem cores, and in drivers that want to accomplish some common task that will affect all instances of the driver. For example, many devices use Cyclic Redundancy Check (CRC) to detect data corruption. If the polynomial used to build the CRC table is the same for all devices that the driver supports, why should every call to probe() generate a new CRC table? We could instead generate it once in the init function, and have it available as a global table for all instances. Is that only theory? No, let’s see an example from the ad74115.c (an ADC/DAC under IIO):

#define AD74115_CRC_POLYNOMIAL			0x7
DECLARE_CRC8_TABLE(ad74115_crc8_table);
//...
static int __init ad74115_register_driver(struct spi_driver *spi)
{
	crc8_populate_msb(ad74115_crc8_table, AD74115_CRC_POLYNOMIAL);

	return spi_register_driver(spi);
}

The init function just populates the CRC table, and then registers the driver. Easy! And as the global variable will be released automatically when the module gets unloaded, there’s not even need for an explicit exit function. Will you never find exit functions out of subsystem core-level code and old drivers? Sometimes you will, if you have to explicitly release some resources you allocated in the init function.

As you can see, the driver is registered for a specific bus (SPI), as that is the Linux driver model, where drivers are bound to devices through a common bus they both support. The bus acts as an intermediary, matching drivers and devices based on compatibility. For instance, in this case, the SPI bus ensures the ad74413r driver is only loaded for devices that communicate using the SPI protocol. What if there is no bus? Then the driver uses the platform bus, a generic mechanism on Linux to handle devices that are not attached to a physical bus.

I know, this chapter had a little bit of theory, and you might still be a bit confused. If you did not get these concepts yet, don’t worry, we will revisit them again later when I provide an example that you can test on your computer without additional hardware. But first, let’s see an example of a simplified upstream driver, and how to build it.

3. Our first driver template

We know what a typical driver should look like, and we have understood why we will have a probe function, and maybe no init function. Then we should be ready to write our first, simple driver!

For our first driver, we are going to use a very simple layout with all the sections we discussed, but reduced to the bare minimum to be able to communicate with a real device. In this case, I will use the Unbranded dummy1234, a completely new device that communicates via I2C, that is only able to provide its ID, which is stored in an internal register.

Believe it or not, it just took 66 (including 13 empty lines!) lines of code to implement a Linux device driver able to communicate with real hardware over I2C. And that includes the license, a multi-line description, and even some dispensable pointers to show basic memory allocation and have a data struct ready to add more interesting stuff in the future.

The following snippet is the code of our first driver template with some comments inline. If you would like to copy the clean version without comments, you will find it here.

//---------------------------- 1. LICENSE -----------------------------------//
// SPDX-License-Identifier: GPL-2.0+
//---------------------------- 2. DESCRIPTION -------------------------------//
/*
 * UNBRANDED DUMMY1234 ID Provider
 *
 * Copyright (c) 2025, Hacker Bikepacker <hacker.bikepacker@gmail.com>
 */
//---------------------------- 3. HEADERs -----------------------------------//
#include <linux/err.h>
#include <linux/i2c.h>
#include <linux/module.h>

//---------------------------- 4. DEFINITIONS -------------------------------//
#define DUMMY1234_REG_ID 0x14

//---------------------------- 5. GLOBAL DATA -------------------------------//

//These pointers are not really necessary, but somehow convenient to only pass
//a pointer to this struct around.
struct dummy1234_data {
	struct i2c_client *client;
	struct device *dev;
};

//---------------------------- 6. FUNCTIONS ---------------------------------//
static void dummy1234_read_id(struct dummy1234_data *data)
{
	int ret;
	u8 reg;

	//We (master) receive 1 byte (the register) from the slave (dummy1234)
	ret = i2c_master_recv(data->client, &reg, 1);
	if (ret < 0) {
		//don't use printk(), and if possible, pr_() either.
		//dev_{dbg,info,notice,warn,err,alert,crit,emerg}() produce better logs
		//because they are bound to a device (data->dev)
		dev_err(data->dev, "failed to read ID\n");
		return;
	}
	dev_info(data->dev, "ID = 0x%02x\n", reg);
}

//Our entry point, the initialization is carried out here
static int dummy1234_probe(struct i2c_client *client)
{
	struct device *dev = &client->dev;
	struct dummy1234_data *data;

	//Allocate memory for the data struct. devm_ means "device managed",
	//and when the device is gone, the memory will be automatically released.
	//GFP_KERNEL is a GFP (Get Free Pages) flag for typical kernel-internal
	//allocation. It is the most common flag in such drivers.
	data = devm_kzalloc(dev, sizeof(*data), GFP_KERNEL);
	if (!data)
		return -ENOMEM;

	data->client = client;
	data->dev = dev;

	dummy1234_read_id(data);

	return 0;
}

//---------------------------- 7. DRIVER INITIALIZATION ---------------------//

//This string will be used to match an i2c device with the driver. More info
//about this mechanism in my article "I2C on Linux".
//The empty element at the end is a guard to identify the end of the array.
static const struct i2c_device_id dummy1234_id[] = {
	{ "dummy1234" },
	{ }
};
MODULE_DEVICE_TABLE(i2c, dummy1234_id);

//This i2c driver is initiaized here: module name, probe, i2c, and i2c ID.
//We will initialize more members in the future ;)
static struct i2c_driver dummy1234_driver = {
	.driver = {
		.name = "dummy1234",
	},
	.probe = dummy1234_probe,
	.id_table = dummy1234_id,
};
module_i2c_driver(dummy1234_driver);

//---------------------------- 8. MODULE MACROS -----------------------------//
MODULE_AUTHOR("Hacker Bikepacker <hacker.bikepacker@gmail.com>");
MODULE_DESCRIPTION("DUMMY1234 ID provider");
//For simplicity GPL is always used even if it is GPL v2
MODULE_LICENSE("GPL");

4. Kconfig and Makefile

Forget it, we are not going to write our own Makefile, and we won’t need uname -r. That’s again something you usually need for out-of-tree code to compile against a certain kernel. For in-tree code like ours (you still don’t know, but our code is indeed in-tree), there is already an infrastructure that we can (i.e. must) use.

In a nutshell, there is a hierarchical structure of Makefiles (i.e. building rules) that Kbuild (the kernel builder) follows to, depending on the kernel configuration (by means of the Kconfig files), include certain objects into the building process (built-in, or as a loadable module). We, driver developers, are more interested in the Kconfig and Makefile within the folder our driver resides. For example, if our driver will reside under drivers/misc, we will have to edit the Kconfig and Makefile under that path to add the building rule and the configuration option to include our driver. Given that dummy1234 has a rather random functionality, we will actually add it to that subfolder.

Ok, so what do we have to do to make things work and build our driver? Usually not more than opening the two files, copying from an existing driver, pasting, and renaming. First, we will add the configuration option to drivers/misc/Kconfig: a relatively short entry with the following structure:

config [MODULE]
	(usually) tristate "Title"
	depends on [MODULE] (one dependency per line)
	select [MODULE] (one dependency per line)
	default [condition] (only if there is a clear default)
	help
	  Say Y here if you want to build a driver for the [device]

	  To compile this driver as a module, choose M here: the
	  module will be called [module_name].

The fields are self-explanatory, yet there are some things to consider:

tristate means that the config accepts three possible states: not selected (n), module (m), and yes (y: built-in). Most drivers are tristate because they can be modules and built in, but there are more possibilities like bool (yes/no), int, or string. If they must be built-in in all cases, then you won’t even find an entry in the Kconfig, because it will be configured like that in the Makefile.
module_name is the name we assigned when we initialized the driver, in our case dummy1234.

Our driver only depends on I2C to run, and it does not select any additional module. There is no special need for a default, and we do want to have a tristate config. Therefore, we can just add the following entry to the Kconfig in alphabetical order:

config DUMMY1234
	tristate "DUMMY1234 ID provider"
	depends on I2C
	help
	  Say Y here if you want to build a driver for the Unbranded
	  Dummy1234 ID provider.

	  To compile this driver as a module, choose M here: the
	  module will be called dummy1234.

As you can see, the explanation was more heavy going than the pretty straightforward implementation 😂

What about the Makefile? Good news: in many cases, it is a single-liner. Just copy the rule from a different driver, rename the bit after CONFIG_ to what you added after config in the Kconfig, and rename the object accordingly:

obj-$(CONFIG_DUMMY1234)		+= dummy1234.o

After adding this line, we will be ready to configure our module: our tristate CONFIG_DUMMY1234 can be either m (obj-m, build a module, what you usually see in a Makefile for out-of-tree modules), n (not included), or y (obj-y: built-in). That’s all we need to know for now, but if you are interested in the build process and the elements involved, please read the official documentation for more detailed information.

5. Building the module

We are all set to build the kernel! To keep things simple, we are going to build a very small kernel with minimal configuration that will build in a few seconds:

Set all configurations to n: make allnoconfig. We just want to build our driver as fast as possible!
Start menuconfig (old-school user interface): make menuconfig.
Select the only dependency we have, I2C. You can either navigate through the menus (Device Drivers → I2C support ---> → I2C support), or simply look for it: press /, type I2C (or i2c, it is not case-sensitive) and press the number you see on the left. In this case, it will be 1 because it is the first entry on the list.
Select DUMMY1234. Feel free to try m (you will also have to select MODULES for that) and y.
Save and Exit.

The moment of truth:

$ `make -j$(nproc)`
...
CC      drivers/misc/dummy1234.o
...
Kernel: arch/x86/boot/bzImage is ready

Happy days! You could also try make M=drivers/misc/ clean && make C=1 -j$(nproc) to clean the misc/ directory and check the quality of your code with the sparse static analysis tool. Then you will see an additional line in the logs like CHECK drivers/misc/dummy1234.c. Another useful tool is smatch, which I covered in this article.

5. Testing the driver

In order to cover all hardware combinations, or at least the most common ones, this section (which is an add-on, not really part of the scope of the article) would have to be way longer than the rest of the sections combined. Therefore, I will use this section to test the driver as it is in my simple setup with a Raspberry Pi, and I will give you some hints to build something similar for your particular case. Moreover, I wisely chose an I2C device for the example to offer much more detailed information thanks to my article about I2C that I previously mentioned, and that will be handy here as well.

The first thing we are going to do is build a proper kernel tailored for the target system. I have explained the process for a Raspberry Pi here, so I won’t repeat myself. If you are using a different platform, you will need to find out how to build and load your own kernel for it. If you have a common platform, it will not be that difficult, and you will have to do it anyway for your budding career as a Linux developer 😉

Once you have built the kernel with the new little driver (built-in or as a module), you will be ready to test if it works. In my case, I have selected the module as built-in (y, m will work too), and I have connected an I2C device (it does not matter which one, and I decided not to advertise any specific brand if they don’t support me) whose address is 0x29, and stores its ID in the register 0x14 (hence why I chose that number for the simple driver). According to the datasheet, 0x01 should be read from that register:

Let’s see what we get!

5.1. Via sysfs attributes

If you know nothing about devicetrees and overlays, you should learn about those topics as soon as possible, and I have these two articles for you to read: devicetree for Raspberry Pi and dt-bindings. For now, we will tell the system about the device manually, via sysfs attributes.

For a detailed description of the following steps, please visit my article about I2C on Linux I mentioned before. Basically, I am checking that the driver gets assigned to the device, and that it reads the ID from the device. Of course, you could read any register as long as the device is ready to operate without previous configurations.

$ i2cdetect -y 1
1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
                        -- -- -- -- -- -- -- --
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
-- -- -- -- -- -- -- -- -- 29 -- -- -- -- -- --
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
-- -- -- -- -- -- -- --
pi@raspberrypi:~ $ dmesg | grep dummy1234
pi@raspberrypi:~ $ sudo bash -c "echo dummy1234 0x29 > /sys/bus/i2c/devices/i2c-1/new_device"
pi@raspberrypi:~ $ i2cdetect -y 1
1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
                        -- -- -- -- -- -- -- --
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
-- -- -- -- -- -- -- -- -- UU -- -- -- -- -- --
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
-- -- -- -- -- -- -- --
pi@raspberrypi:~ $ dmesg | grep dummy1234
[  117.016840] dummy1234 1-0029: ID = 0x01
[  117.016995] i2c i2c-1: new_device: Instantiated device dummy1234 at 0x29

Wonderful! But do we have to manually register the device every time? Obviously not, and the devicetree is the standard solution in embedded systems.

5.2. Via devicetree (DT)

I already mentioned the documents you have to read first if you know nothing about this topic, so I will just show you the node you need to add to your DT under the I2C bus node for our simple device:

dummy1234: dummy1234@29 {
	compatible = "dummy1234";
	reg = <0x29>;
	status = "okay";
};

Use the right I2C address for your device, load your new DT, and the driver will be automatically matched with the device. Cool!

If you have some experience with DTs, you might be wondering why I did not add an of_device_id table to list the compatible, which in this case would be “unbranded,dummy1234”. The answer is for simplicity. The i2c_device_id table will be used as a fallback, so it will work anyway, but a proper driver should have both tables, and if the manufacturer has not been added to the list yet, the series with the driver should first include a patch that adds it to avoid warnings when compiling the bindings. We will discuss such topics in another article.

5.3. Suggestions for different hardware

If you do have a device to play with, and it does not have an I2C protocol, I would recommend you to go for the DT approach without thinking too much about other options. You will save time twice: first because you won’t have to learn how to access the device from userspace, and then because eventually you will need a DT node anyway, either for production or just for ease of use. As I said before, one of your first tasks after bringing up your SBC should be learning how to load a new DT and its support for DT overlays.

For devices that use a different protocol like SPI, you will be able to adapt the simple driver by using SPI structures and functions, but the general structure will look very similar.If your device does not have a bus, the next section will help.

6. Revisiting init/exit vs probe/remove and platform devices

If you have made it to this section, you basically know how to program a Linux device driver, and you should be ready to add more complexity and functionality. I will help you with that in the next episodes, but before we come to an end, I would like to give you another simple example that you can try on your computer without any additional hardware. I hope it will help you understand better the difference between init and probe, exit and remove, and what a platform device is.

This time I have programmed an example with two kernel modules: a device generator, and a device driver. The former will make use of init and exit to register and unregister a platform device, and the latter will contain the probe and remove functions associated with a matched device (the one that the other module registered).

The device generator looks more like the kernel modules of the trivial examples you might have already seen: it has entry and exit functions, and some operations are carried out within them. But these operations do something more interesting than just calling printk(): they register and unregister a platform device. As this example is intended to be run with no additional hardware, a platform device is great because there is no need for a real bus to connect with, and we don’t need to simulate hardware either.

But please, don’t leap to the conclusion that platform stuff is only meant for legacy drivers, misc and virtual hardware. There are many modern devices (especially in embedded systems) that don’t have access to a communication bus, and they only need a couple of signals (e.g. a reset signal) to operate. For example, some on-chip USB hubs partially fall into this category, and they require two different devices to work: a platform device, and a USB device. I have added support for such devices to the mainline kernel in the past, and we will discuss such cases in a future episode. Memory-mapped devices often fall into this category as well, and you will see that even your PC has multiple platform devices under /sys/devices/platform/. If you want to learn more about platform devices and drivers, take a look at the official documentation.

Now let’s get back to our example. This is the code of the platform device generator, which by now should be self-explanatory:

// SPDX-License-Identifier: GPL-2.0+
/*
 * UNBRANDED DUMMY1234 Device Generator
 *
 * Copyright (c) 2025, Hacker Bikepacker <hacker.bikepacker@gmail.com>
 */

#include <linux/device.h>
#include <linux/module.h>
#include <linux/mod_devicetable.h>
#include <linux/platform_device.h>

static struct platform_device *dummy1234_dev;
static const char *device_name = "dummy1234_device";

static int __init dummy1234_init(void)
{
	pr_info("%s: registering %s\n", __func__, device_name);

	dummy1234_dev = platform_device_register_simple(device_name, -1, NULL, 0);
	if (IS_ERR(dummy1234_dev)) {
		pr_err("failed to register %s\n", device_name);
		return PTR_ERR(dummy1234_dev);
	}

	return 0;
}

static void __exit dummy1234_exit(void)
{
	pr_info("%s: unregistering %s\n", __func__, device_name);

	platform_device_unregister(dummy1234_dev);
}

module_init(dummy1234_init);
module_exit(dummy1234_exit);

MODULE_AUTHOR("Hacker Bikepacker <hacker.bikepacker@gmail.com>");
MODULE_DESCRIPTION("DUMMY1234 DEVICE GENERATOR");
MODULE_LICENSE("GPL");

The second module is basically the one we programmed for an I2C device, but adapted for a platform device. Key for the matching is the device_id, which is the name of the device the other module generates. It also includes a remove function to illustrate when it comes into action, and that will clearly show when it would be necessary:

// SPDX-License-Identifier: GPL-2.0+
/*
 * UNBRANDED DUMMY1234 ID Provider
 *
 * Copyright (c) 2025, Hacker Bikepacker <hacker.bikepacker@gmail.com>
 */

#include <linux/device.h>
#include <linux/module.h>
#include <linux/mod_devicetable.h>
#include <linux/platform_device.h>

struct dummy1234_data {
	struct device *dev;
};

static void dummy1234_read_id(struct dummy1234_data *data)
{
	dev_info(data->dev, "ID = 1234 (dummy)\n");
}

static int dummy1234_probe(struct platform_device *pdev)
{
	struct dummy1234_data *data;

	dev_info(&pdev->dev, "Calling %s :D\n", __func__);

	data = devm_kzalloc(&pdev->dev, sizeof(*data), GFP_KERNEL);
	if (!data)
		return -ENOMEM;

	data->dev = &pdev->dev;

	dummy1234_read_id(data);

	return 0;
}

static int dummy1234_remove(struct platform_device *pdev)
{
	dev_info(&pdev->dev, "Device removed!\n");

	return 0;
}

static const struct platform_device_id dummy1234_id_table[] = {
	{ "dummy1234_device" },
	{ }
};

MODULE_DEVICE_TABLE(platform, dummy1234_id_table);


static struct platform_driver dummy1234_driver = {
	.driver = {
		.name = "dummy1234_driver",
	},
	.probe = dummy1234_probe,
	.remove = dummy1234_remove,
	.id_table = dummy1234_id_table,
};

module_platform_driver(dummy1234_driver);

MODULE_AUTHOR("Hacker Bikepacker <hacker.bikepacker@gmail.com>");
MODULE_DESCRIPTION("DUMMY1234 ID provider");
MODULE_LICENSE("GPL");

As you can see, most of the code is either boilerplate, or some renaming to work with a platform device. Easy!

I have created a simple Makefile as well, that you can use to play around without wasting any time:

KERNEL_DIR ?= /lib/modules/$(shell uname -r)/build
PWD := $(shell pwd)

obj-m := dummy1234.o dummy1234_dev.o

all:
	make -C $(KERNEL_DIR) M=$(PWD) modules

clean:
	make -C $(KERNEL_DIR) M=$(PWD) clean

load-dummy1234:
	sudo insmod dummy1234.ko

unload-dummy1234:
	sudo rmmod dummy1234

load-dummy1234-dev:
	sudo insmod dummy1234_dev.ko

unload-dummy1234-dev:
	sudo rmmod dummy1234_dev

.PHONY: all clean load-dummy1234 unload-dummy1234 load-dummy1234-dev unload-dummy1234-dev

These modules will be built against the kernel your machine is running. If you have not installed its kernel headers yet, that’s a simple task:

sudo apt install linux-headers-$(uname -r)

The commands to build, load and unload the modules are also straightforward:

Compile both modules: make
Load device generator make load-dummy1234-plat-dev
Load device driver make load-dummy1234-plat-drv
Unload device generator make unload-dummy1234-plat-dev
Unload device driver make unload-dummy1234-plat-drv
Clean generated modules: make clean

Alright, let’s see them in action! I will first compile both modules, then load the device driver, dummy1234_plat_drv.ko (.ko = kernel object), and then dummy1234_plat_dev.ko, also checking if the device really appears under /sys/devices/platform/ with some attributes. Finally, I will unload the two modules in the inverse order I loaded them:

$ make
make -C /lib/modules/6.8.0-51-generic/build M=/home/jc/dummy1234 modules
make[1]: Entering directory '/usr/src/linux-headers-6.8.0-51-generic'
warning: the compiler differs from the one used to build the kernel
  The kernel was built by: x86_64-linux-gnu-gcc-13 (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
  You are using:           gcc-13 (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
  CC [M]  /home/jc/dummy1234/dummy1234_plat_drv.o
  CC [M]  /home/jc/dummy1234/dummy1234_plat_dev.o
  MODPOST /home/jc/dummy1234/Module.symvers
  CC [M]  /home/jc/dummy1234/dummy1234_plat_drv.mod.o
  LD [M]  /home/jc/dummy1234/dummy1234_plat_drv.ko
  BTF [M] /home/jc/dummy1234/dummy1234_plat_drv.ko
Skipping BTF generation for /home/jc/dummy1234/dummy1234_plat_drv.ko due to unavailability of vmlinux
  CC [M]  /home/jc/dummy1234/dummy1234_plat_dev.mod.o
  LD [M]  /home/jc/dummy1234/dummy1234_plat_dev.ko
  BTF [M] /home/jc/dummy1234/dummy1234_plat_dev.ko
Skipping BTF generation for /home/jc/dummy1234/dummy1234_plat_dev.ko due to unavailability of vmlinux
make[1]: Leaving directory '/usr/src/linux-headers-6.8.0-51-generic'

$ make load-dummy1234-plat-drv
sudo insmod dummy1234_plat_drv.ko

$ make load-dummy1234-plat-dev
sudo insmod dummy1234_plat_dev.ko

$ ll /sys/devices/platform/dummy1234_device/
drwxr-xr-x root root   0 B  Wed Jan 22 21:49:33 2025  .
drwxr-xr-x root root   0 B  Wed Jan 22 18:44:56 2025  ..
.rw-r--r-- root root 4.0 KB Wed Jan 22 21:49:53 2025  driver_override
.r--r--r-- root root 4.0 KB Wed Jan 22 21:49:53 2025  modalias
drwxr-xr-x root root   0 B  Wed Jan 22 21:49:53 2025  power
lrwxrwxrwx root root   0 B  Wed Jan 22 21:49:53 2025  subsystem ⇒ ../../../bus/platform
.rw-r--r-- root root 4.0 KB Wed Jan 22 21:49:53 2025  uevent

$ make unload-dummy1234-plat-dev
sudo rmmod dummy1234_plat_dev

$ make unload-dummy1234-plat-drv
sudo rmmod dummy1234_plat_drv

These are the messages you will see coming with journalctl:

$ journalctl -f | grep dummy1234
TTY=pts/4 ; PWD=/home/jc/dummy1234 ; USER=root ; COMMAND=/usr/sbin/insmod dummy1234_plat_drv.ko
TTY=pts/4 ; PWD=/home/jc/dummy1234 ; USER=root ; COMMAND=/usr/sbin/insmod dummy1234_plat_dev.ko
dummy1234_init: registering dummy1234_device
dummy1234_driver dummy1234_device: Calling dummy1234_probe :D
dummy1234_driver dummy1234_device: ID = 1234 (dummy)
TTY=pts/4 ; PWD=/home/jc/dummy1234 ; USER=root ; COMMAND=/usr/sbin/rmmod dummy1234_plat_dev
dummy1234_exit: unregistering dummy1234_device
dummy1234_driver dummy1234_device: Device removed!
TTY=pts/4 ; PWD=/home/jc/dummy1234 ; USER=root ; COMMAND=/usr/sbin/rmmod dummy1234_plat_drv

Everything worked as expected: the device driver was loaded, and the probe function was not called because there was no matching device yet. After creating the device (which could be found under /sys/devices/platform/), the probe function was triggered. Then, after unregistering the device (i.e. unloading the “device” module), the remove function within the device driver was triggered. Is it not great? You could have N devices, and the probe function would be triggered for every one of them, allocating per-device resources. That is the beauty of the Linux Device Model.

If you want to experiment with these modules, you can find them here to copy their code more easily: dummy1234_plat_drv.c, dummy1234_plat_dev.c, Makefile.

This has been a pretty long article, but I wanted to settle the basics and key concepts for once and for all… until the device model in Linux changes 😆 If you have been able to follow this article until the end, you are definitely ready to write your own modern Linux device drivers, and you should be able to identify legacy drivers right away. From now on, I will focus on useful functionality for your drivers: power management, regmaps, interrupt handling, interactions with other subsystems, and much more. Stay tuned!

Equipment for Embedded Linux Development

Wed, 15 Jan 2025 16:00:00 +0000

A few people who follow my articles –especially the new series about Linux device drivers– have asked me about the hardware they should buy for embedded Linux development. And that did not surprise me, because we all love getting new toys to hack around with. This article addresses such an interesting topic, and walks through the basic components that everyone should have to learn embedded Linux as well as some nice to have stuff that will make your life much easier. I have added some more advanced alternatives for experienced developers as well, so everyone can get something useful out of it.

But I don’t want to throw a random list of articles without saying why and when you will need them; instead, I would like to add some value with my own, honest insights. I will explain why and when you will need them, from the point of view of an embedded Linux developer and Linux kernel contributor, both professionally and as a hobbyist, and based on my own experience. I hope you will find something useful!

Content:

Shopping list
Single Board Computer (SBC)
Power Supply
Breadboards
Multimeter
Wiring/Connectivity
Soldering Station
Oscilloscope
Matrix boards and SMD adapters
Peripherals
Logic analyzer

Shopping list

In a nutshell, that’s what every embedded Linux developer should have at home, at work and in their pockets just in case:

A Single Board Computer (SBC).
A power supply.
Breadboards.
A multimeter.
Basic wiring/connectivity.

And then, depending on your budget and needs, you could/should get:

A soldering station.
An oscilloscope.
Matrix boards and SMD adapters.
Peripherals, lots of peripherals!
Logic analyzer

Single Board Computer (SBC)

This is by far the most important element. It is a board built around a System on Chip (SoC), which is where the CPU, GPU, memory, and other essential peripherals live together in harmony (well, most of the time). It’s the brain of modern embedded systems, powering everything from your smart toaster to industrial robots. Many manufacturers offer SBCs that simplify access to SoCs, providing developers with a ready-to-use platform that eliminates the need for custom hardware design. They offer a convenient way to prototype, test, and deploy embedded systems quickly, with built-in support for peripherals and connectivity, making them ideal for accelerating development.

Currently, the main architecture used in embedded systems is by far ARM64. Therefore, for a complete beginner, I would start with an ARM64-based SoC: you will find a lot of documentation, and most of the things will work out of the box. I always recommend Raspberry Pi and BeagleBoard SBCs because they are the first choices by hobbyists, and you will never lack of online support. They are not the only ARM64 options, though, and I am going to show you a few of them.

If you’re feeling brave and adventurous, you might want to go for a RISC-V SoC—an open architecture that’s incredibly promising (its fans will tell you it’s not just promising; it’s already a reality :D). I will tell you more about it later as well.

When available, I have added two options for each SBC: bare SBC (you will need to buy some extra stuff like a power supply, a case, wires, etc. and in some cases, solder pins) and all-in-one kit (the easiest option: you get everything you need and are ready to hack within minutes).

Raspberry Pi

Probably the most common SBC among hobbyists, it is supported by many distributions, and you will find a lot of information online as well as an infinite number of projects for them. There are also hundreds, if not thousands, of plug&play peripherals for them. I have three Raspberry Pi SBCs myself: Raspberry Pi 4 model B, 5, and zero 2 w, and every one of them has its pros and cons.

The Raspberry Pi 4 and 5 are the most convenient ones to have on your desk for several reasons: greater power, connectivity, and more convenient boot options like USB boot (booting from a USB memory) and network boot (what most professional embedded Linux developers use) thanks to their Ethernet connectivity. The Raspberry Pi 5 is by far the most powerful one (hence why it needs heat sinks), and you will be covered for many years without hardware upgrades. If you are a bit on a budget, the Raspberry Pi 4 is still completely fine, and you will find a lot of documentation for it.

But if you’re looking for a low-power, low-price solution, the Raspberry Pi Zero 2 W is probably a more interesting option. Keep in mind that it’s quite limited in terms of RAM and connectors, but for example, if you’re going on a trip (like me, bikepacking!), it’s ideal due to its size, weight, and low power consumption. The following table summarizes the main features, pros and cons of the three models:

Feature	Raspberry Pi 4 Model B	Raspberry Pi 5	Raspberry Pi Zero 2 W
CPU	Quad-core Cortex-A72 @ 1.5GHz	Quad-core Cortex-A76 @ 2.4GHz	Quad-core Cortex-A53 @ 1GHz
GPU	VideoCore VI	VideoCore VII	VideoCore IV
RAM	2GB, 4GB, 8GB	4GB, 8GB, 16GB	512MB
Connectivity	Wi-Fi, Bluetooth, Gb Eth.	Wi-Fi, Bluetooth, 2.5Gb Eth.	Wi-Fi, Bluetooth
USB Ports	2x USB 3.0, 2x USB 2.0	2x USB 3.0, 2x USB 2.0	1x Micro USB
Power Supply	USB-C (5V/3A)	USB-C (5V/5A)	Micro USB (5V/2.5A)
GPIO Pins	40	40	40
Pros	Versatile, stable and well tested	Performance, fast Eth., lifespan	Compact, low power
Cons	Bulky, less efficient	Price, power, heat dissipation	Low RAM, slow CPU

By the way, often a kit is the only way to get the Raspberry Pi 5 due to supply shortages.

Buy on Amazon: Raspberry Pi 4 model B

Buy on Amazon: Raspberry Pi 4 model B Kit

Buy on Amazon: Raspberry Pi 5

Buy on Amazon: Raspberry Pi 5 Kit

Buy on Amazon: Raspberry Pi zero 2 w

Buy on Amazon: Raspberry Pi zero 2 w Kit

BeagleBoard

Although BeagleBoard SBCs are usually less powerful than their equivalent in the Raspberry Pi world, they offer several advantages that make them shine, especially among high-end embedded developers. There are a lot of sources of information and projects for them too (maybe not as many as for Raspberry Pi, but often coming from experts), and for Linux kernel developers it is awesome because you can run the mainline kernel without a full out-of-tree patchstack or a kernel fork like on a Raspberry Pi (that fork is well maintained and up-to-date, and I have successfully developed multiple drivers based on it, but it’s worth to mention).

I don’t have any BeagleBoard at the moment, but I have used different models in the past (and probably again in the near future), and they are reliable, and nice to work with. Maybe they are not as powerful as other models in this article, but you will save some current consumption while enjoying a SoC that the open source community loves. Moreover, you will have zero issues to follow my articles and tutorials with this board, because everything on it just works.

The recommended models at the moment (ARM64) are the iconic BeagleBone Black, and the more powerful BeaglePlay. As with the Raspberry Pi, both have pros and cons, and any of them will be alright for most applications. As I have experience with the BeagleBone Black, and it is more affordable and widely used, that’s the one I will recommend you. There are better places to buy this board than Amazon, so it’s better that you take a look at this (non-commercial) link to the official site because their official sellers offer better prices.

Orange/Banana Pi & co.

These SoCs offer much more power and features than the ones I mentioned before. Then this article should be me saying: get one of these and move on, right? Well, there’s always a tradeoff. First, there is way less information online, and sometimes you will be on your own. That’s usually fun and challenging, but also time-consuming and sometimes terrifying, especially for beginners. Second, upstream support for these boards is usually worse because there are fewer users, and you might get stuck in vendor-specific code to make use of all their features. But I must say that they are catching up very quickly, because many companies are basing their designs on cheap, yet very powerful SoCs like these. I have seen a lot of activity in the mainline kernel from different companies, and they are doing it for good reasons.

Many of those cheap SBCs are built upon Rockchip SoCs, and I have professional experience with ARM64 Rockchip-based SoCs. They are well suited for commercial projects because of their lower cost and many features, but I can tell you that the documentation and customer support is way worse than the other ARM64 SoCs. In the end, everything will work with some extra effort from your side and the community, but be ready to stumble upon unexpected issues from time to time. If you would like to get one of the most powerful SBCs for a consumer level, the following Orange Pi 5 Plus is even more powerful than many consumer workstations: 8 cores (Quad Core CortexA76 + Quad Core CortexA55), 16 GB LPDD4 RAM, 8K codec… a beast, really.

Buy on Amazon: GeekPi Orange Pi 5 Plus

Another SBC I would also like to mention in this section are the Radxa Rock, which is based on Rockchip as well. I have been able to run Armbian (and even Ubuntu) on the 5B at work, and most of their features worked out of the box, but as soon as you delve into specific hardware features, you notice that there is still a lot of work in progress upstream. But maybe that’s exactly what you are looking for! As a cheaper and simpler alternative, the Radxa 4SE from the link below features the RK3339 SoC, which has been supported by the Linux kernel for quite a while. This SBC is probably one of the most interesting ones to have something powerful in the middle-range price, featuring an Arm Mali T860 MP4 GPU.

Buy on Amazon: Radxa Rock 4C+

Attention on Radxa products has been increasing for some time, and not only due to the Rock models. The latest hit is without a doubt the Radxa Orion O6, which is out of the scope of this article, as it is a more advanced board and relatively new. My advice: if you are new to the Linux embedded world, be realistic and stick to simpler devices!

RISC-V SoCs

Recently, I acquired a board based on a chip with RISC-V architecture, but I’m still testing it and figuring out what works and what doesn’t. Since there aren’t many well-established alternatives in the market yet, and I only have experience with my board, I’ll be a bit cautious in this section.

There is an increasing number of RISC-V boards entering the market, catering to hobbyists, developers, and researchers. Even though I have not seen many commercial products based on RISC-V myself, I guess it will not take long until they get a relevant position in many industries. Among the most popular options, the HiFive Unmatched is the one that gets mentioned over and over again when you look for RISC-V boards. Despite the significant supply issues with the HiFive Unmatched, those who managed to get their hands on one highlight its excellent Linux support and impressive performance. I can’t tell myself, but as we’ll see shortly, we can take an “other fruit”-style approach from a Chinese company to acquire a powerful RISC-V SBC. Just to mention one of manufacturers that appeared when we discussed ARM64 boards, BeagleBoard is also venturing into the RISC-V space with the BeagleV.

What board do I have then? One of those xxxFive, the one that is getting more and more attention these days, and it’s available: the StarFive VisionFive 2 (link below). It features a quad-core StarFive JH7110 RISC-V processor running at up to 1.5 GHz, paired with 8 GB of LPDDR4 RAM. There should be a 4GB version according to the specs, but I could not find it on the market. Linux support is ever–growing, with distributions like Debian and Ubuntu offering compatibility. I have to admit that installing Ubuntu required some work (apparently, some revisions don’t, check here if it would apply to yours), and a few core blocks like the GPU did not work the last time I tried. It will come, but it might take some time. Who knows, maybe you will be the one who brings more support for the whole community!

Buy on Amazon: StarFive VisionFive 2

Buy on Amazon: StarFive VisionFive 2 Kit

Power Supply

Having a bench power supply on your desk is awesome: you can select a wide range of voltage and current levels quickly and with high precision, being able to measure total power consumption. But they are not (easily) portable, and in many cases, especially when you are at the beginning of your journey to become an embedded Linux developer, you will be able to manage with a simple power supply for the SoC and the power outputs it provides, perhaps with another simple power supply for any piece of hardware connected to the system that requires more than what the SoC can supply.

If you are going to rely on the SoC to feed the rest of the system, then you will just need an AC/DC converter with ~18W USB output, and depending on what you want to develop, maybe a lithium battery, which I will leave out of this article because it would be a topic on its own. If you bought a starter kit like the ones I mentioned, you will have received a power supply as well. For those who have acquired a bare SoC, I would recommend you one with an integrated switch for convenience. Believe me, you will regret saving a couple of bucks here and having to connect and disconnect the power supply every time.

Buy on Amazon: Power Supply 5V with ON/OFF switch

Should you prefer a bench power supply, I would recommend you a simple, digital one that will cover almost all cases. For more professional environments, I would opt for the typical device you will find in an electronics lab: one with three independent, high-precision outputs (10mV/10mA is more than enough), and additional measurement/analysis features.

Buy on Amazon: DC Power Supply one output

Buy on Amazon: DC Power Supply three outputs

Breadboards

One of those things where you don’t have to spend much money, or even think what model would suit your needs. Apart from their size, they are all more or less the same. Just get a kit with a couple of boards and a bunch of jumper wires, and you will be good to go.

Buy on Amazon: Breadboards + Jumper wires

Multimeter

This is a very important device that you should have from the beginning. There are many different models out there, but in general you will not need anything special. A super basic multimeter will be able to measure current, voltage, resistance and continuity (audible is always nicer), and you will seldom need anything else.

Buy on Amazon: Cost-effective multimeter

It’s also true that better models don’t cost a fortune either, and you will get a more robust and compact device with a few more cool functions like frequency and duty cycle measurements, which are even more useful if you don’t have an oscilloscope. These devices should last many years, and the first you buy will stay around long. Personally, I have a Crenova MS8233D, and I am very happy with it:

Buy on Amazon: Crenova MS8233D multimeter

Wiring/Connectivity

Every project has its own connectivity requirements, and I can’t cover all kinds of cables, adapters, and wireless modules. My advice: take some time to evaluate what you are going to need from the very beginning, and order it as soon as possible. And don’t forget debugging capabilities! For example, SSH is often enough to debug an embedded system, but I like serial communication to see early messages as they are generated or get easy and quick access to the bootloader. And oh surprise, not all embedded systems are capable of wireless communication! The cable I always use for serial communication is the following one, which has two different connectors on the TTL-side, and the color code described on the USB-side.

Buy on Amazon: USB to TTL cable

For quick prototyping, some jumper wires (like the ones I mentioned when I talked about Breadboards) are always nice to have, and they are really cheap.

Soldering Station

This is another simple item unless you really want to solder small and complex stuff, which at least in the beginning will seldom be the case. Just get a regular, inexpensive soldering kit with the bare-minimum features: adjustable temperature, interchangeable iron tips and a stable base to put the soldering iron when you are doing something else. I can guarantee you that you, and especially your desk, will regret getting a soldering iron with no base. Here’s a low-budget kit that will be enough for easy tasks:

Buy on Amazon: Basic soldering set

If you are planning to solder more often, of if you want to suffer less while desoldering, you could (and probably should) get something slightly more advanced. But really, there is no need to get too fancy, as there are other elements where investing some extra money will make a bigger difference. A soldering station with a hot air tool and digital adjustment will be enough for almost everyone. This one will be a perfect tool for almost everyone:

Buy on Amazon: Advanced soldering set

Additionally, I would recommend you to get helping hands because they are worthy of their name and make soldering a bit easier. Other cheap but useful items like flux paste and solder wick will also come in handy to make soldering and desoldering even easier.

Buy on Amazon: Helping hand soldering

Buy on Amazon: Flux

Buy on Amazon: Solder wick

Oscilloscope

This is one of the most expensive items. They are great to have, and once you have one, you will be using it all the time. But if you don’t need to measure digital signals, or only DC, leave this device for the future. Not having any means to measure clocks, data signals and combinations of them is not optimal, but many hobbyists do as much as they can with the multimeter and hope for the best. As I mentioned before when I discussed multimeters, many offer basic information like frequency and duty cycle for low-frequency signals. On the other hand, there are nowadays relatively cheap oscilloscopes with a bandwidth of ~100MHz and sample rates around 500Msa/s, like the one below:

Buy on Amazon: Cost-efficient oscilloscope

That will be alright for low-frequency —and most medium-frequency– applications. If you would like to invest in a better device with many more analysis features and higher sample rates for better resolution, I could recommend you Siglent, which is the brand I use at work. The oscilloscopes we have cost at least 2 grand, some of them much more, but there are also more affordable models from the same manufacturer that will have you well covered. For example, the Siglent SDS1104X-E provides 4 channels, it reaches 1Gsa/s, and you will be able to analyze several serial protocols like I2C and SPI.

Buy on Amazon: Siglent SDS1104X-E

If you have enough with 2 channels, and you would rather have a bandwidth of 200 MHz and save some money, the Siglent SDS1202X-E would be a great option:

Buy on Amazon: Siglent SDS1202X-E

Matrix boards and SMD adapters

The SMD equivalent to the breadboard we saw before. As I said before, you will not find many through-hole devices these days beyond breakout boards, and SMD is a very common alternative. And again, keep it simple: get regular matrix boards (single or double-sided, 2.54mm pitch, you can use them for through-hole devices too) that you will use to quickly assemble look-alike PCBs, and some basic SMD adapters for common packages. In the end, if your prototype will turn into a product or at least a more robust system, you (or someone else) will have to design a PCB. But until then, these solutions will increase your productivity in the development/prototyping phase a lot!

Needless to say, you will need access to a soldering iron, but as we saw before when we discussedsoldering stations, there are affordable options out there.

Buy on Amazon: Matrix boards

Buy on Amazon: SMD Adapters

Peripherals

I will leave this one to you. Look for any peripherals that could help you develop an interesting project: maybe some touchscreen to receive input from the user, a Bluetooth module… whatever you like. No matter what you choose, expanding your SBC with more devices will be fun, and you will gather a lot of experience.

I will only add that if you are looking for hardware to program your first Linux device driver, check out this article for more information about how to choose one and optimize your time. For me, getting new peripherals and integrating them into a running system is a very entertaining hobby, and I enjoy it a lot!

Logic analyzer

I covered this in more detail in this article about I2C on Linux, but given that a logic analyzer is a fairly common tool in the embedded world, I would like to mention it here as well. In short, a logic analyzer is a tool used to capture and analyze digital signals, particularly useful in debugging and verifying serial protocols like I2C, SPI, or UART in embedded systems. Oscilloscopes, even those with a serial bus decoder like the last two ones that I mentioned before, are sometimes limited when it comes to trace long transactions, filter packets, and analyze the communication at the protocol level. A cheap option (under $15) is the AZDelivery Logic Analyzer:

Buy on Amazon: AZDelivery Logic Analyzer

This logic analyzer can sniff a bus, and thanks to its USB end, transmit the information to your PC to inspect it by means of some signal analysis software suite. A well-known, open-source option is sigrok, that supports different frontends like PulseView and sigrok-cli for CLI lovers, and it even packages a library to integrate it into your projects. It might be worth mentioning that AZDelivery also provides its own open-source libraries for Raspberry Pi to make things even easier.

Maybe you think that buying everything from the list will cost you and arm and a leg, and it’s true that some things are not exactly cheap. But the basic package is indeed affordable, and it will provide you everything you need to become an embedded Linux developer. Moreover, most of these articles are meant to last for many years: Oscilloscopes, multimeters, and soldering irons over 10 years old are very common.

Once you get the bare minimum to start hacking, the learning process will be fun and pretty similar to what you will then do as a professional embedded Linux developer, which is a highly sought-after and well-paid profile… unless you dedicate yourself to bikepacking, of course 😅 And most of the hardware, including the SBCs will stay there for your future projects. All things considered, it sounds like a pretty good investment 😉

Linux Drivers 1 - Hardware Considerations and Setup

Mon, 06 Jan 2025 16:00:00 +0000

The day has finally arrived: I’m starting a series on how to write your first device driver for Linux. And no, I’m not talking about out-of-tree modules or examples based on the wonderful but now outdated Linux Device Drivers, 3rd Edition. I’m talking about drivers that use modern APIs and are upstream-ready, so the community can maintain and keep them up-to-date. Thus, I will always assume that we are writing code to send upstream and become a Linux kernel contributor :star:

I’ll base all my explanations on code you can find in the mainline kernel; whenever possible, from my own drivers or those I’ve contributed to by implementing new features or fixing bugs. The hardware I will use for the examples (SoC and devices) will be simple, and affordable. But don’t worry, you will be able to apply the same concepts to the most complex devices and their drivers as well because Linux loves. Linux loves its code to be homogeneous, at least as much as possible. I’ll tell you more about this in a few minutes.

Regarding the code, it should make no difference whether you are programming a driver for a big tech company or for your little project at home, and you will have to apply the same basic rules regardless the complexity of the hardware. Therefore, I will focus on the key concepts, so you can then build upon for your particular case. I am planning to cover some more advanced features once we have mastered the basics, and hopefully there will be something new to learn even for experienced kernel hackers.

Since there’s a lot to cover, and I want to dive into many details, I expect this series to be relatively long. My goal is that by the end of the series, and apart from analyzing multiples snippets from a different drivers and subsystems, we will have gone through a real driver over 1000 lines of code without skipping any details. In this first chapter, I’ll focus on working with the datasheet and choosing the right subsystem for a device.

Enough for an introduction, let’s get into it!

Content:

0. Is the device already supported?
1. Skim the datasheet
2. Choose the appropriate kernel subsystem
3. Read the subsystem documentation BEFORE you start
4. Use a modern driver as a template: DON’T reinvent the wheel
5. Is your hardware ready?
6. What’s next?

0. Is the device already supported?

Checking if a given device is already supported by Linux before adding it to a new design is always a good idea to save time and efforts, and also to avoid being a fool and working for weeks/months on a device that is already supported, or requires some minimal modifications and additions to an existing driver to be supported.

Therefore, the first step should be obvious: look for the name of your devices (and variations of them) in the code. Basic tools like git grep -i will help. You could also take a look at the existing drivers from the subsystems that would cover your devices (more about subsystems later). Another source of information are the devicetree bindings, where you will find supported devices and their properties. Believe me, it is less work than you think (it will take you 5–10 minutes at most), and if you don’t do it, you will waste loads of time.

I wrote an article about extending drivers that you might use if you find a driver that would require minimal modifications to support your device. From now on, we will suppose that the device is still not supported, and adding it to an existing driver would not be worth it e.g. too many modifications and generic code make drivers more complex and less maintainable.

1. Skim the datasheet

If you don’t know what a datasheet or an application note is, don’t worry, we’re about to cover that. On the other hand, if you’re more of a hardware developer, you’ve definitely worked with these kinds of documents before. But watch out, you’ll need to look at them from a kernel developer’s perspective, not a hardware engineer’s. If you have experience with microcontrollers, that’s going to help a lot to write your first driver and translate the datasheet into software, especially if you’ve worked with Zephyr. But even so, there will be quite a few differences, so make sure to pay attention to the details.

A datasheet is simply a document that describes an electronic device. Among other things, it outlines its physical properties (e.g., form factor, operating temperature), electrical characteristics (e.g., supply voltage, operating frequency), and often—but not always—guidelines on how to interface with it and make it work as part of a system. I say “not always” because sometimes that information is split into one or more additional documents, usually referred to as application notes.

Let’s be pragmatic: the first step is to take a quick look at the datasheet for the device we want to control. If there are application notes, you can leave them for later. We’re just looking for a general understanding of what it does, its basic functionalities, and how it communicates with the outside world. This alone will give you a solid idea of what we’ll need from Linux—whether through our driver or existing code that handles some of it.

As you can imagine, the physical properties aren’t all that relevant to the kernel (though they are for circuit design!), and most electrical characteristics won’t be critical in most cases (though surprises can happen, of course). Unless the device is particularly unique, focus your initial read on the most basic electrical requirements (power supply, clocks, dedicated pins) to make sure your system will be able to handle it. Then take a look at what might be interesting for the software: the communication protocol it uses, its main features, and how it stores and delivers information.

Let’s take the HDC3020 temperature/humidity sensor from Texas Instruments that I co-developed at the LKMP as an example. Click on the link: you will see that the datasheet and the application note (in this case, user guide) are very easy to find. Although you will find many more documents under “Technical documentation” covering a wide range of topics, a few of them will only be relevant if you are designing the hardware around the sensor. Interesting topic, but not what we are covering here!

Alright, let’s open the datasheet. The first page is always a short summary to show everyone how great the device is, and how many invaluable features you will get. Half of it will be irrelevant at this stage, but you will get some interesting bullet points as well:

It has two internal sensors: one for the humidity, and one for the temperature.
It uses I2C as its communication protocol, and it provides CRC checksums.
There are programmable interrupts, and a dedicated pin for it.
There is an external reset.
Apparently, there are multiple measurement modes.
There is a “Typical Application” block that shows how to connect the sensor.

That last point is an important one at this stage: it shows you what connections between the device and your system will be required. Your first goal will be making those connections properly. That’s often easy, sometimes trivial… and seldom a project on its own, especially for beginners. More about that later. In this example, you will just have to find out how to connect the power supply (VDD and GND), the I2C signals (clock and data), and reserve at least two GPIOs for the interrupt and the reset.

Taken from the datasheet

If you scroll down and ignore physical and electrical characteristics, you will soon find that there are indeed multiple measurement modes, all of them relatively slow (up to 10 measurements per second), a table of commands, which seems to be the way to communicate with the device via I2C, and a status register to provide information about the operating state of the device. There is indeed much more information, but with just those points I mentioned, we should be able to manage at least 50% of the device functionality.

You will have to read the datasheet and application notes more thoroughly later on, and sometimes a little note will make you rework a big chunk of code, but that’s part of the job. For now, what you just did is enough work to carry on!

2. Choose the appropriate kernel subsystem

Now that we have a general idea of what the device does, this step should be very easy. Often it is obvious by the nature of the device, and usually there are similar devices already supported upstream. But that is not always the case, and I have already seen a couple of driver proposals that first had to be rewritten before sending them again, this time to the right subsystem.

At least one time it happened because the author did not follow my step no. 0 before writing a new driver: all similar devices belonged to a different subsystem. Moreover, the code was basically useless because none of the modern APIs had been used, but we will leave that for step no. 4.

If you were tempted to send all devices with SPI protocol to the SPI subsystem, stop right there! This is not how things work: the SPI (the same apply to many other communication protocols) subsystem is in charge of providing the SPI functionality to consumer drivers, which will reside under different subsystems. To select the right subsystem, you only have to think about the main functionality of your device. If it is a Real Time Clock (RTC), it does not matter what protocol it uses: it is an RTC and there will probably be a subsystem that covers such devices. Indeed, there is the rtc subsystem under drivers/rtc/

Could you still get it wrong, even if you followed my advice? Yes, in some particular cases, you could. Sometimes there is (slight) overlapping between subsystems, and some devices are not that easy to classify. For example, I have seen a couple of temperature drivers that were first sent to the Industrial Input/Output (iio) subsystem, but ended up being rewritten for the HW Monitor (hwmon) subsystem. The typical applications for the sensor and its operating frequency are often the deciding factors in such cases.

You could ask why the HDC3020 was added to iio and not hwmon, when it can’t take more than 10 measurements per second in its fastest operation mode. The line is rather thin, and this device would work under hwmon just fine, especially considering that it (still) does not support triggered buffers (I will explain them later in this series). On the other hand, it is a general purpose sensor that is not specifically designed to monitor hardware (like the temperature sensor on a motherboard), and similar humidity/temperature sensors like the HDC100 and HDC2010 are already under iio. Having said that, such edge cases are pretty uncommon.

3. Read the subsystem documentation BEFORE you start

Now that we know the right subsystem for our device, we need to find out what the subsystem will expect from us and our code. What is the minimum functionality we have to provide? What kind of documentation and/or tests are expected to be part of the series? Is there anything else I should know before sending my patches to that subsystem?

Many subsystems provide specific documentation to address such questions, and many more. Unfortunately, some subsystems don’t, so you will just have to follow the general rules and recommnedations like the coding style or the guide to submit patches. Either way, I would strongly recommend you to keep an eye on existing drivers to gather common patterns and conventions (e.g. the input subsystem prefers the ubiquitous ret variable to be called error… whatever, just follow suit and let the reviewers focus on what really matters).

Another source of information that I would recommend you to have present is the ABI. If your driver has to expose attributes to userspace, make sure that you follow the definitions under Documentation/ABI/. They are grouped in files whose name resemble the path you would follow to find them. The ABI applies for all exposed attributes, not only for device drivers. For example, you will find the description of /sys/power/state in Documentation/ABI/testing/sysfs-power. Similarly, if you are interested in attributes that USB-C devices expose under /sys/class/typec, you will find them in Documentation/ABI/testing/sysfs-class-typec. And if you need to define a new attribute, make sure that you add it to the right document in that folder, and following the existing structure. That’s something that people often forget, and I have found myself adding missing attributes to hwmon and iio a few times after looking for their descriptions for a driver that would require them. By the way, don’t ever break userspace and stick to the ABI!

4. Use a modern driver as a template: DON’T reinvent the wheel

We will start working with driver code in the next article, but if you can’t wait, make sure that you get the right template. No, you are not going to freestyle your driver. If you want to train your creativity, take a pottery course or paint the walls of your room.

At least for device drivers, you are not supposed to start from a scratch. Instead, you are expected to use the kernel features and APIs as well as common code and patterns found within the subsystem. Do you want to start with a completely empty file? That’s fine, but in no less than 90% of the cases, your code should look like existing upstream code with little variations to address the unique features of your hardware. There is a lot of reliable code that has been used (i.e. tested) for many people during years, so don’t waste your time reinventing the wheel, and simply enjoy what you get for free. Don’t worry, even if you use existing APIs and recycle code, you will have good chances of programming new, buggy code :wink:

Of course, programming a device driver is not just copying, renaming and pasting, and often your drivers will require features that are still not available. Sometimes you will implement a tailored solution for your driver, but there will be occasions where you will add new features to the subsystem core, so other drivers can use them as well. It could even happen that the feature you need should be provided by another subsystem (e.g. power, clock, etc.), and in those cases you will end up working on multiple subsystems at the same time. That is always a great way to learn!

Why did I say that you should base your driver on new ones? Older drivers have been tested and patched for much longer, and they should be more reliable, shouldn’t they? Well… that will depend on how often the driver was used. I have found bugs in old drivers that made me wonder how many people were still using them, and I could confirm that some features had never been used, because they simply did not work and even made the driver crash. I would say that the history of a driver matters way more than its age, and drivers with many patches have usually been more thoroughly tested by multiple users.

Moreover, new drivers will have been reviewed under the latest standards, and they will probably conform to the state-of-the-art APIs. It’s very common that people submit new drivers with old code that has been inherited from legacy drivers, and rewriting those parts to use newer standards is time-consuming (more iterations, testing, etc.).

5. Is your hardware ready?

Anything that has to do with touching hardware is feared by many software developers :laughing: Depending on your answers to these questions, your fear will be justified or not:

Are you looking for ideas and don’t you know what device you will get yet?
Do you know what device you need, but you did not order it yet?
Do you have the hardware, and is it ready to use?
You only have a beetle-looking chip. No idea how to connect it to the system?

Let’s discuss all cases.

Looking for ideas

If you just would like to learn how to program a device driver, keep the hardware simple, and you will have fewer problems to worry about. If the device is already integrated in the system you are going to work on, or if you find an evaluation board, you will be ready to focus on the software from the beginning. For example, a couple of LKMP mentees bought cheap breakout boards that could be plugged into a SoC with very simple wiring.

If your device looks like these, your life will be much easier

I would also recommend you to look for a device you would like to integrate in a personal project, or even build a project upon the device. If you do so, everything you will learn and implement will have some real use as well as a lot of testing from your side. For example, I have provided support for multiple sensors I wanted to add to a little, fancy weather station. Once you know what kind of device would suit your needs, spend some time searching online (e.g. distributors like DigiKey, Farnell, RS, or more specific for hobbyist boards like Adafruit or SparkFun). Even though Linux supports thousands of devices, it won’t take you long until you find an interesting device that is still not supported.

I know what device I need

In that case, consider what options you have according to how much time you would like to invest in setting up the hardware. If your answer is 0–5 minutes, look for a breakout board and the appropriate wiring (see the section above). If you want to learn more about hardware design, or you did not have an easier option, the next section might help you. Either way, make sure you read 0. Is the device already supported?

While waiting for your device to arrive, you might want to look for ways to simulate its functionality and start programming and “soft-testing”. For example, I mentioned some options for I2C devices in this article. That’s a good and cheap way to get some basic driver before you get access to the hardware, but some more advanced features will have to wait. Anyway, never send a driver upstream without having tested it with real hardware!

Beetle-looking chip

If you already have the device, but it is neither integrated into the system, nor plug & play, setting it up will be a bit more laborious, but (usually) feasible even for hobbyist and beginners. First check if the datasheet provides information about how to connect it to the system. Sometimes you will even find a dedicated document for that. You might find out that you still need some capacitors, resistors, adapters or wires… hopefully not much. Depending on how much you know about hardware, how simple/standard the package is, and how motivated you are, you have different options:

a) Mount your device on a protoboard + SMD adapter if required. Unfortunately for hobbyists, through-hole packages are becoming rare. But you could still use a breadboard if you find an SMD adapter that fits your device. You will have to solder the device, but the process is very simple, and you will be able to connect your device to the system with cheap and easy-to-use jumper wires. By the way, I have seen videos of people soldering BGA packages with hot air and loads of flux, but I would not be surprised if that did not work even after 10 tries.

b) Design your own PCB. I would recommend you KiCAD, an open-source, multi-platform EDA, for the design, and a quick search for a cheap manufacturer in your region. For example, I have used multi-cb, which is rather popular in Europe, and everything worked smoothly. Prototypes are usually manually assembled/soldered, but if you don’t have access to a soldering station or the task would be too complex for your skills, look for a manufacturer that also offers assembling. You might have to make some “adjustments” to the HW anyway, so minimal soldering skills will always be welcome. If the package is not simple, this approach will be the only realistic one.

There are many KiCAD-based OS projects, like this one

c) Use your imagination and HW-hacking skills. I have chosen this one a couple of times when I had very simple hardware but no compatible adapter, and too little patience to go for the cleaner solution with a PCB. Some solutions that have worked for me were soldering the device on a protoboard or hacking an adapter with a scalpel and some ultra-thin magnet wires to make it suitable for my device. Don’t expect to have the most noise-, vibration- and shock-resistant design ever, though :wink: But it will work for a prototype until you find a better solution. What I would not recommend you is soldering wires to the pins, because that seldom works, and you will have to re-solder many times. Be hacky, but not that hacky.

I recently found a bug in the driver of the veml6030, and I only had the bare chips within reach. After ~20 minutes of nasty hardware hacking, I was able to validate the fix

6. What’s next?

For the next articles I am going to assume that you have a Linux system, preferably based on a well documented and devicetree-based architecture such as ARM64 and RISC-V. I will be using a Raspberry Pi (ARM64) myself, as I have already written articles about how to set it for Linux kernel development. Please check them out if you are going to use a Raspberry Pi as well to save time, but you will be able to follow this series with any other SoC, almost 1:1.

I will also assume that you are ready to connect your device to the system, but that should be no problem after reading this article 😜

Test Rides Before the Big Adventure

Sun, 22 Dec 2024 08:00:00 +0000

Although I’m someone who tends to leave a lot to improvisation, I’m not so carefree as to skip testing the gear I’ll take on the BIG JOURNEY. Since we* have had pretty much everything we consider necessary for a while now, we’ve been able to do a few short trips near where we live to see if we are missing anything or carrying too much. And as we live near many borders, those short adventures took us to a couple of countries. Here’s how our recent rides through Austria, Germany, Liechtenstein, and Switzerland went!

*I will be bikepacking with my girlfriend, Rebecca, who also took part in the first two trips I am going to talk about.

Content:

1. First Test: Lake Walen and Toggenburg
2. Allgäu: Better on a Bike
3. Testing the Saddle on Long Rides
4. Steel on Steep Climbs

1. First Test: Lake Walen and Toggenburg

The first round (early Spring 2024) to test the equipment was also an excuse to ride along the beautiful Lake Walen (in German, Walensee) and the breathtaking Churfirsten mountain range. I have been on top of some of those peaks –even in the Winter–, and the view from them is always breathtaking. But this time I wanted to encircle them.

Walen Lake from the Frümsel: awesome sight

After cycling the 25 km that separate me from Liechtenstein, we crossed this tiny country via the paved bike path that runs alongside the Rhine. Traveling it from north to south is less than 30 km with no elevation gain, so in about two hours, you’re already heading back out toward Switzerland in the direction of Sargans. This time, we only passed through without stopping, but I’ll share more about it later.

From Sargans, and still on bike paths so far, we reached Lake Walen. The day had been quite cloudy, but little by little, the sky began to clear. The Churfirsten range came into view, and everything was going perfectly.

Things started to go a bit downhill when we discovered that the bike path running along the lake was closed for construction, forcing us to ride on a parallel road with heavy traffic, which somewhat broke the magic. Fortunately, after just over an hour, we managed to return to the section of the bike path that wasn’t under construction. By then, it was time to start looking for a place to sleep i.e., a good spot to camp.

Although it depends on each Canton and municipality, wild camping in Switzerland is only allowed under certain circumstances: one single night, in the mountains above the tree line, outside protected areas, and never in groups. Camping near Lake Walen is only permitted in designated campsites in Walenstadt, Murg, and Gäsi. I’m completely against banning responsible camping in non-protected areas, and where I camp usually ends up cleaner when I leave than when I arrived, as I pick up the trash I find nearby. You might think I always wild camp, and if I did, it would be in quiet and secluded places where I would have never had any issues 😉

The next morning, the clouds had almost completely disappeared, and we continued without setbacks toward the Toggenburg region. This day had a lot more elevation gain than the previous one, but without steep gradients, making it manageable for anyone used to cycling. Toggenburg, and especially the surrounding mountains like the Alpstein, is a paradise for climbing and hiking. If you ever get the chance, don’t hesitate to visit!

After crossing the valley from end to end, all that was left was to descend back toward the Rhine Valley and home. In theory, the gear was fine, but it still needed a bit more testing: we had brought a gas stove instead of the gasoline one we planned to use for the trip, we hadn’t filtered water, and the weather had been good so far, so we still needed to ensure we were equipped for the cold. This last point wasn’t tested on a bike trip but rather on our trip to Iceland a month later, where we camped almost daily.

2. Allgäu: Better on a Bike

The second test was in the summer: a three-day loop through southern Germany, specifically in the Allgäu region. It’s a place that might seem very monotonous if you drive through it, but on a bike, it truly shines: green hills, quiet villages, little-traveled paths, and the Alps always in the background. While it’s not a physically demanding region, it’s far from flat. I’d still call this route a “pleasure trip,” but I wouldn’t tackle it on a fixie 😉

This time we had some extra equipment that we were going to take with us for our adventure: a water filter, a gasoline stove, and in my case, a little stick to hang flags. I could tell you that they add visibility and therefore security on the road, but the truth is that they are a good icebreaker: people see where you are from and where your trip started, and it catches more attention than just a fully-packed bike.

After crossing the border between Austria and Germany along the shores of Lake Constance (Bodensee in German), we headed inland. As soon as you leave Lake Constance, where the bustle of bikes and pedestrians is often overwhelming, tranquility takes center stage: few people outside the urban areas and little traffic by German standards.

Before reaching Isny, we decided to find a quiet spot to spend the night, but not before testing out the gasoline stove (loaded with white gas). It’s a bit more complex than the gas stove and initially feels a little intimidating, but it’s much more powerful and boils water in the blink of an eye!

We thought that staying away from the bushes and trees would be a good idea to have more space around the tent, but I have to admit that it wasn’t. The next morning, everything we had left outside was soaked with dew, and the outer layer of the tent was covered in droplets, just like everything else around us… except under the trees! We quickly realized what to do next time to save ourselves the trouble of drying everything out.

Moving on, we passed through the small and charming town of Isny before heading toward Bad Waldsee, the northernmost point of our route. Personally, I liked Isny more, even though it doesn’t have a lake-surrounded center. Still, it was a nice place to grab some ice cream and test the water filter, which in our case is a gravity-powered one. It’s not the fastest, but it can filter a lot of water at once and requires little maintenance.

A few kilometers later, we found our next hotel with a million stars. Confirmed: under a tree, everything stays much drier.

Our third and final stage first took us to Ravensburg, where we had a second breakfast of the day. This city is much larger than the others we passed through, but it has a beautiful center worth seeing, especially its many towers. From Ravensburg, with a little stop along the way to make some coffee, we headed back to Lake Constance and then home. Once again, we were happy with the gear we’re planning to take on the trip, though I did end up with a slightly sore backside!

3. Testing the Saddle on Long Rides

For the next trip, a long weekend ride to Zurich, I went solo and packed light: no tent, just a thin sleeping bag. In the end, I didn’t even use the stove, but hey, I had to put something in the panniers!

The start of the route was similar to the first one: through Liechtenstein and alongside Lake Walen, with the difference that this time the bike path was open, making the stretch by the lake much more enjoyable. I planned to cover more kilometers each day, so I continued all the way to Lake Zurich (Zürichsee), which is quite long and ends in Zurich itself. I decided to find a quiet spot by the lake around 20 km before reaching the city, and since it was warm and the place was lovely, I had a very pleasant bivouac.

In case it’s useful to you, the place is called Ürikon, and it consists of a tiny pier and an even smaller park that ends abruptly at the water with a step leading into the lake. It’s so close to the water that when I sat on the grass to read for a while, a boat passing in the distance created a tiny wave that ended up splashing me. Move your stuff one meter away to keep it dry!

There are not many better places in the area

Most of the shore is private: big houses with their own pier

Despite the 120 kilometers from the day before, I woke up early and feeling pretty good, though my backside was starting to hurt more than it should. I took advantage of the morning to ride into Zurich and arrived there quite early. It was Sunday, so I had the city almost to myself for a little while.

Zurich

Right after leaving Zurich, I faced the only and not very demanding climb of the route, followed by a rather boring stretch until I reached Lake Constance on its Swiss side, passing through Winterthur. By this point, my backside was in agony, and I was already sure that saddle wasn’t going to make it to the big trip.

I’ve cycled along this side of the lake a couple of times, so I know it fairly well. I didn’t stop much, as there were still quite a few kilometers left to get home. Most of the riders around the lake find the German side more beautiful, but apart from the touristic towns like Meersburg or Lindau, I would say that the Swiss side is more pleasant to ride, especially because you only cycle on bike paths, and it is way less crowded.

In the end, the second stage was about 170 km, at least 100 of which were spent cursing the damn saddle 😂

This trip showed me that I would be able to cover long distances if required, especially if I got another saddle, which I did shortly after.

4. Steel on Steep Climbs

On flat sections or downhill stretches, I don’t notice much difference compared to aluminum or carbon bikes, but uphill… well, let’s just say things get interesting. That’s why I wanted to try a ride with more elevation gain, ideally in one continuous stretch.

After browsing the map for inspiration, I decided to tackle the Kunkels Pass by way of the Tamina Valley, then loop back through Chur and the St. Luzisteig Pass. It was an area I wasn’t very familiar with but seemed intriguing, so I packed up (this time nearly everything I plan to take on the trip, as the Autumn is rather chilly around here) and set off once again toward Liechtenstein/Switzerland.

By the way, even though gaining altitude when it’s already quite cold in the valley might not sound like the best idea, low fog is quite common in the Alps during autumn, particularly near Lake Constance. Often, all it takes is climbing a bit higher to break through the fog, find blue skies, and enjoy the sunlight.

On this first day, I encountered dense fog until I passed through Liechtenstein and reached Bad Ragaz, Switzerland, where the climb up the valley toward the Kunkels Pass begins. Most cyclists in the area, typically riding road bikes, take a less-traveled road toward Pfäfers. However, there’s a lesser-known alternative: the Valenserstrasse. This road, closed to cars without special permits, is even more interesting. The pavement has seen better days, and the route is steep, but it winds through a dense forest, making it especially appealing for a bikepacker like me. I have to admit, I worked up quite a sweat on the steeper sections, but I managed to reach the main road intersection without putting a foot down.

Views in the Tamina Valley

Whichever route you choose, if you’re passing through and have the time, check out the Tamina Gorge (Taminaschlucht), a stunning natural wonder carved out by the Tamina River. This time, I didn’t stop for side trips since I wasn’t sure how long the climb to the pass would take, and the sun was already setting earlier than I’d like.

Continuing along a narrow, lightly trafficked road on the other side of the valley—with plenty of road cyclists for company—the route climbs gradually and steadily until you reach a tiny village called Vättis. From there, cars disappear, and you get the feeling the summit of the pass is just around the corner. That might have been true for the numerous cyclists on professional-grade, state-of-the-art bikes overtaking me, but not so much for someone hauling their belongings on a regular bike 😅

The scenery at this point is absolutely breathtaking, with particularly striking mountains, a mix of green meadows, patches of early snow, and dense forests. After climbing for several hours from Bad Ragaz (no less than three), I finally reached the Kunkels Pass, which, ironically, was almost the only shaded spot on the entire climb due to a nearby hill. I am not going to lie, the climb was exhausting, and I was glad that from there I only had to ride down. After snapping a few photos and taking in the views, I began the descent toward Tamins, heading in the direction of Chur.

The descent on this side of the pass is less scenic than the ascent, to be honest. There are fewer road bikes here, as the surface isn’t fully paved. While manageable on a regular bike, you need to pay attention to the terrain rather than just admiring the view. Back in the lower valleys, the route follows bike paths—many unpaved—but stays close to a highway, bringing some unavoidable noise. The best approach is to keep moving quickly and head straight for Chur.

Chur’s city center is charming and worth a quick visit, especially since it’s easy to explore on a bike. By then, it was getting dark, and I didn’t plan to spend the night in the city. I continued riding to escape the bustle and find a quiet spot to spend the night.

For the second day, I had planned a route that would take me slightly off the most direct path to visit a couple of local villages, Malans and Meienfeld, before climbing to St. Luzisteig on my way back toward Liechtenstein.

The route winds through vineyards, with gentle ups and downs. The villages are very small and charming, perfect for cycling through with almost no traffic. Luckily, the valley here opens up a bit, and in autumn, the sun still reaches it relatively early, so I managed to warm up a little before tackling the climb to the St. Luzisteig “Pass.” I say “Pass” in quotation marks because the name feels a bit pretentious in this case.

Although it lies at the foot of Liechtenstein’s tallest mountains (you can hike to some of them from here), the climb itself is modest—only about 200 meters of elevation gain from Meienfeld. The road is quiet, well-maintained, and free of steep gradients. The highlight comes at the start of the descent, where the road literally passes through a small fortification on the way to Balzers. This is the first village on the Liechtenstein side, and it also boasts a picturesque castle you can ride up to on your bike.

To avoid always sticking to the bike path along the Rhine, I decided to ride at the foot of the mountains, weaving through meadows and farmland and passing through Vaduz, Liechtenstein’s tiny capital. This “city” gets an outsized amount of tourism from people wanting to “check off” another country or stamp collectors visiting the local museum. The most striking feature is the prince’s castle, perched above the city. You’ll pass it if you’re heading toward the higher-altitude villages like Triesenberg and Malbun. The latter is a great starting point for hiking routes but requires a solid climb to reach.

Gradually, I made my way back toward Austria and home, feeling more convinced than ever that there’s no pass in the world I couldn’t tackle with this bike—even if I have to drag it!

New Lifestyle - Hacker Bikepacker

Sun, 15 Dec 2024 08:00:00 +0000

I’m packing my bags and off to discover the world by bike. And no, I haven’t hit my head! I’ve been thinking about this for a long time, and now I’m going to make it a reality.

Content:

1. WTF??
2. The plan
- No more FOSS hacking?
3. Can I help you?
4. Building site

1. WTF??

I have a stable life, both personal and professional. Why would I then change everything to live day to day, without knowing where I will be the next day? At this point, one would expect that the changes in my life would be less radical and more oriented towards what is usually considered “successful”, such as a bigger house, a more buoyant bank account, or a better-paid job. In fact, for an engineer like me, you are almost socially pressured to follow that path: success means working for a big company and having a 6-figure salary. I have nothing against those goals and maybe one day they will attract my attention, but that is not the case yet. I am looking for time and freedom.

Although I am not one of those who work 80 hours a week (far from it), a job with a stable schedule and the obligations of daily life prevent us from having total freedom of our time. Even though in appearance you are free to do what you want, in practice you are pressured by multiple agents: bosses, clients, social conventions, etc. I know that part of life perfectly, like almost everyone else. Now I want to explore another path: to live my own adventure every day as I decide.

Why bikepacking? I am not discovering anything new here: it’s fun, healthy, eco-friendly and lets you connect more deeply with the places you pass through. This is not the first time I do it, but by far the longest, most exciting and adventurous.

2. The plan

The plan is to plan as little as possible :laughing: I will leave Europe (it is yet to be decided whether from Austria, Italy, or Slovenia) towards Asia with no fixed finish line. I have always been interested in Central Asia, of which I only know Uzbekistan, but we will see.

The departure date is March 1st, and the trip will last (in principle…) a full year. There is no predetermined distance, and the accommodation will always be with me in the form of a tent. By the way, that’s a tent I bought and used for the first time in Hawaii last year, and since then it has visited some other interesting places like Iceland. I hope it will get to know a few more countries!

No more FOSS hacking?

Are you joking? I love FOSS hacking! I am still figuring out how, but that has to be part of the journey. And I am definitely packing a SoC to play around! Although I will spend most of my time cycling, discovering places and meeting new people, I will often have some time after pitching the tent to read, write, and code. I would like to keep on contributing to the Linux kernel as well as writing articles, both technical and about the journey. Actually, I would love to post more regularly, now that a regular job is out of the equation.

Moreover, I would like to meet other Open Source enthusiasts in person to share ideas and talk about our common passion. I have already contacted (and been contacted by) some FOSS communities and individuals, and I am looking forward to meeting them. If you live somewhere around the places I will cycle through, stay tuned to take part or send me a message if you want to meet and geek around!

The ultimate success in that respect would be bringing new people to the FOSS community to continue what I have been doing as a blogger and as a co-mentor at the LKMP. If you know any groups that might be interested (e.g. cultural associations, schools, DIY clubs), please give me their contact, so I can meet them and “sell” them the FOSS culture :laughing: I would do it with pleasure and 100% for free.

The following slide from my presentation at the Open Source Summit Europe 2024 in Vienna summarizes pretty well what I have in mind:

3. Can I help you?

Definitely! Depending on your situation and willingness, you can help me in multiple ways:

Locals

A big part of this adventure is about discovering new places, cultures, and ways of seeing life. And the best way to do that is, of course, by connecting with the people who live there. It would be wonderful if you could take some time to show me your world from your perspective! If you can also help with more practical matters, like suggesting which paths to take, what to visit, or offering me a roof, so my tent doesn’t get wet on a rainy day, I would be really grateful.

Advice from a similar experience

Although I would love to hear from any bikepacking experience, those who did some software development along the way (I know there have been some) or generated public content regularly could really help me: hardware/software tools, ways to better communicate… whatever you found handy!

Reward my content

If my posts on this site or on social media helped you, and you would like to acknowledge my work, please consider contributing back. You will find a “Buy me a coffee” button at the end of every article as well as in the new Support me section, where you can also support me via PayPal. What I say for your patches to FOSS projects is valid here too: any contribution, no matter its size, is always awesome.

If none of the options above suits you, you can always help me by increasing the reach of my content. If you share it on social media, with your friends or colleagues, you are already helping me. Not reaching out is a very common cause of death among sources of digital content, and you could save this one.

4. Building site

If you regularly visit this page, you will have noticed that it looks slightly different. I am still working on it, but the idea is to keep a minimalistic design while improving the looks and navigation features. My apologies for any inconsistencies you might find, I am learning along the way :smiley: Feel free to send me feedback from the things that are still off.

Now there are two separate sections for tech and bikepacking articles, as the audience for these two areas don’t necessarily overlap. If you are interested in both, the home site will list all the articles in chronological order.

Now that I will be publishing content more regularly, I would like to offer more options on social media. I have already added some links to my new profiles on Facebook and YouTube in preparation for the future. You will not find much yet, but you can click on “follow” and move on, so you will be notified when I start publishing stuff.

There is also a new section to subscribe to my newsletter. So far, I have only used LinkedIn to tell my contacts that new articles were available. That is far from optimal, as not everyone uses that platform regularly. Now you can get it in a more reliable fashion via email. I will only use the newsletter for what the name hints: don’t worry about spam, promotions, or anything like that. Just notifications about new content :innocent:

I’ve also added some features to make navigating the site easier: links to previous and next posts, quick access to different sections, and a rapid search feature to help you find what you’re looking for in just a second.

I’ll keep improving the site whenever I can find some time to enhance the reader experience. That’s it, I hope you enjoy reading my articles, and as always, any suggestions are more than welcome!

Smoothing out Communication in FOSS Projects

Wed, 06 Nov 2024 08:00:00 +0000

Most of the articles about Open Source (including mine) only focus on technical aspects: coding, statistics, results, etc. But as we are not machines (just yet), I think that the human aspects involved in such projects should have some visibility as well. In the end, we are interacting with communities and their individuals, who have their perspectives and communication skills.

What sets communication apart in FOSS projects is the potential number of people involved and the visibility that the conversations can achieve. As long as everything goes well, this communication is wonderful, and a great source of information for the entire community. But when it goes awry, and someone decides to deviate from the cordial and professional tone that one expects, such visibility can lead to someone feeling intimidated, and even humiliated. Since none of us want these things to happen, I’d like to contribute my small share and discuss a few points that could help new as well as more established community members.

First, I would like to emphasize that at least 99.9% of the interactions in FOSS projects are smooth; usually polite and professional, and from time to time even welcoming and friendly. If only the worst examples reach the public, it’s just because they are exceptional… and because that’s what sells :joy: Although such cases are uncommon, one day you might find yourself being the target of someone’s rage, or even worse, the one who put a community member to shame.

I must also emphasize that I am not an expert in communication or personal skills. I am just above average when it comes to being friendly, talkative, and assertive. That might not be saying much when you are comparing yourself with the average developer :wink: but it has been enough to successfully manage all conflicts I have faced without leaving any “corpse” behind, and also support a couple of LKMP mentees who were a bit overwhelmed with some feedback they got from their first contributions to the Linux kernel. I even attended some Summer courses about soft skills at the university, and oh surprise, I was the only student from any engineering faculty. That’s an area where you won’t see many techies showing up, although that is changing now that many big tech companies explicitly require such skills.

I will split this article into two sections: a first one for those who just joined a FOSS community, and a second one for experienced community members who might profit from smoother discussions upstream.

Content:

1. Newcomers
2. Community Leaders

1. Newcomers

I am not going to show bad examples to keep them from getting publicity, but let’s suppose you have sent a patch upstream with the best intentions, and you get a reply that you consider aggressive or even offensive. You might have never received such replies in your professional or academic interactions, and maybe they are almost impossible to happen where you come from. It should have never happened, but there it is. Let’s see what you could/should do at this point.

1.1 Don’t take it personally

The person who sent you that is neither a relative nor a close friend. It’s someone you probably don’t know, and not someone who really matters in your life. And the other way round applies too: that person doesn’t know you, and they are only replying to some (probably unknown) e-mail address. You are being criticized for some message/piece of work that does not define you personally or professionally. Of course, you could take it personally and believe you have been punched in the face, but your chin doesn’t hurt, does it? Instead, you could take it for what it is: an overreaction that is out of place. As usual with virtual interactions, nonverbal communication is missing, there may have been some misinterpretations (not everyone’s mother language is English!), and the sender might not even be aware of having gone a bit too far.

1.2 Don’t reply immediately

I know absolutely nothing about guns, but I have always heard that you should keep guns and ammo in different places, so you have time to think before you shoot. Well, we can apply the bottom line to immediate replies too. If you have the slightest feelings about a public message you just received, DON'T REPLY JUST YET. Put your email client in the background, keep on doing whatever you are doing, and give yourself time to think. Digest what you just read, and get back a few moments later with a more structured reply. Every comment you send to public forums/mailing lists will stay there for a long time, and you might regret later having replied in the heat of the moment.

1.3. Be honest: did you mess up?

Beginners make a lot of mistakes for obvious reasons, and some of them annoy people because they are repetitive. The project documentation points them out, and you should have read about them before going live. Even in that case, a rude reply is not the way to go, but maybe you can still learn something new, or improve your workflow for the next attempt. If you messed up, consider adding a very short apology at the beginning of your next reply. It costs nothing, and then you can move back to the technical discussion. I have a good example of that: the typical HTML + company signature to a mailing list where only plain text is allowed.

I have been contributing to FOSS projects regularly with my professional email address for more than a year, and I made sure that our IT department keeps my emails untouched when they leave the company infrastructure. That worked well until last week I sent my very first patch to libcamera, a very cool project where I am an utter beginner. To my surprise, my email reached the mailing list in a fully-featured HTML fashion and a huge signature letting everyone know that my company was joining some event. A filter had been modified, and my address had been removed from it. Of course, I couldn’t have foreseen that, as it still worked a couple of days before. Not the best way to introduce yourself to an Open Source community, right? To be honest, I was expecting some unfriendly reply, but I got a polite comment letting me know what I had done wrong, and the rest of the discussion was only technical. I apologized for the trouble, made sure that it won’t happen again (or at least, not any soon), and carried on discussing the technical aspects. Here’s the link to the discussion.

If you, on the other hand, get criticized after making a technical mistake like giving a completely wrong solution to an issue, don’t be too hard on yourself. We all make mistakes, and if you are a beginner, many community members understand things better than you because they have the background and experience you are still missing. Take it as an opportunity to learn from people who know much more than you, and get ready to do better next time. Abandoning the community with your tail between your legs is the worst approach you could take.

1.3. Gather the technical content

Get back to the email and check if there is relevant feedback about the topic you are interested in. If there is none, and keeping the previous section in mind, you can simply ask the emitter to stick to the technical discussion, or ignore such a pointless message. But maybe there is something useful that you could use to turn the discussion in the direction you want, or even improve your original message or piece of code.

It’s time to leave the non-technical comments behind: stop tormenting yourself, gather the valuable information and try to work out how that could fit into a better technical solution.

1.4. Neutral, polite, and focused reply

Once you have decided to reply and what you actually want to reply, try to set an example by replying politely and focused on the technical aspects of the matter. The emitter might be a key stakeholder like a project maintainer, and you want to take the discussion back to the issue you are addressing, keeping all doors open for your current contribution and any others you might come up in the future. Read the Code of Conduct of the project (for example, the Linux Kernel Contributor Covenant Code of Conduct [https://docs.kernel.org/process/code-of-conduct.html]), and try to follow the recommendations, which are usually just common sense.

I would recommend you to leave the offending bits of the message you received uncommented unless there is a good reason to address them. Otherwise, the discussion could get stuck at the points you wanted to leave behind.

1.5 It went too far

If the feedback you got clearly violates the Code of Conduct, you are entitled to report it to the project maintainers. Of course, you should not let pass any flagrant abuse. But such cases are really, really rare, so make sure that you are definitely facing one of them first. Usually, the critiques come in the form of questioning your technical skills, your commitment to the project, and similar things that are unnecessary, but not disobeying the Code of Conduct… or at least not so clearly that you should report them.

Keep in mind that FOSS projects are also open for trolls and fake accounts. If you get ‘trolled’ (e.g. by fake-looking accounts or users with 0 contributions), don’t waste any time replying: just ignore the message and move on.

2. Community Leaders

No one is going to question that you are an expert in your area, maybe a truly master with unbelievable technical abilities. But now that you have reached that point on the technical side, why don’t you make the most of your interactions with FOSS Communities to improve your soft skills? Believe it or not, they could boost your professional career even further, and they will make you reach your technical goals more easily. Such skills only require some practice, which you will have in the discussions upstream anyway, and their principles are way easier to learn than a new programming language or technology. Even for a geek like you :wink:

2.1. Follow “Good Practices”

It goes without saying that you should follow the Code of Conduct of your project, if there is any. If there is not, it’s never too late to implement one! But let’s suppose there is one. There is usually a list of things that you should not do, and you are probably following those. That’s great, but often there is also a section about positive standards that are highly recommended. And that section does not have so many followers :laughing: Once again, let’s take the Linux kernel documentation as an example:

Examples of behavior that contributes to creating a positive environment include:

    Using welcoming and inclusive language

    Being respectful of differing viewpoints and experiences

    Gracefully accepting constructive criticism

    Focusing on what is best for the community

    Showing empathy towards other community members

Would you relate? Awesome! Otherwise, you might have found some room for improvement in your interpersonal skills. You wouldn’t believe how much better your comment will look if it starts with a “Welcome to the community” instead of “Your patch sucks” right before addressing the issues. Not only that, you will be setting better standards for the rest of the community, as people often mimic what they see from more experienced members.

2.2. You were once a noob, remember?

Maybe it was long ago, but hopefully you still remember when you joined your first FOSS project. How many beginner mistakes did you make until you got where you are now? Did everyone shout at you every time you messed up? Well, now you have a great opportunity to offer a better experience to the new generations. Not just because it will motivate newcomers to stay around and eventually improve your beloved project, but also because being nice to people is a good thing on its own. Now you that you are on the other side, probably in a privileged position within the community, you can help the community grow by making new members feel welcome.

Personally, I don’t get offended easily, and unpleasant comments just roll off my back. But every person has different sensitivities, and there is no need to make anyone feel uncomfortable, as if that would make them learn faster. No, it doesn’t, they will learn from polite and professional feedback just the same. I have had one case from the LKMP where a mentee would cross out some subsystem because the feedback was always too harsh, and the very same mentee had no problems in other subsystems. Don’t miss helping hands!

If you really think that someone needs a “punishment”, why don’t you lecture them by showing them how things really are? Think of that as an investment: that person will have some solid ground to work on a better approach instead of sending you “crap” again, and everyone who will read your public comment will send you better stuff.

I have a couple of good examples for you, from two Linux kernel maintainers I had never interacted with:

This one where I got lectured about some memory allocation by Kees Cook, explaining why my patch was a fix even though the bug could not be triggered.
This one where Nathan Chancellor dissected a misleading bug report from Clang about an unused variable (only used for a sizeof()).

As you can see, these well-known members provided detailed, technical-oriented replies, which helped me learn (I could reference Nathan’s comment shortly after for a similar issue), and everyone can access in the future.

2.4 Do you hate people? Automate replies

If you became a software developer to stay away from people and have zero interest in soft skills, that’s absolutely respectable. A FOSS project is not a social club, and you might only be interested in the technology that it covers.

But why don’t you optimize your time and let bots reply to anything that does not make sense to you? You can set filters, CI/CD pipelines, and combine them with automatic replies… a lot of cool stuff to keep humans out of the way until they come up with something useful. For example, many Linux kernel maintainers have implemented such facilities (not necessarily because they don’t like people, but because it saves loads of time), and a neutral, automated reply is better than one from an angry human. A win/win situation!

That’s it for today. If you would like to share some positive interactions in a FOSS community, please leave a comment!

Tips and Tricks to Build the Linux Kernel

Mon, 02 Sep 2024 20:00:00 +0000

Building the Linux kernel from scratch is very easy: you only need the source code (git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git), a few tools to be able to compile it (the list of minimal requirements is here), and simple commands like make defconfig && make.

That should be enough to generate a new kernel image, and many developers never go beyond that: make clean no matter what has been changed, and maybe make -j4 (for some reason, probably a copy+paste, it is always 4 :laughing:) to speed things up. That’s alright, but compiling the kernel takes time, and it exhausts machine resources quickly.

In this article, I discuss some basic notions every kernel developer should have as well as a few tricks/tools to compile the kernel faster, and potentially with a not-so-powerful machine.

Content:

1. Basics from the basics: use make properly
2. The obvious tip: only include what your kernel needs
3. GCC vs Clang: the compiler matters
4. ccache: recycle previous builds to compile faster
5. zram: you don’t need infinite RAM to build the kernel
6. distcc: build on multiple machines in parallel

1. Basics from the basics: use make properly

First things first: the Makefile in the root directory contains a help section, and make help is the first command you should use. There is no need to google the different configurations, how to compile with LLVM, or how to build the documentation. Everything is there, including what make clean, make mrproper, and make distclean mean:

$ make help
Cleaning targets:
  clean		  - Remove most generated files but keep the config and
                    enough build support to build external modules
  mrproper	  - Remove all generated files + config + various backup files
  distclean	  - mrproper + remove editor backup and patch files

You will seldom need mrproper and distclean, and if you are only working on a couple of modules, you don’t want to Remove most generated files either. Actually, make carries out incremental compilations, and if you only modified one file, only that file and the ones that depend on it will be compiled again if you call make without cleaning anything. If for whatever reason you want to clean/build a specific subsystem (more precisely, a folder where a Makefile resides), simply use the M= argument: make M=drivers/iio/humidity/ [clean].

About the ubiquitous -j4: that is telling the compiler to spawn 4 threads to run tasks in parallel. But why 4 and not 40 or 4000? That will depend on your machine and the current workload. If the workload is low (you are about to compile the Linux kernel, maybe not the best moment to overload your machine…), you usually want to have at least as many threads as cores your machine has, possible a few more to optimize workloads. In a nutshell, the nproc (number of processors) environment variable is usually passed instead of a hard-coded number, so you don’t have to edit your command depending on the machine you are using. Common values are -j$(nproc) and -j$((2 * $(nproc))). Simply try what works better for you, but don’t go for something like -j$((1000 * $(nproc))) because that will generate too much context switching (we will see later how the RAM consumption skyrockets), no gains at all, and the compilation will probably crash. For example, my machine compiles much faster with -j$(nproc) than with -j$((2 * $(nproc))), and I have to type less characters :laughing:

2. The obvious tip: only include what your kernel needs

Another thing that beginners inherit from 1-minute tutorials: always use make defconfig or even make allyesconfig. If you read the previous section, you should have already tried make help, which lists all possible configuration targets:

$ make help
# [...]
Configuration targets:
  config          - Update current config utilising a line-oriented program
  nconfig         - Update current config utilising a ncurses menu based program
  menuconfig      - Update current config utilising a menu based program
  xconfig         - Update current config utilising a Qt based front-end
  gconfig         - Update current config utilising a GTK+ based front-end
  oldconfig       - Update current config utilising a provided .config as base
  localmodconfig  - Update current config disabling modules not loaded
                    except those preserved by LMC_KEEP environment variable
  localyesconfig  - Update current config converting local mods to core
                    except those preserved by LMC_KEEP environment variable
  defconfig	      - New config with default from ARCH supplied defconfig
  savedefconfig   - Save current config as ./defconfig (minimal config)
  allnoconfig	  - New config where all options are answered with no
  allyesconfig	  - New config where all options are accepted with yes
  allmodconfig	  - New config selecting modules when possible
  alldefconfig    - New config with all symbols set to default
  randconfig	  - New config with random answer to all options
  yes2modconfig	  - Change answers from yes to mod if possible
  mod2yesconfig	  - Change answers from mod to yes if possible
  mod2noconfig	  - Change answers from mod to no if possible
  listnewconfig   - List new options
  helpnewconfig   - List new options and help text
  olddefconfig	  - Same as oldconfig but sets new symbols to their
                    default value without prompting
  tinyconfig	  - Configure the tiniest possible kernel
  testconfig	  - Run Kconfig unit tests (requires python3 and pytest)

One of them is indeed defconfig, and often it’s what you want to use if you don’t care much about the kernel configuration you are going to use (e.g. when writing a tutorial like this one, or the first time you build the kernel, and you don’t really know what you are doing). But defconfig will enable a bunch of features that you might not need for your quick test (e.g. running a minimal kernel on Qemu, like virtme-ng does… stuff for another article) or on your target machine (e.g. an embedded device). More features means more to compile, and more memory to store the resulting image.

Let’s see a simple example. First, I will build the mainline kernel fox x86_64 defconfig, and then tinyconfig, both with LLVM/Clang 18.1.8 (we will talk about GCC vs Clang later):

$ make defconfig
$ time make -j$(nproc)
real	4m43.643s
$ ls -lh arch/x86/boot/bzImage | awk '{print $5}'
13M

$ make tinyconfig
$ time make -j$(nproc)
real	0m31.355s
$ ls -lh arch/x86/boot/bzImage | awk '{print $5}'
581K

The tinyconfig build was ~9 times faster, and the image is ~22 times smaller. That shows how much room there is in between to build a tailored kernel. And if you are asking yourself “why so much trouble for 3 minutes?”, the results could have been much more extreme. Let’s see how long it takes to compile with allyesconfig:

$ make allyesconfig
$ time make -j$(nproc)
real	60m39.751s

Fine-tuning a kernel configuration is not trivial, and it usually takes a lot of time. But playing around with different configuration targets and then adding a couple of features you would like to try is not, and that will show you the basics to build upon. Bottom line: pay attention to your kernel configuration!

3. GCC vs Clang: the compiler matters

I have talked several times about why I find Clang (actually, the LLVM toolchain) a very interesting option to compile the Linux kernel: better diagnostics (in my experience, it tends to catch more bugs), and more sanitizers (e.g. you still need Clang >= 14 to use KMSAN). If you want to try the whole LLVM toolchain, you only have to pass LLVM=1 (either to make or as an environment variable), and if you only want to use the Clang, use the CC argument as follows: CC=clang. Easy-peasy!

On the other hand, I must say that on my machine, with the versions I installed, and the configurations I tested, GCC compiles a bit faster. Again, let’s give defconfig and allyesconfig a chance:

$ gcc --version
gcc (Ubuntu 13.2.0-23ubuntu4) 13.2.0

$ make defconfig
$ time make -j$(nproc)
real	3m24.128s

$ make allyesconfig
$ time make -j$(nproc)
real	47m23.330s

These results are much better than the ones I got in the previous section with LLVM/Clang. Does that mean that GCC is always faster? Not really. Actually, I have read multiple times that LLVM/Clang is often faster, so I suppose that there are many cases where it is. I am just saying that trying both is very easy, and in your particular case choosing one compiler over the other might make a big difference. Just give both a shot!

4. ccache: recycle previous builds to compile faster

Ccache is a C/C++ compiler cache that significantly speeds up the recompilation process. What it does is simple, yet clever: it stores the results of previous compilations and reuses as much as possible when similar compilations are requested again. By avoiding unnecessary recompilations, ccache can dramatically reduce build times, especially in large projects with many source files. A large project in C with many source files? The Linux kernel sounds like a perfect fit!

You simply have to install ccache, and pass it to the make command. How? A simple way (the one I use) is via the CC argument I mentioned before to specify the compiler, like CC="ccache gcc" or CC="ccache clang. This time I will build the same allyesconfig kernel a few times with LLVM/Clang to compare the results.

By the way, the official documentation to build Linux with LLVM mentions ccache and the need to set KBUILD_BUILD_TIMESTAMP to '' (actually, to a deterministic value) in order to avoid 100% cache misses, so I will follow suit here.

A normal compilation without ccache took ~60 minutes, as we saw in previous sections. Let’s clean and compile again, this time with ccache. I will make sure that there is nothing cached from previous compilations with ccache -C, and I will also zero the statistics with the -z argument (the -s argument will show the ccache statistics: size, hits, and misses):

$ ccache -Cz
Clearing... 100.0% [==============================================]
Statistics zeroed
$ ccache -s
Local storage:
  Cache size (GiB): 0.0 / 5.0 ( 0.00%)
$ make clean
$ time KBUILD_BUILD_TIMESTAMP='' make CC="ccache clang" -j$(nproc)
real	78m9.046s

Even worse! Well, that makes sense because nothing was cached, and the cacheable calls had to be stored. The real gain comes when you clean and build again:

$ make clean
$ time KBUILD_BUILD_TIMESTAMP='' make CC="ccache clang" -j$(nproc)
real	9m52.276s
$ ccache -s
Cacheable calls:    25679 / 30365 (84.57%)
  Hits:             25601 / 25679 (99.70%)
    Direct:         25595 / 25601 (99.98%)
    Preprocessed:       6 / 25601 ( 0.02%)
  Misses:              78 / 25679 ( 0.30%)
Uncacheable calls:   4686 / 30365 (15.43%)
Local storage:
  Cache size (GiB):   1.2 /   5.0 (24.55%)
  Hits:             25601 / 25679 (99.70%)
  Misses:              78 / 25679 ( 0.30%)

That looks much better. In this case nothing changed between the last two compilations, and almost all calls turned into cache hits.

In principle, you will modify multiple files before building again, and the hit rate will decrease. Hence why ccache is especially powerful if you usually work with the same kernel version, and only modify a few files every time (e.g. if you work with a stable kernel, or a given kernel version).

Let’s see what happens when I pull the changes made in a single day (some relatively quiet day between v6.11-rc4 and v6.11-rc5) in the master branch of the mainline kernel:

$ git pull
Updating 86987d84b968..928f79a188aa
Fast-forward
 MAINTAINERS                                    |  2 +-
 arch/loongarch/include/asm/dma-direct.h        | 11 -----------
 arch/loongarch/include/asm/hw_irq.h            |  2 ++
 arch/loongarch/include/asm/kvm_vcpu.h          |  1 -
 arch/loongarch/kernel/fpu.S                    |  4 ++++
 arch/loongarch/kernel/irq.c                    |  3 ---
 arch/loongarch/kvm/switch.S                    |  4 ++++
 arch/loongarch/kvm/timer.c                     |  7 -------
 arch/loongarch/kvm/vcpu.c                      |  2 +-
 drivers/platform/x86/amd/pmc/pmc.c             |  3 +++
 drivers/platform/x86/asus-nb-wmi.c             | 20 +++++++++++++++++++-
 drivers/platform/x86/asus-wmi.h                |  1 +
 drivers/platform/x86/x86-android-tablets/dmi.c |  1 -
 fs/attr.c                                      | 14 +++++++++++---
 fs/btrfs/bio.c                                 | 26 ++++++++++++++++++--------
 fs/btrfs/fiemap.c                              |  2 +-
 fs/btrfs/qgroup.c                              |  2 ++
 fs/btrfs/space-info.c                          | 17 +++++------------
 fs/btrfs/space-info.h                          |  2 +-
 fs/nfsd/nfs4state.c                            | 51 +++++++++++++++++++++++++++++++++------------------
 fs/nfsd/nfs4xdr.c                              |  6 ++++--
 fs/nfsd/state.h                                |  2 +-
 include/linux/fs.h                             |  1 +
 23 files changed, 112 insertions(+), 72 deletions(-)
 delete mode 100644 arch/loongarch/include/asm/dma-direct.h

Not many modifications, but we are using allyesconfig, and some of them like include/linux/fs.h will affect many files. Let’s see what happens then:

$ make clean
$ time KBUILD_BUILD_TIMESTAMP='' make CC="ccache clang" -j$(nproc)
real	76m51.382s

Once again, slower than without ccache, and only slightly faster than with an empty cache. There were not many cache hits, and even though a second compilation got back to very high cache hits, ccache might not be the best option if you always work with the latest mainline kernel.

As I said, if you only work on modules that are not required by many other elements of the kernel, ccache is indeed interesting. For example, I edited two of my device drivers (drivers/iio/humidity/hdc3020.c and drivers/hwmon/chipcap2.c, nothing depends on them), and the results are much better:

$ make clean
$ time KBUILD_BUILD_TIMESTAMP='' make CC="ccache clang" -j$(nproc)
real	9m53.534s
$ ccache -s
Cacheable calls:    25679 / 30365 (84.57%)
  Hits:             25599 / 25679 (99.69%)
    Direct:         25599 / 25599 (100.0%)
    Preprocessed:       0 / 25599 ( 0.00%)
  Misses:              80 / 25679 ( 0.31%)
Uncacheable calls:   4686 / 30365 (15.43%)
Local storage:
  Cache size (GiB):   1.8 /   5.0 (35.58%)
  Hits:             25599 / 25679 (99.69%)
  Misses:              80 / 25679 ( 0.31%)

Only two misses more than if you recompile without changing anything, which makes sense. I would recommend you to try ccache within your usual workflow to check if it makes sense for you or not, because if it does, you really will save much time.

Note that the cache size will increase with new cacheable calls, and at some point you might want to either clean the cache as we did above, increase the size (-M [SIZE]), or carry out any other operation supported by ccache. Anyway, ccache --help is your friend.

If you really want to squeeze ccache to get the most of it, take a look at its very complete documentation here. I am nothing more than a casual user whenever I work on a specific kernel version for several weeks. Maybe you will find some tricks I don’t know, and I will be pleased to update this tutorial and my own workflow. I doubt that this will happen, even if someone finds better tricks after reading this article… yet I still have a little bit of faith in humanity :grin:

5. zram: you don’t need infinite RAM to build the kernel

I did not know this trick until a mentee from the LKMP had issues to compile a “fat” kernel on a machine with limited resources i.e. RAM, and the compilation would crash every time. Fortunately, I never faced that issue myself, but if I had, I would have probably tried to increase the size of the swap memory, which would have been a partition of some slow memory (HDD/SSD). But that mentee had tried that as well, and the compilation would crash anyway. In the end, the hard disk is by no means as fast as the RAM, and if the lack of memory is severe enough, that workaround might fail.

Good news, old and not-so-powerful machines can profit from zram to make the most of their RAM. In a nutshell, zram is a kernel module (you will find it under drivers/block/zram) that gives you some breathing room by creating a compressed block device in your RAM. Essentially, it lets you use part of your RAM as compressed storage, which can be a lifesaver when you’re pushing your system’s memory limits. Instead of upgrading your hardware or dealing with sluggish performance, zram allows you to squeeze more out of the RAM you already have.

So how does this magic work? When you set up zram, it creates a virtual device in your memory where data is compressed before it’s stored. This means you can store more in the same amount of RAM, and your system stays more responsive, even when you’re doing something as demanding as building the Linux kernel. The canonical approach would be loading the zram module (which some distros seem to provide by default, some others will require you to install a package like sudo apt install linux-modules-extra-$(uname -r)), configuring it via sysfs attributes, and activating it. The official kernel documentation walks through those steps in detail, so there is no need to plagiarize here.

There is indeed a friendlier approach thanks to zram-tools, at least for Debian-based distros. Not that loading a module should be a problem (because you DO know how to load/unload kernel modules, don’t you??), but this package slightly simplifies the configuration process.

Actually, there is even a second package called zram-config that in the end seems to do more or less the same. I installed both, and zram-tools provides a nice systemd service (zramswap), and a man page too. I have been using zram-tools for a couple of days, and I am not missing any feature. Its documentation is not very extensive, but fair enough, so I will stick to it. I will not cover other use cases like mounting zram as a /tmp or /var storage, and for us, it will only be fast, extra swap memory.

As I am experimenting with zram, stressing tools, and compression algorithms at the moment, this section might get updated in the future with my typical, not-so-reliable measurements. But to get some taste of what you can expect from zram, I have tested it on a virtual machine with ~4G of RAM and 8 CPUs by compiling the kernel with -j$((10 * $(nproc))). As I said before, spawning so many threads consumes too much RAM, and the compilation will crash without zram, or even with a small amount of RAM assigned to zram.

Let’s try first with the default value of 10%. As you can see, the total size of the swap memory is now slightly bigger (~400M thanks to that new device /dev/zram0):

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           3.6Gi       1.0Gi       2.1Gi        41Mi       798Mi       2.6Gi
Swap:          4.4Gi          0B       4.4Gi

$ swapon
NAME       TYPE        SIZE USED PRIO
/swap.img  file          4G   0B   -2
/dev/zram0 partition 371.3M   0B  100

# you can also use zramctl without arguments to list your zram devices:
$ zramctl
NAME       ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 zstd        371.3M   4K   59B   20K       8 [SWAP]

zram with 10% of total RAM, right after crashing

Now I will try again, but using the example from Debian to give zram 60% of the total RAM and use the zstd compression algorithm:

$ echo -e "ALGO=zstd\nPERCENT=60" | sudo tee -a /etc/default/zramswap
ALGO=zstd
PERCENT=60
$ sudo service zramswap reload

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           3.6Gi       1.1Gi       2.0Gi        42Mi       800Mi       2.6Gi
Swap:          6.2Gi          0B       6.2Gi

$ zramctl /dev/zram0
NAME       ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 zstd          2.2G   4K   59B   20K       8 [SWAP]

Now that the swap memory increased, and a big chunk (2.2G) is actually RAM memory, things look different:

zram with 60%: barely responsive, yet alive!

Of course, zram is no panacea for all cases, and it has some downsides: less RAM without the overhead of the compression/decompression operations, and according to the documentation, about 0.1% of the size of the disk gets eaten by zram even when not in use. The expected compression ratio is 2:1, so even in the most extreme case, you won’t be able to go further than that. In short, if you have no memory issues, you would be better off without zram.

Although I don’t have memory issues at the moment, I am very interested in this feature and how far I can get with it, because I will need it for a “project” where I will only have access to a low-power machine to compile the kernel, at least for a year. More about it soon…

6. distcc: build on multiple machines in parallel

So, you’ve just read about zram and how it helps when your system’s low on resources. What if you do have loads of resources, but scattered across multiple machines that are being underutilized at the moment? Would it not be great if they could work together and boost your kernel compilation? Well, that’s exactly what distcc achieves.

Distcc is a tool that, according to its official site, distributes the compilation of a project across several machines in a network, splitting the workload. Apparently, you don’t even need to run the same OS on the machines you are going to use, but given that my use case is pretty clear (compiling the Linux kernel on a Linux-based OS), I am going to show you an example with two machines that run the same OS (Ubuntu 24.04.1 LTS), and the same versions of all the tools that are required to build the kernel (compiler, linker, etc.).

distcc supports both GCC and Clang as its back-ends (it seems that GCC is the main focus), which is great for kernel hackers. For this example, I am going to use the same version of GCC as before (13.2.0), and distcc 3.4 (the version I got installed via apt) on the two machines. localhost will be the machine where the build process will be triggered or client, and sloth will be the server that will receive requests to process (i.e. compile) from localhost.

Note that sloth is much less powerful than localhost (1/2 RAM, 1/2 CPUs, and slower), and there will be performance losses anyway due to network traffic (the best real performance with two identical machines seems to be ~1.89). Nevertheless, there should be some gains compared to a single machine. I am going to compile allyesconfig again to have some more work for the two machines.

First, and without pretending to write a tutorial about distcc, a very simple configuration that you can escalate for N servers:

(One-time step) Set static IP addresses: not mandatory, but nice to have if you don’t want to remember and write IP addresses every time. Simply give fixed IP addresses to your machine(s) (e.g. access to your router, typically 192.168.0.1 or 192.168.1.1, look for DHCP settings, and choose IP addresses by providing the MAC address of the interface, which you can get with the ip addr command).
(One-time step) Use the static addresses to provide names to the servers and add them to /etc/hosts like this:
```
$ cat /etc/hosts
127.0.0.1 localhost
192.168.0.15 sloth
```
From the man page: “On each of the servers, run distccd –daemon with –allow options to restrict access”. You can simply run distccd --daemon and the default will work for a local network. I only have one server, so I will run that command on sloth.
Adapted from the man page: Put the names of the servers in your environment: export DISTCC_HOSTS='localhost sloth'
Build. In this case I know that I have ~40 CPUs in total, so let’s keep it simple: make -j40 CC=distcc

After a couple of seconds, the network traffic to sloth increases drastically, and the CPU load will rise as well. If you want to know what’s going on behind the scenes, you can install distccmon-gnome to get a minimalistic UI. It is so minimalistic, that you only have to open it, and it will automatically start showing you what distcc is doing:

At the very least, useful to take screenshots for your reports

$ make allyesconfig
$ export DISTCC_HOSTS='localhost sloth'
$ time make -j40 CC=distcc
real	40m46.212s

In my particular case, which is probably not the most optimal one (only one server, Wi-Fi connections, and not particularly fast), the compilation time of allyesconfig dropped from ~47 minutes to ~41 minutes, which is good… although not enough for me to set it up every time I want to build a “fat” kernel. But users with multiple servers and a fast connection will probably love distcc.

If you are planning to use distcc on a regular basis, and potentially over a large number of servers, I would recommend you to read its man page carefully. You will find valuable information about the -j option we discussed before. Another interesting option might be the pump mode via distcc-pump, which will make distcc send the included header files as well to allow for on-sever preprocessing. According to the man page, by using the pump mode, the compilation could be an order of magnitude faster. I gave it a quick try to see what happens, and it did not even work on my machines with the version I have. But I did not invest any time on it, so maybe you will make it work. If you do, please let me know how.

And now it’s your turn to tell me your best tricks to build the Linux kernel quickly, and efficiently. This article is already very long, but I will be pleased to add another section. Whatever it takes for the community!

Catching runtime bugs with Valgrind and (HW)ASan

Mon, 19 Aug 2024 13:00:00 +0000

Without setting a precedent, this time I am going to talk about some userland tools. In particular, I would like to discuss how to catch memory-related runtime bugs with Valgrind and AddressSanitizer (ASan). How come? Because I have mentioned equivalent tools for the Linux kernel in other articles, and I would like to cover them in more detail. But I have noticed that many beginners were completely unaware of any dynamic analysis tool, and that must be fixed immediately.

These tools are almost a must if you code in C/C++, and if you use them wisely, they will save you from rewriting thousands (potentially millions) of lines of C/C++ code in Rust :laughing: Believe it or not, that’s not the only way to fix existing code, even if some rustaceans tell you otherwise. Apparently, some sorts of crabs are able to process words like lifetime and trait, but not realistic or overkill…

Content:

1. A buggy program
2. Attaching Valgrind to a binary
3. Faster with (HW)ASan
4. Attaching Valgrind to systemd services
5. What about undefined behavior, threads, or flow integrity?
6. Support for sanitizers in Meson and Qt

1. A buggy program

I have prepared this very simple program in C that generates a number of memory-related bugs on demand: use-after-free, buffer overflows, etc. You can use it to trigger the bug you are interested in, and see what the reports look like and what information you can expect. All bugs will be caught by Valgrind/ASan, and it should be easy to apply what you learned to any other program, simple or complex.

Note that the compiler will catch some bugs when you build the binary, but just because they are too obvious. You should get used to look for warnings in the compilation process, but keeping in mind that the compiler is not the most powerful tool to catch bugs, and that is usually not enough. In fact, half of the bugs are not noticed by GCC, and clangd does not find them either.

Static analysis is not enough!

2. Attaching Valgrind to a binary

Alright, let’s give the program a try! I will use Valgrind first to see if it is able to catch the bugs on the fly. By the way, the ‘valgrind’ package is available in all distros I know, just install it. Valgrind could not be easier to use: valgrind [--args] ./program. It accepts a bunch of arguments, some of them already set to default values that cover the most common use cases (e.g. memcheck as the default tool). You can see the whole list by running valgrind --help.

First, an example without arguments:

# ignore the warnings, we want to have those bugs in the code!
$ gcc -o bug-generator bug-generator.c
$ valgrind ./bug-generator
==16416== Memcheck, a memory error detector
==16416== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==16416== Using Valgrind-3.22.0 and LibVEX; rerun with -h for copyright info
==16416== Command: ./bug-generator
==16416==

Select a bug to generate:
1. Use after free
2. Heap buffer overflow
3. Stack buffer overflow
4. Global buffer overflow
5. Use after return
6. Use after scope
7. Use uninitialized
8. 1000 memory leaks
X. Exit
Enter your choice: 8
Done

Valgrind has started tracking our program (whose PID is the number at the beginning of every line). Then the menu is displayed, and I have generated 1000 memory leaks, which is one of the bugs that the compiler did not catch. Valgrind did not complain yet, because the memory will be definitely lost once the program ends. Until then, it will be in use (but already wasted). Let’s end the program and see what happens:

Select a bug to generate:
1. Use after free
2. Heap buffer overflow
3. Stack buffer overflow
4. Global buffer overflow
5. Use after return
6. Use after scope
7. Use uninitialized
8. 1000 memory leaks
X. Exit
Enter your choice: X
==16416==
==16416== HEAP SUMMARY:
==16416==     in use at exit: 4,000 bytes in 1,000 blocks
==16416==   total heap usage: 1,002 allocs, 2 frees, 6,048 bytes allocated
==16416==
==16416== LEAK SUMMARY:
==16416==    definitely lost: 4,000 bytes in 1,000 blocks
==16416==    indirectly lost: 0 bytes in 0 blocks
==16416==      possibly lost: 0 bytes in 0 blocks
==16416==    still reachable: 0 bytes in 0 blocks
==16416==         suppressed: 0 bytes in 0 blocks
==16416== Rerun with --leak-check=full to see details of leaked memory
==16416==
==16416== For lists of detected and suppressed errors, rerun with: -s
==16416== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Once we program ended, Valgrind generated a report where we can see that some memory was leaked. In this case, 4000 bytes were leaked. That matches 1000 times the size of an int, which is exactly what the program allocated. As the whole allocation was not carried out at once, we have 1000 tiny blocks of wasted memory. On the other hand, the report does not give much more information: good to know that there is a memory leak, but you usually want to know where. Actually, Valgrind did not even treat the memory leak as an error. But Valgrind can do better than that, and the report gave us a hint: Rerun with –leak-check=full to see details of leaked memory. Let’s try that:

==21198== HEAP SUMMARY:
==21198==     in use at exit: 4,000 bytes in 1,000 blocks
==21198==   total heap usage: 1,002 allocs, 2 frees, 6,048 bytes allocated
==21198==
==21198== 4,000 bytes in 1,000 blocks are definitely lost in loss record 1 of 1
==21198==    at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==21198==    by 0x1094C1: memory_leak (in /home/jc/test/bug-generator)
==21198==    by 0x109655: main (in /home/jc/test/bug-generator)
==21198==
==21198== LEAK SUMMARY:
==21198==    definitely lost: 4,000 bytes in 1,000 blocks
==21198==    indirectly lost: 0 bytes in 0 blocks
==21198==      possibly lost: 0 bytes in 0 blocks
==21198==    still reachable: 0 bytes in 0 blocks
==21198==         suppressed: 0 bytes in 0 blocks
==21198==
==21198== For lists of detected and suppressed errors, rerun with: -s
==21198== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

This time the memory leak is an error, and we even got a backtrace that we could use to find the bug. Unfortunately, the line numbers are not displayed. Why? Because the program was not built with debug symbols. That will usually be the case in production, and you will have to make the most of what you get from Valgrind. We are not in production, so let’s add debug symbols and run the same test again:

$ gcc -g -o bug-generator bug-generator.c

The new backtrace looks like this:

==21898== 4,000 bytes in 1,000 blocks are definitely lost in loss record 1 of 1
==21898==    at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==21898==    by 0x1094C1: memory_leak (bug-generator.c:53)
==21898==    by 0x109655: main (bug-generator.c:99)

The memory allocation happened in line 53 within memory_leak(), and it was never freed.

3. Faster with (HW)ASan

That’s all great, but if you can re-compile the program (often you can’t or don’t want to), you could use ASan instead of Valgrind. Why would you do that? Because Valgrind heavily affects runtime, and according to the Valgrind documentation, some programs could run up to 30 times slower. According to its own documentation, the average slowdown of the program due to ASan is around 2x. That is very convenient when your program needs to meet some timing constraints. For example, I have used both Valgrind and ASan to analyze more complex programs with Gstreamer video pipelines, and with Valgrind I was limited to ~4 frames per second. On the other hand, I could easily reach 30 fps with ASan.

In order to run our program with ASan, you don’t need to install anything unless you are using an incredibly old compiler (LLVM < 3.1 or GCC < 4.8). You simply have to compile the program with a new argument: -fsanitize=address.

$ gcc -g -fsanitize=address -o bug-generator bug-generator.c
$ ./bug-generator

This time you can execute the program alone, and after running the same test and finishing the program, you will get a report like this one:

=================================================================
==35553==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 4000 byte(s) in 1000 object(s) allocated from:
    #0 0x7986bbefbb37 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x55cad6b98863 in memory_leak /home/jc/test/bug-generator.c:53
    #2 0x55cad6b98acd in main /home/jc/test/bug-generator.c:99
    #3 0x7986bba2a1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #4 0x7986bba2a28a in __libc_start_main_impl ../csu/libc-start.c:360
    #5 0x55cad6b98244 in _start (/home/jc/test/bug-generator+0x2244) (BuildId: 2fc43751cfb164be24514984620f6f1b3afa2ce9)

SUMMARY: AddressSanitizer: 4000 byte(s) leaked in 1000 allocation(s).

Pretty much the same, right? And the execution time of the loop dropped from ~8 ms with Valgrind to ~1.6 ms with ASan. 5 times faster, not bad! But it comes at a price: the program size and the memory usage have increased. The former is usually not relevant, but still: the size of a trivial program like the one we are using here went from 19 KB to 33 KB. That might not be an issue on your computer, but a set of much bigger programs running on a tiny embedded system could be a different story.

If your program is running on arm64 (apparently, there is some sort of support for x86_64, but very restricted), you could still try HWASAN (hardware-assisted AddressSanitizer) to reduce memory usage. You only need to pass -fsanitize=hwaddress instead, but as stated in the documentation, it has a couple of drawbacks. Read the documentation carefully to save time when things don’t work as expected.

For the sake of completeness, I cross-compiled the same program for aarch64, and I got the following binary sizes:

aarch64-linux-gnu-gcc -g -o bug-generator bug-generator.c: 72 KB
aarch64-linux-gnu-gcc -g -fsanitize=address -o bug-generator bug-generator.c: 83 KB
aarch64-linux-gnu-gcc -g -fsanitize=hwaddress -o bug-generator bug-generator.c: 75 KB

The one compiled with -fsanitize=hwaddress is indeed smaller, but in most cases that reduction will be negligible. On the other hand, the RAM requirements are apparently the real gain. I have tested both programs multiple times on a Raspberry Pi, and the results were consistent and promising:

THIS IS NOT A SERIOUS TEST! But it seems to work...

I am going to ignore the fact that the binary for arm64 is way bigger than the one for x86_64 with the arguments I passed, because that is out of the scope of this article. ABIs, compiler optimizations… stuff for another article! If you are interested in this topic, I would recommend you to run the same experiment with clang (the original target for ASan) instead of gcc… a couple of surprises are waiting for you :wink:

Valgrind currently supports more architectures than ASan. On some architectures you will have no choice, no matter if you can re-compile the program or not. Moreover, Valgrind not only offers debuggers, but also multiple profiling tools. Believe me, Valgrind is a really awesome, yet ofter underutilized tool suite!

4. Attaching Valgrind to systemd services

Usually you won’t run your program manually, and especially in embedded systems, it will probably be run as a systemd service. In that case, attaching Valgrind to it is still very simple. You just have to find the .service file that references the binary to be executed, and add valgrind to the ExecStart line:

# Before:
# ExecStart=$YOUR_PATH_TO_THE_BINARY/$YOUR_BINARY
# After:
ExecStart=valgrind $YOUR_PATH_TO_THE_BINARY/$YOUR_BINARY

Then you will have to reload the service file, and restart the service:

systemctl daemon-reload
systemctl restart $YOUR_SERVICE

You can stop it by simply calling systemctl stop $YOUR_SERVICE, and journalctl will show you the Valgrind report. Basically the same we did before, but with systemd stuff in between.

If you are using ASan, you don’t have to modify the service file because it is already running within the program. Just restart the service, and let it pick the re-compiled binary.

5. What about undefined behavior, threads, or flow integrity?

I have restricted this article to address sanitizers, but there are so many others that this article could have become a book to cover them all. Now that you know how they work, using a different one (or several at the same time, if they are compatible) is as simple as editing the arguments you pass to Valgrind or the compiler.

There is almost a sanitizer for every kind of bug/security issue you could imagine, and new sanitizers appear every now and then. But some of them are widely used, and you should at least know that they exist (and probably use them regularly!). Here’s a short list with a few sanitizers you could use to instrument your code:

Thread Sanitizer: -fsanitize=thread, for data races. By the way, Valgrind also offers two thread debuggers: Helgrind and DRD.
Memory Sanitizer (MSan): -fsanitize=memory, for uninitialized values.
Undefined Behavior (UBSan): -fsanitize=undefined, for… yeah, undefined behavior, like dividing by zero.
Control Flow Integrity (CFI): -fsanitize=cfi, to abort the program if vulnerabilities that could be used to modify the program’s control flow are found.

If you want to learn more about those sanitizers, and some others, please refer to the official Clang and Valgrind documentation, so I don’t have to reinvent the wheel.

6. Support for sanitizers in Meson and Qt

In order to ease and enforce their usage, many tools have integrated support for sanitizers, so you can use their own configuration variables to include them in your applications. I will just mention Qt and Meson because they are widely used in embedded systems, and I have used them lately, so I can confirm that the code can be instrumented very easily. But I am pretty sure that you will find similar approaches within many other common tools.

To instrument your Qt applications, pass the required flags (sanitizer and then the list of sanitizers) to the CONFIG variable, usually within a .pro file. For example, you could include and the thread sanitizer and UBSan as follows:

CONFIG += sanitizer sanitize_thread sanitize_undefined
Meson provides the b_sanitize base option (e.g. -Db_sanitize=undefined for UBSAn), and of course you could pass the -fsanitize argument “manually” with something like add_project_arguments('-fsanitize=undefined', language: 'c') for UBSan in a C project.

As you can see, the only excuse you really have to refuse using sanitizers is laziness. And if you still decide not to use them, don’t complain the next time you find yourself looking for a needle in a haystack!

Linux Kernel - Mi first 100 patches in the mainline kernel

Sat, 13 Jul 2024 10:00:00 +0000

I have said several times in different articles that contributing to the Linux kernel is both satisfying and rewarding.

Some people have asked me if by rewarding I mean that you get some kind of public recognition from the community, maybe some sort of digital award you can add to your resume. Today I saw that some of my bug fixes were applied to some of the latest 6.10 release candidates, and apparently I won’t have to wait for the next merge window to reach the 100-patch mark in the mainline Linux kernel. Now I can tell you what happens when you get that far.

If you just sent your first patches, but nothing special happened, keep on working hard, and the awards will come…

Would you like to know what happens when you get 100 patches applied to the mainline Linux kernel? Alright, I am going to tell you a secret: you receive a certificate for your unrivalled achievement from the Linux Foundation with a dedication by Linus Torvalds!

Look at mine:

. . . . . . . .

You could not possibly believe that you get a real one!

Jokes aside, I believe that tracking the status of your patchsets as well as your overall progression is not a bad idea as long as you don’t get obsessed with it. In this article I would like to summarize what I have done so far, show some simple mechanisms to follow your contributions, and my opinion about the value of having X patches applied to the Linux kernel.

Content:

1. Following the status of your contributions
2. Simple metrics: what do my contributions look like?
3. Quantity vs Quality: hard to measure
4. My opinion about such metrics

1. Following the status of your contributions

That is something you should aim for. Sending series and forgetting about them 10 minutes later is not a great idea because receiving feedback is highly probable, but by no means guaranteed. Not only because your series could be ignored for a number of reasons, but also because it could have been applied without you being told. That does not happen often, but a few of my patches went that way. Spam filters and other issues related to the email provider are also a real thing. Therefore, I would recommend you to check the status of your work upstream from time to time.

Apart from receiving direct feedback from reviewers and maintainers via email, there are several ways to track the status of your contributions. The most basic, yet reliable one, is git log. You just have to check out the branch where you would expect your commits to be (e.g. master branch of mainline, linux-next, or a subsystem tree), and run git log --author="your_name or email with any extra options you might require. Of course, you can write your own scripts based on git log to make it more powerful and tailored to your needs: number of commits since a given version, number of fixes, etc.

If you do so, you will probably see sporadically how many commits you have in the kernel tree. In my case I have written my own scripts to follow the progression of my mentees from the Linux Kernel Mentorship Program (LKMP) (of course I know what you are breaking apart, even when you “forget” to CC me :poop:), and I can use them to track mine as well.

The kernel used to be a black box to me!

There are some other options out there like subsystem patchworks. Many subsystems use patchwork to keep a list of all patches sent to their mailing lists, and it is sometimes the fastest approach to see the status of all patchsets, including yours: you can see their general state (New, Superseded, Accepted…), filter them, or check what tags they got.

Patchwork: Linux USB

I must say that some subsystems simply ignore their patchwork (the tags like Reviewed-by are automatically tracked, though), and the tool itself is not perfect. For example, I have contributed to USB with two accounts, but I could only get the list for one of them. If you know a trick to list multiple authors, please let me know and I will update this point.

Another tool you probably know already is lore.kernel.org, a mailing list archive and interface for all Linux kernel discussions. Among many other things, you can track your submissions and all the replies you received. In principle, you should have received them as a reply to your email, but if for whatever reason you didn’t, it will be on lore.kernel.org for sure. And if you already access lore.kernel.org on a regular basis, lei (officially available for multiple distros) is a nice interface that will automate some steps you might have been repeating over and over again.

Now, let’s see an example about what metrics someone can get quickly and without much effort to get some rough idea about what has been done and accepted so far. I will also discuss the differences between contributing as a hobbyist and as a “professional”.

2. Simple metrics: what do my contributions look like?

I have two email addresses that I use to send patches: a personal one for my contributions as a hobbyist, and the company email address for my contributions as a “professional”. 75% of my contributions were made as a hobbyist, and when I look at linux-next, this proportion will not change much within the next weeks. How come only 25% of my contributions were made at work? Simply because as an embedded systems engineer, I have several tasks apart from Linux kernel development such as bringing up SoC-based boards, and BSP development.

Anyway, the difference between the patches coming from work and the homemade ones is noticeable. The first group is more focused on adding hardware support for some specific functionality we need for our systems, and it is basically limited to 3-4 subsystems. The second one is rather random: a bunch of bug fixes, a couple of new drivers for sensors I use in personal projects, and cleanups while learning new stuff (often spin-offs from my articles) in 8-10 subsystems. I do that for fun and to learn, so I don’t stick to any specific area.

How many patches belong to each category I mentioned? I took a quick look at the git logs, and it’s more or less as follows:

Bug fixes: 20% -> if your fixes start with “fix…” or use the “Fixes:” tag (they should), they are easy to find.
New drivers/devices: 10% -> you usually know what drivers you authored, also easy to track.
New features: 30% -> less obvious to find on a list, but mine is still not that huge.
Cleanups: 40% -> basically the rest.

Those numbers are tricky, though: most of the cleanup patches only modify a couple of lines of code (dt-binding conversions aside), while new drivers add several hundreds. New features (e.g. extending a driver to support some functionality the hardware offers) usually require something between 20 and 100 lines, and bug fixes are often single-liners. Is the removal of an explicit void * cast, making a struct const, or adding a missing .gitignore exactly the same amount of work as adding a new driver? Obviously not.

Should we then count the number of modified lines instead? That takes me to the next section.

3. Quantity vs Quality: hard to measure

Apparently, I have added/modified/deleted around 5400 lines of code, which results in ~54 lines/patch. Is that much? Well, ask someone who only submits fully-functional device drivers, and everything under 250 lines/patch will be peanuts. Ask “professional” bug fixers (they do exist), and they will probably say that they seldom need to modify more than 5-10 lines per patch.

If we follow a “the more lines, the better” reasoning, someone could ask if an easy dt-binding conversion (a few dozens of lines, often straightforward) is much more valuable than fixing a bug in a single line of code. Personally, I value the bug fix more because it usually implies deep understating of the code to identify the issue and find the right fix as well as knowing debugging tools and methodologies.

What matters then? As usual, the answer is it depends. Raw numbers don’t say much, but we are developers and even if we never wrote code for the Linux kernel, we all have a feeling about what a low-impact patch is, and what a high-impact patch could mean. Although that sounds rather subjective, and in many cases it is, some cases are clear: deleting 50 unused variables is something the compiler will (in principle) do anyway, and dropping average power consumption in arm64 by 10% sounds like an amazing achievement.

Having sent many patches or having modified many lines of code has some value by itself because it means that at least you know the workflow and the tools required to contribute to the Linux kernel, and you are probably familiar with some subsystem(s). Nevertheless, nothing beats our own judgment based on the code itself. Code neither lies nor pretends!

4. My opinion about such metrics

If you are a consultant or your company can profit from such advertising, paying attention to the number of patches/lines added to the Linux kernel is a good idea. Actually, most of the service providers that work on OSS use their contributions to advertise themselves, and they even publish reports with their weekly/monthly/annual contributions. You have probably seen such reports on LinkedIn or at the beginning of their talks in OSS conferences.

That makes perfect sense, and it’s even good for the community because they have to contribute to stay visible and keep reputation. No wonder why some low-impact cleanup series with 40+ patches (add const, drop unused, etc.) come from those companies. They have to deliver high-impact code too (which they do, and a lot), but showing up at the top-end of the lists is not to be neglected either. There is even this Kernel Patch Statistic, where patches and modified lines are listed by author, but also by company. Once again, competition is always welcome.

I remember a podcast where Greg K-H talked about that site, mentioning that someone had to do the low-impact cleanup, which in the end generates many patches for companies and individuals to climb they way up the list :laughing:

What about the rest of kernel hackers? I can speak for myself. First I had to reach 5-10 patches to graduate from the LKMP, so I counted every single contribution. Since then, the numbers don’t matter much to me, because as I said, I believe impact is key. At the beginning it was a good way to check if I was progressing or not, and the first marks (1, 5, 10, 20) really worked as motivators.

But once you reach a higher pace and start sending patches to several subsystems, you forget about that without noticing. Getting a new patch applied is always good news, but when you are working on multiple, unrelated series at the same time, you just move on and go for the next one.

On the other hand, I would never criticize someone who counts every patch and wants to reach X patches by the end of the year or whatever. If that helps to get a better kernel, thanks a lot for your work and don’t let anyone tell you what should matter to you!

That’s all for today. Reviewing my contributions so far was interesting for me to see what I have been doing lately. It is not that long that I sent my first patch, and I am happy about my learning curve and the variety of my patches. But this analysis was enough for good; time to hack again!

Rust in the Linux Kernel - Configuration

Sat, 22 Jun 2024 13:00:00 +0000

After spending some days learning how some compiler attributes (__cleanup, __counted_by) work in the Linux kernel, and fixing multiple bugs by using them properly, I have decided to move on, and start learning how Rust is used in the Linux kernel.

I have to admit that I know very little Rust. I completed the Rustlings almost a year ago, and since then I have been programming small userspace applications in Rust to learn the basics. But to be honest, that was only an intermediate step before applying Rust to lower-level stuff: microcontrollers and the Linux kernel. I will leave bare-metal stuff for another occasion, because it is a very interesting topic.

In this article, I am going to show you some very simple steps you can follow to configure Rust in the Linux kernel without stumbling upon version incompatibilities. The official documentation provides a detailed Quick Start guide, but it does not cover the installation of 3rd-party tools like LLVM/Clang and Rust itself. If you want to save some time, let me show you how you can get the right tools and versions without getting into too much trouble.

Most of the steps are distro-agnostic, but everything is mainly tailored for myself (Debian/Ubuntu). If you have a non-debian-based distro, simply install the latest builds wherever a script uses apt under the hood. Let’s get into it!

Content:

1. Requirements
2. Configuration
3. Build kernel and Rust documentation
4. Start playing around

1. Requirements

I am going to assume that you have installed all requirements to build the Linux kernel without Rust. If you haven’t, you are probably putting the cart before the horse. Anyway, please refer to the official documentation.

1.1. LLVM/Clang

The official documentation states that the LLVM toolchain is better supported than GCC. I have transitioned to LLVM/Clang as my default compiler long time ago, but let’s assume we have not installed LLVM/Clang yet*. Instead of installing from the distro packages (apt install & co), which are usually too old for what is required here, we are going to use a very convenient and official script for it:

# this script requires apt. Download and install a release build otherwise!
wget https://apt.llvm.org/llvm.sh
chmod +x llvm.sh
# don't be a Scrooge and get 'all' packages
sudo ./llvm.sh all

This will install the latest released version, which at the moment of writing is LLVM/Clang-18.

Note that if you run clang --version, you will get an error because the script did not create generic symlinks like sudo apt install clang would have done. It simply installs an LLVM/Clang version by adding the required repository and calling apt, and it is up to you to decide which one to use. But you may want to have generic names of the tools, unless you are willing to tell every program what version it should use (e.g. clang-format). For example, you could build the kernel with LLVM-18 like this: make LLVM=-18 instead of make LLVM=1.

There are multiple options to get generic names like creating symlinks under /usr/bin, or using the update-alternatives command. But you will have to do that for multiple binaries, and you could forget some of them. In my opinion the easiest solution is prepending /lib/llvm-18/bin to $PATH, because the generic names are already included, and I can choose a different version by editing 1-2 digits.

echo 'export PATH=/lib/llvm-18/bin:$PATH' >> ~/.bashrc
source ~/.bashrc
clang --version
Ubuntu clang version 18.1.8 (++20240615103833+3b5b5c1ec4a3-1~exp1~20240615223845.150)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /lib/llvm-18/bin

*If you do have LLVM/Clang installed on your machine, prepending the path to the desired version will still work. update-alternatives might also make sense if later (and often) you want to switch to a different version quickly, and without editing any file. The advantage of that is that it will also update the links to the man pages: man clang instead of man clang-18. Not a great deal, but maybe you care about that. Personally, an updated $PATH is all I need.

1.2. Rust

Once again, let’s get the latest version with an official command, this time from rustup.rs:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

You will be prompted with the following menu:

Keep it simple: Enter.

Everything should work fine, and you will be notified at the end:

Again, keep it simple: restart the shell.

This script has installed rustup, rustc and cargo.

1.3. bindgen

Apparently, bindgen is used to generate the bindings to the C code in the kernel. Good to know, let’s install it as stated in the documentation:

cargo install --locked --version $(scripts/min-tool-version.sh bindgen) bindgen-cli

As long as the min-tool-version.sh is up-to-date (it should), we will get the right version. In my case bindgen-cli v0.65.1 was installed.

1.4. New worktree

We are going to add a new worktree to isolate everything we will do for Rust. This is not a hard requirement, but don’t forget that Rust is still experimental, and you probably don’t want to add Rust by default to your kernel builds. Let’s add the worktree and move to its root directory:

# use the name you prefer instead of rust-playground
git worktree add rust-playground && cd $_

Now we are ready to configure the tools and the kernel. It didn’t hurt so far, did it?

2. Configuration

First, we are going to check if we meet the requirements to use Rust within the kernel:

make LLVM=1 rustavailable

***
*** Rust compiler 'rustc' is too new. This may or may not work.
***   Your version:     1.79.0
***   Expected version: 1.78.0
***
***
*** Source code for the 'core' standard library could not be found
*** at '/home/jc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/lib.rs'.
***
***
*** Please see Documentation/rust/quick-start.rst for details
*** on how to set up the Rust support.
***

Don’t panic, that was expected. We are using a too new version of the Rust compiler, and we are missing the rust-src component.

A “may or may not work” sounds risky, so let’s set the expected version with the following command, which only affects the working directory, not the version you will get for any other project:

rustup override set $(scripts/min-tool-version.sh rustc)

What is that rust-src component anyway, and how do I know we need that? I didn’t make that up, it’s in the official documentation:

The Rust standard library source is required because the build system will cross-compile core and alloc.
If rustup is being used, run:

rustup component add rust-src

Alright, let’s do that:

rustup component add rust-src
info: downloading component 'rust-src'
info: installing component 'rust-src'

At the moment of writing, we don’t have to add any other component (rustfmt, clippy, etc. were automatically installed).

And now, let’s try again:

# consider exporing LLVM=1 for the whole session
make LLVM=1 rustavailable
Rust is available!

Great! Are we done? Almost. We have to enable Rust support, and we are good to go:

make LLVM=1 menuconfig

Select General setup —> (first entry in the main menu), and then Rust support:

<Save>, <Exit>, Enjoy.

3. Build kernel and Rust documentation

We are done with the heavy work. To compile the kernel, simply run:

# consider using -jN (N = $(nproc), 8, etc.)
make LLVM=1
# ...
Kernel: arch/x86/boot/bzImage is ready  (#1)

We can also build the Rust documentation with a single command:

make LLVM=1 rustdoc

The generated documentation provides a lot of information about the supported crates, and it can be opened with your web browser:

xdg-open Documentation/output/rust/rustdoc/kernel/index.html

A lot of valuable information, don't ignore it!

4. Start playing around

It is up to you to decide what comes next. I can only recommend you to read the official documentation that I already mentioned, the rustdoc we just generated, and take a look at the existing examples under samples/rust.

Just a few tips from the official documentation for the sake of completeness before we call it a day:

The Rust support code can be found under rust/
To add sample modules: Kernel hacking -> Sample kernel code -> Rust samples
Additional configuration (not much yet, though): Kernel hacking -> Rust hacking
If you use the rust-analyzer language server, you probably want to generate the rust-project.json configuration file: make LLVM=1 rust-analyzer

I still don’t know what I am going to do with Rust, but as I like learning in depth before taking action, I will be busy with the existing code for a while. Nonetheless, if you have something in mind where we could collaborate, I will be pleased to learn about your projects.

Linux Kernel Development - Automatic Cleanup 2/2

Mon, 17 Jun 2024 19:00:00 +0000

This second and last episode about the automatic cleanup mechanisms in the Linux kernel covers the concept of classes, macros being used to automatically release resources like mutexes, and the ongoing work to increase its coverage. If you know nothing or very little about the cleanup compiler attribute and how it works, please take a look at the first episode, and then get back to this one.

The goal is to ensure that even beginners with only basic knowledge of C are not left behind, while also providing useful information for experienced developers who may not be familiar with the topics covered in this article. I have included very basic examples for the beginners, and slightly more advanced mechanisms for the experienced developers. Please pick whatever you find useful for your level, and if you notice anything I could improve to make things clearer, please provide your feedback!

Content:

1. Classes in the kernel
2. Automatic mutex handling
3. Other cleanup macros in the Linux kernel
4. Ongoing work

1. Classes in the kernel

Don’t get scared (or too excited), the Linux kernel has not adopted C++. Classes in the Linux kernel are not exactly what you might know from other programming languages: they are built upon the concepts we have seen so far to increase resource management automation.

Think of them as objects that provide a constructor and a destructor. If you know nothing about classes, constructors, and destructors, it does not matter: we are simply going to define a new type (typedef in C) out of an existing one, and create an automatic initializer for new objects of that type (the constructor). The destructor is the function for the automatic cleanup we have already learned. I will use a rather loose terminology throughout this article to make sure that any pure C programmer understands everything at first (or maybe second) glance.

At the moment of writing, classes don’t support fancy features you might know from other programming languages, and they might never do, so keep calm and continue thinking in C terminology.

How do we define a class, and how do we declare objects of the class? Once again, the macros defined in include/linux/cleanup.h are the way to go. In particular, we are going to analyze DEFINE_CLASS and CLASS:

// Macro to define classes:
#define DEFINE_CLASS(_name, _type, _exit, _init, _init_args...)		\
typedef _type class_##_name##_t;					\
static inline void class_##_name##_destructor(_type *p)			\
{ _type _T = *p; _exit; }						\
static inline _type class_##_name##_constructor(_init_args)		\
{ _type t = _init; return t; }

// Macro to declare an object of the class:
#define CLASS(_name, var)						\
	class_##_name##_t var __cleanup(class_##_name##_destructor) =	\
		class_##_name##_constructor

Before you give up: the macros are easier than they look, and in the end they do nothing more than defining a class and instantiating objects of that class. Not only that, you will find the following trivial example in the same header file, which I am going to digest for you:

DEFINE_CLASS(fdget, struct fd, fdput(_T), fdget(fd), int fd)

 CLASS(fdget, f)(fd);
 if (!f.file)
 	return -EBADF;

 // use 'f' without concern

This example shows how to use a class to automatically manage file references, but it could be any other resource you would like to get and then release, like a mutex (more about it in the next section). fdget is the class name, struct fd is the type we are going to use for our typedef, _exit is the destructor, _init the constructor, and _init_args... the arguments we want to pass to the constructor.

In this case, a class named fdget is defined, which is nothing more than a struct fd (some struct defined somewhere, it doesn’t matter) that calls fdget() with int fd as the argument when an object of the class is created. When the object (variable) goes out of scope, fdput() is automatically called.

If you are still lost, I got you. Let’s expand the CLASS macro based on what was passed to DEFINE_CLASS by simply replacing the parameters in the macro definitions with the ones used for the example:

/* DEFINE_CLASS(fdget, struct fd, fdput(_T), fdget(fd), int fd) expands to: */
// 1. typedef:
typedef struct fd class_fdget_t;
// 2. destructor:
static inline void class_fdget_destructor(struct fd *p) { struct fd _T = *p; fdput(_T); }
// 3. constructor:
static inline struct fd class_fdget_constructor(int fd) { struct fd t = fdget(fd); return t; }

/* CLASS(fdget, f)(fd) expands to: */
class_fdget_t f __cleanup(class_fdget_destructor) = class_fdget_constructor(fd);
// That looks similar to what we saw in the first episode, doesn't it?
// variable declaration + cleanup macro, and some initialization.

We have simply declared an object called f of the fdget class, whose type is class_fdget_t (basically a struct fd with new superpowers), by means of the CLASS macro. We have initialized f with our constructor that received fd as the argument for the initialization. The fd file reference will be automatically released when f goes out of scope with the mechanism we already know (here the destructor function), so we don’t have to worry about forgetting a call to fdput() every time f goes out of scope.

In principle, you will seldom (if ever) call DEFINE_CLASS yourself. The class definition is only made once, and then all users can simply use CLASS to declare the objects and used them as required.

Still not clear? I am running out of ideas, but I may have one more for total noobs. Let’s rewrite the program from the first episode to include the new macros and hence support classes. Feel free to copy the code and experiment with it:

/* class.c */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

struct foo {
	char *name;
};

static inline void foo_cleaner(struct foo *foo)
{
	printf("Bye %s\n", foo->name);
	free(foo->name);
	free(foo);
}

static inline struct foo *foo_builder(const char *name)
{
	struct foo *foo = malloc(sizeof(*foo));

	foo->name = strdup(name);
	printf("Hello %s\n", foo->name);

	return foo;
}

/************** OUR "API" internals **************/
// Macro to define classes:
// 1. typedef
// 2. destructor
// 3. constructor
#define DEFINE_CLASS(_name, _type, _exit, _init, _init_args...)		\
typedef _type class_##_name##_t;					\
static inline void class_##_name##_destructor(_type *p)			\
{ _type _T = *p; _exit; }						\
static inline _type class_##_name##_constructor(_init_args)		\
{ _type t = _init; return t; }

// Definition of the foo class
DEFINE_CLASS(foo, struct foo *,  foo_cleaner(_T), foo_builder(name), const char *name)
// __free() is used in the CLASS macro to set the destructor
#define __free(_name)	__attribute ((__cleanup__(_name)))

/******************* OUR "API"  *******************/
// Macro to declare an object of the class:
#define CLASS(_name, var)						\
	class_##_name##_t var __free(class_##_name##_destructor) =	\
		class_##_name##_constructor
/*************************************************/

int main()
{
	CLASS(foo, f)("Javier Carrasco");

	printf("Nice to meet you, %s\n", f->name);

	return 0;
}

Now let’s compile and run the program. I will use Valgrind to make sure that there are no memory leaks:

$ clang -g -o class class.c
$ valgrind ./class
==8111== Memcheck, a memory error detector
==8111== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==8111== Using Valgrind-3.21.0 and LibVEX; rerun with -h for copyright info
==8111== Command: ./class
==8111==
Hello Javier Carrasco
Nice to meet you, Javier Carrasco
Bye Javier Carrasco
==8111==
==8111== HEAP SUMMARY:
==8111==     in use at exit: 0 bytes in 0 blocks
==8111==   total heap usage: 3 allocs, 3 frees, 1,048 bytes allocated
==8111==
==8111== All heap blocks were freed -- no leaks are possible
==8111==
==8111== For lists of detected and suppressed errors, rerun with: -s
==8111== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

No memory leaks, 0 errors, and we got the expected output. Awesome!

By the way, why did Valgrind find 3 allocs and 3 frees when there are only two calls to malloc() and two calls to free()? I am sure that you can figure it out :wink:

(Leave this your second reading if you are struggling) If you open the cleanup header file, you will find another macro between DEFINE_CLASS and CLASS called EXTEND_CLASS. It is used to get a new class with its own constructor out of an existing class, and at the moment of writing is not used out of cleanup.h. That does not mean that it is dead code, as we will see in the next section, when conditional mutexes are created out of blocking mutexes.

Great, now that we know how classes work in the Linux kernel, let’s see a fairly common case: the mutex class.

2. Automatic mutex handling

Although it may have great potential, you will not find many real cases where DEFINE_CLASS is used in the kernel (there are currently 6 calls outside cleanup.h). Nonetheless, its usage for mutex handling is becoming pretty common in several subsystems, even though the macros are not called like we did in the previous section. Surprise, surprise: there are some more macros in cleanup.h, and the ones we are going to see now are specific for mutexes. Don’t panic, they are simple wrappers around the macros we just saw:

How do we define a new mutex (from now on, guard) class? With DEFINE_GUARD:
```
#define DEFINE_GUARD(_name, _type, _lock, _unlock) \
DEFINE_CLASS(_name, _type, if (_T) { _unlock; }, ({ _lock; _T; }), _type _T); \
static inline void * class_##_name##_lock_ptr(class_##_name##_t *_T) \
{ return *_T; }
```
Once again, you will seldom define a class for mutexes yourself. There will probably be one already that suits your needs. And if not, you will only have to use DEFINE_GUARD once for the required type.
How do we declare a new object of the mutex class? With guard:
```
#define guard(_name) \
 CLASS(_name, __UNIQUE_ID(guard))
```

As you might have guessed, there is already a class definition for the regular mutex struct mutex (in include/linux/mutex.h). That means that we can simplify the use of that mutex for a given scope with a single line like this: guard(mutex)(&my_mutex);, which will expand to what we know: a variable declaration + cleanup macro, and some initialization. In this case, the initialization is a mutex lock, and the cleanup is a mutex unlock. Exactly what we want.

You will find several examples in the kernel, for example in the IIO subsystem, where I used this mechanism for the hdc3020 and veml6075 drivers. Not releasing a taken mutex is terrible, and not that difficult to forget if the scope has multiple exit points (return, goto). The guard macro is short and simple, it works as expected, and having a reliable mechanism to avoid deadlocks gives peace of mind. By the way, you will find other macros that automate cleanups in that subsystem, because as we will see later, the maintainer is the author of some of them.

Before we move to the next section, I would like to mention another guard macro that you will find in the kernel, and is really nice: scoped_guard. It works like guard, but instead of applying for the current scope, it only applies for the immediate compound statement i.e. the next instruction or instructions within curly brackets:

scoped_guard(mutex)(&my_mutex)
    // do something

scoped_guard(mutex)(&my_mutex) {
    // do a bunch of things
}

Often you want to release the mutex as soon as the resource is available again, and there is no need to wait until the end of the current scope is reached. There has already been some refactoring to use scoped guards where the regular ones were used. Be wise and use the right tool for the job!

(Leave this for your second reading if you are struggling) DEFINE_COND_GUARD is a wrapper around EXTEND_CLASS to define conditional mutexes, which are out of the scope of this article. In a nutshell, and simplifying things, they return if the lock operation did not work instead of sleeping until the operation is permitted. There is also the scoped_cond_guard macro to work with scoped conditional mutexes, so you have multiple choices. Moreover, some subsystems use their own wrappers for specific tasks they carry out on a regular basis. But once you understand the basic macros, everything else is just syntactic sugar.

It might be worth mentioning that there are some additional macros (DEFINE_LOCK_GUARD_*, you will find them at the end of cleanup.h) to create guards with types for locks that don’t have a type themselves (e.g. RCU, where DEFINE_LOCK_GUARD_0, that receives no type argument, is used for that purpose. See rcupdate.h), or require a pointer to access/manipulate some data for the locking/unlocking (e.g. interrupt flags in spin_lock_irqsave).

3. Other cleanup macros in the Linux kernel

If you liked the scoped_guard we saw in the previous section, you are going to enjoy the following macros. It is fairly common in the Linux kernel that the lifetime of an object (whatever the object is, think of some allocated resources) is given by a refcount (to count references to the object). When that refcount reaches zero, no one needs the object anymore, and it can be released. It is often the case, that an iterator (e.g. a pointer) is used in a loop to iterate over a series of objects, incrementing the object’s refcount as long as it is in use to ensure its availability, and decrementing it at the end of the iteration. That sounds like a good fit for some kind of custom-scoped magic, doesn’t it?

A good example could be the iteration over nodes from a device tree. I have already talked about device trees in other articles like this one, so please check it out if you know absolutely nothing about them. I try to make the explanation generic anyway, so you can keep on reading.

The refcount handling in loops has been automated long time ago by keeping the refcount of the current object greater than zero as long as it is in use, repeating that operation until there are no more objects to iterate over, but there is a trick: that automation only works as long as the end of the loop is reached. But often we don’t want to iterate over all possible nodes, right? We could have found the one we were looking for, or even want to exit after some error occurs.

In those cases, the object’s refcount must be manually decremented to reach its original value before the iteration, and avoid a memory leak by preventing the object’s refcount to ever be zero (e.g. after removing a module). Manual intervention is always dangerous. Actually, I have fixed multiple bugs in such loops while writing this article by just looking for examples, like this one, this one, or this one. Even though a wrong refcount handling does not directly lead to a leak as long as the object stays in use, what will happen with that object in the future is often difficult to control and predict. By now you should be convinced that when used with care, the cleanup attribute is a great addition to the kernel.

The new macros to handle such iterative tasks have the same name as the old ones, plus the _scoped suffix. Their implementation is also similar, and by now you should understand the difference without any explanation from my side. Let’s see for example device_for_each_child_node() and device_for_each_child_node_scoped(), which was recently added by Jonathan Cameron (IIO maintainer):

#define device_for_each_child_node(dev, child)				\
	for (child = device_get_next_child_node(dev, NULL); child;	\
	     child = device_get_next_child_node(dev, child))

#define device_for_each_child_node_scoped(dev, child)			\
	for (struct fwnode_handle *child __free(fwnode_handle) =	\
		device_get_next_child_node(dev, NULL);			\
	     child; child = device_get_next_child_node(dev, child))

Exactly, whatever is going on within that loop, the __free() macro is the key to automatically release that fwnode_handle child after every iteration, no matter what that child is. As I said, it has something to do with inner nodes of some device from a device tree, but the important thing is to understand the pattern. Of course, there must be a DEFINE_FREE for the fwnode_handle to define the cleanup function, but that is also something we have already learned in the first episode.

Let’s see how you can easily mess up with the non-scoped version of these macros, and how easy the fixes with the scoped macros can be (not always, though… case by case!). In this case, I used for_each_child_node_scoped() to fix this memory leak in the sun50i cpufreq driver:

static bool dt_has_supported_hw(void)
{
	bool has_opp_supported_hw = false;
	struct device *cpu_dev;

	cpu_dev = get_cpu_device(0);
	if (!cpu_dev)
		return false;

	struct device_node *np __free(device_node) =
		dev_pm_opp_of_get_opp_desc_node(cpu_dev);
	if (!np)
		return false;

	for_each_child_of_node(np, opp) {
		if (of_find_property(opp, "opp-supported-hw", NULL)) {
			has_opp_supported_hw = true;
			break; // early exit with no fwnode_handle_put(opp) to decrement refcount!
		}
	}

	return has_opp_supported_hw;
}

Attach _scoped to for_each_child_of_node, and the bug is fixed. And if another early exit is added in the future, we are still safe. Is that not great?

4. Ongoing work

Even though the scoped versions of a number of such macros are already available in the mainline kernel, there are still others that only have the non-scoped variant. They are waiting for a use case where it makes sense (e.g. fixing a bug where manual release is missing), and therefore the number of members of the _scoped family will probably grow in the next months/years. Similarly, there are many objects that don’t have their own __free() or an associated class, and I bet that some of them could profit from that as well.

I sent myself this patch (still being discussed, mainly for other reasons than the cleanup mechanism itself) a few days ago to fix a bug in the hwmon ltc2992 driver where the current fwnode_for_each_available_child_node is not correctly used. The people who were involved in the design and implementation of the automatic cleanup mechanisms paved the path, and now we can profit from it to add new use cases like this one. Props to them!

The Linux Kernel Mentorship Program from the Linux Foundation is (among many others) also pushing the use of the __free() macro forward by refactoring existing code where error paths increase the risk of missing a memory deallocator. This task was proposed by Julia Lawall during a mentoring session about Coccinelle, and several patches have already been accepted. I must admit that some subsystems were more enthusiastic than others, but in general they are making their way to the kernel.

To be honest, I understand why some maintainers are pushing back. I refactored some easy code myself to be able to review my mentees’ work, and since then I restricted myself to adding the feature to new code or where it fixes real bugs and the improvement is undeniable. Some refactoring where a bunch of goto jumps vanishes is in my opinion a good patch to apply, but we have already seen that there are many things to consider to avoid regressions. A few maintainers still don’t feel comfortable with the auto cleanup stuff (some were completely unaware of it), and they could miss some potential regressions when reviewing patches. If you want to do some refactoring anyway, please be very careful, and make sure you understand the internals. I hope my articles helped a bit!

Apart from the ongoing work I mentioned, I suppose there is much more being discussed in multiple mailing lists. I will let you investigate on your own. Maybe you are even working on some extensions I don’t know… I would love to hear about it!

Hopefully the ongoing work will continue increasing code quality and above all, delivering a better kernel to the end user.

Linux Kernel Development - Automatic Cleanup 1/2

Mon, 03 Jun 2024 10:00:00 +0000

One of the most common criticisms of the C programming language is that dynamically allocated objects are not automatically released. And those who say this are right: memory leaks are a very common issue in C code, including the Linux kernel. Does that mean that C is useless, and the whole kernel should be rewritten in Rust as soon as possible? Definitely not, and even though some code is being rewritten in Rust, the great majority of the new code added with every release is still in C, and that will not change any soon. Instead, we should try to mitigate current pitfalls with new solutions… or simply start using the existing ones, like the Linux kernel recently did.

Content:

1. Background: underutilized cleanup compiler attribute
2. Walkthrough: the __free() macro step by step
3. Return valid memory, but keep on using auto cleanup!
4. Initialize your variables, and fear any “goto”
5. Why Rust then?
6. Why 1/2?

1. Background: underutilized cleanup compiler attribute

Note: this section paraphrases/plagiarizes/summarizes some code from include/linux/cleanup.h as well as this article by Jonathan Corbet on LWN.net, which I strongly recommend. Here I will just digest the key points and add some code snippets for complete noobs :wink:

Both GCC and Clang support a variable attribute called cleanup, which adds a “callback” that runs when the variable goes out of scope. If you never used compiler attributes, it is as simple as placing __attribute__ with the required attribute inside brackets right after the variable declaration. The cleanup attribute expects a function that takes a pointer to the variable type:

void cleanup_function(int *foo) {}

int foo __attibute (__cleanup__(cleanup_fuction));

Whenever foo goes out of scope, cleanup_function will be called. Here is a very simple example you can try and tweak:

/* cleanup.c */
#include <stdio.h>
#include <stdlib.h>

void cleanup_function(int *foo)
{
	printf("foo: I do! The answer is %d.\n", *foo);
}

int main()
{
	int foo __attribute ((__cleanup__(cleanup_function))) = 42;

	printf("What's the answer to everything?\n");
	printf("main: I don't know, bye!\n");

	return 0;
}

Let’s compile and run:

gcc -o cleanup cleanup.c
./cleanup
What's the answer to everything?
main: I don't know, bye!
foo: I do! The answer is 42.

If you can print a message, you can do more interesting stuff like freeing allocated memory, unlocking mutexes, and so on. I bet you are starting to grasp what we are aiming to achieve with this attribute.

Is that new magic? Well, the cleanup attribute exists in GCC since v3.3, which was released in 2003! Ok, then we have been using it in the Linux kernel for decades, right? Not really. It was first introduced in 2023 by Peter Zijlstra. Why? I don’t know. Maybe no one thought about it before, maybe the community did not like the approach… If someone knows, please leave a comment.

You could argue that adding the attribute to every variable that requires some cleanup does not look beautiful, and it is laborious. Yet, I still believe it should be used way more often in C projects that already use compiler extensions. And as we will see in a bit, there are some ways to save a few strokes, especially in the long run.

In case you are wondering why I talked about the cleanup attribute, but then I used __cleanup__ instead: the underscores are optional, and they can save you from collisions in case an existing macro is called cleanup. You can read more about attribute syntax in the official documentation.

The next section is my attempt to help you understand how this attribute is used in the Linux kernel to automatically release memory, and what happens under the hood. Why would you want to know what happens under the hood? Because if you don’t really know what you are doing, you will probably introduce bugs, and you will never be able to extend the mechanism to other use cases. As you will see in a bit, you can easily mess up, and new use cases for this mechanism will arise every now and then. Bear in mind that this feature is rather new in the kernel, and new macros are under review to extend its usage. Bleeding-edge development with 20-year-old compiler attributes!

2. Walkthrough: the __free() macro step by step

The cleanup attribute is really cool, and we are about to see an example where we will use it to free dynamically allocated memory. But typing the whole __attribute ((__cleanup__(cleanup_function))) is too tedious.

Instead, we could define a much shorter macro to call the cleanup function. I know that macros look like black magic for beginners, but don’t worry: they are usually nothing more than syntactic sugar for more complex code.

For example, we could save some typing with a macro like the following __free():

//wherever __free() is used, replace it with this long expression
#define __free(func)	__attribute ((__cleanup__(func)))

Let’s use our new macro to automatically free memory when a variable goes out of scope:

/* free1.c */
#include <stdio.h>
#include <stdlib.h>

#define __free(func)	__attribute ((__cleanup__(func)))

void cleanup_function(int **foo)
{
	printf("Avoiding a memory leak ;)\n");
	free(*foo);
}

int main()
{
	int *foo __free(cleanup_function) = malloc(10*sizeof(*foo));

	// Some (probably buggy ;)) code

	return 0;
}

I said before that clang also supports the cleanup attribute, didn’t I? Let’s see:

clang -o free1 free1.c
./free1
Avoiding a memory leak ;)

Awesome! Are we done? Well, we still have to pass the cleanup function to our macro. Letting everyone use their own cleanup function for a given type does not make much sense, let alone in a huge project like the Linux kernel. Furthermore, no one wants to memorize the name of the cleanup function for every type. Offering a simple API that hides the cleanup mechanism would be more efficient. The API user could simply call __free(type), and the rest would be transparent. Let’s add a new macro to generate the cleanup functions according to the type:

#define DEFINE_FREE(_name, _type, _free) \
	static inline void __free_##_name(void *p) {_type _T = *(_type *)p; _free; }

Wow, wow! What was that? Is that really new syntactic sugar? Yes, it is. Every time you use the DEFINE_FREE() macro, you have to pass a name to generate the cleanup function, the variable type, and the free mechanism. The name could be anything, but the variable type sounds reasonable, so we end up with a cleanup function called __free_type() like __free_int() or __free_foo(). We have used the handy ## preprocessing operator for that. The type is obviously the variable type we want to free, and the free mechanism is the code we want to execute in the cleanup function.

Some beginners might have found a different kind of black magic in the macro we just defined: void pointers. But again, don’t worry. They are useful and actually not that complex. We will use a void pointer to pass any pointer (e.g. a pointer to the variable type), and avoid cumbersome double pointers like the int **foo we used in the last example. Give it a try with void *foo instead, and everything should work like before.

Now that we have our DEFINE_FREE() macro, we can adapt our previous __free() macro to get the generated cleanup function instead:

#define __free(_name)	__attribute ((__cleanup__(__free_##_name)))

And now, another example, this time with cleanup functions for int * and a slightly more complex type strcut foo *:

/* free2.c */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

//Some more complex type
struct foo {
	char *name;
};

static inline void foo_cleaner(struct foo *foo)
{
	printf("Bye %s\n", foo->name);
	free(foo->name);
	free(foo);
}

/************** OUR "API" internals **************/
#define DEFINE_FREE(_name, _type, _free) \
	static inline void __free_##_name(void *p) {_type _T = *(_type *)p; _free; }

//Our cleaner for int *
DEFINE_FREE(int, int *, printf("Bye dynamic int = %d\n", *_T); free(_T))
//Our cleaner for struct foo *
DEFINE_FREE(foo, struct foo *, if(_T) foo_cleaner(_T))
/*************************************************/

/******************* OUR "API"  *******************/
#define __free(_name)	__attribute ((__cleanup__(__free_##_name)))
/*************************************************/

int main()
{
	int *pint __free(int) = malloc(sizeof(*pint));
	struct foo *pfoo __free(foo) = malloc(sizeof(*pfoo));

	*pint = 42;
	pfoo->name = strdup("Javier Carrasco");

	// Some code

	return 0;
}

This time we have programmed the cleanup in two different ways: directly in the DEFINE_FREE() macro for int * types (a simple free), and as a separate function for struct foo *, because a bit more code was needed. Alright, let’s see if it works:

clang -o free2 free2.c
./free2
Bye Javier Carrasco
Bye dynamic int = 42

Note that the order in which the cleanup functions are executed is not random. In fact, they are executed in the reverse order in which the variables with the cleanup attribute are declared in the scope. This is not relevant here, but it’s something to keep in mind.

If you understood everything I told you in this section, congratulations: the macros I used were taken from the Linux kernel*, and you don’t have to learn the same thing twice. If you are still trying to understand how attributes and macros work, don’t panic: a second, more thorough read will definitely help. Just don’t give up!

* The __free() macro from the kernel is not exactly like mine. Instead, an intermediate step is used to get the compiler-specific syntax for the cleanup attribute, increasing flexibility. What you will actually find in the kernel is #define __free(_name) __cleanup(__free_##_name), where __cleanup(__free_##_name) is defined as follows for Clang (in compiler-clang.h): #define __cleanup(func) __maybe_unused __attribute__((__cleanup__(func)))

3. Return valid memory, but keep on using auto cleanup!

What if the allocated memory is required later in the code? Will the cleanup attribute free memory anyway? Of course, it will, as soon as the variable goes out of scope. Does that mean that we can’t use that attribute in that case? No, someone has thought about it already.

We can use the cleanup feature like we did until now to automatically free memory in the error paths, and use a specific macro to return the pointer instead of freeing it. Yet another macro? Yes, and we will see a few more in the next chapter. Get used to them and stop complaining, because they are here to stay, and actually they are really useful.

We have two simple macros at our disposal: no_free_ptr() and return_ptr(). The first one uses a second pointer to store the memory address, setting the original pointer to NULL*. Why? Because the cleanup is going to step in anyway at the end of the variable’s scope. We are not inhibiting the cleanup, at least not explicitly, but the original variable (pointer) will be NULL and nothing will be done. The second variable (pointer) does not use the cleanup attribute, and the memory will not be freed. The return_ptr() macro is just a convenient return no_free_ptr(p).

If you have a function that returns a pointer to some memory, but only if nothing goes wrong, a simple return_ptr(your_pointer) at the end will be enough, and the error paths where the memory should be freed will be covered by the __free() like before. Of course, the caller will be then in charge of freeing the memory when it is no longer needed. If you forget the return_ptr() macro, you will be returning freed memory, which is definitely bad.

no_free_ptr() can be used on its own, and you will find several examples in the kernel. For example this patch uses that macro because memdup_user() frees memory itself if something goes wrong and returns a PTR_ERR() (error pointer). In that case, you don’t want to free the error pointer (big mistake), but propagate the error without giving up automatic deallocation otherwise.

Update: as Harshit Mogalapalli pointed out (thanks for your feedback!), there’s this recent patch by Dan Carpenter that fixes such potential misuse for objects that are directly released with kfree or kvfree by checking too if the pointer is an error pointer (if (!IS_ERR_OR_NULL(_T) instead of if(_T)). As you can see, this area is under heavy development!

* Technically, no_free_ptr() uses a second macro for it, called __get_and_null_ptr(). This is where the real black magic happens by means of two GCC extensions: ({}) and __auto_type. The first extension is used to define an expression that “returns” a value (__val), and the second one, more obvious, increases flexibility to work with different pointers. This second macro is only used internally, and as I mentioned, it just moves the content of the pointer with automatic cleanup to a new variable. You will find __get_and_null_ptr() in include/linux/cleanup.h as well.

The Linux kernel uses GCC extensions A LOT.

4. Initialize your variables, and fear any “goto”

Some mistake I have seen several times when beginners use this feature for the first time is variable declaration without initialization (think of RAII in other programming languages like C++). What does that mean? Look at the following snippet:

{
	struct foo *pfoo __free(foo);
	goto fail;
	pfoo = malloc(sizeof(*pfoo));
fail:
	return -1;
}

That goto fail is jumping over the variable initialization! When the end of the scope is reached, some random address will be freed! We should have initialized the variable when we declared it. If there was no value for the initialization, we should initialize it to NULL. A more subtle case is the following:

/* fail.c */
#include <stdio.h>
#include <stdlib.h>

void cleanup_function(int *foo)
{
	printf("I should not run, or at least print 42... %d.\n", *foo);
}

int main()
{
	goto fail;
	int foo __attribute ((__cleanup__(cleanup_function))) = 42;
fail:
	return 0;
}

Apparently, we have initialized the variable. But it should not matter, because the execution will jump over it, right? Unfortunately, this is not how things work. Let’s compile and run:

gcc -o fail fail.c
(ins) $ ./fail
I should not run, or at least print 42... 0.
# NEVER assume that uninitialized variables are set to 0!

Shit! What happened? The goto does not change the scope/reach of the variable (the block where it is declared), but the initialization does take place where we initialize them. Therefore, declaring a variable that uses the cleanup attribute will basically force us to refactor any previous goto in the same scope, and no new goto instructions should be added before the variable declaration. Thus, declaring them at the beginning of the scope and setting them to NULL when the desired initialization is not possible (because the value is still not available at that point) could avoid such problems.

By the way, that issue was obvious to me because clangd complained about it:

I wonder why LSPs are not mandatory...

The error message is a bit cryptic, but if we compile the same program with clang, we will get more feedback:

clang -o fail fail.c
fail.c:12:2: error: cannot jump from this goto statement to its label
        goto fail;
        ^
fail.c:13:6: note: jump bypasses initialization of variable with __attribute__((cleanup))
        int foo __attribute ((__cleanup__(cleanup_function))) = 42;
            ^
1 error generated.

Am I suggesting that clang (16.0.6) is better than gcc (13.2.0)? In this particular case… yes, by far! :laughing:

This would have not happened if we had used a return instead of a goto. Why? Because the compiler would have reached the return without finding the variable declaration, and it would have not been included in that branch. Moral: if you are using a goto to reach a label that only returns a value, simply return the value.

Jumping around is fun when you are a kid, but good programmers try to hold back. Use goto when you really need it, but be careful, or you will twist your ankle!

5. Why Rust then?

If we can use those macros to automate cleanups, we are making C completely safe! Let’s remove Rust from the kernel and keep on improving our beloved C. Well, there are still many other scenarios where nasty bugs can be (very) easily programmed in C. Ask your favorite chat AI: you will get a huge list of possible mistakes.

Furthermore, the cleanup attribute and the macros based on it can also be misused, and I will give you a couple of real examples… first of all, I believe that refactoring stable code is not always a good idea, even to increase code safety “for future modifications”. We have had a case in the LFX Mentorship where a mentee refactored some code to add the __free() macro without adding the required return_ptr(), and that mistake kept the kernel from booting. The patch had to be reverted, and several people had to get involved to solve the issue as soon as possible. This is how bad refactoring existing code can go. No one notices until it’s too late and shit hits the fan. Don’t you believe me? Then take a look at this recent fix for a regression while refactoring some code in the IIO subsystem to include the cleanup feature. This time, an experienced kernel developer (way, way more experienced than me) introduced the bug, and apparently, this regression could even cause skin burns! :fire: Moral of the story: be cautious because anyone can mess up, test your patch as much as you can, and favor new code or real bugs to introduce this feature until you master it.

On the other hand, I am sure that there are still many bugs hidden in the code that could be avoided with this feature. Actually, I fixed myself this memory leak in a driver from the cpufreq subsystem by adding a simple __free()*. It will take a while until everyone gets used to it, but all things considered, I would say that introducing the cleanup attribute was a good idea.

Anyway, competition is always good (like having GCC and Clang competing to become your preferred C compiler), Rust has been a reality in many successful projects for years, and (at least in theory) it could make kernel development more appealing for new generations. So let’s keep on increasing safety for C while letting Rust find its own way into the kernel.

* If you find a similar bug, consider splitting the fix into two patches: one with the “traditional” fix (e.g. adding the missing kfree()), and a second one with the cleanup attribute. This feature is not available in many stable kernels, and the first patch will be easier to backport.

6. Why 1/2?

This article is getting a bit long, and there are still many macros and use cases I did not mention: classes (yes, classes in C!), scoped loops, usage with mutexes… Some of that stuff is so new in the kernel that there is still not much real code to use as an example, but fortunately enough to learn the key concepts and stay up-to-date when it comes to Linux kernel development.

That’s all for today. Please send me a message if you find any inaccuracy, and I will fix it as soon as possible. Stay tuned and don’t miss the next chapter, whenever I get it finished…

Linux Kernel Development - Firmware API

Thu, 30 May 2024 00:00:00 +0000

The vast majority of devices supported by the Linux kernel require little to no intervention to start running: maybe powering up (e.g. via regulators), de-asserting a reset or accessing a few configuration registers. But a number of them are complex enough to require their own firmware to offer full (or custom) functionality. Some of such devices expect a pre-programmed persistent memory connected to them, but others provide some mechanism to receive their firmware from a software application. A few devices even offer both possibilities.

The first option is obviously simpler to develop: some extra hardware, and the firmware loading will just work. But extra hardware means extra costs, and that is often a key factor, let alone if you are designing an embedded device. Moreover, updating the attached memory with new versions might be more complex than updating the device itself.

On the other hand, programming a custom tool to load firmware for a specific device sounds wearisome as well as dangerous: accessing hardware from userspace, interfering with the device driver… and even if the manufacturer provides a firmware loader (seldom open source), it will not be easy to integrate with the rest of the system (including the device driver), especially if you are working with embedded systems. There must be a better way to do it.

Fortunately, the kernel provides an API to integrate firmware updates into device drivers, offering update automation, fallback mechanisms, and even control via sysfs. No nasty hacks or closed-source tools required! In this article I will discuss how the Firmware API works, and how you can integrate it in your device driver development.

Content:

1. How does it work?
2. Real example: TPS6598x PD controller driver
3. Other things to consider

Note: the official documentation is a much more complete source. It covers many more uses cases (like UEFI support), but it might be a bit overwhelming if you know nothing about this magic. Here I just summarized the key points to grasp the mechanism at a glance and understand a real example with mainline kernel code.

1. How does it work?

The basic mechanism is very simple:

1.1. Fetch

First, we have to provide the file name. The name could simply be a hard-coded string in the driver code (several drivers do that), but there are other options. In the example we are going to see, the file name is a property in the devicetree. That kind of breaks the rule that the devicetree describes hardware, but it increases flexibility. Actually, there are other cases where the rule becomes blurry, like with the ubiquitous wakeup-source property. But that is not the topic we are covering here, and I did not make the decision anyway, so let’s move on. To be honest, my first idea was using the devicetree, then hard-coding the name, and in the end someone sent the devicetree solution before I sent anything upstream :grin:

We will do that by means of the following call:

request_firmware(&fw, name, dev);

where name is the string with the file name, and dev the device for which the firmware is being loaded (e.g. a struct device *dev in a driver that represents the device, often embedded into a driver-specific struct).

And what is fw? The object that the API will populate (hence why it is passed by reference):

const struct firmware *fw;

// from include/linux/firmware.h:
struct firmware {
	size_t size;
	const u8 *data;

	/* firmware loader private fields */
	void *priv;
};

As you can see, it is a very simple structure, and we only care about the data itself (const u8 *data) and its size (size_t size).

We don’t have to provide the file location, because the API defines where it should be stored:

fw_path_para - module parameter - default is empty so this is ignored
/lib/firmware/updates/UTS_RELEASE/
/lib/firmware/updates/
/lib/firmware/UTS_RELEASE/
/lib/firmware/

Of course, you should have placed the binary there before, probably when you built your rootfs. Unless you have a good reason to pass the path as a parameter, the /lib/firmware paths are often preferred.

If the file is found in any of those locations (the list is ordered by priority, so the first match will stop the fetch), fw will be populated with the binary data and its size in bytes. We will use that information to implement the update mechanism.

Risky tip: you could also include the firmware in the Linux image (i.e. built-in). You will find how to do it here, but be careful: you don’t want to include proprietary firmware for legal reasons. Moreover, it has some other drawbacks like having to rebuild the kernel for a new firmware, but maybe it fits your needs (e.g. speed or early availability).

1.2. Data transmission

The update mechanism is strongly device-dependent, and the firmware API cannot know all mechanisms for all devices. Instead, the update implementation will be up to you, following the device application notes. You will have to send the firmware to the device over the supported protocol (typically, but not only, I2C and SPI). Don’t worry, the update mechanisms are usually very simple, and no rocket science is required. Sometimes a single command to trigger a burst transmission or a simple loop to send multiple packets will be enough.

1.3. Cleanup

Once we are done with the firmware update, we will not need fw anymore, so we can release the data with a simple call:

release_firmware(fw);

2. Real example: TPS6598x PD controller driver

Let’s put everything into action and develop some functionality for a device driver! This time we are going to play with a TPS65987 PD controller, whose driver I mentioned in my last article where we learned about device-specific data. This device requires the firmware to be loaded every time it gets powered up, and we will integrate the upload mechanism into the driver’s probe function.

By the way, while I was writing that previous article I found that the data retrieval from this driver was suboptimal, and I sent this simple patch upstream to simplify it. Writing articles is fun, and sometimes even useful :grin: Actually I have contributed several times to this driver (adding the reset signal or fixing a couple of bugs like this one, ported to stable kernels too), and some more patches are getting ready for submission, but the relevant contribution that matters here is the implementation of the firmware transmission for this device.

We already know that the update mechanism is device specific, and we also know (from the previous article, check it out!) how to provide the driver with device-specific data and functions. Let’s use our “deep knowledge” to fill the same steps we saw in the previous section, this time with real kernel code.

2.1. Fetch

This part is common to any device the driver might support. The only difference might be the file name, which (as I mentioned before) is defined in the devicetree. Therefore, we can have a single wrapper to populate our firmware structure and make the data available for its transmission to the device. This wrapper only reads the file name, and calls request_firmware() like we did before:

/*
 * We don't care much about the tps object, it's the top structure that contains
 * struct device *dev.
 */
static int tps_request_firmware(struct tps6598x *tps, const struct firmware **fw)
{
	const char *firmware_name;
	int ret;

	// Function to read the firmware name from the devicetree
	ret = device_property_read_string(tps->dev, "firmware-name",
					  &firmware_name);
	if (ret)
		return ret;

	ret = request_firmware(fw, firmware_name, tps->dev);
	if (ret) {
		dev_err(tps->dev, "failed to retrieve \"%s\"\n", firmware_name);
		return ret;
	}

	// To make sure that the file is not empty
	if ((*fw)->size == 0) {
		release_firmware(*fw);
		ret = -EINVAL;
	}

	return ret;
}

At this point, and if no errors occurred (file name not found in devicetree, file not found or empty), we should have fw ready for the next step.

2.2 Data transmission

This is the device-specific part. The tps6598x driver currently supports firmware updates for two different devices: the tps25750 and the tps6598x. Although both communicate via I2C, their update mechanisms are completely different. Therefore, the cleanest solution is adding per-device update functions that work with the fw object to meet their unique requirements (again, check out my article about extending drivers to support new devices). Here I am going to show you the (relevant part of the) implementation for the tps6598x, which is the one I know better. You can then take a look at the implementation for the tps25750, which might be relevant for you if your device supports bursts (e.g. sending the whole firmware at once).

The tps6598x requires a “start” command to enter in patch mode, and a “complete” command to enter normal operation after the update. That is irrelevant for the data transmission, so I will simplify the real function a little bit. We will focus on what really matter for this article.

Data transfers are carried out by means of the “download” command (“PTCd”), which accepts up to 64 bytes of data. The firmware size for this device is usually around 16 KB, which means that several “download” commands will have to be sent. A bit tedious, but the device does not offer any other option apart from connecting an SPI flash memory. Alright, let’s program a loop to send the data:

static int tps6598x_apply_patch(struct tps6598x *tps)
{
	// variable declaration

	// the wrapper from the previous step
	ret = tps_request_firmware(tps, &fw, &firmware_name);
	if (ret)
		return ret;

	// "start" command: if it is ok, update can start

	/* ------- RELEVANT STUFF (simplified) ------- */
	bytes_left = fw->size;
	while (bytes_left) {
		in_len = min(bytes_left, TPS_MAX_LEN);
		ret = exec_cmd(tps, "PTCd", in_len, fw->data + copied_bytes);
		if (ret) {
			dev_err(tps->dev, "Patch download failed (%d)\n", ret);
			goto release_fw;
		}
		copied_bytes += in_len;
		bytes_left -= in_len;
	}
	/* ------------------------------------------ */

	// "complete" command: if it is ok, update succeeded

	dev_info(tps->dev, "Firmware update succeeded\n");

	// firmware release, next step ;)

	return ret;
};

The algorithm is very simple: send fw->size bytes in packets of 64 bytes from fw->data until there are no bytes left. If the last packet is smaller than 64 bytes, send whatever is left. If there is nothing left to send, happy days!

2.3. Cleanup

No tricks or surprises, just release_firmware(fw) and a label to reach the release in the error paths (the goto from the previous step):

release_fw:
	release_firmware(fw);

Done!

3. Other things to consider

If you have a candidate to profit from the kernel firmware API, read the device datasheet and application notes very carefully before you even start thinking about the implementation. Not only because the transfer mechanism might be very complex (that would be more fun!), but also because you might find several hardware-related challenges that could require a new hardware design. Typical examples are boot sequences and bootstrap pins to access the update mode (the tps6598x has such pins).

Another issue that you should keep in mind is timing. If the device driver tries to access a file from a filesystem that is still not available, it will fail. Why would that even happen? If your driver starts running before the filesystem with the firmware gets mounted, you will probably experience that. In such cases, you could consider either including the firmware file in your initramfs, or building the device driver as a module to store it in the same filesystem. Another option would be a fallback mechanism, which would provide a trigger via sysfs. There is even an additional API called Firmware Upload API to increase control over firmware updates via sysfs and monitor its status. I never used this API myself, but apparently you could use it to automate firmware updates from userspace. As I said, the API is much broader than what I can cover in a single article.

But not all points to consider are so negative/dangerous! The API is very well documented, and there are other functions that might fit your needs better than the simple request_firmware() we saw in this article (there are a bunch of alternatives to request firmware for different use cases, all documented here), so you will probably find an accurate solution for your needs (maybe an asynchronous version?). Furthermore, you will find a lot of examples in different kernel subsystems (like bluetooth or net/wireless) to “get inspired” from.

The last thing I would like to mention should be obvious, but still: if you invest time to extend a driver in order to get firmware update support, share your work with the community. Apart from the typical reasons why you should do it (improvements, fixes, updates), relying on an out-of-tree solution can lead to surprises if you update the kernel, and the driver gets an update mechanism that does not comply with yours (like the hard-coded filename against the devicetree property). I have written a complete series about how to become a Linux kernel contributor, no excuses! Be wise and collaborate with the community :wink:

Linux Kernel Development - Extending Drivers for new Devices

Thu, 25 Apr 2024 10:00:00 +0000

Although writing a new device driver is fun, it is often time-consuming (programming, testing, maintaining), and sometimes a source of trouble (damn bugs!). Therefore, and whenever possible, recycling as much upstream code as possible is a good idea, and it is indeed common practice.

But what if the device is so similar to a supported one that you find yourself doing nothing but renaming functions and variables, modifying a couple of values, and feeling that you just plagiarized an existing driver? In that case, there is no need to re-invent the wheel, and there are easy ways to add support for new devices in existing drivers. Let’s see how it works with real examples.

Content:

1. The trivial case: identical hardware
2. Not exactly the same, but pretty similar
3. One driver to rule them all (and in the darkness bind them)

1. The trivial case: identical hardware

Sometimes you will find devices that are produced by different manufacturers with no modifications beyond the device name. Other devices are just replacements for discontinued ones, being 100% compatible from a software point of view. In such cases, you could even use an existing driver that supports the same device with a different name. But doing things right in this case is straightforward: usually adding a new compatible string is enough.

Let’s see an example, where I added support for The Maxim MAX30101, which is a replacement for the already supported –but no longer recommended for new designs– MAX30105. The “support” consists of literally two new lines in the driver, which is actually the new compatible in two structures: one for i2c_device_id (I2C IDs, where we also indicate that it is treated as a max30105), and one for of_device_id (the strings used in a device tree). Don’t panic just yet, more about these structures in the next section!

The whole patch to support a new device (but don't forget to update bindings!)

2. Not exactly the same, but pretty similar

As you can imagine, hardware designers are also happy when they can recycle existing blocks to produce a new device with minimal effort. Why would you design a 3-channel UV light sensor from scratch, when you have already designed a 3-channel RGB color sensor? Instead, you can change the photodiodes and keep the rest: I2C interface, registers, conversion block, etc. That is not only faster, but also safer: the previous device has already been tested by the customers for months/years.

Faster and safer sounds good, and we also want to recycle stuff. If we have a driver for that RGB sensor, we have at least 90% of the driver for the UV sensor. We just need to add the gains for the new photodiodes, and the rest should just work as it did before. I did not choose a random example, so let’s see this in action with two real devices that the Linux kernel supports: the AMS AS73211 XYZ color sensor, which has been supported since 2020, and the AMS AS7331 UV light sensor, which I recently added to the original driver.

In a case like this, we will need a new compatible again, but also some device-specific code (e.g. the new gains aka scales). The infrastructure to provide that code is already there, and it makes use of good old pointers for it. I have already mentioned structures used to provide compatibles like of_device_id and i2c_device_id, which include a field to pass custom data:

/*
 * Struct used for matching a device
 */
struct of_device_id {
	char	name[32];
	char	type[32];
	char	compatible[128];
	const void *data;
};

We are going to use that const void *data pointer to tell the driver what it requires for a given compatible like this:

static const struct of_device_id as73211_of_match[] = {
	{ .compatible = "ams,as73211", &as73211_spec },
	{ .compatible = "ams,as7331", &as7331_spec },
	{ }
};
MODULE_DEVICE_TABLE(of, as73211_of_match);

as73211_spec and as7331_spec are the structures where the device-specific data is stored, and basically the only non-boilerplate code to support the new device:

/**
 * struct as73211_spec_dev_data - device-specific data
 * @intensity_scale:  Function to retrieve intensity scale values.
 * @channels:         Device channels.
 * @num_channels:     Number of channels of the device.
 */
struct as73211_spec_dev_data {
	int (*intensity_scale)(struct as73211_data *data, int chan, int *val, int *val2);
	struct iio_chan_spec const *channels;
	int num_channels;
};

static const struct as73211_spec_dev_data as73211_spec = {
	.intensity_scale = as73211_intensity_scale,
	.channels = as73211_channels,
	.num_channels = ARRAY_SIZE(as73211_channels),
};

static const struct as73211_spec_dev_data as7331_spec = {
	.intensity_scale = as7331_intensity_scale,
	.channels = as7331_channels,
	.num_channels = ARRAY_SIZE(as7331_channels),
};

As you can see, the structure contains a pointer to a function to provide some device-specific logic. That is really useful if the new device differs a bit more from the original one and needs something more than a couple of new constants to work. In this particular case the device-specific logic is trivial, as it only retrieves the right intensity scale depending on the channel, but there is no limit to the complexity it could contain. If you want to see a more complex case, I can recommend you the tps6598x PD controller, where I added support for a device-specific firmware update. In the end, the mechanism is the same and the complexity is hidden in the device-specific code.

How do we retrieve the data we stored in the structures? Easy! We just have to add a pointer to the driver data for the device-specific configuration, and assign the right value in the probe function by means of device_get_match_data() and the compatible being matched:

struct as73211_data {
	struct i2c_client *client;
	// ...
	const struct as73211_spec_dev_data *spec_dev;
};

// ...

static int as73211_probe(struct i2c_client *client)
{
	// ...
	data->spec_dev = i2c_get_match_data(client);
	if (!data->spec_dev)
		return -EINVAL;
	// ...
}

Hey, why did you use i2c_get_match_data() instead of device_get_match_data()? Are you tricking me again? A little trick does not hurt :wink: Actually, the first thing i2c_get_match_data() does is calling device_get_match_data(), and if it fails, it tries to match the I2C IDs contained in the i2c_device_id table. This is an I2C device that implements such an ID table, where the same approach with pointers is used to provide the device-specific data:

static const struct i2c_device_id as73211_id[] = {
	{ "as73211", (kernel_ulong_t)&as73211_spec },
	{ "as7331", (kernel_ulong_t)&as7331_spec },
	{ }
};
MODULE_DEVICE_TABLE(i2c, as73211_id);

This time explicit casting was used because the data pointer type is not void (actually it is an unsigned long integer, and not a pointer, which looks kind of hacky… I know). Would you have used explicit casting otherwise, e.g. (void *)? You would not be the first one, and I almost did when copying code, but the reviewer noticed in time. In order to avoid spreading that unnecessary explicit casting, I removed it from the kernel with this dull series.

A meticulous reader might have found a few more tricks in the code snippets: why does of_device_id from the trivial case (max30102) not have device data (i.e. the data pointer is not assigned), and why is the explicit cast missing in i2c_device_id? In that case, a simple enum (promoted to an integer type without an explicit cast) is enough to handle device-specific code, and there is no need for more complex solutions. The enum value can be stored in i2c_device_id and then retrieved with i2c_client_get_device_id(). Nevertheless, if the driver gets extended to support more devices with bigger differences, then the approach we have seen in this section would be a much cleaner solution than a switch(ID) or if (ID == XX) in the probe function.

Once support for the new device has been added, you should test it to make sure that everything works. Provided there were no bugs in the original code, the only thing that could be wrong is the device-specific code, which is a tiny fraction of the driver. Testing and debugging is always time-consuming, and having a reliable basis for most of the functionality for free is more than welcome. Sometimes you can even test (at least partially) the original device the driver supports by faking the compatible in the device tree. In case of an I2C device, you can also use the i2c-stub module to simulate devices, as I explained in my article about I2C on Linux.

3. One driver to rule them all (and in the darkness bind them)

You might be tempted to take this approach to the extreme and program THE DRIVER: a single file with a million lines and code for any device you want to use. There are many reasons not to do that, but the most obvious is the one that matters for this article: maintainability.

The technique we just saw is widely used in the kernel, but (ideally) only where it makes sense. If you find yourself recycling a driver to the point where the only thing the devices have in common is the driver’s name and maybe the communication protocol, you are obviously doing something wrong. Simple drivers are easier to understand, update and fix. If you are making a simple driver more complex, and the common code is minimal, you are only making things worse. Because there is little common code, you are not reusing much, and therefore not profiting from existing reliable code. What’s the point, then? In that case, writing a new driver might be wiser. But that is beyond the scope of this article. Stay tuned!

Trivia: How do you know if Linux supports a given device? The fastest and easiest option is to look for the part number with git grep.

What if you are looking for almost identical devices, or the same one with a different name? Easy, use some common sense: look for similar devices from the same manufacturer and check if they are supported, or simply take a look at bindings or drivers in the subsystem the device would belong to, which sometimes are grouped by functionality. It will take you a couple of minutes, which is way shorter and less frustrating than writing a new driver for a device Linux already supports.

Even if you ignore my advice and start writing a new driver right away, you will probably find out while copying code from a driver that does exactly what you need, for a device that seems to do the same as yours. Yeah, you stubborn mule!

Tip 1: look for the first digits of the part number to account for variants and new generations like AD740[0123…].

Tip 2: knowing a bit about the device you want to use always helps to identify a driver that already supports it partially or even completely. The starting point should always be the datasheet.

Tip 3: even if you are sure that two devices are identical, asking the manufacturer is not a bad idea. For example, I did so to ensure that the max30101 was a 1:1 replacement for the max30105 from the point of view of a driver because the pinout was slightly different, and the device description too. I received a reply within a couple of days (manufacturers usually reply promptly, either via email or in their forums) that confirmed my assumption.

Linux Kernel Development - Smatch

Tue, 09 Apr 2024 10:00:00 +0000

If you are one of those who think that static analysis is for the weak, you are also one of those who commit silly mistakes that could have been easily avoided! Give up your arrogance for once and let Smatch help you a little bit :wink:

Content:

1. Yet another cpp checker?
2. Setup
3. How to use and short demo

1. Yet another cpp checker?

If you, on the other hand, use code checkers to catch obvious bugs, you might be wondering why you would need yet another tool to complain about that unused variable. There are some well-known tools out there like clangd, cppcheck, etc., that already do that.

If that was the case, I would have not written this article in the first place. You are right, you don’t need a hundred tools telling you exactly the same, most of the times false positives (cppcheck is the master of the false positives: try to check the whole kern,el, and you will see). But that is not the case, or at least not 100%. Smatch is mainly focused on the Linux kernel (it can be used for other projects, though), and it is able to carry out deeper and kernel-tailored analysis that will potentially unveil bugs that other general-purpose tools might overlook. I also like that it is based on another reliable tool like sparse, and it is actively maintained by Dan Carpenter, who also reports and fixes a lot of bugs in the Linux kernel.

How do I know that smatch catches more bugs? Because I use several static analysis tools in my workflow, and smatch is often the winner. Why do I still use the others? Because they give me immediate feedback while coding, and that is a big time-saver. And who knows if they will ever find something that smatch overlooked… If I can profit from all of them, why should I refuse and wait until someone reaches me to tell me that their tools found something I missed? As you will see during my demo, I know what I am talking about.

2. Setup

The setup I am going to show you is documented in the project, so you can clone the repository and follow the steps from Documentation/smatch.rst, which I converted to from the original .txt version recently after fixing some trivial inaccuracies. A minor contribution, but happy to help anyway!

You can find the repository of the project in several locations, and GitHub is one of them. Let’s clone it first:

git clone https://github.com/error27/smatch.git

If you are too lazy to look for the documentation, just install the dependencies:

# For Debian & co:
apt-get install gcc make sqlite3 libsqlite3-dev libdbd-sqlite3-perl libssl-dev libtry-tiny-perl
# For Fedora & co:
yum install gcc make sqlite3 sqlite-devel sqlite perl-DBD-SQLite openssl-devel perl-Try-Tiny

and build the project with the following complex commands:

cd smatch && make

You don’t have to install anything else as you will run smatch from its directory. Nonetheless, there is another step that is highly recommended: building a cross function database. That will help smatch find more bugs, so let’s do it:

cd ~/path/to/kernel_dir
~/path/to/smatch_dir/smatch_scripts/build_kernel_data.sh

That will take a while depending on your machine and your kernel config, and it should be rebuilt from time to time to keep accuracy, but it is definitely worth it.

3. How to use and short demo

First, a copy+paste from the documentation I mentioned before. You can run smatch for the whole kernel, a specific directory, or a single file:

# Run these commands from the kernel root directory.
# whole kernel:
~/path/to/smatch_dir/smatch_scripts/test_kernel.sh
# directory:
~/path/to/smatch_dir/smatch_scripts/kchecker drivers/whatever/
# file:
~/path/to/smatch_dir/smatch_scripts/kchecker drivers/whatever/file.c

And now let’s take a look at a very simple, yet real example. I am going to tell you a true story about how I started to use smatch.

I have mentioned my kernel driver for the Amphenol Chipcap 2 humidity sensor that was applied to the hwmon subsystem in previous articles. A few days after it got applied, and fortunately before it reached the mainline kernel, I got an email from Dan telling me that he (i.e. smatch) had found an issue in my code.

The first thing I thought when I saw the issue was: how did I miss this silly mistake? Ok, I am a human… but clangd and cppcheck?? Maybe I have some misconfiguration… But then I saw that my tools were catching slightly simpler bugs, as expected. The next step was obviously getting smatch to check if it could find the issue, but it didn’t. Why? Because I had not built the cross function database yet. After building it, smatch found the issue as expected, so I fixed it and made sure that I did not miss any other similar bugs in the rest of the code.

The bug smatch found was a trivial ‘uninitialized variable’, which is obvious when some(one/ tool) tells you, but otherwise easy to overlook. The following code snippet shows two similar functions: the first one requests a threaded interrupt if irq_ready is not 0, and the second one does more or less the same for irq_low and irq_high.

clangd is only unhappy about the first ret

Thanks to the error message, I could fix the first one on the fly and keep on programming, which is always nice. On the other hand, the second ret seems ok for clangd. The logic is only slightly more complex (a second if clause), but in this case, that is enough to fool clang.

Let’s see what cppcheck has to say about the same file. I will use the –enable=all option to catch more stuff:

$ cppcheck --enable=all chipcap2.c
Checking chipcap2.c ...
chipcap2.c:143:6: style: The scope of the variable 'err' can be reduced. [variableScope]
 int err;
     ^
chipcap2.c:292:16: style: The scope of the variable 'timeout' can be reduced. [variableScope]
 unsigned long timeout;
               ^
chipcap2.c:371:16: style: The scope of the variable 'timeout' can be reduced. [variableScope]
 unsigned long timeout;
               ^
chipcap2.c:668:9: warning: Uninitialized variable: ret [uninitvar]
 return ret;
        ^
chipcap2.c:659:22: note: Assuming condition is false
 if (data->irq_ready > 0) {
                     ^
chipcap2.c:668:9: note: Uninitialized variable: ret
 return ret;
        ^

Basically the same: only the first uninitialized variable (more verbose), plus variable scope checks that in the kernel are usually false positives, because the common practice is declaring variables a the beginning of the function. Alright, let’s call smatch to save the day:

$ ~/repos/smatch/smatch_scripts/kchecker drivers/hwmon/chipcap2.c
  CHECK   scripts/mod/empty.c
  CALL    scripts/checksyscalls.sh
  DESCEND objtool
  INSTALL libsubcmd_headers
  CC      drivers/hwmon/chipcap2.o
  CHECK   drivers/hwmon/chipcap2.c
drivers/hwmon/chipcap2.c:668 cc2_request_ready_irq() error: uninitialized symbol 'ret'.
drivers/hwmon/chipcap2.c:701 cc2_request_alarm_irqs() error: uninitialized symbol 'ret'.

Bingo! Smatch reported the lines where the two uninitialized ret variables are used (in this case, returned). Not only that, it did not report useless information.

And that’s basically it for a start. If you want to learn more about smatch (e.g. how to add new checks), I could never tell you as much as the project maintainer. Check out his blog, which is also a good place to learn about static analysis in general.

Oddly enough, I did not know that Dan would join us for a mentoring session at the LKMP to talk about smatch, and that session took place a few minutes before I published this article :grin: If you would like to attend such mentoring sessions and learn from the most experienced kernel developers, click on this link and learn how to join the Linux Kernel Mentoring Program.

One disclaimer before I say goodbye, because I can see it coming: I would not be particularly surprised if it was possible to configure clangd and cppcheck to increase their coverage, but it is a fact that they didn’t catch that bug out of the box, and smatch did. But if you know how to configure them to even surpass smatch (for Linux kernel code analysis), please let me know because I am always looking for new improvements to my workflow. Feedback is always welcome, and not only from tools :smiley: