First Kernel Modules
Learning the Basic of Kernel Modules
I have wanted to write a kernel module for some time in hopes of expanding my understanding of computer systems. In addition to my full time job and my evening classes, I am still working on studying the CMU course CS 213 and its assignments. However, I wanted to take a break from that and do something a bit more hands on.
Because I am limited in time I cannot dedicate myself to working through a full book on either Linux or FreeBSD driver development, I need something a bit simpler to play with. One of my favourite channels on youtube is Johannes 4GNU Linux decided to recently restart their tutorial series Let’s code a Linux Driver for the Raspberry Pi. This is the perfect series to work through in the evenings when I need a break from work, classes, and the general responsibilities of life 1. However, I do not simply want to follow along with Johannes’s tutorials, blindly typing into the .c files whatever he types. Instead I will pursue a practice of reinventing the wheel so to speak. For every driver tutorial I will not only do it in Linux, but will also recreate it in FreeBSD!
While it may seem silly to double up the work on a less popular *nix system, I hope that by recreating basic Linux drivers on FreeBSD I will learn more about both systems. I found this to be the case when I recently tried to get a Linux printer binary working on FreeBSD.
I will be working on a Raspberry Pi 4 Rev 1.4 with 2 gigabytes of RAM. The Pi 4 has an ARM Cortex-A72 64 bit quad core processor running at a default clock speed of 1.5Ghz. Because I do not have any additional cooling systems for the Pi, I do not plan on overclocking it or stressing it too hard. However, the CPU did get a lot hotter than expected during today’s setup and my solution was to open up the window to the the cold Canadian winter. It worked to cool down the Pi, but also my entire room and myself. Getting a cooling device for the Pi is something I will be something to look at in the future. Additionally, as the Pi’s has a different boot process than my x86 laptop, I am excited to learn how the boot processes differ between the two systems. Hopefully I can use what I learn about the boot process for some of my future shenanigans2.
Getting Setup
For this I will need either an SD card or a USB stick that contains an image of both Raspian OS 64-Bit, the Raspberry Pi fork of Debian, specifically the Lite version without the desktop environment, and of FreeBSD 14.2 for AArch64. These are both easy enough to get from their respective websites, or by running the respective commands.
#raspberry pi os
$ wget https://downloads.raspberrypi.com/raspios_lite_arm64/images/raspios_lite_arm64-2024-11-19/2024-11-19-raspios-bookworm-arm64-lite.img.xz
#FreeBSD 14.2
$ wget https://download.freebsd.org/releases/arm64/aarch64/ISO-IMAGES/14.2/FreeBSD-14.2-RELEASE-arm64-aarch64-RPI.img.xz
Hello World Style Kernel Module for Raspian OS 64-Bit Lite
I started working on getting Raspian OS setup as it is the recommended operating system for the Pi. The first steps were to extract the .xz file, which was done with the -d flag to decompress and get rid of the .xz file I downloaded.
$ xz -d raspios_lite_arm64-2024-11-19/2024-11-19-raspios-bookworm-arm64-lite.img.xz
This gave me a .img file that I could write to an SD card using the DD command. Using an SD to USB adapter, the device to which I would be writing was /dev/da1
$ sudo dd if=raspios_lite_arm64-2024-11-19/2024-11-19-raspios-bookworm-arm64-lite.img of=/dev/da1 bs=1M status=progress
Once the files had finished writting to the SD card I could eject it, pop it into the Raspberry Pi, plug in the Pi’s power cable and start using the device. After getting it setup as a headless device, I started SSH’ing into it.
Following this I updated the repo, upgraded the packages, as well as upgraded the firmware for the pi by running the following commands
sudo apt update && sudo apt upgrade -y
sudo rpi-update
sudo reboot
Next, I needed to install the kernel headers for the Pi
sudo apt install -y raspberrypi-kernel-headers
Following Johannes’s first tutorial, I made a new directory for a test module with a C file and a Makefile inside of it.
$ mkdir 01_hello
$ touch 01_hello/hello.c 01_hello/Makefile
The file hello.c had the following code
1#include <linux/module.h>
2#include <linux/init.h>
3
4static int __init my_init(void)
5{
6 printk("Hello and get some popcorn! You're in the kernel!\n");
7 return 0;
8}
9
10static void __exit my_exit(void){
11 printk("Fair well from the kernel!\n");
12}
13
14module_init(my_init);
15module_exit(my_exit);
16
17MODULE_LICENSE("GPL");
This code, more or less ripped off directly from Johannes does the following things. The first is the init function my_init(void), which prints a simple message to kernel, particular to dmesg, and returns 0 upon completion. This function is called by the module_init function upon loading the module into the kernel. Similarly there is the exit function my_exit(void) which also prints a message to the kernel. This function is called by module_exit() when the module leaves the kernel. The last little bit, MODULE_LICENSE("GPL"); is a macro that states the module is made under the GPL license. Not only that, within kernel modules it is possible to use EXPORT_SYMBOL_GPL() to mark the system symbols as being only available to modules which support GPL compatible licenses.
The Makefile had the following
1obj-m += hello.o
2
3all:
4 make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules
5clean:
6 make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean
obj-m specifies that the object file created will bie compiled into a loadable kernel object file, a .ko file.
For all: the Makefile will first change the working directory to /lib/modules/$(shell uname -r)/build and run a make command there. M=$(PWD)will tell the build system where the module source code, hello.c, is in the present working directory. Finallymoduleswill tell the kernel module build system to build using the module files specified byobj-m. If this works, inside of the directory I should have a file named hello.ko.
Running make I get the following notice
$ make
make -C /lib/modules/6.6.77-v8+/build M=/home/jholloway/01_hello modules
make[1]: *** /lib/modules/6.6.77-v8+/build: No such file or directory. Stop.
make: *** [Makefile:4: all] Error 2
Entering uname -r in the shell confirms that I am on version 6.6.77-v8+ of the kernel
$ uname -r
6.6.77-v8+
Looking inside of the director /lib/modules/ I can see that I have headers for a number of different kernel versions. Inside of the directories for the other versions I could see a build directory, but not for /lib/modules/6.6.74+rpt-rpi-v8/, my version of the kernel.
$ ls /lib/modules/
6.1.0-31-arm64 6.6.51+rpt-rpi-2712 6.6.74+rpt-rpi-2712 6.6.77-v8+
6.1.21-v8+ 6.6.51+rpt-rpi-v8 6.6.74+rpt-rpi-v8 6.6.77-v8-16k+
$ ls /lib/modules/6.6.74+rpt-rpi-v8/
build modules.builtin modules.dep modules.softdep
kernel modules.builtin.alias.bin modules.dep.bin modules.symbols
modules.alias modules.builtin.bin modules.devname modules.symbols.bin
modules.alias.bin modules.builtin.modinfo modules.order source
jholloway@raspberrypi:~/01_hello $ ls /lib/modules/6.6.77-v8+/
kernel modules.builtin.alias.bin modules.dep.bin modules.symbols
modules.alias modules.builtin.bin modules.devname modules.symbols.bin
modules.alias.bin modules.builtin.modinfo modules.order
modules.builtin modules.dep modules.softdep
The issue was that by running sudo rpi-update I had updated the kernel to a never version for which the apt package manger did not have kernel build directory. Normally you could run sudo apt install linux-headers-$(uname -r) to install the headers, but doing so said that it could not find packages wit this name. Checking manually, I entered apt search linux-headers-6.6.77-v8+ but did not return any results.
I would need to download and compile the headers myself.
Creating the module headers
To create the module headers I would need to clone the latest git repo for the Raspberry Pi OS
jholloway@raspberrypi:~ $ git clone --depth=1 --branch rpi-6.6.y https://github.com/raspberrypi/linux
The RaspberryPi/Linux repository contains the source tree for the latest Raspberry Pi kernel builds. I made sure to clone the branch rpi-6.6.y as it was the branch for my version of the Raspian OS Linux Kernel.
Next I would need to enter the directory and tell make to write the configuration scripts for the correct architecture, aarm64, and the default configuration for the BMC2721 chipset that is used on the Raspberry Pi.
jholloway@raspberrypi:~ $ cd linux/
jholloway@raspberrypi:~/linux $ make ARCH=arm64 bcm2711_defconfig
HOSTCC scripts/basic/fixdep
HOSTCC scripts/kconfig/conf.o
HOSTCC scripts/kconfig/confdata.o
HOSTCC scripts/kconfig/expr.o
LEX scripts/kconfig/lexer.lex.c
YACC scripts/kconfig/parser.tab.[ch]
HOSTCC scripts/kconfig/lexer.lex.o
HOSTCC scripts/kconfig/menu.o
HOSTCC scripts/kconfig/parser.tab.o
HOSTCC scripts/kconfig/preprocess.o
HOSTCC scripts/kconfig/symbol.o
HOSTCC scripts/kconfig/util.o
HOSTLD scripts/kconfig/conf
After this I needed to tell make to prepare the build environment for building modules without fully compiling the kernel.
jholloway@raspberrypi:~/linux $ make modules_prepare
UPD include/generated/timeconst.h
CC kernel/bounds.s
UPD include/generated/bounds.h
CC arch/arm64/kernel/asm-offsets.s
UPD include/generated/asm-offsets.h
CALL scripts/checksyscalls.sh
CHKSHA1 include/linux/atomic/atomic-arch-fallback.h
CHKSHA1 include/linux/atomic/atomic-instrumented.h
CHKSHA1 include/linux/atomic/atomic-long.h
LDS arch/arm64/kernel/vdso/vdso.lds
CC arch/arm64/kernel/vdso/vgettimeofday.o
AS arch/arm64/kernel/vdso/note.o
AS arch/arm64/kernel/vdso/sigreturn.o
LD arch/arm64/kernel/vdso/vdso.so.dbg
VDSOSYM include/generated/vdso-offsets.h
OBJCOPY arch/arm64/kernel/vdso/vdso.so
LDS scripts/module.lds
make modules_prepare gets the source tree ready for building external modules, essential build files and header files. Unfortunately, this alone was not enough. When I tried to make the kernel module (with a Makefile directing to ~/linux/ to test the recently compile modules I had made) I received the following error message:
$ make
make -C /lib/modules/6.6.77-v8+/build M=/home/jholloway/01_hello modules
make[1]: Entering directory '/home/jholloway/linux'
CC [M] /home/jholloway/01_hello/hello.o
MODPOST /home/jholloway/01_hello/Module.symvers
WARNING: Module.symvers is missing.
Modules may not have dependencies or modversions.
You may get many unresolved symbol errors.
You can set KBUILD_MODPOST_WARN=1 to turn errors into warning
if you want to proceed at your own risk.
ERROR: modpost: "_printk" [/home/jholloway/01_hello/hello.ko] undefined!
ERROR: modpost: "module_layout" [/home/jholloway/01_hello/hello.ko] undefined!
make[3]: *** [scripts/Makefile.modpost:145: /home/jholloway/01_hello/Module.symvers] Error 1
make[2]: *** [/home/jholloway/linux/Makefile:1873: modpost] Error 2
make[1]: *** [Makefile:234: __sub-make] Error 2
make[1]: Leaving directory '/home/jholloway/linux'
make: *** [Makefile:4: all] Error 2
Running make modules_prepare did not create the file Module.symvers which is required for building kernel modules. The file Module.symver is a text file generated after building the full kernel or kernel modules, that contains a listing of all of the exported symbols. It is a plain text file that consists of four columns:
| CRC Value | Symbol Name | Module | Export Type |
|---|
jholloway@raspberrypi:~/linux $ cat Module.symvers | head
0xf7370f56 system_state vmlinux EXPORT_SYMBOL
0xbea5ff1e static_key_initialized vmlinux EXPORT_SYMBOL_GPL
0xc2e587d1 reset_devices vmlinux EXPORT_SYMBOL
0xba497f13 loops_per_jiffy vmlinux EXPORT_SYMBOL
0x252fa4d1 init_uts_ns vmlinux EXPORT_SYMBOL_GPL
0x43f92edd wait_for_initramfs vmlinux EXPORT_SYMBOL_GPL
0x4cb23996 init_task vmlinux EXPORT_SYMBOL
0x8946ea72 fpsimd_context_busy vmlinux EXPORT_SYMBOL
0x8fd180e7 kernel_neon_begin vmlinux EXPORT_SYMBOL_GPL
0xa8a8110c kernel_neon_end vmlinux EXPORT_SYMBOL_GPL
The first column, the CRC Value is the Control and Compatability value for the symbol. It is a checksum value that is used by the kernel to determine if the module is compatabile. It does so by checking a similar value in the module itself. If the values match, the kernel will load the module. If not, the kernel will refuse to load it.
The second column is the symbol name for the built-in modules created during the kernel build process. It serves as an identifier for functions, variables or datastructures that are exported by the kernel or kernel module for other modules to use.
The third column, module will specify if the symbol is from the kernel itself, identified by vmlinux, or if it comes from a internal module. If the latter it will state the name of the .ko file which contains the symbol.
Lastly is the export type will specify whether or not the symbol can be exported to other modules. If the symbol is set to EXPORT_SYMBOL it is allowed to be exported to any other module loaded into the kernel. If the symbol is set to EXPORT_SYMBOL_GPL it is exported only to modules that have been licensed under a compatable open source license (as in the case of our simple hello.c kernel file).
The nature of Module.symvers is very interesting and developing a better understanding of kernel modules, symbols and contrasting how they are managed on Linux vs FreeBSD is something I would love to study more in the future. However, I first needed to generate a new Module.symvers file to get the basic module working.
jholloway@raspberrypi:~/linux$ make -j16 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- modules
CALL scripts/checksyscalls.sh
CC certs/system_keyring.o
CC ipc/compat.o
CC mm/filemap.o
...[lines omitted for brevity]...
This took a long time for the Pi to complete.
Not only was it time consuming, but the Raspberry Pi got hot to touch! Checking it’s temperature the processor was around 79°C and steadily rising. Worried it was going to overhead and shut off, I proped the little device on its side near an open window, the cold winter air blowing past it as the temperature in the room noticably dropped. Eventually the Pi’s processor settled around 73°C, but were I to continue working on this and building kernel modules I would need to invest in active cooling.
Compilation took a long time. I stopped paying attention as the clock went on and Id decided I needed to go to the store to get groceries. While I could have used the -j flag to specify more threads for compilation, compiling large binaries on the Pi is slow and not something I want to do again. Can I do it on my FreeBSD laptop?
Finally with Module.sysvers compiled I could now copy the required files to /lib/modules/6.6.74+rpt-rpi-v8/build
jholloway@raspberrypi:~/linux $ sudo mkdir /lib/modules/6.6.74+rpt-rpi-v8/build
jholloway@raspberrypi:~/linux $ sudo cp -r arch/ include/ Makefile Module.symvers scripts/ /tools/ /lib/modules/6.6.74+rpt-rpi-v8/build
Now I wase able to compile the basic hello world style kernel module without any errors and see the .ko file in the directory.
jholloway@raspberrypi:~/linux $ cd ~/01_hello/
jholloway@raspberrypi:~/01_hello $ make
make -C /lib/modules/6.6.77-v8+/build M=/home/jholloway/01_hello modules
make[1]: Entering directory '/usr/lib/modules/6.6.77-v8+/build'
LD [M] /home/jholloway/01_hello/hello.ko
make[1]: Leaving directory '/usr/lib/modules/6.6.77-v8+/build'
jholloway@raspberrypi:~/01_hello $ ls
hello.c hello.ko hello.mod hello.mod.c hello.mod.o hello.o Makefile modules.order Module.symvers
To test if the module works as planned, it will be loaded into the kernel with the command insmod and then removed from the kernel with rmmod. For each of these, it will write the respective my_init() and my_exit() messages to dmesg.
jholloway@raspberrypi:~/01_hello $ sudo insmod hello.ko
jholloway@raspberrypi:~/01_hello $ dmesg | tail -n 1
[38366.380071] Hello and enjoy some popcorn! You're in the kernel!
jholloway@raspberrypi:~/01_hello $ dmesg | tail -n 1
[38428.565139] Fair well from the kernel!
Finnally the module was build and it worked! The next goals were to write a basic hello world style message for FreeBSD and then create a module that performed some basic IO with the Pi’s GPIO pins. But first I wanted to see if I could use a more powerful machine to make the required header files and modules.
Creating the module headers for 6.6.77-v8+ on FreeBSD
I run FreeBSD on my laptop primarily to challenge myself and learn about different systems. I know people use FreeBSD jails and chroots for cross compilation, is it something I can do? The build process on the Pi took a lot of time, if I could do it on a more powerful machine it would be a huge advantage for this project.
First I entered a Linux chroot based upon an Ubuntu userland.
[jholloway@j_holloway ~]$ sudo chroot /compat/ubuntu/ /bin/bash
Once inside, I made sure apt had installed the required packages.
linuxHolloway@j_holloway:~$ sudo apt install build-essential crossbuild-essential-armhf libncurses-dev bison flex libssl-dev gcc-aarch64-linux-gnu bc
As well as used git to download the latest Raspberry Pi source code
linuxHolloway@j_holloway:~$ git clone --depth=1 --branch rpi-6.6.y https://github.com/raspberrypi/linux
Next would be entering the downloaded directory and preparing it for writing modules. Unlike the Pi, I couldn’t just use the same command of make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcm2711_defconfig, instead I would have to specify a compiler for cross compilation.
linuxHolloway@j_holloway:~/linux$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcm2711_defconfig
HOSTCC scripts/basic/fixdep
HOSTCC scripts/kconfig/conf.o
HOSTCC scripts/kconfig/confdata.o
HOSTCC scripts/kconfig/expr.o
LEX scripts/kconfig/lexer.lex.c
YACC scripts/kconfig/parser.tab.[ch]
HOSTCC scripts/kconfig/lexer.lex.o
HOSTCC scripts/kconfig/menu.o
HOSTCC scripts/kconfig/parser.tab.o
HOSTCC scripts/kconfig/preprocess.o
HOSTCC scripts/kconfig/symbol.o
HOSTCC scripts/kconfig/util.o
HOSTLD scripts/kconfig/conf
#
# configuration written to .config
#
linuxHolloway@j_holloway:~/linux$
Once again I would need to do something different than the Pi. I couldn’t just run make modules_prepare. I would have to specify the architecture as well as the cross compiler.
linuxHolloway@j_holloway:~/linux$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- modules_prepare
UPD include/generated/timeconst.h
CC kernel/bounds.s
UPD include/generated/bounds.h
CC arch/arm64/kernel/asm-offsets.s
UPD include/generated/asm-offsets.h
CALL scripts/checksyscalls.sh
CHKSHA1 include/linux/atomic/atomic-arch-fallback.h
CHKSHA1 include/linux/atomic/atomic-instrumented.h
CHKSHA1 include/linux/atomic/atomic-long.h
LDS arch/arm64/kernel/vdso/vdso.lds
CC arch/arm64/kernel/vdso/vgettimeofday.o
AS arch/arm64/kernel/vdso/note.o
AS arch/arm64/kernel/vdso/sigreturn.o
LD arch/arm64/kernel/vdso/vdso.so.dbg
VDSOSYM include/generated/vdso-offsets.h
OBJCOPY arch/arm64/kernel/vdso/vdso.so
LDS scripts/module.lds
The output was very similar to the Pi.
I suspected that I also needed to create Module.symvers in the chroot, just like I had to do for the Pi. I would run the make modules_prepare and make modules commands, but with the additonal flags for specifying cross compilation. I also wanted to throw in the flag -j16 to specify all the cores in my laptop. I want ths to be faster than on the Raspberry Pi.
linuxHolloway@j_holloway:~/linux$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- modules_prepare
CALL scripts/checksyscalls.sh
linuxHolloway@j_holloway:~/linux$ make -j16 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- modules
CALL scripts/checksyscalls.sh
CC certs/system_keyring.o
CC ipc/compat.o
CC mm/filemap.o
CC io_uring/io_uring.o
AS arch/arm64/lib/clear_page.o
CC security/keys/gc.o
...[lines omitted for brevity]...
LD [M] net/vmw_vsock/vmw_vsock_virtio_transport_common.ko
LD [M] net/vmw_vsock/vsock_loopback.ko
LD [M] net/nsh/nsh.ko
linuxHolloway@j_holloway:~/linux$
So, so much faster!!!
Can I write the same Linux kernel module on my FreeBSD system and move it to Linux?
I made the directory 01_hello on my FreeBSD system with the following C file and Make file
1#include <linux/module.h>
2#include <linux/init.h>
3
4static int __init my_init(void)
5{
6 printk("Hello, this Linux kernel module was written on FreeBSD!\n");
7 return 0;
8}
9
10static void __exit my_exit(void){
11 printk("Fair well from the Linux kernel!\n");
12}
13
14module_init(my_init);
15module_exit(my_exit);
16
17MODULE_LICENSE("GPL");
1obj-m += hello_from_FreeBSD.o
2
3all:
4 make -C /home/linuxHolloway/linux/ M=$(PWD) modules
5clean:
6 make -C /home/linuxHolloway/linux/ M=$(PWD) clean
From the command line, I had to once again specify the target architecture and compiler version for cross compilation. Afterwards it successfully produced a kernel module for Linux for AArch64
linuxHolloway@j_holloway:~/01_hello$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-
make -C /home/linuxHolloway/linux/ M=/home/linuxHolloway/01_hello modules
make[1]: Entering directory '/home/linuxHolloway/linux'
CC [M] /home/linuxHolloway/01_hello/hello_from_FreeBSD.o
MODPOST /home/linuxHolloway/01_hello/Module.symvers
CC [M] /home/linuxHolloway/01_hello/hello_from_FreeBSD.mod.o
LD [M] /home/linuxHolloway/01_hello/hello_from_FreeBSD.ko
make[1]: Leaving directory '/home/linuxHolloway/linux'
linuxHolloway@j_holloway:~/01_hello$ ls
Makefile hello_from_FreeBSD.c hello_from_FreeBSD.mod hello_from_FreeBSD.mod.o modules.order
Module.symvers hello_from_FreeBSD.ko hello_from_FreeBSD.mod.c hello_from_FreeBSD.o
linuxHolloway@j_holloway:~/01_hello$ file hello_from_FreeBSD.ko
hello_from_FreeBSD.ko: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=f1d6ca3752ca8bbe1598506ac48ef228666e4356, not stripped
I copied the file over to the Pi and it was recognized by the system. I loaded it with the command
jholloway@raspberrypi:~/01_hello $ sudo insmod hello_from_FreeBSD.ko
jholloway@raspberrypi:~/01_hello $ dmesg | tail -n 1
[24995.880959] Hello, this Linux kernel module was written on FreeBSD!
jholloway@raspberrypi:~/01_hello $ sudo rmmod hello_from_FreeBSD.ko
jholloway@raspberrypi:~/01_hello $ dmesg | tail -n 1
[25180.332642] Fair well from the Linux kernel!
Despite being written on a different operating system with different architecture, it worked as if it was natively written on the Raspian OS!
It is far more convienent to write future Linux kernel modules on my main machine. It is faster, has a GUI desktop, multiple IDE’s to choose from and over all an environment I am familair with. Should I update the Pi to a newer kernel once again, building the headers on here is much faster. Plus you never know if the skills I learned for cross compilation will come in handy for future schemes.
Hello World Style Kernel Module for FreeBSD 14.2 AAarch64
The first step for this half of the project was to simply boot FreeBSD onto the Raspberry Pi. Like for Raspian OS, the first step would be to write the .img file to an SD card.
$ sudo dd if=FreeBSD-14.2-RELEASE-arm64-aarch64-RPI.img of=/dev/da1 bs=1M status=progress
$ sudo sync
Now it’s time to pop the card into the Raspberry Pi and watch it boot up into FreeBSD. After it is done booting, we can ssh into it.
Unfortunately, FreeBSD is lacking when it comes to drivers and Broadcom has not provided a WiFi driver for FreeBSD. My solution is to use a Vonets VAP11G-300 Mini as WiFi Bridge. While not the fastest, it should meet the needs of this simple project.
Trouble First Time Booting
The first time I tried loading FreeBSD onto the Pi I encountered an error where the bootloader could not find the kernel, and would just time out. I tried to recreate the issue by flashing the FreeBSD image to another SD card, but this time the error did not occur; it loaded straight into the operating system. I can only assume it was an issue related to the older firmware on the Pi that I had updated while playing around in Raspian OS.
Should you encounter an error booting into FreeBSD fromt he aarch64 image, it is likely due to either the firmware on the Pi or the loader in the image. Hackacad has a fantastic, step-by-step guide on how to work through this issue by editing the .img file to upload a custom EFI loader. It is what I used the first-time to get FreeBSD setup, although it seems I no longer need to use the process after updating the Pi. That said, exiting the image files to change how the Raspberry Pi is an issue I would like to discuss more in the future. While trying to troubleshoot the issue and understand why the steps on Hackacad’s blog post worked, I spent a lot of time learning about the boot process for the Raspberry Pi. In addition to this, I read up on how bootloaders such as U-boot are used on embedded ARM devices. Understanding its boot process is important when one wants to write custom images for the device, and it will be an topic I will discuss soon.
Writing a Basic Kernel Module
For this simple module, the plan is to essentially recreate what was done for the Pi, but for FreeBSD on the Pi. Unlike for Raspian OS, we do not need to install header files. This is because FreeBSD is a “complete” operating system, where it is built as one large application. By contrast, Linux distrobutions like Raspian OS are a combination of applications, bundled together with the Linux kernel. As a result, sometime the applications may not always be in sync, as we saw with apt not having the linux headers for version 6.6.77-v8+ of the kernel. While we do not need to install the headers on FreeBSD, we will need to ensure that the system source is installed onto the machine in the directory /usr/src. Unfortunately the image we downloaded does not contain the source tree .
jholloway@BSD-PI4:/usr/src $ sudo git clone --branch releng/14.2 https://git.FreeBSD.org/src.git
Password:
Cloning into 'src'...
remote: Enumerating objects: 4616234, done.
remote: Counting objects: 100% (487433/487433), done.
remote: Compressing objects: 100% (108128/108128), done.
remote: Total 4616234 (delta 439766), reused 393241 (delta 377390), pack-reused 4128801 (from 1)
Receiving objects: 100% (4616234/4616234), 1.78 GiB | 5.67 MiB/s, done.
Resolving deltas: 100% (3649296/3649296), done.
Checking objects: 100% (16777216/16777216), done.
Updating files: 100% (99900/99900), done.
jholloway@BSD-PI4:/usr/src $ ls
CONTRIBUTING.md Makefile.sys.inc contrib libexec sys
COPYRIGHT ObsoleteFiles.inc crypto release targets
LOCKS README.md etc rescue tests
MAINTAINERS RELNOTES gnu sbin tools
Makefile UPDATING include secure usr.bin
Makefile.inc1 bin kerberos5 share usr.sbin
Makefile.libcompat cddl lib stand
This took some time to clone with the slow WiFi bridge. Perhaps cross-compilation on the laptop is in order again?
Like on Raspian, we will create a directory with a C file and a Makefile inside of it.
jholloway@BSD-PI4:~ $ mkdir 01_hello
jholloway@BSD-PI4:~ $ cd 01_hello/
jholloway@BSD-PI4:~/01_hello $ touch hello.c Makefile
The C file will be based upon KLD Skeleton from Chapter 9 of the FreeBSD Architecture Handbook
1#include <sys/types.h>
2#include <sys/errno.h>
3#include <sys/param.h>
4#include <sys/module.h>
5#include <sys/kernel.h>
6#include <sys/systm.h>
7
8static int example_handler(struct module *module, int what, void *arg)
9{
10 int error = 0;
11
12 switch(what) {
13 case MOD_LOAD:
14 printf("Module loaded into FreeBSD kernel\n");
15 break;
16 case MOD_UNLOAD:
17 printf("Module removed from the FreeBSD kernel\n");
18 break;
19 default:
20 error = EOPNOTSUPP;
21 break;
22 }
23 return(error);
24}
25
26static moduledata_t mod = {
27 "hello",
28 example_handler,
29 NULL
30};
31
32DECLARE_MODULE(hello, mod, SI_SUB_KLD, SI_ORDER_ANY);
The layout of this file is different to that of the Linux kernel module. Rather than have two sepperate functions for init and exit, the various events are bundled into the event handler example_handler, with they type of event being the int value passed in as the argument what. Inside of this function we have written a switch statement for the type of event. When the module is loaded it will print one line of text and a different line when unloaded. In the event that neither of these two events occur an error value is set and returned to the user. Unlike Linux, which has printk() for writing to kernel space and dmesg, we will be using a printf() function. The printf() function is different that the regular printf found in stdio.h, this one comes from sys/systm.h and writes into kernel space.
Next the eventhandler is added to a moduledate_t datastructure. The struct has three fields: the first is the name of the module (“hello”), the second is the module’s event handler (example_handler), while the third is a pointer to private data. At the moment, we’ve set this value to NULL.
Finnally we use the DECLARE_MODULE macro to bundle everything together and register the event handler with the system. There are four parameters for DECLARE_MODULE. The first is the module name and the second is the moduledatat structure we had previously defined. The third is the _sub argument which specifies the kernel subsystem, in this example it is set to SI_SUB_KLD, specifying that it belongs to the loadable kernel module subsystem. Finally, the fourth parameters is the order of initialization, which in this simple module is set to any.
For the Makefile:
1SRCS=hello.c
2KMOD=hello
3
4.include <bsd.kmod.mk>
Inside of the directory, we can run make and produce the .ko file which we can then load into the kernel
jholloway@BSD-PI4:~/01_hello $ make
Warning: Object directory not changed from original /home/jholloway/01_hello
cc -O2 -pipe -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc -include /home/jholloway/01_hello/opt_global.h -I. -I/usr/src/sys -I/usr/src/sys/contrib/ck/include -fno-common -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fPIC -fdebug-prefix-map=./machine=/usr/src/sys/arm64/include -MD -MF.depend.hello.o -MThello.o -mgeneral-regs-only -ffixed-x18 -mno-outline-atomics -ffreestanding -fwrapv -fstack-protector -Wall -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error=tautological-compare -Wno-error=empty-body -Wno-error=parentheses-equality -Wno-error=unused-function -Wno-error=pointer-sign -Wno-error=shift-negative-value -Wno-address-of-packed-member -Wno-format-zero-length -std=gnu99 -c hello.c -o hello.o
ld -m aarch64elf -warn-common --build-id=sha1 --no-relax -r -o hello.kld hello.o
:> export_syms
awk -f /usr/src/sys/conf/kmod_syms.awk hello.kld export_syms | xargs -J% objcopy % hello.kld
ld -m aarch64elf -Bshareable -znotext -znorelro -warn-common --build-id=sha1 --no-relax -o hello.ko hello.kld
objcopy --strip-debug hello.ko
jholloway@BSD-PI4:~/01_hello $ ls
Makefile hello.c hello.ko machine
export_syms hello.kld hello.o opt_global.h
The kernel module can be loaded and unloaded with the commands kldload and kldunload respectively.
jholloway@BSD-PI4:~/01_hello $ sudo kldload ./hello.ko
jholloway@BSD-PI4:~/01_hello $ dmesg | tail -n 1
Module loaded into FreeBSD kernel
jholloway@BSD-PI4:~/01_hello $ sudo kldunload ./hello.ko
jholloway@BSD-PI4:~/01_hello $ dmesg | tail -n 1
Module removed from the FreeBSD kernel
jholloway@BSD-PI4:~/01_hello $
Cross Compiling From x86 to AArch64
I wasn’t crazy about having to slowly download the src tree onto my Pi with it’s slow WiFi birdge. A solution to this was to cross compile for FreeBSD from the more powerful x86 laptop, with it’s speedy wired internet, over to the Raspberry Pi.
As I am currently running FreeBSD 14.2 on my laptop I don’t need to worry about writing for different versions of the kernel (although that would be a fun exercise), and I alredy have the source tree in /usr/src from the installation.
I rewrote the basic kernel module on my laptop, but with a change in the printf() statements to show that it was compilied on a different machine.
1#include <sys/types.h>
2#include <sys/errno.h>
3#include <sys/param.h>
4#include <sys/module.h>
5#include <sys/kernel.h>
6#include <sys/systm.h>
7
8
9static int loader(struct module *module, int what, void *arg)
10{
11 int error = 0;
12
13 switch(what) {
14 case MOD_LOAD:
15 printf("Module cross compiled from x86 loaded into FreeBSD kernel\n");
16 break;
17 case MOD_UNLOAD:
18 printf("Module removed from the FreeBSD kernel\n");
19 break;
20 default:
21 error = EOPNOTSUPP;
22 break;
23 }
24 return(error);
25}
26
27static moduledata_t mod = {
28 "hello_from_x86",
29 loader,
30 NULL
31};
32
33DECLARE_MODULE(hello_from_x86, mod, SI_SUB_KLD, SI_ORDER_ANY);
The Makefile had to be changed to specify that the compiler was to cross compile for aarch64-unknown-freebsd, set the machine target architecturem, as well as to include the src directory for AArch64, /usr/src/sys/arm64/include
1KMOD=hello_from_x86
2SRCS=hello_from_x86.c
3
4CC=clang --target=aarch64-unknown-freebsd
5
6CFLAGS+= -I/usr/src/sys/arm64/include
7
8MACHINE_ARCH=aarch64
9MACHINE=arm64
10
11.include <bsd.kmod.mk>
We can now run make in our source directory.
[jholloway@j_holloway ~/WorkSpace/FreeBSD_Device_Drivers/01_hello]$ make
machine -> /usr/src/sys/arm64/include
touch opt_global.h
Warning: Object directory not changed from original /home/jholloway/WorkSpace/FreeBSD_Device_Drivers/01_hello
clang --target=aarch64-unknown-freebsd -O2 -pipe -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc -I/usr/src/sys/arm64/include -include /home/jholloway/WorkSpace/FreeBSD_Device_Drivers/01_hello/opt_global.h -I. -I/usr/src/sys -I/usr/src/sys/contrib/ck/include -fno-common -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fPIC -fdebug-prefix-map=./machine=/usr/src/sys/arm64/include -MD -MF.depend.hello_from_x86.o -MThello_from_x86.o -mgeneral-regs-only -ffixed-x18 -mno-outline-atomics -ffreestanding -fwrapv -fstack-protector -Wall -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error=tautological-compare -Wno-error=empty-body -Wno-error=parentheses-equality -Wno-error=unused-function -Wno-error=pointer-sign -Wno-error=shift-negative-value -Wno-address-of-packed-member -Wno-format-zero-length -std=gnu99 -c hello_from_x86.c -o hello_from_x86.o
ld -m aarch64elf -warn-common --build-id=sha1 --no-relax -r -o hello_from_x86.kld hello_from_x86.o
:> export_syms
awk -f /usr/src/sys/conf/kmod_syms.awk hello_from_x86.kld export_syms | xargs -J% objcopy % hello_from_x86.kld
ld -m aarch64elf -Bshareable -znotext -znorelro -warn-common --build-id=sha1 --no-relax -o hello_from_x86.ko hello_from_x86.kld
objcopy --strip-debug hello_from_x86.ko
[jholloway@j_holloway ~/WorkSpace/FreeBSD_Device_Drivers/01_hello]$ file hello_from_x86.ko
hello_from_x86.ko: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=3f49b398e84d78f0cf08bf620686822a2c1dfd03, not stripped
As we can see from the output of the file command, the file hello_from_x86.c is compiled for an AArch64 architecture. When the binary is transferred to the Raspberry Pi, it can be loaded into the kernel as if it were compilied natively.
jholloway@BSD-PI4:~ $ sudo kldload ./hello_from_x86.ko
jholloway@BSD-PI4:~ $ dmesg | tail -n 1
Module cross compiled from x86 loaded into FreeBSD kernel
jholloway@BSD-PI4:~ $ sudo kldunload ./hello_from_x86.ko
jholloway@BSD-PI4:~ $ dmesg | tail -n 1
Module removed from the FreeBSD kernel
Conclusions
Even though it was in some ways easier to write a simple kernel module on FreeBSD as we did not need to build the headers, it was easier to cross-compile for Raspian OS. The challenge for the Raspian OS was to simply build the correct kernel header files for the version of the kernel that had been installed for Raspian OS. Compiling compatabile binaries inside of a Linux chroot was also very straight forward.
I found it more difficult to cross compile the FreeBSD kernel modules from x86 to the Pi. this is for two reasons. The first is that building kernel modules for Linux has far more reasources online, it’s easier to find help when you get stuck and most instructions still work on the FreeBSD Linux chroot. The second is that FreeBSD uses LLVM and Clang for compiling C binaries. The build system was slightly different when cross compiling, and I was often having issues where I would specify the target architecture with MACHINE_ARCH=aarch64 and MACHINE=arm64, yet the binary would either default to compiling to x86 or would encounter a compilation error. It was only throught a lot of trial and error that I was succesful in figuring out the proper rules for the Makefile to successfully cross-compile.
This experience taught me a lot about cross compiling, the challenges working with different build systems, and the differences between writing the most basic of kernel modules for FreeBSD and Linux. Additonally, because I had spent some time struggling to boot FreeBSD on the Pi (only to not have the issue occur when I tried to recreate it for this blog post), I learned a lot about how the Raspberry Pi boot sequence works.
Going forward I want to continue writing more complex kernel modules, following the blueprints laid out by Johannes 4GNU Linux. Additonally, after studying the boot sequence of the Pi and of ARM systems, I hope to succeed in booting AArch64 FreeBSD on my new Radxa Rock 2F systems on a chip. However, that is for the future. At the moment I need to continue studying for my midterms next week…although I’ve already had some fun creating little system calls to turn an LED on and off.
-
Raising all of my cats takes a lot of time and energy. So much kitty litter… ↩︎
-
I have a pair of Radxa Rock 2F systems on a chip coming in the mail. I have no idea why I ordered them, I was just bored. But by God, I will find a use for them, even if it is just learning how to boot FreeBSD on them! ↩︎