Pine fun - Networking

September 3 2021 by Norman Feske

This article walks through the challenge of porting a network driver from the Linux kernel to Genode. It thereby exemplifies Genode's device-driver environment approach for the reuse of unmodified Linux kernel code, touches crucial technicalities of the Linux kernel, and provides practical clues.

After getting acquainted with running a hand-tuned Linux kernel on the Pine-A64-LTS board (Taking Linux out for a Walk) and narrowing our focus to the parts of our target hardware that are relevant for networking (Pruning device trees), it is time to turn our attention to Genode again. The goal is to transplant the network driver from the Linux kernel to a Genode driver component.

For reference, the result of the work described herein can be found at the genode-allwinner repository.

Overview

The activity of porting a Linux driver is too elaborate for one swift step. To get a sense of measurable progress, it is useful and motivating to define intermediate goals that can be wrapped up one by one.

1. Creating a minimal Linux kernel configuration that accommodates barely more than the single driver we are interested in, similar to what we did in the Taking Linux out for a walk. While pursuing this goal, one can solely keep the focus on the Linux kernel configuration.

2. Getting a tangible feeling for the targeted device and the interplay with other devices. Looking at the pruned device tree as described here is a good aid. Look at the cobweb of device nodes and try to make a mental picture out of it. E.g., in the case of the network driver, we have to consider the ethernet PHY, the actual network controller (emac), and spot the dependency from certain clocks and voltages as potential risks. To double-check the findings, it is recommended to test-drive Linux with the pruned device tree to see if it is still able to operate the device.

3. Creating the initial source skeleton of the driver component and successfully compiling and linking Linux code into a first executable ELF binary. During this step, one can focus solely on the build system, symbols, and compilation units. One doesn't have to understand the code in order to link it. This step is described in Sections Directory structure of the driver component and Identifying Linux source codes of interest.

4. Creation of a test scenario (run script) and a convenient work flow to execute and quickly update the binary. At this step, which is covered by Section Executable testbed, we are merely concerned with the relationships between components without looking inside them. For the work flow, it is satisfying to string together a few convenient shell commands to ease ones life.

5. Complete the execution of low-level Linux initcalls until the first device resources are requested. Here, one has to get close to the Linux code but can ignore any hardware-specific concerns.

6. Moving the work flow over to the real hardware leveraging the capabilities of Genode's run tool, and possibly tweaking it using custom plugins. The focus is on the use of shell commands and glueing them together using Tcl script snippets, possibly even automating the powering of the target hardware to make one's life better.

7. Successively completing all Linux initcalls including those of the driver code. One has to iteratively boot the target hardware, look at the log output, compare the log with the Linux kernel log obtained beforehand. Whenever both logs diverge in non-plausible ways, investigate and instrument the code of both native Linux and Genode. In other words, taking a deep dive into the Linux kernel code, adding additional Linux subsystems, curating dummy functions, supplementing custom emulation code, and extending the platform-driver's configuration whenever encountering the driver's request for additional hardware resources. This step is addressed by Sections Linux initcalls and The lx_emul building blocks.

8. Once all initcalls are executed, provoke a small operation of the driver. For example, exercising the link detection of the network driver by plugging/unplugging the cable, comparing the resulting log output with native Linux. For the first time, the driver's actual functionality and its interplay with the physical world comes into focus. During the iterative debugging and learning, the tips and tricks given by Sections Linux caveats and Debugging and development hints may be of help.

9. Get the driver's core functionality in the form of a self-sufficient program to work. In the network driver's case, this would be the determination of the device's MAC address as well as the transmission and reception of network packets. At this stage, the driver remains co-located with the test code in one program. In other words, the test program uses the Linux kernel's internal APIs directly. For the network driver, one can conveniently use Linux' builtin DHCP support as test program as described in below in Section Using Linux' built-in DHCP support as networking test.

10. Adding the Genode service interface to the driver, e.g., by using the building blocks of the genode_c_api. To test this integration, the test scenario must be enhanced by a separate component that interacts with the driver component using a Genode session interface. Knowing that the Linux code and the device is operational, one can focus solely on the Genode integration at this stage. This step is covered in more detail in Section Connecting the driver with a Genode session interface.

11. Once the driver works reliably in a minimalistic setting, it is time to expose it to regular networking scenarios by packaging it into a form that is digestible by the arsenal of existing run scripts. The packaging step is covered in Section Packaging the driver.

12. With non-trivial work loads at hand, one can take a critical look at the driver's behavior, in particular at its performance, and optimize it if desired.

13. Finally, one can wrap up the work by cleaning up the code, potentially consolidating parts shared with other drivers, reviewing the result, and documenting the component.

When broken up into these steps - each with a different focus and a tangible intermediate result - the work can be conducted in manageable pieces and can even passed-on between developers.

Directory structure of the driver component

Being a network driver, it naturally should reside somewhere under src/drivers/nic/. To avoid possible path ambiguities with network drivers hosted in other Genode source repositories, placing the driver inside a uniquely named subdirectory is a good practice. In our case - with the driver being referred to as "EMAC" - we settle on the directory src/drivers/nic/emac/. This directory will host the following files, for which we can initially create skeleton versions based on one of the already existing drivers. E.g., one may take the emac driver as reference when porting a new network driver.

Build magic

When reviewing the target.mk file of the emac driver, it is obvious that there must be more to the build rules than those few dull lines. The magic happens in the lib-import file import-a64_lx_emul.mk. This file is supplemented to the build process because the target.mk specifies a64_lx_emul as a library dependency.

 LIBS = base a64_lx_emul

The content of import-a64_lx_emul.mk is worth studying. It lists several building blocks of the lx_kit and lx_emul libraries that are used across all Linux drivers ported for the A64 SoC, it defines the compiler flags used for building Linux C code, and it obtains the list of C source files from the driver's sources.list file. Furthermore, it evaluates the BOARDS and DTS_EXTRACT variables to generate driver-specific device-tree binary files. Given this mechanism, the target.mk of a driver solely needs to declare the supported BOARDS and the driver-specific selection of device-tree nodes to produce a ready-to-use dtb file, e.g.,

 BOARDS := pine_a64lts
 DTS_EXTRACT(pine_a64lts) := --select emac

Finally, the import file contains a number of tweaks and quirks such as disabling certain warnings for individual compilation units or generating build artifacts that are implicitly generated by the Linux build system (crc32table.h).

Generated Linux headers

The Linux build system generates a number of header files when preparing a build directory. When compiling Linux code outside the Linux build system - as we do - those headers are badly missed. This is where the a64_linux_generated library comes in. This pseudo library has the sole purpose of creating a Linux build directory with the generated headers we need. It takes the same Linux kernel configuration as used for the a64_linux target we discussed earlier in Taking Linux out for a Walk.

Identifying Linux source codes of interest

How to spot the files needed to drive our network device among the many thousands of C files found in the Linux source tree?

Taking the device tree as our guide

In the remainder of this article, we refer an all-encompassing device-tree source (DTS) file flat_pine64lts.dts for our board. This file can be extracted from the Linux kernel source via the C preprocessor as described here.

Given the flat_pine64lts.dts file, let Genode's dts/extract tool (Pruning device trees) guide our attention:

$ .../tool/dts/extract --select emac flat_pine64_lts.dts \
                      | grep "compatible = "
  compatible = "fixed-clock";
  compatible = "fixed-clock";
  compatible = "simple-bus";
   compatible = "allwinner,sun50i-a64-system-control";
    compatible = "mmio-sram";
     compatible = "allwinner,sun50i-a64-sram-c";
    compatible = "mmio-sram";
     compatible = "allwinner,sun50i-a64-sram-c1",
   compatible = "allwinner,sun50i-a64-ccu";
   compatible = "allwinner,sun50i-a64-pinctrl";
   compatible = "allwinner,sun50i-a64-emac";
    compatible = "snps,dwmac-mdio";
   compatible = "arm,gic-400";
   compatible = "allwinner,sun50i-a64-rtc",
   compatible = "allwinner,sun50i-a64-r-intc",
   compatible = "allwinner,sun50i-a64-r-ccu";
   compatible = "allwinner,sun50i-a64-r-pinctrl";
   compatible = "allwinner,sun8i-a23-rsb";
  compatible = "x-powers,axp803";
 compatible = "pine64,sopine-baseboard", "pine64,sopine",
  compatible = "ethernet-phy-ieee802.3-c22";
 compatible = "pine64,pine64-lts", "allwinner,sun50i-r18",

Note that some compatible attributes span multiple lines (the lines ending with a comma). So it's probably best to manually inspect the device-tree source to get the full information.

Recap that we already used the compatible device-node attributes earlier to connect the dots between the device tree and kernel-configuration options. Analogously, we can use those attribute values to look up the associated source codes.

Let's take "snps,dwmac-mdio" as an pattern to grep Linux source tree:

 linux$ grep -rl "snps,dwmac-mdio" drivers
 drivers/net/ethernet/stmicro/stmmac/stmmac_platform.c

By looking at the Makefile where the driver code is located, we can immediately spot the set of driver sources declared by looking at the listed object files. This information is all we need to expand our sources.list file.

 ...
 drivers/net/ethernet/stmicro/stmmac/stmmac_platform.c
 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
 drivers/net/ethernet/stmicro/stmmac/stmmac_ethtool.c
 drivers/net/ethernet/stmicro/stmmac/stmmac_mdio.c
 drivers/net/ethernet/stmicro/stmmac/ring_mode.c
 drivers/net/ethernet/stmicro/stmmac/chain_mode.c
 drivers/net/ethernet/stmicro/stmmac/dwmac_lib.c
 drivers/net/ethernet/stmicro/stmmac/dwmac1000_core.c
 drivers/net/ethernet/stmicro/stmmac/dwmac1000_dma.c
 drivers/net/ethernet/stmicro/stmmac/dwmac100_core.c
 drivers/net/ethernet/stmicro/stmmac/dwmac100_dma.c
 drivers/net/ethernet/stmicro/stmmac/enh_desc.c
 drivers/net/ethernet/stmicro/stmmac/norm_desc.c
 drivers/net/ethernet/stmicro/stmmac/mmc_core.c
 drivers/net/ethernet/stmicro/stmmac/stmmac_hwtstamp.c
 drivers/net/ethernet/stmicro/stmmac/stmmac_ptp.c
 drivers/net/ethernet/stmicro/stmmac/dwmac4_descs.c
 drivers/net/ethernet/stmicro/stmmac/dwmac4_dma.c
 drivers/net/ethernet/stmicro/stmmac/dwmac4_lib.c
 drivers/net/ethernet/stmicro/stmmac/dwmac4_core.c
 drivers/net/ethernet/stmicro/stmmac/dwmac5.c
 drivers/net/ethernet/stmicro/stmmac/hwif.c
 drivers/net/ethernet/stmicro/stmmac/stmmac_tc.c
 drivers/net/ethernet/stmicro/stmmac/dwxgmac2_core.c
 drivers/net/ethernet/stmicro/stmmac/dwxgmac2_dma.c
 drivers/net/ethernet/stmicro/stmmac/dwxgmac2_descs.c

Taking the Linux build directory as our guide

Alternatively to taking the device-tree as the starting point, the build directory of our bare-bones Linux kernel contains instructive information, specifically the object files that went into the kernel.

 build/arm_v8a$ find a64_linux/ -name "*.o"

Remember that we have previously slimmed down the Linux kernel configuration as far as we could, keeping only the bare minimum needed for networking. So the list of object files found in the Linux build directory serves as a reasonably small superset of the compilation units that are of interest to us. Any compilation unit not listed cannot be important.

By combining both perspectives, taking the compatible device-nodes attributes as cues while using the Linux kernel's object files as plausibility check, our mental picture of the driver code and its dependencies becomes more and more clear and our sources.list file grows.

Build test

With the sources.list enriched with the list of Linux source codes we are interested in, let's have the Genode build system chew a bit on our driver:

 build/arm_v8a$ make drivers/emac
  ...
  Program drivers/nic/emac/emac_nic_drv
    COMPILE  arch/arm64/lib/memchr.o
    COMPILE  arch/arm64/lib/memcmp.o
    ...
    COMPILE  drivers/net/ethernet/stmicro/stmmac/stmmac_main.o
    COMPILE  drivers/net/ethernet/stmicro/stmmac/stmmac_platform.o
    COMPILE  drivers/net/ethernet/stmicro/stmmac/stmmac_mdio.o
    COMPILE  drivers/net/ethernet/stmicro/stmmac/stmmac_ptp.o
    COMPILE  drivers/net/ethernet/stmicro/stmmac/stmmac_tc.o
    COMPILE  dummies.o
    COMPILE  main.o
    LINK     emac_nic_drv
 drivers/net/ethernet/stmicro/stmmac/stmmac_main.o: in function `stmmac_cmdline_opt':
                        stmmac_main.c:5379: undefined reference to `strsep'
 ...
 ... many more undefined references
 ...

We see the Linux source code being picked up and compiled! The build system backs out not before the linking stage. During linking, however, the many inter-dependencies of the driver code from the rest of the Linux kernel become visible in the form of "undefined reference" errors.

Tying the loose ends to make the linker happy

In principle, we could inspect and resolve each of those linking errors manually. But given the volume of error messages, this would be tedious. In this situation, Genode's dde_linux/create_dummies tool comes to the rescue.

Let's remember the location of our driver's generated_dummies.c file in an environment variable named DUMMY_FILE, which will later be evaluated by the create_dummies tool:

build/arm_v8a$ export DUMMY_FILE=/path/to/src/drivers/emac/generated_dummies.c

With the DUMMY_FILE defined, we can instruct the tool to fill the file with a dummy implementation for each of the unresolved references reported by the linker:

build/arm_v8a$ echo > $DUMMY_FILE ;\
               ../../tool/dde_linux/create_dummies generate \
                                    LINUX_KERNEL_DIR=a64_linux \
                                    TARGET=drivers/nic/emac ;\
               make drivers/nic/emac

In the command line above, we first wipe any prior content from the file, then invoke the create_dummies tool, followed by another build of the driver with the new version of generated file. The create_dummies tool is described in more detail at a dedicated article.

With the dummies generated, the linking of the executable binary succeeds! This won't be the last time of issuing the command. In fact, each time after adding or removing any content of the sources.list file, it is best to update generated_dummies.c using the command above.

Executable testbed

With a first executable binary built, it is time to give it a first run. At this point, we are not yet concerned about accessing any actual hardware. We merely want to obtain the first life sign of the component and see any hint of Linux initialization code being executed. The following run script named after the driver - in our case a64_emac_drv.run is an appropriate name - can serve as our initial test scenario.

 build { core init timer drivers/platform drivers/nic/emac }

 create_boot_directory

 install_config {
   <config>
     <parent-provides>
       <service name="LOG"/>
       <service name="PD"/>
       <service name="CPU"/>
       <service name="ROM"/>
       <service name="IO_MEM"/>
       <service name="IRQ"/>
     </parent-provides>

     <default caps="100"/>

     <start name="timer">
       <resource name="RAM" quantum="1M"/>
       <route> <any-service> <parent/> </any-service> </route>
       <provides> <service name="Timer"/> </provides>
     </start>

     <start name="platform_drv">
       <resource name="RAM" quantum="1M"/>
       <provides> <service name="Platform"/> </provides>
       <route> <any-service> <parent/> </any-service> </route>
       <config devices_rom="config">
         <policy label="emac_nic_drv -> ">
         </policy>
       </config>
     </start>

     <start name="emac_nic_drv">
       <resource name="RAM" quantum="1M"/>
       <route>
         <service name="ROM"> <parent/> </service>
         <service name="CPU"> <parent/> </service>
         <service name="PD">  <parent/> </service>
         <service name="LOG"> <parent/> </service>
         <service name="Timer">    <child name="timer"/> </service>
         <service name="Platform"> <child name="platform_drv"/> </service>
       </route>
       <config/>
     </start>

   </config>
 }

 build_boot_image { core ld.lib.so init timer platform_drv
                    emac_nic_drv emac-pine_a64lts.dtb }

 run_genode_until forever

The emac_nic_drv is connected to the platform_drv but no device resources are assigned to the corresponding policy yet. In fact, the device hardware remains completely untouched, which allows us to execute the run script for an arbitrary boards, in particular Qemu. The use of Qemu instead of the targeted board at this early stage is convenient.

The dtb file emac-pine_a64lts.dtb as integrated into the boot image is a side-product of building the emac driver. It is created according to the definition of the BOARDS and DTS_EXTRACT variables in the driver's target.mk file.

Linux initcalls

When executing the run script exemplified above, we are greeted with the following log output.

 kernel initialized
 ROM modules:
  ROM: [0000000040120000,0000000040120469) config
  ROM: [0000000040009000,000000004000a000) core_log
  ROM: [0000000040149000,000000004014a094) emac-pine_a64lts.dtb
  ROM: [0000000040207000,000000004023cbe8) emac_nic_drv
  ROM: [000000004023d000,00000000402841f8) init
  ROM: [0000000040159000,0000000040206880) ld.lib.so
  ROM: [0000000040121000,0000000040148d30) platform_drv
  ROM: [0000000040007000,0000000040008000) platform_info
  ROM: [000000004014b000,0000000040158710) timer

 Genode 21.05-10-g51f02a668d9
 2010 MiB RAM and 64533 caps assigned to init
 [init -> emac_nic_drv] Error: Initcall __initcall_initialize_ptr_randomearly unknown in initcall database!
 [init -> emac_nic_drv] Error: Initcall __initcall_init_jiffies_clocksource1 unknown in initcall database!
 [init -> emac_nic_drv] Error: Initcall __initcall_stmmac_init6 unknown in initcall database!
 [init -> emac_nic_drv] Error: Initcall __initcall_sync_state_resume_initcall7 unknown in initcall database!
 [init -> emac_nic_drv] Error: Initcall __initcall_devlink_class_init2 unknown in initcall database!
 [init -> emac_nic_drv] Error: Function kmem_cache_init not implemented yet!
 [init -> emac_nic_drv] Backtrace follows:
 [init -> emac_nic_drv] 0x1024034
 [init -> emac_nic_drv] 0x1016898
 [init -> emac_nic_drv] 0x1022498
 [init -> emac_nic_drv] Will sleep forever...

The messages "Error: Initcall ... unknown in initcall database!" tell us that the Linux code we incorporated into our component features initcalls that are unknown to the lx_emul execution environment. Hence, lx_emul won't know the order, in which those calls must be executed. More information about the initcall mechanism is available in a dedicated article.

For us, it is important to know that the initcall order is supplemented to the lx_emul library via the header file at src/include/lx_emul/initcall_order.h. The content of this file depends on the Linux kernel configuration. Fortunately, we won't need to maintain this header file by hand. Instead, the handy extract_initcall_order tool allows us to generate this file from the System.map of a built Linux kernel:

 build/arm_v8a$ ../../tool/dde_linux/extract_initcall_order extract \
        LINUX_KERNEL_DIR=a64_linux
        HEADER_FILE=/path/to/drivers/src/include/lx_emul/initcall_order.h

Note that the directory specified at LINUX_KERNEL_DIR must contain a built Linux kernel, specifically the System.map file.

When re-running the run script with the updated initcall_order.h, the initcall-related errors should disappear.

The lx_emul building blocks

A high-level overview of the anatomy of a DDE-Linux-based driver is provided in the release documentation of Genode 21.08. The lx_emul library provides three kinds of build blocks.

First, it provides a custom C interface for low-level mechanisms of the runtime. The corresponding functions are prefixed with lx_emul_. The interface is provided at dde_linux/include/lx_emul/.

Second, it provides alternative implementations of low-level Linux subsystems. Those implementations reside at dde_linux/src/lib/lx_emul/shadow/. For example, shadow/mm/slub.c is an alternative to Linux' mm/slub.c that provides the same binary interface but implements it by the means of the lx_emul mechanisms.

And third, it provides a few shadow headers at dde_linux/src/include/lx_emul/shadow/ that strip away or tweak a few unpleasant parts of the Linux-internal interfaces. In particular, it redirects Linux' original initcall mechanism to the use of lx_emul_register_initcall and it hides low-level aspects of the memory model that are incompatible with Genode. The latter is concerned with the conversion between virtual addresses, struct page pointers, and DMA addresses.

Iterative crafting of the driver's runtime environment

The message "Error: Function ... not implemented yet!" (in the log output above) followed by a backtrace is triggered by one of the dummy implementations in generated_dummies.c. It tells us that we need to replace this particular dummy with either

• A dummy implementation in the manually curated dummies.c with the lx_emul_trace_and_stop call replaced by a meaningful return value, or

• An actual implementation of the function in lx_emul.c, or

• Additional Linux source codes incorporated by extending the sources.list.

The decision must be taken on a case-by-case basis. To take the decision, it is worthwhile to inspect the existing drivers. Drivers of the same kind (network, framebuffer) tend to show similar patterns across SoCs.

Our job at this stage is the repeated execution of the run script while resolving one unimplemented function in each iteration. Sometimes, this process requires a deep dive into parts of the Linux kernel architecture in order to asses the potential impact of the called dummy on the correct functioning of the driver. Sometimes mere intuition may guide us. In any case, the backtrace printed in the log output can be immensely helpful. You may remember the addr2line utility from a previous episode. It accepts an arbitrary sequence of numbers as standard input when started as follows:

build/arm_v8a$ /usr/local/genode/tool/current/bin/genode-aarch64-addr2line \
                                          -e drivers/nic/emac/emac_nic_drv

To process the list of hexadecimal numbers appearing in the log, I use to copy the numbers from the terminal using a rectangular selection (by pressing the control key while selecting an area with the mouse) and pasting the content into the addr2line instance.

Moving to the target hardware

At one point, we will ultimately reach a point where the driver tries to obtain access to device resources.

[init -> emac_nic_drv] Error: memory-mapped I/O resource 0x1c00000 (size=0x1000) unavailable

It's time to move the development from Qemu to the actual target hardware.

In order to grant the driver access to the requested resource, we first look up the requested address in the flat_pine64lts.dts file. In the example above, the range belongs to a device called syscon. With the information found in the device node, we can enrich the platform driver's configuration accordingly.

 <config>
   <device name="syscon" type="allwinner,sun50i-a64-system-control">
     <io_mem address="0x1c00000" size="0x1000"/>
   </device>
   <policy label="emac_nic_drv -> " info="yes">
     <device name="syscon"/>
   </policy>
 </config>

The change consist of two parts. First, a <device> is declared. Note that the type corresponds to the compatible attribute of the DTS device node. Second, the <policy> for the emac_nic_drv is changed to grant access of this device to the driver. It is important to set the info attribute to "yes", which allows the driver to read the device meta information given in the <device> node.

Besides memory-mapped I/O registers, device interrupts are the second type of hardware resource of interest for device drivers. Sooner or later during the driver initialization, we encounter a message like the following.

 [init -> emac_nic_drv] Error: irq 114 unavailable

The driver requests GIC interrupt 114. To find out what's behind this number, the flat_pine64lts.dts file is the right place for seeking the ground truth. Note that interrupt numbers as found in DTS files correspond to GIC interrupts numbers minus 32. So GIC interrupt 114 appears as number 114 - 32 = 82. A search in the DTS file for this number leads us to the matching device.

 emac: ethernet@1c30000 {
   ...
   reg = <0x01c30000 0x10000>;
   ...
   interrupts = <0 82 4>;
   ...
 };

This information is all we need to craft a corresponding <device> node for the platform-driver configuration.

 <device name="emac" type="allwinner,sun50i-a64-emac">
   <io_mem address="0x1c30000" size="0x10000"/>;
   <irq number="114"/>
 </device>

Linux caveats

In the past, we repeatedly encountered two kinds of trip-wires that one should always keep in the back of the mind, namely Linux linker-script magic and global variables.

A few kernel mechanisms depend of special support at the linker-script level, most notably various flavours of initcall mechanisms, sometimes disguised as a global table (__clk_of_table, __irqchip_of_table) magically created by scattered table entries assigned to a special linker section. In contrast to Linux, we cannot rely on linker-level mechanisms if we want to keep using Genode's regular linker script. The lx_emul environment takes care of the initcall flavors that we encountered so far using the pattern described here. But we know that there exist more categories of initcalls. In the event that a certain initialization function is unexpectedly not called, it is worth skimming the symbols of generated_dummies.o for variables with table in their name. Another example for linker magic is the aliasing between the jiffies and jiffies64 variables. Both variables must refer to the same memory location (on little-endian architectures). This concrete issue is covered by the lib/import/import-a64_lx_emul.mk file.

The second trip wire is the presence of global variables that are specially initialized by compilation units not featured in sources.list. In this case, the generated_dummies.c creates default-initialized variable instances, which can break innocent library functions in subtle ways. For example, lib/hexdump.c contains the following global variable:

 const char hex_asc_upper[] = "0123456789ABCDEF";
 EXPORT_SYMBOL(hex_asc_upper);

This variable is implicitly used by lib/vsprintf.c when printing "%d" format strings. If default-initialized, a digit is wrongly rendered as null (terminating the string) instead of the corresponding ASCII value, producing all kinds of funny effects down the road. The global variable defined in lib/ctypes.c is equally important. If default-initialized, toupper and strcasecmp won't work as expected, breaking the program logic when used as condition.

As a rule of thumb, when encountering erratic behavior, one should look out for global variables in generated_dummies.c and investigate their purpose.

Debugging and development hints

Enabling Linux debug messages

Several parts of the Linux kernel are garnished with debug messages that reveal valuable insights of the kernel's behavior beyond the regular log messages. The easiest way to obtain those messages for a given compilation unit is adding the following line right at the beginning of the source file, above the first #include directive:

 #define DEBUG

When booting Linux, one has to supply "debug" as kernel-command line argument. When running the driver as Genode component, no further precaution is needed.

With respect to debug instrumentation, the following compilation units are particularly fruitful:

Logging the execution of initcalls

To unveil the execution sequence of initcalls and for relating messages and backtraces printed in the log with the corresponding Linux code, two little instrumentations are of great help.

Obtaining backtraces of blocked Linux tasks

The Linux kernel code is not executed in a straight linear fashion but in the form of many kernel threads that interact with each other using a variety of mechanisms such as work queues. The lx_emul runtime implements a cooperative task-execution model that folds all Linux kernel threads into a single flow of control. To see what the Linux kernel threads are doing and in particular the situation when they enter a blocking state, the following instrumentation in dde_linux/src/lib/lx_kit/task.cc is invaluable.

 #include <os/backtrace.h>

 void Task::block()
 {
   log("Task::block: ", _name);
   backtrace();
   ...

The Task::block method is called whenever a Linux kernel thread blocks. By adding the two lines at the beginning of the method, we get hold of the situation at each single task switch. It prints the name of the blocked kernel thread along with the backtrace of the thread.

Another suitable point for an instrumentation is the Task::run method. By printing the _name after the _setjmp branch, one can obtain the sequence of resumed (as opposed to blocked) kernel threads.

 void Task::run()
 {
   if (_setjmp(_saved_env))
     return;

   log("Task::run: ", _name);
   ...

De-referenced null pointers

The Linux kernel is anything but short of function pointers and callbacks. Hence, sooner or later during the development, one may be faced with a de-referenced null pointer like this:

 no RM attachment (READ pf_addr=0x18c pf_ip=0x1008fa0 from pager_object: pd='init -> emac_nic_drv' thread='ep')
 Warning: page fault, pager_object: pd='init -> emac_nic_drv' thread='ep' ip=0x1008fa0 fault-addr=0x18c type=no-page

The very small page-fault address (pf_addr) hints at a de-referenced null pointer. The first impulse is looking up the instruction pointer pf_ip in the driver's binary using objdump.

 build/arm_v8a$ /usr/local/genode/tool/current/bin/genode-aarch64-objdump \
                            -lSd drivers/nic/emac/emac_nic_drv | less

In less, when searching for the instruction pointer value (1008fa0), one can see the surroundings of the offending code.

 1008f9c:       aa0003f3        mov     x19, x0
.../src/linux/drivers/base/regmap/regmap.c:2720
        if (!IS_ALIGNED(reg, map->reg_stride))
 1008fa0:       b9418c00        ldr     w0, [x0, #396]

Could map be a null pointer? If so, why? When looking into the code at the displayed coordinates regmap.c at line 2720, we encounter the function regmap_read as a suitable point for instrumentation.

#include <lx_emul.h>
...
int regmap_read(struct regmap *map, unsigned int reg, unsigned int *val)
{
  int ret;
  printk("regmap_read map=%p\n", map);
  if (!map)
    lx_emul_trace_and_stop(__func__);

  if (!IS_ALIGNED(reg, map->reg_stride))
    return -EINVAL;

The printk should validate our hypothesis that map is indeed a null pointer - just to double-check. The lx_emul_trace_and_stop call gives us the backtrace in this interesting situation. When running the code with these instrumentations in place, the following output appears.

[init -> emac_nic_drv] regmap_read map=0
[init -> emac_nic_drv] Error: Function regmap_read not implemented yet!
[init -> emac_nic_drv] Backtrace follows:
[init -> emac_nic_drv] 0x1008fc0
[init -> emac_nic_drv] 0x1009044
[init -> emac_nic_drv] 0x100a9b8
[init -> emac_nic_drv] 0x10076c0
[init -> emac_nic_drv] 0x10060f8
[init -> emac_nic_drv] 0x1006938
[init -> emac_nic_drv] 0x10069a4
[init -> emac_nic_drv] 0x1001320
[init -> emac_nic_drv] 0x1001ac4
[init -> emac_nic_drv] 0x10074a0
[init -> emac_nic_drv] 0x1056f28
[init -> emac_nic_drv] 0x10481e8
[init -> emac_nic_drv] 0x10580f8

Thanks to the backtrace, we can track where the map pointer comes from, ultimately ending up at the call of syscon_regmap_lookup_by_phandle. In our case, this function was (wrongly) stubbed with a dummy function returning NULL. As a way to double-check that the return value of this function indeed corresponds to the de-referenced null pointer, one can tweak the return value a little, returning a smallish magic number.

struct regmap * syscon_regmap_lookup_by_phandle(struct device_node * np,const char * property)
{
  return (struct regmap *)0x550;
}

In the next run, we can observe that the page-fault address indeed changed from 0x18c to 0x6dc.

 no RM attachment (READ pf_addr=0x6dc pf_ip=0x1008fc0 from pager_object: pd='init -> emac_nic_drv' thread='ep') 
 Warning: page fault, pager_object: pd='init -> emac_nic_drv' thread='ep' ip=0x1008fc0 fault-addr=0x6dc type=no-page

Apparently, we cannot simply shortcut the syscon_regmap_lookup_by_phandle function.

Ruling out potential cache-coherency issues

Once the driver starts to interact with the device hardware, additional uncertainties enter the picture. The most uncertain uncertainty is certainly cache coherency. Nowadays, Linux drivers preferably use cached page mappings for DMA buffers and manage the coherency between the device's perspective and the CPU's perspective on those buffers via explicit cache-management (flush, invalidate) operations. This cache management happens behind the surface of functions like dma_map_page_attrs.

To rule out the presence of cache coherency issues, we can force the driver to use uncached mappings only by tweaking the allocators at dde_linux/src/include/lx_kit/env.h. By changing the CACHED argument of the memory member to UNCACHED, all memory dynamically allocated by Linux kernel code will be backed by uncached memory.

Should the driver work with this tweak, one can be pretty sure to have hit a cache-coherency issue, likely missing the correct implementation of a dma_map / dma_unmap operation. Should the driver still does not work, the problem lies somewhere else. Now would be the time to suspect cosmic rays.

Using Linux' built-in DHCP support as networking test

Before equipping the driver with a Genode session interface, it is recommended to first execute its core functionality as a standalone program. For a network driver, the core functionality is the transmission and reception of network packets.

The Linux kernel features builtin support for obtaining an IPv4 network configuration at boot time via DHCP. The network-configuration protocol involves the successful transmission and reception of multiple network packets, and its completion is indicated by the IP address printed in the kernel log. In other words, DHCP is the ideal first test workload for the driver. It requires the following kernel configuration options:

 INET
 IP_PNP
 IP_PNP_DHCP

The implementation resides in net/ipv4/ipconfig.c, which must be added to the sources.list file of the driver. When being part of a regular Linux kernel, this code evaluates the kernel command line, namely the option "ip=dhcp". Since the lx_emul environment has no notion of a kernel command line, we can manually force the code to issue the DHCP request by modifying the implementation of the ip_auto_config function by adding calls to skb_init and ip_auto_config_setup at the beginning of the function (right after the local variable declarations).

static int ip_auto_config_setup(char *addrs);

static int __init ip_auto_config(void)
{
  ...
  skb_init();
  ip_auto_config_setup("dhcp");
   ...

Capturing network traffic

There exist many ways to capture network traffic for observing the interchange of DHCP protocol messages at the DHCP-server side. The tshark tool is particularly nice. For capturing the traffic related to the MAC address of my board, the following command line does an excellent job:

 tshark -i eno1 -t ad -Y 'eth.addr == 02:ba:fe:7b:59:38'

Using flood ping as a rudimentary stability check

Once the driver has reached a seemingly operational state, having successfully completed DHCP, it is a good time to put some stress on the driver. As a litmus test, its interesting to see if a flood ping brings the driver to its knees:

 sudo ping -f -c 1000 -s 1000 <ip-address>

Cross-correlation against the Linux kernel behavior

Sharing one Linux kernel configuration among both our bare-bones Linux kernel and the ported driver code allows for the detailed cross-correlation of the driver behavior. This consistency is fostered by the src/a64_linux/target.inc file that is used by both the a64_linux target (configuring and building a Linux kernel) and each driver component (via the a64_linux_generated library). This way, any instrumentation of the Linux kernel code can be quickly tested in both execution environments.

Connecting the driver with a Genode session interface

Once we have validated that the driver is able to send and receive network packets via the DHCP test, the time is ripe for integrating it into the Genode environment. This integration comes down to two aspects. First, the test scenario must be changed to move the network application into a component separate from the driver, and second, the driver must interact with Genode's uplink session interface.

The first part, the network application, can be accommodated by the NIC router component. For reference, the following <start> node creates an instance of the NIC router that issues a DHCP request once an uplink appears, and prints the obtained IP address in the log.

 <start name="nic_router" caps="200">
   <resource name="RAM" quantum="10M"/>
   <provides>
     <service name="Nic"/>
     <service name="Uplink"/>
   </provides>
   <route>
     <service name="Timer"> <child name="timer"/> </service>
     <any-service> <parent/> </any-service>
   </route>
   <config verbose_domain_state="yes" dhcp_discover_timeout_sec="1">
     <policy label_prefix="emac_nic_drv" domain="uplink"/>
     <domain name="uplink"/>
   </config>
 </start>

Of course, the driver must be able to reach the NIC router, which can be achieved adding the following session route to the driver's <start> node.

 <start name="emac_nic_drv" caps="2000">
   ...
   <route>
     ...
     <service name="Uplink"> <child name="nic_router"/> </service>
     ...
   </route>
 </start>

The second part - bridging the gap between the Linux kernel code and Genode's session interface - can best be addressed by the genode_c_api/uplink.h API and an implementation of the driver's lx_user.c, which connects the genode_c_api with the Linux netdevice interface.

Packaging the driver

The final step is the packaging of the driver to make it available to a broad range of Genode scenarios, in particular the run scripts based on the drivers_nic subsystem such as libports/run/fetchurl_lwip.run.

The packaging is assisted by the dde_linux/list_dependencies tool. It determines the list of header dependencies for our driver by examining dependency (.d) files. For reference, the nic/emac/dep.list file is generated via following command (the paths are abbreviated).

build/arm_v8a$ ../../tool/dde_linux/list_dependencies \
               TARGET_DIR=drivers/nic/emac \
               LINUX_KERNEL_DIR=/path/to/linux/source/ \
               SOURCE_LIST_FILE=.../allwinner/src/drivers/nic/emac/source.list \
               DEP_LIST_FILE=.../allwinner/src/drivers/nic/emac/dep.list \
               generate

As reference for the recipe files needed, the depot recipes at allwinner/recipes/ are helpful. Their roles are as follows:

While crafting those recipes, it is best to use a dummy depot user "x" so that the intermediate results can easily be removed from the depot afterwards. For reference, the following command extracts the archives from the source tree and builds the binaries for the arm_v8a architecture. The process of developing the recipes comes down to repeatedly issuing this command and extending the recipes until the binary archives are successfully built.

genode$ ./tool/depot/create x/pkg/arm_v8a/drivers_nic-pine_a64lts \
                            -j8 \
                            FORCE=1 \
                            UPDATE_VERSIONS=1

What's next?

In the next episode, we will move our focus from the Pine-A64-LTS board to the PinePhone. Read on...