Zynq guide #1 - getting started
This is the first article of a series about the Zynq-7000 SoC from Xilinx. Though not a particularly new SoC, it is still quite unique as it features an FPGA in conjunction with a Cortex-A9 CPU. This opens a bunch of interesting application scenarios since the FPGA can act as a customisable co-processor or hardware accelerator. In this article, I will guide you through the very basic board bring-up to get a first sign of life from this hardware with Genode.
I already had my first contact with the Zynq-7000 SoC a few years ago. This is also when the basic support was contributed to Genode. Yet, complete board support, including the FPGA, as well as security aspects and componentisation were never fully addressed. Since I recently got my hands on a few Zynq-based boards again and started digging into these topics, I want to share my experiences in this article series.
In this article, I begin with the foundations about the platform and a guide on how to enable the support of a new Zynq-based board in Genode. Note, that step-by-step instructions for already supported Zynq boards are readily available in the genode-zynq repository.
Zynq 101
Before continuing with the hands-on part of this article, I will briefly introduce the SoC and what challenges it holds when it comes to bringing Genode to this platform. In case you are already familiar with this and just want to get started, you may easily skip this part and fast-forward to Bringing Genode to the USRP E310.
FPGA foundations
If you never heard of FPGA, it stands for field-programmable gate array. From a software perspective, FPGAs can more generally be termed programmable logic. The Xilinx documents therefore refer to it as PL and to the CPU as the processing system (PS). From a hardware perspective, as the name suggests, FPGAs are some kind of regular structure of logic gates (gate array) that is programmable in the field. It is basically composed of logic cells and block RAM. A logic cell is made programmable by look-up tables that store a custom logic function for a fixed number of logic inputs. Though typically slower than an ASIC (application-specific integrated circuit), FPGAs are great for prototyping and if only a few devices are needed.
An FPGA itself does not store its configuration persistently but must be configured at boot time or run time. The binary format storing the configuration is called the bitstream. It is typically generated by vendor-specific tools from hardware-description languages such as VHDL or Verilog. Modern FPGAs even support partial reconfiguration which keeps parts of the FPGA operational while reprogramming other parts.
The Zynq-7000 family
The Xilinx Zynq-7000 SoC is actually an entire family of SoCs. More specifically, there are single and dual-core Cortex-A9 (Armv7-A) variants as well as varying FPGA sizes. The size of an FPGA is basically determined by the number of logic cells, I/O pins and the amount of block RAM.
Depending on the speed grade, a Zynq-7000 SoC can run up to 1GHz. Most boards I have seen, however, tend to run with 667MHz. Although this seems quite low for a CPU, the possibility of hardware acceleration by the FPGA can outweigh the limited CPU frequency.
Luckily, all Zynq SoCs share the same set of on-chip peripherals. For instance, there are DDR memory, CAN, SPI, UART, I2C, GPIO and DMA controllers. There is also a USB 2.0 host, device and OTG controller as well as two Gigabit Ethernet and SD/SDIO controllers. If you want a more detailed insight into the SoC, I can highly recommend the TRM aka ug585 from Xilinx.
Note, that the Zynq-7000 series must not be confused with the newer Zynq UltraScale+ SoC series, which bases on a Cortex-A53 (Armv8-A). I will use "Zynq" synonymously for the Zynq-7000 SoC in this article series.
Zynq-based boards
There is a large variety of boards featuring a Zynq-7000 SoC on the market. Depending on the application domain, the boards come with a different number of peripheral interfaces and are covering a wide price range. Most common application areas are audio/video processing and software-defined radio.
On the low-budget end of the spectrum, there are a couple of single-core devices like the Cora-Z7 by Digilent or the Minized by Avnet, both being sold for under 100USD/EUR. Both vendors also have dual-core variants with different FPGA sizes in their portfolio, e.g. the Zybo-Z7, the Picozed, the Microzed, the Ultrazed or the Zedboard. They typically sell for a few hundred USD/EUR if they are still available. On the other end of the spectrum, hardware-enthusiasts can also spend a couple of thousand bucks on the Xilinx ZC702 or Xilinx ZC706 evaluation board.
Some boards combine the Zynq with additional computing hardware or peripherals. For example, there is the Parallella board by Adapteva that also features a 16-core Epiphany processor. Interestingly, despite the extra hardware, the Parallella is still available for a very low budget. If you are into software-defined radios and are willing to spend a few thousand USD/EUR, you probably want to have a look at the Universal Software Defined Peripherals (USRPs) E310/E312/E320 from Ettus Research.
The devices named above are just an excerpt of what you find on the market. If you are interested in conducting your own research, you may have a look at the Xilinx web page. When comparing specs, in addition to looking at peripherals and connectivity, you probably want to check the speed grade, the number of cores, the size of the FPGA and the amount of DDR memory as well. Another essential aspect is the quality of documentation: Do you have to read the schematics or is there some prose that explains how things are interconnected on the board? I prefer both ;-)
Challenges ahead
If you have been following Norman’s article series on the Pinephone or Stefan’s journey with the MNT Reform, you already know that bringing Genode to a new hardware platform typically involves writing or porting the necessary device drivers.
Fortunately, there is no reverse engineering necessary for the Zynq’s on-chip peripherals since everything is well-documented in the ug585. Even better, Xilinx not only provides Linux drivers but also a bunch of baremetal drivers and libraries for their devices. For the Zynq-7000, we therefore have the choice between writing device drivers from scratch, porting baremetal drivers, or porting Linux drivers. Besides the basic support for the SoC, the most essential peripherals are the Ethernet controller for network connectivity and the SD controller for persistent storage.
Additionally, we are going to break new ground with the FPGA. It not only involves additional tooling to build custom bitstreams but also raises the question of how to securely integrate and interface the PL in Genode. Given that the Zynq-7000 does not comprise a system MMU but the FPGA has full access to the system bus, a rogue DMA controller can easily compromise the entire system.
Bringing Genode to the USRP E310
Alright, let’s start with the more fun hands-on part. I recently got my hands on a new Zynq board, the USRP E310 from Ettus Research. All Zynq boards I had on my desk so far either came with a pre-built SD card or with a download link for an SD card image. On the SD card, you typically find u-boot installed to boot up a Linux system since Xilinx provides the corresponding tools called PetaLinux for this. Some boards, e.g. the Zybo-Z7, also have a demo program in their on-board flash memory. Before doing any modifications, it is therefore a good habit to check that the board boots up the reference scenario correctly. One aspect to keep in mind when doing this is that the Zynq can boot from flash memory, SD card or JTAG. The boot mode is typically controlled by a jumper. In case of the USRP, it boots from an SD card by default, which was already shipped with the device.
Since the Zynq is an embedded platform, the reference scenarios rarely do any graphical output. In order to check whether the system is booting up correctly, I therefore needed to attach the board’s UART to my host system. Since I’m working from a Linux VM on Sculpt, I also had to pass through the corresponding USB device to my VM. When the board is connected and powered, a /dev/ttyUSB0 device pops up in my Linux system to which I connect with screen:
sudo screen /dev/ttyUSB0 115200
Note: You must hit "Ctrl+A k" to exit screen.
As Norman already described in his first pine fun article, it is a good advice to dump some information from the working Linux system before actually starting with Genode:
#> cat /proc/cpuinfo ... #> cat /proc/meminfo ... #> cat /proc/iomem ... #> cat /proc/interrupts ... #> dmesg ...
I stored these files on my Linux system for future reference. Time will tell if we actually need them.
Compiling my own u-boot
Although the Zynq boards are shipped with a pre-compiled u-boot, I wanted to compile it on my own because of several reasons: Since some boards are already around for quite a few years, the u-boot versions deployed on various boards may differ significantly and therefore have different feature sets and configurations. Furthermore, I noticed that some deployments use the u-boot SPL (secondary program loader) and some use the Xilinx FSBL (first-stage boot loader) as a first stage. In order to prevent adapting Genode to an entire ecosystem of u-boot deployments, I thus opted for integrating the compilation of u-boot into the Genode build system and thereby have a similar deployment on every device.
As a first step, I tried compiling u-boot from scratch for the USRP. The upstream repository already knows a bunch of Zynq boards, except any USRP. Yet, due to the similarity between the Zynq boards, adding another board is only a matter of adding a (reduced) device tree and a ps7_init_gpl.c. Adding the device tree is pretty straightforward as we already have a dts file for reference on the shipped SD card. Looking at the existing dts files at arch/arm/dts/zynq-* in the u-boot repository, however, u-boot only needs to know a couple of devices. I thus copy-pasted the file of an existing board and modified it to reflect the properties from the dts file from Ettus. The resulting zynq-usrp-e31x.dts looks as follows:
// SPDX-License-Identifier: GPL-2.0+
/*
* Copyright (C) 2011 - 2015 Xilinx
* Copyright (C) 2012 National Instruments Corp.
*/
/dts-v1/;
#include "zynq-7000.dtsi"
/ {
model = "NI Ettus Research USRP E31x-3";
compatible = "ettus,e31x", "xlnx,zynq-7000";
aliases {
ethernet0 = &gem0;
serial0 = &uart0;
spi0 = &qspi;
mmc0 = &sdhci0;
};
memory@0 {
device_type = "memory";
reg = <0x0 0x40000000>;
};
chosen {
bootargs = "";
stdout-path = "serial0:115200n8";
};
};
&clkc {
ps-clk-frequency = <33333333>;
};
&gem0 {
status = "okay";
phy-mode = "rgmii-id";
phy-handle = <ðernet_phy>;
ethernet_phy: ethernet-phy@0 {
reg = <0>;
device_type = "ethernet-phy";
};
};
&qspi {
u-boot,dm-pre-reloc;
status = "okay";
};
&sdhci0 {
u-boot,dm-pre-reloc;
status = "okay";
};
&uart0 {
u-boot,dm-pre-reloc;
status = "okay";
};
The Zynq-7000 has multiple UART, Ethernet and SD controllers, hence we must choose which to use. Since the devices are already defined in zynq-7000.dtsi, this is only a matter of replacing any occurence of &uart1 with &uart0 for instance. In addition to the devices, we must also check and adapt the ps-clk-frequency and the memory size, which we can both steal from the shipped dts file. Before being able to compile u-boot, I also needed to add zynq-usrp-e31x.dtb to dtb-$(CONFIG_ARCH_ZYNQ) in arch/arm/dts/Makefile.
With these small changes, we are already able to build u-boot by executing make DEVICE_TREE=zynq-usrp-e31x CROSS_COMPILE=/usr/local/genode/tool/21.05/bin/genode-arm-. By default, u-boot uses a standard initialisation routine for the Zynq-7000 SoC but allows this to be replaced by adding a ps7_init_gpl.c file under board/xilinx/zynq/zynq-usrp-e31x/. This file is generated if you build your own FPGA design with the Xilinx tools, which will be a topic of a future article. For the USRP, I found the file in the uhd repository from Ettus Research. As it turned out, one can re-use the file with only a few adaptations, i.e. changing the #include directive and removing all functions except ps7_init() and ps7_post_config().
Having a minimal patch to u-boot’s upstream repository, the path has been cleared for integrating it into Genode’s build system. Since, Norman already went down this road, I was able to use his result as a blueprint. What I found a useful addition to his solution, though, was adding the option DEFAULT_ENV_FILE="./env.txt, which enables providing a custom built-in configuration. With the following env.txt, u-boot first tries to find a uEnv.txt file on the SD card and running uenvcmd if it is defined. After that, it tries to boot the /boot/uImage file from the SD card:
bootcmd=run uenvboot; run sdboot
bootenv=boot/uEnv.txt
image=boot/uImage
loadbootenv_addr=0x30000000
loadbootenv=load mmc 0:1 ${loadbootenv_addr} ${bootenv}
load_addr=0x2000000
importbootenv=env import -t ${loadbootenv_addr}
sd_uEnvtxt_existence_test=test -e mmc 0:1 ${bootenv}
sd_image_existence_test=test -e mmc 0:1 ${image}
uenvboot=if run sd_uEnvtxt_existence_test; then run loadbootenv; \
run importbootenv; fi; if test -n $uenvcmd; \
then echo Running uenvcmd ...; run uenvcmd; fi
sdboot=if run sd_image_existence_test; then load mmc 0:1 ${load_addr} ${image}; \
bootm start ${load_addr}; bootm loados; bootm go; fi
All of this is already integrated into the genode-zynq repository, so that we are able to build SD card images containing our customised u-boot.
Building an SD card image
By having u-boot support in the genode-zynq repository, we are able to build our first SD card image.
First, if you haven’t already done so, obtain a clone of the repository within the main genode repository, prepare the zynq_uboot port and create a build directory:
genode #> git clone https://github.com/genodelabs/genode-zynq.git repos/zynq genode #> ./tool/ports/prepare_port zynq_uboot genode #> ./tool/create_builddir arm_v7a
Now, uncomment the zynq repo in build/arm_v7a/etc/build.conf so that it contains this line
REPOSITORIES += $(GENODE_DIR)/repos/zynq
From the build directory, we are now able to build u-boot for the USRP and write it to an SD card as follows. Note, that each Zynq board must be enabled in the src/u-boot/zynq/target.mk.
build/arm_v7a #> make u-boot/zynq BOARD=zynq_usrp_e31x
[...]
CONVERT SD-card image u-boot/zynq/zynq_usrp_e31x.img
build/arm_v7a #> sudo dd if=u-boot/zynq/zynq_usrp_e31x.img of=/dev/mmcblkX bs=1M conv=fsync
The created image merely contains u-boot without any uImage to start, however, by booting up the device with the newly populated SD card, we are able to acquire some very useful information, namely the clock settings:
Zynq> clk dump
clk frequency
armpll 1333333320
ddrpll 1066666656
iopll 999999990
cpu_6or4x 666666660
cpu_3or2x 333333330
cpu_2x 222222220
cpu_1x 111111110
ddr2x 355555552
ddr3x 533333328
dci 10158730
lqspi 33333333
smc 22222222
pcap 199999998
gem0 124999999
gem1 16666667
fclk0 99999999
fclk1 199999998
fclk2 249999998
fclk3 41666666
can0 8000000
can1 8000000
sdio0 50000000
sdio1 50000000
uart0 99999999
uart1 99999999
spi0 166666665
spi1 166666665
dma 222222220
usb0_aper 111111110
usb1_aper 111111110
gem0_aper 111111110
gem1_aper 111111110
sdio0_aper 111111110
sdio1_aper 111111110
spi0_aper 111111110
spi1_aper 111111110
can0_aper 111111110
can1_aper 111111110
i2c0_aper 111111110
i2c1_aper 111111110
uart0_aper 111111110
uart1_aper 111111110
gpio_aper 111111110
lqspi_aper 111111110
smc_aper 111111110
swdt 111111110
dbg_trc 66666666
dbg_apb 66666666
That’s quite a list. These clocks are calculated by u-boot based on the ps-clk-frequency and the register settings done by our ps7_init_gpl.c. We will need some of the values in the next step when adding the board-support files to the genode-zynq repository.
Adding the board support files
In order to build the base-hw kernel for a new board, we must add the corresponding board support files. Since Norman has already explained this in much detail, I will keep this section very brief and Zynq-specific. Moreover, due to the similarity between the different Zynq boards, it is mostly a matter of copy-pasting.
First, let’s create the makefiles for bootstrap and core in lib/mk/spec/arm_v7/. For bootstrap, the board-generic part is found in the file bootstrap-hw-zynq.inc, so we get away with filling the file bootstrap-hw-zynq_usrp_e31x.mk with:
REP_INC_DIR += src/bootstrap/board/zynq_usrp_e31x NR_OF_CPUS = 2 include $(REP_DIR)/lib/mk/spec/arm_v7/bootstrap-hw-zynq.inc
The same story holds for core, so we can generate core-hw-zynq_usrp_e31x.mk as follows:
#> sed s/bootstrap/core/ lib/mk/spec/arm_v7/bootstrap-hw-zynq_usrp_e31x.mk \
> lib/mk/spec/arm_v7/core-hw-zynq_usrp_e31x.mk
Second, let’s have a look at the board-specific code in src/include/hw/spec/arm. The zynq.h is shared among all boards and contains the generic definitions. Thus, zynq_usrp_e31x.h only needs to contain the board-specific definitions: the RAM size, which UART to use and the clock frequencies. From our dts, we already know that the RAM size is 1GB and that UART0 is supposed to be used. The clocks we get convieniently from our "clk dump". This is the result:
#include <hw/spec/arm/zynq.h>
namespace Zynq_usrp_e31x {
using namespace Zynq;
enum {
CPU_1X_CLOCK = 111111110,
CPU_3X2X_CLOCK = 3*CPU_1X_CLOCK,
UART_CLOCK = 100*1000*1000,
UART_BASE = UART_0_MMIO_BASE,
RAM_0_SIZE = 0x40000000, /* 1GiB */
CORTEX_A9_PRIVATE_TIMER_CLK = CPU_3X2X_CLOCK,
CORTEX_A9_PRIVATE_TIMER_DIV = 100,
};
};
It is worth noting that the CORTEX_A9_PRIVATE_TIMER_CLK always runs with CPU_3X2X_CLOCK while its prescaler is set by the base-hw kernel according to the value of CORTEX_A9_PRIVATE_TIMER_DIV.
The following files only contain definitions and instantiations that depend on zynq_usrp_e31x.h. They are easily created by taking the ones from an existing Zynq board and replacing the board names.
-
src/include/hw/spec/arm/zynq_usrp_e31x_board.h
-
src/bootstrap/board/zynq_usrp_e31x/board.h
-
src/core/board/zynq_usrp_e31x/board.h
Last, we follow the same approach for the board-property directory board/zynq_usrp_e31x and the recipe recipes/src/base-hw-zynq_usrp_e31x.
With all the ingredients in place, we can build and compile our first Genode application. Let’s build a new SD card image hosting the log run script. In the arm_v7a build directory, we add the following lines to our etc/build.conf
RUN_OPT_usrp = --include image/uboot RUN_OPT_usrp += --include image/zynq_uboot_sdcard BOARD_RUN_OPT(zynq_usrp_e31x) = $(RUN_OPT_usrp)
This instructs the run tool to create a uImage to be booted by u-boot and to inject this uImage into the SD card base image that we already built. Make sure that you have mkimage (provided by u-boot-tools on Ubuntu) and mcopy (provided by mtools) available on your system. With these modifications, we are set up for building run/log and writing the result to our SD card.
build/arm_v7a #> make run/log BOARD=zynq_usrp_e31x KERNEL=hw [...] Created SD-card image file var/run/log.img build/arm_v7a$ sudo dd if=var/run/log.img of=/dev/mmcblkX bs=1M conv=fsync
Enabling TFTP boot
During development, fiddling with micro SD cards is very cumbersome. I therefore recommend activating TFTP boot.
First, we install tftp-hpa on the development system. Configuration may differ depending on your Linux distribution. On my Archlinux installation, I added the line TFTPD_ARGS="/srv/tftp" to the /etc/conf.d/tftpd and enabled the service via systemctl:
sudo systemctl enable tftpd sudo systemctl start tftpd
Second, since my Linux is actually running in a VM on Sculpt, I also added a forwarding rule to /config/nic_router
<domain name="uplink"> [...] <udp-forward port="69" domain="default" to="10.0.1.2" /> </domain>
Here, 10.0.1.2 is the IP that has been assigned to the VM.
Third, we must modify the build.conf a bit to instruct the build system to populate the tftp directory with the uImage and automatically connect to the serial console:
RUN_OPT_usrp = --include image/uboot RUN_OPT_usrp += --include load/tftp RUN_OPT_usrp += --load-tftp-base-dir /srv/tftp/usrp RUN_OPT_usrp += --load-tftp-absolute RUN_OPT_usrp += --include log/serial RUN_OPT_usrp += --log-serial-cmd "picocom -b 115200 /dev/ttyUSB0" BOARD_RUN_OPT(zynq_usrp_e31x) = $(RUN_OPT_usrp)
In a final step, we create a uEnv.txt file that will instruct u-boot to fetch the uImage from our tftp server:
serverip=x.x.x.x
kernel_img=/srv/tftp/usrp/uImage
uenvcmd=tftpboot ${load_addr} ${serverip}:${kernel_img} && \
bootm start && bootm loados && bootm go
Here, you must insert the IP address of your host computer. Also note, that kernel_img corresponds to the --load-tftp-base-dir set in our build.conf.
You can simply copy this uEnv.txt file to the /boot/ directory of the mounted SD card. I like to do this from within u-boot without removing the SD card from the device by running the following commands:
Zynq> tftpboot 0x10000000 x.x.x.x:/srv/tftp/usrp/uEnv.txt
Filename '/srv/tftp/usrp/uEnv.txt'.
Load address: 0x10000000
Loading: #
18.6 KiB/s
done
Bytes transferred = 154 (9a hex)
Zynq> fatwrite mmc 0 0x10000000 boot/uEnv.txt ${filesize}
Of course, do not forget to insert the tftp server’s IP address for "x.x.x.x" and to make the uEnv.txt available on your tftp server.
Alright, with these preparations, it’s time to try out another run script:
build/arm_v7a #> make run/timer_accuracy BOARD=zynq_usrp_e31x KERNEL=hw
Once "Terminal ready" appears, you just need to hit the power button of the USRP to (re)boot the device. With this setup, we not only omit touching any SD card but also enjoy all benefits of run scripts. In the example of the timer_accuracy, the run script parses the log output and compares the reported timing with the host system.
Next steps
Now, that we have set up our environment for efficient development, we can actually get started with enabling the peripherals. In the next series of this article, I am going to shed light on how to use the FPGA.
Edited (2022-02-09): Changed genode-zynq repository URL.
Edited (2023-01-19): Changed path of uImage and uEnv.txt on SD card.