Site logo
Stories around the Genode Operating System RSS feed
Martin Stein avatar

Integrating and running automated tests - Part 2

In this article I'd like to give a very practical guide about how you can create, integrate and run your custom test scenarios on Genode. This is the second of two parts. If you have missed the first part, you may want to read it first.

Step 2: The source archives

Now that I have written my new my_expat test component, I could hastily try to run the Depot-Autopilot run script to evaluate it. But the script would reasonably complain as follows:

 Error: unable to guess version of 'test-my_expat' archive

This is because the Genode depot system wouldn't find any recipe to create an archive named test-my_expat. This archive is more precisely speaking a package that should contain the top level description of my test scenario - which ingredients are needed, how to integrate them, and how to interpret the test results. But actually I am missing even more, because the above mentioned test ingredients themselves must also be available as archives. Besides the my_expat component, I want, for example, also an Init component to act as runtime environment. And an archive for the my_expat component creates further dependencies. Because the component makes use of shared libraries like expat and I wouldn't want to mix a shared library in an archive of a component that uses it but rather have it in a dedicated archive. So, the link between my component and a ready-to-use test scenario can be depicted as a tree where the leafes are the actual content (e.g. source code):

   +-> recipes/src/test-my_expat --+-> src/test/my_expat
   |                               |
   +-> recipes/src/expat           +-> recipes/api/expat -> ...
   |                               |
   +-> recipes/src/init            +-> recipes/api/libc -> ...
   +-> ...

Fortunately, all archives that my test requires, except pkg/test-my_expat and src/test-my_expat, already exist. So, I start with the src/test-my_expat archive:

 mkdir -p repos/libports/recipes/src/test-my_expat
 vim repos/libports/recipes/src/test-my_expat/

The archive shall trigger the creation of the test-my_expat binary. In order to achive this, it needs, first of all, the corresponding file:

 # file repos/libports/recipes/src/test-my_expat/
   mkdir -p src/test/my_expat
   cp $(REP_DIR)/src/test/my_expat/ src/test/my_expat/

Then I add a bogus initial version and hash:

 echo 1970-01-01 0 > repos/libports/recipes/src/test-my_expat/hash

And give the recipe a first try:

 tool/depot/create genodelabs/bin/x86_64/test-my_expat \

This silently fixes my bogus hash file and then complains about some missing library makefiles. Sidenote: If, at some point, the create tool doesn't act as described in this article, you may try to reset your depot directory and try again (be aware that this deletes all your previously created archives, so, you might want to do this in a temporary copy of your genode directory):

 rm -rf depot && git checkout HEAD depot

Now, back to the main track: I can see now that the source archive was created as expected but no binary could be compiled from this:

 ls depot/genodelabs/src/test-my_expat/<LATEST_VERSION>/src/test/expat
 ls depot/genodelabs/bin/x86_64/test-my_expat/<LATEST_VERSION>

I can solve the missing libraries by simply referencing the - as mentioned earlier - already existing API archives:

 export USED_APIS=libports/recipes/src/test-my_expat/used_apis
 echo base  >  $USED_APIS
 echo expat >> $USED_APIS
 echo posix >> $USED_APIS
 echo libc  >> $USED_APIS

When running the create command again, I get a bit further:

  Library ldso-startup
  Library expat
  Library ld
  Library libc
  Library posix
  Library libm
  Library base
  Program test/expat/test-my_expat
    LINK     test-my_expat
  genode-x86-g++: error: main.o: No such file or directory

Of course, I didn't copy the corresponding C++ file from test/my_expat to the source archive (I did this to ensure that I don't copy stuff to the source archive that isn't needed). Let's enhance the file accordingly:

 # file repos/libports/recipes/src/test-my_expat/
   mkdir -p src/test/my_expat
   cp $(REP_DIR)/src/test/my_expat/ src/test/my_expat/
   cp $(REP_DIR)/src/test/my_expat/   src/test/my_expat/

Now, the create command produces the desired result:

 ls depot/genodelabs/bin/x86_64/test-my_expat/<LATEST_VERSION>

If you want to know more about archives and the depot, you may read this online documentation or section 5.5 in the Genode Foundations book.

Step 3: The package archives

Let's integrate the test into a package archive. First, I create the package skeleton:

 mkdir -p libports/recipes/pkg/test-my_expat
 echo 1970-01-01 0 > repos/libports/recipes/pkg/test-my_expat/hash
 echo "my expat test" > repos/libports/recipes/pkg/test-my_expat/README

Then, I declare which archives must be build for my test package:

 export ARCHIVES=repos/libports/recipes/pkg/test-my_expat/archives
 echo "_/src/init"          >  $ARCHIVES
 echo "_/src/test-my_expat" >> $ARCHIVES
 echo "_/src/expat"         >> $ARCHIVES
 echo "_/src/libc"          >> $ARCHIVES
 echo "_/src/vfs"           >> $ARCHIVES
 echo "_/src/posix"         >> $ARCHIVES

Finally, I have to describe the runtime of my test. This is done through the runtime XML file. The top level of this file describes the runtime component itself (Init in my case):

 <!-- excerpt from repos/libports/recipes/pkg/test-my_expat/runtime -->
 <runtime ram="32M" caps="1000" binary="init">

Then, I add four sub-nodes:

 <!-- excerpt from repos/libports/recipes/pkg/test-my_expat/runtime -->
 <runtime ram="32M" caps="1000" binary="init">
   <config>   ... </config>
   <content>  ... </content>
   <requires> ... </requires>
   <events>   ... </events>

Into the <config> sub-node, I put the Init configuration that causes the Init to start and connect the components for my test properly:

 <!-- excerpt from repos/libports/recipes/pkg/test-my_expat/runtime -->
     <service name="ROM"/>
     <service name="PD"/>
     <service name="RM"/>
     <service name="CPU"/>
     <service name="LOG"/>
     <service name="IO_PORT"/>
     <service name="IRQ"/>
     <service name="Timer"/>
     <any-service> <parent/> <any-child/> </any-service>
   <default caps="200"/>
   <start name="test-my_expat">
     <resource name="RAM" quantum="2M"/>
         <inline name="config"><config>
             <test_tag test_attribute="test_value" />
         <dir name="dev"> <log/> </dir>
       <libc stdout="/dev/log"/>

While creating this configuration, I already recognize, that later the Depot Autopilot must make some files - here the expat program and library binaries - availabe to the runtime. This is what the <content> node is for:

 <!-- excerpt from repos/libports/recipes/pkg/test-my_expat/runtime -->
   <rom label=""/>
   <rom label=""/>
   <rom label=""/>
   <rom label=""/>
   <rom label=""/>
   <rom label=""/>
   <rom label="test-my_expat"/>

As you may have noticed in the configuration, instead of spawning a timer driver, I've routed the Timer service to the parent. As it would be cumbersome to start required drivers for each test anew, the Depot-Autopilot run-script starts drivers outside the tests and hands-in the required sessions. Thus, I have to tell the outer world that I need a Timer via the <requires> tag:

 <!-- excerpt from repos/libports/recipes/pkg/test-my_expat/runtime -->
 <requires> <timer/> </requires>

Last but not least, I want to tell the Depot-Autopilot component how to evaluate the result of my test. I can tell it to inspect the LOG sessions for bad or good string patterns and install a timeout to make sure that my test doesn't block the whole system. All this goes to the <events> tag:

 <!-- excerpt from repos/libports/recipes/pkg/test-my_expat/runtime -->
   <log     meaning="succeeded">
     [init -> test-my_expat]  start of element: config*
     [init -> test-my_expat]  start of element: test_tag*
     [init -> test-my_expat]  attribute: name='test_attribute', value='test_value'*
     [init -> test-my_expat]  end of element: test_tag*
     [init -> test-my_expat]  end of element: config*
     "test-my_expat" exited with exit value 0
   <log     meaning="failed">Error: </log>
   <timeout meaning="failed" sec="10" />

This is all. I can successfully create my package now:

 tool/depot/create genodelabs/pkg/x86_64/test-my_expat \
 ls depot/genodelabs/pkg/test-my_expat/<LATEST_VERSION>
   archives  README  runtime

Step 4: Running the Depot Autopilot

First, I'd like to give my test a try without being disturbed by the whole battery of available tests:

 KERNEL=nova TEST_PKGS=test-my_expat TEST_SRCS= make run/depot_autopilot

Most propably, this will complain about some missing archives and provide a depot/create command line that will fix the problem. I do as advised (for more convenience, I also add UPDATE_VERSIONS=1 to the command line) and as soon as all archives are built, I retry to run the Depot-Autopilot command. The scenario starts and through the serial output the Depot Autopilot tells me something like:

 --- Run "test-my_expat" (max 10 sec) ---

 4.340 [init -> test-my_expat] Warning: rtc not configured, returning 0
 4.396 [init -> test-my_expat]  start of element: config
 4.406 [init -> test-my_expat]  start of element: test_tag
 4.412 [init -> test-my_expat]  attribute: name='test_attribute', value='test_value'
 4.420 [init -> test-my_expat]  end of element: test_tag
 4.429 [init -> test-my_expat]  end of element: config
 4.644 [init] child "test-my_expat" exited with exit value 0

  test-my_expat                   ok         4.644  log "[init -> test-my_expat]  start o ..."

 --- Finished after 6.358 sec ---

  test-my_expat                   ok         4.644  log "[init -> test-my_expat]  start o ..."

 succeeded: 1 failed: 0 skipped: 0

I might want to debug the test. In this case, the additional environment variables considered by the run script become helpful:

 KERNEL=nova \
   TEST_PKGS=test-my_expat \
   TEST_BUILDS="test/expat init" \
   TEST_MODULES="test-expat init" \
   TEST_REPEAT=until_failed make

You can read about all this in more detail in the README file of the Depot Autopilot:


In the end, I can add my test permanently to the Depot-Autopilot run-script by enhancing the default packages list:

 # excerpt from repos/gems/run/
 set default_test_pkgs {