Quick Start

Warning

This is the documentation for an old version of tbot. Please head over to

for the latest docs!

Commandline

First of all, install tbot. Instructions are here: Installation.

Let’s get started! To check if the installation went smoothly, try running tbot’s selftests:

bash-4.4$ tbot selftest
tbot starting ...
├─Calling selftest ...
│   ├─Calling testsuite ...
│   │   ├─Calling selftest_failing ...
│   │   │   ├─Calling inner ...
│   │   │   │   └─Fail. (0.000s)
│   │   │   └─Done. (0.001s)
│   │   ├─Calling selftest_uname ...
│   │   │   └─Done. (0.003s)
│   │   ├─Calling selftest_user ...
│   │   │   └─Done. (0.001s)
[...]
│   │   ├─Calling selftest_board_linux ...
│   │   │   ├─Skipped because no board available.
│   │   │   └─Done. (0.000s)
[...]
│   │   ├─Calling selftest_board_linux_bad_console ...
│   │   │   ├─POWERON (test)
│   │   │   ├─LINUX (test-linux)
│   │   │   ├─POWEROFF (test)
│   │   │   └─Done. (0.460s)
│   │   ├─────────────────────────────────────────
│   │   │ Success: 17/17 tests passed
│   │   └─Done. (4.675s)
│   └─Done. (4.742s)
├─────────────────────────────────────────
└─SUCCESS (4.845s)

If you feel adventurous, there are even more selftests that check if the built-in testcases work as intended:

bash-4.4$ tbot selftest_tc
tbot starting ...
├─Calling selftest_tc ...
│   ├─Calling testsuite ...
│   │   ├─Calling selftest_tc_git_checkout ...
│   │   │   ├─Calling git_prepare ...
│   │   │   │   ├─Setting up test repo ...
│   │   │   │   ├─Calling commit ...
│   │   │   │   │   └─Done. (0.009s)
│   │   │   │   ├─Creating test patch ...
│   │   │   │   ├─Calling commit ...
│   │   │   │   │   └─Done. (0.004s)
│   │   │   │   ├─Resetting repo ...
│   │   │   │   └─Done. (0.126s)
│   │   │   ├─Cloning repo ...
│   │   │   ├─Make repo dirty ...
│   │   │   ├─Add dirty commit ...
│   │   │   ├─Calling commit ...
│   │   │   │   └─Done. (0.004s)
│   │   │   └─Done. (0.321s)
[...]
│   │   ├─Calling selftest_tc_build_toolchain ...
│   │   │   ├─Creating dummy toolchain ...
│   │   │   ├─Attempt using it ...
│   │   │   └─Done. (0.153s)
│   │   ├─────────────────────────────────────────
│   │   │ Success: 6/6 tests passed
│   │   └─Done. (3.679s)
│   └─Done. (3.764s)
├─────────────────────────────────────────
└─SUCCESS (3.894s)

tbot also allows you to run multiple testcases at once:

bash-4.4$ tbot selftest selftest_tc
tbot starting ...
├─Calling selftest ...
│   ├─Calling testsuite ...
[...]
│   │   ├─────────────────────────────────────────
│   │   │ Success: 17/17 tests passed
│   │   └─Done. (4.788s)
│   └─Done. (4.873s)
├─Calling selftest_tc ...
│   ├─Calling testsuite ...
[...]
│   │   ├─────────────────────────────────────────
│   │   │ Success: 6/6 tests passed
│   │   └─Done. (3.390s)
│   └─Done. (3.459s)
├─────────────────────────────────────────
└─SUCCESS (8.453s)

If you want an overview of the available testcases, use this command:

$ tbot --list-testcases

The output you saw during the testcase runs was just a rough overview of what is going on. That might not be detailed enough for you. By adding -v, tbot will show all commands as they are executed. Add another one: -vv and you will also see command outputs!

bash-4.4$ tbot selftest_path_stat -vv
tbot starting ...
├─Calling selftest_path_stat ...
│   ├─Setting up test files ...
[...]
│   ├─[local] test -S /tmp/tbot-wd/nonexistent
│   ├─Checking stat results ...
│   ├─[local] stat -t /dev
│   │    ## /dev 4060 0 41ed 0 0 6 1025 20 0 0 1547723442 1547715500 1547715500 0 4096 system_u:object_r:device_t:s0
[...]
│   └─Done. (0.145s)
├─────────────────────────────────────────
└─SUCCESS (0.278s)

Note

There is one more verbosity level: -vvv. This is for debugging, if something doesn’t quite work. It shows you all communication happening, both directions. Try it if you want to, but be prepared: It will look quite messy!

One more commandline feature before we dive into python code: If you are afraid of a destructive command, you can run tbot with --interactive:

bash-4.4$ tbot selftest_uname -vi
tbot starting ...
├─Calling selftest_uname ...
│   ├─[local] uname -a
OK [Y/n]? Y
│   └─Done. (2.721s)
├─────────────────────────────────────────
└─SUCCESS (2.848s)

Now tbot will kindly ask you before running each command! (See? -emacs wouldn’t answer as nicely!)

Testcases

Ok, commandline isn’t all that fun. Let’s dive deeper! Some code please!

1
2
3
4
5
import tbot

@tbot.testcase
def hello_world():
    tbot.log.message("Hello World!")

This is tbot’s hello world. Stick this code into a file named tc.py. Now, if you check the list of testcases (tbot --list-testcases), hello_world pops up. Run it!

bash-4.4$ tbot hello_world
tbot starting ...
├─Calling hello_world ...
│   ├─Hello World!
│   └─Done. (0.000s)
├─────────────────────────────────────────
└─SUCCESS (0.127s)

Hello tbot!

Note

I am sure at least one person reading this will be offended by being told how to name their file. Why tc.py? I prefer calling it my_most_amazing_testcases.py!

Fear not, you can do just that! You just need to tell tbot about it. Instead of the above command, run:

$ tbot -t my_most_amazing_testcases.py hello_world

You can also include all python files in a directory with -T.

Well, before writing actual tests, I need to explain a few things: In tbot, testcases are basically python functions. This means you can call them just like python functions! From other testcases! How about the following?

1
2
3
4
5
6
7
8
9
import tbot

@tbot.testcase
def greet(name: str) -> None:
    tbot.log.message(f"Hello {name}!!")

@tbot.testcase
def greet_tbot() -> None:
    greet("tbot")

If you now call greet_tbot, you can see in the output that it calls greet.

But wait! If you try calling greet directly, it fails! Of course, because greet has a parameter. As previously mentioned, testcases are python functions, so naturally, they can also have parameters. There are two ways to “fix” this:

  1. Specifying a default value for the parameter:

    1
2
3
4
5
    import tbot

@tbot.testcase
def greet(name: str = "World") -> None:
    tbot.log.message(f"Hello {name}!!")

  2. Setting a value for the parameter! That’s right, you can set the parameter from the commandline. It looks like this:

    bash-4.4$ tbot greet -p name=\"tbot\"
tbot starting ...
├─Parameters:
│     name       = 'tbot'
├─Calling greet ...
│   ├─Hello tbot!!
│   └─Done. (0.000s)
├─────────────────────────────────────────
└─SUCCESS (0.238s)

    Note the escaped quotes around \"tbot\". They are necessary because the value is eval()-uated internally. This is done to allow you to set values of any type with ease. Any python expression goes! (Also evil ones, be careful …)

As you’ll see later on, there are cases where you should have default values and ones where it doesn’t make sense. You’ll have to decide individually …

One more thing: You’d expect a testcase to somehow be able to show whether it succeeded. In tbot, a testcase that returns normally passes and one that raises an Exception has failed. This is pretty convenient: You can easily catch failures by using a try-block and your testcases will also automatically fail if anything unexpected happens.

Machines

Next up: Machines! Machines are what tbot is made for. Let’s take a look at the diagram from the landing page again:

_images/tbot.svg

Lab-host? It’s a machine! Buildhost? Just as well! The boards you are testing? You guessed it!

Let’s start simple though: Just run a command on the lab-host:

1
2
3
4
5
6
7
8
import tbot

@tbot.testcase
def greet_user() -> None:
    with tbot.acquire_lab() as lh:
        name = lh.exec0("id", "--user", "--name").strip()

        tbot.log.message(f"Hello {name}!")

Now try:

bash-4.4$ tbot greet_user -v
tbot starting ...
├─Calling greet_user ...
│   ├─[local] id --user --name
│   ├─Hello hws!
│   └─Done. (0.070s)
├─────────────────────────────────────────
└─SUCCESS (0.173s)

As you can see, tbot ran id --user --name to find your name. You might be curious about the [local] part: That’s the machine tbot ran the command on. By default, the lab-host is your localhost. We’ll see later how to change that.

There are quite a few new things in the sample above. Let’s go through them one by one:

  • tbot.acquire_lab(): This is a function provided by tbot that returns the selected lab-host.

  • with tbot.acquire_lab() as lh:: Each machine is a context manager. To get access, you need to enter its context and as soon as you leave it the connection is destroyed. If you haven’t heard about context managers before, take a look at Python with Context Managers. They are really useful!

  • lh.exec0(): This is a function to run a command. Specifically exec-utes it and checks whether the return value is 0. There are also others, for example, lh.test() which returns True if the command succeeded and False otherwise.

  • All command executing methods take one parameter per commandline argument. Each one will be properly escaped: lh.exec0("echo", "!?#;>&<") would print !?#;>&<, no manual quoting needed!

  • lh.exec0() returns a string which I call .strip() on. The reason is that most commands include a trailing newline (\n). I don’t want that in the name so I remove it.

Machines have quite an extensive set of functionality that is definitely worth checking out. Link is here:

Todo

Machine docs

One more feature I want to mention in this quick guide: Most machines have an interactive() method. This method will connect the channel to the terminal and allows you to directly enter commands. You can use it to make tbot do some work, then do something manually. Like a symbiotic development process. It really makes you a lot more productive if you embrace this idea! There is also a testcase to call it from the commandline:

bash-4.4$ tbot interactive_lab
tbot starting ...
├─Calling interactive_lab ...
│   ├─Entering interactive shell ...

local: /tmp> whoami
hws
local: /tmp> exit

│   ├─Exiting interactive shell ...
│   └─Done. (49.746s)
├─────────────────────────────────────────
└─SUCCESS (49.851s)

Configuration

Up until now we did everything on our localhost. That’s boring! tbot allows you to easily use a lab-host that you can connect to via SSH for example. To do that you have to write a small config file. There’s a twist though! The config file is actually a python module. In this module, you create a class for your lab-host. If you have some special features on your lab-host you can add them in there just as well!

The simplest config looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
import tbot
from tbot.machine import linux

class AwesomeLab(linux.lab.SSHLabHost):
    name = "awesome-lab"
    hostname = "awesome.lab.com"

    @property
    def workdir(self):
        return linux.Workdir.athome(self, "tbot-workdir")


# tbot will check for `LAB`, don't forget to set it!
LAB = AwesomeLab

Of course, you’ll have to adjust this a little. tbot will try to connect to the host hostname. It will query ~/.ssh/config for a username and key. (You need to be able to connect to hostname without a password!)

Try using your config now!

$ tbot -l <name-of-lab-config>.py interactive_lab

Congratulations! You now have a remote session on your lab-host! You could also run some selftest to verify that tbot can run these commands on your new lab-host as well:

bash-4.4$ tbot -l lab.py selftest_path_integrity -vv
tbot starting ...
├─Calling selftest_path_integrity ...
│   ├─Logging in on hws@78.79.32.85:22 ...
│   ├─[awesome-lab] echo ${HOME}
│   │    ## /home/hws
│   ├─[awesome-lab] test -d /home/hws/tbot-workdir
│   ├─[awesome-lab] mkdir -p /home/hws/tbot-workdir
│   ├─Logging in on hws@78.79.32.85:22 ...
│   ├─[awesome-lab] mkdir -p /home/hws/tbot-workdir/folder
│   ├─[awesome-lab] test -d /home/hws/tbot-workdir/folder
│   ├─[awesome-lab] uname -a >/home/hws/tbot-workdir/folder/file.txt
│   ├─[awesome-lab] test -f /home/hws/tbot-workdir/folder/file.txt
│   ├─[awesome-lab] rm -r /home/hws/tbot-workdir/folder
│   ├─[awesome-lab] test -e /home/hws/tbot-workdir/folder/file.txt
│   ├─[awesome-lab] test -e /home/hws/tbot-workdir/folder
│   └─Done. (2.833s)
├─────────────────────────────────────────
└─SUCCESS (2.959s)

As you can see, now it says [awesome-lab] in front of the commands. tbot is running commands remotely!

This was just a simple example … Configs can get a lot bigger and a lot more complex. Take a look at their docs for more info!

Hardware Interaction

We haven’t even talked to actual hardware yet! Let’s change that. Unfortunately, as each device is different, you’ll have to figure out a few things yourself.

First Step: Another config file. The board needs to be configured in a second file. Let’s start simple:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
import tbot
from tbot.machine import board, channel, linux

class SomeBoard(board.Board):
    name = "some-board"

    def poweron(self) -> None:
        """Procedure to turn power on."""

        # You can access the labhost as `self.lh`
        # In this case I have a simple command to toggle power.
        self.lh.exec0("remote_power", "bbb", "on")

        # If you can't automatically toggle power,
        # you have to insert some marker here that reminds you
        # to manually toggle power.  How about:
        tbot.log.message("Turn power on now!")

    def poweroff(self) -> None:
        """Procedure to turn power off."""
        self.lh.exec0("remote_power", "bbb", "off")

    def connect(self) -> channel.Channel:
        """Connect to the boards serial interface."""

        # `lh.new_channel` creates a new channel and runs the
        # given command to connect.  Your command should just
        # connect its tty to the serial console like rlogin,
        # telnet, picocom or kermit do.  The minicom behavior
        # will not work.
        return self.lh.new_channel("connect", "bbb")

# tbot will check for `BOARD`, don't forget to set it!
BOARD = SomeBoard

If you did everything correctly, this should be enough to get a serial connection running. Try this:

$ tbot -l lab.py -b my-board.py interactive_board -vv

You should see the board starting to boot. If not, go back and check manually if the commands by themselves work.

Next up we will add config for the Linux running on the board. I’ll skip U-Boot in this quick guide for simplicity. Here’s the full new config:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
import tbot
from tbot.machine import board, channel, linux

class SomeBoard(board.Board):
    name = "some-board"

    def poweron(self) -> None:
        self.lh.exec0("remote_power", "bbb", "on")

    def poweroff(self) -> None:
        self.lh.exec0("remote_power", "bbb", "off")

    def connect(self) -> channel.Channel:
        return self.lh.new_channel("connect", "bbb")

# Linux machine
# We use a `LinuxStandaloneMachine` in this case, because we
# do not care about U-Boot.
class SomeBoardLinux(board.LinuxStandaloneMachine[SomeBoard]):
    # Username for logging in once linux has booted
    username = "root"
    # Password.  If you don't need a password, set this to `None`
    password = "~ysu0dbi"
    # Specifying the shell type is really important!  Else you will
    # see weird things happening.  Login manually once to find out
    # which shell you are running and then set it here.
    shell = linux.shell.Ash

BOARD = SomeBoard
# You need to set `LINUX` now as well.
LINUX = SomeBoardLinux

Again, adjust it as necessary. If you are unsure about some parameters, you can check in the interactive_board session.

If you set everything correctly, you should be able to run:

$ tbot -l lab.py -b my-board.py interactive_linux -vv

You now have a shell on the board! As before, you can also try running a selftest:

$ tbot -l lab.py -b my-board.py selftest_board_linux -vv

bash-4.4$ tbot -l lab.py -b my-board.py selftest_board_linux -vv
tbot starting ...
├─Calling selftest_board_linux ...
│   ├─Logging in on hws@78.79.32.85:22 ...
│   ├─[awesome-lab] connect bbb
│   ├─[awesome-lab] remote_power bbb -l
│   │    ## bbb             off
│   ├─POWERON (bbb)
│   ├─[awesome-lab] remote_power bbb on
│   │    ## Power on   bbb: OK
│   ├─UBOOT (bbb-uboot)
│   │    <>
│   │    <> U-Boot 2018.11-00191-gd73d81fd85 (Nov 20 2018 - 06:01:01 +0100)
│   │    <>
│   │    <> CPU  : AM335X-GP rev 2.1
│   │    <> Model: TI AM335x BeagleBone Black
│   │    <> DRAM:  512 MiB
│   │    <> NAND:  0 MiB
│   │    <> MMC:   OMAP SD/MMC: 0, OMAP SD/MMC: 1
│   │    <> Loading Environment from FAT... ** No partition table - mmc 0 **
│   │    <> No USB device found
│   │    <> <ethaddr> not set. Validating first E-fuse MAC
│   │    <> Net:   eth0: ethernet@4a100000
│   ├─LINUX (bbb-linux)
│   ├─[bbb-uboot] setenv serverip 192.168.1.1
│   ├─[bbb-uboot] setenv netmask 255.255.255.0
│   ├─[bbb-uboot] setenv ipaddr 192.168.1.10
│   ├─[bbb-uboot] mw 0x81000000 0 0x4000
│   ├─[bbb-uboot] setenv rootpath /opt/core-image-lsb-sdk-generic-armv7a-hf
│   ├─[bbb-uboot] run netnfsboot
│   │    <> Booting from network ... with nfsargs ...
│   │    <> link up on port 0, speed 100, full duplex
│   │    <> TFTP from server 192.168.1.1; our IP address is 192.168.1.10
│   │    <> Load address: 0x82000000
│   │    <> Loading: #################################################################
│   │    <>    ########################
│   │    <>    2.9 MiB/s
│   │    <> done
│   │    <> Bytes transferred = 9883000 (96cd78 hex)
│   │    <> link up on port 0, speed 100, full duplex
│   │    <> TFTP from server 192.168.1.1; our IP address is 192.168.1.10
│   │    <> Load address: 0x88000000
│   │    <> Loading: #####
│   │    <>    1.1 MiB/s
│   │    <> done
│   │    <> Bytes transferred = 64051 (fa33 hex)
│   │    <> ## Flattened Device Tree blob at 88000000
│   │    <>    Booting using the fdt blob at 0x88000000
│   │    <>    Loading Device Tree to 8ffed000, end 8ffffa32 ... OK
│   │    <>
│   │    <> Starting kernel ...
│   │    <>
│   │    <> [    0.000000] Booting Linux on physical CPU 0x0
│   │    <> [    0.000000] Linux version 4.9.126 (build@denx) (gcc version 7.2.1 20171011 (Linaro GCC 7.2-2017.11) ) #1 SMP PREEMPT Wed Dec 12 03:12:29 CET 2018
│   │    <> [    0.000000] CPU: ARMv7 Processor [413fc082] revision 2 (ARMv7), cr=10c5387d
│   │    <> [    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache                                              Hello there ;)
│   │    <> [    0.000000] OF: fdt:Machine model: TI AM335x BeagleBone Black
│   │    <> [    0.000000] efi: Getting EFI parameters from FDT:
│   │    <> [    0.000000] efi: UEFI not found.
│   │    <> [    0.000000] cma: Reserved 48 MiB at 0x9c800000
[...]
│   │    <> Poky (Yocto Project Reference Distro) 2.4 generic-armv7a-hf /dev/ttyS0
│   │    <>
│   │    <> generic-armv7a-hf login: root
│   ├─Calling selftest_machine_shell ...
│   │   ├─Testing command output ...
│   │   ├─[bbb-linux] echo 'Hello World'
│   │   │    ## Hello World
│   │   ├─[bbb-linux] echo '$?' '!#'
│   │   │    ## $? !#
[...]
│   │   └─Done. (3.355s)
│   ├─POWEROFF (bbb)
│   ├─[pollux] remote_power bbb off
│   │    ## Power off  bbb: OK
│   └─Done. (44.150s)
├─────────────────────────────────────────
└─SUCCESS (44.624s)

Nice!

Last part of this guide will be interacting with the board from a testcase. It’s pretty straight forward:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
import tbot

@tbot.testcase
def test_board() -> None:
    # Get access to the lab-host as before
    with tbot.acquire_lab() as lh:
        # This context is for the "hardware".  Once you enter
        # it, the board will be powered on and as soon as
        # you exit it, it will be turned off again.
        with tbot.acquire_board(lh) as b:
            # This is the context for the "LinuxMachine".
            # Entering it means tbot will listen to the
            # board booting and give you a machine handle
            # as soon as the shell is available.
            with tbot.acquire_linux(b) as lnx:
                lnx.exec0("uname", "-a")

Those two additional indentation levels aren’t nice - We can refactor the code to look like this (I showed the explicit version first so you can see what is going on):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
import contextlib
import tbot

@tbot.testcase
def test_board() -> None:
    with contextlib.ExitStack() as cx:
        lh  = cx.enter_context(tbot.acquire_lab())
        b   = cx.enter_context(tbot.acquire_board(lh))
        lnx = cx.enter_context(tbot.acquire_linux(b))

        lnx.exec0("uname", "-a")

There is still one issue with this design: Let’s pretend this is a test to check some board functionality. Maybe you have quite a few testcases that each check different parts. Now, we want to call all of them from some “master” test, so we can test everything at once.

The issue we will run into is that each testcase will A) reconnect to the lab-host and B) powercycle the board. This will be very very slow! We can do better!

The idea is that testcases take the lab and board as optional parameters. This allows reusing the old connection and won’t powercycle the board for each test (if you need powercycling, you can of course do it like above):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
import contextlib
import typing
import tbot
from tbot.machine import linux, board

@tbot.testcase
def test_board(
    lab: typing.Optional[linux.LabHost] = None,
    board_linux: typing.Optional[board.LinuxMachine] = None,
) -> None:
    with contextlib.ExitStack() as cx:
        if board_linux is None:
            lh  = cx.enter_context(lab or tbot.acquire_lab())
            b   = cx.enter_context(tbot.acquire_board(lh))
            lnx = cx.enter_context(tbot.acquire_linux(b))
        else:
            lnx = board_linux

        lnx.exec0("uname", "-a")

@tbot.testcase
def call_it() -> None:
    with tbot.acquire_lab() as lh:
        test_board(lh)

You can still call test_board from the commandline, but call_it works as well!

You will probably need this pattern quite a lot. I have compiled a page of this and similar patterns that you can easily copy to your code: Recipes

That’s it for the quick-start. I hope I got you hooked! The next step is to look deeper into each individual part. Docs are here:

  • TODO - In depth docs for machines, paths and a lot more.

  • Configuration - Everything you need to know about tbot’s configuration.

  • Recipes - As mentioned above, a list of “testcase templates”.

  • Logging - I didn’t mention in this quick-start guide, but tbot as extensive logging facilities!

  • Building Projects - tbot has some helpers for compiling code on a machine called “build-host”.

  • tbot Module - API Reference (also Machines, Linux and Board).

  • tbot.tc Module - tbot’s builtin testcases