leandrof/u-boot
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

tools/tbot: update README

Browse Source 
refer in the README to tbots webpage, and delete
the README in tools/tbot, as the latest documentation
for tbot is on this webpage.

Signed-off-by: Heiko Schocher <hs@denx.de>
This commit is contained in:
Heiko Schocher 2017-06-09 06:13:34 +02:00 committed by Tom Rini
parent f8dbc0734d
commit 630dfede22
5 changed files with 1 additions and 685 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
test/README
View File

@ -46,7 +46,7 @@ tbot
Tbot provides a way to execute tests on target hardware. It is intended for
trying out both U-Boot and Linux (and potentially other software) on a
number of boards automatically. It can be used to create a continuous test
environment. See tools/tbot/README for more information.
environment. See http://www.tbot.tools for more information.
Ad-hoc tests

195
tools/tbot/README
View File

@ -1,195 +0,0 @@
# Copyright (c) 2016 DENX Software Engineering GmbH
# Heiko Schocher <hs@denx.de>
#
# SPDX-License-Identifier: GPL-2.0+
#
What is tbot ?
==============
tbot is a tool for executing testcases on boards.
Source code found on [1]
Based on DUTS [2]
written in python
Basic Ideas of tbot
===================
(see also the figure:
https://github.com/hsdenx/tbot/blob/master/doc/tbot_structure.png )
- Virtual laboratory (VL)
VL is the basic environment that groups:
- [a number of] boards - target devices on which tbot executes testcases.
- one Lab PC
- Test case (TC):
A piece of python code, which uses the tbot class from [1].
Tbot provides functions for sending shell commands and parsing the
shell commands output.
Tbot waits endless for a shell commands end (detected through reading
the consoles prompt).
A TC can also call other TC-es.
remark:
Tbot not really waits endless, for a shell commands end, instead
tbot starts a watchdog in the background, and if it triggers, tbot
ends the TC as failed. In the tbot beginning there was a lot of
timeouts / retry cases, but it turned out, that waiting endless
is robust and easy ...
- Host PC (where tbot runs, currently only linux host tested)
must not a powerful machine (For example [3], I use a
raspberry pi for running tbot and buildbot)
- Lab PC:
- Host PC connects through ssh to the Lab PC
-> so it is possible to test boards, which
are not at the same place as the Host PC.
(Lab PC and Host PC can be the same of course)
-> maybe we can setup a Testsystem, which does nightly
U-Boot/Linux builds and test from current mainline U-Boot
on boards wherever they are accessible.
- necessary tasks a Lab PC must deliver:
- connect to boards console through a shell command.
- power on/off boards through a shell command
- detect the current power state of a board through
a shell command
- optional tasks:
- tftp server (for example loading images)
- nfs server (used as rootfs for linux kernels)
- Internet access for example for downloading
U-Boot source with git.
- toolchains installed for compiling source code
-> a linux machine is preffered.
- currently only Lab PC with an installed linux supported/tested.
- Boards(s):
the boards on which shell commands are executed.
- Board state:
equals to the software, the board is currently running.
Currently tbot supports 2 board states:
- "u-boot", if the board is running U-Boot
- "linux", if the board is running a linux kernel
It should be easy to add other board states to tbot, see
https://github.com/hsdenx/tbot/tree/master/src/lab_api/state_[u-boot/linux].py
A board state is detected through analysing the boards
shell prompt. In linux, tbot sets a special tbot prompt,
in U-Boot the prompt is static, and configurable in tbot through
a board config file.
A TC can say in which board state it want to send shell commands.
Tbot tries to detect the current board state, if board is not in
the requested board state, tbot tries to switch into the correct
state. If this fails, the TC fails.
It is possible to switch in a single TC between board states.
- Events
tbot creates while executing testcases so called events.
After tbot ended with the testcase it can call event_backends,
which convert the events to different formats. more info:
https://github.com/hsdenx/tbot/blob/master/doc/README.event
demo for a event backend:
http://xeidos.ddns.net/tests/test_db_auslesen.php
- tbot cmdline parameters:
$ python2.7 src/common/tbot.py --help
Usage: tbot.py [options]
Options:
-h, --help show this help message and exit
-c CFGFILE, --cfgfile=CFGFILE
the tbot common configfilename
-l LOGFILE, --logfile=LOGFILE
the tbot logfilename, if default, tbot creates a
defaultnamelogfile
-t TC, --testcase=TC the testcase which should be run
-v, --verbose be verbose, print all read/write to stdout
-w WORKDIR, --workdir=WORKDIR
set workdir, default os.getcwd()
$
tbot needs the following files for proper execution:
- tbot board configuration file (option -c):
A board configuration file contains settings tbot needs to
connect to the Lab PC and board specific variable settings
for testcases.
- name of the logfile tbot creates (option -l)
defaultname: 'log/' + now.strftime("%Y-%m-%d-%H-%M") + '.log'
- tbots working directory (option -w)
- the testcasename tbot executes (option -t)
You are interested and want to use tbot?
If so, please read on the file:
tools/tbot/README.install
If not read [3] ;-)
Heiko Schocher <hs@denx.de>
v1 2016.01.22
--------------
[1] https://github.com/hsdenx/tbot
[2] http://www.denx.de/wiki/DUTS/DUTSDocs
[3] automated Testsetup with buildbot and tbot doing cyclic tests
(buildbot used for starting tbot TC and web presentation of the
results, all testing done through tbot):
http://xeidos.ddns.net/buildbot/tgrid
Host PC in Letkes/hungary
VL in munich/germany
Fancy things are done here, for example:
- http://xeidos.ddns.net/buildbot/builders/smartweb_dfu/builds/43/steps/shell/logs/tbotlog
(I try to cleanup the logfile soon, so it is not so filled with crap ;-)
A first step see here:
http://xeidos.ddns.net/buildbot/builders/smartweb_dfu/builds/45/steps/shell/logs/tbotlog
(same TC now with the new loglevel = 'CON' ... not yet perfect)
Executed steps:
- clone u-boot.git
- set toolchain
- get a list of patchwork patches from my U-Boots ToDo list
- download all of them, and check them with checkpatch
and apply them to u-boot.git
- compile U-Boot for the smartweb board
- install the resulting images on the smartweb board
- boot U-boot
- test DFU
- more TC should be added here for testing U-Boot
- automatic "git bisect"
https://github.com/hsdenx/tbot/blob/master/src/tc/tc_board_git_bisect.py
http://xeidos.ddns.net/buildbot/builders/tqm5200s/builds/3/steps/shell/logs/tbotlog
If a current U-Boot image not works on the tqm5200 board
this TC can be started. It starts a "git bisect" session,
and compiles for each step U-Boot, install it on the tqm5200
board, and tests if U-Boot works !
At the end, it detects the commit, which breaks the board
This TC is not dependend on U-Boot nor on a special board. It
needs only 3 variables:
tb.board_git_bisect_get_source_tc: TC which gets the source tree, in which
"git bisect" should be executed
tb.board_git_bisect_call_tc: TC which gets called every "git bisect" step,
which executes commands for detecting if current source code is OK or not.
This could be a TC which compiles U-Boot, install it on the board and
executes TC on the new booted U-Boot image. ! Board maybe gets borken,
as not all U-Boot images work, so you must have a TC which install U-Boot
image for example through a debugger.
tb.board_git_bisect_good_commit: last nown good commit id

62
tools/tbot/README-ToDo
View File

@ -1,62 +0,0 @@
# Copyright (c) 2016 DENX Software Engineering GmbH
# Heiko Schocher <hs@denx.de>
#
# SPDX-License-Identifier: GPL-2.0+
#
ToDo list for tbot
==================
please look also into the tbot ToDo list.
https://github.com/hsdenx/tbot/blob/master/ToDo
- cleanup tbot code:
- remove all retry / timeout pieces of code
- clean up tbot function names, as I am not good in
giving function a understandable name ;-)
- as I am not a python programmer, cleanup whole tbot code
- introduce a "layering" like yocto do, so U-Boot TC can integrated
into U-Boot source code.
Proposal:
introduce subdirs in "src/tc"
lab: all lab specific stuff
lab/common: common lab stuff (for example ssh handling)
lab/ssh_std: ssh_std specific stuff
u-boot: all u-boot tests
u-boot/common: common u-boot tc
u-boot/duts: DUTS tc
u-boot-dxr2: all u-boot dxr2 board specific tc
board: board tc
board/common: common board tc
board/dxr2: all tc for dxr2 board
linux: all linux tc
linux/common: common linux tc
linux/dxr2
- move U-Boot special TC to U-Boot source
-> need a mechanism in tbot, how it gets automatically for example
U-Boot TC from U-Boot source...
-> add a consistency checker
- simplify tbot log output (seperate a lot of output which is currently
in INFO logging level, to another logging level)
started (new loglevel "CON", whih prints read/write from console only), see:
https://github.com/hsdenx/tbot/commit/b4ab2567ad8c19ad53f785203159d3c8465a21c6
- make the timestamp configurable
- Open more than 2 filehandles ?
Do we need for more complex TC more than 2 filehandles?
- Find a way to document all TC and document all variables they use in an
automated way.
- write a lot of more TC
- get U-Boot configuration settings from current U-Boot code and use
them in U-Boot TC-es

117
tools/tbot/README.create_a_new_testcase
View File

@ -1,117 +0,0 @@
# Copyright (c) 2016 DENX Software Engineering GmbH
# Heiko Schocher <hs@denx.de>
#
# SPDX-License-Identifier: GPL-2.0+
#
write a new testcase
=====================
A TC is written in python, so you can use python as usual. For accessing
the boards console, use functions from the tbotlib, therefore
First import the tbotlib with the line:
from tbotlib import tbot
If your TC uses variables, please add a line which adds them to
the log file (for debugging purposes):
logging.info("args: %s ...", tb.varname, ...)
Say tbot, for which board state your TC is valid with:
tb.set_board_state("u-boot")
Then you are ready ... and you can use the tbotlib funtions
for writting/reading to the boards console.
Big fat warning:
A TC must worry about to end only if a board has finished the shell
command!
Not following this rule, will end in unpredictable behaviour.
(hopefully) useful tbotlib functions
====================================
- set the board state, you want to test
tb.set_board_state(state)
states are: "u-boot" or "linux"
If tbot could not set the board state, tbot ends with failure.
- write a command to the boards console:
tb.eof_write_con(command):
write the command to the boards console. If this
fails, tbot ends with failure
- write a command to boards console and wait for prompt:
tb.eof_write_cmd(fd, command):
fd: filedescriptor which is used, use tb.channel_con for boards console
command: command which is written to fd
Wait endless for board prompt
- write a list of commands to boards console:
tb.eof_write_cmd_list(fd, cmdlist):
fd: filedescriptor which is used, use tb.channel_con for boards console
cmdlist: python list of commandstrings which is written to fd
- wait for boards prompt:
tb.eof_read_end_state_con(retry):
retry: deprecated, not used anymore, cleanup needed here...
tbot waits endless for the boards prompt
- write a command, wait for prompt and check, if a string is read
tb.write_cmd_check(fd, cmd, string):
fd: filedescriptor which is used, use tb.channel_con for boards console
cmd: command, which is send to fd
string: string which should be read from fd
return value:
True, if string is read and tbot got back boards prompt
False, else
tb.eof_write_cmd_check(fd, cmd, string):
same as tb.write_cmd_check(fd, cmd, string) except, that tbot
ends immediately with Failure, if string is not read.
- read until prompt and search strings:
tb.readline_and_search_strings(fd, strings):
fd: filedescriptor which is used, use tb.channel_con for boards console
strings: python list of strings, which can be read
If one of this strings is read, this function return the index, which
string is read. This function shoud be called in a while loop,
until this function returns 'prompt'
- read a line from filedescriptor:
not recommended to use, as the TC must check, if tprompt is read for every
readen line. Also TC must ensure, that it ends only, if prompt is read.
tb.read_line(fd, retry)
fd: filedescriptor which is used, use tb.channel_con for boards console
retry: retry of trying to reead a line
return values:
True, if a line is read. Readen line in tb.buf[fd]
False, if something read, but not a complete line
None, if nothing is read
check if string contains prompt with:
tb.is_end_fd(fd, string)
fd: filedescriptor which is used, use tb.channel_con for boards console
string: buffer, in which a prompt gets searched.
- calling other TC:
eof_call_tc(name):
call another TC from "src/tc"
if the called TC fails with failure, tbot ends with failure
call_tc(name):
call another TC from "src/tc"
if the TC which call_tc calls fails, call_tc() returns False, else True
There are more functions, but for writting TC this should be enough. But
its software, so new useful functions can always pop up.
Heiko Schocher <hs@denx.de>
v1 2016.01.23

310
tools/tbot/README.install
View File

@ -1,310 +0,0 @@
# Copyright (c) 2016 DENX Software Engineering GmbH
# Heiko Schocher <hs@denx.de>
#
# SPDX-License-Identifier: GPL-2.0+
#
install tbot on your PC (linux only tested):
============================================
- get the source code:
$ git clone https://github.com/hsdenx/tbot.git
[...]
$
cd into the tbot directory.
- you need the for running tbot the python paramiko module, see:
http://www.paramiko.org/installing.html
paramiko is used for handling ssh sessions, and open filedescriptors
on a ssh connection. Tbot open a ssh connection to a "lab PC" and
opens on that connection 2 filehandles, one for control functions
and one for the connection to the boards console. May it is worth
to think about to open more filehandles and use them in tbot, but
thats a point in the Todo list ...
See [1] for more infos about tbot principles.
- prepare a directory for storing the logfiles
and pass it with the commandline option "-l"
to tbot. Default is the directory "log" in the tbot
root (don;t forget to create it, if you want to use it)
- If your VL is not yet in tbot source, integrate it
(This task has only to be done once for your VL):
A VL has, as described in [2] "necessary tasks for a Lab PC" explained,
3 tasks:
a) power on/off the board
b) get power state of the board
c) connect to the boards console
As tbot sends only shell commands (also to the Lab PC)
this tasks must be executable through shell commands on your
Lab PC:
Task a) power on/off board:
default TC for this task is:
https://github.com/hsdenx/tbot/blob/master/src/tc/tc_lab_denx_power.py
- now copy this file to for example
cp src/tc/tc_lab_denx_power.py src/tc/tc_lab_denx_power_XXX.py
(replace XXX to a proper value)
and adapt the "remote_power" command from the denx lab to your needs.
As this TC powers on the board for all your boards in your VL,
you can differ between the boards through the tbot class
variable "tb.boardlabpowername" (which is in the default case the
same as "tb.boardname"), but you may need to name the power target
with an other name than boardname, so you can configure this case.
The power state "tb.power_state" which the TC has to set
is "on" for power on, or "off" for power off.
If switching on the power is successful, call "tb.end_tc(True)"
else "tb.end_tc(False)"
- set in your board config file:
self.tc_lab_denx_power_tc = 'tc_lab_denx_power_XXX.py'
Task b) power on/off board:
default TC for this task is:
https://github.com/hsdenx/tbot/blob/master/src/tc/tc_lab_denx_get_power_state.py
- now copy this file to for example
(replace XXX to a proper value)
cp src/tc/tc_lab_denx_get_power_state.py src/tc/tc_lab_denx_get_power_state_XXX.py
and adapt the commands to your needs.
If the power of the board is on, call "tb.end_tc(True)"
else "tb.end_tc(False)"
- set in your board config file:
self.tc_lab_denx_get_power_state_tc = 'tc_lab_denx_get_power_state_XXX.py'
Task c) connect to the boards console:
default TC for this task is:
https://github.com/hsdenx/tbot/blob/master/src/tc/tc_lab_denx_connect_to_board.py
- now copy this file to for example
(replace XXX to a proper value)
cp src/tc/tc_lab_denx_connect_to_board.py src/tc/tc_lab_denx_connect_to_board_XXX.py
and adapt the commands to your needs.
If connect fails end this TC with "tb.end_tc(False)"
else call "tb.end_tc(True)"
If you want to use kermit for connecting to the boards console, you
can use:
https://github.com/hsdenx/tbot/blob/master/src/tc/tc_workfd_connect_with_kermit.py
Example for such a board in the VL from denx:
self.tc_lab_denx_connect_to_board_tc = 'tc_workfd_connect_with_kermit.py'
https://github.com/hsdenx/tbot/blob/master/tbot_dxr2.cfg#L24
Hopefully this works for you too.
- set in your board config file:
self.tc_lab_denx_connect_to_board_tc = 'tc_lab_denx_connect_to_board_XXX.py'
remarks while writting this:
- Currently there is only the denx VL. Original idea was to include
other VL through a seperate class/file in
https://github.com/hsdenx/tbot/tree/master/src/lab_api
but it turned out, that if we say "ssh" is the standard way to connect
to a VL, we can integrate the VL specific tasks through testcases, see
above, so we should do:
- rename the "denx" API to a more general name.
This is a point on my ToDo list ... done, renamed to 'ssh_std'
- the VL specific configuration may moved from the board config files
and should be collected in VL specific config files, which boards
config file simple include.
- prepare password.py file:
This file contains all passwords tbot needs (for example for
linux login on the boards)
tbot searches this file in the tbot root directory.
It is a simple python file, for example:
# passwords for the lab
if (board == 'lab'):
if (user == 'hs'):
password = 'passwordforuserhs'
if (user == 'root'):
password = 'passwordforrootuser'
# passwords for the boards
elif (board == 'mcx'):
if (user == 'root'):
password = 'passwordformcxrootfs'
else:
if (user == 'root'):
password = ''
- prepare board config file
Each board which is found in the VL needs a tbot configuration file
pass the config file name with the option '-c' to tbot, tbot searches
in the root dir for them.
board Example (dxr2 board):
https://github.com/hsdenx/tbot/blob/master/tbot_dxr2.cfg
Necessary variables:
line 3: boardname, here it is the "etamin" board
no default value, must be set.
line 4: boardlabname: name used for connecting to the board
may differ from tb.boardname, default tb.boardname
line 5: boardlabpowername: name used for power on/off
may differ from tb.boardname, default tb.boardname
line 6: tftpboardname: name used for tftp subdir (from where
U-Boot loads images for example).
may differ from tb.boardname, default tb.boardname
line 7: labprompt: linux prompt tbot sets
no defaultvalue, must be set (maybe we should introduce
"ttbott" as default ...
line 8: debug: If True, adds debug output on the tbot shell
line 9: debugstatus: enable status debug output on the shell
line 10: ip: Where tbot finds the Lab PC
line 11: user: As which user does tbot logs into the Lab PC
line 12: accept_all: passed to paramiko, accept all connections
line 13: keepalivetimout: passed to paramiko, timeout for sending
keepalive message.
line 14: channel_timeout: passed to paramiko
line 15: loglevel: tbots loglevel for adding entries into the logfile.
line 17: wdt_timeout: timeout in seconds for tbots watchdog.
Watchdog gets triggered if prompt get read.
line 24: tc_lab_denx_connect_to_board_tc: Which TC is used for
connecting to the boards console the TC, here:
https://github.com/hsdenx/tbot/blob/master/src/tc/tc_workfd_connect_with_kermit.py
line 27: uboot_prompt: boards U-Boot prompt
line 28: linux_prompt: boards linux prompt
Now comes a list of variables TC needs, this vary from which TC
you start on the board.
Thats it ... you now can call tbot and hopefully, it works ;-)
Find an example log [3] for calling simple U-Boot TC for setting
an U-Boot Environmentvariable.
If you have problems in setting tbot up, please contact me
(and may give me ssh access to your Lab PC ;-)
If you have running your first TC [3], you may want to write now your own
TC (and hopefully share them), so continue with:
u-boot:tools/tbot/README.create_a_new_testcase
Heiko Schocher <hs@denx.de>
v2 2016.04.26
--------------
[1] tbot Dokumentation:
[2] u-boot:/tools/tbot/README
https://github.com/hsdenx/tbot/blob/master/README.md
tbot-devel@googlegroups.com
[3] Example for a first U-Boot TC which should always work:
(with commandline option "-v" for verbose output):
hs@localhost:tbot [event-devel] $ python2.7 src/common/tbot.py -c tbot_dxr2.cfg -t tc_ub_setenv.py -v -l log/tbot.log
**** option cfg: tbot_dxr2.cfg log: log/tbot.log tc: tc_ub_setenv.py v 1
('CUR WORK PATH: ', '/home/hs/data/Entwicklung/tbot')
('CFGFILE ', 'tbot_dxr2.cfg')
('LOGFILE ', '/home/hs/data/Entwicklung/tbot/log/tbot.log')
tb_ctrl: Last login: Mon Apr 25 14:52:42 2016 from 87.97.29.27
*************************************************************
BDI2000 Assignment: (last updated: 2015-11-20 12:30 MET)
bdi1 => techem bdi2 => cetec_mx25 bdi3 => lpc3250
bdi4 => - bdi5 => --Rev.B!-- bdi6 => tqm5200s
bdi7 => [stefano] bdi8 => smartweb bdi9 => sigmatek-nand
bdi10 => pcm052 bdi11 => socrates bdi12 => aristainetos
bdi13 => imx53 bdi14 => ib8315 bdi15 => cairo
bdi16 => g2c1 bdi17 => lwe090 bdi18 => symphony
bdi19 => dxr2 bdi20 => ima3-mx6 bdi21 => sama5d3
bdi98 => - bdi99 => - bdi0 => -
Please power off unused systems when you leave! Thanks, wd.
*************************************************************
tb_ctrl: pollux:~ hs $
tb_ctrl: export PS1=ttbott
ttbott
tb_ctrl: stty cols 200
ttbott
tb_ctrl: export TERM=vt200
ttbott
tb_ctrl: echo $COLUMNS
200
ttbott
tb_con: Last login: Tue Apr 26 06:28:59 2016 from 87.97.29.27
*************************************************************
BDI2000 Assignment: (last updated: 2015-11-20 12:30 MET)
bdi1 => techem bdi2 => cetec_mx25 bdi3 => lpc3250
bdi4 => - bdi5 => --Rev.B!-- bdi6 => tqm5200s
bdi7 => [stefano] bdi8 => smartweb bdi9 => sigmatek-nand
bdi10 => pcm052 bdi11 => socrates bdi12 => aristainetos
bdi13 => imx53 bdi14 => ib8315 bdi15 => cairo
bdi16 => g2c1 bdi17 => lwe090 bdi18 => symphony
bdi19 => dxr2 bdi20 => ima3-mx6 bdi21 => sama5d3
bdi98 => - bdi99 => - bdi0 => -
Please power off unused systems when you leave! Thanks, wd.
*************************************************************
tb_con: pollux:~ hs $
tb_con: export PS1=ttbot
tb_con: t
ttbott
tb_con: stty cols 200
ttbott
tb_con: export TERM=vt200
ttbott
tb_con: echo $COLUMNS
200
ttbott
tb_con: ssh hs@lena
tb_con: hs@lena's password:
tb_con:
tb_con: Last login: Mon Apr 25 07:03:29 2016 from 192.168.1.1
tb_con: [hs@lena ~]$
tb_con: export PS1=ttbott
ttbott
tb_con: stty cols 200
ttbott
tb_con: export TERM=vt200
ttbott
tb_con: echo $COLUMNS
200
ttbott
tb_con: kermit
C-Kermit 8.0.211, 10 Apr 2004, for Linux
Copyright (C) 1985, 2004,
Trustees of Columbia University in the City of New York.
Type ? or HELP for help.
(/home/hs/) C-Kermit>
tb_con: set line /dev/ttyUSB0
(/home/hs/) C-Kermit>
tb_con: set speed 115200
/dev/ttyUSB0, 115200 bps
(/home/hs/) C-Kermit>
tb_con: set flow-control none
(/home/hs/) C-Kermit>
tb_con: set carrier-watch off
(/home/hs/) C-Kermit>
tb_con: connect
Connecting to /dev/ttyUSB0, speed 115200
Escape character: Ctrl-\ (ASCII 28, FS): enabled
Type the escape character followed by C to get back,
or followed by ? to see other options.
----------------------------------------------------
tb_con: <INTERRUPT>
U-Boot#
tb_con: U-Boot#
U-Boot#
tb_con: setenv Heiko Schocher
U-Boot#
tb_con: printenv Heiko
Heiko=Schocher
U-Boot#
[('tc_workfd_ssh.py', 1, 0), ('tc_workfd_connect_with_kermit.py', 1, 0), ('tc_ub_setenv.py', 1, 0)]
End of TBOT: success
hs@localhost:tbot [event-devel] $