1 DEVICE DRIVER TESTS (DDT) for embedded linux
2 ================================================================================
4 1) GOALS:
5 ================================================================================
6 * Validate the functionality, performance and stability of Device Drivers
7 on embedded Linux systems.
8 * Validate performance and stability of the whole embedded system.
9 * Maximize use of open source software and contribute to improve it.
10 * Maximize code reuse across different platforms.
11 * Make it easy to support new embedded platforms.
13 2) ARCHITECTURE HIGHLIGHTS:
14 ================================================================================
15 * Based on LTP http://ltp.sourceforge.net/
16 * Support dynamic selection/filtering of test cases based on platform
17 * Support test parameters overrides based on platform
19 3) HOW TO ADD NEW DDT TESTS:
20 ================================================================================
21 You only need to add files to two directories:
22 testcases/ddt: Contains the test logic files (either c code or shell script)
23 runtest/ddt : Contains the test scenario files. It calls the test logic files.
25 * If your tests requires compiling c sources, create following directory
26 testcases/ddt/<your sources dir name>/
27 Add your c sources to the directory created above and make sure that you have
28 a top level Makefile. (Check Makefile of existing directories for examples)
29 Please note that you don't need to add c sources for external tools (such as
30 iperf, lmbench, etc.), instead external tools should be added to the OE/Arago
31 recipe that builds the test filesystem image.
32 * If your tests use shell scripts, add the shell scripts to the
33 testcases/ddt/scripts/ directory
34 * Write a new test scenario file under runtest/ddt/. Also add a new line with
35 the name of your new scenario file to scenario_groups/default-ddt.
37 4) DDT TESTS IMPLEMENTATION GUIDELINES
38 ================================================================================
39 4.1) TEST CASES
40 * Write you test code using c and/or linux shell scripts.
41 * Beware that the linux shell script should be compatible with the ash shell
42 available in busybox.
43 * Avoid hard-coding values that are specific to your platform, instead try to
44 determine the right value to use using the information provided by Linux
45 ( /sys and proc/ are your friends)
46 * When not possible to determine values dynamically, define default values and
47 provide appropriate overrides based on either ARCH, DRIVER, SOC or MACHINE.
48 Try to define the override value at the most generic level possible.
49 ARCH, DRIVER, SOC and MACHINE variables are exported by ltprun script.
50 * Avoid adding logic to define values dynamically in your c code. If the code
51 needs to use different values depending on the platform being tested, then
52 make that value a command line argument and write a shell script wraper
53 that calls your c executable. The shell script wrapper should use the
54 override logic define above to determine the right value to pass to the
55 c executable.
56 * Use the default shell script template available at
57 testcases/ddt/scripts/TEMPLATE as a starting point to develop your script.
58 * Instead of developing huge shell scripts, try to partition the logic into
59 several smaller scripts, each such smaller script should execute some
60 well-defined task and ideally could be used by other people developing
61 tests. Before you start writing scripts, check the available scripts to see
62 if the logic you need is already available. Scripts that return a value are
63 named get_*
64 * If your script return a single value, use echo to return the value. If your
65 script need to return multiple values, consider breaking it into smaller
66 scripts. If absolutely necessary to return multiple values then write the
67 name=value pairs (one per line) to file descriptor 10.
68 See testcases/ddt/scripts/TEMPLATE for an example.
71 4.2) TEST SCENARIOS
72 * Follow LTP guidelines. The test scenario file is basically made of one or
73 more test step lines. Each test step line have following format
74 <TAG> <COMMANDS>, where
75 TAG is a string that identifies the test step.
76 Use following convention to named TAGs so that the test cases can be
77 selectively run based on AREA, SCOPE and/or TYPE.
78 <AREA>_<SCOPE>_<TYPE>_<OPT_ID>,
79 i.e. “NAND_S_FUNC_RW_8K”, “NAND_M_PERF_ALL-SIZES”
80 The SCOPE tags are used to indicate the amount of time require to run
81 the tests, giving users ability to filter test cases based on estimated
82 execution time.
83 SCOPE TAGS:
84 'XS', 'S', 'M', 'L', 'XL', 'XXL' (for eXtra Small, Small, Medium, etc.
85 Just imagine you are buying clothes ;)
86 We used the following rough guidelines to determine test duration based
87 on scope:
88 XS: <= 1 min
89 S: <= 10 mins
90 M: <= 1 hour
91 L: <= 4 hours
92 XL: <= 24 hours
93 XXL: <= 1 week
94 TYPE TAGS:
95 ‘FUNC’, ‘PERF’, ‘STRESS’, ‘USECASE’, ‘COMPLIANCE’, ‘MODULAR’, ‘DOC’
96 COMMANDS is a list of one or more shell commands separated by semicolon (;),
97 the test step will pass if the commands return zero, otherwise it fails.
99 * Use the default test scenario file template available at
100 runtest/ddt/TEMPLATE as a starting point to develop your test scenario.
102 * Use the @requires annotation to specify ARCH, DRIVER, SOC and/or MACHINE
103 requirements to run the test scenario. You can use (), &&, ||, * to specify
104 the test requirements. Examples:
106 @requires /net/eth/* && spi_master
107 To run this test the platform must have an ethernet driver and a
108 spi_master driver
110 @requires am3517-evm
111 This test can only be run on an am3517 EVM.
113 @requires (mmc_host || nand) && armv*
114 This test requires mmc or nand drivers and an ARM architecture
116 * Use the @setup_requires annotation to specify test setup requirements.
117 Some test cases like USB and Video capture requires special peripherals, such as
118 USB flash drives, DVD players, video cameras, etc., to be connected to the DUT.
119 Using @setup_requires the test developer highlights such test setup requirements.
120 This information might be used by test automation frameworks to allocate test requests
121 to DUTs that have the appropriate peripherals connected to them.
122 Please follow the naming conventions identified in section 9) of this document
123 if the test scenario needs to identify any setup requirements.
125 You can use underscore(_) to seperate multiple @setup_requires. Examples:
127 @setup_requires usbhostvideo_usbhostaudio
128 This test requires usbhostvideo setup and usbhostaudio setup.
130 5) HOW TO ADD NEW PLATFORMS
131 ================================================================================
132 * Copy the default platform file available at platforms/TEMPLATE to
133 platforms/<your platform>. <your platform> name is typically the evm name
134 * Please use the following names for platform files and to compare against
135 $MACHINE when writing ltp-ddt scripts:
137 am180x-evm arago-armv7 dm355-evm dm814x-evm
138 am181x-evm dm365-evm dm816x-evm
139 am3517-evm beagleboard dm368-evm
140 am37x-evm c6a814x-evm dm37x-evm omap3evm
141 am387x-evm c6a816x-evm dm6446-evm tnetv107x-evm
142 am389x-evm da830-omapl137-evm dm6467-evm am335x-evm
143 arago-armv5 da850-omapl138-evm dm6467t-evm beaglebone
145 * Modify your platform file based on the capabilities supported by the new evm
146 The platform file identifies the architecture, the SoC, the evm and the
147 supported drivers. The supported drivers lines follow a variation of the
148 hierarchy used in /sys/class but it is not exactly the same. Hence it is
149 important to use the platforms/TEMPLATE file as your starting point.
150 Please note the first 3 lines of the platform file MUST identify, the
151 architecture, SoC and EVM respectively, follow by one or more driver lines.
152 Typically the architecture and machine name used in the platform file are
153 the ones reported by uname -a.
154 Sample platform file:
155 armv7l
156 am3517
157 am3517-evm
158 net/eth/davinci_emac
159 nand/omap2-nand
160 ehci/ehci-omap
161 i2c-adapter/i2c_omap
162 mmc_host/mmci-omap-hs
163 rtc/rtc-s35390a
164 watchdog/omap_wdt
165 ...
166 * You might need to define new override values for your new platform in some
167 test case files (see section 4.1 above for details). A reasonable strategy
168 is to try to run an existing test plan and then analyze the test failures
169 to determine probable test cases where you need to define override values.
171 6) Building LTP-DDT
172 ================================================================================
174 6.1) Environment Setup for LTP-DDT build
175 ================================================================================
176 In order to cross compile LTP-DDT, a cross-compiler tool-chain is required.
177 We typically use either Linaro Toolchain or Arago Toolchain.
178 To Install Linaro toolchain, please see http://arago-project.org/wiki/index.php/Setting_Up_Build_Environment#Cross-compile_toolchain
179 To Install Arago toolchain:
180 wget and untar http://software-dl.ti.com/sdoemb/sdoemb_public_sw/arago_toolchain/2011_09/exports/arago-2011.09-armv7a-linux-gnueabi-sdk.tar.bz2
181 Set the environment variable PATH to contain the binaries of the toolchain cross-compiler tool-chain.
182 For example, in bash:
183 $ export PATH=/opt/arago-2011.09/armv7a/bin:$PATH
184 Test if PATH is set properly by running following command
185 * For Arago Toolchain
186 $ arm-arago-linux-gnueabi-gcc -v
187 * For Linaro Toolchain
188 $ arm-linux-gnueabihf-gcc -v
190 6.2) HOW TO CROSS-COMPILE AND INSTALL
191 ================================================================================
192 Note: CROSS_COMPILE may be set differently than the examples showing in this section.
193 for Arago Toolchain: CROSS_COMPILE=arm-none-linux-gnueabi-;
194 for Linaro Toolchain: CROSS_COMPILE=arm-linux-gnueabihf-
196 * cd to project root directory
197 * $ cp include/config.h.default include/config.h
198 * $ cp include/mk/config.mk.default include/mk/config.mk
199 * $ cp include/mk/features.mk.default include/mk/features.mk
200 * echo "ltp-ddt_x.x.x" > ChangeLog
201 * Update testcases/ddt/alsa_test_suite/Makefile with alsa include and
202 lib paths.
203 For example:
204 ALSA_INCPATH=/home/useraccount/usr/include
205 ALSA_LIBPATH=/home/useraccount/usr/lib
206 (In case the filesystem does not contain ALSA include and lib refer step 7
207 for installation)
208 NOTE: Alternatively instead of updating the Makefile user can supply
209 ALSA_INCPATH and ALSA_LIBPATH as arguements to make
210 * cd to kernel diretory and compile kernel if kernel is not compiled yet.
211 Then, do headers_install in kernel install directory
212 e.g.
213 $cd linux-xx.xx.xx.xx
214 $make CROSS_COMPILE=arm-none-linux-gnueabi- distclean
215 $make CROSS_COMPILE=arm-none-linux-gnueabi- <defconfig>
216 $make CROSS_COMPILE=arm-none-linux-gnueabi- uImage modules
217 $make CROSS_COMPILE=arm-none-linux-gnueabi- headers_install
218 * cd to ltp-ddt root directory do
219 $ make CROSS_COMPILE=<CC PREFIX> SKIP_IDCHECK=1 KERNEL_INC=<Kernel
220 include directory> KERNEL_USR_INC=<Kernel usr include directory>
221 PLATFORM=<Platform name>
222 For Example:
223 $ make CROSS_COMPILE=arm-none-linux-gnueabi- SKIP_IDCHECK=1 KERNEL_USR_INC=/home/
224 a0850405local/oe/arago-tmp/sysroots/armv7te-none-linux-gnueabi/kernel/usr/include KERNEL_INC=/home/a0850405local/oe/arago-tmp/sysroots/armv5te-none-linux-gnueabi/kernel/include PLATFORM=am335x-evm
225 Also build ltp-ddt test modules like edma and gpio
226 $ make CROSS_COMPILE=<CC PREFIX> SKIP_IDCHECK=1 KERNEL_INC=<Kernel
227 include directory> KERNEL_USR_INC=<Kernel usr include directory>
228 PLATFORM=<Platform name> modules
229 $ sudo make DESTDIR=<destination directory> SKIP_IDCHECK=1 PLATFORM=<Platform name> install
230 For example:
231 $sudo make DESTDIR=/opt/filesystem/home/root/ltp-ddt-install SKIP_IDCHECK=1 PLATFORM=am335x-evm install
233 7) HOW TO INSTALL ALSA INCLUDES AND LIBS
234 Install following packages on to your filesystem
235 alsa-dev (e.g. alsa-dev_1.0.18-r0.1_armv5te.ipk)
236 alsa-lib-dev (e.g. alsa-lib-dev_1.0.18-r0.1_armv5te.ipk)
237 libasound2 (e.g. libasound2_1.0.18-r0.1_armv5te.ipk)
238 To unpack the .ipk files use following commands
239 $ ar -p alsa-dev_1.0.18-r0.1_armv5te.ipk data.tar.gz | tar -zx
240 $ ar -p libasound2_1.0.18-r0.1_armv5te.ipk data.tar.gz | tar -zx
241 $ ar -p alsa-lib-dev_1.0.18-r0.1_armv5te.ipk data.tar.gz | tar -zx
242 (http://arago-project.org/wiki/index.php/Crosscompiling_Outside_of_Arago)
244 8) HOW TO RUN TESTS
245 ================================================================================
246 * Run DDT tests the same way you run LTP tests. Use ltprun program and pass to
247 it the test scenario file in the runtest directory (option -f) to run and the
248 platform (option -P) to use. For example:
249 ./runltp -P am335x-evm -f ddt/lmbench
250 The platform name specified with -P option must exist in the platforms/ dir.
251 It is also possible to run tests without -P option, in such case the ltprun
252 script won't filter test cases and it is possible that tests cases not
253 supported by the platform you are running on will be called.
254 * In addition to selecting test scenarios using -f option, users can also
255 filter test cases using -s PATTERN and -S SKIPFILE options. These options
256 select or skip test cases based on the test case TAG specified in the test
257 scenario file.
258 * The runltp script have lot of options. Some useful ones for stress tests are:
259 -t DURATION: Define duration of the test in s,m,h,d.
260 -x INSTANCES: Run multiple test instances in parallel.
261 -c <options>: Run test under additional background CPU load
262 -D <options>: Run test under additional background load on Secondary storage
263 -m <options>: Run test under additional background load on Main memory
264 -i <options>: Run test under additional background load on IO Bus
265 -n : Run test with network traffic in background.
267 9) @setup_requires Naming Conventions
268 ================================================================================
269 As discussed in section 4.2 above, @setup_requires is used to highlight test scenarios' setup
270 requirements. The following @setup_requires tags are currently defined.
271 If required, please feel free to define a new tag below and send a patch to the Opentest mailing list.
273 * Video Related:
274 usbhostmsc, usbhosthid_mouse, usbhosthid_keyboard, usbhostaudio, usbhostvideo, usbslavefbs, usbslavecdc, usbslaverndis, usbslaveasync
276 * USB Related:
277 usbhostmsc, usbhosthid, usbhostaudio, usbhostvideo, usbslavefbs, usbslavecdc, usbslaverndis, usbslaveasync
279 * Storage Related:
280 sd, sdhc, mmc, sata
283 10) HELP/SUPPORT
284 ================================================================================
285 Please submit your queries or comments to the opentest mailing list at
286 opentest@arago-project.org