path: root/testing.tex

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% %%
%% Copyright (C) 2013 Texas Instruments Incorporated - http://www.ti.com %%
%% %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{Testing}
\label{chap:lcpd-tests}

\paragraph{}
As was discussed in chapter \ref{chap:definition-of-done}, a backlog item 
must be \textbf{tested} and the test code should be \textbf{submitted} to
system test before we can call the backlog item done.
Every backlog item should have a test entry (i.e. \textit{add test} functionality) in VersionOne.
If the developer needs help from system test developing/integrating the test
code then he/she \textbf{MUST} assign the test entry to him/herself and a member
of the system test team.

\section{Test Code}
\paragraph{}
LCPD's core-sdk filesystem must be used for kernel testing. LCPD's core-sdk 
image comes with various test-related applications pre-installed:
\begin{itemize}
    \item ltp-ddt
    \item iperf
    \item bonnie++ 
    \item hdparm
    \item iozone3
    \item lmbench
    \item rt-tests
    \item evtest
    \item libdrm-tests
    \item and others...
\end{itemize}
So users are encouraged to check if there is already a test case or test 
utility available in the core-sdk filesystem before they start developing a new one.
It is recommended that new test cases are developed in LTP-DDT \ref{sec:ltp-ddt} project.

\subsection{Submitting test code}
\label{sec:sub-test-code}
\paragraph{}
The preferred way to submit test code to system test is to submit ltp-ddt patches to
opentest@arago-project.org mailing list.  

\paragraph{}
Once a test have been submitted, the systest team will take care of integrating it
into Opentest.

\subsection{LTP-DDT}
\label{sec:ltp-ddt}

\subsection{Getting LTP-DDT}
\paragraph{}
There are couple of ways to get LTP-DDT sources:
\begin{enumerate}
    \item Clone it from http://arago-project.org/git/projects/test-automation/ltp-ddt.git
    \item Use \textit{linux-devtest.py -U} to get it cloned automatically
\end{enumerate}

\subsection{Creating new LTP-DDT tests}
\paragraph{}
The recommended LTP-DDT development flow is to clone ltp-ddt, build and install it
on a nfs filesystem, so you can quickly check your LTP-DDT changes on a real target.
You can get a core-sdk filesystem tarball that you can use to setup your nfs at
\href{http://lcpd.dal.design.ti.com/core-sdk/}{LCPD core sdk}

\paragraph{}
Please refer to \href{http://arago-project.org/git/projects/?p=test-automation/ltp-ddt.git;a=blob;f=README-DDT#l19}{LTP-DDT README} sections 3 and 4 for detailed information about creating new LTP-DDT tests.

\paragraph{}
Please refer to \href{http://arago-project.org/git/projects/?p=test-automation/ltp-ddt.git;a=blob;f=README-DDT#l163}{LTP-DDT README} sections 6 and 7 for LTP-DDT build instructions.

\subsection{Running LTP-DDT tests}
\paragraph{}
Please refer to \href{http://arago-project.org/git/projects/test-automation/ltp-ddt.git?p=projects/test-automation/ltp-ddt.git;a=blob;f=README-DDT;#l223}{LTP-DDT README} section 8.

\section{Test Automation Frameworks}
\subsection{Opentest}
\label{sec:Opentest}
\paragraph{}
\href{http://arago-project.org/wiki/index.php/Opentest}{Opentest} test automation
framework is used to manage test plans, test results, test hardware and test 
execution.

\paragraph{}
You don't need Opentest to develop LTP-DDT test cases. You don't need Opentest 
to run LTP-DDT tests either. We anticipate that most kernel developers won't 
need to use Opentest on a regular basis, but instead they will run LTP-DDT tests
directly on their Device Under Test (Board or EVM). 

\paragraph{}
You will only need Opentest when you:
\begin{itemize}
  \item want to run tests on boards that you don't have.
  \item want to run complex tests that require test equipment.
  \item want to run testplans defined in Testlink
  \item want to store results in Testlink
  \item want to request multiple test runs and don't want to baby-sit them, you just want to \textbf{Click-and-Forget}
\end{itemize}

\paragraph{}
System test is responsible for integrating LTP-DDT test cases into Opentest 
test plans. 
System test will also develop/integrate complex test cases.

\subsection{Installing Opentest}
\paragraph{}
Check \href{http://arago-project.org/wiki/index.php/OpentestOtherSrvLinuxDevtest}{Opentest Installation}
\paragraph{}
\textbf{IMPORTANT:} Typical LCPD users should not install complete Opentest framework
in their desktop but instead we recommend just installing Options 2 (Staf) and
8 (Command Line Tools).