A test developer is not required to use special API provided by T2C file generation targets in the tests. Sometimes API provided by the chosen testing framework is enough. Nevertheless, T2C API may prove handy, especially requirement-related facilities as well as API for message output.

For the first three of the targets listed, API is declared in t2c_api.h which is automatically #include'd in the resulting source file (the appropriate #include directive comes from the main template of test_group group). For c_minimal target, API is declared and defined in the source file of the tests itself.

Other T2C targets are not required to provide the same API or even any API at all.

Unless specifically stated, the API macros and functions described here are intended to be used in the parameterized test scenarios themselves, that is in <CODE> and <FINALLY> subsections of <BLOCK> sections in a .t2c file.

If TRUE and FALSE constants are not provided by the chosen testing framework or the system under tests, they are provided by T2C API.

FALSE is defined as 0, TRUE - as !FALSE.

Synopsis:

INIT_FAILED(<message>);

INIT_FAILED() should be used in <STARTUP> section to indicate that the initialization has failed for some reason. <message> should describe what happened (can be a constant string) - it will be output to the logging facility used by the tests (stderr, a log file, etc.).

After INIT_FAILED() is called, the instructions following it in the <STARTUP> section will not be executed. Execution of the tests from this .t2c file will be cancelled. The contents of <CLEANUP> section (if it is not empty) will be executed anyway.

Example:

    mod = (FT_Module)FT_Get_Module(library, "otvalid");
    if( mod == NULL )
    {
        INIT_FAILED("OpenType validation module is not available.");
    }

Synopsis:

TRACE(<format>, ...);
TRACE0(<message>);

TRACE() outputs the formatted message to the logging facility used by the tests. This is like what printf() does for stdout except that TRACE() should be passed at least one argument besides the format string <format>.

<format> is a format string of the same kind that printf() uses.

Unlike printf(), TRACE() may end the message with a newline character (if applicable for the logging facility used).

TRACE0() is like TRACE() except it takes the only argument, the message to be output.

Examples:

    TRACE("The element with index %d has value %d.", i, val[i]);
    TRACE0("The operation completed successfully.");

Synopsis:

REQ("<ID> [; ...]", <message>, <expression>);
REQva("<ID> [; ...]", <expression>, <format>, ...); 

REQ() and REQva() are used in the test scenarios (<CODE> section) to check requirements (assertions) and report failures. These macros differ only in the way they prepare a message to be output in case of a failure.

Both macros receive the list of identifiers of requirements checked there and an expression to check.

If <expression> evaluates to nonzero, the check passes, that is, the requirements listed in the first argument of the macros seem to hold. In this case, REQ() and REQva() may report that the requirements were checked. Then execution of the test goes on.

If <expression> evaluates to zero, it indicates a failure, namely a violation of at least one requrement listed in the first argument of the macro. In this case, REQ() and REQva() do the following.

REQ() outputs the message specified in its second argument verbatim to the logging facility used. The message should describe what happened rather than just repeat the text of the corresponding requirements. The message can be left blank if necessary.

REQva() creates and outputs the formatted message using the specified format string and the arguments after it. The effect is as if <format> and all arguments following it were passed to TRACE().

Then both macros report that the specified requirement is violated (or at least one of the requirements if several of them are specified in the list). In addition, the text of the requirement(s) is loaded from the catalogue of requirements and also output to the logging facility (except in c_minimal target). After that, the rest of the code from <CODE> section is not executed but the code from <FINALLY> section is executed. It is similar to “fatal assertions” used in some testing frameworks. Test verdict is set to “FAIL” in this case.

Examples:

    REQ("g_array_append_vals.02", 
        "g_array_append_vals returned NULL", 
        new_ga != NULL);

    REQva("g_array_append_vals.08", 
        ga->len == len,
        "The length of the array is %d while it should be %d.",
        (int)ga->len, len);

It is possible to provide more than one REQ()/REQva() for the same requirement (probably with different expressions to check) in a test scenario.

Ideally, a single requirement should be checked in each REQ()/REQva() call. Sometimes, however, some of the requirements are tightly coupled and it can be difficult to tell exactly which one of them is violated if the checked expression evaluated to 0. This may happen, for example, in the tests for getter/setter functions and in more complex cases.

To handle this, the identifiers of the requirements involved should be listed in REQ()/REQva() separated by semicolons (spaces are allowed before and after a semicolon), for example:

## {ext.g_array_new.06}: "The size of the array is 0 upon creation."
## {g_array_new.08}: "'len' is the length of the array."

    REQ("ext.g_array_new.06; g_array_new.08", 
        "'len' field is not 0 upon creation of the array", 
        ga->len == 0);

The requirements for the applications rather than for the system under test (i.e. those with app-prefixed IDs, app-requirements) can also be checked using REQ()/REQva(). Such REQ()/REQva() calls are usually placed somewhere in the beginning of the test scenario. Unlike the requirements for the system under test, violation of an app-requirement indicates an error in the test itself.

REQ()/REQva() usually handle app-requirements in a different way than the ordinary requirements. They often set different test verdicts in these cases, for example, UNRESOLVED if an app-requirement is violated, FAIL if a requirement for the system under test is violated. The messages output by these macros can also be different.

App-requirements and the requirements for the system under test should not be combined in the same list of IDs in a REQ()/REQva() call. It usually makes no sense anyway.

Requirements for the system under test with ext-prefixed IDs (ext-requirements) are not checked by default. To put it more exactly, REQ()/REQva() calls for such requirements are considered passed by default. If there are several IDs of requrements listed in REQ()/REQva() and at least one of them is an ext-requirement, this REQ()/REQva() is considered passed.

To turn on checking of ext-requirements, CHECK_EXT_REQS must be defined when compiling the tests. To do this, before generating the test sources from .t2c files, add -DCHECK_EXT_REQS compiler option via a parameter in the configuration file for the suite or by adjusting the template of a per-test makefile.

A special expression TODO_REQ() can be used in REQ() (and REQva()) in place of <expression> argument to mark this REQ() as not yet implemented. Such REQ()/REQva() calls are ignored (no-ops) when the tests are executed.

Synopsis:

RETURN;

RETURN ends the test without changing its the verdict (result code). The verdict is PASS by default and can be changed, for example, by REQ(), ABORT_TEST() and other macros.

Synopsis:

ABORT_TEST(<message>);

This macro should be used in a test to abort its execution if something wrong happens: a failure to prepare test situation or the means to analyse the resulting state of the system under test, etc. <message> should describe the situation (can be a constant string).

Example:

    ga = g_array_new(FALSE, TRUE, sizeof(TYPE));
    if (ga == NULL)
    {
        ABORT_TEST("g_array_new() returned NULL.");
    }

This macro may set the verdict of the test to UNRESOLVED (at least for some T2C targets) unless it has been already set to FAIL.

Synopsis:

ABORT_UNSUPPORTED(<message>);

This macro should be used in a test to abort its execution if an optional feature to be checked is not supported by the system under test. <message> should describe the situation (can be a constant string).

Example:

    if (!is_mode_supported(2048, 1536, 64))
    {
        ABORT_UNSUPPORTED("Optional video mode 2048x1536 (64bpp) is not supported");
    }

This macro may set the verdict of the test to UNSUPPORTED (at least for some T2C targets) unless it has been already set to FAIL, UNRESOLVED, UNTESTED or FIP.

Synopsis:

ABORT_UNTESTED(<message>);

This macro should be used in a test to abort its execution if an mandatory feature to be checked is not supported by the system under test. <message> should describe the situation (can be a constant string).

Example:

    if (check_dll_support() == DLL_NOT_SUPPORTED)
    {
        ABORT_UNTESTED("Dynamically loaded libraries are not supported. "
            "Aborting the test.");
    }

This macro may set the verdict of the test to UNTESTED (at least for some T2C targets) unless it has been already set to FAIL, UNRESOLVED, UNSUPPORTED or FIP.

Synopsis:

ABORT_FIP(<message>);

This macro should be used in a test to abort its execution and indicate that the test has FIP ("Further Information Provided") status. It means that more information is necesary to get about the system under test to decide whether the test passed or not. <message> should describe the situation and, preferably, specify which additional data is necessary (<message> can be a constant string).

Example:

    if (!have_a_clue)
    {
        ABORT_FIP("To determine if the test passed, the following " 
            "additional information is needed: ...");
    }

This macro may set the verdict of the test to FIP (at least for some T2C targets) unless it has been already set to FAIL or UNRESOLVED.

Synopsis:

TEST_FAILED(<message>);

This macro aborts test execution and sets the result of the test to FAIL. Most of the time, you do not need this macro because test failures are detected and handled using REQ and REQva macros. TEST_FAILED() is provided for those rare cases in which using REQ/REQva is inconvenient or even inappropriate (a failure which is not tied to a particular requirement or a group of requirements).

Example:

    if (system_behaves_wrong)
    {
        TEST_FAILED("The following parameters of the system "
            "have wrong values: ...");
    }

Synopsis:

T2C_GET_DATA_PATH(<relative_path>);

This macro constructs the path to the specified test data (file or directory) and returns the resulting string (see also Section 6.5, “Access to test data directories from the code”). NULL is returned if there is not enough memory to perform the operation.

The test data directory depends on the T2C file generation target used. It is <$SUITE_ROOT_DIR$>/testdata/<$T2C_FILE_NAME$> (see Section 5.7.2, “Special parameters defined by the file generator”) for c_tet, c_standalone and c_gtest_nested targets. For c_minimal target, it is the current working directory.

<relative_path> is a path to the data relative to the test data directory.

The resulting path is absolute for c_tet, c_standalone and c_gtest_nested targets and relative for c_minimal target.

The returned pointer should be freed when no longer needed.

Example:

    char* path = T2C_GET_DATA_PATH("gttmodule.so");
    if (path == NULL)
    {
        ABORT_TEST("Out of memory.");
    }
    
    // Now the path can be used in the test
    ...
    
    // Free memory occupied by the path 
    // when it is no longer needed.
    free(path); 

Synopsis:

T2C_GET_TEST_PATH();

T2C_GET_TEST_PATH() returns the path to the executable file containing the current test.

The resulting path is absolute for c_tet, c_standalone and c_gtest_nested targets and relative for c_minimal target.

The returned pointer should be freed when no longer needed.

Example:

    char* self = T2C_GET_TEST_PATH();
    if (path == NULL)
    {
        ABORT_TEST("Out of memory.");
    }
    
    // Now the path can be used in the test
    TRACE("Path to this test: %s", path);
    ...
    
    // Free memory occupied by the path 
    // when it is no longer needed.
    free(path); 

The process of building and installing the test suites created using c_tet and c_standalone targets is quite similar. They use different “backends” to execute the tests and collect their results (TETWare Lite and the simple testing framework provided by T2C) and the generated sources of the tests reflect that. But, from the user's point of view, the process of building and installing the test suite is quite similar. Unless specifically stated, what is stated in this section, refers to the both targets.

The goal of these targets is to provide a package that may contain several test suites at once and can be built and installed using configure - make - make install process. The makefiles in the test package are indended to be used with GNU make.

The typical structure of a single test suite is shown here.

After the source files, Makefiles and other files have been generated (using gen_files.sh> script or the file generator directly), the test package should be processed by autoconf and automake to produce the remaining files necessary to build and install the tests. One way to do this is to execute the following command from the root directory of the test package (the very directory where configure.ac is located):

autoreconf --install

Among other things, configure script and the top-level Makefile will be created. Now the source package with the tests can be prepared:

./configure && make distcheck

If these commands succeed, a .tar.bz2 archive (the test source package) will be created, ready for distribution.

c_gtest_nested target allows to create an Autotools-compatible test package that can be nested in the package under test or can be used separately.

The only top-level file of the test package distributed with T2C for this target is gen_files.sh script (see c_gtest_nested/other/ subdirectory of T2C source package). It is not installed along with T2C, just distributed. The script should be placed to the root directory of the test package.

When called with name of the target as an argument (gen_files.sh c_gtest_nested), this script invokes T2C file generator to create appropriate files for the suite (both common and per-test ones). It also creates directory t2c_api/c_gtest_nested/ and copies the sources of the appropriate T2C API library there.

gen_files.sh clean deletes all files generated by t2c as well as t2c_api directory with all its contents.

This target also requires that m4 and testdata subdirectories exist in the root directory of the test suite.

m4 subdirectory can be left empty, Autotools will place appropriate files there when autoreconf is called (see below).

testdata subdirectory can be used to store the data needed by the tests (see Section 6.5, “Access to test data directories from the code” for details.) The subdirectory should at least contain Makefile.am file. This file may contain instructions to prepare the data for before the tests are executed via make check or make test. If this is not necessary, the file can be simply left blank.

After the source files, Makefiles and other files have been generated (using gen_files.sh script or the file generator directly), the test package should be processed by autoconf and automake to produce the remaining files necessary to build and install the tests. If the test package is nested in the package under test, this will be done by calling these tools for the package under test. This will happen if the nested package is properly specified in AC_CONFIG_SUBDIRS in configure.ac and SUBDIRS in Makefile in the package under test - see the tutorial for details and examples.

If you would like to use the test package separately, that is, not nesting it in the package under test, the following command called from the root directory of the test package with execute appropriate autotools:

autoreconf --install

Among other things, configure script and Makefiles for the suite and for each group of tests will be created.

If the test package is nested in the package under test, the former will be configured and built when these operations are performed for the latter. They will usually reside in a single archive.

If the test package is to be used separately, the following commands can be used to prepare the package:

./configure && make distcheck

If these commands succeed, a .tar.bz2 archive (the test source package) will be created, ready for distribution.

To build the tests from such package, the usual “configure - make” process can be used. VPATH builds are also supported.

As usual with Autotools-compatible packages, custom compiler and linker options can be specified in CFLAGS, CXXFLAGS, LDFLAGS and other variables when configuring the package.

There are no separate files with T2C API implementation for this target (everything is in the source file of the tests). Neither are there any auxiliary files.

The goal is to provide just a C/C++ source + makefile for it. That is, no complex test harness, only a plain C or C++ program containing little except the contents of the corresponding .t2c file. This can be used to debug the tests, to thoroughly analyse the behaviour of the system under test in some particular cases, to provide example programs for bug reports, etc.

Therefore many features are not supported in c_minimal target including requirement catalogues, execution in separate processes, time limits, etc.

To build the group of tests generated from a .t2c file using c_minimal target, change current directory to where the source file of this group of tests is located (usually tests/<t2c_file_name>/ subdirectory in the test suite) and execute make.

make clean can be used for cleaning up, make debug - for building the tests with compiler options better suitable for debugging.

The tests are executed using TETWare Lite tools in this case. These tools require TET_ROOT environment variable to be defined. This variable should contain the path to the directory where TETWare Lite is installed. For example, the test case controller (tcc) should be located in ${TET_ROOT}/bin/ directory.

To run the tests from all suites in the test package, execute run_tests.sh script. This sample script is provided with T2C in the same package and should be copied to the test package before using (see the description of the auxiliary files in Section 4.2.1, “c_tet and c_standalone”). The script will call TETWare tools in appropriate environment and with appropriate parameters.

To run the tests from a particular suite contained in the package, specify the name of the suite (the name of the corresponding subdirectory) as an argument to run_tests.sh. For example, assume the test package contains three test suites: glib-t2c, gobject-t2c and pango-t2c. The following command will execute only the tests from gobject-t2c:

./run_tests.sh gobject-t2c

Actually, a part of the name is also acceptable. For example, the following command will execute the tests from gobject-t2c and pango-t2c:

./run_tests.sh go

The results of the tests are in a log file in special format - a so called TET journal. It is located in results/0001e/ subdirectory of the main directory of the test package. Consult TETWare manuals on how to read and analyse such journals.

Similar to c_tet target, the either all the tests or the tests from a particular suite can be executed using run_tests.sh script.

The results of the tests (including all the messages they output via TRACE or the like) are collected in results/test.journal file. A fragment of such file is presented below.

Executing the tests from the following group:
/home/spectre/tests/temp/desktop-t2c/glib-t2c/tests/glib_arrays/glib_arrays
---------------------------------------------------------------------------
[Startup begins]
[Startup ends]
---------------------------------------------------------------------------
<...skipped...>
---------------------------------------------------------------------------
[Test #30 begins]
    Test target(s):
    g_array_set_size
    g_array_new
    g_array_sized_new
    --------
    Checked requirement: {g_array_set_size.02}
    Checked requirement: {g_array_set_size.02}
    Checked requirement: {g_array_set_size.01}
    Checked requirement: {g_array_set_size.08}
    Checked requirement: {g_array_set_size.03}
    Checked requirement: {g_array_new.03}
[Test #30 ends]
[Result] PASSED
---------------------------------------------------------------------------
<...skipped...>
---------------------------------------------------------------------------
[Cleanup begins]
[Cleanup ends]
---------------------------------------------------------------------------

Currently, no special means are provided to calculate how many tests passed or failed, etc. To find this out, the following commands can be used:

grep "^\[Result\] PASSED" test.journal | wc -l

The command above calculates the number of the individual tests that passed. Replace “PASSED” with “FAILED”, “UNRESOLVED” or any other supported test verdict to find out the number of tests that ended with this verdict.

The total number of test executed can be calculated in a similar way:

grep "^\[Result\]" test.journal | wc -l

A group of tests generated from the same .t2c file can be executed separately if necessary. The application containing the group of tests is located in tests/<group_name>/ subdirectory of the test suite and has the same name as the group. Example: <…>/glib-t2c/tests/glib_arrays/glib_arrays.

This application can be executed in an ordinary way from the directory where it resides:

./glib_arrays

This will run all the tests from “glib_arrays” group. The results (in the same format as in the example above) will be stored in a .log file in the same directory.

It is possible to run only selected tests from the group. To do this, specify the indexes of the tests to execute as the arguments. Example:

./glib_arrays 10 15 1 7

Here, the tests #10, #15, #1 and #7 will be executed, in this order. The code from <STARTUP> section of the corresponding .t2c file will be executed before the first of these tests (#10), the code from <CLEANUP> section - after the last one (#7).

The first test in the group has index of 1. The index of each test can be found in the source file of the group of tests, in the comments before the test. Look for “This is the test #…” sentence there. Example:

///////////////////////////////////////////////////////////////////
// This is the test #30
//
// Target(s):
//    g_module_supported
//
// Parameters:
//    none
///////////////////////////////////////////////////////////////////

A group of tests prepared using c_minimal target can be executed in a similar way as the one prepared using c_standalone (see above). That is, if you run the executable built for a group of tests, all tests from this group will be executed. If the indexes of the tests are listed as the arguments of the test executable, only these tests will run, in the order the indexes are specified.

No log file is used in c_minimal target. The messages from the tests are simply output to stdout and stderr.

If the test package is nested in the package under test, the tests will be run when make check is executed for the latter which is often enough.

In addition make check or make test (these commands are equivalent) can be executed from the root directory of the test package to run all the tests. make will report failure if at least one individual test fails.

A log file for each group of tests will be stored in XML-based format in the same directory as the test executable. The log file has extension “xml” and the same name as that directory (in fact, it is the same name as of the .t2c file the tests were generated from).

To generate a complete report on this test run, execute make test-report from the root directory of the test package. This way test-report.xml will be assembled from the logs for the groups of tests and test-report.html will be created - a report in HTML format that can be viewed in a web browser.

make test and make test-report can be called for a single group of tests as well to run the tests and prepare reports in XML and HTML formats for this group only. To do this, you can execute these commands from the directory where the executable for this group resides (tests/<group_name> subdirectory of the test package).

make test will run the tests from this test group and create log file <group_name>.xml in the same directory.

make test-report will run the same instructions as make test first (“test” is a prerequisite for “test-report” - see the Makefile.am) and then it will prepare the report in HTML format (<group_name>.html) from the log file.

Copyright information and license notes can be specified using parameters of the test suite, COPYRIGHT_HOLDER and LICENSE. The templates for c_tet, c_standalone and c_gtest_nested targets refer to these parameters, so the corresponding information will appear in the generated source files.

Suppose you set COPYRIGHT_HOLDER and LICENSE parameters the following way in the configuration file of the suite:

COPYRIGHT_HOLDER = MyGreatCompany, Inc

LICENSE =>>
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. 
<<

As a result, each test source file created from a .t2c file for this test suite will have a notice in the comments at its beginning similar to the following one:

/**************************************************************************
Copyright (C) 2010 MyGreatCompany, Inc. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
**************************************************************************/

Target-specific parameters of the tests are described here. These parameters can be specified in the configuration file of a test suite and, if applicable, in the configuration files of the groups of tests. If a parameter is not specified, it is considered to have an empty value.

# Custom compiler options
COMPILER_FLAGS=`pkg-config --cflags glib-2.0`

# Custom linker options
LINKER_FLAGS=`pkg-config --libs glib-2.0`

# Maximum time in seconds each test is allowed to run
WAIT_TIME=60

# Copyright holder
COPYRIGHT_HOLDER = Some Company

# A boilerplate notice to appear at the top of each test source file
# generated by T2C for this test suite.
LICENSE =>>
Licensed under some license.
<<

############################################################################
# mysuite.cfg - common configuration parameters for the test suite
# for "MySuperLibrary" package
############################################################################

# Name of the test suite
TEST_SUITE_NAME = mysuite-t2c

# Version of the test suite
TEST_SUITE_VERSION = 0.1.0_alpha2

# Bug report address of the test suite (typically, email or web address)
TEST_SUITE_BUGREPORT = http://bugtracker.somesite.org/sample-tests/

# Copyright holder (will appear in the generated sources, etc.)
COPYRIGHT_HOLDER = Some Company Ltd

# Name of the package under test (it will appear in the comments only)
PACKAGE_UNDER_TEST = My Super Library 3.1.x

# Extra options for the compiler and linker common for all the tests in the 
# suite. 
COMMON_COMPILER_FLAGS = -DCHECK_EXT_REQS \
    `pkg-config --cflags MySuperLibrary-3.1 gtk+-2.0`
COMMON_LINKER_FLAGS = 
COMMON_LIBS = `pkg-config --libs MySuperLibrary-3.1 gtk+-2.0`

# To provide additional test-specific options, specify them in the following
# parameters in the .cfg file for the respective group of tests: 
# ADD_COMPILER_FLAGS, ADD_LINKER_FLAGS, ADD_LIBS

# Source and include directories of the package under test (can be the same)
# [NB] $(top_srcdir) is used here to refer to the top source directory of
# the test package rather than the package under test.
BASE_PACKAGE_SRC_DIR = $(top_srcdir)/../../msl/src
BASE_PACKAGE_INC_DIR = $(top_srcdir)/../../msl/include

# These parameters are usually provided in the .cfg file of the group of tests.
#ADD_SOURCES = 
#ADD_HEADERS = 

# A boilerplate notice to appear at the top of each test source file
# generated by T2C for this test suite.
LICENSE =>>
Licensed under some license.
<<