Now we should define which parameters our tests will use. Sometimes this is done in parallel with development of the test scenarios that use these parameters. At least, the set of chosen parameters and the sets of their values are often revisited when the test scenario is being developed.

Using test parameters we can vary the conditions in which the functions under test are to be checked in the test scenario. There are only general recommendations on how to choose the test parameters and their values. Ideally, the number of the parameters and the sets of their values should not be very large for the tests to remain clear. But they should be chosen so as to check the functions under test in all (or the most of) situations where it is necessary.

In addition, larger number of values of the parameters result in larger source files created by T2C file generator from the .t2c file: for each set of values an individual test is created from the corresponding <BLOCK> section. If the source files get very large, there can be problems with building the executable files of the tests from them.

As far as the example in this case study is concerned, we can start with the following parameters of the tests (suggested names of the parameter are in parentheses).

Assume that “DATA” will be the first parameter of the tests (#0) and “TYPE” - the second one (#1). Now we can specify the names of the parameters in the <DEFINE> section of the first <BLOCK> for g_array_append_vals():

<DEFINE>
#define DATA    <%0%>
#define TYPE    <%1%>
</DEFINE>

<%...%> markers denote the places where the value of the parameter with a given number will be inserted. We can refer to the parameters as <%0%> and <%1%> in the source of the test scenario but using symbolic names DATA and TYPE will make the tests more readable and clear.

Similarly, we can fill <DEFINE> section of the second <BLOCK> for g_array_append_vals() as follows, connecting the symbolic names of parameters with their numbers:

<DEFINE>
#define DATA    <%0%>
#define VALS    <%1%>
#define TYPE    <%2%>
</DEFINE>

Choosing the “right” values of the test parameters can be tricky. Basic recommendations on how to do this are given in this section.

For each parameter, there is a set of values it may take. For example, the values allowed for “DATA” and “VALS” parameters mentioned earlier are various C-style arrays that may contain the elements of a given type. “TYPE” parameter is the type of the array element and it can be a built-in C type like int, char and so on. It can also be a user-defined type, for example, a struct or union of some kind.

For now, assume we consider the parameters that do not depend on one another. For each such parameter, it can be helpful to split the set of values it may take into so called equivalence classes. For all the values from a single equivalence class, we could expect that the functions under test behave in a somewhat similar way. Actually, it is up to the developer's knowledge and experience, what to consider similar here.

Sometimes the description of a function can give us a hint about which equivalence classes could be identified. Assume the function under test has one argument (arg) that can take any value of type int. It is a common practice to make the value of such argument a parameter of the tests for this function.

The description of the function may state, for example, that it does one thing if arg is not greater than 0, another thing if arg is 777, or some other thing for the remaining positive values of arg. Sometimes this is not stated explicitly but the developer can imagine that based on the previous experience, some general considerations, the description of the related functions, etc.

We should also take into account that the values of type int can take values from INT_MIN to INT_MAX inclusive.

To prepare a set of equivalence classes for the values of arg, it is recommended to consider the “boundary points” of the ranges of values (like 0, 1, 776, INT_MAX, etc.) as separate classes in addition to the inner points of these ranges.

From each equivalence class, at least one representative value should be selected that will be used in the tests as the value of the respective parameter. If a class consists of only one value, the value will be the representative. If there is more than one value in a class, representative values can be selected in an arbitrary way, we give no specific recommendations here.

So, the equivalence classes and the sample representative values for arg parameter can be defined as shown in Table 4.2, “Equivalence classes and sample representative values”.

Now let us return to the tests for g_array_append_vals() considered in this case study and choose the values for DATA and TYPE parameters described above. The set of equivalence classes we are going to define for each of these parameters is by no means the only one possible. You may come up with different criteria for splitting the whole set of allowed values of a parameter into the equivalence classes.

The possible values for DATA parameter are C-style arrays of appropriate type. We will consider the following two equivalence classes: an empty array and non-empty arrays.

The possible values of TYPE are the names of built-in or user-defined types allowed by C programming language. The analysis of the documentation for “Glib Arrays” allows to suggest that the actual type of the element probably does not matter. It is the size of the value (the number of bytes it occupies in memory) that matters: it affects the layout of the array elements in memory, etc.

So, we could place each type having the same size to the same equivalence class. There would be a great number of equivalence classes this way. To make things more manageable, we can try to reduce the number of classes. For example, we can consider the types with size greater than, say, 10 bytes a single equivalence class. This, of course, may result in less thorough tests. In general, it is up to the developer, how to define equivalence classes to provide sufficient quality of the tests.

It is recommended to place definitions of the user-defined types in <GLOBAL> section of the .t2c file or in the header files included in this file.

Now that we have chosen the parameters of the tests as well as their values, we can set these values in the <VALUES> subsections at the end of the appropriate <BLOCK> sections.

An example from the first <BLOCK> for g_array_append_vals():

<BLOCK>
<TARGETS>
    g_array_append_vals
</TARGETS>
...
<FINALLY>
</FINALLY>

<VALUES>
    {5, -1, 0, 33}
    int
</VALUES>

<VALUES>
    {5, 1, 0, 33}
    char
</VALUES>

<VALUES>
    {5.0, 1.1, 0.7, 33.3}
    double
</VALUES>

<VALUES>
    {}
    int
</VALUES>
...
</BLOCK>

Often some variables are declared (and usually initialized with default values) at the beginning of the test, for example:

    GArray* ga = NULL;
    GArray* new_ga = NULL;
    TYPE data[] = DATA;
    int len = sizeof(data) / sizeof(TYPE);
    int req_ok = TRUE;

In C++ or with a C compiler that supports C99 standard (“ISO/IEC 9899 Programming Languages - C”), local variables can also be declared immediately before they are used, not just at the beginning of the scope. It is allowed in the tests too in these cases.

It is still recommended to declare and initialize pointers and other types of handles to the resources at the beginning of the test. The pointers can be set to NULL, other types of handles - to their default values. This helps avoid a subtle problem described below that may lead to difficult to find bugs.

REQ(), REQva(), ABORT_*() and some other macros in T2C API eventually pass execution to the contents of <FINALLY> section in some cases. Suppose we have the following fragment in a test scenario:

<CODE>
    GArray* ga = NULL;
    char* pmem; // uninitialized pointer
    ...    
    ga = g_array_new(FALSE, TRUE, sizeof(TYPE));
    if (ga == NULL)
    {
        ABORT_TEST("g_array_new() returned NULL.");
    }
    ...
    pmem = (char*)malloc(1024);
    if (pmem == NULL)
    {
         ABORT_TEST("Out of memory.");
    }
    // pmem is now used somehow
    ...
</CODE>
<FINALLY>
    ...
    free(pmem); // may result in a crash
    ...
</FINALLY>

Now suppose g_array_new() fails in this test for some reason and returns NULL. ABORT_TEST() will be called and will pass execution right to the contents of <FINALLY> section. free(pmem) will be called but pmem has not been initialized. So calling free() may result in a memory access violation or some other error or it may not, depending on what garbage pmem contains at that point.

This would not happen if pmem was set to NULL at the beginning. In <FINALLY>, it should also be checked whether pmem is NULL or not (that is, whether we need to call free() for pmem or not):

<CODE>
    GArray* ga = NULL;
    char* pmem = NULL; // explicit initialization with NULL
    ...    
    ga = g_array_new(FALSE, TRUE, sizeof(TYPE));
    if (ga == NULL)
    {
        ABORT_TEST("g_array_new() returned NULL.");
    }
    ...
    pmem = (char*)malloc(1024);
    if (pmem == NULL)
    {
         ABORT_TEST("Out of memory.");
    }
    // pmem is now used somehow
    ...
</CODE>
<FINALLY>
    ...
    if (pmem != NULL)
    {
        free(pmem); // OK
    }
    ...
</FINALLY>

Before the function under test can be called, appropriate conditions for it should be prepared (test situation): environment in the system under test, the arguments to pass to the function, etc.

When checking g_array_append_vals(), it is necessary to create a GArray structure to operate on. A pointer to this structure will be later passed to g_array_append_vals() as the first argument (ga).

    ga = g_array_new(FALSE, TRUE, sizeof(TYPE));
    if (ga == NULL)
    {
        ABORT_TEST("g_array_new() returned NULL.");
    }

It is important to check whether the preparation of the test situation succeeded. For example. it is taken into account in the example above that g_array_new() may fail to create a GArray structure and may return NULL. The results of the functions allocating resources (allocating memory, opening files, creating connections, etc.) should be checked too.

ABORT_TEST() macro is often used at this stage to indicate that the test failed to prepare the conditions required to call the function under test (see the fragment of the test above). It usually makes no sense to continue execution of the test in this case (except for cleaning up), hence ABORT_TEST() outputs the specified message and passes execution right to the contents of <FINALLY> section.

Now that all the conditions have been prepared, we can call the function under test:

    new_ga = g_array_append_vals(ga, data, len);

Ideally, each test scenario should call the function under test only once unless the goal is to check how the function behaves if called two or more times. Several calls of the function under test are also allowed in a single scenario, if the function is called in different situations each time and it was checked in all these situations except one in other test scenarios.

Consider the tests for g_array_append_vals(). The first test scenario (the one that checks appending to an empty array) calls the function just once. So if the tests from this <BLOCK> pass, we could assume that g_array_append_vals() works well in that situation. This is used in the second scenario where appending to a non-empty array is tested. First, g_array_append_vals() is used to prepare the array to operate on. It is called for an empty array for this purpose and we have checked in the previous scenario that it works correctly in this case, so it is acceptable. Then g_array_append_vals is called for the prepared arguments as the function under test. This is one of the situations when calling the function under test several times is acceptable.

Usually if a test scenario calls the function under test more than once for no obvious reason, it may indicate that the scenario can (and should) be split into smaller ones, each calling it just once.

Making test scenarios reasonably small (“atomic”) this way may help eliminate the unexpected influence the calls of the function under test may have on each other. That is, a large number of <BLOCK> sections containing reasonably atomic test scenarios is often better than a small number of <BLOCK> sections in each of which several tests scenarios are lumped together. In the former case, the tests are usually clearer and easier to manage and maintain.

The results of execution of the function(s) under test are analysed at this stage. The test scenario should now find out if any of the coppesponding requirements are violated (“failed requirements”). REQ() and REQva() macros are called to check the requirements and report success or failure. In the first <BLOCK> for g_array_append_vals(), this is done as follows.

    /*
     * the GArray.
     *
     * [The function returns a pointer to the modified GArray.]
     */
    REQ("g_array_append_vals.02", "g_array_append_vals returned NULL", new_ga);
    REQ("g_array_append_vals.02", 
        "The returned GArray pointer does not match the original one.", 
        new_ga == ga);

    /*
     * Adds len elements onto the end of the array.
     */
    TRACE("The length of the array is %d (should be %d).", 
        ga->len, old_len + data_len);
    REQ("g_array_append_vals.08", "", ga->len == old_len + data_len);
    
    for (int i = 0; i < ga->len; i++)
    {
        if (i < old_len) // the first old_len elements must not change
        {
            if(g_array_index(ga, TYPE, i) != vals[i])
            {
                req_ok = FALSE;
                TRACE("The element with index %d has incorrect value.", i);
                break;
            }
        }
        else  
        {
            if (g_array_index(ga, TYPE, i) != data[i + old_len])
            {
                req_ok = FALSE;
                TRACE("The element with index %d has incorrect value.", i);
                break;
            }
        }
    }
    
    REQ("g_array_append_vals.01", "", req_ok);

Note that if REQ detects a violation of a requirement, it outputs appropriate messages to the log file along with the text of the requirement and then passes execution directly to the contents of <FINALLY> section. That is, the remaining instructions in the <CODE> section will not be executed, the remaining requirements will not be checked in this test.

The requirement with ID “g_array_append_vals.02” is checked using two REQ() calls. The first one checks if the pointer returned by g_array_append_vals() is not NULL, the second one - if the pointer is the same as the original one (ga). This example test scenario is a bit contrived. Actually, it would be enough to leave only the second REQ. ga must not be NULL at that point, so if new_ga was NULL, new_ga == ga would be false. Violation of the requirement would be detected anyway.

Still, the above example allows to show that it is possible to use more than one REQ() call for a requirement to check different aspects of it. This may result in a series of messages like the one shown below which may be confusing.

Checked requirement: {some_func.01}
...
Checked requirement: {some_func.01}
...
Checked requirement: {some_func.01}
...
Checked requirement: {some_func.01}
...
Requirement failed: {some_func.01}

This is OK. The messages above just mean that the first four REQ() calls found no errors in the aspects of “some_func.01” requirement they checked and the last one did.

So it is not necessary to combine all expressions to be checked for a requirement in a single REQ(). Neither is it mandatory to provide several REQ() calls for it. It is up to the developer to decide. As usual, the clearer the tests and the messages they output are, the better.

Now consider the following fragment:

    TRACE("The length of the array is %d (should be %d).", 
        ga->len, old_len + data_len);
    REQ("g_array_append_vals.08", "", ga->len == old_len + data_len);

Suppose we need to output the expected and the actual length of the array only if they are not equal, that is, only if the requirement is violated. We could rewrite the above fragment as follows:

    if (ga->len != old_len + data_len)
    {
        TRACE("The length of the array is %d (should be %d).", 
            ga->len, old_len + data_len);
    }
    REQ("g_array_append_vals.08", "", ga->len == old_len + data_len);

There is a more elegant solution that, in addition, allows to avoid comparing the actual and the expected lengths twice: REQva() macro (“va” implies variable argument list). With REQva(), the above fragment could be rewritten as follows:

    REQva("g_array_append_vals.08", 
        ga->len == old_len + data_len,
        "The length of the array is %d (should be %d).",
        ga->len, old_len + data_len);

There is another subtle issue worth mentioning here. Caution should be taken when using REQ() and REQva() in the branches of code depending on the result of the function under test. The following example demonstrates the issue.

    ...
    new_ga = g_array_append_vals(ga, data, len);
    
    if (new_ga != ga)
    {
        REQ("g_array_append_vals.02", 
            "The returned GArray pointer does not match the original one.", 
            new_ga == ga);
    }
    else
    {
        REQva("g_array_append_vals.08", 
            ga->len == old_len + data_len,
            "The length of the array is %d (should be %d).",
            ga->len, old_len + data_len);
    }
    ...

At first, it seems there is no problem here. If g_array_append_vals() returns a pointer different from ga, the first branch will be taken (if (new_ga != ga) {...}) and REQ() located there will report an error, cleanup instructions will be performed and the test will end. So far, so good.

Let us see what happens if g_array_append_vals() returns the same pointer that was passed in (ga). else {...} branch will be taken this time, “g_array_append_vals.08” will be checked there, etc. But REQ() will not be called for “g_array_append_vals.02” in this case. So there will be no evidence in the log file that requirement “g_array_append_vals.02” was actually checked. When the log file is analysed, it may seem that the requirement is still not checked, which is not the case. So some information about the test for this requirement that can be valuable will be lost.

The example above could be rewritten as follows to fix the problem.

    ...
    new_ga = g_array_append_vals(ga, data, len);
    
    if (new_ga != ga)
    {
        REQ("g_array_append_vals.02", 
            "The returned GArray pointer does not match the original one.", 
            new_ga == ga);
    }
    else
    {
        REQ("g_array_append_vals.02", 
            "The returned GArray pointer does not match the original one.", 
            new_ga == ga);
            
        REQva("g_array_append_vals.08", 
            ga->len == old_len + data_len,
            "The length of the array is %d (should be %d).",
            ga->len, old_len + data_len);
    }
    ...

That is, REQ() for “g_array_append_vals.02” is placed in both branches. This works, but it is a not very elegant solution. It can be error-prone as well because you must remember to keep these REQ() calls in both branches in sync from now on. It is better to avoid placing REQ() calls in the branches that are taken depending on the results of the function under test.

If there are resources allocated in this test scenario that have not been released yet, they should be finally released at this stage: files should be closed, memory - deallocated, etc. It is recommended to use <FINALLY> section of the corresponding <BLOCK> for this purpose.

Consider the first test scenario for g_array_append_vals(). The array (ga) that was used in the test should now be destroyed. But before calling g_array_free to do this, we should check if it was really created. As it was mentioned above (Section 6.7.1, “Declarations and definitions used by the tests”), the execution may pass to the contents of <FINALLY> section before some of the objects used by the test are created. So the destruction of the array should be done as follows (taking into account that ga was initialized with NULL):

    if (ga != NULL)
    {
        g_array_free(ga, TRUE);
    }