NoesisGUI

Unit Testing Guide

This guide pretends to describe how unit tests are used in Noesis Engine, why they are useful and what rules you must follow when implementing then.

Implementing Unit Tests

How to implement an Unit Test

A Unit Test is a simple class deriving from BaseComponent and implementing the IUnitTest interface. This is the minimum code to declare a test

#include <Noesis.h>
#include <NsCore/BaseComponent.h>
#include <NsCore/IUnitTest.h>
#include <NsCore/Reflection.h>


namespace Noesis
{
namespace Gui
{

class VisualTest: public Core::BaseComponent, public Core::IUnitTest
{
public:
    /// From IUnitTest
    //@{
    void RunTest();
    //@}

private:
    NS_DECLARE_REFLECTION(VisualTest, Core::BaseComponent)
};

}
}

Unit Tests must test for positive results (testing that the result is ok) and for negative results (testing that and error is returned in those scenarios that are not considered valid). The following macros facilitates the testing process (defined in NsCore/UnitTestMacros.h)

Macro Purpose
NS_UNITTEST_CHECK(condition) Checks the condition is true, raising an exception if false
NS_UNITTEST_MUST_FAIL(expression) Check an expression that must throw. If an exception is not thrown, the macro raises its own to notify the user

Care must be taken with code that asserts in Debug throwing exceptions, because in Release the assert expressions are not evaluated. Tests must run correctly in Debug, Release, or any other project configuration. What follows is a simple test implementation

#include "VisualTest.h"
#include <NsCore/UnitTestMacros.h>
#include <NsCore/TypeId.h>
#include <NsCore/Category.h>
#include <NsGui/Visual.h>


////////////////////////////////////////////////////////////////////////////////////////////////////
void VisualTest::RunTest()
{
    Ptr<Gui::Visual> visual = NsCreateComponent<Gui::Visual>();

    NS_UNITTEST_CHECK(visual->GetParent() == 0);
    NS_UNITTEST_MUST_FAIL(visual->SetParent(visual));
}

////////////////////////////////////////////////////////////////////////////////////////////////////
NS_IMPLEMENT_REFLECTION(VisualTest)
{
    NsMeta<TypeId>("VisualTest");
    NsMeta<Category>("UnitTest");
    NsImpl<IUnitTest>();
}

Unit Tests are registered in the Component Factory with the category of UnitTest, so it's possible to enumerate all the registered tests at any moment.

NS_REGISTER_REFLECTION(Gui, Core)
{
    NS_REGISTER_COMPONENT(VisualTest, NSS(UnitTest))
}

Installing mockups for kernel systems

For some tests you may need getting access to global kernel systems. Inside a unit test it cannot be guaranteed that a kernel system has been initialized. For that purpose the class KernelSystemOverrideRAII is provided to override or install mockup implementations of kernel systems. For example:

void XamlReaderTest::RunTest()
{
    KernelSystemOverrideRAII overrideUISystem(NSS(UISystem), NsCreateComponent<UISystemMockup>());

    // Executes the test with a UISystem mockup installed
    {
        // ...
    }

}

External files

Unit Tests must not depend on external files. If a test needs data that is stored in files (for example, the ImageImporterTest needs some png files to check its functionality) these files must be added to the C++ code as an array of bytes, using the File2Array tool.

The data arrays should be used directly from memory, for example through a MemoryStream implementation, but in case that a disk file is mandatory, a temp file can be created in the execution directory and then deleted when the test has finished (DllTest does this).

How to invoke an Unit Test

The registration in the Component Factory allows the creation and execution of tests without having direct access to their code declaration. The easiest way to run tests is using the UnitTestSystem. It allows to execute a desired test by name, or to execute all the registered tests. For the option of executing all tests, an additional parameter indicates to create an XML file in the Bin directory with the result of the execution. This can be useful to send to a continuous integration machine

Ptr<IUnitTestSystem> unitTestSystem = NsGetSystem(NSS(UnitTestSystem));

// Run one test
IUnitTestSystem::TestResult result;
if (!unitTestSystem->RunTest(NSS(VisualTest), &result))
{
    NS_WARNING(NST("VisualTest failed!. Error is '%s'"), result.exception.GetDescription());
}


NsUInt total = 0;
NsUInt failed = 0;
Ptr<IIterator<const IUnitTestSystem::TestResult&> > it;

// Run all tests creatint the XML file also
it = unitTestSystem->RunAllTests(total, failed, true);

if (failed)
{
    NS_WARNING(NST("%d/%d tests failed"), failed, total);

    // Print info about the failing tests
    while (!it->End())
    {
        NS_INFO_SECTION(NST("Test %hs failed"), it->Get().testName.GetStr());
        it->Get().error.DumpExceptionToLog(LogSeverity_Critical, true);
        it->Get().error.ClearCriticalFlag();
        it->Advance();
    }
}

Another option is using the command line tool located in the package Core/Tester

PS W:\Noesis\NoesisSDK\Bin> .\Debug.Core.Tester.exe run xamlreadertest

------------------------------------------------------------------------------
 Tester Tool v0.1
------------------------------------------------------------------------------

> NoesisApplication Run
  Test XamlReaderTest sucessfully passed
  1/1 test passed

Guidelines

  • Each class submitted to the repository must be accompanied by its corresponding Unit Test. Even private classes must be tested. The philosophy behind Unit Testing is that each class must be isolated and tested independently.

    For example in the Core/Serialization package you can find a test for the MetaData class that is used privately by the Serialization Manager. This class was designed and implemented in parallel with a test. And when all the functionality was ready it was incorporated inside the SerializationManager implementation.

  • Unit Tests must be implemented in parallel with the class implementation. This point is very important to make a good test. Whenever you start implementing a new class you create a empty test. In that test you start designing the class and testing each new functionality as it is being implemented. This guarantee that all the methods are tested inside the unit test.

  • Unit Tests must test all the lines of the corresponding class. 100% code coverage is the ideal for an unit test. This implies that all functions must be invoked (directly or indirectly), all the branches in each conditional code (if, switch, etc) must be tested and each template member must be instantiated at least for one type.

  • Unit Tests must cover edge cases. For example:

    • Testing than a default constructed class has valid values.

      MetaData metaData;
      
      NS_UNITTEST_CHECK(metaData.GetRoot()->GetNumFields() == 0);
      
      MetaDataIterator it = metaData.GetIterator();
      NS_UNITTEST_CHECK(it.End());
      
    • Testing that empty containers inside a class do not crash any public function. And if it must crash, this must be tested too.

      // GetDomain
      NS_UNITTEST_CHECK(configSystem->GetDomain(NSS(TestDomain)) == domain);
      NS_UNITTEST_CHECK(configSystem->TryGetDomain(NSS(Null)) == 0);
      NS_UNITTEST_MUST_FAIL(configSystem->GetDomain(NSS(Null)));
      
    • Testing that adding a repeat key to a container value raises an exception (in those cases where this is necessary)

      // AddProperties
      domain->AddPropertyInt(NSS(int), 3);
      domain->AddPropertyFloat(NSS(float), 3.0f);
      domain->AddPropertyBool(NSS(bool), true);
      domain->AddPropertyString(NSS(string), NST("Three"));
      
      NS_UNITTEST_MUST_FAIL(domain->AddPropertyInt(NSS(int), 3));
      NS_UNITTEST_MUST_FAIL(domain->AddPropertyFloat(NSS(float), 3.0f));
      NS_UNITTEST_MUST_FAIL(domain->AddPropertyFloat(NSS(bool), true));
      NS_UNITTEST_MUST_FAIL(domain->AddPropertyString(NSS(string), NST("Three")));
      
  • Unit Tests are not code examples. Although they serve as a useful code reference, a unit test must cover all the functionality with usually result in dense code. See for example Math3DTest

  • Unit Tests must isolate the current class being tested from the rest of the classes. Sometimes this is not possible because a class uses the interface of another class. In those case the recommended pattern is implementing a mockup. The mockup provides an empty implementation. For example,

    // Mockup implementation of a IStream to be used by this test
    class Noesis::Core::MockupStream: public BaseComponent, public IStream
    {
    public:
        MockupStream(): mReadPos(0), mWritePos(0) {}
    
        /// From IStream
        //@{
        NsBool CanSeek() const
        {
            return false;
        }
    
        void SetPosition(NsUInt64)
        {
            NS_ERROR(NST("Can't seek"));
        }
    
        NsUInt64 GetPosition() const
        {
            NS_ERROR(NST("Can't seek"));
        }
    
        void SetLength(NsUInt64)
        {
            NS_ERROR(NST("Can't seek"));
        }
    
        NsUInt64 GetLength() const
        {
            NS_ERROR(NST("Can't seek"));
        }
    
        NsBool CanRead() const
        {
            return true;
        }
    
        void Read(void* buffer, NsSize size)
        {
            NS_ASSERT(size <= sizeof(mBuffer) - mReadPos);
            memcpy(buffer, mBuffer + mReadPos, size);
            mReadPos += static_cast<NsInt>(size);
        }
    
        NsBool CanWrite() const
        {
            return true;
        }
    
        void Write(const void* buffer, NsSize size)
        {
            NS_ASSERT(size <= sizeof(mBuffer) - mWritePos);
            memcpy(mBuffer + mWritePos, buffer, size);
            mWritePos += static_cast<NsInt>(size);
        }
    
        void Flush() {}
        void Close() {}
        //@}
    
        void Reset()
        {
            mReadPos = 0;
            mWritePos = 0;
        }
    
    private:
        // Internal buffer
        NsByte mBuffer[32];
        NsInt mReadPos;
        NsInt mWritePos;
    
        NS_DECLARE_REFLECTION(MockupStream, BaseComponent)
    };
    
    ////////////////////////////////////////////////////////////////////////////////////////////////////
    void BinaryFormatterTest::RunTest()
    {
        EqualTest(NsCreateComponent<BinaryWriter>(), NsCreateComponent<BinaryReader>(),
            NsCreateComponent<MockupStream>());
        EqualTest(NsCreateComponent<BinaryWriterSwapper>(), NsCreateComponent<BinaryReaderSwapper>(),
            NsCreateComponent<MockupStream>());
    
        NotEqualTest(NsCreateComponent<BinaryWriterSwapper>(), NsCreateComponent<BinaryReader>(),
            NsCreateComponent<MockupStream>());
        NotEqualTest(NsCreateComponent<BinaryWriter>(), NsCreateComponent<BinaryReaderSwapper>(),
            NsCreateComponent<MockupStream>());
    }
    
  • The execution of a test must not alter the global state of the running program. This has several implications like for example that no global kernel systems may be used in a unit test. Another implication is that classes optimized for memory usually does not store a used KernelSystem and whenever they need it, it is acquired from the Kernel. This kind of implementation is not testable because it is using a global kernel system. The recommended solution for this is having two implementations (using a template parameter or a virtual function), in the normal implementation the kernel system is acquired from the Kernel, in the test implementation the kernelsystem is stored inside the class. This stored kernelsystem is created locally inside the test. See the TaskSystemTest and how BaseTask is implemented to be testable inside an unit test.

    In the current implementation of Unit Tests the isolation from the running program is not 100%. For example, the Memory Manager and Symbol Manager is shared between the running the test and the running framework. This is not desirable and probably will be fixed in the future.

© 2017 Noesis Technologies