NoesisGUI

Integrating Third Parties into Noesis Engine

Everything about Third Parties you always wanted to ask, yet never dared to. :)

Introduction

Third Parties are external libraries that are not being developed inside Noesis. The modular structure of Noesis allows for easily integrating external work avoiding reinventing the wheel. Examples of third parties are:

  • FreeType
  • FreeImage
  • OpenAL
  • Python

There are three characteristic points that differentiate a third party package from a Noesis package:

  • Third parties are stored inside the 3rdParty directory
  • Third parties are stored in the repository as binaries + source code. The libraries and binaries are precompiled and commited to Noesis Repository.
  • The development of the third party is not tracked inside Noesis Repository. Normally, only stable versions are deployed to the repository

Configuration

Third parties are located inside the NoesisSDK3rdParty directory. Each package is stored in a directory. In the root of that directory a ThirdParty.def file can be found. That file configures the third party library for proper integration in Noesis. For example, the ThirdParty.def file for FreeType is the following:

#
# ThirdParty definition file
#

[global]

DESC = FreeType, A Free, High-Quality, and Portable Font Engine
WEB = http://www.freetype.org
LICENSE = http://www.freetype.org/FTL.TXT
INCLUDE = freetype-2.3.5/include

LIB = freetype-2.3.5/objs

# Changes to the package
#   The version compiled is Release Multithread. PDB generation have been activated (vc80.pdb)

Most of the settings are self-explanatory, but they are described here one by one

Description

A description of the package is specified here.

Web

A web link to the home page related to the project must be specified here.

License

This field is reserved to describe the license under which this package is being used in Noesis. A clear description or, even better, a link to the licence must be found here. It is very important to describe this field correctly. Whenever possible free or non restrictive licences must be used. Licences supporting source code are preferred. More restrictive licenses can incur in a extra costs to users of Noesis. (TODO: this must be managed by NsToolkit)

Include

A list of paths (relative to the main folder of this library) where include files are stored. This paths are automatically added to packages depending from this library. For example, with the definition of Include found above a package using this 3rdParty could write

////////////////////////////////////////////////////////////////////////////////////////////////////
/// (c) Copyright 2007 Noesis Engine. All Rights Reserved.
/// $Id: Font.h 380 2008-05-30 01:07:53Z jsantos $
////////////////////////////////////////////////////////////////////////////////////////////////////


#ifndef __DRAWING_FONT_H__
#define __DRAWING_FONT_H__


#include <NsCore/BaseComponent.h>
#include <NsCore/Reflection.h>
#include <NsDrawing/IFont.h>

#include <hash_map>

#include <ft2build.h>
#include FT_FREETYPE_H

Lib

A list of paths (relative to the main folder of this library) where library files are stored. Library files are not automatically linked in the dependant packages. The path is automatically added to the project, but the link must still be forced. For example, following with the above example

#pragma comment(lib, "freetype235MT.lib")

Some projects have different directories for each build configuration. This can be specified in the definition file as follows

[Debug]
LIB = expat-2.0.1/win32/bin/debug

[Release]
LIB = expat-2.0.1/win32/bin/release

Bin

A list of binary files (relative to the main folder of this library) that must be copied to the /NoesisSDK/bin directory whenever this 3rdParty is used. The copy is done with the --dist3rd option of NSToolkit. All the binaries that are needed for proper use of the package must be copied: dynamic libraries, debug symbols, etc.

Example,

BIN = Original_Files_From_SDK/bin/Win32/wrap_oal.dll,
      OpenAL_Trunk_1471/OpenAL-Windows/Router/Release/Win32/OpenAL32.dll,
      OpenAL_Trunk_1471/OpenAL-Windows/Router/Release/Win32/OpenAL32.pdb

If the Thirdparty is compiled as an static library this field must not declared and can be omitted from the definition file.

Comments

The end of the file is reserved for comments. Any change done to the ThirdParty must be commented here. For example:

# Modifications done to the original project
#
# For ALL projects
#   Debug
#     Runtime library: MT Debug DLL
#   Release
#     Runtime library: DLL
#     Enhanced instruction set: SSE
#
# For ROUTER TEST project
#   Debug
#     Additional include diretories: ..\..\include\al
#     Additional library diretories: ..\router\debug\win32
#   Release
#     Additional include diretories: ..\..\include\al
#     Additional library diretories: ..\router\release\win32
#
# NOTE: Could compile the code for Unicode, but there is no explicit project for this in the
#       current solution
# NOTE: There is no code for Creative EFX-Util.lib, so the version of the SDK is included

Pre-Compiling ThirdParties

Third parties must be precompiled in the repository. That way, they are not being compiled each time a Rebuild All command is executed. And that makes sense because third parties do not often change. The following rules must be followed when compiling the 3rdParties:

  • If possible, always compile as dynamic library instead of static library. [General -> Configuration Type]
  • Always compile with CRT in Multithread DLL (Debug or Release). This is the official CRT mode supported by all the packages and the option that better integrates with the memory manager. [C/C++ -> Code Generation -> Runtime Library]
  • Enable Whole program optimization. [General -> Whole Program Optimization + Linker -> Optimization -> Link Time Code Generation]
  • Enable utilization of SIMD Extensions for 32-bit builds and SIMD Extensions 2 for 64-bit builds. [C/C++ -> Code Generation -> Enable Enhanced Instruction Set]
  • Symbols must me properly configured. No warnings must appear when linking with the ThirdParty
  • For static libraries, the symbol files must be stored in the same directory that the lib. [C/C++ -> Program Database File Name]
  • For dynamic libraries, symbol files must be specified in the BIN option of the definition file. That way, they will be copied to the /Bin directory. [Linker -> Debugging -> Generate Debug Info + Generate Program Database File]
  • Respect as much as possible the project settings provided by the ThirdParty author. This permits easier upgrading. Any change done to the default settings must be clearly described in the definition file.
  • Normally, only an optimized release configuration must be generated. In that case, the Release CRT library must be discarded when linking a package in debug. This can be done with the following commands
#ifdef _DEBUG
// FreeType library is compiled in Release. When being used by a Debug Package we disable
//  Release runtime libraries to avoid conflicts
#pragma comment(linker, "/NODEFAULTLIB:MSVCRT.lib")
#endif

Subversion Integration

Third parties are managed in the repository as described in the subversion documentation: Vendor branches:

  • Original (unmodified) libraries are stored in Noesis/3rdParty (outside NoesisSDK). Inside that directory you will find a folder for each version committed and the folder 'current' with the latest integrated version. Note that this folder is not include in the data that is branched when a new NoesisSDK branch is created.
  • Inside NoesisSDK there is another 3rdParty directory (Noesis/NoesisSDK/3rdParty). Each 3rdParty has its own directory inside this folder. At the root of this directory, you can find the ThirdParty.def described above and a 'local' folder that contains an integration of the latest version being used by the branch and optionally the required changes needed in Noesis for the 3rdParty to work. This structure is repeated for each branch. This way branched versions of NoesisSDK may use different versions of each 3rdParty.

Upgrading a ThirdParty to a new version

  • As described in Vendor branches, it is recommended to use the svn_load_dirs.pl script. It is located in the root of the 3rdParty directory. For it to work you need to install the following prerequisites:
  • ActivePerl, because the script is written in perl

  • A diff utility. It must be added to the PATH environment

  • A subversion client is needed. By default, the svn_load_dirs.pl has been modified to use the svn.exe located in the NsToolkit:

    # Specify the location of the svn command.
    my $svn = 'W:\Noesis\NoesisSDK\NSToolkit\Extra\Svn\svn.exe';
    
  • Unpack the new version of the 3rdparty in a new directory. In our example we are going to unpack it to a folder called 'drop'.

  • Then apply the following command (note that the drop folder is a local directory and the current folder is a remote directory. You do not need to checkout the current folder):

    W:\Noesis\3rdParty\NVTT> perl ..\svn_load_dirs.pl -t 2.0.6
      svn://svn.noesis/Noesis/3rdParty/NVTT current drop
    
  • The script will show you a list with the files that are going to be added and deleted. Before performing this operation it gives you the chance to detect files that have been renamed. It is very important to manually detect these renames to optimize repository storage and bandwidth in the server. The script will be unable to continue if the case-sensitivity of one or more files has changed. You must solve this problem manually before invoking the script. This is the recommended option given by the tortoise manual:

    It is important to rename it within subversion. Just renaming in the explorer will corrupt your working copy!

    • Commit the changes in your working copy.
    • Rename the file from UPPERcase to upperCASE directly in the repository using the repository browser.
    • Update your working copy.
  • The final step is to integrate the new library inside the NoesisSDK branch (a merge operation from Noesis/3rdParty to Noesis/NoesisSDK/3rdParty). During this merge, conflicts between the new library version and the local modifications must be resolved.

© 2017 Noesis Technologies