Shared Libraries - Binary Components

Table of Contents

1 C-Interoperability and Shared Libraries - Binary Components

1.1 Overview

Shared libraries are compiled object-code exporting functions, data structures and classes which can be used by many different programs. They provide many benefits such as: faster compilation time; less disk space usage since each program using the library do not need a copy of the library source code or its compiled-code; ability to update the library and provide security fixes to client applications without recompilation, just by replacing the library file.

On Windows, shared libraries are called DLLs which stands for Dynamic Linked Libraries, on Unix-like operating systems such as Linux, BSD and MacOSX, they are called SO - shared objects or DSO - dynamic shared objects.

Table 1: Comparison of shared libraries across different operating systems
Operating Long Name Short name File Binary Format
System     Extension  
Windows Dynamic Linked Library DLL .dll PE32/PE64 - Portable Executable
Linux (Dynamic) Shared Object DSO or SO .so ELF or ELF64 (for 64 bits processors)
BSD (Dynamic) Shared Object DSO or SO .so ELF or ELF64
MacOSX - dylib .dylib or .so MachO
  • Shared libraries are similar to executables, however they do not have the main() function entry point.

In addition to C and C++, shared libraries, containing functions with C-linkage, can also be used as binary components (libraries) and be consumed from higher level programming languages such as Python, Ruby, Java or C# through some foreign-function interface. In Python C-functions can be imported from shared libraries using the cytpes library; in C# C-functions can be consumed using the P-invoke API and in Java it is possible to use C-functions from shared libraries using the JNI (Java Native Interface) or JNA, which is easier to use.

Note:

  • Shared libraries are specific to a particular operating system and are not part of C++ standard.

Use Cases and advantages:

  • Multiple programs can reuse the same library reducing disk space.
  • Faster compilation once the shared library is compiled.
  • Changes in the library such as security fixes, updates and new features may not require client programs recompilation if the ABI is not broken.
  • Better for security updates. For instance, if a program is statically compiled with open-ssl library, a program recompilation would be required to update the library to a new version with security fixes for some vulnerability. It could even be a greater security problem and waste of space, if all application depending on open-ssl were statically liked with this library. If a program uses the open-sll shared library, the library can be updated just by replacing a file and the program it will not need to be recompiled as long as there are no ABI breaking changes.
  • Plugin system => The main program can contain an interface class called IInterface and load its implementations from shared libraries plugins. The implementation classes provided by the plugins or shared library files allow the program to be updated and extended at runtime without recompilation.
  • Cross language interoperability => Functions using C-linkage can be called from any high level programming language with foreign function interface such as Python with ctypes library. C++ functions and classes cannot be called due to name mangling and compiler ABI issues. The workaround to this problem is to create a C-interface or functions with C-linkage for classes, functions and namespaces. Instances of a class can be passed around as opaque pointers (void*).

Problems:

  • Due to C++ lack of standard ABI Application Binary Interface, classes, STL containers, functions without C-linkage and namespaces may not be reusable with a client program compiled with different compiler than used to build the library. However, even in this case, there is still the benefit of faster compilation time if the shared library and the client application are part of the same project.
    • TL;DR => Classes, STL and functions without C-linkage of shared library cannot be used with a different compiler than the one used to build the DLL.
  • Cross-compiling interoperability - The only way to ensure that a shared library can work with all compilers is by using a functions with C-linkage (functions with extern "C") annotation and creating C-linkage functions wrappers to classes, STL containers and classes.
  • Classes can be made compatible among different compilers by using an interface class, a class with only pure virtual member functions, using only compatible types in the methods signature or declaration, therefore it cannot use STL container types such as std::string or std::vector. However, each implementation of an interface class can use STL containers internally.

MSDN Documentation

Further Reading

1.2 Design Guidelines for C-Interfaces

Macro for C-linkage and exporting symbols

Instead of:

// Does not work on Unix-like OSes. 
// Only works on compilers with supporting this MSVC (Visual VC++) compiler-extension. 
extern "C" __declspec(dllexport)  
double SomeFunctionWithCLinkage(const char* name) {
  ... function code ... 
}

Better export the symbol using a macro for ensuring cross-platform portability:

  • Note: Names of macros should be unique in order to avoid name collisions. This is why the macros are prefixed with library name.
  • Note: __declspec(dllexport) => MSVC (Visual C++) compiler extension only supported on MSVC and Mingw. (No in C++ ISO standard)
  • Note: __attribute__ ((visibility ("default"))) => GCC/Clang compiler extension used on Unix-like Oses for making symbols hidden by default.
#if defined(__cplusplus)
  // Code being compiled with C++ compiler 
  #define MATHTOOLS_EXTERN_C extern "C"
#else 
  // Code being compiled with C compiler (Not C++ compiler)
  #define MATHTOOLS_EXTERN_C 
#endif 

#if defined(_WIN32)
  // MS Windows DLLs (*.dll)
  #define MATHTOOLS_EXPORT_C MATHTOOLS_EXTERN_C __declspec(dllexport)
#else 
  // Unix-like Shared Object (.so) operating systems and GCC.
  #define MATHTOOLS_EXPORT_C MATHTOOLS_EXTERN_C __attribute__ ((visibility ("default")))
#endif 

So, the function code becomes:

MATHTOOLS_EXPORT_C
double SomeFunctionWithCLinkage(const char* name) 
{
  ... function code ... 
}

Name Prefix

Functions with C-linkage are always in the global namespace, therefore every function needs an unique name in order to avoid name clashing which can result in undefined behavior. A good practice to avoid those name collisions is to use a common prefix before in the name of every function, for instance:

Instead of:

MATHTOOLS_EXPORT_C const char* GetVersion();
MATHTOOLS_EXPORT_C double RootSolver(double x0, double (* function) double);

// Or in multi-line style:
MATHTOOLS_EXPORT_C 
const char* GetVersion();

MATHTOOLS_EXPORT_C 
double RootSolver(double x0, double (* function) double);

Better:

// All functions contains the library name as a prefix in order to avoid 
// name clashes.
MATHTOOLS_EXPORT_C const char* mathtools_GetVersion();

MATHTOOLS_EXPORT_C 
double mathtools_RootSolver(double x0, double (* function) double);

Naming Schema for C-wrapers or C-interfaces for C++ classes

Sample class:

class Plotter{
  struct impl; 
  std::unique_ptr<impl> m_pimpl;    
public: 
  Plotter();
  Plotter(double width, double height);
  Plotter(Plotter const& rhs) = default;
 ~Plotter() = default;

 void addPoint(double x, double y);
 void clear();
};

C-wrappers schema:

  • C-interface for default constructor
MATHTOOLS_EXPORT_C
Plotter* mathtools_Plotter_new() 
{
   return new (std::nothrow) Plotter();
}
  • C-interface or C-Wrapepr for second constructor:
MATHTOOLS_EXPORT_C
Plotter* mathtools_Plotter_new2(double width, double height) 
{
   return new (std::nothrow) Plotter(width, height);
}
  • C-interface for copy constructor:
// Wrapper for copy constructor 
MATHTOOLS_EXPORT_C
Plotter* mathtools_Plotter_new_copy(Plotter* hPlotter) 
{
   return new (std::nothrow) Plotter(*hPlotter);
}
  • C-interface for destructor:
MATHTOOLS_EXPORT_C
void mathtools_Plotter_delete(Plotter* hPlotter) 
{
  delete hPlotter;
}
  • C-interface for class method.
    • As C-interfaces cannot throw exceptions, the last parameter is used for returning an error code.
// Schema: <Library Name>_<Class Name>_<method>
MATHTOOLS_EXPORT_C
void mathtools_Plotter_addPoint(Plotter* hPlotter, double x, double y)
{
   hPlotter->addPoint(x, y);
}

Exceptions and Error Handling

  • DO NOT: Throw exceptions in functions with C-linkage (extern "C") as exceptions are incompatibles among different compilers. Moreover, the C-language cannot handle exceptions.
  • DO: Instead of exceptions, return an error code enumeration as a function return value or as function parameter.

Example: If a function throws exceptions such as the function mathtools_Plotter_addPoint, the exceptions should be caught and the function should return an error code eihter as return value or function parameter.

MATHTOOLS_EXPORT_C
void mathtools_Plotter_addPoint(Plotter* hPlotter, double x, double y, int* ErrorCode)
{
    *ErrorCode = ErrorEnum::OK;
    try { 
       // Method addPoint throws exceptions 
       hPlotter->addPoint(x, y);
    } catch(Exception1 const& ex){
       *ErrorCode = ErrorEnum::NO_SPACE_ERROR; // Set with value of some enum 
    } catch(Exception2 const& ex){
       *ErrorCode = ErrorEnum::SOME_ERROR_ENUM;
   }
}

Passing std::string STL Object across a DLL boundary

Type synonym:

// h prefix stands for handle 
using hString = std::string*;

C-interface for std::string default constructor.

LIBRARY_EXPORT_C  
hString libprefix_string_new()
{
   // std::nothrow => Do not throw std::bad_alloc, just return 
   // a null pointer on failure.
   return new (std::nothrow) std::string();   
}

C-interface for std::string copy-constructor.

LIBRARY_EXPORT_C  
hString libprefix_string_new_copy(hString hstr)
{
   // std::nothrow => Do not throw std::bad_alloc, just return 
   // a null pointer on failure.
   return new (std::nothrow) std::string(*hstr);   
}

C-interface for std::string overloaded constructor.

LIBRARY_EXPORT_C  
hString libprefix_string_new2(const char* text)
{
   return new (std::nothrow) std::string(text);   
}

C-interface for std::string destructor.

// Style 1: 
LIBRARY_EXPORT_C 
hString libprefix_string_delete(hString hstr)
{
   delete hstr;
}

// Style 2: 
LIBRARY_EXPORT_C hString 
libprefix_string_delete(hString hstr)
{
   delete hstr;
}

// Style 3: 
LIBRARY_EXPORT_C 
hString 
libprefix_string_delete(hString hstr)
{
   delete hstr;
}

// Style 4: Trailing return-type.
LIBRARY_EXPORT_C 
auto libprefix_string_delete(hString hstr) -> hString 
{
   delete hstr;
}

C-interface for std::string getter, returns const char*.

LIBRARY_EXPORT_C 
const char*
libprefix_string_get(hString hstr)
{
   return hstr->data();
}

C-interface for std::string setter.

LIBRARY_EXPORT_C 
void 
libprefix_string_set(hString hstr, const char* text)
{
   *hstr = text; 
}

C-interface for std::string concat.

LIBRARY_EXPORT_C 
void 
libprefix_string_conat(hString hstr, const char* text)
{
   *hstr = *hstr + text; 
}

C-function to print std::string object

LIBRARY_EXPORT_C 
void 
libprefix_string_print(hString hstr)
{
   std::cout << *hstr << "\n";
}

C-wrapper to method std::string::clear()

LIBRARY_EXPORT_C 
void 
libprefix_string_clear(hString hstr)
{
   hstr->clear();
}

Passing std::vector<double> across a DLL boundary

Handle type:

using hVectorD = std::vector<double>*;

Constructors:

LIBRARY_EXPORT_C 
hVectorD libprefix_vectorD_new()
{
  return new (std::nothrow) std::vector<double>();
}

LIBRARY_EXPORT_C 
hVectorD libprefix_vectorD_new2(size_t n, double x)
{
  return new (std::nothrow) std::vector<double>(n, x);
}

LIBRARY_EXPORT_C 
hVectorD libprefix_vectorD_new3(size_t n, double [] array)
{
  // Iterator-range constructor 
  return new (std::nothrow) std::vector<double>(array, array + n);
}

Destructor:

LIBRARY_EXPORT_C 
void libprefix_vectorD_delete(hVectorD hvec)
{
  delete hvec;
}

Get size:

LIBRARY_EXPORT_C 
size_t libprefix_vectorD_size(hVectorD hvec)
{
  hvec->size();
}

Getter:

LIBRARY_EXPORT_C 
double libprefix_vectorD_get(hVectorD hvec, size_t n)
{
  return hvec->operator[](n);
}

Setter:

LIBRARY_EXPORT_C 
void libprefix_vectorD_set(hVectorD hvec, size_t n, double x)
{
  hvec->operator[](n) = x;
}

Element-wise fundamental math function 'sin()':

// Returns a new vector Out[i],  Out[i] = sin(Input[i]), i = 0, 1, 2... Input.size() 
LIBRARY_EXPORT_C 
hVector libprefix_vectorD_sin(hVectorD hvec)
{
   auto result = new (std::nothrow) std::vector<double>(hvec->size());
   for(size_t i = 0; i <= hvec->size(); i++)
     (*result)[i] = std::sin((*hvec)[i]);
   return result;
}

// Returns a new vector Out[i],  Out[i] = exp(Input[i]), i = 0, 1, 2... Input.size() 
LIBRARY_EXPORT_C 
hVector libprefix_vectorD_exp(hVectorD hvec)
{
   auto result = new (std::nothrow) std::vector<double> (hvec->size());
   for(size_t i = 0; i <= hvec->size(); i++)
     (*result)[i] = std::exp((*hvec)[i]);
   return result;
}

Higher order C-function to apply a function to each element:

// Returns a new vector Out[i],  Out[i] = exp(Input[i]), i = 0, 1, 2... Input.size() 
LIBRARY_EXPORT_C 
hVector 
libprefix_vectorD_map( 
             hVectorD hvec              /* Vector handle */
            ,double Function (double))  /* Function pointer to math function */                                      
{
   auto result = new (std::nothrow) std::vector<double>(hvec->size());
   for(size_t i = 0; i <= hvec->size(); i++)
     (*result)[i] = Function((*hvec)[i]);
   return result;
}

See also:

1.3 Cross platform Shared Library, C-Interfaces and Language Interoperability

1.3.1 Overview

Source:

Shared library Source Code

Sample Client C++ Program:

Sample Python 3 Client code: (Python3 module wrapper library)

GIST - Better for online view:

Compile the DLL libtest.dll and client program with MSVC

Shared Library

$ cl.exe testlib.cpp /EHsc /LD /nologo user32.lib

C++ Client program (client code)

$ cl.exe /EHsc client1.cpp /Fe:client1.exe testlib.lib && out.exe

Compile the DLL libtest.dll and client program with Mingw/GCC

Shared Library

$ g++ testlib.cpp -o testlib.dll -g -fvisibility=hidden -shared -std=c++14 -Wall

C++ Client program

$ g++ client.cpp -o client.exe -g -std=c++14 testlib.dll -Wall

Compile and run from GIST

Clone the GIST (Gisthub).

# Clone GIST: 
$ git clone https://gist.github.com/caiorss/2bba4c50866d9467aaa8c7792b337f71 dllex1
Cloning into 'dllex1'...
remote: Enumerating objects: 66, done.
remote: Counting objects: 100% (66/66), done.
remote: Compressing objects: 100% (47/47), done.
remote: Total 66 (delta 36), reused 45 (delta 19), pack-reused 0
Unpacking objects: 100% (66/66), done.

# Enter directory containing sources
$ cd dllex1/

$ tree .
.
├── build.bat
├── client1.cpp
├── CMakeLists.txt
├── Makefile
├── pywrapper.py
├── testlib.cpp
└── testlib.hpp

0 directories, 7 files

Build shared library on Linux.

  • For Linux, build DLL and client code with make linux.
$ make linux
echo "Build shared library"
Build shared library
g++ testlib.cpp -o libtestlib.so  -std=c++14 -fPIC -shared -Wall
echo "Build client code 1"
Build client code 1
g++ client1.cpp -o client1.bin -std=c++14 libtestlib.so 

$ file libtestlib.so 
libtestlib.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), 
dynamically linked, BuildID[sha1]=52a16d16991d7aa71763140e97a1422c1387711d, not stripped

$ file client1.bin 
client1.bin: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, 
interpreter /lib64/ld-linux-x86-64.so.2, 
for GNU/Linux 3.2.0, BuildID[sha1]=1a4885e112f72372cfd66816a564f49a24110c50, not stripped

Build shared library on Windows:

  • Just run the batch script build-windows.bat or double click at it.
$ build-windows.bat
**********************************************************************
** Visual Studio 2017 Developer Command Prompt v15.5.6
** Copyright (c) 2017 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
testlib.cpp
   Creating library testlib.lib and object testlib.exp
Microsoft (R) C/C++ Optimizing Compiler Version 19.12.25835 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

client1.cpp
Microsoft (R) Incremental Linker Version 14.12.25835.0
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:client1.exe
client1.obj
testlib.lib
   Creating library client1.lib and object client1.exp
 [StaticObject] => Initialize DLL
testlib.cpp:48: <DllMain> DLL Loaded into the process => PID = 1240
testlib.cpp:54: <DllMain> DLL attached to process.

=== EXPERIMENT 1 ===> Import C++ functions from DLL
client1.cpp:38: <main> Main process starts here.
 => Linalg::norm(xs) 7.4162
=> xs = [5]( 1, 2, 3, 4, 5,  )
=== EXPERIMENT 2 ===> Import class from DLL
 Instance created with name = Dummy

 .. ... ...  .. ... ...  .. ... ...  .. ... ... 

Run C++ client code:

$ ./client1.bin 
 [StaticObject] => Initialize DLL

=== EXPERIMENT 1 ===> Import C++ functions from DLL
client1.cpp:38: <main> Main process starts here.
 => Linalg::norm(xs) 7.4162
=> xs = [5]( 1, 2, 3, 4, 5,  )
=== EXPERIMENT 2 ===> Import class from DLL
 Instance created with name = Dummy
 Counter set to value = 100
cls.getName() = Dummy
    cls.get() = 100

... ... ... 

Run Python client code:

$ make py
python3 pywrapper.py
 [INFO] libpath = libtestlib.so
 [StaticObject] => Initialize DLL
Intializing library
Library initialized OK.

 .. ... ... ... 

1.3.2 Header macros

Sources:

teslib.hpp / Heade File - Macros

The header files uses the following macros to reduce the boilerplate necessary to export the library functions and classes.

  • file: testlib.hpp => Macros for exporting functions and classes.

Macro: EXPORT_CPP

/** Macro EXPORT_CPP makes a symbol visible. */
#if defined(_WIN32)
  #define EXPORT_CPP __declspec(dllexport)
#else
  #define EXPORT_CPP __attribute__ ((visibility ("default")))
  // If not compiled on Windows, remove declspec compiler extension.
  #ifndef __declspec
    // For GCC only - make exported symbol visible symbol 
    #define __declspec(param) __attribute__ ((visibility ("default")))
  #endif
#endif 

The macro EXPORT_CPP annotates the symbol as visible by expanding to:

// On Windows EXPORT_CPP becomes 
__declspec(dllexport)

// On Unix-like OSes, Linux, BSD, OSX ... 
 __attribute__ ((visibility ("default")))

So a function int SomeFunction(double x, double y) becomes:

 // Makes this symbol visible (exported) in the library (DLL)
 EXPORT_CPP int SomeFunction(double , double);

// On Windows: 
__declspec(dllexport) int SomeFunction(double , double);

// On Unix-like operating systems or Linux: 
__attribute__ ((visibility ("default"))) int SomeFunction(double , double);

Macro EXPORT_C:

/* Macro EXPORT_C is used for exporting symbol with C-linkage, it
 * means, without name mangling */
#ifdef __cplusplus
  // extern "C" - Indicates that a given symbol/function has C-linkage and
  // does not have name mangling.
  // 
  // On Linux EXPORT_C becomes
  //  => extern "C" __attribute__ ((visibility ("default")))
  // 
  // On Windows EXPORT_C becomes 
  //  => extern "C" __declspec(dllexport)
  #define EXPORT_C extern "C" EXPORT_CPP
#else
  // If a C-compiler uses this header, remove 'extern "C"'
  #define EXPORT_C  EXPORT_CPP
#endif 

Export symbols with C-linkage, it means that the function is exported without name decoration or name mangling. The function symbol matches exactly its name. Functions with C-linkage can be called by C-code; C++ code compiled with other compilers without ABI issues and foreign function interfaces of scripting languages such as Python Ctypes or C# Pinvoke. Note: If a function is defined with C-linkage, it can only use C-compatible types. So, it is no possible to define functions with C-linkage with overload, default parameter or non C-compatible arguments.

  • On Windows, this macro is expanded to:
EXPORT_C int functionWith_C_linkage(MyClass* pointerToClass, int x, double z);

// On Windows it becomes: 
extern "C" __declspec(dllexport)  int functionWith_C_linkage(MyClass* pointerToClass, int x, double z);

// This function could also be declared as:
extern "C" __declspec(dllexport) 
int functionWith_C_linkage(MyClass* pointerToClass, int x, double z);
  • On Unix-like OSes or linux, the macro EXPORT_C expands to:
EXPORT_C int functionWith_C_linkage(MyClass* pointerToClass, int x, double z);

// On Linux, it becomes:
extern "C" __attribute__ ((visibility ("default"))) int functionWith_C_linkage(MyClass* pointerToClass, int x, double z);

// This function could also be declared as:
extern "C" __attribute__ ((visibility ("default"))) 
int functionWith_C_linkage(MyClass* pointerToClass, int x, double z);

On Unix-like operating system (GCC or Clang compilers), all functions in an object-code (aka compiled code) becomes visible by default what can lead to undefined behavior when there are name clashes due to repeated symbols. With appropriate compiler options such as -fvisibility=hidden, the symbols can be made private by default as happens in Windows where is necessary to annotate the symbol as visible explicitly with __declspec(dllexport). On Unix-like OSes for GCC and Clang compiler the annotation for making the symbol visible is __attribute__ ((visibility ("default"))). This statement has no effect if the appropriate compiler flags are not set.

1.3.3 Namespace Linalg

Functions of Namespace Linalg

Namespace containing sample linear algebra functions, the function norm computes an Euclidean norm of a N-dimension vector, the function linTransform computes a linear transformation performing the computation v[i] * a + b for each vector element.

On Windows, all DLL symbols, such as functions, variables and clases, are private by default, they are not exported by default as happens in Unix-like OSes shared libraries. Therefore, in order to a function be exported in a Windows shared library, it is necessary to use the MSVC or visual C++ compiler extension __declspec(dllexport) before the function declaration.

  • Interface => File: testlib.hpp
#ifdef __cplusplus

namespace Linalg {
        EXPORT_CPP
        double norm(const std::vector<double>& xs);

        EXPORT_CPP std::vector<double>
        linTransform(
                double a,
                double b,
                std::vector<double>& xs
                );

        EXPORT_CPP
        std::ostream&
        printVector(std::ostream& os, std::vector<double>& xs); 
}
#endif 

The compiler directives #ifdef __cplusplus … #endif are used for disabling this code block when the header is used by a C-compiler as the C-language does not support classes, namespaces and many other C++ features.

  • file: testlib.cpp - Implementation
// Linear algebra tools 
 namespace Linalg{
         EXPORT_CPP 
          double norm(const std::vector<double>& xs){
                double sum = 0.0;
                for(const auto& x : xs) sum += x * x;
                return std::sqrt(sum);
        }

        EXPORT_CPP
        std::vector<double>
        linTransform(double a, double b, std::vector<double>& xs){
                std::vector<double> out(xs.size());
                for(size_t i = 0; i < xs.size(); i++){      
                        out[i] = a * xs[i] + b;
                }
                return out;
        }

        EXPORT_CPP
        std::ostream&
        printVector(std::ostream& os, std::vector<double>& xs){
                os << "[" << xs.size() << "]( ";
                for(const auto&  x: xs)
                        os << x << ", ";
                return os << " )";
        }   
}

1.3.4 C-interface of Namespace Linalg

C-interface of Namespace Linalg

A C-interface is necessary for ensuring that the DLL can be called from a C-program or a foreign function interface of a higher level language.

  • File: testlib.hpp => Function declaration.
// ======= C-interface for Linalg namespace =========//

/** Handle or opaque pointer for std::vector<double> */
typedef void* hVectorD;

/* ----- C-Wrappers for Linalg namespace ---- */

EXPORT_C 
double testlib_vectorD_Linalg_norm(hVectorD hv);

EXPORT_C 
void testlib_vectorD_Linalg_printVector(const char* name, hVectorD hv);

  • File: testlib.cpp => C-interface function implementations.

The type synonym hVectorD is handle or a opaque poiter for the type std::vector<double>. It is a workaround, to pass std::vector<double> accross a DLL functions with C-linkage and use std::vector<double> from C programs or foreign function interfaces.

//=========== C-wrappers ---------------///
// Handler for double vector 
using hVectorD = void*;
using pVectorD = std::vector<double>*;

/** C-wrapper for vector<double> constructor */
EXPORT_C  
hVectorD testlib_vectorD_make0(size_t n, double x){
    return new std::vector<double>(n, x);
}

/** C-wrapper for range constructor */
EXPORT_C
hVectorD testlib_vectorD_make1(size_t n, double array []){
    return new std::vector<double>(array, array + n);
}

/** C-wrapper for setting elements of vector<double> */
EXPORT_C 
void testlib_vectorD_set(hVectorD hv, size_t n, double x){
   reinterpret_cast<pVectorD>(hv)->operator[](n) = x;
}

/** C-wrapper for vector<double> destructor */
EXPORT_C void testlib_vectorD_delete(hVectorD hv){
    delete reinterpret_cast<pVectorD>(hv);
}

/** C-wrapepr for Linalg::norm function */
EXPORT_C 
double testlib_vectorD_Linalg_norm(hVectorD hv){
    return Linalg::norm(*reinterpret_cast<pVectorD>(hv));
}

EXPORT_C 
void testlib_vectorD_Linalg_printVector(const char* name, hVectorD hv){
    std::cout << name << " = ";
    Linalg::printVector(std::cout, *reinterpret_cast<pVectorD>(hv));
    std::cout << std::endl;
}

1.3.5 Non-polymorphic Class - SampleClass

  • File: testlib.hpp => Class declaration.

In order to a class be exported in a Windows' shared library, it requires the annotation __declspec(dllexport).

Note: It will not be possible to compiled client program using this class with a compiler different from the used to build the shared library libtest.dll. It happens because, the C++ ABI - Application Binary Interface is not the same accross different compilers. If the DLL is compiled with MSVC, the only way to use this class from a program compiled using MingW or even a different version of MSVC is by using a C-interface (functions with C-linkage) and opaque pointers (void* pointers).

// ======= Non-polymorphic class exported by DLL =========//

#ifdef __cplusplus
// Non-polymorphic class 
class EXPORT_CPP SampleClass{
public:
        SampleClass();
        SampleClass(const std::string& name);   
        ~SampleClass();
        std::string getName() const;
        int get();
        void set(int n);    
private:
        std::string m_name;
        int m_counter;  
};
#endif 
  • File: testlib.cpp => Class implementation.
SampleClass::SampleClass(const std::string& name)
   : m_name(name), m_counter(0)
{
    std::cout << " Instance created with name = " << m_name << std::endl;
}
/** Delegated constructor on right-hand-side */
SampleClass::SampleClass(): SampleClass("unnamed"){}   

SampleClass::~SampleClass(){
    std::string text = std::string("SampleClass => name = ") + m_name +  " deleted";
    DbgTrace(text);
}

std::string SampleClass::getName() const {
    return m_name;
}
int SampleClass::SampleClass::get(){        
    return m_counter;
}
void SampleClass::set(int n){
    std::cout << " Counter set to value = " << n << std::endl;
    m_counter = n;
}   

1.3.6 C-interface for SampleClass

  • File: testlib.hpp => Functions declarations.
/* ----- C-Wrappers for SampleClass namespace ---- */
using hSampleClass = void*;

/** Nullable constructor zero-arg constructor */
extern "C" __declspec(dllexport)
hSampleClass
testlib_SampleClass_make0();

/** Other constructor */
EXPORT_C hSampleClass testlib_SampleClass_make1(const char* name);

/** Destructor */
EXPORT_C
void
testlib_SampleClass_delete(hSampleClass hnd);

/** Wrapper for get method */
EXPORT_C
int
testlib_SampleClass_get(hSampleClass hnd);

/** Wrapper for set method */
EXPORT_C
void
testlib_SampleClass_set(hSampleClass hnd, int n);

EXPORT_C
const char*
testlib_SampleClass_getName(hSampleClass hnd);

  • File: testlib.cpp => Functions implementation.
/* ----- C-Wrappers for SampleClass namespace ---- */

using hSampleClass = void*;

/** Nullable constructor zero-arg constructor */
EXPORT_C 
hSampleClass
testlib_SampleClass_make0();

/** Other constructor */
EXPORT_C hSampleClass testlib_SampleClass_make1(const char* name);

/** Destructor */
EXPORT_C
void
testlib_SampleClass_delete(hSampleClass hnd);

/** Wrapper for get method */
EXPORT_C
int
testlib_SampleClass_get(hSampleClass hnd);

/** Wrapper for set method */
EXPORT_C
void
testlib_SampleClass_set(hSampleClass hnd, int n);

EXPORT_C
const char*
testlib_SampleClass_getName(hSampleClass hnd);

1.3.7 Polymorphic class InterfaceClass

  • File: testlib.hpp => Class declaration.

As the name implies, this class is an interface class, which is a class containing only pure virtual functions (abstract methods). As a result, it is declared only in the header file and cannot be instatiated directly.

This class uses only C-types to be binary compatible with different compilers, if it used STL containers such as string or any other non-compatible C-type, it would not be possible to compile a C++ program using Mingw/GCC against this DLL built with MSVC.

// Polymorphic Interface class binary compatible across different
// compilers as it does not use any STL container on the interface.
#ifdef __cplusplus
struct InterfaceClass{
        /* Returns class unique ID */
        virtual const char* getID() const = 0;
        /** Set class internal state */
        virtual void setName(const char* name) = 0;
        virtual const char* getName() = 0;

        /** Virtual constructor */
        virtual ~InterfaceClass() = default;
        // virtual ~InterfaceClass();
};
#else
  #define InterfaceClass void 
#endif 
  • File: testlib.cpp => class ImplementationA of interface InterfaceClass.
class ImplementationA: public InterfaceClass
{
private:
    std::string m_name;
public:
    static constexpr const char* class_id = "ImplementationA";

    ImplementationA(): m_name("Unammed-A"){ }
    ImplementationA(const std::string& name)
       : m_name(name){} 
    ~ImplementationA(){
         std::cout << " [INFO] ImplementationA deleted => name = "
                   << m_name
                   << " ; type = " << class_id
                   << std::endl;
    }
    const char* getID() const {     
        return class_id;
    }
    void setName(const char* name) {
        m_name = name;
    }
    const char* getName() {
        return m_name.c_str();
    }
};
  • File: testlib.cpp => class ImplementationB of interface InterfaceClass.
class ImplementationB: public InterfaceClass
{
private:
    std::string m_name;
public:
    static constexpr const char* class_id = "ImplementationB";

    ImplementationB(): m_name("Unammed-B"){ }
    ImplementationB(const std::string& name)
       : m_name(name){} 
    ~ImplementationB(){
        std::cout << " [INFO] ImplementationB deleted => name = "
                  << m_name
                  << " ; type = " << class_id
                  << std::endl;
    }
    const char* getID() const {     
            return class_id;
    }
    void setName(const char* name) {
            m_name = name;
    }
    const char* getName() {
            return m_name.c_str();
    }
};

  • File: testlib.hpp => C-interface and factory function to load implementations from the DLL.
/** Factory function */
EXPORT_C InterfaceClass* teslib_InterfaceClass_factory(const char* class_id);

/** C-wrapper for destructor */
EXPORT_C void testlib_InterfaceClass_delete(InterfaceClass* hinst);

/** C-wrapper for getID method */
EXPORT_C const char* testlib_InterfaceClass_getID(InterfaceClass* hinst);

EXPORT_C void testlib_InterfaceClass_setName(InterfaceClass* hinst, const char* name);

EXPORT_C const char* testlib_InterfaceClass_getName(InterfaceClass* hinst);
  • File: testlib.cpp => C-interfaces and factory functions definitions.
EXPORT_C InterfaceClass*
teslib_InterfaceClass_factory(const char* class_id)
{
    auto s = std::string(class_id);
    if(s == "ImplementationA")
            return new ImplementationA();
    if(s == "ImplementationB")
            return new ImplementationB();
    return nullptr;
}

EXPORT_C void testlib_InterfaceClass_delete(InterfaceClass* hinst)
{
    delete hinst;
}
EXPORT_C
const char* testlib_InterfaceClass_getID(InterfaceClass* hinst)
{
    return hinst->getID();
}
EXPORT_C
void testlib_InterfaceClass_setName(InterfaceClass* hinst, const char* name)
{
    hinst->setName(name);
}
EXPORT_C
const char* testlib_InterfaceClass_getName(InterfaceClass* hinst){
    return hinst->getName();
}

1.3.8 DLL startup function DLLMain

When a Windows shared library is loaded at compile-time or at runtime by some process, the function DLLMain is always invoked, it is similar to the function main() from an executable.

Note: logging statements with printf, std::cout, std::cerr will not printed if the DLL is loaded by some non-console program, a program compiled to the window subsystem. Therefore, the easiest way to log is to use the OutputDebugString function which output can viewed by the DebugView sysinternal tool.

  • File: testlib.cpp => DLL main function.
/** - DLL Entry point - main function of DLL which is executed when 
    the DLL is loaded by some process. 
 */
#if defined(_WIN32)
extern "C" __declspec(dllexport)
BOOL APIENTRY DllMain (HINSTANCE hInst, DWORD reason, LPVOID lpReserved)
{
    std::string text =
            std::string("DLL Loaded into the process => PID = ")
            + std::to_string(::GetCurrentProcessId());
    WindbgTrace(text);
    DbgTrace(text);

    switch (reason)
    {
    case DLL_PROCESS_ATTACH:
        WindbgTrace("DLL attached to process");
        DbgTrace("DLL attached to process.");           
      break;
    case DLL_PROCESS_DETACH:
         WindbgTrace("=> DLL detached.");
         DbgTrace("=> DLL attached");
      break;
    case DLL_THREAD_ATTACH:
        WindbgTrace("DLL attached to thread");
        DbgTrace("DLL detached to thread.");
      break;
    case DLL_THREAD_DETACH:
        WindbgTrace("DLL detached from thread");
        DbgTrace("DLL detached from thread.");
      break;
    }

  return TRUE;
}
#endif 

Another more portable way to peform some way action when the DLL is loaded by a process or unloaded is to use a static object. The startup task is executed int the object's constructor and the finalization task, which happens when the process ends, is executed at object's destructor. This approach may a better replacement for DLLMain function, which is specific for Windows, as this technique works both on Windows and any other Unix-like operating system.

  • File: testlib.cpp => Static Object for replacing DLLMain

The class _StaticObject and a static instance of it is defined within an anonymous namespace to make them private to the compilation unit testlib.cpp and not allow them to be used from anywhere else.

// Class private to this compilation unit - cannot be accessed from
// any other file
namespace {
   class _StaticObject{     
   public:
       using Action = std::function<void ()>;
       Action m_end;

       _StaticObject(Action init, Action end)
               :  m_end(end)
       {
               init();
       }
       ~_StaticObject(){ m_end(); }
   };

   // Static object for replacing DLLMain 
   auto initDLL = _StaticObject(
      []{
            std::cout << " [StaticObject] => Initialize DLL"
                       << std::endl;                        
      },
      []{   
           std::cout << " [StaticObject] => Shutdown DLL"
                     << std::endl;                              
      });
}

1.3.9 DLL entry point for run32dll.exe

This function is a DLL entry point for the Windows program rundll32.exe which can execute this function with:

$ rundll32.exe testlib.dll,entryPoint1
$ rundll32.exe testlib.dll,entryPoint1 arg0 arg1 arg2 ... argn 

When the DLL testlib.dll is run with run32dll calling the function entryPoint1, it will display a message box.

Note: It is an optional function which could be used for installing the DLL in the system or creating some entry in Windows' registry.

  • File: testlib.cpp
#if defined(_WIN32)
extern "C" __declspec(dllexport)
void entryPoint1(HWND hwn, HINSTANCE hinst, LPSTR cmdLine, int nCmdShow){  
     DbgDisp(cmdLine);
     OutputDebugString("Rudll32 called entryPoint1()");
     MessageBoxA(NULL, "DLL ENTRY POINT", "Entry point 1", 0);
}
#endif 

1.3.10 C++ Client program - client1.cpp

File client1.cpp

Includes:

  • Note: (#include "testlib.hpp") a header file between quotes, indicates to the compiler that the header is in the same directory as the source file client1.cpp.
#include <iostream>
#include <ostream>
#include <vector>
#include <string>

#include "testlib.hpp"

Declaration of functions with C-linkages not declarated in the header testlib.hpp

extern "C" hVectorD testlib_vectorD_make0(size_t n, double);
extern "C" hVectorD testlib_vectorD_make1(size_t n, double array []);
extern "C" void testlib_vectorD_delete(hVectorD hv);

Main function - Experiment 1 and Experiment 2:

#ifndef DISABLE
std::cout << "\n=== EXPERIMENT 1 ===> Import C++ functions from DLL" << std::endl;

DbgTrace("Main process starts here.");
std::vector<double> xs{1.0, 2.0, 3.0, 4.0, 5.0};
std::cout << " => Linalg::norm(xs) " << Linalg::norm(xs) << std::endl;
std::cout << "=> xs = "; Linalg::printVector(std::cout, xs); std::cout << std::endl;

std::cout << "=== EXPERIMENT 2 ===> Import class from DLL" << std::endl;
auto cls = SampleClass("Dummy");
cls.set(100);
std::cout << "cls.getName() = " << cls.getName() << std::endl;
std::cout << "    cls.get() = " << cls.get() << std::endl;
#endif // -- eof DISABLE flag

Main Function - Experiment 3

//=========>> Load functions and classes using C-interface ==============//

std::cout << "\n== EXPERIMENT 3 ===> Import C-functions from DLL - C-interface" << std::endl;

double arr [] = {1, 2, 3, 4, 5};
hVectorD v1 = testlib_vectorD_make1(5, arr);
testlib_vectorD_Linalg_printVector("v1", v1);
std::cout << "norm(v1) = " << testlib_vectorD_Linalg_norm(v1) << std::endl;

testlib_vectorD_delete(v1);

Main Functions - Experiment 4

std::cout << "\n== EXPERIMENT 4 ===> Non-polymorphic class with C-interface " << std::endl;

hSampleClass hcls = testlib_SampleClass_make1("[EXPERIMENT4]ClassHandle-OOP-C-API");
std::cout << "[EXPERIMENT 4] hcls.getName() = " << testlib_SampleClass_getName(hcls) << std::endl;
testlib_SampleClass_set(hcls, 100);
std::cout << "[EXPERIMENT 4] hcls.get()     = " << testlib_SampleClass_get(hcls) << std::endl;
testlib_SampleClass_set(hcls, 200);
std::cout << "[EXPERIMENT 4] hcls.get()     = " << testlib_SampleClass_get(hcls) << std::endl;

testlib_SampleClass_delete(hcls);   

Main Functions - Experiment 5

std::cout << "\n== EXPERIMENT 5 ===> Load polymorphic classes from DLL " << std::endl;

InterfaceClass* hinstA = teslib_InterfaceClass_factory("ImplementationA");
InterfaceClass* hinstB = teslib_InterfaceClass_factory("ImplementationB");

std::cout << " => hinstA->getID() = " << hinstA->getID() << std::endl;
std::cout << " => hinstA->getID() = " << hinstB->getID() << std::endl;
hinstA->setName("ClassA-implA");
hinstB->setName("ClassB-implB");
std::cout << " => hinstA->getName() = " << hinstA->getID() << std::endl;
std::cout << " => hinstB->getName() = " << hinstB->getID() << std::endl;

// Note: If delete is used directly to delete hinstA and hinstB,
// a segmentatin fault will happen whenc compiling with Mingw/GCC
testlib_InterfaceClass_delete(hinstA);
testlib_InterfaceClass_delete(hinstB);

std::cout << " [INFO] After deleting instances" << std::endl;

DbgTrace("Program ended OK.");

Compile and run client code with MSVC

Note: Both the DLL and the client code are compiled with MSVC-2017 64 bits target.

$ cl.exe /EHsc client.cpp /nologo /Fe:client1.exe testlib.lib && client1.exe

# Output: 
client.cpp
   Creating library client1.lib and object client1.exp
 [StaticObject] => Initialize DLL
testlib.cpp:39: <DllMain> DLL Loaded into the process => PID = 1112
testlib.cpp:45: <DllMain> DLL attached to process.

=== EXPERIMENT 1 ===> Import C++ functions from DLL
client.cpp:33: <main> Main process starts here.
 => Linalg::norm(xs) 7.4162
=> xs = [5]( 1, 2, 3, 4, 5,  )
=== EXPERIMENT 2 ===> Import class from DLL
 Instance created with name = Dummy
 Counter set to value = 100
cls.getName() = Dummy
    cls.get() = 100

== EXPERIMENT 3 ===> Import C-functions from DLL - C-interface
v1 = [5]( 1, 2, 3, 4, 5,  )
norm(v1) = 7.4162

== EXPERIMENT 4 ===> Non-polymorphic class with C-interface 
 Instance created with name = [EXPERIMENT4]ClassHandle-OOP-C-API
[EXPERIMENT 4] hcls.getName() = [EXPERIMENT4]ClassHandle-OOP-C-API
 Counter set to value = 100
[EXPERIMENT 4] hcls.get()     = 100
 Counter set to value = 200
[EXPERIMENT 4] hcls.get()     = 200
testlib.cpp:159: <SampleClass::~SampleClass> SampleClass => name = [EXPERIMENT4]ClassHandle-OOP-C-API deleted

== EXPERIMENT 5 ===> Load polymorphic classes from DLL 
 => hinstA->getID() = ImplementationA
 => hinstA->getID() = ImplementationB
 => hinstA->getName() = ImplementationA
 => hinstB->getName() = ImplementationB
 [INFO] ImplementationA deleted => name = ClassA-implA ; type = ImplementationA
 [INFO] ImplementationB deleted => name = ClassB-implB ; type = ImplementationB
 [INFO] After deleting instances
client.cpp:87: <main> Program ended OK.
testlib.cpp:159: <SampleClass::~SampleClass> SampleClass => name = Dummy deleted
testlib.cpp:39: <DllMain> DLL Loaded into the process => PID = 1112
testlib.cpp:49: <DllMain> => DLL attached
 [StaticObject] => Shutdown DLL

[FAILURE] ABI - Issue - Compile with Mingw client1.cpp against testlib.dll built with MSVC

The client code is compiled with Mingw and the library was built with MSVC-2017 for 64 bits target.

Build and run client [FAILURE]:

$ g++ client1.cpp -g -o client1-gcc.exe -std=c++14 testlib.dll  && client1-gcc.exe
C:\Users\archbox\AppData\Local\Temp\cckNg9eZ.o: In function `main':
client1.cpp:35: undefined reference to `Linalg::norm(std::vector<double, std::allocator<double> > const&)'
client1.cpp:36: undefined reference to `Linalg::printVector(std::ostream&, std::vector<double, std::allocator<double> >&)'
client1.cpp:39: undefined reference to `SampleClass::SampleClass(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
client1.cpp:40: undefined reference to `SampleClass::set(int)'
client1.cpp:41: undefined reference to `SampleClass::getName[abi:cxx11]() const'
client1.cpp:42: undefined reference to `SampleClass::get()'
client1.cpp:39: undefined reference to `SampleClass::~SampleClass()'
client1.cpp:39: undefined reference to `SampleClass::~SampleClass()'
collect2.exe: error: ld returned 1 exit status
Compilation exited abnormally with code 1 at Sun Dec 16 17:41:47

The compilation fails because C++ does not have a standard and stable ABI - Application Binary Interface, as a result, it is not possible to use an object-code (here - any type of compiled code) built by a different compiler than the current one. The ABI comprises the name mangling schema or name decoration schema which is compiler-dependent, padding, class memory layout and so on. In addition to ABI issues, the STL - Standard Template Library implementations may not be compatible and also not be the same across different compilers.

The only way to ensure that a DLL can work with all possible compilers is to use a functions with C-linkage (extern "C") with opaque pointers (void*) for passing around classes and wrapped STL containers. Classes can be used by different compilers, only if they are interface classes (classes with only pure virtual functions) containing only C-compatible types in the declaration.

Compile with Mingw client1.cpp against testlib.dll built with MSVC

The compilation works when compiling with the custom flag -DDISABLE. The directive #ifndef DISABLE … #endif removes all usages of the namespace Linalg and the class SampleClass. This flag makes main the function use only functions with C-linkage and the interface class InterfaceClass.

$ g++ client.cpp -g -o client-gcc.exe -std=c++14 testlib.dll -DDISABLE   && client-gcc.exe

   [StaticObject] => Initialize DLL
  testlib.cpp:39: <DllMain> DLL Loaded into the process => PID = 5724
  testlib.cpp:45: <DllMain> DLL attached to process.

  == EXPERIMENT 3 ===> Import C-functions from DLL - C-interface
  v1 = [5]( 1, 2, 3, 4, 5,  )
  norm(v1) = 7.4162

  == EXPERIMENT 4 ===> Non-polymorphic class with C-interface 
   Instance created with name = [EXPERIMENT4]ClassHandle-OOP-C-API
  [EXPERIMENT 4] hcls.getName() = [EXPERIMENT4]ClassHandle-OOP-C-API
   Counter set to value = 100
  [EXPERIMENT 4] hcls.get()     = 100
   Counter set to value = 200
  [EXPERIMENT 4] hcls.get()     = 200
  testlib.cpp:159: <SampleClass::~SampleClass> SampleClass => name = [EXPERIMENT4]ClassHandle-OOP-C-API deleted

  == EXPERIMENT 5 ===> Load polymorphic classes from DLL 
   => hinstA->getID() = ImplementationA
   => hinstA->getID() = ImplementationB
   => hinstA->getName() = ImplementationA
   => hinstB->getName() = ImplementationB
   [INFO] ImplementationA deleted => name = ClassA-implA ; type = ImplementationA
   [INFO] ImplementationB deleted => name = ClassB-implB ; type = ImplementationB
   [INFO] After deleting instances
  client.cpp:87: <main> Program ended OK.
  testlib.cpp:39: <DllMain> DLL Loaded into the process => PID = 5724
  testlib.cpp:49: <DllMain> => DLL attached
   [StaticObject] => Shutdown DLL

1.3.11 Load DLL in Python3 REPL

Functions with C-linkage can be called by Python using the ctypes library. The following code presents how it can be done:

Table for C types / Python ctypes conversion

C-type Python C-types C-type  
char ctypes.c_char char* ctypes.POINTER(ctypes.c_char)
int ctypes.c_int int* ctypes.POINTER(ctypes.c_int)
size_t ctypes.c_int -  
double ctypes.c_double double* ctypes.POINTER(ctypes.c_double)
void None void* ctypes.c_void_p
       
       
  • Get a Python string from a const char* ptr or char* ptr, use ctypes.string_at(ptr).
  • Pass a python string to an API with const char* (ctypes.POINTER(ctypes.c_char)) as argument, use Function("string-argument".encode('utf-8'))

Python C-types documentation and further reading:

Load the library:

C:\Users\archbox\Desktop\experiments
λ "C:\Users\archbox\Miniconda3\pkgs\python-3.6.5-h0c2934d_0\python"
Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

>>> import ctypes

# This messages is print by the DLLMain function 
>>> lib = ctypes.cdll.LoadLibrary("testlib.dll")
 [StaticObject] => Initialize DLL
testlib.cpp:39: <DllMain> DLL Loaded into the process => PID = 4280
testlib.cpp:45: <DllMain> DLL attached to process.
>>> testlib.cpp:39: <DllMain> DLL Loaded into the process => PID = 4280
testlib.cpp:53: <DllMain> DLL detached to thread.

Check whether exported functions with C-linkage exist:

>>> lib.testlib_vectorD_make0
<_FuncPtr object at 0x0000021BFCAFF388>

>>> lib.testlib_vectorD_Linalg_printVector
<_FuncPtr object at 0x0000021BFCAFF458>
>>>

>>> lib.testlib_vectorD_delete
<_FuncPtr object at 0x0000021BFCAFF528>

# Function which does not exist.
>>> lib.testlib_InterfaceClass_do_not_exist
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\archbox\Miniconda3\pkgs\python-3.6.5-h0c2934d_0\lib\ctypes\__init__.py", line 361, in __getattr__
    func = self.__getitem__(name)
  File "C:\Users\archbox\Miniconda3\pkgs\python-3.6.5-h0c2934d_0\lib\ctypes\__init__.py", line 366, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: function 'testlib_InterfaceClass_do_not_exist' not found

Load std::vector<double> and Linalg namespace wrapper functions

Set up the std::vector<double> functions to be imported.

  • Note: hVectorD => Handle for std::vector<double> is the same as void* or an opaque pointer.
# Funciton: hVectorD testlib_vectorD_make0(size_t n, double x)
# Set function arguments
lib.testlib_vectorD_make0.argtypes = [ctypes.c_int, ctypes.c_double]
# Set return type 
lib.testlib_vectorD_make0.restype  = ctypes.c_void_p

# void testlib_vectorD_Linalg_printVector(const char* name, hVectorD hv)
lib.testlib_vectorD_Linalg_printVector.argtypes = [ctypes.POINTER(ctypes.c_char), ctypes.c_void_p ]
lib.testlib_vectorD_Linalg_printVector.restype = None 

# Set vector elements hv[n] = x
# void testlib_vectorD_set(hVectorD hv, size_t n, double x)
lib.testlib_vectorD_set.argtypes = [ctypes.c_void_p, ctypes.c_int, ctypes.c_double]
lib.testlib_vectorD_set.restype  = None

Testing C-interface functions for std::vector<double>:

# Creating a std::vector<double> with 4 elements equal to 3.0 
>>> vec1 = lib.testlib_vectorD_make0(4, 3.0)

# This number is the memory address held by the opaque pointer,
# it cannot be used directly. 
>>> vec1
2319184200912
>>>
# ========== Modify vector elements ======== #
# Printarray2 the wrapped std::vector<double>
>>> lib.testlib_vectorD_Linalg_printVector("vec1".encode('utf-8'), vec1)
vec1 = [4]( 3, 3, 3, 3,  )
>>>

>>> lib.testlib_vectorD_set(vec1, 0, 5.0)
>>> lib.testlib_vectorD_set(vec1, 1, 10.5)
>>> lib.testlib_vectorD_set(vec1, 2, 4.78)

>>> lib.testlib_vectorD_Linalg_printVector("vec1".encode('utf-8'), vec1)
vec1 = [4]( 5, 10.5, 4.78, 3,  )

Compute vector Euclidian norm:

# Wrapper for function 
# double Linalg::norm(const std::vector<double>& xs)
# double testlib_vectorD_Linalg_norm(hVectorD hv)
lib.testlib_vectorD_Linalg_norm.argtypes = [ ctypes.c_void_p ]
lib.testlib_vectorD_Linalg_norm.restype  = ctypes.c_double

>>> lib.testlib_vectorD_Linalg_printVector("vec1".encode('utf-8'), vec1)
vec1 = [4]( 5, 10.5, 4.78, 3,  )

>>> lib.testlib_vectorD_Linalg_norm(vec1)
12.926654632966722

# Check if calculations are right 
>>> math.sqrt(5 * 5 + 10.5 * 10.5 + 4.78 * 4.78 + 3 * 3)
12.926654632966722
>>>

Delete vector vec1 using its destructor function:

# Function: void testlib_vectorD_delete(hVectorD hv)
lib.testlib_vectorD_delete.argtypes = [ ctypes.c_void_p ]
lib.testlib_vectorD_delete.restype = None

>>> lib.testlib_vectorD_delete(vec1)

Load wrapper function for interface class InterfaceClass

Load factory function:

  • void* teslib_InterfaceClass_factory(const char* class_id)
# InterfaceClass* teslib_InterfaceClass_factory(const char* class_id)
lib.teslib_InterfaceClass_factory.argtypes = [ ctypes.POINTER(ctypes.c_char) ]
lib.teslib_InterfaceClass_factory.restype  = ctypes.c_void_p

# const char* testlib_InterfaceClass_getID(InterfaceClass* hinst)
lib.testlib_InterfaceClass_getID.argtypes = [ ctypes.c_void_p ]
lib.testlib_InterfaceClass_getID.restype  = ctypes.POINTER(ctypes.c_char)

Creating class instances and testing member function C-wrappers:

>>> hinstA = lib.teslib_InterfaceClass_factory("ImplementationA".encode('utf-8'))
>>> hinstA
2319184196864
>>>

>>> hinstB = lib.teslib_InterfaceClass_factory("ImplementationB".encode('utf-8'))
>>> hinstB
2319184198016

>>> s1 = lib.testlib_InterfaceClass_getID(hinstA)
>>> s1
<ctypes.LP_c_char object at 0x0000021BFBBF14C8>
>>> ctypes.string_at(s1)
b'ImplementationA'
>>>

>>> s2 = lib.testlib_InterfaceClass_getID(hinstB)
>>> ctypes.string_at(s2)
b'ImplementationB'

>>> ctypes.string_at(lib.testlib_InterfaceClass_getID(hinstB))
b'ImplementationB'
>>>

Load more C-wrappers for member function of class InterfaceClass:

# const char* testlib_InterfaceClass_getName(InterfaceClass* hinst) 
lib.testlib_InterfaceClass_getName.argtypes = [ ctypes.c_void_p ]
lib.testlib_InterfaceClass_getName.restype  = ctypes.POINTER(ctypes.c_char)

# void testlib_InterfaceClass_setName(InterfaceClass* hinst, const char* name)
>>> lib.testlib_InterfaceClass_setName.argtypes = [ ctypes.c_void_p, ctypes.POINTER(ctypes.c_char) ]
>>> lib.testlib_InterfaceClass_setName.restype = None

>>> lib.testlib_InterfaceClass_getName(hinstA)
<ctypes.LP_c_char object at 0x0000021BFBBF1548>

>>> ctypes.string_at(lib.testlib_InterfaceClass_getName(hinstA))
b'Unammed-A'

>>> ctypes.string_at(lib.testlib_InterfaceClass_getName(hinstB))
b'Unammed-B'

>>> lib.testlib_InterfaceClass_setName(hinstA, "Instance-ClassA1".encode('utf-8'))
>>> ctypes.string_at(lib.testlib_InterfaceClass_getName(hinstA))
b'Instance-ClassA1'

>>> lib.testlib_InterfaceClass_setName(hinstB, "Instance-B1".encode('utf-8'))
>>> ctypes.string_at(lib.testlib_InterfaceClass_getName(hinstB))
b'Instance-B1'

Load C-functions wrappers for the InterfaceClass destructor and dispose both instances hinstA and hinstB:

# Destructor function:
# void testlib_InterfaceClass_delete(InterfaceClass* hinst)
>>> lib.testlib_InterfaceClass_delete.argtypes = [ ctypes.c_void_p ]
>>> lib.testlib_InterfaceClass_delete.restype  = None

# Dispose class hinstA, delete this object 
>>> lib.testlib_InterfaceClass_delete(hinstA)
 [INFO] ImplementationA deleted => name = Instance-ClassA1 ; type = ImplementationA

>>> lib.testlib_InterfaceClass_delete(hinstB)
 [INFO] ImplementationB deleted => name = Instance-B1 ; type = ImplementationB

Exit python3 REPL:

>>> exit()
testlib.cpp:39: <DllMain> DLL Loaded into the process => PID = 4280
testlib.cpp:49: <DllMain> => DLL attached
 [StaticObject] => Shutdown DLL

1.3.12 Python 3 Client Code - Wrapper Module

All the boilerplate code required to load the shared library testlib.dll can be eliminated by crafting a python module file and wrapper classes.

Source:

Module Initialization:

  • File: pywrapper.py

The function _config initializes the wrapper module, loading the DLL - share library testlib.dll into the current Python3 interpreter process and setting up the C-functions (functions exported with C-linkage, without name mangling) exported by the library.

import ctypes

def _getSharedLibrary(libname):
    import sys
    import os
    libfile = libname 
    if sys.platform == "linux" or sys.platform == "linux2":
        libfile = "lib" + libname + ".so"
    elif sys.platform == "darwin":
        libfile =  libname + ".dylyb"
    elif  sys.platform == "win32":
        libfile = libname + ".dll"
    libpath = os.path.join(os.path.dirname(__file__), libfile)
    print(" [INFO] libpath = " + libpath)
    return libpath

# _lib = ctypes.cdll.LoadLibrary("testlib")
_lib = ctypes.cdll.LoadLibrary(_getSharedLibrary("testlib"))

# Startup ctypes FFI - Foreign Function Interface 
def _config():
    print("Intializing library")
    # ======= std::vector<double> and Linalg:: namespace ==========##

    # hVectorD testlib_vectorD_make0(size_t n, double x)
    _lib.testlib_vectorD_make0.argtypes = [ctypes.c_int, ctypes.c_double]
    _lib.testlib_vectorD_make0.restype  = ctypes.c_void_p

    # hVectorD testlib_vectorD_make1(size_t n, double array [])
    _lib.testlib_vectorD_make1.argtypes = [ctypes.c_int, ctypes.POINTER(ctypes.c_double)]
    _lib.testlib_vectorD_make1.restype  = ctypes.c_void_p    

_config()

The class VectorD is a wrapper class for the STL class std::vector<double> and the namespace Linalg containing linear algebra functions.

class VectorD:
    def __init__(self, handle):
        self.hnd = ctypes.c_void_p(handle)
        self.name = "std::vector<double> vx"

    @classmethod
    def fromValue(cls, size, x):
        return VectorD(_lib.testlib_vectorD_make0(size, x))

    @classmethod
    def fromArray(cls, array):
        carray_size_n = ctypes.c_double * len(array)
        return VectorD(_lib.testlib_vectorD_make1(len(array), carray_size_n(*array)))

    # Destructor 
    def __del__(self):
        print(" [TRACE] - Vector disposed - C++ Destructor invoked Ok.")
        _lib.testlib_vectorD_delete(self.hnd)

    def setName(self, name):
        self.name = name

    # Display vector 
    def disp(self):
        _lib.testlib_vectorD_Linalg_printVector(self.name.encode('utf-8'), self.hnd)

    # Set element at nth position 
    def set(self, idx, x):
        _lib.testlib_vectorD_set(self.hnd, idx, x)

    def norm(self):
        return _lib.testlib_vectorD_Linalg_norm(self.hnd)

The class CPPInterfaceClass is a wrapper for the C++ interface class InterfaceClass exported by the DLL. The python wrapper class contains factory methods .factory, .makeA for loading the implementation class ImplementationA and the method .makeB for loading the implementation ImplementationB.

# Proxy for C++ Interface class in the shared library 
class CPPInterfaceClass:
    # Constructor 
    def __init__(self, handle):
        self.hnd = ctypes.c_void_p(handle)

    # Destructor 
    def __del__(self):
        # Call C++ destructor
        # print(" [__del__] => self.hnd = " + str(self.hnd))
        _lib.testlib_InterfaceClass_delete(self.hnd)

    @classmethod
    def factory(cls, classID):
        return CPPInterfaceClass(_lib.teslib_InterfaceClass_factory(classID.encode('utf-8')))

    @classmethod
    def makeA(cls):              
        "Instantiate the class ImplementationA from the DLL."
        return CPPInterfaceClass.factory("ImplementationA")

    @classmethod
    def makeB(cls):              
        "Instantiate the class ImplementationB from the DLL."
        return CPPInterfaceClass.factory("ImplementationB")

    def getType(self):
        return ctypes.string_at(_lib.testlib_InterfaceClass_getID(self.hnd)).decode('utf-8')

    def getName(self):
        return ctypes.string_at(_lib.testlib_InterfaceClass_getName(self.hnd)).decode('utf-8')

    def setName(self, name):
        _lib.testlib_InterfaceClass_setName(self.hnd, name.encode('utf-8'))

    # String representation 
    def __str__(self):
        s   = "CInterfaceClass ; type = " + self.getType()
        s +=  " - name =  " + self.getName() + "\n"
        return s

    # Make class printable in the REPL
    def __repr__(self):
        return self.__str__()    

Function test1() runs an example code:

def test1():
    print("\n ========  Test 1 - std::vector<double> wrapper and Linalg module ======")
    v1 = VectorD.fromValue(4, 3.5)
    print(" [*]=> Before changing std::vector<double> object")
    v1.disp()
    print("v1.norm() = " + str(v1.norm()))
    print()
    print(" [*]=> After changing std::vector<double> object")
    v1.set(0, 5); v1.set(1, 2.6); v1.set(2, 9.81); v1.set(3, 3.76)
    v1.disp()
    print("v1.norm() " + str(v1.norm()))
    print()

    print("\n ========  Interface class 'InterfaceClass' ======")
    clsA = CPPInterfaceClass.makeA()
    print("clsA = " + str(clsA))
    print("clsA.getType() = " + clsA.getType())

    clsB = CPPInterfaceClass.makeB()
    print("clsB = " + str(clsB))
    print("clsB.getType() = " + clsB.getType())    

if __name__ == "__main__":    
    test1()

Usage in Python3 REPL:

This step assumes that the shared library file (testlib.dll on Windows or libtestlib.so on Unix-like OS or Linux) is in the same directory as the python script pywrapper.py.

  • Import module
$ C:/Users/archbox/Miniconda3/pkgs/python-3.6.5-h0c2934d_0/python
Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

>>> from testlib import VectorD
 [StaticObject] => Initialize DLL
testlib.cpp:39: <DllMain> DLL Loaded into the process => PID = 5776
testlib.cpp:45: <DllMain> DLL attached to process.
Intializing library
Library initialized OK.

>>> from testlib import CPPInterfaceClass
  • Testing class VectorD - wrapper for std::vector<double>
>>> v1 = VectorD.fromValue(5, 3.0)
>>> v1.disp()
std::vector<double> vx = [5]( 3, 3, 3, 3, 3,  )

>>> v1.setName("v1")
>>> v1.disp()
v1 = [5]( 3, 3, 3, 3, 3,  )

>>> v1.norm()
6.708203932499369

>>> v1.set(0, 5)
>>> v1.set(1, 4.5)
>>> v1.set(2, 9.8)
>>> v1.set(3, 18.4)

>>> v1.disp()
v1 = [5]( 5, 4.5, 9.8, 18.4, 3,  )
>>>
>>> v1.norm()
22.10995251012539

>>> v1 = 100
 [TRACE] - Vector disposed - C++ Destructor invoked Ok.

>>> v2 = VectorD.fromArray([4.0, 10.25, 9.6, 3, 10, 6, 15])
>>> v2.disp()
std::vector<doubl> vx = [7]( 4, 10.25, 9.6, 3, 10, 6, 15,  )

>>> v2.norm()
24.15

>>> v2.set(0, 100)
>>> v2.set(1, 200)

>>> v2.disp()
std::vector<double> vx = [7]( 100, 200, 9.6, 3, 10, 6, 15,  )

>>> v2.norm()
224.63784186997523
>>>

  • Testing class CPPInterfaceClass, proxy for C++ classes InterfaceClass, ImplementationA and ImplementationB.
>>> from testlib import CPPInterfaceClass

# ======== Load ImplementationA  ============== # 
#
>>> a = CPPInterfaceClass.makeA()
>>> a
CInterfaceClass ; type = ImplementationA - name =  Unammed-A

>>> a.getType()
'ImplementationA'
>>> a.getName()
'Unammed-A'
>>> a.setName("Instance-of-ImplA")
>>> a
CInterfaceClass ; type = ImplementationA - name =  Instance-of-ImplA

# ======== Load ImplementationB  ============== # 
#
>>> b = CPPInterfaceClass.makeB()
>>> b
CInterfaceClass ; type = ImplementationB - name =  Unammed-B

>>> b.setName("Instance-of-class-B")
>>> b.getName()
'Instance-of-class-B'
>>>
>>> b.getType()
'ImplementationB'

  • Exit REPL.
>>> exit()
 [TRACE] - Vector disposed - C++ Destructor invoked Ok.
 [INFO] ImplementationB deleted => name = Instance-of-class-B ; type = ImplementationB
testlib.cpp:39: <DllMain> DLL Loaded into the process => PID = 5776
testlib.cpp:49: <DllMain> => DLL attached
 [StaticObject] => Shutdown DLL

1.3.13 CSharp Client Code - PInvoke

The services provided by the DLL can be consumed from C# or any other .NET language via Pinvoke-API which is the Foreign-Function Interface of .NET platform. The Pinvoke API can import C-function or C++ demangled functions exported with extern "C".

C++ classes, namespaces and functions without C-linkage (extern "C") cannot be loaded directly by P-Invoke or any other Foreign Function-Interface as C++ does not define a standard and ABI - Application Binary Interface, every compiler has its own name mangling schema, padding bytes and class memory layout for non-POD (Plain-Old data) types. A workaround to call C++ from an FFI is to defined C-interface comprised of functions with C-linkage and opaque pointers (void*) for passing class instances around or invoking C++ functions.

File: client-sharp.cs

Compiling and running:

$ csc.exe client-csharp.cs && client-csharp.exe

Wrapper for class std::vector<double> and namespace Linalg.

class VectorD{
        // Constructor Function:
        // hVectorD testlib_vectorD_make0(size_t n, double x)
        // void* testlib_vectorD_make0(size_t n, double x)
        [DllImport("testlib.dll")]
        public static extern
        IntPtr testlib_vectorD_make0(int size, double x);

        [DllImport("testlib.dll")]
        public static extern
        IntPtr testlib_vectorD_make1(int n, double[] x);

        // Destructor function:
        // void testlib_vectorD_delete(hVectorD hv)
        // void testlib_vectorD_delete(void* hv)
        [DllImport("testlib.dll")]
        public static extern
        IntPtr testlib_vectorD_delete(IntPtr hv);

        // void testlib_vectorD_Linalg_printVector(const char* name, hVectorD hv)
        // void testlib_vectorD_Linalg_printVector(const char* name, vector* hv)
        [DllImport("testlib.dll")]
        public static extern
        void testlib_vectorD_Linalg_printVector(string name, IntPtr hv);

        // double testlib_vectorD_Linalg_norm(hVectorD hv)
        // double testlib_vectorD_Linalg_norm(void* hv)
        [DllImport("testlib.dll")]
        public static extern
        double testlib_vectorD_Linalg_norm(IntPtr hv);

        // void testlib_vectorD_set(hVectorD hv, size_t n, double x)
        // void testlib_vectorD_set(void* hv, size_t n, double x)
        [DllImport("testlib.dll")]
        public static extern
        void testlib_vectorD_set(IntPtr hv, int n, double x);

        // ------------ Objet Oriented Wrapper --------//

        // Hadle 
        private IntPtr m_handle;

        public VectorD(int size, double x){
                m_handle = testlib_vectorD_make0(size, x);
        }
        public VectorD(double[] array){
                m_handle = testlib_vectorD_make1(array.Length, array);
        }        
        // Finalized destructor 
        ~VectorD(){
                testlib_vectorD_delete(m_handle);
        }
        public void print(string name = "std::vector<double>"){
                testlib_vectorD_Linalg_printVector(name, m_handle);
        }
        public void set(int n, double x){
                testlib_vectorD_set(m_handle, n, x);
        }
        public double norm(){
                return testlib_vectorD_Linalg_norm(m_handle);
        }
}

Wrapper for C++ interface class 'InterfaceClass' and its implementations, class ImplementationA and ImplementationB:

class CPPInterfaceClass{
        // Handle type => Opaque pointer for InterfaceClass instances.
        // using HInterf = IntPtr;

        // Factory function for loading functions from this interface.
        //-----------------------------------------------------------
        // InterfaceClass* teslib_InterfaceClass_factory(const char* class_id)
        // void** teslib_InterfaceClass_factory(const char* class_id)
        [DllImport("testlib.dll")]
        private static extern
        IntPtr teslib_InterfaceClass_factory(string class_id);

        // void testlib_InterfaceClass_delete(InterfaceClass* hinst)
        // void testlib_InterfaceClass_delete(void*)
        [DllImport("testlib.dll")]
        private static extern
        void testlib_InterfaceClass_delete(IntPtr hinst);

        // char* testlib_InterfaceClass_getID(InterfaceClass* hinst)
        [DllImport("testlib.dll")]        
        private static extern IntPtr testlib_InterfaceClass_getID(IntPtr hinst);

        // void testlib_InterfaceClass_setName(InterfaceClass* hinst, const char* name)
        // Note: Charset.UNICODE does not work as it is UTF16 (Wide Unicode), not UTF-8
        // which is the C++ default unicode type.
        [DllImport("testlib.dll", CharSet = CharSet.Ansi)]
        private static extern
        void testlib_InterfaceClass_setName(IntPtr hinst, string name);

        // char* testlib_InterfaceClass_getName(InterfaceClass* hinst)
        [DllImport("testlib.dll")]
        private static extern
        IntPtr testlib_InterfaceClass_getName(IntPtr hinst);

        private IntPtr m_handle;

        private CPPInterfaceClass(IntPtr handle){
                m_handle = handle;
        }

        ~CPPInterfaceClass(){
                testlib_InterfaceClass_delete(m_handle);
        }

        /** Creates an instance of the the C++ implementation ImplementationA (class) from the 
         * C++ interface: 'InterfaceClass' 
         */
        public static CPPInterfaceClass ImplementationA(){
                return new CPPInterfaceClass(teslib_InterfaceClass_factory("ImplementationA"));
        }
        /** Creates an instance of the C++ implementationa ImplementationB (class) */
        public static CPPInterfaceClass ImplementationB(){
                return new CPPInterfaceClass(teslib_InterfaceClass_factory("ImplementationB"));
        }

        // Get type of the wrapped C++ class 
        public string GetID(){
                IntPtr p = testlib_InterfaceClass_getID(m_handle);
                return Marshal.PtrToStringAnsi(p);
        }

        // Get/Set name of current instance 
        public string Name
        {
                get{
                        IntPtr p = testlib_InterfaceClass_getName(m_handle);
                        return Marshal.PtrToStringAnsi(p);
                }
                set{
                        testlib_InterfaceClass_setName(m_handle, value);
                }
        }
}
  • Main function - Experiment 1:
Console.WriteLine(" ===== EXPERIMENT 1 = Using C-functions directly ==");
Console.WriteLine(" [CSharp] Loading Native DLL C++ Shared Library");                
IntPtr vectorObject = VectorD.testlib_vectorD_make0(5, 3.0);
Console.WriteLine(" [CSharp] Opaque pointer = ", vectorObject.ToString());
VectorD.testlib_vectorD_Linalg_printVector("vectorX", vectorObject);


double x = VectorD.testlib_vectorD_Linalg_norm(vectorObject);
Console.WriteLine(" [CSharp] Vector norm = " + x.ToString());

Console.WriteLine(" [CSharp] End application");
VectorD.testlib_vectorD_delete(vectorObject);

Output:

===== EXPERIMENT 1 = Using C-functions directly ==
 [CSharp] Loading Native DLL C++ Shared Library
 [StaticObject] => Initialize DLL
testlib.cpp:48: <DllMain> DLL Loaded into the process => PID = 1320
testlib.cpp:54: <DllMain> DLL attached to process.
 [CSharp] Opaque pointer = 
vectorX = [5]( 3, 3, 3, 3, 3,  )
 [CSharp] Vector norm = 6.70820393249937
 [CSharp] End application
  • Main function - Experiment 2:
Console.WriteLine(" ===== EXPERIMENT 2 = Using OOP wrapper for std::vector ===");
Console.WriteLine(" ==> Before changing vector");
VectorD v1 = new VectorD(10, 3.5);
v1.print();
v1.print("vector_v1");
Console.WriteLine("v1.norm() = " + v1.norm().ToString());

Console.WriteLine(" ==> After changing vector");
v1.set(1, 10.0); v1.set(2, 5.53); v1.set(3, 8.96);
v1.set(4, -10.34); v1.set(8, 80.54);
v1.print("v1_changed");

Console.WriteLine(" ==> Creating Vector from Array");

VectorD v2 = new VectorD(new double[] {4.5, -8.84, 78.23, 652.3, 34.56, 45.12});
v2.print("v2");
Console.WriteLine("v2.norm() = " + v2.norm().ToString());

Output:

 ===== EXPERIMENT 2 = Using OOP wrapper for std::vector ===
 ==> Before changing vector
std::vector<double> = [10]( 3.5, 3.5, 3.5, 3.5, 3.5, 3.5, 3.5, 3.5, 3.5, 3.5,  )
vector_v1 = [10]( 3.5, 3.5, 3.5, 3.5, 3.5, 3.5, 3.5, 3.5, 3.5, 3.5,  )
v1.norm() = 11.0679718105893
 ==> After changing vector
v1_changed = [10]( 3.5, 10, 5.53, 8.96, -10.34, 3.5, 3.5, 3.5, 80.54, 3.5,  )
 ==> Creating Vector from Array
v2 = [6]( 4.5, -8.84, 78.23, 652.3, 34.56, 45.12,  )
v2.norm() = 659.502711518307
  • Main function - Experiment 3:
Console.WriteLine(" ===== EXPERIMENT 3 = OOP wrapper for InterfaceClass  ===");
CPPInterfaceClass instA = CPPInterfaceClass.ImplementationA();
CPPInterfaceClass instB = CPPInterfaceClass.ImplementationB();

// Console.WriteLine("instA.Name = " + instA.Name);
Console.WriteLine("instA.GetID() = " + instA.GetID());
Console.WriteLine("instB.GetID() = " + instB.GetID());

Console.WriteLine(" **=> Before changing");
Console.WriteLine("instA.Name = " + instA.Name);
Console.WriteLine("instB.Name = " + instB.Name);
Console.WriteLine("\n **=> Before changing");
instA.Name = "Instance-of-ImplA";
instB.Name = "Instance-of-ImplB";
Console.WriteLine("instA.Name = " + instA.Name);
Console.WriteLine("instB.Name = " + instB.Name);

Console.WriteLine(" ===== END  ===========================");

Output:

 ===== EXPERIMENT 3 = OOP wrapper for InterfaceClass  ===
instA.GetID() = ImplementationA
instB.GetID() = ImplementationB
 **=> Before changing
instA.Name = Unammed-A
instB.Name = Unammed-B

 **=> Before changing
instA.Name = Instance-of-ImplA
instB.Name = Instance-of-ImplB
 ===== END  ===========================
testlib.cpp:48: <DllMain> DLL Loaded into the process => PID = 1320
testlib.cpp:62: <DllMain> DLL detached to thread.
 [INFO] ImplementationB deleted => name = Instance-of-ImplB ; type = ImplementationB
 [INFO] ImplementationA deleted => name = Instance-of-ImplA ; type = ImplementationA
testlib.cpp:48: <DllMain> DLL Loaded into the process => PID = 1320
testlib.cpp:58: <DllMain> => DLL attached
 [StaticObject] => Shutdown DLL

1.3.14 View symbols exported by the shared library

  1. View exported symbols on Windows (dumpbin)

    Location of dumpbin tool

    The symbols exported by the DLL can viewed with the dumpbin tool available from the MSVC developer console.

    $ where dumpbin.exe
    C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.12.25827\bin\Hostx64\x64\dumpbin.exe
    

    Show DLL exported symbols

    $ dumpbin.exe /exports testlib.dll 
    

    Output:

    Microsoft (R) COFF/PE Dumper Version 14.12.25835.0
    Copyright (C) Microsoft Corporation.  All rights reserved.
    
    
    Dump of file testlib.dll
    
    File Type: DLL
    
      Section contains the following exports for testlib.dll
    
        00000000 characteristics
        FFFFFFFF time date stamp
            0.00 version
               1 ordinal base
              31 number of functions
              31 number of names
    
        ordinal hint RVA      name
    
              1    0 00005C50 ??0SampleClass@@QEAA@AEBV0@@Z
              2    1 00001650 ??0SampleClass@@QEAA@AEBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@Z
              3    2 000015E0 ??0SampleClass@@QEAA@XZ
              4    3 000016E0 ??1SampleClass@@QEAA@XZ
              5    4 00007280 ??4SampleClass@@QEAAAEAV0@AEBV0@@Z
              6    5 00001880 ?get@SampleClass@@QEAAHXZ
              7    6 00001830 ?getName@SampleClass@@QEBA?AV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@XZ
              8    7 00001370 ?linTransform@Linalg@@YA?AV?$vector@NV?$allocator@N@std@@@std@@NNAEAV23@@Z
              9    8 000012D0 ?norm@Linalg@@YANAEBV?$vector@NV?$allocator@N@std@@@std@@@Z
             10    9 00001480 ?printVector@Linalg@@YAAEAV?$basic_ostream@DU?$char_traits@D@std@@@std@@AEAV23@AEAV?$vector@NV?$allocator@N@std@@@3@@Z
             11    A 000018A0 ?set@SampleClass@@QEAAXH@Z
             12    B 00001D50 DllMain
             13    C 00002720 entryPoint1
             14    D 00002800 main
             15    E 00001B10 teslib_InterfaceClass_factory
             16    F 00001C60 testlib_InterfaceClass_delete
             17   10 00001CC0 testlib_InterfaceClass_getID
             18   11 00001D20 testlib_InterfaceClass_getName
             19   12 00001CF0 testlib_InterfaceClass_setName
             20   13 00001A10 testlib_SampleClass_delete
             21   14 00001A60 testlib_SampleClass_get
             22   15 00001AB0 testlib_SampleClass_getName
             23   16 00001900 testlib_SampleClass_make0
             24   17 00001950 testlib_SampleClass_make1
             25   18 00001A80 testlib_SampleClass_set
             26   19 00001560 testlib_vectorD_Linalg_norm
             27   1A 00001580 testlib_vectorD_Linalg_printVector
             28   1B 000026D0 testlib_vectorD_delete
             29   1C 00002590 testlib_vectorD_make0
             30   1D 00002610 testlib_vectorD_make1
             31   1E 00002690 testlib_vectorD_set
    
      Summary
    
            3000 .data
            4000 .pdata
           18000 .rdata
            1000 .reloc
           2F000 .text
    

    Note that:

    • Functions with C-linkage are not mangled, their symbols match their names, for instance:
     15    E 00001B10 teslib_InterfaceClass_factory
    16    F 00001C60 testlib_InterfaceClass_delete
    17   10 00001CC0 testlib_InterfaceClass_getID
    18   11 00001D20 testlib_InterfaceClass_getName
    19   12 00001CF0 testlib_InterfaceClass_setName
    20   13 00001A10 testlib_SampleClass_delete
    21   14 00001A60 testlib_SampleClass_get
    22   15 00001AB0 testlib_SampleClass_getName
    
    • Functions, member functions, namespaces and everything without C-linkage are mangled, the compiler generate an unique name for the generated symbols:
     1    0 00005C50 ??0SampleClass@@QEAA@AEBV0@@Z
     2    1 00001650 ??0SampleClass@@QEAA@AEBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@Z
     3    2 000015E0 ??0SampleClass@@QEAA@XZ
     4    3 000016E0 ??1SampleClass@@QEAA@XZ
     5    4 00007280 ??4SampleClass@@QEAAAEAV0@AEBV0@@Z
     6    5 00001880 ?get@SampleClass@@QEAAHXZ
     7    6 00001830 ?getName@SampleClass@@QEBA?AV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@XZ
     8    7 00001370 ?linTransform@Linalg@@YA?AV?$vector@NV?$allocator@N@std@@@std@@NNAEAV23@@Z
     9    8 000012D0 ?norm@Linalg@@YANAEBV?$vector@NV?$allocator@N@std@@@std@@@Z
    10    9 00001480 ?printVector@Linalg@@YAAEAV?$basic_ostream@DU?$char_traits@D@std@@@std@@AEAV23@AEAV?$vector@NV?$allocator@N@std@@@3@@Z
    11    A 000018A0 ?set@SampleClass@@QEAAXH@Z
    

    It is possible to demangle those symbols with the tool - https://demangler.com/

    • Symbol: ??0SampleClass@@QEAA@AEBV0@@Z demangled:
    public: __cdecl SampleClass::SampleClass(class SampleClass const & __ptr64) __ptr64
    
    • Symbol: ?get@SampleClass@@QEAAHXZ demangled:
    public: int __cdecl SampleClass::get(void) __ptr64
    
    • Symbol: ?linTransform@Linalg@@YA?AV?$vector@NV?$allocator@N@std@@@std@@NNAEAV23@@Z demangled:
    class std::vector<double,class std::allocator<double> > __cdecl Linalg::linTransform(double,double,class std::vector<double,class std::allocator<double> > & __ptr64)
    
  2. View exports symbols on Unix-like OSes (nm)

    View exported symbols with nm:

    • Note that only functions annotated with extern "C" such as teslib_InterfaceClass_factory are not mangled.
    $ nm -D libtestlib.so 
    
    000000000020a3d8 B __bss_start
                     U __cxa_atexit
                     w __cxa_finalize
                     U __cxa_pure_virtual
    000000000020a3d8 D _edata
    000000000020a440 B _end
    000000000000710c T _fini
                     w __gmon_start__
    ... ...   ... ...   ... ...   ... ...   ... ... 
    0000000000005512 T teslib_InterfaceClass_factory
    0000000000005629 T testlib_InterfaceClass_delete
    0000000000005656 T testlib_InterfaceClass_getID
    00000000000056a8 T testlib_InterfaceClass_getName
    0000000000005677 T testlib_InterfaceClass_setName
    000000000000545c T testlib_SampleClass_delete
    000000000000548f T testlib_SampleClass_get
    00000000000054cc T testlib_SampleClass_getName
    
    ... ... ... ...   ... ... ... ... 
    
    0000000000005076 T _ZN11SampleClassC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
    0000000000005106 T _ZN11SampleClassC2Ev
    0000000000005196 T _ZN11SampleClassD1Ev
    
    

    View exported symbols demangled (without name mangling):

     $ nm -CD libtestlib.so 
     000000000020a3d8 B __bss_start
                      U __cxa_atexit
                      w __cxa_finalize
    
      ... ... ...  ... ... ...  ... ... ...  ... ... ... 
    
     0000000000004fe7 T testlib_vectorD_Linalg_norm
     000000000000500f T testlib_vectorD_Linalg_printVector
     0000000000004e52 T testlib_vectorD_make0
     0000000000004ee2 T testlib_vectorD_make1
     0000000000004f80 T testlib_vectorD_set
                      U _Unwind_Resume
                      U operator delete(void*)
                      U operator delete(void*, unsigned long)
     00000000000052fe T SampleClass::get()
     0000000000005310 T SampleClass::set(int)
    
      ...  ...  ...  ...  ...  ...  ... 
    
     00000000000070ac W double* std::__copy_move<false, true, std::random_access_iterator_tag>::__copy_m<double>(double const*, double const*, double*)
     0000000000006cfe W void std::_Destroy_aux<true>::__destroy<double*>(double*, double*)
     0000000000006c74 W std::_Vector_base<double, std::allocator<double> >::_M_allocate(unsigned long)
     0000000000006a44 W std::_Vector_base<double, std::allocator<double> >::_Vector_impl::_Vector_impl(std::allocator<double> const&)
     0000000000006a44 W std::_Vector_base<double, std::allocator<double> >::_Vector_impl::_Vector_impl(std::allocator<double> const&)
     000000000000681a W std::_Vector_base<double, std::allocator<double> >::_Vector_impl::~_Vector_impl()
     000000000000681a W std::_Vector_base<double, std::allocator<double> >::_Vector_impl::~_Vector_impl()
     0000000000006ae8 W std::_Vector_base<double, std::allocator<double> >::_M_deallocate(double*, unsigned long)
     0000000000006a8e W std::_Vector_base<double, std::allocator<double> >::_M_create_storage(unsigned long)
     0000000000006926 W std::_Vector_base<double, std::allocator<double> >::_M_get_Tp_allocator()
     0000000000006836 W std::_Vector_base<double, std::allocator<double> >::_Vector_base(unsigned long, std::allocator<double> const&)
     00000000000069d2 W std::_Vector_base<double, std::allocator<double> >::_Vector_base(std::allocator<double> const&)
     0000000000006836 W std::_Vector_base<double, std::allocator<double> >::_Vector_base(unsigned long, std::allocator<double> const&)
     00000000000069d2 W std::_Vector_base<double, std::allocator<double> >::_Vector_base(std::allocator<double> const&)
    
    ... ... ... ... ... 
    
    0000000000006706 W std::function<void ()>::function(std::function<void ()> const&)
    0000000000006706 W std::function<void ()>::function(std::function<void ()> const&)
    00000000000062b2 W std::function<void ()>::~function()
    00000000000062b2 W std::function<void ()>::~function()
                     U std::ios_base::Init::Init()
                     U std::ios_base::Init::~Init()
    

1.4 Plugins and Dynamically Loading Classes

1.4.2 API used for loading shared libraries dynamically

Unix C-API for loading shared libraries (shared objects)

// Returns handle to shared library 
void* dlopen(const char* fileName, int flags);

// Close handle 
int   dclose(void* handle)

// Get symbol (C-function or function with C-linkage)
void* dlsym(void* handle, const char* FunctionName)

Windows APIs: (Reference MSDN)

// Ansi version 
HMODULE LoadLibraryA( LPCSTR lpLibFileName);  
// Wide Unicode UTF-16 
HMODULE LoadLibraryW(LPCWSTR lpLibFileName);

// FAPROC => Returns Pointer to function casted as void*
FARPROC GetProcAddress(HMODULE hModule, LPCSTR  lpProcName);

void FreeLibrary(HMODULE hModule);

1.4.3 Example

  1. Overview

    This section presents a simple plugin system which allows loading classes at runtime from dynamically-loaded shared libraries. This approach allows adding new functionality to a software without recompilation or modification of the source code. The plugin system can load any class matching a user supplied interface without knowing anything about a particular class implementation.

    In this example: The plugin PluginA, which the file name is PluginA.dll on Windows or PluginA.so on Linux, exports the classes 'Log' and 'Exp' that implements the interface class IMathFunction. The application loads those classes only knowing the interface IMathFunction. This design allows to add more third-party plugins and more classes matching any known interface class.

    General Consideration for an Interface Class

    • MUST NOT have member variables.
    • MUST have only pure virtual member functions, in other words, only abstract methods.

    ABI - Application Binary Interface Considerations for an Interface Class

    • SHOULD use only C-compatible types in the method signatures in order to ensure that the interface to be compatible with multiple compilers. Note: It is very likely that the application and the plugins (aka addons) may be developed with different compilers without a common ABI - Application Binary Interface.
    • SHOULD NOT use any use any STL type or container in the class method signatures, although the implementation classes provided by the plugins can use them internally. The reason to avoid the use of the STL is that the STL implementation is compiler-dependent what can lead to liking error when linking object-codes generated by different compilers.
    • The STL and non-C compatible can be used in interface classes' methods signatures only if the main application and the plugins are developed by the same team with the same compiler version.

    Fragile-Base-class Problem and class ABI

    • Avoid changing interfaces - Any interface class already being used by third parties plugins should not be changed, otherwise it will break the interface class ABI and disrupt third parties' plugins as they will no longer be binary compatible with the interface. As a result, all plugins would have to be recompiled.
    • Fragile Base Class Problem: - Some changes that can break a base class ABI requiring recompilation of all derived classes are:
      • Add new virtual member functions, aka virtual methods
      • Change the order of virtual member functions
      • Add new member variables
      • Change the order or type of member variables
    • Solution: The solution to fragile base class problem in this case is: don't modify interface classes relied by third-party plugins and add new interface classes or extend the existing ones instead of modifying them.

    Source Code:

  2. File: psdk/interfaces.hpp

    This interface class provides plugin metadata and a factory function (Factory) which allows to instantiate any class. The method Factory takes as argument the name of a class provided by the plugin and returns an instance of it casted as a void pointer (opaque pointer).

    #ifndef _INTERFACES_HPP_
    #define _INTERFACES_HPP_
    
    struct IPluginFactory{
         /** Get Plugin Name */
         virtual const char* Name() const = 0 ;
         /** Get Plugin Version */
         virtual const char* Version() const = 0;
    
         virtual size_t NumberOfClasses() const = 0;
         virtual const char* GetClassName(size_t index) const = 0;
    
         /** Instantiate a class from its name */
         virtual void* Factory(const char* className) const = 0;
    };
    
    #endif 
    
  3. File: psdk/factory.hpp

    Headers:

    ifndef _FACTORY_HPP_
    #define _FACTORY_HPP_
    
    #include <string> 
    #include <functional>
    #include <map>
    #include <vector>
    #include <algorithm>
    
    #include "interfaces.hpp"
    
    /** Macro EXPORT_CPP makes a symbol visible. */
    #if defined(_WIN32)
      // MS-Windows NT 
      #define PSDK_PLUGIN_EXPORT_C extern "C" __declspec(dllexport) 
    #else
      // Unix-like OSes
      #define PSDK_PLUGIN_EXPORT_C extern "C" __attribute__ ((visibility ("default")))
    #endif 
    

    Class PluginFactory: Implementation of the interface IPluginFactory.

    class PluginFactory: public IPluginFactory
    {
           // Constructor database
           using CtorItem = std::pair<std::string, std::function<void* ()>>;
           using CtorDB = std::vector<CtorItem>;
           std::string m_name;
           std::string m_version;   
           CtorDB m_ctordb; 
    public: 
            PluginFactory(const char* name, const char* version):
                m_name(name),
                m_version(version)
            {
            }
    
            /** Get Plugin Name */
            const char* Name() const
            {
                return m_name.data();
            }
    
             /** Get Plugin Version */
            const char* Version() const
            {
                return m_version.data();
            }
    
            virtual size_t NumberOfClasses() const
            {
                 return m_ctordb.size();
            }
            virtual const char* GetClassName(size_t index) const
            {
                return m_ctordb[index].first.data();
            }
    
            /** Instantiate a class from its name */
            void* Factory(const char* className) const
            {
                auto it = std::find_if(m_ctordb.begin(), m_ctordb.end(),
                                       [&className](const auto& p){
                                               return p.first == className;
                                       });
                if(it == m_ctordb.end())
                        return nullptr;
                return it->second();
            }
    
            template<typename AClass>
            PluginFactory& registerClass(std::string const& name)
            {
                 auto constructor = []{ return new (std::nothrow) AClass; };
                 m_ctordb.push_back(std::make_pair(name, constructor));
                 return *this;
            }       
    };
    #endif 
    
  4. File: psdk/loader.hpp

    Headers:

    ifndef _LOADER_HPP_
    #define _LOADER_HPP_
    
    #include <cassert>
    #include <map>
    #include <memory>
    
    #include "interfaces.hpp"
    
    // Unix
    #if defined(_WIN32)
       #include <windows.h>
    #elif defined(__unix__)
      // APIs: dlopen, dlclose, dlopen 
      #include <dlfcn.h>
    #else
      #error "Not supported operating system"
    #endif 
    
    #ifdef GetClassName 
      #undef GetClassName 
    #endif
    

    Class Plugin:

    • This class loads a shared library dynamically with dlopen on Unix or LoadLibrary on Windows and gets a function pointer to the symbol GetPluginFactory which returns a pointer to a statically allocated IPluginFactory object that allows instantiating at runtime any registered class in the plugin.
    class Plugin
    {   
    public:
      // Function pointer to DLL entry-point
      using GetPluginInfo_fp = IPluginFactory* (*) ();
    
      void*        m_hnd        = nullptr;
      std::string  m_file       = "";
      bool         m_isLoaded = false;
      IPluginFactory* m_info     = nullptr;
    
      Plugin()
      {
      }
    
      explicit Plugin(std::string file)
      {
            m_file = std::move(file);
           #if !defined(_WIN32)
              m_hnd = ::dlopen(m_file.c_str(), RTLD_LAZY);
           #else
              m_hnd  = (void*) ::LoadLibraryA(m_file.c_str());
           #endif 
              m_isLoaded = true;
              assert(m_hnd != nullptr);
           #if !defined(_WIN32)
             auto dllEntryPoint =
                 reinterpret_cast<GetPluginInfo_fp>(dlsym(m_hnd, "GetPluginFactory"));
           #else
             auto dllEntryPoint =
                 reinterpret_cast<GetPluginInfo_fp>(GetProcAddress((HMODULE) m_hnd, "GetPluginFactory"));
           #endif 
           assert(dllEntryPoint != nullptr);
           // Retrieve plugin metadata from DLL entry-point function 
           m_info = dllEntryPoint();
      }
    
      ~Plugin()
      {
              this->Unload();
      }
      Plugin(Plugin const&) = delete;
      Plugin& operator=(const Plugin&) = delete;
    
      Plugin(Plugin&& rhs) 
      {
           m_isLoaded   = std::move(rhs.m_isLoaded);
           m_hnd        = std::move(rhs.m_hnd);
           m_file       = std::move(rhs.m_file);
           m_info      = std::move(rhs.m_info);
      } 
      Plugin& operator=(Plugin&& rhs)
      {
           std::swap(rhs.m_isLoaded, m_isLoaded);
           std::swap(rhs.m_hnd,  m_hnd);
           std::swap(rhs.m_file, m_file);
           std::swap(rhs.m_info, m_info);
           return *this;
      }
    
      IPluginFactory* GetInfo() const
      {
         return m_info;
      }
    
      void* CreateInstance(std::string const& className)
      {
          return m_info->Factory(className.c_str());
      }
    
      bool isLoaded() const
      {
          return m_isLoaded;
      }
    
      void Unload()
      {
          if(m_hnd != nullptr) {
             #if !defined(_WIN32)
                ::dlclose(m_hnd);
             #else
                ::FreeLibrary((HMODULE) m_hnd);
              #endif 
              m_hnd = nullptr;
              m_isLoaded = false;
          }
      }
    };
    

    Class PluginManager:

    class PluginManager
    {
    public:
          using PluginMap = std::map<std::string, Plugin>;
          // Plugins database 
          PluginMap m_plugindb;
    
          PluginManager()
          {                             
          }
          std::string GetExtension() const 
          {
             std::string ext;
             #if defined (_WIN32)        
                   ext = ".dll"; // Windows 
             #elif defined(__unix__) && !defined(__apple__)
                   ext = ".so";  // Linux, BDS, Solaris and so on. 
             #elif defined(__apple__)
                   ext = ".dylib"; // MacOSX 
             #else   
               #error "Not implemented for this platform"
             #endif 
                  return ext;
          }
    
          IPluginFactory* addPlugin(const std::string& name)
          {         
                  std::string fileName = name + GetExtension();
                  m_plugindb[name] = Plugin(fileName);
                  return m_plugindb[name].GetInfo();
          }
    
          IPluginFactory* GetPluginFactory(const char* pluginName)
          {
                  auto it = m_plugindb.find(pluginName);
                  if(it == m_plugindb.end())
                          return nullptr;
                  return it->second.GetInfo();
          }
    
          void* CreateInstance(std::string pluginName, std::string className)
          {
                  auto it = m_plugindb.find(pluginName);
                  if(it == m_plugindb.end())
                          return nullptr;       
                  return it->second.CreateInstance(className);
          }
    
          template<typename T>
          std::shared_ptr<T>
          CreateInstanceAs(std::string pluginName, std::string className)
          {
                  void* pObj = CreateInstance(std::move(pluginName), std::move(className));
                  return std::shared_ptr<T>(reinterpret_cast<T*>(pObj));
          }
    
    }; /* --- End of class PluginManager --- */
    
    
    #endif 
    
  5. File: PluginA.cpp

    Sample plugin file exporting classes Exp and Log which implement the interface IMathFunction. When compiled on Windows, the file name becomes PluginA.dll, when compiled on Linux, it becomes PluginA.so.

    Header:

    #include <iostream>
    #include <cmath>
    #include <psdk/factory.hpp>
    

    Interface class IMathFunction:

    class IMathFunction
    {
    public:
        virtual const char* Name() const = 0;
        virtual double  Eval(double x) const = 0;   
        virtual ~IMathFunction() = default;
    };
    

    Sample classes exported by the Plugin: implementations

    class Exp: public IMathFunction
    {
    public:
        Exp() = default;
        const char* Name() const {
           return "Exp";
        }
        double Eval(double x) const {
           return std::exp(x);
        }
    };
    
    class Log: public IMathFunction
    {
    public:
        Log() = default;
        const char* Name() const {
           return "Log";
        }
        double Eval(double x) const {
           return std::log(x);
        }
    };
    

    Plugin Entry Point:

    • This function returns a PluginFactory object which contains the shared library metadata and a factory function for instantiating the exported classes Exp and Log. On Unix-like operating system, this global function is loaded by through the API dlopen and dlsym. On Windows, this entry point is loaded through the Win32 APIs LoadLibrary and GetProcAddress.
    PSDK_PLUGIN_EXPORT_C
    auto GetPluginFactory() -> IPluginFactory*
    {
       static PluginFactory pinfo = []{
                 auto p = PluginFactory("PluginA", "0.1-alpha");
                 p.registerClass<Exp>("Exp");
                 p.registerClass<Log>("Log");
                 return p;
            }();
            return &pinfo;
    }
    
  6. File: main.cpp

    Main application that loads the sample plugin PluginA, file name PluginA.so (Linux) or PluginA.dll (Windows).

    Headers and interface:

    #include <iostream>
    #include <psdk/loader.hpp> 
    
    class IMathFunction
    {
    public:
         virtual const char* Name() const = 0;
         virtual double  Eval(double x) const = 0;  
         virtual ~IMathFunction() = default;
    };
    

    Main Function:

    Create a plugin manager object (ma), a repository of plugins:

    PluginManager ma;
    

    Load the plugin file at current directory, on Windows it loads the shared library file PluginA.dll and on Linux, it loads the file PluginA.so (aka shared object)

    IPluginFactory* infoA = ma.addPlugin("PluginA");
    assert(infoA != nullptr);
    

    Print plugin metadata:

    std::cout << " ---- Plugin Information --------- " << "\n"
              << "  =>              Name = " << infoA->Name() << "\n"
              << "  =>           Version = " << infoA->Version() << "\n"
              << "  => Number of classes = " << infoA->NumberOfClasses()
              << "\n\n";
    
    std::cout << "Classes exported by the Plugin: (PluginA) " << "\n";
    
    for(size_t n = 0; n < infoA->NumberOfClasses(); n++)
    {
       std::cout << " -> Exported Class: " << infoA->GetClassName(n) << "\n";
    }
    

    Load class 'Exp' from the plugin or module 'PluginA' and casting it to the interface IMathFunction.

    // Type of pExp: std::shared_ptr<IMathFunction>
    auto pExp = ma.CreateInstanceAs<IMathFunction>("PluginA", "Exp");
    assert(pExp != nullptr);
    
    std::cout << "pExp->Name()    = " << pExp->Name() << "\n";
    std::cout << "pExp->Eval(3.0) = " << pExp->Eval(3.0) << "\n";
    

    Load class 'Log' from plugin 'PluginA':

    auto pLog = ma.CreateInstanceAs<IMathFunction>("PluginA", "Log");
    std::cout << "pLog->Name()    = " << pLog->Name() << "\n";
    std::cout << "pLog->Eval(3.0) = " << pLog->Eval(3.0) << "\n";
    

    Compilation on Linux:

    # Build application 
    $ g++ main.cpp -o main.bin -I. -Wall -std=c++1z -ldl
    
    # Build the Plugin  DLL - Shared library 
    $ g++ PluginA.cpp -o PluginA.so -I. -Wall -std=c++1z -shared -fPIC
    
    # Test the application 
    $ ./main.bin 
    

    Compile on Linux with CMake

    $ git clone https://github.com/caiorss/sample-cpp-plugin
    $ cd sample-cpp-plugin
    $ cmake -Bbuild -H.
    $ cmake --build build --target install
    
    $ ls bin/
    main.bin*  PluginA.so*
    
    $ ./main.bin 
    

    Compile On Windows with CMake

    $ git clone https://github.com/caiorss/sample-cpp-plugin
    $ cd sample-cpp-plugin
    $ cmake -H. -Bbuild2 -DCMAKE_BUILD_TYPE=DEBUG -G "Visual Studio 15 2017 Win64"
    $ cmake --build build2 --target install
    $ cd bin 
    
    $ ls
    PluginA.dll*  PluginA.lib  PluginA.so  main.bin  main.exe*
    
    $ main.exe # Load PluginA.dll 
    

    Program Output:

    $ ./main.bin 
     [TRACE] Shared library PluginA loaded OK.
     ---- Plugin Information --------- 
      =>              Name = PluginA
      =>           Version = 0.1-alpha
      => Number of classes = 2
    
    Classes exported by the Plugin: (PluginA) 
     -> Exported Class: Exp
     -> Exported Class: Log
    pExp->Name()    = Exp
    pExp->Eval(3.0) = 20.0855
    pLog->Name()    = Log
    pLog->Eval(3.0) = 1.09861
     [TRACE] Shared library PluginA unloaded OK.
    (base) 
    

1.5 Return string from a C++ function to Python Ctypes

Example: Create a function with C linkage or C-API which returns a dynamically allocated string to the caller.

Necessary C-string functions:

Build shared library - DLL (cstr.so):

$ clang++ cstr.cpp -o cstr.so -std=c++1z -O0 -g -shared -fPIC -std=c++1z 

File cstr.cpp

Headers:

#include <iostream>
#include <string>
#include <sstream>
#include <cstring>     // std::strncpy
#include <cmath>

#if defined(_WIN32)
  //  MS-Windows 
  #define EXPORT_C  extern "C" __declexpec(dllexport)
#else
  // Unix
  #define EXPORT_C  extern "C" 
#endif 

Function - version A:

/** Design 1: Buffer allocated by the caller.
 * @param[in]     x            Float point number 
 * @param[in,out] buffer       Buffer allocated by the caller. 
 * @param[in]     buffer_size  Size of buffer allocated by the caller in bytes.
 * @return                     Size of copied array.
 */
EXPORT_C size_t 
square_root_as_stringA(double x, char* buffer, size_t buffer_size)
{
     std::string ss;
     ss = ss + "The square root of x = " + std::to_string(x)
             + " is equal to " + std::to_string(sqrt(x));
     std::strncpy(buffer, ss.data(), buffer_size);
     return ss.size();
}

Function - version B:

/** Design 2: Buffer allocated by the caller. If the buffer is null, the function 
 *            returns the required buffer size.
 * @param[in]     x            Float point number 
 * @param[in,out] buffer       Buffer allocated by the caller. 
 * @param[in]     buffer_size  Size of buffer allocated by the caller in bytes.
 * @return                     Size of copied array or required buffer size.                                
 */
EXPORT_C size_t 
square_root_as_stringB(double x, char* buffer, size_t buffer_size)
{
     std::string ss;
     ss = ss + "The square root of x = " + std::to_string(x)
             + " is equal to " + std::to_string(sqrt(x));
     if(buffer == nullptr)
             return ss.size();
     std::strncpy(buffer, ss.data() /* const char* */, buffer_size);
     return ss.size();
}

Function - version C:

/** Design3: Buffer allocated by callee (this code) and released by the
 * caller. */
EXPORT_C char* 
square_root_as_stringC(double x){
     std::string ss;
     ss = ss + "The square root of x = " + std::to_string(x)
             + " is equal to " + std::to_string(sqrt(x));
     return strdup(ss.data());
}

EXPORT_C void 
cstring_delete(char* pCstring)
{
    std::cout << " [INFO] C-string deleted, memory released. OK." << "\n";
    std::free(pCstring);
}

file: cstrp.py (Python Wrapepr)

Initial configuration:

import ctypes 

CString   = ctypes.POINTER(ctypes.c_char)
STDString = ctypes.c_void_p 
CVoid     = None 
dll = ctypes.cdll.LoadLibrary("cstr.so")

Wrapper for function A:

print("\n EXPERIMENT1 => Call square_root_as_stringA")
print("=============================================")
dll.square_root_as_stringA.argtypes = [ ctypes.c_double,
                                        CString,
                                        ctypes.c_size_t ]
dll.square_root_as_stringA.restype  = ctypes.c_size_t

buf_size = 1024 # 1 Kbyte = 1024 bytes
buf      = ctypes.create_string_buffer(buf_size)
n_read   = dll.square_root_as_stringA(100.45, buf, buf_size)

print("Size = ", n_read, " ; RESULT = ", buf.value)

Wrapper for function B:

print("\n EXPERIMENT 2 => Call square_root_as_stringB")
print("=============================================")
dll.square_root_as_stringB.argtypes = [ ctypes.c_double,
                                        CString,
                                        ctypes.c_size_t ]
dll.square_root_as_stringB.restype  = ctypes.c_size_t

x = 200.45
required_size = dll.square_root_as_stringB(x, ctypes.c_char_p(0), 0)
buf           = ctypes.create_string_buffer(required_size)
dll.square_root_as_stringB(x, buf, required_size)
print(" RESULT = ", buf.value)

Wrapper for function C:

print("\n EXPERIMENT 3 => Call square_root_as_stringC")
print("=============================================")

dll.square_root_as_stringC.argtypes = [ ctypes.c_double ]
dll.square_root_as_stringC.restype  = ctypes.c_char_p

dll.cstring_delete.argtypes = [ ctypes.c_char_p ]
dll.cstring_delete.restype  = CVoid

hCString = dll.square_root_as_stringC(167.42)
print(" RESULT = ", hCString)

Program output

$ python3 cstr.py 

 EXPERIMENT1 => Call square_root_as_stringA
=============================================
Size =  55  ; RESULT =  b'The square root of x = 100.450000 is equal to 10.022475'

 EXPERIMENT 2 => Call square_root_as_stringB
=============================================
 RESULT =  b'The square root of x = 200.450000 is equal to 14.158037'

 EXPERIMENT 3 => Call square_root_as_stringC
=============================================
 RESULT =  b'The square root of x = 167.420000 is equal to 12.939088'

2 Python Native Extension API

2.1 Overview

Many programming languages provide a C-native interfaces such as Python C-API or Java Native Interface where it is possible to create native-code libraries in C without overhead of virtual machine code interpretation. The advantage of a code for a C native interface (native API) of some programming language is the easier usage and better integration with the target language. The drawback for developing some native library with this approach is that the library will be tight coupled with the language native C API, as a result, it will be hard to port or call the code from other programming language. In addition, it may require deeper knowledge of the native C API which often is not well documented.

Definition:

  • A programming language native interface API is a C-API which allows creating libraries almost entirely in C compiled as shared libraries and load them in the same way as ordinary libraries.

Native Interface API X Foreign Function Interface based on Libffi

  • Advantages of Native Interface over FFI
    • Better performance
    • Enhanced integration
    • Easier to use
  • Drawbacks of Native Interface over FFI:
    • Most native APIs are not well documented and they are specific for a given programming language.
    • Requires in-deep knowledge of the native C-API and the target programming language.
    • Hard to reuse the code with other languages and even with C++.
    • Code tightly coupled with the Native Interface API.
  • Advantages of FFI over Native Interface:
    • The code does not need to conform to any native interface C-API and can be called from any programming language. The shortcoming of this approach is the need of C-interfaces (extern "C") for the C++ code and a FFI wrapper.
    • Most of FFIs are based on the library LibFFI which is used by Python (ctypes), Ruby, Java JNA, Haskell and so on. Therefore, the concept of creating FFI wrappers for C or C++ libraries with C-interfaces can be reused on all languages with LibFFI bindings.

Some Programming language specific Native Interface APIs (C-APIs):

Other approaches for developing shared libraries for Python native API:

  • Pybind11 header-only library
    • => Provides a C++-friendly syntax and interface to the Python native C API. It allows building Python native modules and port C++ code in a easier and faster way than dealing directly with Python C-API.
  • SWIG wrapper generator
    • => SWIG can parse C++ headers and interface files and generate C++ binding code to the native API of many programming languages such as Python, Ruby, Java and so on.
  • Cython
    • => Python-like language that compile to C code (generate C-code). The disadvantage is the tight coupling to the Python C-API.

Further documentation and reading:

Documentation:

General:

Creating a native Python module:

Python (CPython) debugging with GDB:

Reverse engineering:

2.2 Python Raw C API

2.2.1 Module Code

  1. Sources

    Module Source:

    Compilation:

    # GCC - GNU C/C++ compiler 
    $ g++ mymodule.cpp -o mymodule.so -g -std=c++1z -fPIC -shared -I/usr/include/python3.6m  
    
    # Clang LLVM 
    $ clang++ mymodule.cpp -o mymodule.so -g -std=c++1z -fPIC -shared -I/usr/include/python3.6m  
    
  2. File: setup.py

    Python script used for building and installing the Python module.

    from distutils.core import setup, Extension
    
    mymodule = Extension('mymodule', sources = ['mymodule.cpp'])
    
    setup(
        name        = 'mymodule',
        version     = '1.0',
        description = 'Sample python C-API exploration',
        ext_modules = [mymodule]
    )
    
  3. File: mymodule.cpp

    Heades:

    #include <iostream>
    #include <string>
    #include <functional>
    #include <iomanip>
    
    // Solve Mingw error: '::hyport' has not been declared 
    #include <math.h>
    
    //#define hypot _hypot
    
    /// C-API header to Python Native Interface API, similar to Java JNI
    /// (Java Native Interface)
    #include <Python.h>
    
    #ifdef _WIN32
      #include <process.h> // Exports _getpid()
    #else
      #include <unistd.h>  // Exports getpid()
    #endif
    

    Global object for DLL initialization (optional):

    • The global object dllinit_hook is used for logging to stdout when the DLL is loaded or unloaded by the python process.
    struct DLLInitialization
    {
         DLLInitialization(){
             int pid;
             #ifdef _WIN32
                pid = _getpid();
             #else
                pid = getpid();       
             #endif 
             std::cerr << " [TRACE] Python module DLL loaded by process PID = <"
                               << pid << "> "
                               << std::endl;
             std::cerr << " [TRACE] Attach the debugger with: $ gdb -pid=" << pid << "\n";
         }
         ~DLLInitialization(){
             std::cerr << " [TRACE] DLL native DLL unloaded OK." << std::endl;
         }  
    };
    
    DLLInitialization dllinit_hook;
    

    Exposed functions declarations:

    /** ======== List of functions exposed to Python =========== */
    
    PyObject*
    exposedFunction(PyObject* self, PyObject* args);
    
    PyObject* testArguments(PyObject* self, PyObject* args);
    
    auto printRange(PyObject* self, PyObject* args) -> PyObject*;
    auto taylorSeriesExp(PyObject* self, PyObject* args) -> PyObject*;
    PyObject* returnTuple(PyObject* self, PyObject* args);
    PyObject* returnDictionary(PyObject* self, PyObject* args);
    PyObject* tabulateFunction(PyObject* self, PyObject* args);
    PyObject* computeStatistics(PyObject* self, PyObject* args);
    

    Table of exported functions to the Python module:

    static PyMethodDef ModuleFunctions [] =
    {
            { "exposedFunction", exposedFunction, METH_VARARGS,
              "Documentation or docstring of function exposedFunction1." }
    
            ,{"testArguments", testArguments, METH_VARARGS,
              "Test python arguments: Signature testArguments(int, float, double, const char*)"}
    
            // Function without docstring: printRange(int i, double x)
            ,{ "printRange", printRange, METH_VARARGS, nullptr }
    
            ,{ "taylorSeriesExp", &taylorSeriesExp, METH_VARARGS,
               "taylorSeriesExp(double x, size_t maxiter, double tol) -> double"
               "\n Computes exponential of a given value with taylor serie approximation."
               "\n Formula reference:  https://www.mathsisfun.com/algebra/taylor-series.html"
            }
            ,{"returnTuple", &returnTuple, METH_VARARGS, nullptr}
            ,{"returnDictionary", &returnDictionary, METH_VARARGS, nullptr}
            ,{"computeStatistics", &computeStatistics, METH_VARARGS, nullptr}
            ,{"tabulateFunction", tabulateFunction, METH_VARARGS,
              "Tabulate some mathematical function or callable object"}
            // Sentinel value used to indicate the end of function listing.
            // All function listing must end with this value.
            ,{nullptr, nullptr, 0, nullptr}                                 
    };
    

    Module definition:

    /* Module definition */
    static struct PyModuleDef ModuleDefinitions {
        PyModuleDef_HEAD_INIT,
       // Module name as string 
       "mymodule",
       // Module documentation (docstring)
       "A sample C++ native-code module for python3.",
       -1,
       // Functions exposed to the module 
       ModuleFunctions
    };
    
    

    Module initialization function:

    • Note: This function must contain the name in the format PyInit_<MODULE NAME> where module name is the same name as the module file name <MODULE NAME>.so (Linux) or <MODULE NAME>.pyd on Windows.
    /** Module Initialization function: must have this name schema
     *  PyInit_<ModuleName> where ModuleName is the same base name of the 
     *  shared library ModuleName.so (on Linux) or ModuleName.pyd (on Windows)
     */
    PyMODINIT_FUNC PyInit_mymodule(void)
    {
       Py_Initialize();
       PyObject* pModule = PyModule_Create(&ModuleDefinitions);
       PyModule_AddObject(pModule, "version", Py_BuildValue("s", "version 0.1-Alpha"));
       return pModule;
    }
    

    Module Functions Implementations

    All functions exposed to the Python C-API must have the same type signature of the function pointer PyExposedFunction or the same type signature of the function MyExposedFunction.

     using PyExposedFunction = PyObject* (*) (PyObject* self, PyObject* args)
    
     // Example: 
     PyObject* MyExposedFunction(PyObject* self, PyObject* args )
     {
        // On Failure return null 
        return nullptr;  
    
        return /* Some instance of PyObject */;
    }
    

    Notes:

    • The function takes always two arguments self, and args (tuple object containing arguments passed to the function)
    • The exposed function always return an object PyObject.
    • All Python objects are heap-allocated.
    • In Python everything is an object: integers, double, classes, class instances, modules and functions and so on. All those types have the same type PyObject in the C-API.
    • Functions exposed to Python runtime should not throw a C++ exceptions, instead they should return null (nullptr) that causes a runtime error or exception on Python side.

    Function: exposedFunction

    • Function that takes a void argument and returns nothing.
    PyObject*
    exposedFunction(PyObject* self, PyObject* args)
    {
         std::cout << " =>> Hello word Python from C++ side." << "\n";
         // All python functions that returns anything
         // should end with this macro 
         Py_RETURN_NONE;
    }
    

    Function: testArguments

    • Function that takes many arguments used to demonstrate how to extract tuple arguments.
    PyObject* testArguments(PyObject* self, PyObject* args)
    {
          int         abool;
          int         aint;
          float       afloat;
          double      adouble;
          const char* aword;
          // Parse function arguments 
          if(!PyArg_ParseTuple(args, "pifds", &abool, &aint, &afloat, &adouble, &aword))
              return nullptr;
    
          std::cerr << " Function: [" << __FUNCTION__ << "]" 
                    << " ==> Number of arguments = " << PyTuple_GET_SIZE(args) << "\n";
    
          std::string str = aword;
          str = "'Received string = " + str + "'";
    
          std::cout << " Received Python Arguments " << "\n";
          std::cout << std::boolalpha;
          std::cout << "abool[bool] = " << (bool) abool << "\n"
                            << " abool[int] = " << abool << "\n"
                            << "      aint  = " << aint << "\n"
                            << "     afloat = " << afloat << "\n"
                            << "    adouble = " << adouble << "\n"
                            << "    aword   = " << str << "\n"
                            << "\n";        
          return Py_BuildValue("s", str.data());
    }
    
    

    Function PyArg_ParseTuple:

    • The function PyArg_ParseTuple is used for extracting function arguments from the tuple object (args).
    • The letters in the string parameter "pifds" means:
      • p => Extract first argument from tuple object as boolean (integer 0 or 1) storing the value in the variable abool.
      • i => Extract second argument from tuple object as integer storing the value in the variable 'aint'.
      • f => Extract third argument as float (float point 32 bits). The value is then stored in the variable 'afloat'.
      • d => Extract fourth argument as double (float point 64 bits)
      • s => Extract fith argument as string.
    • If the function fails, it must return null which causes a runtime error on Python side.
    int         abool;
    int         aint;
    float       afloat;
    double      adouble;
    const char* aword;
    // Parse function arguments 
    if(!PyArg_ParseTuple(args, "pifds", &abool, &aint, &afloat, &adouble, &aword))
        return nullptr;
    

    Function Py_BuildValue("s", str.data());

    • This function is used for building return values to the python side. It can return any python object, such as string, number, class instance or multiple values as a tuple.
    // Return None in the python side 
    return Py_BuildValue("");
    
    // Return a single value as string string 
    return Py_BuildValue("s", str.data());
    return Py_BuildValue("s", "A string to python side");
    
    // Return integer 
    return Py_BuildValue("i", 1000);
    
    // Return float  
    return Py_BuildValue("f", 3.465);
    
    // Return double   
    return Py_BuildValue("d", 140.346);
    
    // Return a tuple with a single double value: (3.156)
    return Py_BuildValue("(d)", 3.156);
    
    // Return a tuple with a single double and integer: (3.156, 100)
    return Py_BuildValue("(di)", 3.156, 100);
    
    // Return a tuple with a single double, integer and string
    // tuple object: (3.156, 100, "a string")
    return Py_BuildValue("(dis)", 3.156, 100, "a string");
    

    Function: printRange

    • Pseudo-Python signature: printRange(int n, float x) => void
    auto printRange(PyObject* self, PyObject* args) -> PyObject*
    {
        std::cerr << " =>> Printing numeric range " << "\n";    
        int    n;
        float  x;
        // Extract function argument from tuple argument object
        // "if" => i - Extract integer
        //      => f - Extract float (IEEE754 32 bits float point)
        if(!PyArg_ParseTuple(args, "if", &n, &x)){
            PyErr_SetString(PyExc_RuntimeError, "Invalid argument");
            // Always return null on failure 
            return nullptr; 
        }
    
        if(n <= 0){
            PyErr_SetString( PyExc_RuntimeError
                            ,"Invalid argument: n supposed to be greater than zero.");
            return nullptr;
        }   
    
        for(int k = 0; k < n; k++)
           std::cout << " k[" << k << "] = " << 3 * k + x << "\n";  
        Py_RETURN_NONE;
    }
    
    

    Function: returnTuple

    • Return multiple values as tuple.
    /** Function that returns multiple values as tuple object */
    PyObject* returnTuple(PyObject* self, PyObject* args)
    {
         double x;
         double y;
         int    n;
         if(!PyArg_ParseTuple(args, "ddi", &x, &y, &n))
                 return nullptr;
         // Build tuple object containing
         // (d: double, d: double, i: integer, s: string)
         return Py_BuildValue("(ddis)", 2 * x + y * n, x + 2 * y - n, n, "Result"); 
    }
    

    Function: returnDictionary

    • Return a dictionary (hash-table)
    /** Return a dictionary object */
    PyObject* returnDictionary(PyObject* self, PyObject* args)
    {
         double x;
         double y;
         int    n;
         if(!PyArg_ParseTuple(args, "ddi", &x, &y, &n))
                 return nullptr;
    
         // Create new dictionary object 
         PyObject* pDict = PyDict_New();
    
         PyObject* p = nullptr;
         // Fill dictionary
         p = Py_BuildValue("d", 2 * x + y * n);
         PyDict_SetItemString(pDict, "alpha", p);
         Py_DECREF(p);
    
         p = Py_BuildValue("i", 3 * n) ;
         PyDict_SetItemString(pDict, "size", p);
         Py_DECREF(p);
    
         p = Py_BuildValue("s", "Some string") ;
         PyDict_SetItemString(pDict, "beta", p);
         Py_DECREF(p);  
    
         return pDict;
    }
    

    Function: computeStatistics

    • Computes statistics sum and mean for any iterable object containing double values such as list.
    PyObject* computeStatistics(PyObject* self, PyObject* args)
    {
         std::cerr << " [TRACE] " << __FILE__ << "(" << __LINE__ << ") - "
                           << "Calling function: " << __FUNCTION__ << "\n";
    
         std::cerr << " [TRACE] Arguments = "; PyObject_Print(args, stdout, 0); std::cerr << "\n";
    
         PyObject* pSeq;
         // Parse function argument as sequence
         if(!PyArg_ParseTuple(args, "O", &pSeq))
                 return nullptr;
    
         pSeq = PySequence_Fast(pSeq, "Expected iterable arguments");
         if(pSeq == nullptr) return nullptr;
    
         int numberOfElements = PySequence_Fast_GET_SIZE(pSeq);
         std::cerr << " [TRACE] numberOfElements = " << numberOfElements << "\n";
    
         PyObject* pItem = nullptr;
         double x;
         double sum;
    
         for(int n = 0; n < numberOfElements; n++)
         {
             pItem = PySequence_Fast_GET_ITEM(pSeq, n);
             if(!pItem) {
                 Py_DECREF(pSeq);
                 return nullptr;
             }
             // Print item 
             std::cout << "item[" << n << "] = ";
             PyObject_Print(pItem, stdout, 0);
             std::cout << "\n";
    
             x = PyFloat_AsDouble(pItem);
             if(PyErr_Occurred() != nullptr){
                     PyErr_SetString(PyExc_TypeError, "Error: expected float point.");
                     return nullptr;
             }
             sum += x;
         }
    
         double mean = sum / numberOfElements;
         std::cout << "\n (C++) Statics Result " << "\n";
         std::cout << "---------------" << "\n";
         std::cout << " Sum  = " << sum << "\n";
         std::cout << " Mean = " << mean << "\n";
    
         Py_DECREF(pSeq);
         return Py_BuildValue(""); // Return None 
    }
    
    

    Function: taylorSeriesExp

    • This function computes the exponential function approximation using its taylor series expansion. Formula at: taylor series
    auto taylorSeriesExp(PyObject* self, PyObject* args) -> PyObject*
    {
         double x;
         size_t maxiter;  // Maximum number of iterations 
         double tol;      // Tolerance
    
         // Parse function arguments 
         if(!PyArg_ParseTuple(args, "did", &x, &maxiter, &tol))
                 return nullptr;
    
         // Validate function arguments
    
         if(tol <= 0 || tol > 1.0){
                 PyErr_SetString( PyExc_RuntimeError
                                ,"Invalid tolerance, expected in range (0, 1]");
                 return nullptr;
         }
    
         // Compute exponential taylor series available at
         // https://www.mathsisfun.com/algebra/taylor-series.html
         unsigned long factorial = 1;
         double        xpower    = 1.0;
         double        sum       = 0.0;
         double        term      = 0.0;
         size_t        idx       = 1;
         do{
              term      = xpower / factorial;
              sum       = sum + term;
              xpower    = xpower * x ;
              factorial = factorial * idx;      
              idx++;
         } while(idx <= maxiter && std::abs(term) > std::abs(sum) * tol );
         // Return float point constnat NAN (Not a Number)
         if(idx >= maxiter){
                 std::cerr << " [ERROR] Series does not converge." << "\n";
                 return Py_BuildValue("d", NAN);
         }
         return Py_BuildValue("d", sum);
    }
    

    Function: tabulateFunction

    • This function takes a callable object (callback) which can be a function, lambda function, callable object (object with method __call__) and so on.

    Pseudo Python method signature using C++ notation:

    using Callback = std::function<double (double)>;
    void tabulateFunction(Callback callback, double xmin, double xmax, double step)
    

    Function code:

    PyObject* tabulateFunction(PyObject* self, PyObject* args)
    {
         PyObject* pObj = nullptr;
         double xmin, xmax, xstep;
    
         if(!PyArg_ParseTuple(args, "Oddd", &pObj, &xmin, &xmax, &xstep))
              return nullptr;
         if(pObj == nullptr) {
              PyErr_SetString(PyExc_RuntimeError, "Error: invalid None object.");
              return nullptr;
         }
         PyObject* pArgs  = nullptr;
         PyObject* pResult = nullptr;
         double y = 0.0;
    
         std::cout << std::fixed << std::setprecision(4);
    
         std::cout << "Tabulating range: "
                   << " ; xmin = " << xmin
                   << " ; xmax = " << xmax
                   << " ; step = " << xstep
                   << "\n";
    
         for(double x = xmin; x <= xmax; x += xstep )
         {
              pArgs  = Py_BuildValue("(d)", x);
              pResult = PyEval_CallObject(pObj, pArgs);
              y = PyFloat_AsDouble(pResult);
              if(PyErr_Occurred() != nullptr){
                      PyErr_SetString(PyExc_RuntimeError, "Error: Invalid float point.");
                      return nullptr;
              }
              std::cout << std::setw(8) << x << std::setw(10) << y  << "\n";
         }
         Py_RETURN_NONE;
    }
    
    

    Extract python callable object (callback) pObj and the parameters xmin, xmax and xstep:

    PyObject* pObj = nullptr;
    double xmin, xmax, xstep;
    // Extract parameters 
    // O => pObj 
    // d => xmin 
    // d => xmax 
    if(!PyArg_ParseTuple(args, "Oddd", &pObj, &xmin, &xmax, &xstep))
         return nullptr;
    

    Annotated code block in the for-loop:

    // Argument (tuple object) of callable object 
    pArgs  = Py_BuildValue("(d)", x);
    
    // Call callable object (method __call__)
    pResult = PyEval_CallObject(pObj, pArgs);
    
    // Try extracting result as double 
    y = PyFloat_AsDouble(pResult);
    
    // Check whether any error has happened and abort the operation in this case. 
    if(PyErr_Occurred() != nullptr){
         PyErr_SetString(PyExc_RuntimeError, "Error: Invalid float point.");
         return nullptr;
    }
    // Display table row 
    std::cout << std::setw(8) << x << std::setw(10) << y  << "\n";
    

2.2.2 Compiling and Running Native Module

Clone repository:

$ git clone https://gist.github.com/caiorss/bc7dc373d87b828029eb0e0ec048d91e python-module 
$ cd python-module 

Manual Compilation

Compile the native module mymodule.so on Linux or any other Unix-like OS:

$ clang++ mymodule.cpp -o mymodule.so -g -std=c++1z -fPIC -shared -I/usr/include/python3.6m  

Compile the native module mymodule.pyd on Windows OS with Mingw compiler:

$  g++ mymodule.cpp -o mymodule.pyd -g -std=c++1z -shared -fPIC \
   -I"C:\Users\archbox\Miniconda3\pkgs\python-3.6.5-h0c2934d_0\include" \
   C:\Users\archbox\Miniconda3\pkgs\python-3.6.5-h0c2934d_0\libs\Python36.lib

Compilation with setup.py

Build only:

  • $ python.exe setup.py build
$ python.exe setup.py build 

$ cd build\
lib.win-amd64-3.6\   temp.win-amd64-3.6\
F:\python-module (master -> origin)
λ cd build\lib.win-amd64-3.6\

$ ls
mymodule.cp36-win_amd64.pyd*

$ C:\Users\archbox\Miniconda3\python.exe
Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import mymodule as m
 [TRACE] Python module DLL loaded by process PID = <620>
 [TRACE] Attach the debugger with: $ gdb -pid=620

Install:

$ python.exe setup.py install

 $ C:\Users\archbox\Miniconda3\python.exe setup.py install
 running install
 running build
 running build_ext
 building 'mymodule' extension
 creating build\temp.win-amd64-3.6
 creating build\temp.win-amd64-3.6\Release
 C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe /c /nologo /Ox
  /W3 /GL /DNDEBUG /MD -IC:\Users\archbox\Miniconda3\include -IC:\Users\archbox\Miniconda3\include "-IC:\Program Files (x86)\Mi
 crosoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\ATLMFC\include" ... ... 
 ... ....   ... ....   ... ....   ... ....   ... ....   ... ....   ... ....  
 ... ....   ... ....   ... ....   ... ....   ... ....   ... ....   ... ....  
 Finished generating code
 running install_lib
 copying build\lib.win-amd64-3.6\mymodule.cp36-win_amd64.pyd -> C:\Users\archbox\Miniconda3\Lib\site-packages
 running install_egg_info
 Writing C:\Users\archbox\Miniconda3\Lib\site-packages\mymodule-1.0-py3.6.egg-info

Check generated file on Linux:

$ file mymodule.so 

$ du -h mymodule.so 
136K    mymodule.so
136K    total

View symbols:

$ nm  mymodule.so 
0000000000209518 B __bss_start
0000000000005500 t __clang_call_terminate
0000000000209518 b completed.7347
                 U __cxa_atexit@@GLIBC_2.2.5
                 U __cxa_begin_catch@@CXXABI_1.3
                 U __cxa_call_unexpected@@CXXABI_1.3
                 U __cxa_end_catch@@CXXABI_1.3
                 w __cxa_finalize@@GLIBC_2.2.5
                 U __cxa_rethrow@@CXXABI_1.3
0000000000004160 t __cxx_global_var_initmymodule.cpp_
00000000000041a0 t __cxx_global_var_initmymodule.cpp_.1
00000000000041f0 t deregister_tm_clones
000000000020951a B dllinit_hook
.. ... ... ... ... ... ... ... 

Load module in Python 3 REPL:

$ python3
Python 3.7.1 (default, Dec 14 2018, 19:28:38) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> import mymodule as m
 [TRACE] Python module DLL loaded by process PID = <5236> 
 [TRACE] Attach the debugger with: $ gdb -pid=5236
>>> 
>>> 

Module Docstring (documentation):

>>> help(m)
Help on module mymodule:

NAME
    mymodule - A sample C++ native-code module for python3.

FUNCTIONS
    computeStatistics(...)

    exposedFunction(...)
        Documentation or docstring of function exposedFunction1.

    printRange(...)

    returnDictionary(...)

    returnTuple(...)

    tabulateFunction(...)
        Tabulate some mathematical function or callable object

    taylorSeriesExp(...)
        taylorSeriesExp(double x, size_t maxiter, double tol) -> double
        Computes exponential of a given value with taylor serie approximation.
        Formula reference:  https://www.mathsisfun.com/algebra/taylor-series.html

    testArguments(...)
        Test python arguments: Signature testArguments(int, float, double, const char*)

DATA
    version = 'version 0.1-Alpha'

FILE
    /home/archbox/shared/python-module/mymodule.so

Module version:

>>> m.version
'version 0.1-Alpha'
>>> 

Call function exposedFunction()

>>> m.exposedFunction()
 =>> Hello word Python from C++ side.
>>> 

Call function testArguments:

>>> s1 = m.testArguments(False, 100, 10.9361, 0.344, "from Python to C++")
 Function: [testArguments] ==> Number of arguments = 5
 Received Python Arguments 
abool[bool] = false
 abool[int] = 0
      aint  = 100
     afloat = 10.9361
    adouble = 0.344
    aword   = 'Received string = from Python to C++'

>>> s1
"'Received string = from Python to C++'"
>>> 

>>> 
>>> s2 = m.testArguments(True, 90, -0.56e6, 9.344e5, "something else more")
 Function: [testArguments] ==> Number of arguments = 5
 Received Python Arguments 
abool[bool] = true
 abool[int] = 1
      aint  = 90
     afloat = -560000
    adouble = 934400
    aword   = 'Received string = something else more'

>>> s2
"'Received string = something else more'"

Call function printRange()

>>> m.printRange(5, 4.5)
 =>> Printing numeric range 
 k[0] = 4.5
 k[1] = 7.5
 k[2] = 10.5
 k[3] = 13.5
 k[4] = 16.5

>>> m.printRange(-5, 4.5)
 =>> Printing numeric range 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: Invalid argument: n supposed to be greater than zero.

>>> m.printRange(0, 4.5)
 =>> Printing numeric range 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: Invalid argument: n supposed to be greater than zero.

Calll function returnTuple:

>>> tpl = m.returnTuple(3.5, 4.2, 10)

>>> tpl
(49.0, 1.9000000000000004, 10, 'Result')

>>> tpl[0]
49.0

>>> tpl[1]
1.9000000000000004

>>> tpl[2]
10

>>> m.returnTuple(8.5, 4.2, 20)
(101.0, -3.1000000000000014, 20, 'Result')

Call functon returnDictionary:

>>> d = m.returnDictionary(8.5, 8.62, 10)

>>> d
{'alpha': 103.19999999999999, 'size': 30, 'beta': 'Some string'}

>>> d["alpha"]
103.19999999999999

>>> d["beta"]
'Some string'

>>> d["size"]
30

Call function computeStatistics:

>>> xs = [4, 1.2, 9.5, 10.5, 3.4, 5.10, 6.0]

>>> xs
[4, 1.2, 9.5, 10.5, 3.4, 5.1, 6.0]

>>> m.computeStatistics(xs)
 [TRACE] mymodule.cpp(268) - Calling function: computeStatistics
 [TRACE] Arguments = ([4, 1.2, 9.5, 10.5, 3.4, 5.1, 6.0],)
 [TRACE] numberOfElements = 7
item[0] = 4
item[1] = 1.2
item[2] = 9.5
item[3] = 10.5
item[4] = 3.4
item[5] = 5.1
item[6] = 6.0

 (C++) Statics Result 
---------------
 Sum  = 39.7
 Mean = 5.67143
>>> 

Call function taylorSerieExp:

>>> help(m.taylorSeriesExp)
Help on built-in function taylorSeriesExp in module mymodule:

taylorSeriesExp(...)
    taylorSeriesExp(double x, size_t maxiter, double tol) -> double
    Computes exponential of a given value with taylor serie approximation.
    Formula reference:  https://www.mathsisfun.com/algebra/taylor-series.html

>>> import math

# Exact value 
>>> math.exp(3.5)
33.11545195869231

# Approximation 
>>> m.taylorSeriesExp(3.5, 10, 0.001)
 [ERROR] Series does not converge.
nan
>>> m.taylorSeriesExp(3.5, 100, 0.001)
33.10588185071678
>>> 

# Exact value 
>>> math.exp(5.0)
148.4131591025766

# Approximation 
>>> m.taylorSeriesExp(5.0, 100, 0.001)
148.37958007973663

>>> m.taylorSeriesExp(5.0, 100, 0.0001)
148.41021027504306
>>> 

Call function: tabulateFunction which takes a callable object as argumet.

>>> help(m.tabulateFunction)
Help on built-in function tabulateFunction in module mymodule:

tabulateFunction(...)
    Tabulate some mathematical function or callable object

Pass ordinary functions:

  • Note: Python ordinary functions are objects with __call__ method:
import math 

>>> math.sqrt.__call__(25)
5.0

>>> math.sqrt.__call__(125)
11.180339887498949
>>> 

>>> m.tabulateFunction(math.exp, -5.0, 5.0, 1.0)
Tabulating range:  ; xmin = -5.0000 ; xmax = 5.0000 ; step = 1.0000
 -5.0000    0.0067
 -4.0000    0.0183
 -3.0000    0.0498
 -2.0000    0.1353
 -1.0000    0.3679
  0.0000    1.0000
  1.0000    2.7183
  2.0000    7.3891
  3.0000   20.0855
  4.0000   54.5982
  5.0000  148.4132
>>> 

>>> a.tabulateFunction(math.sqrt, 0.0, 25.0, 5.0)
Tabulating range:  ; xmin = 0.0000 ; xmax = 25.0000 ; step = 5.0000
  0.0000    0.0000
  5.0000    2.2361
 10.0000    3.1623
 15.0000    3.8730
 20.0000    4.4721
 25.0000    5.0000

Pass lambda functions:

>>> m.tabulateFunction(lambda x: x, 0.0, 25.0, 4.0)
Tabulating range:  ; xmin = 0.0000 ; xmax = 25.0000 ; step = 4.0000
  0.0000    0.0000
  4.0000    4.0000
  8.0000    8.0000
 12.0000   12.0000
 16.0000   16.0000
 20.0000   20.0000
 24.0000   24.0000
>>> 

>>> m.tabulateFunction(lambda x: 3 * x + 5, 0.0, 25.0, 4.0)
Tabulating range:  ; xmin = 0.0000 ; xmax = 25.0000 ; step = 4.0000
  0.0000    5.0000
  4.0000   17.0000
  8.0000   29.0000
 12.0000   41.0000
 16.0000   53.0000
 20.0000   65.0000
 24.0000   77.0000

Class LinearFun:

class LinearFun:
    def __init__(self, a, b):
        self.a = a
        self.b = b
    def __call__(self, x):
        return self.a * x + self.b
    def eval(self, x):
        return self.a * x + self.b

Pass a callable object:

>>> lfun = LinearFun(5, 4)
>>> lfun(3)
19
>>> lfun(6)
34
>>> lfun.eval(3)
19
>>> lfun.eval(6)
34

>>> m.tabulateFunction(lfun, 0.0, 25.0, 4.0)
Tabulating range:  ; xmin = 0.0000 ; xmax = 25.0000 ; step = 4.0000
  0.0000    4.0000
  4.0000   24.0000
  8.0000   44.0000
 12.0000   64.0000
 16.0000   84.0000
 20.0000  104.0000
 24.0000  124.0000
>>> 
>>> 

>>> a.tabulateFunction(lfun.eval, 0.0, 25.0, 4.0)
Tabulating range:  ; xmin = 0.0000 ; xmax = 25.0000 ; step = 4.0000
  0.0000    4.0000
  4.0000   24.0000
  8.0000   44.0000
 12.0000   64.0000
 16.0000   84.0000
 20.0000  104.0000
 24.0000  124.0000

>>> lfun.a 
5
>>> lfun.a  = 0
>>> 
>>> m.tabulateFunction(lfun.eval, 0.0, 25.0, 4.0)
Tabulating range:  ; xmin = 0.0000 ; xmax = 25.0000 ; step = 4.0000
  0.0000    4.0000
  4.0000    4.0000
  8.0000    4.0000
 12.0000    4.0000
 16.0000    4.0000
 20.0000    4.0000
 24.0000    4.0000

2.3 Native Modules or C++ binding with Pybind11

2.3.1 Overview

Pybind11 is a C++ header-only library that simplifies the creation of Python binding to existing C++ libraries or creating Python native modules. This library is easier and less cumbersome to use than Python raw C API due to the library declarative and intuitive syntax and lots of C++ wrappers.

Use cases:

  • Create high-performance python modules in C++
  • Create binding to existing C or C++ libraries.
  • Embed Python in a C++ application
  • Access to operating system APIs not exposed in Python standard library

Documentation:

Github Repository:

Conan package reference:

Headers:

  • <pybind11/pybind11.h>
    • Main header, that should be always included
  • <pybind11/stl.h>
    • Provides implicit convesion from STL containers to Python types.
  • <pybind11/functional.h>

2.3.2 Example

  1. Overview

    The following sample CMake/conan project contains many PyBind11 examples about binding existing C++ functions and classes and access or call Python objects and APIs from C++.

  2. File: CMakeLists.txt

    Note: In order to use this CMakeLists.txt file, it is necessary an installation of conan package manager available in $PATH environment variable, in other words, accessible from command line.

    #      Author: Caio Rodrigues 
    # Description:  Sample CMake + Conan + Pybind11 project
    #        Note:  Requires Conan and Python Development files installed 
    #----------------------------------------------------------
    
    cmake_minimum_required(VERSION 2.8)
    project(pybind11-experiments)
    set(CMAKE_CXX_STANDARD 17)
    
    
    # ============  Conan Boostrap  ======================#
    
    # Download automatically, you can also just copy the conan.cmake file
    if(NOT EXISTS "${CMAKE_BINARY_DIR}/conan.cmake")
       message(STATUS "Downloading conan.cmake from https://github.com/conan-io/cmake-conan")
       file(DOWNLOAD "https://github.com/conan-io/cmake-conan/raw/v0.13/conan.cmake"
                     "${CMAKE_BINARY_DIR}/conan.cmake")
    endif()
    
    include(${CMAKE_BINARY_DIR}/conan.cmake)
    
    conan_cmake_run(REQUIRES   pybind11/2.2.4@conan/stable
                    BASIC_SETUP
                    BUILD missing)
    
    #============ Additional Commands =============#
    
    # Copy target file to current directory whenerver it is rebuilt
    function(copy_after_build TARGET_NAME )
        # Note: CMAKE_CURRENT_LIST_DIR is the directory where is this
        # CMakeLists.txt file.
        set(DESTDIR ${CMAKE_CURRENT_LIST_DIR}/bin/)
        file(MAKE_DIRECTORY ${DESTDIR})
    
        # Copy binary file to <CMakeLists.txt didctory>./bin
        # after target is compiled.
        add_custom_command(TARGET ${TARGET_NAME} POST_BUILD
          COMMAND ${CMAKE_COMMAND} -E copy
                                   $<TARGET_FILE:${TARGET_NAME}> ${DESTDIR}
                                   )
    endfunction()
    
    
    #----------------------------------------------#
    #    Target Configurations                     #
    #----------------------------------------------#
    
    find_package(pybind11 CONFIG REQUIRED)
    
    pybind11_add_module(SampleModule SampleModule.cpp)
    # Copy binary to ./bin directory after module is built
    copy_after_build(SampleModule)
    
    # Target1: cashflow - Python module basics
    #.....................................
    pybind11_add_module(cashflow cashflow.cpp)
    copy_after_build(cashflow)
    
  3. File: SampleModule.cpp
    1. Full Source
    2. Header and Namespaces

      Header files:

      #include <iostream>
      #include <string>
      #include <vector>
      #include <iomanip>
      #include <cmath>
      
      #include <pybind11/pybind11.h>
      #include <pybind11/stl.h>
      #include <pybind11/functional.h>
      

      Namespaces:

      namespace py = pybind11;
      using namespace pybind11::literals;
      
    3. Bindings C++ Functions, enums and Classes

      Bindings to C++ functions, enums and classes without any knowledge of Python API.

      • Function add:
      int add(int i, int j)
      {
          return i + j;
      }
      

      Registration:

      PYBIND11_MODULE(SampleModule, m) {
         // optional module docstring
         m.doc() = "Sample Python built with C++ CeePlusPlus ";
      
         // Register an ordinary C++ function
         m.def("add",                                 // Function Name in Python side
               &add,                                  // Function Pointer
               "A function which adds two numbers",   // Docstring
               py::arg("x"),                          // Name of argument 1
               py::arg("y") = 10);                    // Name of argument 2 with default value
       ... ...   ... ...   ... ...   ... ...   ... ... 
      }
      
      • Function vectorCumulativeSum
      std::vector<double>
      vectorCumulativeSum(std::vector<double> const& xs)
      {
          std::vector<double> out(xs.size());
          double sum = 0.0;
          for(size_t i = 0; i < xs.size(); i++)
          {
              sum += xs[i];
              out[i] = sum;
          }
          return out;
      }
      

      Registration:

      PYBIND11_MODULE(SampleModule, m) {
       ...   ...   ...   ...   ...
        m.def("vectorCumulativeSum", &vectorCumulativeSum);    
       ...   ...   ...   ...   ...
      } 
      
      • Function tabulate1
      void tabulate1(double start, double stop, double step,
                     std::function<double (double)> func)
      {
          if(start > stop || step <= 0)
              throw std::runtime_error(" Error: Invalid domain "
                                       "\n Expected: start < stop && step > 0 "
                                       );
      
          std::cout << std::setprecision(5) << std::fixed;
          for(double x = start; x <= stop; x+= step)
              std::cout << std::setw(10) << x
                        << std::setw(10) << func(x)
                        << std::endl;
      }
      

      Registration:

      PYBIND11_MODULE(SampleModule, m) {
       ...   ...   ...   ...   ...
        m.def("tabulate1", &tabulate1);
       ...   ...   ...   ...   ...
      } 
      
      • Enum class MathErrorCode:
      enum class MathErrorCode: std::uint32_t
      {
           OK           = 0x00
         , OVERFLOW     = 0x01
         , UNDERFLOW    = 0x04
         , NOT_A_NUMBER = 0x08
      };
      

      Registration:

      PYBIND11_MODULE(SampleModule, m) {
       ...   ...   ...   ...   ...
      
        py::enum_<MathErrorCode>(m, "MathErrorCode", py::arithmetic())
           .value("OK",           MathErrorCode::OK)
           .value("OVERFLOW",     MathErrorCode::OVERFLOW)
           .value("UNDEFLOW",     MathErrorCode::UNDERFLOW)
           .value("NOT_A_NUMBER", MathErrorCode::NOT_A_NUMBER);
          // .export_values();
       ...   ...   ...   ...   ...
      } 
      
      • Class LinearFunctor:
      // Sample "function-object class"
      class LinearFunctor
      {
      public:
          double A = 0, B = 0;
      
          LinearFunctor();
          LinearFunctor(double a, double b): A(a), B(b){ }
      
          double GetA() const   { return A; }
          void   SetA(double a) { A = a; }
          double GetB() const   { return B; }
          void   SetB(double b) { B = b; }
      
          void show() const
          {
              std::cout << " LinearFunction: y(x) = A * x + B" << std::endl;
              std::cout << " => A = " << this->A << " ; B = " << this->B << std::endl;
          }
          std::string toString() const
          {
              std::stringstream ss;
              ss << " LinearFunction: y(x) = A * x + B" << std::endl;
              ss << " => A = " << this->A << " ; B = " << this->B << std::endl;
              return ss.str();
          }
          // Function-call operator
          double operator()(double x)
          {
              return A * x + B;
          }
      };
      

      Registration:

      PYBIND11_MODULE(SampleModule, m) {
        ... ...   ... ...   ... ...   ... .
      
          // Register LinearFunction
          py::class_<LinearFunctor>(m, "LinearFunctor")
                  .def(py::init<double, double>())             // Register overloaded consructor
                  .def("GetA", &LinearFunctor::GetA)            // Reister method GetA()
                  .def("GetB", &LinearFunctor::GetB)            // Register method GetB()
                  .def("SetA", &LinearFunctor::SetA)            // Reister method GetA()
                  .def("SetB", &LinearFunctor::SetB)            // Register method GetB()
                  .def("show", &LinearFunctor::show)            // Register method show
                  .def("call", &LinearFunctor::operator())      // Register function-call operator with name 'call'
                  .def("__call__", &LinearFunctor::operator ()) // Register fun-call operator
                  .def("__repr__", &LinearFunctor::toString)    // Register strin representation
                  .def_readwrite("A", &LinearFunctor::A)        // Register field A
                  .def_readwrite("B", &LinearFunctor::B);       // Register field B
      
        ... ... 
      }
      
    4. Functions with knowledge of Python API
      • Function computeFormula
        • This function takes a Python tuple py::args as argument and returns a Python dictionary py::dict
      // Sample function that takes a python tuple as argument and
      // returns a dictionary.
      py::dict
      computeFormula(py::args args)
      {
          py::print(" [INFO] Number of args = {s1} ; args = {s2}",
                    "s1"_a = args.size(), "s2"_a = args);
          std::cout << " [INFO] Arguments are = " << py::str(args) << std::endl;
          int    n = py::cast<int>(args[0]);
          double x = py::cast<double>(args[1]);
          double y = py::cast<double>(args[2]);
      
          py::dict d{ "n"_a =  n, "2x"_a = 2.0 * x, "addxy"_a = (x + y)};
          d["m"] = 5.0 * x + 4.5 * y;
          py::print(" [INFO] Dictionary d = {d} ", "d"_a = d);
          return d;
      }
      

      Registration:

      m.def("computeFormula", &computeFormula);
      
      • Function showDictionary
      // Function that takes a Python dictionary as argument
      void showDictionary(py::dict map)
      {
          std::cout << " ====== Show dictonary =========== " << std::endl;
          for(auto const& item : map)
              std::cout << " key = "
                        << item.first << " ; "
                        << item.second
                        << std::endl;
      }
      

      Registration:

      m.def("showDictionary",
            &showDictionary,
            "Display dictionary in console" );    
      
      • Function pyFuncArgsKwargs
        • Sample function that takes a python tuple (py::args) and named arguments py::kwargs.
      // Sample Python function taking a tuple and dictionary as argument
      void pyFuncArgsKwargs(py::args args, py::kwargs kwargs)
      {
          double x = py::cast<double>(args[0]);
          int    n = py::cast<double>(args[1]);
          bool   verbose = false;
      
          py::print(" [TRACE]   args = ", args);
          py::print(" [TRACE] kwargs = ", kwargs);
      
          std::cout << std::boolalpha;
      
          if(kwargs.contains("verbose"))
          {
              verbose = py::cast<bool>(kwargs["verbose"]);
          }
          std::cout << " [INFO] verbose option = "
                    << verbose << std::endl;
      
          std::cout << " sqrt(x) = " << std::sqrt(x)
                    << " ; 2 * n = " << 2 * n
                    << std::endl;
      }
      

      Registration:

      m.def("dictionaryArgument", &dictionaryArgument);
      
      • Function: dictionaryArgument
      // Sample function that takes a Python dictionary returning a
      // Python tuple as argument.
      py::tuple
      dictionaryArgument(py::dict dict)
      {
          std::cout << " >> Dictionary = " << py::str(dict) << std::endl;
          auto xx = dict["x"];
          auto yy = dict["y"];
          double x = py::cast<double>(xx);
          double y = py::cast<double>(yy);
          std::string name = py::cast<std::string>(dict["name"]);
          double z = std::sqrt(x * x + y * y);
      
          // Return a tuple on python side
          py::tuple tpl = py::make_tuple(x, y, z, name);
          py::print(" [INFO] Tuple = {} ", tpl);
          return tpl;
      }
      

      Registration:

      m.def("pyFuncArgsKwargs",
            &pyFuncArgsKwargs,
            "Sample Python function taking a tuple");
      
      • Function: callPyObjectMethod
        • This funtion takes any Python object as argument and prints its attributed .name and call its method named ".call". If the object does not have those fields and methods, a runtime error will happen.
      // Call Python Objec's Method (Duck typing)
      void callPyObjectMethod(py::object obj, double k)
      {
          // Print objec's name attribute
          py::print(" [TRACE] obj.name = ", obj.attr("name"));
      
          std::string name = py::cast<std::string>(obj.attr("name"));
          std::cout << " [COUT] object.name = " << name << std::endl;
      
          // Invoke object's method .call
          auto call = obj.attr("call");
          py::print(" [TRACE] obj.call(6.0) = ", call(6.0));
          double x = py::cast<double>(call(k));
          py::print(" [TRACE] 100.0 * objct.call(x) = ", 100.0 * x);
      }
      

      Registration:

      m.def("callPyObjectMethod", &callPyObjectMethod);
      
    5. Module Registration of Functions and classes
      PYBIND11_MODULE(SampleModule, m) {
          // optional module docstring
          m.doc() = "Sample Python built with C++ CeePlusPlus ";
      
          //----------- Register C++ Functions --------//
          //
      
          // Register an ordinary C++ function
          m.def("add",                                 // Function Name in Python side
                &add,                                  // Function Pointer
                "A function which adds two numbers",   // Docstring
                py::arg("x"),                          // Name of argument 1
                py::arg("y") = 10);                    // Name of argument 2 with default value
      
          m.def("cppLambda"
                ,[](double x, double y){ return 3.0 * x + y;}
                ,"A C++ lambda object or functor"
                //,py::arg("x"), py::args("y") = 15
          );
      
          m.def("vectorCumulativeSum", &vectorCumulativeSum);
      
          m.def("tabulate1", &tabulate1);
      
       ... ...  ... ...  ... ...  ... ...  ... ... 
       }
      
  4. Compile and Running
    1. Compile

      The project can be compiled from any IDE that supports CMake or from command line either by invoking CMake or a C++ compiler manually. All its need to build from an IDE is to open the CMakeList.txt.

      Clone Repository:

      $ cd /tmp 
      $ git clone https://github.com/caiorss/example-project-pybind11 pybind11
      $ cd pybind11 
      
      $ tree .
      .
      ├── bin
      │   └── test-script.py
      ├── cashflow.cpp
      ├── CMakeLists.txt
      └── SampleModule.cpp
      
      1 directory, 4 files
      

      Build from QTCreator IDE:

      • Just open the file CMakeLists.txt and click at the hammer button or open this file from command line. Similar procedure can be used with other IDEs. After compiled, the Python modules are copied to directory ./bin.
      $ qtcreator CMakeLists.txt &
      

      Build from command line:

      • Set configuration. Note: Debug build type has debugging symbols enabled and optmization disabled.
      $ cmake -H. -Bbuild -DCMAKE_BUILD_TYPE=Debug
      
      -- Conan: Automatic detection of conan settings from cmake
      -- Conan: Settings= -s;build_type=Debug;-s;compiler=gcc;-s;compiler.version=7;-s;compiler.libcxx=libstdc++11
       ...  ...  ...  ...  ...  ...  ...  ...  ...  ... 
      
      -- Build files have been written to: /tmp/pybind11/build
      
      • Compile:
      $ cmake --build build --target
      
      Scanning dependencies of target SampleModule [ 25%] Building CXX
      object CMakeFiles/SampleModule.dir/SampleModule.cpp.o [ 50%] Linking
      CXX shared module lib/SampleModule.cpython-37m-x86_64-linux-gnu.so [
      50%] Built target SampleModule
      
      ... ... ... ... ... ... ... ... ... ... 
      
      • The Cmake function copy_after_build copies the python native modules (shared libraries *.so files on Linux or *.pyd on Windows) to the directory ./bin
      # Module files 
      $ ls bin/
      cashflow.cpython-37m-x86_64-linux-gnu.so*      test-script.py
      SampleModule.cpython-37m-x86_64-linux-gnu.so*
      
      $ file bin/SampleModule.cpython-37m-x86_64-linux-gnu.so 
      bin/SampleModule.cpython-37m-x86_64-linux-gnu.so: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, with debug_info, not stripped
      
    2. Running

      Enter at directory where is the Python native module SampleModule, in this case, the file: SampleModule.cpython-37m-x86_64-linux-gnu.so

      $ cd bin/
      
      $ ls
      cashflow.cpython-37m-x86_64-linux-gnu.so*      test-script.py
      SampleModule.cpython-37m-x86_64-linux-gnu.so*
      
      $ python3
      Python 3.7.1 (default, Dec 14 2018, 19:28:38) 
      [GCC 7.3.0] :: Anaconda, Inc. on linux
      Type "help", "copyright", "credits" or "license" for more information.
      

      Import SampleModule from REPL:

      >>> import SampleModule as t
      

      Get module help:

      >>> help(t)
      
      Help on module SampleModule:
      
      NAME
          SampleModule - Sample Python built with C++ CeePlusPlus
      
      CLASSES
          pybind11_builtins.pybind11_object(builtins.object)
              LinearFunctor
              MathErrorCode
       ....      ....      ....      ....      ....     
      
      
      FUNCTIONS
          add(...) method of builtins.PyCapsule instance
              add(x: int, y: int=10) -> int
      
              A function which adds two numbers
      
          callPyObjectMethod(...) method of builtins.PyCapsule instance
              callPyObjectMethod(arg0: object, arg1: float) -> None
      
          computeFormula(...) method of builtins.PyCapsule instance
              computeFormula(*args) -> dict
      
          cppLambda(...) method of builtins.PyCapsule instance
              cppLambda(arg0: float, arg1: float) -> float
      
              A C++ lambda object or functor
      
          dictionaryArgument(...) method of builtins.PyCapsule instance
      
       ... ...  ... ...  ... ...  ... ...  ... ...  ... ... 
      

      Test function and classes without knowledge of Python API

      • Function add
      >>> help(t.add)
      
      add(...) method of builtins.PyCapsule instance
          add(x: int, y: int=10) -> int
      
          A function which adds two numbers
      
      
      >>> t.add(20)
      30
      >>> t.add(100)
      110
      >>> t.add(15, 25)
      40
      
      • Function vectorCumulativeSum
      >>> help(t.vectorCumulativeSum)
      
      vectorCumulativeSum(...) method of builtins.PyCapsule instance
          vectorCumulativeSum(arg0: List[float]) -> List[float]
      
      >>> t.vectorCumulativeSum([1, 10, 5, -8, 25, 10])
      [1.0, 11.0, 16.0, 8.0, 33.0, 43.0]
      
      • Function tabulate1
      >>> t.tabulate1(0, 5, 1, lambda x: x * x - 5 * x  + 8)
         0.00000   8.00000
         1.00000   4.00000
         2.00000   2.00000
         3.00000   2.00000
         4.00000   4.00000
         5.00000   8.00000
      
      >>> import math
      >>> t.tabulate1(0, 5, 1, lambda x: math.exp(x))
         0.00000   1.00000
         1.00000   2.71828
         2.00000   7.38906
         3.00000  20.08554
         4.00000  54.59815
         5.00000 148.41316
      
      >>> t.tabulate1(0, -5, 1, lambda x: x * x - 5 * x  + 8)
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
      RuntimeError:  Error: Invalid domain 
       Expected: start < stop && step > 0 
      
      • Enum MathError
      >>> status = t.MathErrorCode.OK
      
      >>> status == t.MathErrorCode.OK 
      True
      
      >>> status = t.MathErrorCode.OVERFLOW
      
      >>> status == t.MathErrorCode.OK 
      False
      
      >>> status == t.MathErrorCode.OVERFLOW
      True
      
      >>> int(t.MathErrorCode.OVERFLOW)
      1
      
      >>> int(t.MathErrorCode.NOT_A_NUMBER)
      8
      
      • Class LinearFunctor
      >>> lfun = t.LinearFunctor(10, 5)
      
      >>> lfun
       LinearFunction: y(x) = A * x + B
       => A = 10 ; B = 5
      
      >>> lfun.show()
       LinearFunction: y(x) = A * x + B
       => A = 10.00000 ; B = 5.00000
      >>> 
      
      >>> lfun(4)
      45.0
      
      >>> lfun(5)
      55.0
      
      >>> lfun(10)
      105.0
      >>> 
      
      // Get coefficients 
      //-----------------------
      >>> lfun.A
      10.0
      
      >>> lfun.B
      5.0
      
      >>> lfun.GetA()
      10.0
      
      >>> lfun.GetB()
      5.0
      >>> 
      
      // Change coefficients
      //-------------------
      >>> lfun.SetA(-5)
      
      >>> lfun.SetB(15)
      
      >>> lfun
       LinearFunction: y(x) = A * x + B
       => A = -5 ; B = 15
      
      >>> lfun.call(5.64)
      -13.2
      
      >>> lfun(5.64)
      -13.2
      >>> 
      

      Test function and classes with knowledge of Python API

      • Function: computeFormula
      >>> result = t.computeFormula(100, 5.67, 9.71)
       [INFO] Number of args = {s1} ; args = {s2}
       [INFO] Arguments are = (100, 5.67, 9.71)
       [INFO] Dictionary d = {d} 
      
      >>> result
      {'n': 100, '2x': 11.34, 'addxy': 15.38, 'm': 72.04500000000002}
      
      >>> result["n"]
      100
      
      >>> result["2x"]
      11.34
      
      • Function: showDictionary
      >>> t.showDictionary({'x': 10.34, 'y': "numerical electronic integrator", "name": "ARM-CPU"})
       ====== Show dictonary =========== 
       key = x ; 10.34
       key = y ; numerical electronic integrator
       key = name ; ARM-CPU
      >>> 
      
      >>> t.showDictionary(100)
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
      TypeError: showDictionary(): incompatible function arguments. The following argument types are supported:
          1. (arg0: dict) -> None
      
      Invoked with: 100
      
      • Function: pyFuncArgsKwargs
      >>> t.pyFuncArgsKwargs(2.7824, 200)
       [TRACE]   args =  (2.7824, 200)
       [TRACE] kwargs =  {}
       [INFO] verbose option = false
       sqrt(x) = 1.66805 ; 2 * n = 400
      
      >>> t.pyFuncArgsKwargs(2.7824, 200, verbose = True)
       [TRACE]   args =  (2.7824, 200)
       [TRACE] kwargs =  {'verbose': True}
       [INFO] verbose option = true
       sqrt(x) = 1.66805 ; 2 * n = 400
      
      >>> t.pyFuncArgsKwargs(2.7824, 200, verbose = False)
       [TRACE]   args =  (2.7824, 200)
       [TRACE] kwargs =  {'verbose': False}
       [INFO] verbose option = false
       sqrt(x) = 1.66805 ; 2 * n = 400
      
      
      >>> t.pyFuncArgsKwargs(2.7824, 200, verbose = "aaa")
       [TRACE]   args =  (2.7824, 200)
       [TRACE] kwargs =  {'verbose': 'aaa'}
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
      RuntimeError: Unable to cast Python instance of type <class 'str'> to C++ type 'bool'
      
      • Function: dictionaryArgument
      >>> result = t.dictionaryArgument({'x': 10.5, 'y': 5.6, 'name': "DUMMY"});
       >> Dictionary = {'x': 10.5, 'y': 5.6, 'name': 'DUMMY'}
       [INFO] Tuple = {}  (10.5, 5.6, 11.899999999999999, 'DUMMY')
      
      >>> result
      (10.5, 5.6, 11.899999999999999, 'DUMMY')
      
      >>> result[0]
      10.5
      
      >>> result[1]
      5.6
      
      >>> result[2]
      11.899999999999999
      >>> 
      
      • Function: callPyObjectMethod
      class Dummy:
        def __init__(self, name):
            self.name = name
      
        def call(self, x):
            print(" I was called by C++.")
            return 4.5 * x + 10
      
      >>> obj = Dummy("A dummy object")
      
      >>> t.callPyObjectMethod(obj, 4.5)
       [TRACE] obj.name =  A dummy object
       [COUT] object.name = A dummy object
       I was called by C++.
       [TRACE] obj.call(6.0) =  37.0
       I was called by C++.
       [TRACE] 100.0 * objct.call(x) =  3025.0
      
      
      >>> t.callPyObjectMethod(obj, 10.0)
       [TRACE] obj.name =  A dummy object
       [COUT] object.name = A dummy object
       I was called by C++.
       [TRACE] obj.call(6.0) =  37.0
       I was called by C++.
       [TRACE] 100.0 * objct.call(x) =  5500.0
      

3 R Language (RLang) Native Extension API

3.1 R Raw C API

3.1.1 Overview

R Language C data structures

Reference: http://adv-r.had.co.nz/C-interface.html

Data Structure Description  
Vector    
REALSXP Numeric Vector  
INTSXP Integer Vector  
LGSXP Logical Vector  
STRSXP Character Vector  
CPLSXP Complex Vectors  
     
VECSXP List  
     
Null    
NILSXP Null  
     
Misc    
CLOSXP Function (closure)  
ENVSXP Environment  
LISTSXP Pair list  
DOTSXP    
SYMSXP names/symbols  
     
Internal Objects Note: Only used by C functions, not R functions.  
LANGSXP Language constructs  
CHARSXP -  
PROMSXP -  
EXPRSXP Expressions  

Fuctions for coercing Scalars:

  • asLogical(xs): LGSXP -> int
  • asInteger(xs): INTSXP -> int
  • asReal(xs): REALSXP -> double
  • CHAR(asChar(x)): STRSXP -> const char*

R Types:

  • All numbers are float point (double 64 bits)
  • Scalar data are vectors, a single number is vector of size one.

Useful References:

3.1.2 Install R language and development tools

Install R development libraries with Anaconda Python Distribution

# Install R language, interpreter 
$ conda install r-base   

# Install R development libraries 
$ conda install r-devtools -c r

Locate where the headers:

$ find ~/opt/anaconda3/ -name "Rin*.h"
/home/dummy/opt/anaconda3/lib/R/include/Rinterface.h
/home/dummy/opt/anaconda3/lib/R/include/Rinternals.h
/home/dummy/opt/anaconda3/pkgs/r-base-3.5.1-h1e0a451_2/lib/R/include/Rinterface.h
/home/dummy/opt/anaconda3/pkgs/r-base-3.5.1-h1e0a451_2/lib/R/include/Rinternals.h


$ ls ~/opt/anaconda3/lib/R/include/
R_ext/     Rdefines.h   R.h           Rinternals.h  Rversion.h
Rconfig.h  Rembedded.h  Rinterface.h  Rmath.h       S.h

3.1.3 Example using R native C API

  1. Source

    File:

    Compilation:

    • Manual compilation
    $ clang++ rnative.cpp -o rnative.so -std=c++1z -g -O0 -Wall -shared -fPIC -I/home/archbox/opt/anaconda3/lib/R/include/
    
    • Automatic with R command line tool.
    $ R CMD SHLIB rnative.cpp 
    

    Source Code

    Headers:

    #include <iostream>
    #include <iomanip>
    #include <cmath>
    #include <vector> 
    
    #include <R.h>
    #include <Rinternals.h>
    #include <Rmath.h>
    

    Global object for logging when the library is loaded or unloaded.

    struct DLLIntializer{
         DLLIntializer()
         {
             std::cerr << "[TRACE] - DLL loaded into process OK." << std::endl;
         }
         ~DLLIntializer()
         {
             std::cerr << "[TRACE] - DLL unloaded from process OK." << std::endl;
         }
    };
    
    DLLIntializer init_hook{};
    

    Function: basicFunction

    • Computes the 10.0 * x + 4.0 * y where x and y are scalars (vectors of size one)
    • Note 1: Every argument of a R-function is a SEXP (S-expression structure) that can be a vector, scalar, matrix, R function and so on.
    • Note 2: Every scalar in R is a vector of size 1.
    extern "C"
    SEXP basicFunction(SEXP x, SEXP y)
    {
        std::cerr << " [TRACE] Calling R from C++" << std::endl;
        // Allocate a vector of size 1 of type real (double)
        // Remember: in R, a scalar is a vector of size one  
        SEXP out = PROTECT(allocVector(REALSXP, 1));
        // asReal(x) => Interprets the first element of the
        // vector x as a double 
        REAL(out)[0] = 10.0 * asReal(x) + 4.0 * asReal(y);
        //     UNPROTECT(< Number of allocations> )
        // OR: UNPROTECT(< Number of protects> )
        UNPROTECT(1);   
        return out;
    }
    
    

    Function: GetVersion

    • Returns library verion.
    extern "C"
    SEXP GetVersion()
    {
        SEXP version;
        // Allocate 1 string 
        PROTECT(version = allocVector(STRSXP, 1));
        SET_STRING_ELT(version, 0, ::mkChar("version 0.1"));
        UNPROTECT(1);
        return version;
    }
    

    Function: Metric

    • Display vector statistics
    extern "C"
    auto metrics(SEXP xs) -> SEXP 
    {
       size_t size = ::length(xs);
       double* vec =  REAL(xs);
       double sumsq  = 0.0;
       double sum    = 0.0;
       double x;
       for(auto i = 0; i < size; i++){
           x = vec[i];
           sumsq = sumsq + x * x;
           sum   = sum + x;
       }
       std::cout << std::setprecision(4) << std::fixed;
       std::cout << std::endl;
       std::cout << " =>    Mean = " << sum / size << std::endl;
       std::cout << " =>    Norm = " << std::sqrt(sumsq) << std::endl;
       std::cout << " =>    Size = " << size << std::endl;
       // If the function returns nothing, it should return R_NilValue 
       return R_NilValue;
    }
    
    

    Function: linearcomb

    • Return a linear combination between vectors.
    /** Computes the equation: 
      * For each i = 0, 1, 2, ... N - 1
      *   zs_vector[i] = a * xs_vector[i] + b * ys_vector[i] + c 
      *
      * Returns: Vector zs_vector[i]
      */
    extern "C"
    SEXP linearcomb(SEXP xs_vector, SEXP ys_vector, SEXP a, SEXP b, SEXP c)
    {
         size_t size = ::length(xs_vector);
         // Allocate output vector of same size 
         SEXP output = PROTECT(allocVector(REALSXP, size));
         double *pout, *pxs, *pys;
         pout = REAL(output);
         pxs  = REAL(xs_vector);
         pys  = REAL(ys_vector);
         for(size_t i = 0; i < size; i++)
                 pout[i] = asReal(a) * pxs[i] + asReal(b) * pys[i] + asReal(c);
         UNPROTECT(1);
         return output;
    }
    
  2. Load into R REPL

    Load shared library:

    > dyn.load("rnative.so")
    [TRACE] - DLL loaded into process OK.
    

    Call function basicFunction

    > .Call("basicFunction", 10, 5)
     [TRACE] Calling R from C++
    [1] 120
    

    Call function metrics:

    > .Call("metrics", 10)
     => Average = 10.0000
     =>    Norm = 10.0000
     =>    Size = 1
    NULL
    
    > .Call("metrics", c(4.5, 8.0, 6, -10.0))
     => Average = 2.1250
     =>    Norm = 14.8408
     =>    Size = 4
    NULL
    

    Call function linearcomb:

    • Test function linearcomb which perform linear combination between two vectors:
    > .Call("linearcomb", c(1, 2, 3, 4), c(4, 5, 6, 9), 10, 2, 0)
    [1] 18 30 42 58
    
    > .Call("linearcomb", c(1, 2, 3, 4), c(4, 5, 6, 9), 10, 2, 5)
    [1] 23 35 47 63
    
    > .Call("linearcomb", 4, 6, 10, 1, 2)
    [1] 48
    
    > .Call("linearcomb", 4, 6, 10, 5, 2)
    [1] 72
    

    End of the interative session:

    > quit()
    Save workspace image? [y/n/c]: n
    [TRACE] - DLL unloaded from process OK.
    

    Those functions need wrappers for making usage easier:

    File: rnative.R

    dyn.load("rnative.so")
    
    basicFunction <- function(x, y){  .Call("basicFunction", x, y) }
    
    getVersion <- function(){ .Call("GetVersion")}
    
    metrics <- function(xs){ .Call("metrics", xs) }
    
    # Compute: a * xs + b * ys + c
    linearcomb <- function(xs, ys, a, b, c){
        .Call("linearcomb", xs, ys, a, b, c)
    }
    
    
    testPackage <- function(){
        print(sprintf("basicFunction(4, 5) = %.3f", basicFunction(4, 5)))
        print(sprintf("basicFunction(10, 6) = %.3f", basicFunction(10, 6)))
    
        metrics(8)
        metrics(c(3.4, 9.0, 10.0, -15.75, 20.65))
    }
    

    This script can be loaded into the REPL with:

    > source("rnative.R")
    [TRACE] - DLL loaded into process OK.
    
    > getVersion()
    [1] "version 0.1"
    
    > metrics(c(4, 5, 10.5, 8.65, 5))
    
     =>    Mean = 6.6300
     =>    Norm = 15.8453
     =>    Size = 5
    NULL
    > 
    
    > quit()
    Save workspace image? [y/n/c]: n
    [TRACE] - DLL unloaded from process OK.
    

3.2 Rcpp Library for easier C++ Integration

3.2.1 Overview

Rcpp is a C++ library that makes easier to write native R functions in C++ and also to create bindings to existing C++ libraries.

Rcpp

Official Web Sites:

Doxygen - C++ API Docs:

Further Reading:

Lots of examples:

Repository:

Questions:

R Language Limitations

  • GIL - Global Interpreter Lock of garbage collector
  • Single thread
  • Interpretation overhead
  • Vectorized operations can cost many passes over the input data and be slower than a single loop.

Use Cases for Rcpp package and reasons to write R functions in C++

  • Create R wrappers for existing C++ libraries.
  • Create high performance R packages in C++
  • Access operating system functionality not exposed in the language.
  • Embed R interpreter into some application.
  • R Functions or algorithms written in C++ can be reused with other programming languages or in production systems.

Other RCPP Libraries in CRAN:

Misc:

  • Rcpp Modules => Rcpp extension wrapper library based on Boost.Python library which has a declarative and intituitive syntax for wrapping C++ libraries into Python.
  • Rcpp11 => Library similar to Rcpp Modules and based on Python Pybind11.
  • RProtBuf (Google's Protobuf)

Linear Algebra Libraries:

General Scientific Libraries

  • RcppGSL => (GNU Scientific Library Wrapper)

Install Rcpp Package

> install.packages(c("Rcpp", "RInside", "inline"))

RCPP Available Functions

Note: The RCPP convenience functions are vectorized, therefore they can operate on numeric vector or matrices without explicit loops.

Convenience Functions in Description  
Rcpp namespace    
Fundamental Transcendatal Function    
Rcpp::exp Exponential e^x  
Rcpp::log Natural logarithm base e = 2.718282  
Rcpp::log1p Natural logarithm, however, log(1 + x), where x is a small number  
Rcpp::log10 Logarith base 10  
Rcpp::sqrt Square root  
     
     
Statistcs    
Rcpp::sd Standard Deviation  
Rcpp::mean Mean, or average  
     
Rcpp::sum Sum of input values  
Rcpp::cumsum Cummulative sum  
Rcpp::diff    
Rcpp::pmin Return paralallel maxima of the input values  
Rcpp::pmax Return paralallel minima of the input values  
     
….. and more …    

RCPP Classes

RCPP Class R type typeof(<OBJ>)  
Vector    
Rcpp::IntegerVector integer  
Rcpp::NumericVector numeric  
Rcpp::LogicalVector logical  
Rcpp::CharacterVector character  
Rcpp::RawVector raw…  
Rcpp::ComplexVector complex…  
     
Matrix    
Rcpp::IntegerMatrix    
Rcpp::NumericMatrix    
Rcpp::LogicalMatrix    
Rcpp::RawMatrix    
Rcpp::ComplexMatrix    
     
Other Types    
Rcpp::Environment environment  
Rcpp::Function function (closure)  
Rcpp::XPtr externalptr  
Rcpp::Language Language  

3.2.2 Example - Native R library in C++ with Rcpp

  1. Source Files

    Source Files

    Compilation

    Manual:

    $ clang++ rcxx.cpp -o rcxx.so -std=c++1z -g -O0 -Wall -shared -fPIC \
       -I/home/archbox/opt/anaconda3/lib/R/include/ \
       -I/home/archbox/opt/anaconda3/lib/R/library/Rcpp/include/
    

    Build with CMake:

    $ cmake -Bbuild -H. 
    
    $ cmake --build build --target 
    Scanning dependencies of target rcxx
    [ 50%] Building CXX object CMakeFiles/rcxx.dir/rcxx.cpp.o
    [100%] Linking CXX shared library rcxx.so
    [100%] Built target r
    
    $ ls build/
    CMakeFiles/  CMakeCache.txt  cmake_install.cmake  Makefile  rcxx.so*
    
    $ cp build/rcxx.so . 
    

    File: rcxx.cpp

    Headers:

    #include <iostream>
    #include <cmath>
    #include <iomanip>
    #include <algorithm>
    #include <string>
    #include <map>
    
    #include <Rcpp.h>
    

    Global object for logging when the DLL is loaded or unloaded.

    struct DLLIntializer{
        DLLIntializer()
        {
            std::cerr << "[TRACE] - DLL loaded into process OK." << std::endl;
        }
        ~DLLIntializer()
        {
            std::cerr << "[TRACE] - DLL unloaded from process OK." << std::endl;
        }
    };
    
    // Global object allocated in process' data segment for
    // DLL logging during loading and unloading. 
    DLLIntializer init_hook{};
    

    Function libraryVersion

    • Returns the library version as string.
    RcppExport
    auto libraryVersion() -> SEXP
    {
        return Rcpp::wrap("version 1.0 - alpha");
    }
    

    Function linearcomb

    • Performs the transformation:
    • ys[i] = a * xs[i] + b for every i = 0, 1, 2 … N - 1
    /** Compute equation: 
     * for_each(i) => ys[i] = a * xs[i] + b 
     * return ys 
     */
    RcppExport
    SEXP linearcomb(SEXP a, SEXP b, SEXP xs)
    {
        Rcpp::NumericVector vx(xs);    
        double aa = Rcpp::as<double>(a);
        double bb = Rcpp::as<double>(b);
        Rcpp::NumericVector out(vx.size()); 
        size_t n = vx.size();
        // std::cerr << " [TRACE] Size of xs = " << n << std::endl;
        for(auto i = 0; i < vx.size(); i++)
           out[i] = aa * vx[i] + bb;
        return out;
    }
    

    Function: computeStatistics

    RcppExport
    auto computeStatistics(SEXP xs) -> SEXP
    {
        Rcpp::NumericVector vx{xs};
        double sum = 0.0;
        double sumsq = 0.0;
    
        std::for_each(vx.begin(), vx.end(), [&](double x)
        {
            sum   += x;
            sumsq += x * x;
        });
    
        auto result = std::map<std::string, double>
        {
            {"sum",   sum},
            {"sumsq", sumsq},
            {"mean",  sum / vx.size()},
            {"norm",  std::sqrt(sumsq)}
        };
        return Rcpp::wrap(result);
    }
    

    Function: computeStatistics2:

    RcppExport
    auto computeStatistics2(SEXP xs) -> SEXP
    {
        Rcpp::NumericVector vx{xs};
        double sum = 0.0;
        double sumsq = 0.0;
    
        std::for_each(vx.begin(), vx.end(), [&](double x)
        {
            sum   += x;
            sumsq += x * x;
        });
        return Rcpp::List::create(
                        Rcpp::Named("sum", sum)
                       ,Rcpp::Named("sumsq", sumsq)
                       ,Rcpp::Named("mean", sum / vx.size())
                       ,Rcpp::Named("norm", std::sqrt(sumsq))
                    );
    }
    

    Function tabulate:

    RcppExport
    SEXP tabulate(SEXP mfunc, SEXP mvec)
    {
        Rcpp::Function     func = mfunc;
        Rcpp::NumericVector vec = mvec;
        std::cout << std::setprecision(4) << std::fixed;
        for(auto const& x: vec)
            std::cout << std::setw(10) << x
                      << std::setw(10) << Rcpp::as<double>(func(x))
                      << std::endl;
        // Return nothing on R side
        return R_NilValue;
    }
    

    Function: vectorOperations

    • This function performs arithmetic operation on two input R numeric vectors.
    /* Compute:
     *
     *  FOR i = 0 TO N DO:
     *    out[i] = sqrt(va[i] * 4.0 + vb[i] * 5.0
     *  RETURN ouy
     *
     * Note: The vectorized operation are not efficient
     * Rcpp::sqrt(4.0 * va + 5.0 * vb) is equivaent to:
     * Rcpp::sqrt(operatior*(4.0, va) + operator*(5.0, vb))
     * Rcpp::sqrt(operator+(operatior*(4.0, va), operator*(5.0, vb)))
     *
     * Every operator function call is 1 loop, so, the total number
     * of passes over the input data is equal to:
     *  =>  1 (SQRT Call) + 3 (Number of operators) = 4
     *
     * This function requires 4 for-loops over the data,
     * thus a single for-loop is more efficient.
     */
    RcppExport
    SEXP vectorOperations(SEXP sa, SEXP sb)
    {
        using Vector = Rcpp::NumericVector;
        Vector va = sa;
        Vector vb = sb;
        Vector result = Rcpp::sqrt(4.0 * va + 5.0 * vb);
        return result;
    }
    

    Function: ShowNormalRandoms

    • This code demonstrates how to call R functions such as rnorm from C++ side. It takes as parameter the number of normally distributed random numbers that will be generated with mean 20.0 and standard deviation 5.0
    RcppExport
    SEXP ShowNormalRandoms(SEXP s_size)
    {
        size_t n = Rcpp::as<size_t>(s_size);
        std::cout << " [INFO] size = " << n << std::endl;
    
        // Get R function rnorm for generating N normally distributed
        // random numbers
        Rcpp::Function rnorm("rnorm");
    
        // Call R built-in function rnorm
        // that generates n normally distributed random numbers with
        // mean 20.0 and standard deviation std = 5.0
        //---------------------------------------------
        // NOTE: Every R fuction returns an S-expression struct
        SEXP sexp = rnorm(Rcpp::Named("n", n),
                          Rcpp::Named("sd", 5.0),
                          Rcpp::Named("mean", 20.0)
                          );
        // Convert s-expression to numeric vector
        Rcpp::NumericVector result{sexp};
    
        std::cout << std::setprecision(5) << std::fixed;
        size_t k = 0;
        std::for_each(result.begin(), result.end(),
                      [&](auto x) -> void
                       {
                        std::cout << std::setw(10) << k++
                                  << std::setw(10) << x
                                  << std::endl;
                       });
        // std::for_each(result.begin())
        return R_NilValue;
    }
    
  2. Loading into R REPL

    Interactive Session:

    > library(Rcpp)
    
    > # Load shared library into current REPL 
    > dyn.load("rcxx.so")
    [TRACE] - DLL loaded into process OK.
    

    Call function libraryVersion:

    > .Call("libraryVersion")
    [1] "version 1.0 - alpha"
    > 
    
    # Better 
    > .Call("libraryVersion", PACKAGE = "rcxx")
    [1] "version 1.0 - alpha"
    
    > .Call("libraryVersion", PACKAGE = "rm")
    Error in .Call("libraryVersion", PACKAGE = "rm") : 
      "libraryVersion" not available for .Call() for package "rm"
    

    Call function linearcomb

    > # Call function from shared library 
    > .Call("linearcomb", 10.0, 4.0, c(4.5, 9.8, 5.6, 10.0), PACKAGE = "rcxx")
    [1]  49 102  60 104
    
    > .Call("linearcomb", 10.0, 0, c(4.5, 9.8, 5.6, 10.0), PACKAGE = "rcxx")
    [1]  45  98  56 100
    

    Call function computeStatistics

    > .Call("computeStatistics", c(4.5, 3.5, 9.81, 4.6))
         mean      norm       sum     sumsq 
      5.60250  12.24321  22.41000 149.89610 
    > 
    
    > out = .Call("computeStatistics", c(4.5, 3.5, 9.81, 4.6), PACKAGE = "rcxx")
    > out
         mean      norm       sum     sumsq 
      5.60250  12.24321  22.41000 149.89610 
    
    > out["mean"]
      mean 
    5.6025 
    
    > out["norm"]
        norm 
    12.24321 
    
    > out["sum"]
      sum 
    22.41 
    

    Call function computeStatistics2

    > res = .Call("computeStatistics2", c(4.5, 3.5, 9.81, 4.6), PACAKGE = "rcxx")
    
    > res
    $sum
    [1] 22.41
    
    $sumsq
    [1] 149.8961
    
    $mean
    [1] 5.6025
    
    $norm
    [1] 12.24321
    
    > res$sum
    [1] 22.41
    
    > res$mean
    [1] 5.6025
    > 
    > 100 * res$mean
    [1] 560.25
    

    Call function tabulate

    > .Call("tabulate", sqrt, c(4.5, 3.5, 9.81, 4.6))
        4.5000    2.1213
        3.5000    1.8708
        9.8100    3.1321
        4.6000    2.1448
    NULL
    
    > .Call("tabulate", function(x){ 10 * x + 5 }, c(4.5, 3.5, 9.81, 4.6))
        4.5000   50.0000
        3.5000   40.0000
        9.8100  103.1000
        4.6000   51.0000
    NULL
    > 
    

    Call function vectorOperations

    > .Call("vectorOperations", c(4.0, 3.0, 2.0, 6.0), c(10.0, 5, 8, 4))
    [1] 8.124038 6.082763 6.928203 6.633250
    
    > xs = .Call("vectorOperations", c(4.0, 3.0, 2.0, 6.0), c(10.0, 5, 8, 4))
    
    > xs
    [1] 8.124038 6.082763 6.928203 6.633250
    
    > xs * xs
    [1] 66 37 48 44
    
    > sqrt(4.0 * 4.0 + 5.0 * 10.0)
    [1] 8.124038
    > 
    

    Call function ShowNormalRandoms

    • This function invokes the R built-in function rnorm from the C++ side and displays the result on the screen.
    > .Call("ShowNormalRandoms", 5, PACKAGE = "rcxx")
     [INFO] size = 5
             0  15.61839
             1  25.08418
             2  22.76518
             3  17.96007
             4  17.60298
    NULL
    
    > .Call("ShowNormalRandoms", 10, PACKAGE = "rcxx")
     [INFO] size = 10
             0  17.92591
             1  16.47821
             2  24.35774
             3  19.33816
             4  22.70113
             5  17.37169
             6  14.92140
             7  11.94084
             8  14.29367
             9  12.16308
    NULL
    

3.2.3 References

Created: 2021-06-04 Fri 15:08

Validate