Shared Libraries - Binary Components
Table of Contents
- 1. C-Interoperability and Shared Libraries - Binary Components
- 1.1. Overview
- 1.2. Design Guidelines for C-Interfaces
- 1.3. Cross platform Shared Library, C-Interfaces and Language Interoperability
- 1.3.1. Overview
- 1.3.2. Header macros
- 1.3.3. Namespace Linalg
- 1.3.4. C-interface of Namespace Linalg
- 1.3.5. Non-polymorphic Class - SampleClass
- 1.3.6. C-interface for SampleClass
- 1.3.7. Polymorphic class InterfaceClass
- 1.3.8. DLL startup function DLLMain
- 1.3.9. DLL entry point for run32dll.exe
- 1.3.10. C++ Client program - client1.cpp
- 1.3.11. Load DLL in Python3 REPL
- 1.3.12. Python 3 Client Code - Wrapper Module
- 1.3.13. CSharp Client Code - PInvoke
- 1.3.14. View symbols exported by the shared library
- 1.4. Plugins and Dynamically Loading Classes
- 1.5. Return string from a C++ function to Python Ctypes
- 2. Python Native Extension API
- 3. R Language (RLang) Native Extension API
1 C-Interoperability and Shared Libraries - Binary Components
1.1 Overview
Shared libraries are compiled object-code exporting functions, data structures and classes which can be used by many different programs. They provide many benefits such as: faster compilation time; less disk space usage since each program using the library do not need a copy of the library source code or its compiled-code; ability to update the library and provide security fixes to client applications without recompilation, just by replacing the library file.
On Windows, shared libraries are called DLLs which stands for Dynamic Linked Libraries, on Unix-like operating systems such as Linux, BSD and MacOSX, they are called SO - shared objects or DSO - dynamic shared objects.
Operating | Long Name | Short name | File | Binary Format |
---|---|---|---|---|
System | Extension | |||
Windows | Dynamic Linked Library | DLL | .dll | PE32/PE64 - Portable Executable |
Linux | (Dynamic) Shared Object | DSO or SO | .so | ELF or ELF64 (for 64 bits processors) |
BSD | (Dynamic) Shared Object | DSO or SO | .so | ELF or ELF64 |
MacOSX | - | dylib | .dylib or .so | MachO |
- Shared libraries are similar to executables, however they do not have the main() function entry point.
In addition to C and C++, shared libraries, containing functions with C-linkage, can also be used as binary components (libraries) and be consumed from higher level programming languages such as Python, Ruby, Java or C# through some foreign-function interface. In Python C-functions can be imported from shared libraries using the cytpes library; in C# C-functions can be consumed using the P-invoke API and in Java it is possible to use C-functions from shared libraries using the JNI (Java Native Interface) or JNA, which is easier to use.
Note:
- Shared libraries are specific to a particular operating system and are not part of C++ standard.
Use Cases and advantages:
- Multiple programs can reuse the same library reducing disk space.
- Faster compilation once the shared library is compiled.
- Changes in the library such as security fixes, updates and new features may not require client programs recompilation if the ABI is not broken.
- Better for security updates. For instance, if a program is statically compiled with open-ssl library, a program recompilation would be required to update the library to a new version with security fixes for some vulnerability. It could even be a greater security problem and waste of space, if all application depending on open-ssl were statically liked with this library. If a program uses the open-sll shared library, the library can be updated just by replacing a file and the program it will not need to be recompiled as long as there are no ABI breaking changes.
- Plugin system => The main program can contain an interface class called IInterface and load its implementations from shared libraries plugins. The implementation classes provided by the plugins or shared library files allow the program to be updated and extended at runtime without recompilation.
- Cross language interoperability => Functions using C-linkage can be called from any high level programming language with foreign function interface such as Python with ctypes library. C++ functions and classes cannot be called due to name mangling and compiler ABI issues. The workaround to this problem is to create a C-interface or functions with C-linkage for classes, functions and namespaces. Instances of a class can be passed around as opaque pointers (void*).
Problems:
- Due to C++ lack of standard ABI Application Binary Interface,
classes, STL containers, functions without C-linkage and namespaces
may not be reusable with a client program compiled with different
compiler than used to build the library. However, even in this
case, there is still the benefit of faster compilation time if the
shared library and the client application are part of the same
project.
- TL;DR => Classes, STL and functions without C-linkage of shared library cannot be used with a different compiler than the one used to build the DLL.
- Cross-compiling interoperability - The only way to ensure that a shared library can work with all compilers is by using a functions with C-linkage (functions with extern "C") annotation and creating C-linkage functions wrappers to classes, STL containers and classes.
- Classes can be made compatible among different compilers by using an interface class, a class with only pure virtual member functions, using only compatible types in the methods signature or declaration, therefore it cannot use STL container types such as std::string or std::vector. However, each implementation of an interface class can use STL containers internally.
MSDN Documentation
- Exporting C++ Functions for Use in C-Language Executables
- Exporting C Functions for Use in C or C++ Language Executables
Further Reading
1.2 Design Guidelines for C-Interfaces
Macro for C-linkage and exporting symbols
Instead of:
// Does not work on Unix-like OSes. // Only works on compilers with supporting this MSVC (Visual VC++) compiler-extension. extern "C" __declspec(dllexport) double SomeFunctionWithCLinkage(const char* name) { ... function code ... }
Better export the symbol using a macro for ensuring cross-platform portability:
- Note: Names of macros should be unique in order to avoid name collisions. This is why the macros are prefixed with library name.
- Note:
__declspec(dllexport)
=> MSVC (Visual C++) compiler extension only supported on MSVC and Mingw. (No in C++ ISO standard) - Note:
__attribute__ ((visibility ("default")))
=> GCC/Clang compiler extension used on Unix-like Oses for making symbols hidden by default.
#if defined(__cplusplus) // Code being compiled with C++ compiler #define MATHTOOLS_EXTERN_C extern "C" #else // Code being compiled with C compiler (Not C++ compiler) #define MATHTOOLS_EXTERN_C #endif #if defined(_WIN32) // MS Windows DLLs (*.dll) #define MATHTOOLS_EXPORT_C MATHTOOLS_EXTERN_C __declspec(dllexport) #else // Unix-like Shared Object (.so) operating systems and GCC. #define MATHTOOLS_EXPORT_C MATHTOOLS_EXTERN_C __attribute__ ((visibility ("default"))) #endif
So, the function code becomes:
MATHTOOLS_EXPORT_C double SomeFunctionWithCLinkage(const char* name) { ... function code ... }
Name Prefix
Functions with C-linkage are always in the global namespace, therefore every function needs an unique name in order to avoid name clashing which can result in undefined behavior. A good practice to avoid those name collisions is to use a common prefix before in the name of every function, for instance:
Instead of:
MATHTOOLS_EXPORT_C const char* GetVersion(); MATHTOOLS_EXPORT_C double RootSolver(double x0, double (* function) double); // Or in multi-line style: MATHTOOLS_EXPORT_C const char* GetVersion(); MATHTOOLS_EXPORT_C double RootSolver(double x0, double (* function) double);
Better:
// All functions contains the library name as a prefix in order to avoid // name clashes. MATHTOOLS_EXPORT_C const char* mathtools_GetVersion(); MATHTOOLS_EXPORT_C double mathtools_RootSolver(double x0, double (* function) double);
Naming Schema for C-wrapers or C-interfaces for C++ classes
Sample class:
class Plotter{ struct impl; std::unique_ptr<impl> m_pimpl; public: Plotter(); Plotter(double width, double height); Plotter(Plotter const& rhs) = default; ~Plotter() = default; void addPoint(double x, double y); void clear(); };
C-wrappers schema:
- C-interface for default constructor
MATHTOOLS_EXPORT_C Plotter* mathtools_Plotter_new() { return new (std::nothrow) Plotter(); }
- C-interface or C-Wrapepr for second constructor:
MATHTOOLS_EXPORT_C Plotter* mathtools_Plotter_new2(double width, double height) { return new (std::nothrow) Plotter(width, height); }
- C-interface for copy constructor:
// Wrapper for copy constructor MATHTOOLS_EXPORT_C Plotter* mathtools_Plotter_new_copy(Plotter* hPlotter) { return new (std::nothrow) Plotter(*hPlotter); }
- C-interface for destructor:
MATHTOOLS_EXPORT_C void mathtools_Plotter_delete(Plotter* hPlotter) { delete hPlotter; }
- C-interface for class method.
- As C-interfaces cannot throw exceptions, the last parameter is used for returning an error code.
// Schema: <Library Name>_<Class Name>_<method> MATHTOOLS_EXPORT_C void mathtools_Plotter_addPoint(Plotter* hPlotter, double x, double y) { hPlotter->addPoint(x, y); }
Exceptions and Error Handling
- DO NOT: Throw exceptions in functions with C-linkage (extern "C") as exceptions are incompatibles among different compilers. Moreover, the C-language cannot handle exceptions.
- DO: Instead of exceptions, return an error code enumeration as a function return value or as function parameter.
Example: If a function throws exceptions such as the function
mathtools_Plotter_addPoint
, the exceptions should be caught and the
function should return an error code eihter as return value or
function parameter.
MATHTOOLS_EXPORT_C void mathtools_Plotter_addPoint(Plotter* hPlotter, double x, double y, int* ErrorCode) { *ErrorCode = ErrorEnum::OK; try { // Method addPoint throws exceptions hPlotter->addPoint(x, y); } catch(Exception1 const& ex){ *ErrorCode = ErrorEnum::NO_SPACE_ERROR; // Set with value of some enum } catch(Exception2 const& ex){ *ErrorCode = ErrorEnum::SOME_ERROR_ENUM; } }
Passing std::string STL Object across a DLL boundary
Type synonym:
// h prefix stands for handle using hString = std::string*;
C-interface for std::string default constructor.
LIBRARY_EXPORT_C hString libprefix_string_new() { // std::nothrow => Do not throw std::bad_alloc, just return // a null pointer on failure. return new (std::nothrow) std::string(); }
C-interface for std::string copy-constructor.
LIBRARY_EXPORT_C hString libprefix_string_new_copy(hString hstr) { // std::nothrow => Do not throw std::bad_alloc, just return // a null pointer on failure. return new (std::nothrow) std::string(*hstr); }
C-interface for std::string overloaded constructor.
LIBRARY_EXPORT_C hString libprefix_string_new2(const char* text) { return new (std::nothrow) std::string(text); }
C-interface for std::string destructor.
// Style 1: LIBRARY_EXPORT_C hString libprefix_string_delete(hString hstr) { delete hstr; } // Style 2: LIBRARY_EXPORT_C hString libprefix_string_delete(hString hstr) { delete hstr; } // Style 3: LIBRARY_EXPORT_C hString libprefix_string_delete(hString hstr) { delete hstr; } // Style 4: Trailing return-type. LIBRARY_EXPORT_C auto libprefix_string_delete(hString hstr) -> hString { delete hstr; }
C-interface for std::string getter, returns const char*.
LIBRARY_EXPORT_C const char* libprefix_string_get(hString hstr) { return hstr->data(); }
C-interface for std::string setter.
LIBRARY_EXPORT_C void libprefix_string_set(hString hstr, const char* text) { *hstr = text; }
C-interface for std::string concat.
LIBRARY_EXPORT_C void libprefix_string_conat(hString hstr, const char* text) { *hstr = *hstr + text; }
C-function to print std::string object
LIBRARY_EXPORT_C void libprefix_string_print(hString hstr) { std::cout << *hstr << "\n"; }
C-wrapper to method std::string::clear()
LIBRARY_EXPORT_C void libprefix_string_clear(hString hstr) { hstr->clear(); }
Passing std::vector<double> across a DLL boundary
Handle type:
using hVectorD = std::vector<double>*;
Constructors:
LIBRARY_EXPORT_C hVectorD libprefix_vectorD_new() { return new (std::nothrow) std::vector<double>(); } LIBRARY_EXPORT_C hVectorD libprefix_vectorD_new2(size_t n, double x) { return new (std::nothrow) std::vector<double>(n, x); } LIBRARY_EXPORT_C hVectorD libprefix_vectorD_new3(size_t n, double [] array) { // Iterator-range constructor return new (std::nothrow) std::vector<double>(array, array + n); }
Destructor:
LIBRARY_EXPORT_C void libprefix_vectorD_delete(hVectorD hvec) { delete hvec; }
Get size:
LIBRARY_EXPORT_C size_t libprefix_vectorD_size(hVectorD hvec) { hvec->size(); }
Getter:
LIBRARY_EXPORT_C double libprefix_vectorD_get(hVectorD hvec, size_t n) { return hvec->operator[](n); }
Setter:
LIBRARY_EXPORT_C void libprefix_vectorD_set(hVectorD hvec, size_t n, double x) { hvec->operator[](n) = x; }
Element-wise fundamental math function 'sin()':
// Returns a new vector Out[i], Out[i] = sin(Input[i]), i = 0, 1, 2... Input.size() LIBRARY_EXPORT_C hVector libprefix_vectorD_sin(hVectorD hvec) { auto result = new (std::nothrow) std::vector<double>(hvec->size()); for(size_t i = 0; i <= hvec->size(); i++) (*result)[i] = std::sin((*hvec)[i]); return result; } // Returns a new vector Out[i], Out[i] = exp(Input[i]), i = 0, 1, 2... Input.size() LIBRARY_EXPORT_C hVector libprefix_vectorD_exp(hVectorD hvec) { auto result = new (std::nothrow) std::vector<double> (hvec->size()); for(size_t i = 0; i <= hvec->size(); i++) (*result)[i] = std::exp((*hvec)[i]); return result; }
Higher order C-function to apply a function to each element:
// Returns a new vector Out[i], Out[i] = exp(Input[i]), i = 0, 1, 2... Input.size() LIBRARY_EXPORT_C hVector libprefix_vectorD_map( hVectorD hvec /* Vector handle */ ,double Function (double)) /* Function pointer to math function */ { auto result = new (std::nothrow) std::vector<double>(hvec->size()); for(size_t i = 0; i <= hvec->size(); i++) (*result)[i] = Function((*hvec)[i]); return result; }
See also:
1.3 Cross platform Shared Library, C-Interfaces and Language Interoperability
1.3.1 Overview
Source:
Shared library Source Code
Sample Client C++ Program:
Sample Python 3 Client code: (Python3 module wrapper library)
GIST - Better for online view:
Compile the DLL libtest.dll and client program with MSVC
Shared Library
$ cl.exe testlib.cpp /EHsc /LD /nologo user32.lib
C++ Client program (client code)
$ cl.exe /EHsc client1.cpp /Fe:client1.exe testlib.lib && out.exe
Compile the DLL libtest.dll and client program with Mingw/GCC
Shared Library
$ g++ testlib.cpp -o testlib.dll -g -fvisibility=hidden -shared -std=c++14 -Wall
C++ Client program
$ g++ client.cpp -o client.exe -g -std=c++14 testlib.dll -Wall
Compile and run from GIST
Clone the GIST (Gisthub).
# Clone GIST: $ git clone https://gist.github.com/caiorss/2bba4c50866d9467aaa8c7792b337f71 dllex1 Cloning into 'dllex1'... remote: Enumerating objects: 66, done. remote: Counting objects: 100% (66/66), done. remote: Compressing objects: 100% (47/47), done. remote: Total 66 (delta 36), reused 45 (delta 19), pack-reused 0 Unpacking objects: 100% (66/66), done. # Enter directory containing sources $ cd dllex1/ $ tree . . ├── build.bat ├── client1.cpp ├── CMakeLists.txt ├── Makefile ├── pywrapper.py ├── testlib.cpp └── testlib.hpp 0 directories, 7 files
Build shared library on Linux.
- For Linux, build DLL and client code with make linux.
$ make linux echo "Build shared library" Build shared library g++ testlib.cpp -o libtestlib.so -std=c++14 -fPIC -shared -Wall echo "Build client code 1" Build client code 1 g++ client1.cpp -o client1.bin -std=c++14 libtestlib.so $ file libtestlib.so libtestlib.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=52a16d16991d7aa71763140e97a1422c1387711d, not stripped $ file client1.bin client1.bin: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=1a4885e112f72372cfd66816a564f49a24110c50, not stripped
Build shared library on Windows:
- Just run the batch script build-windows.bat or double click at it.
$ build-windows.bat ********************************************************************** ** Visual Studio 2017 Developer Command Prompt v15.5.6 ** Copyright (c) 2017 Microsoft Corporation ********************************************************************** [vcvarsall.bat] Environment initialized for: 'x64' testlib.cpp Creating library testlib.lib and object testlib.exp Microsoft (R) C/C++ Optimizing Compiler Version 19.12.25835 for x64 Copyright (C) Microsoft Corporation. All rights reserved. client1.cpp Microsoft (R) Incremental Linker Version 14.12.25835.0 Copyright (C) Microsoft Corporation. All rights reserved. /out:client1.exe client1.obj testlib.lib Creating library client1.lib and object client1.exp [StaticObject] => Initialize DLL testlib.cpp:48: <DllMain> DLL Loaded into the process => PID = 1240 testlib.cpp:54: <DllMain> DLL attached to process. === EXPERIMENT 1 ===> Import C++ functions from DLL client1.cpp:38: <main> Main process starts here. => Linalg::norm(xs) 7.4162 => xs = [5]( 1, 2, 3, 4, 5, ) === EXPERIMENT 2 ===> Import class from DLL Instance created with name = Dummy .. ... ... .. ... ... .. ... ... .. ... ...
Run C++ client code:
$ ./client1.bin [StaticObject] => Initialize DLL === EXPERIMENT 1 ===> Import C++ functions from DLL client1.cpp:38: <main> Main process starts here. => Linalg::norm(xs) 7.4162 => xs = [5]( 1, 2, 3, 4, 5, ) === EXPERIMENT 2 ===> Import class from DLL Instance created with name = Dummy Counter set to value = 100 cls.getName() = Dummy cls.get() = 100 ... ... ...
Run Python client code:
$ make py python3 pywrapper.py [INFO] libpath = libtestlib.so [StaticObject] => Initialize DLL Intializing library Library initialized OK. .. ... ... ...
1.3.2 Header macros
Sources:
teslib.hpp / Heade File - Macros
The header files uses the following macros to reduce the boilerplate necessary to export the library functions and classes.
- file: testlib.hpp => Macros for exporting functions and classes.
Macro: EXPORT_CPP
/** Macro EXPORT_CPP makes a symbol visible. */ #if defined(_WIN32) #define EXPORT_CPP __declspec(dllexport) #else #define EXPORT_CPP __attribute__ ((visibility ("default"))) // If not compiled on Windows, remove declspec compiler extension. #ifndef __declspec // For GCC only - make exported symbol visible symbol #define __declspec(param) __attribute__ ((visibility ("default"))) #endif #endif
The macro EXPORT_CPP annotates the symbol as visible by expanding to:
// On Windows EXPORT_CPP becomes __declspec(dllexport) // On Unix-like OSes, Linux, BSD, OSX ... __attribute__ ((visibility ("default")))
So a function int SomeFunction(double x, double y) becomes:
// Makes this symbol visible (exported) in the library (DLL) EXPORT_CPP int SomeFunction(double , double); // On Windows: __declspec(dllexport) int SomeFunction(double , double); // On Unix-like operating systems or Linux: __attribute__ ((visibility ("default"))) int SomeFunction(double , double);
Macro EXPORT_C:
/* Macro EXPORT_C is used for exporting symbol with C-linkage, it * means, without name mangling */ #ifdef __cplusplus // extern "C" - Indicates that a given symbol/function has C-linkage and // does not have name mangling. // // On Linux EXPORT_C becomes // => extern "C" __attribute__ ((visibility ("default"))) // // On Windows EXPORT_C becomes // => extern "C" __declspec(dllexport) #define EXPORT_C extern "C" EXPORT_CPP #else // If a C-compiler uses this header, remove 'extern "C"' #define EXPORT_C EXPORT_CPP #endif
Export symbols with C-linkage, it means that the function is exported without name decoration or name mangling. The function symbol matches exactly its name. Functions with C-linkage can be called by C-code; C++ code compiled with other compilers without ABI issues and foreign function interfaces of scripting languages such as Python Ctypes or C# Pinvoke. Note: If a function is defined with C-linkage, it can only use C-compatible types. So, it is no possible to define functions with C-linkage with overload, default parameter or non C-compatible arguments.
- On Windows, this macro is expanded to:
EXPORT_C int functionWith_C_linkage(MyClass* pointerToClass, int x, double z); // On Windows it becomes: extern "C" __declspec(dllexport) int functionWith_C_linkage(MyClass* pointerToClass, int x, double z); // This function could also be declared as: extern "C" __declspec(dllexport) int functionWith_C_linkage(MyClass* pointerToClass, int x, double z);
- On Unix-like OSes or linux, the macro EXPORT_C expands to:
EXPORT_C int functionWith_C_linkage(MyClass* pointerToClass, int x, double z); // On Linux, it becomes: extern "C" __attribute__ ((visibility ("default"))) int functionWith_C_linkage(MyClass* pointerToClass, int x, double z); // This function could also be declared as: extern "C" __attribute__ ((visibility ("default"))) int functionWith_C_linkage(MyClass* pointerToClass, int x, double z);
On Unix-like operating system (GCC or Clang compilers), all
functions in an object-code (aka compiled code) becomes visible by
default what can lead to undefined behavior when there are name
clashes due to repeated symbols. With appropriate compiler options
such as -fvisibility=hidden, the symbols can be made private by
default as happens in Windows where is necessary to annotate the
symbol as visible explicitly with __declspec(dllexport). On Unix-like
OSes for GCC and Clang compiler the annotation for making the symbol
visible is __attribute__ ((visibility ("default")))
. This statement
has no effect if the appropriate compiler flags are not set.
1.3.3 Namespace Linalg
Functions of Namespace Linalg
Namespace containing sample linear algebra functions, the function norm computes an Euclidean norm of a N-dimension vector, the function linTransform computes a linear transformation performing the computation v[i] * a + b for each vector element.
On Windows, all DLL symbols, such as functions, variables and clases,
are private by default, they are not exported by default as happens in
Unix-like OSes shared libraries. Therefore, in order to a function be
exported in a Windows shared library, it is necessary to use the MSVC or visual C++
compiler extension __declspec(dllexport)
before the function declaration.
- Interface => File: testlib.hpp
#ifdef __cplusplus namespace Linalg { EXPORT_CPP double norm(const std::vector<double>& xs); EXPORT_CPP std::vector<double> linTransform( double a, double b, std::vector<double>& xs ); EXPORT_CPP std::ostream& printVector(std::ostream& os, std::vector<double>& xs); } #endif
The compiler directives #ifdef __cplusplus … #endif are used for disabling this code block when the header is used by a C-compiler as the C-language does not support classes, namespaces and many other C++ features.
- file: testlib.cpp - Implementation
// Linear algebra tools namespace Linalg{ EXPORT_CPP double norm(const std::vector<double>& xs){ double sum = 0.0; for(const auto& x : xs) sum += x * x; return std::sqrt(sum); } EXPORT_CPP std::vector<double> linTransform(double a, double b, std::vector<double>& xs){ std::vector<double> out(xs.size()); for(size_t i = 0; i < xs.size(); i++){ out[i] = a * xs[i] + b; } return out; } EXPORT_CPP std::ostream& printVector(std::ostream& os, std::vector<double>& xs){ os << "[" << xs.size() << "]( "; for(const auto& x: xs) os << x << ", "; return os << " )"; } }
1.3.4 C-interface of Namespace Linalg
C-interface of Namespace Linalg
A C-interface is necessary for ensuring that the DLL can be called from a C-program or a foreign function interface of a higher level language.
- File: testlib.hpp => Function declaration.
// ======= C-interface for Linalg namespace =========// /** Handle or opaque pointer for std::vector<double> */ typedef void* hVectorD; /* ----- C-Wrappers for Linalg namespace ---- */ EXPORT_C double testlib_vectorD_Linalg_norm(hVectorD hv); EXPORT_C void testlib_vectorD_Linalg_printVector(const char* name, hVectorD hv);
- File: testlib.cpp => C-interface function implementations.
The type synonym hVectorD is handle or a opaque poiter for the type std::vector<double>. It is a workaround, to pass std::vector<double> accross a DLL functions with C-linkage and use std::vector<double> from C programs or foreign function interfaces.
//=========== C-wrappers ---------------/// // Handler for double vector using hVectorD = void*; using pVectorD = std::vector<double>*; /** C-wrapper for vector<double> constructor */ EXPORT_C hVectorD testlib_vectorD_make0(size_t n, double x){ return new std::vector<double>(n, x); } /** C-wrapper for range constructor */ EXPORT_C hVectorD testlib_vectorD_make1(size_t n, double array []){ return new std::vector<double>(array, array + n); } /** C-wrapper for setting elements of vector<double> */ EXPORT_C void testlib_vectorD_set(hVectorD hv, size_t n, double x){ reinterpret_cast<pVectorD>(hv)->operator[](n) = x; } /** C-wrapper for vector<double> destructor */ EXPORT_C void testlib_vectorD_delete(hVectorD hv){ delete reinterpret_cast<pVectorD>(hv); } /** C-wrapepr for Linalg::norm function */ EXPORT_C double testlib_vectorD_Linalg_norm(hVectorD hv){ return Linalg::norm(*reinterpret_cast<pVectorD>(hv)); } EXPORT_C void testlib_vectorD_Linalg_printVector(const char* name, hVectorD hv){ std::cout << name << " = "; Linalg::printVector(std::cout, *reinterpret_cast<pVectorD>(hv)); std::cout << std::endl; }
1.3.5 Non-polymorphic Class - SampleClass
- File: testlib.hpp => Class declaration.
In order to a class be exported in a Windows' shared library, it
requires the annotation __declspec(dllexport)
.
Note: It will not be possible to compiled client program using this class with a compiler different from the used to build the shared library libtest.dll. It happens because, the C++ ABI - Application Binary Interface is not the same accross different compilers. If the DLL is compiled with MSVC, the only way to use this class from a program compiled using MingW or even a different version of MSVC is by using a C-interface (functions with C-linkage) and opaque pointers (void* pointers).
// ======= Non-polymorphic class exported by DLL =========// #ifdef __cplusplus // Non-polymorphic class class EXPORT_CPP SampleClass{ public: SampleClass(); SampleClass(const std::string& name); ~SampleClass(); std::string getName() const; int get(); void set(int n); private: std::string m_name; int m_counter; }; #endif
- File: testlib.cpp => Class implementation.
SampleClass::SampleClass(const std::string& name) : m_name(name), m_counter(0) { std::cout << " Instance created with name = " << m_name << std::endl; } /** Delegated constructor on right-hand-side */ SampleClass::SampleClass(): SampleClass("unnamed"){} SampleClass::~SampleClass(){ std::string text = std::string("SampleClass => name = ") + m_name + " deleted"; DbgTrace(text); } std::string SampleClass::getName() const { return m_name; } int SampleClass::SampleClass::get(){ return m_counter; } void SampleClass::set(int n){ std::cout << " Counter set to value = " << n << std::endl; m_counter = n; }
1.3.6 C-interface for SampleClass
- File: testlib.hpp => Functions declarations.
/* ----- C-Wrappers for SampleClass namespace ---- */ using hSampleClass = void*; /** Nullable constructor zero-arg constructor */ extern "C" __declspec(dllexport) hSampleClass testlib_SampleClass_make0(); /** Other constructor */ EXPORT_C hSampleClass testlib_SampleClass_make1(const char* name); /** Destructor */ EXPORT_C void testlib_SampleClass_delete(hSampleClass hnd); /** Wrapper for get method */ EXPORT_C int testlib_SampleClass_get(hSampleClass hnd); /** Wrapper for set method */ EXPORT_C void testlib_SampleClass_set(hSampleClass hnd, int n); EXPORT_C const char* testlib_SampleClass_getName(hSampleClass hnd);
- File: testlib.cpp => Functions implementation.
/* ----- C-Wrappers for SampleClass namespace ---- */ using hSampleClass = void*; /** Nullable constructor zero-arg constructor */ EXPORT_C hSampleClass testlib_SampleClass_make0(); /** Other constructor */ EXPORT_C hSampleClass testlib_SampleClass_make1(const char* name); /** Destructor */ EXPORT_C void testlib_SampleClass_delete(hSampleClass hnd); /** Wrapper for get method */ EXPORT_C int testlib_SampleClass_get(hSampleClass hnd); /** Wrapper for set method */ EXPORT_C void testlib_SampleClass_set(hSampleClass hnd, int n); EXPORT_C const char* testlib_SampleClass_getName(hSampleClass hnd);
1.3.7 Polymorphic class InterfaceClass
- File: testlib.hpp => Class declaration.
As the name implies, this class is an interface class, which is a class containing only pure virtual functions (abstract methods). As a result, it is declared only in the header file and cannot be instatiated directly.
This class uses only C-types to be binary compatible with different compilers, if it used STL containers such as string or any other non-compatible C-type, it would not be possible to compile a C++ program using Mingw/GCC against this DLL built with MSVC.
// Polymorphic Interface class binary compatible across different // compilers as it does not use any STL container on the interface. #ifdef __cplusplus struct InterfaceClass{ /* Returns class unique ID */ virtual const char* getID() const = 0; /** Set class internal state */ virtual void setName(const char* name) = 0; virtual const char* getName() = 0; /** Virtual constructor */ virtual ~InterfaceClass() = default; // virtual ~InterfaceClass(); }; #else #define InterfaceClass void #endif
- File: testlib.cpp => class ImplementationA of interface InterfaceClass.
class ImplementationA: public InterfaceClass { private: std::string m_name; public: static constexpr const char* class_id = "ImplementationA"; ImplementationA(): m_name("Unammed-A"){ } ImplementationA(const std::string& name) : m_name(name){} ~ImplementationA(){ std::cout << " [INFO] ImplementationA deleted => name = " << m_name << " ; type = " << class_id << std::endl; } const char* getID() const { return class_id; } void setName(const char* name) { m_name = name; } const char* getName() { return m_name.c_str(); } };
- File: testlib.cpp => class ImplementationB of interface InterfaceClass.
class ImplementationB: public InterfaceClass { private: std::string m_name; public: static constexpr const char* class_id = "ImplementationB"; ImplementationB(): m_name("Unammed-B"){ } ImplementationB(const std::string& name) : m_name(name){} ~ImplementationB(){ std::cout << " [INFO] ImplementationB deleted => name = " << m_name << " ; type = " << class_id << std::endl; } const char* getID() const { return class_id; } void setName(const char* name) { m_name = name; } const char* getName() { return m_name.c_str(); } };
- File: testlib.hpp => C-interface and factory function to load implementations from the DLL.
/** Factory function */ EXPORT_C InterfaceClass* teslib_InterfaceClass_factory(const char* class_id); /** C-wrapper for destructor */ EXPORT_C void testlib_InterfaceClass_delete(InterfaceClass* hinst); /** C-wrapper for getID method */ EXPORT_C const char* testlib_InterfaceClass_getID(InterfaceClass* hinst); EXPORT_C void testlib_InterfaceClass_setName(InterfaceClass* hinst, const char* name); EXPORT_C const char* testlib_InterfaceClass_getName(InterfaceClass* hinst);
- File: testlib.cpp => C-interfaces and factory functions definitions.
EXPORT_C InterfaceClass* teslib_InterfaceClass_factory(const char* class_id) { auto s = std::string(class_id); if(s == "ImplementationA") return new ImplementationA(); if(s == "ImplementationB") return new ImplementationB(); return nullptr; } EXPORT_C void testlib_InterfaceClass_delete(InterfaceClass* hinst) { delete hinst; } EXPORT_C const char* testlib_InterfaceClass_getID(InterfaceClass* hinst) { return hinst->getID(); } EXPORT_C void testlib_InterfaceClass_setName(InterfaceClass* hinst, const char* name) { hinst->setName(name); } EXPORT_C const char* testlib_InterfaceClass_getName(InterfaceClass* hinst){ return hinst->getName(); }
1.3.8 DLL startup function DLLMain
When a Windows shared library is loaded at compile-time or at runtime by some process, the function DLLMain is always invoked, it is similar to the function main() from an executable.
Note: logging statements with printf, std::cout, std::cerr will not printed if the DLL is loaded by some non-console program, a program compiled to the window subsystem. Therefore, the easiest way to log is to use the OutputDebugString function which output can viewed by the DebugView sysinternal tool.
- File: testlib.cpp => DLL main function.
/** - DLL Entry point - main function of DLL which is executed when the DLL is loaded by some process. */ #if defined(_WIN32) extern "C" __declspec(dllexport) BOOL APIENTRY DllMain (HINSTANCE hInst, DWORD reason, LPVOID lpReserved) { std::string text = std::string("DLL Loaded into the process => PID = ") + std::to_string(::GetCurrentProcessId()); WindbgTrace(text); DbgTrace(text); switch (reason) { case DLL_PROCESS_ATTACH: WindbgTrace("DLL attached to process"); DbgTrace("DLL attached to process."); break; case DLL_PROCESS_DETACH: WindbgTrace("=> DLL detached."); DbgTrace("=> DLL attached"); break; case DLL_THREAD_ATTACH: WindbgTrace("DLL attached to thread"); DbgTrace("DLL detached to thread."); break; case DLL_THREAD_DETACH: WindbgTrace("DLL detached from thread"); DbgTrace("DLL detached from thread."); break; } return TRUE; } #endif
Another more portable way to peform some way action when the DLL is loaded by a process or unloaded is to use a static object. The startup task is executed int the object's constructor and the finalization task, which happens when the process ends, is executed at object's destructor. This approach may a better replacement for DLLMain function, which is specific for Windows, as this technique works both on Windows and any other Unix-like operating system.
- File: testlib.cpp => Static Object for replacing DLLMain
The class _StaticObject and a static instance of it is defined within an anonymous namespace to make them private to the compilation unit testlib.cpp and not allow them to be used from anywhere else.
// Class private to this compilation unit - cannot be accessed from // any other file namespace { class _StaticObject{ public: using Action = std::function<void ()>; Action m_end; _StaticObject(Action init, Action end) : m_end(end) { init(); } ~_StaticObject(){ m_end(); } }; // Static object for replacing DLLMain auto initDLL = _StaticObject( []{ std::cout << " [StaticObject] => Initialize DLL" << std::endl; }, []{ std::cout << " [StaticObject] => Shutdown DLL" << std::endl; }); }
1.3.9 DLL entry point for run32dll.exe
This function is a DLL entry point for the Windows program rundll32.exe which can execute this function with:
$ rundll32.exe testlib.dll,entryPoint1 $ rundll32.exe testlib.dll,entryPoint1 arg0 arg1 arg2 ... argn
When the DLL testlib.dll is run with run32dll calling the function entryPoint1, it will display a message box.
Note: It is an optional function which could be used for installing the DLL in the system or creating some entry in Windows' registry.
- File: testlib.cpp
#if defined(_WIN32) extern "C" __declspec(dllexport) void entryPoint1(HWND hwn, HINSTANCE hinst, LPSTR cmdLine, int nCmdShow){ DbgDisp(cmdLine); OutputDebugString("Rudll32 called entryPoint1()"); MessageBoxA(NULL, "DLL ENTRY POINT", "Entry point 1", 0); } #endif
1.3.10 C++ Client program - client1.cpp
File client1.cpp
Includes:
- Note: (#include "testlib.hpp") a header file between quotes, indicates to the compiler that the header is in the same directory as the source file client1.cpp.
#include <iostream> #include <ostream> #include <vector> #include <string> #include "testlib.hpp"
Declaration of functions with C-linkages not declarated in the header testlib.hpp
extern "C" hVectorD testlib_vectorD_make0(size_t n, double); extern "C" hVectorD testlib_vectorD_make1(size_t n, double array []); extern "C" void testlib_vectorD_delete(hVectorD hv);
Main function - Experiment 1 and Experiment 2:
#ifndef DISABLE std::cout << "\n=== EXPERIMENT 1 ===> Import C++ functions from DLL" << std::endl; DbgTrace("Main process starts here."); std::vector<double> xs{1.0, 2.0, 3.0, 4.0, 5.0}; std::cout << " => Linalg::norm(xs) " << Linalg::norm(xs) << std::endl; std::cout << "=> xs = "; Linalg::printVector(std::cout, xs); std::cout << std::endl; std::cout << "=== EXPERIMENT 2 ===> Import class from DLL" << std::endl; auto cls = SampleClass("Dummy"); cls.set(100); std::cout << "cls.getName() = " << cls.getName() << std::endl; std::cout << " cls.get() = " << cls.get() << std::endl; #endif // -- eof DISABLE flag
Main Function - Experiment 3
//=========>> Load functions and classes using C-interface ==============// std::cout << "\n== EXPERIMENT 3 ===> Import C-functions from DLL - C-interface" << std::endl; double arr [] = {1, 2, 3, 4, 5}; hVectorD v1 = testlib_vectorD_make1(5, arr); testlib_vectorD_Linalg_printVector("v1", v1); std::cout << "norm(v1) = " << testlib_vectorD_Linalg_norm(v1) << std::endl; testlib_vectorD_delete(v1);
Main Functions - Experiment 4
std::cout << "\n== EXPERIMENT 4 ===> Non-polymorphic class with C-interface " << std::endl; hSampleClass hcls = testlib_SampleClass_make1("[EXPERIMENT4]ClassHandle-OOP-C-API"); std::cout << "[EXPERIMENT 4] hcls.getName() = " << testlib_SampleClass_getName(hcls) << std::endl; testlib_SampleClass_set(hcls, 100); std::cout << "[EXPERIMENT 4] hcls.get() = " << testlib_SampleClass_get(hcls) << std::endl; testlib_SampleClass_set(hcls, 200); std::cout << "[EXPERIMENT 4] hcls.get() = " << testlib_SampleClass_get(hcls) << std::endl; testlib_SampleClass_delete(hcls);
Main Functions - Experiment 5
std::cout << "\n== EXPERIMENT 5 ===> Load polymorphic classes from DLL " << std::endl; InterfaceClass* hinstA = teslib_InterfaceClass_factory("ImplementationA"); InterfaceClass* hinstB = teslib_InterfaceClass_factory("ImplementationB"); std::cout << " => hinstA->getID() = " << hinstA->getID() << std::endl; std::cout << " => hinstA->getID() = " << hinstB->getID() << std::endl; hinstA->setName("ClassA-implA"); hinstB->setName("ClassB-implB"); std::cout << " => hinstA->getName() = " << hinstA->getID() << std::endl; std::cout << " => hinstB->getName() = " << hinstB->getID() << std::endl; // Note: If delete is used directly to delete hinstA and hinstB, // a segmentatin fault will happen whenc compiling with Mingw/GCC testlib_InterfaceClass_delete(hinstA); testlib_InterfaceClass_delete(hinstB); std::cout << " [INFO] After deleting instances" << std::endl; DbgTrace("Program ended OK.");
Compile and run client code with MSVC
Note: Both the DLL and the client code are compiled with MSVC-2017 64 bits target.
$ cl.exe /EHsc client.cpp /nologo /Fe:client1.exe testlib.lib && client1.exe # Output: client.cpp Creating library client1.lib and object client1.exp [StaticObject] => Initialize DLL testlib.cpp:39: <DllMain> DLL Loaded into the process => PID = 1112 testlib.cpp:45: <DllMain> DLL attached to process. === EXPERIMENT 1 ===> Import C++ functions from DLL client.cpp:33: <main> Main process starts here. => Linalg::norm(xs) 7.4162 => xs = [5]( 1, 2, 3, 4, 5, ) === EXPERIMENT 2 ===> Import class from DLL Instance created with name = Dummy Counter set to value = 100 cls.getName() = Dummy cls.get() = 100 == EXPERIMENT 3 ===> Import C-functions from DLL - C-interface v1 = [5]( 1, 2, 3, 4, 5, ) norm(v1) = 7.4162 == EXPERIMENT 4 ===> Non-polymorphic class with C-interface Instance created with name = [EXPERIMENT4]ClassHandle-OOP-C-API [EXPERIMENT 4] hcls.getName() = [EXPERIMENT4]ClassHandle-OOP-C-API Counter set to value = 100 [EXPERIMENT 4] hcls.get() = 100 Counter set to value = 200 [EXPERIMENT 4] hcls.get() = 200 testlib.cpp:159: <SampleClass::~SampleClass> SampleClass => name = [EXPERIMENT4]ClassHandle-OOP-C-API deleted == EXPERIMENT 5 ===> Load polymorphic classes from DLL => hinstA->getID() = ImplementationA => hinstA->getID() = ImplementationB => hinstA->getName() = ImplementationA => hinstB->getName() = ImplementationB [INFO] ImplementationA deleted => name = ClassA-implA ; type = ImplementationA [INFO] ImplementationB deleted => name = ClassB-implB ; type = ImplementationB [INFO] After deleting instances client.cpp:87: <main> Program ended OK. testlib.cpp:159: <SampleClass::~SampleClass> SampleClass => name = Dummy deleted testlib.cpp:39: <DllMain> DLL Loaded into the process => PID = 1112 testlib.cpp:49: <DllMain> => DLL attached [StaticObject] => Shutdown DLL
[FAILURE] ABI - Issue - Compile with Mingw client1.cpp against testlib.dll built with MSVC
The client code is compiled with Mingw and the library was built with MSVC-2017 for 64 bits target.
Build and run client [FAILURE]:
$ g++ client1.cpp -g -o client1-gcc.exe -std=c++14 testlib.dll && client1-gcc.exe C:\Users\archbox\AppData\Local\Temp\cckNg9eZ.o: In function `main': client1.cpp:35: undefined reference to `Linalg::norm(std::vector<double, std::allocator<double> > const&)' client1.cpp:36: undefined reference to `Linalg::printVector(std::ostream&, std::vector<double, std::allocator<double> >&)' client1.cpp:39: undefined reference to `SampleClass::SampleClass(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' client1.cpp:40: undefined reference to `SampleClass::set(int)' client1.cpp:41: undefined reference to `SampleClass::getName[abi:cxx11]() const' client1.cpp:42: undefined reference to `SampleClass::get()' client1.cpp:39: undefined reference to `SampleClass::~SampleClass()' client1.cpp:39: undefined reference to `SampleClass::~SampleClass()' collect2.exe: error: ld returned 1 exit status Compilation exited abnormally with code 1 at Sun Dec 16 17:41:47
The compilation fails because C++ does not have a standard and stable ABI - Application Binary Interface, as a result, it is not possible to use an object-code (here - any type of compiled code) built by a different compiler than the current one. The ABI comprises the name mangling schema or name decoration schema which is compiler-dependent, padding, class memory layout and so on. In addition to ABI issues, the STL - Standard Template Library implementations may not be compatible and also not be the same across different compilers.
The only way to ensure that a DLL can work with all possible compilers is to use a functions with C-linkage (extern "C") with opaque pointers (void*) for passing around classes and wrapped STL containers. Classes can be used by different compilers, only if they are interface classes (classes with only pure virtual functions) containing only C-compatible types in the declaration.
Compile with Mingw client1.cpp against testlib.dll built with MSVC
The compilation works when compiling with the custom flag -DDISABLE. The directive #ifndef DISABLE … #endif removes all usages of the namespace Linalg and the class SampleClass. This flag makes main the function use only functions with C-linkage and the interface class InterfaceClass.
$ g++ client.cpp -g -o client-gcc.exe -std=c++14 testlib.dll -DDISABLE && client-gcc.exe [StaticObject] => Initialize DLL testlib.cpp:39: <DllMain> DLL Loaded into the process => PID = 5724 testlib.cpp:45: <DllMain> DLL attached to process. == EXPERIMENT 3 ===> Import C-functions from DLL - C-interface v1 = [5]( 1, 2, 3, 4, 5, ) norm(v1) = 7.4162 == EXPERIMENT 4 ===> Non-polymorphic class with C-interface Instance created with name = [EXPERIMENT4]ClassHandle-OOP-C-API [EXPERIMENT 4] hcls.getName() = [EXPERIMENT4]ClassHandle-OOP-C-API Counter set to value = 100 [EXPERIMENT 4] hcls.get() = 100 Counter set to value = 200 [EXPERIMENT 4] hcls.get() = 200 testlib.cpp:159: <SampleClass::~SampleClass> SampleClass => name = [EXPERIMENT4]ClassHandle-OOP-C-API deleted == EXPERIMENT 5 ===> Load polymorphic classes from DLL => hinstA->getID() = ImplementationA => hinstA->getID() = ImplementationB => hinstA->getName() = ImplementationA => hinstB->getName() = ImplementationB [INFO] ImplementationA deleted => name = ClassA-implA ; type = ImplementationA [INFO] ImplementationB deleted => name = ClassB-implB ; type = ImplementationB [INFO] After deleting instances client.cpp:87: <main> Program ended OK. testlib.cpp:39: <DllMain> DLL Loaded into the process => PID = 5724 testlib.cpp:49: <DllMain> => DLL attached [StaticObject] => Shutdown DLL
1.3.11 Load DLL in Python3 REPL
Functions with C-linkage can be called by Python using the ctypes library. The following code presents how it can be done:
Table for C types / Python ctypes conversion
C-type | Python C-types | C-type | |
---|---|---|---|
char | ctypes.c_char | char* | ctypes.POINTER(ctypes.c_char) |
int | ctypes.c_int | int* | ctypes.POINTER(ctypes.c_int) |
size_t | ctypes.c_int | - | |
double | ctypes.c_double | double* | ctypes.POINTER(ctypes.c_double) |
void | None | void* | ctypes.c_void_p |
- Get a Python string from a const char* ptr or char* ptr, use ctypes.string_at(ptr).
- Pass a python string to an API with const char* (ctypes.POINTER(ctypes.c_char)) as argument, use Function("string-argument".encode('utf-8'))
Python C-types documentation and further reading:
- ctypes — A foreign function library for Python — Python 3.7.2rc1 documentation
- Extending Python With C Libraries and the “ctypes” Module – dbader.org
- ctypes.CDLL Python Example
- Calling C functions from Python – part 1 – using ctypes – Yi Zhang's MSDN Blog
- Python Programming/Extending with ctypes - Wikibooks, open books for an open world
- Using C from Python: How to create a ctypes wrapper - Scientific IT-Systems
Load the library:
C:\Users\archbox\Desktop\experiments λ "C:\Users\archbox\Miniconda3\pkgs\python-3.6.5-h0c2934d_0\python" Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import ctypes # This messages is print by the DLLMain function >>> lib = ctypes.cdll.LoadLibrary("testlib.dll") [StaticObject] => Initialize DLL testlib.cpp:39: <DllMain> DLL Loaded into the process => PID = 4280 testlib.cpp:45: <DllMain> DLL attached to process. >>> testlib.cpp:39: <DllMain> DLL Loaded into the process => PID = 4280 testlib.cpp:53: <DllMain> DLL detached to thread.
Check whether exported functions with C-linkage exist:
>>> lib.testlib_vectorD_make0 <_FuncPtr object at 0x0000021BFCAFF388> >>> lib.testlib_vectorD_Linalg_printVector <_FuncPtr object at 0x0000021BFCAFF458> >>> >>> lib.testlib_vectorD_delete <_FuncPtr object at 0x0000021BFCAFF528> # Function which does not exist. >>> lib.testlib_InterfaceClass_do_not_exist Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Users\archbox\Miniconda3\pkgs\python-3.6.5-h0c2934d_0\lib\ctypes\__init__.py", line 361, in __getattr__ func = self.__getitem__(name) File "C:\Users\archbox\Miniconda3\pkgs\python-3.6.5-h0c2934d_0\lib\ctypes\__init__.py", line 366, in __getitem__ func = self._FuncPtr((name_or_ordinal, self)) AttributeError: function 'testlib_InterfaceClass_do_not_exist' not found
Load std::vector<double> and Linalg namespace wrapper functions
Set up the std::vector<double> functions to be imported.
- Note: hVectorD => Handle for std::vector<double> is the same as void* or an opaque pointer.
# Funciton: hVectorD testlib_vectorD_make0(size_t n, double x) # Set function arguments lib.testlib_vectorD_make0.argtypes = [ctypes.c_int, ctypes.c_double] # Set return type lib.testlib_vectorD_make0.restype = ctypes.c_void_p # void testlib_vectorD_Linalg_printVector(const char* name, hVectorD hv) lib.testlib_vectorD_Linalg_printVector.argtypes = [ctypes.POINTER(ctypes.c_char), ctypes.c_void_p ] lib.testlib_vectorD_Linalg_printVector.restype = None # Set vector elements hv[n] = x # void testlib_vectorD_set(hVectorD hv, size_t n, double x) lib.testlib_vectorD_set.argtypes = [ctypes.c_void_p, ctypes.c_int, ctypes.c_double] lib.testlib_vectorD_set.restype = None
Testing C-interface functions for std::vector<double>:
# Creating a std::vector<double> with 4 elements equal to 3.0 >>> vec1 = lib.testlib_vectorD_make0(4, 3.0) # This number is the memory address held by the opaque pointer, # it cannot be used directly. >>> vec1 2319184200912 >>> # ========== Modify vector elements ======== # # Printarray2 the wrapped std::vector<double> >>> lib.testlib_vectorD_Linalg_printVector("vec1".encode('utf-8'), vec1) vec1 = [4]( 3, 3, 3, 3, ) >>> >>> lib.testlib_vectorD_set(vec1, 0, 5.0) >>> lib.testlib_vectorD_set(vec1, 1, 10.5) >>> lib.testlib_vectorD_set(vec1, 2, 4.78) >>> lib.testlib_vectorD_Linalg_printVector("vec1".encode('utf-8'), vec1) vec1 = [4]( 5, 10.5, 4.78, 3, )
Compute vector Euclidian norm:
# Wrapper for function # double Linalg::norm(const std::vector<double>& xs) # double testlib_vectorD_Linalg_norm(hVectorD hv) lib.testlib_vectorD_Linalg_norm.argtypes = [ ctypes.c_void_p ] lib.testlib_vectorD_Linalg_norm.restype = ctypes.c_double >>> lib.testlib_vectorD_Linalg_printVector("vec1".encode('utf-8'), vec1) vec1 = [4]( 5, 10.5, 4.78, 3, ) >>> lib.testlib_vectorD_Linalg_norm(vec1) 12.926654632966722 # Check if calculations are right >>> math.sqrt(5 * 5 + 10.5 * 10.5 + 4.78 * 4.78 + 3 * 3) 12.926654632966722 >>>
Delete vector vec1 using its destructor function:
# Function: void testlib_vectorD_delete(hVectorD hv) lib.testlib_vectorD_delete.argtypes = [ ctypes.c_void_p ] lib.testlib_vectorD_delete.restype = None >>> lib.testlib_vectorD_delete(vec1)
Load wrapper function for interface class InterfaceClass
Load factory function:
- void* teslib_InterfaceClass_factory(const char* class_id)
# InterfaceClass* teslib_InterfaceClass_factory(const char* class_id) lib.teslib_InterfaceClass_factory.argtypes = [ ctypes.POINTER(ctypes.c_char) ] lib.teslib_InterfaceClass_factory.restype = ctypes.c_void_p # const char* testlib_InterfaceClass_getID(InterfaceClass* hinst) lib.testlib_InterfaceClass_getID.argtypes = [ ctypes.c_void_p ] lib.testlib_InterfaceClass_getID.restype = ctypes.POINTER(ctypes.c_char)
Creating class instances and testing member function C-wrappers:
>>> hinstA = lib.teslib_InterfaceClass_factory("ImplementationA".encode('utf-8')) >>> hinstA 2319184196864 >>> >>> hinstB = lib.teslib_InterfaceClass_factory("ImplementationB".encode('utf-8')) >>> hinstB 2319184198016 >>> s1 = lib.testlib_InterfaceClass_getID(hinstA) >>> s1 <ctypes.LP_c_char object at 0x0000021BFBBF14C8> >>> ctypes.string_at(s1) b'ImplementationA' >>> >>> s2 = lib.testlib_InterfaceClass_getID(hinstB) >>> ctypes.string_at(s2) b'ImplementationB' >>> ctypes.string_at(lib.testlib_InterfaceClass_getID(hinstB)) b'ImplementationB' >>>
Load more C-wrappers for member function of class InterfaceClass:
# const char* testlib_InterfaceClass_getName(InterfaceClass* hinst) lib.testlib_InterfaceClass_getName.argtypes = [ ctypes.c_void_p ] lib.testlib_InterfaceClass_getName.restype = ctypes.POINTER(ctypes.c_char) # void testlib_InterfaceClass_setName(InterfaceClass* hinst, const char* name) >>> lib.testlib_InterfaceClass_setName.argtypes = [ ctypes.c_void_p, ctypes.POINTER(ctypes.c_char) ] >>> lib.testlib_InterfaceClass_setName.restype = None >>> lib.testlib_InterfaceClass_getName(hinstA) <ctypes.LP_c_char object at 0x0000021BFBBF1548> >>> ctypes.string_at(lib.testlib_InterfaceClass_getName(hinstA)) b'Unammed-A' >>> ctypes.string_at(lib.testlib_InterfaceClass_getName(hinstB)) b'Unammed-B' >>> lib.testlib_InterfaceClass_setName(hinstA, "Instance-ClassA1".encode('utf-8')) >>> ctypes.string_at(lib.testlib_InterfaceClass_getName(hinstA)) b'Instance-ClassA1' >>> lib.testlib_InterfaceClass_setName(hinstB, "Instance-B1".encode('utf-8')) >>> ctypes.string_at(lib.testlib_InterfaceClass_getName(hinstB)) b'Instance-B1'
Load C-functions wrappers for the InterfaceClass destructor and dispose both instances hinstA and hinstB:
# Destructor function: # void testlib_InterfaceClass_delete(InterfaceClass* hinst) >>> lib.testlib_InterfaceClass_delete.argtypes = [ ctypes.c_void_p ] >>> lib.testlib_InterfaceClass_delete.restype = None # Dispose class hinstA, delete this object >>> lib.testlib_InterfaceClass_delete(hinstA) [INFO] ImplementationA deleted => name = Instance-ClassA1 ; type = ImplementationA >>> lib.testlib_InterfaceClass_delete(hinstB) [INFO] ImplementationB deleted => name = Instance-B1 ; type = ImplementationB
Exit python3 REPL:
>>> exit() testlib.cpp:39: <DllMain> DLL Loaded into the process => PID = 4280 testlib.cpp:49: <DllMain> => DLL attached [StaticObject] => Shutdown DLL
1.3.12 Python 3 Client Code - Wrapper Module
All the boilerplate code required to load the shared library testlib.dll can be eliminated by crafting a python module file and wrapper classes.
Source:
- File: file:src/dlls/example-windows1/pywrapper.py
- Gist: Gisthub
Module Initialization:
- File: pywrapper.py
The function _config initializes the wrapper module, loading the DLL - share library testlib.dll into the current Python3 interpreter process and setting up the C-functions (functions exported with C-linkage, without name mangling) exported by the library.
import ctypes def _getSharedLibrary(libname): import sys import os libfile = libname if sys.platform == "linux" or sys.platform == "linux2": libfile = "lib" + libname + ".so" elif sys.platform == "darwin": libfile = libname + ".dylyb" elif sys.platform == "win32": libfile = libname + ".dll" libpath = os.path.join(os.path.dirname(__file__), libfile) print(" [INFO] libpath = " + libpath) return libpath # _lib = ctypes.cdll.LoadLibrary("testlib") _lib = ctypes.cdll.LoadLibrary(_getSharedLibrary("testlib")) # Startup ctypes FFI - Foreign Function Interface def _config(): print("Intializing library") # ======= std::vector<double> and Linalg:: namespace ==========## # hVectorD testlib_vectorD_make0(size_t n, double x) _lib.testlib_vectorD_make0.argtypes = [ctypes.c_int, ctypes.c_double] _lib.testlib_vectorD_make0.restype = ctypes.c_void_p # hVectorD testlib_vectorD_make1(size_t n, double array []) _lib.testlib_vectorD_make1.argtypes = [ctypes.c_int, ctypes.POINTER(ctypes.c_double)] _lib.testlib_vectorD_make1.restype = ctypes.c_void_p _config()
The class VectorD is a wrapper class for the STL class std::vector<double> and the namespace Linalg containing linear algebra functions.
class VectorD: def __init__(self, handle): self.hnd = ctypes.c_void_p(handle) self.name = "std::vector<double> vx" @classmethod def fromValue(cls, size, x): return VectorD(_lib.testlib_vectorD_make0(size, x)) @classmethod def fromArray(cls, array): carray_size_n = ctypes.c_double * len(array) return VectorD(_lib.testlib_vectorD_make1(len(array), carray_size_n(*array))) # Destructor def __del__(self): print(" [TRACE] - Vector disposed - C++ Destructor invoked Ok.") _lib.testlib_vectorD_delete(self.hnd) def setName(self, name): self.name = name # Display vector def disp(self): _lib.testlib_vectorD_Linalg_printVector(self.name.encode('utf-8'), self.hnd) # Set element at nth position def set(self, idx, x): _lib.testlib_vectorD_set(self.hnd, idx, x) def norm(self): return _lib.testlib_vectorD_Linalg_norm(self.hnd)
The class CPPInterfaceClass is a wrapper for the C++ interface class InterfaceClass exported by the DLL. The python wrapper class contains factory methods .factory, .makeA for loading the implementation class ImplementationA and the method .makeB for loading the implementation ImplementationB.
# Proxy for C++ Interface class in the shared library class CPPInterfaceClass: # Constructor def __init__(self, handle): self.hnd = ctypes.c_void_p(handle) # Destructor def __del__(self): # Call C++ destructor # print(" [__del__] => self.hnd = " + str(self.hnd)) _lib.testlib_InterfaceClass_delete(self.hnd) @classmethod def factory(cls, classID): return CPPInterfaceClass(_lib.teslib_InterfaceClass_factory(classID.encode('utf-8'))) @classmethod def makeA(cls): "Instantiate the class ImplementationA from the DLL." return CPPInterfaceClass.factory("ImplementationA") @classmethod def makeB(cls): "Instantiate the class ImplementationB from the DLL." return CPPInterfaceClass.factory("ImplementationB") def getType(self): return ctypes.string_at(_lib.testlib_InterfaceClass_getID(self.hnd)).decode('utf-8') def getName(self): return ctypes.string_at(_lib.testlib_InterfaceClass_getName(self.hnd)).decode('utf-8') def setName(self, name): _lib.testlib_InterfaceClass_setName(self.hnd, name.encode('utf-8')) # String representation def __str__(self): s = "CInterfaceClass ; type = " + self.getType() s += " - name = " + self.getName() + "\n" return s # Make class printable in the REPL def __repr__(self): return self.__str__()
Function test1() runs an example code:
def test1(): print("\n ======== Test 1 - std::vector<double> wrapper and Linalg module ======") v1 = VectorD.fromValue(4, 3.5) print(" [*]=> Before changing std::vector<double> object") v1.disp() print("v1.norm() = " + str(v1.norm())) print() print(" [*]=> After changing std::vector<double> object") v1.set(0, 5); v1.set(1, 2.6); v1.set(2, 9.81); v1.set(3, 3.76) v1.disp() print("v1.norm() " + str(v1.norm())) print() print("\n ======== Interface class 'InterfaceClass' ======") clsA = CPPInterfaceClass.makeA() print("clsA = " + str(clsA)) print("clsA.getType() = " + clsA.getType()) clsB = CPPInterfaceClass.makeB() print("clsB = " + str(clsB)) print("clsB.getType() = " + clsB.getType()) if __name__ == "__main__": test1()
Usage in Python3 REPL:
This step assumes that the shared library file (testlib.dll on Windows or libtestlib.so on Unix-like OS or Linux) is in the same directory as the python script pywrapper.py.
- Import module
$ C:/Users/archbox/Miniconda3/pkgs/python-3.6.5-h0c2934d_0/python Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> >>> from testlib import VectorD [StaticObject] => Initialize DLL testlib.cpp:39: <DllMain> DLL Loaded into the process => PID = 5776 testlib.cpp:45: <DllMain> DLL attached to process. Intializing library Library initialized OK. >>> from testlib import CPPInterfaceClass
- Testing class VectorD - wrapper for std::vector<double>
>>> v1 = VectorD.fromValue(5, 3.0) >>> v1.disp() std::vector<double> vx = [5]( 3, 3, 3, 3, 3, ) >>> v1.setName("v1") >>> v1.disp() v1 = [5]( 3, 3, 3, 3, 3, ) >>> v1.norm() 6.708203932499369 >>> v1.set(0, 5) >>> v1.set(1, 4.5) >>> v1.set(2, 9.8) >>> v1.set(3, 18.4) >>> v1.disp() v1 = [5]( 5, 4.5, 9.8, 18.4, 3, ) >>> >>> v1.norm() 22.10995251012539 >>> v1 = 100 [TRACE] - Vector disposed - C++ Destructor invoked Ok. >>> v2 = VectorD.fromArray([4.0, 10.25, 9.6, 3, 10, 6, 15]) >>> v2.disp() std::vector<doubl> vx = [7]( 4, 10.25, 9.6, 3, 10, 6, 15, ) >>> v2.norm() 24.15 >>> v2.set(0, 100) >>> v2.set(1, 200) >>> v2.disp() std::vector<double> vx = [7]( 100, 200, 9.6, 3, 10, 6, 15, ) >>> v2.norm() 224.63784186997523 >>>
- Testing class CPPInterfaceClass, proxy for C++ classes InterfaceClass, ImplementationA and ImplementationB.
>>> from testlib import CPPInterfaceClass # ======== Load ImplementationA ============== # # >>> a = CPPInterfaceClass.makeA() >>> a CInterfaceClass ; type = ImplementationA - name = Unammed-A >>> a.getType() 'ImplementationA' >>> a.getName() 'Unammed-A' >>> a.setName("Instance-of-ImplA") >>> a CInterfaceClass ; type = ImplementationA - name = Instance-of-ImplA # ======== Load ImplementationB ============== # # >>> b = CPPInterfaceClass.makeB() >>> b CInterfaceClass ; type = ImplementationB - name = Unammed-B >>> b.setName("Instance-of-class-B") >>> b.getName() 'Instance-of-class-B' >>> >>> b.getType() 'ImplementationB'
- Exit REPL.
>>> exit() [TRACE] - Vector disposed - C++ Destructor invoked Ok. [INFO] ImplementationB deleted => name = Instance-of-class-B ; type = ImplementationB testlib.cpp:39: <DllMain> DLL Loaded into the process => PID = 5776 testlib.cpp:49: <DllMain> => DLL attached [StaticObject] => Shutdown DLL
1.3.13 CSharp Client Code - PInvoke
The services provided by the DLL can be consumed from C# or any other .NET language via Pinvoke-API which is the Foreign-Function Interface of .NET platform. The Pinvoke API can import C-function or C++ demangled functions exported with extern "C".
C++ classes, namespaces and functions without C-linkage (extern "C") cannot be loaded directly by P-Invoke or any other Foreign Function-Interface as C++ does not define a standard and ABI - Application Binary Interface, every compiler has its own name mangling schema, padding bytes and class memory layout for non-POD (Plain-Old data) types. A workaround to call C++ from an FFI is to defined C-interface comprised of functions with C-linkage and opaque pointers (void*) for passing class instances around or invoking C++ functions.
File: client-sharp.cs
Compiling and running:
$ csc.exe client-csharp.cs && client-csharp.exe
Wrapper for class std::vector<double> and namespace Linalg.
class VectorD{ // Constructor Function: // hVectorD testlib_vectorD_make0(size_t n, double x) // void* testlib_vectorD_make0(size_t n, double x) [DllImport("testlib.dll")] public static extern IntPtr testlib_vectorD_make0(int size, double x); [DllImport("testlib.dll")] public static extern IntPtr testlib_vectorD_make1(int n, double[] x); // Destructor function: // void testlib_vectorD_delete(hVectorD hv) // void testlib_vectorD_delete(void* hv) [DllImport("testlib.dll")] public static extern IntPtr testlib_vectorD_delete(IntPtr hv); // void testlib_vectorD_Linalg_printVector(const char* name, hVectorD hv) // void testlib_vectorD_Linalg_printVector(const char* name, vector* hv) [DllImport("testlib.dll")] public static extern void testlib_vectorD_Linalg_printVector(string name, IntPtr hv); // double testlib_vectorD_Linalg_norm(hVectorD hv) // double testlib_vectorD_Linalg_norm(void* hv) [DllImport("testlib.dll")] public static extern double testlib_vectorD_Linalg_norm(IntPtr hv); // void testlib_vectorD_set(hVectorD hv, size_t n, double x) // void testlib_vectorD_set(void* hv, size_t n, double x) [DllImport("testlib.dll")] public static extern void testlib_vectorD_set(IntPtr hv, int n, double x); // ------------ Objet Oriented Wrapper --------// // Hadle private IntPtr m_handle; public VectorD(int size, double x){ m_handle = testlib_vectorD_make0(size, x); } public VectorD(double[] array){ m_handle = testlib_vectorD_make1(array.Length, array); } // Finalized destructor ~VectorD(){ testlib_vectorD_delete(m_handle); } public void print(string name = "std::vector<double>"){ testlib_vectorD_Linalg_printVector(name, m_handle); } public void set(int n, double x){ testlib_vectorD_set(m_handle, n, x); } public double norm(){ return testlib_vectorD_Linalg_norm(m_handle); } }
Wrapper for C++ interface class 'InterfaceClass' and its implementations, class ImplementationA and ImplementationB:
class CPPInterfaceClass{ // Handle type => Opaque pointer for InterfaceClass instances. // using HInterf = IntPtr; // Factory function for loading functions from this interface. //----------------------------------------------------------- // InterfaceClass* teslib_InterfaceClass_factory(const char* class_id) // void** teslib_InterfaceClass_factory(const char* class_id) [DllImport("testlib.dll")] private static extern IntPtr teslib_InterfaceClass_factory(string class_id); // void testlib_InterfaceClass_delete(InterfaceClass* hinst) // void testlib_InterfaceClass_delete(void*) [DllImport("testlib.dll")] private static extern void testlib_InterfaceClass_delete(IntPtr hinst); // char* testlib_InterfaceClass_getID(InterfaceClass* hinst) [DllImport("testlib.dll")] private static extern IntPtr testlib_InterfaceClass_getID(IntPtr hinst); // void testlib_InterfaceClass_setName(InterfaceClass* hinst, const char* name) // Note: Charset.UNICODE does not work as it is UTF16 (Wide Unicode), not UTF-8 // which is the C++ default unicode type. [DllImport("testlib.dll", CharSet = CharSet.Ansi)] private static extern void testlib_InterfaceClass_setName(IntPtr hinst, string name); // char* testlib_InterfaceClass_getName(InterfaceClass* hinst) [DllImport("testlib.dll")] private static extern IntPtr testlib_InterfaceClass_getName(IntPtr hinst); private IntPtr m_handle; private CPPInterfaceClass(IntPtr handle){ m_handle = handle; } ~CPPInterfaceClass(){ testlib_InterfaceClass_delete(m_handle); } /** Creates an instance of the the C++ implementation ImplementationA (class) from the * C++ interface: 'InterfaceClass' */ public static CPPInterfaceClass ImplementationA(){ return new CPPInterfaceClass(teslib_InterfaceClass_factory("ImplementationA")); } /** Creates an instance of the C++ implementationa ImplementationB (class) */ public static CPPInterfaceClass ImplementationB(){ return new CPPInterfaceClass(teslib_InterfaceClass_factory("ImplementationB")); } // Get type of the wrapped C++ class public string GetID(){ IntPtr p = testlib_InterfaceClass_getID(m_handle); return Marshal.PtrToStringAnsi(p); } // Get/Set name of current instance public string Name { get{ IntPtr p = testlib_InterfaceClass_getName(m_handle); return Marshal.PtrToStringAnsi(p); } set{ testlib_InterfaceClass_setName(m_handle, value); } } }
- Main function - Experiment 1:
Console.WriteLine(" ===== EXPERIMENT 1 = Using C-functions directly =="); Console.WriteLine(" [CSharp] Loading Native DLL C++ Shared Library"); IntPtr vectorObject = VectorD.testlib_vectorD_make0(5, 3.0); Console.WriteLine(" [CSharp] Opaque pointer = ", vectorObject.ToString()); VectorD.testlib_vectorD_Linalg_printVector("vectorX", vectorObject); double x = VectorD.testlib_vectorD_Linalg_norm(vectorObject); Console.WriteLine(" [CSharp] Vector norm = " + x.ToString()); Console.WriteLine(" [CSharp] End application"); VectorD.testlib_vectorD_delete(vectorObject);
Output:
===== EXPERIMENT 1 = Using C-functions directly == [CSharp] Loading Native DLL C++ Shared Library [StaticObject] => Initialize DLL testlib.cpp:48: <DllMain> DLL Loaded into the process => PID = 1320 testlib.cpp:54: <DllMain> DLL attached to process. [CSharp] Opaque pointer = vectorX = [5]( 3, 3, 3, 3, 3, ) [CSharp] Vector norm = 6.70820393249937 [CSharp] End application
- Main function - Experiment 2:
Console.WriteLine(" ===== EXPERIMENT 2 = Using OOP wrapper for std::vector ==="); Console.WriteLine(" ==> Before changing vector"); VectorD v1 = new VectorD(10, 3.5); v1.print(); v1.print("vector_v1"); Console.WriteLine("v1.norm() = " + v1.norm().ToString()); Console.WriteLine(" ==> After changing vector"); v1.set(1, 10.0); v1.set(2, 5.53); v1.set(3, 8.96); v1.set(4, -10.34); v1.set(8, 80.54); v1.print("v1_changed"); Console.WriteLine(" ==> Creating Vector from Array"); VectorD v2 = new VectorD(new double[] {4.5, -8.84, 78.23, 652.3, 34.56, 45.12}); v2.print("v2"); Console.WriteLine("v2.norm() = " + v2.norm().ToString());
Output:
===== EXPERIMENT 2 = Using OOP wrapper for std::vector === ==> Before changing vector std::vector<double> = [10]( 3.5, 3.5, 3.5, 3.5, 3.5, 3.5, 3.5, 3.5, 3.5, 3.5, ) vector_v1 = [10]( 3.5, 3.5, 3.5, 3.5, 3.5, 3.5, 3.5, 3.5, 3.5, 3.5, ) v1.norm() = 11.0679718105893 ==> After changing vector v1_changed = [10]( 3.5, 10, 5.53, 8.96, -10.34, 3.5, 3.5, 3.5, 80.54, 3.5, ) ==> Creating Vector from Array v2 = [6]( 4.5, -8.84, 78.23, 652.3, 34.56, 45.12, ) v2.norm() = 659.502711518307
- Main function - Experiment 3:
Console.WriteLine(" ===== EXPERIMENT 3 = OOP wrapper for InterfaceClass ==="); CPPInterfaceClass instA = CPPInterfaceClass.ImplementationA(); CPPInterfaceClass instB = CPPInterfaceClass.ImplementationB(); // Console.WriteLine("instA.Name = " + instA.Name); Console.WriteLine("instA.GetID() = " + instA.GetID()); Console.WriteLine("instB.GetID() = " + instB.GetID()); Console.WriteLine(" **=> Before changing"); Console.WriteLine("instA.Name = " + instA.Name); Console.WriteLine("instB.Name = " + instB.Name); Console.WriteLine("\n **=> Before changing"); instA.Name = "Instance-of-ImplA"; instB.Name = "Instance-of-ImplB"; Console.WriteLine("instA.Name = " + instA.Name); Console.WriteLine("instB.Name = " + instB.Name); Console.WriteLine(" ===== END ===========================");
Output:
===== EXPERIMENT 3 = OOP wrapper for InterfaceClass === instA.GetID() = ImplementationA instB.GetID() = ImplementationB **=> Before changing instA.Name = Unammed-A instB.Name = Unammed-B **=> Before changing instA.Name = Instance-of-ImplA instB.Name = Instance-of-ImplB ===== END =========================== testlib.cpp:48: <DllMain> DLL Loaded into the process => PID = 1320 testlib.cpp:62: <DllMain> DLL detached to thread. [INFO] ImplementationB deleted => name = Instance-of-ImplB ; type = ImplementationB [INFO] ImplementationA deleted => name = Instance-of-ImplA ; type = ImplementationA testlib.cpp:48: <DllMain> DLL Loaded into the process => PID = 1320 testlib.cpp:58: <DllMain> => DLL attached [StaticObject] => Shutdown DLL
1.3.14 View symbols exported by the shared library
- View exported symbols on Windows (dumpbin)
Location of dumpbin tool
The symbols exported by the DLL can viewed with the dumpbin tool available from the MSVC developer console.
$ where dumpbin.exe C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.12.25827\bin\Hostx64\x64\dumpbin.exe
Show DLL exported symbols
$ dumpbin.exe /exports testlib.dll
Output:
Microsoft (R) COFF/PE Dumper Version 14.12.25835.0 Copyright (C) Microsoft Corporation. All rights reserved. Dump of file testlib.dll File Type: DLL Section contains the following exports for testlib.dll 00000000 characteristics FFFFFFFF time date stamp 0.00 version 1 ordinal base 31 number of functions 31 number of names ordinal hint RVA name 1 0 00005C50 ??0SampleClass@@QEAA@AEBV0@@Z 2 1 00001650 ??0SampleClass@@QEAA@AEBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@Z 3 2 000015E0 ??0SampleClass@@QEAA@XZ 4 3 000016E0 ??1SampleClass@@QEAA@XZ 5 4 00007280 ??4SampleClass@@QEAAAEAV0@AEBV0@@Z 6 5 00001880 ?get@SampleClass@@QEAAHXZ 7 6 00001830 ?getName@SampleClass@@QEBA?AV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@XZ 8 7 00001370 ?linTransform@Linalg@@YA?AV?$vector@NV?$allocator@N@std@@@std@@NNAEAV23@@Z 9 8 000012D0 ?norm@Linalg@@YANAEBV?$vector@NV?$allocator@N@std@@@std@@@Z 10 9 00001480 ?printVector@Linalg@@YAAEAV?$basic_ostream@DU?$char_traits@D@std@@@std@@AEAV23@AEAV?$vector@NV?$allocator@N@std@@@3@@Z 11 A 000018A0 ?set@SampleClass@@QEAAXH@Z 12 B 00001D50 DllMain 13 C 00002720 entryPoint1 14 D 00002800 main 15 E 00001B10 teslib_InterfaceClass_factory 16 F 00001C60 testlib_InterfaceClass_delete 17 10 00001CC0 testlib_InterfaceClass_getID 18 11 00001D20 testlib_InterfaceClass_getName 19 12 00001CF0 testlib_InterfaceClass_setName 20 13 00001A10 testlib_SampleClass_delete 21 14 00001A60 testlib_SampleClass_get 22 15 00001AB0 testlib_SampleClass_getName 23 16 00001900 testlib_SampleClass_make0 24 17 00001950 testlib_SampleClass_make1 25 18 00001A80 testlib_SampleClass_set 26 19 00001560 testlib_vectorD_Linalg_norm 27 1A 00001580 testlib_vectorD_Linalg_printVector 28 1B 000026D0 testlib_vectorD_delete 29 1C 00002590 testlib_vectorD_make0 30 1D 00002610 testlib_vectorD_make1 31 1E 00002690 testlib_vectorD_set Summary 3000 .data 4000 .pdata 18000 .rdata 1000 .reloc 2F000 .text
Note that:
- Functions with C-linkage are not mangled, their symbols match their names, for instance:
15 E 00001B10 teslib_InterfaceClass_factory 16 F 00001C60 testlib_InterfaceClass_delete 17 10 00001CC0 testlib_InterfaceClass_getID 18 11 00001D20 testlib_InterfaceClass_getName 19 12 00001CF0 testlib_InterfaceClass_setName 20 13 00001A10 testlib_SampleClass_delete 21 14 00001A60 testlib_SampleClass_get 22 15 00001AB0 testlib_SampleClass_getName
- Functions, member functions, namespaces and everything without C-linkage are mangled, the compiler generate an unique name for the generated symbols:
1 0 00005C50 ??0SampleClass@@QEAA@AEBV0@@Z 2 1 00001650 ??0SampleClass@@QEAA@AEBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@Z 3 2 000015E0 ??0SampleClass@@QEAA@XZ 4 3 000016E0 ??1SampleClass@@QEAA@XZ 5 4 00007280 ??4SampleClass@@QEAAAEAV0@AEBV0@@Z 6 5 00001880 ?get@SampleClass@@QEAAHXZ 7 6 00001830 ?getName@SampleClass@@QEBA?AV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@XZ 8 7 00001370 ?linTransform@Linalg@@YA?AV?$vector@NV?$allocator@N@std@@@std@@NNAEAV23@@Z 9 8 000012D0 ?norm@Linalg@@YANAEBV?$vector@NV?$allocator@N@std@@@std@@@Z 10 9 00001480 ?printVector@Linalg@@YAAEAV?$basic_ostream@DU?$char_traits@D@std@@@std@@AEAV23@AEAV?$vector@NV?$allocator@N@std@@@3@@Z 11 A 000018A0 ?set@SampleClass@@QEAAXH@Z
It is possible to demangle those symbols with the tool - https://demangler.com/
- Symbol: ??0SampleClass@@QEAA@AEBV0@@Z demangled:
public: __cdecl SampleClass::SampleClass(class SampleClass const & __ptr64) __ptr64
- Symbol: ?get@SampleClass@@QEAAHXZ demangled:
public: int __cdecl SampleClass::get(void) __ptr64
- Symbol: ?linTransform@Linalg@@YA?AV?$vector@NV?$allocator@N@std@@@std@@NNAEAV23@@Z demangled:
class std::vector<double,class std::allocator<double> > __cdecl Linalg::linTransform(double,double,class std::vector<double,class std::allocator<double> > & __ptr64)
- View exports symbols on Unix-like OSes (nm)
View exported symbols with nm:
- Note that only functions annotated with extern "C" such as teslib_InterfaceClass_factory are not mangled.
$ nm -D libtestlib.so 000000000020a3d8 B __bss_start U __cxa_atexit w __cxa_finalize U __cxa_pure_virtual 000000000020a3d8 D _edata 000000000020a440 B _end 000000000000710c T _fini w __gmon_start__ ... ... ... ... ... ... ... ... ... ... 0000000000005512 T teslib_InterfaceClass_factory 0000000000005629 T testlib_InterfaceClass_delete 0000000000005656 T testlib_InterfaceClass_getID 00000000000056a8 T testlib_InterfaceClass_getName 0000000000005677 T testlib_InterfaceClass_setName 000000000000545c T testlib_SampleClass_delete 000000000000548f T testlib_SampleClass_get 00000000000054cc T testlib_SampleClass_getName ... ... ... ... ... ... ... ... 0000000000005076 T _ZN11SampleClassC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE 0000000000005106 T _ZN11SampleClassC2Ev 0000000000005196 T _ZN11SampleClassD1Ev
View exported symbols demangled (without name mangling):
$ nm -CD libtestlib.so 000000000020a3d8 B __bss_start U __cxa_atexit w __cxa_finalize ... ... ... ... ... ... ... ... ... ... ... ... 0000000000004fe7 T testlib_vectorD_Linalg_norm 000000000000500f T testlib_vectorD_Linalg_printVector 0000000000004e52 T testlib_vectorD_make0 0000000000004ee2 T testlib_vectorD_make1 0000000000004f80 T testlib_vectorD_set U _Unwind_Resume U operator delete(void*) U operator delete(void*, unsigned long) 00000000000052fe T SampleClass::get() 0000000000005310 T SampleClass::set(int) ... ... ... ... ... ... ... 00000000000070ac W double* std::__copy_move<false, true, std::random_access_iterator_tag>::__copy_m<double>(double const*, double const*, double*) 0000000000006cfe W void std::_Destroy_aux<true>::__destroy<double*>(double*, double*) 0000000000006c74 W std::_Vector_base<double, std::allocator<double> >::_M_allocate(unsigned long) 0000000000006a44 W std::_Vector_base<double, std::allocator<double> >::_Vector_impl::_Vector_impl(std::allocator<double> const&) 0000000000006a44 W std::_Vector_base<double, std::allocator<double> >::_Vector_impl::_Vector_impl(std::allocator<double> const&) 000000000000681a W std::_Vector_base<double, std::allocator<double> >::_Vector_impl::~_Vector_impl() 000000000000681a W std::_Vector_base<double, std::allocator<double> >::_Vector_impl::~_Vector_impl() 0000000000006ae8 W std::_Vector_base<double, std::allocator<double> >::_M_deallocate(double*, unsigned long) 0000000000006a8e W std::_Vector_base<double, std::allocator<double> >::_M_create_storage(unsigned long) 0000000000006926 W std::_Vector_base<double, std::allocator<double> >::_M_get_Tp_allocator() 0000000000006836 W std::_Vector_base<double, std::allocator<double> >::_Vector_base(unsigned long, std::allocator<double> const&) 00000000000069d2 W std::_Vector_base<double, std::allocator<double> >::_Vector_base(std::allocator<double> const&) 0000000000006836 W std::_Vector_base<double, std::allocator<double> >::_Vector_base(unsigned long, std::allocator<double> const&) 00000000000069d2 W std::_Vector_base<double, std::allocator<double> >::_Vector_base(std::allocator<double> const&) ... ... ... ... ... 0000000000006706 W std::function<void ()>::function(std::function<void ()> const&) 0000000000006706 W std::function<void ()>::function(std::function<void ()> const&) 00000000000062b2 W std::function<void ()>::~function() 00000000000062b2 W std::function<void ()>::~function() U std::ios_base::Init::Init() U std::ios_base::Init::~Init()
1.4 Plugins and Dynamically Loading Classes
1.4.1 Overview
A Plugin system allows loading software components, classes and functions at runtime without from shared libraries without needing to compile or rebuild the main application.
A plugin architecture provides the following benefits:
- Dynamically load class at runtime reducing compilation time.
- Allow third-parties to create new plugins, addons, extensions and so on.
- Hot-swap or replace plugins at runtime.
- Faster compile-time
References:
- Plugins in CPP: Dynamically Linking Shared Objects · christopherpoole/cppplugin Wiki · GitHub
- How to Use the Dynamic Link Library in C++ Linux (gcc compiler)? | Technology of Computing
- C++ Dynamic Loading of Shared Objects at Runtime - Programming - 0x00sec - The Home of the Hacker
- SOFA – Create your plugin
- The Geochemist's Workbench® Plug-in: Wrapper
- A Proposal to add Classes and Functions Required for Dynamic Library Load
- Plugin Architecture – Nuclex Games Blog
- Andy Nicholas » Blog Archive » A Simple Plugin Architecture For C++
Other Implementations:
- GitHub - sourcey/pluga: Simple C++ cross-platform plugin system
- DynObj - C++ Cross Platform Plugin Objects - CodeProject
- Building a Simple C++ Cross-platform Plugin System | Sourcey
- Making a Plugin System - C++ Articles
- Using Dynamic Link Libraries (DLL) to Create Plug-Ins - 3D Game Engine Programming3D Game Engine Programming
- https://github.com/jbandela/cppcomponents (Similar to Windows' COM componets)
- C++ patterns: static registration
1.4.2 API used for loading shared libraries dynamically
Unix C-API for loading shared libraries (shared objects)
// Returns handle to shared library void* dlopen(const char* fileName, int flags); // Close handle int dclose(void* handle) // Get symbol (C-function or function with C-linkage) void* dlsym(void* handle, const char* FunctionName)
Windows APIs: (Reference MSDN)
// Ansi version HMODULE LoadLibraryA( LPCSTR lpLibFileName); // Wide Unicode UTF-16 HMODULE LoadLibraryW(LPCWSTR lpLibFileName); // FAPROC => Returns Pointer to function casted as void* FARPROC GetProcAddress(HMODULE hModule, LPCSTR lpProcName); void FreeLibrary(HMODULE hModule);
1.4.3 Example
- Overview
This section presents a simple plugin system which allows loading classes at runtime from dynamically-loaded shared libraries. This approach allows adding new functionality to a software without recompilation or modification of the source code. The plugin system can load any class matching a user supplied interface without knowing anything about a particular class implementation.
In this example: The plugin PluginA, which the file name is PluginA.dll on Windows or PluginA.so on Linux, exports the classes 'Log' and 'Exp' that implements the interface class IMathFunction. The application loads those classes only knowing the interface IMathFunction. This design allows to add more third-party plugins and more classes matching any known interface class.
General Consideration for an Interface Class
- MUST NOT have member variables.
- MUST have only pure virtual member functions, in other words, only abstract methods.
ABI - Application Binary Interface Considerations for an Interface Class
- SHOULD use only C-compatible types in the method signatures in order to ensure that the interface to be compatible with multiple compilers. Note: It is very likely that the application and the plugins (aka addons) may be developed with different compilers without a common ABI - Application Binary Interface.
- SHOULD NOT use any use any STL type or container in the class method signatures, although the implementation classes provided by the plugins can use them internally. The reason to avoid the use of the STL is that the STL implementation is compiler-dependent what can lead to liking error when linking object-codes generated by different compilers.
- The STL and non-C compatible can be used in interface classes' methods signatures only if the main application and the plugins are developed by the same team with the same compiler version.
Fragile-Base-class Problem and class ABI
- Avoid changing interfaces - Any interface class already being used by third parties plugins should not be changed, otherwise it will break the interface class ABI and disrupt third parties' plugins as they will no longer be binary compatible with the interface. As a result, all plugins would have to be recompiled.
- Fragile Base Class Problem: - Some changes that can break a base
class ABI requiring recompilation of all derived classes are:
- Add new virtual member functions, aka virtual methods
- Change the order of virtual member functions
- Add new member variables
- Change the order or type of member variables
- Solution: The solution to fragile base class problem in this case is: don't modify interface classes relied by third-party plugins and add new interface classes or extend the existing ones instead of modifying them.
Source Code:
- Repository: github/sample-cpp-plugin
- File: psdk/interfaces.hpp
This interface class provides plugin metadata and a factory function (Factory) which allows to instantiate any class. The method Factory takes as argument the name of a class provided by the plugin and returns an instance of it casted as a void pointer (opaque pointer).
#ifndef _INTERFACES_HPP_ #define _INTERFACES_HPP_ struct IPluginFactory{ /** Get Plugin Name */ virtual const char* Name() const = 0 ; /** Get Plugin Version */ virtual const char* Version() const = 0; virtual size_t NumberOfClasses() const = 0; virtual const char* GetClassName(size_t index) const = 0; /** Instantiate a class from its name */ virtual void* Factory(const char* className) const = 0; }; #endif
- File: psdk/factory.hpp
Headers:
ifndef _FACTORY_HPP_ #define _FACTORY_HPP_ #include <string> #include <functional> #include <map> #include <vector> #include <algorithm> #include "interfaces.hpp" /** Macro EXPORT_CPP makes a symbol visible. */ #if defined(_WIN32) // MS-Windows NT #define PSDK_PLUGIN_EXPORT_C extern "C" __declspec(dllexport) #else // Unix-like OSes #define PSDK_PLUGIN_EXPORT_C extern "C" __attribute__ ((visibility ("default"))) #endif
Class PluginFactory: Implementation of the interface IPluginFactory.
class PluginFactory: public IPluginFactory { // Constructor database using CtorItem = std::pair<std::string, std::function<void* ()>>; using CtorDB = std::vector<CtorItem>; std::string m_name; std::string m_version; CtorDB m_ctordb; public: PluginFactory(const char* name, const char* version): m_name(name), m_version(version) { } /** Get Plugin Name */ const char* Name() const { return m_name.data(); } /** Get Plugin Version */ const char* Version() const { return m_version.data(); } virtual size_t NumberOfClasses() const { return m_ctordb.size(); } virtual const char* GetClassName(size_t index) const { return m_ctordb[index].first.data(); } /** Instantiate a class from its name */ void* Factory(const char* className) const { auto it = std::find_if(m_ctordb.begin(), m_ctordb.end(), [&className](const auto& p){ return p.first == className; }); if(it == m_ctordb.end()) return nullptr; return it->second(); } template<typename AClass> PluginFactory& registerClass(std::string const& name) { auto constructor = []{ return new (std::nothrow) AClass; }; m_ctordb.push_back(std::make_pair(name, constructor)); return *this; } }; #endif
- File: psdk/loader.hpp
Headers:
ifndef _LOADER_HPP_ #define _LOADER_HPP_ #include <cassert> #include <map> #include <memory> #include "interfaces.hpp" // Unix #if defined(_WIN32) #include <windows.h> #elif defined(__unix__) // APIs: dlopen, dlclose, dlopen #include <dlfcn.h> #else #error "Not supported operating system" #endif #ifdef GetClassName #undef GetClassName #endif
Class Plugin:
- This class loads a shared library dynamically with dlopen on Unix or LoadLibrary on Windows and gets a function pointer to the symbol GetPluginFactory which returns a pointer to a statically allocated IPluginFactory object that allows instantiating at runtime any registered class in the plugin.
class Plugin { public: // Function pointer to DLL entry-point using GetPluginInfo_fp = IPluginFactory* (*) (); void* m_hnd = nullptr; std::string m_file = ""; bool m_isLoaded = false; IPluginFactory* m_info = nullptr; Plugin() { } explicit Plugin(std::string file) { m_file = std::move(file); #if !defined(_WIN32) m_hnd = ::dlopen(m_file.c_str(), RTLD_LAZY); #else m_hnd = (void*) ::LoadLibraryA(m_file.c_str()); #endif m_isLoaded = true; assert(m_hnd != nullptr); #if !defined(_WIN32) auto dllEntryPoint = reinterpret_cast<GetPluginInfo_fp>(dlsym(m_hnd, "GetPluginFactory")); #else auto dllEntryPoint = reinterpret_cast<GetPluginInfo_fp>(GetProcAddress((HMODULE) m_hnd, "GetPluginFactory")); #endif assert(dllEntryPoint != nullptr); // Retrieve plugin metadata from DLL entry-point function m_info = dllEntryPoint(); } ~Plugin() { this->Unload(); } Plugin(Plugin const&) = delete; Plugin& operator=(const Plugin&) = delete; Plugin(Plugin&& rhs) { m_isLoaded = std::move(rhs.m_isLoaded); m_hnd = std::move(rhs.m_hnd); m_file = std::move(rhs.m_file); m_info = std::move(rhs.m_info); } Plugin& operator=(Plugin&& rhs) { std::swap(rhs.m_isLoaded, m_isLoaded); std::swap(rhs.m_hnd, m_hnd); std::swap(rhs.m_file, m_file); std::swap(rhs.m_info, m_info); return *this; } IPluginFactory* GetInfo() const { return m_info; } void* CreateInstance(std::string const& className) { return m_info->Factory(className.c_str()); } bool isLoaded() const { return m_isLoaded; } void Unload() { if(m_hnd != nullptr) { #if !defined(_WIN32) ::dlclose(m_hnd); #else ::FreeLibrary((HMODULE) m_hnd); #endif m_hnd = nullptr; m_isLoaded = false; } } };
Class PluginManager:
class PluginManager { public: using PluginMap = std::map<std::string, Plugin>; // Plugins database PluginMap m_plugindb; PluginManager() { } std::string GetExtension() const { std::string ext; #if defined (_WIN32) ext = ".dll"; // Windows #elif defined(__unix__) && !defined(__apple__) ext = ".so"; // Linux, BDS, Solaris and so on. #elif defined(__apple__) ext = ".dylib"; // MacOSX #else #error "Not implemented for this platform" #endif return ext; } IPluginFactory* addPlugin(const std::string& name) { std::string fileName = name + GetExtension(); m_plugindb[name] = Plugin(fileName); return m_plugindb[name].GetInfo(); } IPluginFactory* GetPluginFactory(const char* pluginName) { auto it = m_plugindb.find(pluginName); if(it == m_plugindb.end()) return nullptr; return it->second.GetInfo(); } void* CreateInstance(std::string pluginName, std::string className) { auto it = m_plugindb.find(pluginName); if(it == m_plugindb.end()) return nullptr; return it->second.CreateInstance(className); } template<typename T> std::shared_ptr<T> CreateInstanceAs(std::string pluginName, std::string className) { void* pObj = CreateInstance(std::move(pluginName), std::move(className)); return std::shared_ptr<T>(reinterpret_cast<T*>(pObj)); } }; /* --- End of class PluginManager --- */ #endif
- File: PluginA.cpp
Sample plugin file exporting classes Exp and Log which implement the interface IMathFunction. When compiled on Windows, the file name becomes PluginA.dll, when compiled on Linux, it becomes PluginA.so.
Header:
#include <iostream> #include <cmath> #include <psdk/factory.hpp>
Interface class IMathFunction:
class IMathFunction { public: virtual const char* Name() const = 0; virtual double Eval(double x) const = 0; virtual ~IMathFunction() = default; };
Sample classes exported by the Plugin: implementations
class Exp: public IMathFunction { public: Exp() = default; const char* Name() const { return "Exp"; } double Eval(double x) const { return std::exp(x); } }; class Log: public IMathFunction { public: Log() = default; const char* Name() const { return "Log"; } double Eval(double x) const { return std::log(x); } };
Plugin Entry Point:
- This function returns a PluginFactory object which contains the shared library metadata and a factory function for instantiating the exported classes Exp and Log. On Unix-like operating system, this global function is loaded by through the API dlopen and dlsym. On Windows, this entry point is loaded through the Win32 APIs LoadLibrary and GetProcAddress.
PSDK_PLUGIN_EXPORT_C auto GetPluginFactory() -> IPluginFactory* { static PluginFactory pinfo = []{ auto p = PluginFactory("PluginA", "0.1-alpha"); p.registerClass<Exp>("Exp"); p.registerClass<Log>("Log"); return p; }(); return &pinfo; }
- File: main.cpp
Main application that loads the sample plugin PluginA, file name PluginA.so (Linux) or PluginA.dll (Windows).
Headers and interface:
#include <iostream> #include <psdk/loader.hpp> class IMathFunction { public: virtual const char* Name() const = 0; virtual double Eval(double x) const = 0; virtual ~IMathFunction() = default; };
Main Function:
Create a plugin manager object (ma), a repository of plugins:
PluginManager ma;
Load the plugin file at current directory, on Windows it loads the shared library file PluginA.dll and on Linux, it loads the file PluginA.so (aka shared object)
IPluginFactory* infoA = ma.addPlugin("PluginA"); assert(infoA != nullptr);
Print plugin metadata:
std::cout << " ---- Plugin Information --------- " << "\n" << " => Name = " << infoA->Name() << "\n" << " => Version = " << infoA->Version() << "\n" << " => Number of classes = " << infoA->NumberOfClasses() << "\n\n"; std::cout << "Classes exported by the Plugin: (PluginA) " << "\n"; for(size_t n = 0; n < infoA->NumberOfClasses(); n++) { std::cout << " -> Exported Class: " << infoA->GetClassName(n) << "\n"; }
Load class 'Exp' from the plugin or module 'PluginA' and casting it to the interface IMathFunction.
// Type of pExp: std::shared_ptr<IMathFunction> auto pExp = ma.CreateInstanceAs<IMathFunction>("PluginA", "Exp"); assert(pExp != nullptr); std::cout << "pExp->Name() = " << pExp->Name() << "\n"; std::cout << "pExp->Eval(3.0) = " << pExp->Eval(3.0) << "\n";
Load class 'Log' from plugin 'PluginA':
auto pLog = ma.CreateInstanceAs<IMathFunction>("PluginA", "Log"); std::cout << "pLog->Name() = " << pLog->Name() << "\n"; std::cout << "pLog->Eval(3.0) = " << pLog->Eval(3.0) << "\n";
Compilation on Linux:
# Build application $ g++ main.cpp -o main.bin -I. -Wall -std=c++1z -ldl # Build the Plugin DLL - Shared library $ g++ PluginA.cpp -o PluginA.so -I. -Wall -std=c++1z -shared -fPIC # Test the application $ ./main.bin
Compile on Linux with CMake
$ git clone https://github.com/caiorss/sample-cpp-plugin $ cd sample-cpp-plugin $ cmake -Bbuild -H. $ cmake --build build --target install $ ls bin/ main.bin* PluginA.so* $ ./main.bin
Compile On Windows with CMake
$ git clone https://github.com/caiorss/sample-cpp-plugin $ cd sample-cpp-plugin $ cmake -H. -Bbuild2 -DCMAKE_BUILD_TYPE=DEBUG -G "Visual Studio 15 2017 Win64" $ cmake --build build2 --target install $ cd bin $ ls PluginA.dll* PluginA.lib PluginA.so main.bin main.exe* $ main.exe # Load PluginA.dll
Program Output:
$ ./main.bin [TRACE] Shared library PluginA loaded OK. ---- Plugin Information --------- => Name = PluginA => Version = 0.1-alpha => Number of classes = 2 Classes exported by the Plugin: (PluginA) -> Exported Class: Exp -> Exported Class: Log pExp->Name() = Exp pExp->Eval(3.0) = 20.0855 pLog->Name() = Log pLog->Eval(3.0) = 1.09861 [TRACE] Shared library PluginA unloaded OK. (base)
1.5 Return string from a C++ function to Python Ctypes
Example: Create a function with C linkage or C-API which returns a dynamically allocated string to the caller.
- Source: Gist with all files
Necessary C-string functions:
Build shared library - DLL (cstr.so):
$ clang++ cstr.cpp -o cstr.so -std=c++1z -O0 -g -shared -fPIC -std=c++1z
File cstr.cpp
Headers:
#include <iostream> #include <string> #include <sstream> #include <cstring> // std::strncpy #include <cmath> #if defined(_WIN32) // MS-Windows #define EXPORT_C extern "C" __declexpec(dllexport) #else // Unix #define EXPORT_C extern "C" #endif
Function - version A:
/** Design 1: Buffer allocated by the caller. * @param[in] x Float point number * @param[in,out] buffer Buffer allocated by the caller. * @param[in] buffer_size Size of buffer allocated by the caller in bytes. * @return Size of copied array. */ EXPORT_C size_t square_root_as_stringA(double x, char* buffer, size_t buffer_size) { std::string ss; ss = ss + "The square root of x = " + std::to_string(x) + " is equal to " + std::to_string(sqrt(x)); std::strncpy(buffer, ss.data(), buffer_size); return ss.size(); }
Function - version B:
/** Design 2: Buffer allocated by the caller. If the buffer is null, the function * returns the required buffer size. * @param[in] x Float point number * @param[in,out] buffer Buffer allocated by the caller. * @param[in] buffer_size Size of buffer allocated by the caller in bytes. * @return Size of copied array or required buffer size. */ EXPORT_C size_t square_root_as_stringB(double x, char* buffer, size_t buffer_size) { std::string ss; ss = ss + "The square root of x = " + std::to_string(x) + " is equal to " + std::to_string(sqrt(x)); if(buffer == nullptr) return ss.size(); std::strncpy(buffer, ss.data() /* const char* */, buffer_size); return ss.size(); }
Function - version C:
/** Design3: Buffer allocated by callee (this code) and released by the * caller. */ EXPORT_C char* square_root_as_stringC(double x){ std::string ss; ss = ss + "The square root of x = " + std::to_string(x) + " is equal to " + std::to_string(sqrt(x)); return strdup(ss.data()); } EXPORT_C void cstring_delete(char* pCstring) { std::cout << " [INFO] C-string deleted, memory released. OK." << "\n"; std::free(pCstring); }
file: cstrp.py (Python Wrapepr)
Initial configuration:
import ctypes CString = ctypes.POINTER(ctypes.c_char) STDString = ctypes.c_void_p CVoid = None dll = ctypes.cdll.LoadLibrary("cstr.so")
Wrapper for function A:
print("\n EXPERIMENT1 => Call square_root_as_stringA") print("=============================================") dll.square_root_as_stringA.argtypes = [ ctypes.c_double, CString, ctypes.c_size_t ] dll.square_root_as_stringA.restype = ctypes.c_size_t buf_size = 1024 # 1 Kbyte = 1024 bytes buf = ctypes.create_string_buffer(buf_size) n_read = dll.square_root_as_stringA(100.45, buf, buf_size) print("Size = ", n_read, " ; RESULT = ", buf.value)
Wrapper for function B:
print("\n EXPERIMENT 2 => Call square_root_as_stringB") print("=============================================") dll.square_root_as_stringB.argtypes = [ ctypes.c_double, CString, ctypes.c_size_t ] dll.square_root_as_stringB.restype = ctypes.c_size_t x = 200.45 required_size = dll.square_root_as_stringB(x, ctypes.c_char_p(0), 0) buf = ctypes.create_string_buffer(required_size) dll.square_root_as_stringB(x, buf, required_size) print(" RESULT = ", buf.value)
Wrapper for function C:
print("\n EXPERIMENT 3 => Call square_root_as_stringC") print("=============================================") dll.square_root_as_stringC.argtypes = [ ctypes.c_double ] dll.square_root_as_stringC.restype = ctypes.c_char_p dll.cstring_delete.argtypes = [ ctypes.c_char_p ] dll.cstring_delete.restype = CVoid hCString = dll.square_root_as_stringC(167.42) print(" RESULT = ", hCString)
Program output
$ python3 cstr.py EXPERIMENT1 => Call square_root_as_stringA ============================================= Size = 55 ; RESULT = b'The square root of x = 100.450000 is equal to 10.022475' EXPERIMENT 2 => Call square_root_as_stringB ============================================= RESULT = b'The square root of x = 200.450000 is equal to 14.158037' EXPERIMENT 3 => Call square_root_as_stringC ============================================= RESULT = b'The square root of x = 167.420000 is equal to 12.939088'
2 Python Native Extension API
2.1 Overview
Many programming languages provide a C-native interfaces such as Python C-API or Java Native Interface where it is possible to create native-code libraries in C without overhead of virtual machine code interpretation. The advantage of a code for a C native interface (native API) of some programming language is the easier usage and better integration with the target language. The drawback for developing some native library with this approach is that the library will be tight coupled with the language native C API, as a result, it will be hard to port or call the code from other programming language. In addition, it may require deeper knowledge of the native C API which often is not well documented.
Definition:
- A programming language native interface API is a C-API which allows creating libraries almost entirely in C compiled as shared libraries and load them in the same way as ordinary libraries.
Native Interface API X Foreign Function Interface based on Libffi
- Advantages of Native Interface over FFI
- Better performance
- Enhanced integration
- Easier to use
- Drawbacks of Native Interface over FFI:
- Most native APIs are not well documented and they are specific for a given programming language.
- Requires in-deep knowledge of the native C-API and the target programming language.
- Hard to reuse the code with other languages and even with C++.
- Code tightly coupled with the Native Interface API.
- Advantages of FFI over Native Interface:
- The code does not need to conform to any native interface C-API and can be called from any programming language. The shortcoming of this approach is the need of C-interfaces (extern "C") for the C++ code and a FFI wrapper.
- Most of FFIs are based on the library LibFFI which is used by Python (ctypes), Ruby, Java JNA, Haskell and so on. Therefore, the concept of creating FFI wrappers for C or C++ libraries with C-interfaces can be reused on all languages with LibFFI bindings.
Some Programming language specific Native Interface APIs (C-APIs):
- Java Native Interface JNI
- Python C-API
- Ruby C-API
- Lua C-API
- OCaml
- GNU OCtave (Open source implementation of Matlab - Mathworks)
Other approaches for developing shared libraries for Python native API:
- Pybind11 header-only library
- => Provides a C++-friendly syntax and interface to the Python native C API. It allows building Python native modules and port C++ code in a easier and faster way than dealing directly with Python C-API.
- SWIG wrapper generator
- => SWIG can parse C++ headers and interface files and generate C++ binding code to the native API of many programming languages such as Python, Ruby, Java and so on.
- Cython
- => Python-like language that compile to C code (generate C-code). The disadvantage is the tight coupling to the Python C-API.
Further documentation and reading:
Documentation:
- Porting Extension Modules to Python 3 — Python 3.7.2 documentation
- Parsing Python Arguments — Python Extension Patterns 0.1.0 documentation
- ActivePython 3.6.6 Documentation
- Extending and Embedding the Python Interpreter — Python 3.7.2 documentation
General:
- Common Object Structures — Python 3.7.2 documentation
- Python Types and C-Structures — NumPy v1.15 Manual
- 7.3.1 String Objects
- Code Objects — Python 2.7.15 documentation
- The Very High Level Layer — Python 2.7.15 documentation
- Create and call python function from string via C API - Stack Overflow
- Embed Python scripting in C applications
- Python C API: PyEval_CallFunction? - Stack Overflow
- How do I call an object's method from C?
Creating a native Python module:
- Write C++ extensions for Python - Visual Studio | Microsoft Docs
- Calling C functions from Python - part 2 - writing CPython extensions using Python/C API | yizhang82’s blog
Python (CPython) debugging with GDB:
- 22. gdb Support — Python Developer's Guide
- Mixed-mode debugging for Python - Visual Studio | Microsoft Docs
Reverse engineering:
2.2 Python Raw C API
2.2.1 Module Code
- Sources
Module Source:
- File: file:src/dlls/mymodule.cpp
- Gist: mymodule.cpp
Compilation:
# GCC - GNU C/C++ compiler $ g++ mymodule.cpp -o mymodule.so -g -std=c++1z -fPIC -shared -I/usr/include/python3.6m # Clang LLVM $ clang++ mymodule.cpp -o mymodule.so -g -std=c++1z -fPIC -shared -I/usr/include/python3.6m
- File: setup.py
Python script used for building and installing the Python module.
from distutils.core import setup, Extension mymodule = Extension('mymodule', sources = ['mymodule.cpp']) setup( name = 'mymodule', version = '1.0', description = 'Sample python C-API exploration', ext_modules = [mymodule] )
- File: mymodule.cpp
Heades:
#include <iostream> #include <string> #include <functional> #include <iomanip> // Solve Mingw error: '::hyport' has not been declared #include <math.h> //#define hypot _hypot /// C-API header to Python Native Interface API, similar to Java JNI /// (Java Native Interface) #include <Python.h> #ifdef _WIN32 #include <process.h> // Exports _getpid() #else #include <unistd.h> // Exports getpid() #endif
Global object for DLL initialization (optional):
- The global object dllinit_hook is used for logging to stdout when the DLL is loaded or unloaded by the python process.
struct DLLInitialization { DLLInitialization(){ int pid; #ifdef _WIN32 pid = _getpid(); #else pid = getpid(); #endif std::cerr << " [TRACE] Python module DLL loaded by process PID = <" << pid << "> " << std::endl; std::cerr << " [TRACE] Attach the debugger with: $ gdb -pid=" << pid << "\n"; } ~DLLInitialization(){ std::cerr << " [TRACE] DLL native DLL unloaded OK." << std::endl; } }; DLLInitialization dllinit_hook;
Exposed functions declarations:
/** ======== List of functions exposed to Python =========== */ PyObject* exposedFunction(PyObject* self, PyObject* args); PyObject* testArguments(PyObject* self, PyObject* args); auto printRange(PyObject* self, PyObject* args) -> PyObject*; auto taylorSeriesExp(PyObject* self, PyObject* args) -> PyObject*; PyObject* returnTuple(PyObject* self, PyObject* args); PyObject* returnDictionary(PyObject* self, PyObject* args); PyObject* tabulateFunction(PyObject* self, PyObject* args); PyObject* computeStatistics(PyObject* self, PyObject* args);
Table of exported functions to the Python module:
static PyMethodDef ModuleFunctions [] = { { "exposedFunction", exposedFunction, METH_VARARGS, "Documentation or docstring of function exposedFunction1." } ,{"testArguments", testArguments, METH_VARARGS, "Test python arguments: Signature testArguments(int, float, double, const char*)"} // Function without docstring: printRange(int i, double x) ,{ "printRange", printRange, METH_VARARGS, nullptr } ,{ "taylorSeriesExp", &taylorSeriesExp, METH_VARARGS, "taylorSeriesExp(double x, size_t maxiter, double tol) -> double" "\n Computes exponential of a given value with taylor serie approximation." "\n Formula reference: https://www.mathsisfun.com/algebra/taylor-series.html" } ,{"returnTuple", &returnTuple, METH_VARARGS, nullptr} ,{"returnDictionary", &returnDictionary, METH_VARARGS, nullptr} ,{"computeStatistics", &computeStatistics, METH_VARARGS, nullptr} ,{"tabulateFunction", tabulateFunction, METH_VARARGS, "Tabulate some mathematical function or callable object"} // Sentinel value used to indicate the end of function listing. // All function listing must end with this value. ,{nullptr, nullptr, 0, nullptr} };
Module definition:
/* Module definition */ static struct PyModuleDef ModuleDefinitions { PyModuleDef_HEAD_INIT, // Module name as string "mymodule", // Module documentation (docstring) "A sample C++ native-code module for python3.", -1, // Functions exposed to the module ModuleFunctions };
Module initialization function:
- Note: This function must contain the name in the format
PyInit_<MODULE NAME>
where module name is the same name as the module file name <MODULE NAME>.so (Linux) or <MODULE NAME>.pyd on Windows.
/** Module Initialization function: must have this name schema * PyInit_<ModuleName> where ModuleName is the same base name of the * shared library ModuleName.so (on Linux) or ModuleName.pyd (on Windows) */ PyMODINIT_FUNC PyInit_mymodule(void) { Py_Initialize(); PyObject* pModule = PyModule_Create(&ModuleDefinitions); PyModule_AddObject(pModule, "version", Py_BuildValue("s", "version 0.1-Alpha")); return pModule; }
Module Functions Implementations
All functions exposed to the Python C-API must have the same type signature of the function pointer PyExposedFunction or the same type signature of the function MyExposedFunction.
using PyExposedFunction = PyObject* (*) (PyObject* self, PyObject* args) // Example: PyObject* MyExposedFunction(PyObject* self, PyObject* args ) { // On Failure return null return nullptr; return /* Some instance of PyObject */; }
Notes:
- The function takes always two arguments self, and args (tuple object containing arguments passed to the function)
- The exposed function always return an object PyObject.
- All Python objects are heap-allocated.
- In Python everything is an object: integers, double, classes, class instances, modules and functions and so on. All those types have the same type PyObject in the C-API.
- Functions exposed to Python runtime should not throw a C++ exceptions, instead they should return null (nullptr) that causes a runtime error or exception on Python side.
Function: exposedFunction
- Function that takes a void argument and returns nothing.
PyObject* exposedFunction(PyObject* self, PyObject* args) { std::cout << " =>> Hello word Python from C++ side." << "\n"; // All python functions that returns anything // should end with this macro Py_RETURN_NONE; }
Function: testArguments
- Function that takes many arguments used to demonstrate how to extract tuple arguments.
PyObject* testArguments(PyObject* self, PyObject* args) { int abool; int aint; float afloat; double adouble; const char* aword; // Parse function arguments if(!PyArg_ParseTuple(args, "pifds", &abool, &aint, &afloat, &adouble, &aword)) return nullptr; std::cerr << " Function: [" << __FUNCTION__ << "]" << " ==> Number of arguments = " << PyTuple_GET_SIZE(args) << "\n"; std::string str = aword; str = "'Received string = " + str + "'"; std::cout << " Received Python Arguments " << "\n"; std::cout << std::boolalpha; std::cout << "abool[bool] = " << (bool) abool << "\n" << " abool[int] = " << abool << "\n" << " aint = " << aint << "\n" << " afloat = " << afloat << "\n" << " adouble = " << adouble << "\n" << " aword = " << str << "\n" << "\n"; return Py_BuildValue("s", str.data()); }
Function PyArg_ParseTuple:
- The function PyArg_ParseTuple is used for extracting function arguments from the tuple object (args).
- The letters in the string parameter "pifds" means:
- p => Extract first argument from tuple object as boolean (integer 0 or 1) storing the value in the variable abool.
- i => Extract second argument from tuple object as integer storing the value in the variable 'aint'.
- f => Extract third argument as float (float point 32 bits). The value is then stored in the variable 'afloat'.
- d => Extract fourth argument as double (float point 64 bits)
- s => Extract fith argument as string.
- If the function fails, it must return null which causes a runtime error on Python side.
int abool; int aint; float afloat; double adouble; const char* aword; // Parse function arguments if(!PyArg_ParseTuple(args, "pifds", &abool, &aint, &afloat, &adouble, &aword)) return nullptr;
Function Py_BuildValue("s", str.data());
- This function is used for building return values to the python side. It can return any python object, such as string, number, class instance or multiple values as a tuple.
// Return None in the python side return Py_BuildValue(""); // Return a single value as string string return Py_BuildValue("s", str.data()); return Py_BuildValue("s", "A string to python side"); // Return integer return Py_BuildValue("i", 1000); // Return float return Py_BuildValue("f", 3.465); // Return double return Py_BuildValue("d", 140.346); // Return a tuple with a single double value: (3.156) return Py_BuildValue("(d)", 3.156); // Return a tuple with a single double and integer: (3.156, 100) return Py_BuildValue("(di)", 3.156, 100); // Return a tuple with a single double, integer and string // tuple object: (3.156, 100, "a string") return Py_BuildValue("(dis)", 3.156, 100, "a string");
Function: printRange
- Pseudo-Python signature: printRange(int n, float x) => void
auto printRange(PyObject* self, PyObject* args) -> PyObject* { std::cerr << " =>> Printing numeric range " << "\n"; int n; float x; // Extract function argument from tuple argument object // "if" => i - Extract integer // => f - Extract float (IEEE754 32 bits float point) if(!PyArg_ParseTuple(args, "if", &n, &x)){ PyErr_SetString(PyExc_RuntimeError, "Invalid argument"); // Always return null on failure return nullptr; } if(n <= 0){ PyErr_SetString( PyExc_RuntimeError ,"Invalid argument: n supposed to be greater than zero."); return nullptr; } for(int k = 0; k < n; k++) std::cout << " k[" << k << "] = " << 3 * k + x << "\n"; Py_RETURN_NONE; }
Function: returnTuple
- Return multiple values as tuple.
/** Function that returns multiple values as tuple object */ PyObject* returnTuple(PyObject* self, PyObject* args) { double x; double y; int n; if(!PyArg_ParseTuple(args, "ddi", &x, &y, &n)) return nullptr; // Build tuple object containing // (d: double, d: double, i: integer, s: string) return Py_BuildValue("(ddis)", 2 * x + y * n, x + 2 * y - n, n, "Result"); }
Function: returnDictionary
- Return a dictionary (hash-table)
/** Return a dictionary object */ PyObject* returnDictionary(PyObject* self, PyObject* args) { double x; double y; int n; if(!PyArg_ParseTuple(args, "ddi", &x, &y, &n)) return nullptr; // Create new dictionary object PyObject* pDict = PyDict_New(); PyObject* p = nullptr; // Fill dictionary p = Py_BuildValue("d", 2 * x + y * n); PyDict_SetItemString(pDict, "alpha", p); Py_DECREF(p); p = Py_BuildValue("i", 3 * n) ; PyDict_SetItemString(pDict, "size", p); Py_DECREF(p); p = Py_BuildValue("s", "Some string") ; PyDict_SetItemString(pDict, "beta", p); Py_DECREF(p); return pDict; }
Function: computeStatistics
- Computes statistics sum and mean for any iterable object containing double values such as list.
PyObject* computeStatistics(PyObject* self, PyObject* args) { std::cerr << " [TRACE] " << __FILE__ << "(" << __LINE__ << ") - " << "Calling function: " << __FUNCTION__ << "\n"; std::cerr << " [TRACE] Arguments = "; PyObject_Print(args, stdout, 0); std::cerr << "\n"; PyObject* pSeq; // Parse function argument as sequence if(!PyArg_ParseTuple(args, "O", &pSeq)) return nullptr; pSeq = PySequence_Fast(pSeq, "Expected iterable arguments"); if(pSeq == nullptr) return nullptr; int numberOfElements = PySequence_Fast_GET_SIZE(pSeq); std::cerr << " [TRACE] numberOfElements = " << numberOfElements << "\n"; PyObject* pItem = nullptr; double x; double sum; for(int n = 0; n < numberOfElements; n++) { pItem = PySequence_Fast_GET_ITEM(pSeq, n); if(!pItem) { Py_DECREF(pSeq); return nullptr; } // Print item std::cout << "item[" << n << "] = "; PyObject_Print(pItem, stdout, 0); std::cout << "\n"; x = PyFloat_AsDouble(pItem); if(PyErr_Occurred() != nullptr){ PyErr_SetString(PyExc_TypeError, "Error: expected float point."); return nullptr; } sum += x; } double mean = sum / numberOfElements; std::cout << "\n (C++) Statics Result " << "\n"; std::cout << "---------------" << "\n"; std::cout << " Sum = " << sum << "\n"; std::cout << " Mean = " << mean << "\n"; Py_DECREF(pSeq); return Py_BuildValue(""); // Return None }
Function: taylorSeriesExp
- This function computes the exponential function approximation using its taylor series expansion. Formula at: taylor series
auto taylorSeriesExp(PyObject* self, PyObject* args) -> PyObject* { double x; size_t maxiter; // Maximum number of iterations double tol; // Tolerance // Parse function arguments if(!PyArg_ParseTuple(args, "did", &x, &maxiter, &tol)) return nullptr; // Validate function arguments if(tol <= 0 || tol > 1.0){ PyErr_SetString( PyExc_RuntimeError ,"Invalid tolerance, expected in range (0, 1]"); return nullptr; } // Compute exponential taylor series available at // https://www.mathsisfun.com/algebra/taylor-series.html unsigned long factorial = 1; double xpower = 1.0; double sum = 0.0; double term = 0.0; size_t idx = 1; do{ term = xpower / factorial; sum = sum + term; xpower = xpower * x ; factorial = factorial * idx; idx++; } while(idx <= maxiter && std::abs(term) > std::abs(sum) * tol ); // Return float point constnat NAN (Not a Number) if(idx >= maxiter){ std::cerr << " [ERROR] Series does not converge." << "\n"; return Py_BuildValue("d", NAN); } return Py_BuildValue("d", sum); }
Function: tabulateFunction
- This function takes a callable object (callback) which can be a
function, lambda function, callable object (object with method
__call__
) and so on.
Pseudo Python method signature using C++ notation:
using Callback = std::function<double (double)>; void tabulateFunction(Callback callback, double xmin, double xmax, double step)
Function code:
PyObject* tabulateFunction(PyObject* self, PyObject* args) { PyObject* pObj = nullptr; double xmin, xmax, xstep; if(!PyArg_ParseTuple(args, "Oddd", &pObj, &xmin, &xmax, &xstep)) return nullptr; if(pObj == nullptr) { PyErr_SetString(PyExc_RuntimeError, "Error: invalid None object."); return nullptr; } PyObject* pArgs = nullptr; PyObject* pResult = nullptr; double y = 0.0; std::cout << std::fixed << std::setprecision(4); std::cout << "Tabulating range: " << " ; xmin = " << xmin << " ; xmax = " << xmax << " ; step = " << xstep << "\n"; for(double x = xmin; x <= xmax; x += xstep ) { pArgs = Py_BuildValue("(d)", x); pResult = PyEval_CallObject(pObj, pArgs); y = PyFloat_AsDouble(pResult); if(PyErr_Occurred() != nullptr){ PyErr_SetString(PyExc_RuntimeError, "Error: Invalid float point."); return nullptr; } std::cout << std::setw(8) << x << std::setw(10) << y << "\n"; } Py_RETURN_NONE; }
Extract python callable object (callback) pObj and the parameters xmin, xmax and xstep:
PyObject* pObj = nullptr; double xmin, xmax, xstep; // Extract parameters // O => pObj // d => xmin // d => xmax if(!PyArg_ParseTuple(args, "Oddd", &pObj, &xmin, &xmax, &xstep)) return nullptr;
Annotated code block in the for-loop:
// Argument (tuple object) of callable object pArgs = Py_BuildValue("(d)", x); // Call callable object (method __call__) pResult = PyEval_CallObject(pObj, pArgs); // Try extracting result as double y = PyFloat_AsDouble(pResult); // Check whether any error has happened and abort the operation in this case. if(PyErr_Occurred() != nullptr){ PyErr_SetString(PyExc_RuntimeError, "Error: Invalid float point."); return nullptr; } // Display table row std::cout << std::setw(8) << x << std::setw(10) << y << "\n";
2.2.2 Compiling and Running Native Module
Clone repository:
$ git clone https://gist.github.com/caiorss/bc7dc373d87b828029eb0e0ec048d91e python-module $ cd python-module
Manual Compilation
Compile the native module mymodule.so on Linux or any other Unix-like OS:
$ clang++ mymodule.cpp -o mymodule.so -g -std=c++1z -fPIC -shared -I/usr/include/python3.6m
Compile the native module mymodule.pyd on Windows OS with Mingw compiler:
$ g++ mymodule.cpp -o mymodule.pyd -g -std=c++1z -shared -fPIC \ -I"C:\Users\archbox\Miniconda3\pkgs\python-3.6.5-h0c2934d_0\include" \ C:\Users\archbox\Miniconda3\pkgs\python-3.6.5-h0c2934d_0\libs\Python36.lib
Compilation with setup.py
Build only:
- $ python.exe setup.py build
$ python.exe setup.py build $ cd build\ lib.win-amd64-3.6\ temp.win-amd64-3.6\ F:\python-module (master -> origin) λ cd build\lib.win-amd64-3.6\ $ ls mymodule.cp36-win_amd64.pyd* $ C:\Users\archbox\Miniconda3\python.exe Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> >>> import mymodule as m [TRACE] Python module DLL loaded by process PID = <620> [TRACE] Attach the debugger with: $ gdb -pid=620
Install:
- $ python.exe setup.py install
- For uninstall see: question 1550226
$ python.exe setup.py install $ C:\Users\archbox\Miniconda3\python.exe setup.py install running install running build running build_ext building 'mymodule' extension creating build\temp.win-amd64-3.6 creating build\temp.win-amd64-3.6\Release C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\Users\archbox\Miniconda3\include -IC:\Users\archbox\Miniconda3\include "-IC:\Program Files (x86)\Mi crosoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\ATLMFC\include" ... ... ... .... ... .... ... .... ... .... ... .... ... .... ... .... ... .... ... .... ... .... ... .... ... .... ... .... ... .... Finished generating code running install_lib copying build\lib.win-amd64-3.6\mymodule.cp36-win_amd64.pyd -> C:\Users\archbox\Miniconda3\Lib\site-packages running install_egg_info Writing C:\Users\archbox\Miniconda3\Lib\site-packages\mymodule-1.0-py3.6.egg-info
Check generated file on Linux:
$ file mymodule.so $ du -h mymodule.so 136K mymodule.so 136K total
View symbols:
$ nm mymodule.so 0000000000209518 B __bss_start 0000000000005500 t __clang_call_terminate 0000000000209518 b completed.7347 U __cxa_atexit@@GLIBC_2.2.5 U __cxa_begin_catch@@CXXABI_1.3 U __cxa_call_unexpected@@CXXABI_1.3 U __cxa_end_catch@@CXXABI_1.3 w __cxa_finalize@@GLIBC_2.2.5 U __cxa_rethrow@@CXXABI_1.3 0000000000004160 t __cxx_global_var_initmymodule.cpp_ 00000000000041a0 t __cxx_global_var_initmymodule.cpp_.1 00000000000041f0 t deregister_tm_clones 000000000020951a B dllinit_hook .. ... ... ... ... ... ... ...
Load module in Python 3 REPL:
$ python3 Python 3.7.1 (default, Dec 14 2018, 19:28:38) [GCC 7.3.0] :: Anaconda, Inc. on linux Type "help", "copyright", "credits" or "license" for more information. >>> >>> import mymodule as m [TRACE] Python module DLL loaded by process PID = <5236> [TRACE] Attach the debugger with: $ gdb -pid=5236 >>> >>>
Module Docstring (documentation):
>>> help(m) Help on module mymodule: NAME mymodule - A sample C++ native-code module for python3. FUNCTIONS computeStatistics(...) exposedFunction(...) Documentation or docstring of function exposedFunction1. printRange(...) returnDictionary(...) returnTuple(...) tabulateFunction(...) Tabulate some mathematical function or callable object taylorSeriesExp(...) taylorSeriesExp(double x, size_t maxiter, double tol) -> double Computes exponential of a given value with taylor serie approximation. Formula reference: https://www.mathsisfun.com/algebra/taylor-series.html testArguments(...) Test python arguments: Signature testArguments(int, float, double, const char*) DATA version = 'version 0.1-Alpha' FILE /home/archbox/shared/python-module/mymodule.so
Module version:
>>> m.version
'version 0.1-Alpha'
>>>
Call function exposedFunction()
>>> m.exposedFunction()
=>> Hello word Python from C++ side.
>>>
Call function testArguments:
>>> s1 = m.testArguments(False, 100, 10.9361, 0.344, "from Python to C++") Function: [testArguments] ==> Number of arguments = 5 Received Python Arguments abool[bool] = false abool[int] = 0 aint = 100 afloat = 10.9361 adouble = 0.344 aword = 'Received string = from Python to C++' >>> s1 "'Received string = from Python to C++'" >>> >>> >>> s2 = m.testArguments(True, 90, -0.56e6, 9.344e5, "something else more") Function: [testArguments] ==> Number of arguments = 5 Received Python Arguments abool[bool] = true abool[int] = 1 aint = 90 afloat = -560000 adouble = 934400 aword = 'Received string = something else more' >>> s2 "'Received string = something else more'"
Call function printRange()
>>> m.printRange(5, 4.5) =>> Printing numeric range k[0] = 4.5 k[1] = 7.5 k[2] = 10.5 k[3] = 13.5 k[4] = 16.5 >>> m.printRange(-5, 4.5) =>> Printing numeric range Traceback (most recent call last): File "<stdin>", line 1, in <module> RuntimeError: Invalid argument: n supposed to be greater than zero. >>> m.printRange(0, 4.5) =>> Printing numeric range Traceback (most recent call last): File "<stdin>", line 1, in <module> RuntimeError: Invalid argument: n supposed to be greater than zero.
Calll function returnTuple:
>>> tpl = m.returnTuple(3.5, 4.2, 10) >>> tpl (49.0, 1.9000000000000004, 10, 'Result') >>> tpl[0] 49.0 >>> tpl[1] 1.9000000000000004 >>> tpl[2] 10 >>> m.returnTuple(8.5, 4.2, 20) (101.0, -3.1000000000000014, 20, 'Result')
Call functon returnDictionary:
>>> d = m.returnDictionary(8.5, 8.62, 10) >>> d {'alpha': 103.19999999999999, 'size': 30, 'beta': 'Some string'} >>> d["alpha"] 103.19999999999999 >>> d["beta"] 'Some string' >>> d["size"] 30
Call function computeStatistics:
>>> xs = [4, 1.2, 9.5, 10.5, 3.4, 5.10, 6.0] >>> xs [4, 1.2, 9.5, 10.5, 3.4, 5.1, 6.0] >>> m.computeStatistics(xs) [TRACE] mymodule.cpp(268) - Calling function: computeStatistics [TRACE] Arguments = ([4, 1.2, 9.5, 10.5, 3.4, 5.1, 6.0],) [TRACE] numberOfElements = 7 item[0] = 4 item[1] = 1.2 item[2] = 9.5 item[3] = 10.5 item[4] = 3.4 item[5] = 5.1 item[6] = 6.0 (C++) Statics Result --------------- Sum = 39.7 Mean = 5.67143 >>>
Call function taylorSerieExp:
>>> help(m.taylorSeriesExp) Help on built-in function taylorSeriesExp in module mymodule: taylorSeriesExp(...) taylorSeriesExp(double x, size_t maxiter, double tol) -> double Computes exponential of a given value with taylor serie approximation. Formula reference: https://www.mathsisfun.com/algebra/taylor-series.html >>> import math # Exact value >>> math.exp(3.5) 33.11545195869231 # Approximation >>> m.taylorSeriesExp(3.5, 10, 0.001) [ERROR] Series does not converge. nan >>> m.taylorSeriesExp(3.5, 100, 0.001) 33.10588185071678 >>> # Exact value >>> math.exp(5.0) 148.4131591025766 # Approximation >>> m.taylorSeriesExp(5.0, 100, 0.001) 148.37958007973663 >>> m.taylorSeriesExp(5.0, 100, 0.0001) 148.41021027504306 >>>
Call function: tabulateFunction which takes a callable object as argumet.
>>> help(m.tabulateFunction) Help on built-in function tabulateFunction in module mymodule: tabulateFunction(...) Tabulate some mathematical function or callable object
Pass ordinary functions:
- Note: Python ordinary functions are objects with
__call__
method:
import math >>> math.sqrt.__call__(25) 5.0 >>> math.sqrt.__call__(125) 11.180339887498949 >>> >>> m.tabulateFunction(math.exp, -5.0, 5.0, 1.0) Tabulating range: ; xmin = -5.0000 ; xmax = 5.0000 ; step = 1.0000 -5.0000 0.0067 -4.0000 0.0183 -3.0000 0.0498 -2.0000 0.1353 -1.0000 0.3679 0.0000 1.0000 1.0000 2.7183 2.0000 7.3891 3.0000 20.0855 4.0000 54.5982 5.0000 148.4132 >>> >>> a.tabulateFunction(math.sqrt, 0.0, 25.0, 5.0) Tabulating range: ; xmin = 0.0000 ; xmax = 25.0000 ; step = 5.0000 0.0000 0.0000 5.0000 2.2361 10.0000 3.1623 15.0000 3.8730 20.0000 4.4721 25.0000 5.0000
Pass lambda functions:
>>> m.tabulateFunction(lambda x: x, 0.0, 25.0, 4.0) Tabulating range: ; xmin = 0.0000 ; xmax = 25.0000 ; step = 4.0000 0.0000 0.0000 4.0000 4.0000 8.0000 8.0000 12.0000 12.0000 16.0000 16.0000 20.0000 20.0000 24.0000 24.0000 >>> >>> m.tabulateFunction(lambda x: 3 * x + 5, 0.0, 25.0, 4.0) Tabulating range: ; xmin = 0.0000 ; xmax = 25.0000 ; step = 4.0000 0.0000 5.0000 4.0000 17.0000 8.0000 29.0000 12.0000 41.0000 16.0000 53.0000 20.0000 65.0000 24.0000 77.0000
Class LinearFun:
class LinearFun: def __init__(self, a, b): self.a = a self.b = b def __call__(self, x): return self.a * x + self.b def eval(self, x): return self.a * x + self.b
Pass a callable object:
>>> lfun = LinearFun(5, 4) >>> lfun(3) 19 >>> lfun(6) 34 >>> lfun.eval(3) 19 >>> lfun.eval(6) 34 >>> m.tabulateFunction(lfun, 0.0, 25.0, 4.0) Tabulating range: ; xmin = 0.0000 ; xmax = 25.0000 ; step = 4.0000 0.0000 4.0000 4.0000 24.0000 8.0000 44.0000 12.0000 64.0000 16.0000 84.0000 20.0000 104.0000 24.0000 124.0000 >>> >>> >>> a.tabulateFunction(lfun.eval, 0.0, 25.0, 4.0) Tabulating range: ; xmin = 0.0000 ; xmax = 25.0000 ; step = 4.0000 0.0000 4.0000 4.0000 24.0000 8.0000 44.0000 12.0000 64.0000 16.0000 84.0000 20.0000 104.0000 24.0000 124.0000 >>> lfun.a 5 >>> lfun.a = 0 >>> >>> m.tabulateFunction(lfun.eval, 0.0, 25.0, 4.0) Tabulating range: ; xmin = 0.0000 ; xmax = 25.0000 ; step = 4.0000 0.0000 4.0000 4.0000 4.0000 8.0000 4.0000 12.0000 4.0000 16.0000 4.0000 20.0000 4.0000 24.0000 4.0000
2.3 Native Modules or C++ binding with Pybind11
2.3.1 Overview
Pybind11 is a C++ header-only library that simplifies the creation of Python binding to existing C++ libraries or creating Python native modules. This library is easier and less cumbersome to use than Python raw C API due to the library declarative and intuitive syntax and lots of C++ wrappers.
Use cases:
- Create high-performance python modules in C++
- Create binding to existing C or C++ libraries.
- Embed Python in a C++ application
- Access to operating system APIs not exposed in Python standard library
Documentation:
Github Repository:
Conan package reference:
Headers:
- <pybind11/pybind11.h>
- Main header, that should be always included
- <pybind11/stl.h>
- Provides implicit convesion from STL containers to Python types.
- <pybind11/functional.h>
2.3.2 Example
- Overview
The following sample CMake/conan project contains many PyBind11 examples about binding existing C++ functions and classes and access or call Python objects and APIs from C++.
- File: CMakeLists.txt
Note: In order to use this CMakeLists.txt file, it is necessary an installation of conan package manager available in $PATH environment variable, in other words, accessible from command line.
# Author: Caio Rodrigues # Description: Sample CMake + Conan + Pybind11 project # Note: Requires Conan and Python Development files installed #---------------------------------------------------------- cmake_minimum_required(VERSION 2.8) project(pybind11-experiments) set(CMAKE_CXX_STANDARD 17) # ============ Conan Boostrap ======================# # Download automatically, you can also just copy the conan.cmake file if(NOT EXISTS "${CMAKE_BINARY_DIR}/conan.cmake") message(STATUS "Downloading conan.cmake from https://github.com/conan-io/cmake-conan") file(DOWNLOAD "https://github.com/conan-io/cmake-conan/raw/v0.13/conan.cmake" "${CMAKE_BINARY_DIR}/conan.cmake") endif() include(${CMAKE_BINARY_DIR}/conan.cmake) conan_cmake_run(REQUIRES pybind11/2.2.4@conan/stable BASIC_SETUP BUILD missing) #============ Additional Commands =============# # Copy target file to current directory whenerver it is rebuilt function(copy_after_build TARGET_NAME ) # Note: CMAKE_CURRENT_LIST_DIR is the directory where is this # CMakeLists.txt file. set(DESTDIR ${CMAKE_CURRENT_LIST_DIR}/bin/) file(MAKE_DIRECTORY ${DESTDIR}) # Copy binary file to <CMakeLists.txt didctory>./bin # after target is compiled. add_custom_command(TARGET ${TARGET_NAME} POST_BUILD COMMAND ${CMAKE_COMMAND} -E copy $<TARGET_FILE:${TARGET_NAME}> ${DESTDIR} ) endfunction() #----------------------------------------------# # Target Configurations # #----------------------------------------------# find_package(pybind11 CONFIG REQUIRED) pybind11_add_module(SampleModule SampleModule.cpp) # Copy binary to ./bin directory after module is built copy_after_build(SampleModule) # Target1: cashflow - Python module basics #..................................... pybind11_add_module(cashflow cashflow.cpp) copy_after_build(cashflow)
- File: SampleModule.cpp
- Full Source
- File: SampleModule.cpp
- Header and Namespaces
Header files:
#include <iostream> #include <string> #include <vector> #include <iomanip> #include <cmath> #include <pybind11/pybind11.h> #include <pybind11/stl.h> #include <pybind11/functional.h>
Namespaces:
namespace py = pybind11; using namespace pybind11::literals;
- Bindings C++ Functions, enums and Classes
Bindings to C++ functions, enums and classes without any knowledge of Python API.
- Function add:
int add(int i, int j) { return i + j; }
Registration:
PYBIND11_MODULE(SampleModule, m) { // optional module docstring m.doc() = "Sample Python built with C++ CeePlusPlus "; // Register an ordinary C++ function m.def("add", // Function Name in Python side &add, // Function Pointer "A function which adds two numbers", // Docstring py::arg("x"), // Name of argument 1 py::arg("y") = 10); // Name of argument 2 with default value ... ... ... ... ... ... ... ... ... ... }
- Function vectorCumulativeSum
std::vector<double> vectorCumulativeSum(std::vector<double> const& xs) { std::vector<double> out(xs.size()); double sum = 0.0; for(size_t i = 0; i < xs.size(); i++) { sum += xs[i]; out[i] = sum; } return out; }
Registration:
PYBIND11_MODULE(SampleModule, m) { ... ... ... ... ... m.def("vectorCumulativeSum", &vectorCumulativeSum); ... ... ... ... ... }
- Function tabulate1
void tabulate1(double start, double stop, double step, std::function<double (double)> func) { if(start > stop || step <= 0) throw std::runtime_error(" Error: Invalid domain " "\n Expected: start < stop && step > 0 " ); std::cout << std::setprecision(5) << std::fixed; for(double x = start; x <= stop; x+= step) std::cout << std::setw(10) << x << std::setw(10) << func(x) << std::endl; }
Registration:
PYBIND11_MODULE(SampleModule, m) { ... ... ... ... ... m.def("tabulate1", &tabulate1); ... ... ... ... ... }
- Enum class MathErrorCode:
enum class MathErrorCode: std::uint32_t { OK = 0x00 , OVERFLOW = 0x01 , UNDERFLOW = 0x04 , NOT_A_NUMBER = 0x08 };
Registration:
PYBIND11_MODULE(SampleModule, m) { ... ... ... ... ... py::enum_<MathErrorCode>(m, "MathErrorCode", py::arithmetic()) .value("OK", MathErrorCode::OK) .value("OVERFLOW", MathErrorCode::OVERFLOW) .value("UNDEFLOW", MathErrorCode::UNDERFLOW) .value("NOT_A_NUMBER", MathErrorCode::NOT_A_NUMBER); // .export_values(); ... ... ... ... ... }
- Class LinearFunctor:
// Sample "function-object class" class LinearFunctor { public: double A = 0, B = 0; LinearFunctor(); LinearFunctor(double a, double b): A(a), B(b){ } double GetA() const { return A; } void SetA(double a) { A = a; } double GetB() const { return B; } void SetB(double b) { B = b; } void show() const { std::cout << " LinearFunction: y(x) = A * x + B" << std::endl; std::cout << " => A = " << this->A << " ; B = " << this->B << std::endl; } std::string toString() const { std::stringstream ss; ss << " LinearFunction: y(x) = A * x + B" << std::endl; ss << " => A = " << this->A << " ; B = " << this->B << std::endl; return ss.str(); } // Function-call operator double operator()(double x) { return A * x + B; } };
Registration:
PYBIND11_MODULE(SampleModule, m) { ... ... ... ... ... ... ... . // Register LinearFunction py::class_<LinearFunctor>(m, "LinearFunctor") .def(py::init<double, double>()) // Register overloaded consructor .def("GetA", &LinearFunctor::GetA) // Reister method GetA() .def("GetB", &LinearFunctor::GetB) // Register method GetB() .def("SetA", &LinearFunctor::SetA) // Reister method GetA() .def("SetB", &LinearFunctor::SetB) // Register method GetB() .def("show", &LinearFunctor::show) // Register method show .def("call", &LinearFunctor::operator()) // Register function-call operator with name 'call' .def("__call__", &LinearFunctor::operator ()) // Register fun-call operator .def("__repr__", &LinearFunctor::toString) // Register strin representation .def_readwrite("A", &LinearFunctor::A) // Register field A .def_readwrite("B", &LinearFunctor::B); // Register field B ... ... }
- Functions with knowledge of Python API
- Function computeFormula
- This function takes a Python tuple py::args as argument and returns a Python dictionary py::dict
// Sample function that takes a python tuple as argument and // returns a dictionary. py::dict computeFormula(py::args args) { py::print(" [INFO] Number of args = {s1} ; args = {s2}", "s1"_a = args.size(), "s2"_a = args); std::cout << " [INFO] Arguments are = " << py::str(args) << std::endl; int n = py::cast<int>(args[0]); double x = py::cast<double>(args[1]); double y = py::cast<double>(args[2]); py::dict d{ "n"_a = n, "2x"_a = 2.0 * x, "addxy"_a = (x + y)}; d["m"] = 5.0 * x + 4.5 * y; py::print(" [INFO] Dictionary d = {d} ", "d"_a = d); return d; }
Registration:
m.def("computeFormula", &computeFormula);
- Function showDictionary
// Function that takes a Python dictionary as argument void showDictionary(py::dict map) { std::cout << " ====== Show dictonary =========== " << std::endl; for(auto const& item : map) std::cout << " key = " << item.first << " ; " << item.second << std::endl; }
Registration:
m.def("showDictionary", &showDictionary, "Display dictionary in console" );
- Function pyFuncArgsKwargs
- Sample function that takes a python tuple (py::args) and named arguments py::kwargs.
// Sample Python function taking a tuple and dictionary as argument void pyFuncArgsKwargs(py::args args, py::kwargs kwargs) { double x = py::cast<double>(args[0]); int n = py::cast<double>(args[1]); bool verbose = false; py::print(" [TRACE] args = ", args); py::print(" [TRACE] kwargs = ", kwargs); std::cout << std::boolalpha; if(kwargs.contains("verbose")) { verbose = py::cast<bool>(kwargs["verbose"]); } std::cout << " [INFO] verbose option = " << verbose << std::endl; std::cout << " sqrt(x) = " << std::sqrt(x) << " ; 2 * n = " << 2 * n << std::endl; }
Registration:
m.def("dictionaryArgument", &dictionaryArgument);
- Function: dictionaryArgument
// Sample function that takes a Python dictionary returning a // Python tuple as argument. py::tuple dictionaryArgument(py::dict dict) { std::cout << " >> Dictionary = " << py::str(dict) << std::endl; auto xx = dict["x"]; auto yy = dict["y"]; double x = py::cast<double>(xx); double y = py::cast<double>(yy); std::string name = py::cast<std::string>(dict["name"]); double z = std::sqrt(x * x + y * y); // Return a tuple on python side py::tuple tpl = py::make_tuple(x, y, z, name); py::print(" [INFO] Tuple = {} ", tpl); return tpl; }
Registration:
m.def("pyFuncArgsKwargs", &pyFuncArgsKwargs, "Sample Python function taking a tuple");
- Function: callPyObjectMethod
- This funtion takes any Python object as argument and prints its attributed .name and call its method named ".call". If the object does not have those fields and methods, a runtime error will happen.
// Call Python Objec's Method (Duck typing) void callPyObjectMethod(py::object obj, double k) { // Print objec's name attribute py::print(" [TRACE] obj.name = ", obj.attr("name")); std::string name = py::cast<std::string>(obj.attr("name")); std::cout << " [COUT] object.name = " << name << std::endl; // Invoke object's method .call auto call = obj.attr("call"); py::print(" [TRACE] obj.call(6.0) = ", call(6.0)); double x = py::cast<double>(call(k)); py::print(" [TRACE] 100.0 * objct.call(x) = ", 100.0 * x); }
Registration:
m.def("callPyObjectMethod", &callPyObjectMethod);
- Function computeFormula
- Module Registration of Functions and classes
PYBIND11_MODULE(SampleModule, m) { // optional module docstring m.doc() = "Sample Python built with C++ CeePlusPlus "; //----------- Register C++ Functions --------// // // Register an ordinary C++ function m.def("add", // Function Name in Python side &add, // Function Pointer "A function which adds two numbers", // Docstring py::arg("x"), // Name of argument 1 py::arg("y") = 10); // Name of argument 2 with default value m.def("cppLambda" ,[](double x, double y){ return 3.0 * x + y;} ,"A C++ lambda object or functor" //,py::arg("x"), py::args("y") = 15 ); m.def("vectorCumulativeSum", &vectorCumulativeSum); m.def("tabulate1", &tabulate1); ... ... ... ... ... ... ... ... ... ... }
- Full Source
- Compile and Running
- Compile
The project can be compiled from any IDE that supports CMake or from command line either by invoking CMake or a C++ compiler manually. All its need to build from an IDE is to open the CMakeList.txt.
Clone Repository:
$ cd /tmp $ git clone https://github.com/caiorss/example-project-pybind11 pybind11 $ cd pybind11 $ tree . . ├── bin │ └── test-script.py ├── cashflow.cpp ├── CMakeLists.txt └── SampleModule.cpp 1 directory, 4 files
Build from QTCreator IDE:
- Just open the file CMakeLists.txt and click at the hammer button or open this file from command line. Similar procedure can be used with other IDEs. After compiled, the Python modules are copied to directory ./bin.
$ qtcreator CMakeLists.txt &
Build from command line:
- Set configuration. Note: Debug build type has debugging symbols enabled and optmization disabled.
$ cmake -H. -Bbuild -DCMAKE_BUILD_TYPE=Debug -- Conan: Automatic detection of conan settings from cmake -- Conan: Settings= -s;build_type=Debug;-s;compiler=gcc;-s;compiler.version=7;-s;compiler.libcxx=libstdc++11 ... ... ... ... ... ... ... ... ... ... -- Build files have been written to: /tmp/pybind11/build
- Compile:
$ cmake --build build --target Scanning dependencies of target SampleModule [ 25%] Building CXX object CMakeFiles/SampleModule.dir/SampleModule.cpp.o [ 50%] Linking CXX shared module lib/SampleModule.cpython-37m-x86_64-linux-gnu.so [ 50%] Built target SampleModule ... ... ... ... ... ... ... ... ... ...
- The Cmake function copy_after_build copies the python native modules (shared libraries *.so files on Linux or *.pyd on Windows) to the directory ./bin
# Module files $ ls bin/ cashflow.cpython-37m-x86_64-linux-gnu.so* test-script.py SampleModule.cpython-37m-x86_64-linux-gnu.so* $ file bin/SampleModule.cpython-37m-x86_64-linux-gnu.so bin/SampleModule.cpython-37m-x86_64-linux-gnu.so: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, with debug_info, not stripped
- Running
Enter at directory where is the Python native module SampleModule, in this case, the file: SampleModule.cpython-37m-x86_64-linux-gnu.so
$ cd bin/ $ ls cashflow.cpython-37m-x86_64-linux-gnu.so* test-script.py SampleModule.cpython-37m-x86_64-linux-gnu.so* $ python3 Python 3.7.1 (default, Dec 14 2018, 19:28:38) [GCC 7.3.0] :: Anaconda, Inc. on linux Type "help", "copyright", "credits" or "license" for more information.
Import SampleModule from REPL:
>>> import SampleModule as t
Get module help:
>>> help(t) Help on module SampleModule: NAME SampleModule - Sample Python built with C++ CeePlusPlus CLASSES pybind11_builtins.pybind11_object(builtins.object) LinearFunctor MathErrorCode .... .... .... .... .... FUNCTIONS add(...) method of builtins.PyCapsule instance add(x: int, y: int=10) -> int A function which adds two numbers callPyObjectMethod(...) method of builtins.PyCapsule instance callPyObjectMethod(arg0: object, arg1: float) -> None computeFormula(...) method of builtins.PyCapsule instance computeFormula(*args) -> dict cppLambda(...) method of builtins.PyCapsule instance cppLambda(arg0: float, arg1: float) -> float A C++ lambda object or functor dictionaryArgument(...) method of builtins.PyCapsule instance ... ... ... ... ... ... ... ... ... ... ... ...
Test function and classes without knowledge of Python API
- Function add
>>> help(t.add) add(...) method of builtins.PyCapsule instance add(x: int, y: int=10) -> int A function which adds two numbers >>> t.add(20) 30 >>> t.add(100) 110 >>> t.add(15, 25) 40
- Function vectorCumulativeSum
>>> help(t.vectorCumulativeSum) vectorCumulativeSum(...) method of builtins.PyCapsule instance vectorCumulativeSum(arg0: List[float]) -> List[float] >>> t.vectorCumulativeSum([1, 10, 5, -8, 25, 10]) [1.0, 11.0, 16.0, 8.0, 33.0, 43.0]
- Function tabulate1
>>> t.tabulate1(0, 5, 1, lambda x: x * x - 5 * x + 8) 0.00000 8.00000 1.00000 4.00000 2.00000 2.00000 3.00000 2.00000 4.00000 4.00000 5.00000 8.00000 >>> import math >>> t.tabulate1(0, 5, 1, lambda x: math.exp(x)) 0.00000 1.00000 1.00000 2.71828 2.00000 7.38906 3.00000 20.08554 4.00000 54.59815 5.00000 148.41316 >>> t.tabulate1(0, -5, 1, lambda x: x * x - 5 * x + 8) Traceback (most recent call last): File "<stdin>", line 1, in <module> RuntimeError: Error: Invalid domain Expected: start < stop && step > 0
- Enum MathError
>>> status = t.MathErrorCode.OK >>> status == t.MathErrorCode.OK True >>> status = t.MathErrorCode.OVERFLOW >>> status == t.MathErrorCode.OK False >>> status == t.MathErrorCode.OVERFLOW True >>> int(t.MathErrorCode.OVERFLOW) 1 >>> int(t.MathErrorCode.NOT_A_NUMBER) 8
- Class LinearFunctor
>>> lfun = t.LinearFunctor(10, 5) >>> lfun LinearFunction: y(x) = A * x + B => A = 10 ; B = 5 >>> lfun.show() LinearFunction: y(x) = A * x + B => A = 10.00000 ; B = 5.00000 >>> >>> lfun(4) 45.0 >>> lfun(5) 55.0 >>> lfun(10) 105.0 >>> // Get coefficients //----------------------- >>> lfun.A 10.0 >>> lfun.B 5.0 >>> lfun.GetA() 10.0 >>> lfun.GetB() 5.0 >>> // Change coefficients //------------------- >>> lfun.SetA(-5) >>> lfun.SetB(15) >>> lfun LinearFunction: y(x) = A * x + B => A = -5 ; B = 15 >>> lfun.call(5.64) -13.2 >>> lfun(5.64) -13.2 >>>
Test function and classes with knowledge of Python API
- Function: computeFormula
>>> result = t.computeFormula(100, 5.67, 9.71) [INFO] Number of args = {s1} ; args = {s2} [INFO] Arguments are = (100, 5.67, 9.71) [INFO] Dictionary d = {d} >>> result {'n': 100, '2x': 11.34, 'addxy': 15.38, 'm': 72.04500000000002} >>> result["n"] 100 >>> result["2x"] 11.34
- Function: showDictionary
>>> t.showDictionary({'x': 10.34, 'y': "numerical electronic integrator", "name": "ARM-CPU"}) ====== Show dictonary =========== key = x ; 10.34 key = y ; numerical electronic integrator key = name ; ARM-CPU >>> >>> t.showDictionary(100) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: showDictionary(): incompatible function arguments. The following argument types are supported: 1. (arg0: dict) -> None Invoked with: 100
- Function: pyFuncArgsKwargs
>>> t.pyFuncArgsKwargs(2.7824, 200) [TRACE] args = (2.7824, 200) [TRACE] kwargs = {} [INFO] verbose option = false sqrt(x) = 1.66805 ; 2 * n = 400 >>> t.pyFuncArgsKwargs(2.7824, 200, verbose = True) [TRACE] args = (2.7824, 200) [TRACE] kwargs = {'verbose': True} [INFO] verbose option = true sqrt(x) = 1.66805 ; 2 * n = 400 >>> t.pyFuncArgsKwargs(2.7824, 200, verbose = False) [TRACE] args = (2.7824, 200) [TRACE] kwargs = {'verbose': False} [INFO] verbose option = false sqrt(x) = 1.66805 ; 2 * n = 400 >>> t.pyFuncArgsKwargs(2.7824, 200, verbose = "aaa") [TRACE] args = (2.7824, 200) [TRACE] kwargs = {'verbose': 'aaa'} Traceback (most recent call last): File "<stdin>", line 1, in <module> RuntimeError: Unable to cast Python instance of type <class 'str'> to C++ type 'bool'
- Function: dictionaryArgument
>>> result = t.dictionaryArgument({'x': 10.5, 'y': 5.6, 'name': "DUMMY"}); >> Dictionary = {'x': 10.5, 'y': 5.6, 'name': 'DUMMY'} [INFO] Tuple = {} (10.5, 5.6, 11.899999999999999, 'DUMMY') >>> result (10.5, 5.6, 11.899999999999999, 'DUMMY') >>> result[0] 10.5 >>> result[1] 5.6 >>> result[2] 11.899999999999999 >>>
- Function: callPyObjectMethod
class Dummy: def __init__(self, name): self.name = name def call(self, x): print(" I was called by C++.") return 4.5 * x + 10 >>> obj = Dummy("A dummy object") >>> t.callPyObjectMethod(obj, 4.5) [TRACE] obj.name = A dummy object [COUT] object.name = A dummy object I was called by C++. [TRACE] obj.call(6.0) = 37.0 I was called by C++. [TRACE] 100.0 * objct.call(x) = 3025.0 >>> t.callPyObjectMethod(obj, 10.0) [TRACE] obj.name = A dummy object [COUT] object.name = A dummy object I was called by C++. [TRACE] obj.call(6.0) = 37.0 I was called by C++. [TRACE] 100.0 * objct.call(x) = 5500.0
- Compile
3 R Language (RLang) Native Extension API
3.1 R Raw C API
3.1.1 Overview
R Language C data structures
Reference: http://adv-r.had.co.nz/C-interface.html
Data Structure | Description | |
---|---|---|
Vector | ||
REALSXP | Numeric Vector | |
INTSXP | Integer Vector | |
LGSXP | Logical Vector | |
STRSXP | Character Vector | |
CPLSXP | Complex Vectors | |
VECSXP | List | |
Null | ||
NILSXP | Null | |
Misc | ||
CLOSXP | Function (closure) | |
ENVSXP | Environment | |
LISTSXP | Pair list | |
DOTSXP | ||
SYMSXP | names/symbols | |
Internal Objects | Note: Only used by C functions, not R functions. | |
LANGSXP | Language constructs | |
CHARSXP | - | |
PROMSXP | - | |
EXPRSXP | Expressions |
Fuctions for coercing Scalars:
- asLogical(xs): LGSXP -> int
- asInteger(xs): INTSXP -> int
- asReal(xs): REALSXP -> double
- CHAR(asChar(x)): STRSXP -> const char*
R Types:
- All numbers are float point (double 64 bits)
- Scalar data are vectors, a single number is vector of size one.
Useful References:
- R Internals - C API Documentation
- R Native API Cheat Sheet
- Foreign function | R Documentation
- The Need for Speed Part 1: Building an R Package with Fortran (or C) | Strange Attractors
- R's C interface · Advanced R.
- Use of C++ in Packages - The R Blog
- Three ways to call C/C++ from R | R-bloggers
- Integrate C Language Functions into R With an R Package | NCEAS
3.1.2 Install R language and development tools
Install R development libraries with Anaconda Python Distribution
# Install R language, interpreter $ conda install r-base # Install R development libraries $ conda install r-devtools -c r
Locate where the headers:
$ find ~/opt/anaconda3/ -name "Rin*.h"
/home/dummy/opt/anaconda3/lib/R/include/Rinterface.h
/home/dummy/opt/anaconda3/lib/R/include/Rinternals.h
/home/dummy/opt/anaconda3/pkgs/r-base-3.5.1-h1e0a451_2/lib/R/include/Rinterface.h
/home/dummy/opt/anaconda3/pkgs/r-base-3.5.1-h1e0a451_2/lib/R/include/Rinternals.h
$ ls ~/opt/anaconda3/lib/R/include/
R_ext/ Rdefines.h R.h Rinternals.h Rversion.h
Rconfig.h Rembedded.h Rinterface.h Rmath.h S.h
3.1.3 Example using R native C API
- Source
File:
Compilation:
- Manual compilation
$ clang++ rnative.cpp -o rnative.so -std=c++1z -g -O0 -Wall -shared -fPIC -I/home/archbox/opt/anaconda3/lib/R/include/
- Automatic with R command line tool.
$ R CMD SHLIB rnative.cpp
Source Code
Headers:
#include <iostream> #include <iomanip> #include <cmath> #include <vector> #include <R.h> #include <Rinternals.h> #include <Rmath.h>
Global object for logging when the library is loaded or unloaded.
struct DLLIntializer{ DLLIntializer() { std::cerr << "[TRACE] - DLL loaded into process OK." << std::endl; } ~DLLIntializer() { std::cerr << "[TRACE] - DLL unloaded from process OK." << std::endl; } }; DLLIntializer init_hook{};
Function: basicFunction
- Computes the 10.0 * x + 4.0 * y where x and y are scalars (vectors of size one)
- Note 1: Every argument of a R-function is a SEXP (S-expression structure) that can be a vector, scalar, matrix, R function and so on.
- Note 2: Every scalar in R is a vector of size 1.
extern "C" SEXP basicFunction(SEXP x, SEXP y) { std::cerr << " [TRACE] Calling R from C++" << std::endl; // Allocate a vector of size 1 of type real (double) // Remember: in R, a scalar is a vector of size one SEXP out = PROTECT(allocVector(REALSXP, 1)); // asReal(x) => Interprets the first element of the // vector x as a double REAL(out)[0] = 10.0 * asReal(x) + 4.0 * asReal(y); // UNPROTECT(< Number of allocations> ) // OR: UNPROTECT(< Number of protects> ) UNPROTECT(1); return out; }
Function: GetVersion
- Returns library verion.
extern "C" SEXP GetVersion() { SEXP version; // Allocate 1 string PROTECT(version = allocVector(STRSXP, 1)); SET_STRING_ELT(version, 0, ::mkChar("version 0.1")); UNPROTECT(1); return version; }
Function: Metric
- Display vector statistics
extern "C" auto metrics(SEXP xs) -> SEXP { size_t size = ::length(xs); double* vec = REAL(xs); double sumsq = 0.0; double sum = 0.0; double x; for(auto i = 0; i < size; i++){ x = vec[i]; sumsq = sumsq + x * x; sum = sum + x; } std::cout << std::setprecision(4) << std::fixed; std::cout << std::endl; std::cout << " => Mean = " << sum / size << std::endl; std::cout << " => Norm = " << std::sqrt(sumsq) << std::endl; std::cout << " => Size = " << size << std::endl; // If the function returns nothing, it should return R_NilValue return R_NilValue; }
Function: linearcomb
- Return a linear combination between vectors.
/** Computes the equation: * For each i = 0, 1, 2, ... N - 1 * zs_vector[i] = a * xs_vector[i] + b * ys_vector[i] + c * * Returns: Vector zs_vector[i] */ extern "C" SEXP linearcomb(SEXP xs_vector, SEXP ys_vector, SEXP a, SEXP b, SEXP c) { size_t size = ::length(xs_vector); // Allocate output vector of same size SEXP output = PROTECT(allocVector(REALSXP, size)); double *pout, *pxs, *pys; pout = REAL(output); pxs = REAL(xs_vector); pys = REAL(ys_vector); for(size_t i = 0; i < size; i++) pout[i] = asReal(a) * pxs[i] + asReal(b) * pys[i] + asReal(c); UNPROTECT(1); return output; }
- Load into R REPL
Load shared library:
> dyn.load("rnative.so") [TRACE] - DLL loaded into process OK.
Call function basicFunction
> .Call("basicFunction", 10, 5) [TRACE] Calling R from C++ [1] 120
Call function metrics:
> .Call("metrics", 10) => Average = 10.0000 => Norm = 10.0000 => Size = 1 NULL > .Call("metrics", c(4.5, 8.0, 6, -10.0)) => Average = 2.1250 => Norm = 14.8408 => Size = 4 NULL
Call function linearcomb:
- Test function linearcomb which perform linear combination between two vectors:
> .Call("linearcomb", c(1, 2, 3, 4), c(4, 5, 6, 9), 10, 2, 0) [1] 18 30 42 58 > .Call("linearcomb", c(1, 2, 3, 4), c(4, 5, 6, 9), 10, 2, 5) [1] 23 35 47 63 > .Call("linearcomb", 4, 6, 10, 1, 2) [1] 48 > .Call("linearcomb", 4, 6, 10, 5, 2) [1] 72
End of the interative session:
> quit() Save workspace image? [y/n/c]: n [TRACE] - DLL unloaded from process OK.
Those functions need wrappers for making usage easier:
File: rnative.R
dyn.load("rnative.so") basicFunction <- function(x, y){ .Call("basicFunction", x, y) } getVersion <- function(){ .Call("GetVersion")} metrics <- function(xs){ .Call("metrics", xs) } # Compute: a * xs + b * ys + c linearcomb <- function(xs, ys, a, b, c){ .Call("linearcomb", xs, ys, a, b, c) } testPackage <- function(){ print(sprintf("basicFunction(4, 5) = %.3f", basicFunction(4, 5))) print(sprintf("basicFunction(10, 6) = %.3f", basicFunction(10, 6))) metrics(8) metrics(c(3.4, 9.0, 10.0, -15.75, 20.65)) }
This script can be loaded into the REPL with:
> source("rnative.R") [TRACE] - DLL loaded into process OK. > getVersion() [1] "version 0.1" > metrics(c(4, 5, 10.5, 8.65, 5)) => Mean = 6.6300 => Norm = 15.8453 => Size = 5 NULL > > quit() Save workspace image? [y/n/c]: n [TRACE] - DLL unloaded from process OK.
3.2 Rcpp Library for easier C++ Integration
3.2.1 Overview
Rcpp is a C++ library that makes easier to write native R functions in C++ and also to create bindings to existing C++ libraries.
Rcpp
Official Web Sites:
Doxygen - C++ API Docs:
Further Reading:
- Rcpp: Seamless R and C++ Integration [MUST READ]
- (Dirk Eddelbuettel and Romain François)
- Rcpp · Advanced R.
- 25 Rewriting R code in C++ | Advanced R
- Exposing C++ functions and classes with Rcpp modules
Lots of examples:
Repository:
Questions:
R Language Limitations
- GIL - Global Interpreter Lock of garbage collector
- Single thread
- Interpretation overhead
- Vectorized operations can cost many passes over the input data and be slower than a single loop.
Use Cases for Rcpp package and reasons to write R functions in C++
- Create R wrappers for existing C++ libraries.
- Create high performance R packages in C++
- Access operating system functionality not exposed in the language.
- Embed R interpreter into some application.
- R Functions or algorithms written in C++ can be reused with other programming languages or in production systems.
Other RCPP Libraries in CRAN:
Misc:
- Rcpp Modules => Rcpp extension wrapper library based on Boost.Python library which has a declarative and intituitive syntax for wrapping C++ libraries into Python.
- Rcpp11 => Library similar to Rcpp Modules and based on Python
Pybind11.
- Repository: https://github.com/Rcpp11/Rcpp11
- RProtBuf (Google's Protobuf)
Linear Algebra Libraries:
- Rcpp Armadillo => Wraps C++ armadillo library linear algebra
- Rcpp Eigen => (Eigen Library)
General Scientific Libraries
- RcppGSL => (GNU Scientific Library Wrapper)
Install Rcpp Package
> install.packages(c("Rcpp", "RInside", "inline"))
RCPP Available Functions
Note: The RCPP convenience functions are vectorized, therefore they can operate on numeric vector or matrices without explicit loops.
Convenience Functions in | Description | |
---|---|---|
Rcpp namespace | ||
Fundamental Transcendatal Function | ||
Rcpp::exp | Exponential e^x | |
Rcpp::log | Natural logarithm base e = 2.718282 | |
Rcpp::log1p | Natural logarithm, however, log(1 + x), where x is a small number | |
Rcpp::log10 | Logarith base 10 | |
Rcpp::sqrt | Square root | |
Statistcs | ||
Rcpp::sd | Standard Deviation | |
Rcpp::mean | Mean, or average | |
Rcpp::sum | Sum of input values | |
Rcpp::cumsum | Cummulative sum | |
Rcpp::diff | ||
Rcpp::pmin | Return paralallel maxima of the input values | |
Rcpp::pmax | Return paralallel minima of the input values | |
….. and more … |
RCPP Classes
RCPP Class | R type typeof(<OBJ>) | |
---|---|---|
Vector | ||
Rcpp::IntegerVector | integer | |
Rcpp::NumericVector | numeric | |
Rcpp::LogicalVector | logical | |
Rcpp::CharacterVector | character | |
Rcpp::RawVector | raw… | |
Rcpp::ComplexVector | complex… | |
Matrix | ||
Rcpp::IntegerMatrix | ||
Rcpp::NumericMatrix | ||
Rcpp::LogicalMatrix | ||
Rcpp::RawMatrix | ||
Rcpp::ComplexMatrix | ||
Other Types | ||
Rcpp::Environment | environment | |
Rcpp::Function | function (closure) | |
Rcpp::XPtr | externalptr | |
Rcpp::Language | Language |
3.2.2 Example - Native R library in C++ with Rcpp
- Source Files
Source Files
- File: rcxx.cpp
- File: CMakeLists.txt
Compilation
Manual:
$ clang++ rcxx.cpp -o rcxx.so -std=c++1z -g -O0 -Wall -shared -fPIC \ -I/home/archbox/opt/anaconda3/lib/R/include/ \ -I/home/archbox/opt/anaconda3/lib/R/library/Rcpp/include/
Build with CMake:
$ cmake -Bbuild -H. $ cmake --build build --target Scanning dependencies of target rcxx [ 50%] Building CXX object CMakeFiles/rcxx.dir/rcxx.cpp.o [100%] Linking CXX shared library rcxx.so [100%] Built target r $ ls build/ CMakeFiles/ CMakeCache.txt cmake_install.cmake Makefile rcxx.so* $ cp build/rcxx.so .
File: rcxx.cpp
Headers:
#include <iostream> #include <cmath> #include <iomanip> #include <algorithm> #include <string> #include <map> #include <Rcpp.h>
Global object for logging when the DLL is loaded or unloaded.
struct DLLIntializer{ DLLIntializer() { std::cerr << "[TRACE] - DLL loaded into process OK." << std::endl; } ~DLLIntializer() { std::cerr << "[TRACE] - DLL unloaded from process OK." << std::endl; } }; // Global object allocated in process' data segment for // DLL logging during loading and unloading. DLLIntializer init_hook{};
Function libraryVersion
- Returns the library version as string.
RcppExport auto libraryVersion() -> SEXP { return Rcpp::wrap("version 1.0 - alpha"); }
Function linearcomb
- Performs the transformation:
- ys[i] = a * xs[i] + b for every i = 0, 1, 2 … N - 1
/** Compute equation: * for_each(i) => ys[i] = a * xs[i] + b * return ys */ RcppExport SEXP linearcomb(SEXP a, SEXP b, SEXP xs) { Rcpp::NumericVector vx(xs); double aa = Rcpp::as<double>(a); double bb = Rcpp::as<double>(b); Rcpp::NumericVector out(vx.size()); size_t n = vx.size(); // std::cerr << " [TRACE] Size of xs = " << n << std::endl; for(auto i = 0; i < vx.size(); i++) out[i] = aa * vx[i] + bb; return out; }
Function: computeStatistics
RcppExport auto computeStatistics(SEXP xs) -> SEXP { Rcpp::NumericVector vx{xs}; double sum = 0.0; double sumsq = 0.0; std::for_each(vx.begin(), vx.end(), [&](double x) { sum += x; sumsq += x * x; }); auto result = std::map<std::string, double> { {"sum", sum}, {"sumsq", sumsq}, {"mean", sum / vx.size()}, {"norm", std::sqrt(sumsq)} }; return Rcpp::wrap(result); }
Function: computeStatistics2:
RcppExport auto computeStatistics2(SEXP xs) -> SEXP { Rcpp::NumericVector vx{xs}; double sum = 0.0; double sumsq = 0.0; std::for_each(vx.begin(), vx.end(), [&](double x) { sum += x; sumsq += x * x; }); return Rcpp::List::create( Rcpp::Named("sum", sum) ,Rcpp::Named("sumsq", sumsq) ,Rcpp::Named("mean", sum / vx.size()) ,Rcpp::Named("norm", std::sqrt(sumsq)) ); }
Function tabulate:
RcppExport SEXP tabulate(SEXP mfunc, SEXP mvec) { Rcpp::Function func = mfunc; Rcpp::NumericVector vec = mvec; std::cout << std::setprecision(4) << std::fixed; for(auto const& x: vec) std::cout << std::setw(10) << x << std::setw(10) << Rcpp::as<double>(func(x)) << std::endl; // Return nothing on R side return R_NilValue; }
Function: vectorOperations
- This function performs arithmetic operation on two input R numeric vectors.
/* Compute: * * FOR i = 0 TO N DO: * out[i] = sqrt(va[i] * 4.0 + vb[i] * 5.0 * RETURN ouy * * Note: The vectorized operation are not efficient * Rcpp::sqrt(4.0 * va + 5.0 * vb) is equivaent to: * Rcpp::sqrt(operatior*(4.0, va) + operator*(5.0, vb)) * Rcpp::sqrt(operator+(operatior*(4.0, va), operator*(5.0, vb))) * * Every operator function call is 1 loop, so, the total number * of passes over the input data is equal to: * => 1 (SQRT Call) + 3 (Number of operators) = 4 * * This function requires 4 for-loops over the data, * thus a single for-loop is more efficient. */ RcppExport SEXP vectorOperations(SEXP sa, SEXP sb) { using Vector = Rcpp::NumericVector; Vector va = sa; Vector vb = sb; Vector result = Rcpp::sqrt(4.0 * va + 5.0 * vb); return result; }
Function: ShowNormalRandoms
- This code demonstrates how to call R functions such as rnorm from C++ side. It takes as parameter the number of normally distributed random numbers that will be generated with mean 20.0 and standard deviation 5.0
RcppExport SEXP ShowNormalRandoms(SEXP s_size) { size_t n = Rcpp::as<size_t>(s_size); std::cout << " [INFO] size = " << n << std::endl; // Get R function rnorm for generating N normally distributed // random numbers Rcpp::Function rnorm("rnorm"); // Call R built-in function rnorm // that generates n normally distributed random numbers with // mean 20.0 and standard deviation std = 5.0 //--------------------------------------------- // NOTE: Every R fuction returns an S-expression struct SEXP sexp = rnorm(Rcpp::Named("n", n), Rcpp::Named("sd", 5.0), Rcpp::Named("mean", 20.0) ); // Convert s-expression to numeric vector Rcpp::NumericVector result{sexp}; std::cout << std::setprecision(5) << std::fixed; size_t k = 0; std::for_each(result.begin(), result.end(), [&](auto x) -> void { std::cout << std::setw(10) << k++ << std::setw(10) << x << std::endl; }); // std::for_each(result.begin()) return R_NilValue; }
- Loading into R REPL
Interactive Session:
> library(Rcpp) > # Load shared library into current REPL > dyn.load("rcxx.so") [TRACE] - DLL loaded into process OK.
Call function libraryVersion:
> .Call("libraryVersion") [1] "version 1.0 - alpha" > # Better > .Call("libraryVersion", PACKAGE = "rcxx") [1] "version 1.0 - alpha" > .Call("libraryVersion", PACKAGE = "rm") Error in .Call("libraryVersion", PACKAGE = "rm") : "libraryVersion" not available for .Call() for package "rm"
Call function linearcomb
> # Call function from shared library > .Call("linearcomb", 10.0, 4.0, c(4.5, 9.8, 5.6, 10.0), PACKAGE = "rcxx") [1] 49 102 60 104 > .Call("linearcomb", 10.0, 0, c(4.5, 9.8, 5.6, 10.0), PACKAGE = "rcxx") [1] 45 98 56 100
Call function computeStatistics
> .Call("computeStatistics", c(4.5, 3.5, 9.81, 4.6)) mean norm sum sumsq 5.60250 12.24321 22.41000 149.89610 > > out = .Call("computeStatistics", c(4.5, 3.5, 9.81, 4.6), PACKAGE = "rcxx") > out mean norm sum sumsq 5.60250 12.24321 22.41000 149.89610 > out["mean"] mean 5.6025 > out["norm"] norm 12.24321 > out["sum"] sum 22.41
Call function computeStatistics2
> res = .Call("computeStatistics2", c(4.5, 3.5, 9.81, 4.6), PACAKGE = "rcxx") > res $sum [1] 22.41 $sumsq [1] 149.8961 $mean [1] 5.6025 $norm [1] 12.24321 > res$sum [1] 22.41 > res$mean [1] 5.6025 > > 100 * res$mean [1] 560.25
Call function tabulate
> .Call("tabulate", sqrt, c(4.5, 3.5, 9.81, 4.6)) 4.5000 2.1213 3.5000 1.8708 9.8100 3.1321 4.6000 2.1448 NULL > .Call("tabulate", function(x){ 10 * x + 5 }, c(4.5, 3.5, 9.81, 4.6)) 4.5000 50.0000 3.5000 40.0000 9.8100 103.1000 4.6000 51.0000 NULL >
Call function vectorOperations
> .Call("vectorOperations", c(4.0, 3.0, 2.0, 6.0), c(10.0, 5, 8, 4)) [1] 8.124038 6.082763 6.928203 6.633250 > xs = .Call("vectorOperations", c(4.0, 3.0, 2.0, 6.0), c(10.0, 5, 8, 4)) > xs [1] 8.124038 6.082763 6.928203 6.633250 > xs * xs [1] 66 37 48 44 > sqrt(4.0 * 4.0 + 5.0 * 10.0) [1] 8.124038 >
Call function ShowNormalRandoms
- This function invokes the R built-in function rnorm from the C++ side and displays the result on the screen.
> .Call("ShowNormalRandoms", 5, PACKAGE = "rcxx") [INFO] size = 5 0 15.61839 1 25.08418 2 22.76518 3 17.96007 4 17.60298 NULL > .Call("ShowNormalRandoms", 10, PACKAGE = "rcxx") [INFO] size = 10 0 17.92591 1 16.47821 2 24.35774 3 19.33816 4 22.70113 5 17.37169 6 14.92140 7 11.94084 8 14.29367 9 12.16308 NULL
3.2.3 References
- Unofficial Rcpp API Documentation - The Coatless Professor
- Cran - RCPP - Rcpp: Seamless R and C++ Integration
- Roman François. Integrating R with C++: Rcpp, RInside and RProtoBuf
- Roman François. R / C++
- Hyun Min Kang - Biostatistics 615/815 Lecture 12 - Interfacing C++ and R