CPP / C++ Notes - C++ Tooling

Table of Contents

1 Tooling

1.1 Online Tools

1.1.1 Compilers   compiler

  • https://coliru.stacked-crooked.com/
    • The only online compiler which supports C++17 and the experimental C++20.
  • http://rextester.com/
    • The main selling point of this online tool is that it allows to save code snippets and run them again.
    • Online C++ compiler, provides GCC, CLANG and Visual Studio Compiler, also known as MSVC (VC++ - Visual C++).
    • Supports Boost libraries.
  • https://repl.it/languages/cpp11
    • Oline compiler (GCC / C++11 only) with REPL support.
  • https://wandbox.org/ (Best)
    • Supports C++11, C++14, C++17 and C++20a (draft).
    • Supports Boost libraries.
    • Supported Languages other C++:
      • Non-managed languages: C, Rust, D, Go C#, CMake, Nim, Haskell, Nim
      • Managed languages: C#, Java, R, Scala, Swift, PHP, Ruby, SQL and so on.

1.1.2 Compiler Explorers   compiler

  • https://godbolt.org/ [BEST]
    • Compiler explorer - allows to explore the assembly generated by a wide variety of compilers, including, GNU GCC, Clang, MSVC and so on.
  • https://www.onlinegdb.com [BEST]
    • Online GNU Debugger
  • https://cppinsights.io
    • "C++ Insights is a clang-based tool which does a source to source transformation. Its goal is to make things visible which normally, and intentionally, happen behind the scenes. It's about the magic the compiler does for us to make things work. Or looking through the classes of a compiler. You can see, for example the transformation of a lamda, range-based for-loop or auto. Of course, you can transform any other C++ snippet. "
  • https://macro.virhex.com/
    • Macro expander tool
  • https://demangler.com/
    • Symbol demangler for MSVC, GCC and Clang.
  • http://quick-bench.com
    • Tool for micro benchamrks
  • http://metashell.org/index.html
    • Tool for visualizing template expansion

1.1.3 Web Assembly

1.1.4 Assembly

1.1.5 Emulators

  • ARMv7 DEC1-Soc CPU Emulator
    • https://cpulator.01xz.net/?sys=arm-de1soc
    • https://cpulator.01xz.net/doc/
    • "CPUlator is a simulator and debugger of a computer system (CPU, memory, and I/O devices) that runs inside a web browser. Simulation allows running and debugging programs without needing to use a hardware board. When you load a program onto the simulator, the program will behave (almost) identically to the same program running on real hardware. The simulated systems are based on the computer systems from the Altera University Program (Nios II and ARMv7) or SPIM (MIPS). A simulator has several advantages over hardware. It is more convenient and easily accessible than a hardware board. Also, compared to hardware, a simulator has greater visibility into the system state and can warn about suspicious behaviours, such as self-modifying code or violating calling conventions, making debugging easier."
  • Unicorn.js - The Unicorn emulator framework, now available for JavaScript.
    • https://alexaltea.github.io/unicorn.js/
    • https://news.ycombinator.com/item?id=13570281
    • "Unicorn.js is a port of the Unicorn emulator framework for JavaScript, done with Emscripten. It's released as a 19 MB JavaScript file supporting the architectures: ARM, ARM64, M68K, MIPS, SPARC, and x86. Alternatively, per-platform Unicorn.js releases are also available here. Follow the Readme to build Unicorn.js manually. Unicorn is a lightweight multi-architecture CPU emulator framework originally developed by Nguyen Anh Quynh et al. and released under GPLv2."

1.1.6 Misc Tools

Regex Testers

BNF Grammar Testing Tools for Building Parsers

UML Ascii/Text Drawing Tools

1.1.7 Database SQL RDBMS

SQL Simulator / Playground

Tools for creating database schema

Tools for translating SQL Dialects

1.1.8 Latex - Equations and Math Formulas

1.2 Compilers and toolchains

1.2.1 MINGW/GCC and Clang

Clang / LLVM

Windows / GCC-MingW

  • MinGW Compiler [Official Web Site]
  • mingw-w64 - GCC for Windows 64 & 32 bits [Download]
  • Mingw - newen.net Distribution (GCC Ported for Windows)
    • "My MinGW distribution ("distro") is x64-native and currently contains GCC 9.2.0 and Boost 1.71.0. - The components of this distro fall into four categories: Essentials: The components of MinGW itself. They are required in order to run GCC; Libraries: Useful C and C++ libraries; Utilities: Programs used by programmers; Utilities (Binary): More programs used by programmers. Unlike everything else, I obtained these executables from their official websites instead of building them myself."

1.2.2 Web - WebAssembly / WASM / asm.js

  • Emscripten - Emscripten: An LLVM-to-Web Compiler
    • https://github.com/emscripten-core/emscripten
    • "Emscripten compiles C and C++ to WebAssembly using LLVM and Binaryen. Emscripten output can run on the Web, in Node.js, and in wasm runtimes. Emscripten provides Web support for popular portable APIs such as OpenGL and SDL2, allowing complex graphical native applications to be ported, such as the Unity game engine and Google Earth. It can probably port your codebase, too! While Emscripten mostly focuses on compiling C and C++ using Clang, it can be integrated with other LLVM-using compilers (for example, Rust has Emscripten integration, with the wasm32-unknown-emscripten and asmjs-unknown-emscripten targets)."
  • Emcsripten SDK
  • Cheerp - Cheerp - a C/C++ compiler for Web applications
    • https://leaningtech.com/cheerp/
    • "Cheerp is a C/C++ compiler for the Web, based and integrated into the LLVM/clang infrastructure, and featuring numerous custom optimisation steps to maximise performance and minimise size of the compiled JavaScript/WebAssembly output. As such, Cheerp is the best performing, most optimised C++ to WebAssembly compiler available on the market Cheerp is used mainly to port existing C/C++ libraries and applications to HTML5, but can also be used to write high-performance Web applications and components from scratch. C/C++ gets optimised and compiled into JavaScript/WebAssembly, and can easily be deployed as part of a web page."

1.2.3 Mobile Devices / Android

Android (Cross compiler for Linux running on ARM or MIPS)

Side Note: Android is "Linux" with BSD-license LibC and user-space utilities provided by toybox.

1.2.4 Embedded Systems / Microcontrollers / SBC

1.2.5 Embedded Systems / Pre-built GCC Toolchains

1.3 General

1.4 Static code analysers   quality qa bug

Tools for performing static analysis in the code and catching bugs easily go unnoticed.

  • Cppcheck
    • "Cppcheck is a static analysis tool for C/C++ code. It provides unique code analysis to detect bugs and focuses on detecting undefined behaviour and dangerous coding constructs. The goal is to detect only real errors in the code (i.e. have very few false positives)."
    • Web Site: http://cppcheck.sourceforge.net/
  • Clang Analyser
    • "The Clang Static Analyzer is a source code analysis tool that finds bugs in C, C++, and Objective-C programs. Currently it can be run either as a standalone tool or within Xcode. The standalone tool is invoked from the command line, and is intended to be run in tandem with a build of a codebase. The analyzer is 100% open source and is part of the Clang project. Like the rest of Clang, the analyzer is implemented as a C++ library that can be used by other tools and applications."
    • Web Site: https://clang-analyzer.llvm.org
  • Splint - Annotation-Assisted Lightweight Static Checking
    • "Splint is a tool for statically checking C programs for security vulnerabilities and coding mistakes. With minimal effort, Splint can be used as a better lint. If additional effort is invested adding annotations to programs, Splint can perform stronger checking than can be done by any standard lint."
    • Web Site: https://splint.org
  • OCLint
    • "OCLint is a static code analysis tool for improving quality and reducing defects by inspecting C, C++ and Objective-C code and looking for potential problems like: Possible bugs - empty if/else/try/catch/finally statements; Unused code - unused local variables and parameters; Complicated code - high cyclomatic complexity, NPath complexity and high NCSS; Redundant code - redundant if statement and useless parentheses; Code smells - long method and long parameter list; Bad practices - inverted logic and parameter reassignment"
  • PVS-Studio [PROFESSIONAL TOOL]

1.5 Sanitizers

  • google/sanitizers
    • "This project is the home for Sanitizers: AddressSanitizer, MemorySanitizer, ThreadSanitizer, LeakSanitizer, and more The actual code resides in the LLVM repository. Here we keep extended documentation, bugs and some helper code."
  • AddressSanitizer - AddressSanitizer (aka ASan) is a memory error detector for C/C++.
  • MemorySanitizer
  • HWASAN
  • UBSAN - Undefined Behavior Sanitizer

See also:

1.6 Debuggers

1.6.1 General

  • rr-project - Reverse Debugging
    • "rr aspires to be your primary C/C++ debugging tool for Linux, replacing — well, enhancing — gdb. You record a failure once, then debug the recording, deterministically, as many times as you want. The same execution is replayed every time. rr also provides efficient reverse execution under gdb. Set breakpoints and data watchpoints and quickly reverse-execute to where they were hit. rr works on real applications and is used by many developers to fix real bugs. It makes debugging hard bugs much easier, but also speeds up debugging of easy bugs."
    • rr features:
      • Low overhead compared to other similar tools, especially on mostly-single-threaded workloads
      • Supports recording and replay of all kinds of applications: Firefox, Chrome, QEMU, LibreOffice, Go programs, …
      • Record, replay and debug multiple-process workloads, including entire containers
      • Works with gdb scripting and IDE integration
      • Durable, compact traces that can be ported between machines
      • Chaos mode to make intermittent bugs more reproducible

1.6.2 User interfaces for GDB or LLDB

1.6.3 Memory Leak Debuggers   quality qa bug

See also:

1.7 Profiling Tools - Performance Measurement Tools   qa quality speed perf

1.8 Code Navigation and Exploration Tools

Tools for navigating, exploring and undestanding large and unfamiliar code bases.

  • Doxygen [FREE]
    • Doxygen can generate an html cross reference of an unfamiliar code base containing embedded code which allows navigating and exploring the code in any web browser. Disadvantage: harder to use than proper tools built for this purpose. Another issue is that the code must be properly annotated with Doxygen markup comments.
  • cscope [FREE]
  • codequery [FREE]
    • "This is a tool to index, then query or search C, C++, Java, Python, Ruby, Go and Javascript source code. It builds upon the databases of cscope and Exuberant ctags. It can also work with Universal ctags, which is a drop-in replacement for Exuberant ctags. The databases of cscope and ctags would be processed by the cqmakedb tool to generate the CodeQuery database file. The CodeQuery database file can be viewed and queried using the codequery GUI tool."
    • Features: "Combines the best of both cscope and ctags; Faster due to the use of sqlite for the CodeQuery database; Cross-platform GUI tool; Fast auto-completion of search term; Case-insensitive, partial keyword search - wildcard search supported * and ?; Exact match search; Filter search results by file path; File viewer with syntax highlighting, for UTF-8 encoded source files; Ability to open viewed file in an external editor or IDE; Visualization of function call graph and class inheritance based on search term; Visualization graphs can be saved to PNG or Graphviz DOT files"
  • Source Trail [NOW OPEN SOURCE] [BEST]
  • CPPDepend [COMMERCIAL]
    • Features: Allows visual navigation and querying the code with an SQL-like syntax.

Tools for code indexing and cross-reference database for text editors:

  • clang-ctags
    • "Generate tag file for C++ source code, using the clang C++ compiler's parsing libraries. Unlike other ctags implementations, clang-ctags uses a real C++ compiler (clang) to parse source files, allowing for more accurate indexing. (C++ is notoriously difficult to parse, and other ctags implementations rely on heuristics to disambiguate certain constructs.) Unlike other implementations, clang-ctags only understands C and C++ source files; and because clang-ctags needs to run each source file through the C pre-processor, its usage is somewhat more complicated than other ctags implementations."
  • rtags
    • "RTags is a client/server application that indexes C/C++ code and keeps a persistent file-based database of references, declarations, definitions, symbolnames etc. There’s also limited support for ObjC/ObjC++. It allows you to find symbols by name (including nested class and namespace scope). Most importantly we give you proper follow-symbol and find-references support. We also have neat little things like rename-symbol, integration with clang’s “fixits” (https://clang.llvm.org/diagnostics.html). We also integrate with flymake using clang’s vastly superior errors and warnings. Since RTags constantly will reindex “dirty” files you get live updates of compiler errors and warnings. Since we already know how to compile your sources we have a way to quickly bring up the preprocessed output of the current source file in a buffer."
  • GNU Global
    • "GNU GLOBAL is a source code tagging system that works the same way across diverse environments, such as Emacs editor, Vi editor, Less viewer, Bash shell, various web browsers, etc. You can locate various objects, such as functions, macros, structs, classes, in your source files and move there easily. It is useful for hacking on large projects which contain many sub-directories, many #ifdef and many main() functions. It is similar to ctags or etags, but is different from them in the following two points: independence of any editor capability to treat definition and reference It runs in UNIX (POSIX) compatible operating system, like GNU and BSD."

1.9 Compilation Database Tools

  • BEAR - *Build EAR
  • CompileDB
    • https://github.com/nickdiego/compiledb
    • "Tool for generating Clang's JSON Compilation Database file for GNU make-based build systems. It's aimed mainly at non-cmake (cmake already generates compilation database) large codebases. Inspired by projects like YCM-Generator and Bear, but faster (mainly with large projects), since in most cases it doesn't need a clean build (as the mentioned tools do) to generate the compilation database file, to achieve this it uses the make options such as -n/–dry-run and -k/–keep-going to extract the compile commands. Also, it's more cross-compiling friendly than YCM-generator's fake-toolchanin approach."

1.10 Clang/LLVM tools

  • LLDB Debugger
    • "LLDB is a next generation, high-performance debugger. It is built as a set of reusable components which highly leverage existing libraries in the larger LLVM Project, such as the Clang expression parser and LLVM disassembler. LLDB is the default debugger in Xcode on macOS and supports debugging C, Objective-C and C++ on the desktop and iOS devices and simulator."
  • clang-tidy - Automating the formatting of C or C++ code according to some style.
  • clang-query
  • clang-refactor - Clang Refactoring engine
    • "This document describes the design of Clang’s refactoring engine and provides a couple of examples that show how various primitives in the refactoring API can be used to implement different refactoring actions. The LibTooling library provides several other APIs that are used when developing a refactoring action. Refactoring engine can be used to implement local refactorings that are initiated using a selection in an editor or an IDE. You can combine AST matchers and the refactoring engine to implement refactorings that don’t lend themselves well to source selection and/or have to query ASTs for some particular nodes."
  • clang-rename
    • "clang-rename is a C++ refactoring tool. Its purpose is to perform efficient renaming actions in large-scale projects such as renaming classes, functions, variables, arguments, namespaces etc. The tool is in a very early development stage, so you might encounter bugs and crashes. Submitting reports with information about how to reproduce the issue to the LLVM bugtracker will definitely help the project. If you have any ideas or suggestions, you might want to put a feature request there."

1.11 Documentation Generator

1.12 Unix/POSIX and Linux Development

1.12.1 Manpages offline doc

  • jeaye/stdman - Formatted C++20 stdlib man pages (cppreference).
    • stdman is a tool that parses archived HTML files from cppreference and generates groff-formatted manual pages for Unix-based systems. The goal is to provide excellent formatting for easy readability. stdman has been tested on Linux and OS X.
  • tldr - Simplified and community-driven man pages.
  • sinclairtarget/um - "um is a command-line utility for creating and maintaining your own set of man-like help pages. It is available for MacOS (via Homebrew) and Linux (via AUR in Arch, otherwise via Linuxbrew)."

1.12.2 GNU Tools

The following tools are not only useful for debugging, diagnosing and assisting development, they are also useful for reverse engineering.

Note:

  • object files: executables and shared libraries (*.so files), generally with ELF format.
Tool / Command Description
file identify file type by its header bytes
nm list symbols exported by executables
c++filt demangles C++ symbols
objdump show detailed information about object files
readlef show information about ELF object files.
strings show human readable characters in the executable
strip discard symbols
ptrace and strace trace system calls
gdb GNU Debugger
lsof Check which files a process is using
Directory /etc/proc/<pid> Allows to check which files and resources a process is using
   

Command Line Hexadecimal Visualizer

  • hexdump
  • od

GNU Tools

1.12.3 Unix self-extracting installers

  • makeself - "makeself - Make self-extractable archives on Unix. makeself.sh is a small shell script that generates a self-extractable compressed tar archive from a directory. The resulting file appears as a shell script (many of those have a .run suffix), and can be launched as is. The archive will then uncompress itself to a temporary directory and an optional arbitrary command will be executed (for example an installation script). This is pretty similar to archives generated with WinZip Self-Extractor in the Windows world. Makeself archives also include checksums for integrity self-validation (CRC and/or MD5/SHA256 checksums)."

1.12.4 Mac OSX Only

  • auriamg/macdylibbundler
    • "Utility to ease bundling libraries into executables for OSX. dylibbundler is a small command-line programs that aims to make bundling .dylibs as easy as possible. It automatically determines which dylibs are needed by your program, copies these libraries inside the app bundle, and fixes both them and the executable to be ready for distribution… all this with a single command on the teminal! It will also work if your program uses plug-ins that have dependencies too."

1.12.5 Other tools

  • shmcat
    • This is a simple tool that dumps shared memory segments (System V and POSIX), files and text. It might be useful when you have to debug programs that use shared memory. Dumps System V (shmget, shmat API) and POSIX (shm_open…) shared memory.

1.13 Tools for Making Portable Linux Binaries

  • AppImageKit
    • "Using AppImageKit you can package desktop applications as AppImages that run on common Linux-based operating systems, such as RHEL, CentOS, Ubuntu, Fedora, Debian and derivatives. Package desktop applications as AppImages that run on common Linux-based operating systems, such as RHEL, CentOS, openSUSE, SLED, Ubuntu, Fedora, debian and derivatives. Join #AppImage on irc.freenode.net"
  • linuxdeployqt - QT C++ Applications
    • Makes Linux applications self-contained by copying in the libraries and plugins that the application uses, and optionally generates an AppImage. Can be used for Qt and other applications
    • "This Linux Deployment Tool, linuxdeployqt, takes an application as input and makes it self-contained by copying in the resources that the application uses (like libraries, graphics, and plugins) into a bundle. The resulting bundle can be distributed as an AppDir or as an AppImage to users, or can be put into cross-distribution packages. It can be used as part of the build process to deploy applications written in C, C++, and other compiled languages with systems like CMake, qmake, and make. When used on Qt-based applications, it can bundle a specific minimal subset of Qt required to run the application."
  • linuxdeploy
    • "appimagetool, the tool creating AppImages, expects directories in a specific format that will then be converted into the final AppImage. This format is called AppDir. It is not very difficult to understand, but creating AppDirs for arbitrary applications tends to be a very repetitive task. Also, bundling all the dependencies properly can be a quite difficult task. It seems like there is a need for tools which simplify these tasks. linuxdeploy is designed to be an AppDir maintenance tool. It provides extensive functionalities to create and bundle AppDirs for applications. It features a plugin system that allows for easy bundling of frameworks and creating output bundles such as AppImages with little effort."
  • Holy Build Box - For command line apps and servers.
    • Note: It only works Linux x86 and x64 and does not work with graphical applications with GTk, Qt, SDL or OpenGL. It is only intended for headless applications such as servers or daemons.
    • "Holy Build Box is a system for building "portable" binaries for Linux: binaries that work on pretty much any Linux distribution. This works by providing an easy-to-use, isolated, tightly-controlled compilation environment that is designed for producing portable binaries. Holy Build Box can produce x86 and x86-64 binaries."
    • Holy Build Box is made for developers that fit the following criteria:
      • You want to distribute binaries for your C/C++ applications to end users.
      • There are many people in your userbase with low to medium system administration skill, and you want your users to be able to easily use your applications, so you want to avoid asking them to compile from source.
      • You are wary of having to invest a disproportionate amount of time into building platform-specific packages. You want something that works for the majority of your users, without having to invest too much time.
      • You want your users to be easily able to use the latest versions of your software, so waiting for distributions to package your application is not an option.
  • steam-runtime - For games (developed by Valve company)
    • Note: It creates a chroot for custom compilation.
    • "A binary compatible runtime environment for Steam applications on Linux. This release of the steam-runtime SDK marks a change to a chroot environment used for building apps. A chroot environment is a standalone Linux environment rooted somewhere in your file system. All processes that run within the root run relative to that rooted environment. It is possible to install a differently versioned distribution within a root, than the native distribution. For example, it is possible to install an Ubuntu 12.04 chroot environment on an Ubuntu 14.04 system. Tools and utilities for building apps can be installed in the root using standard package management tools, since from the tool's perspective it is running in a native Linux environment. This makes it well suited for an SDK environment."

See also:

1.14 Windows Development Tools

1.14.1 Package managers

  • chocolatey - Command line package manager that can install applications and packages from command line. It works in a similar fashion to MacOSX's homebrew, it does not store any software, the repository just contains recipes for installing the applications.
  • Scoop - Similar to chocolatey, but it stores the recipes in Github repositories.

1.14.2 Sysinternals suite

  • SysInternals - A collection of debugging tools which allows to introspect processes and get informations such as loaded DLLs, files opened by a process, network activity and so on.
  • Sysinternals / dumpbin
    • Show informations Windows native executables and shared libraries, specifically about PE32 object code. For instance, it can show symbols exported by a DLL, DLLs used by an executable and functions imported by the executable from DLLs.
  • Sysinternals / procmon
    • Process mionitor
  • Sysinternals / procexp
    • Process explorer
  • Sysinternals / Winobj
    • WinObj accesses and display information on the NT Object Manager's name space

1.14.3 Installers and unpackers

Installers

Tools for creating installers for Windows applications.

  • NSIS - "NSIS (Nullsoft Scriptable Install System) is a professional open source system to create Windows installers. It is designed to be as small and flexible as possible and is therefore very suitable for internet distribution. Being a user's first experience with your product, a stable and reliable installer is an important component of successful software. With NSIS you can create such installers that are capable of doing everything that is needed to setup your software. NSIS is script-based and allows you to create the logic to handle even the most complex installation tasks. Many plug-ins and scripts are already available: you can create web installers, communicate with Windows and other software components, install or update shared components and more."
  • Inno Setup - "Inno Setup is a free installer for Windows programs by Jordan Russell and Martijn Laan"
    • Source: https://github.com/jrsoftware/issrc
    • Support for every Windows release since 2006, including: Windows 10, Windows 10 on ARM, Windows Server 2016, Windows 8.1, Windows 8, Windows Server 2012, Windows 7, Windows Server 2008 R2, Windows Server 2008, and Windows Vista. (No service packs are required.)
    • Extensive support for both administrative and non administrative installations.
    • Supports creation of a single EXE to install your program for easy online distribution. Disk spanning is also supported.

Unpackers

  • innoextract - "A tool to unpack installers created by Inno Setup."
  • msitools - "msitools is a set of programs to inspect and build Windows Installer (.MSI) files. It is based on libmsi, a portable library to read and write .MSI files. libmsi in turn is a port of (and a subset of) Wine's implementation of the Windows Installer."

1.15 Cross compilation tools

  • MXE - M Cross Compiling Environment (Linux/UNIX to Windows)
    • https://mxe.cc
    • Brief: Cross compiling from Unix-like OSes (Linux, MacOSX or BSD) to Windows NT 32/64 bits.
    • "MXE (M cross environment) is a GNU Makefile that compiles a cross compiler and cross compiles many free libraries such as SDL and Qt. Thus, it provides a nice cross compiling environment for various target platforms, which:""
      • is designed to run on any Unix system;
      • builds many free libraries in addition to the cross compiler
      • can also build just a subset of the packages, and automatically builds their dependencies
      • downloads all needed packages and verifies them by their checksums
      • is able to update the version numbers of all packages automatically
      • directly uses source packages, thus ensuring the whole build mechanism is transparent
      • has pre-compiled binaries that can be used in Continuous Integration systems
      • … … more
  • Dockcross (UNIX to Embedded Systems, Mobile/Android and Windows-Mingw)
    • https://github.com/dockcross/dockcross
    • Brief: Cross compiling toolchains in Docker images for Linux-ARM; Android-ARM(32 and 64 bits), Many Linux (portable linux binaries with GLIBC forward compatibility), Windows with MinGW.
    • Features:
      • Pre-built and configured toolchains for cross compiling
      • Most images also contain an emulator for the target system.
      • Clean separation of build tools, source code, and build artifacts.
      • Commands in the container are run as the calling user, so that any created files have the expected ownership, (i.e. not root).
      • Make variables (CC, LD etc) are set to point to the appropriate tools in the container.
      • Recent CMake and ninja are precompiled.
      • Conan.io can be used as a package manager.
      • Toolchain files configured for CMake.
      • Current directory is mounted as the container's workdir, /work.
      • Works with the Docker for Mac and Docker for Windows.
  • MSVC Docker
    • https://github.com/paleozogt/MSVCDocker
    • Brief: MSVC via Wine in Docker
    • CI with MSVC is unnecessarily difficult. Can't we just use Docker? It turns out we can– by running MSVC in Wine. Lots of folks have tried to do this over the years [1], but the setup is involved and fiddly. But scripting complicated setups is what Docker was made for! The big blocker to getting MSVC in Wine is that even though the software itself works under Wine, the installers don't. We dodge that problem by using Vagrant to drive the MSVC installer in a real Windows OS within VirtualBox, export a snapshot of the installation, and then Docker copies the snapshot into Wine.

1.16 State Machine Code Generators

State machine are ubiquitous in software for broad range of application domains such as games, game animations, embedded systems, network protocols, parsers and so on. The following tools can generate C and C++ state machine code from graphical user interface or from a domain specific language.

  • Ragel State Machine Compiler
    • "Ragel compiles executable finite state machines from regular languages. Ragel targets C, C++ and ASM. Ragel state machines can not only recognize byte sequences as regular expression machines do, but can also execute code at arbitrary points in the recognition of a regular language. Code embedding is done using inline operators that do not disrupt the regular language syntax. The core language consists of standard regular expression operators (such as union, concatenation and Kleene star) and action embedding operators. The user’s regular expressions are compiled to a deterministic state machine and the embedded actions are associated with the transitions of the machine. Understanding the formal relationship between regular expressions and deterministic finite automata is key to using Ragel effectively."
  • SMC: The State Machine Compiler
    • "Now you put your state diagram in one file using an easy-to-understand language. SMC generates the state pattern classes for you. No more hand-maintained transition matrices. No more widely scattered switch statements. Instead, the state diagram is in one place, coded directly from the picture to the SMC language and is easily maintained. SMC uses the state pattern to its fullest extent. In the real world, events don't always happen when they should. Dealing with unexpected events is a must for a robust application. By combining virtual methods with the state pattern, SMC allows you to define "Default" transitions - transitions which allow your objects to handle unexpected events, recover and continue providing service (rather than crashing, burning and getting you into trouble). SMC is a Java application. That means SMC will work on any platform where Java 1.7.0 or better is supported."
  • State machine generator & state diagram editor - StateForge
    • "State machine code generators: Tranform a state machine description into C#, Java or C++ source code. UML semantics: hierarchical, concurrent, asynchronous, entry/exit and history state. An extended version of state pattern is generated."
  • yasmine - yasmine - the C++ UML state machine framework
    • "yasmine is a C++ state machine framework developed by Seadex. The latest stable version is 1.5.3. The very first beta version (0.1.0) was released on June 20, 2016. Originally only supporting C++11 support for C++03 was added with Beta 6 (0.6.0). Version 1.0.0, the first stable version, was released January 20, 2017. The complete version history is available for you, too. yasmine is not just yet another state machine framework! It is an easy to use high level UML state machine abstraction that nicely integrates with modern object oriented C++ code. She is also highly portable. yasmine is in active use under Linux and Windows. Hardware platforms she's running on include x86, ARM, and Blackfin. Builds are actively tested for MSVC, gcc, and Clang."

1.18 Forensics Tools

1.18.1 Editors for structured binary Data

1.18.2 Network Protocols Debugging

Testing socket clients or servers:

Http Requests:

Websockets:

Network capture and packet inspection, low level analysis:

  • wireshark
    • "Wireshark is the world’s foremost and widely-used network protocol analyzer. It lets you see what’s happening on your network at a microscopic level and is the de facto (and often de jure) standard across many commercial and non-profit enterprises, government agencies, and educational institutions. Wireshark development thrives thanks to the volunteer contributions of networking experts around the globe and is the continuation of a project started by Gerald Combs in 1998."
  • tcpdump
    • "tcpdump, a powerful command-line packet analyzer; and libpcap, a portable C/C++ library for network traffic capture."
  • tcpflow
    • "tcpflow is a program that captures data transmitted as part of TCP connections (flows), and stores the data in a way that is convenient for protocol analysis and debugging. Each TCP flow is stored in its own file. Thus, the typical TCP flow will be stored in two files, one for each direction. tcpflow can also process stored 'tcpdump' packet flows."
    • See: TCPflow - Analyze and Debug Network Traffic in Linux

Certificates SSL/TSL:

  • mkcert - "mkcert is a simple tool for making locally-trusted development certificates. It requires no configuration."
  • minica - "minica is a small, simple CA intended for use in situations where the CA operator also operates each host where a certificate will be used."
  • certigo - "Certigo is a utility to examine and validate certificates to help with debugging SSL/TLS issues. Supports all common file formats: Certigo can read and dump certificates in various formats. It can automatically detect and read from X.509 (DER/PEM), JCEKS, PKCS7 and PKCS12 files. Certificates can be dumped to a human-readable format, a set of PEM blocks, or a JSON object for use in scripting."

1.18.3 Debuggers for introspection

  • scout - Scout - Instruction based research debugger
    • "Scout is an extendable basic debugger that was designed for use in those cases that there is no built-in debugger / gdb-stub in the debugee process / firmware. The debugger is intended to be used by security researchers in various scenarios, such as: Collecting information on the address space of the debuggee - recon phase and exploit development; Exploring functionality of the original executable by accessing and executing selected code snippets; Adding and testing new functionality using custom debugger instructions; We have successfully used "Scout" as a debugger in a Linux Kernel setup, and in several embedded firmware research projects, and so we believe that it's extendable API could prove handy for other security researchers in their research projects."
  • IDA - Interactive Disassemble and debugger designed for reverse engineering. In addition Linux, Windows and OSX binaries, IDA can also disassemble firmwares.
  • OllyDBG (Windows-only) - Debugger designed for reverse engineering with better capabilities to operate without debugging symbols.
  • x64dbg (Windows-only) - An open-source x64/x32 debugger for windows.

1.18.4 Tools for analysing executable formats PE, ELF, MachO

  • hte - "Viewer/editor/analyzer for executables"
    • Supported object file formats: common object file format (COFF/XCOFF32); executable and linkable format (ELF); linear executables (LE); standard dos executables (MZ); new executables (NE); portable executables (PE32, PE64); Mach exe/link format (MachO); … …
    • https://github.com/sebastianbiallas/ht
  • pev
    • "pev the PE file analysis toolkit fast, scriptable, multiplatform, feature-rich, free and open source."
  • CFF Explorer
    • PE (Portable Executable) viewer and editor with an intuitive graphical user interface.
  • dwarf-ng
    • "dwarf-ng is a powerful object file manipulation tool and hex editor in the spirit of gdb. with dwarf-ng you can read and edit all the file's section headers as well as the raw data. With dwarf-ng you can create and customize new file's header and it can be used as a compiler back-end to create executables/object files. dwarf-ng also permits to inject easily new headers and pieces of code/data into the file. dwarf-ng currently handles Raw ELF (Elf32, Elf64), PE (Portable executables PE32, PE+) and Mach-O (os x executables 32&64bit)".
  • dwarfutils
    • "Dump and produce DWARF debug information in ELF objects."
  • patchelf
    • "PatchELF is a small utility to modify the dynamic linker and RPATH of ELF executables. Dynamically linked ELF executables always specify a dynamic linker or interpreter, which is a program that actually loads the executable along with all its dynamically linked libraries. (The kernel just loads the interpreter, not the executable.) For example, on a Linux/x86 system the ELF interpreter is typically the file /lib/ld-linux.so.2. It is sometimes necessary to use a different ELF interpreter, say, when you want to test a version of Glibc installed in a location other than /lib. And in the Nix Packages collection we build our own Glibc, which we obviously want to use. Building programs with a different interpreter is a matter of using the linker flag -dynamic-linker: … …"
  • osslsigncode
    • "Platform-independent tool for Authenticode signing of PE(EXE/SYS/DLL/etc), CAB and MSI files - uses OpenSSL and libcurl. It also supports timestamping (Authenticode and RFC3161)."

1.18.5 Firmware Extraction

1.18.6 Windows NT

  • Dependency Walker
  • Resource Hacker
  • RPC View
    • "A free and powerful tool to explore and decompile all RPCfunctionalities present on a Microsoft system."
    • http://rpcview.org/
  • API Monitor
    • "Monitor is a free software that lets you monitor and control API calls made by applications and services. Its a powerful tool for seeing how applications and services work or for tracking down problems that you have in your own applications."
    • http://www.rohitab.com/apimonitor
  • Winapi Override
    • "WinAPIOverride is an advanced api monitoring software for 32 and 64 bits processes. You can monitor and/or override any function of a process. This can be done for API functions or executable internal functions. It tries to fill the gap between classical API monitoring softwares and debuggers. It can break targeted application before or after a function call, allowing memory or registers changes; and it can directly call functions of the targeted application."
    • http://jacquelin.potier.free.fr/winapioverride32/
  • NtTrace (Strace for Windows)
    • "NtTrace provides a simple trace facility for the Windows Native API. It is roughly equivalent to strace on Linux. The native API is the interface between the application space and the OS kernel; this API is provided by ntdll.dll. It is not very well documented, and changes between versions of Windows, but tracing execution of an application at this level can provide a clear view of its use of the operating system. NtTrace uses the debugging interface on Windows to intercept the returns from the native API and display the input arguments and return code. Return codes are translated to Window error code and error messages where possible."
    • http://www.howzatt.demon.co.uk/NtTrace/
    • https://github.com/rogerorr/NtTrace

1.18.7 Framework and Toolboxes

  • radare Framework - A suite of reversing enegineering tools.
    • Features:
      • Disassemble (and assemble for) many different architectures
      • Debug with local native and remote debuggers (gdb, rap, webui, r2pipe, winedbg, windbg)
      • Run on Linux, *BSD, Windows, OSX, Android, iOS, Solaris and Haiku
      • Perform forensics on filesystems and data carving
      • Be scripted in Python, Javascript, Go and more
      • Support collaborative analysis using the embedded webserver
      • Visualize data structures of several file types
      • Patch programs to uncover new features or fix vulnerabilities
      • Use powerful analysis capabilities to speed up reversing
      • Aid in software exploitation
  • radare2-docker - Dockerized version of Radare2.
  • mxmssh/drltrace - Drltrace is a library calls tracer for Windows and Linux applications.
    • "Drltrace is a dynamic API calls tracer for Windows and Linux applications. Drltrace is built on top of DynamoRIO dynamic binary instrumentation framework. Drltrace was initialy implemented by Derek Bruening and distributed with DynamoRIO and DrMemory frameworks. This repository contains a standalone version of drltrace with additional scripts and materials on how to use it for malware analysis. The release build can be downloaded here."
  • Frida - Dynamic instrumentation toolkit for developers, reverse-engineers, and security researchers. Features:
    • "Scriptable: Inject your own scripts into black box processes. Hook any function, spy on crypto APIs or trace private application code, no source code needed. Edit, hit save, and instantly see the results. All without compilation steps or program restarts."
    • "Portable: Works on Windows, macOS, GNU/Linux, iOS, Android, and QNX. Install the Node.js bindings from npm, grab a Python package from PyPI, or use Frida through its Swift bindings, .NET bindings, Qt/Qml bindings, or C API."

1.18.8 Misc

1.19 Version Control System and GIT

Command line repository managers for Github:

Gisthub Command Line Client:

1.20 Fonts For Programming

  • droid-sans-mono-sw
    • "Custom version of the font "Droid Sans Mono", improved for Programming. Some characters have been modified which makes it easier to code. (Including parenthesis size, dotted zero, At sign, Dollar, Tilde, comma/semicolon/dot sizes and a few other characters)."
  • ProgrammingFonts
    • "This is a collection of programming fonts,just share this with the programmers.Now there are 33 kinds of fantastic fonts! http://www.programmingfonts.com."

Created: 2021-06-04 Fri 15:08

Validate