Linux and Unix system programming

Table of Contents

1 Unix System Programming focused on Linux

1.1 Overview

1.1.1 Unix-like Operating Systems

Unix-like operating system are a family of operating system based on the AT&T Unix operating system, developed in the 1970's by Ken Thompson, Dennis Ritchie and others. Later AT&T licensed Unix to third-party vendors, what lead to many proprietary Unix variants.

Some Unix-like operating systems are

Desktop or Server

  • Linux-based or GNU-Linux based operating systems. (Note: Linux is NOT an operating system, it is just the kernel.) Although, the kernel is an essential component of an O.S., it is useless on its own, without all the additional supporting software such as: the GNU userland tools, bootloaders (Grub, Syslinux, Das U-boot), X11 - Windows Systems and so on. Some GNU-Linux operating systems are: Debian Linux; Ubuntu Linux; Fedora Linux; Centos Linux; Gentoo Linux; Arch Linux and more.
    • Somes places where Linux-based operating system is found:
      • Data centers
      • Most cloud virtualized servers.
      • Super Computers for HPC - High Performance Computing
      • HPC clusters - High Performance Computing cluster
      • Some embedded systems with network connectivity such as: high end routers - wifi access point; security cameras; payment terminals; network printers and so on.
      • Billions of mobile devices since Andoid is also based on Linux Kernel.
      • Desktop computers of enthusiasts.
    • Advantages:
      • Largest server and cloud virtual machines install-base.
      • Best hardware support among Unix-like operating systems.
      • Largest open source community.
      • Good support from embedded systems vendors.
      • Transparency: the system exposes many of its internals as virtual-file systems, which only exist on the main memory (volatile memory).
      • Stable and well documented system-calls.
      • The kernel receives significant contribution from many large companies, such as Intel, AMD, Nvidia, Red Hat, IBM, Google, Linar, Samsung and so on.
    • Drawbacks:
      • GLIBC - The GLIBC, which that encapsulates kernel system calls and implements basic C functions, such as memcpy and strcpy, which is relied by almost all applications, does not expose all kernel system call. Other significant problem, is the lack of backward compatibility as GLIBC has breaking changes on every release, as a result native code applications or native executables may not work when moved or copied to other Linux machines.
      • Lack of C API for many functionalities => Unlike other Unix-like operating systems and BSD-variants, Linux kernel lacks user-space C APIs (subroutines) for many of its functionalities. Instead of exposing these facilities as C APIs through the GlibC library, the kernel exposes them as virtual-file systems. For instance, instead of providing an user-space API (C subroutine) for listing all processes, the kernel exposes all processes via /proc virtual file system, which allows user-space applications to query process information just by reading and parsing files. Besides process information API, the Cgroups functionality also lcaks a user-space C API, instead it only has a virtual file system API.
      • Fragmentation among Linux distributions which makes harder to develop software for Linux-based systems as every distribution has different versions of GLIBC, different versions of shared libraries and different package managers. The fragmentation could be reduced if Linux distributions could share binaries without recompilation and applications were developed in a self-contained way.
      • Dependency hell: On most Linux distributions, it is only possible to install a single version of some application from package managers. Applications are also installed in a scattered way on many directories, for instance executables on /bin, /usr/bin, libraries on /lib, /lib64 and so on. If one install an application that relies on shared libraries newer from the ones installed on the system and the older libraries are overriden, applications relying on the older libraries may no longer work as the symbols which they need may no longer be available on the newer libraries.
  • Mac OSX / Apple Inc. (Desktop-only) [BSD Variant]
    • Most successful and widespread Unix-like operating system on Desktop. MacOSX is based on NeXTSETEP operating system, which is derived from BSD Unix (BSD - Berkley Software Distribution) and Mach Kernel, created at Carnegie Mellon University. Even though the graphics stack is built on top of Objective-C, MacOSX still shares a great deal of user-space APIs (Application Programming Interfaces) with BSD derived operating systems, such as: basic Unix system calls, open(), read(), write(); KQueue and poll() io multiplexing; pthread - POSIX threads and so on.
    • Advantages:
      • One of the best graphical user interfaces among Unix-like operating systems; consistent UX - user experience; high level GUI graphical user interface stack, that does not rely on X11. Unlike other unix-like, which applications are installed with files scattered in many different directories such as /bin, /lib, /usr/lib and so on, MacOSX uses self-contained applications app bundles, which are directories ending with .app suffix containing the application and executables and all dependencies in a single directory. App bundles allows installation by dragging and dropping its folder to the /Applications directory and uninstall by just removing the app bundle directory.
    • Drawbacks:
      • Proprietary.
      • Operating system is bundled with the hardware. A potential user cannot choose his or her own hardware.
      • Not allowed to be virtualized on the cloud on non-Apple hardware.
      • Little care about backward and forward compatibility, applications that works today, may not work tomorrow without active maintenance as the operating system vendor is not afraid of breaking backward compatibility.
  • FreeBSD
    • Unix-style operating system based on Unix BSD (Berkeley Software Distribution). Unlike Linux, the FreeBSD is a full-featured operating system, with almost everything maintained by the same team. Some noteworthy features of FreeBSD are: BSD permissive license; the high quality network stack that still outmatches Linux one, for instance BSD was used by Whatsapp, Yahoo and Netflix; PF - Firewall - Packet Filter firewall; ZFS highly reliable and scalable file system; kqueue scalable IO multiplexing API; full-featured documentation; Ports collection; BSD Jails sandboxing which is similar to Docker and other Linux containers; bhyve - hypervisor for visualization; own userland tooling; Linux Emulation - FreeBSD is able to run Linux native executables without recompilation via system call emulation; more cohesive and less fragmented than Linux-systems as everything is almost maintained by the same team.
    • Potential Problems: less hardware support than GNU/Linux systems; smaller community; less corporate support and less developers than Linux, for instance Linux Kernel often receives signifcant contribution from large companies such as Red Hat, IBM, Intel, Nvidia, Qualcom and so on; SVN version control system (note: FreeBDS is slowly moving to GIT). The hardware compatibility drawback that affects Desktop is not significant for usage of FreeBDS on server machines or on cloud virtual machines.
    • See:
  • OpenBSD - Similar to FreeBSD, but with security-aware design.
  • NetBSD
  • DragonFly BSD
  • Minix - Open source small operating system created for teaching about operating system concepts and designed to be understandable by a single person.
  • OpenIndiana - Derived from Illumos operating system, which is a fork of Sun Microsystem's Open Solaris.
  • Redox-OS (Written in Rust language) - "Redox is a Unix-like Operating System written in Rust, aiming to bring the innovations of Rust to a modern microkernel and full set of applications."
    • Unlike, Linux and other Unix-like flavours, Redox-OS has its own graphical user interface, which is not based on x11 - X Windows System.
  • Dahlia OS (dahaliaOS) - Operating system based on GNU/Linux and Google's (Alphabet Inc) Fuchsia OS, which one of the main noteworthy feature is the capability-based security.
  • Plan 9 (Bell Labs) [EXPERIMENTAL] - "Plan 9 from Bell Labs is a experimental operating system developed at Bell Labs at 1980s by Ken Thompson, Rob Pike, Dave Presotto, and Phil Winterbottom. Plan9 takes the concept of "everything is a file" to the next level. What sets Plan9 apart from other Unix variants is that "everything is almost a file", sockets are represented as files in PROCFS file system; GUI - graphical user interfaces are also represented as files.
  • Inferno-OS (Plan-9 variant) [EXPERIMENTAL] - "Inferno® is a distributed operating system, originally developed at Bell Labs, but now developed and maintained by Vita Nuova® as Free Software. Applications written in Inferno's concurrent programming language, Limbo, are compiled to its portable virtual machine code (Dis), to run anywhere on a network in the portable environment that Inferno provides. Unusually, that environment looks and acts like a complete operating system. Inferno represents services and resrouces in a file-like name hiearchy. Programs access them using only the file operations open, read/write, and close. `Files' are not just stored data, but represent devices, network and protocol interfaces, dynamic data sources, and services."

Server or Mainframes (mostly discontinued)

  • Solaris (from Sun Microsystems)
  • z/OS / IBM - Unix-variant for mainframes.
  • AIX / IBM
  • HP-UX
  • IRIX
  • Xenix - Discontinued Microsoft Unix variant.

Mobile (touch devices)

  • Android (Linux-based OS, but without GPL on the userland).
    • Unlike Desktop Linux distributions, Android uses the C runtime library bionic, derived from BSD, instead of GLIBC. Other significant differences is that Android does not uses X11 (X Windows System) as graphical user interface, instead the operating system uses the framebuffer directly. Shell command line tools, such as mv, ls, cp, and others, come from toybox (BSD license), instead of busybox or other GNU tools.
  • iOS / Apple

Embedded Systems

  • Embedded-Linux
    • Embedded Linux is not a single distribution or operating system. It is a highly customized Linux-based system comprised of many custom components, namely: bootloaders (das uboot, most popular); busybox or toybox as replacement for GNU coreutils (command line tolls: bash, ls, mv, …).
    • Embedded Linux systems can be found in: network routers; network printers; many devices with web interface; security cameras; Amazon's Alexa; Mobile devices (Android can be regarded as an embedded Linux system) and on many other devices.
  • QNX
  • Minix
  • NetBSD
  • LynxOS

1.1.2 Common Features of Unix-like Operating Systems

Nowadays, there may be many difference between Unix-like operating systems, but they still have the following features in common:

  • Command line shell
    • In the early days of computing, the shell or command line interpreter was the primary means of interaction between users and computers. A unix shell is used both as a scripting language and as an interactive command line interpreter for user interaction, launching processes, launching daemons (aka services), manipulating files and controlling the operating system.
    • Common Unix shells:
      • ash (Almiquistt Shell)
      • c-shell
      • dash (Debian Almiquist Shell)
      • tcsh
      • ksh (Korn Shell)
      • Tcl
      • bash (Bourn Shell) [Most used, most popular]
      • zsh [Most popular]
      • fish [Most popular]
  • Command Line Applications (Command-Line Centric)
    • mv; cp; rm; mkdir; ls; telnet; ssh; … and so on.
  • Terminal Emulation
    • Early Unix-like operating systems only ran in big and expensive mainframe computers that were simultaneously shared by many users via physical dumb terminals attached through serial cables (RS232 - UART). The first terminals were teletype printers (tty) where users could type commands and the teletype printer would print the the command output in a paper type. Later, dumb CRT (Cathode-Ray Tube) terminals were adopted. Those machines were comprised of a keyboard attached to a CRT video display as single unit attached to a mainframe via a serial cable. Physical Terminals, such as the once popular DEC VT100, are long gone, but the are still emulated by unix-based operating systems. For instance, most of those systems provide the following types of terminal emulation:
    • => Virtual console => Kernel built-in terminal emulator, which can be accessed without any graphical user interface by typing Ctrl + Alt + F1, Ctrl + Alt + F[N] on Linux. BSD has different keybinds for the virtual console. Note: Mac-OSX does not have virtual console.
    • => Graphical Terminal Emulator => Example: Xterm (X11); Gnome Terminal; Terminator; iTerm (MacOSX).
    • => Serial console => Allows accessing a command line shell (bash, for instance) through a serial cable (RS232 / UART). Serial console is still used for accessing servers or embedded systems.
  • Multiple Simultaneous Users
    • Most unix-based systems support multiple simultaneous user accounts through: serial terminal (serial RS232 cable); serial dumb terminals (mainframe heritage); telnet obsolete and unsafe; SSH (Secure Shell Server) protocol over the internet or intranet or X11 (X Windows Systems) for graphical remote access. Most editions of Windows NT operating systems also support remote access via remote desktop, but do not support multiple simultaneous users like Unix systems, this feature is only available on Windows NT server edition.
  • Text-centric
    • Unix-like operating system has the tradition of storing configuration in human-readable text files, instead of binary files as Windows does. The benefits of using text files are the better reusability, searchability and reproducibility of configurations. All that one needs to reproduce the configuration in another machine is to copy the configuration file.
    • Many applications also use text for configuration instead of graphical user interfaces.
    • Examples:
      • Linux-based operating system store many of their configuration as text files in the '/etc' directory. For instance, the file '/etc/fstab' controls the disk partitions mounted during boot time. As opposite to Unix-tradition, Windows and most of its applications store configuration in the registry file which is a binary file.
      • Applications such as bash, vi, emacs store configuration in text files placed in default locations which are read during the initialization.
  • POSIX Features / POSIX - (Portable Operating System Interface) standard
    • POSIX System Calls
      • open(), read(), close() …
    • POSIX Functions encapsulating system calls
      • open(), read(), close(), mmap(), dlopen(), dlclose() …
    • POSIX Threads (_PThreads_) - Thread C API
    • POSIX Shells
  • BSD TCP/IP Sockets => The TCP/IP network technology stack was first implemented on Unix BSD (Berkley Software Distribution) and later became the reference implementation for network stacks.
  • Hierarchical file system
    • File system with paths like: '/' (root directory); '.' (dot) as the current directory; '/home/user/data' (Linux); '/home/Users/somthing' (MacOSX) and so on.
  • Symbolic Link (Symlink)
    • Allows file and directories to be accessed from other directories as they were in that location.
  • Virtual File Systems
    • Many unix-like operating system provide virtual file-system (in-memory file systems) which exposes kernel and system information to user-space applications as human-readable text files. For instance, Linux has the PROCFS (/proc directory) file system which exposes information about all the running processes. PROCFS also has the file /proc/cpuinfo which contains information about the current machine microprocessor, such as: CPU cores; number of hyper threads; CPU features and capabilities.
  • Everything is a file (or better everything is a file descriptor)
    • Many operating such as reading/writing sockets, pipes and so on, use the same read/write system-calls for file descriptors.
    • Hardware represented as device-files
    • Kernel exposes information as text files (Linux)
    • Disk partitions are represented as files /dev/sda, /dev/sdb1, /dev/sdb2 …
    • see: file descriptor
  • Device Files (a.k.a device nodes)
    • Device files are pseudo-files created in virtual file systems which allows user-space applications to interact with the hardware peripherals by using reading and writing system-calls or just by reading and writing files. This feature is particularly important for industrial and embedded system applications as it allows reading sensor data, such as ADC - Analog-To-Digital Converters or digital input, just by reading files and controlling actuators, such as motors, solenoids or valves just by writing to files.
    • Note A: On Linux-based, operating systems, most device files are in the virtual file systems devfs (/dev) and and sysfs (/sys) directory.
    • Note B: Not all device files are associated to real hardware devices. The device-files /dev/null, /dev/zero, /dev/random, /dev/urandom, /dev/kmsg, /dev/tty0 are not associated to any real hardware.
    • Example 1: It allows user-space application to read and write from/to serial port devices by just reading or writing to the device file /dev/ttyS1, /dev/ttyUSB0, …
    • Example 2: In single-board computers SBCs (embedded systems) such as Beagle Bone Black or Raspberry PI, both which contains SOC (System-On-Chip) embedded processors, it is possible to control GPIOs (General Purpose IO) which are digital IOs, and consequently any device attached to them, such as lights, LEDs, motors or valves, just by writing to the corresponding GPIO device file (/sys/class/gpio/gpio10/value). By writing 1 to the this file, the LED is turned on, by writing 0, the LED is turned off. The device-file feature allows controlling the hardware with any programming language, including, shell scripts, python, ruby, standard C++ (without volatile and embedded bare-metal restrictions) and so on.
    • Common Linux Device Files associated to hardware
      • /dev/mem
        • => Device file that represents the physical RAM memory. It can be used for reading MMIO (Memory-Mapped IO) devices.
      • /dev/lp0, /dev/lp1 => Parallel ports
      • /dev/ttyS0, /dev/ttyS1, … => Serial Ports
      • /dev/ttyUSB0, /dev/ttyUSB1, … => USB-to-serial converters such as FTDI
      • /dev/sys/class/gpio/gpio10/export => GPIO / General Purpose Digital IO
      • /dev/sys/class/gpio/gpio10/value
      • … … …
      • /dev/input/mice => Binary file contains mouse position, mouse buttons and and so on. The mouse position can be obtained by reading this file, without any special API.
      • /dev/fb0, /dev/fb1 … => Framebuffer - allows user-space applications to control the graphics card and draw on the screen by just writing to a file. The framebuffer is the lowest level GUI in Linux kernel. Most embedded linux applications draw directly on framebuffer without X11 (X-Windows System).
    • Linux Device Files not associated to hardware:
      • /dev/kmsg
        • => charavter device file containing kernel messages. The file is read by the command line tool '$ dmesg'.
      • /dev/stdout => Console stdout
      • /dev/stdin => Console stdin
      • /dev/stderr => Console stderr
      • /dev/random => File which generates random numbers
      • /dev/urandom => File which generates random numbers
      • /dev/null => File used for discarding stdout or stderr of command line applications.
      • /dev/zero => Generate zero-values
      • /dev/null

Quotes

  • Ken Thompson

I think the major good idea in Unix was its clean and simple interface: open, close, read, and write.

This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.

  • Brian Kernighan

Unix has, I think for many years, had a reputation as being difficult to learn and incomplete. Difficult to learn means that the set of shared conventions, and things that are assumed about the way it works, and the basic mechanisms, are just different from what they are in other systems.

  • Rob Pike - Member of the Unix design team; creator of GO (Golang) and Plan-9 operating system.

Even though the UNIX system introduces a number of innovative programs and techniques, no single program or idea makes it work well. Instead, what makes it effective is the approach to programming, a philosophy of using the computer. Although that philosophy can't be written down in a single sentence, at its heart is the idea that the power of a system comes more from the relationships among programs than from the programs themselves. Many UNIX programs do quite trivial things in isolation, but, combined with other programs, become general and useful tools.

See also

Unix History:

  • The Origin of the Shell - Multics
  • GitHub - yvesnrb/Thompson-Shell
    • Original source code for the Sixth Edition (V6) UNIX Thompson shell.
  • GitHub - susam/tucl
    • The first-ever paper on the Unix shell written by Ken Thompson in 1976 scanned, transcribed, and redistributed with permissio.
  • THE UNIX COMMAND LANGUAGE | The UNIX Command Language (Ken Thompson)
  • GitHub - dspinellis/unix-history-repo
    • "The history and evolution of the Unix operating system is made available as a revision management repository, covering the period from its inception in 1970 as a 2.5 thousand line kernel and 26 commands, to 2018 as a widely-used 30 million line system. The 1.5GB repository contains about half a million commits and more than two thousand merges. The repository employs Git system for its storage and is hosted on GitHub. It has been created by synthesizing with custom software 24 snapshots of systems developed at Bell Labs, the University of California at Berkeley, and the 386BSD team, two legacy repositories, and the modern repository of the open source FreeBSD system. In total, about one thousand individual contributors are identified, the early ones through primary research. The data set can be used for empirical research in software engineering, information systems, and software archaeology."
  • AT&T Archives: The UNIX Operating System
  • UNIX: Making Computers Easier To Use -
    • AT&T Archives film from 1982, Bell Laboratories
  • LINUX's History by Linus Torvalds
  • "What UNIX Cost Us" - Benno Rice (LCA 2020)
    • "UNIX is a hell of a thing. From starting as a skunkworks project in Bell Labs to accidentally dominating the computer industry it's a huge part of the landscape that we work within. The thing is, was it the best thing we could have had? What could have been done better? Join me for a bit of meditation on what else existed then, what was gained, what was lost, and what could (and should) be re-learned."
    • => Criticism of the mindset: "Everything is a file or a file descriptor. Not everything should be a file."

General Unix Concepts:

Papers and mail lists:

Usage of Linux Device Files for controlling hardware (reading Sensors and manipulating actuators):

Terminal Emulation and Serial Console

Plan 9 Operating System

1.1.3 Components of Linux-based operating systems (Linux Distros)

Widely Used GNU and Kernel-Organization Tools

GNU tools are pervarsive on most Linux distributions either as essential command line utilities or building tools:

  • GNU Toolchain (GNU Compilers and GDB Debugger)
    • GDB => GNU Debugger
    • GCC => C, ObjectiveC and C++ compiler
    • GFortran => GNU Fortran Compiler
    • Ada => GNU Ada compiler
    • … …
  • GNU Binutils - Collection of tools for manipulating object-files:
    • ld => GNU linke
    • nm => Shows symbols exported by object code file or shared library.
    • as => GNU assembler
    • objdump => Dump object file symbols, can be used for disassembly ELF object-files.
    • c++-filt => Demangle (aka decode) C++ symbols
    • readelf => Red information about ELF object-files.
    • strings => List printable strings from any binary file.
  • GNU Building System (GNU Autotools, GNU automake and Libtool)
  • GNU Core Utils (coreutils) - GNU Core Utilities - Comprises essential Unix command line utilities such as: ls, cat, grep, less and so on.
  • util-linux (Maintened by Linux Kernel Organization) => Set of command applications not provided by GNU tools:
    • $ mount => Command line application for mounting file systems
    • $ fdisk => Command for creating and manipulating disk partitions
    • $ getopt => Parse command line arguments
    • $ cal => Prints calendar
    • $ hexdump => Dump binary files in hexadecimal format.
    • $ logger => Sends message to syslog
    • $ chcpu => Configure CPUs in a multi-processor system.
    • $ dmesg => Display kernel messages (content of device file /dev/kmsg)
    • $ findmnt => Find mounted file systems.
    • $ lsmode => List loaded kernel modules
    • $ lsblk => List block devices (disks and partitions) and mount points.
    • $ lsscsi => List SCSI devices, hard disks.
    • $ lscpu => Display CPU information: architecture, operating mode, number of cores and so on. This information comes from the file /proc/cpuinfo
    • $ lslocks => List local file system locks.
    • $ lsusbs => List USB devices
    • $ lspci => Show information about devices attached to PCI bus.
    • $ lslogin => Display information about known users in the system.
    • $ lsof => List open files.
    • … and much more …

Components of a Linux-based Operating System (Generic IBM-PC Architecture)

  • First-stage Boot Loader (Built-in, firmware)
    • BIOS - Basic Input/Output System
    • UEFI - Unified Extensible Firmware Interface
  • Second-stage Boot Loader => Loads the operating system
    • Examples:
      • Grub (Mostly used on Desktop)
      • Syslinux (Mostly used for booting CDROM ISO images, Linux Live CDs)
  • Linux Kernel (Written in C with proprietary GNU GCC compiler extension)
    • One of the most important parts of GNU/Linux desktop and server distribution. The kernel provides many essential services that user-space applications take for granted such as: standardized and documented system-calls; process and thread scheduling; network stack; virtual memory management; virtual file systems; uniform interfaces for accessing hardware either as character device, block device or network device for network cards.
  • CRT - C-Runtime Library
    • Encapsulates kernel system-calls and basic services as C APIs for user-space applications and implements C functions required by C standards.
    • Example:
      • GLIBC (GNU C-library, most used C library by Linux Desktop and server distributions). Maintained by: FSF - Free Software Foundation.
      • Bionic - CRT library used by Android.
      • MUSL - CRT library used for embedded systems and static linking.
  • User-space utilities
    • Shell - command line interpreter.
    • Basic Command Line Tools (ls, mv, pwd, ln, rm, file, …)
      • GNU Coreutils
      • Busybox => Is a single binary containing all command line tools as sub-commands. Busybox is widely used on Embedded-Linux systems.
    • GUI - Graphical User Interface Stack (Not part of Kernel) - all GUI applications are built on top of kernel framebuffer.
      • X11 - X Windows System
      • GTK - Gimp Toolkit
      • QT
    • Package Managers

Components of a Embedded Linux System

  • Cross Compiler Toolchain: Mostly GCC - GNU C Compiler
    • Note: The Linux Kernel uses many C proprietary extensions of GCC compiler, therefore it is almost impossible to compile the kernel using any other compiler.
  • Boot Loader => Loads the operating system
  • Linux Kernel (GPL v2 License)
    • Among other-things, the kernel provides: virtual memory management; processes; kernel-space (native) threads; kernel modules (device drivers) API and many other essential features.
  • CRT - C Runtime Library
    • uLibC
    • MUSL
    • Bionic (Android C library)
    • GLIBC
  • User-Space Utilities (Replacemente for GNU CoreUtils)
    • busybox (GPL v2 License) - Single binary executable containing many common unix command line utitlities in a single executable, such as: unix shell, ls, mv, cp, rm, zip, unzip, … and so on.
      • "BusyBox combines tiny versions of many common UNIX utilities into a single small executable. It provides replacements for most of the utilities you usually find in GNU fileutils, shellutils, etc. The utilities in BusyBox generally have fewer options than their full-featured GNU cousins; however, the options that are included provide the expected functionality and behave very much like their GNU counterparts. BusyBox provides a fairly complete environment for any small or embedded system."
    • toybox (BSD License) => Used on Android.
      • "Toybox combines many common Linux command line utilities together into a single BSD-licensed executable. It's simple, small, fast, and reasonably standards-compliant (POSIX-2008 and LSB 4.1). Toybox's main goal is to make Android self-hosting by improving Android's command line utilities so it can build an installable Android Open Source Project image entirely from source under a stock Android system. After a talk at the 2013 Embedded Linux Conference explaining this plan (outline, video), Google merged toybox into AOSP and began shipping toybox in Android Mashmallow."
  • BSP - Board Support Package
    • Device drivers or kernel modules provided by SOC (System-On-Chip) manafucaturer. Most device drivers maps on-board peripherals, such as UART (serial communication device); GPIO - General Purporse IO; CAN - Controller Area Network or ADC - Analog-To-Digital Converter, to device files which allows user-space applications to control the hardware by just reading and writing files.
    • The device-files feature allows using standard C or C++ for interacting the aforementioned peripherals and any physical devices attached to them such as pumps, electric motors, , valves, sensors, accelerometers or gyroscopes. The device-file feature also allows controling the hardware with any scripting language such as: ruby, python, unix shell script (bash, ash, sh, …) …
  • GUI - Graphical User Interface
    • Framebuffer - Linux Kernel Device file which allows user-space applications to control graphical display.
    • In-house GUI on top framebuffer (Lowest level)
    • SDL (LGPL) - http://www.libsdl.org/ - Library that came from Game Development.
    • QT on top of framebuffer
    • X11 - X Windows System (Rarely used, too heavy.)

See

1.2 Command Line Essentials

1.2.1 Overview

Useful reminder of common built-in command line applications for many unix-like operating systems, including Linux-distros; BSD-variants and MacOSX.

1.2.2 Commands summary table

The following table contains the commands for Linux and other Unix-like operating systems that may be useful for troubleshooting and debugging.

Command Description  
System status    
$ uname -a Display operating system, kernel version and uptime.  
$ uptime Tells how long the machine is running.  
$ w Show logged users and pseudo terminal sessions.  
     
Current user    
$ whoami Show current logged user.  
$ id Show current user ID and GID  
$ groups Show groups that the current user belongs to.  
$ locale Show user locale (local language settings)  
$ sudo su Log as root (shoud be avoided)  
$ sudo <COMMAND> Run command as root user.  
     
Enviroment Variables    
$ env Show all enviroment variables.  
$ echo $PATH Shows path environment variable  
$ echo $HOME Shows HOME enviroment variable, akin to Windows %USERPROFILE%  
$ echo $TERM Show type of terminal emulator, dumb, vt100, xterm, and so on.  
     
Process management    
$ which <COMMAND> Show path to given command or executable such as GCC.  
$ ps -ef List all running processes.  
$ ps -aux List all running processes  
$ top Terminal user interface for process monitoring  
$ atop Monitor CPU and IO performance  
$ iotop Monitor only disk IO speed and performance.  
$ htop Similar to htop, but with colors and more functionality.  
$ kill <PROCESS-PID> Terminate a process given its PID (unique identifier)  
$ pgrep <PATTERN> –list-full List all processes PIDs and paths that match a given pattern such as name.  
$ pkill <PROCESS-NAME> Terminate all processes with a given name  
$ pkill nvim Terminate all neovim processes (executable name is 'nvim')  
     
Hardware information    
$ free -m Shows the amount of RAM memory usage and remaining memory.  
$ lscpu Display information about machine CPUs or processor capabilities.  
$ lspci List all PCI card (network cards, video cards and so on)  
$ lsusb List all devices connected to USB.  
$ lsmod List all kernel loaded modules (aka device drivers)  
$ hostnamectl Show hostname, Machine-ID, Boot-ID, Operating system and architecture.  
# demidecode Show hardware serial numbers.  
     
Network Interfaces    
$ ip addr Display all network interfaces  
$ ip route Display default gateway and route.  
$ ifconfig Display all network interfaces, mac and IP addresses  
$ iwconfig Display only Wifi network interfaces.  
$ arp -a Shows ARP table.  
# netstat -anp    
$ ping <HOSTNAME> Check network connectivity with ping.  
$ traceroute <HOSTNAME> Traceroute checking for network connectivity.  
# netstat -tulnp Show all TCP/IP and TCP/UDP connections, ports and related processes.  
# netstat -tulnp | grep -i listen Show all TCP/IP and TCP/UDP listening (passive socket, server sockets).  
$ ss List all TCP/IP, TCP/UDP and unix domain sockets connections statistics.  
$ ss –process –tcp –all List all TCP/IP connections.  
$ ss –process –tcp –listen List only TCP/IP listening sockets and connections.  
$ ss –process –udp –listen List only UDP/IP listening sockets and connections.  
$ ss –process –raw List all raw sockets. The flag (–process) shows the process that socket belongs to.  
$ ss dst <HOSTNAME> –process List all connections related to a given hostname, for instance, 192.168.0.110  
$ ss -4 Display only Ipv4 connections  
$ ss -6 Display only Ipv6 connections  
     
Disks, partitions and mount-points    
$ df -H Display used disk space of all partitions and VSFs  
# fdisk -l List all disks and partitions  
$ blkid List all disks (block devices), labels and their UUID  
$ mount List mout points.  
$ mount | grep -i /dev/sd List only disk mount points.  
     
     
Systemd - Init system    
$ systemctl list-units List systemd running units  
$ systemclt –failed List systemd failed units.  
$ systemctl list-unit-files List systemd unit files.  
     
$ systemctl start <SERVICE> Start a daemon, aka background service, akin to Windows service  
$ systemctl stop <SERVICE> Stop a service.  
$ systemctl enable <SERVICE> Restart a service.  
$ systemctl stop <SERVICE> Stop a service.  
$ systemctl status <SERVICE> Display service status  
     
System logs    
$ journalctl -b Show kernel boot messages.  
$ dmesg -T Kernel messages with human-redable timestamp.  
$ dmesg -T –level=err Shows only kernel messages that reports error.  
     
Package manager    
apt-get -y install <APP> Install an application on Ubuntu, Debian with Apt package manager.  
yum install <APP> Install app on any Read-Hat Linux based distribution.  
dnf install <APP> Install app on Fedora using DNF package manager.  
apk add <APP> Install app. on Alpine.  
     
X Windows System    
pkill -15 Xorg Reset X11 X windows System display server.  
setxkbmap -option Reset capslock customization.  
setxkbmap -option ctrl:swapcaps [EMACS FRIENDLY] Swap ctrl and capslock (X11 only, Linux, BSD and so on)  
setxkbmap -option caps:swapescape [VIM FRIENDLY ] Swap ESC and capslock (X11 only)  

Notes:

  • Commands started with '$' dolar sign does not require administrative privilege.
  • Commands started with '#' symbol require administrative or root privilege and should be run as root user or by using sudo.
  • VSF - Stands for Virtual File System such as udev, tmpfs and so on.
  • ifconfig command, for displaying information about network interfaces, usage is no longer recommended due to it be deprecated on most modern Linux distribution. Ifconfig has been superseded by ip command. However, the ifconfig is still used on BSD-variants, MacOSX and older Unix-like operating systems.
  • iwconfig is also deprecated on most Linux distributions.
  • The command '$ ip addr' is useful for obtaining the IP address of a home network router and browse its web interface using a web browser.
  • UUID stands for Univeral Unique IDentifier
  • The applications, atop, iotop and htop are not installed on most Linux distributions by default, however they can be obtainted through the package manager.
  • The applications atop and iotop are essential for monitoring CPU and disk IO of http servers, database servers and etc.

1.2.3 Basic Commands

  • => Change current prompt (works for Bash and ZSH shells)
$ export PS1=" $ >> "

$ >> command typed by user  
output line 0 
output line 1 
... ... ... 
output line N-1
  • pwd => Get current working directory
$ pwd
/home/mxpkf8/Downloads
  • cd - Change the current directory
$ cd /Applications # OSX 
$ cd ~/Downloads 
$ cd /home/user/Downloads 
$ cd /Users/user/Downloads

# Enter in Home directory: 
#   /Users/<USER> in Mac OSX 
#   /home/<USER/ in Linux                 
$ cd $HOME
  • ls - List directory

Linux: 

$ ls /

bin@   dev/  home/  lib64@       media/  opt/   root/  sbin@  sys/  usr/
boot/  etc/  lib@   lost+found/  mnt/    proc/  run/   srv/   tmp/  var/

Mac OSX: 

$ ls -l /System/Applications/Utilities/Terminal.app/Contents

total 32
-rw-r--r--   1 root  wheel  7998 Apr 17 22:57 Info.plist
drwxr-xr-x   3 root  wheel    96 May 27 20:40 MacOS
-rw-r--r--   1 root  wheel     8 Apr 17 22:57 PkgInfo
drwxr-xr-x  74 root  wheel  2368 May 27 20:30 Resources
drwxr-xr-x   3 root  wheel    96 Apr 17 22:57 _CodeSignature
-rw-r--r--   1 root  wheel   510 Apr 17 22:58 version.plist
  • rm - Delete files or directory
# Delete file 
$ rm file.txt 

# Delete directory dir2 and all sub-directories of dir2 
$ rm -rf ./dir1/dir2

# Delete directory dir2 and all sub-directories of dir2 in verbose mode 
$ rm -rf -v ./dir1/dir2
  • cat - Display text file content

Linux: 

$ cat /proc/uptime 
11960.35 36214.14

$ cat /proc/filesystems 
nodev   sysfs
nodev   tmpfs
nodev   bdev
nodev   proc
nodev   cgroup
nodev   cgroup2
.. ... ... ... 
... ... ... ..

Mac OSX: 

 $ cat /System/Applications/Utilities/Terminal.app/Contents/Info.plist  

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>ATSApplicationFontsPath</key>
        <string>Fonts</string>
        <key>BuildMachineOSBuild</key>
        <string>18A391024</string>
  ... ...   ... ...   ... ...   ... ...   ... ...   ... ...   ... ... 
  ... ...   ... ...   ... ...   ... ...   ... ...   ... ...   ... ...

1.2.4 Creating files and directories from command line

  • mkdir - Create directory (aka folder)
# Create single directory/folder 'somedir' at current directory
$ mkdir somedir

# Create single directory/folder 'somedir' at /tmp path
$ mkdir -p /tmp/somedir

# Create directory-tree root-folder, root-folder/sub1 and root-folder/sub1/sub2 
$ mkdir -p /Users/unix/root-folder/sub1/sub2

# Create directory tree 
$ mkdir -p mydir/{subA,subB}/d1/d2

$ tree mydir
mydir
├── subA
│   └── d1
│       └── d2
└── subB
    └── d1
        └── d2
  • touch - Create empty files:
$ touch file1.txt somefile.cpp CMakeLists.txt
  • Create multi-line file with echo
$ echo line1 > file.txt
$ echo line1 > file.txt
$ echo line3 >> file.txt

$ cat file.txt 
line1
line2
line3
  • Create multi-line files with cat-EOF trick.
# Create a file with some content 
$ cat > script.sh <<EOF
#!/usr/bin/env sh 

echo " => Showing root directory"
ls /

EOF

$ cat script.sh 
#!/usr/bin/env sh 

echo " => Showing root directory"
ls /

$ sh script.sh 
 => Showing root directory
bin   dev  home  lib64       media  opt   root  sbin  sys  usr
boot  etc  lib   lost+found  mnt    proc  run   srv   tmp  var

1.2.5 Files and executables

  • which - Get path to executable in $PATH environment variable.
$ which xdg-open
/usr/bin/xdg-open

$ which zsh
/usr/bin/zsh

$ which brave
/home/mxpkf8/Applications/brave/brave
  • file - Identify text or binary files by they "magic-number" (uniquely identifying byte sequence)

Linux: 

 $ file /usr/bin/bash
 /usr/bin/bash: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV)
 , dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2
 .. ... ... ... ... ... 

$ file vs_buildtools__80950065.1591646379.exe 
vs_buildtools__80950065.1591646379.exe: PE32 executable (GUI) Intel 80386, for MS Windows

Mac OSX: 

$ file /bin/zsh 
/bin/zsh: Mach-O 64-bit executable x86_64

$ file $HOME/Desktop/Console 
/Users/unix/Desktop/Console: MacOS Alias file

$ file /System/Applications/Utilities/Terminal.app/Contents/MacOS/Terminal
/System/Applications/Utilities/Terminal.app/Contents/MacOS/Terminal: Mach-O 64-bit executable x86_64
  • Open file with default system Application / xdg-open (Linux with X11, BSD with X11); open (MacOSX)

Linux: 

# Open Desktop directory with default system file manager 
$ xdg-open $HOME/Desktop 

# Open file image.png with default system image viewer 
$ xdg-open image.png

Mac OSX: 

# Open /Volumes directory with default system file manager (finder)
$ open /Volumes 

# Open file CmakeLists.txt with default system text editor 
$ open CMakeLists.txt 

$ open disk-image.dmg

1.2.6 System information

  • uname => Get Kernel Version

Linux: 

$ uname -a
Linux localhost.localdomain 5.6.6-300.fc32.x86_64 #1 SMP Tue Apr 21 13:44:19 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Mac OSX Catalina: 

$ uname -a 
Darwin ghosts-iMac-Pro.local 19.5.0 Darwin Kernel Version 19.5.0: Tue May 26 20:41:44 PDT 2020; root:xnu-6153.121.2~2/RELEASE_X86_64 x86_64
  • uptime - Shows how long the machine is running.
# Linux 
$ >> uptime
13:43:36 up  2:22,  1 user,  load average: 1.31, 1.00, 0.99

# Mac OSX
13:44  up  1:48, 2 users, load averages: 1.17 1.30 1.35
  • whoami - Show current user (Note: it also works on Windows)
$ whoami
user_ml7abfg
  • w - Show logged users

Linux: 

$ w
 13:37:31 up  2:16,  1 user,  load average: 0.63, 0.87, 0.98
USER     TTY        LOGIN@   IDLE   JCPU   PCPU WHAT
user_ml7abfg   :0        11:22   ?xdm?  43:26   0.00s /usr/libexec/gdm-x-session --register-session --

Mac OSX: 

$ w 
13:38  up  1:42, 2 users, load averages: 1.86 1.53 1.44
USER     TTY      FROM              LOGIN@  IDLE WHAT
unix     console  -                11:58    1:39 -
unix     s000     -                13:32       - w
  • ps - Show running processes

Linux: 

$ ps -ef 
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 11:21 ?        00:00:18 /usr/lib/systemd/systemd --switched-root --system --deserialize 29
root           2       0  0 11:21 ?        00:00:00 [kthreadd]
root           3       2  0 11:21 ?        00:00:00 [rcu_gp]
root           4       2  0 11:21 ?        00:00:00 [rcu_par_gp]
 ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ... 
 ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ... 
mxpkf8      2383    2382  0 11:22 ?        00:00:00 dbus-broker --log 4 --controller 9 --machine-
mxpkf8      2387    1946  0 11:22 ?        00:00:01 /usr/libexec/at-spi2-registryd --use-gnome-se
mxpkf8      2392       1  0 11:22 ?        00:00:00 /usr/bin/gpg-agent --sh --daemon --write-env-
mxpkf8      2393    2149  0 11:22 tty2     00:01:18 xfwm4
 ... ... ... ... ... ...  ... ... ... ... ... ... 
 ... ... ... ... ... ...  ... ... ... ... ... ...

Mac OSX: 

$ ps -ef 
  UID   PID  PPID   C STIME   TTY           TIME CMD
    0     1     0   0 11:56AM ??         0:16.58 /sbin/launchd
    0    86     1   0 11:58AM ??         0:01.13 /usr/sbin/syslogd
    0    87     1   0 11:58AM ??         0:03.44 /usr/libexec/UserEventAgent (System)
    0    90     1   0 11:58AM ??         0:01.11 /System/Library/PrivateFrameworks/Uninstall.framework/Resources/uninstalld
    0    91     1   0 11:58AM ??         0:05.95 /usr/libexec/kextd
    0    92     1   0 11:58AM ??         0:09.31 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/FSEvents.framework/Versions/A/Support/fseventsd
    0    93     1   0 11:58AM ??         0:00.32 /System/Library/PrivateFrameworks/MediaRemo
 ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ... 
 ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ... 
  501  6102     1   0  1:29PM ??         0:00.26 /System/Library/CoreServices/CoreServicesUIAgent.app/Contents/MacOS/CoreServicesUIAgent
  501  6813     1   0  1:45PM ??         0:01.12 /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/CVMCompiler 2
    0  6795   319   0  1:32PM ttys000    0:00.06 login -pf unix
  501  6796  6795   0  1:32PM ttys000    0:00.38 -zsh
    0  6816  6796   0  1:46PM ttys000    0:00.01 ps -ef
  • env - Show environment variables

Linux: 

$ env
IMSETTINGS_INTEGRATE_DESKTOP=yes
SHELL=/bin/bash
SESSION_MANAGER=local/unix:@/tmp/.ICE-unix/2149,unix/unix:/tmp/.ICE-unix/2149
WINDOWID=52428803
COLORTERM=truecolor
XDG_CONFIG_DIRS=/etc/xdg
HISTCONTROL=ignoredups
XDG_MENU_PREFIX=xfce-
 ... .. ... ..  ... .. ... ..  ... .. ...
 ... .. ... ..  ... .. ... ..  ... .. ...

Mac OSX: 

$ env 

TMPDIR=/var/folders/jh/w00bgbps79bddlbqwkv31d9m0000gn/T/
XPC_FLAGS=0x0
LaunchInstanceID=433A53AA-AB93-4CF7-9700-BA15B57FFDAC
TERM=xterm-256color
LANG=en_US.UTF-8
 ... ... ... ... ... ... ... 
 ... ... ... ... ... ... ... 
SHELL=/bin/zsh
HOME=/Users/unix
LOGNAME=unix
USER=unix
PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
SHLVL=1
PWD=/Volumes/data/unix-explore
OLDPWD=/Volumes/data
PS1= $> 
_=/usr/bin/env

1.3 Widely used functions for strings and buffers in low level codes

This section provides a recapitulation of the most widely used C functions in low level code bases and legacy C codes. Those functions are in the header <string.h> for C or <cstring> for C++.

String Manipulation

  • strlen() => Get length of null-terminated character array.
    • Docs: $ man strlen
size_t strlen(const char *s);

root [39] const char* st1 = "unix posix irix linux";
root [40] const char* st2 = "bsd msdos";

root [41] strlen(st1)
(unsigned long) 21

root [42] strlen(st2)
(unsigned long) 9
  • strdup() => Copy a C-string allocating the copy with malloc.
char* strdup(const char *s);

root [39] const char* st1 = "unix posix irix linux";

root [43] char* p_copy = strdup(st1)
(char *) "unix posix irix linux"

// Release memory for heap-allocated string 
root [47] free(p_copy)
  • strcmp() => Compare two C-strings null-terminated (terminated with '\0' or 0x00 null char) array of characters. This function returns 0 if the two strings are equal.
    • Docs: $ man strcmp
int strcmp(const char *s1, const char *s2);

// ------- USAGE --------------------------//
if ( strmcp(STRING_A, STRING_B) == 0 )
    puts(" => Strings are equal");
else 
    puts(" => Strins are not equal");

// ------ CERN's Root REPL Session -------// 
root [0] #include <string.h>

root [1] const char* s1 = "hello"; 
root [2] const char* s2 = "world"; 
root [3] const char* s3 = "world C++17 c11 ADA spark"; 
root [4] const char* s4 = "hello"; 

root [5] strcmp(s1, s2)
(int) -15

root [6] strcmp(s1, s3)
(int) -15

root [7] strcmp(s1, s4)
(int) 0
  • strcasecmp() => Compare two string ignoring case sensitivity. This function returns 0 if the string arguments are equal. Othewise, it returns anything other than 0.
int strcasecmp(const char *string1, const char *string2);

// ---- USAGE ------------------------------------------// 

if( strcasecmp(stringA, stringB) == 0 )
   printf(" Strings are equal Ok. \n");
else 
   printf(" Strings are not equal. \n");

// ----- CERN's ROOT REPL session ----------------------// 

// Strings not equal 
root [21] strcasecmp("app", "Application")
(int) -108

// Strings are equal 
root [22] strcasecmp("application", "Application")
(int) 0

root [23] strcasecmp("AppLiCatION", "Application")
(int) 0
root [24]
  • sprintf() => Print string to a buffer. (Headers: <cstdio> in C++ or <stdio.h> in C). Note: this function is vulnerable to buffer overflow, if the data copied to the buffer is larger than its size.
int sprintf(char *str, const char *format, ...);

// ---- CERN's ROOT REPL session ----------------------// 

root [1] char buffer[300];
root [2] memset(buffer, 0x00, 300)

root [4] sprintf(buffer, "The root of %d is equal to %f", 2, 2.0);

root [5] buffer
(char [300]) "The root of 2 is equal to 2.000000\0\0\0\0\0\0\0\.... ... \0"

root [7] std::cout << " buffer = <" << buffer << ">" << '\n';
buffer = <The root of 2 is equal to 2.000000>
  • snprintf() => Print string to a buffer limiting the number of characters that are copied to the buffer. The limitation of the number characters copied helps protecting against buffer overflow vulnerabilities.
int snprintf(char *str, size_t size, const char *format, ...);

// ---- CERN's ROOT REPL session ----------------------------//
root [0] char buf[200];
root [1] memset(buf, '\0', 200); // Initialize buffer

root [6] snprintf(buf, 10, "The square root of %f is equal to %.3f ", 10.0, sqrt(10.0));
root [7] 
root [7] buf
(char [200]) "The squar\0\0\0\0\0... ...\0\0\0"

root [8] snprintf(buf, 25, "The square root of %f is equal to %.3f ", 10.0, sqrt(10.0));
root [9] 
root [9] buf
(char [200]) "The square root of 10.00\0\0\0...\0\0"


root [10] snprintf(buf, 125, "The square root of %f is equal to %.3f ", 10.0, sqrt(10.0));
root [11] 
root [11] buf
(char [200]) "The square root of 10.000000 is equal to 3.162 \0\0\0..\0\0"
  • dprintf => Print to file descriptor. It can be used for formatted printing to any file descriptor, including sockets, pipes, regular files and standard streams (stdout and stderr).
int dprintf(int fd, const char *format, ...);

// -- EXAMPLE =>> CERN's ROOT REPL Session ---------------// 

root [6] dprintf(STDOUT_FILENO, " [INFO] x = %d ; sqrt(3) = %.3f \n", 100, sqrt(3.0))
 [INFO] x = 100 ; sqrt(3) = 1.732 

root [5] dprintf(STDERR_FILENO, " [INFO] x = %d ; sqrt(3) = %.3f \n", 100, sqrt(3.0))
 [INFO] x = 100 ; sqrt(3) = 1.732 
(int) 35
  • asprintf => Print allocated string. Similar to snprintf, but it stores the output in a large enough heap-allocated (dynamically allocated) buffer.
int asprintf(char **strp, const char *fmt, ...);

// ==>> EXAMPLE =>> CERN's ROOT REPL Session ============//

// Output 
root [7] char* output;

root [8] asprintf(&output, " [TRACE] x = %d ; sin(3.1415) = %.5f ; s = %s \n", -261, sin(3.1415), "Hello world")
(int) 61

root [9] output 
(char *) " [TRACE] x = -261 ; sin(3.1415) = 0.00009 ; s = Hello world "

root [10] puts(output)
 [TRACE] x = -261 ; sin(3.1415) = 0.00009 ; s = Hello world 

// Release allocated memory., 
root [11] free(output)
root [12]
  • strcpy() => Copy string 'src' parameter (null-terminated array of characters) to a buffer.
    • Docs: $ man strcpy
    • Note: It is vulnerable to buffer overflow, if the size of the string is bigger than the buffer. This function is often used to for copying string literals to a buffer.
char* strcpy( char*       p_buffer      // Pointer to buffer (array of chars)
             ,const char* c_string_src  // C-string 
           );

root [9] char buffer[200];

root [10] buffer
(char [200]) "\0\0\0\0\0\0\0\0\0\0\0\0\0\0 ...... \0\0\0\0"

root [11] strcpy(buffer, "Hello world C++17 C++20 ADA Rust")
(char *) "Hello world C++17 C++20 ADA Rust"

root [12] buffer
(char [200]) "Hello world C++17 C++20 ADA Rust\0\0\0\0\0\0\0 ... ... \0\0\0"
  • strncpy() => Copy N characters from string to buffer.
    • Docs: $ man strncpy
    • Linux mapage: "The strcpy() function copies the string pointed to by src, including the terminating null byte ('\0'), to the buffer pointed to by dest. The strings may not overlap, and the destination string dest must be large enough to receive the copy. Beware of buffer overruns! (See BUGS.)"
char* strncpy( char*       dest // Pointer to buffer 
             , const char* src  // C-string (null-terminated) that will be copied 
             , size_t      n    // Number of chars (bytes) from src that will be copied
             );

// ---------- CERN's ROOT Repl Session ------------//

root [13] char buff1[30];
root [14] char buff2[30];

root [15] const char* cstr = "Hello world C++17 embedded systems real time control systems";

root [16] buff1
(char [30]) "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"

root [17] buff2
(char [30]) "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"

root [18] strncpy(buff1, cstr, 10)
(char *) "Hello worl"

root [19] buff1 
(char [30]) "Hello worl\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"

root [20] strncpy(buff2, cstr, 25)
(char *) "Hello world C++17 embedde"

root [21] buff2
(char [30]) "Hello world C++17 embedde\0\0\0"
  • strcat() => Manpage: "The strcat() function appends the src string to the dest string, over‐ writing the terminating null byte ('\0') at the end of dest, and then adds a terminating null byte. The strings may not overlap, and the dest string must have enough space for the result. If dest is not large enough, program behavior is unpredictable; buffer overruns (aka buffer overflow) are a favorite avenue for attacking secure program"
    • Note: This function is unsafe and can introduce buffer overflow vulnerability if the size of the src string is not limited.
char *strcat(char *dest, const char *src);

// ------------- CERN Root REPL  ------------------//
const char* null_terminated_char_array1= "C++1z KALMAN FILTER sensor fusion data";
char buffer1[100] = "Append cstring: ";

root [3] buffer1
(char [100]) "Append cstring: \0\0\0\0\0\0\0\0\0\0\0\0\0...\0\0"


root [4] strcat(buffer1, null_terminated_char_array1)
(char *) "Append cstring: C++1z KALMAN FILTER sensor fusion data"

root [5] buffer1
(char [100]) "Append cstring: C++1z KALMAN FILTER sensor fusion data\0\0....\0\0"

root [6] strcat(buffer1, " string literal ")
(char *) "Append cstring: C++1z KALMAN FILTER sensor fusion data string literal "

root [7] buffer1 
(char [100]) "Append cstring: C++1z KALMAN FILTER sensor fusion data string literal \0\0\0\0...\0\0"
  • strncat() => Similar to strcat, but it limits the number of character to be copied from src to a buffer.
char* strncat(char *dest, const char *src, size_t n);

// ------------ CERN's Root REPL session ----------------------// 
// 

// Stack-allocated buffer 
root [8] char buff[100];
root [11] memset(buff, 100, 0x00); // Initialize buffer

root [12] const char* str1 = "Hello world APL C++1z ADA RUST DLang OCaml";

root [13] buff
(char [100]) "\0\0\0\0\0\0\0\0\0\0\0\0\0\0...\0\0"

root [14] strncat(buff, str1, 10)
(char *) "Hello worl"


root [15] buff
(char [100]) "Hello worl\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0 .. \0\0"

root [16] strncat(buff, str1, 20)
(char *) "Hello worlHello world APL C++1"

root [17] buff
(char [100]) "Hello worlHello world APL C++1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0....\0\0"

root [18] strncat(buff, str1 + 25, 30)
(char *) "Hello worlHello world APL C++1 RUST DLang OCaml"

root [19] buff
(char [100]) "Hello worlHello world APL C++1 RUST DLang OCaml\0\0...\0\0"
  • strtok => Break string into tokens.
char* strtok(char* str, const char* delim);

// ---- Usage Example ------------------// 

root [8] const char* sep = ","; // Separators
root [9] char text[500] = "BRL , CAD ,, USD ,,, EUR , YEN , JPY";

root [10] char* tok = strtok(text, sep)
(char *) "BRL "

root [11] while( tok != nullptr) { printf(" => token = %s \n", tok); tok = strtok(nullptr, sep); }
 => token = BRL  
 => token =  CAD  
 => token =  USD  
 => token =  EUR  
 => token =  YEN  
 => token =  JPY
  • strtoul() => Attempt to parse an unsigned integer from a string. On failure, this function sets the errno variable to ERANGE.
unsigned long int strtoul (const char* str, char** endptr, int base);

// --------------- Example -----------------------// 

root [42] auto n = strtoul("1001001", nullptr, 2)
(unsigned long) 73

root [43] auto n = strtoul("1001001", nullptr, 10)
(unsigned long) 1001001

root [44] auto n = strtoul("1001001", nullptr, 16)
(unsigned long) 16781313

root [45] auto n = strtoul("1001001", nullptr, 8)
(unsigned long) 262657

root [46] auto n = strtoul("AFBC01", nullptr, 16)
(unsigned long) 11516929

// Failure 
root [47] auto n = strtoul("XAFBC01", nullptr, 16)
(unsigned long) 0

root [54] auto n = strtoul("AFBC01 rest of text", nullptr, 16)
(unsigned long) 11516929

root [55] auto n = strtoul("  AFBC01 rest of text", nullptr, 16)
(unsigned long) 11516929

Buffer Manipulation

  • memchr() => Locate a byte in a buffer or memory region. This function returns a pointer to the first occurency of the search byte and returns null pointer if the byte is not found.
 void* memchr(const void* buffer, int byte_searched, size_t buffer_size);

// ----- Example: CERN's ROOT REPL Session ----------------// 

root [19] char mybuffer[200] = "hello world 9 10 36 buffer byte array 9 0 %";

// Pointer to first occurrency of character '9' (byte)
root [22] p 
(char *) "9 10 36 buffer byte array 9 0 %"

root [23] p[0]
(char) '9'
root [24] 

// Position (aka index) where the byte is found 
root [35] ptrdiff_t idx = p - mybuffer 
(long) 12

root [37] mybuffer[idx]
(char) '9'

// Next ocurrency 
root [41] p = (char*) memchr(mybuffer + idx + 1, '9', 200 - idx)
(char *) "9 0 %"
  • memset() => Fill some memory area (buffer) with a constant byte.
void*  memset(void *s, int c, size_t n);

// ------ CERN's ROOT REPL session -------------// 
// 

// Create a heap-allocated buffer containing 6 characters. 
root [0] char* char_heap_buffer = (char*) malloc(6 * sizeof(char))
(char *) "@\x0e" "e\x01"

root [1] char_heap_buffer[0]
(char) '@'

root [2] char_heap_buffer[1]
(char) '0x0e'

root [3] char_heap_buffer[2]
(char) 'e'

// Intialize buffer char_hepa_buffer 
root [9] memset(char_heap_buffer, 'z', 6)
(void *) 0x128c760

root [10] char_heap_buffer[0]
(char) 'z'

root [11] char_heap_buffer[1]
(char) 'z'

root [12] char_heap_buffer[2]
(char) 'z'

root [13] char_heap_buffer[3]
(char) 'z'

root [14] char_heap_buffer 
(char *) "zzzzzz"

root [15] free(char_heap_buffer)
  • memcmp() => Compare the first N bytes of two buffers. It returns 0 if all the first N bytes from two buffers are equal. Note: This function can only compare buffers containing POD data (Plain-Old Data) without any internal pointer or heap-allocated data.
  int memcmp(  const void* s1 // Pointer to the beginning of buffer 1 
             , const void* s2 // Pointer to the beginning of buffer 2 
             , size_t n       // Number of bytes that will be compared 
           );  

// -------- CERN's ROOT REPL Session -----------------------// 
 root [22] uint8_t bf1[] = { 0x01, 0xAB, 0x2C, 0x3C, 0xFF };
 root [26] uint8_t bf2[] = { 0x01, 0xAB, 0x2C, 0x3C, 0xFF, 0x85, 0x2A, 0x45 };
 root [30] uint8_t bfx[] = { 0x3A, 0xF8, 0x25 } 

 // Result: Non-zero => First 3 bytes of two buffers are not equal. 
 root [32] memcmp(bf1, bfx, 3)
 (int) -57

 // Result: Zero => First 5 bytes of two buffers are equal. 
 root [36] memcmp(bf1, bf2, 5)
 (int) 0

 // Result: Zero => First 2 bytes of two buffers are equal. 
 root [37] memcmp(bf1, bf2, 2)
 (int) 0

 root [13] free(heap_buffer)
  • memcpy() => Copy N bytes from a memory area (or buffer) src to a memory area dest. This function can only copy POD (Plain Old Data) types, it cannot copy anything which contains pointers or heap-allocated data.
void* memcpy(void* dest, const void* src, size_t n);

// ------- CERN's ROOT Repl session ---------------// 
// 

// ---- Char array --------//
root [51] char buf1[10] = { 'h', 'e', 'l', 'l', 'o', ' ', 'c', 'p', 'p', '\0' }
(char [10]) "hello cpp"

root [53] char buf2[15] 
(char [15]) "\0\0\0\0\0\0\0\0\0\0\0\0\0"

root [54] memcpy(buf2, buf1, 10 * sizeof(char))
(void *) 0x7f5c5cd7d1b5

root [55] buf2
(char [15]) "hello cpp\0\0\0\0"

// ---- Int array -------//
root [56] int arr1[5] = { 10, 20, 25, 100, -2}
(int [5]) { 10, 20, 25, 100, -2 }

root [57] int arr2[10]
(int [10]) { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }

root [58] memcpy(arr2, arr1, 5 * sizeof(int))
(void *) 0x7f5c5cd7d1f0

root [59] arr2
(int [10]) { 10, 20, 25, 100, -2, 0, 0, 0, 0, 0 }
  • bzero() => Zero-initialize some memory area. In the words of manpage: "The bzero() function erases the data in the n bytes of the memory starting at the location pointed to by s, by writing zeros (bytes containing '\0') to that area."
void bzero(void *s, size_t n);

// ------------ CERN's ROOT REPL session --------------//
//    

// Heap-allocated buffer with 5 integers
root [1] int* heap_buffer = (int*) malloc(5 * sizeof(int)) 
(int *) 0x2c50ea0

root [2] heap_buffer[0]
(int) 37832720

root [3] heap_buffer[1]
(int) 0

root [4] heap_buffer[2]
(int) 51513872

// Initialize buffer, filling it with zeroes. 
root [7] bzero(heap_buffer, 5 * sizeof(int))

root [8] heap_buffer
(int *) 0x2c50ea0

root [9] heap_buffer[0]
(int) 0

root [10] heap_buffer[1]
(int) 0

root [11] heap_buffer[2]
(int) 0

root [12] heap_buffer[4]
(int) 0
  • bcopy() => Manpage: "The bcopy() function copies n bytes from src to dest. The result is correct, even when both areas overlap."
    • Note: Despite the manpage state that the function is deprecated it is still used by many legacy codes, so it is still worth knowing about it.
void bcopy(const void *src, void *dest, size_t n);

// ----------- CERN's ROOT Repl Session --------------// 

root [0] char buffer[50];
root [1] bzero(buffer, 50 * sizeof(char))

root [2] buffer 
(char [50]) "\0\0\0\0\0\0\0\0\0\0\0... ...\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"

// Null terminated array of chars
root [3] const char* src1 = "This is the buffer 1"; 

root [4] bcopy(src1, buffer, 10)

root [5] buffer
(char [50]) "This is th\0\0\0\0\0\0\0\0... ...0\0\0\0\0\0\0"

root [6] bcopy(src1, buffer, 25)

root [7] buffer
(char [50]) "This is the buffer 1\0\0\0\0\x14\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"

root [8] bcopy("control system Modelica / state space model / SIMULINK", buffer, 40)

root [9] buffer
(char [50]) "control system Modelica / state space mo\0\0\0\0\0\0\0\0"

1.4 Map of user-space APIs of Unix-like Operating Systems

It is worth knowing where to find common Unix and POSIX (Portable Operatin System Interface) C APIs, which are common to many UNIX-based operating systems, such as Linux, BSD-variants, MacOSX, QNX and so on. In order to enhance the discoverability, this section lists selected bookmarks to widely used Unix, POSIX and system-specific Application Programing Interfaces (APIS).

Primitive IO (Read, Open, Close - Sytem Calls)

  • open(2) - Open a file returning a file descriptor.
  • read(2) - Read data from a file descriptor to a buffer.
  • write(2) - Write data from buffer to file descriptor.
  • close(2) - Close file descriptor.
  • lseek(2) - Position offset for reading writing operation. This useful for binary files, where some data are located at specific positions.
  • Scatter, gather IO - (reav, writev) - Scatter, Gatter or Vectored IO.

File Descriptor Manipulation

System Information and User Accounts

Network / BSD Sockets

  • IPv4 Protocol - Internet Protocol Version 4
  • IPv6 Protocol - Internet Protocol Version 6
  • Unix Domain Socket - Inter-process communication which is a replacement for IP protocol on local machine. Unix Domain Sockets has less overhead than IP protocol on local host. This protocl is useful for providing network transparency. This API is used by Docker daemon on local machine, DBUS (Inter Process Communication BUS common on Linux Desktops systems) and X11 (X Windows Systems).
  • TCP Protocol - Transmission Control Protocol (Most used protocol over the internet.)
  • UDP Protocol - User Datagram Protocol
  • SCTP Protocol - (Internet Stream Control Transmission Protocol) - New protocol, combining the realibility of TCP with message boundary feature of UDP. Note: This type of protocol is still not availability on Windows NT or Windows CE operating systems, therefore the usage of SCTP is not portable yet.
  • socket(2) - Create a socket endpoint.
  • recv(2) - (read) Receive data from socket.
  • send(2) - (write) Send data to socket
  • read(2) - The read() system call also works with sockets.
  • write(2) - Write sycall also works with sockets. It sends data from some buffer to socket.
  • connect(2) - Used by a TCP client socket to initiate connection to a TCP server socket (a.k.a passive socket or listening socket) running in local or remote machine.
  • accept(2) - Used by a TCP server socket (a.k.a passive socket) to accept connection from a client socket.
  • bind(2) - Used by a TCP server socket to bind to some TCP port (16 bits unsigned number from 0 to 65535). Note: Ports which number is less than 1024 requires administrative or root privilege.
  • listen(2) - (Server socket) - Listen connection.
  • getsockopt(2) - Get and set socket options.
  • gethostbyname(3) - Query DNS, get IPv4 or IPv6 network address from hostname.
  • byteorder(3) (htonl, htons, nothl, ntohs) - Convert from/to network and host machine endianess for numerical values.

IO Multiplexing (aka non-blocking IO)

Note: As most of those APIs are not standardized. In order to ensure code portability, it is better to use C++ libraries that encapsulates multiplexed IO (Input/Output) APIs rather than use them directly. Some of those libraries are: Boost.ASIO (C++ library) and LibUV (C library).

Standardized APIs:

  • select(2) - (Standardized, common to Linux, BSD and MacOSX)
  • poll(2) (Linux Docs); poll (BSD Docs) - (Standardized, common to Linux, BSD and MacOSX)

Non-standardized APIs:

  • epoll(7) - [BEST] Linux-only API for IO multiplexing API.
  • kqueue - [BEST] BSD-Only API for IO multiplexing. This API is available at: FreeBSD; NetBSD; OpenBSD; DragonFlyBSD and MacOSX.

Threads and Concurrency

Process Control APIs

  • execve(2) => Run a process (Execv system call)
  • fexecve(3) => Similar to execve, but executes program (any Unix native-code executable or script starting with shebgang #! such as #!/usr/bin/python …) from file descriptor.
  • fork() => Fork system call, often used alongside Execv.
  • exit(3) => Exit system call. Force termination of current process.
  • wait(2) => Wait for a process.
  • getpid() => Get process unique identifier, PID
  • getppid() => Get parent process PID.
  • getpgrp() => Get process group IDs
  • getsid(2) => Get process session ID.
  • chdir => Current working directory
  • environ(7) => All environment variables from current process.
  • kill(2) => Send UNIX signal to process. It can also be used to request or force process termination.

IPC - Inter Process Communication (Except socket)

  • mmap(2) (BSD Docs) => Map file descriptor to process virtual memory or allocate memory. This API can also be used for shared memory inter-process communication.
  • mmap(2) (Linux Docs)
  • mmap (Blackberry)

Memory Management

Dynamic Loading of Shared Libraries at Runtime

Note: shared libraries (Windows DLLs - Dynamic Linked Libraries) are called SO (Shared Object) or DSO (Dynamic Shared Object) on Unix-based systems.

  • dlopen(3) - Load shared library in process address space.
  • dlclose(3) - Close shared library.
  • dlerror(3) - Get error diagnostics for dlopen() API.
  • dlsym(3) - Load symbol, function pointer or variable, from shared library.
  • dlvsym(3) - Similar to dlsym, but allows loading an specific version of a symbol.
  • dlinfo(3)
  • dladdr(3)
  • dl_iterate_phdr(3) - Iterate over the list of shared libraries loaded by the current process.

Functionalities for supporting containers and sandboxing

Containers are a lightweight virtualization alternative to virtual machines, that uses fewer system resources, such as memory and CPU time as unlike virtual machines, containers share the same kernel. Containers are nothing more than sandboxed processes with restricted access to system resources. On Linux, container runtimes, such as Docker, Podman (Red-Hat) or LXC implement containers with the following APIs: cgroups API for restricting CPU time and memory usage; chroot or restricting the directories that a sandboxed processes can access and making them look like the process' root directory; Linux user, pid, mount, network and namespaces for restricting user privilege, access to system directories and network.

Low Level Terminal API

Terminal Escape and Control Sequences

Articles related to pseudo-terminal device API:

Utilities for PTY - Pseudo-Terminal Interface

Linux-specific (VFS) Virtual File System APIs

GENERAL:

SYSFS VFS:

  • SYSFS - VFS that exports Kernel objects. Some files in SYSFS are mapped to real hardware such as GPIOs; I2C; CAN bus; PCI Bus; USB bus and so on.
  • The sysfs Filesystem - Patrick Mochel

PROCFS VFS:

  • PROCFS - Procfs /proc file system.
  • /PROC/SYS - Directory for manipulating Kernel runtime parameters (sysctl tool)

TMPFS VFS:

Linux System Calls (Assembly)

System V Calling Conventions

Calling conventions adopted by most Unix-like Operating Systems for C-compatible APIs.

1.5 Creating Portable Binaries for Linux / GLIBC Dependency Hell

1.5.1 Overview

One of the greatest challenges of Linux desktop is building and deploying portable applications and native executable which work on any Linux distribution. Even if an application totally statically linked against all dependencies or it is bundled with all its shared libraries dependencies via LD_LIBRARY_PATH environment variable or RPATH, the program may still fail to run in other Linux distribution due to the GLIBC (GNU C library runtime) dependency. A GLIBC is forward compatible, but not backward compatible, the GLIBC runtime failure is more likely to happen if the application was built on a Linux distribution using a newer version of GLIBC, but the program is deployed in a distribution using an older version of GLIBC.

  • Side Note 1:
    • Applications built with GO (Golang) programming language are less likely to be affected by the GLIBC-issue, as GO compiler often does not link against the GLIBC and builds statically linked binaries which can be deployed everywhere.
  • Side Note 2:
    • The GLIBC issue does not only affect C++, all programming languages which generate native Linux executable, for instance, D-lang, Rust, OCaml, Haskell may also be affected due to the GLIBC compatibility problem. Besides executables, the GLIBC-dependency hell can also impact shared libraries, also known as share objects, and consequently native libraries used by programming languages such as Python, Ruby, Java and so on.
  • Side Note 3:
    • Even AppImage, which is a solution for distributing binaries on Linux is affected by the GLIBC-problem. As a result, the AppImage documentation recommends building AppImage on Linux distributions with older versions of GlibC.
  • Side Node 4:
    • The GLIBC can be bypassed if the application invokes system-calls directly by using assembly or inline assembly. On Linux, this is possible because the system calls are stable and well documented. This might be the approach used by Golang for avoiding the GLIBC dependency.

Possible Solutions:

  1. Link against an older version of GLIBC
    • Build the application in a system with the oldest possible version of GLIBC. System in this context means: a linux virtual machine; a chroot environment or a docker image with an older version of GLIBC.
  2. Link against MUSL (CRT - C Runtime Library)
    • The MUSL library, is an alternative to GLIBC, which was designed for embedded systems and static linking, allows building statically linked binaries that does not suffer from the GLIBC compatibility problems. The drawback of this procedure is that dynamic loading with dlopen API (dlopen, dlclose, dlsym functions) is still not supported.
  3. Deploy via Docker
    • Deploy the application via docker image. While this solution is acceptable for compilers, building environments, servers and command line applications, Docker is not suitable for games or Desktop GUI (Graphical User Interface) applications intended to be used by non-technical users.
    • Advantage:
      • Pack application, dependencies and configuration.
      • The deployment environment can be reproduced everywhere.
  4. Distribute as source and recompile from source on every deployment machine.
  5. Distribute the application via Linux distribution repositories
    • Distribute the application via official Linux distribution repositories by creating distribution-specific packages for every supported distribution. It means creating (.deb) Debian packages for Debian-based distributions; (.rpm) packages for Fedora and Centos; (.apk) alpine packages for Alpine.

Further Reading

General reading about Linux CRT (C Runtime Library), GlibC and alternatives:

  • API/ABI changes review for glibc
  • Comparison of C/POSIX standard library implementations for Linux
    • Brief: "The table below and notes which follow are a comparison of some of the different standard library implementations available for Linux, with a particular focus on the balance between feature-richness and bloat. I have tried to be fair and objective, but as I am the author of musl, that may have influenced my choice of which aspects to compare. Future directions for this comparison include detailed performance benchmarking and inclusion of additional library implementations, especially Google's Bionic and other BSD libc ports."
  • A turning point for GNU libc - LWN.net - Jonathan Corbet
    • Brief: "The kernel may be the core of a Linux system, but neither users nor applications deal with the kernel directly. Instead, almost all interactions with the kernel are moderated through the C library, which is charged with providing a standards-compliant interface to the kernel's functionality. There are a number of C library implementations available, but, outside of the embedded sphere, most Linux systems use the GNU C library, often just called "glibc." The development project behind glibc has a long and interesting history which took a new turn with the dissolution of its steering committee on March 26. … …"
  • C library system-call wrappers, or the lack thereof = LWN.net - Jonathan Corbet
    • Brief: "User-space developers may be accustomed to thinking of system calls as direct calls into the kernel. Indeed, the first edition of The C Programming Language described read() and write() as "a direct entry into the operating system". In truth, user-level "system calls" are just functions in the C library like any other. But what happens when the developers of the C library refuse to provide access to system calls they don't like? … "
  • ELF Symbol Versioning - Ulrich Drepper
    • Brief: "The symbol versioning implementation used on Linux with glibc 2.1 o up is an extension of Sun's versioning. It provides most of the functionality Sun has plus one decisive new element: symbol-level versioning with multiple definitions of a symbol. The implementation allows every DSO to either use versions for their symbols or not. … …"
  • Choosing System C library - Khem Raj (Comcast) - Embedded Linux Conference Europe 2014 Düsseldorf Germany.
  • Wikidev - C Library
    • Brief: "The C standard library provides string manipulation (string.h), basic I/O (stdio.h), memory allocation (stdlib.h), and other basic functionality to C programs. The interface is described in the C standard, with further additions described in POSIX as well as vendor extensions. On Unix platforms, the library is named libc and is linked automatically into every executable. … ."
  • Running new applications on old glibc - (lightofdawn)
    • Brief: "Glibc (short for GNU Libc, or GNU C Library) is a library that provides the interface between application programs and the Linux kernel. Although its official name is the "C" library (library for programs written in the "C" language), virtually all dynamically linked program binaries depend on it - it is the de-facto system library in almost all Linux operating systems. … "
  • linux - Multiple glibc libraries on a single host - Stack Overflow
    • Brief: "Multiple glibc libraries on a single host. My linux (SLES-8) server currently has glibc-2.2.5-235, but I have a program which won't work on this version and requires glibc-2.3.3. Is it possible to have multiple glibcs installed on the same host? This is the error I get when I run my program on the old glibc: …"
  • musl libc Wiki [EMBEDDED-SYSTEMS] (Alternative to GLIBC - allows creating fully statically linked and self-contained binaries)
    • Brief: "musl is a C standard library implementation for Linux. This is a wiki maintained by the enthusiastic user community of musl. Some of musl’s major advantages over glibc and uClibc/uClibc-ng are its size, correctness, static linking support, and clean code."
  • GitHub - ifduyue/musl - Unofficial mirror of MUSL (CRT)
  • musl libc - Projects using musl - List of projects using MUSL libC instead of GLIBC (GNU C Library)
  • The Newlib Homepage [EMBEDDED-SYSTEMS] (Alternative to GLIBC)
    • Brief: "Newlib is a C library intended for use on embedded systems. It is a conglomeration of several library parts, all under free software licenses that make them easily usable on embedded products. Newlib is only available in source form. It can be compiled for a wide array of processors, and will usually work on any architecture with the addition of a few low-level routines."
  • GitHub - hwoarang/uClibc [EMBEDDED-SYSTEMS] (Alternative to GLIBC)
    • Brief: "uClibc (aka µClibc/pronounced yew-see-lib-see) is a C library for developing embedded Linux systems. It is much smaller than the GNU C Library, but nearly all applications supported by glibc also work perfectly with uClibc. Porting applications from glibc to uClibc typically involves just recompiling the source code. uClibc even supports shared libraries and threading. It currently runs on standard Linux and MMU-less (also known as µClinux) systems with support for alpha, ARM, cris, e1, h8300, i386, i960, m68k, microblaze, mips/mipsel, PowerPC, SH, SPARC, and v850 processors. …"

System Calls and GlibC:

Utilities to Patch and modify ELF binaries and other general utilities:

  • GitHub - NixOS/patchelf
    • Brief: "PatchELF is a simple utility for modifying existing ELF executables and libraries. In particular, it can do the following: Change the dynamic loader ("ELF interpreter") of executables; Change the RPATH of executables and libraries; Shrink the RPATH of executables and libraries."
  • GitHub - wheybags/glibc_version_header (Utility for ignoring ELF symbol version - note: it is not safe).
    • Brief: "Build portable Linux binaries, no more linker errors on users' older machines from incompatible glibc versions. Essentially, this is a tool that allows you to specify the glibc version that you want to link against, regardless of what version is installed on your machine. This allows you to make portable Linux binaries, without having to build your binaries on an ancient distro (which is the current standard practice). "
  • GitHub - emk/rust-musl-builder
    • Brief: ": Docker images for compiling static Rust binaries using musl-libc and musl-gcc, with static versions of useful C libraries. Supports openssl and diesel crates. "

Reports of GLIBC dependency hell (Version mismatch):

1.5.2 The GLIBC Problem

GLIBC - Problem Illustration with a Sample Project

GIST Containing all files used in this experiment:

File: Makefile 

build:
        cmake --config Debug -H. -B_build2
        cmake --build _build2 --target

File: CMakeLists.txt 

cmake_minimum_required(VERSION 3.9)
project(Simple_Cmake_Project)

set(CMAKE_CXX_STANDARD 17)     
set(CMAKE_VERBOSE_MAKEFILE ON)

#========== Targets Configurations ============#
        add_executable( filesys filesys.cpp )
target_link_libraries( filesys stdc++fs)

File: filesys.cpp 

#include <iostream>
#include <string>
#include <iterator>
#include <iomanip>
#include <filesystem>

namespace fs = std::filesystem;

template<typename Range, typename Function>
auto dotimes(size_t n, Range&& iterable, Function fun)
{
    size_t i = 0;
    auto it = fs::begin(iterable);
    auto end = fs::end(iterable);
    while(i < n && it != end ){
            fun(it);
            ++it;
            i++;
    }
}

int main(){

     std::cout << std::boolalpha;
     std::cout << "\n ===== Listing directory /etc =====" << std::endl;
     // Show first 10 files of directory /etc 
     dotimes(10, fs::directory_iterator("/etc"),
             [](auto p){
                  auto path = p->path();
                  std::cout << std::left
                            << std::setw(0) << path.filename().string()
                            << " " << std::setw(35)
                            << std::right << std::setw(40) << path
                            << std::endl;               
             });        
     return EXIT_SUCCESS;
}

Steps for reproducing the problem

STEP 1: Get current machine infomation.

  • GLIBC Version is: 2.34-2.fc32
# Current machine 
$ cat /etc/fedora-release 
Fedora release 32 (Thirty Two)

# Current GLIBC Version 
$ ld --version
GNU ld version 2.34-2.fc32
... ... ... ...

STEP 2: Build the application. 

$ git clone https://gist.github.com/0557cd0fa1d5370723b015e443a7c036 gist && cd gist 
$ cmake --config Debug -H. -B_build1
$ cmake --build _build1 --target 

$ file _build1/filesys 
_build1/filesys: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, 
interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=d1a3628f794c66bcfb29303cfa4a528910f4e3e2
, for GNU/Linux 3.2.0, not stripped

STEP 3: Run in the current machine (newer version of GLIBC) 

$ _build1/filesys 

 ===== Listing directory /etc =====
chrony.conf                       "/etc/chrony.conf"
xl2tpd                            "/etc/xl2tpd"
group                             "/etc/group"
profile.d                         "/etc/profile.d"
kde4rc                            "/etc/kde4rc"
resolv.conf.8LUAL0                "/etc/resolv.conf.8LUAL0"
geoclue                           "/etc/geoclue"
bash_completion.d                 "/etc/bash_completion.d"
cups                              "/etc/cups"
mime.types                        "/etc/mime.types"

STEP 4: Attempt to run the binary in a distribution using an older version of GLIBC. The distribution used was a QEMU-KVM virtual machine running MX-Linux 19.2, distribution based on Debian, with GLIBC version .

GLIBC Failure on MX-Linux Virtual Machine: 

#---------- Failure!! GLIC Incompa ------------------------------------#
 $ _build1/filesys 
 _build1/filesys: /lib/x86_64-linux-gnu/libstdc++.so.6: version 
  'GLIBCXX_3.4.26' not found (required by _build1/filesys)

Virtual Machine Information: 

#----------- Machine Info (MX Linux 19.2) -------------#
# Kernel version 
$ uname -r
4.19.0-9-amd64

# Distro name 
$ cat /etc/issue 
Welcome to MX 19.2 (patito feo) 64-bit! Powered by Debian.
... ... ... ... ............ .... ... ... .. 

$ ld --version
GNU ld (GNU Binutils for Debian) 2.31.1
... ... ... ... ... ... ... ... ... ... ... ...

1.5.3 Solution 1 - via linking against older GLIBC

As GLIBC is forward compatible, but not backward compatible. The solution for this issue is to build the application on a system with the oldest possible version of GLIBC. System in this context, means: a Linux virtual machine; a Chroot environment; or Docker image with an older version of GLIBC. Other alternative is to replace GLIBC with MUSL CRT (C Runtime library), but the drawback is that MUSL does not support dynamic loading (_dlopen_, dlsym APIs).

The solution adopted for solving this issue was to use a Docker image based on holy-build-box, which is a Docker image based on Centos 6 Linux distribution containing a modern version of GCC compiler, CMake and an older version of GLIBC.

See:

File: Dockerfile (Available at gist url) 

# $ docker run -it --rm -v $PWD:/work -w /work phusion/holy-build-box-64:latest bash 
FROM phusion/holy-build-box-64:latest

ENTRYPOINT [ "/hbb_exe/activate-exec", "bash"  ]

Build docker image: 

# Run at the directory where is Dockerfile 
$ docker build -f Dockerfile -t linux-build .

Enter in the docker image shell: 

$ docker run --rm -it -v $PWD:/cwd -w /cwd linux-build 
Holy build box activated
Prefix: /hbb_exe
CFLAGS/CXXFLAGS: -g -O2 -fvisibility=hidden -I/hbb_exe/include 
LDFLAGS: -L/hbb_exe/lib -static-libstdc++
STATICLIB_CFLAGS: -g -O2 -fvisibility=hidden -I/hbb_exe/include 
SHLIB_CFLAGS: -g -O2 -fvisibility=hidden -I/hbb_exe/include 
SHLIB_LDFLAGS: -L/hbb_exe/lib -static-libstdc++

[root@88c9630a2008 cwd]# ls
_build1  CMakeLists.txt   Dockerfile   filesys.cpp   Makefile

Building the application: 

[root@88c9630a2008 cwd]# make 
cmake --config Debug -H. -B_build2
-- Configuring done
-- Generating done
-- Build files have been written to: /cwd/_build2
cmake --build _build2 --target 
gmake[1]: Entering directory '/cwd/_build2'
/hbb/bin/cmake -S/cwd -B/cwd/_build2 --check-build-system CMakeFiles/Makefile.cmake 0
... ... ... ... ... ... ... ... ... 
... ... ... ... ... ... ... ... ... ... ... ... 

[root@88c9630a2008 cwd]# ls _build2
CMakeCache.txt  CMakeFiles  cmake_install.cmake  filesys  Makefile

Running the application _build2/filesys in MX-Linux: 

# Built on Fedora 32 with newer version of GLIBC [FAILURE]
$ _build1/filesys
_build1/filesys: /lib/x86_64-linux-gnu/libstdc++.so.6: 
version 'GLIBCXX_3.4.26' not found (required by _build1/filesys)


# Built on Centos6 docker with older version of GLIBC [OK, WORKS]
#--------------------------------------
$ _build2/filesys

 ===== Listing directory /etc =====
.java                             "/etc/.java"
.pwd.lock                         "/etc/.pwd.lock"
ImageMagick-6                     "/etc/ImageMagick-6"
NetworkManager                    "/etc/NetworkManager"
UPower                            "/etc/UPower"
X11                               "/etc/X11"
acpi                              "/etc/acpi"
adduser.conf                      "/etc/adduser.conf"
adjtime                           "/etc/adjtime"
alsa                              "/etc/alsa"

1.5.4 Solution 2 - via linking against MUSL libC

Other alternative to avoid the GLIBC dependency problem is to build the application statically linking against MUSL - LibC which is an CRT (C runtime library), initially developed for embedded systems, but that has become an alternative to GLIBC. The easiest way to build statically linking against MUSL is to use an Alpine docker image:

  • Note: CRT - is a central component of Unix-like operating systems. It encapsulates system-calls and other operating system services for user-space applications.

See:

File: MuslBuilder.docker (Available at gist) 

FROM alpine:latest
RUN apk add musl cmake make g++

Building:

Buildilg the docker containing MUSL and development tools: 

$ git clone https://gist.github.com/0557cd0fa1d5370723b015e443a7c036 && cd gist 
$ docker build -f MuslBuilder.docker -t musl-builder .

Running a container of this docker image: 

$ docker run -it --rm -v $(pwd):/cwd -w /cwd -e UID=$(id -u) -e GID=$(id -g) musl-builder 
/cwd $ cmake --config Debug -H. -B_build-musl -DCMAKE_EXE_LINKER_FLAGS="-static -Os"
/cwd $ cmake --build _build-musl --target

Exiting from the container and check the executable: 

$ du -h _build-musl/filesys 
8.7M    _build-musl/filesys
8.7M    total

$ file _build-musl/filesys 
_build-musl/filesys: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV)
, statically linked, with debug_info, not stripped

# Check dependencies 
$ ldd _build-musl/filesys 
        statically linked

Attempt to run from MX-Linux 19.2 (Older GLIBC) 

$ _build-musl/filesys 

 ===== Listing directory /etc =====
.java                             "/etc/.java"
.pwd.lock                         "/etc/.pwd.lock"
ImageMagick-6                     "/etc/ImageMagick-6"
NetworkManager                    "/etc/NetworkManager"
UPower                            "/etc/UPower"
X11                               "/etc/X11"
acpi                              "/etc/acpi"
adduser.conf                      "/etc/adduser.conf"
adjtime                           "/etc/adjtime"
alsa                              "/etc/alsa"

1.5.5 Portable binary using Golang

Applications built with GO programming language are not affect by GLIBC compatibility problems since the GO compiler implementation performs system calls without relying on the GLIBC. This feature allows deploying GO binaries in any Linux distribution without worrying about GLIBC binary compatibility.

Further Reading

Demonstration

File: goapp.go 

package main 

import (
    "fmt"
    "io/ioutil"
    "net/http"
    _ "bufio"
    _ "log"
)

func main() {

    fmt.Println(" => Started Ok")

    // ------- List root directory ---------//
    fmt.Println(" ---- List Root Directory ------")

    // No error handler for the sake of breviety. 
    files, _ := ioutil.ReadDir("/")

    for _, fn := range files { 
        fmt.Println(" => " + fn.Name() )         
    } 

    // ---- Http Request ------------------// 
    fmt.Println(" ------- Http Request -------------")
    resp, _ := http.Get("https://www.httpbin.org/get")
    body, _ := ioutil.ReadAll(resp.Body)
    fmt.Println(string(body))
}

Compile sample GO application: 

 # Compiled  
 $ go build goapp.go    

 # Remove debug symbols 
 $ strip goapp 

 # Get file size in megabytes    
 $ >> du -h goapp
 4.9M    goapp

 # Check binary 
 $ >> file goapp
 goapp: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter 
 /lib64/ld-linux-x86-64.so.2, 
 Go BuildID=rXh5UDymAYyqH6QHp4N5/KktyZonaNBes2MLq_8eV/4lmCJtW64uAqpYg_AIb_/5S4-otZPFzRnjGyq3bBL
, not stripped

# Check shared libraries dependencies 
 $ >> ldd goapp
        linux-vdso.so.1 (0x00007ffee9b99000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc87c0c6000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fc87befc000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fc87c10d000)

Run application in Fedora 32 distribution (Newer version of GLIBC): 

 $ >> ./goapp 
 => Started Ok
 ---- List Root Directory ------
 => .autorelabel
 => bin
 => boot
 => dev
 => etc
 ... ... ... ... ... ... ... ... ... 
 ... ... ... ... ... ... ... ... ... 
 => usr
 => var
 ------- Http Request -------------
{
  "args": {}, 
  "headers": {
    "Accept-Encoding": "gzip", 
    "Host": "www.httpbin.org", 
    "User-Agent": "Go-http-client/2.0", 
    "X-Amzn-Trace-Id": "Root=2-5f5650e9-4830a22ac821b598c22defbc"
  }, 
  "url": "https://www.httpbin.org/get"
}

Run application in MX-Linux distribution, based on Debian (Older version of GLIBC): 

$ ./goapp
 => Started Ok
 ---- List Root Directory ------
 => .cache
 => .config
 => .fehbg
 => bin
 => boot
 ... ... ... ... .... ... ... ... ... .... ... 
 ... ... ... ... .... ... ... ... ... .... ... 
 => var
 => vmlinuz
 ------- Http Request -------------
{
  "args": {}, 
  "headers": {
    "Accept-Encoding": "gzip", 
    "Host": "www.httpbin.org", 
    "User-Agent": "Go-http-client/2.0", 
    "X-Amzn-Trace-Id": "Root=1-5f56523a-6cfd5015acb4421e26947afe"
  }, 
  "url": "https://www.httpbin.org/get"
}

1.6 Fully Statically linked executables for embedded Linux systems

1.6.1 Overview

This sections presents a procedure for cross-compiling a fully statically linked executable, without any dependency, that can be deployed on any ARMV7-based Embedded Linux system, including Beaglebone Black development board and Android (Armv7 based processor).

This procedure uses muslcc/i686:arm-linux-musleabi docker image which contains a cross-compiling toolchain capable of compiling C or C++ applications for Linux kernel running on Armv7 based processors. This toolchain also uses Musl libC, instead of glibC (GNU C runtime library). Musl allows building fully statically applications that does not rely on any system dependency. As a result, applications, built with Musl, can be run on any Linux distribution without being affected by GNU GlibC mismatch problems which arises when an application or object-code, that was built linked against an newer version of GlibC, is run on a system using an older version of GlibC resulting in a runtime linking error.

References and reading:

1.6.2 Project Files

All files are available at:

File: Dockerfil.docker 

FROM muslcc/i686:armv7l-linux-musleabihf
RUN  apk add make cmake 

# ENTRYPOINT ["cmake"]

File: Makefile

  • Makefile for running project commands and documenting the building process.
DOCKER_COMMAND=docker run -e UID=$(shell id -u) -e GID=$(shell id -g) --volume=$(shell pwd):/cwd  --workdir=/cwd musl-armv7

# Builds docker image: musl-armv7
docker:
        docker build -f Dockerfile.docker -t musl-armv7 .

# Build testing application 
build:
        ${DOCKER_COMMAND} cmake -H. -B_build-armv7 -DCMAKE_BUILD_TYPE=Debug
        ${DOCKER_COMMAND} cmake --build _build-armv7 --target 

clean:
        rm -rf -v _build-armv7

File: CMakeLists.txt 

cmake_minimum_required(VERSION 3.9)
project(embedded-linux-armv7)

set(CMAKE_CXX_STANDARD 17)     
set(CMAKE_VERBOSE_MAKEFILE ON)

#====== Global compiler and linker settings ======#

# (-static)           => Fully statically link Musl-LibC
# (-Wl,--gc-sections) => Remove unused code
#                        (requires: -fdata-sections and -ffunction-sections)
set(CMAKE_EXE_LINKER_FLAGS "-static -Wl,--gc-sections")

# Separate data and code sections in order to reduce binary size. 
add_definitions( -fdata-sections -ffunction-sections )

# Copy target executable, for instance my-exe to my-exe.debug 
# and strip debugging symbols from my-exe.
# 
# Reference: https://cmake.org/pipermail/cmake/2011-October/046743.html
macro(STRIP_DEBUG_SYMBOLS target)
  ADD_CUSTOM_COMMAND(TARGET ${target} POST_BUILD
    COMMAND ${CMAKE_COMMAND} -E copy $<TARGET_FILE:${target}> ${CMAKE_BINARY_DIR}/${target}.debug
    COMMAND ${CMAKE_STRIP} -g $<TARGET_FILE:${target}>
  )
endmacro()

#========== Targets Configurations ============#

        add_executable( app portable-app.cpp )
 target_link_libraries( app stdc++fs)
   STRIP_DEBUG_SYMBOLS( app )

File: portable-app.cpp 

#include <iostream>
#include <string>
#include <algorithm>
#include <iomanip>
#include <filesystem>
#include <fstream> 
#include <cassert> 

namespace fs = std::filesystem;

int main(int argc, char** argv)
{

  assert( argc >= 2 && "Supposed to have at least one arguments");
  std::cout << std::boolalpha;
  std::string command = argv[1];

  if(command == "version")
  { 
      auto ifs = std::ifstream("/proc/version");
      assert( ifs.good() && "File supposed to exist" );  
      std::cout << ifs.rdbuf();
      return EXIT_SUCCESS;
  }

  std::cout << "\n ===== Listing directory /etc =====" << std::endl;

  if(command == "list")
  {
      // Show first 10 files of directory /etc 
    auto iter = fs::directory_iterator(argv[2]);

    std::for_each(fs::begin(iter), fs::end(iter)
                 ,[](auto p){
                   auto path = p.path();
                   std::cout << std::left
                             << std::setw(0) << path.filename().string()
                             << " " << std::setw(35)
                             << std::right << std::setw(40) << path
                             << std::endl;              
                 });    
    return EXIT_FAILURE;    
  }

  return EXIT_SUCCESS;
}

1.6.3 Building and running

STEP 1: Downloading the source code. 

$ git clone https://gist.github.com/b411be3b2a4364c7d4de18edd37da6a3 musl-armv7 && cd musl-armv7

STEP 2: Building the docker image: 

$ docker build -f Dockerfile.docker -t musl-armv7 .

Sending build context to Docker daemon  7.168kB
Step 1/2 : FROM muslcc/i686:armv7l-linux-musleabihf
 ---> 4141d3dcca3f
Step 2/2 : RUN  apk add make cmake
 ---> Using cache
 ---> c18ba0aa2335
Successfully built c18ba0aa2335
Successfully tagged musl-armv7:latest

STEP 3: Cross-compile the sample application:

Cross-compiling from command line: 

# Run the docker shell (sh)
$ docker run --interactive --tty -e UID=$(id -u) -e GID=$(id -g) --volume=$(pwd):/cwd  --workdir=/cwd musl-armv7
/cwd # 

# Build the application 
/cwd $ cmake -H. -B_build-armv7 -DCMAKE_BUILD_TYPE=Debug
/cwd $ cmake --build _build-armv7 --target

# Exit the docker shell 
/cwd $ cmake --build _build-armv7 --target

Cross-compile in a single step using the helper Makefile: 

$ make build

Check the compiled executables: 

$ ls _build-armv7/
CMakeFiles/  app*  app.debug*  CMakeCache.txt  cmake_install.cmake  Makefile

# Version with debug symbols 
$ file _build-armv7/app
_build-armv7/app: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, not stripped

$ du -h _build-armv7/app
736K    _build-armv7/app
736K    total

# Version without debugging symbols 
$ file _build-armv7/app.debug 
_build-armv7/app.debug: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV)
, statically linked, with debug_info, not stripped

$ du -h _build-armv7/app.debug 
9.4M    _build-armv7/app.debug
9.4M    total

STEP 4: Attempt to run the built executable on different machines.

Attempt to run the previous executables on the host machine (x86-64 processor - 64 bits): 

$ _build-armv7/app
bash: _build-armv7/app: cannot execute binary file: Exec format error

$ _build-armv7/app.debug 
bash: _build-armv7/app.debug: cannot execute binary file: Exec format error

Run ARMv7 Linux binary on a x86-64 host machine using QEMU emulator: (OwnYourBits) 

# Install Fedora Packages 
$ sudo dnf install -y qemu-user.x86_64 qemu-user-static.x86_64

# Run the Armv7 binary using QEMU-user 
$ qemu-arm ./_build-armv7/app version 
Linux version 5.8.4-200.fc32.x86_64 (mockbuild@bkernel01.iad2.fedoraproject.org) ... ... 

$ qemu-arm ./_build-armv7/app list /

 ===== Listing directory /etc =====
srv                                   "/srv"
sys                                   "/sys"
opt                                   "/opt"
run                                   "/run"
media                                 "/media"
bin                                   "/bin"
.autorelabel                          "/.autorelabel"
root                                  "/root"
var                                   "/var"
etc                                   "/etc"
tmp                                   "/tmp"

Run application in Beaglebone Black (Armv7) embedded linux development board, similar to Raspberry-Pi, but with more IO ports, peripherals and ADC analog-to-digital converters: 

debian@beaglebone:~$ file app
app: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, not stripped

debian@beaglebone:~$ du -h app
736K    app

debian@beaglebone:~$ file app
app: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, not stripped
debian@beaglebone:~$ 

debian@beaglebone:~$ ./app 
Assertion failed: argc >= 2 && "Supposed to have at least one arguments" (/cwd/portable-app.cpp: main: 14)
Aborted
debian@beaglebone:~$ 

debian@beaglebone:~$ ./app version 
Linux version 4.14.71-ti-r80 (root@b2-am57xx-beagle-x15-2gb) (gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)) #1 SMP PREEMPT Fri Oct 5 23:50:11 UTC 2018
debian@beaglebone:~$ 

debian@beaglebone:~$ ./app list /

 ===== Listing directory /etc =====
sys                                   "/sys"
opt                                   "/opt"
sbin                                  "/sbin"
home                                  "/home"
dev                                   "/dev"
nfs-uEnv.txt                          "/nfs-uEnv.txt"
usr                                   "/usr"
bin                                   "/bin"
etc                                   "/etc"
proc                                  "/proc"
boot                                  "/boot"
 ... ... ...  ... ... ...  ... ... ... 
 ... ... ...  ... ... ...  ... ... ...

Run in Android Armv-7 phone via ADB (Android Debug Bridge): 

# Transver application to phone via ADB 
 $ adb push _build-armv7/app.debug /data/local/tmp 
_build-armv7/app.debug: 1 file pushed. 11.1 MB/s (9793936 bytes in 0.842s)

# Enter ADB shell and run the application. 
 $ >> adb shell 
bali:/ $ cd /data/local/tmp

bali:/data/local/tmp $ file app.debug                                                                                                       
app.debug: ELF executable, 32-bit LSB arm, static, not stripped

bali:/data/local/tmp $ du -h app.debug
9.3M    app.debug

134|bali:/data/local/tmp $ ./app.debug version                                                                                              
Linux version 4.4.146+ (jenkins@jenkins-bali-wxg)  ... ... 
bali:/data/local/tmp $ 


1|bali:/data/local/tmp $ ./app.debug list /dev                                                                                              

 ===== Listing directory /etc =====
wmtWifi                           "/dev/wmtWifi"
stpgps                            "/dev/stpgps"
fm                                "/dev/fm"
stpbt                             "/dev/stpbt"
stpwmt                            "/dev/stpwmt"
wmtdetect                         "/dev/wmtdetect"
radio                             "/dev/radio"
ttyGS3                            "/dev/ttyGS3"
ttyGS2                            "/dev/ttyGS2"
ttyGS1                            "/dev/ttyGS1"
ttyGS0                            "/dev/ttyGS0"
mtp_usb                           "/dev/mtp_usb"
usb_accessory                     "/dev/usb_accessory"
 ... ... ...  ... ... ...  ... ... ...  ... ... ...

1.7 Creating command line applications with CLI11 C++ Library

1.7.1 Overview

Despite that GUI Graphical user interfaces applications are more used nowadays than CLI command line applications, it still worth developing CLI applications. Among other things, the benefits of CLI applications over GUI can be summarized as:

  • More precision
    • They are more precise than GUI for performing task. It is easier to guide someone by showing the command line steps rather than describing all buttons, menus, context menus that someone should click in order to accomplish some task.
  • Scriptability
    • CLI applications are scriptable and they can be automated; called from other applications or scripts.
  • Reproducibility
    • Unlike GUIs, CLI applications are reproducible and the sequence of steps for performing some task can be stored as text and later replayed by pasting the commands in a terminal or running them from a script.
  • They can be combined
    • Unlike GUI, CLI applications can be combined with each other via stdout, stderr redirection. Example: CLI tools such as 'grep', 'echo', 'ls', 'find', 'awk', 'sed' are often combined with each other in shell scripts for system configuration or automation.

Anotomy of CLI Applications

Simple command line application: 

$ ./<COMMAND> <P0> <P1> ... <PN> --flag0 --flag1 --switch0 value0 --switch1 value1 --switch value2 ... 

# Or: 
$ ./<COMMAND> <P0> <P1> ... <PN> \
     --flag0 --flag1 \
     --switch0 value0 --switch1 value1 --switch value2 ... 

$ ./<COMMAND> <P0> <P1> ... <PN> --flag0 --flag1 \
     --switch0=value0 --switch1=value1 --switch=value2 ...

Command line application with multiple subcommands: (i.e: git, docker) 

$ ./<COMMAND> <SUBCOMMAND> <P0> <P1> ... <PN> \
     --flag0 --flag1 \
     --switch0 value0 --switch1 value1 --switch value2 ... 

# OR: 
$ ./<COMMAND> <SUBCOMMAND> <P0> <P1> ... <PN> \
     --flag0 --flag1 \
     --switch0=value0 --switch1=value1 --switch=value2 ...

Where:

  • <COMMAND>
    • Appplication or executable.
  • <SUBCOMMAND>
    • Metaphor: an action or verb performed on subjects (positional arguments).
    • Subcommand, similar to 'git checkout' or'docker run'. A subcommand or command represents an
  • <P0>, <P1>, …, <PN>
    • Metaphor: subjects
    • Positional arguments (without '-' dash prefix) - they are essential (required) arguments for running an application. If they are not provided, an error happens.
  • -flag0 -flag1 ….
    • Metaphor: adjectives
    • A flag is an optional command line switch which represents a boolean value. The usage of a flag sets the represented value as true and absence, sets the represented value as false.
  • –switch0=value0 –switch1=value1 ….
    • Metaphor: adjectives
    • Switches are optional command line arguments associated to some program state.

Example - Breaking down command line parts: 

$ docker run -i -t --rm --volume=$PWD:/cwd -w=/cwd xdxd/docker-binwalk bash
  • Command:
    • docker
  • Sub-command:
    • run
  • Positional arguments:
    • (xdxd/docker-binwalk) and (bash)
  • Flags:
    • (-i), (-t), (–rm)
  • Switches:
    • (–volume), (-w)

General recommendations

  • DO ONLY ONE THING => The application should only do one thing. If it needs to perform more than one task, the best choice is to create a subcommand for every task.
  • STATUS CODE => The status code allows scripts or any other application calling the current one to know whether the process execution was successful.
    • => Return status code '0' (EXIT_SUCCESS) when the applications is terminated successfully
    • => Return any other value different than zero in case of failure.
  • DEFAULT VALUES
    • Use reasonable default values for command line switches values and flags.
    • Show the default values of command line switches
  • EXAMPLES
    • Add commands that shows usage examples.
  • SIMPLE TASKS SHOULD BE SIMPLE
    • Create subcommands for the most common use-cases for making simple tasks simple: make simple things simple and complex things possible (borrowed from Alan Kay).
  • STDERR - for non essential information
    • Print non essential, logging or debug information to stderr, standard error output as it allows discarding separating the stdout (standard output) from stderr, redirecting stderr to file or discarding it by redirecting stderr to /dev/null.
  • VERSION (–version)
    • Add switch for printing application version.
  • DRY-RUN (–dry-run)
    • Create a flag option (–dry-run) if the command performs any permanent non-reversible change, such as remove file. When the application receives the –dry-run option, the application only shows the non-reversible actions that would be executed, without actually performing them.

Subcommand for common tasks (Simple tasks should be simple):

For instance, an alternative docker client could have a subcommand 'shell' for just running the shell as entrypoint. $ dockw shell IMAGE could be translated as the following docker command line. The subcommand 'shell' could assume that the current directory ($PWD) is mounted to the directory /cwd in the docker image and this option could be changed using the command line switch --workdir=<SOME_DIR>. 

# 'dockw shell MY_DOCKER_IMAGE ' subcommand from a hypothetical dockw tool 
# is translated as: 
$ docker run --rm -it -v $PWD:/cwd -w /cwd --entrypoint=sh MY_DOCKER_IMAGE

1.7.2 Sample application with multiple sub-comamnds

The following code contains a sample CLI application containing multiple sub-commands:.

GIST:

CLI11 Library Documentation:

Files

File: CMakeLists.txt 

cmake_minimum_required(VERSION 3.9)
project(cliapp)

#========== Global Configurations =============#
#----------------------------------------------#

set(CMAKE_CXX_STANDARD 17)     
set(CMAKE_VERBOSE_MAKEFILE ON)
set(CMAKE_CXX_EXTENSIONS OFF)

# ------------ Download CPM CMake Script ----------------#

## Automatically donwload and use module CPM.cmake
file(DOWNLOAD https://raw.githubusercontent.com/TheLartians/CPM.cmake/v0.26.2/cmake/CPM.cmake
                 "${CMAKE_BINARY_DIR}/CPM.cmake")
include("${CMAKE_BINARY_DIR}/CPM.cmake")

#----------- Add dependencies --------------------------#

CPMAddPackage(
    NAME cli11
    URL  https://github.com/CLIUtils/CLI11/archive/v1.9.0.zip
    DOWNLOAD_ONLY YES
)

include_directories( ${cli11_SOURCE_DIR}/include )

message([TRACE] " cli11_SOURCE_DIR =  ${cli11_SOURCE_DIR} ")

#----------- Set targets -------------------------------#

add_executable(cliapp cliapp.cpp)
target_link_libraries(cliapp stdc++fs)

install( TARGETS cliapp 
         RUNTIME DESTINATION bin)

File: cliapp.cpp 

#include <iostream>
#include <string> 
#include <vector> 
#include <functional>
#include <filesystem>
#include <functional>

#include <CLI/CLI.hpp>


using namespace std::string_literals;
namespace fs = std::filesystem;

struct Command_list
{
    std::string m_path = ".";
    bool m_filter_file = false;
    bool m_filter_dir  = false; 
    bool m_recursive   = false; 

    Command_list(CLI::App& app, std::string name)
    {
        this->install(app, name);
    }

    void install(CLI::App& app, std::string name)
    {
        CLI::App* cmd = app.add_subcommand(
            name, "List directory"
        );

        cmd->callback([this](){
            try {
                this->list_directory();
                // Status code  0 => OK finished successfuly 
                return true; 
            } catch(std::exception const& ex) {
                std::cout << " Error: " << ex.what() << '\n';
                return false; 
            }
        });

        cmd->add_option("<PATH>",        m_path,        "Directory to be listed." )->required();
        cmd->add_flag("-f,--file",       m_filter_file, "Only list files");
        cmd->add_flag("-d,--directory",  m_filter_dir,  "Only list directories");
        cmd->add_flag("-r,--recursive",  m_recursive,   "List direcotyr in recursive way");
    }

private:

    void list_directory()
    {
        if(m_filter_file && m_filter_dir)
            throw std::runtime_error("Error: --file and --dir flags cannot be simultaneously true.");

        using Predfun = std::function<bool (fs::path const&)>;
        auto predicate = Predfun{};

        auto iterate_dir = [&predicate](auto&& iterable){
            for(auto const& path :  iterable )
            {
                if(predicate(path)) std::printf("  => %s\n", path.path().c_str()); 
            }
        };

        using ptr = bool (*) (fs::path const&);

        if(!m_filter_file && !m_filter_dir) 
            predicate = [](fs::path const& p){ return true; };

        if(m_filter_file) 
            predicate = (ptr) &fs::is_regular_file;

        if(m_filter_dir)
            predicate = (ptr) &fs::is_directory;

        if(!m_recursive)
            iterate_dir( fs::directory_iterator(m_path) );
        else 
            iterate_dir( fs::recursive_directory_iterator(m_path) );
    }   
};

class MiniWebServer
{
    int         m_port = 8080; 
    std::string m_host = "0.0.0.0";
    std::string m_path = "."; 
    bool        m_auth = false; 
public:
    // Set server TCP Port 
    void set_port(int port         ){ m_port = port; }
    // Hostnames that server will listen to 
    void set_host(std::string host ){ m_host = host; }
    // Set path containing server data 
    void set_path(std::string path ){ m_path = path; }
    void set_auth(bool flag        ){ m_auth = flag; }

    int  get_port(){ return m_port; }

    void run_server()
    {
        std::printf(" [INFO] Running web server => port = %d ; host = %s; path = %s \n"
            , m_port, m_host.c_str(), m_path.c_str());
        std::printf(" [INFO] Server has authentication => %s \n", m_auth ? "TRUE" : "FALSE" );
    }
};


template<typename T, typename Klass>
auto bind_setter(Klass& cls, void (Klass::* setter) (T&&) )
{
    return [&](T&& q){ 
        (cls.*setter)( std::forward(q) ); 
    };
}

template<typename T, typename Klass>
auto bind_setter(std::shared_ptr<Klass> const& cls, void (Klass::* setter) (T) )
{
    return [cls, setter](T q){ ((*cls).*setter)(q); };
}

void command_server( CLI::App&  app)
{    

    CLI::App* cmd = app.add_subcommand(
        "server", "Run HTTP server for sharing files from some folder."
    );

    cmd->footer("Run local file sharing web server");

    // Created with shared_ptr in order to the object survive this scope
    // and avoid reference to destroyed object when it is referenced from callback. 
    auto server = std::make_shared<MiniWebServer>();

    cmd->callback([=]
    {
        std::cout << " [TRACE] ----- Callback invoked OK ------ \n";
        if(server->get_port() < 0 || server->get_port() > 65535)
        {
            std::cerr << " [ERROR] " << " Invalid TCP port range. " << '\n';
            // Returns non-zero status code
            return false;
        }
        server->run_server();
        return true;
    });

    cmd->add_option_function<std::string>(
          "<PATH>"
        , bind_setter<std::string>(server, &MiniWebServer::set_path)
        , "Directory to be shared in the web server."
    )->required();

    cmd->add_option_function<int>( 
        "-p,--port"
      , [=](auto q){ server->set_port(q); }
      , "Server TCP port, default: 8080"
    );

    cmd->add_option_function<std::string>( 
        "--host"
      , [=](auto q){ server->set_host(q); }
      , "Hostname that server will listen to (default: 0.0.0.0)"
    );

    cmd->add_flag( 
        "-a,--auth"
      // , [](auto q){ server->set_auth(q); }        
       , bind_setter<bool>(server, &MiniWebServer::set_auth)
       , "Shows the command line parameters passed to QEMU."
    );   
}

void command_version( CLI::App&  app, std::string name)
{
    auto cmd = app.add_subcommand(name, "Show application version.");

    cmd->callback([=]{
        std::printf("cliapp version 0.1 - Your favorite U-NIX swiss army knife \n");

        // Return true for idnicating successful status code 
        return true; 
    });
}

int main(int argc, char** argv)
{
    CLI::App app("cliapp");        
    app.footer("\n Command line demo toolbox.");

    command_version(app, "version");
    command_version(app, "v");
    Command_list cmd_list1(app, "ls");
    Command_list cmd_list2(app, "list");
    command_server(app); 

    // ----------- Parse Arguments ---------------//
    try
    {
        if(argc == 1){ 
            std::cout << app.help() << "\n";
            return 0;
         }
        app.require_subcommand();
        app.validate_positionals();
        app.parse(argc, argv);
    } catch(const CLI::ParseError &e)
    {

        return app.exit(e);
    } catch (std::exception& ex)
    {
        std::cout << ex.what() << std::endl;
        return EXIT_FAILURE;
    }

    return 0;
}

Building on MacOSX

Install GCC 9.0 via brew. Note: Apple's default Clang compiler does not have the C++17 file system library. 

$ brew install gcc@9

Get the code: 

$ git clone https://gist.github.com/caiorss/1b6826b9722ba07667eefed2f82d667e src 
$ cd src

Override CMake default compiler: 

$ export CC=/usr/local/Cellar/gcc@9/9.3.0/bin/gcc-9
$ export CXX=/usr/local/Cellar/gcc@9/9.3.0/bin/g++-9

Configuration step: 

$ cmake --config Release -H. -B_build
  ... .... ... ... .. ... ...
-- Checking whether CXX compiler supports OSX deployment target flag - yes
-- Check for working CXX compiler: /usr/local/Cellar/gcc@9/9.3.0/bin/g++-9
-- Check for working CXX compiler: /usr/local/Cellar/gcc@9/9.3.0/bin/g++-9 - works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- CPM: adding package cli11@ ()

[TRACE] cli11_SOURCE_DIR =  /Volumes/data/cliapp/_build/_deps/cli11-src 
-- Configuring done
-- Generating done
-- Build files have been written to: /Volumes/data/cliapp/_build

Building step: 

$ cmake --build _build --target all 
 ... ... ... ... ... ... ... ... 
 ... ... ... ... ... ... ... ... 
[ 50%] Building CXX object CMakeFiles/cliapp.dir/cliapp.cpp.o
/usr/local/Cellar/gcc@9/9.3.0/bin/g++-9   -I/Volumes/data/cliapp/_build/_deps/cli11-src/include  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk   -std=c++17 -o CMakeFiles/cliapp.dir/cliapp.cpp.o -c /Volumes/data/cliapp/cliapp.cpp
[100%] Linking CXX executable cliapp
/usr/local/Cellar/cmake/3.17.3/bin/cmake -E cmake_link_script CMakeFiles/cliapp.dir/link.txt --verbose=1
/usr/local/Cellar/gcc@9/9.3.0/bin/g++-9   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -Wl,-search_paths_first -Wl,-headerpad_max_install_names  CMakeFiles/cliapp.dir/cliapp.cpp.o  -o cliapp  -lstdc++fs 
[100%] Built target cliapp
/usr/local/Cellar/cmake/3.17.3/bin/cmake -E cmake_progress_start /Volumes/data/cliapp/_build/CMakeFiles 0

Check executable: 

$ file _build/cliapp
_build/cliapp: Mach-O 64-bit executable x86_64

Show command line help: 

$ _build/cliapp 

cliapp
Usage: [OPTIONS] [SUBCOMMAND]

Options:
  -h,--help                   Print this help message and exit

Subcommands:
  version                     Show application version.
  ls                          List directory
  list                        List directory
  server                      Run HTTP server for sharing files from some folder.


 Command line demo toolbox.

Show help for 'server' subcommands: 

$ _build/cliapp server -h
Run HTTP server for sharing files from some folder.
Usage: _build/cliapp server [OPTIONS] <PATH>

Positionals:
  <PATH> TEXT REQUIRED        Directory to be shared in the web server.

Options:
  -h,--help                   Print this help message and exit
  -p,--port INT               Server TCP port, default: 8080
  --host TEXT                 Hostname that server will listen to (default: 0.0.0.0)
  -a,--auth                   Shows the command line parameters passed to QEMU.

Run local file sharing web server

Show help for 'ls' subcommand: 

$ _build/cliapp ls -h
List directory
Usage: _build/cliapp ls [OPTIONS] <PATH>

Positionals:
  <PATH> TEXT REQUIRED        Directory to be listed.

Options:
  -h,--help                   Print this help message and exit
  -f,--file                   Only list files
  -d,--directory              Only list directories
  -r,--recursive              List direcotyr in recursive way


 Command line demo toolbox.

Run server subcommand: 

$ _build/cliapp server /var/www 
 [TRACE] ----- Callback invoked OK ------ 
 [INFO] Running web server => port = 8080 ; host = 0.0.0.0; path = /var/www 
 [INFO] Server has authentication => FALSE 

$ _build/cliapp server /var/www --port 9010 --host 127.0.0.1
 [TRACE] ----- Callback invoked OK ------ 
 [INFO] Running web server => port = 9010 ; host = 127.0.0.1; path = /var/www 
 [INFO] Server has authentication => FALSE 

$ _build/cliapp server /var/www --port 9010 --host 127.0.0.1 --auth
 [TRACE] ----- Callback invoked OK ------ 
 [INFO] Running web server => port = 9010 ; host = 127.0.0.1; path = /var/www 
 [INFO] Server has authentication => TRUE 

$ _build/cliapp server /var/www --port=9010 --host=127.0.0.1 --auth
 [TRACE] ----- Callback invoked OK ------ 
 [INFO] Running web server => port = 9010 ; host = 127.0.0.1; path = /var/www 
 [INFO] Server has authentication => TRUE

Run ls subcommand: 

$ _build/cliapp ls
<PATH> is required
Run with --help for more information.


$ _build/cliapp ls /System/Applications
  => /System/Applications/Siri.app
  => /System/Applications/Music.app
  => /System/Applications/FindMy.app
  => /System/Applications/QuickTime Player.app
  => /System/Applications/Chess.app
  => /System/Applications/Photo Booth.app
  => /System/Applications/Books.app
... ... ... ... ... ... ... ... 
... ... ... ... ... ... ... ... ... 

$ _build/cliapp ls /System/Applications/Utilities/Terminal.app --file --recursive
... ... ... ... ... ... ... ... 
... ... ... ... ... ... ... ... ... 

  => /System/Applications/Utilities/Terminal.app/Contents/Resources/ca.lproj/TTFindPanel.strings
  => /System/Applications/Utilities/Terminal.app/Contents/Info.plist
  => /System/Applications/Utilities/Terminal.app/Contents/PkgInfo
  => /System/Applications/Utilities/Terminal.app/Contents/version.plist

1.8 System information via virtual file systems

1.8.1 Overview

The Linux kernel provides two official APIs (Application Programming Interfaces) for user-space applications. The system-calls, that are stable and well documented and often made available via GLIBC (GNU C library). And VSF - virtual file system, which are in-memory pseudo file systems which allow user-space applications to retrieve system information; tune kernel runtime parameters and control device drivers by reading and writing VSF files.

The following code demonstrates, retrives several system information by reading files in Linux's /proc (procfs) and /sys (sysfs) virtual file systems.

Useful files in procfs or (/proc) file system

Some files that contains useful system information are:

  • /proc/version
    • => Linux Kernel Version
  • /proc/cpuinfo
    • => Contains information about current CPU, such as vendor; architecture; supported vector (SIMD) extensions; number of cores and so on.
  • /proc/mounts (command: $ mounts )
    • => Lists all mount points and mounted file systems.
  • /proc/partitions
    • => Contain all system partitons (sda, sda1, …, sda[N])
  • /proc/filesystems
    • => Lists all supported file system types by the current kernel.
  • /proc/modules (command: $ lsmod)
    • => Contains listing of all kernel modules loaded by the kernel. The CLI tool lsmod uses this file to display the loaded modules.
  • /proc/meminfo (command: $ free -m)
    • => Contains information about current memory.
  • /proc/uptime (command: $ uptime)
    • => Stores the CPU uptime, how long the machine is running since boot time.
  • /proc/loadvg (command: $ uptime)
    • => Shows system load average.
    • See:
  • proc/sys/kernel (Directory) - The tool sysctl, for tweaking Kernel parameters, manipulates the file in this directory.
    • => This directory contains, several files which allows tuning kernel parameters at runtime, by just reading or writing to those files.
  • /proc/sys/kernel/hostname
    • => Contains current machine hostname, that can be changed by writiing to this file.
  • /proc/sys/kernel/ostype
    • => Operating system type, always set to "Linux".
  • /proc/sys/kernel/osrelease
    • => Linux kernel version.
  • /proc/sys/kernel/sysrq
    • => If this file is set to '1', it allows using the magic SysRQ key makes possible to kill hogging processes when the machine hangs or freeze due to high memory usage.

Further Reading

General:

Kernel Runtime Settings (/proc/sys/kernel) and sysctl tool:

Linux Load Average (/proc/loadavg)

1.8.2 Sample Code

Code available at:

File: linux-info.cpp 

/* Summary: Gets Linux runtime information from VSF virtual file system. 
 *
 */
#include <iostream>
#include <fstream> 
#include <sstream>
#include <string> 
#include <iomanip>  // std::seprecision, std::fixed 

// See: https://man7.org/linux/man-pages/man2/sysinfo.2.html
#include <sys/sysinfo.h>

void        get_memory_info();
void        show_sysinfo1();
void        show_uptime();
void        show_mount_points();
void        show_kernel_runtime_config();
std::string readFile(std::string const& file);

int main(int argc, char** argv)
{
      if(argc < 2){
          std::cerr << " [ERROR] Usage: $ " << argv[0] << " <COMMAND> " << '\n';
          return EXIT_FAILURE;
      }

      auto command = std::string{ argv[1] };

      if(command == "version"){
           std::cout << "===== Kernel Version ====================================" << '\n';
           auto version = readFile("/proc/version");
           std::cout << " System version = " << version << '\n';
           return EXIT_SUCCESS;
      }

      if(command == "memory")
      {
           std::cout << " ==== Memory (Reported by command $ free -m ) ========" << '\n';
           get_memory_info();
           return EXIT_SUCCESS;
       }

      // Using the header: <sys/sysinfo.h>  
      // Note: this information is reported by the command $ free -m 
      if(command == "sysinfo1")
      {
           std::cout << " ===== System Info 1 (Reported by $ free -m and $ uptime ) === " << '\n';
           show_sysinfo1();
           return EXIT_SUCCESS;
      }

      if(command == "uptime")
      {
           std::cout << "===== System uptime since boot (Reported by $ uptime) ====" << '\n';
           show_uptime();
           return EXIT_SUCCESS;
      }

      if(command == "load-avg")
      {
           std::cout << " ==== System Load Average ================================" << '\n';
           auto loadavg = readFile("/proc/loadavg");
           std::cout << "Load average = " << loadavg << '\n';   
           return EXIT_SUCCESS;
      }

      if(command == "mount")
      {
           std::cout << " ====== System mount points ( Reported by command $ mount ) ===== " << '\n';
           show_mount_points();
           return EXIT_SUCCESS;
      }

      if(command == "sysctl")
      {
           std::cout << " ===== Kernel Runtime Information (Reported by $ sysctl ) ======= " << '\n';
           show_kernel_runtime_config();
           return EXIT_SUCCESS;
      }

      if(command == "dmesg")
      {
           std::cout << " ===== Kernel debug message (Reported by $ dmesg ) ==========" << '\n';
           auto ifs = std::ifstream("/dev/kmsg");
           std::string line;
           while( std::getline(ifs, line) ){
                   std::cout << line << '\n';
           }
      }

      if(command == "cpuinfo")
      {
           std::cout << "======= CPU Information =====================================" << '\n';

           auto ifs = std::ifstream("/proc/cpuinfo");
           std::string line;
           while( std::getline(ifs, line) ){
                   std::cout << line << '\n';
           }        
      }

      return EXIT_SUCCESS;
}

void get_memory_info()
{
     constexpr const char* meminfo_file = "/proc/meminfo";
     // factor = 1 Gigabytes = 1024 * 1024 kbytes 
     constexpr double factor = 1024 * 1024;

     auto ifs = std::ifstream{meminfo_file};
     if(!ifs.good()){
             throw std::runtime_error("Error: unable to memory-info file.");
     }
     std::string line, label;
     std::uint64_t value; 
     while( std::getline(ifs, line) )
     {      
          std::stringstream ss{line};   
          ss >> label >> value;

          if(label == "MemTotal:")
                  std::cout << "     Total memory (Gb) = " << (value / factor) << '\n';

          if(label == "MemAvailable:")      
                  std::cout << " Memory Available (Gb) = " << (value / factor) << '\n';
     }
}

void show_sysinfo1()
{
     struct sysinfo info;
     ::sysinfo(&info);

     // 1 Gigabyte = 1024 megabytes = 1024 * 1024 kbytes = 1024 * 1024 * 1024 bytes;
     constexpr double factor = 1024 * 1024 * 1024;
     constexpr std::uint64_t one_day_to_seconds = 24 * 60 * 60;

     std::cout << " [*] System uptime since boot (seconds) = " << info.uptime << '\n'
                       << " [*] System uptime since boot    (days) = " << info.uptime    / one_day_to_seconds << '\n'
                       << " [*]              Total RAM memory (Gb) = " << info.totalram  / factor << '\n'
                       << " [*]              Free  RAM memory (Gb) = " << info.freeram   / factor << '\n'
                       << " [*]                    Total SWAP (Gb) = " << info.totalswap / factor << '\n'                 
                       << " [*]                     Free SWAP (Gb) = " << info.freeswap  / factor << '\n'                                 
                       << " [*]        Number of processes running = " << info.procs << '\n'
                       << '\n';
}

// Reported by command $ uptime 
void show_uptime()
{   
     auto ifs = std::ifstream("/proc/uptime");
     if( !ifs.good() ){ throw std::runtime_error("Error: unable to open uptime file "); }
     double seconds;
     ifs >> seconds; 
     uint64_t factor = 24 * 60 * 60;
     std::cout << "   Uptime in hours = " << seconds / ( 60 * 60) << '\n';
     std::cout << "    Uptime in days = " << seconds / factor << '\n';
}


void show_kernel_runtime_config()
{
    auto show_field= [](std::string const& label, std::string const& file)
    {
         std::cout << " [*] " 
                   << std::setw(40) << std::right << (label + " => ")
                   << std::setw(50) << std::left  << readFile(file)
                   << '\n';
    };

    std::cout << '\n';
    show_field("Machine Hostname",                "/proc/sys/kernel/hostname");
    show_field("Kernel Version",                  "/proc/sys/kernel/osrelease");    
    show_field("Os type",                         "/proc/sys/kernel/ostype");
    show_field("Boot unique ID identifier",       "/proc/sys/kernel/random/boot_id");
    show_field("Date which kernel was compiled",  "/proc/sys/kernel/version");
    show_field("Sysrq enabled (Emergency keys)",  "/proc/sys/kernel/sysrq");
    show_field("Max number of Cgroups namespaces", "/proc/sys/user/max_cgroup_namespaces");
    show_field("Max number of Mount namespaces",   "/proc/sys/user/max_net_namespaces");
}


void show_network_interfaces()
{
    throw std::runtime_error("Error: not implemented yet.");
}

// Show mounted file systems, but ignore 'cgroup', 'overlay', 'fusectl'   
void show_mount_points()
{
     auto ifs = std::ifstream("/proc/mounts");
     if(!ifs){ 
             throw std::runtime_error("Error: unable to open file /proc/mounts"); 
     }
     std::cout << '\n';
     std::cout << "    " 
               << std::setw(20) << std::left << "File system" 
               << std::setw(20) << std::left << "Type"
               << std::setw(10) << std::left << "Mount point"
               << '\n';

     std::cout << "    " 
               << std::setw(20) << std::left << "------------" 
               << std::setw(20) << std::left << "----"
               << std::setw(10) << std::left << "-----------"
               << '\n';

     std::string line, fsystem, type, mount_directory;  

     while( std::getline(ifs, line) )
     {
        auto ss = std::stringstream(line);
        ss >> fsystem >> mount_directory >> type;
        if( type != "cgroup" && type != "overlay" && type != "cgroup2")
        {
            std::cout << "   " 
                      << std::setw(20) << std::left << fsystem 
                      << std::setw(20) << std::left << type 
                      << std::setw(10) << std::left << mount_directory 
                      << '\n';
        }
     }
}

// Requires: <string>, <stream>, <sstream>
std::string readFile(std::string const& file) 
{
    std::ifstream is(file);
    if( !is.good() ){
        throw std::runtime_error("Error: stream has errors.");
    }
    std::stringstream ss;
    ss << is.rdbuf();   
        std::string m;
        // Remove ending line character '\n' or '\r\n'.
        std::getline(ss, m);
    return m;
}

1.8.3 Running

Building: 

$ g++ linux-info.cpp -o linux-info -std=c++1z -Wall -Wextra -g

Running (Tested on Linux distribution PopOS Live CD running in a virtual machine)

Get Kernel Version: 

 $ >> ./linux-info 
 [ERROR] Usage: $ ./linux-info <COMMAND> 

 $ >> ./linux-info version # Get Kernel version
===== Kernel Version ====================================
 System version = Linux version 5.4.0-7634-generic (buildd@lcy01-amd64-003) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)) #38~1591219791~20.04~6b1c5de-Ubuntu SMP Thu Jun 4 02:56:10 UTC 2
 $ >>

Get RAM memory available (equivalent to $ free -m) 

$ >> ./linux-info memory # RAM memory 
==== Memory (Reported by command $ free -m ) ========
    Total memory (Gb) = 1.9175
Memory Available (Gb) = 1.01926

System uptime and RAM memory available: 

$ >> ./linux-info sysinfo1 
===== System Info 1 (Reported by $ free -m and $ uptime ) === 
[*] System uptime since boot (seconds) = 760
[*] System uptime since boot    (days) = 0
[*]              Total RAM memory (Gb) = 1.9175
[*]              Free  RAM memory (Gb) = 0.20657
[*]                    Total SWAP (Gb) = 0
[*]                     Free SWAP (Gb) = 0
[*]        Number of processes running = 444

Load average: 

 $ >> ./linux-info load-avg
 ==== System Load Average ================================
Load average = 0.27 0.28 0.28 4/443 2532

Mounted file systems (file: /proc/mounts) 

$ >> ./linux-info mount 
====== System mount points ( Reported by command $ mount ) ===== 

   File system         Type                Mount point
   ------------        ----                -----------
  sysfs               sysfs               /sys      
  proc                proc                /proc     
  udev                devtmpfs            /dev      
  devpts              devpts              /dev/pts  
  tmpfs               tmpfs               /run      
  /dev/sr0            iso9660             /cdrom    
  /dev/loop0          squashfs            /rofs     
  securityfs          securityfs          /sys/kernel/security
  tmpfs               tmpfs               /dev/shm  
  tmpfs               tmpfs               /run/lock 

 ...  ...  ...  ...  ...  ...  ...  ...  ...  ... 
 ...  ...  ...  ...  ...  ...  ...  ...  ...  ... 

  debugfs             debugfs             /sys/kernel/debug
  tracefs             tracefs             /sys/kernel/tracing
  fusectl             fusectl             /sys/fs/fuse/connections
  configfs            configfs            /sys/kernel/config
  tmpfs               tmpfs               /tmp      
  tmpfs               tmpfs               /run/user/999
  gvfsd-fuse          fuse.gvfsd-fuse     /run/user/999/gvfs
  /dev/fuse           fuse                /run/user/999/doc
  /dev/fuse           fuse                /root/.cache/doc
  gvfsd-fuse          fuse.gvfsd-fuse     /root/.cache/gvfs
  binfmt_misc         binfmt_misc         /proc/sys/fs/binfmt_misc

Kernel runtime information (directory /proc/sys/kernel) 

$ >> ./linux-info sysctl # Show some kernel runtime options
===== Kernel Runtime Information (Reported by $ sysctl ) ======= 

[*]                     Machine Hostname => pop-os                                            
[*]                       Kernel Version => 5.4.0-7634-generic                                
[*]                              Os type => Linux                                             
[*]            Boot unique ID identifier => 756989d9-f596-4e4f-ac50-d5c3026d041a              
[*]       Date which kernel was compiled => #38~1591219791~20.04~6b1c5de-Ubuntu SMP Thu Jun 4 02:56:10 UTC 2
[*]       Sysrq enabled (Emergency keys) => 176                                               
[*]     Max number of Cgroups namespaces => 7490                                              
[*]       Max number of Mount namespaces => 7490

Kernel messages (file: /dev/kmsg) 

 $ >> ./linux-info dmesg

 ===== Kernel debug message (Reported by $ dmesg ) ==========
5,0,0,-;Linux version 5.4.0-7634-generic (buildd@lcy01-amd64-003) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)) #38~1591219791~20.04~6b1c5de-Ubuntu SMP Thu Jun 4 02:56:10 UTC 2 (Ubuntu 5.4.0-7634.38~1591219791~20.04~6b1c5de-generic 5.4.41)
6,1,0,-;Command line: BOOT_IMAGE=/casper_pop-os_20.04_amd64_intel_debug_19/vmlinuz.efi initrd=/casper_pop-os_20.04_amd64_intel_debug_19/initrd.gz boot=casper live-media-path=/casper_pop-os_20.04_amd64_intel_debug_19 hostname=pop-os username=pop-os noprompt  ---
6,2,0,-;KERNEL supported cpus:
6,3,0,-;  Intel GenuineIntel
  ....   ....   ....   ....   ....   ....   .... 
  ....   ....   ....   ....   ....   ....   .... 

 DEVICE=+hdaudio:hdaudioC0D0
6,649,16644546,-;snd_hda_codec_generic hdaudioC0D0:    inputs:
 SUBSYSTEM=hdaudio
 DEVICE=+hdaudio:hdaudioC0D0
7,650,25338660,-;rfkill: input handler disabled
5,651,40422354,-;random: crng init done
5,652,40422357,-;random: 7 urandom warning(s) missed due to ratelimiting

Testing with Fedora 32 x86-64 (64 bits)

Get mount points (file: /proc/mounts) 

$ ./linux-info mount
 ====== System mount points ( Reported by command $ mount ) ===== 

    File system         Type                Mount point
    ------------        ----                -----------
   sysfs               sysfs               /sys      
   proc                proc                /proc     
   devtmpfs            devtmpfs            /dev      
   securityfs          securityfs          /sys/kernel/security
   tmpfs               tmpfs               /dev/shm  
   devpts              devpts              /dev/pts  
   tmpfs               tmpfs               /run      
   tmpfs               tmpfs               /sys/fs/cgroup
   pstore              pstore              /sys/fs/pstore
   efivarfs            efivarfs            /sys/firmware/efi/efivars
   none                bpf                 /sys/fs/bpf
   configfs            configfs            /sys/kernel/config
   /dev/sda3           ext4                /         
   systemd-1           autofs              /proc/sys/fs/binfmt_misc
   tmpfs               tmpfs               /tmp      
   fusectl             fusectl             /sys/fs/fuse/connections
   /dev/sda1           vfat                /boot/efi 
   sunrpc              rpc_pipefs          /var/lib/nfs/rpc_pipefs
   tmpfs               tmpfs               /run/user/42
   tmpfs               tmpfs               /run/user/1000
   gvfsd-fuse          fuse.gvfsd-fuse     /run/user/1000/gvfs
   nsfs                nsfs                /run/docker/netns/77290c7a4863
   binfmt_misc         binfmt_misc         /proc/sys/fs/binfmt_misc

1.9 Disk Space Usage on Linux and BSD-variants

The following code gets the disk space usage on Linux-derived operating systems by iterating over the file /proc/mounts for determining mounted file systems and then uses the statfs() function for getting disk usage information from each file system mount point. Unlike, Linux On BSD variants such, as FreeBSD, OpenBSD and Mac-OSX, there is no PROCFS virtual file system. On those operating systems, the information about disk space usage can be obtained in a single step by using the function getmntinfo().

Functions used

  • Documentation: $ man statfs
  • The functions statfs and fstatfs provide information about mounted file systems. The parameter buf, must be allocated by the caller. When the function fails, it returns (-1).
#include <sys/vfs.h>    /* or <sys/statfs.h> */

int statfs(const char *path, struct statfs *buf);
int fstatfs(int fd, struct statfs *buf);

struct statfs {
    __fsword_t f_type;    /* Type of filesystem (see below) */
    __fsword_t f_bsize;   /* Optimal transfer block size */
    fsblkcnt_t f_blocks;  /* Total data blocks in filesystem */
    fsblkcnt_t f_bfree;   /* Free blocks in filesystem */
    fsblkcnt_t f_bavail;  /* Free blocks available to
                             unprivileged user */
    fsfilcnt_t f_files;   /* Total file nodes in filesystem */
    fsfilcnt_t f_ffree;   /* Free file nodes in filesystem */
    fsid_t     f_fsid;    /* Filesystem ID */
    __fsword_t f_namelen; /* Maximum length of filenames */
    __fsword_t f_frsize;  /* Fragment size (since Linux 2.6) */
    __fsword_t f_flags;   /* Mount flags of filesystem
                             (since Linux 2.6.36) */
    __fsword_t f_spare[xxx];
                    /* Padding bytes reserved for future use */
};

Sample Code

File: df-size.cpp 

// ----- Get Disk Space Usage -------------// 

#include <iostream>
#include <vector>
#include <string>
#include <algorithm>
#include <functional>
#include <iomanip>
#include <fstream>
#include <cmath> 

// ------ Unix Headers ----//
#include <unistd.h>
#include <sys/vfs.h>    /* Get statfs() */
#include <limits.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <cstring>

struct mount_point
{
    // File System 
    std::string fsystem; 
    // Mounting directroy 
    std::string mount_directory;
    // File system type 
    std::string type;
};

template<typename MountpointConsumer> 
void iterate_mount_points(MountpointConsumer&& consumer);

auto read_symlink(std::string const& path) -> std::string ;

int main(int argc, char** argv)
{

   iterate_mount_points([](mount_point const& mnt)
   {
       struct statfs fs;
       const char* mount_dir = mnt.mount_directory.c_str();

       // =>>> List of file system that will be ignored. 
       // 'static' keyworkd =>  Variable allocated only once, when 
       // this function is called for the first time. 
       static const auto ignored_fsystem = std::vector<std::string>{
             "cgroup", "cgroup2", "proc", "sysfs", "tmpfs", "devtmpfs"
           , "configfs", "bpf", "mqueue", "hugetlbfs", "tracefs"
           , "fuse.gvfsd-fuse", "rpc_pipefs", "debugfs"
           , "overlay", "securityfs", "devpts", "pstore", "efivarfs"
           , "autofs", "binfmt_misc", "tmpfs", "fusectl", "nsfs"
           , "selinuxfs", "functionfs"
        };

       // Ignore the following types of file-system, mostly VSF (Virtual File Systems)
       // or in-memory file systems.
       auto iter = std::find(ignored_fsystem.cbegin(), ignored_fsystem.cend(), mnt.type );
       if( iter != ignored_fsystem.end() ) { return; }

       if( statfs(mount_dir, &fs) == -1 )
       {
           std::cerr << " [ERROR] Unable to get file system statistics for: " 
                     << " Mount point => " << mnt.mount_directory 
                     << " / " << mnt.fsystem.c_str() 
                     << '\n';
           return;
       }

       const size_t block_size = fs.f_bsize;

        // Total disk space in Gigabytes 
       const double total_space = static_cast<double>(fs.f_blocks) * block_size / (1024 * 1024 * 1024);
       // Free disk space in Gigabytes 
       const double available_space = static_cast<double>(fs.f_bavail) * block_size / (1024 * 1024 * 1024);
       // Disk space usage in Gigabytes 
       const double used_space = total_space - available_space;

       // Disk space usage in percent (%) 
       const double used_space_pct = (1.0 - available_space / total_space) * 100.0;

       auto fsystem_abspath = read_symlink(mnt.fsystem);

       std::cout << "      File System: " << fsystem_abspath      << '\n'
                 << "             Type: " << mnt.type             << '\n'       
                 << "      Mount Point: " << mnt.mount_directory  << '\n';
        if(total_space > 1.0)
        {
            std::cout << " Total space (Gb): " << total_space            << '\n'
                      << "  Free space (Gb): " << available_space        << '\n'
                      << "  Used space (Gb): " << used_space             << '\n'                      
                      << "   Used space (%): " << used_space_pct <<  "%" << '\n'
                      << '\n';
        } else {
            std::cout << "  Free space (Mb): " << available_space * 1024  << '\n'
                      << " Total space (Mb): " << total_space * 1024      << '\n'
                      << "  Used space (Gb): " << used_space * 1024       << '\n'                                            
                      << "   Used space (%): " << used_space_pct << "%"   << '\n'
                      << '\n';

        }
   });

        return 0;
}


// ----------------------------------------------------------------------//

template<typename MountpointConsumer> 
void iterate_mount_points(MountpointConsumer&& consumer)
{
    auto ifs = std::ifstream("/proc/mounts");
    if(!ifs) throw std::runtime_error("Error: unable to open file /proc/mounts");     
    std::string line, fsystem, mount_directory, type; 

    while( std::getline(ifs, line) )
    {
        auto ss = std::stringstream(line);
        ss >> fsystem >> mount_directory >> type;
        consumer( mount_point{ fsystem, mount_directory, type } );
        //if( type != "cgroup" && type != "overlay" && type != "cgroup2")        
    }
}

// Get absolute path to file or symbolic link 
auto read_symlink(std::string const& path) -> std::string 
{
    // Create a buffer with size PATH_MAX + 1 filled with 0 ('\0'), null characters
    std::string buffer(PATH_MAX, 0);
    // ssize_t readlink(const char *pathname, char *buf, size_t bufsiz);
    ssize_t nread = ::readlink(path.c_str(), &buffer[0], PATH_MAX);
    if(nread == -1){ return path; }
    buffer.resize(nread);
    return buffer;
}

Building and Running on Fedora Linux 32 x86-64 

$ >> g++ df-size.cpp -o df-size.bin -std=c++1z -Wall -Wextra -g 

$ >> ./df-size.bin 
     File System: /dev/sda3
            Type: ext4
     Mount Point: /
Total space (Gb): 49.06
 Free space (Gb): 23.9494
 Used space (Gb): 25.1106
  Used space (%): 51.1835%

     File System: /dev/sda4
            Type: ext4
     Mount Point: /home
Total space (Gb): 846.607
 Free space (Gb): 510.182
 Used space (Gb): 336.424
  Used space (%): 39.738%

     File System: /dev/sda1
            Type: vfat
     Mount Point: /boot/efi
 Free space (Mb): 279.422
Total space (Mb): 299.82
 Used space (Gb): 20.3984
  Used space (%): 6.80355%

Disk space via (df) command line tool: 

 $ >> df -h 
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        7.8G     0  7.8G   0% /dev
tmpfs           7.8G  315M  7.5G   4% /dev/shm
tmpfs           7.8G  2.1M  7.8G   1% /run
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/sda3        50G   23G   24G  49% /
tmpfs           7.8G   11M  7.8G   1% /tmp
/dev/sda4       847G  294G  511G  37% /home
/dev/sda1       300M   21M  280M   7% /boot/efi
tmpfs           1.6G   16K  1.6G   1% /run/user/42
tmpfs           1.6G   80K  1.6G   1% /run/user/1000

Cross-compiling to Linux Armv7 and run on Android via ADB

Building: 

$ export DOCKER_COMMAND="docker run -e UID=$(id -u) -e GID=$(id -g) --volume=$PWD:/cwd --workdir=/cwd muslcc/i686:armv7l-linux-musleabihf "
$ $DOCKER_COMMAND g++ df-size.cpp -o df-size.armv7 -std=c++1z -static -Wall -Wextra
$ $DOCKER_COMMAND strip df-size.armv7 

$ file df-size.armv7 
df-size.armv7: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, stripped

$ du -h df-size.armv7 
820K    df-size.armv7
820K    total

Transfer executable to Android: 

$ adb push df-size.armv7 /data/local/tmp
df-size.armv7: 1 file pushed. 12.1 MB/s (839348 bytes in 0.066s)

Running on Android vai adb shell: 

$ adb shell
bali:/ $ 
bali:/ $ cd /data/local/tmp
bali:/data/local/tmp $ 

# Confirm disk space usage metrics 
bali:/data/local/tmp $ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/root             2.6G  2.2G  450M  84% /
tmpfs                 931M  1.3M  930M   1% /dev
tmpfs                 931M     0  931M   0% /mnt
/dev/block/dm-1       1.4G  207M  1.2G  15% /vendor
/dev/block/mmcblk0p39  23G   12G   11G  53% /data
/dev/block/mmcblk0p38 402M  6.4M  383M   2% /cache
/data/media            23G   12G   11G  53% /storage/emulated

# Run application 
bali:/data/local/tmp $ ./df-size.armv7  2> /dev/null                                                         
      File System: /dev/root
             Type: ext4
      Mount Point: /
 Total space (Gb): 2.66402
  Free space (Gb): 0.439049
  Used space (Gb): 2.22497
   Used space (%): 83.5193%

      File System: /dev/block/dm-1
             Type: ext4
      Mount Point: /vendor
 Total space (Gb): 1.45304
  Free space (Gb): 1.23495
  Used space (Gb): 0.218094
   Used space (%): 15.0094%

      File System: /dev/block/mmcblk0p39
             Type: f2fs
      Mount Point: /data
 Total space (Gb): 23.4941
  Free space (Gb): 11.2018
  Used space (Gb): 12.2923
   Used space (%): 52.3209%

      File System: /dev/block/mmcblk0p38
             Type: ext4
      Mount Point: /cache
  Free space (Mb): 383.078
 Total space (Mb): 402.445
  Used space (Gb): 19.3672
   Used space (%): 4.81238%

      File System: /dev/block/mmcblk0p35
             Type: ext4
      Mount Point: /product
  Free space (Mb): 444.348
 Total space (Mb): 476.59
  Used space (Gb): 32.2422
   Used space (%): 6.76519%

      File System: /data/media
             Type: sdcardfs
      Mount Point: /storage/emulated
 Total space (Gb): 23.4941
  Free space (Gb): 11.2018
  Used space (Gb): 12.2923
   Used space (%): 52.3209%

Further Reading

1.10 Low level open, read, write, close - IO sycalls overview

The open(), read(), write() and close() system calls are the hallmark of Unix-based operating system as they embody the concept "every is a file" or "everything is a file descriptor". Those fundamental system calls are able to operate over any type of file descriptors such as regular files; symbolic links; in-memory files; anonymous pipes; named pipes; sockets; unix domain sockets IPC (Inter-Process Communication); character device files representing hardware; block device files, which represents non-volatile storage device and so on.

It is worth noting that, those system-calls are not used directly by user-space applications. Instead, applications uses C function wrappers provided by the CRT (C Runtime Library), which is GLIBC in most Linux distributions.

Special File Descriptors

  • STDOUT_FILENO => File descritor for stdout (process standard output)
  • STDERR_FILENO => File descritor for stderr (process standard error output)
  • STDIN_FILENO => File descritor for stdin (process standard input)

Headers 

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

Functions

  • close() => Close a file descriptor.
    • Doc: $ man 2 close
int close(int fd);
  • open() => Encapsulates open system call => Returns a file descritor number. When fails, it returns (-1) setting the global variable errno.
    • Doc: $ man 2 open
int open(const char *pathname, int flags);
  • creat() => Create file. Note: the name is not misspelled. There is really a missing 'e' letter.
    • Doc: $ man 2 creat
int creat(const char *pathname, mode_t mode)
  • read() => Read bytes from a file descriptor into a buffer, returning the number of bytes read. If there is an error, the functions returns (-1) setting the global variable errno.
    • Doc: $ man 2 read
    • Note: If the number of bytes returned is zero, it means that no more bytes can be read in the case of file or that socket connection was closed by the counterpart (socket server or client).
    • Note: The number of bytes returned can smaller than the buffer size in the case of socket file descriptors (partial read).
    • Note: The buffer argument can be a pointer to any POD (Plain-Old Data) type allocated by the calling code. A POD type, does not have any internal pointers or STL containers.
    • Note: A negative return value (-1), as in most Unix APIs, indicates an error. In order to get more information about the error, the calling code must check the errno global variable.
ssize_t read(int fd, void *buf, size_t count);
  • write() => Write N bytes from a buffer to a file descriptor.
    • Doc: $ man 2 write
ssize_t write(int fd, const void *buf, size_t count);

1.11 Low level IO - read() syscall

Summary:

  • Read data from file descriptor to a buffer. Where buffer a pointer to any storage location within the process virtual memory. The 'buffer' argument can be a pointer to any POD type plain-old data, which continuous allocated in the memory, without any internal pointer.
  • This system call can be used with any type of file descriptors, namely: ordinary file descriptors; standard input from terminal, stdin (character device); sockets; unix domain sockets; anonymous pipes; named pipes; character device files, which often represent hardware; and block devices which represent mounted file systems.
#include <unistd.h> 

ssize_t read(int fd, void* buffer, size_t len);

ssize_t read(  int    fd     // => File descriptor          
             , void*  buf    // => Pointer to buffer 
                             //   It can be a pointer to any POD type, plain-old data.
                             //   without any internal pointers.                                  
             , size_t len    // => Buffer size or length in bytes.
           );

Possible Return Values

The system call wrapper function read() can return the following results:

  1. Number of bytes read, which may be less or equal to the buffer size. The bytes read are stored in the buffer.
  2. Return value: (0) indicating EOF (End-Of-File) - there is nor more bytes to be read.
  3. Return value: (-1) and errno set to EINTR indicating that before any byte was read, a signal was set to the current process. In this case, read() can be called again.
  4. Return value: (-1) and errno set to EAGAIN with a file descriptor in non-blocking mode. This case indicates that a read operation on the file descriptor would block the current thread while there is no data available.
  5. Return value: (-1) in the case of failure, in this case, the thread-local storage variable errno is set.

Semantics of read()

  • Ordinary file descriptor
    • A call to read() never blocks the current thread and returns 0 indicating (EOF) when the end of file is reached.
  • STDIN_FILENO (value 0) file descriptor, which represents the console/terminal input.
    • A call to read() blocks the current thread while there is no bytes to be read and returns (0) indicating EOF when the user types the (CTRL D) sequence.
  • Sockets file descriptors in blocking mode (default):
    • A call to read blocks the curren thread while there is no bytes available in the kernel-side buffer. When a read() call returns 0, it indicates that no more bytes can be read as the other side (client or server) has closed the connection and by using close() or shutdown() calls.

Handling partial-reads

The following algorithm reads all bytes until the buffer is filled for handling partial reads which can happen with socket file descriptors in blocking mode. Example about partial read: The server-side socket sends 1000 bytes, then the client-side socket counterpart performs a read(fd_server, buffer, 1000) call. The number of bytes returned on the first call may be less than the buffer size, for instance it could be 200. Then the client-side socket would have to continue to read all bytes until the buffer is filled. 

 //----------------------------------------//
 //  Inputs                                //
 //----------------------------------------//

 // Buffer length in bytes 
 const size_t buffer_len = 400;
 char  buffer[buffer_len];

 //----------------------------------------//
 //  Algorithm                             //
 //----------------------------------------//

 // Pre-condition 
 assert( bufffer_len <= SSIZE_MAX  );
 size_t  len  = buffer_len;
 ssize_t nr   = -1;
 char*   pbuf = buffer; 

 while(  // Stop iterations when buffer is filled.
         len != 0  
         // Stop interations when 'nr' is EOF, there is no more bytes left 
         // or the socket connection has been closed. 
         && ( nr = read(fd, pbuf, len) ) != 0 ) 
{
     if(ret == -1 && errno == EINTR){ continue; } 
     if(ret == -1) { 
         perror(" An error has happened, check ERRNO variable. \n");
         break;
     }

     len  = len - nr; 
     pbuf = pbuf + nr;
 }

1.12 Low level IO - open(), read(), write() syscall functions

This code demonstrates the usage of the open(), read(), write(), close() low-level IO library-calls for file descriptors which encapsulates system-calls with the same name.

GIST:

File: unix-low-level-io.cpp 

#include <iostream>

#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

#include <cstring> // Import: char* strerror(in errnum);

const char* 
errno_to_cstring(int err)
{
    // No such file or directory 
    if(err == ENOENT) return "ENOENT";
    // Operation not permitted 
    if(err == EPERM)  return "EPERM";
    // Onput/Output error 
    if(err == EIO)    return "EIO";

    if(err == EAGAIN) return "EAGAIN";
    if(err == EPERM)  return "EPERM";
    if(err == EPIPE)  return "EPIPE";

    return "<UNKNOWN>";
}   


/** Check whether file descriptor is regular file */
bool fd_is_regular_file(int fd)
{       
     struct stat fd_info; 
     // int fstat(int fd, struct stat *statbuf);
     int r = fstat(fd, &fd_info);
     return S_ISREG(fd_info.st_mode);
}

bool fd_is_directory(int fd)
{
    struct stat fd_info; 
    // int fstat(int fd, struct stat *statbuf);
    int r = fstat(fd, &fd_info);
    return S_ISDIR(fd_info.st_mode);    
}

void print_errno_details(int err)
{
    std::fprintf(stderr , "\n   =>    errno(int) = %d" 
                          "\n   =>    errno(str) = %s"
                          "\n   => errno message = %s \n"
                          , err, errno_to_cstring(err), strerror(err));
     std::fflush(stderr);
}

int main(int argc, char** argv)
{
    std::puts(" [INFO] Program started. ");

    if(argc < 3){
            std::fprintf(stderr, " Usage:                             \n");
            std::fprintf(stderr, "  => To read a file:                \n");
            std::fprintf(stderr, "    $ %s file <FILE>                \n", argv[0]);
            std::fprintf(stderr, "  => To read stdin (console input): \n");
            std::fprintf(stderr, "    $ %s file -stdin                \n", argv[0]);
            return 0;
    }

    // Compare two c-strings return 0 (zero) when they are equal.
    // int strcmp(const char *s1, const char *s2)
    if( strcmp(argv[1], "file") != 0 )
    {
            std::fprintf(stderr, " [ERROR] Expected command file. \n");
            return EXIT_FAILURE;
    }

    // Variable for holding a file descriptor 
    int fd; 

    // The library-call open() attempts to open a file  and returns a "file-descriptor" 
    // (integer number ) when the operation is successful. The library-call 
    // returns (-1) when the operation fails. 
    // Note: It encapsulates the 'open' system call. 
    //  
    if( strcmp(argv[2], "-stdin") == 0)
            fd = STDIN_FILENO; 
    else 
            fd = open(argv[2], O_RDONLY);

    if(fd == -1){
            // Get error flag 'errno' to get more details about current error.
            int err = errno;
            std::fprintf(stderr ," [ERROR] Failed to open file. ");
            print_errno_details(err);
            return EXIT_FAILURE;
    }

    std::fprintf(stdout, " [INFO] ?? File is regular file = %s \n"
                              , fd_is_regular_file(fd) ? "TRUE" : "FALSE"  );
    std::fprintf(stdout, " [INFO] ?? File is directory file = %s \n"
                              , fd_is_directory(fd) ? "TRUE" : "FALSE"  );                
    // Flush file => Force changes to be immeditely written.
    std::fflush(stdout);

    // Buffer maximum size in bytes 
    constexpr size_t BUFFER_MAX_SIZE = 200;     
    char buffer[BUFFER_MAX_SIZE];

    // Stream BUFFER_MAX_SIZE bytes from file descriptor 
    // to STDOUT_FILENO (file descriptor).
    //---------------------------------------------------
    ssize_t ret; 
    do {
            ret = read(fd, buffer, BUFFER_MAX_SIZE);        
            if(ret == -1) {
                    int err = errno; 
                    std::fprintf(stderr, " [ERROR] An error has happened => ");
                    print_errno_details(err);
                    close(fd);
                    return EXIT_FAILURE;
            }       
            ::write(STDOUT_FILENO, buffer, ret);
    } while( ret != 0);

    // Always close the file descriptor.
    close(fd);

    return 0;
}

Building: 

$ g++ unix-low-level-io.cpp -o unix-low-level-io.bin -std=c++1z -Wall -Wextra

Running:

  • Run 1
$ >> ./unix-low-level-io.bin 
[INFO] Program started. 
Usage:                             
 => To read a file:                
   $ ./unix-low-level-io.bin file <FILE>                
 => To read stdin (console input): 
   $ ./unix-low-level-io.bin file -stdin
  • Run 2:
 $ >> ./unix-low-level-io.bin file /proc/filesystems 
 [INFO] Program started. 
 [INFO] ?? File is regular file = TRUE 
 [INFO] ?? File is directory file = FALSE 
nodev   sysfs
nodev   tmpfs
nodev   bdev
nodev   proc
nodev   cgroup
nodev   cgroup2
... ... ... 

$ >> ./unix-low-level-io.bin file /etc/resolv.conf 
 [INFO] Program started. 
 [INFO] ?? File is regular file = TRUE 
 [INFO] ?? File is directory file = FALSE 
# Generated by NetworkManager
nameserver 194.165.12.10
  • Run 3: (Error)
$ >> ./unix-low-level-io.bin file /etc/resosad
[INFO] Program started. 
[ERROR] Failed to open file. 
  =>    errno(int) = 2
  =>    errno(str) = ENOENT
  => errno message = No such file or directory 

$ >> ./unix-low-level-io.bin file /etc/shadow
[INFO] Program started. 
[ERROR] Failed to open file. 
  =>    errno(int) = 13
  =>    errno(str) = <UNKNOWN>
  => errno message = Permission denied 

$ >> ./unix-low-level-io.bin file /
[INFO] Program started. 
[INFO] ?? File is regular file = FALSE 
[INFO] ?? File is directory file = TRUE 
[ERROR] An error has happened => 
  =>    errno(int) = 21
  =>    errno(str) = <UNKNOWN>
  => errno message = Is a directory
  • Run 4 - read stdin file descriptor STDIN_FILENO
 $ >> ./unix-low-level-io.bin file -stdin
 [INFO] Program started. 
 [INFO] ?? File is regular file = FALSE 
 [INFO] ?? File is directory file = FALSE 


Hello world
Hello world
 Unix-linux file descriptors - Low level IO
 Unix-linux file descriptors - Low level IO

# User types Ctrl+D to close STDIN

1.13 Low level IO - creat()

Creates a file using Unix-low level IO function creat() which encapsulates the creat() system-call.

File: unix-creat.cpp 

#include <iostream>
#include <string> 

#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <cstring>

void print_errno_details(int err);

int main(int argc, char** argv)
{
    // File is created with read, write permissions for owner 
    // ,groups and others
    mode_t mode = S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH;

    int fd = creat("/tmp/my-sample-file.txt", mode);
    if(fd  == -1){
        print_errno_details(errno);
        return EXIT_FAILURE;
    }

    constexpr size_t BUFFER_SIZE = 500;
    char buffer[BUFFER_SIZE];

    // Fill the whole buffer with '\0' null char characters
    memset(buffer, '\0', BUFFER_SIZE);

    // strcpy => Copy string literal to buffer. 
    strcpy(buffer, " [LINE 0] Write this message to buffer\n");
    fprintf(stdout, " Buffer size (non blank chars) = %zu \n", strlen(buffer));
    // Write strlen(buffer) bytes to file descriptor 
    write(fd, buffer, strlen(buffer));

    strcpy(buffer, " [LINE 1] Unix, BSD, OSX, LINUX, AIX, POSIX\n");
    fprintf(stdout, " Buffer size (non blank chars) = %zu \n", strlen(buffer));
    fprintf(stdout, " Buffer content = '%s' \n", buffer);
    // Write strlen(buffer) bytes to file descriptor 
    write(fd, buffer, strlen(buffer));

    strcpy(buffer, " [LINE 2] FreeRTOS Linux RTAI Real-time RTMES QNX\n");
    write(fd, buffer, strlen(buffer));

    close(fd);
    return EXIT_SUCCESS;
}

// Ruires: #include <string>
void print_errno_details(int err)
{
        std::fprintf(stderr , "\n   =>    errno(int) = %d" 
                              "\n   => errno message = %s \n"
                            , err, strerror(err));
        std::fflush(stderr);
}

Building: 

$ g++ unix-creat.cpp -o unix-creat.bin -std=c++1z -Wall -Wextra -g

Running: 

 $ >> ./unix-creat.bin 
 Buffer size (non blank chars) = 39 
 Buffer size (non blank chars) = 44 
 Buffer content = ' [LINE 1] Unix, BSD, OSX, LINUX, AIX, POSIX
'

Content of generated file: 

$ >> cat /tmp/my-sample-file.txt 
[LINE 0] Write this message to buffer
[LINE 1] Unix, BSD, OSX, LINUX, AIX, POSIX
[LINE 2] FreeRTOS Linux RTAI Real-time RTMES QNX

Trace library-calls with ltrace application:

  • $ ltrace ./unix-creat.bin
$ >> ltrace ./unix-creat.bin

_ZNSt8ios_base4InitC1Ev(0x4040a9, 0xffff, 0x7ffcfe326598, 224) = 0
__cxa_atexit(0x4010e0, 0x4040a9, 0x402008, 6)                  = 0
creat(0x402010, 420, 0x7ffcfe326598, 256)                      = 3
memset(0x7ffcfe326280, '\0', 500)                              = 0x7ffcfe326280
strlen(" [LINE 0] Write this message to "...)                  = 39
fprintf(0x7f95f2612500, " Buffer size (non blank chars) ="..., 39 Buffer size (non blank chars) = 39 
) = 37
strlen(" [LINE 0] Write this message to "...)                  = 39
write(3, " [LINE 0] Write this message to "..., 39)            = 39
strlen(" [LINE 1] Unix, BSD, OSX, LINUX,"...)                  = 44
fprintf(0x7f95f2612500, " Buffer size (non blank chars) ="..., 44 Buffer size (non blank chars) = 44 
) = 37
fprintf(0x7f95f2612500, " Buffer content = '%s' \n", " [LINE 1] Unix, BSD, OSX, LINUX,"... Buffer content = ' [LINE 1] Unix, BSD, OSX, LINUX, AIX, POSIX
' 
) = 66
strlen(" [LINE 1] Unix, BSD, OSX, LINUX,"...)                  = 44
write(3, " [LINE 1] Unix, BSD, OSX, LINUX,"..., 44)            = 44
strlen(" [LINE 2] FreeRTOS Linux RTAI Re"...)                  = 50
write(3, " [LINE 2] FreeRTOS Linux RTAI Re"..., 50)            = 50
close(3)                                                       = 0
_ZNSt8ios_base4InitD1Ev(0x4040a9, 0, 0x4010e0, 1)              = 0x7f95f2965e40
+++ exited (status 0) +++

Trace system-calls with strace application:

  • $ strace ./unix-creat.bin
$ strace ./unix-creat.bin 

execve("./unix-creat.bin", ["./unix-creat.bin"], 0x7fff489b8e40 /* 83 vars */) = 0
brk(NULL)                               = 0x57c000
arch_prctl(0x3001 /* ARCH_??? */, 0x7fff751bd910) = -1 EINVAL (Invalid argument)
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
 ... ... ... ... ... ... ... ... ... ... ... ... ... ...
 ... ... ... ... ... ... ... ... ... ... ... ... ... ...

mprotect(0x7fed4e032000, 4096, PROT_READ) = 0
munmap(0x7fed4dfe4000, 136250)          = 0
brk(NULL)                               = 0x57c000
brk(0x59d000)                           = 0x59d000
creat("/tmp/my-sample-file.txt", 0644)  = 3
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0xc), ...}) = 0
write(1, " Buffer size (non blank chars) ="..., 37) = 37
write(3, " [LINE 0] Write this message to "..., 39) = 39
write(1, " Buffer size (non blank chars) ="..., 37) = 37
write(1, " Buffer content = ' [LINE 1] Uni"..., 63) = 63
write(1, "' \n", 3)                     = 3
write(3, " [LINE 1] Unix, BSD, OSX, LINUX,"..., 44) = 44
write(3, " [LINE 2] FreeRTOS Linux RTAI Re"..., 50) = 50
close(3)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

1.14 Reading/Listings directory contents

API Documentations:

Functions used: 

DIR *opendir(const char *name);

int readdir(   unsigned int fd
             , struct old_linux_dirent* dirp,
             , unsigned int count);

File: unix-readdir.cpp 

#include <iostream>

// #include <string>
#include <sys/types.h>
#include <dirent.h>  // Get function opendir
#include <errno.h>

#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>


template<typename Callback>
void iterate_directory(const std::string& path, Callback&& callback)
{
    DIR *dir;
    struct dirent *dp;

    dir = opendir(path.c_str()) ;

    // To determine the cause of error - It is necessary to check the error code.
    if (dir == nullptr) throw std::runtime_error("Error: Cannot read directory");

    while ((dp = readdir(dir)) != nullptr) 
    {
       if(path == "/")
            callback(dp->d_name);
       else 
            callback(path + "/" + dp->d_name);      
    };
    closedir(dir);
}

int main(int argc, char** argv)
{
     std::cout << " List directory contents " << std::endl;

     if(argc < 2)
     {
         std::cout << "Usage: ./unix-readdir <DIRECTORY>" << "\n";
         return EXIT_SUCCESS;
     }

     int idx = 0;
     iterate_directory(argv[1], [&idx](auto const& path)
     {
         std::cout << "  [" << idx++  << "] path => " << path << "\n";
     });
     return EXIT_SUCCESS;
}

Building on Linux: 

$ g++ unix-readdir.cpp -o unix-readdir.bin -std=c++1z -g -Wall -Wextra

Building on MacOSX: 

# Note: It is not (GNU GCC), g++ command refers to CLang 
$ g++ unix-readdir.cpp -o unix-readdir.bin -std=c++1z -g -Wall -Wextra 

# Same as previous command (CLang compiler)
$ clang++ unix-readdir.cpp -o unix-readdir.bin -std=c++1z -g -Wall -Wextra 

$ file unix-readdir.bin 
unix-readdir.bin: Mach-O 64-bit executable x86_64

$ du -h unix-readdir.bin
 44K    unix-readdir.bin

# List shared libraries dependencies: 
$ otool -L unix-readdir.bin 
unix-readdir.bin:
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 902.1.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)

Running on Linux: 

$ >> ./unix-readdir.bin /
 List directory contents 
  [0] path => srv
  [1] path => sys
  [2] path => ..
  [3] path => opt
  [4] path => .
  [5] path => run
  [6] path => media
  ... ... ... 
  ... .. .... ... ... 

 $ >> ./unix-readdir.bin /boot 
 List directory contents 
  [0] path => /boot/..
  [1] path => /boot/.
  [2] path => /boot/initramfs-5.6.12-300.fc32.x86_64.img
  [3] path => /boot/grub2
  [4] path => /boot/vmlinuz-0-rescue-a1ac43b933e24659bba5edf2b9cec1e1
    ... ... ...     ... ... ...     ... ... ...     ... ... ...

Running on MacOSX Catalina VM:

  • Note: on MacOSX, the command g++ is not GCC, it refers to Clang.
./unix-readdir /
 List directory contents 
  [0] path => .
  [1] path => ..
  [2] path => home
  [3] path => usr
  [4] path => .DS_Store
  [5] path => bin
  [6] path => sbin
  [7] path => .file
  [8] path => etc
  [9] path => var
  [10] path => Library
  [11] path => System
  [12] path => .VolumeIcon.icns
  [13] path => .fseventsd
  [14] path => private
  [15] path => .vol
  [16] path => Users
  [17] path => Applications
  [18] path => opt
  [19] path => dev
  [20] path => Volumes
  [21] path => tmp
  [22] path => cores

 ./unix-readdir /Applications/iTerm.app/Contents
  List directory contents 
   [0] path => /Applications/iTerm.app/Contents/.
   [1] path => /Applications/iTerm.app/Contents/..
   [2] path => /Applications/iTerm.app/Contents/CodeResources
   [3] path => /Applications/iTerm.app/Contents/_CodeSignature
   [4] path => /Applications/iTerm.app/Contents/MacOS
   [5] path => /Applications/iTerm.app/Contents/Resources
   [6] path => /Applications/iTerm.app/Contents/XPCServices
   [7] path => /Applications/iTerm.app/Contents/Frameworks
   [8] path => /Applications/iTerm.app/Contents/Info.plist
   [9] path => /Applications/iTerm.app/Contents/PkgInfo

1.15 Process Inputs, Outputs and States and PROCFS on Linux

Any process may have the following inputs, outputs and states:

Some Processes States:

  • Native executable used for creating the process. Every process has a native executable. A python process is the Python interpreter (python.exe on Windows) running python scripts. A Java process is the Java virtual machine (java.exe) running java *.class bytecodes or *.jar java packages.
  • CWD => Current Working Directory
    • There is a single current working directory for every process. On Unix, the functions getcwd() and chdir() gets and sets the current working directory.
  • IP => Instruction Pointer (process context) => CPU registry that stores the current machine instruction to be run.
  • SP => Stack Poitner (process context) => CPU registry used for keeping track of the process call stack.
  • Note: all thoses states can be viewed and manipulated with a debugger, such as GDB, LLDB or Windows Debugger. A debugger can monitor, explore and manipulate the virtual memory of any process.

Inputs:

  • Stdin (Standard Input) => User input read from console in some shell (cmd.exe on Windows) or bash, zsh on Unix-like OSes.
  • Command line argument passed to main function
  • Environment variables set during the process instantiation with env or export commands on Unix-like operating systems.
  • Configuration files => read at a default fixed location such as /.bashrc (bash configuration file at ~/home/user/.bashrc/ on Linux ); ~/.vimrc (vim configuration file)
  • Windows Registry entry for configuration during process initialization
  • Input files read after initialization.

Outputs:

  • Stdout (Standard Output) => Process output print by functions such as printf; std::cout << … visible in a terminal. (Unix heritage)
  • Stderr (Standard Error Ouput) => Process error or logging output print by default in a terminal. stderr exists for allowing separating the output (stdout) from the error or logging output (stderr).
  • Statuds code => Value returned by function int main(int argc, char** argv)
    • Value 0 indicates success and that the process terminated successfuly.
    • Value different than zero indicates that the process terminated with errors.
  • Output file(s)

Both Input/Output:

  • Sockets
  • IPC Inter Process Communication
  • GUI - Graphical User Interface
         Command line arguments            
         int arc, char** argv (input)                  Configuration file: 
           >>-------+                         +--<<--  Input file read at fixed location
                     \                       /         Or Windows Registry entry 
                      \                      |
                      |                      |    
                      \/                    \ /   
                     +---------------------------------+
                     |   Process:                      |      
                     |                                 +--------->> Stdout (output)
         ---------->>|     EXE = /bin/some/executable  |            
       Stdin (input) |     PID = 9381 (Unique ID)      +--------->> Stderr (output)
                     |     CWD = /home/user/some/data  |       
                     |                                 |
                     |     IP - Instruction Pointer    +--------->>> Process status code: (output)
         ---------->>|     SP - Stack pointer          |    
Environment          +-----------------------+--------+    int main(int argc, char** argv)
Variables (input)                            |              {
                                             |                  .... ... ...
export PATH=$PATH:/opt/bin/program/bin       |                  int status_code = 0 
export APP_LOG_LEVEL=INFO                    |                  return status_code;
                                            \ /              }
                                      Output File

Example: View Process Information on Linux

All mentioned process internal states can be viewed with a debugger such as GDB (GNU Debugger). On Linux, they can also be accessed with the /proc pseudo-file system created by the Linux kernel which exposes many processes informations.

Example: View information about the process which the PID is 19529

  • Start a Kotlin REPL process and get a PID:
$ kotlinc-jvm
Welcome to Kotlin version 1.3.50 (JRE 1.8.0_162-b12)
Type :help for help, :quit for quit
>>> 
>>> java.lang.management.ManagementFactory.getRuntimeMXBean().getName()
res1: kotlin.String! = 23127@localhost.localdomain
>>>
  • Proc-File system for PID 23127
$ ls /proc/23127
attr/       cmdline          loginuid       personality   statm
fd/         comm             maps           projid_map    status
fdinfo/     coredump_filter  mem            root@         syscall
map_files/  cpuset           mountinfo      sched         timers
net/        cwd@             mounts         schedstat     timerslack_ns
ns/         environ          mountstats     sessionid     uid_map
task/       exe@             numa_maps      setgroups     wchan
autogroup   gid_map          oom_adj        smaps
auxv        io               oom_score      smaps_rollup
cgroup      latency          oom_score_adj  stack
clear_refs  limits           pagemap        stat
  • Get process executable:
    • File: /proc/<PID>/exe is a symbolic link to the process' native executable.
$ readlink /proc/23127/exe 
/home/archbox/opt/java/bin/java
  • Get process current directory:
    • File: /proc/<PID>/cwd is a symbolic link to process' current directory.
$ readlink /proc/23127/cwd
/home/archbox/projects/appdroid/build
  • Get command line arguments used to start the process.
    • File: /proc/<PID>/cmdline contains the command line arguments used to start the process, passed to main(int argc, char** argv).
$ cat /proc/23127/cmdline | strings -1

/home/archbox/opt/java/bin/java
-Xmx256M
-Xms32M
-noverify
-cp
/home/archbox/.sdkman/candidates/kotlin/1.3.50/lib/kotlin-preloader.jar
org.jetbrains.kotlin.preloading.Preloader
-cp
/home/archbox/.sdkman/candidates/kotlin/1.3.50/lib/kotlin-compiler.jar
org.jetbrains.kotlin.cli.jvm.K2JVMCompiler
  • Get process environment variables.
    • File: /proc/<PID>/environ contains process' environment variables.
$ cat /proc/23127/environ | strings -1

QTINC=/usr/lib64/qt-3.3/include
LC_ALL=en_US.UTF-8
...    ...    ...    ...    ... 
...    ...    ...    ...    ... 
USERNAME=archbox
JAVA_HOME=/home/archbox/opt/java
KDEDIRS=/usr
KOTLIN_HOME=/home/archbox/.sdkman/candidates/kotlin/1.3.50
XDG_VTNR=2
SSH_AUTH_SOCK=/tmp/ssh-7z2L0gyqQeeP/agent.3961
ROOTSYS=/home/archbox/opt/root
XDG_SESSION_ID=2
MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl
ENV=/usr/share/Modules/init/profile.sh
DESKTOP_SESSION=lxqt
IMSETTINGS_MODULE=none
PWD=/home/archbox/projects/appdroid/build
  • List all process' file descriptors.
$ ls /proc/23127/fd
0@  10@  12@  14@  16@  18@  2@   21@  24@  28@  3@   31@  4@  6@  8@
1@  11@  13@  15@  17@  19@  20@  22@  27@  29@  30@  33@  5@  7@  9@

$ ls -l /proc/23127/fd
total 0
lrwx------ 1 archbox archbox 64 Nov  4 12:11 0 -> /dev/pts/4
lrwx------ 1 archbox archbox 64 Nov  4 12:11 1 -> /dev/pts/4
lr-x------ 1 archbox archbox 64 Nov  4 12:11 10 -> /home/archbox/.sdkman/candidates/kotlin/1.3.50/lib/kotlin-script-runtime.jar
lr-x------ 1 archbox archbox 64 Nov  4 12:11 11 -> /home/archbox/.sdkman/candidates/kotlin/1.3.50/lib/trove4j.jar
lr-x------ 1 archbox archbox 64 Nov  4 12:11 12 -> /home/archbox/opt/java/jre/lib/jsse.jar
lr-x------ 1 archbox archbox 64 Nov  4 12:11 13 -> /dev/random
lr-x------ 1 archbox archbox 64 Nov  4 12:11 14 -> /dev/urandom
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... 
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... 
lr-x------ 1 archbox archbox 64 Nov  4 12:11 8 -> /home/archbox/.sdkman/candidates/kotlin/1.3.50/lib/kotlin-stdlib.jar
lr-x------ 1 archbox archbox 64 Nov  4 12:11 9 -> /home/archbox/.sdkman/candidates/kotlin/1.3.50/lib/kotlin-reflect.jar

$ cat /proc/23127/maps | awk '{print $6}' | grep '\.so' | sort | uniq

/home/archbox/opt/java/jre/lib/amd64/libjava.so
/home/archbox/opt/java/jre/lib/amd64/libmanagement.so
...    ...    ...    ...    ...    ... 
...    ...    ...    ...    ...    ... 
/usr/lib64/libc-2.27.so
/usr/lib64/libdl-2.27.so
/usr/lib64/libm-2.27.so
/usr/lib64/libnss_files-2.27.so
/usr/lib64/libpthread-2.27.so
/usr/lib64/librt-2.27.so
/usr/lib64/libutil-2.27.so

See also:

1.16 Information about file permissions, owner and group - fstat and stat syscalls

1.16.1 Overview

The following family of common UNIX functions allows obtaining information about file permissions, owner, group and stick bits, namely, set-uid and set-gid special bits. 

#include <sys/stat.h>

// Obtain permission and file information from file path 
int stat(const char * path, struct stat* sb);

// Obtain permission and file information from file path 
int lstat(const char* path, struct stat* sb);

// Obtain permission and file information from file-descriptor 
int fstat(int fd, struct stat *sb);

int fstatat(int fd, const char* path, struct stat *sb, int flag);

Fstat manpage: 

struct stat {
    dev_t     st_dev;         /* ID of device containing file */
    ino_t     st_ino;         /* Inode number */
    mode_t    st_mode;        /* File type and mode */
    nlink_t   st_nlink;       /* Number of hard links */
    uid_t     st_uid;         /* User ID of owner */
    gid_t     st_gid;         /* Group ID of owner */
    dev_t     st_rdev;        /* Device ID (if special file) */
    off_t     st_size;        /* Total size, in bytes */
    blksize_t st_blksize;     /* Block size for filesystem I/O */
    blkcnt_t  st_blocks;      /* Number of 512B blocks allocated */

    /* Since Linux 2.6, the kernel supports nanosecond
       precision for the following timestamp fields.
       For the details before Linux 2.6, see NOTES. */

    struct timespec st_atim;  /* Time of last access */
    struct timespec st_mtim;  /* Time of last modification */
    struct timespec st_ctim;  /* Time of last status change */

#define st_atime st_atim.tv_sec      /* Backward compatibility */
#define st_mtime st_mtim.tv_sec
#define st_ctime st_ctim.tv_sec
};

See also:

Documentation:

1.16.2 Sample code

File: file-info.cpp 

#include <iostream> 
#include <string> 
#include <functional>
#include <cstring> 

// -- Unix-specific Headers -------//
#include <fcntl.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>

#include <pwd.h>
#include <grp.h>


std::string 
get_descriptor_type(struct stat const& fs)
{
    int ftype = fs.st_mode & S_IFMT;
    if(ftype == S_IFDIR)  return "Directory";
    if(ftype == S_IFREG)  return "Regular file";
    if(ftype == S_IFLNK)  return "Symbolic link";
    if(ftype == S_IFBLK)  return "Block device file";
    if(ftype == S_IFCHR)  return "Character device file";    
    if(ftype == S_IFIFO)  return "FIFO or pipe"; 
    if(ftype == S_IFSOCK) return "Socket";
    return "Unknown";    
}; 

int main(int argc, char** argv)
{
    if(argc != 2) {
        std::printf(" Usage: %s <FILE> \n", argv[0]);
        return 0;
    }

    struct stat fs; 
    struct passwd* fs_user = nullptr; 
    struct group*  fs_group = nullptr;


    //int fd = open(argv[1], O_RDONLY);

    if( ::stat(argv[1], &fs ) == -1) {
        std::fprintf(stderr, "Error: unable to open file => error = %s \n", strerror(errno) );
        return -1;
    }

    if( ( fs_user = ::getpwuid(fs.st_uid)) == nullptr )
    {
        std::fprintf(stderr, " Error: unable to get owner information \n");
        return -1;
    }

    if( ( fs_group = ::getgrgid(fs.st_gid)) == nullptr )
    {
        std::fprintf(stderr, " Error: unable to get group information \n");
        return -1;
    }


    // Print boolean as 'true' or 'false'
    std::cout << std::boolalpha;
    std::cout << "  [*]  File size in bytes: " << fs.st_size << '\n';
    std::cout << "  [*]  File size in Kilo bytes: " << static_cast<double>(fs.st_size) / 1024 << '\n';
    std::cout << "  [*]  File size in Mega bytes: " << static_cast<double>(fs.st_size) / (1024 * 1024) << '\n';
    std::cout << "  [*]           File type: " << get_descriptor_type(fs) << '\n';

    std::cout << "\n TEST FILE TYPE \n";
    std::cout << "  [*] ?      Is regular file =>> " << static_cast<bool>( S_ISREG(fs.st_mode) ) << '\n';
    std::cout << "  [*] ?         Is directory =>> " << static_cast<bool>( S_ISDIR(fs.st_mode) ) << '\n';
    std::cout << "  [*] ?     Is symbolic link =>> " << static_cast<bool>( S_ISLNK(fs.st_mode) ) << '\n';
    std::cout << "  [*] ?     Is block device  =>> " << static_cast<bool>( S_ISBLK(fs.st_mode) ) << '\n';
    std::cout << "  [*] ? Is character device  =>> " << static_cast<bool>( S_ISCHR(fs.st_mode) ) << '\n';

    std::cout << "\n ACCESS TIME \n";
    std::cout << "  [*] Time =>   last change: " << ctime(&fs.st_ctime);
    std::cout << "  [*] Time =>   last access: " << ctime(&fs.st_atime);
    std::cout << "  [*] Time => last modified: " << ctime(&fs.st_mtime);

    std::cout <<  "\n OWNER and GROUP \n";
    std::cout << "  [*] Owner: uid = " << fs.st_uid << " ;  user = " << fs_user->pw_name  << '\n';
    std::cout << "  [*] Group: gid = " << fs.st_gid << " ; group = " << fs_group->gr_name  << '\n';

    std::cout << "\n FILE PERMISSIONS \n";
    std::cout << "  =>>  Stick Bits: " << (fs.st_mode & S_ISUID ? "suid" : "-") 
              << "  "                  << (fs.st_mode & S_ISGID ? "sgid" : "-")
              << '\n';
    std::cout << "  =>>  Owner: " << (fs.st_mode & S_IRUSR  ? "r" : "-")   // File is readable   (can be read)
              << " "         << (fs.st_mode & S_IWUSR  ? "w" : "-")        // File is writeable  (can be written)
              << " "         << (fs.st_mode & S_IXUSR  ? "x" : "-")        // File is executable (can be executed - execv syscall)
              << '\n';
    std::cout << "  =>>  Group: " << (fs.st_mode & S_IRGRP  ? "r" : "-")  
              << " "            << (fs.st_mode & S_IWGRP  ? "w" : "-")  
              << " "            << (fs.st_mode & S_IXGRP  ? "x" : "-")
              << '\n';
    std::cout << "  =>> Others: " << (fs.st_mode & S_IROTH  ? "r" : "-")  
              << " "             << (fs.st_mode & S_IWOTH  ? "w" : "-")  
              << " "             << (fs.st_mode & S_IXOTH  ? "x" : "-")
              << '\n';    
    std::cout << " \n- Note: (r - read) ; (w - write); (x - execute) ; (suid - set-UID bit set) ; (sgid - set-GID bit set) \n";

    return 0;
}

Building: 

# Build using GNU GCC (G++) compiler 
$ g++ file-info.cpp -o file-info -std=c++1z -g -Wall -Wextra

# Build using LLVM/Clang compiler 
$ clang++ file-info.cpp -o file-info -std=c++1z -g -Wall -Wextra

Running:

  • Inspect directory /etc
$ ./file-info /etc
  [*]  File size in bytes: 12288
  [*]  File size in Kilo bytes: 12
  [*]  File size in Mega bytes: 0.0117188
  [*]           File type: Directory

 TEST FILE TYPE 
  [*] ?      Is regular file =>> false
  [*] ?         Is directory =>> true
  [*] ?     Is symbolic link =>> false
  [*] ?     Is block device  =>> false
  [*] ? Is character device  =>> false

 ACCESS TIME 
  [*] Time =>   last change: Sun Aug  2 05:19:31 2020
  [*] Time =>   last access: Wed Aug  5 04:50:20 2020
  [*] Time => last modified: Sun Aug  2 05:19:31 2020

 OWNER and GROUP 
  [*] Owner: uid = 0 ;  user = root
  [*] Group: gid = 0 ; group = root

 FILE PERMISSIONS 
  =>>  Stick Bits: -  -
  =>>  Owner: r w x
  =>>  Group: r - x
  =>> Others: r - x

- Note: (r - read) ; (w - write); (x - execute) ; (suid - set-UID bit set) ; (sgid - set-GID bit set)
  • Inspect executable file: /bin/sudo
$ ./file-info /bin/sudo
  [*]  File size in bytes: 182456
  [*]  File size in Kilo bytes: 178.18
  [*]  File size in Mega bytes: 0.174004
  [*]           File type: Regular file

 TEST FILE TYPE 
  [*] ?      Is regular file =>> true
  [*] ?         Is directory =>> false
  [*] ?     Is symbolic link =>> false
  [*] ?     Is block device  =>> false
  [*] ? Is character device  =>> false

 ACCESS TIME 
  [*] Time =>   last change: Tue May 12 01:52:08 2020
  [*] Time =>   last access: Sun Aug  2 05:20:57 2020
  [*] Time => last modified: Fri Mar 27 05:50:36 2020

 OWNER and GROUP 
  [*] Owner: uid = 0 ;  user = root
  [*] Group: gid = 0 ; group = root

 FILE PERMISSIONS 
  =>>  Stick Bits: suid  -
  =>>  Owner: - - x
  =>>  Group: - - x
  =>> Others: - - x

- Note: (r - read) ; (w - write); (x - execute) ; (suid - set-UID bit set) ; (sgid - set-GID bit set)
  • Inspect regular file: ~/bin/fserver.jsh
$ ./file-info ~/bin/fserver.jsh 
  [*]  File size in bytes: 9968346
  [*]  File size in Kilo bytes: 9734.71
  [*]  File size in Mega bytes: 9.50656
  [*]           File type: Regular file

 TEST FILE TYPE 
  [*] ?      Is regular file =>> true
  [*] ?         Is directory =>> false
  [*] ?     Is symbolic link =>> false
  [*] ?     Is block device  =>> false
  [*] ? Is character device  =>> false

 ACCESS TIME 
  [*] Time =>   last change: Tue May 12 12:35:32 2020
  [*] Time =>   last access: Tue Aug  4 03:21:06 2020
  [*] Time => last modified: Mon May 11 22:52:14 2020

 OWNER and GROUP 
  [*] Owner: uid = 1000 ;  user = myuser 
  [*] Group: gid = 1000 ; group = myuser 

 FILE PERMISSIONS 
  =>>  Stick Bits: -  -
  =>>  Owner: r w x
  =>>  Group: r w -
  =>> Others: r - -

- Note: (r - read) ; (w - write); (x - execute) ; (suid - set-UID bit set) ; (sgid - set-GID bit set)
  • Inspect block device file /dev/sda that represets the main disk partition.
$ ./file-info /dev/sda
  [*]  File size in bytes: 0
  [*]  File size in Kilo bytes: 0
  [*]  File size in Mega bytes: 0
  [*]           File type: Block device file

 TEST FILE TYPE 
  [*] ?      Is regular file =>> false
  [*] ?         Is directory =>> false
  [*] ?     Is symbolic link =>> false
  [*] ?     Is block device  =>> true
  [*] ? Is character device  =>> false

 ACCESS TIME 
  [*] Time =>   last change: Wed Jul  8 13:48:48 2020
  [*] Time =>   last access: Wed Jul  8 13:49:09 2020
  [*] Time => last modified: Wed Jul  8 13:48:48 2020

 OWNER and GROUP 
  [*] Owner: uid = 0 ;  user = root
  [*] Group: gid = 6 ; group = disk

 FILE PERMISSIONS 
  =>>  Stick Bits: -  -
  =>>  Owner: r w -
  =>>  Group: r w -
  =>> Others: - - -

- Note: (r - read) ; (w - write); (x - execute) ; (suid - set-UID bit set) ; (sgid - set-GID bit set)
  • Inspect character device file /dev/mem that represents the physical memory (RAM memory).
$ ./file-info /dev/mem
  [*]  File size in bytes: 0
  [*]  File size in Kilo bytes: 0
  [*]  File size in Mega bytes: 0
  [*]           File type: Character device file

 TEST FILE TYPE 
  [*] ?      Is regular file =>> false
  [*] ?         Is directory =>> false
  [*] ?     Is symbolic link =>> false
  [*] ?     Is block device  =>> false
  [*] ? Is character device  =>> true

 ACCESS TIME 
  [*] Time =>   last change: Wed Jul  8 13:48:49 2020
  [*] Time =>   last access: Wed Jul  8 13:48:49 2020
  [*] Time => last modified: Wed Jul  8 13:48:49 2020

 OWNER and GROUP 
  [*] Owner: uid = 0 ;  user = root
  [*] Group: gid = 9 ; group = kmem

 FILE PERMISSIONS 
  =>>  Stick Bits: -  -
  =>>  Owner: r w -
  =>>  Group: r - -
  =>> Others: - - -

- Note: (r - read) ; (w - write); (x - execute) ; (suid - set-UID bit set) ; (sgid - set-GID bit set)

1.17 Information about current process

File: current-process.cpp 

#include <iostream> 
#include <string> 
#include <cstring> 

#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <limits.h> // PATH_MAX constant (macro)

auto get_cwd() -> std::string;
auto set_cwd(const std::string& path) -> void;
auto read_symlink(std::string const& path) -> std::string;

int main(int argc, char** argv)
{
    // Unique process ID (identifier) number 
    fprintf(stdout, "   => Process PID = %d \n", ::getpid());

    // PID for parent process
    fprintf(stdout, "   => Process PID = %d \n", ::getppid());

    // Current directory 
    fprintf(stdout, "   => Current directory = %s \n", get_cwd().c_str());

    fprintf(stdout, "\n Change the curren directory to /etc \n");
    set_cwd("/etc");
    fprintf(stdout, "   => Current directory = %s \n", get_cwd().c_str());

    // Note: It only works on Linux. The directory /proc/self is a pseudo-directory 
    //       whith pseudo-files containing information about the current process. 
    // 
    // the file /proc/self/exe is a symbolic link to the current executable.
    std::string exe_file = read_symlink("/proc/self/exe");
    fprintf(stdout, "  => Absolute Path of current executable = %s \n", exe_file.c_str());

    // Current directory 
    std::string exe_dir = read_symlink("/proc/self/cwd");
    fprintf(stdout, "  => Current directory of this executable = %s \n", exe_dir.c_str());

    fprintf(stdout, " [INFO] Finish Ok. \n");
    return 0;
}

// ----------- Functions Implementations ------------// 
// 

/** Get current working directory of current process */
auto get_cwd() -> std::string 
{
    char* buffer = ::getcwd(nullptr, 0);
    auto cwd = std::string(buffer);
    // Note buffer was allocated with malloc 
    free(buffer);
    return cwd;
}

/** Set current working directory for current process. */
auto set_cwd(const std::string& path) -> void 
{
     int status = ::chdir(path.c_str());
     if(status < 0)
         throw std::runtime_error("Failed to change directory.");
}

/** Read value of symbolic link 
 *  
 * Requires: <limits.h>, <sys/types.h>, <sys/stats.h>
 */
auto read_symlink(std::string const& path) -> std::string 
{
    // Create a buffer with size PATH_MAX + 1 filled with 0 ('\0'), null characters
    std::string buffer(PATH_MAX, 0);
    // ssize_t readlink(const char *pathname, char *buf, size_t bufsiz);
    ssize_t nread = ::readlink(path.c_str(), &buffer[0], PATH_MAX);
    if(nread == -1){
        fprintf(stderr, " Error: %s \n", strerror(errno));
        throw std::runtime_error("Error: unable to read symlink. Check 'errno' variable");
    }
    buffer.resize(nread);
    return buffer;
}

Building: 

$ >> g++ current-process.cpp -o current-process.elf -Wall -Wextra -ggdb

Running: 

$ >> ./current-process.elf 
  => Process PID = 719155 
  => Process PID = 678179 
  => Current directory = /home/mxpkf8/temp-projects/unix-explore 

Change the curren directory to /etc 
  => Current directory = /etc 
 => Absolute Path of current executable = /home/mxpkf8/temp-projects/unix-explore/current-process.elf 
 => Current directory of this executable = /etc 
[INFO] Finish Ok.

1.18 Dynamic Loading Shared Libraries / dlopen() API

1.18.1 Overview

The dlopen API provide access to the dynamic linker services allowing the current process to load and unload shared objects or shared libraries in its address space (virtual memory).

This API for loading shared libraries is not only available on Linux, it also can be found in FreeBSD, NetBSD, MacOSX, Android and so on.

Use cases:

  • Load new code at-runtime
  • Load third-party code a runtime
  • Plugin systems, extensions or addons.
  • Load native code extensions in languages with interpreters written in C, such as Python, Ruby, Lua and so on. Note: Python native extensions are shared libraries.

The Dlopen() API is available on most Unix-like operating systems, such as:

  • Linux
  • MacOSX
  • FreeBDS
  • NetBSD
  • OpenBSD
  • QNX

Documentation:

Further Reading:

Headers and libraries:

  • Header: #include <dlfcn.h>
  • Linking: (-ldl) flag

Function dlopen():

  • Doc: $ man dlopen
  • Loads a shared library and returns a handle or opaque pointer casted as void pointer. The term opaque means that the implementation is hidden and the pointer is only meant to be passed around to other functions.
  • Note: When this function fails, it returns null pointer.
void* dlopen(const char* filename, int flags);

Function dlsym():

  • Doc: $ man dlsym
  • Obtains address of a shared library symbol from handle (from dlopen).
  • Params:
    • handle => Shared library handle obtained from dlopen
    • symbol => Name of symbol to be loaded.
  • Return:
    • Address of symbol casted as void*. It can be a function-pointer or a pointer to global variable. If the symbol is not found, the function returns a null pointer.
void* dlsym(void* handle, const char* symbol);

Function dlvsym():

  • Doc: $ man dlvsym
    • Loads a specific version of a symbol.
void* dlvsym(void *handle, char *symbol, char *version);

Function dlclose():

  • Doc: $ man dlclose()
int dlclose(void *handle);

Function dlerror()

  • Obtaines error messages from dlopen API.
  • Doc: $ man dlerror()
char* dlerror(void);

Simplified type signatures with a C++-friendly notation 

// Opaque pointer for shared library (shared object) handle 
using HND = void* ; 
// Opaque pointer for shared library symbol 
using SYM = void* ; 

HND  dlopen( const char* filename, int flags);
SYM   dlsym( HND handle, const char* symbol);
SYM  dlvsym( HND handle, char *symbol, char *version);
int dlclose( HND handle );

Usage example 

#include <dlfcn.h>

// Opaque pointer for shared library (shared object) handle 
using HND = void* ; 
// Opaque pointer for shared library symbol 
using SYM = void* ; 

void load_library()
{
    // Load shared library 
    HND hnd = dlopen("/path/to/shared-library.so", RTLD_LAZY | RTLD_GLOBAL); 
    if(hnd == nullptr)
    {
       std::cerr << "  [ERROR] " << dlerror() << "\n";
       throw std::runtime_error("Error: unable to load shared library"); 
    }

    // Load symbol 
    SYM hsym = dlsym(hnd, "name_of_function"); 

    // if(!hsym)
    if(hsym == nullptr)
    {
       throw std::runtime_error("Error: symbol not found"); 
    }

    // Type alias for function pointer in modern C++ 
    //  => Note: The function to be loaded can only have C-linkage. 
    using name_of_function_t = void (*) (int param0, double param1, const char* param3);

    // Type alias for function pointer in old C++ (C++98)
    typedef void (*name_of_function_t) (int param0, double param1, const char* param3);       

    auto name_of_function_ptr = reinterpret_cast<name_of_function_t>(hsym); 

    // Call loaded function (Function pointer)
    name_of_function_ptr(100, 2.51, "string");

    // Unload shared library 
    dlclose(hsym);
}

1.18.2 Loading Gtk2/Gtk3 - low level

The following sample code is a GTk3 or GTK2 GUI graphical user interface application with GTK functions loaded from a GTK shared library, without any compile-time linking. The advantage of this approach is the greater portability across different versions of GTk and the ability to switch between GT3 and GT2.

GIST:

File: CMakeLists.txt 

cmake_minimum_required(VERSION 3.9)
project(gtk-sample)

#========== Global Configurations =============#
#----------------------------------------------#

set(CMAKE_CXX_STANDARD 17)     
set(CMAKE_VERBOSE_MAKEFILE ON)

       add_executable( gtk-dlopen gtk-dlopen.cpp)
target_link_libraries( gtk-dlopen dl )

File: gtk-dlopen.cpp 

#include <iostream>
#include <cstring> 
#include <cassert> 

// Uses: dlopen(), dlclose(), ... 
#include <dlfcn.h>

// ---- Copied from GTK headers ------// 
typedef char            gchar;
typedef short           gshort;
typedef long            glong;
typedef int             gint;
typedef gint            gboolean;
typedef unsigned char   guchar;
typedef unsigned short  gushort;
typedef unsigned long   gulong;
typedef unsigned int    guint;
typedef float           gfloat;
typedef double          gdouble;
typedef void*           gpointer;
typedef const void*     gconstpointer;

enum class GtkWindowType
{
    GTK_WINDOW_TOPLEVEL,
    GTK_WINDOW_POPUP
};

enum class GConnectFlags
{
  G_CONNECT_AFTER   = 1 << 0,
  G_CONNECT_SWAPPED = 1 << 1
};

template<typename TFun>
TFun load_symbol(void* hnd, std::string symbol)
{
    void* hSym = dlsym(hnd, symbol.c_str());
    if(hSym == nullptr)
    {
        std::string msg = std::string(" [Error] symbol not found: ") + symbol;
        throw std::runtime_error(msg);
    }    
    return reinterpret_cast<TFun>(hSym);
}

// Opaque type (aka incomplete type)
struct GClosure;

// -------- Function Pointers type aliases -------------------//
using GCallback      = void  (*) (void);
using GClosureNotify = void  (*) (gpointer data, GClosure* closure);
using g_signal_connect_data_t =  gulong (*) (   gpointer instance
                                              , const gchar*    detailed_signal
                                              , GCallback       c_handler
                                              , gpointer        data
                                              , GClosureNotify  destroy_data
                                              , GConnectFlags   connect_flags
                                          );

int main(int argc, char** argv)
{    
    std::string shared_lib =  [&]() -> std::string 
    {
        if(argc < 2) return "libgtk-3.so.0";
        return argv[1];
    }();

    std::cout << " [INFO] Loading shared library: " << shared_lib << "\n";

    //  void *dlopen(const char *filename, int flags);

    // Handle to shared library 
    void* hnd = dlopen(shared_lib.c_str(), RTLD_NOW | RTLD_GLOBAL);
    if(hnd == nullptr){ fprintf(stderr, "%s\n", dlerror());  }
    assert( hnd != nullptr );

    void* hSym = nullptr; 

    // --------- Load gtk_init_check function pointer ------- // 
    hSym = dlsym(hnd, "gtk_init_check");
    assert(hSym != nullptr);
    using gtk_init_check_t = gboolean (*) (int* argc, char*** argv);
    auto gtk_init_check = reinterpret_cast<gtk_init_check_t>(hSym);   

    // ------- Load remaining function pointers (symbols) --------// 
    //-----------------------------------------------------------//

    // opaque pointer 
    struct GtkWidget;

    using gkt_window_new_t = GtkWidget* (*) (int);
    auto gtk_window_new  = load_symbol<gkt_window_new_t>(hnd, "gtk_window_new");    
    auto gtk_widget_show = load_symbol<void (*) (GtkWidget*)>(hnd, "gtk_widget_show");
    auto gtk_main        = load_symbol<void (*) ()>(hnd, "gtk_main");

    using gtk_window_set_title_t = void (*) (GtkWidget* window, const gchar* title);
    auto gtk_window_set_title = load_symbol<gtk_window_set_title_t>(hnd, "gtk_window_set_title");

    using gtk_widget_set_size_t = void (*) (GtkWidget*, gint, gint);
    auto gtk_widget_set_size = load_symbol<gtk_widget_set_size_t>(hnd, "gtk_widget_set_size_request");

    auto gtk_main_quit = load_symbol<void (*) ()>(hnd, "gtk_main_quit");

    auto gtk_signal_connect_data 
          = load_symbol<g_signal_connect_data_t>(hnd, "g_signal_connect_data");

    /** ------- Build Window GUI - Graphical User Interface ----------**/

    // Call function pointer 
    gtk_init_check(&argc, &argv);

    GtkWidget* window = gtk_window_new( (int) GtkWindowType::GTK_WINDOW_TOPLEVEL);
    gtk_widget_set_size(window, 400, 500);
    gtk_window_set_title(window, "My GTK Window");
    gtk_widget_show(window);    

    gtk_signal_connect_data(  window         // Widget 
                            , "destroy"      // Event name 
                            , gtk_main_quit  // Callback                             
                            , nullptr        // Pointer to data (closure )
                            , nullptr 
                            , (GConnectFlags) 0 // GConnect flags 
                            );

    std::cout << " [INFO] Window running. OK! " << "\n";
    gtk_main();

    // Always close shared library handler.
    dlclose(hnd);

    std::cout << " [INFO] Shutdown gracefully. Ok. \n" ;
    return EXIT_SUCCESS;
}

Building: 

$ git clone https://gist.github.com/d81dd9c422dc93ad4104b5e79259afdf && cd gist 
$ g++ gtk-dlopen.cpp -o gtk-dlopen.bin -std=c++1z -ldl -ggdb -Wall -Wextra

Checking the dependencies of gtk-dlopen.bin 

$ >> ldd gtk-dlopen.bin 
       linux-vdso.so.1 (0x00007ffd23fc4000)
       libdl.so.2 => /lib64/libdl.so.2 (0x00007f6a78428000)
       libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f6a78238000)
       libm.so.6 => /lib64/libm.so.6 (0x00007f6a780f2000)
       libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f6a780d7000)
       libc.so.6 => /lib64/libc.so.6 (0x00007f6a77f0d000)
       /lib64/ld-linux-x86-64.so.2 (0x00007f6a78453000)

Running: (load GTK shared library /lib64/libgtk-3.so.0) 

$ >> ./gtk-dlopen.bin 
[INFO] Loading shared library: /lib64/libgtk-3.so.0
[INFO] Window running. OK! 
[INFO] Shutdown gracefully. Ok.

Running: (load GK2 shared library ) 

$ >> ./gtk-dlopen.bin /usr/lib64/libgtk-x11-2.0.so 
[INFO] Loading shared library: /usr/lib64/libgtk-x11-2.0.so
[INFO] Window running. OK! 
[INFO] Shutdown gracefully. Ok.

Verify exported symbols by gtk3 shared library: 

$ >> nm -D /lib64/libgtk-3.so.0 | grep init
       ... ... ... ... ..
       ... ... ... ... ..
       U g_async_initable_new_async
       U g_async_initable_new_finish
       U g_datalist_init
       U g_hash_table_iter_init
       U g_initable_get_type
       U g_initable_new
       U g_initially_unowned_get_type
       U g_mutex_init
       U g_once_init_enter
      ... ... ... ... ..
      ... ... ... ... ..

Screenshot:

gtk-dlopen-screen1.png

Figure 1: Screenshot of GTk Window created using dlopen() API

1.18.3 Loading Gtk2/Gtk3 - high level C++ wrapper

This proof-of-concept code contains a high-level wrapper class SharedLibrary for abstracting away the dlopen() API. This class is used for loading GTK3 or GTk2 shared library at runtime and then showing modal notifications dialogs. The advantage of the dynamic loading approach is the flexibility, portability and easier deployment as the binary does not depends on any particular GTK shared library version.

Note:

  • The code of the class SharedLibrary can be reused on any other unix-like operating system with dlopen() API, such as FreeBDS, NetBSD, MacOSX and so on.
  • The Windows API for dynamic loading is LoadLibrary(), GetProcAddress().

GIST:

Project Files

File: CMakeLists.txt 

cmake_minimum_required(VERSION 3.9)
project(gtk-dynamic-load-dialog)

set(CMAKE_CXX_STANDARD 17)     
set(CMAKE_VERBOSE_MAKEFILE ON)

# ---- TARGETS Configuration ---------------#
       add_executable( gtk-dialog gtk-dialog.cpp gtk_types.hpp)
target_link_libraries( gtk-dialog dl )

File: gtk-dialog.cpp 

#include <iostream>
#include <string> 
#include <cstring> 
#include <cassert> 

// Uses: dlopen(), dlclose(), ... 
#include <dlfcn.h>

#include "gtk_types.hpp"

/** Requires header: <dlfcn.h> and linking flag (-ldl) */
class SharedLibrary
{
public:
    using HND  = void*; 
    using HSYM = void*;
private:
    HND         m_hnd  = nullptr; 
    std::string m_lib = "" ; 
    int         m_flags = 0;
public:
    SharedLibrary(){}    
    ~SharedLibrary(){ this->unload(); }

    // Constructor delegation 
    SharedLibrary(std::string library)
        : SharedLibrary(library, RTLD_NOW | RTLD_GLOBAL)
    {  }

    SharedLibrary(std::string library, int flags)
    {
        this->load(library, flags);
    }

    // Disable copy constructor and copy-assignment operator 
    SharedLibrary(SharedLibrary const &) = delete;
    SharedLibrary& operator=(SharedLibrary const &) = delete;

    // Move constructor 
    SharedLibrary(SharedLibrary&& rhs) 
    {
        std::swap(m_hnd, rhs.m_hnd);
        std::swap(m_lib, rhs.m_lib);
        std::swap(m_flags, rhs.m_flags);
    }

    // Move assignment operator 
    SharedLibrary& operator=(SharedLibrary&& rhs) 
    {
        this->unload();
        std::swap(m_hnd, rhs.m_hnd);
        std::swap(m_lib, rhs.m_lib);
        std::swap(m_flags, rhs.m_flags);
        return *this;
    }

    HND         handler()   const { return m_hnd; }    
    bool        is_loaded() const { return m_hnd != nullptr; }    
    std::string library()   const { return m_lib; }

    /** Retrieve symbol from shared library */
    HSYM symbol_ptr(std::string name) const 
    {
        return ::dlsym(m_hnd, name.c_str());
    }

    /** @brief Get symbol from shared library
     * Note: The caller is resposible for checking if the symbol is (null) nullptr.
     */
    template<typename TFun>
    TFun symbol(std::string name) const 
    {
        HSYM sym = ::dlsym(m_hnd, name.c_str());
        return reinterpret_cast<TFun>(sym);
    }

    /** @brief Attempts to load symbol and throws exception, if symbol is not found. */
    template<typename TFun>
    TFun symbol_or_fail(std::string name) const 
    {
        HSYM sym = ::dlsym(m_hnd, name.c_str());

        if(!sym){ 
            std::string msg = " [ERROR] Dlopen / dlsym() => unable find symbol";
            msg = msg + " <" + name + ">";
            throw std::runtime_error(msg); 
        }
        return reinterpret_cast<TFun>(sym);
    }

    void load(std::string library, int flags)
    {
        if(!this->is_loaded()) { this->unload(); }
        m_lib   = library;
        m_flags = flags;
        m_hnd = dlopen(library.c_str(), RTLD_NOW | RTLD_GLOBAL);        
        if(m_hnd == nullptr){ 
            std::string msg = " [ERROR] DlopenError: ";
            msg = msg + dlerror();
            throw std::runtime_error(msg);
        }
    }    

    void reload() 
    {
        this->unload();
        this->load(m_lib, m_flags);
    }

    void unload() 
    {
        if(!this->is_loaded()) { return; }
        ::dlclose(m_hnd);        
        m_hnd = nullptr;
    }

}; // --- End of class SharedLibrary() ---- //

using namespace GtkPP::fpointer;
using namespace GtkPP::enums;

struct GtkLib
{
    SharedLibrary slib;

    GtkLib(std::string library)
        : slib( std::move(SharedLibrary(library)) )
    { 
        gtk_init_check         = slib.symbol_or_fail<gtk_init_check_t>("gtk_init_check");       
        gtk_main_quit          = slib.symbol_or_fail<gtk_main_quit_t>("gtk_main_quit");
        // gtk_main               = slib.symbol_or_fail<void (*) ()>("gtk_main");
        gtk_message_dialog_new = slib.symbol_or_fail<gtk_message_dialog_new_t>("gtk_message_dialog_new");
        gtk_widget_destroy     = slib.symbol_or_fail<gtk_widget_destroy_t>("gtk_widget_destroy");
        gtk_dialog_run         = slib.symbol_or_fail<gtk_dialog_run_t>("gtk_dialog_run");
        gtk_window_set_title   = slib.symbol<gtk_window_set_title_t>("gtk_window_set_title");

    }

    gtk_init_check_t         gtk_init_check; 
    gtk_main_quit_t          gtk_main_quit; 
    // gtk_main_t               gtk_main;
    gtk_message_dialog_new_t gtk_message_dialog_new;
    gtk_widget_destroy_t     gtk_widget_destroy;
    gtk_dialog_run_t         gtk_dialog_run;
    gtk_window_set_title_t   gtk_window_set_title;

    // Message box notification 
    void msgbox(  std::string title
                , std::string text 
                , GtkMessageType mtype = GtkMessageType::GTK_MESSAGE_INFO 
                ) 
    {
       GtkWidget* dialog = this->gtk_message_dialog_new(  nullptr 
                                                , (int) GtkDialogFlags::GTK_DIALOG_MODAL
                                                , (int) mtype 
                                                , (int) GtkButtonsType::GTK_BUTTONS_OK
                                                , text.c_str()
                                                );                                            
        assert( dialog != nullptr );    
        this->gtk_window_set_title(dialog, title.c_str());
        this->gtk_dialog_run(dialog);
        std::cout << " [INFO] Dialog destroyed OK " << "\n";
        this->gtk_widget_destroy(dialog);   
    }
};


int main(int argc, char** argv)
{        
    // User can select his own GTK shared library, if it is not 
    // found. He can also select GTK2 shared library. 
    auto shared_lib =  [&]() -> std::string 
    {
        if(argc < 2) return "libgtk-3.so.0";
        return argv[1];
    }();

    std::cout << " [INFO] Loading shared library: " << shared_lib << "\n";   
    auto gtk = GtkLib{shared_lib};   

    // Alwas start the library before calling any GTK function 
    gtk.gtk_init_check(&argc, &argv);    

    gtk.msgbox("User notification", "Download completed. Ok.",     GtkMessageType::GTK_MESSAGE_INFO );    
    gtk.msgbox("Runtime error",     "KERNEL PANIC!! Reboot NOW!!", GtkMessageType::GTK_MESSAGE_ERROR );        

    std::cout << " [INFO] Shutdown gracefully. Ok. \n" ;
    return EXIT_SUCCESS;
}

File: gtk_types.hpp 

// ------ C++ Wrappers for dynamically loading GTK ----- //
#ifndef _GTK_TYPES_HPP
#define _GTK_TYPES_HPP

namespace GtkPP 
{
    namespace base {
        using gchar    = char;
        using gshort   = short;
        using glong    = long;
        using gint     = int;
        using gboolean = gint;
        using guchar   = unsigned short;
        using gulong   = unsigned long;
        using guint    = unsigned int;
        using gfloat   = float;
        using gdouble  = double;
        using gpointer = void*;          
        using gconstpointer = const void*;

    };
    // ------- Opaque Types / incomplete types ---------//
    namespace widget {        
        struct GtkWidget;
        struct GClosure;
        struct GtkWidget;
        struct GtkWindow;
        struct GtkDialog;
    };

    namespace enums {
        enum class GtkWindowType
        {
            GTK_WINDOW_TOPLEVEL,
            GTK_WINDOW_POPUP
        };

        enum class GConnectFlags
        {
        G_CONNECT_AFTER = 1 << 0,
        G_CONNECT_SWAPPED   = 1 << 1
        };

        enum class GtkMessageType
        {
        GTK_MESSAGE_INFO,
        GTK_MESSAGE_WARNING,
        GTK_MESSAGE_QUESTION,
        GTK_MESSAGE_ERROR,
        GTK_MESSAGE_OTHER
        };

        enum class GtkButtonsType
        {
        GTK_BUTTONS_NONE,
        GTK_BUTTONS_OK,
        GTK_BUTTONS_CLOSE,
        GTK_BUTTONS_CANCEL,
        GTK_BUTTONS_YES_NO,
        GTK_BUTTONS_OK_CANCEL
        };

        enum class GtkDialogFlags
        {
        GTK_DIALOG_MODAL               = 1 << 0,
        GTK_DIALOG_DESTROY_WITH_PARENT = 1 << 1,
        GTK_DIALOG_USE_HEADER_BAR      = 1 << 2
        };
    };

    /** @brief Function pointers type aliases  */
    namespace fpointer 
    {
        using namespace base;
        using namespace widget;
        using enums::GConnectFlags;

        using GtkDialogFlags_t = int; 
        using GtkMessageType_t = int; 
        using GtkButtonsType_t = int;

        using gtk_init_check_t = gboolean (*) (int* argc, char*** argv);
        using gtk_dialog_run_t       = gint (*) (void* dialog);
        using gtk_widget_destroy_t   = void (*) (GtkWidget* widget);
        using gtk_window_set_title_t = void (*) (GtkWidget* window, const gchar* title);
        using gtk_main_quit_t        = void (*) ();

        using gtk_window_set_title_t = void (*) (GtkWidget* window, const gchar* title);


        using GCallback      = void  (*) (void);
        using GClosureNotify = void  (*) (gpointer data, GClosure* closure);
        using g_signal_connect_data_t =  gulong (*) ( gpointer        instance
                                                    , const gchar*    detailed_signal
                                                    , GCallback       c_handler
                                                    , gpointer          data
                                                    , GClosureNotify  destroy_data
                                                    , GConnectFlags connect_flags
                                                    );


        using gtk_message_dialog_new_t = GtkWidget* (*) ( GtkWindow       *parent,
                                                          GtkDialogFlags_t  flags,
                                                          GtkMessageType_t  type,
                                                          GtkButtonsType_t  buttons,
                                                          const gchar*    message_format
                                                        );


    };

};

#endif

Building and running: 

# Download 
$ >> git clone https://gist.github.com/6bbb3b9491d4fad01d3875a559b7365b gtk-dialog && cd gtk-dialog 

# Configure 
$ >> cmake --config Debug -H. -B_build

# Build 
$ >> cmake --build _build --target all 

# Run 
$ >> _build/gtk-dialog 
[INFO] Loading shared library: libgtk-3.so.0
[INFO] Dialog destroyed OK 
[INFO] Dialog destroyed OK 
[INFO] Shutdown gracefully. Ok.

Screenshots:

gtk-dlopen-screen2.png

Figure 2: Gtk modal dialog screenshot 1

gtk-dlopen-screen3.png

Figure 3: Gtk modal dialog screenshot 2

1.18.4 Loading Gtk2/Gtk3 using D DLang language

The following code creates Gtk graphics user interface by loading Gtk function pointers from Gtk shared library using D (DLang) programming language.

File: gtk_dlopen.d 

import core.stdc.stdio;
import io = std.stdio;
import std.range: empty;
import std.algorithm : map, joiner;
import std.string;
import str = std.string;
import std.array: array;
import std.conv: to;
import DLL = core.sys.posix.dlfcn;
import core.sys.posix.dlfcn;


struct ButtonState 
{
  int counter = 0;
  gtk_message_dialog_new_t gtk_message_dialog_new = null;
  gtk_window_set_title_t   gtk_window_set_title   = null; 
  gtk_dialog_run_t         gtk_dialog_run         = null; 
  gtk_widget_destroy_t     gtk_widget_destroy     = null;
}

extern(C) 
void buttonCallback(GtkWidget* widget, gpointer data)
{
  auto state = cast(ButtonState*) data;
  state.counter = state.counter + 1;
  io.writeln(" [TRACE] Button Called => Counter  = ", state.counter);
  auto message = " [INFO] Counter changed to " ~ to!string(state.counter);
  auto dialog = state.gtk_message_dialog_new( null
                                            , GtkDialogFlags.GTK_DIALOG_MODAL
                                            , GtkMessageType.GTK_MESSAGE_INFO
                                            , GtkButtonsType.GTK_BUTTONS_OK
                                            , message.toStringz
                                            );
  state.gtk_window_set_title(dialog, "Gtk Info dialog from DLang");
  // io.writeln(" [TRACE] Run message dialog ");
  state.gtk_dialog_run(dialog);
  // Dispose GTk object 
  state.gtk_widget_destroy(dialog);
}

int main(string[] args)
{
  printf(" [TRACE] Application started Ok. \n");

  void* dll = DLL.dlopen("libgtk-3.so.0", DLL.RTLD_GLOBAL | DLL.RTLD_LAZY);
  if (!dll)
  {
      fprintf(stderr, " [ERROR] dlopen error: %s\n", DLL.dlerror());
      return 1;
  }

  //-------------------------------------------//
  //    Load GTK Function pointers             //
  //-------------------------------------------//

  auto gtk_init_check               = cast(gtk_init_check_t)( dll.dlsym("gtk_init_check") );
  auto gtk_window_new               = cast(gtk_window_new_t)( dll.dlsym("gtk_window_new") );
  auto gtk_widget_show              = cast(gtk_widget_show_t)( dll.dlsym("gtk_widget_show") );
  auto gtk_main                     = cast(gtk_main_t)( dll.dlsym("gtk_main") );
  auto gtk_window_set_title         = cast(gtk_window_set_title_t)( dll.dlsym("gtk_window_set_title") );
  auto gtk_widget_set_size          = cast(gtk_widget_set_size_t)( dll.dlsym("gtk_widget_set_size_request") );
  auto gtk_main_quit                = cast(gtk_main_quit_t)( dll.dlsym("gtk_main_quit") );
  auto g_signal_connect_data        = cast(g_signal_connect_data_t)( dll.dlsym("g_signal_connect_data") );
  auto gtk_widget_show_all          = cast(gtk_widget_show_all_t)( dll.dlsym("gtk_widget_show_all") );
  auto gtk_button_new_with_label    = cast(gtk_button_new_with_label_t)( dll.dlsym("gtk_button_new_with_label") );
  auto gtk_container_add            = cast(gtk_container_add_t)( dll.dlsym("gtk_container_add") );
  auto gtk_widget_set_size_request  = cast(gtk_widget_set_size_request_t)( dll.dlsym("gtk_widget_set_size_request") );
  auto gtk_dialog_run               = cast(gtk_dialog_run_t)( dll.dlsym("gtk_dialog_run") );
  auto gtk_message_dialog_new       = cast(gtk_message_dialog_new_t)( dll.dlsym("gtk_message_dialog_new") );
  auto gtk_widget_destroy           = cast(gtk_widget_destroy_t)( dll.dlsym("gtk_widget_destroy") );
  auto gtk_get_major_version        = cast(gtk_get_major_version_t)( dll.dlsym("gtk_get_major_version") );
  assert(gtk_message_dialog_new != null);

  if( gtk_get_major_version == null)
  {
    printf(" [TRACE] Running Gtk - 2.X \n");
  } else {
    printf(" [TRACE] Running Gtk - 3.X \n");
  }

  //------------------------------------//
  //      Initialize GTK                //
  //------------------------------------//

  int n = cast(int) args.length;
  auto argarr = args.map!( a => (a.dup ~ '\0' ).ptr ).array;
  auto argv = argarr.ptr;
  // This function must always be called before 
  // calling any other GTK function/procedure. 
  gtk_init_check(&n, &argv);

  //------------------------------------//
  //      Build GTK Widgets             //
  //------------------------------------//

  auto window = gtk_window_new( GtkWindowType.GTK_WINDOW_TOPLEVEL );
  assert( window != null );
  gtk_widget_set_size_request(window, 399, 500);
  gtk_window_set_title(window, "My Gtk Window from Dlang (D)".toStringz );
  auto button = gtk_button_new_with_label("Click me now");
  assert( button != null );
  gtk_container_add(window, button);
  gtk_widget_show_all( window );

  //--------------------------------------//
  //      Set up event handlers           //
  //--------------------------------------//

  // Call gtk_main_quit() function when the user closes the window. 
  g_signal_connect_data(  window
                        ,"destroy".toStringz
                        , gtk_main_quit
                        , null
                        , null
                        , cast(GConnectFlags) 0
                        );

  ButtonState state;
  state.counter                = 100;
  state.gtk_message_dialog_new = gtk_message_dialog_new;              
  state.gtk_widget_destroy     = gtk_widget_destroy;
  state.gtk_window_set_title   = gtk_window_set_title;   
  state.gtk_dialog_run         = gtk_dialog_run;
  gtk_widget_destroy           = gtk_widget_destroy; 

  g_signal_connect_data( button
                       , "clicked".toStringz
                       , &buttonCallback
                       , &state  
                       , null  
                       , cast(GConnectFlags) 0 
                       );

  printf(" [TRACE] Connect button event ");

  //--------------------------------------//
  //        Run Event Loop                //
  //--------------------------------------//
  // Blocks current thread and run GTK event loop.
  gtk_main(); 

  return 0;
}

    // ------------------------------------------ //

alias gchar    = char;
alias gshort   = short;
alias glong    = long;
alias gint     = int;
alias gboolean = gint;
alias guchar   = ushort; // unsigned short 
alias gulong   = ulong;  // unsigned long 
alias guint    = uint;
alias gfloat   = float;
alias gdouble  = double;
alias gpointer = void*;          
alias gconstpointer = const void*;

alias GClosure = void*;
alias GtkWidget = void;
alias GtkWindow = void;
alias GtkDialog = void;

enum GtkWindowType {
   GTK_WINDOW_TOPLEVEL = 0 
  ,GTK_WINDOW_POPUP    = 1
}

enum GConnectFlags {
    G_CONNECT_AFTER   = 1
  , G_CONNECT_SWAPPED = 2 
}

enum GtkMessageType {
    GTK_MESSAGE_INFO
  , GTK_MESSAGE_WARNING
  , GTK_MESSAGE_QUESTION
  , GTK_MESSAGE_ERROR
  , GTK_MESSAGE_OTHER
}

enum  GtkDialogFlags {
      GTK_DIALOG_MODAL               = 1 << 0
    , GTK_DIALOG_DESTROY_WITH_PARENT = 1 << 1
    , GTK_DIALOG_USE_HEADER_BAR      = 1 << 2
}

enum  GtkButtonsType
{
    , GTK_BUTTONS_NONE
    , GTK_BUTTONS_OK
    , GTK_BUTTONS_CLOSE
    , GTK_BUTTONS_CANCEL
    , GTK_BUTTONS_YES_NO
    , GTK_BUTTONS_OK_CANCEL
}


alias gtk_init_check_t = extern(C) gboolean function (int* argc, char*** argv);

alias gtk_dialog_run_t       = extern(C) gint function (void* dialog);
alias gtk_widget_destroy_t   = extern(C) void function (GtkWidget* widget);
alias gtk_main_quit_t        = extern(C) void function ();

alias gtk_window_set_title_t = extern(C) void function (GtkWidget* window, const gchar* title);

// alias GCallback      = extern(C)  void function ();
alias GCallback = extern(C) void function(GtkWidget* widget, gpointer data);

alias GClosureNotify = extern(C)  void function (gpointer data, GClosure* closure);


alias gtk_window_new_t = extern(C) GtkWidget* function (int);  

alias gtk_widget_show_t = extern(C) void function(GtkWidget*);

alias gtk_widget_set_size_t =  extern(C) void function (GtkWidget* widget, gint width, gint height);

alias gtk_widget_show_all_t = extern(C) void function (GtkWidget* widget);

alias gtk_main_t =  extern(C) void function();

alias gtk_button_new_with_label_t = extern(C) GtkWidget* function(const char* label);

alias gtk_container_add_t =  extern(C) void function(GtkWidget* container, GtkWidget* widget);


alias gtk_widget_set_size_request_t = extern(C) void function(GtkWidget* widget, gint width, gint height); 


alias gtk_get_major_version_t = extern(C) void function ();

alias g_signal_connect_data_t =  extern(C) gulong function ( 
                                              gpointer        instance
                                            , const gchar*    detailed_signal
                                            , void*           c_handler 
                                           // , GCallback       c_handler
                                            , gpointer        data
                                            , GClosureNotify  destroy_data
                                            , GConnectFlags   connect_flags
                                            );


alias gtk_message_dialog_new_t = extern(C) GtkWidget* function ( 
                                      GtkWindow       *parent,
                                      GtkDialogFlags  flags,
                                      GtkMessageType  type,
                                      GtkButtonsType  buttons,
                                      const gchar*    message_format
                                    );

Building: 

$ dmd -g gtk_dlopen.d

Building with docker (run the compiler without installing it): 

$  docker run --rm --interactive --tty --volume="$PWD":/pwd --workdir=/pwd dlangchina/dlang-dmd dmd -g gtk_dlopen.d

Running: 

$ ./gtk_dlopen

gtk_dlopen_dlang.png

Figure 4: Gtk Gui built using Dlang

1.18.5 Loading Gtk2/Gtk3 using Nim language

The following code creates a sample Gtk2 or Gtk3 graphics interface by dynamic loading Gtk symbols using dynlib Nim module, a dlopen()/dlclose() cross-platform wrapper, without linking against neither Gtk2 nor Gtk3 shared libraries at compile-time.

Reading:

File: gtk_dlopen.nim 

import std/dynlib  
import os

type
   gchar         = cchar    # char 
   gshort        = cshort   # short 
   glong         = clong    # long 
   gint          = cint     # int 
   gboolean      = gint     
   guchar        = cuchar   # unsigned char   
   gushort       = cushort  # unsigned short  
   gulong        = culong   # unsigned long   
   guint         = cint     # unsigned int    
   gfloat        = cfloat   
   gdouble       = cdouble  
   gpointer      = pointer  # void*  (Opaque pointer)
   gconstpointer = pointer  # const void*     

type 
   GtkWidget      = pointer  
   GConnectFlags  = cint     
   GtkDialogFlags = cint
   GtkMessageType = cint   
   GtkButtonsType = cint  

# Enumeration: GtkWindowType
const 

    GTK_WINDOW_TOPLEVEL = 0 
    GTK_WINDOW_POPUP    = 1 

    # Enumeration: GConnectFlags 
    G_CONNECT_AFTER   = 1
    G_CONNECT_SWAPPED = 2

# Enumeration: GtkMessageType
const 
    GTK_MESSAGE_INFO     = 0
    GTK_MESSAGE_WARNING  = 1
    GTK_MESSAGE_QUESTION = 2
    GTK_MESSAGE_ERROR    = 3
    GTK_MESSAGE_OTHER    = 4

# Enumeration: GtkDialogFlags  
const 
    GTK_DIALOG_MODAL               = 1
    GTK_DIALOG_DESTROY_WITH_PARENT = 2 
    GTK_DIALOG_USE_HEADER_BAR      = 4

# Enumeration: GtkButtonsType
const 
    GTK_BUTTONS_NONE      = 0 
    GTK_BUTTONS_OK        = 1 
    GTK_BUTTONS_CLOSE     = 2
    GTK_BUTTONS_CANCEL    = 3
    GTK_BUTTONS_YES_NO    = 4
    GTK_BUTTONS_OK_CANCEL = 5

  #-------------------------------------------------#
  #         GTK Function Pointers Declarations      #
  #-------------------------------------------------#

type 
    # using gtk_init_check_t = gboolean (*) (int* argc, char*** argv);
    # See: https://stackoverflow.com/questions/55646129/nim-argv-equivalent 
    gtk_init_check_t = proc (argc: ptr cint, argv: cstringArray): gboolean {.gcsafe, stdcall.} 

    # using gkt_window_new_t = GtkWidget* (*) (int);
    gtk_window_new_t = proc(param: cint): GtkWidget {.gcsafe, stdcall.}

    # void (*) (GtkWidget*)
    gtk_widget_show_t = proc(widget: GtkWidget): void {.gcsafe, stdcall.}

    # void (*) ()
    gtk_main_t = proc(): void {.gcsafe, stdcall.}

    # using gtk_window_set_title_t = void (*) (GtkWidget* window, const gchar* title);
    gtk_window_set_title_t = proc(window: GtkWidget, title: cstring): void {.gcsafe, stdcall.}

    # using gtk_widget_set_size_t = void (*) (GtkWidget*, gint, gint);
    gtk_widget_set_size_t = proc(widget: GtkWidget, width: gint, height: gint): void {.gcsafe, stdcall.}

    # void gtk_widget_show_all ( GtkWidget* widget)
    gtk_widget_show_all_t = proc (widget: GtkWidget): void {.gcsafe, stdcall.}

    # GtkWidget* gtk_button_new_with_label ( const gchar* label)
    gtk_button_new_with_label_t = proc(label: cstring): GtkWidget {.gcsafe, stdcall.}

    # void gtk_container_add ( GtkContainer* container, GtkWidget* widget)
    # See: https://docs.gtk.org/gtk3/method.Container.add.html 
    gtk_container_add_t = proc(container: GtkWidget, widget: GtkWidget ): void {.gcsafe, stdcall.}

    # void gtk_widget_set_size_request ( GtkWidget* widget, gint width, gint height)
    # See: https://docs.gtk.org/gtk3/method.Widget.set_size_request.html 
    gtk_widget_set_size_request_t = proc( widget: GtkWidget
                                        , width:  gint
                                        , height: gint)  {.gcsafe, stdcall.}

    # void (*) () 
    gtk_main_quit_t = proc(): void {.gcsafe, stdcall.}

    # void* (void pointer - opaque pointer)
    GClosure = pointer

    #  void  (*) (void);
    GCallback = proc(): void {.gcsafe, stdcall.}

    # void  (*) (gpointer data, GClosure* closure);
    GClosureNotify = proc(data: gpointer, closure: GClosure): void 

    ClickedCallback = proc(): void  {.gcsafe, stdcall.}

    g_signal_connect_data_t = proc( instance:        gpointer
                                  , detailed_signal: cstring
                                  , c_handler:       GCallback
                                  , data:            gpointer
                                  , destroy_data:    GClosureNotify
                                  , connect_flag:    GConnectFlags 
                                  ): gulong {.gcsafe, stdcall.}

    gtk_message_dialog_new_t = proc( parent:         GtkWidget
                                   , flags:          GtkDialogFlags
                                   , ttype:          GtkMessageType
                                   , buttons:        GtkButtonsType
                                   , message_format: cstring 
                                  ): GtkWidget {.gcsafe, stdcall.}

    # gint (*) (void* dialog);
    gtk_dialog_run_t = proc(dialog: GtkWidget): gint {.gcsafe, stdcall.}

    # using gtk_widget_destroy_t = void (*) (GtkWidget* widget);
    gtk_widget_destroy_t = proc(widget: GtkWidget): void  {.gcsafe, stdcall.}

    # guint gtk_get_major_version (void)
    gtk_get_major_version_t = proc(): void {.gcsafe, stdcall.}

  #---------------------------------------------------#
  # Load Gtk function GTK from shared library         #
  #                                                   #
  # GTK functions' type signature can be searched at: #
  #   -  https://docs.gtk.org/gtk3                    #
  #---------------------------------------------------#

proc loadSym(lib: dynlib.LibHandle, symbol: string): pointer =  
    echo " [TRACE] Loading symbol = " & symbol
    let p = lib.symAddr(symbol)
    assert p != nil, ("Cannot load symbol = " & symbol)
    return p


echo "\n\n ======== Program started Ok =============="

let dll = os.getEnv("GTK_LIB", "libgtk-3.so.0")

let lib = dynlib.loadLib(dll)
assert lib != nil, "Cannot load shared library"
echo " [TRACE] Shared library " & dll & " loaded Ok."

let gtk_init_check               = cast[gtk_init_check_t]( lib.loadSym("gtk_init_check") )
let gtk_window_new               = cast[gtk_window_new_t]( lib.loadSym("gtk_window_new") )
let gtk_widget_show              = cast[gtk_widget_show_t]( lib.loadSym("gtk_widget_show") )
let gtk_main                     = cast[gtk_main_t]( lib.loadSym("gtk_main") )
let gtk_window_set_title         = cast[gtk_window_set_title_t]( lib.loadSym("gtk_window_set_title") )
let gtk_widget_set_size          = cast[gtk_widget_set_size_t]( lib.loadSym("gtk_widget_set_size_request") )
let gtk_main_quit                = cast[gtk_main_quit_t]( lib.loadSym("gtk_main_quit") )
let g_signal_connect_data        = cast[g_signal_connect_data_t]( lib.loadSym("g_signal_connect_data") )
let gtk_widget_show_all          = cast[gtk_widget_show_all_t]( lib.loadSym("gtk_widget_show_all") )
let gtk_button_new_with_label    = cast[gtk_button_new_with_label_t]( lib.loadSym("gtk_button_new_with_label") )
let gtk_container_add            = cast[gtk_container_add_t]( lib.loadSym("gtk_container_add") )
let gtk_widget_set_size_request  = cast[gtk_widget_set_size_request_t]( lib.loadSym("gtk_widget_set_size_request") )
let gtk_dialog_run               = cast[gtk_dialog_run_t]( lib.loadSym("gtk_dialog_run") )
let gtk_message_dialog_new       = cast[gtk_message_dialog_new_t]( lib.loadSym("gtk_message_dialog_new") )
let gtk_widget_destroy           = cast[gtk_widget_destroy_t]( lib.loadSym("gtk_widget_destroy") )
let gtk_get_major_version        = cast[gtk_get_major_version_t]( lib.symAddr("gtk_get_major_version") )


if gtk_get_major_version == nil:
    echo " [TRACE] Running Gtk - 2.X"
else: 
    echo " [TRACE] Running Gtk - 3.X"

# gtk_main_quit() wrapper for logging
proc gtk_app_quit(): void  {.gcsafe, stdcall.}  = 
    echo " [TRACE] Quit GTK - Shutdown GTK event loop and unblocks current thread."
    gtk_main_quit()

proc on_clicked(button: GtkWidget, callback: ClickedCallback): void = 
    echo "I was clicked"
    discard g_signal_connect_data(button, "clicked", callback, nil, nil, cint 0)

proc msgbox_info(parent: GtkWidget, title: cstring, text: cstring): void  = 
    let dialog = gtk_message_dialog_new( parent, GTK_DIALOG_MODAL, GTK_MESSAGE_INFO
                                       , GTK_BUTTONS_OK, text)
    assert dialog != nil 
    gtk_window_set_title dialog, title
    discard gtk_dialog_run dialog 
    gtk_widget_destroy dialog 

# Start GTK widgets 
proc gtk_start(): void = 
    var nargv =  newSeq[string](paramCount())
    var x = 0
    while x < paramCount():
       nargv[x] = paramStr(x+1)  # first is program name
       x += 1
    var argv: cStringArray = nargv.allocCStringArray()
    var argc: cint = cint paramCount()
    # Call  gboolean gtk_init_check (int* argc, char*** argv);
    discard gtk_init_check(addr argc, argv)
    echo " [TRACE] Gtk started. Ok."


  #---------------------------------------------------#
  #             Create GTK GUI                        #
  #---------------------------------------------------#

echo " [TRACE] Shared library library loaded Ok."

# Encapsulates gtk_init_check(int &argc, char **argv)
# gtk_init_check must always be called before calling any other GTK 
# function, otherwise a runtime error will happen.
gtk_start()

let window: GtkWidget = gtk_window_new( GTK_WINDOW_TOPLEVEL )
assert window != nil 

# Alternative ways to write the same code in the next non-comment line: 
#  =>> gtk_widget_set_size( window, 400, 500)
#  =>> gtk_widget_set_size  window, 400, 500 
#  =>> window.gtk_widget_set_size( 400, 500 )
#  =>> window.gtk_widget_set_size 400, 500 
window.gtk_widget_set_size 400, 500

window.gtk_window_set_title "My GTK Windows called from Nim"


let button = gtk_button_new_with_label("Click me now!")
assert button != nil, "Failed to create button"
button.gtk_widget_set_size_request 70, 30
gtk_container_add(window, button)

# gtk_widget_show( window ) ->> UFS - Universal Function Call
# window.gtk_widget_show()
gtk_widget_show_all(window)

# Discard annotation discards the function return value 
discard g_signal_connect_data( window, "destroy", gtk_app_quit, nil, nil, cint 0)

var counter = 0

# Set button callback using a lambda syntax sugar 
# for better code asthetics and readability
button.on_clicked do:
    counter = counter + 1
    let message = " [INFO] Button was clicked " & $counter & " times. "
    echo message
    msgbox_info(nil, "Notification", message)

# Run GTK event loop and block current thread 
gtk_main()

echo " [TRACE] GTK application shutdown OK."

Screenshot

nim_dlopen.png

Figure 5: Gtk2/Gtk3 graphics interface created using NIM language

Building and Running

Building: 

$ nim c gtk_dlopen.nim

Running loading Gtk 3: 

 $ ./gtk_dlopen

 ======== Program started Ok ==============
 [TRACE] Shared library libgtk-3.so.0 loaded Ok.
 [TRACE] Loading symbol = gtk_init_check
 [TRACE] Loading symbol = gtk_window_new
 [TRACE] Loading symbol = gtk_widget_show
 [TRACE] Loading symbol = gtk_main
 [TRACE] Loading symbol = gtk_window_set_title
  ... ... ...    ... ... ...   ... ... ...   ... ... ... 
  ... ... ...   ... ... ...   ... ... ...   ... ... ... 
 [TRACE] Loading symbol = gtk_message_dialog_new
 [TRACE] Loading symbol = gtk_widget_destroy
 [TRACE] Running Gtk - 3.X
 [TRACE] Shared library library loaded Ok.
 [TRACE] Gtk started. Ok.
I was clicked
 [INFO] Button was clicked 1 times. 
 [INFO] Button was clicked 2 times. 
 [INFO] Button was clicked 3 times. 
 [INFO] Button was clicked 4 times. 
 [INFO] Button was clicked 5 times. 
 [INFO] Button was clicked 6 times. 
 [TRACE] Quit GTK - Shutdown GTK event loop and unblocks current thread.
 [TRACE] GTK application shutdown OK.

Running loading Gtk 2: 

 $ >> env GTK_LIB="libgtk-x11-2.0.so.0" ./gtk_dlopen

 ======== Program started Ok ==============
 [TRACE] Shared library libgtk-x11-2.0.so.0 loaded Ok.
 [TRACE] Loading symbol = gtk_init_check
 [TRACE] Loading symbol = gtk_window_new
 [TRACE] Loading symbol = gtk_widget_show
 [TRACE] Loading symbol = gtk_main
 [TRACE] Loading symbol = gtk_window_set_title
 [TRACE] Loading symbol = gtk_widget_set_size_request

 ....   ....  ....  ....  ....  ....  .... 
 ....   ....  ....  ....  ....  ....  .... 

 [TRACE] Running Gtk - 2.X
 [TRACE] Shared library library loaded Ok.
 [TRACE] Gtk started. Ok.
I was clicked
 [INFO] Button was clicked 1 times. 
 [INFO] Button was clicked 2 times. 

 ....   ....  ....  ....  ....  ....  .... 
 ....   ....  ....  ....  ....  ....  .... 

 [INFO] Button was clicked 15 times. 
 [TRACE] Quit GTK - Shutdown GTK event loop and unblocks current thread.
 [TRACE] GTK application shutdown OK.

1.19 Library calls hooking/interception with LD_PRELOAD

1.19.1 Overview

The dynamic linker of many Unix-like operating systems, including Linux-based ones and BSD variants, provide a LD_PRELOAD mechanism, which allows intercepting and overriding (aka hooking) library calls performed by applications withou any source code modification. A library call can be overriden by setting the LD_PRELOAD environment variable with a colon (':') separated list containing full paths of shared libraries containing symbols with the same name as the overriden functions.

Some use-cases of this feature are:

  • Override library calls and modify the behavior of library calls.
  • Performance analysis.
  • Instrument library calls.
  • Log library-calls
  • Trace memory allocation by overrading calloc, malloc and free functions.
  • Patch executables without any source code modification or recompilation.

Notes and limitations:

  • The dynamic linker LD_PRELOAD feature do not intercept kernel's system-calls, it can only intercept library calls (aka function calls) or GlibC (GNU C library) or any other CRT (C - Runtime Library) wrappers for system calls, informally called syscalls.
  • LD_PRELOAD trick cannot intercept functions of executables owned by root and with SETUID bit set.

1.19.2 Intercepting open() and fopen() library calls

All sources available at GIST:

File: build.sh (Build script) 

#!/usr/bin/env sh

# Build unix/Linux executable
gcc unix-executable.c -o unix-executable.bin

# Build Linux shared library: hook-files-a.so
gcc hook-files-a.c -o hook-files-a.so -fpic -shared -ldl -Wall

# Build Linux shared library: hook-files-b.so
g++ hook-files-b.cpp -o hook-files-b.so -fpic -Wall -shared -ldl

File: hook-files-a.c / C-version (Shared library, aka shared object, hook-files-a.so) 

#define _GNU_SOURCE // Enables RTLD_NEXT constant.

#include <stdio.h>
#include <assert.h>
#include <stdint.h>  

// ----- Unix-specific headers ------- //
#include <dlfcn.h>

// Annotation for internal symbols not visible outside
// this compilation unit (static storage class).
#define HIDDEN static 

// Anotation for calling function whenever the shared library
// is loaded by some process. Note: this is a specific GNU 
// language extension. 
#define ON_LIBRARY_LOADED __attribute__((constructor))

typedef uint32_t pid_t;
typedef uint32_t mode_t;

extern  pid_t getpid(void);

__attribute__((constructor)) static 
void init_library1()
{
  fprintf(stderr, " [INFO] init_library1() / Library loaded Ok. \n");
}

HIDDEN ON_LIBRARY_LOADED 
void init_library2()
{
  fprintf(stderr, " [INFO] init_library2() / Library loaded Ok. \n");
}

/** Intercepts (aka hooks) and overrides open() function 
 *  from libC (GLIBC) that encapsulates open() system-call 
 *  (aka syscall).
 *---------------------------------------------------------*/
int open(const char* pathname, int flags, mode_t mode)
{
   // Type alias for function pointer with same
   // signature as the open() function. 
   typedef int (* open_t) (  const char* pathname
                           , int         flags
                           , mode_t      mode);

   // Load old symbol of original C function that was overriden
   open_t open_real = (open_t) dlsym(RTLD_NEXT, "open");

   // Sanity checking => calls abort() if symbol is not found.
   assert( open_real );

   // Log intercepted functional call
   // pid =>> Process ID of process that loaded this
   // shared library. 
   fprintf( stderr
            , " [TRACE] pid = %d / open() ->> File = %s \n"
            , getpid(), pathname);

   // Pass control back to intercepted (hooked) function. 
   return open_real(pathname, flags, mode);
}


FILE* fopen64(const char* filename, const char* type)
{
      typedef FILE* (* fopen64_t) (  const char* filename
                                   , const char* type);

      fopen64_t old_fun = (fopen64_t) dlsym(RTLD_NEXT, "fopen64");
      assert( old_fun != NULL);

      fprintf( stderr
               , " [TRACE] pid = %d / fopen64() ->> File = %s \n"
               , getpid(), filename);     

      return old_fun(filename, type);
}

File: hook-files-b.cpp / C++-version (Shared library hook-files-b.so ) 

#include <iostream>
#include <cassert>
// Equivalent to C-header <stdint.h> std::uint32_t ... and so on.
#include <cstdint>  

// ----- Unix-specific headers ------- //

#include <dlfcn.h>

// Annotqtion for defining functions with C-linkage
// that must conform to C ABI and calling conventions.
#define EXTERN_C extern "C"

// Annotation for internal symbols not visible outside
// this compilation unit (static storage class).
#define HIDDEN static 

// Anotation for calling function whenever the shared library
// is loaded by some process. Note: this is a specific GNU 
// language extension. 
#define ON_LIBRARY_LOADED __attribute__((constructor))

// This symbol will be resolved at linking-time by the Linker 
extern "C" pid_t getpid(void);


__attribute__((constructor)) static 
void init_library1()
{
  fprintf( stderr," [INFO] init_library1() / Library loaded Ok. \n");
}

HIDDEN ON_LIBRARY_LOADED 
void init_library2()
{
  fprintf( stderr, " [INFO] init_library2() / Library loaded Ok. \n");
}

/** Intercepts (aka hooks) and overrides open() function 
 *  from libC (GLIBC) that encapsulates open() system-call 
 *  (aka syscall).
 *---------------------------------------------------------*/
EXTERN_C
int open(const char* pathname, int flags, mode_t mode)
{
   // Type alias for function pointer with same
   // signature as the open() function. 
   using open_t = int (*) (  const char* pathname
                           , int         flags
                           , mode_t      mode);

   // Load old symbol of original C function that was overriden
   open_t open_real = (open_t) dlsym(RTLD_NEXT, "open");

   // Sanity checking => calls abort() if symbol is not found.
   assert( open_real != nullptr);

   // Log intercepted functional call
   // pid =>> Process ID of process that loaded this
   // shared library. 
   std::fprintf( stderr
                , " [TRACE] pid = %d / open() ->> File = %s \n"
                , getpid(), pathname);

   // Pass control back to intercepted (hooked) function. 
   return open_real(pathname, flags, mode);
}

EXTERN_C
FILE* fopen64(const char* filename, const char* type)
{
    using fopen64_real = FILE* (*) ( const char* filename
                                   , const char* type);

    auto old_fun = (fopen64_real) dlsym(RTLD_NEXT, "fopen64");
    assert( old_fun != nullptr);

     std::fprintf( stderr
                , " [TRACE] pid = %d / fopen64() ->> File = %s \n"
                , getpid(), filename);     

    return old_fun(filename, type);
}

Building executable and shared libraries (aka shared objects - 'unix terminology'): 

$ >> git clone https://gist.github.com/751fad7bf7707949e824ed2c19a73752 api-hooking && cd api-hooking 
$ >> sh build.sh

$ ls
build.sh  hook-files-a.c  hook-files-b.cpp  unix-executable.c

# Check generated object-codes:
#------------------------------------------------------------------#
$ >> file hook-files-a.so 
hook-files-a.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV) ... ...

$ >> file hook-files-b.so
hook-files-b.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV) ... ...

$ >> file unix-executable.bin 
unix-executable.bin: ELF 64-bit LSB executable, x86-64, ... ... ...

Running executable unix-executable.bin 

 $ >> ./unix-executable.bin  
unix-executable.bin: unix-executable.c:8: main: Assertion `argc == 2 && "Missing file argument"' failed.
fish: “./unix-executable.bin” terminated by signal SIGABRT (Abort)

$ >> ./unix-executable.bin /proc/devices 
[LINE] 1 = Character devices:
 [LINE] 2 =   1 mem
 [LINE] 3 =   4 /dev/vc/0
 [LINE] 4 =   4 tty
... ... ... ... ... 
... ... ... ... ... 
 [LINE] 67 = 253 device-mapper
 [LINE] 68 = 254 mdp
 [LINE] 69 = 259 blkext

Intercepting library-call (ak function) fopen64() using LD_PRELOAD. 

# Using hook-files-a.so (C version)
#-------------------------------------------------
$ >> env LD_PRELOAD=$PWD/hook-files-a.so ./unix-executable.bin /proc/devices 
[INFO] init_library1() / Library loaded Ok. 
[INFO] init_library2() / Library loaded Ok. 
[TRACE] pid = 231690 / fopen64() ->> File = /proc/devices 
[LINE] 1 = Character devices:
 [LINE] 2 =   1 mem
 [LINE] 3 =   4 /dev/vc/0
 [LINE] 4 =   4 tty
 [LINE] 5 =   4 ttyS
 [LINE] 6 =   5 /dev/tty
 [LINE] 7 =   5 /dev/console
 ... ... ... ... 
... ... ... ... ... 

# Using hook-files-b.so (C++ version)
#-------------------------------------------------
$ >> env LD_PRELOAD=$PWD/hook-files-b.so ./unix-executable.bin /proc/devices
[INFO] init_library1() / Library loaded Ok. 
[INFO] init_library2() / Library loaded Ok. 
[TRACE] pid = 231790 / fopen64() ->> File = /proc/devices 
[LINE] 1 = Character devices:
 [LINE] 2 =   1 mem
 [LINE] 3 =   4 /dev/vc/0
 [LINE] 4 =   4 tty
 [LINE] 5 =   4 ttyS
 [LINE] 6 =   5 /dev/tty
 [LINE] 7 =   5 /dev/console
... ...  ... ...  ... ...  ... ...  ... ... 
... ...  ... ...  ... ...  ... ...  ... ...

Intercepting library-call (ak function) fopen64() using LD_PRELOAD, but discarding stdout (standard console output) by redirecting it to /dev/null in order to only display the stderr logged by the replacement functions. 

# C version 
$ >> env LD_PRELOAD=$PWD/hook-files-a.so ./unix-executable.bin /proc/devices > /dev/null
[INFO] init_library1() / Library loaded Ok. 
[INFO] init_library2() / Library loaded Ok. 
[TRACE] pid = 231957 / fopen64() ->> File = /proc/devices 

# C+++ version 
$ >> env LD_PRELOAD=$PWD/hook-files-b.so ./unix-executable.bin /proc/devices > /dev/null
[INFO] init_library1() / Library loaded Ok. 
[INFO] init_library2() / Library loaded Ok. 
[TRACE] pid = 231994 / fopen64() ->> File = /proc/devices

Tracing library calls with ltrace utility: 

 $ >> ltrace ./unix-executable.bin /proc/devices
fopen64("/proc/devices", "r")                         = 0x19722a0
getline(0x7fff02a9a640, 0x7fff02a9a638, 0x19722a0, 0x7fff02a9a638) = 19
fprintf(0x7f50e3769500, " [LINE] %d = %s ", 1, "Character devices:\n" [LINE] 1 = Character devices:
) = 32
getline(0x7fff02a9a640, 0x7fff02a9a638, 0x19722a0, 0x7fff02a9a638) = 8
fprintf(0x7f50e3769500, " [LINE] %d = %s ", 2, "  1 mem\n"  [LINE] 2 =   1 mem
) = 21
getline(0x7fff02a9a640, 0x7fff02a9a638, 0x19722a0, 0x7fff02a9a638) = 14
fprintf(0x7f50e3769500, " [LINE] %d = %s ", 3, "  4 /dev/vc/0\n"  [LINE] 3 =   4 /dev/vc/0
) = 27
getline(0x7fff02a9a640, 0x7fff02a9a638, 0x19722a0, 0x7fff02a9a638) = 8
fprintf(0x7f50e3769500, " [LINE] %d = %s ", 4, "  4 tty\n"  [LINE] 4 =   4 tty
  ... ... ...   ... ... ...   ... ... ...   ... ... ...   ... ... ...   ... ... ... 
  ... ... ...   ... ... ...   ... ... ...   ... ... ...   ... ... ...   ... ... ... 
getline(0x7fff02a9a640, 0x7fff02a9a638, 0x19722a0, 0x7fff02a9a638) = -1
free(0x1972480)                                       = <void>
fclose(0x19722a0)                                     = 0
 +++ exited (status 0) +++

Tracing system-calls calls with strace utility: 

$ >> strace ./unix-executable.bin /proc/devices

execve("./unix-executable.bin", ["./unix-executable.bin", "/proc/devices"], 0x7ffc6f5c0ba8 /* 83 vars */) = 0
 ... ... ... ...  ... ... ... ... ... ... ... ... 
 ... ... ... ...  ... ... ... ... ... ... ... ... 

openat(AT_FDCWD, "/proc/devices", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
read(3, "Character devices:\n  1 mem\n  4 /"..., 1024) = 694
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0xa), ...}) = 0
write(1, " [LINE] 1 = Character devices:\n", 31 [LINE] 1 = Character devices:
) = 31
write(1, "  [LINE] 2 =   1 mem\n", 21  [LINE] 2 =   1 mem
)  = 21
write(1, "  [LINE] 3 =   4 /dev/vc/0\n", 27  [LINE] 3 =   4 /dev/vc/0
) = 27
write(1, "  [LINE] 4 =   4 tty\n", 21  [LINE] 4 =   4 tty
)  = 21
 ... ... ...  ... ... ...  ... ... ...  ... ... ...  ... ... ... 
 ... ... ...  ... ... ...  ... ... ...  ... ... ...  ... ... ...

Determining files opened by uptime executable. 

 $ which uptime
 /usr/bin/uptime

 $ file $(which uptime)
 /usr/bin/uptime: ELF 64-bit LSB shared object, x86-64, ... ... 

# Indicates how long the machine is running: 
$ >> uptime 
15:45:45 up 3 days, 19:39,  1 user,  load average: 0.32, 0.59, 0.58

# Run 1 
$ >> env LD_PRELOAD=$PWD/hook-files-b.so uptime
[INFO] init_library1() / Library loaded Ok. 
[INFO] init_library2() / Library loaded Ok. 
[TRACE] pid = 232426 / open() ->> File = /proc/uptime 
[TRACE] pid = 232426 / open() ->> File = /proc/loadavg 
15:46:39 up 3 days, 19:40,  1 user,  load average: 0.55, 0.61, 0.59

# Run 2 - Discard stdout, preserving only stderr 
$ >> env LD_PRELOAD=$PWD/hook-files-b.so uptime > /dev/null
[INFO] init_library1() / Library loaded Ok. 
[INFO] init_library2() / Library loaded Ok. 
[TRACE] pid = 232456 / open() ->> File = /proc/uptime 
[TRACE] pid = 232456 / open() ->> File = /proc/loadavg 

$ >> cat /proc/uptime 
330114.72 1249234.16

$ >> cat /proc/loadavg 
0.40 0.55 0.57 1/1838 232504

Determining files opened by free executable which indicates the amount of free RAM memory in megabytes in the current machine: 

$ free -m
              total        used        free      shared  buff/cache   available
Mem:          15897        6603        2236        1768        7057        7378
Swap:             0           0           0

# Intercept library calls 
#--------------------------------------------------------------------
$ >> env LD_PRELOAD=$PWD/hook-files-b.so free -m
 [INFO] init_library1() / Library loaded Ok. 
 [INFO] init_library2() / Library loaded Ok. 
 [TRACE] pid = 232965 / open() ->> File = /proc/meminfo 
              total        used        free      shared  buff/cache   available
Mem:          15897        6601        2238        1768        7058        7380
Swap:             0           0           0

# Intercept library calls and discard stdout
# , keeping only stderr (standard error output)
#--------------------------------------------------------------------
 $ >> env LD_PRELOAD=$PWD/hook-files-b.so free -m > /dev/null 
 [INFO] init_library1() / Library loaded Ok. 
 [INFO] init_library2() / Library loaded Ok. 
 [TRACE] pid = 233016 / open() ->> File = /proc/meminfo

# Inspect file /proc/meminfo 
#--------------------------------------------------------------------
 $ >> cat /proc/meminfo 
MemTotal:       16279408 kB
MemFree:         2271428 kB
MemAvailable:    7552036 kB
Buffers:         1178620 kB
 ... ... ... ... ... ... 
 ... ... ... ... ... ... 
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      604380 kB
DirectMap2M:    12916736 kB
DirectMap1G:     4194304 kB

1.19.3 Intercepting dlopen() library calls

The subroutine dlopen() is used for loading shared libraries, plugins and native libraries, in the case of languages such as Python, Ruby, Java, TCL and so on. By intercepting library calls to dlopen(), it is possible check the shared libraries loaded by a given process.

File: dlopen_hook.cpp 

#include <iostream>
#include <cassert>
// Equivalent to C-header <stdint.h> std::uint32_t ... and so on.
#include <cstdint>  

// ----- Unix-specific headers ------- //

#include <dlfcn.h>

// Annotation for defining functions with C-linkage
// that must conform to C ABI and calling conventions.
#define EXTERN_C extern "C"

// This symbol will be resolved at linking-time by the Linker 
extern "C" pid_t getpid(void);

__attribute__((constructor)) static 
void init_library1()
{
    std::fprintf(stderr, " [TRACE] Library loaded on process =>> pid = %d \n\n", getpid());
}


/** Intercept dlopen() library call. 
 *
 ********************************************/
#if 1
EXTERN_C
void* dlopen(const char* filename, int flags)
{
   // Function type signature - must have the same type signature of
   // the intercepted function. 
   using fun_t = void* (*) (const char* filename, int flags);

      // Load old symbol of original C function that was overriden
   fun_t dlopen_real = (fun_t) dlsym(RTLD_NEXT, "dlopen");

   std::fprintf(  stderr, " [TRACE] dlopen() [PID = %d] - library = '%s'  Ok \n"
                            , getpid(), filename);

   return dlopen_real(filename, flags);
}
#endif

Build: 

 $ >> g++ dlopen_hook.cpp -o dlopen_hook.so -fpic -Wl,--soname=./dlopen_hook.so -shared -Wall -ldl     

 # Identify binary format 
 $ >> file dlopen_hook.so 
dlopen_hook.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked
, BuildID[sha1]=e31a1a84b50c2a4e8c72f4de484f8e245be59b67, not stripped

 #  View dependencies 
 $ >> ldd dlopen_hook.so 
        linux-vdso.so.1 (0x00007ffcb6da7000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f64bb201000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f64bb020000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f64bae2e000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f64bb227000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f64bacdf000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f64bacc4000)

Experiment 1: View shared libraries loaded by a Python process (python rpl - read-print-eval-loop). 

 $ >> env LD_PRELOAD=$PWD/dlopen_hook.so python3
 [TRACE] Library loaded on process =>> pid = 53633 

Python 3.8.5 (default, Jul 28 2020, 12:59:40) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
 [TRACE] dlopen() [PID = 53633] - library = '/usr/lib/python3.8/lib-dynload/readline.cpython-38-x86_64-linux-gnu.so'  Ok 
>>> 


>>> import os 
>>> 
>>> os.getpid()
53633

>>> import dbus
 [TRACE] dlopen() [PID = 53633] - library = '/usr/lib/python3/dist-packages/_dbus_bindings.cpython-38-x86_64-linux-gnu.so'  Ok 
>>> 

>>> import numpy
 [TRACE] dlopen() [PID = 53633] - library = '/home/h03f8/.local/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so'  Ok 
 [TRACE] dlopen() [PID = 53633] - library = '/home/h03f8/.local/lib/python3.8/site-packages/numpy/core/_multiarray_tests.cpython-38-x86_64-linux-gnu.so'  Ok 
 [TRACE] dlopen() [PID = 53633] - library = '/usr/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so'  Ok 
 [TRACE] dlopen() [PID = 53633] - library = '(null)'  Ok 
 [TRACE] dlopen() [PID = 53633] - library = '/home/h03f8/.local/lib/python3.8/site-packages/numpy/linalg/lapack_lite.cpython-38-x86_64-linux-gnu.so'  Ok 
 [TRACE] dlopen() [PID = 53633] - library = '/home/h03f8/.local/lib/python3.8/site-packages/numpy/linalg/_umath_linalg.cpython-38-x86_64-linux-gnu.so'  Ok 
 [TRACE] dlopen() [PID = 53633] - library = '/home/h03f8/.local/lib/python3.8/site-packages/numpy/fft/_pocketfft_internal.cpython-38-x86_64-linux-gnu.so'  Ok 
 [TRACE] dlopen() [PID = 53633] - library = '/home/h03f8/.local/lib/python3.8/site-packages/numpy/random/mtrand.cpython-38-x86_64-linux-gnu.so'  Ok 
 [TRACE] dlopen() [PID = 53633] - library = '/home/h03f8/.local/lib/python3.8/site-packages/numpy/random/bit_generator.cpython-38-x86_64-linux-gnu.so'  Ok 
 [TRACE] dlopen() [PID = 53633] - library = '/home/h03f8/.local/lib/python3.8/site-packages/numpy/random/_common.cpython-38-x86_64-linux-gnu.so'  Ok 
 [TRACE] dlopen() [PID = 53633] - library = '/home/h03f8/.local/lib/python3.8/site-packages/numpy/random/_bounded_integers.cpython-38-x86_64-linux-gnu.so'  Ok 
 [TRACE] dlopen() [PID = 53633] - library = '/home/h03f8/.local/lib/python3.8/site-packages/numpy/random/_mt19937.cpython-38-x86_64-linux-gnu.so'  Ok 
 [TRACE] dlopen() [PID = 53633] - library = '/home/h03f8/.local/lib/python3.8/site-packages/numpy/random/_philox.cpython-38-x86_64-linux-gnu.so'  Ok 
 [TRACE] dlopen() [PID = 53633] - library = '/home/h03f8/.local/lib/python3.8/site-packages/numpy/random/_pcg64.cpython-38-x86_64-linux-gnu.so'  Ok 
 [TRACE] dlopen() [PID = 53633] - library = '/home/h03f8/.local/lib/python3.8/site-packages/numpy/random/_sfc64.cpython-38-x86_64-linux-gnu.so'  Ok 
 [TRACE] dlopen() [PID = 53633] - library = '/home/h03f8/.local/lib/python3.8/site-packages/numpy/random/_generator.cpython-38-x86_64-linux-gnu.so'  Ok 
 [TRACE] dlopen() [PID = 53633] - library = '/usr/lib/python3.8/lib-dynload/_opcode.cpython-38-x86_64-linux-gnu.so'  Ok

Experiment 2: View shared libraries loaded by a Ruby process (Ruby repl irb). 

 $ >> env LD_PRELOAD=$PWD/dlopen_hook.so irb
 [TRACE] Library loaded on process =>> pid = 53705 

 [TRACE] dlopen() [PID = 53705] - library = '/usr/lib/x86_64-linux-gnu/ruby/2.7.0/enc/encdb.so'  Ok 
 [TRACE] dlopen() [PID = 53705] - library = '/usr/lib/x86_64-linux-gnu/ruby/2.7.0/enc/trans/transdb.so'  Ok 
 [TRACE] dlopen() [PID = 53705] - library = '/usr/lib/x86_64-linux-gnu/ruby/2.7.0/monitor.so'  Ok 
 [TRACE] dlopen() [PID = 53705] - library = '/usr/lib/x86_64-linux-gnu/ruby/2.7.0/ripper.so'  Ok 
 [TRACE] dlopen() [PID = 53705] - library = '/usr/lib/x86_64-linux-gnu/ruby/2.7.0/readline.so'  Ok 
 [TRACE] dlopen() [PID = 53705] - library = '/usr/lib/x86_64-linux-gnu/ruby/2.7.0/io/console.so'  Ok 
 [TRACE] dlopen() [PID = 53705] - library = '/usr/lib/x86_64-linux-gnu/ruby/2.7.0/pathname.so'  Ok 
 [TRACE] dlopen() [PID = 53705] - library = '/usr/lib/x86_64-linux-gnu/ruby/2.7.0/etc.so'  Ok 
irb(main):001:0>

Experiment 3: View shared libraries loaded by Kate editor. 

$ >> env LD_PRELOAD=$PWD/dlopen_hook.so kate
[TRACE] Library loaded on process =>> pid = 54041 

[TRACE] dlopen() [PID = 54041] - library = '/usr/lib/x86_64-linux-gnu/qt5/plugins/platforms/libqxcb.so.avx2'  Ok 
[TRACE] dlopen() [PID = 54041] - library = '/usr/lib/x86_64-linux-gnu/qt5/plugins/platforms/libqxcb.so'  Ok 
[TRACE] dlopen() [PID = 54041] - library = 'libXcursor.so.1'  Ok 
[TRACE] dlopen() [PID = 54041] - library = '/usr/lib/x86_64-linux-gnu/qt5/plugins/platformthemes/KDEPlasmaPlatformTheme.so.avx2'  Ok 
[TRACE] dlopen() [PID = 54041] - library = '/usr/lib/x86_64-linux-gnu/qt5/plugins/platformthemes/KDEPlasmaPlatformTheme.so'  Ok 

...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ... 
...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ... 

[TRACE] dlopen() [PID = 54041] - library = 'libGLX_mesa.so.0'  Ok 
[TRACE] dlopen() [PID = 54041] - library = 'libGLX_mesa.so.0'  Ok 
[TRACE] dlopen() [PID = 54041] - library = '/usr/lib/x86_64-linux-gnu/dri/tls/iris_dri.so'  Ok 
[TRACE] dlopen() [PID = 54041] - library = '/usr/lib/x86_64-linux-gnu/dri/iris_dri.so'  Ok 
[TRACE] dlopen() [PID = 54041] - library = '/usr/lib/x86_64-linux-gnu/qt5/plugins/konsolepart.so.avx2'  Ok 
[TRACE] dlopen() [PID = 54041] - library = '/usr/lib/x86_64-linux-gnu/qt5/plugins/konsolepart.so'  Ok

1.20 Launching processes with exec and fork system-call wrappers

This code demonstrates the usage of exec and fork system-calls wrapper functions, namely, execvp() and fork() functions.

Relevant functions signature

Fork

  • Linux Manpage: "fork() creates a new process by duplicating the calling process. The new process is referred to as the child process. The calling process is referred to as the parent process."
  • Note: The function 'fork' is not the system call fork, but a C wrapper around this sytem call provided by the C runtime library. (GLIBC - GNU C library in the machine where this was tested.)
pid_t fork(void);

Execvp (wraps exec system call)

  • The exec system calls is always called when a new process is launched/created. It replaces the process image of the current process with a new one from the launched executable.
  • Note: In addition to native executables, on Unix-like operating systems, an executable can also be any scripting file starting with shebang as first line such as "#!/usr/bin/env sh"
int execvp(const char *file, char *const argv[]);

Waitpid => Waits for a state of change of a process.

  • It can be used for waiting for process termination.
pid_t waitpid(pid_t pid, int *wstatus, int options);

Documentation

Further Reading

Sample Code

GIST:

File: unix-process.cpp 

// ------ File: unix-process.cpp -----------------------------------------------------------//
// Description: Shows hows to launch processes on Unix with Exec and Fork syscall wrappers.
//-------------------------------------------------------------------------------------------//

#include <iostream>
#include <vector>

// ----- Unix/Linux headers -----// 
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/wait.h>

#include <cstring> // Import: char* strerror(in errnum);

// -------------- Declarations --------------------------//

void print_errno_details(int err);
void execvp_test();
void execvp_cpp(std::string app, std::vector<std::string> args);

// Launch a new process without terminate this process. 
// It combines fork + exec system-calls. 
void fork_exec(std::string app, std::vector<std::string> args);

// -------------  MAIN() ------------------------------//

int main(int argc, char** argv)
{
     std::puts(" [INFO] Program started. ");

     if(argc < 2){
        std::printf(" Usage:  ./unix-process <OPTION>");
        return EXIT_SUCCESS;
     }

     std::string opt = argv[1];

     if(opt == "0")
     {
        execvp_test();
     }
     // Test execvp 
     if(opt == "1")
     {
         execvp_cpp("thunar", { "/etc" } );
         return EXIT_SUCCESS;
     }
     // Fork-exec 
     if(opt == "2")
     {
         fork_exec("konsole", { "-e", "tmux", "a"} );
     }  

     std::puts(" [TRACE] Finish execution. ");
     return 0;
}

// ------------- Definitions ------------------------//

void print_errno_details(int err)
{
        std::fprintf(stderr ,"\n   =>    errno(int) = %d" 
                             "\n   => errno message = %s \n"
                            , err, strerror(err));
        std::fflush(stderr);
}


// Test exec system-call wrapper function (execvp)
void execvp_test()
{    
        const char* app    = "thunar";

        const char* args[] = {  app     // Process name, can be anything 
                               , "/etc"  // Arguments passed to the process 
                               , nullptr // Always terminate the argument array with null pointer 
                              };

        // Encapsulates execv system call. 
        // int execvp(const char *file, char *const argv[]);
        if( execvp(app, (char *const *) args) == -1)
        {
              std::fprintf(stderr, " Error: unable to launch process");
              print_errno_details(errno);
              throw std::runtime_error("Error: failed to launch process");
        }

}

// C++ wrapper for the exevp() library-call 
// It replaces the current process image with a new one 
// from other executable. 
void execvp_cpp(  std::string app
                , std::vector<std::string> args)
{
     std::vector<const char*> pargs;
     pargs.reserve(args.size() + 1);
     pargs.push_back(app.c_str());
     for(auto const& a: args){ pargs.push_back(a.c_str()); }
     pargs.push_back(nullptr);

     // Signature: int execvp(const char *file, char *const argv[]);

     // execvp(app.c_str(), execvp(app.c_str(), (char* const *) pargs.data() )
     int status = execvp(app.c_str(), (char* const*) pargs.data() );
     if( status == -1)
     {
          std::fprintf(stderr, " Error: unable to launch process");
          print_errno_details(errno);
          throw std::runtime_error("Error: failed to launch process");
     }
}

void fork_exec(std::string app, std::vector<std::string> args)
{
     std::printf(" [TRACE] <BEFORE FORK> PID of parent process = %d \n", getpid());

     // PID of child process (copy of this process)
     pid_t pid = fork();

     if(pid == -1)
     {
         std::fprintf(stderr, "Error: unable to launch process");
         print_errno_details(errno);
         throw std::runtime_error("Error: unable to launch process");
     }
     if(pid == 0){
          std::printf(" [TRACE] Running on child process => PID_CHILD = %d \n", getpid());  

          // Close file descriptors, in order to disconnect the process from the terminal.
          // This procedure allows the process to be launched as a daemon (aka service).
          close(STDOUT_FILENO);
          close(STDERR_FILENO);
          close(STDIN_FILENO );

          // Execvp system call, replace the image of this process with a new one
          // from an executable. 
          execvp_cpp(app, args);        
          return;
     }

     std::printf(" [TRACE] <AFTER FORK> PID of parent process = %d \n", getpid());

     // pid_t waitpid(pid_t pid, int *wstatus, int options);
     int status;

     std::printf(" [TRACE] Waiting for child process to finish. ");

     // Wait for child process termination.
     // From header: #include <sys/wait.h>
     if(waitpid(pid, &status, 0) == -1)
     {
         print_errno_details(errno);
         throw std::runtime_error("Error: cannot wait for child process");
     }

     std::printf(" [TRACE] Child process has been terminated Ok.");
    // -------- Parent process ----------------//
}

Building: 

$ >> g++ unix-process.cpp -o unix-process.bin -std=c++1z -Wall -Wextra -g

Running: 

~/t/unix-explore
$ >> ./unix-process.bin 0
[INFO] Program started. 

~/t/unix-explore
$ >> ./unix-process.bin 1
[INFO] Program started. 

~/t/unix-explore
$ >> ./unix-process.bin 2
[INFO] Program started. 
[TRACE] <BEFORE FORK> PID of parent process = 774929 
[TRACE] <AFTER FORK> PID of parent process = 774929 
[TRACE] Running on child process => PID_CHILD = 774930 
[TRACE] Waiting for child process to finish.  [TRACE] Child process has been terminated Ok. [TRACE] Finish execution.

1.21 Reading subprocess output via Popen()

The convenience function popen(), which is a combination of fork() + exec() and pipe() functions, allows reading a subprocess output. Note: this function is also available on Windows.

Header:

  • <stdio.h> in C
  • <cstdio> in C++

Documentation

Functions

Popen:

  • This function executes: 'sh -c <COMMAND>' via, fork() + exec() and pipe(), returning a file stream. Note: sh is a unix-shell.
FILE* popen(const char* command, const char* mode);

Pclose 

int  pclose(FILE* stream);

Sample Application

File: popen-test.cpp 

#include <iostream>
#include <string> 

// Provides; popen(), pclose()
#include <cstdio>

int main(int argc, char** argv)
{
     std::puts("\n === EXPERIMENT 1 - popen() using buffer ========\n");
     {
         // Equivalent to: fork() + exec() + pipe()
         FILE* fd = popen("ls /", "r");

         if(fd == nullptr){
            fprintf(stderr, "Error: failed to run command. Check ERRNO variable. \n");
            return EXIT_FAILURE;
         }

         constexpr size_t BUFFER_SIZE = 500;
         // Buffer initialized with null char 
         char buffer[BUFFER_SIZE] = {0};

         while( fgets(buffer, BUFFER_SIZE, fd) != nullptr )
         {
            fprintf(stdout, "%s", buffer);
         }

         pclose(fd);        
     }

     std::puts("\n === EXPERIMENT 2 - popen() reading line-by-line ========\n");
     {
         FILE* fd = popen("ps -ef", "r");
         char*   pline = nullptr; 
         size_t  size  = 0; 
         ssize_t nread = 0;
         int     count = 0;
         do {
            nread = getdelim(&pline, &size, '\n', fd);
            std::cout << "  [" << count++ << "]" << " => " << pline;
         } while( nread != -1 && count < 8);
         pclose(fd);
     }

     return EXIT_SUCCESS;
}

Building: 

$ g++ popen-test.cpp -o popen-test -std=c++1z -Wall -Wextra -g

Running: 

  $ ./popen-test 

   === EXPERIMENT 1 - popen() using buffer ========

  bin
  boot
  dev
  etc
  home
. .... ... ... ... 
  ... ... ... ... ...
  usr
  var

   === EXPERIMENT 2 - popen() reading line-by-line ========

    [0] => UID          PID    PPID  C STIME TTY          TIME CMD
    [1] => root           1       0  0 Jun08 ?        00:48:25 /usr/lib/systemd/systemd ... ... ...
    [2] => root           2       0  0 Jun08 ?        00:00:00 [kthreadd]
    [3] => root           3       2  0 Jun08 ?        00:00:00 [rcu_gp]
    [4] => root           4       2  0 Jun08 ?        00:00:00 [rcu_par_gp]
    [5] => root           6       2  0 Jun08 ?        00:00:00 [kworker/0:0H-events_highpri]
    [6] => root           9       2  0 Jun08 ?        00:00:00 [mm_percpu_wq]
    [7] => root          10       2  0 Jun08 ?        00:00:54 [ksoftirqd/0]

1.22 Unix Pipes - Inter Process Communication

Unix pipe is an inter-process communication facility which allows the output of a process to be the input of a another. It can be used for reading the output of a subprocess.

Documentation

Main Functions

Function pipe()

  • Encapsulates: pipe() system-call
  • Linux Manpage: "pipe() creates a pipe, a unidirectional data channel that can be used for interprocess communication. The array pipefd is used to return two file descriptors referring to the ends of the pipe. pipefd[0] refers to the read end of the pipe. pipefd[1] refers to the write end of the pipe."
  • pipefd[0] => File descriptor for reading the pipe.
  • pipefd[1] => File descriptor for writing to the pipe.
#include <unistd.h>

int pipe(int pipefd[2]);

Function dup2()

  • Encapsulates: dup2() system-call
  • Duplicate file descriptor, used for IO redirection.
  • dup2( fd1, STDOUT_FILENO ) => Example: redirect current process' stdout to file descriptor fd1.
int dup2(int oldfd, int newfd);

Sample Code

This code spawns a subprocess using fork() + exec() and uses pipes to read the subprocess output which are stdout and stderr of the "ls -l -a /".

File: pipe-test.cpp 

#include <iostream>
#include <string> 

// ---- Unix headers --------// 
#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>


int main(int argc, char** argv)
{   
    /** Application to be called or its absolute path */
    const char* app = "ls";

    // Arguments ('-l' '-a' '/') for ls -l -a /
    const char* args [] = { /* First argument - process name (can be anything)   */  
                             "ls" 
                            /* Arguments passed to application */
                            , "-l", "-a", "/"
                            /* Sentinel value - always nullptr */
                            , nullptr  
                         };

    constexpr size_t PIPE_READ = 0;
    constexpr size_t PIPE_WRITE = 1;

    int fd[2];

    // ----- Create pipe IPC channel --------//
    if( pipe(fd) == -1){
       std::fprintf(stderr, "Error: cannot create pipe \n");
       return EXIT_FAILURE;
    }

    // ----------- Fork() ----------------//

    pid_t cpid = fork();
    if(cpid == -1)
    {
         std::fprintf(stderr, "Error: unable to launch process \n");
         return EXIT_FAILURE;         
    }
    if(cpid == 0)
    {
        /** ------- BEGIN of child process ------ **/
        std::printf(" [TRACE] Child process => PID_CHILD = %d \n", getpid());       

        // Connect child process' stdin to read-end of the pipe.
        dup2( fd[PIPE_READ], STDIN_FILENO );

        // Redirect child process' stdout (STDOUT_FILENO) to the write-end of the pipe. 
        dup2( fd[PIPE_WRITE], STDOUT_FILENO );
        // Redirect child process' stderr to write-end of the pipe.
        dup2( fd[PIPE_WRITE], STDERR_FILENO );

        // Run process - exec() syscall => the new process image 
        // has the same PID as the child process. 
        execvp(app, (char *const *) args);
        return 0;          
        /** ------- END of child process ------ **/
     }

     // ---- Continue parent process ---------//

     std::printf(" [TRACE] <AFTER FORK> PID of parent process = %d \n", getpid());
     constexpr size_t BUFSIZE = 200;
     ssize_t nread = 0;
     char buffer[BUFSIZE];

     // Close unnused end of the pipe.
     close(fd[PIPE_WRITE]);

     // Write bytes read from the pipe (write-end), written by child process
     // , to the stdout.
     do {   
          nread = read( fd[PIPE_READ], &buffer, BUFSIZE);
          write(STDOUT_FILENO, buffer, nread);      
     } while ( nread > 0);

     std::printf(" [TRACE] Waiting for child process termination. \n");

     close(fd[PIPE_READ]);

     // Wait for child process termination 
     ::wait(nullptr);
     return EXIT_SUCCESS;

     } // -- End of main() --- //

Building: 

$ >> g++ pipe-test.cpp -o pipe-test.bin -std=c++1z -g -Wall -Wextra

Running: 

 $ >> ./pipe-test.bin 
 [TRACE] <AFTER FORK> PID of parent process = 1352693 
 [TRACE] Child process => PID_CHILD = 1352694 
total 72
dr-xr-xr-x.  18 root root  4096 May 17 21:36 .
dr-xr-xr-x.  18 root root  4096 May 17 21:36 ..
-rw-r--r--    1 root root     0 May 17 21:36 .autorelabel
lrwxrwxrwx.   1 root root     7 Jan 28 15:30 bin -> usr/bin
dr-xr-xr-x.   6 root root  4096 May 17 15:17 boot
drwxr-xr-x   21 root root  4200 Jun 28 19:22 dev
... ... ... ... ... ... ... ... ... ... 
... ... ... ... ... ... ... ... ... ... ... ... 
lrwxrwxrwx.   1 root root     8 Jan 28 15:30 sbin -> usr/sbin
drwxr-xr-x.   2 root root  4096 May 12 14:50 srv
dr-xr-xr-x   13 root root     0 Jun  8 11:58 sys
drwxrwxrwt   31 root root   680 Jun 30 04:02 tmp
drwxr-xr-x.  12 root root  4096 Apr 22 19:35 usr
drwxr-xr-x.  22 root root  4096 Apr 22 19:39 var
 [TRACE] Waiting for child process termination.

1.23 Monitor file system changes with Inotify API

1.23.1 Overview

Inotify is an exclusive Linux kernel API for file monitoring, which allows watching file system changes events such as file copy, moving, creation, renaming and deletion without any wasteful and inefficient polling. BSD-variants, such as MacOSX and FreeBSD use the kqueue API.

Inotify API limitations:

  • Cannot monitor network mounted file systems.
  • Unable to watch /proc (PROCFS) virtual file systems (VSF)
  • Cannot monitor sub-directories. It is necessary to add new Inotify watch instances for every sub-directory to be monitored.

Some use-cases of file system monitoring via Inotify API are:

  • File system automation
  • Building system automation
  • Automatic recompilation whenever a dependency changes
  • Reloading application if some configuration file changes.
  • Reloading of game asset if some file change.
  • Executing some action, command or process when a file system event happens.

Functions and Structures

Headers:

Functions:

  • inotify_init() => Returns a file descriptor (numerical value) for an inotify instance. On failure, returns -1 and sets the errno thread-local-storage global variable.
int inotify_init(void) 
int inotify_init1(int flags);
  • inotify_watch_add() => Add new file system watch.
    • Parameters:
      • fd => File descriptor of inotify instance, created with inotify_init function.
      • pathname => Path to directory or file what will be monitored.
      • mask => bitwise mask of flags that selects the types of events that will be monitored.
      • RETURN => Returns (-1) on failure and any other value when the operation is successful.
int inotify_add_watch(int fd, const char* pathname, uint32_t mask);
  • inotify_rm_watch() => Removes file system watch.
    • Parameters:
      • fd => Inotify file descritor created with inotify_init() function.
      • wd => Watch file descriptor created with inotify_watch_add.
int inotify_rm_watch(int fd, int wd);

Structure: 

struct inotify_event {
    int      wd;       /* Watch (watcher file descriptor) descriptor */
    uint32_t mask;     /* Mask of events */
    uint32_t cookie;   /* Unique cookie associating related
                          events (for rename(2)) */
    uint32_t len;      /* Size of name field => File or directory name. */
    char     name[];   /* Optional null-terminated name */
};

References

1.23.2 Sample code

All files available at:

Files

File: inotify-test.cpp 

#include <iostream>
#include <string> 
#include <cstring>  // strerror()
#include <memory>
#include <cassert>
#include <map>

// ---- Unix-specific headers ----------//
#include <unistd.h>
#include <limits.h>

// ---- Linux-specific headers --------//
#include <sys/inotify.h>

constexpr int FAILURE = -1;

void exit_on_failure(int fd, const char* error_message)
{
    if(fd != FAILURE){ return; }
    int err = errno;
    std::fprintf( stderr, " [ERROR] %s ; errno = %d ; errno-msg = %s \n"
                  , error_message, err, strerror(err));
    // Execute exit system call and terminate this process.
    exit(1);
}

int main(int argc, char** argv)
{

    // Create inotify instance, returning file descriptor 
    int fd_inotify = ::inotify_init();
    exit_on_failure(fd_inotify, "Cannot create Inotify instance");

    using WatcheFileDescriptor = int;
    using DirectoryPath       = std::string;
    auto watch_table          = std::map<WatcheFileDescriptor, DirectoryPath>{};

    if(argc < 2){
        std::fprintf(stderr, " [ERROR] Requires one or more directories to be watched. \n");
        return EXIT_FAILURE;
    }

    for(int i = 1; i < argc; i++)
    {
        const char* directory = argv[i];

        // Create watch instance, returning a file descriptor for the watcher 
        int fd_watch = inotify_add_watch(fd_inotify, directory, IN_ALL_EVENTS);
        // Error checking 
        std::string error_msg = std::string("Cannot create watcher for this directory: ") + directory;
        exit_on_failure(fd_watch, error_msg.c_str());
        watch_table[fd_watch] = directory; 

        std::fprintf( stdout, " [INFO] Watching directory => wd = %d ; directory = %s \n"
                    , fd_watch, directory);
    }

    // Maximum number of events 
    constexpr size_t MAX_EVENTS = 1024;
    // Size of data structure inotify_event in bytes 
    constexpr size_t EVENT_SIZE = sizeof(struct inotify_event);
    // Lenght of buffer for events storage in bytes 
    constexpr size_t BUFFER_LENGTH = MAX_EVENTS * ( EVENT_SIZE + NAME_MAX + 1 );
    std::uint8_t     buffer[BUFFER_LENGTH];

    // -------------- Inotify event loop ----------------//
    for(;;)
    {
        // std::fprintf(stderr, " [INFO] Waiting for Inotify event \n");        
        // nn_read => signed number of bytes read from the file descriptor
        //            It is (-1) in the case of errrors.
        ssize_t nn_read = read(fd_inotify, buffer, BUFFER_LENGTH);
        assert(nn_read != -1 && "Placeholder for future error handling.");        
        ssize_t offset = 0;

        // Iterate over all events 
        while( offset < nn_read )
        {
            auto event = reinterpret_cast<inotify_event*>(buffer + offset);        
            auto directory = watch_table[event->wd];

            // Create file
            if(event->mask & IN_CREATE &&  !(event->mask & IN_ISDIR) )
                std::fprintf(stdout
                            , " [INFO] CREATE file event => (file = %s ) ; (dir = %s) \n"
                            , event->name, directory.c_str());

            // Create directory event 
            if(event->mask & IN_CREATE && event->mask & IN_ISDIR )
                std::fprintf( stdout
                            , " [INFO] CREATE directory  event => (file = %s ) ; (dir = %s)  \n"
                            , event->name, directory.c_str());

            // Delete file or directory event. 
            if(event->mask & IN_DELETE)
                std::fprintf( stdout, " [INFO] DELETE file event  => (file = %s ) ; (dir = %s)  \n"
                            , event->name, directory.c_str());

            // Modify file 
            if( event->mask & IN_MODIFY  && !(event->mask & IN_ISDIR) )
                std::fprintf( stdout
                            , " [INFO] MODIFY file event   => (file = %s ) ; (dir = %s) \n"
                            , event->name, directory.c_str());

            // Event happens when file system is unmounted. 
            if(event->mask & IN_IGNORED)
                std::fprintf( stdout
                            , " [INFO] IGNRED file monitor removed or file system unmounted => dir = %s \n"
                            , directory.c_str());

            offset = offset + EVENT_SIZE + event->len;
        }

    } // --------- End of Inotify event loop() --------------//

    //free(event);
    std::fprintf(stdout, " [INFO] Terminate application gracefully \n\n");
    close(fd_inotify);
    return 0;
}

Building: 

$ git clone https://gist.github.com/6e3fc8a39a3a6089ad7002738e4d509a inotify-gist && cd inotify-gist 
$ g++ inotify-test.cpp -o inotify-test.bin -std=c++1z -Wall -Wextra -g

$ file inotify-test.bin 
inotify-test.bin: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), .... ...

Running:

Run the following command from terminal emulator 1: 

$ ./inotify-test.bin /tmp/testing  ~/Downloads 
 [INFO] Watching directory => wd = 1 ; directory = /tmp/testing 
 [INFO] Watching directory => wd = 2 ; directory = /home/unixuser/Downloads 

 [INFO] CREATE file event => (file = file1.txt ) ; (dir = /tmp/testing) 
 [INFO] CREATE file event => (file = file2.txt ) ; (dir = /tmp/testing) 
 [INFO] CREATE file event => (file = file3.txt ) ; (dir = /tmp/testing) 
 [INFO] CREATE file event => (file = file4.txt ) ; (dir = /tmp/testing) 
 [INFO] DELETE file event  => (file = file1.txt ) ; (dir = /tmp/testing)  
 [INFO] CREATE file event => (file = testing-files1.txt ) ; (dir = /home/unixuser/Downloads) 
 [INFO] CREATE file event => (file = testing-files2.txt ) ; (dir = /home/unixuser/Downloads) 
 [INFO] CREATE file event => (file = testing-files3.txt ) ; (dir = /home/unixuser/Downloads) 
 [INFO] CREATE file event => (file = testing-files4.txt ) ; (dir = /home/unixuser/Downloads) 
 [INFO] DELETE file event  => (file = file2.txt ) ; (dir = /tmp/testing)  
 [INFO] DELETE file event  => (file = file3.txt ) ; (dir = /tmp/testing)  
 [INFO] DELETE file event  => (file = file4.txt ) ; (dir = /tmp/testing)  
 [INFO] MODIFY file event   => (file = testing-files1.txt ) ; (dir = /home/unixuser/Downloads) 
 [INFO] MODIFY file event   => (file = testing-files2.txt ) ; (dir = /home/unixuser/Downloads) 
 [INFO] MODIFY file event   => (file = testing-files2.txt ) ; (dir = /home/unixuser/Downloads)

Manipulate the watched directories from terminal emulator 2: 

$ mkdir -p /tmp/testing & cd /tmp/testing 

$ >> touch file1.txt file2.txt file3.txt file4.txt

$ >> rm -rf file1.txt 

$ >> touch ~/Downloads/testing-files{1, 2, 3, 4}.txt 

$ >> rm -rf -v *.txt  
removed 'file2.txt'
removed 'file3.txt'
removed 'file4.txt'

$ >> echo "change file" >> ~/Downloads/testing-files1.txt 

$ >> echo "change file" >> ~/Downloads/testing-files2.txt

$ >> echo "change file" >> ~/Downloads/testing-files2.txt

1.24 mmap - Memory Mapping

1.24.1 Overview

File mapping is an operating system mechanism which maps a disk file to a process virtual memory. This operating system feature allows accessing the file as it was in the process memory.

Benefits:

  • Transparent access to file, allows accessing the file as it was an ordinary memory. (the file is accessible by pointer)
  • Good for processing large files which does not fit in the machine RAM memory.

Use-cases:

  • Read large files
  • IPC - Inter-Process Communication
  • Process complicated binary files
  • Modify complex binary files. The file can be modified by just changing a memory, in other words, modifying the contents of a memory address.
  • Implement JIT - Just-In-Time compiler and execute assembly code at runtime.

The mmap API is available in most Unix-like operating systems:

  • Linux
  • BSD-family: MacOSX, FreeBDS, NetBSD, OpenBSD
  • Solaris
  • QNX, AIX

Mmap API functions

File descriptors: 

#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

// Returns file descriptor of a disk-file 
int open(const char *pathname, int flags);

// Close file descriptor (release resource)
int close(int fd);

Mmap:

  • Doc: $ man mmap
  • Linux Manpage Description: "mmap() creates a new mapping in the virtual address space of the calling process. The starting address for the new mapping is specified in addr. The length argument speci‐ fies the length of the mapping (which must be greater than 0)"
  • Argument: prot (Memory Protection)
    • PROT_READ => Pages may be written (most used)
    • PROT_WRITE => Pages may be written (allows changing the file by writing to the mapped memory)
    • PROT_NONE => Pages may not be accessed
    • PROT_EXEC => Pages may be executed => Allows executing machine code (assembly) at runtime. Use case: JIT - Just-In-Time compiler
  • Argument: flags
    • MAP_SHARED => Multiple processes can share the same file mapping. This flag is used for IPC - Inter Process Communication.
    • MAP_PRIVATE => Only the current process can access the file mapping.
#include <sys/mman.h>

void* mmap(  void*   addr    // (often nullptr) Address that the file will be mapped 
           , size_t  length  // Length of file mapping (often the file size)
           , int     prot    // Flags for memory protection 
           , int     flags   // Flags for process access conttrol (private | shared)
           , int     fd      // File descriptor to be mapped into virtual memory
           , off_t   offset  // (often zero) Offset from the beginning of the file 
          );

munmap:

  • Doc: $ man munmap
  • Linux Manpage doc: "The munmap() system call deletes the mappings for the specified address range, and causes further references to addresses within the range to generate invalid memory ref‐ erences. The region is also automatically unmapped when the process is terminated. On the other hand, closing the file descriptor does not unmap the region."
  • Param addr: Address of memory mapping (value returned by mmap)
  • Param length: length of file mapping, often it is the file size.
// For mmap and munmap 
#include <sys/mman.h>     

int munmap( void *addr, size_t length);

Hyperlinks to manpage documentation of related-functions:

References and further reading

General:

MMAP - File Memory Mapping for other programming languages

1.24.2 Example (C) - mmap for reading Windows PE32 files

This code uses mmap API (memory mapped files) for reading metadata from PE (Portable Executable) - Windows native executable files or windows object-code.

GIST:

File: unix-mmap.c 

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h> 
#include <assert.h>

// --- Unix/Posix headers ---------//
#include <unistd.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>

#define IMAGE_SIZEOF_SHORT_NAME            8
#define IMAGE_NUMBEROF_DIRECTORY_ENTRIES  16

typedef int32_t  LONG;
typedef uint16_t WORD;
typedef uint32_t DWORD;
typedef uint8_t  BYTE;
typedef uint64_t ULONGLONG;;

// Source: <winttt.h>  => Original: typedef struct _IMAGE_DOS_HEADER
typedef struct {            
    WORD   e_magic;                     
    WORD   e_cblp;                   
    WORD   e_cp;                     
    WORD   e_crlc;                   
    WORD   e_cparhdr;                
    WORD   e_minalloc;               
    WORD   e_maxalloc;               
    WORD   e_ss;                     
    WORD   e_sp;                     
    WORD   e_csum;                   
    WORD   e_ip;                     
    WORD   e_cs;                     
    WORD   e_lfarlc;                 
    WORD   e_ovno;                   
    WORD   e_res[4];                 
    WORD   e_oemid;                  
    WORD   e_oeminfo;                
    WORD   e_res2[10];               
    //  Offset to the PE header from the beginning of the file. 
    LONG   e_lfanew;                    
  } IMAGE_DOS_HEADER;

 typedef struct {
    DWORD Signature; 
    WORD  Machine;
    WORD  NumberOfSections;
    DWORD TimeDateStamp;
    DWORD PointerToSymbolTable;
    DWORD NumberOfSymbols;
    WORD  SizeOfOptionalHeader;
    WORD  Characteristics;
 } PE_HEADER;


ssize_t get_file_size(int fd)
{
    struct stat file_stat; 
    if( fstat(fd, &file_stat) == -1 )
        return -1;
    return file_stat.st_size;
}

int main(int argc, char** argv)
{   
    // ----------- Validate arguments ----------------------------//
    if(argc < 2)
    {
        fprintf(stderr, "Usage: /unix-mmap <FILE> \n");
        return EXIT_FAILURE;
    }

    // ----------- Get file descriptor ----------------------------//

    // Get read-only file descriptor of file 
    int fd = open(argv[1], O_RDONLY);
    if(fd == -1){
        fprintf(stderr, " Error: unable to open file. check ERRNO variable \n");
        return EXIT_FAILURE;
    }

    ssize_t size = get_file_size(fd);
    if(size == -1){
        fprintf(stderr, " Error: unable to get file size \n");
        return EXIT_FAILURE;
    }

    // ----------- Map file in to process' virtual memory ---------------------------//

    void* fmap = mmap(    NULL        /* Often set to zero, aka nullpointer           */
                        , size        /* Size of file mapping in bytes                */
                        , PROT_READ   /* Open in read-only mode                       */
                        , MAP_PRIVATE /* Only this process can access this memory-map */
                        , fd          /* File descriptor of file to be memory mapped  */
                        , 0x00        /* Offset from the beggining of the file        */
                     );

    if( fmap == MAP_FAILED)
    {
        fprintf(stderr, " Error: memory mapped failed. check ERRNO \n");
        return EXIT_FAILURE;
    }

    // --------- Process file -----------------------------------------// 

    unsigned char* bmap = (unsigned char*) fmap; 

    // char file_signature [2] = { bmap[0], bmap[1], bmap[2], bmap[3], 0x00 };
    printf(" File signature = %c %c 0x%X 0x%X \n", bmap[0], bmap[1], bmap[2], bmap[3]);

    IMAGE_DOS_HEADER* dos_header = (IMAGE_DOS_HEADER*) fmap;
    PE_HEADER*        pe_header  = fmap + dos_header->e_lfanew;

    assert( pe_header->Signature == 0x4550 );

    printf("\n ========= [DOS HEADER] =================\n");
    printf("   MAGIC NUMBER = 0x%X (hex) \n", dos_header->e_magic);
    printf("   MAGIC NUMBER = %c %c  (str) \n", dos_header->e_magic & 0xFF, (dos_header->e_magic >> 8) & 0xFF);
    printf("       e_lfanew = 0x%X \n",      dos_header->e_lfanew);

    printf("\n ======== [PE - Header] ================\n");
    printf(" Note: if machine is 0x8664 => the processor is AMD-Intel x86-64 \n");
    printf("\n");
    printf("          Signature = 0x%X \n", pe_header->Signature);
    printf("            Machine = 0x%X \n", pe_header->Machine);
    printf(" Number of sections = %d \n",   pe_header->NumberOfSections);

    // ---------- Release Resource ------------------------------------// 

    // Unmap memory segment 
    munmap(fmap, size);
    // Release file-descriptor resource
    close(fd);

    return 0;
} // --- End of main() ----//

Build: 

$ gcc unix-mmap.c -o unix-mmap-c.bin -Wall -Wextra -g

Running: 

$ ./unix-mmap-c.bin notepad.exe 
 File signature = M Z 0x90 0x0 

 ========= [DOS HEADER] =================
   MAGIC NUMBER = 0x5A4D (hex) 
   MAGIC NUMBER = M Z  (str) 
       e_lfanew = 0xF8 

 ======== [PE - Header] ================
 Note: if machine is 0x8664 => the processor is AMD-Intel x86-64 

          Signature = 0x4550 
            Machine = 0x8664 
 Number of sections = 7

1.24.3 Example (C++) - mmap for reading Windows PE32 files

GIST:

File: unix-mmap.cpp 

#include <iostream>
#include <string> 
#include <fstream> 
#include <cstdint> 
#include <cassert>

// --- Unix/Posix headers ---------//
#include <unistd.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>

constexpr size_t IMAGE_SIZEOF_SHORT_NAME = 8;
constexpr size_t IMAGE_NUMBEROF_DIRECTORY_ENTRIES = 16;

// using LONG      = long;
using LONG      = std::int32_t;
using WORD      = std::uint16_t;  // unsigned short;
using DWORD     = std::uint32_t;  // unsigned long; 
using BYTE      = std::uint8_t;   //unsigned char;
using ULONGLONG = std::uint64_t;  // unsigned long long 

// Source: <winttt.h>  => Original: typedef struct _IMAGE_DOS_HEADER
struct IMAGE_DOS_HEADER {            
    WORD   e_magic;                     
    WORD   e_cblp;                   
    WORD   e_cp;                     
    WORD   e_crlc;                   
    WORD   e_cparhdr;                
    WORD   e_minalloc;               
    WORD   e_maxalloc;               
    WORD   e_ss;                     
    WORD   e_sp;                     
    WORD   e_csum;                   
    WORD   e_ip;                     
    WORD   e_cs;                     
    WORD   e_lfarlc;                 
    WORD   e_ovno;                   
    WORD   e_res[4];                 
    WORD   e_oemid;                  
    WORD   e_oeminfo;                
    WORD   e_res2[10];               
    //  Offset to the PE header from the beginning of the file. 
    LONG   e_lfanew;                    
  };

 struct PE_HEADER
 {
    DWORD Signature; 
    WORD  Machine;
    WORD  NumberOfSections;
    DWORD TimeDateStamp;
    DWORD PointerToSymbolTable;
    DWORD NumberOfSymbols;
    WORD  SizeOfOptionalHeader;
    WORD  Characteristics;
 };


ssize_t get_file_size(int fd)
{
    struct stat file_stat; 
    if( fstat(fd, &file_stat) == -1 )
        throw std::runtime_error("Error: unable to get file size");
    return file_stat.st_size;
}

int main(int argc, char** argv)
{   
    // ----------- Validate arguments ----------------------------//
    if(argc < 2)
    {
        std::fprintf(stderr, "Usage: /unix-mmap <FILE> \n");
        return EXIT_FAILURE;
    }

    // ----------- Get file descriptor ----------------------------//

    // Get read-only file descriptor of file 
    int fd = open(argv[1], O_RDONLY);
    if(fd == -1){
        std::fprintf(stderr, " Error: unable to open file. check ERRNO variable \n");
        return EXIT_FAILURE;
    }

    ssize_t size = get_file_size(fd);

    // ----------- Map file in to process' virtual memory ---------------------------//

    void* fmap = mmap(    nullptr     /* Often set to zero, aka nullpointer           */
                        , size        /* Size of file mapping in bytes                */
                        , PROT_READ   /* Open in read-only mode                       */
                        , MAP_PRIVATE /* Only this process can access this memory-map */
                        , fd          /* File descriptor of file to be memory mapped  */
                        , 0x00        /* Offset from the beggining of the file        */
                     );

    if( fmap == MAP_FAILED)
    {
        std::fprintf(stderr, " Error: memory mapped failed. check ERRNO \n");
        return EXIT_FAILURE;
    }

    // --------- Process file -----------------------------------------// 

    const auto bmap = reinterpret_cast<unsigned char*>(fmap); 

    // char file_signature [2] = { bmap[0], bmap[1], bmap[2], bmap[3], 0x00 };
    std::printf(" File signature = %c %c 0x%X 0x%X \n", bmap[0], bmap[1], bmap[2], bmap[3]);

    IMAGE_DOS_HEADER* dos_header = reinterpret_cast<IMAGE_DOS_HEADER*>(fmap);
    PE_HEADER*        pe_header  = reinterpret_cast<PE_HEADER*>( (uintptr_t) fmap + dos_header->e_lfanew);

    assert( pe_header->Signature == 0x4550 );

    std::printf("\n ========= [DOS HEADER] =================\n");
    std::printf("   MAGIC NUMBER = 0x%X (hex) \n", dos_header->e_magic);
    std::printf("   MAGIC NUMBER = %c %c  (str) \n", dos_header->e_magic & 0xFF, (dos_header->e_magic >> 8) & 0xFF);
    std::printf("       e_lfanew = 0x%X \n",      dos_header->e_lfanew);

    std::printf("\n ======== [PE - Header] ================\n");
    std::printf(" Note: if machine is 0x8664 => the processor is AMD-Intel x86-64 \n");
    std::printf("\n");
    std::printf("          Signature = 0x%X \n", pe_header->Signature);
    std::printf("            Machine = 0x%X \n", pe_header->Machine);
    std::printf(" Number of sections = %d \n",   pe_header->NumberOfSections);


    // ---------- Release Resource ------------------------------------// 

    // Unmap memory segment 
    munmap(fmap, size);
    // Release file-descriptor resource
    close(fd);

    return 0;
} // --- End of main() ----//

Building: 

$ clang++ unix-mmap.cpp -o unix-mmap.bin -std=c++1z -Wall -Wextra -g

Running: 

$ ./unix-mmap.bin ipconfig.exe 
 File signature = M Z 0x90 0x0 

 ========= [DOS HEADER] =================
   MAGIC NUMBER = 0x5A4D (hex) 
   MAGIC NUMBER = M Z  (str) 
       e_lfanew = 0xE8 

 ======== [PE - Header] ================
 Note: if machine is 0x8664 => the processor is AMD-Intel x86-64 

          Signature = 0x4550 
            Machine = 0x8664 
 Number of sections = 6 


$ ./unix-mmap.bin notepad.exe 
 File signature = M Z 0x90 0x0 

 ========= [DOS HEADER] =================
   MAGIC NUMBER = 0x5A4D (hex) 
   MAGIC NUMBER = M Z  (str) 
       e_lfanew = 0xF8 

 ======== [PE - Header] ================
 Note: if machine is 0x8664 => the processor is AMD-Intel x86-64 

          Signature = 0x4550 
            Machine = 0x8664 
 Number of sections = 7

1.24.4 Example (C++) - mmap for reading Windows PE32 with class

This code uses a class FileMapping for encapsulating Unix memory-mapped files which simplifies the usage of the mmap feature and makes the code cleaner and safer.

GIST:

File: unix-mmap-class.cpp 

#include <iostream>
#include <string> 
#include <fstream> 
#include <cstdint> 
#include <cassert>

// --- Unix/Posix headers ---------//
#include <unistd.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>

constexpr size_t IMAGE_SIZEOF_SHORT_NAME = 8;
constexpr size_t IMAGE_NUMBEROF_DIRECTORY_ENTRIES = 16;

// using LONG      = long;
using LONG      = std::int32_t;
using WORD      = std::uint16_t;  // unsigned short;
using DWORD     = std::uint32_t;  // unsigned long; 
using BYTE      = std::uint8_t;   //unsigned char;
using ULONGLONG = std::uint64_t;  // unsigned long long 

// Source: <winttt.h>  => Original: typedef struct _IMAGE_DOS_HEADER
struct IMAGE_DOS_HEADER {            
    WORD   e_magic;                     
    WORD   e_cblp;                   
    WORD   e_cp;                     
    WORD   e_crlc;                   
    WORD   e_cparhdr;                
    WORD   e_minalloc;               
    WORD   e_maxalloc;               
    WORD   e_ss;                     
    WORD   e_sp;                     
    WORD   e_csum;                   
    WORD   e_ip;                     
    WORD   e_cs;                     
    WORD   e_lfarlc;                 
    WORD   e_ovno;                   
    WORD   e_res[4];                 
    WORD   e_oemid;                  
    WORD   e_oeminfo;                
    WORD   e_res2[10];               
    //  Offset to the PE header from the beginning of the file. 
    LONG   e_lfanew;                    
  };

 struct PE_HEADER
 {
    DWORD Signature; 
    WORD  Machine;
    WORD  NumberOfSections;
    DWORD TimeDateStamp;
    DWORD PointerToSymbolTable;
    DWORD NumberOfSymbols;
    WORD  SizeOfOptionalHeader;
    WORD  Characteristics;
 };

/** Class for encapsulating memory mapped files*/
class FileMapping
{
    int   m_fd      = -1; 
    void* m_addr    = nullptr; 
    ssize_t m_size  = -1;

    ssize_t get_file_size(int fd)
    {
        struct stat file_stat; 
        if( fstat(fd, &file_stat) == -1 )
            throw std::runtime_error("Error: unable to get file size");
        return file_stat.st_size;
    }

public: 
    // Disable copy constructor 
    FileMapping(FileMapping const&) = delete;
    // Disable copy assignment operator 
    FileMapping& operator=(FileMapping const&) = delete;    

    /** 
     * @param file_path - File to be memory-mapped to current process. 
     */
    FileMapping(std::string file_path)
    {
        // Get read-only file descriptor of file 
        m_fd = open(file_path.c_str(), O_RDONLY);        
        if(m_fd == -1){
            throw std::runtime_error("Unable to open file");
        }      

        m_size = get_file_size(m_fd);
        if(m_size == -1)
            throw std::runtime_error("Unable to get file size");


        m_addr = ::mmap( nullptr     
                        , m_size       
                        , PROT_READ   
                        , MAP_PRIVATE 
                        , m_fd        
                        , 0x00        
                    );
        if( m_addr == MAP_FAILED)
        {
            throw std::runtime_error("Error: failed to map file to memory");
        }

    }

    ~FileMapping()
    {
        // Unmap memory segment 
        munmap(m_addr, m_size);
        // Release file-descriptor resource
        close(m_fd);

        m_fd = -1;
        m_addr = nullptr;
        m_size = -1;
    }

    /** @brief Returns pointer file mapping address. */
    void* addr() const { return m_addr; }  

    /** @brief  Get casted pointer to an offset of the file mapping address. */
    template<typename T>
    T addr_rel(std::ptrdiff_t offset) const { 
        return reinterpret_cast<T>( reinterpret_cast<uintptr_t>(m_addr) + offset);  
    }

}; // --- End of class FileMapping ---- //


int main(int argc, char** argv)
{   
    // ----------- Validate arguments ----------------------------//
    if(argc < 2)
    {
        std::fprintf(stderr, "Usage: /unix-mmap <FILE> \n");
        return EXIT_FAILURE;
    }

    FileMapping fmap(argv[1]);

    // --------- Process file -----------------------------------------// 

    const auto bmap = fmap.addr_rel<unsigned char*>(0x00);

    // char file_signature [2] = { bmap[0], bmap[1], bmap[2], bmap[3], 0x00 };
    std::printf(" File signature = %c %c 0x%X 0x%X \n", bmap[0], bmap[1], bmap[2], bmap[3]);

    IMAGE_DOS_HEADER* dos_header = fmap.addr_rel<IMAGE_DOS_HEADER*>(0x00);
    PE_HEADER*        pe_header  = fmap.addr_rel<PE_HEADER*>( dos_header->e_lfanew);

    assert( pe_header->Signature == 0x4550 );

    std::printf("\n ========= [DOS HEADER] =================\n");
    std::printf("   MAGIC NUMBER = 0x%X (hex) \n", dos_header->e_magic);
    std::printf("   MAGIC NUMBER = %c %c  (str) \n", dos_header->e_magic & 0xFF, (dos_header->e_magic >> 8) & 0xFF);
    std::printf("       e_lfanew = 0x%X \n",      dos_header->e_lfanew);

    std::printf("\n ======== [PE - Header] ================\n");
    std::printf(" Note: if machine is 0x8664 => the processor is AMD-Intel x86-64 \n");
    std::printf("\n");
    std::printf("          Signature = 0x%X \n", pe_header->Signature);
    std::printf("            Machine = 0x%X \n", pe_header->Machine);
    std::printf(" Number of sections = %d \n",   pe_header->NumberOfSections);

    return 0;
} // --- End of main() ----//

Building: 

$ g++ unix-mmap-class.cpp -o unix-mmap-class.bin -std=c++1z -g -Wall -Wextra

Running: 

$ ./unix-mmap-class.bin notepad.exe 
 File signature = M Z 0x90 0x0 

 ========= [DOS HEADER] =================
   MAGIC NUMBER = 0x5A4D (hex) 
   MAGIC NUMBER = M Z  (str) 
       e_lfanew = 0xF8 

 ======== [PE - Header] ================
 Note: if machine is 0x8664 => the processor is AMD-Intel x86-64 

          Signature = 0x4550 
            Machine = 0x8664 
 Number of sections = 7

1.24.5 Example - mmap - Running machine code at runtime

The mmap posix system call and the APIs mprotect can be used for allocating memory with executable permission which allows writing and executing machine code at runtime. This feature is widely used by JIT (Just-In-Time) compilers of interpreted languages such as Julia and Javascript Engine V8, from Chrome browser, for improving the runtime performance and reducing the interpretation overhead.

Additional notes related to: mmap, assembly and machine code.

  • Note 1: The function mmap is also available on other Unix-like operating systems such as MacOSx, BSD-variants (FreeBSD, OpenBSD, NetBSD, DragonFly BSD) and so on.
  • Note 2: The the assembly x64 ISA (x86-64 AMD or Intel) is the same on all operating system that supports this ISA (Instruction Set Architecture). However, an object-code (machine code) from one operating system will not be compatible with another different operating system due the difference between the object-code binary format (Linux uses ELF; MacOSX uses MachO; Windows uses PE32); system calls; calling conventions and system shared libraries.
  • Note 3: On Windows NT kernels the equivalent functions are: VirtualAlloc (equivalent to mmap) and VirtualProtect (mprotect).
  • Note 4: Most Unix system uses the System V calling convention for x86-64 bits while Windows NT 64 bits uses a different callng convention.

Sample Code

The following sample application mmap-loader.cpp loads machine code (compiled assembly) for two functions, at runtime. The machine codes were extract from compiled C application with objdump GNU binutils tool.

GIST (Sources files):

File: mmap-loader.cpp 

 // =======>>> Load and execute machine code at runtime with mmap <<<======
#include <iostream>
#include <string>
#include <fstream>
#include <sstream>
#include <cstring>
#include <cstdlib>
#include <vector>

#include <sys/mman.h>

using byte = uint8_t;

/** Load machine code into process virtual memory */
void* load_machine_code(byte* buffer, std::size_t size);
void* load_machine_code(std::vector<byte> const& data);

std::vector<byte>
hexstr_to_bytes(std::string const& hexcode);

template<typename T>
inline T load_mcode(std::vector<byte> const& data)
{
     return reinterpret_cast<T>( load_machine_code(data) );
}


int main(int argc, char** argv)
{
    std::puts("\n ====== Load add_numbers from machine code ================");
    // 'extern' means that the variable will be resolved by the Linker
    extern std::string add_numbers_str;

    // Type alias for function pointer
    using add_numbers_t = int (*) (int, int);

    auto add_numbers_code = hexstr_to_bytes(add_numbers_str);

    // Load machine code for function add_numbers() at runtime.
    auto add_numbers  = load_mcode<add_numbers_t>(add_numbers_code);

    std::cout << " [RESULT] add_numbers(20, 100) = " << add_numbers(20, 100) << '\n';
    std::cout << " [RESULT] add_numbers(206, 782) = " << add_numbers(206, 782) << '\n';


    std::puts("\n ======== Load factorial() from machine code =============");
    extern std::vector<byte> factorial_code;

    // Function pointer alias
    using factorial_t = int (*) (int);
    auto factorial_func = load_mcode<factorial_t>(factorial_code);

    std::cout << " [RESULT] factorial(5) = " << factorial_func(5) << '\n';
    std::cout << " [RESULT] factorial(6) = " << factorial_func(6)  << '\n';
    std::cout << " [RESULT] factorial(7) = " << factorial_func(7)  << '\n';

    std::fprintf(stdout, " [TRACE] Process terminated gracefully Ok. \n");
    return 0;
}

//===========================================================//
//          I M P L E M E N T A T I O N S                    //
//===========================================================//

void* load_machine_code(const byte* buffer, std::size_t size)
{
    // Create anonymous mapping (not backed by file), a memory that will
    // be later marked as executable.
    void* pmap = ::mmap(  nullptr
                        , size
                        , PROT_READ | PROT_WRITE
                        , MAP_PRIVATE | MAP_ANONYMOUS
                        , -1, 0 );

    if(pmap == MAP_FAILED){ throw std::runtime_error("Failed create memory map");  }
        // Copy machine code to memory map
    std::memcpy( pmap, buffer, size);

    // Mark memory as executable, but not writable.
    if( mprotect(pmap, size, PROT_READ | PROT_EXEC ) == -1 )
        throw std::runtime_error("Faile to change memory protection flags");

    return pmap;
}


std::vector<byte>
hexstr_to_bytes(std::string const& hexcode)
{
    std::vector<byte> bytes;
    bytes.reserve(hexcode.size());
    char* token = strtok((char*) hexcode.data(), "\\x");
    while( token != nullptr ){
        char byte = (char) strtol(token, nullptr, 16);
        // ss << byte;
        bytes.push_back(byte);
        token  = strtok(nullptr, "\\x");
    }
    return bytes;
}

void* load_machine_code(std::vector<byte> const& data)
{
    return load_machine_code(&data[0], data.size());
}

// int add_numners(int x, int y)
//---------------------------------------------------
// 0000000000401152 <add_numbers>:
//   401152:       55                      push   %rbp
//   401153:       48 89 e5                mov    %rsp,%rbp
//   401156:       48 89 f8                mov    %rdi,%rax
//   401159:       48 01 f0                add    %rsi,%rax
//   40115c:       5d                      pop    %rbp
//   40115d:       c3                      retq
//
// Machine encoded as ASCII string
std::string add_numbers_str = "\\x55\\x48\\x89\\xe5\\x48\\x89\\xf8\\x48\\x01\\xf0\\x5d\\xc3";

std::vector<byte> factorial_code = {
          0x55                                      // push   rbp
        , 0x48, 0x89, 0xe5                          // mov    rbp,rsp
        , 0x89, 0x7d, 0xec                          // mov    DWORD PTR [rbp-0x14],edi
        , 0xc7, 0x45, 0xfc, 0x01, 0x00, 0x00, 0x00  // mov    DWORD PTR [rbp-0x4],0x1
        , 0xc7, 0x45, 0xf8, 0x01, 0x00, 0x00, 0x00  // mov    DWORD PTR [rbp-0x8],0x1
        , 0xeb, 0x0e                                // jmp    40115b <factorial+0x25>
        , 0x8b, 0x45, 0xfc                          // mov    eax,DWORD PTR [rbp-0x4]
        , 0x0f, 0xaf, 0x45, 0xf8                    // imul   eax,DWORD PTR [rbp-0x8]
        , 0x89, 0x45, 0xfc                          // mov    DWORD PTR [rbp-0x4],eax
        , 0x83, 0x45, 0xf8, 0x01                    // add    DWORD PTR [rbp-0x8],0x1
        , 0x8b, 0x45, 0xf8                          // mov    eax,DWORD PTR [rbp-0x8]
        , 0x3b, 0x45, 0xec                          // cmp    eax,DWORD PTR [rbp-0x14]
        , 0x7c, 0xea                                // jl     40114d <factorial+0x17>
        , 0x8b, 0x45, 0xfc                          // mov    eax,DWORD PTR [rbp-0x4]
        , 0x5d                                      // pop    rbp
        , 0xc3                                      // ret 
};

Building and running on Linux

Building: 

$ >> g++ mmap-loader.cpp -o mmap-loader.bin -std=c++1z -Wall -Wextra -g

Running: 

$ >> ./mmap-loader.bin

====== Load add_numbers from machine code ================
[RESULT] add_numbers(20, 100) = 120
[RESULT] add_numbers(206, 782) = 988

======== Load factorial() from machine code =============
[RESULT] factorial(5) = 24
[RESULT] factorial(6) = 120
[RESULT] factorial(7) = 720
[TRACE] Process terminated gracefully Ok.

Building Running on MacOSX

Building: 

$ clang++ mmap-loader.cpp -o mmap-loader.bin -std=c++1z -Wall -Wextra -g


$ file mmap-loader.bin
mmap-loader.bin: Mach-O 64-bit executable x86_64

$ otool -L mmap-loader.bin
mmap-loader.bin:
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 902.1.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)

Running:

  • Note: It works on MacOSX because the Linux, MacOSX and other Unix-based sytems use the System V x86-64 calling convention and the machine code does not use any C library function.
$ ./mmap-loader.bin

====== Load add_numbers from machine code ================
[RESULT] add_numbers(20, 100) = 120
[RESULT] add_numbers(206, 782) = 988

======== Load factorial() from machine code =============
[RESULT] factorial(5) = 24
[RESULT] factorial(6) = 120
[RESULT] factorial(7) = 720
[TRACE] Process terminated gracefully Ok.

Running the application in GDB debugger on Linux

Load application in debugger: 

 $ >> gdb ./mmap-loader.bin --silent
Reading symbols from ./mmap-loader.bin...

Set breaking point: 

# Set breaking point 
(gdb) b main
Breakpoint 1 at 0x4023f6: file mmap-loader.cpp, line 30.

Start running: 

(gdb) r
Starting program: /home/user/asm-projects/asm/mmap-loader.bin 
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.31-2.fc32.x86_64

Breakpoint 1, main (argc=1, argv=0x7fffffffd348) at mmap-loader.cpp:30
30              std::puts("\n ====== Load add_numbers from machine code ================");
Missing separate debuginfos, use: dnf debuginfo-install libgcc-10.2.1-1.fc32.x86_64 libstdc++-10.2.1-1.fc32.x86_64

Execute multiple steps: 

(gdb) n

 ====== Load add_numbers from machine code ================
37              auto add_numbers_code = hexstr_to_bytes(add_numbers_str);
(gdb) 
40              auto add_numbers  = load_mcode<add_numbers_t>(add_numbers_code);

(gdb) n
42              std::cout << " [RESULT] add_numbers(20, 100) = " << add_numbers(20, 100) << '\n';
(gdb) 

(gdb) p add_numbers_code 
$1 = std::vector of length 12, capacity 48 = {85 'U', 72 'H', 137 '\211', 229 '\345', 72 'H', 137 '\211', 
  248 '\370', 72 'H', 1 '\001', 240 '\360', 93 ']', 195 '\303'}

(gdb) p add_numbers.size()
Attempt to extract a component of a value that is not a struct or union

(gdb) p add_numbers_code.size()
$2 = 12

(gdb) ptype add_numbers
type = int (*)(int, int)

(gdb) ptype add_numbers_code 
type = std::vector<unsigned char>

Call the function pointer add_numbers() 

(gdb) call add_numbers(20, 100)
$3 = 120

(gdb) call add_numbers(222, -100)
$4 = 122

(gdb) call add_numbers(222, -1000)
$5 = -778

Show memory contents as machine code at function pointer memory location: 

# Bytes at the function pointer memory location 
# (memory allocated with mmap)
(gdb) x/12b add_numbers
0x7ffff7ffb000: 0x55    0x48    0x89    0xe5    0x48    0x89    0xf8    0x48
0x7ffff7ffb008: 0x01    0xf0    0x5d    0xc3

# Show assembly x86-64 (64 bits) in AT&T syntax 
(gdb) x/6i add_numbers
   0x7ffff7ffb000:      push   %rbp
   0x7ffff7ffb001:      mov    %rsp,%rbp
   0x7ffff7ffb004:      mov    %rdi,%rax
   0x7ffff7ffb007:      add    %rsi,%rax
   0x7ffff7ffb00a:      pop    %rbp
   0x7ffff7ffb00b:      retq   
(gdb) 

(gdb) set disassembly-flavor intel

# Show assembly x86-64 (64 bits) in Intel syntax 
(gdb) x/6i add_numbers
   0x7ffff7ffb000:      push   rbp
   0x7ffff7ffb001:      mov    rbp,rsp
   0x7ffff7ffb004:      mov    rax,rdi
   0x7ffff7ffb007:      add    rax,rsi
   0x7ffff7ffb00a:      pop    rbp
   0x7ffff7ffb00b:      ret    
(gdb)

Continue execution until process termination: 

(gdb) n
 [RESULT] add_numbers(20, 100) = 120
43      std::cout << " [RESULT] add_numbers(206, 782) = " << add_numbers(206, 782) << '\n';
(gdb) c
Continuing.
 [RESULT] add_numbers(206, 782) = 988

 ======== Load factorial() from machine code =============
 [RESULT] factorial(5) = 24
 [RESULT] factorial(6) = 120
 [RESULT] factorial(7) = 720
 [TRACE] Process terminated gracefully Ok. 
[Inferior 1 (process 933348) exited normally]
(gdb)

Procedure for Extracting the sample machine code with Objdump

This procedure presents how the sample machine code was extracted with objdump tool.

File: functions.c 

#include <stdio.h>

int factorial(int n)
{
    int prod = 1; 
    for(int i = 1; i < n; i++){ 
        prod = prod * i;
    }
    return prod;
}

int main()
{
    printf(" factorial(5) = %d \n", factorial(5));
    printf(" factorial(6) = %d \n", factorial(6));
    return 0;
}

Build: 

$ >> gcc functions.c -o functions.bin -Wall -Wextra -O0

~/t/asm
$ >> ./functions.bin
factorial(5) = 24 
factorial(6) = 120

Extract assembly and machine code (encoded as hexadecimal bytes): 

 $ objdump functions.bin -M intel -d 

functions.bin:     file format elf64-x86-64


Disassembly of section .init:

0000000000401000 <_init>:
  401000:	f3 0f 1e fa          	endbr64 
  401004:	48 83 ec 08          	sub    rsp,0x8
  401008:	48 8b 05 e9 2f 00 00 	mov    rax,QWORD PTR [rip+0x2fe9]        # 403ff8 <__gmon_start__>

 ...  ...  ...  ...  ...  ...  ...  ...  ... 
 ...  ...  ...  ...  ...  ...  ...  ...  ... 

0000000000401126 <factorial>:
  401126:	55                   	push   rbp
  401127:	48 89 e5             	mov    rbp,rsp
  40112a:	89 7d ec             	mov    DWORD PTR [rbp-0x14],edi
  40112d:	c7 45 fc 01 00 00 00 	mov    DWORD PTR [rbp-0x4],0x1
  401134:	c7 45 f8 01 00 00 00 	mov    DWORD PTR [rbp-0x8],0x1
  40113b:	eb 0e                	jmp    40114b <factorial+0x25>
  40113d:	8b 45 fc             	mov    eax,DWORD PTR [rbp-0x4]
  401140:	0f af 45 f8          	imul   eax,DWORD PTR [rbp-0x8]
  401144:	89 45 fc             	mov    DWORD PTR [rbp-0x4],eax
  401147:	83 45 f8 01          	add    DWORD PTR [rbp-0x8],0x1
  40114b:	8b 45 f8             	mov    eax,DWORD PTR [rbp-0x8]
  40114e:	3b 45 ec             	cmp    eax,DWORD PTR [rbp-0x14]
  401151:	7c ea                	jl     40113d <factorial+0x17>
  401153:	8b 45 fc             	mov    eax,DWORD PTR [rbp-0x4]
  401156:	5d                   	pop    rbp
  401157:	c3                   	ret    

 .... ...  .... ...  .... ...  .... ...  .... ...  .... ... 
 .... ...  .... ...  .... ...  .... ...  .... ...  .... ...

1.25 POSIX Shared Memory

1.25.1 Overview

POSIX (Portable Operarting System Interface) shared memory is a IPC inter-process communication API which allows multiple processes to share the same physical memory segment and transfer data without any copying overhead, unlike pipes, message queues, TCP/IP sockets, UDP/IP sockets or unix domain sockets.

Unix Shared Memory APIs

Unix-based operating systems, including Linux, BSD variants and MacOSX have the following APIs for shared memory.

  • MMAP - Memory Mapped Files (mmap)
    • Functions: mmap(), ftruncate(), munmap()
  • System V shared memory API
    • Functions: shmget(), shmat(), ftok()
  • Posix Shared Memory shm (similar to mmap)
    • Functions: shm_open(), ftruncate(), mmap().

On Linux, the POSIX shared memory is implemented as /dev/shm/<SHARED-MEMORY-NAME> memory mapped file which is an ordinary file created in the /dev/shm pseudo file system, that only exists in the volatile RAM memory. Regardless of the shared memory API, they are prone to race-condition problems if two processes attempt to write to the memory simultaneously which can lead to data corruption and unpredictable behavior. As a result, locks, semaphores and other mutual-exclusion concurrency primitives are necessary to synchronize the access to the shared memory in order to avoid race conditions and data corruption.

Limitations and potential problems

The following limitation for objects allocation are noteworthy:

  • Objects allocated in the shared memory should be contiguous and have any internal indirection or any internal pointer, in other words, they should be POD types, plain-old-data.
  • It is not possible to allocate objects containing STL containers without special allocators.
  • It is also not possible to allocate polymorphic objects, objects implemeting some C++ interface class with virtual member functions.

Headers

  • #include <sys/mman.h>
  • #include <sys/types.h>
  • #include <sys/stat.h>
  • #include <fcntl.h>
  • #include <unistd.h>

Functions 

// Open or create a new shared memory segment 
int shm_open(const char* name, int oflag, mode_t mode);

// Remove shared memory segment 
int shm_unlink(const char* name);

// Truncate or resize file to specific lenght 
int ftruncate(int fd, off_t length);

 // Map a file to current process address space 
 void *mmap(  void*  addr    // Process virtual address where the file will be mapped 
            , size_t length  // Length in bytes of the file mapping 
            , int    prot    // Protection flags 
            , int    flags   // Read-Write type of access 
            , int    fd      // File descriptor to be mapped 
            , off_t  offset  // Offset from beginning of file to be mapped. 
          );

// Unmap file mapping from current 
int munmap(void* addr, size_t length);

Documentation and Further Reading

1.25.2 Sample Code

This sample code aims to experiment POSIX shared memory subsystem. This sample program has two functionalities, a server functionality, which allocates an object in the shared memory and a client functionality which loads and modifies the object allocated in the shared memory segment.

Sources available at:

File: shmem1.cpp 

#include <iostream>
#include <string>
#include <cassert>
#include <cstring>

// ----- Unix and Posix headers ---//
#include <unistd.h>
#include <sys/types.h> 

#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>

// Implementation of object allocated in Shared memory 
struct Object
{
    int    x = 0;
    double y = 26.2124214;
    char   dataset[400] = "Hello world";

    Object(){
        std::cout << " [TRACE] Object created Ok \n" << '\n';
    }

    void set_x(int x)      { this->x = x; }
    void set_y(double y)  { this->y = y; }

    void set_str(const char* value)  
    {
        mempcpy(dataset, value, 200);
    }

    void show() const  
    {
        std::cout << " [TRACE] Shared memory object =>> "
                  << " ; x = "    << x 
                  << " ; y = "    << y 
                  << " ; str =  " << dataset 
                  << '\n';
    }
};

constexpr int         FAILURE            = -1;
constexpr size_t      STORAGE_SIZE       = sizeof(Object);
constexpr const char* shared_memory_name = "/shared-data";

// Provides shared memory object.
void server_side(const char* shm_name);
// Consumes shared memory object.
void client_side(const char* shm_name);

int main(int argc, char** argv)
{
    std::cout << " [INFO] Process Unique Identifier PID = " << ::getpid() << '\n';
    std::cout << " [INFO] Storage size in bytes = " << STORAGE_SIZE << '\n';

    if(argc < 2){
        std::cerr << " [ERROR] Expected command argument. " << '\n';
        return EXIT_FAILURE;
    }
    auto command = std::string{argv[1]};

    if(command == "server" ){ server_side(shared_memory_name);  }
    if(command == "client" ){ client_side(shared_memory_name);  }

        return 0;
}

// ========= I M P L E M E N  T A T I O N S =================//

void server_side(const char* shm_name)
{

    // Remove shared memory segment if it already exists.
    ::shm_unlink(shm_name);

    // Attempt to create shared  memory segment
    // On Linux this segment corresponds to the file: '/dev/shm/<SHM-NAME>'
    int fd = shm_open(  shm_name
                      , O_CREAT | O_EXCL | O_RDWR
                      , S_IRUSR | S_IWUSR
                      );

    // assert => Placeholder for future error handling
    assert( fd != FAILURE && "Failed to create shared memory segment" );

    // Resize shared memory segment
    assert( ::ftruncate(fd, STORAGE_SIZE) != FAILURE );

    // Map shared memory segment into process address space
    void* pmap = ::mmap( nullptr                  // Most of the time set to nullptr 
                       , STORAGE_SIZE          // Size of memory mapping
                       , PROT_READ | PROT_WRITE   // Allows reading and writing operations
                       , MAP_SHARED               // This flag makes this segment visible by other processes.
                       , fd                       // File descriptor
                       , 0x00                     // Offset from beggining of file
                       );

    if(pmap == MAP_FAILED){ perror("mmap"); exit(EXIT_FAILURE);  }

    // Allocate object in shared memory segment
    // using placement 'new operator'
    Object* pObj = new (pmap) Object;
    assert( pObj != nullptr );

    std::cout << " [BEFORE] Before typing RETURN " << '\n';
    pObj->show();

    std::cout << "\n\n Type return to exit" << '\n';
    std::getchar();

    std::cout << " [BEFORE] After typing RETURN " << '\n';
    pObj->show();

    // Mmap cleanup procedure 
    assert( ::munmap(pmap, STORAGE_SIZE) != FAILURE );

    // Remove shared memory segment 
    ::shm_unlink(shm_name);
}

void client_side(const char* shm_name)
{
    // Attempt to create shared  memory segment
    int fd = ::shm_open( shm_name,  O_RDWR, 0);

    // assert => Placeholder for future error handling
    assert( fd != - 1 && "Failed to open shared memory segment" );

    // Map shared memory segment into process address space
    void* pmap = ::mmap( nullptr                  // Most of the time to NULL or nullptr 
                       , sizeof STORAGE_SIZE      // Size of memory mapping
                       , PROT_READ | PROT_WRITE   // Allows reading and writing operations
                       , MAP_SHARED               // This flag makes this segment visible by other processes.
                       , fd                       // File descriptor
                       , 0x00                     // Offset from beggining of file
                       );

    if(pmap == MAP_FAILED){ perror("mmap"); exit(EXIT_FAILURE);  }

    /* Load object from shared memory knowing only its interface 
     * (pointer to interface class), without knowing its exact 
     * implementation or type. 
     */
    Object* pObj = reinterpret_cast<Object*>(pmap);
    std::cout << " Previous object value " << '\n';
    pObj->show();

    int x; 
    double y; 
    std::string str;
    std::cout << "Enter x, y and str: ";

    std::cin >> x >> y >> str;
    pObj->set_x(x);
    pObj->set_y(y);
    pObj->set_str(str.c_str());
    pObj->show();

    // assert( ::munmap(pmap, STORAGE_SIZE) != FAILURE );
}

Building: 

$ >> g++ shmem1.cpp -o shmem1.bin -std=c++1z -g -O0 -Wall -Wextra -lrt

Run as a server in terminal 1: 

$ >> ./shmem1.bin server 
[INFO] Process Unique Identifier PID = 81749
[INFO] Storage size in bytes = 416
[TRACE] Object created Ok 

[BEFORE] Before typing RETURN 
[TRACE] Shared memory object =>>  ; x = 0 ; y = 26.2124 ; str =  Hello world


Type return to exit

[BEFORE] After typing RETURN 
[TRACE] Shared memory object =>>  ; x = 26234 ; y = 3.15156 ; str =  cpp-shared-memory-allocation-shared-object

Run as a client in terminal 2: 

 $ >> ./shmem1.bin client 
 [INFO] Process Unique Identifier PID = 81777
 [INFO] Storage size in bytes = 416
 Previous object value 
 [TRACE] Shared memory object =>>  ; x = 0 ; y = 26.2124 ; str =  Hello world
Enter x, y and str: 20 -90.25122 hello-world-cpp1z
 [TRACE] Shared memory object =>>  ; x = 20 ; y = -90.2512 ; str =  hello-world-cpp1z

 $ >> ./shmem1.bin client
 [INFO] Process Unique Identifier PID = 81811
 [INFO] Storage size in bytes = 416
 Previous object value 
 [TRACE] Shared memory object =>>  ; x = 20 ; y = -90.2512 ; str =  hello-world-cpp1z
Enter x, y and str: 26234 3.15156324 cpp-shared-memory-allocation-shared-object
 [TRACE] Shared memory object =>>  ; x = 26234 ; y = 3.15156 ; str =  cpp-shared-memory-allocation-shared-object

 $ >> ./shmem1.bin client
 [INFO] Process Unique Identifier PID = 81837
 [INFO] Storage size in bytes = 416
 Previous object value 
 [TRACE] Shared memory object =>>  ; x = 26234 ; y = 3.15156 ; str =  cpp-shared-memory-allocation-shared-object
Enter x, y and str: ^C⏎

While the server is running, the following command outputs can be observed: 

 $ >> ls /dev/shm/shared-data
/dev/shm/shared-data

 $ >> file /dev/shm/shared-data
/dev/shm/shared-data: data

 $ >> cat /dev/shm/shared-data 
zf=���f6        @cpp-shared-memory-allocation-shared-object1�y-allocation-s��⏎ 

# ---- Process mappings -------------------------// 

 $ >> pmap 81749
81749:   ./shmem1.bin server
0000000000400000      8K r---- shmem1.bin
0000000000402000      4K r-x-- shmem1.bin
0000000000403000      4K r---- shmem1.bin
0000000000404000      4K r---- shmem1.bin
 .... ...  .... ...  .... ...  .... ...  .... ... 
 .... ...  .... ...  .... ...  .... ... 
00007faddd82f000     32K r---- ld-2.31.so
00007faddd837000      4K rw-s- shared-data
00007faddd838000      4K r---- ld-2.31.so
00007faddd839000      4K rw--- ld-2.31.so
00007faddd83a000      4K rw---   [ anon ]
00007ffe9af95000    136K rw---   [ stack ]
00007ffe9afc0000     16K r----   [ anon ]
00007ffe9afc4000      8K r-x--   [ anon ]
ffffffffff600000      4K r-x--   [ anon ]
 total             5944K

Attempt to run application in client mode after the server process is terminated. 

 $ >> ./shmem1.bin client
 [INFO] Process Unique Identifier PID = 82176
 [INFO] Storage size in bytes = 416
shmem1.bin: shmem1.cpp:128: void client_side(const char*): Assertion `fd != - 1 && "Failed to open shared memory segment"' failed.
fish: “./shmem1.bin client” terminated by signal SIGABRT (Abort)

1.26 memfd_create syscall - Loading from memory

This sample code illustrates how the Linux system call memfd_create can be used for loading shared libraries and executable directly from memory without writing them to disk.

Documentation:

Download incbin.h for embedding files in the executable memfd_test.bin 

$ curl -O -L  "https://raw.githubusercontent.com/graphitemaster/incbin/8cefe46d5380bf5ae4b4d87832d811f6692aae44/incbin.h"

Files

File: memfd_test.cpp 

#include <iostream>
#include <string>

#include <cstring>
#include <cassert>

#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>
#include <linux/memfd.h>
#include <sys/mman.h>
#include <sys/syscall.h>
#include <dlfcn.h>

// Incbin will embed the file named 'binary' in this executable.
#define INCBIN_STYLE INCBIN_STYLE_SNAKE
#define INCBIN_PREFIX g_
#include "incbin.h"
INCBIN(asset, "executable.bin");
INCBIN(shlib, "shlib.so");

// #include <sys/memfd.h>

// Execute ELF64 binary from memory
void execv_mem(const char* file_name, const std::byte* buffer, size_t buffer_size)
{

    int fd = memfd_create(file_name, MFD_ALLOW_SEALING);

    assert( fd != - 1);
    // Write from memory to anonymous file
    write(fd, buffer, buffer_size );
    char path[100];
    memset(path, 0x00, 100);
    sprintf(path, "/proc/self/fd/%d", fd);

// close(fd);

    // - Process name can be anything -----------
    const char* process_name = "ghost";
    const char* args[] = { process_name, nullptr };
    execvp(path, (char* const*) args);
}

// Load shared library from memory without writing to disk!!
void* dlopen_mem(const char* file_name, const std::byte* buffer, size_t buffer_size)
{

    int fd = memfd_create(file_name, MFD_ALLOW_SEALING);

    assert( fd != - 1);
    // Write from memory to anonymous file
    write(fd, buffer, buffer_size );
    char path[100];
    memset(path, 0x00, 100);
    sprintf(path, "/proc/self/fd/%d", fd);


    void* handle = dlopen(path, RTLD_LAZY | RTLD_GLOBAL);
    return handle;
}


using tfunc = int (*) (const char* param);

int main(int argc, char** argv)
{
    // --------- Load library from memory ---------------//
    //
    void* hnd = dlopen_mem("dummyname", (std::byte*) g_shlib_data, g_shlib_size );
    assert(hnd != nullptr);

    tfunc exported_func = (tfunc) dlsym(hnd, "exported_function");
    assert(exported_func != nullptr);

    exported_func("Passed to memory loaded library Ok");

    // --------- Run executable from memory ------------------//
    //
    fprintf(stderr, " [TRACE] Executing ELF64 file from memory. Ok \n");
    execv_mem("anon-file", (std::byte*) g_asset_data, g_asset_size);
    // execvp(path, (char* const*) args);
}

File: app.cpp (Executable to be run from memory) 

#include <iostream>

int main()
{
    std::cout << " [*] Hello world / Hallo Welt! (Deustche) " << std::endl;
    return 0;
}

File: shlib.cpp (Shared library to be loaded from memory) 

#include <iostream>
#include <cstring>

#define EXTERN_C extern "C"

EXTERN_C
int exported_function(const char* param)
{
    fprintf(stderr, " [TRACE] (Library) User parameter = %s \n", param);
    return 1002 + strlen(param);
}

Building

Build the shared library: 

$ g++ shlib.cpp -o shlib.so -fPIC -shared -ldl -Wall -g

Build the executable: 

$ g++ app.cpp -o executable.bin

Build the loader: 

$ g++ memfd_test.cpp -o memfd_test.bin -ldl -std=c++1z -Wall -Wextra -g

Running

After the loader is built, delete shlib.so and executable.bin binary artifacts: 

$ rm shlib.so executable.bin

Finally, run the loader: 

$ ./memfd_test.bin
[TRACE] (Library) User parameter = Passed to memory loaded library Ok 
[TRACE] Executing ELF64 file from memory. Ok 
[*] Hello world / Hallo Welt! (Deustche)

1.27 Syslog - System Message Logging

1.27.1 Overview

Several Unix-like operating systems provide a system message logging, syslog system which allows applications, such as servers and daemons (a.k.a system services) to send logging messages reporting events, auditing and debugging information to a central location, a syslog server which can be in the same machine as the application or in a remote host.

The syslog system has a client-server architecture, comprised of two main parts, the client which is any application that sends the logging message and the server which receives and consolidates client messages via Unix Domain Sockets on local machine; UDP/IP or TCP/IP if the server is located in a remote machine.

Some motivations for using syslog are:

  • Logging consolidation in a central location - which improves discoverability and makes easier to debug, filter and audit applications that runs without any graphical interface.
  • Logging persistence
  • Standard logging API, implemented by many Unix-like operating systems such as GNU/Linux distributions; GNU/Linux with systemd; FreeBSD; OpenBSD and MacOSX.
  • Operating system agnostic. The syslog server or daemon only relies on TCP/UDP, TCP/IP or Unix-Domain sockets, which makes possible to implement a syslog system, even on non-Unix-like operating system.
  • Familiarity of system administrators.
  • Integration to system-d on GNU/Linux distributions.
  • Network infrastructure equipment => Many common network equipment provide facilities for sending auditing information to syslog servers.
  • Multiple ways of visualization. => Unlike printf-like statements, logging can be viewed and inspected from any terminal emulators, graphical applications and also from remote machines. Even for desktop applications, does not run in terminals, logging via syslog is a more convenient way for diagnosing and debugging the application control flow.

Potential use-cases:

  • Logging for auditing and debugging in daemon (services) applications that runs in background and do not have any graphical user interface. Some of those applications are http servers, ftp servers, ssh servers, authentication systems and so on.

Syslog C API

API Documentation:

Syslog Protocol Technical Standard:

Headers:

  • #include <syslog.h>
  • #include <stdarg.h>

Functions:

  • openlog() => Set logging configuration and attributes that affects calls to the logging functions.
    • ident => Unique tag name identifier which allows identifying and filtering the application that logging messages belong.
    • logopt => Logging options bitwise-or flags that speficy the logging behavior.
    • facility => Parameter the assigns thae default category that all log messages will be related to. Some possible values are: LOG_USER (default) - generic user-level messages ;LOG_AUTH - log messages are related to login and authentication via ssh, terminal and so on; LOG_CRON - Cron clock messages; LOG_FTP - ftp daemon; LOG_KERN for Kernel messages.
void
openlog(  const char* ident       // Application unique identifierr 
        , int         logopt      
        , int      facility    // Gr
       );
  • closelog() => Disable logging messages.
void closelog(void);
  • syslog() => Send logging messages to the syslog daemon.
    • priority => Flag that indicates the logging severity. It can be: LOG_INFO for informational messages; LOG_DEBUG for debugging messages; LOG_WARNING for warnings; LOG_ERR for errros and so on. See full documentation: syslog
    • message => Logging message.
void syslog(int priority, const  char* message, ...);

void vsyslog(int priority, const char* message, va_list args);

Further Reading

1.27.2 Usage example

File: syslog-test.cpp 

#include <iostream>
#include <cmath>

// ---- Unix-specific headers ---------//
#include <syslog.h>
#include <unistd.h> // sleep() 

int main(int argc, char** argv)
{

    // Unique identifier 
    const char* ident = "MYSERVER";

    int logopt =  // Include PID (process unique identifier number)
                  // in the log messages
                  LOG_PID    
                | LOG_ODELAY 
                // Print log messages to console (terminal) stderr 
                | LOG_PERROR 
                ;

    // Configure logger  
    openlog(ident, logopt, LOG_USER);

    syslog(LOG_INFO, " Logging has started Ok.");
    syslog(LOG_DEBUG, "Starting server main loop");

    for(int i = 0; i < 10; i++)
    {
        syslog(LOG_INFO, "Loop counter value = %d - sqrt(%d) = %f"
               , i, i, std::sqrt(i));
        // Sleep for 1 second 
        sleep(1);
    }

    syslog(LOG_WARNING, "Main loop terminated.");

    // Shutdown logging 
    closelog();
    return 0;
}

Building: 

$ g++ syslog-test.cpp -o syslog-test -Wall -Wextra -g -std=c++1z

Running: (Terminal 1) 

 $ >> build/test-syslog 
MYSERVER[1668452]:  Logging has started Ok.
MYSERVER[1668452]: Starting server main loop
MYSERVER[1668452]: Loop counter value = 0 - sqrt(0) = 0.000000
MYSERVER[1668452]: Loop counter value = 1 - sqrt(1) = 1.000000
MYSERVER[1668452]: Loop counter value = 2 - sqrt(2) = 1.414214
MYSERVER[1668452]: Loop counter value = 3 - sqrt(3) = 1.732051
MYSERVER[1668452]: Loop counter value = 4 - sqrt(4) = 2.000000
MYSERVER[1668452]: Loop counter value = 5 - sqrt(5) = 2.236068
MYSERVER[1668452]: Loop counter value = 6 - sqrt(6) = 2.449490
MYSERVER[1668452]: Loop counter value = 7 - sqrt(7) = 2.645751
MYSERVER[1668452]: Loop counter value = 8 - sqrt(8) = 2.828427
MYSERVER[1668452]: Loop counter value = 9 - sqrt(9) = 3.000000
MYSERVER[1668452]: Main loop terminated.

Monitoring via journalctl: (Terminal 2)

In a second terminal run the command '$ journalctl -f' that displays the newest logging entries. 

 $ >> journalctl -f
-- Logs begin at Mon 2020-05-11 22:09:13 -03. --
Nov 27 14:20:39 localhost.localdomain dnf[1666375]: Fedora 32 - x86_64                               21 kB/s | 3.3 kB     00:00
Nov 27 14:20:40 localhost.localdomain dnf[1666375]: RPM Fusion for Fedora 32 - Nonfree - NVIDIA Dri  23 kB/s |  11 kB     00:00
  ...   ...   ...   ...   ...   ... 
  ...   ...   ...   ...   ...   ... 

Nov 27 14:50:44 localhost.localdomain MYSERVER[1668452]:  Logging has started Ok.
Nov 27 14:50:44 localhost.localdomain MYSERVER[1668452]: Starting server main loop
Nov 27 14:50:44 localhost.localdomain MYSERVER[1668452]: Loop counter value = 0 - sqrt(0) = 0.000000
Nov 27 14:50:45 localhost.localdomain MYSERVER[1668452]: Loop counter value = 1 - sqrt(1) = 1.000000
Nov 27 14:50:46 localhost.localdomain MYSERVER[1668452]: Loop counter value = 2 - sqrt(2) = 1.414214
Nov 27 14:50:47 localhost.localdomain MYSERVER[1668452]: Loop counter value = 3 - sqrt(3) = 1.732051
Nov 27 14:50:48 localhost.localdomain MYSERVER[1668452]: Loop counter value = 4 - sqrt(4) = 2.000000
Nov 27 14:50:49 localhost.localdomain MYSERVER[1668452]: Loop counter value = 5 - sqrt(5) = 2.236068
Nov 27 14:50:50 localhost.localdomain MYSERVER[1668452]: Loop counter value = 6 - sqrt(6) = 2.449490
Nov 27 14:50:51 localhost.localdomain MYSERVER[1668452]: Loop counter value = 7 - sqrt(7) = 2.645751
Nov 27 14:50:52 localhost.localdomain MYSERVER[1668452]: Loop counter value = 8 - sqrt(8) = 2.828427
Nov 27 14:50:53 localhost.localdomain MYSERVER[1668452]: Loop counter value = 9 - sqrt(9) = 3.000000
Nov 27 14:50:54 localhost.localdomain MYSERVER[1668452]: Main loop terminated.

View only logging entries related to tag MYSERVER: (Approach 1) 

 $ >> journalctl -f -t MYSERVER
-- Logs begin at Mon 2020-05-11 22:09:13 -03. --
Nov 27 14:50:45 localhost.localdomain MYSERVER[1668452]: Loop counter value = 1 - sqrt(1) = 1.000000
Nov 27 14:50:46 localhost.localdomain MYSERVER[1668452]: Loop counter value = 2 - sqrt(2) = 1.414214
Nov 27 14:50:47 localhost.localdomain MYSERVER[1668452]: Loop counter value = 3 - sqrt(3) = 1.732051
Nov 27 14:50:48 localhost.localdomain MYSERVER[1668452]: Loop counter value = 4 - sqrt(4) = 2.000000
Nov 27 14:50:49 localhost.localdomain MYSERVER[1668452]: Loop counter value = 5 - sqrt(5) = 2.236068
Nov 27 14:50:50 localhost.localdomain MYSERVER[1668452]: Loop counter value = 6 - sqrt(6) = 2.449490
Nov 27 14:50:51 localhost.localdomain MYSERVER[1668452]: Loop counter value = 7 - sqrt(7) = 2.645751
Nov 27 14:50:52 localhost.localdomain MYSERVER[1668452]: Loop counter value = 8 - sqrt(8) = 2.828427
Nov 27 14:50:53 localhost.localdomain MYSERVER[1668452]: Loop counter value = 9 - sqrt(9) = 3.000000
Nov 27 14:50:54 localhost.localdomain MYSERVER[1668452]: Main loop terminated.

View only log entries related to tag MYSERVER: (Approach 2) 

 $ >> journalctl SYSLOG_IDENTIFIER=MYSERVER
-- Logs begin at Mon 2020-05-11 22:09:13 -03, end at Fri 2020-11-27 14:52:01 -03. --
Nov 27 14:50:44 localhost.localdomain MYSERVER[1668452]:  Logging has started Ok.
Nov 27 14:50:44 localhost.localdomain MYSERVER[1668452]: Starting server main loop
Nov 27 14:50:44 localhost.localdomain MYSERVER[1668452]: Loop counter value = 0 - sqrt(0) = 0.000000
Nov 27 14:50:45 localhost.localdomain MYSERVER[1668452]: Loop counter value = 1 - sqrt(1) = 1.000000
Nov 27 14:50:46 localhost.localdomain MYSERVER[1668452]: Loop counter value = 2 - sqrt(2) = 1.414214
Nov 27 14:50:47 localhost.localdomain MYSERVER[1668452]: Loop counter value = 3 - sqrt(3) = 1.732051
Nov 27 14:50:48 localhost.localdomain MYSERVER[1668452]: Loop counter value = 4 - sqrt(4) = 2.000000
Nov 27 14:50:49 localhost.localdomain MYSERVER[1668452]: Loop counter value = 5 - sqrt(5) = 2.236068
Nov 27 14:50:50 localhost.localdomain MYSERVER[1668452]: Loop counter value = 6 - sqrt(6) = 2.449490
Nov 27 14:50:51 localhost.localdomain MYSERVER[1668452]: Loop counter value = 7 - sqrt(7) = 2.645751
Nov 27 14:50:52 localhost.localdomain MYSERVER[1668452]: Loop counter value = 8 - sqrt(8) = 2.828427
Nov 27 14:50:53 localhost.localdomain MYSERVER[1668452]: Loop counter value = 9 - sqrt(9) = 3.000000
Nov 27 14:50:54 localhost.localdomain MYSERVER[1668452]: Main loop terminated.

View log messages in JSON format (machine readable): 

$ >> journalctl SYSLOG_IDENTIFIER=MYSERVER -o json-pretty
{
        "_MACHINE_ID" : "a1ac43b933e24659bba5edf2b9cec1e1",
        "_CMDLINE" : "build/test-syslog",
        "_UID" : "1000",
        "_HOSTNAME" : "localhost.localdomain",
        "_SYSTEMD_SLICE" : "user-1000.slice",
        "PRIORITY" : "6",
        "_COMM" : "test-syslog",
        "_EXE" : "/home/myunixuser/temp-projects/quickjs-project/build/test-syslog",
        "SYSLOG_FACILITY" : "1",
        "MESSAGE" : " Logging has started Ok.",
        "_SOURCE_REALTIME_TIMESTAMP" : "1606499444168590",
        "_GID" : "1000",
        "_SYSTEMD_CGROUP" : "/user.slice/user-1000.slice/session-6.scope",
        "_SYSTEMD_USER_SLICE" : "-.slice",
        "_AUDIT_LOGINUID" : "1000",
        "_SYSTEMD_SESSION" : "6",
        "__CURSOR" : "s=c5bc71b70e1e4196ad228c9854261d9a;i=a55c3;b=367eba6384324bf9a1b17ed43352d82e;m=218d3e06963;t=5b51a4cb397e2;x=ce1952db56e33f8b",
        "__REALTIME_TIMESTAMP" : "1606499444168674",
        "_PID" : "1668452",
        "_BOOT_ID" : "367eba6384324bf9a1b17ed43352d82e",
        "_TRANSPORT" : "syslog",
        "SYSLOG_IDENTIFIER" : "MYSERVER",
        "_AUDIT_SESSION" : "6",
        "__MONOTONIC_TIMESTAMP" : "2305657170275",
        "_SYSTEMD_UNIT" : "session-6.scope",
        "_CAP_EFFECTIVE" : "0",
        "SYSLOG_PID" : "1668452",
        "_SYSTEMD_OWNER_UID" : "1000",
        "SYSLOG_TIMESTAMP" : "Nov 27 14:50:44 ",
        "_SYSTEMD_INVOCATION_ID" : "b25ee3d518c54e5c805521823f22d60d"
}
{
        "_SYSTEMD_OWNER_UID" : "1000",
        "SYSLOG_IDENTIFIER" : "MYSERVER",
        "_SOURCE_REALTIME_TIMESTAMP" : "1606499444168718",
        "_PID" : "1668452",
        "_UID" : "1000",
        "_CMDLINE" : "build/test-syslog",
        "_GID" : "1000",
        "_AUDIT_SESSION" : "6",
        "MESSAGE" : "Starting server main loop",
        "__MONOTONIC_TIMESTAMP" : "2305657481811",
        "PRIORITY" : "7",
        "__CURSOR" : "s=c5bc71b70e1e4196ad228c9854261d9a;i=a55c4;b=367eba6384324bf9a1b17ed43352d82e;m=218d3e52a53;t=5b51a4cb858d3;x=f796bcc2cb08f71e",
        "_HOSTNAME" : "localhost.localdomain",
        "_SYSTEMD_INVOCATION_ID" : "b25ee3d518c54e5c805521823f22d60d",
        "_MACHINE_ID" : "a1ac43b933e24659bba5edf2b9cec1e1",
        "_BOOT_ID" : "367eba6384324bf9a1b17ed43352d82e",
        "_SYSTEMD_CGROUP" : "/user.slice/user-1000.slice/session-6.scope",
        "SYSLOG_FACILITY" : "1",
        "_SYSTEMD_USER_SLICE" : "-.slice",
        "_SYSTEMD_UNIT" : "session-6.scope",
        "_SYSTEMD_SLICE" : "user-1000.slice",
        "__REALTIME_TIMESTAMP" : "1606499444480211",
        "_TRANSPORT" : "syslog",
        "_EXE" : "/home/myunixuser/temp-projects/quickjs-project/build/test-syslog",
        "_COMM" : "test-syslog",
        "SYSLOG_PID" : "1668452",
        "_CAP_EFFECTIVE" : "0",
        "SYSLOG_TIMESTAMP" : "Nov 27 14:50:44 ",
        "_AUDIT_LOGINUID" : "1000",
        "_SYSTEMD_SESSION" : "6"
}
 ... ... ... ... 
 ... ... ...

See also:

1.28 Sockets - TCP/IP and Unix Domain Sockets

1.28.1 Overview

Sockets are inter process communication mechanism, intruduced by Unix BSD 4.1 circa 1982, which allows multiple processes running at different machines to communicate across TCP/IP network.

Main Implementations

  • BSD Sockets => Pervarsive on Unix-like operating systems: BSD, Linux, MacOSX, ….,
  • Winsocks => Microsft Windows only (Windows NT and Windows CE families)

Domain and Connection Type

A socket endpoint is defined by its domain and connection type.

Most common socket domains:

  • AF_INET => IPv4 - Internet Protocol [Most used]
  • AF_INET6 => IPv6 - Intenert Protocol (IPv6)
  • AF_UNIX => Unix Domain socket on local machine [Most used]

Most common connection Types:

  • SOCK_STREAM => TCP - Transmission Control Protocol
  • SOCK_DGRAM => UDP - User Datagram Protocol
  • SOCK_RAW => Raw network protocol (Requires root permission)

Socket server or socket client:

  • socket client
    • => Connects to socket server and always starts the connection. Example: curl; web browser (http client); netcat
  • socket server
    • => Listen some port and waits client connection. It can handle one or more clients. Example: Http server, nginx, ssh server.

TCP/IP connection tuple:

  • Hostname or IP address (IPv4 or IPv6)
    • 0.0.0.0 / address used by a socket server for listening all hosts.
    • 127.0.0.1 / Localhost - refers to the local (current) machine
    • 'localhost' / Localhost - refers to the local (current) machine, same as (127.0.0.1)
  • Port number: 16 bits number

Common TCP/IP Ports:

  • 20, 21 - FTP server
  • 22 - SSH server
  • 23 - Telnet (SSH predecessor)
  • 80 - HTTP server
  • 443 - HTTPS (HTTP + SSL/TSL) - Http with SSL encryption
  • 143 - IMAP - Mail
  • 25 - SMTP - Simple Mail Transfer Protocol
  • More at: List of TCP and UDP port numbers

Further Reading

1.28.2 BSD Socket API

The BSD socket API is the reference-implementation for network sockets inter-process communication mechanism, which powers the internet. The BSD socket API was initially implemented on BSD Unix and nowadays is used by the most Unix-based operating systems, namely, Linux, BSD-variants (FreeBSD, NetBSD, OpenBSD) and MacOSX. The Windows sockets (Winsocks) implementation is also based on BSD sockets API, although one of the main differences is that on the Winsocks API, sockets are not file descriptors and Unix read(),/write() calls or windows equivalent APIs cannot be used with sockets.

Headers:

  • #include <sys/types.h>
  • #include <sys/socket.h>
  • #include <netdb.h>

Functions for creating and disposing sockets:

  • socket()
    • Linux Manpage: "socket() creates an endpoint for communication and returns a file descriptor that refers to that endpoint. The file descriptor returned by a successful call will be the lowest-numbered file descriptor not currently open for the process."
    • Note: returns -1 when an error happens and sets the errno thread-local global variable.
int socket(int domain, int type, int protocol);

// --------- Example ----------------------------// 

// Usage for TCP/IPv4 
int sockfd = socket(AF_INET, SOCK_STREAM, 0); 

// Placeholder for future error handling 
assert( sockfd != -1 && "Unable to create socket." );

// Error handling: 
if(sockfd == -1){ 
    perror("Error: failed to create a socket connect. Check errno"); 
    exit(EXIT_FAILURE);
}

Usages for instantiating sockets: 

/************ TCP / IP ******************************/
//--------------------------------------------------//

// Usage for TCP/IPv4 
int sockfd = socket(AF_INET, SOCK_STREAM, 0); 

// Usage for TCP/IPv6 
int sockfd = socket(AF_INET6, SOCK_STREAM, 0);

/************* UDP / IP ******************************/
//--------------------------------------------------//

// Usage for UDP/IPv4 
int sockfd = socket(AF_INET, SOCK_DGRAM, 0); 

// Usage for UDP/IPv6 
int sockfd = socket(AF_INET6, SOCK_DGRAM, 0); 

/*********** SCTP / IP *****************************/
//--------------------------------------------------//

// Usage for SCTP / IPv4 - (SCTP - Stream Control Protocol)
int fd  = socket(PF_INET, SOCK_STREAM, IPPROTO_SCTP);

/***** Unix Domain Sockets (Only for local host) *****/
//--------------------------------------------------//

// Usage for Unix socket with TCP 
int sockfd = socket(AF_UNIX, SOCK_STREAM, 0);

// Usage for Unix socket with UDP 
int sockfd = socket(AF_UNIX, SOCK_DGRAM, 0);
  • close() => Close file descriptor or socket file descriptor.
int close(int fd);

Functions for setting a socket as client-socket:

  • connect() => Connect a client socket to a given address. On error, (-1) is returned. On success, 0 is returned.
int connect(int sockfd, const struct sockaddr *addr, socklen_t addrlen);

Functions for setting up a socket as a server socket (aka passive socket, listening socket)

  • bind() [SOCKET SERVER] => Bind a given port for listening income connections. (used for socket servers)
int bind(int sockfd, const struct sockaddr* addr, socklen_t addrlen);

 struct sockaddr {
     sa_family_t sa_family;
     char        sa_data[14];
 };
  • listen() [SOCKET SERVER]
    • Linux Manpage: "listen() marks the socket referred to by sockfd as a passive socket, that is, as a socket that will be used to accept incoming connection requests using accept(2)."
    • Linux Manpage: "The backlog argument defines the maximum length to which the queue of pending connections for sockfd may grow. If a connection request arrives when the queue is full, the client may receive an error with an indication of ECONNRE‐ FUSED or, if the underlying protocol supports retransmission, the request may be ignored so that a later reattempt at connection succeeds."
int listen(int sockfd, int backlog);
  • setsockopt() => Manipulate socket options.
    • Doc: $ man setsockopt
int setsockopt(int sockfd, int level, int optname, const void *optval, socklen_t optlen);

Functions for sending and receiving messages

Functions send, and recev works on all implementations of sockets BSD Sockets, used by Unix-based operating systems, and Windows sockets.

  • send() => Send a buffer to a socket.
    • Doc: $ man send
// Returns number of bytes received 
ssize_t send(  int         sockfd  // Socket file descriptor 
             , const void* buf     // Pointer to buffer (any POD - Plain Old Data)
             , size_t      len     // Buffer length (or size) in bytes 
             , int         flags   // Flags (bitwise OR flags)
            );
  • recv() => Receive a message from a socket. Read 'len' bytes from socket and store data in a buffer. Note: the buffer argument is a pointer to any POD (Plain-Old Data) type, contiguos allocated data without any internal pointer.
    • Doc: $ man recv
    • This function returns the number of bytes read which the maximum value is the buffer length.
      • The function returns (-1) on error and sets the errno global variable for indicating the failure.
      • The return value (0) indicates that the connection was closed either by the current calling code with close() function applied on the socket file descriptor or by the counterpart socket located in the remote host.
      • If a recev() call is made, and the counterpart socket did not send any data with send() or write(), the call to recev() blocks the current thread if the socket file descriptor is set in blocking mode (default).
      • A call to recev() may not receive all bytes available, even though the peer socket has send a message larger than the buffer size. This situation is called partial-read that must be handled by the calling code.
ssize_t recv(  int       sockfd   // Socket file descriptor 
              , void*    buffer   // Pointer to buffer (POD type) 
              , size_t   len      // Buffer length or size in bytes 
              , int      flags    // Flags (bitwise OR flags
             );

Functions read(), write() also works with sockets file descriptors. However this feature is not portable to windows sockets implementation.

  • read() => Similar as recev(), but it is only available with BSD-style. The usage of read() is not portable to Windows sockets.
ssize_t read(  int    fd       // File descriptor or socket file descriptor 
             , void*  buffer   // Pointer to buffer (any POD type)
             , size_t size     // Buffer size in bytes 
            );
  • write() => Similar to write(), but not portable to Windows sockets. It writes data from buffer to any type of file descriptor and returns the number of bytes sent to the peer socket or returns (-1) on failure setting errno global variable.
ssize_t write( int          fd      // File descriptor or socket file descriptor 
              , const void* buffer  // Pointer to buffer (any POD type)
              , size_t      size    // Buffer size in bytes 
              );

Endianess conversion for binary data exchange of numeric types

One of the pitfalls of binary data exchange is the endianess, the order which bytes of numeric types are placed in the memory. A code assuming that the server and client side have the same endianess may eventually fail and iterpret a numeric type in a incorrect way if the server or client runs in a machine (computer architecture) with a different endianess. In order to both socket communication endpoints interpret the data in a correct way, both have to agree about the data representation (_external representation_) which is the big-endian or network-byte-order. See also: (IBM endian-independent code)

The BSD socket API provides the following functions for converting between machine (host) and network-order endianess of numeric types. Note: those functions are not portable and are neither part of C standard nor part of POSIX standard.

Header:

  • #include <arpa/inet.h>

Functions that converts from host to network endianess:

  • htonl() => Conversion from host (local machine) to network-byte-order endianess. It takes a number in local machine byte-order and returns a number with network-byte-order (big endian) endianess. This function should be used whenever the calling code needs to write to the network via write() or send() calls.
    • Mneumonic: h-to-n-l (Host to network long)
  uint32_t htonl(uint32_t hostlong);

  // --------- Usage example ------------// 
  //                                     // 

  // ==> Number in host (local machine) endianess. 
  uint32_t message_size = 500; 
  // ==> Number in network-byte-order (big-endian)
  uint32_t data = htonl(message_size); 

 if( send(sockfd, &data, sizeof(uint32_t), 0) == -1 )
 { /* Handler error */ }

// OR: 

 if( write(sockfd, &data, sizeof(uint32_t)) == -1 )
 { /* Handler error */ }
  • htons() => Conversion from host to network-byte-order.
    • Mneumonic: h-to-n-s => (Host to network short)
uint16_t htons(uint16_t hostshort);

// --------- Usage Example -------------// 
// 

 // ==> Number in host (local machine) endianess. 
  uint16_t message_size = 500; 
  // ==> Number in network-byte-order (big-endian)
  uint16_t data = htonl(message_size); 

 if( send(sockfd, &data, sizeof(uint16_t), 0) == -1 )
 { /* Handler error */ }

Functions that converts from network to host endianess:

  • ntohl() => Takes a number in network-endianess as argument and returns a number in local-machine endiness. This funtion should be used whenever the calling code is reading data from the network via read() or recev() calls.
    • Mneumonic: n-to-h-l (Network to Host Long)
uint32_t ntohl(uint32_t netlong);

// ------ Usage example / Reading data from network  ----------// 
uint32_t buffer; 

// Buffer variable will receive the data network-byte-order 
// or big-endian.
if( recv(sockfd, &buffer,  sizeof(uint32_t), 0 ) == -1 )
{  /** Error handling ... here */  }

// OR: 
if( read(sockfd, &buffer,  sizeof(uint32_t) ) == -1 )
{  /** Error handling ... here */  }

// Number in host endiness (byte order)
uint32_t message_size = nothl(buffer);

std::fprintf(" Message size in bytes: %u \n", message_size);
  • ntohs() => Similar to ntohl, but uses with short type, or 16 bit integers.
    • Mneumonic: n-to-h-s (Network to Host Short)
uint16_t ntohs(uint16_t netshort);

// ----------- Usage -----------------// 

 uint16_t number; 
 if( recv(sockfd, &number,  sizeof(uint32_t), 0 ) == -1 )
 {  /** Error handling ... here */  }

 number = ntohs(number);

Functions for resolving hostnames (DNS Query)

  • gethostbyname()
    • Translate hostname or IPv4 into IPv4 address. This function performs a DNS query.
struct hostent*  gethostbyname(const char* name);

struct hostent {
    char  *h_name;            /* official name of host */
    char **h_aliases;         /* alias list */
    int    h_addrtype;        /* host address type */
    int    h_length;          /* length of address */
    char **h_addr_list;       /* list of addresses */
}
#define h_addr h_addr_list[0] /* for backward compatibility */

1.28.3 Network Programming Pitfalls [DRAFT]

There are many pitfalls and edge-cases that a network-related can fall in to. This section aims to outline those issues that affect network code.

  • Lack of error checking
    • => Most BSD-socket APIs and Unix APIs return (-1) when an error happens and sets the errno global variable. If error is not properly checked, the calling code may be subject to silent and subtle failure without any warning. During the development assertions, assert(), statements can be used as temporary placeholders for error checking.
  • TCP message boundary
    • => The TCP protocol does not have message boundary, instead TCP is a byte stream oritend protocol. Threfore, there is no way to determine where a message begins or ends. For instance, if a sender sends three messages of 100 bytes via send() or write() functions and the peer socket attempts to read using three calls to read() or recev() using 1000 bytes long buffer, only a single a message will be received and the second call read() or recev() may block the current thread.
    • Solutions:
      • => Use a fixed size message (buffer with constant lenght) on both communication endpoints.
      • => Use custom delimiters for distinguishing the beginning and end of messages.
      • => Send messages prefixed with number of bytes to be read.
      • => Replace TCP with SCTP (Stream Control Transmission Protocol) which combines the best features of TCP and UDP. It provides both the TCP realibility and UDP message-boundary feature.
  • Partial-Reads
    • => Socket functions such recev() or read() may return the number of bytes read smaller than the buffer size, even though there is still bytes avaiable and data sent has a fixed size agreed by both socket endpoints. This edge-case scenario is called partial-read, a robust network code should be keeping reading the file descriptor until the buffer is filled or the connection is closed.
  • Partial-Writes
    • => Functions write() or send() may also be affected by similar edge-case to partial-reads and the number of bytes written returned by write() or send() may be smaller than the buffer size. In this case, the calling code, should be keeping sending the remaining parts of the buffer until all data sent over the network.
  • Endianess (Byte order for numeric types)
    • => In order to proper exchange a numeric type, both ends of a connection must agree about data endianess, or byte-order representation, since different processor architectures may have different byte order data representation of numeric types. In order to avoid a data being read in a wrong way, both sides of the connection must use the same endianess, which is the network-byte-order (big endian). So, a numeric value in machine endianess (default) should be converted to network-byte-order (big-endian) before being sent to the over the network, or written to the file descriptor. And a numeric value received from the network (socket file descriptor) must be converted from network-byte-order to the current machine endianess.
  • Fix-width integers
    • The fundamental integer types in C and C++, int, long, short, long long are not guaranteed to have the same size in all machine. For an instance, on Windows NT 64 bits the type 'long' has 32 bits, while on Linux, the type 'long' has 64 bits. If a socket server running on Linux attempts to send a value with long type to a client socket running on a Windows machine, the value will be interpreted in a incorrect way in the Windows machine. Data misinterpretation such as the previous mentioned one can be avoided by using fixed width integers instead of fundamental integer types as fix-width integers such as uint8_t, uint16_t, uint32_t, uint64_t, int8_t and so on, are guaranteed to have the same size in all machines
    • Summary: User fix-width integers (headers: <cstdint> for C++ or <stdint.h> for C) instead of fundamental types for avoiding data interpretation disagreement.
  • Sending and receiving structured data_
    • Structs or C++ classes should not be sent or received directly via read(), recev(), write() or send() functions due to many issues related to ABI, machine endianess, type-sizes and message-boundary. The first shortcoming is that the buffer argument of the mentioned functions can only be a pointer to POD type, which is a contiguous allocated type without any internal pointer. Other shortcoming are the machine endianess and compilers ABIs (Application Binary Interfaces) which may lead to the data to corruption and and data misinterpretation.
    • Solution: Convert and serialize each member of struct or class to an external data representation agreed by both connection endpoints.

Further Reading

1.28.4 Command Line Tools for Network Debugging

Show Routing Table

Linux: 

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.0.1        0.0.0.0         UG    600    0        0 wlp2s0
10.0.0.0        0.0.0.0         255.255.255.0   U     600    0        0 wlp2s0
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 wlp2s0

Linux: 

$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.0.0.1        0.0.0.0         UG        0 0          0 wlp2s0
10.0.0.0        0.0.0.0         255.255.255.0   U         0 0          0 wlp2s0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 wlp2s0

Mac OSX: 

$ netstat -rn

Routing tables

Internet:
Destination        Gateway            Flags        Netif Expire
default            10.0.2.2           UGSc           en0       
10.0.2/24          link#8             UCS            en0      !
10.0.2.2/32        link#8             UCS            en0      !
10.0.2.2           52:55:a:0:2:2      UHLWIir        en0   1161
10.0.2.3           link#8             UHLWIi         en0      !
10.0.2.15/32       link#8             UCS            en0      !
10.0.2.15          52:54:0:9:49:17    UHLWI          lo0       
127                127.0.0.1          UCS            lo0       
127.0.0.1          127.0.0.1          UH             lo0       
169.254            link#8             UCS            en0      !
224.0.0/4          link#8             UmCS           en0      !
224.0.0.251        1:0:5e:0:0:fb      UHmLWI         en0       
255.255.255.255/32 link#8             UCS            en0      !

Internet6:
Destination                             Gateway                         Flags         Netif Expire
default                                 fe80::2%en0                     UGc             en0       
default                                 fe80::%utun0                    UGcI          utun0       
default                                 fe80::%utun1                    UGcI          utun1       
::1                                     ::1                             UHL             lo0       
fe80::%lo0/64                           fe80::1%lo0                     UcI             lo0       
fe80::1%lo0                             link#1                          UHLI            lo0       
fe80::%en0/64                           link#8                          UCI             en0       
fe80::2%en0                             52:56:0:0:0:2                   UHLWIir         en0       
fe80::cb9:425:579:f7ca%en0              52:54:0:9:49:17                 UHLI            lo0       
fe80::%utun0/64                         fe80::69e6:edfe:e4dc:ccb3%utun0 UcI           utun0       
fe80::69e6:edfe:e4dc:ccb3%utun0         link#9                          UHLI            lo0       
fe80::%utun1/64                         fe80::1b23:740d:d7d0:b07d%utun1 UcI           utun1       
fe80::1b23:740d:d7d0:b07d%utun1         link#10                         UHLI            lo0       
fec0::/64                               link#8                          UC              en0       
fec0::2                                 52:56:0:0:0:2                   UHLWI           en0       
fec0::cee:fd13:3be4:8411                52:54:0:9:49:17                 UHL             lo0       
fec0::2d6d:1074:dc19:b28a               52:54:0:9:49:17                 UHL             lo0       
ff01::%lo0/32                           ::1                             UmCI            lo0       
ff01::%en0/32                           link#8                          UmCI            en0       
ff01::%utun0/32                         fe80::69e6:edfe:e4dc:ccb3%utun0 UmCI          utun0       
ff01::%utun1/32                         fe80::1b23:740d:d7d0:b07d%utun1 UmCI          utun1       
ff02::%lo0/32                           ::1                             UmCI            lo0       
ff02::%en0/32                           link#8                          UmCI            en0       
ff02::%utun0/32                         fe80::69e6:edfe:e4dc:ccb3%utun0 UmCI          utun0       
ff02::%utun1/32                         fe80::1b23:740d:d7d0:b07d%utun1 UmCI          utun1

Network interfaces

For newer Linux distributions:

  • The following command shows that the current machine has three network interfaces. lo is a virtual network interface that corresponds to localhost (127.0.0.1); enps10 => corresponds to a ethernet network card, wbhich is not active; and finally wlp2s0 which is a wifi network card, which the IPv4 address on the local network is 10.0.0.175 and IPv6 address is fe81::88a:6b9c:fcef:4e32.
$ ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever

2: enp1s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 9a:8b:8c:f2:b3:d4 brd ff:ff:ff:ff:ff:ff

3: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 9a:8c:8a:ef:ff:0c brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.175/24 brd 10.0.0.255 scope global dynamic noprefixroute wlp2s0
       valid_lft 84718sec preferred_lft 84718sec
    inet6 fe81::88a:6b9c:fcef:4e32/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

For BSD, MacOSX and older Linux Distributions.

  • Note: The ifconfig cli application is deprecated on most Linux distributions and was replaced by the ip command..
# On MacOSX VM 
$ ifconfig

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
        options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP>
        inet 127.0.0.1 netmask 0xff000000 
        inet6 ::1 prefixlen 128 
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
        nd6 options=201<PERFORMNUD,DAD>

gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280

stf0: flags=0<> mtu 1280

UHC29: flags=0<> mtu 0

UHC93: flags=0<> mtu 0

EHC253: flags=0<> mtu 0

UHC61: flags=0<> mtu 0

en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=424<VLAN_MTU,TSO4,CHANNEL_IO>
        ether 51:53:2a:f8:49:27 
        inet6 fe20::cc9:4a5:549:f7ca%en0 prefixlen 64 secured scopeid 0x8 
        inet6 fec0::cab:f513:8be4:9251 prefixlen 64 autoconf secured 
        inet6 fec0::2c6a:1276:dc19:b29a prefixlen 64 autoconf temporary 
        inet 10.0.2.20 netmask 0xffffff00 broadcast 10.0.2.255
        nd6 options=201<PERFORMNUD,DAD>
        media: autoselect (1000baseT <full-duplex>)
        status: active

utun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
        inet6 fe80::69f6:edfe:84dc:acb3%utun0 prefixlen 64 scopeid 0x9 
        nd6 options=201<PERFORMNUD,DAD>

utun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 2000
        inet6 fe80::1b2c:7424:d7d0:ca7d%utun1 prefixlen 64 scopeid 0xa 
        nd6 options=201<PERFORMNUD,DAD>

Display all listening TCP sockets and open ports (socket servers) 

 $ >> netstat --listening -tnp
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:9000            0.0.0.0:*               LISTEN      142308/python       
tcp        0      0 192.168.100.1:53        0.0.0.0:*               LISTEN      -                   
tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:9056            0.0.0.0:*               LISTEN      434583/./poll-serve 
tcp6       0      0 :::9070                 :::*                    LISTEN      131190/java         
tcp6       0      0 :::8080                 :::*                    LISTEN      100425/java         
tcp6       0      0 ::1:631                 :::*                    LISTEN      -                   
tcp6       0      0 :::9080                 :::*                    LISTEN      100272/java

List all UDP connections 

$ netstat --udp
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
udp        0      0 localhost.locald:bootpc _gateway:bootps         ESTABLISHED

Display all Listening TCP/IP, UDP/IP and Unix Domain Sockets 

 $ >> netstat --listening -p
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:cslistener      0.0.0.0:*               LISTEN      142308/python       
tcp        0      0 localhost.locald:domain 0.0.0.0:*               LISTEN      -                   
tcp        0      0 localhost.locald:domain 0.0.0.0:*               LISTEN      -    
  ...   ...   ...   ...   ...   ...   ...   ...   ... 
  ...   ...   ...   ...   ...   ...   ...   ...   ...   ... 

unix  2      [ ACC ]     STREAM     LISTENING     45857    1720/systemd         /run/user/1000/systemd/private
unix  2      [ ACC ]     STREAM     LISTENING     116650   -                    @/tmp/.X11-unix/X0
unix  2      [ ACC ]     STREAM     LISTENING     125379   7281/Xorg            @/tmp/.X11-unix/X1
unix  2      [ ACC ]     STREAM     LISTENING     780071   7994/emacs           /run/user/1000/emacs/server
unix  2      [ ACC ]     STREAM     LISTENING     45863    1720/systemd         /run/user/1000/bus
unix  2      [ ACC ]     STREAM     LISTENING     45866    1720/systemd         /run/user/1000/pipewire-0
unix  2      [ ACC ]     STREAM     LISTENING     45868    1720/systemd         /run/user/1000/pulse/native
unix  2      [ ACC ]     STREAM     LISTENING     2443571  388596/code --no-sa  /run/user/1000/vscode-1d972a5c-1.45.1-main.sock
unix  2      [ ACC ]     STREAM     LISTENING     77108    3932/tmux            /tmp//tmux-1000/default
unix  2      [ ACC ]     STREAM     LISTENING     3152439  456731/code          /run/user/1000/vscode-git-be7a37b432.sock
unix  2      [ ACC ]     STREAM     LISTENING     36623    -                    @/tmp/dbus-00sao3pj

Show ARP table 

$ arp 
Address                  HWtype  HWaddress           Flags Mask            Iface
_gateway                 ether   21:fe:1d:78:d4:28   C                     wl_1s22f0u4
10.0.1.113               ether   7b:3f:24:57:2b:9c   C                     wlp1s22f0u4
10.0.1.156               ether   80:92:26:87:ab:8f   C

1.28.5 Simple TCP/IP client socket

Description:

  • This sample code contains a simple TCP/IP client socket application that connects to a netcat running as server.

GIST Containing source:

File: client-sock1.cpp 

#include <iostream>
#include <cassert> // Use: assert() for marking placeholders for future error handling.
#include <string>  // std::string 
#include <cstring> // memcpy, memset and so on

// ----- Unix/Linux headers -----// 
#include <unistd.h>
#include <fcntl.h>
#include <sys/types.h>    
#include <sys/socket.h>
#include <netdb.h>
#include <limits.h>

constexpr int OPERATION_FAILURE = -1;

// Read buffer from file descriptor handling partial reads.
// Note: It should only be used if the counterpart socket, sends 
// a buffer of equal to the size of this 'size' parameter.
void read_buffer(int fd, void* buffer, size_t size);

// Write buffer to file descriptor handling partial writes;
void write_buffer(int fd, const void* buffer, size_t size);

// Send text with '\n' UNIX (EOL) end line character.
void write_line(int fd, std::string const& text);

int main(int argc, char** argv)
{
    // ======= Server port and hostname configuration ===========//

     // The port should be a 16 bits number 
     // (Ports less than 1024 requires administrative/root privilege.)
     std::uint16_t port = 8095;
     // '127.0.0.1' for local host (LO Network interface)
     const char* hostname = "127.0.0.1";

     // ======= Create socket file descriptor =====================//

     int sockfd = socket (  AF_INET       // domain:   IPv4
                          , SOCK_STREAM   // type:     TCP 
                          , 0             // protocol: (value 0) 
                         );
    // Returns (-1) on failure      
    assert( sockfd != OPERATION_FAILURE);    

    // =========== Query Address (DNS query) =======================//

     // Note: the keyword 'struct' is not necessary here (redundant). 
     struct hostent* h = gethostbyname(hostname);
     assert(h != nullptr);

     struct sockaddr_in sa; 
     // memset(&sa, 0x00, sizeof(sa));
     // set address 
    std::memcpy(&sa.sin_addr.s_addr, h->h_addr, h->h_length);

     //bcopy( h->h_addr, &sa.sin_addr.s_addr, h->h_length );

     sa.sin_family = AF_INET;
     // The function htons convert a number to big-endian format. 
     sa.sin_port   = htons(port); 

    // ======== Attempt connection to server ========================//

    int conn_result = connect( // Socket file descriptor 
                               sockfd
                               // Pointer sockaddr => Structure containing host address 
                             , reinterpret_cast<sockaddr*>(&sa)
                               // Size of sockaddr buffer in bytes 
                             , sizeof(sockaddr_in));

    if( conn_result == OPERATION_FAILURE )
    {
        std::fprintf(stderr, " [ERROR] Unable to connect to server. \n");
        exit(EXIT_FAILURE);
    }

    //============= Session Loop ===================================//

    // --- Send a buffer (untyped memory location) ----------// 
    char start_message[500];
    std::memset(start_message, '\0', 500);
    std::strcpy(start_message, " [CLIENT] <MESSAGE 1> Hi there, server, I am the client socket \n");    
    write_buffer( sockfd, start_message, strlen(start_message) );

    // --- Send text --------------------------------------//
    write_line(sockfd, " [CLIENT] <MESSAGE 2> Hi there, server, I was sent by the client-side \n");

    constexpr size_t BUFFER_SIZE = 500;
    char buffer[BUFFER_SIZE];

    // ---------- Session Loop ------------------------------//
    for(;;)
    {        
        // Clean buffer before each iteratiion 
        std::memset(buffer, 0x00, BUFFER_SIZE);

        std::fprintf(stdout, " [TRACE] Waiting for server input. " 
                             " =>> Type something in the server netcat terminal and hit RETURN. \n");

        ssize_t r = read(sockfd, buffer, BUFFER_SIZE);
        if(r == 0){
            std::fprintf(stdout, " [TRACE] Server has closed the connection. " 
                               "User types CTRL + D in the server-side terminal. \n");
            break;
        } 
        if( r == -1){
            std::fprintf(stdout, " [TRACE] An error has happend Check ERRNO. ");
            break;
        } 

        std::fprintf(stdout, " [SERVER DATA] Server has sent: %s \n", buffer);

        // Send data back to server 
        write_buffer(sockfd, buffer, BUFFER_SIZE);

    } // --- End of session loop ---------------------------//

    std::fprintf(stderr, " [INFO] Session shutdown OK \n");

    close(sockfd);
    return 0;
}

    // ------------------------------------------------------//
    //          I M P L E M E N T A T I O N S                //
    //-------------------------------------------------------//

void read_buffer(int fd, void* buffer, size_t size)
{
   // Pre-condition 
   assert( size <= SSIZE_MAX  );

   size_t  len  = size;
   ssize_t nr   = -1;
   // Note: std::byte (C++17) is a better replacement for char* or std::uint8_t 
   // as std::byte is more effective for conveying the intent.
   auto pbuf = reinterpret_cast<std::byte*>(buffer); 

   std::fprintf(stderr, " [TRACE] Waiting input \n" );

   while(  len != 0 && ( nr = read(fd, pbuf, len) ) != 0 ) 
   {

       if(nr == -1 && errno == EINTR){ continue; } 
       if(nr == -1) { 
           throw std::runtime_error("An error has happened, check ERRNO ");
           break;
       }
       len  = len - nr; 
       pbuf = pbuf + nr;

       std::cerr << " [TRACE] len = " << len << " ; nr = " << nr << '\n';
   }

   std::fprintf(stderr, " [TRACE] Loop finished. \n" );
}

void write_buffer(int fd, const void* buffer, size_t size)
{
   // Pre-condition 
   assert( size <= SSIZE_MAX  );
   size_t  len  = size;
   ssize_t nr   = -1;
   auto pbuf = reinterpret_cast<const std::uint8_t*>(buffer); 

  while(  len != 0 && ( nr = write(fd, pbuf, len) ) != 0 ) 
  {
      if(nr == -1 && errno == EINTR){ continue; } 
      if(nr == -1) { 
          throw std::runtime_error("An error has happened, check ERRNO ");
          break;
      }
      len  = len - nr; 
      pbuf = pbuf + nr;
   }
}

void write_line(int fd, std::string const& text)
{
    write_buffer(fd, text.data(), text.size());
    char ch = '\n';
    write(fd, &ch, 1);
}

Building: 

$ g++ sock-client1.cpp -std=c++1z -Wall -Wextra -g -O0

Run netcat as server in terminal 1 (server-side) listening port 8095: 

 $ >> nc -v -l 8095
Ncat: Version 7.80 ( https://nmap.org/ncat )
Ncat: Listening on :::8095
Ncat: Listening on 0.0.0.0:8095
Ncat: Connection from 127.0.0.1.
Ncat: Connection from 127.0.0.1:33180.
 [CLIENT] <MESSAGE 1> Hi there, server, I am the client socket 
 [CLIENT] <MESSAGE 2> Hi there, server, I was sent by the client-side 

Message to be sent to client-side
Message to be sent to client-side
Hello world client side
Hello world client side
I am the server - I provide network services such as tensor calculations
I am the server - I provide network services such as tensor calculations
I will type CTRL + D and shutdown the connection after this message
I will type CTRL + D and shutdown the connection after this message

Run sock-client1.bin in terminal 2 connecting to port 8095 from local host: 

$ ./sock-client1.bin 
 [TRACE] Waiting for server input.  =>> Type something in the server netcat terminal and hit RETURN. 
 [SERVER DATA] Server has sent: Message to be sent to client-side

 [TRACE] Waiting for server input.  =>> Type something in the server netcat terminal and hit RETURN. 
 [SERVER DATA] Server has sent: Hello world client side

 [TRACE] Waiting for server input.  =>> Type something in the server netcat terminal and hit RETURN. 
 [SERVER DATA] Server has sent: I am the server - I provide network services such as tensor calculations

 [TRACE] Waiting for server input.  =>> Type something in the server netcat terminal and hit RETURN. 
 [SERVER DATA] Server has sent: I will type CTRL + D and shutdown the connection after this message

 [TRACE] Waiting for server input.  =>> Type something in the server netcat terminal and hit RETURN. 
 [TRACE] Server has closed the connection. User types CTRL + D in the server-side terminal. 
 [INFO] Session shutdown OK

1.28.6 Simple TCP/IP socket echo server

Description:

  • Sample code for a single-thread TCP/IP socket server listening at port 8095.

GIST:

File: socket-echo-server.cpp 

// File: socket-echo-server.cpp 
// Desc: Simple socket server with a single thread. 

#include <iostream>
#include <string> 
#include <vector>
#include <cassert> 
#include <cstring>  // memcpy 

// ----- Unix/Linux headers -----// 
#include <unistd.h>
#include <fcntl.h>
#include <sys/types.h>    
#include <sys/socket.h>
#include <netdb.h>

// -------------  MAIN() ------------------------------//

int main(int argc, char** argv)
{
     std::puts(" [INFO] Program started. ");

     // The port should be a 16 bits number (Note: binding to some low port numbers 
     // requires administrative/root privilege.)
     std::uint16_t port = 8095;

     // The address can be: 
     //   '0.0.0.0'   for listening all hosts 
     //   '127.0.0.1' for local host 
     const char* hostname = "0.0.0.0";

     // --- Create socket file descriptor -----------------------//

     int sockfd = socket (  AF_INET       // domain:   IPv4
                          , SOCK_STREAM   // type:     TCP 
                          , 0             // protocol: (value 0) 
                         );
     // Returns (-1) on failure 
     assert( sockfd != - 1);

     // ---------- query address ------------------------------//

     // Note: the keyword 'struct' is not necessary here (redundant). 
     struct hostent* h = gethostbyname(hostname);
     assert(h != nullptr);

     struct sockaddr_in sa; 
     // memset(&sa, 0x00, sizeof(sa));
     // set address 
     memcpy(&sa.sin_addr.s_addr, h->h_addr, h->h_length);

     //bcopy( h->h_addr, &sa.sin_addr.s_addr, h->h_length );

     sa.sin_family = AF_INET;
     // The function htons convert a number to big-endian format. 
     sa.sin_port   = htons(port); 

     // ----- Bind to Port and wait for client connections ---------//

     if( ::bind(sockfd, (sockaddr *) &sa, sizeof(sa)) == -1)
     {
         fprintf(stderr, "Error: unable to bind socket \n");
         return EXIT_FAILURE;
     }

     // Enables binding to the same address 
     int enable_reuseaddr = 1;
     if ( setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &enable_reuseaddr, sizeof(int)) < 0 )
     {
         fprintf(stderr, "Error: unable to set socket option. \n");
         return EXIT_FAILURE;
     }

     fprintf(stderr, " [TRACE] Listening client connection \n");

     int backlog = 5;
     assert( listen(sockfd, backlog) != -1 );

     // --------------- Server Main Loop --------------------------//

     // Infinite loop where client connections are handled
     while(true)
     {  
         /* [[ ============== BEGIN client loop =========== ]] */

         struct sockaddr_in client_sa; 
         socklen_t addrlen = sizeof(sockaddr_in); 

         fprintf(stderr, " [TRACE] Waiting for client connection. \n");
         int sock_client = accept(sockfd, (sockaddr *) &client_sa, &addrlen );

         if(sock_client == -1)
         {
              fprintf(stderr, " Error: failure to handle client socket. Check errno \n");
              close(sock_client);
              continue;
         }
         const char* welcome_msg = "\n => [INFO] Hello world client side!! \n";     

         // Send buffer content to client socket 
         send(sock_client, welcome_msg, strlen(welcome_msg), 0);

         char buffer[300];

         while(true){
             // Read message from client socket 
             ssize_t n = recv(sock_client, &buffer, 300, 0);
             if(n == 0){ break; }

             std::fprintf(stdout, " [INFO] Received: ");            
             ::write(STDOUT_FILENO, buffer, n);

             // Send content back 
             send(sock_client, buffer, n, 0);

             // Compare 4 first bytes of buffer and 'quit' 
             if( strncmp(buffer, "quit", 4) == 0)
             {
                 fprintf(stderr, " [TRACE] Shutdown connection. Ok. \n");
                 // Close client connection 
                 shutdown(sock_client, SHUT_RDWR);
                 close(sock_client);
                 break;
             }          
         }

     } /* [[ ============== END client loop =========== ]] */

     // Release resource => RAII (Resource-Acquisition-Is-Initialization) fits here. 
     close(sockfd);
     return 0;
}

Building: 

$ g++ socket-echo-server.cpp -o socket-echo-server.elf -std=c++1z -Wall -Wextra

Running server: 

$ ./socket-echo-server.elf 
 [INFO] Program started. 
 [TRACE] Listening client connection 
 [TRACE] Waiting for client connection. 
c++1z ada spark high performance
c++2z c++20 C++17 c++14 C89 c90 ancient unix
quit
 [TRACE] Shutdown connection. Ok. 
 [TRACE] Waiting for client connection.

Running client: (netcat tool) 

$ nc localhost 8095

 => [INFO] Hello world client side!! 
c++1z ada spark high performance
c++1z ada spark high performance
c++2z c++20 C++17 c++14 C89 c90 ancient unix
c++2z c++20 C++17 c++14 C89 c90 ancient unix
quit
quit

^C

1.28.7 Simple Unix-domain socket echo server

Unix domain-socket (not confuse with BSD socket) is an inter-process mechanism that allows usage of socket communication on local machine with less overhead than TCP/IP socket. A significat difference between IP sockets and Unix sockets is that unlike IP-based sockets, Unix domain sockets use file path or abstract namespace not bound to file system as address instead of

The most important usages of Unix-domain sockets are:

  • Implementing network transparency.
  • IPC - Inter Process Network Communication,
  • Privilege separation (Security)

Known usage

  • DBUS - Linux DBUS
  • X11 - X Windows Systems uses Unix Domain Sockets locally for enabling network transparency. A code using Unix Domain socket needs does not need many changes for using IP-based sockets and operate over the network.
  • Docker Container Engine - Docker uses Unix Domain Sockets for privilege separation and network transparency. The docker damenon, running with root (administrative) privilege and the docker client, running without administrative privilege, communicate via Unix domain sockets on local machine and or via TCP/IP sockets over the ĩnternt.

See also

Sample Code

GIST:

File: unix-domain-echo-server.cpp 

// File: socket-echo-server.cpp 
// Desc: Simple socket server with a single thread. 

#include <iostream>
#include <string> 
#include <vector>
#include <cassert> 
#include <cstring>  // memcpy 

// ----- Unix/Linux headers -----// 
#include <unistd.h>
#include <fcntl.h>
#include <sys/types.h>    
#include <sys/socket.h>
#include <sys/un.h>

// -------------  MAIN() ------------------------------//

int main(int argc, char** argv)
{
     std::puts(" [INFO] Program started. ");
     const char* unix_sock_file = "/tmp/unix-socket-file";

     // --- Create socket file descriptor -----------------------//

     int sockfd = socket (  AF_UNIX, SOCK_STREAM, 0);
     assert( sockfd != - 1);

     struct sockaddr_un sa; 
     sa.sun_family = AF_UNIX; 
     strcpy( sa.sun_path, unix_sock_file);  

     // Remove file if it exists.   
     unlink(unix_sock_file);

     // ----- Bind to unix domain socket and wait for client connections ---------//
     if( ::bind(sockfd, (sockaddr *) &sa, sizeof(sa)) == -1)
     {
         int err = errno; 
         fprintf(stderr, "Error: unable to bind socket \n");
         fprintf(stderr, " [ERROR] => %s", strerror(err));
         return EXIT_FAILURE;
     }

     // Enables binding to the same address 
     int enable_reuseaddr = 1;
     if ( setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &enable_reuseaddr, sizeof(int)) < 0 )
     {
         fprintf(stderr, "Error: unable to set socket option. \n");
         return EXIT_FAILURE;
     }


     fprintf(stderr, " [TRACE] Listening client connection \n");

     int backlog = 5;
     assert( listen(sockfd, backlog) != -1 );

     // --------------- Server Main Loop --------------------------//

     // Infinite loop where client connections are handled
     while(true)
     {  
           /* [[ ============== BEGIN client loop =========== ]] */

           struct sockaddr_un client_sa; 
           socklen_t addrlen = sizeof(sockaddr_un); 

           fprintf(stderr, " [TRACE] Waiting for client connection. \n");
           int sock_client = accept(sockfd, (sockaddr *) &client_sa, &addrlen );

           if(sock_client == -1)
           {
                   fprintf(stderr, " Error: failure to handle client socket. Check errno \n");
                   close(sock_client);
                   continue;
           }
           const char* welcome_msg = "\n => [INFO] Hello world client side!! \n";       

           // Send buffer content to client socket 
           send(sock_client, welcome_msg, strlen(welcome_msg), 0);

           char buffer[300];

           while(true){
                 // Read message from client socket 
                 ssize_t n = recv(sock_client, &buffer, 300, 0);
                 if(n == 0){ break; }

                 std::fprintf(stdout, " [INFO] Received: ");            
                 ::write(STDOUT_FILENO, buffer, n);

                 // Send content back 
                 send(sock_client, buffer, n, 0);

                 // Compare 4 first bytes of buffer and 'quit' 
                 if( strncmp(buffer, "quit", 4) == 0)
                 {
                    fprintf(stderr, " [TRACE] Shutdown connection. Ok. \n");
                    // Close client connection 
                    shutdown(sock_client, SHUT_RDWR);
                    close(sock_client);
                    break;
                 }          
         }

   } /* [[ ============== END client loop =========== ]] */

   // Release resource => RAII (Resource-Acquisition-Is-Initialization) fits here. 
   close(sockfd);
   return 0;
}

Build: 

$ g++ unix-domain-echo-server.cpp -o unix-domain-echo-server.bin -std=c++1z -Wall -Wextra -g

Run server: 

$ ./unix-domain-echo-server.bin 
 [INFO] Program started. 
 [TRACE] Listening client connection 
 [TRACE] Waiting for client connection. 
unix irix posix linux macosx kernel
c++1z c++14 c++20 ADA spark ADA rust C++ Lisp

quit
 [TRACE] Shutdown connection. Ok. 
 [TRACE] Waiting for client connection.

Run client (netcat) : 

$ nc -v -U /tmp/unix-socket-file 
Ncat: Version 7.80 ( https://nmap.org/ncat )
Ncat: Connected to /tmp/unix-socket-file.

 => [INFO] Hello world client side!! 
unix irix posix linux macosx kernel
unix irix posix linux macosx kernel
c++1z c++14 c++20 ADA spark ADA rust C++ Lisp
c++1z c++14 c++20 ADA spark ADA rust C++ Lisp


quit
quit
^C

Check unix-socket file: 

$ file /tmp/unix-socket-file 
/tmp/unix-socket-file: socket

$ ls -l /tmp/unix-socket-file 
srwxrwxr-x 1 mxpkf8 mxpkf8 0 Jul  2 01:22 /tmp/unix-socket-file=

1.28.8 Telnet-like application

The following code is a sample telent-like application, which provide remote access to interactive shells (REPLs read-eval-print-loop) from local machine, where the application is running. The application can work in server or client mode. When running in client mode, it attempts to connect to remote netcat server. When running in server mode, the application accepts connection from any netcat server. Interactive shells are applications such as: irb (Ruby shell); ash shell; TCL shell and so on.

File: rsh-server.cpp 

// Desc: Simple telnet-like server.

#include <iostream>
#include <string> 
#include <vector>
#include <cassert> 
#include <cstring>  // memcpy 

// ----- Unix/Linux headers -----// 
#include <unistd.h>
#include <fcntl.h>
#include <sys/types.h>    
#include <sys/socket.h>
#include <netdb.h>
#include <sys/types.h>
#include <sys/wait.h>

constexpr int UNIX_FAILURE = -1;

// Create server socket and return its file descriptor.
int  create_server (const char* hostname, std::uint16_t port, int backlog);

// Spawn a process and redirect its standard streams
// to socket (stderr, stdin, stdout)
void spawn_redirect (int sockfd, const char* executable);

// Send text message to socket
void send_msg (int sockfd, const char* message);

// Connect to remote server and return client socket file descriptor.
int socket_connect(const char* hostname, std::uint16_t port);

// Handle session between server and client.
void session_handler(int sockfd);

// -------------  MAIN() ------------------------------//

int main(int argc, char** argv)
{
    // Placeholder for future error handling.
     assert( argc == 4 && " Requires at least three arguments. " );

     int sockfd = -1;
     auto command  = std::string{ argv[1] };
     auto hostname = std::string{ argv[2] };
     int port      = std::stoi(argv[3]);

     // Run as client mode => Forwarding the shell to the server.
     if(command == "client") for(;;) {
         sockfd = socket_connect(hostname.c_str(), port);

         if( sockfd > 0 )
            std::fprintf(stderr, " [OK   ] Connected to server. \n");
         else
            std::fprintf(stderr, " [ERROR] Conncection failure. Retry. \n");

         session_handler(sockfd);
         sleep(2);
     } // ---- End of client command ---- //

     // Run in server mode => Liste connection.
     if(command == "server")
     {
         sockfd = create_server(hostname.c_str(), port, 5);
         assert( sockfd != -1 );

         // ============= Server Event Loop =============//
         for(;;){

             struct sockaddr_in client_sa;
             socklen_t addrlen = sizeof(sockaddr_in);

             fprintf(stderr, " [TRACE] Waiting for client connection. \n");
             int sock_client = accept(sockfd, (sockaddr *) &client_sa, &addrlen );

             if(sock_client == -1)
             {
                 fprintf(stderr, " Error: failure to handle client socket. Check errno \n");
                 close(sock_client);
                 continue;
             }

             session_handler(sock_client);

         } // --- End of for(;;) server loop ------//

     } // --- End of server command ---//


     // Release resource => RAII (Resource-Acquisition-Is-Initialization) fits here.
     close(sockfd);
     return 0;
}


   // =============================================//
   //          D E F I N I T I O N S               //
   //==============================================//

int create_server(const char* hostname, std::uint16_t port, int backlog)
{
    // --- Create socket file descriptor -----------------------//

     int sockfd = socket ( AF_INET, SOCK_STREAM, 0);

     // Future error handling.
     // Returns (-1) on failure
     assert( sockfd != - 1);

     // ---------- query address ------------------------------//
     // Note: the keyword 'struct' is not necessary here (redundant).
     struct hostent* h = gethostbyname(hostname);
     assert(h != nullptr);

     struct sockaddr_in sa;
     memcpy(&sa.sin_addr.s_addr, h->h_addr, h->h_length);
     sa.sin_family = AF_INET;
     // The function htons convert a number to big-endian format.
     sa.sin_port   = htons(port);

     // ----- Bind to Port and wait for client connections ---------//
     if( ::bind(sockfd, (sockaddr *) &sa, sizeof(sa)) == -1)
     {
         fprintf(stderr, "Error: unable to bind socket \n");
         return EXIT_FAILURE;
     }
     // Enables binding to the same address
     int enable_reuseaddr = 1;
     if ( setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &enable_reuseaddr, sizeof(int)) < 0 )
     {
         fprintf(stderr, "Error: unable to set socket option. \n");
         return EXIT_FAILURE;
     }

     fprintf(stderr, " [TRACE] Listening client connection \n");
     assert( listen(sockfd, backlog) != -1 );

     return sockfd;
}

int socket_connect(const char* hostname, std::uint16_t port)
{
    int sockfd =  socket (  AF_INET, SOCK_STREAM, 0);
    // assert() => Placeholder for future error handling.
    assert( sockfd != UNIX_FAILURE );

    // Query DNS
    struct hostent* h = gethostbyname(hostname);
    assert( h != nullptr );

    struct sockaddr_in sa;
    std::memcpy(&sa.sin_addr.s_addr, h->h_addr, h->h_length);
    sa.sin_family = AF_INET;
    sa.sin_port   = htons(port);

    int res = connect( sockfd
                     , reinterpret_cast<sockaddr*>(&sa)
                     , sizeof(sockaddr_in));

    if( res == -1){ return -1; }
    return sockfd;
}


void send_msg(int sockfd, const char* message)
{
    send(sockfd, message, strlen(message), 0);
}


void spawn_redirect(int sockfd, const char* executable)
{
    pid_t pid = fork();

    switch (pid) {
    case -1:
        std::fprintf(stderr, "Error: unable to fork process. ");
        exit(EXIT_FAILURE);
        break;
        // ----------- Child Process ------------//
    case 0: {

        std::fprintf( stderr, " [TRACE] Spawning process (PID = %d ) shell: %s \n"
                    , ::getpid(), executable);

        // Redirect standard streams to socket file descriptor
        dup2(sockfd, STDOUT_FILENO);
        dup2(sockfd, STDIN_FILENO );
        dup2(sockfd, STDERR_FILENO);

        const char* args [] = {
            "rshell"    // Process name (arbitrary)
            , nullptr   // Sentinel value - (always NULL pointer)
        };

        // Spawn a new process.
        int status = execvp( executable, const_cast<char* const*>(args) );
        if( status == UNIX_FAILURE ){
            int err = errno;
            send_msg(sockfd, " [SERVER ERROR] Unable to locate REPL \n");
            send_msg(sockfd, strerror(err));
            close(sockfd);
            exit(EXIT_FAILURE);
        }

    }
        break;
        //----------- Parent Process ------------//
    default:
        // Wait for child process termination.
        ::waitpid(pid, nullptr, 0);
        break;
    }
}

void session_handler(int sockfd)
{
    constexpr size_t SIZE = 500;
    char buffer[500];
    // Always initialize buffer for avoiding undefined behavior.
    std::memset(buffer, 0x00, SIZE);

    send_msg(sockfd, "Enter the shell to be spawned: ");
    if( ::recv(sockfd, buffer, SIZE, 0) <= 0)
    {
        std::fprintf(stderr, "Connection error or peer closed connection \n");
        return;
    }

    std::fprintf(stderr, " [TRACE] Waiting for child process termination. \n");

    // Remove '\n' end line character if it is found.
    if( char* ptr_char = (char*) ::memchr(buffer, '\n', SIZE) )
    { *ptr_char = '\0'; }

    // Fork a new process (run executable indicated by the buffer).
    spawn_redirect(sockfd, buffer);

    std::fprintf(stderr, " [TRACE] Child process terminated Ok. \n");
}

Running on MacOSX

Building on MacOSX Catalina: 

$ g++ rsh-server.cpp -o rsh-server.osx -std=c++1z -Wall -Wextra -g

$ file rsh-server.osx
rsh-server.osx: Mach-O 64-bit executable x86_64

$ otool -L rsh-server.osx
rsh-server.osx:
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 902.1.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)

Run terminal A => (Remote Linux Machine, netcat as server): 

$ >> rlwrap netcat  -l 9070
Enter the shell to be spawned: sh

# User typed: $ whoami  
whoami 
unix

# User typed: $ uname -a 
uname -a
Darwin ghosts-iMac-Pro.local 19.5.0 Darwin Kernel Version 19.5.0: Tue May 26 20:41:44 PDT 2020; root:xnu-6153.121.2~2/RELEASE_X86_64 x86_64

# User typed: $ ls / 
ls /
Applications
Library
System
Users
Volumes
bin
cores
dev
etc
home
opt
private
sbin
tmp
usr
var

# User typed: $ ls /Applications 
ls /Applications 
Safari.app
Utilities
iTerm.app

# User typed : '$ osalang' for listing all supported 
# OSA (Open Scripting Architecture) languages.
osalang 
AppleScript
JavaScript
Generic Scripting System

# User typeed: # osascript -e .... ... 
osascript -e 'display dialog "My Dialog in OSA AppleScripting Automation!" '
Unable to create basic Accelerated OpenGL renderer.
Unable to create basic Accelerated OpenGL renderer.
Core Image is now using the software OpenGL renderer. This will be slow.
button returned:OK

# User typed: $ osascript -e .... 
osascript -e 'tell app "Finder" to make new Finder window'
Finder window id 117

 # User typed: $ osascript -e .... 
osascript -e 'display notification "Temperature 31 celsius." with title "Wheather report"'


# Type Ctrl + C to shutdown server

Run terminal B => (MacOSX, run rsh-server in client mode): 

./rsh-server.osx client 10.0.2.2 9070 
 [OK   ] Connected to server. 
 [TRACE] Waiting for child process termination. 
 [TRACE] Spawning process (PID = 1569 ) shell: sh 
 [TRACE] Child process terminated Ok. 
 [ERROR] Conncection failure. Retry. 
Connection error or peer closed connection 
 [ERROR] Conncection failure. Retry. 
Connection error or peer closed connection 
 [ERROR] Conncection failure. Retry. 
Connection error or peer closed connection 
 [OK   ] Connected to server.

1.28.9 Telnet-like application with pseudo-terminal devices

The previous telnet-like application is not able to display colors or handle interactive applications which expect to be running in a terminal device, such as sudo, python and so on.

This rewrite of the previous code solves those shortcomings by using UNIX pseudo-terminal device pair, comprised of master and slave pseudo terminal devices. In this new code version, the slave pseudo terminal device is connected to the remote application's standard streams stdin, stdout and stederr. The master pseudo terminal device is connected to the socket. Any message received by the socket is forwarded to the master pseudo-terminal device, then the slave pseudo-terminal devices forwards this data to the process input. Any process output, stderr or stdout, are relayed to the slave pseudo terminal device, then the master pseudo terminal device forwards to the socket that sends the data over the network.

The pseudo terminal devices application programming interface from Unix-like systems is one of the most complex ones due to its long history and sparse documentation. This API for instance, has RS232 UART baud rate parameters that dates back to the mainframe era where terminals was a single unit comprised of a screen and a keyboard attached to a mainframe via a serial cable.

Concepts involved in the Terminal API

  • UNIX signals
  • TTY
  • PTY - Pseudo terminal device
  • Shell
    • Interactive intepreter REPL (read-eval-print-loop), such as bash or python.
  • Controlling terminal
  • Terminal emulator (Example: Xterm, Gnome terminal)
  • Child process
  • Parent process
  • Session leader (often the shell)
  • Session - set of or one or more process groups.
  • Process group (also called 'job')
  • Foreground process
  • Background process
  • Pseudo-terminal modes:
    • Canonical mode (default) => allso called non raw mode. Reads line, by line. Provides default command line history, echoing and allows moving the cursor forward.
    • Non-canonical => also called raw mode => This mode does not provide, history, character echoing, cursor backward movement and so on. Used by Emacs and VI and other interactive terminal applications.
  • Applications that cannot work without pseudo terminals:
    • ssh
    • telnet => Predecessor of ssh
    • script => Records shell session to a file
    • rlogin
    • GNU screen
    • Tmux => Terminal multiplexer
    • sudo => For elevating user privilege.

File: rshell.cpp 

// Desc: Simple telnet-like server using pseudo terminal devices.
#include <iostream>
#include <string>
#include <vector>
#include <cassert>
#include <cstring>  // memcpy

// ----- Unix/Linux headers -----//
#include <unistd.h>
#include <fcntl.h>
#include <sys/types.h>
#include <sys/socket.h>

#include <netdb.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <poll.h>


// ------ UNIX Pseudo Terminal Device API headers ---------//
//
#include <termios.h>

#if defined(__linux__)
// On Linux
#include <pty.h>
#else
// On BSDs =>> MacOSX, FreeBSD, OpenBSD
#include <util.h>
#include <sys/ioctl.h>
#endif


constexpr int UNIX_FAILURE = -1;

// Create server socket and return its file descriptor.
int  create_server (const char* hostname, std::uint16_t port, int backlog);

// Spawn a process and redirect its standard streams
// to socket (stderr, stdin, stdout)
void spawn_redirect (int sockfd, const char* executable);

// Send text message to socket
void send_msg (int sockfd, const char* message);

// Connect to remote server and return client socket file descriptor.
int socket_connect(const char* hostname, std::uint16_t port);

// Handle session between server and client.
void session_handler(int sockfd, const char* executable);

void set_raw_terminal(int pty_fd);

// -------------  MAIN() ------------------------------//

int main(int argc, char** argv)
{

    if(argc != 4)
    {
        fprintf(stderr, " Usage: $ %s client <HOSTNAME> <PORT> \n", argv[0]);
        fprintf(stderr, " Usage: $ %s server <HOSTNAME> <PORT> \n", argv[0]);
        return EXIT_FAILURE;
    }

    int sockfd = -1;
    auto command  = std::string{ argv[1] };
    auto hostname = std::string{ argv[2] };
    int port      = std::stoi(argv[3]);

    const char* shell_executable = "bash";

    // Run as client mode => Forwarding the shell to the server.
    if(command == "client") for(;;) {
          sockfd = socket_connect(hostname.c_str(), port);

          if( sockfd > 0 )
               std::fprintf(stderr, " [OK   ] Connected to server. \n");
          else
                 std::fprintf(stderr, " [ERROR] Conncection failure. Retry. \n");

          session_handler(sockfd, shell_executable);
          sleep(2);
    } // ---- End of client command ---- //

    // Run in server mode => Liste connection.
    if(command == "server")
    {
        sockfd = create_server(hostname.c_str(), port, 5);
        assert( sockfd != -1 );

        // ============= Server Event Loop =============//
        for(;;){

            struct sockaddr_in client_sa;
            socklen_t addrlen = sizeof(sockaddr_in);

            fprintf(stderr, " [TRACE] Waiting for client connection. \n");
            int sock_client = accept(sockfd, (sockaddr *) &client_sa, &addrlen );

            if(sock_client == -1)
            {
                fprintf(stderr, " Error: failure to handle client socket. Check errno \n");
                close(sock_client);
                continue;
            }

            session_handler(sock_client, shell_executable);

        } // --- End of for(;;) server loop ------//

    } // --- End of server command ---//


    // Release resource => RAII (Resource-Acquisition-Is-Initialization) fits here.
    close(sockfd);
    return 0;

 } //  --- End of main() ----- //


// =============================================//
//          D E F I N I T I O N S               //
//==============================================//

int create_server(const char* hostname, std::uint16_t port, int backlog)
{
    // --- Create socket file descriptor -----------------------//

    int sockfd = socket ( AF_INET, SOCK_STREAM, 0);

    // Future error handling.
    // Returns (-1) on failure
    assert( sockfd != - 1);

    // ---------- query address ------------------------------//
    // Note: the keyword 'struct' is not necessary here (redundant).
    struct hostent* h = gethostbyname(hostname);
    assert(h != nullptr);

    struct sockaddr_in sa;
    memcpy(&sa.sin_addr.s_addr, h->h_addr, h->h_length);
    sa.sin_family = AF_INET;
    // The function htons convert a number to big-endian format.
    sa.sin_port   = htons(port);

    // ----- Bind to Port and wait for client connections ---------//
    if( ::bind(sockfd, (sockaddr *) &sa, sizeof(sa)) == -1)
    {
        fprintf(stderr, "Error: unable to bind socket \n");
        return EXIT_FAILURE;
    }
    // Enables binding to the same address
    int enable_reuseaddr = 1;
    if ( setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &enable_reuseaddr, sizeof(int)) < 0 )
    {
        fprintf(stderr, "Error: unable to set socket option. \n");
        return EXIT_FAILURE;
    }

    fprintf(stderr, " [TRACE] Listening client connection \n");
    assert( listen(sockfd, backlog) != -1 );

    return sockfd;
}

int socket_connect(const char* hostname, std::uint16_t port)
{
    int sockfd =  socket (  AF_INET, SOCK_STREAM, 0);
    // assert() => Placeholder for future error handling.
    assert( sockfd != UNIX_FAILURE );

    // Query DNS
    struct hostent* h = gethostbyname(hostname);
    assert( h != nullptr );

    struct sockaddr_in sa;
    std::memcpy(&sa.sin_addr.s_addr, h->h_addr, h->h_length);
    sa.sin_family = AF_INET;
    sa.sin_port   = htons(port);

    int res = connect( sockfd
                       , reinterpret_cast<sockaddr*>(&sa)
                       , sizeof(sockaddr_in));

    if( res == -1){ return -1; }
    return sockfd;
}


void send_msg(int sockfd, const char* message)
{
    send(sockfd, message, strlen(message), 0);
}

void session_handler(int sockfd, const char* executable)
{
    char name[500];
    memset(name, 0x00, 500);

    // File descriptor of master pseudo terminal (PTY) pair.
    // The master pseudo terminal is connected to the parent process.
    int pty_master = -1;

    // File descriptor of slave pseudo terminal (PTY) pair.
    // The slave pseudo terminal is connected to the child process (shell).
    //
    int pty_slave  = -1;

    assert( openpty(&pty_master, &pty_slave, &name[0], nullptr, nullptr) != -1 );

    fprintf(stderr, " [TRACE] Slave  PTY fd = %d ; file = %s \n", pty_slave,  ttyname(pty_slave));
    fprintf(stderr, " [TRACE] Master PTY fd = %d ; file = %s \n", pty_master, ttyname(pty_master));

    int flag = 1;
    // Note: ioctl() returns 0 on success.
    assert( ioctl(pty_master, TIOCPKT, &flag ) != -1);
    // assert( ioctl(pty_master, FIONBIO, &flag ) != -1);
    set_raw_terminal(pty_slave);

    struct winsize win;
    assert( ioctl(pty_master, TIOCGWINSZ, &win) != -1);
    fprintf(stderr, " [TRACE] Window size of cols = %d ; rows = %d \n", win.ws_col, win.ws_row);

    // Fork a child process
    pid_t pid = fork();

    switch (pid)
    {
    case -1:
        std::fprintf(stderr, "Error: unable to fork process. ");
        exit(EXIT_FAILURE);
        break;
        // ----------- Child Process ------------//
    case 0: {

        std::fprintf( stderr, " [TRACE] Spawning process (PID = %d ) pty = %s ; shell: %s \n"
                      , ::getpid(), name, executable);

        // Turn the current terminal into the session leader of the child process.
        assert( setsid() > 0);

        // Redirect process (stdin, stdout, stderr) to the pseudo terminal slave device (PTY).
        assert( dup2(pty_slave, STDOUT_FILENO) != -1);
        assert( dup2(pty_slave, STDIN_FILENO ) != -1);
        assert( dup2(pty_slave, STDERR_FILENO) != -1);

        const char* args [] = {
            executable    // Process name (arbitrary)
            , nullptr     // Sentinel value - (always NULL pointer)
        };

        // Spawn a new process.
        int status = execvp( executable, const_cast<char* const*>(args) );
        if( status == UNIX_FAILURE ){
            int err = errno;
            send_msg(sockfd, " [SERVER ERROR] Unable to locate REPL \n");
            send_msg(sockfd, strerror(err));
            close(sockfd);
            exit(EXIT_FAILURE);
        }

    }
        break;
        //----------- Parent Process ------------//
    default:

        // Buffer size in bytes.
        constexpr size_t BUFFER_SIZE = 1024;
        char buffer[BUFFER_SIZE];

        constexpr  size_t N = 2;
        struct pollfd pollv[N];

        // Register socket and pseudo-terminal master side.
        pollv[0].fd      = sockfd;
        pollv[0].events  = POLLIN;
        pollv[0].revents = 0;
        pollv[1].fd      = pty_master;
        pollv[1].events  = POLLIN;
        pollv[1].revents = 0;

        // --------- Connection session loop -----------------------------//
        //
        for(;;)
        {
            // Block current thread until any file descriptor becomes
            // ready for read operation.
            int nfds = ::poll( pollv, N, -1);
            if(nfds < 0){
                fprintf(stderr, " [TRACE] An error has happened. Abort. ");
                std::terminate();
            }


            // Event that happens when socket becomes ready (counterparty sends a message)
            if( pollv[0].revents & POLLIN ) {
                size_t n = read(sockfd, buffer, BUFFER_SIZE);
                if(n == 0){
                    fprintf(stderr, " [TRACE] Peer closed connection \n");
                    close(pty_master);
                    break;
                }
                // Echoe command output to the server terminal
                write(STDOUT_FILENO, buffer, n);

                // Write socket remote input (stdin) to the master pseudo-terminal device (PTY)
                assert( write(pty_master, buffer, n) > 0);
            }

            // Event that happens when the process connected to the slave-end of pseudo
            // terminal (PTY) sends a message.
            if( pollv[1].revents & POLLIN ) {
                size_t nn = read(pty_master, buffer, BUFFER_SIZE);
                assert( nn > 0 );

                // Relay process output from master pseudo terminal to the socket file descriptor.
                assert( write(sockfd, buffer, nn) > 0);
            }

        } // --- End of session loop ------//

        break;
    }
}


// Set pseudo terminal device in raw mode (not cooked mode, default)
// The algorithm was extracted from the following Python code:
//  =>>  https://github.com/python/cpython/blob/711381dfb09fbd434cc3b404656f7fd306161a64/Lib/tty.py#L18
void set_raw_terminal(int pty_fd)
{
    // Saved terminal settings
    struct termios tty_old;

    // Current terminal settings
    struct termios tty_new;

    // Save the default parameters of the slave side of the PTY
    int rc = tcgetattr(pty_fd, &tty_old);
    assert( rc != -1 );

    tty_new = tty_old;
    cfmakeraw (&tty_new);
    tty_new.c_iflag = tty_new.c_iflag & ~(BRKINT | ICRNL | INPCK | ISTRIP | IXON) ;
    tty_new.c_oflag = tty_new.c_oflag & ~(OPOST);
    tty_new.c_cflag = tty_new.c_cflag & ~(CSIZE | PARENB);
    tty_new.c_cflag = tty_new.c_cflag | CS8;
    tty_new.c_lflag = tty_new.c_lflag & ~(ECHO | ICANON | IEXTEN | ISIG);
    tty_new.c_cc[VMIN] = 1;
    tty_new.c_cc[VTIME] = 0;

    tcsetattr (pty_fd, TCSANOW, &tty_new);
    // tcsetattr (pty_slave, TCSANOW, &tty_new);
}

Building

This application is able to be build on Linux, BSD-variants or MacOSX. In this case it was compiled on a MacOSX VM: 

$ clang++ rshell.cpp -o rshell-osx.bin -std=c++1z -Wall -Wextra

$ file rshell-osx.bin
rshell.bin: Mach-O 64-bit executable x86_64

$ otool -L rshell-osx.bin
rshell.bin:
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 902.1.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)

Running

Terminal 1: Run netcat as server in a Linux machine: 

$ nc -l 9056

The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.
bash-3.2$
bash-3.2$ uname -a
Darwin ghosts-iMac-Pro.local 19.5.0 Darwin Kernel Version 19.5.0: ... ... 

bash-3.2$ pwd
/Volumes/data/macosx

bash-3.2$ ls /
Applications    Volumes         etc             sbin
Library         bin             home            tmp
System          cores           opt             usr
Users           dev             private         var
bash-3.2$


bash-3.2$ osascript -l JavaScript -i

>> ObjC.import("Foundation")
ObjC.import("Foundation")
=> undefined
>> ObjC.import("Cocoa")
ObjC.import("Cocoa")
=> undefined
>>

>> console.log("Hello world - Hola mundo - Ola mundo")
console.log("Hello world - Hola mundo - Ola mundo")
Hello world - Hola mundo - Ola mundo
=> undefined
>>

>> $.NSLog("Mac OSX Objective C is awesome => GNUSTep")
$.NSLog("Mac OSX Objective C is awesome => GNUSTep")
2020-12-22 14:19:01.001 osascript[8194:730902] Mac OSX Objective C is awesome => GNUSTep


>> z = $.NSNumber.numberWithFloat(10.2323)
z = $.NSNumber.numberWithFloat(10.2323)
=> $(10.2322998046875)

>>  $.NSLog(" z = %@", z)
 $.NSLog(" z = %@", z)
2020-12-22 14:20:47.164 osascript[8194:730902]  z = 10.2323


>>  alert = $.NSAlert.alloc.init
 alert = $.NSAlert.alloc.init
=> [id NSAlert]

>>  alert.messageText = "Hello world Alert button"
 alert.messageText = "Hello world Alert button"
=> "Hello world Alert button"

>> alert.runModal(); $.NSLog(" Finished")
alert.runModal(); $.NSLog(" Finished")

Terminal 2: Run rshell as client in a MacOSX VM: 

$  ./rshell-osx.bin client 10.0.2.2 9056 
[OK   ] Connected to server. 
[TRACE] Slave  PTY fd = 5 ; file = /dev/ttys021 
[TRACE] Master PTY fd = 4 ; file = (null) 
[TRACE] Window size of cols = 0 ; rows = 0 
[TRACE] Spawning process (PID = 8191 ) pty = /dev/ttys021 ; shell: bash 
 ... ..   ... ..   ... ..   ... ..   ... ..   ... .. 
  ... ..   ... ..   ... ..   ... ..   ... ..   ... ..   ... ..

1.29 Sockets with SSL/TSL - Secure Communication

1.29.1 Overview

Many widely network protocols, such as HTTP, FTP, Telnet or SMTP, were designed without any security considerations, without any type of encryption which makes the data transmitted by those protocols vulnerable to eavesdropping; tampering, the data can be changed by third party during the transmission; spoofing (impersonation); MIT (Man-In-The Middle attack) a third-party actor can impersonate both sides and modify the data sent over the connection; and integrity violation.

The TLS (Transport Layer Security) and its predecessor SSL (Secure Socket Layer) are the bed rocks of internet security, they are network protocol stacks designed for encapsulating the TCP protocol and solve its major security shortcomings by providing confidentiality through encryption, a third party will only be able to view obfuscated non-readable binary data (blobs); non-repudiation and authentication through certificates and public key infrastructure (PKI); data integrity as property of encryption, both sides of communication channel are able to detect any data tampering.

The TSL protcol stack is comprised of basically two parts, the TLS _handshake protocol, that takes care of authentication, certificate verification and shared key generation and exchange; and the TLS record protocol, that takes care of encapsulating the TCP protocol and providing encrypted communication. When the connection is opened, the TSL hanshaking takes place and the following sequence of events happen: both sides negotiate the TLS version; decides the ciphers that will be used; the client checks server certificate by using a trusted certificate authority; the server may also check the client certificate, but this feature is semdom used; generate a shared encryption key (symmetric encryption) for exchanging encryoted data. After the TLS handshake, the TLS record protocol is used for encapsulating TCP-based protocols, such as HTTP, FTP or SMTP by providing data encryption by using the generated shared (symmetric) key, which is unique for every connection.

Remarks

  • TLS and SSL are essential for internet security. Without it any third party agent, including ISP (Internet Service Providers), would be able to intercept and view the communication between the web browser and web servers.
  • Despite being widely used for providing encryption for the http protocol ( https:/www.site.com URLs, instead of http://www.site.com), the TLS protocol can also encapsulate any other TCP-based network protocol.
  • APIs based on OpenSSL or other implementation of the TLS protocol stack provide encapsulation and transparent encryption and decryption for TCP-based protocols where the calling code does not need to care about encryption details.
  • The private key should never be disclosed, otherwise any third party agent, that obtained the key, will be able to intercept and decrypt the communication.
  • Always keep openSSL library updated with the latest security fixes. This library is a fundamental piece of internet security. Any OpenSSLL vulnerability will make anything that depends on this library vulnerable.
  • Avoiding creating custom or proprietary encryption as cryptography is business critical and requires deeper knowledge of number theory, random numbers, discrete mathematics and abstract algebra.

Some network protocols that relies on TLS/SSL

  • HTTPS (HTTP + SSL/TLS), port 443 => Http protocol encapsulated with TLS/SSL. Web sites with https URL, for instance https://www.httpbin.org, are using TLS.
  • FTPS (FTP + SSL/TLS) => FTP encapsulated with TLS/SSL
  • OpenVPN
  • SSTP (Secure Socket Tunneling Protocol)
  • DoH (DNS over TLS)

Some Implementation of TLS and SSL Protocol Stack

  • OpenSSL
    • Reference-implementation and first implementation of the SSL and TLS network protocol stack. In addition to the shared library provided by OpenSSL suite, it also provide command line tools for creating and validating certificates; creating self-signed certificates and check connection certificate; creating digital signature and so on.
    • Note: OpenSSL is cross-platform and can be used on all platforms, including Windows.
    • Note: In many cases, web applications use OpenSSL indirectly through Nginx reverse proxy where Nginx is used for providing TLS security.
  • GNU TLS
    • License: LGPL
    • GNU Implementation of TLS protocol stack.
  • LibreSSL
    • Fork of OpenSSL created by OpenBSD developers.
  • BoringSSL
    • Google's implementation of TLS/SSL.
  • Mbed TLS (Embedded Systems)
    • Lightweight implementation derived from Mbed operating system for embedded systems.
    • Repository: GitHub - ARMmbed/mbedtls
  • wolfSSL (Embedded Systems)
    • License: Dual license, GPL3 and commercial for closed source applications.
    • Highly secure implementation, designed for embedded systems. Unlike other libraries, WolfSSL has certification for meeting regulatory requirements.
    • Repository: https://github.com/wolfSSL/wolfssl
  • Botan
    • License: BSD
    • Repository: https://github.com/randombit/botan
    • "Botan's goal is to be the best option for cryptography in C++ by offering the tools necessary to implement a range of practical systems, such as TLS protocol, X.509 certificates, modern AEAD ciphers, PKCS#11 and TPM hardware support, password hashing, and post quantum crypto schemes. A Python binding is included, and several other language bindings are available. It is used in many open source and commercial products. The library is accompanied by a featureful command line interface."
  • Rust TLS
    • TLS protocol stack implemented in Rust that conforms to OpenSSL API.
  • BearSSL - Main
    • "BearSSL is an implementation of the SSL/TLS protocol (RFC 5246) written in C. It aims at offering the following features: Be correct and secure. In particular, insecure protocol versions and choices of algorithms are not supported, by design; cryptographic algorithm implementations are constant-time by default. Be small, both in RAM and code footprint. For instance, a minimal server implementation may fit in about 20 kilobytes of compiled code and 25 kilobytes of RAM. Be highly portable. BearSSL targets not only “big” operating systems like Linux and Windows, but also small embedded systems and even special contexts like bootstrap code. Be feature-rich and extensible. SSL/TLS has many defined cipher suites and extensions; BearSSL should implement most of them, and allow extra algorithm implementations to be added afterwards, possibly from third parties."
  • Schannel [WINDOWS-ONLY]
    • Microsft Windows-only implementation of SSL and TSL protocol stack.

Useful Terminology and Concepts

  • IETF - Internet Engineering Task Force
  • RFC - Request For Comment
  • SSL - Socket Secure Layer (predecessor of TLS, first used by Netscape Web Browser)
  • TLS - Transport Layer Security (over TCP) - provides security for TCP based protocols, such as HTTP, FTP, SMTP and so on.
  • DTLS - Datagram Transport Layer Security (over UDP) - provides security for UDP based protocols.
  • PKI - Public Key Infrastructure
  • Non-Repudiation (Used for digital signature and data integrity)
    • If the message recipient has the data received from the sender, the data hash created by the sender's private key, the sender's public key and the receiver is really sure about the public key ownership validated by a CA - Certificate Authority. The sender, owner of the private key, cannot deny that he or she really sent the message and cannot deny anything about the message content.
  • Shared-Key Cryptography (a.k.a Symmetric Key Cryptography)
    • The same key is used for data encryption and decryption. The usage of shared key cryptography is unsafe for communication over an unsafe channels as the encryption keys need to be send over an unsafe channel before the encrypted communication takes place.
    • Applications and use-cases: bulk data encryption for hiding and obfuscating data from unauthorized access; password file encryption; file system encryption; disk encryption; encrypted backup and private key encryption. Shared key cryptography can be used for encrypted communication. However, before the encrypted communication takes place, the key may be transmitted in an unencrypted channel which makes the key vulnerable to interception.
  • Public-Key Cryptography (a.k.a Asymmetric key cryptography)
    • Unlike the shared key cryptography, where the same key is used for data encryption and decryption. The public-key cryptography uses the public key for encryption and private key for decryption. The publick-key cryptography allows communication between two agents without them exchanging their encryption keys (private keys), they only need the public key from each other. Given two agents Alice and Bob, if Alice wants to send a encryted message to Bob, she encrypts the message with Bob's public key and he will be able to decrypt the encrypted data using his private key, which only he owns and knows. If Bob wants to send an encryted message, he encrypts the message with Alice's public key and Bob will be able to decrypt the message using his private key. Although, the public-key cyrptography is safer than shared-key cryptogaphy, public-key algorithms are more computationally expensive than shared key algorithms. Public key cryptography is not enough for secure communication. A third party agent can still impersonate Bob and Alice by sending a bogus Alice's public key to Bob and a bogus Bob's public key to Alice. Now the agent can intercept and decrypt the communication between both sides. This situation is called MITM (Man-In-The-Middle) attack. The MITM could be avoided if Bob was sure that Alice's public really belongs to her and Alice was sure that Bob's public key really belongs to him. What gives the assurance about the public key ownership and identity bound to the key is the PKI public-key infrastructure or the CAs (Certificate Authorities).
    • Applications and use-cases: secure communication over unsafe channels without disclosing private keys; digital signature for legal documents and contracts; data integrity detection, any data modification can be detected by both sides; authentication using the private key, if the server has the client's public key; blockchain is based on public-key cryptography; cryptocurrencies such as Bitcon or Litecoin are also based on public key cryptography; TLS and SSL uses public-key cryptography for exchanging the shared key without disclosing it.
  • X.509 Certificates - The certificate encapsulates is standardized format by RFC 5280 that encapsulates the identification metadata, server or client public key and digital signatures from certificate authorities for validating the identification linked to the certificate. In order to a TLS communication to be established, at least one side needs a TLS certificate, which is in most of the cases only the server. However, client-isde certificate can also be used for client authentication.
  • CHF - Crytography Hash Function => Cryptography hash are deterministic, non-reversible and one-way functions that uniquely map any amount of data, taken as input, to a fixed-width output, named hash value, message-digest or checksum. Some of those algorithms are: MD5, SHA-1, SHA-2, SHA-3, SHA-256, Argon-2, PBKDF2, Bcrypt and so on. Some applications of hash functions are:
    • Uniquely identifying a file: The computed hash value is unique for each file, as a result it uniquely identifies a file. For instance, when comparing two files at different machines, it is faster to compare their hash values rather than downloading the counterpart file and checking the equality of all bytes.
    • File integrity checking: given a file and its previously computed hash, an user can verify if the file is not corrupted or was not tampered by comparing the previously computed hash value and the current file hash value. If the computed value and the already known hash matches, then the file was not currupted and not tampered.
    • Data Checksum: if a data is transmistted with its checksum (hash value), it is possible to detect any data change or corruption by computing the received data hash value (checksum) and comparing with the received hash value.
    • Secure password storage for web applications. If only the hash value of user passwords are storead and the database is leaked, whoever accesses the hashes will not be able to reuse it on other websites.
  • Digital Signature
    • The digital signature of a file or data is produced by generating the hash of this file or data through some hash algorithm and encryting the hash with the sender's private key. Then, the data and its digital signature are sent to the recipient which can check the message data integrity and that the messsage was relly sent by whom the recipient claims to be. The recipient can verify the message by peforming the following process: first decrypt the digital signature with the sender's public key obtaining the message hash; finally the it is compared with the message hash computed by the recipient. If the expected hash and computed hash values match, then the message was really sent by the who the sender claims to be.
    • Most cryptography applications, including PGP, GPUPG, TLS software suits already check the public key automatically.
    • While digital signature ensures data integrity, the non-repudiation assurance which is only possible if the recipient is really sure that public key really belongs to the sender.
    • The recipient can sure about the public key ownership if the following conditions are met: he or she has the sender's public key; the public key is signed by a certificate authority that is trusted by both sides; and the recipient has the public key of the certifcate authority and he or she is really sure about the authenticity of the certificate authority public key.
  • CA - Certificate Authority (PKI - Public Key Infrastructure)
    • Provides non-repudiation, it allows the receiver of the message, for instance a Web browser, to be sure about sender, which could be a the http server, identity and the sender is really the intended communication counterparty. The CA chain also solves the MITM problem. Web browsers use a built-in set of certificate authorities trusted by the browser vendor, which are used for validating certificates from web sites (http servers).
  • SRP - Secure Remote Password
  • CRL - Certificate Revocation List
  • RSA - Rivest–Shamir–Adleman public-key algorithm.
  • ECC - Elliptic Curve Cryptography
  • ECDSA - Elliptic Curve Digital Signature Algorithm
  • MAC - Message Authentication Code
  • HMAC - Keyed-Hashing for Message Authentication
  • DES - Data Encryption Standard
  • BER - Basic Enconding Rules (Related to X509 Certificates)
  • DER - Distinguished Encoding Rules (Related to X509 Certificates)
  • PEM - Privacy Enhanced Mail
  • AES - Advanced Encryption Standard
  • CBC - Cipher Block Chaining Mode
  • OFB - Output FeedBack Mode
  • ECB - Electronic Code Book Mode

Public Key Cryptography Worflow for secure communication 

                     Message from Bob to Alice 


                Bob side                                           Alice side
        .................................                       ..........................................
                                        .                       .
Unencrypted         +-------------+     .                       .      +--------------+  Unencrypted  
(plain text)        |  Alice's    |     .   Encrypted data      .      |  Alice's     |  data (plain text)    
-------->>>-------->+ Public      +--------->>>--------->>>-----.---->>+ Private key  +--------->>>
                    |    Key      |     .   (channel)           .      |              |
                    +-------------+     .                       .      +--------------+
                        |               .                       .              |
                Bob encrypts the        .                       .      Alice decrypts the                 
                message to Alice        .                       .      message using her
                using her public key.   .                       .       private key


                     Message from Alice to Bob 


                Bob side                                            Alice side
        .................................                       ..........................................
                                        .                       .
Unencrypted         +-------------+     .                       .      +--------------+  Unencrypted  
(plain text)        |  Bob's      |     .   Encrypted data      .      |  Bob's       |  data (plain text)    
---<<----------<<---+  private     +---------<<---------<<<-----.--<<--+ public key   +--------<<<----
                    |    Key      |     .                       .      |              |
                    +-------------+     .                       .      +--------------+
                        |               .                       .              |
                Bob decrypts the        .                       .      Alice encrypts the                 
                message using his       .                       .      message using Bob's
                private key.            .                       .      public key.

SSL/TLS Workflow for secure communication 

      Client-Side                                       Server Side 
          |                                                 |
          |   (1) Intiates the communication                |
          +-------------->>>>-------------------------------+
          |                                                 |
          |  (2) Check server certificate and its           |
          |      digital signatures signed by               |
          |      certificate authorities.                   |
          |      Performs negotiation (TLS handshake)       |
          +-----<<<-----------------<<<---------------------+
          |                                                 |
`         |                                                 |
          |  (3) Generate session key (shared key)          |
          |      and sends it to server                     |
          |                                                 |
          |   +------------+   Encrypted               +------------+   Decrypted 
          |   |   Sever's  | session key (shared key)  |   Sever's  | Session key 
          +--->  PUB. KEY  +-------->>>--------------->+  PRIV. KEY +-------->>
          |   |            |                           |            | 
          |   +------------+                           +------------+
          |                                                 |
          |                                                 |
          |  Message from client to server                  |
          |  is encrypted with the                          |
          |  session key                                    |                                                                              
          +--------------->>>>---------------------------->>+
          |                                                 |
          |  Message from server to client                  |
          |  is encrypted with the                          |
          |  session key.                                   |                                                                              
          +<<<----------<<<-------------<<------------------+
          |                                                 |
          |                                                 |
          |                                                 |

Digital Signature Workflow

Use-cases:

  • Ensuring that message (plain text, data or file) sent was not tampered (data integrity) and that it really was authored by the holder of the private key corresponding to the public key on the recipient-side.
  • Digital signature of legal documents for providing non-repudiation assurance and guarantee their authenticity.
  • Trusted execution or app notarization: Only applications with digital signature matching public keys (built-in) trusted by the system vendor, are allowed to run. Examples: MacOSX app notarization; Android notarization; digital signature of Windows executable; UEFI firmware and so etc.
  • Password-less login. Assuming that the server-side has the public key of the client-side. The server sends a challenge to the client. The challenge can be any random data. Then the client sends the data digital signature to the server, which checks whether it matches the data and the public key. If that is the case, then the user is authenticated and is allowed to sign in. If the digital signature fails, the user authentication fails.
  • Signature of TLS/SSL certificates by CA - certificate authorities.
  Bobs sends a message to its digital signature to Alice. 


                         Data, plain text
                      or file authored by Bob.
                                |
               +------<<--------+----->>--------+
               │                                │
               │                                │
               ▼                                │
               │                                │
        ┌──────▼───────┐                        │
        │     Hash     │                        │
        │   Function   │                        │
        │              │                        │
        └──────┬───────┘                        │
               │                                │
               │                                │
               ▼ Data hash                      │
               │                                │
               │                                │
        ┌──────▼───────┐                        │
        │ Bob's private│                        │
        │     key      │                        │
        │              │                        │ 
        └─────┬────────┘                        │  (message)
              │  digital signature              │ Unencrypted
              │                                 │   data
              │  Encrypted                      │
              │  data hash                      │
              │                                 ▼
              │                                 │
              ▼            Bob side (sender)    │
--------------│---------------------------------│---------
              │     Insecure channel            │
--------------│---------------------------------│----------
              │                                 │
              │           Alice side            │
              │           (Recipient)           │
        ┌─────▼─────────┐                 ┌─────▼──────────┐
        │ Bob's public  │                 │                │
        │     key       │                 │ Hash Function  │
        │               │                 │                │
        │               │                 │                │
        └─────┬─────────┘                 └─────┬──────────┘
              │                                 │
              │Data hash (A)                    │   Data hash
              │                                 │   computed by Alice (B)
              │                                 │
              │                                 │
              │                                 │
              └────>>>>────[ = ]───<<<──────────┘
                             |
                             | Compare hashes extracted from 
                             | digital signature (A)
                             | and the hash computed by Alice (B). 
                               If both hashes are equal, then Alice   
                               can be sure that the data was really authored
                                        by Bob and it was not tamperd.

Passwordless Challenge-Response Authentication with Public-Key Cryptography

  • Authentication refers to checking the user identity and only allowing access if the identity provided matches the user account. Although the implementation of password authentication method is simple, this authentication method has several shortcomings. Shorter passwords might be easy to guess and longer password are easy to forget. Adversaries could attempt to impersonate users by taking advantage of leaked credentials from web sites that fail properly store them as the same password may be used on many other web sites. Another drawback is that users can be tricked into disclosing them. A safer alternative to this authentication method is the challenge-response authentication with public-key. In this authentication method, the server has the user account's public key and when an authentication attempt happens, the server sends a challenge, that may be random generated data to the client-side which must send a valid digital signature of the data to the server. If the signature matches the data and the public key, access is granted, otherwise access is denied and the authentication attempt fails. The benefits of this approach over the password-based authentication are that: the user does not need to create and remember another unique password for every new subscription or account; no private data is sent to the server-side; there is no way to trick users into providing the password as no shared secret is ever sent over the network; replay attacks are not possible since the digital signature is unique on every authentication attempt. The disadvantage of this technique is that user may lose access if the private key is lost or someone else obtains the private key. The security of this technique can be improved even further if the private key is stored in a external hardware capable of generating digital signature of input data. The private key can also be encrypted using a shared key cryptography with passphrase or pin as the shared key.
  • Note: The difference between password and passphrase or pin is that passphrase and pin are never sent to the server-side.
  • Another variant of challenge-response authentication process, which is used by ssh, is the following workflow: 1 - the server has the account public key; 2 - the server generates a random data, encrypts it with the public key and sends the encrypted data to the client-side; 3 - in order to authenticate, the client-side must decrypt the data and send it back; 4 - the server grants access if the decrypted data sent by the client-side is identical to the generated data.
  • Examples:

Workflow: 


  Client-side                                      Server-side 
(has private key)                          (has public key or certificate)
      |                                                |
      |  (1) - Attempts to autenticate (login)         |
      |       provide username or account name         |
      +----------------->>>------------------------->>>+ 
      |                                                |
      |  (2) Server sends challenge (random number)    |
      | unique for every auth. attempt                 |
      |----<<-----------<<<----------------------------+
      |                                                |
     \ /                                              \ /
      |  (3) Signs the data and sends and              |
      |      sends the digital signature to server     |   
      +---------->>>-------------------->>----------->>+
      |                                                |
      |  (4)  If the digital signature matches         |
      |      the public key and plain-text data,       |
      |      access is granted.                        |
      +---<<-------------------------------------------+
      |                                                |
     \ /                                              \ /

1.29.2 Further Reading

Technical standards and specifications:

SSL/TSL Concepts:

Cryptography Concepts:

Digital Signature:

Challenge-Reponse authentication with public key:

OpenSSL API

DoH - DNS over HTTPS (DNS over TLS)

1.29.3 Open SSL commands

View certificate of some sever, given its hostname 

 $ >> openssl s_client -showcerts -connect www.httpbin.org:443 < /dev/null 

CONNECTED(00000003)
depth=2 C = US, O = Amazon, CN = Amazon Root CA 1
verify return:1
depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
verify return:1
depth=0 CN = httpbin.org
verify return:1
---
Certificate chain
 0 s:CN = httpbin.org
   i:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
-----BEGIN CERTIFICATE-----
MIIFbjCCBFagAwIBAgIQC6tW9S/J9yHIw1v8WOnMPDANBgkqhkiG9w0BAQsFADBG
MQswCQYDVQQGEwJVUzEPMA0GA1UEChMGQW1hem9uMRUwEwYDVQQLEwxTZXJ2ZXIg
Q0EgMUIxDzANBgNVBAMTBkFtYXpvbjAeFw0yMDAxMTgwMDAwMDBaFw0yMTAyMTgx
MjAwMDBaMBYxFDASBgNVBAMTC2h0dHBiaW4ub3JnMIIBIjANBgkqhkiG9w0BAQEF
 .... ... ...
 ........ 

qPGwissH0W8HgYWhKVISkN2ui55RgbHXQHYDX0uYgGK6iRMxHHlOR1vOqRgEvVGF
5g8CcK3EzritKaHkD6bf0pnSE/E7cjKXzgB4l+58dsNcIVo9YgID4xYGS6paetso
XHE=
-----END CERTIFICATE-----

  ... ... ... ... ... . 
  ... ... ... ... ... ... 

---
Server certificate
subject=CN = httpbin.org

issuer=C = US, O = Amazon, OU = Server CA 1B, CN = Amazon

---
No client certificate CA names sent
Peer signing digest: SHA512
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 5509 bytes and written 443 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: F63AEC9287D557EE6FE770BE0DB46CC0B26991704DEE81E9071377849356F0D6
    Session-ID-ctx: 
    Master-Key: 9FA0D73281EBBD07DEF167E5517207BDB80CCFC22D8C53DE9E0354D50A01B5279C51C3C55BE972497329E3E70BE73C43
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 43200 (seconds)
    TLS session ticket:
    0000 - d0 62 5a 94 f7 f7 a8 e1-c9 40 e9 ff bc ff e1 d2   .bZ......@......
    0010 - 02 08 f4 aa 80 60 d7 e3-1f 85 1e d1 ba e3 b8 75   .....`.........u
    0020 - 20 8f 7f 21 62 b9 72 a1-23 f5 05 8d 61 db b2 2d    ..!b.r.#...a..-
    0030 - 20 71 38 0d 70 30 30 3d-98 ed 64 b1 66 85 75 eb    q8.p00=..d.f.u.
    0040 - a6 e2 fe 13 9d 86 aa f4-43 42 be b3 86 2c a2 30   ........CB...,.0
    0050 - 7c 90 f1 1b 84 b8 58 31-72 3e c2 92 83 d2 4a 33   |.....X1r>....J3
    0060 - a1 76 f7 f4 c1 f1 6a a9-bc ce f7 c4 b6 81 28 2f   .v....j.......(/
    0070 - 76 10 c1 ca a7 5f cf 16-0e 40 ee 09 ab b9 2b d1   v...._...@....+.
    0080 - 05 85 ed 48 30 2b 84 2c-f4 1d b6 61 08 b5 66 ff   ...H0+.,...a..f.
    0090 - c6 3c d3 ac e4 bb 8a de-70 73 af d9 d0 e4 f4 65   .<......ps.....e
    00a0 - 1e 42 74 0d 8b 71 a7 a1-58 84 a3 d1 3b 4e 8c 24   .Bt..q..X...;N.$
    00b0 - 64 46 2f ce 5a a9 a3 a0-64 23 46 36 73 92 50 2f   dF/.Z...d#F6s.P/

    Start Time: 1607966129
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---
DONE

Generate certificate

  • Generates the private key mykey.pem, which should never be disclosed, and the certificate mycert.pem.
$ openssl req -x509 -nodes -days 5000 -newkey rsa:1024 -keyout mykey.pem -out mycert.pem

View Certificate 

$ openssl x509 -text -in mycert.pem

More commands at:

1.29.4 TCP/IP Client Socket with TLS (OpenSSL)

This sample code contains a TCP/IP client socket using TLS encapsulation for secure communication.

OpenSSL Documentation:

All codes and files available at:

File: CMakeLists.txt 

cmake_minimum_required(VERSION 3.9)
project(Sample_SLL)

#========== Global Configurations =============#
#----------------------------------------------#

set(CMAKE_CXX_STANDARD 17)     
set(CMAKE_VERBOSE_MAKEFILE ON )

#----------- Set targets -------------------------------#

find_package( OpenSSL REQUIRED )

       add_executable( ssl-test ssl-test.cpp    )
target_link_libraries( ssl-test ${OPENSSL_LIBRARIES} )

File: ssl-test.cpp 

#include <iostream>
#include <string>
#include <cstring> // Equivalent to #include <string.h> in C
#include <cassert> // assert() macro.

// ----- Unix/Linux headers -----// 
#include <unistd.h>
#include <fcntl.h>
#include <sys/types.h>    
#include <sys/socket.h>
#include <netdb.h>
#include <limits.h>
#include <tuple>

// ----- OpenSSL headers -------//
#include <openssl/ssl.h>
#include <openssl/bio.h>
#include <openssl/err.h>

constexpr int FAILURE     = -1;
constexpr int SSL_FAILURE = -1;

using SSL_Tuple = std::tuple<SSL_CTX*, SSL*>;

// Create and return a client socket and connecting to a hostname.
int socket_connect(const char* hostname, std::uint16_t port);

// Instantiate SSL data structures.
SSL_Tuple setup_ssl();

// Shows certificate information. 
void display_certificate(SSL* ssl);

int main(int argc, char** argv)
{
    if(argc < 3)
    {
        std::fprintf(stderr, " Usage: %s <HOSTNAME> <PORT> \n", argv[0]);
        return EXIT_FAILURE;
    }

    uint16_t    port     = std::stod(argv[2]);
    const char* hostname = argv[1];

    const char* trust_cert_path = X509_get_default_cert_dir();

    std::fprintf( stderr, " [TRACE] X509 Trust Certificate Directory: %s \n"
                , trust_cert_path);

    // =========== SSL/TSL Initialization ===============//
    //---------------------------------------------------// 

    // Requires C++ 17
    auto [ctx, ssl] = setup_ssl();

    //============ Connect to server =====================// 
    //----------------------------------------------------//

    int sockfd = socket_connect(hostname, port);

    // Encapsulate TCP protcol 
    SSL_set_fd(ssl, sockfd);

    if( SSL_connect(ssl) == SSL_FAILURE )
    {
        std::fprintf(stderr, " [ERROR] Failure to setup SSL/TSL connection");
        ERR_print_errors_fp(stderr);
        std::abort();
    }

    std::fprintf(stdout, " [INFO] Connected to the server Ok. ");

    // From OpenSSL library
    display_certificate(ssl);

    // Check certifcate 
    if( SSL_get_verify_result(ssl) == X509_V_OK ) {
        std::fprintf(stdout, " [INFO] Certificate validated. Ok. \n");
    } else { 
        std::fprintf(stdout, " [ERROR] Certificate validation. => May be self signed. \n");
        ERR_print_errors_fp(stderr);
    }


    // ========= Session Loop =============================//
    //-----------------------------------------------------//

    constexpr size_t BUFFER_SIZE = 500;
    // Null character '\0' initialized buffer 
    char buffer[BUFFER_SIZE] = {0}; 

    for(;;)
    {
        ssize_t size = SSL_read(ssl, buffer, BUFFER_SIZE);

        // Return 0 bytes when peer (server) closes connection.
        if(size == 0)
        {
            std::fprintf(stderr, " [INFO] Peer closed the connection. \n");
            break;
        }
        if(size == -1){
            std::fprintf(stderr, " [ERROR] Failed to read socket. \n");
        }
        std::string msg = " [SERVER] " + std::string(buffer, buffer + size);
        std::cout << msg << '\n';

        // Write back message to server.
        SSL_write(ssl, msg.data(), msg.size() );
    }

    // ========= Dispose Resources ========================//
    //-----------------------------------------------------//

    // Dispose resource (file descriptor)
    close(sockfd);
    // Release SSL context 
    SSL_CTX_free(ctx);
}

    //==================================================//
    //          I M P L E M E N T A T I O N S           // 
    //==================================================// 

int 
socket_connect(const char* hostname, std::uint16_t port)
{
    int sockfd =  socket (  AF_INET, SOCK_STREAM, 0);
    // assert() => Placeholder for future error handling.
    assert( sockfd != FAILURE );

    // Query DNS 
    struct hostent* h = gethostbyname(hostname);
    assert( h != nullptr );

    struct sockaddr_in sa; 
    std::memcpy(&sa.sin_addr.s_addr, h->h_addr, h->h_length);
    sa.sin_family = AF_INET;
    sa.sin_port   = htons(port); 

    int res = connect(  sockfd
                      , reinterpret_cast<sockaddr*>(&sa)
                      , sizeof(sockaddr_in));
    assert( res != FAILURE && "Failure to connect => Check errno. " );

    return sockfd;
}

SSL_Tuple
setup_ssl()
{
    SSL_load_error_strings();
    ERR_load_BIO_strings();
    OpenSSL_add_all_algorithms();
    SSL_library_init();

    // Instantiate SSL method 
    const SSL_METHOD* method = SSLv23_method();
    assert( method != nullptr && "Failure to load SSL method.");

    // Create SSL context (ctx)
    SSL_CTX* ctx = SSL_CTX_new( method );
    if( ctx == nullptr )
    {
        std::fprintf(stderr, " [ERROR] Failed to create SSL context \n");
        ERR_print_errors_fp(stderr);
        std::abort();
    }

    if( !SSL_CTX_set_default_verify_paths(ctx) )
    {
        std::fprintf(stderr, " [ERROR] Failure to set default trusted CA paths");
        ERR_print_errors_fp(stderr);
        exit(EXIT_FAILURE);
    }

    SSL* ssl = SSL_new(ctx);
    assert( ssl != nullptr );

    return SSL_Tuple{ ctx, ssl };
}

void 
display_certificate(SSL* ssl)
{
    X509* cert = SSL_get_peer_certificate(ssl);
    if( cert == nullptr )
    {
        std::fprintf(stderr, " [ERROR] Failed to get certificate \n");
        ERR_print_errors_fp(stderr);
        return;
    }    

    char* out  = nullptr;

    long version = X509_get_version(cert);
    std::fprintf(stdout, " [CERTIFICATE] version = %d \n", version);

    out = X509_NAME_oneline( X509_get_subject_name(cert), 0, 0);
    std::fprintf(stdout, " [CERTIFICATE] name = %s \n", out);
    free(out);

    out = X509_NAME_oneline( X509_get_issuer_name(cert), 0, 0);
    std::fprintf(stdout, " [CERTIFICATE] issuer = %s \n", out);
    free(out);

    X509_free(cert);
}

Install OpenSSL Dependency

Debian and Ubuntu Linux: 

$ sudo apt-get install libssl-dev

Fedora Linux: 

$ sudo dnf install -y openssl-devel.x86_64

Building

Building manually: 

$ >> git clone https://gist.github.com/bb884fd36b8930cb7358133c2afe9b56 proj && cd proj 
$ >> g++ ssl-test.cpp -o ssl-test -std=c++1z -Wall -Wextra -lssl -lcrypto

Building with CMake: 

 $ >> git clone https://gist.github.com/bb884fd36b8930cb7358133c2afe9b56 proj && cd proj 

 $ >> ls
cert.pem  CMakeLists.txt  key.pem  ssl-test.cpp

 $ >> cmake -H. -B_build -DCMAKE_BUILD_TYPE=Debug
 $ >> cmake --build _build --target all

 $ >> file _build/ssl-test 
_build/ssl-test: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), 
dynamically linked, interpreter /lib64/ld-linux-x86-64.so.
2, BuildID[sha1]=ff0d3e3765841a53a13dab7b700726fcbd92fe56, for GNU/Linux 3.2.0, with debug_info, not stripped

Generate a sample certificate for testing purposes

Create a certificate (cert.pem) and private key (key.pem) and answer the prompt. 

$ openssl req -x509 -nodes -days 5000 -newkey rsa:1024 -keyout key.pem -out cert.pem

View certificate: 

 $ >> openssl x509 -text -in cert.pem

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            57:5b:cc:50:94:fe:5f:e8:cb:7e:44:43:80:50:76:27:47:61:ba:7e
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = UK, ST = Bristol, L = Yorkshire, O = some company, OU = dummy department, CN = www.myserver.com, emailAddress = company@mail.co.de
        Validity
            Not Before: Dec  9 22:25:09 2020 GMT
            Not After : Aug 18 22:25:09 2034 GMT
        Subject: C = UK, ST = Bristol, L = Yorkshire, O = some company, OU = dummy department, CN = www.myserver.com, emailAddress = company@mail.co.de
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (1024 bit)
                Modulus:
                    00:f3:73:c4:8c:86:6e:d5:06:a7:86:5d:1d:2f:f3:
                    e7:7e:cd:f5:30:b5:c7:21:47:c2:8b:3f:f4:f3:09:
                    30:27:7e:5f:9d:a9:44:b9:5d:e4:50:4b:fe:d7:41:
                    d0:17:da:72:a6:99:f7:25:c5:e1:f1:1f:ee:47:69:
                    d0:c2:47:63:f7:f1:4a:ff:aa:3c:66:57:29:35:a4:
                    c8:9d:b2:cf:44:e4:c4:15:21:8b:4d:25:61:c3:ee:
                    c4:47:50:88:b6:19:d7:77:06:39:d8:6f:34:f0:51:
                    9f:86:78:2e:5c:d9:a5:e6:33:c4:29:a4:f8:8e:01:
                    c0:c7:27:fe:73:bf:ea:3d:2f
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                52:48:8A:A3:78:49:BD:65:A5:E2:C9:EC:86:FF:05:93:D1:7F:A4:C4
            X509v3 Authority Key Identifier: 
                keyid:52:48:8A:A3:78:49:BD:65:A5:E2:C9:EC:86:FF:05:93:D1:7F:A4:C4

            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         b8:ee:be:e6:17:5f:9f:be:61:9f:05:c8:d2:58:f7:86:70:84:
         5f:4b:0c:2a:e8:e5:19:03:20:62:6f:93:34:0c:65:af:68:ac:
         95:ac:33:84:7e:3e:04:54:69:26:a3:f6:60:61:25:38:f2:0c:
         24:94:fc:68:5a:f4:02:e1:61:8c:a7:bd:63:1d:39:e0:b9:f7:
         d7:77:ea:1e:5b:10:13:29:85:26:bf:a0:dc:e6:7d:4f:ff:73:
         26:e4:d7:92:f3:be:6c:5b:55:c9:6f:e3:4e:1a:a6:b0:a8:ce:
         59:e8:4c:21:b3:b7:03:7c:26:71:a4:61:8d:24:9b:cf:d8:d4:
         8a:4b
-----BEGIN CERTIFICATE-----
MIIDJDCCAo2gAwIBAgIUV1vMUJT+X+jLfkRDgFB2J0dhun4wDQYJKoZIhvcNAQEL
BQAwgaMxCzAJBgNVBAYTAlVLMRAwDgYDVQQIDAdCcmlzdG9sMRIwEAYDVQQHDAlZ
b3Jrc2hpcmUxFTATBgNVBAoMDHNvbWUgY29tcGFueTEZMBcGA1UECwwQZHVtbXkg
ZGVwYXJ0bWVudDEZMBcGA1UEAwwQd3d3Lm15c2VydmVyLmNvbTEhMB8GCSqGSIb3
DQEJARYSY29tcGFueUBtYWlsLmNvLmRlMB4XDTIwMTIwOTIyMjUwOVoXDTM0MDgx
ODIyMjUwOVowgaMxCzAJBgNVBAYTAlVLMRAwDgYDVQQIDAdCcmlzdG9sMRIwEAYD
VQQHDAlZb3Jrc2hpcmUxFTATBgNVBAoMDHNvbWUgY29tcGFueTEZMBcGA1UECwwQ
ZHVtbXkgZGVwYXJ0bWVudDEZMBcGA1UEAwwQd3d3Lm15c2VydmVyLmNvbTEhMB8G
CSqGSIb3DQEJARYSY29tcGFueUBtYWlsLmNvLmRlMIGfMA0GCSqGSIb3DQEBAQUA
A4GNADCBiQKBgQDzc8SMhm7VBqeGXR0v8+d+zfUwtcchR8KLP/TzCTAnfl+dqUS5
XeRQS/7XQdAX2nKmmfclxeHxH+5HadDCR2P38Ur/qjxmVyk1pMidss9E5MQVIYtN
JWHD7sRHUIi2Gdd3BjnYbzTwUZ+GeC5c2aXmM8QppPiOAcDHJ/5zv+o9LwIDAQAB
o1MwUTAdBgNVHQ4EFgQUUkiKo3hJvWWl4snshv8Fk9F/pMQwHwYDVR0jBBgwFoAU
UkiKo3hJvWWl4snshv8Fk9F/pMQwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0B
AQsFAAOBgQC47r7mF1+fvmGfBcjSWPeGcIRfSwwq6OUZAyBib5M0DGWvaKyVrDOE
fj4EVGkmo/ZgYSU48gwklPxoWvQC4WGMp71jHTnguffXd+oeWxATKYUmv6Dc5n1P
/3Mm5NeS875sW1XJb+NOGqawqM5Z6Ewhs7cDfCZxpGGNJJvP2NSKSw==
-----END CERTIFICATE-----

Run Netcat server as TLS server using the generated certificate

Netcat terminal emulator: 

 $ >> nc --verbose  --ssl-key key.pem --ssl-cert cert.pem --listen 9051
Ncat: Version 7.80 ( https://nmap.org/ncat )
Ncat: Listening on :::9051
Ncat: Listening on 0.0.0.0:9051
Ncat: Connection from 127.0.0.1.
Ncat: Connection from 127.0.0.1:60202.
User types message in server terminal, the client sends them back to server
 [SERVER] User types message in server terminal, the client sends them back to server
Turn on control valve 
 [SERVER] Turn on control valve
Shutdown cooling system
 [SERVER] Shutdown cooling system
Solve control equations
 [SERVER] Solve control equations
Type Ctrl + C at server terminal to shutdown the connection.^C⏎

TLS Client terminal emulator: 

$ ./build/ssl-test localhost 9051 
 [TRACE] X509 Trust Certificate Directory: /etc/pki/tls/certs 
 [INFO] Connected to the server Ok.  [CERTIFICATE] version = 2 
 [CERTIFICATE] name = /C=UK/ST=Bristol/L=Yorkshire/O=some company/OU=dummy department/CN=www.myserver.com/emailAddress=company@mail.co.de 
 [CERTIFICATE] issuer = /C=UK/ST=Bristol/L=Yorkshire/O=some company/OU=dummy department/CN=www.myserver.com/emailAddress=company@mail.co.de 
 [ERROR] Certificate validation. => May be self signed. 
 [SERVER] User types message in server terminal, the client sends them back to server

 [SERVER] Turn on control valve

 [SERVER] Shutdown cooling system

 [SERVER] Solve control equations

 [INFO] Peer closed the connection.

Attempt to connect to web sites using HTTPS - HTTP/TLS (port 443) 

 $ >> build/ssl-test www.yahoo.com 443
 [TRACE] X509 Trust Certificate Directory: /etc/pki/tls/certs 
 [INFO] Connected to the server Ok.  [CERTIFICATE] version = 2 
 [CERTIFICATE] name = /C=US/ST=California/L=Sunnyvale/O=Oath Inc/CN=*.www.yahoo.com 
 [CERTIFICATE] issuer = /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA 
 [INFO] Certificate validated. Ok. 
^C^C⏎  

 $ >> build/ssl-test www.bing.com 443
 [TRACE] X509 Trust Certificate Directory: /etc/pki/tls/certs 
 [INFO] Connected to the server Ok.  [CERTIFICATE] version = 2 
 [CERTIFICATE] name = /CN=www.bing.com 
 [CERTIFICATE] issuer = /C=US/O=Microsoft Corporation/CN=Microsoft RSA TLS CA 02 
 [INFO] Certificate validated. Ok. 
^C⏎

1.30 Sockets with Multiplexed IO APIs (poll, epoll) [DRAFT]

1.30.1 Overview

A recurrent need in network socket programming is to handle multiple incoming client socket connections that need to be managed simultaneously. For instance, if a blocking read operation is performed on a client socket from a http(web) server, the server will not be able to serve the remaining client sockets while the read operation is ongoing. Several solutions have adopted for solving this issue, such as:

  • Blocking IO (+) One process per connection
    • Peform fork() system call, creating a child process wich is a copy of current process and let it manage the socket file descriptor.
    • Drawbacks: Complex, expensive and slow. A process has lots of operating system resources associated with it, including virtual memory, file descriptors and so on. Another issue is the complexity from IPC - Inter-Process Communication.
  • Blocking IO (+) One thread per connection
    • Spawn/create a new thread for every connection.
    • Drawbacks: Although, this approach is less expensive than open-process-per-connection method, it is still expensive and non-scalable, specially for handling thousands of connections.
  • Blocking IO (+) Thread Pool
    • Reuse a set of pre-existing threads for managing each connection and avoid the overhead of creating and destroying too many threads.
    • Drawbacks: Long running connections should have a timeout in order to not block a thread pool and make it unable to server the remaining client socket requests. This method is less scalable than multiplexed IO.
  • Multiplexed IO [BEST]
    • Many operating systems provide multiplexed IO facilities, such as select, poll and epoll on Linux, keque on BSD-variants and Mac-OSX and IOCP on Windows. Multiplexed IO allow managing multiple file descriptors of any type in a single thread, consequently this feature allows handling multiple client socket connetions with far less overhead than using one thread for every connection. This type IO API is that makes the scalability of Nginx web server and NodeJS.
    • Multiplexed IO works in the following way: first the application register the file descriptors it is interested in monitoring; then the calling code calls a multiplexed IO function that blocks the current thread during some amount of time or indefinitely, the IO multiplexing function only returns when any of the file descriptors are ready to be read or written. The calling code handles each file descriptor by iterating over the data structure returned by the IO multiplexing function which contains the file descriptors events.
    • Note: It is possible to run multiple multiplexed-IO event loops in different threads for increasing responsiveness and scalability.
    • Drawbacks:
      • Event-driven with possible callback hell which can be avoided with user-space threads (aka: coroutines, goroutines, green threads).
      • Any operation which takes a significant time needs to be run in another thread, otherwise the threads handling the connection will be blocked and unable to server any other client request.
    • Wrappers:
      • Multiplexed IO APIs are not standardized. As a result, several high level wrappers have been created for providing an uniform and cross-platform API for multiplexed IO. Some of those wrapper libraries are:
        • Boost.ASIO - C++ library for scalable and cross-platform network programming.
        • libUV - C library used by NodeJS; Julia language and so on.
        • libev
        • libEvent
    • Note: Many documents also calls multiplexed IO as event-driven IO, non-blocking IO and so on.
    • Note: The EPool API cannot monitor file descriptors of regular files. For this case, the Linux kernel provides the iNotify API.

See:

1.30.2 Poll() multiplexed IO API

Poll is a standardized multiplexing IO API common to a wide variety of Unix-like operating systems, including Linux-based operating systems, BSD variants and MacOSX. Poll() API is able to monitor multiple file descriptors in a single thread without any wasteful polling. The API blocks the current thread for a certain amount of time and returns until any monitored file descriptor is ready for read or write IO operations.

Observations:

  • Multiplexed IO is often also called (there is no a standard terminology yet):
    • multiplexed IO (Input/Output)
    • asynchronous IO
    • async IO
    • event-driven IO
    • event-based IO
    • non-blocking IO (although, it really do blocks the current thread.)
  • Poll() is able to monitor:
    • Standard terminal IO file descriptors (STDIN_FILENO, STDOUT_FILENO, STDERR_FILENO)
    • socket file descriptors
    • terminal file descriptors
    • anonymous pipes
    • named pipes
    • character device files
  • Poll() is not able to monitor file descriptor for regular files. In this case, the poll() should be used alongside inotify (Linux-only).

Comparison with Epoll

  • The Poll() API is less scalable and less efficient than the Epoll() API since that, the Poll API requires client code iterating over all monitored file descriptors in order to find which ones are ready for IO operation. Despite Poll() API shortcomings, the advantage of this API over its comparison counterpat is the higher availability on a broad range of Unix-like operating systems such as MacOSX, BSD variants, QNX and so on. While Epoll API is only available on Linux kernel.

Header

  • #include <poll.h>

Functions

  • poll() => Blocks the current thread for a certain amount time until one or more file descriptor is ready for IO operations. If the timeout parameter is (-1), the calling thread is blocked indefinitely until some IO event happens.
int poll(struct pollfd *fds, nfds_t nfds, int timeout);

int poll( // Array containing monitored file descriptors.   
          // Note: Array alllocated by the caller. 
          struct pollfd* fds     
          // Number of monitored file descriptors                                     
          , nfds_t         nfds    
          // Timeout for which poll() blocks the current thread.
          // in milliseconds. If the timeout is (-1), the current 
          // thread is blocked indefinitely. 
          , int            timeout                                     
        );

Data Structures 

struct pollfd {
        // [INPUT] fd => Monitored file descriptor. 
        // can be set to (-1) when the calling code 
        // is no longer interested in the file descriptor. 
        int fd;

        // [INPUT] Flags (bitwise-OR) that selects the types of IO events that  
        // will be monitored for this file descriptor.
        short events;

        // [OUTPUT] Flags (bitwise-OR) which contains the IO events
        // that happened to the file descriptor. Those flags are 
        // set by the function poll(). 
        short revents;
};

Usage 

 constexpr size_t nfds = 10;

 // Caller allocates arrays containing the monitored file descriptors.
 struct pollfd pollv[nfds]; 
 memset(pollv, 0x00, sizeof(pollfd) * nfds);

 // === Caller register file descriptors and events it is interested ======// 
 // 

 // Register STDIN (Standard terminal input)
 pollv[0].fd     = STDIN_FILENO; 
 pollv[0].events = POLLIN | POLLER | POLLHUP; 

 // Register socket server 
 pollv[0].fd     = server_socket_fd; 
 pollv[0].events = POLLIN | POLLER | POLLHUP | POLLOUT; 

 ... .... ... ...  

//===================================================// 
//   Run event loop / IO-multiplexing loop           // 
//===================================================//

for(;;)
{
   // This thread is blocked indefintely until an IO event happens.
   int ret = poll(pollv, nfds, -1);

   // --- Check return value -------//
   if(ret == 0){
      fprintf(stderr, " [TRACE] Timeout \n");
      continue;
   }
   if(ret == -1){
      fprintf(stderr, " [ERROR] Non recoverable error happened. Check ERRNO. Abort() \n");
      exit(EXIT_FAILURE);
   }

   // Iterate over IO events 
   for(int n = 0; n < nfds ; n++)
   {
       auto& pfd = pollv[n];
       // Skip elements for which any event happened. 
       if(pfd.revents == 0){ continue; }

       if(pfd.fd == STDIN_FILENO && pfd.revents & POLLIN)
       {
           // .... User typed something in the console. 
           continue; 
       }

       // A remote machine (client-side) attempts to connect to server. 
       if(pfd.fd == server_socket_fd)
       {
           // .... Income request for connecting to server ... // 
           // Register client socket 
           continue;
        }

       // Event that happens when client socket is ready for read() operation. 
       if(pfd & POLLIN){
             // Read bytes from client socket file descriptor. 
             ssize_t n = read(pfd.fd, buffer, 500);
             if(n == 1){  perror("read() error"); exit(EXIT_FAILURE); }
             if(n == 0){  
                 fprintf(stderr, " [TRACE] Client socket closed connection. \n");
                 close(pfd.fd);
                 // The file descriptor is set to (-1) when the calling code 
                 // is no longer interested in monitoring the descriptor. 
                 pfd.fd = -1;
              }

             // ....  Write to STDOUT ... // 
             write(STDOUT_FILENO, buffer, n);
       }
   }

References

1.30.3 Poll() sample code

In the following sample code, the Poll() API is used for monitoring IO events from standard terminal input (stdin); socket server and connected client sockets. Anything typed by the user on ther terminal is immediately forwarded to the client sockets.

File: poll-server.cpp 

#include <iostream>
#include <string> 
#include <vector>
#include <cassert> 
#include <cstring>  // memcpy 

// ----- Unix/Linux headers -----// 
#include <unistd.h>
#include <fcntl.h>
#include <sys/types.h>    
#include <sys/socket.h>
#include <netdb.h>
#include <limits.h>
#include <sys/ioctl.h>

#include <poll.h>

// -------------  MAIN() ------------------------------//

struct sresult{
    std::string msg;
    ssize_t     size;
};

sresult read_string(int fd, size_t buffer_size);
void    write_buffer(int fd, const void* buffer, size_t size);
void    write_buffer(int fd, const void* buffer, size_t size);
void    write_string(int fd, std::string const& text);

constexpr int UNIX_FAILURE = -1;
constexpr int BLOCK_INFINITE_TIMEOUT = -1;


int main(int argc, char** argv)
{
    std::fprintf(stderr, " [TRACE] Process Unique Identifier  PID = %d \n", ::getpid());

    // ======= Server configuration =========// 
    std::uint16_t port     = 9056;
     const char*  hostname = "0.0.0.0";

    // ====== Create Socket File Descriptor =============// 

     int sockfd = socket (  AF_INET       // domain:   IPv4
                          , SOCK_STREAM   // type:     TCP 
                          , 0             // protocol: (value 0) 
                         );

    std::fprintf(stderr, " [TRACE] sockfd = %d \n", sockfd);

    // Placeholder for future error handling. 
    // Returns (-1) on failure 
    assert( sockfd != - 1);

    // Note: t 'struct' is redundant. 
     struct hostent* h = gethostbyname(hostname);
     assert(h != nullptr);
     struct sockaddr_in sa; 
     memcpy(&sa.sin_addr.s_addr, h->h_addr, h->h_length);
     sa.sin_family = AF_INET;
     sa.sin_port   = htons(port); 

     // ----- Bind to Port and wait for client connections ---------//
     // Enables binding to the same address 
     int enable_reuseaddr = 1;
     if ( setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &enable_reuseaddr, sizeof(int)) < 0 )
     {
         fprintf(stderr, "Error: unable to set socket option. \n");
         return EXIT_FAILURE;
     }

     if( ::bind(sockfd, (sockaddr *) &sa, sizeof(sa)) == -1)
     {
         fprintf(stderr, "Error: unable to bind socket \n");
         return EXIT_FAILURE;
     }

    int backlog = 5;
    assert( listen(sockfd, backlog) != UNIX_FAILURE && " Non-recoverable failure. Check errno. " );

    // ================ Server Main Loop ===============//

    // This heap-allocated array contains all file 
    // descriptors that will be monitored. 
    auto pollv = std::vector<struct pollfd>{};    

    struct pollfd pfd;

    // Register STDIN file descriptor 
    pfd.fd      = STDIN_FILENO;
    pfd.events  = POLLIN;
    pfd.revents = 0;
    pollv.push_back(pfd);

    // Register socket server file descriptor 
    pfd.fd      = sockfd;
    pfd.events  = POLLIN;
    pfd.revents = 0;
    pollv.push_back(pfd);

    std::cout << " [TRACE] pollv.size() = " << pollv.size() << '\n';

    // ======= Server Event Loop / IO Multiplexing Loop =======//
    //
    for(;;)
    {

        std::fprintf(stderr, " [TRACE] Waiting IO events. \n");

        // poll() Blocks the current thread until any of the file descriptors become ready 
        // for IO (Input/Output) operations. On failure, the function returns -1 setting ERRNO.
        // On success, the function poll() returns the number of file descriptors (nfds)
        // ready to be written or read. 
        // 
        int nfds = ::poll( &pollv[0], pollv.size(), BLOCK_INFINITE_TIMEOUT);
        std::fprintf(stderr, " [TRACE] File descriptors ready. OK => nfds = %d \n", nfds);        

        if(nfds == UNIX_FAILURE)
        {
            std::fprintf(stderr, " [ERROR] errno = %d / %s \n", errno, strerror(errno) );
            exit(EXIT_FAILURE);
        }

        assert(nfds > 0 && "Nfds supposed to be greater than zero.");

        // Iterate over ready file descriptors 
        for(auto& pf : pollv)
        {   
            // Skip entries for which there is no IO event.          
            if(pf.revents == 0){ continue; }

            // Event that happens when user types something in the console.
            if(pf.fd == STDIN_FILENO)
            {
                std::string line;
                std::getline(std::cin, line);
                std::fprintf(stdout, " [EVENT] User entered: %s \n", line.c_str());

                // Send STDIN message to all connected socket clients.
                for(int i = 2; i < pollv.size(); i++){
                    int fd = pollv[i].fd;
                    if(fd == -1){ continue; }
                    write_string(fd, " [STDIN] " + line + "\n");
                }

                // Skip the parts below this statement. 
                continue;   
            }

            // Event happens when a socket client attempts to connect. 
            // // if(pf.fd == sockfd && ((pf.revents & POLLIN) == POLLIN)  )
            if( pf.fd == sockfd )
            {   
                sockaddr_in client_sa;     
                socklen_t addrlen = sizeof(sockaddr_in);
                int client_fd = accept(sockfd, nullptr, nullptr);

                //if(client_fd == UNIX_FAILURE){ break; }
                assert( client_fd != UNIX_FAILURE && " Non recoverable failure. \n" );

                std::fprintf(stderr, " [TRACE] Received client connection => client_fd = %d \n", client_fd);     

                // Register client socket file descriptor 
                struct pollfd pfd;
                pfd.fd      = client_fd;
                pfd.events  = POLLIN | POLLHUP;
                pfd.revents = 0;
                pollv.push_back(pfd);            

                // Greet client socket
                write_string(client_fd, " [SERVER] Hi, client socket. Welcome to the server. \n\n");

                continue;
            }

            // Event that happens when any client socket sends a message.
            if(pf.revents & POLLIN)
            {
                auto [msg, size] = read_string(pf.fd, 500);            
                assert( size >= 0 && "  Number of bytes returned supposed to be grater or equal to zero " );

                // Socket returns 0 bytes when disconnected 
                if(size == 0){
                    std::fprintf(stdout, " [EVENT] Client socket fd = %d closed connection \n", pf.fd);
                    // Disable file descriptor 
                    close(pf.fd);
                    pf.fd = -1;
                    continue;
                }
                std::fprintf(stdout, " [EVENT] fd = %d ; message = %s ", pf.fd, msg.c_str());

                write_string(pf.fd, "\n [ECHO] fd = " + std::to_string(pf.fd) + " / " + msg + "\n");
                continue;
            }

        } // ---- End of for() pf iterations --------//

    } // -------- End of server pf loop -----------------------//

    close(sockfd);
    return 0;
}

    // ------------------------------------------------------//
    //          I M P L E M E N T A T I O N S                //
    //-------------------------------------------------------//

sresult
read_string(int fd, size_t buffer_size)
{
    // Create a string initialized with null characters 
    std::string str(buffer_size, 0x00);
    ssize_t n  = read(fd, &str[0], buffer_size);
    if(n > 0){ str.resize(n); }
    return sresult{ str, n };
}


void write_buffer(int fd, const void* buffer, size_t size)
{
   // Pre-condition 
   assert( size <= SSIZE_MAX  );
   size_t  len  = size;
   ssize_t nr   = -1;
   auto pbuf = reinterpret_cast<const std::uint8_t*>(buffer); 

  while(  len != 0 && ( nr = write(fd, pbuf, len) ) != 0 ) 
  {
      if(nr == -1 && errno == EINTR){ continue; } 
      if(nr == -1) { 
          throw std::runtime_error("An error has happened, check ERRNO ");
          break;
      }
      len  = len - nr; 
      pbuf = pbuf + nr;
   }
}

void write_string(int fd, std::string const& text)
{
    write_buffer(fd, text.data(), text.length());
}

Building: 

$ g++ poll-server.cpp -o poll-server.bin -std=c++1z -Wall -Wextra -g -O0

Running server in terminal 1: 

 $ >> ./poll-server.bin 
 [TRACE] Process Unique Identifier  PID = 434583 
 [TRACE] sockfd = 3 
 [TRACE] pollv.size() = 2
 [TRACE] Waiting IO events. 
 [TRACE] File descriptors ready. OK => nfds = 1 
 [TRACE] Received client connection => client_fd = 4 
 [TRACE] Waiting IO events. 
 [TRACE] File descriptors ready. OK => nfds = 1 
 [TRACE] Received client connection => client_fd = 5 
 [TRACE] Waiting IO events. 
 [TRACE] File descriptors ready. OK => nfds = 1 
 [EVENT] fd = 4 ; message = I am the client from the terminal 1
 [TRACE] Waiting IO events. 
 [TRACE] File descriptors ready. OK => nfds = 1 
  [EVENT] fd = 5 ; message = I am the client from the terminal 2
 [TRACE] Waiting IO events. 
This message typed in the server terminal (stdin) will be set to all client sockets.
 [TRACE] File descriptors ready. OK => nfds = 1 
  [EVENT] User entered: This message typed in the server terminal (stdin) will be set to all client sockets. 
 [TRACE] Waiting IO events. 
 [TRACE] File descriptors ready. OK => nfds = 1 
 [EVENT] fd = 4 ; message = Turn on valve control system right away.
 [TRACE] Waiting IO events. 
 [TRACE] File descriptors ready. OK => nfds = 1 
  [EVENT] fd = 5 ; message = Shutdown colling system right away.
 [TRACE] Waiting IO events. 
Operation failure (typed by user)
 [TRACE] File descriptors ready. OK => nfds = 1 
  [EVENT] User entered: Operation failure (typed by user) 
 [TRACE] Waiting IO events. 
 [TRACE] File descriptors ready. OK => nfds = 1 
 [EVENT] fd = 4 ; message = Typed CTRL + C after this message
 [TRACE] Waiting IO events. 
 [TRACE] File descriptors ready. OK => nfds = 1 
  [EVENT] Client socket fd = 4 closed connection 
 [TRACE] Waiting IO events. 
 [TRACE] File descriptors ready. OK => nfds = 1 
 [EVENT] fd = 5 ; message = Type Ctrl+C for disconnecting from server (terminal 2)
 [TRACE] Waiting IO events. 
 [TRACE] File descriptors ready. OK => nfds = 1 
  [EVENT] Client socket fd = 5 closed connection 
 [TRACE] Waiting IO events.

Running netcat client (TCP client socket) in terminal 2: 

$ nc localhost 9056
 [SERVER] Hi, client socket. Welcome to the server. 

I am the client from the terminal 1

 [ECHO] fd = 4 / I am the client from the terminal 1

 [STDIN] This message typed in the server terminal (stdin) will be set to all client sockets.
Turn on valve control system right away.

 [ECHO] fd = 4 / Turn on valve control system right away.

 [STDIN] Operation failure (typed by user)
Typed CTRL + C after this message

 [ECHO] fd = 4 / Typed CTRL + C after this message

^C

Running netcat client (TCP client socket) in terminal 3: 

$ nc localhost 9056
 [SERVER] Hi, client socket. Welcome to the server. 

I am the client from the terminal 2

 [ECHO] fd = 5 / I am the client from the terminal 2

 [STDIN] This message typed in the server terminal (stdin) will be set to all client sockets.
Shutdown colling system right away.

 [ECHO] fd = 5 / Shutdown colling system right away.

 [STDIN] Operation failure (typed by user)
Type Ctrl+C for disconnecting from server (terminal 2)

 [ECHO] fd = 5 / Type Ctrl+C for disconnecting from server (terminal 2)

^C

1.30.4 Epoll() multiplexed IO API

Epoll is a highly scalable and efficient Linux-only multiplexed IO API which allows monitoring an unlimted amount of file descriptors, and consequentely file descriptors related to regular files, device files and sockets. This API is that makes the scalability of NodeJS and Nginx web server possible.

Manpage:

Header file:

  • #include <sys/epoll.h>

Structs:

  • The events is a Or bitmaks, some of the flags are:
    • EPOLLIN => "The associated file is available for read(2) operations."
    • EPOLLOUT => "The associated file is available for write(2) operations."
    • EPOLLHUP => "Hang up happened on the associated file descriptor."
    • EPOLLET => "Requests edge-triggered notification for the associated file descriptor. The default behavior for epoll is level-triggered. See epoll(7) for more detailed information about edge triggered and level-triggered notification."
typedef union epoll_data {
    void        *ptr;
    int          fd;
    uint32_t     u32;
    uint64_t     u64;
} epoll_data_t;

struct epoll_event {
    uint32_t     events;      /* Epoll events */
    epoll_data_t data;        /* User data variable */
};

Functions:

  • epoll_create1(int) => Creates epoll object, returns epoll file descriptor.
    • "If flags is 0, then, other than the fact that the obsolete size argument is dropped, epoll_create1() is the same as epoll_create(). The following value can be included in flags to obtain different behavior:"
 int epoll_create1(int flags);

 // ------ Usage ---------------------//
 // Epoll file descriptor 
 int epfd; 
 if( (epfd = epoll_create1(0)) == -1 )
 { 
    perror("Unable to get epoll file descriptor. Check ERRNO variable."); 
    exit(1);
 }

// .... Use Epoll ....

// Close when no longer needed. 
close(epfd);
  • epoll_ctl() => Modify, remove or add file descriptors to the interest list, set of monitored file descriptors.
    • Doc: $ man epoll_ctl
    • Doc: Manpage
    • Some values of 'op':
int epoll_ctl(  int epfd                     // Epoll file descriptor (Epoll object)
              , int op                       // Operation to be performed 
              , int fd                       // File descritor 
              , struct epoll_event *event    // Pointer to epoll_event 
             );

//---------------------------------------------------//
// Usage example for registering file descriptor     //
// add file descriptor to interest list.             // 
// --------------------------------------------------//

int fd = STDIN_FILENO; // Stdin, console 
epoll_event ev; 
// Monitor when file descriptor is available for read operations.
ev.events  = EPOLLIN;
// Monitor when file descriptor is available for write operations.
ev.events  = EPOLLIN | EPOLLOUT;
ev.data.fd = fd; 
if( epoll_ctrl(epfd, EPOLL_CTL_ADD, fd, &ev ) == -1 )
{  perror("Error: failure. Check errno.");
   exit(1);
}

//---------------------------------------------------//
// Usage example for removing file descriptor        //
// --------------------------------------------------//
epoll_event ev; 
ev.events  = EPOLLIN;
ev.data.fd = fd; 
if( epoll_ctrl(epfd, EPOLL_CTL_DEL, fd,  ) == -1 )
{  perror("Error: failure. Check errno.");
   exit(1);
}
  • epoll_wait() - IO multiplexing function - blocks the current thread for some amount of time until an event related to a monitored file descriptor has happened.
    • Manpage: "wait for an I/O event on an epoll file descriptor"
    • epfd => Epoll file descriptor
    • events => (Output) Pointer to array containing the events related to the monitored file descriptors. This array is allocated by the calling code.
    • maxevents => Maximum number of monitored events (often the size of events array).
    • timeout => Number of milliseconds that epoll_wait() blocks the current thread. If this value is (-1), the calling thread is indefinitely until some IO event happens.
int epoll_wait(int epfd, struct epoll_event *events,
               int maxevents, int timeout);

Usage example of epoll_wait() with level triggered. 

// ============ Usage for servers ===================// 
// 
void Epoll_fd_option(int epoll_fd, int fd, int events)
 {
     epoll_event ev; 
     ev.events  = events;
     ev.data.fd = fd; 
     int res = epoll_ctl(epoll_fd, EPOLL_CTL_ADD, fd, &ev);
     if(res == -1){
         perror("Error: EPoll / unable to register file descritor");
     }    
}

constexpr size_t MAX_EVENTS = 10;     
int server = socket(AF_INET, SOCK_STREAM, 0);
 .. .... ... ...

// Create epoll object 
int epfd = epoll_create1(0);
epoll_event notifications[MAX_EVENTS];

// Monitor incoming connections 
Epoll_fd_option(epfd, server, EPOLLIN);

// Monitor stdin (console file descriptor)
Epoll_fd_option(epfd, STDIN_FILENO, EPOLLIN);

charr buffer[300];

for(;;) {
     // Current thread blocked until any event happens.
     // Returns number of evetns 
     int nfsd = epoll_wait(epfd, notifications, MAX_EVENTS, -1); 

     for(int i = 0; i < nfsd; i++)
     {
         epoll_event ev = notifications[i];

         if(ev.data.fd == server)
         {
              // Accept connection return a client socket 
              int client_fd = accept(server, (sockaddr *) &client.sa, &addrlen);
              Epoll_fd_option(epfdm, client_fd, EPOLLIN);
              continue;
         }

        // Client socket ready for reading operations.
        if(ev.events & EPOLLIN)
        {
            // Read 300 bytes from client socket file descriptor to buffer 
            int n = read(ev.dada.fd, buffer, 300);

            if(n == 0){ close(ev.data.fd); continue; }

            // Send back n bytes to client socket 
            write(ev.data.fd, buffer, n);
            continue;
        }

        if(ev.events & EPOLLOUT)
        { 
           // Perform write operations 
           ... ... .... ...                 
        }

     } // --- End for --- //

} // --- End for --- //

close(epfd);

1.30.5 Epoll() example - sample code

GIST:

Description:

  • Sample TCP/IPv4 network server using epoll() multiplexed IO API for managing STDIN (console input) and multiple client sockets in a single thread. The server also sends any message typed in server console to all connected clients.
  • Note: It uses level triggered IO.

File: multiplex-epoll-server.cpp 

// File: socket-echo-server.cpp
// Desc: Simple socket server with a single thread.

#include <iostream>
#include <string>
#include <vector>
#include <cassert>
#include <cstring> // memcpy

// ----- Unix/Linux headers -----//
#include <unistd.h>
#include <fcntl.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netdb.h>
#include <signal.h>

// Only available on Linux
#include <sys/epoll.h> 

struct Socket 
{
    int           sockfd; 
    sockaddr_in   sa;     
};

struct RecvMsg
{
    ssize_t     size;
    std::string msg;
};

void socket_close(Socket& sock);

void    fd_write(int fd, std::string const& text);
RecvMsg fd_read(int fd, size_t buffer_max = 500);

/** Make a server socket */
Socket socket_make_server(uint16_t port, const char* hostname = "0.0.0.0");
/** Accept connetion from a server socket and returns a client socket */
Socket socket_accept(Socket& server); 

/** Register file descriptor to be monitored by Epoll */
void Epoll_register_fd(int epoll_fd, int fd)
{
    epoll_event ev; 
    ev.events  = EPOLLIN;
    ev.data.fd = fd; 
    int res = epoll_ctl(epoll_fd, EPOLL_CTL_ADD, fd, &ev);
    if(res == -1){
        throw std::runtime_error("Error: EPoll / unable to register file descritor");
    }    
}

/** Remove file descriptor from monitoring list. */
void Epoll_remove_fd(int epoll_fd, int fd)
{
    epoll_event ev;
    int res = epoll_ctl(epoll_fd, EPOLL_CTL_DEL, fd, &ev);
}

/*  Multiplex IO - Blocks current thread until any of monitored file descriptors 
 *  has 
 */
int Epoll_wait(int epoll_fd, size_t max_events, std::vector<epoll_event>& events)
{
    // Returns number of events received 
    int nfd = epoll_wait( epoll_fd     // Epoll file descripotr 
                        , &events[0]   // [output] Array containing received events 
                        , max_events   // Maximum number of events 
                        , -1           // Timeout (-1) => Block indefinitely
                        );

    return nfd;
}


int main(int argc, char **argv)
{
    std::puts(" [INFO] Program started. ");

    // --- Create socket file descriptor -----------------------//
    Socket server = socket_make_server(9026, "0.0.0.0");

    constexpr size_t MAX_EVENTS = 10;     
    int epfd = epoll_create1(0);
    assert(epfd != - 1);
    auto events = std::vector<epoll_event>(MAX_EVENTS);

    // Register stdin 
    Epoll_register_fd(epfd, STDIN_FILENO);

    // Register socket server 
    Epoll_register_fd(epfd, server.sockfd);

    // Contains client sockets file descriptors 
    std::vector<Socket> clients; 


    // Server-accept infinite loop 
    for(;;)
    {
        std::printf(" [TRACE] Epoll waiting for events \n");

        int nfsd = Epoll_wait(epfd, MAX_EVENTS, events);            
        assert( nfsd != - 1);

        std::printf(" [TRACE] Epoll events arrived => nfsd = %d \n", nfsd);

        for(int i = 0; i < nfsd; i++)
        {
            auto ev = events[i];

            std::printf(" ----------------------------------------------------\n");

            if( (ev.events & EPOLLERR) || (ev.events & EPOLLHUP) )
            {
                std::printf(" [ERROR] Epoll error has happened. \n");
                continue;
            }

            // STDIN event => It happens after user types something and
            // hit RETURN on terminal.
            if( ev.data.fd == STDIN_FILENO )
            {
                std::printf(" [TRACE] Received STDIN event \n");
                std::string line;
                std::getline(std::cin, line);
                std::cout << " \n STDIN = " + line << "\n";

                // Send typed message to all client sockets
                for(Socket& cli: clients)
                    fd_write(cli.sockfd, " STDIN = " + line + "\n");

                // Quit server 
                if(line == "exit"){ 
                    std::puts(" [INFO] Shutdown server OK.");
                    return EXIT_SUCCESS;
                }
                continue;
            }

            // Event that happens when there is a new connection.
            if(ev.data.fd == server.sockfd)
            {                
                Socket client = socket_accept(server);        
                Epoll_register_fd(epfd, client.sockfd);

                std::printf(" [TRACE] Accept client connection => ID = %d \n", client.sockfd);                
                fd_write(client.sockfd,  "\n => [SERVER INFO] Hello world client side!! \n");
                // socket_nonblock(client);
                clients.push_back(client);
                continue;
            }

            // Event that happens when some client socket sends data. 
            // EPOLIN event means that the file descriptor is ready for input. 
            if(ev.events & EPOLLIN)            
            {
               int fd = ev.data.fd;
               std::printf("\n [TRACE] Receive client event => ID = %d \n", fd);
               RecvMsg resp = fd_read(fd, 500);                                             
               if(resp.size == 0)
               {
                   std::printf("\n [CLIENT ID = %d] =>> Disconnected from server. \n", fd);
                   close(fd);
                   continue;
               }
               std::printf("\n [CLIENT ID = %d] =>> size = %d ; msg = %s \n", fd, resp.size, resp.msg.c_str());
               fd_write(fd, " [SERVER ECHO] => ID = " + std::to_string(fd) + " => " + resp.msg);
            }

        }


    } /* [[ ============== END client loop =========== ]] */

    socket_close(server);
    return 0;

} // --- End of main() -------------//

// ----------- Function Implementations ----------------------------------------//

void 
socket_close(Socket& sock)
{
    ::close(sock.sockfd);
    sock.sockfd = -1;    
}


void fd_write(int fd, std::string const& text)
{
    ::write(fd, text.c_str(), text.size());
}

RecvMsg fd_read(int fd, size_t buffer_max)
{
    std::string buffer(buffer_max, 0x00);
    ssize_t n = read(fd, &buffer[0], buffer_max);
    buffer.resize(n);
    return RecvMsg{n, buffer};
}


Socket 
socket_make_server(uint16_t port, const char* hostname)
{
    int sockfd = socket(AF_INET, SOCK_STREAM, 0);
    // Returns (-1) on failure
    assert(sockfd != -1);

    // ---------- query address ------------------------------//
    // Note: the keyword 'struct' is not necessary here (redundant).
    struct hostent *h = gethostbyname(hostname);
    assert(h != nullptr);
    struct sockaddr_in sa;
    memcpy(&sa.sin_addr.s_addr, h->h_addr, h->h_length);
    sa.sin_family = AF_INET;
    // The function htons convert a number to big-endian format.
    sa.sin_port = htons(port);

    // ----- Bind to Port and wait for client connections ---------//
    if (::bind(sockfd, (sockaddr *)&sa, sizeof(sa)) == -1)            
        throw std::runtime_error("Error: unable to bind socket \n");


    // Enables binding to the same address
    int enable_reuseaddr = 1;
    if (setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &enable_reuseaddr, sizeof(int)) < 0)
        throw std::runtime_error("Error: unable to set socket option. \n");

    // Ignore SIGPIPE signal which would cause abnormal termination of socket server
    // Requires: #include <signal.h> header.
    signal(SIGPIPE, SIG_IGN);

    fprintf(stderr, " [TRACE] Listening client connection \n");
    int backlog = 5;
    assert(listen(sockfd, backlog) != -1);

    return Socket{ sockfd, sa };
}

Socket 
socket_accept(Socket& server)
{
    Socket client;
    socklen_t addrlen = sizeof(sockaddr_in);
    client.sockfd = accept(server.sockfd, (sockaddr *) &client.sa, &addrlen);

    if (client.sockfd == -1)
    {
        fprintf(stderr, " Error: failure to handle client socket. Check errno \n");
        close(client.sockfd);
    }
    return client;
}

Building: 

g++ multiplex-epoll-server.cpp -o out.bin -std=c++1z -Wall -Wextra

Running: (Server - terminal 1) 

 ./out.bin 

 [INFO] Program started. 
 [TRACE] Listening client connection 
 [TRACE] Epoll waiting for events 
 [TRACE] Epoll events arrived => nfsd = 1 
 ----------------------------------------------------
 [TRACE] Accept client connection => ID = 7 
 [TRACE] Epoll waiting for events 
 [TRACE] Epoll events arrived => nfsd = 1 
 ----------------------------------------------------

 [TRACE] Receive client event => ID = 7 

 [CLIENT ID = 7] =>> size = 19 ; msg = hello world server

 [TRACE] Epoll waiting for events 
 [TRACE] Epoll events arrived => nfsd = 1 
 ----------------------------------------------------
 [TRACE] Accept client connection => ID = 8 
 [TRACE] Epoll waiting for events 
 [TRACE] Epoll events arrived => nfsd = 1 
 ----------------------------------------------------

 [TRACE] Receive client event => ID = 8 

 [CLIENT ID = 8] =>> size = 30 ; msg = connect server from client 2 

 [TRACE] Epoll waiting for events 
message typed in stdin is sent to all client sockets.
 [TRACE] Epoll events arrived => nfsd = 1 
 ----------------------------------------------------
 [TRACE] Received STDIN event 

 STDIN = message typed in stdin is sent to all client sockets.
 [TRACE] Epoll waiting for events 
 [TRACE] Epoll events arrived => nfsd = 1 
 ----------------------------------------------------

 [TRACE] Receive client event => ID = 8 

 [CLIENT ID = 8] =>> size = 24 ; msg = from client 2 to server

 [TRACE] Epoll waiting for events 
 [TRACE] Epoll events arrived => nfsd = 1 
 ----------------------------------------------------

Running: (Client 1 netcat from terminal 2) 

 $ >> nc -v localhost 9026
Ncat: Version 7.80 ( https://nmap.org/ncat )
Ncat: Connection to ::1 failed: Connection refused.
Ncat: Trying next address...
Ncat: Connected to 127.0.0.1:9026.

 => [SERVER INFO] Hello world client side!! 
hello world server
 [SERVER ECHO] => ID = 7 => hello world server
 STDIN = message typed in stdin is sent to all client sockets.
from client 1 to server
 [SERVER ECHO] => ID = 7 => from client 1 to server
^C⏎

Running: (Client 2 netcat from terminal 3) 

$ nc -v localhost  9026
Ncat: Version 7.80 ( https://nmap.org/ncat )
Ncat: Connection to ::1 failed: Connection refused.
Ncat: Trying next address...
Ncat: Connected to 127.0.0.1:9026.

 => [SERVER INFO] Hello world client side!! 
connect server from client 2 
 [SERVER ECHO] => ID = 8 => connect server from client 2 
 STDIN = message typed in stdin is sent to all client sockets.
from client 2 to server
 [SERVER ECHO] => ID = 8 => from client 2 to server
^C

1.31 Ptrace - system call

1.31.1 Overview

Ptrace is common system-call implemented by many Unix-like operating systems, such as Linux and FreeBSD, which provides facilities for manipulating and observing other processes. The purpose of this sytem call is to support the implementation of debuggers such as GDB (GNU Debugger), that use this sytem call for examining, modifying and manipulating the memory of debugged processes. Besides GDB, the ptrace system call is also used by the other tools like ltrace, for tracing library calls, and by strace, for tracing system calls. Although, ptrace has similar capabilities among many different Unix-based operating systems, ptrace is not a standardized system call, therefore there are many specific details such as type signature and request parameters that are operating-system dependent.

Documentation

Ptrace capabilities

  • Attach to forked child process.
  • Attach to an already running process.
  • Read/write tracee process' virtual memory ( represented by the file /proc/<PID>/mem )
  • Get/Set tracee's process CPU registers.
  • Red/Write tracee's process virtual memory
  • Intercept and modify systems calls invoked by the monitored process.
  • Trace library calls
  • Set breaking points

Known usage

  • GDB (GNU Debugger)
  • strace => Utility for tracing system calls.
  • ltrace => Utility for tracing library calls.

Further Reading

1.31.2 Linux Ptrace-related functions

Function: waitpid():

  • Wait for state change of some process. This function returns (-1) on failure.
  • Manpage:
  • Headers:
    • #include <sys/types.h>
    • #include <sys/wait.h>
pid_t waitpid(  pid_t pid       // (Process ID) Unique numeric idefier of monitored process.
              , int*  wstatus   // Allocated by caller. 
              , int   options
            );

  // ==== U S A G E -  E X A M P L E S ============ //

 // Wait until any child process terminated. 
 // -------------------------------------------
 int status = 0;
 waitpid(-1, &status, 0); 

 // Check whether process has terminated normally. 
 //-----------------------------------------------
 int status = 0;
 waitpid(8922, &status, 0);    
 if( WIFEXITED(status) ){ /* Process terminated normally. */ }

Function ptrace():

  • Allows observing and manipulating other processes, such as forked child processes or processes already running.
  • Manpage:
  • Headers:
    • <sys/ptrace.h>
    • <linux/ptrace.h>
    • <linux/tracehook.h>
  • Parameters:
    • request => Determines the action to be performed.
    • pid => Pid of tracee (monitored) process.
    • addr => Address within the tracee virtual memory. This parameter is null (nullptr in C++, or NULL in C) for most of request parameters
    • data => Data send to tracee's process virtual memory. For most request parameters, this parameter is null.
    • RETURN
      • => Return value, returns (-1) on failure setting errno variable. The meaning of the return value depends on the request parameter that was used.
// Returns (-1) on failure. 
long  ptrace(  int    request  // Request or command 
             , pid_t  pid      // PID of tracee or monitored pro ess 
             , void*  addr     // Address within the tracee's process virtual memory.        
             , void*  data     // Data parameter, used by some Ptrace 
          );

Summary for some possible values for the ptrace's request parameters:

  • PTRACE_ATTACH
    • Attach to an already running process given its PID (Unique numeric id identifier)
  • PTRACE_DETACH
    • Detach from attached process.
  • PTRACE_TRACEME
    • Used for tracing a child process spawned with fork() library-call. (Note: fork() on Linux is not a system call)
  • PTRACE_PEEKDATA
    • Read data from tracee's process memory.
  • PTRACE_POKEDATA
    • Write/set data to some tracee's process location (addr parameter)
  • PTRACE_GETREGS
    • Obtain current CPU registers from tracee process.
  • PTRACE_SETREGS
    • Set tracee process' CPU registers.
  • PTRACE_SYSCALL
    • Used for tracing and modifying tracee process' system calls.

Struct: user_regs_struct

  • => Contains CPU registers (context) of monitored process. The fields struct 'user_regs_struct', which represents the CPU registers used by the monitored process, are dependent on the current ISA (Instruction Set Architecture) of current machine.
  • Headers:
    • <sys/user.h>
// Fields of user_regs_struct for x86-64 ISA (AMD64 ISA)
struct user_regs_struct
{
  __extension__ unsigned long long int r15;
  __extension__ unsigned long long int r14;
  __extension__ unsigned long long int r13;
  __extension__ unsigned long long int r12;
  __extension__ unsigned long long int rbp;
  __extension__ unsigned long long int rbx;
  __extension__ unsigned long long int r11;
  __extension__ unsigned long long int r10;
  __extension__ unsigned long long int r9;
  __extension__ unsigned long long int r8;
  __extension__ unsigned long long int rax;
  __extension__ unsigned long long int rcx;
  __extension__ unsigned long long int rdx;
  __extension__ unsigned long long int rsi;
  __extension__ unsigned long long int rdi;
  __extension__ unsigned long long int orig_rax;
  __extension__ unsigned long long int rip;
  __extension__ unsigned long long int cs;
  __extension__ unsigned long long int eflags;
  __extension__ unsigned long long int rsp;
  __extension__ unsigned long long int ss;
  __extension__ unsigned long long int fs_base;
  __extension__ unsigned long long int gs_base;
  __extension__ unsigned long long int ds;
  __extension__ unsigned long long int es;
  __extension__ unsigned long long int fs;
  __extension__ unsigned long long int gs;
};

1.31.3 Example - trace system calls of a running process

The following sample code, traces system and logs system calls performed by an already running process.

GIST:

File: ptrace-attach.cpp

  • Note A: the function ptrace_get_string() reads a string buffer (character array) from the tracee's process virtual memory. The buffer is defined by memory address the initial character and the buffer size as number of bytes.
  • Note B: the function ptrace_get_string_null obtains a null-terminated ('\0') character array from the tracee's process virtual memory.
#include <iostream>
#include <string>
#include <sstream> 
#include <cstring> // memcpy, memset, ...
#include <cassert> // assert() statement 
#include <vector>
#include <functional>
#include <map> 

// --- Unix-specific headers -----// 
#include <unistd.h> 
#include <sys/ptrace.h>
#include <sys/reg.h>
#include <sys/wait.h>
#include <sys/types.h>
#include <sys/syscall.h>
#include <sys/user.h>

#include <fcntl.h>

constexpr int FAILURE = -1;

enum class WaitResult
{
      STOPPED     = 0
    , TERMINATED  = 1
};

/** Ptrace wrapper that terminates the current process there is any error. */
long _ptrace(int request, pid_t pid, void* addr, void* data);

WaitResult     ptrace_wait_syscall( pid_t pid );
std::string      ptrace_get_string( pid_t pid, std::uintptr_t addr, std::size_t size );
std::string ptrace_get_string_null( pid_t pid, std::uintptr_t addr );

int main(int argc, char** argv)
{
    if(argc < 2)
    {
        std::fprintf(stderr, " Usage: $ %s  <PID> \n", argv[0]);
        return EXIT_FAILURE;
    }

    // Next line: throws exception. 
    pid_t pid  = std::stoi(argv[1]);

    // Attempt to attach to process. 
    _ptrace(PTRACE_ATTACH, pid, nullptr, nullptr);
    std::fprintf(stderr, " [TRACE] Attached to process. Ok \n");

    // Fix PTRACE sigrap problem. That happens after attaching to child 
    // process with PTRACE_ATTACH and the child process receives a SIGTRAP signal.
    // See: https://stackoverflow.com/questions/44630382
    _ptrace( PTRACE_SETOPTIONS, pid, nullptr, (void*) PTRACE_O_TRACECLONE);

    // Makes possible to distinguish system-call related stops from other
    // types of stops. 
    _ptrace( PTRACE_SETOPTIONS, pid, nullptr, (void*) PTRACE_O_TRACESYSGOOD);

    // CPU registers (context) of the monitored process 
    struct user_regs_struct regs; 

    /// ========== Ptrace event loop ==============//

    std::fprintf(stderr, " [TRACE] Start event loop. Ok. \n");
    for(;;)
    {
        // Intercept system call entry 
        if(  ptrace_wait_syscall(pid) == WaitResult::TERMINATED)
        { 
            std::fprintf(stderr, " [TRACE] (1) Monitored process has terminated. \n");
            break;
        }

        // Intercept system call exit 
        if( ptrace_wait_syscall(pid) == WaitResult::TERMINATED)
        {                  
            std::fprintf(stderr, " [TRACE] (2) Monitored process has terminated. \n");
            break;
        }

        // Get  CPU register for the current process. 
        _ptrace( PTRACE_GETREGS, pid, nullptr, &regs);

        // Get system call number
        int syscall = regs.orig_rax;

        // System call => open() 
        // int openat( int dirfd              // Register RDI (1st syscall arg.)
        //           , const char *pathname   // Register RSI (2nd syscall arg.) 
        //           , int flags              // Register RSI (3rd syscall arg.)
        //           );
        if(syscall == SYS_openat)
        {   
            auto ptr  = uintptr_t{regs.rsi};
            auto file = ptrace_get_string_null(pid, regs.rsi);  
            std::fprintf(stderr, " [TRACE] openat() system call found => file = %s \n", file.c_str());
            continue;
        }

        // => Note: Linux system calls for x64 is the System V x86-64/ calling convention. 
        // 
        // System call: write(   int fd        // register RDI (1st syscall argument)
        //                     , void* buffer  // register RSI (2nd syscall argument)
        //                     , size_t size   // register RDX (3rd syscall argument)
        //                    ) 
        if(syscall == SYS_write)
        {

            int       write_fd   = regs.rdi; // File descriptor of write system call 
            uintptr_t write_ptr  = regs.rsi; // Pointer to buffer.             
            size_t    write_size = regs.rdx; // Buffer size.              

            auto buffer_text = ptrace_get_string(pid, write_ptr, write_size);  

            std::fprintf( stderr
                            , " [TRACE] Write system (syscall = %d ) " 
                            " fd = %d ; buffer_ptr = 0x%x ; size = %zu ; \n => buffer_text = '%s' \n\n"
                            , syscall
                            , write_fd, write_ptr, write_size, buffer_text.c_str());    

            continue;        
        }   


    } // ====== End of ptrace event loop ====// 

    return 0;
} // --- End of main() ------- //

        // ============================================== //
        //       I M P L E M E N T A T I O N S            //
        //------------------------------------------------//

long 
_ptrace(int request, pid_t pid, void* addr, void* data)
{
    long r = ptrace( (__ptrace_request) request, pid, addr, data);
    if(r == -1){
        std::stringstream ss;
        ss  << " [PTRACE FAILURE] " 
            << " ; errno = " << errno 
            << " ; msg = '" << strerror(errno) 
            << "' \n";        
        throw std::runtime_error(ss.str());
    }
    return r;
}

WaitResult 
ptrace_wait_syscall(pid_t pid)
{
    long result;
    int  status;

    for(;;){
        result = _ptrace(PTRACE_SYSCALL, pid, nullptr, nullptr);
        waitpid(pid, &status, 0);

        // This predicate holds when the monitored process
        // has terminated.
        if( WIFEXITED(status) )
        { return WaitResult::TERMINATED; }

        // Credits and further reading:  
        //   => https://ops.tips/gists/using-c-to-inspect-linux-syscalls
        if( WIFSTOPPED(status) && WSTOPSIG(status) & 0x80 )
        { return WaitResult::STOPPED; }
    }
}

std::string
ptrace_get_string(pid_t pid, std::uintptr_t addr, std::size_t size) 
{
    constexpr size_t long_size = sizeof(long);

    union ptrace_data {
        long data;
        char bytes[long_size];
    };

    int   n      = 0;
    int   steps  = size / sizeof(long);
    auto  pdata  = ptrace_data{};
    auto  result = std::string(size + 1, 0);
    char* paddr  = &result[0];

    assert(result.length() == size + 1);

    while(n < steps)
    {
        pdata.data = _ptrace( PTRACE_PEEKDATA, pid
                            , reinterpret_cast<void*>(addr + n * long_size)
                            , nullptr
                            );
        std::memcpy(paddr, pdata.bytes, long_size);
        n++;
        paddr = paddr + long_size;

    }
    // '%' integer remainder 
    steps = size % long_size;

    if( steps != 0 ) {
        pdata.data = _ptrace( PTRACE_PEEKDATA, pid
                            , reinterpret_cast<void*>(addr + n * long_size)
                            , nullptr
                            );
        memcpy(paddr, pdata.bytes, steps);
    }

    result[size] = '\0';

    return result;
}

std::string
ptrace_get_string_null(pid_t pid, std::uintptr_t addr) 
{
    constexpr size_t long_size = sizeof(long);

    union ptrace_data {
        long data;
        char bytes[long_size];
    };

    int   nread  = 0;
    auto  pdata  = ptrace_data{};
    auto  result = std::string{""};

    std::memset(&pdata.bytes[0], 0x00, long_size * sizeof(char));

    for(;;)
    {
        pdata.data = _ptrace( PTRACE_PEEKDATA
                            , pid
                            , reinterpret_cast<void*>(addr + nread)
                            , nullptr
                            );                

        result.resize(result.size() + long_size + 1);
        char* pinit = &result[0];
        memcpy(pinit + nread, &pdata.data, 1 * sizeof(long));
        nread += long_size;

        //  If the null terminating character '\0' (0x00) is found
        // , then exit the current loop.
        for(size_t i = 0; i < long_size; i++)
            if( pdata.bytes[i] == '\0' ) { goto exit_loop; }
    } 

exit_loop:
    return result;
}

Building: 

$ git clone https://gist.github.com/339b00fc8ab1b3d1d46ed9167ccbaeeb gist && cd gist 
$ g++ ptrace-attach.cpp -o ptrace-attach.bin -std=c++1z -Wall -Wextra -g

(TERMINAL EMULATOR 1) => Run python3 interactive interpreter on terminal emulator 1, the run the patrace-attach.bin applicaton on terminal 2 and attach to the PID of Python interpreter. After the ptrace monitor application is attached to the current process, the ptrace monitor application will log open() and write() system calls performed by the python process. 

 $ >> python
Python 3.8.5 (default, Aug 12 2020, 00:00:00) 
[GCC 10.2.1 20200723 (Red Hat 10.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import os
>>> os.getpid() # =>> Get Process ID unique identifier and attach to it from other terminal
2073259

>>> print("Hello world Ptrace")
Hello world Ptrace

>>> import dbus

>>> fd = open("/tmp/logging.txt", "w")
>>> 

>>> fd = open("/tmp/logging.txt", "w")
>>> fd.write("Hello world Python logging"); fd.flush()
26
>>>

(TERMINAL EMULATOR 2) => Run ptrace-attach.bin on terminal emulator 2 and attach to the Python interpreter process. 

$ ./ptrace-attach.bin 2073259
 [TRACE] Attached to process. Ok 
 [TRACE] Start event loop. Ok. 

 [TRACE] Write system (syscall = 1 )  fd = 1 ; buffer_ptr = 0xf75f7010 ; size = 1 ; 
 => buffer_text = '
' 

 [TRACE] Write system (syscall = 1 )  fd = 1 ; buffer_ptr = 0xf751a6d0 ; size = 19 ; 
 => buffer_text = 'Hello world Ptrace
' 

 [TRACE] Write system (syscall = 1 )  fd = 1 ; buffer_ptr = 0xf75f7010 ; size = 4 ; 
 => buffer_text = '>>> ' 

  ... ... .. ...   ... ... .. ...   ... ... .. ... 
  ... ... .. ...   ... ... .. ...   ... ... .. ... 

 [TRACE] openat() system call found => file = /usr/lib64/python3.8/site-packages/dbus/__pycache__/__init__.cpython-38.pyc 
 [TRACE] openat() system call found => file = /usr/lib64/python3.8/site-packages/dbus 
 [TRACE] openat() system call found => file = /usr/lib64/python3.8/site-packages/dbus/__pycache__/_compat.cpython-38.pyc 
 [TRACE] openat() system call found => file = /usr/lib64/python3.8/site-packages/dbus/__pycache__/exceptions.cpython-38.pyc 
 [TRACE] openat() system call found => file = /usr/lib64/python3.8/site-packages/dbus/__pycache__/types.cpython-38.pyc 
 [TRACE] openat() system call found => file = /usr/lib64/python3.8/site-packages/_dbus_bindings.so 
 [TRACE] openat() system call found => file = /home/mxpkf8/dlang/dmd-2.092.0/linux/lib64/libdbus-1.so.3 
 [TRACE] openat() system call found => file = /etc/ld.so.cache 
 [TRACE] openat() system call found => file = /lib64/libdbus-1.so.3 
 [TRACE] openat() system call found => file = /home/mxpkf8/dlang/dmd-2.092.0/linux/lib64/libsystemd.so.0 
 [TRACE] openat() system call found => file = /lib64/libsystemd.so.0 
 [TRACE] openat() system call found => file = /home/mxpkf8/dlang/dmd-2.092.0/linux/lib64/librt.so.1 
 [TRACE] openat() system call found => file = /lib64/librt.so.1 
 [TRACE] openat() system call found => file = /home/mxpkf8/dlang/dmd-2.092.0/linux/lib64/liblzma.so.5 
 [TRACE] openat() system call found => file = /lib64/liblzma.so.5 
   ....    ....    ....    ....    .... 
   ....    ....    ....    ....    ....    .... 

 [TRACE] openat() system call found => file = /tmp/logging.txt 
 [TRACE] Write system (syscall = 1 )  fd = 1 ; buffer_ptr = 0xf75f7010 ; size = 4 ; 
 => buffer_text = '>>> ' 

  ... ... ... ... ... ... ... ... ... ... 
  ... ... ... .. ... ... ... ... .. ... .. . .. . . 

 [TRACE] Write system (syscall = 1 )  fd = 3 ; buffer_ptr = 0xf7643390 ; size = 26 ; 
 => buffer_text = 'Hello world Python logging' 

 [TRACE] Write system (syscall = 1 )  fd = 1 ; buffer_ptr = 0xf75f7010 ; size = 4 ; 
 => buffer_text = '>>> '

1.32 X11 - X Windows System

1.32.1 Overview

Most Unix-like operating systems, including Linux and BSD variants, except Apple's MacOSX, use X-Windows System, also called X11, which is a network-transparent window system developed at project Athena (MIT) in 1984 and nowadays it is led by the X.Org foundation.

The server or display server (XServer) draws windows and widgets on the human operator display and receives input events from mouse and keyboard on behalf of one or more local or remote client applications (XClients). This architecture enables network transparency, which is the ability to operate either locally or over network. In practical terms, network transparency makes possible to interact with windows from graphics applications (x11 clients) running on remote servers as they were local applications and local windows. A X11 client appplication communicates with the XServer by using the X Windows System (X11) network protocol via unix socket if the client application is running on the user machine or via TCP/IP sockets if the client application is running on a remote server.

Main parts:

  • X Server (Server-side, located at user's machine, draws on user's Monitor/Screen)
    • => Runs on the user (human operator) machine, draws windows and widgets for connected applications x-clients running on local or remote machines.
  • X Client => Application running on user machine or remote server.
    • => Initiates the connection and makes network requests to the server, which draws the application's widgets.
  • X11 - Network Protocol
    • => Techinical specification or technical standard about how x-server and x-clients should communicate.

The X-Client (local or remote application) can communicate with the X-Server by using:

  • Direct X11 protocol network requests via TCP/IP socket or Unix-sockets, in case of applications running on the local machine.
  • XLib C Library => Encapsulates the X11 network protocol.
  • XCb C Library => (Lower level than XLib) Similar to XLib, but has a smaller footprint and lower latency, better suport for thread.

Notes and considerations:

  • Unlike on Windows and Mac-OSX, the graphics interface is not part of neither Linux kernel nor BSD-variants' kernels. The ability to run headless, without any graphics, makes Linux-based operating systems suitable for network server and embedded systems where a built-in GUI could cause higher overhead and system footprint.
  • X11 protocol does not specify styling, widgets (menu, buttons and so on). Those features are provided by window managers and GUI Graphical User Interface toolkits such as, GTK, Qt, Motif, FLTK, and Java Swing (Java Only).
  • X11 is not widely used on Embedded Linux systems as it is on Linux desktop systems. Instead many embedded linux operate directly on top of Framebuffer. The following approaches can be used for creating graphical user interfaces on top of framebuffer: in-house libraries, EGL (Embedded OpenGL), SDL or Qt toolkit on top of frambebuffer.
  • Although Android operating system is based on Linux kernel, unlike the desktop GNU/Linux distribution counterparts, Android user interface is built directly on top of framebuffer.
  • It is not worth developing any application on top of the X-Windows System Protocol as it is low level, lacks widget abstractions and may eventually be replaced by Wayland on major Linux distributions. A better approach is to use higher level libraries such as Qt Widgets, Qt/Quick QML, KDE Libraries (Qt variants), Gtk, Gnome (Gtk variants) or Java Swing as any application built on top of those libraries will be able to work in any operating system using Wayland.
  • Despite being network transparent, X11 protocol is unsafe for usage over the internet due to the protocol be unencrypted which makes it vulnerable to MITM, snooping, spoofing and so on. A workaround for safer usage over the network is to use X11 via SSH (Secure Shell) forwarding. Note: On most Linux Distributions and BSD variants, X11 server does not use TCP/IP connections, instead it uses only Unix-domain sockets.
  • Another problem that affects X11 servers is the _lack of isolation of X11 clients. Any remote X11 client or a x11 client running a container connected to the current X11 server is able to take screenshots, capture type keys and list windows. Some solutions for this isolation issue are: 1 - Only connect to trusted remote x11 client applications; replace X11 with VNC or RDP - Remote Desktop and etc; use Xephyr x11 server that runs as a X11 window for isolating the current x11 server.

X11 Server and Clients 

                                          User Input 
                                      keyboard, mouse ... 
                                              | 
                                              |                                                  
                                             \ /                                                            
                                  +-------------------------+         
  Display <---- Framebuffer  <--- |  X11 Server             | <<----- (X Client 2)
(User's monitor)                  | on user local machine   |         Java Swing 
                                  |                         |         Application 
                                  +-----------+-------------+      running on remote 
                                             / \                  machine and connected
                                              |                   via TCP/IP socket.
                                              |
                                           (X Client 1)
                                          Firefox Browser 
                                      on user local machine                                                                                                     
                                  and connected via Unix domain sockets

See also:

X11 X Windows System

Wayland (Future Replacemente for X11)

X11 client alternative Implementations:

  • python-xlib (Python) - Alternative implentation for X11 client in pure Python.
  • xgb (Go, Golang) - Alternative implementation for X11 client in pure GO, aka Golang language.

1.32.2 X11 Utilities

The following X11 utilities are widely used on many Linux distributions for a wide variety of purposes, including configuration, window manipulation, debugging and desktop automation.

Authentication and authorization:

  • xinit - Program that allows starting Xorg (X Server) manually from command line.
  • xauth - Display X server authorization information.
  • xhost - Arch wiki: "The xhost program is used to add and delete host names or user names to the list allowed to make connections to the X server. In the case of hosts, this provides a rudimentary form of privacy control and security. It is only sufficient for a workstation (single user) environment, although it does limit the worst abuses. Environments which require more sophisticated measures should implement the user-based mechanism or use the hooks in the protocol for passing other authentication data to the server."

Monitor Configuration:

  • xrandr - utility for configuring screen/monitor orientation, resolution, dual monitor and so on. By default, it shows the current displays resolutions.
  • xcalib - Allows inverting monitor/screen color. Useful for avoiding eye strain due to long hours starring on screens.
    • Usage: $ xcalib -i -a
  • xrandr-invert-colors - Similar to xcalib, but works with multiple screens. Note: it is not available on most Linux distributions, thus this utility must be installed from source.

Window manipulation and Desktop Automation:

  • xdotool - Utility application for manipulating and automating x11 windows, similar to Microsft Windows' Autohotkey tool for desktop automation.
  • wmctrl - A command line tool to interact with an EWMH/NetWM compatible X Window Manager.
  • ydotool - Similar to xdotool, but for Wayland.
  • xclip - Copy and paste from/to terminal to clipboard.
  • Xpra - "Xpra is 'GNU Screen for X': it allows you to run X programs, usually on a remote host, direct their display to your local machine, and then to disconnect from these programs and reconnect from the same or another machine, without losing any state. It gives you remote access to individual applications."
  • xmessage => Graphical Message box notification
    • Examle: $ xmsessage -center "Some message"

General Utilities:

  • xterm - Reference implementation for X11 terminal emulators.
  • xkill - command line application that allows forcefully terminating any graphical program by calling $ xkill from command line and clicking at the window whose process will be terminated.
  • xdg-open - Open file or directory from command line with default system application. For instance, $ xdg-open file.xls, where file.xls is a Excel spreadsheet, opens the file with Libreoffice, if this application is available in the sytem.
  • xclip - Allows copying stdin contents to clipboard and pasting contents from clibpboard to stdout.

X11 Preference and configuration:

  • xset - X-Server user preference and display power management.
    • Disable display power management and stop screen from blanking:
      • $ xset dpms 0 0 0 && xset s noblank && xset s off
  • xmodmap - Modify keymaps and pointer buttons mapping in X Windows System X11.
  • setxkbmap - Modify Keyboard - allows changing the keyboard to English, Portuguese, Greek, Cyrillic and so on.
  • xinput - Configure and test X input devices such as keyboard, mouse, touchpad and etc.
  • xev (X event monitor) "Xev creates a window and then asks the X server to send it events whenever anything happens to the window (such as it being moved, resized, typed in, clicked in, etc.). You can also attach it to an existing window. It is useful for seeing what causes events to occur and to display the information that they contain; it is essentially a debugging and development tool, and should not be needed in normal usage."
  • xwinfo - Utility for getting display information about windows. It provides a cursor that shows information about any window that user clicks at.
  • xsetroot - Root window parametter settings.
  • xprop - Similar to xwinfo, but displays font property.
  • xvinfo - Display capabilities of video adapters.
  • xdriinfo - Get confituration of DRI (direct rendering interface) device drivers.

X11 protocol message tracing/monitor tools (Note: most non-standard tools, non commonly available in most distros.)

  • xscope - "XSCOPE is a program to monitor the connections between the X11 window server and a client program. xscope runs as a separate process. By adjusting the host and/or display number that a X11 client attaches to, the client is attached to xscope instead of X11. xscope attaches to X11 as if it were the client. All bytes from the client are sent to xscope which passes them on to X11; All bytes from X11 are sent to xscope which sends them on to the client. xscope is transparent to the client and X11.".
  • x11vis - "x11vis aims to be a useful tool for debugging X11 clients. Its main focus is on being as visual and interactive as possible. Replacing X11 atom, window, font or pixmap IDs with human readable names is the most basic functionality. Contrary to other tools like xtrace, x11vis will display human readable IDs on all requests, no matter at which point in time the information gets available. To help you cope with lots of data, x11vis displays only the most important information by default, automatically folds sequences of boring requests, allows you to filter by packet type or client and provides so-called markers between you can navigate. Also, instead of just assigning a number to each connection, x11vis displays the command line with which that client was started. To learn more, please see the manual.".
  • xmon - "A monitor for the X Window System core protocol. Xmon monitors the messages sent between an X11 server and a number of X11 clients and displays the contents of the messages in an interactively selectable level of detail."
  • xtruss - Tool for tracing/monitoring X11 network protocol.
  • Wireshark - Tool that allows debugging and tracing any network protocol and also allows inspecting USB protocol. (Note: scriptable in Lua programming language.)

Animation:

  • xclock - Clock animation.
  • xeyes - Eye animation that follows the mouse pointer.
  • xlogo - X windows System "X" logo picture.

1.32.3 Setting keyboard mode via setxkbmap

There many cases where there is the need for changing the keyboard settings, such as testing a Live-CD linux distribution, that oftens uses the American keyboard settings by default, even though the keyboard model is non-English; type in languages that does not uses the lantin script or practice foreign languages that uses a different type of keyboard configuration. The command line tool setxkbmap is useful for setting the keyboard language mode in X11 X Windows System environment.

Note: The setxkmap does not work on Wayland environment. This tool only works on X11 (X Windows System).

Some commands for changing keyboard layout:

  • Reset Capslock/Escape/Ctrl customization
    • $ setxkbmap -option
  • Swap Ctrl and Capslock (Friendly to Emacs users):
    • $ setxkbmap -option ctrl:swapcaps
  • Swap ESC (Escape) and Capslock (Friendly to Vi Users):
    • $ setxkbmap -option caps:swapescape
  • APL Keyboard Layout (for APL Programming Language)
    • $ setxkbmap apl
  • American English keyboard layout:
    • $ setxkbmap us
  • American English keyboard layout with dvorak keyboard:
    • $ setxkbmap us -variant dvorak
  • American and Greek keyboard layouts. (use-case: typing math formulas)
    • $ setxkbmap -layout us,gr -option grp:alt_shift_toggle
    • Type Shift + Alt for toggling the layout to Greek keyboard layout. This configuration is useful for typing math formulas.
  • American and Greek keyboard layout. (use-case: typing math formulas)
    • $ setxkbmap -layout us,gr -option grp:win_switch
    • Hold Windows-key and the corresponding key to type in Greek. For instance, by holding Windows-Key and typing A, the letter alpha (α) is typed instead of (a). If the same steps is applied to the letter (b), the letter beta (β) is typed. Finally, release the Windows key for switching back to American keyboard layout.
  • American English and APL keyboard Layout (use-case: type APL symbols)
    • $ setxkbmap -layout us,apl -option grp:win_switch
  • French language keyboard layout:
    • $ setxkbmap fr
  • German language keyboard layout.
    • $ setxkbmap de
    • Note: German umlaut letters (ö, ä, ü, ß) are hard to type without this layout.)
  • Greek language keyboard layout: (α, β, η, κ, λ, ς, ε, σ, Σ)
    • $ setxkbmap gr
  • APL keyboard Layout (For APL programming language)
    • $ setxkbmap apl
    • APL symbols, such as ⍺, ⌈, ⌊, ⊂, ⊃, ∩, ⊥, ⊤, ○,≠,∇ ,are hard to type without this layout.
  • Polish languag keyboard layout
    • $ setxkbmap pl
  • Bulgarian language keyboard layout (Cyrillic Script):
    • $ setxkbmap bg
  • Ukranian language keyboard layout (Cyrillic Script):
    • $ setxkmap ua
  • Russian language keyboard layout (Cyrillic Script):
    • $ setxkmap ru
  • Hebrew language keyboard layout:
    • $ setxkbmap il
  • Turkısh Language keyboard layout:
    • $ setxkbmap tr
  • Spanish language - Latin America keyboard layout:
    • $ setxkbmap latam
    • Makes easier to type Spanish grammar symbols (?, ¿, ¡, ñ)
  • Spanish/English keyboard layout:
    • $ /setxkbmap -layout latam,us -option grp:alt_shift_toggle
  • Portuguese language keyboard layout
    • $ setxkbmap br
  • Portuguese language and APL keyboard layouts, switching layout is similar to American keyboard.
    • $ setxkbmap -layout br,apl -option grp:win_switch
  • Portuguese language and Greek keyboard layouts (For typing math formulas)
    • $ setxkbmap -layout br,gr -option grp:win_switch
  • Portuguse/American English keyboard layout: (Use-case: programming)
    • $ setxkbmap -layout br,us -option grp:alt_shift_toggle
    • Note: The American English keyboard layout is more convenient for programming due to characters such as quotes, double quotes, colon, semicolon, slash (/ and \) and bar be closer to the left fingers.

See also:

1.32.4 Sample X11 Window via Xlib

This sample Xlib applications draw a window with a rectangle and a string. Note: Xlib lacks widgets.

Code available at:

File: CMakeLists.txt 

cmake_minimum_required(VERSION 3.9)
project(Simple_Cmake_Project)

#========== Global Configurations =============#
#----------------------------------------------#

set(CMAKE_CXX_STANDARD 17)     
set(CMAKE_VERBOSE_MAKEFILE ON)

find_package(X11 REQUIRED)

#========== Targets Configurations ============#

       add_executable( xlib-demo xlib-demo.cpp )
target_link_libraries( xlib-demo ${X11_LIBRARIES} )

# add_executable(app2 application2.cpp)

File: xlib-demo.cpp 

#include <iostream>
#include <string> 
#include <cstring>
#include <cassert> 

#include <X11/Xlib.h>
#include <cstdlib>

void abort_on_failure(bool predicate, const char* message = "Abort runtime.")
{
    if(predicate) return;
    std::fprintf(stderr, " [ABORT] %s", message);
    abort();
}

int main(int argc, char** argv)
{
    /** If the value passed to XOpenDisplay is null, the XClient (Xlib) attempts 
     * to connect to the display server designated by the environment variable  
     * $DISPLAY, which is often set to the :1.0 or the local machine. 
     *------------------------------------------------------------------------*/
    Display* dpy = XOpenDisplay(nullptr);
    abort_on_failure(dpy != nullptr, "Cannot open display.");

    /** ----- Query Information about display server ------------------------*/

    int version = XProtocolVersion(dpy);
    std::cout << " [INFO] X11 protocol version = " << version << '\n';

    const char* vendor = XServerVendor(dpy);
    std::cout << " [INFO] X11 XServer vendor = " << vendor << '\n';

    const char* disp_str = XDisplayString(dpy);
    std::cout << " [INFO] X11 Display string = " << disp_str << '\n';

    int n_screen = XScreenCount(dpy);
    std::cout << " [INFO] Number of screens = " << n_screen << '\n';

    /** ----- Draw Screen ------------------------------------------------*/

    int screen = DefaultScreen(dpy);

    Window wnd = XCreateSimpleWindow(
                     dpy                         // Display 
                   , RootWindow(dpy, screen)     // Parent Window 
                   , 10, 20                      // Position (x - horizontal, y - vertical) from top-left corner 
                   , 400, 500                    // Dimensions (width, height)
                   , 1                           // Border width 
                   , BlackPixel(dpy, screen)     // Border 
                   , WhitePixel(dpy, screen)     // Background color 
                   );

    /* Casting void means that the return value (type int) is being ignored. */
    (void) XStoreName(dpy, wnd, "X Windows-System X11");
    // Makes window visible 
    XMapWindow(dpy, wnd);
    XFlush(dpy);

    XEvent event;

    // Select events that the application is interested in. 
    // Only those events will notify the application. (XClient)
    auto mask  =   ExposureMask        
                | ButtonPressMask     // Event: mouse button press
                | KeyPressMask        // Event: key press (keyboard)
                | KeyReleaseMask      // Event: key release (keyboard)
                //| PointerMotionMask   // Event: Mouse Move
                ;
    XSelectInput(dpy, wnd, mask);

    // Process window close event in X11 event loop for(;;)
    Atom wnd_del = XInternAtom(dpy, "WM_DELETE_WINDOW", 0);
    XSetWMProtocols(dpy, wnd, &wnd_del, 1);

    const char* message = "Draw some string on X11 - X windows system.";

    /* ------- X11 (X Windows System) - Event loop ------------ */

    for(;;)
    {
         std::fprintf(stderr, "\n [TRACE] Waiting event. \n");;

         // Get next X11 event      
         XNextEvent(dpy, &event);

         if(event.type == ClientMessage){
                 fprintf(stderr, " [Event] User closed window. => Shutdown \n");
                 break;
         }

         if(event.type == Expose)    
                 fprintf(stderr,  " [Event] Expose event. \n");

         if(event.type == KeyPress)    
                 fprintf(stderr,  " [Event] Key pressed. \n");

         if(event.type == ButtonPress) 
                 fprintf(stderr,  " [Event] Mouse button pressed. \n");

         if(event.type == PointerMotionMask) 
                 fprintf(stderr,  " [Event] Mouse button pressed. \n");     

         // X11 Graphics context, useful for drawing 
         GC gctx = DefaultGC(dpy, screen);

         XFillRectangle( dpy, wnd, gctx
                        , 20, 50    // (x, y) position from top-left corner origin 
                        , 100, 200  // width and height 
                        );

         GC gctx2 = DefaultGC(dpy, screen);                    

         XDrawString(  dpy     // Pointer to display object 
                      , wnd     // Pointer to window 
                      , gctx    // Graphics context 
                      , 90, 300 // (x, y) Position where string will be drawn (from top-left corner)
                      , message
                      , strlen(message) ); 

    } // ----- End of event loop -------- // 

    fprintf(stderr, " [TRACE] User closed window. Shutdown application. Ok \n");

    XDestroyWindow(dpy, wnd);

    // Always close display 
    XCloseDisplay(dpy);

    return 0;
}

Building: 

 $ git clone https://gist.github.com/f7f0b892011ad987bfed531336dbae89 temp && cd temp 
 $ cmake -H. -B_build -DCMAKE_BUILD_TYPE=Debug -G Ninja
 $ cmake --build _build --target all 
[2/2] Linking CXX executable xlib-demo

Running: 

$ >> _build/xlib-demo 
[INFO] X11 protocol version = 11
[INFO] X11 XServer vendor = Fedora Project
[INFO] X11 Display string = :1.0
[INFO] Number of screens = 1

[TRACE] Waiting event. 
[Event] Expose event. 

[TRACE] Waiting event. 

[TRACE] Waiting event. 
[Event] Mouse button pressed. 
... ...     ... ...     ... ...     ... ...    
... ...     ... ...     ... ...

User Interface Screenshot:

x11-gui-sample1.png

Figure 6: Xlib GUI User Interface.

1.32.5 Query X11 information via Xlib

The following sample application uses Xlib for querying X11 windows, window information and terminating the Window process.

Documentation of relevant functions and further reading:

Files

All files available at:

File: CMakeLists.txt 

cmake_minimum_required(VERSION 3.9)
project(X11-Xtoll-Project)

#========== Global Configurations =============#
#----------------------------------------------#

set(CMAKE_CXX_STANDARD 17)     
set(CMAKE_VERBOSE_MAKEFILE ON)

find_package(X11 REQUIRED)

#========== Targets Configurations ============#

       add_executable( x11-tool x11-tool.cpp )
target_link_libraries( x11-tool ${X11_LIBRARIES} )

File: x11-tool.cpp 

#include <iostream>
#include <string> 
#include <cstring>
#include <cassert> 

// ------ Unix Headers (needed by read_symlink) ------// 
#include <unistd.h>
#include <limits.h>
#include <sys/types.h>
#include <signal.h>

// ------ X11 Headers  ------//
#include <X11/Xlib.h>
#include <X11/Xatom.h>

#include <cstdlib>

std::string  read_symlink      (std::string const& path);
int          x11_get_property_int32(const char* property_name, Display* dpy, Window root_window);
std::string  x11_get_property_str  (const char* property_name, Display* dpy, Window root_window);

template<typename Window_Consumer>
void x11_iterate_child_windows(  Display* dpy
                               , Window root_window
                               , Window_Consumer&& consumer);


std::string  x11_window_name(Display* dpy, Window wnd);
int          x11_window_get_pid(Display* dpy, Window wnd);
bool         x11_show_window_attributes(Display* dpy, Window wnd);
void         x11_show_window_properties(Display* dpy, Window wnd);

int main(int argc, char** argv)
{
    // Print booleans as 'true' or 'false' instead of 1 or 0
    std::cout << std::boolalpha;

    Display* dpy = XOpenDisplay(nullptr);    
    // Placeholder for future error checking in non-prototype code. 
    assert( dpy != nullptr && "Cannot open display" );

    const char* disp_str = XDisplayString(dpy);
    assert( disp_str != nullptr && "Failed to get display string" );
        std::cout << " [INFO] X11 Display string = " << disp_str << '\n';

    // Get root window 
    Window root = DefaultRootWindow(dpy);
    //assert( root != nullptr && "Failed to get root window");

    if(argc < 2){
        std::cerr << " [USAGE] $ " << argv[0] << " <COMMAND> " << '\n';
        return EXIT_FAILURE;
    }

    auto command = std::string{argv[1]};

    // List all windows and their window ids (unique identifiers)
    if(command == "list")
    {
        x11_iterate_child_windows(dpy, root,[](Display* dpy, Window wnd, int index)
        {
            if( !x11_show_window_attributes(dpy, wnd)) { return; }
            x11_show_window_properties(dpy, wnd);  
        });    
        return EXIT_SUCCESS;
    }

    if(command == "winfo")
    {
        // WARNING: Throws std::invalid_argument exception on parsing failure.
        int window_id = std::stoi(argv[2]);
        x11_show_window_attributes(dpy, window_id);
        x11_show_window_properties(dpy, window_id);
        return EXIT_SUCCESS;
    }

    // Kill window given its unique identifier (window id)
    if(command == "kill")
    {
        // WARNING: Throws std::invalid_argument exception on parsing failure.
        int window_id = std::stoi(argv[2]);
        // Attempt to get window process ID 
        int pid = x11_window_get_pid(dpy, window_id);
        if(pid < 0){
            std::cerr << " [ERROR] Unable to determine window process ID" << '\n';
            return EXIT_FAILURE;
        }
        std::cout << " Kill process with PID = " << pid << '\n';
        // Force process termination 
        kill(pid, SIGTERM);
        kill(pid, SIGKILL);
        return EXIT_SUCCESS;
    }

    XCloseDisplay(dpy);
    return 0;
}

/*================================================================*\
 *             I M P L E M E N T A T I O N S                      *
 *================================================================*/

// Get absolute path to file or symbolic link 
// Requires: #<unistd.h>, <limits.h>
std::string 
read_symlink(std::string const& path)
{
    // Create a buffer with size PATH_MAX + 1 filled with 0 ('\0'), null characters
    std::string buffer(PATH_MAX, 0);
    // ssize_t readlink(const char *pathname, char *buf, size_t bufsiz);
    ssize_t nread = ::readlink(path.c_str(), &buffer[0], PATH_MAX);
    if(nread == -1){ throw std::runtime_error("Error: unable to read symbolic link"); }
    buffer.resize(nread);
    return buffer;
}


int 
x11_get_property_int32(const char* property_name, Display* dpy, Window root_window)
{
    Atom p = XInternAtom(dpy, property_name, false);
    Atom actual_type;
    int format;
    unsigned long num_items, bytes_after;
    unsigned char* data = nullptr;

    int status = XGetWindowProperty(  dpy, root_window, p, 0L, 1024L
                                    , false,   XA_CARDINAL, &actual_type                                         
                                    , &format, &num_items, &bytes_after                                 
                                    , &data 
                                    );

    if( status != 0 || num_items < 1) 
    {
        //throw std::runtime_error("Error: failed to get property");        
        return -1;
    }
    int value =   data[0] | (data[1] << 8) 
                | (data[2] << 16) | (data[3] << 24);
    XFree(data);
    return value;
}

std::string 
x11_get_property_str(const char* property_name, Display* dpy, Window root_window)
{
    Atom p = XInternAtom(dpy, property_name, false);
    Atom actual_type;
    int format;
    unsigned long num_items, bytes_after;
    unsigned char* data = nullptr;

    int status = XGetWindowProperty(  dpy, root_window, p, 0L, 1024L
                                    , false,   XA_STRING, &actual_type                                         
                                    , &format, &num_items, &bytes_after                                 
                                    , &data 
                                    );

    if( status != 0 || num_items < 1) 
    {
        //throw std::runtime_error("Error: failed to get property");        
        return "<NONE>";
    }
    std::string value(data, data + num_items);
    XFree(data);
    return value;
}

template<typename Window_Consumer>
void x11_iterate_child_windows( Display*          dpy
                               ,Window            root_window
                               ,Window_Consumer&& consumer)
{
   // ----- Return parameters of function XQueryTree() -------//
    // Note: the memory is allocated and owned by the calling code.
    //       function.
    // Note: '{}' curly brackets gurantees default initialization as 'nullptr'
    Window root_return{};
    Window parent_return{};
    // List of childre window to be returned 
    Window* children_return{};
    // Number of children windows 
    unsigned int nchildren_return{};

    int result =  XQueryTree( 
                    dpy                // Display 
                  , root_window        // Root window
                  , &root_return       // Root window return 
                  , &parent_return     // Return paramameter for pointer to parent window 
                  , &children_return   // List of children to be returned 
                  , &nchildren_return  // Number of children elements in (children_return )
                ) ;

    assert( result != 0 && "Failed to query windows");

    std::cout << " [INFO] Number of children  windows = " << nchildren_return << '\n';

    for(auto n = 0; n < nchildren_return; n++ )
    {
        consumer(dpy, children_return[n], n);
    }

    // Dispose, release memory allocated for children windows return 
    XFree(children_return);
}


// Get Window name 
std::string 
x11_window_name(Display* dpy, Window wnd)
{
    char* window_name;

    if( !XFetchName(dpy, wnd, &window_name) )
    { return ""; }

    auto output = std::string{window_name};
    XFree(window_name);
    return output;
}

// Note: It only works for X-client applications running on local machine.
int x11_window_get_pid(Display* dpy, Window wnd)
{
    return x11_get_property_int32("_NET_WM_PID", dpy, wnd);
}

bool x11_show_window_attributes(Display* dpy, Window wnd)
{
    XWindowAttributes attr;

    if( !XGetWindowAttributes(dpy, wnd, &attr) )
    {
        std::fprintf(stderr, "Error, failed to get window (wnd = %zu) attribute \n"
                            , wnd);
        //exit(EXIT_FAILURE);
        return false;
    }  

    auto window_name = x11_window_name(dpy, wnd);
    if(window_name == ""){ return false; }

    auto is_visible = attr.map_state == IsViewable;
    std::cout << "\n\n [INFO] " 
                << " ; window_id = " << wnd 
                << " ; Visible = " << is_visible  
                << " ; Width  = "  << attr.width 
                << " ; Height = "  << attr.height                 
                << '\n';  
    std::cout << " => Window Name = " << window_name << '\n';

    return true;
}

void x11_show_window_properties(Display* dpy, Window wnd)
{
    int pid = x11_get_property_int32("_NET_WM_PID", dpy, wnd);

    std::cout << " => Window PID (Process ID) = "        
                << pid 
                << '\n';
    try
    {
        auto path_to_executable_procfs = std::string("/proc/") + std::to_string(pid) + "/exe";            
        std::cout << " => Window program (executable) = " 
                  << read_symlink(path_to_executable_procfs) << '\n';            
    } catch(const std::exception& ex)
    {
        std::cerr << " [ERROR] " <<  ex.what() << '\n';
    }

    std::cout << " => WM_NAME = " 
                << x11_get_property_str("WM_NAME", dpy, wnd) 
                << '\n';

    std::cout << " => WM_CLASS = " 
                << x11_get_property_str("WM_CLASS", dpy, wnd) 
                << '\n';

    std::cout << " => WM_COMMAND = " 
                << x11_get_property_str("WM_COMMAND", dpy, wnd) 
                << '\n';                  

    // Hostname of machine where the window application is located. 
    // A window may belong to an application (X-client) running in a 
    // remote machine.
    std::cout << " => WM_CLIENT_MACHINE = " 
                << x11_get_property_str("WM_CLIENT_MACHINE", dpy, wnd) 
                << '\n';
}

Building: 

$ git clone https://gist.github.com/639df76864d014ead12936fbd361be73 gist && cd gist 
$ cmake -H. -B_build -DCMAKE_BUILD_TYPE=Debug 
$ cmake --build _build --target all 

$ >> _build/x11-tool 
[INFO] X11 Display string = :1.0
[USAGE] $ _build/x11-tool <COMMAND>

List Windows: 

$ >> _build/x11-tool list
[INFO] X11 Display string = :1.0
[INFO] Number of children  windows = 116


[INFO]  ; window_id = 2097153 ; Visible = false ; Width  = 10 ; Height = 10
=> Window Name = xfce4-session
=> Window PID (Process ID) = 1715
=> Window program (executable) = /usr/bin/xfce4-session
=> WM_NAME = xfce4-session
=> WM_CLASS = xfce4-sessionXfce4-session
=> WM_COMMAND = xfce4-session
=> WM_CLIENT_MACHINE = localhost.localdomain

 ... ....   ... ....   ... .... 
 ... ....   ... ....   ... .... 

[INFO]  ; window_id = 69206017 ; Visible = false ; Width  = 10 ; Height = 10
=> Window Name = Terminal
=> Window PID (Process ID) = 778969
=> Window program (executable) = /usr/libexec/gnome-terminal-server
=> WM_NAME = Terminal
=> WM_CLASS = gnome-terminal-serverGnome-terminal-server
=> WM_COMMAND = gnome-terminal-server
=> WM_CLIENT_MACHINE = localhost.localdomain


[INFO]  ; window_id = 69206018 ; Visible = false ; Width  = 10 ; Height = 10
=> Window Name = Terminal
=> Window PID (Process ID) = 778969
=> Window program (executable) = /usr/libexec/gnome-terminal-server
=> WM_NAME = Terminal
=> WM_CLASS = <NONE>
=> WM_COMMAND = <NONE>
=> WM_CLIENT_MACHINE = localhost.localdomain

List information for a single window: 

$ >> _build/x11-tool winfo 69206017 
[INFO] X11 Display string = :1.0


[INFO]  ; window_id = 69206017 ; Visible = false ; Width  = 10 ; Height = 10
=> Window Name = Terminal
=> Window PID (Process ID) = 778969
=> Window program (executable) = /usr/libexec/gnome-terminal-server
=> WM_NAME = Terminal
=> WM_CLASS = gnome-terminal-serverGnome-terminal-server
=> WM_COMMAND = gnome-terminal-server
=> WM_CLIENT_MACHINE = localhost.localdomain

Check information via xprop x11 utility tools: 

 $ >> xprop -id 69206017 
WM_CLASS(STRING) = "gnome-terminal-server", "Gnome-terminal-server"
WM_COMMAND(STRING) = { "gnome-terminal-server" }
WM_CLIENT_LEADER(WINDOW): window id # 0x4200001
_NET_WM_PID(CARDINAL) = 778969
WM_LOCALE_NAME(STRING) = "en_US.UTF-8"
WM_CLIENT_MACHINE(STRING) = "localhost.localdomain"
WM_NORMAL_HINTS(WM_SIZE_HINTS):
                program specified size: 10 by 10
WM_PROTOCOLS(ATOM): protocols  WM_DELETE_WINDOW, WM_TAKE_FOCUS, _NET_WM_PING
WM_ICON_NAME(STRING) = "Terminal"
_NET_WM_ICON_NAME(UTF8_STRING) = "Terminal"
WM_NAME(STRING) = "Terminal"
_NET_WM_NAME(UTF8_STRING) = "Terminal"

# Get the executable that created this window 
 $ >> readlink -f /proc/778969/exe 
/usr/libexec/gnome-terminal-server

Terminate process that created the previous window: 

$ >> _build/x11-tool kill 69206017 
[INFO] X11 Display string = :1.0
Kill process with PID = 778969

1.32.6 Further reading and references

Implementations of X11 Client

Implementations of X11 Server

  • X Server Source Code / Gitlab [REFERENCE IMPLEMENTATION]
    • https://gitlab.freedesktop.org/xorg/xserver
    • "The X server accepts requests from client applications to create windows, which are (normally rectangular) "virtual screens" that the client program can draw into. Windows are then composed on the actual screen by the X server (or by a separate composite manager) as directed by the window manager, which usually communicates with the user via graphical controls such as buttons and draggable titlebars and borders."
  • The XFree86 - Project Inc - Major implementation of XServer for X11
    • https://www.xfree86.org/
    • "The XFree86 Project, Inc is a global volunteer organization which produces XFree86®, the freely redistributable open-source implementation of the X Window System continuously since 1992. XFree86 runs primarily on UNIX® and UNIX-like operating systems like Linux, all of the BSD variants, Sun Solaris both native 32 and 64 bit support, Solaris x86, Mac OS X (via Darwin) as well as other platforms like OS/2 and Cygwin. What XFree86 does, is provide a client/server interface between the display hardware (those physical things like the mouse, keyboard, and video displays) and the desktop environment, (this is typically called a window manager as it deals with how X is displayed i.e. the overall appearance). Yet X it goes beyond that and also gives the infrastructure and a standardized application interface (API). All of this which makes XFree86 platform-independent, network-transparent and fully extensible. In short, XFree86 is the premier open source X11-based desktop infrastructure.
  • Xvfb (X11 Server with virtual framebuffer)
  • Xephyr (X11 server implementation for running X11 clients with isolation)
  • XWayland (Wayland quasi-backward compatible implementation of X11 server)
    • https://wayland.freedesktop.org/xserver.html
    • Summary: X Server implementation for Microsft Windows NT operating system which allows using remote Linux or BSD graphical applications via X11 X Windows System network protocol.
  • Xming X Server for Windows (X11 server for Windows NT)
  • X-Win32 (X11 server for Windows NT) [COMMERCIAL]
    • https://www.starnet.com/info/xwin32_windows.php
    • Summary: "X-Win32 is the most widely used X server on the market. Run Linux on Windows at LAN-like speed. Features: Clarity: Offering picture-perfect rendering, X-Win32 is widely used in IC design; Speed: Outperforms even your local console when rendering 3D graphics; Reliability: X-Win32 has been constantly developed for the last 25 years; Affordable: X-Win32 is affordably priced with discounts for volume, government and educational licenses; Support: Your license and maintenance purchase gives you exclusive access to our fast, responsive support team."
  • XQuartz - X11 implementation for Apple's MacOSX
    • https://www.xquartz.org/
    • Summary: " The XQuartz project is an open-source effort to develop a version of the X.Org X Window System that runs on macOS. Together with supporting libraries and applications, it forms the X11.app that Apple shipped with OS X versions 10.5 through 10.7."
  • Microwindows or the Nano-X Window System
    • https://github.com/ghaerr/microwindows
    • http://www.microwindows.org/
    • Summary: "Microwindows or Nano-X is a small graphical windowing system that implements both Win32 and Nano-X (X11-like) APIs for clipped graphics drawing in windows on Linux, Mac OS X, EMSCRIPTEN, Android and other platforms. It is Open Source and licenced under the the Mozilla Public License. For creating GUIs, the Nuklear immediate mode GUI, Win32 builtin controls, and TinyWidget's controls based on Nano-X are included. FLTK can be used with the X11 compability library NX11."

X Windows System

  • Athena Widget Set - C Language Interface (libXaw Version 1.0.13) - X Consortium Standard
    • https://www.x.org/releases/current/doc/libXaw/libXaw.html
    • "The X Toolkit is made up of two distinct pieces, the Xt Intrinsics and a widget set. The Athena widget set is a sample implementation of a widget set built upon the Intrinsics. In the X Toolkit, a widget is the combination of an X window or subwindow and its associated input and output semantics. Because the Intrinsics provide the same basic functionality to all widget sets it may be possible to use widgets from the Athena widget set with other widget sets based upon the Intrinsics. Since widget sets may also implement private protocols, all functionality may not be available when mixing and matching widget sets. For information about the Intrinsics, see the X Toolkit Intrinsics - C Language Interface. The Athena widget set is a library package layered on top of the Intrinsics and Xlib that provides a set of user interface tools sufficient to build a wide variety of applications. This layer extends the basic abstractions provided by X and provides the next layer of functionality primarily by supplying a cohesive set of sample widgets. Although the Intrinsics are a Consortium standard, there is no standard widget set."

X11 - X Windows System Concepts

Low Level Graphics Stack - kernel and embedded Linux systems

Wayland (Note: Eventually may replace X Windows System)

  • Wayland v/s Xorg : How Are They Similar & How Are They Different
  • Wayland (display server protocol)
  • Book: The Wayland Protocol
    • https://wayland-book.com/introduction.html
    • "Wayland is the next-generation display server for Unix-like systems, designed and built by the alumni of the venerable Xorg server, and is the best way to get your application windows onto your user's screens. Readers who have worked with X11 in the past will be pleasantly surprised by Wayland's improvements, and those who are new to graphics on Unix will find it a flexible and powerful system for building graphical applications and desktops. This book will help you establish a firm understanding of the concepts, design, and implementation of Wayland, and equip you with the tools to build your own Wayland client and server applications. Over the course of your reading, we'll build a mental model of Wayland and establish the rationale that went into its design. Within these pages you should find many "aha!" moments as the intuitive design choices of Wayland become clear, which should help to keep the pages turning. Welcome to the future of open source graphics!"

Created: 2022-06-21 Tue 10:30

Validate