Modernizing GNU Make's Meta Log System

Abstract

For larger software projects, it is increasingly important that builds are visible and auditable. While GNU Make can be used for both legacy and modern projects, it has historically offered limited visibility of command executions, dependency evaluations, and file system accesses. Additionally, this inhibits debugging, reproducible builds, and optimization for improved performance. This article presents a complete rethink of GNU Make based upon an improved meta-log system. This enables us to transform it from a rule executor to a build engine with rich observability of its inner workings and performance. Information in the meta-log provides observability across several axes: command executions with full arguments, working directories (for path resolution), state transitions of dependencies (with change indicators), captured output streams, and detailed analysis of use of the file system via system-call monitoring. The modular architecture allows for separation of command logging, dependency state tracking, and behavioral file access analysis. Their respective observability capabilities can be enabled or disabled as needed for diagnostics or performance considerations. The tool configuration allows a range of observability/overhead, including disabling globally in production builds, targeting specific source files with detailed tracing, and masking the program behavior of non-target programs. The tool has produced substantial benefit in practice, substantially reducing the required proof effort and having stable performance. The tool has also detected bugs with little training. Taking cues from distributed data processing systems, an augmented meta-log can offer modern build-level requirements like reproducibility across machines, distributed build coordination, and smart automation. Implementation studies show that while logging all operations adds important overhead for I/O-heavy workloads, the log's size can be considerably reduced through smart instrumentation without sacrificing its diagnostic value. Such modernization brings GNU Make up to parity with other modern build systems while keeping the original intent of being lightweight and extensible, which has kept it in use for the last 40 years of software engineering development.