Pipapo

An Expert System for Building Software

Existing Build systems are stupid! Most ones have some strong parts but fail in doing other things just right and all of them are more or less painful to use.

What do we want

  1. Proper dependency tracking, rebuild only whats needed
    This is what a casual user expects, albeit most systems get this thing managed somehow, most fail in less spectacular ways (just rebuilds too much, rarely fail)

  2. Track the right (all) things
    Make and many other systems track only files or maybe even the source files. For delivering correct results one has to acknowledge external files like compilers, libraries too and tracking of control options and environment variables is important too.

  3. Build the dependency graph on demand
    Tests for system features and existence of libraries as well as scanning sourcefile dependencies can be part of the dependency graph and extending it dynamically on demand. This speeds up partial builds, because only whats really required will be inferred.

  4. Software configuration, based on the current system/architecture and user supplied configuration
    While the first point is just some practical exercise in graph theory, this is really the holy grail of software building.

  5. Simple configuration with 'NO' redundancy
    This is really the thing where ALL exiting build systems I know off fail! The user has to maintain tests for available features manually, after such a test he has to setup his configuration variables in some ways, getting them stuffed into the language he compiles and finally using a lot #if HAVE_FOO things in his source.

  6. Knowledge about the system
    When the user includes foo.h the build system should know if this this needs a libfoo for linking, certainly for standard libraries, maybe for common libraries, probably some generic tests and finally user supplied configuraton for special cases

  7. Proper caching of configuration results
    Most tools offer this, but they notoriously fail with invalidating the cache, which leads to spurious bugs to be fixed by the user deleting the entire cache.

  8. Caching of compilation results
    This is quite optional there are external tools like ccache which offer this functionality. But scons shows how this can be useful within the build system. 1. Proper parallel/distributed builds

    Most systems have support for that, but it is easy to get this wrong,
    because dependencies are not tracked as a whole (SUBDIRS= ...)

How a Build System becomes smart

A Build System should 'know' how to do things right. This should include platform specific quirks and any known issues. This should make it an ''Expert System for Building Software'. It should be able to deduce ''ALL' prerequisites from the given sources. The user should only provide configuration for things which can not be deduced.

Analyze the source in well defined way. Lets show:

TODO broken asciidocing

// 1.

// 2.

// 3.

// 4.

// 5.
  1. We include a common config.h file, this is a special file generated by the build system which defines the shared configuration. When there is no such config.h included, the build system should generate the necessary -D flags.

  2. We see that the system header ‘math.h` is unconditionally included, this means that it 'must’ be available on the systen. Further the build system should have the knowledge that linking this source requires -lm on some platforms. This should be handled automatically!

  3. The user wants to provide an alternative here (HAVE_FOO_H) so the build system should generate a test for the availability of foo.h, note that the #include statement refines this test to prefer the local source tree instead system headers.

  4. This defines an optional test for a feature which can be enabled (--with-valgrind). Valgrind installs its headers under its own subdir (/usr/include/valgrind/) this is also a case which should be automatically handled by the build system and providing a -I/usr/include/valgrind flag to the preprocessor.

  5. buildsystem is a placeholder here, I just want to show that the user should be able to pass some metainformation to the build system via pragmas (or comments for other languages), the point is that we want a single point of interest, with the option to put that near the source where it is used and not scattered about several files.

Details

  • dependencies should not only be file (filetime) dependencies but generic. A compilation result depends on configuration features (CFLAGS) and the used compiler as well.

  • always use a complete dependency graph, never sub-parts of it