
Chapter 3 Additional build examples
Let's explain the OMake build model a bit more.
One issue that dominates this discussion is that OMake is based on global project analysis. That
means you define a configuration for the entire project, and you run one instance of omake.
For single-directory projects this doesn't mean much. For multi-directory projects it means a lot.
With GNU make, you would usually invoke the make
program recursively for each directory in
the project. For example, suppose you had a project with some project root directory, containing a
directory of sources src
, which in turn contains subdirectories lib
and main
.
So your project looks like this nice piece of ASCII art.
my_project/
|--> Makefile
`--> src/
|---> Makefile
|---> lib/
| |---> Makefile
| `---> source files...
`---> main/
|---> Makefile
`---> source files...
Typically, with GNU make, you would start an instance of make
in my_project/
; this
would in term start an instance of make
in the src/
directory; and this would start
new instances in lib/
and main/
. Basically, you count up the number of
Makefile
s in the project, and that is the number of instances of make
processes that
will be created.
The number of processes is no big deal with today's machines (sometimes contrary the the author's opinion, we
no longer live in the 1970s). The problem with the scheme was that each make
process had a
separate configuration, and it took a lot of work to make sure that everything was consistent.
Furthermore, suppose the programmer runs make
in the main/
directory, but the
lib/
is out-of-date. In this case, make
would happily crank away, perhaps trying to
rebuild files in lib/
, perhaps just giving up.
With OMake this changes entirely. Well, not entirely. The source structure is quite similar, we
merely add some Os to the ASCII art.
my_project/
|--> OMakeroot (or Root.om)
|--> OMakefile
`--> src/
|---> OMakefile
|---> lib/
| |---> OMakefile
| `---> source files...
`---> main/
|---> OMakefile
`---> source files...
The role of each <dir>/OMakefile
plays the same role as each <dir>/Makefile
: it
describes how to build the source files in <dir>
. The OMakefile retains much of syntax and
structure of the Makefile, but in most cases it is much simpler.
One minor difference is the presence of the OMakeroot in the project root. The main purpose of this
file is to indicate where the project root is in the first place (in case omake
is
invoked from a subdirectory). The OMakeroot
serves as the bootstrap file; omake starts by
reading this file first. Otherwise, the syntax and evaluation of OMakeroot
is no different
from any other OMakefile
.
The big difference is that OMake performs a global analysis. Here is what happens
when omake
starts.
-
omake locates that OMakeroot file, and reads it.
- Each OMakefile points to its subdirectory OMakefiles using the .SUBDIRS target.
For example,
my_project/OMakefile
has a rule,
.SUBDIRS: src
and the my_project/src/OMakefile
has a rule,
.SUBDIRS: lib main
omake
uses these rules to read and evaluate every OMakefile
in the project.
Reading and evaluation is fast. This part of the process is cheap.
- Now that the entire configuration is read,
omake
determines which files are out-of-date
(using a global analysis), and starts the build process. This may take a while, depending on what
exactly needs to be done.
There are several advantages to this model. First, since analysis is global, it is much easier to
ensure that the build configuration is consistent–after all, there is only one configuration.
Another benefit is that the build configuration is inherited, and can be re-used, down the
hierarchy. Typically, the root OMakefile
defines some standard boilerplate and
configuration, and this is inherited by subdirectories that tweak and modify it (but do not need to
restate it entirely). The disadvantage of course is space, since this is global analysis after all.
In practice rarely seems to be a concern; omake takes up much less space than your web browser even
on large projects.
Some notes to the GNU/BSD make user.
-
OMakefiles are a lot like Makefiles. The syntax is similar, and there many of the builtin
functions are similar. However, the two build systems are not the same. Some evil features (in the authors'
opinions) have been dropped in OMake, and some new features have been added.
- OMake works the same way on all platforms, including Win32. The standard configuration does
the right thing, but if you care about porting your code to multiple platforms, and you use some
tricky features, you may need to condition parts of your build config on the
$(OSTYPE)
variable.
- A minor issue is that OMake dependency analysis is based on MD5 file digests. That is,
dependencies are based on file contents, not file modification times. Say goodbye to
false rebuilds based on spurious timestamp changes and mismatches between local time and fileserver
time.
3.1 OMakeroot vs. OMakefile
Before we begin with examples, let's ask the first question, “What is the difference between the
project root OMakeroot and OMakefile?” A short answer is, there is no difference, but you must
have an OMakeroot file (or Root.om file).
However, the normal style is that OMakeroot is boilerplate and is more-or-less the same for all
projects. The OMakefile is where you put all your project-specific stuff.
To get started, you don't have to do this yourself. In most cases you just perform the following
step in your project root directory.
-
Run
omake --install
in your project root.
This will create the initial OMakeroot and OMakefile files that you can edit to get started.
3.2 An example C project
To begin, let's start with a simple example. Let's say that we have a full directory tree,
containing the following files.
my_project/
|--> OMakeroot
|--> OMakefile
`--> src/
|---> OMakefile
|---> lib/
| |---> OMakefile
| |---> ouch.c
| |---> ouch.h
| `---> bandaid.c
`---> main/
|---> OMakefile
|---> horsefly.c
|---> horsefly.h
`---> main.c
Here is an example listing.
my_project/OMakeroot:
# Include the standard configuration for C applications
open build/C
# Process the command-line vars
DefineCommandVars()
# Include the OMakefile in this directory.
.SUBDIRS: .
my_project/OMakefile:
# Set up the standard configuration
CFLAGS += -g
# Include the src subdirectory
.SUBDIRS: src
my_project/src/OMakefile:
# Add any extra options you like
CFLAGS += -O2
# Include the subdirectories
.SUBDIRS: lib main
my_project/src/lib/OMakefile:
# Build the library as a static library.
# This builds libbug.a on Unix/OSX, or libbug.lib on Win32.
# Note that the source files are listed _without_ suffix.
StaticCLibrary(libbug, ouch bandaid)
my_project/src/main/OMakefile:
# Some files include the .h files in ../lib
INCLUDES += ../lib
# Indicate which libraries we want to link against.
LIBS[] +=
../lib/libbug
# Build the program.
# Builds horsefly.exe on Win32, and horsefly on Unix.
# The first argument is the name of the executable.
# The second argument is an array of object files (without suffix)
# that are part of the program.
CProgram(horsefly, horsefly main)
# Build the program by default (in case omake is called
# without any arguments). EXE is defined as .exe on Win32,
# otherwise it is empty.
.DEFAULT: horsefly$(EXE)
Most of the configuration here is defined in the file build/C.om
(which is part of the OMake
distribution). This file takes care of a lot of work, including:
-
Defining the
StaticCLibrary
and CProgram
functions, which describe the canonical
way to build C libraries and programs.
- Defining a mechanism for scanning each of the source programs to discover dependencies.
That is, it defines .SCANNER rules for C source files.
Variables are inherited down the hierarchy, so for example, the value of CFLAGS in
src/main/OMakefile is “-g -O2
”.
3.3 An example OCaml project
Let's repeat the example, assuming we are using OCaml instead of C.
This time, the directory tree looks like this.
my_project/
|--> OMakeroot
|--> OMakefile
`--> src/
|---> OMakefile
|---> lib/
| |---> OMakefile
| |---> ouch.ml
| |---> ouch.mli
| `---> bandaid.ml
`---> main/
|---> OMakefile
|---> horsefly.ml
|---> horsefly.mli
`---> main.ml
The listing is only a bit different.
my_project/OMakeroot:
# Include the standard configuration for OCaml applications
open build/OCaml
# Process the command-line vars
DefineCommandVars()
# Include the OMakefile in this directory.
.SUBDIRS: .
my_project/OMakefile:
# Set up the standard configuration
OCAMLFLAGS += -Wa
# Do we want to use the bytecode compiler,
# or the native-code one? Let's use both for
# this example.
NATIVE_ENABLED = true
BYTE_ENABLED = true
# Include the src subdirectory
.SUBDIRS: src
my_project/src/OMakefile:
# Include the subdirectories
.SUBDIRS: lib main
my_project/src/lib/OMakefile:
# Let's do aggressive inlining on native code
OCAMLOPTFLAGS += -inline 10
# Build the library as a static library.
# This builds libbug.a on Unix/OSX, or libbug.lib on Win32.
# Note that the source files are listed _without_ suffix.
OCamlLibrary(libbug, ouch bandaid)
my_project/src/main/OMakefile:
# These files depend on the interfaces in ../lib
OCAMLINCLUDES += ../lib
# Indicate which libraries we want to link against.
OCAML_LIBS[] +=
../lib/libbug
# Build the program.
# Builds horsefly.exe on Win32, and horsefly on Unix.
# The first argument is the name of the executable.
# The second argument is an array of object files (without suffix)
# that are part of the program.
OCamlProgram(horsefly, horsefly main)
# Build the program by default (in case omake is called
# without any arguments). EXE is defined as .exe on Win32,
# otherwise it is empty.
.DEFAULT: horsefly$(EXE)
In this case, most of the configuration here is defined in the file build/OCaml.om
. In this
particular configuration, files in my_project/src/lib
are compiled aggressively with the
option -inline 10
, but files in my_project/src/lib
are compiled normally.
3.4 Handling new languages
The previous two examples seem to be easy enough, but they rely on the OMake standard library (the
files build/C
and build/OCaml
) to do all the work. What happens if we want to write a
build configuration for a language that is not already supported in the OMake standard library?
For this example, let's suppose we are adopting a new language. The language uses the standard
compile/link model, but is not in the OMake standard library. Specifically, let's say we have the
following setup.
-
Source files are defined in files with a
.cat
suffix (for Categorical Abstract Terminology).
.cat
files are compiled with the catc
compiler to produce .woof
files
(Wicked Object-Oriented Format).
.woof
files are linked by the catc
compiler with the -c
option to produce
a .dog
executable (Digital Object Group). The catc
also defines a -a
option to
combine several .woof
files into a library.
- Each
.cat
can refer to other source files. If a source file a.cat
contains a
line open b
, then a.cat
depends on the file b.woof
, and a.cat
must be
recompiled if b.woof
changes. The catc
function takes a -I
option to define a
search path for dependencies.
To define a build configuration, we have to do three things.
-
Define a
.SCANNER
rule for discovering dependency information for the source files.
- Define a generic rule for compiling a
.cat
file to a .woof
file.
- Define a rule (as a function) for linking
.woof
files to produce a .dog
executable.
Initially, these definitions will be placed in the project root OMakefile
.
3.4.1 Defining a default compilation rule
Let's start with part 2, defining a generic compilation rule. We'll define the build rule as an
implicit rule. To handle the include path, we'll define a variable CAT_INCLUDES
that
specifies the include path. This will be an array of directories. To define the options, we'll use
a lazy variable (Section 6.5). In case there
are any other standard flags, we'll define a CAT_FLAGS
variable.
# Define the catc command, in case we ever want to override it
CATC = catc
# The default flags are empty
CAT_FLAGS =
# The directories in the include path (empty by default)
INCLUDES[] =
# Compute the include options from the include path
PREFIXED_INCLUDES[] = $`(mapprefix -I, $(INCLUDES))
# The default way to build a .woof file
%.woof: %.cat
$(CATC) $(PREFIXED_INCLUDES) $(CAT_FLAGS) -c $<
The final part is the build rule itself, where we call the catc
compiler with the include
path, and the CAT_FLAGS
that have been defined. The $<
variable represents the source
file.
3.4.2 Defining a rule for linking
For linking, we'll define another rule describing how to perform linking. Instead of defining an
implicit rule, we'll define a function that describes the linking step. The function will take two
arguments; the first is the name of the executable (without suffix), and the second is the files to
link (also without suffixes). Here is the code fragment.
# Optional link options
CAT_LINK_FLAGS =
# The function that defines how to build a .dog program
CatProgram(program, files) =
# Add the suffixes
file_names = $(addsuffix .woof, $(files))
prog_name = $(addsuffix .dog, $(files))
# The build rule
$(prog_name): $(file_names)
$(CATC) $(PREFIXED_INCLUDES) $(CAT_FLAGS) $(CAT_LINK_FLAGS) -o $@ $+
# Return the program name
value $(prog_name)
The CAT_LINK_FLAGS
variable is defined just in case we want to pass additional flags specific
to the link step. Now that this function is defined, whenever we want to define a rule for building
a program, we simply call the rule. The previous implicit rule specifies how to compile each source file,
and the CatProgram
function specifies how to build the executable.
# Build a rover.dog program from the source
# files neko.cat and chat.cat.
# Compile it by default.
.DEFAULT: $(CatProgram rover, neko chat)
3.4.3 Dependency scanning
That's it, almost. The part we left out was automated dependency scanning. This is one of the
nicer features of OMake, and one that makes build specifications easier to write and more robust.
Strictly speaking, it isn't required, but you definitely want to do it.
The mechanism is to define a .SCANNER
rule, which is like a normal rule, but it specifies how
to compute dependencies, not the target itself. In this case, we want to define a .SCANNER
rule of the following form.
.SCANNER: %.woof: %.cat
<commands>
This rule specifies that a .woof
file may have additional dependencies that can be extracted
from the corresponding .cat
file by executing the <commands>
. The result of
executing the <commands>
should be a sequence of dependencies in OMake format, printed to the
standard output.
As we mentioned, each .cat
file specifies dependencies on .woof
files with an
open
directive. For example, if the neko.cat
file contains a line open chat
,
then neko.woof
depends on chat.woof
. In this case, the <commands>
should print
the following line.
neko.woof: chat.woof
For an analogy that might make this clearer, consider the C programming language, where a .o
file is produced by compiling a .c
file. If a file foo.c
contains a line like
#include "fum.h"
, then foo.c
should be recompiled whenever fum.h
changes. That
is, the file foo.o
depends on the file fum.h
. In the OMake parlance, this is
called an implicit dependency, and the .SCANNER
<commands>
would print a line
like the following.
foo.o: fum.h
Now, returning to the animal world, to compute the dependencies of neko.woof
, we
should scan neko.cat
, line-by-line, looking for lines of the form open <name>
. We
could do this by writing a program, but it is easy enough to do it in omake
itself. We can
use the builtin awk
function (Section 9.11.5) to scan the source file. One slight complication
is that the dependencies depend on the INCLUDE
path. We'll use the
find-in-path
function (Section 9.2.6) to find them. Here we go.
.SCANNER: %.woof: %.cat
section
# Scan the file
deps[] =
awk($<)
case $'^open'
deps[] += $2
export
# Remove duplicates, and find the files in the include path
deps = $(find-in-path $(INCLUDES), $(set $(deps)))
# Print the dependencies
println($"$@: $(deps)")
Let's look at the parts. First, the entire body is defined in a section
because we are
computing it internally, not as a sequence of shell commands.
We use the deps
variable to collect all the dependencies. The awk
function scans the
source file ($<
) line-by-line. For lines that match the regular expression ^open
(meaning that the line begins with the word open
), we add the second word on the line to the
deps
variable. For example, if the input line is open chat
, then we would add the
chat
string to the deps
array. All other lines in the source file are ignored.
Next, the $(set $(deps))
expression removes any duplicate values in the deps
array
(sorting the array alphabetically in the process). The find-in-path
function then finds the
actual location of each file in the include path.
The final step is print the result as the string $"$@: $(deps)"
The quotations are added to
flatten the deps
array to a simple string.
3.4.4 Pulling it all together
To complete the example, let's pull it all together into a single project, much like our previous
example.
my_project/
|--> OMakeroot
|--> OMakefile
`--> src/
|---> OMakefile
|---> lib/
| |---> OMakefile
| |---> neko.cat
| `---> chat.cat
`---> main/
|---> OMakefile
`---> main.cat
The listing for the entire project is as follows. Here, we also include a function
CatLibrary
to link several .woof
files into a library.
my_project/OMakeroot:
# Process the command-line vars
DefineCommandVars()
# Include the OMakefile in this directory.
.SUBDIRS: .
my_project/OMakefile:
########################################################################
# Standard config for compiling .cat files
#
# Define the catc command, in case we ever want to override it
CATC = catc
# The default flags are empty
CAT_FLAGS =
# The directories in the include path (empty by default)
INCLUDES[] =
# Compute the include options from the include path
PREFIXED_INCLUDES[] = $`(mapprefix -I, $(INCLUDES))
# Dependency scanner for .cat files
.SCANNER: %.woof: %.cat
section
# Scan the file
deps[] =
awk($<)
case $'^open'
deps[] += $2
export
# Remove duplicates, and find the files in the include path
deps = $(find-in-path $(INCLUDES), $(set $(deps)))
# Print the dependencies
println($"$@: $(deps)")
# The default way to compile a .cat file
%.woof: %.cat
$(CATC) $(PREFIXED_INCLUDES) $(CAT_FLAGS) -c $<
# Optional link options
CAT_LINK_FLAGS =
# Build a library for several .woof files
CatLibrary(lib, files) =
# Add the suffixes
file_names = $(addsuffix .woof, $(files))
lib_name = $(addsuffix .woof, $(lib))
# The build rule
$(lib_name): $(file_names)
$(CATC) $(PREFIXED_INCLUDES) $(CAT_FLAGS) $(CAT_LINK_FLAGS) -a $@ $+
# Return the program name
value $(lib_name)
# The function that defines how to build a .dog program
CatProgram(program, files) =
# Add the suffixes
file_names = $(addsuffix .woof, $(files))
prog_name = $(addsuffix .dog, $(program))
# The build rule
$(prog_name): $(file_names)
$(CATC) $(PREFIXED_INCLUDES) $(CAT_FLAGS) $(CAT_LINK_FLAGS) -o $@ $+
# Return the program name
value $(prog_name)
########################################################################
# Now the program proper
#
# Include the src subdirectory
.SUBDIRS: src
my_project/src/OMakefile:
.SUBDIRS: lib main
my_project/src/lib/OMakefile:
CatLibrary(cats, neko chat)
my_project/src/main/OMakefile:
# Allow includes from the ../lib directory
INCLUDES[] += ../lib
# Build the program
.DEFAULT: $(CatProgram main, main ../cats)
Some notes. The configuration in the project OMakeroot
defines the standard configuration, including
the dependency scanner, the default rule for compiling source files, and functions for building
libraries and programs.
These rules and functions are inherited by subdirectories, so the .SCANNER
and build rules
are used automatically in each subdirectory, so you don't need to repeat them.
3.4.5 Finishing up
At this point we are done, but there are a few things we can consider.
First, the rules for building cat programs is defined in the project OMakefile
. If you had
another cat project somewhere, you would need to copy the OMakeroot
(and modify it as
needed). Instead of that, you should consider moving the configuration to a shared library
directory, in a file like Cat.om
. That way, instead of copying the code, you could include
the shared copy with an OMake command open Cat
. The share directory should be added to your
OMAKEPATH
environment variable to ensure that omake
knows how to find it.
Better yet, if you are happy with your work, consider submitting it as a standard configuration (by
sending a request to omake@metaprl.org
) so that others can make use of it too.
3.5 Collapsing the hierarchy, .SUBDIRS bodies
Some projects have many subdirectories that all have the same configuration. For instance, suppose
you have a project with many subdirectories, each containing a set of images that are to be composed
into a web page. Apart from the specific images, the configuration of each file is the same.
To make this more concrete, suppose the project has four subdirectories page1
, page2
,
page3
, and page4
. Each contains two files image1.jpg
and image2.jpg
that are part of a web page generated by a program genhtml
.
Instead of of defining a OMakefile
in each directory, we can define it as a body to the
.SUBDIRS
command.
.SUBDIRS: page1 page2 page3 page4
index.html: image1.jpg image2jpg
genhtml $+ > $@
The body of the .SUBDIRS
is interpreted exactly as if it were the OMakefile
, and it
can contain any of the normal statements. The body is evaluated in the subdirectory for each
of the subdirectories. We can see this if we add a statement that prints the current directory
($(CWD)
).
.SUBDIRS: page1 page2 page3 page4
println($(absname $(CWD)))
index.html: image1.jpg image2jpg
genhtml $+ > $@
# prints
/home/jyh/.../page1
/home/jyh/.../page2
/home/jyh/.../page3
/home/jyh/.../page4
3.5.1 Using glob patterns
Of course, this specification is quite rigid. In practice, it is likely that each subdirectory will
have a different set of images, and all should be included in the web page. One of the easier
solutions is to use one of the directory-listing functions, like
glob
(Section 9.4.1) or ls
(Section 9.4.2).
The glob
function takes a shell pattern, and returns an array of
file with matching filenames in the current directory.
.SUBDIRS: page1 page2 page3 page4
IMAGES = $(glob *.jpg)
index.html: $(IMAGES)
genhtml $+ > $@
3.5.2 Simplified sub-configurations
Another option is to add a configuration file in each of the subdirectories that defines
directory-specific information. For this example, we might define a file BuildInfo.om
in
each of the subdirectories that defines a list of images in that directory. The .SUBDIRS
line is similar, but we include the BuildInfo file.
.SUBDIRS: page1 page2 page3 page4
include BuildInfo # Defines the IMAGES variable
index.html: $(IMAGES)
genhtml $+ > $@
Where we might have the following configurations.
page1/BuildInfo.om:
IMAGES[] = image.jpg
page2/BuildInfo.om:
IMAGES[] = ../common/header.jpg winlogo.jpg
page3/BuildInfo.om:
IMAGES[] = ../common/header.jpg unixlogo.jpg daemon.jpg
page4/BuildInfo.om:
IMAGES[] = fee.jpg fi.jpg foo.jpg fum.jpg
3.5.3 Computing the subdirectory list
The other hardcoded specification is the list of subdirectories page1
, ..., page4
.
Rather than editing the project OMakefile
each time a directory is added, we could compute it
(again with glob
).
.SUBDIRS: $(glob page*)
index.html: $(glob *.jpg)
genhtml $+ > $@
Alternately, the directory structure may be hierarchical. Instead of using glob
, we could
use the subdirs
function, returns each of the directories in a hierarchy. For example, this
is the result of evaluating the subdirs
function in the omake project root. The P
option, passed as the first argument, specifies that the listing is “proper,” it should not
include the omake
directory itself.
osh> subdirs(P, .)
- : <array
/home/jyh/.../omake/mk : Dir
/home/jyh/.../omake/RPM : Dir
...
/home/jyh/.../omake/osx_resources : Dir>
Using subdirs
, our example is now as follows.
.SUBDIRS: $(subdirs P, .)
index.html: $(glob *.jpg)
genhtml $+ > $@
In this case, every subdirectory will be included in the project.
If we are using the BuildInfo.om
option. Instead of including every subdirectory, we could
include only those that contain a BuildInfo.om
file. For this purpose, we can use the
find
function, which traverses the directory hierarchy looking for files that match a test
expression. In our case, we want to search for files with the name BuildInfo.om
.
Here is an example call.
osh> FILES = $(find . -name BuildInfo.om)
- : <array
/home/jyh/.../omake/doc/html/BuildInfo.om : File
/home/jyh/.../omake/src/BuildInfo.om : File
/home/jyh/.../omake/tests/simple/BuildInfo.om : File>
osh> DIRS = $(dirof $(FILES))
- : <array
/home/jyh/.../omake/doc/html : Dir
/home/jyh/.../omake/src : Dir
/home/jyh/.../omake/tests/simple : Dir>
In this example, there are three BuildInfo.om
files, in the doc/html
, src
, and
tests/simple
directories. The dirof
function returns the directories for each of the
files.
Returning to our original example, we modify it as follows.
.SUBDIRS: $(dirof $(find . -name BuildInfo.om))
include BuildInfo # Defines the IMAGES variable
index.html: $(IMAGES)
genhtml $+ > $@
3.5.4 Temporary directories
Sometimes, your project may include temporary directories–directories where you place intermediate
results. these directories are deleted whenever the project is cleanup up. This means, in
particular, that you can't place an OMakefile
in a temporary directory, because it will be
removed when the directory is removed.
Instead, if you need to define a configuration for any of these directories, you will need to define
it using a .SUBDIRS
body.
section
CREATE_SUBDIRS = true
.SUBDIRS: tmp
# Compute an MD5 digest
%.digest: %.comments
echo $(digest $<) > $@
# Extract comments from the source files
%.comments: ../src/%.src
grep '^#' $< > $@
.DEFAULT: foo.digest
.PHONY: clean
clean:
rm -rf tmp
In this example, we define the CREATE_SUBDIRS
variable as true, so that the tmp
directory will be created if it does not exist. The .SUBDIRS
body in this example is a bit
contrived, but it illustrates the kind of specification you might expect. The clean
phony-target indicates that the tmp
directory should be removed when the project is cleaned
up.