More documentation and some test to what is written in documentation.

[omk.git] / doc / manual.texinfo

\input texinfo   @c -*-texinfo-*-
@c %**start of header
@setfilename omk
@settitle Ocera Make System
@c %**end of header

@copying
Manual for Ocera Make System (OMK)

Copyright @copyright{} 2007 Michal Sojka, Pavel Pisa
@end copying

@titlepage
@title Ocera Make System Manual
@page
@vskip 0pt plus 1filll
@insertcopying
@end titlepage

@contents

@ifnottex
@node Top
@top Ocera Make System

@insertcopying
@end ifnottex


@node Overview of OMK
@chapter OMK Overview
@cindex overview

OMK is an advanced make system written entirely in GNU make. Compiling
software using OMK requires only GNU make binary and standard UNIX
utilities (@command{sh}, @command{sed} and
@command{tr}@footnote{@command{tr} is needed only for OMK to be
compatible with MinGW.}) installed. OMK aims to be developer friendly; to
use OMK, you do not need to understand (sometimes) cryptic syntax of
makefile commands.

OMK handles properly dependencies of source files and libraries, so it
is not necessary to recompile the whole project if only some files are
changed. Also OMK greatly simplifies compilation of projects, where
source files are spread between multiple directories. It is also very
useful in combining components (libraries) from different projects to a
single project.

You can use OMK on all platforms where you can run GNU Make including
Cygwin and MinGW. MS DOS was not tested.

@c @section Features

@c @itemize
@c @item
@c Easy to use for beginners.
@c @item
@c Automatic handling of dependencies.
@c @item
@c Supported host platforms: all Un*x operating system including Linux,
@c Cygwin, MS DOS and maybe others.
@c @end itemize

@section Why to Use OMK?

@section Quick Start

If you get some sources, which are distributed with OMK, usually the
following commands are sufficient to compile the whole project.

@example
@verbatim
make default-config
make
@end verbatim
@end example

To use OMK in your own project, follow these steps:

@enumerate
@item
  Take appropriate @file{Makefile.rules}, put it together with leaf
  @file{Makefile} to the root directory of your project.
@item
  Create @file{Makefile.omk} files in all directories you want to
  compile something. Please refer to chapter ... to learn what to write
  in @file{Makefile.omk} files.
@item
  Run @command{make omkize} in the root directory.
@end enumerate

Your project is now ready to compile.


@node History
@section History

OMK was originally written by Pavel Pisa as a solution to have one
common make system for OCERA project, where we needed to compile
user-space programs, Linux kernel modules and RT Linux modules in one
package. Although this system was not accepted for the whole OCERA
project. Several individual developers (mostly from Czech Technical
University) liked it and started to use it.

As a number of projects using OMK grew it was necessary to modularize
the make system to support more ``targets''. Michal Sojka took care
about the process of modularization.

@chapter Original README

@b{Important notice:} This make system uses features found in recent
versions of GNU Make program. If you encounter problems with package
building, check, that you use correct version of Make program.  The
Make older than version 3.80, could not be used.  Even Make version
3.80 has annoying bug which causes building fail with misleading
message "virtual memory exhausted".  Please, upgrade at least to
version 3.81 of GNU Make.

There is list of features which we want to solve with our make system:
@itemize
@item
Central @file{Makefile.rules} for most of components of a bigger project.

FIXME (our CAN framework includes more libraries common with our other
projects, we need to separate some utility libraries etc.)
@item
The rules in more spread Makefiles are way to the hell (update for
different kernel, RT-Linux etc would be nightmare in other case).
@item
Make system should allow to freely move cross-dependant components in
directory structure without need to update users of moved component (I
hate somethink like @option{-I../../sched/rtlshwq/include} in CAN makefiles for
example. If a component is renamed or version is added to then name,
all Makefiles in CAN will require update).
@item
Make system should be able to compile mutually cross-dependant
libraries and should ensure, that change in one component sources or
headers would result in relink or rebuild in components linked against
that library or including modified header file.
@item
Make system has to enable compilation out of OCERA full source tree
(we would lost many users of particular components in other case).
@item
Compile should be able to do all above work without need to install
any files before successful finish of build.
@item
Because we use some libraries for RT-Linux build and user-space build,
we need to solve how to compile from same sources to both targets.
@item
The build system should allow to call make for particular source
subdirectory. Time of recursive make through all subdirectories is
unacceptable.
@item
Make system should enable to build out of sources tree (else clean or
working with CVS sandbox gets fussy and simultaneous multiple targets
gets problematic).
@item
It would be good, if there is a possibility to call make from
read-only media sources.
@item
Make system should store results of build in some separate directory
structure to simple install and testing.
@item
Makefiles in sources directories should be simple.
@end itemize

There is probably only one alternative fully supporting above requirements
and it is GNU Autoheader...Automake...Autoconf... system.
But it is complicated and requires big amount of support files.
It would be acceptable if it could be easily used for OCERA framework.
But there are important show stoppers for that system:
@itemize
@item
It would require deep revision of all OCERA CVS contents and agreement
on this would be problematic
@item
This system is not well prepared for dual compilation for Linux and
RT-Linux sub-targets. It would mean many changes in default autoconf
setup to support this. Probably simplest way would be to rebuild GCC
tool chain for something like i586-elf-rtlinux.  This would require
even more space for OCERA development.
@end itemize

The problem calls for same solution, which would have minimal impact
on other components and would be elegant and would be maintainable
and small, because our main goal is components development and not
make systems development.

There is result of our trial. It is OMK make system.
The @file{Makefile} and @file{Makefile.omk} files should be in all source
directories. Common @file{Makefile.rules} file is required in the toplevel
sources directory. Alternatively this file could be moved
to link tree pointing into readonly media or can be anywhere
else if @code{MAKERULES_DIR} and @code{SOURCES_DIR} are specified.

Syntax of Makefile.omk files is for usual cases compatible
to Automake's Makefile.am descriptions. There are specific targets
for RT-Linux and Linux kernel related stuff

Makefile.omk user defined variables
@vtable @code
@item SUBDIRS
list of subdirectories intended for make from actual directory
@item lib_LIBRARIES
list of the user-space libraries
@item shared_LIBRARIES
list of the user-space shared libraries
@item kernel_LIBRARIES
list of the kernel-space libraries
@item rtlinux_LIBRARIES
list of the RT-Linux kernel-space libraries
@item include_HEADERS  
list of the user-space header files
@item nobase_include_HEADERS 
headers copied even with directory part
@item kernel_HEADERS   
list of the kernel-space  header files
@item rtlinux_HEADERS  
list of the RT-Linux kernel-space  header files
@item bin_PROGRAMS     
list of the require binary programs
@item utils_PROGRAMS   
list of the development utility programs
@item kernel_MODULES   
list of the kernel side modules/applications
@item rtlinux_MODULES  
list of RT-Linux the kernel side modules/applications
@item xxx_SOURCES      
list of specific target sources
@item INCLUDES         
additional include directories and defines for user-space
@item kernel_INCLUDES  
additional include directories and defines for kernel-space
@item rtlinux_INCLUDES 
additional include directories and defines for RT-Linux
@item default_CONFIG   
list of default config assignments CONFIG_XXX=y/n ...
@end vtable

The Makefile is same for all sources directories and is only 14 lines
long.  It is there only for convenience reasons to enable call "make"
from local directory. It contains code which locates
@file{Makefile.rules} in actual or any parent directory. With standard
BASH environment it works such way, that if you get into sources
directory over symbolic links, it is able to unwind yours steps back
=> you can make links to readonly media component directories, copy
@file{Makefile.rules}, Makefile and toplevel Makefile.omk, adjust
Makefile.omk to contain only required components and then call make in
top or even directories after crossing from your tree to readonly
media.

The system compiles all files out of source directories.  The actual
version of system is adapted even for OCERA tree mode if
@code{OCERA_DIR} variable is defined in @file{Makefile.rules}

There are next predefined directory name components, which can be
adapted if required

@table @code
@item BUILD_DIR_NAME = _build
        prefix of directory, where temporary build files are stored
@item COMPILED_DIR_NAME = _compiled
        prefix of directory, where final compilation results are stored
@item GROUP_DIR_NAME = yyy
        this is used for separation of build sub-trees in OCERA environment
        where more @file{Makefile.rules} is spread in the tree
@end table

Next directories are used:

@table @code
@item KERN_BUILD_DIR   := $(MAKERULES_DIR)/$(BUILD_DIR_NAME)/kern
        directory to store intermediate files for kernel-space targets
@item USER_BUILD_DIR   := $(MAKERULES_DIR)/$(BUILD_DIR_NAME)/user
        directory to store intermediate files for user-space targets

@item USER_INCLUDE_DIR := $(MAKERULES_DIR)/$(COMPILED_DIR_NAME)/include
        directory to store exported include files which should be installed later
        on user-space include path
@item USER_LIB_DIR     := $(MAKERULES_DIR)/$(COMPILED_DIR_NAME)/lib
        same for user-pace libraries
@item USER_UTILS_DIR   := $(MAKERULES_DIR)/$(COMPILED_DIR_NAME)/bin-utils
        utilities for testing, which would not probably be installed
@item USER_BIN_DIR     := $(MAKERULES_DIR)/$(COMPILED_DIR_NAME)/bin
        binaries, which should go into directory on standard system PATH
        (/usr/local/bin, /usr/bin or $(prefix)/bin)

@item KERN_INCLUDE_DIR := $(MAKERULES_DIR)/$(COMPILED_DIR_NAME)/include-kern
        directory to store exported include files which should be installed later
        on kernel-space include path
@item KERN_LIB_DIR     := $(MAKERULES_DIR)/$(COMPILED_DIR_NAME)/lib-kern
        same for kernel-pace libraries
@item KERN_MODULES_DIR := $(MAKERULES_DIR)/$(COMPILED_DIR_NAME)/modules
        builded modules for Linux kernel or RT-Linux system
@end table

There is more recursive passes through directories to enable
mutual dependant libraries and binaries to compile.
Next passes are defined

@table @samp
@item default-config
generates @file{config.omk-default} or xxx-default (FIXME) configuration file
@item check-dir 
checks and creates required build directories
@item include-pass   
copies header files to @code{USER_INCLUDE_DIR} and @code{KERN_INCLUDE_DIR}
@item library-pass
builds objects in USER_BUILD_DIR/@var{relative path} and creates libraries
in USER_LIB_DIR
@item binary-pass and utils-pass
links respective binaries in USER_@{BIN,UTILS@}_DIR directory. If some
object file is missing it compiles it in USER_BUILD_DIR/@var{relative path}
@item kernel-lib-pass
builds libraries for kernel space targets
@item kernel-pass 
builds kernel modules
@end table

The amount of passes is relatively high and consumes some time.  But
only other way to support all required features is to assemble one big
toplevel Makefile, which would contain all components and targets
cross-dependencies.

Drawbacks of designed make system
@itemize
@item
the system is not as fast as we would like
@item
it lacks Autoconf and configure extensive support for many systems
from UNIX to DOS and WINDOWS
@item
it does not contain support for checking existence of target
libraries and functionalities as GNU Autoconf
@item
it is heavily dependant on GNU MAKE program. But it would not be big
problem, because even many commercial applications distribute GNU MAKE
with them to be able to work in non-friendly systems
@item
the key drawback is dependence on recent MAKE version 3.80 and better
and even version 3.80 of MAKE has important bug, which has been
corrected in newer sources (FIXME)
@end itemize

The last point is critical. I have not noticed it first, because
I use Slackware-9.2 and it contains latest released version 
of MAKE (version 3.80).
The problem appears when I have tried to build bigger libraries.
There is bug in version 3.80, which results in misleading
error "Virtual memory exhausted". It is known bug with ID 1517

@smallexample
* long prerequisite inside eval(call()) => vm exhausted, Paul D. Smith
@end smallexample


I have optimized some rules to not push memory to the edge,
but there could be still issues with 3.80 version.

I have downloaded latest MAKE CVS sources. The compilation required
separate lookup and download for .po files and full Autoheader... cycle.
I have put together package similar to release. Only ./configure --prefix=...
and make is required. CVS sources contains version 3.81beta1.
You can download prepared sources archive from
  @indicateurl{http://paulandlesley.org/make/make-3.81beta1.tar.bz2}
Or you can get our local copy from
  @indicateurl{http://cmp.felk.cvut.cz/~pisa/can/make-3.81beta1.tar.gz}

The archive contains even "make" binary build by me, which should work
on other Linux distributions as well.  Older version of MAKE (3.79.x
released about year 2000) found on Mandrake and RedHat are not
sufficient and do not support eval feature.  I do not expect, that
Debian would be more up-to-date or contain fixes to MAKE vm exhausted
bug.

The local CTU archive with our CAN components prepared for inclusion
into OCERA SF CVS could be found in my "can" directory

  @indicateurl{http://cmp.felk.cvut.cz/~pisa/can/ocera-can-031212.tar.gz}

The code should build for user-space with new make on most of Linux distros
when make is updated.

If you want to test compile for RT-Linux targets, line

@example
#RTL_DIR := /home/cvs/ocera/ocera-build/kernel/rtlinux
@end example

in @file{Makefile.rules} has to be activated and updated
to point RT-Linux directory containing "rtl.mk".
There is only one library ("ulutrtl") and test utility compiled for RT-Linux
(@file{can/utils/ulut/ul_rtlchk.c}).

The next line, if enabled, controls compilation in OCERA project tree

@example
#OCERA_DIR := $(shell ( cd -L $(MAKERULES_DIR)/../../.. ; pwd -L ) )
@end example

The LinCAN driver has been updated to compile out of source directories.

Please, check, if you could compile CAN package and help us with integration
into OCERA SF CVS. Send your comments and objections. 

The OMK system has been adapted to support actual OCERA configuration process.
I am not happy with ocera.mk mix of defines and poor two or three rules,
but OMK is able to overcome that.

The OMK system has integrated rules (default-config) to build default
configuration file. The file is named @file{config.omk-default} for
the stand-alone compilation.  The name corresponds to OCERA config +
"-default" if OCERA_DIR is defined.  This file contains statements
from all @code{default_CONFIG} lines in all @file{Makefile.omk}.  The
file should be used for building of own @file{config.omk} file, or as
list for all options if Kconfig is used.

@chapter OMK User's Manual

@section Basic Concepts

The main concept of OMK is very simple. In the root directory of the
projects resides a file called @file{Makefile.rules}. This file contains
all compilation rules needed for compilation. There are different
@file{Makefile.rules} for different platforms (Unix, RTEMS, system-less,
...). In every subdirectory a @file{Makefile.omk} is stored. This file
specifies what should be done in this directory (e.g. compile a program
from several source files). Its syntax is very simple -- see the
following sections.

In theory, this could everything, what is needed to compile a project. In
reality, you want to be able to start compilation simply by typing
@command{make} in some directory. Since make searches by default for a
file @file{Makefile}, in every directory, there is small
@file{Makefile}, whose only task is to find @file{Makefile.rules} and
include it.

The compilation process itself is comprised of several passes. The pass
traverse the whole directory structure@footnote{In future, we are
planning some optimization that allows OMK to traverse the directories
only once and thus decrease compilation time.} and does some
task. Typically, these passes are:
@table @dfn
@item include-pass
    This pass takes all include files marked for ``export'' and copies
    (or links) them to the @file{include} directory (usually under
    @file{_compiled} directory). Also, during this pass, generated
    headers are generated according to the current configuration. See
    @ref{Configuration and Conditional Compilation} for details.
@item library-pass
    During this pass, all include files are in place, so all libraries
    can be compiled.
@item binary-pass
    Finally, programs can be compiled and linked against libraries
    created in the previous pass.
@end table

The results of compilation are stored under the @file{_compiled}
directory. This directory is structured as a classical Unix filesystem
(it contains directories like @file{bin}, @file{lib} and @file{include})
and can be directly copied to the target device or to some directory
(e.g. @file{/usr/local}).

Besides @file{_compiled} directory, there in a @file{_build}
directory. Under this directory are stored some temporary files and
intermediate compilation products (object files, dependency files etc.).

@section Dependency Tracking

OMK automatically handles tracking of dependencies of compiled
projects. It uses gcc's @option{-M@var{x}} options to do this for
object files. This way, whenever you change some header file, OMK
recompiles only those files, where the changed header was really
included.

Dependencies are also kept for libraries and binaries. OMK parses linker
map files, so a change to some library causes recompilation of all
programs using this library.

@section Compiling Simple Programs

To tell OMK to compile a program, you need to set some variables in
@file{Makefile.omk} (usually) in the directory where program sources are
located.

In the example bellow a program @command{test} will be compiled from
source @file{test.c}.

@example
@verbatiminclude ../tests/programs/Makefile.omk
@end example

@noindent The variables are:

@defvar bin_PROGRAMS
Contains a list of names (whitespace separated) of programs to be
compiled in this directory.
@end defvar

@defvar @var{program name}_SOURCES
For every program name in @code{bin_PROGRAMS}, this variable contains a
list of sources that are needed to compile the executable. OMK uses an
extension of the filename to determine the compiler to compile this
source.
@end defvar

@defvar @var{program name}_LIBS
This variable contains a list of libraries the program @var{program
name} needs to be linked to.
@end defvar

@defvar LOADLIBES
This variable contains a list of libraries all programs in this
directory needs to be linked to.
@end defvar

@defvar INCLUDES
Directives passed to the C or C++ compiler with additional directories
to be searched for header files. This variable applies to all
compilations invoked in the current directory.
@example
INCLUDES = -Imy_include_dir
@end example
@end defvar

@defvar DEFS
Directives passed to the C or C++ compiler with preprocessor macro
definitions. This variable applies to all compilations invoked in the
current directory.
@example
DEFS = -DDEBUG=1
@end example
@end defvar



@c FIXME: INCLUDES variable should not be set by rtlinux rules.


@section Libraries

LN_INCLUDES

@section Multiple Directories

ALL_OMK_SUBDIRS
SUBDIRS

@node Configuration and Conditional Compilation
@section Configuration and Conditional Compilation

@section Running OMK under Windows OS

@section Troubleshooting

Renaming of some file => dependency problems.

Manual compilation ... V=1



@chapter OMK Reference

@chapter OMK Development



@node Variable Index
@unnumbered Variable Index

@printindex vr

@node Concept Index
@unnumbered Concept Index

@printindex cp

@bye