[Dune-devel] Buildsystem evolution in dune 2.9

[Simon Praetorius]

Dear developers,

The recent release of dune 2.8 opens the door for some more involved 
changes in dune. For example, we want to get the python bindings enabled 
by default with a new way of handling python in dune, see e.g. !960. 
Another field of development is the cmake build system - we have spoken 
a lot about this and I want to give you a road map for the next steps in 
this mail. My goal is to make small steps that can be reviewed and to 
stay backwards compatible

1. Cleanup:

- We have set the cmake required version to 3.13. This can be used to 
bring lots of code to the recent standard and to eliminate several 
workarounds (e.g. !1010, !1008, !862)
- The implementation of the cmake system is grown over time and lots of 
good and useful functionality was added, but this not always 
consistently. There are lost macros that are not used at all, there are 
even invalid functions that could not have been used, there is no naming 
convention, and most of the code is packed in one big cmake file with 
about 1200 lines of code, half of the functions/macros undocumented and 
in random order, not so easy to read. I suggest to clean this up in two 
steps:
   a) Split the big file `DuneMacros.cmake` into parts containing 
functions with the same purpose, e.g., `dune_project` and 
`finalize_dune_project` into a file `DuneProject.cmake`. Group by public 
(interface) functions and internal implementation details.
   b) Introduce the naming convention that ALL dune cmake macros and 
functions must start with "dune_"
One could either split up these two steps or make a combined approach, 
see !957. Renaming functions introduces some difficulties. For all 
renamed functions the old one should still be included but with a 
deprecation. Remaining problem: what if a code whats to use multiple 
dune-common versions and needs to call a cmake function that is renamed 
in a recent release? This cannot be solved easily and needs some version 
checks like we have as macros DUNE_VERSION_LT(...) and friends also in 
cmake.

2. Targets communicate and propagate properties:

Since cmake 3.x targets get more and more important. A target is a 
container for all information necessary to compile code. It contains 
build information and usage information. And we can fill these 
properties. Two steps are already prepared in this direction, but this 
should just be the start.

- Set the required c++ standard version as property of dunecommon 
target. Every library and executable that links against dunecommon gets 
the same standard requirement automatically. See !862.
- Other properties like include-directories and external link-libraries 
can also be made a property of the target, see e.g., !944

3. Put more information into <module>-config.cmake

Whenever a `find_package(<module>)` is called in cmake, a corresponding 
`<module>-config.cmake` is search and executed. This file should contain 
everything necessary to use that module and to link against it. If we 
could make the information stored in there independent of the actual 
dune build-system, it would make using dune modules with cmake much 
easier. Currently we have our own replacement for the <module>-config 
files: Dune<Module>Macros.cmake. Essentially this file does both, it 
configures the current module and also any downstream module after the 
find_package call. Parts of this file should be put either in the main 
CMakeLists file or in the corresponding <module>-config.cmake file.

This is still work in progress and no final solution exists

4. Handling of config.h files

- We have discussed some ideas how to replace/change the current 
situation, see #234.
   a) Either we create module specific config files (e.g. 
module-config.h) that must be included in all header files of that 
module (not the source files)
   b) or we find a way to put all the preprocessor defines into 
compile-options or so-called response files (generated automatically by 
cmake)
- Each solution has some drawbacks but also advantages over the current 
situation. Both approaches are tested in merge requests, see !924 and !925

5. Automatic generation of PkgConfig files

This topic is a bit related to 3. and 4. We have to fix the config.h 
problem first. But then is the question whether we can generate 
pkg-config files like dune-<module>.pc that also contains every 
information necessary to work with that module, all necessary compile 
flags and all dependent libraries/modules also with a pkg-config file. 
This would allow to configure dune modules with any build system we 
like, even with simple makefiles or very performant meson-build


The cleanup step is very important for me to get a better overview over 
the large buildsystem. It is often hard to see the dependencies between 
different mechanisms. In the long run it might be preferable to 
eliminate some functionality and replace it with a better or simpler 
alternative. Maybe it will not be necessary to parse a dune.module file 
within cmake if that information could be encoded directly within the 
main CMakeLists files. The `find_package` utility of cmake has already a 
powerful search procedure implemented. Maybe it will not be necessary 
anymore to use dunecontrol for resolving dependencies if we could let 
cmake figure this out automatically. But I'm not yet sure about this. 
Finally, I would like to be able to use a so-called "super-build" 
approach or the "Fetch-Content" utilities. to be completely independent 
of dunecontrol for the configuration and building of all dune modules. 
But this is still a long way to go.

While we have small and large system-tests included in our continuous 
integration and git workflow, it might still be that these buildsystem 
changes break some code. The buildsystem is very fundamental and it is 
not so easy to see all dependencies and consequences from the beginning. 
That is why it is good that we have a recent stable release. If we go 
some of the steps illustrated above, please try to give feedback if 
something does not work as expected in your setup. We can always go one 
step back and rethink about an approach.


I hope I could give you an idea of some of the next steps and I also 
hope that you will help me to review the more concrete merge requests 
and to think about some details.

Best regards,
Simon

-- 
Dr. Simon Praetorius
Technische Universität Dresden
Institute of Scientific Computing
phone: +49 351 463-34432
mail: simon.praetorius at tu-dresden.de
web: https://tu-dresden.de/mn/math/wir/das-institut/beschaeftigte/simon-praetorius

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dune-project.org/pipermail/dune-devel/attachments/20210909/059abae3/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5204 bytes
Desc: S/MIME Cryptographic Signature
URL: <https://lists.dune-project.org/pipermail/dune-devel/attachments/20210909/059abae3/attachment.bin>