[Dune] Dune and spack

Tue Apr 28 12:37:50 CEST 2020

Hi everyone,

I'm not an expert in spack, just had a quick look into it, but I'm 
really interested in having something like this.

About a year ago, I worked on another packaging system - easybuild - to 
provide dune on our HPC cluster. And naturally, I faced similar 
problems. My decision was to provide just a very small collection of 
variants (e.g. different compiler toolchains, or with/without MPI) and 
followed the principle: provide as many external packages/dependencies 
as possible.  So the user can decide later in his own project which 
packages to use.

The configurations were done in a way that each module defines CMake 
variables and exports those in a environment variable CMAKE_FLAGS in a 
module load step. When compiling modules depending on this already build 
dune-module, the environment CMAKE_FLAGS variable was appended to module 
dependent own flags and passed to CMake. And at the end of the 
configuration this new module also exports its own flags to the 
CMAKE_FLAGS variable. This worked fine for me.

See the repository

https://gitlab.mn.tu-dresden.de/spraetor/dune-easybuild

for an idea of what I've done for easybuild. There are several branches 
with different configurations. It is not cleaned up, though.

Best,
Simon

Am 28.04.20 um 12:17 schrieb Christian Engwer:
> Hi,
>
> this all sound rather nasty. Can we work around this by specifying the
> correct cmake variables from spack and passing them to dependent
> modules?
>
> Christian
>
> On Tue, Apr 28, 2020 at 11:59:53AM +0200, Gauthier Folzan wrote:
>> Hi Dominic,
>> indeed, I realized that I needed to define the dune variants everywhere
>> (what I did more or less).
>> I think defining a common group of variants everywhere is not a so big deal
>> but the big problem is to propagate these variants across the dependent
>> modules.
>> The approach found in some spack packages is to write all possible
>> combinations but this is cumbersome and I think this is what you mean by
>> not elegant. Am I right ?
>> In a first approach, I wanted to have at least a first package to be able
>> to install dune (at least for me), so what I did is to set every variant to
>> true to have a consistent set of dune modules but I agree this is not a
>> solution.
>> I found some discussions about variant propagation in spack forums but
>> nothing seems to have been merged in the main branch.
>> To be sure, don't you have the same problem of flags coherence across
>> modules using only cmake but compiling one by one the dune modules ?
>> Regards,
>> Gauthier
>>
>> On Wed, Apr 22, 2020 at 11:05 AM Dominic Kempf <dominic.r.kempf at gmail.com>
>> wrote:
>>
>>> Hey Gauthier,
>>>
>>> sorry for jumping in late.
>>>
>>> Christian has it right here, I was looking into Spack as a way of setting
>>> up our CI and maybe moving towards testing variability in user landscapes
>>> as well. The main reason I stopped working on it was that the following
>>> particularity (some people call it a bug) of the Dune CMake build system
>>> clashes with Spack:
>>> Each Dune module reruns cmake find scripts for dependencies of all Dune
>>> modules it depends on instead of importing the full configuration of the
>>> upstream modules. Due to this behaviour only modules that were built in the
>>> same environment and with the same flags can be safely used together (as
>>> their cmake find scripts found the same things).
>>> For Spack, that means that all variants that you define need to be
>>> redundantly defined across all modules, and dependencies between those need
>>> to be specified manually. That is neither elegant nor feasible in
>>> maintenance.
>>> The Dune developers have voiced their willingness to abandon this
>>> behaviour, but a volunteer to do such a tedious, unrewarding work is yet to
>>> be found.
>>>
>>> Best,
>>> Dominic
>>>
>>> On Wed, Apr 1, 2020 at 3:29 PM Gauthier Folzan <gauthier.folzan at gmail.com>
>>> wrote:
>>>
>>>> Hi Christian,
>>>> here is the link : https://github.com/gauthier12/dune_spack_repo.git
>>>> It is still work in progress and I am still testing so I don't guarantee
>>>> anything but it seems to work on ubuntu, centos and archlinux for
>>>> dune-pyhon with dune-common, dune-geometry, dune-grid, dune-istl and
>>>> dune-uggrid
>>>> Once you have spack installed, installing dune is quite simple, just add
>>>> the repo and install dune
>>>> spack repo add dune_spack_repo
>>>> spack install dune-python
>>>> Cheers
>>>> Gauthier
>>>>
>>>> On Tue, Mar 31, 2020 at 4:46 PM Christian Engwer <
>>>> christian.engwer at uni-muenster.de> wrote:
>>>>
>>>>> Hi Gauthier,
>>>>>
>>>>> welcome to Dune,
>>>>>
>>>>>> Does someone use spack to compile Dune ? If it can be useful to
>>>>> someone,
>>>>>> the spack repo is on github.
>>>>> I think nobody is actively using spack for Dune currently, Dominik had
>>>>> been workin on spack packages... I think his intention was to ease the
>>>>> CI setup.
>>>>>
>>>>> On the other hand, if spack packages are available, it might become
>>>>> more appealing for other people ;-) Can you post a link to your repo?
>>>>>
>>>>> Thanks
>>>>> Christian
>>>>>
>>>> _______________________________________________
>>>> Dune mailing list
>>>> Dune at lists.dune-project.org
>>>> https://lists.dune-project.org/mailman/listinfo/dune
>>>
>> _______________________________________________
>> Dune mailing list
>> Dune at lists.dune-project.org
>> https://lists.dune-project.org/mailman/listinfo/dune
>
-- 
Dr. Simon Praetorius
Technische Universität Dresden
Institute of Scientific Computing
phone: +49 351 463-34432
mail: simon.praetorius@tu-dresden.de