[Dune] Dune and spack

[Gauthier Folzan]

Dear Dominic,
I totally agree with you :
- the first solution is not appropriate (I didn't even mention it on my
previous mail)
- there are some tiring spack half-implemented, ill-documented spack
features

Concerning the "resource" feature, it seems to be in spack since a long
time. It used in more than 40 packages in the builtin repo (there is ~4k
packages) so it is not a lot of them but among the package using it, there
are some common products (perl, git, gcc, trilinos, ruby, slepc...). So I
don't know if we can consider it mature or commonly used.
To be pragmatic, I think I can make a first example to see if it is doing
the job and to enable you to make your mind about this approach, what do
you think ?
Regards,
Gauthier

On Wed, Apr 29, 2020 at 5:41 PM Dominic Kempf <dominic.r.kempf at gmail.com>
wrote:

> Dear Olivier,
>
> thanks for putting so much work into this.
>
> I have looked at the Github issue (for reference, it is this one:
> https://github.com/spack/spack/pull/391).
> The first presented answer is exactly what made me stop working on this, I
> do not think that is an appropriate
> solution for handling this (exponentially growing) problem.
>
> Concerning resource directives: I have never heard of them and I wasn't
> able to quickly find documentation
> on how they are meant to work! Do you have an example how such a thing
> would look like? Also, is it a mature
> feature of spack? I found myself struggling more than once with
> half-implemented or abandoned features of
> spack that were not clearly advertised as such (e.g. spack setup) so I am
> a bit scared of niche features in spack.
>
> Best,
> Dominic
>
> On Wed, Apr 29, 2020 at 4:31 PM Gauthier Folzan <gauthier.folzan at gmail.com>
> wrote:
>
>> Hi,
>> I asked advices on the spack github and I had an interesting response
>> from Massimiliano Culpo :
>>
>> Maybe a better solution, if it makes sense for Dune, is to treat Dune
>> modules as resources. This means having a single "Dune" package with
>> multiple resource directives. These resources might be activated by a
>> corresponding variant and Dune should be able to install them on request.
>>
>>
>> It was obvious for me to have one spack package by dune module. I didn't
>> think about it but having one unique package makes sense. It ensures
>> options coherence and version coherence between the modules within a
>> package installation. Spack is already multiplying packages installations
>> with different variant, so it's not a problem to have multiple and
>> independent dune installation with different activated modules if needed.
>>
>> What do you think?
>>
>> Regards,
>>
>> Gauthier
>>
>> On Tue, Apr 28, 2020 at 1:19 PM Oliver Sander <
>> oliver.sander at tu-dresden.de> wrote:
>>
>>> Hi,
>>>
>>> as this topic seems to interest quite a number of people, can we not
>>> seize the momentum and actually work on improving Dune towards
>>> the Spack usecase?  I am not a build system expert, but I have yet
>>> to meet somebody who is happy with the current state of module
>>> dependency tracking.  What would a reasonable first step be?
>>> How difficult would it be, e.g., to have Dune modules provide
>>> at least rudimentary config-file packages?
>>>
>>> Best,
>>> Oliver
>>>
>>> On 28.04.20 12:17, Christian Engwer wrote:
>>> > Hi,
>>> >
>>> > this all sound rather nasty. Can we work around this by specifying the
>>> > correct cmake variables from spack and passing them to dependent
>>> > modules?
>>> >
>>> > Christian
>>> >
>>> > On Tue, Apr 28, 2020 at 11:59:53AM +0200, Gauthier Folzan wrote:
>>> >> Hi Dominic,
>>> >> indeed, I realized that I needed to define the dune variants
>>> everywhere
>>> >> (what I did more or less).
>>> >> I think defining a common group of variants everywhere is not a so
>>> big deal
>>> >> but the big problem is to propagate these variants across the
>>> dependent
>>> >> modules.
>>> >> The approach found in some spack packages is to write all possible
>>> >> combinations but this is cumbersome and I think this is what you mean
>>> by
>>> >> not elegant. Am I right ?
>>> >> In a first approach, I wanted to have at least a first package to be
>>> able
>>> >> to install dune (at least for me), so what I did is to set every
>>> variant to
>>> >> true to have a consistent set of dune modules but I agree this is not
>>> a
>>> >> solution.
>>> >> I found some discussions about variant propagation in spack forums but
>>> >> nothing seems to have been merged in the main branch.
>>> >> To be sure, don't you have the same problem of flags coherence across
>>> >> modules using only cmake but compiling one by one the dune modules ?
>>> >> Regards,
>>> >> Gauthier
>>> >>
>>> >> On Wed, Apr 22, 2020 at 11:05 AM Dominic Kempf <
>>> dominic.r.kempf at gmail.com>
>>> >> wrote:
>>> >>
>>> >>> Hey Gauthier,
>>> >>>
>>> >>> sorry for jumping in late.
>>> >>>
>>> >>> Christian has it right here, I was looking into Spack as a way of
>>> setting
>>> >>> up our CI and maybe moving towards testing variability in user
>>> landscapes
>>> >>> as well. The main reason I stopped working on it was that the
>>> following
>>> >>> particularity (some people call it a bug) of the Dune CMake build
>>> system
>>> >>> clashes with Spack:
>>> >>> Each Dune module reruns cmake find scripts for dependencies of all
>>> Dune
>>> >>> modules it depends on instead of importing the full configuration of
>>> the
>>> >>> upstream modules. Due to this behaviour only modules that were built
>>> in the
>>> >>> same environment and with the same flags can be safely used together
>>> (as
>>> >>> their cmake find scripts found the same things).
>>> >>> For Spack, that means that all variants that you define need to be
>>> >>> redundantly defined across all modules, and dependencies between
>>> those need
>>> >>> to be specified manually. That is neither elegant nor feasible in
>>> >>> maintenance.
>>> >>> The Dune developers have voiced their willingness to abandon this
>>> >>> behaviour, but a volunteer to do such a tedious, unrewarding work is
>>> yet to
>>> >>> be found.
>>> >>>
>>> >>> Best,
>>> >>> Dominic
>>> >>>
>>> >>> On Wed, Apr 1, 2020 at 3:29 PM Gauthier Folzan <
>>> gauthier.folzan at gmail.com>
>>> >>> wrote:
>>> >>>
>>> >>>> Hi Christian,
>>> >>>> here is the link :
>>> https://github.com/gauthier12/dune_spack_repo.git
>>> >>>> It is still work in progress and I am still testing so I don't
>>> guarantee
>>> >>>> anything but it seems to work on ubuntu, centos and archlinux for
>>> >>>> dune-pyhon with dune-common, dune-geometry, dune-grid, dune-istl and
>>> >>>> dune-uggrid
>>> >>>> Once you have spack installed, installing dune is quite simple,
>>> just add
>>> >>>> the repo and install dune
>>> >>>> spack repo add dune_spack_repo
>>> >>>> spack install dune-python
>>> >>>> Cheers
>>> >>>> Gauthier
>>> >>>>
>>> >>>> On Tue, Mar 31, 2020 at 4:46 PM Christian Engwer <
>>> >>>> christian.engwer at uni-muenster.de> wrote:
>>> >>>>
>>> >>>>> Hi Gauthier,
>>> >>>>>
>>> >>>>> welcome to Dune,
>>> >>>>>
>>> >>>>>> Does someone use spack to compile Dune ? If it can be useful to
>>> >>>>> someone,
>>> >>>>>> the spack repo is on github.
>>> >>>>>
>>> >>>>> I think nobody is actively using spack for Dune currently, Dominik
>>> had
>>> >>>>> been workin on spack packages... I think his intention was to ease
>>> the
>>> >>>>> CI setup.
>>> >>>>>
>>> >>>>> On the other hand, if spack packages are available, it might become
>>> >>>>> more appealing for other people ;-) Can you post a link to your
>>> repo?
>>> >>>>>
>>> >>>>> Thanks
>>> >>>>> Christian
>>> >>>>>
>>> >>>> _______________________________________________
>>> >>>> Dune mailing list
>>> >>>> Dune at lists.dune-project.org
>>> >>>> https://lists.dune-project.org/mailman/listinfo/dune
>>> >>>
>>> >>>
>>> >
>>> >> _______________________________________________
>>> >> Dune mailing list
>>> >> Dune at lists.dune-project.org
>>> >> https://lists.dune-project.org/mailman/listinfo/dune
>>> >
>>> >
>>>
>>> _______________________________________________
>> Dune mailing list
>> Dune at lists.dune-project.org
>> https://lists.dune-project.org/mailman/listinfo/dune
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dune-project.org/pipermail/dune/attachments/20200501/fa0cd265/attachment.htm>