[Dune] Dune and spack

Wed May 13 12:14:32 CEST 2020

Dear Gauthier,

I tried your prototype and indeed this sounds like a very promising
approach to Spack and Dune.
Thanks a lot for starting this!
I have created a repository on our Gitlab instance, where we try to keep
things related to Dune:
https://gitlab.dune-project.org/spack/dune-spack
and sent you an invitation to join. If you find more time to work on this
project, we can iterate a bit
more in that repository.

For all others: I keep you updated on the project when I did more
experiments.

Best,
Dominic

On Sun, May 10, 2020 at 10:03 PM Gauthier Folzan <gauthier.folzan at gmail.com>
wrote:

> Dear Dominic,
> If you want to have a look, I have pushed a prototype package on the repo
> https://github.com/gauthier12/dune_spack_repo
> It handles dune modules as variants and will fetch needed modules with the
> resource option.
> The package takes into account the different dune modules dependencies and
> download needed modules, by example asking to install dune-grid will
> install dune-geometry too.
> This is done by overloading the "_get_needed_resources".
> The compilation and linking uses dunecontrol so it is the standard
> installation procedure described on the website.
> It is a prototype so not all modules or options are available and
> specifying the version is not supported but it is functional at least for
> version 2.7.
> I hope It can be useful.
> Regards
> Gauthier
>
>
>
> On Mon, May 4, 2020 at 10:31 AM Dominic Kempf <dominic.r.kempf at gmail.com>
> wrote:
>
>> Dear Gauthier,
>>
>> If you are willing and able to invest the time to make a small prototype,
>> I would be more than happy!
>>
>> Best,
>> Dominic
>>
>> On Fri, May 1, 2020 at 3:10 PM Gauthier Folzan <gauthier.folzan at gmail.com>
>> wrote:
>>
>>> Dear Dominic,
>>> I totally agree with you :
>>> - the first solution is not appropriate (I didn't even mention it on my
>>> previous mail)
>>> - there are some tiring spack half-implemented, ill-documented spack
>>> features
>>>
>>> Concerning the "resource" feature, it seems to be in spack since a long
>>> time. It used in more than 40 packages in the builtin repo (there is ~4k
>>> packages) so it is not a lot of them but among the package using it, there
>>> are some common products (perl, git, gcc, trilinos, ruby, slepc...). So I
>>> don't know if we can consider it mature or commonly used.
>>> To be pragmatic, I think I can make a first example to see if it is
>>> doing the job and to enable you to make your mind about this approach, what
>>> do you think ?
>>> Regards,
>>> Gauthier
>>>
>>> On Wed, Apr 29, 2020 at 5:41 PM Dominic Kempf <dominic.r.kempf at gmail.com>
>>> wrote:
>>>
>>>> Dear Olivier,
>>>>
>>>> thanks for putting so much work into this.
>>>>
>>>> I have looked at the Github issue (for reference, it is this one:
>>>> https://github.com/spack/spack/pull/391).
>>>> The first presented answer is exactly what made me stop working on
>>>> this, I do not think that is an appropriate
>>>> solution for handling this (exponentially growing) problem.
>>>>
>>>> Concerning resource directives: I have never heard of them and I wasn't
>>>> able to quickly find documentation
>>>> on how they are meant to work! Do you have an example how such a thing
>>>> would look like? Also, is it a mature
>>>> feature of spack? I found myself struggling more than once with
>>>> half-implemented or abandoned features of
>>>> spack that were not clearly advertised as such (e.g. spack setup) so I
>>>> am a bit scared of niche features in spack.
>>>>
>>>> Best,
>>>> Dominic
>>>>
>>>> On Wed, Apr 29, 2020 at 4:31 PM Gauthier Folzan <
>>>> gauthier.folzan at gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>> I asked advices on the spack github and I had an interesting response
>>>>> from Massimiliano Culpo :
>>>>>
>>>>> Maybe a better solution, if it makes sense for Dune, is to treat Dune
>>>>> modules as resources. This means having a single "Dune" package with
>>>>> multiple resource directives. These resources might be activated by a
>>>>> corresponding variant and Dune should be able to install them on request.
>>>>>
>>>>>
>>>>> It was obvious for me to have one spack package by dune module. I
>>>>> didn't think about it but having one unique package makes sense. It ensures
>>>>> options coherence and version coherence between the modules within a
>>>>> package installation. Spack is already multiplying packages installations
>>>>> with different variant, so it's not a problem to have multiple and
>>>>> independent dune installation with different activated modules if needed.
>>>>>
>>>>> What do you think?
>>>>>
>>>>> Regards,
>>>>>
>>>>> Gauthier
>>>>>
>>>>> On Tue, Apr 28, 2020 at 1:19 PM Oliver Sander <
>>>>> oliver.sander at tu-dresden.de> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> as this topic seems to interest quite a number of people, can we not
>>>>>> seize the momentum and actually work on improving Dune towards
>>>>>> the Spack usecase?  I am not a build system expert, but I have yet
>>>>>> to meet somebody who is happy with the current state of module
>>>>>> dependency tracking.  What would a reasonable first step be?
>>>>>> How difficult would it be, e.g., to have Dune modules provide
>>>>>> at least rudimentary config-file packages?
>>>>>>
>>>>>> Best,
>>>>>> Oliver
>>>>>>
>>>>>> On 28.04.20 12:17, Christian Engwer wrote:
>>>>>> > Hi,
>>>>>> >
>>>>>> > this all sound rather nasty. Can we work around this by specifying
>>>>>> the
>>>>>> > correct cmake variables from spack and passing them to dependent
>>>>>> > modules?
>>>>>> >
>>>>>> > Christian
>>>>>> >
>>>>>> > On Tue, Apr 28, 2020 at 11:59:53AM +0200, Gauthier Folzan wrote:
>>>>>> >> Hi Dominic,
>>>>>> >> indeed, I realized that I needed to define the dune variants
>>>>>> everywhere
>>>>>> >> (what I did more or less).
>>>>>> >> I think defining a common group of variants everywhere is not a so
>>>>>> big deal
>>>>>> >> but the big problem is to propagate these variants across the
>>>>>> dependent
>>>>>> >> modules.
>>>>>> >> The approach found in some spack packages is to write all possible
>>>>>> >> combinations but this is cumbersome and I think this is what you
>>>>>> mean by
>>>>>> >> not elegant. Am I right ?
>>>>>> >> In a first approach, I wanted to have at least a first package to
>>>>>> be able
>>>>>> >> to install dune (at least for me), so what I did is to set every
>>>>>> variant to
>>>>>> >> true to have a consistent set of dune modules but I agree this is
>>>>>> not a
>>>>>> >> solution.
>>>>>> >> I found some discussions about variant propagation in spack forums
>>>>>> but
>>>>>> >> nothing seems to have been merged in the main branch.
>>>>>> >> To be sure, don't you have the same problem of flags coherence
>>>>>> across
>>>>>> >> modules using only cmake but compiling one by one the dune modules
>>>>>> ?
>>>>>> >> Regards,
>>>>>> >> Gauthier
>>>>>> >>
>>>>>> >> On Wed, Apr 22, 2020 at 11:05 AM Dominic Kempf <
>>>>>> dominic.r.kempf at gmail.com>
>>>>>> >> wrote:
>>>>>> >>
>>>>>> >>> Hey Gauthier,
>>>>>> >>>
>>>>>> >>> sorry for jumping in late.
>>>>>> >>>
>>>>>> >>> Christian has it right here, I was looking into Spack as a way of
>>>>>> setting
>>>>>> >>> up our CI and maybe moving towards testing variability in user
>>>>>> landscapes
>>>>>> >>> as well. The main reason I stopped working on it was that the
>>>>>> following
>>>>>> >>> particularity (some people call it a bug) of the Dune CMake build
>>>>>> system
>>>>>> >>> clashes with Spack:
>>>>>> >>> Each Dune module reruns cmake find scripts for dependencies of
>>>>>> all Dune
>>>>>> >>> modules it depends on instead of importing the full configuration
>>>>>> of the
>>>>>> >>> upstream modules. Due to this behaviour only modules that were
>>>>>> built in the
>>>>>> >>> same environment and with the same flags can be safely used
>>>>>> together (as
>>>>>> >>> their cmake find scripts found the same things).
>>>>>> >>> For Spack, that means that all variants that you define need to be
>>>>>> >>> redundantly defined across all modules, and dependencies between
>>>>>> those need
>>>>>> >>> to be specified manually. That is neither elegant nor feasible in
>>>>>> >>> maintenance.
>>>>>> >>> The Dune developers have voiced their willingness to abandon this
>>>>>> >>> behaviour, but a volunteer to do such a tedious, unrewarding work
>>>>>> is yet to
>>>>>> >>> be found.
>>>>>> >>>
>>>>>> >>> Best,
>>>>>> >>> Dominic
>>>>>> >>>
>>>>>> >>> On Wed, Apr 1, 2020 at 3:29 PM Gauthier Folzan <
>>>>>> gauthier.folzan at gmail.com>
>>>>>> >>> wrote:
>>>>>> >>>
>>>>>> >>>> Hi Christian,
>>>>>> >>>> here is the link :
>>>>>> https://github.com/gauthier12/dune_spack_repo.git
>>>>>> >>>> It is still work in progress and I am still testing so I don't
>>>>>> guarantee
>>>>>> >>>> anything but it seems to work on ubuntu, centos and archlinux for
>>>>>> >>>> dune-pyhon with dune-common, dune-geometry, dune-grid, dune-istl
>>>>>> and
>>>>>> >>>> dune-uggrid
>>>>>> >>>> Once you have spack installed, installing dune is quite simple,
>>>>>> just add
>>>>>> >>>> the repo and install dune
>>>>>> >>>> spack repo add dune_spack_repo
>>>>>> >>>> spack install dune-python
>>>>>> >>>> Cheers
>>>>>> >>>> Gauthier
>>>>>> >>>>
>>>>>> >>>> On Tue, Mar 31, 2020 at 4:46 PM Christian Engwer <
>>>>>> >>>> christian.engwer at uni-muenster.de> wrote:
>>>>>> >>>>
>>>>>> >>>>> Hi Gauthier,
>>>>>> >>>>>
>>>>>> >>>>> welcome to Dune,
>>>>>> >>>>>
>>>>>> >>>>>> Does someone use spack to compile Dune ? If it can be useful to
>>>>>> >>>>> someone,
>>>>>> >>>>>> the spack repo is on github.
>>>>>> >>>>>
>>>>>> >>>>> I think nobody is actively using spack for Dune currently,
>>>>>> Dominik had
>>>>>> >>>>> been workin on spack packages... I think his intention was to
>>>>>> ease the
>>>>>> >>>>> CI setup.
>>>>>> >>>>>
>>>>>> >>>>> On the other hand, if spack packages are available, it might
>>>>>> become
>>>>>> >>>>> more appealing for other people ;-) Can you post a link to your
>>>>>> repo?
>>>>>> >>>>>
>>>>>> >>>>> Thanks
>>>>>> >>>>> Christian
>>>>>> >>>>>
>>>>>> >>>> _______________________________________________
>>>>>> >>>> Dune mailing list
>>>>>> >>>> Dune at lists.dune-project.org
>>>>>> >>>> https://lists.dune-project.org/mailman/listinfo/dune
>>>>>> >>>
>>>>>> >>>
>>>>>> >
>>>>>> >> _______________________________________________
>>>>>> >> Dune mailing list
>>>>>> >> Dune at lists.dune-project.org
>>>>>> >> https://lists.dune-project.org/mailman/listinfo/dune
>>>>>> >
>>>>>> >
>>>>>>
>>>>>> _______________________________________________
>>>>> Dune mailing list
>>>>> Dune at lists.dune-project.org
>>>>> https://lists.dune-project.org/mailman/listinfo/dune
>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dune-project.org/pipermail/dune/attachments/20200513/2ef7b711/attachment.htm>