[Dune] Dune and spack

Sun May 10 22:02:40 CEST 2020

Dear Dominic,
If you want to have a look, I have pushed a prototype package on the repo
https://github.com/gauthier12/dune_spack_repo
It handles dune modules as variants and will fetch needed modules with the
resource option.
The package takes into account the different dune modules dependencies and
download needed modules, by example asking to install dune-grid will
install dune-geometry too.
This is done by overloading the "_get_needed_resources".
The compilation and linking uses dunecontrol so it is the standard
installation procedure described on the website.
It is a prototype so not all modules or options are available and
specifying the version is not supported but it is functional at least for
version 2.7.
I hope It can be useful.
Regards
Gauthier

On Mon, May 4, 2020 at 10:31 AM Dominic Kempf <dominic.r.kempf at gmail.com>
wrote:

> Dear Gauthier,
>
> If you are willing and able to invest the time to make a small prototype,
> I would be more than happy!
>
> Best,
> Dominic
>
> On Fri, May 1, 2020 at 3:10 PM Gauthier Folzan <gauthier.folzan at gmail.com>
> wrote:
>
>> Dear Dominic,
>> I totally agree with you :
>> - the first solution is not appropriate (I didn't even mention it on my
>> previous mail)
>> - there are some tiring spack half-implemented, ill-documented spack
>> features
>>
>> Concerning the "resource" feature, it seems to be in spack since a long
>> time. It used in more than 40 packages in the builtin repo (there is ~4k
>> packages) so it is not a lot of them but among the package using it, there
>> are some common products (perl, git, gcc, trilinos, ruby, slepc...). So I
>> don't know if we can consider it mature or commonly used.
>> To be pragmatic, I think I can make a first example to see if it is doing
>> the job and to enable you to make your mind about this approach, what do
>> you think ?
>> Regards,
>> Gauthier
>>
>> On Wed, Apr 29, 2020 at 5:41 PM Dominic Kempf <dominic.r.kempf at gmail.com>
>> wrote:
>>
>>> Dear Olivier,
>>>
>>> thanks for putting so much work into this.
>>>
>>> I have looked at the Github issue (for reference, it is this one:
>>> https://github.com/spack/spack/pull/391).
>>> The first presented answer is exactly what made me stop working on this,
>>> I do not think that is an appropriate
>>> solution for handling this (exponentially growing) problem.
>>>
>>> Concerning resource directives: I have never heard of them and I wasn't
>>> able to quickly find documentation
>>> on how they are meant to work! Do you have an example how such a thing
>>> would look like? Also, is it a mature
>>> feature of spack? I found myself struggling more than once with
>>> half-implemented or abandoned features of
>>> spack that were not clearly advertised as such (e.g. spack setup) so I
>>> am a bit scared of niche features in spack.
>>>
>>> Best,
>>> Dominic
>>>
>>> On Wed, Apr 29, 2020 at 4:31 PM Gauthier Folzan <
>>> gauthier.folzan at gmail.com> wrote:
>>>
>>>> Hi,
>>>> I asked advices on the spack github and I had an interesting response
>>>> from Massimiliano Culpo :
>>>>
>>>> Maybe a better solution, if it makes sense for Dune, is to treat Dune
>>>> modules as resources. This means having a single "Dune" package with
>>>> multiple resource directives. These resources might be activated by a
>>>> corresponding variant and Dune should be able to install them on request.
>>>>
>>>>
>>>> It was obvious for me to have one spack package by dune module. I
>>>> didn't think about it but having one unique package makes sense. It ensures
>>>> options coherence and version coherence between the modules within a
>>>> package installation. Spack is already multiplying packages installations
>>>> with different variant, so it's not a problem to have multiple and
>>>> independent dune installation with different activated modules if needed.
>>>>
>>>> What do you think?
>>>>
>>>> Regards,
>>>>
>>>> Gauthier
>>>>
>>>> On Tue, Apr 28, 2020 at 1:19 PM Oliver Sander <
>>>> oliver.sander at tu-dresden.de> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> as this topic seems to interest quite a number of people, can we not
>>>>> seize the momentum and actually work on improving Dune towards
>>>>> the Spack usecase?  I am not a build system expert, but I have yet
>>>>> to meet somebody who is happy with the current state of module
>>>>> dependency tracking.  What would a reasonable first step be?
>>>>> How difficult would it be, e.g., to have Dune modules provide
>>>>> at least rudimentary config-file packages?
>>>>>
>>>>> Best,
>>>>> Oliver
>>>>>
>>>>> On 28.04.20 12:17, Christian Engwer wrote:
>>>>> > Hi,
>>>>> >
>>>>> > this all sound rather nasty. Can we work around this by specifying
>>>>> the
>>>>> > correct cmake variables from spack and passing them to dependent
>>>>> > modules?
>>>>> >
>>>>> > Christian
>>>>> >
>>>>> > On Tue, Apr 28, 2020 at 11:59:53AM +0200, Gauthier Folzan wrote:
>>>>> >> Hi Dominic,
>>>>> >> indeed, I realized that I needed to define the dune variants
>>>>> everywhere
>>>>> >> (what I did more or less).
>>>>> >> I think defining a common group of variants everywhere is not a so
>>>>> big deal
>>>>> >> but the big problem is to propagate these variants across the
>>>>> dependent
>>>>> >> modules.
>>>>> >> The approach found in some spack packages is to write all possible
>>>>> >> combinations but this is cumbersome and I think this is what you
>>>>> mean by
>>>>> >> not elegant. Am I right ?
>>>>> >> In a first approach, I wanted to have at least a first package to
>>>>> be able
>>>>> >> to install dune (at least for me), so what I did is to set every
>>>>> variant to
>>>>> >> true to have a consistent set of dune modules but I agree this is
>>>>> not a
>>>>> >> solution.
>>>>> >> I found some discussions about variant propagation in spack forums
>>>>> but
>>>>> >> nothing seems to have been merged in the main branch.
>>>>> >> To be sure, don't you have the same problem of flags coherence
>>>>> across
>>>>> >> modules using only cmake but compiling one by one the dune modules ?
>>>>> >> Regards,
>>>>> >> Gauthier
>>>>> >>
>>>>> >> On Wed, Apr 22, 2020 at 11:05 AM Dominic Kempf <
>>>>> dominic.r.kempf at gmail.com>
>>>>> >> wrote:
>>>>> >>
>>>>> >>> Hey Gauthier,
>>>>> >>>
>>>>> >>> sorry for jumping in late.
>>>>> >>>
>>>>> >>> Christian has it right here, I was looking into Spack as a way of
>>>>> setting
>>>>> >>> up our CI and maybe moving towards testing variability in user
>>>>> landscapes
>>>>> >>> as well. The main reason I stopped working on it was that the
>>>>> following
>>>>> >>> particularity (some people call it a bug) of the Dune CMake build
>>>>> system
>>>>> >>> clashes with Spack:
>>>>> >>> Each Dune module reruns cmake find scripts for dependencies of all
>>>>> Dune
>>>>> >>> modules it depends on instead of importing the full configuration
>>>>> of the
>>>>> >>> upstream modules. Due to this behaviour only modules that were
>>>>> built in the
>>>>> >>> same environment and with the same flags can be safely used
>>>>> together (as
>>>>> >>> their cmake find scripts found the same things).
>>>>> >>> For Spack, that means that all variants that you define need to be
>>>>> >>> redundantly defined across all modules, and dependencies between
>>>>> those need
>>>>> >>> to be specified manually. That is neither elegant nor feasible in
>>>>> >>> maintenance.
>>>>> >>> The Dune developers have voiced their willingness to abandon this
>>>>> >>> behaviour, but a volunteer to do such a tedious, unrewarding work
>>>>> is yet to
>>>>> >>> be found.
>>>>> >>>
>>>>> >>> Best,
>>>>> >>> Dominic
>>>>> >>>
>>>>> >>> On Wed, Apr 1, 2020 at 3:29 PM Gauthier Folzan <
>>>>> gauthier.folzan at gmail.com>
>>>>> >>> wrote:
>>>>> >>>
>>>>> >>>> Hi Christian,
>>>>> >>>> here is the link :
>>>>> https://github.com/gauthier12/dune_spack_repo.git
>>>>> >>>> It is still work in progress and I am still testing so I don't
>>>>> guarantee
>>>>> >>>> anything but it seems to work on ubuntu, centos and archlinux for
>>>>> >>>> dune-pyhon with dune-common, dune-geometry, dune-grid, dune-istl
>>>>> and
>>>>> >>>> dune-uggrid
>>>>> >>>> Once you have spack installed, installing dune is quite simple,
>>>>> just add
>>>>> >>>> the repo and install dune
>>>>> >>>> spack repo add dune_spack_repo
>>>>> >>>> spack install dune-python
>>>>> >>>> Cheers
>>>>> >>>> Gauthier
>>>>> >>>>
>>>>> >>>> On Tue, Mar 31, 2020 at 4:46 PM Christian Engwer <
>>>>> >>>> christian.engwer at uni-muenster.de> wrote:
>>>>> >>>>
>>>>> >>>>> Hi Gauthier,
>>>>> >>>>>
>>>>> >>>>> welcome to Dune,
>>>>> >>>>>
>>>>> >>>>>> Does someone use spack to compile Dune ? If it can be useful to
>>>>> >>>>> someone,
>>>>> >>>>>> the spack repo is on github.
>>>>> >>>>>
>>>>> >>>>> I think nobody is actively using spack for Dune currently,
>>>>> Dominik had
>>>>> >>>>> been workin on spack packages... I think his intention was to
>>>>> ease the
>>>>> >>>>> CI setup.
>>>>> >>>>>
>>>>> >>>>> On the other hand, if spack packages are available, it might
>>>>> become
>>>>> >>>>> more appealing for other people ;-) Can you post a link to your
>>>>> repo?
>>>>> >>>>>
>>>>> >>>>> Thanks
>>>>> >>>>> Christian
>>>>> >>>>>
>>>>> >>>> _______________________________________________
>>>>> >>>> Dune mailing list
>>>>> >>>> Dune at lists.dune-project.org
>>>>> >>>> https://lists.dune-project.org/mailman/listinfo/dune
>>>>> >>>
>>>>> >>>
>>>>> >
>>>>> >> _______________________________________________
>>>>> >> Dune mailing list
>>>>> >> Dune at lists.dune-project.org
>>>>> >> https://lists.dune-project.org/mailman/listinfo/dune
>>>>> >
>>>>> >
>>>>>
>>>>> _______________________________________________
>>>> Dune mailing list
>>>> Dune at lists.dune-project.org
>>>> https://lists.dune-project.org/mailman/listinfo/dune
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dune-project.org/pipermail/dune/attachments/20200510/3bf71e53/attachment.htm>