[Dune] Dune and spack

Mon May 4 10:31:31 CEST 2020

Dear Gauthier,

If you are willing and able to invest the time to make a small prototype, I
would be more than happy!

Best,
Dominic

On Fri, May 1, 2020 at 3:10 PM Gauthier Folzan <gauthier.folzan at gmail.com>
wrote:

> Dear Dominic,
> I totally agree with you :
> - the first solution is not appropriate (I didn't even mention it on my
> previous mail)
> - there are some tiring spack half-implemented, ill-documented spack
> features
>
> Concerning the "resource" feature, it seems to be in spack since a long
> time. It used in more than 40 packages in the builtin repo (there is ~4k
> packages) so it is not a lot of them but among the package using it, there
> are some common products (perl, git, gcc, trilinos, ruby, slepc...). So I
> don't know if we can consider it mature or commonly used.
> To be pragmatic, I think I can make a first example to see if it is doing
> the job and to enable you to make your mind about this approach, what do
> you think ?
> Regards,
> Gauthier
>
> On Wed, Apr 29, 2020 at 5:41 PM Dominic Kempf <dominic.r.kempf at gmail.com>
> wrote:
>
>> Dear Olivier,
>>
>> thanks for putting so much work into this.
>>
>> I have looked at the Github issue (for reference, it is this one:
>> https://github.com/spack/spack/pull/391).
>> The first presented answer is exactly what made me stop working on this,
>> I do not think that is an appropriate
>> solution for handling this (exponentially growing) problem.
>>
>> Concerning resource directives: I have never heard of them and I wasn't
>> able to quickly find documentation
>> on how they are meant to work! Do you have an example how such a thing
>> would look like? Also, is it a mature
>> feature of spack? I found myself struggling more than once with
>> half-implemented or abandoned features of
>> spack that were not clearly advertised as such (e.g. spack setup) so I am
>> a bit scared of niche features in spack.
>>
>> Best,
>> Dominic
>>
>> On Wed, Apr 29, 2020 at 4:31 PM Gauthier Folzan <
>> gauthier.folzan at gmail.com> wrote:
>>
>>> Hi,
>>> I asked advices on the spack github and I had an interesting response
>>> from Massimiliano Culpo :
>>>
>>> Maybe a better solution, if it makes sense for Dune, is to treat Dune
>>> modules as resources. This means having a single "Dune" package with
>>> multiple resource directives. These resources might be activated by a
>>> corresponding variant and Dune should be able to install them on request.
>>>
>>>
>>> It was obvious for me to have one spack package by dune module. I didn't
>>> think about it but having one unique package makes sense. It ensures
>>> options coherence and version coherence between the modules within a
>>> package installation. Spack is already multiplying packages installations
>>> with different variant, so it's not a problem to have multiple and
>>> independent dune installation with different activated modules if needed.
>>>
>>> What do you think?
>>>
>>> Regards,
>>>
>>> Gauthier
>>>
>>> On Tue, Apr 28, 2020 at 1:19 PM Oliver Sander <
>>> oliver.sander at tu-dresden.de> wrote:
>>>
>>>> Hi,
>>>>
>>>> as this topic seems to interest quite a number of people, can we not
>>>> seize the momentum and actually work on improving Dune towards
>>>> the Spack usecase?  I am not a build system expert, but I have yet
>>>> to meet somebody who is happy with the current state of module
>>>> dependency tracking.  What would a reasonable first step be?
>>>> How difficult would it be, e.g., to have Dune modules provide
>>>> at least rudimentary config-file packages?
>>>>
>>>> Best,
>>>> Oliver
>>>>
>>>> On 28.04.20 12:17, Christian Engwer wrote:
>>>> > Hi,
>>>> >
>>>> > this all sound rather nasty. Can we work around this by specifying the
>>>> > correct cmake variables from spack and passing them to dependent
>>>> > modules?
>>>> >
>>>> > Christian
>>>> >
>>>> > On Tue, Apr 28, 2020 at 11:59:53AM +0200, Gauthier Folzan wrote:
>>>> >> Hi Dominic,
>>>> >> indeed, I realized that I needed to define the dune variants
>>>> everywhere
>>>> >> (what I did more or less).
>>>> >> I think defining a common group of variants everywhere is not a so
>>>> big deal
>>>> >> but the big problem is to propagate these variants across the
>>>> dependent
>>>> >> modules.
>>>> >> The approach found in some spack packages is to write all possible
>>>> >> combinations but this is cumbersome and I think this is what you
>>>> mean by
>>>> >> not elegant. Am I right ?
>>>> >> In a first approach, I wanted to have at least a first package to be
>>>> able
>>>> >> to install dune (at least for me), so what I did is to set every
>>>> variant to
>>>> >> true to have a consistent set of dune modules but I agree this is
>>>> not a
>>>> >> solution.
>>>> >> I found some discussions about variant propagation in spack forums
>>>> but
>>>> >> nothing seems to have been merged in the main branch.
>>>> >> To be sure, don't you have the same problem of flags coherence across
>>>> >> modules using only cmake but compiling one by one the dune modules ?
>>>> >> Regards,
>>>> >> Gauthier
>>>> >>
>>>> >> On Wed, Apr 22, 2020 at 11:05 AM Dominic Kempf <
>>>> dominic.r.kempf at gmail.com>
>>>> >> wrote:
>>>> >>
>>>> >>> Hey Gauthier,
>>>> >>>
>>>> >>> sorry for jumping in late.
>>>> >>>
>>>> >>> Christian has it right here, I was looking into Spack as a way of
>>>> setting
>>>> >>> up our CI and maybe moving towards testing variability in user
>>>> landscapes
>>>> >>> as well. The main reason I stopped working on it was that the
>>>> following
>>>> >>> particularity (some people call it a bug) of the Dune CMake build
>>>> system
>>>> >>> clashes with Spack:
>>>> >>> Each Dune module reruns cmake find scripts for dependencies of all
>>>> Dune
>>>> >>> modules it depends on instead of importing the full configuration
>>>> of the
>>>> >>> upstream modules. Due to this behaviour only modules that were
>>>> built in the
>>>> >>> same environment and with the same flags can be safely used
>>>> together (as
>>>> >>> their cmake find scripts found the same things).
>>>> >>> For Spack, that means that all variants that you define need to be
>>>> >>> redundantly defined across all modules, and dependencies between
>>>> those need
>>>> >>> to be specified manually. That is neither elegant nor feasible in
>>>> >>> maintenance.
>>>> >>> The Dune developers have voiced their willingness to abandon this
>>>> >>> behaviour, but a volunteer to do such a tedious, unrewarding work
>>>> is yet to
>>>> >>> be found.
>>>> >>>
>>>> >>> Best,
>>>> >>> Dominic
>>>> >>>
>>>> >>> On Wed, Apr 1, 2020 at 3:29 PM Gauthier Folzan <
>>>> gauthier.folzan at gmail.com>
>>>> >>> wrote:
>>>> >>>
>>>> >>>> Hi Christian,
>>>> >>>> here is the link :
>>>> https://github.com/gauthier12/dune_spack_repo.git
>>>> >>>> It is still work in progress and I am still testing so I don't
>>>> guarantee
>>>> >>>> anything but it seems to work on ubuntu, centos and archlinux for
>>>> >>>> dune-pyhon with dune-common, dune-geometry, dune-grid, dune-istl
>>>> and
>>>> >>>> dune-uggrid
>>>> >>>> Once you have spack installed, installing dune is quite simple,
>>>> just add
>>>> >>>> the repo and install dune
>>>> >>>> spack repo add dune_spack_repo
>>>> >>>> spack install dune-python
>>>> >>>> Cheers
>>>> >>>> Gauthier
>>>> >>>>
>>>> >>>> On Tue, Mar 31, 2020 at 4:46 PM Christian Engwer <
>>>> >>>> christian.engwer at uni-muenster.de> wrote:
>>>> >>>>
>>>> >>>>> Hi Gauthier,
>>>> >>>>>
>>>> >>>>> welcome to Dune,
>>>> >>>>>
>>>> >>>>>> Does someone use spack to compile Dune ? If it can be useful to
>>>> >>>>> someone,
>>>> >>>>>> the spack repo is on github.
>>>> >>>>>
>>>> >>>>> I think nobody is actively using spack for Dune currently,
>>>> Dominik had
>>>> >>>>> been workin on spack packages... I think his intention was to
>>>> ease the
>>>> >>>>> CI setup.
>>>> >>>>>
>>>> >>>>> On the other hand, if spack packages are available, it might
>>>> become
>>>> >>>>> more appealing for other people ;-) Can you post a link to your
>>>> repo?
>>>> >>>>>
>>>> >>>>> Thanks
>>>> >>>>> Christian
>>>> >>>>>
>>>> >>>> _______________________________________________
>>>> >>>> Dune mailing list
>>>> >>>> Dune at lists.dune-project.org
>>>> >>>> https://lists.dune-project.org/mailman/listinfo/dune
>>>> >>>
>>>> >>>
>>>> >
>>>> >> _______________________________________________
>>>> >> Dune mailing list
>>>> >> Dune at lists.dune-project.org
>>>> >> https://lists.dune-project.org/mailman/listinfo/dune
>>>> >
>>>> >
>>>>
>>>> _______________________________________________
>>> Dune mailing list
>>> Dune at lists.dune-project.org
>>> https://lists.dune-project.org/mailman/listinfo/dune
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dune-project.org/pipermail/dune/attachments/20200504/65260584/attachment.htm>