[Dune] FW: H5Fed into Dune as a grid manager, comments, ideas ???
Oliver Sander
sander at mi.fu-berlin.de
Mon Nov 16 10:35:25 CET 2009
Hi Benedikt!
>
> When we then proceeded with its development we realized that optimum use of H5Fed in our
> codes requires tight integration into Dune; for example we need checkpoint-restart capability, especially for
> larger machines such as the Cray where run time is often limited to 48 hours, larger jobs then must be
> resubmitted to go through.
>
At the last Dune meeting in Berlin it was decided that checkpointing
should be part of
the Dune interface. A proposal for the precise interface methods should
have been worked out,
unfortunately, to my knowledge, this never happened. Nevertheless, at
least ALUGrid does
provide checkpointing. Implementing checkpointing for UGGrid would need
a few days
of serious work, but it is not impossible.
> Also, we do need the capability of reading large meshes and associated dats (volume id, boundary id)
> NOW, especially for larger macro grids (large meaning n * 10^5 tets). While we intensely use
> the DGF format, this does not scale as other users have come to realize. Eventually, this prevents
> us from running the problems that we must run: waiting for hours sometimes until the mesh
> and its parameters are read, is just not an option.
>
I am just working on a set of patches that allows to read grids of this
size in UGGrid using
the grid factory (i.e., gmsh, AmiraMesh, etc). My test grid has 600.000
elements and has
been provided by Bodo Erdmann and Rainer Roitzsch from ZIB. Reading it
takes about
1 minute on my old laptop. The renumbering problem
that makes DGF so slow is not an issue, because the UGGrid wrapper
renumbers all UG
entities such that their indices match the order of creation.
> Further, we think that reading large meshes should really be done in parallel, i.e. not reading
> the whole mesh on the master node (memory bottleneck, especially on future supercomputers
> which may have hundreds of thousands of cores but quite little memory per core; e.g. our
> Cray XT5 in Manno has 22 kCores but only 1.3 GByte per core...)
>
That is a good point. I think the Dune grid interface should cover this
case and we need
to discuss how.
> Eventually, H5Fed will not stop short of only having tets as elements. Whereas we now focus
> on tetrahedra, we foresee its extension to hexahedra and maybe also hybrid meshes,
> combining tets and hexahedra.
>
This is all available in UGGrid.
> To make a long story short, we need a grid manager and a data storage format that can handle
> practical meshes (while the macro grid may have some 10^5 tets, the leaf grid may have hundreds
> of millions of tets eventually, this number may sound huge but it is foreseeable).
>
Again, this is all available in UGGrid. I know that UGGrid still has
some deficiencies
concering parallel computing, but nothing prevents us from fixing this
if you make
specific requests.
> Let me make one point very clear: we would like to contribute the Dune community
> to this actively in order to make it a success. In no way this is designed to introduce
> competition, but really it is intended to solve our comp. needs, together with Dune
> since we think Dune to be one of the greatest project, if not the greatest , in this field
>
I don't perceive you guys as competition. What you do is still Dune,
after all.
But I worry that you spend a lot of effort on duplicating
functionality. I think
you could have fixed parallelism and checkpointing in UGGrid with a fraction
of the time that you spend (and will continue to spend) on your own
flexible, parallel
multielement grid manager.
Having said that it would still be nice if we could have a look at your
code.
Is it available as a Dune module?
Yours,
Oliver
>
> Have a nice weekend! Benedikt
>
>
>
>
>
>
>
>
>
> ------------------------------------------------------------------------------------------------------------------
> Benedikt Oswald, Dr. sc. techn., dipl. El. Ing. ETH, www.psi.ch, Computational Accelerator Scientist
> Paul Scherrer Institute (PSI), CH-5232 Villigen, Suisse, benedikt.oswald at psi.ch, +41(0)56 310 32 12
> "Passion is required for any great work, and for the Revolution passion and audacity are required in big doses.",
> Ernesto 'Che' Guevara, Letter to his parents.
> http://maxwell.psi.ch/amaswiki/index.php/User:BenediktOswald
> ------------------------------------------------------------------------------------------------------------------
>
>
>
>
> -----Original Message-----
> From: Oliver Sander [mailto:sander at mi.fu-berlin.de]
> Sent: Fri 13.11.2009 16:05
> To: Oswald Benedikt
> Cc: dune at dune-project.org
> Subject: Re: [Dune] FW: H5Fed into Dune as a grid manager, comments, ideas ???
>
> Hi Benedikt!
> So far I thought that what you guys did was developing a general
> and powerful file format for fe grids and data. Now all of a sudden
> it is a grid. My impression reading your email below is that you are
> reinventing the wheel. What can your grid do that ALUGrid or UGGrid
> cannot do already? Why spend the time to write yet another grid
> manager (yagm)? If the problem is that there is currently no way to
> get a hierarchical grid from your file into a ALU/Alberta/UGGrid then
> we should discuss the possibility of extending the GridFactory interface.
>
> Yours,
> Oliver
>
> Oswald Benedikt schrieb:
>
>>
>> Hi Christian, yes, local refinement with hanging nodes is is already
>> available in the current H5Fed (which has no connection to Dune yet)
>> including tags and adjacencies; i.e. we can attach and read / write
>> data to any top. entity on any level; data may be scalar, vector and
>> matrices of type double, complex, integer.
>>
>> Maybe we should use derive from the grid factory class
>> and add additional member functions ?
>>
>> Benedikt
>>
>>
>>
>>
>> -----Original Message-----
>> From: Christian Engwer [mailto:christi at uni-hd.de]
>> Sent: Fri 13.11.2009 14:03
>> To: Oswald Benedikt
>> Cc: Markus Blatt; dune at dune-project.org
>> Subject: Re: [Dune] H5Fed into Dune as a grid manager, comments, ideas ???
>>
>> On Fri, Nov 13, 2009 at 01:56:26PM +0100, Oswald Benedikt wrote:
>>
>>
>>> indeed yes, the idea is to fully connect/integrate H5Fed into Dune so that we can also use
>>> H5Fed as a grid manager. So far H5Fed works in serial mode, without connection to Dune,
>>> and has been tested up to about 2 million tets, but tests are ongoing and of course this
>>> number of tets makes more sense in parallel aynway.
>>>
>>> One of the questions is then if we should adhere to the grid factory approach as do the
>>> other grid managers ?
>>>
>>>
>> Well, the grid factory is only for coarse grid creation. I suggest to
>> support this interface, but you will need more. Do you intend to
>> implement a full grid manager with local refinement etc? In this case
>> you will also want to add read/write methods for your "internal" data
>> format.
>>
>> Christian
>>
>>
>>
>>> Greetings, Benedikt
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: dune-bounces+benedikt.oswald=psi.ch at dune-project.org on behalf of Markus Blatt
>>> Sent: Fri 13.11.2009 13:51
>>> To: dune at dune-project.org
>>> Subject: Re: [Dune] H5Fed into Dune as a grid manager, comments, ideas ???
>>>
>>> Hi,
>>>
>>> On Thu, Nov 12, 2009 at 05:17:06PM +0100, Oswald Benedikt wrote:
>>>
>>>
>>>> NOW, we want to integrate H5Fed into Dune as a (parallel) grid manager,
>>>> in a way like other grid managers are connected to Dune:
>>>>
>>>>
>>>>
>>> Up to now I thought that you are just concerned about reading and
>>> writing data. Do I interpret your comment correctly in sense that you
>>> will provide a full Dune grid based on H5fed? Or am I just not
>>> comprehending your sentence?
>>>
>>> Markus
>>>
>>> _______________________________________________
>>> Dune mailing list
>>> Dune at dune-project.org
>>> http://lists.dune-project.org/mailman/listinfo/dune
>>>
>>>
>>> _______________________________________________
>>> Dune mailing list
>>> Dune at dune-project.org
>>> http://lists.dune-project.org/mailman/listinfo/dune
>>>
>>>
>>>
>>
>> _______________________________________________
>> Dune mailing list
>> Dune at dune-project.org
>> http://lists.dune-project.org/mailman/listinfo/dune
>>
>>
>
>
>
--
************************************************************************
* Oliver Sander ** email: sander at mi.fu-berlin.de *
* Freie Universität Berlin ** phone: + 49 (30) 838 75348 *
* Institut für Mathematik ** URL : page.mi.fu-berlin.de/~sander *
* Arnimallee 6 ** -------------------------------------*
* 14195 Berlin, Germany ** Member of MATHEON (www.matheon.de) *
************************************************************************
More information about the Dune
mailing list