[Dune] Fwd: Re: How to implement point to point communication

Aleksejs Fomins aleksejs.fomins at lspr.ch
Thu Oct 16 16:38:50 CEST 2014 

Dear Andreas,

I think you guessed right, but let me write it again just to be sure. We
want to do our own partitioning for 2 reasons:
1) The elements will initially be stored on processes in an irregular
way. I assume that creating a grid from that chaos would cause the
hostgrid to do a lot of unnecessary work, as the elements would be
scattered over the domain, opposed to making a nice connected subdomain.

2) Curvilinear elements can, in principle, have different interpolation
orders in the same mesh. Thus, the amount of computation per element may
vary. Main causes are
* Computation of global-to-local and local-to-global maps, as well as
integration over elements will be progressively more expensive with
increasing interpolation order
* When doing FEM calculations involving basis functions, it is
reasonable to assume that a highly curved element would require more
basis functions to describe the fine internal processes than the
straight-sided one. Roughly speaking, the number of interpolation
functions increases quadratically with the polynomial order, so the
number of matrix elements could increase quartically.


To summarize, we need a partitioning algorithm that would be able to
manage weights, and run before AluGridFactory.createGrid(). Whether or
not it uses parmetis is not important, I speak of parmetis is because it
is the only thing I have experience with at the moment.

---------------------------------------------------------------------
The strategy for the CurvilinearGridFactory would then be as follows:

1) Add all vertices on process (global index mandatory)

2) Read all elements on process (type, interpOrder, vertexIndexVector
mandatory)

3) Read all boundarySegments on process (type, interpOrder,
vertexIndexVector mandatory, relatedElement optional). It is mandatory
to insert all boundarySegments. At the moment it is also mandatory that
each process only inserts boundarySegments which are connected to its
elements, because otherwise it is not trivial how to communicate them
afterwards. Therefore, when inserting the boundary segment we also ask
to insert the index of the element the boundary is a face to. It is
reasonable to assume that the reader would have to calculate this
information when figuring out which boundaries to read from the file.

4) call createGrid. Have a vector of coordinates, elements and
boundartysegments

4.1) Call partition method, receive which element should go to which
process.

4.2) Communicate all elements and boundarySegments such that each
element only has elements that belong to it. Boundary segments are
communicated together with the elements they are a face to.

4.3) Each process deletes all vertices it does not need any more

4.4) For each element and boundarySegment - compute corners, insert
corners and linear element to hostgrid

4.5) Compute and insert processBoundaries to HostGrid

4.6) Create HostGrid. Delete HostGridFactory

4.7) Create MetaGrid(HostGrid). This would mainly correspond to locating
elements in ALUGrid and making maps to link them with the curvilinear
elements in the metagrid.

----------------------------------------------------------------

I see your point that making a general MetaGridFactory is a very nice
idea. I guess that I could create a generic factory and then create a
CurvilinearGridFactory to inherit from it, as it would have some
separate methods.

Regards,
Aleksejs

On 10/16/2014 10:48 AM, Andreas Dedner wrote:
> Hi.
> 
> To make one thing clear from the start, I think this is a great project
> and very useful for dune.
> That was for me always clear. But what I was missing was a clear idea
> what the plan was, i.e.,
> a list like the one you just wrote. Of course there are many dune
> projects going on where I have at
> best an idea of what the aim is and no more - and that is fine. But to
> provide any useful input,
> a clearer idea helps and just discussing it with Peter does not provide
> me or others with a clear picture.
> 
> Even after reading your description of the project I am still a bit
> unsure about the partitioning.
> To summarize my understanding:
> In one case, one has a partitioned gmsh file. Then repartitioning is
> probably not required?
> In the other case, the gmsh file is on one process only and the gmsh
> reader adds everything into
> the (meta)gridfactory. Is now the idea that the gridfactory first stores
> the inserted element/vertices
> itself (so does not call the insert methods on the hostgrid). Then in
> the createGrid method
> the inserted elements are partitioned and then each element calls the
> insert method on the host
> gridfactory for its own elements? That sounds like a very useful
> "MetaGridFactory" in its own right, i.e.,
> independent of a CurvilinearGeometryGrid. But perhaps/I misunderstood
> the idea. /
> If that is the idea, I would suggest to consider a callback approach
> similar to the one we are using
> for the repartitioning in ALUGrid. So the method createGrid gets a
> callback object which it can ask
> for a partition number to which to send the process. That would mean
> that the code would not be
> restricted to ParMetis.
> 
> Best
> Andreas
> 
> On 16/10/14 08:47, Benedikt Oswald wrote:
>> Dear Dune,
>>
>> for ca. 3 months now we have been working to implement a higher order
>> curvilinear grid manager,
>> using the concept of the metagrid on top of a host grid.
>>
>> In particular, Aleksejs Fomins is in charge of this project within
>> LSPR AG.
>>
>> I should also mention that the sources of this project are publicly
>> available on github
>> (clone of dune-geometry, dune-curvilineargrid)
>>
>>
>> The plan has been as follows:
>> =======================
>>
>> 1) implement the curvilinear geometry in the class LagrangeGeometry
>> which handles
>>      the curvilinear, tetrahedral geometry using Lagrange polynomials
>>
>>      we comment that at present we implement up to order 5 but in
>> principle
>>      we can implement higher orders as well, if required.
>>
>>      status: implemented & tested
>>
>>
>> 2) use the concept of the meta grid, embodied in the GeometryGrid
>>
>>
>> 3) after discussions with Peter Bastian, we decided to create a new
>> Dune module,
>>      i.e.dune-curvilineargrid that uses sources from GeometryGrid
>> which have been
>>      renamed to reflect the new module name; in particular,
>> GeometryGrid in its
>>      present form simply does not do what is needed.
>>
>>      We have decided to use the new dune-alugrid module as the host
>> grid and
>>      we appreciate the ALUGrid effort very much.
>>
>>      status: in progress
>>
>>
>> 4) we wish to avoid the bottleneck of reading the whole mesh on the
>> master node only
>>      and have therefore implement a parallel gmsh reader which reads
>> the full curvilinear
>>      gmsh .msh format, including tags and everything
>>
>>     status: operational
>>
>>
>> 5) as a consequence, we need to repartition the mesh before we create
>> the grid,both
>>      that is the reason why Aleksejs asked for advice on how to
>> communicate elements
>>      between the MPI processes;  in fact, we have found a solution and
>> use the well
>>      known CLINK protocol
>>
>>
>> 6) as a result, we estimate that we will have a first version of the
>> parallel dune-curviineargrid
>>      module operational in December;
>>
>>      the Dune community is very welcome to test it and comment on it.
>>
>>
>> We really appreciate the support given to us from the Dune mailing
>> list and we appreciate
>> the Dune effort enormously. On the other hand, we openly admit that,
>> sometimes in the past,
>> we perceived certain comments as a tad 'grossfürstlich'.
>>
>>
>> with the best of intentions and wishes for a wonderful day,
>>
>> Benedikt
>>
>>
>>
>>
>>
>>
>> --------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> Dr. sc. techn. Benedikt Oswald - first engineer - LSPR AG - phone -
>> +41 43 366 90 74
>> Technoparkstrasse 1, CH-8005 Zürich, benedikt.oswald at lspr.ch - labor
>> vincit omnia improbus
>> --------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>>
>>
>>
>>
>> _______________________________________________
>> Dune mailing list
>> Dune at dune-project.org
>> http://lists.dune-project.org/mailman/listinfo/dune
> 
> 
> 
> 
> _______________________________________________
> Dune mailing list
> Dune at dune-project.org
> http://lists.dune-project.org/mailman/listinfo/dune
> 





-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 551 bytes
Desc: OpenPGP digital signature
URL: <https://lists.dune-project.org/pipermail/dune/attachments/20141016/5907d3af/attachment.sig>

More information about the Dune mailing list