<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 16/10/14 15:38, Aleksejs Fomins
wrote:<br>
</div>
<blockquote cite="mid:543FD87A.4030400@lspr.ch" type="cite">
<pre wrap="">Dear Andreas,
I think you guessed right, but let me write it again just to be sure. We
want to do our own partitioning for 2 reasons:
1) The elements will initially be stored on processes in an irregular
way. I assume that creating a grid from that chaos would cause the
hostgrid to do a lot of unnecessary work, as the elements would be
scattered over the domain, opposed to making a nice connected subdomain.</pre>
</blockquote>
Yes - that would cause a lot of overhead for alugrid<br>
<blockquote cite="mid:543FD87A.4030400@lspr.ch" type="cite">
<pre wrap="">
2) Curvilinear elements can, in principle, have different interpolation
orders in the same mesh. Thus, the amount of computation per element may
vary. Main causes are
* Computation of global-to-local and local-to-global maps, as well as
integration over elements will be progressively more expensive with
increasing interpolation order
* When doing FEM calculations involving basis functions, it is
reasonable to assume that a highly curved element would require more
basis functions to describe the fine internal processes than the
straight-sided one. Roughly speaking, the number of interpolation
functions increases quadratically with the polynomial order, so the
number of matrix elements could increase quartically.</pre>
</blockquote>
This could be handled by using e.g. the option to provide weights
during the call<br>
to ALUGrid loadbalancing.<br>
<blockquote cite="mid:543FD87A.4030400@lspr.ch" type="cite">
<pre wrap="">
To summarize, we need a partitioning algorithm that would be able to
manage weights, and run before AluGridFactory.createGrid(). Whether or
not it uses parmetis is not important, I speak of parmetis is because it
is the only thing I have experience with at the moment.</pre>
</blockquote>
Agree. Just one option to consider: the partitioning could also be a
preprocessing step/<br>
So instead of doing it within a dune structure one could simply have
a "partition.cc" which<br>
does the partitioning and then writes partitioned gmsh files for
your parallel reader which<br>
at least avoid your first point and are also correctly balanced
w.r.t. your second point.<br>
In principal your approach is a preprocessing step anyway so one
doesn't necessarily want to do this <br>
every time a simulation is started anyway. On the other hand if the
whole thing is contained within a<br>
extra program partition.cc then runtime does perhaps not play such
an important role and you<br>
could use the partitioning algorithm and parallel gridfactory
methods available in ALUGrid and except the<br>
additional overhead caused for example by your first point. If you
use the BackupRestore<br>
facility then you do not even need to write partitioned gmsh files.
If I understand you correctly<br>
that would already work with what you have - if that approach then
turns out not to be feasible you<br>
can still add a partitioning into the GridFactory. But perhaps that
approach does not harmonize well with<br>
your workflow.<br>
Andreas<br>
<blockquote cite="mid:543FD87A.4030400@lspr.ch" type="cite">
<pre wrap="">
---------------------------------------------------------------------
The strategy for the CurvilinearGridFactory would then be as follows:
1) Add all vertices on process (global index mandatory)
2) Read all elements on process (type, interpOrder, vertexIndexVector
mandatory)
3) Read all boundarySegments on process (type, interpOrder,
vertexIndexVector mandatory, relatedElement optional). It is mandatory
to insert all boundarySegments. At the moment it is also mandatory that
each process only inserts boundarySegments which are connected to its
elements, because otherwise it is not trivial how to communicate them
afterwards. Therefore, when inserting the boundary segment we also ask
to insert the index of the element the boundary is a face to. It is
reasonable to assume that the reader would have to calculate this
information when figuring out which boundaries to read from the file.
4) call createGrid. Have a vector of coordinates, elements and
boundartysegments
4.1) Call partition method, receive which element should go to which
process.
4.2) Communicate all elements and boundarySegments such that each
element only has elements that belong to it. Boundary segments are
communicated together with the elements they are a face to.
4.3) Each process deletes all vertices it does not need any more
4.4) For each element and boundarySegment - compute corners, insert
corners and linear element to hostgrid
4.5) Compute and insert processBoundaries to HostGrid
4.6) Create HostGrid. Delete HostGridFactory
4.7) Create MetaGrid(HostGrid). This would mainly correspond to locating
elements in ALUGrid and making maps to link them with the curvilinear
elements in the metagrid.
----------------------------------------------------------------
I see your point that making a general MetaGridFactory is a very nice
idea. I guess that I could create a generic factory and then create a
CurvilinearGridFactory to inherit from it, as it would have some
separate methods.
Regards,
Aleksejs
On 10/16/2014 10:48 AM, Andreas Dedner wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi.
To make one thing clear from the start, I think this is a great project
and very useful for dune.
That was for me always clear. But what I was missing was a clear idea
what the plan was, i.e.,
a list like the one you just wrote. Of course there are many dune
projects going on where I have at
best an idea of what the aim is and no more - and that is fine. But to
provide any useful input,
a clearer idea helps and just discussing it with Peter does not provide
me or others with a clear picture.
Even after reading your description of the project I am still a bit
unsure about the partitioning.
To summarize my understanding:
In one case, one has a partitioned gmsh file. Then repartitioning is
probably not required?
In the other case, the gmsh file is on one process only and the gmsh
reader adds everything into
the (meta)gridfactory. Is now the idea that the gridfactory first stores
the inserted element/vertices
itself (so does not call the insert methods on the hostgrid). Then in
the createGrid method
the inserted elements are partitioned and then each element calls the
insert method on the host
gridfactory for its own elements? That sounds like a very useful
"MetaGridFactory" in its own right, i.e.,
independent of a CurvilinearGeometryGrid. But perhaps/I misunderstood
the idea. /
If that is the idea, I would suggest to consider a callback approach
similar to the one we are using
for the repartitioning in ALUGrid. So the method createGrid gets a
callback object which it can ask
for a partition number to which to send the process. That would mean
that the code would not be
restricted to ParMetis.
Best
Andreas
On 16/10/14 08:47, Benedikt Oswald wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Dear Dune,
for ca. 3 months now we have been working to implement a higher order
curvilinear grid manager,
using the concept of the metagrid on top of a host grid.
In particular, Aleksejs Fomins is in charge of this project within
LSPR AG.
I should also mention that the sources of this project are publicly
available on github
(clone of dune-geometry, dune-curvilineargrid)
The plan has been as follows:
=======================
1) implement the curvilinear geometry in the class LagrangeGeometry
which handles
the curvilinear, tetrahedral geometry using Lagrange polynomials
we comment that at present we implement up to order 5 but in
principle
we can implement higher orders as well, if required.
status: implemented & tested
2) use the concept of the meta grid, embodied in the GeometryGrid
3) after discussions with Peter Bastian, we decided to create a new
Dune module,
i.e.dune-curvilineargrid that uses sources from GeometryGrid
which have been
renamed to reflect the new module name; in particular,
GeometryGrid in its
present form simply does not do what is needed.
We have decided to use the new dune-alugrid module as the host
grid and
we appreciate the ALUGrid effort very much.
status: in progress
4) we wish to avoid the bottleneck of reading the whole mesh on the
master node only
and have therefore implement a parallel gmsh reader which reads
the full curvilinear
gmsh .msh format, including tags and everything
status: operational
5) as a consequence, we need to repartition the mesh before we create
the grid,both
that is the reason why Aleksejs asked for advice on how to
communicate elements
between the MPI processes; in fact, we have found a solution and
use the well
known CLINK protocol
6) as a result, we estimate that we will have a first version of the
parallel dune-curviineargrid
module operational in December;
the Dune community is very welcome to test it and comment on it.
We really appreciate the support given to us from the Dune mailing
list and we appreciate
the Dune effort enormously. On the other hand, we openly admit that,
sometimes in the past,
we perceived certain comments as a tad 'grossfürstlich'.
with the best of intentions and wishes for a wonderful day,
Benedikt
--------------------------------------------------------------------------------------------------------------------------------------------------------------
Dr. sc. techn. Benedikt Oswald - first engineer - LSPR AG - phone -
+41 43 366 90 74
Technoparkstrasse 1, CH-8005 Zürich, <a class="moz-txt-link-abbreviated" href="mailto:benedikt.oswald@lspr.ch">benedikt.oswald@lspr.ch</a> - labor
vincit omnia improbus
--------------------------------------------------------------------------------------------------------------------------------------------------------------
_______________________________________________
Dune mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Dune@dune-project.org">Dune@dune-project.org</a>
<a class="moz-txt-link-freetext" href="http://lists.dune-project.org/mailman/listinfo/dune">http://lists.dune-project.org/mailman/listinfo/dune</a>
</pre>
</blockquote>
<pre wrap="">
_______________________________________________
Dune mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Dune@dune-project.org">Dune@dune-project.org</a>
<a class="moz-txt-link-freetext" href="http://lists.dune-project.org/mailman/listinfo/dune">http://lists.dune-project.org/mailman/listinfo/dune</a>
</pre>
</blockquote>
<pre wrap="">
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Dune mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Dune@dune-project.org">Dune@dune-project.org</a>
<a class="moz-txt-link-freetext" href="http://lists.dune-project.org/mailman/listinfo/dune">http://lists.dune-project.org/mailman/listinfo/dune</a>
</pre>
</blockquote>
<br>
</body>
</html>