[Dune-devel] SOCIS 2014: overlapping shwartz operator

[Markus Blatt]

Hi Marco,

sorry for the late reply. I was at a conference the first half of the
week and postponed my reply unti afterwards. Unfortunately, due to the
air conditioning I caught a severe cold and instead of being able to
concentrate and work, I spend rest of the week recovering in bed with
hot tea while outside the sun was shining and it was above 30 degrees
Celsius. 


On Mon, Jul 14, 2014 at 05:42:41PM +0000, Agnese, Marco wrote:
> Hi DUNEers,
> I have some trouble implementing what Markus asked me. What I have done so far can be found at the following link
> 
> https://github.com/magnese/dune-istl/blob/threads/dune/istl/test/threadsmpitest.cc
> 
> This example uses both threads and mpi to assemble the stiffness matrix and the rhs term.
> Markus asked me to:
> >1. modify your assembly routine such that each thread sets up local
> >  matrix and a local vector with some overlap.
> >
> >   What I mean by this is the following. Partition the unknows between
> >   the threads (the owner region). Then augment the partition by
> >   adding all unknowns to the set that are connected by a non-zero
> >  entry of the global matrix to an unknown already in the initial
> >   partition. The added unknowns form the overlap/copy part of the thread.
> > 2. Create the corresponding parallel index sets.
> > 3. Perform a parallel matrix vector product using
> >  OverlappingSchwarzOperator.
> 
> First of all I would like to know if I understood correctly the overlaps. For example for the RHS, each process has exactly one element overlapped which is the node shared by the two processes which shares the node (analogously for the stiffness matrix there is also one overlapped element). Is it correct?
> 

Yes that is correct.

> Secondly, I have created the the ParallelIndexSet imposing the overlapped element as ghosts in both processes. The communication is performed coupling a forward with an add policy and a backward with a copy policy. The values on the b vector are set correctly after this but I don't know if should be done in another way which avoids a double communication. 
> 
> Moreover, I haven't created the IndexSet for the stiffness matrix yet since I am not sure how to do it. I suppose that I have to map the couple of index i,j to an unique index k (something like this k=i*size_matrix+j) and use this to perform an update of the entries as I have done for b. Is it correct or there is a better way?
>

There is no need to do something for the matrix here.
It should suffice to use one index set that describes the
decomposition of the degrees of freedom between the processes. This
index set also describes the decomposition of the matrix ( which has
an entry for every nonzero entry A_ij of the global matrix, wheren i
and j are global indices that are part of the local index set.) The
way the local matrix is constructed each row owned by the process
knows all nonzero entries from the row of the global matrix.
This should also be describe in more detail in "C++ Components
Describing Parallel Domain Decomposition and Communication".

> Finally for what concern the OverlappingSchwartzoperator I am not
> sure how to use it. I suppose that in
> Dune::OverlappingSchwarzOperator< M, X, Y, C > the matrix type M,X,Y
> are already a parallel matrix and parallel vectors ( not the
> sequential structures which I have used). Am I right?Can I just
> create a parallel matrix/vector which is composed by the local
> matrices/vectors which I have assembled using the ParallelIndexSet
> created? 

There is no such thing as a parallel matrix or vector in dune-istl. We
always use sequential data structures together with the parallel index
sets. For OverlappingSchwarzOperator the parallel index sets are
passed implicitly by the type C in the template parameter list. Up to
now this was always OwnerOverlapCopyCommunication which is purely
based on MPI currently. This has to change of course and is one of
your major tasks. In addition to the parallel index set it uses
RemoteIndices, and BufferedCommunicator underneath. Both currently use
only MPI. 

You should try redesigning RemoteIndices such that we
can tell it to use either MPI or threads (later even both?) to compute
the remote information. An idea would be to have a class that is
agnostic of the parallel programming system used and simply stores the
remote index information. Then we could have a class or subclass with a template
tag for the parallel programming system (MPI, thread, hybrid,
whatever) with specializations which are responsible for computing the
remote index information. Later on the tag can be used by
OwnerOverlapCopyCommunication to choose and construct the right
communicator.

This is still based on the fact that we do not need to exchange
RemoteIndices with something else. This is surely a way to go, but
later it might turn out that we want to do something more smart
(e.g. no ghosts between threads). Therefore it might be good idea to
aim for a flexible design that later alows us to use another class for
RemoteIndices.

Please dive into RemoteIndices and check whether my reasoning is
sound. Then sit down and create a cool design for the problem.

In order to speed up project coordination: What about a regular google
hangout for questions? Maybe Tuesdays at 14:00 UCT? 


Markus
-- 
Do you need more support with DUNE or HPC in general? 

Dr. Markus Blatt - HPC-Simulation-Software & Services http://www.dr-blatt.de
Hans-Bunte-Str. 8-10, 69123 Heidelberg, Germany
Tel.: +49 (0) 160 97590858
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <https://lists.dune-project.org/pipermail/dune-devel/attachments/20140721/307fe1c8/attachment.sig>