[Dune] Parallel AMG on up to 65536 cores on Hector

[Markus Blatt]

Hey,

On Tue, Feb 28, 2012 at 11:20:53PM +0000, Eike Mueller wrote:
> 
> Without ParMETIS, each processor will coarsen its local region and
> data is only pulled together on one processor to solve the problem
> on the coarsest level. I can see that this causes all-to-all
> communication, so might be a problem for parallel scalability (when
> I say my code stopped scaling, I have not looked in detail at what
> causes the problem, of course, so this is just a wild guess)?
> I presume that in my case, even if each process only has one dof
> left (but maybe more, if the lower limit is 2000), then the
> agglomerated problem, which is solved on one processor, still has
> 32768 dofs, whereas on smaller processor counts it was much smaller?

Exactly. And you send a lot of embarrassingly small messages to and from
one process, which does not scale.

> 
> If ParMETIS is installed, as soon as the lower limit of dof on one
> process is reached, data will be pulled together on a smaller number
> of processes, where each now has more than 2000 dofs, then
> coarsening continues until the dofs per process fall below 2000
> again and the process is repeated until we end up with one process
> with less than 2000 dofs. So the size of problem that is solved on
> the coarsest level does not grow with the process count. ParMETIS is
> used (or rather the METIS subroutines inside it - which makes me
> believe I do not need to install Metis in addition to ParMETIS), to
> work out the best way of pulling data together, i.e. partition the
> problem between a decreasing number of processors on the coarser
> levels.
>
Again yes. ParMETIS suffices.

Scaling will be better but not on par with the acual solve phase.
 
> If I switch off data agglomeration, what happens at the coarsest level?
> 

You solve using BiCGSTAB preconditioned with the parallel smoother of
AMG. If you have jumps in your coefficients, this will be a problem.


Markus

-- 
Do you need more support with DUNE or HPC in general? 

Dr. Markus Blatt - HPC-Simulation-Software & Services http://www.dr-blatt.de
Hans-Bunte-Str. 8-10, 69123 Heidelberg, Germany
Tel.: +49 (0) 160 97590858  Fax: +49 (0)322 1108991658