[dune-pdelab] Scalability issue in overlapping solvers

[Linus Seelinger]

Hello Aswin,

GenEO is a two-level Schwarz method, so for a small number of subdomains the first level of the preconditioner is still fairly good. You really have to go to larger numbers of subdomains; then the additive Schwarz method involved becomes insufficient (regarding parallel scalability) and you see the effect of the coarse space. I think the first point I saw some effect was around 5 subdomains per direction, but more will certainly be clearer. You can use oversubscription (i.e. more processes than you have processor cores) for testing on your machine.

You see dim(V_H) in the output "Global basis size B=8". If you need to access that in your code, look for the SubdomainProjectedCoarseSpace class, it has a suitable method for that.

Not sure where the YaspGrid error comes from even though you refine the grid. You can check in the vtk output though (it writes non-overlapping per-subdomain files you can access directly). Make sure that no grid elements are shared between neighbors of neighbors (including overlaps!) and adjust grid size accordingly.

Currently we don't have proper support for hyperelastic models in dune-composites, but that is something we want to look at in the future. DUNE certainly has the necessary underlying methods. If you are interested in that topic, collaborations are certainly welcome!

But, taking a step back, what is the specific PDE problem you are ultimately interested in? What does the hyperelastic material look like, how many grid cells will you need, how many processor cores do you have available etc.? If you only want to run that on around 16 processes and the PDE is not too ill-conditioned, something like AMG or more basic methods might work as well (probably even faster) and would be easier to apply.

Overall, maybe it's a better approach to first set up your actual PDE problem, solve it with a direct solver like UMFPack (does not scale well, but is extremely robust), and only look for more advanced solvers when you know exactly how the simpler ones perform.

Best,

Linus

Am Dienstag, 11. Januar 2022, 13:09:48 CET schrieb Aswin vs:
> Sir,
> As you suggested, I am currently looking into the GenEO examples given in
> the dune-composite-master. Still I couldn't see any scalability in the test
> example.
> 
> In the test Example01b I have edited the line Partitioning[0] = 2 and
> Partitioning[0] = 4 for 2 and 4 numbers of processors, respectively. If I
> change it to Partitioning[0] = 8 then I am getting the error " YaspGrid
> does not support degrees of freedom shared by more than immediately
> neighboring subdomains." Then I changed the refineBaseGrid = 2, but it had
> the same error.
> 
> Presently, I am using the desktop having Intel Core™ i7-10700 CPU @ 2.90GHz
> × 1 processor, that has 8/16 number of cores/threads.
> 
> The outputs of the simple runs are given below for your reference. In each
> case, I found the rate is also not strictly decreasing. The  number of
> iterations is increasing and the overall time is also  increasing.
> 
> Can you suggest to me how to run these examples efficiently? Also I would
> like to know how to print the dim(V_H) as you mentioned in the paper.
> 
> Thank you for your time and consideration,
> 
> $ mpirun -np 2 ./Example01b
> *NumRegions
> 1
> Number of Regions = 1
> maxPly = 23
> periodic = 000
> === Building Geometry
> Elements in x direction
> Number of elements per processor: 1925
> Number of nodes per processor: 2592
> Grid transformation complete
> Grid view set up
> Piecewise quadratic serendipity elements
> Starting solve with Geneo Preconditioner
> Eigenvalue threshhold: 0.00769231
> Process 1 picked 6 eigenvectors
> Process 0 picked 2 eigenvectors
> Matrix setup
> Global basis size B=8
> Matrix setup finished: M=0.0490446
> Geneo setup time 23.4082
> === Dune::CGSolver
> Min eigv estimate: 0.0676291
> Max eigv estimate: 2.99961
> Condition estimate: 44.3538
> === rate=0.534884, T=2.99187, TIT=0.135994, IT=22
> Solver: CG
> Preconditioner: GenEO
> Subdomain Solver: UMFPack
> =================
> Solver Converged: 1Solution of my Problem = 0.000166984
> 
> 
> $ mpirun -np 4 ./Example01b
> *NumRegions
> 1
> Number of Regions = 1
> maxPly = 23
> periodic = 000
> === Building Geometry
> Elements in x direction
> Number of elements per processor: 1050
> Number of nodes per processor: 1512
> Grid transformation complete
> Grid view set up
> Piecewise quadratic serendipity elements
> Starting solve with Geneo Preconditioner
> Eigenvalue threshhold: 0.00909091
> Process 2 picked 6 eigenvectors
> Process 3 picked 6 eigenvectors
> Process 1 picked 6 eigenvectors
> Process 0 picked 1 eigenvectors
> Matrix setup
> Global basis size B=19
> Matrix setup finished: M=0.119121
> Geneo setup time 24.9791
> === Dune::CGSolver
> Min eigv estimate: 0.0249189
> Max eigv estimate: 2.9999
> Condition estimate: 120.386
> === rate=0.769396, T=5.08444, TIT=0.110531, IT=46
> Solver: CG
> Preconditioner: GenEO
> Subdomain Solver: UMFPack
> =================
> Solver Converged: 1Solution of my Problem = 0.000166984
> 
> 
> 
> 
> Thank you.
> 
> On Mon, Jan 3, 2022 at 11:41 PM Linus Seelinger <
> linus.seelinger at iwr.uni-heidelberg.de> wrote:
> 
> > Hi Aswin,
> >
> > first of all, I think you might be mislead by how periodic boundaries are
> > handled in DUNE. Periodic boundaries (at least using YASPGrid) require a
> > parallel run (i.e. more than one MPI rank), since they essentially use the
> > same overlapping communication framework that otherwise handles "regular"
> > overlaps between subdomains. Think of a 2D grid, gluing together the
> > periodic ends; in the resulting cylinder shape, subdomains at periodic
> > boundaries are just regular neighbors and can be handled accordingly.
> >
> > Long story short, I think the sequential (-np 1) run does not give you a
> > correct solution (write to VTK and check the output to confirm) and is
> > therefore not a good reference. The other runs look not as bad regarding
> > scalability.
> >
> > If you still need better scalability, you might look into more advanced
> > methods. The appropriate choice then depends a lot on the particular
> > problem you want to solve, so more detail from you would be helpful.
> >
> > One example might be GenEO, which we could scale up to around 16000 cores
> > (see https://doi.org/10.1007/978-3-030-43229-4_11 ). Might be a bit
> > overkill though depending on what you want to do.
> >
> > Best,
> >
> > Linus
> >
> > Am Montag, 3. Januar 2022, 11:09:14 CET schrieb Aswin vs:
> >
> > > Hello,
> >
> > >
> >
> > > Can somebody suggest me how to get good scalability while using the
> >
> > > overlapping solvers in DUNE. I tried the following test example in DUNE
> >
> > > PDELab, but not getting good scalability.
> >
> > > Thank you.
> >
> > >
> >
> > > $  mpirun -np 1 ./test-heat-instationary-periodic
> >
> > > .
> >
> > > .
> >
> > > .
> >
> > > STAGE 2 time (to):   1.5000e-02.
> >
> > > === matrix setup skipped (matrix already allocated)
> >
> > > === matrix assembly (max) 0.9944 s
> >
> > > === residual assembly (max) 0.4861 s
> >
> > > === solving (reduction: 1e-10) === Dune::BiCGSTABSolver
> >
> > > === rate=0.8397, T=12.57, TIT=0.09524, IT=132
> >
> > > 12.68 s
> >
> > > ::: timesteps           2 (2)
> >
> > > ::: nl iterations     565 (565)
> >
> > > ::: lin iterations    565 (565)
> >
> > > ::: assemble time    8.0477e+00 (8.0477e+00)
> >
> > > ::: lin solve time   5.3414e+01 (5.3414e+01)
> >
> > >
> > ---------------------------------------------------------------------------------------
> >
> > > $ mpirun -np 2 ./testheat-instationary-periodic
> >
> > > .
> >
> > > .
> >
> > > .
> >
> > > STAGE 2 time (to):   1.5000e-02.
> >
> > > === matrix setup skipped (matrix already allocated)
> >
> > > === matrix assembly (max) 0.5053 s
> >
> > > === residual assembly (max) 0.2465 s
> >
> > > === solving (reduction: 1e-10) === Dune::BiCGSTABSolver
> >
> > > === rate=0.9268, T=26.95, TIT=0.08895, IT=303
> >
> > > 27.05 s
> >
> > > ::: timesteps           2 (2)
> >
> > > ::: nl iterations    1254 (1254)
> >
> > > ::: lin iterations   1254 (1254)
> >
> > > ::: assemble time    4.0910e+00 (4.0910e+00)
> >
> > > ::: lin solve time   1.1201e+02 (1.1201e+02)
> >
> > >
> > ---------------------------------------------------------------------------------------
> >
> > > $ mpirun -np 4 ./testheat-instationary-periodic
> >
> > > .
> >
> > > .
> >
> > > .
> >
> > > STAGE 2 time (to):   1.5000e-02.
> >
> > > === matrix setup skipped (matrix already allocated)
> >
> > > === matrix assembly (max) 0.271 s
> >
> > > === residual assembly (max) 0.1318 s
> >
> > > === solving (reduction: 1e-10) === Dune::BiCGSTABSolver
> >
> > > === rate=0.9232, T=26.02, TIT=0.0894, IT=291
> >
> > > 26.11 s
> >
> > > ::: timesteps           2 (2)
> >
> > > ::: nl iterations    1249 (1249)
> >
> > > ::: lin iterations   1249 (1249)
> >
> > > ::: assemble time    2.1746e+00 (2.1746e+00)
> >
> > > ::: lin solve time   1.1165e+02 (1.1165e+02)
> >
> > >
> > ---------------------------------------------------------------------------------------
> >
> > > $ mpirun -np 8 ./testheat-instationary-periodic
> >
> > > .
> >
> > > .
> >
> > > .
> >
> > > STAGE 2 time (to):   1.5000e-02.
> >
> > > === matrix setup skipped (matrix already allocated)
> >
> > > === matrix assembly (max) 0.1772 s
> >
> > > === residual assembly (max) 0.08259 s
> >
> > > === solving (reduction: 1e-10) === Dune::BiCGSTABSolver
> >
> > > === rate=0.9288, T=30.81, TIT=0.09751, IT=316
> >
> > > 30.89 s
> >
> > > ::: timesteps           2 (2)
> >
> > > ::: nl iterations    1329 (1329)
> >
> > > ::: lin iterations   1329 (1329)
> >
> > > ::: assemble time    1.3485e+00 (1.3485e+00)
> >
> > > ::: lin solve time   1.2796e+02 (1.2796e+02)
> >
> > >
> >
> > >
> >
> > >
> >
> >
> > _______________________________________________
> > dune-pdelab mailing list
> > dune-pdelab at lists.dune-project.org
> > https://lists.dune-project.org/mailman/listinfo/dune-pdelab
> >
>