<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body><p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;"><span style="background-color:#ffffff;"><span style="font-family:Ubuntu;"><span style="color:#232629;">Hi Aswin,</span></span></span></p>
<br /><p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;"><span style="background-color:#ffffff;"><span style="color:#232629;">first of all, I think you might be mislead by how periodic boundaries are handled in DUNE. Periodic boundaries (at least using YASPGrid) require a parallel run (i.e. more than one MPI rank), since they essentially use the same overlapping communication framework that otherwise handles "regular" overlaps between subdomains. Think of a 2D grid, gluing together the periodic ends; in the resulting cylinder shape, subdomains at periodic boundaries are just regular neighbors and can be handled accordingly.</span></span></p>
<br /><p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;"><span style="background-color:#ffffff;"><span style="color:#232629;">Long story short, I think the sequential (-np 1) run does not give you a correct solution (write to VTK and check the output to confirm) and is therefore not a good reference. The other runs look not as bad regarding scalability.</span></span></p>
<br /><p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;"><span style="background-color:#ffffff;"><span style="color:#232629;">If you still need better scalability, you might look into more advanced methods. The appropriate choice then depends a lot on the particular problem you want to solve, so more detail from you would be helpful.</span></span></p>
<br /><p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;"><span style="color:#232629;"><span style="background-color:#ffffff;">One example might be GenEO, which we could scale up to around 16000 cores (see</span> </span><a href="https://doi.org/10.1007/978-3-030-43229-4_11">https://doi.org/10.1007/978-3-030-43229-4_11</a><span style="color:#232629;"> ). Might be a bit overkill though depending on what you want to do.</span></p>
<br /><p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;"><span style="background-color:#ffffff;"><span style="color:#232629;">Best,</span></span></p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;"><span style="background-color:#ffffff;"><span style="color:#232629;">Linus</span></span></p>
<br /><p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">Am Montag, 3. Januar 2022, 11:09:14 CET schrieb Aswin vs:</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> Hello,</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> </p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> Can somebody suggest me how to get good scalability while using the</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> overlapping solvers in DUNE. I tried the following test example in DUNE</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> PDELab, but not getting good scalability.</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> Thank you.</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> </p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> $ mpirun -np 1 ./test-heat-instationary-periodic</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> .</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> .</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> .</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> STAGE 2 time (to): 1.5000e-02.</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> === matrix setup skipped (matrix already allocated)</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> === matrix assembly (max) 0.9944 s</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> === residual assembly (max) 0.4861 s</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> === solving (reduction: 1e-10) === Dune::BiCGSTABSolver</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> === rate=0.8397, T=12.57, TIT=0.09524, IT=132</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> 12.68 s</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> ::: timesteps 2 (2)</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> ::: nl iterations 565 (565)</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> ::: lin iterations 565 (565)</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> ::: assemble time 8.0477e+00 (8.0477e+00)</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> ::: lin solve time 5.3414e+01 (5.3414e+01)</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> ---------------------------------------------------------------------------------------</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> $ mpirun -np 2 ./testheat-instationary-periodic</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> .</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> .</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> .</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> STAGE 2 time (to): 1.5000e-02.</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> === matrix setup skipped (matrix already allocated)</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> === matrix assembly (max) 0.5053 s</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> === residual assembly (max) 0.2465 s</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> === solving (reduction: 1e-10) === Dune::BiCGSTABSolver</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> === rate=0.9268, T=26.95, TIT=0.08895, IT=303</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> 27.05 s</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> ::: timesteps 2 (2)</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> ::: nl iterations 1254 (1254)</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> ::: lin iterations 1254 (1254)</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> ::: assemble time 4.0910e+00 (4.0910e+00)</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> ::: lin solve time 1.1201e+02 (1.1201e+02)</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> ---------------------------------------------------------------------------------------</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> $ mpirun -np 4 ./testheat-instationary-periodic</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> .</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> .</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> .</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> STAGE 2 time (to): 1.5000e-02.</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> === matrix setup skipped (matrix already allocated)</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> === matrix assembly (max) 0.271 s</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> === residual assembly (max) 0.1318 s</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> === solving (reduction: 1e-10) === Dune::BiCGSTABSolver</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> === rate=0.9232, T=26.02, TIT=0.0894, IT=291</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> 26.11 s</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> ::: timesteps 2 (2)</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> ::: nl iterations 1249 (1249)</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> ::: lin iterations 1249 (1249)</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> ::: assemble time 2.1746e+00 (2.1746e+00)</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> ::: lin solve time 1.1165e+02 (1.1165e+02)</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> ---------------------------------------------------------------------------------------</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> $ mpirun -np 8 ./testheat-instationary-periodic</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> .</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> .</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> .</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> STAGE 2 time (to): 1.5000e-02.</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> === matrix setup skipped (matrix already allocated)</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> === matrix assembly (max) 0.1772 s</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> === residual assembly (max) 0.08259 s</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> === solving (reduction: 1e-10) === Dune::BiCGSTABSolver</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> === rate=0.9288, T=30.81, TIT=0.09751, IT=316</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> 30.89 s</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> ::: timesteps 2 (2)</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> ::: nl iterations 1329 (1329)</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> ::: lin iterations 1329 (1329)</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> ::: assemble time 1.3485e+00 (1.3485e+00)</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> ::: lin solve time 1.2796e+02 (1.2796e+02)</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> </p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> </p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> </p>
<br /><br /></body>
</html>