<div dir="ltr"><div id="gmail-:1d6" class="gmail-Ar gmail-Au gmail-Ao" style="display:block"><div id="gmail-:1d2" class="gmail-Am gmail-Al editable gmail-LW-avf gmail-tS-tW gmail-tS-tY" aria-label="Message Body" role="textbox" aria-multiline="true" style="direction:ltr;min-height:376px" tabindex="1"><div>Sir,</div><div>As you suggested, I am currently looking into the GenEO examples given in the dune-composite-master. Still I couldn't see any scalability in the test example. <br></div><div><br></div><div>In the test Example01b I have edited the line Partitioning[0] = 2 and Partitioning[0] = 4 for 2 and 4 numbers of processors, respectively. If I change it to Partitioning[0] = 8 then I am getting the error " YaspGrid does not support degrees of freedom shared by more than immediately neighboring subdomains." Then I changed the refineBaseGrid = 2, but it had the same error. <br></div><div><br></div><div>Presently, I am using the desktop having Intel Core™ i7-10700 CPU @ 2.90GHz × 1 processor, that has 8/16 number of cores/threads.</div><div><br></div><div> The outputs of the simple runs are given below for your reference. In each case, I found the rate is also not strictly decreasing. The number of iterations is increasing and the overall time is also increasing.</div><div><br></div><div>Can you suggest to me how to run these examples efficiently? Also I would like to know how to print the dim(V_H) as you mentioned in the paper. <br></div><div><br></div><div>Thank you for your time and consideration,<br></div><div><br></div><div>$ mpirun -np 2 ./Example01b </div>*NumRegions<br>1<br>Number of Regions = 1<br>maxPly = 23<br>periodic = 000<br>=== Building Geometry<br>Elements in x direction <br>Number of elements per processor: 1925<br>Number of nodes per processor: 2592<br>Grid transformation complete<br>Grid view set up<br>Piecewise quadratic serendipity elements<br>Starting solve with Geneo Preconditioner<br>Eigenvalue threshhold: 0.00769231<br>Process 1 picked 6 eigenvectors<br>Process 0 picked 2 eigenvectors<br>Matrix setup<br>Global basis size B=8<br>Matrix setup finished: M=0.0490446<br>Geneo setup time 23.4082<br>=== Dune::CGSolver<br>Min eigv estimate: 0.0676291<br>Max eigv estimate: 2.99961<br>Condition estimate: 44.3538<br>=== rate=0.534884, T=2.99187, TIT=0.135994, IT=22<br>Solver: CG<br>Preconditioner: GenEO<br>Subdomain Solver: UMFPack<br>=================<br><div>Solver Converged: 1Solution of my Problem = 0.000166984</div><div><br></div><div><br></div><div>$ mpirun -np 4 ./Example01b <br>*NumRegions<br>1<br>Number of Regions = 1<br>maxPly = 23<br>periodic = 000<br>=== Building Geometry<br>Elements in x direction <br>Number of elements per processor: 1050<br>Number of nodes per processor: 1512<br>Grid transformation complete<br>Grid view set up<br>Piecewise quadratic serendipity elements<br>Starting solve with Geneo Preconditioner<br>Eigenvalue threshhold: 0.00909091<br>Process 2 picked 6 eigenvectors<br>Process 3 picked 6 eigenvectors<br>Process 1 picked 6 eigenvectors<br>Process 0 picked 1 eigenvectors<br>Matrix setup<br>Global basis size B=19<br>Matrix setup finished: M=0.119121<br>Geneo setup time 24.9791<br>=== Dune::CGSolver<br>Min eigv estimate: 0.0249189<br>Max eigv estimate: 2.9999<br>Condition estimate: 120.386<br>=== rate=0.769396, T=5.08444, TIT=0.110531, IT=46<br>Solver: CG<br>Preconditioner: GenEO<br>Subdomain Solver: UMFPack<br>=================<br>Solver Converged: 1Solution of my Problem = 0.000166984</div><div><br></div><div><br></div><div><br></div><div><br></div>Thank you.<br clear="all"></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 3, 2022 at 11:41 PM Linus Seelinger <<a href="mailto:linus.seelinger@iwr.uni-heidelberg.de">linus.seelinger@iwr.uni-heidelberg.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div><p style="margin:0px">Hi Aswin,</p>
<br><p style="margin:0px">first of all, I think you might be mislead by how periodic boundaries are handled in DUNE. Periodic boundaries (at least using YASPGrid) require a parallel run (i.e. more than one MPI rank), since they essentially use the same overlapping communication framework that otherwise handles "regular" overlaps between subdomains. Think of a 2D grid, gluing together the periodic ends; in the resulting cylinder shape, subdomains at periodic boundaries are just regular neighbors and can be handled accordingly.</p>
<br><p style="margin:0px">Long story short, I think the sequential (-np 1) run does not give you a correct solution (write to VTK and check the output to confirm) and is therefore not a good reference. The other runs look not as bad regarding scalability.</p>
<br><p style="margin:0px">If you still need better scalability, you might look into more advanced methods. The appropriate choice then depends a lot on the particular problem you want to solve, so more detail from you would be helpful.</p>
<br><p style="margin:0px">One example might be GenEO, which we could scale up to around 16000 cores (see <a href="https://doi.org/10.1007/978-3-030-43229-4_11" target="_blank">https://doi.org/10.1007/978-3-030-43229-4_11</a> ). Might be a bit overkill though depending on what you want to do.</p>
<br><p style="margin:0px">Best,</p>
<br><p style="margin:0px">Linus</p>
<br><p style="margin:0px">Am Montag, 3. Januar 2022, 11:09:14 CET schrieb Aswin vs:</p>
<p style="margin:0px">> Hello,</p>
<p style="margin:0px">> </p>
<p style="margin:0px">> Can somebody suggest me how to get good scalability while using the</p>
<p style="margin:0px">> overlapping solvers in DUNE. I tried the following test example in DUNE</p>
<p style="margin:0px">> PDELab, but not getting good scalability.</p>
<p style="margin:0px">> Thank you.</p>
<p style="margin:0px">> </p>
<p style="margin:0px">> $ mpirun -np 1 ./test-heat-instationary-periodic</p>
<p style="margin:0px">> .</p>
<p style="margin:0px">> .</p>
<p style="margin:0px">> .</p>
<p style="margin:0px">> STAGE 2 time (to): 1.5000e-02.</p>
<p style="margin:0px">> === matrix setup skipped (matrix already allocated)</p>
<p style="margin:0px">> === matrix assembly (max) 0.9944 s</p>
<p style="margin:0px">> === residual assembly (max) 0.4861 s</p>
<p style="margin:0px">> === solving (reduction: 1e-10) === Dune::BiCGSTABSolver</p>
<p style="margin:0px">> === rate=0.8397, T=12.57, TIT=0.09524, IT=132</p>
<p style="margin:0px">> 12.68 s</p>
<p style="margin:0px">> ::: timesteps 2 (2)</p>
<p style="margin:0px">> ::: nl iterations 565 (565)</p>
<p style="margin:0px">> ::: lin iterations 565 (565)</p>
<p style="margin:0px">> ::: assemble time 8.0477e+00 (8.0477e+00)</p>
<p style="margin:0px">> ::: lin solve time 5.3414e+01 (5.3414e+01)</p>
<p style="margin:0px">> ---------------------------------------------------------------------------------------</p>
<p style="margin:0px">> $ mpirun -np 2 ./testheat-instationary-periodic</p>
<p style="margin:0px">> .</p>
<p style="margin:0px">> .</p>
<p style="margin:0px">> .</p>
<p style="margin:0px">> STAGE 2 time (to): 1.5000e-02.</p>
<p style="margin:0px">> === matrix setup skipped (matrix already allocated)</p>
<p style="margin:0px">> === matrix assembly (max) 0.5053 s</p>
<p style="margin:0px">> === residual assembly (max) 0.2465 s</p>
<p style="margin:0px">> === solving (reduction: 1e-10) === Dune::BiCGSTABSolver</p>
<p style="margin:0px">> === rate=0.9268, T=26.95, TIT=0.08895, IT=303</p>
<p style="margin:0px">> 27.05 s</p>
<p style="margin:0px">> ::: timesteps 2 (2)</p>
<p style="margin:0px">> ::: nl iterations 1254 (1254)</p>
<p style="margin:0px">> ::: lin iterations 1254 (1254)</p>
<p style="margin:0px">> ::: assemble time 4.0910e+00 (4.0910e+00)</p>
<p style="margin:0px">> ::: lin solve time 1.1201e+02 (1.1201e+02)</p>
<p style="margin:0px">> ---------------------------------------------------------------------------------------</p>
<p style="margin:0px">> $ mpirun -np 4 ./testheat-instationary-periodic</p>
<p style="margin:0px">> .</p>
<p style="margin:0px">> .</p>
<p style="margin:0px">> .</p>
<p style="margin:0px">> STAGE 2 time (to): 1.5000e-02.</p>
<p style="margin:0px">> === matrix setup skipped (matrix already allocated)</p>
<p style="margin:0px">> === matrix assembly (max) 0.271 s</p>
<p style="margin:0px">> === residual assembly (max) 0.1318 s</p>
<p style="margin:0px">> === solving (reduction: 1e-10) === Dune::BiCGSTABSolver</p>
<p style="margin:0px">> === rate=0.9232, T=26.02, TIT=0.0894, IT=291</p>
<p style="margin:0px">> 26.11 s</p>
<p style="margin:0px">> ::: timesteps 2 (2)</p>
<p style="margin:0px">> ::: nl iterations 1249 (1249)</p>
<p style="margin:0px">> ::: lin iterations 1249 (1249)</p>
<p style="margin:0px">> ::: assemble time 2.1746e+00 (2.1746e+00)</p>
<p style="margin:0px">> ::: lin solve time 1.1165e+02 (1.1165e+02)</p>
<p style="margin:0px">> ---------------------------------------------------------------------------------------</p>
<p style="margin:0px">> $ mpirun -np 8 ./testheat-instationary-periodic</p>
<p style="margin:0px">> .</p>
<p style="margin:0px">> .</p>
<p style="margin:0px">> .</p>
<p style="margin:0px">> STAGE 2 time (to): 1.5000e-02.</p>
<p style="margin:0px">> === matrix setup skipped (matrix already allocated)</p>
<p style="margin:0px">> === matrix assembly (max) 0.1772 s</p>
<p style="margin:0px">> === residual assembly (max) 0.08259 s</p>
<p style="margin:0px">> === solving (reduction: 1e-10) === Dune::BiCGSTABSolver</p>
<p style="margin:0px">> === rate=0.9288, T=30.81, TIT=0.09751, IT=316</p>
<p style="margin:0px">> 30.89 s</p>
<p style="margin:0px">> ::: timesteps 2 (2)</p>
<p style="margin:0px">> ::: nl iterations 1329 (1329)</p>
<p style="margin:0px">> ::: lin iterations 1329 (1329)</p>
<p style="margin:0px">> ::: assemble time 1.3485e+00 (1.3485e+00)</p>
<p style="margin:0px">> ::: lin solve time 1.2796e+02 (1.2796e+02)</p>
<p style="margin:0px">> </p>
<p style="margin:0px">> </p>
<p style="margin:0px">> </p>
<br><br></div>
_______________________________________________<br>
dune-pdelab mailing list<br>
<a href="mailto:dune-pdelab@lists.dune-project.org" target="_blank">dune-pdelab@lists.dune-project.org</a><br>
<a href="https://lists.dune-project.org/mailman/listinfo/dune-pdelab" rel="noreferrer" target="_blank">https://lists.dune-project.org/mailman/listinfo/dune-pdelab</a><br>
</blockquote></div>