<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
Dear Aswin,<br>
<br>
admittedly your results are a bit strange.<br>
<br>
- The speedup in assembly (results in your previous mail) seem to be
ok.<br>
<br>
- Your processor Intel Core™ i7-10700 CPU @ 2.90GHz has only two
memory channels,<br>
so I would not expect a speedup much beyond 2 but, this might
explain the numbers when you<br>
go from from 2 to 4 processors. I am also not sure about the Turbo
mode. Can you switch it off?<br>
On the other hand, the assembly seems to be fine.<br>
<br>
- In your old mail with the inexact solver in additive Schwarz you
had also included one processor<br>
but there was hardly any speedup from 1 to 2. <br>
<br>
You might go back to that ISTLBackend_OVLP_BCGS_ILU0 Solver and try
different problem sizes<br>
(e.g. such that the local problem in one process fits into cache)
and inner iterations (to increase the<br>
ration of computation to communication) and see what you get. <br>
<br>
You might also want to try the ISTLBackend_BCGS_AMG_SSOR solver.
This one can use also only one subdomain.<br>
<br>
Best regards,<br>
<br>
Peter<br>
<br>
<br>
<div class="moz-cite-prefix">Am 11.01.22 um 13:09 schrieb Aswin vs:<br>
</div>
<blockquote type="cite"
cite="mid:CADDEfOCsjPEpbS3YqBqSfa6z+fVQ17NDi7MB8r+HddMdpRk1NA@mail.gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div id="gmail-:1d6" class="gmail-Ar gmail-Au gmail-Ao"
style="display:block">
<div id="gmail-:1d2" class="gmail-Am gmail-Al editable
gmail-LW-avf gmail-tS-tW gmail-tS-tY" aria-label="Message
Body" role="textbox" aria-multiline="true"
style="direction:ltr;min-height:376px" tabindex="1">
<div>Sir,</div>
<div>As you suggested, I am currently looking into the GenEO
examples given in the dune-composite-master. Still I
couldn't see any scalability in the test example. <br>
</div>
<div><br>
</div>
<div>In the test Example01b I have edited the line
Partitioning[0] = 2 and Partitioning[0] = 4 for 2 and 4
numbers of processors, respectively. If I change it to
Partitioning[0] = 8 then I am getting the error " YaspGrid
does not support degrees of freedom shared by more than
immediately neighboring subdomains." Then I changed the
refineBaseGrid = 2, but it had the same error. <br>
</div>
<div><br>
</div>
<div>Presently, I am using the desktop having Intel Core™
i7-10700 CPU @ 2.90GHz × 1 processor, that has 8/16 number
of cores/threads.</div>
<div><br>
</div>
<div> The outputs of the simple runs are given below for
your reference. In each case, I found the rate is also not
strictly decreasing. The number of iterations is
increasing and the overall time is also increasing.</div>
<div><br>
</div>
<div>Can you suggest to me how to run these examples
efficiently? Also I would like to know how to print the
dim(V_H) as you mentioned in the paper. <br>
</div>
<div><br>
</div>
<div>Thank you for your time and consideration,<br>
</div>
<div><br>
</div>
<div>$ mpirun -np 2 ./Example01b </div>
*NumRegions<br>
1<br>
Number of Regions = 1<br>
maxPly = 23<br>
periodic = 000<br>
=== Building Geometry<br>
Elements in x direction <br>
Number of elements per processor: 1925<br>
Number of nodes per processor: 2592<br>
Grid transformation complete<br>
Grid view set up<br>
Piecewise quadratic serendipity elements<br>
Starting solve with Geneo Preconditioner<br>
Eigenvalue threshhold: 0.00769231<br>
Process 1 picked 6 eigenvectors<br>
Process 0 picked 2 eigenvectors<br>
Matrix setup<br>
Global basis size B=8<br>
Matrix setup finished: M=0.0490446<br>
Geneo setup time 23.4082<br>
=== Dune::CGSolver<br>
Min eigv estimate: 0.0676291<br>
Max eigv estimate: 2.99961<br>
Condition estimate: 44.3538<br>
=== rate=0.534884, T=2.99187, TIT=0.135994, IT=22<br>
Solver: CG<br>
Preconditioner: GenEO<br>
Subdomain Solver: UMFPack<br>
=================<br>
<div>Solver Converged: 1Solution of my Problem = 0.000166984</div>
<div><br>
</div>
<div><br>
</div>
<div>$ mpirun -np 4 ./Example01b <br>
*NumRegions<br>
1<br>
Number of Regions = 1<br>
maxPly = 23<br>
periodic = 000<br>
=== Building Geometry<br>
Elements in x direction <br>
Number of elements per processor: 1050<br>
Number of nodes per processor: 1512<br>
Grid transformation complete<br>
Grid view set up<br>
Piecewise quadratic serendipity elements<br>
Starting solve with Geneo Preconditioner<br>
Eigenvalue threshhold: 0.00909091<br>
Process 2 picked 6 eigenvectors<br>
Process 3 picked 6 eigenvectors<br>
Process 1 picked 6 eigenvectors<br>
Process 0 picked 1 eigenvectors<br>
Matrix setup<br>
Global basis size B=19<br>
Matrix setup finished: M=0.119121<br>
Geneo setup time 24.9791<br>
=== Dune::CGSolver<br>
Min eigv estimate: 0.0249189<br>
Max eigv estimate: 2.9999<br>
Condition estimate: 120.386<br>
=== rate=0.769396, T=5.08444, TIT=0.110531, IT=46<br>
Solver: CG<br>
Preconditioner: GenEO<br>
Subdomain Solver: UMFPack<br>
=================<br>
Solver Converged: 1Solution of my Problem = 0.000166984</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
Thank you.<br clear="all">
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Jan 3, 2022 at 11:41
PM Linus Seelinger <<a
href="mailto:linus.seelinger@iwr.uni-heidelberg.de"
moz-do-not-send="true" class="moz-txt-link-freetext">linus.seelinger@iwr.uni-heidelberg.de</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p style="margin:0px">Hi Aswin,</p>
<br>
<p style="margin:0px">first of all, I think you might be
mislead by how periodic boundaries are handled in DUNE.
Periodic boundaries (at least using YASPGrid) require a
parallel run (i.e. more than one MPI rank), since they
essentially use the same overlapping communication
framework that otherwise handles "regular" overlaps
between subdomains. Think of a 2D grid, gluing together
the periodic ends; in the resulting cylinder shape,
subdomains at periodic boundaries are just regular
neighbors and can be handled accordingly.</p>
<br>
<p style="margin:0px">Long story short, I think the
sequential (-np 1) run does not give you a correct
solution (write to VTK and check the output to confirm)
and is therefore not a good reference. The other runs look
not as bad regarding scalability.</p>
<br>
<p style="margin:0px">If you still need better scalability,
you might look into more advanced methods. The appropriate
choice then depends a lot on the particular problem you
want to solve, so more detail from you would be helpful.</p>
<br>
<p style="margin:0px">One example might be GenEO, which we
could scale up to around 16000 cores (see <a
href="https://doi.org/10.1007/978-3-030-43229-4_11"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://doi.org/10.1007/978-3-030-43229-4_11</a> ).
Might be a bit overkill though depending on what you want
to do.</p>
<br>
<p style="margin:0px">Best,</p>
<br>
<p style="margin:0px">Linus</p>
<br>
<p style="margin:0px">Am Montag, 3. Januar 2022, 11:09:14
CET schrieb Aswin vs:</p>
<p style="margin:0px">> Hello,</p>
<p style="margin:0px">> </p>
<p style="margin:0px">> Can somebody suggest me how to
get good scalability while using the</p>
<p style="margin:0px">> overlapping solvers in DUNE. I
tried the following test example in DUNE</p>
<p style="margin:0px">> PDELab, but not getting good
scalability.</p>
<p style="margin:0px">> Thank you.</p>
<p style="margin:0px">> </p>
<p style="margin:0px">> $ mpirun -np 1
./test-heat-instationary-periodic</p>
<p style="margin:0px">> .</p>
<p style="margin:0px">> .</p>
<p style="margin:0px">> .</p>
<p style="margin:0px">> STAGE 2 time (to): 1.5000e-02.</p>
<p style="margin:0px">> === matrix setup skipped (matrix
already allocated)</p>
<p style="margin:0px">> === matrix assembly (max) 0.9944
s</p>
<p style="margin:0px">> === residual assembly (max)
0.4861 s</p>
<p style="margin:0px">> === solving (reduction: 1e-10)
=== Dune::BiCGSTABSolver</p>
<p style="margin:0px">> === rate=0.8397, T=12.57,
TIT=0.09524, IT=132</p>
<p style="margin:0px">> 12.68 s</p>
<p style="margin:0px">> ::: timesteps 2 (2)</p>
<p style="margin:0px">> ::: nl iterations 565 (565)</p>
<p style="margin:0px">> ::: lin iterations 565 (565)</p>
<p style="margin:0px">> ::: assemble time 8.0477e+00
(8.0477e+00)</p>
<p style="margin:0px">> ::: lin solve time 5.3414e+01
(5.3414e+01)</p>
<p style="margin:0px">>
---------------------------------------------------------------------------------------</p>
<p style="margin:0px">> $ mpirun -np 2
./testheat-instationary-periodic</p>
<p style="margin:0px">> .</p>
<p style="margin:0px">> .</p>
<p style="margin:0px">> .</p>
<p style="margin:0px">> STAGE 2 time (to): 1.5000e-02.</p>
<p style="margin:0px">> === matrix setup skipped (matrix
already allocated)</p>
<p style="margin:0px">> === matrix assembly (max) 0.5053
s</p>
<p style="margin:0px">> === residual assembly (max)
0.2465 s</p>
<p style="margin:0px">> === solving (reduction: 1e-10)
=== Dune::BiCGSTABSolver</p>
<p style="margin:0px">> === rate=0.9268, T=26.95,
TIT=0.08895, IT=303</p>
<p style="margin:0px">> 27.05 s</p>
<p style="margin:0px">> ::: timesteps 2 (2)</p>
<p style="margin:0px">> ::: nl iterations 1254 (1254)</p>
<p style="margin:0px">> ::: lin iterations 1254 (1254)</p>
<p style="margin:0px">> ::: assemble time 4.0910e+00
(4.0910e+00)</p>
<p style="margin:0px">> ::: lin solve time 1.1201e+02
(1.1201e+02)</p>
<p style="margin:0px">>
---------------------------------------------------------------------------------------</p>
<p style="margin:0px">> $ mpirun -np 4
./testheat-instationary-periodic</p>
<p style="margin:0px">> .</p>
<p style="margin:0px">> .</p>
<p style="margin:0px">> .</p>
<p style="margin:0px">> STAGE 2 time (to): 1.5000e-02.</p>
<p style="margin:0px">> === matrix setup skipped (matrix
already allocated)</p>
<p style="margin:0px">> === matrix assembly (max) 0.271 s</p>
<p style="margin:0px">> === residual assembly (max)
0.1318 s</p>
<p style="margin:0px">> === solving (reduction: 1e-10)
=== Dune::BiCGSTABSolver</p>
<p style="margin:0px">> === rate=0.9232, T=26.02,
TIT=0.0894, IT=291</p>
<p style="margin:0px">> 26.11 s</p>
<p style="margin:0px">> ::: timesteps 2 (2)</p>
<p style="margin:0px">> ::: nl iterations 1249 (1249)</p>
<p style="margin:0px">> ::: lin iterations 1249 (1249)</p>
<p style="margin:0px">> ::: assemble time 2.1746e+00
(2.1746e+00)</p>
<p style="margin:0px">> ::: lin solve time 1.1165e+02
(1.1165e+02)</p>
<p style="margin:0px">>
---------------------------------------------------------------------------------------</p>
<p style="margin:0px">> $ mpirun -np 8
./testheat-instationary-periodic</p>
<p style="margin:0px">> .</p>
<p style="margin:0px">> .</p>
<p style="margin:0px">> .</p>
<p style="margin:0px">> STAGE 2 time (to): 1.5000e-02.</p>
<p style="margin:0px">> === matrix setup skipped (matrix
already allocated)</p>
<p style="margin:0px">> === matrix assembly (max) 0.1772
s</p>
<p style="margin:0px">> === residual assembly (max)
0.08259 s</p>
<p style="margin:0px">> === solving (reduction: 1e-10)
=== Dune::BiCGSTABSolver</p>
<p style="margin:0px">> === rate=0.9288, T=30.81,
TIT=0.09751, IT=316</p>
<p style="margin:0px">> 30.89 s</p>
<p style="margin:0px">> ::: timesteps 2 (2)</p>
<p style="margin:0px">> ::: nl iterations 1329 (1329)</p>
<p style="margin:0px">> ::: lin iterations 1329 (1329)</p>
<p style="margin:0px">> ::: assemble time 1.3485e+00
(1.3485e+00)</p>
<p style="margin:0px">> ::: lin solve time 1.2796e+02
(1.2796e+02)</p>
<p style="margin:0px">> </p>
<p style="margin:0px">> </p>
<p style="margin:0px">> </p>
<br>
<br>
</div>
_______________________________________________<br>
dune-pdelab mailing list<br>
<a href="mailto:dune-pdelab@lists.dune-project.org"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">dune-pdelab@lists.dune-project.org</a><br>
<a
href="https://lists.dune-project.org/mailman/listinfo/dune-pdelab"
rel="noreferrer" target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://lists.dune-project.org/mailman/listinfo/dune-pdelab</a><br>
</blockquote>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
dune-pdelab mailing list
<a class="moz-txt-link-abbreviated" href="mailto:dune-pdelab@lists.dune-project.org">dune-pdelab@lists.dune-project.org</a>
<a class="moz-txt-link-freetext" href="https://lists.dune-project.org/mailman/listinfo/dune-pdelab">https://lists.dune-project.org/mailman/listinfo/dune-pdelab</a>
</pre>
</blockquote>
<br>
</body>
</html>