<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div>Hi Steffen,</div><div><br></div><div>METIS was not that big of a deal, thanks to you.</div><div>SuperLU is still making some trouble as explained below:</div><br><div><div>Am 25.06.2014 um 14:20 schrieb Steffen Müthing <<a href="mailto:steffen.muething@iwr.uni-heidelberg.de">steffen.muething@iwr.uni-heidelberg.de</a>>:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">Hi Lukas,<br><br>(Even though the problem is reported against PDELab, I’ll keep this on the main<br>mailing list, because the actual issues are not PDELab-related)<br><br>Am 25.06.2014 um 10:55 schrieb Lukas Riedel <<a href="mailto:riedel-lukas@gmx.de">riedel-lukas@gmx.de</a>>:<br><br><blockquote type="cite">Dear Developers,<br>i ran into two problems creating a code for parallel computation with PDELab. It works, but I would like DUNE to use ParMETIS and SuperLU as it requests.<br><br>OS: Mac OS X 10.9.3 Mavericks<br>GCC: gcc (MacPorts gcc49 4.9-20140416_2) 4.9.0 20140416 (prerelease)<br>G++: g++ (MacPorts gcc49 4.9-20140416_2) 4.9.0 20140416 (prerelease)<br>clang:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>clang version 3.5.0 (trunk 210448)<br><span class="Apple-tab-span" style="white-space: pre;"> </span>Target: x86_64-apple-darwin13.2.0<br><span class="Apple-tab-span" style="white-space: pre;"> </span>Thread model: posix<br>CC:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Apple LLVM version 5.1 (clang-503.0.40) (based on LLVM 3.4svn)<br><span class="Apple-tab-span" style="white-space: pre;"> </span>Target: x86_64-apple-darwin13.2.0<br><span class="Apple-tab-span" style="white-space: pre;"> </span>Thread model: posix<br>DUNE-modules:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>dune-common.............: version 2.3.0<br><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>dune-geometry...........: version 2.3.0<br><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>dune-grid...............: version 2.3.0<br><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>dune-istl...............: version 2.3.0<br><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>dune-localfunctions.....: version 2.3.0<br><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>dune-pdelab.............: version 2.0-dev<br><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>dune-typetree...........: version 2.3-dev<br><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>ALBERTA.................: version 2.0<br><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>ALUGrid.................: version 1.52 (parallel)<br><br>I attached the output of ./dunecontrol as a txt-file.<br><br>I run the following command inside the DUNE-directory:<br><span class="Apple-tab-span" style="white-space: pre;"> </span>$ ./dune-common/bin/dunecontrol --only=<module> --opts=parallel-debug.opts all<br><br>I use the parallel-debug.opts file provided by the archive of the DUNE course in March 2014:<br><span class="Apple-tab-span" style="white-space: pre;"> </span># use these options for configure if no options a provided on the cmdline<br><span class="Apple-tab-span" style="white-space: pre;"> </span>CONFIGURE_FLAGS="--prefix=/Users/lriedel/opt --disable-documentation -disable-mpiruntest --enable-parallel --with-alugrid=/Users/lriedel/opt/alugrid —with alberta=/Users/lriedel/opt/alberta CXX=g++ CC=gcc CXXFLAGS=\"-O0 -g -Wall -Wno-deprecated-declarations -Wa,-q\" CFLAGS=\"-O0 -g -Wall -Wno-deprecated-declarations -Wa,-q\""<br><span class="Apple-tab-span" style="white-space: pre;"> </span>MAKE_FLAGS=„all"<br><br><br>Problems:<br><br>1. I installed METIS and ParMETIS and DUNE recognizes them when building a module via ./dunecontrol , but while running the program on multiple processes I still receive the following output:<br><br>TIME STEP [Alexander (order 2)] 1 time (from): 0.0000e+00 dt: 1.0000e-01 time (to): 1.0000e-01<br>STAGE 1 time (to): 2.9289e-02.<br> Initial defect: 9.5196e-06<br>Successive accumulation of data on coarse levels only works with ParMETIS installed. Fell back to accumulation to one domain on coarsest level<br> Newton iteration 1. New defect: 7.2254e-06. Reduction (this): 7.5900e-01. Reduction (total): 7.5900e-01<br>Successive accumulation of data on coarse levels only works with ParMETIS installed. Fell back to accumulation to one domain on coarsest level<br> Newton iteration 2. New defect: 4.1631e-07. Reduction (this): 5.7618e-02. Reduction (total): 4.3732e-02<br><br>I am using the following linear solver backend (currently suppressing SuperLU):<span class="Apple-converted-space"> </span><br><br> typedef Dune::PDELab::ISTLBackend_BCGS_AMG_SSOR<IGO> LS;<br> LS ls(gfs,5000,0,false,false);<br><br>followed by the Newton Solver<br><br> typedef Dune::PDELab::Newton<IGO,LS,U> PDESOLVER;<br> PDESOLVER pdesolver(igo,ls);<br> pdesolver.setReassembleThreshold(0.0);<br> pdesolver.setVerbosityLevel(2);<br> pdesolver.setReduction(1e-10);<br> pdesolver.setMinLinearReduction(1e-4);<br> pdesolver.setMaxIterations(6);<br> pdesolver.setLineSearchMaxIterations(10);<br><br>and a OneStepMethod<br><br> Dune::PDELab::Alexander2Parameter<Real> alex2;<br> Dune::PDELab::OneStepMethod<Real,IGO,PDESOLVER,U,U> osm(alex2,igo,pdesolver);<br><br>Do I have to tell DUNE explicitly that it shall use ParMETIS?<br></blockquote><br>If you want your program to use ParMETIS, you have to add the PARMETIS_{CPPFLAGS,LDFLAGS,LIBS} to the respective<br>automake variables for your program (take a look at dune-pdelab-howto/src/two-phase/Makefile.am for an example). The<br>PARMETIS_CPPFLAGS set a special macro ENABLE_PARMETIS to 1. Without that macro set, dune-istl will not activate<br>ParMETIS support regardless of whether ParMETIS was found or not.<br><br>This requirement of having to add $(PKGNAME)_CPPFLAGS is pretty common in the DUNE build system BTW, you also need<br>it for stuff like SuperLU, GMP etc.<br><br><blockquote type="cite"><br><br>2. I do not know where to install SuperLU.<br><br>I installed it to Users/lriedel/opt/SuperLU_4.3/ as this is the directory mentioned as „prefix“ in the opts-file, but DUNE can obviously not find it (see output-file)<br>DUNE seems to look for it inside /usr/local/SRC but it is only possible to install the newest version of SuperLU inside a folder structure as follows: ../SuperLU_4.3/SRC/<br>Is there a way to specify where I installed SuperLU so that DUNE can find its libraries?<br></blockquote><br>Well, SuperLU’s build system is very limited and doesn’t really support installing stuff anywhere. It can be configured to a certain<br>extent by editing the file make.inc in the root of the SuperLU source tree. I don’t know how other people handle it, but I (working on<br>OS X like you) usually set the variable SuperLUroot in that file to something like $(HOME)/local/SuperLU-4.3. That will cause the library<br>to be built in $(HOME)/local/SuperLU-4.3/lib. But as mentioned, the build system isn’t very clever, so you have to manually create that<br>directory before calling make with "mkdir -p $(HOME)/local/SuperLU-4.3/lib". In order to keep stuff together, I then create a directory<br>$(HOME)/local/SuperLU-4.3/include and copy all header files from the directory SRC in the SuperLU source tree to that directory.<br>Afterwards, you should run "chmod +r *“ in the include/ directory because one of the headers has weird permissions. Finally I add a<br>symlink SRC -> include to $(HOME)/local/SuperLU-4.3.<br></div></blockquote><div><br></div><div>You need to specify this last command. I built a symlink to the directory that i put all the *.h files into:</div><div><br></div><div><div style="margin: 0px; font-size: 11px; font-family: Menlo;">usr/local/SRC -> ../../Users/lriedel/opt/SuperLU_4.3/include </div><div style="margin: 0px; font-size: 11px; font-family: Menlo;"><br></div><div style="margin: 0px;">Since I did not know better, I also but the SuperLU_DIST header files into this folder.</div><div style="margin: 0px;">Now DUNE can find the header-files, but it still is not content (using the configure flags you mentioned below):</div><div style="margin: 0px;"><br></div><div style="margin: 0px;"><div style="margin: 0px; font-size: 11px; font-family: Menlo;">checking slu_ddefs.h usability... yes</div><div style="margin: 0px; font-size: 11px; font-family: Menlo;">checking slu_ddefs.h presence... yes</div><div style="margin: 0px; font-size: 11px; font-family: Menlo;">checking for slu_ddefs.h... yes</div><div style="margin: 0px; font-size: 11px; font-family: Menlo;">checking for dgssvx... no</div><div style="margin: 0px; font-size: 11px; font-family: Menlo;">checking SuperLU... no</div><div style="margin: 0px; font-size: 11px; font-family: Menlo;">checking for SuperLUDist library... /usr/local/SRC</div><div style="margin: 0px; font-size: 11px; font-family: Menlo;">checking superlu_ddefs.h usability... yes</div><div style="margin: 0px; font-size: 11px; font-family: Menlo;">checking superlu_ddefs.h presence... yes</div><div style="margin: 0px; font-size: 11px; font-family: Menlo;">checking for superlu_ddefs.h... yes</div><div style="margin: 0px; font-size: 11px; font-family: Menlo;">checking for pdgssvx in -lsuperlu-mpi... no</div><div style="margin: 0px; font-size: 11px; font-family: Menlo;">configure: WARNING: libsuperlu-mpi not found</div><div style="margin: 0px; font-size: 11px; font-family: Menlo;">checking static SuperLUDist library superlu_mpi.a in "/usr/local/"... failed</div><div style="margin: 0px; font-size: 11px; font-family: Menlo;">checking SuperLUDist in /usr/local... failed</div><div><br></div><div>For OpenMPI I also need SuperLU_DIST (right?). Where do I have to put this and how do I have to link it? The problem is that the „superlu_ddefs.h“ and „slu_ddefs.h“ are from the two different packages of SuperLU, but DUNE seems to look for them in the same directory. If so, how shall I put both packages together?</div><div>Lastly, even after making both packages, there does not seem to be a „superlu_mpi.a“ file on my whole computer.</div></div></div><br><blockquote type="cite"><div style="font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br>After installing SuperLU that way, you still have to tell DUNE where to find SuperLU by adding<br>"—with-superlu=$(HOME)/local/SuperLU-4.3 --with-superlu-lib=libsuperlu_4.3.a“ to the DUNE CONFIGURE_FLAGS.<br><br></div></blockquote><blockquote type="cite"><div style="font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br>Hope that helps…<br><br>Steffen<br><br><br><blockquote type="cite"><br><br>Thank you and best regards<br>Lukas Riedel<br><br><br><dunecontrol-output.txt><br>_______________________________________________<br>Dune mailing list<br><a href="mailto:Dune@dune-project.org">Dune@dune-project.org</a><br><a href="http://lists.dune-project.org/mailman/listinfo/dune">http://lists.dune-project.org/mailman/listinfo/dune</a></blockquote></div></blockquote><br></div><div><br></div><div>Thanks a lot</div><div>Lukas</div><br></body></html>