[Dune] Parallel Computation: ISTLBackend requests ParMETIS and SuperLU
Lukas Riedel
riedel-lukas at gmx.de
Wed Jun 25 19:46:24 CEST 2014
Hi Steffen,
METIS was not that big of a deal, thanks to you.
SuperLU is still making some trouble as explained below:
Am 25.06.2014 um 14:20 schrieb Steffen Müthing <steffen.muething at iwr.uni-heidelberg.de>:
> Hi Lukas,
>
> (Even though the problem is reported against PDELab, I’ll keep this on the main
> mailing list, because the actual issues are not PDELab-related)
>
> Am 25.06.2014 um 10:55 schrieb Lukas Riedel <riedel-lukas at gmx.de>:
>
>> Dear Developers,
>> i ran into two problems creating a code for parallel computation with PDELab. It works, but I would like DUNE to use ParMETIS and SuperLU as it requests.
>>
>> OS: Mac OS X 10.9.3 Mavericks
>> GCC: gcc (MacPorts gcc49 4.9-20140416_2) 4.9.0 20140416 (prerelease)
>> G++: g++ (MacPorts gcc49 4.9-20140416_2) 4.9.0 20140416 (prerelease)
>> clang: clang version 3.5.0 (trunk 210448)
>> Target: x86_64-apple-darwin13.2.0
>> Thread model: posix
>> CC: Apple LLVM version 5.1 (clang-503.0.40) (based on LLVM 3.4svn)
>> Target: x86_64-apple-darwin13.2.0
>> Thread model: posix
>> DUNE-modules: dune-common.............: version 2.3.0
>> dune-geometry...........: version 2.3.0
>> dune-grid...............: version 2.3.0
>> dune-istl...............: version 2.3.0
>> dune-localfunctions.....: version 2.3.0
>> dune-pdelab.............: version 2.0-dev
>> dune-typetree...........: version 2.3-dev
>> ALBERTA.................: version 2.0
>> ALUGrid.................: version 1.52 (parallel)
>>
>> I attached the output of ./dunecontrol as a txt-file.
>>
>> I run the following command inside the DUNE-directory:
>> $ ./dune-common/bin/dunecontrol --only=<module> --opts=parallel-debug.opts all
>>
>> I use the parallel-debug.opts file provided by the archive of the DUNE course in March 2014:
>> # use these options for configure if no options a provided on the cmdline
>> CONFIGURE_FLAGS="--prefix=/Users/lriedel/opt --disable-documentation -disable-mpiruntest --enable-parallel --with-alugrid=/Users/lriedel/opt/alugrid —with alberta=/Users/lriedel/opt/alberta CXX=g++ CC=gcc CXXFLAGS=\"-O0 -g -Wall -Wno-deprecated-declarations -Wa,-q\" CFLAGS=\"-O0 -g -Wall -Wno-deprecated-declarations -Wa,-q\""
>> MAKE_FLAGS=„all"
>>
>>
>> Problems:
>>
>> 1. I installed METIS and ParMETIS and DUNE recognizes them when building a module via ./dunecontrol , but while running the program on multiple processes I still receive the following output:
>>
>> TIME STEP [Alexander (order 2)] 1 time (from): 0.0000e+00 dt: 1.0000e-01 time (to): 1.0000e-01
>> STAGE 1 time (to): 2.9289e-02.
>> Initial defect: 9.5196e-06
>> Successive accumulation of data on coarse levels only works with ParMETIS installed. Fell back to accumulation to one domain on coarsest level
>> Newton iteration 1. New defect: 7.2254e-06. Reduction (this): 7.5900e-01. Reduction (total): 7.5900e-01
>> Successive accumulation of data on coarse levels only works with ParMETIS installed. Fell back to accumulation to one domain on coarsest level
>> Newton iteration 2. New defect: 4.1631e-07. Reduction (this): 5.7618e-02. Reduction (total): 4.3732e-02
>>
>> I am using the following linear solver backend (currently suppressing SuperLU):
>>
>> typedef Dune::PDELab::ISTLBackend_BCGS_AMG_SSOR<IGO> LS;
>> LS ls(gfs,5000,0,false,false);
>>
>> followed by the Newton Solver
>>
>> typedef Dune::PDELab::Newton<IGO,LS,U> PDESOLVER;
>> PDESOLVER pdesolver(igo,ls);
>> pdesolver.setReassembleThreshold(0.0);
>> pdesolver.setVerbosityLevel(2);
>> pdesolver.setReduction(1e-10);
>> pdesolver.setMinLinearReduction(1e-4);
>> pdesolver.setMaxIterations(6);
>> pdesolver.setLineSearchMaxIterations(10);
>>
>> and a OneStepMethod
>>
>> Dune::PDELab::Alexander2Parameter<Real> alex2;
>> Dune::PDELab::OneStepMethod<Real,IGO,PDESOLVER,U,U> osm(alex2,igo,pdesolver);
>>
>> Do I have to tell DUNE explicitly that it shall use ParMETIS?
>
> If you want your program to use ParMETIS, you have to add the PARMETIS_{CPPFLAGS,LDFLAGS,LIBS} to the respective
> automake variables for your program (take a look at dune-pdelab-howto/src/two-phase/Makefile.am for an example). The
> PARMETIS_CPPFLAGS set a special macro ENABLE_PARMETIS to 1. Without that macro set, dune-istl will not activate
> ParMETIS support regardless of whether ParMETIS was found or not.
>
> This requirement of having to add $(PKGNAME)_CPPFLAGS is pretty common in the DUNE build system BTW, you also need
> it for stuff like SuperLU, GMP etc.
>
>>
>>
>> 2. I do not know where to install SuperLU.
>>
>> I installed it to Users/lriedel/opt/SuperLU_4.3/ as this is the directory mentioned as „prefix“ in the opts-file, but DUNE can obviously not find it (see output-file)
>> DUNE seems to look for it inside /usr/local/SRC but it is only possible to install the newest version of SuperLU inside a folder structure as follows: ../SuperLU_4.3/SRC/
>> Is there a way to specify where I installed SuperLU so that DUNE can find its libraries?
>
> Well, SuperLU’s build system is very limited and doesn’t really support installing stuff anywhere. It can be configured to a certain
> extent by editing the file make.inc in the root of the SuperLU source tree. I don’t know how other people handle it, but I (working on
> OS X like you) usually set the variable SuperLUroot in that file to something like $(HOME)/local/SuperLU-4.3. That will cause the library
> to be built in $(HOME)/local/SuperLU-4.3/lib. But as mentioned, the build system isn’t very clever, so you have to manually create that
> directory before calling make with "mkdir -p $(HOME)/local/SuperLU-4.3/lib". In order to keep stuff together, I then create a directory
> $(HOME)/local/SuperLU-4.3/include and copy all header files from the directory SRC in the SuperLU source tree to that directory.
> Afterwards, you should run "chmod +r *“ in the include/ directory because one of the headers has weird permissions. Finally I add a
> symlink SRC -> include to $(HOME)/local/SuperLU-4.3.
You need to specify this last command. I built a symlink to the directory that i put all the *.h files into:
usr/local/SRC -> ../../Users/lriedel/opt/SuperLU_4.3/include
Since I did not know better, I also but the SuperLU_DIST header files into this folder.
Now DUNE can find the header-files, but it still is not content (using the configure flags you mentioned below):
checking slu_ddefs.h usability... yes
checking slu_ddefs.h presence... yes
checking for slu_ddefs.h... yes
checking for dgssvx... no
checking SuperLU... no
checking for SuperLUDist library... /usr/local/SRC
checking superlu_ddefs.h usability... yes
checking superlu_ddefs.h presence... yes
checking for superlu_ddefs.h... yes
checking for pdgssvx in -lsuperlu-mpi... no
configure: WARNING: libsuperlu-mpi not found
checking static SuperLUDist library superlu_mpi.a in "/usr/local/"... failed
checking SuperLUDist in /usr/local... failed
For OpenMPI I also need SuperLU_DIST (right?). Where do I have to put this and how do I have to link it? The problem is that the „superlu_ddefs.h“ and „slu_ddefs.h“ are from the two different packages of SuperLU, but DUNE seems to look for them in the same directory. If so, how shall I put both packages together?
Lastly, even after making both packages, there does not seem to be a „superlu_mpi.a“ file on my whole computer.
>
> After installing SuperLU that way, you still have to tell DUNE where to find SuperLU by adding
> "—with-superlu=$(HOME)/local/SuperLU-4.3 --with-superlu-lib=libsuperlu_4.3.a“ to the DUNE CONFIGURE_FLAGS.
>
>
> Hope that helps…
>
> Steffen
>
>
>>
>>
>> Thank you and best regards
>> Lukas Riedel
>>
>>
>> <dunecontrol-output.txt>
>> _______________________________________________
>> Dune mailing list
>> Dune at dune-project.org
>> http://lists.dune-project.org/mailman/listinfo/dune
Thanks a lot
Lukas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dune-project.org/pipermail/dune/attachments/20140625/6bfdb321/attachment.htm>
More information about the Dune
mailing list