[dune-pdelab] [Dune] Parallel Computation: ISTLBackend requests ParMETIS and SuperLU

[Steffen Müthing]

Hi Lukas,

(Even though the problem is reported against PDELab, I’ll keep this on the main
mailing list, because the actual issues are not PDELab-related)

Am 25.06.2014 um 10:55 schrieb Lukas Riedel <riedel-lukas at gmx.de>:

> Dear Developers,
> i ran into two problems creating a code for parallel computation with PDELab. It works, but I would like DUNE to use ParMETIS and SuperLU as it requests.
> 
> OS: Mac OS X 10.9.3 Mavericks
> GCC: gcc (MacPorts gcc49 4.9-20140416_2) 4.9.0 20140416 (prerelease)
> G++: g++ (MacPorts gcc49 4.9-20140416_2) 4.9.0 20140416 (prerelease)
> clang: 	clang version 3.5.0 (trunk 210448)
> 	Target: x86_64-apple-darwin13.2.0
> 	Thread model: posix
> CC: 		Apple LLVM version 5.1 (clang-503.0.40) (based on LLVM 3.4svn)
> 	Target: x86_64-apple-darwin13.2.0
> 	Thread model: posix
> DUNE-modules: 	dune-common.............: version 2.3.0
> 		dune-geometry...........: version 2.3.0
> 		dune-grid...............: version 2.3.0
> 		dune-istl...............: version 2.3.0
> 		dune-localfunctions.....: version 2.3.0
> 		dune-pdelab.............: version 2.0-dev
> 		dune-typetree...........: version 2.3-dev
> 		ALBERTA.................: version 2.0
> 		ALUGrid.................: version 1.52 (parallel)
> 
> I attached the output of ./dunecontrol as a txt-file.
> 
> I run the following command inside the DUNE-directory:
> 	$ ./dune-common/bin/dunecontrol --only=<module> --opts=parallel-debug.opts all
> 
> I use the parallel-debug.opts file provided by the archive of the DUNE course in March 2014:
> 	# use these options for configure if no options a provided on the cmdline
> 	CONFIGURE_FLAGS="--prefix=/Users/lriedel/opt --disable-documentation -disable-mpiruntest --enable-parallel --with-alugrid=/Users/lriedel/opt/alugrid —with alberta=/Users/lriedel/opt/alberta CXX=g++ CC=gcc CXXFLAGS=\"-O0 -g -Wall -Wno-deprecated-declarations -Wa,-q\" CFLAGS=\"-O0 -g -Wall -Wno-deprecated-declarations -Wa,-q\""
> 	MAKE_FLAGS=„all"
> 
> 
> Problems:
> 
> 1. I installed METIS and ParMETIS and DUNE recognizes them when building a module via ./dunecontrol , but while running the program on multiple processes I still receive the following output:
> 
> TIME STEP [Alexander (order 2)]      1 time (from):   0.0000e+00 dt:   1.0000e-01 time (to):   1.0000e-01
> STAGE 1 time (to):   2.9289e-02.
>   Initial defect:   9.5196e-06
> Successive accumulation of data on coarse levels only works with ParMETIS installed.  Fell back to accumulation to one domain on coarsest level
>   Newton iteration  1.  New defect:   7.2254e-06.  Reduction (this):   7.5900e-01.  Reduction (total):   7.5900e-01
> Successive accumulation of data on coarse levels only works with ParMETIS installed.  Fell back to accumulation to one domain on coarsest level
>   Newton iteration  2.  New defect:   4.1631e-07.  Reduction (this):   5.7618e-02.  Reduction (total):   4.3732e-02
> 
> I am using the following linear solver backend (currently suppressing SuperLU): 
> 
>   typedef Dune::PDELab::ISTLBackend_BCGS_AMG_SSOR<IGO> LS;
>   LS ls(gfs,5000,0,false,false);
> 
> followed by the Newton Solver
> 
>   typedef Dune::PDELab::Newton<IGO,LS,U> PDESOLVER;
>   PDESOLVER pdesolver(igo,ls);
>   pdesolver.setReassembleThreshold(0.0);
>   pdesolver.setVerbosityLevel(2);
>   pdesolver.setReduction(1e-10);
>   pdesolver.setMinLinearReduction(1e-4);
>   pdesolver.setMaxIterations(6);
>   pdesolver.setLineSearchMaxIterations(10);
> 
> and a OneStepMethod
> 
>   Dune::PDELab::Alexander2Parameter<Real> alex2;
>   Dune::PDELab::OneStepMethod<Real,IGO,PDESOLVER,U,U> osm(alex2,igo,pdesolver);
> 
> Do I have to tell DUNE explicitly that it shall use ParMETIS?

If you want your program to use ParMETIS, you have to add the PARMETIS_{CPPFLAGS,LDFLAGS,LIBS} to the respective
automake variables for your program (take a look at dune-pdelab-howto/src/two-phase/Makefile.am for an example). The
PARMETIS_CPPFLAGS set a special macro ENABLE_PARMETIS to 1. Without that macro set, dune-istl will not activate
ParMETIS support regardless of whether ParMETIS was found or not.

This requirement of having to add $(PKGNAME)_CPPFLAGS is pretty common in the DUNE build system BTW, you also need
it for stuff like SuperLU, GMP etc.

> 
> 
> 2. I do not know where to install SuperLU.
> 
> I installed it to Users/lriedel/opt/SuperLU_4.3/ as this is the directory mentioned as „prefix“ in the opts-file, but DUNE can obviously not find it (see output-file)
> DUNE seems to look for it inside /usr/local/SRC but it is only possible to install the newest version of SuperLU inside a folder structure as follows: ../SuperLU_4.3/SRC/
> Is there a way to specify where I installed SuperLU so that DUNE can find its libraries?

Well, SuperLU’s build system is very limited and doesn’t really support installing stuff anywhere. It can be configured to a certain
extent by editing the file make.inc in the root of the SuperLU source tree. I don’t know how other people handle it, but I (working on
OS X like you) usually set the variable SuperLUroot in that file to something like $(HOME)/local/SuperLU-4.3. That will cause the library
to be built in $(HOME)/local/SuperLU-4.3/lib. But as mentioned, the build system isn’t very clever, so you have to manually create that
directory before calling make with "mkdir -p $(HOME)/local/SuperLU-4.3/lib". In order to keep stuff together, I then create a directory
$(HOME)/local/SuperLU-4.3/include and copy all header files from the directory SRC in the SuperLU source tree to that directory.
Afterwards, you should run "chmod +r *“ in the include/ directory because one of the headers has weird permissions. Finally I add a
symlink SRC -> include to $(HOME)/local/SuperLU-4.3.

After installing SuperLU that way, you still have to tell DUNE where to find SuperLU by adding
"—with-superlu=$(HOME)/local/SuperLU-4.3 --with-superlu-lib=libsuperlu_4.3.a“ to the DUNE CONFIGURE_FLAGS.


Hope that helps…

Steffen


> 
> 
> Thank you and best regards
> Lukas Riedel
> 
> 
> <dunecontrol-output.txt>
> _______________________________________________
> Dune mailing list
> Dune at dune-project.org
> http://lists.dune-project.org/mailman/listinfo/dune

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://lists.dune-project.org/pipermail/dune-pdelab/attachments/20140625/4ca090d2/attachment.sig>