[Dune-devel] [SOCIS 2014]: C++11 threads vs POSIX threads

[Steffen Müthing]

Hi everyone,

I know I’m a little late to the party, but I wanted to share the discussions we had
in the EXADune project (http://conan.iwr.uni-heidelberg.de/research/sppexa/)
regarding threading and what we went for in the end.

As part of that project, we are working on transitioning from the MPI-only model
that we have right now to a hybrid approach with one MPI rank per UMA node
(e.g. per CPU socket) and multithreading within the UMA node. Another aim is
to improve the support for SIMD.

Right now, we have a working implementation of multi-threaded vectors, matrices
and simple preconditiioners as well as a thread-parallel version of the PDELab
assembler.

At the beginning of the project, we considered a number of approaches to multi-threading:

- OpenMP. Nobody really liked that idea because it has poor support for C++(11), there is
  way too much magic hidden in the runtime and apart from that, everybody thought that
  pragmas are ugly ;-)

- Plain threads using either POSIX or C++11. As was already pointed out in the thread, this
  approach has the advantage of portability. It also gives us maximum control over what is
  happening. On the other hand, it requires us to write all the infrastructure ourselves, stuff
  like a thread pool, a scheduler (for more complex work-sharing constructs) etc.

- The Intel Threading Building Blocks library (http://www.threadingbuildingblocks.org). TBB
  provides a higher-level abstraction on top of threads, based around the idea of tasks that
  are executed by the TBB scheduler. It is a native C++ library that provides several levels of
  abstraction, from coding tasks manually and building a dependency graph of tasks that are
  then executed with as much parallelism as possible to high-level parallel_for, parallel_reduce
  et al. At the highest abstraction level, you just pass in the code to execute (as a functor, a
  function pointer or a lambda function) and an iteration range that can be split into subranges
  which are then executed in parallel. The library is licensed identically to the DUNE core
  modules and supports x86, x64 and POWER (maybe more, I’m not sure). It also contains
  allocators that provide better scalability than glibc in multi-threaded settings.
  We liked about it that
  - It’s C++, no mucking around with pragmas
  - Support for C++11, but also fallback to POSIX
  - contains stuff like thread pools, schedulers, task graphs etc.
  - convenient high-level interface
  - still allows for lower-level control like thread-pinning
  - well thought-out concepts for stuff like parallelizing task launches and cache-oblivious algorithms
  But of course, there are also disadvantages
  - It’s another dependency
  - It’s produced by a company (Intel)
  - Proprietary API that newcomers will have to learn first
  - Not as much control as with plain threads


In the end, we settled on TBB because it just offered too many components that we would have
had to reinvent otherwise (and that are really not trivial to get right).

I don’t know who complicated the parallel algorithms in this project are going to be, but I’d suggest
you take a look at TBB as well. The other issue of course is that we will want to port our developments
in EXADune back to the normal repositories in the future, and having two different APIs would be
rather inconvenient (not least because you then need to make sure that things like the thread pools
play along nicely).

On the other hand, I do of course realize that adding a new dependency to dune-common is a decision
that shouldn’t be taken light-heartedly. That said, both Peter and me would be in favor of using TBB as
our common threading API.

Best
Steffen


Am 10.06.2014 um 18:20 schrieb Agnese, Marco <m.agnese13 at imperial.ac.uk>:

> Hi Christoph,
> you are right.
> 
> I go through the source and the documentation of TinyThread++1.1. We can use C++11 when is supported and include the TinyThread library (which is only 3 files) to have a backwards compatibility. 
> 
> TinyThread++1.1 does NOT support all the capabilities of C++11 but only a subset. We can try to use only the subset of features.
> Cheers
> Marco 
> ________________________________________
> From: 7135b8d6-3e78-dd11-9c85-005056a46000 at iws.uni-stuttgart.de [7135b8d6-3e78-dd11-9c85-005056a46000 at iws.uni-stuttgart.de] on behalf of Christoph Grüninger [christoph.grueninger at iws.uni-stuttgart.de]
> Sent: Sunday, June 08, 2014 4:17 PM
> To: Agnese, Marco; dune-devel at dune-project.org
> Subject: Re: [Dune-devel] [SOCIS 2014]: C++11 threads vs POSIX threads
> 
> Hi Marco,
> 
>> I prefer the C++11 way like you. Moreover I see threads just
>> an additional feature since they do not add any functionality
>> apart a performance boost.
> 
> But the performance boost is most important on supercomputer
> which have good multi-threading possibilities. And on these
> computers the backwards compatibility is must crucial. Thus,
> your argument is kind of weak.
> 
>> Obviously libraries could help but I always prefer the STL
>> to external ones since it is standardized.
>> Since every C++ developer is familiar with it, it is more
>> simple for everyone to contribute to the code without
>> spending time learning new libraries.
> 
> Please have a closer look at the libraries I named. They seem
> to implement the C++11 threading for compilers that do not
> natively offer these capabilities.
> 
> Bye
> Christoph
> 
> --
> Das Kommt darauf an, in welchen Zeiträumen du
> rechnest. Für Geologen: ja, in naher Zukunft
> [Stefan Seyfried in suse-laptop]
> _______________________________________________
> Dune-devel mailing list
> Dune-devel at dune-project.org
> http://lists.dune-project.org/mailman/listinfo/dune-devel
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://lists.dune-project.org/pipermail/dune-devel/attachments/20140614/67e025e2/attachment.sig>