[Dune-devel] [SOCIS 2014]: C++11 threads vs POSIX threads

Markus Blatt markus at dr-blatt.de
Mon Jun 16 11:13:15 CEST 2014 

Hi Steffen,

On Sat, Jun 14, 2014 at 07:05:42PM +0200, Steffen Müthing wrote:
> I know I’m a little late to the party, but I wanted to share the discussions we had
> in the EXADune project (http://conan.iwr.uni-heidelberg.de/research/sppexa/)
> regarding threading and what we went for in the end.

Late, indeed. Anyway, this is the first time that there is any kind of
public input from this project (which seems to be rather confidential
for whatever reason). 

> As part of that project, we are working on transitioning from the MPI-only model
> that we have right now to a hybrid approach with one MPI rank per UMA node
> (e.g. per CPU socket) and multithreading within the UMA node. Another aim is
> to improve the support for SIMD.
> 
> Right now, we have a working implementation of multi-threaded vectors, matrices
> and simple preconditiioners as well as a thread-parallel version of the PDELab
> assembler.
> 

This is kind of perpendicular to our project goal, which does not aim
to change the underlying programming paradigm. Actually, we just want
to abuse threads for message passing based on the parallel index sets.
This is of course much less ambituous than your project but will most
likely be achievable in the short project time. One goal is to let
people use DUNE (or at least the linear solvers/AMG) in parallel
without having MPI available. As the changes are not very intrusive
(no solver code will be changed) we hope to have this in master very
soon. 

> At the beginning of the project, we considered a number of approaches to multi-threading:
> 
> - OpenMP. Nobody really liked that idea because it has poor support for C++(11), there is
>   way too much magic hidden in the runtime and apart from that, everybody thought that
>   pragmas are ugly ;-)

All true. But it is an open standard and that is a _big_ advantage.
 
> - Plain threads using either POSIX or C++11. As was already pointed out in the thread, this
>   approach has the advantage of portability. It also gives us maximum control over what is
>   happening. On the other hand, it requires us to write all the infrastructure ourselves, stuff
>   like a thread pool, a scheduler (for more complex work-sharing constructs) etc.
> 

Does it? I would reckon that there are already small libraries out there for
this. Anyway, as our goal is different we probably will not need any
of these functionalities.

> - The Intel Threading Building Blocks library (http://www.threadingbuildingblocks.org). TBB
>   provides a higher-level abstraction on top of threads, based around the idea of tasks that
>   are executed by the TBB scheduler. It is a native C++ library that provides several levels of
>   abstraction, from coding tasks manually and building a dependency graph of tasks that are
>   then executed with as much parallelism as possible to high-level parallel_for, parallel_reduce
>   et al. At the highest abstraction level, you just pass in the code to execute (as a functor, a
>   function pointer or a lambda function) and an iteration range that can be split into subranges
>   which are then executed in parallel. The library is licensed identically to the DUNE core
>   modules and supports x86, x64 and POWER (maybe more, I’m not sure). It also contains
>   allocators that provide better scalability than glibc in multi-threaded settings.
>   We liked about it that
>   - It’s C++, no mucking around with pragmas
>   - Support for C++11, but also fallback to POSIX
>   - contains stuff like thread pools, schedulers, task graphs etc.
>   - convenient high-level interface
>   - still allows for lower-level control like thread-pinning
>   - well thought-out concepts for stuff like parallelizing task launches and cache-oblivious algorithms
>   But of course, there are also disadvantages
>   - It’s another dependency
>   - It’s produced by a company (Intel)
>   - Proprietary API that newcomers will have to learn first
>   - Not as much control as with plain threads
> 

It is a pity, that there was no discussion outside of the inner circle
of your project on this. Given the fact that you are aiming to port
this back this might have been beneficial. (I miss the old DUNE days
of open development. When did it vanish?).

IMHO the biggest problem is not that it is developed by a company but
that it is developed by a hardware company. It is naive to think that
there is any other agenda then increasing the market share of their
hardware/compilers. Looking at their hardware support page it is
evident that this is  the case. Open Source is just like free
or reduced academic licences for proprietory code a vehicle to drive
this. You might have closed some future doors here. Just imagine AMD
(where most of DUNE was run on in the early days) would have convinced
us to use a propietory API....

> 
> In the end, we settled on TBB because it just offered too many components that we would have
> had to reinvent otherwise (and that are really not trivial to get right).
> 
> I don’t know who complicated the parallel algorithms in this project are going to be, but I’d suggest
> you take a look at TBB as well. The other issue of course is that we will want to port our developments
> in EXADune back to the normal repositories in the future, and having two different APIs would be
> rather inconvenient (not least because you then need to make sure that things like the thread pools
> play along nicely).
> 

Like I said our concept is kind of perpendicular. Therefore the risk
might be smaller than you think. 

Even if it was not, there is no way for people not directly involved
in the project to keep track of it. Therefore it is simply impossible
to consider the implications. Postulating this at this late stage
(this project has been proposed since 2013 whithout any kind comments
or commitments) seems a bit unfair. Our projects have been open for
discussions from the start.

Anyway, maybe rethinking our communication policy would be a good
start. Of course this takes time, but the core modules are only as
good as they currently are because we have taken this effort during the
last years. They are in very stable state and we all can be proud to
have achieved this. This should not be jeopardized by aiming at fast
scientific/academic breakthrough at the expense of
sustainability. Some of these days all this code has to be maintained by
somebody and there are only very few among us who currently do this
regularly (mostly for charity). Nevertheless this is a very (if not
the most) important part in a software cycle. Of course there is no
funding from agencies for this and because of this it should be
considered during the development. Otherwise DUNE will become just
another academic software project, which is not what we set out for more
than 10 years ago.

I am a bit (actually a great deal) worried that the code quality is
currently changing. (Up to now this was only evident at some non-core
modules that should be very important to users). I am sure that you guys
are doing a great job. But you are giving away the opportunity of
outside input into your work at an early stage rather lightheaded for
whatever reason. I am not even sure that the current state of the
Exa-DUNE repositories is according to DUNE's licence. The last time
I checked they were not even readable by DUNE core developers. Even if
it is governed by the licence I would consider it fair to give people,
that you base your development on, some insight. The current approach
is promoting irritations and more among fellow developers. This does
not serve DUNE well. 

Let me quote a saying A. Stepanov (from my porous memory):
We have to discuss code if we are to develop into a
profession. ... The case for these guys (big search engine company)
is helpless. They (at least) think they are the experts. If you are in
such a state of mind, you cannot progress any more.

God forbid that we ever reach that kind of state. But on some days I
am under the impression that we are unfortunately closer than before.

> On the other hand, I do of course realize that adding a new dependency to dune-common is a decision
> that shouldn’t be taken light-heartedly. That said, both Peter and me would be in favor of using TBB as
> our common threading API.
> 

I am not favor as long as Marco does not insist on it. In my opinion the
learning curve for this project is too high and I am confident that we
can get TBB to get along with standard threads if someone ever ports
back Exa-DUNE to a stable state. Anybody who does not agree with this
can give convincing arguments and should be prepared to invest his
precious time in tutoring at least parts of this project. That is more
than welcome from my side.

Regards,

Markus

> Best
> Steffen
> 
> 
> Am 10.06.2014 um 18:20 schrieb Agnese, Marco <m.agnese13 at imperial.ac.uk>:
> 
> > Hi Christoph,
> > you are right.
> > 
> > I go through the source and the documentation of TinyThread++1.1. We can use C++11 when is supported and include the TinyThread library (which is only 3 files) to have a backwards compatibility. 
> > 
> > TinyThread++1.1 does NOT support all the capabilities of C++11 but only a subset. We can try to use only the subset of features.
> > Cheers
> > Marco 
> > ________________________________________
> > From: 7135b8d6-3e78-dd11-9c85-005056a46000 at iws.uni-stuttgart.de [7135b8d6-3e78-dd11-9c85-005056a46000 at iws.uni-stuttgart.de] on behalf of Christoph Grüninger [christoph.grueninger at iws.uni-stuttgart.de]
> > Sent: Sunday, June 08, 2014 4:17 PM
> > To: Agnese, Marco; dune-devel at dune-project.org
> > Subject: Re: [Dune-devel] [SOCIS 2014]: C++11 threads vs POSIX threads
> > 
> > Hi Marco,
> > 
> >> I prefer the C++11 way like you. Moreover I see threads just
> >> an additional feature since they do not add any functionality
> >> apart a performance boost.
> > 
> > But the performance boost is most important on supercomputer
> > which have good multi-threading possibilities. And on these
> > computers the backwards compatibility is must crucial. Thus,
> > your argument is kind of weak.
> > 
> >> Obviously libraries could help but I always prefer the STL
> >> to external ones since it is standardized.
> >> Since every C++ developer is familiar with it, it is more
> >> simple for everyone to contribute to the code without
> >> spending time learning new libraries.
> > 
> > Please have a closer look at the libraries I named. They seem
> > to implement the C++11 threading for compilers that do not
> > natively offer these capabilities.
> > 
> > Bye
> > Christoph
> > 
> 



> _______________________________________________
> Dune-devel mailing list
> Dune-devel at dune-project.org
> http://lists.dune-project.org/mailman/listinfo/dune-devel


-- 
Do you need more support with DUNE or HPC in general? 

Dr. Markus Blatt - HPC-Simulation-Software & Services http://www.dr-blatt.de
Hans-Bunte-Str. 8-10, 69123 Heidelberg, Germany
Tel.: +49 (0) 160 97590858
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <https://lists.dune-project.org/pipermail/dune-devel/attachments/20140616/2ca55fb8/attachment.sig>

More information about the Dune-devel mailing list