[Dune] Dynamic load balancing with ALUGrid
Dedner, Andreas
A.S.Dedner at warwick.ac.uk
Fri Feb 22 12:43:39 CET 2013
I assumed so far that the global ids are available during gather/scatter but probably
they are not since they are constructed through the dune bindings after the load balance
is complete. So in gather/scatter only local ids seem to be available. As Martin said,
Bernd seems to be the first who tried to use global ids together with the callback load balancing.
I guess this has to be chalked down as a 'known bug' in alu at the moment....
Andreas
________________________________________
From: Christian Engwer [christian.engwer at uni-muenster.de]
Sent: 22 February 2013 11:33
To: Dedner, Andreas; bernd at iws.uni-stuttgart.de; dune at dune-project.org
Subject: RE: [Dune] Dynamic load balancing with ALUGrid
"Dedner, Andreas" <A.S.Dedner at warwick.ac.uk> schrieb:
>I could guess most of that but I am struggeling with `gid Alsop'...
UPS! Android is not the best platform for writing emails... 'globalId also'
Christian
>________________________________________
>From: Christian Engwer [christian.engwer at uni-muenster.de]
>Sent: 22 February 2013 11:18
>To: Dedner, Andreas; Dedner, Andreas; bernd at iws.uni-stuttgart.de;
>dune at dune-project.org
>Subject: Re: [Dune] Dynamic load balancing with ALUGrid
>
>"Dedner, Andreas" <A.S.Dedner at warwick.ac.uk> schrieb:
>
>>Hi.
>>That is one reason why the loadbalance method takes a callback object
>>with gather and scatter like communicate. The grid then makes sure
>that
>>these methods are called on the right entities.
>
>So, doesn't this allow top update the gid while looping through the
>scatter and providing the correct gid Alsop during load-balance?
>
>In case this is not possible I'd suggest to at least throw an exception
>when along for the gid.
>
>Christian
>
>> So while gathering the
>>data on one processor its local idset can be used and the same holds
>on
>>the receiving process. In the grid-howto there is an example showing a
>>possible way to use this with the persistent containers.
>>Andreas
>>
>>Bernd Flemisch <bernd at iws.uni-stuttgart.de> wrote:
>>Hi Martin,
>>
>>thank you for your answer. This indeed sounds a lot earlier sier and
>>like the
>>solution I originally wanted to achieve. Nevertheless, I am a bit
>>puzzled now. I deliberately preferred global over local Ids, since
>>"A local id set provides ids that are unique within one process but
>two
>>entities in different processes may have the same id," see
>>http://www.dune-project.org/doc/doxygen/dune-grid-html/class_dune_1_1_id_set.html#_details
>>
>>So I thought that with local Ids it could happen that an entity is
>>moved
>>from one process to another, where another entity with the same Id
>>already exists. And then this would not work out. Or am I missing
>>something?
>>
>>Kind regards
>>Bernd
>>
>>On 02/22/2013 09:13 AM, Martin Nolte wrote:
>>> Hi Bernd,
>>>
>>> I guess you are the first to use global ids during load balancing
>and
>>> it seems you hit on a bug in ALUGrid (please file a complete bug
>>> report on this).
>>>
>>> Actually, the solution we use is far simpler. Store your data in a
>>> std::map using the LocalIdSet (or use a PersistentContainer). This
>>> way, your data will always be accessible, even during load
>balancing.
>>> Then you write a data handle (warning: you communicate on the entire
>>> hierarchy). That's it.
>>>
>>> Let me give a bit of explaination: Behind the scenes, ALUGrid will
>>> pack data for a macro element to be sent to another process. To this
>>> end, it will traverse the tree and pack the element information and
>>> your data (calling gather). When all macro elements have been
>packed,
>>> the data is sent to the new processes. Here, they are unpacked,
>i.e.,
>>> the element is constructed and all scatter is called. During
>scatter,
>>> the ids will already be valid, so you can use them to access your
>>data
>>> container (though the global id might be buggy, as you stated).
>>>
>>> One final opinion: GlobalIds are an artificial DUNE construct.
>>> Implementing them might be quite complicated and expensive. This is
>>> the case for ALUGrid, where they are generated in the DUNE bindings.
>>> Hence, they have to communicated during load balancing (expensive).
>>> So, if you ask ALUGrid for the global id set once, you got to pay
>>this
>>> price. There is no way of telling how long this id set is to be used
>>> (because it is returned by reference).
>>>
>>> Best,
>>>
>>> Martin
>>>
>>>
>>>
>>>
>>>
>>> On 02/22/2013 07:35 AM, Bernd Flemisch wrote:
>>>> Hi,
>>>>
>>>> sorry for the noise again, but I found a working solution. So if
>you
>>>> already
>>>> started to brood over the problem, you may relax.
>>>>
>>>> I assume that the Ids are still valid during gather and will become
>>>> valid
>>>> again and with the same values after load balance. I simply gather
>>>> the Ids
>>>> together with the data and scatter both of them. In a postprocess
>>>> after load
>>>> balance, I use this (now valid) information to write the values
>into
>>the
>>>> corresponding vector.
>>>>
>>>> This works for several grids and numbers of processes, so I hope
>for
>>>> all of
>>>> them. I attach the file.
>>>>
>>>> If you think this is important enough to become a test in
>>>> dune/grid/test and
>>>> feel responsible for evaluating and committing it, I am happy to
>>>> provide a
>>>> patch and to improve it together with you.
>>>>
>>>> Thanks to Andreas and Martin for the hints which helped to achieve
>>>> this. Kind
>>>> regards
>>>> Bernd
>>>>
>>>> On Thu, 21 Feb 2013 19:12:44 +0100
>>>> Bernd Flemisch <bernd at iws.uni-stuttgart.de> wrote:
>>>>> Actually, the Ids are the same again after the load balancing step
>>>>> is over.
>>>>> But apparently not during scatter, where I would need it. Hmmm...
>>>>>
>>>>> On Thu, 21 Feb 2013 17:55:51 +0100
>>>>> Bernd Flemisch <bernd at iws.uni-stuttgart.de> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> only a few months have passed and I found the time to look at
>this
>>>>>> again. I
>>>>>> tried to follow Andreas' suggestion and work with Ids rather than
>>>>>> indices,
>>>>>> since they should be persistent. I was hoping that "persistent"
>>>>>> means that,
>>>>>> if an element is moved from one process to another, it keeps its
>>>>>> global Id.
>>>>>> Looking at
>>>>>>
>>http://www.dune-project.org/doc/doxygen/dune-grid-html/class_dune_1_1_id_set.html#_details
>>>>>>
>>>>>>
>>>>>> does not necessarily say that (I think).
>>>>>>
>>>>>> I set up a test for my efforts, see the attached file. The coarse
>>>>>> grid is
>>>>>> 5x1x1. The two left elements are interior on process 1, the three
>>>>>> right
>>>>>> ones interior on process 2. The two left elements get refined.
>>After
>>>>>> adaption, loadBalance with my DataHandle is called. The three
>>right
>>>>>> elements are all moved to process 1, so that after the balancing
>>step,
>>>>>> process 1 holds all elements, while process 0 has none.
>>>>>>
>>>>>> During load balance, I print the cell centers, global Ids, and
>>>>>> gathered/scattered values on each process. This gives the
>>following:
>>>>>> Process 0:
>>>>>> 0: 0.5 0.5 0.5, ([2,3,8,14],8,0): gather 0
>>>>>> 0: 0.7 0.5 0.5, ([3,4,9,15],8,0): gather 1
>>>>>> 0: 0.9 0.5 0.5, ([4,5,10,16],8,0): gather 2
>>>>>> Process 1:
>>>>>> 1: 0.5 0.5 0.5, ([2,3,8,14],71,0): scatter 0
>>>>>> 1: 0.7 0.5 0.5, ([2,3,8,14],70,0): scatter 1
>>>>>> 1: 0.9 0.5 0.5, ([2,3,8,14],69,0): scatter 2
>>>>>> It seems to be correct that information from three elements is
>>sent
>>>>>> from
>>>>>> process 0 and received for the corresponding elements by process
>>1.
>>>>>> However, the global Ids are changed. Is this correct? Then how tf
>>>>>> can I
>>>>>> find out which elements correspond to each other? Or am I still
>>doing
>>>>>> something wrong?
>>>>>>
>>>>>> Thank you for having another look. Kind regards
>>>>>> Bernd
>>>>>>
>>>>>> On Sat, 27 Oct 2012 07:43:48 +0000
>>>>>> "Dedner, Andreas" <A.S.Dedner at warwick.ac.uk> wrote:
>>>>>>> Is it not even so that one can not use an undexset at all? Load
>>>>>>> balancing
>>>>>>> changes the grid so the indexsets are not valid. You need to use
>>id
>>>>>>> containers.
>>>>>>> Andreas
>>>>>>>
>>>>>>> Martin Nolte <nolte at mathematik.uni-freiburg.de> wrote:
>>>>>>> Hi Bernd,
>>>>>>>
>>>>>>> I have an educated guess for you (don't blame me, if I'm wrong):
>>>>>>>
>>>>>>> In normal communication, you only communicate data on a certain
>>>>>>> grid view.
>>>>>>> Hence
>>>>>>> you can use the index set to address data stored on that grid
>>view.
>>>>>>>
>>>>>>> When doing load balancing, you rather communicate data on the
>>entire
>>>>>>> hierarchy.
>>>>>>> Therefore, your entity might not be contained an index set
>>>>>>> (leading to this
>>>>>>> cryptic assertion). So, if you want to communicate data only on
>a
>>>>>>> certain
>>>>>>> grid
>>>>>>> view, you have to add a call to indexSet.contains to the gather
>/
>>>>>>> scatter
>>>>>>> / size
>>>>>>> methods. Usually, fixedSize will be false in such situations (I
>>>>>>> think ALUGrid
>>>>>>> ignores this anyway, so it should not matter, here).
>>>>>>>
>>>>>>> Just a side note: In my opionion, a normal communicate method
>>>>>>> exchanging
>>>>>>> data on
>>>>>>> the entire hierarchy would be nice, too. But alas, this method
>>would
>>>>>>> naturally
>>>>>>> be "grid().communicate( dataHandle, interface, direction )".
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Martin
>>>>>>>
>>>>>>> On 10/26/2012 01:32 PM, Bernd Flemisch wrote:
>>>>>>>> Hey Andreas,
>>>>>>>>
>>>>>>>> thank you for your fast answer. This indeed solves my second
>>issue.
>>>>>>>> However, the
>>>>>>>> first one remains. I attach an updated test file.
>>>>>>>>
>>>>>>>> Kind regards
>>>>>>>> Bernd
>>>>>>>>
>>>>>>>> On 10/26/2012 01:08 PM, Dedner, Andreas wrote:
>>>>>>>>> Hi Bernd
>>>>>>>>>
>>>>>>>>> You should be able to cast the derived class onto the
>interface
>>>>>>>>> class
>>>>>>>>> and that triggers the call to the correct method.
>>>>>>>>>
>>>>>>>>> Andreas
>>>>>>>>>
>>>>>>>>> ________________________________________
>>>>>>>>> From: dune-bounces+a.s.dedner=warwick.ac.uk at dune-project.org
>>>>>>>>> [dune-bounces+a.s.dedner=warwick.ac.uk at dune-project.org] on
>>>>>>>>> behalf of Bernd
>>>>>>>>> Flemisch [bernd at iws.uni-stuttgart.de]
>>>>>>>>> Sent: 26 October 2012 11:53
>>>>>>>>> To: Dune
>>>>>>>>> Subject: [Dune] Dynamic load balancing with ALUGrid
>>>>>>>>>
>>>>>>>>> Dear Dune,
>>>>>>>>>
>>>>>>>>> I try to use dynamic load balancing with ALUGrid. The grid
>>>>>>>>> itself gets
>>>>>>>>> balanced nicely, but I don't succeed in getting data balanced.
>>>>>>>>>
>>>>>>>>> From dune/grid/alugrid/3d/grid.hh lines 701ff, I guess that I
>>>>>>>>> can pack
>>>>>>>>> a "usual" data handle which is used for communication into a
>>>>>>>>> ALUGridLoadBalanceDataHandle and call loadBalance with that
>>>>>>>>> handle. My
>>>>>>>>> initial attempt to do this fails with
>>>>>>>>> test-dynamiclb:
>>>>>>>>> ../../../dune/grid/alugrid/common/defaultindexsets.hh:218:
>>>>>>>>> Dune::DefaultIndexSet<GridImp, IteratorImp>::IndexType
>>>>>>>>> Dune::DefaultIndexSet<GridImp, IteratorImp>::index(const
>>typename
>>>>>>>>> GridImp::Codim<cd>::Entity&) const [with int cd = 0; GridImp =
>>>>>>>>> Dune::ALU3dGrid<(Dune::ALU3dGridElementType)7u>; IteratorImp =
>>>>>>>>> Dune::EntityIterator<0, const
>>>>>>>>> Dune::ALU3dGrid<(Dune::ALU3dGridElementType)7u>,
>>>>>>>>> Dune::ALU3dGridLeafIterator<0,
>(Dune::PartitionIteratorType)4u,
>>>>>>>>> const
>>>>>>>>> Dune::ALU3dGrid<(Dune::ALU3dGridElementType)7u> > >;
>>>>>>>>> Dune::DefaultIndexSet<GridImp, IteratorImp>::IndexType =
>>>>>>>>> unsigned int;
>>>>>>>>> typename GridImp::Codim<cd>::Entity = Dune::Entity<0, 3, const
>>>>>>>>> Dune::ALU3dGrid<(Dune::ALU3dGridElementType)7u>,
>>>>>>>>> Dune::ALU3dGridEntity>]: Assertion `indexContainer( cd )[ en
>>>>>>>>> ].index()
>>>>>>>>> >= 0' failed.
>>>>>>>>>
>>>>>>>>> I attach my effort, where I use a data handle copied from the
>>>>>>>>> grid-howto. It would be great if you could take a look.
>>>>>>>>>
>>>>>>>>> I have a second issue: I guess that I should be able to call
>>>>>>>>> loadBalance
>>>>>>>>> directly with a usual data handle, so that grid.hh lines 701ff
>>>>>>>>> takes
>>>>>>>>> care of it and packs it for me into a
>>ALUGridLoadBalanceDataHandle
>>>>>>>>> before sending it to the real loadBalance method. However,
>>since
>>>>>>>>> my data
>>>>>>>>> handle is only inherited from CommDataHandleIF, the packing
>>>>>>>>> routine is
>>>>>>>>> not triggered but the real loadBalance method is called
>>directly
>>>>>>>>> leading
>>>>>>>>> to a compilation error. How can I trigger the execution of the
>>>>>>>>> packing
>>>>>>>>> routine for such a derived data handle?
>>>>>>>>>
>>>>>>>>> Thank you for having a look. Kind regards
>>>>>>>>> Bernd
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>_____________________________________________________________________
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Bernd Flemisch phone: +49 711 685 69162
>>>>>>>>> IWS, Universität Stuttgart fax: +49 711 685 60430
>>>>>>>>> Pfaffenwaldring 61 email: bernd at iws.uni-stuttgart.de
>>>>>>>>> D-70569 Stuttgart url:
>>>>>>>>>
>>www.hydrosys.uni-stuttgart.de<http://www.hydrosys.uni-stuttgart.de>
>>>>>>>>>
>>_____________________________________________________________________
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Dune mailing list
>>>>>>>> Dune at dune-project.org
>>>>>>>> http://lists.dune-project.org/mailman/listinfo/dune
>>>>>>>
>>>>>>> --
>>>>>>> Dr. Martin Nolte <nolte at mathematik.uni-freiburg.de>
>>>>>>>
>>>>>>> Universität Freiburg phone:
>>>>>>> +49-761-203-5630
>>>>>>> Abteilung für angewandte Mathematik fax:
>>>>>>> +49-761-203-5632
>>>>>>> Hermann-Herder-Straße 10
>>>>>>> 79104 Freiburg, Germany
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Dune mailing list
>>>>>>> Dune at dune-project.org
>>>>>>> http://lists.dune-project.org/mailman/listinfo/dune
>>>>>>>
>>>>>>
>>>>>> ___________________________________________________________
>>>>>>
>>>>>> Bernd Flemisch phone: +49 711 685 69162
>>>>>> IWS, Universitaet Stuttgart fax: +49 711 685 67020
>>>>>> Pfaffenwaldring 61 email: bernd at iws.uni-stuttgart.de
>>>>>> D-70569 Stuttgart url:
>>www.hydrosys.uni-stuttgart.de<http://www.hydrosys.uni-stuttgart.de>
>>>>>> ___________________________________________________________
>>>>>
>>>>> ___________________________________________________________
>>>>>
>>>>> Bernd Flemisch phone: +49 711 685 69162
>>>>> IWS, Universitaet Stuttgart fax: +49 711 685 67020
>>>>> Pfaffenwaldring 61 email: bernd at iws.uni-stuttgart.de
>>>>> D-70569 Stuttgart url:
>>www.hydrosys.uni-stuttgart.de<http://www.hydrosys.uni-stuttgart.de>
>>>>> ___________________________________________________________
>>>>>
>>>>> _______________________________________________
>>>>> Dune mailing list
>>>>> Dune at dune-project.org
>>>>> http://lists.dune-project.org/mailman/listinfo/dune
>>>>
>>>> ___________________________________________________________
>>>>
>>>> Bernd Flemisch phone: +49 711 685 69162
>>>> IWS, Universitaet Stuttgart fax: +49 711 685 67020
>>>> Pfaffenwaldring 61 email: bernd at iws.uni-stuttgart.de
>>>> D-70569 Stuttgart url:
>>www.hydrosys.uni-stuttgart.de<http://www.hydrosys.uni-stuttgart.de>
>>>> ___________________________________________________________
>>>>
>>>>
>>>> _______________________________________________
>>>> Dune mailing list
>>>> Dune at dune-project.org
>>>> http://lists.dune-project.org/mailman/listinfo/dune
>>>>
>>>
>>
>>
>>--
>>_____________________________________________________________________
>>
>>Bernd Flemisch phone: +49 711 685 69162
>>IWS, Universität Stuttgart fax: +49 711 685 60430
>>Pfaffenwaldring 61 email: bernd at iws.uni-stuttgart.de
>>D-70569 Stuttgart url:
>>www.hydrosys.uni-stuttgart.de<http://www.hydrosys.uni-stuttgart.de>
>>_____________________________________________________________________
>>
>>
>>_______________________________________________
>>Dune mailing list
>>Dune at dune-project.org
>>http://lists.dune-project.org/mailman/listinfo/dune
>>
>>
>>
>>------------------------------------------------------------------------
>>
>>_______________________________________________
>>Dune mailing list
>>Dune at dune-project.org
>>http://lists.dune-project.org/mailman/listinfo/dune
More information about the Dune
mailing list