[Dune] Dynamic load balancing with ALUGrid
Bernd Flemisch
bernd at iws.uni-stuttgart.de
Fri Feb 22 13:18:28 CET 2013
I filed a bug report FS#1258:
http://www.dune-project.org/flyspray/index.php?do=details&task_id=1258
Bernd
On 02/22/2013 12:43 PM, Dedner, Andreas wrote:
> I assumed so far that the global ids are available during gather/scatter but probably
> they are not since they are constructed through the dune bindings after the load balance
> is complete. So in gather/scatter only local ids seem to be available. As Martin said,
> Bernd seems to be the first who tried to use global ids together with the callback load balancing.
> I guess this has to be chalked down as a 'known bug' in alu at the moment....
>
> Andreas
> ________________________________________
> From: Christian Engwer [christian.engwer at uni-muenster.de]
> Sent: 22 February 2013 11:33
> To: Dedner, Andreas; bernd at iws.uni-stuttgart.de; dune at dune-project.org
> Subject: RE: [Dune] Dynamic load balancing with ALUGrid
>
> "Dedner, Andreas" <A.S.Dedner at warwick.ac.uk> schrieb:
>
>> I could guess most of that but I am struggeling with `gid Alsop'...
> UPS! Android is not the best platform for writing emails... 'globalId also'
>
> Christian
>
>> ________________________________________
>> From: Christian Engwer [christian.engwer at uni-muenster.de]
>> Sent: 22 February 2013 11:18
>> To: Dedner, Andreas; Dedner, Andreas; bernd at iws.uni-stuttgart.de;
>> dune at dune-project.org
>> Subject: Re: [Dune] Dynamic load balancing with ALUGrid
>>
>> "Dedner, Andreas" <A.S.Dedner at warwick.ac.uk> schrieb:
>>
>>> Hi.
>>> That is one reason why the loadbalance method takes a callback object
>>> with gather and scatter like communicate. The grid then makes sure
>> that
>>> these methods are called on the right entities.
>> So, doesn't this allow top update the gid while looping through the
>> scatter and providing the correct gid Alsop during load-balance?
>>
>> In case this is not possible I'd suggest to at least throw an exception
>> when along for the gid.
>>
>> Christian
>>
>>> So while gathering the
>>> data on one processor its local idset can be used and the same holds
>> on
>>> the receiving process. In the grid-howto there is an example showing a
>>> possible way to use this with the persistent containers.
>>> Andreas
>>>
>>> Bernd Flemisch <bernd at iws.uni-stuttgart.de> wrote:
>>> Hi Martin,
>>>
>>> thank you for your answer. This indeed sounds a lot earlier sier and
>>> like the
>>> solution I originally wanted to achieve. Nevertheless, I am a bit
>>> puzzled now. I deliberately preferred global over local Ids, since
>>> "A local id set provides ids that are unique within one process but
>> two
>>> entities in different processes may have the same id," see
>>> http://www.dune-project.org/doc/doxygen/dune-grid-html/class_dune_1_1_id_set.html#_details
>>>
>>> So I thought that with local Ids it could happen that an entity is
>>> moved
>> >from one process to another, where another entity with the same Id
>>> already exists. And then this would not work out. Or am I missing
>>> something?
>>>
>>> Kind regards
>>> Bernd
>>>
>>> On 02/22/2013 09:13 AM, Martin Nolte wrote:
>>>> Hi Bernd,
>>>>
>>>> I guess you are the first to use global ids during load balancing
>> and
>>>> it seems you hit on a bug in ALUGrid (please file a complete bug
>>>> report on this).
>>>>
>>>> Actually, the solution we use is far simpler. Store your data in a
>>>> std::map using the LocalIdSet (or use a PersistentContainer). This
>>>> way, your data will always be accessible, even during load
>> balancing.
>>>> Then you write a data handle (warning: you communicate on the entire
>>>> hierarchy). That's it.
>>>>
>>>> Let me give a bit of explaination: Behind the scenes, ALUGrid will
>>>> pack data for a macro element to be sent to another process. To this
>>>> end, it will traverse the tree and pack the element information and
>>>> your data (calling gather). When all macro elements have been
>> packed,
>>>> the data is sent to the new processes. Here, they are unpacked,
>> i.e.,
>>>> the element is constructed and all scatter is called. During
>> scatter,
>>>> the ids will already be valid, so you can use them to access your
>>> data
>>>> container (though the global id might be buggy, as you stated).
>>>>
>>>> One final opinion: GlobalIds are an artificial DUNE construct.
>>>> Implementing them might be quite complicated and expensive. This is
>>>> the case for ALUGrid, where they are generated in the DUNE bindings.
>>>> Hence, they have to communicated during load balancing (expensive).
>>>> So, if you ask ALUGrid for the global id set once, you got to pay
>>> this
>>>> price. There is no way of telling how long this id set is to be used
>>>> (because it is returned by reference).
>>>>
>>>> Best,
>>>>
>>>> Martin
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 02/22/2013 07:35 AM, Bernd Flemisch wrote:
>>>>> Hi,
>>>>>
>>>>> sorry for the noise again, but I found a working solution. So if
>> you
>>>>> already
>>>>> started to brood over the problem, you may relax.
>>>>>
>>>>> I assume that the Ids are still valid during gather and will become
>>>>> valid
>>>>> again and with the same values after load balance. I simply gather
>>>>> the Ids
>>>>> together with the data and scatter both of them. In a postprocess
>>>>> after load
>>>>> balance, I use this (now valid) information to write the values
>> into
>>> the
>>>>> corresponding vector.
>>>>>
>>>>> This works for several grids and numbers of processes, so I hope
>> for
>>>>> all of
>>>>> them. I attach the file.
>>>>>
>>>>> If you think this is important enough to become a test in
>>>>> dune/grid/test and
>>>>> feel responsible for evaluating and committing it, I am happy to
>>>>> provide a
>>>>> patch and to improve it together with you.
>>>>>
>>>>> Thanks to Andreas and Martin for the hints which helped to achieve
>>>>> this. Kind
>>>>> regards
>>>>> Bernd
>>>>>
>>>>> On Thu, 21 Feb 2013 19:12:44 +0100
>>>>> Bernd Flemisch <bernd at iws.uni-stuttgart.de> wrote:
>>>>>> Actually, the Ids are the same again after the load balancing step
>>>>>> is over.
>>>>>> But apparently not during scatter, where I would need it. Hmmm...
>>>>>>
>>>>>> On Thu, 21 Feb 2013 17:55:51 +0100
>>>>>> Bernd Flemisch <bernd at iws.uni-stuttgart.de> wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> only a few months have passed and I found the time to look at
>> this
>>>>>>> again. I
>>>>>>> tried to follow Andreas' suggestion and work with Ids rather than
>>>>>>> indices,
>>>>>>> since they should be persistent. I was hoping that "persistent"
>>>>>>> means that,
>>>>>>> if an element is moved from one process to another, it keeps its
>>>>>>> global Id.
>>>>>>> Looking at
>>>>>>>
>>> http://www.dune-project.org/doc/doxygen/dune-grid-html/class_dune_1_1_id_set.html#_details
>>>>>>>
>>>>>>> does not necessarily say that (I think).
>>>>>>>
>>>>>>> I set up a test for my efforts, see the attached file. The coarse
>>>>>>> grid is
>>>>>>> 5x1x1. The two left elements are interior on process 1, the three
>>>>>>> right
>>>>>>> ones interior on process 2. The two left elements get refined.
>>> After
>>>>>>> adaption, loadBalance with my DataHandle is called. The three
>>> right
>>>>>>> elements are all moved to process 1, so that after the balancing
>>> step,
>>>>>>> process 1 holds all elements, while process 0 has none.
>>>>>>>
>>>>>>> During load balance, I print the cell centers, global Ids, and
>>>>>>> gathered/scattered values on each process. This gives the
>>> following:
>>>>>>> Process 0:
>>>>>>> 0: 0.5 0.5 0.5, ([2,3,8,14],8,0): gather 0
>>>>>>> 0: 0.7 0.5 0.5, ([3,4,9,15],8,0): gather 1
>>>>>>> 0: 0.9 0.5 0.5, ([4,5,10,16],8,0): gather 2
>>>>>>> Process 1:
>>>>>>> 1: 0.5 0.5 0.5, ([2,3,8,14],71,0): scatter 0
>>>>>>> 1: 0.7 0.5 0.5, ([2,3,8,14],70,0): scatter 1
>>>>>>> 1: 0.9 0.5 0.5, ([2,3,8,14],69,0): scatter 2
>>>>>>> It seems to be correct that information from three elements is
>>> sent
>>>>>>> from
>>>>>>> process 0 and received for the corresponding elements by process
>>> 1.
>>>>>>> However, the global Ids are changed. Is this correct? Then how tf
>>>>>>> can I
>>>>>>> find out which elements correspond to each other? Or am I still
>>> doing
>>>>>>> something wrong?
>>>>>>>
>>>>>>> Thank you for having another look. Kind regards
>>>>>>> Bernd
>>>>>>>
>>>>>>> On Sat, 27 Oct 2012 07:43:48 +0000
>>>>>>> "Dedner, Andreas" <A.S.Dedner at warwick.ac.uk> wrote:
>>>>>>>> Is it not even so that one can not use an undexset at all? Load
>>>>>>>> balancing
>>>>>>>> changes the grid so the indexsets are not valid. You need to use
>>> id
>>>>>>>> containers.
>>>>>>>> Andreas
>>>>>>>>
>>>>>>>> Martin Nolte <nolte at mathematik.uni-freiburg.de> wrote:
>>>>>>>> Hi Bernd,
>>>>>>>>
>>>>>>>> I have an educated guess for you (don't blame me, if I'm wrong):
>>>>>>>>
>>>>>>>> In normal communication, you only communicate data on a certain
>>>>>>>> grid view.
>>>>>>>> Hence
>>>>>>>> you can use the index set to address data stored on that grid
>>> view.
>>>>>>>> When doing load balancing, you rather communicate data on the
>>> entire
>>>>>>>> hierarchy.
>>>>>>>> Therefore, your entity might not be contained an index set
>>>>>>>> (leading to this
>>>>>>>> cryptic assertion). So, if you want to communicate data only on
>> a
>>>>>>>> certain
>>>>>>>> grid
>>>>>>>> view, you have to add a call to indexSet.contains to the gather
>> /
>>>>>>>> scatter
>>>>>>>> / size
>>>>>>>> methods. Usually, fixedSize will be false in such situations (I
>>>>>>>> think ALUGrid
>>>>>>>> ignores this anyway, so it should not matter, here).
>>>>>>>>
>>>>>>>> Just a side note: In my opionion, a normal communicate method
>>>>>>>> exchanging
>>>>>>>> data on
>>>>>>>> the entire hierarchy would be nice, too. But alas, this method
>>> would
>>>>>>>> naturally
>>>>>>>> be "grid().communicate( dataHandle, interface, direction )".
>>>>>>>>
>>>>>>>> Best,
>>>>>>>>
>>>>>>>> Martin
>>>>>>>>
>>>>>>>> On 10/26/2012 01:32 PM, Bernd Flemisch wrote:
>>>>>>>>> Hey Andreas,
>>>>>>>>>
>>>>>>>>> thank you for your fast answer. This indeed solves my second
>>> issue.
>>>>>>>>> However, the
>>>>>>>>> first one remains. I attach an updated test file.
>>>>>>>>>
>>>>>>>>> Kind regards
>>>>>>>>> Bernd
>>>>>>>>>
>>>>>>>>> On 10/26/2012 01:08 PM, Dedner, Andreas wrote:
>>>>>>>>>> Hi Bernd
>>>>>>>>>>
>>>>>>>>>> You should be able to cast the derived class onto the
>> interface
>>>>>>>>>> class
>>>>>>>>>> and that triggers the call to the correct method.
>>>>>>>>>>
>>>>>>>>>> Andreas
>>>>>>>>>>
>>>>>>>>>> ________________________________________
>>>>>>>>>> From: dune-bounces+a.s.dedner=warwick.ac.uk at dune-project.org
>>>>>>>>>> [dune-bounces+a.s.dedner=warwick.ac.uk at dune-project.org] on
>>>>>>>>>> behalf of Bernd
>>>>>>>>>> Flemisch [bernd at iws.uni-stuttgart.de]
>>>>>>>>>> Sent: 26 October 2012 11:53
>>>>>>>>>> To: Dune
>>>>>>>>>> Subject: [Dune] Dynamic load balancing with ALUGrid
>>>>>>>>>>
>>>>>>>>>> Dear Dune,
>>>>>>>>>>
>>>>>>>>>> I try to use dynamic load balancing with ALUGrid. The grid
>>>>>>>>>> itself gets
>>>>>>>>>> balanced nicely, but I don't succeed in getting data balanced.
>>>>>>>>>>
>>>>>>>>>> From dune/grid/alugrid/3d/grid.hh lines 701ff, I guess that I
>>>>>>>>>> can pack
>>>>>>>>>> a "usual" data handle which is used for communication into a
>>>>>>>>>> ALUGridLoadBalanceDataHandle and call loadBalance with that
>>>>>>>>>> handle. My
>>>>>>>>>> initial attempt to do this fails with
>>>>>>>>>> test-dynamiclb:
>>>>>>>>>> ../../../dune/grid/alugrid/common/defaultindexsets.hh:218:
>>>>>>>>>> Dune::DefaultIndexSet<GridImp, IteratorImp>::IndexType
>>>>>>>>>> Dune::DefaultIndexSet<GridImp, IteratorImp>::index(const
>>> typename
>>>>>>>>>> GridImp::Codim<cd>::Entity&) const [with int cd = 0; GridImp =
>>>>>>>>>> Dune::ALU3dGrid<(Dune::ALU3dGridElementType)7u>; IteratorImp =
>>>>>>>>>> Dune::EntityIterator<0, const
>>>>>>>>>> Dune::ALU3dGrid<(Dune::ALU3dGridElementType)7u>,
>>>>>>>>>> Dune::ALU3dGridLeafIterator<0,
>> (Dune::PartitionIteratorType)4u,
>>>>>>>>>> const
>>>>>>>>>> Dune::ALU3dGrid<(Dune::ALU3dGridElementType)7u> > >;
>>>>>>>>>> Dune::DefaultIndexSet<GridImp, IteratorImp>::IndexType =
>>>>>>>>>> unsigned int;
>>>>>>>>>> typename GridImp::Codim<cd>::Entity = Dune::Entity<0, 3, const
>>>>>>>>>> Dune::ALU3dGrid<(Dune::ALU3dGridElementType)7u>,
>>>>>>>>>> Dune::ALU3dGridEntity>]: Assertion `indexContainer( cd )[ en
>>>>>>>>>> ].index()
>>>>>>>>>>> = 0' failed.
>>>>>>>>>> I attach my effort, where I use a data handle copied from the
>>>>>>>>>> grid-howto. It would be great if you could take a look.
>>>>>>>>>>
>>>>>>>>>> I have a second issue: I guess that I should be able to call
>>>>>>>>>> loadBalance
>>>>>>>>>> directly with a usual data handle, so that grid.hh lines 701ff
>>>>>>>>>> takes
>>>>>>>>>> care of it and packs it for me into a
>>> ALUGridLoadBalanceDataHandle
>>>>>>>>>> before sending it to the real loadBalance method. However,
>>> since
>>>>>>>>>> my data
>>>>>>>>>> handle is only inherited from CommDataHandleIF, the packing
>>>>>>>>>> routine is
>>>>>>>>>> not triggered but the real loadBalance method is called
>>> directly
>>>>>>>>>> leading
>>>>>>>>>> to a compilation error. How can I trigger the execution of the
>>>>>>>>>> packing
>>>>>>>>>> routine for such a derived data handle?
>>>>>>>>>>
>>>>>>>>>> Thank you for having a look. Kind regards
>>>>>>>>>> Bernd
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>
>>> _____________________________________________________________________
>>>>>>>>>>
>>>>>>>>>> Bernd Flemisch phone: +49 711 685 69162
>>>>>>>>>> IWS, Universität Stuttgart fax: +49 711 685 60430
>>>>>>>>>> Pfaffenwaldring 61 email: bernd at iws.uni-stuttgart.de
>>>>>>>>>> D-70569 Stuttgart url:
>>>>>>>>>>
>>> www.hydrosys.uni-stuttgart.de<http://www.hydrosys.uni-stuttgart.de>
>>> _____________________________________________________________________
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Dune mailing list
>>>>>>>>> Dune at dune-project.org
>>>>>>>>> http://lists.dune-project.org/mailman/listinfo/dune
>>>>>>>> --
>>>>>>>> Dr. Martin Nolte <nolte at mathematik.uni-freiburg.de>
>>>>>>>>
>>>>>>>> Universität Freiburg phone:
>>>>>>>> +49-761-203-5630
>>>>>>>> Abteilung für angewandte Mathematik fax:
>>>>>>>> +49-761-203-5632
>>>>>>>> Hermann-Herder-Straße 10
>>>>>>>> 79104 Freiburg, Germany
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Dune mailing list
>>>>>>>> Dune at dune-project.org
>>>>>>>> http://lists.dune-project.org/mailman/listinfo/dune
>>>>>>>>
>>>>>>> ___________________________________________________________
>>>>>>>
>>>>>>> Bernd Flemisch phone: +49 711 685 69162
>>>>>>> IWS, Universitaet Stuttgart fax: +49 711 685 67020
>>>>>>> Pfaffenwaldring 61 email: bernd at iws.uni-stuttgart.de
>>>>>>> D-70569 Stuttgart url:
>>> www.hydrosys.uni-stuttgart.de<http://www.hydrosys.uni-stuttgart.de>
>>>>>>> ___________________________________________________________
>>>>>> ___________________________________________________________
>>>>>>
>>>>>> Bernd Flemisch phone: +49 711 685 69162
>>>>>> IWS, Universitaet Stuttgart fax: +49 711 685 67020
>>>>>> Pfaffenwaldring 61 email: bernd at iws.uni-stuttgart.de
>>>>>> D-70569 Stuttgart url:
>>> www.hydrosys.uni-stuttgart.de<http://www.hydrosys.uni-stuttgart.de>
>>>>>> ___________________________________________________________
>>>>>>
>>>>>> _______________________________________________
>>>>>> Dune mailing list
>>>>>> Dune at dune-project.org
>>>>>> http://lists.dune-project.org/mailman/listinfo/dune
>>>>> ___________________________________________________________
>>>>>
>>>>> Bernd Flemisch phone: +49 711 685 69162
>>>>> IWS, Universitaet Stuttgart fax: +49 711 685 67020
>>>>> Pfaffenwaldring 61 email: bernd at iws.uni-stuttgart.de
>>>>> D-70569 Stuttgart url:
>>> www.hydrosys.uni-stuttgart.de<http://www.hydrosys.uni-stuttgart.de>
>>>>> ___________________________________________________________
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Dune mailing list
>>>>> Dune at dune-project.org
>>>>> http://lists.dune-project.org/mailman/listinfo/dune
>>>>>
>>>
>>> --
>>> _____________________________________________________________________
>>>
>>> Bernd Flemisch phone: +49 711 685 69162
>>> IWS, Universität Stuttgart fax: +49 711 685 60430
>>> Pfaffenwaldring 61 email: bernd at iws.uni-stuttgart.de
>>> D-70569 Stuttgart url:
>>> www.hydrosys.uni-stuttgart.de<http://www.hydrosys.uni-stuttgart.de>
>>> _____________________________________________________________________
>>>
>>>
>>> _______________________________________________
>>> Dune mailing list
>>> Dune at dune-project.org
>>> http://lists.dune-project.org/mailman/listinfo/dune
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> Dune mailing list
>>> Dune at dune-project.org
>>> http://lists.dune-project.org/mailman/listinfo/dune
>
>
--
_____________________________________________________________________
Bernd Flemisch phone: +49 711 685 69162
IWS, Universität Stuttgart fax: +49 711 685 60430
Pfaffenwaldring 61 email: bernd at iws.uni-stuttgart.de
D-70569 Stuttgart url: www.hydrosys.uni-stuttgart.de
_____________________________________________________________________
More information about the Dune
mailing list