[Dune] Dynamic load balancing with ALUGrid
Dedner, Andreas
A.S.Dedner at warwick.ac.uk
Fri Feb 22 12:29:57 CET 2013
I could guess most of that but I am struggeling with `gid Alsop'...
________________________________________
From: Christian Engwer [christian.engwer at uni-muenster.de]
Sent: 22 February 2013 11:18
To: Dedner, Andreas; Dedner, Andreas; bernd at iws.uni-stuttgart.de; dune at dune-project.org
Subject: Re: [Dune] Dynamic load balancing with ALUGrid
"Dedner, Andreas" <A.S.Dedner at warwick.ac.uk> schrieb:
>Hi.
>That is one reason why the loadbalance method takes a callback object
>with gather and scatter like communicate. The grid then makes sure that
>these methods are called on the right entities.
So, doesn't this allow top update the gid while looping through the scatter and providing the correct gid Alsop during load-balance?
In case this is not possible I'd suggest to at least throw an exception when along for the gid.
Christian
> So while gathering the
>data on one processor its local idset can be used and the same holds on
>the receiving process. In the grid-howto there is an example showing a
>possible way to use this with the persistent containers.
>Andreas
>
>Bernd Flemisch <bernd at iws.uni-stuttgart.de> wrote:
>Hi Martin,
>
>thank you for your answer. This indeed sounds a lot earlier sier and
>like the
>solution I originally wanted to achieve. Nevertheless, I am a bit
>puzzled now. I deliberately preferred global over local Ids, since
>"A local id set provides ids that are unique within one process but two
>entities in different processes may have the same id," see
>http://www.dune-project.org/doc/doxygen/dune-grid-html/class_dune_1_1_id_set.html#_details
>
>So I thought that with local Ids it could happen that an entity is
>moved
>from one process to another, where another entity with the same Id
>already exists. And then this would not work out. Or am I missing
>something?
>
>Kind regards
>Bernd
>
>On 02/22/2013 09:13 AM, Martin Nolte wrote:
>> Hi Bernd,
>>
>> I guess you are the first to use global ids during load balancing and
>> it seems you hit on a bug in ALUGrid (please file a complete bug
>> report on this).
>>
>> Actually, the solution we use is far simpler. Store your data in a
>> std::map using the LocalIdSet (or use a PersistentContainer). This
>> way, your data will always be accessible, even during load balancing.
>> Then you write a data handle (warning: you communicate on the entire
>> hierarchy). That's it.
>>
>> Let me give a bit of explaination: Behind the scenes, ALUGrid will
>> pack data for a macro element to be sent to another process. To this
>> end, it will traverse the tree and pack the element information and
>> your data (calling gather). When all macro elements have been packed,
>> the data is sent to the new processes. Here, they are unpacked, i.e.,
>> the element is constructed and all scatter is called. During scatter,
>> the ids will already be valid, so you can use them to access your
>data
>> container (though the global id might be buggy, as you stated).
>>
>> One final opinion: GlobalIds are an artificial DUNE construct.
>> Implementing them might be quite complicated and expensive. This is
>> the case for ALUGrid, where they are generated in the DUNE bindings.
>> Hence, they have to communicated during load balancing (expensive).
>> So, if you ask ALUGrid for the global id set once, you got to pay
>this
>> price. There is no way of telling how long this id set is to be used
>> (because it is returned by reference).
>>
>> Best,
>>
>> Martin
>>
>>
>>
>>
>>
>> On 02/22/2013 07:35 AM, Bernd Flemisch wrote:
>>> Hi,
>>>
>>> sorry for the noise again, but I found a working solution. So if you
>>> already
>>> started to brood over the problem, you may relax.
>>>
>>> I assume that the Ids are still valid during gather and will become
>>> valid
>>> again and with the same values after load balance. I simply gather
>>> the Ids
>>> together with the data and scatter both of them. In a postprocess
>>> after load
>>> balance, I use this (now valid) information to write the values into
>the
>>> corresponding vector.
>>>
>>> This works for several grids and numbers of processes, so I hope for
>>> all of
>>> them. I attach the file.
>>>
>>> If you think this is important enough to become a test in
>>> dune/grid/test and
>>> feel responsible for evaluating and committing it, I am happy to
>>> provide a
>>> patch and to improve it together with you.
>>>
>>> Thanks to Andreas and Martin for the hints which helped to achieve
>>> this. Kind
>>> regards
>>> Bernd
>>>
>>> On Thu, 21 Feb 2013 19:12:44 +0100
>>> Bernd Flemisch <bernd at iws.uni-stuttgart.de> wrote:
>>>> Actually, the Ids are the same again after the load balancing step
>>>> is over.
>>>> But apparently not during scatter, where I would need it. Hmmm...
>>>>
>>>> On Thu, 21 Feb 2013 17:55:51 +0100
>>>> Bernd Flemisch <bernd at iws.uni-stuttgart.de> wrote:
>>>>> Hi,
>>>>>
>>>>> only a few months have passed and I found the time to look at this
>>>>> again. I
>>>>> tried to follow Andreas' suggestion and work with Ids rather than
>>>>> indices,
>>>>> since they should be persistent. I was hoping that "persistent"
>>>>> means that,
>>>>> if an element is moved from one process to another, it keeps its
>>>>> global Id.
>>>>> Looking at
>>>>>
>http://www.dune-project.org/doc/doxygen/dune-grid-html/class_dune_1_1_id_set.html#_details
>>>>>
>>>>>
>>>>> does not necessarily say that (I think).
>>>>>
>>>>> I set up a test for my efforts, see the attached file. The coarse
>>>>> grid is
>>>>> 5x1x1. The two left elements are interior on process 1, the three
>>>>> right
>>>>> ones interior on process 2. The two left elements get refined.
>After
>>>>> adaption, loadBalance with my DataHandle is called. The three
>right
>>>>> elements are all moved to process 1, so that after the balancing
>step,
>>>>> process 1 holds all elements, while process 0 has none.
>>>>>
>>>>> During load balance, I print the cell centers, global Ids, and
>>>>> gathered/scattered values on each process. This gives the
>following:
>>>>> Process 0:
>>>>> 0: 0.5 0.5 0.5, ([2,3,8,14],8,0): gather 0
>>>>> 0: 0.7 0.5 0.5, ([3,4,9,15],8,0): gather 1
>>>>> 0: 0.9 0.5 0.5, ([4,5,10,16],8,0): gather 2
>>>>> Process 1:
>>>>> 1: 0.5 0.5 0.5, ([2,3,8,14],71,0): scatter 0
>>>>> 1: 0.7 0.5 0.5, ([2,3,8,14],70,0): scatter 1
>>>>> 1: 0.9 0.5 0.5, ([2,3,8,14],69,0): scatter 2
>>>>> It seems to be correct that information from three elements is
>sent
>>>>> from
>>>>> process 0 and received for the corresponding elements by process
>1.
>>>>> However, the global Ids are changed. Is this correct? Then how tf
>>>>> can I
>>>>> find out which elements correspond to each other? Or am I still
>doing
>>>>> something wrong?
>>>>>
>>>>> Thank you for having another look. Kind regards
>>>>> Bernd
>>>>>
>>>>> On Sat, 27 Oct 2012 07:43:48 +0000
>>>>> "Dedner, Andreas" <A.S.Dedner at warwick.ac.uk> wrote:
>>>>>> Is it not even so that one can not use an undexset at all? Load
>>>>>> balancing
>>>>>> changes the grid so the indexsets are not valid. You need to use
>id
>>>>>> containers.
>>>>>> Andreas
>>>>>>
>>>>>> Martin Nolte <nolte at mathematik.uni-freiburg.de> wrote:
>>>>>> Hi Bernd,
>>>>>>
>>>>>> I have an educated guess for you (don't blame me, if I'm wrong):
>>>>>>
>>>>>> In normal communication, you only communicate data on a certain
>>>>>> grid view.
>>>>>> Hence
>>>>>> you can use the index set to address data stored on that grid
>view.
>>>>>>
>>>>>> When doing load balancing, you rather communicate data on the
>entire
>>>>>> hierarchy.
>>>>>> Therefore, your entity might not be contained an index set
>>>>>> (leading to this
>>>>>> cryptic assertion). So, if you want to communicate data only on a
>>>>>> certain
>>>>>> grid
>>>>>> view, you have to add a call to indexSet.contains to the gather /
>>>>>> scatter
>>>>>> / size
>>>>>> methods. Usually, fixedSize will be false in such situations (I
>>>>>> think ALUGrid
>>>>>> ignores this anyway, so it should not matter, here).
>>>>>>
>>>>>> Just a side note: In my opionion, a normal communicate method
>>>>>> exchanging
>>>>>> data on
>>>>>> the entire hierarchy would be nice, too. But alas, this method
>would
>>>>>> naturally
>>>>>> be "grid().communicate( dataHandle, interface, direction )".
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> Martin
>>>>>>
>>>>>> On 10/26/2012 01:32 PM, Bernd Flemisch wrote:
>>>>>>> Hey Andreas,
>>>>>>>
>>>>>>> thank you for your fast answer. This indeed solves my second
>issue.
>>>>>>> However, the
>>>>>>> first one remains. I attach an updated test file.
>>>>>>>
>>>>>>> Kind regards
>>>>>>> Bernd
>>>>>>>
>>>>>>> On 10/26/2012 01:08 PM, Dedner, Andreas wrote:
>>>>>>>> Hi Bernd
>>>>>>>>
>>>>>>>> You should be able to cast the derived class onto the interface
>>>>>>>> class
>>>>>>>> and that triggers the call to the correct method.
>>>>>>>>
>>>>>>>> Andreas
>>>>>>>>
>>>>>>>> ________________________________________
>>>>>>>> From: dune-bounces+a.s.dedner=warwick.ac.uk at dune-project.org
>>>>>>>> [dune-bounces+a.s.dedner=warwick.ac.uk at dune-project.org] on
>>>>>>>> behalf of Bernd
>>>>>>>> Flemisch [bernd at iws.uni-stuttgart.de]
>>>>>>>> Sent: 26 October 2012 11:53
>>>>>>>> To: Dune
>>>>>>>> Subject: [Dune] Dynamic load balancing with ALUGrid
>>>>>>>>
>>>>>>>> Dear Dune,
>>>>>>>>
>>>>>>>> I try to use dynamic load balancing with ALUGrid. The grid
>>>>>>>> itself gets
>>>>>>>> balanced nicely, but I don't succeed in getting data balanced.
>>>>>>>>
>>>>>>>> From dune/grid/alugrid/3d/grid.hh lines 701ff, I guess that I
>>>>>>>> can pack
>>>>>>>> a "usual" data handle which is used for communication into a
>>>>>>>> ALUGridLoadBalanceDataHandle and call loadBalance with that
>>>>>>>> handle. My
>>>>>>>> initial attempt to do this fails with
>>>>>>>> test-dynamiclb:
>>>>>>>> ../../../dune/grid/alugrid/common/defaultindexsets.hh:218:
>>>>>>>> Dune::DefaultIndexSet<GridImp, IteratorImp>::IndexType
>>>>>>>> Dune::DefaultIndexSet<GridImp, IteratorImp>::index(const
>typename
>>>>>>>> GridImp::Codim<cd>::Entity&) const [with int cd = 0; GridImp =
>>>>>>>> Dune::ALU3dGrid<(Dune::ALU3dGridElementType)7u>; IteratorImp =
>>>>>>>> Dune::EntityIterator<0, const
>>>>>>>> Dune::ALU3dGrid<(Dune::ALU3dGridElementType)7u>,
>>>>>>>> Dune::ALU3dGridLeafIterator<0, (Dune::PartitionIteratorType)4u,
>>>>>>>> const
>>>>>>>> Dune::ALU3dGrid<(Dune::ALU3dGridElementType)7u> > >;
>>>>>>>> Dune::DefaultIndexSet<GridImp, IteratorImp>::IndexType =
>>>>>>>> unsigned int;
>>>>>>>> typename GridImp::Codim<cd>::Entity = Dune::Entity<0, 3, const
>>>>>>>> Dune::ALU3dGrid<(Dune::ALU3dGridElementType)7u>,
>>>>>>>> Dune::ALU3dGridEntity>]: Assertion `indexContainer( cd )[ en
>>>>>>>> ].index()
>>>>>>>> >= 0' failed.
>>>>>>>>
>>>>>>>> I attach my effort, where I use a data handle copied from the
>>>>>>>> grid-howto. It would be great if you could take a look.
>>>>>>>>
>>>>>>>> I have a second issue: I guess that I should be able to call
>>>>>>>> loadBalance
>>>>>>>> directly with a usual data handle, so that grid.hh lines 701ff
>>>>>>>> takes
>>>>>>>> care of it and packs it for me into a
>ALUGridLoadBalanceDataHandle
>>>>>>>> before sending it to the real loadBalance method. However,
>since
>>>>>>>> my data
>>>>>>>> handle is only inherited from CommDataHandleIF, the packing
>>>>>>>> routine is
>>>>>>>> not triggered but the real loadBalance method is called
>directly
>>>>>>>> leading
>>>>>>>> to a compilation error. How can I trigger the execution of the
>>>>>>>> packing
>>>>>>>> routine for such a derived data handle?
>>>>>>>>
>>>>>>>> Thank you for having a look. Kind regards
>>>>>>>> Bernd
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>_____________________________________________________________________
>>>>>>>>
>>>>>>>>
>>>>>>>> Bernd Flemisch phone: +49 711 685 69162
>>>>>>>> IWS, Universität Stuttgart fax: +49 711 685 60430
>>>>>>>> Pfaffenwaldring 61 email: bernd at iws.uni-stuttgart.de
>>>>>>>> D-70569 Stuttgart url:
>>>>>>>>
>www.hydrosys.uni-stuttgart.de<http://www.hydrosys.uni-stuttgart.de>
>>>>>>>>
>_____________________________________________________________________
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Dune mailing list
>>>>>>> Dune at dune-project.org
>>>>>>> http://lists.dune-project.org/mailman/listinfo/dune
>>>>>>
>>>>>> --
>>>>>> Dr. Martin Nolte <nolte at mathematik.uni-freiburg.de>
>>>>>>
>>>>>> Universität Freiburg phone:
>>>>>> +49-761-203-5630
>>>>>> Abteilung für angewandte Mathematik fax:
>>>>>> +49-761-203-5632
>>>>>> Hermann-Herder-Straße 10
>>>>>> 79104 Freiburg, Germany
>>>>>>
>>>>>> _______________________________________________
>>>>>> Dune mailing list
>>>>>> Dune at dune-project.org
>>>>>> http://lists.dune-project.org/mailman/listinfo/dune
>>>>>>
>>>>>
>>>>> ___________________________________________________________
>>>>>
>>>>> Bernd Flemisch phone: +49 711 685 69162
>>>>> IWS, Universitaet Stuttgart fax: +49 711 685 67020
>>>>> Pfaffenwaldring 61 email: bernd at iws.uni-stuttgart.de
>>>>> D-70569 Stuttgart url:
>www.hydrosys.uni-stuttgart.de<http://www.hydrosys.uni-stuttgart.de>
>>>>> ___________________________________________________________
>>>>
>>>> ___________________________________________________________
>>>>
>>>> Bernd Flemisch phone: +49 711 685 69162
>>>> IWS, Universitaet Stuttgart fax: +49 711 685 67020
>>>> Pfaffenwaldring 61 email: bernd at iws.uni-stuttgart.de
>>>> D-70569 Stuttgart url:
>www.hydrosys.uni-stuttgart.de<http://www.hydrosys.uni-stuttgart.de>
>>>> ___________________________________________________________
>>>>
>>>> _______________________________________________
>>>> Dune mailing list
>>>> Dune at dune-project.org
>>>> http://lists.dune-project.org/mailman/listinfo/dune
>>>
>>> ___________________________________________________________
>>>
>>> Bernd Flemisch phone: +49 711 685 69162
>>> IWS, Universitaet Stuttgart fax: +49 711 685 67020
>>> Pfaffenwaldring 61 email: bernd at iws.uni-stuttgart.de
>>> D-70569 Stuttgart url:
>www.hydrosys.uni-stuttgart.de<http://www.hydrosys.uni-stuttgart.de>
>>> ___________________________________________________________
>>>
>>>
>>> _______________________________________________
>>> Dune mailing list
>>> Dune at dune-project.org
>>> http://lists.dune-project.org/mailman/listinfo/dune
>>>
>>
>
>
>--
>_____________________________________________________________________
>
>Bernd Flemisch phone: +49 711 685 69162
>IWS, Universität Stuttgart fax: +49 711 685 60430
>Pfaffenwaldring 61 email: bernd at iws.uni-stuttgart.de
>D-70569 Stuttgart url:
>www.hydrosys.uni-stuttgart.de<http://www.hydrosys.uni-stuttgart.de>
>_____________________________________________________________________
>
>
>_______________________________________________
>Dune mailing list
>Dune at dune-project.org
>http://lists.dune-project.org/mailman/listinfo/dune
>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Dune mailing list
>Dune at dune-project.org
>http://lists.dune-project.org/mailman/listinfo/dune
More information about the Dune
mailing list