[Dune] Memory consumption for parallel unstructured grids

[Paul Maidl]

Dear Dune-developers,

in my previous mail I asked about the memory consumption of unstructured meshes in Dune, that are used in parallel. 
Thanks a lot to Christian Engwer and Oliver Sander for both of your answers.

To answer your questions I did some more test (Test script is attached) with some meshes (bigger than in my first mail):
- I used meshes with approximately 1e6, 5.8e6 and 9.3e6 elements
- All meshes were tested with 1, 4, 8, 16, and 32 threads
- The memory consumption was tracked over time using pidstat. In all results I used the RSS value as memory consumption, but the VSZ value shows very similar trends
- The load balancing between the threads works really well in the sense that all threads have very similar number of elements (a visualization of the partition for the largest mesh on 32 threads is attached in partitionUGGrid.png)
- For this partition each thread has around 300.000 interior elements and 13.000 ghost elements 

Here are my results:
- For higher number of threads the memory consumption of the grid increases. Not as much as I originally thought but still by a factor of 2 even for the largest mesh
- In varyingMeshesUggrid.pdf the memory consumption for the different mesh sizes is plotted over runtime for varying number of threads. One can see a peek during the distribution of the grid (which is, what one can expect, I think), but also during the (bit small) working phase at the end the difference between the small and high numbers of threads remains.
- To have a more detailed look, the memory consumption of all threads is plotted for the largest mesh run on 32 threads (memoryPerThreadUggrid.pdf). This shows one thread (main thread), that reads the grid and distributes it. During this time it consumes a lot more memory then all other threads. This is expected. However even after finishing the distribution its memory consumption stays much higher than for all other threads although it should only store the same amount of elements as the rest does. This is something I do not understand yet.
- The effect of the previous point is the same for the dune AluGrid implementation (see memoryPerThreadAlugrid.pdf), so it does not seem to be an error in the grid implementation. 
- The effect is also independent of the reader of the grid (Gmsh reader and .vtu reader were tested). 

My questions to this are:
- Are there any additional data that thread 0 has to store compared to the rest, which would explain this behaviour?
- Is there a way to overcome this?

Thanks for your help in advance and best regards
Paul Maidl



-------------- next part --------------
A non-text attachment was scrubbed...
Name: dune-uggridTest.cc
Type: application/octet-stream
Size: 3585 bytes
Desc: not available
URL: <https://lists.dune-project.org/pipermail/dune/attachments/20250627/d64bef9c/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: partitionUGGrid.png
Type: image/png
Size: 270583 bytes
Desc: not available
URL: <https://lists.dune-project.org/pipermail/dune/attachments/20250627/d64bef9c/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: varyingMeshesUggrid.pdf
Type: application/pdf
Size: 96745 bytes
Desc: not available
URL: <https://lists.dune-project.org/pipermail/dune/attachments/20250627/d64bef9c/attachment-0003.pdf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: memoryPerThreadUggrid.pdf
Type: application/pdf
Size: 30269 bytes
Desc: not available
URL: <https://lists.dune-project.org/pipermail/dune/attachments/20250627/d64bef9c/attachment-0004.pdf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: memoryPerThreadAlugrid.pdf
Type: application/pdf
Size: 29374 bytes
Desc: not available
URL: <https://lists.dune-project.org/pipermail/dune/attachments/20250627/d64bef9c/attachment-0005.pdf>