[Dune-devel] JIT compiling of python code uses wrong compile_commands

[Dedner, Andreas]

Great - I don't quite remember what the deal was with the PACKAGENAME. I think its the name of the
'namespace package', i.e., dune in your case which is the default if nor provided.
I  think it is needed to allow for a different package naming convention as required for dumux for example.
Andreas
________________________________
From: Simon Praetorius <simon.praetorius at tu-dresden.de>
Sent: 03 April 2025 22:27
To: dune-devel at lists.dune-project.org <dune-devel at lists.dune-project.org>
Cc: Dedner, Andreas <A.S.Dedner at warwick.ac.uk>
Subject: Re: [Dune-devel] JIT compiling of python code uses wrong compile_commands


Ok, I might have found the problem. It was a small detail in you mail, Andres, that pointed me in a direction to investigate:


There should be a file " in the builddir of dune-vtk under python/dune/data/dune-vtk.cmake.", but I have found mine in a slightly different directory, in python/vtk/data/dune-vtk.cmake. First, I though, this might have been a typo, this seems to be the problem.


How is this file generated? I don't know exactly, but it might be somehow in the dune_python_configure_bindings() macro. This macro was changed some time ago and I had to update the call from the old function to the new one. There was not indication what the new arguments mean, in relation to the old macro parameters, so my solution in dune-vtk/python/CMakeLists.txt was


dune_python_configure_bindings(
PATH "."
PACKAGENAME vtk
CMAKE_METADATA_FLAGS DUNE_OPTS_FILE
)


I named the package "vtk". This seems to be a problem. I don't know where this "PACKAGENAME" argument might be used, but it generates the file in the wrong directory, which is then not found in the whole machinery.

Removing this argument made the file dune-vtk.cmake appear in the correct directory and also the compile_command.json (and the CMakeLists file that it is generated from) contain the dune-vtk directory.


Thanks for the hints and the questions where to start looking.


Best,
Simon


Am 03.04.25 um 23:12 schrieb Simon Praetorius:

Hi Andreas,


thanks for the hints already. I have followed your questions (see below), but still one step in the middle seems to fail.

My first question would be:
what does build/run-in-dune-venv.py pip list | grep dune give you? Most importantly does it list dune-vtk?

Yes, dune-vtk is listed there


If that is found by pip then check the `dune-py/CMakeLists.txt` file if the line set(dune-vtk_DIR PATH) is there.

The dune-py/CMakeLists.txt does **not** contain the vtk dir or anything else from dune-vtk (e.g. no HAVE_DUNE_VTK variable)


The path comes from the metadata file which should be in the builddir of dune-vtk under python/dune/data/dune-vtk.cmake.

The dune-vtk.cmake file exists and contains the following lines (and a few more)


DEPBUILDDIRS=/opt/sources/dune/dune-vtk/build/gcc14-debug;/opt/sources/dune/dune-common/build/gcc14-debug;/opt/sources/dune/dune-geometry/build/gcc14-debug;/opt/sources/dune/dune-localfunctions/build/gcc14-debug;/opt/sources/dune/dune-uggrid/build/gcc14-debug;/opt/sources/dune/dune-grid/build/gcc14-debug;/opt/sources/dune/dune-istl/build/gcc14-debug;/opt/sources/dune/dune-typetree/build/gcc14-debug;/opt/sources/dune/dune-functions/build/gcc14-debug;/opt/sources/dune/dune-alugrid/build/gcc14-debug;/opt/sources/dune/dune-foamgrid/build/gcc14-debug
DEPS=dune-vtk dune-common dune-geometry dune-localfunctions dune-uggrid dune-grid dune-istl dune-typetree dune-functions dune-alugrid dune-foamgrid


This indicates that the metadata is correct, but somehow ignored.


Only after all that looks correct I would check the compile_commands.json file.

This file also does not contain the required include directories of dune-vtk.


That is generated in python/dune/generator/cmakebuilder.py:547
by compiling a empty cc and extracting the information from the files that cmake generated.

Since the CMakeLists.txt file in the dune-py dir is incomplete, the generated compile_commands cannot be correct either. So, now the question is, how is this CMakeLists file generated. How can I inspect or activate the logging tools? And where to find the log output? Maybe there is an information that helps.



I guess this step depends on internal structures of cmake generated files (i.e. link.txt' and could possibly fail with some newer CMake version.

Best
Andreas
________________________________
From: Dune-devel <dune-devel-bounces at lists.dune-project.org><mailto:dune-devel-bounces at lists.dune-project.org> on behalf of Simon Praetorius <simon.praetorius at tu-dresden.de><mailto:simon.praetorius at tu-dresden.de>
Sent: 03 April 2025 20:04
To: dune-devel at lists.dune-project.org<mailto:dune-devel at lists.dune-project.org> <dune-devel at lists.dune-project.org><mailto:dune-devel at lists.dune-project.org>
Subject: [Dune-devel] JIT compiling of python code uses wrong compile_commands


Hi everyone,

I have posted a question already in the Dune support channel, but maybe here in the mailing list are more python experts:

I am currently trying to run a simple Python example in the dune-vtk module, i.e. dune/python/test/testvtk.py. I have built everything without a user specified venv, using the default that is created in the build directory. To run the code, I use the script build/run-in-dune-venv.py in my build directory build/. The problem I have is: The JIT compilation step uses an incomplete list of include directories. In particular, the include dirs from dune-vtk are not there. This can be seen in the compile_commands.json file, which is located in the dune-py directory in the dune-common build directory build/dune-venv/.cache/dune-py.

The error I get is "fatal error: dune/python/vtk/writer.hh: file or directory not found".

Can anyone tell me how this compile_commands.json file is generated? I am trying to figure out why it does not include all the include directories of the modules involved.

Best, Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dune-project.org/pipermail/dune-devel/attachments/20250403/4d362e1d/attachment-0001.htm>