[Dune-devel] [GSoC] Performance testing: more detailed schedule

[Christian Engwer]

Dear Miha,

I have some new replies :-)

I looked into the code and

a) ran into a problem:

christi at sansibar:~/Uni/Dune/dune-perftest$ ./dune/perftest/perftest.py perftest.ini
Traceback (most recent call last):
  File "./dune/perftest/perftest.py", line 86, in <module>
    test_project(sys.argv[1])
  File "./dune/perftest/perftest.py", line 30, in test_project
    if not CONFIG_TOP_SECTION in config:
TypeError: argument of type 'instance' is not iterable

You must be conservative with the python features you use!

b) I _really_ recommend using the buildsystem to invoke the
   test program.
   - I have the following problem: I use out-of-source builds, thus the
     python script is not at a predictable position relative the test
     prog. The same holds for the ini file.
   - If you want to extract buildsystem information like the compiler
     and its options, you don't want to parse the Makefile
     yourself... just use the Makefile
   - The buildsystem already knows about all applications and all
     tests. In order to avoid duplicate data, it is again best to let
     the buildsystem execute the 
   An other option would be to at least use the buildsystem to
   generate the final configuration. In this case we should discuss
   the way the config file structured right now.

c) for some reason I'm not able to see the results graph :-(
   I guess this is due to your move to external csv files. For local
   display I believe you have to include the data into the html file.

d) I think it would be better to generate html files for the
   different programs and modes. Build and run may need significantly
   different resources. The same holds for the different applications.

e) It is also necessary to detect failed runs and leave broken
   builds/runs out of the statistics.

Keep up the nice work!

I suggest to try the buildsystem (aka autotools) integration asap. If
you have any questions pose them on the list. I'll try to answer
directly and perhaps some of my fellow developers can also help you.

Please fell free to differ in your opinion and propose different
options.

Coming back to some older topics...

> So, my timeline for this week is
>  - convert the docs to Doxygen, and make it optional
>  - add memory consumption to graphs (already done).
>  - automated (as in, just calling one command) running of built-in tests.
>  - statistics (only mean, stdev for outliers, and linear trends)

How is your progress?

Especially point 3 seems tricky to me if you don't integrate with the buildsytem...

Cheers
Christian

On Sun, Aug 11, 2013 at 09:57:25PM +0200, Christian Engwer wrote:
> Dear Miha,
> 
> I'm back again and try to complete some of the
> questions/anwers.
> 
> > should be similar for most other VCS's. I'm not sure about getting the
> > compiler and its flags automatically, though.
> > I'll see what I can do.
> 
> The easiest was is to use the build-system itself. The generated
> Makefiles know appropriate compiler options etc... You can have a look
> at dune-common/am/checklog where I added additional Make targets for
> the checklog output. As this requires changes to the buildsystem, I
> suggested to only support autotools atm as it is our "official"
> buildsystem.
> 
> Cheers
> Christian
> 
> PS: I planed to look into your updated code tomorrow and will come
> back to you then!
> 
> _______________________________________________
> Dune-devel mailing list
> Dune-devel at dune-project.org
> http://lists.dune-project.org/mailman/listinfo/dune-devel
> 

-- 
Prof. Dr. Christian Engwer 
Institut für Numerische und Angewandte Mathematik
Fachbereich Mathematik und Informatik der Universität Münster
Einsteinstrasse 62
48149 Münster

E-Mail	christian.engwer at uni-muenster.de
Telefon	+49 251 83-35067
FAX		+49 251 83-32729