[Dune-devel] [GSoC2013][Proposal] Performance testing

[Miha Čančula]

Hello!

I know this topic already got some interest, so let me be the first to
write a proposal here. I will add it to Google's site as soon as I get some
feedback. I would mostly like some clarifications on the infrastructure
(hardware) side of things.

*Introduction*

My name is Miha Čančula and I'm a Physics student from Slovenia. I am
interested in Dune, especially the software testing project.

My field is computational physics, so I write a lot of numerical code, but
I've never heard about Dune until the GSoC listing. This is too bad,
because I could probably use it for school projects. Apart from physics, I
like programming free software, and have a special fondness for software
testing. I write code in a variety of languages (in rough order of
preference: C++, Python, Octave, Java, JavaScript, ..., VBA, BibTeX
Anonymous Forth-Like Language).

I have participated in GSoC for the last two years, once providing
visualizations for a data mining program, and once adding a unit testing
module to the KDevelop IDE.

*The Project

*
This projects consists of four basic parts: A number of example benchmarks
to run, a script to run them and store results, a web-based dashboard for
statistics and reporting, and infrastructure for running the performance
tests.

The first part is most integral to DUNE, but is probably the easiest. I
will create some executables that make use of various features of DUNE.
This includes various allocations, common linear algebra operations, etc.
As I get more familiar with DUNE, I will be able to come up with other
examples. Some of the existing tests are good candidates.

The test running environment will measure spent time and consumed memory
while both compiling and running these benchmarks. I am a long-time Linux
user, so I have a lot of experience with Linux tools. However, in order to
allow testing on other systems, I will try to make the script
cross-platform. The results will be then stored in a database where the
web-based dashboard can collect them.

A web dashboard would be similar to the existing one at
http://autobuild.dune-project.org/buildlog. Unfortunately, the current site
is very slow, and I don't know why that it. Apart from simply displaying
the results, the performance testing dashboard will also need to perform
basic statistics on the results. This means extracting trends and
identifying outliers, but also comparisons between different architectures
and some measure of scalability. I have some web programming experience,
but only with Python and Django. I would prefer to write a dashboard in
Python, because it has great numerical tools, but could also adapt to
writing in Perl if really needed.

I do not know how busy the autobuild machines are at the moment, but
performance testing would definitely involve more system load that plain
unit testing. I would like to re-use the same testing machines for
performance testing as well. Unfortunately, it only has GCC at the moment.
I don't know how those machines are administered, but I hope installing at
least different versions of GCC and CLang is possible. I also have no idea
how your project is funded, so I can't really say what would be the best
infrastructure for this. Alternatively, the tests could run on developers'
machines. This way there is surely more diverse data (a lot more
configurations), but there could hardly be continuous testing this way.
There is also a third way of powering on a couple of virtual machines
periodically, however I don't like this, because it's expensive and
resources sharing may affect performance.

*Questions*

I think I have the first three part roughly worked out. Of course I will
probably need help, and I welcome any input now. I will also add some
details, as well as a projected timeline, before adding the proposal to
Google's site.

But I would like to know more about the infrastructure part. Could anyone
tell me what kind of machines are currently in use? Do you have any plans
to add new ones, either actual machines or virtual?

Finally, do you have any other tips for me?

Thank you,
Miha Čančula
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dune-project.org/pipermail/dune-devel/attachments/20130424/5b3ca837/attachment.htm>