[Dune-devel] [GSoC2013][Proposal] Performance testing

[Christian Engwer]

Hi Miha,

> I know this topic already got some interest, so let me be the first to
> write a proposal here. I will add it to Google's site as soon as I get some
> feedback. I would mostly like some clarifications on the infrastructure
> (hardware) side of things.

regarding the hardware there is nothing to rely on. First of all you
should be able to run test on a given machine with given hardware
(just one seq. process). Later we might want to extend this to
parallel tests.

I'd say the best bet is to assume a stable hardware and clear the
history when the hardware changes, as it is nearly impossible to get
comparable results on different hardware (remember the bogomips?)

> *Introduction*
> 
> My name is Miha Čančula and I'm a Physics student from Slovenia. I am
> interested in Dune, especially the software testing project.
> 
> My field is computational physics, so I write a lot of numerical code, but
> I've never heard about Dune until the GSoC listing. This is too bad,
> because I could probably use it for school projects. Apart from physics, I
> like programming free software, and have a special fondness for software
> testing. I write code in a variety of languages (in rough order of
> preference: C++, Python, Octave, Java, JavaScript, ..., VBA, BibTeX
> Anonymous Forth-Like Language).
> 
> I have participated in GSoC for the last two years, once providing
> visualizations for a data mining program, and once adding a unit testing
> module to the KDevelop IDE.
> 
> *The Project
> 
> *
> This projects consists of four basic parts: A number of example benchmarks
> to run, a script to run them and store results, a web-based dashboard for
> statistics and reporting, and infrastructure for running the performance
> tests.
> 
> The first part is most integral to DUNE, but is probably the easiest. I
> will create some executables that make use of various features of DUNE.
> This includes various allocations, common linear algebra operations, etc.
> As I get more familiar with DUNE, I will be able to come up with other
> examples. Some of the existing tests are good candidates.
> 
> The test running environment will measure spent time and consumed memory
> while both compiling and running these benchmarks. I am a long-time Linux
> user, so I have a lot of experience with Linux tools. However, in order to
> allow testing on other systems, I will try to make the script
> cross-platform. The results will be then stored in a database where the
> web-based dashboard can collect them.
> 
> A web dashboard would be similar to the existing one at
> http://autobuild.dune-project.org/buildlog. Unfortunately, the current site
> is very slow, and I don't know why that it. Apart from simply displaying
> the results, the performance testing dashboard will also need to perform
> basic statistics on the results. This means extracting trends and
> identifying outliers, but also comparisons between different architectures
> and some measure of scalability. I have some web programming experience,
> but only with Python and Django. I would prefer to write a dashboard in
> Python, because it has great numerical tools, but could also adapt to
> writing in Perl if really needed.

regarding the buildlog, a phd in Heidelberg wanted to give this a
more little love. And in general I don't see a problem in having an
other tool for the performance test. Actually I'd even prefer having
something seperate and preferably such that it is possible to run it
easily on a local machine, to allow developers of other modules to use
it them selfs.

> I do not know how busy the autobuild machines are at the moment, but
> performance testing would definitely involve more system load that plain
> unit testing.

not necessarily

> I would like to re-use the same testing machines for
> performance testing as well. Unfortunately, it only has GCC at the moment.
> I don't know how those machines are administered, but I hope installing at
> least different versions of GCC and CLang is possible. I also have no idea
> how your project is funded, so I can't really say what would be the best
> infrastructure for this. Alternatively, the tests could run on developers'
> machines. This way there is surely more diverse data (a lot more
> configurations), but there could hardly be continuous testing this way.
> There is also a third way of powering on a couple of virtual machines
> periodically, however I don't like this, because it's expensive and
> resources sharing may affect performance.

We should first get things in a good shape then we can easily add more
test-setups and more compilers.

Running test on developer machines should definitely be an option. Perhaps
the developer could collect statistics in some local DB (e.g. sqlite)
and generate stic html pages to inspect the performance plot?!

> *Questions*
> 
> I think I have the first three part roughly worked out. Of course I will
> probably need help, and I welcome any input now. I will also add some
> details, as well as a projected timeline, before adding the proposal to
> Google's site.
> 
> But I would like to know more about the infrastructure part. Could anyone
> tell me what kind of machines are currently in use? Do you have any plans
> to add new ones, either actual machines or virtual?

In the first part we would not add more machines, but the system
itself should be able to handle changing setups. The current buildlog
system does not keep a configuration of the different platforms, but
generates the list on-the-fly. One issue is that some of the machines
are virtual, so time measurement is a bit more tricky.

For really short test (very short unit-tests) it might even be
interesting to run tests using the callgrind tool of valgrind and
count cpu instructions.

> Finally, do you have any other tips for me?

Not directly ;-) Keep in mind, that you not only have to convince us,
but google also has to grant the slot.

Cheers Christian