[Dune-devel] [GSoC2013][Proposal] Performance testing

[Miha Čančula]

Thank you for your answers. I wrote a detailed proposal on the GSoC page,
and I tried to make it so everything can be run locally (on one machine),
but the results will be able to be sent to a central website as well. My
proposal is here:
https://google-melange.appspot.com/gsoc/proposal/review/google/gsoc2013/noughmad/74002.



2013/4/25 Christian Engwer <christian.engwer at uni-muenster.de>

> Hi Miha,
>
> > I know this topic already got some interest, so let me be the first to
> > write a proposal here. I will add it to Google's site as soon as I get
> some
> > feedback. I would mostly like some clarifications on the infrastructure
> > (hardware) side of things.
>
> regarding the hardware there is nothing to rely on. First of all you
> should be able to run test on a given machine with given hardware
> (just one seq. process). Later we might want to extend this to
> parallel tests.
>
> I'd say the best bet is to assume a stable hardware and clear the
> history when the hardware changes, as it is nearly impossible to get
> comparable results on different hardware (remember the bogomips?)
>
> > *Introduction*
> >
> > My name is Miha Čančula and I'm a Physics student from Slovenia. I am
> > interested in Dune, especially the software testing project.
> >
> > My field is computational physics, so I write a lot of numerical code,
> but
> > I've never heard about Dune until the GSoC listing. This is too bad,
> > because I could probably use it for school projects. Apart from physics,
> I
> > like programming free software, and have a special fondness for software
> > testing. I write code in a variety of languages (in rough order of
> > preference: C++, Python, Octave, Java, JavaScript, ..., VBA, BibTeX
> > Anonymous Forth-Like Language).
> >
> > I have participated in GSoC for the last two years, once providing
> > visualizations for a data mining program, and once adding a unit testing
> > module to the KDevelop IDE.
> >
> > *The Project
> >
> > *
> > This projects consists of four basic parts: A number of example
> benchmarks
> > to run, a script to run them and store results, a web-based dashboard for
> > statistics and reporting, and infrastructure for running the performance
> > tests.
> >
> > The first part is most integral to DUNE, but is probably the easiest. I
> > will create some executables that make use of various features of DUNE.
> > This includes various allocations, common linear algebra operations, etc.
> > As I get more familiar with DUNE, I will be able to come up with other
> > examples. Some of the existing tests are good candidates.
> >
> > The test running environment will measure spent time and consumed memory
> > while both compiling and running these benchmarks. I am a long-time Linux
> > user, so I have a lot of experience with Linux tools. However, in order
> to
> > allow testing on other systems, I will try to make the script
> > cross-platform. The results will be then stored in a database where the
> > web-based dashboard can collect them.
> >
> > A web dashboard would be similar to the existing one at
> > http://autobuild.dune-project.org/buildlog. Unfortunately, the current
> site
> > is very slow, and I don't know why that it. Apart from simply displaying
> > the results, the performance testing dashboard will also need to perform
> > basic statistics on the results. This means extracting trends and
> > identifying outliers, but also comparisons between different
> architectures
> > and some measure of scalability. I have some web programming experience,
> > but only with Python and Django. I would prefer to write a dashboard in
> > Python, because it has great numerical tools, but could also adapt to
> > writing in Perl if really needed.
>
> regarding the buildlog, a phd in Heidelberg wanted to give this a
> more little love. And in general I don't see a problem in having an
> other tool for the performance test. Actually I'd even prefer having
> something seperate and preferably such that it is possible to run it
> easily on a local machine, to allow developers of other modules to use
> it them selfs.
>
> > I do not know how busy the autobuild machines are at the moment, but
> > performance testing would definitely involve more system load that plain
> > unit testing.
>
> not necessarily
>
> > I would like to re-use the same testing machines for
> > performance testing as well. Unfortunately, it only has GCC at the
> moment.
> > I don't know how those machines are administered, but I hope installing
> at
> > least different versions of GCC and CLang is possible. I also have no
> idea
> > how your project is funded, so I can't really say what would be the best
> > infrastructure for this. Alternatively, the tests could run on
> developers'
> > machines. This way there is surely more diverse data (a lot more
> > configurations), but there could hardly be continuous testing this way.
> > There is also a third way of powering on a couple of virtual machines
> > periodically, however I don't like this, because it's expensive and
> > resources sharing may affect performance.
>
> We should first get things in a good shape then we can easily add more
> test-setups and more compilers.
>
> Running test on developer machines should definitely be an option. Perhaps
> the developer could collect statistics in some local DB (e.g. sqlite)
> and generate stic html pages to inspect the performance plot?!
>
> > *Questions*
> >
> > I think I have the first three part roughly worked out. Of course I will
> > probably need help, and I welcome any input now. I will also add some
> > details, as well as a projected timeline, before adding the proposal to
> > Google's site.
> >
> > But I would like to know more about the infrastructure part. Could anyone
> > tell me what kind of machines are currently in use? Do you have any plans
> > to add new ones, either actual machines or virtual?
>
> In the first part we would not add more machines, but the system
> itself should be able to handle changing setups. The current buildlog
> system does not keep a configuration of the different platforms, but
> generates the list on-the-fly. One issue is that some of the machines
> are virtual, so time measurement is a bit more tricky.
>
> For really short test (very short unit-tests) it might even be
> interesting to run tests using the callgrind tool of valgrind and
> count cpu instructions.
>
> > Finally, do you have any other tips for me?
>
> Not directly ;-) Keep in mind, that you not only have to convince us,
> but google also has to grant the slot.
>
> Cheers Christian
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dune-project.org/pipermail/dune-devel/attachments/20130427/e9fa61cf/attachment.htm>