[Dune-devel] [GSoC] Performance testing: more detailed schedule

[Miha Čančula]

Hello, Christian!

I'm sorry for not replying earlier, my girlfriend came back from a
conference and we had to visit her family. I read email regularly, but I
didn't do anything since the weekend because I have my development setup at
home. I also get few chances to reply, so I hope to address all the points
now.

1.) Python's configparser changed during the 2->3 transition, but I didn't
know the changes were that significant. I'll get back home tomorrow, and I
will test your changes then. According to the docs, the get() syntax is
available in the new version, so we can go with this one.

2.) Using the buildsystem to call the measurement routine. I agree, this is
probably the best way to get all the needed compiler information. Of
course, it should be easy to use, so a single macro or target (per measured
program) should be used. I will try to do it in automake, but would it be
ok if I first write a CMake macro, get the details worked out, and then do
the same with autotools? Considering the use of two build systems, I this
part would be a pretty thin wrapper, just getting the compiler info and
passing it to python.

3.) Split HTML files. I agree, this would be a good idea. I don't know if
it works for you, but I added table filtering, so you can choose to only
display Run, Compile or all measurements. I would keep the total file, but
I can also generate type-specific ones.

4.) No graphs in resIults. I don't know why that should be. Yes, the data
is in a separate CSV file, but in my testing, it was always loaded
correctly. There might be some errors in handling paths, I'll check when I
get home.

5.) Automatic testing. The program now parses a configuration file
(perftest.ini) and runs the tests described in it. This means you still
have to manually run a command, but it's only one command per run. I
suppose "make perftest" is easier to remember than "perftest.py
perftest.ini", so some integration with the buildsystem is needed. However,
I wouldn't run performance tests automatically every time the program is
compiled, this would take too much time.

6.) My progress. I added statistics for finding outliers, you may notice in
the results some rows are different colors depending on their distance from
the mean. Currently all points with at least 1-sigma deviation are marked,
but I don't think that should be the case. There are more rigorous
definition of outliers. The documentation uses Doxygen,
but I haven't converted all the actual docstrings yet. The graphs show both
memory consumption and time spent, and there are separate graphs for
compilation and running.


2013/8/14 Christian Engwer <christian.engwer at uni-muenster.de>

> Hi Miha,
>
> I didn't yet get a response on my mail from the weekend. Is everythin
> all right?
>
> Christian
>
> On Tue, Aug 13, 2013 at 09:22:58AM +0200, Christian Engwer wrote:
> > > a) ran into a problem:
> > >
> > > christi at sansibar:~/Uni/Dune/dune-perftest$
> ./dune/perftest/perftest.py perftest.ini
> > > Traceback (most recent call last):
> > >   File "./dune/perftest/perftest.py", line 86, in <module>
> > >     test_project(sys.argv[1])
> > >   File "./dune/perftest/perftest.py", line 30, in test_project
> > >     if not CONFIG_TOP_SECTION in config:
> > > TypeError: argument of type 'instance' is not iterable
> >
> > the attached patch fixed the problems for me... not sure what it will
> > do to python3?!
> >
> > Christian
>
> > diff --git a/dune/perftest/perftest.py b/dune/perftest/perftest.py
> > index f477589..e541a3e 100755
> > --- a/dune/perftest/perftest.py
> > +++ b/dune/perftest/perftest.py
> > @@ -27,28 +27,30 @@ def test_project(config_file):
> >    config = configparser.ConfigParser()
> >    config.read(config_file)
> >
> > -  if not CONFIG_TOP_SECTION in config:
> > +  if not CONFIG_TOP_SECTION in config.sections():
> >      raise SyntaxError("The configuration file must have a [DUNE
> Perftest] section")
> >
> > -  program_name = config[CONFIG_TOP_SECTION][KEY_PROGRAM_NAME]
> > -  output_file = config[CONFIG_TOP_SECTION][KEY_OUTPUT]
> > +  program_name = config.get(CONFIG_TOP_SECTION,KEY_PROGRAM_NAME)
> > +  output_file = config.get(CONFIG_TOP_SECTION,KEY_OUTPUT)
> >
> >    revision = None
> > -  if KEY_VCS in config[CONFIG_TOP_SECTION]:
> > -    revision = get_revision(dir_name,
> config[CONFIG_TOP_SECTION][KEY_VCS])
> > +  if KEY_VCS in config.items(CONFIG_TOP_SECTION):
> > +    revision = get_revision(dir_name,
> config.get(CONFIG_TOP_SECTION,KEY_VCS))
> >
> > -  for section in config:
> > +  for section in config.sections():
> > +    print "Parsing section " + section
> >      if section == CONFIG_TOP_SECTION:
> >        continue
> >
> > -    if not KEY_COMMAND in config[section]:
> > +    if not config.has_option(section, KEY_COMMAND):
> > +      print("No Entry '" + KEY_COMMAND + "' ... Skipping sections " +
> section)
> >        continue;
> >
> > -    command_type = config[section][KEY_COMMAND_TYPE] if
> KEY_COMMAND_TYPE in config[section] else section
> > +    command_type = config.get(section,KEY_COMMAND_TYPE) if
> config.has_option(section, KEY_COMMAND_TYPE) else section
> >
> > -    if KEY_PRE_COMMAND in config[section]:
> > -      subprocess.call(config[section][KEY_PRE_COMMAND].split())
> > -    data = measure(program_name, config[section][KEY_COMMAND].split(),
> command_type=command_type, revision=revision)
> > +    if config.has_option(section, KEY_PRE_COMMAND):
> > +      subprocess.call(config.get(section,KEY_PRE_COMMAND).split())
> > +    data = measure(program_name,
> config.get(section,KEY_COMMAND).split(), command_type=command_type,
> revision=revision)
> >
> >      log_file = os.path.join(dir_name, "%s_%s.log" % (output_file,
> section))
> >      db_file = os.path.join(dir_name, "%s.db" % output_file)
>
> > _______________________________________________
> > Dune-devel mailing list
> > Dune-devel at dune-project.org
> > http://lists.dune-project.org/mailman/listinfo/dune-devel
>
>
> --
> Prof. Dr. Christian Engwer
> Institut für Numerische und Angewandte Mathematik
> Fachbereich Mathematik und Informatik der Universität Münster
> Einsteinstrasse 62
> 48149 Münster
>
> E-Mail  christian.engwer at uni-muenster.de
> Telefon +49 251 83-35067
> FAX             +49 251 83-32729
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dune-project.org/pipermail/dune-devel/attachments/20130814/1f5ec0de/attachment.htm>