<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">2013/8/17 Christian Engwer <span dir="ltr"><<a href="mailto:christian.engwer@uni-muenster.de" target="_blank">christian.engwer@uni-muenster.de</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Miha,<br>
<br>
it seems you are making good progress with the build-system<br>
integration :-)<br></blockquote><div>I don't think it's that good. The CMake integration works well, and is pretty intuitive (just call add_performance_test(<name> <target>)), but I still don't know how to do the same using automake. There is no equivalent (that I know of) of add_custom_target() or add_dependency(). <br>
<br>I've been thinking of using intermediate files and a wildcard target (pattern rule), so that you would add something like<br><br></div><div>perftest_myexample: myexample perftest_myexample.log<br><br></div><div>and there would be a general rule<br>
<br></div><div>%_perftest.log:<br></div><div> <run the test here><br><br></div><div>If will try such a rule today, unless you have a better suggestion. <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
It would be great, to have a short readme in doc, which explains, how<br>
the tests interact with the buildsystem and what the user is supposed<br>
to do.<br></blockquote><div>Yes, of course. But first the interaction should work :) <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Then I have some additional comments...<br>
- all test should be performed in the build directory, this is then<br>
also the place where results should be stored.<br>
- it would be convenient to have all results in a seperate<br>
subdirectory (of the build dir) and then the html files again in a<br>
sub dir.<br></blockquote><div>I agree, will make it that way. <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
- in order to be able to integrate the perftests with the autobuild<br>
services, it is necessary to seperate data aquisition and<br>
storage. Therefore it ise necessary to write some kind of log files<br>
which can then be transferred to the server (or is this already<br>
integrated?).<br></blockquote><div>This is somewhat separated right now. There are separate python modules, but so far the main script calls them all in sequence. <br></div><div>Log files are generated in any case, and are optionally deleted afterwards. There shouldn't be problems there. <br>
</div><div>The flow of data is like this: (measure -> logfile), (logfile -> sqlite database), (database -> HTML). <br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
- when we have a large set of tests, an overview file would help.</blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Ciao<br>
<span class="HOEnZb"><font color="#888888">Christian<br>
</font></span><div class="im HOEnZb"><br>
On Thu, Aug 15, 2013 at 05:06:39PM +0200, Miha Čančula wrote:<br>
</div><div class="HOEnZb"><div class="h5">> Apparently Chrome is more security-consious than Firefox and refuses to<br>
> load local files via XMLHttpRequest. If I put it in the same file, then<br>
> graphs work (at least in Chromium, don't have any other browsers to check).<br>
> I just pushed the change.<br>
><br>
> I'm tackling CMake right now, I hope to get at least the compiler and flags<br>
> today.<br>
><br>
><br>
> 2013/8/14 Christian Engwer <<a href="mailto:christian.engwer@uni-muenster.de">christian.engwer@uni-muenster.de</a>><br>
><br>
> > Hi Miha,<br>
> ><br>
> > > 2.) Using the buildsystem to call the measurement routine. I agree, this<br>
> > is<br>
> > > probably the best way to get all the needed compiler information. Of<br>
> > > course, it should be easy to use, so a single macro or target (per<br>
> > measured<br>
> > > program) should be used. I will try to do it in automake, but would it be<br>
> > > ok if I first write a CMake macro, get the details worked out, and then<br>
> > do<br>
> > > the same with autotools? Considering the use of two build systems, I this<br>
> > > part would be a pretty thin wrapper, just getting the compiler info and<br>
> > > passing it to python.<br>
> ><br>
> > This would be viable... in particular this would mean that you create<br>
> > a suitable ini file during the configure/cmake run.<br>
> ><br>
> > > 3.) Split HTML files. I agree, this would be a good idea. I don't know if<br>
> > > it works for you, but I added table filtering, so you can choose to only<br>
> > > display Run, Compile or all measurements. I would keep the total file,<br>
> > but<br>
> > > I can also generate type-specific ones.<br>
> > ><br>
> > > 4.) No graphs in resIults. I don't know why that should be. Yes, the data<br>
> > > is in a separate CSV file, but in my testing, it was always loaded<br>
> > > correctly. There might be some errors in handling paths, I'll check when<br>
> > I<br>
> > > get home.<br>
> ><br>
> > I think the problem is due to security reasons... I tried with<br>
> > chromium und epiphany, which are both webkit based and I assumed that<br>
> > the browser does not allow to reload local files from java-script. As<br>
> > many users will have chrome as browser, we must be able to hanle<br>
> > this and thus I suggest to include the csv again into the html file.<br>
> ><br>
> > > 5.) Automatic testing. The program now parses a configuration file<br>
> > > (perftest.ini) and runs the tests described in it. This means you still<br>
> > > have to manually run a command, but it's only one command per run. I<br>
> > > suppose "make perftest" is easier to remember than "perftest.py<br>
> > > perftest.ini", so some integration with the buildsystem is needed.<br>
> > However,<br>
> > > I wouldn't run performance tests automatically every time the program is<br>
> > > compiled, this would take too much time.<br>
> ><br>
> > No, we should not run the test every time. I would expect that you can<br>
> > enable/disable the automatic data collection. Using a single "make" is<br>
> > not really good, as we want to have seperate data for all our<br>
> > compilation units.<br>
> ><br>
> > In my vision, we constantly collect data (when enables, e.g. on the<br>
> > build-server) and then we have individual html files per make<br>
> > lib/exe. Within this html file we can now select which data to look<br>
> > at, using the filter approach. This will limit the number of files<br>
> > required and make sure they don't grow too far.<br>
> ><br>
> > > 6.) My progress. I added statistics for finding outliers, you may notice<br>
> > in<br>
> > > the results some rows are different colors depending on their distance<br>
> > from<br>
> > > the mean. Currently all points with at least 1-sigma deviation are<br>
> > marked,<br>
> > > but I don't think that should be the case. There are more rigorous<br>
> > > definition of outliers. The documentation uses Doxygen,<br>
> > > but I haven't converted all the actual docstrings yet. The graphs show<br>
> > both<br>
> > > memory consumption and time spent, and there are separate graphs for<br>
> > > compilation and running.<br>
> ><br>
> > I guess I will see this in detail, once the data is included again :-)<br>
> ><br>
> > Christian<br>
> ><br>
<br>
</div></div><div class="HOEnZb"><div class="h5">--<br>
Prof. Dr. Christian Engwer<br>
Institut für Numerische und Angewandte Mathematik<br>
Fachbereich Mathematik und Informatik der Universität Münster<br>
Einsteinstrasse 62<br>
48149 Münster<br>
<br>
E-Mail <a href="mailto:christian.engwer@uni-muenster.de">christian.engwer@uni-muenster.de</a><br>
Telefon <a href="tel:%2B49%20251%2083-35067" value="+492518335067">+49 251 83-35067</a><br>
FAX <a href="tel:%2B49%20251%2083-32729" value="+492518332729">+49 251 83-32729</a><br>
</div></div></blockquote></div><br></div></div>