performance profiling infrastructure #67

christoff-buerger · 2016-09-13T20:13:22Z

Systematic Performance measurements are helpful to evaluate the automatic attribute caching of RACR, pinpoint improvement opportunities and track performance regressions. Scheme distributions often provide tools for detailed code profiling, like function execution and garbage collection times. The parameterisation of example applications for different usage scenarios with different performance-load characteristics like size of abstract syntax tree, number of attributes, size of attribute caches or frequency and impact of rewrites still has to be provisioned however. A generic performance profiling infrastructure, which enables the generation of test series according to user-specified program arguments and the classification and visualisation of respective measurement results, would be very useful.

The functionalities to provide are:

a configuration language for the specification of measurement-templates defining the inputs and measured outputs, and their respective names and types, a valid measurement must satisfy
a test series generator executing and measuring RACR-programs according to given value ranges; generated test series have to be repeatable
a measurement browser enabling the collection and filtering of previous measurements in measurement-compilations; the different measurements of repeated test series have to be combined in an empirically and statistically meaningful way
a visualisation generator to plot measurement-compilations; generated plots have to be labeled according to argument types and names
an integrity checker to validate stored measurement-compilations against configuration-templates

…ed make-table script adding command line argument checks. Also moved all profiling scripts in the profiling directory.

…ed make-measurements script adding command line options to select a profiling configuration and save a rerun script. Added sanity checks for provided arguments. Ensured, that temporary resources are properly created and cleaned up.

… that /dev/null as rerun script for make-measurements works (even as root). Also added the initialisation of generated rerun scripts.

…ecks for the types and values of measurement parameters (make-measurements script); integer value parameters now trigger iteration-step and -adjustment configurations and respective sanity checks of provided values.

…ript to parse profiling configurations and extended them with measurement result configuration section.

…surements.bash now handles not-a-number arguments.

…pport to specify an execution script to profiling configurations, the parsing of such extended configurations and the execution and logging of execution results in make-measurements.bash.

…ror column, indicating exceptions while profiling, to measurement tables. Respectively, make-measurements catches such exceptions. Profiling therefore now is robust; unexpected errors of profiled programs do not disrupt measurement tables and are properly logged.

… flag to make-measurements; the flag forces the script to fail if any measurement fails. Otherwise, the script executes all measurements, even if one fails.

…rofiling for atomic petrinets.

…age comment and error checks to run script for atomic petrinets profiling .

…ofiling scripts into separate directory (profiling-scripts) and shortened their names (make-measurements.bash -> measure.bash, make-table.bash -> record.bash, parse-profiling-configuration.bash -> configure.bash). The recording script now fixes the measurement table it uses in case the result pipline is premature empty like in case of user aborted measurements (for example via CTRL-C).

…abled analysis caching as parameter to the atomic petrinets profiling case study; it now can be turned on and off. Likewise, enabled analysis caching can now be triggered for the fUML Activity Diagrams example.

…tract.bash script, which can be used to extract rows from measurement tables according to constraints (work in progress).

…progress on extract script; added actual extraction code and split it in static and generated Scheme part.

…progress on extract script. It now also queries for error status, measurement date and result extraction operators. Several operators can be given for each column. The actual extraction script is generated as a temporary file, automatically deleted when finished.

…progress on extract script. All global extrema are extracted (if several rows share a global extrema, they are all preserved). Layout of measurement tables simplified to ease automatic processing and generation.

…rking version of extract script. Duplicated entries are now automatically deleted. Local extrema are considered equal if all their columns, ignoring measurement dates, are the same. This implies, that rows are extracted according to a local extrema, if, and only if, the value of the extrema column of a row is the min/max within the set of all rows whose column values are equal except the measurement date column. Global extrema do not have such a column-uniqueness selection criteria. Rows are extracted according to a global extrema when the value of their respective extrema column is the min/max within the set of ALL rows. In summary, a row is extracted if it satisfies ALL conditions of ALL its columns and, if there are any extrema selectors, has an extrema.

…g in extraction script regarding the selection of local extrema. Extraction is tested and working. Further, the measurement date is now the first parameter, whereas the status (aborted, failed, succeeded) is the first result; the columns of measurement tables are reordered accordingly.

…ucture (i.e., everything except visualisation: configuration language, test series generator, measurement browser).

… least one is given; it now also creates containing directories of the measurements table given via -t flag if such do not exist yet. Recording of measurement results now checks that the table given as target has -- in case it already exists -- a correct header considering the used profiling configuration.

…ectness w.r.t. the given profiling configuration (since extraction scripts already check all given source tables and recording will always produce correct tables, table integrity is always guaranteed even in in case of measurement abortion, errors or exceptional termination while measuring etc). The golden rule is: Tables are only updated iff everything is correctly configured, i.e., correct execution can be guaranteed, and resulting tables can be guaranteed to satisfy the given configuration.

…ew measurements or extraction results; table-integrity can also be checked explicitly via a new 'check-tables.bash' script. In case of errors, tables are never changed.

…s and the recording callee is now handled correctly and more robust and not anymore based on 'hopeful long enough sleep times to let the recording finish before killing it'.

…script abortion timing.

…ork in progress).

…ite the first few elements if there are less than already existing new entries read (in case of measure.bash this was a bug because the array is read in a loop).

…Thh:mm:ssZ) format; this enables lexical ordering and standard conformant third party processing of dates.

christoff-buerger added enhancement high user interface maintainability labels Sep 13, 2016

christoff-buerger added this to the RACR version 0.8.5: infrastructure edition (fortitudinem per suburbium et regionem) milestone Sep 13, 2016

christoff-buerger self-assigned this Sep 13, 2016

christoff-buerger added a commit that referenced this issue Sep 18, 2016

Working on issue #67 (performance profiling infrastructure): Refactor…

c2beb19

…ed make-table script adding command line argument checks. Also moved all profiling scripts in the profiling directory.

christoff-buerger added a commit that referenced this issue Sep 20, 2016

Working on issue #67 (performance profiling infrastructure): Ensured,…

c71f18c

… that /dev/null as rerun script for make-measurements works (even as root). Also added the initialisation of generated rerun scripts.

christoff-buerger added a commit that referenced this issue Oct 7, 2016

Working on issue #67 (performance profiling infrastructure): Added sc…

52e27f6

…ript to parse profiling configurations and extended them with measurement result configuration section.

christoff-buerger added a commit that referenced this issue Oct 8, 2016

Working on issue #67 (performance profiling infrastructure): make-mea…

52edde3

…surements.bash now handles not-a-number arguments.

christoff-buerger added a commit that referenced this issue Nov 15, 2016

Working on issue #67 (performance profiling infrastructure): Set-up p…

7aad8ac

…rofiling for atomic petrinets.

christoff-buerger added a commit that referenced this issue Nov 17, 2016

Working on issue #67 (performance profiling infrastructure): Added us…

d239d5f

…age comment and error checks to run script for atomic petrinets profiling .

christoff-buerger added a commit that referenced this issue Nov 25, 2016

Working on issue #67 (performance profiling infrastructure): Added ex…

4936929

…tract.bash script, which can be used to extract rows from measurement tables according to constraints (work in progress).

christoff-buerger added a commit that referenced this issue Nov 29, 2016

Working on issue #67 (performance profiling infrastructure): Work in …

d0081a2

…progress on extract script; added actual extraction code and split it in static and generated Scheme part.

christoff-buerger mentioned this issue Oct 18, 2017

clear bundling of deployment scripts and generated deployment artefacts #82

Closed

christoff-buerger added a commit that referenced this issue Sep 16, 2019

#67: Finished text-based facilities of performance profiling infrastr…

72a513f

…ucture (i.e., everything except visualisation: configuration language, test series generator, measurement browser).

christoff-buerger added a commit that referenced this issue Mar 1, 2020

#67: Measurement dates are now in UTC standard format.

cc017ab

christoff-buerger mentioned this issue Mar 2, 2020

script for temporary file creation #89

Closed

christoff-buerger removed the high label Mar 2, 2020

christoff-buerger added a commit that referenced this issue May 8, 2020

#67: Robustness improvements taking care of absolute corner cases of …

2397e6e

…script abortion timing.

christoff-buerger added a commit that referenced this issue May 14, 2020

#61,#67,#89: Fixed spelling errors.

3f2c8fd

christoff-buerger added a commit that referenced this issue Sep 24, 2021

#67: First steps on providing profiling plots in 2D/3D with plotly (w…

73f6bf1

…ork in progress).

christoff-buerger added a commit that referenced this issue Oct 3, 2021

#67: First steps on providing profiling plots in 2D/3D with plotly (w…

ee80184

…ork in progress).

christoff-buerger mentioned this issue Oct 15, 2021

script for atomic file locking, enabling atomic script execution and exclusive resource allocation (mutex) #93

Closed

christoff-buerger mentioned this issue Oct 25, 2021

thread-safe RACR-library/Scheme program execution #96

Closed

christoff-buerger added a commit that referenced this issue Oct 27, 2021

#67: Unified spelling (measurementS table).

9b75496

christoff-buerger mentioned this issue Oct 29, 2021

thread-safe performance profiling infrastructure #97

Closed

christoff-buerger added a commit that referenced this issue Sep 15, 2023

#67: Changed name of used test table.

5cb29b9

christoff-buerger added a commit that referenced this issue Dec 5, 2023

#67: Write measurement date in ISO 8601-1:2019/Amd 1:2022 (YYYY-MM-DD…

d6f678d

…Thh:mm:ssZ) format; this enables lexical ordering and standard conformant third party processing of dates.

christoff-buerger mentioned this issue Feb 11, 2024

test suite should reuse performance profiling scripts for logging and performance testing #101

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

performance profiling infrastructure #67

performance profiling infrastructure #67

christoff-buerger commented Sep 13, 2016 •

edited

Loading

performance profiling infrastructure #67

performance profiling infrastructure #67

Comments

christoff-buerger commented Sep 13, 2016 • edited Loading

christoff-buerger commented Sep 13, 2016 •

edited

Loading