Multi-version metric analysis

From FetchWiki

Jump to: navigation, search

This use case attempts to support quality assurance personnel (and potentially also managers) in objectively analyzing quantitative differences among multiple versions of a given software system. E.g., this use case supports the following activities:

  • Evaluation of the trend of various maintainability indicators (complexity, coupling, cohesion).
  • Confirmation of the effect of refactoring efforts (e.g., targeting specific outliers).

Contents

[edit] Characteristic Information

  • Goal in Context: User requests an overview of quantitative differences among multiple versions of a given software system.
  • Preconditions: an abstract model is available in multiple versions
  • Success End Condition: User receives plots and summary data comparing the versions.
  • Failed End Condition: User receives message explaining failed attempts.
  • Primary Actor: Q&A (typically Tool Guru)
  • Trigger: User launches generation script.

[edit] Main Success Scenario

put here the steps of the scenario from trigger to goal delivery, and any cleanup after

1. User computes a set of metrics using an RML script (e.g., the Chidamber and Kemerer metric suite using computeCKmetrics.rml) for each abstract model of the different versions of the system. The user stores the output files in an empty directory.

2. The user executes the aggregateVersionMetricData.py script, which aggregates the metric data on all versions in a single file per metric. Once again, the user stores the output files in an empty directory.

3. The user executes the reportVersionMetrics.R script, which creates boxplots for each metric across the multiple versions. Aside a normal boxplot including outliers, a separate boxplot is generated for each metric, limiting the range of the boxplot to the first outlier.

[edit] Example Scenario

In this example scenario, we start with two versions of the TeamSpirit case, for which the RSF model is present in fetch/samples/teamspirit/teamspirit.rsf. Instead of generating a second version, we will just duplicate this RSF model, thereby comparing one version against the same version.

We assume that you are located in the fetch directory. We use the Linux x86 binaries for the tools used.

mkdir ~/versionMetrics
./bin/crocopat-linux-x86 scripts/rml/computeCKmetrics.rml < samples/teamspirit/teamspirit.rsf > ~/versionMetrics/v0.1.txt
cp ~/versionMetrics/v0.1.txt ~/versionMetrics/v0.2.txt # We compare one version against the same version.
mkdir ~/metricsData
python scripts/postprocessing/aggregateVersionMetricData.py ~/versionMetrics ~/metricsData

Output:

Versions: ['v0.1', 'v0.2']
Exporting cbo data to file /home/bdubois/metricsData/cbo.txt... [done]
Exporting lcom data to file /home/bdubois/metricsData/lcom.txt... [done]
Exporting noc data to file /home/bdubois/metricsData/noc.txt... [done]
Exporting rfc data to file /home/bdubois/metricsData/rfc.txt... [done]
Exporting dit data to file /home/bdubois/metricsData/dit.txt... [done]

We then continue:

mkdir ~/plots
export INPUT_DIR=~/metricsData
export OUTPUT_DIR=~/plots
R --slave --no-save < scripts/R/reportVersionMetrics.R

You will then find out that the directory ~/plots consists of the following files (for each metric):

  • <metric-name>.normalRange.ps (in case there are high outliers)
  • <metric-name>.ps

[edit] Extensions

put here the extensions, one at a time, each referring to the step of the main scenario

[edit] Sub-variations

put here the sub-variations that will cause eventual bifurcation in the scenario

step or variation number - list of sub-variations

[edit] Related Information

  • Priority: Optional
  • Frequency: release-wise

[edit] Open Issues

list of issues about this use case awaiting decisions

[edit] Status

The scripts mentioned above automate each of the steps. The sequence of steps still remains a manual procedure.

[edit] Scripts

The scripts can be mastered as follows.

[edit] computeCKmetrics.rml

input 
RSF model (of a single version of the software system)
output 
ASCII file containing a table of metric values for each class.
Sample output of computeCKmetrics.rml
ID name cbo lcom noc rfc dit
3976 'Foo::Bar' 1 18 0 5 0
3976 'Foo::Baz' 1 12 0 4 0
Weighted Methods per Class (WMC) (currently missing)
Measure of the total complexity implemented by the methods in that class. Forces:
  • Maintenance: The larger the number of methods and their complexity, the more time and effort is required for maintenance.
  • Reuse: The larger the number of methods and complexity, the less likely a class will be reused.
Depth of Inheritance Tree (DIT) 
Measure of how many ancestor classes can potentially affect this class. Hierbij wordt een afweging gemaakt tussen volgende forces:
  • Complexity: The deeper a class is in the hierarchy, the greater the number of methods it is likely to inherit, making it more complex to predict its behavior.
  • Reuse: The deeper a particular class is in the hierarchy, the greater the potential reuse of inherited methods.
Number of Children (NOC) 
Measure of how many subclasses are going to inherit the methods of the parent class. Forces:
  • Reuse: The greater the number of children, the greater the reuse.
  • Testing: A class with a large number of children may require more testing of the methods in that class.
Coupling between Object Classes (CBO) 
Measure of how many classes are coupled via method or instance variable interaction. Forces:
  • Reuse: The more independent a class is, the easier it is to reuse it in another application.
  • Maintenance: The larger the number of couples, the higher the sensitivity to changes in other parts of the design.
  • Testing: The higher the inter-object class coupling, the more rigorous testing needs to be.
Response for a Class (RFC) 
Measure of the set of potentially executed methods in response to a message received by an object of that class. Forces:
  • Testng: If a larger number of methods can be invoked, the testing and debugging becomes more complicated.
  • Complexity: The larger the number of methods that can be invoked from a class, the greater the complexity.
Lack of Cohesion in Methods (LCOM) 
Measure of the absence of similarity among methods with regard to attribute usage. This metric is an inverse cohesion metric: it measures the absence of cohesion. Forces:
  • Complexity: high lack of cohesion increases complexity.


[edit] aggregateVersionMetricData.py

input 
  1. inputDir -- directory containing ASCII files for metric-tables generated from computeCKmetrics.rml
  2. outputDir -- directory in which the output will be stored
The names of the files in the inputDir are important:
  • The extension needs to be .txt. All other files are ignored.
  • The name of the file, excluding the .txt. extension will be used as the name of the version. Accordingly, we advise file names similar to v0.1.txt.
output 
ASCII file for each metric, containing a table of metric values for each version.
Sample output of aggregateVersionMetricData.rml, e.g., cbo.txt
v0.1 v0.2 v0.3
5 6 7
4 4 5
The order of the various version-columns is random. Accordingly, one may not assume that the data on a row represents multiple versions of the same class.

[edit] reportVersionMetrics.R

input 
None. However, the following environment variables need to be available:
  1. INPUT_DIR -- directory containing ASCII files for metric-tables generated from computeCKmetrics.rml
  2. OUTPUT_DIR -- directory in which the output will be stored
The names of the files in the INPUT_DIR are important:
  • The file name represents the name of the metric value, with a ".txt" extension.
output 
Two boxplots for each metric:
  1. Boxplot indicating all outliers. Often, however, statistics cannot be interpreted using this boxplot (e.g., median, quartiles), since outliers will typically be so large that they minimize the visibility of the actual box plot.
  2. Boxplot with a range limited to [0, minOutlier], with minOutlier representing the minimum outlier across all versions.
Personal tools