Aspect-Oriented Programming evaluated: A Study on the Impact that Aspect-Oriented Programming can have on Software Development Productivity

Table of Contents

Abstract

Zusammenfassung

Directory of Figures

Directory of Tables

Directory of Listings

Table of Contents

1. Introduction

2. Motivation and Problem Description

3. Experiment Background
3.1. A short summary on Empiricism and Experimentation
3.1.1. What is meant by Empiricism?
3.1.2. Research and Experimentation methods
3.1.2.1. Case Studies or Benchmarks
3.1.2.2. Field Studies
3.1.2.3. Controlled Experiments
3.1.2.4. Polls
3.1.2.5. Meta-Studies
3.1.3. Empirical research in Software Engineering – Specifics and Dangers
3.2. Aspect-Oriented Programming
3.2.1. Aspect-Orientation in General
3.2.2. AspectJ – a short Introduction

4. The Experiment
4.1. Experiment Buildup
4.1.1. The planned Course of the Experiment
4.1.2. The Questionnaire
4.1.3. The Hard- and Software used in the Experiment
4.1.3.1. The Application used for Editing
4.1.3.2. The Development Environment and Hardware
4.1.4. The Tasks
4.1.4.1. Task1: The Logging Task
4.1.4.2. Task2: The Parameter Null Task
4.1.4.3. Task3: The Synchronization Task
4.1.4.4. Task4: The Check Player Argument Task
4.1.4.5. Task5: The Notify Observers Task
4.1.4.6. Task6: The Observers Null Check Task
4.1.4.7. Task7: The Refresh Constraint Task
4.1.4.8. Task8: The Label Value Check Task
4.1.4.9. Task9: The Current Level Check Task
4.2. Implementation of the Experiment

5. Experiment Analysis and Results
5.1. Data Processing and Preparation
5.2. Data analysis and presentation
5.2.1. The Logging Task
5.2.2. The Parameter Null Task
5.2.3. The Synchronization Task
5.2.4. The Player Check Task
5.2.5. The Notify Observers Task
5.2.6. The Observers Null Task
5.2.7. The Refresh Constraint Task
5.2.8. The Label Value Check Task
5.2.9. The Level Check Task
5.2.10. Results of the Development Times and Descriptive Statistics
5.2.11. Statistical Tests on the Results
5.2.12. Doing a Break-Even Analysis
5.2.13. Participant Grouping

6. Discussion
6.1. Thoughts on Validity
6.1.1. Internal Validity
6.1.2. External Validity
6.2. General Discussion

7. Related Work

8. Conclusion

9. Appendix
9.1. The questionnaire (German)
9.2. The aspect-oriented task descriptions (German)
9.3. The object-oriented task descriptions (German)

10. References

Directory of Figures

Figure 4-1 – A simplified UML Diagram of the target application

Figure 5-1 – A sample diagram showing progress per time in seconds depending on parameter count of methods for participant 16 and the Logging task

Figure 5-2 – A sample diagram showing overall unaltered progress of participant 16 on the Logging task

Figure 5-3 A sample diagram showing progress per time in seconds depending on parameter count of methods for participant 16 and the Logging task after data cleansing

Figure 5-4 sample diagram showing overall unaltered progress of participant 16 on the Logging task after data cleansing

Figure 5-5 - A sample diagram showing overall unaltered progress of participant 16 on the Parameter Null task

Figure 5-6 - A sample diagram showing overall unaltered progress of participant 16 on the Synchronization task

Figure 5-7 - A sample diagram showing overall unaltered progress of participant 16 on the Player Check task

Figure 5-8 - A sample diagram showing overall unaltered progress of participant 16 on the Notify Observers task

Figure 5-9 - A sample diagram showing overall unaltered progress of participant 16 on the Observers Null task, using the copy and paste approach

Figure 5-10 - Figure 5-11 - A sample diagram showing overall unaltered progress of participant 2 on the Observers Null task, using the find and replace approach

Figure 5-12 - A sample diagram showing overall unaltered progress of participant 16 on the Refresh Constraint task

Figure 5-13 - A sample diagram showing overall unaltered progress of participant 16 on the Label Value Check task

Figure 5-14 – A sample diagram showing overall unaltered progress of participant 16 on the Level Check task

Figure 5-15 - Sum of times for the object-oriented and aspect-oriented solutions and the ratio

Figure 5-16 – A scatter diagram showing development times for all participants using aspect-orientation on the larger three tasks

Figure 5-17 - A scatter diagram showing development times for all participants using object-orientation on the larger three tasks

Figure 5-18 - A scatter diagram showing development times for all participants using aspect-orientation on the smaller six tasks

Figure 5-19 - A scatter diagram showing development times for all participants using object-orientation on the smaller six tasks

Figure 5-20 – Normalized diagram for participant 16 and the logging task

Figure 5-21 – Scatter diagram for the distribution of the break-even values for all tasks

Figure 9-1 – Page 1 of the questionnaire

Figure 9-2 – Page 2 of the questionnaire

Directory of Tables

Table 5-1 – Regression equations and determination coefficients for all participants for the Logging task using unaltered data

Table 5-2 Regression equations and determination coefficients for all participants for the Logging task using cleansed data

Table 5-3 - Regression equations and determination coefficients for all participants for the Parameter Null task using unaltered data and cleansed data

Table 5-4 - Regression equations and determination coefficients for all participants for the Synchronization task using unaltered data and cleansed data

Table 5-5 - Regression equations and determination coefficients for all participants for the Player Check task using unaltered data

Table 5-6 - Regression equations and determination coefficients for all participants for the Notify Observers task using unaltered data

Table 5-7 - Regression equations and determination coefficients for all participants for the Observers Null task using unaltered data

Table 5-8 - Regression equations and determination coefficients for all participants for the Refresh Constraint task using unaltered data

Table 5-9 - Regression equations and determination coefficients for all participants for the Label Value Check task using unaltered data

Table 5-10 - Regression equations and determination coefficients for all participants for the Level Check task using unaltered data

Table 5-11 – Development times for every participant and every task, measured in seconds

Table 5-12 – Differences between aspect-oriented and object-oriented development times

Table 5-13 – Descriptive statistics for the development times of all participants

Table 5-14 – Ratios of aspect-oriented to object-oriented development time

Table 5-15 – Statistical functions on the ratio data

Table 5-16 – Results of the statistical tests for normal distribution

Table 5-17 – Results of the Wilcoxon-test

Table 5-18 – Break-even values for all participants and all tasks

Table 5-19 – Descriptive Statistics on the break-even values

Table 5-20 – Answers to the questionnaire

Table 5-21 – Three tables showing the developer categorizations

Table 5-22 – Descriptive statistics values for the participant groupings

Table 5-23 – Results for normal distribution tests for the advanced group

Table 5-24 – Results for the paired T-tests on the tasks of the advanced group that are supposedly normally distributed

Table 5-25 – Results for the Wilcoxon-test on all tasks for the advanced group

Table 5-26 - Results for normal distribution tests for the average group

Table 5-27 - Results for the paired T-tests on the tasks of the average group that are supposedly normally distributed

Table 5-28 - Results for the Wilcoxon-test on all tasks for the average group

Table 5-29 - Results for normal distribution tests for the novice group

Table 5-30 - Results for the paired T-tests on the tasks of the novice group that are supposedly normally distributed

Table 5-31 - Results for the Wilcoxon-test on all tasks for the novices group

Directory of Listings

Listing 3-1 – An example aspect in AspectJ Syntax

Listing 4-1 – An example statement of the Logging Task using only object-oriented programming

Listing 4-2 – A possible AspectJ Solution for the Logging Task

Listing 4-3 – The null-reference check using pure object-orientation

Listing 4-4 – A possible AspectJ solution for the null-reference checks

Listing 4-5 – The code template for the synchronization task

Listing 4-6 – The AspectJ solution for the synchronization task

Listing 4-7 – An example object-oriented solution for the check player argument task

Listing 4-8 – Possible AspectJ solution for the check player argument task

Listing 4-9 – An object-oriented example for the notify observers task

Listing 4-10 – An AspectJ solution for the notify observers task

Listing 4-11 - An object-oriented example for the observers null check task

Listing 4-12 - An AspectJ solution for the observers null check task

Listing 4-13 –Possible solution for the refresh constraint task using object-orientation

Listing 4-14 - An AspectJ solution for the refresh constraint task

Listing 4-15 - Possible solution for the label value check task using object-orientation

Listing 4-16 - An AspectJ solution for the label value check task

Listing 4-17 - Possible solution for the current level check task using object-orientation

Listing 4-18 - An AspectJ solution for the current level check task

Abstract

This book describes and evaluates a controlled experiment on aspect-oriented programming. The experiment was designed to make the overall performance of developers using object-orientation and aspect-orientation on a number of tasks comparable. The primary focus of the experiment laid on whether aspect-orientation has a positive impact on the development time when comparing its performance with the performance of the object-oriented approach on the same task.

Zusammenfassung

Dieses Buch beschreibt und wertet ein kontrolliertes Experiment aus. Das Experiment wurde aufgesetzt, um die allgemeine Arbeitsleistung von Entwicklern bei der Nutzung des objektorientierten und des aspektorientierten Ansatzes für eine Anzahl an Aufgaben vergleichbar zu machen. Dabei wurde der primäre Fokus darauf gelegt messbar zu machen, ob die Aspektorientierung einen positiven Einfluss auf die Entwicklungszeit im Vergleich zur Objektorientierung bei der gleichen Aufgabe hat.

1. Introduction

“Time is money” is what many people state when considering the aspect of time in modern businesses. Many cost prediction models and actual prices for various products (especially in the area where human service and creativity are playing a major role) are based on time as a central factor. This applies to a large part of the software industry, where the time which developers need to finish the software is a critical factor in almost any software project. The technique of aspect-oriented programming could present a possibility to save a large amount of time, especially for redundant code in larger projects.

This work introduces a controlled experiment that analyzes the development costs in terms of additional development time, caused by the specification of redundant code in the object-oriented programming language Java in comparison to the aspect-oriented programming language AspectJ, which is essentially an add-on for the Java language. Chapter two describes the motivation and the background of the study, trying to argue on the importance of empirical research in this area. Chapter three summarizes some historical background about empiricism, the methods associated with today’s empirical research (like controlled experiments), and a short introduction to aspect-oriented programming and the AspectJ implementation of that technique. Chapter four explains the experiment’s setup, the specific tasks and their different solutions in aspect-oriented and object-oriented programming. Chapter five explains and presents most of the experiment results from the raw data to the aggregated forms and the statistical and explorative approaches that were done with the data. After a discussion of the experiments results and validity thoughts in chapter six, chapter seven summarizes some related work in the field of studies on aspect-orientation. Finally, chapter eight concludes this work.

2. Motivation and Problem Description

Aspect-oriented programming has been introduced as a major step forward in software development. Some even proposed that

"AOP could be the next step in the steady evolution of the OO paradigm, or perhaps it will evolve into a completely new paradigm independent of OO"^[1],

making it possible to solve difficult problems affecting larger parts of software in less time than it would take using exclusively procedural or object-oriented programming. From looking at its conception, it seems especially well-suited to avoid writing redundant and repeating lines of code which appear on different occasions in the program's code. The study described in this work was therefore focused on redundant code and repetitive tasks, where the progress can be measured easily in time.

All of this sounds like it is a great achievement for the field of software engineering, but as so often in the young computer science, it has still not been tried to strengthen these claims using empirical studies and controlled experiments. Regrettably, this does not only apply to aspect-orientation as a relatively new technique, but to nearly all software engineering methods and techniques. In software engineering, empirical research makes up only a very small amount of all research work and publications. This might be seen as a dangerous development when simultaneously looking at the great impact the field of computer science and business information systems has on everyday life. Basili^[2] and Tichy^[3] both argue in that direction. Today, vehicles, weapons, machines, air planes, space craft and nearly every other electrical or mechanical system is somehow controlled, observed or at least influenced by software. Modern technology has grown exponentially and towards being nearly completely software relied in such a short time span (about 30 years), that it might be considered extremely careless not to try to validate the use of software engineering methods, techniques, languages and products using empirical research. Not to mention the potential cost and loss of time that could be avoided by differentiating good from bad methods (even if results can only be based on the situation and context). For software development is still a process dominated by human creativity and brains and therefore costs for personnel make up the bulk of the development budgets (attempts on using industrial methods for producing software still have to prove their worth). A lot of time and money is wasted on projects that are never finished or finished way after deadline, on faulty software and the correction of these faults, on software that is not accepted by the client or user, not to mention potential risks and costs through security leaks and their exploitation. Unfortunately, there have been nearly no studies on the cost of writing redundant code either.

So to make an own bold claim: There is a huge lack of empirical research in software engineering to back up many assumptions that are made about methods and techniques. One reason might be that empirical research always consumes time and resources, most of the time more than literature study and collecting arguments does. Even for the object-oriented paradigm^[4], which has been around for quite some time now, there have been only a handful of studies and experiments trying to strengthen the object-oriented claim of being better suited as an approach to programming^[5]. The rationale behind this is often the statement that it fits the way the human brain works. Most people would agree that this argument makes sense and it definitely does. Still, some more studies backing that statement up need to be done.

Hence, this experiment was motivated by all the above grievances and the need for more empirical research. The main motivation was to try to back up or falsify the assumption that aspect-orientation decreases the time needed to write redundant code, but in a rather explorative matter, not trying to concentrate on rejecting or hardening one single hypothesis. One of the central questions was: When would aspect-orientation be the technique providing an advantage in time, and when would the plain object-oriented approach be the better choice and would it even be possible to tell at all? To go a little further, it was tried to find a break-even point depending on task complexity and redundancy, which could be taken as an estimated predictor for anyone trying to assess which technique to use in a given situation, which is why a very fine-grained and basic approach was taken. Another part of the motivation was that as a side-effect of the overall data collection procedure, a large set of data for the process and time consumption of writing redundant code in object-oriented Java was gathered and could be analyzed and used in further studies.

There are also other facets of development that aspect-orientation has an impact on. For example, aspects provide clean design separation and modularization of the so-called crosscutting concerns, which are parts of the software that cannot be cleanly encapsulated into one single class or component (because they crosscut multiple components and code needs to be inserted at these spots to have the crosscutting work as wanted). Modularization of crosscutting concerns can enhance readability, maintainability and flexibility of applications. But aspects also add an increasing amount of complexity to a program. Especially if aspects get tangled into each other, more than one aspect weaves itself into the same place of code and on top of that the dynamic features of aspects are used, which can be very confusing to write and understand. This can make debugging applications crosscut by many aspects a burden or even nearly impossible. Some of the studies mentioned in the related work chapter are concerned with these facets of aspect-orientation and (Highley, et al., 1999) provides a critical analysis of the aspect-oriented approach in general. Thus, this work will only focus on the code redundancy removal and time saving.

3. Experiment Background

3.1. A short summary on Empiricism and Experimentation

3.1.1. What is meant by Empiricism?

The term empiricism is derived from a Greek word meaning “based on experience” and is used in two different contexts: The philosophical and the scientific.

In philosophy, a theory of knowledge^[6] says that any real knowledge that can be gained has to be gathered by experiencing and perceiving things with the human senses. This ultimately implies that human beings cannot gain knowledge and ideas out of themselves just by “thinking”, which is the concept of another field of philosophy. These two schools of thinking are called empiricism (knowledge arises from experience) and rationalism (knowledge can be gained by reasoning). As a matter of fact, many would agree that they do not exist in the real world in their most radical interpretation, and as a discussion of them is far out of the scope of this thesis, they will not be covered deeper. Interested readers might want to take a look into philosophical literature on both matters.

The more common term of empiricism is used in the modern sciences, its meaning being derived and closely related to the philosophical term. It is used to characterize the methodology of research which tries to strengthen or falsify (it can never prove) scientific theories and hypotheses using experiments or observations of reality, which ultimately leads back to the philosophical notion of learning by experience and observation. Empirical research is especially widespread in the natural and social sciences, as well as in medicine and the pharmaceutical area of research.

3.1.2. Research and Experimentation methods

This chapter’s information is primarily based on the work of Prechelt^[7] (and the summary of Josupeit-Walter^[8] ), who concentrated his evaluation and description of empirical research methods on these methods fit to be used in software engineering. The social sciences use other and more detailed categorizations for these methods and are in many matters more precise in their description and implementation.^[9] Nevertheless, for this work, the focus will stay on the methods presented by Prechelt. Most of these research methods are based on observation, either through qualitative manual approaches or automated quantitative ones.

3.1.2.1. Case Studies or Benchmarks

Case studies are commonly used to evaluate tools or methods using a controlled but typical environment for their implementation. They mostly consist of one or more tasks or use cases the participants have to fulfill using certain methods or tools, sometimes to evaluate a single method or tool or sometimes to compare two different approaches. The results can be qualitative or quantitative, depending on the implementation of the study. Case studies, in contrast to controlled experiments (explained in more detail later), do not try to keep all possible variable factors constant, which means that all the factors influencing the outcome of the study can possibly have a great impact on the results and make them very indeterministic or complicate the reconstruction of their development because of the many interdependencies. All the same, case studies are still useful for a rough and efficient evaluation of certain approaches or technologies.

To summarize, their advantages lie in their ease of implementation and general application possibilities, while their drawbacks lie in their sometimes unreliable results possibly not giving insights or conclusions.

A special case of case studies are benchmarks, which are standardized case studies which only result in quantitative data. Benchmarks are commonly used to create results that can be compared directly to other implementations of the same benchmark (as it is done in many hardware tests, where a benchmark program is run on computers with different hardware to make their overall performance or the performance of a single device comparable) or to create template data that represents a threshold which certain devices, tools or methods have to fulfill for quality testing.

3.1.2.2. Field Studies

Unlike case studies, benchmarks or controlled experiments, field studies are implemented in the field, meaning real software projects in the industry and are designed as an accompanying observation of specific factors, processes, behaviors or situations in these projects. Field studies try not to influence the observed projects and processes as to not distort the results. They have the advantage that they can be used to observe even complex situations, which would be too time-consuming or complicated to implement in a controlled environment, or if the scientists want to observe unaltered real life situations. But these advantages lead to a drawback comparable to that of case studies, but in a much stronger way: Their results are really hard to transform into a general hypothesis and the whole field study description can get very complex because of the complex circumstances.

3.1.2.3. Controlled Experiments

Of all research methods, controlled experiments try to exert the most control on the experiment’s implementation and circumstances. In the optimal case, only one or a few factors are left open as variables, everything else is kept as a constant. These experiments are the hardest to design and implement, as they require thorough planning and disciplined implementation. The setup is generally well defined and only these factors which are the focus of the experiments observations are kept variable. Controlled experiments therefore have a high validity and can easily be reproduced many times (when following the exact setup and implementation), producing reliable and comparable results. Their biggest disadvantage is their large cost in time and work for preparation, buildup and evaluation.

3.1.2.4. Polls

Polls (sometimes called interviews) are easy to implement and to evaluate, and can therefore be used to evaluate facts over a large number of people. However, they suffer from results that are hard to interpret and can be very unreliable, as every submitted answer is a completely subjective rating or evaluation of the specific person. And as many persons tend to have large number of different opinions and views, polls tend to have a large variability in the range of answers, especially when open^[10] questions are used. Poll results tend to be more reliably the more persons are included and if the range of questioned persons is representative for certain groups (like an even ratio of software developers and project managers).

3.1.2.5. Meta-Studies

Meta studies take a number of other studies on a certain topic and try to evaluate whether there are differences or similarities between these studies’ results and whether there are certain gaps in all data and if unanswered questions remain. They tend to produce less work than most other studies, as anyone carrying out a meta-study mainly has to do investigative work on the results and essays of the original experiments’ executors. Their most important aspect is the resulting summary of a possible large field of other works, which can be used by other scientists to get an overview of the current level of research on the topic.

3.1.3. Empirical research in Software Engineering – Specifics and Dangers

As stated above, software development is still a process dominated by human creativity and brains. Its mechanisms therefore do still elude a complete understanding and are very hard to measure and capture through observation and data collection. As Prechelt has written in his book, there are many specifics of empirical research and controlled experiments to be considered in software engineering. He states that for many controlled experiments, the most important variable to control is the variation among all participants' approaches and performance on a problem (which is an especially big variation for programming or modeling tasks). The wide range of experience with modeling, programming, programming languages and development tools between software developers, which is the very nature of software development and still eludes any quantitative way of measuring it, makes the results of empirical experiments generally hard to predict or interpret. Empirical research in software engineering is still at its beginning and researchers are still far from being able to handle and control these variations in a way that would make them able to produce very reliable results in most situations. These are some of the reasons why computer scientists tend to stay away from empirical research (Tichy summarizes 16 excuses used most to avoid experimentation in software engineering in his paper that was already cited above: (Tichy, 1997)).

Even the object-oriented approach, which is currently the most used in industry and academics, has not been validated thoroughly^[11]. Some even argue that there are still problems in the idea of object-orientation.^[12]

3.2. Aspect-Oriented Programming

3.2.1. Aspect-Orientation in General

Object-oriented Programming has had an amazing triumphal procession in the past years, both in the academic world as well as in the industrial development practice. It still has its drawbacks and is sometimes not sufficient for solving a specific set of problems. In 1997, Kiczales and his colleagues published the paper (Kiczales, et al., 1997) which introduced aspect-oriented programming as a modified approach on solving specific problems in software development. The idea was that certain functional parts of software crosscut an application or at least large parts of it (like logging, which is the most worn example for aspect-orientation, tracing, security, synchronization or other functions). Today, these specific functions are commonly called crosscutting concerns. Using the object-oriented approach, developers had a hard time implementing these crosscutting concerns seamlessly into their programs, because their nature prevented a clean separation of concerns and ultimately lead to tangled and difficult to read code (imagine an example where each method call that had to be logged for debugging purposes needed a separate logging statement which had to be inserted manually into the code). This code was also very tough to maintain and change, as the calls to these functions were scattered across the rest of the code and one central code change to solve the problem (which is usually one of the main benefits of object-orientation, the encapsulation of functionality) was not possible. All these drawbacks lead to the idea of aspect-oriented programming, where the so called aspects replace the tangled and scattered fragments in the old code by one central isolated point, effectively modularizing the crosscutting concern in the code. For the logging example, this aspect could be given the task of calling the logging functionality (which might be a method of a class providing this function) on all occasions the developer wants it to. This makes it easy for the developer to have every single method call in the program be logged without having to insert logging statements into the code to log itself.

3.2.2. AspectJ – a short Introduction

AspectJ^[13] is the implementation of the aspect-oriented add-on for the well-known and widely used programming language Java. It provides language constructs and mechanisms that implement the aspect-oriented crosscutting concerns using aspects. The paper (Kiczales, et al., 2001) presents an overview of AspectJ and its constructs, as well as how to use them. Some of these mechanisms will be explained here, but for a deeper introduction, the AspectJ Tutorial of the AspectJ project team^[14] provides a more sophisticated resource.

AspectJ introduces the aspect language construct into Java, which is defined in practically the same way as a standard Java class and provides the language unit which encapsulates and modularizes crosscutting functionality. Kiczales and his colleagues differentiate the crosscutting mechanisms of AspectJ into dynamic crosscutting, meaning being able to run additional code at well-defined points during program execution, and static crosscutting, meaning the extension of existing types through new operations^[15]. There is some confusion concerning the meaning of the terms dynamic and static crosscutting, as some seem to use these terms in different contexts for different concepts, others use them the way Kiczales and his colleagues did. So it cannot be clearly stated whether these definitions of dynamic and static crosscutting are deprecated today. For reasons of simplicity, this AspectJ introduction focuses on the concepts which were originally meant by the notion of dynamic crosscutting: Running aspect code at well-defined points in the program.^[16]

These well-defined points are commonly called join points and can represent different occasions during program execution, like a method call, an exception being thrown or the reading of a class field. A set of join points is called a pointcut (like all method calls in a certain class). Another interpretation might be to say that a join point is an incarnation of a certain pointcut definition, somehow like the relation of an object to a class. The code that is to be executed on the execution of a pointcut is called advice. The following code example shows the different concepts and their syntax in AspectJ:

Abbildung in dieser Leseprobe nicht enthalten

Listing 3-1 – An example aspect in AspectJ Syntax

On a first look, the syntax of the aspect frame does not differ much from that of a class: It has a name, a package, can have imports and is defined the same as a class (it also gets its own source file), except for the keyword aspect that is used. An aspect can also have fields and methods, like the aspect from the example. These are the similarities between an aspect and a class, but more interesting are the aspect specific constructs: There is a pointcut definition called methodCall, which hooks itself onto all calls to public methods of the class MyClass which return a String. The method name, number of parameters and the parameter types are not relevant for this pointcut, as wildcards are used for the definition. Pointcuts use constructs called primitives, which represent certain points in program execution, like the call of a method (in the example, the call primitive is used). These wildcards and primitives make AspectJ a very powerful tool, interested readers should refer to the AspectJ documentation for more on wildcards and primitives as well as more AspectJ syntax. Hence, the pointcut stands for a set of join points in the program, specifically all occasions of calls to methods which return a String in the class MyClass. The last construct in the example is the advice, which represents the code to be executed on certain occasions. In this case, a line is printed to the console every time before a method which fulfills the primitive of the methodCall pointcut is executed, indicated by the before keyword. Another possibility would be to use the after or around keywords, which can run the advice code after or instead of the original method call.

The process of putting together the object-oriented and aspect-oriented code is called weaving, where the weaver-process inserts the connection to the aspect advice code at the designated join points. This definition of weaving is rather rudimentary, but shall suffice in the context of this work, as a more exact description and explanation can be found in (Hanenberg, 2005), which is generally a good source of information on aspect-orientation concepts and background.

4. The Experiment

4.1. Experiment Buildup

4.1.1. The planned Course of the Experiment

The experiment in question was planned as a group of small sized programming tasks for the participants, consisting of two main assignments which each consisted of the same nine tasks. The nine tasks were designed so that each of them had different variables to influence their editing, which are described separately for each task below. Each of these assignments had to be fulfilled using the plain object-oriented approach (as the control language) as well as using only an aspect to solve the same problems. This means all participants had to do all nine tasks twice, some started with the object-oriented assignment, some with the aspect-oriented and then, after finishing, had to solve the same nine tasks using the other technique. It was randomly chosen who had to start with which assignment, but it was made sure that an even number of participants started with each. When using object-oriented programming, they were not allowed to use an aspect and while using aspect-oriented programming, they were not allowed to modify the original code and were permitted to modify their aspect only. The object-oriented tasks could only be solved by writing heavily redundant code, as can be seen in the descriptions of the specific tasks below. All participants, no matter which assignment they were given first, took part in a short AspectJ Tutorial of about 60-90 minutes in which they could try out some example exercises to get used to AspectJ syntax and handling. The tutorial concentrated only on a practical introduction to these parts and concepts of AspectJ which were needed to solve the tasks in the study while completely ignoring all remaining concepts of AspectJ. All in all, the experiment was planned to take each participant approximately 5 hours to complete, but no hard time limit was set.

4.1.2. The Questionnaire

Before beginning the study, all participants had to fill out an electronic questionnaire where they had to self-assess their skills and experience through various questions, mostly on a range from one to six. It included questions about general programming skills, Java, Eclipse or AspectJ experience and furthermore contained open slots for the participants to fill in any additional experience they had with different programming languages or techniques like logical programming. The data from the questionnaire was meant to be used later when trying to find out if specific previous knowledge influenced the participants' progress in the study. The whole original questionnaire can be found in the appendix.

4.1.3. The Hard- and Software used in the Experiment

4.1.3.1. The Application used for Editing

A self-written game designed explicitly for the experiment was used as the target application the participants had to work on, consisting of nine classes within three packages with 110 methods, eight constructors, and 37 instance variables written in pure Java (version 1.6). Each class was specified in its own file. The game has a small graphical user interface with an underlying event-based model-view-controller^[17] architecture (see Figure 4-1) and was modified in large parts to fit the experiments requirements.

Abbildung in dieser Leseprobe nicht enthalten

Figure 4-1 – A simplified UML Diagram of the target application

Essentially, the application is a simple text-based labyrinth game, where the player navigates through a labyrinth, walking towards the goal field while trying to avoid traps which are hidden on some fields. It has a JFrame called LabyrinthFrame, which acts as the view in the model-view-controller architecture and which is responsible for every type of feedback the user gets. Its registered listener for all events and actions is the GameManager class, which also acts as the core of the game, where the main game logic is controlled. The GameManager is called by the GameRunner class, which owns the main method and uses the FileAccess class, which provides functionality for reading level data from files. The GameObject, Player, Trap, GameLevel and LevelField classes represent the underlying model of the game.

4.1.3.2. The Development Environment and Hardware

A set of Lenovo Thinkpad^[18] laptops was used as computers for the study and every participant was provided with a mouse to make the most efficient programming possible. All had the same configuration, which consisted of the integrated development environment and the task workspaces, the batch files for starting the tasks (which were used to make starting eclipse using different workspaces easier), a database^[19] for the data to be automatically logged during the experiment and a screen logging software^[20].

As an integrated development environment, Eclipse Ganymede^[21] was used, along with the AspectJ Development Tools plugin^[22]. For logging reasons, a self-written development trace plugin (which is the detailed topic of the following work: (Josupeit-Walter, 2009)) had been added to Eclipse, which wrote the current workspace status into a database every time the user made a short break in editing the code (about two seconds of inactivity were needed to trigger it), saved the workspace or ran a test case. That way, a large base of data for evaluation could be generated for every task every participant had edited. Additionally, the screen logger installed on each computer was started before the participants began working on the tasks. Its videos were used for data redundancy purposes and to give additional data and detailed evaluation should the data in the database not be enlightening enough to reconstruct what a participant did.

These additional programs, especially the latency of eclipse that was induced by the regular workspace saving created some adverse conditions for the participants, as performance and reaction time of the Eclipse environment were significantly decreased during the experiment and the mouse pointer was flickering due to the screen logger. But as this was the case for every participant, the odds were still even and the effect applies to all data.

4.1.4. The Tasks

All tasks given to the participants had to be done in the order given to them. They received some sheets of paper which had the problem descriptions and instructions for each task on them, along with some general hints. Each task was started using a named batch file, which itself started Eclipse using the specific workspace for that task. The corresponding workspaces for each task also contained a set of test cases^[23] the participants were instructed to use to evaluate their progress on the current task. That way they had the possibility to find out whether they were done with the current task and where they still had errors or unfinished work. Only if all test cases succeeded they were allowed to begin the next task. After each task, Eclipse had to be closed and the following task started using the next batch file. The first three larger tasks were supposed to provide an easy means of achieving better results using aspect-orientation, while for the smaller tasks it was predictable that plain object-orientation would generally achieve better results.

In the following chapters, all nine tasks are explained in more detail:

4.1.4.1. Task1: The Logging Task

The first (and also the largest) task of the study was to add a logging-feature to the application, where each method (but no constructors) had to be supplemented with a call to the logger at the beginning of the method. For this, a corresponding logger-interface was provided which expected the name of the class where the method was declared in, the method name, the return type, an array of its actual parameter instances and an array of type String with the formal parameter names. An example for a log call is given below in Listing 4-1.

Abbildung in dieser Leseprobe nicht enthalten

Listing 4-1 – An example statement of the Logging Task using only object-oriented programming

For a public method m in class C with parameter types int and A (and the corresponding parameter names i and a) and the return type R the corresponding invocation of the logger in pure Java that needed to be defined by the developer looks like shown above. The expected type names are simple names, i.e. no package names were expected by the logger. The consumed time by this task consists of the time for navigating to the right position in the code and writing the corresponding piece of code. Typically, developers do this via copy and paste, where they copy a previous logger-call and then copy the corresponding data into it. The object-oriented solution to this task was a very repetitive procedure, as the log statement had to be written or copied and modified for every method, making up 110 statements to create altogether. It can be assumed that the required time also depends on the number of parameters for each method, because for each parameter two changed entries (compared to a previous log-invocation) needed to be specified (one for the parameter type and one for the actual parameter), which makes this a potential variable on the performance, even if its impact may be small. Within the 110 methods there were 45 without parameters, 39 with one parameter, 20 with two parameters, one with three parameters, four with four parameters and one with seven parameters.

The same task had to be solved in AspectJ, too. Thereto, the aspect definition, consisting of the keyword aspect, an aspect name and the corresponding brackets were given to the participant.

A good (and short) AspectJ solution for this task was to specify a pointcut that refers to the target classes via their package descriptions and a corresponding advice that reads the method signature and the actual parameters from thisJoinPoint. An example for such a piece of code that needed to be specified by each participant is shown in Listing 4-2.

Abbildung in dieser Leseprobe nicht enthalten

Listing 4-2 – A possible AspectJ Solution for the Logging Task

The participants did not get any hints about how to specify this aspect. Hence, this piece of code required some creative reasoning about AspectJ: The participants needed to think about how to use the language constructs of AspectJ in this situation. As a consequence, large variations of this piece of code were probable.

4.1.4.2. Task2: The Parameter Null Task

The second task was to add null reference-checks to all non-primitive parameters of all methods in the application (without constructors), which included strings and enumerations instances. In case one of the non-primitive parameters was null, an InvalidGameStateException should be thrown (which was part of the application and supplied in a separate package). For a method m in class C with parameter types int, A and B (and the corresponding parameter names i, a and b) the corresponding null-reference check that needed to be defined by the developer looks like shown in Listing 4-3.

Abbildung in dieser Leseprobe nicht enthalten

Listing 4-3 – The null-reference check using pure object-orientation

Altogether, 34 methods needed to be adapted. Similar to the first task, it can be assumed that the required time also depends on the variable number of non-primitive parameter types (in addition to the time required for typing the code for the first time), because even with copy and paste, the parameters still had to be renamed, added or removed. Within the 34 methods there were 29 methods with one, four methods with two and one with three non-primitive parameter types.

Again, in order to specify a corresponding aspect, a template for this aspect (without pointcut and advice) was given. In fact, a good solution could have the same pointcut as the one from the log-task. Within the corresponding advice it seems reasonable to reflect on the method’s parameter types via the signature passed by thisJoinPoint (see Listing 4-4).

Abbildung in dieser Leseprobe nicht enthalten

Listing 4-4 – A possible AspectJ solution for the null-reference checks

Again, the participants did not get any hint about how to implement the corresponding aspect. Consequently, a large variety of different implementations among the participants is probable. For example, AspectJ solutions which did not differ between primitive and reference types were perfectly valid, as primitive types could not have been null anyway.

4.1.4.3. Task3: The Synchronization Task

The third task was to add synchronization code to all methods within the three classes GameObject, Player, Trap and GameLevel. Thereto, the participants were explicitly asked to make use of the synchronize block provided by Java for which the task description already contained a template as shown in Listing 4-5. Altogether, 52 methods needed to be adapted by the participants.

Abbildung in dieser Leseprobe nicht enthalten

Listing 4-5 – The code template for the synchronization task

In contrast to the previous tasks it was not necessary to modify the inserted code, while in the previous examples some parameters, etc. had to be added, removed or modified, this code comes "as is". Nevertheless, a difference is that it is still no plain copy & paste approach: the participant has to navigate to the beginning of a method to add the synchronized-phrase as well as to the end of a method in order to insert the closing bracket. From that perspective, the length of the method might be an influencing variable factor on the time required to fulfill this task. However, the method length in our example application always fit on the display. Consequently, the influence of method length is considered not to be substantial and is neglected in further discussions.

Abbildung in dieser Leseprobe nicht enthalten

Listing 4-6 – The AspectJ solution for the synchronization task

For specifying the AspectJ version of this task, the participants needed to be familiar with the around and proceed language constructs of AspectJ. In contrast to the previous tasks there are not that many reasonable alternative solutions. The only possible differences manifest in whether the target object reference is being exposed by the pointcut definition (like specified in Listing 4-6) or whether thisJoinPoint is being used for this purpose.

4.1.4.4. Task4: The Check Player Argument Task

After the first three tasks, which consumed the largest part of the time, the check player task was a very short one. The GameManager class has four update methods which take a Player object as an argument. The task was to check for each update method whether this argument equals the Player object instance that the current GameManager references via its player attribute. If the instances would not match, an InvalidGameStateException had to be thrown. An example code excerpt is shown in Listing 3-1 below.

Abbildung in dieser Leseprobe nicht enthalten

Listing 4-7 – An example object-oriented solution for the check player argument task

Because there are only four methods that had to be edited, this task was easy to do. The only variable in its editing could have been the time needed to find the four methods in the GameManager class, the code itself could be written the first time and copied for the other three times. But still, the approach could vary. This task was one of the tasks where the object-oriented approach was assumed to take much less time, because of the small number of methods to be edited.

Using AspectJ, the trick was to get a reference to the instance of the GameManager class which held the player attribute that had to be compared via this or target and use either context binding via the AspectJ args construct or the getArgs method of thisJoinPoint to get hold of a reference to the Player argument. A rough example solution could have looked like the following:

Abbildung in dieser Leseprobe nicht enthalten

Listing 4-8 – Possible AspectJ solution for the check player argument task

A more elegant approach would have been the usage of the args keyword to bind the Player argument. But again, the pointcut and advice definitions could vary between all participants and still be legitimate solutions which solved the task.

4.1.4.5. Task5: The Notify Observers Task

Fifth task among the nine was the notify observers task, where the participants had to insert a call to the notifyObservers method of the GameObject and Player classes each time an instance variable of these classes was set in one of their methods. This had to happen every time an instance variable was set, not just at the end of the method were the setting had happened. This made sure that even in methods where more than one variable was set the call to notifyObservers was done the according number of times, too (an example is shown below). All in all, 19 methods were subject of such editing.

Abbildung in dieser Leseprobe nicht enthalten

Listing 4-9 – An object-oriented example for the notify observers task

The time needed for this task was influenced by the variables of the time for finding all set appearances of instance variables in the methods. The notifyObservers call could be copied into the code once it was written manually for the first time or could be written each time using the help of the Eclipse auto-completion feature.

One of the reasons for this approach was to keep the semantics the same as those from the AspectJ solution which had to be used for the task. Again, different solutions were possible. The following code shows a pointcut which made use of the fact that Player was a subclass of GameObject and binds the current instance via this.

Abbildung in dieser Leseprobe nicht enthalten

Listing 4-10 – An AspectJ solution for the notify observers task

The above AspectJ construct makes sure that the notifyObservers method is called every time an instance variable of GameObject or Player is set, just like the task description specifies. With enough understanding of the AspectJ mechanics, it was possible to finish this task quite fast, even if the pure copy and paste approach of the object-orientation was very fast and easy to do itself, either.

4.1.4.6. Task6: The Observers Null Check Task

Immediately after the notify observers task the observers null check task had to be done. It was meant to complement the notify observers task, because now the participants had to add a null reference check before the call of the notifyObservers method in the classes GameObject and Player to make sure their observers list was not null. 15 methods had to be edited that way. When taking the example from above, a code sample could look like the following:

Abbildung in dieser Leseprobe nicht enthalten

Listing 4-11 - An object-oriented example for the observers null check task

As can be seen, the semantics were kept the same as the previous task, originating from the same cause of keeping them close to those of the aspect, requiring a null check before each call to notifyObservers. The time needed for this task was, like the previous task, influenced by the variables of the time for finding all notifyObservers calls in the code (which should have been a little easier now, as the participants already edited the same code parts in the previous task). Again, after writing the necessary code once, it could be copied and pasted for all other times.

Abbildung in dieser Leseprobe nicht enthalten

Listing 4-12 - An AspectJ solution for the observers null check task

The pointcut definition could be kept simple, as only the calls to the notifyObservers method needed to be captured and it was sufficient to make a null reference check on the observers instance variable of the object in the advice. Different solutions were possible, like using explicit pointcut definitions for the two classes or the execution primitive along with this.

4.1.4.7. Task7: The Refresh Constraint Task

Task number seven is called the refresh constraint task, where the goal is to check integer values which are passed to all methods of the LabyrinthFrame class whose names begin with refresh. Those methods are responsible for passing these integer values on to JLabel fields which present these values to the user. These integer values have to be checked whether they are of a value smaller than (-1), which would be an invalid value and therefore an InvalidGameStateException should be thrown in the case. Listing 4-13 demonstrates an example.

Abbildung in dieser Leseprobe nicht enthalten

Listing 4-13 –Possible solution for the refresh constraint task using object-orientation

Participants only had to write the if-statement once and could copy it for the next time. A possible variable could be the number of parameters the methods had, where participants had to use an or-clause or copy and edit two if-statements to fulfill the task. Altogether, eight methods had to be edited with these refresh constraints.

Aspects that fulfilled this task were a bit trickier, because of the boxing^[24] behavior in Java. Either the participants avoided the unboxing procedure by using different pointcuts, where the args primitive removed the need to unbox the integer values, or they used the getArgs method and unboxed the values for checking, as is shown in the aspect example below:

Abbildung in dieser Leseprobe nicht enthalten

Listing 4-14 - An AspectJ solution for the refresh constraint task

The aspect attaches itself to all methods of LabyrinthFrame starting with refresh, uses the getArgs method to obtain the arguments, unboxes the values and does the check. As can be seen, the code for unboxing the integer values out of the Object array references looks rather ugly. The Object has to be cast to the type Integer (which is the so called wrapper class for integer values), and after this, the intValue method has to be used to acquire the real primitive integer value which is boxed in the Integer type object. Because the fact that this code is ugly and such occurrences of boxing and unboxing do not appear in the day to day tasks of developers too often, many participants possibly would not have known how to write the code for the unboxing procedure. To eliminate this large factor, which would possibly have consumed a lot of time and made the code more error prone, a small hint was given in the tasks description along with a code sample on how to use unboxing. With this hint, this variable should have been eliminated.

4.1.4.8. Task8: The Label Value Check Task

Penultimate among the tasks was task eight, the label value check task. During this task, the participants had to write code which checks the value of a JLabel for correctness just after it is set. Essentially, it was only needed to call the getText method of the JLabel and check its returned string for correctness. Correctness was defined as a number between zero and nine or an empty string as the last character. Participants were not required to write the code which performed the real string check themselves, they were rather provided with a convenience method called isValidLabelString, which performed the check and returned true or false. On return of a false, an InvalidGameStateException had to be thrown. Ultimately, the only code fragments that needed to be written were like the following:

Abbildung in dieser Leseprobe nicht enthalten

Listing 4-15 - Possible solution for the label value check task using object-orientation

Such a code fragment had to be inserted into six methods altogether. This time there were practically no variables, as the code could be copied and pasted and only the name of the label had to be altered in the if-statement.

In contrast to the object-oriented solution, the aspects that fulfilled this task where probably the hardest to write of all nine tasks. Participants needed to get a reference to either the instance of the LabyrinthFrame object (to be able to call the isValidLabelString method, which is not static) and the reference to the JLabel whose text had been set.

Abbildung in dieser Leseprobe nicht enthalten

Listing 4-16 - An AspectJ solution for the label value check task

This aspect probably required the most thinking and trickiest use of AspectJ. Just getting the string argument for the refresh method was not sufficient to fulfill this task, as it was important that the check happened after the call to the setText method of the label. The participants needed to have understood the concepts of both the this and target primitives. Another approach would have been to create a new LabyrinthFrame instance just to get access to the isValidLabelString method, so that only the JLabel reference would have been needed to be acquired via pointcut.

4.1.4.9. Task9: The Current Level Check Task

The last task to be done was the current level check task. Goal of the task was to check whether the level boundaries of a new level which was loaded where greater than the boundaries of the current level. The point for the check was every time the currentlevel instance variable of the GameManager class was set in the program code. Participants did not have to write the check logic themselves, as a convenience method called checkLevelConstraints was provided for the task, which took the current and the new level as arguments and returned a true or a false. On return of a false, an InvalidGameStateException had to be thrown.

Abbildung in dieser Leseprobe nicht enthalten

Listing 4-17 - Possible solution for the current level check task using object-orientation

The object-oriented solution did not require much thinking and could be copied and pasted once a correct solution had been typed once. No adjustments had to be made in the code. The only variable was the time needed to search for the occurrences in the GameManager code where the currentlevel instance variable was set.

The aspect which solved this task needed to hook itself onto the setting of the currentlevel instance variable via the set primitive and get the reference to the GameManager instance via this (as the checkLevelConstraints method is not static).

Abbildung in dieser Leseprobe nicht enthalten

Listing 4-18 - An AspectJ solution for the current level check task

The aspect needed to get the reference to the level to be set using the getArgs method and a cast to GameLevel. And it contained an additional pitfall, as the currentlevel variable could be null in specific situations in the program flow, which required a check against not null.

4.2. Implementation of the Experiment

The implementation of the experiment was started by a phase where possible participants were recruited using personal contact, e-mail newsletters and forums which are commonly frequented by computer science students and students of related sciences. As the participants were not paid anything for their participation and time consumption of the experiment was communicated as about 4-5 hours, feedback from possibly interested participants was relatively low. At the end of the implementation phase, 24 people had partaken in the experiment. Most of them were computer science or business information systems students, but also one software developer from the industry and a few university teachers.

The first difficulty after making contact was to find dates and times to which the participants had time to spare. It was impossible to get a group larger than two or three participants at a single time and day, which made the whole process of giving the tutorial, supervising the fulfillment of the tasks, saving data and videos and preparing the computer for the next participant a very time consuming one. The experiment implementation phase was planned to take two weeks at first, but had to be extended to nearly five weeks, because of the scheduling problems with all participants.

During the passes of the first participants, a problem with the original timing plan for the tasks revealed itself, as the time the development tracing tool added to the overall time had not been taken into account and now lead to an increased time needed to finish all the tasks. Additionally, most of the participants had large problems in solving the aspect-oriented tasks and it took them a lot longer than originally estimated. In the end, only a small group of participants stayed in the projected time limit of about four to five hours overall experiment time.

As another result of this huge time multiplication, the videos captured by the screen logger became very big, much bigger than assumed and some turned out to be partially corrupted because of that fact.

Another problem originated from some strange hang-ups the Eclipse environment sometimes suffered right after starting a new task, requiring the Eclipse GUI to be shutdown and restarted. This lead to the problem of multiple session ids for one task of some participants, as these session ids were generated and written into the database each time a participant started a new task. As a best case this meant that there existed exactly 18 session ids (9 aspect-oriented tasks, 9 object-oriented) for each participant after the experiment. Because of the crashes, which sometimes even happed while working on a task, for some participants multiple session ids were created for some tasks. This has lead to additional problems in the analysis phase.

Two of the participants did not manage to finish working on both task sets. Because of a lack of time, one only did the object-oriented part and one the aspect-oriented. The progress of two other participants was not saved into the database correctly because of a wrongly configured file for the development trace tool. All in all, 20 fully usable datasets were available after the experiment, along with one additional object-oriented dataset and one aspect-oriented from the two participants who could not bring up the time to finish all tasks.

5. Experiment Analysis and Results

Preparation and processing of the gathered data for later evaluation was started even while the experiment implementation phase was still running. The whole gathered data taken together formed a huge mass of information, which was way too overwhelming when only considering the goal of this experiment, as the development tracing tool was bound to save the complete workspace into the database every few seconds, along with a lot of additional data like keyboard input, shortcuts or eclipse commands. Description of the steps for data preparation, processing and analysis are only focused on the data relevant for this experiment.

5.1. Data Processing and Preparation

As the interest of this experiment was primarily to measure the successes per time (where successes being the number of fulfilled test cases) for each task and programming technique to possibly create a time series afterwards, it was important to only extract the data needed for that process from the database. First, a time interval had to be chosen at which the development tracing tool would extract the required data from the database, because extracting every so called snapshot of a workspace from the database would have taken a huge amount of time. As a first interval, 30 seconds were chosen. For the extraction procedure, the tool needed to be supplied with the requested time interval and the session id, which uniquely identified one eclipse session for one task of one user, as already stated above. A list was created manually which mapped the specific task and related programming technique to the corresponding session id or session ids (sometimes an Eclipse crash right in the middle of working on a task required more than one session to be provided to the tool for extraction of one tasks complete result). In some worst cases, the correct session ids could only be found out by looking into the screen captured videos and making a time stamp comparison between the video and the sessions in the database. After that, the tool was started via automatically created batch files for each of these sessions and extracted the required snapshots (a snapshot is a buildable version of the eclipse workspace of the corresponding task) from the database for the 30 seconds interval.

After the snapshots for the tasks were available, the tool needed to iterate over these snapshots, build each one and run all the test cases on each of these and write the results into the database, so that a “successes per time”-series could be created. To create even more fine-grained data, evaluation aspects were written which printed out additional data for each test case run into a stream which was written into the database along with the other data and the result of success or failure for the specific test case. Because errors which the compiled snapshots sometimes contained initially compromised this evaluation, resulting in faulty data, it was necessary to write additional pointcuts and advice into the evaluation aspects which encapsulated the invocation of the test case methods and made sure all exceptions were caught and did not harm the evaluation process. For the smaller tasks, it became necessary to use a smaller time interval of five seconds, because the 30 seconds interval that was initially chosen created too rough time series for these tasks.

For the aspect-oriented results, it soon became clear that no useful time lines could be created from them, as there was no real order apparent they were following. Most of them consisted just of many intervals where no successes were achieved at all, and the next 30 seconds lead to the complete success in all test cases. As this was the supposed and expected nature of the aspect results, the thought of creating time series diagrams for the aspect-oriented tasks had to be abandoned. Therefore, it was necessary to have at least a precise idea of what time was needed exactly for the aspect-oriented tasks, which is why a new database procedure was created, which used the data from the database to extract the exact number of seconds a participant required for one task in aspect-orientation. As a starting point, the first time a participant edited the workspace was chosen, as it was assumed that at this time the task description had been read and understood and working on the aspect had began. As the ending time, the first timestamp where all test cases for this task succeeded was taken. The difference between both was taken as the time needed for that task. As this approach seemed to generate precise results, it was taken to calculate the object-oriented overall editing time, too.

Because correlation values were below seventy percent on some result sets, a look into the corresponding video was taken (if available) and the possible source of the bad correlation (like taking a wrong approach and then correcting it, resulting in long time spans where no successes where generated) was noted along with the data. And because there were some inconsistencies in the data (manifesting in above mentioned bad correlation), like participants doing a short break which was not noticed, making a pause in thinking, etc. there was the idea of cleaning the data from these possible disturbances. So, a database procedure was written to remove all entries from a result set where the ratio of changed successes per time differed more than eighty percent from the average number of successes done in the complete time. The idea behind this was that while doing a redundant task in code over and over again, the average time for each success should be nearly identical when removing all disturbance factors like users taking a break or thinking. This breaks the data down to the raw editing of the code and pushes correlation even further, as can be seen from the cleansed data in the diagrams and tables shown later. This data cleansing could only be done for the larger tasks, as the smaller task results consisted of too few entries at all and too many were removed by the procedure, generating mostly unusable data.

The data from the questionnaire was also evaluated and used to try to categorize the participants depending on the answers they provided for each question. To be able to better create groups based on skill or experience profiles, the participants were asked an additional question regarding their overall experience with reflection techniques, which supposedly aided in solving the aspect-oriented tasks.

[...]

^[1] (Highley, et al., 1999), p. 2

^[2] (Basili, et al., 2007) and (Basili, 2007), for example. Basili and his colleagues have written many publications on the role and use of empirical studies and experimentation in software engineering

^[3] (Tichy, 1997)

^[4] Most people use the word paradigm as a replacement for technique, approach or concept, even if it differs from the word's original meaning. As its original meaning was to represent an underlying concept or a principle agreed by everyone on the matter, it can still be used in software engineering, but people should be aware of its usage in a wrong sense.

^[5] The paper of Josupeit-Walter (Josupeit-Walter, 2008) summarizes most of the studies on object-orientation. None of them really managed to back everything that is said about its benefits.

^[6] The whole field of theory of knowledge concerns itself with knowledge, its nature, how it is gained, etc. See http://en.wikipedia.org/wiki/Epistemology

^[7] (Prechelt, 2001)

^[8] (Josupeit-Walter, 2008)

^[9] Interesting books to read for anyone interested in detailed information on empirical research methods: (Bortz, et al., 2002), which is very thorough and precise, (Rogge, 1995), which gives a good summary and short explanation or (Christensen, 1977), which is more focused on the experimental approach.

^[10] Open questions leave the answer to the reader, closed questions give a list of concrete answers or a range of ordinal ratings the reader has to choose from (some allow to pick only one answer, some allow more than one).

^[11] See (Josupeit-Walter, 2008) or (Prechelt, 2001) for a summary of empirical research on object-orientation.

^[12] Two papers which go into that direction are (Jones, 1994) and the follow-up (Steidley, 1994)

^[13] As of the time of this work, AspectJ was available as version 1.6.2

^[14] Their web site can be found at http://www.eclipse.org/aspectj/

^[15] (Kiczales, et al., 2001), p.3

^[16] (Hanenberg, 2005) provides a thorough description of dynamic and static features.

^[17] More information on the Model-View-Controller design pattern can be found here: http://en.wikipedia.org/wiki/Model-view-controller

^[18] Specifically, they were R60’s

^[19] The database server was PostgreSQL in version 8.3: http://www.postgresql.org/

^[20] CamStudio Version 2.00: http://camstudio.org/

^[21] Which is Eclipse version 3.4 and can be found on http://www.eclipse.org

^[22] Version 1.6.0 of the AJDT was used: http://www.eclipse.org/ajdt/

^[23] Which were written for JUnit, a tool for running test cases: http://www.junit.org/

^[24] Encapsulating a primitive value type (like integer) into a reference type (for example of type Object or its corresponding wrapper class Integer) is called boxing, where the primitive value of the integer is moved to the heap and replaced by a reference to it on the stack.