Reducing Effort in Software Developers: Impact of Merging and Repositioning of Static Analysis Alarms

Project Objective

To understand the impact of merging and repositioning static analysis warnings in programming environments on the developers’ warning inspection time, understanding accuracy, and eye movements, and to identify if there is a relationship between a programmer’s spatial abilities and their problem-solving skills.

The research sought to determine if there is a relationship between a programmer’s spatial abilities and their problem-solving skills, as well as whether specific treatments could reduce the cognitive load and improve efficiency in addressing alarms.

Methods Used

• Employed a within-subjects study design in which two treatment conditions, repositioning and merging alarms, were applied to assess their effects. The participants were asked to work on general alarm inspection tasks as well.
• Deployed a survey containing the study questions though the Qualtrics platform.
• Methods included quantitative analysis to find the effect of the treatments on accuracy, and explore the relationship between cognitive abilities and comprehension scores.
• Employed eye tracking technology to collect data on developers’ gaze patterns and fixations while performing manual inspection tasks on software code
• Quantitative analysis methods were applied to compare visual effort metrics and task accuracy between the treatment groups.

Role
Principal Researcher

I framed the research questions, chose the methodologies, analyzed the data, and presented the findings in one research paper and my PhD dissertation.

Key Findings

1. Reduced Visual Effort: Repositioning and merging alarms significantly reduced visual effort in tasks requiring examination of a single file.
2. Complication of Navigation: In multi-file tasks, the treatments did not significantly impact visual effort or accuracy. The participants had a more difficult time navigating through the project and finding relevant parts of code, no matter the treatment.
3. Performance and Cognitive Abilities: A moderate positive correlation was observed between spatial cognition and comprehension score in the survey study, indicating the importance of spatial cognition compared to working memory.
4. Challenges of C Programming: Most participants found the study somewhat challenging, based on our post-questionnaire results.

Process

• Preparation: Developed a research plan focusing on studying the effects of merging and repositioning of static alarms, through a large-scale survey and eye tracking. Selected relevant code inspection tasks and set up the eye tracking equipment.
• Recruitment: 249 participants were recruited through Mechanical Turk and links sharing on social media for the survey study, and 27 participants (including undergraduate, graduate students, and industry professionals, and 50% of the participants were female) who had experience with the C programming language were recruited for the eye tracking study.
• Execution: Conducted the online study using Qualtrics and cleaned up the data using Python scripts after determining criteria for data validity. Ran the eye tracking study using eye tracking equipment (Tobii Pro TX-300) in the SERESL lab, with participants being assigned to various treatment conditions. The i-trace plugin was connected to the Eclipse IDE to collect line and token level eye tracking data from the code reading process, and its toolkit was used to generate fixations.
• Analysis: Extracted fixation metrics from the eye tracking data using Python scripts, and analyzed the data using statistical tests like Pearson’s correlation, Mann-Whitney U tests, and other relevant methods. Data was analyzed with the Python programming language and its statistical packages.
• Reporting: Combined findings were documented and presented in the form of research papers and presentations, using data visualizations, including graphs, charts, and figures.

Recommendations

• Usability of static analysis tools can be improved by repositioning and merging of static analysis alarms.
• Repositioning and merging alarms can be effective in reducing visual effort for software developers inspecting individual code files, drawing focus to the most relevant parts of the code.
• These treatments can address the issues of high number of alarms (through merging) and the presentation of alarms (through repositioning and merging).
• The combined findings indicated that participants’ spatial cognitive abilities correlated with their comprehension skills and accuracy.

Visuals

The following images show the relevant parts of task code and the the participants’ gaze patterns on the relevant code blocks while working on the manual inspection tasks in the form of Alpscarfs.

The Alpscarf shows that all participants fixated on the relevant parts of the code, especially Block 3 that contains the assignment of the variable under assertion in Block 1. They also kept revisiting Block 1 to check the assertion.

The following images show the the participants’ gaze patterns on the relevant code blocks while working on the repositioning treatment tasks and tasks without treatment in the form of Alpscarfs.

The Alpscarfs shows that all participants fixated on the relevant parts of the code in the non-treatment and treatment tasks, but in the task with treatment most of the focus is on Block 5, which is the most relevant block for solving the task. The Alpscarf for the task without treatment showed more visual effort and searching through code from the participants to find the relevant information to solve the task.

Reflection

I identified potential reasons for inconclusive results regarding manual effort reduction and visual effort. Future directions involve using more advanced tools, techniques, and a broader range of tasks with multiple files to draw comprehensive conclusions regarding the effects of alarms merging, repositioning, and visual effort in software development.