Assessing the Effect of Programming Language and Task Type on Eye Movements of Computer Science Students

Project Objective

The objective of this research was to investigate how programming language (Python and C++) and task type (Bug Finding and Feature Addition) impact the eye movements of computer science students. The study aimed to understand how students visually process code when performing these programming tasks in these two languages.

Methods Used

• Employed eye-tracking technology to collect gaze data during code reading and programming tasks.
• Analyzed the results of conducted controlled experiments with computer science students.
• Utilized statistical analysis to identify patterns and differences in eye movements.

Role
Researcher

I analyzed and visualized data, and presented the findings.

Key Findings

1. Bug Fixing vs. Feature Addition: Participants had fixated significantly longer on different elements during bug fixing tasks compared to feature addition tasks. This suggests that bug fixing is a more challenging task.
2. Attention to Surrounding Code: Participants tended to look at lines adjacent to the one containing the bug more frequently before focusing on the buggy line itself. This indicates a pattern of reviewing nearby code when debugging.
3. Feature Addition Success: Participants who successfully added a new feature made their first edit earlier in the task compared to those who failed to add the feature. Early edits were associated with task success.
4. Python vs. C++: Tasks in Python and C++ had similar overall fixation duration and counts when adjusted for character count, indicating that participants spent a similar amount of time and attention on both languages.
5. Console Output: Participants spent more time fixating on the console output when working on Python tasks. This suggests that console output may play a more significant role in Python programming tasks.
6. Task Type vs. Programming Language: Overall, the type of task (bug fixing vs. feature addition) had a more substantial effect on the overall fixation duration and count compared to the choice of programming language (Python vs. C++).

Process

• Preparation: Collaboratively defined and revised research questions and hypotheses.
• Recruitment: Selected 14 computer science undergraduate and graduate students from a US university as participants.
• Execution: Conducted eye-tracking experiments in a controlled lab environment.
• Analysis: Analyzed eye movement data using statistical methods and visualization techniques. I used Mann-Whitney U and Wilcoxon signed-rank tests to compare different metrics (accuracy, time, and normalized eye tracking fixation metrics). Used R to perform statistical analysis and visualize fixations.
• Reporting: Presented research findings through an academic paper, accepted in ACM Transactions on Computing Education.

Recommendations

• Suggested that task type and complexity should be taken into account when designing programming assignments.
• Students should be encouraged to make edits in the code in order to successfully add features.
• Programming educators should contain more code reading exercises to get students more familiar with different styles of coding, not only their own.
• The research contributed valuable insights to the field of computer science education, helping educators tailor their teaching approaches.

Visuals

The following plots show the participants’ gaze patterns on lines while working on the bug finding tasks in the form of Alpscarfs . The Alpscarfs show that the line containing the bug was more frequently fixated on in the Python task. We can see a more chronological reading pattern in the participants who answered the C++ task incorrectly. The participants did not read the code in order while working on the Python bug finding task.

Reflection

The research project provided valuable insights into the cognitive processes of computer science students while programming. It highlighted the importance of considering programming language and task complexity in educational contexts. In future research, we plan to explore more diverse participant groups and further investigate the implications of these findings on programming pedagogy.