publications | Lucas Layman, PhD

2024

Generative AI and the Future of Software Testing – A Virtual Roundtable

Lucas Layman, and Ron Vetter

IEEE Computer, Jan 2024

2023

A Controlled Experiment on the Impact of Intrusion Detection False Alarm Rate on Analyst Performance

Lucas Layman, and William Roden

In Proceedings of the Human Factors and Ergonomics Society (HFES) 67th Int’l Meeting, Oct 2023

arXiv
Interdisciplinary Collaboration among Nursing and Computer Science to Refine a Pharmacogenetics Clinical Decision Support Tool via Mobile Application

Crystal Heath Dodson, and Lucas Layman

Computers, Informatics, Nursing, Jun 2023

2022

A Portable, Bluetooth-Enabled, Isometric Neck Assessment System

Lucas Layman, Alexander T McDaniel, and Lindsey H Schroeder

In Proceedings of IEEE SoutheastCon 2022, Jun 2022
Coastal Eco Explorer: A Mobile Application for Ecology Education

Lucas Layman, Amy R Taylor, Dennis S Kubasko Jr., and 2 more authors

In Proceedings of IEEE SoutheastCon 2022, Jun 2022
Refinement of a Pharmacogenomics App for Dosing Guidelines for Oncology: Findings from the Usability Evaluation

Crystal Dodson, and Lucas Layman

Annals of Translational Medicine, Dec 2022

2020

Cry Wolf: Toward an Experimentation Platform and Dataset for Human Factors in Cyber Security Analysis

William Roden, and Lucas Layman

In Proceedings of the 2020 ACM Southeast Conference (ACMSE 2020), Dec 2020

arXiv
Coastal Eco Explorer

Lucas Layman, Kinsley Sigmund, Amy R. Taylor, and 2 more authors

Dec 2020
Suitability of SCS1 as a Pre-CS2 Assessment Instrument: A Comparison with Short Deliberate-practice Questions

Yang Song, Yunkai Xiao, Jonathan Stephens, and 3 more authors

In Proceedings of the 2020 ACM Southeast Conference (ACMSE 2020), Dec 2020
Toward Predicting Success and Failure in CS2 : A Mixed-Method Analysis

Lucas Layman, Yang Song, and Curry Guinn

In Proceedings of the 2020 ACM Southeast Conference (ACMSE 2020), Dec 2020

Abs

Factors driving success and failure in CS1 are the subject of much study but less so for CS2. This paper investigates the transition from CS1 to CS2 in search of leading indicators of success in CS2. Both CS1 and CS2 at the University of North Carolina Wilmington (UNCW) are taught in Python with annual enrollments of 300 and 150 respectively. In this paper, we report on the following research questions: (1) Are CS1 grades indicators of CS2 grades? (2) Does a quantitative relationship exist between CS2 course grade and a modified version of the SCS1 concept inventory? (3) What are the most challenging aspects of CS2, and how well does CS1 prepare students for CS2 from the student’s perspective? We provide a quantitative analysis of 2300 CS1 and CS2 course grades from 2013–2019. In Spring 2019, we administered a modified version of the SCS1 concept inventory to 44 students in the first week of CS2. Further, 69 students completed an exit questionnaire at the conclusion of CS2 to gain qualitative student feedback on their challenges in CS2 and on how well CS1 prepared them for CS2. We find that 56% of students’ grades were lower in CS2 than CS1, 18% improved their grades, and 26% earned the same grade. Of the changes, 62% were within one grade point. We find a statistically significant correlation between the modified SCS1 score and CS2 grade points. Students identify linked lists and class/object concepts among the most challenging. Student feedback on CS2 challenges and the adequacy of their CS1 preparations identify possible avenues for improving the CS1-CS2 transition.

2019

The Cry Wolf IDS Simulator - An environment for conducting controlled experiments of cyber security analysis tasks

William Roden, and Lucas Layman

Dec 2019
The Cry Wolf Dataset - A repository of simulated IDS alerts for experimentation

Williams Roden, and Lucas Layman

Dec 2019
How to trick the Borg: threat models against manual and automated techniques for detecting network attacks

Carl Sabottke, Daniel Chen, Lucas Layman, and 1 more author

Computers & Security, Mar 2019

Abs Website

Cyber attackers constantly craft new attacks previously unknown to the security community. There are two approaches for detecting such attacks: (1) employing human analysts who can observe the data and identify anomalies that correspond to malicious intent; and (2) utilizing unsupervised automated techniques, such as clustering, that do not rely on ground truth. We conduct a security analysis of the two approaches, utilizing attacks against a real-world website. Through two experiments—a user study with 65 security analysts and an experimental analysis of attack discovery using DBSCAN clustering—we compare the strategies and features employed by human analysts and clustering system for detecting attacks. Building on these observations, we propose threat models for the human analysis process and for the unsupervised techniques when operating in adversarial settings. Based on our analysis, we propose and evaluate two attacks against the DBSCAN clustering algorithm and a defense. Finally, we discuss the implications of our insights for hybrid systems that utilize the strengths of automation and of human analysis to complement their respective weaknesses.

2017

Less is More: Minimizing Code Reorganization using XTREE

Rahul Krishna, Tim Menzies, and Lucas Layman

Information and Software Technology, Mar 2017

Abs

\textcopyright 2017 Elsevier B.V. Context: Developers use bad code smells to guide code reorganization. Yet developers, textbooks, tools, and researchers disagree on which bad smells are important. How can we offer reliable advice to developers about which bad smells to fix? Objective: To evaluate the likelihood that a code reorganization to address bad code smells will yield improvement in the defect-proneness of the code. Method: We introduce XTREE, a framework that analyzes a historical log of defects seen previously in the code and generates a set of useful code changes. Any bad smell that requires changes outside of that set can be deprioritized (since there is no historical evidence that the bad smell causes any problems). Evaluation: We evaluate XTREE’s recommendations for bad smell improvement against recommendations from previous work (Shatnawi, Alves, and Borges) using multiple data sets of code metrics and defect counts. Results: Code modules that are changed in response to XTREE’s recommendations contain significantly fewer defects than recommendations from previous studies. Further, XTREE endorses changes to very few code metrics, so XTREE requires programmers to do less work. Further, XTREE’s recommendations are more responsive to the particulars of different data sets. Finally XTREE’s recommendations may be generalized to identify the most crucial factors affecting multiple datasets (see the last figure in paper). Conclusion: Before undertaking a code reorganization based on a bad smell report, use a framework like XTREE to check and ignore any such operations that are useless; i.e. ones which lack evidence in the historical record that it is useful to make that change. Note that this use case applies to both manual code reorganizations proposed by developers as well as those conducted by automatic methods.

2016

Are delayed issues harder to resolve? Revisiting cost-to-fix of defects throughout the lifecycle

Tim Menzies, William Nichols, Forrest Shull, and 1 more author

Empirical Software Engineering: An International Journal, Mar 2016

Abs

\textcopyright 2016, Springer Science+Business Media New York. Many practitioners and academics believe in a delayed issue effect (DIE); i.e. the longer an issue lingers in the system, the more effort it requires to resolve. This belief is often used to justify major investments in new development processes that promise to retire more issues sooner. This paper tests for the delayed issue effect in 171 software projects conducted around the world in the period from 2006–2014. To the best of our knowledge, this is the largest study yet published on this effect. We found no evidence for the delayed issue effect; i.e. the effort to resolve issues in a later phase was not consistently or substantially greater than when issues were resolved soon after their introduction. This paper documents the above study and explores reasons for this mismatch between this common rule of thumb and empirical data. In summary, DIE is not some constant across all projects. Rather, DIE might be an historical relic that occurs intermittently only in certain kinds of projects. This is a significant result since it predicts that new development processes that promise to faster retire more issues will not have a guaranteed return on investment (depending on the context where applied), and that a long-held truth in software engineering should not be considered a global truism.
Topic modeling of NASA space system problem reports research in practice

L. Layman, A.P. Nikora, J. Meek, and 1 more author

In Proceedings - 13th Working Conference on Mining Software Repositories, MSR 2016, Mar 2016

Abs

\textcopyright 2016 ACM. Problem reports at NASA are similar to bug reports: they capture defects found during test, post-launch operational anomalies, and document the investigation and corrective action of the issue. These artifacts are a rich source of lessons learned for NASA, but are expensive to analyze since problem reports are comprised primarily of natural language text. We apply topic modeling to a corpus of NASA problem reports to extract trends in testing and operational failures. We collected 16,669 problem reports from six NASA space flight missions and applied Latent Dirichlet Allocation topic modeling to the document corpus. We analyze the most popular topics within and across missions, and how popular topics changed over the lifetime of a mission. We find that hardware material and flight software issues are common during the integration and testing phase, while ground station software and equipment issues are more common during the operations phase. We identify a number of challenges in topic modeling for trend analysis: 1) that the process of sele cting the topic modeling parameters lacks definitive guidance, 2) defining semantically-meaningful topic labels requires nontrivial effort and domain expertise, 3) topic models derived from the combined corpus of the six missions were biased toward the larger missions, and 4) topics must be semantically distinct as well as cohesive to be useful. Nonetheless, topic modeling can identify problem themes within missions and across mission lifetimes, providing useful feedback to engineers and project managers.
Topic Modeling of NASA Space System Problem Reports

Lucas Layman, Allen P. Nikora, Joshua Meek, and 1 more author

In Proceedings of the 13th International Conference on Mining Software Repositories (MSR ’16), Mar 2016

2015

Ask the Engineers: Exploring Repertory Grids and Personal Constructs for Software Data Analysis

Lucas Layman, Carolyn Seaman, Davide Falessi, and 1 more author

In 8th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE 2015), Mar 2015

Abs

\textcopyright 2015 IEEE. Maturity in software projects is often equated with data-driven predictability. However, data collection is expensive and measuring all variables that may correlate with project outcome is neither practical nor feasible. In contrast, a project engineer can identify a handful of factors that he or she believes influence the success of a project. The challenge is to quantify engineers’ insights in a way that is useful for data analysis. In this exploratory study, we investigate the repertory grid technique for this purpose. The repertory grid technique is an interview-based procedure for eliciting ’constructs’ (e.g., Adhering to coding standards) that individuals believe influence a worldly phenomenon (e.g., What makes a high-quality software project) by comparing example elements from their past (e.g., Projects they have worked on). We investigate the relationship between objective metrics of project performance and repertory grid constructs elicited from eight software engineers. Our results show correlations between the engineers’ subjective constructs and the objective project outcome measures. This suggests that repertory grids may be of benefit in developing models of project outcomes, particularly when project data is limited.
LACE2: Better Privacy-Preserving Data Sharing for Cross Project Defect Prediction

Fayola Peters, Tim Menzies, and Lucas Layman

In 37th International Conference on Software Engineering (ICSE ’15), Mar 2015

Abs

\textcopyright 2015 IEEE. Before a community can learn general principles, it must share individual experiences. Data sharing is the fundamental step of cross project defect prediction, i.e. the process of using data from one project to predict for defects in another. Prior work on secure data sharing allowed data owners to share their data on a single-party basis for defect prediction via data minimization and obfuscation. However the studied method did not consider that bigger data required the data owner to share more of their data. In this paper, we extend previous work with LACE2 which reduces the amount of data shared by using multi-party data sharing. Here data owners incrementally add data to a cache passed among them and contribute "interesting" data that are not similar to the current content of the cache. Also, before data owner i passes the cache to data owner j, privacy is preserved by applying obfuscation algorithms to hide project details. The experiments of this paper show that (a) LACE2 is comparatively less expensive than the single-party approach and (b) the multiparty approach of LACE2 yields higher privacy than the prior approach without damaging predictive efficacy (indeed, in some cases, LACE2 leads to better defect predictors).
Applying Software Data Analysis in Industry Contexts: When Research Meets Reality

M. Diep, L. Esker, D. Falessi, and 3 more authors

Mar 2015

Abs

\textcopyright 2015 Elsevier Inc. All rights reserved. Software data analytics is key for helping stakeholders make decisions, and thus establishing a measurement and data analysis program is a recognized best practice within the software industry. However, practical implementation of measurement programs and analytics in industry is challenging. In this chapter, we discuss real-world challenges that arise during the implementation of a software measurement and analytics program. We also report lessons learned for overcoming these challenges and best practices for practical, effective data analysis in industry. The lessons learned provide guidance for researchers who wish to collaborate with industry partners in data analytics, as well as for industry practitioners interested in setting up and realizing the benefits of an effective measurement program.

2014

Human Factors in Webserver Log File Analysis: A Controlled Experiment on Investigating Malicious Activity

Lucas Layman, Sylvain David Diffo, and Nico Zazworka

In Proc. of the 2014 Symposium and Bootcamp on the Science of Security (HotSoS ’14), Mar 2014

Abs

While automated methods are the first line of defense for detecting attacks on webservers, a human agent is required to understand the attacker’s intent and the attack process. The goal of this research is to understand the value of various log fields and the cognitive processes by which log information is grouped, searched, and correlated. Such knowledge will enable the development of human-focused log file investigation technologies. We performed controlled experiments with 65 subjects (IT professionals and novices) who investigated excerpts from six Webserver log files. Quantitative and qualitative data were gathered to: 1) analyze subject accuracy in identifying malicious activity; 2) identify the most useful pieces of log file information; and 3) understand the techniques and strategies used by subjects to process the information. Statistically significant effects were observed in the accuracy of identifying attacks and time taken depending on the type of attack. Systematic differences were also observed in the log fields used by high-performing and low-performing groups. The findings include: 1) new insights into how specific log data fields are used to effectively assess potentially malicious activity; 2) obfuscating factors in log data from a human cognitive perspective; and 3) practical implications for tools to support log file investigations. Copyright 2014 ACM.
A Methodology for Exposing Risk in Achieving Emergent System Properties

Lucas Layman, Victor R Basili, and Marvin V Zelkowitz

Transactions on Software Engineering Methodology, Mar 2014

Abs

Determining whether systems achieve desired emergent properties, such as safety or reliability, requires an analysis of the system as a whole, often in later development stages when changes are difficult and costly to implement. In this article we propose the Process Risk Indicator (PRI) methodology for analyzing and evaluating emergent properties early in the development cycle. A fundamental assumption of system engineering is that risk mitigation processes reduce system risks, yet these processes may also be a source of risk: (1) processes may not be appropriate for achieving the desired emergent property; or (2) processes may not be followed appropriately. PRI analyzes development process artifacts (e.g., designs pertaining to reliability or safety analysis reports) to quantify process risks that may lead to higher system risk.We applied PRI to the hazard analysis processes of a network-centric, Department of Defense system-of-systems and two NASA spaceflight projects to assess the risk of not achieving one such emergent property, software safety, during the early stages of the development lifecycle. The PRI methodology was used to create measurement baselines for process indicators of software safety risk, to identify risks in the hazard analysis process, and to provide feedback to projects for reducing these risks. \textcopyright 2014 ACM.
InViz: Instant visualization of security attacks

L. Layman, and N. Zazworka

In ACM International Conference Proceeding Series, Mar 2014

Abs

The InViz tool is a functional prototype that provides graphical visualizations of log file events to support real-time attack investigation. Through visualization, both experts and novices in cybersecurity can analyze patterns of application behavior and investigate potential cybersecurity attacks. The goal of this research is to identify and evaluate the cybersecurity information to visualize that reduces the amount of time required to perform cyber forensics.

2013

Technical Debt: Showing the Way for Better Transfer of Empirical Results

Forrest Shull, Davide Falessi, Carolyn Seaman, and 2 more authors

In Perspectives on the Future of Software Engineering: Essays in Honor of Dieter Rombach, Mar 2013

Abs

\textcopyright 2013 Springer-Verlag Berlin Heidelberg. All rights are reserved. In this chapter, we discuss recent progress and opportunities in empirical software engineering by focusing on a particular technology, Technical Debt (TD), which ties together many recent developments in the field. Recent advances in TD research are providing empiricists the chance to make more sophisticated recommendations that have observable impact on practice. TD uses a financial metaphor and provides a framework for articulating the notion of tradeoffs between the short-term benefits and the long-term costs of software development decisions. TD is seeing an explosion of interest in the practitioner community, and research in this area is quickly having an impact on practice. We argue that this is due to several strands of empirical research reaching a level of maturity that provides useful benefits to practitioners, who in turn provide excellent data to researchers. They key is providing observable benefit to practitioners, such as the ability to tie technical debt measures to business goals, and the ability to articulate more sophisticated value-based propositions regarding how to prioritize rework. TD is an interesting case study in how the maturing field of empirical software engineering research is paying dividends. It is only a little hyperbolic to call this a watershed moment for empirical study, where many areas of progress are coming to a head at the same time.
Debugging Revisited: Toward Understanding the Debugging Needs of Contemporary Software Developers

Lucas Layman, Madeline Diep, Meiyappan Nagappan, and 3 more authors

In 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement, Oct 2013

Abs

We know surprisingly little about how professional developers define debugging and the challenges they face in industrial environments. To begin exploring professional debugging challenges and needs, we conducted and analyzed interviews with 15 professional software engineers at Microsoft. The goals of this study are: 1) to understand how professional developers currently use information and tools to debug; 2) to identify new challenges in debugging in contemporary software development domains (web services, multithreaded/multicore programming); and 3) to identify the improvements in debugging support desired by these professionals that are needed from research. The interviews were coded to identify the most common information resources, techniques, challenges, and needs for debugging as articulated by the developers. The study reveals several debugging challenges faced by professionals, including: 1) the interaction of hypothesis instrumentationand software environment as a source of debugging difficulty; 2) the impact of log file information on accurate debugging of web services; and 3) the mismatch between the sequential human thought process and the non-sequential execution of multithreaded environments as source of difficulty. The interviewees also describe desired improvements to tools to support debugging, many of which have been discussed in research but not transitioned to practice.
Local versus Global Lessons for Defect Prediction and Effort Estimation

Tim Menzies, Andrew Butcher, David Cok, and 5 more authors

IEEE Transactions on Software Engineering, Jun 2013

Abs

Existing research is unclear on how to generate lessons learned for defect prediction and effort estimation. Should we seek lessons that are global to multiple projects or just local to particular projects? This paper aims to comparatively evaluate local versus global lessons learned for effort estimation and defect prediction. We applied automated clustering tools to effort and defect datasets from the PROMISE repository. Rule learners generated lessons learned from all the data, from local projects, or just from each cluster. The results indicate that the lessons learned after combining small parts of different data sources (i.e., the clusters) were superior to either generalizations formed over all the data or local lessons formed from particular projects. We conclude that when researchers attempt to draw lessons from some historical data source, they should 1) ignore any existing local divisions into multiple sources, 2) cluster across all available data, then 3) restrict the learning of lessons to the clusters from other sources that are nearest to the test data.
Automated classification of NASA anomalies using natural language processing techniques

Davide Falessi, and Lucas Layman

In 2013 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), Nov 2013

Abs

NASA anomaly databases are rich resources of software failure data in the field. These data are often captured in natural language that is not appropriate for trending or statistical analyses. This fast abstract describes a feasibility study of applying 60 natural language processing techniques for automatically classifying anomaly data to enable trend analyses.
Using Amazon’s Mechanical Turk for User Studies: Eight Things You Need to Know

Lucas Layman, and Gunnar Sigurdsson

In Proceedings of the 7th International Symposium on Empirical Software Engineering and Measurement (ESEM 2013), Nov 2013

Abs

Amazon’s Mechanical Turk is a crowdsourcing technology that enables requesters to create tasks to be completed by human agents in exchange for compensation. Researchers in computer science have successfully used this service to quickly reach large numbers of subjects for a relatively low cost. However, the Mechanical Turk’s model and policies introduce several experimental limitations and threats that must be controlled. In this short paper, we describe limitations imposed using Amazon’s Mechanical Turk during an experiment on cyber-attack investigation techniques. While the experiment was successful, we were forced to change our experimental design and had to recover from some costly mistakes. The goal of this short paper is to identify these limitations and pitfalls and provide eight considerations for experimental design so that other researchers can maximize the benefits of using the Mechanical Turk as a research platform.

2012

Toward Baselining Software Anomalies in NASA Missions

Lucas Layman, Marvin Zelkowitz, Victor Basili, and 1 more author

In 2012 IEEE 23rd International Symposium on Software Reliability Engineering Workshops, Nov 2012

Abs

In this fast abstract, we provide preliminary findings from an analysis of 14,500 spacecraft anomalies from unmanned NASA missions. We provide some baselines for the distributions of software vs. non-software anomalies in spaceflight systems, the risk ratings of software anomalies, and the corrective actions associated with software anomalies.

2011

A Case Study of Measuring Process Risk for Early Insights into Software Safety

Lucas Layman, Victor R Basili, Marvin V Zelkowitz, and 1 more author

In Proceedings of the 33rd ACM/IEEE International Conference on Software Engineering (ICSE ’11), Nov 2011

Abs

In this case study, we examine software safety risk in three flight hardware systems in NASA’s Constellation spaceflight program. We applied our Technical and Process Risk Measurement (TPRM) methodology to the Constellation hazard analysis process to quantify the technical and process risks involving software safety in the early design phase of these projects. We analyzed 154 hazard reports and collected metrics to measure the prevalence of software in hazards and the specificity of descriptions of software causes of hazardous conditions. We found that 49-70% of 154 hazardous conditions could be caused by software or software was involved in the prevention of the hazardous condition. We also found that 12-17% of the 2013 hazard causes involved software, and that 23-29% of all causes had a software control. The application of the TRPM methodology identified process risks in the application of the hazard analysis process itself that may lead to software safety risk. \textcopyright 2011 ACM.
A Methodology for Exposing Software Development Risk in Emergent System Properties

Victor R. Basili, Lucas Layman, and Marvin V. Zelkowitz

Nov 2011

2010

Obtaining Valid Safety Data for Software Safety Measurement and Process Improvement

Victor R Basili, Marvin V Zelkowitz, Lucas Layman, and 2 more authors

In Proceedings of the 4th ACM/IEEE International Symposium on Empirical Soiftware Engineering and Measurement (ESEM ’10), Nov 2010

Abs

We report on a preliminary case study to examine software safety risk in the early design phase of the NASA Constellation spaceflight program. Our goal is to provide NASA quality assurance managers with information regarding the ongoing state of software safety across the program. We examined 154 hazard reports created during the preliminary design phase of three major flight hardware systems within the Constellation program. Our purpose was two-fold: 1) to quantify the relative importance of software with respect to system safety; and 2) to identify potential risks due to incorrect application of the safety process, deficiencies in the safety process, or the lack of a defined process. One early outcome of this work was to show that there are structural deficiencies in collecting valid safety data that make software safety different from hardware safety. In our conclusions we present some of these deficiencies. \textcopyright 2010 ACM.
A Methodology for Mapping System Engineering Challenges to Recommended Approaches

Lucas Layman, Forrest Shull, Paul Componation, and 7 more authors

In Proceedings of the 4th Annual IEEE International Systems Conference, Nov 2010

Abs

Our current research is focused on identifying system engineering approaches that address four key development challenges in a tightly constrained, rapid reaction environment: 1) changing and emerging requirements; 2) conflicting stakeholder priorities; 3) concurrent sustainment and development activities; and 4) integration of independently evolving components. We are building a concept map of the key elements that form a strategic bridge between development challenges and the specific methods, processes, and tools that successfully address those challenges. In this paper, we present our methodology for constructing a robust mapping that incorporates interviews, surveys, and rigorous analysis methods. We summarize the results from interviews with sponsor personnel, the results of a best practices survey of 116 professionals, and qualitative analysis of the survey responses. \textcopyright2010 IEEE.
The Role and Quality of Software Safety in the NASA Constellation Program

Lucas Layman, Victor R. Basili, and Marvin V. Zelkowitz

Nov 2010
How Effective is Test Driven Development?

Burak Turhan, Lucas Layman, Madeline Diep, and 2 more authors

In Making Software: What Really Works, and Why We Believe It, Nov 2010
What Do We Know about Test-Driven Development

Forrest Shull, Grigori Melnik, Burak Turhan, and 3 more authors

IEEE Software, Nov 2010

Abs

What if someone argued that one of your basic conceptions about how to develop software was misguided? What would it take to change your mind? That’s essentially the dilemma faced by advocates of test-driven development (TDD). The TDD paradigm argues that the basic cycle of developing code and then testing it to make sure it does what it’s supposed to do-something drilled into most of us from the time we began learning software development- isn’t the most effective approach. TDD replaces the traditional "code then test" cycle. First, you develop test cases for a small increment of functionality; then you write code that makes those tests run correctly. After each increment, you refactor the code to maintain code quality.

2009

Evaluation of Systems Engineering Methods, Processes and Tools on Department of Defense and Intelligence Community Programs

R. Turner, F. Shull, B. Boehm, and 11 more authors

Nov 2009

Abs

Final Technical Report
Coordination in large-scale software teams

Andrew Begel, Nachiappan Nagappan, Christopher Poile, and 1 more author

In 2009 ICSE Workshop on Cooperative and Human Aspects on Software Engineering, May 2009

2008

Addressing Diverse Needs through a Balance of Agile and Plan-driven Software Development Methodologies in the Core Software Engineering Course

Lucas Layman, Laurie Williams, Kelli Slaten, and 2 more authors

International Journal of Engineering Education, May 2008

Abs

The software industry uses a mixture of plan-driven and agile techniques, and educators must prepare students for industry needs while creating an effective educational environment that appeals to a diverse student population. We describe the undergraduate course in software engineering at North Carolina State University, which teaches both agile and plan-driven practices while emphasizing collaborative and active learning. We present demographics, personality types, and learning styles from 400 students, and provide statistical analyses and student testimonials on the impact of our course. Students have reacted favorably to the course and are better prepared to meet the diverse needs of industry. \textcopyright 2008 TEMPUS Publications.
Iterative identification of fault-prone binaries using in-process metrics

Lucas Layman, Gunnar Kudrjavets, and Nachiappan Nagappan

In Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement - ESEM ’08, Oct 2008
Information Needs of Developers for Program Comprehension during Software Maintenance Tasks

Lucas Layman

Oct 2008
Mining software effort data: A preliminary analysis of Visual Studio Team System Data

Lucas Layman, Nachiappan Nagappan, Sam Guckenheimer, and 2 more authors

In Proceedings of the 2008 International Working Conference on Mining software repositories - MSR ’08, May 2008
Eleven Guidelines for Implementing Pair Programming in the Classroom

Laurie Williams, D. Scott McCrickard, Lucas Layman, and 1 more author

In Agile 2008 Conference, May 2008

Abs

Utilizing pair programming in the classroom requires specific classroom management techniques. We have created nine guidelines for successfully implementing pair programming in the classroom. These guidelines are based on pair programming experiences spanning seven years and over one thousand students at North Carolina State University. In Fall 2007, pair programming was adopted in the undergraduate human-computer interaction (HCI) course at Virginia Tech. We present the pair programming guidelines in the context of the HCI course, discuss how the guidelines were implemented, and evaluate the general applicability and sufficiency of the guidelines. We find that eight of the nine guidelines were applicable to the Virginia Tech experience. We amended our peer evaluation guideline to account for constantly supervised pairing, as was the case at Virginia Tech. We add two guidelines stating that a pair should always be working toward a common goal and that pairs should be encouraged to find their own answers to increase their independence and self-confidence.
MimEc: Intelligent User Notification of Faults in the Eclipse IDE

Lucas M Layman, Laurie A Williams, and Robert St. Amant

In 1st Workshop on Cooperative and Human Aspects of Software Engineering (CHASE ’08), May 2008

2007

Toward Reducing Fault Fix Time: Understanding Developer Behavior for the Design of Automated Fault Detection Tools

Lucas Layman, Laurie Williams, and Robert St. Amant

In First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007), Sep 2007

Abs

The longer a fault remains in the code from the time it was injected, the more time it will take to fix the fault. Increasingly, automated fault detection (AFD) tools are providing developers with prompt feedback on recently-introduced faults to reduce fault fix time. If however, the frequency and content of this feedback does not match the developer’s goals and/or workflow, the developer may ignore the information. We conducted a controlled study with 18 developers to explore what factors are used by developers to decide whether or not to address a fault when notified of the error. The findings of our study lead to several conjectures about the design of AFD tools to effectively notify developers of faults in the coding phase. The AFD tools should present fault information that is relevant to the primary programming task with accurate and precise descriptions. The fault severity and the specific timing of fault notification should be customizable. Finally, the AFD tool must be accurate and reliable to build trust with the developer.
Lab Partners: If They’re Good Enough for the Sciences, Why Aren’t They Good Enough for Us?

Laurie Williams, and Lucas Layman

In 20th Conference on Software Engineering Education and Training (CSEET’07), Jul 2007

Abs

Despite many professed benefits of collaboration, some computer science educators feel students need to master work individually, particularly in the courses early in the curriculum that feed into software engineering courses. In the natural sciences, however, students almost always work with one or more partners in the laboratory. What can computer science educators learn about collaborative lab settings from our natural science counterparts? We conducted a survey of science and computer science educators to compare views and use of collaboration in their classes. The positive and negative aspects of collaboration, as reported by the natural science educators, are strikingly similar to those of computer science educators. These results suggest that computer science educators should be more open to the use of collaborative labs, as is done in the natural sciences, for the overall benefit to students.
On the Impact of a Collaborative Pedagogy on African American Millennial Students in Software Engineering

Laurie Williams, Lucas Layman, Kelli M. Slaten, and 2 more authors

In 29th International Conference on Software Engineering (ICSE’07), May 2007

Abs

Millennial students (those born after 1982), particularly African Americans and women, have demonstrated a propensity toward collaborative activities. We conducted a collective case study at North Carolina State University and North Carolina A&T to ascertain the role of collaboration and social interaction in attracting and retaining students in information technology. Responses from semi-structured interviews with 11 representative African American students in these classes were coded and analyzed. The responses from these minority students were used to evolve a social interaction model. The conjectures generated from the model suggest that pair programming and agile software methodologies effectively create a collaborative environment that is desirable to Millennial students, male and female, and, with the new evidence, minority and majority. Additionally, the African American Millennial students enjoy learning from their peers and believe that a collaborative environment better prepares them for the "real world".
Personality types, learning styles, and an agile approach to software engineering education

L. Layman, T. Cornwell, and L. Williams

In Proceedings of the Thirty-Seventh SIGCSE Technical Symposium on Computer Science Education, May 2007

Abs

This paper describes an initiative at North Carolina State University in which the undergraduate software engineering class was restructured in layout and in presentation. The change was made from a lecture-based coursed that followed the waterfall method to a lab-oriented course emphasizing practical tools and agile processes. We examine the new course layout from the perspective of Myers-Briggs personality types and Felder-Silverman learning styles to discuss how the new software engineering class format appeals to a wide variety of students. The new course format resulted in some of the highest student evaluations in recent course history. It is now the standard for the undergraduate software engineering course at the university and has since been used in other North Carolina institutions. Copyright 2006 ACM.
Note to self: Make Assignments Meaningful

Lucas Layman, Laurie Williams, and Kelli Slaten

In Proceedings of the 28th SIGCSE Technical Symposium on Computer Science Education, Mar 2007
On the Impact of a Collaborative Pedagogy on African-American Millennial Students in Software Engineering

Laurie Williams, Lucas Layman, Kelli M Slaten, and 2 more authors

In , Mar 2007

2006

Examining the Compatibility of Student Pair Programmers

Laurie Williams, Lucas Layman, Jason Osborne, and 1 more author

In AGILE 2006 (AGILE’06), Mar 2006

Abs

Pair programming has been shown to be beneficial for both students and teaching staff in university courses. A two-phased study of 1350 students was conducted at North Carolina State University from 2002-2005 to determine if teaching staff can proactively form compatible pairs based upon any of the following factors: personality type, learning style, skill level, programming self esteem, work ethic, or time management preference. We examined compatibility among freshmen, advanced undergraduate and graduate student pair programmers. We have found that overall 93% of students are compatible with their partners. Students notably preferred to pair with a partner that he or she perceived to be of similar or higher skill level to them, which can be predicted by grouping students with similar grade point average. Additionally, pairs comprised of a sensor and an intuitor learning style seem to be compatible, and pairs with differing work ethic are generally not compatible
Intelligent User Notifaction to Expedite Awareness of Fault Code

Lucas Layman

In International Doctoral Symposium on Empirical Software Engineering (IDoESE ’06), Mar 2006
Changing Students’ Perceptions: An Analysis of the Supplementary Benefits of Collaborative Software Development

L. Layman

In 19th Conference on Software Engineering Education and Training (CSEET’06), Mar 2006

Abs

Collaborative work has been in use as an instructional tool to increase student understanding through collaborative learning and to improve student performance in computer science courses. However, little work has been done to understand how the act of collaboration, through pair programming or group work, impacts a student’s knowledge of the benefits and difficulties of collaborative work experience in collaborative work is essential preparation for professional software development. A study was conducted at North Carolina State University to assess changes in advanced undergraduate students’ perceptions of pair programming and collaboration. Student personality types, learning styles, and other characteristics were gathered during two semesters of an undergraduate software engineering course. The study found that, after experiencing pair programming, most students indicated a stronger preference to work with another student, believed that pairing made them more organized, and believed that pairing saved time on homework assignments. Students who disliked their collaborative experiences were predominantly reflective learners, introverts, and strong coders. Those students also cited that a non-participatory partner and difficulties scheduling meeting times outside of the classroom were primary reasons for disliking pair programming. Personality type and learning style had little effect on the changes in perceptions of collaboration
Essential Communication Practices for Extreme Programming in a Global Software Development Team

Lucas Layman, Laurie Williams, Daniela Damian, and 1 more author

Information and Software Technology, Mar 2006
Personality Types, Learning Styles, and an Agile Approach to Software Engineering Education

Lucas Layman, Travis Cornwell, and Laurie Williams

In Proceedings of the 37th SIGCSE Technical Symposium on Computer Science Education, Mar 2006
Motivations and Measurements in an Agile Case Study

Lucas Layman, Laurie Williams, and Lynn Cunningham

Journal of Systems Architecture, Mar 2006

2005

How and Why Collaborative Software Development Impacts the Software Engineering Course

Lucas Layman, Laurie Williams, Jason Osborne, and 8 more authors

In Proceedings Frontiers in Education 35th Annual Conference, Mar 2005

Abs

This paper presents the results of an initial quantitative investigation to assess a variety of factors that potentially affect the collaborative software development experience. This research was conducted with 119 students in two undergraduate software engineering classes at North Carolina State University. A survey was administered where students could reflect on their collaborative experiences. We analyzed these factors for interrelationships as well as for correlations with performance in the course, grade point average, and SAT scores. Our findings support the components of the proposed Social Interaction Model of Pair Programming. The substantiation of the Social Interaction Model of Pair Programming values suggests that they should be considered in course planning. We also find that work ethic and self-perceived programming ability positively correlate with GPA
Undergraduate Student Perceptions of Pair Programming and Agile Software Methodologies: Verifying a Model of Social Interaction

Kelli M Slaten, Maria Droujkova, Sarah B Berenson, and 2 more authors

In Agile Development Conference (ADC’05), Mar 2005

Abs

One of the reasons that undergraduate students, particularly women and minorities, can become disenchanted with computer science education is because software development is wrongly characterized as a solitary activity. We conducted a collective case study in a software engineering course at North Carolina State University to ascertain the effects of a collaborative pedagogy intervention on student perceptions. The pedagogy intervention was based upon the practices of agile software development with a focus on pair programming. Six representative students in the course participated in the study. Their perspectives helped validate a social interaction model of student views. The findings suggest that pair programming and agile software methodologies contribute to more effective learning opportunities for computer science students and that students understand and appreciate these benefits.
Rational Unified Process Evaluation Framework Version 1.0

Williams Krebs, Chih-Wei Ho, Laurie Williams, and 1 more author

Mar 2005
Personality Profiles and Learning Styles of Advanced Undergraduate Computer Science Students

Lucas Layman, Travis Cornwell, Laurie Williams, and 1 more author

Mar 2005
On establishing the essential components of a technology-dependent framework

Laurie Williams, Lucas Layman, and Pekka Abrahamsson

ACM SIGSOFT Software Engineering Notes, Jul 2005

2004

Extreme Programming Evaluation Framework for Object-Oriented Languages – Version 1.1

L Williams, W Krebs, L Layman, and 2 more authors

Jul 2004
Toward a Framework for Evaluating Extreme Programming

Laurie Williams, William Krebs, Lucas Layman, and 2 more authors

In Proceedings of the 8th International Conference on Evaluation and Assessment in Software Engineering (EASE ’04), Jul 2004

Abs

Software organizations are progressively adopting the development practices associated with the extreme programming (XP) methodology. Most reports on the efficacy of these practices are anecdotal. This paper provides a benchmark measurement framework for researchers and practitioners to express concretely the XP practices the organization has selected to adopt and/or modify, and the outcome thereof. The framework enables the necessary meta-analysis for combining families of case studies. The results of running framework-based case studies in various contexts will eventually constitute a body of knowledge of systematic, empirical evaluations of XP and its practices. Additionally, this benchmark provides a baseline framework that can be adapted for industrial case studies of other technologies and processes. To provide a foundation on the use of the framework, we present the initial validation of our XP evaluation framework based upon a year-long study of an IBM team that adopted a subset of XP practices.
Empirical investigation of the impact of extreme programming practices on software projects

Lucas Layman

In Companion to the 19th annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications - OOPSLA ’04, Oct 2004
Exploring Extreme Programming in Context: An Industrial Case Study

L Layman, L Williams, and L Cunningham

In Agile Development Conference 2004 (ADC’04), Oct 2004
Identifying Potential Deficiencies in Agile Requirements Engineering Practices

L Layman

Oct 2004
Motivations and Measurements in an Agile Case Study

L Layman, L Williams, and L Cunningham

In , Oct 2004
Exploring the Use of a Safe Subset of Extreme Programming: An Industrial Case Study

Laurie Williams, Lucas Layman, William Krebs, and 1 more author

Oct 2004

2003

Toward an XP Evaluation Framework

Laurie Williams, William Krebs, Lucas Layman, and 1 more author

Oct 2003
Are decomposition slices clones?

K. Gallagher, and L. Layman

In Proceedings of the 11th International Workshop on Program Comprehension (IWPC ’03), Oct 2003

Abs

When computing program slices on all variables in a system, we observed that many of these slices are the same. This leads to the question: Are we looking at software clones? We discuss the genesis of this phenomena and present some of the data observations that led to the question. The answer to our query is not immediately clear. We end by presenting arguments both pro and con. Supporting the affirmative, we observed that some slice-clones are evidently the result of the usual genesis of software clones: failure to note appropriate abstractions. Also, slice-clones assist in program comprehension by coalescing into one program fragment the computations on many different variables. Opposing the proposition, we note that slice-clones do not arise due to programmer intent or the copying of existing idioms.