Dag Sjøberg

• Sjøberg, Dag & Bergersen, Gunnar Rye (2023). Improving the Reporting of Threats to Construct Validity. In Babar, Muhammad Ali (Eds.), EASE '23: Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering. Association for Computing Machinery (ACM). ISSN 979-8-4007-0044-6. doi: 10.1145/3593434.3593449. Full text in Research Archive Show summary

  The use of social robots in public spaces is becoming increasingly popular due to their ability to provide personalized services to users. However, the convergence of different technologies and software applications has raised concerns regarding security requirements, standards, and regulations. Specifically, there are significant concerns about the evolving threat landscape for software applications in public settings, where social robots interact without supervision and are in direct contact with threat actors. During the development of social robots software, developers and practitioners need practical tools to continuously assess their products’ security profiles. This paper presents a preventive approach to the dynamic evolving security landscape of Social Robots in Public Spaces (SRPS) using design science research (DSR) methodology to develop a security framework. The study investigates security threats, vulnerabilities, and risks associated with SRPS software development and analyzes existing related frameworks to design a security framework for SRPS software developers. The research aims to provide insights into the security aspects of SRPS software application development processes and contribute to developing effective security frameworks to mitigate evolving risks and ensure secure operation and acceptance in public spaces.

• Florea, Raluca Madalina; Stray, Viktoria & Sjøberg, Dag (2023). On the roles of software testers: An exploratory study. Journal of Systems and Software. ISSN 0164-1212. 204. doi: 10.1016/j.jss.2023.111742. Full text in Research Archive
• Holgeid, Knut Kjetil; Krogstie, John; Mikalef, Patrick; Saur, Eirik Eide & Sjøberg, Dag (2022). Benefits management and Information Technology work distribution. IET Software. ISSN 1751-8806. 16(4), p. 438–454. doi: 10.1049/sfw2.12062. Full text in Research Archive
• Sjøberg, Dag & Bergersen, Gunnar Rye (2022). Construct Validity in Software Engineering. IEEE Transactions on Software Engineering. ISSN 0098-5589. 49(3), p. 1374–1396. doi: 10.1109/TSE.2022.3176725.
• Soares de Toledo, Saulo; Martini, Antonio; Nguyen, Phu Hong & Sjøberg, Dag (2022). Accumulation and Prioritization of Architectural Debt in Three Companies Migrating to Microservices. IEEE Access. ISSN 2169-3536. 10, p. 37422–37445. doi: 10.1109/ACCESS.2022.3158648. Full text in Research Archive
• Gjelsten, Børge Kile; Bergersen, Gunnar R.; Sjøberg, Dag & Cutts, Quintin (2021). No Gender Difference in CS1 Grade for Students with Programming from High School: An Exploratory Study, Koli Calling '21: Proceedings of the 21st Koli Calling International Conference on Computing Education Research. ACM Digital Library. ISSN 978-1-4503-8488-9. doi: 10.1145/3488042.3488071.
• Soares de Toledo, Saulo; Martini, Antonio; Sjøberg, Dag; Przybyszewska, Agata & Skov Frandsen, Johannes (2021). Reducing Incidents in Microservices by Repaying Architectural Technical Debt, Proceedings of 2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE (Institute of Electrical and Electronics Engineers). ISSN 978-1-6654-2705-0. doi: 10.1109/SEAA53835.2021.00033. Full text in Research Archive
• Holgeid, Knut Kjetil; Jørgensen, Magne; Sjøberg, Dag & Krogstie, John (2021). Benefits management in software development: A systematic review of empirical studies. IET Software. ISSN 1751-8806. 15(1), p. 1–24. doi: 10.1049/sfw2.12007. Full text in Research Archive
• Soares De Toledo, Saulo; Martini, Antonio & Sjøberg, Dag (2021). Identifying architectural technical debt, principal, and interest in microservices: A multiple-case study. Journal of Systems and Software. ISSN 0164-1212. 177. doi: 10.1016/j.jss.2021.110968. Full text in Research Archive Show summary

  Background: Using a microservices architecture is a popular strategy for software organizations to deliver value to their customers fast and continuously. However, scientific knowledge on how to manage architectural debt in microservices is scarce. Objectives: In the context of microservices applications, this paper aims to identify architectural technical debts (ATDs), their costs, and their most common solutions. Method: We conducted an exploratory multiple case study by conducting 25 interviews with practitioners working with microservices in seven large companies. Results: We found 16 ATD issues, their negative impact (interest), and common solutions to repay each debt together with the related costs (principal). Two examples of critical ATD issues found were the use of shared databases that, if not properly planned, leads to potential breaks on services every time the database schema changes and bad API designs, which leads to coupling among teams. We identified ATDs occurring in different domains and stages of development and created a map of the relationships among those debts. Conclusion: The findings may guide organizations in developing microservices systems that better manage and avoid architectural debts.

• Soares De Toledo, Saulo; Martini, Antonio & I. K. Sjøberg, Dag (2020). Improving Agility by Managing Shared Libraries in Microservices. In Paasivaara, Maria & Kruchten, Philippe (Ed.), Lecture Notes in Business Information Processing, vol 396. Springer Berlin/Heidelberg. ISSN 978-3-030-58857-1. p. 195–202. doi: 10.1007/978-3-030-58858-8_20. Full text in Research Archive Show summary

  Using microservices is a way of supporting an agile architecture. However, if the microservices development is not properly managed, the teams’ development velocity may be affected, reducing agility and increasing architectural technical debt. This paper investigates how to manage the use of shared libraries in microservices to improve agility during development. We interviewed practitioners from four large international companies involved in microservices projects to identify problems when using shared libraries. Our results show that the participating companies had issues with shared libraries as follows: coupling among teams, delays on fixes due to overhead on libraries development teams, and need to maintain many versions of the libraries. Our results highlight that the use of shared libraries may hinder agility on microservices. Thus, their use should be restricted to situations where shared libraries cannot be replaced by a microservice and the costs of replicating the code on each service is very high.

• Stray, Viktoria; Moe, Nils Brede & Sjøberg, Dag (2020). Daily Stand-Up Meetings: Start Breaking the Rules! IEEE Software. ISSN 0740-7459. 37(3), p. 70–77. doi: 10.1109/MS.2018.2875988. Full text in Research Archive Show summary

  Daily stand-up meetings are commonly used for software teams to collaborate and exchange information, but conducting them in a way that benefits the whole team can be challenging. We describe factors that can affect meetings and propose recommendations for improving them.

• Soares De Toledo, Saulo; Martini, Antonio; Przybyszewska, Agata & Sjøberg, Dag (2019). Architectural Technical Debt in Microservices: A Case Study in a Large Company. In Avgeriou, Paris & Schmid, Klaus (Ed.), Proceedings of the Second International Conference on Technical Debt. IEEE Press. ISSN 978-1-7281-3371-3. doi: 10.1109/TechDebt.2019.00026. Full text in Research Archive Show summary

  Introduction: Software companies aim to achieve continuous delivery to constantly provide value to their customers. A popular strategy is to use microservices architecture. However, such an architecture is also subject to debt, which hinders the continuous delivery process and thus negatively affects the software released to the customers. Objectives: The aim of this study is to identify issues, solutions and risks related to Architecture Technical Debt in microservices. Method: We conducted an exploratory case study of a real life project with about 1000 services in a large, international company. Through qualitative analysis of documents and interviews, we investigated Architecture Technical Debt in the communication layer of a system with microservices architecture. Results: Our main contributions are a list of Architecture Technical Debt issues specific for the communication layer in a system with microservices architecture, as well as their associated negative impact (interest), a solution to repay the debt and the its cost (principal). Among the found Architecture Technical Debt issues were the existence of business logic in the communication layer and a high amount of point-to-point connections between services. The studied solution consists of the implementation of different canonical models specific to different domains, the removal of business logic from the communication layer, and migration from services to use the communication layer correctly. We also contributed with a list of possible risks that can affect the payment of the debt, as lack of funding and inadequate prioritization. Conclusion: We found issues, solutions and possible risks that are specific for microservices architectures not yet encountered in the current literature. Our results may be useful for practitioners that want to avoid or repay Technical Debt in their microservices architecture.

• Sjøberg, Dag (2018). An empirical study of WIP in kanban teams. In Oivo, Markku; Fernández, Daniel Méndez & Mockus, Audris (Ed.), Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM, Oulu, Finland — October 11 - 12, 2018. Association for Computing Machinery (ACM). ISSN 978-1-4503-5823-1. p. 13.1–13.8. doi: 10.1145/3239235.3239238. Show summary

  Background: Limiting the amount of Work-In-Progress (WIP) is considered a fundamental principle in Kanban software development. However, no published studies from real cases exist that indicate what an optimal WIP limit should be. Aims: The primary aim is to study the effect of WIP on the performance of a Kanban team. The secondary aim is to illustrate methodological challenges when attempting to identify an optimal or appropriate WIP limit. Method: A quantitative case study was conducted in a software company that provided information about more than 8,000 work items developed over four years by five teams. Relationships between WIP, lead time and productivity were analyzed. Results: WIP correlates with lead time; that is, lower WIP indicates shorter lead times, which is consistent with claims in the literature. However, WIP also correlates with productivity, which is inconsistent with the claim in the literature that a low WIP (still above a certain threshold) will improve productivity. The collected data set did not include sufficient information to measure aspects of quality. There are several threats to the way productivity was measured. Conclusions: Indicating an optimal WIP limit is difficult in the studied company because a changing WIP gives contrasting results on different team performance variables. Because the effect of WIP has not been quantitatively examined before, this study clearly needs to be replicated in other contexts. In addition, studies that include other team performance variables, such as various aspects of quality, are requested. The methodological challenges illustrated in this paper need to be addressed.

• Feldt, robert; Zimmermann, Thomas; Bergersen, Gunnar R.; Falessi, Davide; Jedlitschka, Andreas & Juristo, Natalia [Show all 12 contributors for this article] (2018). Four commentaries on the use of students and professionals in empirical software engineering experiments. Empirical Software Engineering. ISSN 1382-3256. 23(6), p. 3801–3820. doi: 10.1007/s10664-018-9655-0. Show summary

  The relative pros and cons of using students or practitioners in experiments in empirical software engineering have been discussed for a long time and continue to be an important topic. Following the recent publication of “Empirical software engineering experts on the use of students and professionals in experiments” by Falessi, Juristo, Wohlin, Turhan, Münch, Jedlitschka, and Oivo (EMSE, February 2018) we received a commentary by Sjøberg and Bergersen. Given that the topic is of great methodological interest to the community and requires nuanced treatment, we invited two editorial board members, Martin Shepperd and Per Runeson, respectively, to provide additional views

• Lindsjørn, Yngve; Bergersen, Gunnar R.; Dingsøyr, Torgeir & Sjøberg, Dag (2018). Teamwork Quality and Team Performance: Exploring Differences Between Small and Large Agile Projects. In Garbajosa, Juan; Wang, Xiaofeng & Aguiar, Ademar (Ed.), Agile Processes in Software Engineering and Extreme Programming, 19th International Conference, XP 2018, Proceedings. Springer. ISSN 978-3-319-91601-9. p. 267–274. doi: 10.1007/978-3-319-91602-6_19. Full text in Research Archive Show summary

  Agile principles were originally developed for small projects but are now widely used in larger projects with hundreds of developers. Teamwork quality is essential in any development work, but how does teamwork quality differ in small and large agile projects? We report from an explorative survey with 64 agile teams and 320 team members and team leaders, from 31 teams in small projects and 33 teams in large projects. For small projects, teamwork quality was considered by both team members and team leaders to primarily affect product quality. For large projects, the effect of teamwork quality on product quality was positive when it was rated by team members but was negative when rated by team leaders. At a finer granularity, the six dimensions of teamwork quality that we investigated affected team performance differently in small and large projects. These findings question to what extent findings from previous studies on teamwork in agile development in small projects apply to large projects.

• Sjøberg, Dag (2016). The relationship between software process, context and outcome. Lecture Notes in Computer Science (LNCS). ISSN 0302-9743. 10027, p. 3–11. doi: 10.1007/978-3-319-49094-6_1. Show summary

  Most practitioners and researchers agree that when developing software, process affects product, and the usefulness of a process depends on the context. However, which processes are most useful for a specific company or project is generally unknown. When studying the relation between context, process and product, one challenge is that experiments often lack realism, which makes the transfer of results to industry difficult. In contrast, most of the important factors vary beyond the researcher’s control in case studies, which makes it difficult to identify cause and effect relationships. This paper reports a study where realism was combined with control over certain context and process factors. Four companies developed the same system, and the price varied by a factor of six. Certain patterns of relationships were expected (expensive company, low cost, schedule overrun); others were unexpected (cheap company, maintainable system because of small code). The community needs to identify the most important relationships among process, context and outcome.

• Lindsjørn, Yngve; Sjøberg, Dag; Dingsøyr, Torgeir; Bergersen, Gunnar R. & Dybå, Tore (2016). Teamwork Quality and Project Success in Software Development: A Survey of Agile Development Teams. Journal of Systems and Software. ISSN 0164-1212. 122, p. 274–286. doi: 10.1016/j.jss.2016.09.028. Full text in Research Archive Show summary

  Small, self-directed teams are central in agile development. This article investigates the effect of team- work quality on team performance, learning and work satisfaction in agile software teams, and whether this effect differs from that of traditional software teams. A survey was administered to 477 respondents from 71 agile software teams in 26 companies and analyzed using structural equation modeling. A posi- tive effect of teamwork quality on team performance was found when team members and team leaders rated team performance. In contrast, a negligible effect was found when product owners rated team per- formance. The effect of teamwork quality on team members ́learning and work satisfaction was strongly positive, but was only rated by the team members. Despite claims of the importance of teamwork in ag- ile teams, this study did not find teamwork quality to be higher than in a similar survey on traditional teams. The effect of teamwork quality on team performance was only marginally greater for the agile teams than for the traditional teams.

• Sjøberg, Dag; Bergersen, Gunnar R. & Dybå, Tore (2016). Why Theory Matters. In Menzies, Tim; Williams, Laurie & Zimmermann, Thomas D. (Ed.), Perspectives on Data Science for Software Engineering. Elsevier. ISSN 9780128042069. p. 29–33. doi: 10.1016/B978-0-12-804206-9.00006-4. Show summary

  It is relatively easy to generate and acquire much data from software engineering activities. The challenge is to obtain meaning from the data that represents something true, rather than spurious. To increase knowledge and insight, more theories should be built and used.

• Dybå, Tore; Bergersen, Gunnar R. & Sjøberg, Dag (2016). Evidence-Based Software Engineering. In Menzies, Tim; Williams, Laurie & Zimmermann, Thomas D. (Ed.), Perspectives on Data Science for Software Engineering. Elsevier. ISSN 9780128042069. p. 149–153. doi: 10.1016/B978-0-12-804206-9.00029-5. Show summary

  A decade ago, Kitchenham, Dybå and Jørgensen coined the term and provided the foundations for evidence-based software engineering (EBSE). A trilogy of papers was written for researchers, practitioners, and educators. They suggested that practitioners consider EBSE as a mechanism to support and improve their technology adoption decisions, and that researchers should use systematic literature reviews as a methodology for performing unbiased aggregation of empirical results. This spurred significant international activity, and a renewed focus on research methods and theory, and on the future of empirical methods in SE research.

• Jørgensen, Magne; Dybå, Tore; Liestøl, Knut & Sjøberg, Dag (2016). Incorrect results in software engineering experiments: How to improve research practices. Journal of Systems and Software. ISSN 0164-1212. 116, p. 133–145. doi: 10.1016/j.jss.2015.03.065. Full text in Research Archive Show summary

  Context The trustworthiness of research results is a growing concern in many empirical disciplines. Aim The goals of this paper are to assess how much the trustworthiness of results reported in software engineering experiments is affected by researcher and publication bias, given typical statistical power and significance levels, and to suggest improved research practices. Method First, we conducted a small-scale survey to document the presence of researcher and publication biases in software engineering experiments. Then, we built a model that estimates the proportion of correct results for different levels of researcher and publication bias. A review of 150 randomly selected software engineering experiments published in the period 2002–2013 was conducted to provide input to the model. Results The survey indicates that researcher and publication bias is quite common. This finding is supported by the observation that the actual proportion of statistically significant results reported in the reviewed papers was about twice as high as the one expected assuming no researcher and publication bias. Our models suggest a high proportion of incorrect results even with quite conservative assumptions. Conclusion Research practices must improve to increase the trustworthiness of software engineering experiments. A key to this improvement is to avoid conducting studies with unsatisfactory low statistical power.

• Stray, Viktoria; Sjøberg, Dag & Dybå, Tore (2016). The Daily Stand-up Meeting: A Grounded Theory Study. Journal of Systems and Software. ISSN 0164-1212. 114, p. 101–124. doi: 10.1016/j.jss.2016.01.004. Full text in Research Archive Show summary

  The daily stand-up meeting is one of the most used agile practices but has rarely been the subject of empirical research. The present study aims to identify how daily stand-up meetings are conducted and what the attitudes towards them are. A grounded theory study with 12 software teams in three companies in Malaysia, Norway, Poland and the United Kingdom was conducted. We interviewed 60 people, observed 79 daily stand-up meetings and collected supplementary data. The factors that contributed the most to a positive attitude towards the daily stand-up meeting were information sharing with the team and the opportunity to discuss and solve problems. The factors that contributed the most to a negative attitude were status reporting to the manager and that the frequency of the meeting was perceived to be too high and the duration too long. Based on our results, we developed a grounded theory of daily stand-up meetings and proposed empirically based recommendations and guidelines on how to organize them. Organizations should be aware of the factors that may affect the attitude towards daily stand-up meetings and should consider our recommendations and guidelines to make this agile practice as valuable as possible.

• Bergersen, Gunnar R.; Sjøberg, Dag & Dybå, Tore (2014). Construction and Validation of an Instrument for Measuring Programming Skill. IEEE Transactions on Software Engineering. ISSN 0098-5589. 40(12), p. 1163–1184. doi: 10.1109/TSE.2014.2348997. Show summary

  Skilled workers are crucial to the success of software development. The current practice in research and industry for assessing programming skills is mostly to use proxy variables of skill, such as education, experience, and multiple-choice knowledge tests. There is as yet no valid and efficient way to measure programming skill. The aim of this research is to develop a valid instrument that measures programming skill by inferring skill directly from the performance on programming tasks. Over two days, 65 professional developers from eight countries solved 19 Java programming tasks. Based on the developers' performance, the Rasch measurement model was used to construct the instrument. The instrument was found to have satisfactory (internal) psychometric properties and correlated with external variables in compliance with theoretical expectations. Such an instrument has many implications for practice, for example, in job recruitment and project allocation.

• Stray, Viktoria Gulliksen; Lindsjørn, Yngve & Sjøberg, Dag (2013). Obstacles to Efficient Daily Meetings in Agile Development Projects: A Case Study. In Falessi, Davide (Eds.), ACM / IEEE International Symposium on Empirical Software Engineering and Measurement. Baltimore, 10-11 Oct. 2013. IEEE (Institute of Electrical and Electronics Engineers). ISSN 978-0-7695-5056-5. p. 95–102. doi: 10.1109/ESEM.2013.30. Show summary

  Context: Most of the software organizations that use agile methods organize daily team meetings. Aim: Our aim was to understand how daily meetings are conducted and identify obstacles that reduce their efficiency. Method: We observed 56 daily meetings and conducted 21 interviews in three different teams in two countries. We used the repertory grid technique in the interviews and to analyze the results. Results: We identified thirteen obstacles. The four most prominent ones were: (1) The daily meetings lasted too long (on average, 22 minutes instead of the scheduled 15 minutes). (2) In the meetings that were not self-organized, team members reported to the Scrum Master instead of sharing information among all team members. (3) The interruption caused by daily meetings required substantially more time than the actual meeting time due to overhead before and after the meetings. (4) Several team members had negative attitudes towards the daily meetings. Conclusion: Organizers of daily meetings should evaluate whether the obstacles we have identified are present in their organization and consider our suggestions to remove or reduce these obstacles.

• Kitchenham, Barbara; Sjøberg, Dag; Dybå, Tore; Brereton, O. Pearl; Budgen, David & Höst, Martin [Show all 7 contributors for this article] (2013). Trends in the Quality of Human-Centric Software Engineering Experiments-A Quasi-Experiment. IEEE Transactions on Software Engineering. ISSN 0098-5589. 39(7), p. 1002–1017. doi: 10.1109/TSE.2012.76. Show summary

  Context: Several text books and papers published between 2000 and 2002 have attempted to introduce experimental design and statistical methods to software engineers undertaking empirical studies. Objective: This paper investigates whether there has been an increase in the quality of human-centric experimental and quasi-experimental journal papers over the time period 1993 to 2010. Method: Seventy experimental and quasi-experimental papers published in four general software engineering journals in the years 1992-2002 and 2006-2010 were each assessed for quality by three empirical software engineering researchers using two quality assessment methods (a questionnaire-based method and a subjective overall assessment). Regression analysis was used to assess the relationship between paper quality and the year of publication, publication date group (before 2003 and after 2005), source journal, average coauthor experience, citation of statistical text books and papers, and paper length. The results were validated both by removing papers for which the quality score appeared unreliable and using an alternative quality measure. Results: Paper quality was significantly associated with year, citing general statistical texts, and paper length (p <; 0.05). Paper length did not reach significance when quality was measured using an overall subjective assessment. Conclusions: The quality of experimental and quasi-experimental software engineering papers appears to have improved gradually since 1993.

• Sjøberg, Dag; Yamashita, Aiko Amparo Fallas; Anda, Bente Cecilie D; Mockus, Audris & Dybå, Tore (2013). Quantifying the Effect of Code Smells on Maintenance Effort. IEEE Transactions on Software Engineering. ISSN 0098-5589. 39(8), p. 1144–1156. doi: 10.1109/TSE.2012.89. Show summary

  Context: Code smells are assumed to indicate bad design that leads to less maintainable code. However, this assumption has not been investigated in controlled studies with professional software developers. Aim: This paper investigates the relationship between code smells and maintenance effort. Method: Six developers were hired to perform three maintenance tasks each on four functionally equivalent Java systems originally implemented by different companies. Each developer spent three to four weeks. In total, they modified 298 Java files in the four systems. An Eclipse IDE plug-in measured the exact amount of time a developer spent maintaining each file. Regression analysis was used to explain the effort using file properties, including the number of smells. Results: None of the 12 investigated smells was significantly associated with increased effort after we adjusted for file size and the number of changes; Refused Bequest was significantly associated with decreased effort. File size and the number of changes explained almost all of the modeled variation in effort. Conclusion: The effects of the 12 smells on maintenance effort were limited. To reduce maintenance effort, a focus on reducing code size and the work practices that limit the number of changes may be more beneficial than refactoring code smells.

• Dybå, Tore; Sjøberg, Dag & Cruzes, Daniela Soares (2012). What works for whom, where, when, and why?: on the role of context in empirical software engineering. In Runeson, Per; Høst, Martin; Mendes, Emilia; Andrews, Anneliese Amschler & Harrison, Rachel (Ed.), ESEM '12. Proceedings of the ACM-IEEE International Symposium on Empirical Software Engineering and Measurement. IEEE conference proceedings. ISSN 978-1-4503-1056-7. p. 19–28. doi: 10.1145/2372251.2372256. Show summary

  Context is a central concept in empirical software engineering. It is one of the distinctive features of the discipline and it is an in-dispensable part of software practice. It is likely responsible for one of the most challenging methodological and theoretical problems: study-to-study variation in research findings. Still, empirical software engineering research is mostly concerned with attempts to identify universal relationships that are independent of how work settings and other contexts interact with the processes important to software practice. The aim of this paper is to provide an overview of how context affects empirical research and how empirical software engineering research can be better 'contextualized' in order to provide a better understanding of what works for whom, where, when, and why. We exemplify the importance of context with examples from recent systematic reviews and offer recommendations on the way forward.

• Sjøberg, Dag; Anda, Bente Cecilie D & Mockus, Audris (2012). Questioning software maintenance metrics: a comparative case study. In Runeson, Per; Høst, Martin; Mendes, Emilia; Andrews, Anneliese Amschler & Harrison, Rachel (Ed.), ESEM '12. Proceedings of the ACM-IEEE International Symposium on Empirical Software Engineering and Measurement. IEEE conference proceedings. ISSN 978-1-4503-1056-7. p. 107–110. doi: 10.1145/2372251.2372269.
• Sjøberg, Dag; Johnsen, Anders & Solberg, Jørgen (2012). Quantifying the Effect of Using Kanban versus Scrum: A Case Study. IEEE Software. ISSN 0740-7459. 29(5), p. 47–53. doi: 10.1109/MS.2012.110.
• Kitchenham, Barbara Ann; Sjøberg, Dag; Dybå, Tore; Pfahl, Dietmar Alfred; Brereton, Pearl & Budgen, David [Show all 8 contributors for this article] (2012). Three empirical studies on the agreement of reviewers about the quality of software engineering experiments. Information and Software Technology. ISSN 0950-5849. 54(8), p. 804–819. doi: 10.1016/j.infsof.2011.11.008.
• Bergersen, Gunnar R. & Sjøberg, Dag (2012). Evaluating Methods and Technologies in Software Engineering with Respect to Developers’ Skill Level. In Mendes, Emilia & Genero, Marcela (Ed.), Proceedings, 16th International Conference on Evaluation and Assessment in Software Engineering. IET Research Journals. ISSN 978-1-84919-541-6. p. 101–110.
• Bergersen, Gunnar; Hannay, Jo Erskine; Sjøberg, Dag; Dybå, Tore & Karahasanovic, Amela (2011). Inferring Skill from Tests of Programming Performance: Combining Time and Quality, Proceedings of International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE conference proceedings. ISSN 978-1-4577-2203-5. p. 305–314. doi: 10.1109/esem.2011.39. Show summary

  The skills of software developers are important to the success of software projects. Also, when studying the general effect of a tool or method, it is important to control for individual differences in skill. However, the way skill is assessed is often ad hoc, or based on unvalidated methods. According to established test theory, validated tests of skill should infer skill levels from well-defined performance measures on multiple, small, representative tasks. In this respect, we show how time and quality, which are often analyzed separately, can be combined as task performance and subsequently be aggregated as an approximation of skill. Our results show significant positive correlations between our proposed measures of skill and other variables, such as seniority, lines of code written, and self-evaluated expertise. The method for combining time and quality is a promising first step to measuring programming skill in both industry and research settings.

• Olbrich, Steffen; Cruzes, Daniela Soares & Sjøberg, Dag (2010). Are all code smells harmful? A study of God Classes and Brain Classes in the evolution of three open source systems. Proceedings - Conferense on Software Maintenance. ISSN 1063-6773. 1, p. 1–10. doi: 10.1109/ICSM.2010.5609564.
• Hannay, Jo Erskine; Arisholm, Erik; Engvik, Harald & Sjøberg, Dag (2010). Effects of Personality on Pair Programming. IEEE Transactions on Software Engineering. ISSN 0098-5589. 36(1), p. 61–80. doi: 10.1109/TSE.2009.41.
• Sjøberg, Dag & Grimstad, Stein (2010). Software Engineering: Why - How - What - What's Next. In Tveito, Aslak; Bruaset, Are Magnus & Lysne, Olav (Ed.), Simula Research Laboratory - by thinking constantly about it. Springer. ISSN 978-3-642-01155-9. p. 363–368.
• Sjøberg, Dag (2010). The Industry is our Lab – Organization and Conducting of Empirical Studies in Software Engineering at Simula. In Tveito, Aslak; Bruaset, Are Magnus & Lysne, Olav (Ed.), Simula Research Laboratory - by thinking constantly about it. Springer. ISSN 978-3-642-01155-9. p. 443–458.
• Sjøberg, Dag; Odberg, Erik & Warlo, Borger (2010). The Challenge of Assessing and Controlling Complexity in a Large Portfolio of Software Systems. In Hall, Tracy; Høst, Martin; Niazi, Mahmood; Mishra, Alok & Falessi, Davide (Ed.), Product Focused Software Development and Process Improvement (PROFES 2010, Limerick, Ireland, June 21-23. 2010). Universitalia. ISSN 978-88-6507-067-3. p. 75–78.
• Kitchenham, B.; Sjøberg, Dag; Brereton, O. Pearl; Budgen, David; Dybå, Tore & Høst, Martin [Show all 8 contributors for this article] (2010). Can we evaluate the quality of software engineering experiments? In Morisio, Maurizio (Eds.), Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement. IEEE (Institute of Electrical and Electronics Engineers). ISSN 978-1-4503-0039-1.
• Sjøberg, Dag (2010). Confronting the Myth of Rapid Obsolescence in Computing Research. Communications of the ACM. ISSN 0001-0782. 53(9), p. 62–67. doi: 10.1145/1810891.1810911.
• Følstad, Asbjørn; Anda, Bente Cecilie D & Sjøberg, Dag (2010). The usability inspection performance of work-domain experts: An empirical study. Interacting with computers. ISSN 0953-5438. 22(2), p. 75–87. doi: 10.1016/j.intcom.2009.09.001.
• Hannay, Jo Erskine; Dybå, Tore; Arisholm, Erik & Sjøberg, Dag (2009). The effectiveness of pair programming: A meta-analysis. Information and Software Technology. ISSN 0950-5849. 51(7), p. 1110–1122. doi: 10.1016/j.infsof.2009.02.001.
• Yamashita, Aiko Amparo Fallas; Benestad, Hans Christian; Anda, Bente Cecilie D; Arnstad, P.E.; Sjøberg, Dag & Moonen, L. (2009). Using Concept Mapping for Maintainability Assessments. In Carver, Jeffrey (Eds.), 3rd IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM 2009). IEEE (Institute of Electrical and Electronics Engineers). ISSN 978-1-4244-4842-5. p. 378–389.
• Kampenes, Vigdis By; Dybå, T.; Hannay, Jo Erskine & Sjøberg, Dag (2009). A systematic review of quasi-experiments in software engineering. Information and Software Technology. ISSN 0950-5849. 51(1), p. 71–82. doi: 10.1016/j.infsof.2008.04.006.
• Karahasanovic, Amela; Hinkel, Unni Nyhamar; Sjøberg, Dag & Thomas, R. (2009). Comparing of feedback-collection and think-aloud methods in program comprehension studies. Behaviour and Information Technology. ISSN 0144-929X. 28(2), p. 139–164. doi: 10.1080/01449290701682761.
• Anda, Bente Cecilie D; Sjøberg, Dag & Mockus, Audris (2009). Variability and Reproducibility in Software Engineering: A Study of Four Companies that Developed the Same System. IEEE Transactions on Software Engineering. ISSN 0098-5589. 35(3), p. 407–429. doi: 10.1109/TSE.2008.89.
• Sjøberg, Dag; Dybå, T.; Anda, Bente Cecilie D & Hannay, Jo Erskine (2008). Building Theories in Software Engineering. In Shull, Forrest; Singer, Janice & Sjøberg, Dag (Ed.), Advanced Topics in Empirical Software Engineering. Springer. ISSN 978-1-84800-043-8.
• Sjøberg, Dag; Dybå, Tore & Jørgensen, Magne (2007). The Future of Empirical Methods in Software Engineering Research. In Briand, Lionel & Wolf, Alexander (Ed.), Future of Software Engineering. IEEEE-CS. ISSN 0-7695-2829-5. p. 358–378.
• Sjøberg, Dag (2007). Knowledge Acquisition in Software Engineering Requires Sharing of Data and Artifacts, Empirical Software Engineering Issues: Critical Assessment and Future Directions. Springer. ISSN 978-3-540-71300-5. p. 77–82.
• Sjøberg, Dag (2007). Documenting Theories, Empirical Software Engineering Issues: Critical Assessment and Future Directions. Springer. ISSN 978-3-540-71300-5. p. 111–114.
• Basili, Victor R.; Zelkowitz, Marvin V.; Sjøberg, Dag & Johnson, Philip (2007). Protocols in the use of Empirical Software Engineering Artifacts. Empirical Software Engineering. ISSN 1382-3256. 12(1), p. 107–119.
• Dybå, Tore; Arisholm, Erik; Sjøberg, Dag; Hannay, J.E. & Shull, Forrest (2007). Are Two Heads Better than One? On the Effectiveness of Pair Programming. IEEE Software. ISSN 0740-7459. 24(6), p. 12–15.
• Arisholm, Erik; Gallis, Hans; Dybå, Tore & Sjøberg, Dag (2007). Evaluating Pair Programming with Respect to System Complexity and Programmer Expertise. IEEE Transactions on Software Engineering. ISSN 0098-5589. 333(2), p. 65–86.
• Kampenes, Vigdis By; Dyba, T; Hannay, Jo Erskine & Sjøberg, Dag (2007). A systematic review of effect size in software engineering experiments. Information and Software Technology. ISSN 0950-5849. 49, p. 1073–1086. doi: 10.1016/j.infsof.2007.02.015.
• Hannay, Jo Erskine; Sjøberg, Dag & Dybå, Tore (2007). A Systematic Review of Theory Use in Software Engineering Experiments. IEEE Transactions on Software Engineering. ISSN 0098-5589. 33(2), p. 87–107.
• Jørgensen, Magne & Sjøberg, Dag (2006). Expert Estimation of Software Development Work. In Madhavji, Nazim; Fernandez-Ramli, Juan & Perry, Dewayne (Ed.), Software Evolution and Feedback: Theory and Practice. IEEE Press. ISSN 0-470-87180-6. p. 523–527.
• Dybå, Tore; Kampenes, Vigdis By & Sjøberg, Dag (2006). A systematic review of statistical power in software engineering experiments. Information and Software Technology. ISSN 0950-5849. 48(8), p. 745–755.
• Karahasanovic, Amela; Fjuk, Annita; Sjøberg, Dag & Thomas, R. (2006). Revealing object-oriented comprehension by feedback collection. In Fjuk, Annita; Karahasanovic, Amela & Kaasbøll, Jens Johan (Ed.), Comprehensive Object-Oriented Learning: The Learner's Perspective. Informing Science Press. ISSN 83-922337-4-3. p. 111–130.
• Conradi, Reidar; Dybå, Tore; Sjøberg, Dag I. K. & Ulsund, Tor (2006). Lessons Learned and Recommendations from two Large Norwegian SPI Programmes. In Conradi, Reidar; Dybå, Tore; Sjøberg, Dag I. K. & Ulsund, Tor (Ed.), Software Process Improvement: Results and Experience from the Field. Springer. ISSN 3-540-32178-0. p. 3–18. Show summary

  Software development is an experimental discipline, i.e. somewhat unpredictable. This suggests that software processes improvement should be based on the continuous iteration of characterization, goal setting, selection of improved technology, monitoring and analysis of its effects. This paper describes experiences from the empirical studies in two large SPI programmes in Norway. Five main lessons were learned: 1) It is a challenge for the industrial partners to invest enough resources in SPI activities. 2) The research partners must learn to know the companies, and 3) they must work as a multicompetent and coherent unit towards them. 4) Any SPI initiative must show visible, short-term payoff. 5) Establishing a solid baseline from which to improve is unrealistic. Based on these lessons, a set of operational recommendations for other researchers in the area are proposed. Also published in Flavio Oquendo (Ed.), 9th Int'l. Workshop on Software Process Technology, Helsinki, Finland, ISBN 3-540-40764-2

• Krogstie, John; Jahr, Arthur & Sjøberg, Dag I. K. (2006). A longitudinal study of development and maintenance in Norway: Report from the 2003 investigation. Information and Software Technology. ISSN 0950-5849. 48, p. 993–1005.
• Benestad, Hans Christian; Arisholm, Erik & Sjøberg, Dag (2005). How to Recruit Professionals as Subjects in Software Engineering Experiments. In Hustad, E.; Munkvold, B.E.; Rolland, K & Flak, L.S. (Ed.), IRIS (Information Systems Research in Scandinavia). Department of Information Systems. ISSN 82-7117-554-8.
• Anda, Bente Cecilie D & Sjøberg, Dag (2005). Investigating the Role of Use Cases in the Construction of Class Diagrams. Empirical Software Engineering. ISSN 1382-3256. 10(3), p. 285–309.
• Karahasanovic, Amela; Anda, Bente Cecilie D; Arisholm, Erik; Hove, SE; Jørgensen, Magne & Sjøberg, Dag [Show all 7 contributors for this article] (2005). Collecting Feedback during Software Engineering Experiments. Empirical Software Engineering. ISSN 1382-3256. 10(2), p. 113–147.
• Sjøberg, Dag; Hannay, J.E.; Kampenes, V.B.; Hansen, O; Karahasanovic, Amela & Rekdal, A.C. [Show all 7 contributors for this article] (2005). A Survey of Controlled Experiments in Software Engineering. IEEE Transactions on Software Engineering. ISSN 0098-5589. 31(9), p. 733–753.
• Karahasanovic, Amela; Fjuk, A; Sjøberg, Dag & Thomas, R. (2004). A Controlled Experiment to Evaluate the Reactivity and Usefulness of the Think-Aloud Tool. In Khosrowpour, M (Eds.), Information Resources Management Association International Conference (IRMA'04), Computer Interaction Track. Idea Group Publishing. p. 1033–1034.
• Jørgensen, Magne & Sjøberg, Dag (2004). Generalization and Theory-Building in Software Engineering Research. In Linkman, Stephen (Eds.), Empirical Assessment in Software Engineering (EASE2004). IEEE (Institute of Electrical and Electronics Engineers). ISSN 0-7695-2163-0. p. 29–36.
• Arisholm, Erik & Sjøberg, Dag (2004). Evaluating the Effect of a Delegated versus Centralized Control Style on the Maintainability of Object-Oriented Software. IEEE Transactions on Software Engineering. ISSN 0098-5589. 30(8), p. 521–534.
• Jørgensen, Magne & Sjøberg, Dag (2004). The impact of customer expectation on software development effort estimates. International Journal of Project Management. ISSN 0263-7863. 22(4), p. 317–325.

View all works in Cristin

• Shull, Forrest; Singer, Janice & Sjøberg, Dag (2008). Advanced Topics in Empirical Software Engineering. Springer. ISBN 978-1-84800-043-8. 388 p.
• Conradi, Reidar; Dybå, Tore; Sjøberg, Dag I. K. & Ulsund, Tor (2006). Software Process Improvement: Results and Experience from the Field. Springer. ISBN 3-540-32178-0. 413 p. Show summary

  This book reports on results and experience from three Norwegian, industrial R&D project on Software Process Improvement (SPI) - SPIQ, PROFIT and SPIKE - with emphasis on the latter. The book is organized in five sections, with totally 20 previously published papers. Section 1 is about SPI principles/general, and has 5 papers. Section 2: Knowledge Management for SPI and has 5 papers. Section 3 is about Process Modelling and Electronic Process Guides and has 4 papers. Section 4 is about Estimation Methods and has 3 papers. Section 5 is about Empirical Studies in OO and Component-based Systems and has 3 papers. The audience is software professionals as well as researchers and graduate students.

View all works in Cristin

• Sjøberg, Dag (2023). Tall opplyser. [Newspaper]. Klassekampen.
• Sjøberg, Dag (2023). AI Engineer. [Business/trade/industry journal]. kode24.
• Sjøberg, Dag (2023). Studie antyder at IT-fag ikke bør undervises på engelsk. Det blir ikke lett, mener professor. Digi.no.
• Bergersen, Gunnar R. & Sjøberg, Dag Ingar Kondrup (2023). Education of software developers in a fast-growing job market: Giving more students with "non-standard" backgrounds an opportunity. Show summary

  Software development is a fast-growing occupation worldwide and has grown by 50% in Norway during the last five years. Unlike many other professional occupations, system developers are seldom required to have a formal education to work professionally. For example, anyone with the right skills can build open-source software or offer freelance services worldwide on a project-by-project basis. Currently, educational institutions cannot educate enough developers to fill new jobs. However, there is also a discrepancy between what type of students that academic educations will admit and, potentially, could admit. In this context, a student with a "standard" background would enter a programme after upper secondary education with all the study requirements in place and a goal to complete a BSc or MSc. In contrast, a student with a "non-standard" background would typically be older, have prior work or education experience, and have different interests and motivations to study a few selected topics. However, this may not be possible as many would-be students lack advanced math, a formal requirement for attending many courses. We report results on an ongoing longitudinal study of software development students (N=854) in our department over four years. In addition to analyzing prior grades, math specialization, and programming-related electives from upper secondary school, the students provided self-reports on their prior experience, interests, motivations, and aspirations for studying software development. We also contrast these variables with grades and study credit production and show that a substantial group of students with non-standard backgrounds appear to do equally well or better than the students with standard backgrounds. We conclude that even though commonly established requirements for studying software development should be retained for students with standard backgrounds, one should revise how students with non-standard backgrounds are admitted. As calls for greater diversity in society linger, we highlight the importance of greater flexibility in university requirements for studying software development so that the software industry's increasing needs are met to a greater extent than it is today.

• Sjøberg, Dag (2022). Keynote: Construct Validity in Software Engineering.
• Sjøberg, Dag (2021). Utenlandske forskere: Jobbene nordmenn ikke vil ha. [Newspaper]. Aftenposten.
• Sjøberg, Dag (2018). Forskning er skriving. Forskerforum. ISSN 0800-1715. p. 43–43.
• Tobiassen, Markus; Bergersen, Gunnar R. & Sjøberg, Dag (2016). Ser butikk i it-flauser. [Newspaper]. Dagens Næringsliv.
• Sjøberg, Dag (2016). Programmering i skolen er hverken frelse eller rør. [Newspaper]. Aftenposten.
• Sjøberg, Dag (2015). Programmering bør være allmennkunnskap. [Newspaper]. Dagens Næringsliv.
• Sjøberg, Dag (2015). Bruk av IT-konsulenter i offentlig sektor. [Newspaper]. Bergens Tidende.
• Sjøberg, Dag (2015). Vil ikke forske på it. [Newspaper]. Dagens Næringsliv.
• Sjøberg, Dag (2014). Spekulativ sykkelforskning. Forskning.no. ISSN 1891-635X.
• Sjøberg, Dag (2013). Viktigste faktorer for å redusere teknisk gjeld.
• Sjøberg, Dag (2013). Smidig må finne sin form. [Newspaper]. Computerworld.
• Sjøberg, Dag (2013). Kjemper innbitt mot stammespråket. [Newspaper]. Computerworld.
• Sjøberg, Dag (2012). Organisering av offentlige IT-prosjekter. [Newspaper]. Aftenposten.
• Sjøberg, Dag (2012). Har brukt 425 millioner på IT-konsulenter. [Newspaper]. Aftenposten.
• Sjøberg, Dag (2012). Smidig forskning. [Newspaper]. Computerworld.
• Sjøberg, Dag (2012). Smidig utvikling krever opplæring. [Newspaper]. Computerworld.
• Sjøberg, Dag (2012). Skandalene er kommet for å bli. [Newspaper]. Aftenposten.
• Sjøberg, Dag (2012). Kompleksiteten vil bli enorm dersom man skal samle alt på ett sted. [Newspaper]. Aftenposten.
• Sjøberg, Dag (2011). En kvantitativ studie av Kanban versus Scrum.
• Sjøberg, Dag (2011). Suksess med Kanban. [Newspaper]. Computerworld.
• Sjøberg, Dag (2011). Empirical software research: an interview with Dag Sjøberg, University of Oslo, Norway. ACM Ubiquity. ISSN 1530-2180. doi: 10.1145/1998372.1998374.
• Mockus, Audris; Anda, Bente Cecilie D & Sjøberg, Dag (2010). Experiences from Replicating a Case Study to Investigate Reproducibility of Software Development.
• Sjøberg, Dag; Dybå, Tore; Conradi, Reidar; Hoff, Per Morten & Gundersen, Christer (2010). Derfor trenger Norge mer IKT-forskning. ComputerWorld Norge. ISSN 1501-6595. p. 24–24. Show summary

  Dette er første motkronikk av 3. des. 2010 til startkronikk i Computerworld 19. nov. 2010: "http://www.idi.ntnu.no/grupper/su/publ/reidar/cw.19.11.startkronikk.pdf". Se også Conradi's andre motkronikk i Computerworld, 17. des. 2010: http://www.idi.ntnu.no/grupper/su/publ/reidar/cw.17.12.motkronikk2-rc.pdf.

• Sjøberg, Dag (2010). IT-forskningen er treg. Forskerforum. ISSN 0800-1715. p. 32–33.
• Sjøberg, Dag (2010). Universitetet i Oslo satser på Smidig. [Newspaper]. ComputerWorld.
• Benestad, Hans Christian; Arisholm, Erik & Sjøberg, Dag (2005). How to Recruit Professionals as Subjects in Software Engineering Experiments.
• LaToza, Thomas D; Ko, Amy J.; Shepherd, David C; Sjøberg, Dag & Xie, Benjamin (2023). Theories of Programming (Dagstuhl Seminar 22231). Schloss Dagstuhl - Leibniz-Zentrum fur Informatik. Full text in Research Archive
• Bergersen, Gunnar R.; Sjøberg, Dag & Dybå, Tore (2015). Measuring programming skill. Akademika forlag. ISSN 1501-7710.

View all works in Cristin