Lu Xiao
Abstract
Abstract
Mocking frameworks provide convenient APIs, which create mock objects, manipulate their behavior, and verify their execution, for the purpose of isolating test dependencies in unit testing. This study contributes an in-depth empirical study of whether and how mocking frameworks are used in Apache projects. The key findings and insights of this study include: First, mocking frameworks are widely used in 66% of Apache Java projects, with Mockito, EasyMock, and PowerMock being the top three most popular frameworks. Larger-scale and more recent projects tend to observe a stronger need to use mocking frameworks. This underscores the importance of mocking in practice and related future research. Second, mocking is overall practiced quite selectively in software projects—not all test files use mocking, nor all dependencies of a test target are mocked. It calls for more future research to gain a more systematic understanding of when and what to mock to provide formal guidance to practitioners. On top of this, the intensity of mocking in different projects shows different trends in the projects’ evolution history—implying the compound effects of various factors, such as the pace of a project’s growth, the available resources, time pressure, and priority, etc. This points to an important future research direction in facilitating best mocking practices in software evolution. Furthermore, we revealed the most frequently used APIs in the three most popular frameworks, organized based on the function types. The top five APIs in each functional type of the three mocking frameworks usually take the majority (78% to 100%) of usage in Apache projects. This indicates that developers can focus on these APIs to quickly learn the common usage of these mocking frameworks. We further investigated informal methods of mocking, which do not rely on any mocking framework. These informal mocking methods point to potential sub-optimal mocking practices that could be improved, as well as limitations of existing mocking frameworks. Finally, we conducted a developer survey to collect additional insights regarding the above analysis based on their experience, which complements our analysis based on repository mining. Overall, this study offers practitioners profound empirical knowledge of how mocking frameworks are used in practice and sheds light on future research directions to enhancing mocking in practice.
-
Apache PDFBox | a java PDF library.
https://pdfbox.apache.org/
Google Scholar
-
Apache software foundation projects list.
https://projects.apache.org/projects.html
. Accessed 8 May 2023
Google Scholar
-
Arcuri A, Fraser G, Just R (2017) Private api access and functional mocking in automated unit test generation. In: 2017 IEEE international conference on software testing, verification and validation (ICST), IEEE, pp 126–137
Google Scholar
-
Barker D (2016) Web content management: systems, features, and best practices. “ O’Reilly Media, Inc.”
Google Scholar
-
Bertolino A (2007) Software testing research: achievements, challenges, dreams. In: Future of software engineering (FOSE’07), IEEE, pp 85–103.
https://doi.org/10.1109/FOSE.2007.25
Google Scholar
Digital Library
-
Briney K (2015) Data Management for Researchers: Organize, maintain and share your data for research success. Pelagic Publishing Ltd
Google Scholar
-
Buyya R, Dastjerdi AV (2016) Internet of Things: Principles and paradigms. Elsevier
Google Scholar
-
Assessing the health of open source communities
Computer
2006
39
5
89
91
10.1109/MC.2006.152
Google Scholar
Digital Library
-
Daka E, Fraser G (2014) A survey on unit testing practices and problems. In: 2014 IEEE 25th International symposium on software reliability engineering, IEEE, pp 201–211.
https://doi.org/10.1109/ISSRE.2014.11
Google Scholar
Digital Library
-
dev: Apache wink – index.
https://wink.apache.org/
Google Scholar
-
Apache and eclipse: comparing open source project incubators
IEEE Soft
2007
24
6
90
98
10.1109/MS.2007.157
Google Scholar
Digital Library
-
EasyMock.
https://easymock.org/
Google Scholar
-
Fazzini M, Choi C, Copia JM, Lee G, Kakehi Y, Gorla A, Orso A (2022) Use of test doubles in android testing: an in-depth investigation. In: Proceedings of the 44th international conference on software engineering, pp 2266–2278
Google Scholar
-
Fazzini M, Gorla A, Orso A (2020) A framework for automated test mocking of mobile apps. In: 2020 35th IEEE/ACM International conference on automated software engineering (ASE), IEEE, pp 1204–1208
Google Scholar
-
How important are high response rates for college surveys?
Rev Higher Educ
2017
40
2
245
265
10.1353/rhe.2017.0003
Google Scholar
Cross Ref
-
Freeman S, Mackinnon T, Pryce N, Walnes J (2004) jmock: supporting responsibility-based design with mock objects. In: Companion to the 19th annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications, pp 4–5
Google Scholar
-
Freeman S, Mackinnon T, Pryce N, Walnes J (2004) Mock roles, not objects. In: Companion to the 19th annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications, pp 236–246.
https://doi.org/10.1145/1028664.1028765
Google Scholar
Digital Library
-
Garousi V, Zhi J (2013) A survey of software testing practices in Canada. J Syst Softw 86(5):1354–1376.
https://doi.org/10.1016/j.jss.2012.12.051
.
https://www.sciencedirect.com/science/article/pii/S0164121212003561
Google Scholar
Digital Library
-
Henderson F (2017) Software engineering at google.
arXiv:1702.01715
Google Scholar
-
Hunt A, Thomas D (2004) Pragmatic unit testing in c# with nunit. Pragmatic Programmers
Google Scholar
-
Ieee standard glossary of software engineering terminology (1990) IEEE Std 610(12–1990):1–84.
https://doi.org/10.1109/IEEESTD.1990.101064
Google Scholar
Cross Ref
-
JUnit 5.
https://junit.org/junit5/
Google Scholar
-
Kaner C, Falk J, Nguyen HQ (1999) Testing computer software. John Wiley & Sons
Google Scholar
-
Karlesky M, Williams G, Bereza W, Fletcher M (2007) Mocking the embedded world: test-driven development, continuous integration, and design patterns. Proc. Emb. Systems Conf, CA, USA, pp 1518–1532
Google Scholar
-
Kim SS (2016) Mocking embedded hardware for software validation. Ph.D. thesis
Google Scholar
-
List of releases for the project.
https://github.com/gzhao9/Mock-Apache-Empirical-Study/blob/main/RQ1/project%20tags_info.csv
Google Scholar
-
Marri MR, Xie T, Tillmann N, De Halleux J, Schulte W (2009) An empirical study of testing file-system-dependent software with mock objects. In: 2009 ICSE Workshop on automation of software test, IEEE, pp 149–153.
https://doi.org/10.1007/s10664-018-9663-0
Google Scholar
Digital Library
-
Maven Repository: org.springframework » spring-mock.
https://mvnrepository.com/artifact/org.springframework/spring-mock
Google Scholar
-
Mockito release notes.
https://code.google.com/archive/p/mockito/wikis/ReleaseNotes.wiki
. Accessed 8 May 2023
Google Scholar
-
mockito.
https://site.mockito.org/
Google Scholar
-
Two case studies of open source software development: Apache and mozilla
ACM Trans Softw Eng Methodol (TOSEM)
2002
11
3
309
346
10.1145/567793.567795
Google Scholar
Digital Library
-
Mockus A, Fielding RT, Herbsleb J (2000) A case study of open source software development: the apache server. In: Proceedings of the 22nd international conference on software engineering, pp 263–272
Google Scholar
-
moq.
https://github.com/moq/moq4
Google Scholar
-
Mostafa S, Wang X (2014) An empirical study on the usage of mocking frameworks in software testing. In: 2014 14th international conference on quality software, IEEE, pp 127–132.
https://doi.org/10.1109/QSIC.2014.19
Google Scholar
Digital Library
-
Myers GJ, Badgett T, Thomas TM, Sandler C (2004) The art of software testing, vol 2. Wiley Online Library.
https://doi.org/10.1002/9781119202486
Google Scholar
Cross Ref
-
Nandigam J, Gudivada VN, Hamou-Lhadj A, Tao Y (2009) Interface-based object-oriented design with mock objects. In: 2009 Sixth international conference on information technology: new generations, IEEE, pp 713–718.
https://doi.org/10.1109/ITNG.2009.268
Google Scholar
Digital Library
-
NMock: A Dynamic Mock Object Library for .NET.
https://nmock.sourceforge.net/
Google Scholar
-
Pereira G, Hora A (2020) Assessing mock classes: an empirical study. In: 2020 IEEE International conference on software maintenance and evolution (ICSME), IEEE, pp 453–463.
https://doi.org/10.1109/ICSME46990.2020.00050
Google Scholar
Cross Ref
-
PowerMock framework site.
https://powermock.github.io/
Google Scholar
-
PowerMock framework site.
https://powermock.github.io/
Google Scholar
-
Projects by category in apache software foundation.
https://projects.apache.org/projects.html?category
Google Scholar
-
Rigby PC, German DM, Storey MA (2008) Open source software peer review practices: a case study of the apache server. In: Proceedings of the 30th international conference on Software engineering, pp 541–550
Google Scholar
-
A survey of unit testing practices
IEEE Softw
2006
23
4
22
29
10.1109/MS.2006.91
Google Scholar
Digital Library
-
The apache software foundation: Brian behlendorf
Computer
2012
45
10
8
9
10.1109/MC.2012.355
Google Scholar
Digital Library
-
Solms F, Marshall L (2016) Contract-based mocking for services-oriented development. In: Proceedings of the annual conference of the south african institute of computer scientists and information technologists, pp 1–8
Google Scholar
-
Mock objects for testing java systems
Empirical Softw Eng
2019
24
3
1461
1498
10.1007/s10664-018-9663-0
Google Scholar
Digital Library
-
Spadini D, Aniche M, Bruntink M, Bacchelli A (2017) To mock or not to mock? an empirical study on mocking practices. In: 2017 IEEE/ACM 14th International conference on mining software repositories (MSR), IEEE, pp 402–412.
https://doi.org/10.1109/MSR.2017.61
Google Scholar
Digital Library
-
Svensgård S, Henriksson J (2017) Mocking saas cloud for testing
Google Scholar
-
Taneja K, Zhang Y, Xie T (2010) Moda: automated test generation for database applications via mock objects. In: Proceedings of the IEEE/ACM international conference on Automated software engineering, pp 289–292
Google Scholar
-
Unittest.mock - mock object library.
https://docs.python.org/3/library/unittest.mock.html#module-unittest.mock
Google Scholar
-
Wang X (2021) Understanding and facilitating the usage of mocking frameworks for test dependency isolation. Ph.D. thesis, Stevens Institute of Technology
Google Scholar
-
Wang X, Xiao L, Yu T, Woepse A, Wong S (2022) From inheritance to mockito: An automatic refactoring approach. IEEE Trans Softw Eng 1–23.
https://doi.org/10.1109/TSE.2022.3231850
Google Scholar
Digital Library
-
Wang X, Xiao L, Yu T, Woepse A, Wong S (2021) An automatic refactoring framework for replacing test-production inheritance by mocking mechanism. In: Proceedings of the 29th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering, pp 540–552
Google Scholar
-
Web E (2013) Eclipse JDT™(Java development tools).
https://projects.eclipse.org/projects/eclipse.jdt
Google Scholar
-
Weiss M, Moroiu G, Zhao P (2006) Evolution of open source communities. In: Open source systems: IFIP working group 2.13 foundation on open source software, June 8–10, 2006, Como, Italy 2, Springer, pp 21–32
Google Scholar
-
Welcome to the Apache Struts project.
https://struts.apache.org/
Google Scholar
-
Why is it so bad to mock classes? — stackoverflow.com.
https://stackoverflow.com/questions/1595166/why-is-it-so-bad-to-mock-classes
. Accessed 19 Jul 2023
Google Scholar
-
Zhu H, Wei L, Wen M, Liu Y, Cheung SC, Sheng Q, Zhou C (2020) Mocksniffer: characterizing and recommending mocking decisions for unit tests. In: Proceedings of the 35th IEEE/ACM international conference on automated software engineering, pp 436–447
Google Scholar
-
Published in