Droidbugs: A new benchmark to evaluate automated repair methods for Android apps
Automated program repair (APR) is an emerging field of research that sets out to develop methods of fixing software bugs automatically, without intervention by human programmers. After development, APR techniques are typically evaluated on several benchmarks, procedures that assess the performance of a computer program or system by running a series of standard tests.
Researchers at Universidade Federal de Goias and I4Soft have recently developed an introductory benchmark based on the analyses of 360 open projects for Android, each of them with more than 5,000 downloads. This benchmark, presented in a study pre-published on arXiv, contains 13 single bugs classified by the type of test that effectively exposed them.
"Our work is placed in the context of a very recent research field called automatic program repair," Celso Camilo, one of the researchers who carried out the study, told TechXplore. "Some automatic repair solutions have already been proposed and evaluated using specialized benchmarks. However, despite the huge use of Android applications, there is no benchmark designed to evaluate automated program repair techniques for mobile projects."
To address this shortcoming, the researchers developed DroidBugs, a new benchmark with real and reproducible bugs from mobile projects. This benchmark could help to evaluate new automatic program repair programs for Android apps.
"DroidBugs is a set of well-defined profiles of bugs in real open source Android applications," Camilo explained. "It organizes and provides 13 bugs, seven revealed by simulating an Android device and six analyzing the source code only. The benchmark presents the location of the bug in the source code, the buggy and fixed versions of the source code, and the test suite used to expose the errors."
Using Astor4Android, an APR tool developed by Camilo's research group at I4Soft, as well as two common fault localization strategies, the researchers tried to locate and fix bugs in Android applications. While doing this, they encountered a number of challenges that could be addressed by future studies.
"We provided the first public benchmark for automated program repair in the context of Android development," Camilo said. "We also applied existing APR algorithms and reported results on automatically localizing and repairing the bugs in the benchmark. With such findings, we were able to point out important challenges to producing a relevant benchmark based on open source Android projects, which may be useful for future efforts in this research field."
The study carried out by Camilo and his colleagues could lay the foundations for the development of further benchmarks to evaluate APR techniques for Android mobile applications. The researchers are now planning to broaden their research and explore new ways of improving DroidBugs.
"We intend to extend the types of bugs by increasing the number of applications covered by the benchmark," Camilo said. "We also plan to perform a deep investigation of the challenges for current APR techniques in fixing bugs of Android apps."
More information: DroidBugs: An Android Benchmark for Automatic Program Repair. arXiv:1809.07353 [cs.SE]. arxiv.org/abs/1809.07353
© 2018 Tech Xplore