2024-10-10, 17:30–18:00 (Europe/Luxembourg), Room C1.03.05
We present Metamorphic Security Testing (MST), a testing automation approach that integrates test input generation strategies inspired by mutational fuzzing and automatically detects vulnerabilities by verifying general relations that should always hold between the outputs produced with valid and mutated inputs. It enables engineers to specify metamorphic relations (MRs) that capture how to modify a valid input as an attacker would do and specify the relation between the outputs of the valid and modified input. We developed a framework that integrates an Eclipse plugin to specify MRs and automatically tests the system. Further, we defined a catalogue of 76 system-agnostic MRs for Web systems that can discover 45% of the vulnerability types concerning the violations of MITRE security design principles. Our empirical results show that MST can detect 85% of the vulnerabilities in Joomla and Jenkins with few false alarms. Ongoing research includes the automated generation of MRs from specification documents by leveraging large language models.
Security testing aims to verify that the software meets its security properties. In modern systems, the effectiveness of security testing is complicated by a wide input space and vulnerabilities being detectable only by exploring a specific portion of such input space (e.g., a combination of corner cases). In turn, to effectively perform security testing, it is necessary to exercise the system with a very large set of inputs. Full automation is thus required to lower costs and increase security testing effectiveness. However, to achieve such automation, it is not sufficient to automatically derive test inputs, but it is necessary to address the oracle problem, which refers to the challenge of distinguishing correct from incorrect behaviour for a given system input; for example, in a Web system, determine the correctness of the response received after a specific HTTP GET request performed by a certain user. Traditional testing approaches require the manual specification of the expected output for each input, which is infeasible when a large number of inputs is automatically generated. To address these challenges, we propose Metamorphic Security Testing (MST), a metamorphic testing approach that integrates test input generation strategies inspired by mutational fuzzing and alleviates the oracle problem by verifying general relations that should always hold between the outputs generated by valid and mutated inputs. It enables engineers to specify metamorphic relations (MRs) that capture how to modify a valid input as an attacker would do and specify the relation between the outputs of the valid and invalid input. We defined a domain-specific language accompanied by an Eclipse editor. Also, we developed a framework that automatically collects the input data and transforms the MRs into executable Java code to automatically perform security testing. It automatically tests software systems to detect vulnerabilities based on the relations and collected data. Currently, we provide a catalogue of 76 system-agnostic MRs to automate security testing in Web systems. It covers 39% of the OWASP security testing activities not automated by state-of-the-art techniques; further, our MRs can automatically discover 45% of the vulnerability types due to violations of MITRE security design principles. We evaluated MST effectiveness and scalability with two well-known Web systems (i.e., Jenkins and Joomla). It automatically detected 85% of their vulnerabilities and showed a high specificity (99.81% of the generated inputs do not lead to a false positive). Finally, we demonstrated the applicability of the approach to test robotic systems and desktop applications. Ongoing research includes the automated generation of MRs from specification documents by leveraging large language models.
Fabrizio PASTORE received his PhD degree from the University of Milano – Bicocca, Italy, in 2010. His research interests are in software engineering, focusing on software testing, and program analysis. Fabrizio Pastore has worked as Assistant professor at the University of Milano – Bicocca, as Research associate at SnT, and as Post-doctoral researcher at the University of Lugano (Switzerland). Fabrizio is now part of the Software Verification and Validation group.