Fuzzing to Identify Undiscovered Bugs in Scientific Software

Abstract

Scientific software is defined as software that aids in research, testing or design of scientific models that are used to explain and predict the behavior of real objects or systems in a variety of scientific disciplines. Bugs in scientific software can cause large-scale consequences, for example, error-prone scientific results can lead to ineffectual projects and embarrassing retractions, incorrect financial transactions can cause tremendous monetary loss, etc. However, scientific software is typically developed by novices who are not well-equipped with bug finding tools. In this research, we investigate methods to systematically find undiscovered bugs in scientific software packages written in Julia. Julia is a programming language designed specifically for scientific and numeric computing. The goal of our research is to help scientific software developers find undiscovered bugs using fuzzing. Fuzzing is defined as a software testing technique which incorporates automated random input generation and injection to a software in the hopes of triggering an error condition or fault. We investigate 20 open-source Julia repositories collected from GitHub and apply fuzzing on these repositories. We have developed fuzzers in Python for four programs and have done some preliminary analysis on the documented crash reports. We find that there are several unhandled exception conditions in them, for example, we found 5 bugs in the Julia FFTW package, and 4 bugs in the Julia HTTP package.