BOOTABLE: Bioinformatics benchmark tool suite for applications and hardware

Highlights

• Implementation of a benchmark tool suite based on bioinformatics applications.

• Convenient installation and usage design.

• Resource consumption and scaling investigation of any desired tool.

• Generic tool integration.

Abstract

The interest in analyzing biological data on a large scale has grown over the last years. Bioinformatics applications play an important role when it comes to the analysis of huge amounts of data. Due to the large amount of biological data and/or large problem spaces a considerable amount of computing resources is required to answer the raised research questions. In order to estimate which underlying hardware might be the most suitable for the bioinformatics tools applied, a well-defined benchmark suite is required. Such a benchmark suite can get useful in the case of purchasing hardware and even further for larger projects with the goal to establish a bioinformatics compute infrastructure. With this paper we present BOOTABLE, our bioinformatic benchmark suite. BOOTABLE currently contains seven popular and widely used bioinformatic applications representing a broad spectrum of usage characteristics. It further includes an automated installation procedure and all required datasets. Furthermore it includes functionalities to test any desired application with regards to resource consumption and scaling behavior.

Introduction

The success of sequencing technologies led to a large growth of genomics data on the scale of tera- and petabytes. At the same time, the demand for analytical methods that can analyze the created data on a large scale increased. The research field of bioinformatics has become more and more important to provide algorithms, methods and applications for researchers to examine the created data and extract meaningful parts to answer biological questions. In order to handle the huge amount of data and analyze them, a rather large amount of computing resources is required. Bioinformatic algorithms and applications mostly try to solve NP-Hard problems or harder [1]. In order to solve such problems in a relevant time or find a sufficient approximation, large computing resources are necessary. But also other research topics settled in the field of bioinformatics, or around it, benefit from large available computing resources like molecular dynamics simulation in computational chemistry or machine learning methods. These fields also handle large datasets, albeit not in the range of petabytes but the calculations for the simulation and training tasks are very complex and would benefit from large computing resources. No matter which discipline in bioinformatics or related fields you are focused on, there is a large demand for computer-aided methods requiring computational resources. The required resources can be available as a commercial computing cloud like [2], [3], [4] or academic clouds like the de.NBI cloud [5], bwCloud [6], or a hybrid one like Helix Nebula [7]. No matter how the computational resources are provided, at some point it has to be decided which kind of hardware has to be procured.

Usually the required computational hardware has to be purchased/acquired or you have the choice between already existing hardware. To decide which hardware should be used, it would make sense to test your common bioinformatics applications or at least the topic of the used tools like sequence analysis tools [8], protein folding prediction tools [9] or molecular dynamics simulation tools [10]. Questions raised could be for example: How robust is the underlying hardware? How fast can it deliver the application results? Is the number of compute cores more important than the clock rate? How much Random-Access Memory (RAM) is required? All these and many more questions need to be considered in order to purchase and use computing hardware in an efficient and cost-saving way.

To test different hardware you will need some kind of test environment consisting of a range of tools and suitable datasets. All these base elements combined would lead to a so-called benchmarking suite that handles all the measurements. In order to get comparable results a benchmark suite has to fulfill some constraints. First, every tested tool needs to be installed in exactly the same version and in the same way. Second, if it needs to be compiled, it must be the same compiler with the same parameters to exclude any variation and therefore deviations in the measured values. In addition to the applications, the data used to run the desired tools, also has to be the same and even further the datasets has to be chosen carefully, because their size or complexity will have a direct impact on the runtime. If we summarize all these requirements above we are talking about a reproducible deployment that must guarantee a minimum standard of equality. Such a proposed benchmark tool would have applications in different real world scenarios aside from a hardware procurement process. A choice of examples are presented in the following.

In order to buy and test hardware you normally contact the hardware vendor of your trust. The hardware vendor or reseller, in most cases, has no clue about your used applications whether on how to install them, which datasets are required, neither how to execute them. Furthermore, hardware vendors might have other opinions than a researcher about which results of a benchmark are meaningful for the performance of the used tools. Of course you could explain them how to install and run the applications and what you would like to know in the end, but this can get time-consuming.

An other point from the sight of the hardware operator is to focus on a well formed utilization of the underlying hardware in terms that requested resources are used wisely by the user and not wasted and therefore prevent other users from using the resources. In order to give users an estimation about which amount of resources would make sense, the operator will need some measurements of the used tools. For large and commercial providers this might not be interesting as the users pay for the allocated resources but for academic and also more specialized compute centers, supporting a specific community and have to care about their resources this might be worth the effort to investigate the scaling behavior of the used tools and applications.

Even the hardware vendors directly would benefit from a benchmark suite. They would not have to care about how to install the tools, run the tools or find suitable datasets for them. This possibility could lead to systems specialized on specific workloads, especially for bioinformatics.

From the view of application developers it would be a real help to test a tool regarding the utilized resources and also the scaling behavior to see where the limits or bottlenecks are and eventually to overcome them. For example it would not make sense to increase the throughput of an application or speed up the underlying algorithm if the bottleneck is the read/write speed of the disk. But to reveal a limit like this, a supporting tool that does all the measurements would be helpful.