Efficient DNA sequence partitioning using probabilistic subsets and hypergraphs

Sequence clustering is an important computational step in numerous bioinformatics applications such as high-throughput immune system characterization, marker-based diversity profiling, and reduced-complexity population genetic experiments. However, clustering very large datasets produced by high-throughput sequencers is CPU- and memory-intensive. As such, the analysis of large datasets is challenging and often requires compromising clustering sensitivity for speed. To address this limitation, we propose a new probabilistic data partitioning technique that divides large datasets of sequences of equal length into smaller, non-overlapping subsets that can subsequently be clustered using higher sensitivity algorithms at lower computational costs. In addition to substantially lowering resources requirements, this partitioning step provides an opportunity for effortless parallelization of DNA sequence clustering, and for the independent processing of each subset using more accurate algorithms. Our results show that our algorithm, which we implemented in a program called SLYMFAST can be used to cluster in a few hours datasets that would otherwise take weeks to cluster without partitioning first.