Abstract
In this paper we present a parallel version for the algorithm BFQzip, we introduced in [Guerrini et al., BIOSTEC – BIOINFORMATICS 2022], that modifies the bases and quality scores components taking into account both information at the same time, while preserving variant calling. The resulting FASTQ file achieves better compression than the original data. Here, we introduce a strategy that splits the FASTQ file into t blocks and processes them in parallel independently by using the BFQzip algorithm. The resulting blocks with the modified bases and smoothed qualities are merged (in order) and compressed. We show that our strategy can improve the compression ratio of large FASTQ files by taking advantage of the redundancy of reads. When splitting into blocks, the reads belonging to the same portion of the genome could end up in different blocks. Therefore, we analyze how reordering reads before splitting the input FASTQ can improve the compression ratio as the number of threads increases. We also propose a paired-end mode that allows to exploit the paired-end information by processing blocks of FASTQ files in pairs.
Availability: The software is freely available at https://github.com/veronicaguerrini/BFQzip
An earlier version of this contribution appeared in the Proceedings of the 15th International Joint Conference on Biomedical Engineering Systems and Technologies: Bioinformatics [26].
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
By using the code/library at https://github.com/nicolaprezza/bwt2lcp.
- 2.
Note that the value Q does not depend on the cluster analyzed only in strategy (a).
- 3.
We recall that a paired-end read is a pair consisting of two reads (or mates), such that the second read occurs in the genome at a known distance after the first one.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
SNPs calling pipeline available at https://github.com/veronicaguerrini/BFQzip/blob/main/variant_calling/pipeline_SNPsCall.sh.
References
Sensitivity of string compressors and repetitiveness measures (2023). https://doi.org/10.1016/j.ic.2022.104999
Abouelhoda, M.I., Kurtz, S., Ohlebusch, E.: Replacing suffix trees with enhanced suffix arrays. J. Discrete Algorithms 2(1), 53–86 (2004). https://doi.org/10.1016/S1570-8667(03)00065-0
Adjeroh, D., Bell, T., Mukherjee, A.: The Burrows-Wheeler Transform: Data Compression, Suffix Arrays, and Pattern Matching. Springer, New York (2008). https://doi.org/10.1007/978-0-387-78909-5
Bauer, M., Cox, A., Rosone, G.: Lightweight algorithms for constructing and inverting the BWT of string collections. Theor. Comput. Sci. 483, 134–148 (2013). https://doi.org/10.1016/j.tcs.2012.02.002
Benoit, G., et al.: Reference-free compression of high throughput sequencing data with a probabilistic de Bruijn graph. BMC Bioinform. 16, 1–14 (2015). https://doi.org/10.1186/s12859-015-0709-7
Bentley, J.W., Gibney, D., Thankachan, S.V.: On the complexity of BWT-runs minimization via alphabet reordering. In: ESA. LIPIcs, vol. 173, pp. 15:1–15:13. Schloss Dagstuhl-Leibniz-Zentrum für Informatik, Dagstuhl, Germany (2020). https://doi.org/10.4230/LIPIcs.ESA.2020.15
Bonizzoni, P., Della Vedova, G., Pirola, Y., Previtali, M., Rizzi, R.: Multithread multistring Burrows-Wheeler transform and longest common prefix array. J. Comput. Biol. 26(9), 948–961 (2019). https://doi.org/10.1089/cmb.2018.0230
Bonomo, S., Mantaci, S., Restivo, A., Rosone, G., Sciortino, M.: Sorting conjugates and suffixes of words in a multiset. Int. J. Found. Comput. Sci. 25(08), 1161–1175 (2014)
Boucher, C., Cenzato, D., Lipták, Z., Rossi, M., Sciortino, M.: Computing the original eBWT faster, simpler, and with less memory. In: Lecroq, T., Touzet, H. (eds.) SPIRE 2021. LNCS, vol. 12944, pp. 129–142. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86692-1_11
Burrows, M., Wheeler, D.: A Block Sorting data Compression Algorithm. Technical report DIGITAL System Research Center (1994)
Cazaux, B., Rivals, E.: Linking BWT and XBW via Aho-Corasick automaton: applications to run-length encoding. In: CPM. LIPIcs, vol. 128, pp. 24:1–24:20. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, Dagstuhl, Germany (2019). https://doi.org/10.4230/LIPIcs.CPM.2019.24
Cenzato, D., Lipták, Z.: A theoretical and experimental analysis of BWT variants for string collections. In: Bannai, H., Holub, J. (eds.) 33rd Annual Symposium on Combinatorial Pattern Matching (CPM 2022). Leibniz International Proceedings in Informatics (LIPIcs), vol. 223, pp. 25:1–25:18. Schloss Dagstuhl - Leibniz-Zentrum für Informatik (2022). https://doi.org/10.4230/LIPIcs.CPM.2022.25
Chandak, S., Tatwawadi, K., Ochoa, I., Hernaez, M., Weissman, T.: SPRING: a next-generation compressor for FASTQ data. Bioinformatics 35(15), 2674–2676 (2018)
Chandak, S., Tatwawadi, K., Weissman, T.: Compression of genomic sequencing reads via hash-based reordering: algorithm and analysis. Bioinformatics 34(4), 558–567 (2017). https://doi.org/10.1093/bioinformatics/btx639
Cleary, J., Witten, I.: Data compression using adaptive coding and partial string matching. IEEE Trans. Commun. 32(4), 396–402 (1984). https://doi.org/10.1109/TCOM.1984.1096090
Cox, A., Bauer, M., Jakobi, T., Rosone, G.: Large-scale compression of genomic sequence databases with the Burrows-Wheeler transform. Bioinformatics 28(11), 1415–1419 (2012). https://doi.org/10.1093/bioinformatics/bts173
Cox, A.J., Jakobi, T., Rosone, G., Schulz-Trieglaff, O.B.: Comparing DNA sequence collections by direct comparison of compressed text indexes. In: Raphael, B., Tang, J. (eds.) WABI 2012. LNCS, vol. 7534, pp. 214–224. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33122-0_17
Deorowicz, S.: Fqsqueezer: k-mer-based compression of sequencing data. Sci. Rep. 10(1), 1–9 (2020)
DePristo, M.A., et al.: A framework for variation discovery and genotyping using next-generation DNA sequencing data. Nat. Genet. 43(5), 491–498 (2011). https://doi.org/10.1038/ng.806
Egidi, L., Louza, F.A., Manzini, G., Telles, G.P.: External memory BWT and LCP computation for sequence collections with applications. Algorithms Mol. Biol. 14(1), 6:1–6:15 (2019). https://doi.org/10.1186/s13015-019-0140-0
Egidi, L., Manzini, G.: Lightweight BWT and LCP merging via the gap algorithm. In: Fici, G., Sciortino, M., Venturini, R. (eds.) SPIRE 2017. LNCS, vol. 10508, pp. 176–190. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67428-5_15
Ferragina, P., Manzini, G.: Opportunistic data structures with applications. In: FOCS, pp. 390–398. IEEE Computer Society (2000). https://doi.org/10.1109/SFCS.2000.892127
Giuliani, S., Inenaga, S., Lipták, Z., Prezza, N., Sciortino, M., Toffanello, A.: Novel results on the number of runs of the Burrows-Wheeler-transform. In: Bureš, T., et al. (eds.) SOFSEM 2021. LNCS, vol. 12607, pp. 249–262. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-67731-2_18
Greenfield, D.L., Stegle, O., Rrustemi, A.: GeneCodeq: quality score compression and improved genotyping using a Bayesian framework. Bioinformatics 32(20), 3124–3132 (2016). https://doi.org/10.1093/bioinformatics/btw385
Guerrini, V., Louza, F., Rosone, G.: Metagenomic analysis through the extended Burrows-Wheeler transform. BMC Bioinform. 21, 21–25 (2020). https://doi.org/10.1186/s12859-020-03628-w
Guerrini., V., Louza., F., Rosone., G.: Lossy compressor preserving variant calling through extended BWT. In: Proceedings of the 15th International Joint Conference on Biomedical Engineering Systems and Technologies - BIOINFORMATICS, pp. 38–48. INSTICC, SciTePress (2022). https://doi.org/10.5220/0010834100003123
Hach, F., Numanagić, I., Alkan, C., Sahinalp, S.C.: SCALCE: boosting sequence compression algorithms using locally consistent encoding. Bioinformatics 28(23), 3051–3057 (2012). https://doi.org/10.1093/bioinformatics/bts593
Holland, R.C., Lynch, N.: Sequence squeeze: an open contest for sequence compression. GigaScience 2(1), 2047–217X (2013). https://doi.org/10.1186/2047-217X-2-5
Janin, L., Rosone, G., Cox, A.J.: Adaptive reference-free compression of sequence quality scores. Bioinformatics 30(1), 24–30 (2014). https://doi.org/10.1093/bioinformatics/btt257
Li, H.: Aligning sequence reads, clone sequences and assembly contigs with BWA-MEM. arXiv:1303.3997 (2013)
Li, H.: Fast construction of FM-index for long sequence reads. Bioinformatics 30(22), 3274–3275 (2014). https://doi.org/10.1093/bioinformatics/btu541, source code: https://github.com/lh3/ropebwt2
Li, H., Durbin, R.: Fast and accurate short read alignment with Burrows-Wheeler transform. Bioinform. 25(14), 1754–1760 (2009). https://doi.org/10.1093/bioinformatics/btp324
Louza, F.A., Telles, G.P., Gog, S., Prezza, N., Rosone, G.: gsufsort: constructing suffix arrays, LCP arrays and BWTs for string collections. Algorithms Mol. Biol. 15, 1–5 (2020)
Louza, F.A., Gog, S., Telles, G.P.: Inducing enhanced suffix arrays for string collections. Theor. Comput. Sci. 678, 22–39 (2017). https://doi.org/10.1016/j.tcs.2017.03.039
Mäkinen, V., Navarro, G., Sirén, J., Välimäki, N.: Storage and retrieval of highly repetitive sequence collections. J. Comput. Biol. 17(3), 281–308 (2010)
Manber, U., Myers, G.: Suffix arrays: a new method for on-line string searches. In: ACM-SIAM SODA, pp. 319–327 (1990)
Mantaci, S., Restivo, A., Rosone, G., Sciortino, M.: An extension of the Burrows-Wheeler Transform. Theoret. Comput. Sci. 387(3), 298–312 (2007)
Mantaci, S., Restivo, A., Rosone, G., Sciortino, M., Versari, L.: Measuring the clustering effect of BWT via RLE. Theor. Comput. Sci. 698, 79–87 (2017). https://doi.org/10.1016/j.tcs.2017.07.015
Moffat, A.: Implementing the PPM data compression scheme. IEEE Trans. Commun. 38(11), 1917–1921 (1990). https://doi.org/10.1109/26.61469
Na, J.C., et al.: FM-index of alignment with gaps. Theor. Comput. Sci. 710, 148–157 (2018)
Navarro, G.: Indexing highly repetitive string collections, part I: repetitiveness measures. ACM Comput. Surv. 54(2), 29:1–29:31 (2021)
Navarro, G.: Indexing highly repetitive string collections, part II: compressed indexes. ACM Comput. Surv. 54(2), 26:1–26:32 (2021)
Ochoa, I., Hernaez, M., Goldfeder, R., Weissman, T., Ashley, E.: Effect of lossy compression of quality scores on variant calling. Brief. Bioinform. 18(2), 183–194 (2016). https://doi.org/10.1093/bib/bbw011
Prezza, N., Pisanti, N., Sciortino, M., Rosone, G.: Variable-order reference-free variant discovery with the Burrows-Wheeler transform. BMC Bioinform. 21, 1–20 (2020). https://doi.org/10.1186/s12859-020-03586-3
Prezza, N., Rosone, G.: Space-efficient computation of the LCP array from the Burrows-Wheeler transform. In: Annual Symposium on Combinatorial Pattern Matching (CPM). vol. 128. Schloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing (2019). https://doi.org/10.4230/LIPIcs.CPM.2019.7
Prezza, N., Pisanti, N., Sciortino, M., Rosone, G.: SNPs detection by eBWT positional clustering. Algorithms Mol. Biol. 14(1), 3 (2019). https://doi.org/10.1186/s13015-019-0137-8
Prezza, N., Rosone, G.: Space-efficient construction of compressed suffix trees. Theoret. Comput. Sci. 852, 138–156 (2021). https://doi.org/10.1016/j.tcs.2020.11.024
Restivo, A., Rosone, G.: Balancing and clustering of words in the Burrows-Wheeler transform. Theor. Comput. Sci. 412(27), 3019–3032 (2011). https://doi.org/10.1016/j.tcs.2010.11.040
Roguski, L., Ochoa, I., Hernaez, M., Deorowicz, S.: FaStore: a space-saving solution for raw sequencing data. Bioinformatics 34(16), 2748–2756 (2018)
Rosone, G., Sciortino, M.: The Burrows-Wheeler transform between data compression and combinatorics on words. In: Bonizzoni, P., Brattka, V., Löwe, B. (eds.) CiE 2013. LNCS, vol. 7921, pp. 353–364. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39053-1_42
Shibuya, Y., Comin, M.: Better quality score compression through sequence-based quality smoothing. BMC Bioinform. 20-S(9), 302:1–302:11 (2019). https://doi.org/10.1186/s12859-019-2883-5
Simpson, J.T., Durbin, R.: Efficient construction of an assembly string graph using the FM-index. Bioinformatics 26(12), 367–373 (2010)
Yu, Y.W., Yorukoglu, D., Peng, J., Berger, B.: Quality score compression improves genotyping accuracy. Nat. Biotechnol. 33(3), 240–243 (2015). https://doi.org/10.1038/nbt.3170
Acknowledgments
The authors thank Prof. Nalvo Almeida (UFMS, Brazil) for granting access to the computer used in the experiments.
Funding
Work partially supported by MIUR, the Italian Ministry of Education, University and Research, under PRIN Project n. 20174LF3T8 AHeAD (“Efficient Algorithms for HArnessing Networked Data”). and by PNRR - M4C2 - Investimento 1.5, Ecosistema dell’Innovazione ECS00000017 - “THE - Tuscany Health Ecosystem” - Spoke 6 “Precision medicine & personalized healthcare”, funded by the European Commission under the NextGeneration EU programme.
F.A.L. acknowledges the financial support of CNPq (grant number 406418/2021-7) and FAPEMIG (grant number APQ-01217-22).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Guerrini, V., Louza, F.A., Rosone, G. (2023). Parallel Lossy Compression for Large FASTQ Files. In: Roque, A.C.A., et al. Biomedical Engineering Systems and Technologies. BIOSTEC 2022. Communications in Computer and Information Science, vol 1814. Springer, Cham. https://doi.org/10.1007/978-3-031-38854-5_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-38854-5_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-38853-8
Online ISBN: 978-3-031-38854-5
eBook Packages: Computer ScienceComputer Science (R0)