Abstract
Detection of highly similar sequences within genomic collections has a number of applications, including the assembly of expressed sequence tag data, genome comparison, and clustering sequence collections for improved search speed and accuracy. While several approaches exist for this task, they are becoming infeasible — either in space or in time — as genomic collections continue to grow at a rapid pace. In this paper we present an approach based on document fingerprinting for identifying highly similar sequences. Our approach uses a modest amount of memory and executes in a time roughly proportional to the size of the collection. We demonstrate substantial speed improvements compared to the CD-HIT algorithm, the most successful existing approach for clustering large protein sequence collections.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Altschul, S., Gish, W., Miller, W., Myers, E., Lipman, D.: Basic local alignment search tool. Journal of Molecular Biology 215(3), 403–410 (1990)
Altschul, S., Madden, T., Schaffer, A., Zhang, J., Zhang, Z., Miller, W., Lipman, D.: Gapped BLAST and PSI–BLAST: A new generation of protein database search programs. Nucleic Acids Research 25(17), 3389–3402 (1997)
Bernstein, Y., Zobel, J.: A scalable system for identifying co-derivative documents. In: Apostolico, A., Melucci, M. (eds.) Proc. String Processing and Information Retrieval Symposium (SPIRE), Padova, Italy. Springer, Heidelberg (2004)
Bernstein, Y., Zobel, J.: Redundant documents and search effectiveness. In: Chowdhury, A., Fuhr, N., Ronthaler, M., Schek, H., Teiken, W. (eds.) Proc. CIKM conference, Bremen, Germany, pp. 736–743. ACM Press, New York (2005)
Brin, S., Davis, J., Garc´ıa-Molina, H.: Copy detection mechanisms for digital documents. In: Proceedings of the ACM SIGMOD Annual Conference, pp. 398–409 (1995)
Broder, A.Z., Glassman, S.C., Manasse, M.S., Zweig, G.: Syntactic clustering of the web. Computer Networks and ISDN Systems 29(8-13), 1157–1166 (1997)
Buckley, C., Voorhees, E.M.: Evaluating evaluation measure stability. In: Proc. ACM SIGIR conference, pp. 33–40. ACM Press, New York (2000)
Burke, J., Davison, D., Hide, W.: d2 cluster: A validated method for clustering EST and full-length DNA sequences. Genome Research 9(11), 1135–1142 (1999)
Cameron, M., Williams, H.E., Cannane, A.: Improved gapped alignment in BLAST. IEEE Transactions on Computational Biology and Bioinformatics 1(3), 116–129 (2004)
Cameron, M., Williams, H.E., Cannane, A.: A deterministic finite automaton for faster protein hit detection in BLAST. Journal of Computational Biology (2005) (to appear)
Chandonia, J., Hon, G., Walker, N., Conte, L.L., Koehl, P., Levitt, M., Brenner, S.: The ASTRAL compendium in 2004. Nucleic Acids Research 32, D189–D192 (2004)
Chao, K., Pearson, W., Miller, W.: Aligning two sequences within a specified diagonal band. Computer Applications in the Biosciences 8(5), 481–487 (1992)
Fetterly, D., Manasse, M., Najork, M.: On the evolution of clusters of near-duplicate web pages. In: Baeza-Yates, R. (ed.) Proc. 1st Latin American Web Congress, pp. 37–45. IEEE, Santiago (2003)
Grossi, R., Vitter, J.S.: Compressed suffix arrays and suffix trees with applications to text indexing and string matching (extended abstract). In: STOC 2000: Proceedings of the thirty-second annual ACM symposium on Theory of computing, pp. 397–406. ACM Press, New York (2000)
Gusfield, D.: Algorithms on Strings, Trees, and Sequences. Cambridge University Press, Cambridge (1997)
Heintze, N.: Scalable document fingerprinting. In: 1996 USENIX Workshop on Electronic Commerce (1996)
Holm, L., Sander, C.: Removing near-neighbour redundancy from large protein sequence collections. Bioinformatics 14(5), 423–429 (1998)
Kurtz, S., Phillippy, A., Delcher, A., Smoot, M., Shumway, M., Antonescu, C., Salzberg, S.: Versatile and open software for comparing large genomes. Genome Biology 5(2) (2004)
Li, W., Jaroszewski, L., Godzik, A.: Clustering of highly homologous sequences to reduce the size of large protein databases. Bioinformatics 17(3), 282–283 (2001a)
Li, W., Jaroszewski, L., Godzik, A.: Tolerating some redundancy significantly speeds up clustering of large protein databases. Bioinformatics 18, 77–82 (2001b)
Li, W., Jaroszewski, L., Godzik, A.: Sequence clustering strategies improve remote homology recognitions while reducing search times. Protein Engineering 15(8), 643–649 (2002)
Malde, K., Coward, E., Jonassen, I.: Fast sequence clustering using a suffix array algorithm. Bioinformatics 19(10), 1221–1226 (2003)
Manber, U.: Finding similar files in a large file system, in Proceedings of the USENIX Winter, Technical Conference, San Fransisco, CA, USA, pp. 1–10 (1994)
Manber, U., Myers, G.: Suffix arrays: a new method for on-line string searches. SIAM Journal on Computing 22(5), 935–948 (1993)
Park, J., Holm, L., Heger, A., Chothia, C.: RSDB: representative sequence databases have high information content. Bioinformatics 16(5), 458–464 (2000)
Pearson, W., Lipman, D.: Improved tools for biological sequence comparison. Proceedings of the National Academy of Sciences USA 85(8), 2444–2448 (1988)
Shivakumar, N., García-Molina, H.: Finding near-replicas of documents on the web. In: WEBDB: International Workshop on the World Wide Web and Databases, WebDB. Springer, Heidelberg (1999)
Smith, T., Waterman, M.: Identification of common molecular subsequences. Journal of Molecular Biology 147(1), 195–197 (1981)
Witten, I.H., Moffat, A., Bell, T.C.: Managing Gigabytes: Compressing and Indexing Documents and Images. Morgan Kauffman, San Francisco (1999)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Bernstein, Y., Cameron, M. (2006). Fast Discovery of Similar Sequences in Large Genomic Collections. In: Lalmas, M., MacFarlane, A., Rüger, S., Tombros, A., Tsikrika, T., Yavlinsky, A. (eds) Advances in Information Retrieval. ECIR 2006. Lecture Notes in Computer Science, vol 3936. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11735106_38
Download citation
DOI: https://doi.org/10.1007/11735106_38
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-33347-0
Online ISBN: 978-3-540-33348-7
eBook Packages: Computer ScienceComputer Science (R0)