Skip to main content
SpringerLink
Log in

Storage and Retrieval of Large Data Sets: Dimensionality Reduction and Nearest Neighbour Search

  • Conference paper
Contemporary Computing (IC3 2012)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 306))

Included in the following conference series:

Abstract

Storing and querying are two important issues that need to be addressed while designing an information retrieval system for a large and high-dimensional data set. In this work, we discuss about tackling such data, specifically about the nearest neighbour search and the efficient storage layout to store such data. The data set used in the current work has been taken from an online source called ZINC, a repository for drug like chemical structures. Processing a high dimensional data is a tough task hence dimensionality reduction should be employed. Here for dimensionality reduction is achieved through a filter-based feature selection method, based on correlation fractal dimension (CFD) discrimination measure, is used. The number of dimensions using the correlation fractal dimension are reduced from 58 to 7. To identify the nearest neighbours for a given chemical structure Tanimoto similarity coefficient is used with these reduced set of features. The nearest neighbours identified using the Tanimoto measure are stored in a storage layout known as modified inverted file. Nearest neighbours for a query can be retrieved back from the storage layout, with just one read operation from the data file thereby reducing the time for retrieval.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Cormen, T.H., Leiserson, C.E., Rivest, R.L.: Introduction to algorithms. The MIT Press, London (2000)

    Google Scholar 

  2. Fukunaga, K.: Introduction to Statistical Pattern Recognition. Academic Press Professional, Inc., San Diego (1990)

    MATH  Google Scholar 

  3. Pearson, K.: On lines and planes of closest t to systems of points in space. Philiosophical Magazine 2, 559–572 (1901)

    Article  Google Scholar 

  4. Spearman, C.: General intelligence objectively determined and measured. American Journal of Psychology 15, 206–221 (1904)

    Google Scholar 

  5. Torgerson, W.S.: Multidimensional scaling I: Theory and method. Psychometrika 17, 401–419 (1952)

    Article  MATH  MathSciNet  Google Scholar 

  6. Nguyen, D.V., Rocke, D.M.: Tumor classication by partial least squares using microarray gene expression data. Bioinformatics 18, 39–50 (2002)

    Article  Google Scholar 

  7. Maaten, L.V., Postma, E., Herik, J.V.: Dimensionality Reduction: A Comparative Review. Technical report: TiCC TR 2009-005 (2009)

    Google Scholar 

  8. Lejsek, H., Asmundsson, F.H., Jonsson, B.P., Amsaleg, L.: NV-Tree: An Efficient Disk-Based Index for Approximate Search in Very Large High-Dimensional Collections. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 869–883 (2009)

    Article  Google Scholar 

  9. Lin, H.-Y., Huang, P.-W., Hsu, K.-H.: A new indexing method with high storage utilization and retrieval efficiency for large spatial databases. Information and Software Technology 49, 817–826 (2007)

    Article  Google Scholar 

  10. ZINC- A free database for virtual screening, http://www.zinc.docking.org

  11. Sankara Rao, A., Durga Bhavani, S., Sobha Rani, T., Bapi, R.S., Narahari Sastry, G.: Study of Diversity and Similarity of Large Chemical Databases Using Tanimoto Measure. In: Venugopal, K.R., Patnaik, L.M. (eds.) ICIP 2011. CCIS, vol. 157, pp. 40–50. Springer, Heidelberg (2011)

    Chapter  Google Scholar 

  12. Bhavani, S.D., Rani, T.S., Bapi, R.S.: Feature selection using correlation fractal dimension: Issues and applications in binary classification problems. Applied Soft Computing 8, 555–563 (2008)

    Article  Google Scholar 

  13. Garratt, A., Jackson, M., Burden, P., Wallis, J.: A survey of alternative designs for a search engine storage structure. Information and Software Technology 43, 661–677 (2001)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer and Information Sciences, University of Hyderabad, Hyderabad, India

    A. Poorna Chandrasekhar & T. Sobha Rani

Authors
  1. A. Poorna Chandrasekhar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. T. Sobha Rani
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. TASSL, Dept. of Electrical & Computer Engineering, Rutgers the State University of New Jersey, Brett Road, 08854-8058, Piscataway, NJ, USA

    Manish Parashar

  2. Mathematics and Computer Science Division, Argonne National Laboratory, 60439, Argonne, IL, USA

    Dinesh Kaushik

  3. School of Computer Science and Welsh Science Center, Cardiff University, 5 The Parade, CF24 3AA, Cardiff, UK

    Omer F. Rana

  4. Division of Physical Sciences and Engineering, 4700 King Abdullah University of Science and Technology, Room 3221, Al Jazri Building, 23955-6900, Thuwal, Makkah, Saudi Arabia

    Ravi Samtaney

  5. Department of Electrical and Computer Engineering, Stony Brook University, 11794, Stony Brook, New York, USA

    Yuanyuan Yang

  6. Faculty of Engineering and Information Technologies, School of Information Technologies, University of Sydney, 2006, Sydney, NSW, Australia

    Albert Zomaya

Rights and permissions

Reprints and permissions

Copyright information

© 2012 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Chandrasekhar, A.P., Rani, T.S. (2012). Storage and Retrieval of Large Data Sets: Dimensionality Reduction and Nearest Neighbour Search. In: Parashar, M., Kaushik, D., Rana, O.F., Samtaney, R., Yang, Y., Zomaya, A. (eds) Contemporary Computing. IC3 2012. Communications in Computer and Information Science, vol 306. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-32129-0_29

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-32129-0_29

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-32128-3

  • Online ISBN: 978-3-642-32129-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation