Petascale Data Storage Institute @ UC Santa Cruz

The growth rate of data has been rapidly increasing over the past decade. This growth has pushed data quantities to a point where users and administrators have difficulty finding and managing files. UC Santa Cruz researchers are addressing this problem by integrating search and indexing into storage systems. We have designed scalable indexes that enable a number of powerful search capabilities across billions of files. Users can quickly find files with specific properties and issue range, nearest neighbor, or top-k queries, for example, 'which 10 files consume the most storage?' or 'what are my most recently accessed presentation files?'. In addition, queries can be issued across specific sub-trees, across an multiple nodes, or across an entire storage system's history. Our index utilizes data structures specifically designed for the properties of large-scale storage systems, such as, highly skewed data distributions and hierarchical files, that provide much better performance and scalability than existing approaches.