Deduplication

Description

The Storage Systems Research Center is a pioneer in the use of deduplication to consolidate storage by eliminating references to duplicate and near-duplicate data. Deduplication reduces the total amount of storage needed to store information by matching data sequences in new data with identical or similar sequences in already-stored data and storing references to the existing data in place of the new data. However, while this approach can reduce the demand for storage, it introduces new problems. The system must be able to efficiently locate duplicates, a non-trivial task when the storage system contains terabytes to petabytes of data. The system must also restore some of the redundancy removed by deduplication; failing to do so can result in large volumes of data being lost if a single key chunk of data fails. Deduplication potentially reduces performance by spreading data across disk, and can result in security issues.

Status

The Deep Store project initially explored the use of deduplication for archival systems, and we are continuing to explore additional issues such as performance, indexing, reliability, and security for deduplicated data.

Publications

2009

• Deepavali Bhagwat, Kave Eshghi, Darrell D. E. Long, Mark Lillibridge, Extreme Binning: Scalable, Parallel Deduplication for Chunk-based File Backup , Proceedings of the 17th IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS 2009), September 2009.
• Keren Jin, Ethan L. Miller, The Effectiveness of Deduplication on Virtual Machine Disk Images, Proceedings of SYSTOR 2009: The Israeli Experimental Systems Conference, May 2009.
• Mark Lillibridge, Kave Eshghi, Deepavali Bhagwat, Vinay Deolalikar, Greg Trezise, Peter Camble, Sparse Indexing: Large Scale, Inline Deduplication Using Sampling and Locality, Proceedings of the 7th USENIX Conference on File and Storage Technologies (FAST '09), February 2009.

2008

• Mark W. Storer, Kevin Greenan, Darrell D. E. Long, Ethan L. Miller, Secure Data Deduplication, Proceedings of the 4th International Workshop on Storage Security and Survivability (StorageSS 2008), held in conjunction with the 15th ACM Conference on Computer and Communications Security (CCS 2008), October 2008.

2007

• Deepavali Bhagwat, Kave Eshghi, Pankaj Mehra, Content-based Document Routing and Index Partitioning for Scalable Similarity-based Searches in a Large Corpus, Proceedings of the 13th ACM SIGKDD international conference on Knowledge Discovery and Data Mining (KDD '07), August 2007, pages 105-112.

2006

• Deepavali Bhagwat, Kristal Pollack, Darrell D. E. Long, Thomas Schwarz, Ethan L. Miller, Jehan-François Pâris, Providing High Reliability in a Minimum Redundancy Archival Storage System, Proceedings of the 14th IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS '06), September 2006, pages 413-421.
• Lawrence You, Efficient Archival Data Storage, Technical Report UCSC-SSRC-06-04, June 2006. Ph.D. thesis.

2005

• Lawrence You, Kristal Pollack, Darrell D. E. Long, Deep Store: An Archival Storage System Architecture, Proceedings of the 21st International Conference on Data Engineering (ICDE '05), April 2005.

2004

• Lawrence You, Christos Karamanolis, Evaluation of efficient archival storage techniques, Proceedings of the 21st IEEE / 12th NASA Goddard Conference on Mass Storage Systems and Technologies, April 2004.