Skyhook: Towards an Arrow-Native Storage System

Appeared in CCGrid 2022.

Abstract

With the ever-increasing dataset sizes, several file formats such as Parquet, ORC, and Avro have been developed to store data efficiently, save the network, and interconnect bandwidth at the price of additional CPU utilization. However, with the advent of networks supporting 25-100 Gb/s and storage devices delivering 1,000,000 reqs/sec, the CPU has become the bottleneck trying to keep up feeding data in and out of these fast devices. The result is that data access libraries executed on single clients are often CPU-bound and cannot utilize the scale-out benefits of distributed storage systems. One attractive solution to this problem is to offload data-reducing processing and filtering tasks to the storage layer. However, modifying legacy storage systems to support compute offloading is often tedious and requires an extensive understanding of the system internals. Previous approaches re-implemented functionality of data processing frameworks and access libraries for a particular storage system, a duplication of effort that might have to be repeated for different storage systems.

This paper introduces a new design paradigm that allows ex- tending programmable object storage systems to embed existing, widely used data processing frameworks and access libraries into the storage layer with no modifications. In this approach, data processing frameworks and access libraries can evolve independently from storage systems while leveraging distributed storage systems’ scale-out and availability properties. We present Skyhook, an example implementation of our design paradigm using Ceph, Apache Arrow, and Parquet. We provide a brief performance evaluation of Skyhook and discuss key results.

Publication date:
May 2022

Authors:
Jayjeet Chakraborty
Carlos Maltzahn

Projects:

Available media

Full paper text: PDF

Bibtex entry

@inproceedings{Chakraborty-skyhook-22,
  author       = {Jayjeet Chakraborty and Carlos Maltzahn},
  title        = {Skyhook: Towards an Arrow-Native Storage System},
  booktitle    = {CCGrid 2022},
  month        = may,
  year         = {2022},
}