Work-in-Progress: A high-bandwidth snappy decompressor in reconfigurable logic

Jian Fang, Jianyu Chen, Zaid Al-Ars, Peter Hofstee, Jan Hidders

Research output: Chapter in Book/Report/Conference proceedingConference paperResearch

5 Citations (Scopus)

Abstract

While in-memory databases have largely removed I/O as a bottleneck for database operations, loading the data from storage into memory remains a significant limiter to end-to end performance. Snappy is a widely used compression algorithm in the Hadoop ecosystem and in database systems and is an option in often-used file formats such as Parquet and ORC. Compression reduces the amount of data that must be transferred from/to the storage saving both storage space and storage bandwidth. While it is easy for a CPU Snappy decompressor to keep up with the bandwidth of a hard disk drive, when moving to NVMe devices attached with high bandwidth connections such as PCIe Gen4 or OpenCAPI, the decompression speed in a CPU is insufficient. We propose an FPGA-based Snappy decompressor that can process multiple tokens in parallel and operates on each FPGA block ram independently. Read commands are recycled until the read data is valid dramatically reducing control complexity. One instance of our decompression engine takes 9% of the LUTs in the XCKU15P FPGA, and achieves up to 3GB/s (5GB/s) decompression rate from the input (output) side, about an order of magnitude faster than a CPU (single thread). Parquet allows for independent decompression of multiple pages and instantiating eight of these units on a XCKU15P FPGA can keep up with the highest performance interface bandwidths.

Original languageEnglish
Title of host publication2018 International Conference on Hardware/Software Codesign and System Synthesis, CODES+ISSS 2018
Place of PublicationTurin, Italy
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages2
ISBN (Electronic)9781538655627
DOIs
Publication statusPublished - 6 Nov 2018
Event2018 ACM/IEEE International Conference on Hardware/Software Codesign and System Synthesis, CODES+ISSS 2018 - Turin, Italy
Duration: 30 Sep 20185 Oct 2018

Conference

Conference2018 ACM/IEEE International Conference on Hardware/Software Codesign and System Synthesis, CODES+ISSS 2018
CountryItaly
CityTurin
Period30/09/185/10/18

Keywords

  • Decompression
  • FPGA
  • Snappy

Fingerprint Dive into the research topics of 'Work-in-Progress: A high-bandwidth snappy decompressor in reconfigurable logic'. Together they form a unique fingerprint.

Cite this