zlib compression fails on large IOBufs
authorNicholas Ormrod <njormrod@fb.com>
Wed, 26 Nov 2014 00:36:20 +0000 (16:36 -0800)
committerDave Watson <davejwatson@fb.com>
Thu, 11 Dec 2014 15:58:58 +0000 (07:58 -0800)
commiteb9d7cd7754c3f21ec767e261dff0906c35d0cf6
tree0c476d1b6934f629e33e2076bac0b2cc43c5a5f6
parent607e323f3791f3701e61b47e74280deb9aa26a22
zlib compression fails on large IOBufs

Summary:
If a single IOBuf has size exceeding 2^32, then our zlib
compression algorithm fails. Specifically, zlib z_stream.avail_in is
only 32 bytes (I think it's a long?
http://www.gzip.org/zlib/zlib_faq.html#faq32), and so a too-big IOBuf
will overflow the z_stream and cause data loss.

This diff breaks up large IOBufs into smaller chunks.

Test Plan:
fbconfig -r folly && fbmake runtests

Also compressed biggrep's configerator blob, which is how this bug was
caught. It now works. See the associated task.

Reviewed By: robbert@fb.com

Subscribers: trunkagent, sdwilsh, njormrod, folly-diffs@

FB internal diff: D1702925

Tasks: 5648445

Signature: t1:1702925:1416958232:459d498ff1db13e1a20766855e6f2f97da8cde8c
folly/io/Compression.cpp