folly::json: allow skipping invalid UTF8
authorMark McDuff <mcduff@fb.com>
Tue, 17 Dec 2013 03:07:42 +0000 (19:07 -0800)
committerJordan DeLong <jdelong@fb.com>
Fri, 20 Dec 2013 21:06:36 +0000 (13:06 -0800)
commita91e75dbf16f3f3e038dbd828efedd9916d59bf9
treed7c6322202385ba2829ece7bb0304be03d38d6d7
parent484392b014c24f572bf8431dbc94466673c05c68
folly::json: allow skipping invalid UTF8

Summary:
folly::json::serialize by default doesn't check for valid UTF8, and as a result can generate invalid JSON.  There is an option to check for valid UTF8, which throws on an error.

This diff introduces a new option, `skip_invalid`, which replaces invalid chars with U+FFFD. http://en.wikipedia.org/wiki/Specials_(Unicode_block) seems to suggest that this is the correct replacement.

@override-unit-failures

Test Plan: g-unittest

Reviewed By: delong.j@fb.com

FB internal diff: D1102923
folly/json.cpp
folly/json.h
folly/test/JsonTest.cpp