folly.git
5 years agoRemove/make private the default ***Holder constructor to allow compile time detection...
Dan Melnic [Mon, 3 Apr 2017 21:09:47 +0000 (14:09 -0700)]
Remove/make private the default ***Holder constructor to allow compile time detection of ***Holder(mutexPtr_) constructs

Summary: Avoid unintended C++ stuff - this is not the same as the "fleeting rvalue" - this change might break the build but I will fix it

Reviewed By: yfeldblum

Differential Revision: D4814104

fbshipit-source-id: 058a0eac44893c573062fcf5665d4fd022ee64a0

5 years agoatomic_shared_ptr
Dave Watson [Mon, 3 Apr 2017 17:51:19 +0000 (10:51 -0700)]
atomic_shared_ptr

Summary:
A (almost) lock-free atomic_shared_ptr, matching the proposed concurrency TS interface.

See notes at top of file

Reviewed By: davidtgoldblatt

Differential Revision: D4716098

fbshipit-source-id: b9ca2443ba9e227ebb6f40807128073c6e14222a

5 years agoOptimize JSON escaping of ASCII strings v2017.04.03.00
Giuseppe Ottaviano [Sat, 1 Apr 2017 05:17:57 +0000 (22:17 -0700)]
Optimize JSON escaping of ASCII strings

Summary:
`escapeString` is very slow even when there's very little to nothing to escape. This diff adds a fast path to copy sequences of bytes that don't need escaping.
It also optimizes appending escape sequences: `string::push_back` is slow because it has to do a capacity check for every character.

Before:
```
  ============================================================================
  folly/test/JsonOtherTest.cpp                    relative  time/iter  iters/s
  ============================================================================
  jsonSerialize                                              818.55ns    1.22M
  jsonSerializeWithNonAsciiEncoding                            1.35us  738.06K
  jsonSerializeWithUtf8Validation                              1.42us  705.60K
  jsonSerializeAsciiWithUtf8Validation                         3.27us  306.06K
  parseSmallStringWithUtf                                      1.91us  522.38K
  parseNormalString                                            1.51us  660.27K
  parseBigString                                             384.44ns    2.60M
  toJson                                                     480.54ns    2.08M
  ============================================================================
```

After:
```
  ============================================================================
  folly/test/JsonOtherTest.cpp                    relative  time/iter  iters/s
  ============================================================================
  jsonSerialize                                              781.69ns    1.28M
  jsonSerializeWithNonAsciiEncoding                          847.68ns    1.18M
  jsonSerializeWithUtf8Validation                            928.68ns    1.08M
  jsonSerializeAsciiWithUtf8Validation                       199.85ns    5.00M
  parseSmallStringWithUtf                                      1.93us  518.39K
  parseNormalString                                            1.45us  689.11K
  parseBigString                                             378.66ns    2.64M
  toJson                                                     446.38ns    2.24M
  ============================================================================
```

All string escaping benchmarks are slightly faster, and ASCII-only with no escapes is 8x faster.

Reviewed By: luciang, evilmucedin, yfeldblum

Differential Revision: D4793233

fbshipit-source-id: c40d07708bd787799c8c00f9f23a417b862ee9ae

5 years agofix memory leak in case of large number of retries
Mainak Mandal [Sat, 1 Apr 2017 02:01:12 +0000 (19:01 -0700)]
fix memory leak in case of large number of retries

Summary: Infinite retries is something that is often needed for read-modify-write like workflows. The current implementation was creating a nested chain of implicit promises. This manifests as a memory leak after some time. Worse yet, even if it succeeds, it will take a long time to churn thru the chain of promises.

Reviewed By: yfeldblum

Differential Revision: D4770335

fbshipit-source-id: 44b8d6df1084de4514b66919a9838cf2322d6dce

5 years agoavoid UB in StringKeyedCommon for default constructed StringPiece
Wez Furlong [Fri, 31 Mar 2017 06:38:50 +0000 (23:38 -0700)]
avoid UB in StringKeyedCommon for default constructed StringPiece

Summary:
ubsan flags this problem:

```
buck-out/opt-ubsan/gen/folly/__default_headers__#header-mode-symlink-tree-with-header-map,headers/folly/experimental/StringKeyedCommon.h:31:18: runtime error: null pointer passed as argument 2, which is declared to never be null
third-party-buck/gcc-4.9-glibc-2.20-fb/build/glibc/include/string.h:47:28: note: nonnull attribute specified here
    #0 0x5cb88f in std::pair<std::__detail::_Node_iterator<std::pair<folly::Range<char const*> const, facebook::eden::GitIgnore>, false, true>, bool> folly::StringKeyedUnorderedMap<facebook::eden::GitIgnore, folly::Hash, std::equal_to<folly::Range<char const*> >, std::allocator<std::pair<folly::Range<char const*> const, facebook::eden::GitIgnore> > >::emplace<facebook::eden::GitIgnore>(folly::Range<char const*>, facebook::eden::GitIgnore&&) buck-out/opt-ubsan/gen/folly/__default_headers__#header-mode-symlink-tree-with-header-map,headers/folly/experimental/StringKeyedCommon.h:31
    #1 0x5c6652 in facebook::eden::(anonymous namespace)::IgnoreChecker::isIgnored(facebook::eden::detail::RelativePathBase<folly::Range<char const*> >) eden/fs/inodes/Dirstate.cpp:226
    #2 0x5c6037 in facebook::eden::(anonymous namespace)::IgnoreChecker::isIgnored(facebook::eden::detail::RelativePathBase<folly::Range<char const*> >) eden/fs/inodes/Dirstate.cpp:180
    #3 0x5c5173 in facebook::eden::Dirstate::getStatusForExistingDirectory(facebook::eden::detail::RelativePathBase<folly::Range<char const*> >) const eden/fs/inodes/Dirstate.cpp:372
    #4 0x5c4509 in facebook::eden::Dirstate::getStatus() const eden/fs/inodes/Dirstate.cpp:272
    #5 0x50a448 in verifyExpectedDirstate(facebook::eden::Dirstate const*, std::unordered_map<std::basic_fbstring<char, std::char_traits<char>, std::allocator<char>, std::fbstring_core<char> >, facebook::eden::StatusCode, std::hash<std::basic_fbstring<char, std::char_traits<char>, std::allocator<char>, std::fbstring_core<char> > >, std::equal_to<std::basic_fbstring<char, std::char_traits<char>, std::allocator<char>, std::fbstring_core<char> > >, std::allocator<std::pair<std::basic_fbstring<char, std::char_traits<char>, std::allocator<char>, std::fbstring_core<char> > const, facebook::eden::StatusCode> > >&&) eden/fs/inodes/test/DirstateTest.cpp:48
    #6 0x50c70c in Dirstate_addDirectoriesWithMixOfFiles_Test::TestBody() eden/fs/inodes/test/DirstateTest.cpp:220
    #7 0xb45857 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/engshare/third-party2/gtest/1.7.0/src/gtest-1.7.0/./src/gtest.cc:2364
    #8 0xb36784 in testing::Test::Run() /home/engshare/third-party2/gtest/1.7.0/src/gtest-1.7.0/./src/gtest.cc:2437
    #9 0xb36957 in testing::TestInfo::Run() [clone .part.558] /home/engshare/third-party2/gtest/1.7.0/src/gtest-1.7.0/./src/gtest.cc:2612
    #10 0xb36b74 in testing::TestCase::Run() [clone .part.559] /home/engshare/third-party2/gtest/1.7.0/src/gtest-1.7.0/./src/gtest.cc:2587
    #11 0xb3806e in testing::internal::UnitTestImpl::RunAllTests() [clone .part.561] /home/engshare/third-party2/gtest/1.7.0/src/gtest-1.7.0/./src/gtest.cc:4571
    #12 0xb382d9 in testing::UnitTest::Run() /home/engshare/third-party2/gtest/1.7.0/src/gtest-1.7.0/./src/gtest.cc:4519
    #13 0x861ca7 in main third-party-buck/gcc-4.9-glibc-2.20-fb/build/gtest/include/gtest/gtest.h:2326
    #14 0x7f20be0740f5 in __libc_start_main /home/engshare/third-party2/glibc/2.20/src/glibc-2.20/csu/libc-start.c:289
    #15 0x4a552c in _start /home/engshare/third-party2/glibc/2.20/src/glibc-2.20/csu/../sysdeps/x86_64/start.S:122

UndefinedBehaviorSanitizer: undefined-behavior buck-out/opt-ubsan/gen/folly/__default_headers__#header-mode-symlink-tree-with-header-map,headers/folly/experimental/StringKeyedCommon.h:31:18
```

The issue is that `StringPiece` default constructs to `{nullptr, nullptr}` as a valid representation of an empty string.
This is tripping up UBSAN in this case.

The fix is a trivial nullptr check

Reviewed By: igorsugak

Differential Revision: D4791015

fbshipit-source-id: dec7484b29ecb29c17b8dd6a9b0e8093f07d63cb

5 years agoSome constexpr symbols for whether sanitizers are enabled
Yedidya Feldblum [Fri, 31 Mar 2017 04:38:09 +0000 (21:38 -0700)]
Some constexpr symbols for whether sanitizers are enabled

Summary:
[Folly] Some constexpr symbols for whether sanitizers are enabled.

Useful where we need to do or not do some particular thing when sanitizers are enabled or disabled. Such cases do happen. It is better to use a C++ `constexpr` symbol rather than a preprocessor symbol in such cases, where possible.

Reviewed By: meyering

Differential Revision: D4805110

fbshipit-source-id: ca9ce074edef73dce3a04693dda102698c9cd29f

5 years agoFix double-delete in Singleton::make_mock
Yedidya Feldblum [Thu, 30 Mar 2017 22:43:00 +0000 (15:43 -0700)]
Fix double-delete in Singleton::make_mock

Summary:
[Folly] Fix double-`delete` in `Singleton::make_mock`.

In that function, we unconditionally destroy any existing singleton instance of the same singleton. Actually, we must conditionally destroy it - conditionally on one having been created and not yet destroyed.

This problem only rarely appears because it is masked by `delete static_cast<T*>(nullptr)` being a no-op. For example, if we call `make_mock` before actually instantiating the singleton, we call that no-op. The way to make it appear is to instantiate the singleton, and then to call `make_mock` twice in a row. The first call to `make_mock` unconditionally destroys the existing instance (it still should have checked) and the second call does it again, but because the existing instance is not `nullptr`, the second call is a double-delete of a non-`nullptr` instance and crashes. In the simple case, as reproduced in an attached test, the failure is observable with ASAN. In other cases, the double-`free` is tolerated, but the failure may only be observable depending on the singleton object's state - if running the dtor twice fails.

Reviewed By: ivmaykov

Differential Revision: D4798163

fbshipit-source-id: e7b65d030d89225dfdc2fad4c778d3653460806e

5 years agoRemove unneccessary test for &that==this in folly::Function's move assignment operator.
Eric Niebler [Thu, 30 Mar 2017 21:09:34 +0000 (14:09 -0700)]
Remove unneccessary test for &that==this in folly::Function's move assignment operator.

Summary: Self-moves are exceedingly rare and need not preserve the state of the object. They must only leave the object in a valid but unspecified state. By removing the branch, we make the common case (non-self move) faster.

Reviewed By: spacedentist, ot

Differential Revision: D4803486

fbshipit-source-id: 3ef2e1e13cd08d9221ecb154bfb3338b16487717

5 years agoProperly constrain folly::Function's generic conversion constructor and fix its noexc...
Eric Niebler [Thu, 30 Mar 2017 21:09:32 +0000 (14:09 -0700)]
Properly constrain folly::Function's generic conversion constructor and fix its noexcept specification

Summary:
The generic conversion constructor for `folly::Function` was not checking that the source object could successfully be copy/move constructed, leading to some `is_constructible` false positives.

Also, the `noexcept` specification on the `Function` constructor wasn't taking into account that the source object might be copied into the Function, instead of moved. The copy could throw.

Reviewed By: yfeldblum

Differential Revision: D4775037

fbshipit-source-id: f337b41bf9ac431baa9457a501e63c18ca099e57

5 years agoPath variants for get_(ref_)default()
Nick Terrell [Thu, 30 Mar 2017 19:48:11 +0000 (12:48 -0700)]
Path variants for get_(ref_)default()

Summary: Useful for the same reason as the `get_ptr()` path function, but when you want to use a default value.

Reviewed By: luciang, yfeldblum

Differential Revision: D4785728

fbshipit-source-id: 70fd56f9ffa7a9edd6740f6dd712d3a251bf9fb0

5 years agoBacked out changeset d4e3fdfd5775
Igor Sugak [Thu, 30 Mar 2017 01:14:16 +0000 (18:14 -0700)]
Backed out changeset d4e3fdfd5775

Summary:
https://our.intern.facebook.com/intern/sevmanager/view/s/145809/?hc_location=ufi

```
hg backout d4e3fdfd57750c8dcabdb4c01ab3b528c4818624
```

Reviewed By: lbrandy

Differential Revision: D4799568

fbshipit-source-id: 76366040702e68becc7e245d7ce15832f933e10e

5 years agouncurry
Tom Jackson [Wed, 29 Mar 2017 22:17:25 +0000 (15:17 -0700)]
uncurry

Summary: Extending ApplyTuple to support this common functional construct, with good forwarding semantics.

Reviewed By: yfeldblum

Differential Revision: D4787560

fbshipit-source-id: 2c740e448e0cb916abe948b79709d5ecd8ba54bb

5 years agoCut extraneous typename in StringKeyedCommon.h
Yedidya Feldblum [Wed, 29 Mar 2017 17:17:46 +0000 (10:17 -0700)]
Cut extraneous typename in StringKeyedCommon.h

Summary:
[Folly] Cut extraneous typename in `StringKeyedCommon.h`.

In this case, `typename` is optional, and, to avoid confusion, should be removed.

"In this case" means in a template definition, with a qualified non-dependent name.

Reviewed By: ericniebler

Differential Revision: D4792087

fbshipit-source-id: dbb9d2c747432a1f9bc8823e72a7668e78a2fbaa

5 years agofix dead-lock in Future when executor discards function
Sven Over [Wed, 29 Mar 2017 10:48:52 +0000 (03:48 -0700)]
fix dead-lock in Future when executor discards function

Summary:
This diff adds two tests to futures/test/ViaTest.cpp:
viaDummyExecutorFutureSetValueFirst and
viaDummyExecutorFutureSetCallbackFirst. The latter resulted in a
dead-lock before the fix contained in this diff.

It is important that the callback function is destroyed after
it is executed, since it may capture objects (like a Promise)
that should be destroyed (so that e.g. a corresponding Future
throws BrokenPromise). When the callback is executed via an
executor, it is possible that the executor doesn't get around to
executing the task. We shouldn't rely on the task being executed
to do necessary clean-up. That clean-up should (also) happen
when the task (with its captured data) is destroyed (in the
spirit of RIAA).

Reviewed By: djwatson

Differential Revision: D4779215

fbshipit-source-id: d029cf8b8f7b55e1b03357749c5fb62d95986ca7

5 years agoAutomaticCodec
Nick Terrell [Wed, 29 Mar 2017 04:21:02 +0000 (21:21 -0700)]
AutomaticCodec

Summary:
Add codec that does automatic format detection for decompression.
The codec returned by `getCodec(CodecType::AUTOMATIC)` can decompress codecs `LZ4_FRAME`, `ZSTD`, `ZLIB`, `GZIP`, and `LZMA2`.
Additionally, the function `getAutomaticCodec()` allows users to pass their own custom codecs, but the 5 default codecs must also be supported.
We force the default codecs on users because having a common subset of formats is in general useful, and because we want to be able to add codecs to this set.

When compiled in debug mode, on construction it checks that the codecs are compatible.
This helps make sure that users don't accidentally add ambiguous codecs, and if we add a new default codec in the future, tests will fail on ambiguity.

Reviewed By: yfeldblum

Differential Revision: D4760065

fbshipit-source-id: 9e65844aba9f1ae3d5c7b86643931782c06ab3eb

5 years agoCursor::read<T>() initializes return value
Ondrej Lehecka [Wed, 29 Mar 2017 01:58:12 +0000 (18:58 -0700)]
Cursor::read<T>() initializes return value

Summary:
If the compiler doesn't see thru the call to tryRead which always initializes the val variable, it will emit the following error.
I was getting error when compiling my android project:
error: 'val' may be used uninitialized in this function [-Werror=maybe-uninitialized]
This should not have a negative perf impact as smart compilers which inline call to tryRead can elliminate the initialization.

Reviewed By: djwatson

Differential Revision: D4790864

fbshipit-source-id: f353cfe54ca4d056b6ddfc075d00580c9f2d391e

5 years agoOptimize ZlibCodec::doUncompress()
Nick Terrell [Tue, 28 Mar 2017 23:30:48 +0000 (16:30 -0700)]
Optimize ZlibCodec::doUncompress()

Summary:
`ZlibCodec::doUncompress()` always allocates at least 4 MiB when the uncompressed size is unkown.
We can get a 10-20% speed boost for small inputs (as well as significant memory savings) if we dynamically choose the buffer size.

Benchmarks before the change:
  ============================================================================
  experimental/terrelln/CompressionBenchmark.cpp  relative  time/iter  iters/s
  ============================================================================
  uncompress(ZLIB_100)                                         2.96us  338.19K
  uncompress(ZLIB_1000)                                        7.22us  138.44K
  uncompress(ZLIB_10000)                                      36.86us   27.13K
  uncompress(ZLIB_100000)                                    299.70us    3.34K
  uncompress(ZLIB_1000000)                                     2.74ms   365.32
  uncompress(ZLIB_10000000)                                   26.91ms    37.16
  uncompress(ZLIB_100000000)                                 273.92ms     3.65
  ============================================================================
After:
  ============================================================================
  experimental/terrelln/CompressionBenchmark.cpp  relative  time/iter  iters/s
  ============================================================================
  uncompress(ZLIB_100)                                         2.34us  426.68K
  uncompress(ZLIB_1000)                                        6.24us  160.38K
  uncompress(ZLIB_10000)                                      35.52us   28.16K
  uncompress(ZLIB_100000)                                    296.30us    3.37K
  uncompress(ZLIB_1000000)                                     2.74ms   365.25
  uncompress(ZLIB_10000000)                                   27.16ms    36.82
  uncompress(ZLIB_100000000)                                 273.72ms     3.65
  ============================================================================

Reviewed By: yfeldblum

Differential Revision: D4783871

fbshipit-source-id: 0fcce191ea520fd2117551db679c86f5ff734893

5 years agoImplement S_ISDIR()
Christopher Dykes [Tue, 28 Mar 2017 18:21:24 +0000 (11:21 -0700)]
Implement S_ISDIR()

Summary: It's needed downstream in HHVM.

Reviewed By: yfeldblum

Differential Revision: D4785204

fbshipit-source-id: fe81ed8e23497fffeb44b2a366c457b788bcaada

5 years agoChange trylock() to try_lock() in folly::SpinLock to conform to standard Lockable.
Maged Michael [Tue, 28 Mar 2017 15:36:31 +0000 (08:36 -0700)]
Change trylock() to try_lock() in folly::SpinLock to conform to standard Lockable.

Reviewed By: yfeldblum

Differential Revision: D4782707

fbshipit-source-id: 535b42b4f2558cadc78037268d6de81a8bb49840

5 years agoAllow custom sorting function for JSON serializer
Dylan Yudaken [Tue, 28 Mar 2017 15:02:42 +0000 (08:02 -0700)]
Allow custom sorting function for JSON serializer

Summary: In some situations it is useful to be able to sort the JSON keys according to some different scheme for readability, so allow the sorting function to be changed by the serializer_opts

Reviewed By: yfeldblum

Differential Revision: D4782077

fbshipit-source-id: 032fa60a38804452bd1c22c67ba897521cb2cd1d

5 years agoGate LZ4Frame behind version check
Nick Terrell [Mon, 27 Mar 2017 23:14:38 +0000 (16:14 -0700)]
Gate LZ4Frame behind version check

Summary:
D4715918 broke open source builds on Ubuntu 14.04, since it has lz4-r114, but the lz4 frame API was introduced in [r123](https://github.com/lz4/lz4/blob/r123/lz4frame.h).
Put the `LZ4FrameCodec` behind a lz4 version check.
Fixes https://github.com/facebook/fbthrift/issues/209.

Reviewed By: yfeldblum

Differential Revision: D4780830

fbshipit-source-id: 19492a7e6bdd128e610c36b5778274e19eff9548

5 years agoDisallow temporary default values in get_ref_default()
Nick Terrell [Mon, 27 Mar 2017 20:03:02 +0000 (13:03 -0700)]
Disallow temporary default values in get_ref_default()

Summary: Passing a temporary default value to `get_ref_default()` returns a dangling reference when it is used.

Reviewed By: lbrandy, yfeldblum

Differential Revision: D4768769

fbshipit-source-id: 4c58a17dc7662ad553cf88a8544dae20016d2f6f

5 years agoSeveral fixes to the "SWMRList" example in experimental/hazptr. v2017.03.27.00
Arthur O'Dwyer [Sat, 25 Mar 2017 20:43:31 +0000 (13:43 -0700)]
Several fixes to the "SWMRList" example in experimental/hazptr.

Summary:
Fix a correctness bug in "SWMRList.h".

Thanks to Maged for the tip! The old code omitted setting a removed node's
"next" pointer to `nullptr`, which meant that if the writer removed node
A and then node B = A->next while the reader was looking at B, then the reader
might happily keep chasing pointers B->next, B->next->next,... without ever
noticing that it was now on a "dead branch" of the linked list.

(And then there's a bit of a trick: you really do have to remove the node
first and *then* set its "next" pointer to null, because if you do the null
assignment first, you'll truncate the list, which means that some readers
will see a truncated list and return the wrong answer. I've added comments
to try to explain this to future-me.)

Style nit: Avoid doing multiple atomic operations on the same line of code.

----

Modernize the parameter-passing conventions in SWMRList.h.

Pass `T`s by const reference, except in `add` where we're likely to want to
make a copy of the `T` anyway. In that case it would be more "STL-correct"
to supply two different overloads `add(T&&)` and `add(const T&)`, but this
is just an example so it makes sense to keep things simple.

----

Fix an undefined behavior in SWMRList example.

Searching an empty SWMRList<int> always invokes undefined behavior by
comparing an uninitialized `T elem` against `val` on the final line of
the `contains` function. Besides, this patch allows SWMRList to work
with `T`s that aren't default-constructible or even copy-constructible.

----

Closes https://github.com/facebook/folly/pull/566

Reviewed By: djwatson

Differential Revision: D4772359

Pulled By: yfeldblum

fbshipit-source-id: 8f96573530800675cb56006aa91e7a5c5c1fb85d

5 years ago(folly) Add lock holder interface to flat combining. Change an assert to a static_assert.
Maged Michael [Sat, 25 Mar 2017 10:28:15 +0000 (03:28 -0700)]
(folly) Add lock holder interface to flat combining. Change an assert to a static_assert.

Summary:
Two changes:
- Add a lock holder interface to enable users to move exclusive access and implicitly release the held lock.
- Change an assert to a static_assert to check at compile time if a provided function fits in the Function structure without dynamic allocation.

Reviewed By: djwatson

Differential Revision: D4770229

fbshipit-source-id: 89408164c08d7660231a6ca4e37637dd688356cd

5 years agoAdd tryRead() and endian variants
Nick Terrell [Sat, 25 Mar 2017 01:12:06 +0000 (18:12 -0700)]
Add tryRead() and endian variants

Summary:
Add a `tryRead()`, and endian variants, which try to read into an arithmetic type, and if there isn't enough data they return false.
One use case is to quickly check if an IOBuf starts with a certain prefix, benchmarks show that using `tryReadLE()` is 6x faster than using `pullAtMost()` and `memcmp()`.

Reviewed By: yfeldblum

Differential Revision: D4767855

fbshipit-source-id: feb8c61092772933d4b8496b27d464559ff8b827

5 years agoAvoid passing temporary to get_ref_default()
Nick Terrell [Fri, 24 Mar 2017 22:00:07 +0000 (15:00 -0700)]
Avoid passing temporary to get_ref_default()

Summary: `get_ref_default()` won't accept temporary default values after D4768769.

Reviewed By: yfeldblum

Differential Revision: D4769043

fbshipit-source-id: 46b039e13b80721922f69036c6cc02f5642a26f9

5 years agoAllow decompresion without uncompressedSize
Nick Terrell [Fri, 24 Mar 2017 21:30:45 +0000 (14:30 -0700)]
Allow decompresion without uncompressedSize

Summary:
The `LZMA2` codec doesn't allow decompression without passing the uncompressed size.
Every other codec that supports streaming decompression without the uncompressed size by streaming into an IOBuf chain.
Benchmarks show that reducing the `defaultBufferLength` to 256 KiB doesn't slow down decompression of large files, but it speeds up decompression of small files (< 1000 bytes).

Reviewed By: yfeldblum

Differential Revision: D4751571

fbshipit-source-id: 39dbe6754a1ecdc2b7ba3107e9face926d4c98ca

5 years agoOptimize ZSTDCodec::doUncompress()
Nick Terrell [Fri, 24 Mar 2017 19:18:57 +0000 (12:18 -0700)]
Optimize ZSTDCodec::doUncompress()

Summary:
It is ~10% faster to call `ZSTD_decompress()` than use the
streaming API. The streaming API does some extra `memcpy`s that we
can avoid. We are working on improving the speed of the streaming
API in the case where all the data can be processed in one shot,
but that won't be available in the stable ZSTD api for a few versions.

Reviewed By: yfeldblum

Differential Revision: D4731058

fbshipit-source-id: 39026c499c0f5002466097b5afe7e30f850e0ae8

5 years agoAdd LZ4_FRAME codec
Nick Terrell [Fri, 24 Mar 2017 18:08:17 +0000 (11:08 -0700)]
Add LZ4_FRAME codec

Summary:
The LZ4 Frame codec encodes data using the LZ4 frame format.
One advantage of the LZ4 frame format is that it has 4 magic bytes in the header, so users can transparently determine compression type.
It also allows the user to interop with the lz4 command line tool.

Reviewed By: yfeldblum

Differential Revision: D4715918

fbshipit-source-id: 689833fef526b1cfe98685179e7b494380d49cba

5 years agoExpand environment-handling in folly
Phil Willoughby [Fri, 24 Mar 2017 09:05:42 +0000 (02:05 -0700)]
Expand environment-handling in folly

Summary:
Extract the environment-as-STL-map logic out of `test::EnvVarSaver` and into a
separate class `experimental::EnvironmentState` so that other code which needs
to manipulate the environment can do so more easily.

Add routines to set the process environment from the state of an
`EnvironmentState`'s map, and to extract the environment in the forms required
by `Subprocess` and the UNIX routines `execve` etc.

Reviewed By: yfeldblum

Differential Revision: D4713307

fbshipit-source-id: 6b1380dd29b9ba41c97b886814dd3eee91fc1c0f

5 years agoRe-enable io tests
Ted Percival [Thu, 23 Mar 2017 18:30:38 +0000 (11:30 -0700)]
Re-enable io tests

Summary:
Looks like some tests for the `io` directory were left behind when they were moved out of experimental. Re-enabled them in the new location, except for the compression test because it takes a lot longer. It's still built and can be executed manually.

Also added some preprocessor guards around compression codecs that might not be compiled in (and therefore fail during test SetUp).

Depends on #547 because the path to gtest appears in the new Makefile.am's `CPPFLAGS`.
Closes https://github.com/facebook/folly/pull/550

Reviewed By: yfeldblum

Differential Revision: D4631244

Pulled By: Orvid

fbshipit-source-id: 496b2c272e4f7293822f159776a02f7dd0c9e04d

5 years agoMake sure singleton destructor is always called from main thread.
Andrii Grynenko [Thu, 23 Mar 2017 18:17:04 +0000 (11:17 -0700)]
Make sure singleton destructor is always called from main thread.

Reviewed By: yfeldblum, ot

Differential Revision: D4742262

fbshipit-source-id: 7137bb8e1cb39c74e8ba726e76e9a8a6f3e1984c

5 years agoRandomize the Unix socket name in AsyncSocketTest.SendMessageAncillaryData test to...
Maxim Georgiev [Thu, 23 Mar 2017 17:31:34 +0000 (10:31 -0700)]
Randomize the Unix socket name in AsyncSocketTest.SendMessageAncillaryData test to avoid collisions.

Summary:
Our test framework reports frequent failures of AsyncSocketTest.SendMessageAncillaryData. According to the logs the socket fails to bind:

    folly/io/async/test/AsyncSocketTest2.cpp:3098: Failure
    Expected: (bind(lfd, (struct sockaddr*)&addr, sizeof(addr))) != (-1), actual: -1 vs -1
    Bind failed: 98

This diff adds the socket name randomization to avoid name collisions between tests running concurrently on the same test box.

Reviewed By: yfeldblum

Differential Revision: D4758942

fbshipit-source-id: 6066dbc18222a4521c40b2ff218cb7dab8bd789d

5 years agochange EXPECT_EQ(false, ... to EXPECT_FALSE(...; ditto for ASSERT_EQ
Jim Meyering [Thu, 23 Mar 2017 17:18:49 +0000 (10:18 -0700)]
change EXPECT_EQ(false, ... to EXPECT_FALSE(...; ditto for ASSERT_EQ

Summary:
Change every instance of EXPECT_EQ(false, ... to the simpler/shorter EXPECT_FALSE(...
Do the same for ASSERT_EQ, for consistency, and for true/TRUE.

Reviewed By: Orvid

Differential Revision: D4751397

fbshipit-source-id: 045293dbf8be1167f0cb1b85a2f9cd59a6a30f73

5 years agoFlat Combining
Maged Michael [Thu, 23 Mar 2017 14:13:23 +0000 (07:13 -0700)]
Flat Combining

Summary:
Flat combining template that takes the following template parameters:
 T         Concurrent data structure using FC interface
 Mutex Mutex type (default std::mutex)
 Atom  Atomic template (default std::atomic)
 Req    Optional request structure to hold custom info (default dummy type bool)

Flat combining (FC) was introduced in the SPAA 2010 paper Flat Combining and the Synchronization-Parallelism Tradeoff, by Danny Hendler, Itai Incze, Nir Shavit, and Moran Tzafrir.
http://mcg.cs.tau.ac.il/projects/projects/flat-combining

Reviewed By: djwatson

Differential Revision: D4602402

fbshipit-source-id: 38327f752a3e92bb01e5496c321d8c87c818087a

5 years agoFix typo in Singleton
Giuseppe Ottaviano [Wed, 22 Mar 2017 23:02:41 +0000 (16:02 -0700)]
Fix typo in Singleton

Summary: The "l" in "Haskell" is not a singleton. Also, friends don't let friends rely on operator precedence.

Reviewed By: yfeldblum

Differential Revision: D4756653

fbshipit-source-id: 7b5360bfc1b26fac06062e4adbfae3a1e477c2dc

5 years agoFix the nested command line app test helper
Christopher Dykes [Wed, 22 Mar 2017 18:04:04 +0000 (11:04 -0700)]
Fix the nested command line app test helper

Summary: D4719112 changed the signature of the constructor but didn't update this callsite...

Reviewed By: djwatson

Differential Revision: D4754745

fbshipit-source-id: ae538a3ffb64cdc211f23220ea5e0d7bb7eff9b3

5 years agoAndroid should also use a relaxed Singleton
Michael Lee [Wed, 22 Mar 2017 17:41:23 +0000 (10:41 -0700)]
Android should also use a relaxed Singleton

Summary: On Android, we should also use a relaxed singleton

Reviewed By: andriigrynenko

Differential Revision: D4754227

fbshipit-source-id: 157844e618d216b768bbb07f36c380ba2f4c9ab8

5 years agoAdd additional points to configure help output
Brian Smith [Wed, 22 Mar 2017 03:26:02 +0000 (20:26 -0700)]
Add additional points to configure help output

Summary: Sometimes tools need to output a little more help context before or after help.

Reviewed By: yfeldblum

Differential Revision: D4719112

fbshipit-source-id: e6f59312d722e0669053644d862c81758855b538

5 years agoUse the gflags portability header
Christopher Dykes [Wed, 22 Mar 2017 01:42:28 +0000 (18:42 -0700)]
Use the gflags portability header

Summary: The gflags portability header should always be used in Folly. There were a couple of places gflags was being used directly. This fixes that.

Reviewed By: yfeldblum

Differential Revision: D4748674

fbshipit-source-id: ba3c0564a39f870f1f5f9371e32d06eb84591ba7

5 years agofix sorted_vector_{set,map} insert with bad hint
Nathan Bronson [Tue, 21 Mar 2017 21:28:38 +0000 (14:28 -0700)]
fix sorted_vector_{set,map} insert with bad hint

Summary:
sorted_vector_{set,map} can silently drop insert if the hinted
insertion position is too small.  This diff fixes it.

Reviewed By: yfeldblum

Differential Revision: D4747319

fbshipit-source-id: 31e399d07a0b77b700edf034dc723cb997dc8e16

5 years agoAdd IOBuf::cloneCoalesced()
Nick Terrell [Tue, 21 Mar 2017 18:13:36 +0000 (11:13 -0700)]
Add IOBuf::cloneCoalesced()

Summary:
`LZ4Codec::doCompress()` and `doUncompress()` need to coalesce a `const IOBuf*`, so they `clone()` and then `coalesce()`.
It is more efficient to do this in one step, as shown by the benchmarks.

Reviewed By: yfeldblum, simpkins

Differential Revision: D4727802

fbshipit-source-id: f8d5f6187f2ee804301232b3c75b848330169fc5

5 years agoDon't allow singleton use w/o registrationComplete()
Andrii Grynenko [Tue, 21 Mar 2017 16:45:01 +0000 (09:45 -0700)]
Don't allow singleton use w/o registrationComplete()

Summary: Submitting mostly to see what breaks.

Reviewed By: yfeldblum

Differential Revision: D4249032

fbshipit-source-id: d4e3fdfd57750c8dcabdb4c01ab3b528c4818624

5 years agoRe-enable a couple of tests under MSVC
Christopher Dykes [Tue, 21 Mar 2017 13:11:14 +0000 (06:11 -0700)]
Re-enable a couple of tests under MSVC

Summary: VS 2017 is here and the bugs these were hitting have been fixed, so re-enable them.

Reviewed By: yfeldblum

Differential Revision: D4742829

fbshipit-source-id: 898e7cfea9de9ffa813c562c12e0e18e0577c8c3

5 years agoIn AsyncSocketTest.SendMessageFlags test use folly::test::TemporaryFile object instea...
Maxim Georgiev [Tue, 21 Mar 2017 02:31:48 +0000 (19:31 -0700)]
In AsyncSocketTest.SendMessageFlags test use folly::test::TemporaryFile object instead of directly creating a file with O_TMPFILE flag

Summary: O_TMPFILE flag is not available on older Linux systems (supported starting Linux 3.11). Got complains about it on GitHub (https://fburl.com/9d848m7i) Will use folly::test::TemporaryFile instead.

Reviewed By: yfeldblum

Differential Revision: D4729733

fbshipit-source-id: 903563825c5b939e35c61725da559e33d21212c4

5 years agocall static function directly
Louis Brandy [Mon, 20 Mar 2017 21:55:36 +0000 (14:55 -0700)]
call static function directly

Summary: This code looks really weird and is otherwise confusing.

Reviewed By: andriigrynenko

Differential Revision: D4730812

fbshipit-source-id: d9f4e73145e81f085ce959a76cdb568d3ae4eb52

5 years agoGive observer manager threads a name
Christopher Dykes [Mon, 20 Mar 2017 21:51:15 +0000 (14:51 -0700)]
Give observer manager threads a name

Summary: Re-landing D4618623 after it had to be reverted due to it's use of `FixedString`, which had broken GCC 4.8. `FixedString` should now be compiling correctly on GCC 4.8 so this should be fine now.

Reviewed By: yfeldblum

Differential Revision: D4727022

fbshipit-source-id: f85c6952fd82e6cd6042f8055ef54d9422f8b0f9

5 years agoFolly Futures to Python Asyncio Futures Bridge
Jason Fried [Mon, 20 Mar 2017 18:48:49 +0000 (11:48 -0700)]
Folly Futures to Python Asyncio Futures Bridge

Summary:
folly/python/futures.h provides some helper functions for bridging folly::future to asyncio.Future.

folly/python/NotificationQueueExecutor.h is a Driveable executor that has a fileno() method that can be monitored using (select, epoll) to determine if the drive method should be called.

folly/python/executor.pyx is an implementation of a "driver" for the NotificationQueueExecutor from the python asyncio side. It tracks also keeps track of asyncio eventloops to Executor mappings.

the getExecutor() from folly/python/futures.h uses that mapping to return the correct executor for this python thread.

Reviewed By: andriigrynenko, yfeldblum

Differential Revision: D4687029

fbshipit-source-id: e79314606ffa18cb6933fe6b749991bfea646cde

5 years agofolly/test/TimeseriesTest.cpp: avoid shadowing warnings v2017.03.20.00
Jim Meyering [Sun, 19 Mar 2017 02:53:26 +0000 (19:53 -0700)]
folly/test/TimeseriesTest.cpp: avoid shadowing warnings

Summary:
Fix warnings exposed by gcc's -Wshadow-compatible-local.
Rename lambda parameter "ts" to "timeSeries".

This avoids the following errors:

  folly/test/TimeseriesTest.cpp:802:27: error: declaration of 'ts' shadows a previous local [-Werror=shadow-compatible-local]
  folly/test/TimeseriesTest.cpp:809:31: error: shadowed declaration is here [-Werror=shadow-compatible-local]

Reviewed By: yfeldblum, liuxuli

Differential Revision: D4735180

fbshipit-source-id: 59b3fecf4031b3c6615e5b764fbc4ea04754eb50

5 years agoMake EventBaseLocal::getOrCreate work with noncopyable args
Victor Zverovich [Sat, 18 Mar 2017 13:29:19 +0000 (06:29 -0700)]
Make EventBaseLocal::getOrCreate work with noncopyable args

Summary: Use perfect forwarding in `EventBaseLocal::getOrCreate` to make it work with noncopyable but moveable arguments.

Reviewed By: yfeldblum

Differential Revision: D4730566

fbshipit-source-id: fa02348b7a9217fef980ec5e743b5990b9d19e9a

5 years agoPut the full help before the options
Brian Smith [Fri, 17 Mar 2017 05:19:53 +0000 (22:19 -0700)]
Put the full help before the options

Summary: This is more in line with "traditional" output from cli tools on *nix.

Reviewed By: yfeldblum

Differential Revision: D4719055

fbshipit-source-id: 2389a4a2a49f18d4b3d0257f68036f73f8fbc49e

5 years agoAdd hash and equal_to implementations to folly::Uri
Andreas C. Osowski [Fri, 17 Mar 2017 00:35:15 +0000 (17:35 -0700)]
Add hash and equal_to implementations to folly::Uri

Summary:
Adds a default inline implementation for `std::hash` and `std::equal_to` to `folly::Uri`.
Both do the comparison / hash based upon the return value of `folly::Uri::toString`.

Reviewed By: yfeldblum

Differential Revision: D4711506

fbshipit-source-id: f4c2a9de8d4302fd315a9f31329cc8ba9f5e0409

5 years agoAdd support for OpenSSL 1.0
Michael Steinert [Thu, 16 Mar 2017 21:12:55 +0000 (14:12 -0700)]
Add support for OpenSSL 1.0

Summary:
I'm working on a project where the target platform is tiny a bit out of
date:

    $ openssl version
    OpenSSL 1.0.0-fips 29 Mar 2010

For various "reasons" I'm not able to update it at this time, however I
would like to use Proxygen (and hence Folly). This patch allows Folly to
compile with the above version of OpenSSL.
Closes https://github.com/facebook/folly/pull/562

Reviewed By: Orvid

Differential Revision: D4715116

Pulled By: yfeldblum

fbshipit-source-id: be38ffb78f1e5cee971ce8cb81936b7f16efe050

5 years agoAdd getPeerAddress/getLocalAddress folly::SocketAddress() function
JiaJie Ye [Thu, 16 Mar 2017 06:03:20 +0000 (23:03 -0700)]
Add getPeerAddress/getLocalAddress folly::SocketAddress() function

Summary: Add getPeerAddress/getLocalAddress folly::SocketAddress() function

Reviewed By: yfeldblum

Differential Revision: D4706693

fbshipit-source-id: 534e862f4168bc9e7bb02e41eac8e2ced6a02c59

5 years agoBackport std::index_sequence and friends
Christopher Dykes [Wed, 15 Mar 2017 18:37:07 +0000 (11:37 -0700)]
Backport std::index_sequence and friends

Summary:
To GCC 4.8, because it's not *quite* dead yet.

This is needed to get FixedString working properly on GCC 4.8

Reviewed By: yfeldblum

Differential Revision: D4646026

fbshipit-source-id: 075df04e479bd2b11f6538da2ed3e78b59956621

5 years agoFix the pkg-config filter
Michael Steinert [Wed, 15 Mar 2017 16:23:02 +0000 (09:23 -0700)]
Fix the pkg-config filter

Summary:
I encountered a situation where "pkg-config --libs openssl" included
some linker flags, i.e.

    $ pkg-config --libs openssl
    -Wl,-z,relro -lssl -lcrypto -ldl -lz

This resulted in these linker flags being passed to sed. This change
filters out the linker flags from the pkg-config output.
Closes https://github.com/facebook/folly/pull/559

Reviewed By: pixelb

Differential Revision: D4700120

Pulled By: yfeldblum

fbshipit-source-id: 530d524caf965919418d19a1df53d2b418f67717

5 years agoAdd String Support to Compression Codec
Felix Handte [Wed, 15 Mar 2017 15:48:25 +0000 (08:48 -0700)]
Add String Support to Compression Codec

Summary:
This diff adds string-based compress and uncompress support to
folly::io::Codec. By default, the string API is implemented by translating to
and from IOBufs. Room is left for individual implementations to override that
behavior and go straight to the underlying library.

Reviewed By: yfeldblum

Differential Revision: D4690770

fbshipit-source-id: 7e2615da9add8328654505cdda6a2c79583b4558

5 years agoIgnore the generated file folly/libfolly.pc
Michael Steinert [Tue, 14 Mar 2017 17:54:57 +0000 (10:54 -0700)]
Ignore the generated file folly/libfolly.pc

Summary: Closes https://github.com/facebook/folly/pull/560

Reviewed By: pixelb

Differential Revision: D4700198

Pulled By: yfeldblum

fbshipit-source-id: 5e251e5719c6783c9f8a4fd9bab357ed362340c0

5 years agoFix the --disable-follytestmain option
Michael Steinert [Tue, 14 Mar 2017 14:35:08 +0000 (07:35 -0700)]
Fix the --disable-follytestmain option

Summary:
This option is useful when compiling on systems that do not want to
include libunwind.
Closes https://github.com/facebook/folly/pull/558

Reviewed By: andrewjcg

Differential Revision: D4701878

Pulled By: yfeldblum

fbshipit-source-id: 61765e2e04ea1da510f06d7805c2c54381aa771e

5 years agosmall_vector should not require a copy-ctr from its element type unless it needs it.
Dmitry Pleshkov [Tue, 14 Mar 2017 06:33:57 +0000 (23:33 -0700)]
small_vector should not require a copy-ctr from its element type unless it needs it.

Summary:
Use default constructor directly, instead of copying the element introduced by default parameter value.
This behavior is compatible with `std::vector`.

Reviewed By: ot, bixue2010, luciang

Differential Revision: D4700611

fbshipit-source-id: d3bd82c46c2857f5abcbec5dd2a64aaa85eaefe4

5 years agoAdd a way to determine if a compression codec is supported at runtime
Christopher Dykes [Mon, 13 Mar 2017 21:47:40 +0000 (14:47 -0700)]
Add a way to determine if a compression codec is supported at runtime

Summary: Folly supports being compiled without all of the compression libraries, so we need a way to determine at runtime what libraries Folly has been compiled to support.

Reviewed By: yfeldblum

Differential Revision: D4692556

fbshipit-source-id: 0ec459db70a4b1d64f1e07c87a1f51ae584bccd0

5 years agoFill buffer before allocating more
Nick Terrell [Mon, 13 Mar 2017 19:50:55 +0000 (12:50 -0700)]
Fill buffer before allocating more

Summary:
Modify `QueueAppender::pushAtMost(const uint8_t*, size_t)` to fill the end of
the current buffer before it allocates another.

Reviewed By: yfeldblum

Differential Revision: D4687832

fbshipit-source-id: 7c21f4359da9f0cf26d9577abddfd640b0c877e9

5 years agoget_ptr(mapOfMaps, key, key...)
Tom Jackson [Mon, 13 Mar 2017 18:09:00 +0000 (11:09 -0700)]
get_ptr(mapOfMaps, key, key...)

Summary:
Since this is a not-uncommon pattern now:

```lang=cpp
if (auto found1 = get_ptr(m, k1)) {
  if (auto found2 = get_ptr(*found, k2)) {
    return use(*found2);
  }
}
```

This diff enables:

```lang=cpp
if (auto found = get_ptr(m, k1, k2)) {
  return use(*found);
}

```

Note that it works for const and non-const maps, returning a correspondingly mutable pointer.

Reviewed By: luciang, yfeldblum

Differential Revision: D3812701

fbshipit-source-id: 77ace9f5dca6cdc4cae7e6dfb9e5fc1075b5b571

5 years agoUpdate folly::Singleton documentation block
Phil Willoughby [Mon, 13 Mar 2017 10:03:16 +0000 (03:03 -0700)]
Update folly::Singleton documentation block

Summary:
With more experience of using folly::Singleton in our code we have found new
recommendations for how it should be used. This change updates the
documentation block to include those recommendations.

Reviewed By: yfeldblum

Differential Revision: D4688700

fbshipit-source-id: 40350c2330084a28f1dc0744176d0ac8a0e64d39

5 years agoMake EventBaseLoopController only support VirtualEventBase v2017.03.13.00
Andrii Grynenko [Sat, 11 Mar 2017 01:06:58 +0000 (17:06 -0800)]
Make EventBaseLoopController only support VirtualEventBase

Summary: EventBase support is achieved through default VirtualEventBase mechanism.

Reviewed By: yfeldblum

Differential Revision: D4685948

fbshipit-source-id: 15c8c789a55776984aa9087455e3f7b79d7604ad

5 years agoFix data race in IndexedMemPool
Maged Michael [Fri, 10 Mar 2017 23:34:07 +0000 (15:34 -0800)]
Fix data race in IndexedMemPool

Summary:
IndexedMemPool uses regular memory for the global and local next links. There can be concurrent reads and writes of such links. In order to avoid C++ undefined behavior due to the data races, these links should be atomic. Relaxed loads and stores are sufficient for correctness.

Depends on D4680286

Reviewed By: nbronson

Differential Revision: D4680317

fbshipit-source-id: 27c0d888e08bd5d862aee4c6dc6ee393033b32e3

5 years agoFix problems with clock_gettime and OSX < 10.12 + XCode 8
Christopher Dykes [Fri, 10 Mar 2017 21:52:11 +0000 (13:52 -0800)]
Fix problems with clock_gettime and OSX < 10.12 + XCode 8

Summary: It's a pain, so rely on macros to figure a few things about the build configuration.

Reviewed By: yfeldblum

Differential Revision: D4229128

fbshipit-source-id: 78b414c21cae6ba51ade2ab75b117cccad5c608c

5 years agoUse VirtualEventBase in mcrouter
Andrii Grynenko [Fri, 10 Mar 2017 19:40:59 +0000 (11:40 -0800)]
Use VirtualEventBase in mcrouter

Summary:
This switches mcrouter to use VirtualEventBase as the unified way of managing its threads. This allows multiple mcrouters to be run on the same thread-pool, without having to keep separate shutdown logic.

As an immediate win - this eliminates any special shutdown logic from tcc and standalone mcrouter.

Reviewed By: jmswen

Differential Revision: D4570982

fbshipit-source-id: 3974279005150e3e1db16230fde6b5951ae016b9

5 years agoFix use after move in AsyncSSLSocket constructor.
Kyle Nekritz [Fri, 10 Mar 2017 17:36:52 +0000 (09:36 -0800)]
Fix use after move in AsyncSSLSocket constructor.

Summary: The new AsyncSocket is already constructed at this point so just use the event base from it.

Reviewed By: djwatson

Differential Revision: D4689395

fbshipit-source-id: aac898c1c6e6e0e0ffcb20b16bbf2b842cc31d54

5 years agoImplementing a callback interface for folly::AsyncSocket allowing to supply an ancill...
Maxim Georgiev [Fri, 10 Mar 2017 06:19:15 +0000 (22:19 -0800)]
Implementing a callback interface for folly::AsyncSocket allowing to supply an ancillary data buffer with msghdr structure to sendmsg() system call

Summary: Implementing a callback interface for folly::AsyncSocket allowing to supply an ancillary data buffer with msghdr structure to sendmsg() system call.

Reviewed By: afrind

Differential Revision: D4422168

fbshipit-source-id: 29a23b05f704aff796d368f4ac9514c49b7ce578

5 years agoRemove runInLoop
Andrii Grynenko [Fri, 10 Mar 2017 04:34:14 +0000 (20:34 -0800)]
Remove runInLoop

Summary: Existing runInLoop implementation was generally not safe (it did not capture the KeepAlive token). runInLoop() is mostly used in library code though, and often times it's ok not capture the KeepAlive token. It's better to just have the caller make the decision, rather then keep runInLoop() and make it always capture the KeepAlive token.

Reviewed By: yfeldblum

Differential Revision: D4684025

fbshipit-source-id: 65907bbe9c774e2a7b92580d0c0387d346b495d5

5 years agoteach gtest how to pretty-print StringPiece values
Adam Simpkins [Fri, 10 Mar 2017 01:54:21 +0000 (17:54 -0800)]
teach gtest how to pretty-print StringPiece values

Summary:
Even though an ostream operator<<() is defined for StringPiece, gtest doesn't
use it when printing StringPiece values in test failure messages.  Because
StringPiece defines a nested iterator type gtest instead treats StringPiece as
a container, and prints each of its elements (characters) one-by-one.  This is
fairly awkward to read.

This diff defines an explicit PrintTo() function for StringPiece in
folly/test/TestUtils.h.  This makes gtest print StringPiece values nicely if
you include TestUtils.h in your test sources.

Reviewed By: yfeldblum

Differential Revision: D4672257

fbshipit-source-id: 4b39ccc116e5382c29c37c2abe879293d310faf5

5 years agoDetect data race in IndexedMemPool
Maged Michael [Thu, 9 Mar 2017 19:21:50 +0000 (11:21 -0800)]
Detect data race in IndexedMemPool

Summary: IndexedMemPool uses regular memory for the global and local next links. There is a data race as there can be concurrent reads and writes of such links. Added a test for data races.

Reviewed By: nbronson

Differential Revision: D4680286

fbshipit-source-id: 9ef3ed439b9df07d69afe570e516c146caa53a6c

5 years agoReplace MSG_PEEK with a pre-received data interface.
Kyle Nekritz [Thu, 9 Mar 2017 16:25:27 +0000 (08:25 -0800)]
Replace MSG_PEEK with a pre-received data interface.

Summary: MSG_PEEK was difficult if not impossible to use well since we do not provide a way wait for more data to arrive. If you are using setPeek on AsyncSocket, and you do not receive the amount of data you want, you must either abandon your peek attempt, or spin around the event base waiting for more data. This diff replaces the peek interface on AsyncSocket with a pre-received data interface, allowing users to insert data back onto the front of connections after reading some data in another layer.

Reviewed By: djwatson

Differential Revision: D4626315

fbshipit-source-id: c552e64f5b3ac9e40ea3358d65b4b9db848f5d74

5 years agoFixes to always call Singleton's registrationComplete()/initFacebook()
Andrii Grynenko [Wed, 8 Mar 2017 22:13:08 +0000 (14:13 -0800)]
Fixes to always call Singleton's registrationComplete()/initFacebook()

Reviewed By: yfeldblum

Differential Revision: D4671818

fbshipit-source-id: 80154adfef238f0be62da12fb843e22de1ae65ca

5 years agoMark non-captured constexpr variable as static
Christopher Dykes [Wed, 8 Mar 2017 20:38:27 +0000 (12:38 -0800)]
Mark non-captured constexpr variable as static

Summary:
GCC & Clang allow the use of a non-captured constexpr local within a lambda, MSVC does not.

All 3 compilers allow this when the local has static storage.

Reviewed By: yfeldblum

Differential Revision: D4675147

fbshipit-source-id: 3e07eb77731cbd3ebda6aee4b0b57ac19ac34de5

5 years agoDon't define MALLOCX_* if building with JEMalloc
Christopher Dykes [Wed, 8 Mar 2017 18:48:01 +0000 (10:48 -0800)]
Don't define MALLOCX_* if building with JEMalloc

Summary:
With the way it is currently, if `jemalloc.h` is included after `folly/Malloc.h`, `jemalloc.h` ends up redefining these.
This changes it to just include `jemalloc.h` if we know we're compiling with JEMalloc.

Reviewed By: yfeldblum

Differential Revision: D4660285

fbshipit-source-id: 814e36432676e89cf7ba2bc5247b7dde2bfdc14a

5 years agoFix mcrouter-oss build
Anirudh Ramachandran [Wed, 8 Mar 2017 18:10:42 +0000 (10:10 -0800)]
Fix mcrouter-oss build

Summary: D4624754 broke mcrouter-oss because that uses older OpenSSL (1.0.1).

Reviewed By: yfeldblum, jmswen

Differential Revision: D4670767

fbshipit-source-id: f297c8d3e013525bb36bd059eb40c70f0c4c2dcc

5 years agostd::make_unique -> folly::make_unique for older platforms
Jon Maltiel Swenson [Wed, 8 Mar 2017 17:41:26 +0000 (09:41 -0800)]
std::make_unique -> folly::make_unique for older platforms

Summary: Title.

Reviewed By: yfeldblum

Differential Revision: D4674125

fbshipit-source-id: 71ad38d7a52ab1c27fcc7ff14d130978b0ece382

5 years agoDefault VirtualEventBase
Andrii Grynenko [Wed, 8 Mar 2017 00:54:44 +0000 (16:54 -0800)]
Default VirtualEventBase

Summary:
Make each EventBase have a defuault VirtualEventBase which is attached to it and is joined on destruction. Default VirtualEventBase is lazily created on first request.
This makes it trivial to use code switched to VirtualEventBase both with VirtualEventBase and EventBase.

Reviewed By: yfeldblum

Differential Revision: D4644639

fbshipit-source-id: cf28a3632463a1c61404c225ce1186f5a4a062a3

5 years agoDial EventBase loop time logging up to 11
Alexander Blom [Tue, 7 Mar 2017 11:27:21 +0000 (03:27 -0800)]
Dial EventBase loop time logging up to 11

Summary:
There is no need for EventBase to log every loop iteration on a relatively low VLOG level.
Just move it up to 11 like the other verbose logging in EventBase.cpp

Reviewed By: yfeldblum

Differential Revision: D4642379

fbshipit-source-id: 351636afb186ba94b2447b206d7d1389cbdba110

5 years agoAdd a few more methods to OpenSSLCertUtils
Anirudh Ramachandran [Tue, 7 Mar 2017 08:26:12 +0000 (00:26 -0800)]
Add a few more methods to OpenSSLCertUtils

Summary:
Add a few more getters (subject, issuer, notbefore, notafter) to
OpenSSLCertUtils.

Also add an additional API to AsyncSSLSocket to add application-generated
alert strings during the handshake, for e.g., during certificate verification

Reviewed By: knekritz

Differential Revision: D4624754

fbshipit-source-id: f01998b9e0e58b88ece8c6dc51ab590988bf0a8f

5 years agofixing a compile error on windows v2017.03.06.00
Johannes Kopf [Sun, 5 Mar 2017 23:58:16 +0000 (15:58 -0800)]
fixing a compile error on windows

Summary:
This file only compiles with MSVC after I made the change in this diff.

The code was difficult to follow, so I am not sure if the template is needed here?

Reviewed By: yfeldblum

Differential Revision: D4656575

fbshipit-source-id: 19415623bdb1757b573f12cdc9d8544d716ce6f2

5 years agofolly/Foreach.h: avoid shadowing warnings
Jim Meyering [Sun, 5 Mar 2017 18:32:27 +0000 (10:32 -0800)]
folly/Foreach.h: avoid shadowing warnings

Summary:
Fix warnings exposed by gcc's -Wshadow-compatible-local.
Don't use a hard-coded names that would cause shadowing with each nested use of this macro.

This avoids errors like the following:

  folly/Foreach.h:94:18: error: declaration of 'FOR_EACH_privateCount' shadows a previous local [-Werror=shadow-compatible-local]
  folly/Foreach.h:91:12: error: shadowed declaration is here [-Werror=shadow-compatible-local]

Reviewed By: yfeldblum

Differential Revision: D4656783

fbshipit-source-id: d7cd820fb04ccd78242f7487e2b219dc300709b6

5 years agoAdd getTotalConnectTimeout method
Subodh Iyengar [Sun, 5 Mar 2017 01:46:28 +0000 (17:46 -0800)]
Add getTotalConnectTimeout method

Summary:
AsyncSSLSocket can be connected using connect with
total timeouts.

This adds a method to get the total timeout that
was set.

Reviewed By: ngoyal, yfeldblum

Differential Revision: D4656331

fbshipit-source-id: a55ad9f081449d358b8133e9598a2063f625a2e6

5 years agoRemove a RequestContext deadlock
Marc Celani [Sat, 4 Mar 2017 06:27:50 +0000 (22:27 -0800)]
Remove a RequestContext deadlock

Summary:
It is not unusual to write a RequestData that will send a request to a backend service upon destruction (for example, to log the request data). This typically involves calling EventBase::runInEventBaseThread() to do the logging. Normally, if you wait until the RequestContext is destructing for this to happen, everything works fine. However, if you explicitly call clearContextData(), then this can cause a deadlock. clearContextData() holds the write lock on the data_. Imagine the NotificationQueue thread tries to set the context (for some other reason), and it is waiting on the read lock on the data_. It does so while holding a lock on its own queue. Now, the clearContextData() calls the destructor while holding the write lock on data_, and the destructor calls runInEventBaseThread, which grabs the queue's lock. Deadlock.

This can either be fixed in NotificationQueue or RequestContext. I decided to fix it in RequestContext because there may be other deadlocks that don't involve NotificationQueue (see the test plan).

There are other deadlocks in RequestContext if you call write methods in RequestData::onSet or onUnset. But fixing this would require a massive perf regression (I imagine using shared_ptr instead of unique_ptr, and copying the ptrs under readlock and calling the callbacks after releasing the lock - yuck!) So instead, warn people against doing that.

Reviewed By: yfeldblum

Differential Revision: D4655400

fbshipit-source-id: cfebe696953a19dc4dba49976bbcd0ec1363bc42

5 years agoTweak size handling in Endian
Yedidya Feldblum [Fri, 3 Mar 2017 05:32:09 +0000 (21:32 -0800)]
Tweak size handling in Endian

Summary:
[Folly] Tweak size handling in `Endian`.

Either multiply by 8 in one place, or divide by 8 in another place. Go with the latter, because `sizeof` in C++ emits byte sizes, not bit sizes.

Reviewed By: simpkins

Differential Revision: D4645020

fbshipit-source-id: cb78600ba4c20bebc66aed506d4b5d6c378fc998

5 years agoRevert D4618623: Give observer manager threads a name
Christopher Dykes [Thu, 2 Mar 2017 21:44:59 +0000 (13:44 -0800)]
Revert D4618623: Give observer manager threads a name

Summary: This reverts commit 21b86944f513da0700039a3e3736937fd72e8887

Differential Revision: D4618623

fbshipit-source-id: 6ab6498a2ce22895d7052e258d1af216d46ca178

5 years agoAdd a method to AsyncSSLSocket to supply connect timeouts
Subodh Iyengar [Thu, 2 Mar 2017 17:00:14 +0000 (09:00 -0800)]
Add a method to AsyncSSLSocket to supply connect timeouts

Summary:
The current method to supply connect timeouts to AsyncSSLSocket
is to supply only the max connect + ssl connect time.

However in some cases when connect time is known, it is desirable
to supply it so that if connect does not succeed in connect time
we can error out quicker and retry.

This adds a new method for connect time in addition to the total
Connect time.

An alternative to this would be to create a AsyncSocket, connect it
and then pass it's fd to AsyncSSLSocket, however that approach does
not work well when TFO is being used, because TFO State is a part of
the AsyncSocket's state and that is not encapsulated with the fd transfer.

We could move the state around, but that is error prone, and this is much
simpler and isolated to AsyncSSLSocket.

Reviewed By: knekritz

Differential Revision: D4626924

fbshipit-source-id: d802d035efbced68873ab59314d9f0e661e5509b

5 years agoRefactor Endian
Yedidya Feldblum [Thu, 2 Mar 2017 05:20:44 +0000 (21:20 -0800)]
Refactor Endian

Summary:
[Folly] Refactor `Endian`.

Define `swap` for all types, instead of only declaring it for all types, but have its implementation call a function that only has a given set of non-template overloads for a given set of supported types.

The effective change is that now `Endian::swap<std::string>` will fail to compile, rather than compile but fail to link.

And support floating-point types.

Reviewed By: nbronson

Differential Revision: D4615706

fbshipit-source-id: 5034e1e4466b8118a258d917ce8f4094460ca01a

5 years agoFix ASAN integration with detect_stack_use_after_return turned on
Andrii Grynenko [Wed, 1 Mar 2017 18:37:55 +0000 (10:37 -0800)]
Fix ASAN integration with detect_stack_use_after_return turned on

Summary: detect_stack_use_after_return enables fake stack logic. Previous implementation was relying on fiber releasing its fake stack before switch, which is not safe, because that fake stack may actually be in use. This diff implements a hacky workaround by building a freeFakeStack function, which can release fake stack outside of the fiber. Ideally this function should be provided by ASAN itself.

Reviewed By: yfeldblum

Differential Revision: D4631962

fbshipit-source-id: 6213e2a6b3cbc35fae3793fcf86b0614dd8ff825

5 years agoDisable MemoryIdler::unmapUnusedStack with ASAN
Andrii Grynenko [Wed, 1 Mar 2017 18:35:36 +0000 (10:35 -0800)]
Disable MemoryIdler::unmapUnusedStack with ASAN

Summary: It doesn't work nicely with fake stacks (which are used with detect_stack_use_after_return).

Reviewed By: yfeldblum

Differential Revision: D4631995

fbshipit-source-id: f139041314f8f64c0cf0a15874d6a4710fb2a5b6

5 years agorename Range template arg for increased readability
Nathan Bronson [Wed, 1 Mar 2017 17:53:28 +0000 (09:53 -0800)]
rename Range template arg for increased readability

Summary:
Parameterizing Range by the template arg T is potentially
confusing because it violates the typical container pattern of T being
the value type.  This diff uses Range<Iter> consistently in Range.h when
the template arg refers to a fully general iterator.

Reviewed By: yfeldblum

Differential Revision: D4635270

fbshipit-source-id: f70bb2e3e23ab9875fba3dbf7b11492613a037b8

5 years agoNew C++17 backport: folly::enable_shared_from_this
Philipp Unterbrunner [Wed, 1 Mar 2017 06:14:32 +0000 (22:14 -0800)]
New C++17 backport: folly::enable_shared_from_this

Summary:
C++14 has no direct means of creating a std::weak_ptr, one must always create a (temporary) std::shared_ptr first. C++17 adds weak_from_this() to std::enable_shared_from_this to avoid that overhead.
Alas code that is meant to compile under different language versions cannot rely on std::enable_shared_from_this::weak_from_this().
This new utility class uses SFINAE to call std::enable_shared_from_this<T>::weak_from_this() if available. Falls back to std::enable_shared_from_this<T>::shared_from_this() otherwise.
Use as a drop-in replacement for std::enable_shared_from_this where weak_from_this() is desired.

Reviewed By: yfeldblum

Differential Revision: D4616394

fbshipit-source-id: 73bf6cd8852d4a33478a9280cc69c81e7ea73423

5 years agoGive observer manager threads a name
Haijun Zhu [Tue, 28 Feb 2017 06:45:34 +0000 (22:45 -0800)]
Give observer manager threads a name

Summary: As title, convenient for debugging. Asked for from D4542856

Reviewed By: nkg-

Differential Revision: D4618623

fbshipit-source-id: 21b86944f513da0700039a3e3736937fd72e8887

5 years agoUpdate googletest to 1.8.0
Ted Percival [Mon, 27 Feb 2017 23:01:14 +0000 (15:01 -0800)]
Update googletest to 1.8.0

Summary:
`make check` was failing because <gmock.h> is included but not supplied. Updating to googletest 1.8.0 includes both googletest & googlemock, so `make check` now runs (and passes).
Closes https://github.com/facebook/folly/pull/547

Reviewed By: igorsugak

Differential Revision: D4593403

Pulled By: Orvid

fbshipit-source-id: 02f80e7b67e17562ff981e9c80a37f985eb685be

5 years agoMove some tests out of FOLLY_ALLOW_TFO ifdef.
Kyle Nekritz [Mon, 27 Feb 2017 23:00:11 +0000 (15:00 -0800)]
Move some tests out of FOLLY_ALLOW_TFO ifdef.

Summary: I think this ended up encompassing more than intended.

Reviewed By: yfeldblum, maxgeorg

Differential Revision: D4622645

fbshipit-source-id: 26597098a479b85d6ecc64abef2ecc53637ee759

5 years agoDon't call memcpy with empty buffer in IOBuf::copyBuffer().
Kyle Nekritz [Mon, 27 Feb 2017 18:48:39 +0000 (10:48 -0800)]
Don't call memcpy with empty buffer in IOBuf::copyBuffer().

Summary: Fixes "undefined behavior" in ubsan builds when using an empty ByteRange.

Reviewed By: siyengar

Differential Revision: D4613663

fbshipit-source-id: 4b53ba764609acc986340f10613f84585fa697cf

5 years agoKill checks in configure.ac that aren't used
Christopher Dykes [Mon, 27 Feb 2017 03:20:16 +0000 (19:20 -0800)]
Kill checks in configure.ac that aren't used

Summary: They are checks that aren't actually used anywhere to guard anything, so kill them. Many are for things that have long since been standard, or aren't even used.

Reviewed By: yfeldblum

Differential Revision: D4615947

fbshipit-source-id: 2179caff4ef8583c89efedbff3136ae5fc552424

5 years agoUse cstdint instead of inttypes.h
Christopher Dykes [Mon, 27 Feb 2017 03:19:28 +0000 (19:19 -0800)]
Use cstdint instead of inttypes.h

Summary: This is the only place `inttypes.h` is used in Folly and it's used in a context where `cstdint` would have sufficed.

Reviewed By: yfeldblum

Differential Revision: D4619248

fbshipit-source-id: e67dccb7d8bf1ad5f7539d4b039590253e8d8c79