folly.git
5 years agoMake folly::getCurrentThreadId() return a thread ID on OSX
Christopher Dykes [Fri, 14 Apr 2017 20:54:32 +0000 (13:54 -0700)]
Make folly::getCurrentThreadId() return a thread ID on OSX

Summary: It was previously returning a pointer masquerading as a `uint64_t`.

Reviewed By: yfeldblum

Differential Revision: D4888325

fbshipit-source-id: 6cf6ed09f0e7a39a8a2f93d3fa14b06913c27805

5 years agoAdd FOLLY_HAVE_PTRHEAD
Christopher Dykes [Fri, 14 Apr 2017 20:53:52 +0000 (13:53 -0700)]
Add FOLLY_HAVE_PTRHEAD

Summary:
Folly is gaining support for compiling without PThread support on Windows, but we need a way to know that at compile-time for certain APIs.
This also includes changes to guard the only API outside of portability/PThread.h that needs to be guarded on the existence of PThread.

Reviewed By: yfeldblum

Differential Revision: D4889526

fbshipit-source-id: 21175ad90f60a47718c7e2775e3b429b2aad62e2

5 years agoAdd support for building without PThread to the CMake files
Christopher Dykes [Fri, 14 Apr 2017 20:51:43 +0000 (13:51 -0700)]
Add support for building without PThread to the CMake files

Summary: Because, soon it won't be required.

Reviewed By: yfeldblum

Differential Revision: D4892076

fbshipit-source-id: 6bfbf5bf523262659c0af336f659398ae8545e50

5 years agoRemove the old SpinLock implementations
Christopher Dykes [Fri, 14 Apr 2017 18:14:45 +0000 (11:14 -0700)]
Remove the old SpinLock implementations

Summary: They aren't actually needed as the primary implementation is supported on all platforms.

Reviewed By: yfeldblum

Differential Revision: D4882687

fbshipit-source-id: 7208c5d3c1f35b29b0cabb6a20fe030fbf10b131

5 years agoRevert D4805628: [folly] Suppress more warnings for MSVC
Mark Williams [Fri, 14 Apr 2017 16:00:08 +0000 (09:00 -0700)]
Revert D4805628: [folly] Suppress more warnings for MSVC

Summary: This reverts commit 9574ea984e1e3daca01101259687f46effcf3a9f

Differential Revision: D4805628

fbshipit-source-id: 3285317a4c3f2012da1f13e9c91b6c623726c9b7

5 years agoInitialize the singleton-vault early in init()
Yedidya Feldblum [Fri, 14 Apr 2017 07:22:47 +0000 (00:22 -0700)]
Initialize the singleton-vault early in init()

Summary:
Initialize the singleton-vault early in `init()` to avoid ordering problems with other parts of `init()`.

Initialization is done by calling `SingletonVault::singleton()->registrationComplete()`. Do that before (almost) everything, in particular, before parsing args and initializing logging.

Note that this just marks it okay for singletons to be instantiated, but does not by itself instantiate any singletons. However, this is useful if, in some way, parsing args or initializing logging should happen to use singletons.

Reviewed By: andriigrynenko

Differential Revision: D4889777

fbshipit-source-id: 6817db22d38a498cd20e361fc574a146029432ec

5 years agoSuppress more warnings for MSVC
Andrew Krieger [Fri, 14 Apr 2017 07:06:53 +0000 (00:06 -0700)]
Suppress more warnings for MSVC

Summary:
Several other warnings that aren't reasonable to disable globally occur in folly headers.
- Wrap the unreachable code warnings in MSVC specific disable blocks to prevent problems for users.
- Cast to signed types to perform integral negation before casting back to unsigned for bit operations.
- Enable a simpler overload for bool->float conversion than one which attempts float->bool.
- Delete one unneeded undef.

(Note: this ignores all push blocking failures!)

Reviewed By: yfeldblum

Differential Revision: D4805628

fbshipit-source-id: 9574ea984e1e3daca01101259687f46effcf3a9f

5 years agoRemove pthread dependency from CacheLocality.h
Andrew Krieger [Fri, 14 Apr 2017 05:57:20 +0000 (22:57 -0700)]
Remove pthread dependency from CacheLocality.h

Summary:
For getting a thread id for hashing, std::this_thread::get_id()
is just as good and also removes an unneeded pthreads dependency. This enables
using MPMCQueue on Windows without pthreads as well.

Reviewed By: Orvid, yfeldblum

Differential Revision: D4879940

fbshipit-source-id: c4fb5eea165eb450240f94904aa26a10aa394d1b

5 years agoSwitch TimedRWMutex implementation to use lock_guard where possible
Christopher Dykes [Fri, 14 Apr 2017 04:25:29 +0000 (21:25 -0700)]
Switch TimedRWMutex implementation to use lock_guard where possible

Summary: As titled. There are some places where scoping prevents trivial conversion, so I've left those places with explicit lock/unlock calls.

Reviewed By: andriigrynenko, yfeldblum

Differential Revision: D4888812

fbshipit-source-id: c124306e0373ee9eb395cb98f88df63c91523d48

5 years agoUse getCurrentThreadID() rather than pthread_self() in MemoryIdler
Christopher Dykes [Fri, 14 Apr 2017 04:19:08 +0000 (21:19 -0700)]
Use getCurrentThreadID() rather than pthread_self() in MemoryIdler

Summary: Ignoring the questionability of the approach it's being used in, it's better to not have the direct PThread dependency.

Reviewed By: yfeldblum

Differential Revision: D4889245

fbshipit-source-id: da099c6f938dbe98c1b9eeaf4de0a27a2c4d65f1

5 years agoExplicitly use CreateFileA in readlink
Andrew Krieger [Fri, 14 Apr 2017 02:23:50 +0000 (19:23 -0700)]
Explicitly use CreateFileA in readlink

Summary:
In Unicode enabled projects, this errors because CreateFile aliases
CreateFileW, which takes wchar_t* not char*.

Reviewed By: Orvid, yfeldblum

Differential Revision: D4878424

fbshipit-source-id: b44b369c0533e74163f68d95c2bf353584033731

5 years agoSupport folly::getCurrentThreadID() without PThread
Christopher Dykes [Fri, 14 Apr 2017 01:52:20 +0000 (18:52 -0700)]
Support folly::getCurrentThreadID() without PThread

Summary: Just use the native Windows API call instead, as it gives the same result.

Reviewed By: yfeldblum

Differential Revision: D4887587

fbshipit-source-id: adbfd288bea425f1aff46d0685083807490f7f09

5 years agoDon't use pthread_spinlock_t in TimedRWMutex
Christopher Dykes [Fri, 14 Apr 2017 00:56:39 +0000 (17:56 -0700)]
Don't use pthread_spinlock_t in TimedRWMutex

Summary: `TimedMutex` was already using `folly::SpinLock`, so switch `TimedRWMutex` to do the same.

Reviewed By: andriigrynenko

Differential Revision: D4888005

fbshipit-source-id: 6e782347bc22dc186ed41f2e77c6614b8444bae2

5 years agoSwitch includes of PThread to the portability header
Christopher Dykes [Thu, 13 Apr 2017 20:57:26 +0000 (13:57 -0700)]
Switch includes of PThread to the portability header

Summary:
As part of the work towards getting Folly compiling without PThread, it needs to be treated as a non-portable include.
This switches the includes not already covered by other diffs.

Reviewed By: yfeldblum

Differential Revision: D4882777

fbshipit-source-id: d9521564c814f9bcff2fcb358cbb89b3777c327d

5 years agoFix 1/2 of exception_wrapper under MSVC
Christopher Dykes [Thu, 13 Apr 2017 19:45:06 +0000 (12:45 -0700)]
Fix 1/2 of exception_wrapper under MSVC

Summary:
MSVC didn't like referring to members of `exception_wrapper` in the initializer for a `static constexpr` field directly in `exception_wrapper`, so shift the initialization to the actual definition of the fields.
As the fields are only referred to via their address, dropping the `constexpr` loses nothing.

Reviewed By: ericniebler

Differential Revision: D4873939

fbshipit-source-id: 30b690b1ab3f2f7a25b9dc4863b46f64c811797d

5 years agoAdd method to check if SSL Lock is disabled
Neel Goyal [Thu, 13 Apr 2017 19:14:40 +0000 (12:14 -0700)]
Add method to check if SSL Lock is disabled

Summary:
Add a method where users can determine if a SSL lock is disabled.

This can help when it comes to making decisions about things like whether reusing SSL Contexts is safe in multithreaded programs.

Reviewed By: siyengar

Differential Revision: D4875780

fbshipit-source-id: 91e9259fee25856be1b77823559d16d0679bde5b

5 years agoKill a couple of PThread includes
Christopher Dykes [Thu, 13 Apr 2017 18:24:51 +0000 (11:24 -0700)]
Kill a couple of PThread includes

Summary: They aren't actually needed, and are in the way of being able to build Folly without PThread.

Reviewed By: yfeldblum

Differential Revision: D4882410

fbshipit-source-id: e4a48600f79c57bb965a35fab94962a29b8e06d4

5 years agoAdd CMake build system
Christopher Dykes [Thu, 13 Apr 2017 05:25:15 +0000 (22:25 -0700)]
Add CMake build system

Summary: This is for use on Windows only, and does not support Linux or OSX in the slightest. There are probably still a few hacks laying around in it, but it works, and that's good enough for now.

Reviewed By: yfeldblum

Differential Revision: D4873498

fbshipit-source-id: 5952a927037dc1d56c51e71d51ba506d2cd17357

5 years agoAdd bzip2 support
Nick Terrell [Thu, 13 Apr 2017 02:43:02 +0000 (19:43 -0700)]
Add bzip2 support

Summary:
Adds bzip2 support to `folly/io/Compression.h`.
Adds bzip2 to the default set of supported codecs for the `AutomaticCodec`.

Reviewed By: yfeldblum

Differential Revision: D4873771

fbshipit-source-id: d4f4861aef7e4b9efb67095e8892c265b5ae5557

5 years agoFix the last issues with exception_wrapper under MSVC
Christopher Dykes [Thu, 13 Apr 2017 01:19:37 +0000 (18:19 -0700)]
Fix the last issues with exception_wrapper under MSVC

Summary:
Well, MSVC 2017 anyways.
MSVC 2017 wasn't able to handle the decltype in the type specialization, which was also entirely unnecesary.

Reviewed By: yfeldblum

Differential Revision: D4880272

fbshipit-source-id: b6deed72a12d4b58d6eca200e7287e536f94ca30

5 years agoReplace Subprocess::pipe* syntax sugar with Subprocess::Options().pipe*
Alexey Spiridonov [Wed, 12 Apr 2017 21:43:04 +0000 (14:43 -0700)]
Replace Subprocess::pipe* syntax sugar with Subprocess::Options().pipe*

Summary:
This is a bit too magical -- it's not clear that the thing produces an Options object. If you do know that you can chain further option setters off this thing, it's nice, but otherwise, the first impression is "what just happened?".

So, let's have one good way for doing things.

Reviewed By: yfeldblum

Differential Revision: D4863947

fbshipit-source-id: 3dfe83cfc077d47f604f47dcb21149fbaa2d2243

5 years agoDelete | operator for Subprocess::Options
Alexey Spiridonov [Wed, 12 Apr 2017 21:43:03 +0000 (14:43 -0700)]
Delete | operator for Subprocess::Options

Summary:
This operator is WRONG. It has not worked correctly for years, e.g. it lacks support for chdir, and several other options.

The operator is not really useful after C++11. Usually, you should just chain setters, e.g. `Subprocess::Options().pipeStdout().pipeStderr()`. If you must repeatedly mutate options in a fixed way, in the C++11 world you can use a lambda instead.

Reviewed By: yfeldblum

Differential Revision: D4862698

fbshipit-source-id: a2d8ace53424b9232e178cf202cf51beb7b59b12

5 years agoFix zlib + lzma memory usage inefficiency
Nick Terrell [Wed, 12 Apr 2017 19:44:56 +0000 (12:44 -0700)]
Fix zlib + lzma memory usage inefficiency

Summary:
Since IOBuf rounds to a good malloc size, zlib and lzma waste space.
For a 4 MiB `length`, about 1 MiB of the IOBuf is wasted.

Reviewed By: yfeldblum

Differential Revision: D4872830

fbshipit-source-id: 42fc83277b2dae22b75403e0e71c43e8f2b19f21

5 years agocodemod: ASN1_STRING_data -> ASN1_STRING_get0_data
Jude Taylor [Wed, 12 Apr 2017 17:57:54 +0000 (10:57 -0700)]
codemod: ASN1_STRING_data -> ASN1_STRING_get0_data

Reviewed By: anirudhvr

Differential Revision: D4830205

fbshipit-source-id: 55f34c3bddead8a73174c403f76572248c541a10

5 years agoMake folly::StrictConjunction usable under MSVC
Christopher Dykes [Wed, 12 Apr 2017 17:44:00 +0000 (10:44 -0700)]
Make folly::StrictConjunction usable under MSVC

Summary:
Template pack expansion plays much nicer when being expanded into the template parameters of a base class than it does as the template parameters to a templated `using` when under MSVC.
This fixes the other half of `folly::exception_wrapper`.

Reviewed By: yfeldblum

Differential Revision: D4874047

fbshipit-source-id: 6593d10f8737b940bd550353e54f43bc96402a48

5 years agoProvide a copy ctor for Digest that copies the current hash context
Jude Taylor [Wed, 12 Apr 2017 16:40:55 +0000 (09:40 -0700)]
Provide a copy ctor for Digest that copies the current hash context

Summary: opensource changes for t16702532

Reviewed By: yfeldblum

Differential Revision: D4846899

fbshipit-source-id: 67a610ff399e95c7cf1c9c8a5950f79bfc3aabb9

5 years agoAdd memory padding and alignment to prevent false sharing
Artem Lantsev [Wed, 12 Apr 2017 09:16:51 +0000 (02:16 -0700)]
Add memory padding and alignment to prevent false sharing

Summary: This is a follow up commit for the thread https://github.com/facebook/folly/pull/378

Reviewed By: nbronson

Differential Revision: D4860356

fbshipit-source-id: f10a0d12a593c18b1abf94da5b477c524c04f4be

5 years agoSupport linux namespace clone flags
Aravind Anbudurai [Wed, 12 Apr 2017 08:36:29 +0000 (01:36 -0700)]
Support linux namespace clone flags

Summary:
This diffs adds supports on folly::Subprocess to be able to take clone flags and
use them to call clone(2) instead of the default vfork()

I checked that all tests pass when I replace vfork with fork on trunk. So there
isn't anything built assuming the parent is paused for an execve. Correct me if
I am wrong here.

(Note: this ignores all push blocking failures!)

Reviewed By: snarkmaster

Differential Revision: D4853169

fbshipit-source-id: 7e5851df3a49996a4a5dc1945457686dd042e1f4

5 years agoIPAddressV(46) methods to convert IPs to their inverse in-addr.arpa / ip6.arpa repres...
Angelo Failla [Wed, 12 Apr 2017 04:13:21 +0000 (21:13 -0700)]
IPAddressV(46) methods to convert IPs to their inverse in-addr.arpa / ip6.arpa representation

Summary: As per the title, this adds methods to convert IPv(46) to their inverse representation, this is useful for applications that needs to calculate the inverse arpa representation, typically used when making PTR DNS requests.

Reviewed By: Orvid

Differential Revision: D4867502

fbshipit-source-id: 190e5c309b17a633e1c97b077f212ab38725860f

5 years agonon-throwing, non-allocating exception_wrapper
Eric Niebler [Tue, 11 Apr 2017 23:12:59 +0000 (16:12 -0700)]
non-throwing, non-allocating exception_wrapper

Summary:
The purpose of this reimplementation of `exception_wrapper` is threefold:

- Make `exception_wrapper` smaller. It goes from 48 bytes to 24.
- Give it `noexcept` ~~copy and~~ move
- Store small exception objects in an internal buffer; i.e., with zero allocations.

The ultimate goal is to change `folly::Try<T>` to a thin wrapper over `folly::Expected<T, exception_wrapper>`. (Currently, it stores the `exception_wrapper` on the heap.)

As part of this redesign, I:

- Remove `exception_wrapper::getCopied`. The user shouldn't care how the `exception_wrapper` stores the exception.
- Remove `exception_wrapper::operator==`. It was only used in 2 places in test code. The existing semantics (return true IFF two `exception_wrapper`s point to the //same// exception object) prevented the small-object optimization.
- Add new `handle()` API that behaves like cascading `catch` clauses. For instance:
```lang=c++
exception_wrapper ew = ...;
ew.handle(
    [&](const SomeException& e) { /*...*/ },
    [&](const AnotherException& e) { /*...*/ },
    [&](...) { /* catch all*/ }, // yes, lambda with ellipses works!
```
- Add a `type()` member for accessing the `typeid` of the wrapped exception, if it's known or can be determined with a `catch(std::exception&)`.

This table shows the percent improvement for the exception_wrapper_benchmark test:

| Test  | Percent improvement (gcc-5)  | Percent improvement (gcc-4)
| -----  | -----  | -----
| exception_wrapper_create_and_test  | 14.33%    | -6.50%
| exception_wrapper_create_and_test_concurrent | 11.91% | 20.15%
| exception_wrapper_create_and_throw | -0.82% | -0.25%
| exception_wrapper_create_and_cast | 15.02% | 14.31%
| exception_wrapper_create_and_throw_concurrent | 18.37% | 8.03%
| exception_wrapper_create_and_cast_concurrent | 28.18% | -10.77%

The percent win for gcc-5 is 15% on average. The non-throwing tests show a greater win since the cost of actually throwing an exception drowns out the other improvements. (One of the reasons to use `exception_wrapper` is to not need to throw in the first place.) On gcc-4, there is roughly no change since the gcc-4 standard exceptions (`std::runtime_error`, std::logic_error`) are non-conforming since they have throwing copy operations.

Reviewed By: yfeldblum

Differential Revision: D4385822

fbshipit-source-id: 63a8316c2923b29a79f8fa446126a8c37aa32989

5 years agoTreat OpenSSL as a non-portable include
Christopher Dykes [Tue, 11 Apr 2017 21:17:10 +0000 (14:17 -0700)]
Treat OpenSSL as a non-portable include

Summary: On Windows, the OpenSSL headers have to be included in a specific order relative to WinSock. Fixing the issues ad-hock is not a viable way to continue, so treat it as completely non-portable and require all includes of it in Folly to go through the portability header.

Reviewed By: yfeldblum

Differential Revision: D4856858

fbshipit-source-id: 56aca1fb0fe095f41a8af12488c6c2080344603d

5 years agoImplement clearenv in the portability headers
Christopher Dykes [Tue, 11 Apr 2017 19:26:53 +0000 (12:26 -0700)]
Implement clearenv in the portability headers

Summary:
It doesn't exist on OSX or Windows, so implement it.

Closes https://github.com/facebook/folly/issues/567

Reviewed By: yfeldblum

Differential Revision: D4783463

fbshipit-source-id: 0a2586aced7123b797a8e55a3e86124b449634e4

5 years agoFix TLRefCount::useGlobal to work with TSAN
Andrii Grynenko [Tue, 11 Apr 2017 06:30:46 +0000 (23:30 -0700)]
Fix TLRefCount::useGlobal to work with TSAN

Summary: TSAN has a limitation for the number of locks held concurrently - https://github.com/llvm-mirror/compiler-rt/blob/master/lib/sanitizer_common/sanitizer_deadlock_detector.h#L126, so we have to work around that.

Reviewed By: yfeldblum

Differential Revision: D4866477

fbshipit-source-id: 958251e97d91c1c46ef4e907de2cf189fb04f88e

5 years agoFix Fiber::LocalData leak for large fiber locals
Andrii Grynenko [Tue, 11 Apr 2017 06:14:38 +0000 (23:14 -0700)]
Fix Fiber::LocalData leak for large fiber locals

Summary: facepaw

Reviewed By: yfeldblum

Differential Revision: D4856227

fbshipit-source-id: 17a73d4d630c6efe1763bbd8aff8b7a2c7721c27

5 years agoRevert D4832473: [Folly] Disable EnvUtil::setAsCurrentEnvironment() on platforms...
Christopher Dykes [Tue, 11 Apr 2017 02:23:54 +0000 (19:23 -0700)]
Revert D4832473: [Folly] Disable EnvUtil::setAsCurrentEnvironment() on platforms without clearenv()

Summary: This reverts commit a80aabb5a223264746ab45e3138d065bce5fe99c

Differential Revision: D4832473

fbshipit-source-id: 1a78d61f3f09f1064876a786c15601a091f4402b

5 years agoRemove superfluous std::move
Alexander Shaposhnikov [Mon, 10 Apr 2017 22:24:47 +0000 (15:24 -0700)]
Remove superfluous std::move

Summary: Cleanup: remove superfluous std::move for return values.

Reviewed By: yfeldblum

Differential Revision: D4856421

fbshipit-source-id: bf2cb66f3b43eb8f3b183e22b0acd31e0a0c0958

5 years agoFix std::max() call in Compression.cpp
Nick Terrell [Mon, 10 Apr 2017 20:44:04 +0000 (13:44 -0700)]
Fix std::max() call in Compression.cpp

Summary:
`std::max(uint64_t, size_t)` was called.
Fixes https://github.com/facebook/folly/issues/576.

Reviewed By: Orvid

Differential Revision: D4861336

fbshipit-source-id: 1b6f67b291451048ba79d638d2c1184f9245dc0c

5 years agoFix the build... Whoops...
Christopher Dykes [Mon, 10 Apr 2017 19:14:41 +0000 (12:14 -0700)]
Fix the build... Whoops...

Summary: I landed a change to Folly that used a feature I had just re-written in Folly without updating the change. This has broken everything. Whoops.

Reviewed By: yfeldblum

Differential Revision: D4861562

fbshipit-source-id: 05beadbe233d1fdf90fb4435e65b718cdec7f63e

5 years agoRe-work the OpenSSL portability header to be a portability header
Christopher Dykes [Mon, 10 Apr 2017 18:10:21 +0000 (11:10 -0700)]
Re-work the OpenSSL portability header to be a portability header

Summary:
It was defining the same functions as OpenSSL, but it was putting them in the `folly::ssl` namespace and requiring a manual `use namespace` everywhere to actually be able to use it. Proper portability headers require no interaction to function correctly other than including them in the build.
This adjusts it so including it is all that is required to get an OpenSSL 1.1.0 compatible API.
This also re-organizes the code to significantly lower the amount of nesting in the `#if`'s, and puts it in order from oldest APIs to newest, rather than haphazard.

Reviewed By: yfeldblum

Differential Revision: D4856784

fbshipit-source-id: 7c00fdf4a3e51e37287d2b1ef294046f38a96a43

5 years agoClient RequestChannels and folly/python/futures.h
Jason Fried [Mon, 10 Apr 2017 18:05:00 +0000 (11:05 -0700)]
Client RequestChannels and folly/python/futures.h

Summary:
Use RequestChannel to construct Thrift Clients.

use folly/python/futures.h to marry up thrift client rpc futures to
asyncio futures. No gil exchanges

py3 clients are async context managers now.

Reviewed By: ambv, yfeldblum

Differential Revision: D4639677

fbshipit-source-id: 6b400747507446882d3cfcb1cf7e85c3daa07a2e

5 years agoFix folly/test:singleton_test - Singleton.SharedPtrUsage
Andrii Grynenko [Mon, 10 Apr 2017 17:44:31 +0000 (10:44 -0700)]
Fix folly/test:singleton_test - Singleton.SharedPtrUsage

Reviewed By: yfeldblum

Differential Revision: D4856138

fbshipit-source-id: 19eee4f8c4108ef168d82d5e9b1f2d65ff9b03bd

5 years agoRefactor FOLLY_GCC_DISABLE_WARNING to play nice with clang-format v2017.04.10.00
Christopher Dykes [Mon, 10 Apr 2017 01:02:53 +0000 (18:02 -0700)]
Refactor FOLLY_GCC_DISABLE_WARNING to play nice with clang-format

Summary: Currently clang-format cannot be run on any file using the macro because it breaks it. By putting the warning name in a string clang-format no longer breaks it.

Reviewed By: yfeldblum

Differential Revision: D4856945

fbshipit-source-id: ebd2d223dd97f66224e9e2912610057e36287dae

5 years agoAll template params for PriorityMPMCQueue
Yedidya Feldblum [Sat, 8 Apr 2017 14:08:01 +0000 (07:08 -0700)]
All template params for PriorityMPMCQueue

Summary:
[Folly] All template params for `PriorityMPMCQueue`.

Mimic the suite of template params of `MPMCQueue`.

Reviewed By: magedm

Differential Revision: D4857179

fbshipit-source-id: 3d441099c46ea1223155bd3f4cc819595814c47b

5 years agoFix include ordering for OpenSSLPtrTypes.h
Christopher Dykes [Sat, 8 Apr 2017 04:22:45 +0000 (21:22 -0700)]
Fix include ordering for OpenSSLPtrTypes.h

Summary: When including OpenSSL on Windows, the portability header must be included before including any OpenSSL headers.

Reviewed By: yfeldblum

Differential Revision: D4856376

fbshipit-source-id: a8e6533b2ba3c003314ae2426605031fa67be05f

5 years agoQualify a couple local constexpr values as 'static' to allow use from within non...
Christopher Dykes [Sat, 8 Apr 2017 04:15:44 +0000 (21:15 -0700)]
Qualify a couple local constexpr values as 'static' to allow use from within non-capturing lambdas

Summary: MSVC doesn't allow non-static locals to be accessed within a non-capturing lambda, and some were introduced into the json code.

Reviewed By: yfeldblum

Differential Revision: D4856242

fbshipit-source-id: b2db36ca37a79a41237d39cc7e7b839a5416922f

5 years agoFix leak in EliasFanoCodingTest
Giuseppe Ottaviano [Sat, 8 Apr 2017 02:50:22 +0000 (19:50 -0700)]
Fix leak in EliasFanoCodingTest

Summary: `EliasFanoCompressedList` requires explicit freeing.

Reviewed By: lbrandy, philippv, yfeldblum, luciang

Differential Revision: D4855200

fbshipit-source-id: ccee44b6371ae743ef15a72af9b691d5b56e37c2

5 years agoReadMostlySharedPtrTest, subtle test bug caught by -fsanitize-address-use-after-scope
Louis Brandy [Sat, 8 Apr 2017 02:38:33 +0000 (19:38 -0700)]
ReadMostlySharedPtrTest, subtle test bug caught by -fsanitize-address-use-after-scope

Summary: Because these `atomic<int>` are referenced in the destructor of the `ReadMostlyMainPtr`, they must be declared before it, otherwise they are destroyed before it i.

Reviewed By: yfeldblum

Differential Revision: D4854964

fbshipit-source-id: 625a8f890bba166b682ca13558f8bcde86edcc75

5 years agofix EvictingCacheMap test bug found w/ -fsanitize-address-use-after-scope
Louis Brandy [Sat, 8 Apr 2017 02:25:07 +0000 (19:25 -0700)]
fix EvictingCacheMap test bug found w/ -fsanitize-address-use-after-scope

Summary: This test actually stores references to `sum` in objects it puts into the `EvictingCacheMap`. Those references get accessed in the destructor of the objects, which is the destructor of the `EvictingCacheMap`. That means the `sum` variable must outlive (and be declared -before-) the `EvictingCacheMap` it's testing.

Reviewed By: yfeldblum, meyering

Differential Revision: D4854619

fbshipit-source-id: b0a3109278f48d10b4cd0c52a12cb2064f4a00c5

5 years agoFix leak in HMAC_CTX_free compat API
Anirudh Ramachandran [Fri, 7 Apr 2017 18:58:27 +0000 (11:58 -0700)]
Fix leak in HMAC_CTX_free compat API

Summary:
D4406876 replaced EVP_MD_CTX and HMAC_CTX on the stack with heap-allocated
versions. We omitted the _cleanup functions thinking those didn't actually free
anything, but turns out HMAC_CTX_cleanup internally calls EVP_MD_CTX_cleanup
which OPENSSL_frees a bunch of stuff.

Reviewed By: yfeldblum

Differential Revision: D4850388

fbshipit-source-id: 4ef1413a4105f3638140e4d12aeae64afe43f099

5 years agoFix a logic error in StringKeyedTest
Phil Willoughby [Fri, 7 Apr 2017 14:11:55 +0000 (07:11 -0700)]
Fix a logic error in StringKeyedTest

Summary: Make sure that the last use of `s6` is sequenced before it is moved-from.

Reviewed By: nbronson

Differential Revision: D4850480

fbshipit-source-id: 5272c14475ae2c62bc613cfd54f5df482b19d3dc

5 years agoGet rid of try/catch in messageAvailable, which is now noexcept
Alan Frindell [Thu, 6 Apr 2017 21:00:58 +0000 (14:00 -0700)]
Get rid of try/catch in messageAvailable, which is now noexcept

Summary: The problem with catching the exception here is stack information is lost.  Just let it std::terminate if it throws

Reviewed By: ikobzar

Differential Revision: D4831909

fbshipit-source-id: 42139bd7caee0fedff13328d52fa3be1c517e730

5 years agoRequire registrationComplete() not only in dbg builds
Andrii Grynenko [Thu, 6 Apr 2017 17:10:38 +0000 (10:10 -0700)]
Require registrationComplete() not only in dbg builds

Summary: Improve error message and change DFATAL to FATAL.

Reviewed By: yfeldblum, meyering

Differential Revision: D4837840

fbshipit-source-id: 54ae53c9356fc2e85266b037996aa7b2bae3d4e4

5 years agoUpdate StringKeyed... benchmarks to tell the whole story
Phil Willoughby [Thu, 6 Apr 2017 11:03:02 +0000 (04:03 -0700)]
Update StringKeyed... benchmarks to tell the whole story

Summary:
Benchmark each type with its native key type and with its non-native key type.
This allows you to make a more-informed decision about which to use based on
the types you have available at insertion/lookup time.

Reviewed By: yfeldblum

Differential Revision: D4826527

fbshipit-source-id: ee4ef5255d2bdd955e9deb30dc4c28be183cee07

5 years agoIndexedMemPool: Fix race condition on size_ that can cause the destructor to access...
Maged Michael [Thu, 6 Apr 2017 10:04:20 +0000 (03:04 -0700)]
IndexedMemPool: Fix race condition on size_ that can cause the destructor to access nonexistent slots

Summary:
Contention on allocation when there is exactly one slot remaining can cause size_ to exceed actualCapacity_.
Without taking the min of size_ and actualCapacity_ the destructor may try to destroy out-of-bound slots.

Added a test to test/IndexedMemPoolTest.cpp that failed before the fix and passes after the fix.

Reviewed By: nbronson

Differential Revision: D4837251

fbshipit-source-id: a887487727f17eaf2ba66345f40fc91d2fe3bc00

5 years agoMake NotificationQueue::Consumer::messageAvailable noexcept
Alan Frindell [Wed, 5 Apr 2017 23:37:16 +0000 (16:37 -0700)]
Make NotificationQueue::Consumer::messageAvailable noexcept

Summary: There's a comment in that code that states that it *it* noexcept, and that is common for most folly async io callbacks.

Reviewed By: yfeldblum

Differential Revision: D4831800

fbshipit-source-id: 78894ad72504b9dfe540c14b8d7bbb29247aaf87

5 years agoAdd trimAtMost functions
Subodh Iyengar [Wed, 5 Apr 2017 22:15:49 +0000 (15:15 -0700)]
Add trimAtMost functions

Summary:
Add trimAtMost start and trimAtMost end.
These are useful functions to have if you
don't care about trimming an exact amount.

Reviewed By: yfeldblum

Differential Revision: D4658087

fbshipit-source-id: f7dd62c0f4837503e59cdec75fdad433be20306c

5 years agoDisable EnvUtil::setAsCurrentEnvironment() on platforms without clearenv()
Christopher Dykes [Wed, 5 Apr 2017 19:50:28 +0000 (12:50 -0700)]
Disable EnvUtil::setAsCurrentEnvironment() on platforms without clearenv()

Summary:
Implementing `clearenv()` correctly on multiple OS's without relying on underlying implementation details is significantly more complicated, and this is preventing being able to build on OSX.
This function isn't actually used anywhere currently.

Reviewed By: yfeldblum

Differential Revision: D4832473

fbshipit-source-id: a80aabb5a223264746ab45e3138d065bce5fe99c

5 years agouse folly singleton relaxed init on android
Eli Lindsey [Tue, 4 Apr 2017 19:49:07 +0000 (12:49 -0700)]
use folly singleton relaxed init on android

Summary: fixing D4825412

Reviewed By: lbrandy

Differential Revision: D4826815

fbshipit-source-id: 94691ff293dad53b16e7a2f139c1df490a61e565

5 years agoMake some AsyncTest methods virtual to allow mocking them using gtest/gmock
Maxim Georgiev [Tue, 4 Apr 2017 19:16:07 +0000 (12:16 -0700)]
Make some AsyncTest methods virtual to allow mocking them using gtest/gmock

Summary: GMock can't mock class methods that are not virtual. This way classes relying on the non-virtual methods of AsyncSocket can't be efficiently tested using gtest/gmock.

Reviewed By: yfeldblum

Differential Revision: D4806838

fbshipit-source-id: fde852b75f7ac1d468c177e317c516cb0dc333b5

5 years agoFunction.h: remove FOLLY_MSVC_DISABLE_WARNING
Sven Over [Tue, 4 Apr 2017 16:53:30 +0000 (09:53 -0700)]
Function.h: remove FOLLY_MSVC_DISABLE_WARNING

Summary:
We recently removed declarations of Function constructors and
assignment operators that took non-const Function references. They
were necessary at the time to make sure that the templated
constructor that can take any invokable object did not get
selected for copying Functions from non-const lvalues. This,
however, is now provided by some std::enable_if construct.

Reviewed By: ericniebler

Differential Revision: D4818736

fbshipit-source-id: 08197721beef009f6ae482b576df78ec6999d132

5 years agoSupport building with OpenSSL 1.1.0 and BoringSSL
Anirudh Ramachandran [Tue, 4 Apr 2017 06:30:01 +0000 (23:30 -0700)]
Support building with OpenSSL 1.1.0 and BoringSSL

Summary:
More work to get wangle compiling. wangle/facebook/http pulls in
proxygen libs and that's another pain altogether, so this only makes the rest of
wangle build with 1.1.0 and BoringSSL

Depends on D4406876

Reviewed By: ngoyal

Differential Revision: D4767060

fbshipit-source-id: bd6bc6959d04028c84360e434f6bbdb2cde2faac

5 years agoOpenSSL 1.1.0 compatibility
Anirudh Ramachandran [Tue, 4 Apr 2017 06:29:59 +0000 (23:29 -0700)]
OpenSSL 1.1.0 compatibility

Summary: Add more layers to support compiling with OpenSSL 1.1.0

Reviewed By: yfeldblum

Differential Revision: D4406876

fbshipit-source-id: 8eff7e4f8e096605fc9e1093ef533b5afe0ee539

5 years agoDon't allow singleton use w/o registrationComplete()
Andrii Grynenko [Tue, 4 Apr 2017 03:04:44 +0000 (20:04 -0700)]
Don't allow singleton use w/o registrationComplete()

Summary: This is same as D4249032, but for now this is only enabled in dbg and dev builds. I'm planning to bump this to FATAL later, once we fix know problematic main()s.

Reviewed By: lbrandy, yfeldblum

Differential Revision: D4821763

fbshipit-source-id: 5f7930f8cbcb10275d23a89848f1ec8ee34a8020

5 years agoRemove use of generic lambdas
Nick Terrell [Mon, 3 Apr 2017 23:51:30 +0000 (16:51 -0700)]
Remove use of generic lambdas

Summary:
The use of generic lambdas caused compiler errors with gcc 4.8.2.
Fixes https://github.com/facebook/folly/issues/569.

Reviewed By: Orvid, yfeldblum

Differential Revision: D4821115

fbshipit-source-id: 8372ee7695a3d0a1df0d033623618a923c261737

5 years agoFix broken ManualExecutor
Harrison Klaperman [Mon, 3 Apr 2017 23:31:57 +0000 (16:31 -0700)]
Fix broken ManualExecutor

Summary:
The priority queue in the manual executor implementation is backwards. This means that scheduled things run in reverse order, and a later thing will block an earlier thing if you advance to a timestamp in between the two.

This diff fixes the problem and adds tests to confirm the fix. These tests fail on the old implementation.

Reviewed By: yfeldblum

Differential Revision: D4739101

fbshipit-source-id: 6e429828460df5b3c656580568a9ae1eb4009527

5 years agoDon't use macros for FBVector::insert
Christopher Dykes [Mon, 3 Apr 2017 21:51:22 +0000 (14:51 -0700)]
Don't use macros for FBVector::insert

Summary: Macros are not the best solution anymore.

Reviewed By: yfeldblum

Differential Revision: D4270626

fbshipit-source-id: 8acbd5b778d4effcfb587655b4397737bef0d379

5 years agoRemove/make private the default ***Holder constructor to allow compile time detection...
Dan Melnic [Mon, 3 Apr 2017 21:09:47 +0000 (14:09 -0700)]
Remove/make private the default ***Holder constructor to allow compile time detection of ***Holder(mutexPtr_) constructs

Summary: Avoid unintended C++ stuff - this is not the same as the "fleeting rvalue" - this change might break the build but I will fix it

Reviewed By: yfeldblum

Differential Revision: D4814104

fbshipit-source-id: 058a0eac44893c573062fcf5665d4fd022ee64a0

5 years agoatomic_shared_ptr
Dave Watson [Mon, 3 Apr 2017 17:51:19 +0000 (10:51 -0700)]
atomic_shared_ptr

Summary:
A (almost) lock-free atomic_shared_ptr, matching the proposed concurrency TS interface.

See notes at top of file

Reviewed By: davidtgoldblatt

Differential Revision: D4716098

fbshipit-source-id: b9ca2443ba9e227ebb6f40807128073c6e14222a

5 years agoOptimize JSON escaping of ASCII strings v2017.04.03.00
Giuseppe Ottaviano [Sat, 1 Apr 2017 05:17:57 +0000 (22:17 -0700)]
Optimize JSON escaping of ASCII strings

Summary:
`escapeString` is very slow even when there's very little to nothing to escape. This diff adds a fast path to copy sequences of bytes that don't need escaping.
It also optimizes appending escape sequences: `string::push_back` is slow because it has to do a capacity check for every character.

Before:
```
  ============================================================================
  folly/test/JsonOtherTest.cpp                    relative  time/iter  iters/s
  ============================================================================
  jsonSerialize                                              818.55ns    1.22M
  jsonSerializeWithNonAsciiEncoding                            1.35us  738.06K
  jsonSerializeWithUtf8Validation                              1.42us  705.60K
  jsonSerializeAsciiWithUtf8Validation                         3.27us  306.06K
  parseSmallStringWithUtf                                      1.91us  522.38K
  parseNormalString                                            1.51us  660.27K
  parseBigString                                             384.44ns    2.60M
  toJson                                                     480.54ns    2.08M
  ============================================================================
```

After:
```
  ============================================================================
  folly/test/JsonOtherTest.cpp                    relative  time/iter  iters/s
  ============================================================================
  jsonSerialize                                              781.69ns    1.28M
  jsonSerializeWithNonAsciiEncoding                          847.68ns    1.18M
  jsonSerializeWithUtf8Validation                            928.68ns    1.08M
  jsonSerializeAsciiWithUtf8Validation                       199.85ns    5.00M
  parseSmallStringWithUtf                                      1.93us  518.39K
  parseNormalString                                            1.45us  689.11K
  parseBigString                                             378.66ns    2.64M
  toJson                                                     446.38ns    2.24M
  ============================================================================
```

All string escaping benchmarks are slightly faster, and ASCII-only with no escapes is 8x faster.

Reviewed By: luciang, evilmucedin, yfeldblum

Differential Revision: D4793233

fbshipit-source-id: c40d07708bd787799c8c00f9f23a417b862ee9ae

5 years agofix memory leak in case of large number of retries
Mainak Mandal [Sat, 1 Apr 2017 02:01:12 +0000 (19:01 -0700)]
fix memory leak in case of large number of retries

Summary: Infinite retries is something that is often needed for read-modify-write like workflows. The current implementation was creating a nested chain of implicit promises. This manifests as a memory leak after some time. Worse yet, even if it succeeds, it will take a long time to churn thru the chain of promises.

Reviewed By: yfeldblum

Differential Revision: D4770335

fbshipit-source-id: 44b8d6df1084de4514b66919a9838cf2322d6dce

5 years agoavoid UB in StringKeyedCommon for default constructed StringPiece
Wez Furlong [Fri, 31 Mar 2017 06:38:50 +0000 (23:38 -0700)]
avoid UB in StringKeyedCommon for default constructed StringPiece

Summary:
ubsan flags this problem:

```
buck-out/opt-ubsan/gen/folly/__default_headers__#header-mode-symlink-tree-with-header-map,headers/folly/experimental/StringKeyedCommon.h:31:18: runtime error: null pointer passed as argument 2, which is declared to never be null
third-party-buck/gcc-4.9-glibc-2.20-fb/build/glibc/include/string.h:47:28: note: nonnull attribute specified here
    #0 0x5cb88f in std::pair<std::__detail::_Node_iterator<std::pair<folly::Range<char const*> const, facebook::eden::GitIgnore>, false, true>, bool> folly::StringKeyedUnorderedMap<facebook::eden::GitIgnore, folly::Hash, std::equal_to<folly::Range<char const*> >, std::allocator<std::pair<folly::Range<char const*> const, facebook::eden::GitIgnore> > >::emplace<facebook::eden::GitIgnore>(folly::Range<char const*>, facebook::eden::GitIgnore&&) buck-out/opt-ubsan/gen/folly/__default_headers__#header-mode-symlink-tree-with-header-map,headers/folly/experimental/StringKeyedCommon.h:31
    #1 0x5c6652 in facebook::eden::(anonymous namespace)::IgnoreChecker::isIgnored(facebook::eden::detail::RelativePathBase<folly::Range<char const*> >) eden/fs/inodes/Dirstate.cpp:226
    #2 0x5c6037 in facebook::eden::(anonymous namespace)::IgnoreChecker::isIgnored(facebook::eden::detail::RelativePathBase<folly::Range<char const*> >) eden/fs/inodes/Dirstate.cpp:180
    #3 0x5c5173 in facebook::eden::Dirstate::getStatusForExistingDirectory(facebook::eden::detail::RelativePathBase<folly::Range<char const*> >) const eden/fs/inodes/Dirstate.cpp:372
    #4 0x5c4509 in facebook::eden::Dirstate::getStatus() const eden/fs/inodes/Dirstate.cpp:272
    #5 0x50a448 in verifyExpectedDirstate(facebook::eden::Dirstate const*, std::unordered_map<std::basic_fbstring<char, std::char_traits<char>, std::allocator<char>, std::fbstring_core<char> >, facebook::eden::StatusCode, std::hash<std::basic_fbstring<char, std::char_traits<char>, std::allocator<char>, std::fbstring_core<char> > >, std::equal_to<std::basic_fbstring<char, std::char_traits<char>, std::allocator<char>, std::fbstring_core<char> > >, std::allocator<std::pair<std::basic_fbstring<char, std::char_traits<char>, std::allocator<char>, std::fbstring_core<char> > const, facebook::eden::StatusCode> > >&&) eden/fs/inodes/test/DirstateTest.cpp:48
    #6 0x50c70c in Dirstate_addDirectoriesWithMixOfFiles_Test::TestBody() eden/fs/inodes/test/DirstateTest.cpp:220
    #7 0xb45857 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/engshare/third-party2/gtest/1.7.0/src/gtest-1.7.0/./src/gtest.cc:2364
    #8 0xb36784 in testing::Test::Run() /home/engshare/third-party2/gtest/1.7.0/src/gtest-1.7.0/./src/gtest.cc:2437
    #9 0xb36957 in testing::TestInfo::Run() [clone .part.558] /home/engshare/third-party2/gtest/1.7.0/src/gtest-1.7.0/./src/gtest.cc:2612
    #10 0xb36b74 in testing::TestCase::Run() [clone .part.559] /home/engshare/third-party2/gtest/1.7.0/src/gtest-1.7.0/./src/gtest.cc:2587
    #11 0xb3806e in testing::internal::UnitTestImpl::RunAllTests() [clone .part.561] /home/engshare/third-party2/gtest/1.7.0/src/gtest-1.7.0/./src/gtest.cc:4571
    #12 0xb382d9 in testing::UnitTest::Run() /home/engshare/third-party2/gtest/1.7.0/src/gtest-1.7.0/./src/gtest.cc:4519
    #13 0x861ca7 in main third-party-buck/gcc-4.9-glibc-2.20-fb/build/gtest/include/gtest/gtest.h:2326
    #14 0x7f20be0740f5 in __libc_start_main /home/engshare/third-party2/glibc/2.20/src/glibc-2.20/csu/libc-start.c:289
    #15 0x4a552c in _start /home/engshare/third-party2/glibc/2.20/src/glibc-2.20/csu/../sysdeps/x86_64/start.S:122

UndefinedBehaviorSanitizer: undefined-behavior buck-out/opt-ubsan/gen/folly/__default_headers__#header-mode-symlink-tree-with-header-map,headers/folly/experimental/StringKeyedCommon.h:31:18
```

The issue is that `StringPiece` default constructs to `{nullptr, nullptr}` as a valid representation of an empty string.
This is tripping up UBSAN in this case.

The fix is a trivial nullptr check

Reviewed By: igorsugak

Differential Revision: D4791015

fbshipit-source-id: dec7484b29ecb29c17b8dd6a9b0e8093f07d63cb

5 years agoSome constexpr symbols for whether sanitizers are enabled
Yedidya Feldblum [Fri, 31 Mar 2017 04:38:09 +0000 (21:38 -0700)]
Some constexpr symbols for whether sanitizers are enabled

Summary:
[Folly] Some constexpr symbols for whether sanitizers are enabled.

Useful where we need to do or not do some particular thing when sanitizers are enabled or disabled. Such cases do happen. It is better to use a C++ `constexpr` symbol rather than a preprocessor symbol in such cases, where possible.

Reviewed By: meyering

Differential Revision: D4805110

fbshipit-source-id: ca9ce074edef73dce3a04693dda102698c9cd29f

5 years agoFix double-delete in Singleton::make_mock
Yedidya Feldblum [Thu, 30 Mar 2017 22:43:00 +0000 (15:43 -0700)]
Fix double-delete in Singleton::make_mock

Summary:
[Folly] Fix double-`delete` in `Singleton::make_mock`.

In that function, we unconditionally destroy any existing singleton instance of the same singleton. Actually, we must conditionally destroy it - conditionally on one having been created and not yet destroyed.

This problem only rarely appears because it is masked by `delete static_cast<T*>(nullptr)` being a no-op. For example, if we call `make_mock` before actually instantiating the singleton, we call that no-op. The way to make it appear is to instantiate the singleton, and then to call `make_mock` twice in a row. The first call to `make_mock` unconditionally destroys the existing instance (it still should have checked) and the second call does it again, but because the existing instance is not `nullptr`, the second call is a double-delete of a non-`nullptr` instance and crashes. In the simple case, as reproduced in an attached test, the failure is observable with ASAN. In other cases, the double-`free` is tolerated, but the failure may only be observable depending on the singleton object's state - if running the dtor twice fails.

Reviewed By: ivmaykov

Differential Revision: D4798163

fbshipit-source-id: e7b65d030d89225dfdc2fad4c778d3653460806e

5 years agoRemove unneccessary test for &that==this in folly::Function's move assignment operator.
Eric Niebler [Thu, 30 Mar 2017 21:09:34 +0000 (14:09 -0700)]
Remove unneccessary test for &that==this in folly::Function's move assignment operator.

Summary: Self-moves are exceedingly rare and need not preserve the state of the object. They must only leave the object in a valid but unspecified state. By removing the branch, we make the common case (non-self move) faster.

Reviewed By: spacedentist, ot

Differential Revision: D4803486

fbshipit-source-id: 3ef2e1e13cd08d9221ecb154bfb3338b16487717

5 years agoProperly constrain folly::Function's generic conversion constructor and fix its noexc...
Eric Niebler [Thu, 30 Mar 2017 21:09:32 +0000 (14:09 -0700)]
Properly constrain folly::Function's generic conversion constructor and fix its noexcept specification

Summary:
The generic conversion constructor for `folly::Function` was not checking that the source object could successfully be copy/move constructed, leading to some `is_constructible` false positives.

Also, the `noexcept` specification on the `Function` constructor wasn't taking into account that the source object might be copied into the Function, instead of moved. The copy could throw.

Reviewed By: yfeldblum

Differential Revision: D4775037

fbshipit-source-id: f337b41bf9ac431baa9457a501e63c18ca099e57

5 years agoPath variants for get_(ref_)default()
Nick Terrell [Thu, 30 Mar 2017 19:48:11 +0000 (12:48 -0700)]
Path variants for get_(ref_)default()

Summary: Useful for the same reason as the `get_ptr()` path function, but when you want to use a default value.

Reviewed By: luciang, yfeldblum

Differential Revision: D4785728

fbshipit-source-id: 70fd56f9ffa7a9edd6740f6dd712d3a251bf9fb0

5 years agoBacked out changeset d4e3fdfd5775
Igor Sugak [Thu, 30 Mar 2017 01:14:16 +0000 (18:14 -0700)]
Backed out changeset d4e3fdfd5775

Summary:
https://our.intern.facebook.com/intern/sevmanager/view/s/145809/?hc_location=ufi

```
hg backout d4e3fdfd57750c8dcabdb4c01ab3b528c4818624
```

Reviewed By: lbrandy

Differential Revision: D4799568

fbshipit-source-id: 76366040702e68becc7e245d7ce15832f933e10e

5 years agouncurry
Tom Jackson [Wed, 29 Mar 2017 22:17:25 +0000 (15:17 -0700)]
uncurry

Summary: Extending ApplyTuple to support this common functional construct, with good forwarding semantics.

Reviewed By: yfeldblum

Differential Revision: D4787560

fbshipit-source-id: 2c740e448e0cb916abe948b79709d5ecd8ba54bb

5 years agoCut extraneous typename in StringKeyedCommon.h
Yedidya Feldblum [Wed, 29 Mar 2017 17:17:46 +0000 (10:17 -0700)]
Cut extraneous typename in StringKeyedCommon.h

Summary:
[Folly] Cut extraneous typename in `StringKeyedCommon.h`.

In this case, `typename` is optional, and, to avoid confusion, should be removed.

"In this case" means in a template definition, with a qualified non-dependent name.

Reviewed By: ericniebler

Differential Revision: D4792087

fbshipit-source-id: dbb9d2c747432a1f9bc8823e72a7668e78a2fbaa

5 years agofix dead-lock in Future when executor discards function
Sven Over [Wed, 29 Mar 2017 10:48:52 +0000 (03:48 -0700)]
fix dead-lock in Future when executor discards function

Summary:
This diff adds two tests to futures/test/ViaTest.cpp:
viaDummyExecutorFutureSetValueFirst and
viaDummyExecutorFutureSetCallbackFirst. The latter resulted in a
dead-lock before the fix contained in this diff.

It is important that the callback function is destroyed after
it is executed, since it may capture objects (like a Promise)
that should be destroyed (so that e.g. a corresponding Future
throws BrokenPromise). When the callback is executed via an
executor, it is possible that the executor doesn't get around to
executing the task. We shouldn't rely on the task being executed
to do necessary clean-up. That clean-up should (also) happen
when the task (with its captured data) is destroyed (in the
spirit of RIAA).

Reviewed By: djwatson

Differential Revision: D4779215

fbshipit-source-id: d029cf8b8f7b55e1b03357749c5fb62d95986ca7

5 years agoAutomaticCodec
Nick Terrell [Wed, 29 Mar 2017 04:21:02 +0000 (21:21 -0700)]
AutomaticCodec

Summary:
Add codec that does automatic format detection for decompression.
The codec returned by `getCodec(CodecType::AUTOMATIC)` can decompress codecs `LZ4_FRAME`, `ZSTD`, `ZLIB`, `GZIP`, and `LZMA2`.
Additionally, the function `getAutomaticCodec()` allows users to pass their own custom codecs, but the 5 default codecs must also be supported.
We force the default codecs on users because having a common subset of formats is in general useful, and because we want to be able to add codecs to this set.

When compiled in debug mode, on construction it checks that the codecs are compatible.
This helps make sure that users don't accidentally add ambiguous codecs, and if we add a new default codec in the future, tests will fail on ambiguity.

Reviewed By: yfeldblum

Differential Revision: D4760065

fbshipit-source-id: 9e65844aba9f1ae3d5c7b86643931782c06ab3eb

5 years agoCursor::read<T>() initializes return value
Ondrej Lehecka [Wed, 29 Mar 2017 01:58:12 +0000 (18:58 -0700)]
Cursor::read<T>() initializes return value

Summary:
If the compiler doesn't see thru the call to tryRead which always initializes the val variable, it will emit the following error.
I was getting error when compiling my android project:
error: 'val' may be used uninitialized in this function [-Werror=maybe-uninitialized]
This should not have a negative perf impact as smart compilers which inline call to tryRead can elliminate the initialization.

Reviewed By: djwatson

Differential Revision: D4790864

fbshipit-source-id: f353cfe54ca4d056b6ddfc075d00580c9f2d391e

5 years agoOptimize ZlibCodec::doUncompress()
Nick Terrell [Tue, 28 Mar 2017 23:30:48 +0000 (16:30 -0700)]
Optimize ZlibCodec::doUncompress()

Summary:
`ZlibCodec::doUncompress()` always allocates at least 4 MiB when the uncompressed size is unkown.
We can get a 10-20% speed boost for small inputs (as well as significant memory savings) if we dynamically choose the buffer size.

Benchmarks before the change:
  ============================================================================
  experimental/terrelln/CompressionBenchmark.cpp  relative  time/iter  iters/s
  ============================================================================
  uncompress(ZLIB_100)                                         2.96us  338.19K
  uncompress(ZLIB_1000)                                        7.22us  138.44K
  uncompress(ZLIB_10000)                                      36.86us   27.13K
  uncompress(ZLIB_100000)                                    299.70us    3.34K
  uncompress(ZLIB_1000000)                                     2.74ms   365.32
  uncompress(ZLIB_10000000)                                   26.91ms    37.16
  uncompress(ZLIB_100000000)                                 273.92ms     3.65
  ============================================================================
After:
  ============================================================================
  experimental/terrelln/CompressionBenchmark.cpp  relative  time/iter  iters/s
  ============================================================================
  uncompress(ZLIB_100)                                         2.34us  426.68K
  uncompress(ZLIB_1000)                                        6.24us  160.38K
  uncompress(ZLIB_10000)                                      35.52us   28.16K
  uncompress(ZLIB_100000)                                    296.30us    3.37K
  uncompress(ZLIB_1000000)                                     2.74ms   365.25
  uncompress(ZLIB_10000000)                                   27.16ms    36.82
  uncompress(ZLIB_100000000)                                 273.72ms     3.65
  ============================================================================

Reviewed By: yfeldblum

Differential Revision: D4783871

fbshipit-source-id: 0fcce191ea520fd2117551db679c86f5ff734893

5 years agoImplement S_ISDIR()
Christopher Dykes [Tue, 28 Mar 2017 18:21:24 +0000 (11:21 -0700)]
Implement S_ISDIR()

Summary: It's needed downstream in HHVM.

Reviewed By: yfeldblum

Differential Revision: D4785204

fbshipit-source-id: fe81ed8e23497fffeb44b2a366c457b788bcaada

5 years agoChange trylock() to try_lock() in folly::SpinLock to conform to standard Lockable.
Maged Michael [Tue, 28 Mar 2017 15:36:31 +0000 (08:36 -0700)]
Change trylock() to try_lock() in folly::SpinLock to conform to standard Lockable.

Reviewed By: yfeldblum

Differential Revision: D4782707

fbshipit-source-id: 535b42b4f2558cadc78037268d6de81a8bb49840

5 years agoAllow custom sorting function for JSON serializer
Dylan Yudaken [Tue, 28 Mar 2017 15:02:42 +0000 (08:02 -0700)]
Allow custom sorting function for JSON serializer

Summary: In some situations it is useful to be able to sort the JSON keys according to some different scheme for readability, so allow the sorting function to be changed by the serializer_opts

Reviewed By: yfeldblum

Differential Revision: D4782077

fbshipit-source-id: 032fa60a38804452bd1c22c67ba897521cb2cd1d

5 years agoGate LZ4Frame behind version check
Nick Terrell [Mon, 27 Mar 2017 23:14:38 +0000 (16:14 -0700)]
Gate LZ4Frame behind version check

Summary:
D4715918 broke open source builds on Ubuntu 14.04, since it has lz4-r114, but the lz4 frame API was introduced in [r123](https://github.com/lz4/lz4/blob/r123/lz4frame.h).
Put the `LZ4FrameCodec` behind a lz4 version check.
Fixes https://github.com/facebook/fbthrift/issues/209.

Reviewed By: yfeldblum

Differential Revision: D4780830

fbshipit-source-id: 19492a7e6bdd128e610c36b5778274e19eff9548

5 years agoDisallow temporary default values in get_ref_default()
Nick Terrell [Mon, 27 Mar 2017 20:03:02 +0000 (13:03 -0700)]
Disallow temporary default values in get_ref_default()

Summary: Passing a temporary default value to `get_ref_default()` returns a dangling reference when it is used.

Reviewed By: lbrandy, yfeldblum

Differential Revision: D4768769

fbshipit-source-id: 4c58a17dc7662ad553cf88a8544dae20016d2f6f

5 years agoSeveral fixes to the "SWMRList" example in experimental/hazptr. v2017.03.27.00
Arthur O'Dwyer [Sat, 25 Mar 2017 20:43:31 +0000 (13:43 -0700)]
Several fixes to the "SWMRList" example in experimental/hazptr.

Summary:
Fix a correctness bug in "SWMRList.h".

Thanks to Maged for the tip! The old code omitted setting a removed node's
"next" pointer to `nullptr`, which meant that if the writer removed node
A and then node B = A->next while the reader was looking at B, then the reader
might happily keep chasing pointers B->next, B->next->next,... without ever
noticing that it was now on a "dead branch" of the linked list.

(And then there's a bit of a trick: you really do have to remove the node
first and *then* set its "next" pointer to null, because if you do the null
assignment first, you'll truncate the list, which means that some readers
will see a truncated list and return the wrong answer. I've added comments
to try to explain this to future-me.)

Style nit: Avoid doing multiple atomic operations on the same line of code.

----

Modernize the parameter-passing conventions in SWMRList.h.

Pass `T`s by const reference, except in `add` where we're likely to want to
make a copy of the `T` anyway. In that case it would be more "STL-correct"
to supply two different overloads `add(T&&)` and `add(const T&)`, but this
is just an example so it makes sense to keep things simple.

----

Fix an undefined behavior in SWMRList example.

Searching an empty SWMRList<int> always invokes undefined behavior by
comparing an uninitialized `T elem` against `val` on the final line of
the `contains` function. Besides, this patch allows SWMRList to work
with `T`s that aren't default-constructible or even copy-constructible.

----

Closes https://github.com/facebook/folly/pull/566

Reviewed By: djwatson

Differential Revision: D4772359

Pulled By: yfeldblum

fbshipit-source-id: 8f96573530800675cb56006aa91e7a5c5c1fb85d

5 years ago(folly) Add lock holder interface to flat combining. Change an assert to a static_assert.
Maged Michael [Sat, 25 Mar 2017 10:28:15 +0000 (03:28 -0700)]
(folly) Add lock holder interface to flat combining. Change an assert to a static_assert.

Summary:
Two changes:
- Add a lock holder interface to enable users to move exclusive access and implicitly release the held lock.
- Change an assert to a static_assert to check at compile time if a provided function fits in the Function structure without dynamic allocation.

Reviewed By: djwatson

Differential Revision: D4770229

fbshipit-source-id: 89408164c08d7660231a6ca4e37637dd688356cd

5 years agoAdd tryRead() and endian variants
Nick Terrell [Sat, 25 Mar 2017 01:12:06 +0000 (18:12 -0700)]
Add tryRead() and endian variants

Summary:
Add a `tryRead()`, and endian variants, which try to read into an arithmetic type, and if there isn't enough data they return false.
One use case is to quickly check if an IOBuf starts with a certain prefix, benchmarks show that using `tryReadLE()` is 6x faster than using `pullAtMost()` and `memcmp()`.

Reviewed By: yfeldblum

Differential Revision: D4767855

fbshipit-source-id: feb8c61092772933d4b8496b27d464559ff8b827

5 years agoAvoid passing temporary to get_ref_default()
Nick Terrell [Fri, 24 Mar 2017 22:00:07 +0000 (15:00 -0700)]
Avoid passing temporary to get_ref_default()

Summary: `get_ref_default()` won't accept temporary default values after D4768769.

Reviewed By: yfeldblum

Differential Revision: D4769043

fbshipit-source-id: 46b039e13b80721922f69036c6cc02f5642a26f9

5 years agoAllow decompresion without uncompressedSize
Nick Terrell [Fri, 24 Mar 2017 21:30:45 +0000 (14:30 -0700)]
Allow decompresion without uncompressedSize

Summary:
The `LZMA2` codec doesn't allow decompression without passing the uncompressed size.
Every other codec that supports streaming decompression without the uncompressed size by streaming into an IOBuf chain.
Benchmarks show that reducing the `defaultBufferLength` to 256 KiB doesn't slow down decompression of large files, but it speeds up decompression of small files (< 1000 bytes).

Reviewed By: yfeldblum

Differential Revision: D4751571

fbshipit-source-id: 39dbe6754a1ecdc2b7ba3107e9face926d4c98ca

5 years agoOptimize ZSTDCodec::doUncompress()
Nick Terrell [Fri, 24 Mar 2017 19:18:57 +0000 (12:18 -0700)]
Optimize ZSTDCodec::doUncompress()

Summary:
It is ~10% faster to call `ZSTD_decompress()` than use the
streaming API. The streaming API does some extra `memcpy`s that we
can avoid. We are working on improving the speed of the streaming
API in the case where all the data can be processed in one shot,
but that won't be available in the stable ZSTD api for a few versions.

Reviewed By: yfeldblum

Differential Revision: D4731058

fbshipit-source-id: 39026c499c0f5002466097b5afe7e30f850e0ae8

5 years agoAdd LZ4_FRAME codec
Nick Terrell [Fri, 24 Mar 2017 18:08:17 +0000 (11:08 -0700)]
Add LZ4_FRAME codec

Summary:
The LZ4 Frame codec encodes data using the LZ4 frame format.
One advantage of the LZ4 frame format is that it has 4 magic bytes in the header, so users can transparently determine compression type.
It also allows the user to interop with the lz4 command line tool.

Reviewed By: yfeldblum

Differential Revision: D4715918

fbshipit-source-id: 689833fef526b1cfe98685179e7b494380d49cba

5 years agoExpand environment-handling in folly
Phil Willoughby [Fri, 24 Mar 2017 09:05:42 +0000 (02:05 -0700)]
Expand environment-handling in folly

Summary:
Extract the environment-as-STL-map logic out of `test::EnvVarSaver` and into a
separate class `experimental::EnvironmentState` so that other code which needs
to manipulate the environment can do so more easily.

Add routines to set the process environment from the state of an
`EnvironmentState`'s map, and to extract the environment in the forms required
by `Subprocess` and the UNIX routines `execve` etc.

Reviewed By: yfeldblum

Differential Revision: D4713307

fbshipit-source-id: 6b1380dd29b9ba41c97b886814dd3eee91fc1c0f

5 years agoRe-enable io tests
Ted Percival [Thu, 23 Mar 2017 18:30:38 +0000 (11:30 -0700)]
Re-enable io tests

Summary:
Looks like some tests for the `io` directory were left behind when they were moved out of experimental. Re-enabled them in the new location, except for the compression test because it takes a lot longer. It's still built and can be executed manually.

Also added some preprocessor guards around compression codecs that might not be compiled in (and therefore fail during test SetUp).

Depends on #547 because the path to gtest appears in the new Makefile.am's `CPPFLAGS`.
Closes https://github.com/facebook/folly/pull/550

Reviewed By: yfeldblum

Differential Revision: D4631244

Pulled By: Orvid

fbshipit-source-id: 496b2c272e4f7293822f159776a02f7dd0c9e04d

5 years agoMake sure singleton destructor is always called from main thread.
Andrii Grynenko [Thu, 23 Mar 2017 18:17:04 +0000 (11:17 -0700)]
Make sure singleton destructor is always called from main thread.

Reviewed By: yfeldblum, ot

Differential Revision: D4742262

fbshipit-source-id: 7137bb8e1cb39c74e8ba726e76e9a8a6f3e1984c

5 years agoRandomize the Unix socket name in AsyncSocketTest.SendMessageAncillaryData test to...
Maxim Georgiev [Thu, 23 Mar 2017 17:31:34 +0000 (10:31 -0700)]
Randomize the Unix socket name in AsyncSocketTest.SendMessageAncillaryData test to avoid collisions.

Summary:
Our test framework reports frequent failures of AsyncSocketTest.SendMessageAncillaryData. According to the logs the socket fails to bind:

    folly/io/async/test/AsyncSocketTest2.cpp:3098: Failure
    Expected: (bind(lfd, (struct sockaddr*)&addr, sizeof(addr))) != (-1), actual: -1 vs -1
    Bind failed: 98

This diff adds the socket name randomization to avoid name collisions between tests running concurrently on the same test box.

Reviewed By: yfeldblum

Differential Revision: D4758942

fbshipit-source-id: 6066dbc18222a4521c40b2ff218cb7dab8bd789d

5 years agochange EXPECT_EQ(false, ... to EXPECT_FALSE(...; ditto for ASSERT_EQ
Jim Meyering [Thu, 23 Mar 2017 17:18:49 +0000 (10:18 -0700)]
change EXPECT_EQ(false, ... to EXPECT_FALSE(...; ditto for ASSERT_EQ

Summary:
Change every instance of EXPECT_EQ(false, ... to the simpler/shorter EXPECT_FALSE(...
Do the same for ASSERT_EQ, for consistency, and for true/TRUE.

Reviewed By: Orvid

Differential Revision: D4751397

fbshipit-source-id: 045293dbf8be1167f0cb1b85a2f9cd59a6a30f73