5 years agofall back to .debug_info scan in fatal signal handler v2016.10.17.00
Philip Pronin [Sat, 15 Oct 2016 04:44:00 +0000 (21:44 -0700)]
fall back to .debug_info scan in fatal signal handler

We've found clang might be generating incomplete `.debug_aranges`,
while falling back to linear `.debug_info` scan is too expensive and shouldn't
be used by default, we can afford doing that in fatal signal handler.

Also optimize `exception_tracer::printExceptionInfo()` to avoid `LocationInfo`
resolution if `NO_FILE_AND_LINE` is used.

Reviewed By: luciang, ot

Differential Revision: D4020989

fbshipit-source-id: 84172208736b224c19206da48bcb3b5c0b2c67d0

5 years agoBacked out changeset 09c1712854b3
Nicholas Ormrod [Fri, 14 Oct 2016 18:32:14 +0000 (11:32 -0700)]
Backed out changeset 09c1712854b3

Summary: Removing COW from fbstring had adverse memory consequences when sync'd with libgcc. Revert this diff to keep folly and libgcc in sync.

Reviewed By: yfeldblum, luciang

Differential Revision: D4019604

fbshipit-source-id: 80bd31c220098bfab37f0effc90f67876432369d

5 years agoBatchDispatcher exception handling
Shubhanshu Agrawal [Fri, 14 Oct 2016 13:43:42 +0000 (06:43 -0700)]
BatchDispatcher exception handling

Summary: error handling for batch dispatcher, for the case where dispatch function throws.

Reviewed By: yfeldblum

Differential Revision: D4016246

fbshipit-source-id: ac61b873a425128be7839a119591fca10608210a

5 years agonoexcept CollectVariadicContext destructor
Matthieu Martin [Fri, 14 Oct 2016 02:43:20 +0000 (19:43 -0700)]
noexcept CollectVariadicContext destructor

Summary: To be explicit about our assumption from D4013691

Reviewed By: andriigrynenko

Differential Revision: D4015816

fbshipit-source-id: 24f984ac9959d0c4c0aeac34b3dce8dfcf2c1354

5 years agoAdd missing include
Csaba Kertesz [Thu, 13 Oct 2016 21:39:07 +0000 (14:39 -0700)]
Add missing include

String.h references std::vector, but it is not included.

Reviewed By: yfeldblum, meyering

Differential Revision: D4009410

Pulled By: Orvid

fbshipit-source-id: f21ce0d5833f0959a83e75e2fbcbe2beedbb5b6f

5 years agoAvoid setting OPENSSL_IS_{101,102,110} when using BoringSSL.
Zonr Chang [Thu, 13 Oct 2016 21:07:57 +0000 (14:07 -0700)]
Avoid setting OPENSSL_IS_{101,102,110} when using BoringSSL.

Summary: Closes

Reviewed By: anirudhvr

Differential Revision: D4016114

Pulled By: Orvid

fbshipit-source-id: ce9da1e20390a2c834e376a7a06f6002103ff1e4

5 years agofix .debug_aranges parsing
Philip Pronin [Thu, 13 Oct 2016 17:09:36 +0000 (10:09 -0700)]
fix .debug_aranges parsing

Each sequence of tuples within an entires set is terminated with (0,
0) according to the DWARF standard (see 7.20 in

Reviewed By: ot, luciang

Differential Revision: D4014952

fbshipit-source-id: c4b034c94adc4a0d9b052456c02919a54a79eaf1

5 years agoProvide a unpackTryTuple function in folly
Matthieu Martin [Thu, 13 Oct 2016 07:09:32 +0000 (00:09 -0700)]
Provide a unpackTryTuple function in folly

Summary: This feature already exists in detail, move it to public instead.

Reviewed By: andriigrynenko

Differential Revision: D4013691

fbshipit-source-id: 1779cc53d114ddc97993b41e0ad63c104008f6b0

5 years agoAdd update-mutex into Observable
Andrii Grynenko [Thu, 13 Oct 2016 01:30:12 +0000 (18:30 -0700)]
Add update-mutex into Observable

Reviewed By: yfeldblum

Differential Revision: D4013256

fbshipit-source-id: 771f9becfa9e7676439209dfbf4a746c110d629d

5 years agoAccess SSL cipher info with a const reference.
Zonr Chang [Wed, 12 Oct 2016 20:16:27 +0000 (13:16 -0700)]
Access SSL cipher info with a const reference.

BoringSSL returns `const SSL_CIPHER*` from sk_SSL_CIPHER_value().

Reviewed By: yfeldblum

Differential Revision: D4009712

Pulled By: Orvid

fbshipit-source-id: cca2e8cb6aab0a459dca55b8f23ef57fa5c8251f

5 years agoFix call to SSL_SESSION_up_ref().
Zonr Chang [Wed, 12 Oct 2016 20:14:31 +0000 (13:14 -0700)]
Fix call to SSL_SESSION_up_ref().

`SSL_SESSION_up_ref()` in both BoringSSL and OpenSSL 1.1.0 takes a

Reviewed By: yfeldblum

Differential Revision: D4009706

Pulled By: Orvid

fbshipit-source-id: 68ea201821e7a6d5835a79b34187344ba251e239

5 years agoZSTDCodec should properly handle non-shortcut levels
Philip Pronin [Wed, 12 Oct 2016 17:57:13 +0000 (10:57 -0700)]
ZSTDCodec should properly handle non-shortcut levels

Currently only shortcut levels are properly handled, others result in
using level 1.

Reviewed By: yfeldblum

Differential Revision: D4008123

fbshipit-source-id: 37845eeec139007738f99e72ecfb969c6a2e5652

5 years agoPrint why we called std::terminate
Phil Willoughby [Wed, 12 Oct 2016 09:44:45 +0000 (02:44 -0700)]
Print why we called std::terminate

When terminating because we were told to rethrow a nonexistent exception, we
now print that to `std::cerr` for the benefit of users on platforms which do not dump a stack trace on

Reviewed By: yfeldblum

Differential Revision: D4001087

fbshipit-source-id: 5b834cfc75bf42bec06a49ac9db55789a83df180

5 years agoAdd support in enumerate() for iterators with exotic pointers
Giuseppe Ottaviano [Wed, 12 Oct 2016 04:12:59 +0000 (21:12 -0700)]
Add support in enumerate() for iterators with exotic pointers

Summary: I thought we wouldn't need this but it turns out Thrift Frozen2 iterators return proxies (it's proxies all the way down). Thanks to ericniebler for trick.

Reviewed By: yfeldblum

Differential Revision: D4005700

fbshipit-source-id: 1911996afa075c1d819a3aaea2ee924bc2ae2f20

5 years agoFix build on Mac OSX Sierra
Christopher Dykes [Wed, 12 Oct 2016 00:40:01 +0000 (17:40 -0700)]
Fix build on Mac OSX Sierra

There are two changes here.
The first is to eliminate `detail::DEFAULT_CLOCK_ID` from `Benchmark.[cpp|h]` as Sierra defines `clockid_t` as an enum type, which means that calling `clock_gettime(detail::DEFAULT_CLOCK_ID` would fail, because the enums are incompatible. As this was being used as a default, but is not actually changable anywhere, I just got rid of `detail::DEFAULT_CLOCK_ID` entirely.
The second is to move `portability/BitsFunctexcept.cpp` into `libfollybase_la_SOURCES`, because it's needed for generating the fingerprint tables.

Reviewed By: yfeldblum

Differential Revision: D4004843

fbshipit-source-id: b2a9c33f8e516d8eb3cdc5ab093f4946ac9ed37e

5 years agofolly: signal-handler: dynamic cache size, based on number of dynamic-loaded ELF...
Lucian Grijincu [Wed, 12 Oct 2016 00:03:55 +0000 (17:03 -0700)]
folly: signal-handler: dynamic cache size, based on number of dynamic-loaded ELF files

I added kFatalSignalHandlerCacheSize in {D3984649}, and chose a big
number to satisfy most use-cases. But the size increase is non-trivial.

Memory usage:

- before: 5.1 MB in folly::symbolizer::SignalSafeElfCache::SignalSafeElfCache
- after: 80.0 MB in folly::symbolizer::SignalSafeElfCache::SignalSafeElfCache

Switch to a dynamic approach and everyone pays for what they use.
The r_debug structure contains (amongst others) the following fields:

  struct link_map *r_map:
    A linked list of loaded objects.

  enum { RT_CONSISTENT, RT_ADD, RT_DELETE } r_state:
    The current state of the r_map list.  RT_CONSISTENT means that r_map
    is not currently being modified and may safely be inspected.  RT_ADD
    means that an object is being added to r_map, and that the list is
    not guaranteed to be consistent.  Likewise RT_DELETE means that an
    object is being removed from the list.
  /* Call the OS-dependent function to set up life so we can do things like
     file access.  It will call `dl_main' (below) to do all the real work
     of the dynamic linker, and then unwind our frame and run the user
     entry point on the same stack we entered on.  */
  start_addr = _dl_sysdep_start (arg, &dl_main);

  /* Initialize the data structures for the search paths for shared
     objects.  */
  _dl_init_paths (library_path);

  /* Initialize _r_debug.  */
  struct r_debug *r = _dl_debug_initialize (GL(dl_rtld_map).l_addr,
  r->r_state = RT_CONSISTENT;

  /* We start adding objects.  */
  r->r_state = RT_ADD;
  _dl_debug_state ();
  LIBC_PROBE (init_start, 2, LM_ID_BASE, r);


  /* Notify the debugger all new objects are now ready to go.  We must re-get
     the address since by now the variable might be in another object.  */
  r = _dl_debug_initialize (0, LM_ID_BASE);
  r->r_state = RT_CONSISTENT;
  _dl_debug_state ();
  LIBC_PROBE (init_complete, 2, LM_ID_BASE, r);


Reviewed By: bixue2010

Differential Revision: D3996974

fbshipit-source-id: e24d72e3cc0339e4cf1acdd2f4c9a7ebfcfdf739

5 years agofolly/ConcurrentSkipList.h: avoid shadowing warnings (trivial)
Jim Meyering [Tue, 11 Oct 2016 18:25:41 +0000 (11:25 -0700)]
folly/ConcurrentSkipList.h: avoid shadowing warnings (trivial)

Compiling with gcc's proposed -Wshadow-compatible-local option
exposed some fix-worthy warnings. Rename to avoid the shadowing.

Reviewed By: evilmucedin

Differential Revision: D3999026

fbshipit-source-id: 26cb3033ba8c5538cc9217993f2fda6aef954a8f

5 years agoBatchDispatcher add() interface
Shubhanshu Agrawal [Tue, 11 Oct 2016 15:02:40 +0000 (08:02 -0700)]
BatchDispatcher add() interface

Summary: converting add(value&&) to add(value) to make it easier for callsites to use.

Reviewed By: A5he

Differential Revision: D4000416

fbshipit-source-id: c66f4d93d1223286bbc786d17f65ed02a63cc794

5 years agoDisabling failing test for fbandroid
Louis Boval [Tue, 11 Oct 2016 05:11:03 +0000 (22:11 -0700)]
Disabling failing test for fbandroid

Summary: this particular test fails reliably for fbandroid. Modifying the code in fbcode, assuming it will be synced promptly.

Reviewed By: edelron

Differential Revision: D4000722

fbshipit-source-id: fca5e691ea5c972ddd245acee1c17720730539d4

5 years agoRemove memcpy-UB in fbstring
Nicholas Ormrod [Mon, 10 Oct 2016 22:45:25 +0000 (15:45 -0700)]
Remove memcpy-UB in fbstring

The bug report in t13764686 shows that `folly::StringPiece().str()` causes undefined behavior by passing nullptr into memcpy: the default-initialized StringPiece has two null pointers, which will call `string(char*, size)` with nullptr and 0. This gets forwarded to `podCopy` in fbstring, which calls memcpy and has undefined behavior.

The call pattern `string(nullptr, 0)` is not illegal (note: that syntax is ambiguous, as the literal `0` can be converted to a pointer). Prevent this legal call pattern from causing undefined behavior in fbstring.

This diff puts asserts in podCopy to catch errors, and ensures that current podCopy callsites ensure that the pointer is non-null.

Most podCopy callsites do not need to guarded, since the size is guaranteed to be greater than 0 (which will hard crash instead). The interesting callsite is in `initMedium`, because there is a mode (disableSSO) which will allocate empty strings on the heap, so we need to add a guard there.

Reviewed By: luciang

Differential Revision: D3996440

fbshipit-source-id: b311a311973d1d969542245c72035c5b38da58e3

5 years agoUpdate hazard pointers prototype
Maged Michael [Mon, 10 Oct 2016 13:43:52 +0000 (06:43 -0700)]
Update hazard pointers prototype

- Got rid of std::function reclamation functions and added a Deleter template parameter.
- Got rid of the flush() member functions of hazptr_domain
- Added a lock-free non-bool member function to get a protected pointer.
- Implemented the interface changes.
- Changed the order of accesses in reading the shared list of objects vs reading the hazard pointers. I think the previous order would have allowed recently protected objects to be reclaimed incorrectly.
Updated the examples and tests accordingly.

Reviewed By: davidtgoldblatt

Differential Revision: D3981284

fbshipit-source-id: 35ff60da3aea1f67c58d82437dda58f6d8b07bf5

5 years agoMake exception_wrapper::throwException() strict
Phil Willoughby [Mon, 10 Oct 2016 10:55:14 +0000 (03:55 -0700)]
Make exception_wrapper::throwException() strict

The definition of `throwException()` is changed to add the `[[noreturn]]`
attribute for consistency with `std::rethrow_exception`. If the
`exception_wrapper` contains an exception, that is rethrown; if not, then
`std::terminate` is called.

An alternative design which threw an exception if the `exception_wrapper` did not contain an exception was rejected because of the ambiguity it introduced as to whether the exception thrown was wrapped or not.

Benchmarks before:


folly/test/ExceptionWrapperBenchmark.cpp        relative  time/iter  iters/s
exception_ptr_create_and_test                                1.45us  689.01K
exception_wrapper_create_and_test               4337.80%    33.46ns   29.89M
exception_ptr_create_and_test_concurrent                   342.99us    2.92K
exception_wrapper_create_and_test_concurrent     101.41%   338.21us    2.96K
exception_ptr_create_and_throw                               3.05us  327.89K
exception_wrapper_create_and_throw               140.22%     2.17us  459.77K
exception_wrapper_create_and_cast               8956.80%    34.05ns   29.37M
exception_ptr_create_and_throw_concurrent                  372.68us    2.68K
exception_wrapper_create_and_throw_concurrent    102.54%   363.44us    2.75K
exception_wrapper_create_and_cast_concurrent     110.93%   335.97us    2.98K

and after:


folly/test/ExceptionWrapperBenchmark.cpp        relative  time/iter  iters/s
exception_ptr_create_and_test                                1.46us  684.09K
exception_wrapper_create_and_test               4368.84%    33.46ns   29.89M
exception_ptr_create_and_test_concurrent                   341.20us    2.93K
exception_wrapper_create_and_test_concurrent      99.88%   341.60us    2.93K
exception_ptr_create_and_throw                               3.08us  324.93K
exception_wrapper_create_and_throw               106.93%     2.88us  347.46K
exception_wrapper_create_and_cast               9532.05%    32.29ns   30.97M
exception_ptr_create_and_throw_concurrent                  363.73us    2.75K
exception_wrapper_create_and_throw_concurrent    101.04%   360.00us    2.78K
exception_wrapper_create_and_cast_concurrent     104.50%   348.07us    2.87K

Reviewed By: yfeldblum

Differential Revision: D3923939

fbshipit-source-id: 6fc58ac571e59e4122f1fbd194c678e3a3841057

5 years agofolly/Foreach.h: allow FOR_EACH to be nested with no shadowing v2016.10.10.00
Jim Meyering [Sun, 9 Oct 2016 21:53:30 +0000 (14:53 -0700)]
folly/Foreach.h: allow FOR_EACH to be nested with no shadowing

Prior to this change, any nested use of FOR_EACH would
induce a shadowed declaration for each of the two local
state variables it declares.

This makes the names of those variables __LINE__-dependent,
so that there is no shadowing, as long as each nested use
is on a different line.

This also adds a new test that (prior to this change) would fail to
compile with an option like -Werror=shadow-compatible-local.

Since this change relies on cpp token concatenation, I have included
The fix defines a new helper macro, _FE_ANON, to derive each new variable
name. I wondered whether to do this for every other FOR_* macro here, but
since so far, I have encountered more than 10 cases of nested FOR_EACH
uses in a large corpus, but no nesting of any other FOR_* macro, I am
content to do it only for this one.

Reviewed By: yfeldblum

Differential Revision: D3992956

fbshipit-source-id: f26fba89bc661bb9d22747dec0acdcf8c648fb83

5 years agofolly/io/async/HHWheelTimer.cpp: remove decl of unused variable (trivial)
Jim Meyering [Sat, 8 Oct 2016 15:39:18 +0000 (08:39 -0700)]
folly/io/async/HHWheelTimer.cpp: remove decl of unused variable (trivial)

In HHWheelTimer::scheduleTimeout, remove the declaration of the
unused variable, "prev".

Reviewed By: yfeldblum

Differential Revision: D3987409

fbshipit-source-id: 3a9291bf104e6865d608adfb6118609faa0aea96

5 years agoLimit maximum number of iterations in folly::Benchmark with bm_max_iters
Yinghai Lu [Sat, 8 Oct 2016 01:38:14 +0000 (18:38 -0700)]
Limit maximum number of iterations in folly::Benchmark with bm_max_iters

We are using `bm_max_iters` in folly and other part of the code in fbcode but it didn't really exist in Benchmark.cpp. For example:

This patch adds back this functionality.

This diff depends on D3990600.

Reviewed By: yfeldblum

Differential Revision: D3989499

fbshipit-source-id: d2303b2bebb196e84a592d54a72af68171971491

5 years agoClang-format the flag declarations in Benchmark.cpp
Yinghai Lu [Sat, 8 Oct 2016 01:38:12 +0000 (18:38 -0700)]
Clang-format the flag declarations in Benchmark.cpp

Summary: This patch prepares the later diff which will add a new flag. Just make things nice.

Reviewed By: igorsugak

Differential Revision: D3990600

fbshipit-source-id: d4f53a5b44708ca025e4f6615d161b24de685931

5 years agoDisallow COW in FBString
Nicholas Ormrod [Fri, 7 Oct 2016 20:05:44 +0000 (13:05 -0700)]
Disallow COW in FBString

Remove all code that create large strings.

This diff does not remove code that deals with large strings, so that the transition to libgcc will be smoother. Note that the FBSTRING_ASSERT inside of setCapacity will be disabled in the libgcc build.

Reviewed By: luciang, ot

Differential Revision: D3977459

fbshipit-source-id: 09c1712854b31509138c5cc8baf19caf3c715bb6

5 years agofolly: symbolizer: increase default signal-handler symbolizer elf file cache size...
Lucian Grijincu [Fri, 7 Oct 2016 19:58:50 +0000 (12:58 -0700)]
folly: symbolizer: increase default signal-handler symbolizer elf file cache size (make it work with large number of .so)

Bump the default size so we can symbolize symbols in dynamically linked
binaries with lots and lots of shared libraries :)

  * Will not grow; once the capacity is reached, lookups for files that
  * aren't already in the cache will fail (return nullptr).
  class SignalSafeElfCache

Reviewed By: ot, philippv

Differential Revision: D3984649

fbshipit-source-id: acdca9b9adf6c16f3a1556bc3f57b28844cdd9cc

5 years agoFunctionScheduler - return whether shutdown was successful
Mirek Klimos [Fri, 7 Oct 2016 18:18:57 +0000 (11:18 -0700)]
FunctionScheduler - return whether shutdown was successful

Summary: We want to fail fast if we attempt to stop a scheduler that was not running

Reviewed By: yfeldblum

Differential Revision: D3976919

fbshipit-source-id: d8aa8b35aa9e22758e6a7ef64f48a7dd6d990b1c

5 years agoReverted commit D3979179
Alex Yarmula [Thu, 6 Oct 2016 15:51:33 +0000 (08:51 -0700)]
Reverted commit D3979179

When `delayed` is called on the Future, the underlying `futures::sleep` call runs on a timer thread, and the resulting callback is called on the same thread. Therefore, in the following sequence:

  f.via(&someExecutor).within(one_ms).then([&]() { /* [1] */ })

The code in [1] is not running in someExecutor. This can cause confusion by users of the library who expect the initial `via` to be sticky.

This change returns to the prior `Executor` after `delayed` is finished.

Reviewed By: yfeldblum

Differential Revision: D3979179

fbshipit-source-id: e1448f5603f0c9440490ae3bf0e670687f4179f3

5 years agofolly/io: avoid shadowing warnings
Jim Meyering [Thu, 6 Oct 2016 15:07:29 +0000 (08:07 -0700)]
folly/io: avoid shadowing warnings

Summary: Fix shadowing warnings.

Reviewed By: russoue

Differential Revision: D3979165

fbshipit-source-id: 19a3940e210e4d5a698dbc0e6d74b317a74a94fa

5 years agouse previous Executor after delayed
Alex Yarmula [Thu, 6 Oct 2016 14:21:22 +0000 (07:21 -0700)]
use previous Executor after delayed

When `delayed` is called on the Future, the underlying `futures::sleep` call runs on a timer thread, and the resulting callback is called on the same thread. Therefore, in the following sequence:

  f.via(&someExecutor).within(one_ms).then([&]() { /* [1] */ })

The code in [1] is not running in someExecutor. This can cause confusion by users of the library who expect the initial `via` to be sticky.

This change returns to the prior `Executor` after `delayed` is finished.

Reviewed By: yfeldblum

Differential Revision: D3979179

fbshipit-source-id: 936ff5626e8ac377ffb15babf573349466984e3a

5 years agoGate `__returns_nonnull__` attribute to gcc 4.9 or later
Michael Lee [Wed, 5 Oct 2016 23:15:02 +0000 (16:15 -0700)]
Gate `__returns_nonnull__` attribute to gcc 4.9 or later

Summary: The attribute `__returns_nonnull__` is only applicable for gcc 4.9 or later (

Reviewed By: ot

Differential Revision: D3976598

fbshipit-source-id: 3c678a11c5046e5a8b7686327dbb3ebc9b598616

5 years agofolly: deprecate StringPiece::hash
Lucian Grijincu [Wed, 5 Oct 2016 20:49:26 +0000 (13:49 -0700)]
folly: deprecate StringPiece::hash

Summary: It's bad mkey. Says so on the tin can. Alert/annoy its users.

Reviewed By: yfeldblum, ot

Differential Revision: D3973625

fbshipit-source-id: 23f4e16182749d016084fcb378a1170ff402c259

5 years agofolly: kill StringPieceHash
Lucian Grijincu [Wed, 5 Oct 2016 20:46:42 +0000 (13:46 -0700)]
folly: kill StringPieceHash

Summary: no uses remain in folly

Reviewed By: yfeldblum

Differential Revision: D3972507

fbshipit-source-id: 356ddbcf24c217b0f589989ab6cc3910d0f7baf3

5 years agoUpdated example and methodology for using DeterministicSchedule support for auxiliary...
Maged Michael [Wed, 5 Oct 2016 12:18:09 +0000 (05:18 -0700)]
Updated example and methodology for using DeterministicSchedule support for auxiliary data and global invariants

Depends on D3792669

Updating the test and methodology based on the experience with fine-grained testing dynamic MPMCQueue using DeterministicSchedule's support for auxiliary data and global invariants.

Updates D3675447

Reviewed By: djwatson

Differential Revision: D3794217

fbshipit-source-id: d2862895cb8dea120e758beeb24d6ae15191b013

5 years agoFix optional default initialization
Matthieu Martin [Wed, 5 Oct 2016 06:28:35 +0000 (23:28 -0700)]
Fix optional default initialization

Summary: Initialize Optional internal memory, so that "un-initialized memory" tools don't flag its callsites.

Reviewed By: yfeldblum

Differential Revision: D3960462

fbshipit-source-id: 3bd0109959fb93e040fa2e874f586b3508e46dd2

5 years agofolly: experimental: ABI-break: replace deprecated folly::StringPieceHash with folly...
Lucian Grijincu [Wed, 5 Oct 2016 04:28:20 +0000 (21:28 -0700)]
folly: experimental: ABI-break: replace deprecated folly::StringPieceHash with folly::Hash

folly::StringPieceHash has a large comment explaining why not to use it, but it's not marked as deprecated.

It's also conveniently named StringPieceHash and not StringPieceHashDEPRECATED so new uses appear from time to time.

Let's nuke the default before it does more damage.

Reviewed By: yfeldblum

Differential Revision: D3967239

fbshipit-source-id: 04b0564fdf2a70f1195ef2afdff092bc63cde975

5 years agofolly: elfcache: nuke StringPieceHash
Lucian Grijincu [Wed, 5 Oct 2016 00:56:57 +0000 (17:56 -0700)]
folly: elfcache: nuke StringPieceHash

Summary: #accept2ship

Reviewed By: bixue2010

Differential Revision: D3971138

fbshipit-source-id: 25e0aeddebf001dd8e27a4939842915ccf1000ad

5 years agoImplicit construct sorted_vector_set from std::initializer_list
Qinfan Wu [Tue, 4 Oct 2016 20:58:10 +0000 (13:58 -0700)]
Implicit construct sorted_vector_set from std::initializer_list

The following thrift file doesn't work:
  typedef set<i32> (cpp.template = "folly::sorted_vector_set") IntSet
  struct Test {
    1: map<i32, IntSet> m = {7: [1, 2]},

The generated file looks like this:
      : m(std::initializer_list<
          std::pair<const int32_t, folly::sorted_vector_set<int32_t>>>{
          {7, std::initializer_list<int32_t>{1, 2}}}) {}

Given that `std::set` allows implicit construction from `std::initializer_list`, it's probably okay to make `folly::sorted_vector_set` the same.

Reviewed By: ot

Differential Revision: D3968054

fbshipit-source-id: 978d59a7e545e44d603eeb9671815ac7f2c57342

5 years agofolly/io/async/AsyncSocket.cpp: avoid shadowing warning for "rv"
Jim Meyering [Tue, 4 Oct 2016 19:41:07 +0000 (12:41 -0700)]
folly/io/async/AsyncSocket.cpp: avoid shadowing warning for "rv"

Enabling the proposed-for-upstream gcc -Wshadow-local option exposes this shadowing:
folly/io/async/AsyncSocket.cpp: In member function 'virtual void folly::AsyncSocket::connect(folly::AsyncSocket::ConnectCallback*, const folly::SocketAddress&, int, const OptionMap&, const folly::SocketAddress&)':
folly/io/async/AsyncSocket.cpp:421:11: error: declaration of 'rv' shadows a previous local [-Werror=shadow-compatible-local]
folly/io/async/AsyncSocket.cpp:364:9: error: shadowed declaration is here [-Werror=shadow-compatible-local]

Reviewed By: ngoyal, knekritz

Differential Revision: D3963873

fbshipit-source-id: 85b3ab1fc3b200b81615a00d66a183cc9c792a90

5 years agohazptr: Replace friend "swap" with a member function and a non-friend
Arthur O'Dwyer [Tue, 4 Oct 2016 17:24:45 +0000 (10:24 -0700)]
hazptr: Replace friend "swap" with a member function and a non-friend

This matches what the STL does with e.g. std::vector::swap() and std::shared_ptr::swap().

Should be relatively uncontroversial, I would think.

Reviewed By: magedm

Differential Revision: D3963285

Pulled By: yfeldblum

fbshipit-source-id: 3fa6bf77e66fa9a673cb648b0bf87b1db3caa6c8

5 years agofolly/io/Compression.cpp: avoid shadowing warnings for "rc"
Jim Meyering [Tue, 4 Oct 2016 16:11:08 +0000 (09:11 -0700)]
folly/io/Compression.cpp: avoid shadowing warnings for "rc"

gcc's -Wshadow (and the under-review -Wshadow-local) would evoke this:
folly/io/Compression.cpp:650:9: error: declaration of 'rc' shadows a previous local [-Werror=shadow-compatible-local]
folly/io/Compression.cpp:637:7: error: shadowed declaration is here [-Werror=shadow-compatible-local]
folly/io/Compression.cpp:566:9: error: declaration of 'rc' shadows a previous local [-Werror=shadow-compatible-local]
folly/io/Compression.cpp:548:7: error: shadowed declaration is here [-Werror=shadow-compatible-local]
In each case, I removed the "int" from the latter declaration.

Reviewed By: philippv

Differential Revision: D3966308

fbshipit-source-id: 584cb9ffe8ba0e56914223c440efabe9e0de6b17

5 years agoOutline several fbstring/malloc functions
Giuseppe Ottaviano [Tue, 4 Oct 2016 04:39:06 +0000 (21:39 -0700)]
Outline several fbstring/malloc functions

Summary: Outline all the non-trivial constructor and the destructor, `assign()`, and `append()` to reduce code size. The fast path of copy, and moves, are instead still inlined.

Reviewed By: philippv

Differential Revision: D3896612

fbshipit-source-id: 25050d4ba28d25da226a7bc49d5b542947d0c512

5 years agoFix a typo
Giuseppe Ottaviano [Tue, 4 Oct 2016 03:01:55 +0000 (20:01 -0700)]
Fix a typo

Reviewed By: lbrandy

Differential Revision: D3964328

fbshipit-source-id: 0b1dc2d4b2a1565019770668445a624625b8257e

5 years agoadd BatchDispatcher
Shubhanshu Agrawal [Mon, 3 Oct 2016 19:13:32 +0000 (12:13 -0700)]
add BatchDispatcher

This diff adds BatchDispatcher, which can be used to batch values while performing IO.
This would be useful in writing single id code in node, and would be use to batch ids while doing IO at storage adapter.

Differential Revision: D3900404

fbshipit-source-id: f53aa352344ff55674c7544302b6a1b4726214b6

5 years agoOptimize frequently inlined FBString methods
Giuseppe Ottaviano [Mon, 3 Oct 2016 18:58:07 +0000 (11:58 -0700)]
Optimize frequently inlined FBString methods

Almost every method of `fbstring` needs to perform category
dispatching. The category constants are `size_t`, which become 8-byte
immediate values in the dispatching code, so even a simple `if
(category() == Category::isSmall)` is quite large. When inlined
hundreds of thousands of time, it adds up.

This diff redefines the category type to be 1 byte (without changing
the ABI). It also optimizes `size()` and `c_str()` and makes them
branch-free, which probably is not going to have any perf impact but
it saves a few bytes.

Generated code for some small functions:

- `reset()`
48 ba 00 00 00 00 00    movabs $0x1700000000000000,%rdx
00 00 17
48 89 f8                mov    %rdi,%rax
c6 07 00                movb   $0x0,(%rdi)
48 89 57 10             mov    %rdx,0x10(%rdi)

20 bytes

48 89 f8                mov    %rdi,%rax
c6 47 17 17             movb   $0x17,0x17(%rdi)
c6 07 00                movb   $0x0,(%rdi)

10 bytes

- `c_str()`
48 b8 00 00 00 00 00    movabs $0xc000000000000000,%rax
00 00 c0
48 85 47 10             test   %rax,0x10(%rdi)
74 08                   je     401fd8
48 8b 07                mov    (%rdi),%rax
c3                      retq
0f 1f 40 00             nopl   0x0(%rax)
48 89 f8                mov    %rdi,%rax

26 bytes (without the `retq`)

f6 47 17 c0             testb  $0xc0,0x17(%rdi)
48 89 f8                mov    %rdi,%rax
48 0f 45 07             cmovne (%rdi),%rax

11 bytes

- `size()`
48 b8 00 00 00 00 00    movabs $0xc000000000000000,%rax
00 00 c0
48 85 47 10             test   %rax,0x10(%rdi)
74 08                   je     401fa8
48 8b 47 08             mov    0x8(%rdi),%rax
c3                      retq
0f 1f 00                nopl   (%rax)
48 0f be 57 17          movsbq 0x17(%rdi),%rdx
b8 17 00 00 00          mov    $0x17,%eax
48 29 d0                sub    %rdx,%rax

36 bytes (without the `retq`)

0f b6 57 17             movzbl 0x17(%rdi),%edx
b8 17 00 00 00          mov    $0x17,%eax
48 29 d0                sub    %rdx,%rax
48 0f 48 47 08          cmovs  0x8(%rdi),%rax

17 bytes

Reviewed By: philippv

Differential Revision: D3957276

fbshipit-source-id: ef40d82bbbb0456b1044421cd02133c268abe39b

5 years agoswitch back to inline assembly in compression::instructions v2016.10.03.00
Philip Pronin [Mon, 3 Oct 2016 04:25:03 +0000 (21:25 -0700)]
switch back to inline assembly in compression::instructions

D3278901 never worked, GCC (at least 4.9) failed to propagate
`FOLLY_TARGET_ATTRIBUTE("arch=haswell")` down, `blsr` and `select64`
weren't inlined in EF coding on Haswell, showing up in CPU profile.

This diff switches back to inline assembly, removing second dispatching
mechanism (IFUNC; in additional to caller-side `Default` / `Nehalem` / `Haswell`
dispatching) for `compression::instructions`, and disables functionality on
compilers not supporting AT&T syntax (MSVC++) for now.

Added `FOLLY_ALWAYS_INLINE` to make sure failure to inline results in
compilation failure.

Reviewed By: ot

Differential Revision: D3959438

fbshipit-source-id: e40573fbfbf38991caa2cd70293aeaeeec3afad7

5 years agoFix folly::Partial copy/move construction
Sven Over [Sun, 2 Oct 2016 17:22:13 +0000 (10:22 -0700)]
Fix folly::Partial copy/move construction

Any attempt to copy or move an object returned by folly::partial
yields a compiler error because it is invoking the constructor
for wrapping callable objects, which triggers a type mismatch.

This diff fixes that by explicitly naming the default implementations
of copy and move constructors.

This diff also adds additional tests that fail to compile without this fix.

And then this diff also moves the Partial class into folly::detail,
because it is not meant to be named in user code, but only returned
by the folly::partial function.

Reviewed By: mhx

Differential Revision: D3923809

fbshipit-source-id: a8883951afd2a1999acbfffc51296393b058f860

5 years agofolly: use std::__throw_out_of_range
Lucian Grijincu [Fri, 30 Sep 2016 19:12:32 +0000 (12:12 -0700)]
folly: use std::__throw_out_of_range

A ton of these functions are tiny and should be inlined. `throw std::out_of_range("bla")` generates quite a lot of code.

See godbolt from {D3940968}:

          movq    %rdi, %rax
          sarq    $32, %rax
          cmpl    %eax, %edi
          jg      .L20
          leal    1(%rdi), %eax
          subq    $8, %rsp
          call    S::outOfRange() [clone .isra.0]

vs previous implementation

          movq    %rdi, %rdx
          sarq    $32, %rdx
          cmpl    %edx, %edi
          jg      .L14
          leal    1(%rdi), %eax
          pushq   %rbp
          pushq   %rbx
          movl    $16, %edi
          subq    $8, %rsp
          call    __cxa_allocate_exception
          movl    $.LC0, %esi
          movq    %rax, %rdi
          movq    %rax, %rbx
          call    std::out_of_range::out_of_range(char const*)
          movl    std::out_of_range::~out_of_range(), %edx
          movl    typeinfo for std::out_of_range, %esi
          movq    %rbx, %rdi
          call    __cxa_throw
          movq    %rax, %rbp
          movq    %rbx, %rdi
          call    __cxa_free_exception
          movq    %rbp, %rdi
          call    _Unwind_Resume

This pattern is in the standard library as well:
      at(size_type __n)
if (__n >= _Nm)
return _M_instance[__n];

Reviewed By: yfeldblum

Differential Revision: D3947075

fbshipit-source-id: 6e174c725791762a533a534c8482ea9576460b86

5 years agoFind dwarf.h in libdwarf subdirectory
Ted Percival [Fri, 30 Sep 2016 17:40:00 +0000 (10:40 -0700)]
Find dwarf.h in libdwarf subdirectory

libdwarf headers moved from `/usr/include` to `/usr/include/libdwarf` in [dwarfutils 20160613-2](

Adds a check for the library as well, and fixes a tiny typo in the `libunwind` check.

No longer checks for `libdwarf.h`; only `dwarf.h` is used.

Reviewed By: yfeldblum

Differential Revision: D3878961

Pulled By: Orvid

fbshipit-source-id: 7970d33511bf6d351cec2c8077cb07b85f2f8d33

5 years agofolly: Range: detail::throwOutOfRange -> std::__throw_out_of_range
Lucian Grijincu [Thu, 29 Sep 2016 21:40:13 +0000 (14:40 -0700)]
folly: Range: detail::throwOutOfRange -> std::__throw_out_of_range

No need to define our own.

Generated code is similar: (one extra move instruction).

        subq    $8, %rsp
        call    S::outOfRange() [clone .isra.0]

        .string "out of range"
        subq    $8, %rsp
        movl    $.LC1, %edi
        call    std::__throw_out_of_range(char const*)

Reviewed By: Orvid

Differential Revision: D3945578

fbshipit-source-id: c65e9dea55e8f01f51766b2695af68d2bc92c266

5 years agoFix ThreadCachedInt race condition
Dan Schatzberg [Thu, 29 Sep 2016 15:22:50 +0000 (08:22 -0700)]
Fix ThreadCachedInt race condition

Acquire a SharedMutex at ThreadExit to ensure that after unlinking the ThreadEntry from
the list, future accessAllThreads() won't miss a destroying thread.

This is quite a dangerous fix as it changes some lock ordering semantics. ThreadLocal
elements are now destroyed while holding a lock, so if the destruction function
acquires a different lock, ordering must be consistent with other
uses of accessAllThreads().

I've made accessAllThreads() an opt-in feature via a template parameter and changed
all existing uses. I've also fixed a few lock ordering issues that arose due to this

Reviewed By: andriigrynenko

Differential Revision: D3931072

fbshipit-source-id: 4d464408713184080079698df453b95873bb1a6c

5 years agoFix apple bug around TFO writes
Subodh Iyengar [Thu, 29 Sep 2016 13:42:11 +0000 (06:42 -0700)]
Fix apple bug around TFO writes

When using connectx to do TFO, apple has a bug
where the second write after a TFO write will cause
the socket to throw an ENOTCONN error instead of an
EAGAIN. Linux handles this case fine and returns an
EAGAIN, however apple returns ENOTCONN.

We solve this by treating ENOTCONN as an EAGAIN temporarily.

Reviewed By: yfeldblum

Differential Revision: D3942681

fbshipit-source-id: ab4f0b5fd6cdcfe9c584ea00849705a2d739d65f

5 years agoAdd tfo functions for apple
Subodh Iyengar [Thu, 29 Sep 2016 13:42:10 +0000 (06:42 -0700)]
Add tfo functions for apple

Adds TFO functions for apple devices
Also allows android as well by removing that
restriction. Newer versions of android support
TFO, so this allows us to experiment with them

Reviewed By: yfeldblum

Differential Revision: D3942664

fbshipit-source-id: faf439783b018cf7c987a2e3ade5ea6c0c02bf48

5 years agofolly: Range: outline exception throwing
Lucian Grijincu [Thu, 29 Sep 2016 05:33:39 +0000 (22:33 -0700)]
folly: Range: outline exception throwing

Here's some godbolt:

          movq    %rdi, %rax
          sarq    $32, %rax
          cmpl    %eax, %edi
          jg      .L20
          leal    1(%rdi), %eax
          subq    $8, %rsp
          call    S::outOfRange() [clone .isra.0]

vs previous implementation

          movq    %rdi, %rdx
          sarq    $32, %rdx
          cmpl    %edx, %edi
          jg      .L14
          leal    1(%rdi), %eax
          pushq   %rbp
          pushq   %rbx
          movl    $16, %edi
          subq    $8, %rsp
          call    __cxa_allocate_exception
          movl    $.LC0, %esi
          movq    %rax, %rdi
          movq    %rax, %rbx
          call    std::out_of_range::out_of_range(char const*)
          movl    std::out_of_range::~out_of_range(), %edx
          movl    typeinfo for std::out_of_range, %esi
          movq    %rbx, %rdi
          call    __cxa_throw
          movq    %rax, %rbp
          movq    %rbx, %rdi
          call    __cxa_free_exception
          movq    %rbp, %rdi
          call    _Unwind_Resume

Reviewed By: ot

Differential Revision: D3940968

fbshipit-source-id: b47a41e7cdd863fcef099ff3c21860b2979ee6e8

5 years agoAdd unchecked versions of advance(), subtract(), and subpiece() in Range
Giuseppe Ottaviano [Wed, 28 Sep 2016 22:37:33 +0000 (15:37 -0700)]
Add unchecked versions of advance(), subtract(), and subpiece() in Range

`Range` has a somewhat inconsistent API: most methods assume
argument validity as a precondition (mirroring their STL
counterparts), others check the arguments and throw for invalid ones.

Since `Range` is intended as a zero-cost abstraction on top of
pointers/iterators, unchecked methods should be preferred. At this
point however we cannot change the semantics of `advance()` and other
methods. This diff adds new unchecked versions of these methods.

Reviewed By: luciang

Differential Revision: D3938480

fbshipit-source-id: 6952683ee0716aa1584e79584158fbf3e083b52e

Witchakorn Kamolpornwijit [Wed, 28 Sep 2016 20:00:04 +0000 (13:00 -0700)]

The original code `AC_DEFINE([FOLLY_SUPPLY_MISSING_INT128_TRAITS...` resulted in defining the constant FOLLY_FOLLY_SUPPLY_MISSING_INT128_TRAITS in folly-config.h. This patch fix the name of the constant so that the name match with what appear in Traits.h.

Reviewed By: yfeldblum

Differential Revision: D3931349

Pulled By: Orvid

fbshipit-source-id: bd7e7d3580d02134d36706bd5836822135232d26

5 years agofolly/experimental/hazptr: fix gcc 5 build
Maged Michael [Wed, 28 Sep 2016 08:36:57 +0000 (01:36 -0700)]
folly/experimental/hazptr: fix gcc 5 build

Summary: Fix includes in memory_resource.h

Reviewed By: yfeldblum

Differential Revision: D3935848

fbshipit-source-id: 713ce85ad17072779dfa1c667459e367c5e1d4b7

5 years agoSet a default value for slot in SharedMutex.h
Michael Lee [Wed, 28 Sep 2016 00:31:22 +0000 (17:31 -0700)]
Set a default value for slot in SharedMutex.h

`SharedMutexImpl::lockSharedImpl` has a potentially uninitialized access:

  Assume state = 0
  canAlreadyDefer = (state & kMayDefer) != 0 ==> false
  aboveDeferThreshold = (state & kHasS) >= (kNumSharedToStartDeferring - 1) * kIncrHasS ==> false

  if (canAlreadyDefer || (aboveDeferThreshold && !drainInProgress)) ==> false

  line:1452: gotSlot(slot)->compare_exchange_strong(...) uses slot uninitialized

Reviewed By: Orvid

Differential Revision: D3933638

fbshipit-source-id: 0fbce5c00b8b1f34e50c302cb88def97853c5afe

5 years agoIntroducing folly::FunctionRef
Sven Over [Tue, 27 Sep 2016 14:44:32 +0000 (07:44 -0700)]
Introducing folly::FunctionRef

This commit introduces a simple function reference type, similar to
std::reference_wrapper, but the template parameter is the function
signature type rather than the type of the referenced object.

A folly::FunctionRef is cheap to construct as it contains only a
pointer to the referenced callable and a pointer to a function which
invokes the callable.

The user of FunctionRef must be aware of the reference semantics:
storing a copy of a FunctionRef is potentially dangerous and should
be avoided unless the referenced object definitely outlives the
FunctionRef object. Thus any function that accepts a FunctionRef
parameter should only use it to invoke the referenced function and
not store a copy of it. Knowing that FunctionRef itself has reference
semantics, it is generally okay to use it to reference lambdas that
capture by reference.

Reviewed By: ericniebler

Differential Revision: D3277364

fbshipit-source-id: 0a7676919cd240da5b6e1f94cadba4289e0aca28

5 years agoSSL_SESSION wrapper
Anirudh Ramachandran [Mon, 26 Sep 2016 21:55:14 +0000 (14:55 -0700)]

This is a start to wrapping various SSL objects going forward so different
binaries can choose different version of OpenSSL (i.e., BoringSSL, OpenSSL
1.1.0, OpenSSL 1.0.2, etc.). There's no change to the caller - everyone just
uses 'SSLSession', but the implementation details vary

Reviewed By: siyengar

Differential Revision: D3707791

fbshipit-source-id: f895334a768cb7d43b41af40c9bc06be5307cc7f

5 years agoAsyncSSLSocket::getSSLClientCiphers using static map
Anirudh Ramachandran [Mon, 26 Sep 2016 20:14:55 +0000 (13:14 -0700)]
AsyncSSLSocket::getSSLClientCiphers using static map

OpenSSL SSL_METHOD->get_cipher_by_char is not present in either OpenSSL
1.1.0 or BoringSSL. In addition, knekritz reports that time's being spent in
binary searching for cipher names.

Since the ciphercodes and names are (fairly) static, we store these in a static
hash map.

Reviewed By: siyengar

Differential Revision: D3275185

fbshipit-source-id: 08b36f3e73239b415b74c6ecc30ed65832d9ebd0

5 years agoMake a SharedPromise from a Future
Phil Willoughby [Mon, 26 Sep 2016 11:59:05 +0000 (04:59 -0700)]
Make a SharedPromise from a Future

Summary: Makes it easy to split a Future into multiple Futures

Reviewed By: rongrong

Differential Revision: D3885897

fbshipit-source-id: 6ac9fb22444dd828fbdebb44b06bf3d93d0f7583

6 years agoImprove documentation of MPMCQueue size and sizeGuess methods v2016.09.26.00
Eli Pozniansky [Fri, 23 Sep 2016 20:08:23 +0000 (13:08 -0700)]
Improve documentation of MPMCQueue size and sizeGuess methods

Summary: See title.

Reviewed By: nbronson

Differential Revision: D3914188

fbshipit-source-id: dd9ccd0c48911632d229ae675cc40d835ea14724

6 years agoAvoid external linkage call inside __ifunc__
Alisson Gusatti Azzolini [Fri, 23 Sep 2016 01:49:16 +0000 (18:49 -0700)]
Avoid external linkage call inside __ifunc__

In debug mode, CpuId() ends up triggering external linkage from inside this __ifunc__.
However, in PIC, the relocation of external symbols are not ready yet, causing a segfault.

This only reproes in dynamic linking (PIC / so) and dbg mode.

Reviewed By: luciang

Differential Revision: D3895239

fbshipit-source-id: 2b7856c10abb5cfe24736d5bfac28e7e9d0e8272

6 years agoSwitch SocketAddressTest to use TemporaryFile instead of mkstemp
Michael Lee [Thu, 22 Sep 2016 18:09:57 +0000 (11:09 -0700)]
Switch SocketAddressTest to use TemporaryFile instead of mkstemp

Summary: Switch from mkstemp to the slightly more portable `folly::test::TemporaryFile`

Reviewed By: jsedgwick

Differential Revision: D3890411

fbshipit-source-id: e98d1e3a5adae92af1bb36f6213b194f633fab0f

6 years agoFix a couple comments
Giuseppe Ottaviano [Thu, 22 Sep 2016 18:06:31 +0000 (11:06 -0700)]
Fix a couple comments

Reviewed By: nbronson

Differential Revision: D3905865

fbshipit-source-id: 2743af4ae1b34adb0f8e611e672f9b6068430ec9

6 years agofix flaky TimeKeeper unit-test
Tianjiao Yin [Thu, 22 Sep 2016 08:06:15 +0000 (01:06 -0700)]
fix flaky TimeKeeper unit-test

Sometime we have such unit-test failure

folly/futures/test/TimekeeperTest.cpp:134: Failure
Value of: flag
  Actual: false
Expected: true

This diff should make it less flaky, though I am not sure what's the best way to fix this flaky unit-test.

Reviewed By: yfeldblum

Differential Revision: D3889630

fbshipit-source-id: e7486d75fbcb3081d06724d213d4a2cf8942955d

6 years agoAdd logs for TFO succeded
Subodh Iyengar [Thu, 22 Sep 2016 08:01:42 +0000 (01:01 -0700)]
Add logs for TFO succeded

Add logs for TFO to AsyncSocket
and wangle

Reviewed By: knekritz

Differential Revision: D3906104

fbshipit-source-id: 9a79b6f91273f8017a5e0d72fe5bdc8eff645ebc

6 years agoexperimental/observer/detail/ObserverManager.cpp missing gflags
Michael Lee [Thu, 22 Sep 2016 06:29:55 +0000 (23:29 -0700)]
experimental/observer/detail/ObserverManager.cpp missing gflags

Summary: Fix the build due to a missing gflags header

Reviewed By: yfeldblum

Differential Revision: D3905525

fbshipit-source-id: cc09189196a04741162c2d1561545d3fec417e5e

6 years agoFix compilation for libc++ 3.9
Shoaib Meenai [Wed, 21 Sep 2016 15:07:33 +0000 (08:07 -0700)]
Fix compilation for libc++ 3.9

libc++ 3.9 and above define the __throw* functions inside stdexcept, so
defining them ourselves leads to compilation errors when compiling
against libc++ 3.9. Add appropriate preprocessor guards to avoid this.

Reviewed By: meyering

Differential Revision: D3898284

fbshipit-source-id: 435a28c2b3a83ee4d8f5af0df0343c524469011e

6 years agoFix detaching from evb
Subodh Iyengar [Tue, 20 Sep 2016 23:25:32 +0000 (16:25 -0700)]
Fix detaching from evb

When we attach and detach from the
event base threads during connections
in the case of SSL it could result in
a crash while using TFO.

When TFO was enabled connectTimeout_
was set, which was not detached.

In the case when we would
fall back from TFO in case TFO did not
succeed, we would try and schedule a
connect timeout again. However because
we are now scheduled in a new evb thread,
this would cause the scheduleTimeout to
assert that it was not in the correct evb

This fixes the issue by attaching and detaching
connect timeouts as well.

Reviewed By: ngoyal

Differential Revision: D3892490

fbshipit-source-id: 278c0b8029022144cd59366ceb0ce83f0a60a307

6 years agouse folly::Function<void()> in folly::Executor interface
Sven Over [Tue, 20 Sep 2016 22:04:07 +0000 (15:04 -0700)]
use folly::Function<void()> in folly::Executor interface

This diff changes the definition of `folly::Func` from `std::function<void()>`
over to `folly::Function<void()>`. This mostly affects the interface of
`folly::Executor` and derived and related classes. By using
`folly::Function<void()>`, we allow to have lambdas capturing move-only types
passed to executors. This continues an effort to get rid of the potentially
dangerous `folly::MoveWrapper` by allowing to capture move-only types in lambdas
when using `folly::Future`, `folly::EventBase` and now `folly::Executor`.

Reviewed By: yfeldblum

Differential Revision: D3706449

fbshipit-source-id: 11c2cee32cb9f0298c39b7b1695a477777eeb3aa

6 years agoDraft prototype of hazard pointers C++ template library
Maged Michael [Tue, 20 Sep 2016 20:47:17 +0000 (13:47 -0700)]
Draft prototype of hazard pointers C++ template library

Summary: Make draft of hazard pointers prototype public

Reviewed By: djwatson

Differential Revision: D3870280

fbshipit-source-id: e029efa336585055f67687059e10ae11766f8d7f

6 years agoRemove boost::barrier from AsyncUDPSocketTest.
Andrey Ignatov [Tue, 20 Sep 2016 19:14:33 +0000 (12:14 -0700)]
Remove boost::barrier from AsyncUDPSocketTest.

`EventBase` has its own methods to wait till it's ready so we can avoid using
barriers and remove boost dependency.

Reviewed By: mzlee

Differential Revision: D3894408

fbshipit-source-id: f050a982c98c4f672cf295845115686c95fc7919

6 years agoFix fibers asan integration for new asan APIs
Igor Sugak [Tue, 20 Sep 2016 07:51:26 +0000 (00:51 -0700)]
Fix fibers asan integration for new asan APIs

Updating fibers code to use

Patch by andriigrynenko!

Reviewed By: andrewcox

Differential Revision: D3500482

fbshipit-source-id: f51f4bb4ebf0d95a898eb1d4098459aa691acd61

6 years agofix ZSTD support
Philip Pronin [Tue, 20 Sep 2016 03:52:08 +0000 (20:52 -0700)]
fix ZSTD support

Existing logic is broken (unable to correctly handle chained `IOBuf`
in case of both `compress` and `uncompress`) and has unnecessarly strict
`needsUncompressedLength() == true` requirement.

This diff switches `ZSTDCodec` to use streaming to handle chained `IOBuf`,
drops `needsUncompressedLength() == true`.

Reviewed By: luciang

Differential Revision: D3827579

fbshipit-source-id: 0ef6a9ea664ef585d0e181bff6ca17166b28efc2

6 years agoTry using the last Deferred reader slot first
Qi Wang [Mon, 19 Sep 2016 23:48:17 +0000 (16:48 -0700)]
Try using the last Deferred reader slot first

When trying to find an empty deferred reader slot, getting the current CPU can
take quite a few cycles, e.g. >1% CPU on SMC (

Let's track the last slot used by this thread and try that slot first before reading
CPU id and doing the search.

u-benchmark results seem to be improving generally (though a bit noisy and not
sure how much to trust). Results w/ this diff on left side:

Reviewed By: nbronson

Differential Revision: D3857793

fbshipit-source-id: 8b1c005362c82e748a663100f889b0b99dc257fe

6 years agomove `shellQuote` to implementation file v2016.09.19.00
Dominik Gabi [Fri, 16 Sep 2016 22:00:45 +0000 (15:00 -0700)]
move `shellQuote` to implementation file

Summary: Fixing ODR violations...

Reviewed By: simpkins, yfeldblum

Differential Revision: D3875667

fbshipit-source-id: 0d8ec0b48e14fffb7e3e60c0e68e2576b2f58d1e

6 years agoescape `{` in cmd for deprecated `Substring(std::string...)`
Dominik Gabi [Fri, 16 Sep 2016 22:00:43 +0000 (15:00 -0700)]
escape `{` in cmd for deprecated `Substring(std::string...)`

Since `shellify` interprets the command as a format string we need to escape
`{` and `}`.

Reviewed By: ldemailly, simpkins

Differential Revision: D3874605

fbshipit-source-id: f47db387c3a44a3ba1c0c1d4726b7212fcb5ef3e

6 years agoupdate stats APIs to use TimePoint vs Duration correctly
Adam Simpkins [Fri, 16 Sep 2016 19:23:41 +0000 (12:23 -0700)]
update stats APIs to use TimePoint vs Duration correctly

Update the stats APIs to correcly distinguish between TimePoint and Duration

This does leave addValue() and update() APIs in place that accept Duration
values, for backwards compatibility.  These should eventually be removed once
all code has been converted to call the new APIs.

Reviewed By: yfeldblum

Differential Revision: D3808805

fbshipit-source-id: 36d6574ba4a09db7eb9f1a35e47addd3e07f8461

6 years agoReturn rvalue references from &&-qualified members of dynamic
Tom Jackson [Fri, 16 Sep 2016 17:09:32 +0000 (10:09 -0700)]
Return rvalue references from &&-qualified members of dynamic

Summary: Let the caller do a move, don't force one.

Reviewed By: yfeldblum

Differential Revision: D3873129

fbshipit-source-id: 40c6bf564bcbf794830c99ea1248a9c1bb30e9b0

6 years agoReduce footprint of ScribeClient
Vladimir Slaykovskiy [Fri, 16 Sep 2016 09:53:58 +0000 (02:53 -0700)]
Reduce footprint of ScribeClient

I've found all code paths that contributed to the number of threads in new ScribeClient. Here's a list of flags that reduce the number of thread:
  --default_tls_thread_count=1 # was 32
  --has_thrift_dispatcher_reporter=0 # creates a thrift server with a few extra threads
  --observer_manager_pool_size=1 # default is 4

Optimizations in this diff:
- Don't initialize OBCClient object if it is not used
- Added FLAG_observer_manager_pool_size to control size of the pool in ObserverManager

Currently OBC counters are switched off globally in ScribeCliean, it means that new code path that creates SR instances is disabled, but eventually we plan to switch it on. Clients that don't use SR and want to keep their RSS/threads profile low, should pass flags listed above. I'll announce this in ScribeUsers group as well.

Differential Revision: D3870704

fbshipit-source-id: 0efad6b3dc43c072ab11cac7e9461c09532ea11c

6 years agowork around LLVM#30305 in folly::Expected, use unified initialization syntax for...
Eric Niebler [Fri, 16 Sep 2016 00:00:30 +0000 (17:00 -0700)]
work around LLVM#30305 in folly::Expected, use unified initialization syntax for extra goodness

Summary: The behavior of inheriting constructors changed in clang 3.9 (see, and folly::Expected needs to change accordingly. As a drive-by improvement, change all invocations of default constructors to use unified initialization syntax.

Reviewed By: igorsugak

Differential Revision: D3872994

fbshipit-source-id: fdaea8b35980df21b8522e2c3d5a8c3be1d84efa

6 years agofolly/.../ExceptionTracerLib.cpp: provide less DOF fodder (avoid heap use-after-free)
Jim Meyering [Thu, 15 Sep 2016 23:26:34 +0000 (16:26 -0700)]
folly/.../ExceptionTracerLib.cpp: provide less DOF fodder (avoid heap use-after-free)

Before this change, an application would fail consistently with a shutdown-time heap use-after-free (i.e., destructor-order-fiasco when one of the following symbols was used after its static-destruction-triggered free):


Each of those would define a classic meyers singleton.
Since each destructor is trivial, we can fix this by making each a pseudo-leaky (indestructible) meyers singleton instead. Since each static value is never destroyed,
it can never cause a DOF error again.

Differential Revision: D3870740

fbshipit-source-id: 625f5d5268768ca0e4125bed72bc66c53618be29

6 years agoKill unneeded dependency on portability/Memory.h
Christopher Dykes [Thu, 15 Sep 2016 20:00:46 +0000 (13:00 -0700)]
Kill unneeded dependency on portability/Memory.h

Summary: This dependency was added but isn't actually needed, so get rid of it.

Reviewed By: mzlee

Differential Revision: D3863923

fbshipit-source-id: 407d2b3db759e30c37d876dde133fc962daeaeae

6 years agoAdd flag to indicate a session resumption attempt
Neel Goyal [Thu, 15 Sep 2016 19:00:17 +0000 (12:00 -0700)]
Add flag to indicate a session resumption attempt

Summary: Track when session resumption was attempted during connect

Reviewed By: shamdor-fb

Differential Revision: D3861028

fbshipit-source-id: 26ca41084faeeb64666b6833896b9510ef2b4c25

6 years agoa simple first step towards using clocks properly in the stats code
Adam Simpkins [Thu, 15 Sep 2016 18:02:47 +0000 (11:02 -0700)]
a simple first step towards using clocks properly in the stats code

Update the timeseries and histogram classes to accept a clock parameter as a
template parameter, instead of a time duration type.

This is a first step towards transitioning the code to correctly distinguishing
between time_point and duration types.  This defines TimePoint and Duration
type aliases, but does not start using them yet.

In upcoming diffs I will start converting more APIs to correctly use TimePoint
instead of just Duration.

For now the default clock type is folly::LegacyStatsClock, which still uses
std::chrono::seconds as the default duration.  At the moment the stats code is
optimized for second granularity--the addValue() code has a fast path when
called in the same second as the last update.  When using finer granularity
durations this fast path can't be used as often.  I will also send out
subsequent diffs to make the code optimized for updates within the same bucket,
rather than just updates with the exact same time value.

Reviewed By: yfeldblum

Differential Revision: D3807715

fbshipit-source-id: 77696c4f44a8d85e4d6ff84d7656fe7a9709797c

6 years agoignore `$SHELL` in `shellify`
Dominik Gabi [Thu, 15 Sep 2016 14:57:15 +0000 (07:57 -0700)]
ignore `$SHELL` in `shellify`

Why `$SHELL` is a bad idea:

- getenv() is not thread safe. (In practice it should work if other threads aren't calling setenv(), but still seems undesirable if we can avoid it.)
- It seems confusing for the program to have different behavior for different developers.
- Other shells besides /bin/sh may have different quoting behaviors. For instance, I don't think csh allows unescaped newlines inside single quotes.
- SHELL might be set to other non-shell-like programs in some cases. (Say, if this gets run from inside a git or mercurial hook it might end up being set to the restricted git or hg shell that only lets you run specific commands.)
Anyway, this isn't related to your diff, so nothing needs to be done for now,
but I would vote for changing this to always use /bin/sh in a future diff.

Both the C system() call and python's subprocess module appear to
always use /bin/sh and ignore $SHELL.

Reviewed By: simpkins

Differential Revision: D3867047

fbshipit-source-id: dab0e6afbe1c20ff13d9a52f212df95af425dd77

6 years agoformatting support for `Subprocess::shellify`
Dominik Gabi [Thu, 15 Sep 2016 14:57:14 +0000 (07:57 -0700)]
formatting support for `Subprocess::shellify`

Adding formatting support with automatic escaping of parameters. Will
subsequentyl try and make it harder for people to use non-statically determined
format strings for this.

Reviewed By: simpkins

Differential Revision: D3866174

fbshipit-source-id: 80bb5886386692643876116fbf338ca0d6986e6a

6 years agodeprecate `folly::Subprocess(std::string, ...)`
Dominik Gabi [Thu, 15 Sep 2016 14:57:13 +0000 (07:57 -0700)]
deprecate `folly::Subprocess(std::string, ...)`

introducing `Subprocess::shellify` to get around this. Still thinking
about how to make sure people escape arguments passing into that function. I'd
like to have something along the lines of Phabricator's `csprintf` here.

Reviewed By: yfeldblum

Differential Revision: D3857827

fbshipit-source-id: 8afbc9f1c62c62e0fc91782e11b808145b370933

6 years agoMake ElfCache follow .gnu_debuglink
Mark Williams [Wed, 14 Sep 2016 22:48:52 +0000 (15:48 -0700)]
Make ElfCache follow .gnu_debuglink

If you split out debug info into a separate file, folly::Symbolizer
didn't find it, and failed to include file and line number info in

This adds a new open mode which follows the .gnu_debuginfo link, and
uses it from ElfCache and SignalSafeElfCache.

Reviewed By: meyering, luciang

Differential Revision: D3852311

fbshipit-source-id: fec9e57378ae59fa1b82d41a163bb9cfcf9ca23c

6 years agoSwap a newly added include of gtest.h with portability/GTest.h
Christopher Dykes [Wed, 14 Sep 2016 21:12:21 +0000 (14:12 -0700)]
Swap a newly added include of gtest.h with portability/GTest.h

Summary: Because it's needed with the way the portability layer works.

Reviewed By: yfeldblum

Differential Revision: D3863954

fbshipit-source-id: 3b682dcc2d2799f17c61fe16a4391fc9f0c2b127

6 years agoTest for folly::SingletonThreadLocal
Phil Willoughby [Wed, 14 Sep 2016 08:53:09 +0000 (01:53 -0700)]
Test for folly::SingletonThreadLocal

Summary: Validates that we create a distinct singleton for each thread and that each such singleton is destroyed at its owning-thread's exit.

Reviewed By: yfeldblum

Differential Revision: D3849146

fbshipit-source-id: af878b32ecfc6c82b866d7a805e1385d74a8a5f5

6 years agoRemove Benchmark.h dependency in SpookyHash tests
Michael Lee [Tue, 13 Sep 2016 23:02:19 +0000 (16:02 -0700)]
Remove Benchmark.h dependency in SpookyHash tests

Summary: It would seem the folly/Benchmark.h dependency was to pull in glog/logging.h. Include what you use.

Reviewed By: yfeldblum

Differential Revision: D3857112

fbshipit-source-id: 8d5bd105e10eee8f5b6e5d377d58ae84d908ed57

6 years agoOptimize toJson() Performance
Milo Yip [Tue, 13 Sep 2016 18:24:35 +0000 (11:24 -0700)]
Optimize toJson() Performance

Removing string::reserve() which causes O(n^2) penalty.

Fixes #477

Reviewed By: yfeldblum

Differential Revision: D3850509

Pulled By: Orvid

fbshipit-source-id: ecf44c35b2aedadc5385d23c325cacab1abfd02d

6 years agoFix recordStack feature in FiberManager
Stepan Palamarchuk [Tue, 13 Sep 2016 02:49:22 +0000 (19:49 -0700)]
Fix recordStack feature in FiberManager

It appears that a feature of recording exact stack usage on fibers is broken for a long time.
This diff brings it back.

Reviewed By: andriigrynenko

Differential Revision: D3847035

fbshipit-source-id: d9cf532774204b6ac2f40e1b39fd98ea06c08f55

6 years agoFix a bug in ObserverManager destructor
Andrii Grynenko [Tue, 13 Sep 2016 00:53:52 +0000 (17:53 -0700)]
Fix a bug in ObserverManager destructor

Summary: facepaw

Reviewed By: ckwalsh

Differential Revision: D3853363

fbshipit-source-id: f863a3e3ae830256f5cff6c56f23254ff6fdee2a

6 years agoFix TLObserver copy constructor
Andrii Grynenko [Tue, 13 Sep 2016 00:53:09 +0000 (17:53 -0700)]
Fix TLObserver copy constructor

Reviewed By: yfeldblum

Differential Revision: D3853494

fbshipit-source-id: 23f5cadcb55a9307fb2a825fe4c53a03c5978d12