folly.git
3 years agoRemove multi-poster support from Baton v2017.12.11.00
Yedidya Feldblum [Sun, 10 Dec 2017 20:39:24 +0000 (12:39 -0800)]
Remove multi-poster support from Baton

Summary:
[Folly] Remove multi-poster support from `Baton`.

Code which needs multi-poster support may use `SaturatingSemaphore` instead.

Reviewed By: magedm

Differential Revision: D6529661

fbshipit-source-id: d9dc053ca984ef3a404e9361910b0044817d4905

3 years agoFix some clang build failures on aarch64
Andrew Gallagher [Sun, 10 Dec 2017 08:55:21 +0000 (00:55 -0800)]
Fix some clang build failures on aarch64

Reviewed By: Orvid

Differential Revision: D6387412

fbshipit-source-id: 4c1424ed4fa49e69885de8cd72e78169314fc502

3 years agoUnboundedBlockingQueue: Remove extra include
Maged Michael [Sun, 10 Dec 2017 06:31:03 +0000 (22:31 -0800)]
UnboundedBlockingQueue: Remove extra include

Summary: Remove include not needed after recent change

Reviewed By: yfeldblum

Differential Revision: D6529608

fbshipit-source-id: 904072084e180125d68bd1ef6e4dffb2939cd1eb

3 years agoFix signed integer overflow in StaticTracepointTest.cpp
Yedidya Feldblum [Sun, 10 Dec 2017 01:09:24 +0000 (17:09 -0800)]
Fix signed integer overflow in StaticTracepointTest.cpp

Summary:
[Folly] Fix signed integer overflow in `StaticTracepointTest.cpp`.

It is undefined behavior and is caught by UBSAN in some test runs.

Reviewed By: Orvid

Differential Revision: D6528445

fbshipit-source-id: e4a199e9005b57bcd93c8c4b69f37ccb1e072bdf

3 years agoSplit Baton wait methods into fast and slow paths
Yedidya Feldblum [Sat, 9 Dec 2017 17:50:25 +0000 (09:50 -0800)]
Split Baton wait methods into fast and slow paths

Summary:
[Folly] Split `Baton` wait methods into fast and slow paths.

Force-inline the fast paths, and force-outline the slow paths.

Reviewed By: magedm

Differential Revision: D6501992

fbshipit-source-id: 611e26b3cfeef01eb8d3a3500ae3ebc26bee6e86

3 years agoFix UBSAN failure in RangeTest.cpp with libstdc++-v4.9
Yedidya Feldblum [Sat, 9 Dec 2017 17:47:06 +0000 (09:47 -0800)]
Fix UBSAN failure in RangeTest.cpp with libstdc++-v4.9

Summary:
[Folly] Fix UBSAN failure in `RangeTest.cpp` with `libstdc++-v4.9`.

The implementation of `std::array` in `libstdc++-v4.9` has undefined behavior in the case of the zero-length array - dereferencing (but immediately taking the address again) of `nullptr`. This is fixed in `libstdc++-v5`.

Reviewed By: ot

Differential Revision: D6528614

fbshipit-source-id: 26be6bdc28d9e6dc6e6293a4d4ecffd769c01413

3 years agoreplace std::dequeue with UMPMCQueue in UnboundedBlockingQueue
Yedidya Feldblum [Sat, 9 Dec 2017 05:11:03 +0000 (21:11 -0800)]
replace std::dequeue with UMPMCQueue in UnboundedBlockingQueue

Summary: As above. Thanks magedm for giving us this beautiful piece of equipment!

Reviewed By: magedm

Differential Revision: D6488661

fbshipit-source-id: 95aa9646ca1ea937bb1d055e9baa037896c3161e

3 years agoUnboundedQueue: Add LgAlign template parameter - Refactor code
Maged Michael [Fri, 8 Dec 2017 17:25:57 +0000 (09:25 -0800)]
UnboundedQueue: Add LgAlign template parameter - Refactor code

Summary:
- Add a template parameter, LgAlign, to control memory usage. The parameter is used in DynamicBoundedQueue.
- Refactor code.

Reviewed By: yfeldblum

Differential Revision: D6508015

fbshipit-source-id: 6e17b1d8fd900595147dc4217e04d379a13fbdf8

3 years agoMove folly/Bits.h to folly/lang/
Yedidya Feldblum [Fri, 8 Dec 2017 08:24:44 +0000 (00:24 -0800)]
Move folly/Bits.h to folly/lang/

Summary: [Folly] Move `folly/Bits.h` to `folly/lang/`.

Reviewed By: phoad, Orvid

Differential Revision: D6495547

fbshipit-source-id: a93159321df8277f8a4b4f10a5e4e0fc58cb6022

3 years agoprevent folly::Function<T const&()> from being initialized with a function that retur...
Eric Niebler [Fri, 8 Dec 2017 06:36:49 +0000 (22:36 -0800)]
prevent folly::Function<T const&()> from being initialized with a function that returns a prvalue

Summary: If the folly::Function returns by const& and the wrapped function returns a prvalue, it is guaranteed to return a dangling reference. Prevent it.

Reviewed By: yfeldblum, aary

Differential Revision: D6487447

fbshipit-source-id: 61700b4688e29409eefa27f546b31ecac258cfdd

3 years agologging: use raw string literals in config tests
Adam Simpkins [Fri, 8 Dec 2017 03:24:34 +0000 (19:24 -0800)]
logging: use raw string literals in config tests

Summary:
Update string literals with escaped quotes and backslashes to use raw string
literal syntax instead.

Reviewed By: yfeldblum

Differential Revision: D6501426

fbshipit-source-id: 557366f86b360596b13148d03fd33c7df5c3f890

3 years agoFlatCombining: Use SaturatingSemaphore instead of multi-poster and non-blocking Baton
Maged Michael [Thu, 7 Dec 2017 22:41:53 +0000 (14:41 -0800)]
FlatCombining: Use SaturatingSemaphore instead of multi-poster and non-blocking Baton

Summary: Use SaturatingSemaphore instead of multi-poster and non-blocking Baton, in preparation for restoring Baton to being single-poster and with blocking wait().

Reviewed By: davidtgoldblatt

Differential Revision: D6510619

fbshipit-source-id: eed87beabc1b698080b8e41c746d570d30804ce7

3 years agoFix crash in exception_wrapper::get_exception<>
Yedidya Feldblum [Thu, 7 Dec 2017 20:38:59 +0000 (12:38 -0800)]
Fix crash in exception_wrapper::get_exception<>

Summary:
[Folly] Fix crash in `exception_wrapper::get_exception<>`.

When the contained object is unthrown and does not inherit `std::exception`, `get_exception` templated over a type that does not inherit `std::exception` must throw and catch internally and may then return a pointer to the internally thrown object, which has since been deallocated.

Attempting to dereference that pointer is undefined behavior is correctly caught by ASAN as heap-use-after-free.

Fix it by storing objects not inheriting `std::exception` using only the `std::exception_ptr` representation.

As a downside, we no longer get the small-object optimization or the non-throwing optimization for objects which do not inherit `std::exception`. But this is not likely to be terribly concerning in practice.

Reviewed By: ericniebler

Differential Revision: D6504911

fbshipit-source-id: 0065de911733b5cab87be55e7e4e47f0a9c09140

3 years agoUse simpler tags for ctor dispatch in exception_wrapper
Yedidya Feldblum [Thu, 7 Dec 2017 20:38:57 +0000 (12:38 -0800)]
Use simpler tags for ctor dispatch in exception_wrapper

Summary:
[Folly] Use simpler tags for ctor dispatch in `exception_wrapper`.

The tags do not need to be `std::integral_constant`, but can just be arbitrary empty types.

Also format the usage sites of the tags.

Reviewed By: ericniebler

Differential Revision: D6506417

fbshipit-source-id: 377359086d94fc1a8bf8eebf6f8b058ff544829f

3 years agoFix DCHECKs in IOBufQueue
Stepan Palamarchuk [Thu, 7 Dec 2017 15:43:57 +0000 (07:43 -0800)]
Fix DCHECKs in IOBufQueue

Summary: glog actually tries to pull srtings from these pointers when the checks fail, which obfuscates the error

Reviewed By: yfeldblum

Differential Revision: D6506886

fbshipit-source-id: 0283e02aecaa248b84fca1433d3f29f92c37b2e6

3 years agoRemoved dead IsSomeVector
Mehdi Bouaziz [Thu, 7 Dec 2017 10:57:26 +0000 (02:57 -0800)]
Removed dead IsSomeVector

Summary: deadcode

Reviewed By: ddrcoder

Differential Revision: D6498754

fbshipit-source-id: 9dda0ade6928d648b8938a06b185e430753f2318

3 years agoFix SignalHandlerTest with ASAN
Andrii Grynenko [Thu, 7 Dec 2017 01:50:03 +0000 (17:50 -0800)]
Fix SignalHandlerTest with ASAN

Reviewed By: yfeldblum

Differential Revision: D6503762

fbshipit-source-id: f82b88c401979761053004eadd2e4c6a74f05f9c

3 years agologging: allow partial updates to log handler settings
Adam Simpkins [Thu, 7 Dec 2017 01:29:32 +0000 (17:29 -0800)]
logging: allow partial updates to log handler settings

Summary:
This updates the LogHandlerConfig code to allow changing the settings on an
existing log handler without listing all of its existing options from scratch.

This also changes the syntax of the basic log handler configuration string to
use a colon to separate the log handler name+type from the options list.  In
other words, rather than specifying `default=stream,stream=stderr,async=true`
you now say `default=stream:stream=stderr,async=true`.

The primary motivation for this change is to make it easy for users to switch
the async setting for the default log handler on or off.  Callers can now
specify `default:async=true` to easily enable async mode on the default log
handler without having to completely re-list the full settings for the default
handler.

Reviewed By: yfeldblum

Differential Revision: D6494228

fbshipit-source-id: 52a296f800a5456f0c3aa10546298139c8db52fc

3 years agologging: split FileHandlerFactory into two classes
Adam Simpkins [Thu, 7 Dec 2017 01:29:30 +0000 (17:29 -0800)]
logging: split FileHandlerFactory into two classes

Summary:
Split FileHandlerFactory into separate FileHandlerFactory and
StreamHandlerFactory classes, where FileHandlerFactory only handles logging to
files by path name, and StreamHandlerFactory only supports logging to stdout or
stderr.

The primary motivation for this is to allow logging to stdout or stderr in some
cases without allowing arbitrary files to be opened by FileHandlerFactory.
This can be achieved now by registering StreamHandlerFactory but not
FileHandlerFactory.  This makes it safer to allow controlling logging
configuration via command line flags even in setuid binaries.

Reviewed By: yfeldblum

Differential Revision: D6494226

fbshipit-source-id: a3ec371ca4266424d07dff20be18e6e13c057593

3 years agologging: add a StandardLogHandlerFactory helper class
Adam Simpkins [Thu, 7 Dec 2017 01:29:27 +0000 (17:29 -0800)]
logging: add a StandardLogHandlerFactory helper class

Summary:
This moves some of the FileHandlerFactory code out into a new
StandardLogHandlerFactory helper class.  This will make it easier in the future
to add new LogHandlerFactory implementations that create StandardLogHandler
objects.

In particular, I plan to use this soon to split FileHandlerFactory into two
separate classes: one for writing to files on disk and a separate class for
writing to stdout or stderr.

Reviewed By: yfeldblum

Differential Revision: D6494227

fbshipit-source-id: 52e24250d020d21a5395d2a68fa5bd40bb32fbd4

3 years agofix ubsan-detected UMR in TemporaryFile
Louis Brandy [Thu, 7 Dec 2017 01:10:31 +0000 (17:10 -0800)]
fix ubsan-detected UMR in TemporaryFile

Summary:
Move constructors are just like any other constructors. `this` is not in a valid state. You aren't supposed to destroy `this` before move-constructing it.

This was detected by ubsan stress runs.

UBSAN report:

```
[ RUN      ] TemporaryFile.moveCtor
folly/experimental/TestUtil.cpp:91:20: runtime error: load of value 3, which is not a valid value for type 'bool'
 [ failed: 1 ]
 ````

Reviewed By: yfeldblum

Differential Revision: D6501500

fbshipit-source-id: 83bd0525ac2c1f5951f70f5596034bb6cb0ca985

3 years agoRename Baton::timed_wait to try_wait_for and try_wait_until
Yedidya Feldblum [Wed, 6 Dec 2017 19:36:23 +0000 (11:36 -0800)]
Rename Baton::timed_wait to try_wait_for and try_wait_until

Summary:
[Folly] Rename `Baton::timed_wait` to `try_wait_for` and `try_wait_until`.

For consistency with `std::timed_mutex`'s suite of `lock` member functions, as well as `SaturatingSemaphore` and others.

Reviewed By: davidtgoldblatt

Differential Revision: D6493714

fbshipit-source-id: 566c6d5dc0f6db4a0308cbd1972d53794f898383

3 years agoMove folly/Baton.h to folly/synchronization/
Yedidya Feldblum [Wed, 6 Dec 2017 06:03:40 +0000 (22:03 -0800)]
Move folly/Baton.h to folly/synchronization/

Summary: [Folly] Move `folly/Baton.h` to `folly/synchronization/`.

Reviewed By: phoad, Orvid

Differential Revision: D6490282

fbshipit-source-id: 66e2d25ffe3275d576b97b81c1987709000f6649

3 years agoUse boost::filesystem::current_path in ChangeToTempDir
Yedidya Feldblum [Wed, 6 Dec 2017 05:00:02 +0000 (21:00 -0800)]
Use boost::filesystem::current_path in ChangeToTempDir

Summary:
[Folly] Use `boost::filesystem::current_path` in `ChangeToTempDir`.

It will report failures noisily.

Reviewed By: pixelb

Differential Revision: D6493243

fbshipit-source-id: 423dc0e3a46781e9af42fee69060d31085f1a7c6

3 years agoexclude Unix Domain Socket from enableTTLBANotifications
Liang Zhu [Wed, 6 Dec 2017 00:45:14 +0000 (16:45 -0800)]
exclude Unix Domain Socket from enableTTLBANotifications

Reviewed By: yfeldblum

Differential Revision: D6482348

fbshipit-source-id: 5b2d3c984b761a54d5a9eba48953c556ed22cda6

3 years agoFix RequestContext held too long issue in EventBase
Pingjia Shan [Tue, 5 Dec 2017 21:03:44 +0000 (13:03 -0800)]
Fix RequestContext held too long issue in EventBase

Summary:
Upon debugging for the attached task, it appears to me the problem are in two places:
. After a callback has been run, context_ wasn't reset
. After a callback has been canceled, context_ wasn't reset

In this diff:
. Fix these two places.
. Updating unit tests to cover these two cases.

Reviewed By: yfeldblum

Differential Revision: D6465788

fbshipit-source-id: 85b3b29dc80c9f3971c85f302385d41ded44fa0e

3 years agoSplit get_default() into two for deferred default construction and added forwarding...
Aaryaman Sagar [Tue, 5 Dec 2017 18:54:55 +0000 (10:54 -0800)]
Split get_default() into two for deferred default construction and added forwarding to speed up default element construction with rvalues

Summary:
As it stood currently folly::get_default() would unnecessarily
construct a value into the third parameter, which was unnecessary in the code
path where the element was found in the map.  Also the default value can be
forwarded to the return type in the code path where the element is not found
and an rvalue parameter is passed as the default value

Reviewed By: yfeldblum

Differential Revision: D6390315

fbshipit-source-id: ef692b827d5a36751b4eb1e12042869e8fbba2e5

3 years agobitreverse
Dave Watson [Tue, 5 Dec 2017 17:14:34 +0000 (09:14 -0800)]
bitreverse

Summary:
Adds a bitreverse function to Bits.h

AFAIK there is no great way to do this on x86, takes ~4-5ns.
arm has a single instruction we can drop in in the future.

Reviewed By: yfeldblum

Differential Revision: D6459283

fbshipit-source-id: 129db196b2fac95386f601ae57843aa87523b915

3 years agoexclude Unix Domain Socket from setErrMessageCB
Liang Zhu [Tue, 5 Dec 2017 11:49:11 +0000 (03:49 -0800)]
exclude Unix Domain Socket from setErrMessageCB

Summary:
In the latest stable kernel 4.14.3 as of 2017-12-04, unix domain socket does not support MSG_ERRQUEUE. So recvmsg(MSG_ERRQUEUE) will read application data from unix doamin socket as error message, which breaks the message flow in application. This diff disable setErrMessageCB for Unix Domain Socket.

Both [[ https://github.com/torvalds/linux/blob/master/net/ipv4/tcp.c#L1782 | tcp_recvmsg ]] and [[ https://github.com/torvalds/linux/blob/master/net/ipv4/udp.c#L1571 | udp_recvmsg ]] will check flag MSG_ERRQUEUE and read from error queue.  However, there is nothing about MSG_ERRQUEUE in [[ https://github.com/torvalds/linux/blob/master/net/unix/af_unix.c#L2249 | af_unix ]].

Reviewed By: yfeldblum

Differential Revision: D6479465

fbshipit-source-id: eba2c8650e96466f2b361a42ddf90053d65f19bd

3 years agoAdd missing check against SemiFuture in FutureBase ctor
Yedidya Feldblum [Tue, 5 Dec 2017 04:57:02 +0000 (20:57 -0800)]
Add missing check against SemiFuture in FutureBase ctor

Summary: [Folly] Add missing check against `SemiFuture` in `FutureBase` ctor.

Reviewed By: LeeHowes

Differential Revision: D6469021

fbshipit-source-id: 0ca31461afee4bf64725e065a8cb4d5e933d2013

3 years agobetter error handling in MemoryIdler for inside jails
Nathan Bronson [Tue, 5 Dec 2017 00:39:00 +0000 (16:39 -0800)]
better error handling in MemoryIdler for inside jails

Summary:
/proc may be unavailable in restricted environments, which can
cause pthread_getattr_np to fail.  This diff adds checking to the return
code from that function.

Reviewed By: yfeldblum

Differential Revision: D6473789

fbshipit-source-id: de016d32f29edf8410808dfb491987692f39e768

3 years agoImprove QueueAppender/IOBufQueue performance
Stepan Palamarchuk [Mon, 4 Dec 2017 23:08:24 +0000 (15:08 -0800)]
Improve QueueAppender/IOBufQueue performance

Summary:
Currently QueueAppender needs to follow a chain of 4 indirections (QueueAppender->IOBufQueue->IOBuf(head)->IOBuf(tail)->data).
This diff adds a cache of writable tail range in IOBufQueue and allows it to be placed externally.

Before this diff on hot path QueueAppender::write<signed char> was ~167 bytes of code (with majority being actually executed), after this diff it's down to ~30 bytes:
  0x0000000000419d10 <+0>:     mov    (%rdi),%rax
  0x0000000000419d13 <+3>:     cmp    %rax,0x8(%rdi)
  0x0000000000419d17 <+7>:     je     0x419d28 <folly::io::QueueAppender::write<signed char>(signed char)+24>
  0x0000000000419d19 <+9>:     mov    %sil,(%rax)
  0x0000000000419d1c <+12>:    addq   $0x1,(%rdi)
  0x0000000000419d20 <+16>:    retq
  0x0000000000419d21 <+17>:    nopl   0x0(%rax)
  0x0000000000419d28 <+24>:    movsbl %sil,%esi
  0x0000000000419d2c <+28>:    jmpq   0x419ca0 <folly::io::QueueAppender::writeSlow<signed char>(signed char)>

With this diff, Thrift serialization performance is improved up to 2x with production workloads (2x for compact, 3x for binary).

Thrift benchmark output:
Before:
  ============================================================================
  thrift/lib/cpp2/test/ProtocolBench.cpp          relative  time/iter  iters/s
  ============================================================================
  BinaryProtocol_write_Empty                                  58.05ns   17.23M
  BinaryProtocol_write_SmallInt                               75.17ns   13.30M
  BinaryProtocol_write_BigInt                                 74.60ns   13.41M
  BinaryProtocol_write_SmallString                            85.12ns   11.75M
  BinaryProtocol_write_BigString                             802.96ns    1.25M
  BinaryProtocol_write_BigBinary                             174.69ns    5.72M
  BinaryProtocol_write_LargeBinary                           171.81ns    5.82M
  BinaryProtocol_write_Mixed                                 130.97ns    7.64M
  BinaryProtocol_write_SmallListInt                          123.99ns    8.06M
  BinaryProtocol_write_BigListInt                             40.72us   24.56K
  BinaryProtocol_write_BigListMixed                          784.78us    1.27K
  BinaryProtocol_write_LargeListMixed                         98.84ms    10.12
  CompactProtocol_write_Empty                                 64.38ns   15.53M
  CompactProtocol_write_SmallInt                              76.74ns   13.03M
  CompactProtocol_write_BigInt                                83.62ns   11.96M
  CompactProtocol_write_SmallString                           86.05ns   11.62M
  CompactProtocol_write_BigString                            786.18ns    1.27M
  CompactProtocol_write_BigBinary                            184.91ns    5.41M
  CompactProtocol_write_LargeBinary                          182.12ns    5.49M
  CompactProtocol_write_Mixed                                120.89ns    8.27M
  CompactProtocol_write_SmallListInt                         119.74ns    8.35M
  CompactProtocol_write_BigListInt                            43.76us   22.85K
  CompactProtocol_write_BigListMixed                         595.90us    1.68K
  CompactProtocol_write_LargeListMixed                        72.80ms    13.74
  ============================================================================
After:
  ============================================================================
  thrift/lib/cpp2/test/ProtocolBench.cpp          relative  time/iter  iters/s
  ============================================================================
  BinaryProtocol_write_Empty                                  65.97ns   15.16M
  BinaryProtocol_write_SmallInt                               72.31ns   13.83M
  BinaryProtocol_write_BigInt                                 72.67ns   13.76M
  BinaryProtocol_write_SmallString                            77.56ns   12.89M
  BinaryProtocol_write_BigString                             782.07ns    1.28M
  BinaryProtocol_write_BigBinary                             179.69ns    5.57M
  BinaryProtocol_write_LargeBinary                           182.62ns    5.48M
  BinaryProtocol_write_Mixed                                  91.62ns   10.92M
  BinaryProtocol_write_SmallListInt                           96.22ns   10.39M
  BinaryProtocol_write_BigListInt                             19.65us   50.90K
  BinaryProtocol_write_BigListMixed                          245.69us    4.07K
  BinaryProtocol_write_LargeListMixed                         46.56ms    21.48
  CompactProtocol_write_Empty                                 74.44ns   13.43M
  CompactProtocol_write_SmallInt                              80.35ns   12.45M
  CompactProtocol_write_BigInt                                85.30ns   11.72M
  CompactProtocol_write_SmallString                           82.61ns   12.10M
  CompactProtocol_write_BigString                            784.77ns    1.27M
  CompactProtocol_write_BigBinary                            193.20ns    5.18M
  CompactProtocol_write_LargeBinary                          192.53ns    5.19M
  CompactProtocol_write_Mixed                                 99.78ns   10.02M
  CompactProtocol_write_SmallListInt                         104.77ns    9.54M
  CompactProtocol_write_BigListInt                            25.62us   39.03K
  CompactProtocol_write_BigListMixed                         272.42us    3.67K
  CompactProtocol_write_LargeListMixed                        38.32ms    26.09
  ============================================================================

QueueAppender Benchmark output (although not very representative due to a tight loop):
Before:
  ============================================================================
  folly/io/test/QueueAppenderBenchmark.cpp        relative  time/iter  iters/s
  ============================================================================
  write_uint8                                                 10.50us   95.20K
  write_uint16                                                 5.48us  182.49K
  write_uint32                                                 2.73us  366.22K
  push_64b                                                     9.77us  102.36K
  push_1024b                                                 112.87us    8.86K
  append                                                      64.21us   15.57K
  preallocate_postallocate_1b                                 16.34us   61.19K
  preallocate_postallocate_4b                                 15.56us   64.26K
  preallocate_postallocate_32b                                22.17us   45.11K
  preallocate_postallocate_256b                              149.55us    6.69K
  ============================================================================

After:
  ============================================================================
  folly/io/test/QueueAppenderBenchmark.cpp        relative  time/iter  iters/s
  ============================================================================
  write_uint8                                                  8.86us  112.81K
  write_uint16                                                 3.91us  255.68K
  write_uint32                                                 2.08us  481.78K
  push_64b                                                     8.24us  121.39K
  push_1024b                                                 115.50us    8.66K
  append                                                      67.52us   14.81K
  preallocate_postallocate_1b                                 13.86us   72.17K
  preallocate_postallocate_4b                                 11.67us   85.71K
  preallocate_postallocate_32b                                20.35us   49.14K
  preallocate_postallocate_256b                              148.57us    6.73K
  ============================================================================

Reviewed By: yfeldblum

Differential Revision: D6427749

fbshipit-source-id: 8495cc74b6106b15d201e37533ae4c0a1abc9d74

3 years agoadd AtFork::unregisterHandler
Dave Watson [Mon, 4 Dec 2017 20:57:07 +0000 (12:57 -0800)]
add AtFork::unregisterHandler

Summary: Adds an AdFork::unregisterHandler command, such that we can register and unregister fork handlers.

Reviewed By: yfeldblum

Differential Revision: D6439796

fbshipit-source-id: b710152f8cf98371c330b484cd5fe42de44e74ac

3 years agoUnbounded queue
Maged Michael [Mon, 4 Dec 2017 19:07:30 +0000 (11:07 -0800)]
Unbounded queue

Summary:
UnboundedQueue supports:
- SPSC, MPSC, SCMP, MPMC
- Non-waiting, waiting, and timed consumer operations.
- Producers never wait or fail (unless out-of-memory).
- Memory usage grows and shrinks dynamically

```
/// UnboundedQueue supports a variety of options for unbounded
/// dynamically expanding an shrinking queues, including variations of:
/// - Single vs. multiple producers
/// - Single vs. multiple consumers
/// - Blocking vs. spin-waiting
/// - Non-waiting, timed, and waiting consumer operations.
/// Producer operations never wait or fail (unless out-of-memory).
///
/// Template parameters:
/// - T: element type
/// - SingleProducer: true if there can be only one producer at a
///   time.
/// - SingleConsumer: true if there can be only one consumer at a
///   time.
/// - MayBlock: true if consumers may block, false if they only
///   spins. A performance tuning parameter.
/// - LgSegmentSize (default 8): Log base 2 of number of elements per
///   segment. A performance tuning parameter. See below.
///
/// When to use UnboundedQueue:
/// - If a small bound may lead to deadlock or performance degradation
///   under bursty patterns.
/// - If there is no risk of the queue growing too much.
///
/// When not to use UnboundedQueue:
/// - If there is risk of the queue growing too much and a large bound
///   is acceptable, then use DynamicBoundedQueue.
/// - If the queue must not allocate on enqueue or it must have a
///   small bound, then use fixed-size MPMCQueue or (if non-blocking
///   SPSC) ProducerConsumerQueue.
///
/// Template Aliases:
///   USPSCQueue<T, MayBlock, LgSegmentSize>
///   UMPSCQueue<T, MayBlock, LgSegmentSize>
///   USPMCQueue<T, MayBlock, LgSegmentSize>
///   UMPMCQueue<T, MayBlock, LgSegmentSize>
///
/// Functions:
///   Producer operations never wait or fail (unless OOM)
///     void enqueue(const T&);
///     void enqueue(T&&);
///         Adds an element to the end of the queue.
///
///   Consumer operations:
///     void dequeue(T&);
///         Extracts an element from the front of the queue. Waits
///         until an element is available if needed.
///     bool try_dequeue(T&);
///         Tries to extracts an element from the front of the queue
///         if available. Returns true if successful, false otherwise.
///     bool try_dequeue_until(T&, time_point& deadline);
///         Tries to extracts an element from the front of the queue
///         if available until the specified deadline.  Returns true
///         if successful, false otherwise.
///     bool try_dequeue_for(T&, duration&);
///         Tries to extracts an element from the front of the queue
///         if available for for the specified duration.  Returns true
///         if successful, false otherwise.
///
///   Secondary functions:
///     size_t size();
///         Returns an estimate of the size of the queue.
///     bool empty();
///         Returns true only if the queue was empty during the call.
///     Note: size() and empty() are guaranteed to be accurate only if
```

Reviewed By: djwatson

Differential Revision: D6157613

fbshipit-source-id: db423f86d1d0604d22f6b9c71ea0ed08be32e2a1

3 years agoAdd wrapper for getting X509_digest from a cert
Anirudh Ramachandran [Mon, 4 Dec 2017 07:56:21 +0000 (23:56 -0800)]
Add wrapper for getting X509_digest from a cert

Summary:
X509_digest, or the "fingerprint" of the cert, is useful to identify
certificates with the same CN/SANs. This adds an API to get the SHA1 or SHA256
digest of a cert.

Reviewed By: yfeldblum

Differential Revision: D6457917

fbshipit-source-id: 148a5a2270e938b39065e00d7947c2fc57cd7f75

3 years agoStatically allocate futex array v2017.12.04.00
Alastair Daivis [Sun, 3 Dec 2017 01:35:33 +0000 (17:35 -0800)]
Statically allocate futex array

Summary:
AFAICT this is currently the only thing preventing the non-dynamic version of Folly's MPMCQueue from being allocation-free on enqueue and dequeue operations. Allocating this with a constant expression should move the allocation from runtime to link (possibly compile?) time, which will let us use it in allocation sensitive contexts.

Feel free to suggest other reviewers, I couldn't find an existing folly reviewer group.

Reviewed By: yfeldblum, agola11, nbronson

Differential Revision: D6447848

fbshipit-source-id: 86b84b19d62f1e1bcecdb9e757a6dfa90597b084

3 years agoallow fiber executor to override fiber options
David Goode [Sat, 2 Dec 2017 03:56:14 +0000 (19:56 -0800)]
allow fiber executor to override fiber options

Summary: As in title, for eg larger stack size

Reviewed By: yfeldblum

Differential Revision: D6466762

fbshipit-source-id: 303e8732c8bfb64fbd8619163779942fe5f8b5a8

3 years agoadd makeSystemError*() helper functions
Adam Simpkins [Sat, 2 Dec 2017 02:05:54 +0000 (18:05 -0800)]
add makeSystemError*() helper functions

Summary:
Add makeSystemError() helper functions that are similar to the existing
throwSystemError() functions but just return the exception rather than throwing
it.  This is helpful for callers using folly::Expected or folly::Future, where
they need to return an exception type rather than throwing it.

This also includes comments about the fact that this code is using the wrong
error category.  The C++ standard indicates that std::generic_category() for
POSIX errno values.  I am not fixing this as part of this diff, since this
change has the potential to break existing users, and would need a fair amount
of testing first.

Reviewed By: yfeldblum

Differential Revision: D6456771

fbshipit-source-id: 4724b51b8d4a7f513ae70ea1b0c4f0516cfc205f

3 years agologging: fix compilation error on older C++ compilers
Adam Simpkins [Sat, 2 Dec 2017 02:01:59 +0000 (18:01 -0800)]
logging: fix compilation error on older C++ compilers

Summary:
Add an explicit `std::move()` around the return value of
`logConfigToDynamic()`.  This explicit move is required pre-C++14,
but is not required after DR 1579:
http://www.open-std.org/jtc1/sc22/wg21/docs/cwg_defects.html#1579

Newer versions of g++ and clang no longer require the move.  However gcc
versions earlier than 5.0 do require it.

Reviewed By: yfeldblum

Differential Revision: D6466447

fbshipit-source-id: 1b15934490d4966e9b3e5612e0e2ecbb43c979ca

3 years agologging: update initialization code to use the new LogConfig logic
Adam Simpkins [Sat, 2 Dec 2017 00:53:51 +0000 (16:53 -0800)]
logging: update initialization code to use the new LogConfig logic

Summary:
Replace the initLoggingGlogStyle() function with a more generic initLogging()
function that accepts a log config string to be parsed with parseLogConfig().

Reviewed By: bolinfest, yfeldblum

Differential Revision: D6342086

fbshipit-source-id: fb1bffd11f190b70e03e2ccbf2b30be08d655242

3 years agounify FOLLY_SANITIZE existing uses and add a few new
Igor Sugak [Fri, 1 Dec 2017 19:19:44 +0000 (11:19 -0800)]
unify FOLLY_SANITIZE existing uses and add a few new

Reviewed By: yfeldblum

Differential Revision: D6453972

fbshipit-source-id: fa083c0b782188f5f45a2a354045aee03e528829

3 years agoimplement chrono conversions for unusual duration types
Adam Simpkins [Fri, 1 Dec 2017 19:18:17 +0000 (11:18 -0800)]
implement chrono conversions for unusual duration types

Summary:
Implement conversions between std::chrono::duration and POSIX-style time
structures even when neither the numerator nor the denominator of the duration
ratio are 1.

Both of these are done by first converting to an intermediate type where the
numerator is 1, and then using the conversion routines for that case.

Reviewed By: yfeldblum

Differential Revision: D6366647

fbshipit-source-id: 8f9495fb4101cac6d8b4cf0353a679107007b298

3 years agoMove threadlocal_detail::Atfork to its own file
Dave Watson [Fri, 1 Dec 2017 15:09:43 +0000 (07:09 -0800)]
Move threadlocal_detail::Atfork to its own file

Summary: As title

Reviewed By: yfeldblum

Differential Revision: D6440723

fbshipit-source-id: 3168d7bb616ae0ff3fe42f7584c5a255c4953875

3 years agofix ASAN build
Bin Liu [Thu, 30 Nov 2017 23:52:54 +0000 (15:52 -0800)]
fix ASAN build

Summary:
Fix bug in non-jemalloc build.

(Note: this ignores all push blocking failures!)

Reviewed By: igorsugak

Differential Revision: D6451342

fbshipit-source-id: 44725d4e0c685a59325fa3d08877d0f62120c5c2

3 years agoLet SaturatingSemaphore::try_wait be non-const and add ready
Yedidya Feldblum [Thu, 30 Nov 2017 22:56:07 +0000 (14:56 -0800)]
Let SaturatingSemaphore::try_wait be non-const and add ready

Summary:
[Folly] Let `SaturatingSemaphore::try_wait` be non-`const` and add `ready`.

For internal API consistency.

Reviewed By: magedm

Differential Revision: D6450089

fbshipit-source-id: 65b9b723672521710a69719b192eb2922a27b778

3 years agofix build when sanitizers are enabled and jemalloc is disabled
Igor Sugak [Thu, 30 Nov 2017 22:48:24 +0000 (14:48 -0800)]
fix build when sanitizers are enabled and jemalloc is disabled

Summary: Add preprocessor checks to guard jemalloc headers if any sanitizer is enabled.

Reviewed By: yfeldblum

Differential Revision: D6441578

fbshipit-source-id: 85364fca5af33c95cd05f232d1e3f9cbac9a4120

3 years agoSome fixes for custom conversions of enums
Yedidya Feldblum [Thu, 30 Nov 2017 06:13:39 +0000 (22:13 -0800)]
Some fixes for custom conversions of enums

Summary:
[Folly] Some fixes for custom conversions of enums.

Of note, `to` was defined for enum -> all conversions, including enum -> string conversions, but we actually want enum -> string conversions to be done via ADL-discovered toAppend.

Reviewed By: ot

Differential Revision: D6411250

fbshipit-source-id: 852b64309e6adf1c68e5153635cb29632e2d86d4

3 years agoSaturating semaphore
Maged Michael [Thu, 30 Nov 2017 05:12:16 +0000 (21:12 -0800)]
Saturating semaphore

Summary:
SaturatingSemaphore is a flag that allows:
- multiple concurrent posters
- multiple concurrent waiters
- idempotent posting
- non-destructive waiting
- blocking and spinning
- pre-block spin time control

```
/// SaturatingSemaphore is a flag that allows concurrent posting by
/// multiple posters and concurrent non-destructive waiting by
/// multiple waiters.
///
/// A SaturatingSemaphore allows one or more waiter threads to check,
/// spin, or block, indefinitely or with timeout, for a flag to be set
/// by one or more poster threads. By setting the flag, posters
/// announce to waiters (that may be already waiting or will check
/// the flag in the future) that some condition is true. Posts to an
/// already set flag are idempotent.
```

Reviewed By: djwatson

Differential Revision: D6379704

fbshipit-source-id: 59aed76caa2d159639e75425a778a9c63f18f375

3 years agoadd FOLLY_SANITIZE macro
Igor Sugak [Thu, 30 Nov 2017 02:14:04 +0000 (18:14 -0800)]
add FOLLY_SANITIZE macro

Summary: Most of the time we want to guard some code when any of the sanitizers are enabled not just one.

Reviewed By: yfeldblum

Differential Revision: D6429720

fbshipit-source-id: 7157a0a13f2b2acabd36a19c61b9fdbf38ba96b8

3 years agologging: add a LogConfig::update() method
Adam Simpkins [Thu, 30 Nov 2017 01:35:21 +0000 (17:35 -0800)]
logging: add a LogConfig::update() method

Summary:
Add a method for merging the settings from two LogConfig objects.
This allows LogConfig objects to be merged before applying them to the
LoggerDB.  The effects are the same as two sequential LoggerDB::updateConfig()
calls, but without having to apply the intermediate state to the LoggerDB.

Reviewed By: bolinfest

Differential Revision: D6342085

fbshipit-source-id: 0f8a1b7d8d195a80bc74342444dd3152d331fcb6

3 years agologging: add an assert for the FLAG_INHERIT bit
Adam Simpkins [Thu, 30 Nov 2017 01:35:19 +0000 (17:35 -0800)]
logging: add an assert for the FLAG_INHERIT bit

Summary:
Add a static_assert() to check that the FLAG_INHERIT bit does not conflict with
valid LogLevel values.

Reviewed By: bolinfest

Differential Revision: D6341242

fbshipit-source-id: cec284623c8a612a3c54f1b593d169310790616e

3 years agologging: remove comment about LogLevel::ERROR
Adam Simpkins [Thu, 30 Nov 2017 01:35:17 +0000 (17:35 -0800)]
logging: remove comment about LogLevel::ERROR

Summary:
The logging code used to support a LogLevel::ERROR value on non-Windows
platforms.  I removed it in D5288600 to make all platforms consistent, but I
forgot to remove this comment describing it.

Reviewed By: bolinfest

Differential Revision: D6341244

fbshipit-source-id: 054427d342066f354c859b12611b907dc2d4fa35

3 years agologging: add LoggerDB::updateConfig() and resetConfig()
Adam Simpkins [Thu, 30 Nov 2017 01:35:16 +0000 (17:35 -0800)]
logging: add LoggerDB::updateConfig() and resetConfig()

Summary: Add methods for applying config changes from a LogConfig object to the LoggerDB.

Reviewed By: bolinfest

Differential Revision: D6200564

fbshipit-source-id: a25eb99e84b2885bf6853e2222db0d7432a6c37b

3 years agologging: add a LoggerDB::getConfig() method
Adam Simpkins [Thu, 30 Nov 2017 01:35:13 +0000 (17:35 -0800)]
logging: add a LoggerDB::getConfig() method

Summary: Add a method to get the current LogConfig state from the LoggerDB.

Reviewed By: bolinfest

Differential Revision: D6200596

fbshipit-source-id: 3bc57d498a5d25d19099d861376d71ea9f7e4039

3 years agologging: don't clamp the log level to DFATAL in debug builds
Adam Simpkins [Thu, 30 Nov 2017 01:35:11 +0000 (17:35 -0800)]
logging: don't clamp the log level to DFATAL in debug builds

Summary:
Remove the logic that clamps the log level to DFATAL in debug builds.

This behavior logically makes some sense, but results in subtle and potentially
confusing behavior changes.  It seems confusing that after calling
`setLevel(LogLevel::MAX_LEVEL)` on a log category, calling `getLevel()` on that
object would return `DFATAL` rather than `MAX_LEVEL`.  We also weren't clamping
the level to DFATAL consistently: `setLevel()` would clamp the value, but on
construction the initial level was still set to MAX_LEVEL rather than DFATAL.
This resulted in some issues when implementing `LoggerDB::getConfig()` since we
could not consistently detect if a log category was using the default level
settings.  Rather than fix this inconsistency it seems better to simply remove
this clamping behavior.

This means that it is possible for users to disable DFATAL log messages even in
debug builds if they really want to.  Previously this was only allowed in
release builds.  This protection doesn't really seem all that
valuable--presumably most developers won't really want to do this, and if they
really do request this configuration it doesn't seem all that bad to honor it.

Reviewed By: bolinfest, yfeldblum

Differential Revision: D6200569

fbshipit-source-id: 83ef8e2e4d3b61bc5b105038cbe3132979e9ac67

3 years agologging: add a LogHandler::getConfig() method
Adam Simpkins [Thu, 30 Nov 2017 01:35:09 +0000 (17:35 -0800)]
logging: add a LogHandler::getConfig() method

Summary:
Add a method to LogHandler to return its current configuration.  This will
make it possible to query the LoggerDB for its current configuration state.

Reviewed By: bolinfest

Differential Revision: D6200563

fbshipit-source-id: 2b8b9752bbeb26c8aac28d1a73b7e2312fd198c8

3 years agologging: add a LogHandler registry to LoggerDB
Adam Simpkins [Thu, 30 Nov 2017 01:35:08 +0000 (17:35 -0800)]
logging: add a LogHandler registry to LoggerDB

Summary:
Update the LoggerDB to track a list of LogHandlers by name, and
LogHandlerFactories by handler type.

This will be needed to support updating the LoggerDB configuration from a
LogConfig object.

Reviewed By: bolinfest

Differential Revision: D6200562

fbshipit-source-id: e365b4e0df65aa5aaa34e118eb3cee9c9c45cb05

3 years agoSynchronize coupled caches in folly::threadlocal_detail::StaticMeta
Yedidya Feldblum [Wed, 29 Nov 2017 20:12:01 +0000 (12:12 -0800)]
Synchronize coupled caches in folly::threadlocal_detail::StaticMeta

Summary:
[Folly] Synchronize coupled caches in `folly::threadlocal_detail::StaticMeta`.

The caches should be set together, and only together, because they are coupled. This prevents bugs where one function that sets one cache but not the other cache is inlined into the caller in one module, and another function that reads both caches is inlined into the caller in another module.

Reviewed By: djwatson

Differential Revision: D6435175

fbshipit-source-id: 846c4972b40e525f2c04da6e6609c2ad54f674c0

3 years agologging: add a FileHandlerFactory class
Adam Simpkins [Wed, 29 Nov 2017 01:44:55 +0000 (17:44 -0800)]
logging: add a FileHandlerFactory class

Summary:
Add a new LogHandlerFactory interface for creating LogHandler objects from a
LogHandlerConfig.

Also add an initial FileHandlerFactory implementation capable of creating
LogHandler objects that write to a file descriptor.

Reviewed By: bolinfest

Differential Revision: D6200567

fbshipit-source-id: 14b86fc14ad475223aa4b57d45c40638b48c7594

3 years agoAdd unit tests for opening non-ELF files.
Andrii Nakryiko [Wed, 29 Nov 2017 00:48:14 +0000 (16:48 -0800)]
Add unit tests for opening non-ELF files.

Summary:
Just testing that ElfFile returns expected error code and message for
non-ELF files.

Depends on D6410210

Reviewed By: yfeldblum

Differential Revision: D6418365

fbshipit-source-id: aaab3b9f3ca1a12d384ae98a1772b7f640115192

3 years agoFix incorrect usages of folly::Synchronized
Sarang Masti [Tue, 28 Nov 2017 23:51:10 +0000 (15:51 -0800)]
Fix incorrect usages of folly::Synchronized

Summary:
We need to keep LockedPtr returned by rlock/wlock alive for the entire duration
of the loop. Else we are working on a snapshot of the data structure, which
might not be what we want.

Reviewed By: aary, ot, luciang

Differential Revision: D6426613

fbshipit-source-id: 294c0a7b9d756f3116d5cdb3a7b3678f693e0754

3 years agologging: convert assert() checks to FOLLY_SAFE_DCHECK()
Adam Simpkins [Tue, 28 Nov 2017 20:02:02 +0000 (12:02 -0800)]
logging: convert assert() checks to FOLLY_SAFE_DCHECK()

Summary:
Replace all assert() checks in the folly logging code with
FOLLY_SAFE_DCHECK().

Reviewed By: yfeldblum

Differential Revision: D6422885

fbshipit-source-id: 5cb12dd59f2fe5d346f15b9d317abede8654a879

3 years agoUnbreak folly on x86-32
Alexander Kabaev [Tue, 28 Nov 2017 19:42:20 +0000 (11:42 -0800)]
Unbreak folly on x86-32

Summary:
_mm_cvtsi64_si128 and _mm_cvtsi128_si64 intrinsics are available on
x86_64 only, so do not enable SSE instructions when compiling for 32 bits.

Reviewed By: yfeldblum

Differential Revision: D6422271

fbshipit-source-id: a3c5cf3f2ee4b47d5a1edcf93cfff3d1f3fe907e

3 years agoRemove some custom test main()'s in folly/experimental/symbolizer/
Yedidya Feldblum [Tue, 28 Nov 2017 06:18:54 +0000 (22:18 -0800)]
Remove some custom test main()'s in folly/experimental/symbolizer/

Summary:
[Folly] Remove some custom test `main()`'s in `folly/experimental/symbolizer/`.

In one of the two cases, we need to do some extra work to give `ElfFile` knowledge of the program path.

Reviewed By: ot, Orvid, philippv, luciang

Differential Revision: D6420517

fbshipit-source-id: 11c694a8342abb6fff05c368f3cda2e6287ccb4b

3 years agoReset context shared_ptr in AsyncTimeout::cancelTimeout()
Pingjia Shan [Tue, 28 Nov 2017 06:10:59 +0000 (22:10 -0800)]
Reset context shared_ptr in AsyncTimeout::cancelTimeout()

Summary:
This seems to fix issue in the attached task.

`context_` is set in `scheduleTimeout()` and never gets reset. So when an AsyncTimeout object is held across requests by objects like `IOThreadPool0` in I/O Thread Pool used by wangle/acceptor/ConnectionManager, RequestContext object created for the last request is leaked until `IOThreadPool0` handles another request and overrides `context_`. In the issue described in attached task, unit test has single request, next request never comes in, RequestContext doesn't end until test service stops, logging feature relying on RequestContext dtor doesn't get called in time, and thus unit test fails.

Reviewed By: yfeldblum

Differential Revision: D6402268

fbshipit-source-id: 200c6d358dfa6d7d9aa68ab05f6f1c7f4117b0ec

3 years agologging: fix unused variable warning in non-debug builds
Adam Simpkins [Tue, 28 Nov 2017 05:30:58 +0000 (21:30 -0800)]
logging: fix unused variable warning in non-debug builds

Summary:
Use FOLLY_SAFE_DCHECK() rather than assert() to avoid compiler errors about
this variable being unused in builds where assert() statements are disabled.

Reviewed By: bolinfest, yfeldblum

Differential Revision: D6421978

fbshipit-source-id: ab1019fa1fa420856e1bf3bc39c441662c4efd75

4 years agoUse "auto" with GENERATOR instead of std::function
Tom Jackson [Tue, 28 Nov 2017 01:13:37 +0000 (17:13 -0800)]
Use "auto" with GENERATOR instead of std::function

Summary: Now that we're using C++14 more broadly, we should use it to improve generator performance. This speeds up a microbenchmark >5x by removing type erasure.

Reviewed By: philippv

Differential Revision: D6398730

fbshipit-source-id: 5809058a3b5ff0e66fd4b1e8954698944e1a7d09

4 years agoRevise API to load cert/key in SSLContext.
Xiangyu Bu [Mon, 27 Nov 2017 23:37:45 +0000 (15:37 -0800)]
Revise API to load cert/key in SSLContext.

Summary:
When loading cert/key pair, order matters:
(a) Wrong key will fail to load if a cert is loaded;
(b) Wrong cert will succeed to load even if a private key is loaded.

So this diff adds:
(1) SSLContext::checkPrivateKey() -- must call for case (b).
(2) SSLContext::loadCertKeyPairFromBufferPEM() -- use this if one loads both cert and key. Guaranteed to throw if cert/key mismatch.

Reviewed By: yfeldblum

Differential Revision: D6416280

fbshipit-source-id: 8ae370883d46e9b5afb69c506c09fbf7ba82b1b9

4 years agoFix ElfFile crashing when opening short (<64 bytes) files.
Andrii Nakryiko [Mon, 27 Nov 2017 19:31:18 +0000 (11:31 -0800)]
Fix ElfFile crashing when opening short (<64 bytes) files.

Summary:
ElfFile in init() assumes file is long enough to contain complete
ElfHeader, which is 64 bytes long. This is not true for valid non-ELF files like
short scripts. They shouldn't cause crash.

Reviewed By: myreg

Differential Revision: D6410210

fbshipit-source-id: 28fd017d8de17c431d7d006a1655ade8a95994bd

4 years agoUse hazptr_local and hazptr_array
Dave Watson [Mon, 27 Nov 2017 18:50:26 +0000 (10:50 -0800)]
Use hazptr_local and hazptr_array

Summary: Use newest hazptr hotness in concurrenthashmap.  Shaves ~10% off of the single-thread find performance.

Reviewed By: magedm

Differential Revision: D6259947

fbshipit-source-id: 7ecf99d38fdf8e311fca3313137e0fca5af3f165

4 years agomake `usingJEMalloc()` a constexpr when USE_JEMALLOC is defined v2017.11.27.00
Bin Liu [Mon, 27 Nov 2017 06:07:41 +0000 (22:07 -0800)]
make `usingJEMalloc()` a constexpr when USE_JEMALLOC is defined

Summary: This would save some function calls and reduce code size if we know we will use jemalloc.

Reviewed By: yfeldblum

Differential Revision: D6411810

fbshipit-source-id: 9db471678e473945f5f00354c0a1285ecdb91de6

4 years agoAdd folly::hasher support for floating point types
Giuseppe Ottaviano [Sun, 26 Nov 2017 22:43:06 +0000 (14:43 -0800)]
Add folly::hasher support for floating point types

Summary:
Move `folly::hasher` closer to feature parity with `std::hash`.
This is in order to replace some instances of `folly::hash::hash_combine(...)` with `folly::Hash()(...)` (`std::hash` is the identity for integers, which makes it an unsafe default for more sophisticated hash data structures, including open-addressing hash tables).

The implementation is similar to `libstdc++`'s implementation, in that we handle separately the `0` case, because `0` and `-0` have different binary representations but are equal according to `operator==`, and hash the bytes otherwise. It is probably a little faster than `libstdc++`'s implementation, that delegates a out-of-line Murmur hash routine for arbitrary buffers, while this uses a minimal inlineable machine word hashing routine.

Reviewed By: yfeldblum

Differential Revision: D6410713

fbshipit-source-id: 86d9e4ed8da04fffe283949825852e539ec7d5cf

4 years agoQueuedImmediateExecutor tweaks
Yedidya Feldblum [Thu, 23 Nov 2017 19:59:32 +0000 (11:59 -0800)]
QueuedImmediateExecutor tweaks

Summary:
[Folly] `QueuedImmediateExecutor` tweaks.

* Add a leaky meyers singleton instance.
* Make the thread-local queue an instance variable. Callers which want the singleton thread-local queue can use the singleton executor instance instance, for the same effect.
* Simplify the body of `add`, and perform the thread-local lookup only once per invocation.

Reviewed By: djwatson

Differential Revision: D6399067

fbshipit-source-id: 03904885a70c4b943141bd83868414d27232fd6a

4 years agoEnable EventBase switching for AsyncSocket even if it has registered events
Fuat Geleri [Wed, 22 Nov 2017 22:20:42 +0000 (14:20 -0800)]
Enable EventBase switching for AsyncSocket even if it has registered events

Summary:
AsyncSocket will return isDetachable() -> false if there is a read callback set.
So,
 - ignore event registration status
 - unregister the events when detaching and
 - register them back when attaching again.

Reviewed By: afrind, eduardo-elizondo

Differential Revision: D6331787

fbshipit-source-id: fca4e6101f2d5666edbafa09116aa50f34cb084d

4 years agoLink to unwind when using symbolizer
Guo Xiao [Wed, 22 Nov 2017 19:19:03 +0000 (11:19 -0800)]
Link to unwind when using symbolizer

Summary:
Fix #673
Closes https://github.com/facebook/folly/pull/715

Reviewed By: meyering

Differential Revision: D6394731

Pulled By: yfeldblum

fbshipit-source-id: fb92828d0c633278601125dbc64cadf0cdbaa7fe

4 years agoAdd a fast path to folly::ThreadLocal
Dave Watson [Wed, 22 Nov 2017 16:16:04 +0000 (08:16 -0800)]
Add a fast path to folly::ThreadLocal

Summary:
Currently folly::ThreadLocal[Ptr] is pretty heavy-weight for a get():

1) call instance(), take a static init guard, branch
2) call getThreadEntry, check if thread_local is not null, branch
3) check if id < threadEntry->capacity, branch
4) Finally, return threadEntry->elements[id]

If we have real thread_locals, we can do better by caching the capacity directly,
combining all three checks:

1) checkif id < threadLocalCapacityCheck, branch.  If not, do slow path.
2) return threadEntry->elements[id].  Threadentry is never null if capacity > 0, and
    instance() setup work is called during the first getThreadEntry call when threadlocalcapacity == 0.

Reviewed By: yfeldblum

Differential Revision: D6379878

fbshipit-source-id: 4fc7564bbb2f319d65875124026aef28d910ef06

4 years agoFix ASAN failure in FutureDAG test
Andrii Grynenko [Wed, 22 Nov 2017 04:58:05 +0000 (20:58 -0800)]
Fix ASAN failure in FutureDAG test

Summary: dogscience

Reviewed By: yfeldblum

Differential Revision: D6387604

fbshipit-source-id: 3e33da3e3c6801d6e2186637c8fb6adaf1eef6f6

4 years agoAdd -Werror to -Wshadow-local and -Wshadow-compatible-local warning check
Aaron Orenstein [Tue, 21 Nov 2017 22:02:18 +0000 (14:02 -0800)]
Add -Werror to -Wshadow-local and -Wshadow-compatible-local warning check

Summary:
Without -Werror Apple LLVM will print a warning about -Wshadow-local
and -Wshadow-compatible-local but won't fail.  This causes problems for
downstream code which uses -Werror.

Reviewed By: meyering

Differential Revision: D6375771

fbshipit-source-id: 4ac3a4ff46bdb76f1f07b0c8ba479a7d404c8c9b

4 years agologging: add a LogConfig class and parsing code
Adam Simpkins [Tue, 21 Nov 2017 18:44:02 +0000 (10:44 -0800)]
logging: add a LogConfig class and parsing code

Summary:
Add a new LogConfig class to represent the current configuration state of the
LoggerDB.  This also includes code to parse config strings in both JSON and a
simpler more human-writable format (intended primarily for use in command line
arguments).

I generally expect the human-writable format to be used mainly to configure log
levels.  It also supports configuring log handler details as well, but the
format for this data is slightly more cumbersome and will probably be harder
for people to easily remember.

The parsing code is intentionally kept as part of the 'init' library rather
than the core 'logging' library so that other libraries that simply wish to log
messages do not need to depend on it.  For instance, this would allow the folly
JSON library to use the logging library without causing a circular dependency.

Reviewed By: bolinfest

Differential Revision: D6200560

fbshipit-source-id: e4e3c7f941808251b6c7bcbbdac0210118675fb0

4 years agoimprove ThreadLocalBenchmark
Dave Watson [Tue, 21 Nov 2017 15:34:15 +0000 (07:34 -0800)]
improve ThreadLocalBenchmark

Summary:
Benchmark appears to be memory bound, and is affected by different cpus.
Add a new benchmark that does more work.

Reviewed By: yfeldblum

Differential Revision: D6380904

fbshipit-source-id: 5cfbaab70379aa4a2923d957e1147d8486deeff7

4 years agofix hazptr_array move
Dave Watson [Tue, 21 Nov 2017 15:33:09 +0000 (07:33 -0800)]
fix hazptr_array move

Summary: Broken, found via build failures in D6259947

Reviewed By: yfeldblum

Differential Revision: D6374215

fbshipit-source-id: 3a438b5416683c602966c2f029c6b7787acdccaa

4 years agoRevert D6366352: [folly] Split get_default() into two for deferred default construction
Giuseppe Ottaviano [Tue, 21 Nov 2017 07:10:34 +0000 (23:10 -0800)]
Revert D6366352: [folly] Split get_default() into two for deferred default construction

Summary:
This reverts commit db55b944ca63e565997094c11b90c4ebe98531ce

bypass-lint

Differential Revision: D6366352

fbshipit-source-id: e25906409186b077ef9117aa524cc7c86314ae12

4 years agoSplit get_default() into two for deferred default construction
Aaryaman Sagar [Mon, 20 Nov 2017 23:59:34 +0000 (15:59 -0800)]
Split get_default() into two for deferred default construction

Summary:
As it stood currently folly::get_default() would unnecessarily
construct a value into the third parameter, which was unnecessary in the fast
path where the element was found in the map

Reviewed By: yfeldblum

Differential Revision: D6366352

fbshipit-source-id: db55b944ca63e565997094c11b90c4ebe98531ce

4 years agocorrect usage of namespace std for coroutines_trait specialization
Dylan Yudaken [Mon, 20 Nov 2017 23:59:02 +0000 (15:59 -0800)]
correct usage of namespace std for coroutines_trait specialization

Summary: When specializing traits you should not use the libcpp inline std namespace, but rather the plain one.

Reviewed By: yfeldblum

Differential Revision: D6373787

fbshipit-source-id: d62ef9ccbf90c6f952db5f93f2377c920e68a809

4 years agoWork around a bug in MSVC name lookup within templated friend contexts
Christopher Dykes [Mon, 20 Nov 2017 21:44:02 +0000 (13:44 -0800)]
Work around a bug in MSVC name lookup within templated friend contexts

Summary: MSVC has a bug that causes it to think that `once_flag` in this context resolves to the `using` statement further up, rather than `detail::once_flag`.

Reviewed By: yfeldblum

Differential Revision: D6376501

fbshipit-source-id: 481dbd75ed21f3d519bd920258fa743f314668ad

4 years agoNo need for a wrapping structure for posixTimeToDuration
Yedidya Feldblum [Mon, 20 Nov 2017 21:29:48 +0000 (13:29 -0800)]
No need for a wrapping structure for posixTimeToDuration

Summary:
[Folly] No need for a wrapping structure for `posixTimeToDuration`.

We can just use a variant of tag dispatch. In this variant, we pass to `posixTimeToDuration` a default-initialized value of the desired return type and we write overload templates for each possible variant. The argument is used purely for overload resolution and return type deduction, not for its runtime value. It is slightly different from tag dispatch because we do not use separate types which are purely tag types.

Reviewed By: simpkins

Differential Revision: D6371572

fbshipit-source-id: 1987dee31fceec8733caa61495e96489dbf1ca39

4 years agoAdd free-function retire
Dave Watson [Mon, 20 Nov 2017 16:18:27 +0000 (08:18 -0800)]
Add free-function retire

Summary:
Adds a hazptr_retire and domain::retire methods to hazptr.  They both allocate memory instead of inheriting.

This will make implementation of google's cell proposal trivial, vs. D4754972 which felt overwraught.

http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/p0561r0.html

Reviewed By: magedm

Differential Revision: D6361162

fbshipit-source-id: 9f259f434139f960483b2ab7c5190d3807edcf52

4 years agofix SingletonTest v2017.11.20.00
Tianjiao Yin [Mon, 20 Nov 2017 02:51:19 +0000 (18:51 -0800)]
fix SingletonTest

Summary: There is no guarantee that this code will finish in 6 seconds (nor other threads will release singleton in 5 seconds), especially in ASAN mode. Though I don't have better idea, this diff will relax the condition in ASAN mode to make the unit-test less flaky.

Reviewed By: yfeldblum

Differential Revision: D6371692

fbshipit-source-id: 58dd15cc0b3273719314c8b323ba88ee47e8ff61

4 years agoimplement to() conversions for std::chrono to timespec/timeval
Adam Simpkins [Sun, 19 Nov 2017 23:14:20 +0000 (15:14 -0800)]
implement to() conversions for std::chrono to timespec/timeval

Summary:
Add folly::to() conversions to convert between std::chrono::duration or
std::chrono::time_point types and struct timespec or struct timeval types.

To conform to the behavior of the existing arithmetic-to-arithmetic
conversions, this code performs proper overflow checking and throws a
`ConversionError` on overflow.  This unfortunately does make the code rather
complicated compared to a non-checking implementation.

Conversions between some unusual duration types is not implemented yet, and
will fail at compile time if someone tries to use it.  This happens for
durations where neither the numerator nor the denominator of the ratio is 1.
For instance, 7/13ths of a second.

Reviewed By: yfeldblum

Differential Revision: D6356700

fbshipit-source-id: 9dce8ab8f32d8c18089f32c7176a8abf3c3f11f7

4 years agoget rid of redundant calls to RequestContext::saveContext()
Pingjia Shan [Sun, 19 Nov 2017 05:55:44 +0000 (21:55 -0800)]
get rid of redundant calls to RequestContext::saveContext()

Summary:
In the past, these calls were required.

Used to solve static destruction ordering issue.  Any static object that uses RequestContext must call this function in its constructor.

That is when we were using `static folly::ThreadLocal<std::shared_ptr<RequestContext>>`, which was non-leaky.

The problem being addressed is when we have some code of the form:

```lang=c++
void doWork() {
  static EventBase eb;
}
```

But now we are using `SingletonThreadLocal<std::shared_ptr<RequestContext>>`, which is leaky.

So the issue that these calls were there to address seems to have been resolved.

Reviewed By: yfeldblum

Differential Revision: D6332597

fbshipit-source-id: c6aba6620ef2fb3a344ea20f56c8b9c0cdf42c70

4 years agofolly: support FOLLY_FALLTHROUGH on GCC
Pádraig Brady [Sat, 18 Nov 2017 22:25:54 +0000 (14:25 -0800)]
folly: support FOLLY_FALLTHROUGH on GCC

Summary:
This is required to avoid new GCC 7 -Wimplict-fallthrough warnings.
We also update to use the C++17 [[fallthrough]] attribute if supported.

Reviewed By: yfeldblum

Differential Revision: D6367140

fbshipit-source-id: d5380983cb300f944df9c2885d0faa0155994be7

4 years agoreduce the number of iteration when test SharedMutexTest in ASAN mode
Tianjiao Yin [Sat, 18 Nov 2017 14:39:09 +0000 (06:39 -0800)]
reduce the number of iteration when test SharedMutexTest in ASAN mode

Summary: This unit-test takes too long to finish in ASAN mode from heavily loaded system (more than 10 minutes).

Reviewed By: yfeldblum

Differential Revision: D6362111

fbshipit-source-id: b097eff60f88ace4fb869132598806700804e267

4 years agoRemove unnecessary white line
Pingjia Shan [Fri, 17 Nov 2017 17:40:14 +0000 (09:40 -0800)]
Remove unnecessary white line

Summary:
Seems to be causing rendering error:

{F114885894}

Created from Diffusion's 'Open in Editor' feature.

Reviewed By: lskuff

Differential Revision: D6358188

fbshipit-source-id: 3c437007b425bcadb79a71807ad92d520a70e8cb

4 years agofolly/fibers/test/FibersTest.cpp: accommodate ASAN's detect_stack_use_after_return=1
Jim Meyering [Fri, 17 Nov 2017 04:52:38 +0000 (20:52 -0800)]
folly/fibers/test/FibersTest.cpp: accommodate ASAN's detect_stack_use_after_return=1

Summary:
With ASAN enabled (actually, only with ASAN *and* its detect_stack_use_after_return=1 option),
the addTaskFinally test would fail.  This adapts to accommodate the larger stack offsets.
Also, use EXPECT_GT and EXPECT_LT rather than EXPECT_TRUE.

Reviewed By: yfeldblum

Differential Revision: D6353666

fbshipit-source-id: 39e04caffa7b24cde97c749686c7e651a071dcec

4 years agoExpose the time remaining in HHWheelTimer::Callback
Jody Ho [Thu, 16 Nov 2017 07:23:02 +0000 (23:23 -0800)]
Expose the time remaining in HHWheelTimer::Callback

Summary:
We would like to know the time remaining for a scheduled timeout to decide
whether a new event should override the scheduled timeout.

Reviewed By: yfeldblum

Differential Revision: D6334067

fbshipit-source-id: f172d5cd7fc804db5fd53a42d06cadfddf857e22

4 years agoAdd unit test for timeout=0
Subodh Iyengar [Thu, 16 Nov 2017 01:59:20 +0000 (17:59 -0800)]
Add unit test for timeout=0

Summary:
Add unit test for immediate timeout since
we're using it in a few places.

Reviewed By: yfeldblum

Differential Revision: D6327012

fbshipit-source-id: ea80763d080b06e2a43277118d3147dc8016f348

4 years agoFix wrong source name in tests Makefile
Yedidya Feldblum [Wed, 15 Nov 2017 23:52:02 +0000 (15:52 -0800)]
Fix wrong source name in tests Makefile

Summary:
[Folly] Fix wrong source name in tests `Makefile.am`.

Closes #714.

Reviewed By: Orvid

Differential Revision: D6340209

fbshipit-source-id: 88da57fb4d663071eb578efec50c7e162b0c03d1

4 years agoallow small vector to be storage for sorted_vector_map
Benny Chen [Wed, 15 Nov 2017 22:03:27 +0000 (14:03 -0800)]
allow small vector to be storage for sorted_vector_map

Summary:
this is to allow small_vector to be a storage option for sorted_vector_map. Reas
on I want to do this is because in ads there are a lot of small maps where we wo
uld have to allocate separately.

Reviewed By: yfeldblum

Differential Revision: D6318811

fbshipit-source-id: b145d1bef2cbbeb946995aa66b55aaadeb6c54f5

4 years agoadd EXPECT_THROW_RE() and EXPECT_THROW_ERRNO() test macros
Adam Simpkins [Wed, 15 Nov 2017 20:48:56 +0000 (12:48 -0800)]
add EXPECT_THROW_RE() and EXPECT_THROW_ERRNO() test macros

Summary:
Add EXPECT_THROW_RE() and EXPECT_THROW_ERRNO() macros to folly/test/TestUtils.h
These allow more precise checks than the basic EXPECT_THROW() macro provided as
part of gtest.

These macros are being moved into folly from Facebook's eden repository
(https://github.com/facebookexperimental/eden)
This will allow us to use them in folly tests and in other projects that depend
on folly.

Reviewed By: yfeldblum

Differential Revision: D6301760

fbshipit-source-id: 1f434fb5bc9b7859f763171264fb0b2e1b4bda62

4 years agofix asan build
Igor Sugak [Wed, 15 Nov 2017 06:42:28 +0000 (22:42 -0800)]
fix asan build

Reviewed By: yfeldblum

Differential Revision: D6332075

fbshipit-source-id: 26a279b6ee4253a8be8f9f63e6900c2082a5486d