folly.git
6 years agoFix zerocopy AsyncSocket memory leaks
Dan Melnic [Fri, 15 Dec 2017 03:59:27 +0000 (19:59 -0800)]
Fix zerocopy AsyncSocket memory leaks

Summary:
We currently leak ptr entries due to a missing map erase call in AsyncSocket::releaseZeroCopyBuf.
Also addZeroCopyBuf called when the bytesWritten == 0 will cause and extra id to be allocated. This will delay the deletion of the last IOBuf sent via zerocopy. The number of buffers can accumulate over time if the same bytesWritten == 0 situation is encountered.
Found when running the nbd-server zero downtime upgrade during heavy fio traffic.
Add an AsyncSocket::processZeroCopyWriteInProgress so we can drain the zerocopy msg error queue even after we set the ReadCallback to nullptr.

Reviewed By: djwatson

Differential Revision: D6552982

fbshipit-source-id: 3d2fdca84ec3b5fc46c3bed06c0c9ede66ed565a

6 years agoFix xlog docs
Chen Shen [Thu, 14 Dec 2017 23:26:53 +0000 (15:26 -0800)]
Fix xlog docs

Summary: When I was investigating the difference between glog/xlog I encountered this.

Reviewed By: yfeldblum

Differential Revision: D6572117

fbshipit-source-id: 4130deb6cfed7cfb7d2433cdab292481beb2469f

6 years agoUse explicit memory order in Baton::post
Yedidya Feldblum [Thu, 14 Dec 2017 23:07:18 +0000 (15:07 -0800)]
Use explicit memory order in Baton::post

Summary: [Folly] Use explicit memory order in `Baton::post`.

Reviewed By: djwatson

Differential Revision: D6502268

fbshipit-source-id: a379ea711f165e3645ac2127a49824f46225b80a

6 years agoMove security protocol and application protocol APIs to AsyncTransport.
Kyle Nekritz [Thu, 14 Dec 2017 19:32:57 +0000 (11:32 -0800)]
Move security protocol and application protocol APIs to AsyncTransport.

Summary: They belong here rather than AsyncTransportWrapper.

Reviewed By: siyengar

Differential Revision: D6568209

fbshipit-source-id: 0856dba0b2d527ae68d33942b28e2c485a5b1f46

6 years agoDynamic Bounded Queue
Maged Michael [Thu, 14 Dec 2017 18:33:06 +0000 (10:33 -0800)]
Dynamic Bounded Queue

Summary:
DynamicBoundedQueue supports:
- Dynamic memory usage that grows and shrink in proportion to the number of elements in the queue.
- Adjustable capacity that helps throttle pathological cases of producer-consumer imbalance that may lead to excessive memory usage.
- The adjustable capacity can also help prevent deadlock by allowing users to temporarily increase capacity substantially to guarantee accommodating producer requests that cannot wait.
- SPSC, SPMC, MPSC, MPMC variants.
- Blocking and spinning-only variants.
- Inter-operable non-waiting, timed until, timed for, and waiting variants of producer and consumer operations.
- Optional variable element weights.

Reviewed By: djwatson

Differential Revision: D6462123

fbshipit-source-id: 5c7a7d88c3fa273211702af9ff78f6302afd2276

6 years agoUnboundedQueue: Use alignas instead of FOLLY_ALIGNED
Maged Michael [Thu, 14 Dec 2017 02:54:26 +0000 (18:54 -0800)]
UnboundedQueue: Use alignas instead of FOLLY_ALIGNED

Summary: Use alignas instead of FOLLY_ALIGNED

Reviewed By: Orvid

Differential Revision: D6554984

fbshipit-source-id: 5b11ac387faa137aa8dd0c35b6e9bb999cf0bf2d

6 years agoFix AsyncSocketTest.WriteErrorCallbackBytesWritten
Yedidya Feldblum [Thu, 14 Dec 2017 02:00:55 +0000 (18:00 -0800)]
Fix AsyncSocketTest.WriteErrorCallbackBytesWritten

Summary:
[Folly] Fix `AsyncSocketTest.WriteErrorCallbackBytesWritten`.

Thanks to congestion, especially when the tests are run concurrently, the expectations in the test were too restrictive.

If only 20KB are read, it is possible that only 20KB are acknowledged. The expectation was that if 20KB are read and the recv buffer and send buffer are both 8KB and 24KB are written then all 24KB are acknowledged, but congestion control disagrees.

It is possible that any number of bytes are written to the send buffer, from 28KB up to 40KB. And we have to explicitly wait for 28KB to be written even to know that (otherwise we only know that 20KB are written).

Differential Revision: D6550804

fbshipit-source-id: 100d086972c1526b909da0dbb6e609c144d7b17b

6 years agoconstexpr_ceil
Yedidya Feldblum [Thu, 14 Dec 2017 01:08:39 +0000 (17:08 -0800)]
constexpr_ceil

Summary: [Folly] `constexpr_ceil`, an integral rounding-up util.

Reviewed By: Orvid

Differential Revision: D6558042

fbshipit-source-id: 6b42add9bf2e3605baf71391130c2a2c88cc4385

6 years agoMove max_align_v and max_align_t to folly/lang/Align.h
Yedidya Feldblum [Thu, 14 Dec 2017 00:29:37 +0000 (16:29 -0800)]
Move max_align_v and max_align_t to folly/lang/Align.h

Summary:
[Folly] Move `max_align_v` and `max_align_t` to `folly/lang/Align.h`.

And drop the macro `FOLLY_ALIGNED_MAX` - just use `alignas(max_align_v)` instead.

Reviewed By: Orvid

Differential Revision: D6554677

fbshipit-source-id: cacb335b141623506a41e95e0740613374279c6f

6 years agoRemove unused #include <string.h> from folly/Portability.h
Yedidya Feldblum [Wed, 13 Dec 2017 22:53:49 +0000 (14:53 -0800)]
Remove unused #include <string.h> from folly/Portability.h

Summary: [Folly] Remove unused `#include <string.h>` from `folly/Portability.h`.

Reviewed By: Orvid

Differential Revision: D6557374

fbshipit-source-id: 369ef0b622dc057e882ed0f4ad25b47fa3f05100

6 years agoExplicitly refer to the std::chrono namespace to avoid conflicts with the folly:...
Christopher Dykes [Wed, 13 Dec 2017 20:49:43 +0000 (12:49 -0800)]
Explicitly refer to the std::chrono namespace to avoid conflicts with the folly::chrono namespace

Summary:
This is a hard requirement due to an upcoming change in Folly.

The codemod was only run on files that referenced the `std::chrono` namespace with an unqualified `chrono` and also used both the `std` and `folly` namespaces.
```
fbgr -sl "[^:]chrono::" | xargs grep -l "using namespace std;" | xargs grep -l "using namespace folly;" | xargs sed -r -i 's/([^:])chrono::([a-z])/\1std::chrono::\2/g'
```

Reviewed By: yfeldblum

Differential Revision: D6547864

fbshipit-source-id: 4a8230d311edbaa173722a09b2773e8d053fae7e

6 years agotypo in io/Cursor.h
RcRonco [Wed, 13 Dec 2017 19:26:12 +0000 (11:26 -0800)]
typo in io/Cursor.h

Summary: Closes https://github.com/facebook/folly/pull/721

Differential Revision: D6558163

Pulled By: yfeldblum

fbshipit-source-id: ed03aa92b54c61f9980a4759da29910aa77d5db6

6 years agoRCU
Dave Watson [Wed, 13 Dec 2017 16:13:20 +0000 (08:13 -0800)]
RCU

Summary: This adds an RCU implementation, matching http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/p0461r1.pdf as closely as pratical.  This implementation does not require thread registration or quiescence.

Reviewed By: magedm

Differential Revision: D6330631

fbshipit-source-id: 2c729f3a4c0f151cde5d9a599ecd2a2c20c7da55

6 years agologging: minor clean up in Logger.h
Adam Simpkins [Wed, 13 Dec 2017 00:30:16 +0000 (16:30 -0800)]
logging: minor clean up in Logger.h

Summary:
Update the comment describing the Logger class, and remove a `clang-format off`
since recent versions of `clang-format` appear to do a good job formatting this
macro.

Reviewed By: yfeldblum

Differential Revision: D6525996

fbshipit-source-id: 6abbb3740afb53feb2418ca890cd78af1eb5f7a2

6 years agologging: add more documentation
Adam Simpkins [Wed, 13 Dec 2017 00:30:13 +0000 (16:30 -0800)]
logging: add more documentation

Summary: Add several new documents describing the behavior of the folly logging library.

Reviewed By: yfeldblum

Differential Revision: D6525997

fbshipit-source-id: 8ba2ff99c70c7566987b3da321526a16755e72bf

6 years agoSwitch to the try_wait_for and try_wait_until Baton APIs
Yedidya Feldblum [Tue, 12 Dec 2017 06:10:59 +0000 (22:10 -0800)]
Switch to the try_wait_for and try_wait_until Baton APIs

Summary: Switch to the `try_wait_for` and `try_wait_until` `Baton` APIs.

Reviewed By: davidtgoldblatt

Differential Revision: D6532103

fbshipit-source-id: aa3ce64152d167bb9c9cb1f266be0f9f8bd498f5

6 years agoStore filename and provide detailed message on data access assertion failure.
Andrii Nakryiko [Tue, 12 Dec 2017 06:05:33 +0000 (22:05 -0800)]
Store filename and provide detailed message on data access assertion failure.

Summary:
It looks like not having offset/size/filename information is way more
harmful, than storing filename just for the sake of this error message.

Reviewed By: yfeldblum

Differential Revision: D6536616

fbshipit-source-id: 469fbdf1deedd76ebd79cf98716c2c269cb10e4d

6 years agoAssorted tweaks to folly/String.h
Yedidya Feldblum [Tue, 12 Dec 2017 04:19:06 +0000 (20:19 -0800)]
Assorted tweaks to folly/String.h

Summary:
[Folly] Assorted tweaks to `folly/String.h`.

* Use `void_t` in the definition of `IsConvertible`.
* Remove `AllConvertible`. Callers can use `StrictConjunction` instead.
* Move the `static_assert`s from the header to the source file.

Reviewed By: andrewjcg, ot

Differential Revision: D6501515

fbshipit-source-id: 472ecf23bf7f06be211480b0aceea95f7e60dc21

6 years agoFix case where ssl cert does not match key
Neel Goyal [Tue, 12 Dec 2017 02:48:49 +0000 (18:48 -0800)]
Fix case where ssl cert does not match key

Summary: In some cases, SSLContextManager seg faults if a cert and key do not match.  This guards against that case when strictSSL = false, and throws a more useful error in the cases when SSL is required.

Reviewed By: xybu

Differential Revision: D6513964

fbshipit-source-id: 8e63a22b346fd3f2a30d558a3659ab6794c7a105

6 years agofolly::fibers::Baton API consistency with folly::Baton
Yedidya Feldblum [Tue, 12 Dec 2017 00:26:17 +0000 (16:26 -0800)]
folly::fibers::Baton API consistency with folly::Baton

Summary:
[Folly] `folly::fibers::Baton` API consistency with `folly::Baton`.

Specifically, the suite of `wait`, `try_wait`, `try_wait_for`, and `try_wait_until` members and member templates.

Hold onto the `timed_wait` members for now, but mark them deprecated.

Additionally, for consistency, offer main-context function params consistently for all `try_wait_for`, `try_wait_until`, and both variants of `timed_wait`.

Reviewed By: andriigrynenko

Differential Revision: D6531145

fbshipit-source-id: 960fba48716b12b0ef53262786eacab88d8b2375

6 years agoPass ZSTD_CONTENTSIZE_UNKNOWN
Nick Terrell [Tue, 12 Dec 2017 00:21:29 +0000 (16:21 -0800)]
Pass ZSTD_CONTENTSIZE_UNKNOWN

Summary:
Content size 0 as unknown is deprecated.
See code comments for details.

Reviewed By: felixhandte

Differential Revision: D6521582

fbshipit-source-id: 876b94cc0123c1020b456cc8f4cd86997afe0619

6 years agoRemove multi-poster support from Baton v2017.12.11.00
Yedidya Feldblum [Sun, 10 Dec 2017 20:39:24 +0000 (12:39 -0800)]
Remove multi-poster support from Baton

Summary:
[Folly] Remove multi-poster support from `Baton`.

Code which needs multi-poster support may use `SaturatingSemaphore` instead.

Reviewed By: magedm

Differential Revision: D6529661

fbshipit-source-id: d9dc053ca984ef3a404e9361910b0044817d4905

6 years agoFix some clang build failures on aarch64
Andrew Gallagher [Sun, 10 Dec 2017 08:55:21 +0000 (00:55 -0800)]
Fix some clang build failures on aarch64

Reviewed By: Orvid

Differential Revision: D6387412

fbshipit-source-id: 4c1424ed4fa49e69885de8cd72e78169314fc502

6 years agoUnboundedBlockingQueue: Remove extra include
Maged Michael [Sun, 10 Dec 2017 06:31:03 +0000 (22:31 -0800)]
UnboundedBlockingQueue: Remove extra include

Summary: Remove include not needed after recent change

Reviewed By: yfeldblum

Differential Revision: D6529608

fbshipit-source-id: 904072084e180125d68bd1ef6e4dffb2939cd1eb

6 years agoFix signed integer overflow in StaticTracepointTest.cpp
Yedidya Feldblum [Sun, 10 Dec 2017 01:09:24 +0000 (17:09 -0800)]
Fix signed integer overflow in StaticTracepointTest.cpp

Summary:
[Folly] Fix signed integer overflow in `StaticTracepointTest.cpp`.

It is undefined behavior and is caught by UBSAN in some test runs.

Reviewed By: Orvid

Differential Revision: D6528445

fbshipit-source-id: e4a199e9005b57bcd93c8c4b69f37ccb1e072bdf

6 years agoSplit Baton wait methods into fast and slow paths
Yedidya Feldblum [Sat, 9 Dec 2017 17:50:25 +0000 (09:50 -0800)]
Split Baton wait methods into fast and slow paths

Summary:
[Folly] Split `Baton` wait methods into fast and slow paths.

Force-inline the fast paths, and force-outline the slow paths.

Reviewed By: magedm

Differential Revision: D6501992

fbshipit-source-id: 611e26b3cfeef01eb8d3a3500ae3ebc26bee6e86

6 years agoFix UBSAN failure in RangeTest.cpp with libstdc++-v4.9
Yedidya Feldblum [Sat, 9 Dec 2017 17:47:06 +0000 (09:47 -0800)]
Fix UBSAN failure in RangeTest.cpp with libstdc++-v4.9

Summary:
[Folly] Fix UBSAN failure in `RangeTest.cpp` with `libstdc++-v4.9`.

The implementation of `std::array` in `libstdc++-v4.9` has undefined behavior in the case of the zero-length array - dereferencing (but immediately taking the address again) of `nullptr`. This is fixed in `libstdc++-v5`.

Reviewed By: ot

Differential Revision: D6528614

fbshipit-source-id: 26be6bdc28d9e6dc6e6293a4d4ecffd769c01413

6 years agoreplace std::dequeue with UMPMCQueue in UnboundedBlockingQueue
Yedidya Feldblum [Sat, 9 Dec 2017 05:11:03 +0000 (21:11 -0800)]
replace std::dequeue with UMPMCQueue in UnboundedBlockingQueue

Summary: As above. Thanks magedm for giving us this beautiful piece of equipment!

Reviewed By: magedm

Differential Revision: D6488661

fbshipit-source-id: 95aa9646ca1ea937bb1d055e9baa037896c3161e

6 years agoUnboundedQueue: Add LgAlign template parameter - Refactor code
Maged Michael [Fri, 8 Dec 2017 17:25:57 +0000 (09:25 -0800)]
UnboundedQueue: Add LgAlign template parameter - Refactor code

Summary:
- Add a template parameter, LgAlign, to control memory usage. The parameter is used in DynamicBoundedQueue.
- Refactor code.

Reviewed By: yfeldblum

Differential Revision: D6508015

fbshipit-source-id: 6e17b1d8fd900595147dc4217e04d379a13fbdf8

6 years agoMove folly/Bits.h to folly/lang/
Yedidya Feldblum [Fri, 8 Dec 2017 08:24:44 +0000 (00:24 -0800)]
Move folly/Bits.h to folly/lang/

Summary: [Folly] Move `folly/Bits.h` to `folly/lang/`.

Reviewed By: phoad, Orvid

Differential Revision: D6495547

fbshipit-source-id: a93159321df8277f8a4b4f10a5e4e0fc58cb6022

6 years agoprevent folly::Function<T const&()> from being initialized with a function that retur...
Eric Niebler [Fri, 8 Dec 2017 06:36:49 +0000 (22:36 -0800)]
prevent folly::Function<T const&()> from being initialized with a function that returns a prvalue

Summary: If the folly::Function returns by const& and the wrapped function returns a prvalue, it is guaranteed to return a dangling reference. Prevent it.

Reviewed By: yfeldblum, aary

Differential Revision: D6487447

fbshipit-source-id: 61700b4688e29409eefa27f546b31ecac258cfdd

6 years agologging: use raw string literals in config tests
Adam Simpkins [Fri, 8 Dec 2017 03:24:34 +0000 (19:24 -0800)]
logging: use raw string literals in config tests

Summary:
Update string literals with escaped quotes and backslashes to use raw string
literal syntax instead.

Reviewed By: yfeldblum

Differential Revision: D6501426

fbshipit-source-id: 557366f86b360596b13148d03fd33c7df5c3f890

6 years agoFlatCombining: Use SaturatingSemaphore instead of multi-poster and non-blocking Baton
Maged Michael [Thu, 7 Dec 2017 22:41:53 +0000 (14:41 -0800)]
FlatCombining: Use SaturatingSemaphore instead of multi-poster and non-blocking Baton

Summary: Use SaturatingSemaphore instead of multi-poster and non-blocking Baton, in preparation for restoring Baton to being single-poster and with blocking wait().

Reviewed By: davidtgoldblatt

Differential Revision: D6510619

fbshipit-source-id: eed87beabc1b698080b8e41c746d570d30804ce7

6 years agoFix crash in exception_wrapper::get_exception<>
Yedidya Feldblum [Thu, 7 Dec 2017 20:38:59 +0000 (12:38 -0800)]
Fix crash in exception_wrapper::get_exception<>

Summary:
[Folly] Fix crash in `exception_wrapper::get_exception<>`.

When the contained object is unthrown and does not inherit `std::exception`, `get_exception` templated over a type that does not inherit `std::exception` must throw and catch internally and may then return a pointer to the internally thrown object, which has since been deallocated.

Attempting to dereference that pointer is undefined behavior is correctly caught by ASAN as heap-use-after-free.

Fix it by storing objects not inheriting `std::exception` using only the `std::exception_ptr` representation.

As a downside, we no longer get the small-object optimization or the non-throwing optimization for objects which do not inherit `std::exception`. But this is not likely to be terribly concerning in practice.

Reviewed By: ericniebler

Differential Revision: D6504911

fbshipit-source-id: 0065de911733b5cab87be55e7e4e47f0a9c09140

6 years agoUse simpler tags for ctor dispatch in exception_wrapper
Yedidya Feldblum [Thu, 7 Dec 2017 20:38:57 +0000 (12:38 -0800)]
Use simpler tags for ctor dispatch in exception_wrapper

Summary:
[Folly] Use simpler tags for ctor dispatch in `exception_wrapper`.

The tags do not need to be `std::integral_constant`, but can just be arbitrary empty types.

Also format the usage sites of the tags.

Reviewed By: ericniebler

Differential Revision: D6506417

fbshipit-source-id: 377359086d94fc1a8bf8eebf6f8b058ff544829f

6 years agoFix DCHECKs in IOBufQueue
Stepan Palamarchuk [Thu, 7 Dec 2017 15:43:57 +0000 (07:43 -0800)]
Fix DCHECKs in IOBufQueue

Summary: glog actually tries to pull srtings from these pointers when the checks fail, which obfuscates the error

Reviewed By: yfeldblum

Differential Revision: D6506886

fbshipit-source-id: 0283e02aecaa248b84fca1433d3f29f92c37b2e6

6 years agoRemoved dead IsSomeVector
Mehdi Bouaziz [Thu, 7 Dec 2017 10:57:26 +0000 (02:57 -0800)]
Removed dead IsSomeVector

Summary: deadcode

Reviewed By: ddrcoder

Differential Revision: D6498754

fbshipit-source-id: 9dda0ade6928d648b8938a06b185e430753f2318

6 years agoFix SignalHandlerTest with ASAN
Andrii Grynenko [Thu, 7 Dec 2017 01:50:03 +0000 (17:50 -0800)]
Fix SignalHandlerTest with ASAN

Reviewed By: yfeldblum

Differential Revision: D6503762

fbshipit-source-id: f82b88c401979761053004eadd2e4c6a74f05f9c

6 years agologging: allow partial updates to log handler settings
Adam Simpkins [Thu, 7 Dec 2017 01:29:32 +0000 (17:29 -0800)]
logging: allow partial updates to log handler settings

Summary:
This updates the LogHandlerConfig code to allow changing the settings on an
existing log handler without listing all of its existing options from scratch.

This also changes the syntax of the basic log handler configuration string to
use a colon to separate the log handler name+type from the options list.  In
other words, rather than specifying `default=stream,stream=stderr,async=true`
you now say `default=stream:stream=stderr,async=true`.

The primary motivation for this change is to make it easy for users to switch
the async setting for the default log handler on or off.  Callers can now
specify `default:async=true` to easily enable async mode on the default log
handler without having to completely re-list the full settings for the default
handler.

Reviewed By: yfeldblum

Differential Revision: D6494228

fbshipit-source-id: 52a296f800a5456f0c3aa10546298139c8db52fc

6 years agologging: split FileHandlerFactory into two classes
Adam Simpkins [Thu, 7 Dec 2017 01:29:30 +0000 (17:29 -0800)]
logging: split FileHandlerFactory into two classes

Summary:
Split FileHandlerFactory into separate FileHandlerFactory and
StreamHandlerFactory classes, where FileHandlerFactory only handles logging to
files by path name, and StreamHandlerFactory only supports logging to stdout or
stderr.

The primary motivation for this is to allow logging to stdout or stderr in some
cases without allowing arbitrary files to be opened by FileHandlerFactory.
This can be achieved now by registering StreamHandlerFactory but not
FileHandlerFactory.  This makes it safer to allow controlling logging
configuration via command line flags even in setuid binaries.

Reviewed By: yfeldblum

Differential Revision: D6494226

fbshipit-source-id: a3ec371ca4266424d07dff20be18e6e13c057593

6 years agologging: add a StandardLogHandlerFactory helper class
Adam Simpkins [Thu, 7 Dec 2017 01:29:27 +0000 (17:29 -0800)]
logging: add a StandardLogHandlerFactory helper class

Summary:
This moves some of the FileHandlerFactory code out into a new
StandardLogHandlerFactory helper class.  This will make it easier in the future
to add new LogHandlerFactory implementations that create StandardLogHandler
objects.

In particular, I plan to use this soon to split FileHandlerFactory into two
separate classes: one for writing to files on disk and a separate class for
writing to stdout or stderr.

Reviewed By: yfeldblum

Differential Revision: D6494227

fbshipit-source-id: 52e24250d020d21a5395d2a68fa5bd40bb32fbd4

6 years agofix ubsan-detected UMR in TemporaryFile
Louis Brandy [Thu, 7 Dec 2017 01:10:31 +0000 (17:10 -0800)]
fix ubsan-detected UMR in TemporaryFile

Summary:
Move constructors are just like any other constructors. `this` is not in a valid state. You aren't supposed to destroy `this` before move-constructing it.

This was detected by ubsan stress runs.

UBSAN report:

```
[ RUN      ] TemporaryFile.moveCtor
folly/experimental/TestUtil.cpp:91:20: runtime error: load of value 3, which is not a valid value for type 'bool'
 [ failed: 1 ]
 ````

Reviewed By: yfeldblum

Differential Revision: D6501500

fbshipit-source-id: 83bd0525ac2c1f5951f70f5596034bb6cb0ca985

6 years agoRename Baton::timed_wait to try_wait_for and try_wait_until
Yedidya Feldblum [Wed, 6 Dec 2017 19:36:23 +0000 (11:36 -0800)]
Rename Baton::timed_wait to try_wait_for and try_wait_until

Summary:
[Folly] Rename `Baton::timed_wait` to `try_wait_for` and `try_wait_until`.

For consistency with `std::timed_mutex`'s suite of `lock` member functions, as well as `SaturatingSemaphore` and others.

Reviewed By: davidtgoldblatt

Differential Revision: D6493714

fbshipit-source-id: 566c6d5dc0f6db4a0308cbd1972d53794f898383

6 years agoMove folly/Baton.h to folly/synchronization/
Yedidya Feldblum [Wed, 6 Dec 2017 06:03:40 +0000 (22:03 -0800)]
Move folly/Baton.h to folly/synchronization/

Summary: [Folly] Move `folly/Baton.h` to `folly/synchronization/`.

Reviewed By: phoad, Orvid

Differential Revision: D6490282

fbshipit-source-id: 66e2d25ffe3275d576b97b81c1987709000f6649

6 years agoUse boost::filesystem::current_path in ChangeToTempDir
Yedidya Feldblum [Wed, 6 Dec 2017 05:00:02 +0000 (21:00 -0800)]
Use boost::filesystem::current_path in ChangeToTempDir

Summary:
[Folly] Use `boost::filesystem::current_path` in `ChangeToTempDir`.

It will report failures noisily.

Reviewed By: pixelb

Differential Revision: D6493243

fbshipit-source-id: 423dc0e3a46781e9af42fee69060d31085f1a7c6

6 years agoexclude Unix Domain Socket from enableTTLBANotifications
Liang Zhu [Wed, 6 Dec 2017 00:45:14 +0000 (16:45 -0800)]
exclude Unix Domain Socket from enableTTLBANotifications

Reviewed By: yfeldblum

Differential Revision: D6482348

fbshipit-source-id: 5b2d3c984b761a54d5a9eba48953c556ed22cda6

6 years agoFix RequestContext held too long issue in EventBase
Pingjia Shan [Tue, 5 Dec 2017 21:03:44 +0000 (13:03 -0800)]
Fix RequestContext held too long issue in EventBase

Summary:
Upon debugging for the attached task, it appears to me the problem are in two places:
. After a callback has been run, context_ wasn't reset
. After a callback has been canceled, context_ wasn't reset

In this diff:
. Fix these two places.
. Updating unit tests to cover these two cases.

Reviewed By: yfeldblum

Differential Revision: D6465788

fbshipit-source-id: 85b3b29dc80c9f3971c85f302385d41ded44fa0e

6 years agoSplit get_default() into two for deferred default construction and added forwarding...
Aaryaman Sagar [Tue, 5 Dec 2017 18:54:55 +0000 (10:54 -0800)]
Split get_default() into two for deferred default construction and added forwarding to speed up default element construction with rvalues

Summary:
As it stood currently folly::get_default() would unnecessarily
construct a value into the third parameter, which was unnecessary in the code
path where the element was found in the map.  Also the default value can be
forwarded to the return type in the code path where the element is not found
and an rvalue parameter is passed as the default value

Reviewed By: yfeldblum

Differential Revision: D6390315

fbshipit-source-id: ef692b827d5a36751b4eb1e12042869e8fbba2e5

6 years agobitreverse
Dave Watson [Tue, 5 Dec 2017 17:14:34 +0000 (09:14 -0800)]
bitreverse

Summary:
Adds a bitreverse function to Bits.h

AFAIK there is no great way to do this on x86, takes ~4-5ns.
arm has a single instruction we can drop in in the future.

Reviewed By: yfeldblum

Differential Revision: D6459283

fbshipit-source-id: 129db196b2fac95386f601ae57843aa87523b915

6 years agoexclude Unix Domain Socket from setErrMessageCB
Liang Zhu [Tue, 5 Dec 2017 11:49:11 +0000 (03:49 -0800)]
exclude Unix Domain Socket from setErrMessageCB

Summary:
In the latest stable kernel 4.14.3 as of 2017-12-04, unix domain socket does not support MSG_ERRQUEUE. So recvmsg(MSG_ERRQUEUE) will read application data from unix doamin socket as error message, which breaks the message flow in application. This diff disable setErrMessageCB for Unix Domain Socket.

Both [[ https://github.com/torvalds/linux/blob/master/net/ipv4/tcp.c#L1782 | tcp_recvmsg ]] and [[ https://github.com/torvalds/linux/blob/master/net/ipv4/udp.c#L1571 | udp_recvmsg ]] will check flag MSG_ERRQUEUE and read from error queue.  However, there is nothing about MSG_ERRQUEUE in [[ https://github.com/torvalds/linux/blob/master/net/unix/af_unix.c#L2249 | af_unix ]].

Reviewed By: yfeldblum

Differential Revision: D6479465

fbshipit-source-id: eba2c8650e96466f2b361a42ddf90053d65f19bd

6 years agoAdd missing check against SemiFuture in FutureBase ctor
Yedidya Feldblum [Tue, 5 Dec 2017 04:57:02 +0000 (20:57 -0800)]
Add missing check against SemiFuture in FutureBase ctor

Summary: [Folly] Add missing check against `SemiFuture` in `FutureBase` ctor.

Reviewed By: LeeHowes

Differential Revision: D6469021

fbshipit-source-id: 0ca31461afee4bf64725e065a8cb4d5e933d2013

6 years agobetter error handling in MemoryIdler for inside jails
Nathan Bronson [Tue, 5 Dec 2017 00:39:00 +0000 (16:39 -0800)]
better error handling in MemoryIdler for inside jails

Summary:
/proc may be unavailable in restricted environments, which can
cause pthread_getattr_np to fail.  This diff adds checking to the return
code from that function.

Reviewed By: yfeldblum

Differential Revision: D6473789

fbshipit-source-id: de016d32f29edf8410808dfb491987692f39e768

6 years agoImprove QueueAppender/IOBufQueue performance
Stepan Palamarchuk [Mon, 4 Dec 2017 23:08:24 +0000 (15:08 -0800)]
Improve QueueAppender/IOBufQueue performance

Summary:
Currently QueueAppender needs to follow a chain of 4 indirections (QueueAppender->IOBufQueue->IOBuf(head)->IOBuf(tail)->data).
This diff adds a cache of writable tail range in IOBufQueue and allows it to be placed externally.

Before this diff on hot path QueueAppender::write<signed char> was ~167 bytes of code (with majority being actually executed), after this diff it's down to ~30 bytes:
  0x0000000000419d10 <+0>:     mov    (%rdi),%rax
  0x0000000000419d13 <+3>:     cmp    %rax,0x8(%rdi)
  0x0000000000419d17 <+7>:     je     0x419d28 <folly::io::QueueAppender::write<signed char>(signed char)+24>
  0x0000000000419d19 <+9>:     mov    %sil,(%rax)
  0x0000000000419d1c <+12>:    addq   $0x1,(%rdi)
  0x0000000000419d20 <+16>:    retq
  0x0000000000419d21 <+17>:    nopl   0x0(%rax)
  0x0000000000419d28 <+24>:    movsbl %sil,%esi
  0x0000000000419d2c <+28>:    jmpq   0x419ca0 <folly::io::QueueAppender::writeSlow<signed char>(signed char)>

With this diff, Thrift serialization performance is improved up to 2x with production workloads (2x for compact, 3x for binary).

Thrift benchmark output:
Before:
  ============================================================================
  thrift/lib/cpp2/test/ProtocolBench.cpp          relative  time/iter  iters/s
  ============================================================================
  BinaryProtocol_write_Empty                                  58.05ns   17.23M
  BinaryProtocol_write_SmallInt                               75.17ns   13.30M
  BinaryProtocol_write_BigInt                                 74.60ns   13.41M
  BinaryProtocol_write_SmallString                            85.12ns   11.75M
  BinaryProtocol_write_BigString                             802.96ns    1.25M
  BinaryProtocol_write_BigBinary                             174.69ns    5.72M
  BinaryProtocol_write_LargeBinary                           171.81ns    5.82M
  BinaryProtocol_write_Mixed                                 130.97ns    7.64M
  BinaryProtocol_write_SmallListInt                          123.99ns    8.06M
  BinaryProtocol_write_BigListInt                             40.72us   24.56K
  BinaryProtocol_write_BigListMixed                          784.78us    1.27K
  BinaryProtocol_write_LargeListMixed                         98.84ms    10.12
  CompactProtocol_write_Empty                                 64.38ns   15.53M
  CompactProtocol_write_SmallInt                              76.74ns   13.03M
  CompactProtocol_write_BigInt                                83.62ns   11.96M
  CompactProtocol_write_SmallString                           86.05ns   11.62M
  CompactProtocol_write_BigString                            786.18ns    1.27M
  CompactProtocol_write_BigBinary                            184.91ns    5.41M
  CompactProtocol_write_LargeBinary                          182.12ns    5.49M
  CompactProtocol_write_Mixed                                120.89ns    8.27M
  CompactProtocol_write_SmallListInt                         119.74ns    8.35M
  CompactProtocol_write_BigListInt                            43.76us   22.85K
  CompactProtocol_write_BigListMixed                         595.90us    1.68K
  CompactProtocol_write_LargeListMixed                        72.80ms    13.74
  ============================================================================
After:
  ============================================================================
  thrift/lib/cpp2/test/ProtocolBench.cpp          relative  time/iter  iters/s
  ============================================================================
  BinaryProtocol_write_Empty                                  65.97ns   15.16M
  BinaryProtocol_write_SmallInt                               72.31ns   13.83M
  BinaryProtocol_write_BigInt                                 72.67ns   13.76M
  BinaryProtocol_write_SmallString                            77.56ns   12.89M
  BinaryProtocol_write_BigString                             782.07ns    1.28M
  BinaryProtocol_write_BigBinary                             179.69ns    5.57M
  BinaryProtocol_write_LargeBinary                           182.62ns    5.48M
  BinaryProtocol_write_Mixed                                  91.62ns   10.92M
  BinaryProtocol_write_SmallListInt                           96.22ns   10.39M
  BinaryProtocol_write_BigListInt                             19.65us   50.90K
  BinaryProtocol_write_BigListMixed                          245.69us    4.07K
  BinaryProtocol_write_LargeListMixed                         46.56ms    21.48
  CompactProtocol_write_Empty                                 74.44ns   13.43M
  CompactProtocol_write_SmallInt                              80.35ns   12.45M
  CompactProtocol_write_BigInt                                85.30ns   11.72M
  CompactProtocol_write_SmallString                           82.61ns   12.10M
  CompactProtocol_write_BigString                            784.77ns    1.27M
  CompactProtocol_write_BigBinary                            193.20ns    5.18M
  CompactProtocol_write_LargeBinary                          192.53ns    5.19M
  CompactProtocol_write_Mixed                                 99.78ns   10.02M
  CompactProtocol_write_SmallListInt                         104.77ns    9.54M
  CompactProtocol_write_BigListInt                            25.62us   39.03K
  CompactProtocol_write_BigListMixed                         272.42us    3.67K
  CompactProtocol_write_LargeListMixed                        38.32ms    26.09
  ============================================================================

QueueAppender Benchmark output (although not very representative due to a tight loop):
Before:
  ============================================================================
  folly/io/test/QueueAppenderBenchmark.cpp        relative  time/iter  iters/s
  ============================================================================
  write_uint8                                                 10.50us   95.20K
  write_uint16                                                 5.48us  182.49K
  write_uint32                                                 2.73us  366.22K
  push_64b                                                     9.77us  102.36K
  push_1024b                                                 112.87us    8.86K
  append                                                      64.21us   15.57K
  preallocate_postallocate_1b                                 16.34us   61.19K
  preallocate_postallocate_4b                                 15.56us   64.26K
  preallocate_postallocate_32b                                22.17us   45.11K
  preallocate_postallocate_256b                              149.55us    6.69K
  ============================================================================

After:
  ============================================================================
  folly/io/test/QueueAppenderBenchmark.cpp        relative  time/iter  iters/s
  ============================================================================
  write_uint8                                                  8.86us  112.81K
  write_uint16                                                 3.91us  255.68K
  write_uint32                                                 2.08us  481.78K
  push_64b                                                     8.24us  121.39K
  push_1024b                                                 115.50us    8.66K
  append                                                      67.52us   14.81K
  preallocate_postallocate_1b                                 13.86us   72.17K
  preallocate_postallocate_4b                                 11.67us   85.71K
  preallocate_postallocate_32b                                20.35us   49.14K
  preallocate_postallocate_256b                              148.57us    6.73K
  ============================================================================

Reviewed By: yfeldblum

Differential Revision: D6427749

fbshipit-source-id: 8495cc74b6106b15d201e37533ae4c0a1abc9d74

6 years agoadd AtFork::unregisterHandler
Dave Watson [Mon, 4 Dec 2017 20:57:07 +0000 (12:57 -0800)]
add AtFork::unregisterHandler

Summary: Adds an AdFork::unregisterHandler command, such that we can register and unregister fork handlers.

Reviewed By: yfeldblum

Differential Revision: D6439796

fbshipit-source-id: b710152f8cf98371c330b484cd5fe42de44e74ac

6 years agoUnbounded queue
Maged Michael [Mon, 4 Dec 2017 19:07:30 +0000 (11:07 -0800)]
Unbounded queue

Summary:
UnboundedQueue supports:
- SPSC, MPSC, SCMP, MPMC
- Non-waiting, waiting, and timed consumer operations.
- Producers never wait or fail (unless out-of-memory).
- Memory usage grows and shrinks dynamically

```
/// UnboundedQueue supports a variety of options for unbounded
/// dynamically expanding an shrinking queues, including variations of:
/// - Single vs. multiple producers
/// - Single vs. multiple consumers
/// - Blocking vs. spin-waiting
/// - Non-waiting, timed, and waiting consumer operations.
/// Producer operations never wait or fail (unless out-of-memory).
///
/// Template parameters:
/// - T: element type
/// - SingleProducer: true if there can be only one producer at a
///   time.
/// - SingleConsumer: true if there can be only one consumer at a
///   time.
/// - MayBlock: true if consumers may block, false if they only
///   spins. A performance tuning parameter.
/// - LgSegmentSize (default 8): Log base 2 of number of elements per
///   segment. A performance tuning parameter. See below.
///
/// When to use UnboundedQueue:
/// - If a small bound may lead to deadlock or performance degradation
///   under bursty patterns.
/// - If there is no risk of the queue growing too much.
///
/// When not to use UnboundedQueue:
/// - If there is risk of the queue growing too much and a large bound
///   is acceptable, then use DynamicBoundedQueue.
/// - If the queue must not allocate on enqueue or it must have a
///   small bound, then use fixed-size MPMCQueue or (if non-blocking
///   SPSC) ProducerConsumerQueue.
///
/// Template Aliases:
///   USPSCQueue<T, MayBlock, LgSegmentSize>
///   UMPSCQueue<T, MayBlock, LgSegmentSize>
///   USPMCQueue<T, MayBlock, LgSegmentSize>
///   UMPMCQueue<T, MayBlock, LgSegmentSize>
///
/// Functions:
///   Producer operations never wait or fail (unless OOM)
///     void enqueue(const T&);
///     void enqueue(T&&);
///         Adds an element to the end of the queue.
///
///   Consumer operations:
///     void dequeue(T&);
///         Extracts an element from the front of the queue. Waits
///         until an element is available if needed.
///     bool try_dequeue(T&);
///         Tries to extracts an element from the front of the queue
///         if available. Returns true if successful, false otherwise.
///     bool try_dequeue_until(T&, time_point& deadline);
///         Tries to extracts an element from the front of the queue
///         if available until the specified deadline.  Returns true
///         if successful, false otherwise.
///     bool try_dequeue_for(T&, duration&);
///         Tries to extracts an element from the front of the queue
///         if available for for the specified duration.  Returns true
///         if successful, false otherwise.
///
///   Secondary functions:
///     size_t size();
///         Returns an estimate of the size of the queue.
///     bool empty();
///         Returns true only if the queue was empty during the call.
///     Note: size() and empty() are guaranteed to be accurate only if
```

Reviewed By: djwatson

Differential Revision: D6157613

fbshipit-source-id: db423f86d1d0604d22f6b9c71ea0ed08be32e2a1

6 years agoAdd wrapper for getting X509_digest from a cert
Anirudh Ramachandran [Mon, 4 Dec 2017 07:56:21 +0000 (23:56 -0800)]
Add wrapper for getting X509_digest from a cert

Summary:
X509_digest, or the "fingerprint" of the cert, is useful to identify
certificates with the same CN/SANs. This adds an API to get the SHA1 or SHA256
digest of a cert.

Reviewed By: yfeldblum

Differential Revision: D6457917

fbshipit-source-id: 148a5a2270e938b39065e00d7947c2fc57cd7f75

6 years agoStatically allocate futex array v2017.12.04.00
Alastair Daivis [Sun, 3 Dec 2017 01:35:33 +0000 (17:35 -0800)]
Statically allocate futex array

Summary:
AFAICT this is currently the only thing preventing the non-dynamic version of Folly's MPMCQueue from being allocation-free on enqueue and dequeue operations. Allocating this with a constant expression should move the allocation from runtime to link (possibly compile?) time, which will let us use it in allocation sensitive contexts.

Feel free to suggest other reviewers, I couldn't find an existing folly reviewer group.

Reviewed By: yfeldblum, agola11, nbronson

Differential Revision: D6447848

fbshipit-source-id: 86b84b19d62f1e1bcecdb9e757a6dfa90597b084

6 years agoallow fiber executor to override fiber options
David Goode [Sat, 2 Dec 2017 03:56:14 +0000 (19:56 -0800)]
allow fiber executor to override fiber options

Summary: As in title, for eg larger stack size

Reviewed By: yfeldblum

Differential Revision: D6466762

fbshipit-source-id: 303e8732c8bfb64fbd8619163779942fe5f8b5a8

6 years agoadd makeSystemError*() helper functions
Adam Simpkins [Sat, 2 Dec 2017 02:05:54 +0000 (18:05 -0800)]
add makeSystemError*() helper functions

Summary:
Add makeSystemError() helper functions that are similar to the existing
throwSystemError() functions but just return the exception rather than throwing
it.  This is helpful for callers using folly::Expected or folly::Future, where
they need to return an exception type rather than throwing it.

This also includes comments about the fact that this code is using the wrong
error category.  The C++ standard indicates that std::generic_category() for
POSIX errno values.  I am not fixing this as part of this diff, since this
change has the potential to break existing users, and would need a fair amount
of testing first.

Reviewed By: yfeldblum

Differential Revision: D6456771

fbshipit-source-id: 4724b51b8d4a7f513ae70ea1b0c4f0516cfc205f

6 years agologging: fix compilation error on older C++ compilers
Adam Simpkins [Sat, 2 Dec 2017 02:01:59 +0000 (18:01 -0800)]
logging: fix compilation error on older C++ compilers

Summary:
Add an explicit `std::move()` around the return value of
`logConfigToDynamic()`.  This explicit move is required pre-C++14,
but is not required after DR 1579:
http://www.open-std.org/jtc1/sc22/wg21/docs/cwg_defects.html#1579

Newer versions of g++ and clang no longer require the move.  However gcc
versions earlier than 5.0 do require it.

Reviewed By: yfeldblum

Differential Revision: D6466447

fbshipit-source-id: 1b15934490d4966e9b3e5612e0e2ecbb43c979ca

6 years agologging: update initialization code to use the new LogConfig logic
Adam Simpkins [Sat, 2 Dec 2017 00:53:51 +0000 (16:53 -0800)]
logging: update initialization code to use the new LogConfig logic

Summary:
Replace the initLoggingGlogStyle() function with a more generic initLogging()
function that accepts a log config string to be parsed with parseLogConfig().

Reviewed By: bolinfest, yfeldblum

Differential Revision: D6342086

fbshipit-source-id: fb1bffd11f190b70e03e2ccbf2b30be08d655242

6 years agounify FOLLY_SANITIZE existing uses and add a few new
Igor Sugak [Fri, 1 Dec 2017 19:19:44 +0000 (11:19 -0800)]
unify FOLLY_SANITIZE existing uses and add a few new

Reviewed By: yfeldblum

Differential Revision: D6453972

fbshipit-source-id: fa083c0b782188f5f45a2a354045aee03e528829

6 years agoimplement chrono conversions for unusual duration types
Adam Simpkins [Fri, 1 Dec 2017 19:18:17 +0000 (11:18 -0800)]
implement chrono conversions for unusual duration types

Summary:
Implement conversions between std::chrono::duration and POSIX-style time
structures even when neither the numerator nor the denominator of the duration
ratio are 1.

Both of these are done by first converting to an intermediate type where the
numerator is 1, and then using the conversion routines for that case.

Reviewed By: yfeldblum

Differential Revision: D6366647

fbshipit-source-id: 8f9495fb4101cac6d8b4cf0353a679107007b298

6 years agoMove threadlocal_detail::Atfork to its own file
Dave Watson [Fri, 1 Dec 2017 15:09:43 +0000 (07:09 -0800)]
Move threadlocal_detail::Atfork to its own file

Summary: As title

Reviewed By: yfeldblum

Differential Revision: D6440723

fbshipit-source-id: 3168d7bb616ae0ff3fe42f7584c5a255c4953875

6 years agofix ASAN build
Bin Liu [Thu, 30 Nov 2017 23:52:54 +0000 (15:52 -0800)]
fix ASAN build

Summary:
Fix bug in non-jemalloc build.

(Note: this ignores all push blocking failures!)

Reviewed By: igorsugak

Differential Revision: D6451342

fbshipit-source-id: 44725d4e0c685a59325fa3d08877d0f62120c5c2

6 years agoLet SaturatingSemaphore::try_wait be non-const and add ready
Yedidya Feldblum [Thu, 30 Nov 2017 22:56:07 +0000 (14:56 -0800)]
Let SaturatingSemaphore::try_wait be non-const and add ready

Summary:
[Folly] Let `SaturatingSemaphore::try_wait` be non-`const` and add `ready`.

For internal API consistency.

Reviewed By: magedm

Differential Revision: D6450089

fbshipit-source-id: 65b9b723672521710a69719b192eb2922a27b778

6 years agofix build when sanitizers are enabled and jemalloc is disabled
Igor Sugak [Thu, 30 Nov 2017 22:48:24 +0000 (14:48 -0800)]
fix build when sanitizers are enabled and jemalloc is disabled

Summary: Add preprocessor checks to guard jemalloc headers if any sanitizer is enabled.

Reviewed By: yfeldblum

Differential Revision: D6441578

fbshipit-source-id: 85364fca5af33c95cd05f232d1e3f9cbac9a4120

6 years agoSome fixes for custom conversions of enums
Yedidya Feldblum [Thu, 30 Nov 2017 06:13:39 +0000 (22:13 -0800)]
Some fixes for custom conversions of enums

Summary:
[Folly] Some fixes for custom conversions of enums.

Of note, `to` was defined for enum -> all conversions, including enum -> string conversions, but we actually want enum -> string conversions to be done via ADL-discovered toAppend.

Reviewed By: ot

Differential Revision: D6411250

fbshipit-source-id: 852b64309e6adf1c68e5153635cb29632e2d86d4

6 years agoSaturating semaphore
Maged Michael [Thu, 30 Nov 2017 05:12:16 +0000 (21:12 -0800)]
Saturating semaphore

Summary:
SaturatingSemaphore is a flag that allows:
- multiple concurrent posters
- multiple concurrent waiters
- idempotent posting
- non-destructive waiting
- blocking and spinning
- pre-block spin time control

```
/// SaturatingSemaphore is a flag that allows concurrent posting by
/// multiple posters and concurrent non-destructive waiting by
/// multiple waiters.
///
/// A SaturatingSemaphore allows one or more waiter threads to check,
/// spin, or block, indefinitely or with timeout, for a flag to be set
/// by one or more poster threads. By setting the flag, posters
/// announce to waiters (that may be already waiting or will check
/// the flag in the future) that some condition is true. Posts to an
/// already set flag are idempotent.
```

Reviewed By: djwatson

Differential Revision: D6379704

fbshipit-source-id: 59aed76caa2d159639e75425a778a9c63f18f375

6 years agoadd FOLLY_SANITIZE macro
Igor Sugak [Thu, 30 Nov 2017 02:14:04 +0000 (18:14 -0800)]
add FOLLY_SANITIZE macro

Summary: Most of the time we want to guard some code when any of the sanitizers are enabled not just one.

Reviewed By: yfeldblum

Differential Revision: D6429720

fbshipit-source-id: 7157a0a13f2b2acabd36a19c61b9fdbf38ba96b8

6 years agologging: add a LogConfig::update() method
Adam Simpkins [Thu, 30 Nov 2017 01:35:21 +0000 (17:35 -0800)]
logging: add a LogConfig::update() method

Summary:
Add a method for merging the settings from two LogConfig objects.
This allows LogConfig objects to be merged before applying them to the
LoggerDB.  The effects are the same as two sequential LoggerDB::updateConfig()
calls, but without having to apply the intermediate state to the LoggerDB.

Reviewed By: bolinfest

Differential Revision: D6342085

fbshipit-source-id: 0f8a1b7d8d195a80bc74342444dd3152d331fcb6

6 years agologging: add an assert for the FLAG_INHERIT bit
Adam Simpkins [Thu, 30 Nov 2017 01:35:19 +0000 (17:35 -0800)]
logging: add an assert for the FLAG_INHERIT bit

Summary:
Add a static_assert() to check that the FLAG_INHERIT bit does not conflict with
valid LogLevel values.

Reviewed By: bolinfest

Differential Revision: D6341242

fbshipit-source-id: cec284623c8a612a3c54f1b593d169310790616e

6 years agologging: remove comment about LogLevel::ERROR
Adam Simpkins [Thu, 30 Nov 2017 01:35:17 +0000 (17:35 -0800)]
logging: remove comment about LogLevel::ERROR

Summary:
The logging code used to support a LogLevel::ERROR value on non-Windows
platforms.  I removed it in D5288600 to make all platforms consistent, but I
forgot to remove this comment describing it.

Reviewed By: bolinfest

Differential Revision: D6341244

fbshipit-source-id: 054427d342066f354c859b12611b907dc2d4fa35

6 years agologging: add LoggerDB::updateConfig() and resetConfig()
Adam Simpkins [Thu, 30 Nov 2017 01:35:16 +0000 (17:35 -0800)]
logging: add LoggerDB::updateConfig() and resetConfig()

Summary: Add methods for applying config changes from a LogConfig object to the LoggerDB.

Reviewed By: bolinfest

Differential Revision: D6200564

fbshipit-source-id: a25eb99e84b2885bf6853e2222db0d7432a6c37b

6 years agologging: add a LoggerDB::getConfig() method
Adam Simpkins [Thu, 30 Nov 2017 01:35:13 +0000 (17:35 -0800)]
logging: add a LoggerDB::getConfig() method

Summary: Add a method to get the current LogConfig state from the LoggerDB.

Reviewed By: bolinfest

Differential Revision: D6200596

fbshipit-source-id: 3bc57d498a5d25d19099d861376d71ea9f7e4039

6 years agologging: don't clamp the log level to DFATAL in debug builds
Adam Simpkins [Thu, 30 Nov 2017 01:35:11 +0000 (17:35 -0800)]
logging: don't clamp the log level to DFATAL in debug builds

Summary:
Remove the logic that clamps the log level to DFATAL in debug builds.

This behavior logically makes some sense, but results in subtle and potentially
confusing behavior changes.  It seems confusing that after calling
`setLevel(LogLevel::MAX_LEVEL)` on a log category, calling `getLevel()` on that
object would return `DFATAL` rather than `MAX_LEVEL`.  We also weren't clamping
the level to DFATAL consistently: `setLevel()` would clamp the value, but on
construction the initial level was still set to MAX_LEVEL rather than DFATAL.
This resulted in some issues when implementing `LoggerDB::getConfig()` since we
could not consistently detect if a log category was using the default level
settings.  Rather than fix this inconsistency it seems better to simply remove
this clamping behavior.

This means that it is possible for users to disable DFATAL log messages even in
debug builds if they really want to.  Previously this was only allowed in
release builds.  This protection doesn't really seem all that
valuable--presumably most developers won't really want to do this, and if they
really do request this configuration it doesn't seem all that bad to honor it.

Reviewed By: bolinfest, yfeldblum

Differential Revision: D6200569

fbshipit-source-id: 83ef8e2e4d3b61bc5b105038cbe3132979e9ac67

6 years agologging: add a LogHandler::getConfig() method
Adam Simpkins [Thu, 30 Nov 2017 01:35:09 +0000 (17:35 -0800)]
logging: add a LogHandler::getConfig() method

Summary:
Add a method to LogHandler to return its current configuration.  This will
make it possible to query the LoggerDB for its current configuration state.

Reviewed By: bolinfest

Differential Revision: D6200563

fbshipit-source-id: 2b8b9752bbeb26c8aac28d1a73b7e2312fd198c8

6 years agologging: add a LogHandler registry to LoggerDB
Adam Simpkins [Thu, 30 Nov 2017 01:35:08 +0000 (17:35 -0800)]
logging: add a LogHandler registry to LoggerDB

Summary:
Update the LoggerDB to track a list of LogHandlers by name, and
LogHandlerFactories by handler type.

This will be needed to support updating the LoggerDB configuration from a
LogConfig object.

Reviewed By: bolinfest

Differential Revision: D6200562

fbshipit-source-id: e365b4e0df65aa5aaa34e118eb3cee9c9c45cb05

6 years agoSynchronize coupled caches in folly::threadlocal_detail::StaticMeta
Yedidya Feldblum [Wed, 29 Nov 2017 20:12:01 +0000 (12:12 -0800)]
Synchronize coupled caches in folly::threadlocal_detail::StaticMeta

Summary:
[Folly] Synchronize coupled caches in `folly::threadlocal_detail::StaticMeta`.

The caches should be set together, and only together, because they are coupled. This prevents bugs where one function that sets one cache but not the other cache is inlined into the caller in one module, and another function that reads both caches is inlined into the caller in another module.

Reviewed By: djwatson

Differential Revision: D6435175

fbshipit-source-id: 846c4972b40e525f2c04da6e6609c2ad54f674c0

6 years agologging: add a FileHandlerFactory class
Adam Simpkins [Wed, 29 Nov 2017 01:44:55 +0000 (17:44 -0800)]
logging: add a FileHandlerFactory class

Summary:
Add a new LogHandlerFactory interface for creating LogHandler objects from a
LogHandlerConfig.

Also add an initial FileHandlerFactory implementation capable of creating
LogHandler objects that write to a file descriptor.

Reviewed By: bolinfest

Differential Revision: D6200567

fbshipit-source-id: 14b86fc14ad475223aa4b57d45c40638b48c7594

6 years agoAdd unit tests for opening non-ELF files.
Andrii Nakryiko [Wed, 29 Nov 2017 00:48:14 +0000 (16:48 -0800)]
Add unit tests for opening non-ELF files.

Summary:
Just testing that ElfFile returns expected error code and message for
non-ELF files.

Depends on D6410210

Reviewed By: yfeldblum

Differential Revision: D6418365

fbshipit-source-id: aaab3b9f3ca1a12d384ae98a1772b7f640115192

6 years agoFix incorrect usages of folly::Synchronized
Sarang Masti [Tue, 28 Nov 2017 23:51:10 +0000 (15:51 -0800)]
Fix incorrect usages of folly::Synchronized

Summary:
We need to keep LockedPtr returned by rlock/wlock alive for the entire duration
of the loop. Else we are working on a snapshot of the data structure, which
might not be what we want.

Reviewed By: aary, ot, luciang

Differential Revision: D6426613

fbshipit-source-id: 294c0a7b9d756f3116d5cdb3a7b3678f693e0754

6 years agologging: convert assert() checks to FOLLY_SAFE_DCHECK()
Adam Simpkins [Tue, 28 Nov 2017 20:02:02 +0000 (12:02 -0800)]
logging: convert assert() checks to FOLLY_SAFE_DCHECK()

Summary:
Replace all assert() checks in the folly logging code with
FOLLY_SAFE_DCHECK().

Reviewed By: yfeldblum

Differential Revision: D6422885

fbshipit-source-id: 5cb12dd59f2fe5d346f15b9d317abede8654a879

6 years agoUnbreak folly on x86-32
Alexander Kabaev [Tue, 28 Nov 2017 19:42:20 +0000 (11:42 -0800)]
Unbreak folly on x86-32

Summary:
_mm_cvtsi64_si128 and _mm_cvtsi128_si64 intrinsics are available on
x86_64 only, so do not enable SSE instructions when compiling for 32 bits.

Reviewed By: yfeldblum

Differential Revision: D6422271

fbshipit-source-id: a3c5cf3f2ee4b47d5a1edcf93cfff3d1f3fe907e

6 years agoRemove some custom test main()'s in folly/experimental/symbolizer/
Yedidya Feldblum [Tue, 28 Nov 2017 06:18:54 +0000 (22:18 -0800)]
Remove some custom test main()'s in folly/experimental/symbolizer/

Summary:
[Folly] Remove some custom test `main()`'s in `folly/experimental/symbolizer/`.

In one of the two cases, we need to do some extra work to give `ElfFile` knowledge of the program path.

Reviewed By: ot, Orvid, philippv, luciang

Differential Revision: D6420517

fbshipit-source-id: 11c694a8342abb6fff05c368f3cda2e6287ccb4b

6 years agoReset context shared_ptr in AsyncTimeout::cancelTimeout()
Pingjia Shan [Tue, 28 Nov 2017 06:10:59 +0000 (22:10 -0800)]
Reset context shared_ptr in AsyncTimeout::cancelTimeout()

Summary:
This seems to fix issue in the attached task.

`context_` is set in `scheduleTimeout()` and never gets reset. So when an AsyncTimeout object is held across requests by objects like `IOThreadPool0` in I/O Thread Pool used by wangle/acceptor/ConnectionManager, RequestContext object created for the last request is leaked until `IOThreadPool0` handles another request and overrides `context_`. In the issue described in attached task, unit test has single request, next request never comes in, RequestContext doesn't end until test service stops, logging feature relying on RequestContext dtor doesn't get called in time, and thus unit test fails.

Reviewed By: yfeldblum

Differential Revision: D6402268

fbshipit-source-id: 200c6d358dfa6d7d9aa68ab05f6f1c7f4117b0ec

6 years agologging: fix unused variable warning in non-debug builds
Adam Simpkins [Tue, 28 Nov 2017 05:30:58 +0000 (21:30 -0800)]
logging: fix unused variable warning in non-debug builds

Summary:
Use FOLLY_SAFE_DCHECK() rather than assert() to avoid compiler errors about
this variable being unused in builds where assert() statements are disabled.

Reviewed By: bolinfest, yfeldblum

Differential Revision: D6421978

fbshipit-source-id: ab1019fa1fa420856e1bf3bc39c441662c4efd75

6 years agoUse "auto" with GENERATOR instead of std::function
Tom Jackson [Tue, 28 Nov 2017 01:13:37 +0000 (17:13 -0800)]
Use "auto" with GENERATOR instead of std::function

Summary: Now that we're using C++14 more broadly, we should use it to improve generator performance. This speeds up a microbenchmark >5x by removing type erasure.

Reviewed By: philippv

Differential Revision: D6398730

fbshipit-source-id: 5809058a3b5ff0e66fd4b1e8954698944e1a7d09

6 years agoRevise API to load cert/key in SSLContext.
Xiangyu Bu [Mon, 27 Nov 2017 23:37:45 +0000 (15:37 -0800)]
Revise API to load cert/key in SSLContext.

Summary:
When loading cert/key pair, order matters:
(a) Wrong key will fail to load if a cert is loaded;
(b) Wrong cert will succeed to load even if a private key is loaded.

So this diff adds:
(1) SSLContext::checkPrivateKey() -- must call for case (b).
(2) SSLContext::loadCertKeyPairFromBufferPEM() -- use this if one loads both cert and key. Guaranteed to throw if cert/key mismatch.

Reviewed By: yfeldblum

Differential Revision: D6416280

fbshipit-source-id: 8ae370883d46e9b5afb69c506c09fbf7ba82b1b9

6 years agoFix ElfFile crashing when opening short (<64 bytes) files.
Andrii Nakryiko [Mon, 27 Nov 2017 19:31:18 +0000 (11:31 -0800)]
Fix ElfFile crashing when opening short (<64 bytes) files.

Summary:
ElfFile in init() assumes file is long enough to contain complete
ElfHeader, which is 64 bytes long. This is not true for valid non-ELF files like
short scripts. They shouldn't cause crash.

Reviewed By: myreg

Differential Revision: D6410210

fbshipit-source-id: 28fd017d8de17c431d7d006a1655ade8a95994bd

6 years agoUse hazptr_local and hazptr_array
Dave Watson [Mon, 27 Nov 2017 18:50:26 +0000 (10:50 -0800)]
Use hazptr_local and hazptr_array

Summary: Use newest hazptr hotness in concurrenthashmap.  Shaves ~10% off of the single-thread find performance.

Reviewed By: magedm

Differential Revision: D6259947

fbshipit-source-id: 7ecf99d38fdf8e311fca3313137e0fca5af3f165

6 years agomake `usingJEMalloc()` a constexpr when USE_JEMALLOC is defined v2017.11.27.00
Bin Liu [Mon, 27 Nov 2017 06:07:41 +0000 (22:07 -0800)]
make `usingJEMalloc()` a constexpr when USE_JEMALLOC is defined

Summary: This would save some function calls and reduce code size if we know we will use jemalloc.

Reviewed By: yfeldblum

Differential Revision: D6411810

fbshipit-source-id: 9db471678e473945f5f00354c0a1285ecdb91de6

6 years agoAdd folly::hasher support for floating point types
Giuseppe Ottaviano [Sun, 26 Nov 2017 22:43:06 +0000 (14:43 -0800)]
Add folly::hasher support for floating point types

Summary:
Move `folly::hasher` closer to feature parity with `std::hash`.
This is in order to replace some instances of `folly::hash::hash_combine(...)` with `folly::Hash()(...)` (`std::hash` is the identity for integers, which makes it an unsafe default for more sophisticated hash data structures, including open-addressing hash tables).

The implementation is similar to `libstdc++`'s implementation, in that we handle separately the `0` case, because `0` and `-0` have different binary representations but are equal according to `operator==`, and hash the bytes otherwise. It is probably a little faster than `libstdc++`'s implementation, that delegates a out-of-line Murmur hash routine for arbitrary buffers, while this uses a minimal inlineable machine word hashing routine.

Reviewed By: yfeldblum

Differential Revision: D6410713

fbshipit-source-id: 86d9e4ed8da04fffe283949825852e539ec7d5cf

6 years agoQueuedImmediateExecutor tweaks
Yedidya Feldblum [Thu, 23 Nov 2017 19:59:32 +0000 (11:59 -0800)]
QueuedImmediateExecutor tweaks

Summary:
[Folly] `QueuedImmediateExecutor` tweaks.

* Add a leaky meyers singleton instance.
* Make the thread-local queue an instance variable. Callers which want the singleton thread-local queue can use the singleton executor instance instance, for the same effect.
* Simplify the body of `add`, and perform the thread-local lookup only once per invocation.

Reviewed By: djwatson

Differential Revision: D6399067

fbshipit-source-id: 03904885a70c4b943141bd83868414d27232fd6a

6 years agoEnable EventBase switching for AsyncSocket even if it has registered events
Fuat Geleri [Wed, 22 Nov 2017 22:20:42 +0000 (14:20 -0800)]
Enable EventBase switching for AsyncSocket even if it has registered events

Summary:
AsyncSocket will return isDetachable() -> false if there is a read callback set.
So,
 - ignore event registration status
 - unregister the events when detaching and
 - register them back when attaching again.

Reviewed By: afrind, eduardo-elizondo

Differential Revision: D6331787

fbshipit-source-id: fca4e6101f2d5666edbafa09116aa50f34cb084d

6 years agoLink to unwind when using symbolizer
Guo Xiao [Wed, 22 Nov 2017 19:19:03 +0000 (11:19 -0800)]
Link to unwind when using symbolizer

Summary:
Fix #673
Closes https://github.com/facebook/folly/pull/715

Reviewed By: meyering

Differential Revision: D6394731

Pulled By: yfeldblum

fbshipit-source-id: fb92828d0c633278601125dbc64cadf0cdbaa7fe

6 years agoAdd a fast path to folly::ThreadLocal
Dave Watson [Wed, 22 Nov 2017 16:16:04 +0000 (08:16 -0800)]
Add a fast path to folly::ThreadLocal

Summary:
Currently folly::ThreadLocal[Ptr] is pretty heavy-weight for a get():

1) call instance(), take a static init guard, branch
2) call getThreadEntry, check if thread_local is not null, branch
3) check if id < threadEntry->capacity, branch
4) Finally, return threadEntry->elements[id]

If we have real thread_locals, we can do better by caching the capacity directly,
combining all three checks:

1) checkif id < threadLocalCapacityCheck, branch.  If not, do slow path.
2) return threadEntry->elements[id].  Threadentry is never null if capacity > 0, and
    instance() setup work is called during the first getThreadEntry call when threadlocalcapacity == 0.

Reviewed By: yfeldblum

Differential Revision: D6379878

fbshipit-source-id: 4fc7564bbb2f319d65875124026aef28d910ef06

6 years agoFix ASAN failure in FutureDAG test
Andrii Grynenko [Wed, 22 Nov 2017 04:58:05 +0000 (20:58 -0800)]
Fix ASAN failure in FutureDAG test

Summary: dogscience

Reviewed By: yfeldblum

Differential Revision: D6387604

fbshipit-source-id: 3e33da3e3c6801d6e2186637c8fb6adaf1eef6f6

6 years agoAdd -Werror to -Wshadow-local and -Wshadow-compatible-local warning check
Aaron Orenstein [Tue, 21 Nov 2017 22:02:18 +0000 (14:02 -0800)]
Add -Werror to -Wshadow-local and -Wshadow-compatible-local warning check

Summary:
Without -Werror Apple LLVM will print a warning about -Wshadow-local
and -Wshadow-compatible-local but won't fail.  This causes problems for
downstream code which uses -Werror.

Reviewed By: meyering

Differential Revision: D6375771

fbshipit-source-id: 4ac3a4ff46bdb76f1f07b0c8ba479a7d404c8c9b

6 years agologging: add a LogConfig class and parsing code
Adam Simpkins [Tue, 21 Nov 2017 18:44:02 +0000 (10:44 -0800)]
logging: add a LogConfig class and parsing code

Summary:
Add a new LogConfig class to represent the current configuration state of the
LoggerDB.  This also includes code to parse config strings in both JSON and a
simpler more human-writable format (intended primarily for use in command line
arguments).

I generally expect the human-writable format to be used mainly to configure log
levels.  It also supports configuring log handler details as well, but the
format for this data is slightly more cumbersome and will probably be harder
for people to easily remember.

The parsing code is intentionally kept as part of the 'init' library rather
than the core 'logging' library so that other libraries that simply wish to log
messages do not need to depend on it.  For instance, this would allow the folly
JSON library to use the logging library without causing a circular dependency.

Reviewed By: bolinfest

Differential Revision: D6200560

fbshipit-source-id: e4e3c7f941808251b6c7bcbbdac0210118675fb0