folly.git
5 years agoAdd missing uint32 type to folly::ProgramOptions::gFlagAdders
Uladzislau Paulovich [Wed, 21 Jun 2017 13:19:44 +0000 (06:19 -0700)]
Add missing uint32 type to folly::ProgramOptions::gFlagAdders

Summary: Absence of "uint32" type in gFlagAdders map breaks all apps that use gflags of this type (i.e. define a flag with "DEFINE_uint32").

Reviewed By: WillerZ

Differential Revision: D5286720

fbshipit-source-id: c02bd959cb9ea9a47fba1e01429181ba09edf5dd

5 years agoAdd support for getting other threads' names
Giuseppe Ottaviano [Wed, 21 Jun 2017 06:03:57 +0000 (23:03 -0700)]
Add support for getting other threads' names

Summary: Complete the `ThreadName.h` API.

Reviewed By: luciang, Orvid

Differential Revision: D5289160

fbshipit-source-id: a48e61093008039da50b1c568364fa5b8744b401

5 years agologging: reduce the amount of code emitted for log statements
Adam Simpkins [Wed, 21 Jun 2017 02:44:20 +0000 (19:44 -0700)]
logging: reduce the amount of code emitted for log statements

Summary:
This refactors the logging code with the aim of reducing the amount of assembly
code emitted for each log statement, particularly for `XLOG()` statements.
Ideally it should be possible to put debug `XLOG()` statements throughout your
code without having to worry about the performance overhead.  Therefore we
should attempt to make sure that `XLOG()` statements do not generate a lot of
assembly and hurt icache performance.

This diff does not have any code behavior changes--it just moves code around a
bit.  The high-level summary is:

- Move as much code as possible into the LogStreamProcessor constructors and
  destructor.  Make sure these methods are defined in LogStreamProcessor.cpp to
  avoid having them be emitted inline at each log statement.
- Move some of the XLOG level checking logic into separate non-inline functions
  defined in xlog.cpp
- Pass xlog category information around as a pair of (categoryName,
  isOverridden) parameters.  If isOverridden is true then the categoryName
  parameter should be used as the category name directly.  If isOverridden is
  false, then categoryName is a filename that needs to go through filename to
  category name string processing.  This allows the category name processing to
  be done in non-inlined code.

Reviewed By: wez

Differential Revision: D5269976

fbshipit-source-id: 7a7877ddfed66cd27ed82f052330b6aa2be4b37b

5 years agologging: add printf-style logging macros
Adam Simpkins [Wed, 21 Jun 2017 02:44:18 +0000 (19:44 -0700)]
logging: add printf-style logging macros

Summary:
Add new `FB_LOGC()` and `XLOGC()` macros that accept C-style printf format
strings.  (The `FB_LOGF()` and `XLOGF()` macro names are already used for
`folly::format()` style formatting.)

This will make it easier for users to update existing printf-style code to use
this new logging library.

These are in a separate `printf.h` header file that must be explicitly included
to have access to these macros.  The intent is to encourage users to use one of
the other APIs (streaming, append-style, or `folly::format()`) instead of these
printf-like APIs in new code.

Reviewed By: omry

Differential Revision: D5269974

fbshipit-source-id: 56e55f9642bb00806d9b4c762fb6a91778ef6ad3

5 years agologging: make XLOG_GET_CATEGORY() safe for all callers
Adam Simpkins [Wed, 21 Jun 2017 02:44:17 +0000 (19:44 -0700)]
logging: make XLOG_GET_CATEGORY() safe for all callers

Summary:
The `XLOG_GET_CATEGORY()` macro was previously written assuming it was only
used inside `XLOG()` statement.  When used inside the main translation unit
being compiled (e.g., a .cpp file and not a header file), the code assumed that
the file-scope category had already been initialized, since a level check had
presumably already been performed.

However, it is useful in some places for external users (outside of the logging
library itself) to call `XLOG_GET_CATEGORY()`.  In these cases a log level
check may not have been performed yet, so the file-scope category may not be
initialized yet.

This diff renames the existing `XLOG_GET_CATEGORY()` macro to
`XLOG_GET_CATEGORY_INTERNAL()` and adds a new `XLOG_GET_CATEGORY()` macro that
is slower (it always looks up the category by name) but always safe to use.

This also adds a new `XLOG_GET_CATEGORY_NAME()` macro, and renames the existing
`XLOG_SET_CATEGORY()` macro to `XLOG_SET_CATEGORY_NAME()` for API consistency.

Reviewed By: wez

Differential Revision: D5269975

fbshipit-source-id: 373805255823855282fa7a8d4a4d232ba06367f6

5 years agologging: improve the AsyncFileWriterTest discard test
Adam Simpkins [Wed, 21 Jun 2017 02:44:16 +0000 (19:44 -0700)]
logging: improve the AsyncFileWriterTest discard test

Summary:
This improves the test that exercises the AsyncFileWriter message discarding
logic.

Previously each writer thread wrote a fixed number of small messages.  This was
fairly slow, and wasn't always guaranteed to reliably produce discards.
Now each writer thread writes larger messages, which produce discards faster.
The test also automatically stops after 10 separate discard events, so that it
finishes faster (typically a few hundred milliseconds, rather than 5+ seconds).

This also updates the test to use XLOG() internally rather than using fprintf()
to print to stderr, now that the XLOG() diffs have landed.

Reviewed By: wez

Differential Revision: D5261059

fbshipit-source-id: 120224706fee36948ef76efbeb579ccc56400c51

5 years agologging: implement FATAL and DFATAL log levels
Adam Simpkins [Tue, 20 Jun 2017 21:01:59 +0000 (14:01 -0700)]
logging: implement FATAL and DFATAL log levels

Summary:
Add new `FATAL` and `DFATAL` log levels.

Any log message with a level of `FATAL` always crashes the program.  Log
messages with a level of `DFATAL` crash the program in debug build modes.

Before crashing, the code makes sure to flush all LogHandlers, so that the
fatal message is not lost if some of the LogHandlers process messages
asynchronously.  If no LogHandlers were configured, the message is printed to
stderr.

Reviewed By: wez

Differential Revision: D5189497

fbshipit-source-id: c45dbd582fb1c3a962d00effb2967737ef97cc8b

5 years agoUpdate hazard pointer interface to standard proposal P0233R4
Maged Michael [Tue, 20 Jun 2017 19:10:45 +0000 (12:10 -0700)]
Update hazard pointer interface to standard proposal P0233R4

Summary:
Updated to the interface to be in synch with the latest standard proposal in P0233R4 as follows:
- Renamed hazptr_owner as hazptr_holder.
- Combined hazptr_holder member functions set() and clear() as reset().
- Replaced the template parameter A for hazptr_holder member function templates try_protect() and get_protected with atomic<T*>.
- Moved the template parameter T from the class hazptr_holder to its member functions try_protect(), get_protected(), and reset().
- Added a non-template overload of hazptr_holder::reset() with an optional nullptr_t parameter.
- Removed the template parameter T from the free function swap() as hazptr_holder is no longer a template.

Reviewed By: davidtgoldblatt

Differential Revision: D5283863

fbshipit-source-id: 2bc1a09f4f844aa72d9b7dff9c450540bbe09972

5 years agofix folly::FunctionScheduler.cancelFunctionAndWait() hanging issue
Qi Zhou [Tue, 20 Jun 2017 18:36:53 +0000 (11:36 -0700)]
fix folly::FunctionScheduler.cancelFunctionAndWait() hanging issue

Summary:
When
- only one function is scheduled in FunctionScheduler; and
- the function is running while cancelFunctionAndWait() is being called;
FunctionScheduler.cancelFunctionAndWait() will hang forever.  The root cause is that the condition in cancelFunctionAndWait() is incorrect:

```
runningCondvar_.wait(l, [currentFunction, this]() {
  return currentFunction != currentFunction_;
});
```

because currentFunction will not be changed if only one function is in the scheduler.

The fix here is to
- clear currentFunction as nullptr.  This also makes the internal behaviors of cancelFunction() and cancelFunctionAndWait() consistent.
- introduces additional variable to indicate the state of cancelling current function.  After running the function, the background thread will detect cancellation of current function and clear the variable.
- cancelFunctionAndWait() condition variable will wait for the variable to be cleared.

Similarly, cancelAllFunctionsAndWait() also suffers from the same issue.

Unit tests are added to reproduce the issue.

Reviewed By: yfeldblum

Differential Revision: D5271664

fbshipit-source-id: acb223304d3eab23129907ce9ff5e57e55f1e909

5 years agologging: add LoggerDB::flushAllHandlers()
Adam Simpkins [Tue, 20 Jun 2017 18:02:00 +0000 (11:02 -0700)]
logging: add LoggerDB::flushAllHandlers()

Summary:
Add a method to flush all LogHandler objects.

This will be necessary to implement FB_LOG(FATAL), as we will want to flush all
outstanding messages before crashing.

Reviewed By: wez

Differential Revision: D5189501

fbshipit-source-id: faf260b8e71e5dfed4a3b1c1aee32f072bd7b764

5 years agologging: add a LogHandler::flush() call
Adam Simpkins [Tue, 20 Jun 2017 18:01:57 +0000 (11:01 -0700)]
logging: add a LogHandler::flush() call

Summary:
Add a flush() call to the LogHandler interface.  This is needed to implement
`FB_LOG(FATAL)` so that we can flush all LogHandlers before aborting the
program.

Reviewed By: wez

Differential Revision: D5189499

fbshipit-source-id: 75fa4d7e75ea26de5b7383bf7e8d073fb37e9309

5 years agologging: add a NEVER_DISCARD flag to LogWriter
Adam Simpkins [Tue, 20 Jun 2017 18:01:55 +0000 (11:01 -0700)]
logging: add a NEVER_DISCARD flag to LogWriter

Summary:
Add a flag to the LogWriter so we can ensure that particular messages are never
discarded, even when the LogWriter is throttling messages.

This functionality will be necessary to implement `FB_LOG(FATAL)` to that we
can avoid discarding the reason for crashing.

Reviewed By: wez

Differential Revision: D5189498

fbshipit-source-id: dc4322ea5ba449a341cdbdc32afb0ed466019801

5 years agologging: add a small example program
Adam Simpkins [Tue, 20 Jun 2017 18:01:53 +0000 (11:01 -0700)]
logging: add a small example program

Summary:
This adds a small example program which demonstrates using the logging library.

This gives a very basic example of how the library is intended to be used, and
can also be used to play around with controlling the log levels from the
command line argument.

Reviewed By: wez

Differential Revision: D5083104

fbshipit-source-id: ab09c6c88db33065f6e39f35b28014f2a6153cef

5 years agoDelete folly/futures/OpaqueCallbackShunt.h
Christopher Dykes [Tue, 20 Jun 2017 17:11:42 +0000 (10:11 -0700)]
Delete folly/futures/OpaqueCallbackShunt.h

Summary: It is not used anywhere, not even in tests, so kill it.

Reviewed By: yfeldblum

Differential Revision: D5280800

fbshipit-source-id: 7e6a308bf09198548b77dcc1bfacc0ee95eb4887

5 years agoRevert D5278412: [Folly] Merge StringBase.cpp and String.cpp
Christopher Dykes [Tue, 20 Jun 2017 04:23:56 +0000 (21:23 -0700)]
Revert D5278412: [Folly] Merge StringBase.cpp and String.cpp

Summary: This reverts commit 07639e155421f31a6cc7ed16cba2034750e44325

Differential Revision: D5278412

fbshipit-source-id: 4b4d60f1f472ebe2e68e8eea1a31f42bc55c60ed

5 years agoMerge StringBase.cpp and String.cpp
Christopher Dykes [Tue, 20 Jun 2017 00:24:45 +0000 (17:24 -0700)]
Merge StringBase.cpp and String.cpp

Summary: The only reason these were split was because of the use of `folly::format`

Reviewed By: yfeldblum

Differential Revision: D5278412

fbshipit-source-id: 07639e155421f31a6cc7ed16cba2034750e44325

5 years agoSwitch pid_t to be defined as int rather than void*
Christopher Dykes [Tue, 20 Jun 2017 00:23:19 +0000 (17:23 -0700)]
Switch pid_t to be defined as int rather than void*

Summary: Because an `int` plays much much nicer with code already written for Windows than `void*` does. It takes quite a bit of hackery to make it possible with the pthread implementation we support using, but it is possible and is worth the effort.

Reviewed By: yfeldblum

Differential Revision: D5276968

fbshipit-source-id: 4cd0d3120a4f19976e76853ef4b337e96d5005a8

5 years agoMake semaphore.h a non-portable header
Christopher Dykes [Mon, 19 Jun 2017 21:34:21 +0000 (14:34 -0700)]
Make semaphore.h a non-portable header

Summary: On Windows it has include order requirements.

Reviewed By: yfeldblum

Differential Revision: D5277151

fbshipit-source-id: a6cf923d78e5944ac34e885e32a7cf072f904720

5 years agoDocument the preferred alternatives in Foreach.h
Phil Willoughby [Mon, 19 Jun 2017 12:51:34 +0000 (05:51 -0700)]
Document the preferred alternatives in Foreach.h

Reviewed By: yfeldblum

Differential Revision: D5228760

fbshipit-source-id: bcad4237997e611e06c93b476c5b673dc93c0621

5 years agoFix the build of detail/CacheLocality.h on Windows
Christopher Dykes [Mon, 19 Jun 2017 07:04:43 +0000 (00:04 -0700)]
Fix the build of detail/CacheLocality.h on Windows

Summary: It was broken by D5139886 which added a call to `std::__throw_bad_alloc` without first including the correct portability header.

Reviewed By: yfeldblum

Differential Revision: D5273016

fbshipit-source-id: cac1772f41ae5942a29cefa69fb07d9cf94b5aab

5 years agoFix the CMake build with tests enabled
Christopher Dykes [Mon, 19 Jun 2017 05:42:40 +0000 (22:42 -0700)]
Fix the CMake build with tests enabled

Summary: I broke it when I moved the stats tests around.

Reviewed By: yfeldblum

Differential Revision: D5258526

fbshipit-source-id: 70982ffaaed89a548e943e4b011b013c89808666

5 years agoMake DestructorCheck::Safety no-copy, no-move v2017.06.19.00
Alan Frindell [Mon, 19 Jun 2017 04:01:03 +0000 (21:01 -0700)]
Make DestructorCheck::Safety no-copy, no-move

Summary: It's not safe because of how it manipulates the linked list.  The intended usage is to declare it locally on the stack

Reviewed By: yfeldblum

Differential Revision: D5267507

fbshipit-source-id: 07a55eaee2c6a266bb233432f6a3383236594195

5 years agoFix a few exception_wrapper tests under MSVC
Christopher Dykes [Sun, 18 Jun 2017 18:14:46 +0000 (11:14 -0700)]
Fix a few exception_wrapper tests under MSVC

Summary: They were incorrectly comparing against string literals rather than the actual demangled names. MSVC includes `class/struct` as part of the mangled name, so they also appear in the demangled name, in contrast to GCC/Clang, which don't.

Reviewed By: yfeldblum

Differential Revision: D5271087

fbshipit-source-id: 41fa0cebe098b1b935e1f8b1af5882c412cf6254

5 years agoworking and warning tweak for UninitializedMemoryHacks
Nathan Bronson [Sat, 17 Jun 2017 16:49:56 +0000 (09:49 -0700)]
working and warning tweak for UninitializedMemoryHacks

Summary:
Improve documentation and use the correct pragma for warning
on the unsupported path.

Reviewed By: ot

Differential Revision: D5264582

fbshipit-source-id: 710ee46fef6d8f37f665f4bb6f7e4c5dc0b27436

5 years agoRevert D4567233: [Folly] Drop support for Clang < 3.9
Christopher Dykes [Fri, 16 Jun 2017 20:59:23 +0000 (13:59 -0700)]
Revert D4567233: [Folly] Drop support for Clang < 3.9

Summary: This reverts commit cd94c711f0390d249b6736ab292721a477022b02

Differential Revision: D4567233

fbshipit-source-id: a10aff6b4c231666aec9f40ab65e881c3cd870dc

5 years agoImprove get_fiber_manager_map_*() error reporting
Peter DeLong [Fri, 16 Jun 2017 17:32:35 +0000 (10:32 -0700)]
Improve get_fiber_manager_map_*() error reporting

Summary:
get_fiber_manager_map_vevb() and get_fiber_manager_map_evb() don't
provide very useful feedback when called with an empty map (just the former) or
with a program that doesn't have debug symbols (both)

Reviewed By: andriigrynenko

Differential Revision: D5260114

fbshipit-source-id: c44b4e279e5c88dc08507b969339a4befc23d79b

5 years agoFix class member shadowing in folly::ProducerConsumerQueue
Uladzislau Paulovich [Fri, 16 Jun 2017 09:27:40 +0000 (02:27 -0700)]
Fix class member shadowing in folly::ProducerConsumerQueue

Summary: Previous version failed to compile with "-Werror=shadow" flag, this commit fixes the problem.

Reviewed By: WillerZ

Differential Revision: D5255814

fbshipit-source-id: c1252474ed5415b47759022bcbabc78c1639e10a

5 years agoRemove the relative comparison operators on LogLevel
Christopher Dykes [Fri, 16 Jun 2017 08:12:53 +0000 (01:12 -0700)]
Remove the relative comparison operators on LogLevel

Summary: MSVC, GCC, and Clang define the relative comparison operators by default on `enum class` values, however under MSVC defining them explicitly causes the compiler to fail to resolve which operator to use, so just remove them entirely, as they aren't needed.

Reviewed By: simpkins

Differential Revision: D5259835

fbshipit-source-id: 1b91db5de0fc47320daabfdd7132ae910887ff0f

5 years agoDon't declare caught exceptions when not actually used
Christopher Dykes [Fri, 16 Jun 2017 02:47:37 +0000 (19:47 -0700)]
Don't declare caught exceptions when not actually used

Summary: There are a couple of places where the caught exception was being given a name that was never used, generating a warning under MSVC.

Reviewed By: simpkins

Differential Revision: D5260131

fbshipit-source-id: f82c6bd1266f6a4c916594ec3ac94edc9a2e48fe

5 years agoDon't try to use _CrtDbgReport in the logging framework
Christopher Dykes [Thu, 15 Jun 2017 23:54:38 +0000 (16:54 -0700)]
Don't try to use _CrtDbgReport in the logging framework

Summary: Glog doesn't try to, so there's no real reason to try and be smart. Just do the same thing as every other platform for now until we have reason to do otherwise.

Reviewed By: simpkins

Differential Revision: D5259435

fbshipit-source-id: dcdf55f51f2e13945afd351cb45e9c1a22f56218

5 years agoEnsure LogWriter::writeMessage(std::string&&) is in the overload set of ImmediateFile...
Christopher Dykes [Thu, 15 Jun 2017 23:53:09 +0000 (16:53 -0700)]
Ensure LogWriter::writeMessage(std::string&&) is in the overload set of ImmediateFileWriter

Summary: As MSVC correctly warns via C4266, the overload sets of functions in base classes are not part of the overload set in derived classes where you've declared an overload (or override) unless you've explicitly imported it.

Reviewed By: simpkins

Differential Revision: D5260056

fbshipit-source-id: bbbeeea3c13201a3a6eba6e62cfa5a49a9470d43

5 years agoInclude the time portability header in GlogStyleFormatter.cpp
Christopher Dykes [Thu, 15 Jun 2017 23:52:29 +0000 (16:52 -0700)]
Include the time portability header in GlogStyleFormatter.cpp

Summary: It is needed on Windows for the `localtime_r` function.

Reviewed By: simpkins

Differential Revision: D5259944

fbshipit-source-id: e380ecf42252ecabec168cf87668ace3ff4c0a9f

5 years agoExplicitly initialize the base class of LogStream
Christopher Dykes [Thu, 15 Jun 2017 23:51:24 +0000 (16:51 -0700)]
Explicitly initialize the base class of LogStream

Summary:
The standard does not define that `std::ostream` should have a default constructor, so this was invalid and was an error on Windows.
This explicitly initializes the base class by passing a `nullptr`.

Reviewed By: simpkins

Differential Revision: D5259690

fbshipit-source-id: b8914d73ff2682e1a4447b8338860259778b2247

5 years agoFix Observable to not trigger unneccessary refresh if the value didn't change
Andrii Grynenko [Thu, 15 Jun 2017 21:37:41 +0000 (14:37 -0700)]
Fix Observable to not trigger unneccessary refresh if the value didn't change

Differential Revision: D5251218

fbshipit-source-id: 1ceb37dd727e8ac2fd842e2c437cdaa9017221c8

5 years agofix issues with the open source tests
Adam Simpkins [Thu, 15 Jun 2017 18:28:06 +0000 (11:28 -0700)]
fix issues with the open source tests

Summary:
Fix build problems in the open source Makefiles that cause issues when running
`make check`

- The test subdirectory needs to be processed before experimental, since some
  tests in experimental/ depend on libfollytestmain from test/
- The stats/test Makefile had an incorrectly copy-and-pasted line for
  libgtest_la_SOURCES but didn't actually build a libgtest.la library.
- The test/ Makefile defined thread_id_test but forgot to add it to TESTS so
  that it would actually be run as part of "make check"

Reviewed By: Orvid

Differential Revision: D5249132

fbshipit-source-id: 5a71e1f72a39d5407b843a5876891c67238ec006

5 years agologging: add initialization convenience functions
Adam Simpkins [Thu, 15 Jun 2017 18:04:00 +0000 (11:04 -0700)]
logging: add initialization convenience functions

Summary:
Add a logging/Init.h header file with a couple convenience functions for
initializing log levels and log handlers.

This is pretty basic for now, but simplifies usage for small programs that just
want to easily initialize the logging library.

Reviewed By: wez

Differential Revision: D5083106

fbshipit-source-id: 73c1fd00df2eaf506b9c1485d6afd12570412a0f

5 years agologging: add GlogStyleFormatter
Adam Simpkins [Thu, 15 Jun 2017 18:03:58 +0000 (11:03 -0700)]
logging: add GlogStyleFormatter

Summary:
Add a LogFormatter implementation that logs messages using a similar format to
the glog library.

Reviewed By: wez

Differential Revision: D5083108

fbshipit-source-id: 75f0a6b78ce5406b4557d6c4394f033d5e019f71

5 years agologging: add AsyncFileWriter
Adam Simpkins [Thu, 15 Jun 2017 18:03:57 +0000 (11:03 -0700)]
logging: add AsyncFileWriter

Summary:
Add an AsyncFileWriter class that implements the LogWriter interface using a
separate I/O thread to write the log messages to a file descriptor.

This LogWriter implementation ensures that normal process threads will never
block due to logging I/O.  By default it will buffer up to 1MB of data.  If
log messages are generated faster than they can be written to the file, log
messages will be discarded once the buffer limit is exceeded.  The LogWriter
will emit a message into the file recording how many messages were discarded
where the dropped messages should have been.

The downside of this class is that unwritten log messages still in the buffer
will be lost when the program crashes.

Reviewed By: wez

Differential Revision: D5083107

fbshipit-source-id: c67226f4d0726675d480b03eae83a29c5c3431b2

5 years agologging: add ImmediateFileWriter
Adam Simpkins [Thu, 15 Jun 2017 18:03:55 +0000 (11:03 -0700)]
logging: add ImmediateFileWriter

Summary:
Add an ImmediateFileWriter class that implements the LogWriter interface by
immediately writing data it receives to a file descriptor.

This LogWriter implementation can be used by users that want to ensure log
messages are immediately flushed to a file as soon as they are written, at the
expense of possibly having logging I/O block their normal process threads.

Reviewed By: wez

Differential Revision: D5083105

fbshipit-source-id: 15cd071834e1784fe50e1d6d6ce403b9bc907047

5 years agologging: add a LoggerDB::internalWarning() function
Adam Simpkins [Thu, 15 Jun 2017 18:03:52 +0000 (11:03 -0700)]
logging: add a LoggerDB::internalWarning() function

Summary:
A few places in the logging code need to report issues when something goes
wrong in the logging code itself.  This adds a new LoggerDB::internalWarning()
method for use in these places.

This provides a default implementation that logs these warnings with
_CrtDbgReport() on Windows, and writes to stderr on non-Windows platforms.
The default implementation can also be overridden for programs that want some
alternative behavior.

Reviewed By: wez

Differential Revision: D5157731

fbshipit-source-id: e27f5587d58b3941fca8d5b38c8c3edbcb75d7f3

5 years agologging: add new RateLimiter helper class
Adam Simpkins [Thu, 15 Jun 2017 18:03:50 +0000 (11:03 -0700)]
logging: add new RateLimiter helper class

Summary:
Add a new RateLimiter API to the logging library, to support rate limiting
messages in the future.  I have included a single IntervalRateLimiter
implementation in this diff.  In the future we can add more implementations if
necessary, to mimic the functionality available with our older logging code, to
make it easier for users to convert to the new framework.

Note that RateLimiter is inside a `folly::logging` namespace, unlike most of
the other code in the logging library that lives directly in the `folly`
namespace.  I intentionally chose this since RateLimiter is a fairly generic
class name, and I wanted to distinguish it from other possible generic class
names in folly.  On the other hand, most of the other class names already start
with `Log`, so there seems to be no need to put them in a `logging` namespace.

Nothing is using this new API yet, but I will use it for some internal logging
APIs in an upcoming diff.  Later on I also plan to use it to implement
per-LogCategory rate limiting, and possibly per-LogHandler rate limiting.

Reviewed By: wez

Differential Revision: D5162805

fbshipit-source-id: 9b81c2f4544006cd392152a768296bce0c5daaa1

5 years agologging: add LogFormatter and LogWriter interfaces
Adam Simpkins [Thu, 15 Jun 2017 18:03:48 +0000 (11:03 -0700)]
logging: add LogFormatter and LogWriter interfaces

Summary:
This simplifies the LogHandler interface to a single generic `handleMessage()`,
and adds a `StandardLogHandler` implementation that defers to separate
`LogFormatter` and `LogWriter` objects.

The `LogFormatter` class is responsible for serializing the `LogMessage` object
into a string, and `LogWriter` is responsible for then doing something with
the serialized string.

This will make it possible in the future to have separate `LogWriter`
implementations that all share the same log formatting code.  For example, this
will allow separate `LogWriter` implementations for performing file I/O
immediately versus performing I/O asynchronously in a separate thread.

Reviewed By: yfeldblum

Differential Revision: D5083103

fbshipit-source-id: e3f5ece25e260c825d49a5eb30e942973d6b68bf

5 years agologging: add XLOG() and XLOGF() logging macros
Adam Simpkins [Thu, 15 Jun 2017 18:03:46 +0000 (11:03 -0700)]
logging: add XLOG() and XLOGF() logging macros

Summary:
This adds new `XLOG()` and `XLOGF()` macros to the logging library.  These are
similar to `FB_LOG()` and `FB_LOGF()`, but do not require a Logger argument.
Instead, the log category is picked automatically based on the current file
name.  The basic algorithm for picking the category name is to replace
directory separators with '.', and to strip off the filename extension.

For instance, all `XLOG()` statements in `src/foo/bar.cpp` will log to the
category `src.foo.bar`.  This also works correctly in header files: `XLOG()`
statements in `src/foo/mylib.h` will log to `src.foo.mylib`

This should generally result in a good log category hierarchy without the user
having to spend additional time picking category names--we simply re-use the
decisions that they already made for their directory layout.

In general I expect the `XLOG()` macros to be convenient enough that users will
use `XLOG()` in almost all cases rather than using `FB_LOG()`.

The log category name used by `XLOG()` statements can be overridden using
`XLOG_SET_CATEGORY()`, but this only works in .cpp files, not in header files.

Reviewed By: wez

Differential Revision: D4920257

fbshipit-source-id: 7ffafd9a4c87e6fb5eb35d86e0eb86ef1ed5be95

5 years agoAdd thread caching of hazard pointers. Benchmarks. Minor fixes, optimizations, and...
Maged Michael [Thu, 15 Jun 2017 17:43:26 +0000 (10:43 -0700)]
Add thread caching of hazard pointers. Benchmarks. Minor fixes, optimizations, and refactoring.

Summary:
Added support for thread caching of hazard pointers.
Added thread caching benchmarks.
Removed function call from hazptr_domain constexpr constructor.
Optimizations of memory order and code refactoring.

Reviewed By: davidtgoldblatt

Differential Revision: D5249070

fbshipit-source-id: 487fb23abccde228c3c726de4ac8e9f07bfa9498

5 years agoFix a memory leak in 1.1.0 related to initial_ctx
Neel Goyal [Thu, 15 Jun 2017 14:51:57 +0000 (07:51 -0700)]
Fix a memory leak in 1.1.0 related to initial_ctx

Summary: We would always up_ref the ctx before setting it as the initial_ctx.  This causes a leak in 1.1.0 since the initial_ctx isn't set in this version of OpenSSL.  We'll move the up_ref for the initial_ctx into the OpenSSLUtils helper.

Reviewed By: anirudhvr

Differential Revision: D5227823

fbshipit-source-id: b4490b317bd4dc8752a8d7e244fd153100a52aa6

5 years agofix rest of non-portable includes
James Sedgwick [Thu, 15 Jun 2017 06:03:28 +0000 (23:03 -0700)]
fix rest of non-portable includes

Summary: According to internal linter

Reviewed By: Orvid

Differential Revision: D5051010

fbshipit-source-id: febdeca05ac1cf3ad82617169f90912a445cf173

5 years agoFixing opt-asan/ubsan fail for folly contbuild
Jonathan Kron [Wed, 14 Jun 2017 15:46:15 +0000 (08:46 -0700)]
Fixing opt-asan/ubsan fail for folly contbuild

Summary: Flag caused compilation error on unused param, asserts optimized out in opt-asan/ubsan builds.  Replaced with DCHECK as per meyering's advice.

Reviewed By: meyering

Differential Revision: D5246089

fbshipit-source-id: dc0abda91f900dd98af31410f4667c52404997f5

5 years agoAdd fuzz testing
Nick Terrell [Tue, 13 Jun 2017 21:04:08 +0000 (14:04 -0700)]
Add fuzz testing

Summary: `ZSTD_decompress()` doesn't verify the uncompressed length in the frame as I thought, so throw an exception instead of `DCHECK()`.

Reviewed By: meyering

Differential Revision: D5234576

fbshipit-source-id: f72cf085a7267de32ce13553ce7ebbfe3b8a3f05

5 years agoremove misleading comment in Partial.h
Sven Over [Tue, 13 Jun 2017 16:17:06 +0000 (09:17 -0700)]
remove misleading comment in Partial.h

Summary:
The removed comment cites a gcc compiler bug as a reason to spell
out the return type of the three operator() methods. While that
compiler bug can lead to compiler errors in some situations,
it is not the only reason why we need the return type arrow
notation.

If the compiler tries to instantiate operator() with a set of
parameters that the embedded function does not accept, then the
substitution failure is inside decltype, which is not a compiler
error (SFINAE) but just means that the Partial class doesn't define
operator() for that parameter signature. Without decltype, the body
of operator() is ill-formed and that is a compiler error.

Reviewed By: meyering

Differential Revision: D5236537

fbshipit-source-id: 7daa0fbcf128a2c7c5c36532a5402deba394e1b8

5 years agoAdd toFullyQualifiedAppend() methods
Nick Terrell [Tue, 13 Jun 2017 06:12:42 +0000 (23:12 -0700)]
Add toFullyQualifiedAppend() methods

Summary: A common use case for `IPAddress::toFullyQualified()` is to print a `<ip>:<port>` string. It is faster to reserve enough space for both beforehand than create 2 strings.

Reviewed By: yfeldblum

Differential Revision: D5224454

fbshipit-source-id: 4536f89a9d51d39dd9fd970c753ecb8ecced5d22

5 years agoFix decompression of truncated data
Nick Terrell [Tue, 13 Jun 2017 02:00:54 +0000 (19:00 -0700)]
Fix decompression of truncated data

Summary: During decompression, when the data is truncated, `StreamCodec::doUncompress()` loops forever, since it doesn't check forward progress. `Bzip2Codec` does the same.

Reviewed By: chipturner

Differential Revision: D5233052

fbshipit-source-id: 8797a7f06d9afa494eea292a8a5dc980c7571bd0

5 years agologging: if folly::format() fails, also log the arguments
Adam Simpkins [Tue, 13 Jun 2017 01:33:06 +0000 (18:33 -0700)]
logging: if folly::format() fails, also log the arguments

Summary:
If the folly::sformat() call fails in an FB_LOGF() statement, make a
best-effort attempt to log the format arguments as well, in addition to the
format string.  For each argument, folly::to<std::string>() is use if it is
supported for this argument.

This will help ensure that the arguments that were being logged aren't lost
even if the format string was incorrect.

Reviewed By: WillerZ

Differential Revision: D5082978

fbshipit-source-id: 0d56030e639cd7e8f2242bb43646ab4248c6a877

5 years agologging: add support for streaming operators
Adam Simpkins [Tue, 13 Jun 2017 01:33:05 +0000 (18:33 -0700)]
logging: add support for streaming operators

Summary:
Update the logging library so that FB_LOG() and FB_LOGF() also accept
streaming style arguments:

  FB_LOG(logger) << "logs can be written like this now: " << 1234;
  FB_LOG(logger, "both styles can be used") << " together";
  FB_LOGF(logger, "test: {}, {}", 1234, 5678) << " and with FB_LOGF() too";

Streaming support does make the upcoming XLOG() macros more complicated to
implement, since the macro has to expand to a single ternary `?` expression.
However I was able to come up with a solution that works and has minimal
overhead.

Reviewed By: wez

Differential Revision: D5082979

fbshipit-source-id: 11734e39c02ad28aceb44bbfbd137d781caa30cf

5 years agoadd a new logging library
Adam Simpkins [Tue, 13 Jun 2017 01:33:03 +0000 (18:33 -0700)]
add a new logging library

Summary:
This begins adding a new logging library for efficient, hierarchical logging.

This includes the basic library framework, plus a README file with a brief
overview and the motivation for creating a new logging library.

Reviewed By: wez

Differential Revision: D4911867

fbshipit-source-id: 359623e11feeaa547f3ac2c369bf806ee6996554

5 years agofix build due to stats test refactor
Theo Najim [Mon, 12 Jun 2017 18:27:38 +0000 (11:27 -0700)]
fix build due to stats test refactor

Summary: Closes https://github.com/facebook/folly/pull/609

Differential Revision: D5228342

Pulled By: Orvid

fbshipit-source-id: edb81284a0401a4ea267310e1ca01c390bcdfb32

5 years agofix FutexTest v2017.06.12.00
Tianjiao Yin [Sat, 10 Jun 2017 05:03:48 +0000 (22:03 -0700)]
fix FutexTest

Summary: 2_ms seems too short. I am not sure whether it's feasible to check whether thread is waiting for an address. We could wait for longer to reduce false alarm.

Reviewed By: nbronson

Differential Revision: D5220819

fbshipit-source-id: 42f31206e9cb7f9addaa049d0e7cd995f6735f6c

5 years agoLet SubprocessError inherit std::runtime_error
Yedidya Feldblum [Sat, 10 Jun 2017 03:19:01 +0000 (20:19 -0700)]
Let SubprocessError inherit std::runtime_error

Summary:
[Folly] Let `SubprocessError` inherit `std::runtime_error`.

As an added bonus, this gives it `std::runtime_error`'s refcounted string for cheap copies.

Reviewed By: ericniebler

Differential Revision: D5216758

fbshipit-source-id: 43298e06f02cfd88abf2d73f7aa16117a6cb052b

5 years agoReorganize the stats directory
Christopher Dykes [Sat, 10 Jun 2017 02:48:51 +0000 (19:48 -0700)]
Reorganize the stats directory

Summary: The source and tests for the stats directory was spread across folly/detail and folly/test, move them into folly/stats directly instead.

Reviewed By: yfeldblum

Differential Revision: D5216810

fbshipit-source-id: 00a8bb95a4f7830d0bc46b3f914f256a37833b78

5 years agoapply clang-tidy modernize-use-override
Victor Gao [Fri, 9 Jun 2017 18:26:26 +0000 (11:26 -0700)]
apply clang-tidy modernize-use-override

Summary: Use clang-tidy to mechanically add missing `override` and remove redundant `virtual`.

Reviewed By: igorsugak

Differential Revision: D5211868

fbshipit-source-id: 6a85f7c4a543a4c9345ec5b0681a8853707343dc

5 years agoapply clang-tidy modernize-use-override
Victor Gao [Fri, 9 Jun 2017 18:23:25 +0000 (11:23 -0700)]
apply clang-tidy modernize-use-override

Summary: Use clang-tidy to mechanically add missing `override` and remove redundant `virtual`.

Reviewed By: igorsugak

Differential Revision: D5211868

fbshipit-source-id: 4118c4c72f8ec3485507f69679f7e852b3eaeb73

5 years agoSaner test exception in ExceptionWrapperTest.cpp
Yedidya Feldblum [Fri, 9 Jun 2017 06:45:51 +0000 (23:45 -0700)]
Saner test exception in ExceptionWrapperTest.cpp

Summary:
[Folly] Saner test exception in `ExceptionWrapperTest.cpp`.

Make construction do the work and make `what()` free.

Reviewed By: andrewjcg

Differential Revision: D5216707

fbshipit-source-id: bfb4c2473a61ff7da7a3a01adc85facae30d6586

5 years agoSome more OpenSSL 1.1.0 compat APIs
Anirudh Ramachandran [Fri, 9 Jun 2017 06:06:24 +0000 (23:06 -0700)]
Some more OpenSSL 1.1.0 compat APIs

Summary: Add a few more compatibility wrappers for pre-1.1.0 APIs

Reviewed By: yfeldblum

Differential Revision: D5194164

fbshipit-source-id: ae8db08c31370eca729df2927798b6f4d99ee70c

5 years agoApply clang-format to folly/gen/ (template decls)
Yedidya Feldblum [Fri, 9 Jun 2017 05:34:09 +0000 (22:34 -0700)]
Apply clang-format to folly/gen/ (template decls)

Summary: [Folly] Apply `clang-format` to `folly/gen/` (template decls).

Reviewed By: Orvid

Differential Revision: D5216229

fbshipit-source-id: 1659f1944ccde4de39bccd189bb6490395cf29c2

5 years agoFix OpenSSLUtils to not include headers the wrong way
Christopher Dykes [Fri, 9 Jun 2017 04:15:44 +0000 (21:15 -0700)]
Fix OpenSSLUtils to not include headers the wrong way

Summary: This is entirely the wrong way to include these headers, and is being included within an anon namespace so just wrong.

Reviewed By: yfeldblum

Differential Revision: D5215065

fbshipit-source-id: 499df58c0202f7a1d4482eaa6a0e8b2e1535c763

5 years agoApply clang-format to folly/experimental/exception_tracer/ (headers)
Yedidya Feldblum [Fri, 9 Jun 2017 03:59:50 +0000 (20:59 -0700)]
Apply clang-format to folly/experimental/exception_tracer/ (headers)

Summary:
[Folly] Apply `clang-format` to `folly/experimental/exception_tracer/` (headers).

With some manual rearrangement in the places where `clang-format` does awkward things, with the result that `clang-format` over the listed directory becomes a no-op.

Reviewed By: Orvid

Differential Revision: D5215212

fbshipit-source-id: 5570f02c238b1874adbc2ff4150f465c947ad0e6

5 years agoApply clang-format to folly/fibers/
Yedidya Feldblum [Thu, 8 Jun 2017 22:26:53 +0000 (15:26 -0700)]
Apply clang-format to folly/fibers/

Summary:
[Folly] Apply `clang-format` to `folly/fibers/`

With some manual rearrangement in the places where `clang-format` does awkward things, with the result that `clang-format` over `folly/fibers/` becomes a no-op.

Reviewed By: igorsugak

Differential Revision: D5178118

fbshipit-source-id: ae65ff1902666ba9106e18f916bb1d10e6406bf4

5 years agoSwitch unguarded #pragmas to use portability macros
Christopher Dykes [Thu, 8 Jun 2017 21:50:17 +0000 (14:50 -0700)]
Switch unguarded #pragmas to use portability macros

Summary: Otherwise downstream Windows users have to explicitly disable MSVC's warnings about unknown pragmas.

Reviewed By: yfeldblum

Differential Revision: D5211415

fbshipit-source-id: 42871e03895010818c7e1cb6e57c1885970e98c2

5 years agoRefer to nullptr not NULL
Christopher Dykes [Thu, 8 Jun 2017 21:48:12 +0000 (14:48 -0700)]
Refer to nullptr not NULL

Summary:
Folly is a C++ library not a C library, and (almost) universally uses `nullptr` everywhere, so refer to `nullptr` rather than `NULL`.
This also fixes the 1 place in our actual code where we were using `NULL` rather than `nullptr`.

Reviewed By: yfeldblum

Differential Revision: D5204766

fbshipit-source-id: 2a5d5011a28d7d5dd48789f60643a656f51b9732

5 years agoExtend the -Warray-bounds workaround in FixedString.h to gcc-5.
Eric Niebler [Thu, 8 Jun 2017 19:47:55 +0000 (12:47 -0700)]
Extend the -Warray-bounds workaround in FixedString.h to gcc-5.

Summary: FixedString.h has a workaround for a gcc bug regarding erroneous -Werror=array-bounds errors. This commit extends the workaround to the gcc-5 series, which sadly is still broken. Life gets better with gcc-6.

Reviewed By: luciang

Differential Revision: D5210633

fbshipit-source-id: 4fcf080d18ddc573636a3e58b61a97167a8c06ce

5 years agomake extent_hooks static.
Qi Wang [Thu, 8 Jun 2017 17:26:32 +0000 (10:26 -0700)]
make extent_hooks static.

Summary:
Make extent hooks static and avoid calling arena_destroy, in case there are
alive allocations not freed yet.

Reviewed By: yfeldblum

Differential Revision: D5208102

fbshipit-source-id: 612c772347cd90227fa303fd0b059edbaeb7d4e2

5 years agoImplement __builtin_popcount under MSVC
Christopher Dykes [Wed, 7 Jun 2017 23:03:27 +0000 (16:03 -0700)]
Implement __builtin_popcount under MSVC

Summary:
I thought I had already implemented this, but apparently I had only implemented the ll variable. Whoops.
This implements the 32-bit version which fixes the build on Windows.

Reviewed By: yfeldblum

Differential Revision: D5203680

fbshipit-source-id: 02b133db59e232cac586944b0ffc0e8bbf5f533a

5 years agoExclude experimental/hazptr/bench from the globs used in the CMake build
Christopher Dykes [Wed, 7 Jun 2017 23:01:52 +0000 (16:01 -0700)]
Exclude experimental/hazptr/bench from the globs used in the CMake build

Summary: The directory contains benchmarks and multiple `main`'s so shouldn't be included as part of the main folly library.

Reviewed By: yfeldblum

Differential Revision: D5204132

fbshipit-source-id: 7289cb9c2ec292c13df39c9e7b48eca4166ca951

5 years agoPass Try by rvalue ref in onError implementation
Yedidya Feldblum [Wed, 7 Jun 2017 21:39:54 +0000 (14:39 -0700)]
Pass Try by rvalue ref in onError implementation

Summary:
[Folly] Pass `Try` by rvalue ref in `onError` implementation.

As is done in other overloads and in `thenImplementation`. For consistency.

Reviewed By: spacedentist

Differential Revision: D5199302

fbshipit-source-id: c78db0c36fc7a0b846ca5e44fc237422c9203a24

5 years agoUri: switch to std::string over fbstring take 2
Sonia Mar [Wed, 7 Jun 2017 18:30:55 +0000 (11:30 -0700)]
Uri: switch to std::string over fbstring take 2

Summary: Update tuple generated for hash/equal_to for `folly::Uri` to also use std::string

Reviewed By: Orvid

Differential Revision: D5199266

fbshipit-source-id: bcd619dae01ffbf82fbb5c8987b0e87103572452

5 years agoAdd OpenSSLCertUtils functions for DER encoding/decoding.
Kyle Nekritz [Wed, 7 Jun 2017 14:31:37 +0000 (07:31 -0700)]
Add OpenSSLCertUtils functions for DER encoding/decoding.

Reviewed By: anirudhvr

Differential Revision: D5193205

fbshipit-source-id: 5b427ee4f31008518078f5e54e85c0f0f2201da5

5 years agoShrink the implementations of then and onError
Yedidya Feldblum [Wed, 7 Jun 2017 09:33:31 +0000 (02:33 -0700)]
Shrink the implementations of then and onError

Summary:
[Folly] Shrink the implementations of `then` and `onError`.

Just a small refactor.

Reviewed By: spacedentist

Differential Revision: D5180784

fbshipit-source-id: a399d18500a2b4c5d8f24dee54891cca802b4461

5 years agoTry::tryGetExceptionObject
Yedidya Feldblum [Wed, 7 Jun 2017 07:05:51 +0000 (00:05 -0700)]
Try::tryGetExceptionObject

Summary:
[Folly] `Try::tryGetExceptionObject`.

If the `Try` has a `std::exception` for the non-template overload or a `From` where `is_convertible<From*, Ex*>` for the template overload parameterized by `typename Ex`, returns a pointer to it. Otherwise returns `nullptr`.

Reviewed By: spacedentist

Differential Revision: D5180848

fbshipit-source-id: b7593b7459259f9a1e1e78b36d548e55d086e722

5 years agoRemove exception_wrapper::throwException
Yedidya Feldblum [Wed, 7 Jun 2017 05:16:39 +0000 (22:16 -0700)]
Remove exception_wrapper::throwException

Summary:
[Folly] Remove `exception_wrapper::throwException`.

It has been renamed to `throw_exception`.

Reviewed By: Orvid

Differential Revision: D5181005

fbshipit-source-id: 442b3fd9bdaa8170db504a81651cbb085ea50624

5 years agoAdd zlib/gzip streaming support
Nick Terrell [Wed, 7 Jun 2017 04:28:09 +0000 (21:28 -0700)]
Add zlib/gzip streaming support

Summary:
Add streaming interface to the ZlibCodec.
Implement ZlibStreamCodec::doCompress() and ZlibStreamCodec::doUncompress() using the streaming interface. fbgs CodecType::ZLIB and check that no caller requires thread-safety.
I found one caller, but it was fixed in D5090855.

Reviewed By: yfeldblum

Differential Revision: D5169338

fbshipit-source-id: 40478e162143623ad28fd8bc937d0195521f13fe

5 years agoSwitch folly::Uri to use std::string rather than fbstring
Christopher Dykes [Wed, 7 Jun 2017 02:53:27 +0000 (19:53 -0700)]
Switch folly::Uri to use std::string rather than fbstring

Summary: Using fbstring directly is not really worthwhile in the vast majority of cases, especially when most places just convert it into an `std::string` immediately.

Reviewed By: yfeldblum

Differential Revision: D5187835

fbshipit-source-id: 6f7d71612d3765e8b501d2432cd6281369bfe0fa

5 years agoAdd a version of async_test using openssl 1.1.0
Anirudh Ramachandran [Wed, 7 Jun 2017 01:24:35 +0000 (18:24 -0700)]
Add a version of async_test using openssl 1.1.0

Summary:
Now that it's possible, let's add some 1.1.0 tests to avoid regressing
1.1.0 support

Reviewed By: yfeldblum

Differential Revision: D5167246

fbshipit-source-id: ba12414504131697d4e0757c9c340a66f810acd4

5 years agoFix destruction order problem in getCoreAllocator
Giuseppe Ottaviano [Wed, 7 Jun 2017 00:32:12 +0000 (17:32 -0700)]
Fix destruction order problem in getCoreAllocator

Summary: We cannot guarantee the correct destruction order of the Meyers singleton owning the allocator, and the objects allocated with it, so we just leak it using `Indestructible`.

Reviewed By: djwatson

Differential Revision: D5196227

fbshipit-source-id: ec07ab1e21af7814194883b252d45aa36d2a04b1

5 years agoRename exception_wrapper::get_object to get_exception
Yedidya Feldblum [Tue, 6 Jun 2017 22:02:17 +0000 (15:02 -0700)]
Rename exception_wrapper::get_object to get_exception

Summary:
[Folly] Rename `exception_wrapper::get_object` to `get_exception`.

This will be a template overload of the existing non-template member `get_exception`, but it should just work.

Reviewed By: spacedentist

Differential Revision: D5180845

fbshipit-source-id: 28e16e039ec6a7939b38f217b4ad5c6bc803f515

5 years agoadditional minor cleanup to the TimeUtil code
Adam Simpkins [Tue, 6 Jun 2017 21:09:41 +0000 (14:09 -0700)]
additional minor cleanup to the TimeUtil code

Summary:
- Fix TimePoint to use steady_clock rather than system_clock
- Use nanoseconds instead of milliseconds in most locations

This code should probably also be moved into a folly::testing namespace or
something.  This class is pretty specific to unit tests, and `folly::TimePoint`
seems like too generic of a name for it.  However, just to keep things simple
I'm not doing that as part of this diff yet.

Reviewed By: yfeldblum

Differential Revision: D5175630

fbshipit-source-id: 26490fc7ff1b25fb86f09309e81108828cd0f091

5 years agoupdate TimeUtil to use getOSThreadID()
Adam Simpkins [Tue, 6 Jun 2017 21:09:40 +0000 (14:09 -0700)]
update TimeUtil to use getOSThreadID()

Summary:
Use the newly introduced getOSThreadID() function to get a thread ID for
determining thread scheduling information on Linux.

Reviewed By: yfeldblum

Differential Revision: D5173596

fbshipit-source-id: 3e6fa352c3774986b0ac1f93bd6c7e4a29e08471

5 years agofix timeWaiting computation in TimeUtil
Adam Simpkins [Tue, 6 Jun 2017 21:09:39 +0000 (14:09 -0700)]
fix timeWaiting computation in TimeUtil

Summary:
Some of the code in TimeUtil.cpp computes how long the current thread has spent
waiting to be schedule on the CPU, so we can discount this time when testing
timeout behavior.

It appears that this code has been broken for a fairly long time.  Older Linux
kernels reported values in /proc/<pid>/schedstat using jiffies.  However, it
appears that this was changed in Linux 2.6.23 to report values using
nanoseconds.  (It looks to me like this was changed in commit 425e0968a25f,
which purports to simply be moving code from sched.c to sched_stats.h,
but it appears to also have also changed this behavior.)

This updates TimeUtil.cpp to compute the value correctly in modern Linux
kernels.

Reviewed By: yfeldblum

Differential Revision: D5173598

fbshipit-source-id: 98436d3a687400bf68661156ffce93eea7232632

5 years agoCore-local allocator
Dave Watson [Tue, 6 Jun 2017 21:06:53 +0000 (14:06 -0700)]
Core-local allocator

Summary:
Adds a core-local allocator to CacheLocality.  Multiple objects using
cache locality may share the same allocator, and allocate things less than
cacheline sized, without incurring additional false-sharing overhead.

Reviewed By: nbronson, ot

Differential Revision: D5139886

fbshipit-source-id: a9804662d6339829a12e0791f418dabd9678f1bf

5 years agoFix the build of MallctlHelperTest without JEMalloc
Christopher Dykes [Tue, 6 Jun 2017 20:30:54 +0000 (13:30 -0700)]
Fix the build of MallctlHelperTest without JEMalloc

Summary:
`kDecayCmd` was guarded such that it wasn't being defined at all if JEMalloc wasn't available.

Closes https://github.com/facebook/folly/issues/608

Reviewed By: interwq

Differential Revision: D5193721

fbshipit-source-id: 1a2666de70e90eac1e9f2e130d0452b01b3239fa

5 years agoAdd element construction/destruction hooks to IndexedMemPool
Victor Zverovich [Tue, 6 Jun 2017 20:10:49 +0000 (13:10 -0700)]
Add element construction/destruction hooks to IndexedMemPool

Summary: This diff adds a Traits template parameter to IndexedMemPool that allows more control over the lifetime management of individual elements, including mixes of lazy and eager recycle semantics (or colocation of different classes of data inside a single element). It also arranges that an index is not reported as isAllocated() until it has been passed to Traits::initialize and passed to Traits::onAllocate at least once, so code that is traversing valid indexes doesn't need to deal with the post-initialize but pre-onAllocate state (it must still deal with indexes that reported isAllocated() as true but have since been passed to onRecycle). The default behavior is unchanged.

Reviewed By: nbronson

Differential Revision: D5177462

fbshipit-source-id: e7d22c860ab6bf25083977dfb5a63955641c9cfb

5 years agoDrop a dead include of FBString.h from MemoryMapping.h
Christopher Dykes [Tue, 6 Jun 2017 18:00:02 +0000 (11:00 -0700)]
Drop a dead include of FBString.h from MemoryMapping.h

Summary: MemoryMapping doesn't use fbstring directly at all, so this include isn't needed.

Reviewed By: yfeldblum

Differential Revision: D5188209

fbshipit-source-id: d86b876a96343c830295bea1162580215a340815

5 years agoadd missing include to ThreadId.h
Adam Simpkins [Tue, 6 Jun 2017 17:22:59 +0000 (10:22 -0700)]
add missing include to ThreadId.h

Summary:
The syscall() function is defined in <unistd.h>
<sys/syscall.h> apparently only defines IDs to be used with syscall(), but does
not define the syscall() function itself.

This caused build failures for files that included ThreadId.h before unistd.h

Reviewed By: Orvid

Differential Revision: D5189658

fbshipit-source-id: 2ec8ea1d58f3fc14cf458a53ecaa811978527398

5 years agoUninitializedMemoryHacks
Nathan Bronson [Tue, 6 Jun 2017 16:08:51 +0000 (09:08 -0700)]
UninitializedMemoryHacks

Summary:
This diff adds helper functions that can resize std::string or std::vector
without constructing or initializing new elements.  They are designed
for retroactively optimizing code where touching every element twice
(or touching never-used elements once) shows up in profiling, and where
restructuring the code to use fixed-length arrays or IOBuf-s would
be difficult.

Implementations are provided for 5 string implementations (pre-c++11
libstdc++, libstdc++ with SSO, libc++, std::basic_fbstring, and MSVC)
and 3 vector implementations (libstdc++, libc++, and MSVC).  On an
unsupported platform you will hopefully get a #warn if you include
UninitializedMemoryHacks.h followed by a linker error if you actually
use it.

Reviewed By: yfeldblum

Differential Revision: D5102679

fbshipit-source-id: 536c00eabae4cdb8a0affe3e919a372f4dc51ac5

5 years agoUpdate MallctlHelperTest with jemalloc 5 APIs.
Qi Wang [Tue, 6 Jun 2017 06:06:11 +0000 (23:06 -0700)]
Update MallctlHelperTest with jemalloc 5 APIs.

Summary: Update decay time API to be jemalloc 4 & 5 compatible.

Reviewed By: yfeldblum

Differential Revision: D5188574

fbshipit-source-id: 8e3c26c8b6431efdd0ec7ff492a065bddb816b4b

5 years agoadd getOSThreadID()
Adam Simpkins [Tue, 6 Jun 2017 02:07:36 +0000 (19:07 -0700)]
add getOSThreadID()

Summary:
Add a getOSThreadID() call.  This is similar to getCurrentThreadID(), but
returns the OS-level thread ID.

Reviewed By: yfeldblum

Differential Revision: D5173627

fbshipit-source-id: 0e8695ecfc1e382d12526c1b7d1717be114c9f4a

5 years agoSwitch Bits.h to just use the popcount builtins directly
Christopher Dykes [Mon, 5 Jun 2017 23:38:24 +0000 (16:38 -0700)]
Switch Bits.h to just use the popcount builtins directly

Summary:
These builtins are available on all platforms under GCC, using the instruction directly when available and a fallback implementation otherwise.
They are implemented in the builtins portability header for MSVC.

Reviewed By: yfeldblum

Differential Revision: D5185106

fbshipit-source-id: a58305a6b99eb49bd165876a4a60c512a259b225

5 years agoRename folly/detail/Malloc.h to folly/detail/MallocImpl.h
Christopher Dykes [Mon, 5 Jun 2017 23:09:11 +0000 (16:09 -0700)]
Rename folly/detail/Malloc.h to folly/detail/MallocImpl.h

Summary: This makes the name of the header match the source file.

Reviewed By: yfeldblum

Differential Revision: D5186193

fbshipit-source-id: b1f9041fb730eacb91ad167e500c25df8f8ba947

5 years agoLet map-related functions be templated over key types
Yedidya Feldblum [Mon, 5 Jun 2017 21:35:46 +0000 (14:35 -0700)]
Let map-related functions be templated over key types

Summary:
[Folly] Let map-related functions be templated over key types.

This supports the case where map members like `find` may be overloaded or templated on types other than the map's `key_type const&`.

In C++14, standard library map types gain template overloads for `find` and for other members, permitting lookups without constructing `key_type` instances. This is useful primarily when `key_type` is expensive but another object type comparable with `key_type` is inexpensive. As a common example, a `key_type` of `std::string` is expensive in the general case because it allocates storage on the heap but `std::string_view` (C++17), when constructed with non-allocated `char const*, std::size_t`, would be inexpensive.

Reviewed By: terrelln

Differential Revision: D5145879

fbshipit-source-id: 979b75bfe55661aab11d5302da1bcd7830abd5af

5 years agoFixed documentation in folly/Random.h
Peter Goldsborough [Mon, 5 Jun 2017 21:17:02 +0000 (14:17 -0700)]
Fixed documentation in folly/Random.h

Summary:
The docs give `ThreadLocalPRNG rng = Random::threadLocalPRNG()` as an example
of creating a thread local PRNG, but `Random::threadLocalPRNG()` does not/no longer
exist. I think it's just `folly::ThreadLocalPRNG` right now.

Reviewed By: yfeldblum

Differential Revision: D5184992

fbshipit-source-id: 63a9ef62b32fca42088abec419ae53531910cc82

5 years agoFix JemallocNodumpAllocator destructor w/ jemalloc 5.0.
Qi Wang [Mon, 5 Jun 2017 19:45:43 +0000 (12:45 -0700)]
Fix JemallocNodumpAllocator destructor w/ jemalloc 5.0.

Summary: Destroy created arena which has extent_hooks linked.

Reviewed By: jasone

Differential Revision: D5177535

fbshipit-source-id: 7d75f4276fc174c4d4ce1102dfb8ec8c27c1f56d

5 years agoMake StringKeyed* more complete
Phil Willoughby [Mon, 5 Jun 2017 17:58:29 +0000 (10:58 -0700)]
Make StringKeyed* more complete

Summary: Add `count()` and `swap()` methods. Ensure that `operator==` works.

Reviewed By: yfeldblum

Differential Revision: D5169393

fbshipit-source-id: a8025f6fdf251e38b0d2f27733d18967a55c6a15