8 years agoadd support for whenAll to waitWithSemaphore
Matt Dordal [Fri, 23 May 2014 16:19:20 +0000 (09:19 -0700)]
add support for whenAll to waitWithSemaphore

waitWithSemaphore currently doesn't support vector<Try<T>>, unless T is void.
Fix that, and also add a now-required void specialization.

Test Plan:
Add a test that uses vector<Try<bool>>, ensure that the tests compile
(and pass).

Reviewed By:

Subscribers: folly@lists, fugalh

FB internal diff: D1338528

Tasks: 4389473

8 years agoFuture::value() should throw when unset
Simon Martin [Wed, 21 May 2014 20:31:54 +0000 (13:31 -0700)]
Future::value() should throw when unset

Added a test to call Future::value() before the Promise value is set, expecting an exception.
In a dbg build the test failed due on the assertion in Optional::value().
In a opt build the test failed due as no exception was thrown.
There are 2 points where we could throw our exception:
a) Optional::value() - replacing the assertion
b) Future::value()

I'm not sure which location makes the most sense.
With the assertion in Optional it seems that adding the throw here would not be unexpected but this is outside the wangle code.
So as a first pass I've added the throw in Future::value(), and made a new WangleException for this.

Test Plan:
$ fbconfig folly/wangle
$ fbmake runtests

Reviewed By:

Subscribers: folly@lists, fugalh

FB internal diff: D1340886

8 years agoadded missing includes; also fixed lint issue with noexcept
Daniil Burdakov [Wed, 21 May 2014 17:52:03 +0000 (17:52 +0000)]
added missing includes; also fixed lint issue with noexcept

Summary: subj

Test Plan: unit tests

Reviewed By:

Subscribers: folly@lists

FB internal diff: D1341693

8 years agoMore opensource build fixes
Tudor Bosman [Tue, 20 May 2014 21:32:43 +0000 (14:32 -0700)]
More opensource build fixes

- libtool version
- get rid of tiny libraries
- add folly/gen and a bunch of stuff from experimental

Test Plan: built, built a program against it in a ubuntu vm

Reviewed By:

Subscribers: folly@lists, fugalh

FB internal diff: D1339920

8 years agomake BucketedTimeSeries::addValue() honor old timestamps
Adam Simpkins [Wed, 14 May 2014 20:40:33 +0000 (13:40 -0700)]
make BucketedTimeSeries::addValue() honor old timestamps

Previously BucketedTimeSeries()::addValue() documented that it required
time to move forwards.  If it was ever called with a timestamp older
than the most recent one it had seen, it would just use latestTime_ as
the time, and add the value to the most recent bucket.

This changes addValue() so that it always uses the timestamp passed in
by the caller.  If this time value refers to an old bucket that is still
being tracked, the data point will be added to that bucket.  If the time
value is older than the oldest currently tracked bucket, the data point
will be ignored, and false will be returned.

I did consider leaving the current addValue() behavior as-is, and
requiring a separate addHistoricalValue() for when users intentionally
want to try adding old data points.  However, it seems nicer to build
this into the existing addValue() function.  The old behavior of just
replacing the supplied time value seems potentially surprising to users.

This does change the behavior of addValue(), and therefore could affect
the behavior of some programs.  However, up until now no-one should have
been calling addValue() with old time values, as it wouldn't have done
what they want anyway.  I did a brief search through our code base, and
all call sites I saw always called addValue() with the current time.
(Most of the callers use wall clock time, so this change might affect
program behavior if the system time changes after the program starts.
We should ideally change our programs to use monotonic clocks instead.)

Test Plan:
Included a new unit test.

Also compared the timeseries_benchmark results before and after this
change.  Overall this new logic seems to be faster.  For the "all time"
case, the new code is over 2x as fast.  For the normal, non-all-time
case the new code is around 5% faster.

Reviewed By:

Subscribers: doug, folly@lists, net-systems@, exa

FB internal diff: D1338466

8 years agoSome opensource build fixes
Tudor Bosman [Tue, 20 May 2014 15:49:16 +0000 (08:49 -0700)]
Some opensource build fixes

- switch to new versions of ax_boost_*.m4
- versioning in libtool
- better checks in

Test Plan: built in an Ubuntu VM

Reviewed By:

Subscribers: folly@lists

FB internal diff: D1338957

8 years agoBuild up signal handler message before writing
Tudor Bosman [Mon, 19 May 2014 19:13:46 +0000 (12:13 -0700)]
Build up signal handler message before writing

Summary: So it doesn't interleave with whatever other threads write to stderr.

Test Plan: folly/experimental/symbolizer/test

Reviewed By:

Subscribers: folly@lists

FB internal diff: D1337029

8 years agofix waitWithSemaphore return type
Matt Dordal [Mon, 19 May 2014 20:03:28 +0000 (13:03 -0700)]
fix waitWithSemaphore return type

waitWithSemaphore always tried to return a value, which is not what the
underlying implementation did. If the value_type was an object, it would
fail to compile.

Test Plan: unit tests (added one to compile all the variants)

Reviewed By:

Subscribers: folly@lists, fugalh

FB internal diff: D1326916

8 years agoMake EventHandler::isPending const
Alex Landau [Thu, 15 May 2014 20:23:23 +0000 (13:23 -0700)]
Make EventHandler::isPending const

Summary: Because it just queries state

Test Plan: fbmake

Reviewed By:

Subscribers: folly@lists

FB internal diff: D1332397

8 years agoRemoved old FBVector compatibility functions
Nicholas Ormrod [Thu, 15 May 2014 17:36:38 +0000 (10:36 -0700)]
Removed old FBVector compatibility functions

Summary: FBVector still has some code for gcc-4.6. Removed it.

Test Plan:
fbconfig -r folly && fbmake runtests
fbconfig folly/test/stl_test && fbmake runtests (after enabling)

Reviewed By:

Subscribers: folly@lists, sdwilsh

FB internal diff: D1320358

8 years agoRevert "[folly::Subprocess] Set O_CLOEXEC by default when creating pipes to avoid...
Rocky Liu [Tue, 13 May 2014 23:40:54 +0000 (16:40 -0700)]
Revert "[folly::Subprocess] Set O_CLOEXEC by default when creating pipes to avoid race conditions resulting from concurrent Subprocess creations"

Summary: This reverts commit c2f089cf080f2b3effa9efa5e4708b9674437d45.

Test Plan: Compile && folly::Subprocess unit tests

Reviewed By:

FB internal diff: D1327952

8 years agoAlways #define _GNU_SOURCE to pull in pipe2() declarations
Rocky Liu [Tue, 13 May 2014 21:35:37 +0000 (14:35 -0700)]
Always #define _GNU_SOURCE to pull in pipe2() declarations

Summary: [folly::Subprocess] Always #define _GNU_SOURCE to pull in pipe2() declarations

Test Plan: Compile

Reviewed By:

FB internal diff: D1327004

8 years agoAdding a release function for ThreadLocalPtr.
Akshay Vaidya [Fri, 9 May 2014 17:07:05 +0000 (10:07 -0700)]
Adding a release function for ThreadLocalPtr.

ThreadLocalPtr manages the lifecycle of the object that is
stored with it. We have a use case where we sometimes want to transfer ownership
of the stored object to another thread by wrapping them with
unique_ptrs. Adding a release function, similar to to the
unique_ptr::release is the cleanest way for us to transfer ownership.

Test Plan:
I can do some on off testing using a command line tool, but I
was wondering about how to add some unit tests. Not sure when the folly
unit tests were.

Reviewed By:

FB internal diff: D1321588

8 years agoSet O_CLOEXEC by default when creating pipes to avoid race conditions resulting from...
Rocky Liu [Tue, 13 May 2014 18:43:59 +0000 (11:43 -0700)]
Set O_CLOEXEC by default when creating pipes to avoid race conditions resulting from concurrent Subprocess creations

[folly::Subprocess] Set O_CLOEXEC by default when creating pipes to avoid race conditions resulting from concurrent Subprocess creations

If multiple threads are creating Subprocess objects concurrently, the
write side file descriptor of the pipe created in the parent process
might be inherited into other child processes unintentionally and never
closed, causing the parent process to hang while reading from the read
side of its pipe, thinking the other side must have been closed.

The fix to the problem is to create the pipes and set O_CLOEXEC in
a single pipe2 call. Then the child could clear the O_CLOEXEC flag
selectively before calling exec().

Test Plan:
Existing unit tests of Subprocess
Added a new unit test which will hang in Subprocess constructor without
this fix.

Reviewed By:

FB internal diff: D1267396

8 years agoadd waitWithSemaphore to folly::wangle
Matt Dordal [Tue, 13 May 2014 17:37:45 +0000 (10:37 -0700)]
add waitWithSemaphore to folly::wangle

It may be useful to wait for a future to finish. This adds a utility function
to do so, returning a completed future.

NB: While it doesn't matter which thread executes the `then`, there does need
to be two threads. If not, this will deadlock forever. I'm not sure if there's
a way to detect/prevent that.

Test Plan: added some unit tests.

Reviewed By:

FB internal diff: D1319330

8 years agomissing exception
Elizabeth Smith [Tue, 13 May 2014 15:15:07 +0000 (08:15 -0700)]
missing exception

Summary: windows is also missing __throw_bad_alloc

Test Plan: fbconfig -r folly && fbmake runtests

Reviewed By:

FB internal diff: D1318921

8 years agosmall_vector exception safety, part 2
Nicholas Ormrod [Mon, 12 May 2014 18:02:00 +0000 (11:02 -0700)]
small_vector exception safety, part 2

small_vector is now object-exception safe for all container
functions, except for input-iterators.

That's a bold claim; probably incorrect. At the very least, it passes
the same test suite as std::vector and fbvector.

Aside: Clearly, no one uses erase(q1, q2) in the wild.

Facebook: Nothing special.

Test Plan:
fbconfig -r folly && fbmake runtests
fbconfig -r experimental/njormrod/stltest && fbmake runtests

Reviewed By:

FB internal diff: D1319787

8 years agosmall_vector exception safety, part 1
Nicholas Ormrod [Mon, 12 May 2014 17:57:21 +0000 (10:57 -0700)]
small_vector exception safety, part 1

small_vector is now object-exception safe for the general
container functions (N3337 table 96).

An amusing bug: in debug mode, swap would trigger an out-of-bounds
operator[] access. This has been fixed.

Nothing fancy in the non-OSS files.

Test Plan:
fbconfig -r folly && fbmake runtests
fbconfig -r experimental/njormrod/stltest && fbmake runtests

Reviewed By:

FB internal diff: D1319574

8 years ago(wangle) ManualExecutor::waitFor(F&&)
Hans Fugal [Fri, 9 May 2014 22:00:48 +0000 (15:00 -0700)]
(wangle) ManualExecutor::waitFor(F&&)


Test Plan: Using it in D1322350

Reviewed By:

FB internal diff: D1322475

8 years agomakeFuture(Try<T>&&)
James Sedgwick [Fri, 9 May 2014 22:37:42 +0000 (15:37 -0700)]

Add makeFuture variant which extracts the result contained in a Try and sticks it in a Future
One use case:

template <typename Result, typename Op, typename... Args>
Future<Result> wrapper(Op op, Args&&... args) {
// ... do some stuff before...
return op(std::forward<Args>(args)...).then([] (Try<Result>&& t) {
// ... do some stuff after...
return makeFuture<Result>(std::move(t));

With this makeFuture variant, "wrapper" doesn't need to be specialized for when
Result is void

Test Plan: employed in my code, will link to diff when ready

Reviewed By:

FB internal diff: D1318047

8 years agoDelete small_vector's OneBitMutex policy
Nicholas Ormrod [Fri, 9 May 2014 21:33:26 +0000 (14:33 -0700)]
Delete small_vector's OneBitMutex policy

It is unused.

Facebook: All hits from fbgs OneBitMutex have been deleted in this diff.

Test Plan: fbconfig -r folly && fbmake runtests

Reviewed By:

FB internal diff: D1319624

8 years agoHousekeeping
Nicholas Ormrod [Thu, 8 May 2014 23:23:13 +0000 (16:23 -0700)]

Remvoed old fbvector folly/test/stl_test files.
Have kept StlVectorTest, since it is still impressive and useful.

Facebook: n/a

Test Plan:
enable StlVectorTest in the TARGETS
fbconfig -r folly && fbmake runtests

Reviewed By:

FB internal diff: D1320254

8 years agomade folly::gen::member accept pointers to objects as well as references
Daniil Burdakov [Thu, 8 May 2014 11:57:20 +0000 (11:57 +0000)]
made folly::gen::member accept pointers to objects as well as references

Summary: subj

Test Plan: tests

Reviewed By:

FB internal diff: D1318719

8 years agofolly: fbstring: conditionally write terminator in c_str()
Lucian Grijincu [Thu, 8 May 2014 18:50:50 +0000 (11:50 -0700)]
folly: fbstring: conditionally write terminator in c_str()

c_str/data writes a terminator '\0' on each call. When
multiple threads call c_str/data on the same string (e.g. small global
constant) the string's cache line will bounce from cpu to cpu.

Cpus report that most instructions are stalled
(stalled-cycles-frontend/stalled-cycles-backend near 100%).

Fix: check before setting '\0'.

Test Plan:
** BEFORE **

- nothing defined  ** gcc-4.8.1-glibc-2.17-fb

$ perf stat --detailed _build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=100000000
4.10s experimental/lucian/bench/StringTerminatorBenchmark.cpp
Linking _build/opt/experimental/lucian/bench/string_terminator_benchmark...
2.40s _build/opt/experimental/lucian/bench/string_terminator_benchmark
I0507 17:27:39.373522 28745 StringTerminatorBenchmark.cpp:98]  --bm_min_iters=100000000
experimental/lucian/bench/StringTerminatorBenchmark.cpprelative  time/iter  iters/s
static_std_1t                                              934.10ps    1.07G
privat_std_1t                                    100.32%   931.08ps    1.07G
static_fbs_1t                                    100.30%   931.33ps    1.07G
privat_fbs_1t                                    100.42%   930.22ps    1.08G
static_sp__1t                                   86986.52%     1.07ps  931.24G
privat_sp__1t                                   81459.50%     1.15ps  872.07G
static_std_32t                                             392.41ns    2.55M
privat_std_32t                                  15072.69%     2.60ns  384.10M
static_fbs_32t                                   102.41%   383.17ns    2.61M
privat_fbs_32t                                  13643.61%     2.88ns  347.68M
static_sp__32t                                  61032.25%   642.96ps    1.56G
privat_sp__32t                                  74172.72%   529.06ps    1.89G

Performance counter stats for '_build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=100000000':

2417478.451740 task-clock                #   27.005 CPUs utilized
65,374 context-switches          #    0.027 K/sec
4,270 CPU-migrations            #    0.002 K/sec
2,594 page-faults               #    0.001 K/sec
5,261,195,623,299 cycles                    #    2.176 GHz                     [25.09%]
5,072,634,235,091 stalled-cycles-frontend   #   96.42% frontend cycles idle    [30.01%]
3,848,759,938,052 stalled-cycles-backend    #   73.15% backend  cycles idle    [30.01%]
614,983,033,652 instructions              #    0.12  insns per cycle
#    8.25  stalled cycles per insn [25.09%]
152,995,596,390 branches                  #   63.287 M/sec                   [30.01%]
24,501,018 branch-misses             #    0.02% of all branches         [25.94%]
66,298,370,215 L1-dcache-loads           #   27.425 M/sec                   [30.03%]
1,642,188,850 L1-dcache-load-misses     #    2.48% of all L1-dcache hits   [30.02%]
968,142,977 LLC-loads                 #    0.400 M/sec                   [30.02%]
480,786,261 LLC-load-misses           #   49.66% of all LL-cache hits    [30.02%]

89.520960614 seconds time elapsed

- #define FBSTRING_PERVERSE ** gcc-4.8.1-glibc-2.17-fb

$ perf stat --detailed _build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=100000000
4.25s experimental/lucian/bench/StringTerminatorBenchmark.cpp
Linking _build/opt/experimental/lucian/bench/string_terminator_benchmark...
2.45s _build/opt/experimental/lucian/bench/string_terminator_benchmark
I0507 17:24:26.487848 23720 StringTerminatorBenchmark.cpp:98]  --bm_min_iters=100000000
experimental/lucian/bench/StringTerminatorBenchmark.cpprelative  time/iter  iters/s
static_std_1t                                              932.63ps    1.07G
privat_std_1t                                    100.95%   923.85ps    1.08G
static_fbs_1t                                    100.04%   932.23ps    1.07G
privat_fbs_1t                                    101.14%   922.07ps    1.08G
static_sp__1t                                   30418.15%     3.07ps  326.16G
privat_sp__1t                                   56775.39%     1.64ps  608.77G
static_std_32t                                             388.57ns    2.57M
privat_std_32t                                  16465.70%     2.36ns  423.75M
static_fbs_32t                                    96.19%   403.98ns    2.48M
privat_fbs_32t                                  16248.69%     2.39ns  418.16M
static_sp__32t                                  81185.06%   478.63ps    2.09G
privat_sp__32t                                  81470.69%   476.95ps    2.10G

Performance counter stats for '_build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=100000000':

2486009.171842 task-clock                #   27.241 CPUs utilized
64,538 context-switches          #    0.026 K/sec
4,501 CPU-migrations            #    0.002 K/sec
2,863 page-faults               #    0.001 K/sec
5,413,507,764,726 cycles                    #    2.178 GHz                     [25.62%]
5,207,550,928,974 stalled-cycles-frontend   #   96.20% frontend cycles idle    [30.01%]
3,919,627,007,209 stalled-cycles-backend    #   72.40% backend  cycles idle    [30.01%]
685,365,706,158 instructions              #    0.13  insns per cycle
#    7.60  stalled cycles per insn [25.62%]
148,798,151,350 branches                  #   59.854 M/sec                   [30.01%]
24,386,492 branch-misses             #    0.02% of all branches         [26.08%]
72,683,962,699 L1-dcache-loads           #   29.237 M/sec                   [30.02%]
1,687,684,775 L1-dcache-load-misses     #    2.32% of all L1-dcache hits   [30.01%]
989,352,938 LLC-loads                 #    0.398 M/sec                   [30.02%]
484,825,665 LLC-load-misses           #   49.00% of all LL-cache hits    [30.02%]

91.259265010 seconds time elapsed

- #define FBSTRING_CONSERVATIVE ** gcc-4.8.1-glibc-2.17-fb

$ perf stat --detailed _build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=100000000
0.18s experimental/lucian/bench/StringTerminatorBenchmark.cpp
Linking _build/opt/experimental/lucian/bench/string_terminator_benchmark...
2.43s _build/opt/experimental/lucian/bench/string_terminator_benchmark
I0507 17:30:58.246834  1571 StringTerminatorBenchmark.cpp:98]  --bm_min_iters=100000000
experimental/lucian/bench/StringTerminatorBenchmark.cpprelative  time/iter  iters/s
static_std_1t                                                1.44ps  692.61G
privat_std_1t                                    193.01%   748.06fs    1.34T
static_fbs_1t                                    399.47%   361.43fs    2.77T
privat_fbs_1t                                    129.29%     1.12ps  895.50G
static_sp__1t                                     78.99%     1.83ps  547.10G
privat_sp__1t                                     56.22%     2.57ps  389.36G
static_std_32t                                             478.67ps    2.09G
privat_std_32t                                   100.30%   477.26ps    2.10G
static_fbs_32t                                   100.52%   476.21ps    2.10G
privat_fbs_32t                                   100.52%   476.22ps    2.10G
static_sp__32t                                    99.57%   480.76ps    2.08G
privat_sp__32t                                   100.06%   478.40ps    2.09G

Performance counter stats for '_build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=100000000':

198000.887222 task-clock                #   14.738 CPUs utilized
5,016 context-switches          #    0.025 K/sec
2,536 CPU-migrations            #    0.013 K/sec
4,717 page-faults               #    0.024 K/sec
421,850,242,358 cycles                    #    2.131 GHz                     [25.31%]
209,633,195,159 stalled-cycles-frontend   #   49.69% frontend cycles idle    [30.32%]
6,252,451,713 stalled-cycles-backend    #    1.48% backend  cycles idle    [30.21%]
874,096,942,441 instructions              #    2.07  insns per cycle
#    0.24  stalled cycles per insn [25.37%]
218,114,341,006 branches                  # 1101.583 M/sec                   [30.34%]
3,080,267 branch-misses             #    0.00% of all branches         [26.23%]
240,532,859 L1-dcache-loads           #    1.215 M/sec                   [30.73%]
23,151,832 L1-dcache-load-misses     #    9.63% of all L1-dcache hits   [30.61%]
5,283,803 LLC-loads                 #    0.027 M/sec                   [30.60%]
1,079,973 LLC-load-misses           #   20.44% of all LL-cache hits    [30.60%]

13.434898734 seconds time elapsed

- #define FBSTRING_PERVERSE ** gcc-4.8.1-glibc-2.17

$ perf stat --detailed _build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=100000000
I0507 17:37:01.095785 28744 StringTerminatorBenchmark.cpp:98]  --bm_min_iters=100000000
experimental/lucian/bench/StringTerminatorBenchmark.cpprelative  time/iter  iters/s
static_std_1t                                                2.80ps  357.44G
privat_std_1t                                    118.65%     2.36ps  424.09G
static_fbs_1t                                      0.30%   937.44ps    1.07G
privat_fbs_1t                                      0.30%   924.33ps    1.08G
static_sp__1t                                    229.14%     1.22ps  819.03G
privat_sp__1t                                    212.22%     1.32ps  758.55G
static_std_32t                                             662.00ps    1.51G
privat_std_32t                                   134.15%   493.47ps    2.03G
static_fbs_32t                                     1.27%    52.24ns   19.14M
privat_fbs_32t                                    28.02%     2.36ns  423.31M
static_sp__32t                                   138.51%   477.94ps    2.09G
privat_sp__32t                                   138.39%   478.38ps    2.09G

Performance counter stats for '_build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=100000000':

316918.699522 task-clock                #   17.561 CPUs utilized
8,761 context-switches          #    0.028 K/sec
2,190 CPU-migrations            #    0.007 K/sec
5,361 page-faults               #    0.017 K/sec
683,482,498,044 cycles                    #    2.157 GHz                     [25.13%]
488,046,572,692 stalled-cycles-frontend   #   71.41% frontend cycles idle    [30.18%]
269,765,079,972 stalled-cycles-backend    #   39.47% backend  cycles idle    [30.17%]
756,213,442,723 instructions              #    1.11  insns per cycle
#    0.65  stalled cycles per insn [25.17%]
175,643,190,326 branches                  #  554.222 M/sec                   [30.14%]
3,737,621 branch-misses             #    0.00% of all branches         [25.63%]
35,169,313,417 L1-dcache-loads           #  110.973 M/sec                   [30.31%]
215,454,174 L1-dcache-load-misses     #    0.61% of all L1-dcache hits   [30.25%]
148,883,234 LLC-loads                 #    0.470 M/sec                   [30.25%]
48,768,585 LLC-load-misses           #   32.76% of all LL-cache hits    [30.26%]

18.046902132 seconds time elapsed

** AFTER **

- define nothing ** gcc-4.8.1-glibc-2.17-fb

$ perf stat --detailed _build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=100000000
I0507 20:34:15.943284 16852 StringTerminatorBenchmark.cpp:98]  --bm_min_iters=100000000
experimental/lucian/bench/StringTerminatorBenchmark.cpprelative  time/iter  iters/s
static_std_1t                                              936.05ps    1.07G
privat_std_1t                                    100.67%   929.82ps    1.08G
static_fbs_1t                                     98.63%   949.07ps    1.05G
privat_fbs_1t                                    101.56%   921.69ps    1.08G
static_sp__1t                                   40342.12%     2.32ps  430.98G
privat_sp__1t                                   16370.35%     5.72ps  174.89G
static_std_32t                                             409.26ns    2.44M
privat_std_32t                                  16777.41%     2.44ns  409.95M
static_fbs_32t                                  17398.58%     2.35ns  425.13M
privat_fbs_32t                                  17475.47%     2.34ns  427.01M
static_sp__32t                                  85426.96%   479.07ps    2.09G
privat_sp__32t                                  85033.14%   481.29ps    2.08G

Performance counter stats for '_build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=100000000':

1387268.222731 task-clock                #   25.553 CPUs utilized
34,698 context-switches          #    0.025 K/sec
2,770 CPU-migrations            #    0.002 K/sec
3,039 page-faults               #    0.002 K/sec
3,019,637,816,074 cycles                    #    2.177 GHz                     [25.21%]
2,801,192,631,479 stalled-cycles-frontend   #   92.77% frontend cycles idle    [30.02%]
2,020,857,118,698 stalled-cycles-backend    #   66.92% backend  cycles idle    [30.00%]
801,090,224,478 instructions              #    0.27  insns per cycle
#    3.50  stalled cycles per insn [25.20%]
206,099,842,649 branches                  #  148.565 M/sec                   [30.03%]
12,954,936 branch-misses             #    0.01% of all branches         [25.64%]
104,129,435,254 L1-dcache-loads           #   75.061 M/sec                   [30.07%]
883,390,641 L1-dcache-load-misses     #    0.85% of all L1-dcache hits   [30.06%]
516,975,218 LLC-loads                 #    0.373 M/sec                   [30.04%]
255,887,523 LLC-load-misses           #   49.50% of all LL-cache hits    [30.04%]

54.289185806 seconds time elapsed

- #define FBSTRING_PERVERSE ** gcc-4.8.1-glibc-2.17-fb

$ perf stat --detailed _build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=100000000
4.20s experimental/lucian/bench/StringTerminatorBenchmark.cpp
Linking _build/opt/experimental/lucian/bench/string_terminator_benchmark...
2.47s _build/opt/experimental/lucian/bench/string_terminator_benchmark
I0507 20:19:07.272021 18610 StringTerminatorBenchmark.cpp:98]  --bm_min_iters=100000000
experimental/lucian/bench/StringTerminatorBenchmark.cpprelative  time/iter  iters/s
static_std_1t                                              936.15ps    1.07G
privat_std_1t                                    100.84%   928.34ps    1.08G
static_fbs_1t                                     97.92%   956.06ps    1.05G
privat_fbs_1t                                    101.41%   923.14ps    1.08G
static_sp__1t                                       inf%     0.00fs  infinity
privat_sp__1t                                       inf%     0.00fs  infinity
static_std_32t                                             413.49ns    2.42M
privat_std_32t                                  17375.60%     2.38ns  420.22M
static_fbs_32t                                  17443.07%     2.37ns  421.85M
privat_fbs_32t                                  15354.32%     2.69ns  371.33M
static_sp__32t                                  82034.91%   504.05ps    1.98G
privat_sp__32t                                  63367.22%   652.53ps    1.53G

Performance counter stats for '_build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=100000000':

1390538.848103 task-clock                #   25.373 CPUs utilized
37,417 context-switches          #    0.027 K/sec
3,147 CPU-migrations            #    0.002 K/sec
2,876 page-faults               #    0.002 K/sec
3,024,513,016,946 cycles                    #    2.175 GHz                     [25.08%]
2,813,021,108,191 stalled-cycles-frontend   #   93.01% frontend cycles idle    [30.02%]
2,043,124,392,473 stalled-cycles-backend    #   67.55% backend  cycles idle    [30.01%]
774,663,686,661 instructions              #    0.26  insns per cycle
#    3.63  stalled cycles per insn [25.09%]
197,666,485,664 branches                  #  142.151 M/sec                   [30.03%]
15,077,576 branch-misses             #    0.01% of all branches         [25.73%]
104,720,369,589 L1-dcache-loads           #   75.309 M/sec                   [30.05%]
886,090,434 L1-dcache-load-misses     #    0.85% of all L1-dcache hits   [30.04%]
520,015,584 LLC-loads                 #    0.374 M/sec                   [30.03%]
256,990,100 LLC-load-misses           #   49.42% of all LL-cache hits    [30.04%]

54.804099454 seconds time elapsed

- #define FBSTRING_CONSERVATIVE ** gcc-4.8.1-glibc-2.17-fb

$ perf stat --detailed _build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=100000000
4.02s experimental/lucian/bench/StringTerminatorBenchmark.cpp
Linking _build/opt/experimental/lucian/bench/string_terminator_benchmark...
2.45s _build/opt/experimental/lucian/bench/string_terminator_benchmark
I0507 20:31:31.120209 10543 StringTerminatorBenchmark.cpp:98]  --bm_min_iters=100000000
experimental/lucian/bench/StringTerminatorBenchmark.cpprelative  time/iter  iters/s
static_std_1t                                              929.96ps    1.08G
privat_std_1t                                    100.37%   926.53ps    1.08G
static_fbs_1t                                       inf%     0.00fs  infinity
privat_fbs_1t                                       inf%     0.00fs  infinity
static_sp__1t                                       inf%     0.00fs  infinity
privat_sp__1t                                       inf%     0.00fs  infinity
static_std_32t                                             381.41ns    2.62M
privat_std_32t                                  16080.74%     2.37ns  421.62M
static_fbs_32t                                  80498.05%   473.81ps    2.11G
privat_fbs_32t                                  80368.84%   474.57ps    2.11G
static_sp__32t                                  80287.07%   475.05ps    2.11G
privat_sp__32t                                  80410.51%   474.33ps    2.11G

Performance counter stats for '_build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=100000000':

1293727.223839 task-clock                #   25.403 CPUs utilized
33,881 context-switches          #    0.026 K/sec
3,718 CPU-migrations            #    0.003 K/sec
3,829 page-faults               #    0.003 K/sec
2,813,078,184,066 cycles                    #    2.174 GHz                     [25.11%]
2,597,561,032,630 stalled-cycles-frontend   #   92.34% frontend cycles idle    [30.04%]
1,883,652,860,583 stalled-cycles-backend    #   66.96% backend  cycles idle    [30.02%]
800,150,465,648 instructions              #    0.28  insns per cycle
#    3.25  stalled cycles per insn [25.10%]
197,970,559,157 branches                  #  153.023 M/sec                   [30.02%]
15,385,773 branch-misses             #    0.01% of all branches         [25.95%]
36,541,946,374 L1-dcache-loads           #   28.245 M/sec                   [30.08%]
813,994,917 L1-dcache-load-misses     #    2.23% of all L1-dcache hits   [30.07%]
467,241,616 LLC-loads                 #    0.361 M/sec                   [30.07%]
247,980,582 LLC-load-misses           #   53.07% of all LL-cache hits    [30.08%]

50.928965545 seconds time elapsed

Reviewed By:

FB internal diff: D1318048

8 years agoFix buid, missing getHugePageSizeForDevice
Dave Watson [Wed, 7 May 2014 19:58:31 +0000 (12:58 -0700)]
Fix buid, missing getHugePageSizeForDevice

Folly jenkins build has been failing for a couple days:

Due to a dep on an experimental file in D1307044.  Added file to, also needs boost_filesystem, added configure and link check for that too.

Test Plan: build works on ubuntu

Reviewed By:

FB internal diff: D1316855

8 years agoExpression SFINAE fixes in ApplyTuple
Elizabeth Smith [Thu, 8 May 2014 15:20:48 +0000 (08:20 -0700)]
Expression SFINAE fixes in ApplyTuple

MSVC does not support Expression SFINAE

this is a very nice c++11 feature that makes for some nice clean templating

But of course MSVC can't have nice things - it partially implements this when it feels like it, so some will work and some will need the nonsense


There will be more of these little template helper fixes  - they make the code a bit more complex but don't actually change anything when compiled

The accompanying fix in the test also does nothing but work around an MSVC compiler bug where it loses it's mind over the global namespace

Test Plan: fbconfig -r folly && fbmake runtests

Reviewed By:

FB internal diff: D1312700

8 years agotry_and_catch
Marc Celani [Sat, 3 May 2014 01:17:32 +0000 (18:17 -0700)]

Summary: A helper function to do try/catch on multiple exception types and store the ressult in an exception_wrapper. The purpose of this function is to best effort mimic std::current_exception(). Unlike std::current_exception(), we need to specify the types that we expect to see. Rather than writing macros or several lines per exception type to capture the exception into an exception_wrapper, this function makes writing try/catch blocks for this purpose very easy.

Test Plan: unit test

Reviewed By:

FB internal diff: D1308511


8 years agoSmall readability improvements
Alexey Spiridonov [Fri, 25 Apr 2014 22:38:38 +0000 (15:38 -0700)]
Small readability improvements

Summary: I got confused by the error message, and wasn't quite clear on the intent of communicateIOBuf()

Test Plan: waiting for auto-unittests

Reviewed By:

FB internal diff: D1297525

8 years agofix IOBuf self move-assignment
Philip Pronin [Tue, 6 May 2014 22:53:39 +0000 (15:53 -0700)]
fix IOBuf self move-assignment

Summary: Properly handle `this == &other` case.

Test Plan: fbconfig -r folly/test && fbmake runtests_opt -j32

Reviewed By:

FB internal diff: D1314916

8 years agoFix rebase fail
Marc Celani [Wed, 7 May 2014 03:43:41 +0000 (20:43 -0700)]
Fix rebase fail

Summary: whoops

Test Plan: reran unit tests

Reviewed By:

Blame Rev:

8 years agoAllow for folly::exception_wrapper in ClientReceiveState
Marc Celani [Wed, 7 May 2014 02:05:02 +0000 (19:05 -0700)]
Allow for folly::exception_wrapper in ClientReceiveState

This diff allows us to use folly::exception_wrapper in ClientReceiveState. Existing use cases are still supported (crs.exception() still returns an exception_ptr no matter what), but we can now choose to set an exception without throwing first.

On the folly side, add some new functions for making an exception_ptr from an exception_wrapper.

Test Plan: Reran unit tests

Reviewed By:

FB internal diff: D1307027


8 years agoBenchmark specific fixes
Elizabeth Smith [Wed, 7 May 2014 00:41:09 +0000 (17:41 -0700)]
Benchmark specific fixes

benchmark is necessary to get the tests running
primarily replacing a few inline asm items with MSVC intrinsics and do not optimize tricks
also a fix for use of the gcc specific ## with __VAR_ARGS__
although that is primarily intended as a workaround for trailing commas, gcc apparently cuts off all items in the macro afterwards
this was being used for a clever/dirty trick to do one or none for a macro
replaced instead with a version that will work for all other compilers (might need to be expanded for more args, but for now it's working with all current usage)


also fixed a use of max without specifying a template type that was making msvc barf - specifying it (as in the min case) made it compile cleanly

Test Plan: fbconfig -r folly && fbmake runtests

Reviewed By:

FB internal diff: D1313609

8 years agorace in Future destructor
Hans Fugal [Wed, 30 Apr 2014 16:40:08 +0000 (09:40 -0700)]
race in Future destructor

@mnd wrote some Wangle code that would trip up with a `bad_function_call` exception. This Shouldn't Happen™ but the exception comes from trying to call a `std::function` which is null. We pretty thoroughly examined his usage and didn't find any problems, and this patch seems to make the error go away. See #4207781 for more details.

And reasoning about it, it makes sense. Inline comments explain the race.

Test Plan: Alas, I haven't been able to get a minimal repro and therefore a regression unit test. It's a hard race to trigger. I still don't understand why Matt's code does it.

Reviewed By:

FB internal diff: D1304001

8 years agoprintf format checking for msvc
Elizabeth Smith [Tue, 6 May 2014 18:44:31 +0000 (11:44 -0700)]
printf format checking for msvc

sal annotations can be used to do similar (not exact) checking to the functionality provided by the format attribute in gcc
the annotations are done by prefixing the format string with a value which makes the macro definitions a bit messy


Test Plan: fbconfig -r folly && fbmake runtests

Reviewed By:

FB internal diff: D1313653

8 years agoUse winpthreads clock_gettime implementations
Elizabeth Smith [Tue, 6 May 2014 18:39:21 +0000 (11:39 -0700)]
Use winpthreads clock_gettime implementations

No need to reinvent the wheel - winpthreads has the appropriate posix layer in place for time functionality
and we're already using it for pthreads on windows - so just include the right headers to make sure defines are set up when using time functionality


Test Plan: fbconfig -r folly && fbmake runtests

Reviewed By:

FB internal diff: D1313600

8 years agoAdding some msvc specific defines
Elizabeth Smith [Tue, 6 May 2014 18:21:11 +0000 (11:21 -0700)]
Adding some msvc specific defines

msvc puts ssize_t in a stupidly odd place and names it weirdly too
this also takes care of snprintf missing (the semantics are slightly off in the msvc version regarding the return value, but usage in folly is limited and does NOT do the double snprintf call madness so this is safe)
funcsig and pretty function give you roughtly the same thing in compiler specific implementations
strerror_s is msvc's thread save strerror


Test Plan: fbmake runtests

Reviewed By:

FB internal diff: D1291542

8 years agofolly::join takes advantage of StringPiece::size()
Marc Celani [Tue, 6 May 2014 00:58:40 +0000 (17:58 -0700)]
folly::join takes advantage of StringPiece::size()

Summary: folly::join should take advantage of StringPiece::size() when joining StringPieces. This avoids unnecessary resizes when appending values to the output string.

Test Plan: Reran folly unit tests for strings

Reviewed By:

FB internal diff: D1313009


8 years agoHandle event_base_new failure when out of file descriptors.
Zejun Wu [Tue, 6 May 2014 00:08:36 +0000 (17:08 -0700)]
Handle event_base_new failure when out of file descriptors.

Both event_base_new and event_init return nullptr when out of file descriptors.
Using null event_base will result in segfault.

Test Plan: (ulimit -n 50000 && _build/opt/sigma/service/sigma_server --instance_name=si_sigma_push --min_scribe_log_level=0 --allow_status_port_fallback=true --minloglevel=1 --v=0 --feature_objects_limit=1000000 --hbase_default_timeout_ms=250 --max_total_connections_per_region_server=10 --max_retained_connections_per_region_server=10 --tao_default_timeout_ms=5000 --enable_writes_scribe_si_floop=false --enable_writes_all=false --arena_size_limit=268435456 --run_fxl=true)

Reviewed By:

FB internal diff: D1311855

8 years agoignore non-existent mount points in readHugePageSizes
Philip Pronin [Mon, 5 May 2014 17:10:41 +0000 (10:10 -0700)]
ignore non-existent mount points in readHugePageSizes

Some of mount points may not exist (for example, if
application is running from chroot and unshare hasn't been used).


Test Plan: unicorn canary

Reviewed By:

FB internal diff: D1311374

8 years agoAdd defaulted() in Format.h to avoid throwing on missing keys; add string-y dynamic...
Tudor Bosman [Wed, 30 Apr 2014 15:41:45 +0000 (08:41 -0700)]
Add defaulted() in Format.h to avoid throwing on missing keys; add string-y dynamic constructors

Test Plan: folly/test

Reviewed By:

FB internal diff: D1303911

8 years agoExceptionWrapper comments
Marc Celani [Fri, 2 May 2014 03:05:29 +0000 (20:05 -0700)]
ExceptionWrapper comments

Summary: Comments to discuss motivation

Test Plan: its comments

Reviewed By:

FB internal diff: D1308259


8 years agoMove setPosixThreadName & friends to folly.
Louis Brandy [Sat, 26 Apr 2014 00:05:02 +0000 (17:05 -0700)]
Move setPosixThreadName & friends to folly.

Summary: This was previously in thrift (and copied and pasted in several other places, including folly itself). Other potential open-source projects want this basic functionality so lets centralize it in folly instead of having potentially awkward dependencies on thrift (or copy/paste everywhere).

Test Plan: Build all the things. Run the tests.

Reviewed By:

FB internal diff: D1297972

8 years agoFix for Android importing folly/Random.cpp
Bryan Alger [Thu, 1 May 2014 05:41:23 +0000 (22:41 -0700)]
Fix for Android importing folly/Random.cpp

Summary: ext/random causes issues building on android

Test Plan: compiled on fbandroid

Reviewed By:

FB internal diff: D1308224

8 years agoexception_wrapper: now without undefined behavior
Tudor Bosman [Fri, 2 May 2014 02:57:06 +0000 (19:57 -0700)]
exception_wrapper: now without undefined behavior

Converting from std::exception* to void* to T* (where T is not std::exception
but a derived type) is undefined behavior (and will break with multiple or
virtual inheritance). Luckily, there's no need for void* there at all.

Also, don't force make_exception_wrapper to capture by value.

Test Plan: exception_wrapper_test

Reviewed By:

FB internal diff: D1308251


8 years agoMemoryMapping changes: automatically detect huge pages, no more WritableMemoryMapping
Tudor Bosman [Thu, 1 May 2014 20:04:04 +0000 (13:04 -0700)]
MemoryMapping changes: automatically detect huge pages, no more WritableMemoryMapping

Summary: "writable" is now an option to the MemoryMapping constructor.

Test Plan: folly/test, thrift/lib/util/test:FrozenUtilTest, others

Reviewed By:

FB internal diff: D1307044


8 years agoAdd missing files to
Dave Watson [Thu, 1 May 2014 01:26:56 +0000 (18:26 -0700)]
Add missing files to

As title, add newly added files.  fbthrift's jenkins build is failing due to missing futex and lifosem.  Added other ones that looked new

Test Plan: Built folly on ubuntu.  Builds.

Reviewed By:

FB internal diff: D1306401

8 years agoexception wrapper
Marc Celani [Thu, 1 May 2014 17:44:13 +0000 (10:44 -0700)]
exception wrapper

folly::exception_wrapper is a different take on std::exception_ptr to
suit a specific use case.

The good: std::exception_ptr is not templated, so it can easily be used in
different classes without template creep. You can pass errors around between
threads or simply between modules. Rethrowing the exception throws the *proper*
derived class exception, not some base class.

The bad: Getting access to the exception requires throwing, which is expensive.
Many users of popular frameworks that take advantage of std::exception_ptr
check if the exception is set, and if so do some error handling without actually
knowing the type of the exception or logging its message, just to avoid the cost
of rethrowing the exception.

The ugly: Creating an exception_ptr requires throwing the exception as least
once. This is bad in the performance sensitive case where users will not even
inspect the exception.

This class takes advantage of the good while avoiding the requirement to throw.
By using a templated deleter and thrower function, we can create an
exception_wrapper which is properly managed, can be thrown, and can be retrieved
as a void* with a get() function. Users that previously caught exceptions are
now able to dynamically cast to different exception types they formerly caught
to avoid the unwind cost while still getting details about the error.

Test Plan: unit test

Reviewed By:

FB internal diff: D1305470


8 years agoFix Chatty subprocess test, call callback on hangup
Tudor Bosman [Wed, 30 Apr 2014 00:40:42 +0000 (17:40 -0700)]
Fix Chatty subprocess test, call callback on hangup

The Chatty subprocess test incorrectly assumed that we saw EOF on the last
read from the child (that is, read() returned 0). That's not the case for two
reasons: 1. the child is allowed to stall right before it closes its stdout, in
which case we get EAGAIN, and 2. Subprocess::communicate would close the fd
without calling the read callback anyway. Fix both such things.

Test Plan: ran test

Reviewed By:

FB internal diff: D1303215

8 years agoQueueBenchmark set max read at once
Dave Watson [Fri, 11 Apr 2014 18:56:35 +0000 (11:56 -0700)]
QueueBenchmark set max read at once

Summary: Makes a fair comparison between asox queue and notification queue.  THe updated benchmark is just noise in this diff, notificationqueue isn't fast enough (yet) to make a difference.

Test Plan:
fbconfig common/concurrency; fbmake opt

Reviewed By:

FB internal diff: D1272859

8 years agoUse readNoInt/writeNoInt in folly::Subprocess
Andrey Goder [Mon, 28 Apr 2014 20:43:17 +0000 (13:43 -0700)]
Use readNoInt/writeNoInt in folly::Subprocess

Summary: We have these helper methods, but are not using them. What sadness.

Test Plan:
fbconfig -r folly
fbmake runtests

Reviewed By:

FB internal diff: D1299720

8 years agoInclude <random> in folly/Random.h
Peter Griess [Mon, 28 Apr 2014 19:03:41 +0000 (12:03 -0700)]
Include <random> in folly/Random.h

- Its methods are being used; include the header


Test Plan: - Build for iOS

Reviewed By:

FB internal diff: D1299440

8 years agoDefault to using folly::LifoSem
Marc Celani [Mon, 28 Apr 2014 17:09:16 +0000 (10:09 -0700)]
Default to using folly::LifoSem

Unless a service is overloaded, it should be able to clear
its queue frequently. When this happens, threads fall asleep until
more work is available in the queue. Waking up threads in LIFO
order gives us a lot of benefits. First, threads that were most
recently active are more likely to be mapped to the same cpu core
and thereby experience better L1 cache hit rate. Second, we can
madvise away jemalloc arenas on very idle threads. If we wake up
threads in FIFO order, we will never get a thread to remain idle
long enough for this to be worthwhile.

folly::LifoSem does just that. Benchmark in which the queue is
allowed to drain show that we get a substantial increase in
throughput by waking up threads in LIFO order.

Test Plan:
QueueBenchmark results summary:
As expected, benchmarks run faster in the case where the queue is
able to frequently drain itself, particularly in cases where the
number of threads is large. Benchmarks run slower when the
consumers cannot keep up with the producers, particularly when we
reach the queue capacity and we need to synchronize between
producers and consumers. However, in this case I think we care
less about the overhead of the queue itself and more about how
quickly we can shed the actual underlying load.

Reviewed By:

FB internal diff: D1298343

8 years agofixing split_step documentation
Marcelo Juchem [Mon, 7 Apr 2014 18:58:03 +0000 (11:58 -0700)]
fixing split_step documentation

Summary: example results were backwards

Test Plan: unit tests

Reviewed By:

FB internal diff: D1262425

8 years agoSymbolizedFrame::name is already null-terminated
Tudor Bosman [Sun, 20 Apr 2014 15:02:08 +0000 (08:02 -0700)]
SymbolizedFrame::name is already null-terminated

... as it comes from Elf::getSymbolName, which returns null-terminated C
strings. So there's no need to copy it into a fixed-size buffer (and have a
buffer overflow, ouch).

Test Plan: folly/experimental/symbolizer_test, see what else "arc unit" gets us

Reviewed By:

FB internal diff: D1286348

8 years agostd::move-able MemoryMapping
Tom Jackson [Wed, 23 Apr 2014 00:22:09 +0000 (17:22 -0700)]
std::move-able MemoryMapping

Test Plan: unit tests

Reviewed By:

FB internal diff: D1290632

8 years agoSupport addValueAggregated() for histogram
Tian Fang [Mon, 21 Apr 2014 02:23:13 +0000 (19:23 -0700)]
Support addValueAggregated() for histogram

Support addValueAggregated() for histogram to add multiple samples for
one bucket at one time.

Test Plan: Have a client to call the new API and test in production.

Reviewed By:

FB internal diff: D1286528

8 years agoBetter <bits/c++config.h> detection
Elizabeth Smith [Wed, 23 Apr 2014 20:09:53 +0000 (13:09 -0700)]
Better <bits/c++config.h> detection

When worrying about only 2 cpp libraries, detection of "is this libstdc++ or libc++" works, but when throwing in a third option (msvc) this becomes a very brittle way of detecting features.  Also depending on how clang is used it may or may not have this header about

Changed to instead use configuration value for "do we have this header file" for including bits/c++config.h and changed the logic to be identical in all three use cases

Test Plan: fbmake runtests

Reviewed By:

FB internal diff: D1291547

8 years agomsvc packed attribute translation
Elizabeth Smith [Wed, 23 Apr 2014 20:02:51 +0000 (13:02 -0700)]
msvc packed attribute translation

Provide translations for gcc packed

MSVC has a very very ugly syntax for packing using pragmas which is very different from the gcc attribute
this makes it very hard to macro around, but lots of ifdefs are also painful - cleanest of the hackery is to simply have push, pop, and attribute macros for packed that define properly depending on the compiler

Test Plan: fbmake runtests

Reviewed By:

FB internal diff: D1279966

8 years agoFuture API for CacheClientCommon
Andrii Grynenko [Wed, 23 Apr 2014 02:58:46 +0000 (19:58 -0700)]
Future API for CacheClientCommon

Allow mutable functors in Promise::fulfil.

Works both with EventBase and without it (thanks to wangle::Future thread-safety). CacheClientImpl lifetime is guaranteed using shared_ptr.

Test Plan: unit test

Reviewed By:

FB internal diff: D1291024

8 years agofix off-by-one error in IPAddress::longestCommonPrefix()
Adam Simpkins [Fri, 18 Apr 2014 18:53:32 +0000 (11:53 -0700)]
fix off-by-one error in IPAddress::longestCommonPrefix()

Fix an off-by-one error causing an ASAN abort.  When calling
longestCommonPrefix() with a full mask, it would compare one byte past
the end of the address data, and would end up writing past the end of
the ba array on the stack.

Test Plan:
Built with ASAN, ran the unit tests, and verified the ASAN failure was

Reviewed By:

FB internal diff: D1284750

8 years agoMove common/network/IPAddress.h and related to folly/
Anton Likhtarov [Sat, 5 Apr 2014 02:00:31 +0000 (19:00 -0700)]
Move common/network/IPAddress.h and related to folly/

Moving our internal IP/Mac address libraries to folly/

We want to get rid of common/ dependencies in Mcrouter since we're going to open source it. Also looking at the original commit, seems like it's been the intention all along, so I just did it.

I tried to keep dependencies intact as much as possible. Changing projects to use this directly should be in separate diffs.

Test Plan:
Run folly/network and common/network unit tests.

Generate the list of targets with:

fbgs /$FILE.h | cut -f1 -d: | xargs -L1 dirname | cut -f2- -d/ | sort | uniq

Then fbconfig + fbmake. Will fix contbuild failures.

Revert Plan:

Reviewed By:

FB internal diff: D1261089

8 years agoFix type mismatch in IOBuf::getIov()
Peter Griess [Fri, 18 Apr 2014 20:06:07 +0000 (13:06 -0700)]
Fix type mismatch in IOBuf::getIov()

Summary: - On iOS, iovec::iov_len is a size_t, which is a uint32_t; convert

Test Plan:
- fbconfig -r folly && fbmake runtests
- Builds in fbobjc

Reviewed By:

FB internal diff: D1284931

8 years agoFix header ordering lint failures
Nicholas Ormrod [Fri, 18 Apr 2014 17:16:33 +0000 (10:16 -0700)]
Fix header ordering lint failures

Headers whose includes are not sufficient cause problems when included
from other files. An easy way to prevent this is to include the header
first in its associated cpp file. In fact, we have a lint rule for this.
However, there are places where this rule is ignored. This diff fixes
many of them.

Note that a few top-level directories were excluded.

Reviewed By:

FB internal diff: D1281032

8 years agox64 detection missing an include
Elizabeth Smith [Fri, 18 Apr 2014 16:28:00 +0000 (09:28 -0700)]
x64 detection missing an include

Summary: rwspinlock.h was missing the portability.h header which made the x64 detection go wonky, but only shows up when used with isolated rwspinlock use

Test Plan: fbmake runtests

Reviewed By:

FB internal diff: D1284387

Blame Revision: D1282140

8 years agomacro for cross platform x64 detection
Elizabeth Smith [Fri, 18 Apr 2014 15:18:54 +0000 (08:18 -0700)]
macro for cross platform x64 detection

Summary: Another cross platform fun macro - this one detects x64 status and requires some portability.h includes (which should only define portability stuff and not actually include anything)

Test Plan: fbmake runtests

Reviewed By:

FB internal diff: D1282140

8 years agoFix flag naming
Bryan Alger [Thu, 17 Apr 2014 22:06:36 +0000 (15:06 -0700)]
Fix flag naming

Summary: doh

Test Plan: compiled under android

Reviewed By:

FB internal diff: D1283177

8 years agofbstring likely/unlikley ifdef for stuff that doesn't support it
Elizabeth Smith [Thu, 17 Apr 2014 20:36:10 +0000 (13:36 -0700)]
fbstring likely/unlikley ifdef for stuff that doesn't support it

Summary: the fbstring specific unlikely/likely defs were missing the ifdef blocks to avoid breaking things without builtin_expect

Test Plan: fbmake runtests

Reviewed By:

FB internal diff: D1282166

8 years ago(Folly/Gen) Make ranges and sequences take a stepping functor
Hannes Roth [Thu, 17 Apr 2014 16:26:56 +0000 (09:26 -0700)]
(Folly/Gen) Make ranges and sequences take a stepping functor

The functor allows to create ranges and sequences that advance the
iterator by more than 1 per iteration.

Test Plan: Unit test.

Reviewed By:

FB internal diff: D1228065

8 years ago(Folly/Gen) Fix compilation with clang
Hannes Roth [Thu, 17 Apr 2014 15:49:51 +0000 (08:49 -0700)]
(Folly/Gen) Fix compilation with clang

Clang chokes because it tries to instantiate both versions of `from<const int*>`, one of which calls `std::begin(const int*)`, which doesn't work. By casting the intializer list to the right type, it can pick the overload.

Clang, because it makes debugging these templates so much better.

Test Plan: `fbconfig --clang folly/gen/test && fbmake runtests_dbg`

Reviewed By:

FB internal diff: D1280888

8 years agoabstract thread_local support
Elizabeth Smith [Thu, 17 Apr 2014 14:49:10 +0000 (07:49 -0700)]
abstract thread_local support

change from using __thread to using FOLLY_THREAD_LOCAL macro, this will allow abstraction over gcc and msvc implementations of thread local (__thread and __declspec(thread)) which have the same semantices and will also allow drop in replacement of thread_local when compiler support for the feature is complete  This doesn't do anything about apple, however, which still has broken __thread support

This doesn't actually change any implementation for now, simply allows for correct compilation

Test Plan: fbmake runtests

Reviewed By:

FB internal diff: D1278726

8 years agoAdd missing includes for msvc
Elizabeth Smith [Thu, 17 Apr 2014 14:45:39 +0000 (07:45 -0700)]
Add missing includes for msvc

Summary: MSVC is a stricter compiler and needs additional includes explicitly declared

Test Plan: fbmake runtests

Reviewed By:

FB internal diff: D1277004

8 years agoparallel(pipeline)
Tom Jackson [Sat, 5 Apr 2014 00:58:07 +0000 (17:58 -0700)]

Adding `... | parallel(my | pipe | line) | ...` for parallelizing a portion of a generator pipeline.

auto factored = from(values)
| parallel(filter(isEven) | map(square) | sub(count))
| sum;

Work is divided evenly among a fixed number of threads using a `MPMCQueue`.

Test Plan: Unit tests and benchmarks testing for a variety of workloads and performance characteristics, including sub-linear (blocking) workloads, linear (mostly math) workloads, and superlinear (sleeping) workloads to simulate real-world use.

Reviewed By:

FB internal diff: D638551

8 years agoIntroduce LifoSem
Marc Celani [Thu, 17 Apr 2014 00:19:31 +0000 (17:19 -0700)]
Introduce LifoSem

Summary: A LIFO semaphore is useful for building a thread pool because we can keep cpu caches warmer by continually reusing the last used thread. This diff introduces a LIFO semaphore to folly.

Test Plan: unit tests

Reviewed By:

FB internal diff: D1280405

8 years agoFix folly Uri::host() return value for IPv6 address
Jun LI [Wed, 16 Apr 2014 20:58:54 +0000 (13:58 -0700)]
Fix folly Uri::host() return value for IPv6 address

Summary: folly Uri::host() returns leading and trailing brackets for IPv6 host, for example, for URI "http:://[::1]:8080/index.html", host() method returns "[::1]". But square brackets are not part of host in fact, this diff is going to remove brackets for IPv6 host.

Test Plan: fbconfig -r folly/test && fbmake runtests_dbg

Reviewed By:

FB internal diff: D1276513

8 years agoMSVC translation of noreturn attribute
Elizabeth Smith [Wed, 16 Apr 2014 20:49:35 +0000 (13:49 -0700)]
MSVC translation of noreturn attribute

Provide translations for gcc noreturn attribute

__attribute__((noreturn)) is gcc specific, clang understands it on nix systems, but for msvc __declspec(noreturn) is the compiler specific version, which clang will imitate/use on windows.  There was already a FOLLY_NORETURN in portability.h, however because of __declspec limitations it must prefix the declaration, not postfix.  The gcc noreturn attribute does not have this limitation and will work prefixed OR postfixed, so to keep from turning code into spaghetti nonsense, the macro was moved to the front of declations where it is currently used.  This will allow it to work with the proper definitions on clang, clang on windows, gcc, and msvc

Test Plan: fbmake runtests

Reviewed By:

FB internal diff: D1279466

8 years agoMSVC always_inline and noinline __attributes__ translation
Elizabeth Smith [Wed, 16 Apr 2014 20:45:37 +0000 (13:45 -0700)]
MSVC always_inline and noinline __attributes__ translation

Provide translations for gcc always_inline and noinline attribute

Change to using a macro from portability.h for attributes for using always_inline and noinline for cross platform support

Test Plan: fbmake runtests

Reviewed By:

FB internal diff: D1279652

8 years agoUse StringPiece instead of String
Victor Loh [Wed, 16 Apr 2014 04:57:23 +0000 (21:57 -0700)]
Use StringPiece instead of String

Using StringPiece makes it easier than std::string

Facebook: Motivation for this diff is found in D1253595

Test Plan:
Added new unittest

fbconfig folly/io/test && fbmake runtests_opt

Reviewed By:

FB internal diff: D1276185

8 years agouint isn't portable and makes some compilers angry, change to unsigned int
Elizabeth Smith [Wed, 16 Apr 2014 01:52:19 +0000 (18:52 -0700)]
uint isn't portable and makes some compilers angry, change to unsigned int

Summary: As part of the windows port, uint is not defined as a type and is not standard, it makes msvc choke, a simple change to unsigned int fixes the issue

Test Plan: fbmake runtests

Reviewed By:

FB internal diff: D1278487

8 years agoAdd non-const operator[] to TypedIOBuf
Tudor Bosman [Tue, 15 Apr 2014 16:24:11 +0000 (09:24 -0700)]
Add non-const operator[] to TypedIOBuf

Test Plan: used it

Reviewed By:

FB internal diff: D1277063


8 years agofolly: improve ProducerConsumerQueueBenchmark (misplaced doNotOptimizeAway)
Lucian Grijincu [Tue, 15 Apr 2014 03:15:10 +0000 (20:15 -0700)]
folly: improve ProducerConsumerQueueBenchmark (misplaced doNotOptimizeAway)

as per discussion from

After changing the queue length to be a small value (100) I didn't see
improvement from cache alignment of atomics (too much noise). Dropping
that and just fixing the benchmark.

Test Plan: n/a

Reviewed By:

FB internal diff: D1276591

8 years agoFix apparent race in SubprocessTest
Nicholas Ormrod [Mon, 14 Apr 2014 21:21:11 +0000 (14:21 -0700)]
Fix apparent race in SubprocessTest

The code originally contained the line

EXPECT_EQ((rcount == lineCount), r); // EOF iff at lineCount

Which asserted that, on the last iteration, EOF was being read (and thus
r was true).

This line has been causing errors in contbuild. There seem to be two
issues at play.

First, the left hand side of the EXPECT_EQ is always false. rcount is
incremented three lines down, and the (implicit) loop ends when rcount
equals lineCount. This means that, on the last iteration, rcount is
actually one less than lineCount. Hence the check is effectively
asserting that r is false.

The second issue is that r is rarely true. Empirically, a necessary but
not sufficient condition is that there be lots of other processes
running at the same time (such as when running fbconfig -r folly &&
fbmake runtests_opt). This would explain why the test passes during
development but fails in contbuild.

I am guessing that, when there are few processes running, Subprocess is
predictably ceasing to read before whichever other process yields EOF,
and that this does not always happen when there is resource contention.

As an immediate fix, I am asserting that r may only be true on the last
iteration. Could someone (cc @tudorb) who is more familiar with the
intricacies of Subprocess please check to see if this is an idiosyncrasy
of the test, or if there is actually an underlying bug.

Test Plan:
Before the change, run fbconfig -r folly && fbmake runtests_opt
repeatedly. Notice that most of the iterations yield an error from line
376 (which is consistent with contbuild error logs).

After the change, notice that the error disappears.

Reviewed By:

FB internal diff: D1269124

8 years agofolly: gen: pmap: parallel version of map
Lucian Grijincu [Sun, 13 Apr 2014 06:10:11 +0000 (23:10 -0700)]
folly: gen: pmap: parallel version of map

same as map, but runs it's argument in parallel over a number of threads.


Test Plan: added test

Reviewed By:

FB internal diff: D1258364

8 years agoChange child's working directory
Alexey Spiridonov [Thu, 10 Apr 2014 07:26:23 +0000 (00:26 -0700)]
Change child's working directory

Summary: Changing the parent's WD is prone to race conditions of various sorts, and needlessly alters parent state. Python's subprocess also has this feature.

Test Plan: fbmake dbg _bin/folly/test/subprocess_test ; ../_bin/folly/test/subprocess_test

Reviewed By:

FB internal diff: D1269526

8 years agofolly: allow to build with (flex arrays vs. lambda)
Jim Meyering [Fri, 4 Apr 2014 20:14:20 +0000 (13:14 -0700)]
folly: allow to build with (flex arrays vs. lambda)

* folly/test/CacheLocalityTest.cpp (contentionAtWidth): Work around
clang's prohibition against using flexible arrays in a lambda.

Test Plan:
fbconfig --clang --with-project-version clang:dev folly/test \
&& fbmake runtests

Reviewed By:

FB internal diff: D1263620

8 years agoconfigurable alignment for Arena
Philip Pronin [Tue, 8 Apr 2014 18:38:55 +0000 (11:38 -0700)]
configurable alignment for Arena

Summary: Allow specifying custom alignment.

Test Plan: fbconfig -r folly/test && fbmake opt -j32


Reviewed By:

FB internal diff: D1264851

8 years agoMake header compatible with warp
ptarjan [Wed, 9 Apr 2014 04:13:38 +0000 (21:13 -0700)]
Make header compatible with warp

8 years agofix ConcurrentSkipList::Recycler layout
Philip Pronin [Tue, 8 Apr 2014 18:08:02 +0000 (11:08 -0700)]
fix ConcurrentSkipList::Recycler layout

D1261546 introduced regression in `sizeof(ConcurrentSkipList)`.
In case of `NodeAlloc = folly::SysAlloc`, size of an empty `NodeAlloc` struct
is 1 byte, due to the alignment requirements of the next field, it got
translated into wasting 8 bytes.

Test Plan: fbconfig -r folly/test && fbmake opt -j32


Reviewed By:

FB internal diff: D1264730

8 years agofolly: accommodate fact that clang's strlen is not constexpr
Jim Meyering [Tue, 8 Apr 2014 04:11:54 +0000 (21:11 -0700)]
folly: accommodate fact that clang's strlen is not constexpr

* folly/folly-config.h (FOLLY_HAVE_CONSTEXPR_STRLEN): Define to
zero when compiling with clang, since clang's strlen is not constexpr.

Test Plan:
Run this:
fbconfig --clang --with-project-version clang:dev folly/test && fbconfig dbg
and observe that the following no longer strikes:
folly/test/RangeTest.cpp:304:25: error: constexpr variable 'hello1' must be initialized by a constant expression

I also tried using clang-3.4 (the current default),
fbconfig --clang folly/test && fbmake dbg
with these results:

Reviewed By:

FB internal diff: D1262840

8 years agoFix make check
Dave Watson [Fri, 4 Apr 2014 19:17:54 +0000 (12:17 -0700)]
Fix make check

Summary: Fix make check.  Benchmark.cpp was included twice, eventfd doesn't exist anymore.

Test Plan: autoreconf -i; ./configure; make check on ubuntu

Reviewed By:

FB internal diff: D1259706

8 years agofix automake
Dave Watson [Thu, 3 Apr 2014 20:40:58 +0000 (13:40 -0700)]
fix automake

Automake complains there are two Malloc.cpp rules - rename one of them.

Facebook: Actually this file doesn't even build in fbconfig - is it still necessary?

Test Plan: autoreconf -i on ubuntu doens't fail

Reviewed By:

FB internal diff: D1257372

8 years ago(wangle) codegen then() variant tests
Hans Fugal [Mon, 7 Apr 2014 16:57:42 +0000 (09:57 -0700)]
(wangle) codegen then() variant tests

When all the variants I am dreaming of work, there's a little more than
100 tests generated.

Test Plan: fbmake runtests

Reviewed By:

FB internal diff: D1258754

8 years ago(Folly/FBString) Fix compilation time of test with opt
Hannes Roth [Mon, 7 Apr 2014 16:44:38 +0000 (09:44 -0700)]
(Folly/FBString) Fix compilation time of test with opt

Summary: Now compiles in 18 seconds. Took two minutes before.

Test Plan: `fbmake runtests_opt`

Reviewed By:

FB internal diff: D1243689

8 years agoconfigurable node allocator for ConcurrentSkipList
Philip Pronin [Sat, 5 Apr 2014 21:54:37 +0000 (14:54 -0700)]
configurable node allocator for ConcurrentSkipList

Allow specifying `SimpleAllocator` to use for node allocation.
Added optimizations for arena allocators.

Test Plan: fbconfig -r folly/test && fbmake opt -j32

Reviewed By:

FB internal diff: D1261546

8 years agomake ConcurrentSkipList ctor public
Philip Pronin [Sat, 5 Apr 2014 04:50:27 +0000 (21:50 -0700)]
make ConcurrentSkipList ctor public

There is no reason to force heap allocation for

Test Plan: fbconfig -r unicorn/test && fbmake opt -j32


Reviewed By:

FB internal diff: D1261183

8 years agoAdding support for signed integers
Tom Jackson [Fri, 4 Apr 2014 23:55:09 +0000 (16:55 -0700)]
Adding support for signed integers

Summary: For use with Thrift::Frozen, especially.

Test Plan: Unit tests

Reviewed By:

FB internal diff: D1255257

8 years agoBugfix in iterator emplace
Nicholas Ormrod [Fri, 4 Apr 2014 03:02:55 +0000 (20:02 -0700)]
Bugfix in iterator emplace

Subtle bugfix to fbvector

Reproduction requirements:
- emplace into a vector at a non-end position
- the vector has enough capacity for the extra elements
- the value type's internal state is poisoned after a move

What should happen, explained by picture:


want to insert two elements here

FIRST STEP: uninitialized move-construction
x and y are initialized, but could be in an invalid state


SECOND STEP: move-assignment, in reverse order


What actually happens:
An indexing error causes the entire tail (##1234xy##) to be
move-assigned, instead of the intended head of the tail (##1234##).

If the data type's move operator does not invalidate its own data (i.e.
move is essentially a copy, as is the case with atomic types) then the
move assignment of ##y## onto ##Y## and ##x## onto ##X## is basically a
no-op. However, for types that do invalidate their own data when being
the source of a move (e.g. non-empty vectors and fbvectors), then the
end result will have bad data in place of ##X## and ##Y##.

This bug has lain dormant for a while.

Very few people emplace into a vector, and those few who do could be
saved by using a copy-movable type like int.

The original test case did not use a data type which poisoned itself
after being the source of a move, so the tests did not notice any

A new testsuite, which does poison itself, did notice.

Test Plan:
re-enable the test/stl_test mega-test. Run it. All tests

fbconfig -r folly && fbmake runtests_opt

Run fbvector through the new test suite (which initially caught the
error). It now passes.

Reviewed By:

FB internal diff: D1257258

8 years agoA few missing includes
Hans Fugal [Thu, 3 Apr 2014 18:05:32 +0000 (11:05 -0700)]
A few missing includes

Summary: These were required to build folly on OSX.

Test Plan: build

Reviewed By:

FB internal diff: D1256856

8 years ago(wangle) Use an atomic_flag to make FutureObject threadsafe
Hans Fugal [Thu, 3 Apr 2014 15:48:29 +0000 (08:48 -0700)]
(wangle) Use an atomic_flag to make FutureObject threadsafe

There are two operations on `FutureObject` (the backing object that both `Future` and `Promise` front), which are not threadsafe currently. Those are `setContinuation` and `fulfil`. They're not threadsafe because they have a race condition between checking to see whether they should run the continuation and the other side making its modification. This diff introduces a `std::atomic_flag` variable to avoid the race and make these threadsafe.

NB Making `Future`/`Promise` threadsafe does not make this pattern safe:


(In a hypothetical world where `Future::via` is another name for `executeWithSameThread`)
There is a race between the future from `via` and the call of `then` (and between the first and second `then`s), so the `then`s could execute in the far thread as intended, or they could execute in the current thread (probably not what was intended). This diff does not solve that semantic problem of which thread this runs in. But it does make it safer, in that all those continuations will always execute, just maybe not in the thread you had intended.

You may ask what is the benefit if that semantic problem still exists? That's up for debate. I think the safety is worth it: at least everything a n00b throws at it will *work*. The question of thread semantics can iterate. If we end up with thread semantics that really don't need locks (e.g. maybe because we move to an Rx/Later style "launch" at the end of a chain) we can always toss this atomic.

How does this affect performance? I'm doing some experiments and going to write up my findings. Early results indicate that outlier performance is impacted but not by a lot, but don't count those chickens until I hatch them. Part of the reason for sending this diff out is to run windtunnel experiments.

Test Plan: tests, benchmark, close inspection

Reviewed By:

FB internal diff: D1241822

Louis Kruger [Tue, 1 Apr 2014 20:22:42 +0000 (13:22 -0700)]

TIMED_SYNCHRONIZED_CONST is acquiring a read-write lock but releasing a shared lock.  Chaos ensues.

Test Plan: Add unit test, test times out before fix, passes after fix.

Reviewed By:

FB internal diff: D1251518

Blame Revision: D327492

8 years agoAdd ability to silence callbacks for Subprocess::communicate
Tudor Bosman [Tue, 1 Apr 2014 17:36:27 +0000 (10:36 -0700)]
Add ability to silence callbacks for Subprocess::communicate

Subprocess::communicate callbacks are level-triggered, which makes writing
"chatty" communication protocols difficult -- you often want to silence the
write callback until you read the expected message. Add
Subprocess::enableNotifications() for this purpose.

Test Plan: test added

Reviewed By:

FB internal diff: D1251564