From: Vladimir Slaykovskiy <vslaykovsky@fb.com>
Date: Fri, 16 Sep 2016 09:53:58 +0000 (-0700)
Subject: Reduce footprint of ScribeClient
X-Git-Tag: v2016.09.19.00~4
X-Git-Url: http://plrg.eecs.uci.edu/git/?a=commitdiff_plain;h=2adb6eb0cd30c11f58052753f72e1db6bb77f25d;p=folly.git

Reduce footprint of ScribeClient

Summary:
I've found all code paths that contributed to the number of threads in new ScribeClient. Here's a list of flags that reduce the number of thread:
  --default_tls_thread_count=1 # was 32
  --sr2_event_base_pool_size=1
  --has_thrift_dispatcher_reporter=0 # creates a thrift server with a few extra threads
  --observer_manager_pool_size=1 # default is 4

Optimizations in this diff:
- Don't initialize OBCClient object if it is not used
- Added FLAG_observer_manager_pool_size to control size of the pool in ObserverManager

Currently OBC counters are switched off globally in ScribeCliean, it means that new code path that creates SR instances is disabled, but eventually we plan to switch it on. Clients that don't use SR and want to keep their RSS/threads profile low, should pass flags listed above. I'll announce this in ScribeUsers group as well.

Differential Revision: D3870704

fbshipit-source-id: 0efad6b3dc43c072ab11cac7e9461c09532ea11c
---

diff --git a/folly/experimental/observer/detail/ObserverManager.cpp b/folly/experimental/observer/detail/ObserverManager.cpp
index 9f8b7b7f..7cd3832c 100644
--- a/folly/experimental/observer/detail/ObserverManager.cpp
+++ b/folly/experimental/observer/detail/ObserverManager.cpp
@@ -25,8 +25,12 @@ FOLLY_TLS bool ObserverManager::inManagerThread_{false};
 FOLLY_TLS ObserverManager::DependencyRecorder::Dependencies*
     ObserverManager::DependencyRecorder::currentDependencies_{nullptr};
 
+DEFINE_int32(
+    observer_manager_pool_size,
+    4,
+    "How many internal threads ObserverManager should use");
+
 namespace {
-constexpr size_t kCurrentThreadPoolSize{4};
 constexpr size_t kCurrentQueueSize{10 * 1024};
 constexpr size_t kNextQueueSize{10 * 1024};
 }
@@ -34,7 +38,11 @@ constexpr size_t kNextQueueSize{10 * 1024};
 class ObserverManager::CurrentQueue {
  public:
   CurrentQueue() : queue_(kCurrentQueueSize) {
-    for (size_t i = 0; i < kCurrentThreadPoolSize; ++i) {
+    if (FLAGS_observer_manager_pool_size < 1) {
+      LOG(ERROR) << "--observer_manager_pool_size should be >= 1";
+      FLAGS_observer_manager_pool_size = 1;
+    }
+    for (int32_t i = 0; i < FLAGS_observer_manager_pool_size; ++i) {
       threads_.emplace_back([&]() {
         ObserverManager::inManagerThread_ = true;