using RCU rather than SRCU, because RCU is almost always faster
and easier to use than is SRCU.
- If you need to enter your read-side critical section in a
- hardirq or exception handler, and then exit that same read-side
- critical section in the task that was interrupted, then you need
- to srcu_read_lock_raw() and srcu_read_unlock_raw(), which avoid
- the lockdep checking that would otherwise this practice illegal.
-
Also unlike other forms of RCU, explicit initialization
and cleanup is required via init_srcu_struct() and
cleanup_srcu_struct(). These are passed a "struct srcu_struct"
"srcu_expedited": srcu_read_lock(), srcu_read_unlock() and
synchronize_srcu_expedited().
- "srcu_raw": srcu_read_lock_raw(), srcu_read_unlock_raw(),
- and call_srcu().
-
- "srcu_raw_sync": srcu_read_lock_raw(), srcu_read_unlock_raw(),
- and synchronize_srcu().
-
"sched": preempt_disable(), preempt_enable(), and
call_rcu_sched().
srcu_read_lock synchronize_srcu srcu_barrier
srcu_read_unlock call_srcu
- srcu_read_lock_raw synchronize_srcu_expedited
- srcu_read_unlock_raw
- srcu_dereference
+ srcu_dereference synchronize_srcu_expedited
SRCU: Initialization/cleanup
init_srcu_struct
a. Will readers need to block? If so, you need SRCU.
-b. Is it necessary to start a read-side critical section in a
- hardirq handler or exception handler, and then to complete
- this read-side critical section in the task that was
- interrupted? If so, you need SRCU's srcu_read_lock_raw() and
- srcu_read_unlock_raw() primitives.
-
-c. What about the -rt patchset? If readers would need to block
+b. What about the -rt patchset? If readers would need to block
in an non-rt kernel, you need SRCU. If readers would block
in a -rt kernel, but not in a non-rt kernel, SRCU is not
necessary.
-d. Do you need to treat NMI handlers, hardirq handlers,
+c. Do you need to treat NMI handlers, hardirq handlers,
and code segments with preemption disabled (whether
via preempt_disable(), local_irq_save(), local_bh_disable(),
or some other mechanism) as if they were explicit RCU readers?
If so, RCU-sched is the only choice that will work for you.
-e. Do you need RCU grace periods to complete even in the face
+d. Do you need RCU grace periods to complete even in the face
of softirq monopolization of one or more of the CPUs? For
example, is your code subject to network-based denial-of-service
attacks? If so, you need RCU-bh.
-f. Is your workload too update-intensive for normal use of
+e. Is your workload too update-intensive for normal use of
RCU, but inappropriate for other synchronization mechanisms?
If so, consider SLAB_DESTROY_BY_RCU. But please be careful!
-g. Do you need read-side critical sections that are respected
+f. Do you need read-side critical sections that are respected
even though they are in the middle of the idle loop, during
user-mode execution, or on an offlined CPU? If so, SRCU is the
only choice that will work for you.
-h. Otherwise, use RCU.
+g. Otherwise, use RCU.
Of course, this all assumes that you have determined that RCU is in fact
the right tool for your job.
calls and by forcing both kernel threads and interrupts
to execute elsewhere.
+Name: kworker/%u:%d%s (cpu, id, priority)
+Purpose: Execute workqueue requests
+To reduce its OS jitter, do any of the following:
+1. Run your workload at a real-time priority, which will allow
+ preempting the kworker daemons.
+2. Do any of the following needed to avoid jitter that your
+ application cannot tolerate:
+ a. Build your kernel with CONFIG_SLUB=y rather than
+ CONFIG_SLAB=y, thus avoiding the slab allocator's periodic
+ use of each CPU's workqueues to run its cache_reap()
+ function.
+ b. Avoid using oprofile, thus avoiding OS jitter from
+ wq_sync_buffer().
+ c. Limit your CPU frequency so that a CPU-frequency
+ governor is not required, possibly enlisting the aid of
+ special heatsinks or other cooling technologies. If done
+ correctly, and if you CPU architecture permits, you should
+ be able to build your kernel with CONFIG_CPU_FREQ=n to
+ avoid the CPU-frequency governor periodically running
+ on each CPU, including cs_dbs_timer() and od_dbs_timer().
+ WARNING: Please check your CPU specifications to
+ make sure that this is safe on your particular system.
+ d. It is not possible to entirely get rid of OS jitter
+ from vmstat_update() on CONFIG_SMP=y systems, but you
+ can decrease its frequency by writing a large value to
+ /proc/sys/vm/stat_interval. The default value is HZ,
+ for an interval of one second. Of course, larger values
+ will make your virtual-memory statistics update more
+ slowly. Of course, you can also run your workload at
+ a real-time priority, thus preempting vmstat_update().
+ e. If running on high-end powerpc servers, build with
+ CONFIG_PPC_RTAS_DAEMON=n. This prevents the RTAS
+ daemon from running on each CPU every second or so.
+ (This will require editing Kconfig files and will defeat
+ this platform's RAS functionality.) This avoids jitter
+ due to the rtas_event_scan() function.
+ WARNING: Please check your CPU specifications to
+ make sure that this is safe on your particular system.
+ f. If running on Cell Processor, build your kernel with
+ CBE_CPUFREQ_SPU_GOVERNOR=n to avoid OS jitter from
+ spu_gov_work().
+ WARNING: Please check your CPU specifications to
+ make sure that this is safe on your particular system.
+ g. If running on PowerMAC, build your kernel with
+ CONFIG_PMAC_RACKMETER=n to disable the CPU-meter,
+ avoiding OS jitter from rackmeter_do_timer().
+
Name: rcuc/%u
Purpose: Execute RCU callbacks in CONFIG_RCU_BOOST=y kernels.
To reduce its OS jitter, do at least one of the following:
some types of computationally intensive high-performance computing (HPC)
applications and for real-time applications.
-There are two main contexts in which the number of scheduling-clock
-interrupts can be reduced compared to the old-school approach of sending
-a scheduling-clock interrupt to all CPUs every jiffy whether they need
-it or not (CONFIG_HZ_PERIODIC=y or CONFIG_NO_HZ=n for older kernels):
+There are three main ways of managing scheduling-clock interrupts
+(also known as "scheduling-clock ticks" or simply "ticks"):
-1. Idle CPUs (CONFIG_NO_HZ_IDLE=y or CONFIG_NO_HZ=y for older kernels).
+1. Never omit scheduling-clock ticks (CONFIG_HZ_PERIODIC=y or
+ CONFIG_NO_HZ=n for older kernels). You normally will -not-
+ want to choose this option.
-2. CPUs having only one runnable task (CONFIG_NO_HZ_FULL=y).
+2. Omit scheduling-clock ticks on idle CPUs (CONFIG_NO_HZ_IDLE=y or
+ CONFIG_NO_HZ=y for older kernels). This is the most common
+ approach, and should be the default.
-These two cases are described in the following two sections, followed
+3. Omit scheduling-clock ticks on CPUs that are either idle or that
+ have only one runnable task (CONFIG_NO_HZ_FULL=y). Unless you
+ are running realtime applications or certain types of HPC
+ workloads, you will normally -not- want this option.
+
+These three cases are described in the following three sections, followed
by a third section on RCU-specific considerations and a fourth and final
section listing known issues.
-IDLE CPUs
+NEVER OMIT SCHEDULING-CLOCK TICKS
+
+Very old versions of Linux from the 1990s and the very early 2000s
+are incapable of omitting scheduling-clock ticks. It turns out that
+there are some situations where this old-school approach is still the
+right approach, for example, in heavy workloads with lots of tasks
+that use short bursts of CPU, where there are very frequent idle
+periods, but where these idle periods are also quite short (tens or
+hundreds of microseconds). For these types of workloads, scheduling
+clock interrupts will normally be delivered any way because there
+will frequently be multiple runnable tasks per CPU. In these cases,
+attempting to turn off the scheduling clock interrupt will have no effect
+other than increasing the overhead of switching to and from idle and
+transitioning between user and kernel execution.
+
+This mode of operation can be selected using CONFIG_HZ_PERIODIC=y (or
+CONFIG_NO_HZ=n for older kernels).
+
+However, if you are instead running a light workload with long idle
+periods, failing to omit scheduling-clock interrupts will result in
+excessive power consumption. This is especially bad on battery-powered
+devices, where it results in extremely short battery lifetimes. If you
+are running light workloads, you should therefore read the following
+section.
+
+In addition, if you are running either a real-time workload or an HPC
+workload with short iterations, the scheduling-clock interrupts can
+degrade your applications performance. If this describes your workload,
+you should read the following two sections.
+
+
+OMIT SCHEDULING-CLOCK TICKS FOR IDLE CPUs
If a CPU is idle, there is little point in sending it a scheduling-clock
interrupt. After all, the primary purpose of a scheduling-clock interrupt
dyntick-idle mode.
-CPUs WITH ONLY ONE RUNNABLE TASK
+OMIT SCHEDULING-CLOCK TICKS FOR CPUs WITH ONLY ONE RUNNABLE TASK
If a CPU has only one runnable task, there is little point in sending it
a scheduling-clock interrupt because there is no other task to switch to.
+Note that omitting scheduling-clock ticks for CPUs with only one runnable
+task implies also omitting them for idle CPUs.
The CONFIG_NO_HZ_FULL=y Kconfig option causes the kernel to avoid
sending scheduling-clock interrupts to CPUs with a single runnable task,
single runnable SCHED_FIFO task and multiple runnable SCHED_OTHER
tasks, even though these interrupts are unnecessary.
+ And even when there are multiple runnable tasks on a given CPU,
+ there is little point in interrupting that CPU until the current
+ running task's timeslice expires, which is almost always way
+ longer than the time of the next scheduling-clock interrupt.
+
Better handling of these sorts of situations is future work.
o A reboot is required to reconfigure both adaptive idle and RCU
scheduling-clock interrupt going in order to support accurate
timekeeping.
-o If there are adaptive-ticks CPUs, there will be at least one
- CPU keeping the scheduling-clock interrupt going, even if all
- CPUs are otherwise idle.
+o If there might potentially be some adaptive-ticks CPUs, there
+ will be at least one CPU keeping the scheduling-clock interrupt
+ going, even if all CPUs are otherwise idle.
+
+ Better handling of this situation is ongoing work.
+
+o Some process-handling operations still require the occasional
+ scheduling-clock tick. These operations include calculating CPU
+ load, maintaining sched average, computing CFS entity vruntime,
+ computing avenrun, and carrying out load balancing. They are
+ currently accommodated by scheduling-clock tick every second
+ or so. On-going work will eliminate the need even for these
+ infrequent scheduling-clock ticks.
up_out:
up_read(¤t->mm->mmap_sem);
- goto out;
+ goto out_srcu;
}
int kvmppc_core_init_vm(struct kvm *kvm)
__srcu_read_unlock(sp, idx);
}
-/**
- * srcu_read_lock_raw - register a new reader for an SRCU-protected structure.
- * @sp: srcu_struct in which to register the new reader.
- *
- * Enter an SRCU read-side critical section. Similar to srcu_read_lock(),
- * but avoids the RCU-lockdep checking. This means that it is legal to
- * use srcu_read_lock_raw() in one context, for example, in an exception
- * handler, and then have the matching srcu_read_unlock_raw() in another
- * context, for example in the task that took the exception.
- *
- * However, the entire SRCU read-side critical section must reside within a
- * single task. For example, beware of using srcu_read_lock_raw() in
- * a device interrupt handler and srcu_read_unlock() in the interrupted
- * task: This will not work if interrupts are threaded.
- */
-static inline int srcu_read_lock_raw(struct srcu_struct *sp)
-{
- unsigned long flags;
- int ret;
-
- local_irq_save(flags);
- ret = __srcu_read_lock(sp);
- local_irq_restore(flags);
- return ret;
-}
-
-/**
- * srcu_read_unlock_raw - unregister reader from an SRCU-protected structure.
- * @sp: srcu_struct in which to unregister the old reader.
- * @idx: return value from corresponding srcu_read_lock_raw().
- *
- * Exit an SRCU read-side critical section without lockdep-RCU checking.
- * See srcu_read_lock_raw() for more details.
- */
-static inline void srcu_read_unlock_raw(struct srcu_struct *sp, int idx)
-{
- unsigned long flags;
-
- local_irq_save(flags);
- __srcu_read_unlock(sp, idx);
- local_irq_restore(flags);
-}
-
#endif
Accept the default if unsure.
config RCU_NOCB_CPU
- bool "Offload RCU callback processing from boot-selected CPUs (EXPERIMENTAL"
+ bool "Offload RCU callback processing from boot-selected CPUs"
depends on TREE_RCU || TREE_PREEMPT_RCU
default n
help
prompt "Build-forced no-CBs CPUs"
default RCU_NOCB_CPU_NONE
help
- This option allows no-CBs CPUs to be specified at build time.
- Additional no-CBs CPUs may be specified by the rcu_nocbs=
- boot parameter.
+ This option allows no-CBs CPUs (whose RCU callbacks are invoked
+ from kthreads rather than from softirq context) to be specified
+ at build time. Additional no-CBs CPUs may be specified by
+ the rcu_nocbs= boot parameter.
config RCU_NOCB_CPU_NONE
bool "No build_forced no-CBs CPUs"
help
This option does not force any of the CPUs to be no-CBs CPUs.
Only CPUs designated by the rcu_nocbs= boot parameter will be
- no-CBs CPUs.
+ no-CBs CPUs, whose RCU callbacks will be invoked by per-CPU
+ kthreads whose names begin with "rcuo". All other CPUs will
+ invoke their own RCU callbacks in softirq context.
+
+ Select this option if you want to choose no-CBs CPUs at
+ boot time, for example, to allow testing of different no-CBs
+ configurations without having to rebuild the kernel each time.
config RCU_NOCB_CPU_ZERO
bool "CPU 0 is a build_forced no-CBs CPU"
depends on RCU_NOCB_CPU && !NO_HZ_FULL
help
- This option forces CPU 0 to be a no-CBs CPU. Additional CPUs
- may be designated as no-CBs CPUs using the rcu_nocbs= boot
- parameter will be no-CBs CPUs.
+ This option forces CPU 0 to be a no-CBs CPU, so that its RCU
+ callbacks are invoked by a per-CPU kthread whose name begins
+ with "rcuo". Additional CPUs may be designated as no-CBs
+ CPUs using the rcu_nocbs= boot parameter will be no-CBs CPUs.
+ All other CPUs will invoke their own RCU callbacks in softirq
+ context.
Select this if CPU 0 needs to be a no-CBs CPU for real-time
- or energy-efficiency reasons.
+ or energy-efficiency reasons, but the real reason it exists
+ is to ensure that randconfig testing covers mixed systems.
config RCU_NOCB_CPU_ALL
bool "All CPUs are build_forced no-CBs CPUs"
depends on RCU_NOCB_CPU
help
This option forces all CPUs to be no-CBs CPUs. The rcu_nocbs=
- boot parameter will be ignored.
+ boot parameter will be ignored. All CPUs' RCU callbacks will
+ be executed in the context of per-CPU rcuo kthreads created for
+ this purpose. Assuming that the kthreads whose names start with
+ "rcuo" are bound to "housekeeping" CPUs, this reduces OS jitter
+ on the remaining CPUs, but might decrease memory locality during
+ RCU-callback invocation, thus potentially degrading throughput.
Select this if all CPUs need to be no-CBs CPUs for real-time
or energy-efficiency reasons.
struct lockdep_map rcu_sched_lock_map =
STATIC_LOCKDEP_MAP_INIT("rcu_read_lock_sched", &rcu_sched_lock_key);
EXPORT_SYMBOL_GPL(rcu_sched_lock_map);
-#endif
-
-#ifdef CONFIG_DEBUG_LOCK_ALLOC
int debug_lockdep_rcu_enabled(void)
{
.name = "srcu_sync"
};
-static int srcu_torture_read_lock_raw(void) __acquires(&srcu_ctl)
-{
- return srcu_read_lock_raw(&srcu_ctl);
-}
-
-static void srcu_torture_read_unlock_raw(int idx) __releases(&srcu_ctl)
-{
- srcu_read_unlock_raw(&srcu_ctl, idx);
-}
-
-static struct rcu_torture_ops srcu_raw_ops = {
- .init = rcu_sync_torture_init,
- .readlock = srcu_torture_read_lock_raw,
- .read_delay = srcu_read_delay,
- .readunlock = srcu_torture_read_unlock_raw,
- .completed = srcu_torture_completed,
- .deferred_free = srcu_torture_deferred_free,
- .sync = srcu_torture_synchronize,
- .call = NULL,
- .cb_barrier = NULL,
- .stats = srcu_torture_stats,
- .name = "srcu_raw"
-};
-
-static struct rcu_torture_ops srcu_raw_sync_ops = {
- .init = rcu_sync_torture_init,
- .readlock = srcu_torture_read_lock_raw,
- .read_delay = srcu_read_delay,
- .readunlock = srcu_torture_read_unlock_raw,
- .completed = srcu_torture_completed,
- .deferred_free = rcu_sync_torture_deferred_free,
- .sync = srcu_torture_synchronize,
- .call = NULL,
- .cb_barrier = NULL,
- .stats = srcu_torture_stats,
- .name = "srcu_raw_sync"
-};
-
static void srcu_torture_synchronize_expedited(void)
{
synchronize_srcu_expedited(&srcu_ctl);
{ &rcu_ops, &rcu_sync_ops, &rcu_expedited_ops,
&rcu_bh_ops, &rcu_bh_sync_ops, &rcu_bh_expedited_ops,
&srcu_ops, &srcu_sync_ops, &srcu_expedited_ops,
- &srcu_raw_ops, &srcu_raw_sync_ops,
&sched_ops, &sched_sync_ops, &sched_expedited_ops, };
mutex_lock(&fullstop_mutex);
module_param(qhimark, long, 0444);
module_param(qlowmark, long, 0444);
-static ulong jiffies_till_first_fqs = RCU_JIFFIES_TILL_FORCE_QS;
-static ulong jiffies_till_next_fqs = RCU_JIFFIES_TILL_FORCE_QS;
+static ulong jiffies_till_first_fqs = ULONG_MAX;
+static ulong jiffies_till_next_fqs = ULONG_MAX;
module_param(jiffies_till_first_fqs, ulong, 0644);
module_param(jiffies_till_next_fqs, ulong, 0644);
rsp->jiffies_stall = jiffies + ULONG_MAX / 2;
}
-/*
- * Update CPU-local rcu_data state to record the newly noticed grace period.
- * This is used both when we started the grace period and when we notice
- * that someone else started the grace period. The caller must hold the
- * ->lock of the leaf rcu_node structure corresponding to the current CPU,
- * and must have irqs disabled.
- */
-static void __note_new_gpnum(struct rcu_state *rsp, struct rcu_node *rnp, struct rcu_data *rdp)
-{
- if (rdp->gpnum != rnp->gpnum) {
- /*
- * If the current grace period is waiting for this CPU,
- * set up to detect a quiescent state, otherwise don't
- * go looking for one.
- */
- rdp->gpnum = rnp->gpnum;
- trace_rcu_grace_period(rsp->name, rdp->gpnum, "cpustart");
- rdp->passed_quiesce = 0;
- rdp->qs_pending = !!(rnp->qsmask & rdp->grpmask);
- zero_cpu_stall_ticks(rdp);
- }
-}
-
-static void note_new_gpnum(struct rcu_state *rsp, struct rcu_data *rdp)
-{
- unsigned long flags;
- struct rcu_node *rnp;
-
- local_irq_save(flags);
- rnp = rdp->mynode;
- if (rdp->gpnum == ACCESS_ONCE(rnp->gpnum) || /* outside lock. */
- !raw_spin_trylock(&rnp->lock)) { /* irqs already off, so later. */
- local_irq_restore(flags);
- return;
- }
- __note_new_gpnum(rsp, rnp, rdp);
- raw_spin_unlock_irqrestore(&rnp->lock, flags);
-}
-
-/*
- * Did someone else start a new RCU grace period start since we last
- * checked? Update local state appropriately if so. Must be called
- * on the CPU corresponding to rdp.
- */
-static int
-check_for_new_grace_period(struct rcu_state *rsp, struct rcu_data *rdp)
-{
- unsigned long flags;
- int ret = 0;
-
- local_irq_save(flags);
- if (rdp->gpnum != rsp->gpnum) {
- note_new_gpnum(rsp, rdp);
- ret = 1;
- }
- local_irq_restore(flags);
- return ret;
-}
-
/*
* Initialize the specified rcu_data structure's callback list to empty.
*/
}
/*
- * Advance this CPU's callbacks, but only if the current grace period
- * has ended. This may be called only from the CPU to whom the rdp
- * belongs. In addition, the corresponding leaf rcu_node structure's
- * ->lock must be held by the caller, with irqs disabled.
+ * Update CPU-local rcu_data state to record the beginnings and ends of
+ * grace periods. The caller must hold the ->lock of the leaf rcu_node
+ * structure corresponding to the current CPU, and must have irqs disabled.
*/
-static void
-__rcu_process_gp_end(struct rcu_state *rsp, struct rcu_node *rnp, struct rcu_data *rdp)
+static void __note_gp_changes(struct rcu_state *rsp, struct rcu_node *rnp, struct rcu_data *rdp)
{
- /* Did another grace period end? */
+ /* Handle the ends of any preceding grace periods first. */
if (rdp->completed == rnp->completed) {
- /* No, so just accelerate recent callbacks. */
+ /* No grace period end, so just accelerate recent callbacks. */
rcu_accelerate_cbs(rsp, rnp, rdp);
} else {
/* Remember that we saw this grace-period completion. */
rdp->completed = rnp->completed;
trace_rcu_grace_period(rsp->name, rdp->gpnum, "cpuend");
+ }
+ if (rdp->gpnum != rnp->gpnum) {
/*
- * If we were in an extended quiescent state, we may have
- * missed some grace periods that others CPUs handled on
- * our behalf. Catch up with this state to avoid noting
- * spurious new grace periods. If another grace period
- * has started, then rnp->gpnum will have advanced, so
- * we will detect this later on. Of course, any quiescent
- * states we found for the old GP are now invalid.
- */
- if (ULONG_CMP_LT(rdp->gpnum, rdp->completed)) {
- rdp->gpnum = rdp->completed;
- rdp->passed_quiesce = 0;
- }
-
- /*
- * If RCU does not need a quiescent state from this CPU,
- * then make sure that this CPU doesn't go looking for one.
+ * If the current grace period is waiting for this CPU,
+ * set up to detect a quiescent state, otherwise don't
+ * go looking for one.
*/
- if ((rnp->qsmask & rdp->grpmask) == 0)
- rdp->qs_pending = 0;
+ rdp->gpnum = rnp->gpnum;
+ trace_rcu_grace_period(rsp->name, rdp->gpnum, "cpustart");
+ rdp->passed_quiesce = 0;
+ rdp->qs_pending = !!(rnp->qsmask & rdp->grpmask);
+ zero_cpu_stall_ticks(rdp);
}
}
-/*
- * Advance this CPU's callbacks, but only if the current grace period
- * has ended. This may be called only from the CPU to whom the rdp
- * belongs.
- */
-static void
-rcu_process_gp_end(struct rcu_state *rsp, struct rcu_data *rdp)
+static void note_gp_changes(struct rcu_state *rsp, struct rcu_data *rdp)
{
unsigned long flags;
struct rcu_node *rnp;
local_irq_save(flags);
rnp = rdp->mynode;
- if (rdp->completed == ACCESS_ONCE(rnp->completed) || /* outside lock. */
+ if ((rdp->gpnum == ACCESS_ONCE(rnp->gpnum) &&
+ rdp->completed == ACCESS_ONCE(rnp->completed)) || /* w/out lock. */
!raw_spin_trylock(&rnp->lock)) { /* irqs already off, so later. */
local_irq_restore(flags);
return;
}
- __rcu_process_gp_end(rsp, rnp, rdp);
+ __note_gp_changes(rsp, rnp, rdp);
raw_spin_unlock_irqrestore(&rnp->lock, flags);
}
-/*
- * Do per-CPU grace-period initialization for running CPU. The caller
- * must hold the lock of the leaf rcu_node structure corresponding to
- * this CPU.
- */
-static void
-rcu_start_gp_per_cpu(struct rcu_state *rsp, struct rcu_node *rnp, struct rcu_data *rdp)
-{
- /* Prior grace period ended, so advance callbacks for current CPU. */
- __rcu_process_gp_end(rsp, rnp, rdp);
-
- /* Set state so that this CPU will detect the next quiescent state. */
- __note_new_gpnum(rsp, rnp, rdp);
-}
-
/*
* Initialize a new grace period.
*/
WARN_ON_ONCE(rnp->completed != rsp->completed);
ACCESS_ONCE(rnp->completed) = rsp->completed;
if (rnp == rdp->mynode)
- rcu_start_gp_per_cpu(rsp, rnp, rdp);
+ __note_gp_changes(rsp, rnp, rdp);
rcu_preempt_boost_start_gp(rnp);
trace_rcu_grace_period_init(rsp->name, rnp->gpnum,
rnp->level, rnp->grplo,
ACCESS_ONCE(rnp->completed) = rsp->gpnum;
rdp = this_cpu_ptr(rsp->rda);
if (rnp == rdp->mynode)
- __rcu_process_gp_end(rsp, rnp, rdp);
+ __note_gp_changes(rsp, rnp, rdp);
nocb += rcu_future_gp_cleanup(rsp, rnp);
raw_spin_unlock_irq(&rnp->lock);
cond_resched();
static void
rcu_check_quiescent_state(struct rcu_state *rsp, struct rcu_data *rdp)
{
- /* If there is now a new grace period, record and return. */
- if (check_for_new_grace_period(rsp, rdp))
- return;
+ /* Check for grace-period ends and beginnings. */
+ note_gp_changes(rsp, rdp);
/*
* Does this CPU still need to do its part for current grace period?
WARN_ON_ONCE(rdp->beenonline == 0);
- /* Handle the end of a grace period that some other CPU ended. */
- rcu_process_gp_end(rsp, rdp);
-
/* Update RCU state based on any recent quiescent states. */
rcu_check_quiescent_state(rsp, rdp);
if (unlikely(rdp->qlen > rdp->qlen_last_fqs_check + qhimark)) {
/* Are we ignoring a completed grace period? */
- rcu_process_gp_end(rsp, rdp);
- check_for_new_grace_period(rsp, rdp);
+ note_gp_changes(rsp, rdp);
/* Start a new grace period if one not already started. */
if (!rcu_gp_in_progress(rsp)) {
*/
static void __init rcu_init_geometry(void)
{
+ ulong d;
int i;
int j;
int n = nr_cpu_ids;
int rcu_capacity[MAX_RCU_LVLS + 1];
+ /*
+ * Initialize any unspecified boot parameters.
+ * The default values of jiffies_till_first_fqs and
+ * jiffies_till_next_fqs are set to the RCU_JIFFIES_TILL_FORCE_QS
+ * value, which is a function of HZ, then adding one for each
+ * RCU_JIFFIES_FQS_DIV CPUs that might be on the system.
+ */
+ d = RCU_JIFFIES_TILL_FORCE_QS + nr_cpu_ids / RCU_JIFFIES_FQS_DIV;
+ if (jiffies_till_first_fqs == ULONG_MAX)
+ jiffies_till_first_fqs = d;
+ if (jiffies_till_next_fqs == ULONG_MAX)
+ jiffies_till_next_fqs = d;
+
/* If the compile-time values are accurate, just leave. */
if (rcu_fanout_leaf == CONFIG_RCU_FANOUT_LEAF &&
nr_cpu_ids == NR_CPUS)
#define RCU_FORCE_QS 3 /* Need to force quiescent state. */
#define RCU_SIGNAL_INIT RCU_SAVE_DYNTICK
-#define RCU_JIFFIES_TILL_FORCE_QS 3 /* for rsp->jiffies_force_qs */
+#define RCU_JIFFIES_TILL_FORCE_QS (1 + (HZ > 250) + (HZ > 500))
+ /* For jiffies_till_first_fqs and */
+ /* and jiffies_till_next_fqs. */
-#define RCU_STALL_RAT_DELAY 2 /* Allow other CPUs time */
- /* to take at least one */
- /* scheduling clock irq */
- /* before ratting on them. */
+#define RCU_JIFFIES_FQS_DIV 256 /* Very large systems need more */
+ /* delay between bouts of */
+ /* quiescent-state forcing. */
+
+#define RCU_STALL_RAT_DELAY 2 /* Allow other CPUs time to take */
+ /* at least one scheduling clock */
+ /* irq before ratting on them. */
#define rcu_wait(cond) \
do { \
pr_info("\tFour-level hierarchy is enabled.\n");
#endif
if (rcu_fanout_leaf != CONFIG_RCU_FANOUT_LEAF)
- pr_info("\tExperimental boot-time adjustment of leaf fanout to %d.\n", rcu_fanout_leaf);
+ pr_info("\tBoot-time adjustment of leaf fanout to %d.\n", rcu_fanout_leaf);
if (nr_cpu_ids != NR_CPUS)
pr_info("\tRCU restricting CPUs from NR_CPUS=%d to nr_cpu_ids=%d.\n", NR_CPUS, nr_cpu_ids);
#ifdef CONFIG_RCU_NOCB_CPU
have_rcu_nocb_mask = true;
}
#ifdef CONFIG_RCU_NOCB_CPU_ZERO
- pr_info("\tExperimental no-CBs CPU 0\n");
+ pr_info("\tOffload RCU callbacks from CPU 0\n");
cpumask_set_cpu(0, rcu_nocb_mask);
#endif /* #ifdef CONFIG_RCU_NOCB_CPU_ZERO */
#ifdef CONFIG_RCU_NOCB_CPU_ALL
- pr_info("\tExperimental no-CBs for all CPUs\n");
+ pr_info("\tOffload RCU callbacks from all CPUs\n");
cpumask_setall(rcu_nocb_mask);
#endif /* #ifdef CONFIG_RCU_NOCB_CPU_ALL */
#endif /* #ifndef CONFIG_RCU_NOCB_CPU_NONE */
if (have_rcu_nocb_mask) {
cpulist_scnprintf(nocb_buf, sizeof(nocb_buf), rcu_nocb_mask);
- pr_info("\tExperimental no-CBs CPUs: %s.\n", nocb_buf);
+ pr_info("\tOffload RCU callbacks from CPUs: %s.\n", nocb_buf);
if (rcu_nocb_poll)
- pr_info("\tExperimental polled no-CBs CPUs.\n");
+ pr_info("\tPoll for callbacks from no-CBs CPUs.\n");
}
#endif /* #ifdef CONFIG_RCU_NOCB_CPU */
}
*/
if (rdp->completed != rnp->completed &&
rdp->nxttail[RCU_DONE_TAIL] != rdp->nxttail[RCU_NEXT_TAIL])
- rcu_process_gp_end(rsp, rdp);
+ note_gp_changes(rsp, rdp);
if (cpu_has_callbacks_ready_to_invoke(rdp))
cbs_ready = true;
projects / firefly-linux-kernel-4.4.55.git / commitdiff