mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
synced 2024-08-26 10:49:33 +00:00
![Federico Vaga](/assets/img/avatar_default.png)
This patch includes the kernel-hacking translation in Italian (both hacking.rst and locking.rst). It adds also the anchors for the english kernel-hacking documents. Signed-off-by: Federico Vaga <federico.vaga@vaga.pv.it> Signed-off-by: Jonathan Corbet <corbet@lwn.net>
1448 lines
53 KiB
ReStructuredText
1448 lines
53 KiB
ReStructuredText
.. _kernel_hacking_lock:
|
|
|
|
===========================
|
|
Unreliable Guide To Locking
|
|
===========================
|
|
|
|
:Author: Rusty Russell
|
|
|
|
Introduction
|
|
============
|
|
|
|
Welcome, to Rusty's Remarkably Unreliable Guide to Kernel Locking
|
|
issues. This document describes the locking systems in the Linux Kernel
|
|
in 2.6.
|
|
|
|
With the wide availability of HyperThreading, and preemption in the
|
|
Linux Kernel, everyone hacking on the kernel needs to know the
|
|
fundamentals of concurrency and locking for SMP.
|
|
|
|
The Problem With Concurrency
|
|
============================
|
|
|
|
(Skip this if you know what a Race Condition is).
|
|
|
|
In a normal program, you can increment a counter like so:
|
|
|
|
::
|
|
|
|
very_important_count++;
|
|
|
|
|
|
This is what they would expect to happen:
|
|
|
|
|
|
.. table:: Expected Results
|
|
|
|
+------------------------------------+------------------------------------+
|
|
| Instance 1 | Instance 2 |
|
|
+====================================+====================================+
|
|
| read very_important_count (5) | |
|
|
+------------------------------------+------------------------------------+
|
|
| add 1 (6) | |
|
|
+------------------------------------+------------------------------------+
|
|
| write very_important_count (6) | |
|
|
+------------------------------------+------------------------------------+
|
|
| | read very_important_count (6) |
|
|
+------------------------------------+------------------------------------+
|
|
| | add 1 (7) |
|
|
+------------------------------------+------------------------------------+
|
|
| | write very_important_count (7) |
|
|
+------------------------------------+------------------------------------+
|
|
|
|
This is what might happen:
|
|
|
|
.. table:: Possible Results
|
|
|
|
+------------------------------------+------------------------------------+
|
|
| Instance 1 | Instance 2 |
|
|
+====================================+====================================+
|
|
| read very_important_count (5) | |
|
|
+------------------------------------+------------------------------------+
|
|
| | read very_important_count (5) |
|
|
+------------------------------------+------------------------------------+
|
|
| add 1 (6) | |
|
|
+------------------------------------+------------------------------------+
|
|
| | add 1 (6) |
|
|
+------------------------------------+------------------------------------+
|
|
| write very_important_count (6) | |
|
|
+------------------------------------+------------------------------------+
|
|
| | write very_important_count (6) |
|
|
+------------------------------------+------------------------------------+
|
|
|
|
|
|
Race Conditions and Critical Regions
|
|
------------------------------------
|
|
|
|
This overlap, where the result depends on the relative timing of
|
|
multiple tasks, is called a race condition. The piece of code containing
|
|
the concurrency issue is called a critical region. And especially since
|
|
Linux starting running on SMP machines, they became one of the major
|
|
issues in kernel design and implementation.
|
|
|
|
Preemption can have the same effect, even if there is only one CPU: by
|
|
preempting one task during the critical region, we have exactly the same
|
|
race condition. In this case the thread which preempts might run the
|
|
critical region itself.
|
|
|
|
The solution is to recognize when these simultaneous accesses occur, and
|
|
use locks to make sure that only one instance can enter the critical
|
|
region at any time. There are many friendly primitives in the Linux
|
|
kernel to help you do this. And then there are the unfriendly
|
|
primitives, but I'll pretend they don't exist.
|
|
|
|
Locking in the Linux Kernel
|
|
===========================
|
|
|
|
If I could give you one piece of advice: never sleep with anyone crazier
|
|
than yourself. But if I had to give you advice on locking: **keep it
|
|
simple**.
|
|
|
|
Be reluctant to introduce new locks.
|
|
|
|
Strangely enough, this last one is the exact reverse of my advice when
|
|
you **have** slept with someone crazier than yourself. And you should
|
|
think about getting a big dog.
|
|
|
|
Two Main Types of Kernel Locks: Spinlocks and Mutexes
|
|
-----------------------------------------------------
|
|
|
|
There are two main types of kernel locks. The fundamental type is the
|
|
spinlock (``include/asm/spinlock.h``), which is a very simple
|
|
single-holder lock: if you can't get the spinlock, you keep trying
|
|
(spinning) until you can. Spinlocks are very small and fast, and can be
|
|
used anywhere.
|
|
|
|
The second type is a mutex (``include/linux/mutex.h``): it is like a
|
|
spinlock, but you may block holding a mutex. If you can't lock a mutex,
|
|
your task will suspend itself, and be woken up when the mutex is
|
|
released. This means the CPU can do something else while you are
|
|
waiting. There are many cases when you simply can't sleep (see
|
|
`What Functions Are Safe To Call From Interrupts? <#sleeping-things>`__),
|
|
and so have to use a spinlock instead.
|
|
|
|
Neither type of lock is recursive: see
|
|
`Deadlock: Simple and Advanced <#deadlock>`__.
|
|
|
|
Locks and Uniprocessor Kernels
|
|
------------------------------
|
|
|
|
For kernels compiled without ``CONFIG_SMP``, and without
|
|
``CONFIG_PREEMPT`` spinlocks do not exist at all. This is an excellent
|
|
design decision: when no-one else can run at the same time, there is no
|
|
reason to have a lock.
|
|
|
|
If the kernel is compiled without ``CONFIG_SMP``, but ``CONFIG_PREEMPT``
|
|
is set, then spinlocks simply disable preemption, which is sufficient to
|
|
prevent any races. For most purposes, we can think of preemption as
|
|
equivalent to SMP, and not worry about it separately.
|
|
|
|
You should always test your locking code with ``CONFIG_SMP`` and
|
|
``CONFIG_PREEMPT`` enabled, even if you don't have an SMP test box,
|
|
because it will still catch some kinds of locking bugs.
|
|
|
|
Mutexes still exist, because they are required for synchronization
|
|
between user contexts, as we will see below.
|
|
|
|
Locking Only In User Context
|
|
----------------------------
|
|
|
|
If you have a data structure which is only ever accessed from user
|
|
context, then you can use a simple mutex (``include/linux/mutex.h``) to
|
|
protect it. This is the most trivial case: you initialize the mutex.
|
|
Then you can call :c:func:`mutex_lock_interruptible()` to grab the
|
|
mutex, and :c:func:`mutex_unlock()` to release it. There is also a
|
|
:c:func:`mutex_lock()`, which should be avoided, because it will
|
|
not return if a signal is received.
|
|
|
|
Example: ``net/netfilter/nf_sockopt.c`` allows registration of new
|
|
:c:func:`setsockopt()` and :c:func:`getsockopt()` calls, with
|
|
:c:func:`nf_register_sockopt()`. Registration and de-registration
|
|
are only done on module load and unload (and boot time, where there is
|
|
no concurrency), and the list of registrations is only consulted for an
|
|
unknown :c:func:`setsockopt()` or :c:func:`getsockopt()` system
|
|
call. The ``nf_sockopt_mutex`` is perfect to protect this, especially
|
|
since the setsockopt and getsockopt calls may well sleep.
|
|
|
|
Locking Between User Context and Softirqs
|
|
-----------------------------------------
|
|
|
|
If a softirq shares data with user context, you have two problems.
|
|
Firstly, the current user context can be interrupted by a softirq, and
|
|
secondly, the critical region could be entered from another CPU. This is
|
|
where :c:func:`spin_lock_bh()` (``include/linux/spinlock.h``) is
|
|
used. It disables softirqs on that CPU, then grabs the lock.
|
|
:c:func:`spin_unlock_bh()` does the reverse. (The '_bh' suffix is
|
|
a historical reference to "Bottom Halves", the old name for software
|
|
interrupts. It should really be called spin_lock_softirq()' in a
|
|
perfect world).
|
|
|
|
Note that you can also use :c:func:`spin_lock_irq()` or
|
|
:c:func:`spin_lock_irqsave()` here, which stop hardware interrupts
|
|
as well: see `Hard IRQ Context <#hard-irq-context>`__.
|
|
|
|
This works perfectly for UP as well: the spin lock vanishes, and this
|
|
macro simply becomes :c:func:`local_bh_disable()`
|
|
(``include/linux/interrupt.h``), which protects you from the softirq
|
|
being run.
|
|
|
|
Locking Between User Context and Tasklets
|
|
-----------------------------------------
|
|
|
|
This is exactly the same as above, because tasklets are actually run
|
|
from a softirq.
|
|
|
|
Locking Between User Context and Timers
|
|
---------------------------------------
|
|
|
|
This, too, is exactly the same as above, because timers are actually run
|
|
from a softirq. From a locking point of view, tasklets and timers are
|
|
identical.
|
|
|
|
Locking Between Tasklets/Timers
|
|
-------------------------------
|
|
|
|
Sometimes a tasklet or timer might want to share data with another
|
|
tasklet or timer.
|
|
|
|
The Same Tasklet/Timer
|
|
~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Since a tasklet is never run on two CPUs at once, you don't need to
|
|
worry about your tasklet being reentrant (running twice at once), even
|
|
on SMP.
|
|
|
|
Different Tasklets/Timers
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
If another tasklet/timer wants to share data with your tasklet or timer
|
|
, you will both need to use :c:func:`spin_lock()` and
|
|
:c:func:`spin_unlock()` calls. :c:func:`spin_lock_bh()` is
|
|
unnecessary here, as you are already in a tasklet, and none will be run
|
|
on the same CPU.
|
|
|
|
Locking Between Softirqs
|
|
------------------------
|
|
|
|
Often a softirq might want to share data with itself or a tasklet/timer.
|
|
|
|
The Same Softirq
|
|
~~~~~~~~~~~~~~~~
|
|
|
|
The same softirq can run on the other CPUs: you can use a per-CPU array
|
|
(see `Per-CPU Data <#per-cpu-data>`__) for better performance. If you're
|
|
going so far as to use a softirq, you probably care about scalable
|
|
performance enough to justify the extra complexity.
|
|
|
|
You'll need to use :c:func:`spin_lock()` and
|
|
:c:func:`spin_unlock()` for shared data.
|
|
|
|
Different Softirqs
|
|
~~~~~~~~~~~~~~~~~~
|
|
|
|
You'll need to use :c:func:`spin_lock()` and
|
|
:c:func:`spin_unlock()` for shared data, whether it be a timer,
|
|
tasklet, different softirq or the same or another softirq: any of them
|
|
could be running on a different CPU.
|
|
|
|
Hard IRQ Context
|
|
================
|
|
|
|
Hardware interrupts usually communicate with a tasklet or softirq.
|
|
Frequently this involves putting work in a queue, which the softirq will
|
|
take out.
|
|
|
|
Locking Between Hard IRQ and Softirqs/Tasklets
|
|
----------------------------------------------
|
|
|
|
If a hardware irq handler shares data with a softirq, you have two
|
|
concerns. Firstly, the softirq processing can be interrupted by a
|
|
hardware interrupt, and secondly, the critical region could be entered
|
|
by a hardware interrupt on another CPU. This is where
|
|
:c:func:`spin_lock_irq()` is used. It is defined to disable
|
|
interrupts on that cpu, then grab the lock.
|
|
:c:func:`spin_unlock_irq()` does the reverse.
|
|
|
|
The irq handler does not to use :c:func:`spin_lock_irq()`, because
|
|
the softirq cannot run while the irq handler is running: it can use
|
|
:c:func:`spin_lock()`, which is slightly faster. The only exception
|
|
would be if a different hardware irq handler uses the same lock:
|
|
:c:func:`spin_lock_irq()` will stop that from interrupting us.
|
|
|
|
This works perfectly for UP as well: the spin lock vanishes, and this
|
|
macro simply becomes :c:func:`local_irq_disable()`
|
|
(``include/asm/smp.h``), which protects you from the softirq/tasklet/BH
|
|
being run.
|
|
|
|
:c:func:`spin_lock_irqsave()` (``include/linux/spinlock.h``) is a
|
|
variant which saves whether interrupts were on or off in a flags word,
|
|
which is passed to :c:func:`spin_unlock_irqrestore()`. This means
|
|
that the same code can be used inside an hard irq handler (where
|
|
interrupts are already off) and in softirqs (where the irq disabling is
|
|
required).
|
|
|
|
Note that softirqs (and hence tasklets and timers) are run on return
|
|
from hardware interrupts, so :c:func:`spin_lock_irq()` also stops
|
|
these. In that sense, :c:func:`spin_lock_irqsave()` is the most
|
|
general and powerful locking function.
|
|
|
|
Locking Between Two Hard IRQ Handlers
|
|
-------------------------------------
|
|
|
|
It is rare to have to share data between two IRQ handlers, but if you
|
|
do, :c:func:`spin_lock_irqsave()` should be used: it is
|
|
architecture-specific whether all interrupts are disabled inside irq
|
|
handlers themselves.
|
|
|
|
Cheat Sheet For Locking
|
|
=======================
|
|
|
|
Pete Zaitcev gives the following summary:
|
|
|
|
- If you are in a process context (any syscall) and want to lock other
|
|
process out, use a mutex. You can take a mutex and sleep
|
|
(``copy_from_user*(`` or ``kmalloc(x,GFP_KERNEL)``).
|
|
|
|
- Otherwise (== data can be touched in an interrupt), use
|
|
:c:func:`spin_lock_irqsave()` and
|
|
:c:func:`spin_unlock_irqrestore()`.
|
|
|
|
- Avoid holding spinlock for more than 5 lines of code and across any
|
|
function call (except accessors like :c:func:`readb()`).
|
|
|
|
Table of Minimum Requirements
|
|
-----------------------------
|
|
|
|
The following table lists the **minimum** locking requirements between
|
|
various contexts. In some cases, the same context can only be running on
|
|
one CPU at a time, so no locking is required for that context (eg. a
|
|
particular thread can only run on one CPU at a time, but if it needs
|
|
shares data with another thread, locking is required).
|
|
|
|
Remember the advice above: you can always use
|
|
:c:func:`spin_lock_irqsave()`, which is a superset of all other
|
|
spinlock primitives.
|
|
|
|
============== ============= ============= ========= ========= ========= ========= ======= ======= ============== ==============
|
|
. IRQ Handler A IRQ Handler B Softirq A Softirq B Tasklet A Tasklet B Timer A Timer B User Context A User Context B
|
|
============== ============= ============= ========= ========= ========= ========= ======= ======= ============== ==============
|
|
IRQ Handler A None
|
|
IRQ Handler B SLIS None
|
|
Softirq A SLI SLI SL
|
|
Softirq B SLI SLI SL SL
|
|
Tasklet A SLI SLI SL SL None
|
|
Tasklet B SLI SLI SL SL SL None
|
|
Timer A SLI SLI SL SL SL SL None
|
|
Timer B SLI SLI SL SL SL SL SL None
|
|
User Context A SLI SLI SLBH SLBH SLBH SLBH SLBH SLBH None
|
|
User Context B SLI SLI SLBH SLBH SLBH SLBH SLBH SLBH MLI None
|
|
============== ============= ============= ========= ========= ========= ========= ======= ======= ============== ==============
|
|
|
|
Table: Table of Locking Requirements
|
|
|
|
+--------+----------------------------+
|
|
| SLIS | spin_lock_irqsave |
|
|
+--------+----------------------------+
|
|
| SLI | spin_lock_irq |
|
|
+--------+----------------------------+
|
|
| SL | spin_lock |
|
|
+--------+----------------------------+
|
|
| SLBH | spin_lock_bh |
|
|
+--------+----------------------------+
|
|
| MLI | mutex_lock_interruptible |
|
|
+--------+----------------------------+
|
|
|
|
Table: Legend for Locking Requirements Table
|
|
|
|
The trylock Functions
|
|
=====================
|
|
|
|
There are functions that try to acquire a lock only once and immediately
|
|
return a value telling about success or failure to acquire the lock.
|
|
They can be used if you need no access to the data protected with the
|
|
lock when some other thread is holding the lock. You should acquire the
|
|
lock later if you then need access to the data protected with the lock.
|
|
|
|
:c:func:`spin_trylock()` does not spin but returns non-zero if it
|
|
acquires the spinlock on the first try or 0 if not. This function can be
|
|
used in all contexts like :c:func:`spin_lock()`: you must have
|
|
disabled the contexts that might interrupt you and acquire the spin
|
|
lock.
|
|
|
|
:c:func:`mutex_trylock()` does not suspend your task but returns
|
|
non-zero if it could lock the mutex on the first try or 0 if not. This
|
|
function cannot be safely used in hardware or software interrupt
|
|
contexts despite not sleeping.
|
|
|
|
Common Examples
|
|
===============
|
|
|
|
Let's step through a simple example: a cache of number to name mappings.
|
|
The cache keeps a count of how often each of the objects is used, and
|
|
when it gets full, throws out the least used one.
|
|
|
|
All In User Context
|
|
-------------------
|
|
|
|
For our first example, we assume that all operations are in user context
|
|
(ie. from system calls), so we can sleep. This means we can use a mutex
|
|
to protect the cache and all the objects within it. Here's the code::
|
|
|
|
#include <linux/list.h>
|
|
#include <linux/slab.h>
|
|
#include <linux/string.h>
|
|
#include <linux/mutex.h>
|
|
#include <asm/errno.h>
|
|
|
|
struct object
|
|
{
|
|
struct list_head list;
|
|
int id;
|
|
char name[32];
|
|
int popularity;
|
|
};
|
|
|
|
/* Protects the cache, cache_num, and the objects within it */
|
|
static DEFINE_MUTEX(cache_lock);
|
|
static LIST_HEAD(cache);
|
|
static unsigned int cache_num = 0;
|
|
#define MAX_CACHE_SIZE 10
|
|
|
|
/* Must be holding cache_lock */
|
|
static struct object *__cache_find(int id)
|
|
{
|
|
struct object *i;
|
|
|
|
list_for_each_entry(i, &cache, list)
|
|
if (i->id == id) {
|
|
i->popularity++;
|
|
return i;
|
|
}
|
|
return NULL;
|
|
}
|
|
|
|
/* Must be holding cache_lock */
|
|
static void __cache_delete(struct object *obj)
|
|
{
|
|
BUG_ON(!obj);
|
|
list_del(&obj->list);
|
|
kfree(obj);
|
|
cache_num--;
|
|
}
|
|
|
|
/* Must be holding cache_lock */
|
|
static void __cache_add(struct object *obj)
|
|
{
|
|
list_add(&obj->list, &cache);
|
|
if (++cache_num > MAX_CACHE_SIZE) {
|
|
struct object *i, *outcast = NULL;
|
|
list_for_each_entry(i, &cache, list) {
|
|
if (!outcast || i->popularity < outcast->popularity)
|
|
outcast = i;
|
|
}
|
|
__cache_delete(outcast);
|
|
}
|
|
}
|
|
|
|
int cache_add(int id, const char *name)
|
|
{
|
|
struct object *obj;
|
|
|
|
if ((obj = kmalloc(sizeof(*obj), GFP_KERNEL)) == NULL)
|
|
return -ENOMEM;
|
|
|
|
strlcpy(obj->name, name, sizeof(obj->name));
|
|
obj->id = id;
|
|
obj->popularity = 0;
|
|
|
|
mutex_lock(&cache_lock);
|
|
__cache_add(obj);
|
|
mutex_unlock(&cache_lock);
|
|
return 0;
|
|
}
|
|
|
|
void cache_delete(int id)
|
|
{
|
|
mutex_lock(&cache_lock);
|
|
__cache_delete(__cache_find(id));
|
|
mutex_unlock(&cache_lock);
|
|
}
|
|
|
|
int cache_find(int id, char *name)
|
|
{
|
|
struct object *obj;
|
|
int ret = -ENOENT;
|
|
|
|
mutex_lock(&cache_lock);
|
|
obj = __cache_find(id);
|
|
if (obj) {
|
|
ret = 0;
|
|
strcpy(name, obj->name);
|
|
}
|
|
mutex_unlock(&cache_lock);
|
|
return ret;
|
|
}
|
|
|
|
Note that we always make sure we have the cache_lock when we add,
|
|
delete, or look up the cache: both the cache infrastructure itself and
|
|
the contents of the objects are protected by the lock. In this case it's
|
|
easy, since we copy the data for the user, and never let them access the
|
|
objects directly.
|
|
|
|
There is a slight (and common) optimization here: in
|
|
:c:func:`cache_add()` we set up the fields of the object before
|
|
grabbing the lock. This is safe, as no-one else can access it until we
|
|
put it in cache.
|
|
|
|
Accessing From Interrupt Context
|
|
--------------------------------
|
|
|
|
Now consider the case where :c:func:`cache_find()` can be called
|
|
from interrupt context: either a hardware interrupt or a softirq. An
|
|
example would be a timer which deletes object from the cache.
|
|
|
|
The change is shown below, in standard patch format: the ``-`` are lines
|
|
which are taken away, and the ``+`` are lines which are added.
|
|
|
|
::
|
|
|
|
--- cache.c.usercontext 2003-12-09 13:58:54.000000000 +1100
|
|
+++ cache.c.interrupt 2003-12-09 14:07:49.000000000 +1100
|
|
@@ -12,7 +12,7 @@
|
|
int popularity;
|
|
};
|
|
|
|
-static DEFINE_MUTEX(cache_lock);
|
|
+static DEFINE_SPINLOCK(cache_lock);
|
|
static LIST_HEAD(cache);
|
|
static unsigned int cache_num = 0;
|
|
#define MAX_CACHE_SIZE 10
|
|
@@ -55,6 +55,7 @@
|
|
int cache_add(int id, const char *name)
|
|
{
|
|
struct object *obj;
|
|
+ unsigned long flags;
|
|
|
|
if ((obj = kmalloc(sizeof(*obj), GFP_KERNEL)) == NULL)
|
|
return -ENOMEM;
|
|
@@ -63,30 +64,33 @@
|
|
obj->id = id;
|
|
obj->popularity = 0;
|
|
|
|
- mutex_lock(&cache_lock);
|
|
+ spin_lock_irqsave(&cache_lock, flags);
|
|
__cache_add(obj);
|
|
- mutex_unlock(&cache_lock);
|
|
+ spin_unlock_irqrestore(&cache_lock, flags);
|
|
return 0;
|
|
}
|
|
|
|
void cache_delete(int id)
|
|
{
|
|
- mutex_lock(&cache_lock);
|
|
+ unsigned long flags;
|
|
+
|
|
+ spin_lock_irqsave(&cache_lock, flags);
|
|
__cache_delete(__cache_find(id));
|
|
- mutex_unlock(&cache_lock);
|
|
+ spin_unlock_irqrestore(&cache_lock, flags);
|
|
}
|
|
|
|
int cache_find(int id, char *name)
|
|
{
|
|
struct object *obj;
|
|
int ret = -ENOENT;
|
|
+ unsigned long flags;
|
|
|
|
- mutex_lock(&cache_lock);
|
|
+ spin_lock_irqsave(&cache_lock, flags);
|
|
obj = __cache_find(id);
|
|
if (obj) {
|
|
ret = 0;
|
|
strcpy(name, obj->name);
|
|
}
|
|
- mutex_unlock(&cache_lock);
|
|
+ spin_unlock_irqrestore(&cache_lock, flags);
|
|
return ret;
|
|
}
|
|
|
|
Note that the :c:func:`spin_lock_irqsave()` will turn off
|
|
interrupts if they are on, otherwise does nothing (if we are already in
|
|
an interrupt handler), hence these functions are safe to call from any
|
|
context.
|
|
|
|
Unfortunately, :c:func:`cache_add()` calls :c:func:`kmalloc()`
|
|
with the ``GFP_KERNEL`` flag, which is only legal in user context. I
|
|
have assumed that :c:func:`cache_add()` is still only called in
|
|
user context, otherwise this should become a parameter to
|
|
:c:func:`cache_add()`.
|
|
|
|
Exposing Objects Outside This File
|
|
----------------------------------
|
|
|
|
If our objects contained more information, it might not be sufficient to
|
|
copy the information in and out: other parts of the code might want to
|
|
keep pointers to these objects, for example, rather than looking up the
|
|
id every time. This produces two problems.
|
|
|
|
The first problem is that we use the ``cache_lock`` to protect objects:
|
|
we'd need to make this non-static so the rest of the code can use it.
|
|
This makes locking trickier, as it is no longer all in one place.
|
|
|
|
The second problem is the lifetime problem: if another structure keeps a
|
|
pointer to an object, it presumably expects that pointer to remain
|
|
valid. Unfortunately, this is only guaranteed while you hold the lock,
|
|
otherwise someone might call :c:func:`cache_delete()` and even
|
|
worse, add another object, re-using the same address.
|
|
|
|
As there is only one lock, you can't hold it forever: no-one else would
|
|
get any work done.
|
|
|
|
The solution to this problem is to use a reference count: everyone who
|
|
has a pointer to the object increases it when they first get the object,
|
|
and drops the reference count when they're finished with it. Whoever
|
|
drops it to zero knows it is unused, and can actually delete it.
|
|
|
|
Here is the code::
|
|
|
|
--- cache.c.interrupt 2003-12-09 14:25:43.000000000 +1100
|
|
+++ cache.c.refcnt 2003-12-09 14:33:05.000000000 +1100
|
|
@@ -7,6 +7,7 @@
|
|
struct object
|
|
{
|
|
struct list_head list;
|
|
+ unsigned int refcnt;
|
|
int id;
|
|
char name[32];
|
|
int popularity;
|
|
@@ -17,6 +18,35 @@
|
|
static unsigned int cache_num = 0;
|
|
#define MAX_CACHE_SIZE 10
|
|
|
|
+static void __object_put(struct object *obj)
|
|
+{
|
|
+ if (--obj->refcnt == 0)
|
|
+ kfree(obj);
|
|
+}
|
|
+
|
|
+static void __object_get(struct object *obj)
|
|
+{
|
|
+ obj->refcnt++;
|
|
+}
|
|
+
|
|
+void object_put(struct object *obj)
|
|
+{
|
|
+ unsigned long flags;
|
|
+
|
|
+ spin_lock_irqsave(&cache_lock, flags);
|
|
+ __object_put(obj);
|
|
+ spin_unlock_irqrestore(&cache_lock, flags);
|
|
+}
|
|
+
|
|
+void object_get(struct object *obj)
|
|
+{
|
|
+ unsigned long flags;
|
|
+
|
|
+ spin_lock_irqsave(&cache_lock, flags);
|
|
+ __object_get(obj);
|
|
+ spin_unlock_irqrestore(&cache_lock, flags);
|
|
+}
|
|
+
|
|
/* Must be holding cache_lock */
|
|
static struct object *__cache_find(int id)
|
|
{
|
|
@@ -35,6 +65,7 @@
|
|
{
|
|
BUG_ON(!obj);
|
|
list_del(&obj->list);
|
|
+ __object_put(obj);
|
|
cache_num--;
|
|
}
|
|
|
|
@@ -63,6 +94,7 @@
|
|
strlcpy(obj->name, name, sizeof(obj->name));
|
|
obj->id = id;
|
|
obj->popularity = 0;
|
|
+ obj->refcnt = 1; /* The cache holds a reference */
|
|
|
|
spin_lock_irqsave(&cache_lock, flags);
|
|
__cache_add(obj);
|
|
@@ -79,18 +111,15 @@
|
|
spin_unlock_irqrestore(&cache_lock, flags);
|
|
}
|
|
|
|
-int cache_find(int id, char *name)
|
|
+struct object *cache_find(int id)
|
|
{
|
|
struct object *obj;
|
|
- int ret = -ENOENT;
|
|
unsigned long flags;
|
|
|
|
spin_lock_irqsave(&cache_lock, flags);
|
|
obj = __cache_find(id);
|
|
- if (obj) {
|
|
- ret = 0;
|
|
- strcpy(name, obj->name);
|
|
- }
|
|
+ if (obj)
|
|
+ __object_get(obj);
|
|
spin_unlock_irqrestore(&cache_lock, flags);
|
|
- return ret;
|
|
+ return obj;
|
|
}
|
|
|
|
We encapsulate the reference counting in the standard 'get' and 'put'
|
|
functions. Now we can return the object itself from
|
|
:c:func:`cache_find()` which has the advantage that the user can
|
|
now sleep holding the object (eg. to :c:func:`copy_to_user()` to
|
|
name to userspace).
|
|
|
|
The other point to note is that I said a reference should be held for
|
|
every pointer to the object: thus the reference count is 1 when first
|
|
inserted into the cache. In some versions the framework does not hold a
|
|
reference count, but they are more complicated.
|
|
|
|
Using Atomic Operations For The Reference Count
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
In practice, :c:type:`atomic_t` would usually be used for refcnt. There are a
|
|
number of atomic operations defined in ``include/asm/atomic.h``: these
|
|
are guaranteed to be seen atomically from all CPUs in the system, so no
|
|
lock is required. In this case, it is simpler than using spinlocks,
|
|
although for anything non-trivial using spinlocks is clearer. The
|
|
:c:func:`atomic_inc()` and :c:func:`atomic_dec_and_test()`
|
|
are used instead of the standard increment and decrement operators, and
|
|
the lock is no longer used to protect the reference count itself.
|
|
|
|
::
|
|
|
|
--- cache.c.refcnt 2003-12-09 15:00:35.000000000 +1100
|
|
+++ cache.c.refcnt-atomic 2003-12-11 15:49:42.000000000 +1100
|
|
@@ -7,7 +7,7 @@
|
|
struct object
|
|
{
|
|
struct list_head list;
|
|
- unsigned int refcnt;
|
|
+ atomic_t refcnt;
|
|
int id;
|
|
char name[32];
|
|
int popularity;
|
|
@@ -18,33 +18,15 @@
|
|
static unsigned int cache_num = 0;
|
|
#define MAX_CACHE_SIZE 10
|
|
|
|
-static void __object_put(struct object *obj)
|
|
-{
|
|
- if (--obj->refcnt == 0)
|
|
- kfree(obj);
|
|
-}
|
|
-
|
|
-static void __object_get(struct object *obj)
|
|
-{
|
|
- obj->refcnt++;
|
|
-}
|
|
-
|
|
void object_put(struct object *obj)
|
|
{
|
|
- unsigned long flags;
|
|
-
|
|
- spin_lock_irqsave(&cache_lock, flags);
|
|
- __object_put(obj);
|
|
- spin_unlock_irqrestore(&cache_lock, flags);
|
|
+ if (atomic_dec_and_test(&obj->refcnt))
|
|
+ kfree(obj);
|
|
}
|
|
|
|
void object_get(struct object *obj)
|
|
{
|
|
- unsigned long flags;
|
|
-
|
|
- spin_lock_irqsave(&cache_lock, flags);
|
|
- __object_get(obj);
|
|
- spin_unlock_irqrestore(&cache_lock, flags);
|
|
+ atomic_inc(&obj->refcnt);
|
|
}
|
|
|
|
/* Must be holding cache_lock */
|
|
@@ -65,7 +47,7 @@
|
|
{
|
|
BUG_ON(!obj);
|
|
list_del(&obj->list);
|
|
- __object_put(obj);
|
|
+ object_put(obj);
|
|
cache_num--;
|
|
}
|
|
|
|
@@ -94,7 +76,7 @@
|
|
strlcpy(obj->name, name, sizeof(obj->name));
|
|
obj->id = id;
|
|
obj->popularity = 0;
|
|
- obj->refcnt = 1; /* The cache holds a reference */
|
|
+ atomic_set(&obj->refcnt, 1); /* The cache holds a reference */
|
|
|
|
spin_lock_irqsave(&cache_lock, flags);
|
|
__cache_add(obj);
|
|
@@ -119,7 +101,7 @@
|
|
spin_lock_irqsave(&cache_lock, flags);
|
|
obj = __cache_find(id);
|
|
if (obj)
|
|
- __object_get(obj);
|
|
+ object_get(obj);
|
|
spin_unlock_irqrestore(&cache_lock, flags);
|
|
return obj;
|
|
}
|
|
|
|
Protecting The Objects Themselves
|
|
---------------------------------
|
|
|
|
In these examples, we assumed that the objects (except the reference
|
|
counts) never changed once they are created. If we wanted to allow the
|
|
name to change, there are three possibilities:
|
|
|
|
- You can make ``cache_lock`` non-static, and tell people to grab that
|
|
lock before changing the name in any object.
|
|
|
|
- You can provide a :c:func:`cache_obj_rename()` which grabs this
|
|
lock and changes the name for the caller, and tell everyone to use
|
|
that function.
|
|
|
|
- You can make the ``cache_lock`` protect only the cache itself, and
|
|
use another lock to protect the name.
|
|
|
|
Theoretically, you can make the locks as fine-grained as one lock for
|
|
every field, for every object. In practice, the most common variants
|
|
are:
|
|
|
|
- One lock which protects the infrastructure (the ``cache`` list in
|
|
this example) and all the objects. This is what we have done so far.
|
|
|
|
- One lock which protects the infrastructure (including the list
|
|
pointers inside the objects), and one lock inside the object which
|
|
protects the rest of that object.
|
|
|
|
- Multiple locks to protect the infrastructure (eg. one lock per hash
|
|
chain), possibly with a separate per-object lock.
|
|
|
|
Here is the "lock-per-object" implementation:
|
|
|
|
::
|
|
|
|
--- cache.c.refcnt-atomic 2003-12-11 15:50:54.000000000 +1100
|
|
+++ cache.c.perobjectlock 2003-12-11 17:15:03.000000000 +1100
|
|
@@ -6,11 +6,17 @@
|
|
|
|
struct object
|
|
{
|
|
+ /* These two protected by cache_lock. */
|
|
struct list_head list;
|
|
+ int popularity;
|
|
+
|
|
atomic_t refcnt;
|
|
+
|
|
+ /* Doesn't change once created. */
|
|
int id;
|
|
+
|
|
+ spinlock_t lock; /* Protects the name */
|
|
char name[32];
|
|
- int popularity;
|
|
};
|
|
|
|
static DEFINE_SPINLOCK(cache_lock);
|
|
@@ -77,6 +84,7 @@
|
|
obj->id = id;
|
|
obj->popularity = 0;
|
|
atomic_set(&obj->refcnt, 1); /* The cache holds a reference */
|
|
+ spin_lock_init(&obj->lock);
|
|
|
|
spin_lock_irqsave(&cache_lock, flags);
|
|
__cache_add(obj);
|
|
|
|
Note that I decide that the popularity count should be protected by the
|
|
``cache_lock`` rather than the per-object lock: this is because it (like
|
|
the :c:type:`struct list_head <list_head>` inside the object)
|
|
is logically part of the infrastructure. This way, I don't need to grab
|
|
the lock of every object in :c:func:`__cache_add()` when seeking
|
|
the least popular.
|
|
|
|
I also decided that the id member is unchangeable, so I don't need to
|
|
grab each object lock in :c:func:`__cache_find()` to examine the
|
|
id: the object lock is only used by a caller who wants to read or write
|
|
the name field.
|
|
|
|
Note also that I added a comment describing what data was protected by
|
|
which locks. This is extremely important, as it describes the runtime
|
|
behavior of the code, and can be hard to gain from just reading. And as
|
|
Alan Cox says, “Lock data, not code”.
|
|
|
|
Common Problems
|
|
===============
|
|
|
|
Deadlock: Simple and Advanced
|
|
-----------------------------
|
|
|
|
There is a coding bug where a piece of code tries to grab a spinlock
|
|
twice: it will spin forever, waiting for the lock to be released
|
|
(spinlocks, rwlocks and mutexes are not recursive in Linux). This is
|
|
trivial to diagnose: not a
|
|
stay-up-five-nights-talk-to-fluffy-code-bunnies kind of problem.
|
|
|
|
For a slightly more complex case, imagine you have a region shared by a
|
|
softirq and user context. If you use a :c:func:`spin_lock()` call
|
|
to protect it, it is possible that the user context will be interrupted
|
|
by the softirq while it holds the lock, and the softirq will then spin
|
|
forever trying to get the same lock.
|
|
|
|
Both of these are called deadlock, and as shown above, it can occur even
|
|
with a single CPU (although not on UP compiles, since spinlocks vanish
|
|
on kernel compiles with ``CONFIG_SMP``\ =n. You'll still get data
|
|
corruption in the second example).
|
|
|
|
This complete lockup is easy to diagnose: on SMP boxes the watchdog
|
|
timer or compiling with ``DEBUG_SPINLOCK`` set
|
|
(``include/linux/spinlock.h``) will show this up immediately when it
|
|
happens.
|
|
|
|
A more complex problem is the so-called 'deadly embrace', involving two
|
|
or more locks. Say you have a hash table: each entry in the table is a
|
|
spinlock, and a chain of hashed objects. Inside a softirq handler, you
|
|
sometimes want to alter an object from one place in the hash to another:
|
|
you grab the spinlock of the old hash chain and the spinlock of the new
|
|
hash chain, and delete the object from the old one, and insert it in the
|
|
new one.
|
|
|
|
There are two problems here. First, if your code ever tries to move the
|
|
object to the same chain, it will deadlock with itself as it tries to
|
|
lock it twice. Secondly, if the same softirq on another CPU is trying to
|
|
move another object in the reverse direction, the following could
|
|
happen:
|
|
|
|
+-----------------------+-----------------------+
|
|
| CPU 1 | CPU 2 |
|
|
+=======================+=======================+
|
|
| Grab lock A -> OK | Grab lock B -> OK |
|
|
+-----------------------+-----------------------+
|
|
| Grab lock B -> spin | Grab lock A -> spin |
|
|
+-----------------------+-----------------------+
|
|
|
|
Table: Consequences
|
|
|
|
The two CPUs will spin forever, waiting for the other to give up their
|
|
lock. It will look, smell, and feel like a crash.
|
|
|
|
Preventing Deadlock
|
|
-------------------
|
|
|
|
Textbooks will tell you that if you always lock in the same order, you
|
|
will never get this kind of deadlock. Practice will tell you that this
|
|
approach doesn't scale: when I create a new lock, I don't understand
|
|
enough of the kernel to figure out where in the 5000 lock hierarchy it
|
|
will fit.
|
|
|
|
The best locks are encapsulated: they never get exposed in headers, and
|
|
are never held around calls to non-trivial functions outside the same
|
|
file. You can read through this code and see that it will never
|
|
deadlock, because it never tries to grab another lock while it has that
|
|
one. People using your code don't even need to know you are using a
|
|
lock.
|
|
|
|
A classic problem here is when you provide callbacks or hooks: if you
|
|
call these with the lock held, you risk simple deadlock, or a deadly
|
|
embrace (who knows what the callback will do?). Remember, the other
|
|
programmers are out to get you, so don't do this.
|
|
|
|
Overzealous Prevention Of Deadlocks
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Deadlocks are problematic, but not as bad as data corruption. Code which
|
|
grabs a read lock, searches a list, fails to find what it wants, drops
|
|
the read lock, grabs a write lock and inserts the object has a race
|
|
condition.
|
|
|
|
If you don't see why, please stay the fuck away from my code.
|
|
|
|
Racing Timers: A Kernel Pastime
|
|
-------------------------------
|
|
|
|
Timers can produce their own special problems with races. Consider a
|
|
collection of objects (list, hash, etc) where each object has a timer
|
|
which is due to destroy it.
|
|
|
|
If you want to destroy the entire collection (say on module removal),
|
|
you might do the following::
|
|
|
|
/* THIS CODE BAD BAD BAD BAD: IF IT WAS ANY WORSE IT WOULD USE
|
|
HUNGARIAN NOTATION */
|
|
spin_lock_bh(&list_lock);
|
|
|
|
while (list) {
|
|
struct foo *next = list->next;
|
|
del_timer(&list->timer);
|
|
kfree(list);
|
|
list = next;
|
|
}
|
|
|
|
spin_unlock_bh(&list_lock);
|
|
|
|
|
|
Sooner or later, this will crash on SMP, because a timer can have just
|
|
gone off before the :c:func:`spin_lock_bh()`, and it will only get
|
|
the lock after we :c:func:`spin_unlock_bh()`, and then try to free
|
|
the element (which has already been freed!).
|
|
|
|
This can be avoided by checking the result of
|
|
:c:func:`del_timer()`: if it returns 1, the timer has been deleted.
|
|
If 0, it means (in this case) that it is currently running, so we can
|
|
do::
|
|
|
|
retry:
|
|
spin_lock_bh(&list_lock);
|
|
|
|
while (list) {
|
|
struct foo *next = list->next;
|
|
if (!del_timer(&list->timer)) {
|
|
/* Give timer a chance to delete this */
|
|
spin_unlock_bh(&list_lock);
|
|
goto retry;
|
|
}
|
|
kfree(list);
|
|
list = next;
|
|
}
|
|
|
|
spin_unlock_bh(&list_lock);
|
|
|
|
|
|
Another common problem is deleting timers which restart themselves (by
|
|
calling :c:func:`add_timer()` at the end of their timer function).
|
|
Because this is a fairly common case which is prone to races, you should
|
|
use :c:func:`del_timer_sync()` (``include/linux/timer.h``) to
|
|
handle this case. It returns the number of times the timer had to be
|
|
deleted before we finally stopped it from adding itself back in.
|
|
|
|
Locking Speed
|
|
=============
|
|
|
|
There are three main things to worry about when considering speed of
|
|
some code which does locking. First is concurrency: how many things are
|
|
going to be waiting while someone else is holding a lock. Second is the
|
|
time taken to actually acquire and release an uncontended lock. Third is
|
|
using fewer, or smarter locks. I'm assuming that the lock is used fairly
|
|
often: otherwise, you wouldn't be concerned about efficiency.
|
|
|
|
Concurrency depends on how long the lock is usually held: you should
|
|
hold the lock for as long as needed, but no longer. In the cache
|
|
example, we always create the object without the lock held, and then
|
|
grab the lock only when we are ready to insert it in the list.
|
|
|
|
Acquisition times depend on how much damage the lock operations do to
|
|
the pipeline (pipeline stalls) and how likely it is that this CPU was
|
|
the last one to grab the lock (ie. is the lock cache-hot for this CPU):
|
|
on a machine with more CPUs, this likelihood drops fast. Consider a
|
|
700MHz Intel Pentium III: an instruction takes about 0.7ns, an atomic
|
|
increment takes about 58ns, a lock which is cache-hot on this CPU takes
|
|
160ns, and a cacheline transfer from another CPU takes an additional 170
|
|
to 360ns. (These figures from Paul McKenney's `Linux Journal RCU
|
|
article <http://www.linuxjournal.com/article.php?sid=6993>`__).
|
|
|
|
These two aims conflict: holding a lock for a short time might be done
|
|
by splitting locks into parts (such as in our final per-object-lock
|
|
example), but this increases the number of lock acquisitions, and the
|
|
results are often slower than having a single lock. This is another
|
|
reason to advocate locking simplicity.
|
|
|
|
The third concern is addressed below: there are some methods to reduce
|
|
the amount of locking which needs to be done.
|
|
|
|
Read/Write Lock Variants
|
|
------------------------
|
|
|
|
Both spinlocks and mutexes have read/write variants: ``rwlock_t`` and
|
|
:c:type:`struct rw_semaphore <rw_semaphore>`. These divide
|
|
users into two classes: the readers and the writers. If you are only
|
|
reading the data, you can get a read lock, but to write to the data you
|
|
need the write lock. Many people can hold a read lock, but a writer must
|
|
be sole holder.
|
|
|
|
If your code divides neatly along reader/writer lines (as our cache code
|
|
does), and the lock is held by readers for significant lengths of time,
|
|
using these locks can help. They are slightly slower than the normal
|
|
locks though, so in practice ``rwlock_t`` is not usually worthwhile.
|
|
|
|
Avoiding Locks: Read Copy Update
|
|
--------------------------------
|
|
|
|
There is a special method of read/write locking called Read Copy Update.
|
|
Using RCU, the readers can avoid taking a lock altogether: as we expect
|
|
our cache to be read more often than updated (otherwise the cache is a
|
|
waste of time), it is a candidate for this optimization.
|
|
|
|
How do we get rid of read locks? Getting rid of read locks means that
|
|
writers may be changing the list underneath the readers. That is
|
|
actually quite simple: we can read a linked list while an element is
|
|
being added if the writer adds the element very carefully. For example,
|
|
adding ``new`` to a single linked list called ``list``::
|
|
|
|
new->next = list->next;
|
|
wmb();
|
|
list->next = new;
|
|
|
|
|
|
The :c:func:`wmb()` is a write memory barrier. It ensures that the
|
|
first operation (setting the new element's ``next`` pointer) is complete
|
|
and will be seen by all CPUs, before the second operation is (putting
|
|
the new element into the list). This is important, since modern
|
|
compilers and modern CPUs can both reorder instructions unless told
|
|
otherwise: we want a reader to either not see the new element at all, or
|
|
see the new element with the ``next`` pointer correctly pointing at the
|
|
rest of the list.
|
|
|
|
Fortunately, there is a function to do this for standard
|
|
:c:type:`struct list_head <list_head>` lists:
|
|
:c:func:`list_add_rcu()` (``include/linux/list.h``).
|
|
|
|
Removing an element from the list is even simpler: we replace the
|
|
pointer to the old element with a pointer to its successor, and readers
|
|
will either see it, or skip over it.
|
|
|
|
::
|
|
|
|
list->next = old->next;
|
|
|
|
|
|
There is :c:func:`list_del_rcu()` (``include/linux/list.h``) which
|
|
does this (the normal version poisons the old object, which we don't
|
|
want).
|
|
|
|
The reader must also be careful: some CPUs can look through the ``next``
|
|
pointer to start reading the contents of the next element early, but
|
|
don't realize that the pre-fetched contents is wrong when the ``next``
|
|
pointer changes underneath them. Once again, there is a
|
|
:c:func:`list_for_each_entry_rcu()` (``include/linux/list.h``)
|
|
to help you. Of course, writers can just use
|
|
:c:func:`list_for_each_entry()`, since there cannot be two
|
|
simultaneous writers.
|
|
|
|
Our final dilemma is this: when can we actually destroy the removed
|
|
element? Remember, a reader might be stepping through this element in
|
|
the list right now: if we free this element and the ``next`` pointer
|
|
changes, the reader will jump off into garbage and crash. We need to
|
|
wait until we know that all the readers who were traversing the list
|
|
when we deleted the element are finished. We use
|
|
:c:func:`call_rcu()` to register a callback which will actually
|
|
destroy the object once all pre-existing readers are finished.
|
|
Alternatively, :c:func:`synchronize_rcu()` may be used to block
|
|
until all pre-existing are finished.
|
|
|
|
But how does Read Copy Update know when the readers are finished? The
|
|
method is this: firstly, the readers always traverse the list inside
|
|
:c:func:`rcu_read_lock()`/:c:func:`rcu_read_unlock()` pairs:
|
|
these simply disable preemption so the reader won't go to sleep while
|
|
reading the list.
|
|
|
|
RCU then waits until every other CPU has slept at least once: since
|
|
readers cannot sleep, we know that any readers which were traversing the
|
|
list during the deletion are finished, and the callback is triggered.
|
|
The real Read Copy Update code is a little more optimized than this, but
|
|
this is the fundamental idea.
|
|
|
|
::
|
|
|
|
--- cache.c.perobjectlock 2003-12-11 17:15:03.000000000 +1100
|
|
+++ cache.c.rcupdate 2003-12-11 17:55:14.000000000 +1100
|
|
@@ -1,15 +1,18 @@
|
|
#include <linux/list.h>
|
|
#include <linux/slab.h>
|
|
#include <linux/string.h>
|
|
+#include <linux/rcupdate.h>
|
|
#include <linux/mutex.h>
|
|
#include <asm/errno.h>
|
|
|
|
struct object
|
|
{
|
|
- /* These two protected by cache_lock. */
|
|
+ /* This is protected by RCU */
|
|
struct list_head list;
|
|
int popularity;
|
|
|
|
+ struct rcu_head rcu;
|
|
+
|
|
atomic_t refcnt;
|
|
|
|
/* Doesn't change once created. */
|
|
@@ -40,7 +43,7 @@
|
|
{
|
|
struct object *i;
|
|
|
|
- list_for_each_entry(i, &cache, list) {
|
|
+ list_for_each_entry_rcu(i, &cache, list) {
|
|
if (i->id == id) {
|
|
i->popularity++;
|
|
return i;
|
|
@@ -49,19 +52,25 @@
|
|
return NULL;
|
|
}
|
|
|
|
+/* Final discard done once we know no readers are looking. */
|
|
+static void cache_delete_rcu(void *arg)
|
|
+{
|
|
+ object_put(arg);
|
|
+}
|
|
+
|
|
/* Must be holding cache_lock */
|
|
static void __cache_delete(struct object *obj)
|
|
{
|
|
BUG_ON(!obj);
|
|
- list_del(&obj->list);
|
|
- object_put(obj);
|
|
+ list_del_rcu(&obj->list);
|
|
cache_num--;
|
|
+ call_rcu(&obj->rcu, cache_delete_rcu);
|
|
}
|
|
|
|
/* Must be holding cache_lock */
|
|
static void __cache_add(struct object *obj)
|
|
{
|
|
- list_add(&obj->list, &cache);
|
|
+ list_add_rcu(&obj->list, &cache);
|
|
if (++cache_num > MAX_CACHE_SIZE) {
|
|
struct object *i, *outcast = NULL;
|
|
list_for_each_entry(i, &cache, list) {
|
|
@@ -104,12 +114,11 @@
|
|
struct object *cache_find(int id)
|
|
{
|
|
struct object *obj;
|
|
- unsigned long flags;
|
|
|
|
- spin_lock_irqsave(&cache_lock, flags);
|
|
+ rcu_read_lock();
|
|
obj = __cache_find(id);
|
|
if (obj)
|
|
object_get(obj);
|
|
- spin_unlock_irqrestore(&cache_lock, flags);
|
|
+ rcu_read_unlock();
|
|
return obj;
|
|
}
|
|
|
|
Note that the reader will alter the popularity member in
|
|
:c:func:`__cache_find()`, and now it doesn't hold a lock. One
|
|
solution would be to make it an ``atomic_t``, but for this usage, we
|
|
don't really care about races: an approximate result is good enough, so
|
|
I didn't change it.
|
|
|
|
The result is that :c:func:`cache_find()` requires no
|
|
synchronization with any other functions, so is almost as fast on SMP as
|
|
it would be on UP.
|
|
|
|
There is a further optimization possible here: remember our original
|
|
cache code, where there were no reference counts and the caller simply
|
|
held the lock whenever using the object? This is still possible: if you
|
|
hold the lock, no one can delete the object, so you don't need to get
|
|
and put the reference count.
|
|
|
|
Now, because the 'read lock' in RCU is simply disabling preemption, a
|
|
caller which always has preemption disabled between calling
|
|
:c:func:`cache_find()` and :c:func:`object_put()` does not
|
|
need to actually get and put the reference count: we could expose
|
|
:c:func:`__cache_find()` by making it non-static, and such
|
|
callers could simply call that.
|
|
|
|
The benefit here is that the reference count is not written to: the
|
|
object is not altered in any way, which is much faster on SMP machines
|
|
due to caching.
|
|
|
|
Per-CPU Data
|
|
------------
|
|
|
|
Another technique for avoiding locking which is used fairly widely is to
|
|
duplicate information for each CPU. For example, if you wanted to keep a
|
|
count of a common condition, you could use a spin lock and a single
|
|
counter. Nice and simple.
|
|
|
|
If that was too slow (it's usually not, but if you've got a really big
|
|
machine to test on and can show that it is), you could instead use a
|
|
counter for each CPU, then none of them need an exclusive lock. See
|
|
:c:func:`DEFINE_PER_CPU()`, :c:func:`get_cpu_var()` and
|
|
:c:func:`put_cpu_var()` (``include/linux/percpu.h``).
|
|
|
|
Of particular use for simple per-cpu counters is the ``local_t`` type,
|
|
and the :c:func:`cpu_local_inc()` and related functions, which are
|
|
more efficient than simple code on some architectures
|
|
(``include/asm/local.h``).
|
|
|
|
Note that there is no simple, reliable way of getting an exact value of
|
|
such a counter, without introducing more locks. This is not a problem
|
|
for some uses.
|
|
|
|
Data Which Mostly Used By An IRQ Handler
|
|
----------------------------------------
|
|
|
|
If data is always accessed from within the same IRQ handler, you don't
|
|
need a lock at all: the kernel already guarantees that the irq handler
|
|
will not run simultaneously on multiple CPUs.
|
|
|
|
Manfred Spraul points out that you can still do this, even if the data
|
|
is very occasionally accessed in user context or softirqs/tasklets. The
|
|
irq handler doesn't use a lock, and all other accesses are done as so::
|
|
|
|
spin_lock(&lock);
|
|
disable_irq(irq);
|
|
...
|
|
enable_irq(irq);
|
|
spin_unlock(&lock);
|
|
|
|
The :c:func:`disable_irq()` prevents the irq handler from running
|
|
(and waits for it to finish if it's currently running on other CPUs).
|
|
The spinlock prevents any other accesses happening at the same time.
|
|
Naturally, this is slower than just a :c:func:`spin_lock_irq()`
|
|
call, so it only makes sense if this type of access happens extremely
|
|
rarely.
|
|
|
|
What Functions Are Safe To Call From Interrupts?
|
|
================================================
|
|
|
|
Many functions in the kernel sleep (ie. call schedule()) directly or
|
|
indirectly: you can never call them while holding a spinlock, or with
|
|
preemption disabled. This also means you need to be in user context:
|
|
calling them from an interrupt is illegal.
|
|
|
|
Some Functions Which Sleep
|
|
--------------------------
|
|
|
|
The most common ones are listed below, but you usually have to read the
|
|
code to find out if other calls are safe. If everyone else who calls it
|
|
can sleep, you probably need to be able to sleep, too. In particular,
|
|
registration and deregistration functions usually expect to be called
|
|
from user context, and can sleep.
|
|
|
|
- Accesses to userspace:
|
|
|
|
- :c:func:`copy_from_user()`
|
|
|
|
- :c:func:`copy_to_user()`
|
|
|
|
- :c:func:`get_user()`
|
|
|
|
- :c:func:`put_user()`
|
|
|
|
- :c:func:`kmalloc(GFP_KERNEL) <kmalloc>`
|
|
|
|
- :c:func:`mutex_lock_interruptible()` and
|
|
:c:func:`mutex_lock()`
|
|
|
|
There is a :c:func:`mutex_trylock()` which does not sleep.
|
|
Still, it must not be used inside interrupt context since its
|
|
implementation is not safe for that. :c:func:`mutex_unlock()`
|
|
will also never sleep. It cannot be used in interrupt context either
|
|
since a mutex must be released by the same task that acquired it.
|
|
|
|
Some Functions Which Don't Sleep
|
|
--------------------------------
|
|
|
|
Some functions are safe to call from any context, or holding almost any
|
|
lock.
|
|
|
|
- :c:func:`printk()`
|
|
|
|
- :c:func:`kfree()`
|
|
|
|
- :c:func:`add_timer()` and :c:func:`del_timer()`
|
|
|
|
Mutex API reference
|
|
===================
|
|
|
|
.. kernel-doc:: include/linux/mutex.h
|
|
:internal:
|
|
|
|
.. kernel-doc:: kernel/locking/mutex.c
|
|
:export:
|
|
|
|
Futex API reference
|
|
===================
|
|
|
|
.. kernel-doc:: kernel/futex.c
|
|
:internal:
|
|
|
|
Further reading
|
|
===============
|
|
|
|
- ``Documentation/locking/spinlocks.txt``: Linus Torvalds' spinlocking
|
|
tutorial in the kernel sources.
|
|
|
|
- Unix Systems for Modern Architectures: Symmetric Multiprocessing and
|
|
Caching for Kernel Programmers:
|
|
|
|
Curt Schimmel's very good introduction to kernel level locking (not
|
|
written for Linux, but nearly everything applies). The book is
|
|
expensive, but really worth every penny to understand SMP locking.
|
|
[ISBN: 0201633388]
|
|
|
|
Thanks
|
|
======
|
|
|
|
Thanks to Telsa Gwynne for DocBooking, neatening and adding style.
|
|
|
|
Thanks to Martin Pool, Philipp Rumpf, Stephen Rothwell, Paul Mackerras,
|
|
Ruedi Aschwanden, Alan Cox, Manfred Spraul, Tim Waugh, Pete Zaitcev,
|
|
James Morris, Robert Love, Paul McKenney, John Ashby for proofreading,
|
|
correcting, flaming, commenting.
|
|
|
|
Thanks to the cabal for having no influence on this document.
|
|
|
|
Glossary
|
|
========
|
|
|
|
preemption
|
|
Prior to 2.5, or when ``CONFIG_PREEMPT`` is unset, processes in user
|
|
context inside the kernel would not preempt each other (ie. you had that
|
|
CPU until you gave it up, except for interrupts). With the addition of
|
|
``CONFIG_PREEMPT`` in 2.5.4, this changed: when in user context, higher
|
|
priority tasks can "cut in": spinlocks were changed to disable
|
|
preemption, even on UP.
|
|
|
|
bh
|
|
Bottom Half: for historical reasons, functions with '_bh' in them often
|
|
now refer to any software interrupt, e.g. :c:func:`spin_lock_bh()`
|
|
blocks any software interrupt on the current CPU. Bottom halves are
|
|
deprecated, and will eventually be replaced by tasklets. Only one bottom
|
|
half will be running at any time.
|
|
|
|
Hardware Interrupt / Hardware IRQ
|
|
Hardware interrupt request. :c:func:`in_irq()` returns true in a
|
|
hardware interrupt handler.
|
|
|
|
Interrupt Context
|
|
Not user context: processing a hardware irq or software irq. Indicated
|
|
by the :c:func:`in_interrupt()` macro returning true.
|
|
|
|
SMP
|
|
Symmetric Multi-Processor: kernels compiled for multiple-CPU machines.
|
|
(``CONFIG_SMP=y``).
|
|
|
|
Software Interrupt / softirq
|
|
Software interrupt handler. :c:func:`in_irq()` returns false;
|
|
:c:func:`in_softirq()` returns true. Tasklets and softirqs both
|
|
fall into the category of 'software interrupts'.
|
|
|
|
Strictly speaking a softirq is one of up to 32 enumerated software
|
|
interrupts which can run on multiple CPUs at once. Sometimes used to
|
|
refer to tasklets as well (ie. all software interrupts).
|
|
|
|
tasklet
|
|
A dynamically-registrable software interrupt, which is guaranteed to
|
|
only run on one CPU at a time.
|
|
|
|
timer
|
|
A dynamically-registrable software interrupt, which is run at (or close
|
|
to) a given time. When running, it is just like a tasklet (in fact, they
|
|
are called from the ``TIMER_SOFTIRQ``).
|
|
|
|
UP
|
|
Uni-Processor: Non-SMP. (``CONFIG_SMP=n``).
|
|
|
|
User Context
|
|
The kernel executing on behalf of a particular process (ie. a system
|
|
call or trap) or kernel thread. You can tell which process with the
|
|
``current`` macro.) Not to be confused with userspace. Can be
|
|
interrupted by software or hardware interrupts.
|
|
|
|
Userspace
|
|
A process executing its own code outside the kernel.
|