mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
synced 2024-08-24 01:41:39 +00:00
![Kent Overstreet](/assets/img/avatar_default.png)
Immutable biovecs are going to require an explicit iterator. To implement immutable bvecs, a later patch is going to add a bi_bvec_done member to this struct; for now, this patch effectively just renames things. Signed-off-by: Kent Overstreet <kmo@daterainc.com> Cc: Jens Axboe <axboe@kernel.dk> Cc: Geert Uytterhoeven <geert@linux-m68k.org> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> Cc: Paul Mackerras <paulus@samba.org> Cc: "Ed L. Cashin" <ecashin@coraid.com> Cc: Nick Piggin <npiggin@kernel.dk> Cc: Lars Ellenberg <drbd-dev@lists.linbit.com> Cc: Jiri Kosina <jkosina@suse.cz> Cc: Matthew Wilcox <willy@linux.intel.com> Cc: Geoff Levand <geoff@infradead.org> Cc: Yehuda Sadeh <yehuda@inktank.com> Cc: Sage Weil <sage@inktank.com> Cc: Alex Elder <elder@inktank.com> Cc: ceph-devel@vger.kernel.org Cc: Joshua Morris <josh.h.morris@us.ibm.com> Cc: Philip Kelleher <pjk1939@linux.vnet.ibm.com> Cc: Rusty Russell <rusty@rustcorp.com.au> Cc: "Michael S. Tsirkin" <mst@redhat.com> Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> Cc: Jeremy Fitzhardinge <jeremy@goop.org> Cc: Neil Brown <neilb@suse.de> Cc: Alasdair Kergon <agk@redhat.com> Cc: Mike Snitzer <snitzer@redhat.com> Cc: dm-devel@redhat.com Cc: Martin Schwidefsky <schwidefsky@de.ibm.com> Cc: Heiko Carstens <heiko.carstens@de.ibm.com> Cc: linux390@de.ibm.com Cc: Boaz Harrosh <bharrosh@panasas.com> Cc: Benny Halevy <bhalevy@tonian.com> Cc: "James E.J. Bottomley" <JBottomley@parallels.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: "Nicholas A. Bellinger" <nab@linux-iscsi.org> Cc: Alexander Viro <viro@zeniv.linux.org.uk> Cc: Chris Mason <chris.mason@fusionio.com> Cc: "Theodore Ts'o" <tytso@mit.edu> Cc: Andreas Dilger <adilger.kernel@dilger.ca> Cc: Jaegeuk Kim <jaegeuk.kim@samsung.com> Cc: Steven Whitehouse <swhiteho@redhat.com> Cc: Dave Kleikamp <shaggy@kernel.org> Cc: Joern Engel <joern@logfs.org> Cc: Prasad Joshi <prasadjoshi.linux@gmail.com> Cc: Trond Myklebust <Trond.Myklebust@netapp.com> Cc: KONISHI Ryusuke <konishi.ryusuke@lab.ntt.co.jp> Cc: Mark Fasheh <mfasheh@suse.com> Cc: Joel Becker <jlbec@evilplan.org> Cc: Ben Myers <bpm@sgi.com> Cc: xfs@oss.sgi.com Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Len Brown <len.brown@intel.com> Cc: Pavel Machek <pavel@ucw.cz> Cc: "Rafael J. Wysocki" <rjw@sisk.pl> Cc: Herton Ronaldo Krzesinski <herton.krzesinski@canonical.com> Cc: Ben Hutchings <ben@decadent.org.uk> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Guo Chao <yan@linux.vnet.ibm.com> Cc: Tejun Heo <tj@kernel.org> Cc: Asai Thambi S P <asamymuthupa@micron.com> Cc: Selvan Mani <smani@micron.com> Cc: Sam Bradshaw <sbradshaw@micron.com> Cc: Wei Yongjun <yongjun_wei@trendmicro.com.cn> Cc: "Roger Pau Monné" <roger.pau@citrix.com> Cc: Jan Beulich <jbeulich@suse.com> Cc: Stefano Stabellini <stefano.stabellini@eu.citrix.com> Cc: Ian Campbell <Ian.Campbell@citrix.com> Cc: Sebastian Ott <sebott@linux.vnet.ibm.com> Cc: Christian Borntraeger <borntraeger@de.ibm.com> Cc: Minchan Kim <minchan@kernel.org> Cc: Jiang Liu <jiang.liu@huawei.com> Cc: Nitin Gupta <ngupta@vflare.org> Cc: Jerome Marchand <jmarchand@redhat.com> Cc: Joe Perches <joe@perches.com> Cc: Peng Tao <tao.peng@emc.com> Cc: Andy Adamson <andros@netapp.com> Cc: fanchaoting <fanchaoting@cn.fujitsu.com> Cc: Jie Liu <jeff.liu@oracle.com> Cc: Sunil Mushran <sunil.mushran@gmail.com> Cc: "Martin K. Petersen" <martin.petersen@oracle.com> Cc: Namjae Jeon <namjae.jeon@samsung.com> Cc: Pankaj Kumar <pankaj.km@samsung.com> Cc: Dan Magenheimer <dan.magenheimer@oracle.com> Cc: Mel Gorman <mgorman@suse.de>6
1198 lines
54 KiB
Text
1198 lines
54 KiB
Text
Notes on the Generic Block Layer Rewrite in Linux 2.5
|
|
=====================================================
|
|
|
|
Notes Written on Jan 15, 2002:
|
|
Jens Axboe <jens.axboe@oracle.com>
|
|
Suparna Bhattacharya <suparna@in.ibm.com>
|
|
|
|
Last Updated May 2, 2002
|
|
September 2003: Updated I/O Scheduler portions
|
|
Nick Piggin <npiggin@kernel.dk>
|
|
|
|
Introduction:
|
|
|
|
These are some notes describing some aspects of the 2.5 block layer in the
|
|
context of the bio rewrite. The idea is to bring out some of the key
|
|
changes and a glimpse of the rationale behind those changes.
|
|
|
|
Please mail corrections & suggestions to suparna@in.ibm.com.
|
|
|
|
Credits:
|
|
---------
|
|
|
|
2.5 bio rewrite:
|
|
Jens Axboe <jens.axboe@oracle.com>
|
|
|
|
Many aspects of the generic block layer redesign were driven by and evolved
|
|
over discussions, prior patches and the collective experience of several
|
|
people. See sections 8 and 9 for a list of some related references.
|
|
|
|
The following people helped with review comments and inputs for this
|
|
document:
|
|
Christoph Hellwig <hch@infradead.org>
|
|
Arjan van de Ven <arjanv@redhat.com>
|
|
Randy Dunlap <rdunlap@xenotime.net>
|
|
Andre Hedrick <andre@linux-ide.org>
|
|
|
|
The following people helped with fixes/contributions to the bio patches
|
|
while it was still work-in-progress:
|
|
David S. Miller <davem@redhat.com>
|
|
|
|
|
|
Description of Contents:
|
|
------------------------
|
|
|
|
1. Scope for tuning of logic to various needs
|
|
1.1 Tuning based on device or low level driver capabilities
|
|
- Per-queue parameters
|
|
- Highmem I/O support
|
|
- I/O scheduler modularization
|
|
1.2 Tuning based on high level requirements/capabilities
|
|
1.2.1 I/O Barriers
|
|
1.2.2 Request Priority/Latency
|
|
1.3 Direct access/bypass to lower layers for diagnostics and special
|
|
device operations
|
|
1.3.1 Pre-built commands
|
|
2. New flexible and generic but minimalist i/o structure or descriptor
|
|
(instead of using buffer heads at the i/o layer)
|
|
2.1 Requirements/Goals addressed
|
|
2.2 The bio struct in detail (multi-page io unit)
|
|
2.3 Changes in the request structure
|
|
3. Using bios
|
|
3.1 Setup/teardown (allocation, splitting)
|
|
3.2 Generic bio helper routines
|
|
3.2.1 Traversing segments and completion units in a request
|
|
3.2.2 Setting up DMA scatterlists
|
|
3.2.3 I/O completion
|
|
3.2.4 Implications for drivers that do not interpret bios (don't handle
|
|
multiple segments)
|
|
3.2.5 Request command tagging
|
|
3.3 I/O submission
|
|
4. The I/O scheduler
|
|
5. Scalability related changes
|
|
5.1 Granular locking: Removal of io_request_lock
|
|
5.2 Prepare for transition to 64 bit sector_t
|
|
6. Other Changes/Implications
|
|
6.1 Partition re-mapping handled by the generic block layer
|
|
7. A few tips on migration of older drivers
|
|
8. A list of prior/related/impacted patches/ideas
|
|
9. Other References/Discussion Threads
|
|
|
|
---------------------------------------------------------------------------
|
|
|
|
Bio Notes
|
|
--------
|
|
|
|
Let us discuss the changes in the context of how some overall goals for the
|
|
block layer are addressed.
|
|
|
|
1. Scope for tuning the generic logic to satisfy various requirements
|
|
|
|
The block layer design supports adaptable abstractions to handle common
|
|
processing with the ability to tune the logic to an appropriate extent
|
|
depending on the nature of the device and the requirements of the caller.
|
|
One of the objectives of the rewrite was to increase the degree of tunability
|
|
and to enable higher level code to utilize underlying device/driver
|
|
capabilities to the maximum extent for better i/o performance. This is
|
|
important especially in the light of ever improving hardware capabilities
|
|
and application/middleware software designed to take advantage of these
|
|
capabilities.
|
|
|
|
1.1 Tuning based on low level device / driver capabilities
|
|
|
|
Sophisticated devices with large built-in caches, intelligent i/o scheduling
|
|
optimizations, high memory DMA support, etc may find some of the
|
|
generic processing an overhead, while for less capable devices the
|
|
generic functionality is essential for performance or correctness reasons.
|
|
Knowledge of some of the capabilities or parameters of the device should be
|
|
used at the generic block layer to take the right decisions on
|
|
behalf of the driver.
|
|
|
|
How is this achieved ?
|
|
|
|
Tuning at a per-queue level:
|
|
|
|
i. Per-queue limits/values exported to the generic layer by the driver
|
|
|
|
Various parameters that the generic i/o scheduler logic uses are set at
|
|
a per-queue level (e.g maximum request size, maximum number of segments in
|
|
a scatter-gather list, hardsect size)
|
|
|
|
Some parameters that were earlier available as global arrays indexed by
|
|
major/minor are now directly associated with the queue. Some of these may
|
|
move into the block device structure in the future. Some characteristics
|
|
have been incorporated into a queue flags field rather than separate fields
|
|
in themselves. There are blk_queue_xxx functions to set the parameters,
|
|
rather than update the fields directly
|
|
|
|
Some new queue property settings:
|
|
|
|
blk_queue_bounce_limit(q, u64 dma_address)
|
|
Enable I/O to highmem pages, dma_address being the
|
|
limit. No highmem default.
|
|
|
|
blk_queue_max_sectors(q, max_sectors)
|
|
Sets two variables that limit the size of the request.
|
|
|
|
- The request queue's max_sectors, which is a soft size in
|
|
units of 512 byte sectors, and could be dynamically varied
|
|
by the core kernel.
|
|
|
|
- The request queue's max_hw_sectors, which is a hard limit
|
|
and reflects the maximum size request a driver can handle
|
|
in units of 512 byte sectors.
|
|
|
|
The default for both max_sectors and max_hw_sectors is
|
|
255. The upper limit of max_sectors is 1024.
|
|
|
|
blk_queue_max_phys_segments(q, max_segments)
|
|
Maximum physical segments you can handle in a request. 128
|
|
default (driver limit). (See 3.2.2)
|
|
|
|
blk_queue_max_hw_segments(q, max_segments)
|
|
Maximum dma segments the hardware can handle in a request. 128
|
|
default (host adapter limit, after dma remapping).
|
|
(See 3.2.2)
|
|
|
|
blk_queue_max_segment_size(q, max_seg_size)
|
|
Maximum size of a clustered segment, 64kB default.
|
|
|
|
blk_queue_hardsect_size(q, hardsect_size)
|
|
Lowest possible sector size that the hardware can operate
|
|
on, 512 bytes default.
|
|
|
|
New queue flags:
|
|
|
|
QUEUE_FLAG_CLUSTER (see 3.2.2)
|
|
QUEUE_FLAG_QUEUED (see 3.2.4)
|
|
|
|
|
|
ii. High-mem i/o capabilities are now considered the default
|
|
|
|
The generic bounce buffer logic, present in 2.4, where the block layer would
|
|
by default copyin/out i/o requests on high-memory buffers to low-memory buffers
|
|
assuming that the driver wouldn't be able to handle it directly, has been
|
|
changed in 2.5. The bounce logic is now applied only for memory ranges
|
|
for which the device cannot handle i/o. A driver can specify this by
|
|
setting the queue bounce limit for the request queue for the device
|
|
(blk_queue_bounce_limit()). This avoids the inefficiencies of the copyin/out
|
|
where a device is capable of handling high memory i/o.
|
|
|
|
In order to enable high-memory i/o where the device is capable of supporting
|
|
it, the pci dma mapping routines and associated data structures have now been
|
|
modified to accomplish a direct page -> bus translation, without requiring
|
|
a virtual address mapping (unlike the earlier scheme of virtual address
|
|
-> bus translation). So this works uniformly for high-memory pages (which
|
|
do not have a corresponding kernel virtual address space mapping) and
|
|
low-memory pages.
|
|
|
|
Note: Please refer to Documentation/DMA-API-HOWTO.txt for a discussion
|
|
on PCI high mem DMA aspects and mapping of scatter gather lists, and support
|
|
for 64 bit PCI.
|
|
|
|
Special handling is required only for cases where i/o needs to happen on
|
|
pages at physical memory addresses beyond what the device can support. In these
|
|
cases, a bounce bio representing a buffer from the supported memory range
|
|
is used for performing the i/o with copyin/copyout as needed depending on
|
|
the type of the operation. For example, in case of a read operation, the
|
|
data read has to be copied to the original buffer on i/o completion, so a
|
|
callback routine is set up to do this, while for write, the data is copied
|
|
from the original buffer to the bounce buffer prior to issuing the
|
|
operation. Since an original buffer may be in a high memory area that's not
|
|
mapped in kernel virtual addr, a kmap operation may be required for
|
|
performing the copy, and special care may be needed in the completion path
|
|
as it may not be in irq context. Special care is also required (by way of
|
|
GFP flags) when allocating bounce buffers, to avoid certain highmem
|
|
deadlock possibilities.
|
|
|
|
It is also possible that a bounce buffer may be allocated from high-memory
|
|
area that's not mapped in kernel virtual addr, but within the range that the
|
|
device can use directly; so the bounce page may need to be kmapped during
|
|
copy operations. [Note: This does not hold in the current implementation,
|
|
though]
|
|
|
|
There are some situations when pages from high memory may need to
|
|
be kmapped, even if bounce buffers are not necessary. For example a device
|
|
may need to abort DMA operations and revert to PIO for the transfer, in
|
|
which case a virtual mapping of the page is required. For SCSI it is also
|
|
done in some scenarios where the low level driver cannot be trusted to
|
|
handle a single sg entry correctly. The driver is expected to perform the
|
|
kmaps as needed on such occasions using the __bio_kmap_atomic and bio_kmap_irq
|
|
routines as appropriate. A driver could also use the blk_queue_bounce()
|
|
routine on its own to bounce highmem i/o to low memory for specific requests
|
|
if so desired.
|
|
|
|
iii. The i/o scheduler algorithm itself can be replaced/set as appropriate
|
|
|
|
As in 2.4, it is possible to plugin a brand new i/o scheduler for a particular
|
|
queue or pick from (copy) existing generic schedulers and replace/override
|
|
certain portions of it. The 2.5 rewrite provides improved modularization
|
|
of the i/o scheduler. There are more pluggable callbacks, e.g for init,
|
|
add request, extract request, which makes it possible to abstract specific
|
|
i/o scheduling algorithm aspects and details outside of the generic loop.
|
|
It also makes it possible to completely hide the implementation details of
|
|
the i/o scheduler from block drivers.
|
|
|
|
I/O scheduler wrappers are to be used instead of accessing the queue directly.
|
|
See section 4. The I/O scheduler for details.
|
|
|
|
1.2 Tuning Based on High level code capabilities
|
|
|
|
i. Application capabilities for raw i/o
|
|
|
|
This comes from some of the high-performance database/middleware
|
|
requirements where an application prefers to make its own i/o scheduling
|
|
decisions based on an understanding of the access patterns and i/o
|
|
characteristics
|
|
|
|
ii. High performance filesystems or other higher level kernel code's
|
|
capabilities
|
|
|
|
Kernel components like filesystems could also take their own i/o scheduling
|
|
decisions for optimizing performance. Journalling filesystems may need
|
|
some control over i/o ordering.
|
|
|
|
What kind of support exists at the generic block layer for this ?
|
|
|
|
The flags and rw fields in the bio structure can be used for some tuning
|
|
from above e.g indicating that an i/o is just a readahead request, or for
|
|
marking barrier requests (discussed next), or priority settings (currently
|
|
unused). As far as user applications are concerned they would need an
|
|
additional mechanism either via open flags or ioctls, or some other upper
|
|
level mechanism to communicate such settings to block.
|
|
|
|
1.2.1 I/O Barriers
|
|
|
|
There is a way to enforce strict ordering for i/os through barriers.
|
|
All requests before a barrier point must be serviced before the barrier
|
|
request and any other requests arriving after the barrier will not be
|
|
serviced until after the barrier has completed. This is useful for higher
|
|
level control on write ordering, e.g flushing a log of committed updates
|
|
to disk before the corresponding updates themselves.
|
|
|
|
A flag in the bio structure, BIO_BARRIER is used to identify a barrier i/o.
|
|
The generic i/o scheduler would make sure that it places the barrier request and
|
|
all other requests coming after it after all the previous requests in the
|
|
queue. Barriers may be implemented in different ways depending on the
|
|
driver. For more details regarding I/O barriers, please read barrier.txt
|
|
in this directory.
|
|
|
|
1.2.2 Request Priority/Latency
|
|
|
|
Todo/Under discussion:
|
|
Arjan's proposed request priority scheme allows higher levels some broad
|
|
control (high/med/low) over the priority of an i/o request vs other pending
|
|
requests in the queue. For example it allows reads for bringing in an
|
|
executable page on demand to be given a higher priority over pending write
|
|
requests which haven't aged too much on the queue. Potentially this priority
|
|
could even be exposed to applications in some manner, providing higher level
|
|
tunability. Time based aging avoids starvation of lower priority
|
|
requests. Some bits in the bi_rw flags field in the bio structure are
|
|
intended to be used for this priority information.
|
|
|
|
|
|
1.3 Direct Access to Low level Device/Driver Capabilities (Bypass mode)
|
|
(e.g Diagnostics, Systems Management)
|
|
|
|
There are situations where high-level code needs to have direct access to
|
|
the low level device capabilities or requires the ability to issue commands
|
|
to the device bypassing some of the intermediate i/o layers.
|
|
These could, for example, be special control commands issued through ioctl
|
|
interfaces, or could be raw read/write commands that stress the drive's
|
|
capabilities for certain kinds of fitness tests. Having direct interfaces at
|
|
multiple levels without having to pass through upper layers makes
|
|
it possible to perform bottom up validation of the i/o path, layer by
|
|
layer, starting from the media.
|
|
|
|
The normal i/o submission interfaces, e.g submit_bio, could be bypassed
|
|
for specially crafted requests which such ioctl or diagnostics
|
|
interfaces would typically use, and the elevator add_request routine
|
|
can instead be used to directly insert such requests in the queue or preferably
|
|
the blk_do_rq routine can be used to place the request on the queue and
|
|
wait for completion. Alternatively, sometimes the caller might just
|
|
invoke a lower level driver specific interface with the request as a
|
|
parameter.
|
|
|
|
If the request is a means for passing on special information associated with
|
|
the command, then such information is associated with the request->special
|
|
field (rather than misuse the request->buffer field which is meant for the
|
|
request data buffer's virtual mapping).
|
|
|
|
For passing request data, the caller must build up a bio descriptor
|
|
representing the concerned memory buffer if the underlying driver interprets
|
|
bio segments or uses the block layer end*request* functions for i/o
|
|
completion. Alternatively one could directly use the request->buffer field to
|
|
specify the virtual address of the buffer, if the driver expects buffer
|
|
addresses passed in this way and ignores bio entries for the request type
|
|
involved. In the latter case, the driver would modify and manage the
|
|
request->buffer, request->sector and request->nr_sectors or
|
|
request->current_nr_sectors fields itself rather than using the block layer
|
|
end_request or end_that_request_first completion interfaces.
|
|
(See 2.3 or Documentation/block/request.txt for a brief explanation of
|
|
the request structure fields)
|
|
|
|
[TBD: end_that_request_last should be usable even in this case;
|
|
Perhaps an end_that_direct_request_first routine could be implemented to make
|
|
handling direct requests easier for such drivers; Also for drivers that
|
|
expect bios, a helper function could be provided for setting up a bio
|
|
corresponding to a data buffer]
|
|
|
|
<JENS: I dont understand the above, why is end_that_request_first() not
|
|
usable? Or _last for that matter. I must be missing something>
|
|
<SUP: What I meant here was that if the request doesn't have a bio, then
|
|
end_that_request_first doesn't modify nr_sectors or current_nr_sectors,
|
|
and hence can't be used for advancing request state settings on the
|
|
completion of partial transfers. The driver has to modify these fields
|
|
directly by hand.
|
|
This is because end_that_request_first only iterates over the bio list,
|
|
and always returns 0 if there are none associated with the request.
|
|
_last works OK in this case, and is not a problem, as I mentioned earlier
|
|
>
|
|
|
|
1.3.1 Pre-built Commands
|
|
|
|
A request can be created with a pre-built custom command to be sent directly
|
|
to the device. The cmd block in the request structure has room for filling
|
|
in the command bytes. (i.e rq->cmd is now 16 bytes in size, and meant for
|
|
command pre-building, and the type of the request is now indicated
|
|
through rq->flags instead of via rq->cmd)
|
|
|
|
The request structure flags can be set up to indicate the type of request
|
|
in such cases (REQ_PC: direct packet command passed to driver, REQ_BLOCK_PC:
|
|
packet command issued via blk_do_rq, REQ_SPECIAL: special request).
|
|
|
|
It can help to pre-build device commands for requests in advance.
|
|
Drivers can now specify a request prepare function (q->prep_rq_fn) that the
|
|
block layer would invoke to pre-build device commands for a given request,
|
|
or perform other preparatory processing for the request. This is routine is
|
|
called by elv_next_request(), i.e. typically just before servicing a request.
|
|
(The prepare function would not be called for requests that have REQ_DONTPREP
|
|
enabled)
|
|
|
|
Aside:
|
|
Pre-building could possibly even be done early, i.e before placing the
|
|
request on the queue, rather than construct the command on the fly in the
|
|
driver while servicing the request queue when it may affect latencies in
|
|
interrupt context or responsiveness in general. One way to add early
|
|
pre-building would be to do it whenever we fail to merge on a request.
|
|
Now REQ_NOMERGE is set in the request flags to skip this one in the future,
|
|
which means that it will not change before we feed it to the device. So
|
|
the pre-builder hook can be invoked there.
|
|
|
|
|
|
2. Flexible and generic but minimalist i/o structure/descriptor.
|
|
|
|
2.1 Reason for a new structure and requirements addressed
|
|
|
|
Prior to 2.5, buffer heads were used as the unit of i/o at the generic block
|
|
layer, and the low level request structure was associated with a chain of
|
|
buffer heads for a contiguous i/o request. This led to certain inefficiencies
|
|
when it came to large i/o requests and readv/writev style operations, as it
|
|
forced such requests to be broken up into small chunks before being passed
|
|
on to the generic block layer, only to be merged by the i/o scheduler
|
|
when the underlying device was capable of handling the i/o in one shot.
|
|
Also, using the buffer head as an i/o structure for i/os that didn't originate
|
|
from the buffer cache unnecessarily added to the weight of the descriptors
|
|
which were generated for each such chunk.
|
|
|
|
The following were some of the goals and expectations considered in the
|
|
redesign of the block i/o data structure in 2.5.
|
|
|
|
i. Should be appropriate as a descriptor for both raw and buffered i/o -
|
|
avoid cache related fields which are irrelevant in the direct/page i/o path,
|
|
or filesystem block size alignment restrictions which may not be relevant
|
|
for raw i/o.
|
|
ii. Ability to represent high-memory buffers (which do not have a virtual
|
|
address mapping in kernel address space).
|
|
iii.Ability to represent large i/os w/o unnecessarily breaking them up (i.e
|
|
greater than PAGE_SIZE chunks in one shot)
|
|
iv. At the same time, ability to retain independent identity of i/os from
|
|
different sources or i/o units requiring individual completion (e.g. for
|
|
latency reasons)
|
|
v. Ability to represent an i/o involving multiple physical memory segments
|
|
(including non-page aligned page fragments, as specified via readv/writev)
|
|
without unnecessarily breaking it up, if the underlying device is capable of
|
|
handling it.
|
|
vi. Preferably should be based on a memory descriptor structure that can be
|
|
passed around different types of subsystems or layers, maybe even
|
|
networking, without duplication or extra copies of data/descriptor fields
|
|
themselves in the process
|
|
vii.Ability to handle the possibility of splits/merges as the structure passes
|
|
through layered drivers (lvm, md, evms), with minimal overhead.
|
|
|
|
The solution was to define a new structure (bio) for the block layer,
|
|
instead of using the buffer head structure (bh) directly, the idea being
|
|
avoidance of some associated baggage and limitations. The bio structure
|
|
is uniformly used for all i/o at the block layer ; it forms a part of the
|
|
bh structure for buffered i/o, and in the case of raw/direct i/o kiobufs are
|
|
mapped to bio structures.
|
|
|
|
2.2 The bio struct
|
|
|
|
The bio structure uses a vector representation pointing to an array of tuples
|
|
of <page, offset, len> to describe the i/o buffer, and has various other
|
|
fields describing i/o parameters and state that needs to be maintained for
|
|
performing the i/o.
|
|
|
|
Notice that this representation means that a bio has no virtual address
|
|
mapping at all (unlike buffer heads).
|
|
|
|
struct bio_vec {
|
|
struct page *bv_page;
|
|
unsigned short bv_len;
|
|
unsigned short bv_offset;
|
|
};
|
|
|
|
/*
|
|
* main unit of I/O for the block layer and lower layers (ie drivers)
|
|
*/
|
|
struct bio {
|
|
struct bio *bi_next; /* request queue link */
|
|
struct block_device *bi_bdev; /* target device */
|
|
unsigned long bi_flags; /* status, command, etc */
|
|
unsigned long bi_rw; /* low bits: r/w, high: priority */
|
|
|
|
unsigned int bi_vcnt; /* how may bio_vec's */
|
|
struct bvec_iter bi_iter; /* current index into bio_vec array */
|
|
|
|
unsigned int bi_size; /* total size in bytes */
|
|
unsigned short bi_phys_segments; /* segments after physaddr coalesce*/
|
|
unsigned short bi_hw_segments; /* segments after DMA remapping */
|
|
unsigned int bi_max; /* max bio_vecs we can hold
|
|
used as index into pool */
|
|
struct bio_vec *bi_io_vec; /* the actual vec list */
|
|
bio_end_io_t *bi_end_io; /* bi_end_io (bio) */
|
|
atomic_t bi_cnt; /* pin count: free when it hits zero */
|
|
void *bi_private;
|
|
};
|
|
|
|
With this multipage bio design:
|
|
|
|
- Large i/os can be sent down in one go using a bio_vec list consisting
|
|
of an array of <page, offset, len> fragments (similar to the way fragments
|
|
are represented in the zero-copy network code)
|
|
- Splitting of an i/o request across multiple devices (as in the case of
|
|
lvm or raid) is achieved by cloning the bio (where the clone points to
|
|
the same bi_io_vec array, but with the index and size accordingly modified)
|
|
- A linked list of bios is used as before for unrelated merges (*) - this
|
|
avoids reallocs and makes independent completions easier to handle.
|
|
- Code that traverses the req list can find all the segments of a bio
|
|
by using rq_for_each_segment. This handles the fact that a request
|
|
has multiple bios, each of which can have multiple segments.
|
|
- Drivers which can't process a large bio in one shot can use the bi_iter
|
|
field to keep track of the next bio_vec entry to process.
|
|
(e.g a 1MB bio_vec needs to be handled in max 128kB chunks for IDE)
|
|
[TBD: Should preferably also have a bi_voffset and bi_vlen to avoid modifying
|
|
bi_offset an len fields]
|
|
|
|
(*) unrelated merges -- a request ends up containing two or more bios that
|
|
didn't originate from the same place.
|
|
|
|
bi_end_io() i/o callback gets called on i/o completion of the entire bio.
|
|
|
|
At a lower level, drivers build a scatter gather list from the merged bios.
|
|
The scatter gather list is in the form of an array of <page, offset, len>
|
|
entries with their corresponding dma address mappings filled in at the
|
|
appropriate time. As an optimization, contiguous physical pages can be
|
|
covered by a single entry where <page> refers to the first page and <len>
|
|
covers the range of pages (up to 16 contiguous pages could be covered this
|
|
way). There is a helper routine (blk_rq_map_sg) which drivers can use to build
|
|
the sg list.
|
|
|
|
Note: Right now the only user of bios with more than one page is ll_rw_kio,
|
|
which in turn means that only raw I/O uses it (direct i/o may not work
|
|
right now). The intent however is to enable clustering of pages etc to
|
|
become possible. The pagebuf abstraction layer from SGI also uses multi-page
|
|
bios, but that is currently not included in the stock development kernels.
|
|
The same is true of Andrew Morton's work-in-progress multipage bio writeout
|
|
and readahead patches.
|
|
|
|
2.3 Changes in the Request Structure
|
|
|
|
The request structure is the structure that gets passed down to low level
|
|
drivers. The block layer make_request function builds up a request structure,
|
|
places it on the queue and invokes the drivers request_fn. The driver makes
|
|
use of block layer helper routine elv_next_request to pull the next request
|
|
off the queue. Control or diagnostic functions might bypass block and directly
|
|
invoke underlying driver entry points passing in a specially constructed
|
|
request structure.
|
|
|
|
Only some relevant fields (mainly those which changed or may be referred
|
|
to in some of the discussion here) are listed below, not necessarily in
|
|
the order in which they occur in the structure (see include/linux/blkdev.h)
|
|
Refer to Documentation/block/request.txt for details about all the request
|
|
structure fields and a quick reference about the layers which are
|
|
supposed to use or modify those fields.
|
|
|
|
struct request {
|
|
struct list_head queuelist; /* Not meant to be directly accessed by
|
|
the driver.
|
|
Used by q->elv_next_request_fn
|
|
rq->queue is gone
|
|
*/
|
|
.
|
|
.
|
|
unsigned char cmd[16]; /* prebuilt command data block */
|
|
unsigned long flags; /* also includes earlier rq->cmd settings */
|
|
.
|
|
.
|
|
sector_t sector; /* this field is now of type sector_t instead of int
|
|
preparation for 64 bit sectors */
|
|
.
|
|
.
|
|
|
|
/* Number of scatter-gather DMA addr+len pairs after
|
|
* physical address coalescing is performed.
|
|
*/
|
|
unsigned short nr_phys_segments;
|
|
|
|
/* Number of scatter-gather addr+len pairs after
|
|
* physical and DMA remapping hardware coalescing is performed.
|
|
* This is the number of scatter-gather entries the driver
|
|
* will actually have to deal with after DMA mapping is done.
|
|
*/
|
|
unsigned short nr_hw_segments;
|
|
|
|
/* Various sector counts */
|
|
unsigned long nr_sectors; /* no. of sectors left: driver modifiable */
|
|
unsigned long hard_nr_sectors; /* block internal copy of above */
|
|
unsigned int current_nr_sectors; /* no. of sectors left in the
|
|
current segment:driver modifiable */
|
|
unsigned long hard_cur_sectors; /* block internal copy of the above */
|
|
.
|
|
.
|
|
int tag; /* command tag associated with request */
|
|
void *special; /* same as before */
|
|
char *buffer; /* valid only for low memory buffers up to
|
|
current_nr_sectors */
|
|
.
|
|
.
|
|
struct bio *bio, *biotail; /* bio list instead of bh */
|
|
struct request_list *rl;
|
|
}
|
|
|
|
See the rq_flag_bits definitions for an explanation of the various flags
|
|
available. Some bits are used by the block layer or i/o scheduler.
|
|
|
|
The behaviour of the various sector counts are almost the same as before,
|
|
except that since we have multi-segment bios, current_nr_sectors refers
|
|
to the numbers of sectors in the current segment being processed which could
|
|
be one of the many segments in the current bio (i.e i/o completion unit).
|
|
The nr_sectors value refers to the total number of sectors in the whole
|
|
request that remain to be transferred (no change). The purpose of the
|
|
hard_xxx values is for block to remember these counts every time it hands
|
|
over the request to the driver. These values are updated by block on
|
|
end_that_request_first, i.e. every time the driver completes a part of the
|
|
transfer and invokes block end*request helpers to mark this. The
|
|
driver should not modify these values. The block layer sets up the
|
|
nr_sectors and current_nr_sectors fields (based on the corresponding
|
|
hard_xxx values and the number of bytes transferred) and updates it on
|
|
every transfer that invokes end_that_request_first. It does the same for the
|
|
buffer, bio, bio->bi_iter fields too.
|
|
|
|
The buffer field is just a virtual address mapping of the current segment
|
|
of the i/o buffer in cases where the buffer resides in low-memory. For high
|
|
memory i/o, this field is not valid and must not be used by drivers.
|
|
|
|
Code that sets up its own request structures and passes them down to
|
|
a driver needs to be careful about interoperation with the block layer helper
|
|
functions which the driver uses. (Section 1.3)
|
|
|
|
3. Using bios
|
|
|
|
3.1 Setup/Teardown
|
|
|
|
There are routines for managing the allocation, and reference counting, and
|
|
freeing of bios (bio_alloc, bio_get, bio_put).
|
|
|
|
This makes use of Ingo Molnar's mempool implementation, which enables
|
|
subsystems like bio to maintain their own reserve memory pools for guaranteed
|
|
deadlock-free allocations during extreme VM load. For example, the VM
|
|
subsystem makes use of the block layer to writeout dirty pages in order to be
|
|
able to free up memory space, a case which needs careful handling. The
|
|
allocation logic draws from the preallocated emergency reserve in situations
|
|
where it cannot allocate through normal means. If the pool is empty and it
|
|
can wait, then it would trigger action that would help free up memory or
|
|
replenish the pool (without deadlocking) and wait for availability in the pool.
|
|
If it is in IRQ context, and hence not in a position to do this, allocation
|
|
could fail if the pool is empty. In general mempool always first tries to
|
|
perform allocation without having to wait, even if it means digging into the
|
|
pool as long it is not less that 50% full.
|
|
|
|
On a free, memory is released to the pool or directly freed depending on
|
|
the current availability in the pool. The mempool interface lets the
|
|
subsystem specify the routines to be used for normal alloc and free. In the
|
|
case of bio, these routines make use of the standard slab allocator.
|
|
|
|
The caller of bio_alloc is expected to taken certain steps to avoid
|
|
deadlocks, e.g. avoid trying to allocate more memory from the pool while
|
|
already holding memory obtained from the pool.
|
|
[TBD: This is a potential issue, though a rare possibility
|
|
in the bounce bio allocation that happens in the current code, since
|
|
it ends up allocating a second bio from the same pool while
|
|
holding the original bio ]
|
|
|
|
Memory allocated from the pool should be released back within a limited
|
|
amount of time (in the case of bio, that would be after the i/o is completed).
|
|
This ensures that if part of the pool has been used up, some work (in this
|
|
case i/o) must already be in progress and memory would be available when it
|
|
is over. If allocating from multiple pools in the same code path, the order
|
|
or hierarchy of allocation needs to be consistent, just the way one deals
|
|
with multiple locks.
|
|
|
|
The bio_alloc routine also needs to allocate the bio_vec_list (bvec_alloc())
|
|
for a non-clone bio. There are the 6 pools setup for different size biovecs,
|
|
so bio_alloc(gfp_mask, nr_iovecs) will allocate a vec_list of the
|
|
given size from these slabs.
|
|
|
|
The bio_get() routine may be used to hold an extra reference on a bio prior
|
|
to i/o submission, if the bio fields are likely to be accessed after the
|
|
i/o is issued (since the bio may otherwise get freed in case i/o completion
|
|
happens in the meantime).
|
|
|
|
The bio_clone() routine may be used to duplicate a bio, where the clone
|
|
shares the bio_vec_list with the original bio (i.e. both point to the
|
|
same bio_vec_list). This would typically be used for splitting i/o requests
|
|
in lvm or md.
|
|
|
|
3.2 Generic bio helper Routines
|
|
|
|
3.2.1 Traversing segments and completion units in a request
|
|
|
|
The macro rq_for_each_segment() should be used for traversing the bios
|
|
in the request list (drivers should avoid directly trying to do it
|
|
themselves). Using these helpers should also make it easier to cope
|
|
with block changes in the future.
|
|
|
|
struct req_iterator iter;
|
|
rq_for_each_segment(bio_vec, rq, iter)
|
|
/* bio_vec is now current segment */
|
|
|
|
I/O completion callbacks are per-bio rather than per-segment, so drivers
|
|
that traverse bio chains on completion need to keep that in mind. Drivers
|
|
which don't make a distinction between segments and completion units would
|
|
need to be reorganized to support multi-segment bios.
|
|
|
|
3.2.2 Setting up DMA scatterlists
|
|
|
|
The blk_rq_map_sg() helper routine would be used for setting up scatter
|
|
gather lists from a request, so a driver need not do it on its own.
|
|
|
|
nr_segments = blk_rq_map_sg(q, rq, scatterlist);
|
|
|
|
The helper routine provides a level of abstraction which makes it easier
|
|
to modify the internals of request to scatterlist conversion down the line
|
|
without breaking drivers. The blk_rq_map_sg routine takes care of several
|
|
things like collapsing physically contiguous segments (if QUEUE_FLAG_CLUSTER
|
|
is set) and correct segment accounting to avoid exceeding the limits which
|
|
the i/o hardware can handle, based on various queue properties.
|
|
|
|
- Prevents a clustered segment from crossing a 4GB mem boundary
|
|
- Avoids building segments that would exceed the number of physical
|
|
memory segments that the driver can handle (phys_segments) and the
|
|
number that the underlying hardware can handle at once, accounting for
|
|
DMA remapping (hw_segments) (i.e. IOMMU aware limits).
|
|
|
|
Routines which the low level driver can use to set up the segment limits:
|
|
|
|
blk_queue_max_hw_segments() : Sets an upper limit of the maximum number of
|
|
hw data segments in a request (i.e. the maximum number of address/length
|
|
pairs the host adapter can actually hand to the device at once)
|
|
|
|
blk_queue_max_phys_segments() : Sets an upper limit on the maximum number
|
|
of physical data segments in a request (i.e. the largest sized scatter list
|
|
a driver could handle)
|
|
|
|
3.2.3 I/O completion
|
|
|
|
The existing generic block layer helper routines end_request,
|
|
end_that_request_first and end_that_request_last can be used for i/o
|
|
completion (and setting things up so the rest of the i/o or the next
|
|
request can be kicked of) as before. With the introduction of multi-page
|
|
bio support, end_that_request_first requires an additional argument indicating
|
|
the number of sectors completed.
|
|
|
|
3.2.4 Implications for drivers that do not interpret bios (don't handle
|
|
multiple segments)
|
|
|
|
Drivers that do not interpret bios e.g those which do not handle multiple
|
|
segments and do not support i/o into high memory addresses (require bounce
|
|
buffers) and expect only virtually mapped buffers, can access the rq->buffer
|
|
field. As before the driver should use current_nr_sectors to determine the
|
|
size of remaining data in the current segment (that is the maximum it can
|
|
transfer in one go unless it interprets segments), and rely on the block layer
|
|
end_request, or end_that_request_first/last to take care of all accounting
|
|
and transparent mapping of the next bio segment when a segment boundary
|
|
is crossed on completion of a transfer. (The end*request* functions should
|
|
be used if only if the request has come down from block/bio path, not for
|
|
direct access requests which only specify rq->buffer without a valid rq->bio)
|
|
|
|
3.2.5 Generic request command tagging
|
|
|
|
3.2.5.1 Tag helpers
|
|
|
|
Block now offers some simple generic functionality to help support command
|
|
queueing (typically known as tagged command queueing), ie manage more than
|
|
one outstanding command on a queue at any given time.
|
|
|
|
blk_queue_init_tags(struct request_queue *q, int depth)
|
|
|
|
Initialize internal command tagging structures for a maximum
|
|
depth of 'depth'.
|
|
|
|
blk_queue_free_tags((struct request_queue *q)
|
|
|
|
Teardown tag info associated with the queue. This will be done
|
|
automatically by block if blk_queue_cleanup() is called on a queue
|
|
that is using tagging.
|
|
|
|
The above are initialization and exit management, the main helpers during
|
|
normal operations are:
|
|
|
|
blk_queue_start_tag(struct request_queue *q, struct request *rq)
|
|
|
|
Start tagged operation for this request. A free tag number between
|
|
0 and 'depth' is assigned to the request (rq->tag holds this number),
|
|
and 'rq' is added to the internal tag management. If the maximum depth
|
|
for this queue is already achieved (or if the tag wasn't started for
|
|
some other reason), 1 is returned. Otherwise 0 is returned.
|
|
|
|
blk_queue_end_tag(struct request_queue *q, struct request *rq)
|
|
|
|
End tagged operation on this request. 'rq' is removed from the internal
|
|
book keeping structures.
|
|
|
|
To minimize struct request and queue overhead, the tag helpers utilize some
|
|
of the same request members that are used for normal request queue management.
|
|
This means that a request cannot both be an active tag and be on the queue
|
|
list at the same time. blk_queue_start_tag() will remove the request, but
|
|
the driver must remember to call blk_queue_end_tag() before signalling
|
|
completion of the request to the block layer. This means ending tag
|
|
operations before calling end_that_request_last()! For an example of a user
|
|
of these helpers, see the IDE tagged command queueing support.
|
|
|
|
Certain hardware conditions may dictate a need to invalidate the block tag
|
|
queue. For instance, on IDE any tagged request error needs to clear both
|
|
the hardware and software block queue and enable the driver to sanely restart
|
|
all the outstanding requests. There's a third helper to do that:
|
|
|
|
blk_queue_invalidate_tags(struct request_queue *q)
|
|
|
|
Clear the internal block tag queue and re-add all the pending requests
|
|
to the request queue. The driver will receive them again on the
|
|
next request_fn run, just like it did the first time it encountered
|
|
them.
|
|
|
|
3.2.5.2 Tag info
|
|
|
|
Some block functions exist to query current tag status or to go from a
|
|
tag number to the associated request. These are, in no particular order:
|
|
|
|
blk_queue_tagged(q)
|
|
|
|
Returns 1 if the queue 'q' is using tagging, 0 if not.
|
|
|
|
blk_queue_tag_request(q, tag)
|
|
|
|
Returns a pointer to the request associated with tag 'tag'.
|
|
|
|
blk_queue_tag_depth(q)
|
|
|
|
Return current queue depth.
|
|
|
|
blk_queue_tag_queue(q)
|
|
|
|
Returns 1 if the queue can accept a new queued command, 0 if we are
|
|
at the maximum depth already.
|
|
|
|
blk_queue_rq_tagged(rq)
|
|
|
|
Returns 1 if the request 'rq' is tagged.
|
|
|
|
3.2.5.2 Internal structure
|
|
|
|
Internally, block manages tags in the blk_queue_tag structure:
|
|
|
|
struct blk_queue_tag {
|
|
struct request **tag_index; /* array or pointers to rq */
|
|
unsigned long *tag_map; /* bitmap of free tags */
|
|
struct list_head busy_list; /* fifo list of busy tags */
|
|
int busy; /* queue depth */
|
|
int max_depth; /* max queue depth */
|
|
};
|
|
|
|
Most of the above is simple and straight forward, however busy_list may need
|
|
a bit of explaining. Normally we don't care too much about request ordering,
|
|
but in the event of any barrier requests in the tag queue we need to ensure
|
|
that requests are restarted in the order they were queue. This may happen
|
|
if the driver needs to use blk_queue_invalidate_tags().
|
|
|
|
Tagging also defines a new request flag, REQ_QUEUED. This is set whenever
|
|
a request is currently tagged. You should not use this flag directly,
|
|
blk_rq_tagged(rq) is the portable way to do so.
|
|
|
|
3.3 I/O Submission
|
|
|
|
The routine submit_bio() is used to submit a single io. Higher level i/o
|
|
routines make use of this:
|
|
|
|
(a) Buffered i/o:
|
|
The routine submit_bh() invokes submit_bio() on a bio corresponding to the
|
|
bh, allocating the bio if required. ll_rw_block() uses submit_bh() as before.
|
|
|
|
(b) Kiobuf i/o (for raw/direct i/o):
|
|
The ll_rw_kio() routine breaks up the kiobuf into page sized chunks and
|
|
maps the array to one or more multi-page bios, issuing submit_bio() to
|
|
perform the i/o on each of these.
|
|
|
|
The embedded bh array in the kiobuf structure has been removed and no
|
|
preallocation of bios is done for kiobufs. [The intent is to remove the
|
|
blocks array as well, but it's currently in there to kludge around direct i/o.]
|
|
Thus kiobuf allocation has switched back to using kmalloc rather than vmalloc.
|
|
|
|
Todo/Observation:
|
|
|
|
A single kiobuf structure is assumed to correspond to a contiguous range
|
|
of data, so brw_kiovec() invokes ll_rw_kio for each kiobuf in a kiovec.
|
|
So right now it wouldn't work for direct i/o on non-contiguous blocks.
|
|
This is to be resolved. The eventual direction is to replace kiobuf
|
|
by kvec's.
|
|
|
|
Badari Pulavarty has a patch to implement direct i/o correctly using
|
|
bio and kvec.
|
|
|
|
|
|
(c) Page i/o:
|
|
Todo/Under discussion:
|
|
|
|
Andrew Morton's multi-page bio patches attempt to issue multi-page
|
|
writeouts (and reads) from the page cache, by directly building up
|
|
large bios for submission completely bypassing the usage of buffer
|
|
heads. This work is still in progress.
|
|
|
|
Christoph Hellwig had some code that uses bios for page-io (rather than
|
|
bh). This isn't included in bio as yet. Christoph was also working on a
|
|
design for representing virtual/real extents as an entity and modifying
|
|
some of the address space ops interfaces to utilize this abstraction rather
|
|
than buffer_heads. (This is somewhat along the lines of the SGI XFS pagebuf
|
|
abstraction, but intended to be as lightweight as possible).
|
|
|
|
(d) Direct access i/o:
|
|
Direct access requests that do not contain bios would be submitted differently
|
|
as discussed earlier in section 1.3.
|
|
|
|
Aside:
|
|
|
|
Kvec i/o:
|
|
|
|
Ben LaHaise's aio code uses a slightly different structure instead
|
|
of kiobufs, called a kvec_cb. This contains an array of <page, offset, len>
|
|
tuples (very much like the networking code), together with a callback function
|
|
and data pointer. This is embedded into a brw_cb structure when passed
|
|
to brw_kvec_async().
|
|
|
|
Now it should be possible to directly map these kvecs to a bio. Just as while
|
|
cloning, in this case rather than PRE_BUILT bio_vecs, we set the bi_io_vec
|
|
array pointer to point to the veclet array in kvecs.
|
|
|
|
TBD: In order for this to work, some changes are needed in the way multi-page
|
|
bios are handled today. The values of the tuples in such a vector passed in
|
|
from higher level code should not be modified by the block layer in the course
|
|
of its request processing, since that would make it hard for the higher layer
|
|
to continue to use the vector descriptor (kvec) after i/o completes. Instead,
|
|
all such transient state should either be maintained in the request structure,
|
|
and passed on in some way to the endio completion routine.
|
|
|
|
|
|
4. The I/O scheduler
|
|
I/O scheduler, a.k.a. elevator, is implemented in two layers. Generic dispatch
|
|
queue and specific I/O schedulers. Unless stated otherwise, elevator is used
|
|
to refer to both parts and I/O scheduler to specific I/O schedulers.
|
|
|
|
Block layer implements generic dispatch queue in block/*.c.
|
|
The generic dispatch queue is responsible for properly ordering barrier
|
|
requests, requeueing, handling non-fs requests and all other subtleties.
|
|
|
|
Specific I/O schedulers are responsible for ordering normal filesystem
|
|
requests. They can also choose to delay certain requests to improve
|
|
throughput or whatever purpose. As the plural form indicates, there are
|
|
multiple I/O schedulers. They can be built as modules but at least one should
|
|
be built inside the kernel. Each queue can choose different one and can also
|
|
change to another one dynamically.
|
|
|
|
A block layer call to the i/o scheduler follows the convention elv_xxx(). This
|
|
calls elevator_xxx_fn in the elevator switch (block/elevator.c). Oh, xxx
|
|
and xxx might not match exactly, but use your imagination. If an elevator
|
|
doesn't implement a function, the switch does nothing or some minimal house
|
|
keeping work.
|
|
|
|
4.1. I/O scheduler API
|
|
|
|
The functions an elevator may implement are: (* are mandatory)
|
|
elevator_merge_fn called to query requests for merge with a bio
|
|
|
|
elevator_merge_req_fn called when two requests get merged. the one
|
|
which gets merged into the other one will be
|
|
never seen by I/O scheduler again. IOW, after
|
|
being merged, the request is gone.
|
|
|
|
elevator_merged_fn called when a request in the scheduler has been
|
|
involved in a merge. It is used in the deadline
|
|
scheduler for example, to reposition the request
|
|
if its sorting order has changed.
|
|
|
|
elevator_allow_merge_fn called whenever the block layer determines
|
|
that a bio can be merged into an existing
|
|
request safely. The io scheduler may still
|
|
want to stop a merge at this point if it
|
|
results in some sort of conflict internally,
|
|
this hook allows it to do that.
|
|
|
|
elevator_dispatch_fn* fills the dispatch queue with ready requests.
|
|
I/O schedulers are free to postpone requests by
|
|
not filling the dispatch queue unless @force
|
|
is non-zero. Once dispatched, I/O schedulers
|
|
are not allowed to manipulate the requests -
|
|
they belong to generic dispatch queue.
|
|
|
|
elevator_add_req_fn* called to add a new request into the scheduler
|
|
|
|
elevator_former_req_fn
|
|
elevator_latter_req_fn These return the request before or after the
|
|
one specified in disk sort order. Used by the
|
|
block layer to find merge possibilities.
|
|
|
|
elevator_completed_req_fn called when a request is completed.
|
|
|
|
elevator_may_queue_fn returns true if the scheduler wants to allow the
|
|
current context to queue a new request even if
|
|
it is over the queue limit. This must be used
|
|
very carefully!!
|
|
|
|
elevator_set_req_fn
|
|
elevator_put_req_fn Must be used to allocate and free any elevator
|
|
specific storage for a request.
|
|
|
|
elevator_activate_req_fn Called when device driver first sees a request.
|
|
I/O schedulers can use this callback to
|
|
determine when actual execution of a request
|
|
starts.
|
|
elevator_deactivate_req_fn Called when device driver decides to delay
|
|
a request by requeueing it.
|
|
|
|
elevator_init_fn*
|
|
elevator_exit_fn Allocate and free any elevator specific storage
|
|
for a queue.
|
|
|
|
4.2 Request flows seen by I/O schedulers
|
|
All requests seen by I/O schedulers strictly follow one of the following three
|
|
flows.
|
|
|
|
set_req_fn ->
|
|
|
|
i. add_req_fn -> (merged_fn ->)* -> dispatch_fn -> activate_req_fn ->
|
|
(deactivate_req_fn -> activate_req_fn ->)* -> completed_req_fn
|
|
ii. add_req_fn -> (merged_fn ->)* -> merge_req_fn
|
|
iii. [none]
|
|
|
|
-> put_req_fn
|
|
|
|
4.3 I/O scheduler implementation
|
|
The generic i/o scheduler algorithm attempts to sort/merge/batch requests for
|
|
optimal disk scan and request servicing performance (based on generic
|
|
principles and device capabilities), optimized for:
|
|
i. improved throughput
|
|
ii. improved latency
|
|
iii. better utilization of h/w & CPU time
|
|
|
|
Characteristics:
|
|
|
|
i. Binary tree
|
|
AS and deadline i/o schedulers use red black binary trees for disk position
|
|
sorting and searching, and a fifo linked list for time-based searching. This
|
|
gives good scalability and good availability of information. Requests are
|
|
almost always dispatched in disk sort order, so a cache is kept of the next
|
|
request in sort order to prevent binary tree lookups.
|
|
|
|
This arrangement is not a generic block layer characteristic however, so
|
|
elevators may implement queues as they please.
|
|
|
|
ii. Merge hash
|
|
AS and deadline use a hash table indexed by the last sector of a request. This
|
|
enables merging code to quickly look up "back merge" candidates, even when
|
|
multiple I/O streams are being performed at once on one disk.
|
|
|
|
"Front merges", a new request being merged at the front of an existing request,
|
|
are far less common than "back merges" due to the nature of most I/O patterns.
|
|
Front merges are handled by the binary trees in AS and deadline schedulers.
|
|
|
|
iii. Plugging the queue to batch requests in anticipation of opportunities for
|
|
merge/sort optimizations
|
|
|
|
Plugging is an approach that the current i/o scheduling algorithm resorts to so
|
|
that it collects up enough requests in the queue to be able to take
|
|
advantage of the sorting/merging logic in the elevator. If the
|
|
queue is empty when a request comes in, then it plugs the request queue
|
|
(sort of like plugging the bath tub of a vessel to get fluid to build up)
|
|
till it fills up with a few more requests, before starting to service
|
|
the requests. This provides an opportunity to merge/sort the requests before
|
|
passing them down to the device. There are various conditions when the queue is
|
|
unplugged (to open up the flow again), either through a scheduled task or
|
|
could be on demand. For example wait_on_buffer sets the unplugging going
|
|
through sync_buffer() running blk_run_address_space(mapping). Or the caller
|
|
can do it explicity through blk_unplug(bdev). So in the read case,
|
|
the queue gets explicitly unplugged as part of waiting for completion on that
|
|
buffer. For page driven IO, the address space ->sync_page() takes care of
|
|
doing the blk_run_address_space().
|
|
|
|
Aside:
|
|
This is kind of controversial territory, as it's not clear if plugging is
|
|
always the right thing to do. Devices typically have their own queues,
|
|
and allowing a big queue to build up in software, while letting the device be
|
|
idle for a while may not always make sense. The trick is to handle the fine
|
|
balance between when to plug and when to open up. Also now that we have
|
|
multi-page bios being queued in one shot, we may not need to wait to merge
|
|
a big request from the broken up pieces coming by.
|
|
|
|
4.4 I/O contexts
|
|
I/O contexts provide a dynamically allocated per process data area. They may
|
|
be used in I/O schedulers, and in the block layer (could be used for IO statis,
|
|
priorities for example). See *io_context in block/ll_rw_blk.c, and as-iosched.c
|
|
for an example of usage in an i/o scheduler.
|
|
|
|
|
|
5. Scalability related changes
|
|
|
|
5.1 Granular Locking: io_request_lock replaced by a per-queue lock
|
|
|
|
The global io_request_lock has been removed as of 2.5, to avoid
|
|
the scalability bottleneck it was causing, and has been replaced by more
|
|
granular locking. The request queue structure has a pointer to the
|
|
lock to be used for that queue. As a result, locking can now be
|
|
per-queue, with a provision for sharing a lock across queues if
|
|
necessary (e.g the scsi layer sets the queue lock pointers to the
|
|
corresponding adapter lock, which results in a per host locking
|
|
granularity). The locking semantics are the same, i.e. locking is
|
|
still imposed by the block layer, grabbing the lock before
|
|
request_fn execution which it means that lots of older drivers
|
|
should still be SMP safe. Drivers are free to drop the queue
|
|
lock themselves, if required. Drivers that explicitly used the
|
|
io_request_lock for serialization need to be modified accordingly.
|
|
Usually it's as easy as adding a global lock:
|
|
|
|
static DEFINE_SPINLOCK(my_driver_lock);
|
|
|
|
and passing the address to that lock to blk_init_queue().
|
|
|
|
5.2 64 bit sector numbers (sector_t prepares for 64 bit support)
|
|
|
|
The sector number used in the bio structure has been changed to sector_t,
|
|
which could be defined as 64 bit in preparation for 64 bit sector support.
|
|
|
|
6. Other Changes/Implications
|
|
|
|
6.1 Partition re-mapping handled by the generic block layer
|
|
|
|
In 2.5 some of the gendisk/partition related code has been reorganized.
|
|
Now the generic block layer performs partition-remapping early and thus
|
|
provides drivers with a sector number relative to whole device, rather than
|
|
having to take partition number into account in order to arrive at the true
|
|
sector number. The routine blk_partition_remap() is invoked by
|
|
generic_make_request even before invoking the queue specific make_request_fn,
|
|
so the i/o scheduler also gets to operate on whole disk sector numbers. This
|
|
should typically not require changes to block drivers, it just never gets
|
|
to invoke its own partition sector offset calculations since all bios
|
|
sent are offset from the beginning of the device.
|
|
|
|
|
|
7. A Few Tips on Migration of older drivers
|
|
|
|
Old-style drivers that just use CURRENT and ignores clustered requests,
|
|
may not need much change. The generic layer will automatically handle
|
|
clustered requests, multi-page bios, etc for the driver.
|
|
|
|
For a low performance driver or hardware that is PIO driven or just doesn't
|
|
support scatter-gather changes should be minimal too.
|
|
|
|
The following are some points to keep in mind when converting old drivers
|
|
to bio.
|
|
|
|
Drivers should use elv_next_request to pick up requests and are no longer
|
|
supposed to handle looping directly over the request list.
|
|
(struct request->queue has been removed)
|
|
|
|
Now end_that_request_first takes an additional number_of_sectors argument.
|
|
It used to handle always just the first buffer_head in a request, now
|
|
it will loop and handle as many sectors (on a bio-segment granularity)
|
|
as specified.
|
|
|
|
Now bh->b_end_io is replaced by bio->bi_end_io, but most of the time the
|
|
right thing to use is bio_endio(bio, uptodate) instead.
|
|
|
|
If the driver is dropping the io_request_lock from its request_fn strategy,
|
|
then it just needs to replace that with q->queue_lock instead.
|
|
|
|
As described in Sec 1.1, drivers can set max sector size, max segment size
|
|
etc per queue now. Drivers that used to define their own merge functions i
|
|
to handle things like this can now just use the blk_queue_* functions at
|
|
blk_init_queue time.
|
|
|
|
Drivers no longer have to map a {partition, sector offset} into the
|
|
correct absolute location anymore, this is done by the block layer, so
|
|
where a driver received a request ala this before:
|
|
|
|
rq->rq_dev = mk_kdev(3, 5); /* /dev/hda5 */
|
|
rq->sector = 0; /* first sector on hda5 */
|
|
|
|
it will now see
|
|
|
|
rq->rq_dev = mk_kdev(3, 0); /* /dev/hda */
|
|
rq->sector = 123128; /* offset from start of disk */
|
|
|
|
As mentioned, there is no virtual mapping of a bio. For DMA, this is
|
|
not a problem as the driver probably never will need a virtual mapping.
|
|
Instead it needs a bus mapping (dma_map_page for a single segment or
|
|
use dma_map_sg for scatter gather) to be able to ship it to the driver. For
|
|
PIO drivers (or drivers that need to revert to PIO transfer once in a
|
|
while (IDE for example)), where the CPU is doing the actual data
|
|
transfer a virtual mapping is needed. If the driver supports highmem I/O,
|
|
(Sec 1.1, (ii) ) it needs to use __bio_kmap_atomic and bio_kmap_irq to
|
|
temporarily map a bio into the virtual address space.
|
|
|
|
|
|
8. Prior/Related/Impacted patches
|
|
|
|
8.1. Earlier kiobuf patches (sct/axboe/chait/hch/mkp)
|
|
- orig kiobuf & raw i/o patches (now in 2.4 tree)
|
|
- direct kiobuf based i/o to devices (no intermediate bh's)
|
|
- page i/o using kiobuf
|
|
- kiobuf splitting for lvm (mkp)
|
|
- elevator support for kiobuf request merging (axboe)
|
|
8.2. Zero-copy networking (Dave Miller)
|
|
8.3. SGI XFS - pagebuf patches - use of kiobufs
|
|
8.4. Multi-page pioent patch for bio (Christoph Hellwig)
|
|
8.5. Direct i/o implementation (Andrea Arcangeli) since 2.4.10-pre11
|
|
8.6. Async i/o implementation patch (Ben LaHaise)
|
|
8.7. EVMS layering design (IBM EVMS team)
|
|
8.8. Larger page cache size patch (Ben LaHaise) and
|
|
Large page size (Daniel Phillips)
|
|
=> larger contiguous physical memory buffers
|
|
8.9. VM reservations patch (Ben LaHaise)
|
|
8.10. Write clustering patches ? (Marcelo/Quintela/Riel ?)
|
|
8.11. Block device in page cache patch (Andrea Archangeli) - now in 2.4.10+
|
|
8.12. Multiple block-size transfers for faster raw i/o (Shailabh Nagar,
|
|
Badari)
|
|
8.13 Priority based i/o scheduler - prepatches (Arjan van de Ven)
|
|
8.14 IDE Taskfile i/o patch (Andre Hedrick)
|
|
8.15 Multi-page writeout and readahead patches (Andrew Morton)
|
|
8.16 Direct i/o patches for 2.5 using kvec and bio (Badari Pulavarthy)
|
|
|
|
9. Other References:
|
|
|
|
9.1 The Splice I/O Model - Larry McVoy (and subsequent discussions on lkml,
|
|
and Linus' comments - Jan 2001)
|
|
9.2 Discussions about kiobuf and bh design on lkml between sct, linus, alan
|
|
et al - Feb-March 2001 (many of the initial thoughts that led to bio were
|
|
brought up in this discussion thread)
|
|
9.3 Discussions on mempool on lkml - Dec 2001.
|
|
|