License cleanup: add SPDX GPL-2.0 license identifier to files with no license
Many source files in the tree are missing licensing information, which
makes it harder for compliance tools to determine the correct license.
By default all files without license information are under the default
license of the kernel, which is GPL version 2.
Update the files which contain no license information with the 'GPL-2.0'
SPDX license identifier. The SPDX identifier is a legally binding
shorthand, which can be used instead of the full boiler plate text.
This patch is based on work done by Thomas Gleixner and Kate Stewart and
Philippe Ombredanne.
How this work was done:
Patches were generated and checked against linux-4.14-rc6 for a subset of
the use cases:
- file had no licensing information it it.
- file was a */uapi/* one with no licensing information in it,
- file was a */uapi/* one with existing licensing information,
Further patches will be generated in subsequent months to fix up cases
where non-standard license headers were used, and references to license
had to be inferred by heuristics based on keywords.
The analysis to determine which SPDX License Identifier to be applied to
a file was done in a spreadsheet of side by side results from of the
output of two independent scanners (ScanCode & Windriver) producing SPDX
tag:value files created by Philippe Ombredanne. Philippe prepared the
base worksheet, and did an initial spot review of a few 1000 files.
The 4.13 kernel was the starting point of the analysis with 60,537 files
assessed. Kate Stewart did a file by file comparison of the scanner
results in the spreadsheet to determine which SPDX license identifier(s)
to be applied to the file. She confirmed any determination that was not
immediately clear with lawyers working with the Linux Foundation.
Criteria used to select files for SPDX license identifier tagging was:
- Files considered eligible had to be source code files.
- Make and config files were included as candidates if they contained >5
lines of source
- File already had some variant of a license header in it (even if <5
lines).
All documentation files were explicitly excluded.
The following heuristics were used to determine which SPDX license
identifiers to apply.
- when both scanners couldn't find any license traces, file was
considered to have no license information in it, and the top level
COPYING file license applied.
For non */uapi/* files that summary was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 11139
and resulted in the first patch in this series.
If that file was a */uapi/* path one, it was "GPL-2.0 WITH
Linux-syscall-note" otherwise it was "GPL-2.0". Results of that was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 WITH Linux-syscall-note 930
and resulted in the second patch in this series.
- if a file had some form of licensing information in it, and was one
of the */uapi/* ones, it was denoted with the Linux-syscall-note if
any GPL family license was found in the file or had no licensing in
it (per prior point). Results summary:
SPDX license identifier # files
---------------------------------------------------|------
GPL-2.0 WITH Linux-syscall-note 270
GPL-2.0+ WITH Linux-syscall-note 169
((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) 21
((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) 17
LGPL-2.1+ WITH Linux-syscall-note 15
GPL-1.0+ WITH Linux-syscall-note 14
((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) 5
LGPL-2.0+ WITH Linux-syscall-note 4
LGPL-2.1 WITH Linux-syscall-note 3
((GPL-2.0 WITH Linux-syscall-note) OR MIT) 3
((GPL-2.0 WITH Linux-syscall-note) AND MIT) 1
and that resulted in the third patch in this series.
- when the two scanners agreed on the detected license(s), that became
the concluded license(s).
- when there was disagreement between the two scanners (one detected a
license but the other didn't, or they both detected different
licenses) a manual inspection of the file occurred.
- In most cases a manual inspection of the information in the file
resulted in a clear resolution of the license that should apply (and
which scanner probably needed to revisit its heuristics).
- When it was not immediately clear, the license identifier was
confirmed with lawyers working with the Linux Foundation.
- If there was any question as to the appropriate license identifier,
the file was flagged for further research and to be revisited later
in time.
In total, over 70 hours of logged manual review was done on the
spreadsheet to determine the SPDX license identifiers to apply to the
source files by Kate, Philippe, Thomas and, in some cases, confirmation
by lawyers working with the Linux Foundation.
Kate also obtained a third independent scan of the 4.13 code base from
FOSSology, and compared selected files where the other two scanners
disagreed against that SPDX file, to see if there was new insights. The
Windriver scanner is based on an older version of FOSSology in part, so
they are related.
Thomas did random spot checks in about 500 files from the spreadsheets
for the uapi headers and agreed with SPDX license identifier in the
files he inspected. For the non-uapi files Thomas did random spot checks
in about 15000 files.
In initial set of patches against 4.14-rc6, 3 files were found to have
copy/paste license identifier errors, and have been fixed to reflect the
correct identifier.
Additionally Philippe spent 10 hours this week doing a detailed manual
inspection and review of the 12,461 patched files from the initial patch
version early this week with:
- a full scancode scan run, collecting the matched texts, detected
license ids and scores
- reviewing anything where there was a license detected (about 500+
files) to ensure that the applied SPDX license was correct
- reviewing anything where there was no detection but the patch license
was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
SPDX license was correct
This produced a worksheet with 20 files needing minor correction. This
worksheet was then exported into 3 different .csv files for the
different types of files to be modified.
These .csv files were then reviewed by Greg. Thomas wrote a script to
parse the csv files and add the proper SPDX tag to the file, in the
format that the file expected. This script was further refined by Greg
based on the output to detect more types of files automatically and to
distinguish between header and source .c files (which need different
comment types.) Finally Greg ran the script using the .csv files to
generate the patches.
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-11-01 14:07:57 +00:00
|
|
|
/* SPDX-License-Identifier: GPL-2.0 */
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
#ifndef BLK_MQ_H
|
|
|
|
#define BLK_MQ_H
|
|
|
|
|
|
|
|
#include <linux/blkdev.h>
|
2016-09-17 14:38:44 +00:00
|
|
|
#include <linux/sbitmap.h>
|
2020-12-03 01:26:36 +00:00
|
|
|
#include <linux/lockdep.h>
|
2021-09-20 12:33:28 +00:00
|
|
|
#include <linux/scatterlist.h>
|
2021-10-16 22:38:14 +00:00
|
|
|
#include <linux/prefetch.h>
|
2022-11-01 15:00:47 +00:00
|
|
|
#include <linux/srcu.h>
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
|
|
|
|
struct blk_mq_tags;
|
2014-09-25 15:23:47 +00:00
|
|
|
struct blk_flush_queue;
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
|
2021-09-20 12:33:28 +00:00
|
|
|
#define BLKDEV_MIN_RQ 4
|
2021-10-05 10:23:27 +00:00
|
|
|
#define BLKDEV_DEFAULT_RQ 128
|
2021-09-20 12:33:28 +00:00
|
|
|
|
2022-09-21 21:19:54 +00:00
|
|
|
enum rq_end_io_ret {
|
|
|
|
RQ_END_IO_NONE,
|
|
|
|
RQ_END_IO_FREE,
|
|
|
|
};
|
|
|
|
|
|
|
|
typedef enum rq_end_io_ret (rq_end_io_fn)(struct request *, blk_status_t);
|
2021-09-20 12:33:28 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* request flags */
|
|
|
|
typedef __u32 __bitwise req_flags_t;
|
|
|
|
|
|
|
|
/* drive already may have started this one */
|
|
|
|
#define RQF_STARTED ((__force req_flags_t)(1 << 1))
|
|
|
|
/* may not be passed by ioscheduler */
|
|
|
|
#define RQF_SOFTBARRIER ((__force req_flags_t)(1 << 3))
|
|
|
|
/* request for flush sequence */
|
|
|
|
#define RQF_FLUSH_SEQ ((__force req_flags_t)(1 << 4))
|
|
|
|
/* merge of different types, fail separately */
|
|
|
|
#define RQF_MIXED_MERGE ((__force req_flags_t)(1 << 5))
|
|
|
|
/* track inflight for MQ */
|
|
|
|
#define RQF_MQ_INFLIGHT ((__force req_flags_t)(1 << 6))
|
|
|
|
/* don't call prep for this one */
|
|
|
|
#define RQF_DONTPREP ((__force req_flags_t)(1 << 7))
|
|
|
|
/* vaguely specified driver internal error. Ignored by the block layer */
|
|
|
|
#define RQF_FAILED ((__force req_flags_t)(1 << 10))
|
|
|
|
/* don't warn about errors */
|
|
|
|
#define RQF_QUIET ((__force req_flags_t)(1 << 11))
|
|
|
|
/* elevator private data attached */
|
|
|
|
#define RQF_ELVPRIV ((__force req_flags_t)(1 << 12))
|
|
|
|
/* account into disk and partition IO statistics */
|
|
|
|
#define RQF_IO_STAT ((__force req_flags_t)(1 << 13))
|
|
|
|
/* runtime pm request */
|
|
|
|
#define RQF_PM ((__force req_flags_t)(1 << 15))
|
|
|
|
/* on IO scheduler merge hash */
|
|
|
|
#define RQF_HASHED ((__force req_flags_t)(1 << 16))
|
|
|
|
/* track IO completion time */
|
|
|
|
#define RQF_STATS ((__force req_flags_t)(1 << 17))
|
|
|
|
/* Look at ->special_vec for the actual data payload instead of the
|
|
|
|
bio chain. */
|
|
|
|
#define RQF_SPECIAL_PAYLOAD ((__force req_flags_t)(1 << 18))
|
|
|
|
/* The per-zone write lock is held for this request */
|
|
|
|
#define RQF_ZONE_WRITE_LOCKED ((__force req_flags_t)(1 << 19))
|
|
|
|
/* already slept for hybrid poll */
|
|
|
|
#define RQF_MQ_POLL_SLEPT ((__force req_flags_t)(1 << 20))
|
|
|
|
/* ->timeout has been called, don't expire again */
|
|
|
|
#define RQF_TIMED_OUT ((__force req_flags_t)(1 << 21))
|
2021-10-15 15:44:38 +00:00
|
|
|
/* queue has elevator attached */
|
|
|
|
#define RQF_ELV ((__force req_flags_t)(1 << 22))
|
2022-07-06 12:03:50 +00:00
|
|
|
#define RQF_RESV ((__force req_flags_t)(1 << 23))
|
2021-09-20 12:33:28 +00:00
|
|
|
|
|
|
|
/* flags that prevent us from merging requests: */
|
|
|
|
#define RQF_NOMERGE_FLAGS \
|
|
|
|
(RQF_STARTED | RQF_SOFTBARRIER | RQF_FLUSH_SEQ | RQF_SPECIAL_PAYLOAD)
|
|
|
|
|
|
|
|
enum mq_rq_state {
|
|
|
|
MQ_RQ_IDLE = 0,
|
|
|
|
MQ_RQ_IN_FLIGHT = 1,
|
|
|
|
MQ_RQ_COMPLETE = 2,
|
|
|
|
};
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Try to put the fields that are referenced together in the same cacheline.
|
|
|
|
*
|
|
|
|
* If you modify this structure, make sure to update blk_rq_init() and
|
|
|
|
* especially blk_mq_rq_ctx_init() to take care of the added fields.
|
|
|
|
*/
|
|
|
|
struct request {
|
|
|
|
struct request_queue *q;
|
|
|
|
struct blk_mq_ctx *mq_ctx;
|
|
|
|
struct blk_mq_hw_ctx *mq_hctx;
|
|
|
|
|
2022-07-14 18:06:32 +00:00
|
|
|
blk_opf_t cmd_flags; /* op and common flags */
|
2021-09-20 12:33:28 +00:00
|
|
|
req_flags_t rq_flags;
|
|
|
|
|
|
|
|
int tag;
|
|
|
|
int internal_tag;
|
|
|
|
|
2021-10-15 21:03:52 +00:00
|
|
|
unsigned int timeout;
|
|
|
|
|
2021-09-20 12:33:28 +00:00
|
|
|
/* the following two fields are internal, NEVER access directly */
|
|
|
|
unsigned int __data_len; /* total data len */
|
|
|
|
sector_t __sector; /* sector cursor */
|
|
|
|
|
|
|
|
struct bio *bio;
|
|
|
|
struct bio *biotail;
|
|
|
|
|
2021-10-06 12:34:11 +00:00
|
|
|
union {
|
|
|
|
struct list_head queuelist;
|
|
|
|
struct request *rq_next;
|
|
|
|
};
|
2021-09-20 12:33:28 +00:00
|
|
|
|
|
|
|
struct block_device *part;
|
|
|
|
#ifdef CONFIG_BLK_RQ_ALLOC_TIME
|
|
|
|
/* Time that the first bio started allocating this request. */
|
|
|
|
u64 alloc_time_ns;
|
|
|
|
#endif
|
|
|
|
/* Time that this request was allocated for this IO. */
|
|
|
|
u64 start_time_ns;
|
|
|
|
/* Time that I/O was submitted to the device. */
|
|
|
|
u64 io_start_time_ns;
|
|
|
|
|
|
|
|
#ifdef CONFIG_BLK_WBT
|
|
|
|
unsigned short wbt_flags;
|
|
|
|
#endif
|
|
|
|
/*
|
|
|
|
* rq sectors used for blk stats. It has the same value
|
|
|
|
* with blk_rq_sectors(rq), except that it never be zeroed
|
|
|
|
* by completion.
|
|
|
|
*/
|
|
|
|
unsigned short stats_sectors;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Number of scatter-gather DMA addr+len pairs after
|
|
|
|
* physical address coalescing is performed.
|
|
|
|
*/
|
|
|
|
unsigned short nr_phys_segments;
|
|
|
|
|
|
|
|
#ifdef CONFIG_BLK_DEV_INTEGRITY
|
|
|
|
unsigned short nr_integrity_segments;
|
|
|
|
#endif
|
|
|
|
|
|
|
|
#ifdef CONFIG_BLK_INLINE_ENCRYPTION
|
|
|
|
struct bio_crypt_ctx *crypt_ctx;
|
blk-crypto: rename blk_keyslot_manager to blk_crypto_profile
blk_keyslot_manager is misnamed because it doesn't necessarily manage
keyslots. It actually does several different things:
- Contains the crypto capabilities of the device.
- Provides functions to control the inline encryption hardware.
Originally these were just for programming/evicting keyslots;
however, new functionality (hardware-wrapped keys) will require new
functions here which are unrelated to keyslots. Moreover,
device-mapper devices already (ab)use "keyslot_evict" to pass key
eviction requests to their underlying devices even though
device-mapper devices don't have any keyslots themselves (so it
really should be "evict_key", not "keyslot_evict").
- Sometimes (but not always!) it manages keyslots. Originally it
always did, but device-mapper devices don't have keyslots
themselves, so they use a "passthrough keyslot manager" which
doesn't actually manage keyslots. This hack works, but the
terminology is unnatural. Also, some hardware doesn't have keyslots
and thus also uses a "passthrough keyslot manager" (support for such
hardware is yet to be upstreamed, but it will happen eventually).
Let's stop having keyslot managers which don't actually manage keyslots.
Instead, rename blk_keyslot_manager to blk_crypto_profile.
This is a fairly big change, since for consistency it also has to update
keyslot manager-related function names, variable names, and comments --
not just the actual struct name. However it's still a fairly
straightforward change, as it doesn't change any actual functionality.
Acked-by: Ulf Hansson <ulf.hansson@linaro.org> # For MMC
Reviewed-by: Mike Snitzer <snitzer@redhat.com>
Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Link: https://lore.kernel.org/r/20211018180453.40441-4-ebiggers@kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2021-10-18 18:04:52 +00:00
|
|
|
struct blk_crypto_keyslot *crypt_keyslot;
|
2021-09-20 12:33:28 +00:00
|
|
|
#endif
|
|
|
|
|
|
|
|
unsigned short ioprio;
|
|
|
|
|
|
|
|
enum mq_rq_state state;
|
2021-10-14 20:39:59 +00:00
|
|
|
atomic_t ref;
|
2021-09-20 12:33:28 +00:00
|
|
|
|
|
|
|
unsigned long deadline;
|
|
|
|
|
2021-10-15 21:03:52 +00:00
|
|
|
/*
|
|
|
|
* The hash is used inside the scheduler, and killed once the
|
|
|
|
* request reaches the dispatch list. The ipi_list is only used
|
|
|
|
* to queue the request for softirq completion, which is long
|
|
|
|
* after the request has been unhashed (and even removed from
|
|
|
|
* the dispatch list).
|
|
|
|
*/
|
|
|
|
union {
|
|
|
|
struct hlist_node hash; /* merge hash */
|
|
|
|
struct llist_node ipi_list;
|
|
|
|
};
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The rb_node is only used inside the io scheduler, requests
|
|
|
|
* are pruned when moved to the dispatch queue. So let the
|
|
|
|
* completion_data share space with the rb_node.
|
|
|
|
*/
|
|
|
|
union {
|
|
|
|
struct rb_node rb_node; /* sort/lookup */
|
|
|
|
struct bio_vec special_vec;
|
|
|
|
void *completion_data;
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Three pointers are available for the IO schedulers, if they need
|
|
|
|
* more they have to dynamically allocate it. Flush requests are
|
|
|
|
* never put on the IO scheduler. So let the flush fields share
|
|
|
|
* space with the elevator data.
|
|
|
|
*/
|
|
|
|
union {
|
|
|
|
struct {
|
|
|
|
struct io_cq *icq;
|
|
|
|
void *priv[2];
|
|
|
|
} elv;
|
|
|
|
|
|
|
|
struct {
|
|
|
|
unsigned int seq;
|
|
|
|
struct list_head list;
|
|
|
|
rq_end_io_fn *saved_end_io;
|
|
|
|
} flush;
|
|
|
|
};
|
|
|
|
|
2021-09-20 12:33:28 +00:00
|
|
|
union {
|
|
|
|
struct __call_single_data csd;
|
|
|
|
u64 fifo_time;
|
|
|
|
};
|
|
|
|
|
|
|
|
/*
|
|
|
|
* completion callback.
|
|
|
|
*/
|
|
|
|
rq_end_io_fn *end_io;
|
|
|
|
void *end_io_data;
|
|
|
|
};
|
|
|
|
|
2022-07-14 18:06:30 +00:00
|
|
|
static inline enum req_op req_op(const struct request *req)
|
|
|
|
{
|
|
|
|
return req->cmd_flags & REQ_OP_MASK;
|
|
|
|
}
|
2021-09-20 12:33:28 +00:00
|
|
|
|
|
|
|
static inline bool blk_rq_is_passthrough(struct request *rq)
|
|
|
|
{
|
|
|
|
return blk_op_is_passthrough(req_op(rq));
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline unsigned short req_get_ioprio(struct request *req)
|
|
|
|
{
|
|
|
|
return req->ioprio;
|
|
|
|
}
|
|
|
|
|
|
|
|
#define rq_data_dir(rq) (op_is_write(req_op(rq)) ? WRITE : READ)
|
|
|
|
|
|
|
|
#define rq_dma_dir(rq) \
|
|
|
|
(op_is_write(req_op(rq)) ? DMA_TO_DEVICE : DMA_FROM_DEVICE)
|
|
|
|
|
2022-01-05 17:05:15 +00:00
|
|
|
#define rq_list_add(listptr, rq) do { \
|
|
|
|
(rq)->rq_next = *(listptr); \
|
|
|
|
*(listptr) = rq; \
|
|
|
|
} while (0)
|
|
|
|
|
|
|
|
#define rq_list_pop(listptr) \
|
|
|
|
({ \
|
|
|
|
struct request *__req = NULL; \
|
|
|
|
if ((listptr) && *(listptr)) { \
|
|
|
|
__req = *(listptr); \
|
|
|
|
*(listptr) = __req->rq_next; \
|
|
|
|
} \
|
|
|
|
__req; \
|
|
|
|
})
|
|
|
|
|
|
|
|
#define rq_list_peek(listptr) \
|
|
|
|
({ \
|
|
|
|
struct request *__req = NULL; \
|
|
|
|
if ((listptr) && *(listptr)) \
|
|
|
|
__req = *(listptr); \
|
|
|
|
__req; \
|
|
|
|
})
|
|
|
|
|
|
|
|
#define rq_list_for_each(listptr, pos) \
|
|
|
|
for (pos = rq_list_peek((listptr)); pos; pos = rq_list_next(pos))
|
|
|
|
|
2022-01-05 17:05:16 +00:00
|
|
|
#define rq_list_for_each_safe(listptr, pos, nxt) \
|
|
|
|
for (pos = rq_list_peek((listptr)), nxt = rq_list_next(pos); \
|
|
|
|
pos; pos = nxt, nxt = pos ? rq_list_next(pos) : NULL)
|
|
|
|
|
2022-01-05 17:05:15 +00:00
|
|
|
#define rq_list_next(rq) (rq)->rq_next
|
|
|
|
#define rq_list_empty(list) ((list) == (struct request *) NULL)
|
|
|
|
|
2022-01-05 17:05:17 +00:00
|
|
|
/**
|
|
|
|
* rq_list_move() - move a struct request from one list to another
|
|
|
|
* @src: The source list @rq is currently in
|
|
|
|
* @dst: The destination list that @rq will be appended to
|
|
|
|
* @rq: The request to move
|
|
|
|
* @prev: The request preceding @rq in @src (NULL if @rq is the head)
|
|
|
|
*/
|
2022-01-07 00:52:28 +00:00
|
|
|
static inline void rq_list_move(struct request **src, struct request **dst,
|
2022-01-05 17:05:17 +00:00
|
|
|
struct request *rq, struct request *prev)
|
|
|
|
{
|
|
|
|
if (prev)
|
|
|
|
prev->rq_next = rq->rq_next;
|
|
|
|
else
|
|
|
|
*src = rq->rq_next;
|
|
|
|
rq_list_add(dst, rq);
|
|
|
|
}
|
|
|
|
|
2022-09-20 20:06:26 +00:00
|
|
|
/**
|
|
|
|
* enum blk_eh_timer_return - How the timeout handler should proceed
|
|
|
|
* @BLK_EH_DONE: The block driver completed the command or will complete it at
|
|
|
|
* a later time.
|
|
|
|
* @BLK_EH_RESET_TIMER: Reset the request timer and continue waiting for the
|
|
|
|
* request to complete.
|
|
|
|
*/
|
2021-09-20 12:33:28 +00:00
|
|
|
enum blk_eh_timer_return {
|
2022-09-20 20:06:26 +00:00
|
|
|
BLK_EH_DONE,
|
|
|
|
BLK_EH_RESET_TIMER,
|
2021-09-20 12:33:28 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
#define BLK_TAG_ALLOC_FIFO 0 /* allocate starting from 0 */
|
|
|
|
#define BLK_TAG_ALLOC_RR 1 /* allocate starting from last allocated tag */
|
|
|
|
|
2018-04-20 08:29:51 +00:00
|
|
|
/**
|
2019-10-22 00:07:24 +00:00
|
|
|
* struct blk_mq_hw_ctx - State for a hardware queue facing the hardware
|
|
|
|
* block device
|
2018-04-20 08:29:51 +00:00
|
|
|
*/
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
struct blk_mq_hw_ctx {
|
|
|
|
struct {
|
2019-10-22 00:07:24 +00:00
|
|
|
/** @lock: Protects the dispatch list. */
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
spinlock_t lock;
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @dispatch: Used for requests that are ready to be
|
|
|
|
* dispatched to the hardware but for some reason (e.g. lack of
|
|
|
|
* resources) could not be sent to the hardware. As soon as the
|
|
|
|
* driver can send new requests, requests at this list will
|
|
|
|
* be sent first for a fairer dispatch.
|
|
|
|
*/
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
struct list_head dispatch;
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @state: BLK_MQ_S_* flags. Defines the state of the hw
|
|
|
|
* queue (active, scheduled to restart, stopped).
|
|
|
|
*/
|
|
|
|
unsigned long state;
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
} ____cacheline_aligned_in_smp;
|
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @run_work: Used for scheduling a hardware queue run at a later time.
|
|
|
|
*/
|
2017-04-10 15:54:54 +00:00
|
|
|
struct delayed_work run_work;
|
2019-10-22 00:07:24 +00:00
|
|
|
/** @cpumask: Map of available CPUs where this hctx can run. */
|
2014-04-09 16:18:23 +00:00
|
|
|
cpumask_var_t cpumask;
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @next_cpu: Used by blk_mq_hctx_next_cpu() for round-robin CPU
|
|
|
|
* selection from @cpumask.
|
|
|
|
*/
|
2014-05-07 16:26:44 +00:00
|
|
|
int next_cpu;
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @next_cpu_batch: Counter of how many works left in the batch before
|
|
|
|
* changing to the next CPU.
|
|
|
|
*/
|
2014-05-07 16:26:44 +00:00
|
|
|
int next_cpu_batch;
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/** @flags: BLK_MQ_F_* flags. Defines the behaviour of the queue. */
|
|
|
|
unsigned long flags;
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @sched_data: Pointer owned by the IO scheduler attached to a request
|
|
|
|
* queue. It's up to the IO scheduler how to use this pointer.
|
|
|
|
*/
|
2017-01-17 13:03:22 +00:00
|
|
|
void *sched_data;
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @queue: Pointer to the request queue that owns this hardware context.
|
|
|
|
*/
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
struct request_queue *queue;
|
2019-10-22 00:07:24 +00:00
|
|
|
/** @fq: Queue of requests that need to perform a flush operation. */
|
2014-09-25 15:23:47 +00:00
|
|
|
struct blk_flush_queue *fq;
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @driver_data: Pointer to data owned by the block driver that created
|
|
|
|
* this hctx
|
|
|
|
*/
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
void *driver_data;
|
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @ctx_map: Bitmap for each software queue. If bit is on, there is a
|
|
|
|
* pending request in that software queue.
|
|
|
|
*/
|
2016-09-17 14:38:44 +00:00
|
|
|
struct sbitmap ctx_map;
|
2014-05-19 15:23:55 +00:00
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @dispatch_from: Software queue to be used when no scheduler was
|
|
|
|
* selected.
|
|
|
|
*/
|
blk-mq-sched: improve dispatching from sw queue
SCSI devices use host-wide tagset, and the shared driver tag space is
often quite big. However, there is also a queue depth for each lun(
.cmd_per_lun), which is often small, for example, on both lpfc and
qla2xxx, .cmd_per_lun is just 3.
So lots of requests may stay in sw queue, and we always flush all
belonging to same hw queue and dispatch them all to driver.
Unfortunately it is easy to cause queue busy because of the small
.cmd_per_lun. Once these requests are flushed out, they have to stay in
hctx->dispatch, and no bio merge can happen on these requests, and
sequential IO performance is harmed.
This patch introduces blk_mq_dequeue_from_ctx for dequeuing a request
from a sw queue, so that we can dispatch them in scheduler's way. We can
then avoid dequeueing too many requests from sw queue, since we don't
flush ->dispatch completely.
This patch improves dispatching from sw queue by using the .get_budget
and .put_budget callbacks.
Reviewed-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-10-14 09:22:30 +00:00
|
|
|
struct blk_mq_ctx *dispatch_from;
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @dispatch_busy: Number used by blk_mq_update_dispatch_busy() to
|
|
|
|
* decide if the hw_queue is busy using Exponential Weighted Moving
|
|
|
|
* Average algorithm.
|
|
|
|
*/
|
2018-07-03 15:03:16 +00:00
|
|
|
unsigned int dispatch_busy;
|
blk-mq-sched: improve dispatching from sw queue
SCSI devices use host-wide tagset, and the shared driver tag space is
often quite big. However, there is also a queue depth for each lun(
.cmd_per_lun), which is often small, for example, on both lpfc and
qla2xxx, .cmd_per_lun is just 3.
So lots of requests may stay in sw queue, and we always flush all
belonging to same hw queue and dispatch them all to driver.
Unfortunately it is easy to cause queue busy because of the small
.cmd_per_lun. Once these requests are flushed out, they have to stay in
hctx->dispatch, and no bio merge can happen on these requests, and
sequential IO performance is harmed.
This patch introduces blk_mq_dequeue_from_ctx for dequeuing a request
from a sw queue, so that we can dispatch them in scheduler's way. We can
then avoid dequeueing too many requests from sw queue, since we don't
flush ->dispatch completely.
This patch improves dispatching from sw queue by using the .get_budget
and .put_budget callbacks.
Reviewed-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-10-14 09:22:30 +00:00
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/** @type: HCTX_TYPE_* flags. Type of hardware queue. */
|
2018-10-29 19:13:29 +00:00
|
|
|
unsigned short type;
|
2019-10-22 00:07:24 +00:00
|
|
|
/** @nr_ctx: Number of software queues. */
|
2018-10-29 19:13:29 +00:00
|
|
|
unsigned short nr_ctx;
|
2019-10-22 00:07:24 +00:00
|
|
|
/** @ctxs: Array of software queues. */
|
2018-07-03 15:03:16 +00:00
|
|
|
struct blk_mq_ctx **ctxs;
|
2014-05-09 15:36:49 +00:00
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/** @dispatch_wait_lock: Lock for dispatch_wait queue. */
|
2018-06-25 11:31:47 +00:00
|
|
|
spinlock_t dispatch_wait_lock;
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @dispatch_wait: Waitqueue to put requests when there is no tag
|
|
|
|
* available at the moment, to wait for another try in the future.
|
|
|
|
*/
|
2017-11-09 15:32:43 +00:00
|
|
|
wait_queue_entry_t dispatch_wait;
|
2019-10-22 00:07:24 +00:00
|
|
|
|
|
|
|
/**
|
|
|
|
* @wait_index: Index of next available dispatch_wait queue to insert
|
|
|
|
* requests.
|
|
|
|
*/
|
2014-06-18 05:12:35 +00:00
|
|
|
atomic_t wait_index;
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @tags: Tags owned by the block driver. A tag at this set is only
|
|
|
|
* assigned when a request is dispatched from a hardware queue.
|
|
|
|
*/
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
struct blk_mq_tags *tags;
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @sched_tags: Tags owned by I/O scheduler. If there is an I/O
|
|
|
|
* scheduler associated with a request queue, a tag is assigned when
|
|
|
|
* that request is allocated. Else, this member is not used.
|
|
|
|
*/
|
2017-01-17 13:03:22 +00:00
|
|
|
struct blk_mq_tags *sched_tags;
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/** @queued: Number of queued requests. */
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
unsigned long queued;
|
2019-10-22 00:07:24 +00:00
|
|
|
/** @run: Number of dispatched requests. */
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
unsigned long run;
|
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/** @numa_node: NUMA node the storage adapter has been connected to. */
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
unsigned int numa_node;
|
2019-10-22 00:07:24 +00:00
|
|
|
/** @queue_num: Index of this hardware queue. */
|
2015-01-07 17:44:04 +00:00
|
|
|
unsigned int queue_num;
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @nr_active: Number of active requests. Only used when a tag set is
|
|
|
|
* shared across request queues.
|
|
|
|
*/
|
2014-05-13 21:10:52 +00:00
|
|
|
atomic_t nr_active;
|
|
|
|
|
2020-05-29 13:53:15 +00:00
|
|
|
/** @cpuhp_online: List to store request if CPU is going to die */
|
|
|
|
struct hlist_node cpuhp_online;
|
2019-10-22 00:07:24 +00:00
|
|
|
/** @cpuhp_dead: List to store request if some CPU die. */
|
2016-09-22 14:05:17 +00:00
|
|
|
struct hlist_node cpuhp_dead;
|
2019-10-22 00:07:24 +00:00
|
|
|
/** @kobj: Kernel object for sysfs. */
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
struct kobject kobj;
|
2015-11-05 17:44:55 +00:00
|
|
|
|
2017-05-04 14:17:21 +00:00
|
|
|
#ifdef CONFIG_BLK_DEBUG_FS
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @debugfs_dir: debugfs directory for this hardware queue. Named
|
|
|
|
* as cpu<cpu_number>.
|
|
|
|
*/
|
2017-05-04 14:17:21 +00:00
|
|
|
struct dentry *debugfs_dir;
|
2019-10-22 00:07:24 +00:00
|
|
|
/** @sched_debugfs_dir: debugfs directory for the scheduler. */
|
2017-05-04 14:24:40 +00:00
|
|
|
struct dentry *sched_debugfs_dir;
|
2017-05-04 14:17:21 +00:00
|
|
|
#endif
|
2017-06-20 18:15:38 +00:00
|
|
|
|
2020-03-10 04:26:16 +00:00
|
|
|
/**
|
|
|
|
* @hctx_list: if this hctx is not in use, this is an entry in
|
|
|
|
* q->unused_hctx_list.
|
|
|
|
*/
|
blk-mq: always free hctx after request queue is freed
In normal queue cleanup path, hctx is released after request queue
is freed, see blk_mq_release().
However, in __blk_mq_update_nr_hw_queues(), hctx may be freed because
of hw queues shrinking. This way is easy to cause use-after-free,
because: one implicit rule is that it is safe to call almost all block
layer APIs if the request queue is alive; and one hctx may be retrieved
by one API, then the hctx can be freed by blk_mq_update_nr_hw_queues();
finally use-after-free is triggered.
Fixes this issue by always freeing hctx after releasing request queue.
If some hctxs are removed in blk_mq_update_nr_hw_queues(), introduce
a per-queue list to hold them, then try to resuse these hctxs if numa
node is matched.
Cc: Dongli Zhang <dongli.zhang@oracle.com>
Cc: James Smart <james.smart@broadcom.com>
Cc: Bart Van Assche <bart.vanassche@wdc.com>
Cc: linux-scsi@vger.kernel.org,
Cc: Martin K . Petersen <martin.petersen@oracle.com>,
Cc: Christoph Hellwig <hch@lst.de>,
Cc: James E . J . Bottomley <jejb@linux.vnet.ibm.com>,
Reviewed-by: Hannes Reinecke <hare@suse.com>
Tested-by: James Smart <james.smart@broadcom.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-04-30 01:52:27 +00:00
|
|
|
struct list_head hctx_list;
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
};
|
|
|
|
|
2019-09-30 23:00:45 +00:00
|
|
|
/**
|
2019-10-22 00:07:24 +00:00
|
|
|
* struct blk_mq_queue_map - Map software queues to hardware queues
|
2019-09-30 23:00:45 +00:00
|
|
|
* @mq_map: CPU ID to hardware queue index map. This is an array
|
|
|
|
* with nr_cpu_ids elements. Each element has a value in the range
|
|
|
|
* [@queue_offset, @queue_offset + @nr_queues).
|
|
|
|
* @nr_queues: Number of hardware queues to map CPU IDs onto.
|
|
|
|
* @queue_offset: First hardware queue to map onto. Used by the PCIe NVMe
|
|
|
|
* driver to map each hardware queue type (enum hctx_type) onto a distinct
|
|
|
|
* set of hardware queues.
|
|
|
|
*/
|
2018-10-29 19:06:14 +00:00
|
|
|
struct blk_mq_queue_map {
|
|
|
|
unsigned int *mq_map;
|
|
|
|
unsigned int nr_queues;
|
2018-10-24 19:16:11 +00:00
|
|
|
unsigned int queue_offset;
|
2018-10-29 19:06:14 +00:00
|
|
|
};
|
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* enum hctx_type - Type of hardware queue
|
|
|
|
* @HCTX_TYPE_DEFAULT: All I/O not otherwise accounted for.
|
|
|
|
* @HCTX_TYPE_READ: Just for READ I/O.
|
|
|
|
* @HCTX_TYPE_POLL: Polled I/O of any kind.
|
|
|
|
* @HCTX_MAX_TYPES: Number of types of hctx.
|
|
|
|
*/
|
2018-12-02 16:46:16 +00:00
|
|
|
enum hctx_type {
|
2019-10-22 00:07:24 +00:00
|
|
|
HCTX_TYPE_DEFAULT,
|
|
|
|
HCTX_TYPE_READ,
|
|
|
|
HCTX_TYPE_POLL,
|
2018-12-02 16:46:16 +00:00
|
|
|
|
|
|
|
HCTX_MAX_TYPES,
|
2018-10-29 19:06:14 +00:00
|
|
|
};
|
|
|
|
|
2019-09-30 23:00:45 +00:00
|
|
|
/**
|
|
|
|
* struct blk_mq_tag_set - tag set that can be shared between request queues
|
2023-02-17 09:29:10 +00:00
|
|
|
* @ops: Pointers to functions that implement block driver behavior.
|
2019-09-30 23:00:45 +00:00
|
|
|
* @map: One or more ctx -> hctx mappings. One map exists for each
|
|
|
|
* hardware queue type (enum hctx_type) that the driver wishes
|
|
|
|
* to support. There are no restrictions on maps being of the
|
|
|
|
* same size, and it's perfectly legal to share maps between
|
|
|
|
* types.
|
|
|
|
* @nr_maps: Number of elements in the @map array. A number in the range
|
|
|
|
* [1, HCTX_MAX_TYPES].
|
|
|
|
* @nr_hw_queues: Number of hardware queues supported by the block driver that
|
|
|
|
* owns this data structure.
|
|
|
|
* @queue_depth: Number of tags per hardware queue, reserved tags included.
|
|
|
|
* @reserved_tags: Number of tags to set aside for BLK_MQ_REQ_RESERVED tag
|
|
|
|
* allocations.
|
|
|
|
* @cmd_size: Number of additional bytes to allocate per request. The block
|
|
|
|
* driver owns these additional bytes.
|
|
|
|
* @numa_node: NUMA node the storage adapter has been connected to.
|
|
|
|
* @timeout: Request processing timeout in jiffies.
|
|
|
|
* @flags: Zero or more BLK_MQ_F_* flags.
|
|
|
|
* @driver_data: Pointer to data owned by the block driver that created this
|
|
|
|
* tag set.
|
|
|
|
* @tags: Tag sets. One tag set per hardware queue. Has @nr_hw_queues
|
|
|
|
* elements.
|
2021-10-05 10:23:39 +00:00
|
|
|
* @shared_tags:
|
|
|
|
* Shared set of tags. Has @nr_hw_queues elements. If set,
|
|
|
|
* shared by all @tags.
|
2019-09-30 23:00:45 +00:00
|
|
|
* @tag_list_lock: Serializes tag_list accesses.
|
|
|
|
* @tag_list: List of the request queues that use this tag set. See also
|
|
|
|
* request_queue.tag_set_list.
|
2022-11-01 15:00:47 +00:00
|
|
|
* @srcu: Use as lock when type of the request queue is blocking
|
|
|
|
* (BLK_MQ_F_BLOCKING).
|
2019-09-30 23:00:45 +00:00
|
|
|
*/
|
2014-04-15 20:14:00 +00:00
|
|
|
struct blk_mq_tag_set {
|
2023-02-17 09:29:10 +00:00
|
|
|
const struct blk_mq_ops *ops;
|
2018-10-29 19:06:14 +00:00
|
|
|
struct blk_mq_queue_map map[HCTX_MAX_TYPES];
|
2019-09-30 23:00:45 +00:00
|
|
|
unsigned int nr_maps;
|
|
|
|
unsigned int nr_hw_queues;
|
|
|
|
unsigned int queue_depth;
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
unsigned int reserved_tags;
|
2019-09-30 23:00:45 +00:00
|
|
|
unsigned int cmd_size;
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
int numa_node;
|
|
|
|
unsigned int timeout;
|
2019-09-30 23:00:45 +00:00
|
|
|
unsigned int flags;
|
2014-04-15 20:14:00 +00:00
|
|
|
void *driver_data;
|
|
|
|
|
|
|
|
struct blk_mq_tags **tags;
|
2014-05-13 21:10:52 +00:00
|
|
|
|
2021-10-05 10:23:39 +00:00
|
|
|
struct blk_mq_tags *shared_tags;
|
2021-10-05 10:23:37 +00:00
|
|
|
|
2014-05-13 21:10:52 +00:00
|
|
|
struct mutex tag_list_lock;
|
|
|
|
struct list_head tag_list;
|
2022-11-01 15:00:47 +00:00
|
|
|
struct srcu_struct *srcu;
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
};
|
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* struct blk_mq_queue_data - Data about a request inserted in a queue
|
|
|
|
*
|
|
|
|
* @rq: Request pointer.
|
|
|
|
* @last: If it is the last request in the queue.
|
|
|
|
*/
|
2014-10-29 17:14:52 +00:00
|
|
|
struct blk_mq_queue_data {
|
|
|
|
struct request *rq;
|
|
|
|
bool last;
|
|
|
|
};
|
|
|
|
|
2022-07-06 12:03:53 +00:00
|
|
|
typedef bool (busy_tag_iter_fn)(struct request *, void *);
|
2015-11-05 17:44:55 +00:00
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* struct blk_mq_ops - Callback functions that implements block driver
|
|
|
|
* behaviour.
|
|
|
|
*/
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
struct blk_mq_ops {
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @queue_rq: Queue a new request from block IO.
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
*/
|
2020-07-28 16:20:36 +00:00
|
|
|
blk_status_t (*queue_rq)(struct blk_mq_hw_ctx *,
|
|
|
|
const struct blk_mq_queue_data *);
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @commit_rqs: If a driver uses bd->last to judge when to submit
|
|
|
|
* requests to hardware, it must define this function. In case of errors
|
|
|
|
* that make us stop issuing further requests, this hook serves the
|
2018-11-28 00:02:25 +00:00
|
|
|
* purpose of kicking the hardware (which the last request otherwise
|
|
|
|
* would have done).
|
|
|
|
*/
|
2020-07-28 16:20:36 +00:00
|
|
|
void (*commit_rqs)(struct blk_mq_hw_ctx *);
|
2018-11-28 00:02:25 +00:00
|
|
|
|
2021-12-03 13:48:53 +00:00
|
|
|
/**
|
|
|
|
* @queue_rqs: Queue a list of new requests. Driver is guaranteed
|
|
|
|
* that each request belongs to the same queue. If the driver doesn't
|
|
|
|
* empty the @rqlist completely, then the rest will be queued
|
|
|
|
* individually by the block layer upon return.
|
|
|
|
*/
|
|
|
|
void (*queue_rqs)(struct request **rqlist);
|
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @get_budget: Reserve budget before queue request, once .queue_rq is
|
2017-10-14 09:22:29 +00:00
|
|
|
* run, it is driver's responsibility to release the
|
|
|
|
* reserved budget. Also we have to handle failure case
|
|
|
|
* of .get_budget for avoiding I/O deadlock.
|
|
|
|
*/
|
2021-01-22 02:33:12 +00:00
|
|
|
int (*get_budget)(struct request_queue *);
|
2020-07-28 16:20:36 +00:00
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @put_budget: Release the reserved budget.
|
|
|
|
*/
|
2021-01-22 02:33:12 +00:00
|
|
|
void (*put_budget)(struct request_queue *, int);
|
2017-10-14 09:22:29 +00:00
|
|
|
|
2021-04-21 15:45:26 +00:00
|
|
|
/**
|
|
|
|
* @set_rq_budget_token: store rq's budget token
|
2021-01-22 02:33:11 +00:00
|
|
|
*/
|
|
|
|
void (*set_rq_budget_token)(struct request *, int);
|
2021-04-21 15:45:26 +00:00
|
|
|
/**
|
|
|
|
* @get_rq_budget_token: retrieve rq's budget token
|
2021-01-22 02:33:11 +00:00
|
|
|
*/
|
|
|
|
int (*get_rq_budget_token)(struct request *);
|
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @timeout: Called on request timeout.
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
*/
|
2022-07-06 12:03:51 +00:00
|
|
|
enum blk_eh_timer_return (*timeout)(struct request *);
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @poll: Called to poll for completion of a specific tag.
|
2015-11-05 17:44:55 +00:00
|
|
|
*/
|
2021-10-12 15:24:29 +00:00
|
|
|
int (*poll)(struct blk_mq_hw_ctx *, struct io_comp_batch *);
|
2015-11-05 17:44:55 +00:00
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @complete: Mark the request as complete.
|
|
|
|
*/
|
2020-07-28 16:20:36 +00:00
|
|
|
void (*complete)(struct request *);
|
2014-02-10 11:24:38 +00:00
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @init_hctx: Called when the block layer side of a hardware queue has
|
|
|
|
* been set up, allowing the driver to allocate/init matching
|
|
|
|
* structures.
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
*/
|
2020-07-28 16:20:36 +00:00
|
|
|
int (*init_hctx)(struct blk_mq_hw_ctx *, void *, unsigned int);
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @exit_hctx: Ditto for exit/teardown.
|
|
|
|
*/
|
2020-07-28 16:20:36 +00:00
|
|
|
void (*exit_hctx)(struct blk_mq_hw_ctx *, unsigned int);
|
2014-04-15 19:59:10 +00:00
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @init_request: Called for every command allocated by the block layer
|
|
|
|
* to allow the driver to set up driver specific data.
|
2014-09-25 15:23:47 +00:00
|
|
|
*
|
|
|
|
* Tag greater than or equal to queue_depth is for setting up
|
|
|
|
* flush request.
|
2014-04-15 19:59:10 +00:00
|
|
|
*/
|
2020-07-28 16:20:36 +00:00
|
|
|
int (*init_request)(struct blk_mq_tag_set *set, struct request *,
|
|
|
|
unsigned int, unsigned int);
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @exit_request: Ditto for exit/teardown.
|
|
|
|
*/
|
2020-07-28 16:20:36 +00:00
|
|
|
void (*exit_request)(struct blk_mq_tag_set *set, struct request *,
|
|
|
|
unsigned int);
|
2019-10-22 00:07:24 +00:00
|
|
|
|
|
|
|
/**
|
|
|
|
* @cleanup_rq: Called before freeing one request which isn't completed
|
|
|
|
* yet, and usually for freeing the driver private data.
|
2019-07-25 02:04:59 +00:00
|
|
|
*/
|
2020-07-28 16:20:36 +00:00
|
|
|
void (*cleanup_rq)(struct request *);
|
2019-07-25 02:04:59 +00:00
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @busy: If set, returns whether or not this queue currently is busy.
|
2018-10-29 16:15:10 +00:00
|
|
|
*/
|
2020-07-28 16:20:36 +00:00
|
|
|
bool (*busy)(struct request_queue *);
|
2018-10-29 16:15:10 +00:00
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @map_queues: This allows drivers specify their own queue mapping by
|
|
|
|
* overriding the setup-time function that builds the mq_map.
|
|
|
|
*/
|
2022-08-15 17:00:43 +00:00
|
|
|
void (*map_queues)(struct blk_mq_tag_set *set);
|
2017-04-26 20:47:56 +00:00
|
|
|
|
|
|
|
#ifdef CONFIG_BLK_DEBUG_FS
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* @show_rq: Used by the debugfs implementation to show driver-specific
|
2017-04-26 20:47:56 +00:00
|
|
|
* information about a request.
|
|
|
|
*/
|
|
|
|
void (*show_rq)(struct seq_file *m, struct request *rq);
|
|
|
|
#endif
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
enum {
|
|
|
|
BLK_MQ_F_SHOULD_MERGE = 1 << 0,
|
2020-08-19 15:20:19 +00:00
|
|
|
BLK_MQ_F_TAG_QUEUE_SHARED = 1 << 1,
|
2020-05-29 13:53:15 +00:00
|
|
|
/*
|
|
|
|
* Set when this device requires underlying blk-mq device for
|
|
|
|
* completing IO:
|
|
|
|
*/
|
|
|
|
BLK_MQ_F_STACKING = 1 << 2,
|
blk-mq: Facilitate a shared sbitmap per tagset
Some SCSI HBAs (such as HPSA, megaraid, mpt3sas, hisi_sas_v3 ..) support
multiple reply queues with single hostwide tags.
In addition, these drivers want to use interrupt assignment in
pci_alloc_irq_vectors(PCI_IRQ_AFFINITY). However, as discussed in [0],
CPU hotplug may cause in-flight IO completion to not be serviced when an
interrupt is shutdown. That problem is solved in commit bf0beec0607d
("blk-mq: drain I/O when all CPUs in a hctx are offline").
However, to take advantage of that blk-mq feature, the HBA HW queuess are
required to be mapped to that of the blk-mq hctx's; to do that, the HBA HW
queues need to be exposed to the upper layer.
In making that transition, the per-SCSI command request tags are no
longer unique per Scsi host - they are just unique per hctx. As such, the
HBA LLDD would have to generate this tag internally, which has a certain
performance overhead.
However another problem is that blk-mq assumes the host may accept
(Scsi_host.can_queue * #hw queue) commands. In commit 6eb045e092ef ("scsi:
core: avoid host-wide host_busy counter for scsi_mq"), the Scsi host busy
counter was removed, which would stop the LLDD being sent more than
.can_queue commands; however, it should still be ensured that the block
layer does not issue more than .can_queue commands to the Scsi host.
To solve this problem, introduce a shared sbitmap per blk_mq_tag_set,
which may be requested at init time.
New flag BLK_MQ_F_TAG_HCTX_SHARED should be set when requesting the
tagset to indicate whether the shared sbitmap should be used.
Even when BLK_MQ_F_TAG_HCTX_SHARED is set, a full set of tags and requests
are still allocated per hctx; the reason for this is that if tags and
requests were only allocated for a single hctx - like hctx0 - it may break
block drivers which expect a request be associated with a specific hctx,
i.e. not always hctx0. This will introduce extra memory usage.
This change is based on work originally from Ming Lei in [1] and from
Bart's suggestion in [2].
[0] https://lore.kernel.org/linux-block/alpine.DEB.2.21.1904051331270.1802@nanos.tec.linutronix.de/
[1] https://lore.kernel.org/linux-block/20190531022801.10003-1-ming.lei@redhat.com/
[2] https://lore.kernel.org/linux-block/ff77beff-5fd9-9f05-12b6-826922bace1f@huawei.com/T/#m3db0a602f095cbcbff27e9c884d6b4ae826144be
Signed-off-by: John Garry <john.garry@huawei.com>
Tested-by: Don Brace<don.brace@microsemi.com> #SCSI resv cmds patches used
Tested-by: Douglas Gilbert <dgilbert@interlog.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-08-19 15:20:24 +00:00
|
|
|
BLK_MQ_F_TAG_HCTX_SHARED = 1 << 3,
|
2016-09-21 16:12:13 +00:00
|
|
|
BLK_MQ_F_BLOCKING = 1 << 5,
|
2021-08-05 17:41:59 +00:00
|
|
|
/* Do not allow an I/O scheduler to be configured. */
|
2017-01-13 21:43:58 +00:00
|
|
|
BLK_MQ_F_NO_SCHED = 1 << 6,
|
2021-08-05 17:41:59 +00:00
|
|
|
/*
|
|
|
|
* Select 'none' during queue registration in case of a single hwq
|
|
|
|
* or shared hwqs instead of 'mq-deadline'.
|
|
|
|
*/
|
|
|
|
BLK_MQ_F_NO_SCHED_BY_DEFAULT = 1 << 7,
|
2015-01-23 21:18:00 +00:00
|
|
|
BLK_MQ_F_ALLOC_POLICY_START_BIT = 8,
|
|
|
|
BLK_MQ_F_ALLOC_POLICY_BITS = 1,
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
|
2014-03-19 21:25:02 +00:00
|
|
|
BLK_MQ_S_STOPPED = 0,
|
2014-05-13 21:10:52 +00:00
|
|
|
BLK_MQ_S_TAG_ACTIVE = 1,
|
2017-01-17 13:03:22 +00:00
|
|
|
BLK_MQ_S_SCHED_RESTART = 2,
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
|
2020-05-29 13:53:15 +00:00
|
|
|
/* hw queue is inactive after all its CPUs become offline */
|
|
|
|
BLK_MQ_S_INACTIVE = 3,
|
|
|
|
|
2014-06-05 21:21:56 +00:00
|
|
|
BLK_MQ_MAX_DEPTH = 10240,
|
2014-05-07 16:26:44 +00:00
|
|
|
|
|
|
|
BLK_MQ_CPU_WORK_BATCH = 8,
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
};
|
2015-01-23 21:18:00 +00:00
|
|
|
#define BLK_MQ_FLAG_TO_ALLOC_POLICY(flags) \
|
|
|
|
((flags >> BLK_MQ_F_ALLOC_POLICY_START_BIT) & \
|
|
|
|
((1 << BLK_MQ_F_ALLOC_POLICY_BITS) - 1))
|
|
|
|
#define BLK_ALLOC_POLICY_TO_MQ_FLAG(policy) \
|
|
|
|
((policy & ((1 << BLK_MQ_F_ALLOC_POLICY_BITS) - 1)) \
|
|
|
|
<< BLK_MQ_F_ALLOC_POLICY_START_BIT)
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
|
2021-10-05 10:23:37 +00:00
|
|
|
#define BLK_MQ_NO_HCTX_IDX (-1U)
|
|
|
|
|
2021-08-16 13:19:05 +00:00
|
|
|
struct gendisk *__blk_mq_alloc_disk(struct blk_mq_tag_set *set, void *queuedata,
|
|
|
|
struct lock_class_key *lkclass);
|
2021-06-02 06:53:18 +00:00
|
|
|
#define blk_mq_alloc_disk(set, queuedata) \
|
|
|
|
({ \
|
|
|
|
static struct lock_class_key __key; \
|
|
|
|
\
|
2021-08-16 13:19:05 +00:00
|
|
|
__blk_mq_alloc_disk(set, queuedata, &__key); \
|
2021-06-02 06:53:18 +00:00
|
|
|
})
|
2022-06-19 06:05:51 +00:00
|
|
|
struct gendisk *blk_mq_alloc_disk_for_queue(struct request_queue *q,
|
|
|
|
struct lock_class_key *lkclass);
|
2014-04-15 20:14:00 +00:00
|
|
|
struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *);
|
2021-06-02 06:53:17 +00:00
|
|
|
int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
|
|
|
|
struct request_queue *q);
|
2022-06-19 06:05:51 +00:00
|
|
|
void blk_mq_destroy_queue(struct request_queue *);
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
|
2014-04-15 20:14:00 +00:00
|
|
|
int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set);
|
2021-06-02 06:53:16 +00:00
|
|
|
int blk_mq_alloc_sq_tag_set(struct blk_mq_tag_set *set,
|
|
|
|
const struct blk_mq_ops *ops, unsigned int queue_depth,
|
|
|
|
unsigned int set_flags);
|
2014-04-15 20:14:00 +00:00
|
|
|
void blk_mq_free_tag_set(struct blk_mq_tag_set *set);
|
|
|
|
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
void blk_mq_free_request(struct request *rq);
|
2015-11-26 08:13:05 +00:00
|
|
|
|
2018-12-18 04:11:17 +00:00
|
|
|
bool blk_mq_queue_inflight(struct request_queue *q);
|
2018-11-08 16:03:51 +00:00
|
|
|
|
2015-11-26 08:13:05 +00:00
|
|
|
enum {
|
2017-11-09 18:49:59 +00:00
|
|
|
/* return when out of requests */
|
|
|
|
BLK_MQ_REQ_NOWAIT = (__force blk_mq_req_flags_t)(1 << 0),
|
|
|
|
/* allocate from reserved pool */
|
|
|
|
BLK_MQ_REQ_RESERVED = (__force blk_mq_req_flags_t)(1 << 1),
|
2020-12-09 05:29:45 +00:00
|
|
|
/* set RQF_PM */
|
|
|
|
BLK_MQ_REQ_PM = (__force blk_mq_req_flags_t)(1 << 2),
|
2015-11-26 08:13:05 +00:00
|
|
|
};
|
|
|
|
|
2022-07-14 18:06:32 +00:00
|
|
|
struct request *blk_mq_alloc_request(struct request_queue *q, blk_opf_t opf,
|
2017-11-09 18:49:59 +00:00
|
|
|
blk_mq_req_flags_t flags);
|
2017-06-20 18:15:39 +00:00
|
|
|
struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
|
2022-07-14 18:06:32 +00:00
|
|
|
blk_opf_t opf, blk_mq_req_flags_t flags,
|
2017-11-09 18:49:59 +00:00
|
|
|
unsigned int hctx_idx);
|
2021-10-16 22:38:14 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Tag address space map.
|
|
|
|
*/
|
|
|
|
struct blk_mq_tags {
|
|
|
|
unsigned int nr_tags;
|
|
|
|
unsigned int nr_reserved_tags;
|
|
|
|
|
|
|
|
atomic_t active_queues;
|
|
|
|
|
|
|
|
struct sbitmap_queue bitmap_tags;
|
|
|
|
struct sbitmap_queue breserved_tags;
|
|
|
|
|
|
|
|
struct request **rqs;
|
|
|
|
struct request **static_rqs;
|
|
|
|
struct list_head page_list;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* used to clear request reference in rqs[] before freeing one
|
|
|
|
* request pool
|
|
|
|
*/
|
|
|
|
spinlock_t lock;
|
|
|
|
};
|
|
|
|
|
|
|
|
static inline struct request *blk_mq_tag_to_rq(struct blk_mq_tags *tags,
|
|
|
|
unsigned int tag)
|
|
|
|
{
|
|
|
|
if (tag < tags->nr_tags) {
|
|
|
|
prefetch(tags->rqs[tag]);
|
|
|
|
return tags->rqs[tag];
|
|
|
|
}
|
|
|
|
|
|
|
|
return NULL;
|
|
|
|
}
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
|
2014-10-30 13:45:11 +00:00
|
|
|
enum {
|
|
|
|
BLK_MQ_UNIQUE_TAG_BITS = 16,
|
|
|
|
BLK_MQ_UNIQUE_TAG_MASK = (1 << BLK_MQ_UNIQUE_TAG_BITS) - 1,
|
|
|
|
};
|
|
|
|
|
|
|
|
u32 blk_mq_unique_tag(struct request *rq);
|
|
|
|
|
|
|
|
static inline u16 blk_mq_unique_tag_to_hwq(u32 unique_tag)
|
|
|
|
{
|
|
|
|
return unique_tag >> BLK_MQ_UNIQUE_TAG_BITS;
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline u16 blk_mq_unique_tag_to_tag(u32 unique_tag)
|
|
|
|
{
|
|
|
|
return unique_tag & BLK_MQ_UNIQUE_TAG_MASK;
|
|
|
|
}
|
|
|
|
|
2019-09-30 08:25:49 +00:00
|
|
|
/**
|
|
|
|
* blk_mq_rq_state() - read the current MQ_RQ_* state of a request
|
|
|
|
* @rq: target request.
|
|
|
|
*/
|
|
|
|
static inline enum mq_rq_state blk_mq_rq_state(struct request *rq)
|
|
|
|
{
|
|
|
|
return READ_ONCE(rq->state);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline int blk_mq_request_started(struct request *rq)
|
|
|
|
{
|
|
|
|
return blk_mq_rq_state(rq) != MQ_RQ_IDLE;
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline int blk_mq_request_completed(struct request *rq)
|
|
|
|
{
|
|
|
|
return blk_mq_rq_state(rq) == MQ_RQ_COMPLETE;
|
|
|
|
}
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
|
2021-02-01 03:49:38 +00:00
|
|
|
/*
|
|
|
|
*
|
|
|
|
* Set the state to complete when completing a request from inside ->queue_rq.
|
|
|
|
* This is used by drivers that want to ensure special complete actions that
|
|
|
|
* need access to the request are called on failure, e.g. by nvme for
|
|
|
|
* multipathing.
|
|
|
|
*/
|
|
|
|
static inline void blk_mq_set_request_complete(struct request *rq)
|
|
|
|
{
|
|
|
|
WRITE_ONCE(rq->state, MQ_RQ_COMPLETE);
|
|
|
|
}
|
|
|
|
|
2021-10-25 07:06:57 +00:00
|
|
|
/*
|
|
|
|
* Complete the request directly instead of deferring it to softirq or
|
|
|
|
* completing it another CPU. Useful in preemptible instead of an interrupt.
|
|
|
|
*/
|
|
|
|
static inline void blk_mq_complete_request_direct(struct request *rq,
|
|
|
|
void (*complete)(struct request *rq))
|
|
|
|
{
|
|
|
|
WRITE_ONCE(rq->state, MQ_RQ_COMPLETE);
|
|
|
|
complete(rq);
|
|
|
|
}
|
|
|
|
|
2014-09-13 23:40:09 +00:00
|
|
|
void blk_mq_start_request(struct request *rq);
|
2017-06-03 07:38:04 +00:00
|
|
|
void blk_mq_end_request(struct request *rq, blk_status_t error);
|
|
|
|
void __blk_mq_end_request(struct request *rq, blk_status_t error);
|
2021-10-08 11:50:46 +00:00
|
|
|
void blk_mq_end_request_batch(struct io_comp_batch *ib);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Only need start/end time stamping if we have iostat or
|
|
|
|
* blk stats enabled, or using an IO scheduler.
|
|
|
|
*/
|
|
|
|
static inline bool blk_mq_need_time_stamp(struct request *rq)
|
|
|
|
{
|
|
|
|
return (rq->rq_flags & (RQF_IO_STAT | RQF_STATS | RQF_ELV));
|
|
|
|
}
|
|
|
|
|
2022-07-06 12:03:50 +00:00
|
|
|
static inline bool blk_mq_is_reserved_rq(struct request *rq)
|
|
|
|
{
|
|
|
|
return rq->rq_flags & RQF_RESV;
|
|
|
|
}
|
|
|
|
|
2021-10-08 11:50:46 +00:00
|
|
|
/*
|
|
|
|
* Batched completions only work when there is no I/O error and no special
|
|
|
|
* ->end_io handler.
|
|
|
|
*/
|
|
|
|
static inline bool blk_mq_add_to_batch(struct request *req,
|
|
|
|
struct io_comp_batch *iob, int ioerror,
|
|
|
|
void (*complete)(struct io_comp_batch *))
|
|
|
|
{
|
2022-10-27 08:57:09 +00:00
|
|
|
if (!iob || (req->rq_flags & RQF_ELV) || ioerror ||
|
|
|
|
(req->end_io && !blk_rq_is_passthrough(req)))
|
2021-10-08 11:50:46 +00:00
|
|
|
return false;
|
2022-09-21 14:24:16 +00:00
|
|
|
|
2021-10-08 11:50:46 +00:00
|
|
|
if (!iob->complete)
|
|
|
|
iob->complete = complete;
|
|
|
|
else if (iob->complete != complete)
|
|
|
|
return false;
|
|
|
|
iob->need_ts |= blk_mq_need_time_stamp(req);
|
|
|
|
rq_list_add(&iob->req_list, req);
|
|
|
|
return true;
|
|
|
|
}
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
|
2016-10-29 00:21:41 +00:00
|
|
|
void blk_mq_requeue_request(struct request *rq, bool kick_requeue_list);
|
2014-05-28 14:08:02 +00:00
|
|
|
void blk_mq_kick_requeue_list(struct request_queue *q);
|
2016-09-14 17:28:30 +00:00
|
|
|
void blk_mq_delay_kick_requeue_list(struct request_queue *q, unsigned long msecs);
|
2020-06-11 06:44:47 +00:00
|
|
|
void blk_mq_complete_request(struct request *rq);
|
2020-06-11 06:44:50 +00:00
|
|
|
bool blk_mq_complete_request_remote(struct request *rq);
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
void blk_mq_stop_hw_queue(struct blk_mq_hw_ctx *hctx);
|
|
|
|
void blk_mq_start_hw_queue(struct blk_mq_hw_ctx *hctx);
|
2013-10-25 13:45:58 +00:00
|
|
|
void blk_mq_stop_hw_queues(struct request_queue *q);
|
2014-04-16 07:44:56 +00:00
|
|
|
void blk_mq_start_hw_queues(struct request_queue *q);
|
2016-12-08 20:19:30 +00:00
|
|
|
void blk_mq_start_stopped_hw_queue(struct blk_mq_hw_ctx *hctx, bool async);
|
2014-04-16 07:44:54 +00:00
|
|
|
void blk_mq_start_stopped_hw_queues(struct request_queue *q, bool async);
|
2017-06-06 15:22:01 +00:00
|
|
|
void blk_mq_quiesce_queue(struct request_queue *q);
|
2022-11-01 15:00:48 +00:00
|
|
|
void blk_mq_wait_quiesce_done(struct blk_mq_tag_set *set);
|
2022-11-01 15:00:49 +00:00
|
|
|
void blk_mq_quiesce_tagset(struct blk_mq_tag_set *set);
|
|
|
|
void blk_mq_unquiesce_tagset(struct blk_mq_tag_set *set);
|
2017-06-06 15:22:03 +00:00
|
|
|
void blk_mq_unquiesce_queue(struct request_queue *q);
|
2017-04-07 18:16:52 +00:00
|
|
|
void blk_mq_delay_run_hw_queue(struct blk_mq_hw_ctx *hctx, unsigned long msecs);
|
2019-10-29 16:59:30 +00:00
|
|
|
void blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async);
|
2015-03-12 03:56:38 +00:00
|
|
|
void blk_mq_run_hw_queues(struct request_queue *q, bool async);
|
2020-04-20 16:24:52 +00:00
|
|
|
void blk_mq_delay_run_hw_queues(struct request_queue *q, unsigned long msecs);
|
2016-03-10 11:58:46 +00:00
|
|
|
void blk_mq_tagset_busy_iter(struct blk_mq_tag_set *tagset,
|
|
|
|
busy_tag_iter_fn *fn, void *priv);
|
2019-07-24 03:48:40 +00:00
|
|
|
void blk_mq_tagset_wait_completed_request(struct blk_mq_tag_set *tagset);
|
2015-01-02 22:05:12 +00:00
|
|
|
void blk_mq_freeze_queue(struct request_queue *q);
|
2014-12-20 00:54:14 +00:00
|
|
|
void blk_mq_unfreeze_queue(struct request_queue *q);
|
2017-03-27 12:06:57 +00:00
|
|
|
void blk_freeze_queue_start(struct request_queue *q);
|
2017-03-01 19:22:10 +00:00
|
|
|
void blk_mq_freeze_queue_wait(struct request_queue *q);
|
2017-03-01 19:22:11 +00:00
|
|
|
int blk_mq_freeze_queue_wait_timeout(struct request_queue *q,
|
|
|
|
unsigned long timeout);
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
|
2022-08-15 17:00:43 +00:00
|
|
|
void blk_mq_map_queues(struct blk_mq_queue_map *qmap);
|
2015-12-18 00:08:14 +00:00
|
|
|
void blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, int nr_hw_queues);
|
|
|
|
|
2017-06-21 17:55:47 +00:00
|
|
|
void blk_mq_quiesce_queue_nowait(struct request_queue *q);
|
2017-06-06 15:22:02 +00:00
|
|
|
|
2018-10-31 23:01:22 +00:00
|
|
|
unsigned int blk_mq_rq_cpu(struct request *rq);
|
|
|
|
|
2020-06-11 06:44:47 +00:00
|
|
|
bool __blk_should_fake_timeout(struct request_queue *q);
|
|
|
|
static inline bool blk_should_fake_timeout(struct request_queue *q)
|
|
|
|
{
|
|
|
|
if (IS_ENABLED(CONFIG_FAIL_IO_TIMEOUT) &&
|
|
|
|
test_bit(QUEUE_FLAG_FAIL_IO, &q->queue_flags))
|
|
|
|
return __blk_should_fake_timeout(q);
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
2019-10-22 00:07:24 +00:00
|
|
|
/**
|
|
|
|
* blk_mq_rq_from_pdu - cast a PDU to a request
|
|
|
|
* @pdu: the PDU (Protocol Data Unit) to be casted
|
|
|
|
*
|
|
|
|
* Return: request
|
|
|
|
*
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
* Driver command data is immediately after the request. So subtract request
|
2019-10-22 00:07:24 +00:00
|
|
|
* size to get back to the original request.
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
*/
|
|
|
|
static inline struct request *blk_mq_rq_from_pdu(void *pdu)
|
|
|
|
{
|
|
|
|
return pdu - sizeof(struct request);
|
|
|
|
}
|
2019-10-22 00:07:24 +00:00
|
|
|
|
|
|
|
/**
|
|
|
|
* blk_mq_rq_to_pdu - cast a request to a PDU
|
|
|
|
* @rq: the request to be casted
|
|
|
|
*
|
|
|
|
* Return: pointer to the PDU
|
|
|
|
*
|
|
|
|
* Driver command data is immediately after the request. So add request to get
|
|
|
|
* the PDU.
|
|
|
|
*/
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
static inline void *blk_mq_rq_to_pdu(struct request *rq)
|
|
|
|
{
|
2015-04-09 21:54:05 +00:00
|
|
|
return rq + 1;
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
#define queue_for_each_hw_ctx(q, hctx, i) \
|
2022-03-08 07:32:19 +00:00
|
|
|
xa_for_each(&(q)->hctx_table, (i), (hctx))
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
|
|
|
|
#define hctx_for_each_ctx(hctx, ctx, i) \
|
2014-01-28 15:09:46 +00:00
|
|
|
for ((i) = 0; (i) < (hctx)->nr_ctx && \
|
|
|
|
({ ctx = (hctx)->ctxs[(i)]; 1; }); (i)++)
|
blk-mq: new multi-queue block IO queueing mechanism
Linux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct
request units for IO. The block layer provides various helper
functionalities to let drivers share code, things like tag
management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the
block layer and IO submitter. Since this bypasses the IO stack,
driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic
request_fn based driver doesn't work well enough. The design dates
back to when both SMP and high IOPS was rare. It has problems with
scaling to bigger machines, and runs into scaling issues even on
smaller machines when you have IOPS in the hundreds of thousands
per device.
The stacked approach is then most often selected as the model
for the driver. But this means that everybody has to re-invent
everything, and along with that we get all the problems again
that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The
design is centered around per-cpu queues for queueing IO, which
then funnel down into x number of hardware submission queues.
We might have a 1:1 mapping between the two, or it might be
an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to
be able to uniquely identify a request both in the driver and
to the hardware. The tagging uses per-cpu caches for freed
tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device
basis. Basically the driver should be able to get a notification,
if a request happens to fail.
- Optional support for non 1:1 mappings between issue and
submission queues. blk-mq can redirect IO completions to the
desired location.
- Support for per-request payloads. Drivers almost always need
to associate a request structure with some driver private
command structure. Drivers can tell blk-mq this at init time,
and then any request handed to the driver will have the
required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model
gets neither of these. Even for high IOPS devices, merging
sequential IO reduces per-command overhead and thus
increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with
the hope being that, as it matures, it can replace both the classic
and stacked model. That would get us back to having just 1 real
model for block devices, leaving the stacked approach to dm/md
devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <shli@fusionio.com>
Alexander Gordeev <agordeev@redhat.com>
Christoph Hellwig <hch@infradead.org>
Mike Christie <michaelc@cs.wisc.edu>
Matias Bjorling <m@bjorling.me>
Jeff Moyer <jmoyer@redhat.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2013-10-24 08:20:05 +00:00
|
|
|
|
2019-07-25 02:04:59 +00:00
|
|
|
static inline void blk_mq_cleanup_rq(struct request *rq)
|
|
|
|
{
|
|
|
|
if (rq->q->mq_ops->cleanup_rq)
|
|
|
|
rq->q->mq_ops->cleanup_rq(rq);
|
|
|
|
}
|
|
|
|
|
2020-11-10 02:24:03 +00:00
|
|
|
static inline void blk_rq_bio_prep(struct request *rq, struct bio *bio,
|
|
|
|
unsigned int nr_segs)
|
|
|
|
{
|
|
|
|
rq->nr_phys_segments = nr_segs;
|
|
|
|
rq->__data_len = bio->bi_iter.bi_size;
|
|
|
|
rq->bio = rq->biotail = bio;
|
|
|
|
rq->ioprio = bio_prio(bio);
|
|
|
|
}
|
|
|
|
|
2020-12-03 01:26:36 +00:00
|
|
|
void blk_mq_hctx_set_fq_lock_class(struct blk_mq_hw_ctx *hctx,
|
|
|
|
struct lock_class_key *key);
|
2020-04-25 07:53:36 +00:00
|
|
|
|
2021-09-20 12:33:28 +00:00
|
|
|
static inline bool rq_is_sync(struct request *rq)
|
|
|
|
{
|
|
|
|
return op_is_sync(rq->cmd_flags);
|
|
|
|
}
|
|
|
|
|
|
|
|
void blk_rq_init(struct request_queue *q, struct request *rq);
|
|
|
|
int blk_rq_prep_clone(struct request *rq, struct request *rq_src,
|
|
|
|
struct bio_set *bs, gfp_t gfp_mask,
|
|
|
|
int (*bio_ctr)(struct bio *, struct bio *, void *), void *data);
|
|
|
|
void blk_rq_unprep_clone(struct request *rq);
|
2022-02-15 10:05:38 +00:00
|
|
|
blk_status_t blk_insert_cloned_request(struct request *rq);
|
2021-09-20 12:33:28 +00:00
|
|
|
|
|
|
|
struct rq_map_data {
|
|
|
|
struct page **pages;
|
|
|
|
unsigned long offset;
|
2022-08-05 22:39:04 +00:00
|
|
|
unsigned short page_order;
|
|
|
|
unsigned short nr_entries;
|
|
|
|
bool null_mapped;
|
|
|
|
bool from_user;
|
2021-09-20 12:33:28 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
int blk_rq_map_user(struct request_queue *, struct request *,
|
|
|
|
struct rq_map_data *, void __user *, unsigned long, gfp_t);
|
2022-09-30 06:27:40 +00:00
|
|
|
int blk_rq_map_user_io(struct request *, struct rq_map_data *,
|
|
|
|
void __user *, unsigned long, gfp_t, bool, int, bool, int);
|
2021-09-20 12:33:28 +00:00
|
|
|
int blk_rq_map_user_iov(struct request_queue *, struct request *,
|
|
|
|
struct rq_map_data *, const struct iov_iter *, gfp_t);
|
|
|
|
int blk_rq_unmap_user(struct bio *);
|
|
|
|
int blk_rq_map_kern(struct request_queue *, struct request *, void *,
|
|
|
|
unsigned int, gfp_t);
|
|
|
|
int blk_rq_append_bio(struct request *rq, struct bio *bio);
|
2022-05-24 12:15:30 +00:00
|
|
|
void blk_execute_rq_nowait(struct request *rq, bool at_head);
|
2021-11-26 12:18:01 +00:00
|
|
|
blk_status_t blk_execute_rq(struct request *rq, bool at_head);
|
2022-08-23 16:14:42 +00:00
|
|
|
bool blk_rq_is_poll(struct request *rq);
|
2021-09-20 12:33:28 +00:00
|
|
|
|
|
|
|
struct req_iterator {
|
|
|
|
struct bvec_iter iter;
|
|
|
|
struct bio *bio;
|
|
|
|
};
|
|
|
|
|
|
|
|
#define __rq_for_each_bio(_bio, rq) \
|
|
|
|
if ((rq->bio)) \
|
|
|
|
for (_bio = (rq)->bio; _bio; _bio = _bio->bi_next)
|
|
|
|
|
|
|
|
#define rq_for_each_segment(bvl, _rq, _iter) \
|
|
|
|
__rq_for_each_bio(_iter.bio, _rq) \
|
|
|
|
bio_for_each_segment(bvl, _iter.bio, _iter.iter)
|
|
|
|
|
|
|
|
#define rq_for_each_bvec(bvl, _rq, _iter) \
|
|
|
|
__rq_for_each_bio(_iter.bio, _rq) \
|
|
|
|
bio_for_each_bvec(bvl, _iter.bio, _iter.iter)
|
|
|
|
|
|
|
|
#define rq_iter_last(bvec, _iter) \
|
|
|
|
(_iter.bio->bi_next == NULL && \
|
|
|
|
bio_iter_last(bvec, _iter.iter))
|
|
|
|
|
|
|
|
/*
|
|
|
|
* blk_rq_pos() : the current sector
|
|
|
|
* blk_rq_bytes() : bytes left in the entire request
|
|
|
|
* blk_rq_cur_bytes() : bytes left in the current segment
|
|
|
|
* blk_rq_sectors() : sectors left in the entire request
|
|
|
|
* blk_rq_cur_sectors() : sectors left in the current segment
|
|
|
|
* blk_rq_stats_sectors() : sectors of the entire request used for stats
|
|
|
|
*/
|
|
|
|
static inline sector_t blk_rq_pos(const struct request *rq)
|
|
|
|
{
|
|
|
|
return rq->__sector;
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline unsigned int blk_rq_bytes(const struct request *rq)
|
|
|
|
{
|
|
|
|
return rq->__data_len;
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline int blk_rq_cur_bytes(const struct request *rq)
|
|
|
|
{
|
2021-10-12 16:18:00 +00:00
|
|
|
if (!rq->bio)
|
|
|
|
return 0;
|
|
|
|
if (!bio_has_data(rq->bio)) /* dataless requests such as discard */
|
|
|
|
return rq->bio->bi_iter.bi_size;
|
|
|
|
return bio_iovec(rq->bio).bv_len;
|
2021-09-20 12:33:28 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static inline unsigned int blk_rq_sectors(const struct request *rq)
|
|
|
|
{
|
|
|
|
return blk_rq_bytes(rq) >> SECTOR_SHIFT;
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline unsigned int blk_rq_cur_sectors(const struct request *rq)
|
|
|
|
{
|
|
|
|
return blk_rq_cur_bytes(rq) >> SECTOR_SHIFT;
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline unsigned int blk_rq_stats_sectors(const struct request *rq)
|
|
|
|
{
|
|
|
|
return rq->stats_sectors;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Some commands like WRITE SAME have a payload or data transfer size which
|
|
|
|
* is different from the size of the request. Any driver that supports such
|
|
|
|
* commands using the RQF_SPECIAL_PAYLOAD flag needs to use this helper to
|
|
|
|
* calculate the data transfer size.
|
|
|
|
*/
|
|
|
|
static inline unsigned int blk_rq_payload_bytes(struct request *rq)
|
|
|
|
{
|
|
|
|
if (rq->rq_flags & RQF_SPECIAL_PAYLOAD)
|
|
|
|
return rq->special_vec.bv_len;
|
|
|
|
return blk_rq_bytes(rq);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Return the first full biovec in the request. The caller needs to check that
|
|
|
|
* there are any bvecs before calling this helper.
|
|
|
|
*/
|
|
|
|
static inline struct bio_vec req_bvec(struct request *rq)
|
|
|
|
{
|
|
|
|
if (rq->rq_flags & RQF_SPECIAL_PAYLOAD)
|
|
|
|
return rq->special_vec;
|
|
|
|
return mp_bvec_iter_bvec(rq->bio->bi_io_vec, rq->bio->bi_iter);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline unsigned int blk_rq_count_bios(struct request *rq)
|
|
|
|
{
|
|
|
|
unsigned int nr_bios = 0;
|
|
|
|
struct bio *bio;
|
|
|
|
|
|
|
|
__rq_for_each_bio(bio, rq)
|
|
|
|
nr_bios++;
|
|
|
|
|
|
|
|
return nr_bios;
|
|
|
|
}
|
|
|
|
|
|
|
|
void blk_steal_bios(struct bio_list *list, struct request *rq);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Request completion related functions.
|
|
|
|
*
|
|
|
|
* blk_update_request() completes given number of bytes and updates
|
|
|
|
* the request without completing it.
|
|
|
|
*/
|
|
|
|
bool blk_update_request(struct request *rq, blk_status_t error,
|
|
|
|
unsigned int nr_bytes);
|
|
|
|
void blk_abort_request(struct request *);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Number of physical segments as sent to the device.
|
|
|
|
*
|
|
|
|
* Normally this is the number of discontiguous data segments sent by the
|
|
|
|
* submitter. But for data-less command like discard we might have no
|
|
|
|
* actual data segments submitted, but the driver might have to add it's
|
|
|
|
* own special payload. In that case we still return 1 here so that this
|
|
|
|
* special payload will be mapped.
|
|
|
|
*/
|
|
|
|
static inline unsigned short blk_rq_nr_phys_segments(struct request *rq)
|
|
|
|
{
|
|
|
|
if (rq->rq_flags & RQF_SPECIAL_PAYLOAD)
|
|
|
|
return 1;
|
|
|
|
return rq->nr_phys_segments;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Number of discard segments (or ranges) the driver needs to fill in.
|
|
|
|
* Each discard bio merged into a request is counted as one segment.
|
|
|
|
*/
|
|
|
|
static inline unsigned short blk_rq_nr_discard_segments(struct request *rq)
|
|
|
|
{
|
|
|
|
return max_t(unsigned short, rq->nr_phys_segments, 1);
|
|
|
|
}
|
|
|
|
|
|
|
|
int __blk_rq_map_sg(struct request_queue *q, struct request *rq,
|
|
|
|
struct scatterlist *sglist, struct scatterlist **last_sg);
|
|
|
|
static inline int blk_rq_map_sg(struct request_queue *q, struct request *rq,
|
|
|
|
struct scatterlist *sglist)
|
|
|
|
{
|
|
|
|
struct scatterlist *last_sg = NULL;
|
|
|
|
|
|
|
|
return __blk_rq_map_sg(q, rq, sglist, &last_sg);
|
|
|
|
}
|
|
|
|
void blk_dump_rq_flags(struct request *, char *);
|
|
|
|
|
|
|
|
#ifdef CONFIG_BLK_DEV_ZONED
|
|
|
|
static inline unsigned int blk_rq_zone_no(struct request *rq)
|
|
|
|
{
|
2022-07-06 07:03:50 +00:00
|
|
|
return disk_zone_no(rq->q->disk, blk_rq_pos(rq));
|
2021-09-20 12:33:28 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static inline unsigned int blk_rq_zone_is_seq(struct request *rq)
|
|
|
|
{
|
2022-07-06 07:03:50 +00:00
|
|
|
return disk_zone_is_seq(rq->q->disk, blk_rq_pos(rq));
|
2021-09-20 12:33:28 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
bool blk_req_needs_zone_write_lock(struct request *rq);
|
|
|
|
bool blk_req_zone_write_trylock(struct request *rq);
|
|
|
|
void __blk_req_zone_write_lock(struct request *rq);
|
|
|
|
void __blk_req_zone_write_unlock(struct request *rq);
|
|
|
|
|
|
|
|
static inline void blk_req_zone_write_lock(struct request *rq)
|
|
|
|
{
|
|
|
|
if (blk_req_needs_zone_write_lock(rq))
|
|
|
|
__blk_req_zone_write_lock(rq);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void blk_req_zone_write_unlock(struct request *rq)
|
|
|
|
{
|
|
|
|
if (rq->rq_flags & RQF_ZONE_WRITE_LOCKED)
|
|
|
|
__blk_req_zone_write_unlock(rq);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline bool blk_req_zone_is_write_locked(struct request *rq)
|
|
|
|
{
|
2022-07-06 07:03:50 +00:00
|
|
|
return rq->q->disk->seq_zones_wlock &&
|
|
|
|
test_bit(blk_rq_zone_no(rq), rq->q->disk->seq_zones_wlock);
|
2021-09-20 12:33:28 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static inline bool blk_req_can_dispatch_to_zone(struct request *rq)
|
|
|
|
{
|
|
|
|
if (!blk_req_needs_zone_write_lock(rq))
|
|
|
|
return true;
|
|
|
|
return !blk_req_zone_is_write_locked(rq);
|
|
|
|
}
|
|
|
|
#else /* CONFIG_BLK_DEV_ZONED */
|
|
|
|
static inline bool blk_req_needs_zone_write_lock(struct request *rq)
|
|
|
|
{
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void blk_req_zone_write_lock(struct request *rq)
|
|
|
|
{
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void blk_req_zone_write_unlock(struct request *rq)
|
|
|
|
{
|
|
|
|
}
|
|
|
|
static inline bool blk_req_zone_is_write_locked(struct request *rq)
|
|
|
|
{
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline bool blk_req_can_dispatch_to_zone(struct request *rq)
|
|
|
|
{
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
#endif /* CONFIG_BLK_DEV_ZONED */
|
|
|
|
|
|
|
|
#endif /* BLK_MQ_H */
|