- Introduce the DM vdo target which provides block-level
deduplication, compression, and thin provisioning. Please see both: Documentation/admin-guide/device-mapper/vdo.rst and Documentation/admin-guide/device-mapper/vdo-design.rst - The DM vdo target handles its concurrency by pinning an IO, and subsequent stages of handling that IO, to a particular VDO thread. This aspect of VDO is "unique" but its overall implementation is very tightly coupled to its mostly lockless threading model. As such, VDO is not easily changed to use more traditional finer-grained locking and Linux workqueues. Please see the "Zones and Threading" section of vdo-design.rst - The DM vdo target has been used in production for many years but has seen significant changes over the past ~6 years to prepare it for upstream inclusion. The codebase is still large but it is isolated to drivers/md/dm-vdo/ and has been made considerably more approachable and maintainable. - Matt Sakai has been added to the MAINTAINERS file to reflect that he will send VDO changes upstream through the DM subsystem maintainers. -----BEGIN PGP SIGNATURE----- iQEzBAABCAAdFiEEJfWUX4UqZ4x1O2wixSPxCi2dA1oFAmXqYlEACgkQxSPxCi2d A1oOAggAlPdoDfU2DWoFxJcCGR4sYrk6n8GOGXZIi5bZeJFaPUoH9EQZUVgsMiEo TSpJugk6KddsXNsKHCuc3ctejNT4FAs3f5pcqWQ+VTR6xXD7xw67hN78arhu4VZH OWzAyKu8fs4NUxjMThLdwMlt2+vOfcuOHAXxcil3kvG6SPcPw1FDde3Jbb5OmgcF z1D9kkUuqH+d46P/dwOsNcr7cMIUviZW+plFnVMKsdJGH/SCyu6TCLYWrwBR1VXW pNylxk4AcsffQdu2Oor6+0ALaH7Wq3kjymNLQlYWt0EibGbBakrVs1A/DIXUZDiX gYfcemQpl74x1vifNtIsUWW9vO1XPg== =2Oof -----END PGP SIGNATURE----- Merge tag 'for-6.9/dm-vdo' of git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm Pull device mapper VDO target from Mike Snitzer: "Introduce the DM vdo target which provides block-level deduplication, compression, and thin provisioning. Please see: Documentation/admin-guide/device-mapper/vdo.rst Documentation/admin-guide/device-mapper/vdo-design.rst The DM vdo target handles its concurrency by pinning an IO, and subsequent stages of handling that IO, to a particular VDO thread. This aspect of VDO is "unique" but its overall implementation is very tightly coupled to its mostly lockless threading model. As such, VDO is not easily changed to use more traditional finer-grained locking and Linux workqueues. Please see the "Zones and Threading" section of vdo-design.rst The DM vdo target has been used in production for many years but has seen significant changes over the past ~6 years to prepare it for upstream inclusion. The codebase is still large but it is isolated to drivers/md/dm-vdo/ and has been made considerably more approachable and maintainable. Matt Sakai has been added to the MAINTAINERS file to reflect that he will send VDO changes upstream through the DM subsystem maintainers" * tag 'for-6.9/dm-vdo' of git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm: (142 commits) dm vdo: document minimum metadata size requirements dm vdo: remove meaningless version number constant dm vdo: remove vdo_perform_once dm vdo block-map: Remove stray semicolon dm vdo string-utils: change from uds_ to vdo_ namespace dm vdo logger: change from uds_ to vdo_ namespace dm vdo funnel-queue: change from uds_ to vdo_ namespace dm vdo indexer: fix use after free dm vdo logger: remove log level to string conversion code dm vdo: document log_level parameter dm vdo: add 'log_level' module parameter dm vdo: remove all sysfs interfaces dm vdo target: eliminate inappropriate uses of UDS_SUCCESS dm vdo indexer: update ASSERT and ASSERT_LOG_ONLY usage dm vdo encodings: update some stale comments dm vdo permassert: audit all of ASSERT to test for VDO_SUCCESS dm-vdo funnel-workqueue: return VDO_SUCCESS from make_simple_work_queue dm vdo thread-utils: return VDO_SUCCESS on vdo_create_thread success dm vdo int-map: return VDO_SUCCESS on success dm vdo: check for VDO_SUCCESS return value from memory-alloc functions ...
This commit is contained in:
commit
61387b8dcf
|
@ -34,6 +34,8 @@ Device Mapper
|
|||
switch
|
||||
thin-provisioning
|
||||
unstriped
|
||||
vdo-design
|
||||
vdo
|
||||
verity
|
||||
writecache
|
||||
zero
|
||||
|
|
|
@ -0,0 +1,633 @@
|
|||
.. SPDX-License-Identifier: GPL-2.0-only
|
||||
|
||||
================
|
||||
Design of dm-vdo
|
||||
================
|
||||
|
||||
The dm-vdo (virtual data optimizer) target provides inline deduplication,
|
||||
compression, zero-block elimination, and thin provisioning. A dm-vdo target
|
||||
can be backed by up to 256TB of storage, and can present a logical size of
|
||||
up to 4PB. This target was originally developed at Permabit Technology
|
||||
Corp. starting in 2009. It was first released in 2013 and has been used in
|
||||
production environments ever since. It was made open-source in 2017 after
|
||||
Permabit was acquired by Red Hat. This document describes the design of
|
||||
dm-vdo. For usage, see vdo.rst in the same directory as this file.
|
||||
|
||||
Because deduplication rates fall drastically as the block size increases, a
|
||||
vdo target has a maximum block size of 4K. However, it can achieve
|
||||
deduplication rates of 254:1, i.e. up to 254 copies of a given 4K block can
|
||||
reference a single 4K of actual storage. It can achieve compression rates
|
||||
of 14:1. All zero blocks consume no storage at all.
|
||||
|
||||
Theory of Operation
|
||||
===================
|
||||
|
||||
The design of dm-vdo is based on the idea that deduplication is a two-part
|
||||
problem. The first is to recognize duplicate data. The second is to avoid
|
||||
storing multiple copies of those duplicates. Therefore, dm-vdo has two main
|
||||
parts: a deduplication index (called UDS) that is used to discover
|
||||
duplicate data, and a data store with a reference counted block map that
|
||||
maps from logical block addresses to the actual storage location of the
|
||||
data.
|
||||
|
||||
Zones and Threading
|
||||
-------------------
|
||||
|
||||
Due to the complexity of data optimization, the number of metadata
|
||||
structures involved in a single write operation to a vdo target is larger
|
||||
than most other targets. Furthermore, because vdo must operate on small
|
||||
block sizes in order to achieve good deduplication rates, acceptable
|
||||
performance can only be achieved through parallelism. Therefore, vdo's
|
||||
design attempts to be lock-free.
|
||||
|
||||
Most of a vdo's main data structures are designed to be easily divided into
|
||||
"zones" such that any given bio must only access a single zone of any zoned
|
||||
structure. Safety with minimal locking is achieved by ensuring that during
|
||||
normal operation, each zone is assigned to a specific thread, and only that
|
||||
thread will access the portion of the data structure in that zone.
|
||||
Associated with each thread is a work queue. Each bio is associated with a
|
||||
request object (the "data_vio") which will be added to a work queue when
|
||||
the next phase of its operation requires access to the structures in the
|
||||
zone associated with that queue.
|
||||
|
||||
Another way of thinking about this arrangement is that the work queue for
|
||||
each zone has an implicit lock on the structures it manages for all its
|
||||
operations, because vdo guarantees that no other thread will alter those
|
||||
structures.
|
||||
|
||||
Although each structure is divided into zones, this division is not
|
||||
reflected in the on-disk representation of each data structure. Therefore,
|
||||
the number of zones for each structure, and hence the number of threads,
|
||||
can be reconfigured each time a vdo target is started.
|
||||
|
||||
The Deduplication Index
|
||||
-----------------------
|
||||
|
||||
In order to identify duplicate data efficiently, vdo was designed to
|
||||
leverage some common characteristics of duplicate data. From empirical
|
||||
observations, we gathered two key insights. The first is that in most data
|
||||
sets with significant amounts of duplicate data, the duplicates tend to
|
||||
have temporal locality. When a duplicate appears, it is more likely that
|
||||
other duplicates will be detected, and that those duplicates will have been
|
||||
written at about the same time. This is why the index keeps records in
|
||||
temporal order. The second insight is that new data is more likely to
|
||||
duplicate recent data than it is to duplicate older data and in general,
|
||||
there are diminishing returns to looking further back in time. Therefore,
|
||||
when the index is full, it should cull its oldest records to make space for
|
||||
new ones. Another important idea behind the design of the index is that the
|
||||
ultimate goal of deduplication is to reduce storage costs. Since there is a
|
||||
trade-off between the storage saved and the resources expended to achieve
|
||||
those savings, vdo does not attempt to find every last duplicate block. It
|
||||
is sufficient to find and eliminate most of the redundancy.
|
||||
|
||||
Each block of data is hashed to produce a 16-byte block name. An index
|
||||
record consists of this block name paired with the presumed location of
|
||||
that data on the underlying storage. However, it is not possible to
|
||||
guarantee that the index is accurate. In the most common case, this occurs
|
||||
because it is too costly to update the index when a block is over-written
|
||||
or discarded. Doing so would require either storing the block name along
|
||||
with the blocks, which is difficult to do efficiently in block-based
|
||||
storage, or reading and rehashing each block before overwriting it.
|
||||
Inaccuracy can also result from a hash collision where two different blocks
|
||||
have the same name. In practice, this is extremely unlikely, but because
|
||||
vdo does not use a cryptographic hash, a malicious workload could be
|
||||
constructed. Because of these inaccuracies, vdo treats the locations in the
|
||||
index as hints, and reads each indicated block to verify that it is indeed
|
||||
a duplicate before sharing the existing block with a new one.
|
||||
|
||||
Records are collected into groups called chapters. New records are added to
|
||||
the newest chapter, called the open chapter. This chapter is stored in a
|
||||
format optimized for adding and modifying records, and the content of the
|
||||
open chapter is not finalized until it runs out of space for new records.
|
||||
When the open chapter fills up, it is closed and a new open chapter is
|
||||
created to collect new records.
|
||||
|
||||
Closing a chapter converts it to a different format which is optimized for
|
||||
reading. The records are written to a series of record pages based on the
|
||||
order in which they were received. This means that records with temporal
|
||||
locality should be on a small number of pages, reducing the I/O required to
|
||||
retrieve them. The chapter also compiles an index that indicates which
|
||||
record page contains any given name. This index means that a request for a
|
||||
name can determine exactly which record page may contain that record,
|
||||
without having to load the entire chapter from storage. This index uses
|
||||
only a subset of the block name as its key, so it cannot guarantee that an
|
||||
index entry refers to the desired block name. It can only guarantee that if
|
||||
there is a record for this name, it will be on the indicated page. Closed
|
||||
chapters are read-only structures and their contents are never altered in
|
||||
any way.
|
||||
|
||||
Once enough records have been written to fill up all the available index
|
||||
space, the oldest chapter is removed to make space for new chapters. Any
|
||||
time a request finds a matching record in the index, that record is copied
|
||||
into the open chapter. This ensures that useful block names remain available
|
||||
in the index, while unreferenced block names are forgotten over time.
|
||||
|
||||
In order to find records in older chapters, the index also maintains a
|
||||
higher level structure called the volume index, which contains entries
|
||||
mapping each block name to the chapter containing its newest record. This
|
||||
mapping is updated as records for the block name are copied or updated,
|
||||
ensuring that only the newest record for a given block name can be found.
|
||||
An older record for a block name will no longer be found even though it has
|
||||
not been deleted from its chapter. Like the chapter index, the volume index
|
||||
uses only a subset of the block name as its key and can not definitively
|
||||
say that a record exists for a name. It can only say which chapter would
|
||||
contain the record if a record exists. The volume index is stored entirely
|
||||
in memory and is saved to storage only when the vdo target is shut down.
|
||||
|
||||
From the viewpoint of a request for a particular block name, it will first
|
||||
look up the name in the volume index. This search will either indicate that
|
||||
the name is new, or which chapter to search. If it returns a chapter, the
|
||||
request looks up its name in the chapter index. This will indicate either
|
||||
that the name is new, or which record page to search. Finally, if it is not
|
||||
new, the request will look for its name in the indicated record page.
|
||||
This process may require up to two page reads per request (one for the
|
||||
chapter index page and one for the request page). However, recently
|
||||
accessed pages are cached so that these page reads can be amortized across
|
||||
many block name requests.
|
||||
|
||||
The volume index and the chapter indexes are implemented using a
|
||||
memory-efficient structure called a delta index. Instead of storing the
|
||||
entire block name (the key) for each entry, the entries are sorted by name
|
||||
and only the difference between adjacent keys (the delta) is stored.
|
||||
Because we expect the hashes to be randomly distributed, the size of the
|
||||
deltas follows an exponential distribution. Because of this distribution,
|
||||
the deltas are expressed using a Huffman code to take up even less space.
|
||||
The entire sorted list of keys is called a delta list. This structure
|
||||
allows the index to use many fewer bytes per entry than a traditional hash
|
||||
table, but it is slightly more expensive to look up entries, because a
|
||||
request must read every entry in a delta list to add up the deltas in order
|
||||
to find the record it needs. The delta index reduces this lookup cost by
|
||||
splitting its key space into many sub-lists, each starting at a fixed key
|
||||
value, so that each individual list is short.
|
||||
|
||||
The default index size can hold 64 million records, corresponding to about
|
||||
256GB of data. This means that the index can identify duplicate data if the
|
||||
original data was written within the last 256GB of writes. This range is
|
||||
called the deduplication window. If new writes duplicate data that is older
|
||||
than that, the index will not be able to find it because the records of the
|
||||
older data have been removed. This means that if an application writes a
|
||||
200 GB file to a vdo target and then immediately writes it again, the two
|
||||
copies will deduplicate perfectly. Doing the same with a 500 GB file will
|
||||
result in no deduplication, because the beginning of the file will no
|
||||
longer be in the index by the time the second write begins (assuming there
|
||||
is no duplication within the file itself).
|
||||
|
||||
If an application anticipates a data workload that will see useful
|
||||
deduplication beyond the 256GB threshold, vdo can be configured to use a
|
||||
larger index with a correspondingly larger deduplication window. (This
|
||||
configuration can only be set when the target is created, not altered
|
||||
later. It is important to consider the expected workload for a vdo target
|
||||
before configuring it.) There are two ways to do this.
|
||||
|
||||
One way is to increase the memory size of the index, which also increases
|
||||
the amount of backing storage required. Doubling the size of the index will
|
||||
double the length of the deduplication window at the expense of doubling
|
||||
the storage size and the memory requirements.
|
||||
|
||||
The other option is to enable sparse indexing. Sparse indexing increases
|
||||
the deduplication window by a factor of 10, at the expense of also
|
||||
increasing the storage size by a factor of 10. However with sparse
|
||||
indexing, the memory requirements do not increase. The trade-off is
|
||||
slightly more computation per request and a slight decrease in the amount
|
||||
of deduplication detected. For most workloads with significant amounts of
|
||||
duplicate data, sparse indexing will detect 97-99% of the deduplication
|
||||
that a standard index will detect.
|
||||
|
||||
The vio and data_vio Structures
|
||||
-------------------------------
|
||||
|
||||
A vio (short for Vdo I/O) is conceptually similar to a bio, with additional
|
||||
fields and data to track vdo-specific information. A struct vio maintains a
|
||||
pointer to a bio but also tracks other fields specific to the operation of
|
||||
vdo. The vio is kept separate from its related bio because there are many
|
||||
circumstances where vdo completes the bio but must continue to do work
|
||||
related to deduplication or compression.
|
||||
|
||||
Metadata reads and writes, and other writes that originate within vdo, use
|
||||
a struct vio directly. Application reads and writes use a larger structure
|
||||
called a data_vio to track information about their progress. A struct
|
||||
data_vio contain a struct vio and also includes several other fields
|
||||
related to deduplication and other vdo features. The data_vio is the
|
||||
primary unit of application work in vdo. Each data_vio proceeds through a
|
||||
set of steps to handle the application data, after which it is reset and
|
||||
returned to a pool of data_vios for reuse.
|
||||
|
||||
There is a fixed pool of 2048 data_vios. This number was chosen to bound
|
||||
the amount of work that is required to recover from a crash. In addition,
|
||||
benchmarks have indicated that increasing the size of the pool does not
|
||||
significantly improve performance.
|
||||
|
||||
The Data Store
|
||||
--------------
|
||||
|
||||
The data store is implemented by three main data structures, all of which
|
||||
work in concert to reduce or amortize metadata updates across as many data
|
||||
writes as possible.
|
||||
|
||||
*The Slab Depot*
|
||||
|
||||
Most of the vdo volume belongs to the slab depot. The depot contains a
|
||||
collection of slabs. The slabs can be up to 32GB, and are divided into
|
||||
three sections. Most of a slab consists of a linear sequence of 4K blocks.
|
||||
These blocks are used either to store data, or to hold portions of the
|
||||
block map (see below). In addition to the data blocks, each slab has a set
|
||||
of reference counters, using 1 byte for each data block. Finally each slab
|
||||
has a journal.
|
||||
|
||||
Reference updates are written to the slab journal. Slab journal blocks are
|
||||
written out either when they are full, or when the recovery journal
|
||||
requests they do so in order to allow the main recovery journal (see below)
|
||||
to free up space. The slab journal is used both to ensure that the main
|
||||
recovery journal can regularly free up space, and also to amortize the cost
|
||||
of updating individual reference blocks. The reference counters are kept in
|
||||
memory and are written out, a block at a time in oldest-dirtied-order, only
|
||||
when there is a need to reclaim slab journal space. The write operations
|
||||
are performed in the background as needed so they do not add latency to
|
||||
particular I/O operations.
|
||||
|
||||
Each slab is independent of every other. They are assigned to "physical
|
||||
zones" in round-robin fashion. If there are P physical zones, then slab n
|
||||
is assigned to zone n mod P.
|
||||
|
||||
The slab depot maintains an additional small data structure, the "slab
|
||||
summary," which is used to reduce the amount of work needed to come back
|
||||
online after a crash. The slab summary maintains an entry for each slab
|
||||
indicating whether or not the slab has ever been used, whether all of its
|
||||
reference count updates have been persisted to storage, and approximately
|
||||
how full it is. During recovery, each physical zone will attempt to recover
|
||||
at least one slab, stopping whenever it has recovered a slab which has some
|
||||
free blocks. Once each zone has some space, or has determined that none is
|
||||
available, the target can resume normal operation in a degraded mode. Read
|
||||
and write requests can be serviced, perhaps with degraded performance,
|
||||
while the remainder of the dirty slabs are recovered.
|
||||
|
||||
*The Block Map*
|
||||
|
||||
The block map contains the logical to physical mapping. It can be thought
|
||||
of as an array with one entry per logical address. Each entry is 5 bytes,
|
||||
36 bits of which contain the physical block number which holds the data for
|
||||
the given logical address. The other 4 bits are used to indicate the nature
|
||||
of the mapping. Of the 16 possible states, one represents a logical address
|
||||
which is unmapped (i.e. it has never been written, or has been discarded),
|
||||
one represents an uncompressed block, and the other 14 states are used to
|
||||
indicate that the mapped data is compressed, and which of the compression
|
||||
slots in the compressed block contains the data for this logical address.
|
||||
|
||||
In practice, the array of mapping entries is divided into "block map
|
||||
pages," each of which fits in a single 4K block. Each block map page
|
||||
consists of a header and 812 mapping entries. Each mapping page is actually
|
||||
a leaf of a radix tree which consists of block map pages at each level.
|
||||
There are 60 radix trees which are assigned to "logical zones" in round
|
||||
robin fashion. (If there are L logical zones, tree n will belong to zone n
|
||||
mod L.) At each level, the trees are interleaved, so logical addresses
|
||||
0-811 belong to tree 0, logical addresses 812-1623 belong to tree 1, and so
|
||||
on. The interleaving is maintained all the way up to the 60 root nodes.
|
||||
Choosing 60 trees results in an evenly distributed number of trees per zone
|
||||
for a large number of possible logical zone counts. The storage for the 60
|
||||
tree roots is allocated at format time. All other block map pages are
|
||||
allocated out of the slabs as needed. This flexible allocation avoids the
|
||||
need to pre-allocate space for the entire set of logical mappings and also
|
||||
makes growing the logical size of a vdo relatively easy.
|
||||
|
||||
In operation, the block map maintains two caches. It is prohibitive to keep
|
||||
the entire leaf level of the trees in memory, so each logical zone
|
||||
maintains its own cache of leaf pages. The size of this cache is
|
||||
configurable at target start time. The second cache is allocated at start
|
||||
time, and is large enough to hold all the non-leaf pages of the entire
|
||||
block map. This cache is populated as pages are needed.
|
||||
|
||||
*The Recovery Journal*
|
||||
|
||||
The recovery journal is used to amortize updates across the block map and
|
||||
slab depot. Each write request causes an entry to be made in the journal.
|
||||
Entries are either "data remappings" or "block map remappings." For a data
|
||||
remapping, the journal records the logical address affected and its old and
|
||||
new physical mappings. For a block map remapping, the journal records the
|
||||
block map page number and the physical block allocated for it. Block map
|
||||
pages are never reclaimed or repurposed, so the old mapping is always 0.
|
||||
|
||||
Each journal entry is an intent record summarizing the metadata updates
|
||||
that are required for a data_vio. The recovery journal issues a flush
|
||||
before each journal block write to ensure that the physical data for the
|
||||
new block mappings in that block are stable on storage, and journal block
|
||||
writes are all issued with the FUA bit set to ensure the recovery journal
|
||||
entries themselves are stable. The journal entry and the data write it
|
||||
represents must be stable on disk before the other metadata structures may
|
||||
be updated to reflect the operation. These entries allow the vdo device to
|
||||
reconstruct the logical to physical mappings after an unexpected
|
||||
interruption such as a loss of power.
|
||||
|
||||
*Write Path*
|
||||
|
||||
All write I/O to vdo is asynchronous. Each bio will be acknowledged as soon
|
||||
as vdo has done enough work to guarantee that it can complete the write
|
||||
eventually. Generally, the data for acknowledged but unflushed write I/O
|
||||
can be treated as though it is cached in memory. If an application
|
||||
requires data to be stable on storage, it must issue a flush or write the
|
||||
data with the FUA bit set like any other asynchronous I/O. Shutting down
|
||||
the vdo target will also flush any remaining I/O.
|
||||
|
||||
Application write bios follow the steps outlined below.
|
||||
|
||||
1. A data_vio is obtained from the data_vio pool and associated with the
|
||||
application bio. If there are no data_vios available, the incoming bio
|
||||
will block until a data_vio is available. This provides back pressure
|
||||
to the application. The data_vio pool is protected by a spin lock.
|
||||
|
||||
The newly acquired data_vio is reset and the bio's data is copied into
|
||||
the data_vio if it is a write and the data is not all zeroes. The data
|
||||
must be copied because the application bio can be acknowledged before
|
||||
the data_vio processing is complete, which means later processing steps
|
||||
will no longer have access to the application bio. The application bio
|
||||
may also be smaller than 4K, in which case the data_vio will have
|
||||
already read the underlying block and the data is instead copied over
|
||||
the relevant portion of the larger block.
|
||||
|
||||
2. The data_vio places a claim (the "logical lock") on the logical address
|
||||
of the bio. It is vital to prevent simultaneous modifications of the
|
||||
same logical address, because deduplication involves sharing blocks.
|
||||
This claim is implemented as an entry in a hashtable where the key is
|
||||
the logical address and the value is a pointer to the data_vio
|
||||
currently handling that address.
|
||||
|
||||
If a data_vio looks in the hashtable and finds that another data_vio is
|
||||
already operating on that logical address, it waits until the previous
|
||||
operation finishes. It also sends a message to inform the current
|
||||
lock holder that it is waiting. Most notably, a new data_vio waiting
|
||||
for a logical lock will flush the previous lock holder out of the
|
||||
compression packer (step 8d) rather than allowing it to continue
|
||||
waiting to be packed.
|
||||
|
||||
This stage requires the data_vio to get an implicit lock on the
|
||||
appropriate logical zone to prevent concurrent modifications of the
|
||||
hashtable. This implicit locking is handled by the zone divisions
|
||||
described above.
|
||||
|
||||
3. The data_vio traverses the block map tree to ensure that all the
|
||||
necessary internal tree nodes have been allocated, by trying to find
|
||||
the leaf page for its logical address. If any interior tree page is
|
||||
missing, it is allocated at this time out of the same physical storage
|
||||
pool used to store application data.
|
||||
|
||||
a. If any page-node in the tree has not yet been allocated, it must be
|
||||
allocated before the write can continue. This step requires the
|
||||
data_vio to lock the page-node that needs to be allocated. This
|
||||
lock, like the logical block lock in step 2, is a hashtable entry
|
||||
that causes other data_vios to wait for the allocation process to
|
||||
complete.
|
||||
|
||||
The implicit logical zone lock is released while the allocation is
|
||||
happening, in order to allow other operations in the same logical
|
||||
zone to proceed. The details of allocation are the same as in
|
||||
step 4. Once a new node has been allocated, that node is added to
|
||||
the tree using a similar process to adding a new data block mapping.
|
||||
The data_vio journals the intent to add the new node to the block
|
||||
map tree (step 10), updates the reference count of the new block
|
||||
(step 11), and reacquires the implicit logical zone lock to add the
|
||||
new mapping to the parent tree node (step 12). Once the tree is
|
||||
updated, the data_vio proceeds down the tree. Any other data_vios
|
||||
waiting on this allocation also proceed.
|
||||
|
||||
b. In the steady-state case, the block map tree nodes will already be
|
||||
allocated, so the data_vio just traverses the tree until it finds
|
||||
the required leaf node. The location of the mapping (the "block map
|
||||
slot") is recorded in the data_vio so that later steps do not need
|
||||
to traverse the tree again. The data_vio then releases the implicit
|
||||
logical zone lock.
|
||||
|
||||
4. If the block is a zero block, skip to step 9. Otherwise, an attempt is
|
||||
made to allocate a free data block. This allocation ensures that the
|
||||
data_vio can write its data somewhere even if deduplication and
|
||||
compression are not possible. This stage gets an implicit lock on a
|
||||
physical zone to search for free space within that zone.
|
||||
|
||||
The data_vio will search each slab in a zone until it finds a free
|
||||
block or decides there are none. If the first zone has no free space,
|
||||
it will proceed to search the next physical zone by taking the implicit
|
||||
lock for that zone and releasing the previous one until it finds a
|
||||
free block or runs out of zones to search. The data_vio will acquire a
|
||||
struct pbn_lock (the "physical block lock") on the free block. The
|
||||
struct pbn_lock also has several fields to record the various kinds of
|
||||
claims that data_vios can have on physical blocks. The pbn_lock is
|
||||
added to a hashtable like the logical block locks in step 2. This
|
||||
hashtable is also covered by the implicit physical zone lock. The
|
||||
reference count of the free block is updated to prevent any other
|
||||
data_vio from considering it free. The reference counters are a
|
||||
sub-component of the slab and are thus also covered by the implicit
|
||||
physical zone lock.
|
||||
|
||||
5. If an allocation was obtained, the data_vio has all the resources it
|
||||
needs to complete the write. The application bio can safely be
|
||||
acknowledged at this point. The acknowledgment happens on a separate
|
||||
thread to prevent the application callback from blocking other data_vio
|
||||
operations.
|
||||
|
||||
If an allocation could not be obtained, the data_vio continues to
|
||||
attempt to deduplicate or compress the data, but the bio is not
|
||||
acknowledged because the vdo device may be out of space.
|
||||
|
||||
6. At this point vdo must determine where to store the application data.
|
||||
The data_vio's data is hashed and the hash (the "record name") is
|
||||
recorded in the data_vio.
|
||||
|
||||
7. The data_vio reserves or joins a struct hash_lock, which manages all of
|
||||
the data_vios currently writing the same data. Active hash locks are
|
||||
tracked in a hashtable similar to the way logical block locks are
|
||||
tracked in step 2. This hashtable is covered by the implicit lock on
|
||||
the hash zone.
|
||||
|
||||
If there is no existing hash lock for this data_vio's record_name, the
|
||||
data_vio obtains a hash lock from the pool, adds it to the hashtable,
|
||||
and sets itself as the new hash lock's "agent." The hash_lock pool is
|
||||
also covered by the implicit hash zone lock. The hash lock agent will
|
||||
do all the work to decide where the application data will be
|
||||
written. If a hash lock for the data_vio's record_name already exists,
|
||||
and the data_vio's data is the same as the agent's data, the new
|
||||
data_vio will wait for the agent to complete its work and then share
|
||||
its result.
|
||||
|
||||
In the rare case that a hash lock exists for the data_vio's hash but
|
||||
the data does not match the hash lock's agent, the data_vio skips to
|
||||
step 8h and attempts to write its data directly. This can happen if two
|
||||
different data blocks produce the same hash, for example.
|
||||
|
||||
8. The hash lock agent attempts to deduplicate or compress its data with
|
||||
the following steps.
|
||||
|
||||
a. The agent initializes and sends its embedded deduplication request
|
||||
(struct uds_request) to the deduplication index. This does not
|
||||
require the data_vio to get any locks because the index components
|
||||
manage their own locking. The data_vio waits until it either gets a
|
||||
response from the index or times out.
|
||||
|
||||
b. If the deduplication index returns advice, the data_vio attempts to
|
||||
obtain a physical block lock on the indicated physical address, in
|
||||
order to read the data and verify that it is the same as the
|
||||
data_vio's data, and that it can accept more references. If the
|
||||
physical address is already locked by another data_vio, the data at
|
||||
that address may soon be overwritten so it is not safe to use the
|
||||
address for deduplication.
|
||||
|
||||
c. If the data matches and the physical block can add references, the
|
||||
agent and any other data_vios waiting on it will record this
|
||||
physical block as their new physical address and proceed to step 9
|
||||
to record their new mapping. If there are more data_vios in the hash
|
||||
lock than there are references available, one of the remaining
|
||||
data_vios becomes the new agent and continues to step 8d as if no
|
||||
valid advice was returned.
|
||||
|
||||
d. If no usable duplicate block was found, the agent first checks that
|
||||
it has an allocated physical block (from step 3) that it can write
|
||||
to. If the agent does not have an allocation, some other data_vio in
|
||||
the hash lock that does have an allocation takes over as agent. If
|
||||
none of the data_vios have an allocated physical block, these writes
|
||||
are out of space, so they proceed to step 13 for cleanup.
|
||||
|
||||
e. The agent attempts to compress its data. If the data does not
|
||||
compress, the data_vio will continue to step 8h to write its data
|
||||
directly.
|
||||
|
||||
If the compressed size is small enough, the agent will release the
|
||||
implicit hash zone lock and go to the packer (struct packer) where
|
||||
it will be placed in a bin (struct packer_bin) along with other
|
||||
data_vios. All compression operations require the implicit lock on
|
||||
the packer zone.
|
||||
|
||||
The packer can combine up to 14 compressed blocks in a single 4k
|
||||
data block. Compression is only helpful if vdo can pack at least 2
|
||||
data_vios into a single data block. This means that a data_vio may
|
||||
wait in the packer for an arbitrarily long time for other data_vios
|
||||
to fill out the compressed block. There is a mechanism for vdo to
|
||||
evict waiting data_vios when continuing to wait would cause
|
||||
problems. Circumstances causing an eviction include an application
|
||||
flush, device shutdown, or a subsequent data_vio trying to overwrite
|
||||
the same logical block address. A data_vio may also be evicted from
|
||||
the packer if it cannot be paired with any other compressed block
|
||||
before more compressible blocks need to use its bin. An evicted
|
||||
data_vio will proceed to step 8h to write its data directly.
|
||||
|
||||
f. If the agent fills a packer bin, either because all 14 of its slots
|
||||
are used or because it has no remaining space, it is written out
|
||||
using the allocated physical block from one of its data_vios. Step
|
||||
8d has already ensured that an allocation is available.
|
||||
|
||||
g. Each data_vio sets the compressed block as its new physical address.
|
||||
The data_vio obtains an implicit lock on the physical zone and
|
||||
acquires the struct pbn_lock for the compressed block, which is
|
||||
modified to be a shared lock. Then it releases the implicit physical
|
||||
zone lock and proceeds to step 8i.
|
||||
|
||||
h. Any data_vio evicted from the packer will have an allocation from
|
||||
step 3. It will write its data to that allocated physical block.
|
||||
|
||||
i. After the data is written, if the data_vio is the agent of a hash
|
||||
lock, it will reacquire the implicit hash zone lock and share its
|
||||
physical address with as many other data_vios in the hash lock as
|
||||
possible. Each data_vio will then proceed to step 9 to record its
|
||||
new mapping.
|
||||
|
||||
j. If the agent actually wrote new data (whether compressed or not),
|
||||
the deduplication index is updated to reflect the location of the
|
||||
new data. The agent then releases the implicit hash zone lock.
|
||||
|
||||
9. The data_vio determines the previous mapping of the logical address.
|
||||
There is a cache for block map leaf pages (the "block map cache"),
|
||||
because there are usually too many block map leaf nodes to store
|
||||
entirely in memory. If the desired leaf page is not in the cache, the
|
||||
data_vio will reserve a slot in the cache and load the desired page
|
||||
into it, possibly evicting an older cached page. The data_vio then
|
||||
finds the current physical address for this logical address (the "old
|
||||
physical mapping"), if any, and records it. This step requires a lock
|
||||
on the block map cache structures, covered by the implicit logical zone
|
||||
lock.
|
||||
|
||||
10. The data_vio makes an entry in the recovery journal containing the
|
||||
logical block address, the old physical mapping, and the new physical
|
||||
mapping. Making this journal entry requires holding the implicit
|
||||
recovery journal lock. The data_vio will wait in the journal until all
|
||||
recovery blocks up to the one containing its entry have been written
|
||||
and flushed to ensure the transaction is stable on storage.
|
||||
|
||||
11. Once the recovery journal entry is stable, the data_vio makes two slab
|
||||
journal entries: an increment entry for the new mapping, and a
|
||||
decrement entry for the old mapping. These two operations each require
|
||||
holding a lock on the affected physical slab, covered by its implicit
|
||||
physical zone lock. For correctness during recovery, the slab journal
|
||||
entries in any given slab journal must be in the same order as the
|
||||
corresponding recovery journal entries. Therefore, if the two entries
|
||||
are in different zones, they are made concurrently, and if they are in
|
||||
the same zone, the increment is always made before the decrement in
|
||||
order to avoid underflow. After each slab journal entry is made in
|
||||
memory, the associated reference count is also updated in memory.
|
||||
|
||||
12. Once both of the reference count updates are done, the data_vio
|
||||
acquires the implicit logical zone lock and updates the
|
||||
logical-to-physical mapping in the block map to point to the new
|
||||
physical block. At this point the write operation is complete.
|
||||
|
||||
13. If the data_vio has a hash lock, it acquires the implicit hash zone
|
||||
lock and releases its hash lock to the pool.
|
||||
|
||||
The data_vio then acquires the implicit physical zone lock and releases
|
||||
the struct pbn_lock it holds for its allocated block. If it had an
|
||||
allocation that it did not use, it also sets the reference count for
|
||||
that block back to zero to free it for use by subsequent data_vios.
|
||||
|
||||
The data_vio then acquires the implicit logical zone lock and releases
|
||||
the logical block lock acquired in step 2.
|
||||
|
||||
The application bio is then acknowledged if it has not previously been
|
||||
acknowledged, and the data_vio is returned to the pool.
|
||||
|
||||
*Read Path*
|
||||
|
||||
An application read bio follows a much simpler set of steps. It does steps
|
||||
1 and 2 in the write path to obtain a data_vio and lock its logical
|
||||
address. If there is already a write data_vio in progress for that logical
|
||||
address that is guaranteed to complete, the read data_vio will copy the
|
||||
data from the write data_vio and return it. Otherwise, it will look up the
|
||||
logical-to-physical mapping by traversing the block map tree as in step 3,
|
||||
and then read and possibly decompress the indicated data at the indicated
|
||||
physical block address. A read data_vio will not allocate block map tree
|
||||
nodes if they are missing. If the interior block map nodes do not exist
|
||||
yet, the logical block map address must still be unmapped and the read
|
||||
data_vio will return all zeroes. A read data_vio handles cleanup and
|
||||
acknowledgment as in step 13, although it only needs to release the logical
|
||||
lock and return itself to the pool.
|
||||
|
||||
*Small Writes*
|
||||
|
||||
All storage within vdo is managed as 4KB blocks, but it can accept writes
|
||||
as small as 512 bytes. Processing a write that is smaller than 4K requires
|
||||
a read-modify-write operation that reads the relevant 4K block, copies the
|
||||
new data over the approriate sectors of the block, and then launches a
|
||||
write operation for the modified data block. The read and write stages of
|
||||
this operation are nearly identical to the normal read and write
|
||||
operations, and a single data_vio is used throughout this operation.
|
||||
|
||||
*Recovery*
|
||||
|
||||
When a vdo is restarted after a crash, it will attempt to recover from the
|
||||
recovery journal. During the pre-resume phase of the next start, the
|
||||
recovery journal is read. The increment portion of valid entries are played
|
||||
into the block map. Next, valid entries are played, in order as required,
|
||||
into the slab journals. Finally, each physical zone attempts to replay at
|
||||
least one slab journal to reconstruct the reference counts of one slab.
|
||||
Once each zone has some free space (or has determined that it has none),
|
||||
the vdo comes back online, while the remainder of the slab journals are
|
||||
used to reconstruct the rest of the reference counts in the background.
|
||||
|
||||
*Read-only Rebuild*
|
||||
|
||||
If a vdo encounters an unrecoverable error, it will enter read-only mode.
|
||||
This mode indicates that some previously acknowledged data may have been
|
||||
lost. The vdo may be instructed to rebuild as best it can in order to
|
||||
return to a writable state. However, this is never done automatically due
|
||||
to the possibility that data has been lost. During a read-only rebuild, the
|
||||
block map is recovered from the recovery journal as before. However, the
|
||||
reference counts are not rebuilt from the slab journals. Instead, the
|
||||
reference counts are zeroed, the entire block map is traversed, and the
|
||||
reference counts are updated from the block mappings. While this may lose
|
||||
some data, it ensures that the block map and reference counts are
|
||||
consistent with each other. This allows vdo to resume normal operation and
|
||||
accept further writes.
|
|
@ -0,0 +1,406 @@
|
|||
.. SPDX-License-Identifier: GPL-2.0-only
|
||||
|
||||
dm-vdo
|
||||
======
|
||||
|
||||
The dm-vdo (virtual data optimizer) device mapper target provides
|
||||
block-level deduplication, compression, and thin provisioning. As a device
|
||||
mapper target, it can add these features to the storage stack, compatible
|
||||
with any file system. The vdo target does not protect against data
|
||||
corruption, relying instead on integrity protection of the storage below
|
||||
it. It is strongly recommended that lvm be used to manage vdo volumes. See
|
||||
lvmvdo(7).
|
||||
|
||||
Userspace component
|
||||
===================
|
||||
|
||||
Formatting a vdo volume requires the use of the 'vdoformat' tool, available
|
||||
at:
|
||||
|
||||
https://github.com/dm-vdo/vdo/
|
||||
|
||||
In most cases, a vdo target will recover from a crash automatically the
|
||||
next time it is started. In cases where it encountered an unrecoverable
|
||||
error (either during normal operation or crash recovery) the target will
|
||||
enter or come up in read-only mode. Because read-only mode is indicative of
|
||||
data-loss, a positive action must be taken to bring vdo out of read-only
|
||||
mode. The 'vdoforcerebuild' tool, available from the same repo, is used to
|
||||
prepare a read-only vdo to exit read-only mode. After running this tool,
|
||||
the vdo target will rebuild its metadata the next time it is
|
||||
started. Although some data may be lost, the rebuilt vdo's metadata will be
|
||||
internally consistent and the target will be writable again.
|
||||
|
||||
The repo also contains additional userspace tools which can be used to
|
||||
inspect a vdo target's on-disk metadata. Fortunately, these tools are
|
||||
rarely needed except by dm-vdo developers.
|
||||
|
||||
Metadata requirements
|
||||
=====================
|
||||
|
||||
Each vdo volume reserves 3GB of space for metadata, or more depending on
|
||||
its configuration. It is helpful to check that the space saved by
|
||||
deduplication and compression is not cancelled out by the metadata
|
||||
requirements. An estimation of the space saved for a specific dataset can
|
||||
be computed with the vdo estimator tool, which is available at:
|
||||
|
||||
https://github.com/dm-vdo/vdoestimator/
|
||||
|
||||
Target interface
|
||||
================
|
||||
|
||||
Table line
|
||||
----------
|
||||
|
||||
::
|
||||
|
||||
<offset> <logical device size> vdo V4 <storage device>
|
||||
<storage device size> <minimum I/O size> <block map cache size>
|
||||
<block map era length> [optional arguments]
|
||||
|
||||
|
||||
Required parameters:
|
||||
|
||||
offset:
|
||||
The offset, in sectors, at which the vdo volume's logical
|
||||
space begins.
|
||||
|
||||
logical device size:
|
||||
The size of the device which the vdo volume will service,
|
||||
in sectors. Must match the current logical size of the vdo
|
||||
volume.
|
||||
|
||||
storage device:
|
||||
The device holding the vdo volume's data and metadata.
|
||||
|
||||
storage device size:
|
||||
The size of the device holding the vdo volume, as a number
|
||||
of 4096-byte blocks. Must match the current size of the vdo
|
||||
volume.
|
||||
|
||||
minimum I/O size:
|
||||
The minimum I/O size for this vdo volume to accept, in
|
||||
bytes. Valid values are 512 or 4096. The recommended value
|
||||
is 4096.
|
||||
|
||||
block map cache size:
|
||||
The size of the block map cache, as a number of 4096-byte
|
||||
blocks. The minimum and recommended value is 32768 blocks.
|
||||
If the logical thread count is non-zero, the cache size
|
||||
must be at least 4096 blocks per logical thread.
|
||||
|
||||
block map era length:
|
||||
The speed with which the block map cache writes out
|
||||
modified block map pages. A smaller era length is likely to
|
||||
reduce the amount of time spent rebuilding, at the cost of
|
||||
increased block map writes during normal operation. The
|
||||
maximum and recommended value is 16380; the minimum value
|
||||
is 1.
|
||||
|
||||
Optional parameters:
|
||||
--------------------
|
||||
Some or all of these parameters may be specified as <key> <value> pairs.
|
||||
|
||||
Thread related parameters:
|
||||
|
||||
Different categories of work are assigned to separate thread groups, and
|
||||
the number of threads in each group can be configured separately.
|
||||
|
||||
If <hash>, <logical>, and <physical> are all set to 0, the work handled by
|
||||
all three thread types will be handled by a single thread. If any of these
|
||||
values are non-zero, all of them must be non-zero.
|
||||
|
||||
ack:
|
||||
The number of threads used to complete bios. Since
|
||||
completing a bio calls an arbitrary completion function
|
||||
outside the vdo volume, threads of this type allow the vdo
|
||||
volume to continue processing requests even when bio
|
||||
completion is slow. The default is 1.
|
||||
|
||||
bio:
|
||||
The number of threads used to issue bios to the underlying
|
||||
storage. Threads of this type allow the vdo volume to
|
||||
continue processing requests even when bio submission is
|
||||
slow. The default is 4.
|
||||
|
||||
bioRotationInterval:
|
||||
The number of bios to enqueue on each bio thread before
|
||||
switching to the next thread. The value must be greater
|
||||
than 0 and not more than 1024; the default is 64.
|
||||
|
||||
cpu:
|
||||
The number of threads used to do CPU-intensive work, such
|
||||
as hashing and compression. The default is 1.
|
||||
|
||||
hash:
|
||||
The number of threads used to manage data comparisons for
|
||||
deduplication based on the hash value of data blocks. The
|
||||
default is 0.
|
||||
|
||||
logical:
|
||||
The number of threads used to manage caching and locking
|
||||
based on the logical address of incoming bios. The default
|
||||
is 0; the maximum is 60.
|
||||
|
||||
physical:
|
||||
The number of threads used to manage administration of the
|
||||
underlying storage device. At format time, a slab size for
|
||||
the vdo is chosen; the vdo storage device must be large
|
||||
enough to have at least 1 slab per physical thread. The
|
||||
default is 0; the maximum is 16.
|
||||
|
||||
Miscellaneous parameters:
|
||||
|
||||
maxDiscard:
|
||||
The maximum size of discard bio accepted, in 4096-byte
|
||||
blocks. I/O requests to a vdo volume are normally split
|
||||
into 4096-byte blocks, and processed up to 2048 at a time.
|
||||
However, discard requests to a vdo volume can be
|
||||
automatically split to a larger size, up to <maxDiscard>
|
||||
4096-byte blocks in a single bio, and are limited to 1500
|
||||
at a time. Increasing this value may provide better overall
|
||||
performance, at the cost of increased latency for the
|
||||
individual discard requests. The default and minimum is 1;
|
||||
the maximum is UINT_MAX / 4096.
|
||||
|
||||
deduplication:
|
||||
Whether deduplication is enabled. The default is 'on'; the
|
||||
acceptable values are 'on' and 'off'.
|
||||
|
||||
compression:
|
||||
Whether compression is enabled. The default is 'off'; the
|
||||
acceptable values are 'on' and 'off'.
|
||||
|
||||
Device modification
|
||||
-------------------
|
||||
|
||||
A modified table may be loaded into a running, non-suspended vdo volume.
|
||||
The modifications will take effect when the device is next resumed. The
|
||||
modifiable parameters are <logical device size>, <physical device size>,
|
||||
<maxDiscard>, <compression>, and <deduplication>.
|
||||
|
||||
If the logical device size or physical device size are changed, upon
|
||||
successful resume vdo will store the new values and require them on future
|
||||
startups. These two parameters may not be decreased. The logical device
|
||||
size may not exceed 4 PB. The physical device size must increase by at
|
||||
least 32832 4096-byte blocks if at all, and must not exceed the size of the
|
||||
underlying storage device. Additionally, when formatting the vdo device, a
|
||||
slab size is chosen: the physical device size may never increase above the
|
||||
size which provides 8192 slabs, and each increase must be large enough to
|
||||
add at least one new slab.
|
||||
|
||||
Examples:
|
||||
|
||||
Start a previously-formatted vdo volume with 1 GB logical space and 1 GB
|
||||
physical space, storing to /dev/dm-1 which has more than 1 GB of space.
|
||||
|
||||
::
|
||||
|
||||
dmsetup create vdo0 --table \
|
||||
"0 2097152 vdo V4 /dev/dm-1 262144 4096 32768 16380"
|
||||
|
||||
Grow the logical size to 4 GB.
|
||||
|
||||
::
|
||||
|
||||
dmsetup reload vdo0 --table \
|
||||
"0 8388608 vdo V4 /dev/dm-1 262144 4096 32768 16380"
|
||||
dmsetup resume vdo0
|
||||
|
||||
Grow the physical size to 2 GB.
|
||||
|
||||
::
|
||||
|
||||
dmsetup reload vdo0 --table \
|
||||
"0 8388608 vdo V4 /dev/dm-1 524288 4096 32768 16380"
|
||||
dmsetup resume vdo0
|
||||
|
||||
Grow the physical size by 1 GB more and increase max discard sectors.
|
||||
|
||||
::
|
||||
|
||||
dmsetup reload vdo0 --table \
|
||||
"0 10485760 vdo V4 /dev/dm-1 786432 4096 32768 16380 maxDiscard 8"
|
||||
dmsetup resume vdo0
|
||||
|
||||
Stop the vdo volume.
|
||||
|
||||
::
|
||||
|
||||
dmsetup remove vdo0
|
||||
|
||||
Start the vdo volume again. Note that the logical and physical device sizes
|
||||
must still match, but other parameters can change.
|
||||
|
||||
::
|
||||
|
||||
dmsetup create vdo1 --table \
|
||||
"0 10485760 vdo V4 /dev/dm-1 786432 512 65550 5000 hash 1 logical 3 physical 2"
|
||||
|
||||
Messages
|
||||
--------
|
||||
All vdo devices accept messages in the form:
|
||||
|
||||
::
|
||||
dmsetup message <target-name> 0 <message-name> <message-parameters>
|
||||
|
||||
The messages are:
|
||||
|
||||
stats:
|
||||
Outputs the current view of the vdo statistics. Mostly used
|
||||
by the vdostats userspace program to interpret the output
|
||||
buffer.
|
||||
|
||||
dump:
|
||||
Dumps many internal structures to the system log. This is
|
||||
not always safe to run, so it should only be used to debug
|
||||
a hung vdo. Optional parameters to specify structures to
|
||||
dump are:
|
||||
|
||||
viopool: The pool of I/O requests incoming bios
|
||||
pools: A synonym of 'viopool'
|
||||
vdo: Most of the structures managing on-disk data
|
||||
queues: Basic information about each vdo thread
|
||||
threads: A synonym of 'queues'
|
||||
default: Equivalent to 'queues vdo'
|
||||
all: All of the above.
|
||||
|
||||
dump-on-shutdown:
|
||||
Perform a default dump next time vdo shuts down.
|
||||
|
||||
|
||||
Status
|
||||
------
|
||||
|
||||
::
|
||||
|
||||
<device> <operating mode> <in recovery> <index state>
|
||||
<compression state> <physical blocks used> <total physical blocks>
|
||||
|
||||
device:
|
||||
The name of the vdo volume.
|
||||
|
||||
operating mode:
|
||||
The current operating mode of the vdo volume; values may be
|
||||
'normal', 'recovering' (the volume has detected an issue
|
||||
with its metadata and is attempting to repair itself), and
|
||||
'read-only' (an error has occurred that forces the vdo
|
||||
volume to only support read operations and not writes).
|
||||
|
||||
in recovery:
|
||||
Whether the vdo volume is currently in recovery mode;
|
||||
values may be 'recovering' or '-' which indicates not
|
||||
recovering.
|
||||
|
||||
index state:
|
||||
The current state of the deduplication index in the vdo
|
||||
volume; values may be 'closed', 'closing', 'error',
|
||||
'offline', 'online', 'opening', and 'unknown'.
|
||||
|
||||
compression state:
|
||||
The current state of compression in the vdo volume; values
|
||||
may be 'offline' and 'online'.
|
||||
|
||||
used physical blocks:
|
||||
The number of physical blocks in use by the vdo volume.
|
||||
|
||||
total physical blocks:
|
||||
The total number of physical blocks the vdo volume may use;
|
||||
the difference between this value and the
|
||||
<used physical blocks> is the number of blocks the vdo
|
||||
volume has left before being full.
|
||||
|
||||
Memory Requirements
|
||||
===================
|
||||
|
||||
A vdo target requires a fixed 38 MB of RAM along with the following amounts
|
||||
that scale with the target:
|
||||
|
||||
- 1.15 MB of RAM for each 1 MB of configured block map cache size. The
|
||||
block map cache requires a minimum of 150 MB.
|
||||
- 1.6 MB of RAM for each 1 TB of logical space.
|
||||
- 268 MB of RAM for each 1 TB of physical storage managed by the volume.
|
||||
|
||||
The deduplication index requires additional memory which scales with the
|
||||
size of the deduplication window. For dense indexes, the index requires 1
|
||||
GB of RAM per 1 TB of window. For sparse indexes, the index requires 1 GB
|
||||
of RAM per 10 TB of window. The index configuration is set when the target
|
||||
is formatted and may not be modified.
|
||||
|
||||
Module Parameters
|
||||
=================
|
||||
|
||||
The vdo driver has a numeric parameter 'log_level' which controls the
|
||||
verbosity of logging from the driver. The default setting is 6
|
||||
(LOGLEVEL_INFO and more severe messages).
|
||||
|
||||
Run-time Usage
|
||||
==============
|
||||
|
||||
When using dm-vdo, it is important to be aware of the ways in which its
|
||||
behavior differs from other storage targets.
|
||||
|
||||
- There is no guarantee that over-writes of existing blocks will succeed.
|
||||
Because the underlying storage may be multiply referenced, over-writing
|
||||
an existing block generally requires a vdo to have a free block
|
||||
available.
|
||||
|
||||
- When blocks are no longer in use, sending a discard request for those
|
||||
blocks lets the vdo release references for those blocks. If the vdo is
|
||||
thinly provisioned, discarding unused blocks is essential to prevent the
|
||||
target from running out of space. However, due to the sharing of
|
||||
duplicate blocks, no discard request for any given logical block is
|
||||
guaranteed to reclaim space.
|
||||
|
||||
- Assuming the underlying storage properly implements flush requests, vdo
|
||||
is resilient against crashes, however, unflushed writes may or may not
|
||||
persist after a crash.
|
||||
|
||||
- Each write to a vdo target entails a significant amount of processing.
|
||||
However, much of the work is paralellizable. Therefore, vdo targets
|
||||
achieve better throughput at higher I/O depths, and can support up 2048
|
||||
requests in parallel.
|
||||
|
||||
Tuning
|
||||
======
|
||||
|
||||
The vdo device has many options, and it can be difficult to make optimal
|
||||
choices without perfect knowledge of the workload. Additionally, most
|
||||
configuration options must be set when a vdo target is started, and cannot
|
||||
be changed without shutting it down completely; the configuration cannot be
|
||||
changed while the target is active. Ideally, tuning with simulated
|
||||
workloads should be performed before deploying vdo in production
|
||||
environments.
|
||||
|
||||
The most important value to adjust is the block map cache size. In order to
|
||||
service a request for any logical address, a vdo must load the portion of
|
||||
the block map which holds the relevant mapping. These mappings are cached.
|
||||
Performance will suffer when the working set does not fit in the cache. By
|
||||
default, a vdo allocates 128 MB of metadata cache in RAM to support
|
||||
efficient access to 100 GB of logical space at a time. It should be scaled
|
||||
up proportionally for larger working sets.
|
||||
|
||||
The logical and physical thread counts should also be adjusted. A logical
|
||||
thread controls a disjoint section of the block map, so additional logical
|
||||
threads increase parallelism and can increase throughput. Physical threads
|
||||
control a disjoint section of the data blocks, so additional physical
|
||||
threads can also increase throughput. However, excess threads can waste
|
||||
resources and increase contention.
|
||||
|
||||
Bio submission threads control the parallelism involved in sending I/O to
|
||||
the underlying storage; fewer threads mean there is more opportunity to
|
||||
reorder I/O requests for performance benefit, but also that each I/O
|
||||
request has to wait longer before being submitted.
|
||||
|
||||
Bio acknowledgment threads are used for finishing I/O requests. This is
|
||||
done on dedicated threads since the amount of work required to execute a
|
||||
bio's callback can not be controlled by the vdo itself. Usually one thread
|
||||
is sufficient but additional threads may be beneficial, particularly when
|
||||
bios have CPU-heavy callbacks.
|
||||
|
||||
CPU threads are used for hashing and for compression; in workloads with
|
||||
compression enabled, more threads may result in higher throughput.
|
||||
|
||||
Hash threads are used to sort active requests by hash and determine whether
|
||||
they should deduplicate; the most CPU intensive actions done by these
|
||||
threads are comparison of 4096-byte data blocks. In most cases, a single
|
||||
hash thread is sufficient.
|
|
@ -6134,6 +6134,14 @@ F: include/linux/device-mapper.h
|
|||
F: include/linux/dm-*.h
|
||||
F: include/uapi/linux/dm-*.h
|
||||
|
||||
DEVICE-MAPPER VDO TARGET
|
||||
M: Matthew Sakai <msakai@redhat.com>
|
||||
M: dm-devel@lists.linux.dev
|
||||
L: dm-devel@lists.linux.dev
|
||||
S: Maintained
|
||||
F: Documentation/admin-guide/device-mapper/vdo*.rst
|
||||
F: drivers/md/dm-vdo/
|
||||
|
||||
DEVLINK
|
||||
M: Jiri Pirko <jiri@resnulli.us>
|
||||
L: netdev@vger.kernel.org
|
||||
|
|
|
@ -634,4 +634,6 @@ config DM_AUDIT
|
|||
Enables audit logging of several security relevant events in the
|
||||
particular device-mapper targets, especially the integrity target.
|
||||
|
||||
source "drivers/md/dm-vdo/Kconfig"
|
||||
|
||||
endif # MD
|
||||
|
|
|
@ -68,6 +68,7 @@ obj-$(CONFIG_DM_ZERO) += dm-zero.o
|
|||
obj-$(CONFIG_DM_RAID) += dm-raid.o
|
||||
obj-$(CONFIG_DM_THIN_PROVISIONING) += dm-thin-pool.o
|
||||
obj-$(CONFIG_DM_VERITY) += dm-verity.o
|
||||
obj-$(CONFIG_DM_VDO) += dm-vdo/
|
||||
obj-$(CONFIG_DM_CACHE) += dm-cache.o
|
||||
obj-$(CONFIG_DM_CACHE_SMQ) += dm-cache-smq.o
|
||||
obj-$(CONFIG_DM_EBS) += dm-ebs.o
|
||||
|
|
|
@ -0,0 +1,17 @@
|
|||
# SPDX-License-Identifier: GPL-2.0-only
|
||||
|
||||
config DM_VDO
|
||||
tristate "VDO: deduplication and compression target"
|
||||
depends on 64BIT
|
||||
depends on BLK_DEV_DM
|
||||
select DM_BUFIO
|
||||
select LZ4_COMPRESS
|
||||
select LZ4_DECOMPRESS
|
||||
help
|
||||
This device mapper target presents a block device with
|
||||
deduplication, compression and thin-provisioning.
|
||||
|
||||
To compile this code as a module, choose M here: the module will
|
||||
be called dm-vdo.
|
||||
|
||||
If unsure, say N.
|
|
@ -0,0 +1,57 @@
|
|||
# SPDX-License-Identifier: GPL-2.0-only
|
||||
|
||||
ccflags-y := -I$(srctree)/$(src) -I$(srctree)/$(src)/indexer
|
||||
|
||||
obj-$(CONFIG_DM_VDO) += dm-vdo.o
|
||||
|
||||
dm-vdo-objs := \
|
||||
action-manager.o \
|
||||
admin-state.o \
|
||||
block-map.o \
|
||||
completion.o \
|
||||
data-vio.o \
|
||||
dedupe.o \
|
||||
dm-vdo-target.o \
|
||||
dump.o \
|
||||
encodings.o \
|
||||
errors.o \
|
||||
flush.o \
|
||||
funnel-queue.o \
|
||||
funnel-workqueue.o \
|
||||
int-map.o \
|
||||
io-submitter.o \
|
||||
logger.o \
|
||||
logical-zone.o \
|
||||
memory-alloc.o \
|
||||
message-stats.o \
|
||||
murmurhash3.o \
|
||||
packer.o \
|
||||
permassert.o \
|
||||
physical-zone.o \
|
||||
priority-table.o \
|
||||
recovery-journal.o \
|
||||
repair.o \
|
||||
slab-depot.o \
|
||||
status-codes.o \
|
||||
string-utils.o \
|
||||
thread-device.o \
|
||||
thread-registry.o \
|
||||
thread-utils.o \
|
||||
vdo.o \
|
||||
vio.o \
|
||||
wait-queue.o \
|
||||
indexer/chapter-index.o \
|
||||
indexer/config.o \
|
||||
indexer/delta-index.o \
|
||||
indexer/funnel-requestqueue.o \
|
||||
indexer/geometry.o \
|
||||
indexer/index.o \
|
||||
indexer/index-layout.o \
|
||||
indexer/index-page-map.o \
|
||||
indexer/index-session.o \
|
||||
indexer/io-factory.o \
|
||||
indexer/open-chapter.o \
|
||||
indexer/radix-sort.o \
|
||||
indexer/sparse-cache.o \
|
||||
indexer/volume.o \
|
||||
indexer/volume-index.o
|
|
@ -0,0 +1,388 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "action-manager.h"
|
||||
|
||||
#include "memory-alloc.h"
|
||||
#include "permassert.h"
|
||||
|
||||
#include "admin-state.h"
|
||||
#include "completion.h"
|
||||
#include "status-codes.h"
|
||||
#include "types.h"
|
||||
#include "vdo.h"
|
||||
|
||||
/**
|
||||
* struct action - An action to be performed in each of a set of zones.
|
||||
* @in_use: Whether this structure is in use.
|
||||
* @operation: The admin operation associated with this action.
|
||||
* @preamble: The method to run on the initiator thread before the action is applied to each zone.
|
||||
* @zone_action: The action to be performed in each zone.
|
||||
* @conclusion: The method to run on the initiator thread after the action is applied to each zone.
|
||||
* @parent: The object to notify when the action is complete.
|
||||
* @context: The action specific context.
|
||||
* @next: The action to perform after this one.
|
||||
*/
|
||||
struct action {
|
||||
bool in_use;
|
||||
const struct admin_state_code *operation;
|
||||
vdo_action_preamble_fn preamble;
|
||||
vdo_zone_action_fn zone_action;
|
||||
vdo_action_conclusion_fn conclusion;
|
||||
struct vdo_completion *parent;
|
||||
void *context;
|
||||
struct action *next;
|
||||
};
|
||||
|
||||
/**
|
||||
* struct action_manager - Definition of an action manager.
|
||||
* @completion: The completion for performing actions.
|
||||
* @state: The state of this action manager.
|
||||
* @actions: The two action slots.
|
||||
* @current_action: The current action slot.
|
||||
* @zones: The number of zones in which an action is to be applied.
|
||||
* @Scheduler: A function to schedule a default next action.
|
||||
* @get_zone_thread_id: A function to get the id of the thread on which to apply an action to a
|
||||
* zone.
|
||||
* @initiator_thread_id: The ID of the thread on which actions may be initiated.
|
||||
* @context: Opaque data associated with this action manager.
|
||||
* @acting_zone: The zone currently being acted upon.
|
||||
*/
|
||||
struct action_manager {
|
||||
struct vdo_completion completion;
|
||||
struct admin_state state;
|
||||
struct action actions[2];
|
||||
struct action *current_action;
|
||||
zone_count_t zones;
|
||||
vdo_action_scheduler_fn scheduler;
|
||||
vdo_zone_thread_getter_fn get_zone_thread_id;
|
||||
thread_id_t initiator_thread_id;
|
||||
void *context;
|
||||
zone_count_t acting_zone;
|
||||
};
|
||||
|
||||
static inline struct action_manager *as_action_manager(struct vdo_completion *completion)
|
||||
{
|
||||
vdo_assert_completion_type(completion, VDO_ACTION_COMPLETION);
|
||||
return container_of(completion, struct action_manager, completion);
|
||||
}
|
||||
|
||||
/* Implements vdo_action_scheduler_fn. */
|
||||
static bool no_default_action(void *context __always_unused)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
/* Implements vdo_action_preamble_fn. */
|
||||
static void no_preamble(void *context __always_unused, struct vdo_completion *completion)
|
||||
{
|
||||
vdo_finish_completion(completion);
|
||||
}
|
||||
|
||||
/* Implements vdo_action_conclusion_fn. */
|
||||
static int no_conclusion(void *context __always_unused)
|
||||
{
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_make_action_manager() - Make an action manager.
|
||||
* @zones: The number of zones to which actions will be applied.
|
||||
* @get_zone_thread_id: A function to get the thread id associated with a zone.
|
||||
* @initiator_thread_id: The thread on which actions may initiated.
|
||||
* @context: The object which holds the per-zone context for the action.
|
||||
* @scheduler: A function to schedule a next action after an action concludes if there is no
|
||||
* pending action (may be NULL).
|
||||
* @vdo: The vdo used to initialize completions.
|
||||
* @manager_ptr: A pointer to hold the new action manager.
|
||||
*
|
||||
* Return: VDO_SUCCESS or an error code.
|
||||
*/
|
||||
int vdo_make_action_manager(zone_count_t zones,
|
||||
vdo_zone_thread_getter_fn get_zone_thread_id,
|
||||
thread_id_t initiator_thread_id, void *context,
|
||||
vdo_action_scheduler_fn scheduler, struct vdo *vdo,
|
||||
struct action_manager **manager_ptr)
|
||||
{
|
||||
struct action_manager *manager;
|
||||
int result = vdo_allocate(1, struct action_manager, __func__, &manager);
|
||||
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
*manager = (struct action_manager) {
|
||||
.zones = zones,
|
||||
.scheduler =
|
||||
((scheduler == NULL) ? no_default_action : scheduler),
|
||||
.get_zone_thread_id = get_zone_thread_id,
|
||||
.initiator_thread_id = initiator_thread_id,
|
||||
.context = context,
|
||||
};
|
||||
|
||||
manager->actions[0].next = &manager->actions[1];
|
||||
manager->current_action = manager->actions[1].next =
|
||||
&manager->actions[0];
|
||||
vdo_set_admin_state_code(&manager->state, VDO_ADMIN_STATE_NORMAL_OPERATION);
|
||||
vdo_initialize_completion(&manager->completion, vdo, VDO_ACTION_COMPLETION);
|
||||
*manager_ptr = manager;
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
const struct admin_state_code *vdo_get_current_manager_operation(struct action_manager *manager)
|
||||
{
|
||||
return vdo_get_admin_state_code(&manager->state);
|
||||
}
|
||||
|
||||
void *vdo_get_current_action_context(struct action_manager *manager)
|
||||
{
|
||||
return manager->current_action->in_use ? manager->current_action->context : NULL;
|
||||
}
|
||||
|
||||
static void finish_action_callback(struct vdo_completion *completion);
|
||||
static void apply_to_zone(struct vdo_completion *completion);
|
||||
|
||||
static thread_id_t get_acting_zone_thread_id(struct action_manager *manager)
|
||||
{
|
||||
return manager->get_zone_thread_id(manager->context, manager->acting_zone);
|
||||
}
|
||||
|
||||
static void preserve_error(struct vdo_completion *completion)
|
||||
{
|
||||
if (completion->parent != NULL)
|
||||
vdo_set_completion_result(completion->parent, completion->result);
|
||||
|
||||
vdo_reset_completion(completion);
|
||||
vdo_run_completion(completion);
|
||||
}
|
||||
|
||||
static void prepare_for_next_zone(struct action_manager *manager)
|
||||
{
|
||||
vdo_prepare_completion_for_requeue(&manager->completion, apply_to_zone,
|
||||
preserve_error,
|
||||
get_acting_zone_thread_id(manager),
|
||||
manager->current_action->parent);
|
||||
}
|
||||
|
||||
static void prepare_for_conclusion(struct action_manager *manager)
|
||||
{
|
||||
vdo_prepare_completion_for_requeue(&manager->completion, finish_action_callback,
|
||||
preserve_error, manager->initiator_thread_id,
|
||||
manager->current_action->parent);
|
||||
}
|
||||
|
||||
static void apply_to_zone(struct vdo_completion *completion)
|
||||
{
|
||||
zone_count_t zone;
|
||||
struct action_manager *manager = as_action_manager(completion);
|
||||
|
||||
VDO_ASSERT_LOG_ONLY((vdo_get_callback_thread_id() == get_acting_zone_thread_id(manager)),
|
||||
"%s() called on acting zones's thread", __func__);
|
||||
|
||||
zone = manager->acting_zone++;
|
||||
if (manager->acting_zone == manager->zones) {
|
||||
/*
|
||||
* We are about to apply to the last zone. Once that is finished, we're done, so go
|
||||
* back to the initiator thread and finish up.
|
||||
*/
|
||||
prepare_for_conclusion(manager);
|
||||
} else {
|
||||
/* Prepare to come back on the next zone */
|
||||
prepare_for_next_zone(manager);
|
||||
}
|
||||
|
||||
manager->current_action->zone_action(manager->context, zone, completion);
|
||||
}
|
||||
|
||||
static void handle_preamble_error(struct vdo_completion *completion)
|
||||
{
|
||||
/* Skip the zone actions since the preamble failed. */
|
||||
completion->callback = finish_action_callback;
|
||||
preserve_error(completion);
|
||||
}
|
||||
|
||||
static void launch_current_action(struct action_manager *manager)
|
||||
{
|
||||
struct action *action = manager->current_action;
|
||||
int result = vdo_start_operation(&manager->state, action->operation);
|
||||
|
||||
if (result != VDO_SUCCESS) {
|
||||
if (action->parent != NULL)
|
||||
vdo_set_completion_result(action->parent, result);
|
||||
|
||||
/* We aren't going to run the preamble, so don't run the conclusion */
|
||||
action->conclusion = no_conclusion;
|
||||
finish_action_callback(&manager->completion);
|
||||
return;
|
||||
}
|
||||
|
||||
if (action->zone_action == NULL) {
|
||||
prepare_for_conclusion(manager);
|
||||
} else {
|
||||
manager->acting_zone = 0;
|
||||
vdo_prepare_completion_for_requeue(&manager->completion, apply_to_zone,
|
||||
handle_preamble_error,
|
||||
get_acting_zone_thread_id(manager),
|
||||
manager->current_action->parent);
|
||||
}
|
||||
|
||||
action->preamble(manager->context, &manager->completion);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_schedule_default_action() - Attempt to schedule the default action.
|
||||
* @manager: The action manager.
|
||||
*
|
||||
* If the manager is not operating normally, the action will not be scheduled.
|
||||
*
|
||||
* Return: true if an action was scheduled.
|
||||
*/
|
||||
bool vdo_schedule_default_action(struct action_manager *manager)
|
||||
{
|
||||
/* Don't schedule a default action if we are operating or not in normal operation. */
|
||||
const struct admin_state_code *code = vdo_get_current_manager_operation(manager);
|
||||
|
||||
return ((code == VDO_ADMIN_STATE_NORMAL_OPERATION) &&
|
||||
manager->scheduler(manager->context));
|
||||
}
|
||||
|
||||
static void finish_action_callback(struct vdo_completion *completion)
|
||||
{
|
||||
bool has_next_action;
|
||||
int result;
|
||||
struct action_manager *manager = as_action_manager(completion);
|
||||
struct action action = *(manager->current_action);
|
||||
|
||||
manager->current_action->in_use = false;
|
||||
manager->current_action = manager->current_action->next;
|
||||
|
||||
/*
|
||||
* We need to check this now to avoid use-after-free issues if running the conclusion or
|
||||
* notifying the parent results in the manager being freed.
|
||||
*/
|
||||
has_next_action =
|
||||
(manager->current_action->in_use || vdo_schedule_default_action(manager));
|
||||
result = action.conclusion(manager->context);
|
||||
vdo_finish_operation(&manager->state, VDO_SUCCESS);
|
||||
if (action.parent != NULL)
|
||||
vdo_continue_completion(action.parent, result);
|
||||
|
||||
if (has_next_action)
|
||||
launch_current_action(manager);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_schedule_action() - Schedule an action to be applied to all zones.
|
||||
* @manager: The action manager to schedule the action on.
|
||||
* @preamble: A method to be invoked on the initiator thread once this action is started but before
|
||||
* applying to each zone; may be NULL.
|
||||
* @action: The action to apply to each zone; may be NULL.
|
||||
* @conclusion: A method to be invoked back on the initiator thread once the action has been
|
||||
* applied to all zones; may be NULL.
|
||||
* @parent: The object to notify once the action is complete or if the action can not be scheduled;
|
||||
* may be NULL.
|
||||
*
|
||||
* The action will be launched immediately if there is no current action, or as soon as the current
|
||||
* action completes. If there is already a pending action, this action will not be scheduled, and,
|
||||
* if it has a parent, that parent will be notified. At least one of the preamble, action, or
|
||||
* conclusion must not be NULL.
|
||||
*
|
||||
* Return: true if the action was scheduled.
|
||||
*/
|
||||
bool vdo_schedule_action(struct action_manager *manager, vdo_action_preamble_fn preamble,
|
||||
vdo_zone_action_fn action, vdo_action_conclusion_fn conclusion,
|
||||
struct vdo_completion *parent)
|
||||
{
|
||||
return vdo_schedule_operation(manager, VDO_ADMIN_STATE_OPERATING, preamble,
|
||||
action, conclusion, parent);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_schedule_operation() - Schedule an operation to be applied to all zones.
|
||||
* @manager: The action manager to schedule the action on.
|
||||
* @operation: The operation this action will perform
|
||||
* @preamble: A method to be invoked on the initiator thread once this action is started but before
|
||||
* applying to each zone; may be NULL.
|
||||
* @action: The action to apply to each zone; may be NULL.
|
||||
* @conclusion: A method to be invoked back on the initiator thread once the action has been
|
||||
* applied to all zones; may be NULL.
|
||||
* @parent: The object to notify once the action is complete or if the action can not be scheduled;
|
||||
* may be NULL.
|
||||
*
|
||||
* The operation's action will be launched immediately if there is no current action, or as soon as
|
||||
* the current action completes. If there is already a pending action, this operation will not be
|
||||
* scheduled, and, if it has a parent, that parent will be notified. At least one of the preamble,
|
||||
* action, or conclusion must not be NULL.
|
||||
*
|
||||
* Return: true if the action was scheduled.
|
||||
*/
|
||||
bool vdo_schedule_operation(struct action_manager *manager,
|
||||
const struct admin_state_code *operation,
|
||||
vdo_action_preamble_fn preamble, vdo_zone_action_fn action,
|
||||
vdo_action_conclusion_fn conclusion,
|
||||
struct vdo_completion *parent)
|
||||
{
|
||||
return vdo_schedule_operation_with_context(manager, operation, preamble, action,
|
||||
conclusion, NULL, parent);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_schedule_operation_with_context() - Schedule an operation on all zones.
|
||||
* @manager: The action manager to schedule the action on.
|
||||
* @operation: The operation this action will perform.
|
||||
* @preamble: A method to be invoked on the initiator thread once this action is started but before
|
||||
* applying to each zone; may be NULL.
|
||||
* @action: The action to apply to each zone; may be NULL.
|
||||
* @conclusion: A method to be invoked back on the initiator thread once the action has been
|
||||
* applied to all zones; may be NULL.
|
||||
* @context: An action-specific context which may be retrieved via
|
||||
* vdo_get_current_action_context(); may be NULL.
|
||||
* @parent: The object to notify once the action is complete or if the action can not be scheduled;
|
||||
* may be NULL.
|
||||
*
|
||||
* The operation's action will be launched immediately if there is no current action, or as soon as
|
||||
* the current action completes. If there is already a pending action, this operation will not be
|
||||
* scheduled, and, if it has a parent, that parent will be notified. At least one of the preamble,
|
||||
* action, or conclusion must not be NULL.
|
||||
*
|
||||
* Return: true if the action was scheduled
|
||||
*/
|
||||
bool vdo_schedule_operation_with_context(struct action_manager *manager,
|
||||
const struct admin_state_code *operation,
|
||||
vdo_action_preamble_fn preamble,
|
||||
vdo_zone_action_fn action,
|
||||
vdo_action_conclusion_fn conclusion,
|
||||
void *context, struct vdo_completion *parent)
|
||||
{
|
||||
struct action *current_action;
|
||||
|
||||
VDO_ASSERT_LOG_ONLY((vdo_get_callback_thread_id() == manager->initiator_thread_id),
|
||||
"action initiated from correct thread");
|
||||
if (!manager->current_action->in_use) {
|
||||
current_action = manager->current_action;
|
||||
} else if (!manager->current_action->next->in_use) {
|
||||
current_action = manager->current_action->next;
|
||||
} else {
|
||||
if (parent != NULL)
|
||||
vdo_continue_completion(parent, VDO_COMPONENT_BUSY);
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
*current_action = (struct action) {
|
||||
.in_use = true,
|
||||
.operation = operation,
|
||||
.preamble = (preamble == NULL) ? no_preamble : preamble,
|
||||
.zone_action = action,
|
||||
.conclusion = (conclusion == NULL) ? no_conclusion : conclusion,
|
||||
.context = context,
|
||||
.parent = parent,
|
||||
.next = current_action->next,
|
||||
};
|
||||
|
||||
if (current_action == manager->current_action)
|
||||
launch_current_action(manager);
|
||||
|
||||
return true;
|
||||
}
|
|
@ -0,0 +1,110 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef VDO_ACTION_MANAGER_H
|
||||
#define VDO_ACTION_MANAGER_H
|
||||
|
||||
#include "admin-state.h"
|
||||
#include "types.h"
|
||||
|
||||
/*
|
||||
* An action_manager provides a generic mechanism for applying actions to multi-zone entities (such
|
||||
* as the block map or slab depot). Each action manager is tied to a specific context for which it
|
||||
* manages actions. The manager ensures that only one action is active on that context at a time,
|
||||
* and supports at most one pending action. Calls to schedule an action when there is already a
|
||||
* pending action will result in VDO_COMPONENT_BUSY errors. Actions may only be submitted to the
|
||||
* action manager from a single thread (which thread is determined when the action manager is
|
||||
* constructed).
|
||||
*
|
||||
* A scheduled action consists of four components:
|
||||
*
|
||||
* preamble
|
||||
* an optional method to be run on the initiator thread before applying the action to all zones
|
||||
* zone_action
|
||||
* an optional method to be applied to each of the zones
|
||||
* conclusion
|
||||
* an optional method to be run on the initiator thread once the per-zone method has been
|
||||
* applied to all zones
|
||||
* parent
|
||||
* an optional completion to be finished once the conclusion is done
|
||||
*
|
||||
* At least one of the three methods must be provided.
|
||||
*/
|
||||
|
||||
/*
|
||||
* A function which is to be applied asynchronously to a set of zones.
|
||||
* @context: The object which holds the per-zone context for the action
|
||||
* @zone_number: The number of zone to which the action is being applied
|
||||
* @parent: The object to notify when the action is complete
|
||||
*/
|
||||
typedef void (*vdo_zone_action_fn)(void *context, zone_count_t zone_number,
|
||||
struct vdo_completion *parent);
|
||||
|
||||
/*
|
||||
* A function which is to be applied asynchronously on an action manager's initiator thread as the
|
||||
* preamble of an action.
|
||||
* @context: The object which holds the per-zone context for the action
|
||||
* @parent: The object to notify when the action is complete
|
||||
*/
|
||||
typedef void (*vdo_action_preamble_fn)(void *context, struct vdo_completion *parent);
|
||||
|
||||
/*
|
||||
* A function which will run on the action manager's initiator thread as the conclusion of an
|
||||
* action.
|
||||
* @context: The object which holds the per-zone context for the action
|
||||
*
|
||||
* Return: VDO_SUCCESS or an error
|
||||
*/
|
||||
typedef int (*vdo_action_conclusion_fn)(void *context);
|
||||
|
||||
/*
|
||||
* A function to schedule an action.
|
||||
* @context: The object which holds the per-zone context for the action
|
||||
*
|
||||
* Return: true if an action was scheduled
|
||||
*/
|
||||
typedef bool (*vdo_action_scheduler_fn)(void *context);
|
||||
|
||||
/*
|
||||
* A function to get the id of the thread associated with a given zone.
|
||||
* @context: The action context
|
||||
* @zone_number: The number of the zone for which the thread ID is desired
|
||||
*/
|
||||
typedef thread_id_t (*vdo_zone_thread_getter_fn)(void *context, zone_count_t zone_number);
|
||||
|
||||
struct action_manager;
|
||||
|
||||
int __must_check vdo_make_action_manager(zone_count_t zones,
|
||||
vdo_zone_thread_getter_fn get_zone_thread_id,
|
||||
thread_id_t initiator_thread_id, void *context,
|
||||
vdo_action_scheduler_fn scheduler,
|
||||
struct vdo *vdo,
|
||||
struct action_manager **manager_ptr);
|
||||
|
||||
const struct admin_state_code *__must_check
|
||||
vdo_get_current_manager_operation(struct action_manager *manager);
|
||||
|
||||
void * __must_check vdo_get_current_action_context(struct action_manager *manager);
|
||||
|
||||
bool vdo_schedule_default_action(struct action_manager *manager);
|
||||
|
||||
bool vdo_schedule_action(struct action_manager *manager, vdo_action_preamble_fn preamble,
|
||||
vdo_zone_action_fn action, vdo_action_conclusion_fn conclusion,
|
||||
struct vdo_completion *parent);
|
||||
|
||||
bool vdo_schedule_operation(struct action_manager *manager,
|
||||
const struct admin_state_code *operation,
|
||||
vdo_action_preamble_fn preamble, vdo_zone_action_fn action,
|
||||
vdo_action_conclusion_fn conclusion,
|
||||
struct vdo_completion *parent);
|
||||
|
||||
bool vdo_schedule_operation_with_context(struct action_manager *manager,
|
||||
const struct admin_state_code *operation,
|
||||
vdo_action_preamble_fn preamble,
|
||||
vdo_zone_action_fn action,
|
||||
vdo_action_conclusion_fn conclusion,
|
||||
void *context, struct vdo_completion *parent);
|
||||
|
||||
#endif /* VDO_ACTION_MANAGER_H */
|
|
@ -0,0 +1,506 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "admin-state.h"
|
||||
|
||||
#include "logger.h"
|
||||
#include "memory-alloc.h"
|
||||
#include "permassert.h"
|
||||
|
||||
#include "completion.h"
|
||||
#include "types.h"
|
||||
|
||||
static const struct admin_state_code VDO_CODE_NORMAL_OPERATION = {
|
||||
.name = "VDO_ADMIN_STATE_NORMAL_OPERATION",
|
||||
.normal = true,
|
||||
};
|
||||
const struct admin_state_code *VDO_ADMIN_STATE_NORMAL_OPERATION = &VDO_CODE_NORMAL_OPERATION;
|
||||
static const struct admin_state_code VDO_CODE_OPERATING = {
|
||||
.name = "VDO_ADMIN_STATE_OPERATING",
|
||||
.normal = true,
|
||||
.operating = true,
|
||||
};
|
||||
const struct admin_state_code *VDO_ADMIN_STATE_OPERATING = &VDO_CODE_OPERATING;
|
||||
static const struct admin_state_code VDO_CODE_FORMATTING = {
|
||||
.name = "VDO_ADMIN_STATE_FORMATTING",
|
||||
.operating = true,
|
||||
.loading = true,
|
||||
};
|
||||
const struct admin_state_code *VDO_ADMIN_STATE_FORMATTING = &VDO_CODE_FORMATTING;
|
||||
static const struct admin_state_code VDO_CODE_PRE_LOADING = {
|
||||
.name = "VDO_ADMIN_STATE_PRE_LOADING",
|
||||
.operating = true,
|
||||
.loading = true,
|
||||
};
|
||||
const struct admin_state_code *VDO_ADMIN_STATE_PRE_LOADING = &VDO_CODE_PRE_LOADING;
|
||||
static const struct admin_state_code VDO_CODE_PRE_LOADED = {
|
||||
.name = "VDO_ADMIN_STATE_PRE_LOADED",
|
||||
};
|
||||
const struct admin_state_code *VDO_ADMIN_STATE_PRE_LOADED = &VDO_CODE_PRE_LOADED;
|
||||
static const struct admin_state_code VDO_CODE_LOADING = {
|
||||
.name = "VDO_ADMIN_STATE_LOADING",
|
||||
.normal = true,
|
||||
.operating = true,
|
||||
.loading = true,
|
||||
};
|
||||
const struct admin_state_code *VDO_ADMIN_STATE_LOADING = &VDO_CODE_LOADING;
|
||||
static const struct admin_state_code VDO_CODE_LOADING_FOR_RECOVERY = {
|
||||
.name = "VDO_ADMIN_STATE_LOADING_FOR_RECOVERY",
|
||||
.operating = true,
|
||||
.loading = true,
|
||||
};
|
||||
const struct admin_state_code *VDO_ADMIN_STATE_LOADING_FOR_RECOVERY =
|
||||
&VDO_CODE_LOADING_FOR_RECOVERY;
|
||||
static const struct admin_state_code VDO_CODE_LOADING_FOR_REBUILD = {
|
||||
.name = "VDO_ADMIN_STATE_LOADING_FOR_REBUILD",
|
||||
.operating = true,
|
||||
.loading = true,
|
||||
};
|
||||
const struct admin_state_code *VDO_ADMIN_STATE_LOADING_FOR_REBUILD = &VDO_CODE_LOADING_FOR_REBUILD;
|
||||
static const struct admin_state_code VDO_CODE_WAITING_FOR_RECOVERY = {
|
||||
.name = "VDO_ADMIN_STATE_WAITING_FOR_RECOVERY",
|
||||
.operating = true,
|
||||
};
|
||||
const struct admin_state_code *VDO_ADMIN_STATE_WAITING_FOR_RECOVERY =
|
||||
&VDO_CODE_WAITING_FOR_RECOVERY;
|
||||
static const struct admin_state_code VDO_CODE_NEW = {
|
||||
.name = "VDO_ADMIN_STATE_NEW",
|
||||
.quiescent = true,
|
||||
};
|
||||
const struct admin_state_code *VDO_ADMIN_STATE_NEW = &VDO_CODE_NEW;
|
||||
static const struct admin_state_code VDO_CODE_INITIALIZED = {
|
||||
.name = "VDO_ADMIN_STATE_INITIALIZED",
|
||||
};
|
||||
const struct admin_state_code *VDO_ADMIN_STATE_INITIALIZED = &VDO_CODE_INITIALIZED;
|
||||
static const struct admin_state_code VDO_CODE_RECOVERING = {
|
||||
.name = "VDO_ADMIN_STATE_RECOVERING",
|
||||
.draining = true,
|
||||
.operating = true,
|
||||
};
|
||||
const struct admin_state_code *VDO_ADMIN_STATE_RECOVERING = &VDO_CODE_RECOVERING;
|
||||
static const struct admin_state_code VDO_CODE_REBUILDING = {
|
||||
.name = "VDO_ADMIN_STATE_REBUILDING",
|
||||
.draining = true,
|
||||
.operating = true,
|
||||
};
|
||||
const struct admin_state_code *VDO_ADMIN_STATE_REBUILDING = &VDO_CODE_REBUILDING;
|
||||
static const struct admin_state_code VDO_CODE_SAVING = {
|
||||
.name = "VDO_ADMIN_STATE_SAVING",
|
||||
.draining = true,
|
||||
.quiescing = true,
|
||||
.operating = true,
|
||||
};
|
||||
const struct admin_state_code *VDO_ADMIN_STATE_SAVING = &VDO_CODE_SAVING;
|
||||
static const struct admin_state_code VDO_CODE_SAVED = {
|
||||
.name = "VDO_ADMIN_STATE_SAVED",
|
||||
.quiescent = true,
|
||||
};
|
||||
const struct admin_state_code *VDO_ADMIN_STATE_SAVED = &VDO_CODE_SAVED;
|
||||
static const struct admin_state_code VDO_CODE_SCRUBBING = {
|
||||
.name = "VDO_ADMIN_STATE_SCRUBBING",
|
||||
.draining = true,
|
||||
.loading = true,
|
||||
.operating = true,
|
||||
};
|
||||
const struct admin_state_code *VDO_ADMIN_STATE_SCRUBBING = &VDO_CODE_SCRUBBING;
|
||||
static const struct admin_state_code VDO_CODE_SAVE_FOR_SCRUBBING = {
|
||||
.name = "VDO_ADMIN_STATE_SAVE_FOR_SCRUBBING",
|
||||
.draining = true,
|
||||
.operating = true,
|
||||
};
|
||||
const struct admin_state_code *VDO_ADMIN_STATE_SAVE_FOR_SCRUBBING = &VDO_CODE_SAVE_FOR_SCRUBBING;
|
||||
static const struct admin_state_code VDO_CODE_STOPPING = {
|
||||
.name = "VDO_ADMIN_STATE_STOPPING",
|
||||
.draining = true,
|
||||
.quiescing = true,
|
||||
.operating = true,
|
||||
};
|
||||
const struct admin_state_code *VDO_ADMIN_STATE_STOPPING = &VDO_CODE_STOPPING;
|
||||
static const struct admin_state_code VDO_CODE_STOPPED = {
|
||||
.name = "VDO_ADMIN_STATE_STOPPED",
|
||||
.quiescent = true,
|
||||
};
|
||||
const struct admin_state_code *VDO_ADMIN_STATE_STOPPED = &VDO_CODE_STOPPED;
|
||||
static const struct admin_state_code VDO_CODE_SUSPENDING = {
|
||||
.name = "VDO_ADMIN_STATE_SUSPENDING",
|
||||
.draining = true,
|
||||
.quiescing = true,
|
||||
.operating = true,
|
||||
};
|
||||
const struct admin_state_code *VDO_ADMIN_STATE_SUSPENDING = &VDO_CODE_SUSPENDING;
|
||||
static const struct admin_state_code VDO_CODE_SUSPENDED = {
|
||||
.name = "VDO_ADMIN_STATE_SUSPENDED",
|
||||
.quiescent = true,
|
||||
};
|
||||
const struct admin_state_code *VDO_ADMIN_STATE_SUSPENDED = &VDO_CODE_SUSPENDED;
|
||||
static const struct admin_state_code VDO_CODE_SUSPENDED_OPERATION = {
|
||||
.name = "VDO_ADMIN_STATE_SUSPENDED_OPERATION",
|
||||
.operating = true,
|
||||
};
|
||||
const struct admin_state_code *VDO_ADMIN_STATE_SUSPENDED_OPERATION = &VDO_CODE_SUSPENDED_OPERATION;
|
||||
static const struct admin_state_code VDO_CODE_RESUMING = {
|
||||
.name = "VDO_ADMIN_STATE_RESUMING",
|
||||
.operating = true,
|
||||
};
|
||||
const struct admin_state_code *VDO_ADMIN_STATE_RESUMING = &VDO_CODE_RESUMING;
|
||||
|
||||
/**
|
||||
* get_next_state() - Determine the state which should be set after a given operation completes
|
||||
* based on the operation and the current state.
|
||||
* @operation The operation to be started.
|
||||
*
|
||||
* Return: The state to set when the operation completes or NULL if the operation can not be
|
||||
* started in the current state.
|
||||
*/
|
||||
static const struct admin_state_code *get_next_state(const struct admin_state *state,
|
||||
const struct admin_state_code *operation)
|
||||
{
|
||||
const struct admin_state_code *code = vdo_get_admin_state_code(state);
|
||||
|
||||
if (code->operating)
|
||||
return NULL;
|
||||
|
||||
if (operation == VDO_ADMIN_STATE_SAVING)
|
||||
return (code == VDO_ADMIN_STATE_NORMAL_OPERATION ? VDO_ADMIN_STATE_SAVED : NULL);
|
||||
|
||||
if (operation == VDO_ADMIN_STATE_SUSPENDING) {
|
||||
return (code == VDO_ADMIN_STATE_NORMAL_OPERATION
|
||||
? VDO_ADMIN_STATE_SUSPENDED
|
||||
: NULL);
|
||||
}
|
||||
|
||||
if (operation == VDO_ADMIN_STATE_STOPPING)
|
||||
return (code == VDO_ADMIN_STATE_NORMAL_OPERATION ? VDO_ADMIN_STATE_STOPPED : NULL);
|
||||
|
||||
if (operation == VDO_ADMIN_STATE_PRE_LOADING)
|
||||
return (code == VDO_ADMIN_STATE_INITIALIZED ? VDO_ADMIN_STATE_PRE_LOADED : NULL);
|
||||
|
||||
if (operation == VDO_ADMIN_STATE_SUSPENDED_OPERATION) {
|
||||
return (((code == VDO_ADMIN_STATE_SUSPENDED) ||
|
||||
(code == VDO_ADMIN_STATE_SAVED)) ? code : NULL);
|
||||
}
|
||||
|
||||
return VDO_ADMIN_STATE_NORMAL_OPERATION;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_finish_operation() - Finish the current operation.
|
||||
*
|
||||
* Will notify the operation waiter if there is one. This method should be used for operations
|
||||
* started with vdo_start_operation(). For operations which were started with vdo_start_draining(),
|
||||
* use vdo_finish_draining() instead.
|
||||
*
|
||||
* Return: true if there was an operation to finish.
|
||||
*/
|
||||
bool vdo_finish_operation(struct admin_state *state, int result)
|
||||
{
|
||||
if (!vdo_get_admin_state_code(state)->operating)
|
||||
return false;
|
||||
|
||||
state->complete = state->starting;
|
||||
if (state->waiter != NULL)
|
||||
vdo_set_completion_result(state->waiter, result);
|
||||
|
||||
if (!state->starting) {
|
||||
vdo_set_admin_state_code(state, state->next_state);
|
||||
if (state->waiter != NULL)
|
||||
vdo_launch_completion(vdo_forget(state->waiter));
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* begin_operation() - Begin an operation if it may be started given the current state.
|
||||
* @waiter A completion to notify when the operation is complete; may be NULL.
|
||||
* @initiator The vdo_admin_initiator_fn to call if the operation may begin; may be NULL.
|
||||
*
|
||||
* Return: VDO_SUCCESS or an error.
|
||||
*/
|
||||
static int __must_check begin_operation(struct admin_state *state,
|
||||
const struct admin_state_code *operation,
|
||||
struct vdo_completion *waiter,
|
||||
vdo_admin_initiator_fn initiator)
|
||||
{
|
||||
int result;
|
||||
const struct admin_state_code *next_state = get_next_state(state, operation);
|
||||
|
||||
if (next_state == NULL) {
|
||||
result = vdo_log_error_strerror(VDO_INVALID_ADMIN_STATE,
|
||||
"Can't start %s from %s",
|
||||
operation->name,
|
||||
vdo_get_admin_state_code(state)->name);
|
||||
} else if (state->waiter != NULL) {
|
||||
result = vdo_log_error_strerror(VDO_COMPONENT_BUSY,
|
||||
"Can't start %s with extant waiter",
|
||||
operation->name);
|
||||
} else {
|
||||
state->waiter = waiter;
|
||||
state->next_state = next_state;
|
||||
vdo_set_admin_state_code(state, operation);
|
||||
if (initiator != NULL) {
|
||||
state->starting = true;
|
||||
initiator(state);
|
||||
state->starting = false;
|
||||
if (state->complete)
|
||||
vdo_finish_operation(state, VDO_SUCCESS);
|
||||
}
|
||||
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
if (waiter != NULL)
|
||||
vdo_continue_completion(waiter, result);
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* start_operation() - Start an operation if it may be started given the current state.
|
||||
* @waiter A completion to notify when the operation is complete.
|
||||
* @initiator The vdo_admin_initiator_fn to call if the operation may begin; may be NULL.
|
||||
*
|
||||
* Return: true if the operation was started.
|
||||
*/
|
||||
static inline bool __must_check start_operation(struct admin_state *state,
|
||||
const struct admin_state_code *operation,
|
||||
struct vdo_completion *waiter,
|
||||
vdo_admin_initiator_fn initiator)
|
||||
{
|
||||
return (begin_operation(state, operation, waiter, initiator) == VDO_SUCCESS);
|
||||
}
|
||||
|
||||
/**
|
||||
* check_code() - Check the result of a state validation.
|
||||
* @valid true if the code is of an appropriate type.
|
||||
* @code The code which failed to be of the correct type.
|
||||
* @what What the code failed to be, for logging.
|
||||
* @waiter The completion to notify of the error; may be NULL.
|
||||
*
|
||||
* If the result failed, log an invalid state error and, if there is a waiter, notify it.
|
||||
*
|
||||
* Return: The result of the check.
|
||||
*/
|
||||
static bool check_code(bool valid, const struct admin_state_code *code, const char *what,
|
||||
struct vdo_completion *waiter)
|
||||
{
|
||||
int result;
|
||||
|
||||
if (valid)
|
||||
return true;
|
||||
|
||||
result = vdo_log_error_strerror(VDO_INVALID_ADMIN_STATE,
|
||||
"%s is not a %s", code->name, what);
|
||||
if (waiter != NULL)
|
||||
vdo_continue_completion(waiter, result);
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* assert_vdo_drain_operation() - Check that an operation is a drain.
|
||||
* @waiter The completion to finish with an error if the operation is not a drain.
|
||||
*
|
||||
* Return: true if the specified operation is a drain.
|
||||
*/
|
||||
static bool __must_check assert_vdo_drain_operation(const struct admin_state_code *operation,
|
||||
struct vdo_completion *waiter)
|
||||
{
|
||||
return check_code(operation->draining, operation, "drain operation", waiter);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_start_draining() - Initiate a drain operation if the current state permits it.
|
||||
* @operation The type of drain to initiate.
|
||||
* @waiter The completion to notify when the drain is complete.
|
||||
* @initiator The vdo_admin_initiator_fn to call if the operation may begin; may be NULL.
|
||||
*
|
||||
* Return: true if the drain was initiated, if not the waiter will be notified.
|
||||
*/
|
||||
bool vdo_start_draining(struct admin_state *state,
|
||||
const struct admin_state_code *operation,
|
||||
struct vdo_completion *waiter, vdo_admin_initiator_fn initiator)
|
||||
{
|
||||
const struct admin_state_code *code = vdo_get_admin_state_code(state);
|
||||
|
||||
if (!assert_vdo_drain_operation(operation, waiter))
|
||||
return false;
|
||||
|
||||
if (code->quiescent) {
|
||||
vdo_launch_completion(waiter);
|
||||
return false;
|
||||
}
|
||||
|
||||
if (!code->normal) {
|
||||
vdo_log_error_strerror(VDO_INVALID_ADMIN_STATE, "can't start %s from %s",
|
||||
operation->name, code->name);
|
||||
vdo_continue_completion(waiter, VDO_INVALID_ADMIN_STATE);
|
||||
return false;
|
||||
}
|
||||
|
||||
return start_operation(state, operation, waiter, initiator);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_finish_draining() - Finish a drain operation if one was in progress.
|
||||
*
|
||||
* Return: true if the state was draining; will notify the waiter if so.
|
||||
*/
|
||||
bool vdo_finish_draining(struct admin_state *state)
|
||||
{
|
||||
return vdo_finish_draining_with_result(state, VDO_SUCCESS);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_finish_draining_with_result() - Finish a drain operation with a status code.
|
||||
*
|
||||
* Return: true if the state was draining; will notify the waiter if so.
|
||||
*/
|
||||
bool vdo_finish_draining_with_result(struct admin_state *state, int result)
|
||||
{
|
||||
return (vdo_is_state_draining(state) && vdo_finish_operation(state, result));
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_assert_load_operation() - Check that an operation is a load.
|
||||
* @waiter The completion to finish with an error if the operation is not a load.
|
||||
*
|
||||
* Return: true if the specified operation is a load.
|
||||
*/
|
||||
bool vdo_assert_load_operation(const struct admin_state_code *operation,
|
||||
struct vdo_completion *waiter)
|
||||
{
|
||||
return check_code(operation->loading, operation, "load operation", waiter);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_start_loading() - Initiate a load operation if the current state permits it.
|
||||
* @operation The type of load to initiate.
|
||||
* @waiter The completion to notify when the load is complete (may be NULL).
|
||||
* @initiator The vdo_admin_initiator_fn to call if the operation may begin; may be NULL.
|
||||
*
|
||||
* Return: true if the load was initiated, if not the waiter will be notified.
|
||||
*/
|
||||
bool vdo_start_loading(struct admin_state *state,
|
||||
const struct admin_state_code *operation,
|
||||
struct vdo_completion *waiter, vdo_admin_initiator_fn initiator)
|
||||
{
|
||||
return (vdo_assert_load_operation(operation, waiter) &&
|
||||
start_operation(state, operation, waiter, initiator));
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_finish_loading() - Finish a load operation if one was in progress.
|
||||
*
|
||||
* Return: true if the state was loading; will notify the waiter if so.
|
||||
*/
|
||||
bool vdo_finish_loading(struct admin_state *state)
|
||||
{
|
||||
return vdo_finish_loading_with_result(state, VDO_SUCCESS);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_finish_loading_with_result() - Finish a load operation with a status code.
|
||||
* @result The result of the load operation.
|
||||
*
|
||||
* Return: true if the state was loading; will notify the waiter if so.
|
||||
*/
|
||||
bool vdo_finish_loading_with_result(struct admin_state *state, int result)
|
||||
{
|
||||
return (vdo_is_state_loading(state) && vdo_finish_operation(state, result));
|
||||
}
|
||||
|
||||
/**
|
||||
* assert_vdo_resume_operation() - Check whether an admin_state_code is a resume operation.
|
||||
* @waiter The completion to notify if the operation is not a resume operation; may be NULL.
|
||||
*
|
||||
* Return: true if the code is a resume operation.
|
||||
*/
|
||||
static bool __must_check assert_vdo_resume_operation(const struct admin_state_code *operation,
|
||||
struct vdo_completion *waiter)
|
||||
{
|
||||
return check_code(operation == VDO_ADMIN_STATE_RESUMING, operation,
|
||||
"resume operation", waiter);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_start_resuming() - Initiate a resume operation if the current state permits it.
|
||||
* @operation The type of resume to start.
|
||||
* @waiter The completion to notify when the resume is complete (may be NULL).
|
||||
* @initiator The vdo_admin_initiator_fn to call if the operation may begin; may be NULL.
|
||||
*
|
||||
* Return: true if the resume was initiated, if not the waiter will be notified.
|
||||
*/
|
||||
bool vdo_start_resuming(struct admin_state *state,
|
||||
const struct admin_state_code *operation,
|
||||
struct vdo_completion *waiter, vdo_admin_initiator_fn initiator)
|
||||
{
|
||||
return (assert_vdo_resume_operation(operation, waiter) &&
|
||||
start_operation(state, operation, waiter, initiator));
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_finish_resuming() - Finish a resume operation if one was in progress.
|
||||
*
|
||||
* Return: true if the state was resuming; will notify the waiter if so.
|
||||
*/
|
||||
bool vdo_finish_resuming(struct admin_state *state)
|
||||
{
|
||||
return vdo_finish_resuming_with_result(state, VDO_SUCCESS);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_finish_resuming_with_result() - Finish a resume operation with a status code.
|
||||
* @result The result of the resume operation.
|
||||
*
|
||||
* Return: true if the state was resuming; will notify the waiter if so.
|
||||
*/
|
||||
bool vdo_finish_resuming_with_result(struct admin_state *state, int result)
|
||||
{
|
||||
return (vdo_is_state_resuming(state) && vdo_finish_operation(state, result));
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_resume_if_quiescent() - Change the state to normal operation if the current state is
|
||||
* quiescent.
|
||||
*
|
||||
* Return: VDO_SUCCESS if the state resumed, VDO_INVALID_ADMIN_STATE otherwise.
|
||||
*/
|
||||
int vdo_resume_if_quiescent(struct admin_state *state)
|
||||
{
|
||||
if (!vdo_is_state_quiescent(state))
|
||||
return VDO_INVALID_ADMIN_STATE;
|
||||
|
||||
vdo_set_admin_state_code(state, VDO_ADMIN_STATE_NORMAL_OPERATION);
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_start_operation() - Attempt to start an operation.
|
||||
*
|
||||
* Return: VDO_SUCCESS if the operation was started, VDO_INVALID_ADMIN_STATE if not
|
||||
*/
|
||||
int vdo_start_operation(struct admin_state *state,
|
||||
const struct admin_state_code *operation)
|
||||
{
|
||||
return vdo_start_operation_with_waiter(state, operation, NULL, NULL);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_start_operation_with_waiter() - Attempt to start an operation.
|
||||
* @waiter the completion to notify when the operation completes or fails to start; may be NULL.
|
||||
* @initiator The vdo_admin_initiator_fn to call if the operation may begin; may be NULL.
|
||||
*
|
||||
* Return: VDO_SUCCESS if the operation was started, VDO_INVALID_ADMIN_STATE if not
|
||||
*/
|
||||
int vdo_start_operation_with_waiter(struct admin_state *state,
|
||||
const struct admin_state_code *operation,
|
||||
struct vdo_completion *waiter,
|
||||
vdo_admin_initiator_fn initiator)
|
||||
{
|
||||
return (check_code(operation->operating, operation, "operation", waiter) ?
|
||||
begin_operation(state, operation, waiter, initiator) :
|
||||
VDO_INVALID_ADMIN_STATE);
|
||||
}
|
|
@ -0,0 +1,178 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef VDO_ADMIN_STATE_H
|
||||
#define VDO_ADMIN_STATE_H
|
||||
|
||||
#include "completion.h"
|
||||
#include "types.h"
|
||||
|
||||
struct admin_state_code {
|
||||
const char *name;
|
||||
/* Normal operation, data_vios may be active */
|
||||
bool normal;
|
||||
/* I/O is draining, new requests should not start */
|
||||
bool draining;
|
||||
/* This is a startup time operation */
|
||||
bool loading;
|
||||
/* The next state will be quiescent */
|
||||
bool quiescing;
|
||||
/* The VDO is quiescent, there should be no I/O */
|
||||
bool quiescent;
|
||||
/* Whether an operation is in progress and so no other operation may be started */
|
||||
bool operating;
|
||||
};
|
||||
|
||||
extern const struct admin_state_code *VDO_ADMIN_STATE_NORMAL_OPERATION;
|
||||
extern const struct admin_state_code *VDO_ADMIN_STATE_OPERATING;
|
||||
extern const struct admin_state_code *VDO_ADMIN_STATE_FORMATTING;
|
||||
extern const struct admin_state_code *VDO_ADMIN_STATE_PRE_LOADING;
|
||||
extern const struct admin_state_code *VDO_ADMIN_STATE_PRE_LOADED;
|
||||
extern const struct admin_state_code *VDO_ADMIN_STATE_LOADING;
|
||||
extern const struct admin_state_code *VDO_ADMIN_STATE_LOADING_FOR_RECOVERY;
|
||||
extern const struct admin_state_code *VDO_ADMIN_STATE_LOADING_FOR_REBUILD;
|
||||
extern const struct admin_state_code *VDO_ADMIN_STATE_WAITING_FOR_RECOVERY;
|
||||
extern const struct admin_state_code *VDO_ADMIN_STATE_NEW;
|
||||
extern const struct admin_state_code *VDO_ADMIN_STATE_INITIALIZED;
|
||||
extern const struct admin_state_code *VDO_ADMIN_STATE_RECOVERING;
|
||||
extern const struct admin_state_code *VDO_ADMIN_STATE_REBUILDING;
|
||||
extern const struct admin_state_code *VDO_ADMIN_STATE_SAVING;
|
||||
extern const struct admin_state_code *VDO_ADMIN_STATE_SAVED;
|
||||
extern const struct admin_state_code *VDO_ADMIN_STATE_SCRUBBING;
|
||||
extern const struct admin_state_code *VDO_ADMIN_STATE_SAVE_FOR_SCRUBBING;
|
||||
extern const struct admin_state_code *VDO_ADMIN_STATE_STOPPING;
|
||||
extern const struct admin_state_code *VDO_ADMIN_STATE_STOPPED;
|
||||
extern const struct admin_state_code *VDO_ADMIN_STATE_SUSPENDING;
|
||||
extern const struct admin_state_code *VDO_ADMIN_STATE_SUSPENDED;
|
||||
extern const struct admin_state_code *VDO_ADMIN_STATE_SUSPENDED_OPERATION;
|
||||
extern const struct admin_state_code *VDO_ADMIN_STATE_RESUMING;
|
||||
|
||||
struct admin_state {
|
||||
const struct admin_state_code *current_state;
|
||||
/* The next administrative state (when the current operation finishes) */
|
||||
const struct admin_state_code *next_state;
|
||||
/* A completion waiting on a state change */
|
||||
struct vdo_completion *waiter;
|
||||
/* Whether an operation is being initiated */
|
||||
bool starting;
|
||||
/* Whether an operation has completed in the initiator */
|
||||
bool complete;
|
||||
};
|
||||
|
||||
/**
|
||||
* typedef vdo_admin_initiator_fn - A method to be called once an admin operation may be initiated.
|
||||
*/
|
||||
typedef void (*vdo_admin_initiator_fn)(struct admin_state *state);
|
||||
|
||||
static inline const struct admin_state_code * __must_check
|
||||
vdo_get_admin_state_code(const struct admin_state *state)
|
||||
{
|
||||
return READ_ONCE(state->current_state);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_set_admin_state_code() - Set the current admin state code.
|
||||
*
|
||||
* This function should be used primarily for initialization and by adminState internals. Most uses
|
||||
* should go through the operation interfaces.
|
||||
*/
|
||||
static inline void vdo_set_admin_state_code(struct admin_state *state,
|
||||
const struct admin_state_code *code)
|
||||
{
|
||||
WRITE_ONCE(state->current_state, code);
|
||||
}
|
||||
|
||||
static inline bool __must_check vdo_is_state_normal(const struct admin_state *state)
|
||||
{
|
||||
return vdo_get_admin_state_code(state)->normal;
|
||||
}
|
||||
|
||||
static inline bool __must_check vdo_is_state_suspending(const struct admin_state *state)
|
||||
{
|
||||
return (vdo_get_admin_state_code(state) == VDO_ADMIN_STATE_SUSPENDING);
|
||||
}
|
||||
|
||||
static inline bool __must_check vdo_is_state_saving(const struct admin_state *state)
|
||||
{
|
||||
return (vdo_get_admin_state_code(state) == VDO_ADMIN_STATE_SAVING);
|
||||
}
|
||||
|
||||
static inline bool __must_check vdo_is_state_saved(const struct admin_state *state)
|
||||
{
|
||||
return (vdo_get_admin_state_code(state) == VDO_ADMIN_STATE_SAVED);
|
||||
}
|
||||
|
||||
static inline bool __must_check vdo_is_state_draining(const struct admin_state *state)
|
||||
{
|
||||
return vdo_get_admin_state_code(state)->draining;
|
||||
}
|
||||
|
||||
static inline bool __must_check vdo_is_state_loading(const struct admin_state *state)
|
||||
{
|
||||
return vdo_get_admin_state_code(state)->loading;
|
||||
}
|
||||
|
||||
static inline bool __must_check vdo_is_state_resuming(const struct admin_state *state)
|
||||
{
|
||||
return (vdo_get_admin_state_code(state) == VDO_ADMIN_STATE_RESUMING);
|
||||
}
|
||||
|
||||
static inline bool __must_check vdo_is_state_clean_load(const struct admin_state *state)
|
||||
{
|
||||
const struct admin_state_code *code = vdo_get_admin_state_code(state);
|
||||
|
||||
return ((code == VDO_ADMIN_STATE_FORMATTING) || (code == VDO_ADMIN_STATE_LOADING));
|
||||
}
|
||||
|
||||
static inline bool __must_check vdo_is_state_quiescing(const struct admin_state *state)
|
||||
{
|
||||
return vdo_get_admin_state_code(state)->quiescing;
|
||||
}
|
||||
|
||||
static inline bool __must_check vdo_is_state_quiescent(const struct admin_state *state)
|
||||
{
|
||||
return vdo_get_admin_state_code(state)->quiescent;
|
||||
}
|
||||
|
||||
bool __must_check vdo_assert_load_operation(const struct admin_state_code *operation,
|
||||
struct vdo_completion *waiter);
|
||||
|
||||
bool vdo_start_loading(struct admin_state *state,
|
||||
const struct admin_state_code *operation,
|
||||
struct vdo_completion *waiter, vdo_admin_initiator_fn initiator);
|
||||
|
||||
bool vdo_finish_loading(struct admin_state *state);
|
||||
|
||||
bool vdo_finish_loading_with_result(struct admin_state *state, int result);
|
||||
|
||||
bool vdo_start_resuming(struct admin_state *state,
|
||||
const struct admin_state_code *operation,
|
||||
struct vdo_completion *waiter, vdo_admin_initiator_fn initiator);
|
||||
|
||||
bool vdo_finish_resuming(struct admin_state *state);
|
||||
|
||||
bool vdo_finish_resuming_with_result(struct admin_state *state, int result);
|
||||
|
||||
int vdo_resume_if_quiescent(struct admin_state *state);
|
||||
|
||||
bool vdo_start_draining(struct admin_state *state,
|
||||
const struct admin_state_code *operation,
|
||||
struct vdo_completion *waiter, vdo_admin_initiator_fn initiator);
|
||||
|
||||
bool vdo_finish_draining(struct admin_state *state);
|
||||
|
||||
bool vdo_finish_draining_with_result(struct admin_state *state, int result);
|
||||
|
||||
int vdo_start_operation(struct admin_state *state,
|
||||
const struct admin_state_code *operation);
|
||||
|
||||
int vdo_start_operation_with_waiter(struct admin_state *state,
|
||||
const struct admin_state_code *operation,
|
||||
struct vdo_completion *waiter,
|
||||
vdo_admin_initiator_fn initiator);
|
||||
|
||||
bool vdo_finish_operation(struct admin_state *state, int result);
|
||||
|
||||
#endif /* VDO_ADMIN_STATE_H */
|
File diff suppressed because it is too large
Load Diff
|
@ -0,0 +1,394 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef VDO_BLOCK_MAP_H
|
||||
#define VDO_BLOCK_MAP_H
|
||||
|
||||
#include <linux/list.h>
|
||||
|
||||
#include "numeric.h"
|
||||
|
||||
#include "admin-state.h"
|
||||
#include "completion.h"
|
||||
#include "encodings.h"
|
||||
#include "int-map.h"
|
||||
#include "statistics.h"
|
||||
#include "types.h"
|
||||
#include "vio.h"
|
||||
#include "wait-queue.h"
|
||||
|
||||
/*
|
||||
* The block map is responsible for tracking all the logical to physical mappings of a VDO. It
|
||||
* consists of a collection of 60 radix trees gradually allocated as logical addresses are used.
|
||||
* Each tree is assigned to a logical zone such that it is easy to compute which zone must handle
|
||||
* each logical address. Each logical zone also has a dedicated portion of the leaf page cache.
|
||||
*
|
||||
* Each logical zone has a single dedicated queue and thread for performing all updates to the
|
||||
* radix trees assigned to that zone. The concurrency guarantees of this single-threaded model
|
||||
* allow the code to omit more fine-grained locking for the block map structures.
|
||||
*
|
||||
* Load operations must be performed on the admin thread. Normal operations, such as reading and
|
||||
* updating mappings, must be performed on the appropriate logical zone thread. Save operations
|
||||
* must be launched from the same admin thread as the original load operation.
|
||||
*/
|
||||
|
||||
enum {
|
||||
BLOCK_MAP_VIO_POOL_SIZE = 64,
|
||||
};
|
||||
|
||||
/*
|
||||
* Generation counter for page references.
|
||||
*/
|
||||
typedef u32 vdo_page_generation;
|
||||
|
||||
extern const struct block_map_entry UNMAPPED_BLOCK_MAP_ENTRY;
|
||||
|
||||
/* The VDO Page Cache abstraction. */
|
||||
struct vdo_page_cache {
|
||||
/* the VDO which owns this cache */
|
||||
struct vdo *vdo;
|
||||
/* number of pages in cache */
|
||||
page_count_t page_count;
|
||||
/* number of pages to write in the current batch */
|
||||
page_count_t pages_in_batch;
|
||||
/* Whether the VDO is doing a read-only rebuild */
|
||||
bool rebuilding;
|
||||
|
||||
/* array of page information entries */
|
||||
struct page_info *infos;
|
||||
/* raw memory for pages */
|
||||
char *pages;
|
||||
/* cache last found page info */
|
||||
struct page_info *last_found;
|
||||
/* map of page number to info */
|
||||
struct int_map *page_map;
|
||||
/* main LRU list (all infos) */
|
||||
struct list_head lru_list;
|
||||
/* free page list (oldest first) */
|
||||
struct list_head free_list;
|
||||
/* outgoing page list */
|
||||
struct list_head outgoing_list;
|
||||
/* number of read I/O operations pending */
|
||||
page_count_t outstanding_reads;
|
||||
/* number of write I/O operations pending */
|
||||
page_count_t outstanding_writes;
|
||||
/* number of pages covered by the current flush */
|
||||
page_count_t pages_in_flush;
|
||||
/* number of pages waiting to be included in the next flush */
|
||||
page_count_t pages_to_flush;
|
||||
/* number of discards in progress */
|
||||
unsigned int discard_count;
|
||||
/* how many VPCs waiting for free page */
|
||||
unsigned int waiter_count;
|
||||
/* queue of waiters who want a free page */
|
||||
struct vdo_wait_queue free_waiters;
|
||||
/*
|
||||
* Statistics are only updated on the logical zone thread, but are accessed from other
|
||||
* threads.
|
||||
*/
|
||||
struct block_map_statistics stats;
|
||||
/* counter for pressure reports */
|
||||
u32 pressure_report;
|
||||
/* the block map zone to which this cache belongs */
|
||||
struct block_map_zone *zone;
|
||||
};
|
||||
|
||||
/*
|
||||
* The state of a page buffer. If the page buffer is free no particular page is bound to it,
|
||||
* otherwise the page buffer is bound to particular page whose absolute pbn is in the pbn field. If
|
||||
* the page is resident or dirty the page data is stable and may be accessed. Otherwise the page is
|
||||
* in flight (incoming or outgoing) and its data should not be accessed.
|
||||
*
|
||||
* @note Update the static data in get_page_state_name() if you change this enumeration.
|
||||
*/
|
||||
enum vdo_page_buffer_state {
|
||||
/* this page buffer is not being used */
|
||||
PS_FREE,
|
||||
/* this page is being read from store */
|
||||
PS_INCOMING,
|
||||
/* attempt to load this page failed */
|
||||
PS_FAILED,
|
||||
/* this page is valid and un-modified */
|
||||
PS_RESIDENT,
|
||||
/* this page is valid and modified */
|
||||
PS_DIRTY,
|
||||
/* this page is being written and should not be used */
|
||||
PS_OUTGOING,
|
||||
/* not a state */
|
||||
PAGE_STATE_COUNT,
|
||||
} __packed;
|
||||
|
||||
/*
|
||||
* The write status of page
|
||||
*/
|
||||
enum vdo_page_write_status {
|
||||
WRITE_STATUS_NORMAL,
|
||||
WRITE_STATUS_DISCARD,
|
||||
WRITE_STATUS_DEFERRED,
|
||||
} __packed;
|
||||
|
||||
/* Per-page-slot information. */
|
||||
struct page_info {
|
||||
/* Preallocated page struct vio */
|
||||
struct vio *vio;
|
||||
/* back-link for references */
|
||||
struct vdo_page_cache *cache;
|
||||
/* the pbn of the page */
|
||||
physical_block_number_t pbn;
|
||||
/* page is busy (temporarily locked) */
|
||||
u16 busy;
|
||||
/* the write status the page */
|
||||
enum vdo_page_write_status write_status;
|
||||
/* page state */
|
||||
enum vdo_page_buffer_state state;
|
||||
/* queue of completions awaiting this item */
|
||||
struct vdo_wait_queue waiting;
|
||||
/* state linked list entry */
|
||||
struct list_head state_entry;
|
||||
/* LRU entry */
|
||||
struct list_head lru_entry;
|
||||
/*
|
||||
* The earliest recovery journal block containing uncommitted updates to the block map page
|
||||
* associated with this page_info. A reference (lock) is held on that block to prevent it
|
||||
* from being reaped. When this value changes, the reference on the old value must be
|
||||
* released and a reference on the new value must be acquired.
|
||||
*/
|
||||
sequence_number_t recovery_lock;
|
||||
};
|
||||
|
||||
/*
|
||||
* A completion awaiting a specific page. Also a live reference into the page once completed, until
|
||||
* freed.
|
||||
*/
|
||||
struct vdo_page_completion {
|
||||
/* The generic completion */
|
||||
struct vdo_completion completion;
|
||||
/* The cache involved */
|
||||
struct vdo_page_cache *cache;
|
||||
/* The waiter for the pending list */
|
||||
struct vdo_waiter waiter;
|
||||
/* The absolute physical block number of the page on disk */
|
||||
physical_block_number_t pbn;
|
||||
/* Whether the page may be modified */
|
||||
bool writable;
|
||||
/* Whether the page is available */
|
||||
bool ready;
|
||||
/* The info structure for the page, only valid when ready */
|
||||
struct page_info *info;
|
||||
};
|
||||
|
||||
struct forest;
|
||||
|
||||
struct tree_page {
|
||||
struct vdo_waiter waiter;
|
||||
|
||||
/* Dirty list entry */
|
||||
struct list_head entry;
|
||||
|
||||
/* If dirty, the tree zone flush generation in which it was last dirtied. */
|
||||
u8 generation;
|
||||
|
||||
/* Whether this page is an interior tree page being written out. */
|
||||
bool writing;
|
||||
|
||||
/* If writing, the tree zone flush generation of the copy being written. */
|
||||
u8 writing_generation;
|
||||
|
||||
/*
|
||||
* Sequence number of the earliest recovery journal block containing uncommitted updates to
|
||||
* this page
|
||||
*/
|
||||
sequence_number_t recovery_lock;
|
||||
|
||||
/* The value of recovery_lock when the this page last started writing */
|
||||
sequence_number_t writing_recovery_lock;
|
||||
|
||||
char page_buffer[VDO_BLOCK_SIZE];
|
||||
};
|
||||
|
||||
enum block_map_page_type {
|
||||
VDO_TREE_PAGE,
|
||||
VDO_CACHE_PAGE,
|
||||
};
|
||||
|
||||
typedef struct list_head dirty_era_t[2];
|
||||
|
||||
struct dirty_lists {
|
||||
/* The number of periods after which an element will be expired */
|
||||
block_count_t maximum_age;
|
||||
/* The oldest period which has unexpired elements */
|
||||
sequence_number_t oldest_period;
|
||||
/* One more than the current period */
|
||||
sequence_number_t next_period;
|
||||
/* The offset in the array of lists of the oldest period */
|
||||
block_count_t offset;
|
||||
/* Expired pages */
|
||||
dirty_era_t expired;
|
||||
/* The lists of dirty pages */
|
||||
dirty_era_t eras[];
|
||||
};
|
||||
|
||||
struct block_map_zone {
|
||||
zone_count_t zone_number;
|
||||
thread_id_t thread_id;
|
||||
struct admin_state state;
|
||||
struct block_map *block_map;
|
||||
/* Dirty pages, by era*/
|
||||
struct dirty_lists *dirty_lists;
|
||||
struct vdo_page_cache page_cache;
|
||||
data_vio_count_t active_lookups;
|
||||
struct int_map *loading_pages;
|
||||
struct vio_pool *vio_pool;
|
||||
/* The tree page which has issued or will be issuing a flush */
|
||||
struct tree_page *flusher;
|
||||
struct vdo_wait_queue flush_waiters;
|
||||
/* The generation after the most recent flush */
|
||||
u8 generation;
|
||||
u8 oldest_generation;
|
||||
/* The counts of dirty pages in each generation */
|
||||
u32 dirty_page_counts[256];
|
||||
};
|
||||
|
||||
struct block_map {
|
||||
struct vdo *vdo;
|
||||
struct action_manager *action_manager;
|
||||
/* The absolute PBN of the first root of the tree part of the block map */
|
||||
physical_block_number_t root_origin;
|
||||
block_count_t root_count;
|
||||
|
||||
/* The era point we are currently distributing to the zones */
|
||||
sequence_number_t current_era_point;
|
||||
/* The next era point */
|
||||
sequence_number_t pending_era_point;
|
||||
|
||||
/* The number of entries in block map */
|
||||
block_count_t entry_count;
|
||||
nonce_t nonce;
|
||||
struct recovery_journal *journal;
|
||||
|
||||
/* The trees for finding block map pages */
|
||||
struct forest *forest;
|
||||
/* The expanded trees awaiting growth */
|
||||
struct forest *next_forest;
|
||||
/* The number of entries after growth */
|
||||
block_count_t next_entry_count;
|
||||
|
||||
zone_count_t zone_count;
|
||||
struct block_map_zone zones[];
|
||||
};
|
||||
|
||||
/**
|
||||
* typedef vdo_entry_callback_fn - A function to be called for each allocated PBN when traversing
|
||||
* the forest.
|
||||
* @pbn: A PBN of a tree node.
|
||||
* @completion: The parent completion of the traversal.
|
||||
*
|
||||
* Return: VDO_SUCCESS or an error.
|
||||
*/
|
||||
typedef int (*vdo_entry_callback_fn)(physical_block_number_t pbn,
|
||||
struct vdo_completion *completion);
|
||||
|
||||
static inline struct vdo_page_completion *as_vdo_page_completion(struct vdo_completion *completion)
|
||||
{
|
||||
vdo_assert_completion_type(completion, VDO_PAGE_COMPLETION);
|
||||
return container_of(completion, struct vdo_page_completion, completion);
|
||||
}
|
||||
|
||||
void vdo_release_page_completion(struct vdo_completion *completion);
|
||||
|
||||
void vdo_get_page(struct vdo_page_completion *page_completion,
|
||||
struct block_map_zone *zone, physical_block_number_t pbn,
|
||||
bool writable, void *parent, vdo_action_fn callback,
|
||||
vdo_action_fn error_handler, bool requeue);
|
||||
|
||||
void vdo_request_page_write(struct vdo_completion *completion);
|
||||
|
||||
int __must_check vdo_get_cached_page(struct vdo_completion *completion,
|
||||
struct block_map_page **page_ptr);
|
||||
|
||||
int __must_check vdo_invalidate_page_cache(struct vdo_page_cache *cache);
|
||||
|
||||
static inline struct block_map_page * __must_check
|
||||
vdo_as_block_map_page(struct tree_page *tree_page)
|
||||
{
|
||||
return (struct block_map_page *) tree_page->page_buffer;
|
||||
}
|
||||
|
||||
bool vdo_copy_valid_page(char *buffer, nonce_t nonce,
|
||||
physical_block_number_t pbn,
|
||||
struct block_map_page *page);
|
||||
|
||||
void vdo_find_block_map_slot(struct data_vio *data_vio);
|
||||
|
||||
physical_block_number_t vdo_find_block_map_page_pbn(struct block_map *map,
|
||||
page_number_t page_number);
|
||||
|
||||
void vdo_write_tree_page(struct tree_page *page, struct block_map_zone *zone);
|
||||
|
||||
void vdo_traverse_forest(struct block_map *map, vdo_entry_callback_fn callback,
|
||||
struct vdo_completion *completion);
|
||||
|
||||
int __must_check vdo_decode_block_map(struct block_map_state_2_0 state,
|
||||
block_count_t logical_blocks, struct vdo *vdo,
|
||||
struct recovery_journal *journal, nonce_t nonce,
|
||||
page_count_t cache_size, block_count_t maximum_age,
|
||||
struct block_map **map_ptr);
|
||||
|
||||
void vdo_drain_block_map(struct block_map *map, const struct admin_state_code *operation,
|
||||
struct vdo_completion *parent);
|
||||
|
||||
void vdo_resume_block_map(struct block_map *map, struct vdo_completion *parent);
|
||||
|
||||
int __must_check vdo_prepare_to_grow_block_map(struct block_map *map,
|
||||
block_count_t new_logical_blocks);
|
||||
|
||||
void vdo_grow_block_map(struct block_map *map, struct vdo_completion *parent);
|
||||
|
||||
void vdo_abandon_block_map_growth(struct block_map *map);
|
||||
|
||||
void vdo_free_block_map(struct block_map *map);
|
||||
|
||||
struct block_map_state_2_0 __must_check vdo_record_block_map(const struct block_map *map);
|
||||
|
||||
void vdo_initialize_block_map_from_journal(struct block_map *map,
|
||||
struct recovery_journal *journal);
|
||||
|
||||
zone_count_t vdo_compute_logical_zone(struct data_vio *data_vio);
|
||||
|
||||
void vdo_advance_block_map_era(struct block_map *map,
|
||||
sequence_number_t recovery_block_number);
|
||||
|
||||
void vdo_update_block_map_page(struct block_map_page *page, struct data_vio *data_vio,
|
||||
physical_block_number_t pbn,
|
||||
enum block_mapping_state mapping_state,
|
||||
sequence_number_t *recovery_lock);
|
||||
|
||||
void vdo_get_mapped_block(struct data_vio *data_vio);
|
||||
|
||||
void vdo_put_mapped_block(struct data_vio *data_vio);
|
||||
|
||||
struct block_map_statistics __must_check vdo_get_block_map_statistics(struct block_map *map);
|
||||
|
||||
/**
|
||||
* vdo_convert_maximum_age() - Convert the maximum age to reflect the new recovery journal format
|
||||
* @age: The configured maximum age
|
||||
*
|
||||
* Return: The converted age
|
||||
*
|
||||
* In the old recovery journal format, each journal block held 311 entries, and every write bio
|
||||
* made two entries. The old maximum age was half the usable journal length. In the new format,
|
||||
* each block holds only 217 entries, but each bio only makes one entry. We convert the configured
|
||||
* age so that the number of writes in a block map era is the same in the old and new formats. This
|
||||
* keeps the bound on the amount of work required to recover the block map from the recovery
|
||||
* journal the same across the format change. It also keeps the amortization of block map page
|
||||
* writes to write bios the same.
|
||||
*/
|
||||
static inline block_count_t vdo_convert_maximum_age(block_count_t age)
|
||||
{
|
||||
return DIV_ROUND_UP(age * RECOVERY_JOURNAL_1_ENTRIES_PER_BLOCK,
|
||||
2 * RECOVERY_JOURNAL_ENTRIES_PER_BLOCK);
|
||||
}
|
||||
|
||||
#endif /* VDO_BLOCK_MAP_H */
|
|
@ -0,0 +1,140 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "completion.h"
|
||||
|
||||
#include <linux/kernel.h>
|
||||
|
||||
#include "logger.h"
|
||||
#include "permassert.h"
|
||||
|
||||
#include "status-codes.h"
|
||||
#include "types.h"
|
||||
#include "vio.h"
|
||||
#include "vdo.h"
|
||||
|
||||
/**
|
||||
* DOC: vdo completions.
|
||||
*
|
||||
* Most of vdo's data structures are lock free, each either belonging to a single "zone," or
|
||||
* divided into a number of zones whose accesses to the structure do not overlap. During normal
|
||||
* operation, at most one thread will be operating in any given zone. Each zone has a
|
||||
* vdo_work_queue which holds vdo_completions that are to be run in that zone. A completion may
|
||||
* only be enqueued on one queue or operating in a single zone at a time.
|
||||
*
|
||||
* At each step of a multi-threaded operation, the completion performing the operation is given a
|
||||
* callback, error handler, and thread id for the next step. A completion is "run" when it is
|
||||
* operating on the correct thread (as specified by its callback_thread_id). If the value of its
|
||||
* "result" field is an error (i.e. not VDO_SUCCESS), the function in its "error_handler" will be
|
||||
* invoked. If the error_handler is NULL, or there is no error, the function set as its "callback"
|
||||
* will be invoked. Generally, a completion will not be run directly, but rather will be
|
||||
* "launched." In this case, it will check whether it is operating on the correct thread. If it is,
|
||||
* it will run immediately. Otherwise, it will be enqueue on the vdo_work_queue associated with the
|
||||
* completion's "callback_thread_id". When it is dequeued, it will be on the correct thread, and
|
||||
* will get run. In some cases, the completion should get queued instead of running immediately,
|
||||
* even if it is being launched from the correct thread. This is usually in cases where there is a
|
||||
* long chain of callbacks, all on the same thread, which could overflow the stack. In such cases,
|
||||
* the completion's "requeue" field should be set to true. Doing so will skip the current thread
|
||||
* check and simply enqueue the completion.
|
||||
*
|
||||
* A completion may be "finished," in which case its "complete" field will be set to true before it
|
||||
* is next run. It is a bug to attempt to set the result or re-finish a finished completion.
|
||||
* Because a completion's fields are not safe to examine from any thread other than the one on
|
||||
* which the completion is currently operating, this field is used only to aid in detecting
|
||||
* programming errors. It can not be used for cross-thread checking on the status of an operation.
|
||||
* A completion must be "reset" before it can be reused after it has been finished. Resetting will
|
||||
* also clear any error from the result field.
|
||||
**/
|
||||
|
||||
void vdo_initialize_completion(struct vdo_completion *completion,
|
||||
struct vdo *vdo,
|
||||
enum vdo_completion_type type)
|
||||
{
|
||||
memset(completion, 0, sizeof(*completion));
|
||||
completion->vdo = vdo;
|
||||
completion->type = type;
|
||||
vdo_reset_completion(completion);
|
||||
}
|
||||
|
||||
static inline void assert_incomplete(struct vdo_completion *completion)
|
||||
{
|
||||
VDO_ASSERT_LOG_ONLY(!completion->complete, "completion is not complete");
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_set_completion_result() - Set the result of a completion.
|
||||
*
|
||||
* Older errors will not be masked.
|
||||
*/
|
||||
void vdo_set_completion_result(struct vdo_completion *completion, int result)
|
||||
{
|
||||
assert_incomplete(completion);
|
||||
if (completion->result == VDO_SUCCESS)
|
||||
completion->result = result;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_launch_completion_with_priority() - Run or enqueue a completion.
|
||||
* @priority: The priority at which to enqueue the completion.
|
||||
*
|
||||
* If called on the correct thread (i.e. the one specified in the completion's callback_thread_id
|
||||
* field) and not marked for requeue, the completion will be run immediately. Otherwise, the
|
||||
* completion will be enqueued on the specified thread.
|
||||
*/
|
||||
void vdo_launch_completion_with_priority(struct vdo_completion *completion,
|
||||
enum vdo_completion_priority priority)
|
||||
{
|
||||
thread_id_t callback_thread = completion->callback_thread_id;
|
||||
|
||||
if (completion->requeue || (callback_thread != vdo_get_callback_thread_id())) {
|
||||
vdo_enqueue_completion(completion, priority);
|
||||
return;
|
||||
}
|
||||
|
||||
vdo_run_completion(completion);
|
||||
}
|
||||
|
||||
/** vdo_finish_completion() - Mark a completion as complete and then launch it. */
|
||||
void vdo_finish_completion(struct vdo_completion *completion)
|
||||
{
|
||||
assert_incomplete(completion);
|
||||
completion->complete = true;
|
||||
if (completion->callback != NULL)
|
||||
vdo_launch_completion(completion);
|
||||
}
|
||||
|
||||
void vdo_enqueue_completion(struct vdo_completion *completion,
|
||||
enum vdo_completion_priority priority)
|
||||
{
|
||||
struct vdo *vdo = completion->vdo;
|
||||
thread_id_t thread_id = completion->callback_thread_id;
|
||||
|
||||
if (VDO_ASSERT(thread_id < vdo->thread_config.thread_count,
|
||||
"thread_id %u (completion type %d) is less than thread count %u",
|
||||
thread_id, completion->type,
|
||||
vdo->thread_config.thread_count) != VDO_SUCCESS)
|
||||
BUG();
|
||||
|
||||
completion->requeue = false;
|
||||
completion->priority = priority;
|
||||
completion->my_queue = NULL;
|
||||
vdo_enqueue_work_queue(vdo->threads[thread_id].queue, completion);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_requeue_completion_if_needed() - Requeue a completion if not called on the specified thread.
|
||||
*
|
||||
* Return: True if the completion was requeued; callers may not access the completion in this case.
|
||||
*/
|
||||
bool vdo_requeue_completion_if_needed(struct vdo_completion *completion,
|
||||
thread_id_t callback_thread_id)
|
||||
{
|
||||
if (vdo_get_callback_thread_id() == callback_thread_id)
|
||||
return false;
|
||||
|
||||
completion->callback_thread_id = callback_thread_id;
|
||||
vdo_enqueue_completion(completion, VDO_WORK_Q_DEFAULT_PRIORITY);
|
||||
return true;
|
||||
}
|
|
@ -0,0 +1,152 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef VDO_COMPLETION_H
|
||||
#define VDO_COMPLETION_H
|
||||
|
||||
#include "permassert.h"
|
||||
|
||||
#include "status-codes.h"
|
||||
#include "types.h"
|
||||
|
||||
/**
|
||||
* vdo_run_completion() - Run a completion's callback or error handler on the current thread.
|
||||
*
|
||||
* Context: This function must be called from the correct callback thread.
|
||||
*/
|
||||
static inline void vdo_run_completion(struct vdo_completion *completion)
|
||||
{
|
||||
if ((completion->result != VDO_SUCCESS) && (completion->error_handler != NULL)) {
|
||||
completion->error_handler(completion);
|
||||
return;
|
||||
}
|
||||
|
||||
completion->callback(completion);
|
||||
}
|
||||
|
||||
void vdo_set_completion_result(struct vdo_completion *completion, int result);
|
||||
|
||||
void vdo_initialize_completion(struct vdo_completion *completion, struct vdo *vdo,
|
||||
enum vdo_completion_type type);
|
||||
|
||||
/**
|
||||
* vdo_reset_completion() - Reset a completion to a clean state, while keeping the type, vdo and
|
||||
* parent information.
|
||||
*/
|
||||
static inline void vdo_reset_completion(struct vdo_completion *completion)
|
||||
{
|
||||
completion->result = VDO_SUCCESS;
|
||||
completion->complete = false;
|
||||
}
|
||||
|
||||
void vdo_launch_completion_with_priority(struct vdo_completion *completion,
|
||||
enum vdo_completion_priority priority);
|
||||
|
||||
/**
|
||||
* vdo_launch_completion() - Launch a completion with default priority.
|
||||
*/
|
||||
static inline void vdo_launch_completion(struct vdo_completion *completion)
|
||||
{
|
||||
vdo_launch_completion_with_priority(completion, VDO_WORK_Q_DEFAULT_PRIORITY);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_continue_completion() - Continue processing a completion.
|
||||
* @result: The current result (will not mask older errors).
|
||||
*
|
||||
* Continue processing a completion by setting the current result and calling
|
||||
* vdo_launch_completion().
|
||||
*/
|
||||
static inline void vdo_continue_completion(struct vdo_completion *completion, int result)
|
||||
{
|
||||
vdo_set_completion_result(completion, result);
|
||||
vdo_launch_completion(completion);
|
||||
}
|
||||
|
||||
void vdo_finish_completion(struct vdo_completion *completion);
|
||||
|
||||
/**
|
||||
* vdo_fail_completion() - Set the result of a completion if it does not already have an error,
|
||||
* then finish it.
|
||||
*/
|
||||
static inline void vdo_fail_completion(struct vdo_completion *completion, int result)
|
||||
{
|
||||
vdo_set_completion_result(completion, result);
|
||||
vdo_finish_completion(completion);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_assert_completion_type() - Assert that a completion is of the correct type.
|
||||
*
|
||||
* Return: VDO_SUCCESS or an error
|
||||
*/
|
||||
static inline int vdo_assert_completion_type(struct vdo_completion *completion,
|
||||
enum vdo_completion_type expected)
|
||||
{
|
||||
return VDO_ASSERT(expected == completion->type,
|
||||
"completion type should be %u, not %u", expected,
|
||||
completion->type);
|
||||
}
|
||||
|
||||
static inline void vdo_set_completion_callback(struct vdo_completion *completion,
|
||||
vdo_action_fn callback,
|
||||
thread_id_t callback_thread_id)
|
||||
{
|
||||
completion->callback = callback;
|
||||
completion->callback_thread_id = callback_thread_id;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_launch_completion_callback() - Set the callback for a completion and launch it immediately.
|
||||
*/
|
||||
static inline void vdo_launch_completion_callback(struct vdo_completion *completion,
|
||||
vdo_action_fn callback,
|
||||
thread_id_t callback_thread_id)
|
||||
{
|
||||
vdo_set_completion_callback(completion, callback, callback_thread_id);
|
||||
vdo_launch_completion(completion);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_prepare_completion() - Prepare a completion for launch.
|
||||
*
|
||||
* Resets the completion, and then sets its callback, error handler, callback thread, and parent.
|
||||
*/
|
||||
static inline void vdo_prepare_completion(struct vdo_completion *completion,
|
||||
vdo_action_fn callback,
|
||||
vdo_action_fn error_handler,
|
||||
thread_id_t callback_thread_id, void *parent)
|
||||
{
|
||||
vdo_reset_completion(completion);
|
||||
vdo_set_completion_callback(completion, callback, callback_thread_id);
|
||||
completion->error_handler = error_handler;
|
||||
completion->parent = parent;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_prepare_completion_for_requeue() - Prepare a completion for launch ensuring that it will
|
||||
* always be requeued.
|
||||
*
|
||||
* Resets the completion, and then sets its callback, error handler, callback thread, and parent.
|
||||
*/
|
||||
static inline void vdo_prepare_completion_for_requeue(struct vdo_completion *completion,
|
||||
vdo_action_fn callback,
|
||||
vdo_action_fn error_handler,
|
||||
thread_id_t callback_thread_id,
|
||||
void *parent)
|
||||
{
|
||||
vdo_prepare_completion(completion, callback, error_handler,
|
||||
callback_thread_id, parent);
|
||||
completion->requeue = true;
|
||||
}
|
||||
|
||||
void vdo_enqueue_completion(struct vdo_completion *completion,
|
||||
enum vdo_completion_priority priority);
|
||||
|
||||
|
||||
bool vdo_requeue_completion_if_needed(struct vdo_completion *completion,
|
||||
thread_id_t callback_thread_id);
|
||||
|
||||
#endif /* VDO_COMPLETION_H */
|
|
@ -0,0 +1,96 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef VDO_CONSTANTS_H
|
||||
#define VDO_CONSTANTS_H
|
||||
|
||||
#include <linux/blkdev.h>
|
||||
|
||||
#include "types.h"
|
||||
|
||||
enum {
|
||||
/*
|
||||
* The maximum number of contiguous PBNs which will go to a single bio submission queue,
|
||||
* assuming there is more than one queue.
|
||||
*/
|
||||
VDO_BIO_ROTATION_INTERVAL_LIMIT = 1024,
|
||||
|
||||
/* The number of entries on a block map page */
|
||||
VDO_BLOCK_MAP_ENTRIES_PER_PAGE = 812,
|
||||
|
||||
/* The origin of the flat portion of the block map */
|
||||
VDO_BLOCK_MAP_FLAT_PAGE_ORIGIN = 1,
|
||||
|
||||
/*
|
||||
* The height of a block map tree. Assuming a root count of 60 and 812 entries per page,
|
||||
* this is big enough to represent almost 95 PB of logical space.
|
||||
*/
|
||||
VDO_BLOCK_MAP_TREE_HEIGHT = 5,
|
||||
|
||||
/* The default number of bio submission queues. */
|
||||
DEFAULT_VDO_BIO_SUBMIT_QUEUE_COUNT = 4,
|
||||
|
||||
/* The number of contiguous PBNs to be submitted to a single bio queue. */
|
||||
DEFAULT_VDO_BIO_SUBMIT_QUEUE_ROTATE_INTERVAL = 64,
|
||||
|
||||
/* The number of trees in the arboreal block map */
|
||||
DEFAULT_VDO_BLOCK_MAP_TREE_ROOT_COUNT = 60,
|
||||
|
||||
/* The default size of the recovery journal, in blocks */
|
||||
DEFAULT_VDO_RECOVERY_JOURNAL_SIZE = 32 * 1024,
|
||||
|
||||
/* The default size of each slab journal, in blocks */
|
||||
DEFAULT_VDO_SLAB_JOURNAL_SIZE = 224,
|
||||
|
||||
/* Unit test minimum */
|
||||
MINIMUM_VDO_SLAB_JOURNAL_BLOCKS = 2,
|
||||
|
||||
/*
|
||||
* The initial size of lbn_operations and pbn_operations, which is based upon the expected
|
||||
* maximum number of outstanding VIOs. This value was chosen to make it highly unlikely
|
||||
* that the maps would need to be resized.
|
||||
*/
|
||||
VDO_LOCK_MAP_CAPACITY = 10000,
|
||||
|
||||
/* The maximum number of logical zones */
|
||||
MAX_VDO_LOGICAL_ZONES = 60,
|
||||
|
||||
/* The maximum number of physical zones */
|
||||
MAX_VDO_PHYSICAL_ZONES = 16,
|
||||
|
||||
/* The base-2 logarithm of the maximum blocks in one slab */
|
||||
MAX_VDO_SLAB_BITS = 23,
|
||||
|
||||
/* The maximum number of slabs the slab depot supports */
|
||||
MAX_VDO_SLABS = 8192,
|
||||
|
||||
/*
|
||||
* The maximum number of block map pages to load simultaneously during recovery or rebuild.
|
||||
*/
|
||||
MAXIMUM_SIMULTANEOUS_VDO_BLOCK_MAP_RESTORATION_READS = 1024,
|
||||
|
||||
/* The maximum number of entries in the slab summary */
|
||||
MAXIMUM_VDO_SLAB_SUMMARY_ENTRIES = MAX_VDO_SLABS * MAX_VDO_PHYSICAL_ZONES,
|
||||
|
||||
/* The maximum number of total threads in a VDO thread configuration. */
|
||||
MAXIMUM_VDO_THREADS = 100,
|
||||
|
||||
/* The maximum number of VIOs in the system at once */
|
||||
MAXIMUM_VDO_USER_VIOS = 2048,
|
||||
|
||||
/* The only physical block size supported by VDO */
|
||||
VDO_BLOCK_SIZE = 4096,
|
||||
|
||||
/* The number of sectors per block */
|
||||
VDO_SECTORS_PER_BLOCK = (VDO_BLOCK_SIZE >> SECTOR_SHIFT),
|
||||
|
||||
/* The size of a sector that will not be torn */
|
||||
VDO_SECTOR_SIZE = 512,
|
||||
|
||||
/* The physical block number reserved for storing the zero block */
|
||||
VDO_ZERO_BLOCK = 0,
|
||||
};
|
||||
|
||||
#endif /* VDO_CONSTANTS_H */
|
|
@ -0,0 +1,59 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef UDS_CPU_H
|
||||
#define UDS_CPU_H
|
||||
|
||||
#include <linux/cache.h>
|
||||
|
||||
/**
|
||||
* uds_prefetch_address() - Minimize cache-miss latency by attempting to move data into a CPU cache
|
||||
* before it is accessed.
|
||||
*
|
||||
* @address: the address to fetch (may be invalid)
|
||||
* @for_write: must be constant at compile time--false if for reading, true if for writing
|
||||
*/
|
||||
static inline void uds_prefetch_address(const void *address, bool for_write)
|
||||
{
|
||||
/*
|
||||
* for_write won't be a constant if we are compiled with optimization turned off, in which
|
||||
* case prefetching really doesn't matter. clang can't figure out that if for_write is a
|
||||
* constant, it can be passed as the second, mandatorily constant argument to prefetch(),
|
||||
* at least currently on llvm 12.
|
||||
*/
|
||||
if (__builtin_constant_p(for_write)) {
|
||||
if (for_write)
|
||||
__builtin_prefetch(address, true);
|
||||
else
|
||||
__builtin_prefetch(address, false);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* uds_prefetch_range() - Minimize cache-miss latency by attempting to move a range of addresses
|
||||
* into a CPU cache before they are accessed.
|
||||
*
|
||||
* @start: the starting address to fetch (may be invalid)
|
||||
* @size: the number of bytes in the address range
|
||||
* @for_write: must be constant at compile time--false if for reading, true if for writing
|
||||
*/
|
||||
static inline void uds_prefetch_range(const void *start, unsigned int size,
|
||||
bool for_write)
|
||||
{
|
||||
/*
|
||||
* Count the number of cache lines to fetch, allowing for the address range to span an
|
||||
* extra cache line boundary due to address alignment.
|
||||
*/
|
||||
const char *address = (const char *) start;
|
||||
unsigned int offset = ((uintptr_t) address % L1_CACHE_BYTES);
|
||||
unsigned int cache_lines = (1 + ((size + offset) / L1_CACHE_BYTES));
|
||||
|
||||
while (cache_lines-- > 0) {
|
||||
uds_prefetch_address(address, for_write);
|
||||
address += L1_CACHE_BYTES;
|
||||
}
|
||||
}
|
||||
|
||||
#endif /* UDS_CPU_H */
|
File diff suppressed because it is too large
Load Diff
|
@ -0,0 +1,670 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef DATA_VIO_H
|
||||
#define DATA_VIO_H
|
||||
|
||||
#include <linux/atomic.h>
|
||||
#include <linux/bio.h>
|
||||
#include <linux/list.h>
|
||||
|
||||
#include "permassert.h"
|
||||
|
||||
#include "indexer.h"
|
||||
|
||||
#include "block-map.h"
|
||||
#include "completion.h"
|
||||
#include "constants.h"
|
||||
#include "dedupe.h"
|
||||
#include "encodings.h"
|
||||
#include "logical-zone.h"
|
||||
#include "physical-zone.h"
|
||||
#include "types.h"
|
||||
#include "vdo.h"
|
||||
#include "vio.h"
|
||||
#include "wait-queue.h"
|
||||
|
||||
/* Codes for describing the last asynchronous operation performed on a vio. */
|
||||
enum async_operation_number {
|
||||
MIN_VIO_ASYNC_OPERATION_NUMBER,
|
||||
VIO_ASYNC_OP_LAUNCH = MIN_VIO_ASYNC_OPERATION_NUMBER,
|
||||
VIO_ASYNC_OP_ACKNOWLEDGE_WRITE,
|
||||
VIO_ASYNC_OP_ACQUIRE_VDO_HASH_LOCK,
|
||||
VIO_ASYNC_OP_ATTEMPT_LOGICAL_BLOCK_LOCK,
|
||||
VIO_ASYNC_OP_LOCK_DUPLICATE_PBN,
|
||||
VIO_ASYNC_OP_CHECK_FOR_DUPLICATION,
|
||||
VIO_ASYNC_OP_CLEANUP,
|
||||
VIO_ASYNC_OP_COMPRESS_DATA_VIO,
|
||||
VIO_ASYNC_OP_FIND_BLOCK_MAP_SLOT,
|
||||
VIO_ASYNC_OP_GET_MAPPED_BLOCK_FOR_READ,
|
||||
VIO_ASYNC_OP_GET_MAPPED_BLOCK_FOR_WRITE,
|
||||
VIO_ASYNC_OP_HASH_DATA_VIO,
|
||||
VIO_ASYNC_OP_JOURNAL_REMAPPING,
|
||||
VIO_ASYNC_OP_ATTEMPT_PACKING,
|
||||
VIO_ASYNC_OP_PUT_MAPPED_BLOCK,
|
||||
VIO_ASYNC_OP_READ_DATA_VIO,
|
||||
VIO_ASYNC_OP_UPDATE_DEDUPE_INDEX,
|
||||
VIO_ASYNC_OP_UPDATE_REFERENCE_COUNTS,
|
||||
VIO_ASYNC_OP_VERIFY_DUPLICATION,
|
||||
VIO_ASYNC_OP_WRITE_DATA_VIO,
|
||||
MAX_VIO_ASYNC_OPERATION_NUMBER,
|
||||
} __packed;
|
||||
|
||||
struct lbn_lock {
|
||||
logical_block_number_t lbn;
|
||||
bool locked;
|
||||
struct vdo_wait_queue waiters;
|
||||
struct logical_zone *zone;
|
||||
};
|
||||
|
||||
/* A position in the arboreal block map at a specific level. */
|
||||
struct block_map_tree_slot {
|
||||
page_number_t page_index;
|
||||
struct block_map_slot block_map_slot;
|
||||
};
|
||||
|
||||
/* Fields for using the arboreal block map. */
|
||||
struct tree_lock {
|
||||
/* The current height at which this data_vio is operating */
|
||||
height_t height;
|
||||
/* The block map tree for this LBN */
|
||||
root_count_t root_index;
|
||||
/* Whether we hold a page lock */
|
||||
bool locked;
|
||||
/* The key for the lock map */
|
||||
u64 key;
|
||||
/* The queue of waiters for the page this vio is allocating or loading */
|
||||
struct vdo_wait_queue waiters;
|
||||
/* The block map tree slots for this LBN */
|
||||
struct block_map_tree_slot tree_slots[VDO_BLOCK_MAP_TREE_HEIGHT + 1];
|
||||
};
|
||||
|
||||
struct zoned_pbn {
|
||||
physical_block_number_t pbn;
|
||||
enum block_mapping_state state;
|
||||
struct physical_zone *zone;
|
||||
};
|
||||
|
||||
/*
|
||||
* Where a data_vio is on the compression path; advance_compression_stage() depends on the order of
|
||||
* this enum.
|
||||
*/
|
||||
enum data_vio_compression_stage {
|
||||
/* A data_vio which has not yet entered the compression path */
|
||||
DATA_VIO_PRE_COMPRESSOR,
|
||||
/* A data_vio which is in the compressor */
|
||||
DATA_VIO_COMPRESSING,
|
||||
/* A data_vio which is blocked in the packer */
|
||||
DATA_VIO_PACKING,
|
||||
/* A data_vio which is no longer on the compression path (and never will be) */
|
||||
DATA_VIO_POST_PACKER,
|
||||
};
|
||||
|
||||
struct data_vio_compression_status {
|
||||
enum data_vio_compression_stage stage;
|
||||
bool may_not_compress;
|
||||
};
|
||||
|
||||
struct compression_state {
|
||||
/*
|
||||
* The current compression status of this data_vio. This field contains a value which
|
||||
* consists of a data_vio_compression_stage and a flag indicating whether a request has
|
||||
* been made to cancel (or prevent) compression for this data_vio.
|
||||
*
|
||||
* This field should be accessed through the get_data_vio_compression_status() and
|
||||
* set_data_vio_compression_status() methods. It should not be accessed directly.
|
||||
*/
|
||||
atomic_t status;
|
||||
|
||||
/* The compressed size of this block */
|
||||
u16 size;
|
||||
|
||||
/* The packer input or output bin slot which holds the enclosing data_vio */
|
||||
slot_number_t slot;
|
||||
|
||||
/* The packer bin to which the enclosing data_vio has been assigned */
|
||||
struct packer_bin *bin;
|
||||
|
||||
/* A link in the chain of data_vios which have been packed together */
|
||||
struct data_vio *next_in_batch;
|
||||
|
||||
/* A vio which is blocked in the packer while holding a lock this vio needs. */
|
||||
struct data_vio *lock_holder;
|
||||
|
||||
/*
|
||||
* The compressed block used to hold the compressed form of this block and that of any
|
||||
* other blocks for which this data_vio is the compressed write agent.
|
||||
*/
|
||||
struct compressed_block *block;
|
||||
};
|
||||
|
||||
/* Fields supporting allocation of data blocks. */
|
||||
struct allocation {
|
||||
/* The physical zone in which to allocate a physical block */
|
||||
struct physical_zone *zone;
|
||||
|
||||
/* The block allocated to this vio */
|
||||
physical_block_number_t pbn;
|
||||
|
||||
/*
|
||||
* If non-NULL, the pooled PBN lock held on the allocated block. Must be a write lock until
|
||||
* the block has been written, after which it will become a read lock.
|
||||
*/
|
||||
struct pbn_lock *lock;
|
||||
|
||||
/* The type of write lock to obtain on the allocated block */
|
||||
enum pbn_lock_type write_lock_type;
|
||||
|
||||
/* The zone which was the start of the current allocation cycle */
|
||||
zone_count_t first_allocation_zone;
|
||||
|
||||
/* Whether this vio should wait for a clean slab */
|
||||
bool wait_for_clean_slab;
|
||||
};
|
||||
|
||||
struct reference_updater {
|
||||
enum journal_operation operation;
|
||||
bool increment;
|
||||
struct zoned_pbn zpbn;
|
||||
struct pbn_lock *lock;
|
||||
struct vdo_waiter waiter;
|
||||
};
|
||||
|
||||
/* A vio for processing user data requests. */
|
||||
struct data_vio {
|
||||
/* The vdo_wait_queue entry structure */
|
||||
struct vdo_waiter waiter;
|
||||
|
||||
/* The logical block of this request */
|
||||
struct lbn_lock logical;
|
||||
|
||||
/* The state for traversing the block map tree */
|
||||
struct tree_lock tree_lock;
|
||||
|
||||
/* The current partition address of this block */
|
||||
struct zoned_pbn mapped;
|
||||
|
||||
/* The hash of this vio (if not zero) */
|
||||
struct uds_record_name record_name;
|
||||
|
||||
/* Used for logging and debugging */
|
||||
enum async_operation_number last_async_operation;
|
||||
|
||||
/* The operations to record in the recovery and slab journals */
|
||||
struct reference_updater increment_updater;
|
||||
struct reference_updater decrement_updater;
|
||||
|
||||
u16 read : 1;
|
||||
u16 write : 1;
|
||||
u16 fua : 1;
|
||||
u16 is_zero : 1;
|
||||
u16 is_discard : 1;
|
||||
u16 is_partial : 1;
|
||||
u16 is_duplicate : 1;
|
||||
u16 first_reference_operation_complete : 1;
|
||||
u16 downgrade_allocation_lock : 1;
|
||||
|
||||
struct allocation allocation;
|
||||
|
||||
/*
|
||||
* Whether this vio has received an allocation. This field is examined from threads not in
|
||||
* the allocation zone.
|
||||
*/
|
||||
bool allocation_succeeded;
|
||||
|
||||
/* The new partition address of this block after the vio write completes */
|
||||
struct zoned_pbn new_mapped;
|
||||
|
||||
/* The hash zone responsible for the name (NULL if is_zero_block) */
|
||||
struct hash_zone *hash_zone;
|
||||
|
||||
/* The lock this vio holds or shares with other vios with the same data */
|
||||
struct hash_lock *hash_lock;
|
||||
|
||||
/* All data_vios sharing a hash lock are kept in a list linking these list entries */
|
||||
struct list_head hash_lock_entry;
|
||||
|
||||
/* The block number in the partition of the UDS deduplication advice */
|
||||
struct zoned_pbn duplicate;
|
||||
|
||||
/*
|
||||
* The sequence number of the recovery journal block containing the increment entry for
|
||||
* this vio.
|
||||
*/
|
||||
sequence_number_t recovery_sequence_number;
|
||||
|
||||
/* The point in the recovery journal where this write last made an entry */
|
||||
struct journal_point recovery_journal_point;
|
||||
|
||||
/* The list of vios in user initiated write requests */
|
||||
struct list_head write_entry;
|
||||
|
||||
/* The generation number of the VDO that this vio belongs to */
|
||||
sequence_number_t flush_generation;
|
||||
|
||||
/* The completion to use for fetching block map pages for this vio */
|
||||
struct vdo_page_completion page_completion;
|
||||
|
||||
/* The user bio that initiated this VIO */
|
||||
struct bio *user_bio;
|
||||
|
||||
/* partial block support */
|
||||
block_size_t offset;
|
||||
|
||||
/*
|
||||
* The number of bytes to be discarded. For discards, this field will always be positive,
|
||||
* whereas for non-discards it will always be 0. Hence it can be used to determine whether
|
||||
* a data_vio is processing a discard, even after the user_bio has been acknowledged.
|
||||
*/
|
||||
u32 remaining_discard;
|
||||
|
||||
struct dedupe_context *dedupe_context;
|
||||
|
||||
/* Fields beyond this point will not be reset when a pooled data_vio is reused. */
|
||||
|
||||
struct vio vio;
|
||||
|
||||
/* The completion for making reference count decrements */
|
||||
struct vdo_completion decrement_completion;
|
||||
|
||||
/* All of the fields necessary for the compression path */
|
||||
struct compression_state compression;
|
||||
|
||||
/* A block used as output during compression or uncompression */
|
||||
char *scratch_block;
|
||||
|
||||
struct list_head pool_entry;
|
||||
};
|
||||
|
||||
static inline struct data_vio *vio_as_data_vio(struct vio *vio)
|
||||
{
|
||||
VDO_ASSERT_LOG_ONLY((vio->type == VIO_TYPE_DATA), "vio is a data_vio");
|
||||
return container_of(vio, struct data_vio, vio);
|
||||
}
|
||||
|
||||
static inline struct data_vio *as_data_vio(struct vdo_completion *completion)
|
||||
{
|
||||
return vio_as_data_vio(as_vio(completion));
|
||||
}
|
||||
|
||||
static inline struct data_vio *vdo_waiter_as_data_vio(struct vdo_waiter *waiter)
|
||||
{
|
||||
if (waiter == NULL)
|
||||
return NULL;
|
||||
|
||||
return container_of(waiter, struct data_vio, waiter);
|
||||
}
|
||||
|
||||
static inline struct data_vio *data_vio_from_reference_updater(struct reference_updater *updater)
|
||||
{
|
||||
if (updater->increment)
|
||||
return container_of(updater, struct data_vio, increment_updater);
|
||||
|
||||
return container_of(updater, struct data_vio, decrement_updater);
|
||||
}
|
||||
|
||||
static inline bool data_vio_has_flush_generation_lock(struct data_vio *data_vio)
|
||||
{
|
||||
return !list_empty(&data_vio->write_entry);
|
||||
}
|
||||
|
||||
static inline struct vdo *vdo_from_data_vio(struct data_vio *data_vio)
|
||||
{
|
||||
return data_vio->vio.completion.vdo;
|
||||
}
|
||||
|
||||
static inline bool data_vio_has_allocation(struct data_vio *data_vio)
|
||||
{
|
||||
return (data_vio->allocation.pbn != VDO_ZERO_BLOCK);
|
||||
}
|
||||
|
||||
struct data_vio_compression_status __must_check
|
||||
advance_data_vio_compression_stage(struct data_vio *data_vio);
|
||||
struct data_vio_compression_status __must_check
|
||||
get_data_vio_compression_status(struct data_vio *data_vio);
|
||||
bool cancel_data_vio_compression(struct data_vio *data_vio);
|
||||
|
||||
struct data_vio_pool;
|
||||
|
||||
int make_data_vio_pool(struct vdo *vdo, data_vio_count_t pool_size,
|
||||
data_vio_count_t discard_limit, struct data_vio_pool **pool_ptr);
|
||||
void free_data_vio_pool(struct data_vio_pool *pool);
|
||||
void vdo_launch_bio(struct data_vio_pool *pool, struct bio *bio);
|
||||
void drain_data_vio_pool(struct data_vio_pool *pool, struct vdo_completion *completion);
|
||||
void resume_data_vio_pool(struct data_vio_pool *pool, struct vdo_completion *completion);
|
||||
|
||||
void dump_data_vio_pool(struct data_vio_pool *pool, bool dump_vios);
|
||||
data_vio_count_t get_data_vio_pool_active_discards(struct data_vio_pool *pool);
|
||||
data_vio_count_t get_data_vio_pool_discard_limit(struct data_vio_pool *pool);
|
||||
data_vio_count_t get_data_vio_pool_maximum_discards(struct data_vio_pool *pool);
|
||||
int __must_check set_data_vio_pool_discard_limit(struct data_vio_pool *pool,
|
||||
data_vio_count_t limit);
|
||||
data_vio_count_t get_data_vio_pool_active_requests(struct data_vio_pool *pool);
|
||||
data_vio_count_t get_data_vio_pool_request_limit(struct data_vio_pool *pool);
|
||||
data_vio_count_t get_data_vio_pool_maximum_requests(struct data_vio_pool *pool);
|
||||
|
||||
void complete_data_vio(struct vdo_completion *completion);
|
||||
void handle_data_vio_error(struct vdo_completion *completion);
|
||||
|
||||
static inline void continue_data_vio(struct data_vio *data_vio)
|
||||
{
|
||||
vdo_launch_completion(&data_vio->vio.completion);
|
||||
}
|
||||
|
||||
/**
|
||||
* continue_data_vio_with_error() - Set an error code and then continue processing a data_vio.
|
||||
*
|
||||
* This will not mask older errors. This function can be called with a success code, but it is more
|
||||
* efficient to call continue_data_vio() if the caller knows the result was a success.
|
||||
*/
|
||||
static inline void continue_data_vio_with_error(struct data_vio *data_vio, int result)
|
||||
{
|
||||
vdo_continue_completion(&data_vio->vio.completion, result);
|
||||
}
|
||||
|
||||
const char * __must_check get_data_vio_operation_name(struct data_vio *data_vio);
|
||||
|
||||
static inline void assert_data_vio_in_hash_zone(struct data_vio *data_vio)
|
||||
{
|
||||
thread_id_t expected = data_vio->hash_zone->thread_id;
|
||||
thread_id_t thread_id = vdo_get_callback_thread_id();
|
||||
/*
|
||||
* It's odd to use the LBN, but converting the record name to hex is a bit clunky for an
|
||||
* inline, and the LBN better than nothing as an identifier.
|
||||
*/
|
||||
VDO_ASSERT_LOG_ONLY((expected == thread_id),
|
||||
"data_vio for logical block %llu on thread %u, should be on hash zone thread %u",
|
||||
(unsigned long long) data_vio->logical.lbn, thread_id, expected);
|
||||
}
|
||||
|
||||
static inline void set_data_vio_hash_zone_callback(struct data_vio *data_vio,
|
||||
vdo_action_fn callback)
|
||||
{
|
||||
vdo_set_completion_callback(&data_vio->vio.completion, callback,
|
||||
data_vio->hash_zone->thread_id);
|
||||
}
|
||||
|
||||
/**
|
||||
* launch_data_vio_hash_zone_callback() - Set a callback as a hash zone operation and invoke it
|
||||
* immediately.
|
||||
*/
|
||||
static inline void launch_data_vio_hash_zone_callback(struct data_vio *data_vio,
|
||||
vdo_action_fn callback)
|
||||
{
|
||||
set_data_vio_hash_zone_callback(data_vio, callback);
|
||||
vdo_launch_completion(&data_vio->vio.completion);
|
||||
}
|
||||
|
||||
static inline void assert_data_vio_in_logical_zone(struct data_vio *data_vio)
|
||||
{
|
||||
thread_id_t expected = data_vio->logical.zone->thread_id;
|
||||
thread_id_t thread_id = vdo_get_callback_thread_id();
|
||||
|
||||
VDO_ASSERT_LOG_ONLY((expected == thread_id),
|
||||
"data_vio for logical block %llu on thread %u, should be on thread %u",
|
||||
(unsigned long long) data_vio->logical.lbn, thread_id, expected);
|
||||
}
|
||||
|
||||
static inline void set_data_vio_logical_callback(struct data_vio *data_vio,
|
||||
vdo_action_fn callback)
|
||||
{
|
||||
vdo_set_completion_callback(&data_vio->vio.completion, callback,
|
||||
data_vio->logical.zone->thread_id);
|
||||
}
|
||||
|
||||
/**
|
||||
* launch_data_vio_logical_callback() - Set a callback as a logical block operation and invoke it
|
||||
* immediately.
|
||||
*/
|
||||
static inline void launch_data_vio_logical_callback(struct data_vio *data_vio,
|
||||
vdo_action_fn callback)
|
||||
{
|
||||
set_data_vio_logical_callback(data_vio, callback);
|
||||
vdo_launch_completion(&data_vio->vio.completion);
|
||||
}
|
||||
|
||||
static inline void assert_data_vio_in_allocated_zone(struct data_vio *data_vio)
|
||||
{
|
||||
thread_id_t expected = data_vio->allocation.zone->thread_id;
|
||||
thread_id_t thread_id = vdo_get_callback_thread_id();
|
||||
|
||||
VDO_ASSERT_LOG_ONLY((expected == thread_id),
|
||||
"struct data_vio for allocated physical block %llu on thread %u, should be on thread %u",
|
||||
(unsigned long long) data_vio->allocation.pbn, thread_id,
|
||||
expected);
|
||||
}
|
||||
|
||||
static inline void set_data_vio_allocated_zone_callback(struct data_vio *data_vio,
|
||||
vdo_action_fn callback)
|
||||
{
|
||||
vdo_set_completion_callback(&data_vio->vio.completion, callback,
|
||||
data_vio->allocation.zone->thread_id);
|
||||
}
|
||||
|
||||
/**
|
||||
* launch_data_vio_allocated_zone_callback() - Set a callback as a physical block operation in a
|
||||
* data_vio's allocated zone and queue the data_vio and
|
||||
* invoke it immediately.
|
||||
*/
|
||||
static inline void launch_data_vio_allocated_zone_callback(struct data_vio *data_vio,
|
||||
vdo_action_fn callback)
|
||||
{
|
||||
set_data_vio_allocated_zone_callback(data_vio, callback);
|
||||
vdo_launch_completion(&data_vio->vio.completion);
|
||||
}
|
||||
|
||||
static inline void assert_data_vio_in_duplicate_zone(struct data_vio *data_vio)
|
||||
{
|
||||
thread_id_t expected = data_vio->duplicate.zone->thread_id;
|
||||
thread_id_t thread_id = vdo_get_callback_thread_id();
|
||||
|
||||
VDO_ASSERT_LOG_ONLY((expected == thread_id),
|
||||
"data_vio for duplicate physical block %llu on thread %u, should be on thread %u",
|
||||
(unsigned long long) data_vio->duplicate.pbn, thread_id,
|
||||
expected);
|
||||
}
|
||||
|
||||
static inline void set_data_vio_duplicate_zone_callback(struct data_vio *data_vio,
|
||||
vdo_action_fn callback)
|
||||
{
|
||||
vdo_set_completion_callback(&data_vio->vio.completion, callback,
|
||||
data_vio->duplicate.zone->thread_id);
|
||||
}
|
||||
|
||||
/**
|
||||
* launch_data_vio_duplicate_zone_callback() - Set a callback as a physical block operation in a
|
||||
* data_vio's duplicate zone and queue the data_vio and
|
||||
* invoke it immediately.
|
||||
*/
|
||||
static inline void launch_data_vio_duplicate_zone_callback(struct data_vio *data_vio,
|
||||
vdo_action_fn callback)
|
||||
{
|
||||
set_data_vio_duplicate_zone_callback(data_vio, callback);
|
||||
vdo_launch_completion(&data_vio->vio.completion);
|
||||
}
|
||||
|
||||
static inline void assert_data_vio_in_mapped_zone(struct data_vio *data_vio)
|
||||
{
|
||||
thread_id_t expected = data_vio->mapped.zone->thread_id;
|
||||
thread_id_t thread_id = vdo_get_callback_thread_id();
|
||||
|
||||
VDO_ASSERT_LOG_ONLY((expected == thread_id),
|
||||
"data_vio for mapped physical block %llu on thread %u, should be on thread %u",
|
||||
(unsigned long long) data_vio->mapped.pbn, thread_id, expected);
|
||||
}
|
||||
|
||||
static inline void set_data_vio_mapped_zone_callback(struct data_vio *data_vio,
|
||||
vdo_action_fn callback)
|
||||
{
|
||||
vdo_set_completion_callback(&data_vio->vio.completion, callback,
|
||||
data_vio->mapped.zone->thread_id);
|
||||
}
|
||||
|
||||
static inline void assert_data_vio_in_new_mapped_zone(struct data_vio *data_vio)
|
||||
{
|
||||
thread_id_t expected = data_vio->new_mapped.zone->thread_id;
|
||||
thread_id_t thread_id = vdo_get_callback_thread_id();
|
||||
|
||||
VDO_ASSERT_LOG_ONLY((expected == thread_id),
|
||||
"data_vio for new_mapped physical block %llu on thread %u, should be on thread %u",
|
||||
(unsigned long long) data_vio->new_mapped.pbn, thread_id,
|
||||
expected);
|
||||
}
|
||||
|
||||
static inline void set_data_vio_new_mapped_zone_callback(struct data_vio *data_vio,
|
||||
vdo_action_fn callback)
|
||||
{
|
||||
vdo_set_completion_callback(&data_vio->vio.completion, callback,
|
||||
data_vio->new_mapped.zone->thread_id);
|
||||
}
|
||||
|
||||
static inline void assert_data_vio_in_journal_zone(struct data_vio *data_vio)
|
||||
{
|
||||
thread_id_t journal_thread = vdo_from_data_vio(data_vio)->thread_config.journal_thread;
|
||||
thread_id_t thread_id = vdo_get_callback_thread_id();
|
||||
|
||||
VDO_ASSERT_LOG_ONLY((journal_thread == thread_id),
|
||||
"data_vio for logical block %llu on thread %u, should be on journal thread %u",
|
||||
(unsigned long long) data_vio->logical.lbn, thread_id,
|
||||
journal_thread);
|
||||
}
|
||||
|
||||
static inline void set_data_vio_journal_callback(struct data_vio *data_vio,
|
||||
vdo_action_fn callback)
|
||||
{
|
||||
thread_id_t journal_thread = vdo_from_data_vio(data_vio)->thread_config.journal_thread;
|
||||
|
||||
vdo_set_completion_callback(&data_vio->vio.completion, callback, journal_thread);
|
||||
}
|
||||
|
||||
/**
|
||||
* launch_data_vio_journal_callback() - Set a callback as a journal operation and invoke it
|
||||
* immediately.
|
||||
*/
|
||||
static inline void launch_data_vio_journal_callback(struct data_vio *data_vio,
|
||||
vdo_action_fn callback)
|
||||
{
|
||||
set_data_vio_journal_callback(data_vio, callback);
|
||||
vdo_launch_completion(&data_vio->vio.completion);
|
||||
}
|
||||
|
||||
static inline void assert_data_vio_in_packer_zone(struct data_vio *data_vio)
|
||||
{
|
||||
thread_id_t packer_thread = vdo_from_data_vio(data_vio)->thread_config.packer_thread;
|
||||
thread_id_t thread_id = vdo_get_callback_thread_id();
|
||||
|
||||
VDO_ASSERT_LOG_ONLY((packer_thread == thread_id),
|
||||
"data_vio for logical block %llu on thread %u, should be on packer thread %u",
|
||||
(unsigned long long) data_vio->logical.lbn, thread_id,
|
||||
packer_thread);
|
||||
}
|
||||
|
||||
static inline void set_data_vio_packer_callback(struct data_vio *data_vio,
|
||||
vdo_action_fn callback)
|
||||
{
|
||||
thread_id_t packer_thread = vdo_from_data_vio(data_vio)->thread_config.packer_thread;
|
||||
|
||||
vdo_set_completion_callback(&data_vio->vio.completion, callback, packer_thread);
|
||||
}
|
||||
|
||||
/**
|
||||
* launch_data_vio_packer_callback() - Set a callback as a packer operation and invoke it
|
||||
* immediately.
|
||||
*/
|
||||
static inline void launch_data_vio_packer_callback(struct data_vio *data_vio,
|
||||
vdo_action_fn callback)
|
||||
{
|
||||
set_data_vio_packer_callback(data_vio, callback);
|
||||
vdo_launch_completion(&data_vio->vio.completion);
|
||||
}
|
||||
|
||||
static inline void assert_data_vio_on_cpu_thread(struct data_vio *data_vio)
|
||||
{
|
||||
thread_id_t cpu_thread = vdo_from_data_vio(data_vio)->thread_config.cpu_thread;
|
||||
thread_id_t thread_id = vdo_get_callback_thread_id();
|
||||
|
||||
VDO_ASSERT_LOG_ONLY((cpu_thread == thread_id),
|
||||
"data_vio for logical block %llu on thread %u, should be on cpu thread %u",
|
||||
(unsigned long long) data_vio->logical.lbn, thread_id,
|
||||
cpu_thread);
|
||||
}
|
||||
|
||||
static inline void set_data_vio_cpu_callback(struct data_vio *data_vio,
|
||||
vdo_action_fn callback)
|
||||
{
|
||||
thread_id_t cpu_thread = vdo_from_data_vio(data_vio)->thread_config.cpu_thread;
|
||||
|
||||
vdo_set_completion_callback(&data_vio->vio.completion, callback, cpu_thread);
|
||||
}
|
||||
|
||||
/**
|
||||
* launch_data_vio_cpu_callback() - Set a callback to run on the CPU queues and invoke it
|
||||
* immediately.
|
||||
*/
|
||||
static inline void launch_data_vio_cpu_callback(struct data_vio *data_vio,
|
||||
vdo_action_fn callback,
|
||||
enum vdo_completion_priority priority)
|
||||
{
|
||||
set_data_vio_cpu_callback(data_vio, callback);
|
||||
vdo_launch_completion_with_priority(&data_vio->vio.completion, priority);
|
||||
}
|
||||
|
||||
static inline void set_data_vio_bio_zone_callback(struct data_vio *data_vio,
|
||||
vdo_action_fn callback)
|
||||
{
|
||||
vdo_set_completion_callback(&data_vio->vio.completion, callback,
|
||||
get_vio_bio_zone_thread_id(&data_vio->vio));
|
||||
}
|
||||
|
||||
/**
|
||||
* launch_data_vio_bio_zone_callback() - Set a callback as a bio zone operation and invoke it
|
||||
* immediately.
|
||||
*/
|
||||
static inline void launch_data_vio_bio_zone_callback(struct data_vio *data_vio,
|
||||
vdo_action_fn callback)
|
||||
{
|
||||
set_data_vio_bio_zone_callback(data_vio, callback);
|
||||
vdo_launch_completion_with_priority(&data_vio->vio.completion,
|
||||
BIO_Q_DATA_PRIORITY);
|
||||
}
|
||||
|
||||
/**
|
||||
* launch_data_vio_on_bio_ack_queue() - If the vdo uses a bio_ack queue, set a callback to run on
|
||||
* it and invoke it immediately, otherwise, just run the
|
||||
* callback on the current thread.
|
||||
*/
|
||||
static inline void launch_data_vio_on_bio_ack_queue(struct data_vio *data_vio,
|
||||
vdo_action_fn callback)
|
||||
{
|
||||
struct vdo_completion *completion = &data_vio->vio.completion;
|
||||
struct vdo *vdo = completion->vdo;
|
||||
|
||||
if (!vdo_uses_bio_ack_queue(vdo)) {
|
||||
callback(completion);
|
||||
return;
|
||||
}
|
||||
|
||||
vdo_set_completion_callback(completion, callback,
|
||||
vdo->thread_config.bio_ack_thread);
|
||||
vdo_launch_completion_with_priority(completion, BIO_ACK_Q_ACK_PRIORITY);
|
||||
}
|
||||
|
||||
void data_vio_allocate_data_block(struct data_vio *data_vio,
|
||||
enum pbn_lock_type write_lock_type,
|
||||
vdo_action_fn callback, vdo_action_fn error_handler);
|
||||
|
||||
void release_data_vio_allocation_lock(struct data_vio *data_vio, bool reset);
|
||||
|
||||
int __must_check uncompress_data_vio(struct data_vio *data_vio,
|
||||
enum block_mapping_state mapping_state,
|
||||
char *buffer);
|
||||
|
||||
void update_metadata_for_data_vio_write(struct data_vio *data_vio,
|
||||
struct pbn_lock *lock);
|
||||
void write_data_vio(struct data_vio *data_vio);
|
||||
void launch_compress_data_vio(struct data_vio *data_vio);
|
||||
void continue_data_vio_with_block_map_slot(struct vdo_completion *completion);
|
||||
|
||||
#endif /* DATA_VIO_H */
|
File diff suppressed because it is too large
Load Diff
|
@ -0,0 +1,120 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef VDO_DEDUPE_H
|
||||
#define VDO_DEDUPE_H
|
||||
|
||||
#include <linux/list.h>
|
||||
#include <linux/timer.h>
|
||||
|
||||
#include "indexer.h"
|
||||
|
||||
#include "admin-state.h"
|
||||
#include "constants.h"
|
||||
#include "statistics.h"
|
||||
#include "types.h"
|
||||
#include "wait-queue.h"
|
||||
|
||||
struct dedupe_context {
|
||||
struct hash_zone *zone;
|
||||
struct uds_request request;
|
||||
struct list_head list_entry;
|
||||
struct funnel_queue_entry queue_entry;
|
||||
u64 submission_jiffies;
|
||||
struct data_vio *requestor;
|
||||
atomic_t state;
|
||||
};
|
||||
|
||||
struct hash_lock;
|
||||
|
||||
struct hash_zone {
|
||||
/* Which hash zone this is */
|
||||
zone_count_t zone_number;
|
||||
|
||||
/* The administrative state of the zone */
|
||||
struct admin_state state;
|
||||
|
||||
/* The thread ID for this zone */
|
||||
thread_id_t thread_id;
|
||||
|
||||
/* Mapping from record name fields to hash_locks */
|
||||
struct int_map *hash_lock_map;
|
||||
|
||||
/* List containing all unused hash_locks */
|
||||
struct list_head lock_pool;
|
||||
|
||||
/*
|
||||
* Statistics shared by all hash locks in this zone. Only modified on the hash zone thread,
|
||||
* but queried by other threads.
|
||||
*/
|
||||
struct hash_lock_statistics statistics;
|
||||
|
||||
/* Array of all hash_locks */
|
||||
struct hash_lock *lock_array;
|
||||
|
||||
/* These fields are used to manage the dedupe contexts */
|
||||
struct list_head available;
|
||||
struct list_head pending;
|
||||
struct funnel_queue *timed_out_complete;
|
||||
struct timer_list timer;
|
||||
struct vdo_completion completion;
|
||||
unsigned int active;
|
||||
atomic_t timer_state;
|
||||
|
||||
/* The dedupe contexts for querying the index from this zone */
|
||||
struct dedupe_context contexts[MAXIMUM_VDO_USER_VIOS];
|
||||
};
|
||||
|
||||
struct hash_zones;
|
||||
|
||||
struct pbn_lock * __must_check vdo_get_duplicate_lock(struct data_vio *data_vio);
|
||||
|
||||
void vdo_acquire_hash_lock(struct vdo_completion *completion);
|
||||
void vdo_continue_hash_lock(struct vdo_completion *completion);
|
||||
void vdo_release_hash_lock(struct data_vio *data_vio);
|
||||
void vdo_clean_failed_hash_lock(struct data_vio *data_vio);
|
||||
void vdo_share_compressed_write_lock(struct data_vio *data_vio,
|
||||
struct pbn_lock *pbn_lock);
|
||||
|
||||
int __must_check vdo_make_hash_zones(struct vdo *vdo, struct hash_zones **zones_ptr);
|
||||
|
||||
void vdo_free_hash_zones(struct hash_zones *zones);
|
||||
|
||||
void vdo_drain_hash_zones(struct hash_zones *zones, struct vdo_completion *parent);
|
||||
|
||||
void vdo_get_dedupe_statistics(struct hash_zones *zones, struct vdo_statistics *stats);
|
||||
|
||||
struct hash_zone * __must_check vdo_select_hash_zone(struct hash_zones *zones,
|
||||
const struct uds_record_name *name);
|
||||
|
||||
void vdo_dump_hash_zones(struct hash_zones *zones);
|
||||
|
||||
const char *vdo_get_dedupe_index_state_name(struct hash_zones *zones);
|
||||
|
||||
u64 vdo_get_dedupe_index_timeout_count(struct hash_zones *zones);
|
||||
|
||||
int vdo_message_dedupe_index(struct hash_zones *zones, const char *name);
|
||||
|
||||
void vdo_set_dedupe_state_normal(struct hash_zones *zones);
|
||||
|
||||
void vdo_start_dedupe_index(struct hash_zones *zones, bool create_flag);
|
||||
|
||||
void vdo_resume_hash_zones(struct hash_zones *zones, struct vdo_completion *parent);
|
||||
|
||||
void vdo_finish_dedupe_index(struct hash_zones *zones);
|
||||
|
||||
/* Interval (in milliseconds) from submission until switching to fast path and skipping UDS. */
|
||||
extern unsigned int vdo_dedupe_index_timeout_interval;
|
||||
|
||||
/*
|
||||
* Minimum time interval (in milliseconds) between timer invocations to check for requests waiting
|
||||
* for UDS that should now time out.
|
||||
*/
|
||||
extern unsigned int vdo_dedupe_index_min_timer_interval;
|
||||
|
||||
void vdo_set_dedupe_index_timeout_interval(unsigned int value);
|
||||
void vdo_set_dedupe_index_min_timer_interval(unsigned int value);
|
||||
|
||||
#endif /* VDO_DEDUPE_H */
|
File diff suppressed because it is too large
Load Diff
|
@ -0,0 +1,275 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "dump.h"
|
||||
|
||||
#include <linux/module.h>
|
||||
|
||||
#include "memory-alloc.h"
|
||||
#include "string-utils.h"
|
||||
|
||||
#include "constants.h"
|
||||
#include "data-vio.h"
|
||||
#include "dedupe.h"
|
||||
#include "funnel-workqueue.h"
|
||||
#include "io-submitter.h"
|
||||
#include "logger.h"
|
||||
#include "types.h"
|
||||
#include "vdo.h"
|
||||
|
||||
enum dump_options {
|
||||
/* Work queues */
|
||||
SHOW_QUEUES,
|
||||
/* Memory pools */
|
||||
SHOW_VIO_POOL,
|
||||
/* Others */
|
||||
SHOW_VDO_STATUS,
|
||||
/* This one means an option overrides the "default" choices, instead of altering them. */
|
||||
SKIP_DEFAULT
|
||||
};
|
||||
|
||||
enum dump_option_flags {
|
||||
/* Work queues */
|
||||
FLAG_SHOW_QUEUES = (1 << SHOW_QUEUES),
|
||||
/* Memory pools */
|
||||
FLAG_SHOW_VIO_POOL = (1 << SHOW_VIO_POOL),
|
||||
/* Others */
|
||||
FLAG_SHOW_VDO_STATUS = (1 << SHOW_VDO_STATUS),
|
||||
/* Special */
|
||||
FLAG_SKIP_DEFAULT = (1 << SKIP_DEFAULT)
|
||||
};
|
||||
|
||||
#define FLAGS_ALL_POOLS (FLAG_SHOW_VIO_POOL)
|
||||
#define DEFAULT_DUMP_FLAGS (FLAG_SHOW_QUEUES | FLAG_SHOW_VDO_STATUS)
|
||||
/* Another static buffer... log10(256) = 2.408+, round up: */
|
||||
#define DIGITS_PER_U64 (1 + sizeof(u64) * 2409 / 1000)
|
||||
|
||||
static inline bool is_arg_string(const char *arg, const char *this_option)
|
||||
{
|
||||
/* convention seems to be case-independent options */
|
||||
return strncasecmp(arg, this_option, strlen(this_option)) == 0;
|
||||
}
|
||||
|
||||
static void do_dump(struct vdo *vdo, unsigned int dump_options_requested,
|
||||
const char *why)
|
||||
{
|
||||
u32 active, maximum;
|
||||
s64 outstanding;
|
||||
|
||||
vdo_log_info("%s dump triggered via %s", VDO_LOGGING_MODULE_NAME, why);
|
||||
active = get_data_vio_pool_active_requests(vdo->data_vio_pool);
|
||||
maximum = get_data_vio_pool_maximum_requests(vdo->data_vio_pool);
|
||||
outstanding = (atomic64_read(&vdo->stats.bios_submitted) -
|
||||
atomic64_read(&vdo->stats.bios_completed));
|
||||
vdo_log_info("%u device requests outstanding (max %u), %lld bio requests outstanding, device '%s'",
|
||||
active, maximum, outstanding,
|
||||
vdo_get_device_name(vdo->device_config->owning_target));
|
||||
if (((dump_options_requested & FLAG_SHOW_QUEUES) != 0) && (vdo->threads != NULL)) {
|
||||
thread_id_t id;
|
||||
|
||||
for (id = 0; id < vdo->thread_config.thread_count; id++)
|
||||
vdo_dump_work_queue(vdo->threads[id].queue);
|
||||
}
|
||||
|
||||
vdo_dump_hash_zones(vdo->hash_zones);
|
||||
dump_data_vio_pool(vdo->data_vio_pool,
|
||||
(dump_options_requested & FLAG_SHOW_VIO_POOL) != 0);
|
||||
if ((dump_options_requested & FLAG_SHOW_VDO_STATUS) != 0)
|
||||
vdo_dump_status(vdo);
|
||||
|
||||
vdo_report_memory_usage();
|
||||
vdo_log_info("end of %s dump", VDO_LOGGING_MODULE_NAME);
|
||||
}
|
||||
|
||||
static int parse_dump_options(unsigned int argc, char *const *argv,
|
||||
unsigned int *dump_options_requested_ptr)
|
||||
{
|
||||
unsigned int dump_options_requested = 0;
|
||||
|
||||
static const struct {
|
||||
const char *name;
|
||||
unsigned int flags;
|
||||
} option_names[] = {
|
||||
{ "viopool", FLAG_SKIP_DEFAULT | FLAG_SHOW_VIO_POOL },
|
||||
{ "vdo", FLAG_SKIP_DEFAULT | FLAG_SHOW_VDO_STATUS },
|
||||
{ "pools", FLAG_SKIP_DEFAULT | FLAGS_ALL_POOLS },
|
||||
{ "queues", FLAG_SKIP_DEFAULT | FLAG_SHOW_QUEUES },
|
||||
{ "threads", FLAG_SKIP_DEFAULT | FLAG_SHOW_QUEUES },
|
||||
{ "default", FLAG_SKIP_DEFAULT | DEFAULT_DUMP_FLAGS },
|
||||
{ "all", ~0 },
|
||||
};
|
||||
|
||||
bool options_okay = true;
|
||||
unsigned int i;
|
||||
|
||||
for (i = 1; i < argc; i++) {
|
||||
unsigned int j;
|
||||
|
||||
for (j = 0; j < ARRAY_SIZE(option_names); j++) {
|
||||
if (is_arg_string(argv[i], option_names[j].name)) {
|
||||
dump_options_requested |= option_names[j].flags;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (j == ARRAY_SIZE(option_names)) {
|
||||
vdo_log_warning("dump option name '%s' unknown", argv[i]);
|
||||
options_okay = false;
|
||||
}
|
||||
}
|
||||
if (!options_okay)
|
||||
return -EINVAL;
|
||||
if ((dump_options_requested & FLAG_SKIP_DEFAULT) == 0)
|
||||
dump_options_requested |= DEFAULT_DUMP_FLAGS;
|
||||
*dump_options_requested_ptr = dump_options_requested;
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Dump as specified by zero or more string arguments. */
|
||||
int vdo_dump(struct vdo *vdo, unsigned int argc, char *const *argv, const char *why)
|
||||
{
|
||||
unsigned int dump_options_requested = 0;
|
||||
int result = parse_dump_options(argc, argv, &dump_options_requested);
|
||||
|
||||
if (result != 0)
|
||||
return result;
|
||||
|
||||
do_dump(vdo, dump_options_requested, why);
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Dump everything we know how to dump */
|
||||
void vdo_dump_all(struct vdo *vdo, const char *why)
|
||||
{
|
||||
do_dump(vdo, ~0, why);
|
||||
}
|
||||
|
||||
/*
|
||||
* Dump out the data_vio waiters on a waitq.
|
||||
* wait_on should be the label to print for queue (e.g. logical or physical)
|
||||
*/
|
||||
static void dump_vio_waiters(struct vdo_wait_queue *waitq, char *wait_on)
|
||||
{
|
||||
struct vdo_waiter *waiter, *first = vdo_waitq_get_first_waiter(waitq);
|
||||
struct data_vio *data_vio;
|
||||
|
||||
if (first == NULL)
|
||||
return;
|
||||
|
||||
data_vio = vdo_waiter_as_data_vio(first);
|
||||
|
||||
vdo_log_info(" %s is locked. Waited on by: vio %px pbn %llu lbn %llu d-pbn %llu lastOp %s",
|
||||
wait_on, data_vio, data_vio->allocation.pbn, data_vio->logical.lbn,
|
||||
data_vio->duplicate.pbn, get_data_vio_operation_name(data_vio));
|
||||
|
||||
for (waiter = first->next_waiter; waiter != first; waiter = waiter->next_waiter) {
|
||||
data_vio = vdo_waiter_as_data_vio(waiter);
|
||||
vdo_log_info(" ... and : vio %px pbn %llu lbn %llu d-pbn %llu lastOp %s",
|
||||
data_vio, data_vio->allocation.pbn, data_vio->logical.lbn,
|
||||
data_vio->duplicate.pbn,
|
||||
get_data_vio_operation_name(data_vio));
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
* Encode various attributes of a data_vio as a string of one-character flags. This encoding is for
|
||||
* logging brevity:
|
||||
*
|
||||
* R => vio completion result not VDO_SUCCESS
|
||||
* W => vio is on a waitq
|
||||
* D => vio is a duplicate
|
||||
* p => vio is a partial block operation
|
||||
* z => vio is a zero block
|
||||
* d => vio is a discard
|
||||
*
|
||||
* The common case of no flags set will result in an empty, null-terminated buffer. If any flags
|
||||
* are encoded, the first character in the string will be a space character.
|
||||
*/
|
||||
static void encode_vio_dump_flags(struct data_vio *data_vio, char buffer[8])
|
||||
{
|
||||
char *p_flag = buffer;
|
||||
*p_flag++ = ' ';
|
||||
if (data_vio->vio.completion.result != VDO_SUCCESS)
|
||||
*p_flag++ = 'R';
|
||||
if (data_vio->waiter.next_waiter != NULL)
|
||||
*p_flag++ = 'W';
|
||||
if (data_vio->is_duplicate)
|
||||
*p_flag++ = 'D';
|
||||
if (data_vio->is_partial)
|
||||
*p_flag++ = 'p';
|
||||
if (data_vio->is_zero)
|
||||
*p_flag++ = 'z';
|
||||
if (data_vio->remaining_discard > 0)
|
||||
*p_flag++ = 'd';
|
||||
if (p_flag == &buffer[1]) {
|
||||
/* No flags, so remove the blank space. */
|
||||
p_flag = buffer;
|
||||
}
|
||||
*p_flag = '\0';
|
||||
}
|
||||
|
||||
/* Implements buffer_dump_function. */
|
||||
void dump_data_vio(void *data)
|
||||
{
|
||||
struct data_vio *data_vio = data;
|
||||
|
||||
/*
|
||||
* This just needs to be big enough to hold a queue (thread) name and a function name (plus
|
||||
* a separator character and NUL). The latter is limited only by taste.
|
||||
*
|
||||
* In making this static, we're assuming only one "dump" will run at a time. If more than
|
||||
* one does run, the log output will be garbled anyway.
|
||||
*/
|
||||
static char vio_completion_dump_buffer[100 + MAX_VDO_WORK_QUEUE_NAME_LEN];
|
||||
static char vio_block_number_dump_buffer[sizeof("P L D") + 3 * DIGITS_PER_U64];
|
||||
static char vio_flush_generation_buffer[sizeof(" FG") + DIGITS_PER_U64];
|
||||
static char flags_dump_buffer[8];
|
||||
|
||||
/*
|
||||
* We're likely to be logging a couple thousand of these lines, and in some circumstances
|
||||
* syslogd may have trouble keeping up, so keep it BRIEF rather than user-friendly.
|
||||
*/
|
||||
vdo_dump_completion_to_buffer(&data_vio->vio.completion,
|
||||
vio_completion_dump_buffer,
|
||||
sizeof(vio_completion_dump_buffer));
|
||||
if (data_vio->is_duplicate) {
|
||||
snprintf(vio_block_number_dump_buffer,
|
||||
sizeof(vio_block_number_dump_buffer), "P%llu L%llu D%llu",
|
||||
data_vio->allocation.pbn, data_vio->logical.lbn,
|
||||
data_vio->duplicate.pbn);
|
||||
} else if (data_vio_has_allocation(data_vio)) {
|
||||
snprintf(vio_block_number_dump_buffer,
|
||||
sizeof(vio_block_number_dump_buffer), "P%llu L%llu",
|
||||
data_vio->allocation.pbn, data_vio->logical.lbn);
|
||||
} else {
|
||||
snprintf(vio_block_number_dump_buffer,
|
||||
sizeof(vio_block_number_dump_buffer), "L%llu",
|
||||
data_vio->logical.lbn);
|
||||
}
|
||||
|
||||
if (data_vio->flush_generation != 0) {
|
||||
snprintf(vio_flush_generation_buffer,
|
||||
sizeof(vio_flush_generation_buffer), " FG%llu",
|
||||
data_vio->flush_generation);
|
||||
} else {
|
||||
vio_flush_generation_buffer[0] = 0;
|
||||
}
|
||||
|
||||
encode_vio_dump_flags(data_vio, flags_dump_buffer);
|
||||
|
||||
vdo_log_info(" vio %px %s%s %s %s%s", data_vio,
|
||||
vio_block_number_dump_buffer,
|
||||
vio_flush_generation_buffer,
|
||||
get_data_vio_operation_name(data_vio),
|
||||
vio_completion_dump_buffer,
|
||||
flags_dump_buffer);
|
||||
/*
|
||||
* might want info on: wantUDSAnswer / operation / status
|
||||
* might want info on: bio / bios_merged
|
||||
*/
|
||||
|
||||
dump_vio_waiters(&data_vio->logical.waiters, "lbn");
|
||||
|
||||
/* might want to dump more info from vio here */
|
||||
}
|
|
@ -0,0 +1,17 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef VDO_DUMP_H
|
||||
#define VDO_DUMP_H
|
||||
|
||||
#include "types.h"
|
||||
|
||||
int vdo_dump(struct vdo *vdo, unsigned int argc, char *const *argv, const char *why);
|
||||
|
||||
void vdo_dump_all(struct vdo *vdo, const char *why);
|
||||
|
||||
void dump_data_vio(void *data);
|
||||
|
||||
#endif /* VDO_DUMP_H */
|
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
|
@ -0,0 +1,307 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "errors.h"
|
||||
|
||||
#include <linux/compiler.h>
|
||||
#include <linux/errno.h>
|
||||
|
||||
#include "logger.h"
|
||||
#include "permassert.h"
|
||||
#include "string-utils.h"
|
||||
|
||||
static const struct error_info successful = { "UDS_SUCCESS", "Success" };
|
||||
|
||||
static const char *const message_table[] = {
|
||||
[EPERM] = "Operation not permitted",
|
||||
[ENOENT] = "No such file or directory",
|
||||
[ESRCH] = "No such process",
|
||||
[EINTR] = "Interrupted system call",
|
||||
[EIO] = "Input/output error",
|
||||
[ENXIO] = "No such device or address",
|
||||
[E2BIG] = "Argument list too long",
|
||||
[ENOEXEC] = "Exec format error",
|
||||
[EBADF] = "Bad file descriptor",
|
||||
[ECHILD] = "No child processes",
|
||||
[EAGAIN] = "Resource temporarily unavailable",
|
||||
[ENOMEM] = "Cannot allocate memory",
|
||||
[EACCES] = "Permission denied",
|
||||
[EFAULT] = "Bad address",
|
||||
[ENOTBLK] = "Block device required",
|
||||
[EBUSY] = "Device or resource busy",
|
||||
[EEXIST] = "File exists",
|
||||
[EXDEV] = "Invalid cross-device link",
|
||||
[ENODEV] = "No such device",
|
||||
[ENOTDIR] = "Not a directory",
|
||||
[EISDIR] = "Is a directory",
|
||||
[EINVAL] = "Invalid argument",
|
||||
[ENFILE] = "Too many open files in system",
|
||||
[EMFILE] = "Too many open files",
|
||||
[ENOTTY] = "Inappropriate ioctl for device",
|
||||
[ETXTBSY] = "Text file busy",
|
||||
[EFBIG] = "File too large",
|
||||
[ENOSPC] = "No space left on device",
|
||||
[ESPIPE] = "Illegal seek",
|
||||
[EROFS] = "Read-only file system",
|
||||
[EMLINK] = "Too many links",
|
||||
[EPIPE] = "Broken pipe",
|
||||
[EDOM] = "Numerical argument out of domain",
|
||||
[ERANGE] = "Numerical result out of range"
|
||||
};
|
||||
|
||||
static const struct error_info error_list[] = {
|
||||
{ "UDS_OVERFLOW", "Index overflow" },
|
||||
{ "UDS_INVALID_ARGUMENT", "Invalid argument passed to internal routine" },
|
||||
{ "UDS_BAD_STATE", "UDS data structures are in an invalid state" },
|
||||
{ "UDS_DUPLICATE_NAME", "Attempt to enter the same name into a delta index twice" },
|
||||
{ "UDS_ASSERTION_FAILED", "Assertion failed" },
|
||||
{ "UDS_QUEUED", "Request queued" },
|
||||
{ "UDS_ALREADY_REGISTERED", "Error range already registered" },
|
||||
{ "UDS_OUT_OF_RANGE", "Cannot access data outside specified limits" },
|
||||
{ "UDS_DISABLED", "UDS library context is disabled" },
|
||||
{ "UDS_UNSUPPORTED_VERSION", "Unsupported version" },
|
||||
{ "UDS_CORRUPT_DATA", "Some index structure is corrupt" },
|
||||
{ "UDS_NO_INDEX", "No index found" },
|
||||
{ "UDS_INDEX_NOT_SAVED_CLEANLY", "Index not saved cleanly" },
|
||||
};
|
||||
|
||||
struct error_block {
|
||||
const char *name;
|
||||
int base;
|
||||
int last;
|
||||
int max;
|
||||
const struct error_info *infos;
|
||||
};
|
||||
|
||||
#define MAX_ERROR_BLOCKS 6
|
||||
|
||||
static struct {
|
||||
int allocated;
|
||||
int count;
|
||||
struct error_block blocks[MAX_ERROR_BLOCKS];
|
||||
} registered_errors = {
|
||||
.allocated = MAX_ERROR_BLOCKS,
|
||||
.count = 1,
|
||||
.blocks = { {
|
||||
.name = "UDS Error",
|
||||
.base = UDS_ERROR_CODE_BASE,
|
||||
.last = UDS_ERROR_CODE_LAST,
|
||||
.max = UDS_ERROR_CODE_BLOCK_END,
|
||||
.infos = error_list,
|
||||
} },
|
||||
};
|
||||
|
||||
/* Get the error info for an error number. Also returns the name of the error block, if known. */
|
||||
static const char *get_error_info(int errnum, const struct error_info **info_ptr)
|
||||
{
|
||||
struct error_block *block;
|
||||
|
||||
if (errnum == UDS_SUCCESS) {
|
||||
*info_ptr = &successful;
|
||||
return NULL;
|
||||
}
|
||||
|
||||
for (block = registered_errors.blocks;
|
||||
block < registered_errors.blocks + registered_errors.count;
|
||||
block++) {
|
||||
if ((errnum >= block->base) && (errnum < block->last)) {
|
||||
*info_ptr = block->infos + (errnum - block->base);
|
||||
return block->name;
|
||||
} else if ((errnum >= block->last) && (errnum < block->max)) {
|
||||
*info_ptr = NULL;
|
||||
return block->name;
|
||||
}
|
||||
}
|
||||
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/* Return a string describing a system error message. */
|
||||
static const char *system_string_error(int errnum, char *buf, size_t buflen)
|
||||
{
|
||||
size_t len;
|
||||
const char *error_string = NULL;
|
||||
|
||||
if ((errnum > 0) && (errnum < ARRAY_SIZE(message_table)))
|
||||
error_string = message_table[errnum];
|
||||
|
||||
len = ((error_string == NULL) ?
|
||||
snprintf(buf, buflen, "Unknown error %d", errnum) :
|
||||
snprintf(buf, buflen, "%s", error_string));
|
||||
if (len < buflen)
|
||||
return buf;
|
||||
|
||||
buf[0] = '\0';
|
||||
return "System error";
|
||||
}
|
||||
|
||||
/* Convert an error code to a descriptive string. */
|
||||
const char *uds_string_error(int errnum, char *buf, size_t buflen)
|
||||
{
|
||||
char *buffer = buf;
|
||||
char *buf_end = buf + buflen;
|
||||
const struct error_info *info = NULL;
|
||||
const char *block_name;
|
||||
|
||||
if (buf == NULL)
|
||||
return NULL;
|
||||
|
||||
if (errnum < 0)
|
||||
errnum = -errnum;
|
||||
|
||||
block_name = get_error_info(errnum, &info);
|
||||
if (block_name != NULL) {
|
||||
if (info != NULL) {
|
||||
buffer = vdo_append_to_buffer(buffer, buf_end, "%s: %s",
|
||||
block_name, info->message);
|
||||
} else {
|
||||
buffer = vdo_append_to_buffer(buffer, buf_end, "Unknown %s %d",
|
||||
block_name, errnum);
|
||||
}
|
||||
} else if (info != NULL) {
|
||||
buffer = vdo_append_to_buffer(buffer, buf_end, "%s", info->message);
|
||||
} else {
|
||||
const char *tmp = system_string_error(errnum, buffer, buf_end - buffer);
|
||||
|
||||
if (tmp != buffer)
|
||||
buffer = vdo_append_to_buffer(buffer, buf_end, "%s", tmp);
|
||||
else
|
||||
buffer += strlen(tmp);
|
||||
}
|
||||
|
||||
return buf;
|
||||
}
|
||||
|
||||
/* Convert an error code to its name. */
|
||||
const char *uds_string_error_name(int errnum, char *buf, size_t buflen)
|
||||
{
|
||||
char *buffer = buf;
|
||||
char *buf_end = buf + buflen;
|
||||
const struct error_info *info = NULL;
|
||||
const char *block_name;
|
||||
|
||||
if (errnum < 0)
|
||||
errnum = -errnum;
|
||||
|
||||
block_name = get_error_info(errnum, &info);
|
||||
if (block_name != NULL) {
|
||||
if (info != NULL) {
|
||||
buffer = vdo_append_to_buffer(buffer, buf_end, "%s", info->name);
|
||||
} else {
|
||||
buffer = vdo_append_to_buffer(buffer, buf_end, "%s %d",
|
||||
block_name, errnum);
|
||||
}
|
||||
} else if (info != NULL) {
|
||||
buffer = vdo_append_to_buffer(buffer, buf_end, "%s", info->name);
|
||||
} else {
|
||||
const char *tmp;
|
||||
|
||||
tmp = system_string_error(errnum, buffer, buf_end - buffer);
|
||||
if (tmp != buffer)
|
||||
buffer = vdo_append_to_buffer(buffer, buf_end, "%s", tmp);
|
||||
else
|
||||
buffer += strlen(tmp);
|
||||
}
|
||||
|
||||
return buf;
|
||||
}
|
||||
|
||||
/*
|
||||
* Translate an error code into a value acceptable to the kernel. The input error code may be a
|
||||
* system-generated value (such as -EIO), or an internal UDS status code. The result will be a
|
||||
* negative errno value.
|
||||
*/
|
||||
int uds_status_to_errno(int error)
|
||||
{
|
||||
char error_name[VDO_MAX_ERROR_NAME_SIZE];
|
||||
char error_message[VDO_MAX_ERROR_MESSAGE_SIZE];
|
||||
|
||||
/* 0 is success, and negative values are already system error codes. */
|
||||
if (likely(error <= 0))
|
||||
return error;
|
||||
|
||||
if (error < 1024) {
|
||||
/* This is probably an errno from userspace. */
|
||||
return -error;
|
||||
}
|
||||
|
||||
/* Internal UDS errors */
|
||||
switch (error) {
|
||||
case UDS_NO_INDEX:
|
||||
case UDS_CORRUPT_DATA:
|
||||
/* The index doesn't exist or can't be recovered. */
|
||||
return -ENOENT;
|
||||
|
||||
case UDS_INDEX_NOT_SAVED_CLEANLY:
|
||||
case UDS_UNSUPPORTED_VERSION:
|
||||
/*
|
||||
* The index exists, but can't be loaded. Tell the client it exists so they don't
|
||||
* destroy it inadvertently.
|
||||
*/
|
||||
return -EEXIST;
|
||||
|
||||
case UDS_DISABLED:
|
||||
/* The session is unusable; only returned by requests. */
|
||||
return -EIO;
|
||||
|
||||
default:
|
||||
/* Translate an unexpected error into something generic. */
|
||||
vdo_log_info("%s: mapping status code %d (%s: %s) to -EIO",
|
||||
__func__, error,
|
||||
uds_string_error_name(error, error_name,
|
||||
sizeof(error_name)),
|
||||
uds_string_error(error, error_message,
|
||||
sizeof(error_message)));
|
||||
return -EIO;
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
* Register a block of error codes.
|
||||
*
|
||||
* @block_name: the name of the block of error codes
|
||||
* @first_error: the first error code in the block
|
||||
* @next_free_error: one past the highest possible error in the block
|
||||
* @infos: a pointer to the error info array for the block
|
||||
* @info_size: the size of the error info array
|
||||
*/
|
||||
int uds_register_error_block(const char *block_name, int first_error,
|
||||
int next_free_error, const struct error_info *infos,
|
||||
size_t info_size)
|
||||
{
|
||||
int result;
|
||||
struct error_block *block;
|
||||
struct error_block new_block = {
|
||||
.name = block_name,
|
||||
.base = first_error,
|
||||
.last = first_error + (info_size / sizeof(struct error_info)),
|
||||
.max = next_free_error,
|
||||
.infos = infos,
|
||||
};
|
||||
|
||||
result = VDO_ASSERT(first_error < next_free_error,
|
||||
"well-defined error block range");
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
if (registered_errors.count == registered_errors.allocated) {
|
||||
/* This should never happen. */
|
||||
return UDS_OVERFLOW;
|
||||
}
|
||||
|
||||
for (block = registered_errors.blocks;
|
||||
block < registered_errors.blocks + registered_errors.count;
|
||||
block++) {
|
||||
if (strcmp(block_name, block->name) == 0)
|
||||
return UDS_DUPLICATE_NAME;
|
||||
|
||||
/* Ensure error ranges do not overlap. */
|
||||
if ((first_error < block->max) && (next_free_error > block->base))
|
||||
return UDS_ALREADY_REGISTERED;
|
||||
}
|
||||
|
||||
registered_errors.blocks[registered_errors.count++] = new_block;
|
||||
return UDS_SUCCESS;
|
||||
}
|
|
@ -0,0 +1,73 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef UDS_ERRORS_H
|
||||
#define UDS_ERRORS_H
|
||||
|
||||
#include <linux/compiler.h>
|
||||
#include <linux/types.h>
|
||||
|
||||
/* Custom error codes and error-related utilities */
|
||||
#define VDO_SUCCESS 0
|
||||
|
||||
/* Valid status codes for internal UDS functions. */
|
||||
enum uds_status_codes {
|
||||
/* Successful return */
|
||||
UDS_SUCCESS = VDO_SUCCESS,
|
||||
/* Used as a base value for reporting internal errors */
|
||||
UDS_ERROR_CODE_BASE = 1024,
|
||||
/* Index overflow */
|
||||
UDS_OVERFLOW = UDS_ERROR_CODE_BASE,
|
||||
/* Invalid argument passed to internal routine */
|
||||
UDS_INVALID_ARGUMENT,
|
||||
/* UDS data structures are in an invalid state */
|
||||
UDS_BAD_STATE,
|
||||
/* Attempt to enter the same name into an internal structure twice */
|
||||
UDS_DUPLICATE_NAME,
|
||||
/* An assertion failed */
|
||||
UDS_ASSERTION_FAILED,
|
||||
/* A request has been queued for later processing (not an error) */
|
||||
UDS_QUEUED,
|
||||
/* This error range has already been registered */
|
||||
UDS_ALREADY_REGISTERED,
|
||||
/* Attempt to read or write data outside the valid range */
|
||||
UDS_OUT_OF_RANGE,
|
||||
/* The index session is disabled */
|
||||
UDS_DISABLED,
|
||||
/* The index configuration or volume format is no longer supported */
|
||||
UDS_UNSUPPORTED_VERSION,
|
||||
/* Some index structure is corrupt */
|
||||
UDS_CORRUPT_DATA,
|
||||
/* No index state found */
|
||||
UDS_NO_INDEX,
|
||||
/* Attempt to access incomplete index save data */
|
||||
UDS_INDEX_NOT_SAVED_CLEANLY,
|
||||
/* One more than the last UDS_INTERNAL error code */
|
||||
UDS_ERROR_CODE_LAST,
|
||||
/* One more than the last error this block will ever use */
|
||||
UDS_ERROR_CODE_BLOCK_END = UDS_ERROR_CODE_BASE + 440,
|
||||
};
|
||||
|
||||
enum {
|
||||
VDO_MAX_ERROR_NAME_SIZE = 80,
|
||||
VDO_MAX_ERROR_MESSAGE_SIZE = 128,
|
||||
};
|
||||
|
||||
struct error_info {
|
||||
const char *name;
|
||||
const char *message;
|
||||
};
|
||||
|
||||
const char * __must_check uds_string_error(int errnum, char *buf, size_t buflen);
|
||||
|
||||
const char *uds_string_error_name(int errnum, char *buf, size_t buflen);
|
||||
|
||||
int uds_status_to_errno(int error);
|
||||
|
||||
int uds_register_error_block(const char *block_name, int first_error,
|
||||
int last_reserved_error, const struct error_info *infos,
|
||||
size_t info_size);
|
||||
|
||||
#endif /* UDS_ERRORS_H */
|
|
@ -0,0 +1,560 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "flush.h"
|
||||
|
||||
#include <linux/mempool.h>
|
||||
#include <linux/spinlock.h>
|
||||
|
||||
#include "logger.h"
|
||||
#include "memory-alloc.h"
|
||||
#include "permassert.h"
|
||||
|
||||
#include "admin-state.h"
|
||||
#include "completion.h"
|
||||
#include "io-submitter.h"
|
||||
#include "logical-zone.h"
|
||||
#include "slab-depot.h"
|
||||
#include "types.h"
|
||||
#include "vdo.h"
|
||||
|
||||
struct flusher {
|
||||
struct vdo_completion completion;
|
||||
/* The vdo to which this flusher belongs */
|
||||
struct vdo *vdo;
|
||||
/* The administrative state of the flusher */
|
||||
struct admin_state state;
|
||||
/* The current flush generation of the vdo */
|
||||
sequence_number_t flush_generation;
|
||||
/* The first unacknowledged flush generation */
|
||||
sequence_number_t first_unacknowledged_generation;
|
||||
/* The queue of flush requests waiting to notify other threads */
|
||||
struct vdo_wait_queue notifiers;
|
||||
/* The queue of flush requests waiting for VIOs to complete */
|
||||
struct vdo_wait_queue pending_flushes;
|
||||
/* The flush generation for which notifications are being sent */
|
||||
sequence_number_t notify_generation;
|
||||
/* The logical zone to notify next */
|
||||
struct logical_zone *logical_zone_to_notify;
|
||||
/* The ID of the thread on which flush requests should be made */
|
||||
thread_id_t thread_id;
|
||||
/* The pool of flush requests */
|
||||
mempool_t *flush_pool;
|
||||
/* Bios waiting for a flush request to become available */
|
||||
struct bio_list waiting_flush_bios;
|
||||
/* The lock to protect the previous fields */
|
||||
spinlock_t lock;
|
||||
/* The rotor for selecting the bio queue for submitting flush bios */
|
||||
zone_count_t bio_queue_rotor;
|
||||
/* The number of flushes submitted to the current bio queue */
|
||||
int flush_count;
|
||||
};
|
||||
|
||||
/**
|
||||
* assert_on_flusher_thread() - Check that we are on the flusher thread.
|
||||
* @flusher: The flusher.
|
||||
* @caller: The function which is asserting.
|
||||
*/
|
||||
static inline void assert_on_flusher_thread(struct flusher *flusher, const char *caller)
|
||||
{
|
||||
VDO_ASSERT_LOG_ONLY((vdo_get_callback_thread_id() == flusher->thread_id),
|
||||
"%s() called from flusher thread", caller);
|
||||
}
|
||||
|
||||
/**
|
||||
* as_flusher() - Convert a generic vdo_completion to a flusher.
|
||||
* @completion: The completion to convert.
|
||||
*
|
||||
* Return: The completion as a flusher.
|
||||
*/
|
||||
static struct flusher *as_flusher(struct vdo_completion *completion)
|
||||
{
|
||||
vdo_assert_completion_type(completion, VDO_FLUSH_NOTIFICATION_COMPLETION);
|
||||
return container_of(completion, struct flusher, completion);
|
||||
}
|
||||
|
||||
/**
|
||||
* completion_as_vdo_flush() - Convert a generic vdo_completion to a vdo_flush.
|
||||
* @completion: The completion to convert.
|
||||
*
|
||||
* Return: The completion as a vdo_flush.
|
||||
*/
|
||||
static inline struct vdo_flush *completion_as_vdo_flush(struct vdo_completion *completion)
|
||||
{
|
||||
vdo_assert_completion_type(completion, VDO_FLUSH_COMPLETION);
|
||||
return container_of(completion, struct vdo_flush, completion);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_waiter_as_flush() - Convert a vdo_flush's generic wait queue entry back to the vdo_flush.
|
||||
* @waiter: The wait queue entry to convert.
|
||||
*
|
||||
* Return: The wait queue entry as a vdo_flush.
|
||||
*/
|
||||
static struct vdo_flush *vdo_waiter_as_flush(struct vdo_waiter *waiter)
|
||||
{
|
||||
return container_of(waiter, struct vdo_flush, waiter);
|
||||
}
|
||||
|
||||
static void *allocate_flush(gfp_t gfp_mask, void *pool_data)
|
||||
{
|
||||
struct vdo_flush *flush = NULL;
|
||||
|
||||
if ((gfp_mask & GFP_NOWAIT) == GFP_NOWAIT) {
|
||||
flush = vdo_allocate_memory_nowait(sizeof(struct vdo_flush), __func__);
|
||||
} else {
|
||||
int result = vdo_allocate(1, struct vdo_flush, __func__, &flush);
|
||||
|
||||
if (result != VDO_SUCCESS)
|
||||
vdo_log_error_strerror(result, "failed to allocate spare flush");
|
||||
}
|
||||
|
||||
if (flush != NULL) {
|
||||
struct flusher *flusher = pool_data;
|
||||
|
||||
vdo_initialize_completion(&flush->completion, flusher->vdo,
|
||||
VDO_FLUSH_COMPLETION);
|
||||
}
|
||||
|
||||
return flush;
|
||||
}
|
||||
|
||||
static void free_flush(void *element, void *pool_data __always_unused)
|
||||
{
|
||||
vdo_free(element);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_make_flusher() - Make a flusher for a vdo.
|
||||
* @vdo: The vdo which owns the flusher.
|
||||
*
|
||||
* Return: VDO_SUCCESS or an error.
|
||||
*/
|
||||
int vdo_make_flusher(struct vdo *vdo)
|
||||
{
|
||||
int result = vdo_allocate(1, struct flusher, __func__, &vdo->flusher);
|
||||
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
vdo->flusher->vdo = vdo;
|
||||
vdo->flusher->thread_id = vdo->thread_config.packer_thread;
|
||||
vdo_set_admin_state_code(&vdo->flusher->state, VDO_ADMIN_STATE_NORMAL_OPERATION);
|
||||
vdo_initialize_completion(&vdo->flusher->completion, vdo,
|
||||
VDO_FLUSH_NOTIFICATION_COMPLETION);
|
||||
|
||||
spin_lock_init(&vdo->flusher->lock);
|
||||
bio_list_init(&vdo->flusher->waiting_flush_bios);
|
||||
vdo->flusher->flush_pool = mempool_create(1, allocate_flush, free_flush,
|
||||
vdo->flusher);
|
||||
return ((vdo->flusher->flush_pool == NULL) ? -ENOMEM : VDO_SUCCESS);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_free_flusher() - Free a flusher.
|
||||
* @flusher: The flusher to free.
|
||||
*/
|
||||
void vdo_free_flusher(struct flusher *flusher)
|
||||
{
|
||||
if (flusher == NULL)
|
||||
return;
|
||||
|
||||
if (flusher->flush_pool != NULL)
|
||||
mempool_destroy(vdo_forget(flusher->flush_pool));
|
||||
vdo_free(flusher);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_get_flusher_thread_id() - Get the ID of the thread on which flusher functions should be
|
||||
* called.
|
||||
* @flusher: The flusher to query.
|
||||
*
|
||||
* Return: The ID of the thread which handles the flusher.
|
||||
*/
|
||||
thread_id_t vdo_get_flusher_thread_id(struct flusher *flusher)
|
||||
{
|
||||
return flusher->thread_id;
|
||||
}
|
||||
|
||||
static void notify_flush(struct flusher *flusher);
|
||||
static void vdo_complete_flush(struct vdo_flush *flush);
|
||||
|
||||
/**
|
||||
* finish_notification() - Finish the notification process.
|
||||
* @completion: The flusher completion.
|
||||
*
|
||||
* Finishes the notification process by checking if any flushes have completed and then starting
|
||||
* the notification of the next flush request if one came in while the current notification was in
|
||||
* progress. This callback is registered in flush_packer_callback().
|
||||
*/
|
||||
static void finish_notification(struct vdo_completion *completion)
|
||||
{
|
||||
struct flusher *flusher = as_flusher(completion);
|
||||
|
||||
assert_on_flusher_thread(flusher, __func__);
|
||||
|
||||
vdo_waitq_enqueue_waiter(&flusher->pending_flushes,
|
||||
vdo_waitq_dequeue_waiter(&flusher->notifiers));
|
||||
vdo_complete_flushes(flusher);
|
||||
if (vdo_waitq_has_waiters(&flusher->notifiers))
|
||||
notify_flush(flusher);
|
||||
}
|
||||
|
||||
/**
|
||||
* flush_packer_callback() - Flush the packer.
|
||||
* @completion: The flusher completion.
|
||||
*
|
||||
* Flushes the packer now that all of the logical and physical zones have been notified of the new
|
||||
* flush request. This callback is registered in increment_generation().
|
||||
*/
|
||||
static void flush_packer_callback(struct vdo_completion *completion)
|
||||
{
|
||||
struct flusher *flusher = as_flusher(completion);
|
||||
|
||||
vdo_increment_packer_flush_generation(flusher->vdo->packer);
|
||||
vdo_launch_completion_callback(completion, finish_notification,
|
||||
flusher->thread_id);
|
||||
}
|
||||
|
||||
/**
|
||||
* increment_generation() - Increment the flush generation in a logical zone.
|
||||
* @completion: The flusher as a completion.
|
||||
*
|
||||
* If there are more logical zones, go on to the next one, otherwise, prepare the physical zones.
|
||||
* This callback is registered both in notify_flush() and in itself.
|
||||
*/
|
||||
static void increment_generation(struct vdo_completion *completion)
|
||||
{
|
||||
struct flusher *flusher = as_flusher(completion);
|
||||
struct logical_zone *zone = flusher->logical_zone_to_notify;
|
||||
|
||||
vdo_increment_logical_zone_flush_generation(zone, flusher->notify_generation);
|
||||
if (zone->next == NULL) {
|
||||
vdo_launch_completion_callback(completion, flush_packer_callback,
|
||||
flusher->thread_id);
|
||||
return;
|
||||
}
|
||||
|
||||
flusher->logical_zone_to_notify = zone->next;
|
||||
vdo_launch_completion_callback(completion, increment_generation,
|
||||
flusher->logical_zone_to_notify->thread_id);
|
||||
}
|
||||
|
||||
/**
|
||||
* notify_flush() - Launch a flush notification.
|
||||
* @flusher: The flusher doing the notification.
|
||||
*/
|
||||
static void notify_flush(struct flusher *flusher)
|
||||
{
|
||||
struct vdo_flush *flush =
|
||||
vdo_waiter_as_flush(vdo_waitq_get_first_waiter(&flusher->notifiers));
|
||||
|
||||
flusher->notify_generation = flush->flush_generation;
|
||||
flusher->logical_zone_to_notify = &flusher->vdo->logical_zones->zones[0];
|
||||
flusher->completion.requeue = true;
|
||||
vdo_launch_completion_callback(&flusher->completion, increment_generation,
|
||||
flusher->logical_zone_to_notify->thread_id);
|
||||
}
|
||||
|
||||
/**
|
||||
* flush_vdo() - Start processing a flush request.
|
||||
* @completion: A flush request (as a vdo_completion)
|
||||
*
|
||||
* This callback is registered in launch_flush().
|
||||
*/
|
||||
static void flush_vdo(struct vdo_completion *completion)
|
||||
{
|
||||
struct vdo_flush *flush = completion_as_vdo_flush(completion);
|
||||
struct flusher *flusher = completion->vdo->flusher;
|
||||
bool may_notify;
|
||||
int result;
|
||||
|
||||
assert_on_flusher_thread(flusher, __func__);
|
||||
result = VDO_ASSERT(vdo_is_state_normal(&flusher->state),
|
||||
"flusher is in normal operation");
|
||||
if (result != VDO_SUCCESS) {
|
||||
vdo_enter_read_only_mode(flusher->vdo, result);
|
||||
vdo_complete_flush(flush);
|
||||
return;
|
||||
}
|
||||
|
||||
flush->flush_generation = flusher->flush_generation++;
|
||||
may_notify = !vdo_waitq_has_waiters(&flusher->notifiers);
|
||||
vdo_waitq_enqueue_waiter(&flusher->notifiers, &flush->waiter);
|
||||
if (may_notify)
|
||||
notify_flush(flusher);
|
||||
}
|
||||
|
||||
/**
|
||||
* check_for_drain_complete() - Check whether the flusher has drained.
|
||||
* @flusher: The flusher.
|
||||
*/
|
||||
static void check_for_drain_complete(struct flusher *flusher)
|
||||
{
|
||||
bool drained;
|
||||
|
||||
if (!vdo_is_state_draining(&flusher->state) ||
|
||||
vdo_waitq_has_waiters(&flusher->pending_flushes))
|
||||
return;
|
||||
|
||||
spin_lock(&flusher->lock);
|
||||
drained = bio_list_empty(&flusher->waiting_flush_bios);
|
||||
spin_unlock(&flusher->lock);
|
||||
|
||||
if (drained)
|
||||
vdo_finish_draining(&flusher->state);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_complete_flushes() - Attempt to complete any flushes which might have finished.
|
||||
* @flusher: The flusher.
|
||||
*/
|
||||
void vdo_complete_flushes(struct flusher *flusher)
|
||||
{
|
||||
sequence_number_t oldest_active_generation = U64_MAX;
|
||||
struct logical_zone *zone;
|
||||
|
||||
assert_on_flusher_thread(flusher, __func__);
|
||||
|
||||
for (zone = &flusher->vdo->logical_zones->zones[0]; zone != NULL; zone = zone->next)
|
||||
oldest_active_generation =
|
||||
min(oldest_active_generation,
|
||||
READ_ONCE(zone->oldest_active_generation));
|
||||
|
||||
while (vdo_waitq_has_waiters(&flusher->pending_flushes)) {
|
||||
struct vdo_flush *flush =
|
||||
vdo_waiter_as_flush(vdo_waitq_get_first_waiter(&flusher->pending_flushes));
|
||||
|
||||
if (flush->flush_generation >= oldest_active_generation)
|
||||
return;
|
||||
|
||||
VDO_ASSERT_LOG_ONLY((flush->flush_generation ==
|
||||
flusher->first_unacknowledged_generation),
|
||||
"acknowledged next expected flush, %llu, was: %llu",
|
||||
(unsigned long long) flusher->first_unacknowledged_generation,
|
||||
(unsigned long long) flush->flush_generation);
|
||||
vdo_waitq_dequeue_waiter(&flusher->pending_flushes);
|
||||
vdo_complete_flush(flush);
|
||||
flusher->first_unacknowledged_generation++;
|
||||
}
|
||||
|
||||
check_for_drain_complete(flusher);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_dump_flusher() - Dump the flusher, in a thread-unsafe fashion.
|
||||
* @flusher: The flusher.
|
||||
*/
|
||||
void vdo_dump_flusher(const struct flusher *flusher)
|
||||
{
|
||||
vdo_log_info("struct flusher");
|
||||
vdo_log_info(" flush_generation=%llu first_unacknowledged_generation=%llu",
|
||||
(unsigned long long) flusher->flush_generation,
|
||||
(unsigned long long) flusher->first_unacknowledged_generation);
|
||||
vdo_log_info(" notifiers queue is %s; pending_flushes queue is %s",
|
||||
(vdo_waitq_has_waiters(&flusher->notifiers) ? "not empty" : "empty"),
|
||||
(vdo_waitq_has_waiters(&flusher->pending_flushes) ? "not empty" : "empty"));
|
||||
}
|
||||
|
||||
/**
|
||||
* initialize_flush() - Initialize a vdo_flush structure.
|
||||
* @flush: The flush to initialize.
|
||||
* @vdo: The vdo being flushed.
|
||||
*
|
||||
* Initializes a vdo_flush structure, transferring all the bios in the flusher's waiting_flush_bios
|
||||
* list to it. The caller MUST already hold the lock.
|
||||
*/
|
||||
static void initialize_flush(struct vdo_flush *flush, struct vdo *vdo)
|
||||
{
|
||||
bio_list_init(&flush->bios);
|
||||
bio_list_merge(&flush->bios, &vdo->flusher->waiting_flush_bios);
|
||||
bio_list_init(&vdo->flusher->waiting_flush_bios);
|
||||
}
|
||||
|
||||
static void launch_flush(struct vdo_flush *flush)
|
||||
{
|
||||
struct vdo_completion *completion = &flush->completion;
|
||||
|
||||
vdo_prepare_completion(completion, flush_vdo, flush_vdo,
|
||||
completion->vdo->thread_config.packer_thread, NULL);
|
||||
vdo_enqueue_completion(completion, VDO_DEFAULT_Q_FLUSH_PRIORITY);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_launch_flush() - Function called to start processing a flush request.
|
||||
* @vdo: The vdo.
|
||||
* @bio: The bio containing an empty flush request.
|
||||
*
|
||||
* This is called when we receive an empty flush bio from the block layer, and before acknowledging
|
||||
* a non-empty bio with the FUA flag set.
|
||||
*/
|
||||
void vdo_launch_flush(struct vdo *vdo, struct bio *bio)
|
||||
{
|
||||
/*
|
||||
* Try to allocate a vdo_flush to represent the flush request. If the allocation fails,
|
||||
* we'll deal with it later.
|
||||
*/
|
||||
struct vdo_flush *flush = mempool_alloc(vdo->flusher->flush_pool, GFP_NOWAIT);
|
||||
struct flusher *flusher = vdo->flusher;
|
||||
const struct admin_state_code *code = vdo_get_admin_state_code(&flusher->state);
|
||||
|
||||
VDO_ASSERT_LOG_ONLY(!code->quiescent, "Flushing not allowed in state %s",
|
||||
code->name);
|
||||
|
||||
spin_lock(&flusher->lock);
|
||||
|
||||
/* We have a new bio to start. Add it to the list. */
|
||||
bio_list_add(&flusher->waiting_flush_bios, bio);
|
||||
|
||||
if (flush == NULL) {
|
||||
spin_unlock(&flusher->lock);
|
||||
return;
|
||||
}
|
||||
|
||||
/* We have flushes to start. Capture them in the vdo_flush structure. */
|
||||
initialize_flush(flush, vdo);
|
||||
spin_unlock(&flusher->lock);
|
||||
|
||||
/* Finish launching the flushes. */
|
||||
launch_flush(flush);
|
||||
}
|
||||
|
||||
/**
|
||||
* release_flush() - Release a vdo_flush structure that has completed its work.
|
||||
* @flush: The completed flush structure to re-use or free.
|
||||
*
|
||||
* If there are any pending flush requests whose vdo_flush allocation failed, they will be launched
|
||||
* by immediately re-using the released vdo_flush. If there is no spare vdo_flush, the released
|
||||
* structure will become the spare. Otherwise, the vdo_flush will be freed.
|
||||
*/
|
||||
static void release_flush(struct vdo_flush *flush)
|
||||
{
|
||||
bool relaunch_flush;
|
||||
struct flusher *flusher = flush->completion.vdo->flusher;
|
||||
|
||||
spin_lock(&flusher->lock);
|
||||
if (bio_list_empty(&flusher->waiting_flush_bios)) {
|
||||
relaunch_flush = false;
|
||||
} else {
|
||||
/* We have flushes to start. Capture them in a flush request. */
|
||||
initialize_flush(flush, flusher->vdo);
|
||||
relaunch_flush = true;
|
||||
}
|
||||
spin_unlock(&flusher->lock);
|
||||
|
||||
if (relaunch_flush) {
|
||||
/* Finish launching the flushes. */
|
||||
launch_flush(flush);
|
||||
return;
|
||||
}
|
||||
|
||||
mempool_free(flush, flusher->flush_pool);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_complete_flush_callback() - Function called to complete and free a flush request, registered
|
||||
* in vdo_complete_flush().
|
||||
* @completion: The flush request.
|
||||
*/
|
||||
static void vdo_complete_flush_callback(struct vdo_completion *completion)
|
||||
{
|
||||
struct vdo_flush *flush = completion_as_vdo_flush(completion);
|
||||
struct vdo *vdo = completion->vdo;
|
||||
struct bio *bio;
|
||||
|
||||
while ((bio = bio_list_pop(&flush->bios)) != NULL) {
|
||||
/*
|
||||
* We're not acknowledging this bio now, but we'll never touch it again, so this is
|
||||
* the last chance to account for it.
|
||||
*/
|
||||
vdo_count_bios(&vdo->stats.bios_acknowledged, bio);
|
||||
|
||||
/* Update the device, and send it on down... */
|
||||
bio_set_dev(bio, vdo_get_backing_device(vdo));
|
||||
atomic64_inc(&vdo->stats.flush_out);
|
||||
submit_bio_noacct(bio);
|
||||
}
|
||||
|
||||
|
||||
/*
|
||||
* Release the flush structure, freeing it, re-using it as the spare, or using it to launch
|
||||
* any flushes that had to wait when allocations failed.
|
||||
*/
|
||||
release_flush(flush);
|
||||
}
|
||||
|
||||
/**
|
||||
* select_bio_queue() - Select the bio queue on which to finish a flush request.
|
||||
* @flusher: The flusher finishing the request.
|
||||
*/
|
||||
static thread_id_t select_bio_queue(struct flusher *flusher)
|
||||
{
|
||||
struct vdo *vdo = flusher->vdo;
|
||||
zone_count_t bio_threads = flusher->vdo->thread_config.bio_thread_count;
|
||||
int interval;
|
||||
|
||||
if (bio_threads == 1)
|
||||
return vdo->thread_config.bio_threads[0];
|
||||
|
||||
interval = vdo->device_config->thread_counts.bio_rotation_interval;
|
||||
if (flusher->flush_count == interval) {
|
||||
flusher->flush_count = 1;
|
||||
flusher->bio_queue_rotor = ((flusher->bio_queue_rotor + 1) % bio_threads);
|
||||
} else {
|
||||
flusher->flush_count++;
|
||||
}
|
||||
|
||||
return vdo->thread_config.bio_threads[flusher->bio_queue_rotor];
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_complete_flush() - Complete and free a vdo flush request.
|
||||
* @flush: The flush request.
|
||||
*/
|
||||
static void vdo_complete_flush(struct vdo_flush *flush)
|
||||
{
|
||||
struct vdo_completion *completion = &flush->completion;
|
||||
|
||||
vdo_prepare_completion(completion, vdo_complete_flush_callback,
|
||||
vdo_complete_flush_callback,
|
||||
select_bio_queue(completion->vdo->flusher), NULL);
|
||||
vdo_enqueue_completion(completion, BIO_Q_FLUSH_PRIORITY);
|
||||
}
|
||||
|
||||
/**
|
||||
* initiate_drain() - Initiate a drain.
|
||||
*
|
||||
* Implements vdo_admin_initiator_fn.
|
||||
*/
|
||||
static void initiate_drain(struct admin_state *state)
|
||||
{
|
||||
check_for_drain_complete(container_of(state, struct flusher, state));
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_drain_flusher() - Drain the flusher.
|
||||
* @flusher: The flusher to drain.
|
||||
* @completion: The completion to finish when the flusher has drained.
|
||||
*
|
||||
* Drains the flusher by preventing any more VIOs from entering the flusher and then flushing. The
|
||||
* flusher will be left in the suspended state.
|
||||
*/
|
||||
void vdo_drain_flusher(struct flusher *flusher, struct vdo_completion *completion)
|
||||
{
|
||||
assert_on_flusher_thread(flusher, __func__);
|
||||
vdo_start_draining(&flusher->state, VDO_ADMIN_STATE_SUSPENDING, completion,
|
||||
initiate_drain);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_resume_flusher() - Resume a flusher which has been suspended.
|
||||
* @flusher: The flusher to resume.
|
||||
* @parent: The completion to finish when the flusher has resumed.
|
||||
*/
|
||||
void vdo_resume_flusher(struct flusher *flusher, struct vdo_completion *parent)
|
||||
{
|
||||
assert_on_flusher_thread(flusher, __func__);
|
||||
vdo_continue_completion(parent, vdo_resume_if_quiescent(&flusher->state));
|
||||
}
|
|
@ -0,0 +1,44 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef VDO_FLUSH_H
|
||||
#define VDO_FLUSH_H
|
||||
|
||||
#include "funnel-workqueue.h"
|
||||
#include "types.h"
|
||||
#include "vio.h"
|
||||
#include "wait-queue.h"
|
||||
|
||||
/* A marker for tracking which journal entries are affected by a flush request. */
|
||||
struct vdo_flush {
|
||||
/* The completion for enqueueing this flush request. */
|
||||
struct vdo_completion completion;
|
||||
/* The flush bios covered by this request */
|
||||
struct bio_list bios;
|
||||
/* The wait queue entry for this flush */
|
||||
struct vdo_waiter waiter;
|
||||
/* Which flush this struct represents */
|
||||
sequence_number_t flush_generation;
|
||||
};
|
||||
|
||||
struct flusher;
|
||||
|
||||
int __must_check vdo_make_flusher(struct vdo *vdo);
|
||||
|
||||
void vdo_free_flusher(struct flusher *flusher);
|
||||
|
||||
thread_id_t __must_check vdo_get_flusher_thread_id(struct flusher *flusher);
|
||||
|
||||
void vdo_complete_flushes(struct flusher *flusher);
|
||||
|
||||
void vdo_dump_flusher(const struct flusher *flusher);
|
||||
|
||||
void vdo_launch_flush(struct vdo *vdo, struct bio *bio);
|
||||
|
||||
void vdo_drain_flusher(struct flusher *flusher, struct vdo_completion *completion);
|
||||
|
||||
void vdo_resume_flusher(struct flusher *flusher, struct vdo_completion *parent);
|
||||
|
||||
#endif /* VDO_FLUSH_H */
|
|
@ -0,0 +1,170 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "funnel-queue.h"
|
||||
|
||||
#include "cpu.h"
|
||||
#include "memory-alloc.h"
|
||||
#include "permassert.h"
|
||||
|
||||
int vdo_make_funnel_queue(struct funnel_queue **queue_ptr)
|
||||
{
|
||||
int result;
|
||||
struct funnel_queue *queue;
|
||||
|
||||
result = vdo_allocate(1, struct funnel_queue, "funnel queue", &queue);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
/*
|
||||
* Initialize the stub entry and put it in the queue, establishing the invariant that
|
||||
* queue->newest and queue->oldest are never null.
|
||||
*/
|
||||
queue->stub.next = NULL;
|
||||
queue->newest = &queue->stub;
|
||||
queue->oldest = &queue->stub;
|
||||
|
||||
*queue_ptr = queue;
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
void vdo_free_funnel_queue(struct funnel_queue *queue)
|
||||
{
|
||||
vdo_free(queue);
|
||||
}
|
||||
|
||||
static struct funnel_queue_entry *get_oldest(struct funnel_queue *queue)
|
||||
{
|
||||
/*
|
||||
* Barrier requirements: We need a read barrier between reading a "next" field pointer
|
||||
* value and reading anything it points to. There's an accompanying barrier in
|
||||
* vdo_funnel_queue_put() between its caller setting up the entry and making it visible.
|
||||
*/
|
||||
struct funnel_queue_entry *oldest = queue->oldest;
|
||||
struct funnel_queue_entry *next = READ_ONCE(oldest->next);
|
||||
|
||||
if (oldest == &queue->stub) {
|
||||
/*
|
||||
* When the oldest entry is the stub and it has no successor, the queue is
|
||||
* logically empty.
|
||||
*/
|
||||
if (next == NULL)
|
||||
return NULL;
|
||||
/*
|
||||
* The stub entry has a successor, so the stub can be dequeued and ignored without
|
||||
* breaking the queue invariants.
|
||||
*/
|
||||
oldest = next;
|
||||
queue->oldest = oldest;
|
||||
next = READ_ONCE(oldest->next);
|
||||
}
|
||||
|
||||
/*
|
||||
* We have a non-stub candidate to dequeue. If it lacks a successor, we'll need to put the
|
||||
* stub entry back on the queue first.
|
||||
*/
|
||||
if (next == NULL) {
|
||||
struct funnel_queue_entry *newest = READ_ONCE(queue->newest);
|
||||
|
||||
if (oldest != newest) {
|
||||
/*
|
||||
* Another thread has already swung queue->newest atomically, but not yet
|
||||
* assigned previous->next. The queue is really still empty.
|
||||
*/
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/*
|
||||
* Put the stub entry back on the queue, ensuring a successor will eventually be
|
||||
* seen.
|
||||
*/
|
||||
vdo_funnel_queue_put(queue, &queue->stub);
|
||||
|
||||
/* Check again for a successor. */
|
||||
next = READ_ONCE(oldest->next);
|
||||
if (next == NULL) {
|
||||
/*
|
||||
* We lost a race with a producer who swapped queue->newest before we did,
|
||||
* but who hasn't yet updated previous->next. Try again later.
|
||||
*/
|
||||
return NULL;
|
||||
}
|
||||
}
|
||||
|
||||
return oldest;
|
||||
}
|
||||
|
||||
/*
|
||||
* Poll a queue, removing the oldest entry if the queue is not empty. This function must only be
|
||||
* called from a single consumer thread.
|
||||
*/
|
||||
struct funnel_queue_entry *vdo_funnel_queue_poll(struct funnel_queue *queue)
|
||||
{
|
||||
struct funnel_queue_entry *oldest = get_oldest(queue);
|
||||
|
||||
if (oldest == NULL)
|
||||
return oldest;
|
||||
|
||||
/*
|
||||
* Dequeue the oldest entry and return it. Only one consumer thread may call this function,
|
||||
* so no locking, atomic operations, or fences are needed; queue->oldest is owned by the
|
||||
* consumer and oldest->next is never used by a producer thread after it is swung from NULL
|
||||
* to non-NULL.
|
||||
*/
|
||||
queue->oldest = READ_ONCE(oldest->next);
|
||||
/*
|
||||
* Make sure the caller sees the proper stored data for this entry. Since we've already
|
||||
* fetched the entry pointer we stored in "queue->oldest", this also ensures that on entry
|
||||
* to the next call we'll properly see the dependent data.
|
||||
*/
|
||||
smp_rmb();
|
||||
/*
|
||||
* If "oldest" is a very light-weight work item, we'll be looking for the next one very
|
||||
* soon, so prefetch it now.
|
||||
*/
|
||||
uds_prefetch_address(queue->oldest, true);
|
||||
WRITE_ONCE(oldest->next, NULL);
|
||||
return oldest;
|
||||
}
|
||||
|
||||
/*
|
||||
* Check whether the funnel queue is empty or not. If the queue is in a transition state with one
|
||||
* or more entries being added such that the list view is incomplete, this function will report the
|
||||
* queue as empty.
|
||||
*/
|
||||
bool vdo_is_funnel_queue_empty(struct funnel_queue *queue)
|
||||
{
|
||||
return get_oldest(queue) == NULL;
|
||||
}
|
||||
|
||||
/*
|
||||
* Check whether the funnel queue is idle or not. If the queue has entries available to be
|
||||
* retrieved, it is not idle. If the queue is in a transition state with one or more entries being
|
||||
* added such that the list view is incomplete, it may not be possible to retrieve an entry with
|
||||
* the vdo_funnel_queue_poll() function, but the queue will not be considered idle.
|
||||
*/
|
||||
bool vdo_is_funnel_queue_idle(struct funnel_queue *queue)
|
||||
{
|
||||
/*
|
||||
* Oldest is not the stub, so there's another entry, though if next is NULL we can't
|
||||
* retrieve it yet.
|
||||
*/
|
||||
if (queue->oldest != &queue->stub)
|
||||
return false;
|
||||
|
||||
/*
|
||||
* Oldest is the stub, but newest has been updated by _put(); either there's another,
|
||||
* retrievable entry in the list, or the list is officially empty but in the intermediate
|
||||
* state of having an entry added.
|
||||
*
|
||||
* Whether anything is retrievable depends on whether stub.next has been updated and become
|
||||
* visible to us, but for idleness we don't care. And due to memory ordering in _put(), the
|
||||
* update to newest would be visible to us at the same time or sooner.
|
||||
*/
|
||||
if (READ_ONCE(queue->newest) != &queue->stub)
|
||||
return false;
|
||||
|
||||
return true;
|
||||
}
|
|
@ -0,0 +1,110 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef VDO_FUNNEL_QUEUE_H
|
||||
#define VDO_FUNNEL_QUEUE_H
|
||||
|
||||
#include <linux/atomic.h>
|
||||
#include <linux/cache.h>
|
||||
|
||||
/*
|
||||
* A funnel queue is a simple (almost) lock-free queue that accepts entries from multiple threads
|
||||
* (multi-producer) and delivers them to a single thread (single-consumer). "Funnel" is an attempt
|
||||
* to evoke the image of requests from more than one producer being "funneled down" to a single
|
||||
* consumer.
|
||||
*
|
||||
* This is an unsynchronized but thread-safe data structure when used as intended. There is no
|
||||
* mechanism to ensure that only one thread is consuming from the queue. If more than one thread
|
||||
* attempts to consume from the queue, the resulting behavior is undefined. Clients must not
|
||||
* directly access or manipulate the internals of the queue, which are only exposed for the purpose
|
||||
* of allowing the very simple enqueue operation to be inlined.
|
||||
*
|
||||
* The implementation requires that a funnel_queue_entry structure (a link pointer) is embedded in
|
||||
* the queue entries, and pointers to those structures are used exclusively by the queue. No macros
|
||||
* are defined to template the queue, so the offset of the funnel_queue_entry in the records placed
|
||||
* in the queue must all be the same so the client can derive their structure pointer from the
|
||||
* entry pointer returned by vdo_funnel_queue_poll().
|
||||
*
|
||||
* Callers are wholly responsible for allocating and freeing the entries. Entries may be freed as
|
||||
* soon as they are returned since this queue is not susceptible to the "ABA problem" present in
|
||||
* many lock-free data structures. The queue is dynamically allocated to ensure cache-line
|
||||
* alignment, but no other dynamic allocation is used.
|
||||
*
|
||||
* The algorithm is not actually 100% lock-free. There is a single point in vdo_funnel_queue_put()
|
||||
* at which a preempted producer will prevent the consumers from seeing items added to the queue by
|
||||
* later producers, and only if the queue is short enough or the consumer fast enough for it to
|
||||
* reach what was the end of the queue at the time of the preemption.
|
||||
*
|
||||
* The consumer function, vdo_funnel_queue_poll(), will return NULL when the queue is empty. To
|
||||
* wait for data to consume, spin (if safe) or combine the queue with a struct event_count to
|
||||
* signal the presence of new entries.
|
||||
*/
|
||||
|
||||
/* This queue link structure must be embedded in client entries. */
|
||||
struct funnel_queue_entry {
|
||||
/* The next (newer) entry in the queue. */
|
||||
struct funnel_queue_entry *next;
|
||||
};
|
||||
|
||||
/*
|
||||
* The dynamically allocated queue structure, which is allocated on a cache line boundary so the
|
||||
* producer and consumer fields in the structure will land on separate cache lines. This should be
|
||||
* consider opaque but it is exposed here so vdo_funnel_queue_put() can be inlined.
|
||||
*/
|
||||
struct __aligned(L1_CACHE_BYTES) funnel_queue {
|
||||
/*
|
||||
* The producers' end of the queue, an atomically exchanged pointer that will never be
|
||||
* NULL.
|
||||
*/
|
||||
struct funnel_queue_entry *newest;
|
||||
|
||||
/* The consumer's end of the queue, which is owned by the consumer and never NULL. */
|
||||
struct funnel_queue_entry *oldest __aligned(L1_CACHE_BYTES);
|
||||
|
||||
/* A dummy entry used to provide the non-NULL invariants above. */
|
||||
struct funnel_queue_entry stub;
|
||||
};
|
||||
|
||||
int __must_check vdo_make_funnel_queue(struct funnel_queue **queue_ptr);
|
||||
|
||||
void vdo_free_funnel_queue(struct funnel_queue *queue);
|
||||
|
||||
/*
|
||||
* Put an entry on the end of the queue.
|
||||
*
|
||||
* The entry pointer must be to the struct funnel_queue_entry embedded in the caller's data
|
||||
* structure. The caller must be able to derive the address of the start of their data structure
|
||||
* from the pointer that passed in here, so every entry in the queue must have the struct
|
||||
* funnel_queue_entry at the same offset within the client's structure.
|
||||
*/
|
||||
static inline void vdo_funnel_queue_put(struct funnel_queue *queue,
|
||||
struct funnel_queue_entry *entry)
|
||||
{
|
||||
struct funnel_queue_entry *previous;
|
||||
|
||||
/*
|
||||
* Barrier requirements: All stores relating to the entry ("next" pointer, containing data
|
||||
* structure fields) must happen before the previous->next store making it visible to the
|
||||
* consumer. Also, the entry's "next" field initialization to NULL must happen before any
|
||||
* other producer threads can see the entry (the xchg) and try to update the "next" field.
|
||||
*
|
||||
* xchg implements a full barrier.
|
||||
*/
|
||||
WRITE_ONCE(entry->next, NULL);
|
||||
previous = xchg(&queue->newest, entry);
|
||||
/*
|
||||
* Preemptions between these two statements hide the rest of the queue from the consumer,
|
||||
* preventing consumption until the following assignment runs.
|
||||
*/
|
||||
WRITE_ONCE(previous->next, entry);
|
||||
}
|
||||
|
||||
struct funnel_queue_entry *__must_check vdo_funnel_queue_poll(struct funnel_queue *queue);
|
||||
|
||||
bool __must_check vdo_is_funnel_queue_empty(struct funnel_queue *queue);
|
||||
|
||||
bool __must_check vdo_is_funnel_queue_idle(struct funnel_queue *queue);
|
||||
|
||||
#endif /* VDO_FUNNEL_QUEUE_H */
|
|
@ -0,0 +1,638 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "funnel-workqueue.h"
|
||||
|
||||
#include <linux/atomic.h>
|
||||
#include <linux/cache.h>
|
||||
#include <linux/completion.h>
|
||||
#include <linux/err.h>
|
||||
#include <linux/kthread.h>
|
||||
#include <linux/percpu.h>
|
||||
|
||||
#include "funnel-queue.h"
|
||||
#include "logger.h"
|
||||
#include "memory-alloc.h"
|
||||
#include "numeric.h"
|
||||
#include "permassert.h"
|
||||
#include "string-utils.h"
|
||||
|
||||
#include "completion.h"
|
||||
#include "status-codes.h"
|
||||
|
||||
static DEFINE_PER_CPU(unsigned int, service_queue_rotor);
|
||||
|
||||
/**
|
||||
* DOC: Work queue definition.
|
||||
*
|
||||
* There are two types of work queues: simple, with one worker thread, and round-robin, which uses
|
||||
* a group of the former to do the work, and assigns work to them in round-robin fashion (roughly).
|
||||
* Externally, both are represented via the same common sub-structure, though there's actually not
|
||||
* a great deal of overlap between the two types internally.
|
||||
*/
|
||||
struct vdo_work_queue {
|
||||
/* Name of just the work queue (e.g., "cpuQ12") */
|
||||
char *name;
|
||||
bool round_robin_mode;
|
||||
struct vdo_thread *owner;
|
||||
/* Life cycle functions, etc */
|
||||
const struct vdo_work_queue_type *type;
|
||||
};
|
||||
|
||||
struct simple_work_queue {
|
||||
struct vdo_work_queue common;
|
||||
struct funnel_queue *priority_lists[VDO_WORK_Q_MAX_PRIORITY + 1];
|
||||
void *private;
|
||||
|
||||
/*
|
||||
* The fields above are unchanged after setup but often read, and are good candidates for
|
||||
* caching -- and if the max priority is 2, just fit in one x86-64 cache line if aligned.
|
||||
* The fields below are often modified as we sleep and wake, so we want a separate cache
|
||||
* line for performance.
|
||||
*/
|
||||
|
||||
/* Any (0 or 1) worker threads waiting for new work to do */
|
||||
wait_queue_head_t waiting_worker_threads ____cacheline_aligned;
|
||||
/* Hack to reduce wakeup calls if the worker thread is running */
|
||||
atomic_t idle;
|
||||
|
||||
/* These are infrequently used so in terms of performance we don't care where they land. */
|
||||
struct task_struct *thread;
|
||||
/* Notify creator once worker has initialized */
|
||||
struct completion *started;
|
||||
};
|
||||
|
||||
struct round_robin_work_queue {
|
||||
struct vdo_work_queue common;
|
||||
struct simple_work_queue **service_queues;
|
||||
unsigned int num_service_queues;
|
||||
};
|
||||
|
||||
static inline struct simple_work_queue *as_simple_work_queue(struct vdo_work_queue *queue)
|
||||
{
|
||||
return ((queue == NULL) ?
|
||||
NULL : container_of(queue, struct simple_work_queue, common));
|
||||
}
|
||||
|
||||
static inline struct round_robin_work_queue *as_round_robin_work_queue(struct vdo_work_queue *queue)
|
||||
{
|
||||
return ((queue == NULL) ?
|
||||
NULL :
|
||||
container_of(queue, struct round_robin_work_queue, common));
|
||||
}
|
||||
|
||||
/* Processing normal completions. */
|
||||
|
||||
/*
|
||||
* Dequeue and return the next waiting completion, if any.
|
||||
*
|
||||
* We scan the funnel queues from highest priority to lowest, once; there is therefore a race
|
||||
* condition where a high-priority completion can be enqueued followed by a lower-priority one, and
|
||||
* we'll grab the latter (but we'll catch the high-priority item on the next call). If strict
|
||||
* enforcement of priorities becomes necessary, this function will need fixing.
|
||||
*/
|
||||
static struct vdo_completion *poll_for_completion(struct simple_work_queue *queue)
|
||||
{
|
||||
int i;
|
||||
|
||||
for (i = queue->common.type->max_priority; i >= 0; i--) {
|
||||
struct funnel_queue_entry *link = vdo_funnel_queue_poll(queue->priority_lists[i]);
|
||||
|
||||
if (link != NULL)
|
||||
return container_of(link, struct vdo_completion, work_queue_entry_link);
|
||||
}
|
||||
|
||||
return NULL;
|
||||
}
|
||||
|
||||
static void enqueue_work_queue_completion(struct simple_work_queue *queue,
|
||||
struct vdo_completion *completion)
|
||||
{
|
||||
VDO_ASSERT_LOG_ONLY(completion->my_queue == NULL,
|
||||
"completion %px (fn %px) to enqueue (%px) is not already queued (%px)",
|
||||
completion, completion->callback, queue, completion->my_queue);
|
||||
if (completion->priority == VDO_WORK_Q_DEFAULT_PRIORITY)
|
||||
completion->priority = queue->common.type->default_priority;
|
||||
|
||||
if (VDO_ASSERT(completion->priority <= queue->common.type->max_priority,
|
||||
"priority is in range for queue") != VDO_SUCCESS)
|
||||
completion->priority = 0;
|
||||
|
||||
completion->my_queue = &queue->common;
|
||||
|
||||
/* Funnel queue handles the synchronization for the put. */
|
||||
vdo_funnel_queue_put(queue->priority_lists[completion->priority],
|
||||
&completion->work_queue_entry_link);
|
||||
|
||||
/*
|
||||
* Due to how funnel queue synchronization is handled (just atomic operations), the
|
||||
* simplest safe implementation here would be to wake-up any waiting threads after
|
||||
* enqueueing each item. Even if the funnel queue is not empty at the time of adding an
|
||||
* item to the queue, the consumer thread may not see this since it is not guaranteed to
|
||||
* have the same view of the queue as a producer thread.
|
||||
*
|
||||
* However, the above is wasteful so instead we attempt to minimize the number of thread
|
||||
* wakeups. Using an idle flag, and careful ordering using memory barriers, we should be
|
||||
* able to determine when the worker thread might be asleep or going to sleep. We use
|
||||
* cmpxchg to try to take ownership (vs other producer threads) of the responsibility for
|
||||
* waking the worker thread, so multiple wakeups aren't tried at once.
|
||||
*
|
||||
* This was tuned for some x86 boxes that were handy; it's untested whether doing the read
|
||||
* first is any better or worse for other platforms, even other x86 configurations.
|
||||
*/
|
||||
smp_mb();
|
||||
if ((atomic_read(&queue->idle) != 1) || (atomic_cmpxchg(&queue->idle, 1, 0) != 1))
|
||||
return;
|
||||
|
||||
/* There's a maximum of one thread in this list. */
|
||||
wake_up(&queue->waiting_worker_threads);
|
||||
}
|
||||
|
||||
static void run_start_hook(struct simple_work_queue *queue)
|
||||
{
|
||||
if (queue->common.type->start != NULL)
|
||||
queue->common.type->start(queue->private);
|
||||
}
|
||||
|
||||
static void run_finish_hook(struct simple_work_queue *queue)
|
||||
{
|
||||
if (queue->common.type->finish != NULL)
|
||||
queue->common.type->finish(queue->private);
|
||||
}
|
||||
|
||||
/*
|
||||
* Wait for the next completion to process, or until kthread_should_stop indicates that it's time
|
||||
* for us to shut down.
|
||||
*
|
||||
* If kthread_should_stop says it's time to stop but we have pending completions return a
|
||||
* completion.
|
||||
*
|
||||
* Also update statistics relating to scheduler interactions.
|
||||
*/
|
||||
static struct vdo_completion *wait_for_next_completion(struct simple_work_queue *queue)
|
||||
{
|
||||
struct vdo_completion *completion;
|
||||
DEFINE_WAIT(wait);
|
||||
|
||||
while (true) {
|
||||
prepare_to_wait(&queue->waiting_worker_threads, &wait,
|
||||
TASK_INTERRUPTIBLE);
|
||||
/*
|
||||
* Don't set the idle flag until a wakeup will not be lost.
|
||||
*
|
||||
* Force synchronization between setting the idle flag and checking the funnel
|
||||
* queue; the producer side will do them in the reverse order. (There's still a
|
||||
* race condition we've chosen to allow, because we've got a timeout below that
|
||||
* unwedges us if we hit it, but this may narrow the window a little.)
|
||||
*/
|
||||
atomic_set(&queue->idle, 1);
|
||||
smp_mb(); /* store-load barrier between "idle" and funnel queue */
|
||||
|
||||
completion = poll_for_completion(queue);
|
||||
if (completion != NULL)
|
||||
break;
|
||||
|
||||
/*
|
||||
* We need to check for thread-stop after setting TASK_INTERRUPTIBLE state up
|
||||
* above. Otherwise, schedule() will put the thread to sleep and might miss a
|
||||
* wakeup from kthread_stop() call in vdo_finish_work_queue().
|
||||
*/
|
||||
if (kthread_should_stop())
|
||||
break;
|
||||
|
||||
schedule();
|
||||
|
||||
/*
|
||||
* Most of the time when we wake, it should be because there's work to do. If it
|
||||
* was a spurious wakeup, continue looping.
|
||||
*/
|
||||
completion = poll_for_completion(queue);
|
||||
if (completion != NULL)
|
||||
break;
|
||||
}
|
||||
|
||||
finish_wait(&queue->waiting_worker_threads, &wait);
|
||||
atomic_set(&queue->idle, 0);
|
||||
|
||||
return completion;
|
||||
}
|
||||
|
||||
static void process_completion(struct simple_work_queue *queue,
|
||||
struct vdo_completion *completion)
|
||||
{
|
||||
if (VDO_ASSERT(completion->my_queue == &queue->common,
|
||||
"completion %px from queue %px marked as being in this queue (%px)",
|
||||
completion, queue, completion->my_queue) == VDO_SUCCESS)
|
||||
completion->my_queue = NULL;
|
||||
|
||||
vdo_run_completion(completion);
|
||||
}
|
||||
|
||||
static void service_work_queue(struct simple_work_queue *queue)
|
||||
{
|
||||
run_start_hook(queue);
|
||||
|
||||
while (true) {
|
||||
struct vdo_completion *completion = poll_for_completion(queue);
|
||||
|
||||
if (completion == NULL)
|
||||
completion = wait_for_next_completion(queue);
|
||||
|
||||
if (completion == NULL) {
|
||||
/* No completions but kthread_should_stop() was triggered. */
|
||||
break;
|
||||
}
|
||||
|
||||
process_completion(queue, completion);
|
||||
|
||||
/*
|
||||
* Be friendly to a CPU that has other work to do, if the kernel has told us to.
|
||||
* This speeds up some performance tests; that "other work" might include other VDO
|
||||
* threads.
|
||||
*/
|
||||
if (need_resched())
|
||||
cond_resched();
|
||||
}
|
||||
|
||||
run_finish_hook(queue);
|
||||
}
|
||||
|
||||
static int work_queue_runner(void *ptr)
|
||||
{
|
||||
struct simple_work_queue *queue = ptr;
|
||||
|
||||
complete(queue->started);
|
||||
service_work_queue(queue);
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Creation & teardown */
|
||||
|
||||
static void free_simple_work_queue(struct simple_work_queue *queue)
|
||||
{
|
||||
unsigned int i;
|
||||
|
||||
for (i = 0; i <= VDO_WORK_Q_MAX_PRIORITY; i++)
|
||||
vdo_free_funnel_queue(queue->priority_lists[i]);
|
||||
vdo_free(queue->common.name);
|
||||
vdo_free(queue);
|
||||
}
|
||||
|
||||
static void free_round_robin_work_queue(struct round_robin_work_queue *queue)
|
||||
{
|
||||
struct simple_work_queue **queue_table = queue->service_queues;
|
||||
unsigned int count = queue->num_service_queues;
|
||||
unsigned int i;
|
||||
|
||||
queue->service_queues = NULL;
|
||||
|
||||
for (i = 0; i < count; i++)
|
||||
free_simple_work_queue(queue_table[i]);
|
||||
vdo_free(queue_table);
|
||||
vdo_free(queue->common.name);
|
||||
vdo_free(queue);
|
||||
}
|
||||
|
||||
void vdo_free_work_queue(struct vdo_work_queue *queue)
|
||||
{
|
||||
if (queue == NULL)
|
||||
return;
|
||||
|
||||
vdo_finish_work_queue(queue);
|
||||
|
||||
if (queue->round_robin_mode)
|
||||
free_round_robin_work_queue(as_round_robin_work_queue(queue));
|
||||
else
|
||||
free_simple_work_queue(as_simple_work_queue(queue));
|
||||
}
|
||||
|
||||
static int make_simple_work_queue(const char *thread_name_prefix, const char *name,
|
||||
struct vdo_thread *owner, void *private,
|
||||
const struct vdo_work_queue_type *type,
|
||||
struct simple_work_queue **queue_ptr)
|
||||
{
|
||||
DECLARE_COMPLETION_ONSTACK(started);
|
||||
struct simple_work_queue *queue;
|
||||
int i;
|
||||
struct task_struct *thread = NULL;
|
||||
int result;
|
||||
|
||||
VDO_ASSERT_LOG_ONLY((type->max_priority <= VDO_WORK_Q_MAX_PRIORITY),
|
||||
"queue priority count %u within limit %u", type->max_priority,
|
||||
VDO_WORK_Q_MAX_PRIORITY);
|
||||
|
||||
result = vdo_allocate(1, struct simple_work_queue, "simple work queue", &queue);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
queue->private = private;
|
||||
queue->started = &started;
|
||||
queue->common.type = type;
|
||||
queue->common.owner = owner;
|
||||
init_waitqueue_head(&queue->waiting_worker_threads);
|
||||
|
||||
result = vdo_duplicate_string(name, "queue name", &queue->common.name);
|
||||
if (result != VDO_SUCCESS) {
|
||||
vdo_free(queue);
|
||||
return -ENOMEM;
|
||||
}
|
||||
|
||||
for (i = 0; i <= type->max_priority; i++) {
|
||||
result = vdo_make_funnel_queue(&queue->priority_lists[i]);
|
||||
if (result != VDO_SUCCESS) {
|
||||
free_simple_work_queue(queue);
|
||||
return result;
|
||||
}
|
||||
}
|
||||
|
||||
thread = kthread_run(work_queue_runner, queue, "%s:%s", thread_name_prefix,
|
||||
queue->common.name);
|
||||
if (IS_ERR(thread)) {
|
||||
free_simple_work_queue(queue);
|
||||
return (int) PTR_ERR(thread);
|
||||
}
|
||||
|
||||
queue->thread = thread;
|
||||
|
||||
/*
|
||||
* If we don't wait to ensure the thread is running VDO code, a quick kthread_stop (due to
|
||||
* errors elsewhere) could cause it to never get as far as running VDO, skipping the
|
||||
* cleanup code.
|
||||
*
|
||||
* Eventually we should just make that path safe too, and then we won't need this
|
||||
* synchronization.
|
||||
*/
|
||||
wait_for_completion(&started);
|
||||
|
||||
*queue_ptr = queue;
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_make_work_queue() - Create a work queue; if multiple threads are requested, completions will
|
||||
* be distributed to them in round-robin fashion.
|
||||
*
|
||||
* Each queue is associated with a struct vdo_thread which has a single vdo thread id. Regardless
|
||||
* of the actual number of queues and threads allocated here, code outside of the queue
|
||||
* implementation will treat this as a single zone.
|
||||
*/
|
||||
int vdo_make_work_queue(const char *thread_name_prefix, const char *name,
|
||||
struct vdo_thread *owner, const struct vdo_work_queue_type *type,
|
||||
unsigned int thread_count, void *thread_privates[],
|
||||
struct vdo_work_queue **queue_ptr)
|
||||
{
|
||||
struct round_robin_work_queue *queue;
|
||||
int result;
|
||||
char thread_name[TASK_COMM_LEN];
|
||||
unsigned int i;
|
||||
|
||||
if (thread_count == 1) {
|
||||
struct simple_work_queue *simple_queue;
|
||||
void *context = ((thread_privates != NULL) ? thread_privates[0] : NULL);
|
||||
|
||||
result = make_simple_work_queue(thread_name_prefix, name, owner, context,
|
||||
type, &simple_queue);
|
||||
if (result == VDO_SUCCESS)
|
||||
*queue_ptr = &simple_queue->common;
|
||||
return result;
|
||||
}
|
||||
|
||||
result = vdo_allocate(1, struct round_robin_work_queue, "round-robin work queue",
|
||||
&queue);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
result = vdo_allocate(thread_count, struct simple_work_queue *,
|
||||
"subordinate work queues", &queue->service_queues);
|
||||
if (result != VDO_SUCCESS) {
|
||||
vdo_free(queue);
|
||||
return result;
|
||||
}
|
||||
|
||||
queue->num_service_queues = thread_count;
|
||||
queue->common.round_robin_mode = true;
|
||||
queue->common.owner = owner;
|
||||
|
||||
result = vdo_duplicate_string(name, "queue name", &queue->common.name);
|
||||
if (result != VDO_SUCCESS) {
|
||||
vdo_free(queue->service_queues);
|
||||
vdo_free(queue);
|
||||
return -ENOMEM;
|
||||
}
|
||||
|
||||
*queue_ptr = &queue->common;
|
||||
|
||||
for (i = 0; i < thread_count; i++) {
|
||||
void *context = ((thread_privates != NULL) ? thread_privates[i] : NULL);
|
||||
|
||||
snprintf(thread_name, sizeof(thread_name), "%s%u", name, i);
|
||||
result = make_simple_work_queue(thread_name_prefix, thread_name, owner,
|
||||
context, type, &queue->service_queues[i]);
|
||||
if (result != VDO_SUCCESS) {
|
||||
queue->num_service_queues = i;
|
||||
/* Destroy previously created subordinates. */
|
||||
vdo_free_work_queue(vdo_forget(*queue_ptr));
|
||||
return result;
|
||||
}
|
||||
}
|
||||
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
static void finish_simple_work_queue(struct simple_work_queue *queue)
|
||||
{
|
||||
if (queue->thread == NULL)
|
||||
return;
|
||||
|
||||
/* Tells the worker thread to shut down and waits for it to exit. */
|
||||
kthread_stop(queue->thread);
|
||||
queue->thread = NULL;
|
||||
}
|
||||
|
||||
static void finish_round_robin_work_queue(struct round_robin_work_queue *queue)
|
||||
{
|
||||
struct simple_work_queue **queue_table = queue->service_queues;
|
||||
unsigned int count = queue->num_service_queues;
|
||||
unsigned int i;
|
||||
|
||||
for (i = 0; i < count; i++)
|
||||
finish_simple_work_queue(queue_table[i]);
|
||||
}
|
||||
|
||||
/* No enqueueing of completions should be done once this function is called. */
|
||||
void vdo_finish_work_queue(struct vdo_work_queue *queue)
|
||||
{
|
||||
if (queue == NULL)
|
||||
return;
|
||||
|
||||
if (queue->round_robin_mode)
|
||||
finish_round_robin_work_queue(as_round_robin_work_queue(queue));
|
||||
else
|
||||
finish_simple_work_queue(as_simple_work_queue(queue));
|
||||
}
|
||||
|
||||
/* Debugging dumps */
|
||||
|
||||
static void dump_simple_work_queue(struct simple_work_queue *queue)
|
||||
{
|
||||
const char *thread_status = "no threads";
|
||||
char task_state_report = '-';
|
||||
|
||||
if (queue->thread != NULL) {
|
||||
task_state_report = task_state_to_char(queue->thread);
|
||||
thread_status = atomic_read(&queue->idle) ? "idle" : "running";
|
||||
}
|
||||
|
||||
vdo_log_info("workQ %px (%s) %s (%c)", &queue->common, queue->common.name,
|
||||
thread_status, task_state_report);
|
||||
|
||||
/* ->waiting_worker_threads wait queue status? anyone waiting? */
|
||||
}
|
||||
|
||||
/*
|
||||
* Write to the buffer some info about the completion, for logging. Since the common use case is
|
||||
* dumping info about a lot of completions to syslog all at once, the format favors brevity over
|
||||
* readability.
|
||||
*/
|
||||
void vdo_dump_work_queue(struct vdo_work_queue *queue)
|
||||
{
|
||||
if (queue->round_robin_mode) {
|
||||
struct round_robin_work_queue *round_robin = as_round_robin_work_queue(queue);
|
||||
unsigned int i;
|
||||
|
||||
for (i = 0; i < round_robin->num_service_queues; i++)
|
||||
dump_simple_work_queue(round_robin->service_queues[i]);
|
||||
} else {
|
||||
dump_simple_work_queue(as_simple_work_queue(queue));
|
||||
}
|
||||
}
|
||||
|
||||
static void get_function_name(void *pointer, char *buffer, size_t buffer_length)
|
||||
{
|
||||
if (pointer == NULL) {
|
||||
/*
|
||||
* Format "%ps" logs a null pointer as "(null)" with a bunch of leading spaces. We
|
||||
* sometimes use this when logging lots of data; don't be so verbose.
|
||||
*/
|
||||
strscpy(buffer, "-", buffer_length);
|
||||
} else {
|
||||
/*
|
||||
* Use a pragma to defeat gcc's format checking, which doesn't understand that
|
||||
* "%ps" actually does support a precision spec in Linux kernel code.
|
||||
*/
|
||||
char *space;
|
||||
|
||||
#pragma GCC diagnostic push
|
||||
#pragma GCC diagnostic ignored "-Wformat"
|
||||
snprintf(buffer, buffer_length, "%.*ps", buffer_length - 1, pointer);
|
||||
#pragma GCC diagnostic pop
|
||||
|
||||
space = strchr(buffer, ' ');
|
||||
if (space != NULL)
|
||||
*space = '\0';
|
||||
}
|
||||
}
|
||||
|
||||
void vdo_dump_completion_to_buffer(struct vdo_completion *completion, char *buffer,
|
||||
size_t length)
|
||||
{
|
||||
size_t current_length =
|
||||
scnprintf(buffer, length, "%.*s/", TASK_COMM_LEN,
|
||||
(completion->my_queue == NULL ? "-" : completion->my_queue->name));
|
||||
|
||||
if (current_length < length - 1) {
|
||||
get_function_name((void *) completion->callback, buffer + current_length,
|
||||
length - current_length);
|
||||
}
|
||||
}
|
||||
|
||||
/* Completion submission */
|
||||
/*
|
||||
* If the completion has a timeout that has already passed, the timeout handler function may be
|
||||
* invoked by this function.
|
||||
*/
|
||||
void vdo_enqueue_work_queue(struct vdo_work_queue *queue,
|
||||
struct vdo_completion *completion)
|
||||
{
|
||||
/*
|
||||
* Convert the provided generic vdo_work_queue to the simple_work_queue to actually queue
|
||||
* on.
|
||||
*/
|
||||
struct simple_work_queue *simple_queue = NULL;
|
||||
|
||||
if (!queue->round_robin_mode) {
|
||||
simple_queue = as_simple_work_queue(queue);
|
||||
} else {
|
||||
struct round_robin_work_queue *round_robin = as_round_robin_work_queue(queue);
|
||||
|
||||
/*
|
||||
* It shouldn't be a big deal if the same rotor gets used for multiple work queues.
|
||||
* Any patterns that might develop are likely to be disrupted by random ordering of
|
||||
* multiple completions and migration between cores, unless the load is so light as
|
||||
* to be regular in ordering of tasks and the threads are confined to individual
|
||||
* cores; with a load that light we won't care.
|
||||
*/
|
||||
unsigned int rotor = this_cpu_inc_return(service_queue_rotor);
|
||||
unsigned int index = rotor % round_robin->num_service_queues;
|
||||
|
||||
simple_queue = round_robin->service_queues[index];
|
||||
}
|
||||
|
||||
enqueue_work_queue_completion(simple_queue, completion);
|
||||
}
|
||||
|
||||
/* Misc */
|
||||
|
||||
/*
|
||||
* Return the work queue pointer recorded at initialization time in the work-queue stack handle
|
||||
* initialized on the stack of the current thread, if any.
|
||||
*/
|
||||
static struct simple_work_queue *get_current_thread_work_queue(void)
|
||||
{
|
||||
/*
|
||||
* In interrupt context, if a vdo thread is what got interrupted, the calls below will find
|
||||
* the queue for the thread which was interrupted. However, the interrupted thread may have
|
||||
* been processing a completion, in which case starting to process another would violate
|
||||
* our concurrency assumptions.
|
||||
*/
|
||||
if (in_interrupt())
|
||||
return NULL;
|
||||
|
||||
if (kthread_func(current) != work_queue_runner)
|
||||
/* Not a VDO work queue thread. */
|
||||
return NULL;
|
||||
|
||||
return kthread_data(current);
|
||||
}
|
||||
|
||||
struct vdo_work_queue *vdo_get_current_work_queue(void)
|
||||
{
|
||||
struct simple_work_queue *queue = get_current_thread_work_queue();
|
||||
|
||||
return (queue == NULL) ? NULL : &queue->common;
|
||||
}
|
||||
|
||||
struct vdo_thread *vdo_get_work_queue_owner(struct vdo_work_queue *queue)
|
||||
{
|
||||
return queue->owner;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_get_work_queue_private_data() - Returns the private data for the current thread's work
|
||||
* queue, or NULL if none or if the current thread is not a
|
||||
* work queue thread.
|
||||
*/
|
||||
void *vdo_get_work_queue_private_data(void)
|
||||
{
|
||||
struct simple_work_queue *queue = get_current_thread_work_queue();
|
||||
|
||||
return (queue != NULL) ? queue->private : NULL;
|
||||
}
|
||||
|
||||
bool vdo_work_queue_type_is(struct vdo_work_queue *queue,
|
||||
const struct vdo_work_queue_type *type)
|
||||
{
|
||||
return (queue->type == type);
|
||||
}
|
|
@ -0,0 +1,51 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef VDO_WORK_QUEUE_H
|
||||
#define VDO_WORK_QUEUE_H
|
||||
|
||||
#include <linux/sched.h> /* for TASK_COMM_LEN */
|
||||
|
||||
#include "types.h"
|
||||
|
||||
enum {
|
||||
MAX_VDO_WORK_QUEUE_NAME_LEN = TASK_COMM_LEN,
|
||||
};
|
||||
|
||||
struct vdo_work_queue_type {
|
||||
void (*start)(void *context);
|
||||
void (*finish)(void *context);
|
||||
enum vdo_completion_priority max_priority;
|
||||
enum vdo_completion_priority default_priority;
|
||||
};
|
||||
|
||||
struct vdo_completion;
|
||||
struct vdo_thread;
|
||||
struct vdo_work_queue;
|
||||
|
||||
int vdo_make_work_queue(const char *thread_name_prefix, const char *name,
|
||||
struct vdo_thread *owner, const struct vdo_work_queue_type *type,
|
||||
unsigned int thread_count, void *thread_privates[],
|
||||
struct vdo_work_queue **queue_ptr);
|
||||
|
||||
void vdo_enqueue_work_queue(struct vdo_work_queue *queue, struct vdo_completion *completion);
|
||||
|
||||
void vdo_finish_work_queue(struct vdo_work_queue *queue);
|
||||
|
||||
void vdo_free_work_queue(struct vdo_work_queue *queue);
|
||||
|
||||
void vdo_dump_work_queue(struct vdo_work_queue *queue);
|
||||
|
||||
void vdo_dump_completion_to_buffer(struct vdo_completion *completion, char *buffer,
|
||||
size_t length);
|
||||
|
||||
void *vdo_get_work_queue_private_data(void);
|
||||
struct vdo_work_queue *vdo_get_current_work_queue(void);
|
||||
struct vdo_thread *vdo_get_work_queue_owner(struct vdo_work_queue *queue);
|
||||
|
||||
bool __must_check vdo_work_queue_type_is(struct vdo_work_queue *queue,
|
||||
const struct vdo_work_queue_type *type);
|
||||
|
||||
#endif /* VDO_WORK_QUEUE_H */
|
|
@ -0,0 +1,293 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "chapter-index.h"
|
||||
|
||||
#include "errors.h"
|
||||
#include "logger.h"
|
||||
#include "memory-alloc.h"
|
||||
#include "permassert.h"
|
||||
|
||||
#include "hash-utils.h"
|
||||
#include "indexer.h"
|
||||
|
||||
int uds_make_open_chapter_index(struct open_chapter_index **chapter_index,
|
||||
const struct index_geometry *geometry, u64 volume_nonce)
|
||||
{
|
||||
int result;
|
||||
size_t memory_size;
|
||||
struct open_chapter_index *index;
|
||||
|
||||
result = vdo_allocate(1, struct open_chapter_index, "open chapter index", &index);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
/*
|
||||
* The delta index will rebalance delta lists when memory gets tight,
|
||||
* so give the chapter index one extra page.
|
||||
*/
|
||||
memory_size = ((geometry->index_pages_per_chapter + 1) * geometry->bytes_per_page);
|
||||
index->geometry = geometry;
|
||||
index->volume_nonce = volume_nonce;
|
||||
result = uds_initialize_delta_index(&index->delta_index, 1,
|
||||
geometry->delta_lists_per_chapter,
|
||||
geometry->chapter_mean_delta,
|
||||
geometry->chapter_payload_bits,
|
||||
memory_size, 'm');
|
||||
if (result != UDS_SUCCESS) {
|
||||
vdo_free(index);
|
||||
return result;
|
||||
}
|
||||
|
||||
index->memory_size = index->delta_index.memory_size + sizeof(struct open_chapter_index);
|
||||
*chapter_index = index;
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
void uds_free_open_chapter_index(struct open_chapter_index *chapter_index)
|
||||
{
|
||||
if (chapter_index == NULL)
|
||||
return;
|
||||
|
||||
uds_uninitialize_delta_index(&chapter_index->delta_index);
|
||||
vdo_free(chapter_index);
|
||||
}
|
||||
|
||||
/* Re-initialize an open chapter index for a new chapter. */
|
||||
void uds_empty_open_chapter_index(struct open_chapter_index *chapter_index,
|
||||
u64 virtual_chapter_number)
|
||||
{
|
||||
uds_reset_delta_index(&chapter_index->delta_index);
|
||||
chapter_index->virtual_chapter_number = virtual_chapter_number;
|
||||
}
|
||||
|
||||
static inline bool was_entry_found(const struct delta_index_entry *entry, u32 address)
|
||||
{
|
||||
return (!entry->at_end) && (entry->key == address);
|
||||
}
|
||||
|
||||
/* Associate a record name with the record page containing its metadata. */
|
||||
int uds_put_open_chapter_index_record(struct open_chapter_index *chapter_index,
|
||||
const struct uds_record_name *name,
|
||||
u32 page_number)
|
||||
{
|
||||
int result;
|
||||
struct delta_index_entry entry;
|
||||
u32 address;
|
||||
u32 list_number;
|
||||
const u8 *found_name;
|
||||
bool found;
|
||||
const struct index_geometry *geometry = chapter_index->geometry;
|
||||
u64 chapter_number = chapter_index->virtual_chapter_number;
|
||||
u32 record_pages = geometry->record_pages_per_chapter;
|
||||
|
||||
result = VDO_ASSERT(page_number < record_pages,
|
||||
"Page number within chapter (%u) exceeds the maximum value %u",
|
||||
page_number, record_pages);
|
||||
if (result != VDO_SUCCESS)
|
||||
return UDS_INVALID_ARGUMENT;
|
||||
|
||||
address = uds_hash_to_chapter_delta_address(name, geometry);
|
||||
list_number = uds_hash_to_chapter_delta_list(name, geometry);
|
||||
result = uds_get_delta_index_entry(&chapter_index->delta_index, list_number,
|
||||
address, name->name, &entry);
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
found = was_entry_found(&entry, address);
|
||||
result = VDO_ASSERT(!(found && entry.is_collision),
|
||||
"Chunk appears more than once in chapter %llu",
|
||||
(unsigned long long) chapter_number);
|
||||
if (result != VDO_SUCCESS)
|
||||
return UDS_BAD_STATE;
|
||||
|
||||
found_name = (found ? name->name : NULL);
|
||||
return uds_put_delta_index_entry(&entry, address, page_number, found_name);
|
||||
}
|
||||
|
||||
/*
|
||||
* Pack a section of an open chapter index into a chapter index page. A range of delta lists
|
||||
* (starting with a specified list index) is copied from the open chapter index into a memory page.
|
||||
* The number of lists copied onto the page is returned to the caller on success.
|
||||
*
|
||||
* @chapter_index: The open chapter index
|
||||
* @memory: The memory page to use
|
||||
* @first_list: The first delta list number to be copied
|
||||
* @last_page: If true, this is the last page of the chapter index and all the remaining lists must
|
||||
* be packed onto this page
|
||||
* @lists_packed: The number of delta lists that were packed onto this page
|
||||
*/
|
||||
int uds_pack_open_chapter_index_page(struct open_chapter_index *chapter_index,
|
||||
u8 *memory, u32 first_list, bool last_page,
|
||||
u32 *lists_packed)
|
||||
{
|
||||
int result;
|
||||
struct delta_index *delta_index = &chapter_index->delta_index;
|
||||
struct delta_index_stats stats;
|
||||
u64 nonce = chapter_index->volume_nonce;
|
||||
u64 chapter_number = chapter_index->virtual_chapter_number;
|
||||
const struct index_geometry *geometry = chapter_index->geometry;
|
||||
u32 list_count = geometry->delta_lists_per_chapter;
|
||||
unsigned int removals = 0;
|
||||
struct delta_index_entry entry;
|
||||
u32 next_list;
|
||||
s32 list_number;
|
||||
|
||||
for (;;) {
|
||||
result = uds_pack_delta_index_page(delta_index, nonce, memory,
|
||||
geometry->bytes_per_page,
|
||||
chapter_number, first_list,
|
||||
lists_packed);
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
if ((first_list + *lists_packed) == list_count) {
|
||||
/* All lists are packed. */
|
||||
break;
|
||||
} else if (*lists_packed == 0) {
|
||||
/*
|
||||
* The next delta list does not fit on a page. This delta list will be
|
||||
* removed.
|
||||
*/
|
||||
} else if (last_page) {
|
||||
/*
|
||||
* This is the last page and there are lists left unpacked, but all of the
|
||||
* remaining lists must fit on the page. Find a list that contains entries
|
||||
* and remove the entire list. Try the first list that does not fit. If it
|
||||
* is empty, we will select the last list that already fits and has any
|
||||
* entries.
|
||||
*/
|
||||
} else {
|
||||
/* This page is done. */
|
||||
break;
|
||||
}
|
||||
|
||||
if (removals == 0) {
|
||||
uds_get_delta_index_stats(delta_index, &stats);
|
||||
vdo_log_warning("The chapter index for chapter %llu contains %llu entries with %llu collisions",
|
||||
(unsigned long long) chapter_number,
|
||||
(unsigned long long) stats.record_count,
|
||||
(unsigned long long) stats.collision_count);
|
||||
}
|
||||
|
||||
list_number = *lists_packed;
|
||||
do {
|
||||
if (list_number < 0)
|
||||
return UDS_OVERFLOW;
|
||||
|
||||
next_list = first_list + list_number--,
|
||||
result = uds_start_delta_index_search(delta_index, next_list, 0,
|
||||
&entry);
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
result = uds_next_delta_index_entry(&entry);
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
} while (entry.at_end);
|
||||
|
||||
do {
|
||||
result = uds_remove_delta_index_entry(&entry);
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
removals++;
|
||||
} while (!entry.at_end);
|
||||
}
|
||||
|
||||
if (removals > 0) {
|
||||
vdo_log_warning("To avoid chapter index page overflow in chapter %llu, %u entries were removed from the chapter index",
|
||||
(unsigned long long) chapter_number, removals);
|
||||
}
|
||||
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
/* Make a new chapter index page, initializing it with the data from a given index_page buffer. */
|
||||
int uds_initialize_chapter_index_page(struct delta_index_page *index_page,
|
||||
const struct index_geometry *geometry,
|
||||
u8 *page_buffer, u64 volume_nonce)
|
||||
{
|
||||
return uds_initialize_delta_index_page(index_page, volume_nonce,
|
||||
geometry->chapter_mean_delta,
|
||||
geometry->chapter_payload_bits,
|
||||
page_buffer, geometry->bytes_per_page);
|
||||
}
|
||||
|
||||
/* Validate a chapter index page read during rebuild. */
|
||||
int uds_validate_chapter_index_page(const struct delta_index_page *index_page,
|
||||
const struct index_geometry *geometry)
|
||||
{
|
||||
int result;
|
||||
const struct delta_index *delta_index = &index_page->delta_index;
|
||||
u32 first = index_page->lowest_list_number;
|
||||
u32 last = index_page->highest_list_number;
|
||||
u32 list_number;
|
||||
|
||||
/* We walk every delta list from start to finish. */
|
||||
for (list_number = first; list_number <= last; list_number++) {
|
||||
struct delta_index_entry entry;
|
||||
|
||||
result = uds_start_delta_index_search(delta_index, list_number - first,
|
||||
0, &entry);
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
for (;;) {
|
||||
result = uds_next_delta_index_entry(&entry);
|
||||
if (result != UDS_SUCCESS) {
|
||||
/*
|
||||
* A random bit stream is highly likely to arrive here when we go
|
||||
* past the end of the delta list.
|
||||
*/
|
||||
return result;
|
||||
}
|
||||
|
||||
if (entry.at_end)
|
||||
break;
|
||||
|
||||
/* Also make sure that the record page field contains a plausible value. */
|
||||
if (uds_get_delta_entry_value(&entry) >=
|
||||
geometry->record_pages_per_chapter) {
|
||||
/*
|
||||
* Do not log this as an error. It happens in normal operation when
|
||||
* we are doing a rebuild but haven't written the entire volume
|
||||
* once.
|
||||
*/
|
||||
return UDS_CORRUPT_DATA;
|
||||
}
|
||||
}
|
||||
}
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
/*
|
||||
* Search a chapter index page for a record name, returning the record page number that may contain
|
||||
* the name.
|
||||
*/
|
||||
int uds_search_chapter_index_page(struct delta_index_page *index_page,
|
||||
const struct index_geometry *geometry,
|
||||
const struct uds_record_name *name,
|
||||
u16 *record_page_ptr)
|
||||
{
|
||||
int result;
|
||||
struct delta_index *delta_index = &index_page->delta_index;
|
||||
u32 address = uds_hash_to_chapter_delta_address(name, geometry);
|
||||
u32 delta_list_number = uds_hash_to_chapter_delta_list(name, geometry);
|
||||
u32 sub_list_number = delta_list_number - index_page->lowest_list_number;
|
||||
struct delta_index_entry entry;
|
||||
|
||||
result = uds_get_delta_index_entry(delta_index, sub_list_number, address,
|
||||
name->name, &entry);
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
if (was_entry_found(&entry, address))
|
||||
*record_page_ptr = uds_get_delta_entry_value(&entry);
|
||||
else
|
||||
*record_page_ptr = NO_CHAPTER_INDEX_ENTRY;
|
||||
|
||||
return UDS_SUCCESS;
|
||||
}
|
|
@ -0,0 +1,61 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef UDS_CHAPTER_INDEX_H
|
||||
#define UDS_CHAPTER_INDEX_H
|
||||
|
||||
#include <linux/limits.h>
|
||||
|
||||
#include "delta-index.h"
|
||||
#include "geometry.h"
|
||||
|
||||
/*
|
||||
* A chapter index for an open chapter is a mutable structure that tracks all the records that have
|
||||
* been added to the chapter. A chapter index for a closed chapter is similar except that it is
|
||||
* immutable because the contents of a closed chapter can never change, and the immutable structure
|
||||
* is more efficient. Both types of chapter index are implemented with a delta index.
|
||||
*/
|
||||
|
||||
/* The value returned when no entry is found in the chapter index. */
|
||||
#define NO_CHAPTER_INDEX_ENTRY U16_MAX
|
||||
|
||||
struct open_chapter_index {
|
||||
const struct index_geometry *geometry;
|
||||
struct delta_index delta_index;
|
||||
u64 virtual_chapter_number;
|
||||
u64 volume_nonce;
|
||||
size_t memory_size;
|
||||
};
|
||||
|
||||
int __must_check uds_make_open_chapter_index(struct open_chapter_index **chapter_index,
|
||||
const struct index_geometry *geometry,
|
||||
u64 volume_nonce);
|
||||
|
||||
void uds_free_open_chapter_index(struct open_chapter_index *chapter_index);
|
||||
|
||||
void uds_empty_open_chapter_index(struct open_chapter_index *chapter_index,
|
||||
u64 virtual_chapter_number);
|
||||
|
||||
int __must_check uds_put_open_chapter_index_record(struct open_chapter_index *chapter_index,
|
||||
const struct uds_record_name *name,
|
||||
u32 page_number);
|
||||
|
||||
int __must_check uds_pack_open_chapter_index_page(struct open_chapter_index *chapter_index,
|
||||
u8 *memory, u32 first_list,
|
||||
bool last_page, u32 *lists_packed);
|
||||
|
||||
int __must_check uds_initialize_chapter_index_page(struct delta_index_page *index_page,
|
||||
const struct index_geometry *geometry,
|
||||
u8 *page_buffer, u64 volume_nonce);
|
||||
|
||||
int __must_check uds_validate_chapter_index_page(const struct delta_index_page *index_page,
|
||||
const struct index_geometry *geometry);
|
||||
|
||||
int __must_check uds_search_chapter_index_page(struct delta_index_page *index_page,
|
||||
const struct index_geometry *geometry,
|
||||
const struct uds_record_name *name,
|
||||
u16 *record_page_ptr);
|
||||
|
||||
#endif /* UDS_CHAPTER_INDEX_H */
|
|
@ -0,0 +1,376 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "config.h"
|
||||
|
||||
#include "logger.h"
|
||||
#include "memory-alloc.h"
|
||||
#include "numeric.h"
|
||||
#include "string-utils.h"
|
||||
#include "thread-utils.h"
|
||||
|
||||
static const u8 INDEX_CONFIG_MAGIC[] = "ALBIC";
|
||||
static const u8 INDEX_CONFIG_VERSION_6_02[] = "06.02";
|
||||
static const u8 INDEX_CONFIG_VERSION_8_02[] = "08.02";
|
||||
|
||||
#define DEFAULT_VOLUME_READ_THREADS 2
|
||||
#define MAX_VOLUME_READ_THREADS 16
|
||||
#define INDEX_CONFIG_MAGIC_LENGTH (sizeof(INDEX_CONFIG_MAGIC) - 1)
|
||||
#define INDEX_CONFIG_VERSION_LENGTH ((int)(sizeof(INDEX_CONFIG_VERSION_6_02) - 1))
|
||||
|
||||
static bool is_version(const u8 *version, u8 *buffer)
|
||||
{
|
||||
return memcmp(version, buffer, INDEX_CONFIG_VERSION_LENGTH) == 0;
|
||||
}
|
||||
|
||||
static bool are_matching_configurations(struct uds_configuration *saved_config,
|
||||
struct index_geometry *saved_geometry,
|
||||
struct uds_configuration *user)
|
||||
{
|
||||
struct index_geometry *geometry = user->geometry;
|
||||
bool result = true;
|
||||
|
||||
if (saved_geometry->record_pages_per_chapter != geometry->record_pages_per_chapter) {
|
||||
vdo_log_error("Record pages per chapter (%u) does not match (%u)",
|
||||
saved_geometry->record_pages_per_chapter,
|
||||
geometry->record_pages_per_chapter);
|
||||
result = false;
|
||||
}
|
||||
|
||||
if (saved_geometry->chapters_per_volume != geometry->chapters_per_volume) {
|
||||
vdo_log_error("Chapter count (%u) does not match (%u)",
|
||||
saved_geometry->chapters_per_volume,
|
||||
geometry->chapters_per_volume);
|
||||
result = false;
|
||||
}
|
||||
|
||||
if (saved_geometry->sparse_chapters_per_volume != geometry->sparse_chapters_per_volume) {
|
||||
vdo_log_error("Sparse chapter count (%u) does not match (%u)",
|
||||
saved_geometry->sparse_chapters_per_volume,
|
||||
geometry->sparse_chapters_per_volume);
|
||||
result = false;
|
||||
}
|
||||
|
||||
if (saved_config->cache_chapters != user->cache_chapters) {
|
||||
vdo_log_error("Cache size (%u) does not match (%u)",
|
||||
saved_config->cache_chapters, user->cache_chapters);
|
||||
result = false;
|
||||
}
|
||||
|
||||
if (saved_config->volume_index_mean_delta != user->volume_index_mean_delta) {
|
||||
vdo_log_error("Volume index mean delta (%u) does not match (%u)",
|
||||
saved_config->volume_index_mean_delta,
|
||||
user->volume_index_mean_delta);
|
||||
result = false;
|
||||
}
|
||||
|
||||
if (saved_geometry->bytes_per_page != geometry->bytes_per_page) {
|
||||
vdo_log_error("Bytes per page value (%zu) does not match (%zu)",
|
||||
saved_geometry->bytes_per_page, geometry->bytes_per_page);
|
||||
result = false;
|
||||
}
|
||||
|
||||
if (saved_config->sparse_sample_rate != user->sparse_sample_rate) {
|
||||
vdo_log_error("Sparse sample rate (%u) does not match (%u)",
|
||||
saved_config->sparse_sample_rate,
|
||||
user->sparse_sample_rate);
|
||||
result = false;
|
||||
}
|
||||
|
||||
if (saved_config->nonce != user->nonce) {
|
||||
vdo_log_error("Nonce (%llu) does not match (%llu)",
|
||||
(unsigned long long) saved_config->nonce,
|
||||
(unsigned long long) user->nonce);
|
||||
result = false;
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
/* Read the configuration and validate it against the provided one. */
|
||||
int uds_validate_config_contents(struct buffered_reader *reader,
|
||||
struct uds_configuration *user_config)
|
||||
{
|
||||
int result;
|
||||
struct uds_configuration config;
|
||||
struct index_geometry geometry;
|
||||
u8 version_buffer[INDEX_CONFIG_VERSION_LENGTH];
|
||||
u32 bytes_per_page;
|
||||
u8 buffer[sizeof(struct uds_configuration_6_02)];
|
||||
size_t offset = 0;
|
||||
|
||||
result = uds_verify_buffered_data(reader, INDEX_CONFIG_MAGIC,
|
||||
INDEX_CONFIG_MAGIC_LENGTH);
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
result = uds_read_from_buffered_reader(reader, version_buffer,
|
||||
INDEX_CONFIG_VERSION_LENGTH);
|
||||
if (result != UDS_SUCCESS)
|
||||
return vdo_log_error_strerror(result, "cannot read index config version");
|
||||
|
||||
if (!is_version(INDEX_CONFIG_VERSION_6_02, version_buffer) &&
|
||||
!is_version(INDEX_CONFIG_VERSION_8_02, version_buffer)) {
|
||||
return vdo_log_error_strerror(UDS_CORRUPT_DATA,
|
||||
"unsupported configuration version: '%.*s'",
|
||||
INDEX_CONFIG_VERSION_LENGTH,
|
||||
version_buffer);
|
||||
}
|
||||
|
||||
result = uds_read_from_buffered_reader(reader, buffer, sizeof(buffer));
|
||||
if (result != UDS_SUCCESS)
|
||||
return vdo_log_error_strerror(result, "cannot read config data");
|
||||
|
||||
decode_u32_le(buffer, &offset, &geometry.record_pages_per_chapter);
|
||||
decode_u32_le(buffer, &offset, &geometry.chapters_per_volume);
|
||||
decode_u32_le(buffer, &offset, &geometry.sparse_chapters_per_volume);
|
||||
decode_u32_le(buffer, &offset, &config.cache_chapters);
|
||||
offset += sizeof(u32);
|
||||
decode_u32_le(buffer, &offset, &config.volume_index_mean_delta);
|
||||
decode_u32_le(buffer, &offset, &bytes_per_page);
|
||||
geometry.bytes_per_page = bytes_per_page;
|
||||
decode_u32_le(buffer, &offset, &config.sparse_sample_rate);
|
||||
decode_u64_le(buffer, &offset, &config.nonce);
|
||||
|
||||
result = VDO_ASSERT(offset == sizeof(struct uds_configuration_6_02),
|
||||
"%zu bytes read but not decoded",
|
||||
sizeof(struct uds_configuration_6_02) - offset);
|
||||
if (result != VDO_SUCCESS)
|
||||
return UDS_CORRUPT_DATA;
|
||||
|
||||
if (is_version(INDEX_CONFIG_VERSION_6_02, version_buffer)) {
|
||||
user_config->geometry->remapped_virtual = 0;
|
||||
user_config->geometry->remapped_physical = 0;
|
||||
} else {
|
||||
u8 remapping[sizeof(u64) + sizeof(u64)];
|
||||
|
||||
result = uds_read_from_buffered_reader(reader, remapping,
|
||||
sizeof(remapping));
|
||||
if (result != UDS_SUCCESS)
|
||||
return vdo_log_error_strerror(result, "cannot read converted config");
|
||||
|
||||
offset = 0;
|
||||
decode_u64_le(remapping, &offset,
|
||||
&user_config->geometry->remapped_virtual);
|
||||
decode_u64_le(remapping, &offset,
|
||||
&user_config->geometry->remapped_physical);
|
||||
}
|
||||
|
||||
if (!are_matching_configurations(&config, &geometry, user_config)) {
|
||||
vdo_log_warning("Supplied configuration does not match save");
|
||||
return UDS_NO_INDEX;
|
||||
}
|
||||
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
/*
|
||||
* Write the configuration to stable storage. If the superblock version is < 4, write the 6.02
|
||||
* version; otherwise write the 8.02 version, indicating the configuration is for an index that has
|
||||
* been reduced by one chapter.
|
||||
*/
|
||||
int uds_write_config_contents(struct buffered_writer *writer,
|
||||
struct uds_configuration *config, u32 version)
|
||||
{
|
||||
int result;
|
||||
struct index_geometry *geometry = config->geometry;
|
||||
u8 buffer[sizeof(struct uds_configuration_8_02)];
|
||||
size_t offset = 0;
|
||||
|
||||
result = uds_write_to_buffered_writer(writer, INDEX_CONFIG_MAGIC,
|
||||
INDEX_CONFIG_MAGIC_LENGTH);
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
/*
|
||||
* If version is < 4, the index has not been reduced by a chapter so it must be written out
|
||||
* as version 6.02 so that it is still compatible with older versions of UDS.
|
||||
*/
|
||||
if (version >= 4) {
|
||||
result = uds_write_to_buffered_writer(writer, INDEX_CONFIG_VERSION_8_02,
|
||||
INDEX_CONFIG_VERSION_LENGTH);
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
} else {
|
||||
result = uds_write_to_buffered_writer(writer, INDEX_CONFIG_VERSION_6_02,
|
||||
INDEX_CONFIG_VERSION_LENGTH);
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
}
|
||||
|
||||
encode_u32_le(buffer, &offset, geometry->record_pages_per_chapter);
|
||||
encode_u32_le(buffer, &offset, geometry->chapters_per_volume);
|
||||
encode_u32_le(buffer, &offset, geometry->sparse_chapters_per_volume);
|
||||
encode_u32_le(buffer, &offset, config->cache_chapters);
|
||||
encode_u32_le(buffer, &offset, 0);
|
||||
encode_u32_le(buffer, &offset, config->volume_index_mean_delta);
|
||||
encode_u32_le(buffer, &offset, geometry->bytes_per_page);
|
||||
encode_u32_le(buffer, &offset, config->sparse_sample_rate);
|
||||
encode_u64_le(buffer, &offset, config->nonce);
|
||||
|
||||
result = VDO_ASSERT(offset == sizeof(struct uds_configuration_6_02),
|
||||
"%zu bytes encoded, of %zu expected", offset,
|
||||
sizeof(struct uds_configuration_6_02));
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
if (version >= 4) {
|
||||
encode_u64_le(buffer, &offset, geometry->remapped_virtual);
|
||||
encode_u64_le(buffer, &offset, geometry->remapped_physical);
|
||||
}
|
||||
|
||||
return uds_write_to_buffered_writer(writer, buffer, offset);
|
||||
}
|
||||
|
||||
/* Compute configuration parameters that depend on memory size. */
|
||||
static int compute_memory_sizes(uds_memory_config_size_t mem_gb, bool sparse,
|
||||
u32 *chapters_per_volume, u32 *record_pages_per_chapter,
|
||||
u32 *sparse_chapters_per_volume)
|
||||
{
|
||||
u32 reduced_chapters = 0;
|
||||
u32 base_chapters;
|
||||
|
||||
if (mem_gb == UDS_MEMORY_CONFIG_256MB) {
|
||||
base_chapters = DEFAULT_CHAPTERS_PER_VOLUME;
|
||||
*record_pages_per_chapter = SMALL_RECORD_PAGES_PER_CHAPTER;
|
||||
} else if (mem_gb == UDS_MEMORY_CONFIG_512MB) {
|
||||
base_chapters = DEFAULT_CHAPTERS_PER_VOLUME;
|
||||
*record_pages_per_chapter = 2 * SMALL_RECORD_PAGES_PER_CHAPTER;
|
||||
} else if (mem_gb == UDS_MEMORY_CONFIG_768MB) {
|
||||
base_chapters = DEFAULT_CHAPTERS_PER_VOLUME;
|
||||
*record_pages_per_chapter = 3 * SMALL_RECORD_PAGES_PER_CHAPTER;
|
||||
} else if ((mem_gb >= 1) && (mem_gb <= UDS_MEMORY_CONFIG_MAX)) {
|
||||
base_chapters = mem_gb * DEFAULT_CHAPTERS_PER_VOLUME;
|
||||
*record_pages_per_chapter = DEFAULT_RECORD_PAGES_PER_CHAPTER;
|
||||
} else if (mem_gb == UDS_MEMORY_CONFIG_REDUCED_256MB) {
|
||||
reduced_chapters = 1;
|
||||
base_chapters = DEFAULT_CHAPTERS_PER_VOLUME;
|
||||
*record_pages_per_chapter = SMALL_RECORD_PAGES_PER_CHAPTER;
|
||||
} else if (mem_gb == UDS_MEMORY_CONFIG_REDUCED_512MB) {
|
||||
reduced_chapters = 1;
|
||||
base_chapters = DEFAULT_CHAPTERS_PER_VOLUME;
|
||||
*record_pages_per_chapter = 2 * SMALL_RECORD_PAGES_PER_CHAPTER;
|
||||
} else if (mem_gb == UDS_MEMORY_CONFIG_REDUCED_768MB) {
|
||||
reduced_chapters = 1;
|
||||
base_chapters = DEFAULT_CHAPTERS_PER_VOLUME;
|
||||
*record_pages_per_chapter = 3 * SMALL_RECORD_PAGES_PER_CHAPTER;
|
||||
} else if ((mem_gb >= 1 + UDS_MEMORY_CONFIG_REDUCED) &&
|
||||
(mem_gb <= UDS_MEMORY_CONFIG_REDUCED_MAX)) {
|
||||
reduced_chapters = 1;
|
||||
base_chapters = ((mem_gb - UDS_MEMORY_CONFIG_REDUCED) *
|
||||
DEFAULT_CHAPTERS_PER_VOLUME);
|
||||
*record_pages_per_chapter = DEFAULT_RECORD_PAGES_PER_CHAPTER;
|
||||
} else {
|
||||
vdo_log_error("received invalid memory size");
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
if (sparse) {
|
||||
/* Make 95% of chapters sparse, allowing 10x more records. */
|
||||
*sparse_chapters_per_volume = (19 * base_chapters) / 2;
|
||||
base_chapters *= 10;
|
||||
} else {
|
||||
*sparse_chapters_per_volume = 0;
|
||||
}
|
||||
|
||||
*chapters_per_volume = base_chapters - reduced_chapters;
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
static unsigned int __must_check normalize_zone_count(unsigned int requested)
|
||||
{
|
||||
unsigned int zone_count = requested;
|
||||
|
||||
if (zone_count == 0)
|
||||
zone_count = num_online_cpus() / 2;
|
||||
|
||||
if (zone_count < 1)
|
||||
zone_count = 1;
|
||||
|
||||
if (zone_count > MAX_ZONES)
|
||||
zone_count = MAX_ZONES;
|
||||
|
||||
vdo_log_info("Using %u indexing zone%s for concurrency.",
|
||||
zone_count, zone_count == 1 ? "" : "s");
|
||||
return zone_count;
|
||||
}
|
||||
|
||||
static unsigned int __must_check normalize_read_threads(unsigned int requested)
|
||||
{
|
||||
unsigned int read_threads = requested;
|
||||
|
||||
if (read_threads < 1)
|
||||
read_threads = DEFAULT_VOLUME_READ_THREADS;
|
||||
|
||||
if (read_threads > MAX_VOLUME_READ_THREADS)
|
||||
read_threads = MAX_VOLUME_READ_THREADS;
|
||||
|
||||
return read_threads;
|
||||
}
|
||||
|
||||
int uds_make_configuration(const struct uds_parameters *params,
|
||||
struct uds_configuration **config_ptr)
|
||||
{
|
||||
struct uds_configuration *config;
|
||||
u32 chapters_per_volume = 0;
|
||||
u32 record_pages_per_chapter = 0;
|
||||
u32 sparse_chapters_per_volume = 0;
|
||||
int result;
|
||||
|
||||
result = compute_memory_sizes(params->memory_size, params->sparse,
|
||||
&chapters_per_volume, &record_pages_per_chapter,
|
||||
&sparse_chapters_per_volume);
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
result = vdo_allocate(1, struct uds_configuration, __func__, &config);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
result = uds_make_index_geometry(DEFAULT_BYTES_PER_PAGE, record_pages_per_chapter,
|
||||
chapters_per_volume, sparse_chapters_per_volume,
|
||||
0, 0, &config->geometry);
|
||||
if (result != UDS_SUCCESS) {
|
||||
uds_free_configuration(config);
|
||||
return result;
|
||||
}
|
||||
|
||||
config->zone_count = normalize_zone_count(params->zone_count);
|
||||
config->read_threads = normalize_read_threads(params->read_threads);
|
||||
|
||||
config->cache_chapters = DEFAULT_CACHE_CHAPTERS;
|
||||
config->volume_index_mean_delta = DEFAULT_VOLUME_INDEX_MEAN_DELTA;
|
||||
config->sparse_sample_rate = (params->sparse ? DEFAULT_SPARSE_SAMPLE_RATE : 0);
|
||||
config->nonce = params->nonce;
|
||||
config->bdev = params->bdev;
|
||||
config->offset = params->offset;
|
||||
config->size = params->size;
|
||||
|
||||
*config_ptr = config;
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
void uds_free_configuration(struct uds_configuration *config)
|
||||
{
|
||||
if (config != NULL) {
|
||||
uds_free_index_geometry(config->geometry);
|
||||
vdo_free(config);
|
||||
}
|
||||
}
|
||||
|
||||
void uds_log_configuration(struct uds_configuration *config)
|
||||
{
|
||||
struct index_geometry *geometry = config->geometry;
|
||||
|
||||
vdo_log_debug("Configuration:");
|
||||
vdo_log_debug(" Record pages per chapter: %10u", geometry->record_pages_per_chapter);
|
||||
vdo_log_debug(" Chapters per volume: %10u", geometry->chapters_per_volume);
|
||||
vdo_log_debug(" Sparse chapters per volume: %10u", geometry->sparse_chapters_per_volume);
|
||||
vdo_log_debug(" Cache size (chapters): %10u", config->cache_chapters);
|
||||
vdo_log_debug(" Volume index mean delta: %10u", config->volume_index_mean_delta);
|
||||
vdo_log_debug(" Bytes per page: %10zu", geometry->bytes_per_page);
|
||||
vdo_log_debug(" Sparse sample rate: %10u", config->sparse_sample_rate);
|
||||
vdo_log_debug(" Nonce: %llu", (unsigned long long) config->nonce);
|
||||
}
|
|
@ -0,0 +1,124 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef UDS_CONFIG_H
|
||||
#define UDS_CONFIG_H
|
||||
|
||||
#include "geometry.h"
|
||||
#include "indexer.h"
|
||||
#include "io-factory.h"
|
||||
|
||||
/*
|
||||
* The uds_configuration records a variety of parameters used to configure a new UDS index. Some
|
||||
* parameters are provided by the client, while others are fixed or derived from user-supplied
|
||||
* values. It is created when an index is created, and it is recorded in the index metadata.
|
||||
*/
|
||||
|
||||
enum {
|
||||
DEFAULT_VOLUME_INDEX_MEAN_DELTA = 4096,
|
||||
DEFAULT_CACHE_CHAPTERS = 7,
|
||||
DEFAULT_SPARSE_SAMPLE_RATE = 32,
|
||||
MAX_ZONES = 16,
|
||||
};
|
||||
|
||||
/* A set of configuration parameters for the indexer. */
|
||||
struct uds_configuration {
|
||||
/* Storage device for the index */
|
||||
struct block_device *bdev;
|
||||
|
||||
/* The maximum allowable size of the index */
|
||||
size_t size;
|
||||
|
||||
/* The offset where the index should start */
|
||||
off_t offset;
|
||||
|
||||
/* Parameters for the volume */
|
||||
|
||||
/* The volume layout */
|
||||
struct index_geometry *geometry;
|
||||
|
||||
/* Index owner's nonce */
|
||||
u64 nonce;
|
||||
|
||||
/* The number of threads used to process index requests */
|
||||
unsigned int zone_count;
|
||||
|
||||
/* The number of threads used to read volume pages */
|
||||
unsigned int read_threads;
|
||||
|
||||
/* Size of the page cache and sparse chapter index cache in chapters */
|
||||
u32 cache_chapters;
|
||||
|
||||
/* Parameters for the volume index */
|
||||
|
||||
/* The mean delta for the volume index */
|
||||
u32 volume_index_mean_delta;
|
||||
|
||||
/* Sampling rate for sparse indexing */
|
||||
u32 sparse_sample_rate;
|
||||
};
|
||||
|
||||
/* On-disk structure of data for a version 8.02 index. */
|
||||
struct uds_configuration_8_02 {
|
||||
/* Smaller (16), Small (64) or large (256) indices */
|
||||
u32 record_pages_per_chapter;
|
||||
/* Total number of chapters per volume */
|
||||
u32 chapters_per_volume;
|
||||
/* Number of sparse chapters per volume */
|
||||
u32 sparse_chapters_per_volume;
|
||||
/* Size of the page cache, in chapters */
|
||||
u32 cache_chapters;
|
||||
/* Unused field */
|
||||
u32 unused;
|
||||
/* The volume index mean delta to use */
|
||||
u32 volume_index_mean_delta;
|
||||
/* Size of a page, used for both record pages and index pages */
|
||||
u32 bytes_per_page;
|
||||
/* Sampling rate for sparse indexing */
|
||||
u32 sparse_sample_rate;
|
||||
/* Index owner's nonce */
|
||||
u64 nonce;
|
||||
/* Virtual chapter remapped from physical chapter 0 */
|
||||
u64 remapped_virtual;
|
||||
/* New physical chapter which remapped chapter was moved to */
|
||||
u64 remapped_physical;
|
||||
} __packed;
|
||||
|
||||
/* On-disk structure of data for a version 6.02 index. */
|
||||
struct uds_configuration_6_02 {
|
||||
/* Smaller (16), Small (64) or large (256) indices */
|
||||
u32 record_pages_per_chapter;
|
||||
/* Total number of chapters per volume */
|
||||
u32 chapters_per_volume;
|
||||
/* Number of sparse chapters per volume */
|
||||
u32 sparse_chapters_per_volume;
|
||||
/* Size of the page cache, in chapters */
|
||||
u32 cache_chapters;
|
||||
/* Unused field */
|
||||
u32 unused;
|
||||
/* The volume index mean delta to use */
|
||||
u32 volume_index_mean_delta;
|
||||
/* Size of a page, used for both record pages and index pages */
|
||||
u32 bytes_per_page;
|
||||
/* Sampling rate for sparse indexing */
|
||||
u32 sparse_sample_rate;
|
||||
/* Index owner's nonce */
|
||||
u64 nonce;
|
||||
} __packed;
|
||||
|
||||
int __must_check uds_make_configuration(const struct uds_parameters *params,
|
||||
struct uds_configuration **config_ptr);
|
||||
|
||||
void uds_free_configuration(struct uds_configuration *config);
|
||||
|
||||
int __must_check uds_validate_config_contents(struct buffered_reader *reader,
|
||||
struct uds_configuration *config);
|
||||
|
||||
int __must_check uds_write_config_contents(struct buffered_writer *writer,
|
||||
struct uds_configuration *config, u32 version);
|
||||
|
||||
void uds_log_configuration(struct uds_configuration *config);
|
||||
|
||||
#endif /* UDS_CONFIG_H */
|
File diff suppressed because it is too large
Load Diff
|
@ -0,0 +1,279 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef UDS_DELTA_INDEX_H
|
||||
#define UDS_DELTA_INDEX_H
|
||||
|
||||
#include <linux/cache.h>
|
||||
|
||||
#include "numeric.h"
|
||||
#include "time-utils.h"
|
||||
|
||||
#include "config.h"
|
||||
#include "io-factory.h"
|
||||
|
||||
/*
|
||||
* A delta index is a key-value store, where each entry maps an address (the key) to a payload (the
|
||||
* value). The entries are sorted by address, and only the delta between successive addresses is
|
||||
* stored in the entry. The addresses are assumed to be uniformly distributed, and the deltas are
|
||||
* therefore exponentially distributed.
|
||||
*
|
||||
* A delta_index can either be mutable or immutable depending on its expected use. The immutable
|
||||
* form of a delta index is used for the indexes of closed chapters committed to the volume. The
|
||||
* mutable form of a delta index is used by the volume index, and also by the chapter index in an
|
||||
* open chapter. Like the index as a whole, each mutable delta index is divided into a number of
|
||||
* independent zones.
|
||||
*/
|
||||
|
||||
struct delta_list {
|
||||
/* The offset of the delta list start, in bits */
|
||||
u64 start;
|
||||
/* The number of bits in the delta list */
|
||||
u16 size;
|
||||
/* Where the last search "found" the key, in bits */
|
||||
u16 save_offset;
|
||||
/* The key for the record just before save_offset */
|
||||
u32 save_key;
|
||||
};
|
||||
|
||||
struct delta_zone {
|
||||
/* The delta list memory */
|
||||
u8 *memory;
|
||||
/* The delta list headers */
|
||||
struct delta_list *delta_lists;
|
||||
/* Temporary starts of delta lists */
|
||||
u64 *new_offsets;
|
||||
/* Buffered writer for saving an index */
|
||||
struct buffered_writer *buffered_writer;
|
||||
/* The size of delta list memory */
|
||||
size_t size;
|
||||
/* Nanoseconds spent rebalancing */
|
||||
ktime_t rebalance_time;
|
||||
/* Number of memory rebalances */
|
||||
u32 rebalance_count;
|
||||
/* The number of bits in a stored value */
|
||||
u8 value_bits;
|
||||
/* The number of bits in the minimal key code */
|
||||
u16 min_bits;
|
||||
/* The number of keys used in a minimal code */
|
||||
u32 min_keys;
|
||||
/* The number of keys used for another code bit */
|
||||
u32 incr_keys;
|
||||
/* The number of records in the index */
|
||||
u64 record_count;
|
||||
/* The number of collision records */
|
||||
u64 collision_count;
|
||||
/* The number of records removed */
|
||||
u64 discard_count;
|
||||
/* The number of UDS_OVERFLOW errors detected */
|
||||
u64 overflow_count;
|
||||
/* The index of the first delta list */
|
||||
u32 first_list;
|
||||
/* The number of delta lists */
|
||||
u32 list_count;
|
||||
/* Tag belonging to this delta index */
|
||||
u8 tag;
|
||||
} __aligned(L1_CACHE_BYTES);
|
||||
|
||||
struct delta_list_save_info {
|
||||
/* Tag identifying which delta index this list is in */
|
||||
u8 tag;
|
||||
/* Bit offset of the start of the list data */
|
||||
u8 bit_offset;
|
||||
/* Number of bytes of list data */
|
||||
u16 byte_count;
|
||||
/* The delta list number within the delta index */
|
||||
u32 index;
|
||||
} __packed;
|
||||
|
||||
struct delta_index {
|
||||
/* The zones */
|
||||
struct delta_zone *delta_zones;
|
||||
/* The number of zones */
|
||||
unsigned int zone_count;
|
||||
/* The number of delta lists */
|
||||
u32 list_count;
|
||||
/* Maximum lists per zone */
|
||||
u32 lists_per_zone;
|
||||
/* Total memory allocated to this index */
|
||||
size_t memory_size;
|
||||
/* The number of non-empty lists at load time per zone */
|
||||
u32 load_lists[MAX_ZONES];
|
||||
/* True if this index is mutable */
|
||||
bool mutable;
|
||||
/* Tag belonging to this delta index */
|
||||
u8 tag;
|
||||
};
|
||||
|
||||
/*
|
||||
* A delta_index_page describes a single page of a chapter index. The delta_index field allows the
|
||||
* page to be treated as an immutable delta_index. We use the delta_zone field to treat the chapter
|
||||
* index page as a single zone index, and without the need to do an additional memory allocation.
|
||||
*/
|
||||
struct delta_index_page {
|
||||
struct delta_index delta_index;
|
||||
/* These values are loaded from the delta_page_header */
|
||||
u32 lowest_list_number;
|
||||
u32 highest_list_number;
|
||||
u64 virtual_chapter_number;
|
||||
/* This structure describes the single zone of a delta index page. */
|
||||
struct delta_zone delta_zone;
|
||||
};
|
||||
|
||||
/*
|
||||
* Notes on the delta_index_entries:
|
||||
*
|
||||
* The fields documented as "public" can be read by any code that uses a delta_index. The fields
|
||||
* documented as "private" carry information between delta_index method calls and should not be
|
||||
* used outside the delta_index module.
|
||||
*
|
||||
* (1) The delta_index_entry is used like an iterator when searching a delta list.
|
||||
*
|
||||
* (2) It is also the result of a successful search and can be used to refer to the element found
|
||||
* by the search.
|
||||
*
|
||||
* (3) It is also the result of an unsuccessful search and can be used to refer to the insertion
|
||||
* point for a new record.
|
||||
*
|
||||
* (4) If at_end is true, the delta_list entry can only be used as the insertion point for a new
|
||||
* record at the end of the list.
|
||||
*
|
||||
* (5) If at_end is false and is_collision is true, the delta_list entry fields refer to a
|
||||
* collision entry in the list, and the delta_list entry can be used as a reference to this
|
||||
* entry.
|
||||
*
|
||||
* (6) If at_end is false and is_collision is false, the delta_list entry fields refer to a
|
||||
* non-collision entry in the list. Such delta_list entries can be used as a reference to a
|
||||
* found entry, or an insertion point for a non-collision entry before this entry, or an
|
||||
* insertion point for a collision entry that collides with this entry.
|
||||
*/
|
||||
struct delta_index_entry {
|
||||
/* Public fields */
|
||||
/* The key for this entry */
|
||||
u32 key;
|
||||
/* We are after the last list entry */
|
||||
bool at_end;
|
||||
/* This record is a collision */
|
||||
bool is_collision;
|
||||
|
||||
/* Private fields */
|
||||
/* This delta list overflowed */
|
||||
bool list_overflow;
|
||||
/* The number of bits used for the value */
|
||||
u8 value_bits;
|
||||
/* The number of bits used for the entire entry */
|
||||
u16 entry_bits;
|
||||
/* The delta index zone */
|
||||
struct delta_zone *delta_zone;
|
||||
/* The delta list containing the entry */
|
||||
struct delta_list *delta_list;
|
||||
/* The delta list number */
|
||||
u32 list_number;
|
||||
/* Bit offset of this entry within the list */
|
||||
u16 offset;
|
||||
/* The delta between this and previous entry */
|
||||
u32 delta;
|
||||
/* Temporary delta list for immutable indices */
|
||||
struct delta_list temp_delta_list;
|
||||
};
|
||||
|
||||
struct delta_index_stats {
|
||||
/* Number of bytes allocated */
|
||||
size_t memory_allocated;
|
||||
/* Nanoseconds spent rebalancing */
|
||||
ktime_t rebalance_time;
|
||||
/* Number of memory rebalances */
|
||||
u32 rebalance_count;
|
||||
/* The number of records in the index */
|
||||
u64 record_count;
|
||||
/* The number of collision records */
|
||||
u64 collision_count;
|
||||
/* The number of records removed */
|
||||
u64 discard_count;
|
||||
/* The number of UDS_OVERFLOW errors detected */
|
||||
u64 overflow_count;
|
||||
/* The number of delta lists */
|
||||
u32 list_count;
|
||||
};
|
||||
|
||||
int __must_check uds_initialize_delta_index(struct delta_index *delta_index,
|
||||
unsigned int zone_count, u32 list_count,
|
||||
u32 mean_delta, u32 payload_bits,
|
||||
size_t memory_size, u8 tag);
|
||||
|
||||
int __must_check uds_initialize_delta_index_page(struct delta_index_page *delta_index_page,
|
||||
u64 expected_nonce, u32 mean_delta,
|
||||
u32 payload_bits, u8 *memory,
|
||||
size_t memory_size);
|
||||
|
||||
void uds_uninitialize_delta_index(struct delta_index *delta_index);
|
||||
|
||||
void uds_reset_delta_index(const struct delta_index *delta_index);
|
||||
|
||||
int __must_check uds_pack_delta_index_page(const struct delta_index *delta_index,
|
||||
u64 header_nonce, u8 *memory,
|
||||
size_t memory_size,
|
||||
u64 virtual_chapter_number, u32 first_list,
|
||||
u32 *list_count);
|
||||
|
||||
int __must_check uds_start_restoring_delta_index(struct delta_index *delta_index,
|
||||
struct buffered_reader **buffered_readers,
|
||||
unsigned int reader_count);
|
||||
|
||||
int __must_check uds_finish_restoring_delta_index(struct delta_index *delta_index,
|
||||
struct buffered_reader **buffered_readers,
|
||||
unsigned int reader_count);
|
||||
|
||||
int __must_check uds_check_guard_delta_lists(struct buffered_reader **buffered_readers,
|
||||
unsigned int reader_count);
|
||||
|
||||
int __must_check uds_start_saving_delta_index(const struct delta_index *delta_index,
|
||||
unsigned int zone_number,
|
||||
struct buffered_writer *buffered_writer);
|
||||
|
||||
int __must_check uds_finish_saving_delta_index(const struct delta_index *delta_index,
|
||||
unsigned int zone_number);
|
||||
|
||||
int __must_check uds_write_guard_delta_list(struct buffered_writer *buffered_writer);
|
||||
|
||||
size_t __must_check uds_compute_delta_index_save_bytes(u32 list_count,
|
||||
size_t memory_size);
|
||||
|
||||
int __must_check uds_start_delta_index_search(const struct delta_index *delta_index,
|
||||
u32 list_number, u32 key,
|
||||
struct delta_index_entry *iterator);
|
||||
|
||||
int __must_check uds_next_delta_index_entry(struct delta_index_entry *delta_entry);
|
||||
|
||||
int __must_check uds_remember_delta_index_offset(const struct delta_index_entry *delta_entry);
|
||||
|
||||
int __must_check uds_get_delta_index_entry(const struct delta_index *delta_index,
|
||||
u32 list_number, u32 key, const u8 *name,
|
||||
struct delta_index_entry *delta_entry);
|
||||
|
||||
int __must_check uds_get_delta_entry_collision(const struct delta_index_entry *delta_entry,
|
||||
u8 *name);
|
||||
|
||||
u32 __must_check uds_get_delta_entry_value(const struct delta_index_entry *delta_entry);
|
||||
|
||||
int __must_check uds_set_delta_entry_value(const struct delta_index_entry *delta_entry, u32 value);
|
||||
|
||||
int __must_check uds_put_delta_index_entry(struct delta_index_entry *delta_entry, u32 key,
|
||||
u32 value, const u8 *name);
|
||||
|
||||
int __must_check uds_remove_delta_index_entry(struct delta_index_entry *delta_entry);
|
||||
|
||||
void uds_get_delta_index_stats(const struct delta_index *delta_index,
|
||||
struct delta_index_stats *stats);
|
||||
|
||||
size_t __must_check uds_compute_delta_index_size(u32 entry_count, u32 mean_delta,
|
||||
u32 payload_bits);
|
||||
|
||||
u32 uds_get_delta_index_page_count(u32 entry_count, u32 list_count, u32 mean_delta,
|
||||
u32 payload_bits, size_t bytes_per_page);
|
||||
|
||||
void uds_log_delta_index_entry(struct delta_index_entry *delta_entry);
|
||||
|
||||
#endif /* UDS_DELTA_INDEX_H */
|
|
@ -0,0 +1,279 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "funnel-requestqueue.h"
|
||||
|
||||
#include <linux/atomic.h>
|
||||
#include <linux/compiler.h>
|
||||
#include <linux/wait.h>
|
||||
|
||||
#include "funnel-queue.h"
|
||||
#include "logger.h"
|
||||
#include "memory-alloc.h"
|
||||
#include "thread-utils.h"
|
||||
|
||||
/*
|
||||
* This queue will attempt to handle requests in reasonably sized batches instead of reacting
|
||||
* immediately to each new request. The wait time between batches is dynamically adjusted up or
|
||||
* down to try to balance responsiveness against wasted thread run time.
|
||||
*
|
||||
* If the wait time becomes long enough, the queue will become dormant and must be explicitly
|
||||
* awoken when a new request is enqueued. The enqueue operation updates "newest" in the funnel
|
||||
* queue via xchg (which is a memory barrier), and later checks "dormant" to decide whether to do a
|
||||
* wakeup of the worker thread.
|
||||
*
|
||||
* When deciding to go to sleep, the worker thread sets "dormant" and then examines "newest" to
|
||||
* decide if the funnel queue is idle. In dormant mode, the last examination of "newest" before
|
||||
* going to sleep is done inside the wait_event_interruptible() macro, after a point where one or
|
||||
* more memory barriers have been issued. (Preparing to sleep uses spin locks.) Even if the funnel
|
||||
* queue's "next" field update isn't visible yet to make the entry accessible, its existence will
|
||||
* kick the worker thread out of dormant mode and back into timer-based mode.
|
||||
*
|
||||
* Unbatched requests are used to communicate between different zone threads and will also cause
|
||||
* the queue to awaken immediately.
|
||||
*/
|
||||
|
||||
enum {
|
||||
NANOSECOND = 1,
|
||||
MICROSECOND = 1000 * NANOSECOND,
|
||||
MILLISECOND = 1000 * MICROSECOND,
|
||||
DEFAULT_WAIT_TIME = 20 * MICROSECOND,
|
||||
MINIMUM_WAIT_TIME = DEFAULT_WAIT_TIME / 2,
|
||||
MAXIMUM_WAIT_TIME = MILLISECOND,
|
||||
MINIMUM_BATCH = 32,
|
||||
MAXIMUM_BATCH = 64,
|
||||
};
|
||||
|
||||
struct uds_request_queue {
|
||||
/* Wait queue for synchronizing producers and consumer */
|
||||
struct wait_queue_head wait_head;
|
||||
/* Function to process a request */
|
||||
uds_request_queue_processor_fn processor;
|
||||
/* Queue of new incoming requests */
|
||||
struct funnel_queue *main_queue;
|
||||
/* Queue of old requests to retry */
|
||||
struct funnel_queue *retry_queue;
|
||||
/* The thread id of the worker thread */
|
||||
struct thread *thread;
|
||||
/* True if the worker was started */
|
||||
bool started;
|
||||
/* When true, requests can be enqueued */
|
||||
bool running;
|
||||
/* A flag set when the worker is waiting without a timeout */
|
||||
atomic_t dormant;
|
||||
};
|
||||
|
||||
static inline struct uds_request *poll_queues(struct uds_request_queue *queue)
|
||||
{
|
||||
struct funnel_queue_entry *entry;
|
||||
|
||||
entry = vdo_funnel_queue_poll(queue->retry_queue);
|
||||
if (entry != NULL)
|
||||
return container_of(entry, struct uds_request, queue_link);
|
||||
|
||||
entry = vdo_funnel_queue_poll(queue->main_queue);
|
||||
if (entry != NULL)
|
||||
return container_of(entry, struct uds_request, queue_link);
|
||||
|
||||
return NULL;
|
||||
}
|
||||
|
||||
static inline bool are_queues_idle(struct uds_request_queue *queue)
|
||||
{
|
||||
return vdo_is_funnel_queue_idle(queue->retry_queue) &&
|
||||
vdo_is_funnel_queue_idle(queue->main_queue);
|
||||
}
|
||||
|
||||
/*
|
||||
* Determine if there is a next request to process, and return it if there is. Also return flags
|
||||
* indicating whether the worker thread can sleep (for the use of wait_event() macros) and whether
|
||||
* the thread did sleep before returning a new request.
|
||||
*/
|
||||
static inline bool dequeue_request(struct uds_request_queue *queue,
|
||||
struct uds_request **request_ptr, bool *waited_ptr)
|
||||
{
|
||||
struct uds_request *request = poll_queues(queue);
|
||||
|
||||
if (request != NULL) {
|
||||
*request_ptr = request;
|
||||
return true;
|
||||
}
|
||||
|
||||
if (!READ_ONCE(queue->running)) {
|
||||
/* Wake the worker thread so it can exit. */
|
||||
*request_ptr = NULL;
|
||||
return true;
|
||||
}
|
||||
|
||||
*request_ptr = NULL;
|
||||
*waited_ptr = true;
|
||||
return false;
|
||||
}
|
||||
|
||||
static void wait_for_request(struct uds_request_queue *queue, bool dormant,
|
||||
unsigned long timeout, struct uds_request **request,
|
||||
bool *waited)
|
||||
{
|
||||
if (dormant) {
|
||||
wait_event_interruptible(queue->wait_head,
|
||||
(dequeue_request(queue, request, waited) ||
|
||||
!are_queues_idle(queue)));
|
||||
return;
|
||||
}
|
||||
|
||||
wait_event_interruptible_hrtimeout(queue->wait_head,
|
||||
dequeue_request(queue, request, waited),
|
||||
ns_to_ktime(timeout));
|
||||
}
|
||||
|
||||
static void request_queue_worker(void *arg)
|
||||
{
|
||||
struct uds_request_queue *queue = arg;
|
||||
struct uds_request *request = NULL;
|
||||
unsigned long time_batch = DEFAULT_WAIT_TIME;
|
||||
bool dormant = atomic_read(&queue->dormant);
|
||||
bool waited = false;
|
||||
long current_batch = 0;
|
||||
|
||||
for (;;) {
|
||||
wait_for_request(queue, dormant, time_batch, &request, &waited);
|
||||
if (likely(request != NULL)) {
|
||||
current_batch++;
|
||||
queue->processor(request);
|
||||
} else if (!READ_ONCE(queue->running)) {
|
||||
break;
|
||||
}
|
||||
|
||||
if (dormant) {
|
||||
/*
|
||||
* The queue has been roused from dormancy. Clear the flag so enqueuers can
|
||||
* stop broadcasting. No fence is needed for this transition.
|
||||
*/
|
||||
atomic_set(&queue->dormant, false);
|
||||
dormant = false;
|
||||
time_batch = DEFAULT_WAIT_TIME;
|
||||
} else if (waited) {
|
||||
/*
|
||||
* We waited for this request to show up. Adjust the wait time to smooth
|
||||
* out the batch size.
|
||||
*/
|
||||
if (current_batch < MINIMUM_BATCH) {
|
||||
/*
|
||||
* If the last batch of requests was too small, increase the wait
|
||||
* time.
|
||||
*/
|
||||
time_batch += time_batch / 4;
|
||||
if (time_batch >= MAXIMUM_WAIT_TIME) {
|
||||
atomic_set(&queue->dormant, true);
|
||||
dormant = true;
|
||||
}
|
||||
} else if (current_batch > MAXIMUM_BATCH) {
|
||||
/*
|
||||
* If the last batch of requests was too large, decrease the wait
|
||||
* time.
|
||||
*/
|
||||
time_batch -= time_batch / 4;
|
||||
if (time_batch < MINIMUM_WAIT_TIME)
|
||||
time_batch = MINIMUM_WAIT_TIME;
|
||||
}
|
||||
current_batch = 0;
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
* Ensure that we process any remaining requests that were enqueued before trying to shut
|
||||
* down. The corresponding write barrier is in uds_request_queue_finish().
|
||||
*/
|
||||
smp_rmb();
|
||||
while ((request = poll_queues(queue)) != NULL)
|
||||
queue->processor(request);
|
||||
}
|
||||
|
||||
int uds_make_request_queue(const char *queue_name,
|
||||
uds_request_queue_processor_fn processor,
|
||||
struct uds_request_queue **queue_ptr)
|
||||
{
|
||||
int result;
|
||||
struct uds_request_queue *queue;
|
||||
|
||||
result = vdo_allocate(1, struct uds_request_queue, __func__, &queue);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
queue->processor = processor;
|
||||
queue->running = true;
|
||||
atomic_set(&queue->dormant, false);
|
||||
init_waitqueue_head(&queue->wait_head);
|
||||
|
||||
result = vdo_make_funnel_queue(&queue->main_queue);
|
||||
if (result != VDO_SUCCESS) {
|
||||
uds_request_queue_finish(queue);
|
||||
return result;
|
||||
}
|
||||
|
||||
result = vdo_make_funnel_queue(&queue->retry_queue);
|
||||
if (result != VDO_SUCCESS) {
|
||||
uds_request_queue_finish(queue);
|
||||
return result;
|
||||
}
|
||||
|
||||
result = vdo_create_thread(request_queue_worker, queue, queue_name,
|
||||
&queue->thread);
|
||||
if (result != VDO_SUCCESS) {
|
||||
uds_request_queue_finish(queue);
|
||||
return result;
|
||||
}
|
||||
|
||||
queue->started = true;
|
||||
*queue_ptr = queue;
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
static inline void wake_up_worker(struct uds_request_queue *queue)
|
||||
{
|
||||
if (wq_has_sleeper(&queue->wait_head))
|
||||
wake_up(&queue->wait_head);
|
||||
}
|
||||
|
||||
void uds_request_queue_enqueue(struct uds_request_queue *queue,
|
||||
struct uds_request *request)
|
||||
{
|
||||
struct funnel_queue *sub_queue;
|
||||
bool unbatched = request->unbatched;
|
||||
|
||||
sub_queue = request->requeued ? queue->retry_queue : queue->main_queue;
|
||||
vdo_funnel_queue_put(sub_queue, &request->queue_link);
|
||||
|
||||
/*
|
||||
* We must wake the worker thread when it is dormant. A read fence isn't needed here since
|
||||
* we know the queue operation acts as one.
|
||||
*/
|
||||
if (atomic_read(&queue->dormant) || unbatched)
|
||||
wake_up_worker(queue);
|
||||
}
|
||||
|
||||
void uds_request_queue_finish(struct uds_request_queue *queue)
|
||||
{
|
||||
if (queue == NULL)
|
||||
return;
|
||||
|
||||
/*
|
||||
* This memory barrier ensures that any requests we queued will be seen. The point is that
|
||||
* when dequeue_request() sees the following update to the running flag, it will also be
|
||||
* able to see any change we made to a next field in the funnel queue entry. The
|
||||
* corresponding read barrier is in request_queue_worker().
|
||||
*/
|
||||
smp_wmb();
|
||||
WRITE_ONCE(queue->running, false);
|
||||
|
||||
if (queue->started) {
|
||||
wake_up_worker(queue);
|
||||
vdo_join_threads(queue->thread);
|
||||
}
|
||||
|
||||
vdo_free_funnel_queue(queue->main_queue);
|
||||
vdo_free_funnel_queue(queue->retry_queue);
|
||||
vdo_free(queue);
|
||||
}
|
|
@ -0,0 +1,31 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef UDS_REQUEST_QUEUE_H
|
||||
#define UDS_REQUEST_QUEUE_H
|
||||
|
||||
#include "indexer.h"
|
||||
|
||||
/*
|
||||
* A simple request queue which will handle new requests in the order in which they are received,
|
||||
* and will attempt to handle requeued requests before new ones. However, the nature of the
|
||||
* implementation means that it cannot guarantee this ordering; the prioritization is merely a
|
||||
* hint.
|
||||
*/
|
||||
|
||||
struct uds_request_queue;
|
||||
|
||||
typedef void (*uds_request_queue_processor_fn)(struct uds_request *);
|
||||
|
||||
int __must_check uds_make_request_queue(const char *queue_name,
|
||||
uds_request_queue_processor_fn processor,
|
||||
struct uds_request_queue **queue_ptr);
|
||||
|
||||
void uds_request_queue_enqueue(struct uds_request_queue *queue,
|
||||
struct uds_request *request);
|
||||
|
||||
void uds_request_queue_finish(struct uds_request_queue *queue);
|
||||
|
||||
#endif /* UDS_REQUEST_QUEUE_H */
|
|
@ -0,0 +1,201 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "geometry.h"
|
||||
|
||||
#include <linux/compiler.h>
|
||||
#include <linux/log2.h>
|
||||
|
||||
#include "errors.h"
|
||||
#include "logger.h"
|
||||
#include "memory-alloc.h"
|
||||
#include "permassert.h"
|
||||
|
||||
#include "delta-index.h"
|
||||
#include "indexer.h"
|
||||
|
||||
/*
|
||||
* An index volume is divided into a fixed number of fixed-size chapters, each consisting of a
|
||||
* fixed number of fixed-size pages. The volume layout is defined by two constants and four
|
||||
* parameters. The constants are that index records are 32 bytes long (16-byte block name plus
|
||||
* 16-byte metadata) and that open chapter index hash slots are one byte long. The four parameters
|
||||
* are the number of bytes in a page, the number of record pages in a chapter, the number of
|
||||
* chapters in a volume, and the number of chapters that are sparse. From these parameters, we can
|
||||
* derive the rest of the layout and other index properties.
|
||||
*
|
||||
* The index volume is sized by its maximum memory footprint. For a dense index, the persistent
|
||||
* storage is about 10 times the size of the memory footprint. For a sparse index, the persistent
|
||||
* storage is about 100 times the size of the memory footprint.
|
||||
*
|
||||
* For a small index with a memory footprint less than 1GB, there are three possible memory
|
||||
* configurations: 0.25GB, 0.5GB and 0.75GB. The default geometry for each is 1024 index records
|
||||
* per 32 KB page, 1024 chapters per volume, and either 64, 128, or 192 record pages per chapter
|
||||
* (resulting in 6, 13, or 20 index pages per chapter) depending on the memory configuration. For
|
||||
* the VDO default of a 0.25 GB index, this yields a deduplication window of 256 GB using about 2.5
|
||||
* GB for the persistent storage and 256 MB of RAM.
|
||||
*
|
||||
* For a larger index with a memory footprint that is a multiple of 1 GB, the geometry is 1024
|
||||
* index records per 32 KB page, 256 record pages per chapter, 26 index pages per chapter, and 1024
|
||||
* chapters for every GB of memory footprint. For a 1 GB volume, this yields a deduplication window
|
||||
* of 1 TB using about 9GB of persistent storage and 1 GB of RAM.
|
||||
*
|
||||
* The above numbers hold for volumes which have no sparse chapters. A sparse volume has 10 times
|
||||
* as many chapters as the corresponding non-sparse volume, which provides 10 times the
|
||||
* deduplication window while using 10 times as much persistent storage as the equivalent
|
||||
* non-sparse volume with the same memory footprint.
|
||||
*
|
||||
* If the volume has been converted from a non-lvm format to an lvm volume, the number of chapters
|
||||
* per volume will have been reduced by one by eliminating physical chapter 0, and the virtual
|
||||
* chapter that formerly mapped to physical chapter 0 may be remapped to another physical chapter.
|
||||
* This remapping is expressed by storing which virtual chapter was remapped, and which physical
|
||||
* chapter it was moved to.
|
||||
*/
|
||||
|
||||
int uds_make_index_geometry(size_t bytes_per_page, u32 record_pages_per_chapter,
|
||||
u32 chapters_per_volume, u32 sparse_chapters_per_volume,
|
||||
u64 remapped_virtual, u64 remapped_physical,
|
||||
struct index_geometry **geometry_ptr)
|
||||
{
|
||||
int result;
|
||||
struct index_geometry *geometry;
|
||||
|
||||
result = vdo_allocate(1, struct index_geometry, "geometry", &geometry);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
geometry->bytes_per_page = bytes_per_page;
|
||||
geometry->record_pages_per_chapter = record_pages_per_chapter;
|
||||
geometry->chapters_per_volume = chapters_per_volume;
|
||||
geometry->sparse_chapters_per_volume = sparse_chapters_per_volume;
|
||||
geometry->dense_chapters_per_volume = chapters_per_volume - sparse_chapters_per_volume;
|
||||
geometry->remapped_virtual = remapped_virtual;
|
||||
geometry->remapped_physical = remapped_physical;
|
||||
|
||||
geometry->records_per_page = bytes_per_page / BYTES_PER_RECORD;
|
||||
geometry->records_per_chapter = geometry->records_per_page * record_pages_per_chapter;
|
||||
geometry->records_per_volume = (u64) geometry->records_per_chapter * chapters_per_volume;
|
||||
|
||||
geometry->chapter_mean_delta = 1 << DEFAULT_CHAPTER_MEAN_DELTA_BITS;
|
||||
geometry->chapter_payload_bits = bits_per(record_pages_per_chapter - 1);
|
||||
/*
|
||||
* We want 1 delta list for every 64 records in the chapter.
|
||||
* The "| 077" ensures that the chapter_delta_list_bits computation
|
||||
* does not underflow.
|
||||
*/
|
||||
geometry->chapter_delta_list_bits =
|
||||
bits_per((geometry->records_per_chapter - 1) | 077) - 6;
|
||||
geometry->delta_lists_per_chapter = 1 << geometry->chapter_delta_list_bits;
|
||||
/* We need enough address bits to achieve the desired mean delta. */
|
||||
geometry->chapter_address_bits =
|
||||
(DEFAULT_CHAPTER_MEAN_DELTA_BITS -
|
||||
geometry->chapter_delta_list_bits +
|
||||
bits_per(geometry->records_per_chapter - 1));
|
||||
geometry->index_pages_per_chapter =
|
||||
uds_get_delta_index_page_count(geometry->records_per_chapter,
|
||||
geometry->delta_lists_per_chapter,
|
||||
geometry->chapter_mean_delta,
|
||||
geometry->chapter_payload_bits,
|
||||
bytes_per_page);
|
||||
|
||||
geometry->pages_per_chapter = geometry->index_pages_per_chapter + record_pages_per_chapter;
|
||||
geometry->pages_per_volume = geometry->pages_per_chapter * chapters_per_volume;
|
||||
geometry->bytes_per_volume =
|
||||
bytes_per_page * (geometry->pages_per_volume + HEADER_PAGES_PER_VOLUME);
|
||||
|
||||
*geometry_ptr = geometry;
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
int uds_copy_index_geometry(struct index_geometry *source,
|
||||
struct index_geometry **geometry_ptr)
|
||||
{
|
||||
return uds_make_index_geometry(source->bytes_per_page,
|
||||
source->record_pages_per_chapter,
|
||||
source->chapters_per_volume,
|
||||
source->sparse_chapters_per_volume,
|
||||
source->remapped_virtual, source->remapped_physical,
|
||||
geometry_ptr);
|
||||
}
|
||||
|
||||
void uds_free_index_geometry(struct index_geometry *geometry)
|
||||
{
|
||||
vdo_free(geometry);
|
||||
}
|
||||
|
||||
u32 __must_check uds_map_to_physical_chapter(const struct index_geometry *geometry,
|
||||
u64 virtual_chapter)
|
||||
{
|
||||
u64 delta;
|
||||
|
||||
if (!uds_is_reduced_index_geometry(geometry))
|
||||
return virtual_chapter % geometry->chapters_per_volume;
|
||||
|
||||
if (likely(virtual_chapter > geometry->remapped_virtual)) {
|
||||
delta = virtual_chapter - geometry->remapped_virtual;
|
||||
if (likely(delta > geometry->remapped_physical))
|
||||
return delta % geometry->chapters_per_volume;
|
||||
else
|
||||
return delta - 1;
|
||||
}
|
||||
|
||||
if (virtual_chapter == geometry->remapped_virtual)
|
||||
return geometry->remapped_physical;
|
||||
|
||||
delta = geometry->remapped_virtual - virtual_chapter;
|
||||
if (delta < geometry->chapters_per_volume)
|
||||
return geometry->chapters_per_volume - delta;
|
||||
|
||||
/* This chapter is so old the answer doesn't matter. */
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Check whether any sparse chapters are in use. */
|
||||
bool uds_has_sparse_chapters(const struct index_geometry *geometry,
|
||||
u64 oldest_virtual_chapter, u64 newest_virtual_chapter)
|
||||
{
|
||||
return uds_is_sparse_index_geometry(geometry) &&
|
||||
((newest_virtual_chapter - oldest_virtual_chapter + 1) >
|
||||
geometry->dense_chapters_per_volume);
|
||||
}
|
||||
|
||||
bool uds_is_chapter_sparse(const struct index_geometry *geometry,
|
||||
u64 oldest_virtual_chapter, u64 newest_virtual_chapter,
|
||||
u64 virtual_chapter_number)
|
||||
{
|
||||
return uds_has_sparse_chapters(geometry, oldest_virtual_chapter,
|
||||
newest_virtual_chapter) &&
|
||||
((virtual_chapter_number + geometry->dense_chapters_per_volume) <=
|
||||
newest_virtual_chapter);
|
||||
}
|
||||
|
||||
/* Calculate how many chapters to expire after opening the newest chapter. */
|
||||
u32 uds_chapters_to_expire(const struct index_geometry *geometry, u64 newest_chapter)
|
||||
{
|
||||
/* If the index isn't full yet, don't expire anything. */
|
||||
if (newest_chapter < geometry->chapters_per_volume)
|
||||
return 0;
|
||||
|
||||
/* If a chapter is out of order... */
|
||||
if (geometry->remapped_physical > 0) {
|
||||
u64 oldest_chapter = newest_chapter - geometry->chapters_per_volume;
|
||||
|
||||
/*
|
||||
* ... expire an extra chapter when expiring the moved chapter to free physical
|
||||
* space for the new chapter ...
|
||||
*/
|
||||
if (oldest_chapter == geometry->remapped_virtual)
|
||||
return 2;
|
||||
|
||||
/*
|
||||
* ... but don't expire anything when the new chapter will use the physical chapter
|
||||
* freed by expiring the moved chapter.
|
||||
*/
|
||||
if (oldest_chapter == (geometry->remapped_virtual + geometry->remapped_physical))
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Normally, just expire one. */
|
||||
return 1;
|
||||
}
|
|
@ -0,0 +1,140 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef UDS_INDEX_GEOMETRY_H
|
||||
#define UDS_INDEX_GEOMETRY_H
|
||||
|
||||
#include "indexer.h"
|
||||
|
||||
/*
|
||||
* The index_geometry records parameters that define the layout of a UDS index volume, and the size and
|
||||
* shape of various index structures. It is created when the index is created, and is referenced by
|
||||
* many index sub-components.
|
||||
*/
|
||||
|
||||
struct index_geometry {
|
||||
/* Size of a chapter page, in bytes */
|
||||
size_t bytes_per_page;
|
||||
/* Number of record pages in a chapter */
|
||||
u32 record_pages_per_chapter;
|
||||
/* Total number of chapters in a volume */
|
||||
u32 chapters_per_volume;
|
||||
/* Number of sparsely-indexed chapters in a volume */
|
||||
u32 sparse_chapters_per_volume;
|
||||
/* Number of bits used to determine delta list numbers */
|
||||
u8 chapter_delta_list_bits;
|
||||
/* Virtual chapter remapped from physical chapter 0 */
|
||||
u64 remapped_virtual;
|
||||
/* New physical chapter where the remapped chapter can be found */
|
||||
u64 remapped_physical;
|
||||
|
||||
/*
|
||||
* The following properties are derived from the ones above, but they are computed and
|
||||
* recorded as fields for convenience.
|
||||
*/
|
||||
/* Total number of pages in a volume, excluding the header */
|
||||
u32 pages_per_volume;
|
||||
/* Total number of bytes in a volume, including the header */
|
||||
size_t bytes_per_volume;
|
||||
/* Number of pages in a chapter */
|
||||
u32 pages_per_chapter;
|
||||
/* Number of index pages in a chapter index */
|
||||
u32 index_pages_per_chapter;
|
||||
/* Number of records that fit on a page */
|
||||
u32 records_per_page;
|
||||
/* Number of records that fit in a chapter */
|
||||
u32 records_per_chapter;
|
||||
/* Number of records that fit in a volume */
|
||||
u64 records_per_volume;
|
||||
/* Number of delta lists per chapter index */
|
||||
u32 delta_lists_per_chapter;
|
||||
/* Mean delta for chapter indexes */
|
||||
u32 chapter_mean_delta;
|
||||
/* Number of bits needed for record page numbers */
|
||||
u8 chapter_payload_bits;
|
||||
/* Number of bits used to compute addresses for chapter delta lists */
|
||||
u8 chapter_address_bits;
|
||||
/* Number of densely-indexed chapters in a volume */
|
||||
u32 dense_chapters_per_volume;
|
||||
};
|
||||
|
||||
enum {
|
||||
/* The number of bytes in a record (name + metadata) */
|
||||
BYTES_PER_RECORD = (UDS_RECORD_NAME_SIZE + UDS_RECORD_DATA_SIZE),
|
||||
|
||||
/* The default length of a page in a chapter, in bytes */
|
||||
DEFAULT_BYTES_PER_PAGE = 1024 * BYTES_PER_RECORD,
|
||||
|
||||
/* The default maximum number of records per page */
|
||||
DEFAULT_RECORDS_PER_PAGE = DEFAULT_BYTES_PER_PAGE / BYTES_PER_RECORD,
|
||||
|
||||
/* The default number of record pages in a chapter */
|
||||
DEFAULT_RECORD_PAGES_PER_CHAPTER = 256,
|
||||
|
||||
/* The default number of record pages in a chapter for a small index */
|
||||
SMALL_RECORD_PAGES_PER_CHAPTER = 64,
|
||||
|
||||
/* The default number of chapters in a volume */
|
||||
DEFAULT_CHAPTERS_PER_VOLUME = 1024,
|
||||
|
||||
/* The default number of sparsely-indexed chapters in a volume */
|
||||
DEFAULT_SPARSE_CHAPTERS_PER_VOLUME = 0,
|
||||
|
||||
/* The log2 of the default mean delta */
|
||||
DEFAULT_CHAPTER_MEAN_DELTA_BITS = 16,
|
||||
|
||||
/* The log2 of the number of delta lists in a large chapter */
|
||||
DEFAULT_CHAPTER_DELTA_LIST_BITS = 12,
|
||||
|
||||
/* The log2 of the number of delta lists in a small chapter */
|
||||
SMALL_CHAPTER_DELTA_LIST_BITS = 10,
|
||||
|
||||
/* The number of header pages per volume */
|
||||
HEADER_PAGES_PER_VOLUME = 1,
|
||||
};
|
||||
|
||||
int __must_check uds_make_index_geometry(size_t bytes_per_page, u32 record_pages_per_chapter,
|
||||
u32 chapters_per_volume,
|
||||
u32 sparse_chapters_per_volume, u64 remapped_virtual,
|
||||
u64 remapped_physical,
|
||||
struct index_geometry **geometry_ptr);
|
||||
|
||||
int __must_check uds_copy_index_geometry(struct index_geometry *source,
|
||||
struct index_geometry **geometry_ptr);
|
||||
|
||||
void uds_free_index_geometry(struct index_geometry *geometry);
|
||||
|
||||
u32 __must_check uds_map_to_physical_chapter(const struct index_geometry *geometry,
|
||||
u64 virtual_chapter);
|
||||
|
||||
/*
|
||||
* Check whether this geometry is reduced by a chapter. This will only be true if the volume was
|
||||
* converted from a non-lvm volume to an lvm volume.
|
||||
*/
|
||||
static inline bool __must_check
|
||||
uds_is_reduced_index_geometry(const struct index_geometry *geometry)
|
||||
{
|
||||
return !!(geometry->chapters_per_volume & 1);
|
||||
}
|
||||
|
||||
static inline bool __must_check
|
||||
uds_is_sparse_index_geometry(const struct index_geometry *geometry)
|
||||
{
|
||||
return geometry->sparse_chapters_per_volume > 0;
|
||||
}
|
||||
|
||||
bool __must_check uds_has_sparse_chapters(const struct index_geometry *geometry,
|
||||
u64 oldest_virtual_chapter,
|
||||
u64 newest_virtual_chapter);
|
||||
|
||||
bool __must_check uds_is_chapter_sparse(const struct index_geometry *geometry,
|
||||
u64 oldest_virtual_chapter,
|
||||
u64 newest_virtual_chapter,
|
||||
u64 virtual_chapter_number);
|
||||
|
||||
u32 __must_check uds_chapters_to_expire(const struct index_geometry *geometry,
|
||||
u64 newest_chapter);
|
||||
|
||||
#endif /* UDS_INDEX_GEOMETRY_H */
|
|
@ -0,0 +1,66 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef UDS_HASH_UTILS_H
|
||||
#define UDS_HASH_UTILS_H
|
||||
|
||||
#include "numeric.h"
|
||||
|
||||
#include "geometry.h"
|
||||
#include "indexer.h"
|
||||
|
||||
/* Utilities for extracting portions of a request name for various uses. */
|
||||
|
||||
/* How various portions of a record name are apportioned. */
|
||||
enum {
|
||||
VOLUME_INDEX_BYTES_OFFSET = 0,
|
||||
VOLUME_INDEX_BYTES_COUNT = 8,
|
||||
CHAPTER_INDEX_BYTES_OFFSET = 8,
|
||||
CHAPTER_INDEX_BYTES_COUNT = 6,
|
||||
SAMPLE_BYTES_OFFSET = 14,
|
||||
SAMPLE_BYTES_COUNT = 2,
|
||||
};
|
||||
|
||||
static inline u64 uds_extract_chapter_index_bytes(const struct uds_record_name *name)
|
||||
{
|
||||
const u8 *chapter_bits = &name->name[CHAPTER_INDEX_BYTES_OFFSET];
|
||||
u64 bytes = (u64) get_unaligned_be16(chapter_bits) << 32;
|
||||
|
||||
bytes |= get_unaligned_be32(chapter_bits + 2);
|
||||
return bytes;
|
||||
}
|
||||
|
||||
static inline u64 uds_extract_volume_index_bytes(const struct uds_record_name *name)
|
||||
{
|
||||
return get_unaligned_be64(&name->name[VOLUME_INDEX_BYTES_OFFSET]);
|
||||
}
|
||||
|
||||
static inline u32 uds_extract_sampling_bytes(const struct uds_record_name *name)
|
||||
{
|
||||
return get_unaligned_be16(&name->name[SAMPLE_BYTES_OFFSET]);
|
||||
}
|
||||
|
||||
/* Compute the chapter delta list for a given name. */
|
||||
static inline u32 uds_hash_to_chapter_delta_list(const struct uds_record_name *name,
|
||||
const struct index_geometry *geometry)
|
||||
{
|
||||
return ((uds_extract_chapter_index_bytes(name) >> geometry->chapter_address_bits) &
|
||||
((1 << geometry->chapter_delta_list_bits) - 1));
|
||||
}
|
||||
|
||||
/* Compute the chapter delta address for a given name. */
|
||||
static inline u32 uds_hash_to_chapter_delta_address(const struct uds_record_name *name,
|
||||
const struct index_geometry *geometry)
|
||||
{
|
||||
return uds_extract_chapter_index_bytes(name) & ((1 << geometry->chapter_address_bits) - 1);
|
||||
}
|
||||
|
||||
static inline unsigned int uds_name_to_hash_slot(const struct uds_record_name *name,
|
||||
unsigned int slot_count)
|
||||
{
|
||||
return (unsigned int) (uds_extract_chapter_index_bytes(name) % slot_count);
|
||||
}
|
||||
|
||||
#endif /* UDS_HASH_UTILS_H */
|
File diff suppressed because it is too large
Load Diff
|
@ -0,0 +1,43 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef UDS_INDEX_LAYOUT_H
|
||||
#define UDS_INDEX_LAYOUT_H
|
||||
|
||||
#include "config.h"
|
||||
#include "indexer.h"
|
||||
#include "io-factory.h"
|
||||
|
||||
/*
|
||||
* The index layout describes the format of the index on the underlying storage, and is responsible
|
||||
* for creating those structures when the index is first created. It also validates the index data
|
||||
* when loading a saved index, and updates it when saving the index.
|
||||
*/
|
||||
|
||||
struct index_layout;
|
||||
|
||||
int __must_check uds_make_index_layout(struct uds_configuration *config, bool new_layout,
|
||||
struct index_layout **layout_ptr);
|
||||
|
||||
void uds_free_index_layout(struct index_layout *layout);
|
||||
|
||||
int __must_check uds_replace_index_layout_storage(struct index_layout *layout,
|
||||
struct block_device *bdev);
|
||||
|
||||
int __must_check uds_load_index_state(struct index_layout *layout,
|
||||
struct uds_index *index);
|
||||
|
||||
int __must_check uds_save_index_state(struct index_layout *layout,
|
||||
struct uds_index *index);
|
||||
|
||||
int __must_check uds_discard_open_chapter(struct index_layout *layout);
|
||||
|
||||
u64 __must_check uds_get_volume_nonce(struct index_layout *layout);
|
||||
|
||||
int __must_check uds_open_volume_bufio(struct index_layout *layout, size_t block_size,
|
||||
unsigned int reserved_buffers,
|
||||
struct dm_bufio_client **client_ptr);
|
||||
|
||||
#endif /* UDS_INDEX_LAYOUT_H */
|
|
@ -0,0 +1,173 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "index-page-map.h"
|
||||
|
||||
#include "errors.h"
|
||||
#include "logger.h"
|
||||
#include "memory-alloc.h"
|
||||
#include "numeric.h"
|
||||
#include "permassert.h"
|
||||
#include "string-utils.h"
|
||||
#include "thread-utils.h"
|
||||
|
||||
#include "hash-utils.h"
|
||||
#include "indexer.h"
|
||||
|
||||
/*
|
||||
* The index page map is conceptually a two-dimensional array indexed by chapter number and index
|
||||
* page number within the chapter. Each entry contains the number of the last delta list on that
|
||||
* index page. In order to save memory, the information for the last page in each chapter is not
|
||||
* recorded, as it is known from the geometry.
|
||||
*/
|
||||
|
||||
static const u8 PAGE_MAP_MAGIC[] = "ALBIPM02";
|
||||
|
||||
#define PAGE_MAP_MAGIC_LENGTH (sizeof(PAGE_MAP_MAGIC) - 1)
|
||||
|
||||
static inline u32 get_entry_count(const struct index_geometry *geometry)
|
||||
{
|
||||
return geometry->chapters_per_volume * (geometry->index_pages_per_chapter - 1);
|
||||
}
|
||||
|
||||
int uds_make_index_page_map(const struct index_geometry *geometry,
|
||||
struct index_page_map **map_ptr)
|
||||
{
|
||||
int result;
|
||||
struct index_page_map *map;
|
||||
|
||||
result = vdo_allocate(1, struct index_page_map, "page map", &map);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
map->geometry = geometry;
|
||||
map->entries_per_chapter = geometry->index_pages_per_chapter - 1;
|
||||
result = vdo_allocate(get_entry_count(geometry), u16, "Index Page Map Entries",
|
||||
&map->entries);
|
||||
if (result != VDO_SUCCESS) {
|
||||
uds_free_index_page_map(map);
|
||||
return result;
|
||||
}
|
||||
|
||||
*map_ptr = map;
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
void uds_free_index_page_map(struct index_page_map *map)
|
||||
{
|
||||
if (map != NULL) {
|
||||
vdo_free(map->entries);
|
||||
vdo_free(map);
|
||||
}
|
||||
}
|
||||
|
||||
void uds_update_index_page_map(struct index_page_map *map, u64 virtual_chapter_number,
|
||||
u32 chapter_number, u32 index_page_number,
|
||||
u32 delta_list_number)
|
||||
{
|
||||
size_t slot;
|
||||
|
||||
map->last_update = virtual_chapter_number;
|
||||
if (index_page_number == map->entries_per_chapter)
|
||||
return;
|
||||
|
||||
slot = (chapter_number * map->entries_per_chapter) + index_page_number;
|
||||
map->entries[slot] = delta_list_number;
|
||||
}
|
||||
|
||||
u32 uds_find_index_page_number(const struct index_page_map *map,
|
||||
const struct uds_record_name *name, u32 chapter_number)
|
||||
{
|
||||
u32 delta_list_number = uds_hash_to_chapter_delta_list(name, map->geometry);
|
||||
u32 slot = chapter_number * map->entries_per_chapter;
|
||||
u32 page;
|
||||
|
||||
for (page = 0; page < map->entries_per_chapter; page++) {
|
||||
if (delta_list_number <= map->entries[slot + page])
|
||||
break;
|
||||
}
|
||||
|
||||
return page;
|
||||
}
|
||||
|
||||
void uds_get_list_number_bounds(const struct index_page_map *map, u32 chapter_number,
|
||||
u32 index_page_number, u32 *lowest_list,
|
||||
u32 *highest_list)
|
||||
{
|
||||
u32 slot = chapter_number * map->entries_per_chapter;
|
||||
|
||||
*lowest_list = ((index_page_number == 0) ?
|
||||
0 : map->entries[slot + index_page_number - 1] + 1);
|
||||
*highest_list = ((index_page_number < map->entries_per_chapter) ?
|
||||
map->entries[slot + index_page_number] :
|
||||
map->geometry->delta_lists_per_chapter - 1);
|
||||
}
|
||||
|
||||
u64 uds_compute_index_page_map_save_size(const struct index_geometry *geometry)
|
||||
{
|
||||
return PAGE_MAP_MAGIC_LENGTH + sizeof(u64) + sizeof(u16) * get_entry_count(geometry);
|
||||
}
|
||||
|
||||
int uds_write_index_page_map(struct index_page_map *map, struct buffered_writer *writer)
|
||||
{
|
||||
int result;
|
||||
u8 *buffer;
|
||||
size_t offset = 0;
|
||||
u64 saved_size = uds_compute_index_page_map_save_size(map->geometry);
|
||||
u32 i;
|
||||
|
||||
result = vdo_allocate(saved_size, u8, "page map data", &buffer);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
memcpy(buffer, PAGE_MAP_MAGIC, PAGE_MAP_MAGIC_LENGTH);
|
||||
offset += PAGE_MAP_MAGIC_LENGTH;
|
||||
encode_u64_le(buffer, &offset, map->last_update);
|
||||
for (i = 0; i < get_entry_count(map->geometry); i++)
|
||||
encode_u16_le(buffer, &offset, map->entries[i]);
|
||||
|
||||
result = uds_write_to_buffered_writer(writer, buffer, offset);
|
||||
vdo_free(buffer);
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
return uds_flush_buffered_writer(writer);
|
||||
}
|
||||
|
||||
int uds_read_index_page_map(struct index_page_map *map, struct buffered_reader *reader)
|
||||
{
|
||||
int result;
|
||||
u8 magic[PAGE_MAP_MAGIC_LENGTH];
|
||||
u8 *buffer;
|
||||
size_t offset = 0;
|
||||
u64 saved_size = uds_compute_index_page_map_save_size(map->geometry);
|
||||
u32 i;
|
||||
|
||||
result = vdo_allocate(saved_size, u8, "page map data", &buffer);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
result = uds_read_from_buffered_reader(reader, buffer, saved_size);
|
||||
if (result != UDS_SUCCESS) {
|
||||
vdo_free(buffer);
|
||||
return result;
|
||||
}
|
||||
|
||||
memcpy(&magic, buffer, PAGE_MAP_MAGIC_LENGTH);
|
||||
offset += PAGE_MAP_MAGIC_LENGTH;
|
||||
if (memcmp(magic, PAGE_MAP_MAGIC, PAGE_MAP_MAGIC_LENGTH) != 0) {
|
||||
vdo_free(buffer);
|
||||
return UDS_CORRUPT_DATA;
|
||||
}
|
||||
|
||||
decode_u64_le(buffer, &offset, &map->last_update);
|
||||
for (i = 0; i < get_entry_count(map->geometry); i++)
|
||||
decode_u16_le(buffer, &offset, &map->entries[i]);
|
||||
|
||||
vdo_free(buffer);
|
||||
vdo_log_debug("read index page map, last update %llu",
|
||||
(unsigned long long) map->last_update);
|
||||
return UDS_SUCCESS;
|
||||
}
|
|
@ -0,0 +1,50 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef UDS_INDEX_PAGE_MAP_H
|
||||
#define UDS_INDEX_PAGE_MAP_H
|
||||
|
||||
#include "geometry.h"
|
||||
#include "io-factory.h"
|
||||
|
||||
/*
|
||||
* The index maintains a page map which records how the chapter delta lists are distributed among
|
||||
* the index pages for each chapter, allowing the volume to be efficient about reading only pages
|
||||
* that it knows it will need.
|
||||
*/
|
||||
|
||||
struct index_page_map {
|
||||
const struct index_geometry *geometry;
|
||||
u64 last_update;
|
||||
u32 entries_per_chapter;
|
||||
u16 *entries;
|
||||
};
|
||||
|
||||
int __must_check uds_make_index_page_map(const struct index_geometry *geometry,
|
||||
struct index_page_map **map_ptr);
|
||||
|
||||
void uds_free_index_page_map(struct index_page_map *map);
|
||||
|
||||
int __must_check uds_read_index_page_map(struct index_page_map *map,
|
||||
struct buffered_reader *reader);
|
||||
|
||||
int __must_check uds_write_index_page_map(struct index_page_map *map,
|
||||
struct buffered_writer *writer);
|
||||
|
||||
void uds_update_index_page_map(struct index_page_map *map, u64 virtual_chapter_number,
|
||||
u32 chapter_number, u32 index_page_number,
|
||||
u32 delta_list_number);
|
||||
|
||||
u32 __must_check uds_find_index_page_number(const struct index_page_map *map,
|
||||
const struct uds_record_name *name,
|
||||
u32 chapter_number);
|
||||
|
||||
void uds_get_list_number_bounds(const struct index_page_map *map, u32 chapter_number,
|
||||
u32 index_page_number, u32 *lowest_list,
|
||||
u32 *highest_list);
|
||||
|
||||
u64 uds_compute_index_page_map_save_size(const struct index_geometry *geometry);
|
||||
|
||||
#endif /* UDS_INDEX_PAGE_MAP_H */
|
|
@ -0,0 +1,739 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "index-session.h"
|
||||
|
||||
#include <linux/atomic.h>
|
||||
|
||||
#include "logger.h"
|
||||
#include "memory-alloc.h"
|
||||
#include "time-utils.h"
|
||||
|
||||
#include "funnel-requestqueue.h"
|
||||
#include "index.h"
|
||||
#include "index-layout.h"
|
||||
|
||||
/*
|
||||
* The index session contains a lock (the request_mutex) which ensures that only one thread can
|
||||
* change the state of its index at a time. The state field indicates the current state of the
|
||||
* index through a set of descriptive flags. The request_mutex must be notified whenever a
|
||||
* non-transient state flag is cleared. The request_mutex is also used to count the number of
|
||||
* requests currently in progress so that they can be drained when suspending or closing the index.
|
||||
*
|
||||
* If the index session is suspended shortly after opening an index, it may have to suspend during
|
||||
* a rebuild. Depending on the size of the index, a rebuild may take a significant amount of time,
|
||||
* so UDS allows the rebuild to be paused in order to suspend the session in a timely manner. When
|
||||
* the index session is resumed, the rebuild can continue from where it left off. If the index
|
||||
* session is shut down with a suspended rebuild, the rebuild progress is abandoned and the rebuild
|
||||
* will start from the beginning the next time the index is loaded. The mutex and status fields in
|
||||
* the index_load_context are used to record the state of any interrupted rebuild.
|
||||
*/
|
||||
|
||||
enum index_session_flag_bit {
|
||||
IS_FLAG_BIT_START = 8,
|
||||
/* The session has started loading an index but not completed it. */
|
||||
IS_FLAG_BIT_LOADING = IS_FLAG_BIT_START,
|
||||
/* The session has loaded an index, which can handle requests. */
|
||||
IS_FLAG_BIT_LOADED,
|
||||
/* The session's index has been permanently disabled. */
|
||||
IS_FLAG_BIT_DISABLED,
|
||||
/* The session's index is suspended. */
|
||||
IS_FLAG_BIT_SUSPENDED,
|
||||
/* The session is handling some index state change. */
|
||||
IS_FLAG_BIT_WAITING,
|
||||
/* The session's index is closing and draining requests. */
|
||||
IS_FLAG_BIT_CLOSING,
|
||||
/* The session is being destroyed and is draining requests. */
|
||||
IS_FLAG_BIT_DESTROYING,
|
||||
};
|
||||
|
||||
enum index_session_flag {
|
||||
IS_FLAG_LOADED = (1 << IS_FLAG_BIT_LOADED),
|
||||
IS_FLAG_LOADING = (1 << IS_FLAG_BIT_LOADING),
|
||||
IS_FLAG_DISABLED = (1 << IS_FLAG_BIT_DISABLED),
|
||||
IS_FLAG_SUSPENDED = (1 << IS_FLAG_BIT_SUSPENDED),
|
||||
IS_FLAG_WAITING = (1 << IS_FLAG_BIT_WAITING),
|
||||
IS_FLAG_CLOSING = (1 << IS_FLAG_BIT_CLOSING),
|
||||
IS_FLAG_DESTROYING = (1 << IS_FLAG_BIT_DESTROYING),
|
||||
};
|
||||
|
||||
/* Release a reference to an index session. */
|
||||
static void release_index_session(struct uds_index_session *index_session)
|
||||
{
|
||||
mutex_lock(&index_session->request_mutex);
|
||||
if (--index_session->request_count == 0)
|
||||
uds_broadcast_cond(&index_session->request_cond);
|
||||
mutex_unlock(&index_session->request_mutex);
|
||||
}
|
||||
|
||||
/*
|
||||
* Acquire a reference to the index session for an asynchronous index request. The reference must
|
||||
* eventually be released with a corresponding call to release_index_session().
|
||||
*/
|
||||
static int get_index_session(struct uds_index_session *index_session)
|
||||
{
|
||||
unsigned int state;
|
||||
int result = UDS_SUCCESS;
|
||||
|
||||
mutex_lock(&index_session->request_mutex);
|
||||
index_session->request_count++;
|
||||
state = index_session->state;
|
||||
mutex_unlock(&index_session->request_mutex);
|
||||
|
||||
if (state == IS_FLAG_LOADED) {
|
||||
return UDS_SUCCESS;
|
||||
} else if (state & IS_FLAG_DISABLED) {
|
||||
result = UDS_DISABLED;
|
||||
} else if ((state & IS_FLAG_LOADING) ||
|
||||
(state & IS_FLAG_SUSPENDED) ||
|
||||
(state & IS_FLAG_WAITING)) {
|
||||
result = -EBUSY;
|
||||
} else {
|
||||
result = UDS_NO_INDEX;
|
||||
}
|
||||
|
||||
release_index_session(index_session);
|
||||
return result;
|
||||
}
|
||||
|
||||
int uds_launch_request(struct uds_request *request)
|
||||
{
|
||||
size_t internal_size;
|
||||
int result;
|
||||
|
||||
if (request->callback == NULL) {
|
||||
vdo_log_error("missing required callback");
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
switch (request->type) {
|
||||
case UDS_DELETE:
|
||||
case UDS_POST:
|
||||
case UDS_QUERY:
|
||||
case UDS_QUERY_NO_UPDATE:
|
||||
case UDS_UPDATE:
|
||||
break;
|
||||
default:
|
||||
vdo_log_error("received invalid callback type");
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
/* Reset all internal fields before processing. */
|
||||
internal_size =
|
||||
sizeof(struct uds_request) - offsetof(struct uds_request, zone_number);
|
||||
// FIXME should be using struct_group for this instead
|
||||
memset((char *) request + sizeof(*request) - internal_size, 0, internal_size);
|
||||
|
||||
result = get_index_session(request->session);
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
request->found = false;
|
||||
request->unbatched = false;
|
||||
request->index = request->session->index;
|
||||
|
||||
uds_enqueue_request(request, STAGE_TRIAGE);
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
static void enter_callback_stage(struct uds_request *request)
|
||||
{
|
||||
if (request->status != UDS_SUCCESS) {
|
||||
/* All request errors are considered unrecoverable */
|
||||
mutex_lock(&request->session->request_mutex);
|
||||
request->session->state |= IS_FLAG_DISABLED;
|
||||
mutex_unlock(&request->session->request_mutex);
|
||||
}
|
||||
|
||||
uds_request_queue_enqueue(request->session->callback_queue, request);
|
||||
}
|
||||
|
||||
static inline void count_once(u64 *count_ptr)
|
||||
{
|
||||
WRITE_ONCE(*count_ptr, READ_ONCE(*count_ptr) + 1);
|
||||
}
|
||||
|
||||
static void update_session_stats(struct uds_request *request)
|
||||
{
|
||||
struct session_stats *session_stats = &request->session->stats;
|
||||
|
||||
count_once(&session_stats->requests);
|
||||
|
||||
switch (request->type) {
|
||||
case UDS_POST:
|
||||
if (request->found)
|
||||
count_once(&session_stats->posts_found);
|
||||
else
|
||||
count_once(&session_stats->posts_not_found);
|
||||
|
||||
if (request->location == UDS_LOCATION_IN_OPEN_CHAPTER)
|
||||
count_once(&session_stats->posts_found_open_chapter);
|
||||
else if (request->location == UDS_LOCATION_IN_DENSE)
|
||||
count_once(&session_stats->posts_found_dense);
|
||||
else if (request->location == UDS_LOCATION_IN_SPARSE)
|
||||
count_once(&session_stats->posts_found_sparse);
|
||||
break;
|
||||
|
||||
case UDS_UPDATE:
|
||||
if (request->found)
|
||||
count_once(&session_stats->updates_found);
|
||||
else
|
||||
count_once(&session_stats->updates_not_found);
|
||||
break;
|
||||
|
||||
case UDS_DELETE:
|
||||
if (request->found)
|
||||
count_once(&session_stats->deletions_found);
|
||||
else
|
||||
count_once(&session_stats->deletions_not_found);
|
||||
break;
|
||||
|
||||
case UDS_QUERY:
|
||||
case UDS_QUERY_NO_UPDATE:
|
||||
if (request->found)
|
||||
count_once(&session_stats->queries_found);
|
||||
else
|
||||
count_once(&session_stats->queries_not_found);
|
||||
break;
|
||||
|
||||
default:
|
||||
request->status = VDO_ASSERT(false, "unknown request type: %d",
|
||||
request->type);
|
||||
}
|
||||
}
|
||||
|
||||
static void handle_callbacks(struct uds_request *request)
|
||||
{
|
||||
struct uds_index_session *index_session = request->session;
|
||||
|
||||
if (request->status == UDS_SUCCESS)
|
||||
update_session_stats(request);
|
||||
|
||||
request->status = uds_status_to_errno(request->status);
|
||||
request->callback(request);
|
||||
release_index_session(index_session);
|
||||
}
|
||||
|
||||
static int __must_check make_empty_index_session(struct uds_index_session **index_session_ptr)
|
||||
{
|
||||
int result;
|
||||
struct uds_index_session *session;
|
||||
|
||||
result = vdo_allocate(1, struct uds_index_session, __func__, &session);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
mutex_init(&session->request_mutex);
|
||||
uds_init_cond(&session->request_cond);
|
||||
mutex_init(&session->load_context.mutex);
|
||||
uds_init_cond(&session->load_context.cond);
|
||||
|
||||
result = uds_make_request_queue("callbackW", &handle_callbacks,
|
||||
&session->callback_queue);
|
||||
if (result != UDS_SUCCESS) {
|
||||
vdo_free(session);
|
||||
return result;
|
||||
}
|
||||
|
||||
*index_session_ptr = session;
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
int uds_create_index_session(struct uds_index_session **session)
|
||||
{
|
||||
if (session == NULL) {
|
||||
vdo_log_error("missing session pointer");
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
return uds_status_to_errno(make_empty_index_session(session));
|
||||
}
|
||||
|
||||
static int __must_check start_loading_index_session(struct uds_index_session *index_session)
|
||||
{
|
||||
int result;
|
||||
|
||||
mutex_lock(&index_session->request_mutex);
|
||||
if (index_session->state & IS_FLAG_SUSPENDED) {
|
||||
vdo_log_info("Index session is suspended");
|
||||
result = -EBUSY;
|
||||
} else if (index_session->state != 0) {
|
||||
vdo_log_info("Index is already loaded");
|
||||
result = -EBUSY;
|
||||
} else {
|
||||
index_session->state |= IS_FLAG_LOADING;
|
||||
result = UDS_SUCCESS;
|
||||
}
|
||||
mutex_unlock(&index_session->request_mutex);
|
||||
return result;
|
||||
}
|
||||
|
||||
static void finish_loading_index_session(struct uds_index_session *index_session,
|
||||
int result)
|
||||
{
|
||||
mutex_lock(&index_session->request_mutex);
|
||||
index_session->state &= ~IS_FLAG_LOADING;
|
||||
if (result == UDS_SUCCESS)
|
||||
index_session->state |= IS_FLAG_LOADED;
|
||||
|
||||
uds_broadcast_cond(&index_session->request_cond);
|
||||
mutex_unlock(&index_session->request_mutex);
|
||||
}
|
||||
|
||||
static int initialize_index_session(struct uds_index_session *index_session,
|
||||
enum uds_open_index_type open_type)
|
||||
{
|
||||
int result;
|
||||
struct uds_configuration *config;
|
||||
|
||||
result = uds_make_configuration(&index_session->parameters, &config);
|
||||
if (result != UDS_SUCCESS) {
|
||||
vdo_log_error_strerror(result, "Failed to allocate config");
|
||||
return result;
|
||||
}
|
||||
|
||||
memset(&index_session->stats, 0, sizeof(index_session->stats));
|
||||
result = uds_make_index(config, open_type, &index_session->load_context,
|
||||
enter_callback_stage, &index_session->index);
|
||||
if (result != UDS_SUCCESS)
|
||||
vdo_log_error_strerror(result, "Failed to make index");
|
||||
else
|
||||
uds_log_configuration(config);
|
||||
|
||||
uds_free_configuration(config);
|
||||
return result;
|
||||
}
|
||||
|
||||
static const char *get_open_type_string(enum uds_open_index_type open_type)
|
||||
{
|
||||
switch (open_type) {
|
||||
case UDS_CREATE:
|
||||
return "creating index";
|
||||
case UDS_LOAD:
|
||||
return "loading or rebuilding index";
|
||||
case UDS_NO_REBUILD:
|
||||
return "loading index";
|
||||
default:
|
||||
return "unknown open method";
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
* Open an index under the given session. This operation will fail if the
|
||||
* index session is suspended, or if there is already an open index.
|
||||
*/
|
||||
int uds_open_index(enum uds_open_index_type open_type,
|
||||
const struct uds_parameters *parameters,
|
||||
struct uds_index_session *session)
|
||||
{
|
||||
int result;
|
||||
char name[BDEVNAME_SIZE];
|
||||
|
||||
if (parameters == NULL) {
|
||||
vdo_log_error("missing required parameters");
|
||||
return -EINVAL;
|
||||
}
|
||||
if (parameters->bdev == NULL) {
|
||||
vdo_log_error("missing required block device");
|
||||
return -EINVAL;
|
||||
}
|
||||
if (session == NULL) {
|
||||
vdo_log_error("missing required session pointer");
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
result = start_loading_index_session(session);
|
||||
if (result != UDS_SUCCESS)
|
||||
return uds_status_to_errno(result);
|
||||
|
||||
session->parameters = *parameters;
|
||||
format_dev_t(name, parameters->bdev->bd_dev);
|
||||
vdo_log_info("%s: %s", get_open_type_string(open_type), name);
|
||||
|
||||
result = initialize_index_session(session, open_type);
|
||||
if (result != UDS_SUCCESS)
|
||||
vdo_log_error_strerror(result, "Failed %s",
|
||||
get_open_type_string(open_type));
|
||||
|
||||
finish_loading_index_session(session, result);
|
||||
return uds_status_to_errno(result);
|
||||
}
|
||||
|
||||
static void wait_for_no_requests_in_progress(struct uds_index_session *index_session)
|
||||
{
|
||||
mutex_lock(&index_session->request_mutex);
|
||||
while (index_session->request_count > 0) {
|
||||
uds_wait_cond(&index_session->request_cond,
|
||||
&index_session->request_mutex);
|
||||
}
|
||||
mutex_unlock(&index_session->request_mutex);
|
||||
}
|
||||
|
||||
static int __must_check save_index(struct uds_index_session *index_session)
|
||||
{
|
||||
wait_for_no_requests_in_progress(index_session);
|
||||
return uds_save_index(index_session->index);
|
||||
}
|
||||
|
||||
static void suspend_rebuild(struct uds_index_session *session)
|
||||
{
|
||||
mutex_lock(&session->load_context.mutex);
|
||||
switch (session->load_context.status) {
|
||||
case INDEX_OPENING:
|
||||
session->load_context.status = INDEX_SUSPENDING;
|
||||
|
||||
/* Wait until the index indicates that it is not replaying. */
|
||||
while ((session->load_context.status != INDEX_SUSPENDED) &&
|
||||
(session->load_context.status != INDEX_READY)) {
|
||||
uds_wait_cond(&session->load_context.cond,
|
||||
&session->load_context.mutex);
|
||||
}
|
||||
|
||||
break;
|
||||
|
||||
case INDEX_READY:
|
||||
/* Index load does not need to be suspended. */
|
||||
break;
|
||||
|
||||
case INDEX_SUSPENDED:
|
||||
case INDEX_SUSPENDING:
|
||||
case INDEX_FREEING:
|
||||
default:
|
||||
/* These cases should not happen. */
|
||||
VDO_ASSERT_LOG_ONLY(false, "Bad load context state %u",
|
||||
session->load_context.status);
|
||||
break;
|
||||
}
|
||||
mutex_unlock(&session->load_context.mutex);
|
||||
}
|
||||
|
||||
/*
|
||||
* Suspend index operation, draining all current index requests and preventing new index requests
|
||||
* from starting. Optionally saves all index data before returning.
|
||||
*/
|
||||
int uds_suspend_index_session(struct uds_index_session *session, bool save)
|
||||
{
|
||||
int result = UDS_SUCCESS;
|
||||
bool no_work = false;
|
||||
bool rebuilding = false;
|
||||
|
||||
/* Wait for any current index state change to complete. */
|
||||
mutex_lock(&session->request_mutex);
|
||||
while (session->state & IS_FLAG_CLOSING)
|
||||
uds_wait_cond(&session->request_cond, &session->request_mutex);
|
||||
|
||||
if ((session->state & IS_FLAG_WAITING) || (session->state & IS_FLAG_DESTROYING)) {
|
||||
no_work = true;
|
||||
vdo_log_info("Index session is already changing state");
|
||||
result = -EBUSY;
|
||||
} else if (session->state & IS_FLAG_SUSPENDED) {
|
||||
no_work = true;
|
||||
} else if (session->state & IS_FLAG_LOADING) {
|
||||
session->state |= IS_FLAG_WAITING;
|
||||
rebuilding = true;
|
||||
} else if (session->state & IS_FLAG_LOADED) {
|
||||
session->state |= IS_FLAG_WAITING;
|
||||
} else {
|
||||
no_work = true;
|
||||
session->state |= IS_FLAG_SUSPENDED;
|
||||
uds_broadcast_cond(&session->request_cond);
|
||||
}
|
||||
mutex_unlock(&session->request_mutex);
|
||||
|
||||
if (no_work)
|
||||
return uds_status_to_errno(result);
|
||||
|
||||
if (rebuilding)
|
||||
suspend_rebuild(session);
|
||||
else if (save)
|
||||
result = save_index(session);
|
||||
else
|
||||
result = uds_flush_index_session(session);
|
||||
|
||||
mutex_lock(&session->request_mutex);
|
||||
session->state &= ~IS_FLAG_WAITING;
|
||||
session->state |= IS_FLAG_SUSPENDED;
|
||||
uds_broadcast_cond(&session->request_cond);
|
||||
mutex_unlock(&session->request_mutex);
|
||||
return uds_status_to_errno(result);
|
||||
}
|
||||
|
||||
static int replace_device(struct uds_index_session *session, struct block_device *bdev)
|
||||
{
|
||||
int result;
|
||||
|
||||
result = uds_replace_index_storage(session->index, bdev);
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
session->parameters.bdev = bdev;
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
/*
|
||||
* Resume index operation after being suspended. If the index is suspended and the supplied block
|
||||
* device differs from the current backing store, the index will start using the new backing store.
|
||||
*/
|
||||
int uds_resume_index_session(struct uds_index_session *session,
|
||||
struct block_device *bdev)
|
||||
{
|
||||
int result = UDS_SUCCESS;
|
||||
bool no_work = false;
|
||||
bool resume_replay = false;
|
||||
|
||||
mutex_lock(&session->request_mutex);
|
||||
if (session->state & IS_FLAG_WAITING) {
|
||||
vdo_log_info("Index session is already changing state");
|
||||
no_work = true;
|
||||
result = -EBUSY;
|
||||
} else if (!(session->state & IS_FLAG_SUSPENDED)) {
|
||||
/* If not suspended, just succeed. */
|
||||
no_work = true;
|
||||
result = UDS_SUCCESS;
|
||||
} else {
|
||||
session->state |= IS_FLAG_WAITING;
|
||||
if (session->state & IS_FLAG_LOADING)
|
||||
resume_replay = true;
|
||||
}
|
||||
mutex_unlock(&session->request_mutex);
|
||||
|
||||
if (no_work)
|
||||
return result;
|
||||
|
||||
if ((session->index != NULL) && (bdev != session->parameters.bdev)) {
|
||||
result = replace_device(session, bdev);
|
||||
if (result != UDS_SUCCESS) {
|
||||
mutex_lock(&session->request_mutex);
|
||||
session->state &= ~IS_FLAG_WAITING;
|
||||
uds_broadcast_cond(&session->request_cond);
|
||||
mutex_unlock(&session->request_mutex);
|
||||
return uds_status_to_errno(result);
|
||||
}
|
||||
}
|
||||
|
||||
if (resume_replay) {
|
||||
mutex_lock(&session->load_context.mutex);
|
||||
switch (session->load_context.status) {
|
||||
case INDEX_SUSPENDED:
|
||||
session->load_context.status = INDEX_OPENING;
|
||||
/* Notify the index to start replaying again. */
|
||||
uds_broadcast_cond(&session->load_context.cond);
|
||||
break;
|
||||
|
||||
case INDEX_READY:
|
||||
/* There is no index rebuild to resume. */
|
||||
break;
|
||||
|
||||
case INDEX_OPENING:
|
||||
case INDEX_SUSPENDING:
|
||||
case INDEX_FREEING:
|
||||
default:
|
||||
/* These cases should not happen; do nothing. */
|
||||
VDO_ASSERT_LOG_ONLY(false, "Bad load context state %u",
|
||||
session->load_context.status);
|
||||
break;
|
||||
}
|
||||
mutex_unlock(&session->load_context.mutex);
|
||||
}
|
||||
|
||||
mutex_lock(&session->request_mutex);
|
||||
session->state &= ~IS_FLAG_WAITING;
|
||||
session->state &= ~IS_FLAG_SUSPENDED;
|
||||
uds_broadcast_cond(&session->request_cond);
|
||||
mutex_unlock(&session->request_mutex);
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
static int save_and_free_index(struct uds_index_session *index_session)
|
||||
{
|
||||
int result = UDS_SUCCESS;
|
||||
bool suspended;
|
||||
struct uds_index *index = index_session->index;
|
||||
|
||||
if (index == NULL)
|
||||
return UDS_SUCCESS;
|
||||
|
||||
mutex_lock(&index_session->request_mutex);
|
||||
suspended = (index_session->state & IS_FLAG_SUSPENDED);
|
||||
mutex_unlock(&index_session->request_mutex);
|
||||
|
||||
if (!suspended) {
|
||||
result = uds_save_index(index);
|
||||
if (result != UDS_SUCCESS)
|
||||
vdo_log_warning_strerror(result,
|
||||
"ignoring error from save_index");
|
||||
}
|
||||
uds_free_index(index);
|
||||
index_session->index = NULL;
|
||||
|
||||
/*
|
||||
* Reset all index state that happens to be in the index
|
||||
* session, so it doesn't affect any future index.
|
||||
*/
|
||||
mutex_lock(&index_session->load_context.mutex);
|
||||
index_session->load_context.status = INDEX_OPENING;
|
||||
mutex_unlock(&index_session->load_context.mutex);
|
||||
|
||||
mutex_lock(&index_session->request_mutex);
|
||||
/* Only the suspend bit will remain relevant. */
|
||||
index_session->state &= IS_FLAG_SUSPENDED;
|
||||
mutex_unlock(&index_session->request_mutex);
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
/* Save and close the current index. */
|
||||
int uds_close_index(struct uds_index_session *index_session)
|
||||
{
|
||||
int result = UDS_SUCCESS;
|
||||
|
||||
/* Wait for any current index state change to complete. */
|
||||
mutex_lock(&index_session->request_mutex);
|
||||
while ((index_session->state & IS_FLAG_WAITING) ||
|
||||
(index_session->state & IS_FLAG_CLOSING)) {
|
||||
uds_wait_cond(&index_session->request_cond,
|
||||
&index_session->request_mutex);
|
||||
}
|
||||
|
||||
if (index_session->state & IS_FLAG_SUSPENDED) {
|
||||
vdo_log_info("Index session is suspended");
|
||||
result = -EBUSY;
|
||||
} else if ((index_session->state & IS_FLAG_DESTROYING) ||
|
||||
!(index_session->state & IS_FLAG_LOADED)) {
|
||||
/* The index doesn't exist, hasn't finished loading, or is being destroyed. */
|
||||
result = UDS_NO_INDEX;
|
||||
} else {
|
||||
index_session->state |= IS_FLAG_CLOSING;
|
||||
}
|
||||
mutex_unlock(&index_session->request_mutex);
|
||||
if (result != UDS_SUCCESS)
|
||||
return uds_status_to_errno(result);
|
||||
|
||||
vdo_log_debug("Closing index");
|
||||
wait_for_no_requests_in_progress(index_session);
|
||||
result = save_and_free_index(index_session);
|
||||
vdo_log_debug("Closed index");
|
||||
|
||||
mutex_lock(&index_session->request_mutex);
|
||||
index_session->state &= ~IS_FLAG_CLOSING;
|
||||
uds_broadcast_cond(&index_session->request_cond);
|
||||
mutex_unlock(&index_session->request_mutex);
|
||||
return uds_status_to_errno(result);
|
||||
}
|
||||
|
||||
/* This will save and close an open index before destroying the session. */
|
||||
int uds_destroy_index_session(struct uds_index_session *index_session)
|
||||
{
|
||||
int result;
|
||||
bool load_pending = false;
|
||||
|
||||
vdo_log_debug("Destroying index session");
|
||||
|
||||
/* Wait for any current index state change to complete. */
|
||||
mutex_lock(&index_session->request_mutex);
|
||||
while ((index_session->state & IS_FLAG_WAITING) ||
|
||||
(index_session->state & IS_FLAG_CLOSING)) {
|
||||
uds_wait_cond(&index_session->request_cond,
|
||||
&index_session->request_mutex);
|
||||
}
|
||||
|
||||
if (index_session->state & IS_FLAG_DESTROYING) {
|
||||
mutex_unlock(&index_session->request_mutex);
|
||||
vdo_log_info("Index session is already closing");
|
||||
return -EBUSY;
|
||||
}
|
||||
|
||||
index_session->state |= IS_FLAG_DESTROYING;
|
||||
load_pending = ((index_session->state & IS_FLAG_LOADING) &&
|
||||
(index_session->state & IS_FLAG_SUSPENDED));
|
||||
mutex_unlock(&index_session->request_mutex);
|
||||
|
||||
if (load_pending) {
|
||||
/* Tell the index to terminate the rebuild. */
|
||||
mutex_lock(&index_session->load_context.mutex);
|
||||
if (index_session->load_context.status == INDEX_SUSPENDED) {
|
||||
index_session->load_context.status = INDEX_FREEING;
|
||||
uds_broadcast_cond(&index_session->load_context.cond);
|
||||
}
|
||||
mutex_unlock(&index_session->load_context.mutex);
|
||||
|
||||
/* Wait until the load exits before proceeding. */
|
||||
mutex_lock(&index_session->request_mutex);
|
||||
while (index_session->state & IS_FLAG_LOADING) {
|
||||
uds_wait_cond(&index_session->request_cond,
|
||||
&index_session->request_mutex);
|
||||
}
|
||||
mutex_unlock(&index_session->request_mutex);
|
||||
}
|
||||
|
||||
wait_for_no_requests_in_progress(index_session);
|
||||
result = save_and_free_index(index_session);
|
||||
uds_request_queue_finish(index_session->callback_queue);
|
||||
index_session->callback_queue = NULL;
|
||||
vdo_log_debug("Destroyed index session");
|
||||
vdo_free(index_session);
|
||||
return uds_status_to_errno(result);
|
||||
}
|
||||
|
||||
/* Wait until all callbacks for index operations are complete. */
|
||||
int uds_flush_index_session(struct uds_index_session *index_session)
|
||||
{
|
||||
wait_for_no_requests_in_progress(index_session);
|
||||
uds_wait_for_idle_index(index_session->index);
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
/* Statistics collection is intended to be thread-safe. */
|
||||
static void collect_stats(const struct uds_index_session *index_session,
|
||||
struct uds_index_stats *stats)
|
||||
{
|
||||
const struct session_stats *session_stats = &index_session->stats;
|
||||
|
||||
stats->current_time = ktime_to_seconds(current_time_ns(CLOCK_REALTIME));
|
||||
stats->posts_found = READ_ONCE(session_stats->posts_found);
|
||||
stats->in_memory_posts_found = READ_ONCE(session_stats->posts_found_open_chapter);
|
||||
stats->dense_posts_found = READ_ONCE(session_stats->posts_found_dense);
|
||||
stats->sparse_posts_found = READ_ONCE(session_stats->posts_found_sparse);
|
||||
stats->posts_not_found = READ_ONCE(session_stats->posts_not_found);
|
||||
stats->updates_found = READ_ONCE(session_stats->updates_found);
|
||||
stats->updates_not_found = READ_ONCE(session_stats->updates_not_found);
|
||||
stats->deletions_found = READ_ONCE(session_stats->deletions_found);
|
||||
stats->deletions_not_found = READ_ONCE(session_stats->deletions_not_found);
|
||||
stats->queries_found = READ_ONCE(session_stats->queries_found);
|
||||
stats->queries_not_found = READ_ONCE(session_stats->queries_not_found);
|
||||
stats->requests = READ_ONCE(session_stats->requests);
|
||||
}
|
||||
|
||||
int uds_get_index_session_stats(struct uds_index_session *index_session,
|
||||
struct uds_index_stats *stats)
|
||||
{
|
||||
if (stats == NULL) {
|
||||
vdo_log_error("received a NULL index stats pointer");
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
collect_stats(index_session, stats);
|
||||
if (index_session->index != NULL) {
|
||||
uds_get_index_stats(index_session->index, stats);
|
||||
} else {
|
||||
stats->entries_indexed = 0;
|
||||
stats->memory_used = 0;
|
||||
stats->collisions = 0;
|
||||
stats->entries_discarded = 0;
|
||||
}
|
||||
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
void uds_wait_cond(struct cond_var *cv, struct mutex *mutex)
|
||||
{
|
||||
DEFINE_WAIT(__wait);
|
||||
|
||||
prepare_to_wait(&cv->wait_queue, &__wait, TASK_IDLE);
|
||||
mutex_unlock(mutex);
|
||||
schedule();
|
||||
finish_wait(&cv->wait_queue, &__wait);
|
||||
mutex_lock(mutex);
|
||||
}
|
|
@ -0,0 +1,85 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef UDS_INDEX_SESSION_H
|
||||
#define UDS_INDEX_SESSION_H
|
||||
|
||||
#include <linux/atomic.h>
|
||||
#include <linux/cache.h>
|
||||
|
||||
#include "thread-utils.h"
|
||||
|
||||
#include "config.h"
|
||||
#include "indexer.h"
|
||||
|
||||
/*
|
||||
* The index session mediates all interactions with a UDS index. Once the index session is created,
|
||||
* it can be used to open, close, suspend, or recreate an index. It implements the majority of the
|
||||
* functions in the top-level UDS API.
|
||||
*
|
||||
* If any deduplication request fails due to an internal error, the index is marked disabled. It
|
||||
* will not accept any further requests and can only be closed. Closing the index will clear the
|
||||
* disabled flag, and the index can then be reopened and recovered using the same index session.
|
||||
*/
|
||||
|
||||
struct __aligned(L1_CACHE_BYTES) session_stats {
|
||||
/* Post requests that found an entry */
|
||||
u64 posts_found;
|
||||
/* Post requests found in the open chapter */
|
||||
u64 posts_found_open_chapter;
|
||||
/* Post requests found in the dense index */
|
||||
u64 posts_found_dense;
|
||||
/* Post requests found in the sparse index */
|
||||
u64 posts_found_sparse;
|
||||
/* Post requests that did not find an entry */
|
||||
u64 posts_not_found;
|
||||
/* Update requests that found an entry */
|
||||
u64 updates_found;
|
||||
/* Update requests that did not find an entry */
|
||||
u64 updates_not_found;
|
||||
/* Delete requests that found an entry */
|
||||
u64 deletions_found;
|
||||
/* Delete requests that did not find an entry */
|
||||
u64 deletions_not_found;
|
||||
/* Query requests that found an entry */
|
||||
u64 queries_found;
|
||||
/* Query requests that did not find an entry */
|
||||
u64 queries_not_found;
|
||||
/* Total number of requests */
|
||||
u64 requests;
|
||||
};
|
||||
|
||||
enum index_suspend_status {
|
||||
/* An index load has started but the index is not ready for use. */
|
||||
INDEX_OPENING = 0,
|
||||
/* The index is able to handle requests. */
|
||||
INDEX_READY,
|
||||
/* The index is attempting to suspend a rebuild. */
|
||||
INDEX_SUSPENDING,
|
||||
/* An index rebuild has been suspended. */
|
||||
INDEX_SUSPENDED,
|
||||
/* An index rebuild is being stopped in order to shut down. */
|
||||
INDEX_FREEING,
|
||||
};
|
||||
|
||||
struct index_load_context {
|
||||
struct mutex mutex;
|
||||
struct cond_var cond;
|
||||
enum index_suspend_status status;
|
||||
};
|
||||
|
||||
struct uds_index_session {
|
||||
unsigned int state;
|
||||
struct uds_index *index;
|
||||
struct uds_request_queue *callback_queue;
|
||||
struct uds_parameters parameters;
|
||||
struct index_load_context load_context;
|
||||
struct mutex request_mutex;
|
||||
struct cond_var request_cond;
|
||||
int request_count;
|
||||
struct session_stats stats;
|
||||
};
|
||||
|
||||
#endif /* UDS_INDEX_SESSION_H */
|
File diff suppressed because it is too large
Load Diff
|
@ -0,0 +1,83 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef UDS_INDEX_H
|
||||
#define UDS_INDEX_H
|
||||
|
||||
#include "index-layout.h"
|
||||
#include "index-session.h"
|
||||
#include "open-chapter.h"
|
||||
#include "volume.h"
|
||||
#include "volume-index.h"
|
||||
|
||||
/*
|
||||
* The index is a high-level structure which represents the totality of the UDS index. It manages
|
||||
* the queues for incoming requests and dispatches them to the appropriate sub-components like the
|
||||
* volume or the volume index. It also manages administrative tasks such as saving and loading the
|
||||
* index.
|
||||
*
|
||||
* The index is divided into a number of independent zones and assigns each request to a zone based
|
||||
* on its name. Most sub-components are similarly divided into zones as well so that requests in
|
||||
* each zone usually operate without interference or coordination between zones.
|
||||
*/
|
||||
|
||||
typedef void (*index_callback_fn)(struct uds_request *request);
|
||||
|
||||
struct index_zone {
|
||||
struct uds_index *index;
|
||||
struct open_chapter_zone *open_chapter;
|
||||
struct open_chapter_zone *writing_chapter;
|
||||
u64 oldest_virtual_chapter;
|
||||
u64 newest_virtual_chapter;
|
||||
unsigned int id;
|
||||
};
|
||||
|
||||
struct uds_index {
|
||||
bool has_saved_open_chapter;
|
||||
bool need_to_save;
|
||||
struct index_load_context *load_context;
|
||||
struct index_layout *layout;
|
||||
struct volume_index *volume_index;
|
||||
struct volume *volume;
|
||||
unsigned int zone_count;
|
||||
struct index_zone **zones;
|
||||
|
||||
u64 oldest_virtual_chapter;
|
||||
u64 newest_virtual_chapter;
|
||||
|
||||
u64 last_save;
|
||||
u64 prev_save;
|
||||
struct chapter_writer *chapter_writer;
|
||||
|
||||
index_callback_fn callback;
|
||||
struct uds_request_queue *triage_queue;
|
||||
struct uds_request_queue *zone_queues[];
|
||||
};
|
||||
|
||||
enum request_stage {
|
||||
STAGE_TRIAGE,
|
||||
STAGE_INDEX,
|
||||
STAGE_MESSAGE,
|
||||
};
|
||||
|
||||
int __must_check uds_make_index(struct uds_configuration *config,
|
||||
enum uds_open_index_type open_type,
|
||||
struct index_load_context *load_context,
|
||||
index_callback_fn callback, struct uds_index **new_index);
|
||||
|
||||
int __must_check uds_save_index(struct uds_index *index);
|
||||
|
||||
void uds_free_index(struct uds_index *index);
|
||||
|
||||
int __must_check uds_replace_index_storage(struct uds_index *index,
|
||||
struct block_device *bdev);
|
||||
|
||||
void uds_get_index_stats(struct uds_index *index, struct uds_index_stats *counters);
|
||||
|
||||
void uds_enqueue_request(struct uds_request *request, enum request_stage stage);
|
||||
|
||||
void uds_wait_for_idle_index(struct uds_index *index);
|
||||
|
||||
#endif /* UDS_INDEX_H */
|
|
@ -0,0 +1,353 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef INDEXER_H
|
||||
#define INDEXER_H
|
||||
|
||||
#include <linux/mutex.h>
|
||||
#include <linux/sched.h>
|
||||
#include <linux/types.h>
|
||||
#include <linux/wait.h>
|
||||
|
||||
#include "funnel-queue.h"
|
||||
|
||||
/*
|
||||
* UDS public API
|
||||
*
|
||||
* The Universal Deduplication System (UDS) is an efficient name-value store. When used for
|
||||
* deduplicating storage, the names are generally hashes of data blocks and the associated data is
|
||||
* where that block is located on the underlying storage medium. The stored names are expected to
|
||||
* be randomly distributed among the space of possible names. If this assumption is violated, the
|
||||
* UDS index will store fewer names than normal but will otherwise continue to work. The data
|
||||
* associated with each name can be any 16-byte value.
|
||||
*
|
||||
* A client must first create an index session to interact with an index. Once created, the session
|
||||
* can be shared among multiple threads or users. When a session is destroyed, it will also close
|
||||
* and save any associated index.
|
||||
*
|
||||
* To make a request, a client must allocate a uds_request structure and set the required fields
|
||||
* before launching it. UDS will invoke the provided callback to complete the request. After the
|
||||
* callback has been called, the uds_request structure can be freed or reused for a new request.
|
||||
* There are five types of requests:
|
||||
*
|
||||
* A UDS_UPDATE request will associate the provided name with the provided data. Any previous data
|
||||
* associated with that name will be discarded.
|
||||
*
|
||||
* A UDS_QUERY request will return the data associated with the provided name, if any. The entry
|
||||
* for the name will also be marked as most recent, as if the data had been updated.
|
||||
*
|
||||
* A UDS_POST request is a combination of UDS_QUERY and UDS_UPDATE. If there is already data
|
||||
* associated with the provided name, that data is returned. If there is no existing association,
|
||||
* the name is associated with the newly provided data. This request is equivalent to a UDS_QUERY
|
||||
* request followed by a UDS_UPDATE request if no data is found, but it is much more efficient.
|
||||
*
|
||||
* A UDS_QUERY_NO_UPDATE request will return the data associated with the provided name, but will
|
||||
* not change the recency of the entry for the name. This request is primarily useful for testing,
|
||||
* to determine whether an entry exists without changing the internal state of the index.
|
||||
*
|
||||
* A UDS_DELETE request removes any data associated with the provided name. This operation is
|
||||
* generally not necessary, because the index will automatically discard its oldest entries once it
|
||||
* becomes full.
|
||||
*/
|
||||
|
||||
/* General UDS constants and structures */
|
||||
|
||||
enum uds_request_type {
|
||||
/* Create or update the mapping for a name, and make the name most recent. */
|
||||
UDS_UPDATE,
|
||||
|
||||
/* Return any mapped data for a name, and make the name most recent. */
|
||||
UDS_QUERY,
|
||||
|
||||
/*
|
||||
* Return any mapped data for a name, or map the provided data to the name if there is no
|
||||
* current data, and make the name most recent.
|
||||
*/
|
||||
UDS_POST,
|
||||
|
||||
/* Return any mapped data for a name without updating its recency. */
|
||||
UDS_QUERY_NO_UPDATE,
|
||||
|
||||
/* Remove any mapping for a name. */
|
||||
UDS_DELETE,
|
||||
|
||||
};
|
||||
|
||||
enum uds_open_index_type {
|
||||
/* Create a new index. */
|
||||
UDS_CREATE,
|
||||
|
||||
/* Load an existing index and try to recover if necessary. */
|
||||
UDS_LOAD,
|
||||
|
||||
/* Load an existing index, but only if it was saved cleanly. */
|
||||
UDS_NO_REBUILD,
|
||||
};
|
||||
|
||||
enum {
|
||||
/* The record name size in bytes */
|
||||
UDS_RECORD_NAME_SIZE = 16,
|
||||
/* The maximum record data size in bytes */
|
||||
UDS_RECORD_DATA_SIZE = 16,
|
||||
};
|
||||
|
||||
/*
|
||||
* A type representing a UDS memory configuration which is either a positive integer number of
|
||||
* gigabytes or one of the six special constants for configurations smaller than one gigabyte.
|
||||
*/
|
||||
typedef int uds_memory_config_size_t;
|
||||
|
||||
enum {
|
||||
/* The maximum configurable amount of memory */
|
||||
UDS_MEMORY_CONFIG_MAX = 1024,
|
||||
/* Flag indicating that the index has one less chapter than usual */
|
||||
UDS_MEMORY_CONFIG_REDUCED = 0x1000,
|
||||
UDS_MEMORY_CONFIG_REDUCED_MAX = 1024 + UDS_MEMORY_CONFIG_REDUCED,
|
||||
/* Special values indicating sizes less than 1 GB */
|
||||
UDS_MEMORY_CONFIG_256MB = -256,
|
||||
UDS_MEMORY_CONFIG_512MB = -512,
|
||||
UDS_MEMORY_CONFIG_768MB = -768,
|
||||
UDS_MEMORY_CONFIG_REDUCED_256MB = -1280,
|
||||
UDS_MEMORY_CONFIG_REDUCED_512MB = -1536,
|
||||
UDS_MEMORY_CONFIG_REDUCED_768MB = -1792,
|
||||
};
|
||||
|
||||
struct uds_record_name {
|
||||
unsigned char name[UDS_RECORD_NAME_SIZE];
|
||||
};
|
||||
|
||||
struct uds_record_data {
|
||||
unsigned char data[UDS_RECORD_DATA_SIZE];
|
||||
};
|
||||
|
||||
struct uds_volume_record {
|
||||
struct uds_record_name name;
|
||||
struct uds_record_data data;
|
||||
};
|
||||
|
||||
struct uds_parameters {
|
||||
/* The block_device used for storage */
|
||||
struct block_device *bdev;
|
||||
/* The maximum allowable size of the index on storage */
|
||||
size_t size;
|
||||
/* The offset where the index should start */
|
||||
off_t offset;
|
||||
/* The maximum memory allocation, in GB */
|
||||
uds_memory_config_size_t memory_size;
|
||||
/* Whether the index should include sparse chapters */
|
||||
bool sparse;
|
||||
/* A 64-bit nonce to validate the index */
|
||||
u64 nonce;
|
||||
/* The number of threads used to process index requests */
|
||||
unsigned int zone_count;
|
||||
/* The number of threads used to read volume pages */
|
||||
unsigned int read_threads;
|
||||
};
|
||||
|
||||
/*
|
||||
* These statistics capture characteristics of the current index, including resource usage and
|
||||
* requests processed since the index was opened.
|
||||
*/
|
||||
struct uds_index_stats {
|
||||
/* The total number of records stored in the index */
|
||||
u64 entries_indexed;
|
||||
/* An estimate of the index's memory usage, in bytes */
|
||||
u64 memory_used;
|
||||
/* The number of collisions recorded in the volume index */
|
||||
u64 collisions;
|
||||
/* The number of entries discarded from the index since startup */
|
||||
u64 entries_discarded;
|
||||
/* The time at which these statistics were fetched */
|
||||
s64 current_time;
|
||||
/* The number of post calls that found an existing entry */
|
||||
u64 posts_found;
|
||||
/* The number of post calls that added an entry */
|
||||
u64 posts_not_found;
|
||||
/*
|
||||
* The number of post calls that found an existing entry that is current enough to only
|
||||
* exist in memory and not have been committed to disk yet
|
||||
*/
|
||||
u64 in_memory_posts_found;
|
||||
/*
|
||||
* The number of post calls that found an existing entry in the dense portion of the index
|
||||
*/
|
||||
u64 dense_posts_found;
|
||||
/*
|
||||
* The number of post calls that found an existing entry in the sparse portion of the index
|
||||
*/
|
||||
u64 sparse_posts_found;
|
||||
/* The number of update calls that updated an existing entry */
|
||||
u64 updates_found;
|
||||
/* The number of update calls that added a new entry */
|
||||
u64 updates_not_found;
|
||||
/* The number of delete requests that deleted an existing entry */
|
||||
u64 deletions_found;
|
||||
/* The number of delete requests that did nothing */
|
||||
u64 deletions_not_found;
|
||||
/* The number of query calls that found existing entry */
|
||||
u64 queries_found;
|
||||
/* The number of query calls that did not find an entry */
|
||||
u64 queries_not_found;
|
||||
/* The total number of requests processed */
|
||||
u64 requests;
|
||||
};
|
||||
|
||||
enum uds_index_region {
|
||||
/* No location information has been determined */
|
||||
UDS_LOCATION_UNKNOWN = 0,
|
||||
/* The index page entry has been found */
|
||||
UDS_LOCATION_INDEX_PAGE_LOOKUP,
|
||||
/* The record page entry has been found */
|
||||
UDS_LOCATION_RECORD_PAGE_LOOKUP,
|
||||
/* The record is not in the index */
|
||||
UDS_LOCATION_UNAVAILABLE,
|
||||
/* The record was found in the open chapter */
|
||||
UDS_LOCATION_IN_OPEN_CHAPTER,
|
||||
/* The record was found in the dense part of the index */
|
||||
UDS_LOCATION_IN_DENSE,
|
||||
/* The record was found in the sparse part of the index */
|
||||
UDS_LOCATION_IN_SPARSE,
|
||||
} __packed;
|
||||
|
||||
/* Zone message requests are used to communicate between index zones. */
|
||||
enum uds_zone_message_type {
|
||||
/* A standard request with no message */
|
||||
UDS_MESSAGE_NONE = 0,
|
||||
/* Add a chapter to the sparse chapter index cache */
|
||||
UDS_MESSAGE_SPARSE_CACHE_BARRIER,
|
||||
/* Close a chapter to keep the zone from falling behind */
|
||||
UDS_MESSAGE_ANNOUNCE_CHAPTER_CLOSED,
|
||||
} __packed;
|
||||
|
||||
struct uds_zone_message {
|
||||
/* The type of message, determining how it will be processed */
|
||||
enum uds_zone_message_type type;
|
||||
/* The virtual chapter number to which the message applies */
|
||||
u64 virtual_chapter;
|
||||
};
|
||||
|
||||
struct uds_index_session;
|
||||
struct uds_index;
|
||||
struct uds_request;
|
||||
|
||||
/* Once this callback has been invoked, the uds_request structure can be reused or freed. */
|
||||
typedef void (*uds_request_callback_fn)(struct uds_request *request);
|
||||
|
||||
struct uds_request {
|
||||
/* These input fields must be set before launching a request. */
|
||||
|
||||
/* The name of the record to look up or create */
|
||||
struct uds_record_name record_name;
|
||||
/* New data to associate with the record name, if applicable */
|
||||
struct uds_record_data new_metadata;
|
||||
/* A callback to invoke when the request is complete */
|
||||
uds_request_callback_fn callback;
|
||||
/* The index session that will manage this request */
|
||||
struct uds_index_session *session;
|
||||
/* The type of operation to perform, as describe above */
|
||||
enum uds_request_type type;
|
||||
|
||||
/* These output fields are set when a request is complete. */
|
||||
|
||||
/* The existing data associated with the request name, if any */
|
||||
struct uds_record_data old_metadata;
|
||||
/* Either UDS_SUCCESS or an error code for the request */
|
||||
int status;
|
||||
/* True if the record name had an existing entry in the index */
|
||||
bool found;
|
||||
|
||||
/*
|
||||
* The remaining fields are used internally and should not be altered by clients. The index
|
||||
* relies on zone_number being the first field in this section.
|
||||
*/
|
||||
|
||||
/* The number of the zone which will process this request*/
|
||||
unsigned int zone_number;
|
||||
/* A link for adding a request to a lock-free queue */
|
||||
struct funnel_queue_entry queue_link;
|
||||
/* A link for adding a request to a standard linked list */
|
||||
struct uds_request *next_request;
|
||||
/* A pointer to the index processing this request */
|
||||
struct uds_index *index;
|
||||
/* Control message for coordinating between zones */
|
||||
struct uds_zone_message zone_message;
|
||||
/* If true, process request immediately by waking the worker thread */
|
||||
bool unbatched;
|
||||
/* If true, continue this request before processing newer requests */
|
||||
bool requeued;
|
||||
/* The virtual chapter containing the record name, if known */
|
||||
u64 virtual_chapter;
|
||||
/* The region of the index containing the record name */
|
||||
enum uds_index_region location;
|
||||
};
|
||||
|
||||
/* Compute the number of bytes needed to store an index. */
|
||||
int __must_check uds_compute_index_size(const struct uds_parameters *parameters,
|
||||
u64 *index_size);
|
||||
|
||||
/* A session is required for most index operations. */
|
||||
int __must_check uds_create_index_session(struct uds_index_session **session);
|
||||
|
||||
/* Destroying an index session also closes and saves the associated index. */
|
||||
int uds_destroy_index_session(struct uds_index_session *session);
|
||||
|
||||
/*
|
||||
* Create or open an index with an existing session. This operation fails if the index session is
|
||||
* suspended, or if there is already an open index.
|
||||
*/
|
||||
int __must_check uds_open_index(enum uds_open_index_type open_type,
|
||||
const struct uds_parameters *parameters,
|
||||
struct uds_index_session *session);
|
||||
|
||||
/*
|
||||
* Wait until all callbacks for index operations are complete, and prevent new index operations
|
||||
* from starting. New index operations will fail with EBUSY until the session is resumed. Also
|
||||
* optionally saves the index.
|
||||
*/
|
||||
int __must_check uds_suspend_index_session(struct uds_index_session *session, bool save);
|
||||
|
||||
/*
|
||||
* Allow new index operations for an index, whether it was suspended or not. If the index is
|
||||
* suspended and the supplied block device differs from the current backing store, the index will
|
||||
* start using the new backing store instead.
|
||||
*/
|
||||
int __must_check uds_resume_index_session(struct uds_index_session *session,
|
||||
struct block_device *bdev);
|
||||
|
||||
/* Wait until all outstanding index operations are complete. */
|
||||
int __must_check uds_flush_index_session(struct uds_index_session *session);
|
||||
|
||||
/* Close an index. This operation fails if the index session is suspended. */
|
||||
int __must_check uds_close_index(struct uds_index_session *session);
|
||||
|
||||
/* Get index statistics since the last time the index was opened. */
|
||||
int __must_check uds_get_index_session_stats(struct uds_index_session *session,
|
||||
struct uds_index_stats *stats);
|
||||
|
||||
/* This function will fail if any required field of the request is not set. */
|
||||
int __must_check uds_launch_request(struct uds_request *request);
|
||||
|
||||
struct cond_var {
|
||||
wait_queue_head_t wait_queue;
|
||||
};
|
||||
|
||||
static inline void uds_init_cond(struct cond_var *cv)
|
||||
{
|
||||
init_waitqueue_head(&cv->wait_queue);
|
||||
}
|
||||
|
||||
static inline void uds_signal_cond(struct cond_var *cv)
|
||||
{
|
||||
wake_up(&cv->wait_queue);
|
||||
}
|
||||
|
||||
static inline void uds_broadcast_cond(struct cond_var *cv)
|
||||
{
|
||||
wake_up_all(&cv->wait_queue);
|
||||
}
|
||||
|
||||
void uds_wait_cond(struct cond_var *cv, struct mutex *mutex);
|
||||
|
||||
#endif /* INDEXER_H */
|
|
@ -0,0 +1,415 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "io-factory.h"
|
||||
|
||||
#include <linux/atomic.h>
|
||||
#include <linux/blkdev.h>
|
||||
#include <linux/err.h>
|
||||
#include <linux/mount.h>
|
||||
|
||||
#include "logger.h"
|
||||
#include "memory-alloc.h"
|
||||
#include "numeric.h"
|
||||
|
||||
/*
|
||||
* The I/O factory object manages access to index storage, which is a contiguous range of blocks on
|
||||
* a block device.
|
||||
*
|
||||
* The factory holds the open device and is responsible for closing it. The factory has methods to
|
||||
* make helper structures that can be used to access sections of the index.
|
||||
*/
|
||||
struct io_factory {
|
||||
struct block_device *bdev;
|
||||
atomic_t ref_count;
|
||||
};
|
||||
|
||||
/* The buffered reader allows efficient I/O by reading page-sized segments into a buffer. */
|
||||
struct buffered_reader {
|
||||
struct io_factory *factory;
|
||||
struct dm_bufio_client *client;
|
||||
struct dm_buffer *buffer;
|
||||
sector_t limit;
|
||||
sector_t block_number;
|
||||
u8 *start;
|
||||
u8 *end;
|
||||
};
|
||||
|
||||
#define MAX_READ_AHEAD_BLOCKS 4
|
||||
|
||||
/*
|
||||
* The buffered writer allows efficient I/O by buffering writes and committing page-sized segments
|
||||
* to storage.
|
||||
*/
|
||||
struct buffered_writer {
|
||||
struct io_factory *factory;
|
||||
struct dm_bufio_client *client;
|
||||
struct dm_buffer *buffer;
|
||||
sector_t limit;
|
||||
sector_t block_number;
|
||||
u8 *start;
|
||||
u8 *end;
|
||||
int error;
|
||||
};
|
||||
|
||||
static void uds_get_io_factory(struct io_factory *factory)
|
||||
{
|
||||
atomic_inc(&factory->ref_count);
|
||||
}
|
||||
|
||||
int uds_make_io_factory(struct block_device *bdev, struct io_factory **factory_ptr)
|
||||
{
|
||||
int result;
|
||||
struct io_factory *factory;
|
||||
|
||||
result = vdo_allocate(1, struct io_factory, __func__, &factory);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
factory->bdev = bdev;
|
||||
atomic_set_release(&factory->ref_count, 1);
|
||||
|
||||
*factory_ptr = factory;
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
int uds_replace_storage(struct io_factory *factory, struct block_device *bdev)
|
||||
{
|
||||
factory->bdev = bdev;
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
/* Free an I/O factory once all references have been released. */
|
||||
void uds_put_io_factory(struct io_factory *factory)
|
||||
{
|
||||
if (atomic_add_return(-1, &factory->ref_count) <= 0)
|
||||
vdo_free(factory);
|
||||
}
|
||||
|
||||
size_t uds_get_writable_size(struct io_factory *factory)
|
||||
{
|
||||
return i_size_read(factory->bdev->bd_inode);
|
||||
}
|
||||
|
||||
/* Create a struct dm_bufio_client for an index region starting at offset. */
|
||||
int uds_make_bufio(struct io_factory *factory, off_t block_offset, size_t block_size,
|
||||
unsigned int reserved_buffers, struct dm_bufio_client **client_ptr)
|
||||
{
|
||||
struct dm_bufio_client *client;
|
||||
|
||||
client = dm_bufio_client_create(factory->bdev, block_size, reserved_buffers, 0,
|
||||
NULL, NULL, 0);
|
||||
if (IS_ERR(client))
|
||||
return -PTR_ERR(client);
|
||||
|
||||
dm_bufio_set_sector_offset(client, block_offset * SECTORS_PER_BLOCK);
|
||||
*client_ptr = client;
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
static void read_ahead(struct buffered_reader *reader, sector_t block_number)
|
||||
{
|
||||
if (block_number < reader->limit) {
|
||||
sector_t read_ahead = min((sector_t) MAX_READ_AHEAD_BLOCKS,
|
||||
reader->limit - block_number);
|
||||
|
||||
dm_bufio_prefetch(reader->client, block_number, read_ahead);
|
||||
}
|
||||
}
|
||||
|
||||
void uds_free_buffered_reader(struct buffered_reader *reader)
|
||||
{
|
||||
if (reader == NULL)
|
||||
return;
|
||||
|
||||
if (reader->buffer != NULL)
|
||||
dm_bufio_release(reader->buffer);
|
||||
|
||||
dm_bufio_client_destroy(reader->client);
|
||||
uds_put_io_factory(reader->factory);
|
||||
vdo_free(reader);
|
||||
}
|
||||
|
||||
/* Create a buffered reader for an index region starting at offset. */
|
||||
int uds_make_buffered_reader(struct io_factory *factory, off_t offset, u64 block_count,
|
||||
struct buffered_reader **reader_ptr)
|
||||
{
|
||||
int result;
|
||||
struct dm_bufio_client *client = NULL;
|
||||
struct buffered_reader *reader = NULL;
|
||||
|
||||
result = uds_make_bufio(factory, offset, UDS_BLOCK_SIZE, 1, &client);
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
result = vdo_allocate(1, struct buffered_reader, "buffered reader", &reader);
|
||||
if (result != VDO_SUCCESS) {
|
||||
dm_bufio_client_destroy(client);
|
||||
return result;
|
||||
}
|
||||
|
||||
*reader = (struct buffered_reader) {
|
||||
.factory = factory,
|
||||
.client = client,
|
||||
.buffer = NULL,
|
||||
.limit = block_count,
|
||||
.block_number = 0,
|
||||
.start = NULL,
|
||||
.end = NULL,
|
||||
};
|
||||
|
||||
read_ahead(reader, 0);
|
||||
uds_get_io_factory(factory);
|
||||
*reader_ptr = reader;
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
static int position_reader(struct buffered_reader *reader, sector_t block_number,
|
||||
off_t offset)
|
||||
{
|
||||
struct dm_buffer *buffer = NULL;
|
||||
void *data;
|
||||
|
||||
if ((reader->end == NULL) || (block_number != reader->block_number)) {
|
||||
if (block_number >= reader->limit)
|
||||
return UDS_OUT_OF_RANGE;
|
||||
|
||||
if (reader->buffer != NULL)
|
||||
dm_bufio_release(vdo_forget(reader->buffer));
|
||||
|
||||
data = dm_bufio_read(reader->client, block_number, &buffer);
|
||||
if (IS_ERR(data))
|
||||
return -PTR_ERR(data);
|
||||
|
||||
reader->buffer = buffer;
|
||||
reader->start = data;
|
||||
if (block_number == reader->block_number + 1)
|
||||
read_ahead(reader, block_number + 1);
|
||||
}
|
||||
|
||||
reader->block_number = block_number;
|
||||
reader->end = reader->start + offset;
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
static size_t bytes_remaining_in_read_buffer(struct buffered_reader *reader)
|
||||
{
|
||||
return (reader->end == NULL) ? 0 : reader->start + UDS_BLOCK_SIZE - reader->end;
|
||||
}
|
||||
|
||||
static int reset_reader(struct buffered_reader *reader)
|
||||
{
|
||||
sector_t block_number;
|
||||
|
||||
if (bytes_remaining_in_read_buffer(reader) > 0)
|
||||
return UDS_SUCCESS;
|
||||
|
||||
block_number = reader->block_number;
|
||||
if (reader->end != NULL)
|
||||
block_number++;
|
||||
|
||||
return position_reader(reader, block_number, 0);
|
||||
}
|
||||
|
||||
int uds_read_from_buffered_reader(struct buffered_reader *reader, u8 *data,
|
||||
size_t length)
|
||||
{
|
||||
int result = UDS_SUCCESS;
|
||||
size_t chunk_size;
|
||||
|
||||
while (length > 0) {
|
||||
result = reset_reader(reader);
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
chunk_size = min(length, bytes_remaining_in_read_buffer(reader));
|
||||
memcpy(data, reader->end, chunk_size);
|
||||
length -= chunk_size;
|
||||
data += chunk_size;
|
||||
reader->end += chunk_size;
|
||||
}
|
||||
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
/*
|
||||
* Verify that the next data on the reader matches the required value. If the value matches, the
|
||||
* matching contents are consumed. If the value does not match, the reader state is unchanged.
|
||||
*/
|
||||
int uds_verify_buffered_data(struct buffered_reader *reader, const u8 *value,
|
||||
size_t length)
|
||||
{
|
||||
int result = UDS_SUCCESS;
|
||||
size_t chunk_size;
|
||||
sector_t start_block_number = reader->block_number;
|
||||
int start_offset = reader->end - reader->start;
|
||||
|
||||
while (length > 0) {
|
||||
result = reset_reader(reader);
|
||||
if (result != UDS_SUCCESS) {
|
||||
result = UDS_CORRUPT_DATA;
|
||||
break;
|
||||
}
|
||||
|
||||
chunk_size = min(length, bytes_remaining_in_read_buffer(reader));
|
||||
if (memcmp(value, reader->end, chunk_size) != 0) {
|
||||
result = UDS_CORRUPT_DATA;
|
||||
break;
|
||||
}
|
||||
|
||||
length -= chunk_size;
|
||||
value += chunk_size;
|
||||
reader->end += chunk_size;
|
||||
}
|
||||
|
||||
if (result != UDS_SUCCESS)
|
||||
position_reader(reader, start_block_number, start_offset);
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
/* Create a buffered writer for an index region starting at offset. */
|
||||
int uds_make_buffered_writer(struct io_factory *factory, off_t offset, u64 block_count,
|
||||
struct buffered_writer **writer_ptr)
|
||||
{
|
||||
int result;
|
||||
struct dm_bufio_client *client = NULL;
|
||||
struct buffered_writer *writer;
|
||||
|
||||
result = uds_make_bufio(factory, offset, UDS_BLOCK_SIZE, 1, &client);
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
result = vdo_allocate(1, struct buffered_writer, "buffered writer", &writer);
|
||||
if (result != VDO_SUCCESS) {
|
||||
dm_bufio_client_destroy(client);
|
||||
return result;
|
||||
}
|
||||
|
||||
*writer = (struct buffered_writer) {
|
||||
.factory = factory,
|
||||
.client = client,
|
||||
.buffer = NULL,
|
||||
.limit = block_count,
|
||||
.start = NULL,
|
||||
.end = NULL,
|
||||
.block_number = 0,
|
||||
.error = UDS_SUCCESS,
|
||||
};
|
||||
|
||||
uds_get_io_factory(factory);
|
||||
*writer_ptr = writer;
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
static size_t get_remaining_write_space(struct buffered_writer *writer)
|
||||
{
|
||||
return writer->start + UDS_BLOCK_SIZE - writer->end;
|
||||
}
|
||||
|
||||
static int __must_check prepare_next_buffer(struct buffered_writer *writer)
|
||||
{
|
||||
struct dm_buffer *buffer = NULL;
|
||||
void *data;
|
||||
|
||||
if (writer->block_number >= writer->limit) {
|
||||
writer->error = UDS_OUT_OF_RANGE;
|
||||
return UDS_OUT_OF_RANGE;
|
||||
}
|
||||
|
||||
data = dm_bufio_new(writer->client, writer->block_number, &buffer);
|
||||
if (IS_ERR(data)) {
|
||||
writer->error = -PTR_ERR(data);
|
||||
return writer->error;
|
||||
}
|
||||
|
||||
writer->buffer = buffer;
|
||||
writer->start = data;
|
||||
writer->end = data;
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
static int flush_previous_buffer(struct buffered_writer *writer)
|
||||
{
|
||||
size_t available;
|
||||
|
||||
if (writer->buffer == NULL)
|
||||
return writer->error;
|
||||
|
||||
if (writer->error == UDS_SUCCESS) {
|
||||
available = get_remaining_write_space(writer);
|
||||
|
||||
if (available > 0)
|
||||
memset(writer->end, 0, available);
|
||||
|
||||
dm_bufio_mark_buffer_dirty(writer->buffer);
|
||||
}
|
||||
|
||||
dm_bufio_release(writer->buffer);
|
||||
writer->buffer = NULL;
|
||||
writer->start = NULL;
|
||||
writer->end = NULL;
|
||||
writer->block_number++;
|
||||
return writer->error;
|
||||
}
|
||||
|
||||
void uds_free_buffered_writer(struct buffered_writer *writer)
|
||||
{
|
||||
int result;
|
||||
|
||||
if (writer == NULL)
|
||||
return;
|
||||
|
||||
flush_previous_buffer(writer);
|
||||
result = -dm_bufio_write_dirty_buffers(writer->client);
|
||||
if (result != UDS_SUCCESS)
|
||||
vdo_log_warning_strerror(result, "%s: failed to sync storage", __func__);
|
||||
|
||||
dm_bufio_client_destroy(writer->client);
|
||||
uds_put_io_factory(writer->factory);
|
||||
vdo_free(writer);
|
||||
}
|
||||
|
||||
/*
|
||||
* Append data to the buffer, writing as needed. If no data is provided, zeros are written instead.
|
||||
* If a write error occurs, it is recorded and returned on every subsequent write attempt.
|
||||
*/
|
||||
int uds_write_to_buffered_writer(struct buffered_writer *writer, const u8 *data,
|
||||
size_t length)
|
||||
{
|
||||
int result = writer->error;
|
||||
size_t chunk_size;
|
||||
|
||||
while ((length > 0) && (result == UDS_SUCCESS)) {
|
||||
if (writer->buffer == NULL) {
|
||||
result = prepare_next_buffer(writer);
|
||||
continue;
|
||||
}
|
||||
|
||||
chunk_size = min(length, get_remaining_write_space(writer));
|
||||
if (data == NULL) {
|
||||
memset(writer->end, 0, chunk_size);
|
||||
} else {
|
||||
memcpy(writer->end, data, chunk_size);
|
||||
data += chunk_size;
|
||||
}
|
||||
|
||||
length -= chunk_size;
|
||||
writer->end += chunk_size;
|
||||
|
||||
if (get_remaining_write_space(writer) == 0)
|
||||
result = uds_flush_buffered_writer(writer);
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
int uds_flush_buffered_writer(struct buffered_writer *writer)
|
||||
{
|
||||
if (writer->error != UDS_SUCCESS)
|
||||
return writer->error;
|
||||
|
||||
return flush_previous_buffer(writer);
|
||||
}
|
|
@ -0,0 +1,64 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef UDS_IO_FACTORY_H
|
||||
#define UDS_IO_FACTORY_H
|
||||
|
||||
#include <linux/dm-bufio.h>
|
||||
|
||||
/*
|
||||
* The I/O factory manages all low-level I/O operations to the underlying storage device. Its main
|
||||
* clients are the index layout and the volume. The buffered reader and buffered writer interfaces
|
||||
* are helpers for accessing data in a contiguous range of storage blocks.
|
||||
*/
|
||||
|
||||
struct buffered_reader;
|
||||
struct buffered_writer;
|
||||
|
||||
struct io_factory;
|
||||
|
||||
enum {
|
||||
UDS_BLOCK_SIZE = 4096,
|
||||
SECTORS_PER_BLOCK = UDS_BLOCK_SIZE >> SECTOR_SHIFT,
|
||||
};
|
||||
|
||||
int __must_check uds_make_io_factory(struct block_device *bdev,
|
||||
struct io_factory **factory_ptr);
|
||||
|
||||
int __must_check uds_replace_storage(struct io_factory *factory,
|
||||
struct block_device *bdev);
|
||||
|
||||
void uds_put_io_factory(struct io_factory *factory);
|
||||
|
||||
size_t __must_check uds_get_writable_size(struct io_factory *factory);
|
||||
|
||||
int __must_check uds_make_bufio(struct io_factory *factory, off_t block_offset,
|
||||
size_t block_size, unsigned int reserved_buffers,
|
||||
struct dm_bufio_client **client_ptr);
|
||||
|
||||
int __must_check uds_make_buffered_reader(struct io_factory *factory, off_t offset,
|
||||
u64 block_count,
|
||||
struct buffered_reader **reader_ptr);
|
||||
|
||||
void uds_free_buffered_reader(struct buffered_reader *reader);
|
||||
|
||||
int __must_check uds_read_from_buffered_reader(struct buffered_reader *reader, u8 *data,
|
||||
size_t length);
|
||||
|
||||
int __must_check uds_verify_buffered_data(struct buffered_reader *reader, const u8 *value,
|
||||
size_t length);
|
||||
|
||||
int __must_check uds_make_buffered_writer(struct io_factory *factory, off_t offset,
|
||||
u64 block_count,
|
||||
struct buffered_writer **writer_ptr);
|
||||
|
||||
void uds_free_buffered_writer(struct buffered_writer *buffer);
|
||||
|
||||
int __must_check uds_write_to_buffered_writer(struct buffered_writer *writer,
|
||||
const u8 *data, size_t length);
|
||||
|
||||
int __must_check uds_flush_buffered_writer(struct buffered_writer *writer);
|
||||
|
||||
#endif /* UDS_IO_FACTORY_H */
|
|
@ -0,0 +1,426 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "open-chapter.h"
|
||||
|
||||
#include <linux/log2.h>
|
||||
|
||||
#include "logger.h"
|
||||
#include "memory-alloc.h"
|
||||
#include "numeric.h"
|
||||
#include "permassert.h"
|
||||
|
||||
#include "config.h"
|
||||
#include "hash-utils.h"
|
||||
|
||||
/*
|
||||
* Each index zone has a dedicated open chapter zone structure which gets an equal share of the
|
||||
* open chapter space. Records are assigned to zones based on their record name. Within each zone,
|
||||
* records are stored in an array in the order they arrive. Additionally, a reference to each
|
||||
* record is stored in a hash table to help determine if a new record duplicates an existing one.
|
||||
* If new metadata for an existing name arrives, the record is altered in place. The array of
|
||||
* records is 1-based so that record number 0 can be used to indicate an unused hash slot.
|
||||
*
|
||||
* Deleted records are marked with a flag rather than actually removed to simplify hash table
|
||||
* management. The array of deleted flags overlays the array of hash slots, but the flags are
|
||||
* indexed by record number instead of by record name. The number of hash slots will always be a
|
||||
* power of two that is greater than the number of records to be indexed, guaranteeing that hash
|
||||
* insertion cannot fail, and that there are sufficient flags for all records.
|
||||
*
|
||||
* Once any open chapter zone fills its available space, the chapter is closed. The records from
|
||||
* each zone are interleaved to attempt to preserve temporal locality and assigned to record pages.
|
||||
* Empty or deleted records are replaced by copies of a valid record so that the record pages only
|
||||
* contain valid records. The chapter then constructs a delta index which maps each record name to
|
||||
* the record page on which that record can be found, which is split into index pages. These
|
||||
* structures are then passed to the volume to be recorded on storage.
|
||||
*
|
||||
* When the index is saved, the open chapter records are saved in a single array, once again
|
||||
* interleaved to attempt to preserve temporal locality. When the index is reloaded, there may be a
|
||||
* different number of zones than previously, so the records must be parcelled out to their new
|
||||
* zones. In addition, depending on the distribution of record names, a new zone may have more
|
||||
* records than it has space. In this case, the latest records for that zone will be discarded.
|
||||
*/
|
||||
|
||||
static const u8 OPEN_CHAPTER_MAGIC[] = "ALBOC";
|
||||
static const u8 OPEN_CHAPTER_VERSION[] = "02.00";
|
||||
|
||||
#define OPEN_CHAPTER_MAGIC_LENGTH (sizeof(OPEN_CHAPTER_MAGIC) - 1)
|
||||
#define OPEN_CHAPTER_VERSION_LENGTH (sizeof(OPEN_CHAPTER_VERSION) - 1)
|
||||
#define LOAD_RATIO 2
|
||||
|
||||
static inline size_t records_size(const struct open_chapter_zone *open_chapter)
|
||||
{
|
||||
return sizeof(struct uds_volume_record) * (1 + open_chapter->capacity);
|
||||
}
|
||||
|
||||
static inline size_t slots_size(size_t slot_count)
|
||||
{
|
||||
return sizeof(struct open_chapter_zone_slot) * slot_count;
|
||||
}
|
||||
|
||||
int uds_make_open_chapter(const struct index_geometry *geometry, unsigned int zone_count,
|
||||
struct open_chapter_zone **open_chapter_ptr)
|
||||
{
|
||||
int result;
|
||||
struct open_chapter_zone *open_chapter;
|
||||
size_t capacity = geometry->records_per_chapter / zone_count;
|
||||
size_t slot_count = (1 << bits_per(capacity * LOAD_RATIO));
|
||||
|
||||
result = vdo_allocate_extended(struct open_chapter_zone, slot_count,
|
||||
struct open_chapter_zone_slot, "open chapter",
|
||||
&open_chapter);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
open_chapter->slot_count = slot_count;
|
||||
open_chapter->capacity = capacity;
|
||||
result = vdo_allocate_cache_aligned(records_size(open_chapter), "record pages",
|
||||
&open_chapter->records);
|
||||
if (result != VDO_SUCCESS) {
|
||||
uds_free_open_chapter(open_chapter);
|
||||
return result;
|
||||
}
|
||||
|
||||
*open_chapter_ptr = open_chapter;
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
void uds_reset_open_chapter(struct open_chapter_zone *open_chapter)
|
||||
{
|
||||
open_chapter->size = 0;
|
||||
open_chapter->deletions = 0;
|
||||
|
||||
memset(open_chapter->records, 0, records_size(open_chapter));
|
||||
memset(open_chapter->slots, 0, slots_size(open_chapter->slot_count));
|
||||
}
|
||||
|
||||
static unsigned int probe_chapter_slots(struct open_chapter_zone *open_chapter,
|
||||
const struct uds_record_name *name)
|
||||
{
|
||||
struct uds_volume_record *record;
|
||||
unsigned int slot_count = open_chapter->slot_count;
|
||||
unsigned int slot = uds_name_to_hash_slot(name, slot_count);
|
||||
unsigned int record_number;
|
||||
unsigned int attempts = 1;
|
||||
|
||||
while (true) {
|
||||
record_number = open_chapter->slots[slot].record_number;
|
||||
|
||||
/*
|
||||
* If the hash slot is empty, we've reached the end of a chain without finding the
|
||||
* record and should terminate the search.
|
||||
*/
|
||||
if (record_number == 0)
|
||||
return slot;
|
||||
|
||||
/*
|
||||
* If the name of the record referenced by the slot matches and has not been
|
||||
* deleted, then we've found the requested name.
|
||||
*/
|
||||
record = &open_chapter->records[record_number];
|
||||
if ((memcmp(&record->name, name, UDS_RECORD_NAME_SIZE) == 0) &&
|
||||
!open_chapter->slots[record_number].deleted)
|
||||
return slot;
|
||||
|
||||
/*
|
||||
* Quadratic probing: advance the probe by 1, 2, 3, etc. and try again. This
|
||||
* performs better than linear probing and works best for 2^N slots.
|
||||
*/
|
||||
slot = (slot + attempts++) % slot_count;
|
||||
}
|
||||
}
|
||||
|
||||
void uds_search_open_chapter(struct open_chapter_zone *open_chapter,
|
||||
const struct uds_record_name *name,
|
||||
struct uds_record_data *metadata, bool *found)
|
||||
{
|
||||
unsigned int slot;
|
||||
unsigned int record_number;
|
||||
|
||||
slot = probe_chapter_slots(open_chapter, name);
|
||||
record_number = open_chapter->slots[slot].record_number;
|
||||
if (record_number == 0) {
|
||||
*found = false;
|
||||
} else {
|
||||
*found = true;
|
||||
*metadata = open_chapter->records[record_number].data;
|
||||
}
|
||||
}
|
||||
|
||||
/* Add a record to the open chapter zone and return the remaining space. */
|
||||
int uds_put_open_chapter(struct open_chapter_zone *open_chapter,
|
||||
const struct uds_record_name *name,
|
||||
const struct uds_record_data *metadata)
|
||||
{
|
||||
unsigned int slot;
|
||||
unsigned int record_number;
|
||||
struct uds_volume_record *record;
|
||||
|
||||
if (open_chapter->size >= open_chapter->capacity)
|
||||
return 0;
|
||||
|
||||
slot = probe_chapter_slots(open_chapter, name);
|
||||
record_number = open_chapter->slots[slot].record_number;
|
||||
|
||||
if (record_number == 0) {
|
||||
record_number = ++open_chapter->size;
|
||||
open_chapter->slots[slot].record_number = record_number;
|
||||
}
|
||||
|
||||
record = &open_chapter->records[record_number];
|
||||
record->name = *name;
|
||||
record->data = *metadata;
|
||||
|
||||
return open_chapter->capacity - open_chapter->size;
|
||||
}
|
||||
|
||||
void uds_remove_from_open_chapter(struct open_chapter_zone *open_chapter,
|
||||
const struct uds_record_name *name)
|
||||
{
|
||||
unsigned int slot;
|
||||
unsigned int record_number;
|
||||
|
||||
slot = probe_chapter_slots(open_chapter, name);
|
||||
record_number = open_chapter->slots[slot].record_number;
|
||||
|
||||
if (record_number > 0) {
|
||||
open_chapter->slots[record_number].deleted = true;
|
||||
open_chapter->deletions += 1;
|
||||
}
|
||||
}
|
||||
|
||||
void uds_free_open_chapter(struct open_chapter_zone *open_chapter)
|
||||
{
|
||||
if (open_chapter != NULL) {
|
||||
vdo_free(open_chapter->records);
|
||||
vdo_free(open_chapter);
|
||||
}
|
||||
}
|
||||
|
||||
/* Map each record name to its record page number in the delta chapter index. */
|
||||
static int fill_delta_chapter_index(struct open_chapter_zone **chapter_zones,
|
||||
unsigned int zone_count,
|
||||
struct open_chapter_index *index,
|
||||
struct uds_volume_record *collated_records)
|
||||
{
|
||||
int result;
|
||||
unsigned int records_per_chapter;
|
||||
unsigned int records_per_page;
|
||||
unsigned int record_index;
|
||||
unsigned int records = 0;
|
||||
u32 page_number;
|
||||
unsigned int z;
|
||||
int overflow_count = 0;
|
||||
struct uds_volume_record *fill_record = NULL;
|
||||
|
||||
/*
|
||||
* The record pages should not have any empty space, so find a record with which to fill
|
||||
* the chapter zone if it was closed early, and also to replace any deleted records. The
|
||||
* last record in any filled zone is guaranteed to not have been deleted, so use one of
|
||||
* those.
|
||||
*/
|
||||
for (z = 0; z < zone_count; z++) {
|
||||
struct open_chapter_zone *zone = chapter_zones[z];
|
||||
|
||||
if (zone->size == zone->capacity) {
|
||||
fill_record = &zone->records[zone->size];
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
records_per_chapter = index->geometry->records_per_chapter;
|
||||
records_per_page = index->geometry->records_per_page;
|
||||
|
||||
for (records = 0; records < records_per_chapter; records++) {
|
||||
struct uds_volume_record *record = &collated_records[records];
|
||||
struct open_chapter_zone *open_chapter;
|
||||
|
||||
/* The record arrays in the zones are 1-based. */
|
||||
record_index = 1 + (records / zone_count);
|
||||
page_number = records / records_per_page;
|
||||
open_chapter = chapter_zones[records % zone_count];
|
||||
|
||||
/* Use the fill record in place of an unused record. */
|
||||
if (record_index > open_chapter->size ||
|
||||
open_chapter->slots[record_index].deleted) {
|
||||
*record = *fill_record;
|
||||
continue;
|
||||
}
|
||||
|
||||
*record = open_chapter->records[record_index];
|
||||
result = uds_put_open_chapter_index_record(index, &record->name,
|
||||
page_number);
|
||||
switch (result) {
|
||||
case UDS_SUCCESS:
|
||||
break;
|
||||
case UDS_OVERFLOW:
|
||||
overflow_count++;
|
||||
break;
|
||||
default:
|
||||
vdo_log_error_strerror(result,
|
||||
"failed to build open chapter index");
|
||||
return result;
|
||||
}
|
||||
}
|
||||
|
||||
if (overflow_count > 0)
|
||||
vdo_log_warning("Failed to add %d entries to chapter index",
|
||||
overflow_count);
|
||||
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
int uds_close_open_chapter(struct open_chapter_zone **chapter_zones,
|
||||
unsigned int zone_count, struct volume *volume,
|
||||
struct open_chapter_index *chapter_index,
|
||||
struct uds_volume_record *collated_records,
|
||||
u64 virtual_chapter_number)
|
||||
{
|
||||
int result;
|
||||
|
||||
uds_empty_open_chapter_index(chapter_index, virtual_chapter_number);
|
||||
result = fill_delta_chapter_index(chapter_zones, zone_count, chapter_index,
|
||||
collated_records);
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
return uds_write_chapter(volume, chapter_index, collated_records);
|
||||
}
|
||||
|
||||
int uds_save_open_chapter(struct uds_index *index, struct buffered_writer *writer)
|
||||
{
|
||||
int result;
|
||||
struct open_chapter_zone *open_chapter;
|
||||
struct uds_volume_record *record;
|
||||
u8 record_count_data[sizeof(u32)];
|
||||
u32 record_count = 0;
|
||||
unsigned int record_index;
|
||||
unsigned int z;
|
||||
|
||||
result = uds_write_to_buffered_writer(writer, OPEN_CHAPTER_MAGIC,
|
||||
OPEN_CHAPTER_MAGIC_LENGTH);
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
result = uds_write_to_buffered_writer(writer, OPEN_CHAPTER_VERSION,
|
||||
OPEN_CHAPTER_VERSION_LENGTH);
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
for (z = 0; z < index->zone_count; z++) {
|
||||
open_chapter = index->zones[z]->open_chapter;
|
||||
record_count += open_chapter->size - open_chapter->deletions;
|
||||
}
|
||||
|
||||
put_unaligned_le32(record_count, record_count_data);
|
||||
result = uds_write_to_buffered_writer(writer, record_count_data,
|
||||
sizeof(record_count_data));
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
record_index = 1;
|
||||
while (record_count > 0) {
|
||||
for (z = 0; z < index->zone_count; z++) {
|
||||
open_chapter = index->zones[z]->open_chapter;
|
||||
if (record_index > open_chapter->size)
|
||||
continue;
|
||||
|
||||
if (open_chapter->slots[record_index].deleted)
|
||||
continue;
|
||||
|
||||
record = &open_chapter->records[record_index];
|
||||
result = uds_write_to_buffered_writer(writer, (u8 *) record,
|
||||
sizeof(*record));
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
record_count--;
|
||||
}
|
||||
|
||||
record_index++;
|
||||
}
|
||||
|
||||
return uds_flush_buffered_writer(writer);
|
||||
}
|
||||
|
||||
u64 uds_compute_saved_open_chapter_size(struct index_geometry *geometry)
|
||||
{
|
||||
unsigned int records_per_chapter = geometry->records_per_chapter;
|
||||
|
||||
return OPEN_CHAPTER_MAGIC_LENGTH + OPEN_CHAPTER_VERSION_LENGTH + sizeof(u32) +
|
||||
records_per_chapter * sizeof(struct uds_volume_record);
|
||||
}
|
||||
|
||||
static int load_version20(struct uds_index *index, struct buffered_reader *reader)
|
||||
{
|
||||
int result;
|
||||
u32 record_count;
|
||||
u8 record_count_data[sizeof(u32)];
|
||||
struct uds_volume_record record;
|
||||
|
||||
/*
|
||||
* Track which zones cannot accept any more records. If the open chapter had a different
|
||||
* number of zones previously, some new zones may have more records than they have space
|
||||
* for. These overflow records will be discarded.
|
||||
*/
|
||||
bool full_flags[MAX_ZONES] = {
|
||||
false,
|
||||
};
|
||||
|
||||
result = uds_read_from_buffered_reader(reader, (u8 *) &record_count_data,
|
||||
sizeof(record_count_data));
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
record_count = get_unaligned_le32(record_count_data);
|
||||
while (record_count-- > 0) {
|
||||
unsigned int zone = 0;
|
||||
|
||||
result = uds_read_from_buffered_reader(reader, (u8 *) &record,
|
||||
sizeof(record));
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
if (index->zone_count > 1)
|
||||
zone = uds_get_volume_index_zone(index->volume_index,
|
||||
&record.name);
|
||||
|
||||
if (!full_flags[zone]) {
|
||||
struct open_chapter_zone *open_chapter;
|
||||
unsigned int remaining;
|
||||
|
||||
open_chapter = index->zones[zone]->open_chapter;
|
||||
remaining = uds_put_open_chapter(open_chapter, &record.name,
|
||||
&record.data);
|
||||
/* Do not allow any zone to fill completely. */
|
||||
full_flags[zone] = (remaining <= 1);
|
||||
}
|
||||
}
|
||||
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
int uds_load_open_chapter(struct uds_index *index, struct buffered_reader *reader)
|
||||
{
|
||||
u8 version[OPEN_CHAPTER_VERSION_LENGTH];
|
||||
int result;
|
||||
|
||||
result = uds_verify_buffered_data(reader, OPEN_CHAPTER_MAGIC,
|
||||
OPEN_CHAPTER_MAGIC_LENGTH);
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
result = uds_read_from_buffered_reader(reader, version, sizeof(version));
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
if (memcmp(OPEN_CHAPTER_VERSION, version, sizeof(version)) != 0) {
|
||||
return vdo_log_error_strerror(UDS_CORRUPT_DATA,
|
||||
"Invalid open chapter version: %.*s",
|
||||
(int) sizeof(version), version);
|
||||
}
|
||||
|
||||
return load_version20(index, reader);
|
||||
}
|
|
@ -0,0 +1,79 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef UDS_OPEN_CHAPTER_H
|
||||
#define UDS_OPEN_CHAPTER_H
|
||||
|
||||
#include "chapter-index.h"
|
||||
#include "geometry.h"
|
||||
#include "index.h"
|
||||
#include "volume.h"
|
||||
|
||||
/*
|
||||
* The open chapter tracks the newest records in memory. Like the index as a whole, each open
|
||||
* chapter is divided into a number of independent zones which are interleaved when the chapter is
|
||||
* committed to the volume.
|
||||
*/
|
||||
|
||||
enum {
|
||||
OPEN_CHAPTER_RECORD_NUMBER_BITS = 23,
|
||||
};
|
||||
|
||||
struct open_chapter_zone_slot {
|
||||
/* If non-zero, the record number addressed by this hash slot */
|
||||
unsigned int record_number : OPEN_CHAPTER_RECORD_NUMBER_BITS;
|
||||
/* If true, the record at the index of this hash slot was deleted */
|
||||
bool deleted : 1;
|
||||
} __packed;
|
||||
|
||||
struct open_chapter_zone {
|
||||
/* The maximum number of records that can be stored */
|
||||
unsigned int capacity;
|
||||
/* The number of records stored */
|
||||
unsigned int size;
|
||||
/* The number of deleted records */
|
||||
unsigned int deletions;
|
||||
/* Array of chunk records, 1-based */
|
||||
struct uds_volume_record *records;
|
||||
/* The number of slots in the hash table */
|
||||
unsigned int slot_count;
|
||||
/* The hash table slots, referencing virtual record numbers */
|
||||
struct open_chapter_zone_slot slots[];
|
||||
};
|
||||
|
||||
int __must_check uds_make_open_chapter(const struct index_geometry *geometry,
|
||||
unsigned int zone_count,
|
||||
struct open_chapter_zone **open_chapter_ptr);
|
||||
|
||||
void uds_reset_open_chapter(struct open_chapter_zone *open_chapter);
|
||||
|
||||
void uds_search_open_chapter(struct open_chapter_zone *open_chapter,
|
||||
const struct uds_record_name *name,
|
||||
struct uds_record_data *metadata, bool *found);
|
||||
|
||||
int __must_check uds_put_open_chapter(struct open_chapter_zone *open_chapter,
|
||||
const struct uds_record_name *name,
|
||||
const struct uds_record_data *metadata);
|
||||
|
||||
void uds_remove_from_open_chapter(struct open_chapter_zone *open_chapter,
|
||||
const struct uds_record_name *name);
|
||||
|
||||
void uds_free_open_chapter(struct open_chapter_zone *open_chapter);
|
||||
|
||||
int __must_check uds_close_open_chapter(struct open_chapter_zone **chapter_zones,
|
||||
unsigned int zone_count, struct volume *volume,
|
||||
struct open_chapter_index *chapter_index,
|
||||
struct uds_volume_record *collated_records,
|
||||
u64 virtual_chapter_number);
|
||||
|
||||
int __must_check uds_save_open_chapter(struct uds_index *index,
|
||||
struct buffered_writer *writer);
|
||||
|
||||
int __must_check uds_load_open_chapter(struct uds_index *index,
|
||||
struct buffered_reader *reader);
|
||||
|
||||
u64 uds_compute_saved_open_chapter_size(struct index_geometry *geometry);
|
||||
|
||||
#endif /* UDS_OPEN_CHAPTER_H */
|
|
@ -0,0 +1,330 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "radix-sort.h"
|
||||
|
||||
#include <linux/limits.h>
|
||||
#include <linux/types.h>
|
||||
|
||||
#include "memory-alloc.h"
|
||||
#include "string-utils.h"
|
||||
|
||||
/*
|
||||
* This implementation allocates one large object to do the sorting, which can be reused as many
|
||||
* times as desired. The amount of memory required is logarithmically proportional to the number of
|
||||
* keys to be sorted.
|
||||
*/
|
||||
|
||||
/* Piles smaller than this are handled with a simple insertion sort. */
|
||||
#define INSERTION_SORT_THRESHOLD 12
|
||||
|
||||
/* Sort keys are pointers to immutable fixed-length arrays of bytes. */
|
||||
typedef const u8 *sort_key_t;
|
||||
|
||||
/*
|
||||
* The keys are separated into piles based on the byte in each keys at the current offset, so the
|
||||
* number of keys with each byte must be counted.
|
||||
*/
|
||||
struct histogram {
|
||||
/* The number of non-empty bins */
|
||||
u16 used;
|
||||
/* The index (key byte) of the first non-empty bin */
|
||||
u16 first;
|
||||
/* The index (key byte) of the last non-empty bin */
|
||||
u16 last;
|
||||
/* The number of occurrences of each specific byte */
|
||||
u32 size[256];
|
||||
};
|
||||
|
||||
/*
|
||||
* Sub-tasks are manually managed on a stack, both for performance and to put a logarithmic bound
|
||||
* on the stack space needed.
|
||||
*/
|
||||
struct task {
|
||||
/* Pointer to the first key to sort. */
|
||||
sort_key_t *first_key;
|
||||
/* Pointer to the last key to sort. */
|
||||
sort_key_t *last_key;
|
||||
/* The offset into the key at which to continue sorting. */
|
||||
u16 offset;
|
||||
/* The number of bytes remaining in the sort keys. */
|
||||
u16 length;
|
||||
};
|
||||
|
||||
struct radix_sorter {
|
||||
unsigned int count;
|
||||
struct histogram bins;
|
||||
sort_key_t *pile[256];
|
||||
struct task *end_of_stack;
|
||||
struct task insertion_list[256];
|
||||
struct task stack[];
|
||||
};
|
||||
|
||||
/* Compare a segment of two fixed-length keys starting at an offset. */
|
||||
static inline int compare(sort_key_t key1, sort_key_t key2, u16 offset, u16 length)
|
||||
{
|
||||
return memcmp(&key1[offset], &key2[offset], length);
|
||||
}
|
||||
|
||||
/* Insert the next unsorted key into an array of sorted keys. */
|
||||
static inline void insert_key(const struct task task, sort_key_t *next)
|
||||
{
|
||||
/* Pull the unsorted key out, freeing up the array slot. */
|
||||
sort_key_t unsorted = *next;
|
||||
|
||||
/* Compare the key to the preceding sorted entries, shifting down ones that are larger. */
|
||||
while ((--next >= task.first_key) &&
|
||||
(compare(unsorted, next[0], task.offset, task.length) < 0))
|
||||
next[1] = next[0];
|
||||
|
||||
/* Insert the key into the last slot that was cleared, sorting it. */
|
||||
next[1] = unsorted;
|
||||
}
|
||||
|
||||
/*
|
||||
* Sort a range of key segments using an insertion sort. This simple sort is faster than the
|
||||
* 256-way radix sort when the number of keys to sort is small.
|
||||
*/
|
||||
static inline void insertion_sort(const struct task task)
|
||||
{
|
||||
sort_key_t *next;
|
||||
|
||||
for (next = task.first_key + 1; next <= task.last_key; next++)
|
||||
insert_key(task, next);
|
||||
}
|
||||
|
||||
/* Push a sorting task onto a task stack. */
|
||||
static inline void push_task(struct task **stack_pointer, sort_key_t *first_key,
|
||||
u32 count, u16 offset, u16 length)
|
||||
{
|
||||
struct task *task = (*stack_pointer)++;
|
||||
|
||||
task->first_key = first_key;
|
||||
task->last_key = &first_key[count - 1];
|
||||
task->offset = offset;
|
||||
task->length = length;
|
||||
}
|
||||
|
||||
static inline void swap_keys(sort_key_t *a, sort_key_t *b)
|
||||
{
|
||||
sort_key_t c = *a;
|
||||
*a = *b;
|
||||
*b = c;
|
||||
}
|
||||
|
||||
/*
|
||||
* Count the number of times each byte value appears in the arrays of keys to sort at the current
|
||||
* offset, keeping track of the number of non-empty bins, and the index of the first and last
|
||||
* non-empty bin.
|
||||
*/
|
||||
static inline void measure_bins(const struct task task, struct histogram *bins)
|
||||
{
|
||||
sort_key_t *key_ptr;
|
||||
|
||||
/*
|
||||
* Subtle invariant: bins->used and bins->size[] are zero because the sorting code clears
|
||||
* it all out as it goes. Even though this structure is re-used, we don't need to pay to
|
||||
* zero it before starting a new tally.
|
||||
*/
|
||||
bins->first = U8_MAX;
|
||||
bins->last = 0;
|
||||
|
||||
for (key_ptr = task.first_key; key_ptr <= task.last_key; key_ptr++) {
|
||||
/* Increment the count for the byte in the key at the current offset. */
|
||||
u8 bin = (*key_ptr)[task.offset];
|
||||
u32 size = ++bins->size[bin];
|
||||
|
||||
/* Track non-empty bins. */
|
||||
if (size == 1) {
|
||||
bins->used += 1;
|
||||
if (bin < bins->first)
|
||||
bins->first = bin;
|
||||
|
||||
if (bin > bins->last)
|
||||
bins->last = bin;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
* Convert the bin sizes to pointers to where each pile goes.
|
||||
*
|
||||
* pile[0] = first_key + bin->size[0],
|
||||
* pile[1] = pile[0] + bin->size[1], etc.
|
||||
*
|
||||
* After the keys are moved to the appropriate pile, we'll need to sort each of the piles by the
|
||||
* next radix position. A new task is put on the stack for each pile containing lots of keys, or a
|
||||
* new task is put on the list for each pile containing few keys.
|
||||
*
|
||||
* @stack: pointer the top of the stack
|
||||
* @end_of_stack: the end of the stack
|
||||
* @list: pointer the head of the list
|
||||
* @pile: array for pointers to the end of each pile
|
||||
* @bins: the histogram of the sizes of each pile
|
||||
* @first_key: the first key of the stack
|
||||
* @offset: the next radix position to sort by
|
||||
* @length: the number of bytes remaining in the sort keys
|
||||
*
|
||||
* Return: UDS_SUCCESS or an error code
|
||||
*/
|
||||
static inline int push_bins(struct task **stack, struct task *end_of_stack,
|
||||
struct task **list, sort_key_t *pile[],
|
||||
struct histogram *bins, sort_key_t *first_key,
|
||||
u16 offset, u16 length)
|
||||
{
|
||||
sort_key_t *pile_start = first_key;
|
||||
int bin;
|
||||
|
||||
for (bin = bins->first; ; bin++) {
|
||||
u32 size = bins->size[bin];
|
||||
|
||||
/* Skip empty piles. */
|
||||
if (size == 0)
|
||||
continue;
|
||||
|
||||
/* There's no need to sort empty keys. */
|
||||
if (length > 0) {
|
||||
if (size > INSERTION_SORT_THRESHOLD) {
|
||||
if (*stack >= end_of_stack)
|
||||
return UDS_BAD_STATE;
|
||||
|
||||
push_task(stack, pile_start, size, offset, length);
|
||||
} else if (size > 1) {
|
||||
push_task(list, pile_start, size, offset, length);
|
||||
}
|
||||
}
|
||||
|
||||
pile_start += size;
|
||||
pile[bin] = pile_start;
|
||||
if (--bins->used == 0)
|
||||
break;
|
||||
}
|
||||
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
int uds_make_radix_sorter(unsigned int count, struct radix_sorter **sorter)
|
||||
{
|
||||
int result;
|
||||
unsigned int stack_size = count / INSERTION_SORT_THRESHOLD;
|
||||
struct radix_sorter *radix_sorter;
|
||||
|
||||
result = vdo_allocate_extended(struct radix_sorter, stack_size, struct task,
|
||||
__func__, &radix_sorter);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
radix_sorter->count = count;
|
||||
radix_sorter->end_of_stack = radix_sorter->stack + stack_size;
|
||||
*sorter = radix_sorter;
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
void uds_free_radix_sorter(struct radix_sorter *sorter)
|
||||
{
|
||||
vdo_free(sorter);
|
||||
}
|
||||
|
||||
/*
|
||||
* Sort pointers to fixed-length keys (arrays of bytes) using a radix sort. The sort implementation
|
||||
* is unstable, so the relative ordering of equal keys is not preserved.
|
||||
*/
|
||||
int uds_radix_sort(struct radix_sorter *sorter, const unsigned char *keys[],
|
||||
unsigned int count, unsigned short length)
|
||||
{
|
||||
struct task start;
|
||||
struct histogram *bins = &sorter->bins;
|
||||
sort_key_t **pile = sorter->pile;
|
||||
struct task *task_stack = sorter->stack;
|
||||
|
||||
/* All zero-length keys are identical and therefore already sorted. */
|
||||
if ((count == 0) || (length == 0))
|
||||
return UDS_SUCCESS;
|
||||
|
||||
/* The initial task is to sort the entire length of all the keys. */
|
||||
start = (struct task) {
|
||||
.first_key = keys,
|
||||
.last_key = &keys[count - 1],
|
||||
.offset = 0,
|
||||
.length = length,
|
||||
};
|
||||
|
||||
if (count <= INSERTION_SORT_THRESHOLD) {
|
||||
insertion_sort(start);
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
if (count > sorter->count)
|
||||
return UDS_INVALID_ARGUMENT;
|
||||
|
||||
/*
|
||||
* Repeatedly consume a sorting task from the stack and process it, pushing new sub-tasks
|
||||
* onto the stack for each radix-sorted pile. When all tasks and sub-tasks have been
|
||||
* processed, the stack will be empty and all the keys in the starting task will be fully
|
||||
* sorted.
|
||||
*/
|
||||
for (*task_stack = start; task_stack >= sorter->stack; task_stack--) {
|
||||
const struct task task = *task_stack;
|
||||
struct task *insertion_task_list;
|
||||
int result;
|
||||
sort_key_t *fence;
|
||||
sort_key_t *end;
|
||||
|
||||
measure_bins(task, bins);
|
||||
|
||||
/*
|
||||
* Now that we know how large each bin is, generate pointers for each of the piles
|
||||
* and push a new task to sort each pile by the next radix byte.
|
||||
*/
|
||||
insertion_task_list = sorter->insertion_list;
|
||||
result = push_bins(&task_stack, sorter->end_of_stack,
|
||||
&insertion_task_list, pile, bins, task.first_key,
|
||||
task.offset + 1, task.length - 1);
|
||||
if (result != UDS_SUCCESS) {
|
||||
memset(bins, 0, sizeof(*bins));
|
||||
return result;
|
||||
}
|
||||
|
||||
/* Now bins->used is zero again. */
|
||||
|
||||
/*
|
||||
* Don't bother processing the last pile: when piles 0..N-1 are all in place, then
|
||||
* pile N must also be in place.
|
||||
*/
|
||||
end = task.last_key - bins->size[bins->last];
|
||||
bins->size[bins->last] = 0;
|
||||
|
||||
for (fence = task.first_key; fence <= end; ) {
|
||||
u8 bin;
|
||||
sort_key_t key = *fence;
|
||||
|
||||
/*
|
||||
* The radix byte of the key tells us which pile it belongs in. Swap it for
|
||||
* an unprocessed item just below that pile, and repeat.
|
||||
*/
|
||||
while (--pile[bin = key[task.offset]] > fence)
|
||||
swap_keys(pile[bin], &key);
|
||||
|
||||
/*
|
||||
* The pile reached the fence. Put the key at the bottom of that pile,
|
||||
* completing it, and advance the fence to the next pile.
|
||||
*/
|
||||
*fence = key;
|
||||
fence += bins->size[bin];
|
||||
bins->size[bin] = 0;
|
||||
}
|
||||
|
||||
/* Now bins->size[] is all zero again. */
|
||||
|
||||
/*
|
||||
* When the number of keys in a task gets small enough, it is faster to use an
|
||||
* insertion sort than to keep subdividing into tiny piles.
|
||||
*/
|
||||
while (--insertion_task_list >= sorter->insertion_list)
|
||||
insertion_sort(*insertion_task_list);
|
||||
}
|
||||
|
||||
return UDS_SUCCESS;
|
||||
}
|
|
@ -0,0 +1,26 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef UDS_RADIX_SORT_H
|
||||
#define UDS_RADIX_SORT_H
|
||||
|
||||
/*
|
||||
* Radix sort is implemented using an American Flag sort, an unstable, in-place 8-bit radix
|
||||
* exchange sort. This is adapted from the algorithm in the paper by Peter M. McIlroy, Keith
|
||||
* Bostic, and M. Douglas McIlroy, "Engineering Radix Sort".
|
||||
*
|
||||
* http://www.usenix.org/publications/compsystems/1993/win_mcilroy.pdf
|
||||
*/
|
||||
|
||||
struct radix_sorter;
|
||||
|
||||
int __must_check uds_make_radix_sorter(unsigned int count, struct radix_sorter **sorter);
|
||||
|
||||
void uds_free_radix_sorter(struct radix_sorter *sorter);
|
||||
|
||||
int __must_check uds_radix_sort(struct radix_sorter *sorter, const unsigned char *keys[],
|
||||
unsigned int count, unsigned short length);
|
||||
|
||||
#endif /* UDS_RADIX_SORT_H */
|
|
@ -0,0 +1,624 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "sparse-cache.h"
|
||||
|
||||
#include <linux/cache.h>
|
||||
#include <linux/delay.h>
|
||||
#include <linux/dm-bufio.h>
|
||||
|
||||
#include "logger.h"
|
||||
#include "memory-alloc.h"
|
||||
#include "permassert.h"
|
||||
|
||||
#include "chapter-index.h"
|
||||
#include "config.h"
|
||||
#include "index.h"
|
||||
|
||||
/*
|
||||
* Since the cache is small, it is implemented as a simple array of cache entries. Searching for a
|
||||
* specific virtual chapter is implemented as a linear search. The cache replacement policy is
|
||||
* least-recently-used (LRU). Again, the small size of the cache allows the LRU order to be
|
||||
* maintained by shifting entries in an array list.
|
||||
*
|
||||
* Changing the contents of the cache requires the coordinated participation of all zone threads
|
||||
* via the careful use of barrier messages sent to all the index zones by the triage queue worker
|
||||
* thread. The critical invariant for coordination is that the cache membership must not change
|
||||
* between updates, so that all calls to uds_sparse_cache_contains() from the zone threads must all
|
||||
* receive the same results for every virtual chapter number. To ensure that critical invariant,
|
||||
* state changes such as "that virtual chapter is no longer in the volume" and "skip searching that
|
||||
* chapter because it has had too many cache misses" are represented separately from the cache
|
||||
* membership information (the virtual chapter number).
|
||||
*
|
||||
* As a result of this invariant, we have the guarantee that every zone thread will call
|
||||
* uds_update_sparse_cache() once and exactly once to request a chapter that is not in the cache,
|
||||
* and the serialization of the barrier requests from the triage queue ensures they will all
|
||||
* request the same chapter number. This means the only synchronization we need can be provided by
|
||||
* a pair of thread barriers used only in the uds_update_sparse_cache() call, providing a critical
|
||||
* section where a single zone thread can drive the cache update while all the other zone threads
|
||||
* are known to be blocked, waiting in the second barrier. Outside that critical section, all the
|
||||
* zone threads implicitly hold a shared lock. Inside it, the thread for zone zero holds an
|
||||
* exclusive lock. No other threads may access or modify the cache entries.
|
||||
*
|
||||
* Chapter statistics must only be modified by a single thread, which is also the zone zero thread.
|
||||
* All fields that might be frequently updated by that thread are kept in separate cache-aligned
|
||||
* structures so they will not cause cache contention via "false sharing" with the fields that are
|
||||
* frequently accessed by all of the zone threads.
|
||||
*
|
||||
* The LRU order is managed independently by each zone thread, and each zone uses its own list for
|
||||
* searching and cache membership queries. The zone zero list is used to decide which chapter to
|
||||
* evict when the cache is updated, and its search list is copied to the other threads at that
|
||||
* time.
|
||||
*
|
||||
* The virtual chapter number field of the cache entry is the single field indicating whether a
|
||||
* chapter is a member of the cache or not. The value NO_CHAPTER is used to represent a null or
|
||||
* undefined chapter number. When present in the virtual chapter number field of a
|
||||
* cached_chapter_index, it indicates that the cache entry is dead, and all the other fields of
|
||||
* that entry (other than immutable pointers to cache memory) are undefined and irrelevant. Any
|
||||
* cache entry that is not marked as dead is fully defined and a member of the cache, and
|
||||
* uds_sparse_cache_contains() will always return true for any virtual chapter number that appears
|
||||
* in any of the cache entries.
|
||||
*
|
||||
* A chapter index that is a member of the cache may be excluded from searches between calls to
|
||||
* uds_update_sparse_cache() in two different ways. First, when a chapter falls off the end of the
|
||||
* volume, its virtual chapter number will be less that the oldest virtual chapter number. Since
|
||||
* that chapter is no longer part of the volume, there's no point in continuing to search that
|
||||
* chapter index. Once invalidated, that virtual chapter will still be considered a member of the
|
||||
* cache, but it will no longer be searched for matching names.
|
||||
*
|
||||
* The second mechanism is a heuristic based on keeping track of the number of consecutive search
|
||||
* misses in a given chapter index. Once that count exceeds a threshold, the skip_search flag will
|
||||
* be set to true, causing the chapter to be skipped when searching the entire cache, but still
|
||||
* allowing it to be found when searching for a hook in that specific chapter. Finding a hook will
|
||||
* clear the skip_search flag, once again allowing the non-hook searches to use that cache entry.
|
||||
* Again, regardless of the state of the skip_search flag, the virtual chapter must still
|
||||
* considered to be a member of the cache for uds_sparse_cache_contains().
|
||||
*/
|
||||
|
||||
#define SKIP_SEARCH_THRESHOLD 20000
|
||||
#define ZONE_ZERO 0
|
||||
|
||||
/*
|
||||
* These counters are essentially fields of the struct cached_chapter_index, but are segregated
|
||||
* into this structure because they are frequently modified. They are grouped and aligned to keep
|
||||
* them on different cache lines from the chapter fields that are accessed far more often than they
|
||||
* are updated.
|
||||
*/
|
||||
struct __aligned(L1_CACHE_BYTES) cached_index_counters {
|
||||
u64 consecutive_misses;
|
||||
};
|
||||
|
||||
struct __aligned(L1_CACHE_BYTES) cached_chapter_index {
|
||||
/*
|
||||
* The virtual chapter number of the cached chapter index. NO_CHAPTER means this cache
|
||||
* entry is unused. This field must only be modified in the critical section in
|
||||
* uds_update_sparse_cache().
|
||||
*/
|
||||
u64 virtual_chapter;
|
||||
|
||||
u32 index_pages_count;
|
||||
|
||||
/*
|
||||
* These pointers are immutable during the life of the cache. The contents of the arrays
|
||||
* change when the cache entry is replaced.
|
||||
*/
|
||||
struct delta_index_page *index_pages;
|
||||
struct dm_buffer **page_buffers;
|
||||
|
||||
/*
|
||||
* If set, skip the chapter when searching the entire cache. This flag is just a
|
||||
* performance optimization. This flag is mutable between cache updates, but it rarely
|
||||
* changes and is frequently accessed, so it groups with the immutable fields.
|
||||
*/
|
||||
bool skip_search;
|
||||
|
||||
/*
|
||||
* The cache-aligned counters change often and are placed at the end of the structure to
|
||||
* prevent false sharing with the more stable fields above.
|
||||
*/
|
||||
struct cached_index_counters counters;
|
||||
};
|
||||
|
||||
/*
|
||||
* A search_list represents an ordering of the sparse chapter index cache entry array, from most
|
||||
* recently accessed to least recently accessed, which is the order in which the indexes should be
|
||||
* searched and the reverse order in which they should be evicted from the cache.
|
||||
*
|
||||
* Cache entries that are dead or empty are kept at the end of the list, avoiding the need to even
|
||||
* iterate over them to search, and ensuring that dead entries are replaced before any live entries
|
||||
* are evicted.
|
||||
*
|
||||
* The search list is instantiated for each zone thread, avoiding any need for synchronization. The
|
||||
* structure is allocated on a cache boundary to avoid false sharing of memory cache lines between
|
||||
* zone threads.
|
||||
*/
|
||||
struct search_list {
|
||||
u8 capacity;
|
||||
u8 first_dead_entry;
|
||||
struct cached_chapter_index *entries[];
|
||||
};
|
||||
|
||||
struct threads_barrier {
|
||||
/* Lock for this barrier object */
|
||||
struct semaphore lock;
|
||||
/* Semaphore for threads waiting at this barrier */
|
||||
struct semaphore wait;
|
||||
/* Number of threads which have arrived */
|
||||
int arrived;
|
||||
/* Total number of threads using this barrier */
|
||||
int thread_count;
|
||||
};
|
||||
|
||||
struct sparse_cache {
|
||||
const struct index_geometry *geometry;
|
||||
unsigned int capacity;
|
||||
unsigned int zone_count;
|
||||
|
||||
unsigned int skip_threshold;
|
||||
struct search_list *search_lists[MAX_ZONES];
|
||||
struct cached_chapter_index **scratch_entries;
|
||||
|
||||
struct threads_barrier begin_update_barrier;
|
||||
struct threads_barrier end_update_barrier;
|
||||
|
||||
struct cached_chapter_index chapters[];
|
||||
};
|
||||
|
||||
static void initialize_threads_barrier(struct threads_barrier *barrier,
|
||||
unsigned int thread_count)
|
||||
{
|
||||
sema_init(&barrier->lock, 1);
|
||||
barrier->arrived = 0;
|
||||
barrier->thread_count = thread_count;
|
||||
sema_init(&barrier->wait, 0);
|
||||
}
|
||||
|
||||
static inline void __down(struct semaphore *semaphore)
|
||||
{
|
||||
/*
|
||||
* Do not use down(semaphore). Instead use down_interruptible so that
|
||||
* we do not get 120 second stall messages in kern.log.
|
||||
*/
|
||||
while (down_interruptible(semaphore) != 0) {
|
||||
/*
|
||||
* If we're called from a user-mode process (e.g., "dmsetup
|
||||
* remove") while waiting for an operation that may take a
|
||||
* while (e.g., UDS index save), and a signal is sent (SIGINT,
|
||||
* SIGUSR2), then down_interruptible will not block. If that
|
||||
* happens, sleep briefly to avoid keeping the CPU locked up in
|
||||
* this loop. We could just call cond_resched, but then we'd
|
||||
* still keep consuming CPU time slices and swamp other threads
|
||||
* trying to do computational work.
|
||||
*/
|
||||
fsleep(1000);
|
||||
}
|
||||
}
|
||||
|
||||
static void enter_threads_barrier(struct threads_barrier *barrier)
|
||||
{
|
||||
__down(&barrier->lock);
|
||||
if (++barrier->arrived == barrier->thread_count) {
|
||||
/* last thread */
|
||||
int i;
|
||||
|
||||
for (i = 1; i < barrier->thread_count; i++)
|
||||
up(&barrier->wait);
|
||||
|
||||
barrier->arrived = 0;
|
||||
up(&barrier->lock);
|
||||
} else {
|
||||
up(&barrier->lock);
|
||||
__down(&barrier->wait);
|
||||
}
|
||||
}
|
||||
|
||||
static int __must_check initialize_cached_chapter_index(struct cached_chapter_index *chapter,
|
||||
const struct index_geometry *geometry)
|
||||
{
|
||||
int result;
|
||||
|
||||
chapter->virtual_chapter = NO_CHAPTER;
|
||||
chapter->index_pages_count = geometry->index_pages_per_chapter;
|
||||
|
||||
result = vdo_allocate(chapter->index_pages_count, struct delta_index_page,
|
||||
__func__, &chapter->index_pages);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
return vdo_allocate(chapter->index_pages_count, struct dm_buffer *,
|
||||
"sparse index volume pages", &chapter->page_buffers);
|
||||
}
|
||||
|
||||
static int __must_check make_search_list(struct sparse_cache *cache,
|
||||
struct search_list **list_ptr)
|
||||
{
|
||||
struct search_list *list;
|
||||
unsigned int bytes;
|
||||
u8 i;
|
||||
int result;
|
||||
|
||||
bytes = (sizeof(struct search_list) +
|
||||
(cache->capacity * sizeof(struct cached_chapter_index *)));
|
||||
result = vdo_allocate_cache_aligned(bytes, "search list", &list);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
list->capacity = cache->capacity;
|
||||
list->first_dead_entry = 0;
|
||||
|
||||
for (i = 0; i < list->capacity; i++)
|
||||
list->entries[i] = &cache->chapters[i];
|
||||
|
||||
*list_ptr = list;
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
int uds_make_sparse_cache(const struct index_geometry *geometry, unsigned int capacity,
|
||||
unsigned int zone_count, struct sparse_cache **cache_ptr)
|
||||
{
|
||||
int result;
|
||||
unsigned int i;
|
||||
struct sparse_cache *cache;
|
||||
unsigned int bytes;
|
||||
|
||||
bytes = (sizeof(struct sparse_cache) + (capacity * sizeof(struct cached_chapter_index)));
|
||||
result = vdo_allocate_cache_aligned(bytes, "sparse cache", &cache);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
cache->geometry = geometry;
|
||||
cache->capacity = capacity;
|
||||
cache->zone_count = zone_count;
|
||||
|
||||
/*
|
||||
* Scale down the skip threshold since the cache only counts cache misses in zone zero, but
|
||||
* requests are being handled in all zones.
|
||||
*/
|
||||
cache->skip_threshold = (SKIP_SEARCH_THRESHOLD / zone_count);
|
||||
|
||||
initialize_threads_barrier(&cache->begin_update_barrier, zone_count);
|
||||
initialize_threads_barrier(&cache->end_update_barrier, zone_count);
|
||||
|
||||
for (i = 0; i < capacity; i++) {
|
||||
result = initialize_cached_chapter_index(&cache->chapters[i], geometry);
|
||||
if (result != UDS_SUCCESS)
|
||||
goto out;
|
||||
}
|
||||
|
||||
for (i = 0; i < zone_count; i++) {
|
||||
result = make_search_list(cache, &cache->search_lists[i]);
|
||||
if (result != UDS_SUCCESS)
|
||||
goto out;
|
||||
}
|
||||
|
||||
/* purge_search_list() needs some temporary lists for sorting. */
|
||||
result = vdo_allocate(capacity * 2, struct cached_chapter_index *,
|
||||
"scratch entries", &cache->scratch_entries);
|
||||
if (result != VDO_SUCCESS)
|
||||
goto out;
|
||||
|
||||
*cache_ptr = cache;
|
||||
return UDS_SUCCESS;
|
||||
out:
|
||||
uds_free_sparse_cache(cache);
|
||||
return result;
|
||||
}
|
||||
|
||||
static inline void set_skip_search(struct cached_chapter_index *chapter,
|
||||
bool skip_search)
|
||||
{
|
||||
/* Check before setting to reduce cache line contention. */
|
||||
if (READ_ONCE(chapter->skip_search) != skip_search)
|
||||
WRITE_ONCE(chapter->skip_search, skip_search);
|
||||
}
|
||||
|
||||
static void score_search_hit(struct cached_chapter_index *chapter)
|
||||
{
|
||||
chapter->counters.consecutive_misses = 0;
|
||||
set_skip_search(chapter, false);
|
||||
}
|
||||
|
||||
static void score_search_miss(struct sparse_cache *cache,
|
||||
struct cached_chapter_index *chapter)
|
||||
{
|
||||
chapter->counters.consecutive_misses++;
|
||||
if (chapter->counters.consecutive_misses > cache->skip_threshold)
|
||||
set_skip_search(chapter, true);
|
||||
}
|
||||
|
||||
static void release_cached_chapter_index(struct cached_chapter_index *chapter)
|
||||
{
|
||||
unsigned int i;
|
||||
|
||||
chapter->virtual_chapter = NO_CHAPTER;
|
||||
if (chapter->page_buffers == NULL)
|
||||
return;
|
||||
|
||||
for (i = 0; i < chapter->index_pages_count; i++) {
|
||||
if (chapter->page_buffers[i] != NULL)
|
||||
dm_bufio_release(vdo_forget(chapter->page_buffers[i]));
|
||||
}
|
||||
}
|
||||
|
||||
void uds_free_sparse_cache(struct sparse_cache *cache)
|
||||
{
|
||||
unsigned int i;
|
||||
|
||||
if (cache == NULL)
|
||||
return;
|
||||
|
||||
vdo_free(cache->scratch_entries);
|
||||
|
||||
for (i = 0; i < cache->zone_count; i++)
|
||||
vdo_free(cache->search_lists[i]);
|
||||
|
||||
for (i = 0; i < cache->capacity; i++) {
|
||||
release_cached_chapter_index(&cache->chapters[i]);
|
||||
vdo_free(cache->chapters[i].index_pages);
|
||||
vdo_free(cache->chapters[i].page_buffers);
|
||||
}
|
||||
|
||||
vdo_free(cache);
|
||||
}
|
||||
|
||||
/*
|
||||
* Take the indicated element of the search list and move it to the start, pushing the pointers
|
||||
* previously before it back down the list.
|
||||
*/
|
||||
static inline void set_newest_entry(struct search_list *search_list, u8 index)
|
||||
{
|
||||
struct cached_chapter_index *newest;
|
||||
|
||||
if (index > 0) {
|
||||
newest = search_list->entries[index];
|
||||
memmove(&search_list->entries[1], &search_list->entries[0],
|
||||
index * sizeof(struct cached_chapter_index *));
|
||||
search_list->entries[0] = newest;
|
||||
}
|
||||
|
||||
/*
|
||||
* This function may have moved a dead chapter to the front of the list for reuse, in which
|
||||
* case the set of dead chapters becomes smaller.
|
||||
*/
|
||||
if (search_list->first_dead_entry <= index)
|
||||
search_list->first_dead_entry++;
|
||||
}
|
||||
|
||||
bool uds_sparse_cache_contains(struct sparse_cache *cache, u64 virtual_chapter,
|
||||
unsigned int zone_number)
|
||||
{
|
||||
struct search_list *search_list;
|
||||
struct cached_chapter_index *chapter;
|
||||
u8 i;
|
||||
|
||||
/*
|
||||
* The correctness of the barriers depends on the invariant that between calls to
|
||||
* uds_update_sparse_cache(), the answers this function returns must never vary: the result
|
||||
* for a given chapter must be identical across zones. That invariant must be maintained
|
||||
* even if the chapter falls off the end of the volume, or if searching it is disabled
|
||||
* because of too many search misses.
|
||||
*/
|
||||
search_list = cache->search_lists[zone_number];
|
||||
for (i = 0; i < search_list->first_dead_entry; i++) {
|
||||
chapter = search_list->entries[i];
|
||||
|
||||
if (virtual_chapter == chapter->virtual_chapter) {
|
||||
if (zone_number == ZONE_ZERO)
|
||||
score_search_hit(chapter);
|
||||
|
||||
set_newest_entry(search_list, i);
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
/*
|
||||
* Re-sort cache entries into three sets (active, skippable, and dead) while maintaining the LRU
|
||||
* ordering that already existed. This operation must only be called during the critical section in
|
||||
* uds_update_sparse_cache().
|
||||
*/
|
||||
static void purge_search_list(struct search_list *search_list,
|
||||
struct sparse_cache *cache, u64 oldest_virtual_chapter)
|
||||
{
|
||||
struct cached_chapter_index **entries;
|
||||
struct cached_chapter_index **skipped;
|
||||
struct cached_chapter_index **dead;
|
||||
struct cached_chapter_index *chapter;
|
||||
unsigned int next_alive = 0;
|
||||
unsigned int next_skipped = 0;
|
||||
unsigned int next_dead = 0;
|
||||
unsigned int i;
|
||||
|
||||
entries = &search_list->entries[0];
|
||||
skipped = &cache->scratch_entries[0];
|
||||
dead = &cache->scratch_entries[search_list->capacity];
|
||||
|
||||
for (i = 0; i < search_list->first_dead_entry; i++) {
|
||||
chapter = search_list->entries[i];
|
||||
if ((chapter->virtual_chapter < oldest_virtual_chapter) ||
|
||||
(chapter->virtual_chapter == NO_CHAPTER))
|
||||
dead[next_dead++] = chapter;
|
||||
else if (chapter->skip_search)
|
||||
skipped[next_skipped++] = chapter;
|
||||
else
|
||||
entries[next_alive++] = chapter;
|
||||
}
|
||||
|
||||
memcpy(&entries[next_alive], skipped,
|
||||
next_skipped * sizeof(struct cached_chapter_index *));
|
||||
memcpy(&entries[next_alive + next_skipped], dead,
|
||||
next_dead * sizeof(struct cached_chapter_index *));
|
||||
search_list->first_dead_entry = next_alive + next_skipped;
|
||||
}
|
||||
|
||||
static int __must_check cache_chapter_index(struct cached_chapter_index *chapter,
|
||||
u64 virtual_chapter,
|
||||
const struct volume *volume)
|
||||
{
|
||||
int result;
|
||||
|
||||
release_cached_chapter_index(chapter);
|
||||
|
||||
result = uds_read_chapter_index_from_volume(volume, virtual_chapter,
|
||||
chapter->page_buffers,
|
||||
chapter->index_pages);
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
chapter->counters.consecutive_misses = 0;
|
||||
chapter->virtual_chapter = virtual_chapter;
|
||||
chapter->skip_search = false;
|
||||
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
static inline void copy_search_list(const struct search_list *source,
|
||||
struct search_list *target)
|
||||
{
|
||||
*target = *source;
|
||||
memcpy(target->entries, source->entries,
|
||||
source->capacity * sizeof(struct cached_chapter_index *));
|
||||
}
|
||||
|
||||
/*
|
||||
* Update the sparse cache to contain a chapter index. This function must be called by all the zone
|
||||
* threads with the same chapter number to correctly enter the thread barriers used to synchronize
|
||||
* the cache updates.
|
||||
*/
|
||||
int uds_update_sparse_cache(struct index_zone *zone, u64 virtual_chapter)
|
||||
{
|
||||
int result = UDS_SUCCESS;
|
||||
const struct uds_index *index = zone->index;
|
||||
struct sparse_cache *cache = index->volume->sparse_cache;
|
||||
|
||||
if (uds_sparse_cache_contains(cache, virtual_chapter, zone->id))
|
||||
return UDS_SUCCESS;
|
||||
|
||||
/*
|
||||
* Wait for every zone thread to reach its corresponding barrier request and invoke this
|
||||
* function before starting to modify the cache.
|
||||
*/
|
||||
enter_threads_barrier(&cache->begin_update_barrier);
|
||||
|
||||
/*
|
||||
* This is the start of the critical section: the zone zero thread is captain, effectively
|
||||
* holding an exclusive lock on the sparse cache. All the other zone threads must do
|
||||
* nothing between the two barriers. They will wait at the end_update_barrier again for the
|
||||
* captain to finish the update.
|
||||
*/
|
||||
|
||||
if (zone->id == ZONE_ZERO) {
|
||||
unsigned int z;
|
||||
struct search_list *list = cache->search_lists[ZONE_ZERO];
|
||||
|
||||
purge_search_list(list, cache, zone->oldest_virtual_chapter);
|
||||
|
||||
if (virtual_chapter >= index->oldest_virtual_chapter) {
|
||||
set_newest_entry(list, list->capacity - 1);
|
||||
result = cache_chapter_index(list->entries[0], virtual_chapter,
|
||||
index->volume);
|
||||
}
|
||||
|
||||
for (z = 1; z < cache->zone_count; z++)
|
||||
copy_search_list(list, cache->search_lists[z]);
|
||||
}
|
||||
|
||||
/*
|
||||
* This is the end of the critical section. All cache invariants must have been restored.
|
||||
*/
|
||||
enter_threads_barrier(&cache->end_update_barrier);
|
||||
return result;
|
||||
}
|
||||
|
||||
void uds_invalidate_sparse_cache(struct sparse_cache *cache)
|
||||
{
|
||||
unsigned int i;
|
||||
|
||||
for (i = 0; i < cache->capacity; i++)
|
||||
release_cached_chapter_index(&cache->chapters[i]);
|
||||
}
|
||||
|
||||
static inline bool should_skip_chapter(struct cached_chapter_index *chapter,
|
||||
u64 oldest_chapter, u64 requested_chapter)
|
||||
{
|
||||
if ((chapter->virtual_chapter == NO_CHAPTER) ||
|
||||
(chapter->virtual_chapter < oldest_chapter))
|
||||
return true;
|
||||
|
||||
if (requested_chapter != NO_CHAPTER)
|
||||
return requested_chapter != chapter->virtual_chapter;
|
||||
else
|
||||
return READ_ONCE(chapter->skip_search);
|
||||
}
|
||||
|
||||
static int __must_check search_cached_chapter_index(struct cached_chapter_index *chapter,
|
||||
const struct index_geometry *geometry,
|
||||
const struct index_page_map *index_page_map,
|
||||
const struct uds_record_name *name,
|
||||
u16 *record_page_ptr)
|
||||
{
|
||||
u32 physical_chapter =
|
||||
uds_map_to_physical_chapter(geometry, chapter->virtual_chapter);
|
||||
u32 index_page_number =
|
||||
uds_find_index_page_number(index_page_map, name, physical_chapter);
|
||||
struct delta_index_page *index_page =
|
||||
&chapter->index_pages[index_page_number];
|
||||
|
||||
return uds_search_chapter_index_page(index_page, geometry, name,
|
||||
record_page_ptr);
|
||||
}
|
||||
|
||||
int uds_search_sparse_cache(struct index_zone *zone, const struct uds_record_name *name,
|
||||
u64 *virtual_chapter_ptr, u16 *record_page_ptr)
|
||||
{
|
||||
int result;
|
||||
struct volume *volume = zone->index->volume;
|
||||
struct sparse_cache *cache = volume->sparse_cache;
|
||||
struct cached_chapter_index *chapter;
|
||||
struct search_list *search_list;
|
||||
u8 i;
|
||||
/* Search the entire cache unless a specific chapter was requested. */
|
||||
bool search_one = (*virtual_chapter_ptr != NO_CHAPTER);
|
||||
|
||||
*record_page_ptr = NO_CHAPTER_INDEX_ENTRY;
|
||||
search_list = cache->search_lists[zone->id];
|
||||
for (i = 0; i < search_list->first_dead_entry; i++) {
|
||||
chapter = search_list->entries[i];
|
||||
|
||||
if (should_skip_chapter(chapter, zone->oldest_virtual_chapter,
|
||||
*virtual_chapter_ptr))
|
||||
continue;
|
||||
|
||||
result = search_cached_chapter_index(chapter, cache->geometry,
|
||||
volume->index_page_map, name,
|
||||
record_page_ptr);
|
||||
if (result != UDS_SUCCESS)
|
||||
return result;
|
||||
|
||||
if (*record_page_ptr != NO_CHAPTER_INDEX_ENTRY) {
|
||||
/*
|
||||
* In theory, this might be a false match while a true match exists in
|
||||
* another chapter, but that's a very rare case and not worth the extra
|
||||
* search complexity.
|
||||
*/
|
||||
set_newest_entry(search_list, i);
|
||||
if (zone->id == ZONE_ZERO)
|
||||
score_search_hit(chapter);
|
||||
|
||||
*virtual_chapter_ptr = chapter->virtual_chapter;
|
||||
return UDS_SUCCESS;
|
||||
}
|
||||
|
||||
if (zone->id == ZONE_ZERO)
|
||||
score_search_miss(cache, chapter);
|
||||
|
||||
if (search_one)
|
||||
break;
|
||||
}
|
||||
|
||||
return UDS_SUCCESS;
|
||||
}
|
|
@ -0,0 +1,46 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef UDS_SPARSE_CACHE_H
|
||||
#define UDS_SPARSE_CACHE_H
|
||||
|
||||
#include "geometry.h"
|
||||
#include "indexer.h"
|
||||
|
||||
/*
|
||||
* The sparse cache is a cache of entire chapter indexes from sparse chapters used for searching
|
||||
* for names after all other search paths have failed. It contains only complete chapter indexes;
|
||||
* record pages from sparse chapters and single index pages used for resolving hooks are kept in
|
||||
* the regular page cache in the volume.
|
||||
*
|
||||
* The most important property of this cache is the absence of synchronization for read operations.
|
||||
* Safe concurrent access to the cache by the zone threads is controlled by the triage queue and
|
||||
* the barrier requests it issues to the zone queues. The set of cached chapters does not and must
|
||||
* not change between the carefully coordinated calls to uds_update_sparse_cache() from the zone
|
||||
* threads. Outside of updates, every zone will get the same result when calling
|
||||
* uds_sparse_cache_contains() as every other zone.
|
||||
*/
|
||||
|
||||
struct index_zone;
|
||||
struct sparse_cache;
|
||||
|
||||
int __must_check uds_make_sparse_cache(const struct index_geometry *geometry,
|
||||
unsigned int capacity, unsigned int zone_count,
|
||||
struct sparse_cache **cache_ptr);
|
||||
|
||||
void uds_free_sparse_cache(struct sparse_cache *cache);
|
||||
|
||||
bool uds_sparse_cache_contains(struct sparse_cache *cache, u64 virtual_chapter,
|
||||
unsigned int zone_number);
|
||||
|
||||
int __must_check uds_update_sparse_cache(struct index_zone *zone, u64 virtual_chapter);
|
||||
|
||||
void uds_invalidate_sparse_cache(struct sparse_cache *cache);
|
||||
|
||||
int __must_check uds_search_sparse_cache(struct index_zone *zone,
|
||||
const struct uds_record_name *name,
|
||||
u64 *virtual_chapter_ptr, u16 *record_page_ptr);
|
||||
|
||||
#endif /* UDS_SPARSE_CACHE_H */
|
File diff suppressed because it is too large
Load Diff
|
@ -0,0 +1,193 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef UDS_VOLUME_INDEX_H
|
||||
#define UDS_VOLUME_INDEX_H
|
||||
|
||||
#include <linux/limits.h>
|
||||
|
||||
#include "thread-utils.h"
|
||||
|
||||
#include "config.h"
|
||||
#include "delta-index.h"
|
||||
#include "indexer.h"
|
||||
|
||||
/*
|
||||
* The volume index is the primary top-level index for UDS. It contains records which map a record
|
||||
* name to the chapter where a record with that name is stored. This mapping can definitively say
|
||||
* when no record exists. However, because we only use a subset of the name for this index, it
|
||||
* cannot definitively say that a record for the entry does exist. It can only say that if a record
|
||||
* exists, it will be in a particular chapter. The request can then be dispatched to that chapter
|
||||
* for further processing.
|
||||
*
|
||||
* If the volume_index_record does not actually match the record name, the index can store a more
|
||||
* specific collision record to disambiguate the new entry from the existing one. Index entries are
|
||||
* managed with volume_index_record structures.
|
||||
*/
|
||||
|
||||
#define NO_CHAPTER U64_MAX
|
||||
|
||||
struct volume_index_stats {
|
||||
/* Nanoseconds spent rebalancing */
|
||||
ktime_t rebalance_time;
|
||||
/* Number of memory rebalances */
|
||||
u32 rebalance_count;
|
||||
/* The number of records in the index */
|
||||
u64 record_count;
|
||||
/* The number of collision records */
|
||||
u64 collision_count;
|
||||
/* The number of records removed */
|
||||
u64 discard_count;
|
||||
/* The number of UDS_OVERFLOWs detected */
|
||||
u64 overflow_count;
|
||||
/* The number of delta lists */
|
||||
u32 delta_lists;
|
||||
/* Number of early flushes */
|
||||
u64 early_flushes;
|
||||
};
|
||||
|
||||
struct volume_sub_index_zone {
|
||||
u64 virtual_chapter_low;
|
||||
u64 virtual_chapter_high;
|
||||
u64 early_flushes;
|
||||
} __aligned(L1_CACHE_BYTES);
|
||||
|
||||
struct volume_sub_index {
|
||||
/* The delta index */
|
||||
struct delta_index delta_index;
|
||||
/* The first chapter to be flushed in each zone */
|
||||
u64 *flush_chapters;
|
||||
/* The zones */
|
||||
struct volume_sub_index_zone *zones;
|
||||
/* The volume nonce */
|
||||
u64 volume_nonce;
|
||||
/* Expected size of a chapter (per zone) */
|
||||
u64 chapter_zone_bits;
|
||||
/* Maximum size of the index (per zone) */
|
||||
u64 max_zone_bits;
|
||||
/* The number of bits in address mask */
|
||||
u8 address_bits;
|
||||
/* Mask to get address within delta list */
|
||||
u32 address_mask;
|
||||
/* The number of bits in chapter number */
|
||||
u8 chapter_bits;
|
||||
/* The largest storable chapter number */
|
||||
u32 chapter_mask;
|
||||
/* The number of chapters used */
|
||||
u32 chapter_count;
|
||||
/* The number of delta lists */
|
||||
u32 list_count;
|
||||
/* The number of zones */
|
||||
unsigned int zone_count;
|
||||
/* The amount of memory allocated */
|
||||
u64 memory_size;
|
||||
};
|
||||
|
||||
struct volume_index_zone {
|
||||
/* Protects the sampled index in this zone */
|
||||
struct mutex hook_mutex;
|
||||
} __aligned(L1_CACHE_BYTES);
|
||||
|
||||
struct volume_index {
|
||||
u32 sparse_sample_rate;
|
||||
unsigned int zone_count;
|
||||
u64 memory_size;
|
||||
struct volume_sub_index vi_non_hook;
|
||||
struct volume_sub_index vi_hook;
|
||||
struct volume_index_zone *zones;
|
||||
};
|
||||
|
||||
/*
|
||||
* The volume_index_record structure is used to facilitate processing of a record name. A client
|
||||
* first calls uds_get_volume_index_record() to find the volume index record for a record name. The
|
||||
* fields of the record can then be examined to determine the state of the record.
|
||||
*
|
||||
* If is_found is false, then the index did not find an entry for the record name. Calling
|
||||
* uds_put_volume_index_record() will insert a new entry for that name at the proper place.
|
||||
*
|
||||
* If is_found is true, then we did find an entry for the record name, and the virtual_chapter and
|
||||
* is_collision fields reflect the entry found. Subsequently, a call to
|
||||
* uds_remove_volume_index_record() will remove the entry, a call to
|
||||
* uds_set_volume_index_record_chapter() will update the existing entry, and a call to
|
||||
* uds_put_volume_index_record() will insert a new collision record after the existing entry.
|
||||
*/
|
||||
struct volume_index_record {
|
||||
/* Public fields */
|
||||
|
||||
/* Chapter where the record info is found */
|
||||
u64 virtual_chapter;
|
||||
/* This record is a collision */
|
||||
bool is_collision;
|
||||
/* This record is the requested record */
|
||||
bool is_found;
|
||||
|
||||
/* Private fields */
|
||||
|
||||
/* Zone that contains this name */
|
||||
unsigned int zone_number;
|
||||
/* The volume index */
|
||||
struct volume_sub_index *sub_index;
|
||||
/* Mutex for accessing this delta index entry in the hook index */
|
||||
struct mutex *mutex;
|
||||
/* The record name to which this record refers */
|
||||
const struct uds_record_name *name;
|
||||
/* The delta index entry for this record */
|
||||
struct delta_index_entry delta_entry;
|
||||
};
|
||||
|
||||
int __must_check uds_make_volume_index(const struct uds_configuration *config,
|
||||
u64 volume_nonce,
|
||||
struct volume_index **volume_index);
|
||||
|
||||
void uds_free_volume_index(struct volume_index *volume_index);
|
||||
|
||||
int __must_check uds_compute_volume_index_save_blocks(const struct uds_configuration *config,
|
||||
size_t block_size,
|
||||
u64 *block_count);
|
||||
|
||||
unsigned int __must_check uds_get_volume_index_zone(const struct volume_index *volume_index,
|
||||
const struct uds_record_name *name);
|
||||
|
||||
bool __must_check uds_is_volume_index_sample(const struct volume_index *volume_index,
|
||||
const struct uds_record_name *name);
|
||||
|
||||
/*
|
||||
* This function is only used to manage sparse cache membership. Most requests should use
|
||||
* uds_get_volume_index_record() to look up index records instead.
|
||||
*/
|
||||
u64 __must_check uds_lookup_volume_index_name(const struct volume_index *volume_index,
|
||||
const struct uds_record_name *name);
|
||||
|
||||
int __must_check uds_get_volume_index_record(struct volume_index *volume_index,
|
||||
const struct uds_record_name *name,
|
||||
struct volume_index_record *record);
|
||||
|
||||
int __must_check uds_put_volume_index_record(struct volume_index_record *record,
|
||||
u64 virtual_chapter);
|
||||
|
||||
int __must_check uds_remove_volume_index_record(struct volume_index_record *record);
|
||||
|
||||
int __must_check uds_set_volume_index_record_chapter(struct volume_index_record *record,
|
||||
u64 virtual_chapter);
|
||||
|
||||
void uds_set_volume_index_open_chapter(struct volume_index *volume_index,
|
||||
u64 virtual_chapter);
|
||||
|
||||
void uds_set_volume_index_zone_open_chapter(struct volume_index *volume_index,
|
||||
unsigned int zone_number,
|
||||
u64 virtual_chapter);
|
||||
|
||||
int __must_check uds_load_volume_index(struct volume_index *volume_index,
|
||||
struct buffered_reader **readers,
|
||||
unsigned int reader_count);
|
||||
|
||||
int __must_check uds_save_volume_index(struct volume_index *volume_index,
|
||||
struct buffered_writer **writers,
|
||||
unsigned int writer_count);
|
||||
|
||||
void uds_get_volume_index_stats(const struct volume_index *volume_index,
|
||||
struct volume_index_stats *stats);
|
||||
|
||||
#endif /* UDS_VOLUME_INDEX_H */
|
File diff suppressed because it is too large
Load Diff
|
@ -0,0 +1,172 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef UDS_VOLUME_H
|
||||
#define UDS_VOLUME_H
|
||||
|
||||
#include <linux/atomic.h>
|
||||
#include <linux/cache.h>
|
||||
#include <linux/dm-bufio.h>
|
||||
#include <linux/limits.h>
|
||||
|
||||
#include "permassert.h"
|
||||
#include "thread-utils.h"
|
||||
|
||||
#include "chapter-index.h"
|
||||
#include "config.h"
|
||||
#include "geometry.h"
|
||||
#include "indexer.h"
|
||||
#include "index-layout.h"
|
||||
#include "index-page-map.h"
|
||||
#include "radix-sort.h"
|
||||
#include "sparse-cache.h"
|
||||
|
||||
/*
|
||||
* The volume manages deduplication records on permanent storage. The term "volume" can also refer
|
||||
* to the region of permanent storage where the records (and the chapters containing them) are
|
||||
* stored. The volume handles all I/O to this region by reading, caching, and writing chapter pages
|
||||
* as necessary.
|
||||
*/
|
||||
|
||||
enum index_lookup_mode {
|
||||
/* Always do lookups in all chapters normally */
|
||||
LOOKUP_NORMAL,
|
||||
/* Only do a subset of lookups needed when rebuilding an index */
|
||||
LOOKUP_FOR_REBUILD,
|
||||
};
|
||||
|
||||
struct queued_read {
|
||||
bool invalid;
|
||||
bool reserved;
|
||||
u32 physical_page;
|
||||
struct uds_request *first_request;
|
||||
struct uds_request *last_request;
|
||||
};
|
||||
|
||||
struct __aligned(L1_CACHE_BYTES) search_pending_counter {
|
||||
u64 atomic_value;
|
||||
};
|
||||
|
||||
struct cached_page {
|
||||
/* Whether this page is currently being read asynchronously */
|
||||
bool read_pending;
|
||||
/* The physical page stored in this cache entry */
|
||||
u32 physical_page;
|
||||
/* The value of the volume clock when this page was last used */
|
||||
s64 last_used;
|
||||
/* The cached page buffer */
|
||||
struct dm_buffer *buffer;
|
||||
/* The chapter index page, meaningless for record pages */
|
||||
struct delta_index_page index_page;
|
||||
};
|
||||
|
||||
struct page_cache {
|
||||
/* The number of zones */
|
||||
unsigned int zone_count;
|
||||
/* The number of volume pages that can be cached */
|
||||
u32 indexable_pages;
|
||||
/* The maximum number of simultaneously cached pages */
|
||||
u16 cache_slots;
|
||||
/* An index for each physical page noting where it is in the cache */
|
||||
u16 *index;
|
||||
/* The array of cached pages */
|
||||
struct cached_page *cache;
|
||||
/* A counter for each zone tracking if a search is occurring there */
|
||||
struct search_pending_counter *search_pending_counters;
|
||||
/* The read queue entries as a circular array */
|
||||
struct queued_read *read_queue;
|
||||
|
||||
/* All entries above this point are constant after initialization. */
|
||||
|
||||
/*
|
||||
* These values are all indexes into the array of read queue entries. New entries in the
|
||||
* read queue are enqueued at read_queue_last. To dequeue entries, a reader thread gets the
|
||||
* lock and then claims the entry pointed to by read_queue_next_read and increments that
|
||||
* value. After the read is completed, the reader thread calls release_read_queue_entry(),
|
||||
* which increments read_queue_first until it points to a pending read, or is equal to
|
||||
* read_queue_next_read. This means that if multiple reads are outstanding,
|
||||
* read_queue_first might not advance until the last of the reads finishes.
|
||||
*/
|
||||
u16 read_queue_first;
|
||||
u16 read_queue_next_read;
|
||||
u16 read_queue_last;
|
||||
|
||||
atomic64_t clock;
|
||||
};
|
||||
|
||||
struct volume {
|
||||
struct index_geometry *geometry;
|
||||
struct dm_bufio_client *client;
|
||||
u64 nonce;
|
||||
size_t cache_size;
|
||||
|
||||
/* A single page worth of records, for sorting */
|
||||
const struct uds_volume_record **record_pointers;
|
||||
/* Sorter for sorting records within each page */
|
||||
struct radix_sorter *radix_sorter;
|
||||
|
||||
struct sparse_cache *sparse_cache;
|
||||
struct page_cache page_cache;
|
||||
struct index_page_map *index_page_map;
|
||||
|
||||
struct mutex read_threads_mutex;
|
||||
struct cond_var read_threads_cond;
|
||||
struct cond_var read_threads_read_done_cond;
|
||||
struct thread **reader_threads;
|
||||
unsigned int read_thread_count;
|
||||
bool read_threads_exiting;
|
||||
|
||||
enum index_lookup_mode lookup_mode;
|
||||
unsigned int reserved_buffers;
|
||||
};
|
||||
|
||||
int __must_check uds_make_volume(const struct uds_configuration *config,
|
||||
struct index_layout *layout,
|
||||
struct volume **new_volume);
|
||||
|
||||
void uds_free_volume(struct volume *volume);
|
||||
|
||||
int __must_check uds_replace_volume_storage(struct volume *volume,
|
||||
struct index_layout *layout,
|
||||
struct block_device *bdev);
|
||||
|
||||
int __must_check uds_find_volume_chapter_boundaries(struct volume *volume,
|
||||
u64 *lowest_vcn, u64 *highest_vcn,
|
||||
bool *is_empty);
|
||||
|
||||
int __must_check uds_search_volume_page_cache(struct volume *volume,
|
||||
struct uds_request *request,
|
||||
bool *found);
|
||||
|
||||
int __must_check uds_search_volume_page_cache_for_rebuild(struct volume *volume,
|
||||
const struct uds_record_name *name,
|
||||
u64 virtual_chapter,
|
||||
bool *found);
|
||||
|
||||
int __must_check uds_search_cached_record_page(struct volume *volume,
|
||||
struct uds_request *request, u32 chapter,
|
||||
u16 record_page_number, bool *found);
|
||||
|
||||
void uds_forget_chapter(struct volume *volume, u64 chapter);
|
||||
|
||||
int __must_check uds_write_chapter(struct volume *volume,
|
||||
struct open_chapter_index *chapter_index,
|
||||
const struct uds_volume_record records[]);
|
||||
|
||||
void uds_prefetch_volume_chapter(const struct volume *volume, u32 chapter);
|
||||
|
||||
int __must_check uds_read_chapter_index_from_volume(const struct volume *volume,
|
||||
u64 virtual_chapter,
|
||||
struct dm_buffer *volume_buffers[],
|
||||
struct delta_index_page index_pages[]);
|
||||
|
||||
int __must_check uds_get_volume_record_page(struct volume *volume, u32 chapter,
|
||||
u32 page_number, u8 **data_ptr);
|
||||
|
||||
int __must_check uds_get_volume_index_page(struct volume *volume, u32 chapter,
|
||||
u32 page_number,
|
||||
struct delta_index_page **page_ptr);
|
||||
|
||||
#endif /* UDS_VOLUME_H */
|
|
@ -0,0 +1,707 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
/**
|
||||
* DOC:
|
||||
*
|
||||
* Hash table implementation of a map from integers to pointers, implemented using the Hopscotch
|
||||
* Hashing algorithm by Herlihy, Shavit, and Tzafrir (see
|
||||
* http://en.wikipedia.org/wiki/Hopscotch_hashing). This implementation does not contain any of the
|
||||
* locking/concurrency features of the algorithm, just the collision resolution scheme.
|
||||
*
|
||||
* Hopscotch Hashing is based on hashing with open addressing and linear probing. All the entries
|
||||
* are stored in a fixed array of buckets, with no dynamic allocation for collisions. Unlike linear
|
||||
* probing, all the entries that hash to a given bucket are stored within a fixed neighborhood
|
||||
* starting at that bucket. Chaining is effectively represented as a bit vector relative to each
|
||||
* bucket instead of as pointers or explicit offsets.
|
||||
*
|
||||
* When an empty bucket cannot be found within a given neighborhood, subsequent neighborhoods are
|
||||
* searched, and one or more entries will "hop" into those neighborhoods. When this process works,
|
||||
* an empty bucket will move into the desired neighborhood, allowing the entry to be added. When
|
||||
* that process fails (typically when the buckets are around 90% full), the table must be resized
|
||||
* and the all entries rehashed and added to the expanded table.
|
||||
*
|
||||
* Unlike linear probing, the number of buckets that must be searched in the worst case has a fixed
|
||||
* upper bound (the size of the neighborhood). Those entries occupy a small number of memory cache
|
||||
* lines, leading to improved use of the cache (fewer misses on both successful and unsuccessful
|
||||
* searches). Hopscotch hashing outperforms linear probing at much higher load factors, so even
|
||||
* with the increased memory burden for maintaining the hop vectors, less memory is needed to
|
||||
* achieve that performance. Hopscotch is also immune to "contamination" from deleting entries
|
||||
* since entries are genuinely removed instead of being replaced by a placeholder.
|
||||
*
|
||||
* The published description of the algorithm used a bit vector, but the paper alludes to an offset
|
||||
* scheme which is used by this implementation. Since the entries in the neighborhood are within N
|
||||
* entries of the hash bucket at the start of the neighborhood, a pair of small offset fields each
|
||||
* log2(N) bits wide is all that's needed to maintain the hops as a linked list. In order to encode
|
||||
* "no next hop" (i.e. NULL) as the natural initial value of zero, the offsets are biased by one
|
||||
* (i.e. 0 => NULL, 1 => offset=0, 2 => offset=1, etc.) We can represent neighborhoods of up to 255
|
||||
* entries with just 8+8=16 bits per entry. The hop list is sorted by hop offset so the first entry
|
||||
* in the list is always the bucket closest to the start of the neighborhood.
|
||||
*
|
||||
* While individual accesses tend to be very fast, the table resize operations are very, very
|
||||
* expensive. If an upper bound on the latency of adding an entry to the table is needed, we either
|
||||
* need to ensure the table is pre-sized to be large enough so no resize is ever needed, or we'll
|
||||
* need to develop an approach to incrementally resize the table.
|
||||
*/
|
||||
|
||||
#include "int-map.h"
|
||||
|
||||
#include <linux/minmax.h>
|
||||
|
||||
#include "errors.h"
|
||||
#include "logger.h"
|
||||
#include "memory-alloc.h"
|
||||
#include "numeric.h"
|
||||
#include "permassert.h"
|
||||
|
||||
#define DEFAULT_CAPACITY 16 /* the number of neighborhoods in a new table */
|
||||
#define NEIGHBORHOOD 255 /* the number of buckets in each neighborhood */
|
||||
#define MAX_PROBES 1024 /* limit on the number of probes for a free bucket */
|
||||
#define NULL_HOP_OFFSET 0 /* the hop offset value terminating the hop list */
|
||||
#define DEFAULT_LOAD 75 /* a compromise between memory use and performance */
|
||||
|
||||
/**
|
||||
* struct bucket - hash bucket
|
||||
*
|
||||
* Buckets are packed together to reduce memory usage and improve cache efficiency. It would be
|
||||
* tempting to encode the hop offsets separately and maintain alignment of key/value pairs, but
|
||||
* it's crucial to keep the hop fields near the buckets that they use them so they'll tend to share
|
||||
* cache lines.
|
||||
*/
|
||||
struct __packed bucket {
|
||||
/**
|
||||
* @first_hop: The biased offset of the first entry in the hop list of the neighborhood
|
||||
* that hashes to this bucket.
|
||||
*/
|
||||
u8 first_hop;
|
||||
/** @next_hop: The biased offset of the next bucket in the hop list. */
|
||||
u8 next_hop;
|
||||
/** @key: The key stored in this bucket. */
|
||||
u64 key;
|
||||
/** @value: The value stored in this bucket (NULL if empty). */
|
||||
void *value;
|
||||
};
|
||||
|
||||
/**
|
||||
* struct int_map - The concrete definition of the opaque int_map type.
|
||||
*
|
||||
* To avoid having to wrap the neighborhoods of the last entries back around to the start of the
|
||||
* bucket array, we allocate a few more buckets at the end of the array instead, which is why
|
||||
* capacity and bucket_count are different.
|
||||
*/
|
||||
struct int_map {
|
||||
/** @size: The number of entries stored in the map. */
|
||||
size_t size;
|
||||
/** @capacity: The number of neighborhoods in the map. */
|
||||
size_t capacity;
|
||||
/* @bucket_count: The number of buckets in the bucket array. */
|
||||
size_t bucket_count;
|
||||
/** @buckets: The array of hash buckets. */
|
||||
struct bucket *buckets;
|
||||
};
|
||||
|
||||
/**
|
||||
* mix() - The Google CityHash 16-byte hash mixing function.
|
||||
* @input1: The first input value.
|
||||
* @input2: The second input value.
|
||||
*
|
||||
* Return: A hash of the two inputs.
|
||||
*/
|
||||
static u64 mix(u64 input1, u64 input2)
|
||||
{
|
||||
static const u64 CITY_MULTIPLIER = 0x9ddfea08eb382d69ULL;
|
||||
u64 hash = (input1 ^ input2);
|
||||
|
||||
hash *= CITY_MULTIPLIER;
|
||||
hash ^= (hash >> 47);
|
||||
hash ^= input2;
|
||||
hash *= CITY_MULTIPLIER;
|
||||
hash ^= (hash >> 47);
|
||||
hash *= CITY_MULTIPLIER;
|
||||
return hash;
|
||||
}
|
||||
|
||||
/**
|
||||
* hash_key() - Calculate a 64-bit non-cryptographic hash value for the provided 64-bit integer
|
||||
* key.
|
||||
* @key: The mapping key.
|
||||
*
|
||||
* The implementation is based on Google's CityHash, only handling the specific case of an 8-byte
|
||||
* input.
|
||||
*
|
||||
* Return: The hash of the mapping key.
|
||||
*/
|
||||
static u64 hash_key(u64 key)
|
||||
{
|
||||
/*
|
||||
* Aliasing restrictions forbid us from casting pointer types, so use a union to convert a
|
||||
* single u64 to two u32 values.
|
||||
*/
|
||||
union {
|
||||
u64 u64;
|
||||
u32 u32[2];
|
||||
} pun = {.u64 = key};
|
||||
|
||||
return mix(sizeof(key) + (((u64) pun.u32[0]) << 3), pun.u32[1]);
|
||||
}
|
||||
|
||||
/**
|
||||
* allocate_buckets() - Initialize an int_map.
|
||||
* @map: The map to initialize.
|
||||
* @capacity: The initial capacity of the map.
|
||||
*
|
||||
* Return: VDO_SUCCESS or an error code.
|
||||
*/
|
||||
static int allocate_buckets(struct int_map *map, size_t capacity)
|
||||
{
|
||||
map->size = 0;
|
||||
map->capacity = capacity;
|
||||
|
||||
/*
|
||||
* Allocate NEIGHBORHOOD - 1 extra buckets so the last bucket can have a full neighborhood
|
||||
* without have to wrap back around to element zero.
|
||||
*/
|
||||
map->bucket_count = capacity + (NEIGHBORHOOD - 1);
|
||||
return vdo_allocate(map->bucket_count, struct bucket,
|
||||
"struct int_map buckets", &map->buckets);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_int_map_create() - Allocate and initialize an int_map.
|
||||
* @initial_capacity: The number of entries the map should initially be capable of holding (zero
|
||||
* tells the map to use its own small default).
|
||||
* @map_ptr: Output, a pointer to hold the new int_map.
|
||||
*
|
||||
* Return: VDO_SUCCESS or an error code.
|
||||
*/
|
||||
int vdo_int_map_create(size_t initial_capacity, struct int_map **map_ptr)
|
||||
{
|
||||
struct int_map *map;
|
||||
int result;
|
||||
size_t capacity;
|
||||
|
||||
result = vdo_allocate(1, struct int_map, "struct int_map", &map);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
/* Use the default capacity if the caller did not specify one. */
|
||||
capacity = (initial_capacity > 0) ? initial_capacity : DEFAULT_CAPACITY;
|
||||
|
||||
/*
|
||||
* Scale up the capacity by the specified initial load factor. (i.e to hold 1000 entries at
|
||||
* 80% load we need a capacity of 1250)
|
||||
*/
|
||||
capacity = capacity * 100 / DEFAULT_LOAD;
|
||||
|
||||
result = allocate_buckets(map, capacity);
|
||||
if (result != VDO_SUCCESS) {
|
||||
vdo_int_map_free(vdo_forget(map));
|
||||
return result;
|
||||
}
|
||||
|
||||
*map_ptr = map;
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_int_map_free() - Free an int_map.
|
||||
* @map: The int_map to free.
|
||||
*
|
||||
* NOTE: The map does not own the pointer values stored in the map and they are not freed by this
|
||||
* call.
|
||||
*/
|
||||
void vdo_int_map_free(struct int_map *map)
|
||||
{
|
||||
if (map == NULL)
|
||||
return;
|
||||
|
||||
vdo_free(vdo_forget(map->buckets));
|
||||
vdo_free(vdo_forget(map));
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_int_map_size() - Get the number of entries stored in an int_map.
|
||||
* @map: The int_map to query.
|
||||
*
|
||||
* Return: The number of entries in the map.
|
||||
*/
|
||||
size_t vdo_int_map_size(const struct int_map *map)
|
||||
{
|
||||
return map->size;
|
||||
}
|
||||
|
||||
/**
|
||||
* dereference_hop() - Convert a biased hop offset within a neighborhood to a pointer to the bucket
|
||||
* it references.
|
||||
* @neighborhood: The first bucket in the neighborhood.
|
||||
* @hop_offset: The biased hop offset to the desired bucket.
|
||||
*
|
||||
* Return: NULL if hop_offset is zero, otherwise a pointer to the bucket in the neighborhood at
|
||||
* hop_offset - 1.
|
||||
*/
|
||||
static struct bucket *dereference_hop(struct bucket *neighborhood, unsigned int hop_offset)
|
||||
{
|
||||
BUILD_BUG_ON(NULL_HOP_OFFSET != 0);
|
||||
if (hop_offset == NULL_HOP_OFFSET)
|
||||
return NULL;
|
||||
|
||||
return &neighborhood[hop_offset - 1];
|
||||
}
|
||||
|
||||
/**
|
||||
* insert_in_hop_list() - Add a bucket into the hop list for the neighborhood.
|
||||
* @neighborhood: The first bucket in the neighborhood.
|
||||
* @new_bucket: The bucket to add to the hop list.
|
||||
*
|
||||
* The bucket is inserted it into the list so the hop list remains sorted by hop offset.
|
||||
*/
|
||||
static void insert_in_hop_list(struct bucket *neighborhood, struct bucket *new_bucket)
|
||||
{
|
||||
/* Zero indicates a NULL hop offset, so bias the hop offset by one. */
|
||||
int hop_offset = 1 + (new_bucket - neighborhood);
|
||||
|
||||
/* Handle the special case of adding a bucket at the start of the list. */
|
||||
int next_hop = neighborhood->first_hop;
|
||||
|
||||
if ((next_hop == NULL_HOP_OFFSET) || (next_hop > hop_offset)) {
|
||||
new_bucket->next_hop = next_hop;
|
||||
neighborhood->first_hop = hop_offset;
|
||||
return;
|
||||
}
|
||||
|
||||
/* Search the hop list for the insertion point that maintains the sort order. */
|
||||
for (;;) {
|
||||
struct bucket *bucket = dereference_hop(neighborhood, next_hop);
|
||||
|
||||
next_hop = bucket->next_hop;
|
||||
|
||||
if ((next_hop == NULL_HOP_OFFSET) || (next_hop > hop_offset)) {
|
||||
new_bucket->next_hop = next_hop;
|
||||
bucket->next_hop = hop_offset;
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* select_bucket() - Select and return the hash bucket for a given search key.
|
||||
* @map: The map to search.
|
||||
* @key: The mapping key.
|
||||
*/
|
||||
static struct bucket *select_bucket(const struct int_map *map, u64 key)
|
||||
{
|
||||
/*
|
||||
* Calculate a good hash value for the provided key. We want exactly 32 bits, so mask the
|
||||
* result.
|
||||
*/
|
||||
u64 hash = hash_key(key) & 0xFFFFFFFF;
|
||||
|
||||
/*
|
||||
* Scale the 32-bit hash to a bucket index by treating it as a binary fraction and
|
||||
* multiplying that by the capacity. If the hash is uniformly distributed over [0 ..
|
||||
* 2^32-1], then (hash * capacity / 2^32) should be uniformly distributed over [0 ..
|
||||
* capacity-1]. The multiply and shift is much faster than a divide (modulus) on X86 CPUs.
|
||||
*/
|
||||
return &map->buckets[(hash * map->capacity) >> 32];
|
||||
}
|
||||
|
||||
/**
|
||||
* search_hop_list() - Search the hop list associated with given hash bucket for a given search
|
||||
* key.
|
||||
* @map: The map being searched.
|
||||
* @bucket: The map bucket to search for the key.
|
||||
* @key: The mapping key.
|
||||
* @previous_ptr: Output. if not NULL, a pointer in which to store the bucket in the list preceding
|
||||
* the one that had the matching key
|
||||
*
|
||||
* If the key is found, returns a pointer to the entry (bucket or collision), otherwise returns
|
||||
* NULL.
|
||||
*
|
||||
* Return: An entry that matches the key, or NULL if not found.
|
||||
*/
|
||||
static struct bucket *search_hop_list(struct int_map *map __always_unused,
|
||||
struct bucket *bucket,
|
||||
u64 key,
|
||||
struct bucket **previous_ptr)
|
||||
{
|
||||
struct bucket *previous = NULL;
|
||||
unsigned int next_hop = bucket->first_hop;
|
||||
|
||||
while (next_hop != NULL_HOP_OFFSET) {
|
||||
/*
|
||||
* Check the neighboring bucket indexed by the offset for the
|
||||
* desired key.
|
||||
*/
|
||||
struct bucket *entry = dereference_hop(bucket, next_hop);
|
||||
|
||||
if ((key == entry->key) && (entry->value != NULL)) {
|
||||
if (previous_ptr != NULL)
|
||||
*previous_ptr = previous;
|
||||
return entry;
|
||||
}
|
||||
next_hop = entry->next_hop;
|
||||
previous = entry;
|
||||
}
|
||||
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_int_map_get() - Retrieve the value associated with a given key from the int_map.
|
||||
* @map: The int_map to query.
|
||||
* @key: The key to look up.
|
||||
*
|
||||
* Return: The value associated with the given key, or NULL if the key is not mapped to any value.
|
||||
*/
|
||||
void *vdo_int_map_get(struct int_map *map, u64 key)
|
||||
{
|
||||
struct bucket *match = search_hop_list(map, select_bucket(map, key), key, NULL);
|
||||
|
||||
return ((match != NULL) ? match->value : NULL);
|
||||
}
|
||||
|
||||
/**
|
||||
* resize_buckets() - Increase the number of hash buckets.
|
||||
* @map: The map to resize.
|
||||
*
|
||||
* Resizes and rehashes all the existing entries, storing them in the new buckets.
|
||||
*
|
||||
* Return: VDO_SUCCESS or an error code.
|
||||
*/
|
||||
static int resize_buckets(struct int_map *map)
|
||||
{
|
||||
int result;
|
||||
size_t i;
|
||||
|
||||
/* Copy the top-level map data to the stack. */
|
||||
struct int_map old_map = *map;
|
||||
|
||||
/* Re-initialize the map to be empty and 50% larger. */
|
||||
size_t new_capacity = map->capacity / 2 * 3;
|
||||
|
||||
vdo_log_info("%s: attempting resize from %zu to %zu, current size=%zu",
|
||||
__func__, map->capacity, new_capacity, map->size);
|
||||
result = allocate_buckets(map, new_capacity);
|
||||
if (result != VDO_SUCCESS) {
|
||||
*map = old_map;
|
||||
return result;
|
||||
}
|
||||
|
||||
/* Populate the new hash table from the entries in the old bucket array. */
|
||||
for (i = 0; i < old_map.bucket_count; i++) {
|
||||
struct bucket *entry = &old_map.buckets[i];
|
||||
|
||||
if (entry->value == NULL)
|
||||
continue;
|
||||
|
||||
result = vdo_int_map_put(map, entry->key, entry->value, true, NULL);
|
||||
if (result != VDO_SUCCESS) {
|
||||
/* Destroy the new partial map and restore the map from the stack. */
|
||||
vdo_free(vdo_forget(map->buckets));
|
||||
*map = old_map;
|
||||
return result;
|
||||
}
|
||||
}
|
||||
|
||||
/* Destroy the old bucket array. */
|
||||
vdo_free(vdo_forget(old_map.buckets));
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
/**
|
||||
* find_empty_bucket() - Probe the bucket array starting at the given bucket for the next empty
|
||||
* bucket, returning a pointer to it.
|
||||
* @map: The map containing the buckets to search.
|
||||
* @bucket: The bucket at which to start probing.
|
||||
* @max_probes: The maximum number of buckets to search.
|
||||
*
|
||||
* NULL will be returned if the search reaches the end of the bucket array or if the number of
|
||||
* linear probes exceeds a specified limit.
|
||||
*
|
||||
* Return: The next empty bucket, or NULL if the search failed.
|
||||
*/
|
||||
static struct bucket *
|
||||
find_empty_bucket(struct int_map *map, struct bucket *bucket, unsigned int max_probes)
|
||||
{
|
||||
/*
|
||||
* Limit the search to either the nearer of the end of the bucket array or a fixed distance
|
||||
* beyond the initial bucket.
|
||||
*/
|
||||
ptrdiff_t remaining = &map->buckets[map->bucket_count] - bucket;
|
||||
struct bucket *sentinel = &bucket[min_t(ptrdiff_t, remaining, max_probes)];
|
||||
struct bucket *entry;
|
||||
|
||||
for (entry = bucket; entry < sentinel; entry++) {
|
||||
if (entry->value == NULL)
|
||||
return entry;
|
||||
}
|
||||
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/**
|
||||
* move_empty_bucket() - Move an empty bucket closer to the start of the bucket array.
|
||||
* @map: The map containing the bucket.
|
||||
* @hole: The empty bucket to fill with an entry that precedes it in one of its enclosing
|
||||
* neighborhoods.
|
||||
*
|
||||
* This searches the neighborhoods that contain the empty bucket for a non-empty bucket closer to
|
||||
* the start of the array. If such a bucket is found, this swaps the two buckets by moving the
|
||||
* entry to the empty bucket.
|
||||
*
|
||||
* Return: The bucket that was vacated by moving its entry to the provided hole, or NULL if no
|
||||
* entry could be moved.
|
||||
*/
|
||||
static struct bucket *move_empty_bucket(struct int_map *map __always_unused,
|
||||
struct bucket *hole)
|
||||
{
|
||||
/*
|
||||
* Examine every neighborhood that the empty bucket is part of, starting with the one in
|
||||
* which it is the last bucket. No boundary check is needed for the negative array
|
||||
* arithmetic since this function is only called when hole is at least NEIGHBORHOOD cells
|
||||
* deeper into the array than a valid bucket.
|
||||
*/
|
||||
struct bucket *bucket;
|
||||
|
||||
for (bucket = &hole[1 - NEIGHBORHOOD]; bucket < hole; bucket++) {
|
||||
/*
|
||||
* Find the entry that is nearest to the bucket, which means it will be nearest to
|
||||
* the hash bucket whose neighborhood is full.
|
||||
*/
|
||||
struct bucket *new_hole = dereference_hop(bucket, bucket->first_hop);
|
||||
|
||||
if (new_hole == NULL) {
|
||||
/*
|
||||
* There are no buckets in this neighborhood that are in use by this one
|
||||
* (they must all be owned by overlapping neighborhoods).
|
||||
*/
|
||||
continue;
|
||||
}
|
||||
|
||||
/*
|
||||
* Skip this bucket if its first entry is actually further away than the hole that
|
||||
* we're already trying to fill.
|
||||
*/
|
||||
if (hole < new_hole)
|
||||
continue;
|
||||
|
||||
/*
|
||||
* We've found an entry in this neighborhood that we can "hop" further away, moving
|
||||
* the hole closer to the hash bucket, if not all the way into its neighborhood.
|
||||
*/
|
||||
|
||||
/*
|
||||
* The entry that will be the new hole is the first bucket in the list, so setting
|
||||
* first_hop is all that's needed remove it from the list.
|
||||
*/
|
||||
bucket->first_hop = new_hole->next_hop;
|
||||
new_hole->next_hop = NULL_HOP_OFFSET;
|
||||
|
||||
/* Move the entry into the original hole. */
|
||||
hole->key = new_hole->key;
|
||||
hole->value = new_hole->value;
|
||||
new_hole->value = NULL;
|
||||
|
||||
/* Insert the filled hole into the hop list for the neighborhood. */
|
||||
insert_in_hop_list(bucket, hole);
|
||||
return new_hole;
|
||||
}
|
||||
|
||||
/* We couldn't find an entry to relocate to the hole. */
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/**
|
||||
* update_mapping() - Find and update any existing mapping for a given key, returning the value
|
||||
* associated with the key in the provided pointer.
|
||||
* @map: The int_map to attempt to modify.
|
||||
* @neighborhood: The first bucket in the neighborhood that would contain the search key
|
||||
* @key: The key with which to associate the new value.
|
||||
* @new_value: The value to be associated with the key.
|
||||
* @update: Whether to overwrite an existing value.
|
||||
* @old_value_ptr: a pointer in which to store the old value (unmodified if no mapping was found)
|
||||
*
|
||||
* Return: true if the map contains a mapping for the key, false if it does not.
|
||||
*/
|
||||
static bool update_mapping(struct int_map *map, struct bucket *neighborhood,
|
||||
u64 key, void *new_value, bool update, void **old_value_ptr)
|
||||
{
|
||||
struct bucket *bucket = search_hop_list(map, neighborhood, key, NULL);
|
||||
|
||||
if (bucket == NULL) {
|
||||
/* There is no bucket containing the key in the neighborhood. */
|
||||
return false;
|
||||
}
|
||||
|
||||
/*
|
||||
* Return the value of the current mapping (if desired) and update the mapping with the new
|
||||
* value (if desired).
|
||||
*/
|
||||
if (old_value_ptr != NULL)
|
||||
*old_value_ptr = bucket->value;
|
||||
if (update)
|
||||
bucket->value = new_value;
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* find_or_make_vacancy() - Find an empty bucket.
|
||||
* @map: The int_map to search or modify.
|
||||
* @neighborhood: The first bucket in the neighborhood in which an empty bucket is needed for a new
|
||||
* mapping.
|
||||
*
|
||||
* Find an empty bucket in a specified neighborhood for a new mapping or attempt to re-arrange
|
||||
* mappings so there is such a bucket. This operation may fail (returning NULL) if an empty bucket
|
||||
* is not available or could not be relocated to the neighborhood.
|
||||
*
|
||||
* Return: a pointer to an empty bucket in the desired neighborhood, or NULL if a vacancy could not
|
||||
* be found or arranged.
|
||||
*/
|
||||
static struct bucket *find_or_make_vacancy(struct int_map *map,
|
||||
struct bucket *neighborhood)
|
||||
{
|
||||
/* Probe within and beyond the neighborhood for the first empty bucket. */
|
||||
struct bucket *hole = find_empty_bucket(map, neighborhood, MAX_PROBES);
|
||||
|
||||
/*
|
||||
* Keep trying until the empty bucket is in the bucket's neighborhood or we are unable to
|
||||
* move it any closer by swapping it with a filled bucket.
|
||||
*/
|
||||
while (hole != NULL) {
|
||||
int distance = hole - neighborhood;
|
||||
|
||||
if (distance < NEIGHBORHOOD) {
|
||||
/*
|
||||
* We've found or relocated an empty bucket close enough to the initial
|
||||
* hash bucket to be referenced by its hop vector.
|
||||
*/
|
||||
return hole;
|
||||
}
|
||||
|
||||
/*
|
||||
* The nearest empty bucket isn't within the neighborhood that must contain the new
|
||||
* entry, so try to swap it with bucket that is closer.
|
||||
*/
|
||||
hole = move_empty_bucket(map, hole);
|
||||
}
|
||||
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_int_map_put() - Try to associate a value with an integer.
|
||||
* @map: The int_map to attempt to modify.
|
||||
* @key: The key with which to associate the new value.
|
||||
* @new_value: The value to be associated with the key.
|
||||
* @update: Whether to overwrite an existing value.
|
||||
* @old_value_ptr: A pointer in which to store either the old value (if the key was already mapped)
|
||||
* or NULL if the map did not contain the key; NULL may be provided if the caller
|
||||
* does not need to know the old value
|
||||
*
|
||||
* Try to associate a value (a pointer) with an integer in an int_map. If the map already contains
|
||||
* a mapping for the provided key, the old value is only replaced with the specified value if
|
||||
* update is true. In either case the old value is returned. If the map does not already contain a
|
||||
* value for the specified key, the new value is added regardless of the value of update.
|
||||
*
|
||||
* Return: VDO_SUCCESS or an error code.
|
||||
*/
|
||||
int vdo_int_map_put(struct int_map *map, u64 key, void *new_value, bool update,
|
||||
void **old_value_ptr)
|
||||
{
|
||||
struct bucket *neighborhood, *bucket;
|
||||
|
||||
if (unlikely(new_value == NULL))
|
||||
return -EINVAL;
|
||||
|
||||
/*
|
||||
* Select the bucket at the start of the neighborhood that must contain any entry for the
|
||||
* provided key.
|
||||
*/
|
||||
neighborhood = select_bucket(map, key);
|
||||
|
||||
/*
|
||||
* Check whether the neighborhood already contains an entry for the key, in which case we
|
||||
* optionally update it, returning the old value.
|
||||
*/
|
||||
if (update_mapping(map, neighborhood, key, new_value, update, old_value_ptr))
|
||||
return VDO_SUCCESS;
|
||||
|
||||
/*
|
||||
* Find an empty bucket in the desired neighborhood for the new entry or re-arrange entries
|
||||
* in the map so there is such a bucket. This operation will usually succeed; the loop body
|
||||
* will only be executed on the rare occasions that we have to resize the map.
|
||||
*/
|
||||
while ((bucket = find_or_make_vacancy(map, neighborhood)) == NULL) {
|
||||
int result;
|
||||
|
||||
/*
|
||||
* There is no empty bucket in which to put the new entry in the current map, so
|
||||
* we're forced to allocate a new bucket array with a larger capacity, re-hash all
|
||||
* the entries into those buckets, and try again (a very expensive operation for
|
||||
* large maps).
|
||||
*/
|
||||
result = resize_buckets(map);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
/*
|
||||
* Resizing the map invalidates all pointers to buckets, so recalculate the
|
||||
* neighborhood pointer.
|
||||
*/
|
||||
neighborhood = select_bucket(map, key);
|
||||
}
|
||||
|
||||
/* Put the new entry in the empty bucket, adding it to the neighborhood. */
|
||||
bucket->key = key;
|
||||
bucket->value = new_value;
|
||||
insert_in_hop_list(neighborhood, bucket);
|
||||
map->size += 1;
|
||||
|
||||
/* There was no existing entry, so there was no old value to be returned. */
|
||||
if (old_value_ptr != NULL)
|
||||
*old_value_ptr = NULL;
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_int_map_remove() - Remove the mapping for a given key from the int_map.
|
||||
* @map: The int_map from which to remove the mapping.
|
||||
* @key: The key whose mapping is to be removed.
|
||||
*
|
||||
* Return: the value that was associated with the key, or NULL if it was not mapped.
|
||||
*/
|
||||
void *vdo_int_map_remove(struct int_map *map, u64 key)
|
||||
{
|
||||
void *value;
|
||||
|
||||
/* Select the bucket to search and search it for an existing entry. */
|
||||
struct bucket *bucket = select_bucket(map, key);
|
||||
struct bucket *previous;
|
||||
struct bucket *victim = search_hop_list(map, bucket, key, &previous);
|
||||
|
||||
if (victim == NULL) {
|
||||
/* There is no matching entry to remove. */
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/*
|
||||
* We found an entry to remove. Save the mapped value to return later and empty the bucket.
|
||||
*/
|
||||
map->size -= 1;
|
||||
value = victim->value;
|
||||
victim->value = NULL;
|
||||
victim->key = 0;
|
||||
|
||||
/* The victim bucket is now empty, but it still needs to be spliced out of the hop list. */
|
||||
if (previous == NULL) {
|
||||
/* The victim is the head of the list, so swing first_hop. */
|
||||
bucket->first_hop = victim->next_hop;
|
||||
} else {
|
||||
previous->next_hop = victim->next_hop;
|
||||
}
|
||||
|
||||
victim->next_hop = NULL_HOP_OFFSET;
|
||||
return value;
|
||||
}
|
|
@ -0,0 +1,39 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef VDO_INT_MAP_H
|
||||
#define VDO_INT_MAP_H
|
||||
|
||||
#include <linux/compiler.h>
|
||||
#include <linux/types.h>
|
||||
|
||||
/**
|
||||
* DOC: int_map
|
||||
*
|
||||
* An int_map associates pointers (void *) with integer keys (u64). NULL pointer values are
|
||||
* not supported.
|
||||
*
|
||||
* The map is implemented as hash table, which should provide constant-time insert, query, and
|
||||
* remove operations, although the insert may occasionally grow the table, which is linear in the
|
||||
* number of entries in the map. The table will grow as needed to hold new entries, but will not
|
||||
* shrink as entries are removed.
|
||||
*/
|
||||
|
||||
struct int_map;
|
||||
|
||||
int __must_check vdo_int_map_create(size_t initial_capacity, struct int_map **map_ptr);
|
||||
|
||||
void vdo_int_map_free(struct int_map *map);
|
||||
|
||||
size_t vdo_int_map_size(const struct int_map *map);
|
||||
|
||||
void *vdo_int_map_get(struct int_map *map, u64 key);
|
||||
|
||||
int __must_check vdo_int_map_put(struct int_map *map, u64 key, void *new_value,
|
||||
bool update, void **old_value_ptr);
|
||||
|
||||
void *vdo_int_map_remove(struct int_map *map, u64 key);
|
||||
|
||||
#endif /* VDO_INT_MAP_H */
|
|
@ -0,0 +1,477 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "io-submitter.h"
|
||||
|
||||
#include <linux/bio.h>
|
||||
#include <linux/kernel.h>
|
||||
#include <linux/mutex.h>
|
||||
|
||||
#include "memory-alloc.h"
|
||||
#include "permassert.h"
|
||||
|
||||
#include "data-vio.h"
|
||||
#include "logger.h"
|
||||
#include "types.h"
|
||||
#include "vdo.h"
|
||||
#include "vio.h"
|
||||
|
||||
/*
|
||||
* Submission of bio operations to the underlying storage device will go through a separate work
|
||||
* queue thread (or more than one) to prevent blocking in other threads if the storage device has a
|
||||
* full queue. The plug structure allows that thread to do better batching of requests to make the
|
||||
* I/O more efficient.
|
||||
*
|
||||
* When multiple worker threads are used, a thread is chosen for a I/O operation submission based
|
||||
* on the PBN, so a given PBN will consistently wind up on the same thread. Flush operations are
|
||||
* assigned round-robin.
|
||||
*
|
||||
* The map (protected by the mutex) collects pending I/O operations so that the worker thread can
|
||||
* reorder them to try to encourage I/O request merging in the request queue underneath.
|
||||
*/
|
||||
struct bio_queue_data {
|
||||
struct vdo_work_queue *queue;
|
||||
struct blk_plug plug;
|
||||
struct int_map *map;
|
||||
struct mutex lock;
|
||||
unsigned int queue_number;
|
||||
};
|
||||
|
||||
struct io_submitter {
|
||||
unsigned int num_bio_queues_used;
|
||||
unsigned int bio_queue_rotation_interval;
|
||||
struct bio_queue_data bio_queue_data[];
|
||||
};
|
||||
|
||||
static void start_bio_queue(void *ptr)
|
||||
{
|
||||
struct bio_queue_data *bio_queue_data = ptr;
|
||||
|
||||
blk_start_plug(&bio_queue_data->plug);
|
||||
}
|
||||
|
||||
static void finish_bio_queue(void *ptr)
|
||||
{
|
||||
struct bio_queue_data *bio_queue_data = ptr;
|
||||
|
||||
blk_finish_plug(&bio_queue_data->plug);
|
||||
}
|
||||
|
||||
static const struct vdo_work_queue_type bio_queue_type = {
|
||||
.start = start_bio_queue,
|
||||
.finish = finish_bio_queue,
|
||||
.max_priority = BIO_Q_MAX_PRIORITY,
|
||||
.default_priority = BIO_Q_DATA_PRIORITY,
|
||||
};
|
||||
|
||||
/**
|
||||
* count_all_bios() - Determine which bio counter to use.
|
||||
* @vio: The vio associated with the bio.
|
||||
* @bio: The bio to count.
|
||||
*/
|
||||
static void count_all_bios(struct vio *vio, struct bio *bio)
|
||||
{
|
||||
struct atomic_statistics *stats = &vio->completion.vdo->stats;
|
||||
|
||||
if (is_data_vio(vio)) {
|
||||
vdo_count_bios(&stats->bios_out, bio);
|
||||
return;
|
||||
}
|
||||
|
||||
vdo_count_bios(&stats->bios_meta, bio);
|
||||
if (vio->type == VIO_TYPE_RECOVERY_JOURNAL)
|
||||
vdo_count_bios(&stats->bios_journal, bio);
|
||||
else if (vio->type == VIO_TYPE_BLOCK_MAP)
|
||||
vdo_count_bios(&stats->bios_page_cache, bio);
|
||||
}
|
||||
|
||||
/**
|
||||
* assert_in_bio_zone() - Assert that a vio is in the correct bio zone and not in interrupt
|
||||
* context.
|
||||
* @vio: The vio to check.
|
||||
*/
|
||||
static void assert_in_bio_zone(struct vio *vio)
|
||||
{
|
||||
VDO_ASSERT_LOG_ONLY(!in_interrupt(), "not in interrupt context");
|
||||
assert_vio_in_bio_zone(vio);
|
||||
}
|
||||
|
||||
/**
|
||||
* send_bio_to_device() - Update stats and tracing info, then submit the supplied bio to the OS for
|
||||
* processing.
|
||||
* @vio: The vio associated with the bio.
|
||||
* @bio: The bio to submit to the OS.
|
||||
*/
|
||||
static void send_bio_to_device(struct vio *vio, struct bio *bio)
|
||||
{
|
||||
struct vdo *vdo = vio->completion.vdo;
|
||||
|
||||
assert_in_bio_zone(vio);
|
||||
atomic64_inc(&vdo->stats.bios_submitted);
|
||||
count_all_bios(vio, bio);
|
||||
bio_set_dev(bio, vdo_get_backing_device(vdo));
|
||||
submit_bio_noacct(bio);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_submit_vio() - Submits a vio's bio to the underlying block device. May block if the device
|
||||
* is busy. This callback should be used by vios which did not attempt to merge.
|
||||
*/
|
||||
void vdo_submit_vio(struct vdo_completion *completion)
|
||||
{
|
||||
struct vio *vio = as_vio(completion);
|
||||
|
||||
send_bio_to_device(vio, vio->bio);
|
||||
}
|
||||
|
||||
/**
|
||||
* get_bio_list() - Extract the list of bios to submit from a vio.
|
||||
* @vio: The vio submitting I/O.
|
||||
*
|
||||
* The list will always contain at least one entry (the bio for the vio on which it is called), but
|
||||
* other bios may have been merged with it as well.
|
||||
*
|
||||
* Return: bio The head of the bio list to submit.
|
||||
*/
|
||||
static struct bio *get_bio_list(struct vio *vio)
|
||||
{
|
||||
struct bio *bio;
|
||||
struct io_submitter *submitter = vio->completion.vdo->io_submitter;
|
||||
struct bio_queue_data *bio_queue_data = &(submitter->bio_queue_data[vio->bio_zone]);
|
||||
|
||||
assert_in_bio_zone(vio);
|
||||
|
||||
mutex_lock(&bio_queue_data->lock);
|
||||
vdo_int_map_remove(bio_queue_data->map,
|
||||
vio->bios_merged.head->bi_iter.bi_sector);
|
||||
vdo_int_map_remove(bio_queue_data->map,
|
||||
vio->bios_merged.tail->bi_iter.bi_sector);
|
||||
bio = vio->bios_merged.head;
|
||||
bio_list_init(&vio->bios_merged);
|
||||
mutex_unlock(&bio_queue_data->lock);
|
||||
|
||||
return bio;
|
||||
}
|
||||
|
||||
/**
|
||||
* submit_data_vio() - Submit a data_vio's bio to the storage below along with
|
||||
* any bios that have been merged with it.
|
||||
*
|
||||
* Context: This call may block and so should only be called from a bio thread.
|
||||
*/
|
||||
static void submit_data_vio(struct vdo_completion *completion)
|
||||
{
|
||||
struct bio *bio, *next;
|
||||
struct vio *vio = as_vio(completion);
|
||||
|
||||
assert_in_bio_zone(vio);
|
||||
for (bio = get_bio_list(vio); bio != NULL; bio = next) {
|
||||
next = bio->bi_next;
|
||||
bio->bi_next = NULL;
|
||||
send_bio_to_device((struct vio *) bio->bi_private, bio);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* get_mergeable_locked() - Attempt to find an already queued bio that the current bio can be
|
||||
* merged with.
|
||||
* @map: The bio map to use for merging.
|
||||
* @vio: The vio we want to merge.
|
||||
* @back_merge: Set to true for a back merge, false for a front merge.
|
||||
*
|
||||
* There are two types of merging possible, forward and backward, which are distinguished by a flag
|
||||
* that uses kernel elevator terminology.
|
||||
*
|
||||
* Return: the vio to merge to, NULL if no merging is possible.
|
||||
*/
|
||||
static struct vio *get_mergeable_locked(struct int_map *map, struct vio *vio,
|
||||
bool back_merge)
|
||||
{
|
||||
struct bio *bio = vio->bio;
|
||||
sector_t merge_sector = bio->bi_iter.bi_sector;
|
||||
struct vio *vio_merge;
|
||||
|
||||
if (back_merge)
|
||||
merge_sector -= VDO_SECTORS_PER_BLOCK;
|
||||
else
|
||||
merge_sector += VDO_SECTORS_PER_BLOCK;
|
||||
|
||||
vio_merge = vdo_int_map_get(map, merge_sector);
|
||||
|
||||
if (vio_merge == NULL)
|
||||
return NULL;
|
||||
|
||||
if (vio->completion.priority != vio_merge->completion.priority)
|
||||
return NULL;
|
||||
|
||||
if (bio_data_dir(bio) != bio_data_dir(vio_merge->bio))
|
||||
return NULL;
|
||||
|
||||
if (bio_list_empty(&vio_merge->bios_merged))
|
||||
return NULL;
|
||||
|
||||
if (back_merge) {
|
||||
return (vio_merge->bios_merged.tail->bi_iter.bi_sector == merge_sector ?
|
||||
vio_merge : NULL);
|
||||
}
|
||||
|
||||
return (vio_merge->bios_merged.head->bi_iter.bi_sector == merge_sector ?
|
||||
vio_merge : NULL);
|
||||
}
|
||||
|
||||
static int map_merged_vio(struct int_map *bio_map, struct vio *vio)
|
||||
{
|
||||
int result;
|
||||
sector_t bio_sector;
|
||||
|
||||
bio_sector = vio->bios_merged.head->bi_iter.bi_sector;
|
||||
result = vdo_int_map_put(bio_map, bio_sector, vio, true, NULL);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
bio_sector = vio->bios_merged.tail->bi_iter.bi_sector;
|
||||
return vdo_int_map_put(bio_map, bio_sector, vio, true, NULL);
|
||||
}
|
||||
|
||||
static int merge_to_prev_tail(struct int_map *bio_map, struct vio *vio,
|
||||
struct vio *prev_vio)
|
||||
{
|
||||
vdo_int_map_remove(bio_map, prev_vio->bios_merged.tail->bi_iter.bi_sector);
|
||||
bio_list_merge(&prev_vio->bios_merged, &vio->bios_merged);
|
||||
return map_merged_vio(bio_map, prev_vio);
|
||||
}
|
||||
|
||||
static int merge_to_next_head(struct int_map *bio_map, struct vio *vio,
|
||||
struct vio *next_vio)
|
||||
{
|
||||
/*
|
||||
* Handle "next merge" and "gap fill" cases the same way so as to reorder bios in a way
|
||||
* that's compatible with using funnel queues in work queues. This avoids removing an
|
||||
* existing completion.
|
||||
*/
|
||||
vdo_int_map_remove(bio_map, next_vio->bios_merged.head->bi_iter.bi_sector);
|
||||
bio_list_merge_head(&next_vio->bios_merged, &vio->bios_merged);
|
||||
return map_merged_vio(bio_map, next_vio);
|
||||
}
|
||||
|
||||
/**
|
||||
* try_bio_map_merge() - Attempt to merge a vio's bio with other pending I/Os.
|
||||
* @vio: The vio to merge.
|
||||
*
|
||||
* Currently this is only used for data_vios, but is broken out for future use with metadata vios.
|
||||
*
|
||||
* Return: whether or not the vio was merged.
|
||||
*/
|
||||
static bool try_bio_map_merge(struct vio *vio)
|
||||
{
|
||||
int result;
|
||||
bool merged = true;
|
||||
struct bio *bio = vio->bio;
|
||||
struct vio *prev_vio, *next_vio;
|
||||
struct vdo *vdo = vio->completion.vdo;
|
||||
struct bio_queue_data *bio_queue_data =
|
||||
&vdo->io_submitter->bio_queue_data[vio->bio_zone];
|
||||
|
||||
bio->bi_next = NULL;
|
||||
bio_list_init(&vio->bios_merged);
|
||||
bio_list_add(&vio->bios_merged, bio);
|
||||
|
||||
mutex_lock(&bio_queue_data->lock);
|
||||
prev_vio = get_mergeable_locked(bio_queue_data->map, vio, true);
|
||||
next_vio = get_mergeable_locked(bio_queue_data->map, vio, false);
|
||||
if (prev_vio == next_vio)
|
||||
next_vio = NULL;
|
||||
|
||||
if ((prev_vio == NULL) && (next_vio == NULL)) {
|
||||
/* no merge. just add to bio_queue */
|
||||
merged = false;
|
||||
result = vdo_int_map_put(bio_queue_data->map,
|
||||
bio->bi_iter.bi_sector,
|
||||
vio, true, NULL);
|
||||
} else if (next_vio == NULL) {
|
||||
/* Only prev. merge to prev's tail */
|
||||
result = merge_to_prev_tail(bio_queue_data->map, vio, prev_vio);
|
||||
} else {
|
||||
/* Only next. merge to next's head */
|
||||
result = merge_to_next_head(bio_queue_data->map, vio, next_vio);
|
||||
}
|
||||
mutex_unlock(&bio_queue_data->lock);
|
||||
|
||||
/* We don't care about failure of int_map_put in this case. */
|
||||
VDO_ASSERT_LOG_ONLY(result == VDO_SUCCESS, "bio map insertion succeeds");
|
||||
return merged;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_submit_data_vio() - Submit I/O for a data_vio.
|
||||
* @data_vio: the data_vio for which to issue I/O.
|
||||
*
|
||||
* If possible, this I/O will be merged other pending I/Os. Otherwise, the data_vio will be sent to
|
||||
* the appropriate bio zone directly.
|
||||
*/
|
||||
void vdo_submit_data_vio(struct data_vio *data_vio)
|
||||
{
|
||||
if (try_bio_map_merge(&data_vio->vio))
|
||||
return;
|
||||
|
||||
launch_data_vio_bio_zone_callback(data_vio, submit_data_vio);
|
||||
}
|
||||
|
||||
/**
|
||||
* __submit_metadata_vio() - Submit I/O for a metadata vio.
|
||||
* @vio: the vio for which to issue I/O
|
||||
* @physical: the physical block number to read or write
|
||||
* @callback: the bio endio function which will be called after the I/O completes
|
||||
* @error_handler: the handler for submission or I/O errors (may be NULL)
|
||||
* @operation: the type of I/O to perform
|
||||
* @data: the buffer to read or write (may be NULL)
|
||||
*
|
||||
* The vio is enqueued on a vdo bio queue so that bio submission (which may block) does not block
|
||||
* other vdo threads.
|
||||
*
|
||||
* That the error handler will run on the correct thread is only true so long as the thread calling
|
||||
* this function, and the thread set in the endio callback are the same, as well as the fact that
|
||||
* no error can occur on the bio queue. Currently this is true for all callers, but additional care
|
||||
* will be needed if this ever changes.
|
||||
*/
|
||||
void __submit_metadata_vio(struct vio *vio, physical_block_number_t physical,
|
||||
bio_end_io_t callback, vdo_action_fn error_handler,
|
||||
blk_opf_t operation, char *data)
|
||||
{
|
||||
int result;
|
||||
struct vdo_completion *completion = &vio->completion;
|
||||
const struct admin_state_code *code = vdo_get_admin_state(completion->vdo);
|
||||
|
||||
|
||||
VDO_ASSERT_LOG_ONLY(!code->quiescent, "I/O not allowed in state %s", code->name);
|
||||
VDO_ASSERT_LOG_ONLY(vio->bio->bi_next == NULL, "metadata bio has no next bio");
|
||||
|
||||
vdo_reset_completion(completion);
|
||||
completion->error_handler = error_handler;
|
||||
result = vio_reset_bio(vio, data, callback, operation | REQ_META, physical);
|
||||
if (result != VDO_SUCCESS) {
|
||||
continue_vio(vio, result);
|
||||
return;
|
||||
}
|
||||
|
||||
vdo_set_completion_callback(completion, vdo_submit_vio,
|
||||
get_vio_bio_zone_thread_id(vio));
|
||||
vdo_launch_completion_with_priority(completion, get_metadata_priority(vio));
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_make_io_submitter() - Create an io_submitter structure.
|
||||
* @thread_count: Number of bio-submission threads to set up.
|
||||
* @rotation_interval: Interval to use when rotating between bio-submission threads when enqueuing
|
||||
* completions.
|
||||
* @max_requests_active: Number of bios for merge tracking.
|
||||
* @vdo: The vdo which will use this submitter.
|
||||
* @io_submitter: pointer to the new data structure.
|
||||
*
|
||||
* Return: VDO_SUCCESS or an error.
|
||||
*/
|
||||
int vdo_make_io_submitter(unsigned int thread_count, unsigned int rotation_interval,
|
||||
unsigned int max_requests_active, struct vdo *vdo,
|
||||
struct io_submitter **io_submitter_ptr)
|
||||
{
|
||||
unsigned int i;
|
||||
struct io_submitter *io_submitter;
|
||||
int result;
|
||||
|
||||
result = vdo_allocate_extended(struct io_submitter, thread_count,
|
||||
struct bio_queue_data, "bio submission data",
|
||||
&io_submitter);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
io_submitter->bio_queue_rotation_interval = rotation_interval;
|
||||
|
||||
/* Setup for each bio-submission work queue */
|
||||
for (i = 0; i < thread_count; i++) {
|
||||
struct bio_queue_data *bio_queue_data = &io_submitter->bio_queue_data[i];
|
||||
|
||||
mutex_init(&bio_queue_data->lock);
|
||||
/*
|
||||
* One I/O operation per request, but both first & last sector numbers.
|
||||
*
|
||||
* If requests are assigned to threads round-robin, they should be distributed
|
||||
* quite evenly. But if they're assigned based on PBN, things can sometimes be very
|
||||
* uneven. So for now, we'll assume that all requests *may* wind up on one thread,
|
||||
* and thus all in the same map.
|
||||
*/
|
||||
result = vdo_int_map_create(max_requests_active * 2,
|
||||
&bio_queue_data->map);
|
||||
if (result != VDO_SUCCESS) {
|
||||
/*
|
||||
* Clean up the partially initialized bio-queue entirely and indicate that
|
||||
* initialization failed.
|
||||
*/
|
||||
vdo_log_error("bio map initialization failed %d", result);
|
||||
vdo_cleanup_io_submitter(io_submitter);
|
||||
vdo_free_io_submitter(io_submitter);
|
||||
return result;
|
||||
}
|
||||
|
||||
bio_queue_data->queue_number = i;
|
||||
result = vdo_make_thread(vdo, vdo->thread_config.bio_threads[i],
|
||||
&bio_queue_type, 1, (void **) &bio_queue_data);
|
||||
if (result != VDO_SUCCESS) {
|
||||
/*
|
||||
* Clean up the partially initialized bio-queue entirely and indicate that
|
||||
* initialization failed.
|
||||
*/
|
||||
vdo_int_map_free(vdo_forget(bio_queue_data->map));
|
||||
vdo_log_error("bio queue initialization failed %d", result);
|
||||
vdo_cleanup_io_submitter(io_submitter);
|
||||
vdo_free_io_submitter(io_submitter);
|
||||
return result;
|
||||
}
|
||||
|
||||
bio_queue_data->queue = vdo->threads[vdo->thread_config.bio_threads[i]].queue;
|
||||
io_submitter->num_bio_queues_used++;
|
||||
}
|
||||
|
||||
*io_submitter_ptr = io_submitter;
|
||||
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_cleanup_io_submitter() - Tear down the io_submitter fields as needed for a physical layer.
|
||||
* @io_submitter: The I/O submitter data to tear down (may be NULL).
|
||||
*/
|
||||
void vdo_cleanup_io_submitter(struct io_submitter *io_submitter)
|
||||
{
|
||||
int i;
|
||||
|
||||
if (io_submitter == NULL)
|
||||
return;
|
||||
|
||||
for (i = io_submitter->num_bio_queues_used - 1; i >= 0; i--)
|
||||
vdo_finish_work_queue(io_submitter->bio_queue_data[i].queue);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_free_io_submitter() - Free the io_submitter fields and structure as needed.
|
||||
* @io_submitter: The I/O submitter data to destroy.
|
||||
*
|
||||
* This must be called after vdo_cleanup_io_submitter(). It is used to release resources late in
|
||||
* the shutdown process to avoid or reduce the chance of race conditions.
|
||||
*/
|
||||
void vdo_free_io_submitter(struct io_submitter *io_submitter)
|
||||
{
|
||||
int i;
|
||||
|
||||
if (io_submitter == NULL)
|
||||
return;
|
||||
|
||||
for (i = io_submitter->num_bio_queues_used - 1; i >= 0; i--) {
|
||||
io_submitter->num_bio_queues_used--;
|
||||
/* vdo_destroy() will free the work queue, so just give up our reference to it. */
|
||||
vdo_forget(io_submitter->bio_queue_data[i].queue);
|
||||
vdo_int_map_free(vdo_forget(io_submitter->bio_queue_data[i].map));
|
||||
}
|
||||
vdo_free(io_submitter);
|
||||
}
|
|
@ -0,0 +1,47 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef VDO_IO_SUBMITTER_H
|
||||
#define VDO_IO_SUBMITTER_H
|
||||
|
||||
#include <linux/bio.h>
|
||||
|
||||
#include "types.h"
|
||||
|
||||
struct io_submitter;
|
||||
|
||||
int vdo_make_io_submitter(unsigned int thread_count, unsigned int rotation_interval,
|
||||
unsigned int max_requests_active, struct vdo *vdo,
|
||||
struct io_submitter **io_submitter);
|
||||
|
||||
void vdo_cleanup_io_submitter(struct io_submitter *io_submitter);
|
||||
|
||||
void vdo_free_io_submitter(struct io_submitter *io_submitter);
|
||||
|
||||
void vdo_submit_vio(struct vdo_completion *completion);
|
||||
|
||||
void vdo_submit_data_vio(struct data_vio *data_vio);
|
||||
|
||||
void __submit_metadata_vio(struct vio *vio, physical_block_number_t physical,
|
||||
bio_end_io_t callback, vdo_action_fn error_handler,
|
||||
blk_opf_t operation, char *data);
|
||||
|
||||
static inline void vdo_submit_metadata_vio(struct vio *vio, physical_block_number_t physical,
|
||||
bio_end_io_t callback, vdo_action_fn error_handler,
|
||||
blk_opf_t operation)
|
||||
{
|
||||
__submit_metadata_vio(vio, physical, callback, error_handler,
|
||||
operation, vio->data);
|
||||
}
|
||||
|
||||
static inline void vdo_submit_flush_vio(struct vio *vio, bio_end_io_t callback,
|
||||
vdo_action_fn error_handler)
|
||||
{
|
||||
/* FIXME: Can we just use REQ_OP_FLUSH? */
|
||||
__submit_metadata_vio(vio, 0, callback, error_handler,
|
||||
REQ_OP_WRITE | REQ_PREFLUSH, NULL);
|
||||
}
|
||||
|
||||
#endif /* VDO_IO_SUBMITTER_H */
|
|
@ -0,0 +1,239 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "logger.h"
|
||||
|
||||
#include <asm/current.h>
|
||||
#include <linux/delay.h>
|
||||
#include <linux/hardirq.h>
|
||||
#include <linux/module.h>
|
||||
#include <linux/printk.h>
|
||||
#include <linux/sched.h>
|
||||
|
||||
#include "errors.h"
|
||||
#include "thread-device.h"
|
||||
#include "thread-utils.h"
|
||||
|
||||
int vdo_log_level = VDO_LOG_DEFAULT;
|
||||
|
||||
int vdo_get_log_level(void)
|
||||
{
|
||||
int log_level_latch = READ_ONCE(vdo_log_level);
|
||||
|
||||
if (unlikely(log_level_latch > VDO_LOG_MAX)) {
|
||||
log_level_latch = VDO_LOG_DEFAULT;
|
||||
WRITE_ONCE(vdo_log_level, log_level_latch);
|
||||
}
|
||||
return log_level_latch;
|
||||
}
|
||||
|
||||
static const char *get_current_interrupt_type(void)
|
||||
{
|
||||
if (in_nmi())
|
||||
return "NMI";
|
||||
|
||||
if (in_irq())
|
||||
return "HI";
|
||||
|
||||
if (in_softirq())
|
||||
return "SI";
|
||||
|
||||
return "INTR";
|
||||
}
|
||||
|
||||
/**
|
||||
* emit_log_message_to_kernel() - Emit a log message to the kernel at the specified priority.
|
||||
*
|
||||
* @priority: The priority at which to log the message
|
||||
* @fmt: The format string of the message
|
||||
*/
|
||||
static void emit_log_message_to_kernel(int priority, const char *fmt, ...)
|
||||
{
|
||||
va_list args;
|
||||
struct va_format vaf;
|
||||
|
||||
if (priority > vdo_get_log_level())
|
||||
return;
|
||||
|
||||
va_start(args, fmt);
|
||||
vaf.fmt = fmt;
|
||||
vaf.va = &args;
|
||||
|
||||
switch (priority) {
|
||||
case VDO_LOG_EMERG:
|
||||
case VDO_LOG_ALERT:
|
||||
case VDO_LOG_CRIT:
|
||||
pr_crit("%pV", &vaf);
|
||||
break;
|
||||
case VDO_LOG_ERR:
|
||||
pr_err("%pV", &vaf);
|
||||
break;
|
||||
case VDO_LOG_WARNING:
|
||||
pr_warn("%pV", &vaf);
|
||||
break;
|
||||
case VDO_LOG_NOTICE:
|
||||
case VDO_LOG_INFO:
|
||||
pr_info("%pV", &vaf);
|
||||
break;
|
||||
case VDO_LOG_DEBUG:
|
||||
pr_debug("%pV", &vaf);
|
||||
break;
|
||||
default:
|
||||
printk(KERN_DEFAULT "%pV", &vaf);
|
||||
break;
|
||||
}
|
||||
|
||||
va_end(args);
|
||||
}
|
||||
|
||||
/**
|
||||
* emit_log_message() - Emit a log message to the kernel log in a format suited to the current
|
||||
* thread context.
|
||||
*
|
||||
* Context info formats:
|
||||
*
|
||||
* interrupt: uds[NMI]: blah
|
||||
* kvdo thread: kvdo12:foobarQ: blah
|
||||
* thread w/device id: kvdo12:myprog: blah
|
||||
* other thread: uds: myprog: blah
|
||||
*
|
||||
* Fields: module name, interrupt level, process name, device ID.
|
||||
*
|
||||
* @priority: the priority at which to log the message
|
||||
* @module: The name of the module doing the logging
|
||||
* @prefix: The prefix of the log message
|
||||
* @vaf1: The first message format descriptor
|
||||
* @vaf2: The second message format descriptor
|
||||
*/
|
||||
static void emit_log_message(int priority, const char *module, const char *prefix,
|
||||
const struct va_format *vaf1, const struct va_format *vaf2)
|
||||
{
|
||||
int device_instance;
|
||||
|
||||
/*
|
||||
* In interrupt context, identify the interrupt type and module. Ignore the process/thread
|
||||
* since it could be anything.
|
||||
*/
|
||||
if (in_interrupt()) {
|
||||
const char *type = get_current_interrupt_type();
|
||||
|
||||
emit_log_message_to_kernel(priority, "%s[%s]: %s%pV%pV\n", module, type,
|
||||
prefix, vaf1, vaf2);
|
||||
return;
|
||||
}
|
||||
|
||||
/* Not at interrupt level; we have a process we can look at, and might have a device ID. */
|
||||
device_instance = vdo_get_thread_device_id();
|
||||
if (device_instance >= 0) {
|
||||
emit_log_message_to_kernel(priority, "%s%u:%s: %s%pV%pV\n", module,
|
||||
device_instance, current->comm, prefix, vaf1,
|
||||
vaf2);
|
||||
return;
|
||||
}
|
||||
|
||||
/*
|
||||
* If it's a kernel thread and the module name is a prefix of its name, assume it is ours
|
||||
* and only identify the thread.
|
||||
*/
|
||||
if (((current->flags & PF_KTHREAD) != 0) &&
|
||||
(strncmp(module, current->comm, strlen(module)) == 0)) {
|
||||
emit_log_message_to_kernel(priority, "%s: %s%pV%pV\n", current->comm,
|
||||
prefix, vaf1, vaf2);
|
||||
return;
|
||||
}
|
||||
|
||||
/* Identify the module and the process. */
|
||||
emit_log_message_to_kernel(priority, "%s: %s: %s%pV%pV\n", module, current->comm,
|
||||
prefix, vaf1, vaf2);
|
||||
}
|
||||
|
||||
/*
|
||||
* vdo_log_embedded_message() - Log a message embedded within another message.
|
||||
* @priority: the priority at which to log the message
|
||||
* @module: the name of the module doing the logging
|
||||
* @prefix: optional string prefix to message, may be NULL
|
||||
* @fmt1: format of message first part (required)
|
||||
* @args1: arguments for message first part (required)
|
||||
* @fmt2: format of message second part
|
||||
*/
|
||||
void vdo_log_embedded_message(int priority, const char *module, const char *prefix,
|
||||
const char *fmt1, va_list args1, const char *fmt2, ...)
|
||||
{
|
||||
va_list args1_copy;
|
||||
va_list args2;
|
||||
struct va_format vaf1, vaf2;
|
||||
|
||||
va_start(args2, fmt2);
|
||||
|
||||
if (module == NULL)
|
||||
module = VDO_LOGGING_MODULE_NAME;
|
||||
|
||||
if (prefix == NULL)
|
||||
prefix = "";
|
||||
|
||||
/*
|
||||
* It is implementation dependent whether va_list is defined as an array type that decays
|
||||
* to a pointer when passed as an argument. Copy args1 and args2 with va_copy so that vaf1
|
||||
* and vaf2 get proper va_list pointers irrespective of how va_list is defined.
|
||||
*/
|
||||
va_copy(args1_copy, args1);
|
||||
vaf1.fmt = fmt1;
|
||||
vaf1.va = &args1_copy;
|
||||
|
||||
vaf2.fmt = fmt2;
|
||||
vaf2.va = &args2;
|
||||
|
||||
emit_log_message(priority, module, prefix, &vaf1, &vaf2);
|
||||
|
||||
va_end(args1_copy);
|
||||
va_end(args2);
|
||||
}
|
||||
|
||||
int vdo_vlog_strerror(int priority, int errnum, const char *module, const char *format,
|
||||
va_list args)
|
||||
{
|
||||
char errbuf[VDO_MAX_ERROR_MESSAGE_SIZE];
|
||||
const char *message = uds_string_error(errnum, errbuf, sizeof(errbuf));
|
||||
|
||||
vdo_log_embedded_message(priority, module, NULL, format, args, ": %s (%d)",
|
||||
message, errnum);
|
||||
return errnum;
|
||||
}
|
||||
|
||||
int __vdo_log_strerror(int priority, int errnum, const char *module, const char *format, ...)
|
||||
{
|
||||
va_list args;
|
||||
|
||||
va_start(args, format);
|
||||
vdo_vlog_strerror(priority, errnum, module, format, args);
|
||||
va_end(args);
|
||||
return errnum;
|
||||
}
|
||||
|
||||
void vdo_log_backtrace(int priority)
|
||||
{
|
||||
if (priority > vdo_get_log_level())
|
||||
return;
|
||||
|
||||
dump_stack();
|
||||
}
|
||||
|
||||
void __vdo_log_message(int priority, const char *module, const char *format, ...)
|
||||
{
|
||||
va_list args;
|
||||
|
||||
va_start(args, format);
|
||||
vdo_log_embedded_message(priority, module, NULL, format, args, "%s", "");
|
||||
va_end(args);
|
||||
}
|
||||
|
||||
/*
|
||||
* Sleep or delay a few milliseconds in an attempt to allow the log buffers to be flushed lest they
|
||||
* be overrun.
|
||||
*/
|
||||
void vdo_pause_for_logger(void)
|
||||
{
|
||||
fsleep(4000);
|
||||
}
|
|
@ -0,0 +1,100 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef VDO_LOGGER_H
|
||||
#define VDO_LOGGER_H
|
||||
|
||||
#include <linux/kern_levels.h>
|
||||
#include <linux/module.h>
|
||||
#include <linux/ratelimit.h>
|
||||
#include <linux/device-mapper.h>
|
||||
|
||||
/* Custom logging utilities for UDS */
|
||||
|
||||
enum {
|
||||
VDO_LOG_EMERG = LOGLEVEL_EMERG,
|
||||
VDO_LOG_ALERT = LOGLEVEL_ALERT,
|
||||
VDO_LOG_CRIT = LOGLEVEL_CRIT,
|
||||
VDO_LOG_ERR = LOGLEVEL_ERR,
|
||||
VDO_LOG_WARNING = LOGLEVEL_WARNING,
|
||||
VDO_LOG_NOTICE = LOGLEVEL_NOTICE,
|
||||
VDO_LOG_INFO = LOGLEVEL_INFO,
|
||||
VDO_LOG_DEBUG = LOGLEVEL_DEBUG,
|
||||
|
||||
VDO_LOG_MAX = VDO_LOG_DEBUG,
|
||||
VDO_LOG_DEFAULT = VDO_LOG_INFO,
|
||||
};
|
||||
|
||||
extern int vdo_log_level;
|
||||
|
||||
#define DM_MSG_PREFIX "vdo"
|
||||
#define VDO_LOGGING_MODULE_NAME DM_NAME ": " DM_MSG_PREFIX
|
||||
|
||||
/* Apply a rate limiter to a log method call. */
|
||||
#define vdo_log_ratelimit(log_fn, ...) \
|
||||
do { \
|
||||
static DEFINE_RATELIMIT_STATE(_rs, \
|
||||
DEFAULT_RATELIMIT_INTERVAL, \
|
||||
DEFAULT_RATELIMIT_BURST); \
|
||||
if (__ratelimit(&_rs)) { \
|
||||
log_fn(__VA_ARGS__); \
|
||||
} \
|
||||
} while (0)
|
||||
|
||||
int vdo_get_log_level(void);
|
||||
|
||||
void vdo_log_embedded_message(int priority, const char *module, const char *prefix,
|
||||
const char *fmt1, va_list args1, const char *fmt2, ...)
|
||||
__printf(4, 0) __printf(6, 7);
|
||||
|
||||
void vdo_log_backtrace(int priority);
|
||||
|
||||
/* All log functions will preserve the caller's value of errno. */
|
||||
|
||||
#define vdo_log_strerror(priority, errnum, ...) \
|
||||
__vdo_log_strerror(priority, errnum, VDO_LOGGING_MODULE_NAME, __VA_ARGS__)
|
||||
|
||||
int __vdo_log_strerror(int priority, int errnum, const char *module,
|
||||
const char *format, ...)
|
||||
__printf(4, 5);
|
||||
|
||||
int vdo_vlog_strerror(int priority, int errnum, const char *module, const char *format,
|
||||
va_list args)
|
||||
__printf(4, 0);
|
||||
|
||||
/* Log an error prefixed with the string associated with the errnum. */
|
||||
#define vdo_log_error_strerror(errnum, ...) \
|
||||
vdo_log_strerror(VDO_LOG_ERR, errnum, __VA_ARGS__)
|
||||
|
||||
#define vdo_log_debug_strerror(errnum, ...) \
|
||||
vdo_log_strerror(VDO_LOG_DEBUG, errnum, __VA_ARGS__)
|
||||
|
||||
#define vdo_log_info_strerror(errnum, ...) \
|
||||
vdo_log_strerror(VDO_LOG_INFO, errnum, __VA_ARGS__)
|
||||
|
||||
#define vdo_log_warning_strerror(errnum, ...) \
|
||||
vdo_log_strerror(VDO_LOG_WARNING, errnum, __VA_ARGS__)
|
||||
|
||||
#define vdo_log_fatal_strerror(errnum, ...) \
|
||||
vdo_log_strerror(VDO_LOG_CRIT, errnum, __VA_ARGS__)
|
||||
|
||||
#define vdo_log_message(priority, ...) \
|
||||
__vdo_log_message(priority, VDO_LOGGING_MODULE_NAME, __VA_ARGS__)
|
||||
|
||||
void __vdo_log_message(int priority, const char *module, const char *format, ...)
|
||||
__printf(3, 4);
|
||||
|
||||
#define vdo_log_debug(...) vdo_log_message(VDO_LOG_DEBUG, __VA_ARGS__)
|
||||
|
||||
#define vdo_log_info(...) vdo_log_message(VDO_LOG_INFO, __VA_ARGS__)
|
||||
|
||||
#define vdo_log_warning(...) vdo_log_message(VDO_LOG_WARNING, __VA_ARGS__)
|
||||
|
||||
#define vdo_log_error(...) vdo_log_message(VDO_LOG_ERR, __VA_ARGS__)
|
||||
|
||||
#define vdo_log_fatal(...) vdo_log_message(VDO_LOG_CRIT, __VA_ARGS__)
|
||||
|
||||
void vdo_pause_for_logger(void);
|
||||
#endif /* VDO_LOGGER_H */
|
|
@ -0,0 +1,373 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "logical-zone.h"
|
||||
|
||||
#include "logger.h"
|
||||
#include "memory-alloc.h"
|
||||
#include "permassert.h"
|
||||
#include "string-utils.h"
|
||||
|
||||
#include "action-manager.h"
|
||||
#include "admin-state.h"
|
||||
#include "block-map.h"
|
||||
#include "completion.h"
|
||||
#include "constants.h"
|
||||
#include "data-vio.h"
|
||||
#include "flush.h"
|
||||
#include "int-map.h"
|
||||
#include "physical-zone.h"
|
||||
#include "vdo.h"
|
||||
|
||||
#define ALLOCATIONS_PER_ZONE 128
|
||||
|
||||
/**
|
||||
* as_logical_zone() - Convert a generic vdo_completion to a logical_zone.
|
||||
* @completion: The completion to convert.
|
||||
*
|
||||
* Return: The completion as a logical_zone.
|
||||
*/
|
||||
static struct logical_zone *as_logical_zone(struct vdo_completion *completion)
|
||||
{
|
||||
vdo_assert_completion_type(completion, VDO_GENERATION_FLUSHED_COMPLETION);
|
||||
return container_of(completion, struct logical_zone, completion);
|
||||
}
|
||||
|
||||
/* get_thread_id_for_zone() - Implements vdo_zone_thread_getter_fn. */
|
||||
static thread_id_t get_thread_id_for_zone(void *context, zone_count_t zone_number)
|
||||
{
|
||||
struct logical_zones *zones = context;
|
||||
|
||||
return zones->zones[zone_number].thread_id;
|
||||
}
|
||||
|
||||
/**
|
||||
* initialize_zone() - Initialize a logical zone.
|
||||
* @zones: The logical_zones to which this zone belongs.
|
||||
* @zone_number: The logical_zone's index.
|
||||
*/
|
||||
static int initialize_zone(struct logical_zones *zones, zone_count_t zone_number)
|
||||
{
|
||||
int result;
|
||||
struct vdo *vdo = zones->vdo;
|
||||
struct logical_zone *zone = &zones->zones[zone_number];
|
||||
zone_count_t allocation_zone_number;
|
||||
|
||||
result = vdo_int_map_create(VDO_LOCK_MAP_CAPACITY, &zone->lbn_operations);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
if (zone_number < vdo->thread_config.logical_zone_count - 1)
|
||||
zone->next = &zones->zones[zone_number + 1];
|
||||
|
||||
vdo_initialize_completion(&zone->completion, vdo,
|
||||
VDO_GENERATION_FLUSHED_COMPLETION);
|
||||
zone->zones = zones;
|
||||
zone->zone_number = zone_number;
|
||||
zone->thread_id = vdo->thread_config.logical_threads[zone_number];
|
||||
zone->block_map_zone = &vdo->block_map->zones[zone_number];
|
||||
INIT_LIST_HEAD(&zone->write_vios);
|
||||
vdo_set_admin_state_code(&zone->state, VDO_ADMIN_STATE_NORMAL_OPERATION);
|
||||
|
||||
allocation_zone_number = zone->thread_id % vdo->thread_config.physical_zone_count;
|
||||
zone->allocation_zone = &vdo->physical_zones->zones[allocation_zone_number];
|
||||
|
||||
return vdo_make_default_thread(vdo, zone->thread_id);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_make_logical_zones() - Create a set of logical zones.
|
||||
* @vdo: The vdo to which the zones will belong.
|
||||
* @zones_ptr: A pointer to hold the new zones.
|
||||
*
|
||||
* Return: VDO_SUCCESS or an error code.
|
||||
*/
|
||||
int vdo_make_logical_zones(struct vdo *vdo, struct logical_zones **zones_ptr)
|
||||
{
|
||||
struct logical_zones *zones;
|
||||
int result;
|
||||
zone_count_t zone;
|
||||
zone_count_t zone_count = vdo->thread_config.logical_zone_count;
|
||||
|
||||
if (zone_count == 0)
|
||||
return VDO_SUCCESS;
|
||||
|
||||
result = vdo_allocate_extended(struct logical_zones, zone_count,
|
||||
struct logical_zone, __func__, &zones);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
zones->vdo = vdo;
|
||||
zones->zone_count = zone_count;
|
||||
for (zone = 0; zone < zone_count; zone++) {
|
||||
result = initialize_zone(zones, zone);
|
||||
if (result != VDO_SUCCESS) {
|
||||
vdo_free_logical_zones(zones);
|
||||
return result;
|
||||
}
|
||||
}
|
||||
|
||||
result = vdo_make_action_manager(zones->zone_count, get_thread_id_for_zone,
|
||||
vdo->thread_config.admin_thread, zones, NULL,
|
||||
vdo, &zones->manager);
|
||||
if (result != VDO_SUCCESS) {
|
||||
vdo_free_logical_zones(zones);
|
||||
return result;
|
||||
}
|
||||
|
||||
*zones_ptr = zones;
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_free_logical_zones() - Free a set of logical zones.
|
||||
* @zones: The set of zones to free.
|
||||
*/
|
||||
void vdo_free_logical_zones(struct logical_zones *zones)
|
||||
{
|
||||
zone_count_t index;
|
||||
|
||||
if (zones == NULL)
|
||||
return;
|
||||
|
||||
vdo_free(vdo_forget(zones->manager));
|
||||
|
||||
for (index = 0; index < zones->zone_count; index++)
|
||||
vdo_int_map_free(vdo_forget(zones->zones[index].lbn_operations));
|
||||
|
||||
vdo_free(zones);
|
||||
}
|
||||
|
||||
static inline void assert_on_zone_thread(struct logical_zone *zone, const char *what)
|
||||
{
|
||||
VDO_ASSERT_LOG_ONLY((vdo_get_callback_thread_id() == zone->thread_id),
|
||||
"%s() called on correct thread", what);
|
||||
}
|
||||
|
||||
/**
|
||||
* check_for_drain_complete() - Check whether this zone has drained.
|
||||
* @zone: The zone to check.
|
||||
*/
|
||||
static void check_for_drain_complete(struct logical_zone *zone)
|
||||
{
|
||||
if (!vdo_is_state_draining(&zone->state) || zone->notifying ||
|
||||
!list_empty(&zone->write_vios))
|
||||
return;
|
||||
|
||||
vdo_finish_draining(&zone->state);
|
||||
}
|
||||
|
||||
/**
|
||||
* initiate_drain() - Initiate a drain.
|
||||
*
|
||||
* Implements vdo_admin_initiator_fn.
|
||||
*/
|
||||
static void initiate_drain(struct admin_state *state)
|
||||
{
|
||||
check_for_drain_complete(container_of(state, struct logical_zone, state));
|
||||
}
|
||||
|
||||
/**
|
||||
* drain_logical_zone() - Drain a logical zone.
|
||||
*
|
||||
* Implements vdo_zone_action_fn.
|
||||
*/
|
||||
static void drain_logical_zone(void *context, zone_count_t zone_number,
|
||||
struct vdo_completion *parent)
|
||||
{
|
||||
struct logical_zones *zones = context;
|
||||
|
||||
vdo_start_draining(&zones->zones[zone_number].state,
|
||||
vdo_get_current_manager_operation(zones->manager), parent,
|
||||
initiate_drain);
|
||||
}
|
||||
|
||||
void vdo_drain_logical_zones(struct logical_zones *zones,
|
||||
const struct admin_state_code *operation,
|
||||
struct vdo_completion *parent)
|
||||
{
|
||||
vdo_schedule_operation(zones->manager, operation, NULL, drain_logical_zone, NULL,
|
||||
parent);
|
||||
}
|
||||
|
||||
/**
|
||||
* resume_logical_zone() - Resume a logical zone.
|
||||
*
|
||||
* Implements vdo_zone_action_fn.
|
||||
*/
|
||||
static void resume_logical_zone(void *context, zone_count_t zone_number,
|
||||
struct vdo_completion *parent)
|
||||
{
|
||||
struct logical_zone *zone = &(((struct logical_zones *) context)->zones[zone_number]);
|
||||
|
||||
vdo_fail_completion(parent, vdo_resume_if_quiescent(&zone->state));
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_resume_logical_zones() - Resume a set of logical zones.
|
||||
* @zones: The logical zones to resume.
|
||||
* @parent: The object to notify when the zones have resumed.
|
||||
*/
|
||||
void vdo_resume_logical_zones(struct logical_zones *zones, struct vdo_completion *parent)
|
||||
{
|
||||
vdo_schedule_operation(zones->manager, VDO_ADMIN_STATE_RESUMING, NULL,
|
||||
resume_logical_zone, NULL, parent);
|
||||
}
|
||||
|
||||
/**
|
||||
* update_oldest_active_generation() - Update the oldest active generation.
|
||||
* @zone: The zone.
|
||||
*
|
||||
* Return: true if the oldest active generation has changed.
|
||||
*/
|
||||
static bool update_oldest_active_generation(struct logical_zone *zone)
|
||||
{
|
||||
struct data_vio *data_vio =
|
||||
list_first_entry_or_null(&zone->write_vios, struct data_vio,
|
||||
write_entry);
|
||||
sequence_number_t oldest =
|
||||
(data_vio == NULL) ? zone->flush_generation : data_vio->flush_generation;
|
||||
|
||||
if (oldest == zone->oldest_active_generation)
|
||||
return false;
|
||||
|
||||
WRITE_ONCE(zone->oldest_active_generation, oldest);
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_increment_logical_zone_flush_generation() - Increment the flush generation in a logical
|
||||
* zone.
|
||||
* @zone: The logical zone.
|
||||
* @expected_generation: The expected value of the flush generation before the increment.
|
||||
*/
|
||||
void vdo_increment_logical_zone_flush_generation(struct logical_zone *zone,
|
||||
sequence_number_t expected_generation)
|
||||
{
|
||||
assert_on_zone_thread(zone, __func__);
|
||||
VDO_ASSERT_LOG_ONLY((zone->flush_generation == expected_generation),
|
||||
"logical zone %u flush generation %llu should be %llu before increment",
|
||||
zone->zone_number, (unsigned long long) zone->flush_generation,
|
||||
(unsigned long long) expected_generation);
|
||||
|
||||
zone->flush_generation++;
|
||||
zone->ios_in_flush_generation = 0;
|
||||
update_oldest_active_generation(zone);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_acquire_flush_generation_lock() - Acquire the shared lock on a flush generation by a write
|
||||
* data_vio.
|
||||
* @data_vio: The data_vio.
|
||||
*/
|
||||
void vdo_acquire_flush_generation_lock(struct data_vio *data_vio)
|
||||
{
|
||||
struct logical_zone *zone = data_vio->logical.zone;
|
||||
|
||||
assert_on_zone_thread(zone, __func__);
|
||||
VDO_ASSERT_LOG_ONLY(vdo_is_state_normal(&zone->state), "vdo state is normal");
|
||||
|
||||
data_vio->flush_generation = zone->flush_generation;
|
||||
list_add_tail(&data_vio->write_entry, &zone->write_vios);
|
||||
zone->ios_in_flush_generation++;
|
||||
}
|
||||
|
||||
static void attempt_generation_complete_notification(struct vdo_completion *completion);
|
||||
|
||||
/**
|
||||
* notify_flusher() - Notify the flush that at least one generation no longer has active VIOs.
|
||||
* @completion: The zone completion.
|
||||
*
|
||||
* This callback is registered in attempt_generation_complete_notification().
|
||||
*/
|
||||
static void notify_flusher(struct vdo_completion *completion)
|
||||
{
|
||||
struct logical_zone *zone = as_logical_zone(completion);
|
||||
|
||||
vdo_complete_flushes(zone->zones->vdo->flusher);
|
||||
vdo_launch_completion_callback(completion,
|
||||
attempt_generation_complete_notification,
|
||||
zone->thread_id);
|
||||
}
|
||||
|
||||
/**
|
||||
* attempt_generation_complete_notification() - Notify the flusher if some generation no
|
||||
* longer has active VIOs.
|
||||
* @completion: The zone completion.
|
||||
*/
|
||||
static void attempt_generation_complete_notification(struct vdo_completion *completion)
|
||||
{
|
||||
struct logical_zone *zone = as_logical_zone(completion);
|
||||
|
||||
assert_on_zone_thread(zone, __func__);
|
||||
if (zone->oldest_active_generation <= zone->notification_generation) {
|
||||
zone->notifying = false;
|
||||
check_for_drain_complete(zone);
|
||||
return;
|
||||
}
|
||||
|
||||
zone->notifying = true;
|
||||
zone->notification_generation = zone->oldest_active_generation;
|
||||
vdo_launch_completion_callback(&zone->completion, notify_flusher,
|
||||
vdo_get_flusher_thread_id(zone->zones->vdo->flusher));
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_release_flush_generation_lock() - Release the shared lock on a flush generation held by a
|
||||
* write data_vio.
|
||||
* @data_vio: The data_vio whose lock is to be released.
|
||||
*
|
||||
* If there are pending flushes, and this data_vio completes the oldest generation active in this
|
||||
* zone, an attempt will be made to finish any flushes which may now be complete.
|
||||
*/
|
||||
void vdo_release_flush_generation_lock(struct data_vio *data_vio)
|
||||
{
|
||||
struct logical_zone *zone = data_vio->logical.zone;
|
||||
|
||||
assert_on_zone_thread(zone, __func__);
|
||||
|
||||
if (!data_vio_has_flush_generation_lock(data_vio))
|
||||
return;
|
||||
|
||||
list_del_init(&data_vio->write_entry);
|
||||
VDO_ASSERT_LOG_ONLY((zone->oldest_active_generation <= data_vio->flush_generation),
|
||||
"data_vio releasing lock on generation %llu is not older than oldest active generation %llu",
|
||||
(unsigned long long) data_vio->flush_generation,
|
||||
(unsigned long long) zone->oldest_active_generation);
|
||||
|
||||
if (!update_oldest_active_generation(zone) || zone->notifying)
|
||||
return;
|
||||
|
||||
attempt_generation_complete_notification(&zone->completion);
|
||||
}
|
||||
|
||||
struct physical_zone *vdo_get_next_allocation_zone(struct logical_zone *zone)
|
||||
{
|
||||
if (zone->allocation_count == ALLOCATIONS_PER_ZONE) {
|
||||
zone->allocation_count = 0;
|
||||
zone->allocation_zone = zone->allocation_zone->next;
|
||||
}
|
||||
|
||||
zone->allocation_count++;
|
||||
return zone->allocation_zone;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_dump_logical_zone() - Dump information about a logical zone to the log for debugging.
|
||||
* @zone: The zone to dump
|
||||
*
|
||||
* Context: the information is dumped in a thread-unsafe fashion.
|
||||
*
|
||||
*/
|
||||
void vdo_dump_logical_zone(const struct logical_zone *zone)
|
||||
{
|
||||
vdo_log_info("logical_zone %u", zone->zone_number);
|
||||
vdo_log_info(" flush_generation=%llu oldest_active_generation=%llu notification_generation=%llu notifying=%s ios_in_flush_generation=%llu",
|
||||
(unsigned long long) READ_ONCE(zone->flush_generation),
|
||||
(unsigned long long) READ_ONCE(zone->oldest_active_generation),
|
||||
(unsigned long long) READ_ONCE(zone->notification_generation),
|
||||
vdo_bool_to_string(READ_ONCE(zone->notifying)),
|
||||
(unsigned long long) READ_ONCE(zone->ios_in_flush_generation));
|
||||
}
|
|
@ -0,0 +1,89 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef VDO_LOGICAL_ZONE_H
|
||||
#define VDO_LOGICAL_ZONE_H
|
||||
|
||||
#include <linux/list.h>
|
||||
|
||||
#include "admin-state.h"
|
||||
#include "int-map.h"
|
||||
#include "types.h"
|
||||
|
||||
struct physical_zone;
|
||||
|
||||
struct logical_zone {
|
||||
/* The completion for flush notifications */
|
||||
struct vdo_completion completion;
|
||||
/* The owner of this zone */
|
||||
struct logical_zones *zones;
|
||||
/* Which logical zone this is */
|
||||
zone_count_t zone_number;
|
||||
/* The thread id for this zone */
|
||||
thread_id_t thread_id;
|
||||
/* In progress operations keyed by LBN */
|
||||
struct int_map *lbn_operations;
|
||||
/* The logical to physical map */
|
||||
struct block_map_zone *block_map_zone;
|
||||
/* The current flush generation */
|
||||
sequence_number_t flush_generation;
|
||||
/*
|
||||
* The oldest active generation in this zone. This is mutated only on the logical zone
|
||||
* thread but is queried from the flusher thread.
|
||||
*/
|
||||
sequence_number_t oldest_active_generation;
|
||||
/* The number of IOs in the current flush generation */
|
||||
block_count_t ios_in_flush_generation;
|
||||
/* The youngest generation of the current notification */
|
||||
sequence_number_t notification_generation;
|
||||
/* Whether a notification is in progress */
|
||||
bool notifying;
|
||||
/* The queue of active data write VIOs */
|
||||
struct list_head write_vios;
|
||||
/* The administrative state of the zone */
|
||||
struct admin_state state;
|
||||
/* The physical zone from which to allocate */
|
||||
struct physical_zone *allocation_zone;
|
||||
/* The number of allocations done from the current allocation_zone */
|
||||
block_count_t allocation_count;
|
||||
/* The next zone */
|
||||
struct logical_zone *next;
|
||||
};
|
||||
|
||||
struct logical_zones {
|
||||
/* The vdo whose zones these are */
|
||||
struct vdo *vdo;
|
||||
/* The manager for administrative actions */
|
||||
struct action_manager *manager;
|
||||
/* The number of zones */
|
||||
zone_count_t zone_count;
|
||||
/* The logical zones themselves */
|
||||
struct logical_zone zones[];
|
||||
};
|
||||
|
||||
int __must_check vdo_make_logical_zones(struct vdo *vdo,
|
||||
struct logical_zones **zones_ptr);
|
||||
|
||||
void vdo_free_logical_zones(struct logical_zones *zones);
|
||||
|
||||
void vdo_drain_logical_zones(struct logical_zones *zones,
|
||||
const struct admin_state_code *operation,
|
||||
struct vdo_completion *completion);
|
||||
|
||||
void vdo_resume_logical_zones(struct logical_zones *zones,
|
||||
struct vdo_completion *parent);
|
||||
|
||||
void vdo_increment_logical_zone_flush_generation(struct logical_zone *zone,
|
||||
sequence_number_t expected_generation);
|
||||
|
||||
void vdo_acquire_flush_generation_lock(struct data_vio *data_vio);
|
||||
|
||||
void vdo_release_flush_generation_lock(struct data_vio *data_vio);
|
||||
|
||||
struct physical_zone * __must_check vdo_get_next_allocation_zone(struct logical_zone *zone);
|
||||
|
||||
void vdo_dump_logical_zone(const struct logical_zone *zone);
|
||||
|
||||
#endif /* VDO_LOGICAL_ZONE_H */
|
|
@ -0,0 +1,438 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include <linux/delay.h>
|
||||
#include <linux/mm.h>
|
||||
#include <linux/sched/mm.h>
|
||||
#include <linux/slab.h>
|
||||
#include <linux/vmalloc.h>
|
||||
|
||||
#include "logger.h"
|
||||
#include "memory-alloc.h"
|
||||
#include "permassert.h"
|
||||
|
||||
/*
|
||||
* UDS and VDO keep track of which threads are allowed to allocate memory freely, and which threads
|
||||
* must be careful to not do a memory allocation that does an I/O request. The 'allocating_threads'
|
||||
* thread_registry and its associated methods implement this tracking.
|
||||
*/
|
||||
static struct thread_registry allocating_threads;
|
||||
|
||||
static inline bool allocations_allowed(void)
|
||||
{
|
||||
return vdo_lookup_thread(&allocating_threads) != NULL;
|
||||
}
|
||||
|
||||
/*
|
||||
* Register the current thread as an allocating thread.
|
||||
*
|
||||
* An optional flag location can be supplied indicating whether, at any given point in time, the
|
||||
* threads associated with that flag should be allocating storage. If the flag is false, a message
|
||||
* will be logged.
|
||||
*
|
||||
* If no flag is supplied, the thread is always allowed to allocate storage without complaint.
|
||||
*
|
||||
* @new_thread: registered_thread structure to use for the current thread
|
||||
* @flag_ptr: Location of the allocation-allowed flag
|
||||
*/
|
||||
void vdo_register_allocating_thread(struct registered_thread *new_thread,
|
||||
const bool *flag_ptr)
|
||||
{
|
||||
if (flag_ptr == NULL) {
|
||||
static const bool allocation_always_allowed = true;
|
||||
|
||||
flag_ptr = &allocation_always_allowed;
|
||||
}
|
||||
|
||||
vdo_register_thread(&allocating_threads, new_thread, flag_ptr);
|
||||
}
|
||||
|
||||
/* Unregister the current thread as an allocating thread. */
|
||||
void vdo_unregister_allocating_thread(void)
|
||||
{
|
||||
vdo_unregister_thread(&allocating_threads);
|
||||
}
|
||||
|
||||
/*
|
||||
* We track how much memory has been allocated and freed. When we unload the module, we log an
|
||||
* error if we have not freed all the memory that we allocated. Nearly all memory allocation and
|
||||
* freeing is done using this module.
|
||||
*
|
||||
* We do not use kernel functions like the kvasprintf() method, which allocate memory indirectly
|
||||
* using kmalloc.
|
||||
*
|
||||
* These data structures and methods are used to track the amount of memory used.
|
||||
*/
|
||||
|
||||
/*
|
||||
* We allocate very few large objects, and allocation/deallocation isn't done in a
|
||||
* performance-critical stage for us, so a linked list should be fine.
|
||||
*/
|
||||
struct vmalloc_block_info {
|
||||
void *ptr;
|
||||
size_t size;
|
||||
struct vmalloc_block_info *next;
|
||||
};
|
||||
|
||||
static struct {
|
||||
spinlock_t lock;
|
||||
size_t kmalloc_blocks;
|
||||
size_t kmalloc_bytes;
|
||||
size_t vmalloc_blocks;
|
||||
size_t vmalloc_bytes;
|
||||
size_t peak_bytes;
|
||||
struct vmalloc_block_info *vmalloc_list;
|
||||
} memory_stats __cacheline_aligned;
|
||||
|
||||
static void update_peak_usage(void)
|
||||
{
|
||||
size_t total_bytes = memory_stats.kmalloc_bytes + memory_stats.vmalloc_bytes;
|
||||
|
||||
if (total_bytes > memory_stats.peak_bytes)
|
||||
memory_stats.peak_bytes = total_bytes;
|
||||
}
|
||||
|
||||
static void add_kmalloc_block(size_t size)
|
||||
{
|
||||
unsigned long flags;
|
||||
|
||||
spin_lock_irqsave(&memory_stats.lock, flags);
|
||||
memory_stats.kmalloc_blocks++;
|
||||
memory_stats.kmalloc_bytes += size;
|
||||
update_peak_usage();
|
||||
spin_unlock_irqrestore(&memory_stats.lock, flags);
|
||||
}
|
||||
|
||||
static void remove_kmalloc_block(size_t size)
|
||||
{
|
||||
unsigned long flags;
|
||||
|
||||
spin_lock_irqsave(&memory_stats.lock, flags);
|
||||
memory_stats.kmalloc_blocks--;
|
||||
memory_stats.kmalloc_bytes -= size;
|
||||
spin_unlock_irqrestore(&memory_stats.lock, flags);
|
||||
}
|
||||
|
||||
static void add_vmalloc_block(struct vmalloc_block_info *block)
|
||||
{
|
||||
unsigned long flags;
|
||||
|
||||
spin_lock_irqsave(&memory_stats.lock, flags);
|
||||
block->next = memory_stats.vmalloc_list;
|
||||
memory_stats.vmalloc_list = block;
|
||||
memory_stats.vmalloc_blocks++;
|
||||
memory_stats.vmalloc_bytes += block->size;
|
||||
update_peak_usage();
|
||||
spin_unlock_irqrestore(&memory_stats.lock, flags);
|
||||
}
|
||||
|
||||
static void remove_vmalloc_block(void *ptr)
|
||||
{
|
||||
struct vmalloc_block_info *block;
|
||||
struct vmalloc_block_info **block_ptr;
|
||||
unsigned long flags;
|
||||
|
||||
spin_lock_irqsave(&memory_stats.lock, flags);
|
||||
for (block_ptr = &memory_stats.vmalloc_list;
|
||||
(block = *block_ptr) != NULL;
|
||||
block_ptr = &block->next) {
|
||||
if (block->ptr == ptr) {
|
||||
*block_ptr = block->next;
|
||||
memory_stats.vmalloc_blocks--;
|
||||
memory_stats.vmalloc_bytes -= block->size;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
spin_unlock_irqrestore(&memory_stats.lock, flags);
|
||||
if (block != NULL)
|
||||
vdo_free(block);
|
||||
else
|
||||
vdo_log_info("attempting to remove ptr %px not found in vmalloc list", ptr);
|
||||
}
|
||||
|
||||
/*
|
||||
* Determine whether allocating a memory block should use kmalloc or __vmalloc.
|
||||
*
|
||||
* vmalloc can allocate any integral number of pages.
|
||||
*
|
||||
* kmalloc can allocate any number of bytes up to a configured limit, which defaults to 8 megabytes
|
||||
* on some systems. kmalloc is especially good when memory is being both allocated and freed, and
|
||||
* it does this efficiently in a multi CPU environment.
|
||||
*
|
||||
* kmalloc usually rounds the size of the block up to the next power of two, so when the requested
|
||||
* block is bigger than PAGE_SIZE / 2 bytes, kmalloc will never give you less space than the
|
||||
* corresponding vmalloc allocation. Sometimes vmalloc will use less overhead than kmalloc.
|
||||
*
|
||||
* The advantages of kmalloc do not help out UDS or VDO, because we allocate all our memory up
|
||||
* front and do not free and reallocate it. Sometimes we have problems using kmalloc, because the
|
||||
* Linux memory page map can become so fragmented that kmalloc will not give us a 32KB chunk. We
|
||||
* have used vmalloc as a backup to kmalloc in the past, and a follow-up vmalloc of 32KB will work.
|
||||
* But there is no strong case to be made for using kmalloc over vmalloc for these size chunks.
|
||||
*
|
||||
* The kmalloc/vmalloc boundary is set at 4KB, and kmalloc gets the 4KB requests. There is no
|
||||
* strong reason for favoring either kmalloc or vmalloc for 4KB requests, except that tracking
|
||||
* vmalloc statistics uses a linked list implementation. Using a simple test, this choice of
|
||||
* boundary results in 132 vmalloc calls. Using vmalloc for requests of exactly 4KB results in an
|
||||
* additional 6374 vmalloc calls, which is much less efficient for tracking.
|
||||
*
|
||||
* @size: How many bytes to allocate
|
||||
*/
|
||||
static inline bool use_kmalloc(size_t size)
|
||||
{
|
||||
return size <= PAGE_SIZE;
|
||||
}
|
||||
|
||||
/*
|
||||
* Allocate storage based on memory size and alignment, logging an error if the allocation fails.
|
||||
* The memory will be zeroed.
|
||||
*
|
||||
* @size: The size of an object
|
||||
* @align: The required alignment
|
||||
* @what: What is being allocated (for error logging)
|
||||
* @ptr: A pointer to hold the allocated memory
|
||||
*
|
||||
* Return: VDO_SUCCESS or an error code
|
||||
*/
|
||||
int vdo_allocate_memory(size_t size, size_t align, const char *what, void *ptr)
|
||||
{
|
||||
/*
|
||||
* The __GFP_RETRY_MAYFAIL flag means the VM implementation will retry memory reclaim
|
||||
* procedures that have previously failed if there is some indication that progress has
|
||||
* been made elsewhere. It can wait for other tasks to attempt high level approaches to
|
||||
* freeing memory such as compaction (which removes fragmentation) and page-out. There is
|
||||
* still a definite limit to the number of retries, but it is a larger limit than with
|
||||
* __GFP_NORETRY. Allocations with this flag may fail, but only when there is genuinely
|
||||
* little unused memory. While these allocations do not directly trigger the OOM killer,
|
||||
* their failure indicates that the system is likely to need to use the OOM killer soon.
|
||||
* The caller must handle failure, but can reasonably do so by failing a higher-level
|
||||
* request, or completing it only in a much less efficient manner.
|
||||
*/
|
||||
const gfp_t gfp_flags = GFP_KERNEL | __GFP_ZERO | __GFP_RETRY_MAYFAIL;
|
||||
unsigned int noio_flags;
|
||||
bool allocations_restricted = !allocations_allowed();
|
||||
unsigned long start_time;
|
||||
void *p = NULL;
|
||||
|
||||
if (unlikely(ptr == NULL))
|
||||
return -EINVAL;
|
||||
|
||||
if (size == 0) {
|
||||
*((void **) ptr) = NULL;
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
if (allocations_restricted)
|
||||
noio_flags = memalloc_noio_save();
|
||||
|
||||
start_time = jiffies;
|
||||
if (use_kmalloc(size) && (align < PAGE_SIZE)) {
|
||||
p = kmalloc(size, gfp_flags | __GFP_NOWARN);
|
||||
if (p == NULL) {
|
||||
/*
|
||||
* It is possible for kmalloc to fail to allocate memory because there is
|
||||
* no page available. A short sleep may allow the page reclaimer to
|
||||
* free a page.
|
||||
*/
|
||||
fsleep(1000);
|
||||
p = kmalloc(size, gfp_flags);
|
||||
}
|
||||
|
||||
if (p != NULL)
|
||||
add_kmalloc_block(ksize(p));
|
||||
} else {
|
||||
struct vmalloc_block_info *block;
|
||||
|
||||
if (vdo_allocate(1, struct vmalloc_block_info, __func__, &block) == VDO_SUCCESS) {
|
||||
/*
|
||||
* It is possible for __vmalloc to fail to allocate memory because there
|
||||
* are no pages available. A short sleep may allow the page reclaimer
|
||||
* to free enough pages for a small allocation.
|
||||
*
|
||||
* For larger allocations, the page_alloc code is racing against the page
|
||||
* reclaimer. If the page reclaimer can stay ahead of page_alloc, the
|
||||
* __vmalloc will succeed. But if page_alloc overtakes the page reclaimer,
|
||||
* the allocation fails. It is possible that more retries will succeed.
|
||||
*/
|
||||
for (;;) {
|
||||
p = __vmalloc(size, gfp_flags | __GFP_NOWARN);
|
||||
if (p != NULL)
|
||||
break;
|
||||
|
||||
if (jiffies_to_msecs(jiffies - start_time) > 1000) {
|
||||
/* Try one more time, logging a failure for this call. */
|
||||
p = __vmalloc(size, gfp_flags);
|
||||
break;
|
||||
}
|
||||
|
||||
fsleep(1000);
|
||||
}
|
||||
|
||||
if (p == NULL) {
|
||||
vdo_free(block);
|
||||
} else {
|
||||
block->ptr = p;
|
||||
block->size = PAGE_ALIGN(size);
|
||||
add_vmalloc_block(block);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (allocations_restricted)
|
||||
memalloc_noio_restore(noio_flags);
|
||||
|
||||
if (unlikely(p == NULL)) {
|
||||
vdo_log_error("Could not allocate %zu bytes for %s in %u msecs",
|
||||
size, what, jiffies_to_msecs(jiffies - start_time));
|
||||
return -ENOMEM;
|
||||
}
|
||||
|
||||
*((void **) ptr) = p;
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
/*
|
||||
* Allocate storage based on memory size, failing immediately if the required memory is not
|
||||
* available. The memory will be zeroed.
|
||||
*
|
||||
* @size: The size of an object.
|
||||
* @what: What is being allocated (for error logging)
|
||||
*
|
||||
* Return: pointer to the allocated memory, or NULL if the required space is not available.
|
||||
*/
|
||||
void *vdo_allocate_memory_nowait(size_t size, const char *what __maybe_unused)
|
||||
{
|
||||
void *p = kmalloc(size, GFP_NOWAIT | __GFP_ZERO);
|
||||
|
||||
if (p != NULL)
|
||||
add_kmalloc_block(ksize(p));
|
||||
|
||||
return p;
|
||||
}
|
||||
|
||||
void vdo_free(void *ptr)
|
||||
{
|
||||
if (ptr != NULL) {
|
||||
if (is_vmalloc_addr(ptr)) {
|
||||
remove_vmalloc_block(ptr);
|
||||
vfree(ptr);
|
||||
} else {
|
||||
remove_kmalloc_block(ksize(ptr));
|
||||
kfree(ptr);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
* Reallocate dynamically allocated memory. There are no alignment guarantees for the reallocated
|
||||
* memory. If the new memory is larger than the old memory, the new space will be zeroed.
|
||||
*
|
||||
* @ptr: The memory to reallocate.
|
||||
* @old_size: The old size of the memory
|
||||
* @size: The new size to allocate
|
||||
* @what: What is being allocated (for error logging)
|
||||
* @new_ptr: A pointer to hold the reallocated pointer
|
||||
*
|
||||
* Return: VDO_SUCCESS or an error code
|
||||
*/
|
||||
int vdo_reallocate_memory(void *ptr, size_t old_size, size_t size, const char *what,
|
||||
void *new_ptr)
|
||||
{
|
||||
int result;
|
||||
|
||||
if (size == 0) {
|
||||
vdo_free(ptr);
|
||||
*(void **) new_ptr = NULL;
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
result = vdo_allocate(size, char, what, new_ptr);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
if (ptr != NULL) {
|
||||
if (old_size < size)
|
||||
size = old_size;
|
||||
|
||||
memcpy(*((void **) new_ptr), ptr, size);
|
||||
vdo_free(ptr);
|
||||
}
|
||||
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
int vdo_duplicate_string(const char *string, const char *what, char **new_string)
|
||||
{
|
||||
int result;
|
||||
u8 *dup;
|
||||
|
||||
result = vdo_allocate(strlen(string) + 1, u8, what, &dup);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
memcpy(dup, string, strlen(string) + 1);
|
||||
*new_string = dup;
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
void vdo_memory_init(void)
|
||||
{
|
||||
spin_lock_init(&memory_stats.lock);
|
||||
vdo_initialize_thread_registry(&allocating_threads);
|
||||
}
|
||||
|
||||
void vdo_memory_exit(void)
|
||||
{
|
||||
VDO_ASSERT_LOG_ONLY(memory_stats.kmalloc_bytes == 0,
|
||||
"kmalloc memory used (%zd bytes in %zd blocks) is returned to the kernel",
|
||||
memory_stats.kmalloc_bytes, memory_stats.kmalloc_blocks);
|
||||
VDO_ASSERT_LOG_ONLY(memory_stats.vmalloc_bytes == 0,
|
||||
"vmalloc memory used (%zd bytes in %zd blocks) is returned to the kernel",
|
||||
memory_stats.vmalloc_bytes, memory_stats.vmalloc_blocks);
|
||||
vdo_log_debug("peak usage %zd bytes", memory_stats.peak_bytes);
|
||||
}
|
||||
|
||||
void vdo_get_memory_stats(u64 *bytes_used, u64 *peak_bytes_used)
|
||||
{
|
||||
unsigned long flags;
|
||||
|
||||
spin_lock_irqsave(&memory_stats.lock, flags);
|
||||
*bytes_used = memory_stats.kmalloc_bytes + memory_stats.vmalloc_bytes;
|
||||
*peak_bytes_used = memory_stats.peak_bytes;
|
||||
spin_unlock_irqrestore(&memory_stats.lock, flags);
|
||||
}
|
||||
|
||||
/*
|
||||
* Report stats on any allocated memory that we're tracking. Not all allocation types are
|
||||
* guaranteed to be tracked in bytes (e.g., bios).
|
||||
*/
|
||||
void vdo_report_memory_usage(void)
|
||||
{
|
||||
unsigned long flags;
|
||||
u64 kmalloc_blocks;
|
||||
u64 kmalloc_bytes;
|
||||
u64 vmalloc_blocks;
|
||||
u64 vmalloc_bytes;
|
||||
u64 peak_usage;
|
||||
u64 total_bytes;
|
||||
|
||||
spin_lock_irqsave(&memory_stats.lock, flags);
|
||||
kmalloc_blocks = memory_stats.kmalloc_blocks;
|
||||
kmalloc_bytes = memory_stats.kmalloc_bytes;
|
||||
vmalloc_blocks = memory_stats.vmalloc_blocks;
|
||||
vmalloc_bytes = memory_stats.vmalloc_bytes;
|
||||
peak_usage = memory_stats.peak_bytes;
|
||||
spin_unlock_irqrestore(&memory_stats.lock, flags);
|
||||
total_bytes = kmalloc_bytes + vmalloc_bytes;
|
||||
vdo_log_info("current module memory tracking (actual allocation sizes, not requested):");
|
||||
vdo_log_info(" %llu bytes in %llu kmalloc blocks",
|
||||
(unsigned long long) kmalloc_bytes,
|
||||
(unsigned long long) kmalloc_blocks);
|
||||
vdo_log_info(" %llu bytes in %llu vmalloc blocks",
|
||||
(unsigned long long) vmalloc_bytes,
|
||||
(unsigned long long) vmalloc_blocks);
|
||||
vdo_log_info(" total %llu bytes, peak usage %llu bytes",
|
||||
(unsigned long long) total_bytes, (unsigned long long) peak_usage);
|
||||
}
|
|
@ -0,0 +1,162 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef VDO_MEMORY_ALLOC_H
|
||||
#define VDO_MEMORY_ALLOC_H
|
||||
|
||||
#include <linux/cache.h>
|
||||
#include <linux/io.h> /* for PAGE_SIZE */
|
||||
|
||||
#include "permassert.h"
|
||||
#include "thread-registry.h"
|
||||
|
||||
/* Custom memory allocation function that tracks memory usage */
|
||||
int __must_check vdo_allocate_memory(size_t size, size_t align, const char *what, void *ptr);
|
||||
|
||||
/*
|
||||
* Allocate storage based on element counts, sizes, and alignment.
|
||||
*
|
||||
* This is a generalized form of our allocation use case: It allocates an array of objects,
|
||||
* optionally preceded by one object of another type (i.e., a struct with trailing variable-length
|
||||
* array), with the alignment indicated.
|
||||
*
|
||||
* Why is this inline? The sizes and alignment will always be constant, when invoked through the
|
||||
* macros below, and often the count will be a compile-time constant 1 or the number of extra bytes
|
||||
* will be a compile-time constant 0. So at least some of the arithmetic can usually be optimized
|
||||
* away, and the run-time selection between allocation functions always can. In many cases, it'll
|
||||
* boil down to just a function call with a constant size.
|
||||
*
|
||||
* @count: The number of objects to allocate
|
||||
* @size: The size of an object
|
||||
* @extra: The number of additional bytes to allocate
|
||||
* @align: The required alignment
|
||||
* @what: What is being allocated (for error logging)
|
||||
* @ptr: A pointer to hold the allocated memory
|
||||
*
|
||||
* Return: VDO_SUCCESS or an error code
|
||||
*/
|
||||
static inline int __vdo_do_allocation(size_t count, size_t size, size_t extra,
|
||||
size_t align, const char *what, void *ptr)
|
||||
{
|
||||
size_t total_size = count * size + extra;
|
||||
|
||||
/* Overflow check: */
|
||||
if ((size > 0) && (count > ((SIZE_MAX - extra) / size))) {
|
||||
/*
|
||||
* This is kind of a hack: We rely on the fact that SIZE_MAX would cover the entire
|
||||
* address space (minus one byte) and thus the system can never allocate that much
|
||||
* and the call will always fail. So we can report an overflow as "out of memory"
|
||||
* by asking for "merely" SIZE_MAX bytes.
|
||||
*/
|
||||
total_size = SIZE_MAX;
|
||||
}
|
||||
|
||||
return vdo_allocate_memory(total_size, align, what, ptr);
|
||||
}
|
||||
|
||||
/*
|
||||
* Allocate one or more elements of the indicated type, logging an error if the allocation fails.
|
||||
* The memory will be zeroed.
|
||||
*
|
||||
* @COUNT: The number of objects to allocate
|
||||
* @TYPE: The type of objects to allocate. This type determines the alignment of the allocation.
|
||||
* @WHAT: What is being allocated (for error logging)
|
||||
* @PTR: A pointer to hold the allocated memory
|
||||
*
|
||||
* Return: VDO_SUCCESS or an error code
|
||||
*/
|
||||
#define vdo_allocate(COUNT, TYPE, WHAT, PTR) \
|
||||
__vdo_do_allocation(COUNT, sizeof(TYPE), 0, __alignof__(TYPE), WHAT, PTR)
|
||||
|
||||
/*
|
||||
* Allocate one object of an indicated type, followed by one or more elements of a second type,
|
||||
* logging an error if the allocation fails. The memory will be zeroed.
|
||||
*
|
||||
* @TYPE1: The type of the primary object to allocate. This type determines the alignment of the
|
||||
* allocated memory.
|
||||
* @COUNT: The number of objects to allocate
|
||||
* @TYPE2: The type of array objects to allocate
|
||||
* @WHAT: What is being allocated (for error logging)
|
||||
* @PTR: A pointer to hold the allocated memory
|
||||
*
|
||||
* Return: VDO_SUCCESS or an error code
|
||||
*/
|
||||
#define vdo_allocate_extended(TYPE1, COUNT, TYPE2, WHAT, PTR) \
|
||||
__extension__({ \
|
||||
int _result; \
|
||||
TYPE1 **_ptr = (PTR); \
|
||||
BUILD_BUG_ON(__alignof__(TYPE1) < __alignof__(TYPE2)); \
|
||||
_result = __vdo_do_allocation(COUNT, \
|
||||
sizeof(TYPE2), \
|
||||
sizeof(TYPE1), \
|
||||
__alignof__(TYPE1), \
|
||||
WHAT, \
|
||||
_ptr); \
|
||||
_result; \
|
||||
})
|
||||
|
||||
/*
|
||||
* Allocate memory starting on a cache line boundary, logging an error if the allocation fails. The
|
||||
* memory will be zeroed.
|
||||
*
|
||||
* @size: The number of bytes to allocate
|
||||
* @what: What is being allocated (for error logging)
|
||||
* @ptr: A pointer to hold the allocated memory
|
||||
*
|
||||
* Return: VDO_SUCCESS or an error code
|
||||
*/
|
||||
static inline int __must_check vdo_allocate_cache_aligned(size_t size, const char *what, void *ptr)
|
||||
{
|
||||
return vdo_allocate_memory(size, L1_CACHE_BYTES, what, ptr);
|
||||
}
|
||||
|
||||
/*
|
||||
* Allocate one element of the indicated type immediately, failing if the required memory is not
|
||||
* immediately available.
|
||||
*
|
||||
* @size: The number of bytes to allocate
|
||||
* @what: What is being allocated (for error logging)
|
||||
*
|
||||
* Return: pointer to the memory, or NULL if the memory is not available.
|
||||
*/
|
||||
void *__must_check vdo_allocate_memory_nowait(size_t size, const char *what);
|
||||
|
||||
int __must_check vdo_reallocate_memory(void *ptr, size_t old_size, size_t size,
|
||||
const char *what, void *new_ptr);
|
||||
|
||||
int __must_check vdo_duplicate_string(const char *string, const char *what,
|
||||
char **new_string);
|
||||
|
||||
/* Free memory allocated with vdo_allocate(). */
|
||||
void vdo_free(void *ptr);
|
||||
|
||||
static inline void *__vdo_forget(void **ptr_ptr)
|
||||
{
|
||||
void *ptr = *ptr_ptr;
|
||||
|
||||
*ptr_ptr = NULL;
|
||||
return ptr;
|
||||
}
|
||||
|
||||
/*
|
||||
* Null out a pointer and return a copy to it. This macro should be used when passing a pointer to
|
||||
* a function for which it is not safe to access the pointer once the function returns.
|
||||
*/
|
||||
#define vdo_forget(ptr) __vdo_forget((void **) &(ptr))
|
||||
|
||||
void vdo_memory_init(void);
|
||||
|
||||
void vdo_memory_exit(void);
|
||||
|
||||
void vdo_register_allocating_thread(struct registered_thread *new_thread,
|
||||
const bool *flag_ptr);
|
||||
|
||||
void vdo_unregister_allocating_thread(void);
|
||||
|
||||
void vdo_get_memory_stats(u64 *bytes_used, u64 *peak_bytes_used);
|
||||
|
||||
void vdo_report_memory_usage(void);
|
||||
|
||||
#endif /* VDO_MEMORY_ALLOC_H */
|
|
@ -0,0 +1,432 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "dedupe.h"
|
||||
#include "logger.h"
|
||||
#include "memory-alloc.h"
|
||||
#include "message-stats.h"
|
||||
#include "statistics.h"
|
||||
#include "thread-device.h"
|
||||
#include "vdo.h"
|
||||
|
||||
static void write_u64(char *prefix, u64 value, char *suffix, char **buf,
|
||||
unsigned int *maxlen)
|
||||
{
|
||||
int count;
|
||||
|
||||
count = scnprintf(*buf, *maxlen, "%s%llu%s", prefix == NULL ? "" : prefix,
|
||||
value, suffix == NULL ? "" : suffix);
|
||||
*buf += count;
|
||||
*maxlen -= count;
|
||||
}
|
||||
|
||||
static void write_u32(char *prefix, u32 value, char *suffix, char **buf,
|
||||
unsigned int *maxlen)
|
||||
{
|
||||
int count;
|
||||
|
||||
count = scnprintf(*buf, *maxlen, "%s%u%s", prefix == NULL ? "" : prefix,
|
||||
value, suffix == NULL ? "" : suffix);
|
||||
*buf += count;
|
||||
*maxlen -= count;
|
||||
}
|
||||
|
||||
static void write_block_count_t(char *prefix, block_count_t value, char *suffix,
|
||||
char **buf, unsigned int *maxlen)
|
||||
{
|
||||
int count;
|
||||
|
||||
count = scnprintf(*buf, *maxlen, "%s%llu%s", prefix == NULL ? "" : prefix,
|
||||
value, suffix == NULL ? "" : suffix);
|
||||
*buf += count;
|
||||
*maxlen -= count;
|
||||
}
|
||||
|
||||
static void write_string(char *prefix, char *value, char *suffix, char **buf,
|
||||
unsigned int *maxlen)
|
||||
{
|
||||
int count;
|
||||
|
||||
count = scnprintf(*buf, *maxlen, "%s%s%s", prefix == NULL ? "" : prefix,
|
||||
value, suffix == NULL ? "" : suffix);
|
||||
*buf += count;
|
||||
*maxlen -= count;
|
||||
}
|
||||
|
||||
static void write_bool(char *prefix, bool value, char *suffix, char **buf,
|
||||
unsigned int *maxlen)
|
||||
{
|
||||
int count;
|
||||
|
||||
count = scnprintf(*buf, *maxlen, "%s%d%s", prefix == NULL ? "" : prefix,
|
||||
value, suffix == NULL ? "" : suffix);
|
||||
*buf += count;
|
||||
*maxlen -= count;
|
||||
}
|
||||
|
||||
static void write_u8(char *prefix, u8 value, char *suffix, char **buf,
|
||||
unsigned int *maxlen)
|
||||
{
|
||||
int count;
|
||||
|
||||
count = scnprintf(*buf, *maxlen, "%s%u%s", prefix == NULL ? "" : prefix,
|
||||
value, suffix == NULL ? "" : suffix);
|
||||
*buf += count;
|
||||
*maxlen -= count;
|
||||
}
|
||||
|
||||
static void write_block_allocator_statistics(char *prefix,
|
||||
struct block_allocator_statistics *stats,
|
||||
char *suffix, char **buf,
|
||||
unsigned int *maxlen)
|
||||
{
|
||||
write_string(prefix, "{ ", NULL, buf, maxlen);
|
||||
/* The total number of slabs from which blocks may be allocated */
|
||||
write_u64("slabCount : ", stats->slab_count, ", ", buf, maxlen);
|
||||
/* The total number of slabs from which blocks have ever been allocated */
|
||||
write_u64("slabsOpened : ", stats->slabs_opened, ", ", buf, maxlen);
|
||||
/* The number of times since loading that a slab has been re-opened */
|
||||
write_u64("slabsReopened : ", stats->slabs_reopened, ", ", buf, maxlen);
|
||||
write_string(NULL, "}", suffix, buf, maxlen);
|
||||
}
|
||||
|
||||
static void write_commit_statistics(char *prefix, struct commit_statistics *stats,
|
||||
char *suffix, char **buf, unsigned int *maxlen)
|
||||
{
|
||||
write_string(prefix, "{ ", NULL, buf, maxlen);
|
||||
/* The total number of items on which processing has started */
|
||||
write_u64("started : ", stats->started, ", ", buf, maxlen);
|
||||
/* The total number of items for which a write operation has been issued */
|
||||
write_u64("written : ", stats->written, ", ", buf, maxlen);
|
||||
/* The total number of items for which a write operation has completed */
|
||||
write_u64("committed : ", stats->committed, ", ", buf, maxlen);
|
||||
write_string(NULL, "}", suffix, buf, maxlen);
|
||||
}
|
||||
|
||||
static void write_recovery_journal_statistics(char *prefix,
|
||||
struct recovery_journal_statistics *stats,
|
||||
char *suffix, char **buf,
|
||||
unsigned int *maxlen)
|
||||
{
|
||||
write_string(prefix, "{ ", NULL, buf, maxlen);
|
||||
/* Number of times the on-disk journal was full */
|
||||
write_u64("diskFull : ", stats->disk_full, ", ", buf, maxlen);
|
||||
/* Number of times the recovery journal requested slab journal commits. */
|
||||
write_u64("slabJournalCommitsRequested : ",
|
||||
stats->slab_journal_commits_requested, ", ", buf, maxlen);
|
||||
/* Write/Commit totals for individual journal entries */
|
||||
write_commit_statistics("entries : ", &stats->entries, ", ", buf, maxlen);
|
||||
/* Write/Commit totals for journal blocks */
|
||||
write_commit_statistics("blocks : ", &stats->blocks, ", ", buf, maxlen);
|
||||
write_string(NULL, "}", suffix, buf, maxlen);
|
||||
}
|
||||
|
||||
static void write_packer_statistics(char *prefix, struct packer_statistics *stats,
|
||||
char *suffix, char **buf, unsigned int *maxlen)
|
||||
{
|
||||
write_string(prefix, "{ ", NULL, buf, maxlen);
|
||||
/* Number of compressed data items written since startup */
|
||||
write_u64("compressedFragmentsWritten : ",
|
||||
stats->compressed_fragments_written, ", ", buf, maxlen);
|
||||
/* Number of blocks containing compressed items written since startup */
|
||||
write_u64("compressedBlocksWritten : ",
|
||||
stats->compressed_blocks_written, ", ", buf, maxlen);
|
||||
/* Number of VIOs that are pending in the packer */
|
||||
write_u64("compressedFragmentsInPacker : ",
|
||||
stats->compressed_fragments_in_packer, ", ", buf, maxlen);
|
||||
write_string(NULL, "}", suffix, buf, maxlen);
|
||||
}
|
||||
|
||||
static void write_slab_journal_statistics(char *prefix,
|
||||
struct slab_journal_statistics *stats,
|
||||
char *suffix, char **buf, unsigned int *maxlen)
|
||||
{
|
||||
write_string(prefix, "{ ", NULL, buf, maxlen);
|
||||
/* Number of times the on-disk journal was full */
|
||||
write_u64("diskFullCount : ", stats->disk_full_count, ", ", buf, maxlen);
|
||||
/* Number of times an entry was added over the flush threshold */
|
||||
write_u64("flushCount : ", stats->flush_count, ", ", buf, maxlen);
|
||||
/* Number of times an entry was added over the block threshold */
|
||||
write_u64("blockedCount : ", stats->blocked_count, ", ", buf, maxlen);
|
||||
/* Number of times a tail block was written */
|
||||
write_u64("blocksWritten : ", stats->blocks_written, ", ", buf, maxlen);
|
||||
/* Number of times we had to wait for the tail to write */
|
||||
write_u64("tailBusyCount : ", stats->tail_busy_count, ", ", buf, maxlen);
|
||||
write_string(NULL, "}", suffix, buf, maxlen);
|
||||
}
|
||||
|
||||
static void write_slab_summary_statistics(char *prefix,
|
||||
struct slab_summary_statistics *stats,
|
||||
char *suffix, char **buf, unsigned int *maxlen)
|
||||
{
|
||||
write_string(prefix, "{ ", NULL, buf, maxlen);
|
||||
/* Number of blocks written */
|
||||
write_u64("blocksWritten : ", stats->blocks_written, ", ", buf, maxlen);
|
||||
write_string(NULL, "}", suffix, buf, maxlen);
|
||||
}
|
||||
|
||||
static void write_ref_counts_statistics(char *prefix, struct ref_counts_statistics *stats,
|
||||
char *suffix, char **buf, unsigned int *maxlen)
|
||||
{
|
||||
write_string(prefix, "{ ", NULL, buf, maxlen);
|
||||
/* Number of reference blocks written */
|
||||
write_u64("blocksWritten : ", stats->blocks_written, ", ", buf, maxlen);
|
||||
write_string(NULL, "}", suffix, buf, maxlen);
|
||||
}
|
||||
|
||||
static void write_block_map_statistics(char *prefix, struct block_map_statistics *stats,
|
||||
char *suffix, char **buf, unsigned int *maxlen)
|
||||
{
|
||||
write_string(prefix, "{ ", NULL, buf, maxlen);
|
||||
/* number of dirty (resident) pages */
|
||||
write_u32("dirtyPages : ", stats->dirty_pages, ", ", buf, maxlen);
|
||||
/* number of clean (resident) pages */
|
||||
write_u32("cleanPages : ", stats->clean_pages, ", ", buf, maxlen);
|
||||
/* number of free pages */
|
||||
write_u32("freePages : ", stats->free_pages, ", ", buf, maxlen);
|
||||
/* number of pages in failed state */
|
||||
write_u32("failedPages : ", stats->failed_pages, ", ", buf, maxlen);
|
||||
/* number of pages incoming */
|
||||
write_u32("incomingPages : ", stats->incoming_pages, ", ", buf, maxlen);
|
||||
/* number of pages outgoing */
|
||||
write_u32("outgoingPages : ", stats->outgoing_pages, ", ", buf, maxlen);
|
||||
/* how many times free page not avail */
|
||||
write_u32("cachePressure : ", stats->cache_pressure, ", ", buf, maxlen);
|
||||
/* number of get_vdo_page() calls for read */
|
||||
write_u64("readCount : ", stats->read_count, ", ", buf, maxlen);
|
||||
/* number of get_vdo_page() calls for write */
|
||||
write_u64("writeCount : ", stats->write_count, ", ", buf, maxlen);
|
||||
/* number of times pages failed to read */
|
||||
write_u64("failedReads : ", stats->failed_reads, ", ", buf, maxlen);
|
||||
/* number of times pages failed to write */
|
||||
write_u64("failedWrites : ", stats->failed_writes, ", ", buf, maxlen);
|
||||
/* number of gets that are reclaimed */
|
||||
write_u64("reclaimed : ", stats->reclaimed, ", ", buf, maxlen);
|
||||
/* number of gets for outgoing pages */
|
||||
write_u64("readOutgoing : ", stats->read_outgoing, ", ", buf, maxlen);
|
||||
/* number of gets that were already there */
|
||||
write_u64("foundInCache : ", stats->found_in_cache, ", ", buf, maxlen);
|
||||
/* number of gets requiring discard */
|
||||
write_u64("discardRequired : ", stats->discard_required, ", ", buf, maxlen);
|
||||
/* number of gets enqueued for their page */
|
||||
write_u64("waitForPage : ", stats->wait_for_page, ", ", buf, maxlen);
|
||||
/* number of gets that have to fetch */
|
||||
write_u64("fetchRequired : ", stats->fetch_required, ", ", buf, maxlen);
|
||||
/* number of page fetches */
|
||||
write_u64("pagesLoaded : ", stats->pages_loaded, ", ", buf, maxlen);
|
||||
/* number of page saves */
|
||||
write_u64("pagesSaved : ", stats->pages_saved, ", ", buf, maxlen);
|
||||
/* the number of flushes issued */
|
||||
write_u64("flushCount : ", stats->flush_count, ", ", buf, maxlen);
|
||||
write_string(NULL, "}", suffix, buf, maxlen);
|
||||
}
|
||||
|
||||
static void write_hash_lock_statistics(char *prefix, struct hash_lock_statistics *stats,
|
||||
char *suffix, char **buf, unsigned int *maxlen)
|
||||
{
|
||||
write_string(prefix, "{ ", NULL, buf, maxlen);
|
||||
/* Number of times the UDS advice proved correct */
|
||||
write_u64("dedupeAdviceValid : ", stats->dedupe_advice_valid, ", ", buf, maxlen);
|
||||
/* Number of times the UDS advice proved incorrect */
|
||||
write_u64("dedupeAdviceStale : ", stats->dedupe_advice_stale, ", ", buf, maxlen);
|
||||
/* Number of writes with the same data as another in-flight write */
|
||||
write_u64("concurrentDataMatches : ", stats->concurrent_data_matches,
|
||||
", ", buf, maxlen);
|
||||
/* Number of writes whose hash collided with an in-flight write */
|
||||
write_u64("concurrentHashCollisions : ",
|
||||
stats->concurrent_hash_collisions, ", ", buf, maxlen);
|
||||
/* Current number of dedupe queries that are in flight */
|
||||
write_u32("currDedupeQueries : ", stats->curr_dedupe_queries, ", ", buf, maxlen);
|
||||
write_string(NULL, "}", suffix, buf, maxlen);
|
||||
}
|
||||
|
||||
static void write_error_statistics(char *prefix, struct error_statistics *stats,
|
||||
char *suffix, char **buf, unsigned int *maxlen)
|
||||
{
|
||||
write_string(prefix, "{ ", NULL, buf, maxlen);
|
||||
/* number of times VDO got an invalid dedupe advice PBN from UDS */
|
||||
write_u64("invalidAdvicePBNCount : ", stats->invalid_advice_pbn_count,
|
||||
", ", buf, maxlen);
|
||||
/* number of times a VIO completed with a VDO_NO_SPACE error */
|
||||
write_u64("noSpaceErrorCount : ", stats->no_space_error_count, ", ",
|
||||
buf, maxlen);
|
||||
/* number of times a VIO completed with a VDO_READ_ONLY error */
|
||||
write_u64("readOnlyErrorCount : ", stats->read_only_error_count, ", ",
|
||||
buf, maxlen);
|
||||
write_string(NULL, "}", suffix, buf, maxlen);
|
||||
}
|
||||
|
||||
static void write_bio_stats(char *prefix, struct bio_stats *stats, char *suffix,
|
||||
char **buf, unsigned int *maxlen)
|
||||
{
|
||||
write_string(prefix, "{ ", NULL, buf, maxlen);
|
||||
/* Number of REQ_OP_READ bios */
|
||||
write_u64("read : ", stats->read, ", ", buf, maxlen);
|
||||
/* Number of REQ_OP_WRITE bios with data */
|
||||
write_u64("write : ", stats->write, ", ", buf, maxlen);
|
||||
/* Number of bios tagged with REQ_PREFLUSH and containing no data */
|
||||
write_u64("emptyFlush : ", stats->empty_flush, ", ", buf, maxlen);
|
||||
/* Number of REQ_OP_DISCARD bios */
|
||||
write_u64("discard : ", stats->discard, ", ", buf, maxlen);
|
||||
/* Number of bios tagged with REQ_PREFLUSH */
|
||||
write_u64("flush : ", stats->flush, ", ", buf, maxlen);
|
||||
/* Number of bios tagged with REQ_FUA */
|
||||
write_u64("fua : ", stats->fua, ", ", buf, maxlen);
|
||||
write_string(NULL, "}", suffix, buf, maxlen);
|
||||
}
|
||||
|
||||
static void write_memory_usage(char *prefix, struct memory_usage *stats, char *suffix,
|
||||
char **buf, unsigned int *maxlen)
|
||||
{
|
||||
write_string(prefix, "{ ", NULL, buf, maxlen);
|
||||
/* Tracked bytes currently allocated. */
|
||||
write_u64("bytesUsed : ", stats->bytes_used, ", ", buf, maxlen);
|
||||
/* Maximum tracked bytes allocated. */
|
||||
write_u64("peakBytesUsed : ", stats->peak_bytes_used, ", ", buf, maxlen);
|
||||
write_string(NULL, "}", suffix, buf, maxlen);
|
||||
}
|
||||
|
||||
static void write_index_statistics(char *prefix, struct index_statistics *stats,
|
||||
char *suffix, char **buf, unsigned int *maxlen)
|
||||
{
|
||||
write_string(prefix, "{ ", NULL, buf, maxlen);
|
||||
/* Number of records stored in the index */
|
||||
write_u64("entriesIndexed : ", stats->entries_indexed, ", ", buf, maxlen);
|
||||
/* Number of post calls that found an existing entry */
|
||||
write_u64("postsFound : ", stats->posts_found, ", ", buf, maxlen);
|
||||
/* Number of post calls that added a new entry */
|
||||
write_u64("postsNotFound : ", stats->posts_not_found, ", ", buf, maxlen);
|
||||
/* Number of query calls that found an existing entry */
|
||||
write_u64("queriesFound : ", stats->queries_found, ", ", buf, maxlen);
|
||||
/* Number of query calls that added a new entry */
|
||||
write_u64("queriesNotFound : ", stats->queries_not_found, ", ", buf, maxlen);
|
||||
/* Number of update calls that found an existing entry */
|
||||
write_u64("updatesFound : ", stats->updates_found, ", ", buf, maxlen);
|
||||
/* Number of update calls that added a new entry */
|
||||
write_u64("updatesNotFound : ", stats->updates_not_found, ", ", buf, maxlen);
|
||||
/* Number of entries discarded */
|
||||
write_u64("entriesDiscarded : ", stats->entries_discarded, ", ", buf, maxlen);
|
||||
write_string(NULL, "}", suffix, buf, maxlen);
|
||||
}
|
||||
|
||||
static void write_vdo_statistics(char *prefix, struct vdo_statistics *stats, char *suffix,
|
||||
char **buf, unsigned int *maxlen)
|
||||
{
|
||||
write_string(prefix, "{ ", NULL, buf, maxlen);
|
||||
write_u32("version : ", stats->version, ", ", buf, maxlen);
|
||||
/* Number of blocks used for data */
|
||||
write_u64("dataBlocksUsed : ", stats->data_blocks_used, ", ", buf, maxlen);
|
||||
/* Number of blocks used for VDO metadata */
|
||||
write_u64("overheadBlocksUsed : ", stats->overhead_blocks_used, ", ",
|
||||
buf, maxlen);
|
||||
/* Number of logical blocks that are currently mapped to physical blocks */
|
||||
write_u64("logicalBlocksUsed : ", stats->logical_blocks_used, ", ", buf, maxlen);
|
||||
/* number of physical blocks */
|
||||
write_block_count_t("physicalBlocks : ", stats->physical_blocks, ", ",
|
||||
buf, maxlen);
|
||||
/* number of logical blocks */
|
||||
write_block_count_t("logicalBlocks : ", stats->logical_blocks, ", ",
|
||||
buf, maxlen);
|
||||
/* Size of the block map page cache, in bytes */
|
||||
write_u64("blockMapCacheSize : ", stats->block_map_cache_size, ", ",
|
||||
buf, maxlen);
|
||||
/* The physical block size */
|
||||
write_u64("blockSize : ", stats->block_size, ", ", buf, maxlen);
|
||||
/* Number of times the VDO has successfully recovered */
|
||||
write_u64("completeRecoveries : ", stats->complete_recoveries, ", ",
|
||||
buf, maxlen);
|
||||
/* Number of times the VDO has recovered from read-only mode */
|
||||
write_u64("readOnlyRecoveries : ", stats->read_only_recoveries, ", ",
|
||||
buf, maxlen);
|
||||
/* String describing the operating mode of the VDO */
|
||||
write_string("mode : ", stats->mode, ", ", buf, maxlen);
|
||||
/* Whether the VDO is in recovery mode */
|
||||
write_bool("inRecoveryMode : ", stats->in_recovery_mode, ", ", buf, maxlen);
|
||||
/* What percentage of recovery mode work has been completed */
|
||||
write_u8("recoveryPercentage : ", stats->recovery_percentage, ", ", buf, maxlen);
|
||||
/* The statistics for the compressed block packer */
|
||||
write_packer_statistics("packer : ", &stats->packer, ", ", buf, maxlen);
|
||||
/* Counters for events in the block allocator */
|
||||
write_block_allocator_statistics("allocator : ", &stats->allocator,
|
||||
", ", buf, maxlen);
|
||||
/* Counters for events in the recovery journal */
|
||||
write_recovery_journal_statistics("journal : ", &stats->journal, ", ",
|
||||
buf, maxlen);
|
||||
/* The statistics for the slab journals */
|
||||
write_slab_journal_statistics("slabJournal : ", &stats->slab_journal,
|
||||
", ", buf, maxlen);
|
||||
/* The statistics for the slab summary */
|
||||
write_slab_summary_statistics("slabSummary : ", &stats->slab_summary,
|
||||
", ", buf, maxlen);
|
||||
/* The statistics for the reference counts */
|
||||
write_ref_counts_statistics("refCounts : ", &stats->ref_counts, ", ",
|
||||
buf, maxlen);
|
||||
/* The statistics for the block map */
|
||||
write_block_map_statistics("blockMap : ", &stats->block_map, ", ", buf, maxlen);
|
||||
/* The dedupe statistics from hash locks */
|
||||
write_hash_lock_statistics("hashLock : ", &stats->hash_lock, ", ", buf, maxlen);
|
||||
/* Counts of error conditions */
|
||||
write_error_statistics("errors : ", &stats->errors, ", ", buf, maxlen);
|
||||
/* The VDO instance */
|
||||
write_u32("instance : ", stats->instance, ", ", buf, maxlen);
|
||||
/* Current number of active VIOs */
|
||||
write_u32("currentVIOsInProgress : ", stats->current_vios_in_progress,
|
||||
", ", buf, maxlen);
|
||||
/* Maximum number of active VIOs */
|
||||
write_u32("maxVIOs : ", stats->max_vios, ", ", buf, maxlen);
|
||||
/* Number of times the UDS index was too slow in responding */
|
||||
write_u64("dedupeAdviceTimeouts : ", stats->dedupe_advice_timeouts,
|
||||
", ", buf, maxlen);
|
||||
/* Number of flush requests submitted to the storage device */
|
||||
write_u64("flushOut : ", stats->flush_out, ", ", buf, maxlen);
|
||||
/* Logical block size */
|
||||
write_u64("logicalBlockSize : ", stats->logical_block_size, ", ", buf, maxlen);
|
||||
/* Bios submitted into VDO from above */
|
||||
write_bio_stats("biosIn : ", &stats->bios_in, ", ", buf, maxlen);
|
||||
write_bio_stats("biosInPartial : ", &stats->bios_in_partial, ", ", buf, maxlen);
|
||||
/* Bios submitted onward for user data */
|
||||
write_bio_stats("biosOut : ", &stats->bios_out, ", ", buf, maxlen);
|
||||
/* Bios submitted onward for metadata */
|
||||
write_bio_stats("biosMeta : ", &stats->bios_meta, ", ", buf, maxlen);
|
||||
write_bio_stats("biosJournal : ", &stats->bios_journal, ", ", buf, maxlen);
|
||||
write_bio_stats("biosPageCache : ", &stats->bios_page_cache, ", ", buf, maxlen);
|
||||
write_bio_stats("biosOutCompleted : ", &stats->bios_out_completed, ", ",
|
||||
buf, maxlen);
|
||||
write_bio_stats("biosMetaCompleted : ", &stats->bios_meta_completed,
|
||||
", ", buf, maxlen);
|
||||
write_bio_stats("biosJournalCompleted : ",
|
||||
&stats->bios_journal_completed, ", ", buf, maxlen);
|
||||
write_bio_stats("biosPageCacheCompleted : ",
|
||||
&stats->bios_page_cache_completed, ", ", buf, maxlen);
|
||||
write_bio_stats("biosAcknowledged : ", &stats->bios_acknowledged, ", ",
|
||||
buf, maxlen);
|
||||
write_bio_stats("biosAcknowledgedPartial : ",
|
||||
&stats->bios_acknowledged_partial, ", ", buf, maxlen);
|
||||
/* Current number of bios in progress */
|
||||
write_bio_stats("biosInProgress : ", &stats->bios_in_progress, ", ",
|
||||
buf, maxlen);
|
||||
/* Memory usage stats. */
|
||||
write_memory_usage("memoryUsage : ", &stats->memory_usage, ", ", buf, maxlen);
|
||||
/* The statistics for the UDS index */
|
||||
write_index_statistics("index : ", &stats->index, ", ", buf, maxlen);
|
||||
write_string(NULL, "}", suffix, buf, maxlen);
|
||||
}
|
||||
|
||||
int vdo_write_stats(struct vdo *vdo, char *buf, unsigned int maxlen)
|
||||
{
|
||||
struct vdo_statistics *stats;
|
||||
int result;
|
||||
|
||||
result = vdo_allocate(1, struct vdo_statistics, __func__, &stats);
|
||||
if (result != VDO_SUCCESS) {
|
||||
vdo_log_error("Cannot allocate memory to write VDO statistics");
|
||||
return result;
|
||||
}
|
||||
|
||||
vdo_fetch_statistics(vdo, stats);
|
||||
write_vdo_statistics(NULL, stats, NULL, &buf, &maxlen);
|
||||
vdo_free(stats);
|
||||
return VDO_SUCCESS;
|
||||
}
|
|
@ -0,0 +1,13 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef VDO_MESSAGE_STATS_H
|
||||
#define VDO_MESSAGE_STATS_H
|
||||
|
||||
#include "types.h"
|
||||
|
||||
int vdo_write_stats(struct vdo *vdo, char *buf, unsigned int maxlen);
|
||||
|
||||
#endif /* VDO_MESSAGE_STATS_H */
|
|
@ -0,0 +1,175 @@
|
|||
// SPDX-License-Identifier: LGPL-2.1+
|
||||
/*
|
||||
* MurmurHash3 was written by Austin Appleby, and is placed in the public
|
||||
* domain. The author hereby disclaims copyright to this source code.
|
||||
*
|
||||
* Adapted by John Wiele (jwiele@redhat.com).
|
||||
*/
|
||||
|
||||
#include "murmurhash3.h"
|
||||
|
||||
static inline u64 rotl64(u64 x, s8 r)
|
||||
{
|
||||
return (x << r) | (x >> (64 - r));
|
||||
}
|
||||
|
||||
#define ROTL64(x, y) rotl64(x, y)
|
||||
static __always_inline u64 getblock64(const u64 *p, int i)
|
||||
{
|
||||
#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
|
||||
return p[i];
|
||||
#elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
|
||||
return __builtin_bswap64(p[i]);
|
||||
#else
|
||||
#error "can't figure out byte order"
|
||||
#endif
|
||||
}
|
||||
|
||||
static __always_inline void putblock64(u64 *p, int i, u64 value)
|
||||
{
|
||||
#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
|
||||
p[i] = value;
|
||||
#elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
|
||||
p[i] = __builtin_bswap64(value);
|
||||
#else
|
||||
#error "can't figure out byte order"
|
||||
#endif
|
||||
}
|
||||
|
||||
/* Finalization mix - force all bits of a hash block to avalanche */
|
||||
|
||||
static __always_inline u64 fmix64(u64 k)
|
||||
{
|
||||
k ^= k >> 33;
|
||||
k *= 0xff51afd7ed558ccdLLU;
|
||||
k ^= k >> 33;
|
||||
k *= 0xc4ceb9fe1a85ec53LLU;
|
||||
k ^= k >> 33;
|
||||
|
||||
return k;
|
||||
}
|
||||
|
||||
void murmurhash3_128(const void *key, const int len, const u32 seed, void *out)
|
||||
{
|
||||
const u8 *data = key;
|
||||
const int nblocks = len / 16;
|
||||
|
||||
u64 h1 = seed;
|
||||
u64 h2 = seed;
|
||||
|
||||
const u64 c1 = 0x87c37b91114253d5LLU;
|
||||
const u64 c2 = 0x4cf5ad432745937fLLU;
|
||||
|
||||
/* body */
|
||||
|
||||
const u64 *blocks = (const u64 *)(data);
|
||||
|
||||
int i;
|
||||
|
||||
for (i = 0; i < nblocks; i++) {
|
||||
u64 k1 = getblock64(blocks, i * 2 + 0);
|
||||
u64 k2 = getblock64(blocks, i * 2 + 1);
|
||||
|
||||
k1 *= c1;
|
||||
k1 = ROTL64(k1, 31);
|
||||
k1 *= c2;
|
||||
h1 ^= k1;
|
||||
|
||||
h1 = ROTL64(h1, 27);
|
||||
h1 += h2;
|
||||
h1 = h1 * 5 + 0x52dce729;
|
||||
|
||||
k2 *= c2;
|
||||
k2 = ROTL64(k2, 33);
|
||||
k2 *= c1;
|
||||
h2 ^= k2;
|
||||
|
||||
h2 = ROTL64(h2, 31);
|
||||
h2 += h1;
|
||||
h2 = h2 * 5 + 0x38495ab5;
|
||||
}
|
||||
|
||||
/* tail */
|
||||
|
||||
{
|
||||
const u8 *tail = (const u8 *)(data + nblocks * 16);
|
||||
|
||||
u64 k1 = 0;
|
||||
u64 k2 = 0;
|
||||
|
||||
switch (len & 15) {
|
||||
case 15:
|
||||
k2 ^= ((u64)tail[14]) << 48;
|
||||
fallthrough;
|
||||
case 14:
|
||||
k2 ^= ((u64)tail[13]) << 40;
|
||||
fallthrough;
|
||||
case 13:
|
||||
k2 ^= ((u64)tail[12]) << 32;
|
||||
fallthrough;
|
||||
case 12:
|
||||
k2 ^= ((u64)tail[11]) << 24;
|
||||
fallthrough;
|
||||
case 11:
|
||||
k2 ^= ((u64)tail[10]) << 16;
|
||||
fallthrough;
|
||||
case 10:
|
||||
k2 ^= ((u64)tail[9]) << 8;
|
||||
fallthrough;
|
||||
case 9:
|
||||
k2 ^= ((u64)tail[8]) << 0;
|
||||
k2 *= c2;
|
||||
k2 = ROTL64(k2, 33);
|
||||
k2 *= c1;
|
||||
h2 ^= k2;
|
||||
fallthrough;
|
||||
|
||||
case 8:
|
||||
k1 ^= ((u64)tail[7]) << 56;
|
||||
fallthrough;
|
||||
case 7:
|
||||
k1 ^= ((u64)tail[6]) << 48;
|
||||
fallthrough;
|
||||
case 6:
|
||||
k1 ^= ((u64)tail[5]) << 40;
|
||||
fallthrough;
|
||||
case 5:
|
||||
k1 ^= ((u64)tail[4]) << 32;
|
||||
fallthrough;
|
||||
case 4:
|
||||
k1 ^= ((u64)tail[3]) << 24;
|
||||
fallthrough;
|
||||
case 3:
|
||||
k1 ^= ((u64)tail[2]) << 16;
|
||||
fallthrough;
|
||||
case 2:
|
||||
k1 ^= ((u64)tail[1]) << 8;
|
||||
fallthrough;
|
||||
case 1:
|
||||
k1 ^= ((u64)tail[0]) << 0;
|
||||
k1 *= c1;
|
||||
k1 = ROTL64(k1, 31);
|
||||
k1 *= c2;
|
||||
h1 ^= k1;
|
||||
break;
|
||||
default:
|
||||
break;
|
||||
};
|
||||
}
|
||||
/* finalization */
|
||||
|
||||
h1 ^= len;
|
||||
h2 ^= len;
|
||||
|
||||
h1 += h2;
|
||||
h2 += h1;
|
||||
|
||||
h1 = fmix64(h1);
|
||||
h2 = fmix64(h2);
|
||||
|
||||
h1 += h2;
|
||||
h2 += h1;
|
||||
|
||||
putblock64((u64 *)out, 0, h1);
|
||||
putblock64((u64 *)out, 1, h2);
|
||||
}
|
|
@ -0,0 +1,15 @@
|
|||
/* SPDX-License-Identifier: LGPL-2.1+ */
|
||||
/*
|
||||
* MurmurHash3 was written by Austin Appleby, and is placed in the public
|
||||
* domain. The author hereby disclaims copyright to this source code.
|
||||
*/
|
||||
|
||||
#ifndef _MURMURHASH3_H_
|
||||
#define _MURMURHASH3_H_
|
||||
|
||||
#include <linux/compiler.h>
|
||||
#include <linux/types.h>
|
||||
|
||||
void murmurhash3_128(const void *key, int len, u32 seed, void *out);
|
||||
|
||||
#endif /* _MURMURHASH3_H_ */
|
|
@ -0,0 +1,78 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef UDS_NUMERIC_H
|
||||
#define UDS_NUMERIC_H
|
||||
|
||||
#include <asm/unaligned.h>
|
||||
#include <linux/kernel.h>
|
||||
#include <linux/types.h>
|
||||
|
||||
/*
|
||||
* These utilities encode or decode a number from an offset in a larger data buffer and then
|
||||
* advance the offset pointer to the next field in the buffer.
|
||||
*/
|
||||
|
||||
static inline void decode_s64_le(const u8 *buffer, size_t *offset, s64 *decoded)
|
||||
{
|
||||
*decoded = get_unaligned_le64(buffer + *offset);
|
||||
*offset += sizeof(s64);
|
||||
}
|
||||
|
||||
static inline void encode_s64_le(u8 *data, size_t *offset, s64 to_encode)
|
||||
{
|
||||
put_unaligned_le64(to_encode, data + *offset);
|
||||
*offset += sizeof(s64);
|
||||
}
|
||||
|
||||
static inline void decode_u64_le(const u8 *buffer, size_t *offset, u64 *decoded)
|
||||
{
|
||||
*decoded = get_unaligned_le64(buffer + *offset);
|
||||
*offset += sizeof(u64);
|
||||
}
|
||||
|
||||
static inline void encode_u64_le(u8 *data, size_t *offset, u64 to_encode)
|
||||
{
|
||||
put_unaligned_le64(to_encode, data + *offset);
|
||||
*offset += sizeof(u64);
|
||||
}
|
||||
|
||||
static inline void decode_s32_le(const u8 *buffer, size_t *offset, s32 *decoded)
|
||||
{
|
||||
*decoded = get_unaligned_le32(buffer + *offset);
|
||||
*offset += sizeof(s32);
|
||||
}
|
||||
|
||||
static inline void encode_s32_le(u8 *data, size_t *offset, s32 to_encode)
|
||||
{
|
||||
put_unaligned_le32(to_encode, data + *offset);
|
||||
*offset += sizeof(s32);
|
||||
}
|
||||
|
||||
static inline void decode_u32_le(const u8 *buffer, size_t *offset, u32 *decoded)
|
||||
{
|
||||
*decoded = get_unaligned_le32(buffer + *offset);
|
||||
*offset += sizeof(u32);
|
||||
}
|
||||
|
||||
static inline void encode_u32_le(u8 *data, size_t *offset, u32 to_encode)
|
||||
{
|
||||
put_unaligned_le32(to_encode, data + *offset);
|
||||
*offset += sizeof(u32);
|
||||
}
|
||||
|
||||
static inline void decode_u16_le(const u8 *buffer, size_t *offset, u16 *decoded)
|
||||
{
|
||||
*decoded = get_unaligned_le16(buffer + *offset);
|
||||
*offset += sizeof(u16);
|
||||
}
|
||||
|
||||
static inline void encode_u16_le(u8 *data, size_t *offset, u16 to_encode)
|
||||
{
|
||||
put_unaligned_le16(to_encode, data + *offset);
|
||||
*offset += sizeof(u16);
|
||||
}
|
||||
|
||||
#endif /* UDS_NUMERIC_H */
|
|
@ -0,0 +1,780 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "packer.h"
|
||||
|
||||
#include <linux/atomic.h>
|
||||
#include <linux/blkdev.h>
|
||||
|
||||
#include "logger.h"
|
||||
#include "memory-alloc.h"
|
||||
#include "permassert.h"
|
||||
#include "string-utils.h"
|
||||
|
||||
#include "admin-state.h"
|
||||
#include "completion.h"
|
||||
#include "constants.h"
|
||||
#include "data-vio.h"
|
||||
#include "dedupe.h"
|
||||
#include "encodings.h"
|
||||
#include "io-submitter.h"
|
||||
#include "physical-zone.h"
|
||||
#include "status-codes.h"
|
||||
#include "vdo.h"
|
||||
#include "vio.h"
|
||||
|
||||
static const struct version_number COMPRESSED_BLOCK_1_0 = {
|
||||
.major_version = 1,
|
||||
.minor_version = 0,
|
||||
};
|
||||
|
||||
#define COMPRESSED_BLOCK_1_0_SIZE (4 + 4 + (2 * VDO_MAX_COMPRESSION_SLOTS))
|
||||
|
||||
/**
|
||||
* vdo_get_compressed_block_fragment() - Get a reference to a compressed fragment from a compressed
|
||||
* block.
|
||||
* @mapping_state [in] The mapping state for the look up.
|
||||
* @compressed_block [in] The compressed block that was read from disk.
|
||||
* @fragment_offset [out] The offset of the fragment within a compressed block.
|
||||
* @fragment_size [out] The size of the fragment.
|
||||
*
|
||||
* Return: If a valid compressed fragment is found, VDO_SUCCESS; otherwise, VDO_INVALID_FRAGMENT if
|
||||
* the fragment is invalid.
|
||||
*/
|
||||
int vdo_get_compressed_block_fragment(enum block_mapping_state mapping_state,
|
||||
struct compressed_block *block,
|
||||
u16 *fragment_offset, u16 *fragment_size)
|
||||
{
|
||||
u16 compressed_size;
|
||||
u16 offset = 0;
|
||||
unsigned int i;
|
||||
u8 slot;
|
||||
struct version_number version;
|
||||
|
||||
if (!vdo_is_state_compressed(mapping_state))
|
||||
return VDO_INVALID_FRAGMENT;
|
||||
|
||||
version = vdo_unpack_version_number(block->header.version);
|
||||
if (!vdo_are_same_version(version, COMPRESSED_BLOCK_1_0))
|
||||
return VDO_INVALID_FRAGMENT;
|
||||
|
||||
slot = mapping_state - VDO_MAPPING_STATE_COMPRESSED_BASE;
|
||||
if (slot >= VDO_MAX_COMPRESSION_SLOTS)
|
||||
return VDO_INVALID_FRAGMENT;
|
||||
|
||||
compressed_size = __le16_to_cpu(block->header.sizes[slot]);
|
||||
for (i = 0; i < slot; i++) {
|
||||
offset += __le16_to_cpu(block->header.sizes[i]);
|
||||
if (offset >= VDO_COMPRESSED_BLOCK_DATA_SIZE)
|
||||
return VDO_INVALID_FRAGMENT;
|
||||
}
|
||||
|
||||
if ((offset + compressed_size) > VDO_COMPRESSED_BLOCK_DATA_SIZE)
|
||||
return VDO_INVALID_FRAGMENT;
|
||||
|
||||
*fragment_offset = offset;
|
||||
*fragment_size = compressed_size;
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
/**
|
||||
* assert_on_packer_thread() - Check that we are on the packer thread.
|
||||
* @packer: The packer.
|
||||
* @caller: The function which is asserting.
|
||||
*/
|
||||
static inline void assert_on_packer_thread(struct packer *packer, const char *caller)
|
||||
{
|
||||
VDO_ASSERT_LOG_ONLY((vdo_get_callback_thread_id() == packer->thread_id),
|
||||
"%s() called from packer thread", caller);
|
||||
}
|
||||
|
||||
/**
|
||||
* insert_in_sorted_list() - Insert a bin to the list.
|
||||
* @packer: The packer.
|
||||
* @bin: The bin to move to its sorted position.
|
||||
*
|
||||
* The list is in ascending order of free space. Since all bins are already in the list, this
|
||||
* actually moves the bin to the correct position in the list.
|
||||
*/
|
||||
static void insert_in_sorted_list(struct packer *packer, struct packer_bin *bin)
|
||||
{
|
||||
struct packer_bin *active_bin;
|
||||
|
||||
list_for_each_entry(active_bin, &packer->bins, list)
|
||||
if (active_bin->free_space > bin->free_space) {
|
||||
list_move_tail(&bin->list, &active_bin->list);
|
||||
return;
|
||||
}
|
||||
|
||||
list_move_tail(&bin->list, &packer->bins);
|
||||
}
|
||||
|
||||
/**
|
||||
* make_bin() - Allocate a bin and put it into the packer's list.
|
||||
* @packer: The packer.
|
||||
*/
|
||||
static int __must_check make_bin(struct packer *packer)
|
||||
{
|
||||
struct packer_bin *bin;
|
||||
int result;
|
||||
|
||||
result = vdo_allocate_extended(struct packer_bin, VDO_MAX_COMPRESSION_SLOTS,
|
||||
struct vio *, __func__, &bin);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
bin->free_space = VDO_COMPRESSED_BLOCK_DATA_SIZE;
|
||||
INIT_LIST_HEAD(&bin->list);
|
||||
list_add_tail(&bin->list, &packer->bins);
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_make_packer() - Make a new block packer.
|
||||
*
|
||||
* @vdo: The vdo to which this packer belongs.
|
||||
* @bin_count: The number of partial bins to keep in memory.
|
||||
* @packer_ptr: A pointer to hold the new packer.
|
||||
*
|
||||
* Return: VDO_SUCCESS or an error
|
||||
*/
|
||||
int vdo_make_packer(struct vdo *vdo, block_count_t bin_count, struct packer **packer_ptr)
|
||||
{
|
||||
struct packer *packer;
|
||||
block_count_t i;
|
||||
int result;
|
||||
|
||||
result = vdo_allocate(1, struct packer, __func__, &packer);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
packer->thread_id = vdo->thread_config.packer_thread;
|
||||
packer->size = bin_count;
|
||||
INIT_LIST_HEAD(&packer->bins);
|
||||
vdo_set_admin_state_code(&packer->state, VDO_ADMIN_STATE_NORMAL_OPERATION);
|
||||
|
||||
for (i = 0; i < bin_count; i++) {
|
||||
result = make_bin(packer);
|
||||
if (result != VDO_SUCCESS) {
|
||||
vdo_free_packer(packer);
|
||||
return result;
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
* The canceled bin can hold up to half the number of user vios. Every canceled vio in the
|
||||
* bin must have a canceler for which it is waiting, and any canceler will only have
|
||||
* canceled one lock holder at a time.
|
||||
*/
|
||||
result = vdo_allocate_extended(struct packer_bin, MAXIMUM_VDO_USER_VIOS / 2,
|
||||
struct vio *, __func__, &packer->canceled_bin);
|
||||
if (result != VDO_SUCCESS) {
|
||||
vdo_free_packer(packer);
|
||||
return result;
|
||||
}
|
||||
|
||||
result = vdo_make_default_thread(vdo, packer->thread_id);
|
||||
if (result != VDO_SUCCESS) {
|
||||
vdo_free_packer(packer);
|
||||
return result;
|
||||
}
|
||||
|
||||
*packer_ptr = packer;
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_free_packer() - Free a block packer.
|
||||
* @packer: The packer to free.
|
||||
*/
|
||||
void vdo_free_packer(struct packer *packer)
|
||||
{
|
||||
struct packer_bin *bin, *tmp;
|
||||
|
||||
if (packer == NULL)
|
||||
return;
|
||||
|
||||
list_for_each_entry_safe(bin, tmp, &packer->bins, list) {
|
||||
list_del_init(&bin->list);
|
||||
vdo_free(bin);
|
||||
}
|
||||
|
||||
vdo_free(vdo_forget(packer->canceled_bin));
|
||||
vdo_free(packer);
|
||||
}
|
||||
|
||||
/**
|
||||
* get_packer_from_data_vio() - Get the packer from a data_vio.
|
||||
* @data_vio: The data_vio.
|
||||
*
|
||||
* Return: The packer from the VDO to which the data_vio belongs.
|
||||
*/
|
||||
static inline struct packer *get_packer_from_data_vio(struct data_vio *data_vio)
|
||||
{
|
||||
return vdo_from_data_vio(data_vio)->packer;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_get_packer_statistics() - Get the current statistics from the packer.
|
||||
* @packer: The packer to query.
|
||||
*
|
||||
* Return: a copy of the current statistics for the packer.
|
||||
*/
|
||||
struct packer_statistics vdo_get_packer_statistics(const struct packer *packer)
|
||||
{
|
||||
const struct packer_statistics *stats = &packer->statistics;
|
||||
|
||||
return (struct packer_statistics) {
|
||||
.compressed_fragments_written = READ_ONCE(stats->compressed_fragments_written),
|
||||
.compressed_blocks_written = READ_ONCE(stats->compressed_blocks_written),
|
||||
.compressed_fragments_in_packer = READ_ONCE(stats->compressed_fragments_in_packer),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* abort_packing() - Abort packing a data_vio.
|
||||
* @data_vio: The data_vio to abort.
|
||||
*/
|
||||
static void abort_packing(struct data_vio *data_vio)
|
||||
{
|
||||
struct packer *packer = get_packer_from_data_vio(data_vio);
|
||||
|
||||
WRITE_ONCE(packer->statistics.compressed_fragments_in_packer,
|
||||
packer->statistics.compressed_fragments_in_packer - 1);
|
||||
|
||||
write_data_vio(data_vio);
|
||||
}
|
||||
|
||||
/**
|
||||
* release_compressed_write_waiter() - Update a data_vio for which a successful compressed write
|
||||
* has completed and send it on its way.
|
||||
|
||||
* @data_vio: The data_vio to release.
|
||||
* @allocation: The allocation to which the compressed block was written.
|
||||
*/
|
||||
static void release_compressed_write_waiter(struct data_vio *data_vio,
|
||||
struct allocation *allocation)
|
||||
{
|
||||
data_vio->new_mapped = (struct zoned_pbn) {
|
||||
.pbn = allocation->pbn,
|
||||
.zone = allocation->zone,
|
||||
.state = data_vio->compression.slot + VDO_MAPPING_STATE_COMPRESSED_BASE,
|
||||
};
|
||||
|
||||
vdo_share_compressed_write_lock(data_vio, allocation->lock);
|
||||
update_metadata_for_data_vio_write(data_vio, allocation->lock);
|
||||
}
|
||||
|
||||
/**
|
||||
* finish_compressed_write() - Finish a compressed block write.
|
||||
* @completion: The compressed write completion.
|
||||
*
|
||||
* This callback is registered in continue_after_allocation().
|
||||
*/
|
||||
static void finish_compressed_write(struct vdo_completion *completion)
|
||||
{
|
||||
struct data_vio *agent = as_data_vio(completion);
|
||||
struct data_vio *client, *next;
|
||||
|
||||
assert_data_vio_in_allocated_zone(agent);
|
||||
|
||||
/*
|
||||
* Process all the non-agent waiters first to ensure that the pbn lock can not be released
|
||||
* until all of them have had a chance to journal their increfs.
|
||||
*/
|
||||
for (client = agent->compression.next_in_batch; client != NULL; client = next) {
|
||||
next = client->compression.next_in_batch;
|
||||
release_compressed_write_waiter(client, &agent->allocation);
|
||||
}
|
||||
|
||||
completion->error_handler = handle_data_vio_error;
|
||||
release_compressed_write_waiter(agent, &agent->allocation);
|
||||
}
|
||||
|
||||
static void handle_compressed_write_error(struct vdo_completion *completion)
|
||||
{
|
||||
struct data_vio *agent = as_data_vio(completion);
|
||||
struct allocation *allocation = &agent->allocation;
|
||||
struct data_vio *client, *next;
|
||||
|
||||
if (vdo_requeue_completion_if_needed(completion, allocation->zone->thread_id))
|
||||
return;
|
||||
|
||||
update_vio_error_stats(as_vio(completion),
|
||||
"Completing compressed write vio for physical block %llu with error",
|
||||
(unsigned long long) allocation->pbn);
|
||||
|
||||
for (client = agent->compression.next_in_batch; client != NULL; client = next) {
|
||||
next = client->compression.next_in_batch;
|
||||
write_data_vio(client);
|
||||
}
|
||||
|
||||
/* Now that we've released the batch from the packer, forget the error and continue on. */
|
||||
vdo_reset_completion(completion);
|
||||
completion->error_handler = handle_data_vio_error;
|
||||
write_data_vio(agent);
|
||||
}
|
||||
|
||||
/**
|
||||
* add_to_bin() - Put a data_vio in a specific packer_bin in which it will definitely fit.
|
||||
* @bin: The bin in which to put the data_vio.
|
||||
* @data_vio: The data_vio to add.
|
||||
*/
|
||||
static void add_to_bin(struct packer_bin *bin, struct data_vio *data_vio)
|
||||
{
|
||||
data_vio->compression.bin = bin;
|
||||
data_vio->compression.slot = bin->slots_used;
|
||||
bin->incoming[bin->slots_used++] = data_vio;
|
||||
}
|
||||
|
||||
/**
|
||||
* remove_from_bin() - Get the next data_vio whose compression has not been canceled from a bin.
|
||||
* @packer: The packer.
|
||||
* @bin: The bin from which to get a data_vio.
|
||||
*
|
||||
* Any canceled data_vios will be moved to the canceled bin.
|
||||
* Return: An uncanceled data_vio from the bin or NULL if there are none.
|
||||
*/
|
||||
static struct data_vio *remove_from_bin(struct packer *packer, struct packer_bin *bin)
|
||||
{
|
||||
while (bin->slots_used > 0) {
|
||||
struct data_vio *data_vio = bin->incoming[--bin->slots_used];
|
||||
|
||||
if (!advance_data_vio_compression_stage(data_vio).may_not_compress) {
|
||||
data_vio->compression.bin = NULL;
|
||||
return data_vio;
|
||||
}
|
||||
|
||||
add_to_bin(packer->canceled_bin, data_vio);
|
||||
}
|
||||
|
||||
/* The bin is now empty. */
|
||||
bin->free_space = VDO_COMPRESSED_BLOCK_DATA_SIZE;
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/**
|
||||
* initialize_compressed_block() - Initialize a compressed block.
|
||||
* @block: The compressed block to initialize.
|
||||
* @size: The size of the agent's fragment.
|
||||
*
|
||||
* This method initializes the compressed block in the compressed write agent. Because the
|
||||
* compressor already put the agent's compressed fragment at the start of the compressed block's
|
||||
* data field, it needn't be copied. So all we need do is initialize the header and set the size of
|
||||
* the agent's fragment.
|
||||
*/
|
||||
static void initialize_compressed_block(struct compressed_block *block, u16 size)
|
||||
{
|
||||
/*
|
||||
* Make sure the block layout isn't accidentally changed by changing the length of the
|
||||
* block header.
|
||||
*/
|
||||
BUILD_BUG_ON(sizeof(struct compressed_block_header) != COMPRESSED_BLOCK_1_0_SIZE);
|
||||
|
||||
block->header.version = vdo_pack_version_number(COMPRESSED_BLOCK_1_0);
|
||||
block->header.sizes[0] = __cpu_to_le16(size);
|
||||
}
|
||||
|
||||
/**
|
||||
* pack_fragment() - Pack a data_vio's fragment into the compressed block in which it is already
|
||||
* known to fit.
|
||||
* @compression: The agent's compression_state to pack in to.
|
||||
* @data_vio: The data_vio to pack.
|
||||
* @offset: The offset into the compressed block at which to pack the fragment.
|
||||
* @compressed_block: The compressed block which will be written out when batch is fully packed.
|
||||
*
|
||||
* Return: The new amount of space used.
|
||||
*/
|
||||
static block_size_t __must_check pack_fragment(struct compression_state *compression,
|
||||
struct data_vio *data_vio,
|
||||
block_size_t offset, slot_number_t slot,
|
||||
struct compressed_block *block)
|
||||
{
|
||||
struct compression_state *to_pack = &data_vio->compression;
|
||||
char *fragment = to_pack->block->data;
|
||||
|
||||
to_pack->next_in_batch = compression->next_in_batch;
|
||||
compression->next_in_batch = data_vio;
|
||||
to_pack->slot = slot;
|
||||
block->header.sizes[slot] = __cpu_to_le16(to_pack->size);
|
||||
memcpy(&block->data[offset], fragment, to_pack->size);
|
||||
return (offset + to_pack->size);
|
||||
}
|
||||
|
||||
/**
|
||||
* compressed_write_end_io() - The bio_end_io for a compressed block write.
|
||||
* @bio: The bio for the compressed write.
|
||||
*/
|
||||
static void compressed_write_end_io(struct bio *bio)
|
||||
{
|
||||
struct data_vio *data_vio = vio_as_data_vio(bio->bi_private);
|
||||
|
||||
vdo_count_completed_bios(bio);
|
||||
set_data_vio_allocated_zone_callback(data_vio, finish_compressed_write);
|
||||
continue_data_vio_with_error(data_vio, blk_status_to_errno(bio->bi_status));
|
||||
}
|
||||
|
||||
/**
|
||||
* write_bin() - Write out a bin.
|
||||
* @packer: The packer.
|
||||
* @bin: The bin to write.
|
||||
*/
|
||||
static void write_bin(struct packer *packer, struct packer_bin *bin)
|
||||
{
|
||||
int result;
|
||||
block_size_t offset;
|
||||
slot_number_t slot = 1;
|
||||
struct compression_state *compression;
|
||||
struct compressed_block *block;
|
||||
struct data_vio *agent = remove_from_bin(packer, bin);
|
||||
struct data_vio *client;
|
||||
struct packer_statistics *stats;
|
||||
|
||||
if (agent == NULL)
|
||||
return;
|
||||
|
||||
compression = &agent->compression;
|
||||
compression->slot = 0;
|
||||
block = compression->block;
|
||||
initialize_compressed_block(block, compression->size);
|
||||
offset = compression->size;
|
||||
|
||||
while ((client = remove_from_bin(packer, bin)) != NULL)
|
||||
offset = pack_fragment(compression, client, offset, slot++, block);
|
||||
|
||||
/*
|
||||
* If the batch contains only a single vio, then we save nothing by saving the compressed
|
||||
* form. Continue processing the single vio in the batch.
|
||||
*/
|
||||
if (slot == 1) {
|
||||
abort_packing(agent);
|
||||
return;
|
||||
}
|
||||
|
||||
if (slot < VDO_MAX_COMPRESSION_SLOTS) {
|
||||
/* Clear out the sizes of the unused slots */
|
||||
memset(&block->header.sizes[slot], 0,
|
||||
(VDO_MAX_COMPRESSION_SLOTS - slot) * sizeof(__le16));
|
||||
}
|
||||
|
||||
agent->vio.completion.error_handler = handle_compressed_write_error;
|
||||
if (vdo_is_read_only(vdo_from_data_vio(agent))) {
|
||||
continue_data_vio_with_error(agent, VDO_READ_ONLY);
|
||||
return;
|
||||
}
|
||||
|
||||
result = vio_reset_bio(&agent->vio, (char *) block, compressed_write_end_io,
|
||||
REQ_OP_WRITE, agent->allocation.pbn);
|
||||
if (result != VDO_SUCCESS) {
|
||||
continue_data_vio_with_error(agent, result);
|
||||
return;
|
||||
}
|
||||
|
||||
/*
|
||||
* Once the compressed write is submitted, the fragments are no longer in the packer, so
|
||||
* update stats now.
|
||||
*/
|
||||
stats = &packer->statistics;
|
||||
WRITE_ONCE(stats->compressed_fragments_in_packer,
|
||||
(stats->compressed_fragments_in_packer - slot));
|
||||
WRITE_ONCE(stats->compressed_fragments_written,
|
||||
(stats->compressed_fragments_written + slot));
|
||||
WRITE_ONCE(stats->compressed_blocks_written,
|
||||
stats->compressed_blocks_written + 1);
|
||||
|
||||
vdo_submit_data_vio(agent);
|
||||
}
|
||||
|
||||
/**
|
||||
* add_data_vio_to_packer_bin() - Add a data_vio to a bin's incoming queue
|
||||
* @packer: The packer.
|
||||
* @bin: The bin to which to add the data_vio.
|
||||
* @data_vio: The data_vio to add to the bin's queue.
|
||||
*
|
||||
* Adds a data_vio to a bin's incoming queue, handles logical space change, and calls physical
|
||||
* space processor.
|
||||
*/
|
||||
static void add_data_vio_to_packer_bin(struct packer *packer, struct packer_bin *bin,
|
||||
struct data_vio *data_vio)
|
||||
{
|
||||
/* If the selected bin doesn't have room, start a new batch to make room. */
|
||||
if (bin->free_space < data_vio->compression.size)
|
||||
write_bin(packer, bin);
|
||||
|
||||
add_to_bin(bin, data_vio);
|
||||
bin->free_space -= data_vio->compression.size;
|
||||
|
||||
/* If we happen to exactly fill the bin, start a new batch. */
|
||||
if ((bin->slots_used == VDO_MAX_COMPRESSION_SLOTS) ||
|
||||
(bin->free_space == 0))
|
||||
write_bin(packer, bin);
|
||||
|
||||
/* Now that we've finished changing the free space, restore the sort order. */
|
||||
insert_in_sorted_list(packer, bin);
|
||||
}
|
||||
|
||||
/**
|
||||
* select_bin() - Select the bin that should be used to pack the compressed data in a data_vio with
|
||||
* other data_vios.
|
||||
* @packer: The packer.
|
||||
* @data_vio: The data_vio.
|
||||
*/
|
||||
static struct packer_bin * __must_check select_bin(struct packer *packer,
|
||||
struct data_vio *data_vio)
|
||||
{
|
||||
/*
|
||||
* First best fit: select the bin with the least free space that has enough room for the
|
||||
* compressed data in the data_vio.
|
||||
*/
|
||||
struct packer_bin *bin, *fullest_bin;
|
||||
|
||||
list_for_each_entry(bin, &packer->bins, list) {
|
||||
if (bin->free_space >= data_vio->compression.size)
|
||||
return bin;
|
||||
}
|
||||
|
||||
/*
|
||||
* None of the bins have enough space for the data_vio. We're not allowed to create new
|
||||
* bins, so we have to overflow one of the existing bins. It's pretty intuitive to select
|
||||
* the fullest bin, since that "wastes" the least amount of free space in the compressed
|
||||
* block. But if the space currently used in the fullest bin is smaller than the compressed
|
||||
* size of the incoming block, it seems wrong to force that bin to write when giving up on
|
||||
* compressing the incoming data_vio would likewise "waste" the least amount of free space.
|
||||
*/
|
||||
fullest_bin = list_first_entry(&packer->bins, struct packer_bin, list);
|
||||
if (data_vio->compression.size >=
|
||||
(VDO_COMPRESSED_BLOCK_DATA_SIZE - fullest_bin->free_space))
|
||||
return NULL;
|
||||
|
||||
/*
|
||||
* The fullest bin doesn't have room, but writing it out and starting a new batch with the
|
||||
* incoming data_vio will increase the packer's free space.
|
||||
*/
|
||||
return fullest_bin;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_attempt_packing() - Attempt to rewrite the data in this data_vio as part of a compressed
|
||||
* block.
|
||||
* @data_vio: The data_vio to pack.
|
||||
*/
|
||||
void vdo_attempt_packing(struct data_vio *data_vio)
|
||||
{
|
||||
int result;
|
||||
struct packer_bin *bin;
|
||||
struct data_vio_compression_status status = get_data_vio_compression_status(data_vio);
|
||||
struct packer *packer = get_packer_from_data_vio(data_vio);
|
||||
|
||||
assert_on_packer_thread(packer, __func__);
|
||||
|
||||
result = VDO_ASSERT((status.stage == DATA_VIO_COMPRESSING),
|
||||
"attempt to pack data_vio not ready for packing, stage: %u",
|
||||
status.stage);
|
||||
if (result != VDO_SUCCESS)
|
||||
return;
|
||||
|
||||
/*
|
||||
* Increment whether or not this data_vio will be packed or not since abort_packing()
|
||||
* always decrements the counter.
|
||||
*/
|
||||
WRITE_ONCE(packer->statistics.compressed_fragments_in_packer,
|
||||
packer->statistics.compressed_fragments_in_packer + 1);
|
||||
|
||||
/*
|
||||
* If packing of this data_vio is disallowed for administrative reasons, give up before
|
||||
* making any state changes.
|
||||
*/
|
||||
if (!vdo_is_state_normal(&packer->state) ||
|
||||
(data_vio->flush_generation < packer->flush_generation)) {
|
||||
abort_packing(data_vio);
|
||||
return;
|
||||
}
|
||||
|
||||
/*
|
||||
* The advance_data_vio_compression_stage() check here verifies that the data_vio is
|
||||
* allowed to be compressed (if it has already been canceled, we'll fall out here). Once
|
||||
* the data_vio is in the DATA_VIO_PACKING state, it must be guaranteed to be put in a bin
|
||||
* before any more requests can be processed by the packer thread. Otherwise, a canceling
|
||||
* data_vio could attempt to remove the canceled data_vio from the packer and fail to
|
||||
* rendezvous with it. Thus, we must call select_bin() first to ensure that we will
|
||||
* actually add the data_vio to a bin before advancing to the DATA_VIO_PACKING stage.
|
||||
*/
|
||||
bin = select_bin(packer, data_vio);
|
||||
if ((bin == NULL) ||
|
||||
(advance_data_vio_compression_stage(data_vio).stage != DATA_VIO_PACKING)) {
|
||||
abort_packing(data_vio);
|
||||
return;
|
||||
}
|
||||
|
||||
add_data_vio_to_packer_bin(packer, bin, data_vio);
|
||||
}
|
||||
|
||||
/**
|
||||
* check_for_drain_complete() - Check whether the packer has drained.
|
||||
* @packer: The packer.
|
||||
*/
|
||||
static void check_for_drain_complete(struct packer *packer)
|
||||
{
|
||||
if (vdo_is_state_draining(&packer->state) && (packer->canceled_bin->slots_used == 0))
|
||||
vdo_finish_draining(&packer->state);
|
||||
}
|
||||
|
||||
/**
|
||||
* write_all_non_empty_bins() - Write out all non-empty bins on behalf of a flush or suspend.
|
||||
* @packer: The packer being flushed.
|
||||
*/
|
||||
static void write_all_non_empty_bins(struct packer *packer)
|
||||
{
|
||||
struct packer_bin *bin;
|
||||
|
||||
list_for_each_entry(bin, &packer->bins, list)
|
||||
write_bin(packer, bin);
|
||||
/*
|
||||
* We don't need to re-sort the bin here since this loop will make every bin have
|
||||
* the same amount of free space, so every ordering is sorted.
|
||||
*/
|
||||
|
||||
check_for_drain_complete(packer);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_flush_packer() - Request that the packer flush asynchronously.
|
||||
* @packer: The packer to flush.
|
||||
*
|
||||
* All bins with at least two compressed data blocks will be written out, and any solitary pending
|
||||
* VIOs will be released from the packer. While flushing is in progress, any VIOs submitted to
|
||||
* vdo_attempt_packing() will be continued immediately without attempting to pack them.
|
||||
*/
|
||||
void vdo_flush_packer(struct packer *packer)
|
||||
{
|
||||
assert_on_packer_thread(packer, __func__);
|
||||
if (vdo_is_state_normal(&packer->state))
|
||||
write_all_non_empty_bins(packer);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_remove_lock_holder_from_packer() - Remove a lock holder from the packer.
|
||||
* @completion: The data_vio which needs a lock held by a data_vio in the packer. The data_vio's
|
||||
* compression.lock_holder field will point to the data_vio to remove.
|
||||
*/
|
||||
void vdo_remove_lock_holder_from_packer(struct vdo_completion *completion)
|
||||
{
|
||||
struct data_vio *data_vio = as_data_vio(completion);
|
||||
struct packer *packer = get_packer_from_data_vio(data_vio);
|
||||
struct data_vio *lock_holder;
|
||||
struct packer_bin *bin;
|
||||
slot_number_t slot;
|
||||
|
||||
assert_data_vio_in_packer_zone(data_vio);
|
||||
|
||||
lock_holder = vdo_forget(data_vio->compression.lock_holder);
|
||||
bin = lock_holder->compression.bin;
|
||||
VDO_ASSERT_LOG_ONLY((bin != NULL), "data_vio in packer has a bin");
|
||||
|
||||
slot = lock_holder->compression.slot;
|
||||
bin->slots_used--;
|
||||
if (slot < bin->slots_used) {
|
||||
bin->incoming[slot] = bin->incoming[bin->slots_used];
|
||||
bin->incoming[slot]->compression.slot = slot;
|
||||
}
|
||||
|
||||
lock_holder->compression.bin = NULL;
|
||||
lock_holder->compression.slot = 0;
|
||||
|
||||
if (bin != packer->canceled_bin) {
|
||||
bin->free_space += lock_holder->compression.size;
|
||||
insert_in_sorted_list(packer, bin);
|
||||
}
|
||||
|
||||
abort_packing(lock_holder);
|
||||
check_for_drain_complete(packer);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_increment_packer_flush_generation() - Increment the flush generation in the packer.
|
||||
* @packer: The packer.
|
||||
*
|
||||
* This will also cause the packer to flush so that any VIOs from previous generations will exit
|
||||
* the packer.
|
||||
*/
|
||||
void vdo_increment_packer_flush_generation(struct packer *packer)
|
||||
{
|
||||
assert_on_packer_thread(packer, __func__);
|
||||
packer->flush_generation++;
|
||||
vdo_flush_packer(packer);
|
||||
}
|
||||
|
||||
/**
|
||||
* initiate_drain() - Initiate a drain.
|
||||
*
|
||||
* Implements vdo_admin_initiator_fn.
|
||||
*/
|
||||
static void initiate_drain(struct admin_state *state)
|
||||
{
|
||||
struct packer *packer = container_of(state, struct packer, state);
|
||||
|
||||
write_all_non_empty_bins(packer);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_drain_packer() - Drain the packer by preventing any more VIOs from entering the packer and
|
||||
* then flushing.
|
||||
* @packer: The packer to drain.
|
||||
* @completion: The completion to finish when the packer has drained.
|
||||
*/
|
||||
void vdo_drain_packer(struct packer *packer, struct vdo_completion *completion)
|
||||
{
|
||||
assert_on_packer_thread(packer, __func__);
|
||||
vdo_start_draining(&packer->state, VDO_ADMIN_STATE_SUSPENDING, completion,
|
||||
initiate_drain);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_resume_packer() - Resume a packer which has been suspended.
|
||||
* @packer: The packer to resume.
|
||||
* @parent: The completion to finish when the packer has resumed.
|
||||
*/
|
||||
void vdo_resume_packer(struct packer *packer, struct vdo_completion *parent)
|
||||
{
|
||||
assert_on_packer_thread(packer, __func__);
|
||||
vdo_continue_completion(parent, vdo_resume_if_quiescent(&packer->state));
|
||||
}
|
||||
|
||||
static void dump_packer_bin(const struct packer_bin *bin, bool canceled)
|
||||
{
|
||||
if (bin->slots_used == 0)
|
||||
/* Don't dump empty bins. */
|
||||
return;
|
||||
|
||||
vdo_log_info(" %sBin slots_used=%u free_space=%zu",
|
||||
(canceled ? "Canceled" : ""), bin->slots_used, bin->free_space);
|
||||
|
||||
/*
|
||||
* FIXME: dump vios in bin->incoming? The vios should have been dumped from the vio pool.
|
||||
* Maybe just dump their addresses so it's clear they're here?
|
||||
*/
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_dump_packer() - Dump the packer.
|
||||
* @packer: The packer.
|
||||
*
|
||||
* Context: dumps in a thread-unsafe fashion.
|
||||
*/
|
||||
void vdo_dump_packer(const struct packer *packer)
|
||||
{
|
||||
struct packer_bin *bin;
|
||||
|
||||
vdo_log_info("packer");
|
||||
vdo_log_info(" flushGeneration=%llu state %s packer_bin_count=%llu",
|
||||
(unsigned long long) packer->flush_generation,
|
||||
vdo_get_admin_state_code(&packer->state)->name,
|
||||
(unsigned long long) packer->size);
|
||||
|
||||
list_for_each_entry(bin, &packer->bins, list)
|
||||
dump_packer_bin(bin, false);
|
||||
|
||||
dump_packer_bin(packer->canceled_bin, true);
|
||||
}
|
|
@ -0,0 +1,122 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef VDO_PACKER_H
|
||||
#define VDO_PACKER_H
|
||||
|
||||
#include <linux/list.h>
|
||||
|
||||
#include "admin-state.h"
|
||||
#include "constants.h"
|
||||
#include "encodings.h"
|
||||
#include "statistics.h"
|
||||
#include "types.h"
|
||||
#include "wait-queue.h"
|
||||
|
||||
enum {
|
||||
DEFAULT_PACKER_BINS = 16,
|
||||
};
|
||||
|
||||
/* The header of a compressed block. */
|
||||
struct compressed_block_header {
|
||||
/* Unsigned 32-bit major and minor versions, little-endian */
|
||||
struct packed_version_number version;
|
||||
|
||||
/* List of unsigned 16-bit compressed block sizes, little-endian */
|
||||
__le16 sizes[VDO_MAX_COMPRESSION_SLOTS];
|
||||
} __packed;
|
||||
|
||||
enum {
|
||||
VDO_COMPRESSED_BLOCK_DATA_SIZE = VDO_BLOCK_SIZE - sizeof(struct compressed_block_header),
|
||||
|
||||
/*
|
||||
* A compressed block is only written if we can pack at least two fragments into it, so a
|
||||
* fragment which fills the entire data portion of a compressed block is too big.
|
||||
*/
|
||||
VDO_MAX_COMPRESSED_FRAGMENT_SIZE = VDO_COMPRESSED_BLOCK_DATA_SIZE - 1,
|
||||
};
|
||||
|
||||
/* * The compressed block overlay. */
|
||||
struct compressed_block {
|
||||
struct compressed_block_header header;
|
||||
char data[VDO_COMPRESSED_BLOCK_DATA_SIZE];
|
||||
} __packed;
|
||||
|
||||
/*
|
||||
* Each packer_bin holds an incomplete batch of data_vios that only partially fill a compressed
|
||||
* block. The bins are kept in a ring sorted by the amount of unused space so the first bin with
|
||||
* enough space to hold a newly-compressed data_vio can easily be found. When the bin fills up or
|
||||
* is flushed, the first uncanceled data_vio in the bin is selected to be the agent for that bin.
|
||||
* Upon entering the packer, each data_vio already has its compressed data in the first slot of the
|
||||
* data_vio's compressed_block (overlaid on the data_vio's scratch_block). So the agent's fragment
|
||||
* is already in place. The fragments for the other uncanceled data_vios in the bin are packed into
|
||||
* the agent's compressed block. The agent then writes out the compressed block. If the write is
|
||||
* successful, the agent shares its pbn lock which each of the other data_vios in its compressed
|
||||
* block and sends each on its way. Finally the agent itself continues on the write path as before.
|
||||
*
|
||||
* There is one special bin which is used to hold data_vios which have been canceled and removed
|
||||
* from their bin by the packer. These data_vios need to wait for the canceller to rendezvous with
|
||||
* them and so they sit in this special bin.
|
||||
*/
|
||||
struct packer_bin {
|
||||
/* List links for packer.packer_bins */
|
||||
struct list_head list;
|
||||
/* The number of items in the bin */
|
||||
slot_number_t slots_used;
|
||||
/* The number of compressed block bytes remaining in the current batch */
|
||||
size_t free_space;
|
||||
/* The current partial batch of data_vios, waiting for more */
|
||||
struct data_vio *incoming[];
|
||||
};
|
||||
|
||||
struct packer {
|
||||
/* The ID of the packer's callback thread */
|
||||
thread_id_t thread_id;
|
||||
/* The number of bins */
|
||||
block_count_t size;
|
||||
/* A list of all packer_bins, kept sorted by free_space */
|
||||
struct list_head bins;
|
||||
/*
|
||||
* A bin to hold data_vios which were canceled out of the packer and are waiting to
|
||||
* rendezvous with the canceling data_vio.
|
||||
*/
|
||||
struct packer_bin *canceled_bin;
|
||||
|
||||
/* The current flush generation */
|
||||
sequence_number_t flush_generation;
|
||||
|
||||
/* The administrative state of the packer */
|
||||
struct admin_state state;
|
||||
|
||||
/* Statistics are only updated on the packer thread, but are accessed from other threads */
|
||||
struct packer_statistics statistics;
|
||||
};
|
||||
|
||||
int vdo_get_compressed_block_fragment(enum block_mapping_state mapping_state,
|
||||
struct compressed_block *block,
|
||||
u16 *fragment_offset, u16 *fragment_size);
|
||||
|
||||
int __must_check vdo_make_packer(struct vdo *vdo, block_count_t bin_count,
|
||||
struct packer **packer_ptr);
|
||||
|
||||
void vdo_free_packer(struct packer *packer);
|
||||
|
||||
struct packer_statistics __must_check vdo_get_packer_statistics(const struct packer *packer);
|
||||
|
||||
void vdo_attempt_packing(struct data_vio *data_vio);
|
||||
|
||||
void vdo_flush_packer(struct packer *packer);
|
||||
|
||||
void vdo_remove_lock_holder_from_packer(struct vdo_completion *completion);
|
||||
|
||||
void vdo_increment_packer_flush_generation(struct packer *packer);
|
||||
|
||||
void vdo_drain_packer(struct packer *packer, struct vdo_completion *completion);
|
||||
|
||||
void vdo_resume_packer(struct packer *packer, struct vdo_completion *parent);
|
||||
|
||||
void vdo_dump_packer(const struct packer *packer);
|
||||
|
||||
#endif /* VDO_PACKER_H */
|
|
@ -0,0 +1,26 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "permassert.h"
|
||||
|
||||
#include "errors.h"
|
||||
#include "logger.h"
|
||||
|
||||
int vdo_assertion_failed(const char *expression_string, const char *file_name,
|
||||
int line_number, const char *format, ...)
|
||||
{
|
||||
va_list args;
|
||||
|
||||
va_start(args, format);
|
||||
|
||||
vdo_log_embedded_message(VDO_LOG_ERR, VDO_LOGGING_MODULE_NAME, "assertion \"",
|
||||
format, args, "\" (%s) failed at %s:%d",
|
||||
expression_string, file_name, line_number);
|
||||
vdo_log_backtrace(VDO_LOG_ERR);
|
||||
|
||||
va_end(args);
|
||||
|
||||
return UDS_ASSERTION_FAILED;
|
||||
}
|
|
@ -0,0 +1,45 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef PERMASSERT_H
|
||||
#define PERMASSERT_H
|
||||
|
||||
#include <linux/compiler.h>
|
||||
|
||||
#include "errors.h"
|
||||
|
||||
/* Utilities for asserting that certain conditions are met */
|
||||
|
||||
#define STRINGIFY(X) #X
|
||||
|
||||
/*
|
||||
* A hack to apply the "warn if unused" attribute to an integral expression.
|
||||
*
|
||||
* Since GCC doesn't propagate the warn_unused_result attribute to conditional expressions
|
||||
* incorporating calls to functions with that attribute, this function can be used to wrap such an
|
||||
* expression. With optimization enabled, this function contributes no additional instructions, but
|
||||
* the warn_unused_result attribute still applies to the code calling it.
|
||||
*/
|
||||
static inline int __must_check vdo_must_use(int value)
|
||||
{
|
||||
return value;
|
||||
}
|
||||
|
||||
/* Assert that an expression is true and return an error if it is not. */
|
||||
#define VDO_ASSERT(expr, ...) vdo_must_use(__VDO_ASSERT(expr, __VA_ARGS__))
|
||||
|
||||
/* Log a message if the expression is not true. */
|
||||
#define VDO_ASSERT_LOG_ONLY(expr, ...) __VDO_ASSERT(expr, __VA_ARGS__)
|
||||
|
||||
#define __VDO_ASSERT(expr, ...) \
|
||||
(likely(expr) ? VDO_SUCCESS \
|
||||
: vdo_assertion_failed(STRINGIFY(expr), __FILE__, __LINE__, __VA_ARGS__))
|
||||
|
||||
/* Log an assertion failure message. */
|
||||
int vdo_assertion_failed(const char *expression_string, const char *file_name,
|
||||
int line_number, const char *format, ...)
|
||||
__printf(4, 5);
|
||||
|
||||
#endif /* PERMASSERT_H */
|
|
@ -0,0 +1,644 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "physical-zone.h"
|
||||
|
||||
#include <linux/list.h>
|
||||
|
||||
#include "logger.h"
|
||||
#include "memory-alloc.h"
|
||||
#include "permassert.h"
|
||||
|
||||
#include "block-map.h"
|
||||
#include "completion.h"
|
||||
#include "constants.h"
|
||||
#include "data-vio.h"
|
||||
#include "dedupe.h"
|
||||
#include "encodings.h"
|
||||
#include "flush.h"
|
||||
#include "int-map.h"
|
||||
#include "slab-depot.h"
|
||||
#include "status-codes.h"
|
||||
#include "vdo.h"
|
||||
|
||||
/* Each user data_vio needs a PBN read lock and write lock. */
|
||||
#define LOCK_POOL_CAPACITY (2 * MAXIMUM_VDO_USER_VIOS)
|
||||
|
||||
struct pbn_lock_implementation {
|
||||
enum pbn_lock_type type;
|
||||
const char *name;
|
||||
const char *release_reason;
|
||||
};
|
||||
|
||||
/* This array must have an entry for every pbn_lock_type value. */
|
||||
static const struct pbn_lock_implementation LOCK_IMPLEMENTATIONS[] = {
|
||||
[VIO_READ_LOCK] = {
|
||||
.type = VIO_READ_LOCK,
|
||||
.name = "read",
|
||||
.release_reason = "candidate duplicate",
|
||||
},
|
||||
[VIO_WRITE_LOCK] = {
|
||||
.type = VIO_WRITE_LOCK,
|
||||
.name = "write",
|
||||
.release_reason = "newly allocated",
|
||||
},
|
||||
[VIO_BLOCK_MAP_WRITE_LOCK] = {
|
||||
.type = VIO_BLOCK_MAP_WRITE_LOCK,
|
||||
.name = "block map write",
|
||||
.release_reason = "block map write",
|
||||
},
|
||||
};
|
||||
|
||||
static inline bool has_lock_type(const struct pbn_lock *lock, enum pbn_lock_type type)
|
||||
{
|
||||
return (lock->implementation == &LOCK_IMPLEMENTATIONS[type]);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_is_pbn_read_lock() - Check whether a pbn_lock is a read lock.
|
||||
* @lock: The lock to check.
|
||||
*
|
||||
* Return: true if the lock is a read lock.
|
||||
*/
|
||||
bool vdo_is_pbn_read_lock(const struct pbn_lock *lock)
|
||||
{
|
||||
return has_lock_type(lock, VIO_READ_LOCK);
|
||||
}
|
||||
|
||||
static inline void set_pbn_lock_type(struct pbn_lock *lock, enum pbn_lock_type type)
|
||||
{
|
||||
lock->implementation = &LOCK_IMPLEMENTATIONS[type];
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_downgrade_pbn_write_lock() - Downgrade a PBN write lock to a PBN read lock.
|
||||
* @lock: The PBN write lock to downgrade.
|
||||
*
|
||||
* The lock holder count is cleared and the caller is responsible for setting the new count.
|
||||
*/
|
||||
void vdo_downgrade_pbn_write_lock(struct pbn_lock *lock, bool compressed_write)
|
||||
{
|
||||
VDO_ASSERT_LOG_ONLY(!vdo_is_pbn_read_lock(lock),
|
||||
"PBN lock must not already have been downgraded");
|
||||
VDO_ASSERT_LOG_ONLY(!has_lock_type(lock, VIO_BLOCK_MAP_WRITE_LOCK),
|
||||
"must not downgrade block map write locks");
|
||||
VDO_ASSERT_LOG_ONLY(lock->holder_count == 1,
|
||||
"PBN write lock should have one holder but has %u",
|
||||
lock->holder_count);
|
||||
/*
|
||||
* data_vio write locks are downgraded in place--the writer retains the hold on the lock.
|
||||
* If this was a compressed write, the holder has not yet journaled its own inc ref,
|
||||
* otherwise, it has.
|
||||
*/
|
||||
lock->increment_limit =
|
||||
(compressed_write ? MAXIMUM_REFERENCE_COUNT : MAXIMUM_REFERENCE_COUNT - 1);
|
||||
set_pbn_lock_type(lock, VIO_READ_LOCK);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_claim_pbn_lock_increment() - Try to claim one of the available reference count increments on
|
||||
* a read lock.
|
||||
* @lock: The PBN read lock from which to claim an increment.
|
||||
*
|
||||
* Claims may be attempted from any thread. A claim is only valid until the PBN lock is released.
|
||||
*
|
||||
* Return: true if the claim succeeded, guaranteeing one increment can be made without overflowing
|
||||
* the PBN's reference count.
|
||||
*/
|
||||
bool vdo_claim_pbn_lock_increment(struct pbn_lock *lock)
|
||||
{
|
||||
/*
|
||||
* Claim the next free reference atomically since hash locks from multiple hash zone
|
||||
* threads might be concurrently deduplicating against a single PBN lock on compressed
|
||||
* block. As long as hitting the increment limit will lead to the PBN lock being released
|
||||
* in a sane time-frame, we won't overflow a 32-bit claim counter, allowing a simple add
|
||||
* instead of a compare-and-swap.
|
||||
*/
|
||||
u32 claim_number = (u32) atomic_add_return(1, &lock->increments_claimed);
|
||||
|
||||
return (claim_number <= lock->increment_limit);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_assign_pbn_lock_provisional_reference() - Inform a PBN lock that it is responsible for a
|
||||
* provisional reference.
|
||||
* @lock: The PBN lock.
|
||||
*/
|
||||
void vdo_assign_pbn_lock_provisional_reference(struct pbn_lock *lock)
|
||||
{
|
||||
VDO_ASSERT_LOG_ONLY(!lock->has_provisional_reference,
|
||||
"lock does not have a provisional reference");
|
||||
lock->has_provisional_reference = true;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_unassign_pbn_lock_provisional_reference() - Inform a PBN lock that it is no longer
|
||||
* responsible for a provisional reference.
|
||||
* @lock: The PBN lock.
|
||||
*/
|
||||
void vdo_unassign_pbn_lock_provisional_reference(struct pbn_lock *lock)
|
||||
{
|
||||
lock->has_provisional_reference = false;
|
||||
}
|
||||
|
||||
/**
|
||||
* release_pbn_lock_provisional_reference() - If the lock is responsible for a provisional
|
||||
* reference, release that reference.
|
||||
* @lock: The lock.
|
||||
* @locked_pbn: The PBN covered by the lock.
|
||||
* @allocator: The block allocator from which to release the reference.
|
||||
*
|
||||
* This method is called when the lock is released.
|
||||
*/
|
||||
static void release_pbn_lock_provisional_reference(struct pbn_lock *lock,
|
||||
physical_block_number_t locked_pbn,
|
||||
struct block_allocator *allocator)
|
||||
{
|
||||
int result;
|
||||
|
||||
if (!vdo_pbn_lock_has_provisional_reference(lock))
|
||||
return;
|
||||
|
||||
result = vdo_release_block_reference(allocator, locked_pbn);
|
||||
if (result != VDO_SUCCESS) {
|
||||
vdo_log_error_strerror(result,
|
||||
"Failed to release reference to %s physical block %llu",
|
||||
lock->implementation->release_reason,
|
||||
(unsigned long long) locked_pbn);
|
||||
}
|
||||
|
||||
vdo_unassign_pbn_lock_provisional_reference(lock);
|
||||
}
|
||||
|
||||
/**
|
||||
* union idle_pbn_lock - PBN lock list entries.
|
||||
*
|
||||
* Unused (idle) PBN locks are kept in a list. Just like in a malloc implementation, the lock
|
||||
* structure is unused memory, so we can save a bit of space (and not pollute the lock structure
|
||||
* proper) by using a union to overlay the lock structure with the free list.
|
||||
*/
|
||||
typedef union {
|
||||
/** @entry: Only used while locks are in the pool. */
|
||||
struct list_head entry;
|
||||
/** @lock: Only used while locks are not in the pool. */
|
||||
struct pbn_lock lock;
|
||||
} idle_pbn_lock;
|
||||
|
||||
/**
|
||||
* struct pbn_lock_pool - list of PBN locks.
|
||||
*
|
||||
* The lock pool is little more than the memory allocated for the locks.
|
||||
*/
|
||||
struct pbn_lock_pool {
|
||||
/** @capacity: The number of locks allocated for the pool. */
|
||||
size_t capacity;
|
||||
/** @borrowed: The number of locks currently borrowed from the pool. */
|
||||
size_t borrowed;
|
||||
/** @idle_list: A list containing all idle PBN lock instances. */
|
||||
struct list_head idle_list;
|
||||
/** @locks: The memory for all the locks allocated by this pool. */
|
||||
idle_pbn_lock locks[];
|
||||
};
|
||||
|
||||
/**
|
||||
* return_pbn_lock_to_pool() - Return a pbn lock to its pool.
|
||||
* @pool: The pool from which the lock was borrowed.
|
||||
* @lock: The last reference to the lock being returned.
|
||||
*
|
||||
* It must be the last live reference, as if the memory were being freed (the lock memory will
|
||||
* re-initialized or zeroed).
|
||||
*/
|
||||
static void return_pbn_lock_to_pool(struct pbn_lock_pool *pool, struct pbn_lock *lock)
|
||||
{
|
||||
idle_pbn_lock *idle;
|
||||
|
||||
/* A bit expensive, but will promptly catch some use-after-free errors. */
|
||||
memset(lock, 0, sizeof(*lock));
|
||||
|
||||
idle = container_of(lock, idle_pbn_lock, lock);
|
||||
INIT_LIST_HEAD(&idle->entry);
|
||||
list_add_tail(&idle->entry, &pool->idle_list);
|
||||
|
||||
VDO_ASSERT_LOG_ONLY(pool->borrowed > 0, "shouldn't return more than borrowed");
|
||||
pool->borrowed -= 1;
|
||||
}
|
||||
|
||||
/**
|
||||
* make_pbn_lock_pool() - Create a new PBN lock pool and all the lock instances it can loan out.
|
||||
*
|
||||
* @capacity: The number of PBN locks to allocate for the pool.
|
||||
* @pool_ptr: A pointer to receive the new pool.
|
||||
*
|
||||
* Return: VDO_SUCCESS or an error code.
|
||||
*/
|
||||
static int make_pbn_lock_pool(size_t capacity, struct pbn_lock_pool **pool_ptr)
|
||||
{
|
||||
size_t i;
|
||||
struct pbn_lock_pool *pool;
|
||||
int result;
|
||||
|
||||
result = vdo_allocate_extended(struct pbn_lock_pool, capacity, idle_pbn_lock,
|
||||
__func__, &pool);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
pool->capacity = capacity;
|
||||
pool->borrowed = capacity;
|
||||
INIT_LIST_HEAD(&pool->idle_list);
|
||||
|
||||
for (i = 0; i < capacity; i++)
|
||||
return_pbn_lock_to_pool(pool, &pool->locks[i].lock);
|
||||
|
||||
*pool_ptr = pool;
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
/**
|
||||
* free_pbn_lock_pool() - Free a PBN lock pool.
|
||||
* @pool: The lock pool to free.
|
||||
*
|
||||
* This also frees all the PBN locks it allocated, so the caller must ensure that all locks have
|
||||
* been returned to the pool.
|
||||
*/
|
||||
static void free_pbn_lock_pool(struct pbn_lock_pool *pool)
|
||||
{
|
||||
if (pool == NULL)
|
||||
return;
|
||||
|
||||
VDO_ASSERT_LOG_ONLY(pool->borrowed == 0,
|
||||
"All PBN locks must be returned to the pool before it is freed, but %zu locks are still on loan",
|
||||
pool->borrowed);
|
||||
vdo_free(pool);
|
||||
}
|
||||
|
||||
/**
|
||||
* borrow_pbn_lock_from_pool() - Borrow a PBN lock from the pool and initialize it with the
|
||||
* provided type.
|
||||
* @pool: The pool from which to borrow.
|
||||
* @type: The type with which to initialize the lock.
|
||||
* @lock_ptr: A pointer to receive the borrowed lock.
|
||||
*
|
||||
* Pools do not grow on demand or allocate memory, so this will fail if the pool is empty. Borrowed
|
||||
* locks are still associated with this pool and must be returned to only this pool.
|
||||
*
|
||||
* Return: VDO_SUCCESS, or VDO_LOCK_ERROR if the pool is empty.
|
||||
*/
|
||||
static int __must_check borrow_pbn_lock_from_pool(struct pbn_lock_pool *pool,
|
||||
enum pbn_lock_type type,
|
||||
struct pbn_lock **lock_ptr)
|
||||
{
|
||||
int result;
|
||||
struct list_head *idle_entry;
|
||||
idle_pbn_lock *idle;
|
||||
|
||||
if (pool->borrowed >= pool->capacity)
|
||||
return vdo_log_error_strerror(VDO_LOCK_ERROR,
|
||||
"no free PBN locks left to borrow");
|
||||
pool->borrowed += 1;
|
||||
|
||||
result = VDO_ASSERT(!list_empty(&pool->idle_list),
|
||||
"idle list should not be empty if pool not at capacity");
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
idle_entry = pool->idle_list.prev;
|
||||
list_del(idle_entry);
|
||||
memset(idle_entry, 0, sizeof(*idle_entry));
|
||||
|
||||
idle = list_entry(idle_entry, idle_pbn_lock, entry);
|
||||
idle->lock.holder_count = 0;
|
||||
set_pbn_lock_type(&idle->lock, type);
|
||||
|
||||
*lock_ptr = &idle->lock;
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
/**
|
||||
* initialize_zone() - Initialize a physical zone.
|
||||
* @vdo: The vdo to which the zone will belong.
|
||||
* @zones: The physical_zones to which the zone being initialized belongs
|
||||
*
|
||||
* Return: VDO_SUCCESS or an error code.
|
||||
*/
|
||||
static int initialize_zone(struct vdo *vdo, struct physical_zones *zones)
|
||||
{
|
||||
int result;
|
||||
zone_count_t zone_number = zones->zone_count;
|
||||
struct physical_zone *zone = &zones->zones[zone_number];
|
||||
|
||||
result = vdo_int_map_create(VDO_LOCK_MAP_CAPACITY, &zone->pbn_operations);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
result = make_pbn_lock_pool(LOCK_POOL_CAPACITY, &zone->lock_pool);
|
||||
if (result != VDO_SUCCESS) {
|
||||
vdo_int_map_free(zone->pbn_operations);
|
||||
return result;
|
||||
}
|
||||
|
||||
zone->zone_number = zone_number;
|
||||
zone->thread_id = vdo->thread_config.physical_threads[zone_number];
|
||||
zone->allocator = &vdo->depot->allocators[zone_number];
|
||||
zone->next = &zones->zones[(zone_number + 1) % vdo->thread_config.physical_zone_count];
|
||||
result = vdo_make_default_thread(vdo, zone->thread_id);
|
||||
if (result != VDO_SUCCESS) {
|
||||
free_pbn_lock_pool(vdo_forget(zone->lock_pool));
|
||||
vdo_int_map_free(zone->pbn_operations);
|
||||
return result;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_make_physical_zones() - Make the physical zones for a vdo.
|
||||
* @vdo: The vdo being constructed
|
||||
* @zones_ptr: A pointer to hold the zones
|
||||
*
|
||||
* Return: VDO_SUCCESS or an error code.
|
||||
*/
|
||||
int vdo_make_physical_zones(struct vdo *vdo, struct physical_zones **zones_ptr)
|
||||
{
|
||||
struct physical_zones *zones;
|
||||
int result;
|
||||
zone_count_t zone_count = vdo->thread_config.physical_zone_count;
|
||||
|
||||
if (zone_count == 0)
|
||||
return VDO_SUCCESS;
|
||||
|
||||
result = vdo_allocate_extended(struct physical_zones, zone_count,
|
||||
struct physical_zone, __func__, &zones);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
for (zones->zone_count = 0; zones->zone_count < zone_count; zones->zone_count++) {
|
||||
result = initialize_zone(vdo, zones);
|
||||
if (result != VDO_SUCCESS) {
|
||||
vdo_free_physical_zones(zones);
|
||||
return result;
|
||||
}
|
||||
}
|
||||
|
||||
*zones_ptr = zones;
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_free_physical_zones() - Destroy the physical zones.
|
||||
* @zones: The zones to free.
|
||||
*/
|
||||
void vdo_free_physical_zones(struct physical_zones *zones)
|
||||
{
|
||||
zone_count_t index;
|
||||
|
||||
if (zones == NULL)
|
||||
return;
|
||||
|
||||
for (index = 0; index < zones->zone_count; index++) {
|
||||
struct physical_zone *zone = &zones->zones[index];
|
||||
|
||||
free_pbn_lock_pool(vdo_forget(zone->lock_pool));
|
||||
vdo_int_map_free(vdo_forget(zone->pbn_operations));
|
||||
}
|
||||
|
||||
vdo_free(zones);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_get_physical_zone_pbn_lock() - Get the lock on a PBN if one exists.
|
||||
* @zone: The physical zone responsible for the PBN.
|
||||
* @pbn: The physical block number whose lock is desired.
|
||||
*
|
||||
* Return: The lock or NULL if the PBN is not locked.
|
||||
*/
|
||||
struct pbn_lock *vdo_get_physical_zone_pbn_lock(struct physical_zone *zone,
|
||||
physical_block_number_t pbn)
|
||||
{
|
||||
return ((zone == NULL) ? NULL : vdo_int_map_get(zone->pbn_operations, pbn));
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_attempt_physical_zone_pbn_lock() - Attempt to lock a physical block in the zone responsible
|
||||
* for it.
|
||||
* @zone: The physical zone responsible for the PBN.
|
||||
* @pbn: The physical block number to lock.
|
||||
* @type: The type with which to initialize a new lock.
|
||||
* @lock_ptr: A pointer to receive the lock, existing or new.
|
||||
*
|
||||
* If the PBN is already locked, the existing lock will be returned. Otherwise, a new lock instance
|
||||
* will be borrowed from the pool, initialized, and returned. The lock owner will be NULL for a new
|
||||
* lock acquired by the caller, who is responsible for setting that field promptly. The lock owner
|
||||
* will be non-NULL when there is already an existing lock on the PBN.
|
||||
*
|
||||
* Return: VDO_SUCCESS or an error.
|
||||
*/
|
||||
int vdo_attempt_physical_zone_pbn_lock(struct physical_zone *zone,
|
||||
physical_block_number_t pbn,
|
||||
enum pbn_lock_type type,
|
||||
struct pbn_lock **lock_ptr)
|
||||
{
|
||||
/*
|
||||
* Borrow and prepare a lock from the pool so we don't have to do two int_map accesses in
|
||||
* the common case of no lock contention.
|
||||
*/
|
||||
struct pbn_lock *lock, *new_lock = NULL;
|
||||
int result;
|
||||
|
||||
result = borrow_pbn_lock_from_pool(zone->lock_pool, type, &new_lock);
|
||||
if (result != VDO_SUCCESS) {
|
||||
VDO_ASSERT_LOG_ONLY(false, "must always be able to borrow a PBN lock");
|
||||
return result;
|
||||
}
|
||||
|
||||
result = vdo_int_map_put(zone->pbn_operations, pbn, new_lock, false,
|
||||
(void **) &lock);
|
||||
if (result != VDO_SUCCESS) {
|
||||
return_pbn_lock_to_pool(zone->lock_pool, new_lock);
|
||||
return result;
|
||||
}
|
||||
|
||||
if (lock != NULL) {
|
||||
/* The lock is already held, so we don't need the borrowed one. */
|
||||
return_pbn_lock_to_pool(zone->lock_pool, vdo_forget(new_lock));
|
||||
result = VDO_ASSERT(lock->holder_count > 0, "physical block %llu lock held",
|
||||
(unsigned long long) pbn);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
*lock_ptr = lock;
|
||||
} else {
|
||||
*lock_ptr = new_lock;
|
||||
}
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
/**
|
||||
* allocate_and_lock_block() - Attempt to allocate a block from this zone.
|
||||
* @allocation: The struct allocation of the data_vio attempting to allocate.
|
||||
*
|
||||
* If a block is allocated, the recipient will also hold a lock on it.
|
||||
*
|
||||
* Return: VDO_SUCCESS if a block was allocated, or an error code.
|
||||
*/
|
||||
static int allocate_and_lock_block(struct allocation *allocation)
|
||||
{
|
||||
int result;
|
||||
struct pbn_lock *lock;
|
||||
|
||||
VDO_ASSERT_LOG_ONLY(allocation->lock == NULL,
|
||||
"must not allocate a block while already holding a lock on one");
|
||||
|
||||
result = vdo_allocate_block(allocation->zone->allocator, &allocation->pbn);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
result = vdo_attempt_physical_zone_pbn_lock(allocation->zone, allocation->pbn,
|
||||
allocation->write_lock_type, &lock);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
if (lock->holder_count > 0) {
|
||||
/* This block is already locked, which should be impossible. */
|
||||
return vdo_log_error_strerror(VDO_LOCK_ERROR,
|
||||
"Newly allocated block %llu was spuriously locked (holder_count=%u)",
|
||||
(unsigned long long) allocation->pbn,
|
||||
lock->holder_count);
|
||||
}
|
||||
|
||||
/* We've successfully acquired a new lock, so mark it as ours. */
|
||||
lock->holder_count += 1;
|
||||
allocation->lock = lock;
|
||||
vdo_assign_pbn_lock_provisional_reference(lock);
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
/**
|
||||
* retry_allocation() - Retry allocating a block now that we're done waiting for scrubbing.
|
||||
* @waiter: The allocating_vio that was waiting to allocate.
|
||||
* @context: The context (unused).
|
||||
*/
|
||||
static void retry_allocation(struct vdo_waiter *waiter, void *context __always_unused)
|
||||
{
|
||||
struct data_vio *data_vio = vdo_waiter_as_data_vio(waiter);
|
||||
|
||||
/* Now that some slab has scrubbed, restart the allocation process. */
|
||||
data_vio->allocation.wait_for_clean_slab = false;
|
||||
data_vio->allocation.first_allocation_zone = data_vio->allocation.zone->zone_number;
|
||||
continue_data_vio(data_vio);
|
||||
}
|
||||
|
||||
/**
|
||||
* continue_allocating() - Continue searching for an allocation by enqueuing to wait for scrubbing
|
||||
* or switching to the next zone.
|
||||
* @data_vio: The data_vio attempting to get an allocation.
|
||||
*
|
||||
* This method should only be called from the error handler set in data_vio_allocate_data_block.
|
||||
*
|
||||
* Return: true if the allocation process has continued in another zone.
|
||||
*/
|
||||
static bool continue_allocating(struct data_vio *data_vio)
|
||||
{
|
||||
struct allocation *allocation = &data_vio->allocation;
|
||||
struct physical_zone *zone = allocation->zone;
|
||||
struct vdo_completion *completion = &data_vio->vio.completion;
|
||||
int result = VDO_SUCCESS;
|
||||
bool was_waiting = allocation->wait_for_clean_slab;
|
||||
bool tried_all = (allocation->first_allocation_zone == zone->next->zone_number);
|
||||
|
||||
vdo_reset_completion(completion);
|
||||
|
||||
if (tried_all && !was_waiting) {
|
||||
/*
|
||||
* We've already looked in all the zones, and found nothing. So go through the
|
||||
* zones again, and wait for each to scrub before trying to allocate.
|
||||
*/
|
||||
allocation->wait_for_clean_slab = true;
|
||||
allocation->first_allocation_zone = zone->zone_number;
|
||||
}
|
||||
|
||||
if (allocation->wait_for_clean_slab) {
|
||||
data_vio->waiter.callback = retry_allocation;
|
||||
result = vdo_enqueue_clean_slab_waiter(zone->allocator,
|
||||
&data_vio->waiter);
|
||||
if (result == VDO_SUCCESS) {
|
||||
/* We've enqueued to wait for a slab to be scrubbed. */
|
||||
return true;
|
||||
}
|
||||
|
||||
if ((result != VDO_NO_SPACE) || (was_waiting && tried_all)) {
|
||||
vdo_set_completion_result(completion, result);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
allocation->zone = zone->next;
|
||||
completion->callback_thread_id = allocation->zone->thread_id;
|
||||
vdo_launch_completion(completion);
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_allocate_block_in_zone() - Attempt to allocate a block in the current physical zone, and if
|
||||
* that fails try the next if possible.
|
||||
* @data_vio: The data_vio needing an allocation.
|
||||
*
|
||||
* Return: true if a block was allocated, if not the data_vio will have been dispatched so the
|
||||
* caller must not touch it.
|
||||
*/
|
||||
bool vdo_allocate_block_in_zone(struct data_vio *data_vio)
|
||||
{
|
||||
int result = allocate_and_lock_block(&data_vio->allocation);
|
||||
|
||||
if (result == VDO_SUCCESS)
|
||||
return true;
|
||||
|
||||
if ((result != VDO_NO_SPACE) || !continue_allocating(data_vio))
|
||||
continue_data_vio_with_error(data_vio, result);
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_release_physical_zone_pbn_lock() - Release a physical block lock if it is held and return it
|
||||
* to the lock pool.
|
||||
* @zone: The physical zone in which the lock was obtained.
|
||||
* @locked_pbn: The physical block number to unlock.
|
||||
* @lock: The lock being released.
|
||||
*
|
||||
* It must be the last live reference, as if the memory were being freed (the
|
||||
* lock memory will re-initialized or zeroed).
|
||||
*/
|
||||
void vdo_release_physical_zone_pbn_lock(struct physical_zone *zone,
|
||||
physical_block_number_t locked_pbn,
|
||||
struct pbn_lock *lock)
|
||||
{
|
||||
struct pbn_lock *holder;
|
||||
|
||||
if (lock == NULL)
|
||||
return;
|
||||
|
||||
VDO_ASSERT_LOG_ONLY(lock->holder_count > 0,
|
||||
"should not be releasing a lock that is not held");
|
||||
|
||||
lock->holder_count -= 1;
|
||||
if (lock->holder_count > 0) {
|
||||
/* The lock was shared and is still referenced, so don't release it yet. */
|
||||
return;
|
||||
}
|
||||
|
||||
holder = vdo_int_map_remove(zone->pbn_operations, locked_pbn);
|
||||
VDO_ASSERT_LOG_ONLY((lock == holder), "physical block lock mismatch for block %llu",
|
||||
(unsigned long long) locked_pbn);
|
||||
|
||||
release_pbn_lock_provisional_reference(lock, locked_pbn, zone->allocator);
|
||||
return_pbn_lock_to_pool(zone->lock_pool, lock);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_dump_physical_zone() - Dump information about a physical zone to the log for debugging.
|
||||
* @zone: The zone to dump.
|
||||
*/
|
||||
void vdo_dump_physical_zone(const struct physical_zone *zone)
|
||||
{
|
||||
vdo_dump_block_allocator(zone->allocator);
|
||||
}
|
|
@ -0,0 +1,115 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef VDO_PHYSICAL_ZONE_H
|
||||
#define VDO_PHYSICAL_ZONE_H
|
||||
|
||||
#include <linux/atomic.h>
|
||||
|
||||
#include "types.h"
|
||||
|
||||
/*
|
||||
* The type of a PBN lock.
|
||||
*/
|
||||
enum pbn_lock_type {
|
||||
VIO_READ_LOCK,
|
||||
VIO_WRITE_LOCK,
|
||||
VIO_BLOCK_MAP_WRITE_LOCK,
|
||||
};
|
||||
|
||||
struct pbn_lock_implementation;
|
||||
|
||||
/*
|
||||
* A PBN lock.
|
||||
*/
|
||||
struct pbn_lock {
|
||||
/* The implementation of the lock */
|
||||
const struct pbn_lock_implementation *implementation;
|
||||
|
||||
/* The number of VIOs holding or sharing this lock */
|
||||
data_vio_count_t holder_count;
|
||||
/*
|
||||
* The number of compressed block writers holding a share of this lock while they are
|
||||
* acquiring a reference to the PBN.
|
||||
*/
|
||||
u8 fragment_locks;
|
||||
|
||||
/* Whether the locked PBN has been provisionally referenced on behalf of the lock holder. */
|
||||
bool has_provisional_reference;
|
||||
|
||||
/*
|
||||
* For read locks, the number of references that were known to be available on the locked
|
||||
* block at the time the lock was acquired.
|
||||
*/
|
||||
u8 increment_limit;
|
||||
|
||||
/*
|
||||
* For read locks, the number of data_vios that have tried to claim one of the available
|
||||
* increments during the lifetime of the lock. Each claim will first increment this
|
||||
* counter, so it can exceed the increment limit.
|
||||
*/
|
||||
atomic_t increments_claimed;
|
||||
};
|
||||
|
||||
struct physical_zone {
|
||||
/* Which physical zone this is */
|
||||
zone_count_t zone_number;
|
||||
/* The thread ID for this zone */
|
||||
thread_id_t thread_id;
|
||||
/* In progress operations keyed by PBN */
|
||||
struct int_map *pbn_operations;
|
||||
/* Pool of unused pbn_lock instances */
|
||||
struct pbn_lock_pool *lock_pool;
|
||||
/* The block allocator for this zone */
|
||||
struct block_allocator *allocator;
|
||||
/* The next zone from which to attempt an allocation */
|
||||
struct physical_zone *next;
|
||||
};
|
||||
|
||||
struct physical_zones {
|
||||
/* The number of zones */
|
||||
zone_count_t zone_count;
|
||||
/* The physical zones themselves */
|
||||
struct physical_zone zones[];
|
||||
};
|
||||
|
||||
bool __must_check vdo_is_pbn_read_lock(const struct pbn_lock *lock);
|
||||
void vdo_downgrade_pbn_write_lock(struct pbn_lock *lock, bool compressed_write);
|
||||
bool __must_check vdo_claim_pbn_lock_increment(struct pbn_lock *lock);
|
||||
|
||||
/**
|
||||
* vdo_pbn_lock_has_provisional_reference() - Check whether a PBN lock has a provisional reference.
|
||||
* @lock: The PBN lock.
|
||||
*/
|
||||
static inline bool vdo_pbn_lock_has_provisional_reference(struct pbn_lock *lock)
|
||||
{
|
||||
return ((lock != NULL) && lock->has_provisional_reference);
|
||||
}
|
||||
|
||||
void vdo_assign_pbn_lock_provisional_reference(struct pbn_lock *lock);
|
||||
void vdo_unassign_pbn_lock_provisional_reference(struct pbn_lock *lock);
|
||||
|
||||
int __must_check vdo_make_physical_zones(struct vdo *vdo,
|
||||
struct physical_zones **zones_ptr);
|
||||
|
||||
void vdo_free_physical_zones(struct physical_zones *zones);
|
||||
|
||||
struct pbn_lock * __must_check vdo_get_physical_zone_pbn_lock(struct physical_zone *zone,
|
||||
physical_block_number_t pbn);
|
||||
|
||||
int __must_check vdo_attempt_physical_zone_pbn_lock(struct physical_zone *zone,
|
||||
physical_block_number_t pbn,
|
||||
enum pbn_lock_type type,
|
||||
struct pbn_lock **lock_ptr);
|
||||
|
||||
bool __must_check vdo_allocate_block_in_zone(struct data_vio *data_vio);
|
||||
|
||||
void vdo_release_physical_zone_pbn_lock(struct physical_zone *zone,
|
||||
physical_block_number_t locked_pbn,
|
||||
struct pbn_lock *lock);
|
||||
|
||||
void vdo_dump_physical_zone(const struct physical_zone *zone);
|
||||
|
||||
#endif /* VDO_PHYSICAL_ZONE_H */
|
|
@ -0,0 +1,224 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "priority-table.h"
|
||||
|
||||
#include <linux/log2.h>
|
||||
|
||||
#include "errors.h"
|
||||
#include "memory-alloc.h"
|
||||
#include "permassert.h"
|
||||
|
||||
#include "status-codes.h"
|
||||
|
||||
/* We use a single 64-bit search vector, so the maximum priority is 63 */
|
||||
#define MAX_PRIORITY 63
|
||||
|
||||
/*
|
||||
* All the entries with the same priority are queued in a circular list in a bucket for that
|
||||
* priority. The table is essentially an array of buckets.
|
||||
*/
|
||||
struct bucket {
|
||||
/*
|
||||
* The head of a queue of table entries, all having the same priority
|
||||
*/
|
||||
struct list_head queue;
|
||||
/* The priority of all the entries in this bucket */
|
||||
unsigned int priority;
|
||||
};
|
||||
|
||||
/*
|
||||
* A priority table is an array of buckets, indexed by priority. New entries are added to the end
|
||||
* of the queue in the appropriate bucket. The dequeue operation finds the highest-priority
|
||||
* non-empty bucket by searching a bit vector represented as a single 8-byte word, which is very
|
||||
* fast with compiler and CPU support.
|
||||
*/
|
||||
struct priority_table {
|
||||
/* The maximum priority of entries that may be stored in this table */
|
||||
unsigned int max_priority;
|
||||
/* A bit vector flagging all buckets that are currently non-empty */
|
||||
u64 search_vector;
|
||||
/* The array of all buckets, indexed by priority */
|
||||
struct bucket buckets[];
|
||||
};
|
||||
|
||||
/**
|
||||
* vdo_make_priority_table() - Allocate and initialize a new priority_table.
|
||||
* @max_priority: The maximum priority value for table entries.
|
||||
* @table_ptr: A pointer to hold the new table.
|
||||
*
|
||||
* Return: VDO_SUCCESS or an error code.
|
||||
*/
|
||||
int vdo_make_priority_table(unsigned int max_priority, struct priority_table **table_ptr)
|
||||
{
|
||||
struct priority_table *table;
|
||||
int result;
|
||||
unsigned int priority;
|
||||
|
||||
if (max_priority > MAX_PRIORITY)
|
||||
return UDS_INVALID_ARGUMENT;
|
||||
|
||||
result = vdo_allocate_extended(struct priority_table, max_priority + 1,
|
||||
struct bucket, __func__, &table);
|
||||
if (result != VDO_SUCCESS)
|
||||
return result;
|
||||
|
||||
for (priority = 0; priority <= max_priority; priority++) {
|
||||
struct bucket *bucket = &table->buckets[priority];
|
||||
|
||||
bucket->priority = priority;
|
||||
INIT_LIST_HEAD(&bucket->queue);
|
||||
}
|
||||
|
||||
table->max_priority = max_priority;
|
||||
table->search_vector = 0;
|
||||
|
||||
*table_ptr = table;
|
||||
return VDO_SUCCESS;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_free_priority_table() - Free a priority_table.
|
||||
* @table: The table to free.
|
||||
*
|
||||
* The table does not own the entries stored in it and they are not freed by this call.
|
||||
*/
|
||||
void vdo_free_priority_table(struct priority_table *table)
|
||||
{
|
||||
if (table == NULL)
|
||||
return;
|
||||
|
||||
/*
|
||||
* Unlink the buckets from any entries still in the table so the entries won't be left with
|
||||
* dangling pointers to freed memory.
|
||||
*/
|
||||
vdo_reset_priority_table(table);
|
||||
|
||||
vdo_free(table);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_reset_priority_table() - Reset a priority table, leaving it in the same empty state as when
|
||||
* newly constructed.
|
||||
* @table: The table to reset.
|
||||
*
|
||||
* The table does not own the entries stored in it and they are not freed (or even unlinked from
|
||||
* each other) by this call.
|
||||
*/
|
||||
void vdo_reset_priority_table(struct priority_table *table)
|
||||
{
|
||||
unsigned int priority;
|
||||
|
||||
table->search_vector = 0;
|
||||
for (priority = 0; priority <= table->max_priority; priority++)
|
||||
list_del_init(&table->buckets[priority].queue);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_priority_table_enqueue() - Add a new entry to the priority table, appending it to the queue
|
||||
* for entries with the specified priority.
|
||||
* @table: The table in which to store the entry.
|
||||
* @priority: The priority of the entry.
|
||||
* @entry: The list_head embedded in the entry to store in the table (the caller must have
|
||||
* initialized it).
|
||||
*/
|
||||
void vdo_priority_table_enqueue(struct priority_table *table, unsigned int priority,
|
||||
struct list_head *entry)
|
||||
{
|
||||
VDO_ASSERT_LOG_ONLY((priority <= table->max_priority),
|
||||
"entry priority must be valid for the table");
|
||||
|
||||
/* Append the entry to the queue in the specified bucket. */
|
||||
list_move_tail(entry, &table->buckets[priority].queue);
|
||||
|
||||
/* Flag the bucket in the search vector since it must be non-empty. */
|
||||
table->search_vector |= (1ULL << priority);
|
||||
}
|
||||
|
||||
static inline void mark_bucket_empty(struct priority_table *table, struct bucket *bucket)
|
||||
{
|
||||
table->search_vector &= ~(1ULL << bucket->priority);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_priority_table_dequeue() - Find the highest-priority entry in the table, remove it from the
|
||||
* table, and return it.
|
||||
* @table: The priority table from which to remove an entry.
|
||||
*
|
||||
* If there are multiple entries with the same priority, the one that has been in the table with
|
||||
* that priority the longest will be returned.
|
||||
*
|
||||
* Return: The dequeued entry, or NULL if the table is currently empty.
|
||||
*/
|
||||
struct list_head *vdo_priority_table_dequeue(struct priority_table *table)
|
||||
{
|
||||
struct bucket *bucket;
|
||||
struct list_head *entry;
|
||||
int top_priority;
|
||||
|
||||
if (table->search_vector == 0) {
|
||||
/* All buckets are empty. */
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/*
|
||||
* Find the highest priority non-empty bucket by finding the highest-order non-zero bit in
|
||||
* the search vector.
|
||||
*/
|
||||
top_priority = ilog2(table->search_vector);
|
||||
|
||||
/* Dequeue the first entry in the bucket. */
|
||||
bucket = &table->buckets[top_priority];
|
||||
entry = bucket->queue.next;
|
||||
list_del_init(entry);
|
||||
|
||||
/* Clear the bit in the search vector if the bucket has been emptied. */
|
||||
if (list_empty(&bucket->queue))
|
||||
mark_bucket_empty(table, bucket);
|
||||
|
||||
return entry;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_priority_table_remove() - Remove a specified entry from its priority table.
|
||||
* @table: The table from which to remove the entry.
|
||||
* @entry: The entry to remove from the table.
|
||||
*/
|
||||
void vdo_priority_table_remove(struct priority_table *table, struct list_head *entry)
|
||||
{
|
||||
struct list_head *next_entry;
|
||||
|
||||
/*
|
||||
* We can't guard against calls where the entry is on a list for a different table, but
|
||||
* it's easy to deal with an entry not in any table or list.
|
||||
*/
|
||||
if (list_empty(entry))
|
||||
return;
|
||||
|
||||
/*
|
||||
* Remove the entry from the bucket list, remembering a pointer to another entry in the
|
||||
* ring.
|
||||
*/
|
||||
next_entry = entry->next;
|
||||
list_del_init(entry);
|
||||
|
||||
/*
|
||||
* If the rest of the list is now empty, the next node must be the list head in the bucket
|
||||
* and we can use it to update the search vector.
|
||||
*/
|
||||
if (list_empty(next_entry))
|
||||
mark_bucket_empty(table, list_entry(next_entry, struct bucket, queue));
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_is_priority_table_empty() - Return whether the priority table is empty.
|
||||
* @table: The table to check.
|
||||
*
|
||||
* Return: true if the table is empty.
|
||||
*/
|
||||
bool vdo_is_priority_table_empty(struct priority_table *table)
|
||||
{
|
||||
return (table->search_vector == 0);
|
||||
}
|
|
@ -0,0 +1,47 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef VDO_PRIORITY_TABLE_H
|
||||
#define VDO_PRIORITY_TABLE_H
|
||||
|
||||
#include <linux/list.h>
|
||||
|
||||
/*
|
||||
* A priority_table is a simple implementation of a priority queue for entries with priorities that
|
||||
* are small non-negative integer values. It implements the obvious priority queue operations of
|
||||
* enqueuing an entry and dequeuing an entry with the maximum priority. It also supports removing
|
||||
* an arbitrary entry. The priority of an entry already in the table can be changed by removing it
|
||||
* and re-enqueuing it with a different priority. All operations have O(1) complexity.
|
||||
*
|
||||
* The links for the table entries must be embedded in the entries themselves. Lists are used to
|
||||
* link entries in the table and no wrapper type is declared, so an existing list entry in an
|
||||
* object can also be used to queue it in a priority_table, assuming the field is not used for
|
||||
* anything else while so queued.
|
||||
*
|
||||
* The table is implemented as an array of queues (circular lists) indexed by priority, along with
|
||||
* a hint for which queues are non-empty. Steven Skiena calls a very similar structure a "bounded
|
||||
* height priority queue", but given the resemblance to a hash table, "priority table" seems both
|
||||
* shorter and more apt, if somewhat novel.
|
||||
*/
|
||||
|
||||
struct priority_table;
|
||||
|
||||
int __must_check vdo_make_priority_table(unsigned int max_priority,
|
||||
struct priority_table **table_ptr);
|
||||
|
||||
void vdo_free_priority_table(struct priority_table *table);
|
||||
|
||||
void vdo_priority_table_enqueue(struct priority_table *table, unsigned int priority,
|
||||
struct list_head *entry);
|
||||
|
||||
void vdo_reset_priority_table(struct priority_table *table);
|
||||
|
||||
struct list_head * __must_check vdo_priority_table_dequeue(struct priority_table *table);
|
||||
|
||||
void vdo_priority_table_remove(struct priority_table *table, struct list_head *entry);
|
||||
|
||||
bool __must_check vdo_is_priority_table_empty(struct priority_table *table);
|
||||
|
||||
#endif /* VDO_PRIORITY_TABLE_H */
|
File diff suppressed because it is too large
Load Diff
|
@ -0,0 +1,316 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef VDO_RECOVERY_JOURNAL_H
|
||||
#define VDO_RECOVERY_JOURNAL_H
|
||||
|
||||
#include <linux/list.h>
|
||||
|
||||
#include "numeric.h"
|
||||
|
||||
#include "admin-state.h"
|
||||
#include "constants.h"
|
||||
#include "encodings.h"
|
||||
#include "flush.h"
|
||||
#include "statistics.h"
|
||||
#include "types.h"
|
||||
#include "wait-queue.h"
|
||||
|
||||
/**
|
||||
* DOC: recovery journal.
|
||||
*
|
||||
* The recovery_journal provides a log of all block mapping and reference count changes which have
|
||||
* not yet been stably written to the block map or slab journals. This log helps to reduce the
|
||||
* write amplification of writes by providing amortization of slab journal and block map page
|
||||
* updates.
|
||||
*
|
||||
* The recovery journal has a single dedicated queue and thread for performing all journal updates.
|
||||
* The concurrency guarantees of this single-threaded model allow the code to omit more
|
||||
* fine-grained locking for recovery journal structures.
|
||||
*
|
||||
* The journal consists of a set of on-disk blocks arranged as a circular log with monotonically
|
||||
* increasing sequence numbers. Three sequence numbers serve to define the active extent of the
|
||||
* journal. The 'head' is the oldest active block in the journal. The 'tail' is the end of the
|
||||
* half-open interval containing the active blocks. 'active' is the number of the block actively
|
||||
* receiving entries. In an empty journal, head == active == tail. Once any entries are added, tail
|
||||
* = active + 1, and head may be any value in the interval [tail - size, active].
|
||||
*
|
||||
* The journal also contains a set of in-memory blocks which are used to buffer up entries until
|
||||
* they can be committed. In general the number of in-memory blocks ('tail_buffer_count') will be
|
||||
* less than the on-disk size. Each in-memory block is also a vdo_completion. Each in-memory block
|
||||
* has a vio which is used to commit that block to disk. The vio's data is the on-disk
|
||||
* representation of the journal block. In addition each in-memory block has a buffer which is used
|
||||
* to accumulate entries while a partial commit of the block is in progress. In-memory blocks are
|
||||
* kept on two rings. Free blocks live on the 'free_tail_blocks' ring. When a block becomes active
|
||||
* (see below) it is moved to the 'active_tail_blocks' ring. When a block is fully committed, it is
|
||||
* moved back to the 'free_tail_blocks' ring.
|
||||
*
|
||||
* When entries are added to the journal, they are added to the active in-memory block, as
|
||||
* indicated by the 'active_block' field. If the caller wishes to wait for the entry to be
|
||||
* committed, the requesting VIO will be attached to the in-memory block to which the caller's
|
||||
* entry was added. If the caller does wish to wait, or if the entry filled the active block, an
|
||||
* attempt will be made to commit that block to disk. If there is already another commit in
|
||||
* progress, the attempt will be ignored and then automatically retried when the in-progress commit
|
||||
* completes. If there is no commit in progress, any data_vios waiting on the block are transferred
|
||||
* to the block's vio which is then written, automatically waking all of the waiters when it
|
||||
* completes. When the write completes, any entries which accumulated in the block are copied to
|
||||
* the vio's data buffer.
|
||||
*
|
||||
* Finally, the journal maintains a set of counters, one for each on disk journal block. These
|
||||
* counters are used as locks to prevent premature reaping of journal blocks. Each time a new
|
||||
* sequence number is used, the counter for the corresponding block is incremented. The counter is
|
||||
* subsequently decremented when that block is filled and then committed for the last time. This
|
||||
* prevents blocks from being reaped while they are still being updated. The counter is also
|
||||
* incremented once for each entry added to a block, and decremented once each time the block map
|
||||
* is updated in memory for that request. This prevents blocks from being reaped while their VIOs
|
||||
* are still active. Finally, each in-memory block map page tracks the oldest journal block that
|
||||
* contains entries corresponding to uncommitted updates to that block map page. Each time an
|
||||
* in-memory block map page is updated, it checks if the journal block for the VIO is earlier than
|
||||
* the one it references, in which case it increments the count on the earlier journal block and
|
||||
* decrements the count on the later journal block, maintaining a lock on the oldest journal block
|
||||
* containing entries for that page. When a block map page has been flushed from the cache, the
|
||||
* counter for the journal block it references is decremented. Whenever the counter for the head
|
||||
* block goes to 0, the head is advanced until it comes to a block whose counter is not 0 or until
|
||||
* it reaches the active block. This is the mechanism for reclaiming journal space on disk.
|
||||
*
|
||||
* If there is no in-memory space when a VIO attempts to add an entry, the VIO will be attached to
|
||||
* the 'commit_completion' and will be woken the next time a full block has committed. If there is
|
||||
* no on-disk space when a VIO attempts to add an entry, the VIO will be attached to the
|
||||
* 'reap_completion', and will be woken the next time a journal block is reaped.
|
||||
*/
|
||||
|
||||
enum vdo_zone_type {
|
||||
VDO_ZONE_TYPE_ADMIN,
|
||||
VDO_ZONE_TYPE_JOURNAL,
|
||||
VDO_ZONE_TYPE_LOGICAL,
|
||||
VDO_ZONE_TYPE_PHYSICAL,
|
||||
};
|
||||
|
||||
struct lock_counter {
|
||||
/* The completion for notifying the owner of a lock release */
|
||||
struct vdo_completion completion;
|
||||
/* The number of logical zones which may hold locks */
|
||||
zone_count_t logical_zones;
|
||||
/* The number of physical zones which may hold locks */
|
||||
zone_count_t physical_zones;
|
||||
/* The number of locks */
|
||||
block_count_t locks;
|
||||
/* Whether the lock release notification is in flight */
|
||||
atomic_t state;
|
||||
/* The number of logical zones which hold each lock */
|
||||
atomic_t *logical_zone_counts;
|
||||
/* The number of physical zones which hold each lock */
|
||||
atomic_t *physical_zone_counts;
|
||||
/* The per-lock counts for the journal zone */
|
||||
u16 *journal_counters;
|
||||
/* The per-lock decrement counts for the journal zone */
|
||||
atomic_t *journal_decrement_counts;
|
||||
/* The per-zone, per-lock reference counts for logical zones */
|
||||
u16 *logical_counters;
|
||||
/* The per-zone, per-lock reference counts for physical zones */
|
||||
u16 *physical_counters;
|
||||
};
|
||||
|
||||
struct recovery_journal_block {
|
||||
/* The doubly linked pointers for the free or active lists */
|
||||
struct list_head list_node;
|
||||
/* The waiter for the pending full block list */
|
||||
struct vdo_waiter write_waiter;
|
||||
/* The journal to which this block belongs */
|
||||
struct recovery_journal *journal;
|
||||
/* A pointer to the current sector in the packed block buffer */
|
||||
struct packed_journal_sector *sector;
|
||||
/* The vio for writing this block */
|
||||
struct vio vio;
|
||||
/* The sequence number for this block */
|
||||
sequence_number_t sequence_number;
|
||||
/* The location of this block in the on-disk journal */
|
||||
physical_block_number_t block_number;
|
||||
/* Whether this block is being committed */
|
||||
bool committing;
|
||||
/* The total number of entries in this block */
|
||||
journal_entry_count_t entry_count;
|
||||
/* The total number of uncommitted entries (queued or committing) */
|
||||
journal_entry_count_t uncommitted_entry_count;
|
||||
/* The number of new entries in the current commit */
|
||||
journal_entry_count_t entries_in_commit;
|
||||
/* The queue of vios which will make entries for the next commit */
|
||||
struct vdo_wait_queue entry_waiters;
|
||||
/* The queue of vios waiting for the current commit */
|
||||
struct vdo_wait_queue commit_waiters;
|
||||
};
|
||||
|
||||
struct recovery_journal {
|
||||
/* The thread ID of the journal zone */
|
||||
thread_id_t thread_id;
|
||||
/* The slab depot which can hold locks on this journal */
|
||||
struct slab_depot *depot;
|
||||
/* The block map which can hold locks on this journal */
|
||||
struct block_map *block_map;
|
||||
/* The queue of vios waiting to make entries */
|
||||
struct vdo_wait_queue entry_waiters;
|
||||
/* The number of free entries in the journal */
|
||||
u64 available_space;
|
||||
/* The number of decrement entries which need to be made */
|
||||
data_vio_count_t pending_decrement_count;
|
||||
/* Whether the journal is adding entries from the increment or decrement waiters queues */
|
||||
bool adding_entries;
|
||||
/* The administrative state of the journal */
|
||||
struct admin_state state;
|
||||
/* Whether a reap is in progress */
|
||||
bool reaping;
|
||||
/* The location of the first journal block */
|
||||
physical_block_number_t origin;
|
||||
/* The oldest active block in the journal on disk for block map rebuild */
|
||||
sequence_number_t block_map_head;
|
||||
/* The oldest active block in the journal on disk for slab journal replay */
|
||||
sequence_number_t slab_journal_head;
|
||||
/* The newest block in the journal on disk to which a write has finished */
|
||||
sequence_number_t last_write_acknowledged;
|
||||
/* The end of the half-open interval of the active journal */
|
||||
sequence_number_t tail;
|
||||
/* The point at which the last entry will have been added */
|
||||
struct journal_point append_point;
|
||||
/* The journal point of the vio most recently released from the journal */
|
||||
struct journal_point commit_point;
|
||||
/* The nonce of the VDO */
|
||||
nonce_t nonce;
|
||||
/* The number of recoveries completed by the VDO */
|
||||
u8 recovery_count;
|
||||
/* The number of entries which fit in a single block */
|
||||
journal_entry_count_t entries_per_block;
|
||||
/* Unused in-memory journal blocks */
|
||||
struct list_head free_tail_blocks;
|
||||
/* In-memory journal blocks with records */
|
||||
struct list_head active_tail_blocks;
|
||||
/* A pointer to the active block (the one we are adding entries to now) */
|
||||
struct recovery_journal_block *active_block;
|
||||
/* Journal blocks that need writing */
|
||||
struct vdo_wait_queue pending_writes;
|
||||
/* The new block map reap head after reaping */
|
||||
sequence_number_t block_map_reap_head;
|
||||
/* The head block number for the block map rebuild range */
|
||||
block_count_t block_map_head_block_number;
|
||||
/* The new slab journal reap head after reaping */
|
||||
sequence_number_t slab_journal_reap_head;
|
||||
/* The head block number for the slab journal replay range */
|
||||
block_count_t slab_journal_head_block_number;
|
||||
/* The data-less vio, usable only for flushing */
|
||||
struct vio *flush_vio;
|
||||
/* The number of blocks in the on-disk journal */
|
||||
block_count_t size;
|
||||
/* The number of logical blocks that are in-use */
|
||||
block_count_t logical_blocks_used;
|
||||
/* The number of block map pages that are allocated */
|
||||
block_count_t block_map_data_blocks;
|
||||
/* The number of journal blocks written but not yet acknowledged */
|
||||
block_count_t pending_write_count;
|
||||
/* The threshold at which slab journal tail blocks will be written out */
|
||||
block_count_t slab_journal_commit_threshold;
|
||||
/* Counters for events in the journal that are reported as statistics */
|
||||
struct recovery_journal_statistics events;
|
||||
/* The locks for each on-disk block */
|
||||
struct lock_counter lock_counter;
|
||||
/* The tail blocks */
|
||||
struct recovery_journal_block blocks[];
|
||||
};
|
||||
|
||||
/**
|
||||
* vdo_get_recovery_journal_block_number() - Get the physical block number for a given sequence
|
||||
* number.
|
||||
* @journal: The journal.
|
||||
* @sequence: The sequence number of the desired block.
|
||||
*
|
||||
* Return: The block number corresponding to the sequence number.
|
||||
*/
|
||||
static inline physical_block_number_t __must_check
|
||||
vdo_get_recovery_journal_block_number(const struct recovery_journal *journal,
|
||||
sequence_number_t sequence)
|
||||
{
|
||||
/*
|
||||
* Since journal size is a power of two, the block number modulus can just be extracted
|
||||
* from the low-order bits of the sequence.
|
||||
*/
|
||||
return vdo_compute_recovery_journal_block_number(journal->size, sequence);
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_compute_recovery_journal_check_byte() - Compute the check byte for a given sequence number.
|
||||
* @journal: The journal.
|
||||
* @sequence: The sequence number.
|
||||
*
|
||||
* Return: The check byte corresponding to the sequence number.
|
||||
*/
|
||||
static inline u8 __must_check
|
||||
vdo_compute_recovery_journal_check_byte(const struct recovery_journal *journal,
|
||||
sequence_number_t sequence)
|
||||
{
|
||||
/* The check byte must change with each trip around the journal. */
|
||||
return (((sequence / journal->size) & 0x7F) | 0x80);
|
||||
}
|
||||
|
||||
int __must_check vdo_decode_recovery_journal(struct recovery_journal_state_7_0 state,
|
||||
nonce_t nonce, struct vdo *vdo,
|
||||
struct partition *partition,
|
||||
u64 recovery_count,
|
||||
block_count_t journal_size,
|
||||
struct recovery_journal **journal_ptr);
|
||||
|
||||
void vdo_free_recovery_journal(struct recovery_journal *journal);
|
||||
|
||||
void vdo_initialize_recovery_journal_post_repair(struct recovery_journal *journal,
|
||||
u64 recovery_count,
|
||||
sequence_number_t tail,
|
||||
block_count_t logical_blocks_used,
|
||||
block_count_t block_map_data_blocks);
|
||||
|
||||
block_count_t __must_check
|
||||
vdo_get_journal_block_map_data_blocks_used(struct recovery_journal *journal);
|
||||
|
||||
thread_id_t __must_check vdo_get_recovery_journal_thread_id(struct recovery_journal *journal);
|
||||
|
||||
void vdo_open_recovery_journal(struct recovery_journal *journal,
|
||||
struct slab_depot *depot, struct block_map *block_map);
|
||||
|
||||
sequence_number_t
|
||||
vdo_get_recovery_journal_current_sequence_number(struct recovery_journal *journal);
|
||||
|
||||
block_count_t __must_check vdo_get_recovery_journal_length(block_count_t journal_size);
|
||||
|
||||
struct recovery_journal_state_7_0 __must_check
|
||||
vdo_record_recovery_journal(const struct recovery_journal *journal);
|
||||
|
||||
void vdo_add_recovery_journal_entry(struct recovery_journal *journal,
|
||||
struct data_vio *data_vio);
|
||||
|
||||
void vdo_acquire_recovery_journal_block_reference(struct recovery_journal *journal,
|
||||
sequence_number_t sequence_number,
|
||||
enum vdo_zone_type zone_type,
|
||||
zone_count_t zone_id);
|
||||
|
||||
void vdo_release_recovery_journal_block_reference(struct recovery_journal *journal,
|
||||
sequence_number_t sequence_number,
|
||||
enum vdo_zone_type zone_type,
|
||||
zone_count_t zone_id);
|
||||
|
||||
void vdo_release_journal_entry_lock(struct recovery_journal *journal,
|
||||
sequence_number_t sequence_number);
|
||||
|
||||
void vdo_drain_recovery_journal(struct recovery_journal *journal,
|
||||
const struct admin_state_code *operation,
|
||||
struct vdo_completion *parent);
|
||||
|
||||
void vdo_resume_recovery_journal(struct recovery_journal *journal,
|
||||
struct vdo_completion *parent);
|
||||
|
||||
block_count_t __must_check
|
||||
vdo_get_recovery_journal_logical_blocks_used(const struct recovery_journal *journal);
|
||||
|
||||
struct recovery_journal_statistics __must_check
|
||||
vdo_get_recovery_journal_statistics(const struct recovery_journal *journal);
|
||||
|
||||
void vdo_dump_recovery_journal_statistics(const struct recovery_journal *journal);
|
||||
|
||||
#endif /* VDO_RECOVERY_JOURNAL_H */
|
File diff suppressed because it is too large
Load Diff
|
@ -0,0 +1,14 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef VDO_REPAIR_H
|
||||
#define VDO_REPAIR_H
|
||||
|
||||
#include "types.h"
|
||||
|
||||
void vdo_replay_into_slab_journals(struct block_allocator *allocator, void *context);
|
||||
void vdo_repair(struct vdo_completion *parent);
|
||||
|
||||
#endif /* VDO_REPAIR_H */
|
File diff suppressed because it is too large
Load Diff
|
@ -0,0 +1,601 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef VDO_SLAB_DEPOT_H
|
||||
#define VDO_SLAB_DEPOT_H
|
||||
|
||||
#include <linux/atomic.h>
|
||||
#include <linux/dm-kcopyd.h>
|
||||
#include <linux/list.h>
|
||||
|
||||
#include "numeric.h"
|
||||
|
||||
#include "admin-state.h"
|
||||
#include "completion.h"
|
||||
#include "data-vio.h"
|
||||
#include "encodings.h"
|
||||
#include "physical-zone.h"
|
||||
#include "priority-table.h"
|
||||
#include "recovery-journal.h"
|
||||
#include "statistics.h"
|
||||
#include "types.h"
|
||||
#include "vio.h"
|
||||
#include "wait-queue.h"
|
||||
|
||||
/*
|
||||
* A slab_depot is responsible for managing all of the slabs and block allocators of a VDO. It has
|
||||
* a single array of slabs in order to eliminate the need for additional math in order to compute
|
||||
* which physical zone a PBN is in. It also has a block_allocator per zone.
|
||||
*
|
||||
* Each physical zone has a single dedicated queue and thread for performing all updates to the
|
||||
* slabs assigned to that zone. The concurrency guarantees of this single-threaded model allow the
|
||||
* code to omit more fine-grained locking for the various slab structures. Each physical zone
|
||||
* maintains a separate copy of the slab summary to remove the need for explicit locking on that
|
||||
* structure as well.
|
||||
*
|
||||
* Load operations must be performed on the admin thread. Normal operations, such as allocations
|
||||
* and reference count updates, must be performed on the appropriate physical zone thread. Requests
|
||||
* from the recovery journal to commit slab journal tail blocks must be scheduled from the recovery
|
||||
* journal thread to run on the appropriate physical zone thread. Save operations must be launched
|
||||
* from the same admin thread as the original load operation.
|
||||
*/
|
||||
|
||||
enum {
|
||||
/* The number of vios in the vio pool is proportional to the throughput of the VDO. */
|
||||
BLOCK_ALLOCATOR_VIO_POOL_SIZE = 128,
|
||||
};
|
||||
|
||||
/*
|
||||
* Represents the possible status of a block.
|
||||
*/
|
||||
enum reference_status {
|
||||
RS_FREE, /* this block is free */
|
||||
RS_SINGLE, /* this block is singly-referenced */
|
||||
RS_SHARED, /* this block is shared */
|
||||
RS_PROVISIONAL /* this block is provisionally allocated */
|
||||
};
|
||||
|
||||
struct vdo_slab;
|
||||
|
||||
struct journal_lock {
|
||||
u16 count;
|
||||
sequence_number_t recovery_start;
|
||||
};
|
||||
|
||||
struct slab_journal {
|
||||
/* A waiter object for getting a VIO pool entry */
|
||||
struct vdo_waiter resource_waiter;
|
||||
/* A waiter object for updating the slab summary */
|
||||
struct vdo_waiter slab_summary_waiter;
|
||||
/* A waiter object for getting a vio with which to flush */
|
||||
struct vdo_waiter flush_waiter;
|
||||
/* The queue of VIOs waiting to make an entry */
|
||||
struct vdo_wait_queue entry_waiters;
|
||||
/* The parent slab reference of this journal */
|
||||
struct vdo_slab *slab;
|
||||
|
||||
/* Whether a tail block commit is pending */
|
||||
bool waiting_to_commit;
|
||||
/* Whether the journal is updating the slab summary */
|
||||
bool updating_slab_summary;
|
||||
/* Whether the journal is adding entries from the entry_waiters queue */
|
||||
bool adding_entries;
|
||||
/* Whether a partial write is in progress */
|
||||
bool partial_write_in_progress;
|
||||
|
||||
/* The oldest block in the journal on disk */
|
||||
sequence_number_t head;
|
||||
/* The oldest block in the journal which may not be reaped */
|
||||
sequence_number_t unreapable;
|
||||
/* The end of the half-open interval of the active journal */
|
||||
sequence_number_t tail;
|
||||
/* The next journal block to be committed */
|
||||
sequence_number_t next_commit;
|
||||
/* The tail sequence number that is written in the slab summary */
|
||||
sequence_number_t summarized;
|
||||
/* The tail sequence number that was last summarized in slab summary */
|
||||
sequence_number_t last_summarized;
|
||||
|
||||
/* The sequence number of the recovery journal lock */
|
||||
sequence_number_t recovery_lock;
|
||||
|
||||
/*
|
||||
* The number of entries which fit in a single block. Can't use the constant because unit
|
||||
* tests change this number.
|
||||
*/
|
||||
journal_entry_count_t entries_per_block;
|
||||
/*
|
||||
* The number of full entries which fit in a single block. Can't use the constant because
|
||||
* unit tests change this number.
|
||||
*/
|
||||
journal_entry_count_t full_entries_per_block;
|
||||
|
||||
/* The recovery journal of the VDO (slab journal holds locks on it) */
|
||||
struct recovery_journal *recovery_journal;
|
||||
|
||||
/* The statistics shared by all slab journals in our physical zone */
|
||||
struct slab_journal_statistics *events;
|
||||
/* A list of the VIO pool entries for outstanding journal block writes */
|
||||
struct list_head uncommitted_blocks;
|
||||
|
||||
/*
|
||||
* The current tail block header state. This will be packed into the block just before it
|
||||
* is written.
|
||||
*/
|
||||
struct slab_journal_block_header tail_header;
|
||||
/* A pointer to a block-sized buffer holding the packed block data */
|
||||
struct packed_slab_journal_block *block;
|
||||
|
||||
/* The number of blocks in the on-disk journal */
|
||||
block_count_t size;
|
||||
/* The number of blocks at which to start pushing reference blocks */
|
||||
block_count_t flushing_threshold;
|
||||
/* The number of blocks at which all reference blocks should be writing */
|
||||
block_count_t flushing_deadline;
|
||||
/* The number of blocks at which to wait for reference blocks to write */
|
||||
block_count_t blocking_threshold;
|
||||
/* The number of blocks at which to scrub the slab before coming online */
|
||||
block_count_t scrubbing_threshold;
|
||||
|
||||
/* This list entry is for block_allocator to keep a queue of dirty journals */
|
||||
struct list_head dirty_entry;
|
||||
|
||||
/* The lock for the oldest unreaped block of the journal */
|
||||
struct journal_lock *reap_lock;
|
||||
/* The locks for each on disk block */
|
||||
struct journal_lock *locks;
|
||||
};
|
||||
|
||||
/*
|
||||
* Reference_block structure
|
||||
*
|
||||
* Blocks are used as a proxy, permitting saves of partial refcounts.
|
||||
*/
|
||||
struct reference_block {
|
||||
/* This block waits on the ref_counts to tell it to write */
|
||||
struct vdo_waiter waiter;
|
||||
/* The slab to which this reference_block belongs */
|
||||
struct vdo_slab *slab;
|
||||
/* The number of references in this block that represent allocations */
|
||||
block_size_t allocated_count;
|
||||
/* The slab journal block on which this block must hold a lock */
|
||||
sequence_number_t slab_journal_lock;
|
||||
/* The slab journal block which should be released when this block is committed */
|
||||
sequence_number_t slab_journal_lock_to_release;
|
||||
/* The point up to which each sector is accurate on disk */
|
||||
struct journal_point commit_points[VDO_SECTORS_PER_BLOCK];
|
||||
/* Whether this block has been modified since it was written to disk */
|
||||
bool is_dirty;
|
||||
/* Whether this block is currently writing */
|
||||
bool is_writing;
|
||||
};
|
||||
|
||||
/* The search_cursor represents the saved position of a free block search. */
|
||||
struct search_cursor {
|
||||
/* The reference block containing the current search index */
|
||||
struct reference_block *block;
|
||||
/* The position at which to start searching for the next free counter */
|
||||
slab_block_number index;
|
||||
/* The position just past the last valid counter in the current block */
|
||||
slab_block_number end_index;
|
||||
|
||||
/* A pointer to the first reference block in the slab */
|
||||
struct reference_block *first_block;
|
||||
/* A pointer to the last reference block in the slab */
|
||||
struct reference_block *last_block;
|
||||
};
|
||||
|
||||
enum slab_rebuild_status {
|
||||
VDO_SLAB_REBUILT,
|
||||
VDO_SLAB_REPLAYING,
|
||||
VDO_SLAB_REQUIRES_SCRUBBING,
|
||||
VDO_SLAB_REQUIRES_HIGH_PRIORITY_SCRUBBING,
|
||||
VDO_SLAB_REBUILDING,
|
||||
};
|
||||
|
||||
/*
|
||||
* This is the type declaration for the vdo_slab type. A vdo_slab currently consists of a run of
|
||||
* 2^23 data blocks, but that will soon change to dedicate a small number of those blocks for
|
||||
* metadata storage for the reference counts and slab journal for the slab.
|
||||
*
|
||||
* A reference count is maintained for each physical block number. The vast majority of blocks have
|
||||
* a very small reference count (usually 0 or 1). For references less than or equal to MAXIMUM_REFS
|
||||
* (254) the reference count is stored in counters[pbn].
|
||||
*/
|
||||
struct vdo_slab {
|
||||
/* A list entry to queue this slab in a block_allocator list */
|
||||
struct list_head allocq_entry;
|
||||
|
||||
/* The struct block_allocator that owns this slab */
|
||||
struct block_allocator *allocator;
|
||||
|
||||
/* The journal for this slab */
|
||||
struct slab_journal journal;
|
||||
|
||||
/* The slab number of this slab */
|
||||
slab_count_t slab_number;
|
||||
/* The offset in the allocator partition of the first block in this slab */
|
||||
physical_block_number_t start;
|
||||
/* The offset of the first block past the end of this slab */
|
||||
physical_block_number_t end;
|
||||
/* The starting translated PBN of the slab journal */
|
||||
physical_block_number_t journal_origin;
|
||||
/* The starting translated PBN of the reference counts */
|
||||
physical_block_number_t ref_counts_origin;
|
||||
|
||||
/* The administrative state of the slab */
|
||||
struct admin_state state;
|
||||
/* The status of the slab */
|
||||
enum slab_rebuild_status status;
|
||||
/* Whether the slab was ever queued for scrubbing */
|
||||
bool was_queued_for_scrubbing;
|
||||
|
||||
/* The priority at which this slab has been queued for allocation */
|
||||
u8 priority;
|
||||
|
||||
/* Fields beyond this point are the reference counts for the data blocks in this slab. */
|
||||
/* The size of the counters array */
|
||||
u32 block_count;
|
||||
/* The number of free blocks */
|
||||
u32 free_blocks;
|
||||
/* The array of reference counts */
|
||||
vdo_refcount_t *counters; /* use vdo_allocate() to align data ptr */
|
||||
|
||||
/* The saved block pointer and array indexes for the free block search */
|
||||
struct search_cursor search_cursor;
|
||||
|
||||
/* A list of the dirty blocks waiting to be written out */
|
||||
struct vdo_wait_queue dirty_blocks;
|
||||
/* The number of blocks which are currently writing */
|
||||
size_t active_count;
|
||||
|
||||
/* A waiter object for updating the slab summary */
|
||||
struct vdo_waiter summary_waiter;
|
||||
|
||||
/* The latest slab journal for which there has been a reference count update */
|
||||
struct journal_point slab_journal_point;
|
||||
|
||||
/* The number of reference count blocks */
|
||||
u32 reference_block_count;
|
||||
/* reference count block array */
|
||||
struct reference_block *reference_blocks;
|
||||
};
|
||||
|
||||
enum block_allocator_drain_step {
|
||||
VDO_DRAIN_ALLOCATOR_START,
|
||||
VDO_DRAIN_ALLOCATOR_STEP_SCRUBBER,
|
||||
VDO_DRAIN_ALLOCATOR_STEP_SLABS,
|
||||
VDO_DRAIN_ALLOCATOR_STEP_SUMMARY,
|
||||
VDO_DRAIN_ALLOCATOR_STEP_FINISHED,
|
||||
};
|
||||
|
||||
struct slab_scrubber {
|
||||
/* The queue of slabs to scrub first */
|
||||
struct list_head high_priority_slabs;
|
||||
/* The queue of slabs to scrub once there are no high_priority_slabs */
|
||||
struct list_head slabs;
|
||||
/* The queue of VIOs waiting for a slab to be scrubbed */
|
||||
struct vdo_wait_queue waiters;
|
||||
|
||||
/*
|
||||
* The number of slabs that are unrecovered or being scrubbed. This field is modified by
|
||||
* the physical zone thread, but is queried by other threads.
|
||||
*/
|
||||
slab_count_t slab_count;
|
||||
|
||||
/* The administrative state of the scrubber */
|
||||
struct admin_state admin_state;
|
||||
/* Whether to only scrub high-priority slabs */
|
||||
bool high_priority_only;
|
||||
/* The slab currently being scrubbed */
|
||||
struct vdo_slab *slab;
|
||||
/* The vio for loading slab journal blocks */
|
||||
struct vio vio;
|
||||
};
|
||||
|
||||
/* A sub-structure for applying actions in parallel to all an allocator's slabs. */
|
||||
struct slab_actor {
|
||||
/* The number of slabs performing a slab action */
|
||||
slab_count_t slab_action_count;
|
||||
/* The method to call when a slab action has been completed by all slabs */
|
||||
vdo_action_fn callback;
|
||||
};
|
||||
|
||||
/* A slab_iterator is a structure for iterating over a set of slabs. */
|
||||
struct slab_iterator {
|
||||
struct vdo_slab **slabs;
|
||||
struct vdo_slab *next;
|
||||
slab_count_t end;
|
||||
slab_count_t stride;
|
||||
};
|
||||
|
||||
/*
|
||||
* The slab_summary provides hints during load and recovery about the state of the slabs in order
|
||||
* to avoid the need to read the slab journals in their entirety before a VDO can come online.
|
||||
*
|
||||
* The information in the summary for each slab includes the rough number of free blocks (which is
|
||||
* used to prioritize scrubbing), the cleanliness of a slab (so that clean slabs containing free
|
||||
* space will be used on restart), and the location of the tail block of the slab's journal.
|
||||
*
|
||||
* The slab_summary has its own partition at the end of the volume which is sized to allow for a
|
||||
* complete copy of the summary for each of up to 16 physical zones.
|
||||
*
|
||||
* During resize, the slab_summary moves its backing partition and is saved once moved; the
|
||||
* slab_summary is not permitted to overwrite the previous recovery journal space.
|
||||
*
|
||||
* The slab_summary does not have its own version information, but relies on the VDO volume version
|
||||
* number.
|
||||
*/
|
||||
|
||||
/*
|
||||
* A slab status is a very small structure for use in determining the ordering of slabs in the
|
||||
* scrubbing process.
|
||||
*/
|
||||
struct slab_status {
|
||||
slab_count_t slab_number;
|
||||
bool is_clean;
|
||||
u8 emptiness;
|
||||
};
|
||||
|
||||
struct slab_summary_block {
|
||||
/* The block_allocator to which this block belongs */
|
||||
struct block_allocator *allocator;
|
||||
/* The index of this block in its zone's summary */
|
||||
block_count_t index;
|
||||
/* Whether this block has a write outstanding */
|
||||
bool writing;
|
||||
/* Ring of updates waiting on the outstanding write */
|
||||
struct vdo_wait_queue current_update_waiters;
|
||||
/* Ring of updates waiting on the next write */
|
||||
struct vdo_wait_queue next_update_waiters;
|
||||
/* The active slab_summary_entry array for this block */
|
||||
struct slab_summary_entry *entries;
|
||||
/* The vio used to write this block */
|
||||
struct vio vio;
|
||||
/* The packed entries, one block long, backing the vio */
|
||||
char *outgoing_entries;
|
||||
};
|
||||
|
||||
/*
|
||||
* The statistics for all the slab summary zones owned by this slab summary. These fields are all
|
||||
* mutated only by their physical zone threads, but are read by other threads when gathering
|
||||
* statistics for the entire depot.
|
||||
*/
|
||||
struct atomic_slab_summary_statistics {
|
||||
/* Number of blocks written */
|
||||
atomic64_t blocks_written;
|
||||
};
|
||||
|
||||
struct block_allocator {
|
||||
struct vdo_completion completion;
|
||||
/* The slab depot for this allocator */
|
||||
struct slab_depot *depot;
|
||||
/* The nonce of the VDO */
|
||||
nonce_t nonce;
|
||||
/* The physical zone number of this allocator */
|
||||
zone_count_t zone_number;
|
||||
/* The thread ID for this allocator's physical zone */
|
||||
thread_id_t thread_id;
|
||||
/* The number of slabs in this allocator */
|
||||
slab_count_t slab_count;
|
||||
/* The number of the last slab owned by this allocator */
|
||||
slab_count_t last_slab;
|
||||
/* The reduced priority level used to preserve unopened slabs */
|
||||
unsigned int unopened_slab_priority;
|
||||
/* The state of this allocator */
|
||||
struct admin_state state;
|
||||
/* The actor for applying an action to all slabs */
|
||||
struct slab_actor slab_actor;
|
||||
|
||||
/* The slab from which blocks are currently being allocated */
|
||||
struct vdo_slab *open_slab;
|
||||
/* A priority queue containing all slabs available for allocation */
|
||||
struct priority_table *prioritized_slabs;
|
||||
/* The slab scrubber */
|
||||
struct slab_scrubber scrubber;
|
||||
/* What phase of the close operation the allocator is to perform */
|
||||
enum block_allocator_drain_step drain_step;
|
||||
|
||||
/*
|
||||
* These statistics are all mutated only by the physical zone thread, but are read by other
|
||||
* threads when gathering statistics for the entire depot.
|
||||
*/
|
||||
/*
|
||||
* The count of allocated blocks in this zone. Not in block_allocator_statistics for
|
||||
* historical reasons.
|
||||
*/
|
||||
u64 allocated_blocks;
|
||||
/* Statistics for this block allocator */
|
||||
struct block_allocator_statistics statistics;
|
||||
/* Cumulative statistics for the slab journals in this zone */
|
||||
struct slab_journal_statistics slab_journal_statistics;
|
||||
/* Cumulative statistics for the reference counters in this zone */
|
||||
struct ref_counts_statistics ref_counts_statistics;
|
||||
|
||||
/*
|
||||
* This is the head of a queue of slab journals which have entries in their tail blocks
|
||||
* which have not yet started to commit. When the recovery journal is under space pressure,
|
||||
* slab journals which have uncommitted entries holding a lock on the recovery journal head
|
||||
* are forced to commit their blocks early. This list is kept in order, with the tail
|
||||
* containing the slab journal holding the most recent recovery journal lock.
|
||||
*/
|
||||
struct list_head dirty_slab_journals;
|
||||
|
||||
/* The vio pool for reading and writing block allocator metadata */
|
||||
struct vio_pool *vio_pool;
|
||||
/* The dm_kcopyd client for erasing slab journals */
|
||||
struct dm_kcopyd_client *eraser;
|
||||
/* Iterator over the slabs to be erased */
|
||||
struct slab_iterator slabs_to_erase;
|
||||
|
||||
/* The portion of the slab summary managed by this allocator */
|
||||
/* The state of the slab summary */
|
||||
struct admin_state summary_state;
|
||||
/* The number of outstanding summary writes */
|
||||
block_count_t summary_write_count;
|
||||
/* The array (owned by the blocks) of all entries */
|
||||
struct slab_summary_entry *summary_entries;
|
||||
/* The array of slab_summary_blocks */
|
||||
struct slab_summary_block *summary_blocks;
|
||||
};
|
||||
|
||||
enum slab_depot_load_type {
|
||||
VDO_SLAB_DEPOT_NORMAL_LOAD,
|
||||
VDO_SLAB_DEPOT_RECOVERY_LOAD,
|
||||
VDO_SLAB_DEPOT_REBUILD_LOAD
|
||||
};
|
||||
|
||||
struct slab_depot {
|
||||
zone_count_t zone_count;
|
||||
zone_count_t old_zone_count;
|
||||
struct vdo *vdo;
|
||||
struct slab_config slab_config;
|
||||
struct action_manager *action_manager;
|
||||
|
||||
physical_block_number_t first_block;
|
||||
physical_block_number_t last_block;
|
||||
physical_block_number_t origin;
|
||||
|
||||
/* slab_size == (1 << slab_size_shift) */
|
||||
unsigned int slab_size_shift;
|
||||
|
||||
/* Determines how slabs should be queued during load */
|
||||
enum slab_depot_load_type load_type;
|
||||
|
||||
/* The state for notifying slab journals to release recovery journal */
|
||||
sequence_number_t active_release_request;
|
||||
sequence_number_t new_release_request;
|
||||
|
||||
/* State variables for scrubbing complete handling */
|
||||
atomic_t zones_to_scrub;
|
||||
|
||||
/* Array of pointers to individually allocated slabs */
|
||||
struct vdo_slab **slabs;
|
||||
/* The number of slabs currently allocated and stored in 'slabs' */
|
||||
slab_count_t slab_count;
|
||||
|
||||
/* Array of pointers to a larger set of slabs (used during resize) */
|
||||
struct vdo_slab **new_slabs;
|
||||
/* The number of slabs currently allocated and stored in 'new_slabs' */
|
||||
slab_count_t new_slab_count;
|
||||
/* The size that 'new_slabs' was allocated for */
|
||||
block_count_t new_size;
|
||||
|
||||
/* The last block before resize, for rollback */
|
||||
physical_block_number_t old_last_block;
|
||||
/* The last block after resize, for resize */
|
||||
physical_block_number_t new_last_block;
|
||||
|
||||
/* The statistics for the slab summary */
|
||||
struct atomic_slab_summary_statistics summary_statistics;
|
||||
/* The start of the slab summary partition */
|
||||
physical_block_number_t summary_origin;
|
||||
/* The number of bits to shift to get a 7-bit fullness hint */
|
||||
unsigned int hint_shift;
|
||||
/* The slab summary entries for all of the zones the partition can hold */
|
||||
struct slab_summary_entry *summary_entries;
|
||||
|
||||
/* The block allocators for this depot */
|
||||
struct block_allocator allocators[];
|
||||
};
|
||||
|
||||
struct reference_updater;
|
||||
|
||||
bool __must_check vdo_attempt_replay_into_slab(struct vdo_slab *slab,
|
||||
physical_block_number_t pbn,
|
||||
enum journal_operation operation,
|
||||
bool increment,
|
||||
struct journal_point *recovery_point,
|
||||
struct vdo_completion *parent);
|
||||
|
||||
int __must_check vdo_adjust_reference_count_for_rebuild(struct slab_depot *depot,
|
||||
physical_block_number_t pbn,
|
||||
enum journal_operation operation);
|
||||
|
||||
static inline struct block_allocator *vdo_as_block_allocator(struct vdo_completion *completion)
|
||||
{
|
||||
vdo_assert_completion_type(completion, VDO_BLOCK_ALLOCATOR_COMPLETION);
|
||||
return container_of(completion, struct block_allocator, completion);
|
||||
}
|
||||
|
||||
int __must_check vdo_acquire_provisional_reference(struct vdo_slab *slab,
|
||||
physical_block_number_t pbn,
|
||||
struct pbn_lock *lock);
|
||||
|
||||
int __must_check vdo_allocate_block(struct block_allocator *allocator,
|
||||
physical_block_number_t *block_number_ptr);
|
||||
|
||||
int vdo_enqueue_clean_slab_waiter(struct block_allocator *allocator,
|
||||
struct vdo_waiter *waiter);
|
||||
|
||||
void vdo_modify_reference_count(struct vdo_completion *completion,
|
||||
struct reference_updater *updater);
|
||||
|
||||
int __must_check vdo_release_block_reference(struct block_allocator *allocator,
|
||||
physical_block_number_t pbn);
|
||||
|
||||
void vdo_notify_slab_journals_are_recovered(struct vdo_completion *completion);
|
||||
|
||||
void vdo_dump_block_allocator(const struct block_allocator *allocator);
|
||||
|
||||
int __must_check vdo_decode_slab_depot(struct slab_depot_state_2_0 state,
|
||||
struct vdo *vdo,
|
||||
struct partition *summary_partition,
|
||||
struct slab_depot **depot_ptr);
|
||||
|
||||
void vdo_free_slab_depot(struct slab_depot *depot);
|
||||
|
||||
struct slab_depot_state_2_0 __must_check vdo_record_slab_depot(const struct slab_depot *depot);
|
||||
|
||||
int __must_check vdo_allocate_reference_counters(struct slab_depot *depot);
|
||||
|
||||
struct vdo_slab * __must_check vdo_get_slab(const struct slab_depot *depot,
|
||||
physical_block_number_t pbn);
|
||||
|
||||
u8 __must_check vdo_get_increment_limit(struct slab_depot *depot,
|
||||
physical_block_number_t pbn);
|
||||
|
||||
bool __must_check vdo_is_physical_data_block(const struct slab_depot *depot,
|
||||
physical_block_number_t pbn);
|
||||
|
||||
block_count_t __must_check vdo_get_slab_depot_allocated_blocks(const struct slab_depot *depot);
|
||||
|
||||
block_count_t __must_check vdo_get_slab_depot_data_blocks(const struct slab_depot *depot);
|
||||
|
||||
void vdo_get_slab_depot_statistics(const struct slab_depot *depot,
|
||||
struct vdo_statistics *stats);
|
||||
|
||||
void vdo_load_slab_depot(struct slab_depot *depot,
|
||||
const struct admin_state_code *operation,
|
||||
struct vdo_completion *parent, void *context);
|
||||
|
||||
void vdo_prepare_slab_depot_to_allocate(struct slab_depot *depot,
|
||||
enum slab_depot_load_type load_type,
|
||||
struct vdo_completion *parent);
|
||||
|
||||
void vdo_update_slab_depot_size(struct slab_depot *depot);
|
||||
|
||||
int __must_check vdo_prepare_to_grow_slab_depot(struct slab_depot *depot,
|
||||
const struct partition *partition);
|
||||
|
||||
void vdo_use_new_slabs(struct slab_depot *depot, struct vdo_completion *parent);
|
||||
|
||||
void vdo_abandon_new_slabs(struct slab_depot *depot);
|
||||
|
||||
void vdo_drain_slab_depot(struct slab_depot *depot,
|
||||
const struct admin_state_code *operation,
|
||||
struct vdo_completion *parent);
|
||||
|
||||
void vdo_resume_slab_depot(struct slab_depot *depot, struct vdo_completion *parent);
|
||||
|
||||
void vdo_commit_oldest_slab_journal_tail_blocks(struct slab_depot *depot,
|
||||
sequence_number_t recovery_block_number);
|
||||
|
||||
void vdo_scrub_all_unrecovered_slabs(struct slab_depot *depot,
|
||||
struct vdo_completion *parent);
|
||||
|
||||
void vdo_dump_slab_depot(const struct slab_depot *depot);
|
||||
|
||||
#endif /* VDO_SLAB_DEPOT_H */
|
|
@ -0,0 +1,278 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef STATISTICS_H
|
||||
#define STATISTICS_H
|
||||
|
||||
#include "types.h"
|
||||
|
||||
enum {
|
||||
STATISTICS_VERSION = 36,
|
||||
};
|
||||
|
||||
struct block_allocator_statistics {
|
||||
/* The total number of slabs from which blocks may be allocated */
|
||||
u64 slab_count;
|
||||
/* The total number of slabs from which blocks have ever been allocated */
|
||||
u64 slabs_opened;
|
||||
/* The number of times since loading that a slab has been re-opened */
|
||||
u64 slabs_reopened;
|
||||
};
|
||||
|
||||
/**
|
||||
* Counters for tracking the number of items written (blocks, requests, etc.)
|
||||
* that keep track of totals at steps in the write pipeline. Three counters
|
||||
* allow the number of buffered, in-memory items and the number of in-flight,
|
||||
* unacknowledged writes to be derived, while still tracking totals for
|
||||
* reporting purposes
|
||||
*/
|
||||
struct commit_statistics {
|
||||
/* The total number of items on which processing has started */
|
||||
u64 started;
|
||||
/* The total number of items for which a write operation has been issued */
|
||||
u64 written;
|
||||
/* The total number of items for which a write operation has completed */
|
||||
u64 committed;
|
||||
};
|
||||
|
||||
/** Counters for events in the recovery journal */
|
||||
struct recovery_journal_statistics {
|
||||
/* Number of times the on-disk journal was full */
|
||||
u64 disk_full;
|
||||
/* Number of times the recovery journal requested slab journal commits. */
|
||||
u64 slab_journal_commits_requested;
|
||||
/* Write/Commit totals for individual journal entries */
|
||||
struct commit_statistics entries;
|
||||
/* Write/Commit totals for journal blocks */
|
||||
struct commit_statistics blocks;
|
||||
};
|
||||
|
||||
/** The statistics for the compressed block packer. */
|
||||
struct packer_statistics {
|
||||
/* Number of compressed data items written since startup */
|
||||
u64 compressed_fragments_written;
|
||||
/* Number of blocks containing compressed items written since startup */
|
||||
u64 compressed_blocks_written;
|
||||
/* Number of VIOs that are pending in the packer */
|
||||
u64 compressed_fragments_in_packer;
|
||||
};
|
||||
|
||||
/** The statistics for the slab journals. */
|
||||
struct slab_journal_statistics {
|
||||
/* Number of times the on-disk journal was full */
|
||||
u64 disk_full_count;
|
||||
/* Number of times an entry was added over the flush threshold */
|
||||
u64 flush_count;
|
||||
/* Number of times an entry was added over the block threshold */
|
||||
u64 blocked_count;
|
||||
/* Number of times a tail block was written */
|
||||
u64 blocks_written;
|
||||
/* Number of times we had to wait for the tail to write */
|
||||
u64 tail_busy_count;
|
||||
};
|
||||
|
||||
/** The statistics for the slab summary. */
|
||||
struct slab_summary_statistics {
|
||||
/* Number of blocks written */
|
||||
u64 blocks_written;
|
||||
};
|
||||
|
||||
/** The statistics for the reference counts. */
|
||||
struct ref_counts_statistics {
|
||||
/* Number of reference blocks written */
|
||||
u64 blocks_written;
|
||||
};
|
||||
|
||||
/** The statistics for the block map. */
|
||||
struct block_map_statistics {
|
||||
/* number of dirty (resident) pages */
|
||||
u32 dirty_pages;
|
||||
/* number of clean (resident) pages */
|
||||
u32 clean_pages;
|
||||
/* number of free pages */
|
||||
u32 free_pages;
|
||||
/* number of pages in failed state */
|
||||
u32 failed_pages;
|
||||
/* number of pages incoming */
|
||||
u32 incoming_pages;
|
||||
/* number of pages outgoing */
|
||||
u32 outgoing_pages;
|
||||
/* how many times free page not avail */
|
||||
u32 cache_pressure;
|
||||
/* number of get_vdo_page() calls for read */
|
||||
u64 read_count;
|
||||
/* number of get_vdo_page() calls for write */
|
||||
u64 write_count;
|
||||
/* number of times pages failed to read */
|
||||
u64 failed_reads;
|
||||
/* number of times pages failed to write */
|
||||
u64 failed_writes;
|
||||
/* number of gets that are reclaimed */
|
||||
u64 reclaimed;
|
||||
/* number of gets for outgoing pages */
|
||||
u64 read_outgoing;
|
||||
/* number of gets that were already there */
|
||||
u64 found_in_cache;
|
||||
/* number of gets requiring discard */
|
||||
u64 discard_required;
|
||||
/* number of gets enqueued for their page */
|
||||
u64 wait_for_page;
|
||||
/* number of gets that have to fetch */
|
||||
u64 fetch_required;
|
||||
/* number of page fetches */
|
||||
u64 pages_loaded;
|
||||
/* number of page saves */
|
||||
u64 pages_saved;
|
||||
/* the number of flushes issued */
|
||||
u64 flush_count;
|
||||
};
|
||||
|
||||
/** The dedupe statistics from hash locks */
|
||||
struct hash_lock_statistics {
|
||||
/* Number of times the UDS advice proved correct */
|
||||
u64 dedupe_advice_valid;
|
||||
/* Number of times the UDS advice proved incorrect */
|
||||
u64 dedupe_advice_stale;
|
||||
/* Number of writes with the same data as another in-flight write */
|
||||
u64 concurrent_data_matches;
|
||||
/* Number of writes whose hash collided with an in-flight write */
|
||||
u64 concurrent_hash_collisions;
|
||||
/* Current number of dedupe queries that are in flight */
|
||||
u32 curr_dedupe_queries;
|
||||
};
|
||||
|
||||
/** Counts of error conditions in VDO. */
|
||||
struct error_statistics {
|
||||
/* number of times VDO got an invalid dedupe advice PBN from UDS */
|
||||
u64 invalid_advice_pbn_count;
|
||||
/* number of times a VIO completed with a VDO_NO_SPACE error */
|
||||
u64 no_space_error_count;
|
||||
/* number of times a VIO completed with a VDO_READ_ONLY error */
|
||||
u64 read_only_error_count;
|
||||
};
|
||||
|
||||
struct bio_stats {
|
||||
/* Number of REQ_OP_READ bios */
|
||||
u64 read;
|
||||
/* Number of REQ_OP_WRITE bios with data */
|
||||
u64 write;
|
||||
/* Number of bios tagged with REQ_PREFLUSH and containing no data */
|
||||
u64 empty_flush;
|
||||
/* Number of REQ_OP_DISCARD bios */
|
||||
u64 discard;
|
||||
/* Number of bios tagged with REQ_PREFLUSH */
|
||||
u64 flush;
|
||||
/* Number of bios tagged with REQ_FUA */
|
||||
u64 fua;
|
||||
};
|
||||
|
||||
struct memory_usage {
|
||||
/* Tracked bytes currently allocated. */
|
||||
u64 bytes_used;
|
||||
/* Maximum tracked bytes allocated. */
|
||||
u64 peak_bytes_used;
|
||||
};
|
||||
|
||||
/** UDS index statistics */
|
||||
struct index_statistics {
|
||||
/* Number of records stored in the index */
|
||||
u64 entries_indexed;
|
||||
/* Number of post calls that found an existing entry */
|
||||
u64 posts_found;
|
||||
/* Number of post calls that added a new entry */
|
||||
u64 posts_not_found;
|
||||
/* Number of query calls that found an existing entry */
|
||||
u64 queries_found;
|
||||
/* Number of query calls that added a new entry */
|
||||
u64 queries_not_found;
|
||||
/* Number of update calls that found an existing entry */
|
||||
u64 updates_found;
|
||||
/* Number of update calls that added a new entry */
|
||||
u64 updates_not_found;
|
||||
/* Number of entries discarded */
|
||||
u64 entries_discarded;
|
||||
};
|
||||
|
||||
/** The statistics of the vdo service. */
|
||||
struct vdo_statistics {
|
||||
u32 version;
|
||||
/* Number of blocks used for data */
|
||||
u64 data_blocks_used;
|
||||
/* Number of blocks used for VDO metadata */
|
||||
u64 overhead_blocks_used;
|
||||
/* Number of logical blocks that are currently mapped to physical blocks */
|
||||
u64 logical_blocks_used;
|
||||
/* number of physical blocks */
|
||||
block_count_t physical_blocks;
|
||||
/* number of logical blocks */
|
||||
block_count_t logical_blocks;
|
||||
/* Size of the block map page cache, in bytes */
|
||||
u64 block_map_cache_size;
|
||||
/* The physical block size */
|
||||
u64 block_size;
|
||||
/* Number of times the VDO has successfully recovered */
|
||||
u64 complete_recoveries;
|
||||
/* Number of times the VDO has recovered from read-only mode */
|
||||
u64 read_only_recoveries;
|
||||
/* String describing the operating mode of the VDO */
|
||||
char mode[15];
|
||||
/* Whether the VDO is in recovery mode */
|
||||
bool in_recovery_mode;
|
||||
/* What percentage of recovery mode work has been completed */
|
||||
u8 recovery_percentage;
|
||||
/* The statistics for the compressed block packer */
|
||||
struct packer_statistics packer;
|
||||
/* Counters for events in the block allocator */
|
||||
struct block_allocator_statistics allocator;
|
||||
/* Counters for events in the recovery journal */
|
||||
struct recovery_journal_statistics journal;
|
||||
/* The statistics for the slab journals */
|
||||
struct slab_journal_statistics slab_journal;
|
||||
/* The statistics for the slab summary */
|
||||
struct slab_summary_statistics slab_summary;
|
||||
/* The statistics for the reference counts */
|
||||
struct ref_counts_statistics ref_counts;
|
||||
/* The statistics for the block map */
|
||||
struct block_map_statistics block_map;
|
||||
/* The dedupe statistics from hash locks */
|
||||
struct hash_lock_statistics hash_lock;
|
||||
/* Counts of error conditions */
|
||||
struct error_statistics errors;
|
||||
/* The VDO instance */
|
||||
u32 instance;
|
||||
/* Current number of active VIOs */
|
||||
u32 current_vios_in_progress;
|
||||
/* Maximum number of active VIOs */
|
||||
u32 max_vios;
|
||||
/* Number of times the UDS index was too slow in responding */
|
||||
u64 dedupe_advice_timeouts;
|
||||
/* Number of flush requests submitted to the storage device */
|
||||
u64 flush_out;
|
||||
/* Logical block size */
|
||||
u64 logical_block_size;
|
||||
/* Bios submitted into VDO from above */
|
||||
struct bio_stats bios_in;
|
||||
struct bio_stats bios_in_partial;
|
||||
/* Bios submitted onward for user data */
|
||||
struct bio_stats bios_out;
|
||||
/* Bios submitted onward for metadata */
|
||||
struct bio_stats bios_meta;
|
||||
struct bio_stats bios_journal;
|
||||
struct bio_stats bios_page_cache;
|
||||
struct bio_stats bios_out_completed;
|
||||
struct bio_stats bios_meta_completed;
|
||||
struct bio_stats bios_journal_completed;
|
||||
struct bio_stats bios_page_cache_completed;
|
||||
struct bio_stats bios_acknowledged;
|
||||
struct bio_stats bios_acknowledged_partial;
|
||||
/* Current number of bios in progress */
|
||||
struct bio_stats bios_in_progress;
|
||||
/* Memory usage stats. */
|
||||
struct memory_usage memory_usage;
|
||||
/* The statistics for the UDS index */
|
||||
struct index_statistics index;
|
||||
};
|
||||
|
||||
#endif /* not STATISTICS_H */
|
|
@ -0,0 +1,94 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "status-codes.h"
|
||||
|
||||
#include "errors.h"
|
||||
#include "logger.h"
|
||||
#include "permassert.h"
|
||||
#include "thread-utils.h"
|
||||
|
||||
const struct error_info vdo_status_list[] = {
|
||||
{ "VDO_NOT_IMPLEMENTED", "Not implemented" },
|
||||
{ "VDO_OUT_OF_RANGE", "Out of range" },
|
||||
{ "VDO_REF_COUNT_INVALID", "Reference count would become invalid" },
|
||||
{ "VDO_NO_SPACE", "Out of space" },
|
||||
{ "VDO_BAD_CONFIGURATION", "Bad configuration option" },
|
||||
{ "VDO_COMPONENT_BUSY", "Prior operation still in progress" },
|
||||
{ "VDO_BAD_PAGE", "Corrupt or incorrect page" },
|
||||
{ "VDO_UNSUPPORTED_VERSION", "Unsupported component version" },
|
||||
{ "VDO_INCORRECT_COMPONENT", "Component id mismatch in decoder" },
|
||||
{ "VDO_PARAMETER_MISMATCH", "Parameters have conflicting values" },
|
||||
{ "VDO_UNKNOWN_PARTITION", "No partition exists with a given id" },
|
||||
{ "VDO_PARTITION_EXISTS", "A partition already exists with a given id" },
|
||||
{ "VDO_INCREMENT_TOO_SMALL", "Physical block growth of too few blocks" },
|
||||
{ "VDO_CHECKSUM_MISMATCH", "Incorrect checksum" },
|
||||
{ "VDO_LOCK_ERROR", "A lock is held incorrectly" },
|
||||
{ "VDO_READ_ONLY", "The device is in read-only mode" },
|
||||
{ "VDO_SHUTTING_DOWN", "The device is shutting down" },
|
||||
{ "VDO_CORRUPT_JOURNAL", "Recovery journal entries corrupted" },
|
||||
{ "VDO_TOO_MANY_SLABS", "Exceeds maximum number of slabs supported" },
|
||||
{ "VDO_INVALID_FRAGMENT", "Compressed block fragment is invalid" },
|
||||
{ "VDO_RETRY_AFTER_REBUILD", "Retry operation after rebuilding finishes" },
|
||||
{ "VDO_BAD_MAPPING", "Invalid page mapping" },
|
||||
{ "VDO_BIO_CREATION_FAILED", "Bio creation failed" },
|
||||
{ "VDO_BAD_MAGIC", "Bad magic number" },
|
||||
{ "VDO_BAD_NONCE", "Bad nonce" },
|
||||
{ "VDO_JOURNAL_OVERFLOW", "Journal sequence number overflow" },
|
||||
{ "VDO_INVALID_ADMIN_STATE", "Invalid operation for current state" },
|
||||
};
|
||||
|
||||
/**
|
||||
* vdo_register_status_codes() - Register the VDO status codes.
|
||||
* Return: A success or error code.
|
||||
*/
|
||||
int vdo_register_status_codes(void)
|
||||
{
|
||||
int result;
|
||||
|
||||
BUILD_BUG_ON((VDO_STATUS_CODE_LAST - VDO_STATUS_CODE_BASE) !=
|
||||
ARRAY_SIZE(vdo_status_list));
|
||||
|
||||
result = uds_register_error_block("VDO Status", VDO_STATUS_CODE_BASE,
|
||||
VDO_STATUS_CODE_BLOCK_END, vdo_status_list,
|
||||
sizeof(vdo_status_list));
|
||||
return (result == UDS_SUCCESS) ? VDO_SUCCESS : result;
|
||||
}
|
||||
|
||||
/**
|
||||
* vdo_status_to_errno() - Given an error code, return a value we can return to the OS.
|
||||
* @error: The error code to convert.
|
||||
*
|
||||
* The input error code may be a system-generated value (such as -EIO), an errno macro used in our
|
||||
* code (such as EIO), or a UDS or VDO status code; the result must be something the rest of the OS
|
||||
* can consume (negative errno values such as -EIO, in the case of the kernel).
|
||||
*
|
||||
* Return: A system error code value.
|
||||
*/
|
||||
int vdo_status_to_errno(int error)
|
||||
{
|
||||
char error_name[VDO_MAX_ERROR_NAME_SIZE];
|
||||
char error_message[VDO_MAX_ERROR_MESSAGE_SIZE];
|
||||
|
||||
/* 0 is success, negative a system error code */
|
||||
if (likely(error <= 0))
|
||||
return error;
|
||||
if (error < 1024)
|
||||
return -error;
|
||||
|
||||
/* VDO or UDS error */
|
||||
switch (error) {
|
||||
case VDO_NO_SPACE:
|
||||
return -ENOSPC;
|
||||
case VDO_READ_ONLY:
|
||||
return -EIO;
|
||||
default:
|
||||
vdo_log_info("%s: mapping internal status code %d (%s: %s) to EIO",
|
||||
__func__, error,
|
||||
uds_string_error_name(error, error_name, sizeof(error_name)),
|
||||
uds_string_error(error, error_message, sizeof(error_message)));
|
||||
return -EIO;
|
||||
}
|
||||
}
|
|
@ -0,0 +1,86 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#ifndef VDO_STATUS_CODES_H
|
||||
#define VDO_STATUS_CODES_H
|
||||
|
||||
#include "errors.h"
|
||||
|
||||
enum {
|
||||
UDS_ERRORS_BLOCK_SIZE = UDS_ERROR_CODE_BLOCK_END - UDS_ERROR_CODE_BASE,
|
||||
VDO_ERRORS_BLOCK_START = UDS_ERROR_CODE_BLOCK_END,
|
||||
VDO_ERRORS_BLOCK_END = VDO_ERRORS_BLOCK_START + UDS_ERRORS_BLOCK_SIZE,
|
||||
};
|
||||
|
||||
/* VDO-specific status codes. */
|
||||
enum vdo_status_codes {
|
||||
/* base of all VDO errors */
|
||||
VDO_STATUS_CODE_BASE = VDO_ERRORS_BLOCK_START,
|
||||
/* we haven't written this yet */
|
||||
VDO_NOT_IMPLEMENTED = VDO_STATUS_CODE_BASE,
|
||||
/* input out of range */
|
||||
VDO_OUT_OF_RANGE,
|
||||
/* an invalid reference count would result */
|
||||
VDO_REF_COUNT_INVALID,
|
||||
/* a free block could not be allocated */
|
||||
VDO_NO_SPACE,
|
||||
/* improper or missing configuration option */
|
||||
VDO_BAD_CONFIGURATION,
|
||||
/* prior operation still in progress */
|
||||
VDO_COMPONENT_BUSY,
|
||||
/* page contents incorrect or corrupt data */
|
||||
VDO_BAD_PAGE,
|
||||
/* unsupported version of some component */
|
||||
VDO_UNSUPPORTED_VERSION,
|
||||
/* component id mismatch in decoder */
|
||||
VDO_INCORRECT_COMPONENT,
|
||||
/* parameters have conflicting values */
|
||||
VDO_PARAMETER_MISMATCH,
|
||||
/* no partition exists with a given id */
|
||||
VDO_UNKNOWN_PARTITION,
|
||||
/* a partition already exists with a given id */
|
||||
VDO_PARTITION_EXISTS,
|
||||
/* physical block growth of too few blocks */
|
||||
VDO_INCREMENT_TOO_SMALL,
|
||||
/* incorrect checksum */
|
||||
VDO_CHECKSUM_MISMATCH,
|
||||
/* a lock is held incorrectly */
|
||||
VDO_LOCK_ERROR,
|
||||
/* the VDO is in read-only mode */
|
||||
VDO_READ_ONLY,
|
||||
/* the VDO is shutting down */
|
||||
VDO_SHUTTING_DOWN,
|
||||
/* the recovery journal has corrupt entries */
|
||||
VDO_CORRUPT_JOURNAL,
|
||||
/* exceeds maximum number of slabs supported */
|
||||
VDO_TOO_MANY_SLABS,
|
||||
/* a compressed block fragment is invalid */
|
||||
VDO_INVALID_FRAGMENT,
|
||||
/* action is unsupported while rebuilding */
|
||||
VDO_RETRY_AFTER_REBUILD,
|
||||
/* a block map entry is invalid */
|
||||
VDO_BAD_MAPPING,
|
||||
/* bio_add_page failed */
|
||||
VDO_BIO_CREATION_FAILED,
|
||||
/* bad magic number */
|
||||
VDO_BAD_MAGIC,
|
||||
/* bad nonce */
|
||||
VDO_BAD_NONCE,
|
||||
/* sequence number overflow */
|
||||
VDO_JOURNAL_OVERFLOW,
|
||||
/* the VDO is not in a state to perform an admin operation */
|
||||
VDO_INVALID_ADMIN_STATE,
|
||||
/* one more than last error code */
|
||||
VDO_STATUS_CODE_LAST,
|
||||
VDO_STATUS_CODE_BLOCK_END = VDO_ERRORS_BLOCK_END
|
||||
};
|
||||
|
||||
extern const struct error_info vdo_status_list[];
|
||||
|
||||
int vdo_register_status_codes(void);
|
||||
|
||||
int vdo_status_to_errno(int error);
|
||||
|
||||
#endif /* VDO_STATUS_CODES_H */
|
|
@ -0,0 +1,22 @@
|
|||
// SPDX-License-Identifier: GPL-2.0-only
|
||||
/*
|
||||
* Copyright 2023 Red Hat
|
||||
*/
|
||||
|
||||
#include "string-utils.h"
|
||||
|
||||
char *vdo_append_to_buffer(char *buffer, char *buf_end, const char *fmt, ...)
|
||||
{
|
||||
va_list args;
|
||||
size_t n;
|
||||
|
||||
va_start(args, fmt);
|
||||
n = vsnprintf(buffer, buf_end - buffer, fmt, args);
|
||||
if (n >= (size_t) (buf_end - buffer))
|
||||
buffer = buf_end;
|
||||
else
|
||||
buffer += n;
|
||||
va_end(args);
|
||||
|
||||
return buffer;
|
||||
}
|
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue