2005-04-16 22:20:36 +00:00
|
|
|
/*
|
|
|
|
* linux/fs/nfs/direct.c
|
|
|
|
*
|
|
|
|
* Copyright (C) 2003 by Chuck Lever <cel@netapp.com>
|
|
|
|
*
|
|
|
|
* High-performance uncached I/O for the Linux NFS client
|
|
|
|
*
|
|
|
|
* There are important applications whose performance or correctness
|
|
|
|
* depends on uncached access to file data. Database clusters
|
2006-03-20 18:44:34 +00:00
|
|
|
* (multiple copies of the same instance running on separate hosts)
|
2005-04-16 22:20:36 +00:00
|
|
|
* implement their own cache coherency protocol that subsumes file
|
2006-03-20 18:44:34 +00:00
|
|
|
* system cache protocols. Applications that process datasets
|
|
|
|
* considerably larger than the client's memory do not always benefit
|
|
|
|
* from a local cache. A streaming video server, for instance, has no
|
2005-04-16 22:20:36 +00:00
|
|
|
* need to cache the contents of a file.
|
|
|
|
*
|
|
|
|
* When an application requests uncached I/O, all read and write requests
|
|
|
|
* are made directly to the server; data stored or fetched via these
|
|
|
|
* requests is not cached in the Linux page cache. The client does not
|
|
|
|
* correct unaligned requests from applications. All requested bytes are
|
|
|
|
* held on permanent storage before a direct write system call returns to
|
|
|
|
* an application.
|
|
|
|
*
|
|
|
|
* Solaris implements an uncached I/O facility called directio() that
|
|
|
|
* is used for backups and sequential I/O to very large files. Solaris
|
|
|
|
* also supports uncaching whole NFS partitions with "-o forcedirectio,"
|
|
|
|
* an undocumented mount option.
|
|
|
|
*
|
|
|
|
* Designed by Jeff Kimmel, Chuck Lever, and Trond Myklebust, with
|
|
|
|
* help from Andrew Morton.
|
|
|
|
*
|
|
|
|
* 18 Dec 2001 Initial implementation for 2.4 --cel
|
|
|
|
* 08 Jul 2002 Version for 2.4.19, with bug fixes --trondmy
|
|
|
|
* 08 Jun 2003 Port to 2.5 APIs --cel
|
|
|
|
* 31 Mar 2004 Handle direct I/O without VFS support --cel
|
|
|
|
* 15 Sep 2004 Parallel async reads --cel
|
2006-03-20 18:44:34 +00:00
|
|
|
* 04 May 2005 support O_DIRECT with aio --cel
|
2005-04-16 22:20:36 +00:00
|
|
|
*
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include <linux/errno.h>
|
|
|
|
#include <linux/sched.h>
|
|
|
|
#include <linux/kernel.h>
|
|
|
|
#include <linux/file.h>
|
|
|
|
#include <linux/pagemap.h>
|
|
|
|
#include <linux/kref.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 08:04:11 +00:00
|
|
|
#include <linux/slab.h>
|
2011-02-21 21:28:34 +00:00
|
|
|
#include <linux/task_io_accounting_ops.h>
|
2012-09-25 06:55:57 +00:00
|
|
|
#include <linux/module.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
#include <linux/nfs_fs.h>
|
|
|
|
#include <linux/nfs_page.h>
|
|
|
|
#include <linux/sunrpc/clnt.h>
|
|
|
|
|
|
|
|
#include <asm/uaccess.h>
|
2011-07-26 23:09:06 +00:00
|
|
|
#include <linux/atomic.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2007-04-10 13:26:35 +00:00
|
|
|
#include "internal.h"
|
2006-03-20 18:44:14 +00:00
|
|
|
#include "iostat.h"
|
2012-04-20 18:47:57 +00:00
|
|
|
#include "pnfs.h"
|
2006-03-20 18:44:14 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
#define NFSDBG_FACILITY NFSDBG_VFS
|
|
|
|
|
2006-12-07 04:33:20 +00:00
|
|
|
static struct kmem_cache *nfs_direct_cachep;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* This represents a set of asynchronous requests that we're waiting on
|
|
|
|
*/
|
2014-09-19 16:48:33 +00:00
|
|
|
struct nfs_direct_mirror {
|
|
|
|
ssize_t count;
|
|
|
|
};
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
struct nfs_direct_req {
|
|
|
|
struct kref kref; /* release manager */
|
2006-03-20 18:44:34 +00:00
|
|
|
|
|
|
|
/* I/O parameters */
|
2006-03-20 18:44:36 +00:00
|
|
|
struct nfs_open_context *ctx; /* file open context info */
|
2010-06-25 20:35:53 +00:00
|
|
|
struct nfs_lock_context *l_ctx; /* Lock context info */
|
2006-03-20 18:44:30 +00:00
|
|
|
struct kiocb * iocb; /* controlling i/o request */
|
2006-03-20 18:44:34 +00:00
|
|
|
struct inode * inode; /* target file of i/o */
|
2006-03-20 18:44:34 +00:00
|
|
|
|
|
|
|
/* completion state */
|
2006-06-28 20:52:45 +00:00
|
|
|
atomic_t io_count; /* i/os we're waiting for */
|
2006-03-20 18:44:34 +00:00
|
|
|
spinlock_t lock; /* protect completion state */
|
2014-09-19 16:48:33 +00:00
|
|
|
|
|
|
|
struct nfs_direct_mirror mirrors[NFS_PAGEIO_DESCRIPTOR_MIRROR_MAX];
|
|
|
|
int mirror_count;
|
|
|
|
|
2006-03-20 18:44:34 +00:00
|
|
|
ssize_t count, /* bytes actually processed */
|
2016-04-01 15:42:29 +00:00
|
|
|
max_count, /* max expected count */
|
2012-09-25 06:55:57 +00:00
|
|
|
bytes_left, /* bytes left to be sent */
|
2015-01-19 04:41:16 +00:00
|
|
|
io_start, /* start of IO */
|
2005-04-16 22:20:36 +00:00
|
|
|
error; /* any reported error */
|
2006-03-20 18:44:43 +00:00
|
|
|
struct completion completion; /* wait for i/o completion */
|
2006-03-20 18:44:36 +00:00
|
|
|
|
|
|
|
/* commit state */
|
2012-04-20 18:47:57 +00:00
|
|
|
struct nfs_mds_commit_info mds_cinfo; /* Storage for cinfo */
|
|
|
|
struct pnfs_ds_commit_info ds_cinfo; /* Storage for cinfo */
|
|
|
|
struct work_struct work;
|
2006-03-20 18:44:36 +00:00
|
|
|
int flags;
|
|
|
|
#define NFS_ODIRECT_DO_COMMIT (1) /* an unstable reply was received */
|
|
|
|
#define NFS_ODIRECT_RESCHED_WRITES (2) /* write verification failed */
|
|
|
|
struct nfs_writeverf verf; /* unstable write verifier */
|
2005-04-16 22:20:36 +00:00
|
|
|
};
|
|
|
|
|
2012-04-20 18:47:57 +00:00
|
|
|
static const struct nfs_pgio_completion_ops nfs_direct_write_completion_ops;
|
|
|
|
static const struct nfs_commit_completion_ops nfs_direct_commit_completion_ops;
|
2006-03-20 18:44:36 +00:00
|
|
|
static void nfs_direct_write_complete(struct nfs_direct_req *dreq, struct inode *inode);
|
2012-04-20 18:47:57 +00:00
|
|
|
static void nfs_direct_write_schedule_work(struct work_struct *work);
|
2006-06-28 20:52:45 +00:00
|
|
|
|
|
|
|
static inline void get_dreq(struct nfs_direct_req *dreq)
|
|
|
|
{
|
|
|
|
atomic_inc(&dreq->io_count);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline int put_dreq(struct nfs_direct_req *dreq)
|
|
|
|
{
|
|
|
|
return atomic_dec_and_test(&dreq->io_count);
|
|
|
|
}
|
|
|
|
|
2014-09-19 16:48:33 +00:00
|
|
|
static void
|
|
|
|
nfs_direct_good_bytes(struct nfs_direct_req *dreq, struct nfs_pgio_header *hdr)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
ssize_t count;
|
|
|
|
|
2016-04-01 15:42:29 +00:00
|
|
|
WARN_ON_ONCE(dreq->count >= dreq->max_count);
|
|
|
|
|
2015-04-09 15:02:16 +00:00
|
|
|
if (dreq->mirror_count == 1) {
|
|
|
|
dreq->mirrors[hdr->pgio_mirror_idx].count += hdr->good_bytes;
|
|
|
|
dreq->count += hdr->good_bytes;
|
|
|
|
} else {
|
|
|
|
/* mirrored writes */
|
|
|
|
count = dreq->mirrors[hdr->pgio_mirror_idx].count;
|
|
|
|
if (count + dreq->io_start < hdr->io_start + hdr->good_bytes) {
|
|
|
|
count = hdr->io_start + hdr->good_bytes - dreq->io_start;
|
|
|
|
dreq->mirrors[hdr->pgio_mirror_idx].count = count;
|
|
|
|
}
|
|
|
|
/* update the dreq->count by finding the minimum agreed count from all
|
|
|
|
* mirrors */
|
|
|
|
count = dreq->mirrors[0].count;
|
2014-09-19 16:48:33 +00:00
|
|
|
|
2015-04-09 15:02:16 +00:00
|
|
|
for (i = 1; i < dreq->mirror_count; i++)
|
|
|
|
count = min(count, dreq->mirrors[i].count);
|
2014-09-19 16:48:33 +00:00
|
|
|
|
2015-04-09 15:02:16 +00:00
|
|
|
dreq->count = count;
|
2015-01-19 04:41:16 +00:00
|
|
|
}
|
2014-09-19 16:48:33 +00:00
|
|
|
}
|
|
|
|
|
2014-05-15 15:56:54 +00:00
|
|
|
/*
|
|
|
|
* nfs_direct_select_verf - select the right verifier
|
|
|
|
* @dreq - direct request possibly spanning multiple servers
|
|
|
|
* @ds_clp - nfs_client of data server or NULL if MDS / non-pnfs
|
2014-09-16 21:35:51 +00:00
|
|
|
* @commit_idx - commit bucket index for the DS
|
2014-05-15 15:56:54 +00:00
|
|
|
*
|
|
|
|
* returns the correct verifier to use given the role of the server
|
|
|
|
*/
|
|
|
|
static struct nfs_writeverf *
|
|
|
|
nfs_direct_select_verf(struct nfs_direct_req *dreq,
|
|
|
|
struct nfs_client *ds_clp,
|
2014-09-16 21:35:51 +00:00
|
|
|
int commit_idx)
|
2014-05-15 15:56:54 +00:00
|
|
|
{
|
|
|
|
struct nfs_writeverf *verfp = &dreq->verf;
|
|
|
|
|
|
|
|
#ifdef CONFIG_NFS_V4_1
|
2015-09-21 22:54:47 +00:00
|
|
|
/*
|
|
|
|
* pNFS is in use, use the DS verf except commit_through_mds is set
|
|
|
|
* for layout segment where nbuckets is zero.
|
|
|
|
*/
|
|
|
|
if (ds_clp && dreq->ds_cinfo.nbuckets > 0) {
|
2014-09-16 21:35:51 +00:00
|
|
|
if (commit_idx >= 0 && commit_idx < dreq->ds_cinfo.nbuckets)
|
|
|
|
verfp = &dreq->ds_cinfo.buckets[commit_idx].direct_verf;
|
2014-05-15 15:56:54 +00:00
|
|
|
else
|
|
|
|
WARN_ON_ONCE(1);
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
return verfp;
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
* nfs_direct_set_hdr_verf - set the write/commit verifier
|
|
|
|
* @dreq - direct request possibly spanning multiple servers
|
|
|
|
* @hdr - pageio header to validate against previously seen verfs
|
|
|
|
*
|
|
|
|
* Set the server's (MDS or DS) "seen" verifier
|
|
|
|
*/
|
|
|
|
static void nfs_direct_set_hdr_verf(struct nfs_direct_req *dreq,
|
|
|
|
struct nfs_pgio_header *hdr)
|
|
|
|
{
|
|
|
|
struct nfs_writeverf *verfp;
|
|
|
|
|
2014-09-16 21:35:51 +00:00
|
|
|
verfp = nfs_direct_select_verf(dreq, hdr->ds_clp, hdr->ds_commit_idx);
|
2014-05-15 15:56:54 +00:00
|
|
|
WARN_ON_ONCE(verfp->committed >= 0);
|
|
|
|
memcpy(verfp, &hdr->verf, sizeof(struct nfs_writeverf));
|
|
|
|
WARN_ON_ONCE(verfp->committed < 0);
|
|
|
|
}
|
|
|
|
|
2016-06-02 01:32:24 +00:00
|
|
|
static int nfs_direct_cmp_verf(const struct nfs_writeverf *v1,
|
|
|
|
const struct nfs_writeverf *v2)
|
|
|
|
{
|
|
|
|
return nfs_write_verifier_cmp(&v1->verifier, &v2->verifier);
|
|
|
|
}
|
|
|
|
|
2014-05-15 15:56:54 +00:00
|
|
|
/*
|
|
|
|
* nfs_direct_cmp_hdr_verf - compare verifier for pgio header
|
|
|
|
* @dreq - direct request possibly spanning multiple servers
|
|
|
|
* @hdr - pageio header to validate against previously seen verf
|
|
|
|
*
|
|
|
|
* set the server's "seen" verf if not initialized.
|
|
|
|
* returns result of comparison between @hdr->verf and the "seen"
|
|
|
|
* verf of the server used by @hdr (DS or MDS)
|
|
|
|
*/
|
|
|
|
static int nfs_direct_set_or_cmp_hdr_verf(struct nfs_direct_req *dreq,
|
|
|
|
struct nfs_pgio_header *hdr)
|
|
|
|
{
|
|
|
|
struct nfs_writeverf *verfp;
|
|
|
|
|
2014-09-16 21:35:51 +00:00
|
|
|
verfp = nfs_direct_select_verf(dreq, hdr->ds_clp, hdr->ds_commit_idx);
|
2014-05-15 15:56:54 +00:00
|
|
|
if (verfp->committed < 0) {
|
|
|
|
nfs_direct_set_hdr_verf(dreq, hdr);
|
|
|
|
return 0;
|
|
|
|
}
|
2016-06-02 01:32:24 +00:00
|
|
|
return nfs_direct_cmp_verf(verfp, &hdr->verf);
|
2014-05-15 15:56:54 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* nfs_direct_cmp_commit_data_verf - compare verifier for commit data
|
|
|
|
* @dreq - direct request possibly spanning multiple servers
|
|
|
|
* @data - commit data to validate against previously seen verf
|
|
|
|
*
|
|
|
|
* returns result of comparison between @data->verf and the verf of
|
|
|
|
* the server used by @data (DS or MDS)
|
|
|
|
*/
|
|
|
|
static int nfs_direct_cmp_commit_data_verf(struct nfs_direct_req *dreq,
|
|
|
|
struct nfs_commit_data *data)
|
|
|
|
{
|
|
|
|
struct nfs_writeverf *verfp;
|
|
|
|
|
|
|
|
verfp = nfs_direct_select_verf(dreq, data->ds_clp,
|
|
|
|
data->ds_commit_index);
|
2014-10-01 16:58:25 +00:00
|
|
|
|
|
|
|
/* verifier not set so always fail */
|
|
|
|
if (verfp->committed < 0)
|
|
|
|
return 1;
|
|
|
|
|
2016-06-02 01:32:24 +00:00
|
|
|
return nfs_direct_cmp_verf(verfp, &data->verf);
|
2014-05-15 15:56:54 +00:00
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/**
|
2006-03-20 18:44:28 +00:00
|
|
|
* nfs_direct_IO - NFS address space operation for direct I/O
|
|
|
|
* @iocb: target I/O control block
|
2016-05-29 23:05:03 +00:00
|
|
|
* @iter: I/O buffer
|
2006-03-20 18:44:28 +00:00
|
|
|
*
|
|
|
|
* The presence of this routine in the address space ops vector means
|
2012-07-31 23:45:12 +00:00
|
|
|
* the NFS client supports direct I/O. However, for most direct IO, we
|
|
|
|
* shunt off direct read and write requests before the VFS gets them,
|
|
|
|
* so this method is only ever called for swap.
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
2016-04-07 15:51:58 +00:00
|
|
|
ssize_t nfs_direct_IO(struct kiocb *iocb, struct iov_iter *iter)
|
2006-03-20 18:44:28 +00:00
|
|
|
{
|
2015-01-19 23:44:29 +00:00
|
|
|
struct inode *inode = iocb->ki_filp->f_mapping->host;
|
|
|
|
|
|
|
|
/* we only support swap file calling nfs_direct_IO */
|
|
|
|
if (!IS_SWAPFILE(inode))
|
|
|
|
return 0;
|
|
|
|
|
2015-02-11 18:56:46 +00:00
|
|
|
VM_BUG_ON(iov_iter_count(iter) != PAGE_SIZE);
|
2012-07-31 23:45:12 +00:00
|
|
|
|
2015-03-16 11:33:52 +00:00
|
|
|
if (iov_iter_rw(iter) == READ)
|
2016-04-07 15:51:58 +00:00
|
|
|
return nfs_file_direct_read(iocb, iter);
|
2015-04-09 18:11:08 +00:00
|
|
|
return nfs_file_direct_write(iocb, iter);
|
2006-03-20 18:44:28 +00:00
|
|
|
}
|
|
|
|
|
2007-05-19 21:22:46 +00:00
|
|
|
static void nfs_direct_release_pages(struct page **pages, unsigned int npages)
|
2006-06-20 16:56:31 +00:00
|
|
|
{
|
2007-05-19 21:22:46 +00:00
|
|
|
unsigned int i;
|
2006-06-28 20:52:45 +00:00
|
|
|
for (i = 0; i < npages; i++)
|
mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros
PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} macros were introduced *long* time
ago with promise that one day it will be possible to implement page
cache with bigger chunks than PAGE_SIZE.
This promise never materialized. And unlikely will.
We have many places where PAGE_CACHE_SIZE assumed to be equal to
PAGE_SIZE. And it's constant source of confusion on whether
PAGE_CACHE_* or PAGE_* constant should be used in a particular case,
especially on the border between fs and mm.
Global switching to PAGE_CACHE_SIZE != PAGE_SIZE would cause to much
breakage to be doable.
Let's stop pretending that pages in page cache are special. They are
not.
The changes are pretty straight-forward:
- <foo> << (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;
- <foo> >> (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;
- PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} -> PAGE_{SIZE,SHIFT,MASK,ALIGN};
- page_cache_get() -> get_page();
- page_cache_release() -> put_page();
This patch contains automated changes generated with coccinelle using
script below. For some reason, coccinelle doesn't patch header files.
I've called spatch for them manually.
The only adjustment after coccinelle is revert of changes to
PAGE_CAHCE_ALIGN definition: we are going to drop it later.
There are few places in the code where coccinelle didn't reach. I'll
fix them manually in a separate patch. Comments and documentation also
will be addressed with the separate patch.
virtual patch
@@
expression E;
@@
- E << (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E
@@
expression E;
@@
- E >> (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E
@@
@@
- PAGE_CACHE_SHIFT
+ PAGE_SHIFT
@@
@@
- PAGE_CACHE_SIZE
+ PAGE_SIZE
@@
@@
- PAGE_CACHE_MASK
+ PAGE_MASK
@@
expression E;
@@
- PAGE_CACHE_ALIGN(E)
+ PAGE_ALIGN(E)
@@
expression E;
@@
- page_cache_get(E)
+ get_page(E)
@@
expression E;
@@
- page_cache_release(E)
+ put_page(E)
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-04-01 12:29:47 +00:00
|
|
|
put_page(pages[i]);
|
2006-03-20 18:44:43 +00:00
|
|
|
}
|
|
|
|
|
2012-04-20 18:47:57 +00:00
|
|
|
void nfs_init_cinfo_from_dreq(struct nfs_commit_info *cinfo,
|
|
|
|
struct nfs_direct_req *dreq)
|
|
|
|
{
|
2016-04-01 17:45:09 +00:00
|
|
|
cinfo->inode = dreq->inode;
|
2012-04-20 18:47:57 +00:00
|
|
|
cinfo->mds = &dreq->mds_cinfo;
|
|
|
|
cinfo->ds = &dreq->ds_cinfo;
|
|
|
|
cinfo->dreq = dreq;
|
|
|
|
cinfo->completion_ops = &nfs_direct_commit_completion_ops;
|
|
|
|
}
|
|
|
|
|
2014-09-19 16:48:33 +00:00
|
|
|
static inline void nfs_direct_setup_mirroring(struct nfs_direct_req *dreq,
|
|
|
|
struct nfs_pageio_descriptor *pgio,
|
|
|
|
struct nfs_page *req)
|
|
|
|
{
|
|
|
|
int mirror_count = 1;
|
|
|
|
|
|
|
|
if (pgio->pg_ops->pg_get_mirror_count)
|
|
|
|
mirror_count = pgio->pg_ops->pg_get_mirror_count(pgio, req);
|
|
|
|
|
|
|
|
dreq->mirror_count = mirror_count;
|
|
|
|
}
|
|
|
|
|
2006-03-20 18:44:31 +00:00
|
|
|
static inline struct nfs_direct_req *nfs_direct_req_alloc(void)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2006-03-20 18:44:31 +00:00
|
|
|
struct nfs_direct_req *dreq;
|
|
|
|
|
2012-04-30 22:31:49 +00:00
|
|
|
dreq = kmem_cache_zalloc(nfs_direct_cachep, GFP_KERNEL);
|
2006-03-20 18:44:31 +00:00
|
|
|
if (!dreq)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
kref_init(&dreq->kref);
|
2006-06-28 20:52:45 +00:00
|
|
|
kref_get(&dreq->kref);
|
2006-03-20 18:44:43 +00:00
|
|
|
init_completion(&dreq->completion);
|
2012-04-20 18:47:57 +00:00
|
|
|
INIT_LIST_HEAD(&dreq->mds_cinfo.list);
|
2014-05-15 15:56:54 +00:00
|
|
|
dreq->verf.committed = NFS_INVALID_STABLE_HOW; /* not set yet */
|
2012-04-20 18:47:57 +00:00
|
|
|
INIT_WORK(&dreq->work, nfs_direct_write_schedule_work);
|
2014-09-19 16:48:33 +00:00
|
|
|
dreq->mirror_count = 1;
|
2006-03-20 18:44:34 +00:00
|
|
|
spin_lock_init(&dreq->lock);
|
2006-03-20 18:44:31 +00:00
|
|
|
|
|
|
|
return dreq;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2007-05-30 16:58:00 +00:00
|
|
|
static void nfs_direct_req_free(struct kref *kref)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
struct nfs_direct_req *dreq = container_of(kref, struct nfs_direct_req, kref);
|
2006-03-20 18:44:36 +00:00
|
|
|
|
2014-11-05 14:36:50 +00:00
|
|
|
nfs_free_pnfs_ds_cinfo(&dreq->ds_cinfo);
|
2010-06-25 20:35:53 +00:00
|
|
|
if (dreq->l_ctx != NULL)
|
|
|
|
nfs_put_lock_context(dreq->l_ctx);
|
2006-03-20 18:44:36 +00:00
|
|
|
if (dreq->ctx != NULL)
|
|
|
|
put_nfs_open_context(dreq->ctx);
|
2005-04-16 22:20:36 +00:00
|
|
|
kmem_cache_free(nfs_direct_cachep, dreq);
|
|
|
|
}
|
|
|
|
|
2007-05-30 16:58:00 +00:00
|
|
|
static void nfs_direct_req_release(struct nfs_direct_req *dreq)
|
|
|
|
{
|
|
|
|
kref_put(&dreq->kref, nfs_direct_req_free);
|
|
|
|
}
|
|
|
|
|
2012-09-25 06:55:57 +00:00
|
|
|
ssize_t nfs_dreq_bytes_left(struct nfs_direct_req *dreq)
|
|
|
|
{
|
|
|
|
return dreq->bytes_left;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(nfs_dreq_bytes_left);
|
|
|
|
|
2006-03-20 18:44:31 +00:00
|
|
|
/*
|
|
|
|
* Collects and returns the final error value/byte-count.
|
|
|
|
*/
|
|
|
|
static ssize_t nfs_direct_wait(struct nfs_direct_req *dreq)
|
|
|
|
{
|
2006-03-20 18:44:34 +00:00
|
|
|
ssize_t result = -EIOCBQUEUED;
|
2006-03-20 18:44:31 +00:00
|
|
|
|
|
|
|
/* Async requests don't wait here */
|
|
|
|
if (dreq->iocb)
|
|
|
|
goto out;
|
|
|
|
|
2007-12-06 21:24:39 +00:00
|
|
|
result = wait_for_completion_killable(&dreq->completion);
|
2006-03-20 18:44:31 +00:00
|
|
|
|
2016-06-17 20:48:22 +00:00
|
|
|
if (!result) {
|
|
|
|
result = dreq->count;
|
|
|
|
WARN_ON_ONCE(dreq->count < 0);
|
|
|
|
}
|
2006-03-20 18:44:31 +00:00
|
|
|
if (!result)
|
2006-03-20 18:44:34 +00:00
|
|
|
result = dreq->error;
|
2006-03-20 18:44:31 +00:00
|
|
|
|
|
|
|
out:
|
|
|
|
return (ssize_t) result;
|
|
|
|
}
|
|
|
|
|
2006-03-20 18:44:31 +00:00
|
|
|
/*
|
2006-06-28 20:52:45 +00:00
|
|
|
* Synchronous I/O uses a stack-allocated iocb. Thus we can't trust
|
|
|
|
* the iocb is still valid here if this is a synchronous request.
|
2006-03-20 18:44:31 +00:00
|
|
|
*/
|
2016-06-23 13:29:47 +00:00
|
|
|
static void nfs_direct_complete(struct nfs_direct_req *dreq)
|
2006-03-20 18:44:31 +00:00
|
|
|
{
|
2013-11-14 16:50:28 +00:00
|
|
|
struct inode *inode = dreq->inode;
|
|
|
|
|
direct-io: only inc/dec inode->i_dio_count for file systems
do_blockdev_direct_IO() increments and decrements the inode
->i_dio_count for each IO operation. It does this to protect against
truncate of a file. Block devices don't need this sort of protection.
For a capable multiqueue setup, this atomic int is the only shared
state between applications accessing the device for O_DIRECT, and it
presents a scaling wall for that. In my testing, as much as 30% of
system time is spent incrementing and decrementing this value. A mixed
read/write workload improved from ~2.5M IOPS to ~9.6M IOPS, with
better latencies too. Before:
clat percentiles (usec):
| 1.00th=[ 33], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34],
| 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 35], 60.00th=[ 35],
| 70.00th=[ 35], 80.00th=[ 35], 90.00th=[ 37], 95.00th=[ 80],
| 99.00th=[ 98], 99.50th=[ 151], 99.90th=[ 155], 99.95th=[ 155],
| 99.99th=[ 165]
After:
clat percentiles (usec):
| 1.00th=[ 95], 5.00th=[ 108], 10.00th=[ 129], 20.00th=[ 149],
| 30.00th=[ 155], 40.00th=[ 161], 50.00th=[ 167], 60.00th=[ 171],
| 70.00th=[ 177], 80.00th=[ 185], 90.00th=[ 201], 95.00th=[ 270],
| 99.00th=[ 390], 99.50th=[ 398], 99.90th=[ 418], 99.95th=[ 422],
| 99.99th=[ 438]
In other setups, Robert Elliott reported seeing good performance
improvements:
https://lkml.org/lkml/2015/4/3/557
The more applications accessing the device, the worse it gets.
Add a new direct-io flags, DIO_SKIP_DIO_COUNT, which tells
do_blockdev_direct_IO() that it need not worry about incrementing
or decrementing the inode i_dio_count for this caller.
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Theodore Ts'o <tytso@mit.edu>
Cc: Elliott, Robert (Server Storage) <elliott@hp.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Jens Axboe <axboe@fb.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2015-04-15 23:05:48 +00:00
|
|
|
inode_dio_end(inode);
|
2013-11-14 16:50:29 +00:00
|
|
|
|
|
|
|
if (dreq->iocb) {
|
2006-03-20 18:44:34 +00:00
|
|
|
long res = (long) dreq->error;
|
2016-06-17 20:48:22 +00:00
|
|
|
if (dreq->count != 0) {
|
2006-03-20 18:44:34 +00:00
|
|
|
res = (long) dreq->count;
|
2016-06-17 20:48:22 +00:00
|
|
|
WARN_ON_ONCE(dreq->count < 0);
|
|
|
|
}
|
2015-02-02 13:49:06 +00:00
|
|
|
dreq->iocb->ki_complete(dreq->iocb, res, 0);
|
2006-03-20 18:44:43 +00:00
|
|
|
}
|
2013-11-14 16:50:29 +00:00
|
|
|
|
2006-03-20 18:44:43 +00:00
|
|
|
complete_all(&dreq->completion);
|
2006-03-20 18:44:31 +00:00
|
|
|
|
2007-05-30 16:58:00 +00:00
|
|
|
nfs_direct_req_release(dreq);
|
2006-03-20 18:44:31 +00:00
|
|
|
}
|
|
|
|
|
2012-05-04 17:54:24 +00:00
|
|
|
static void nfs_direct_readpage_release(struct nfs_page *req)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2013-12-17 17:20:16 +00:00
|
|
|
dprintk("NFS: direct read done (%s/%llu %d@%lld)\n",
|
2016-04-10 05:33:30 +00:00
|
|
|
req->wb_context->dentry->d_sb->s_id,
|
2015-03-17 22:25:59 +00:00
|
|
|
(unsigned long long)NFS_FILEID(d_inode(req->wb_context->dentry)),
|
2012-04-20 18:47:51 +00:00
|
|
|
req->wb_bytes,
|
|
|
|
(long long)req_offset(req));
|
|
|
|
nfs_release_request(req);
|
2008-04-15 20:33:58 +00:00
|
|
|
}
|
|
|
|
|
2012-04-20 18:47:51 +00:00
|
|
|
static void nfs_direct_read_completion(struct nfs_pgio_header *hdr)
|
2008-04-15 20:33:58 +00:00
|
|
|
{
|
2012-04-20 18:47:51 +00:00
|
|
|
unsigned long bytes = 0;
|
|
|
|
struct nfs_direct_req *dreq = hdr->dreq;
|
2008-04-15 20:33:58 +00:00
|
|
|
|
2012-04-20 18:47:51 +00:00
|
|
|
if (test_bit(NFS_IOHDR_REDO, &hdr->flags))
|
|
|
|
goto out_put;
|
2006-03-20 18:44:34 +00:00
|
|
|
|
|
|
|
spin_lock(&dreq->lock);
|
2012-04-20 18:47:51 +00:00
|
|
|
if (test_bit(NFS_IOHDR_ERROR, &hdr->flags) && (hdr->good_bytes == 0))
|
|
|
|
dreq->error = hdr->error;
|
|
|
|
else
|
2014-09-19 16:48:33 +00:00
|
|
|
nfs_direct_good_bytes(dreq, hdr);
|
|
|
|
|
2012-04-20 18:47:51 +00:00
|
|
|
spin_unlock(&dreq->lock);
|
|
|
|
|
2012-05-01 16:49:58 +00:00
|
|
|
while (!list_empty(&hdr->pages)) {
|
|
|
|
struct nfs_page *req = nfs_list_entry(hdr->pages.next);
|
|
|
|
struct page *page = req->wb_page;
|
|
|
|
|
2012-12-12 17:36:31 +00:00
|
|
|
if (!PageCompound(page) && bytes < hdr->good_bytes)
|
|
|
|
set_page_dirty(page);
|
2012-05-01 16:49:58 +00:00
|
|
|
bytes += req->wb_bytes;
|
|
|
|
nfs_list_remove_request(req);
|
|
|
|
nfs_direct_readpage_release(req);
|
2007-05-22 14:22:27 +00:00
|
|
|
}
|
2012-04-20 18:47:51 +00:00
|
|
|
out_put:
|
2006-06-28 20:52:45 +00:00
|
|
|
if (put_dreq(dreq))
|
2016-06-23 13:29:47 +00:00
|
|
|
nfs_direct_complete(dreq);
|
2012-04-20 18:47:51 +00:00
|
|
|
hdr->release(hdr);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2012-04-30 17:40:06 +00:00
|
|
|
static void nfs_read_sync_pgio_error(struct list_head *head)
|
2012-04-20 18:47:44 +00:00
|
|
|
{
|
2012-04-20 18:47:51 +00:00
|
|
|
struct nfs_page *req;
|
2012-04-20 18:47:44 +00:00
|
|
|
|
2012-04-20 18:47:51 +00:00
|
|
|
while (!list_empty(head)) {
|
|
|
|
req = nfs_list_entry(head->next);
|
|
|
|
nfs_list_remove_request(req);
|
|
|
|
nfs_release_request(req);
|
|
|
|
}
|
2012-04-20 18:47:44 +00:00
|
|
|
}
|
|
|
|
|
2012-04-20 18:47:51 +00:00
|
|
|
static void nfs_direct_pgio_init(struct nfs_pgio_header *hdr)
|
|
|
|
{
|
|
|
|
get_dreq(hdr->dreq);
|
|
|
|
}
|
|
|
|
|
|
|
|
static const struct nfs_pgio_completion_ops nfs_direct_read_completion_ops = {
|
2012-04-30 17:40:06 +00:00
|
|
|
.error_cleanup = nfs_read_sync_pgio_error,
|
2012-04-20 18:47:51 +00:00
|
|
|
.init_hdr = nfs_direct_pgio_init,
|
|
|
|
.completion = nfs_direct_read_completion,
|
|
|
|
};
|
|
|
|
|
2006-03-20 18:44:28 +00:00
|
|
|
/*
|
2006-06-28 20:52:45 +00:00
|
|
|
* For each rsize'd chunk of the user's buffer, dispatch an NFS READ
|
|
|
|
* operation. If nfs_readdata_alloc() or get_user_pages() fails,
|
|
|
|
* bail and stop sending more reads. Read length accounting is
|
|
|
|
* handled automatically by nfs_direct_read_result(). Otherwise, if
|
|
|
|
* no requests have been sent, just return an error.
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
|
|
|
|
2014-03-21 08:58:33 +00:00
|
|
|
static ssize_t nfs_direct_read_schedule_iovec(struct nfs_direct_req *dreq,
|
|
|
|
struct iov_iter *iter,
|
|
|
|
loff_t pos)
|
|
|
|
{
|
|
|
|
struct nfs_pageio_descriptor desc;
|
|
|
|
struct inode *inode = dreq->inode;
|
|
|
|
ssize_t result = -EINVAL;
|
|
|
|
size_t requested_bytes = 0;
|
|
|
|
size_t rsize = max_t(size_t, NFS_SERVER(inode)->rsize, PAGE_SIZE);
|
2006-09-08 16:48:54 +00:00
|
|
|
|
2014-06-12 17:30:18 +00:00
|
|
|
nfs_pageio_init_read(&desc, dreq->inode, false,
|
2014-03-21 08:58:33 +00:00
|
|
|
&nfs_direct_read_completion_ops);
|
|
|
|
get_dreq(dreq);
|
|
|
|
desc.pg_dreq = dreq;
|
direct-io: only inc/dec inode->i_dio_count for file systems
do_blockdev_direct_IO() increments and decrements the inode
->i_dio_count for each IO operation. It does this to protect against
truncate of a file. Block devices don't need this sort of protection.
For a capable multiqueue setup, this atomic int is the only shared
state between applications accessing the device for O_DIRECT, and it
presents a scaling wall for that. In my testing, as much as 30% of
system time is spent incrementing and decrementing this value. A mixed
read/write workload improved from ~2.5M IOPS to ~9.6M IOPS, with
better latencies too. Before:
clat percentiles (usec):
| 1.00th=[ 33], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34],
| 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 35], 60.00th=[ 35],
| 70.00th=[ 35], 80.00th=[ 35], 90.00th=[ 37], 95.00th=[ 80],
| 99.00th=[ 98], 99.50th=[ 151], 99.90th=[ 155], 99.95th=[ 155],
| 99.99th=[ 165]
After:
clat percentiles (usec):
| 1.00th=[ 95], 5.00th=[ 108], 10.00th=[ 129], 20.00th=[ 149],
| 30.00th=[ 155], 40.00th=[ 161], 50.00th=[ 167], 60.00th=[ 171],
| 70.00th=[ 177], 80.00th=[ 185], 90.00th=[ 201], 95.00th=[ 270],
| 99.00th=[ 390], 99.50th=[ 398], 99.90th=[ 418], 99.95th=[ 422],
| 99.99th=[ 438]
In other setups, Robert Elliott reported seeing good performance
improvements:
https://lkml.org/lkml/2015/4/3/557
The more applications accessing the device, the worse it gets.
Add a new direct-io flags, DIO_SKIP_DIO_COUNT, which tells
do_blockdev_direct_IO() that it need not worry about incrementing
or decrementing the inode i_dio_count for this caller.
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Theodore Ts'o <tytso@mit.edu>
Cc: Elliott, Robert (Server Storage) <elliott@hp.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Jens Axboe <axboe@fb.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2015-04-15 23:05:48 +00:00
|
|
|
inode_dio_begin(inode);
|
2012-07-31 23:45:12 +00:00
|
|
|
|
2014-03-21 08:58:33 +00:00
|
|
|
while (iov_iter_count(iter)) {
|
|
|
|
struct page **pagevec;
|
|
|
|
size_t bytes;
|
|
|
|
size_t pgbase;
|
|
|
|
unsigned npages, i;
|
2006-06-28 20:52:45 +00:00
|
|
|
|
2014-03-21 08:58:33 +00:00
|
|
|
result = iov_iter_get_pages_alloc(iter, &pagevec,
|
|
|
|
rsize, &pgbase);
|
|
|
|
if (result < 0)
|
|
|
|
break;
|
|
|
|
|
|
|
|
bytes = result;
|
|
|
|
iov_iter_advance(iter, bytes);
|
|
|
|
npages = (result + pgbase + PAGE_SIZE - 1) / PAGE_SIZE;
|
2012-04-20 18:47:51 +00:00
|
|
|
for (i = 0; i < npages; i++) {
|
|
|
|
struct nfs_page *req;
|
2012-05-04 17:47:16 +00:00
|
|
|
unsigned int req_len = min_t(size_t, bytes, PAGE_SIZE - pgbase);
|
2012-04-20 18:47:51 +00:00
|
|
|
/* XXX do we need to do the eof zeroing found in async_filler? */
|
2014-05-15 15:56:45 +00:00
|
|
|
req = nfs_create_request(dreq->ctx, pagevec[i], NULL,
|
2012-04-20 18:47:51 +00:00
|
|
|
pgbase, req_len);
|
|
|
|
if (IS_ERR(req)) {
|
|
|
|
result = PTR_ERR(req);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
req->wb_index = pos >> PAGE_SHIFT;
|
|
|
|
req->wb_offset = pos & ~PAGE_MASK;
|
2014-03-21 08:58:33 +00:00
|
|
|
if (!nfs_pageio_add_request(&desc, req)) {
|
|
|
|
result = desc.pg_error;
|
2012-04-20 18:47:51 +00:00
|
|
|
nfs_release_request(req);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
pgbase = 0;
|
|
|
|
bytes -= req_len;
|
2014-03-21 08:58:33 +00:00
|
|
|
requested_bytes += req_len;
|
2012-04-20 18:47:51 +00:00
|
|
|
pos += req_len;
|
2012-09-25 06:55:57 +00:00
|
|
|
dreq->bytes_left -= req_len;
|
2012-04-20 18:47:51 +00:00
|
|
|
}
|
2012-04-30 17:27:31 +00:00
|
|
|
nfs_direct_release_pages(pagevec, npages);
|
2014-03-21 08:58:33 +00:00
|
|
|
kvfree(pagevec);
|
2007-11-12 17:16:47 +00:00
|
|
|
if (result < 0)
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2012-04-20 18:47:51 +00:00
|
|
|
nfs_pageio_complete(&desc);
|
|
|
|
|
2011-01-21 15:54:57 +00:00
|
|
|
/*
|
|
|
|
* If no bytes were started, return the error, and let the
|
|
|
|
* generic layer handle the completion.
|
|
|
|
*/
|
|
|
|
if (requested_bytes == 0) {
|
direct-io: only inc/dec inode->i_dio_count for file systems
do_blockdev_direct_IO() increments and decrements the inode
->i_dio_count for each IO operation. It does this to protect against
truncate of a file. Block devices don't need this sort of protection.
For a capable multiqueue setup, this atomic int is the only shared
state between applications accessing the device for O_DIRECT, and it
presents a scaling wall for that. In my testing, as much as 30% of
system time is spent incrementing and decrementing this value. A mixed
read/write workload improved from ~2.5M IOPS to ~9.6M IOPS, with
better latencies too. Before:
clat percentiles (usec):
| 1.00th=[ 33], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34],
| 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 35], 60.00th=[ 35],
| 70.00th=[ 35], 80.00th=[ 35], 90.00th=[ 37], 95.00th=[ 80],
| 99.00th=[ 98], 99.50th=[ 151], 99.90th=[ 155], 99.95th=[ 155],
| 99.99th=[ 165]
After:
clat percentiles (usec):
| 1.00th=[ 95], 5.00th=[ 108], 10.00th=[ 129], 20.00th=[ 149],
| 30.00th=[ 155], 40.00th=[ 161], 50.00th=[ 167], 60.00th=[ 171],
| 70.00th=[ 177], 80.00th=[ 185], 90.00th=[ 201], 95.00th=[ 270],
| 99.00th=[ 390], 99.50th=[ 398], 99.90th=[ 418], 99.95th=[ 422],
| 99.99th=[ 438]
In other setups, Robert Elliott reported seeing good performance
improvements:
https://lkml.org/lkml/2015/4/3/557
The more applications accessing the device, the worse it gets.
Add a new direct-io flags, DIO_SKIP_DIO_COUNT, which tells
do_blockdev_direct_IO() that it need not worry about incrementing
or decrementing the inode i_dio_count for this caller.
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Theodore Ts'o <tytso@mit.edu>
Cc: Elliott, Robert (Server Storage) <elliott@hp.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Jens Axboe <axboe@fb.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2015-04-15 23:05:48 +00:00
|
|
|
inode_dio_end(inode);
|
2011-01-21 15:54:57 +00:00
|
|
|
nfs_direct_req_release(dreq);
|
|
|
|
return result < 0 ? result : -EIO;
|
|
|
|
}
|
|
|
|
|
2007-11-12 17:16:47 +00:00
|
|
|
if (put_dreq(dreq))
|
2016-06-23 13:29:47 +00:00
|
|
|
nfs_direct_complete(dreq);
|
2011-01-21 15:54:57 +00:00
|
|
|
return 0;
|
2007-11-12 17:16:47 +00:00
|
|
|
}
|
|
|
|
|
2013-11-14 16:50:31 +00:00
|
|
|
/**
|
|
|
|
* nfs_file_direct_read - file direct read operation for NFS files
|
|
|
|
* @iocb: target I/O control block
|
2014-03-05 02:53:33 +00:00
|
|
|
* @iter: vector of user buffers into which to read data
|
2013-11-14 16:50:31 +00:00
|
|
|
*
|
|
|
|
* We use this function for direct reads instead of calling
|
|
|
|
* generic_file_aio_read() in order to avoid gfar's check to see if
|
|
|
|
* the request starts before the end of the file. For that check
|
|
|
|
* to work, we must generate a GETATTR before each direct read, and
|
|
|
|
* even then there is a window between the GETATTR and the subsequent
|
|
|
|
* READ where the file size could change. Our preference is simply
|
|
|
|
* to do all reads the application wants, and the server will take
|
|
|
|
* care of managing the end of file boundary.
|
|
|
|
*
|
|
|
|
* This function also eliminates unnecessarily updating the file's
|
|
|
|
* atime locally, as the NFS server sets the file's atime, and this
|
|
|
|
* client must read the updated atime from the server back into its
|
|
|
|
* cache.
|
|
|
|
*/
|
2016-04-07 15:51:58 +00:00
|
|
|
ssize_t nfs_file_direct_read(struct kiocb *iocb, struct iov_iter *iter)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2013-11-14 16:50:31 +00:00
|
|
|
struct file *file = iocb->ki_filp;
|
|
|
|
struct address_space *mapping = file->f_mapping;
|
|
|
|
struct inode *inode = mapping->host;
|
2005-04-16 22:20:36 +00:00
|
|
|
struct nfs_direct_req *dreq;
|
2012-08-13 21:15:50 +00:00
|
|
|
struct nfs_lock_context *l_ctx;
|
2013-11-14 16:50:31 +00:00
|
|
|
ssize_t result = -EINVAL;
|
2014-03-05 03:38:00 +00:00
|
|
|
size_t count = iov_iter_count(iter);
|
2013-11-14 16:50:31 +00:00
|
|
|
nfs_add_stats(mapping->host, NFSIOS_DIRECTREADBYTES, count);
|
|
|
|
|
|
|
|
dfprintk(FILE, "NFS: direct read(%pD2, %zd@%Ld)\n",
|
2016-04-07 15:51:58 +00:00
|
|
|
file, count, (long long) iocb->ki_pos);
|
2013-11-14 16:50:31 +00:00
|
|
|
|
|
|
|
result = 0;
|
|
|
|
if (!count)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
task_io_account_read(count);
|
|
|
|
|
|
|
|
result = -ENOMEM;
|
2006-06-28 20:52:45 +00:00
|
|
|
dreq = nfs_direct_req_alloc();
|
2010-06-25 20:35:53 +00:00
|
|
|
if (dreq == NULL)
|
2016-06-03 21:07:19 +00:00
|
|
|
goto out;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2006-03-20 18:44:14 +00:00
|
|
|
dreq->inode = inode;
|
2016-04-01 15:42:29 +00:00
|
|
|
dreq->bytes_left = dreq->max_count = count;
|
2016-04-07 15:51:58 +00:00
|
|
|
dreq->io_start = iocb->ki_pos;
|
2007-08-10 21:44:32 +00:00
|
|
|
dreq->ctx = get_nfs_open_context(nfs_file_open_context(iocb->ki_filp));
|
2012-08-13 21:15:50 +00:00
|
|
|
l_ctx = nfs_get_lock_context(dreq->ctx);
|
|
|
|
if (IS_ERR(l_ctx)) {
|
|
|
|
result = PTR_ERR(l_ctx);
|
2010-06-25 20:35:53 +00:00
|
|
|
goto out_release;
|
2012-08-13 21:15:50 +00:00
|
|
|
}
|
|
|
|
dreq->l_ctx = l_ctx;
|
2006-03-20 18:44:30 +00:00
|
|
|
if (!is_sync_kiocb(iocb))
|
|
|
|
dreq->iocb = iocb;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2016-06-03 21:07:19 +00:00
|
|
|
nfs_start_io_direct(inode);
|
|
|
|
|
2014-03-05 02:53:33 +00:00
|
|
|
NFS_I(inode)->read_io += count;
|
2016-04-07 15:51:58 +00:00
|
|
|
result = nfs_direct_read_schedule_iovec(dreq, iter, iocb->ki_pos);
|
2013-11-14 16:50:33 +00:00
|
|
|
|
2016-06-03 21:07:19 +00:00
|
|
|
nfs_end_io_direct(inode);
|
2013-11-14 16:50:33 +00:00
|
|
|
|
2013-11-14 16:50:31 +00:00
|
|
|
if (!result) {
|
2006-06-28 20:52:45 +00:00
|
|
|
result = nfs_direct_wait(dreq);
|
2013-11-14 16:50:31 +00:00
|
|
|
if (result > 0)
|
2016-04-07 15:51:58 +00:00
|
|
|
iocb->ki_pos += result;
|
2013-11-14 16:50:31 +00:00
|
|
|
}
|
2013-11-14 16:50:33 +00:00
|
|
|
|
2010-06-25 20:35:53 +00:00
|
|
|
out_release:
|
2007-05-30 16:58:00 +00:00
|
|
|
nfs_direct_req_release(dreq);
|
2010-06-25 20:35:53 +00:00
|
|
|
out:
|
2005-04-16 22:20:36 +00:00
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
2014-12-11 18:04:55 +00:00
|
|
|
static void
|
|
|
|
nfs_direct_write_scan_commit_list(struct inode *inode,
|
|
|
|
struct list_head *list,
|
|
|
|
struct nfs_commit_info *cinfo)
|
|
|
|
{
|
2016-04-01 17:45:09 +00:00
|
|
|
spin_lock(&cinfo->inode->i_lock);
|
2014-12-11 18:04:55 +00:00
|
|
|
#ifdef CONFIG_NFS_V4_1
|
|
|
|
if (cinfo->ds != NULL && cinfo->ds->nwritten != 0)
|
|
|
|
NFS_SERVER(inode)->pnfs_curr_ld->recover_commit_reqs(list, cinfo);
|
|
|
|
#endif
|
|
|
|
nfs_scan_commit_list(&cinfo->mds->list, list, cinfo, 0);
|
2016-04-01 17:45:09 +00:00
|
|
|
spin_unlock(&cinfo->inode->i_lock);
|
2014-12-11 18:04:55 +00:00
|
|
|
}
|
|
|
|
|
2006-03-20 18:44:36 +00:00
|
|
|
static void nfs_direct_write_reschedule(struct nfs_direct_req *dreq)
|
|
|
|
{
|
2012-04-20 18:47:57 +00:00
|
|
|
struct nfs_pageio_descriptor desc;
|
|
|
|
struct nfs_page *req, *tmp;
|
|
|
|
LIST_HEAD(reqs);
|
|
|
|
struct nfs_commit_info cinfo;
|
|
|
|
LIST_HEAD(failed);
|
2014-09-19 16:48:33 +00:00
|
|
|
int i;
|
2012-04-20 18:47:57 +00:00
|
|
|
|
|
|
|
nfs_init_cinfo_from_dreq(&cinfo, dreq);
|
2014-12-11 18:04:55 +00:00
|
|
|
nfs_direct_write_scan_commit_list(dreq->inode, &reqs, &cinfo);
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2006-03-20 18:44:36 +00:00
|
|
|
dreq->count = 0;
|
2016-06-02 01:42:32 +00:00
|
|
|
dreq->verf.committed = NFS_INVALID_STABLE_HOW;
|
|
|
|
nfs_clear_pnfs_ds_commit_verifiers(&dreq->ds_cinfo);
|
2014-09-19 16:48:33 +00:00
|
|
|
for (i = 0; i < dreq->mirror_count; i++)
|
|
|
|
dreq->mirrors[i].count = 0;
|
2006-06-28 20:52:45 +00:00
|
|
|
get_dreq(dreq);
|
|
|
|
|
2014-04-16 13:07:21 +00:00
|
|
|
nfs_pageio_init_write(&desc, dreq->inode, FLUSH_STABLE, false,
|
2012-04-20 18:47:57 +00:00
|
|
|
&nfs_direct_write_completion_ops);
|
|
|
|
desc.pg_dreq = dreq;
|
2006-06-20 16:55:45 +00:00
|
|
|
|
2014-09-19 16:48:33 +00:00
|
|
|
req = nfs_list_entry(reqs.next);
|
|
|
|
nfs_direct_setup_mirroring(dreq, &desc, req);
|
2015-12-03 18:57:48 +00:00
|
|
|
if (desc.pg_error < 0) {
|
|
|
|
list_splice_init(&reqs, &failed);
|
|
|
|
goto out_failed;
|
|
|
|
}
|
2014-09-19 16:48:33 +00:00
|
|
|
|
2012-04-20 18:47:57 +00:00
|
|
|
list_for_each_entry_safe(req, tmp, &reqs, wb_list) {
|
|
|
|
if (!nfs_pageio_add_request(&desc, req)) {
|
2012-07-08 14:24:10 +00:00
|
|
|
nfs_list_remove_request(req);
|
2012-04-20 18:47:57 +00:00
|
|
|
nfs_list_add_request(req, &failed);
|
2016-04-01 17:45:09 +00:00
|
|
|
spin_lock(&cinfo.inode->i_lock);
|
2012-04-20 18:47:57 +00:00
|
|
|
dreq->flags = 0;
|
2015-12-03 18:57:48 +00:00
|
|
|
if (desc.pg_error < 0)
|
|
|
|
dreq->error = desc.pg_error;
|
|
|
|
else
|
|
|
|
dreq->error = -EIO;
|
2016-04-01 17:45:09 +00:00
|
|
|
spin_unlock(&cinfo.inode->i_lock);
|
2012-04-20 18:47:57 +00:00
|
|
|
}
|
2012-06-19 17:39:14 +00:00
|
|
|
nfs_release_request(req);
|
2012-04-20 18:47:57 +00:00
|
|
|
}
|
|
|
|
nfs_pageio_complete(&desc);
|
2006-03-20 18:44:36 +00:00
|
|
|
|
2015-12-03 18:57:48 +00:00
|
|
|
out_failed:
|
2012-07-08 14:24:10 +00:00
|
|
|
while (!list_empty(&failed)) {
|
|
|
|
req = nfs_list_entry(failed.next);
|
|
|
|
nfs_list_remove_request(req);
|
2012-05-09 18:04:55 +00:00
|
|
|
nfs_unlock_and_release_request(req);
|
2012-07-08 14:24:10 +00:00
|
|
|
}
|
2006-03-20 18:44:36 +00:00
|
|
|
|
2012-04-20 18:47:57 +00:00
|
|
|
if (put_dreq(dreq))
|
|
|
|
nfs_direct_write_complete(dreq, dreq->inode);
|
2008-04-15 20:56:39 +00:00
|
|
|
}
|
|
|
|
|
2012-04-20 18:47:57 +00:00
|
|
|
static void nfs_direct_commit_complete(struct nfs_commit_data *data)
|
2008-04-15 20:56:39 +00:00
|
|
|
{
|
2012-04-20 18:47:39 +00:00
|
|
|
struct nfs_direct_req *dreq = data->dreq;
|
2012-04-20 18:47:57 +00:00
|
|
|
struct nfs_commit_info cinfo;
|
|
|
|
struct nfs_page *req;
|
2008-04-15 20:56:39 +00:00
|
|
|
int status = data->task.tk_status;
|
|
|
|
|
2012-04-20 18:47:57 +00:00
|
|
|
nfs_init_cinfo_from_dreq(&cinfo, dreq);
|
2008-04-15 20:56:39 +00:00
|
|
|
if (status < 0) {
|
2007-04-14 23:11:52 +00:00
|
|
|
dprintk("NFS: %5u commit failed with error %d.\n",
|
2012-04-20 18:47:57 +00:00
|
|
|
data->task.tk_pid, status);
|
2006-03-20 18:44:36 +00:00
|
|
|
dreq->flags = NFS_ODIRECT_RESCHED_WRITES;
|
2014-05-15 15:56:54 +00:00
|
|
|
} else if (nfs_direct_cmp_commit_data_verf(dreq, data)) {
|
2008-04-15 20:56:39 +00:00
|
|
|
dprintk("NFS: %5u commit verify failed\n", data->task.tk_pid);
|
2006-03-20 18:44:36 +00:00
|
|
|
dreq->flags = NFS_ODIRECT_RESCHED_WRITES;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2008-04-15 20:56:39 +00:00
|
|
|
dprintk("NFS: %5u commit returned %d\n", data->task.tk_pid, status);
|
2012-04-20 18:47:57 +00:00
|
|
|
while (!list_empty(&data->pages)) {
|
|
|
|
req = nfs_list_entry(data->pages.next);
|
|
|
|
nfs_list_remove_request(req);
|
|
|
|
if (dreq->flags == NFS_ODIRECT_RESCHED_WRITES) {
|
|
|
|
/* Note the rewrite will go through mds */
|
2014-09-05 22:20:21 +00:00
|
|
|
nfs_mark_request_commit(req, NULL, &cinfo, 0);
|
2012-06-08 20:48:33 +00:00
|
|
|
} else
|
|
|
|
nfs_release_request(req);
|
2012-05-09 18:04:55 +00:00
|
|
|
nfs_unlock_and_release_request(req);
|
2012-04-20 18:47:57 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
if (atomic_dec_and_test(&cinfo.mds->rpcs_out))
|
|
|
|
nfs_direct_write_complete(dreq, data->inode);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2015-12-31 14:28:06 +00:00
|
|
|
static void nfs_direct_resched_write(struct nfs_commit_info *cinfo,
|
|
|
|
struct nfs_page *req)
|
2012-04-20 18:47:57 +00:00
|
|
|
{
|
2015-12-31 14:28:06 +00:00
|
|
|
struct nfs_direct_req *dreq = cinfo->dreq;
|
|
|
|
|
|
|
|
spin_lock(&dreq->lock);
|
|
|
|
dreq->flags = NFS_ODIRECT_RESCHED_WRITES;
|
|
|
|
spin_unlock(&dreq->lock);
|
|
|
|
nfs_mark_request_commit(req, NULL, cinfo, 0);
|
2012-04-20 18:47:57 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static const struct nfs_commit_completion_ops nfs_direct_commit_completion_ops = {
|
|
|
|
.completion = nfs_direct_commit_complete,
|
2015-12-31 14:28:06 +00:00
|
|
|
.resched_write = nfs_direct_resched_write,
|
2006-03-20 18:44:36 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
static void nfs_direct_commit_schedule(struct nfs_direct_req *dreq)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2012-04-20 18:47:57 +00:00
|
|
|
int res;
|
|
|
|
struct nfs_commit_info cinfo;
|
|
|
|
LIST_HEAD(mds_list);
|
|
|
|
|
|
|
|
nfs_init_cinfo_from_dreq(&cinfo, dreq);
|
|
|
|
nfs_scan_commit(dreq->inode, &mds_list, &cinfo);
|
|
|
|
res = nfs_generic_commit_list(dreq->inode, &mds_list, 0, &cinfo);
|
|
|
|
if (res < 0) /* res == -ENOMEM */
|
|
|
|
nfs_direct_write_reschedule(dreq);
|
2006-03-20 18:44:36 +00:00
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2012-04-20 18:47:57 +00:00
|
|
|
static void nfs_direct_write_schedule_work(struct work_struct *work)
|
2006-03-20 18:44:36 +00:00
|
|
|
{
|
2012-04-20 18:47:57 +00:00
|
|
|
struct nfs_direct_req *dreq = container_of(work, struct nfs_direct_req, work);
|
2006-03-20 18:44:36 +00:00
|
|
|
int flags = dreq->flags;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2006-03-20 18:44:36 +00:00
|
|
|
dreq->flags = 0;
|
|
|
|
switch (flags) {
|
|
|
|
case NFS_ODIRECT_DO_COMMIT:
|
|
|
|
nfs_direct_commit_schedule(dreq);
|
2005-04-16 22:20:36 +00:00
|
|
|
break;
|
2006-03-20 18:44:36 +00:00
|
|
|
case NFS_ODIRECT_RESCHED_WRITES:
|
|
|
|
nfs_direct_write_reschedule(dreq);
|
|
|
|
break;
|
|
|
|
default:
|
2016-06-23 13:29:47 +00:00
|
|
|
nfs_zap_mapping(dreq->inode, dreq->inode->i_mapping);
|
|
|
|
nfs_direct_complete(dreq);
|
2006-03-20 18:44:36 +00:00
|
|
|
}
|
|
|
|
}
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2012-04-20 18:47:57 +00:00
|
|
|
static void nfs_direct_write_complete(struct nfs_direct_req *dreq, struct inode *inode)
|
2006-03-20 18:44:36 +00:00
|
|
|
{
|
2012-04-20 18:47:57 +00:00
|
|
|
schedule_work(&dreq->work); /* Calls nfs_direct_write_schedule_work */
|
2006-03-20 18:44:36 +00:00
|
|
|
}
|
2012-04-20 18:47:57 +00:00
|
|
|
|
|
|
|
static void nfs_direct_write_completion(struct nfs_pgio_header *hdr)
|
|
|
|
{
|
|
|
|
struct nfs_direct_req *dreq = hdr->dreq;
|
|
|
|
struct nfs_commit_info cinfo;
|
2014-06-09 15:48:36 +00:00
|
|
|
bool request_commit = false;
|
2012-04-20 18:47:57 +00:00
|
|
|
struct nfs_page *req = nfs_list_entry(hdr->pages.next);
|
|
|
|
|
|
|
|
if (test_bit(NFS_IOHDR_REDO, &hdr->flags))
|
|
|
|
goto out_put;
|
|
|
|
|
|
|
|
nfs_init_cinfo_from_dreq(&cinfo, dreq);
|
|
|
|
|
|
|
|
spin_lock(&dreq->lock);
|
|
|
|
|
|
|
|
if (test_bit(NFS_IOHDR_ERROR, &hdr->flags)) {
|
|
|
|
dreq->flags = 0;
|
|
|
|
dreq->error = hdr->error;
|
|
|
|
}
|
2014-06-09 15:48:36 +00:00
|
|
|
if (dreq->error == 0) {
|
2014-09-19 16:48:33 +00:00
|
|
|
nfs_direct_good_bytes(dreq, hdr);
|
2014-06-09 15:48:36 +00:00
|
|
|
if (nfs_write_need_commit(hdr)) {
|
2012-04-20 18:47:57 +00:00
|
|
|
if (dreq->flags == NFS_ODIRECT_RESCHED_WRITES)
|
2014-06-09 15:48:36 +00:00
|
|
|
request_commit = true;
|
2012-04-20 18:47:57 +00:00
|
|
|
else if (dreq->flags == 0) {
|
2014-05-15 15:56:54 +00:00
|
|
|
nfs_direct_set_hdr_verf(dreq, hdr);
|
2014-06-09 15:48:36 +00:00
|
|
|
request_commit = true;
|
2012-04-20 18:47:57 +00:00
|
|
|
dreq->flags = NFS_ODIRECT_DO_COMMIT;
|
|
|
|
} else if (dreq->flags == NFS_ODIRECT_DO_COMMIT) {
|
2014-06-09 15:48:36 +00:00
|
|
|
request_commit = true;
|
|
|
|
if (nfs_direct_set_or_cmp_hdr_verf(dreq, hdr))
|
2014-05-15 15:56:54 +00:00
|
|
|
dreq->flags =
|
|
|
|
NFS_ODIRECT_RESCHED_WRITES;
|
2012-04-20 18:47:57 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
spin_unlock(&dreq->lock);
|
|
|
|
|
|
|
|
while (!list_empty(&hdr->pages)) {
|
2014-05-15 15:56:45 +00:00
|
|
|
|
2012-04-20 18:47:57 +00:00
|
|
|
req = nfs_list_entry(hdr->pages.next);
|
|
|
|
nfs_list_remove_request(req);
|
2014-06-09 15:48:36 +00:00
|
|
|
if (request_commit) {
|
2012-05-09 17:54:53 +00:00
|
|
|
kref_get(&req->wb_kref);
|
2014-09-05 22:20:21 +00:00
|
|
|
nfs_mark_request_commit(req, hdr->lseg, &cinfo,
|
|
|
|
hdr->ds_commit_idx);
|
2012-04-20 18:47:57 +00:00
|
|
|
}
|
2012-05-09 18:04:55 +00:00
|
|
|
nfs_unlock_and_release_request(req);
|
2012-04-20 18:47:57 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
out_put:
|
|
|
|
if (put_dreq(dreq))
|
|
|
|
nfs_direct_write_complete(dreq, hdr->inode);
|
|
|
|
hdr->release(hdr);
|
|
|
|
}
|
|
|
|
|
2012-04-30 17:40:06 +00:00
|
|
|
static void nfs_write_sync_pgio_error(struct list_head *head)
|
|
|
|
{
|
|
|
|
struct nfs_page *req;
|
|
|
|
|
|
|
|
while (!list_empty(head)) {
|
|
|
|
req = nfs_list_entry(head->next);
|
|
|
|
nfs_list_remove_request(req);
|
2012-05-09 18:04:55 +00:00
|
|
|
nfs_unlock_and_release_request(req);
|
2012-04-30 17:40:06 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-12-31 16:44:06 +00:00
|
|
|
static void nfs_direct_write_reschedule_io(struct nfs_pgio_header *hdr)
|
|
|
|
{
|
|
|
|
struct nfs_direct_req *dreq = hdr->dreq;
|
|
|
|
|
|
|
|
spin_lock(&dreq->lock);
|
|
|
|
if (dreq->error == 0) {
|
|
|
|
dreq->flags = NFS_ODIRECT_RESCHED_WRITES;
|
|
|
|
/* fake unstable write to let common nfs resend pages */
|
|
|
|
hdr->verf.committed = NFS_UNSTABLE;
|
|
|
|
hdr->good_bytes = hdr->args.count;
|
|
|
|
}
|
|
|
|
spin_unlock(&dreq->lock);
|
|
|
|
}
|
|
|
|
|
2012-04-20 18:47:57 +00:00
|
|
|
static const struct nfs_pgio_completion_ops nfs_direct_write_completion_ops = {
|
2012-04-30 17:40:06 +00:00
|
|
|
.error_cleanup = nfs_write_sync_pgio_error,
|
2012-04-20 18:47:57 +00:00
|
|
|
.init_hdr = nfs_direct_pgio_init,
|
|
|
|
.completion = nfs_direct_write_completion,
|
2015-12-31 16:44:06 +00:00
|
|
|
.reschedule_io = nfs_direct_write_reschedule_io,
|
2012-04-20 18:47:57 +00:00
|
|
|
};
|
|
|
|
|
2014-03-21 08:58:33 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* NB: Return the value of the first error return code. Subsequent
|
|
|
|
* errors after the first one are ignored.
|
|
|
|
*/
|
|
|
|
/*
|
|
|
|
* For each wsize'd chunk of the user's buffer, dispatch an NFS WRITE
|
|
|
|
* operation. If nfs_writedata_alloc() or get_user_pages() fails,
|
|
|
|
* bail and stop sending more writes. Write length accounting is
|
|
|
|
* handled automatically by nfs_direct_write_result(). Otherwise, if
|
|
|
|
* no requests have been sent, just return an error.
|
|
|
|
*/
|
2007-11-12 17:16:47 +00:00
|
|
|
static ssize_t nfs_direct_write_schedule_iovec(struct nfs_direct_req *dreq,
|
2014-03-05 02:53:33 +00:00
|
|
|
struct iov_iter *iter,
|
2014-03-21 08:58:33 +00:00
|
|
|
loff_t pos)
|
2007-11-12 17:16:47 +00:00
|
|
|
{
|
2012-04-20 18:47:57 +00:00
|
|
|
struct nfs_pageio_descriptor desc;
|
2012-05-31 16:22:33 +00:00
|
|
|
struct inode *inode = dreq->inode;
|
2007-11-12 17:16:47 +00:00
|
|
|
ssize_t result = 0;
|
|
|
|
size_t requested_bytes = 0;
|
2014-03-21 08:58:33 +00:00
|
|
|
size_t wsize = max_t(size_t, NFS_SERVER(inode)->wsize, PAGE_SIZE);
|
2007-11-12 17:16:47 +00:00
|
|
|
|
2014-04-16 13:07:21 +00:00
|
|
|
nfs_pageio_init_write(&desc, inode, FLUSH_COND_STABLE, false,
|
2012-04-20 18:47:57 +00:00
|
|
|
&nfs_direct_write_completion_ops);
|
|
|
|
desc.pg_dreq = dreq;
|
2007-11-12 17:16:47 +00:00
|
|
|
get_dreq(dreq);
|
direct-io: only inc/dec inode->i_dio_count for file systems
do_blockdev_direct_IO() increments and decrements the inode
->i_dio_count for each IO operation. It does this to protect against
truncate of a file. Block devices don't need this sort of protection.
For a capable multiqueue setup, this atomic int is the only shared
state between applications accessing the device for O_DIRECT, and it
presents a scaling wall for that. In my testing, as much as 30% of
system time is spent incrementing and decrementing this value. A mixed
read/write workload improved from ~2.5M IOPS to ~9.6M IOPS, with
better latencies too. Before:
clat percentiles (usec):
| 1.00th=[ 33], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34],
| 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 35], 60.00th=[ 35],
| 70.00th=[ 35], 80.00th=[ 35], 90.00th=[ 37], 95.00th=[ 80],
| 99.00th=[ 98], 99.50th=[ 151], 99.90th=[ 155], 99.95th=[ 155],
| 99.99th=[ 165]
After:
clat percentiles (usec):
| 1.00th=[ 95], 5.00th=[ 108], 10.00th=[ 129], 20.00th=[ 149],
| 30.00th=[ 155], 40.00th=[ 161], 50.00th=[ 167], 60.00th=[ 171],
| 70.00th=[ 177], 80.00th=[ 185], 90.00th=[ 201], 95.00th=[ 270],
| 99.00th=[ 390], 99.50th=[ 398], 99.90th=[ 418], 99.95th=[ 422],
| 99.99th=[ 438]
In other setups, Robert Elliott reported seeing good performance
improvements:
https://lkml.org/lkml/2015/4/3/557
The more applications accessing the device, the worse it gets.
Add a new direct-io flags, DIO_SKIP_DIO_COUNT, which tells
do_blockdev_direct_IO() that it need not worry about incrementing
or decrementing the inode i_dio_count for this caller.
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Theodore Ts'o <tytso@mit.edu>
Cc: Elliott, Robert (Server Storage) <elliott@hp.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Jens Axboe <axboe@fb.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2015-04-15 23:05:48 +00:00
|
|
|
inode_dio_begin(inode);
|
2007-11-12 17:16:47 +00:00
|
|
|
|
2014-03-21 08:58:33 +00:00
|
|
|
NFS_I(inode)->write_io += iov_iter_count(iter);
|
|
|
|
while (iov_iter_count(iter)) {
|
|
|
|
struct page **pagevec;
|
|
|
|
size_t bytes;
|
|
|
|
size_t pgbase;
|
|
|
|
unsigned npages, i;
|
|
|
|
|
|
|
|
result = iov_iter_get_pages_alloc(iter, &pagevec,
|
|
|
|
wsize, &pgbase);
|
2007-11-12 17:16:47 +00:00
|
|
|
if (result < 0)
|
|
|
|
break;
|
2014-03-21 08:58:33 +00:00
|
|
|
|
|
|
|
bytes = result;
|
|
|
|
iov_iter_advance(iter, bytes);
|
|
|
|
npages = (result + pgbase + PAGE_SIZE - 1) / PAGE_SIZE;
|
|
|
|
for (i = 0; i < npages; i++) {
|
|
|
|
struct nfs_page *req;
|
|
|
|
unsigned int req_len = min_t(size_t, bytes, PAGE_SIZE - pgbase);
|
|
|
|
|
2014-06-12 17:30:18 +00:00
|
|
|
req = nfs_create_request(dreq->ctx, pagevec[i], NULL,
|
2014-03-21 08:58:33 +00:00
|
|
|
pgbase, req_len);
|
|
|
|
if (IS_ERR(req)) {
|
|
|
|
result = PTR_ERR(req);
|
|
|
|
break;
|
|
|
|
}
|
2014-09-19 16:48:33 +00:00
|
|
|
|
|
|
|
nfs_direct_setup_mirroring(dreq, &desc, req);
|
2015-12-03 18:57:48 +00:00
|
|
|
if (desc.pg_error < 0) {
|
|
|
|
nfs_free_request(req);
|
|
|
|
result = desc.pg_error;
|
|
|
|
break;
|
|
|
|
}
|
2014-09-19 16:48:33 +00:00
|
|
|
|
2014-03-21 08:58:33 +00:00
|
|
|
nfs_lock_request(req);
|
|
|
|
req->wb_index = pos >> PAGE_SHIFT;
|
|
|
|
req->wb_offset = pos & ~PAGE_MASK;
|
|
|
|
if (!nfs_pageio_add_request(&desc, req)) {
|
|
|
|
result = desc.pg_error;
|
|
|
|
nfs_unlock_and_release_request(req);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
pgbase = 0;
|
|
|
|
bytes -= req_len;
|
|
|
|
requested_bytes += req_len;
|
|
|
|
pos += req_len;
|
|
|
|
dreq->bytes_left -= req_len;
|
|
|
|
}
|
|
|
|
nfs_direct_release_pages(pagevec, npages);
|
|
|
|
kvfree(pagevec);
|
|
|
|
if (result < 0)
|
2007-11-12 17:16:47 +00:00
|
|
|
break;
|
|
|
|
}
|
2012-04-20 18:47:57 +00:00
|
|
|
nfs_pageio_complete(&desc);
|
2007-11-12 17:16:47 +00:00
|
|
|
|
2011-01-21 15:54:57 +00:00
|
|
|
/*
|
|
|
|
* If no bytes were started, return the error, and let the
|
|
|
|
* generic layer handle the completion.
|
|
|
|
*/
|
|
|
|
if (requested_bytes == 0) {
|
direct-io: only inc/dec inode->i_dio_count for file systems
do_blockdev_direct_IO() increments and decrements the inode
->i_dio_count for each IO operation. It does this to protect against
truncate of a file. Block devices don't need this sort of protection.
For a capable multiqueue setup, this atomic int is the only shared
state between applications accessing the device for O_DIRECT, and it
presents a scaling wall for that. In my testing, as much as 30% of
system time is spent incrementing and decrementing this value. A mixed
read/write workload improved from ~2.5M IOPS to ~9.6M IOPS, with
better latencies too. Before:
clat percentiles (usec):
| 1.00th=[ 33], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34],
| 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 35], 60.00th=[ 35],
| 70.00th=[ 35], 80.00th=[ 35], 90.00th=[ 37], 95.00th=[ 80],
| 99.00th=[ 98], 99.50th=[ 151], 99.90th=[ 155], 99.95th=[ 155],
| 99.99th=[ 165]
After:
clat percentiles (usec):
| 1.00th=[ 95], 5.00th=[ 108], 10.00th=[ 129], 20.00th=[ 149],
| 30.00th=[ 155], 40.00th=[ 161], 50.00th=[ 167], 60.00th=[ 171],
| 70.00th=[ 177], 80.00th=[ 185], 90.00th=[ 201], 95.00th=[ 270],
| 99.00th=[ 390], 99.50th=[ 398], 99.90th=[ 418], 99.95th=[ 422],
| 99.99th=[ 438]
In other setups, Robert Elliott reported seeing good performance
improvements:
https://lkml.org/lkml/2015/4/3/557
The more applications accessing the device, the worse it gets.
Add a new direct-io flags, DIO_SKIP_DIO_COUNT, which tells
do_blockdev_direct_IO() that it need not worry about incrementing
or decrementing the inode i_dio_count for this caller.
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Theodore Ts'o <tytso@mit.edu>
Cc: Elliott, Robert (Server Storage) <elliott@hp.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Jens Axboe <axboe@fb.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2015-04-15 23:05:48 +00:00
|
|
|
inode_dio_end(inode);
|
2011-01-21 15:54:57 +00:00
|
|
|
nfs_direct_req_release(dreq);
|
|
|
|
return result < 0 ? result : -EIO;
|
|
|
|
}
|
|
|
|
|
2007-11-12 17:16:47 +00:00
|
|
|
if (put_dreq(dreq))
|
|
|
|
nfs_direct_write_complete(dreq, dreq->inode);
|
2011-01-21 15:54:57 +00:00
|
|
|
return 0;
|
2007-11-12 17:16:47 +00:00
|
|
|
}
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/**
|
|
|
|
* nfs_file_direct_write - file direct write operation for NFS files
|
|
|
|
* @iocb: target I/O control block
|
2014-03-05 02:53:33 +00:00
|
|
|
* @iter: vector of user buffers from which to write data
|
2005-04-16 22:20:36 +00:00
|
|
|
*
|
|
|
|
* We use this function for direct writes instead of calling
|
|
|
|
* generic_file_aio_write() in order to avoid taking the inode
|
|
|
|
* semaphore and updating the i_size. The NFS server will set
|
|
|
|
* the new i_size and this client must read the updated size
|
|
|
|
* back into its cache. We let the server do generic write
|
|
|
|
* parameter checking and report problems.
|
|
|
|
*
|
|
|
|
* We eliminate local atime updates, see direct read above.
|
|
|
|
*
|
|
|
|
* We avoid unnecessary page cache invalidations for normal cached
|
|
|
|
* readers of this file.
|
|
|
|
*
|
|
|
|
* Note that O_APPEND is not supported for NFS direct writes, as there
|
|
|
|
* is no atomic O_APPEND write facility in the NFS protocol.
|
|
|
|
*/
|
2015-04-09 18:11:08 +00:00
|
|
|
ssize_t nfs_file_direct_write(struct kiocb *iocb, struct iov_iter *iter)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2013-11-14 16:50:32 +00:00
|
|
|
ssize_t result = -EINVAL;
|
2016-06-23 14:35:48 +00:00
|
|
|
size_t count;
|
2005-04-16 22:20:36 +00:00
|
|
|
struct file *file = iocb->ki_filp;
|
|
|
|
struct address_space *mapping = file->f_mapping;
|
2013-11-14 16:50:32 +00:00
|
|
|
struct inode *inode = mapping->host;
|
|
|
|
struct nfs_direct_req *dreq;
|
|
|
|
struct nfs_lock_context *l_ctx;
|
2015-04-09 18:11:08 +00:00
|
|
|
loff_t pos, end;
|
2007-11-12 17:16:52 +00:00
|
|
|
|
2013-09-16 14:53:17 +00:00
|
|
|
dfprintk(FILE, "NFS: direct write(%pD2, %zd@%Ld)\n",
|
2015-04-09 16:55:47 +00:00
|
|
|
file, iov_iter_count(iter), (long long) iocb->ki_pos);
|
2006-10-01 06:28:46 +00:00
|
|
|
|
2016-06-23 14:35:48 +00:00
|
|
|
result = generic_write_checks(iocb, iter);
|
|
|
|
if (result <= 0)
|
|
|
|
return result;
|
|
|
|
count = result;
|
|
|
|
nfs_add_stats(mapping->host, NFSIOS_DIRECTWRITTENBYTES, count);
|
2015-04-09 16:55:47 +00:00
|
|
|
|
|
|
|
pos = iocb->ki_pos;
|
mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros
PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} macros were introduced *long* time
ago with promise that one day it will be possible to implement page
cache with bigger chunks than PAGE_SIZE.
This promise never materialized. And unlikely will.
We have many places where PAGE_CACHE_SIZE assumed to be equal to
PAGE_SIZE. And it's constant source of confusion on whether
PAGE_CACHE_* or PAGE_* constant should be used in a particular case,
especially on the border between fs and mm.
Global switching to PAGE_CACHE_SIZE != PAGE_SIZE would cause to much
breakage to be doable.
Let's stop pretending that pages in page cache are special. They are
not.
The changes are pretty straight-forward:
- <foo> << (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;
- <foo> >> (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;
- PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} -> PAGE_{SIZE,SHIFT,MASK,ALIGN};
- page_cache_get() -> get_page();
- page_cache_release() -> put_page();
This patch contains automated changes generated with coccinelle using
script below. For some reason, coccinelle doesn't patch header files.
I've called spatch for them manually.
The only adjustment after coccinelle is revert of changes to
PAGE_CAHCE_ALIGN definition: we are going to drop it later.
There are few places in the code where coccinelle didn't reach. I'll
fix them manually in a separate patch. Comments and documentation also
will be addressed with the separate patch.
virtual patch
@@
expression E;
@@
- E << (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E
@@
expression E;
@@
- E >> (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E
@@
@@
- PAGE_CACHE_SHIFT
+ PAGE_SHIFT
@@
@@
- PAGE_CACHE_SIZE
+ PAGE_SIZE
@@
@@
- PAGE_CACHE_MASK
+ PAGE_MASK
@@
expression E;
@@
- PAGE_CACHE_ALIGN(E)
+ PAGE_ALIGN(E)
@@
expression E;
@@
- page_cache_get(E)
+ get_page(E)
@@
expression E;
@@
- page_cache_release(E)
+ put_page(E)
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-04-01 12:29:47 +00:00
|
|
|
end = (pos + iov_iter_count(iter) - 1) >> PAGE_SHIFT;
|
2005-11-30 23:08:17 +00:00
|
|
|
|
2016-06-23 14:35:48 +00:00
|
|
|
task_io_account_write(count);
|
2011-02-21 21:28:34 +00:00
|
|
|
|
2013-11-14 16:50:32 +00:00
|
|
|
result = -ENOMEM;
|
|
|
|
dreq = nfs_direct_req_alloc();
|
|
|
|
if (!dreq)
|
2016-06-03 21:07:19 +00:00
|
|
|
goto out;
|
2006-03-20 18:44:33 +00:00
|
|
|
|
2013-11-14 16:50:32 +00:00
|
|
|
dreq->inode = inode;
|
2016-06-23 14:35:48 +00:00
|
|
|
dreq->bytes_left = dreq->max_count = count;
|
2015-01-19 04:41:16 +00:00
|
|
|
dreq->io_start = pos;
|
2013-11-14 16:50:32 +00:00
|
|
|
dreq->ctx = get_nfs_open_context(nfs_file_open_context(iocb->ki_filp));
|
|
|
|
l_ctx = nfs_get_lock_context(dreq->ctx);
|
|
|
|
if (IS_ERR(l_ctx)) {
|
|
|
|
result = PTR_ERR(l_ctx);
|
|
|
|
goto out_release;
|
|
|
|
}
|
|
|
|
dreq->l_ctx = l_ctx;
|
|
|
|
if (!is_sync_kiocb(iocb))
|
|
|
|
dreq->iocb = iocb;
|
|
|
|
|
2016-06-03 21:07:19 +00:00
|
|
|
nfs_start_io_direct(inode);
|
|
|
|
|
2014-03-21 08:58:33 +00:00
|
|
|
result = nfs_direct_write_schedule_iovec(dreq, iter, pos);
|
2013-11-14 16:50:34 +00:00
|
|
|
|
|
|
|
if (mapping->nrpages) {
|
|
|
|
invalidate_inode_pages2_range(mapping,
|
mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros
PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} macros were introduced *long* time
ago with promise that one day it will be possible to implement page
cache with bigger chunks than PAGE_SIZE.
This promise never materialized. And unlikely will.
We have many places where PAGE_CACHE_SIZE assumed to be equal to
PAGE_SIZE. And it's constant source of confusion on whether
PAGE_CACHE_* or PAGE_* constant should be used in a particular case,
especially on the border between fs and mm.
Global switching to PAGE_CACHE_SIZE != PAGE_SIZE would cause to much
breakage to be doable.
Let's stop pretending that pages in page cache are special. They are
not.
The changes are pretty straight-forward:
- <foo> << (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;
- <foo> >> (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;
- PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} -> PAGE_{SIZE,SHIFT,MASK,ALIGN};
- page_cache_get() -> get_page();
- page_cache_release() -> put_page();
This patch contains automated changes generated with coccinelle using
script below. For some reason, coccinelle doesn't patch header files.
I've called spatch for them manually.
The only adjustment after coccinelle is revert of changes to
PAGE_CAHCE_ALIGN definition: we are going to drop it later.
There are few places in the code where coccinelle didn't reach. I'll
fix them manually in a separate patch. Comments and documentation also
will be addressed with the separate patch.
virtual patch
@@
expression E;
@@
- E << (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E
@@
expression E;
@@
- E >> (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E
@@
@@
- PAGE_CACHE_SHIFT
+ PAGE_SHIFT
@@
@@
- PAGE_CACHE_SIZE
+ PAGE_SIZE
@@
@@
- PAGE_CACHE_MASK
+ PAGE_MASK
@@
expression E;
@@
- PAGE_CACHE_ALIGN(E)
+ PAGE_ALIGN(E)
@@
expression E;
@@
- page_cache_get(E)
+ get_page(E)
@@
expression E;
@@
- page_cache_release(E)
+ put_page(E)
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-04-01 12:29:47 +00:00
|
|
|
pos >> PAGE_SHIFT, end);
|
2013-11-14 16:50:34 +00:00
|
|
|
}
|
|
|
|
|
2016-06-03 21:07:19 +00:00
|
|
|
nfs_end_io_direct(inode);
|
2013-11-14 16:50:34 +00:00
|
|
|
|
2013-11-14 16:50:32 +00:00
|
|
|
if (!result) {
|
|
|
|
result = nfs_direct_wait(dreq);
|
|
|
|
if (result > 0) {
|
|
|
|
iocb->ki_pos = pos + result;
|
2016-04-07 15:52:01 +00:00
|
|
|
/* XXX: should check the generic_write_sync retval */
|
|
|
|
generic_write_sync(iocb, result);
|
2013-11-14 16:50:32 +00:00
|
|
|
}
|
2012-04-20 18:47:57 +00:00
|
|
|
}
|
2013-11-14 16:50:32 +00:00
|
|
|
out_release:
|
|
|
|
nfs_direct_req_release(dreq);
|
2016-06-03 21:07:19 +00:00
|
|
|
out:
|
2013-11-14 16:50:32 +00:00
|
|
|
return result;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2006-03-20 18:44:34 +00:00
|
|
|
/**
|
|
|
|
* nfs_init_directcache - create a slab cache for nfs_direct_req structures
|
|
|
|
*
|
|
|
|
*/
|
NFS: Split fs/nfs/inode.c
As fs/nfs/inode.c is rather large, heterogenous and unwieldy, the attached
patch splits it up into a number of files:
(*) fs/nfs/inode.c
Strictly inode specific functions.
(*) fs/nfs/super.c
Superblock management functions for NFS and NFS4, normal access, clones
and referrals. The NFS4 superblock functions _could_ move out into a
separate conditionally compiled file, but it's probably not worth it as
there're so many common bits.
(*) fs/nfs/namespace.c
Some namespace-specific functions have been moved here.
(*) fs/nfs/nfs4namespace.c
NFS4-specific namespace functions (this could be merged into the previous
file). This file is conditionally compiled.
(*) fs/nfs/internal.h
Inter-file declarations, plus a few simple utility functions moved from
fs/nfs/inode.c.
Additionally, all the in-.c-file externs have been moved here, and those
files they were moved from now includes this file.
For the most part, the functions have not been changed, only some multiplexor
functions have changed significantly.
I've also:
(*) Added some extra banner comments above some functions.
(*) Rearranged the function order within the files to be more logical and
better grouped (IMO), though someone may prefer a different order.
(*) Reduced the number of #ifdefs in .c files.
(*) Added missing __init and __exit directives.
Signed-Off-By: David Howells <dhowells@redhat.com>
2006-06-09 13:34:33 +00:00
|
|
|
int __init nfs_init_directcache(void)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
|
|
|
nfs_direct_cachep = kmem_cache_create("nfs_direct_cache",
|
|
|
|
sizeof(struct nfs_direct_req),
|
2006-03-24 11:16:06 +00:00
|
|
|
0, (SLAB_RECLAIM_ACCOUNT|
|
|
|
|
SLAB_MEM_SPREAD),
|
2007-07-20 01:11:58 +00:00
|
|
|
NULL);
|
2005-04-16 22:20:36 +00:00
|
|
|
if (nfs_direct_cachep == NULL)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2006-03-20 18:44:34 +00:00
|
|
|
/**
|
NFS: Split fs/nfs/inode.c
As fs/nfs/inode.c is rather large, heterogenous and unwieldy, the attached
patch splits it up into a number of files:
(*) fs/nfs/inode.c
Strictly inode specific functions.
(*) fs/nfs/super.c
Superblock management functions for NFS and NFS4, normal access, clones
and referrals. The NFS4 superblock functions _could_ move out into a
separate conditionally compiled file, but it's probably not worth it as
there're so many common bits.
(*) fs/nfs/namespace.c
Some namespace-specific functions have been moved here.
(*) fs/nfs/nfs4namespace.c
NFS4-specific namespace functions (this could be merged into the previous
file). This file is conditionally compiled.
(*) fs/nfs/internal.h
Inter-file declarations, plus a few simple utility functions moved from
fs/nfs/inode.c.
Additionally, all the in-.c-file externs have been moved here, and those
files they were moved from now includes this file.
For the most part, the functions have not been changed, only some multiplexor
functions have changed significantly.
I've also:
(*) Added some extra banner comments above some functions.
(*) Rearranged the function order within the files to be more logical and
better grouped (IMO), though someone may prefer a different order.
(*) Reduced the number of #ifdefs in .c files.
(*) Added missing __init and __exit directives.
Signed-Off-By: David Howells <dhowells@redhat.com>
2006-06-09 13:34:33 +00:00
|
|
|
* nfs_destroy_directcache - destroy the slab cache for nfs_direct_req structures
|
2006-03-20 18:44:34 +00:00
|
|
|
*
|
|
|
|
*/
|
2006-06-27 19:59:15 +00:00
|
|
|
void nfs_destroy_directcache(void)
|
2005-04-16 22:20:36 +00:00
|
|
|
{
|
2006-09-27 08:49:40 +00:00
|
|
|
kmem_cache_destroy(nfs_direct_cachep);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|