License cleanup: add SPDX GPL-2.0 license identifier to files with no license
Many source files in the tree are missing licensing information, which
makes it harder for compliance tools to determine the correct license.
By default all files without license information are under the default
license of the kernel, which is GPL version 2.
Update the files which contain no license information with the 'GPL-2.0'
SPDX license identifier. The SPDX identifier is a legally binding
shorthand, which can be used instead of the full boiler plate text.
This patch is based on work done by Thomas Gleixner and Kate Stewart and
Philippe Ombredanne.
How this work was done:
Patches were generated and checked against linux-4.14-rc6 for a subset of
the use cases:
- file had no licensing information it it.
- file was a */uapi/* one with no licensing information in it,
- file was a */uapi/* one with existing licensing information,
Further patches will be generated in subsequent months to fix up cases
where non-standard license headers were used, and references to license
had to be inferred by heuristics based on keywords.
The analysis to determine which SPDX License Identifier to be applied to
a file was done in a spreadsheet of side by side results from of the
output of two independent scanners (ScanCode & Windriver) producing SPDX
tag:value files created by Philippe Ombredanne. Philippe prepared the
base worksheet, and did an initial spot review of a few 1000 files.
The 4.13 kernel was the starting point of the analysis with 60,537 files
assessed. Kate Stewart did a file by file comparison of the scanner
results in the spreadsheet to determine which SPDX license identifier(s)
to be applied to the file. She confirmed any determination that was not
immediately clear with lawyers working with the Linux Foundation.
Criteria used to select files for SPDX license identifier tagging was:
- Files considered eligible had to be source code files.
- Make and config files were included as candidates if they contained >5
lines of source
- File already had some variant of a license header in it (even if <5
lines).
All documentation files were explicitly excluded.
The following heuristics were used to determine which SPDX license
identifiers to apply.
- when both scanners couldn't find any license traces, file was
considered to have no license information in it, and the top level
COPYING file license applied.
For non */uapi/* files that summary was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 11139
and resulted in the first patch in this series.
If that file was a */uapi/* path one, it was "GPL-2.0 WITH
Linux-syscall-note" otherwise it was "GPL-2.0". Results of that was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 WITH Linux-syscall-note 930
and resulted in the second patch in this series.
- if a file had some form of licensing information in it, and was one
of the */uapi/* ones, it was denoted with the Linux-syscall-note if
any GPL family license was found in the file or had no licensing in
it (per prior point). Results summary:
SPDX license identifier # files
---------------------------------------------------|------
GPL-2.0 WITH Linux-syscall-note 270
GPL-2.0+ WITH Linux-syscall-note 169
((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) 21
((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) 17
LGPL-2.1+ WITH Linux-syscall-note 15
GPL-1.0+ WITH Linux-syscall-note 14
((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) 5
LGPL-2.0+ WITH Linux-syscall-note 4
LGPL-2.1 WITH Linux-syscall-note 3
((GPL-2.0 WITH Linux-syscall-note) OR MIT) 3
((GPL-2.0 WITH Linux-syscall-note) AND MIT) 1
and that resulted in the third patch in this series.
- when the two scanners agreed on the detected license(s), that became
the concluded license(s).
- when there was disagreement between the two scanners (one detected a
license but the other didn't, or they both detected different
licenses) a manual inspection of the file occurred.
- In most cases a manual inspection of the information in the file
resulted in a clear resolution of the license that should apply (and
which scanner probably needed to revisit its heuristics).
- When it was not immediately clear, the license identifier was
confirmed with lawyers working with the Linux Foundation.
- If there was any question as to the appropriate license identifier,
the file was flagged for further research and to be revisited later
in time.
In total, over 70 hours of logged manual review was done on the
spreadsheet to determine the SPDX license identifiers to apply to the
source files by Kate, Philippe, Thomas and, in some cases, confirmation
by lawyers working with the Linux Foundation.
Kate also obtained a third independent scan of the 4.13 code base from
FOSSology, and compared selected files where the other two scanners
disagreed against that SPDX file, to see if there was new insights. The
Windriver scanner is based on an older version of FOSSology in part, so
they are related.
Thomas did random spot checks in about 500 files from the spreadsheets
for the uapi headers and agreed with SPDX license identifier in the
files he inspected. For the non-uapi files Thomas did random spot checks
in about 15000 files.
In initial set of patches against 4.14-rc6, 3 files were found to have
copy/paste license identifier errors, and have been fixed to reflect the
correct identifier.
Additionally Philippe spent 10 hours this week doing a detailed manual
inspection and review of the 12,461 patched files from the initial patch
version early this week with:
- a full scancode scan run, collecting the matched texts, detected
license ids and scores
- reviewing anything where there was a license detected (about 500+
files) to ensure that the applied SPDX license was correct
- reviewing anything where there was no detection but the patch license
was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
SPDX license was correct
This produced a worksheet with 20 files needing minor correction. This
worksheet was then exported into 3 different .csv files for the
different types of files to be modified.
These .csv files were then reviewed by Greg. Thomas wrote a script to
parse the csv files and add the proper SPDX tag to the file, in the
format that the file expected. This script was further refined by Greg
based on the output to detect more types of files automatically and to
distinguish between header and source .c files (which need different
comment types.) Finally Greg ran the script using the .csv files to
generate the patches.
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-11-01 14:07:57 +00:00
|
|
|
// SPDX-License-Identifier: GPL-2.0
|
2017-04-09 17:06:16 +00:00
|
|
|
/*
|
2018-05-07 19:27:21 +00:00
|
|
|
* Copyright (c) 2016-2018 Oracle. All rights reserved.
|
2017-04-09 17:06:16 +00:00
|
|
|
*
|
|
|
|
* Use the core R/W API to move RPC-over-RDMA Read and Write chunks.
|
|
|
|
*/
|
|
|
|
|
2018-05-07 19:27:11 +00:00
|
|
|
#include <rdma/rw.h>
|
|
|
|
|
2017-04-09 17:06:16 +00:00
|
|
|
#include <linux/sunrpc/rpc_rdma.h>
|
|
|
|
#include <linux/sunrpc/svc_rdma.h>
|
|
|
|
#include <linux/sunrpc/debug.h>
|
|
|
|
|
2018-05-07 19:27:11 +00:00
|
|
|
#include "xprt_rdma.h"
|
|
|
|
#include <trace/events/rpcrdma.h>
|
2017-04-09 17:06:16 +00:00
|
|
|
|
|
|
|
#define RPCDBG_FACILITY RPCDBG_SVCXPRT
|
|
|
|
|
2017-06-23 21:18:24 +00:00
|
|
|
static void svc_rdma_write_done(struct ib_cq *cq, struct ib_wc *wc);
|
|
|
|
static void svc_rdma_wc_read_done(struct ib_cq *cq, struct ib_wc *wc);
|
|
|
|
|
2017-04-09 17:06:16 +00:00
|
|
|
/* Each R/W context contains state for one chain of RDMA Read or
|
|
|
|
* Write Work Requests.
|
|
|
|
*
|
|
|
|
* Each WR chain handles a single contiguous server-side buffer,
|
|
|
|
* because scatterlist entries after the first have to start on
|
|
|
|
* page alignment. xdr_buf iovecs cannot guarantee alignment.
|
|
|
|
*
|
|
|
|
* Each WR chain handles only one R_key. Each RPC-over-RDMA segment
|
|
|
|
* from a client may contain a unique R_key, so each WR chain moves
|
|
|
|
* up to one segment at a time.
|
|
|
|
*
|
|
|
|
* The scatterlist makes this data structure over 4KB in size. To
|
|
|
|
* make it less likely to fail, and to handle the allocation for
|
|
|
|
* smaller I/O requests without disabling bottom-halves, these
|
|
|
|
* contexts are created on demand, but cached and reused until the
|
|
|
|
* controlling svcxprt_rdma is destroyed.
|
|
|
|
*/
|
|
|
|
struct svc_rdma_rw_ctxt {
|
|
|
|
struct list_head rw_list;
|
|
|
|
struct rdma_rw_ctx rw_ctx;
|
|
|
|
int rw_nents;
|
|
|
|
struct sg_table rw_sg_table;
|
|
|
|
struct scatterlist rw_first_sgl[0];
|
|
|
|
};
|
|
|
|
|
|
|
|
static inline struct svc_rdma_rw_ctxt *
|
|
|
|
svc_rdma_next_ctxt(struct list_head *list)
|
|
|
|
{
|
|
|
|
return list_first_entry_or_null(list, struct svc_rdma_rw_ctxt,
|
|
|
|
rw_list);
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct svc_rdma_rw_ctxt *
|
|
|
|
svc_rdma_get_rw_ctxt(struct svcxprt_rdma *rdma, unsigned int sges)
|
|
|
|
{
|
|
|
|
struct svc_rdma_rw_ctxt *ctxt;
|
|
|
|
|
|
|
|
spin_lock(&rdma->sc_rw_ctxt_lock);
|
|
|
|
|
|
|
|
ctxt = svc_rdma_next_ctxt(&rdma->sc_rw_ctxts);
|
|
|
|
if (ctxt) {
|
|
|
|
list_del(&ctxt->rw_list);
|
|
|
|
spin_unlock(&rdma->sc_rw_ctxt_lock);
|
|
|
|
} else {
|
|
|
|
spin_unlock(&rdma->sc_rw_ctxt_lock);
|
2019-01-15 21:11:40 +00:00
|
|
|
ctxt = kmalloc(struct_size(ctxt, rw_first_sgl, SG_CHUNK_SIZE),
|
2017-04-09 17:06:16 +00:00
|
|
|
GFP_KERNEL);
|
|
|
|
if (!ctxt)
|
|
|
|
goto out;
|
|
|
|
INIT_LIST_HEAD(&ctxt->rw_list);
|
|
|
|
}
|
|
|
|
|
|
|
|
ctxt->rw_sg_table.sgl = ctxt->rw_first_sgl;
|
|
|
|
if (sg_alloc_table_chained(&ctxt->rw_sg_table, sges,
|
2019-04-28 07:39:30 +00:00
|
|
|
ctxt->rw_sg_table.sgl,
|
|
|
|
SG_CHUNK_SIZE)) {
|
2017-04-09 17:06:16 +00:00
|
|
|
kfree(ctxt);
|
|
|
|
ctxt = NULL;
|
|
|
|
}
|
|
|
|
out:
|
|
|
|
return ctxt;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void svc_rdma_put_rw_ctxt(struct svcxprt_rdma *rdma,
|
|
|
|
struct svc_rdma_rw_ctxt *ctxt)
|
|
|
|
{
|
2019-04-28 07:39:30 +00:00
|
|
|
sg_free_table_chained(&ctxt->rw_sg_table, SG_CHUNK_SIZE);
|
2017-04-09 17:06:16 +00:00
|
|
|
|
|
|
|
spin_lock(&rdma->sc_rw_ctxt_lock);
|
|
|
|
list_add(&ctxt->rw_list, &rdma->sc_rw_ctxts);
|
|
|
|
spin_unlock(&rdma->sc_rw_ctxt_lock);
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* svc_rdma_destroy_rw_ctxts - Free accumulated R/W contexts
|
|
|
|
* @rdma: transport about to be destroyed
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
void svc_rdma_destroy_rw_ctxts(struct svcxprt_rdma *rdma)
|
|
|
|
{
|
|
|
|
struct svc_rdma_rw_ctxt *ctxt;
|
|
|
|
|
|
|
|
while ((ctxt = svc_rdma_next_ctxt(&rdma->sc_rw_ctxts)) != NULL) {
|
|
|
|
list_del(&ctxt->rw_list);
|
|
|
|
kfree(ctxt);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* A chunk context tracks all I/O for moving one Read or Write
|
|
|
|
* chunk. This is a a set of rdma_rw's that handle data movement
|
|
|
|
* for all segments of one chunk.
|
|
|
|
*
|
|
|
|
* These are small, acquired with a single allocator call, and
|
|
|
|
* no more than one is needed per chunk. They are allocated on
|
|
|
|
* demand, and not cached.
|
|
|
|
*/
|
|
|
|
struct svc_rdma_chunk_ctxt {
|
|
|
|
struct ib_cqe cc_cqe;
|
|
|
|
struct svcxprt_rdma *cc_rdma;
|
|
|
|
struct list_head cc_rwctxts;
|
|
|
|
int cc_sqecount;
|
|
|
|
};
|
|
|
|
|
|
|
|
static void svc_rdma_cc_init(struct svcxprt_rdma *rdma,
|
2017-06-23 21:19:29 +00:00
|
|
|
struct svc_rdma_chunk_ctxt *cc)
|
2017-04-09 17:06:16 +00:00
|
|
|
{
|
|
|
|
cc->cc_rdma = rdma;
|
|
|
|
svc_xprt_get(&rdma->sc_xprt);
|
|
|
|
|
|
|
|
INIT_LIST_HEAD(&cc->cc_rwctxts);
|
|
|
|
cc->cc_sqecount = 0;
|
|
|
|
}
|
|
|
|
|
2017-06-23 21:19:29 +00:00
|
|
|
static void svc_rdma_cc_release(struct svc_rdma_chunk_ctxt *cc,
|
|
|
|
enum dma_data_direction dir)
|
2017-04-09 17:06:16 +00:00
|
|
|
{
|
|
|
|
struct svcxprt_rdma *rdma = cc->cc_rdma;
|
|
|
|
struct svc_rdma_rw_ctxt *ctxt;
|
|
|
|
|
|
|
|
while ((ctxt = svc_rdma_next_ctxt(&cc->cc_rwctxts)) != NULL) {
|
|
|
|
list_del(&ctxt->rw_list);
|
|
|
|
|
|
|
|
rdma_rw_ctx_destroy(&ctxt->rw_ctx, rdma->sc_qp,
|
|
|
|
rdma->sc_port_num, ctxt->rw_sg_table.sgl,
|
2017-06-23 21:19:29 +00:00
|
|
|
ctxt->rw_nents, dir);
|
2017-04-09 17:06:16 +00:00
|
|
|
svc_rdma_put_rw_ctxt(rdma, ctxt);
|
|
|
|
}
|
|
|
|
svc_xprt_put(&rdma->sc_xprt);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* State for sending a Write or Reply chunk.
|
|
|
|
* - Tracks progress of writing one chunk over all its segments
|
|
|
|
* - Stores arguments for the SGL constructor functions
|
|
|
|
*/
|
|
|
|
struct svc_rdma_write_info {
|
|
|
|
/* write state of this chunk */
|
|
|
|
unsigned int wi_seg_off;
|
|
|
|
unsigned int wi_seg_no;
|
|
|
|
unsigned int wi_nsegs;
|
|
|
|
__be32 *wi_segs;
|
|
|
|
|
|
|
|
/* SGL constructor arguments */
|
|
|
|
struct xdr_buf *wi_xdr;
|
|
|
|
unsigned char *wi_base;
|
|
|
|
unsigned int wi_next_off;
|
|
|
|
|
|
|
|
struct svc_rdma_chunk_ctxt wi_cc;
|
|
|
|
};
|
|
|
|
|
|
|
|
static struct svc_rdma_write_info *
|
|
|
|
svc_rdma_write_info_alloc(struct svcxprt_rdma *rdma, __be32 *chunk)
|
|
|
|
{
|
|
|
|
struct svc_rdma_write_info *info;
|
|
|
|
|
|
|
|
info = kmalloc(sizeof(*info), GFP_KERNEL);
|
|
|
|
if (!info)
|
|
|
|
return info;
|
|
|
|
|
|
|
|
info->wi_seg_off = 0;
|
|
|
|
info->wi_seg_no = 0;
|
|
|
|
info->wi_nsegs = be32_to_cpup(++chunk);
|
|
|
|
info->wi_segs = ++chunk;
|
2017-06-23 21:19:29 +00:00
|
|
|
svc_rdma_cc_init(rdma, &info->wi_cc);
|
2017-06-23 21:18:24 +00:00
|
|
|
info->wi_cc.cc_cqe.done = svc_rdma_write_done;
|
2017-04-09 17:06:16 +00:00
|
|
|
return info;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void svc_rdma_write_info_free(struct svc_rdma_write_info *info)
|
|
|
|
{
|
2017-06-23 21:19:29 +00:00
|
|
|
svc_rdma_cc_release(&info->wi_cc, DMA_TO_DEVICE);
|
2017-04-09 17:06:16 +00:00
|
|
|
kfree(info);
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* svc_rdma_write_done - Write chunk completion
|
|
|
|
* @cq: controlling Completion Queue
|
|
|
|
* @wc: Work Completion
|
|
|
|
*
|
|
|
|
* Pages under I/O are freed by a subsequent Send completion.
|
|
|
|
*/
|
|
|
|
static void svc_rdma_write_done(struct ib_cq *cq, struct ib_wc *wc)
|
|
|
|
{
|
|
|
|
struct ib_cqe *cqe = wc->wr_cqe;
|
|
|
|
struct svc_rdma_chunk_ctxt *cc =
|
|
|
|
container_of(cqe, struct svc_rdma_chunk_ctxt, cc_cqe);
|
|
|
|
struct svcxprt_rdma *rdma = cc->cc_rdma;
|
|
|
|
struct svc_rdma_write_info *info =
|
|
|
|
container_of(cc, struct svc_rdma_write_info, wi_cc);
|
|
|
|
|
2018-05-07 19:27:16 +00:00
|
|
|
trace_svcrdma_wc_write(wc);
|
|
|
|
|
2017-04-09 17:06:16 +00:00
|
|
|
atomic_add(cc->cc_sqecount, &rdma->sc_sq_avail);
|
|
|
|
wake_up(&rdma->sc_send_wait);
|
|
|
|
|
2019-02-06 17:00:57 +00:00
|
|
|
if (unlikely(wc->status != IB_WC_SUCCESS))
|
2017-04-09 17:06:16 +00:00
|
|
|
set_bit(XPT_CLOSE, &rdma->sc_xprt.xpt_flags);
|
|
|
|
|
|
|
|
svc_rdma_write_info_free(info);
|
|
|
|
}
|
|
|
|
|
2017-06-23 21:18:24 +00:00
|
|
|
/* State for pulling a Read chunk.
|
|
|
|
*/
|
|
|
|
struct svc_rdma_read_info {
|
2018-05-07 19:27:21 +00:00
|
|
|
struct svc_rdma_recv_ctxt *ri_readctxt;
|
2017-06-23 21:18:24 +00:00
|
|
|
unsigned int ri_position;
|
|
|
|
unsigned int ri_pageno;
|
|
|
|
unsigned int ri_pageoff;
|
|
|
|
unsigned int ri_chunklen;
|
|
|
|
|
|
|
|
struct svc_rdma_chunk_ctxt ri_cc;
|
|
|
|
};
|
|
|
|
|
|
|
|
static struct svc_rdma_read_info *
|
|
|
|
svc_rdma_read_info_alloc(struct svcxprt_rdma *rdma)
|
|
|
|
{
|
|
|
|
struct svc_rdma_read_info *info;
|
|
|
|
|
|
|
|
info = kmalloc(sizeof(*info), GFP_KERNEL);
|
|
|
|
if (!info)
|
|
|
|
return info;
|
|
|
|
|
2017-06-23 21:19:29 +00:00
|
|
|
svc_rdma_cc_init(rdma, &info->ri_cc);
|
2017-06-23 21:18:24 +00:00
|
|
|
info->ri_cc.cc_cqe.done = svc_rdma_wc_read_done;
|
|
|
|
return info;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void svc_rdma_read_info_free(struct svc_rdma_read_info *info)
|
|
|
|
{
|
2017-06-23 21:19:29 +00:00
|
|
|
svc_rdma_cc_release(&info->ri_cc, DMA_FROM_DEVICE);
|
2017-06-23 21:18:24 +00:00
|
|
|
kfree(info);
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* svc_rdma_wc_read_done - Handle completion of an RDMA Read ctx
|
|
|
|
* @cq: controlling Completion Queue
|
|
|
|
* @wc: Work Completion
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
static void svc_rdma_wc_read_done(struct ib_cq *cq, struct ib_wc *wc)
|
|
|
|
{
|
|
|
|
struct ib_cqe *cqe = wc->wr_cqe;
|
|
|
|
struct svc_rdma_chunk_ctxt *cc =
|
|
|
|
container_of(cqe, struct svc_rdma_chunk_ctxt, cc_cqe);
|
|
|
|
struct svcxprt_rdma *rdma = cc->cc_rdma;
|
|
|
|
struct svc_rdma_read_info *info =
|
|
|
|
container_of(cc, struct svc_rdma_read_info, ri_cc);
|
|
|
|
|
2018-05-07 19:27:16 +00:00
|
|
|
trace_svcrdma_wc_read(wc);
|
|
|
|
|
2017-06-23 21:18:24 +00:00
|
|
|
atomic_add(cc->cc_sqecount, &rdma->sc_sq_avail);
|
|
|
|
wake_up(&rdma->sc_send_wait);
|
|
|
|
|
|
|
|
if (unlikely(wc->status != IB_WC_SUCCESS)) {
|
|
|
|
set_bit(XPT_CLOSE, &rdma->sc_xprt.xpt_flags);
|
2018-05-07 19:27:32 +00:00
|
|
|
svc_rdma_recv_ctxt_put(rdma, info->ri_readctxt);
|
2017-06-23 21:18:24 +00:00
|
|
|
} else {
|
|
|
|
spin_lock(&rdma->sc_rq_dto_lock);
|
2018-05-07 19:27:21 +00:00
|
|
|
list_add_tail(&info->ri_readctxt->rc_list,
|
2017-06-23 21:18:24 +00:00
|
|
|
&rdma->sc_read_complete_q);
|
svcrpc: fix unlikely races preventing queueing of sockets
In the rpc server, When something happens that might be reason to wake
up a thread to do something, what we do is
- modify xpt_flags, sk_sock->flags, xpt_reserved, or
xpt_nr_rqsts to indicate the new situation
- call svc_xprt_enqueue() to decide whether to wake up a thread.
svc_xprt_enqueue may require multiple conditions to be true before
queueing up a thread to handle the xprt. In the SMP case, one of the
other CPU's may have set another required condition, and in that case,
although both CPUs run svc_xprt_enqueue(), it's possible that neither
call sees the writes done by the other CPU in time, and neither one
recognizes that all the required conditions have been set. A socket
could therefore be ignored indefinitely.
Add memory barries to ensure that any svc_xprt_enqueue() call will
always see the conditions changed by other CPUs before deciding to
ignore a socket.
I've never seen this race reported. In the unlikely event it happens,
another event will usually come along and the problem will fix itself.
So I don't think this is worth backporting to stable.
Chuck tried this patch and said "I don't see any performance
regressions, but my server has only a single last-level CPU cache."
Tested-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: J. Bruce Fields <bfields@redhat.com>
2019-01-11 20:36:40 +00:00
|
|
|
/* Note the unlock pairs with the smp_rmb in svc_xprt_ready: */
|
|
|
|
set_bit(XPT_DATA, &rdma->sc_xprt.xpt_flags);
|
2017-06-23 21:18:24 +00:00
|
|
|
spin_unlock(&rdma->sc_rq_dto_lock);
|
|
|
|
|
|
|
|
svc_xprt_enqueue(&rdma->sc_xprt);
|
|
|
|
}
|
|
|
|
|
|
|
|
svc_rdma_read_info_free(info);
|
|
|
|
}
|
|
|
|
|
2017-04-09 17:06:16 +00:00
|
|
|
/* This function sleeps when the transport's Send Queue is congested.
|
|
|
|
*
|
|
|
|
* Assumptions:
|
|
|
|
* - If ib_post_send() succeeds, only one completion is expected,
|
|
|
|
* even if one or more WRs are flushed. This is true when posting
|
|
|
|
* an rdma_rw_ctx or when posting a single signaled WR.
|
|
|
|
*/
|
|
|
|
static int svc_rdma_post_chunk_ctxt(struct svc_rdma_chunk_ctxt *cc)
|
|
|
|
{
|
|
|
|
struct svcxprt_rdma *rdma = cc->cc_rdma;
|
|
|
|
struct svc_xprt *xprt = &rdma->sc_xprt;
|
2018-07-18 16:25:32 +00:00
|
|
|
struct ib_send_wr *first_wr;
|
|
|
|
const struct ib_send_wr *bad_wr;
|
2017-04-09 17:06:16 +00:00
|
|
|
struct list_head *tmp;
|
|
|
|
struct ib_cqe *cqe;
|
|
|
|
int ret;
|
|
|
|
|
2017-06-23 21:17:24 +00:00
|
|
|
if (cc->cc_sqecount > rdma->sc_sq_depth)
|
|
|
|
return -EINVAL;
|
|
|
|
|
2017-04-09 17:06:16 +00:00
|
|
|
first_wr = NULL;
|
|
|
|
cqe = &cc->cc_cqe;
|
|
|
|
list_for_each(tmp, &cc->cc_rwctxts) {
|
|
|
|
struct svc_rdma_rw_ctxt *ctxt;
|
|
|
|
|
|
|
|
ctxt = list_entry(tmp, struct svc_rdma_rw_ctxt, rw_list);
|
|
|
|
first_wr = rdma_rw_ctx_wrs(&ctxt->rw_ctx, rdma->sc_qp,
|
|
|
|
rdma->sc_port_num, cqe, first_wr);
|
|
|
|
cqe = NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
do {
|
|
|
|
if (atomic_sub_return(cc->cc_sqecount,
|
|
|
|
&rdma->sc_sq_avail) > 0) {
|
2018-07-26 17:36:50 +00:00
|
|
|
ret = ib_post_send(rdma->sc_qp, first_wr, &bad_wr);
|
2018-05-07 19:27:16 +00:00
|
|
|
trace_svcrdma_post_rw(&cc->cc_cqe,
|
|
|
|
cc->cc_sqecount, ret);
|
2017-04-09 17:06:16 +00:00
|
|
|
if (ret)
|
|
|
|
break;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2018-05-07 19:27:16 +00:00
|
|
|
trace_svcrdma_sq_full(rdma);
|
2017-04-09 17:06:16 +00:00
|
|
|
atomic_add(cc->cc_sqecount, &rdma->sc_sq_avail);
|
|
|
|
wait_event(rdma->sc_send_wait,
|
|
|
|
atomic_read(&rdma->sc_sq_avail) > cc->cc_sqecount);
|
2018-05-07 19:27:16 +00:00
|
|
|
trace_svcrdma_sq_retry(rdma);
|
2017-04-09 17:06:16 +00:00
|
|
|
} while (1);
|
|
|
|
|
|
|
|
set_bit(XPT_CLOSE, &xprt->xpt_flags);
|
|
|
|
|
|
|
|
/* If even one was posted, there will be a completion. */
|
|
|
|
if (bad_wr != first_wr)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
atomic_add(cc->cc_sqecount, &rdma->sc_sq_avail);
|
|
|
|
wake_up(&rdma->sc_send_wait);
|
|
|
|
return -ENOTCONN;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Build and DMA-map an SGL that covers one kvec in an xdr_buf
|
|
|
|
*/
|
|
|
|
static void svc_rdma_vec_to_sg(struct svc_rdma_write_info *info,
|
|
|
|
unsigned int len,
|
|
|
|
struct svc_rdma_rw_ctxt *ctxt)
|
|
|
|
{
|
|
|
|
struct scatterlist *sg = ctxt->rw_sg_table.sgl;
|
|
|
|
|
|
|
|
sg_set_buf(&sg[0], info->wi_base, len);
|
|
|
|
info->wi_base += len;
|
|
|
|
|
|
|
|
ctxt->rw_nents = 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Build and DMA-map an SGL that covers part of an xdr_buf's pagelist.
|
|
|
|
*/
|
|
|
|
static void svc_rdma_pagelist_to_sg(struct svc_rdma_write_info *info,
|
|
|
|
unsigned int remaining,
|
|
|
|
struct svc_rdma_rw_ctxt *ctxt)
|
|
|
|
{
|
|
|
|
unsigned int sge_no, sge_bytes, page_off, page_no;
|
|
|
|
struct xdr_buf *xdr = info->wi_xdr;
|
|
|
|
struct scatterlist *sg;
|
|
|
|
struct page **page;
|
|
|
|
|
2017-06-23 21:19:21 +00:00
|
|
|
page_off = info->wi_next_off + xdr->page_base;
|
|
|
|
page_no = page_off >> PAGE_SHIFT;
|
|
|
|
page_off = offset_in_page(page_off);
|
2017-04-09 17:06:16 +00:00
|
|
|
page = xdr->pages + page_no;
|
|
|
|
info->wi_next_off += remaining;
|
|
|
|
sg = ctxt->rw_sg_table.sgl;
|
|
|
|
sge_no = 0;
|
|
|
|
do {
|
|
|
|
sge_bytes = min_t(unsigned int, remaining,
|
|
|
|
PAGE_SIZE - page_off);
|
|
|
|
sg_set_page(sg, *page, sge_bytes, page_off);
|
|
|
|
|
|
|
|
remaining -= sge_bytes;
|
|
|
|
sg = sg_next(sg);
|
|
|
|
page_off = 0;
|
|
|
|
sge_no++;
|
|
|
|
page++;
|
|
|
|
} while (remaining);
|
|
|
|
|
|
|
|
ctxt->rw_nents = sge_no;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Construct RDMA Write WRs to send a portion of an xdr_buf containing
|
|
|
|
* an RPC Reply.
|
|
|
|
*/
|
|
|
|
static int
|
|
|
|
svc_rdma_build_writes(struct svc_rdma_write_info *info,
|
|
|
|
void (*constructor)(struct svc_rdma_write_info *info,
|
|
|
|
unsigned int len,
|
|
|
|
struct svc_rdma_rw_ctxt *ctxt),
|
|
|
|
unsigned int remaining)
|
|
|
|
{
|
|
|
|
struct svc_rdma_chunk_ctxt *cc = &info->wi_cc;
|
|
|
|
struct svcxprt_rdma *rdma = cc->cc_rdma;
|
|
|
|
struct svc_rdma_rw_ctxt *ctxt;
|
|
|
|
__be32 *seg;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
seg = info->wi_segs + info->wi_seg_no * rpcrdma_segment_maxsz;
|
|
|
|
do {
|
|
|
|
unsigned int write_len;
|
|
|
|
u32 seg_length, seg_handle;
|
|
|
|
u64 seg_offset;
|
|
|
|
|
|
|
|
if (info->wi_seg_no >= info->wi_nsegs)
|
|
|
|
goto out_overflow;
|
|
|
|
|
|
|
|
seg_handle = be32_to_cpup(seg);
|
|
|
|
seg_length = be32_to_cpup(seg + 1);
|
|
|
|
xdr_decode_hyper(seg + 2, &seg_offset);
|
|
|
|
seg_offset += info->wi_seg_off;
|
|
|
|
|
|
|
|
write_len = min(remaining, seg_length - info->wi_seg_off);
|
|
|
|
ctxt = svc_rdma_get_rw_ctxt(rdma,
|
|
|
|
(write_len >> PAGE_SHIFT) + 2);
|
|
|
|
if (!ctxt)
|
|
|
|
goto out_noctx;
|
|
|
|
|
|
|
|
constructor(info, write_len, ctxt);
|
|
|
|
ret = rdma_rw_ctx_init(&ctxt->rw_ctx, rdma->sc_qp,
|
|
|
|
rdma->sc_port_num, ctxt->rw_sg_table.sgl,
|
|
|
|
ctxt->rw_nents, 0, seg_offset,
|
|
|
|
seg_handle, DMA_TO_DEVICE);
|
|
|
|
if (ret < 0)
|
|
|
|
goto out_initerr;
|
|
|
|
|
2018-05-07 19:27:11 +00:00
|
|
|
trace_svcrdma_encode_wseg(seg_handle, write_len, seg_offset);
|
2017-04-09 17:06:16 +00:00
|
|
|
list_add(&ctxt->rw_list, &cc->cc_rwctxts);
|
|
|
|
cc->cc_sqecount += ret;
|
|
|
|
if (write_len == seg_length - info->wi_seg_off) {
|
|
|
|
seg += 4;
|
|
|
|
info->wi_seg_no++;
|
|
|
|
info->wi_seg_off = 0;
|
|
|
|
} else {
|
|
|
|
info->wi_seg_off += write_len;
|
|
|
|
}
|
|
|
|
remaining -= write_len;
|
|
|
|
} while (remaining);
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
out_overflow:
|
|
|
|
dprintk("svcrdma: inadequate space in Write chunk (%u)\n",
|
|
|
|
info->wi_nsegs);
|
|
|
|
return -E2BIG;
|
|
|
|
|
|
|
|
out_noctx:
|
|
|
|
dprintk("svcrdma: no R/W ctxs available\n");
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
out_initerr:
|
|
|
|
svc_rdma_put_rw_ctxt(rdma, ctxt);
|
2018-05-07 19:27:16 +00:00
|
|
|
trace_svcrdma_dma_map_rwctx(rdma, ret);
|
2017-04-09 17:06:16 +00:00
|
|
|
return -EIO;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Send one of an xdr_buf's kvecs by itself. To send a Reply
|
|
|
|
* chunk, the whole RPC Reply is written back to the client.
|
|
|
|
* This function writes either the head or tail of the xdr_buf
|
|
|
|
* containing the Reply.
|
|
|
|
*/
|
|
|
|
static int svc_rdma_send_xdr_kvec(struct svc_rdma_write_info *info,
|
|
|
|
struct kvec *vec)
|
|
|
|
{
|
|
|
|
info->wi_base = vec->iov_base;
|
|
|
|
return svc_rdma_build_writes(info, svc_rdma_vec_to_sg,
|
|
|
|
vec->iov_len);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Send an xdr_buf's page list by itself. A Write chunk is
|
|
|
|
* just the page list. a Reply chunk is the head, page list,
|
|
|
|
* and tail. This function is shared between the two types
|
|
|
|
* of chunk.
|
|
|
|
*/
|
|
|
|
static int svc_rdma_send_xdr_pagelist(struct svc_rdma_write_info *info,
|
|
|
|
struct xdr_buf *xdr)
|
|
|
|
{
|
|
|
|
info->wi_xdr = xdr;
|
|
|
|
info->wi_next_off = 0;
|
|
|
|
return svc_rdma_build_writes(info, svc_rdma_pagelist_to_sg,
|
|
|
|
xdr->page_len);
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* svc_rdma_send_write_chunk - Write all segments in a Write chunk
|
|
|
|
* @rdma: controlling RDMA transport
|
|
|
|
* @wr_ch: Write chunk provided by client
|
|
|
|
* @xdr: xdr_buf containing the data payload
|
|
|
|
*
|
|
|
|
* Returns a non-negative number of bytes the chunk consumed, or
|
|
|
|
* %-E2BIG if the payload was larger than the Write chunk,
|
2017-06-23 21:17:24 +00:00
|
|
|
* %-EINVAL if client provided too many segments,
|
2017-04-09 17:06:16 +00:00
|
|
|
* %-ENOMEM if rdma_rw context pool was exhausted,
|
|
|
|
* %-ENOTCONN if posting failed (connection is lost),
|
|
|
|
* %-EIO if rdma_rw initialization failed (DMA mapping, etc).
|
|
|
|
*/
|
|
|
|
int svc_rdma_send_write_chunk(struct svcxprt_rdma *rdma, __be32 *wr_ch,
|
|
|
|
struct xdr_buf *xdr)
|
|
|
|
{
|
|
|
|
struct svc_rdma_write_info *info;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
if (!xdr->page_len)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
info = svc_rdma_write_info_alloc(rdma, wr_ch);
|
|
|
|
if (!info)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
ret = svc_rdma_send_xdr_pagelist(info, xdr);
|
|
|
|
if (ret < 0)
|
|
|
|
goto out_err;
|
|
|
|
|
|
|
|
ret = svc_rdma_post_chunk_ctxt(&info->wi_cc);
|
|
|
|
if (ret < 0)
|
|
|
|
goto out_err;
|
2018-05-07 19:27:11 +00:00
|
|
|
|
|
|
|
trace_svcrdma_encode_write(xdr->page_len);
|
2017-04-09 17:06:16 +00:00
|
|
|
return xdr->page_len;
|
|
|
|
|
|
|
|
out_err:
|
|
|
|
svc_rdma_write_info_free(info);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* svc_rdma_send_reply_chunk - Write all segments in the Reply chunk
|
|
|
|
* @rdma: controlling RDMA transport
|
|
|
|
* @rp_ch: Reply chunk provided by client
|
|
|
|
* @writelist: true if client provided a Write list
|
|
|
|
* @xdr: xdr_buf containing an RPC Reply
|
|
|
|
*
|
|
|
|
* Returns a non-negative number of bytes the chunk consumed, or
|
|
|
|
* %-E2BIG if the payload was larger than the Reply chunk,
|
2017-06-23 21:17:24 +00:00
|
|
|
* %-EINVAL if client provided too many segments,
|
2017-04-09 17:06:16 +00:00
|
|
|
* %-ENOMEM if rdma_rw context pool was exhausted,
|
|
|
|
* %-ENOTCONN if posting failed (connection is lost),
|
|
|
|
* %-EIO if rdma_rw initialization failed (DMA mapping, etc).
|
|
|
|
*/
|
|
|
|
int svc_rdma_send_reply_chunk(struct svcxprt_rdma *rdma, __be32 *rp_ch,
|
|
|
|
bool writelist, struct xdr_buf *xdr)
|
|
|
|
{
|
|
|
|
struct svc_rdma_write_info *info;
|
|
|
|
int consumed, ret;
|
|
|
|
|
|
|
|
info = svc_rdma_write_info_alloc(rdma, rp_ch);
|
|
|
|
if (!info)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
ret = svc_rdma_send_xdr_kvec(info, &xdr->head[0]);
|
|
|
|
if (ret < 0)
|
|
|
|
goto out_err;
|
|
|
|
consumed = xdr->head[0].iov_len;
|
|
|
|
|
|
|
|
/* Send the page list in the Reply chunk only if the
|
|
|
|
* client did not provide Write chunks.
|
|
|
|
*/
|
|
|
|
if (!writelist && xdr->page_len) {
|
|
|
|
ret = svc_rdma_send_xdr_pagelist(info, xdr);
|
|
|
|
if (ret < 0)
|
|
|
|
goto out_err;
|
|
|
|
consumed += xdr->page_len;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (xdr->tail[0].iov_len) {
|
|
|
|
ret = svc_rdma_send_xdr_kvec(info, &xdr->tail[0]);
|
|
|
|
if (ret < 0)
|
|
|
|
goto out_err;
|
|
|
|
consumed += xdr->tail[0].iov_len;
|
|
|
|
}
|
|
|
|
|
|
|
|
ret = svc_rdma_post_chunk_ctxt(&info->wi_cc);
|
|
|
|
if (ret < 0)
|
|
|
|
goto out_err;
|
2018-05-07 19:27:11 +00:00
|
|
|
|
|
|
|
trace_svcrdma_encode_reply(consumed);
|
2017-04-09 17:06:16 +00:00
|
|
|
return consumed;
|
|
|
|
|
|
|
|
out_err:
|
|
|
|
svc_rdma_write_info_free(info);
|
|
|
|
return ret;
|
|
|
|
}
|
2017-06-23 21:18:24 +00:00
|
|
|
|
|
|
|
static int svc_rdma_build_read_segment(struct svc_rdma_read_info *info,
|
|
|
|
struct svc_rqst *rqstp,
|
|
|
|
u32 rkey, u32 len, u64 offset)
|
|
|
|
{
|
2018-05-07 19:27:21 +00:00
|
|
|
struct svc_rdma_recv_ctxt *head = info->ri_readctxt;
|
2017-06-23 21:18:24 +00:00
|
|
|
struct svc_rdma_chunk_ctxt *cc = &info->ri_cc;
|
|
|
|
struct svc_rdma_rw_ctxt *ctxt;
|
|
|
|
unsigned int sge_no, seg_len;
|
|
|
|
struct scatterlist *sg;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
sge_no = PAGE_ALIGN(info->ri_pageoff + len) >> PAGE_SHIFT;
|
|
|
|
ctxt = svc_rdma_get_rw_ctxt(cc->cc_rdma, sge_no);
|
|
|
|
if (!ctxt)
|
|
|
|
goto out_noctx;
|
|
|
|
ctxt->rw_nents = sge_no;
|
|
|
|
|
|
|
|
sg = ctxt->rw_sg_table.sgl;
|
|
|
|
for (sge_no = 0; sge_no < ctxt->rw_nents; sge_no++) {
|
|
|
|
seg_len = min_t(unsigned int, len,
|
|
|
|
PAGE_SIZE - info->ri_pageoff);
|
|
|
|
|
2018-05-07 19:27:21 +00:00
|
|
|
head->rc_arg.pages[info->ri_pageno] =
|
2017-06-23 21:18:24 +00:00
|
|
|
rqstp->rq_pages[info->ri_pageno];
|
|
|
|
if (!info->ri_pageoff)
|
2018-05-07 19:27:21 +00:00
|
|
|
head->rc_page_count++;
|
2017-06-23 21:18:24 +00:00
|
|
|
|
|
|
|
sg_set_page(sg, rqstp->rq_pages[info->ri_pageno],
|
|
|
|
seg_len, info->ri_pageoff);
|
|
|
|
sg = sg_next(sg);
|
|
|
|
|
|
|
|
info->ri_pageoff += seg_len;
|
|
|
|
if (info->ri_pageoff == PAGE_SIZE) {
|
|
|
|
info->ri_pageno++;
|
|
|
|
info->ri_pageoff = 0;
|
|
|
|
}
|
|
|
|
len -= seg_len;
|
|
|
|
|
|
|
|
/* Safety check */
|
|
|
|
if (len &&
|
|
|
|
&rqstp->rq_pages[info->ri_pageno + 1] > rqstp->rq_page_end)
|
|
|
|
goto out_overrun;
|
|
|
|
}
|
|
|
|
|
|
|
|
ret = rdma_rw_ctx_init(&ctxt->rw_ctx, cc->cc_rdma->sc_qp,
|
|
|
|
cc->cc_rdma->sc_port_num,
|
|
|
|
ctxt->rw_sg_table.sgl, ctxt->rw_nents,
|
|
|
|
0, offset, rkey, DMA_FROM_DEVICE);
|
|
|
|
if (ret < 0)
|
|
|
|
goto out_initerr;
|
|
|
|
|
|
|
|
list_add(&ctxt->rw_list, &cc->cc_rwctxts);
|
|
|
|
cc->cc_sqecount += ret;
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
out_noctx:
|
|
|
|
dprintk("svcrdma: no R/W ctxs available\n");
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
out_overrun:
|
|
|
|
dprintk("svcrdma: request overruns rq_pages\n");
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
out_initerr:
|
2018-05-07 19:27:16 +00:00
|
|
|
trace_svcrdma_dma_map_rwctx(cc->cc_rdma, ret);
|
2017-06-23 21:18:24 +00:00
|
|
|
svc_rdma_put_rw_ctxt(cc->cc_rdma, ctxt);
|
|
|
|
return -EIO;
|
|
|
|
}
|
|
|
|
|
2017-08-01 16:00:14 +00:00
|
|
|
/* Walk the segments in the Read chunk starting at @p and construct
|
|
|
|
* RDMA Read operations to pull the chunk to the server.
|
|
|
|
*/
|
2017-06-23 21:18:24 +00:00
|
|
|
static int svc_rdma_build_read_chunk(struct svc_rqst *rqstp,
|
|
|
|
struct svc_rdma_read_info *info,
|
|
|
|
__be32 *p)
|
|
|
|
{
|
2018-07-27 15:18:59 +00:00
|
|
|
unsigned int i;
|
2017-06-23 21:18:24 +00:00
|
|
|
int ret;
|
|
|
|
|
2017-08-01 16:00:14 +00:00
|
|
|
ret = -EINVAL;
|
2017-06-23 21:18:24 +00:00
|
|
|
info->ri_chunklen = 0;
|
2017-08-01 16:00:14 +00:00
|
|
|
while (*p++ != xdr_zero && be32_to_cpup(p++) == info->ri_position) {
|
2017-06-23 21:18:24 +00:00
|
|
|
u32 rs_handle, rs_length;
|
|
|
|
u64 rs_offset;
|
|
|
|
|
|
|
|
rs_handle = be32_to_cpup(p++);
|
|
|
|
rs_length = be32_to_cpup(p++);
|
|
|
|
p = xdr_decode_hyper(p, &rs_offset);
|
|
|
|
|
|
|
|
ret = svc_rdma_build_read_segment(info, rqstp,
|
|
|
|
rs_handle, rs_length,
|
|
|
|
rs_offset);
|
|
|
|
if (ret < 0)
|
|
|
|
break;
|
|
|
|
|
2018-05-07 19:27:11 +00:00
|
|
|
trace_svcrdma_encode_rseg(rs_handle, rs_length, rs_offset);
|
2017-06-23 21:18:24 +00:00
|
|
|
info->ri_chunklen += rs_length;
|
|
|
|
}
|
|
|
|
|
2018-07-27 15:18:59 +00:00
|
|
|
/* Pages under I/O have been copied to head->rc_pages.
|
|
|
|
* Prevent their premature release by svc_xprt_release() .
|
|
|
|
*/
|
|
|
|
for (i = 0; i < info->ri_readctxt->rc_page_count; i++)
|
|
|
|
rqstp->rq_pages[i] = NULL;
|
|
|
|
|
2017-06-23 21:18:24 +00:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Construct RDMA Reads to pull over a normal Read chunk. The chunk
|
2018-05-07 19:27:21 +00:00
|
|
|
* data lands in the page list of head->rc_arg.pages.
|
2017-06-23 21:18:24 +00:00
|
|
|
*
|
2018-05-07 19:27:21 +00:00
|
|
|
* Currently NFSD does not look at the head->rc_arg.tail[0] iovec.
|
2017-06-23 21:18:24 +00:00
|
|
|
* Therefore, XDR round-up of the Read chunk and trailing
|
|
|
|
* inline content must both be added at the end of the pagelist.
|
|
|
|
*/
|
|
|
|
static int svc_rdma_build_normal_read_chunk(struct svc_rqst *rqstp,
|
|
|
|
struct svc_rdma_read_info *info,
|
|
|
|
__be32 *p)
|
|
|
|
{
|
2018-05-07 19:27:21 +00:00
|
|
|
struct svc_rdma_recv_ctxt *head = info->ri_readctxt;
|
2017-06-23 21:18:24 +00:00
|
|
|
int ret;
|
|
|
|
|
|
|
|
ret = svc_rdma_build_read_chunk(rqstp, info, p);
|
|
|
|
if (ret < 0)
|
|
|
|
goto out;
|
|
|
|
|
2018-05-07 19:27:11 +00:00
|
|
|
trace_svcrdma_encode_read(info->ri_chunklen, info->ri_position);
|
|
|
|
|
2018-05-07 19:27:43 +00:00
|
|
|
head->rc_hdr_count = 0;
|
|
|
|
|
2017-08-18 15:12:35 +00:00
|
|
|
/* Split the Receive buffer between the head and tail
|
|
|
|
* buffers at Read chunk's position. XDR roundup of the
|
|
|
|
* chunk is not included in either the pagelist or in
|
|
|
|
* the tail.
|
2017-06-23 21:18:24 +00:00
|
|
|
*/
|
2018-05-07 19:27:21 +00:00
|
|
|
head->rc_arg.tail[0].iov_base =
|
|
|
|
head->rc_arg.head[0].iov_base + info->ri_position;
|
|
|
|
head->rc_arg.tail[0].iov_len =
|
|
|
|
head->rc_arg.head[0].iov_len - info->ri_position;
|
|
|
|
head->rc_arg.head[0].iov_len = info->ri_position;
|
2017-06-23 21:18:24 +00:00
|
|
|
|
svcrdma: Fix Read chunk round-up
A single NFSv4 WRITE compound can often have three operations:
PUTFH, WRITE, then GETATTR.
When the WRITE payload is sent in a Read chunk, the client places
the GETATTR in the inline part of the RPC/RDMA message, just after
the WRITE operation (sans payload). The position value in the Read
chunk enables the receiver to insert the Read chunk at the correct
place in the received XDR stream; that is between the WRITE and
GETATTR.
According to RFC 8166, an NFS/RDMA client does not have to add XDR
round-up to the Read chunk that carries the WRITE payload. The
receiver adds XDR round-up padding if it is absent and the
receiver's XDR decoder requires it to be present.
Commit 193bcb7b3719 ("svcrdma: Populate tail iovec when receiving")
attempted to add support for receiving such a compound so that just
the WRITE payload appears in rq_arg's page list, and the trailing
GETATTR is placed in rq_arg's tail iovec. (TCP just strings the
whole compound into the head iovec and page list, without regard
to the alignment of the WRITE payload).
The server transport logic also had to accommodate the optional XDR
round-up of the Read chunk, which it did simply by lengthening the
tail iovec when round-up was needed. This approach is adequate for
the NFSv2 and NFSv3 WRITE decoders.
Unfortunately it is not sufficient for nfsd4_decode_write. When the
Read chunk length is a couple of bytes less than PAGE_SIZE, the
computation at the end of nfsd4_decode_write allows argp->pagelen to
go negative, which breaks the logic in read_buf that looks for the
tail iovec.
The result is that a WRITE operation whose payload length is just
less than a multiple of a page succeeds, but the subsequent GETATTR
in the same compound fails with NFS4ERR_OP_ILLEGAL because the XDR
decoder can't find it. Clients ignore the error, but they must
update their attribute cache via a separate round trip.
As nfsd4_decode_write appears to expect the payload itself to always
have appropriate XDR round-up, have svc_rdma_build_normal_read_chunk
add the Read chunk XDR round-up to the page_len rather than
lengthening the tail iovec.
Reported-by: Olga Kornievskaia <kolga@netapp.com>
Fixes: 193bcb7b3719 ("svcrdma: Populate tail iovec when receiving")
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Tested-by: Olga Kornievskaia <kolga@netapp.com>
Signed-off-by: J. Bruce Fields <bfields@redhat.com>
2018-02-02 19:28:59 +00:00
|
|
|
/* Read chunk may need XDR roundup (see RFC 8166, s. 3.4.5.2).
|
2017-08-18 15:12:35 +00:00
|
|
|
*
|
svcrdma: Fix Read chunk round-up
A single NFSv4 WRITE compound can often have three operations:
PUTFH, WRITE, then GETATTR.
When the WRITE payload is sent in a Read chunk, the client places
the GETATTR in the inline part of the RPC/RDMA message, just after
the WRITE operation (sans payload). The position value in the Read
chunk enables the receiver to insert the Read chunk at the correct
place in the received XDR stream; that is between the WRITE and
GETATTR.
According to RFC 8166, an NFS/RDMA client does not have to add XDR
round-up to the Read chunk that carries the WRITE payload. The
receiver adds XDR round-up padding if it is absent and the
receiver's XDR decoder requires it to be present.
Commit 193bcb7b3719 ("svcrdma: Populate tail iovec when receiving")
attempted to add support for receiving such a compound so that just
the WRITE payload appears in rq_arg's page list, and the trailing
GETATTR is placed in rq_arg's tail iovec. (TCP just strings the
whole compound into the head iovec and page list, without regard
to the alignment of the WRITE payload).
The server transport logic also had to accommodate the optional XDR
round-up of the Read chunk, which it did simply by lengthening the
tail iovec when round-up was needed. This approach is adequate for
the NFSv2 and NFSv3 WRITE decoders.
Unfortunately it is not sufficient for nfsd4_decode_write. When the
Read chunk length is a couple of bytes less than PAGE_SIZE, the
computation at the end of nfsd4_decode_write allows argp->pagelen to
go negative, which breaks the logic in read_buf that looks for the
tail iovec.
The result is that a WRITE operation whose payload length is just
less than a multiple of a page succeeds, but the subsequent GETATTR
in the same compound fails with NFS4ERR_OP_ILLEGAL because the XDR
decoder can't find it. Clients ignore the error, but they must
update their attribute cache via a separate round trip.
As nfsd4_decode_write appears to expect the payload itself to always
have appropriate XDR round-up, have svc_rdma_build_normal_read_chunk
add the Read chunk XDR round-up to the page_len rather than
lengthening the tail iovec.
Reported-by: Olga Kornievskaia <kolga@netapp.com>
Fixes: 193bcb7b3719 ("svcrdma: Populate tail iovec when receiving")
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Tested-by: Olga Kornievskaia <kolga@netapp.com>
Signed-off-by: J. Bruce Fields <bfields@redhat.com>
2018-02-02 19:28:59 +00:00
|
|
|
* If the client already rounded up the chunk length, the
|
|
|
|
* length does not change. Otherwise, the length of the page
|
|
|
|
* list is increased to include XDR round-up.
|
|
|
|
*
|
|
|
|
* Currently these chunks always start at page offset 0,
|
|
|
|
* thus the rounded-up length never crosses a page boundary.
|
2017-08-18 15:12:35 +00:00
|
|
|
*/
|
svcrdma: Fix Read chunk round-up
A single NFSv4 WRITE compound can often have three operations:
PUTFH, WRITE, then GETATTR.
When the WRITE payload is sent in a Read chunk, the client places
the GETATTR in the inline part of the RPC/RDMA message, just after
the WRITE operation (sans payload). The position value in the Read
chunk enables the receiver to insert the Read chunk at the correct
place in the received XDR stream; that is between the WRITE and
GETATTR.
According to RFC 8166, an NFS/RDMA client does not have to add XDR
round-up to the Read chunk that carries the WRITE payload. The
receiver adds XDR round-up padding if it is absent and the
receiver's XDR decoder requires it to be present.
Commit 193bcb7b3719 ("svcrdma: Populate tail iovec when receiving")
attempted to add support for receiving such a compound so that just
the WRITE payload appears in rq_arg's page list, and the trailing
GETATTR is placed in rq_arg's tail iovec. (TCP just strings the
whole compound into the head iovec and page list, without regard
to the alignment of the WRITE payload).
The server transport logic also had to accommodate the optional XDR
round-up of the Read chunk, which it did simply by lengthening the
tail iovec when round-up was needed. This approach is adequate for
the NFSv2 and NFSv3 WRITE decoders.
Unfortunately it is not sufficient for nfsd4_decode_write. When the
Read chunk length is a couple of bytes less than PAGE_SIZE, the
computation at the end of nfsd4_decode_write allows argp->pagelen to
go negative, which breaks the logic in read_buf that looks for the
tail iovec.
The result is that a WRITE operation whose payload length is just
less than a multiple of a page succeeds, but the subsequent GETATTR
in the same compound fails with NFS4ERR_OP_ILLEGAL because the XDR
decoder can't find it. Clients ignore the error, but they must
update their attribute cache via a separate round trip.
As nfsd4_decode_write appears to expect the payload itself to always
have appropriate XDR round-up, have svc_rdma_build_normal_read_chunk
add the Read chunk XDR round-up to the page_len rather than
lengthening the tail iovec.
Reported-by: Olga Kornievskaia <kolga@netapp.com>
Fixes: 193bcb7b3719 ("svcrdma: Populate tail iovec when receiving")
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Tested-by: Olga Kornievskaia <kolga@netapp.com>
Signed-off-by: J. Bruce Fields <bfields@redhat.com>
2018-02-02 19:28:59 +00:00
|
|
|
info->ri_chunklen = XDR_QUADLEN(info->ri_chunklen) << 2;
|
2017-06-23 21:18:24 +00:00
|
|
|
|
2018-05-07 19:27:21 +00:00
|
|
|
head->rc_arg.page_len = info->ri_chunklen;
|
|
|
|
head->rc_arg.len += info->ri_chunklen;
|
|
|
|
head->rc_arg.buflen += info->ri_chunklen;
|
2017-06-23 21:18:24 +00:00
|
|
|
|
|
|
|
out:
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Construct RDMA Reads to pull over a Position Zero Read chunk.
|
|
|
|
* The start of the data lands in the first page just after
|
|
|
|
* the Transport header, and the rest lands in the page list of
|
2018-05-07 19:27:21 +00:00
|
|
|
* head->rc_arg.pages.
|
2017-06-23 21:18:24 +00:00
|
|
|
*
|
|
|
|
* Assumptions:
|
|
|
|
* - A PZRC has an XDR-aligned length (no implicit round-up).
|
|
|
|
* - There can be no trailing inline content (IOW, we assume
|
|
|
|
* a PZRC is never sent in an RDMA_MSG message, though it's
|
|
|
|
* allowed by spec).
|
|
|
|
*/
|
|
|
|
static int svc_rdma_build_pz_read_chunk(struct svc_rqst *rqstp,
|
|
|
|
struct svc_rdma_read_info *info,
|
|
|
|
__be32 *p)
|
|
|
|
{
|
2018-05-07 19:27:21 +00:00
|
|
|
struct svc_rdma_recv_ctxt *head = info->ri_readctxt;
|
2017-06-23 21:18:24 +00:00
|
|
|
int ret;
|
|
|
|
|
|
|
|
ret = svc_rdma_build_read_chunk(rqstp, info, p);
|
|
|
|
if (ret < 0)
|
|
|
|
goto out;
|
|
|
|
|
2018-05-07 19:27:11 +00:00
|
|
|
trace_svcrdma_encode_pzr(info->ri_chunklen);
|
|
|
|
|
2018-05-07 19:27:21 +00:00
|
|
|
head->rc_arg.len += info->ri_chunklen;
|
|
|
|
head->rc_arg.buflen += info->ri_chunklen;
|
2017-06-23 21:18:24 +00:00
|
|
|
|
2018-05-07 19:27:43 +00:00
|
|
|
head->rc_hdr_count = 1;
|
|
|
|
head->rc_arg.head[0].iov_base = page_address(head->rc_pages[0]);
|
|
|
|
head->rc_arg.head[0].iov_len = min_t(size_t, PAGE_SIZE,
|
|
|
|
info->ri_chunklen);
|
|
|
|
|
|
|
|
head->rc_arg.page_len = info->ri_chunklen -
|
|
|
|
head->rc_arg.head[0].iov_len;
|
2017-06-23 21:18:24 +00:00
|
|
|
|
|
|
|
out:
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* svc_rdma_recv_read_chunk - Pull a Read chunk from the client
|
|
|
|
* @rdma: controlling RDMA transport
|
|
|
|
* @rqstp: set of pages to use as Read sink buffers
|
|
|
|
* @head: pages under I/O collect here
|
|
|
|
* @p: pointer to start of Read chunk
|
|
|
|
*
|
|
|
|
* Returns:
|
|
|
|
* %0 if all needed RDMA Reads were posted successfully,
|
|
|
|
* %-EINVAL if client provided too many segments,
|
|
|
|
* %-ENOMEM if rdma_rw context pool was exhausted,
|
|
|
|
* %-ENOTCONN if posting failed (connection is lost),
|
|
|
|
* %-EIO if rdma_rw initialization failed (DMA mapping, etc).
|
|
|
|
*
|
|
|
|
* Assumptions:
|
|
|
|
* - All Read segments in @p have the same Position value.
|
|
|
|
*/
|
|
|
|
int svc_rdma_recv_read_chunk(struct svcxprt_rdma *rdma, struct svc_rqst *rqstp,
|
2018-05-07 19:27:21 +00:00
|
|
|
struct svc_rdma_recv_ctxt *head, __be32 *p)
|
2017-06-23 21:18:24 +00:00
|
|
|
{
|
|
|
|
struct svc_rdma_read_info *info;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
/* The request (with page list) is constructed in
|
2018-05-07 19:27:21 +00:00
|
|
|
* head->rc_arg. Pages involved with RDMA Read I/O are
|
2017-06-23 21:18:24 +00:00
|
|
|
* transferred there.
|
|
|
|
*/
|
2018-05-07 19:27:21 +00:00
|
|
|
head->rc_arg.head[0] = rqstp->rq_arg.head[0];
|
|
|
|
head->rc_arg.tail[0] = rqstp->rq_arg.tail[0];
|
|
|
|
head->rc_arg.pages = head->rc_pages;
|
|
|
|
head->rc_arg.page_base = 0;
|
|
|
|
head->rc_arg.page_len = 0;
|
|
|
|
head->rc_arg.len = rqstp->rq_arg.len;
|
|
|
|
head->rc_arg.buflen = rqstp->rq_arg.buflen;
|
2017-06-23 21:18:24 +00:00
|
|
|
|
|
|
|
info = svc_rdma_read_info_alloc(rdma);
|
|
|
|
if (!info)
|
|
|
|
return -ENOMEM;
|
|
|
|
info->ri_readctxt = head;
|
2018-05-07 19:27:43 +00:00
|
|
|
info->ri_pageno = 0;
|
|
|
|
info->ri_pageoff = 0;
|
2017-06-23 21:18:24 +00:00
|
|
|
|
|
|
|
info->ri_position = be32_to_cpup(p + 1);
|
|
|
|
if (info->ri_position)
|
|
|
|
ret = svc_rdma_build_normal_read_chunk(rqstp, info, p);
|
|
|
|
else
|
|
|
|
ret = svc_rdma_build_pz_read_chunk(rqstp, info, p);
|
|
|
|
if (ret < 0)
|
2018-07-27 15:18:59 +00:00
|
|
|
goto out_err;
|
2017-06-23 21:18:24 +00:00
|
|
|
|
|
|
|
ret = svc_rdma_post_chunk_ctxt(&info->ri_cc);
|
|
|
|
if (ret < 0)
|
2018-07-27 15:18:59 +00:00
|
|
|
goto out_err;
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
out_err:
|
|
|
|
svc_rdma_read_info_free(info);
|
2017-06-23 21:18:24 +00:00
|
|
|
return ret;
|
|
|
|
}
|