License cleanup: add SPDX GPL-2.0 license identifier to files with no license
Many source files in the tree are missing licensing information, which
makes it harder for compliance tools to determine the correct license.
By default all files without license information are under the default
license of the kernel, which is GPL version 2.
Update the files which contain no license information with the 'GPL-2.0'
SPDX license identifier. The SPDX identifier is a legally binding
shorthand, which can be used instead of the full boiler plate text.
This patch is based on work done by Thomas Gleixner and Kate Stewart and
Philippe Ombredanne.
How this work was done:
Patches were generated and checked against linux-4.14-rc6 for a subset of
the use cases:
- file had no licensing information it it.
- file was a */uapi/* one with no licensing information in it,
- file was a */uapi/* one with existing licensing information,
Further patches will be generated in subsequent months to fix up cases
where non-standard license headers were used, and references to license
had to be inferred by heuristics based on keywords.
The analysis to determine which SPDX License Identifier to be applied to
a file was done in a spreadsheet of side by side results from of the
output of two independent scanners (ScanCode & Windriver) producing SPDX
tag:value files created by Philippe Ombredanne. Philippe prepared the
base worksheet, and did an initial spot review of a few 1000 files.
The 4.13 kernel was the starting point of the analysis with 60,537 files
assessed. Kate Stewart did a file by file comparison of the scanner
results in the spreadsheet to determine which SPDX license identifier(s)
to be applied to the file. She confirmed any determination that was not
immediately clear with lawyers working with the Linux Foundation.
Criteria used to select files for SPDX license identifier tagging was:
- Files considered eligible had to be source code files.
- Make and config files were included as candidates if they contained >5
lines of source
- File already had some variant of a license header in it (even if <5
lines).
All documentation files were explicitly excluded.
The following heuristics were used to determine which SPDX license
identifiers to apply.
- when both scanners couldn't find any license traces, file was
considered to have no license information in it, and the top level
COPYING file license applied.
For non */uapi/* files that summary was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 11139
and resulted in the first patch in this series.
If that file was a */uapi/* path one, it was "GPL-2.0 WITH
Linux-syscall-note" otherwise it was "GPL-2.0". Results of that was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 WITH Linux-syscall-note 930
and resulted in the second patch in this series.
- if a file had some form of licensing information in it, and was one
of the */uapi/* ones, it was denoted with the Linux-syscall-note if
any GPL family license was found in the file or had no licensing in
it (per prior point). Results summary:
SPDX license identifier # files
---------------------------------------------------|------
GPL-2.0 WITH Linux-syscall-note 270
GPL-2.0+ WITH Linux-syscall-note 169
((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) 21
((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) 17
LGPL-2.1+ WITH Linux-syscall-note 15
GPL-1.0+ WITH Linux-syscall-note 14
((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) 5
LGPL-2.0+ WITH Linux-syscall-note 4
LGPL-2.1 WITH Linux-syscall-note 3
((GPL-2.0 WITH Linux-syscall-note) OR MIT) 3
((GPL-2.0 WITH Linux-syscall-note) AND MIT) 1
and that resulted in the third patch in this series.
- when the two scanners agreed on the detected license(s), that became
the concluded license(s).
- when there was disagreement between the two scanners (one detected a
license but the other didn't, or they both detected different
licenses) a manual inspection of the file occurred.
- In most cases a manual inspection of the information in the file
resulted in a clear resolution of the license that should apply (and
which scanner probably needed to revisit its heuristics).
- When it was not immediately clear, the license identifier was
confirmed with lawyers working with the Linux Foundation.
- If there was any question as to the appropriate license identifier,
the file was flagged for further research and to be revisited later
in time.
In total, over 70 hours of logged manual review was done on the
spreadsheet to determine the SPDX license identifiers to apply to the
source files by Kate, Philippe, Thomas and, in some cases, confirmation
by lawyers working with the Linux Foundation.
Kate also obtained a third independent scan of the 4.13 code base from
FOSSology, and compared selected files where the other two scanners
disagreed against that SPDX file, to see if there was new insights. The
Windriver scanner is based on an older version of FOSSology in part, so
they are related.
Thomas did random spot checks in about 500 files from the spreadsheets
for the uapi headers and agreed with SPDX license identifier in the
files he inspected. For the non-uapi files Thomas did random spot checks
in about 15000 files.
In initial set of patches against 4.14-rc6, 3 files were found to have
copy/paste license identifier errors, and have been fixed to reflect the
correct identifier.
Additionally Philippe spent 10 hours this week doing a detailed manual
inspection and review of the 12,461 patched files from the initial patch
version early this week with:
- a full scancode scan run, collecting the matched texts, detected
license ids and scores
- reviewing anything where there was a license detected (about 500+
files) to ensure that the applied SPDX license was correct
- reviewing anything where there was no detection but the patch license
was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
SPDX license was correct
This produced a worksheet with 20 files needing minor correction. This
worksheet was then exported into 3 different .csv files for the
different types of files to be modified.
These .csv files were then reviewed by Greg. Thomas wrote a script to
parse the csv files and add the proper SPDX tag to the file, in the
format that the file expected. This script was further refined by Greg
based on the output to detect more types of files automatically and to
distinguish between header and source .c files (which need different
comment types.) Finally Greg ran the script using the .csv files to
generate the patches.
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-11-01 14:07:57 +00:00
|
|
|
// SPDX-License-Identifier: GPL-2.0
|
2017-01-09 15:55:17 +00:00
|
|
|
/*
|
|
|
|
* Shared Memory Communications over RDMA (SMC-R) and RoCE
|
|
|
|
*
|
|
|
|
* Basic Transport Functions exploiting Infiniband API
|
|
|
|
*
|
|
|
|
* Copyright IBM Corp. 2016
|
|
|
|
*
|
|
|
|
* Author(s): Ursula Braun <ubraun@linux.vnet.ibm.com>
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include <linux/socket.h>
|
|
|
|
#include <linux/if_vlan.h>
|
|
|
|
#include <linux/random.h>
|
|
|
|
#include <linux/workqueue.h>
|
2019-11-16 16:47:29 +00:00
|
|
|
#include <linux/wait.h>
|
2019-11-16 16:47:30 +00:00
|
|
|
#include <linux/reboot.h>
|
2020-07-08 15:05:13 +00:00
|
|
|
#include <linux/mutex.h>
|
2020-12-01 19:20:44 +00:00
|
|
|
#include <linux/list.h>
|
|
|
|
#include <linux/smc.h>
|
2017-01-09 15:55:17 +00:00
|
|
|
#include <net/tcp.h>
|
|
|
|
#include <net/sock.h>
|
|
|
|
#include <rdma/ib_verbs.h>
|
2018-06-05 05:40:19 +00:00
|
|
|
#include <rdma/ib_cache.h>
|
2017-01-09 15:55:17 +00:00
|
|
|
|
|
|
|
#include "smc.h"
|
|
|
|
#include "smc_clc.h"
|
|
|
|
#include "smc_core.h"
|
|
|
|
#include "smc_ib.h"
|
2017-01-09 15:55:19 +00:00
|
|
|
#include "smc_wr.h"
|
2017-01-09 15:55:21 +00:00
|
|
|
#include "smc_llc.h"
|
2017-01-09 15:55:22 +00:00
|
|
|
#include "smc_cdc.h"
|
2017-01-09 15:55:25 +00:00
|
|
|
#include "smc_close.h"
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
#include "smc_ism.h"
|
2020-12-01 19:20:44 +00:00
|
|
|
#include "smc_netlink.h"
|
2021-06-16 14:52:55 +00:00
|
|
|
#include "smc_stats.h"
|
2021-11-01 07:39:16 +00:00
|
|
|
#include "smc_tracepoint.h"
|
2017-01-09 15:55:17 +00:00
|
|
|
|
2017-09-21 07:16:31 +00:00
|
|
|
#define SMC_LGR_NUM_INCR 256
|
|
|
|
#define SMC_LGR_FREE_DELAY_SERV (600 * HZ)
|
2018-05-23 14:38:12 +00:00
|
|
|
#define SMC_LGR_FREE_DELAY_CLNT (SMC_LGR_FREE_DELAY_SERV + 10 * HZ)
|
2017-01-09 15:55:17 +00:00
|
|
|
|
2020-12-01 19:20:49 +00:00
|
|
|
struct smc_lgr_list smc_lgr_list = { /* established link groups */
|
2018-05-18 07:34:11 +00:00
|
|
|
.lock = __SPIN_LOCK_UNLOCKED(smc_lgr_list.lock),
|
|
|
|
.list = LIST_HEAD_INIT(smc_lgr_list.list),
|
|
|
|
.num = 0,
|
|
|
|
};
|
2017-01-09 15:55:21 +00:00
|
|
|
|
2019-12-12 21:35:41 +00:00
|
|
|
static atomic_t lgr_cnt = ATOMIC_INIT(0); /* number of existing link groups */
|
2019-11-16 16:47:29 +00:00
|
|
|
static DECLARE_WAIT_QUEUE_HEAD(lgrs_deleted);
|
|
|
|
|
2018-05-18 07:34:17 +00:00
|
|
|
static void smc_buf_free(struct smc_link_group *lgr, bool is_rmb,
|
|
|
|
struct smc_buf_desc *buf_desc);
|
2020-02-17 15:24:54 +00:00
|
|
|
static void __smc_lgr_terminate(struct smc_link_group *lgr, bool soft);
|
2018-05-03 15:57:38 +00:00
|
|
|
|
2020-05-01 10:48:08 +00:00
|
|
|
static void smc_link_down_work(struct work_struct *work);
|
2020-05-01 10:48:07 +00:00
|
|
|
|
2019-10-09 08:07:44 +00:00
|
|
|
/* return head of link group list and its lock for a given link group */
|
|
|
|
static inline struct list_head *smc_lgr_list_head(struct smc_link_group *lgr,
|
|
|
|
spinlock_t **lgr_lock)
|
|
|
|
{
|
|
|
|
if (lgr->is_smcd) {
|
|
|
|
*lgr_lock = &lgr->smcd->lgr_lock;
|
|
|
|
return &lgr->smcd->lgr_list;
|
|
|
|
}
|
|
|
|
|
|
|
|
*lgr_lock = &smc_lgr_list.lock;
|
|
|
|
return &smc_lgr_list.list;
|
|
|
|
}
|
|
|
|
|
2020-12-01 19:20:39 +00:00
|
|
|
static void smc_ibdev_cnt_inc(struct smc_link *lnk)
|
|
|
|
{
|
|
|
|
atomic_inc(&lnk->smcibdev->lnk_cnt_by_port[lnk->ibport - 1]);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void smc_ibdev_cnt_dec(struct smc_link *lnk)
|
|
|
|
{
|
|
|
|
atomic_dec(&lnk->smcibdev->lnk_cnt_by_port[lnk->ibport - 1]);
|
|
|
|
}
|
|
|
|
|
2018-03-14 10:01:02 +00:00
|
|
|
static void smc_lgr_schedule_free_work(struct smc_link_group *lgr)
|
|
|
|
{
|
|
|
|
/* client link group creation always follows the server link group
|
|
|
|
* creation. For client use a somewhat higher removal delay time,
|
|
|
|
* otherwise there is a risk of out-of-sync link groups.
|
|
|
|
*/
|
2020-09-10 16:48:26 +00:00
|
|
|
if (!lgr->freeing) {
|
2019-10-21 14:13:11 +00:00
|
|
|
mod_delayed_work(system_wq, &lgr->free_work,
|
|
|
|
(!lgr->is_smcd && lgr->role == SMC_CLNT) ?
|
|
|
|
SMC_LGR_FREE_DELAY_CLNT :
|
|
|
|
SMC_LGR_FREE_DELAY_SERV);
|
|
|
|
}
|
2018-03-14 10:01:02 +00:00
|
|
|
}
|
|
|
|
|
2017-01-09 15:55:17 +00:00
|
|
|
/* Register connection's alert token in our lookup structure.
|
|
|
|
* To use rbtrees we have to implement our own insert core.
|
|
|
|
* Requires @conns_lock
|
|
|
|
* @smc connection to register
|
|
|
|
* Returns 0 on success, != otherwise.
|
|
|
|
*/
|
|
|
|
static void smc_lgr_add_alert_token(struct smc_connection *conn)
|
|
|
|
{
|
|
|
|
struct rb_node **link, *parent = NULL;
|
|
|
|
u32 token = conn->alert_token_local;
|
|
|
|
|
|
|
|
link = &conn->lgr->conns_all.rb_node;
|
|
|
|
while (*link) {
|
|
|
|
struct smc_connection *cur = rb_entry(*link,
|
|
|
|
struct smc_connection, alert_node);
|
|
|
|
|
|
|
|
parent = *link;
|
|
|
|
if (cur->alert_token_local > token)
|
|
|
|
link = &parent->rb_left;
|
|
|
|
else
|
|
|
|
link = &parent->rb_right;
|
|
|
|
}
|
|
|
|
/* Put the new node there */
|
|
|
|
rb_link_node(&conn->alert_node, parent, link);
|
|
|
|
rb_insert_color(&conn->alert_node, &conn->lgr->conns_all);
|
|
|
|
}
|
|
|
|
|
2020-05-04 12:18:43 +00:00
|
|
|
/* assign an SMC-R link to the connection */
|
|
|
|
static int smcr_lgr_conn_assign_link(struct smc_connection *conn, bool first)
|
|
|
|
{
|
|
|
|
enum smc_link_state expected = first ? SMC_LNK_ACTIVATING :
|
|
|
|
SMC_LNK_ACTIVE;
|
|
|
|
int i, j;
|
|
|
|
|
|
|
|
/* do link balancing */
|
net/smc: Reset connection when trying to use SMCRv2 fails.
We found a crash when using SMCRv2 with 2 Mellanox ConnectX-4. It
can be reproduced by:
- smc_run nginx
- smc_run wrk -t 32 -c 500 -d 30 http://<ip>:<port>
BUG: kernel NULL pointer dereference, address: 0000000000000014
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 8000000108713067 P4D 8000000108713067 PUD 151127067 PMD 0
Oops: 0000 [#1] PREEMPT SMP PTI
CPU: 4 PID: 2441 Comm: kworker/4:249 Kdump: loaded Tainted: G W E 6.4.0-rc1+ #42
Workqueue: smc_hs_wq smc_listen_work [smc]
RIP: 0010:smc_clc_send_confirm_accept+0x284/0x580 [smc]
RSP: 0018:ffffb8294b2d7c78 EFLAGS: 00010a06
RAX: ffff8f1873238880 RBX: ffffb8294b2d7dc8 RCX: 0000000000000000
RDX: 00000000000000b4 RSI: 0000000000000001 RDI: 0000000000b40c00
RBP: ffffb8294b2d7db8 R08: ffff8f1815c5860c R09: 0000000000000000
R10: 0000000000000400 R11: 0000000000000000 R12: ffff8f1846f56180
R13: ffff8f1815c5860c R14: 0000000000000001 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffff8f1aefd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000014 CR3: 00000001027a0001 CR4: 00000000003706e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>
? mlx5_ib_map_mr_sg+0xa1/0xd0 [mlx5_ib]
? smcr_buf_map_link+0x24b/0x290 [smc]
? __smc_buf_create+0x4ee/0x9b0 [smc]
smc_clc_send_accept+0x4c/0xb0 [smc]
smc_listen_work+0x346/0x650 [smc]
? __schedule+0x279/0x820
process_one_work+0x1e5/0x3f0
worker_thread+0x4d/0x2f0
? __pfx_worker_thread+0x10/0x10
kthread+0xe5/0x120
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>
During the CLC handshake, server sequentially tries available SMCRv2
and SMCRv1 devices in smc_listen_work().
If an SMCRv2 device is found. SMCv2 based link group and link will be
assigned to the connection. Then assumed that some buffer assignment
errors happen later in the CLC handshake, such as RMB registration
failure, server will give up SMCRv2 and try SMCRv1 device instead. But
the resources assigned to the connection won't be reset.
When server tries SMCRv1 device, the connection creation process will
be executed again. Since conn->lnk has been assigned when trying SMCRv2,
it will not be set to the correct SMCRv1 link in
smcr_lgr_conn_assign_link(). So in such situation, conn->lgr points to
correct SMCRv1 link group but conn->lnk points to the SMCRv2 link
mistakenly.
Then in smc_clc_send_confirm_accept(), conn->rmb_desc->mr[link->link_idx]
will be accessed. Since the link->link_idx is not correct, the related
MR may not have been initialized, so crash happens.
| Try SMCRv2 device first
| |-> conn->lgr: assign existed SMCRv2 link group;
| |-> conn->link: assign existed SMCRv2 link (link_idx may be 1 in SMC_LGR_SYMMETRIC);
| |-> sndbuf & RMB creation fails, quit;
|
| Try SMCRv1 device then
| |-> conn->lgr: create SMCRv1 link group and assign;
| |-> conn->link: keep SMCRv2 link mistakenly;
| |-> sndbuf & RMB creation succeed, only RMB->mr[link_idx = 0]
| initialized.
|
| Then smc_clc_send_confirm_accept() accesses
| conn->rmb_desc->mr[conn->link->link_idx, which is 1], then crash.
v
This patch tries to fix this by cleaning conn->lnk before assigning
link. In addition, it is better to reset the connection and clean the
resources assigned if trying SMCRv2 failed in buffer creation or
registration.
Fixes: e49300a6bf62 ("net/smc: add listen processing for SMC-Rv2")
Link: https://lore.kernel.org/r/20220523055056.2078994-1-liuyacan@corp.netease.com/
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Reviewed-by: Tony Lu <tonylu@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2023-05-18 05:14:55 +00:00
|
|
|
conn->lnk = NULL; /* reset conn->lnk first */
|
2020-05-04 12:18:43 +00:00
|
|
|
for (i = 0; i < SMC_LINKS_PER_LGR_MAX; i++) {
|
|
|
|
struct smc_link *lnk = &conn->lgr->lnk[i];
|
|
|
|
|
2020-05-04 12:18:44 +00:00
|
|
|
if (lnk->state != expected || lnk->link_is_asym)
|
2020-05-04 12:18:43 +00:00
|
|
|
continue;
|
|
|
|
if (conn->lgr->role == SMC_CLNT) {
|
|
|
|
conn->lnk = lnk; /* temporary, SMC server assigns link*/
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
if (conn->lgr->conns_num % 2) {
|
|
|
|
for (j = i + 1; j < SMC_LINKS_PER_LGR_MAX; j++) {
|
|
|
|
struct smc_link *lnk2;
|
|
|
|
|
|
|
|
lnk2 = &conn->lgr->lnk[j];
|
2020-05-04 12:18:44 +00:00
|
|
|
if (lnk2->state == expected &&
|
|
|
|
!lnk2->link_is_asym) {
|
2020-05-04 12:18:43 +00:00
|
|
|
conn->lnk = lnk2;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
if (!conn->lnk)
|
|
|
|
conn->lnk = lnk;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
if (!conn->lnk)
|
|
|
|
return SMC_CLC_DECL_NOACTLINK;
|
2020-12-01 19:20:38 +00:00
|
|
|
atomic_inc(&conn->lnk->conn_cnt);
|
2020-05-04 12:18:43 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2017-01-09 15:55:17 +00:00
|
|
|
/* Register connection in link group by assigning an alert token
|
|
|
|
* registered in a search tree.
|
|
|
|
* Requires @conns_lock
|
|
|
|
* Note that '0' is a reserved value and not assigned.
|
|
|
|
*/
|
2020-05-04 12:18:43 +00:00
|
|
|
static int smc_lgr_register_conn(struct smc_connection *conn, bool first)
|
2017-01-09 15:55:17 +00:00
|
|
|
{
|
|
|
|
struct smc_sock *smc = container_of(conn, struct smc_sock, conn);
|
|
|
|
static atomic_t nexttoken = ATOMIC_INIT(0);
|
2020-05-04 12:18:43 +00:00
|
|
|
int rc;
|
2017-01-09 15:55:17 +00:00
|
|
|
|
2020-05-04 12:18:43 +00:00
|
|
|
if (!conn->lgr->is_smcd) {
|
|
|
|
rc = smcr_lgr_conn_assign_link(conn, first);
|
2022-01-06 12:42:08 +00:00
|
|
|
if (rc) {
|
|
|
|
conn->lgr = NULL;
|
2020-05-04 12:18:43 +00:00
|
|
|
return rc;
|
2022-01-06 12:42:08 +00:00
|
|
|
}
|
2020-05-04 12:18:43 +00:00
|
|
|
}
|
2017-01-09 15:55:17 +00:00
|
|
|
/* find a new alert_token_local value not yet used by some connection
|
|
|
|
* in this link group
|
|
|
|
*/
|
|
|
|
sock_hold(&smc->sk); /* sock_put in smc_lgr_unregister_conn() */
|
|
|
|
while (!conn->alert_token_local) {
|
|
|
|
conn->alert_token_local = atomic_inc_return(&nexttoken);
|
|
|
|
if (smc_lgr_find_conn(conn->alert_token_local, conn->lgr))
|
|
|
|
conn->alert_token_local = 0;
|
|
|
|
}
|
|
|
|
smc_lgr_add_alert_token(conn);
|
|
|
|
conn->lgr->conns_num++;
|
2020-04-29 15:10:41 +00:00
|
|
|
return 0;
|
2017-01-09 15:55:17 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Unregister connection and reset the alert token of the given connection<
|
|
|
|
*/
|
|
|
|
static void __smc_lgr_unregister_conn(struct smc_connection *conn)
|
|
|
|
{
|
|
|
|
struct smc_sock *smc = container_of(conn, struct smc_sock, conn);
|
|
|
|
struct smc_link_group *lgr = conn->lgr;
|
|
|
|
|
|
|
|
rb_erase(&conn->alert_node, &lgr->conns_all);
|
2020-12-01 19:20:38 +00:00
|
|
|
if (conn->lnk)
|
|
|
|
atomic_dec(&conn->lnk->conn_cnt);
|
2017-01-09 15:55:17 +00:00
|
|
|
lgr->conns_num--;
|
|
|
|
conn->alert_token_local = 0;
|
|
|
|
sock_put(&smc->sk); /* sock_hold in smc_lgr_register_conn() */
|
|
|
|
}
|
|
|
|
|
2018-10-25 11:25:28 +00:00
|
|
|
/* Unregister connection from lgr
|
2017-01-09 15:55:17 +00:00
|
|
|
*/
|
|
|
|
static void smc_lgr_unregister_conn(struct smc_connection *conn)
|
|
|
|
{
|
|
|
|
struct smc_link_group *lgr = conn->lgr;
|
|
|
|
|
2022-01-13 08:36:41 +00:00
|
|
|
if (!smc_conn_lgr_valid(conn))
|
2019-01-30 17:51:02 +00:00
|
|
|
return;
|
2017-01-09 15:55:17 +00:00
|
|
|
write_lock_bh(&lgr->conns_lock);
|
|
|
|
if (conn->alert_token_local) {
|
|
|
|
__smc_lgr_unregister_conn(conn);
|
|
|
|
}
|
|
|
|
write_unlock_bh(&lgr->conns_lock);
|
|
|
|
}
|
|
|
|
|
2020-12-01 19:20:44 +00:00
|
|
|
int smc_nl_get_sys_info(struct sk_buff *skb, struct netlink_callback *cb)
|
|
|
|
{
|
|
|
|
struct smc_nl_dmp_ctx *cb_ctx = smc_nl_dmp_ctx(cb);
|
|
|
|
char hostname[SMC_MAX_HOSTNAME_LEN + 1];
|
|
|
|
char smc_seid[SMC_MAX_EID_LEN + 1];
|
|
|
|
struct nlattr *attrs;
|
|
|
|
u8 *seid = NULL;
|
|
|
|
u8 *host = NULL;
|
|
|
|
void *nlh;
|
|
|
|
|
|
|
|
nlh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid, cb->nlh->nlmsg_seq,
|
|
|
|
&smc_gen_nl_family, NLM_F_MULTI,
|
|
|
|
SMC_NETLINK_GET_SYS_INFO);
|
|
|
|
if (!nlh)
|
|
|
|
goto errmsg;
|
|
|
|
if (cb_ctx->pos[0])
|
|
|
|
goto errout;
|
|
|
|
attrs = nla_nest_start(skb, SMC_GEN_SYS_INFO);
|
|
|
|
if (!attrs)
|
|
|
|
goto errout;
|
|
|
|
if (nla_put_u8(skb, SMC_NLA_SYS_VER, SMC_V2))
|
|
|
|
goto errattr;
|
|
|
|
if (nla_put_u8(skb, SMC_NLA_SYS_REL, SMC_RELEASE))
|
|
|
|
goto errattr;
|
|
|
|
if (nla_put_u8(skb, SMC_NLA_SYS_IS_ISM_V2, smc_ism_is_v2_capable()))
|
|
|
|
goto errattr;
|
2021-10-16 09:37:51 +00:00
|
|
|
if (nla_put_u8(skb, SMC_NLA_SYS_IS_SMCR_V2, true))
|
|
|
|
goto errattr;
|
2020-12-01 19:20:44 +00:00
|
|
|
smc_clc_get_hostname(&host);
|
|
|
|
if (host) {
|
2021-01-12 16:21:21 +00:00
|
|
|
memcpy(hostname, host, SMC_MAX_HOSTNAME_LEN);
|
|
|
|
hostname[SMC_MAX_HOSTNAME_LEN] = 0;
|
2020-12-01 19:20:44 +00:00
|
|
|
if (nla_put_string(skb, SMC_NLA_SYS_LOCAL_HOST, hostname))
|
|
|
|
goto errattr;
|
|
|
|
}
|
2021-09-14 08:35:06 +00:00
|
|
|
if (smc_ism_is_v2_capable()) {
|
|
|
|
smc_ism_get_system_eid(&seid);
|
2021-01-12 16:21:22 +00:00
|
|
|
memcpy(smc_seid, seid, SMC_MAX_EID_LEN);
|
|
|
|
smc_seid[SMC_MAX_EID_LEN] = 0;
|
2020-12-01 19:20:44 +00:00
|
|
|
if (nla_put_string(skb, SMC_NLA_SYS_SEID, smc_seid))
|
|
|
|
goto errattr;
|
|
|
|
}
|
|
|
|
nla_nest_end(skb, attrs);
|
|
|
|
genlmsg_end(skb, nlh);
|
|
|
|
cb_ctx->pos[0] = 1;
|
|
|
|
return skb->len;
|
|
|
|
|
|
|
|
errattr:
|
|
|
|
nla_nest_cancel(skb, attrs);
|
|
|
|
errout:
|
|
|
|
genlmsg_cancel(skb, nlh);
|
|
|
|
errmsg:
|
|
|
|
return skb->len;
|
|
|
|
}
|
|
|
|
|
2021-10-16 09:37:51 +00:00
|
|
|
/* Fill SMC_NLA_LGR_D_V2_COMMON/SMC_NLA_LGR_R_V2_COMMON nested attributes */
|
|
|
|
static int smc_nl_fill_lgr_v2_common(struct smc_link_group *lgr,
|
|
|
|
struct sk_buff *skb,
|
|
|
|
struct netlink_callback *cb,
|
|
|
|
struct nlattr *v2_attrs)
|
|
|
|
{
|
|
|
|
char smc_host[SMC_MAX_HOSTNAME_LEN + 1];
|
|
|
|
char smc_eid[SMC_MAX_EID_LEN + 1];
|
|
|
|
|
|
|
|
if (nla_put_u8(skb, SMC_NLA_LGR_V2_VER, lgr->smc_version))
|
|
|
|
goto errv2attr;
|
|
|
|
if (nla_put_u8(skb, SMC_NLA_LGR_V2_REL, lgr->peer_smc_release))
|
|
|
|
goto errv2attr;
|
|
|
|
if (nla_put_u8(skb, SMC_NLA_LGR_V2_OS, lgr->peer_os))
|
|
|
|
goto errv2attr;
|
|
|
|
memcpy(smc_host, lgr->peer_hostname, SMC_MAX_HOSTNAME_LEN);
|
|
|
|
smc_host[SMC_MAX_HOSTNAME_LEN] = 0;
|
|
|
|
if (nla_put_string(skb, SMC_NLA_LGR_V2_PEER_HOST, smc_host))
|
|
|
|
goto errv2attr;
|
|
|
|
memcpy(smc_eid, lgr->negotiated_eid, SMC_MAX_EID_LEN);
|
|
|
|
smc_eid[SMC_MAX_EID_LEN] = 0;
|
|
|
|
if (nla_put_string(skb, SMC_NLA_LGR_V2_NEG_EID, smc_eid))
|
|
|
|
goto errv2attr;
|
|
|
|
|
|
|
|
nla_nest_end(skb, v2_attrs);
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
errv2attr:
|
|
|
|
nla_nest_cancel(skb, v2_attrs);
|
|
|
|
return -EMSGSIZE;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int smc_nl_fill_smcr_lgr_v2(struct smc_link_group *lgr,
|
|
|
|
struct sk_buff *skb,
|
|
|
|
struct netlink_callback *cb)
|
|
|
|
{
|
|
|
|
struct nlattr *v2_attrs;
|
|
|
|
|
|
|
|
v2_attrs = nla_nest_start(skb, SMC_NLA_LGR_R_V2);
|
|
|
|
if (!v2_attrs)
|
|
|
|
goto errattr;
|
|
|
|
if (nla_put_u8(skb, SMC_NLA_LGR_R_V2_DIRECT, !lgr->uses_gateway))
|
|
|
|
goto errv2attr;
|
|
|
|
|
|
|
|
nla_nest_end(skb, v2_attrs);
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
errv2attr:
|
|
|
|
nla_nest_cancel(skb, v2_attrs);
|
|
|
|
errattr:
|
|
|
|
return -EMSGSIZE;
|
|
|
|
}
|
|
|
|
|
2020-12-01 19:20:45 +00:00
|
|
|
static int smc_nl_fill_lgr(struct smc_link_group *lgr,
|
|
|
|
struct sk_buff *skb,
|
|
|
|
struct netlink_callback *cb)
|
|
|
|
{
|
|
|
|
char smc_target[SMC_MAX_PNETID_LEN + 1];
|
2021-10-16 09:37:51 +00:00
|
|
|
struct nlattr *attrs, *v2_attrs;
|
2020-12-01 19:20:45 +00:00
|
|
|
|
|
|
|
attrs = nla_nest_start(skb, SMC_GEN_LGR_SMCR);
|
|
|
|
if (!attrs)
|
|
|
|
goto errout;
|
|
|
|
|
|
|
|
if (nla_put_u32(skb, SMC_NLA_LGR_R_ID, *((u32 *)&lgr->id)))
|
|
|
|
goto errattr;
|
|
|
|
if (nla_put_u32(skb, SMC_NLA_LGR_R_CONNS_NUM, lgr->conns_num))
|
|
|
|
goto errattr;
|
|
|
|
if (nla_put_u8(skb, SMC_NLA_LGR_R_ROLE, lgr->role))
|
|
|
|
goto errattr;
|
|
|
|
if (nla_put_u8(skb, SMC_NLA_LGR_R_TYPE, lgr->type))
|
|
|
|
goto errattr;
|
2022-07-14 09:44:05 +00:00
|
|
|
if (nla_put_u8(skb, SMC_NLA_LGR_R_BUF_TYPE, lgr->buf_type))
|
|
|
|
goto errattr;
|
2020-12-01 19:20:45 +00:00
|
|
|
if (nla_put_u8(skb, SMC_NLA_LGR_R_VLAN_ID, lgr->vlan_id))
|
|
|
|
goto errattr;
|
2021-12-28 13:06:10 +00:00
|
|
|
if (nla_put_u64_64bit(skb, SMC_NLA_LGR_R_NET_COOKIE,
|
|
|
|
lgr->net->net_cookie, SMC_NLA_LGR_R_PAD))
|
|
|
|
goto errattr;
|
2021-01-12 16:21:22 +00:00
|
|
|
memcpy(smc_target, lgr->pnet_id, SMC_MAX_PNETID_LEN);
|
|
|
|
smc_target[SMC_MAX_PNETID_LEN] = 0;
|
2020-12-01 19:20:45 +00:00
|
|
|
if (nla_put_string(skb, SMC_NLA_LGR_R_PNETID, smc_target))
|
|
|
|
goto errattr;
|
2021-10-16 09:37:51 +00:00
|
|
|
if (lgr->smc_version > SMC_V1) {
|
|
|
|
v2_attrs = nla_nest_start(skb, SMC_NLA_LGR_R_V2_COMMON);
|
|
|
|
if (!v2_attrs)
|
|
|
|
goto errattr;
|
|
|
|
if (smc_nl_fill_lgr_v2_common(lgr, skb, cb, v2_attrs))
|
|
|
|
goto errattr;
|
|
|
|
if (smc_nl_fill_smcr_lgr_v2(lgr, skb, cb))
|
|
|
|
goto errattr;
|
|
|
|
}
|
2020-12-01 19:20:45 +00:00
|
|
|
|
|
|
|
nla_nest_end(skb, attrs);
|
|
|
|
return 0;
|
|
|
|
errattr:
|
|
|
|
nla_nest_cancel(skb, attrs);
|
|
|
|
errout:
|
|
|
|
return -EMSGSIZE;
|
|
|
|
}
|
|
|
|
|
2020-12-01 19:20:46 +00:00
|
|
|
static int smc_nl_fill_lgr_link(struct smc_link_group *lgr,
|
|
|
|
struct smc_link *link,
|
|
|
|
struct sk_buff *skb,
|
|
|
|
struct netlink_callback *cb)
|
|
|
|
{
|
2021-01-12 16:21:22 +00:00
|
|
|
char smc_ibname[IB_DEVICE_NAME_MAX];
|
2020-12-01 19:20:46 +00:00
|
|
|
u8 smc_gid_target[41];
|
|
|
|
struct nlattr *attrs;
|
|
|
|
u32 link_uid = 0;
|
|
|
|
void *nlh;
|
|
|
|
|
|
|
|
nlh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid, cb->nlh->nlmsg_seq,
|
|
|
|
&smc_gen_nl_family, NLM_F_MULTI,
|
|
|
|
SMC_NETLINK_GET_LINK_SMCR);
|
|
|
|
if (!nlh)
|
|
|
|
goto errmsg;
|
|
|
|
|
|
|
|
attrs = nla_nest_start(skb, SMC_GEN_LINK_SMCR);
|
|
|
|
if (!attrs)
|
|
|
|
goto errout;
|
|
|
|
|
|
|
|
if (nla_put_u8(skb, SMC_NLA_LINK_ID, link->link_id))
|
|
|
|
goto errattr;
|
|
|
|
if (nla_put_u32(skb, SMC_NLA_LINK_STATE, link->state))
|
|
|
|
goto errattr;
|
|
|
|
if (nla_put_u32(skb, SMC_NLA_LINK_CONN_CNT,
|
|
|
|
atomic_read(&link->conn_cnt)))
|
|
|
|
goto errattr;
|
|
|
|
if (nla_put_u8(skb, SMC_NLA_LINK_IB_PORT, link->ibport))
|
|
|
|
goto errattr;
|
|
|
|
if (nla_put_u32(skb, SMC_NLA_LINK_NET_DEV, link->ndev_ifidx))
|
|
|
|
goto errattr;
|
|
|
|
snprintf(smc_ibname, sizeof(smc_ibname), "%s", link->ibname);
|
|
|
|
if (nla_put_string(skb, SMC_NLA_LINK_IB_DEV, smc_ibname))
|
|
|
|
goto errattr;
|
|
|
|
memcpy(&link_uid, link->link_uid, sizeof(link_uid));
|
|
|
|
if (nla_put_u32(skb, SMC_NLA_LINK_UID, link_uid))
|
|
|
|
goto errattr;
|
|
|
|
memcpy(&link_uid, link->peer_link_uid, sizeof(link_uid));
|
|
|
|
if (nla_put_u32(skb, SMC_NLA_LINK_PEER_UID, link_uid))
|
|
|
|
goto errattr;
|
|
|
|
memset(smc_gid_target, 0, sizeof(smc_gid_target));
|
|
|
|
smc_gid_be16_convert(smc_gid_target, link->gid);
|
|
|
|
if (nla_put_string(skb, SMC_NLA_LINK_GID, smc_gid_target))
|
|
|
|
goto errattr;
|
|
|
|
memset(smc_gid_target, 0, sizeof(smc_gid_target));
|
|
|
|
smc_gid_be16_convert(smc_gid_target, link->peer_gid);
|
|
|
|
if (nla_put_string(skb, SMC_NLA_LINK_PEER_GID, smc_gid_target))
|
|
|
|
goto errattr;
|
|
|
|
|
|
|
|
nla_nest_end(skb, attrs);
|
|
|
|
genlmsg_end(skb, nlh);
|
|
|
|
return 0;
|
|
|
|
errattr:
|
|
|
|
nla_nest_cancel(skb, attrs);
|
|
|
|
errout:
|
|
|
|
genlmsg_cancel(skb, nlh);
|
|
|
|
errmsg:
|
|
|
|
return -EMSGSIZE;
|
|
|
|
}
|
|
|
|
|
2020-12-01 19:20:45 +00:00
|
|
|
static int smc_nl_handle_lgr(struct smc_link_group *lgr,
|
|
|
|
struct sk_buff *skb,
|
2020-12-01 19:20:46 +00:00
|
|
|
struct netlink_callback *cb,
|
|
|
|
bool list_links)
|
2020-12-01 19:20:45 +00:00
|
|
|
{
|
|
|
|
void *nlh;
|
2020-12-01 19:20:46 +00:00
|
|
|
int i;
|
2020-12-01 19:20:45 +00:00
|
|
|
|
|
|
|
nlh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid, cb->nlh->nlmsg_seq,
|
|
|
|
&smc_gen_nl_family, NLM_F_MULTI,
|
|
|
|
SMC_NETLINK_GET_LGR_SMCR);
|
|
|
|
if (!nlh)
|
|
|
|
goto errmsg;
|
|
|
|
if (smc_nl_fill_lgr(lgr, skb, cb))
|
|
|
|
goto errout;
|
|
|
|
|
|
|
|
genlmsg_end(skb, nlh);
|
2020-12-01 19:20:46 +00:00
|
|
|
if (!list_links)
|
|
|
|
goto out;
|
|
|
|
for (i = 0; i < SMC_LINKS_PER_LGR_MAX; i++) {
|
|
|
|
if (!smc_link_usable(&lgr->lnk[i]))
|
|
|
|
continue;
|
|
|
|
if (smc_nl_fill_lgr_link(lgr, &lgr->lnk[i], skb, cb))
|
|
|
|
goto errout;
|
|
|
|
}
|
|
|
|
out:
|
2020-12-01 19:20:45 +00:00
|
|
|
return 0;
|
|
|
|
|
|
|
|
errout:
|
|
|
|
genlmsg_cancel(skb, nlh);
|
|
|
|
errmsg:
|
|
|
|
return -EMSGSIZE;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void smc_nl_fill_lgr_list(struct smc_lgr_list *smc_lgr,
|
|
|
|
struct sk_buff *skb,
|
2020-12-01 19:20:46 +00:00
|
|
|
struct netlink_callback *cb,
|
|
|
|
bool list_links)
|
2020-12-01 19:20:45 +00:00
|
|
|
{
|
|
|
|
struct smc_nl_dmp_ctx *cb_ctx = smc_nl_dmp_ctx(cb);
|
|
|
|
struct smc_link_group *lgr;
|
|
|
|
int snum = cb_ctx->pos[0];
|
|
|
|
int num = 0;
|
|
|
|
|
|
|
|
spin_lock_bh(&smc_lgr->lock);
|
|
|
|
list_for_each_entry(lgr, &smc_lgr->list, list) {
|
|
|
|
if (num < snum)
|
|
|
|
goto next;
|
2020-12-01 19:20:46 +00:00
|
|
|
if (smc_nl_handle_lgr(lgr, skb, cb, list_links))
|
2020-12-01 19:20:45 +00:00
|
|
|
goto errout;
|
|
|
|
next:
|
|
|
|
num++;
|
|
|
|
}
|
|
|
|
errout:
|
|
|
|
spin_unlock_bh(&smc_lgr->lock);
|
|
|
|
cb_ctx->pos[0] = num;
|
|
|
|
}
|
|
|
|
|
2020-12-01 19:20:47 +00:00
|
|
|
static int smc_nl_fill_smcd_lgr(struct smc_link_group *lgr,
|
|
|
|
struct sk_buff *skb,
|
|
|
|
struct netlink_callback *cb)
|
|
|
|
{
|
|
|
|
char smc_pnet[SMC_MAX_PNETID_LEN + 1];
|
2023-01-23 18:17:50 +00:00
|
|
|
struct smcd_dev *smcd = lgr->smcd;
|
2020-12-01 19:20:47 +00:00
|
|
|
struct nlattr *attrs;
|
|
|
|
void *nlh;
|
|
|
|
|
|
|
|
nlh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid, cb->nlh->nlmsg_seq,
|
|
|
|
&smc_gen_nl_family, NLM_F_MULTI,
|
|
|
|
SMC_NETLINK_GET_LGR_SMCD);
|
|
|
|
if (!nlh)
|
|
|
|
goto errmsg;
|
|
|
|
|
|
|
|
attrs = nla_nest_start(skb, SMC_GEN_LGR_SMCD);
|
|
|
|
if (!attrs)
|
|
|
|
goto errout;
|
|
|
|
|
|
|
|
if (nla_put_u32(skb, SMC_NLA_LGR_D_ID, *((u32 *)&lgr->id)))
|
|
|
|
goto errattr;
|
2023-01-23 18:17:50 +00:00
|
|
|
if (nla_put_u64_64bit(skb, SMC_NLA_LGR_D_GID,
|
|
|
|
smcd->ops->get_local_gid(smcd),
|
|
|
|
SMC_NLA_LGR_D_PAD))
|
2020-12-01 19:20:47 +00:00
|
|
|
goto errattr;
|
|
|
|
if (nla_put_u64_64bit(skb, SMC_NLA_LGR_D_PEER_GID, lgr->peer_gid,
|
|
|
|
SMC_NLA_LGR_D_PAD))
|
|
|
|
goto errattr;
|
|
|
|
if (nla_put_u8(skb, SMC_NLA_LGR_D_VLAN_ID, lgr->vlan_id))
|
|
|
|
goto errattr;
|
|
|
|
if (nla_put_u32(skb, SMC_NLA_LGR_D_CONNS_NUM, lgr->conns_num))
|
|
|
|
goto errattr;
|
|
|
|
if (nla_put_u32(skb, SMC_NLA_LGR_D_CHID, smc_ism_get_chid(lgr->smcd)))
|
|
|
|
goto errattr;
|
2021-01-12 16:21:22 +00:00
|
|
|
memcpy(smc_pnet, lgr->smcd->pnetid, SMC_MAX_PNETID_LEN);
|
|
|
|
smc_pnet[SMC_MAX_PNETID_LEN] = 0;
|
2020-12-01 19:20:47 +00:00
|
|
|
if (nla_put_string(skb, SMC_NLA_LGR_D_PNETID, smc_pnet))
|
|
|
|
goto errattr;
|
2021-10-16 09:37:51 +00:00
|
|
|
if (lgr->smc_version > SMC_V1) {
|
|
|
|
struct nlattr *v2_attrs;
|
2020-12-01 19:20:47 +00:00
|
|
|
|
2021-10-16 09:37:51 +00:00
|
|
|
v2_attrs = nla_nest_start(skb, SMC_NLA_LGR_D_V2_COMMON);
|
|
|
|
if (!v2_attrs)
|
|
|
|
goto errattr;
|
|
|
|
if (smc_nl_fill_lgr_v2_common(lgr, skb, cb, v2_attrs))
|
|
|
|
goto errattr;
|
|
|
|
}
|
2020-12-01 19:20:47 +00:00
|
|
|
nla_nest_end(skb, attrs);
|
|
|
|
genlmsg_end(skb, nlh);
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
errattr:
|
|
|
|
nla_nest_cancel(skb, attrs);
|
|
|
|
errout:
|
|
|
|
genlmsg_cancel(skb, nlh);
|
|
|
|
errmsg:
|
|
|
|
return -EMSGSIZE;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int smc_nl_handle_smcd_lgr(struct smcd_dev *dev,
|
|
|
|
struct sk_buff *skb,
|
|
|
|
struct netlink_callback *cb)
|
|
|
|
{
|
|
|
|
struct smc_nl_dmp_ctx *cb_ctx = smc_nl_dmp_ctx(cb);
|
|
|
|
struct smc_link_group *lgr;
|
|
|
|
int snum = cb_ctx->pos[1];
|
|
|
|
int rc = 0, num = 0;
|
|
|
|
|
|
|
|
spin_lock_bh(&dev->lgr_lock);
|
|
|
|
list_for_each_entry(lgr, &dev->lgr_list, list) {
|
|
|
|
if (!lgr->is_smcd)
|
|
|
|
continue;
|
|
|
|
if (num < snum)
|
|
|
|
goto next;
|
|
|
|
rc = smc_nl_fill_smcd_lgr(lgr, skb, cb);
|
|
|
|
if (rc)
|
|
|
|
goto errout;
|
|
|
|
next:
|
|
|
|
num++;
|
|
|
|
}
|
|
|
|
errout:
|
|
|
|
spin_unlock_bh(&dev->lgr_lock);
|
|
|
|
cb_ctx->pos[1] = num;
|
|
|
|
return rc;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int smc_nl_fill_smcd_dev(struct smcd_dev_list *dev_list,
|
|
|
|
struct sk_buff *skb,
|
|
|
|
struct netlink_callback *cb)
|
|
|
|
{
|
|
|
|
struct smc_nl_dmp_ctx *cb_ctx = smc_nl_dmp_ctx(cb);
|
|
|
|
struct smcd_dev *smcd_dev;
|
|
|
|
int snum = cb_ctx->pos[0];
|
|
|
|
int rc = 0, num = 0;
|
|
|
|
|
|
|
|
mutex_lock(&dev_list->mutex);
|
|
|
|
list_for_each_entry(smcd_dev, &dev_list->list, list) {
|
|
|
|
if (list_empty(&smcd_dev->lgr_list))
|
|
|
|
continue;
|
|
|
|
if (num < snum)
|
|
|
|
goto next;
|
|
|
|
rc = smc_nl_handle_smcd_lgr(smcd_dev, skb, cb);
|
|
|
|
if (rc)
|
|
|
|
goto errout;
|
|
|
|
next:
|
|
|
|
num++;
|
|
|
|
}
|
|
|
|
errout:
|
|
|
|
mutex_unlock(&dev_list->mutex);
|
|
|
|
cb_ctx->pos[0] = num;
|
|
|
|
return rc;
|
|
|
|
}
|
|
|
|
|
2020-12-01 19:20:45 +00:00
|
|
|
int smcr_nl_get_lgr(struct sk_buff *skb, struct netlink_callback *cb)
|
|
|
|
{
|
2020-12-01 19:20:46 +00:00
|
|
|
bool list_links = false;
|
|
|
|
|
|
|
|
smc_nl_fill_lgr_list(&smc_lgr_list, skb, cb, list_links);
|
|
|
|
return skb->len;
|
|
|
|
}
|
|
|
|
|
|
|
|
int smcr_nl_get_link(struct sk_buff *skb, struct netlink_callback *cb)
|
|
|
|
{
|
|
|
|
bool list_links = true;
|
|
|
|
|
|
|
|
smc_nl_fill_lgr_list(&smc_lgr_list, skb, cb, list_links);
|
2020-12-01 19:20:45 +00:00
|
|
|
return skb->len;
|
|
|
|
}
|
|
|
|
|
2020-12-01 19:20:47 +00:00
|
|
|
int smcd_nl_get_lgr(struct sk_buff *skb, struct netlink_callback *cb)
|
|
|
|
{
|
|
|
|
smc_nl_fill_smcd_dev(&smcd_dev_list, skb, cb);
|
|
|
|
return skb->len;
|
|
|
|
}
|
|
|
|
|
2022-01-06 12:42:08 +00:00
|
|
|
void smc_lgr_cleanup_early(struct smc_link_group *lgr)
|
2020-02-25 15:34:36 +00:00
|
|
|
{
|
2020-05-03 12:38:47 +00:00
|
|
|
spinlock_t *lgr_lock;
|
2020-02-25 15:34:36 +00:00
|
|
|
|
|
|
|
if (!lgr)
|
|
|
|
return;
|
|
|
|
|
2021-12-01 03:02:30 +00:00
|
|
|
smc_lgr_list_head(lgr, &lgr_lock);
|
2020-05-03 12:38:47 +00:00
|
|
|
spin_lock_bh(lgr_lock);
|
|
|
|
/* do not use this link group for new connections */
|
2021-12-01 03:02:30 +00:00
|
|
|
if (!list_empty(&lgr->list))
|
|
|
|
list_del_init(&lgr->list);
|
2020-05-03 12:38:47 +00:00
|
|
|
spin_unlock_bh(lgr_lock);
|
2020-09-10 16:48:26 +00:00
|
|
|
__smc_lgr_terminate(lgr, true);
|
2020-02-25 15:34:36 +00:00
|
|
|
}
|
|
|
|
|
2020-05-04 12:18:46 +00:00
|
|
|
static void smcr_lgr_link_deactivate_all(struct smc_link_group *lgr)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < SMC_LINKS_PER_LGR_MAX; i++) {
|
|
|
|
struct smc_link *lnk = &lgr->lnk[i];
|
|
|
|
|
2021-12-28 09:03:24 +00:00
|
|
|
if (smc_link_sendable(lnk))
|
2020-05-04 12:18:46 +00:00
|
|
|
lnk->state = SMC_LNK_INACTIVE;
|
|
|
|
}
|
2020-07-08 15:05:11 +00:00
|
|
|
wake_up_all(&lgr->llc_msg_waiter);
|
|
|
|
wake_up_all(&lgr->llc_flow_waiter);
|
2020-05-04 12:18:46 +00:00
|
|
|
}
|
|
|
|
|
2018-11-22 09:26:35 +00:00
|
|
|
static void smc_lgr_free(struct smc_link_group *lgr);
|
|
|
|
|
2017-01-09 15:55:17 +00:00
|
|
|
static void smc_lgr_free_work(struct work_struct *work)
|
|
|
|
{
|
|
|
|
struct smc_link_group *lgr = container_of(to_delayed_work(work),
|
|
|
|
struct smc_link_group,
|
|
|
|
free_work);
|
2019-10-09 08:07:44 +00:00
|
|
|
spinlock_t *lgr_lock;
|
2017-01-09 15:55:17 +00:00
|
|
|
bool conns;
|
|
|
|
|
2019-10-09 08:07:44 +00:00
|
|
|
smc_lgr_list_head(lgr, &lgr_lock);
|
|
|
|
spin_lock_bh(lgr_lock);
|
2019-10-21 14:13:11 +00:00
|
|
|
if (lgr->freeing) {
|
|
|
|
spin_unlock_bh(lgr_lock);
|
|
|
|
return;
|
|
|
|
}
|
2017-01-09 15:55:17 +00:00
|
|
|
read_lock_bh(&lgr->conns_lock);
|
|
|
|
conns = RB_EMPTY_ROOT(&lgr->conns_all);
|
|
|
|
read_unlock_bh(&lgr->conns_lock);
|
|
|
|
if (!conns) { /* number of lgr connections is no longer zero */
|
2019-10-09 08:07:44 +00:00
|
|
|
spin_unlock_bh(lgr_lock);
|
2017-01-09 15:55:17 +00:00
|
|
|
return;
|
|
|
|
}
|
2019-10-21 14:13:09 +00:00
|
|
|
list_del_init(&lgr->list); /* remove from smc_lgr_list */
|
2019-10-21 14:13:11 +00:00
|
|
|
lgr->freeing = 1; /* this instance does the freeing, no new schedule */
|
|
|
|
spin_unlock_bh(lgr_lock);
|
|
|
|
cancel_delayed_work(&lgr->free_work);
|
2018-07-25 14:35:33 +00:00
|
|
|
|
2020-05-04 12:18:42 +00:00
|
|
|
if (!lgr->is_smcd && !lgr->terminating)
|
|
|
|
smc_llc_send_link_delete_all(lgr, true,
|
|
|
|
SMC_LLC_DEL_PROG_INIT_TERM);
|
2019-11-14 12:02:41 +00:00
|
|
|
if (lgr->is_smcd && !lgr->terminating)
|
2019-10-21 14:13:11 +00:00
|
|
|
smc_ism_signal_shutdown(lgr);
|
2020-05-04 12:18:46 +00:00
|
|
|
if (!lgr->is_smcd)
|
|
|
|
smcr_lgr_link_deactivate_all(lgr);
|
2019-10-21 14:13:11 +00:00
|
|
|
smc_lgr_free(lgr);
|
2017-01-09 15:55:17 +00:00
|
|
|
}
|
|
|
|
|
2019-10-21 14:13:14 +00:00
|
|
|
static void smc_lgr_terminate_work(struct work_struct *work)
|
|
|
|
{
|
|
|
|
struct smc_link_group *lgr = container_of(work, struct smc_link_group,
|
|
|
|
terminate_work);
|
|
|
|
|
2020-02-17 15:24:54 +00:00
|
|
|
__smc_lgr_terminate(lgr, true);
|
2019-10-21 14:13:14 +00:00
|
|
|
}
|
|
|
|
|
2020-04-29 15:10:39 +00:00
|
|
|
/* return next unique link id for the lgr */
|
|
|
|
static u8 smcr_next_link_id(struct smc_link_group *lgr)
|
|
|
|
{
|
|
|
|
u8 link_id;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
while (1) {
|
2021-11-15 09:45:07 +00:00
|
|
|
again:
|
2020-04-29 15:10:39 +00:00
|
|
|
link_id = ++lgr->next_link_id;
|
|
|
|
if (!link_id) /* skip zero as link_id */
|
|
|
|
link_id = ++lgr->next_link_id;
|
|
|
|
for (i = 0; i < SMC_LINKS_PER_LGR_MAX; i++) {
|
2020-04-29 15:10:43 +00:00
|
|
|
if (smc_link_usable(&lgr->lnk[i]) &&
|
2020-04-29 15:10:39 +00:00
|
|
|
lgr->lnk[i].link_id == link_id)
|
2021-11-15 09:45:07 +00:00
|
|
|
goto again;
|
2020-04-29 15:10:39 +00:00
|
|
|
}
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
return link_id;
|
|
|
|
}
|
|
|
|
|
2020-12-01 19:20:41 +00:00
|
|
|
static void smcr_copy_dev_info_to_link(struct smc_link *link)
|
|
|
|
{
|
|
|
|
struct smc_ib_device *smcibdev = link->smcibdev;
|
|
|
|
|
|
|
|
snprintf(link->ibname, sizeof(link->ibname), "%s",
|
|
|
|
smcibdev->ibdev->name);
|
|
|
|
link->ndev_ifidx = smcibdev->ndev_ifidx[link->ibport - 1];
|
|
|
|
}
|
|
|
|
|
2020-05-03 12:38:40 +00:00
|
|
|
int smcr_link_init(struct smc_link_group *lgr, struct smc_link *lnk,
|
|
|
|
u8 link_idx, struct smc_init_info *ini)
|
2020-04-29 15:10:38 +00:00
|
|
|
{
|
2021-10-16 09:37:46 +00:00
|
|
|
struct smc_ib_device *smcibdev;
|
2020-04-29 15:10:38 +00:00
|
|
|
u8 rndvec[3];
|
|
|
|
int rc;
|
|
|
|
|
2021-10-16 09:37:46 +00:00
|
|
|
if (lgr->smc_version == SMC_V2) {
|
|
|
|
lnk->smcibdev = ini->smcrv2.ib_dev_v2;
|
|
|
|
lnk->ibport = ini->smcrv2.ib_port_v2;
|
|
|
|
} else {
|
|
|
|
lnk->smcibdev = ini->ib_dev;
|
|
|
|
lnk->ibport = ini->ib_port;
|
|
|
|
}
|
|
|
|
get_device(&lnk->smcibdev->ibdev->dev);
|
|
|
|
atomic_inc(&lnk->smcibdev->lnk_cnt);
|
2022-01-13 08:36:42 +00:00
|
|
|
refcount_set(&lnk->refcnt, 1); /* link refcnt is set to 1 */
|
|
|
|
lnk->clearing = 0;
|
2021-10-16 09:37:46 +00:00
|
|
|
lnk->path_mtu = lnk->smcibdev->pattr[lnk->ibport - 1].active_mtu;
|
2020-04-29 15:10:39 +00:00
|
|
|
lnk->link_id = smcr_next_link_id(lgr);
|
2020-04-29 15:10:40 +00:00
|
|
|
lnk->lgr = lgr;
|
2022-01-13 08:36:40 +00:00
|
|
|
smc_lgr_hold(lgr); /* lgr_put in smcr_link_clear() */
|
2020-04-29 15:10:39 +00:00
|
|
|
lnk->link_idx = link_idx;
|
2022-09-06 13:01:39 +00:00
|
|
|
lnk->wr_rx_id_compl = 0;
|
2020-12-01 19:20:39 +00:00
|
|
|
smc_ibdev_cnt_inc(lnk);
|
2020-12-01 19:20:41 +00:00
|
|
|
smcr_copy_dev_info_to_link(lnk);
|
2020-12-01 19:20:38 +00:00
|
|
|
atomic_set(&lnk->conn_cnt, 0);
|
2020-05-04 12:18:47 +00:00
|
|
|
smc_llc_link_set_uid(lnk);
|
2020-05-01 10:48:08 +00:00
|
|
|
INIT_WORK(&lnk->link_down_wrk, smc_link_down_work);
|
2021-10-16 09:37:46 +00:00
|
|
|
if (!lnk->smcibdev->initialized) {
|
|
|
|
rc = (int)smc_ib_setup_per_ibdev(lnk->smcibdev);
|
2020-04-29 15:10:38 +00:00
|
|
|
if (rc)
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
get_random_bytes(rndvec, sizeof(rndvec));
|
|
|
|
lnk->psn_initial = rndvec[0] + (rndvec[1] << 8) +
|
|
|
|
(rndvec[2] << 16);
|
|
|
|
rc = smc_ib_determine_gid(lnk->smcibdev, lnk->ibport,
|
2021-10-16 09:37:48 +00:00
|
|
|
ini->vlan_id, lnk->gid, &lnk->sgid_index,
|
|
|
|
lgr->smc_version == SMC_V2 ?
|
|
|
|
&ini->smcrv2 : NULL);
|
2020-04-29 15:10:38 +00:00
|
|
|
if (rc)
|
|
|
|
goto out;
|
|
|
|
rc = smc_llc_link_init(lnk);
|
|
|
|
if (rc)
|
|
|
|
goto out;
|
|
|
|
rc = smc_wr_alloc_link_mem(lnk);
|
|
|
|
if (rc)
|
|
|
|
goto clear_llc_lnk;
|
|
|
|
rc = smc_ib_create_protection_domain(lnk);
|
|
|
|
if (rc)
|
|
|
|
goto free_link_mem;
|
|
|
|
rc = smc_ib_create_queue_pair(lnk);
|
|
|
|
if (rc)
|
|
|
|
goto dealloc_pd;
|
|
|
|
rc = smc_wr_create_link(lnk);
|
|
|
|
if (rc)
|
|
|
|
goto destroy_qp;
|
2020-07-18 13:06:16 +00:00
|
|
|
lnk->state = SMC_LNK_ACTIVATING;
|
2020-04-29 15:10:38 +00:00
|
|
|
return 0;
|
|
|
|
|
|
|
|
destroy_qp:
|
|
|
|
smc_ib_destroy_queue_pair(lnk);
|
|
|
|
dealloc_pd:
|
|
|
|
smc_ib_dealloc_protection_domain(lnk);
|
|
|
|
free_link_mem:
|
|
|
|
smc_wr_free_link_mem(lnk);
|
|
|
|
clear_llc_lnk:
|
2020-05-05 13:01:20 +00:00
|
|
|
smc_llc_link_clear(lnk, false);
|
2020-04-29 15:10:38 +00:00
|
|
|
out:
|
2020-12-01 19:20:39 +00:00
|
|
|
smc_ibdev_cnt_dec(lnk);
|
2021-10-16 09:37:46 +00:00
|
|
|
put_device(&lnk->smcibdev->ibdev->dev);
|
|
|
|
smcibdev = lnk->smcibdev;
|
2020-04-29 15:10:38 +00:00
|
|
|
memset(lnk, 0, sizeof(struct smc_link));
|
2020-04-29 15:10:43 +00:00
|
|
|
lnk->state = SMC_LNK_UNUSED;
|
2021-10-16 09:37:46 +00:00
|
|
|
if (!atomic_dec_return(&smcibdev->lnk_cnt))
|
|
|
|
wake_up(&smcibdev->lnks_deleted);
|
2022-01-13 08:36:40 +00:00
|
|
|
smc_lgr_put(lgr); /* lgr_hold above */
|
2020-04-29 15:10:38 +00:00
|
|
|
return rc;
|
|
|
|
}
|
|
|
|
|
2017-01-09 15:55:17 +00:00
|
|
|
/* create a new SMC link group */
|
2019-04-12 10:57:26 +00:00
|
|
|
static int smc_lgr_create(struct smc_sock *smc, struct smc_init_info *ini)
|
2017-01-09 15:55:17 +00:00
|
|
|
{
|
|
|
|
struct smc_link_group *lgr;
|
2019-10-09 08:07:43 +00:00
|
|
|
struct list_head *lgr_list;
|
2023-01-23 18:17:52 +00:00
|
|
|
struct smcd_dev *smcd;
|
2017-01-09 15:55:17 +00:00
|
|
|
struct smc_link *lnk;
|
2019-10-09 08:07:44 +00:00
|
|
|
spinlock_t *lgr_lock;
|
2020-04-29 15:10:39 +00:00
|
|
|
u8 link_idx;
|
2017-01-09 15:55:17 +00:00
|
|
|
int rc = 0;
|
2017-01-09 15:55:18 +00:00
|
|
|
int i;
|
2017-01-09 15:55:17 +00:00
|
|
|
|
2019-04-12 10:57:26 +00:00
|
|
|
if (ini->is_smcd && ini->vlan_id) {
|
2020-09-26 10:44:29 +00:00
|
|
|
if (smc_ism_get_vlan(ini->ism_dev[ini->ism_selected],
|
|
|
|
ini->vlan_id)) {
|
2019-04-12 10:57:30 +00:00
|
|
|
rc = SMC_CLC_DECL_ISMVLANERR;
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
goto out;
|
2019-04-12 10:57:30 +00:00
|
|
|
}
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
}
|
|
|
|
|
2017-01-09 15:55:17 +00:00
|
|
|
lgr = kzalloc(sizeof(*lgr), GFP_KERNEL);
|
|
|
|
if (!lgr) {
|
2019-04-12 10:57:30 +00:00
|
|
|
rc = SMC_CLC_DECL_MEM;
|
2019-10-10 08:16:09 +00:00
|
|
|
goto ism_put_vlan;
|
2017-01-09 15:55:17 +00:00
|
|
|
}
|
2020-09-10 16:48:29 +00:00
|
|
|
lgr->tx_wq = alloc_workqueue("smc_tx_wq-%*phN", 0, 0,
|
|
|
|
SMC_LGR_ID_SIZE, &lgr->id);
|
|
|
|
if (!lgr->tx_wq) {
|
|
|
|
rc = -ENOMEM;
|
|
|
|
goto free_lgr;
|
|
|
|
}
|
2019-04-12 10:57:26 +00:00
|
|
|
lgr->is_smcd = ini->is_smcd;
|
2018-05-15 15:05:03 +00:00
|
|
|
lgr->sync_err = 0;
|
2019-10-21 14:13:11 +00:00
|
|
|
lgr->terminating = 0;
|
|
|
|
lgr->freeing = 0;
|
2019-04-12 10:57:26 +00:00
|
|
|
lgr->vlan_id = ini->vlan_id;
|
2022-01-13 08:36:40 +00:00
|
|
|
refcount_set(&lgr->refcnt, 1); /* set lgr refcnt to 1 */
|
2023-02-02 08:26:42 +00:00
|
|
|
init_rwsem(&lgr->sndbufs_lock);
|
|
|
|
init_rwsem(&lgr->rmbs_lock);
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
rwlock_init(&lgr->conns_lock);
|
2017-01-09 15:55:18 +00:00
|
|
|
for (i = 0; i < SMC_RMBE_SIZES; i++) {
|
|
|
|
INIT_LIST_HEAD(&lgr->sndbufs[i]);
|
|
|
|
INIT_LIST_HEAD(&lgr->rmbs[i]);
|
|
|
|
}
|
2020-04-29 15:10:39 +00:00
|
|
|
lgr->next_link_id = 0;
|
2018-05-18 07:34:11 +00:00
|
|
|
smc_lgr_list.num += SMC_LGR_NUM_INCR;
|
|
|
|
memcpy(&lgr->id, (u8 *)&smc_lgr_list.num, SMC_LGR_ID_SIZE);
|
2017-01-09 15:55:17 +00:00
|
|
|
INIT_DELAYED_WORK(&lgr->free_work, smc_lgr_free_work);
|
2019-10-21 14:13:14 +00:00
|
|
|
INIT_WORK(&lgr->terminate_work, smc_lgr_terminate_work);
|
2017-01-09 15:55:17 +00:00
|
|
|
lgr->conns_all = RB_ROOT;
|
2019-04-12 10:57:26 +00:00
|
|
|
if (ini->is_smcd) {
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
/* SMC-D specific settings */
|
2023-01-23 18:17:52 +00:00
|
|
|
smcd = ini->ism_dev[ini->ism_selected];
|
|
|
|
get_device(smcd->ops->get_dev(smcd));
|
2020-09-26 10:44:29 +00:00
|
|
|
lgr->peer_gid = ini->ism_peer_gid[ini->ism_selected];
|
|
|
|
lgr->smcd = ini->ism_dev[ini->ism_selected];
|
|
|
|
lgr_list = &ini->ism_dev[ini->ism_selected]->lgr_list;
|
2019-10-09 08:07:44 +00:00
|
|
|
lgr_lock = &lgr->smcd->lgr_lock;
|
2020-09-26 10:44:31 +00:00
|
|
|
lgr->smc_version = ini->smcd_version;
|
2019-11-14 12:02:40 +00:00
|
|
|
lgr->peer_shutdown = 0;
|
2020-09-26 10:44:29 +00:00
|
|
|
atomic_inc(&ini->ism_dev[ini->ism_selected]->lgr_cnt);
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
} else {
|
|
|
|
/* SMC-R specific settings */
|
2021-10-16 09:37:46 +00:00
|
|
|
struct smc_ib_device *ibdev;
|
|
|
|
int ibport;
|
|
|
|
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
lgr->role = smc->listen_smc ? SMC_SERV : SMC_CLNT;
|
2021-10-16 09:37:46 +00:00
|
|
|
lgr->smc_version = ini->smcr_version;
|
|
|
|
memcpy(lgr->peer_systemid, ini->peer_systemid,
|
2019-04-12 10:57:26 +00:00
|
|
|
SMC_SYSTEMID_LEN);
|
2021-10-16 09:37:46 +00:00
|
|
|
if (lgr->smc_version == SMC_V2) {
|
|
|
|
ibdev = ini->smcrv2.ib_dev_v2;
|
|
|
|
ibport = ini->smcrv2.ib_port_v2;
|
|
|
|
lgr->saddr = ini->smcrv2.saddr;
|
|
|
|
lgr->uses_gateway = ini->smcrv2.uses_gateway;
|
|
|
|
memcpy(lgr->nexthop_mac, ini->smcrv2.nexthop_mac,
|
|
|
|
ETH_ALEN);
|
|
|
|
} else {
|
|
|
|
ibdev = ini->ib_dev;
|
|
|
|
ibport = ini->ib_port;
|
|
|
|
}
|
|
|
|
memcpy(lgr->pnet_id, ibdev->pnetid[ibport - 1],
|
2020-05-01 10:48:06 +00:00
|
|
|
SMC_MAX_PNETID_LEN);
|
2022-10-14 09:34:36 +00:00
|
|
|
rc = smc_wr_alloc_lgr_mem(lgr);
|
|
|
|
if (rc)
|
2021-10-16 09:37:49 +00:00
|
|
|
goto free_wq;
|
2020-04-29 15:10:49 +00:00
|
|
|
smc_llc_lgr_init(lgr, smc);
|
|
|
|
|
2020-04-29 15:10:39 +00:00
|
|
|
link_idx = SMC_SINGLE_LINK;
|
|
|
|
lnk = &lgr->lnk[link_idx];
|
|
|
|
rc = smcr_link_init(lgr, lnk, link_idx, ini);
|
2021-10-16 09:37:49 +00:00
|
|
|
if (rc) {
|
|
|
|
smc_wr_free_lgr_mem(lgr);
|
2020-09-10 16:48:29 +00:00
|
|
|
goto free_wq;
|
2021-10-16 09:37:49 +00:00
|
|
|
}
|
2021-12-28 13:06:09 +00:00
|
|
|
lgr->net = smc_ib_net(lnk->smcibdev);
|
2020-04-29 15:10:38 +00:00
|
|
|
lgr_list = &smc_lgr_list.list;
|
|
|
|
lgr_lock = &smc_lgr_list.lock;
|
2022-07-14 09:44:03 +00:00
|
|
|
lgr->buf_type = lgr->net->smc.sysctl_smcr_buf_type;
|
2019-11-16 16:47:29 +00:00
|
|
|
atomic_inc(&lgr_cnt);
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
}
|
2017-01-09 15:55:17 +00:00
|
|
|
smc->conn.lgr = lgr;
|
2019-10-09 08:07:44 +00:00
|
|
|
spin_lock_bh(lgr_lock);
|
2020-07-20 14:24:29 +00:00
|
|
|
list_add_tail(&lgr->list, lgr_list);
|
2019-10-09 08:07:44 +00:00
|
|
|
spin_unlock_bh(lgr_lock);
|
2017-01-09 15:55:19 +00:00
|
|
|
return 0;
|
|
|
|
|
2020-09-10 16:48:29 +00:00
|
|
|
free_wq:
|
|
|
|
destroy_workqueue(lgr->tx_wq);
|
2017-01-09 15:55:19 +00:00
|
|
|
free_lgr:
|
|
|
|
kfree(lgr);
|
2019-10-10 08:16:09 +00:00
|
|
|
ism_put_vlan:
|
|
|
|
if (ini->is_smcd && ini->vlan_id)
|
2020-09-26 10:44:29 +00:00
|
|
|
smc_ism_put_vlan(ini->ism_dev[ini->ism_selected], ini->vlan_id);
|
2017-01-09 15:55:17 +00:00
|
|
|
out:
|
2019-04-12 10:57:30 +00:00
|
|
|
if (rc < 0) {
|
|
|
|
if (rc == -ENOMEM)
|
|
|
|
rc = SMC_CLC_DECL_MEM;
|
|
|
|
else
|
|
|
|
rc = SMC_CLC_DECL_INTERR;
|
|
|
|
}
|
2017-01-09 15:55:17 +00:00
|
|
|
return rc;
|
|
|
|
}
|
|
|
|
|
2020-05-04 12:18:38 +00:00
|
|
|
static int smc_write_space(struct smc_connection *conn)
|
|
|
|
{
|
|
|
|
int buffer_len = conn->peer_rmbe_size;
|
|
|
|
union smc_host_cursor prod;
|
|
|
|
union smc_host_cursor cons;
|
|
|
|
int space;
|
|
|
|
|
|
|
|
smc_curs_copy(&prod, &conn->local_tx_ctrl.prod, conn);
|
|
|
|
smc_curs_copy(&cons, &conn->local_rx_ctrl.cons, conn);
|
|
|
|
/* determine rx_buf space */
|
|
|
|
space = buffer_len - smc_curs_diff(buffer_len, &cons, &prod);
|
|
|
|
return space;
|
|
|
|
}
|
|
|
|
|
2020-05-30 14:42:37 +00:00
|
|
|
static int smc_switch_cursor(struct smc_sock *smc, struct smc_cdc_tx_pend *pend,
|
|
|
|
struct smc_wr_buf *wr_buf)
|
2020-05-04 12:18:38 +00:00
|
|
|
{
|
|
|
|
struct smc_connection *conn = &smc->conn;
|
|
|
|
union smc_host_cursor cons, fin;
|
|
|
|
int rc = 0;
|
|
|
|
int diff;
|
|
|
|
|
|
|
|
smc_curs_copy(&conn->tx_curs_sent, &conn->tx_curs_fin, conn);
|
|
|
|
smc_curs_copy(&fin, &conn->local_tx_ctrl_fin, conn);
|
|
|
|
/* set prod cursor to old state, enforce tx_rdma_writes() */
|
|
|
|
smc_curs_copy(&conn->local_tx_ctrl.prod, &fin, conn);
|
|
|
|
smc_curs_copy(&cons, &conn->local_rx_ctrl.cons, conn);
|
|
|
|
|
|
|
|
if (smc_curs_comp(conn->peer_rmbe_size, &cons, &fin) < 0) {
|
|
|
|
/* cons cursor advanced more than fin, and prod was set
|
|
|
|
* fin above, so now prod is smaller than cons. Fix that.
|
|
|
|
*/
|
|
|
|
diff = smc_curs_diff(conn->peer_rmbe_size, &fin, &cons);
|
|
|
|
smc_curs_add(conn->sndbuf_desc->len,
|
|
|
|
&conn->tx_curs_sent, diff);
|
|
|
|
smc_curs_add(conn->sndbuf_desc->len,
|
|
|
|
&conn->tx_curs_fin, diff);
|
|
|
|
|
|
|
|
smp_mb__before_atomic();
|
|
|
|
atomic_add(diff, &conn->sndbuf_space);
|
|
|
|
smp_mb__after_atomic();
|
|
|
|
|
|
|
|
smc_curs_add(conn->peer_rmbe_size,
|
|
|
|
&conn->local_tx_ctrl.prod, diff);
|
|
|
|
smc_curs_add(conn->peer_rmbe_size,
|
|
|
|
&conn->local_tx_ctrl_fin, diff);
|
|
|
|
}
|
|
|
|
/* recalculate, value is used by tx_rdma_writes() */
|
|
|
|
atomic_set(&smc->conn.peer_rmbe_space, smc_write_space(conn));
|
|
|
|
|
|
|
|
if (smc->sk.sk_state != SMC_INIT &&
|
|
|
|
smc->sk.sk_state != SMC_CLOSED) {
|
2020-05-30 14:42:37 +00:00
|
|
|
rc = smcr_cdc_msg_send_validation(conn, pend, wr_buf);
|
2020-05-04 12:18:38 +00:00
|
|
|
if (!rc) {
|
2020-09-10 16:48:29 +00:00
|
|
|
queue_delayed_work(conn->lgr->tx_wq, &conn->tx_work, 0);
|
2020-05-04 12:18:38 +00:00
|
|
|
smc->sk.sk_data_ready(&smc->sk);
|
|
|
|
}
|
2020-05-30 14:42:37 +00:00
|
|
|
} else {
|
|
|
|
smc_wr_tx_put_slot(conn->lnk,
|
|
|
|
(struct smc_wr_tx_pend_priv *)pend);
|
2020-05-04 12:18:38 +00:00
|
|
|
}
|
|
|
|
return rc;
|
|
|
|
}
|
|
|
|
|
2021-08-09 09:05:57 +00:00
|
|
|
void smc_switch_link_and_count(struct smc_connection *conn,
|
|
|
|
struct smc_link *to_lnk)
|
2020-12-01 19:20:38 +00:00
|
|
|
{
|
|
|
|
atomic_dec(&conn->lnk->conn_cnt);
|
2022-01-13 08:36:42 +00:00
|
|
|
/* link_hold in smc_conn_create() */
|
|
|
|
smcr_link_put(conn->lnk);
|
2020-12-01 19:20:38 +00:00
|
|
|
conn->lnk = to_lnk;
|
|
|
|
atomic_inc(&conn->lnk->conn_cnt);
|
2022-01-13 08:36:42 +00:00
|
|
|
/* link_put in smc_conn_free() */
|
|
|
|
smcr_link_hold(conn->lnk);
|
2020-12-01 19:20:38 +00:00
|
|
|
}
|
|
|
|
|
2020-05-04 12:18:38 +00:00
|
|
|
struct smc_link *smc_switch_conns(struct smc_link_group *lgr,
|
|
|
|
struct smc_link *from_lnk, bool is_dev_err)
|
|
|
|
{
|
|
|
|
struct smc_link *to_lnk = NULL;
|
2020-05-30 14:42:37 +00:00
|
|
|
struct smc_cdc_tx_pend *pend;
|
2020-05-04 12:18:38 +00:00
|
|
|
struct smc_connection *conn;
|
2020-05-30 14:42:37 +00:00
|
|
|
struct smc_wr_buf *wr_buf;
|
2020-05-04 12:18:38 +00:00
|
|
|
struct smc_sock *smc;
|
|
|
|
struct rb_node *node;
|
|
|
|
int i, rc = 0;
|
|
|
|
|
|
|
|
/* link is inactive, wake up tx waiters */
|
|
|
|
smc_wr_wakeup_tx_wait(from_lnk);
|
|
|
|
|
|
|
|
for (i = 0; i < SMC_LINKS_PER_LGR_MAX; i++) {
|
2020-07-18 13:06:16 +00:00
|
|
|
if (!smc_link_active(&lgr->lnk[i]) || i == from_lnk->link_idx)
|
2020-05-04 12:18:38 +00:00
|
|
|
continue;
|
|
|
|
if (is_dev_err && from_lnk->smcibdev == lgr->lnk[i].smcibdev &&
|
|
|
|
from_lnk->ibport == lgr->lnk[i].ibport) {
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
to_lnk = &lgr->lnk[i];
|
|
|
|
break;
|
|
|
|
}
|
2021-10-07 14:14:40 +00:00
|
|
|
if (!to_lnk || !smc_wr_tx_link_hold(to_lnk)) {
|
2020-05-04 12:18:38 +00:00
|
|
|
smc_lgr_terminate_sched(lgr);
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
again:
|
|
|
|
read_lock_bh(&lgr->conns_lock);
|
|
|
|
for (node = rb_first(&lgr->conns_all); node; node = rb_next(node)) {
|
|
|
|
conn = rb_entry(node, struct smc_connection, alert_node);
|
|
|
|
if (conn->lnk != from_lnk)
|
|
|
|
continue;
|
|
|
|
smc = container_of(conn, struct smc_sock, conn);
|
|
|
|
/* conn->lnk not yet set in SMC_INIT state */
|
|
|
|
if (smc->sk.sk_state == SMC_INIT)
|
|
|
|
continue;
|
|
|
|
if (smc->sk.sk_state == SMC_CLOSED ||
|
|
|
|
smc->sk.sk_state == SMC_PEERCLOSEWAIT1 ||
|
|
|
|
smc->sk.sk_state == SMC_PEERCLOSEWAIT2 ||
|
|
|
|
smc->sk.sk_state == SMC_APPFINCLOSEWAIT ||
|
|
|
|
smc->sk.sk_state == SMC_APPCLOSEWAIT1 ||
|
|
|
|
smc->sk.sk_state == SMC_APPCLOSEWAIT2 ||
|
|
|
|
smc->sk.sk_state == SMC_PEERFINCLOSEWAIT ||
|
|
|
|
smc->sk.sk_state == SMC_PEERABORTWAIT ||
|
|
|
|
smc->sk.sk_state == SMC_PROCESSABORT) {
|
|
|
|
spin_lock_bh(&conn->send_lock);
|
2020-12-01 19:20:38 +00:00
|
|
|
smc_switch_link_and_count(conn, to_lnk);
|
2020-05-04 12:18:38 +00:00
|
|
|
spin_unlock_bh(&conn->send_lock);
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
sock_hold(&smc->sk);
|
|
|
|
read_unlock_bh(&lgr->conns_lock);
|
2020-05-30 14:42:37 +00:00
|
|
|
/* pre-fetch buffer outside of send_lock, might sleep */
|
|
|
|
rc = smc_cdc_get_free_slot(conn, to_lnk, &wr_buf, NULL, &pend);
|
2021-10-07 14:14:40 +00:00
|
|
|
if (rc)
|
|
|
|
goto err_out;
|
2020-05-04 12:18:38 +00:00
|
|
|
/* avoid race with smcr_tx_sndbuf_nonempty() */
|
|
|
|
spin_lock_bh(&conn->send_lock);
|
2020-12-01 19:20:38 +00:00
|
|
|
smc_switch_link_and_count(conn, to_lnk);
|
2020-05-30 14:42:37 +00:00
|
|
|
rc = smc_switch_cursor(smc, pend, wr_buf);
|
2020-05-04 12:18:38 +00:00
|
|
|
spin_unlock_bh(&conn->send_lock);
|
|
|
|
sock_put(&smc->sk);
|
2021-10-07 14:14:40 +00:00
|
|
|
if (rc)
|
|
|
|
goto err_out;
|
2020-05-04 12:18:38 +00:00
|
|
|
goto again;
|
|
|
|
}
|
|
|
|
read_unlock_bh(&lgr->conns_lock);
|
2021-10-07 14:14:40 +00:00
|
|
|
smc_wr_tx_link_put(to_lnk);
|
2020-05-04 12:18:38 +00:00
|
|
|
return to_lnk;
|
2021-10-07 14:14:40 +00:00
|
|
|
|
|
|
|
err_out:
|
|
|
|
smcr_link_down_cond_sched(to_lnk);
|
|
|
|
smc_wr_tx_link_put(to_lnk);
|
|
|
|
return NULL;
|
2020-05-04 12:18:38 +00:00
|
|
|
}
|
|
|
|
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
static void smcr_buf_unuse(struct smc_buf_desc *buf_desc, bool is_rmb,
|
2020-04-30 13:55:45 +00:00
|
|
|
struct smc_link_group *lgr)
|
2020-04-29 15:10:41 +00:00
|
|
|
{
|
2023-02-02 08:26:42 +00:00
|
|
|
struct rw_semaphore *lock; /* lock buffer list */
|
2020-05-01 10:48:05 +00:00
|
|
|
int rc;
|
|
|
|
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
if (is_rmb && buf_desc->is_conf_rkey && !list_empty(&lgr->list)) {
|
2020-04-29 15:10:41 +00:00
|
|
|
/* unregister rmb with peer */
|
2020-05-01 10:48:05 +00:00
|
|
|
rc = smc_llc_flow_initiate(lgr, SMC_LLC_FLOW_RKEY);
|
|
|
|
if (!rc) {
|
|
|
|
/* protect against smc_llc_cli_rkey_exchange() */
|
2023-02-02 08:26:40 +00:00
|
|
|
down_read(&lgr->llc_conf_mutex);
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
smc_llc_do_delete_rkey(lgr, buf_desc);
|
|
|
|
buf_desc->is_conf_rkey = false;
|
2023-02-02 08:26:40 +00:00
|
|
|
up_read(&lgr->llc_conf_mutex);
|
2020-05-01 10:48:05 +00:00
|
|
|
smc_llc_flow_stop(lgr, &lgr->llc_flow_lcl);
|
|
|
|
}
|
2020-04-29 15:10:41 +00:00
|
|
|
}
|
2020-05-01 10:48:05 +00:00
|
|
|
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
if (buf_desc->is_reg_err) {
|
2020-04-29 15:10:41 +00:00
|
|
|
/* buf registration failed, reuse not possible */
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
lock = is_rmb ? &lgr->rmbs_lock :
|
|
|
|
&lgr->sndbufs_lock;
|
2023-02-02 08:26:42 +00:00
|
|
|
down_write(lock);
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
list_del(&buf_desc->list);
|
2023-02-02 08:26:42 +00:00
|
|
|
up_write(lock);
|
2020-04-29 15:10:41 +00:00
|
|
|
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
smc_buf_free(lgr, is_rmb, buf_desc);
|
2020-04-29 15:10:41 +00:00
|
|
|
} else {
|
2023-02-16 06:39:05 +00:00
|
|
|
/* memzero_explicit provides potential memory barrier semantics */
|
|
|
|
memzero_explicit(buf_desc->cpu_addr, buf_desc->len);
|
|
|
|
WRITE_ONCE(buf_desc->used, 0);
|
2020-04-29 15:10:41 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-10-25 11:25:28 +00:00
|
|
|
static void smc_buf_unuse(struct smc_connection *conn,
|
|
|
|
struct smc_link_group *lgr)
|
2017-01-09 15:55:18 +00:00
|
|
|
{
|
2021-12-03 11:33:31 +00:00
|
|
|
if (conn->sndbuf_desc) {
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
if (!lgr->is_smcd && conn->sndbuf_desc->is_vm) {
|
|
|
|
smcr_buf_unuse(conn->sndbuf_desc, false, lgr);
|
|
|
|
} else {
|
2023-02-16 06:39:05 +00:00
|
|
|
memzero_explicit(conn->sndbuf_desc->cpu_addr, conn->sndbuf_desc->len);
|
|
|
|
WRITE_ONCE(conn->sndbuf_desc->used, 0);
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
}
|
2021-12-03 11:33:31 +00:00
|
|
|
}
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
if (conn->rmb_desc) {
|
|
|
|
if (!lgr->is_smcd) {
|
|
|
|
smcr_buf_unuse(conn->rmb_desc, true, lgr);
|
|
|
|
} else {
|
2023-02-16 06:39:05 +00:00
|
|
|
memzero_explicit(conn->rmb_desc->cpu_addr,
|
|
|
|
conn->rmb_desc->len + sizeof(struct smcd_cdc_msg));
|
|
|
|
WRITE_ONCE(conn->rmb_desc->used, 0);
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
}
|
2021-12-03 11:33:31 +00:00
|
|
|
}
|
2017-01-09 15:55:18 +00:00
|
|
|
}
|
|
|
|
|
2017-01-09 15:55:17 +00:00
|
|
|
/* remove a finished connection from its link group */
|
|
|
|
void smc_conn_free(struct smc_connection *conn)
|
|
|
|
{
|
2018-10-25 11:25:28 +00:00
|
|
|
struct smc_link_group *lgr = conn->lgr;
|
|
|
|
|
2022-01-13 08:36:40 +00:00
|
|
|
if (!lgr || conn->freed)
|
|
|
|
/* Connection has never been registered in a
|
|
|
|
* link group, or has already been freed.
|
|
|
|
*/
|
2017-01-09 15:55:17 +00:00
|
|
|
return;
|
2022-01-13 08:36:40 +00:00
|
|
|
|
|
|
|
conn->freed = 1;
|
2022-01-13 08:36:41 +00:00
|
|
|
if (!smc_conn_lgr_valid(conn))
|
2022-01-13 08:36:40 +00:00
|
|
|
/* Connection has already unregistered from
|
|
|
|
* link group.
|
|
|
|
*/
|
|
|
|
goto lgr_put;
|
|
|
|
|
2018-10-25 11:25:28 +00:00
|
|
|
if (lgr->is_smcd) {
|
2019-11-14 12:02:41 +00:00
|
|
|
if (!list_empty(&lgr->list))
|
|
|
|
smc_ism_unset_conn(conn);
|
2018-06-28 17:05:10 +00:00
|
|
|
tasklet_kill(&conn->rx_tsklet);
|
|
|
|
} else {
|
net/smc: fix kernel panic caused by race of smc_sock
A crash occurs when smc_cdc_tx_handler() tries to access smc_sock
but smc_release() has already freed it.
[ 4570.695099] BUG: unable to handle page fault for address: 000000002eae9e88
[ 4570.696048] #PF: supervisor write access in kernel mode
[ 4570.696728] #PF: error_code(0x0002) - not-present page
[ 4570.697401] PGD 0 P4D 0
[ 4570.697716] Oops: 0002 [#1] PREEMPT SMP NOPTI
[ 4570.698228] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.16.0-rc4+ #111
[ 4570.699013] Hardware name: Alibaba Cloud Alibaba Cloud ECS, BIOS 8c24b4c 04/0
[ 4570.699933] RIP: 0010:_raw_spin_lock+0x1a/0x30
<...>
[ 4570.711446] Call Trace:
[ 4570.711746] <IRQ>
[ 4570.711992] smc_cdc_tx_handler+0x41/0xc0
[ 4570.712470] smc_wr_tx_tasklet_fn+0x213/0x560
[ 4570.712981] ? smc_cdc_tx_dismisser+0x10/0x10
[ 4570.713489] tasklet_action_common.isra.17+0x66/0x140
[ 4570.714083] __do_softirq+0x123/0x2f4
[ 4570.714521] irq_exit_rcu+0xc4/0xf0
[ 4570.714934] common_interrupt+0xba/0xe0
Though smc_cdc_tx_handler() checked the existence of smc connection,
smc_release() may have already dismissed and released the smc socket
before smc_cdc_tx_handler() further visits it.
smc_cdc_tx_handler() |smc_release()
if (!conn) |
|
|smc_cdc_tx_dismiss_slots()
| smc_cdc_tx_dismisser()
|
|sock_put(&smc->sk) <- last sock_put,
| smc_sock freed
bh_lock_sock(&smc->sk) (panic) |
To make sure we won't receive any CDC messages after we free the
smc_sock, add a refcount on the smc_connection for inflight CDC
message(posted to the QP but haven't received related CQE), and
don't release the smc_connection until all the inflight CDC messages
haven been done, for both success or failed ones.
Using refcount on CDC messages brings another problem: when the link
is going to be destroyed, smcr_link_clear() will reset the QP, which
then remove all the pending CQEs related to the QP in the CQ. To make
sure all the CQEs will always come back so the refcount on the
smc_connection can always reach 0, smc_ib_modify_qp_reset() was replaced
by smc_ib_modify_qp_error().
And remove the timeout in smc_wr_tx_wait_no_pending_sends() since we
need to wait for all pending WQEs done, or we may encounter use-after-
free when handling CQEs.
For IB device removal routine, we need to wait for all the QPs on that
device been destroyed before we can destroy CQs on the device, or
the refcount on smc_connection won't reach 0 and smc_sock cannot be
released.
Fixes: 5f08318f617b ("smc: connection data control (CDC)")
Reported-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: Dust Li <dust.li@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2021-12-28 09:03:25 +00:00
|
|
|
smc_cdc_wait_pend_tx_wr(conn);
|
2020-05-04 12:18:40 +00:00
|
|
|
if (current_work() != &conn->abort_work)
|
|
|
|
cancel_work_sync(&conn->abort_work);
|
2018-06-28 17:05:10 +00:00
|
|
|
}
|
2019-10-21 14:13:13 +00:00
|
|
|
if (!list_empty(&lgr->list)) {
|
|
|
|
smc_buf_unuse(conn, lgr); /* allow buffer reuse */
|
2022-03-02 13:25:11 +00:00
|
|
|
smc_lgr_unregister_conn(conn);
|
2019-10-21 14:13:13 +00:00
|
|
|
}
|
2018-10-25 11:25:28 +00:00
|
|
|
|
|
|
|
if (!lgr->conns_num)
|
|
|
|
smc_lgr_schedule_free_work(lgr);
|
2022-01-13 08:36:40 +00:00
|
|
|
lgr_put:
|
2022-01-13 08:36:42 +00:00
|
|
|
if (!lgr->is_smcd)
|
|
|
|
smcr_link_put(conn->lnk); /* link_hold in smc_conn_create() */
|
2022-01-13 08:36:40 +00:00
|
|
|
smc_lgr_put(lgr); /* lgr_hold in smc_conn_create() */
|
2017-01-09 15:55:17 +00:00
|
|
|
}
|
|
|
|
|
2020-05-01 10:48:02 +00:00
|
|
|
/* unregister a link from a buf_desc */
|
|
|
|
static void smcr_buf_unmap_link(struct smc_buf_desc *buf_desc, bool is_rmb,
|
|
|
|
struct smc_link *lnk)
|
|
|
|
{
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
if (is_rmb || buf_desc->is_vm)
|
2020-05-01 10:48:02 +00:00
|
|
|
buf_desc->is_reg_mr[lnk->link_idx] = false;
|
|
|
|
if (!buf_desc->is_map_ib[lnk->link_idx])
|
|
|
|
return;
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
|
|
|
|
if ((is_rmb || buf_desc->is_vm) &&
|
|
|
|
buf_desc->mr[lnk->link_idx]) {
|
|
|
|
smc_ib_put_memory_region(buf_desc->mr[lnk->link_idx]);
|
|
|
|
buf_desc->mr[lnk->link_idx] = NULL;
|
|
|
|
}
|
|
|
|
if (is_rmb)
|
2020-05-01 10:48:02 +00:00
|
|
|
smc_ib_buf_unmap_sg(lnk, buf_desc, DMA_FROM_DEVICE);
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
else
|
2020-05-01 10:48:02 +00:00
|
|
|
smc_ib_buf_unmap_sg(lnk, buf_desc, DMA_TO_DEVICE);
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
|
2020-05-01 10:48:02 +00:00
|
|
|
sg_free_table(&buf_desc->sgt[lnk->link_idx]);
|
|
|
|
buf_desc->is_map_ib[lnk->link_idx] = false;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* unmap all buffers of lgr for a deleted link */
|
|
|
|
static void smcr_buf_unmap_lgr(struct smc_link *lnk)
|
|
|
|
{
|
|
|
|
struct smc_link_group *lgr = lnk->lgr;
|
|
|
|
struct smc_buf_desc *buf_desc, *bf;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < SMC_RMBE_SIZES; i++) {
|
2023-02-02 08:26:42 +00:00
|
|
|
down_write(&lgr->rmbs_lock);
|
2020-05-01 10:48:02 +00:00
|
|
|
list_for_each_entry_safe(buf_desc, bf, &lgr->rmbs[i], list)
|
|
|
|
smcr_buf_unmap_link(buf_desc, true, lnk);
|
2023-02-02 08:26:42 +00:00
|
|
|
up_write(&lgr->rmbs_lock);
|
|
|
|
|
|
|
|
down_write(&lgr->sndbufs_lock);
|
2020-05-01 10:48:02 +00:00
|
|
|
list_for_each_entry_safe(buf_desc, bf, &lgr->sndbufs[i],
|
|
|
|
list)
|
|
|
|
smcr_buf_unmap_link(buf_desc, false, lnk);
|
2023-02-02 08:26:42 +00:00
|
|
|
up_write(&lgr->sndbufs_lock);
|
2020-05-01 10:48:02 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static void smcr_rtoken_clear_link(struct smc_link *lnk)
|
|
|
|
{
|
|
|
|
struct smc_link_group *lgr = lnk->lgr;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < SMC_RMBS_PER_LGR_MAX; i++) {
|
|
|
|
lgr->rtokens[i][lnk->link_idx].rkey = 0;
|
|
|
|
lgr->rtokens[i][lnk->link_idx].dma_addr = 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2022-01-13 08:36:42 +00:00
|
|
|
static void __smcr_link_clear(struct smc_link *lnk)
|
2017-01-09 15:55:17 +00:00
|
|
|
{
|
2022-01-13 08:36:40 +00:00
|
|
|
struct smc_link_group *lgr = lnk->lgr;
|
2020-04-29 15:10:43 +00:00
|
|
|
struct smc_ib_device *smcibdev;
|
|
|
|
|
2017-01-09 15:55:19 +00:00
|
|
|
smc_wr_free_link_mem(lnk);
|
2020-12-01 19:20:39 +00:00
|
|
|
smc_ibdev_cnt_dec(lnk);
|
2020-04-29 15:10:38 +00:00
|
|
|
put_device(&lnk->smcibdev->ibdev->dev);
|
2020-04-29 15:10:43 +00:00
|
|
|
smcibdev = lnk->smcibdev;
|
|
|
|
memset(lnk, 0, sizeof(struct smc_link));
|
|
|
|
lnk->state = SMC_LNK_UNUSED;
|
|
|
|
if (!atomic_dec_return(&smcibdev->lnk_cnt))
|
|
|
|
wake_up(&smcibdev->lnks_deleted);
|
2022-01-13 08:36:40 +00:00
|
|
|
smc_lgr_put(lgr); /* lgr_hold in smcr_link_init() */
|
2017-01-09 15:55:17 +00:00
|
|
|
}
|
|
|
|
|
2022-01-13 08:36:42 +00:00
|
|
|
/* must be called under lgr->llc_conf_mutex lock */
|
|
|
|
void smcr_link_clear(struct smc_link *lnk, bool log)
|
|
|
|
{
|
|
|
|
if (!lnk->lgr || lnk->clearing ||
|
|
|
|
lnk->state == SMC_LNK_UNUSED)
|
|
|
|
return;
|
|
|
|
lnk->clearing = 1;
|
|
|
|
lnk->peer_qpn = 0;
|
|
|
|
smc_llc_link_clear(lnk, log);
|
|
|
|
smcr_buf_unmap_lgr(lnk);
|
|
|
|
smcr_rtoken_clear_link(lnk);
|
|
|
|
smc_ib_modify_qp_error(lnk);
|
|
|
|
smc_wr_free_link(lnk);
|
|
|
|
smc_ib_destroy_queue_pair(lnk);
|
|
|
|
smc_ib_dealloc_protection_domain(lnk);
|
|
|
|
smcr_link_put(lnk); /* theoretically last link_put */
|
|
|
|
}
|
|
|
|
|
|
|
|
void smcr_link_hold(struct smc_link *lnk)
|
|
|
|
{
|
|
|
|
refcount_inc(&lnk->refcnt);
|
|
|
|
}
|
|
|
|
|
|
|
|
void smcr_link_put(struct smc_link *lnk)
|
|
|
|
{
|
|
|
|
if (refcount_dec_and_test(&lnk->refcnt))
|
|
|
|
__smcr_link_clear(lnk);
|
|
|
|
}
|
|
|
|
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
static void smcr_buf_free(struct smc_link_group *lgr, bool is_rmb,
|
|
|
|
struct smc_buf_desc *buf_desc)
|
2017-01-09 15:55:18 +00:00
|
|
|
{
|
2020-04-29 15:10:41 +00:00
|
|
|
int i;
|
2018-05-18 07:34:17 +00:00
|
|
|
|
2020-05-01 10:48:02 +00:00
|
|
|
for (i = 0; i < SMC_LINKS_PER_LGR_MAX; i++)
|
|
|
|
smcr_buf_unmap_link(buf_desc, is_rmb, &lgr->lnk[i]);
|
2020-04-29 15:10:40 +00:00
|
|
|
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
if (!buf_desc->is_vm && buf_desc->pages)
|
2018-05-03 16:12:38 +00:00
|
|
|
__free_pages(buf_desc->pages, buf_desc->order);
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
else if (buf_desc->is_vm && buf_desc->cpu_addr)
|
|
|
|
vfree(buf_desc->cpu_addr);
|
2017-07-28 11:56:20 +00:00
|
|
|
kfree(buf_desc);
|
2017-01-09 15:55:18 +00:00
|
|
|
}
|
|
|
|
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
static void smcd_buf_free(struct smc_link_group *lgr, bool is_dmb,
|
|
|
|
struct smc_buf_desc *buf_desc)
|
|
|
|
{
|
2018-06-28 17:05:10 +00:00
|
|
|
if (is_dmb) {
|
|
|
|
/* restore original buf len */
|
|
|
|
buf_desc->len += sizeof(struct smcd_cdc_msg);
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
smc_ism_unregister_dmb(lgr->smcd, buf_desc);
|
2018-06-28 17:05:10 +00:00
|
|
|
} else {
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
kfree(buf_desc->cpu_addr);
|
2018-06-28 17:05:10 +00:00
|
|
|
}
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
kfree(buf_desc);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void smc_buf_free(struct smc_link_group *lgr, bool is_rmb,
|
|
|
|
struct smc_buf_desc *buf_desc)
|
|
|
|
{
|
|
|
|
if (lgr->is_smcd)
|
|
|
|
smcd_buf_free(lgr, is_rmb, buf_desc);
|
|
|
|
else
|
|
|
|
smcr_buf_free(lgr, is_rmb, buf_desc);
|
|
|
|
}
|
|
|
|
|
2017-07-28 11:56:20 +00:00
|
|
|
static void __smc_lgr_free_bufs(struct smc_link_group *lgr, bool is_rmb)
|
2017-01-09 15:55:18 +00:00
|
|
|
{
|
2017-07-28 11:56:20 +00:00
|
|
|
struct smc_buf_desc *buf_desc, *bf_desc;
|
|
|
|
struct list_head *buf_list;
|
2017-01-09 15:55:18 +00:00
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < SMC_RMBE_SIZES; i++) {
|
2017-07-28 11:56:20 +00:00
|
|
|
if (is_rmb)
|
|
|
|
buf_list = &lgr->rmbs[i];
|
|
|
|
else
|
|
|
|
buf_list = &lgr->sndbufs[i];
|
|
|
|
list_for_each_entry_safe(buf_desc, bf_desc, buf_list,
|
2017-01-09 15:55:18 +00:00
|
|
|
list) {
|
2017-07-28 11:56:20 +00:00
|
|
|
list_del(&buf_desc->list);
|
2018-05-18 07:34:17 +00:00
|
|
|
smc_buf_free(lgr, is_rmb, buf_desc);
|
2017-01-09 15:55:18 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-07-28 11:56:20 +00:00
|
|
|
static void smc_lgr_free_bufs(struct smc_link_group *lgr)
|
|
|
|
{
|
|
|
|
/* free send buffers */
|
|
|
|
__smc_lgr_free_bufs(lgr, false);
|
|
|
|
/* free rmbs */
|
|
|
|
__smc_lgr_free_bufs(lgr, true);
|
|
|
|
}
|
|
|
|
|
2022-01-13 08:36:40 +00:00
|
|
|
/* won't be freed until no one accesses to lgr anymore */
|
|
|
|
static void __smc_lgr_free(struct smc_link_group *lgr)
|
|
|
|
{
|
|
|
|
smc_lgr_free_bufs(lgr);
|
|
|
|
if (lgr->is_smcd) {
|
|
|
|
if (!atomic_dec_return(&lgr->smcd->lgr_cnt))
|
|
|
|
wake_up(&lgr->smcd->lgrs_deleted);
|
|
|
|
} else {
|
|
|
|
smc_wr_free_lgr_mem(lgr);
|
|
|
|
if (!atomic_dec_return(&lgr_cnt))
|
|
|
|
wake_up(&lgrs_deleted);
|
|
|
|
}
|
|
|
|
kfree(lgr);
|
|
|
|
}
|
|
|
|
|
2017-01-09 15:55:17 +00:00
|
|
|
/* remove a link group */
|
2018-11-22 09:26:35 +00:00
|
|
|
static void smc_lgr_free(struct smc_link_group *lgr)
|
2017-01-09 15:55:17 +00:00
|
|
|
{
|
2020-04-29 15:10:41 +00:00
|
|
|
int i;
|
|
|
|
|
2020-05-04 12:18:46 +00:00
|
|
|
if (!lgr->is_smcd) {
|
net/smc: llc_conf_mutex refactor, replace it with rw_semaphore
llc_conf_mutex was used to protect links and link related configurations
in the same link group, for example, add or delete links. However,
in most cases, the protected critical area has only read semantics and
with no write semantics at all, such as obtaining a usable link or an
available rmb_desc.
This patch do simply code refactoring, replace mutex with rw_semaphore,
replace mutex_lock with down_write and replace mutex_unlock with
up_write.
Theoretically, this replacement is equivalent, but after this patch,
we can distinguish lock granularity according to different semantics
of critical areas.
Signed-off-by: D. Wythe <alibuda@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2023-02-02 08:26:39 +00:00
|
|
|
down_write(&lgr->llc_conf_mutex);
|
2020-05-04 12:18:46 +00:00
|
|
|
for (i = 0; i < SMC_LINKS_PER_LGR_MAX; i++) {
|
|
|
|
if (lgr->lnk[i].state != SMC_LNK_UNUSED)
|
2020-05-05 13:01:20 +00:00
|
|
|
smcr_link_clear(&lgr->lnk[i], false);
|
2020-05-04 12:18:46 +00:00
|
|
|
}
|
net/smc: llc_conf_mutex refactor, replace it with rw_semaphore
llc_conf_mutex was used to protect links and link related configurations
in the same link group, for example, add or delete links. However,
in most cases, the protected critical area has only read semantics and
with no write semantics at all, such as obtaining a usable link or an
available rmb_desc.
This patch do simply code refactoring, replace mutex with rw_semaphore,
replace mutex_lock with down_write and replace mutex_unlock with
up_write.
Theoretically, this replacement is equivalent, but after this patch,
we can distinguish lock granularity according to different semantics
of critical areas.
Signed-off-by: D. Wythe <alibuda@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2023-02-02 08:26:39 +00:00
|
|
|
up_write(&lgr->llc_conf_mutex);
|
2020-05-04 12:18:46 +00:00
|
|
|
smc_llc_lgr_clear(lgr);
|
|
|
|
}
|
|
|
|
|
2020-09-10 16:48:29 +00:00
|
|
|
destroy_workqueue(lgr->tx_wq);
|
2019-10-09 08:07:45 +00:00
|
|
|
if (lgr->is_smcd) {
|
2020-09-10 16:48:26 +00:00
|
|
|
smc_ism_put_vlan(lgr->smcd, lgr->vlan_id);
|
2023-01-23 18:17:52 +00:00
|
|
|
put_device(lgr->smcd->ops->get_dev(lgr->smcd));
|
2019-10-09 08:07:45 +00:00
|
|
|
}
|
2022-01-13 08:36:40 +00:00
|
|
|
smc_lgr_put(lgr); /* theoretically last lgr_put */
|
|
|
|
}
|
|
|
|
|
|
|
|
void smc_lgr_hold(struct smc_link_group *lgr)
|
|
|
|
{
|
|
|
|
refcount_inc(&lgr->refcnt);
|
|
|
|
}
|
|
|
|
|
|
|
|
void smc_lgr_put(struct smc_link_group *lgr)
|
|
|
|
{
|
|
|
|
if (refcount_dec_and_test(&lgr->refcnt))
|
|
|
|
__smc_lgr_free(lgr);
|
2017-01-09 15:55:17 +00:00
|
|
|
}
|
|
|
|
|
2019-10-21 14:13:13 +00:00
|
|
|
static void smc_sk_wake_ups(struct smc_sock *smc)
|
|
|
|
{
|
|
|
|
smc->sk.sk_write_space(&smc->sk);
|
|
|
|
smc->sk.sk_data_ready(&smc->sk);
|
|
|
|
smc->sk.sk_state_change(&smc->sk);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* kill a connection */
|
2019-11-14 12:02:42 +00:00
|
|
|
static void smc_conn_kill(struct smc_connection *conn, bool soft)
|
2019-10-21 14:13:13 +00:00
|
|
|
{
|
|
|
|
struct smc_sock *smc = container_of(conn, struct smc_sock, conn);
|
|
|
|
|
2019-11-14 12:02:40 +00:00
|
|
|
if (conn->lgr->is_smcd && conn->lgr->peer_shutdown)
|
|
|
|
conn->local_tx_ctrl.conn_state_flags.peer_conn_abort = 1;
|
|
|
|
else
|
|
|
|
smc_close_abort(conn);
|
2019-10-21 14:13:13 +00:00
|
|
|
conn->killed = 1;
|
2019-11-14 12:02:40 +00:00
|
|
|
smc->sk.sk_err = ECONNABORTED;
|
2019-10-21 14:13:13 +00:00
|
|
|
smc_sk_wake_ups(smc);
|
2019-11-14 12:02:41 +00:00
|
|
|
if (conn->lgr->is_smcd) {
|
|
|
|
smc_ism_unset_conn(conn);
|
2019-11-14 12:02:42 +00:00
|
|
|
if (soft)
|
|
|
|
tasklet_kill(&conn->rx_tsklet);
|
|
|
|
else
|
|
|
|
tasklet_unlock_wait(&conn->rx_tsklet);
|
2019-11-14 12:02:46 +00:00
|
|
|
} else {
|
net/smc: fix kernel panic caused by race of smc_sock
A crash occurs when smc_cdc_tx_handler() tries to access smc_sock
but smc_release() has already freed it.
[ 4570.695099] BUG: unable to handle page fault for address: 000000002eae9e88
[ 4570.696048] #PF: supervisor write access in kernel mode
[ 4570.696728] #PF: error_code(0x0002) - not-present page
[ 4570.697401] PGD 0 P4D 0
[ 4570.697716] Oops: 0002 [#1] PREEMPT SMP NOPTI
[ 4570.698228] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.16.0-rc4+ #111
[ 4570.699013] Hardware name: Alibaba Cloud Alibaba Cloud ECS, BIOS 8c24b4c 04/0
[ 4570.699933] RIP: 0010:_raw_spin_lock+0x1a/0x30
<...>
[ 4570.711446] Call Trace:
[ 4570.711746] <IRQ>
[ 4570.711992] smc_cdc_tx_handler+0x41/0xc0
[ 4570.712470] smc_wr_tx_tasklet_fn+0x213/0x560
[ 4570.712981] ? smc_cdc_tx_dismisser+0x10/0x10
[ 4570.713489] tasklet_action_common.isra.17+0x66/0x140
[ 4570.714083] __do_softirq+0x123/0x2f4
[ 4570.714521] irq_exit_rcu+0xc4/0xf0
[ 4570.714934] common_interrupt+0xba/0xe0
Though smc_cdc_tx_handler() checked the existence of smc connection,
smc_release() may have already dismissed and released the smc socket
before smc_cdc_tx_handler() further visits it.
smc_cdc_tx_handler() |smc_release()
if (!conn) |
|
|smc_cdc_tx_dismiss_slots()
| smc_cdc_tx_dismisser()
|
|sock_put(&smc->sk) <- last sock_put,
| smc_sock freed
bh_lock_sock(&smc->sk) (panic) |
To make sure we won't receive any CDC messages after we free the
smc_sock, add a refcount on the smc_connection for inflight CDC
message(posted to the QP but haven't received related CQE), and
don't release the smc_connection until all the inflight CDC messages
haven been done, for both success or failed ones.
Using refcount on CDC messages brings another problem: when the link
is going to be destroyed, smcr_link_clear() will reset the QP, which
then remove all the pending CQEs related to the QP in the CQ. To make
sure all the CQEs will always come back so the refcount on the
smc_connection can always reach 0, smc_ib_modify_qp_reset() was replaced
by smc_ib_modify_qp_error().
And remove the timeout in smc_wr_tx_wait_no_pending_sends() since we
need to wait for all pending WQEs done, or we may encounter use-after-
free when handling CQEs.
For IB device removal routine, we need to wait for all the QPs on that
device been destroyed before we can destroy CQs on the device, or
the refcount on smc_connection won't reach 0 and smc_sock cannot be
released.
Fixes: 5f08318f617b ("smc: connection data control (CDC)")
Reported-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: Dust Li <dust.li@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2021-12-28 09:03:25 +00:00
|
|
|
smc_cdc_wait_pend_tx_wr(conn);
|
2019-11-14 12:02:41 +00:00
|
|
|
}
|
2019-10-21 14:13:13 +00:00
|
|
|
smc_lgr_unregister_conn(conn);
|
2019-10-21 14:13:15 +00:00
|
|
|
smc_close_active_abort(smc);
|
2019-10-21 14:13:13 +00:00
|
|
|
}
|
|
|
|
|
2019-11-14 12:02:41 +00:00
|
|
|
static void smc_lgr_cleanup(struct smc_link_group *lgr)
|
|
|
|
{
|
|
|
|
if (lgr->is_smcd) {
|
|
|
|
smc_ism_signal_shutdown(lgr);
|
|
|
|
} else {
|
2020-05-04 12:18:45 +00:00
|
|
|
u32 rsn = lgr->llc_termination_rsn;
|
|
|
|
|
|
|
|
if (!rsn)
|
|
|
|
rsn = SMC_LLC_DEL_PROG_INIT_TERM;
|
|
|
|
smc_llc_send_link_delete_all(lgr, false, rsn);
|
2020-05-04 12:18:46 +00:00
|
|
|
smcr_lgr_link_deactivate_all(lgr);
|
2019-11-14 12:02:41 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2020-02-17 15:24:53 +00:00
|
|
|
/* terminate link group
|
|
|
|
* @soft: true if link group shutdown can take its time
|
|
|
|
* false if immediate link group shutdown is required
|
|
|
|
*/
|
2019-11-14 12:02:42 +00:00
|
|
|
static void __smc_lgr_terminate(struct smc_link_group *lgr, bool soft)
|
2017-01-09 15:55:17 +00:00
|
|
|
{
|
|
|
|
struct smc_connection *conn;
|
2017-01-09 15:55:25 +00:00
|
|
|
struct smc_sock *smc;
|
2017-01-09 15:55:17 +00:00
|
|
|
struct rb_node *node;
|
|
|
|
|
2018-05-15 15:05:03 +00:00
|
|
|
if (lgr->terminating)
|
|
|
|
return; /* lgr already terminating */
|
2020-05-04 12:18:46 +00:00
|
|
|
/* cancel free_work sync, will terminate when lgr->freeing is set */
|
2023-03-13 10:08:28 +00:00
|
|
|
cancel_delayed_work(&lgr->free_work);
|
2018-05-15 15:05:03 +00:00
|
|
|
lgr->terminating = 1;
|
2017-01-09 15:55:17 +00:00
|
|
|
|
2019-10-21 14:13:10 +00:00
|
|
|
/* kill remaining link group connections */
|
|
|
|
read_lock_bh(&lgr->conns_lock);
|
2017-01-09 15:55:17 +00:00
|
|
|
node = rb_first(&lgr->conns_all);
|
|
|
|
while (node) {
|
2019-10-21 14:13:10 +00:00
|
|
|
read_unlock_bh(&lgr->conns_lock);
|
2017-01-09 15:55:17 +00:00
|
|
|
conn = rb_entry(node, struct smc_connection, alert_node);
|
2017-01-09 15:55:25 +00:00
|
|
|
smc = container_of(conn, struct smc_sock, conn);
|
2019-10-21 14:13:15 +00:00
|
|
|
sock_hold(&smc->sk); /* sock_put below */
|
2019-10-21 14:13:10 +00:00
|
|
|
lock_sock(&smc->sk);
|
2019-11-14 12:02:42 +00:00
|
|
|
smc_conn_kill(conn, soft);
|
2019-10-21 14:13:10 +00:00
|
|
|
release_sock(&smc->sk);
|
2019-10-21 14:13:15 +00:00
|
|
|
sock_put(&smc->sk); /* sock_hold above */
|
2019-10-21 14:13:10 +00:00
|
|
|
read_lock_bh(&lgr->conns_lock);
|
2017-01-09 15:55:17 +00:00
|
|
|
node = rb_first(&lgr->conns_all);
|
|
|
|
}
|
2019-10-21 14:13:10 +00:00
|
|
|
read_unlock_bh(&lgr->conns_lock);
|
2019-11-14 12:02:41 +00:00
|
|
|
smc_lgr_cleanup(lgr);
|
2020-05-04 12:18:46 +00:00
|
|
|
smc_lgr_free(lgr);
|
2017-01-09 15:55:17 +00:00
|
|
|
}
|
|
|
|
|
2020-02-17 15:24:54 +00:00
|
|
|
/* unlink link group and schedule termination */
|
|
|
|
void smc_lgr_terminate_sched(struct smc_link_group *lgr)
|
2018-05-23 14:38:10 +00:00
|
|
|
{
|
2019-10-09 08:07:44 +00:00
|
|
|
spinlock_t *lgr_lock;
|
|
|
|
|
|
|
|
smc_lgr_list_head(lgr, &lgr_lock);
|
|
|
|
spin_lock_bh(lgr_lock);
|
2020-02-17 15:24:52 +00:00
|
|
|
if (list_empty(&lgr->list) || lgr->terminating || lgr->freeing) {
|
2019-10-21 14:13:09 +00:00
|
|
|
spin_unlock_bh(lgr_lock);
|
|
|
|
return; /* lgr already terminating */
|
|
|
|
}
|
|
|
|
list_del_init(&lgr->list);
|
2020-05-04 12:18:46 +00:00
|
|
|
lgr->freeing = 1;
|
2019-10-09 08:07:44 +00:00
|
|
|
spin_unlock_bh(lgr_lock);
|
2020-02-17 15:24:54 +00:00
|
|
|
schedule_work(&lgr->terminate_work);
|
2018-05-23 14:38:10 +00:00
|
|
|
}
|
|
|
|
|
2019-11-14 12:02:42 +00:00
|
|
|
/* Called when peer lgr shutdown (regularly or abnormally) is received */
|
2018-11-20 15:46:41 +00:00
|
|
|
void smc_smcd_terminate(struct smcd_dev *dev, u64 peer_gid, unsigned short vlan)
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
{
|
|
|
|
struct smc_link_group *lgr, *l;
|
|
|
|
LIST_HEAD(lgr_free_list);
|
|
|
|
|
|
|
|
/* run common cleanup function and build free list */
|
2019-10-09 08:07:44 +00:00
|
|
|
spin_lock_bh(&dev->lgr_lock);
|
2019-10-09 08:07:43 +00:00
|
|
|
list_for_each_entry_safe(lgr, l, &dev->lgr_list, list) {
|
|
|
|
if ((!peer_gid || lgr->peer_gid == peer_gid) &&
|
2018-11-20 15:46:41 +00:00
|
|
|
(vlan == VLAN_VID_MASK || lgr->vlan_id == vlan)) {
|
2019-11-14 12:02:40 +00:00
|
|
|
if (peer_gid) /* peer triggered termination */
|
|
|
|
lgr->peer_shutdown = 1;
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
list_move(&lgr->list, &lgr_free_list);
|
2020-05-04 12:18:46 +00:00
|
|
|
lgr->freeing = 1;
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
}
|
|
|
|
}
|
2019-10-09 08:07:44 +00:00
|
|
|
spin_unlock_bh(&dev->lgr_lock);
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
|
|
|
|
/* cancel the regular free workers and actually free lgrs */
|
|
|
|
list_for_each_entry_safe(lgr, l, &lgr_free_list, list) {
|
|
|
|
list_del_init(&lgr->list);
|
2019-11-14 12:02:40 +00:00
|
|
|
schedule_work(&lgr->terminate_work);
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2019-11-14 12:02:42 +00:00
|
|
|
/* Called when an SMCD device is removed or the smc module is unloaded */
|
|
|
|
void smc_smcd_terminate_all(struct smcd_dev *smcd)
|
|
|
|
{
|
|
|
|
struct smc_link_group *lgr, *lg;
|
|
|
|
LIST_HEAD(lgr_free_list);
|
|
|
|
|
|
|
|
spin_lock_bh(&smcd->lgr_lock);
|
|
|
|
list_splice_init(&smcd->lgr_list, &lgr_free_list);
|
|
|
|
list_for_each_entry(lgr, &lgr_free_list, list)
|
|
|
|
lgr->freeing = 1;
|
|
|
|
spin_unlock_bh(&smcd->lgr_lock);
|
|
|
|
|
|
|
|
list_for_each_entry_safe(lgr, lg, &lgr_free_list, list) {
|
|
|
|
list_del_init(&lgr->list);
|
|
|
|
__smc_lgr_terminate(lgr, false);
|
|
|
|
}
|
2019-11-14 12:02:43 +00:00
|
|
|
|
|
|
|
if (atomic_read(&smcd->lgr_cnt))
|
|
|
|
wait_event(smcd->lgrs_deleted, !atomic_read(&smcd->lgr_cnt));
|
2019-11-14 12:02:42 +00:00
|
|
|
}
|
|
|
|
|
2019-11-14 12:02:47 +00:00
|
|
|
/* Called when an SMCR device is removed or the smc module is unloaded.
|
|
|
|
* If smcibdev is given, all SMCR link groups using this device are terminated.
|
|
|
|
* If smcibdev is NULL, all SMCR link groups are terminated.
|
|
|
|
*/
|
|
|
|
void smc_smcr_terminate_all(struct smc_ib_device *smcibdev)
|
|
|
|
{
|
|
|
|
struct smc_link_group *lgr, *lg;
|
|
|
|
LIST_HEAD(lgr_free_list);
|
2020-04-29 15:10:41 +00:00
|
|
|
int i;
|
2019-11-14 12:02:47 +00:00
|
|
|
|
|
|
|
spin_lock_bh(&smc_lgr_list.lock);
|
|
|
|
if (!smcibdev) {
|
|
|
|
list_splice_init(&smc_lgr_list.list, &lgr_free_list);
|
|
|
|
list_for_each_entry(lgr, &lgr_free_list, list)
|
|
|
|
lgr->freeing = 1;
|
|
|
|
} else {
|
|
|
|
list_for_each_entry_safe(lgr, lg, &smc_lgr_list.list, list) {
|
2020-04-29 15:10:41 +00:00
|
|
|
for (i = 0; i < SMC_LINKS_PER_LGR_MAX; i++) {
|
2020-05-01 10:48:09 +00:00
|
|
|
if (lgr->lnk[i].smcibdev == smcibdev)
|
2022-01-16 07:43:42 +00:00
|
|
|
smcr_link_down_cond_sched(&lgr->lnk[i]);
|
2019-11-14 12:02:47 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
spin_unlock_bh(&smc_lgr_list.lock);
|
|
|
|
|
|
|
|
list_for_each_entry_safe(lgr, lg, &lgr_free_list, list) {
|
|
|
|
list_del_init(&lgr->list);
|
2020-05-04 12:18:45 +00:00
|
|
|
smc_llc_set_termination_rsn(lgr, SMC_LLC_DEL_OP_INIT_TERM);
|
2019-11-14 12:02:47 +00:00
|
|
|
__smc_lgr_terminate(lgr, false);
|
net/smc: fix kernel panic caused by race of smc_sock
A crash occurs when smc_cdc_tx_handler() tries to access smc_sock
but smc_release() has already freed it.
[ 4570.695099] BUG: unable to handle page fault for address: 000000002eae9e88
[ 4570.696048] #PF: supervisor write access in kernel mode
[ 4570.696728] #PF: error_code(0x0002) - not-present page
[ 4570.697401] PGD 0 P4D 0
[ 4570.697716] Oops: 0002 [#1] PREEMPT SMP NOPTI
[ 4570.698228] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.16.0-rc4+ #111
[ 4570.699013] Hardware name: Alibaba Cloud Alibaba Cloud ECS, BIOS 8c24b4c 04/0
[ 4570.699933] RIP: 0010:_raw_spin_lock+0x1a/0x30
<...>
[ 4570.711446] Call Trace:
[ 4570.711746] <IRQ>
[ 4570.711992] smc_cdc_tx_handler+0x41/0xc0
[ 4570.712470] smc_wr_tx_tasklet_fn+0x213/0x560
[ 4570.712981] ? smc_cdc_tx_dismisser+0x10/0x10
[ 4570.713489] tasklet_action_common.isra.17+0x66/0x140
[ 4570.714083] __do_softirq+0x123/0x2f4
[ 4570.714521] irq_exit_rcu+0xc4/0xf0
[ 4570.714934] common_interrupt+0xba/0xe0
Though smc_cdc_tx_handler() checked the existence of smc connection,
smc_release() may have already dismissed and released the smc socket
before smc_cdc_tx_handler() further visits it.
smc_cdc_tx_handler() |smc_release()
if (!conn) |
|
|smc_cdc_tx_dismiss_slots()
| smc_cdc_tx_dismisser()
|
|sock_put(&smc->sk) <- last sock_put,
| smc_sock freed
bh_lock_sock(&smc->sk) (panic) |
To make sure we won't receive any CDC messages after we free the
smc_sock, add a refcount on the smc_connection for inflight CDC
message(posted to the QP but haven't received related CQE), and
don't release the smc_connection until all the inflight CDC messages
haven been done, for both success or failed ones.
Using refcount on CDC messages brings another problem: when the link
is going to be destroyed, smcr_link_clear() will reset the QP, which
then remove all the pending CQEs related to the QP in the CQ. To make
sure all the CQEs will always come back so the refcount on the
smc_connection can always reach 0, smc_ib_modify_qp_reset() was replaced
by smc_ib_modify_qp_error().
And remove the timeout in smc_wr_tx_wait_no_pending_sends() since we
need to wait for all pending WQEs done, or we may encounter use-after-
free when handling CQEs.
For IB device removal routine, we need to wait for all the QPs on that
device been destroyed before we can destroy CQs on the device, or
the refcount on smc_connection won't reach 0 and smc_sock cannot be
released.
Fixes: 5f08318f617b ("smc: connection data control (CDC)")
Reported-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: Dust Li <dust.li@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2021-12-28 09:03:25 +00:00
|
|
|
}
|
|
|
|
|
2019-11-16 16:47:29 +00:00
|
|
|
if (smcibdev) {
|
|
|
|
if (atomic_read(&smcibdev->lnk_cnt))
|
|
|
|
wait_event(smcibdev->lnks_deleted,
|
|
|
|
!atomic_read(&smcibdev->lnk_cnt));
|
|
|
|
} else {
|
|
|
|
if (atomic_read(&lgr_cnt))
|
|
|
|
wait_event(lgrs_deleted, !atomic_read(&lgr_cnt));
|
|
|
|
}
|
2019-11-14 12:02:47 +00:00
|
|
|
}
|
|
|
|
|
2020-05-04 12:18:44 +00:00
|
|
|
/* set new lgr type and clear all asymmetric link tagging */
|
|
|
|
void smcr_lgr_set_type(struct smc_link_group *lgr, enum smc_lgr_type new_type)
|
|
|
|
{
|
2020-05-05 13:01:20 +00:00
|
|
|
char *lgr_type = "";
|
2020-05-04 12:18:44 +00:00
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < SMC_LINKS_PER_LGR_MAX; i++)
|
|
|
|
if (smc_link_usable(&lgr->lnk[i]))
|
|
|
|
lgr->lnk[i].link_is_asym = false;
|
2020-05-05 13:01:20 +00:00
|
|
|
if (lgr->type == new_type)
|
|
|
|
return;
|
2020-05-04 12:18:44 +00:00
|
|
|
lgr->type = new_type;
|
2020-05-05 13:01:20 +00:00
|
|
|
|
|
|
|
switch (lgr->type) {
|
|
|
|
case SMC_LGR_NONE:
|
|
|
|
lgr_type = "NONE";
|
|
|
|
break;
|
|
|
|
case SMC_LGR_SINGLE:
|
|
|
|
lgr_type = "SINGLE";
|
|
|
|
break;
|
|
|
|
case SMC_LGR_SYMMETRIC:
|
|
|
|
lgr_type = "SYMMETRIC";
|
|
|
|
break;
|
|
|
|
case SMC_LGR_ASYMMETRIC_PEER:
|
|
|
|
lgr_type = "ASYMMETRIC_PEER";
|
|
|
|
break;
|
|
|
|
case SMC_LGR_ASYMMETRIC_LOCAL:
|
|
|
|
lgr_type = "ASYMMETRIC_LOCAL";
|
|
|
|
break;
|
|
|
|
}
|
2021-12-28 13:06:11 +00:00
|
|
|
pr_warn_ratelimited("smc: SMC-R lg %*phN net %llu state changed: "
|
2020-05-05 13:01:20 +00:00
|
|
|
"%s, pnetid %.16s\n", SMC_LGR_ID_SIZE, &lgr->id,
|
2021-12-28 13:06:11 +00:00
|
|
|
lgr->net->net_cookie, lgr_type, lgr->pnet_id);
|
2020-05-04 12:18:44 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* set new lgr type and tag a link as asymmetric */
|
|
|
|
void smcr_lgr_set_type_asym(struct smc_link_group *lgr,
|
|
|
|
enum smc_lgr_type new_type, int asym_lnk_idx)
|
|
|
|
{
|
|
|
|
smcr_lgr_set_type(lgr, new_type);
|
|
|
|
lgr->lnk[asym_lnk_idx].link_is_asym = true;
|
|
|
|
}
|
|
|
|
|
2020-05-04 12:18:40 +00:00
|
|
|
/* abort connection, abort_work scheduled from tasklet context */
|
|
|
|
static void smc_conn_abort_work(struct work_struct *work)
|
|
|
|
{
|
|
|
|
struct smc_connection *conn = container_of(work,
|
|
|
|
struct smc_connection,
|
|
|
|
abort_work);
|
|
|
|
struct smc_sock *smc = container_of(conn, struct smc_sock, conn);
|
|
|
|
|
2021-09-20 19:18:15 +00:00
|
|
|
lock_sock(&smc->sk);
|
2020-05-04 12:18:40 +00:00
|
|
|
smc_conn_kill(conn, true);
|
2021-09-20 19:18:15 +00:00
|
|
|
release_sock(&smc->sk);
|
2020-05-04 12:18:40 +00:00
|
|
|
sock_put(&smc->sk); /* sock_hold done by schedulers of abort_work */
|
|
|
|
}
|
|
|
|
|
2020-05-01 10:48:07 +00:00
|
|
|
void smcr_port_add(struct smc_ib_device *smcibdev, u8 ibport)
|
|
|
|
{
|
|
|
|
struct smc_link_group *lgr, *n;
|
|
|
|
|
|
|
|
list_for_each_entry_safe(lgr, n, &smc_lgr_list.list, list) {
|
2020-07-18 13:06:14 +00:00
|
|
|
struct smc_link *link;
|
|
|
|
|
2020-05-01 10:48:07 +00:00
|
|
|
if (strncmp(smcibdev->pnetid[ibport - 1], lgr->pnet_id,
|
|
|
|
SMC_MAX_PNETID_LEN) ||
|
|
|
|
lgr->type == SMC_LGR_SYMMETRIC ||
|
2021-12-28 13:06:09 +00:00
|
|
|
lgr->type == SMC_LGR_ASYMMETRIC_PEER ||
|
|
|
|
!rdma_dev_access_netns(smcibdev->ibdev, lgr->net))
|
2020-05-01 10:48:07 +00:00
|
|
|
continue;
|
2020-07-18 13:06:14 +00:00
|
|
|
|
|
|
|
/* trigger local add link processing */
|
|
|
|
link = smc_llc_usable_link(lgr);
|
|
|
|
if (link)
|
|
|
|
smc_llc_add_link_local(link);
|
2020-05-01 10:48:07 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2020-05-01 10:48:08 +00:00
|
|
|
/* link is down - switch connections to alternate link,
|
|
|
|
* must be called under lgr->llc_conf_mutex lock
|
|
|
|
*/
|
|
|
|
static void smcr_link_down(struct smc_link *lnk)
|
|
|
|
{
|
|
|
|
struct smc_link_group *lgr = lnk->lgr;
|
|
|
|
struct smc_link *to_lnk;
|
|
|
|
int del_link_id;
|
|
|
|
|
|
|
|
if (!lgr || lnk->state == SMC_LNK_UNUSED || list_empty(&lgr->list))
|
|
|
|
return;
|
|
|
|
|
2020-05-04 12:18:38 +00:00
|
|
|
to_lnk = smc_switch_conns(lgr, lnk, true);
|
2020-05-01 10:48:08 +00:00
|
|
|
if (!to_lnk) { /* no backup link available */
|
2020-05-05 13:01:20 +00:00
|
|
|
smcr_link_clear(lnk, true);
|
2020-05-01 10:48:08 +00:00
|
|
|
return;
|
|
|
|
}
|
2020-05-04 12:18:44 +00:00
|
|
|
smcr_lgr_set_type(lgr, SMC_LGR_SINGLE);
|
2020-05-01 10:48:08 +00:00
|
|
|
del_link_id = lnk->link_id;
|
|
|
|
|
|
|
|
if (lgr->role == SMC_SERV) {
|
|
|
|
/* trigger local delete link processing */
|
2020-05-03 12:38:50 +00:00
|
|
|
smc_llc_srv_delete_link_local(to_lnk, del_link_id);
|
2020-05-01 10:48:08 +00:00
|
|
|
} else {
|
|
|
|
if (lgr->llc_flow_lcl.type != SMC_LLC_FLOW_NONE) {
|
|
|
|
/* another llc task is ongoing */
|
net/smc: llc_conf_mutex refactor, replace it with rw_semaphore
llc_conf_mutex was used to protect links and link related configurations
in the same link group, for example, add or delete links. However,
in most cases, the protected critical area has only read semantics and
with no write semantics at all, such as obtaining a usable link or an
available rmb_desc.
This patch do simply code refactoring, replace mutex with rw_semaphore,
replace mutex_lock with down_write and replace mutex_unlock with
up_write.
Theoretically, this replacement is equivalent, but after this patch,
we can distinguish lock granularity according to different semantics
of critical areas.
Signed-off-by: D. Wythe <alibuda@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2023-02-02 08:26:39 +00:00
|
|
|
up_write(&lgr->llc_conf_mutex);
|
2020-07-08 15:05:11 +00:00
|
|
|
wait_event_timeout(lgr->llc_flow_waiter,
|
|
|
|
(list_empty(&lgr->list) ||
|
|
|
|
lgr->llc_flow_lcl.type == SMC_LLC_FLOW_NONE),
|
2020-05-01 10:48:08 +00:00
|
|
|
SMC_LLC_WAIT_TIME);
|
net/smc: llc_conf_mutex refactor, replace it with rw_semaphore
llc_conf_mutex was used to protect links and link related configurations
in the same link group, for example, add or delete links. However,
in most cases, the protected critical area has only read semantics and
with no write semantics at all, such as obtaining a usable link or an
available rmb_desc.
This patch do simply code refactoring, replace mutex with rw_semaphore,
replace mutex_lock with down_write and replace mutex_unlock with
up_write.
Theoretically, this replacement is equivalent, but after this patch,
we can distinguish lock granularity according to different semantics
of critical areas.
Signed-off-by: D. Wythe <alibuda@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2023-02-02 08:26:39 +00:00
|
|
|
down_write(&lgr->llc_conf_mutex);
|
2020-05-01 10:48:08 +00:00
|
|
|
}
|
2020-07-18 13:06:10 +00:00
|
|
|
if (!list_empty(&lgr->list)) {
|
2020-07-08 15:05:11 +00:00
|
|
|
smc_llc_send_delete_link(to_lnk, del_link_id,
|
|
|
|
SMC_LLC_REQ, true,
|
|
|
|
SMC_LLC_DEL_LOST_PATH);
|
2020-07-18 13:06:10 +00:00
|
|
|
smcr_link_clear(lnk, true);
|
|
|
|
}
|
2020-07-08 15:05:11 +00:00
|
|
|
wake_up(&lgr->llc_flow_waiter); /* wake up next waiter */
|
2020-05-01 10:48:08 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* must be called under lgr->llc_conf_mutex lock */
|
|
|
|
void smcr_link_down_cond(struct smc_link *lnk)
|
|
|
|
{
|
2021-11-01 07:39:16 +00:00
|
|
|
if (smc_link_downing(&lnk->state)) {
|
|
|
|
trace_smcr_link_down(lnk, __builtin_return_address(0));
|
2020-05-01 10:48:08 +00:00
|
|
|
smcr_link_down(lnk);
|
2021-11-01 07:39:16 +00:00
|
|
|
}
|
2020-05-01 10:48:08 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* will get the lgr->llc_conf_mutex lock */
|
|
|
|
void smcr_link_down_cond_sched(struct smc_link *lnk)
|
|
|
|
{
|
2021-11-01 07:39:16 +00:00
|
|
|
if (smc_link_downing(&lnk->state)) {
|
|
|
|
trace_smcr_link_down(lnk, __builtin_return_address(0));
|
2020-05-01 10:48:08 +00:00
|
|
|
schedule_work(&lnk->link_down_wrk);
|
2021-11-01 07:39:16 +00:00
|
|
|
}
|
2020-05-01 10:48:08 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
void smcr_port_err(struct smc_ib_device *smcibdev, u8 ibport)
|
|
|
|
{
|
|
|
|
struct smc_link_group *lgr, *n;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
list_for_each_entry_safe(lgr, n, &smc_lgr_list.list, list) {
|
|
|
|
if (strncmp(smcibdev->pnetid[ibport - 1], lgr->pnet_id,
|
|
|
|
SMC_MAX_PNETID_LEN))
|
|
|
|
continue; /* lgr is not affected */
|
|
|
|
if (list_empty(&lgr->list))
|
|
|
|
continue;
|
|
|
|
for (i = 0; i < SMC_LINKS_PER_LGR_MAX; i++) {
|
|
|
|
struct smc_link *lnk = &lgr->lnk[i];
|
|
|
|
|
|
|
|
if (smc_link_usable(lnk) &&
|
|
|
|
lnk->smcibdev == smcibdev && lnk->ibport == ibport)
|
|
|
|
smcr_link_down_cond_sched(lnk);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static void smc_link_down_work(struct work_struct *work)
|
|
|
|
{
|
|
|
|
struct smc_link *link = container_of(work, struct smc_link,
|
|
|
|
link_down_wrk);
|
|
|
|
struct smc_link_group *lgr = link->lgr;
|
|
|
|
|
|
|
|
if (list_empty(&lgr->list))
|
|
|
|
return;
|
2020-07-08 15:05:11 +00:00
|
|
|
wake_up_all(&lgr->llc_msg_waiter);
|
net/smc: llc_conf_mutex refactor, replace it with rw_semaphore
llc_conf_mutex was used to protect links and link related configurations
in the same link group, for example, add or delete links. However,
in most cases, the protected critical area has only read semantics and
with no write semantics at all, such as obtaining a usable link or an
available rmb_desc.
This patch do simply code refactoring, replace mutex with rw_semaphore,
replace mutex_lock with down_write and replace mutex_unlock with
up_write.
Theoretically, this replacement is equivalent, but after this patch,
we can distinguish lock granularity according to different semantics
of critical areas.
Signed-off-by: D. Wythe <alibuda@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2023-02-02 08:26:39 +00:00
|
|
|
down_write(&lgr->llc_conf_mutex);
|
2020-05-01 10:48:08 +00:00
|
|
|
smcr_link_down(link);
|
net/smc: llc_conf_mutex refactor, replace it with rw_semaphore
llc_conf_mutex was used to protect links and link related configurations
in the same link group, for example, add or delete links. However,
in most cases, the protected critical area has only read semantics and
with no write semantics at all, such as obtaining a usable link or an
available rmb_desc.
This patch do simply code refactoring, replace mutex with rw_semaphore,
replace mutex_lock with down_write and replace mutex_unlock with
up_write.
Theoretically, this replacement is equivalent, but after this patch,
we can distinguish lock granularity according to different semantics
of critical areas.
Signed-off-by: D. Wythe <alibuda@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2023-02-02 08:26:39 +00:00
|
|
|
up_write(&lgr->llc_conf_mutex);
|
2020-05-01 10:48:08 +00:00
|
|
|
}
|
|
|
|
|
2021-11-24 12:32:37 +00:00
|
|
|
static int smc_vlan_by_tcpsk_walk(struct net_device *lower_dev,
|
|
|
|
struct netdev_nested_priv *priv)
|
|
|
|
{
|
|
|
|
unsigned short *vlan_id = (unsigned short *)priv->data;
|
|
|
|
|
|
|
|
if (is_vlan_dev(lower_dev)) {
|
|
|
|
*vlan_id = vlan_dev_vlan_id(lower_dev);
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Determine vlan of internal TCP socket. */
|
2019-04-12 10:57:26 +00:00
|
|
|
int smc_vlan_by_tcpsk(struct socket *clcsock, struct smc_init_info *ini)
|
2017-01-09 15:55:17 +00:00
|
|
|
{
|
|
|
|
struct dst_entry *dst = sk_dst_get(clcsock->sk);
|
2021-11-24 12:32:37 +00:00
|
|
|
struct netdev_nested_priv priv;
|
2018-05-02 14:56:47 +00:00
|
|
|
struct net_device *ndev;
|
2021-11-24 12:32:37 +00:00
|
|
|
int rc = 0;
|
2017-01-09 15:55:17 +00:00
|
|
|
|
2019-04-12 10:57:26 +00:00
|
|
|
ini->vlan_id = 0;
|
2017-01-09 15:55:17 +00:00
|
|
|
if (!dst) {
|
|
|
|
rc = -ENOTCONN;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
if (!dst->dev) {
|
|
|
|
rc = -ENODEV;
|
|
|
|
goto out_rel;
|
|
|
|
}
|
|
|
|
|
2018-05-02 14:56:47 +00:00
|
|
|
ndev = dst->dev;
|
|
|
|
if (is_vlan_dev(ndev)) {
|
2019-04-12 10:57:26 +00:00
|
|
|
ini->vlan_id = vlan_dev_vlan_id(ndev);
|
2018-05-02 14:56:47 +00:00
|
|
|
goto out_rel;
|
|
|
|
}
|
|
|
|
|
2021-11-24 12:32:37 +00:00
|
|
|
priv.data = (void *)&ini->vlan_id;
|
2018-05-02 14:56:47 +00:00
|
|
|
rtnl_lock();
|
2021-11-24 12:32:37 +00:00
|
|
|
netdev_walk_all_lower_dev(ndev, smc_vlan_by_tcpsk_walk, &priv);
|
2018-05-02 14:56:47 +00:00
|
|
|
rtnl_unlock();
|
2017-01-09 15:55:17 +00:00
|
|
|
|
|
|
|
out_rel:
|
|
|
|
dst_release(dst);
|
|
|
|
out:
|
|
|
|
return rc;
|
|
|
|
}
|
|
|
|
|
2021-10-16 09:37:46 +00:00
|
|
|
static bool smcr_lgr_match(struct smc_link_group *lgr, u8 smcr_version,
|
|
|
|
u8 peer_systemid[],
|
|
|
|
u8 peer_gid[],
|
|
|
|
u8 peer_mac_v1[],
|
2021-12-28 13:06:09 +00:00
|
|
|
enum smc_lgr_role role, u32 clcqpn,
|
|
|
|
struct net *net)
|
2017-01-09 15:55:17 +00:00
|
|
|
{
|
2021-12-28 13:06:09 +00:00
|
|
|
struct smc_link *lnk;
|
2020-04-29 15:10:41 +00:00
|
|
|
int i;
|
|
|
|
|
2021-10-16 09:37:46 +00:00
|
|
|
if (memcmp(lgr->peer_systemid, peer_systemid, SMC_SYSTEMID_LEN) ||
|
2020-04-29 15:10:41 +00:00
|
|
|
lgr->role != role)
|
|
|
|
return false;
|
|
|
|
|
|
|
|
for (i = 0; i < SMC_LINKS_PER_LGR_MAX; i++) {
|
2021-12-28 13:06:09 +00:00
|
|
|
lnk = &lgr->lnk[i];
|
|
|
|
|
|
|
|
if (!smc_link_active(lnk))
|
2020-04-29 15:10:41 +00:00
|
|
|
continue;
|
2021-12-28 13:06:09 +00:00
|
|
|
/* use verbs API to check netns, instead of lgr->net */
|
|
|
|
if (!rdma_dev_access_netns(lnk->smcibdev->ibdev, net))
|
|
|
|
return false;
|
|
|
|
if ((lgr->role == SMC_SERV || lnk->peer_qpn == clcqpn) &&
|
|
|
|
!memcmp(lnk->peer_gid, peer_gid, SMC_GID_SIZE) &&
|
2021-10-16 09:37:46 +00:00
|
|
|
(smcr_version == SMC_V2 ||
|
2021-12-28 13:06:09 +00:00
|
|
|
!memcmp(lnk->peer_mac, peer_mac_v1, ETH_ALEN)))
|
2020-04-29 15:10:41 +00:00
|
|
|
return true;
|
|
|
|
}
|
|
|
|
return false;
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
}
|
2017-01-09 15:55:17 +00:00
|
|
|
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
static bool smcd_lgr_match(struct smc_link_group *lgr,
|
|
|
|
struct smcd_dev *smcismdev, u64 peer_gid)
|
|
|
|
{
|
|
|
|
return lgr->peer_gid == peer_gid && lgr->smcd == smcismdev;
|
2017-01-09 15:55:17 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* create a new SMC connection (and a new link group if necessary) */
|
2019-04-12 10:57:26 +00:00
|
|
|
int smc_conn_create(struct smc_sock *smc, struct smc_init_info *ini)
|
2017-01-09 15:55:17 +00:00
|
|
|
{
|
|
|
|
struct smc_connection *conn = &smc->conn;
|
2021-12-28 13:06:09 +00:00
|
|
|
struct net *net = sock_net(&smc->sk);
|
2019-10-09 08:07:43 +00:00
|
|
|
struct list_head *lgr_list;
|
2017-01-09 15:55:17 +00:00
|
|
|
struct smc_link_group *lgr;
|
|
|
|
enum smc_lgr_role role;
|
2019-10-09 08:07:44 +00:00
|
|
|
spinlock_t *lgr_lock;
|
2017-01-09 15:55:17 +00:00
|
|
|
int rc = 0;
|
|
|
|
|
2020-09-26 10:44:29 +00:00
|
|
|
lgr_list = ini->is_smcd ? &ini->ism_dev[ini->ism_selected]->lgr_list :
|
2020-09-26 10:44:23 +00:00
|
|
|
&smc_lgr_list.list;
|
2020-09-26 10:44:29 +00:00
|
|
|
lgr_lock = ini->is_smcd ? &ini->ism_dev[ini->ism_selected]->lgr_lock :
|
2020-09-26 10:44:23 +00:00
|
|
|
&smc_lgr_list.lock;
|
2020-09-10 16:48:21 +00:00
|
|
|
ini->first_contact_local = 1;
|
2017-01-09 15:55:17 +00:00
|
|
|
role = smc->listen_smc ? SMC_SERV : SMC_CLNT;
|
2020-09-10 16:48:21 +00:00
|
|
|
if (role == SMC_CLNT && ini->first_contact_peer)
|
2017-01-09 15:55:17 +00:00
|
|
|
/* create new link group as well */
|
|
|
|
goto create;
|
|
|
|
|
|
|
|
/* determine if an existing link group can be reused */
|
2019-10-09 08:07:44 +00:00
|
|
|
spin_lock_bh(lgr_lock);
|
2019-10-09 08:07:43 +00:00
|
|
|
list_for_each_entry(lgr, lgr_list, list) {
|
2017-01-09 15:55:17 +00:00
|
|
|
write_lock_bh(&lgr->conns_lock);
|
2019-04-12 10:57:26 +00:00
|
|
|
if ((ini->is_smcd ?
|
2020-09-26 10:44:29 +00:00
|
|
|
smcd_lgr_match(lgr, ini->ism_dev[ini->ism_selected],
|
|
|
|
ini->ism_peer_gid[ini->ism_selected]) :
|
2021-10-16 09:37:46 +00:00
|
|
|
smcr_lgr_match(lgr, ini->smcr_version,
|
|
|
|
ini->peer_systemid,
|
|
|
|
ini->peer_gid, ini->peer_mac, role,
|
2021-12-28 13:06:09 +00:00
|
|
|
ini->ib_clcqpn, net)) &&
|
2017-01-09 15:55:17 +00:00
|
|
|
!lgr->sync_err &&
|
2020-11-18 21:40:37 +00:00
|
|
|
(ini->smcd_version == SMC_V2 ||
|
|
|
|
lgr->vlan_id == ini->vlan_id) &&
|
2020-07-20 14:24:29 +00:00
|
|
|
(role == SMC_CLNT || ini->is_smcd ||
|
2022-03-02 13:25:12 +00:00
|
|
|
(lgr->conns_num < SMC_RMBS_PER_LGR_MAX &&
|
|
|
|
!bitmap_full(lgr->rtokens_used_mask, SMC_RMBS_PER_LGR_MAX)))) {
|
2017-01-09 15:55:17 +00:00
|
|
|
/* link group found */
|
2020-09-10 16:48:21 +00:00
|
|
|
ini->first_contact_local = 0;
|
2017-01-09 15:55:17 +00:00
|
|
|
conn->lgr = lgr;
|
2020-05-04 12:18:43 +00:00
|
|
|
rc = smc_lgr_register_conn(conn, false);
|
2017-01-09 15:55:17 +00:00
|
|
|
write_unlock_bh(&lgr->conns_lock);
|
2020-04-29 15:10:41 +00:00
|
|
|
if (!rc && delayed_work_pending(&lgr->free_work))
|
|
|
|
cancel_delayed_work(&lgr->free_work);
|
2017-01-09 15:55:17 +00:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
write_unlock_bh(&lgr->conns_lock);
|
|
|
|
}
|
2019-10-09 08:07:44 +00:00
|
|
|
spin_unlock_bh(lgr_lock);
|
2020-04-29 15:10:41 +00:00
|
|
|
if (rc)
|
|
|
|
return rc;
|
2017-01-09 15:55:17 +00:00
|
|
|
|
2020-09-10 16:48:21 +00:00
|
|
|
if (role == SMC_CLNT && !ini->first_contact_peer &&
|
|
|
|
ini->first_contact_local) {
|
2017-01-09 15:55:17 +00:00
|
|
|
/* Server reuses a link group, but Client wants to start
|
|
|
|
* a new one
|
|
|
|
* send out_of_sync decline, reason synchr. error
|
|
|
|
*/
|
2019-04-12 10:57:30 +00:00
|
|
|
return SMC_CLC_DECL_SYNCERR;
|
2017-01-09 15:55:17 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
create:
|
2020-09-10 16:48:21 +00:00
|
|
|
if (ini->first_contact_local) {
|
2019-04-12 10:57:26 +00:00
|
|
|
rc = smc_lgr_create(smc, ini);
|
2017-01-09 15:55:17 +00:00
|
|
|
if (rc)
|
|
|
|
goto out;
|
2019-06-26 15:47:49 +00:00
|
|
|
lgr = conn->lgr;
|
|
|
|
write_lock_bh(&lgr->conns_lock);
|
2020-05-04 12:18:43 +00:00
|
|
|
rc = smc_lgr_register_conn(conn, true);
|
2019-06-26 15:47:49 +00:00
|
|
|
write_unlock_bh(&lgr->conns_lock);
|
2022-01-06 12:42:08 +00:00
|
|
|
if (rc) {
|
|
|
|
smc_lgr_cleanup_early(lgr);
|
2020-04-29 15:10:41 +00:00
|
|
|
goto out;
|
2022-01-06 12:42:08 +00:00
|
|
|
}
|
2017-01-09 15:55:17 +00:00
|
|
|
}
|
2022-01-13 08:36:40 +00:00
|
|
|
smc_lgr_hold(conn->lgr); /* lgr_put in smc_conn_free() */
|
2022-01-13 08:36:42 +00:00
|
|
|
if (!conn->lgr->is_smcd)
|
|
|
|
smcr_link_hold(conn->lnk); /* link_put in smc_conn_free() */
|
2022-01-13 08:36:40 +00:00
|
|
|
conn->freed = 0;
|
2017-01-09 15:55:22 +00:00
|
|
|
conn->local_tx_ctrl.common.type = SMC_CDC_MSG_TYPE;
|
2018-02-28 11:44:07 +00:00
|
|
|
conn->local_tx_ctrl.len = SMC_WR_TX_SIZE;
|
2018-05-23 14:38:11 +00:00
|
|
|
conn->urg_state = SMC_URG_READ;
|
net/smc: fix kernel panic caused by race of smc_sock
A crash occurs when smc_cdc_tx_handler() tries to access smc_sock
but smc_release() has already freed it.
[ 4570.695099] BUG: unable to handle page fault for address: 000000002eae9e88
[ 4570.696048] #PF: supervisor write access in kernel mode
[ 4570.696728] #PF: error_code(0x0002) - not-present page
[ 4570.697401] PGD 0 P4D 0
[ 4570.697716] Oops: 0002 [#1] PREEMPT SMP NOPTI
[ 4570.698228] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.16.0-rc4+ #111
[ 4570.699013] Hardware name: Alibaba Cloud Alibaba Cloud ECS, BIOS 8c24b4c 04/0
[ 4570.699933] RIP: 0010:_raw_spin_lock+0x1a/0x30
<...>
[ 4570.711446] Call Trace:
[ 4570.711746] <IRQ>
[ 4570.711992] smc_cdc_tx_handler+0x41/0xc0
[ 4570.712470] smc_wr_tx_tasklet_fn+0x213/0x560
[ 4570.712981] ? smc_cdc_tx_dismisser+0x10/0x10
[ 4570.713489] tasklet_action_common.isra.17+0x66/0x140
[ 4570.714083] __do_softirq+0x123/0x2f4
[ 4570.714521] irq_exit_rcu+0xc4/0xf0
[ 4570.714934] common_interrupt+0xba/0xe0
Though smc_cdc_tx_handler() checked the existence of smc connection,
smc_release() may have already dismissed and released the smc socket
before smc_cdc_tx_handler() further visits it.
smc_cdc_tx_handler() |smc_release()
if (!conn) |
|
|smc_cdc_tx_dismiss_slots()
| smc_cdc_tx_dismisser()
|
|sock_put(&smc->sk) <- last sock_put,
| smc_sock freed
bh_lock_sock(&smc->sk) (panic) |
To make sure we won't receive any CDC messages after we free the
smc_sock, add a refcount on the smc_connection for inflight CDC
message(posted to the QP but haven't received related CQE), and
don't release the smc_connection until all the inflight CDC messages
haven been done, for both success or failed ones.
Using refcount on CDC messages brings another problem: when the link
is going to be destroyed, smcr_link_clear() will reset the QP, which
then remove all the pending CQEs related to the QP in the CQ. To make
sure all the CQEs will always come back so the refcount on the
smc_connection can always reach 0, smc_ib_modify_qp_reset() was replaced
by smc_ib_modify_qp_error().
And remove the timeout in smc_wr_tx_wait_no_pending_sends() since we
need to wait for all pending WQEs done, or we may encounter use-after-
free when handling CQEs.
For IB device removal routine, we need to wait for all the QPs on that
device been destroyed before we can destroy CQs on the device, or
the refcount on smc_connection won't reach 0 and smc_sock cannot be
released.
Fixes: 5f08318f617b ("smc: connection data control (CDC)")
Reported-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: Dust Li <dust.li@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2021-12-28 09:03:25 +00:00
|
|
|
init_waitqueue_head(&conn->cdc_pend_tx_wq);
|
2020-05-04 12:18:40 +00:00
|
|
|
INIT_WORK(&smc->conn.abort_work, smc_conn_abort_work);
|
2019-04-12 10:57:26 +00:00
|
|
|
if (ini->is_smcd) {
|
2018-06-28 17:05:10 +00:00
|
|
|
conn->rx_off = sizeof(struct smcd_cdc_msg);
|
|
|
|
smcd_cdc_rx_init(conn); /* init tasklet for this conn */
|
2020-09-03 19:53:16 +00:00
|
|
|
} else {
|
|
|
|
conn->rx_off = 0;
|
2018-06-28 17:05:10 +00:00
|
|
|
}
|
2017-01-09 15:55:22 +00:00
|
|
|
#ifndef KERNEL_HAS_ATOMIC64
|
|
|
|
spin_lock_init(&conn->acurs_lock);
|
|
|
|
#endif
|
2017-01-09 15:55:17 +00:00
|
|
|
|
|
|
|
out:
|
2019-04-12 10:57:30 +00:00
|
|
|
return rc;
|
2017-01-09 15:55:17 +00:00
|
|
|
}
|
2017-01-09 15:55:18 +00:00
|
|
|
|
2021-08-09 08:10:14 +00:00
|
|
|
#define SMCD_DMBE_SIZES 6 /* 0 -> 16KB, 1 -> 32KB, .. 6 -> 1MB */
|
|
|
|
#define SMCR_RMBE_SIZES 5 /* 0 -> 16KB, 1 -> 32KB, .. 5 -> 512KB */
|
|
|
|
|
|
|
|
/* convert the RMB size into the compressed notation (minimum 16K, see
|
|
|
|
* SMCD/R_DMBE_SIZES.
|
2018-05-18 07:34:14 +00:00
|
|
|
* In contrast to plain ilog2, this rounds towards the next power of 2,
|
|
|
|
* so the socket application gets at least its desired sndbuf / rcvbuf size.
|
|
|
|
*/
|
2021-08-09 08:10:14 +00:00
|
|
|
static u8 smc_compress_bufsize(int size, bool is_smcd, bool is_rmb)
|
2018-05-18 07:34:14 +00:00
|
|
|
{
|
2021-08-09 08:10:14 +00:00
|
|
|
const unsigned int max_scat = SG_MAX_SINGLE_ALLOC * PAGE_SIZE;
|
2018-05-18 07:34:14 +00:00
|
|
|
u8 compressed;
|
|
|
|
|
|
|
|
if (size <= SMC_BUF_MIN_SIZE)
|
|
|
|
return 0;
|
|
|
|
|
2021-08-09 08:10:14 +00:00
|
|
|
size = (size - 1) >> 14; /* convert to 16K multiple */
|
|
|
|
compressed = min_t(u8, ilog2(size) + 1,
|
|
|
|
is_smcd ? SMCD_DMBE_SIZES : SMCR_RMBE_SIZES);
|
|
|
|
|
|
|
|
if (!is_smcd && is_rmb)
|
|
|
|
/* RMBs are backed by & limited to max size of scatterlists */
|
|
|
|
compressed = min_t(u8, compressed, ilog2(max_scat >> 14));
|
|
|
|
|
2018-05-18 07:34:14 +00:00
|
|
|
return compressed;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* convert the RMB size from compressed notation into integer */
|
|
|
|
int smc_uncompress_bufsize(u8 compressed)
|
|
|
|
{
|
|
|
|
u32 size;
|
|
|
|
|
|
|
|
size = 0x00000001 << (((int)compressed) + 14);
|
|
|
|
return (int)size;
|
|
|
|
}
|
|
|
|
|
2017-07-28 11:56:20 +00:00
|
|
|
/* try to reuse a sndbuf or rmb description slot for a certain
|
|
|
|
* buffer size; if not available, return NULL
|
2017-01-09 15:55:18 +00:00
|
|
|
*/
|
2018-05-18 07:34:16 +00:00
|
|
|
static struct smc_buf_desc *smc_buf_get_slot(int compressed_bufsize,
|
2023-02-02 08:26:42 +00:00
|
|
|
struct rw_semaphore *lock,
|
2018-05-18 07:34:16 +00:00
|
|
|
struct list_head *buf_list)
|
2017-01-09 15:55:18 +00:00
|
|
|
{
|
2017-07-28 11:56:20 +00:00
|
|
|
struct smc_buf_desc *buf_slot;
|
2017-01-09 15:55:18 +00:00
|
|
|
|
2023-02-02 08:26:42 +00:00
|
|
|
down_read(lock);
|
2017-07-28 11:56:20 +00:00
|
|
|
list_for_each_entry(buf_slot, buf_list, list) {
|
|
|
|
if (cmpxchg(&buf_slot->used, 0, 1) == 0) {
|
2023-02-02 08:26:42 +00:00
|
|
|
up_read(lock);
|
2017-07-28 11:56:20 +00:00
|
|
|
return buf_slot;
|
2017-01-09 15:55:18 +00:00
|
|
|
}
|
|
|
|
}
|
2023-02-02 08:26:42 +00:00
|
|
|
up_read(lock);
|
2017-01-09 15:55:18 +00:00
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
2017-01-09 15:55:24 +00:00
|
|
|
/* one of the conditions for announcing a receiver's current window size is
|
|
|
|
* that it "results in a minimum increase in the window size of 10% of the
|
|
|
|
* receive buffer space" [RFC7609]
|
|
|
|
*/
|
|
|
|
static inline int smc_rmb_wnd_update_limit(int rmbe_size)
|
|
|
|
{
|
2022-03-01 09:44:00 +00:00
|
|
|
return max_t(int, rmbe_size / 10, SOCK_MIN_SNDBUF / 2);
|
2017-01-09 15:55:24 +00:00
|
|
|
}
|
|
|
|
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
/* map an buf to a link */
|
2020-04-29 15:10:41 +00:00
|
|
|
static int smcr_buf_map_link(struct smc_buf_desc *buf_desc, bool is_rmb,
|
|
|
|
struct smc_link *lnk)
|
|
|
|
{
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
int rc, i, nents, offset, buf_size, size, access_flags;
|
|
|
|
struct scatterlist *sg;
|
|
|
|
void *buf;
|
2020-04-29 15:10:41 +00:00
|
|
|
|
|
|
|
if (buf_desc->is_map_ib[lnk->link_idx])
|
|
|
|
return 0;
|
|
|
|
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
if (buf_desc->is_vm) {
|
|
|
|
buf = buf_desc->cpu_addr;
|
|
|
|
buf_size = buf_desc->len;
|
|
|
|
offset = offset_in_page(buf_desc->cpu_addr);
|
|
|
|
nents = PAGE_ALIGN(buf_size + offset) / PAGE_SIZE;
|
|
|
|
} else {
|
|
|
|
nents = 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
rc = sg_alloc_table(&buf_desc->sgt[lnk->link_idx], nents, GFP_KERNEL);
|
2020-04-29 15:10:41 +00:00
|
|
|
if (rc)
|
|
|
|
return rc;
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
|
|
|
|
if (buf_desc->is_vm) {
|
|
|
|
/* virtually contiguous buffer */
|
|
|
|
for_each_sg(buf_desc->sgt[lnk->link_idx].sgl, sg, nents, i) {
|
|
|
|
size = min_t(int, PAGE_SIZE - offset, buf_size);
|
|
|
|
sg_set_page(sg, vmalloc_to_page(buf), size, offset);
|
|
|
|
buf += size / sizeof(*buf);
|
|
|
|
buf_size -= size;
|
|
|
|
offset = 0;
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
/* physically contiguous buffer */
|
|
|
|
sg_set_buf(buf_desc->sgt[lnk->link_idx].sgl,
|
|
|
|
buf_desc->cpu_addr, buf_desc->len);
|
|
|
|
}
|
2020-04-29 15:10:41 +00:00
|
|
|
|
|
|
|
/* map sg table to DMA address */
|
|
|
|
rc = smc_ib_buf_map_sg(lnk, buf_desc,
|
|
|
|
is_rmb ? DMA_FROM_DEVICE : DMA_TO_DEVICE);
|
|
|
|
/* SMC protocol depends on mapping to one DMA address only */
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
if (rc != nents) {
|
2020-04-29 15:10:41 +00:00
|
|
|
rc = -EAGAIN;
|
|
|
|
goto free_table;
|
|
|
|
}
|
|
|
|
|
2022-07-14 09:44:01 +00:00
|
|
|
buf_desc->is_dma_need_sync |=
|
|
|
|
smc_ib_is_sg_need_sync(lnk, buf_desc) << lnk->link_idx;
|
|
|
|
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
if (is_rmb || buf_desc->is_vm) {
|
|
|
|
/* create a new memory region for the RMB or vzalloced sndbuf */
|
|
|
|
access_flags = is_rmb ?
|
|
|
|
IB_ACCESS_REMOTE_WRITE | IB_ACCESS_LOCAL_WRITE :
|
|
|
|
IB_ACCESS_LOCAL_WRITE;
|
|
|
|
|
|
|
|
rc = smc_ib_get_memory_region(lnk->roce_pd, access_flags,
|
2020-04-29 15:10:41 +00:00
|
|
|
buf_desc, lnk->link_idx);
|
|
|
|
if (rc)
|
|
|
|
goto buf_unmap;
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
smc_ib_sync_sg_for_device(lnk, buf_desc,
|
|
|
|
is_rmb ? DMA_FROM_DEVICE : DMA_TO_DEVICE);
|
2020-04-29 15:10:41 +00:00
|
|
|
}
|
|
|
|
buf_desc->is_map_ib[lnk->link_idx] = true;
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
buf_unmap:
|
|
|
|
smc_ib_buf_unmap_sg(lnk, buf_desc,
|
|
|
|
is_rmb ? DMA_FROM_DEVICE : DMA_TO_DEVICE);
|
|
|
|
free_table:
|
|
|
|
sg_free_table(&buf_desc->sgt[lnk->link_idx]);
|
|
|
|
return rc;
|
|
|
|
}
|
|
|
|
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
/* register a new buf on IB device, rmb or vzalloced sndbuf
|
2020-05-01 10:48:05 +00:00
|
|
|
* must be called under lgr->llc_conf_mutex lock
|
|
|
|
*/
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
int smcr_link_reg_buf(struct smc_link *link, struct smc_buf_desc *buf_desc)
|
2020-05-01 10:48:01 +00:00
|
|
|
{
|
|
|
|
if (list_empty(&link->lgr->list))
|
|
|
|
return -ENOLINK;
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
if (!buf_desc->is_reg_mr[link->link_idx]) {
|
|
|
|
/* register memory region for new buf */
|
|
|
|
if (buf_desc->is_vm)
|
|
|
|
buf_desc->mr[link->link_idx]->iova =
|
|
|
|
(uintptr_t)buf_desc->cpu_addr;
|
|
|
|
if (smc_wr_reg_send(link, buf_desc->mr[link->link_idx])) {
|
|
|
|
buf_desc->is_reg_err = true;
|
2020-05-01 10:48:01 +00:00
|
|
|
return -EFAULT;
|
|
|
|
}
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
buf_desc->is_reg_mr[link->link_idx] = true;
|
2020-05-01 10:48:01 +00:00
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2023-02-02 08:26:42 +00:00
|
|
|
static int _smcr_buf_map_lgr(struct smc_link *lnk, struct rw_semaphore *lock,
|
2020-05-01 10:48:03 +00:00
|
|
|
struct list_head *lst, bool is_rmb)
|
|
|
|
{
|
|
|
|
struct smc_buf_desc *buf_desc, *bf;
|
|
|
|
int rc = 0;
|
|
|
|
|
2023-02-02 08:26:42 +00:00
|
|
|
down_write(lock);
|
2020-05-01 10:48:03 +00:00
|
|
|
list_for_each_entry_safe(buf_desc, bf, lst, list) {
|
|
|
|
if (!buf_desc->used)
|
|
|
|
continue;
|
|
|
|
rc = smcr_buf_map_link(buf_desc, is_rmb, lnk);
|
|
|
|
if (rc)
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
out:
|
2023-02-02 08:26:42 +00:00
|
|
|
up_write(lock);
|
2020-05-01 10:48:03 +00:00
|
|
|
return rc;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* map all used buffers of lgr for a new link */
|
|
|
|
int smcr_buf_map_lgr(struct smc_link *lnk)
|
|
|
|
{
|
|
|
|
struct smc_link_group *lgr = lnk->lgr;
|
|
|
|
int i, rc = 0;
|
|
|
|
|
|
|
|
for (i = 0; i < SMC_RMBE_SIZES; i++) {
|
|
|
|
rc = _smcr_buf_map_lgr(lnk, &lgr->rmbs_lock,
|
|
|
|
&lgr->rmbs[i], true);
|
|
|
|
if (rc)
|
|
|
|
return rc;
|
|
|
|
rc = _smcr_buf_map_lgr(lnk, &lgr->sndbufs_lock,
|
|
|
|
&lgr->sndbufs[i], false);
|
|
|
|
if (rc)
|
|
|
|
return rc;
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2020-05-01 10:48:05 +00:00
|
|
|
/* register all used buffers of lgr for a new link,
|
|
|
|
* must be called under lgr->llc_conf_mutex lock
|
|
|
|
*/
|
2020-05-01 10:48:03 +00:00
|
|
|
int smcr_buf_reg_lgr(struct smc_link *lnk)
|
|
|
|
{
|
|
|
|
struct smc_link_group *lgr = lnk->lgr;
|
|
|
|
struct smc_buf_desc *buf_desc, *bf;
|
|
|
|
int i, rc = 0;
|
|
|
|
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
/* reg all RMBs for a new link */
|
2023-02-02 08:26:42 +00:00
|
|
|
down_write(&lgr->rmbs_lock);
|
2020-05-01 10:48:03 +00:00
|
|
|
for (i = 0; i < SMC_RMBE_SIZES; i++) {
|
|
|
|
list_for_each_entry_safe(buf_desc, bf, &lgr->rmbs[i], list) {
|
|
|
|
if (!buf_desc->used)
|
|
|
|
continue;
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
rc = smcr_link_reg_buf(lnk, buf_desc);
|
|
|
|
if (rc) {
|
2023-02-02 08:26:42 +00:00
|
|
|
up_write(&lgr->rmbs_lock);
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
return rc;
|
|
|
|
}
|
2020-05-01 10:48:03 +00:00
|
|
|
}
|
|
|
|
}
|
2023-02-02 08:26:42 +00:00
|
|
|
up_write(&lgr->rmbs_lock);
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
|
|
|
|
if (lgr->buf_type == SMCR_PHYS_CONT_BUFS)
|
|
|
|
return rc;
|
|
|
|
|
|
|
|
/* reg all vzalloced sndbufs for a new link */
|
2023-02-02 08:26:42 +00:00
|
|
|
down_write(&lgr->sndbufs_lock);
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
for (i = 0; i < SMC_RMBE_SIZES; i++) {
|
|
|
|
list_for_each_entry_safe(buf_desc, bf, &lgr->sndbufs[i], list) {
|
|
|
|
if (!buf_desc->used || !buf_desc->is_vm)
|
|
|
|
continue;
|
|
|
|
rc = smcr_link_reg_buf(lnk, buf_desc);
|
|
|
|
if (rc) {
|
2023-02-02 08:26:42 +00:00
|
|
|
up_write(&lgr->sndbufs_lock);
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
return rc;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
2023-02-02 08:26:42 +00:00
|
|
|
up_write(&lgr->sndbufs_lock);
|
2020-05-01 10:48:03 +00:00
|
|
|
return rc;
|
|
|
|
}
|
|
|
|
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
static struct smc_buf_desc *smcr_new_buf_create(struct smc_link_group *lgr,
|
|
|
|
bool is_rmb, int bufsize)
|
2017-07-28 11:56:21 +00:00
|
|
|
{
|
|
|
|
struct smc_buf_desc *buf_desc;
|
|
|
|
|
|
|
|
/* try to alloc a new buffer */
|
|
|
|
buf_desc = kzalloc(sizeof(*buf_desc), GFP_KERNEL);
|
|
|
|
if (!buf_desc)
|
|
|
|
return ERR_PTR(-ENOMEM);
|
|
|
|
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
switch (lgr->buf_type) {
|
|
|
|
case SMCR_PHYS_CONT_BUFS:
|
|
|
|
case SMCR_MIXED_BUFS:
|
|
|
|
buf_desc->order = get_order(bufsize);
|
|
|
|
buf_desc->pages = alloc_pages(GFP_KERNEL | __GFP_NOWARN |
|
|
|
|
__GFP_NOMEMALLOC | __GFP_COMP |
|
|
|
|
__GFP_NORETRY | __GFP_ZERO,
|
|
|
|
buf_desc->order);
|
|
|
|
if (buf_desc->pages) {
|
|
|
|
buf_desc->cpu_addr =
|
|
|
|
(void *)page_address(buf_desc->pages);
|
|
|
|
buf_desc->len = bufsize;
|
|
|
|
buf_desc->is_vm = false;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
if (lgr->buf_type == SMCR_PHYS_CONT_BUFS)
|
|
|
|
goto out;
|
|
|
|
fallthrough; // try virtually continguous buf
|
|
|
|
case SMCR_VIRT_CONT_BUFS:
|
|
|
|
buf_desc->order = get_order(bufsize);
|
|
|
|
buf_desc->cpu_addr = vzalloc(PAGE_SIZE << buf_desc->order);
|
|
|
|
if (!buf_desc->cpu_addr)
|
|
|
|
goto out;
|
|
|
|
buf_desc->pages = NULL;
|
|
|
|
buf_desc->len = bufsize;
|
|
|
|
buf_desc->is_vm = true;
|
|
|
|
break;
|
|
|
|
}
|
2020-04-29 15:10:41 +00:00
|
|
|
return buf_desc;
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
|
|
|
|
out:
|
|
|
|
kfree(buf_desc);
|
|
|
|
return ERR_PTR(-EAGAIN);
|
2020-04-29 15:10:41 +00:00
|
|
|
}
|
2017-07-28 11:56:21 +00:00
|
|
|
|
2020-04-29 15:10:41 +00:00
|
|
|
/* map buf_desc on all usable links,
|
|
|
|
* unused buffers stay mapped as long as the link is up
|
|
|
|
*/
|
|
|
|
static int smcr_buf_map_usable_links(struct smc_link_group *lgr,
|
|
|
|
struct smc_buf_desc *buf_desc, bool is_rmb)
|
|
|
|
{
|
2022-09-20 06:43:09 +00:00
|
|
|
int i, rc = 0, cnt = 0;
|
2017-07-28 11:56:21 +00:00
|
|
|
|
2020-05-01 10:48:05 +00:00
|
|
|
/* protect against parallel link reconfiguration */
|
2023-02-02 08:26:40 +00:00
|
|
|
down_read(&lgr->llc_conf_mutex);
|
2020-04-29 15:10:41 +00:00
|
|
|
for (i = 0; i < SMC_LINKS_PER_LGR_MAX; i++) {
|
|
|
|
struct smc_link *lnk = &lgr->lnk[i];
|
2017-07-28 11:56:21 +00:00
|
|
|
|
2020-04-29 15:10:43 +00:00
|
|
|
if (!smc_link_usable(lnk))
|
2020-04-29 15:10:41 +00:00
|
|
|
continue;
|
|
|
|
if (smcr_buf_map_link(buf_desc, is_rmb, lnk)) {
|
|
|
|
rc = -ENOMEM;
|
|
|
|
goto out;
|
2017-07-28 11:56:21 +00:00
|
|
|
}
|
2022-09-20 06:43:09 +00:00
|
|
|
cnt++;
|
2017-07-28 11:56:21 +00:00
|
|
|
}
|
2020-04-29 15:10:41 +00:00
|
|
|
out:
|
2023-02-02 08:26:40 +00:00
|
|
|
up_read(&lgr->llc_conf_mutex);
|
2022-09-20 06:43:09 +00:00
|
|
|
if (!rc && !cnt)
|
|
|
|
rc = -EINVAL;
|
2020-04-29 15:10:41 +00:00
|
|
|
return rc;
|
2017-07-28 11:56:21 +00:00
|
|
|
}
|
|
|
|
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
static struct smc_buf_desc *smcd_new_buf_create(struct smc_link_group *lgr,
|
|
|
|
bool is_dmb, int bufsize)
|
|
|
|
{
|
|
|
|
struct smc_buf_desc *buf_desc;
|
|
|
|
int rc;
|
|
|
|
|
|
|
|
/* try to alloc a new DMB */
|
|
|
|
buf_desc = kzalloc(sizeof(*buf_desc), GFP_KERNEL);
|
|
|
|
if (!buf_desc)
|
|
|
|
return ERR_PTR(-ENOMEM);
|
|
|
|
if (is_dmb) {
|
|
|
|
rc = smc_ism_register_dmb(lgr, bufsize, buf_desc);
|
|
|
|
if (rc) {
|
|
|
|
kfree(buf_desc);
|
2020-10-23 18:48:29 +00:00
|
|
|
if (rc == -ENOMEM)
|
|
|
|
return ERR_PTR(-EAGAIN);
|
|
|
|
if (rc == -ENOSPC)
|
|
|
|
return ERR_PTR(-ENOSPC);
|
|
|
|
return ERR_PTR(-EIO);
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
}
|
2018-06-28 17:05:10 +00:00
|
|
|
buf_desc->pages = virt_to_page(buf_desc->cpu_addr);
|
|
|
|
/* CDC header stored in buf. So, pretend it was smaller */
|
|
|
|
buf_desc->len = bufsize - sizeof(struct smcd_cdc_msg);
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
} else {
|
|
|
|
buf_desc->cpu_addr = kzalloc(bufsize, GFP_KERNEL |
|
|
|
|
__GFP_NOWARN | __GFP_NORETRY |
|
|
|
|
__GFP_NOMEMALLOC);
|
|
|
|
if (!buf_desc->cpu_addr) {
|
|
|
|
kfree(buf_desc);
|
|
|
|
return ERR_PTR(-EAGAIN);
|
|
|
|
}
|
|
|
|
buf_desc->len = bufsize;
|
|
|
|
}
|
|
|
|
return buf_desc;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int __smc_buf_create(struct smc_sock *smc, bool is_smcd, bool is_rmb)
|
2017-01-09 15:55:18 +00:00
|
|
|
{
|
2018-05-18 07:34:16 +00:00
|
|
|
struct smc_buf_desc *buf_desc = ERR_PTR(-ENOMEM);
|
2017-01-09 15:55:18 +00:00
|
|
|
struct smc_connection *conn = &smc->conn;
|
|
|
|
struct smc_link_group *lgr = conn->lgr;
|
2017-07-28 11:56:20 +00:00
|
|
|
struct list_head *buf_list;
|
2017-07-28 11:56:14 +00:00
|
|
|
int bufsize, bufsize_short;
|
2023-02-02 08:26:42 +00:00
|
|
|
struct rw_semaphore *lock; /* lock buffer list */
|
2021-06-16 14:52:55 +00:00
|
|
|
bool is_dgraded = false;
|
2017-07-28 11:56:20 +00:00
|
|
|
int sk_buf_size;
|
2017-01-09 15:55:18 +00:00
|
|
|
|
2017-07-28 11:56:20 +00:00
|
|
|
if (is_rmb)
|
|
|
|
/* use socket recv buffer size (w/o overhead) as start value */
|
2022-09-20 09:52:22 +00:00
|
|
|
sk_buf_size = smc->sk.sk_rcvbuf;
|
2017-07-28 11:56:20 +00:00
|
|
|
else
|
|
|
|
/* use socket send buffer size (w/o overhead) as start value */
|
2022-09-20 09:52:22 +00:00
|
|
|
sk_buf_size = smc->sk.sk_sndbuf;
|
2017-07-28 11:56:20 +00:00
|
|
|
|
2021-08-09 08:10:14 +00:00
|
|
|
for (bufsize_short = smc_compress_bufsize(sk_buf_size, is_smcd, is_rmb);
|
2017-07-28 11:56:14 +00:00
|
|
|
bufsize_short >= 0; bufsize_short--) {
|
2017-07-28 11:56:20 +00:00
|
|
|
if (is_rmb) {
|
|
|
|
lock = &lgr->rmbs_lock;
|
|
|
|
buf_list = &lgr->rmbs[bufsize_short];
|
|
|
|
} else {
|
|
|
|
lock = &lgr->sndbufs_lock;
|
|
|
|
buf_list = &lgr->sndbufs[bufsize_short];
|
2017-07-28 11:56:19 +00:00
|
|
|
}
|
2017-07-28 11:56:14 +00:00
|
|
|
bufsize = smc_uncompress_bufsize(bufsize_short);
|
2017-07-28 11:56:15 +00:00
|
|
|
|
2017-07-28 11:56:20 +00:00
|
|
|
/* check for reusable slot in the link group */
|
2018-05-18 07:34:16 +00:00
|
|
|
buf_desc = smc_buf_get_slot(bufsize_short, lock, buf_list);
|
2017-07-28 11:56:20 +00:00
|
|
|
if (buf_desc) {
|
2022-07-14 09:44:01 +00:00
|
|
|
buf_desc->is_dma_need_sync = 0;
|
2021-06-16 14:52:58 +00:00
|
|
|
SMC_STAT_RMB_SIZE(smc, is_smcd, is_rmb, bufsize);
|
|
|
|
SMC_STAT_BUF_REUSE(smc, is_smcd, is_rmb);
|
2017-01-09 15:55:18 +00:00
|
|
|
break; /* found reusable slot */
|
|
|
|
}
|
2017-07-28 11:56:15 +00:00
|
|
|
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
if (is_smcd)
|
|
|
|
buf_desc = smcd_new_buf_create(lgr, is_rmb, bufsize);
|
|
|
|
else
|
|
|
|
buf_desc = smcr_new_buf_create(lgr, is_rmb, bufsize);
|
|
|
|
|
2017-07-28 11:56:21 +00:00
|
|
|
if (PTR_ERR(buf_desc) == -ENOMEM)
|
|
|
|
break;
|
2021-06-16 14:52:55 +00:00
|
|
|
if (IS_ERR(buf_desc)) {
|
|
|
|
if (!is_dgraded) {
|
|
|
|
is_dgraded = true;
|
2021-06-16 14:52:58 +00:00
|
|
|
SMC_STAT_RMB_DOWNGRADED(smc, is_smcd, is_rmb);
|
2021-06-16 14:52:55 +00:00
|
|
|
}
|
2017-07-28 11:56:15 +00:00
|
|
|
continue;
|
2021-06-16 14:52:55 +00:00
|
|
|
}
|
2017-07-28 11:56:16 +00:00
|
|
|
|
2021-06-16 14:52:58 +00:00
|
|
|
SMC_STAT_RMB_ALLOC(smc, is_smcd, is_rmb);
|
|
|
|
SMC_STAT_RMB_SIZE(smc, is_smcd, is_rmb, bufsize);
|
2017-07-28 11:56:20 +00:00
|
|
|
buf_desc->used = 1;
|
2023-02-02 08:26:42 +00:00
|
|
|
down_write(lock);
|
2017-07-28 11:56:20 +00:00
|
|
|
list_add(&buf_desc->list, buf_list);
|
2023-02-02 08:26:42 +00:00
|
|
|
up_write(lock);
|
2017-07-28 11:56:20 +00:00
|
|
|
break; /* found */
|
2017-01-09 15:55:18 +00:00
|
|
|
}
|
2017-07-28 11:56:20 +00:00
|
|
|
|
2017-07-28 11:56:21 +00:00
|
|
|
if (IS_ERR(buf_desc))
|
2020-07-26 18:34:28 +00:00
|
|
|
return PTR_ERR(buf_desc);
|
2017-07-28 11:56:20 +00:00
|
|
|
|
2020-04-29 15:10:41 +00:00
|
|
|
if (!is_smcd) {
|
|
|
|
if (smcr_buf_map_usable_links(lgr, buf_desc, is_rmb)) {
|
net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.
When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.
So this patch is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.
Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:
1) regression in data path, which is brought by additional address
translation of sndbuf by RNIC in Tx. But in general, translating
address through MTT is fast.
Taking 256KB sndbuf and RMB as an example, the comparisons in qperf
latency and bandwidth test with physically and virtually contiguous
buffers are as follows:
- client:
smc_run taskset -c <cpu> qperf <server> -oo msg_size:1:64K:*2\
-t 5 -vu tcp_{bw|lat}
- server:
smc_run taskset -c <cpu> qperf
[latency]
msgsize tcp smcr smcr-use-virt-buf
1 11.17 us 7.56 us 7.51 us (-0.67%)
2 10.65 us 7.74 us 7.56 us (-2.31%)
4 11.11 us 7.52 us 7.59 us ( 0.84%)
8 10.83 us 7.55 us 7.51 us (-0.48%)
16 11.21 us 7.46 us 7.51 us ( 0.71%)
32 10.65 us 7.53 us 7.58 us ( 0.61%)
64 10.95 us 7.74 us 7.80 us ( 0.76%)
128 11.14 us 7.83 us 7.87 us ( 0.47%)
256 10.97 us 7.94 us 7.92 us (-0.28%)
512 11.23 us 7.94 us 8.20 us ( 3.25%)
1024 11.60 us 8.12 us 8.20 us ( 0.96%)
2048 14.04 us 8.30 us 8.51 us ( 2.49%)
4096 16.88 us 9.13 us 9.07 us (-0.64%)
8192 22.50 us 10.56 us 11.22 us ( 6.26%)
16384 28.99 us 12.88 us 13.83 us ( 7.37%)
32768 40.13 us 16.76 us 16.95 us ( 1.16%)
65536 68.70 us 24.68 us 24.85 us ( 0.68%)
[bandwidth]
msgsize tcp smcr smcr-use-virt-buf
1 1.65 MB/s 1.59 MB/s 1.53 MB/s (-3.88%)
2 3.32 MB/s 3.17 MB/s 3.08 MB/s (-2.67%)
4 6.66 MB/s 6.33 MB/s 6.09 MB/s (-3.85%)
8 13.67 MB/s 13.45 MB/s 11.97 MB/s (-10.99%)
16 25.36 MB/s 27.15 MB/s 24.16 MB/s (-11.01%)
32 48.22 MB/s 54.24 MB/s 49.41 MB/s (-8.89%)
64 106.79 MB/s 107.32 MB/s 99.05 MB/s (-7.71%)
128 210.21 MB/s 202.46 MB/s 201.02 MB/s (-0.71%)
256 400.81 MB/s 416.81 MB/s 393.52 MB/s (-5.59%)
512 746.49 MB/s 834.12 MB/s 809.99 MB/s (-2.89%)
1024 1292.33 MB/s 1641.96 MB/s 1571.82 MB/s (-4.27%)
2048 2007.64 MB/s 2760.44 MB/s 2717.68 MB/s (-1.55%)
4096 2665.17 MB/s 4157.44 MB/s 4070.76 MB/s (-2.09%)
8192 3159.72 MB/s 4361.57 MB/s 4270.65 MB/s (-2.08%)
16384 4186.70 MB/s 4574.13 MB/s 4501.17 MB/s (-1.60%)
32768 4093.21 MB/s 4487.42 MB/s 4322.43 MB/s (-3.68%)
65536 4057.14 MB/s 4735.61 MB/s 4555.17 MB/s (-3.81%)
2) regression in buffer initialization and destruction path, which is
brought by additional MR operations of sndbufs. But thanks to link
group buffer reuse mechanism, the impact of this kind of regression
decreases as times of buffer reuse increases.
Taking 256KB sndbuf and RMB as an example, latency of some key SMC-R
buffer-related function obtained by bpftrace are as follows:
Function Phys-bufs Virt-bufs
smcr_new_buf_create() 67154 ns 79164 ns
smc_ib_buf_map_sg() 525 ns 928 ns
smc_ib_get_memory_region() 162294 ns 161191 ns
smc_wr_reg_send() 9957 ns 9635 ns
smc_ib_put_memory_region() 203548 ns 198374 ns
smc_ib_buf_unmap_sg() 508 ns 1158 ns
------------
Test environment notes:
1. Above tests run on 2 VMs within the same Host.
2. The NIC is ConnectX-4Lx, using SRIOV and passing through 2 VFs to
the each VM respectively.
3. VMs' vCPUs are binded to different physical CPUs, and the binded
physical CPUs are isolated by `isolcpus=xxx` cmdline.
4. NICs' queue number are set to 1.
Signed-off-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2022-07-14 09:44:04 +00:00
|
|
|
smcr_buf_unuse(buf_desc, is_rmb, lgr);
|
2020-04-29 15:10:41 +00:00
|
|
|
return -ENOMEM;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-07-28 11:56:20 +00:00
|
|
|
if (is_rmb) {
|
|
|
|
conn->rmb_desc = buf_desc;
|
2017-07-28 11:56:14 +00:00
|
|
|
conn->rmbe_size_short = bufsize_short;
|
2022-09-20 09:52:22 +00:00
|
|
|
smc->sk.sk_rcvbuf = bufsize;
|
2017-01-09 15:55:22 +00:00
|
|
|
atomic_set(&conn->bytes_to_rcv, 0);
|
2018-06-28 17:05:10 +00:00
|
|
|
conn->rmbe_update_limit =
|
|
|
|
smc_rmb_wnd_update_limit(buf_desc->len);
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
if (is_smcd)
|
|
|
|
smc_ism_set_conn(conn); /* map RMB/smcd_dev to conn */
|
2017-01-09 15:55:18 +00:00
|
|
|
} else {
|
2017-07-28 11:56:20 +00:00
|
|
|
conn->sndbuf_desc = buf_desc;
|
2022-09-20 09:52:22 +00:00
|
|
|
smc->sk.sk_sndbuf = bufsize;
|
2017-07-28 11:56:20 +00:00
|
|
|
atomic_set(&conn->sndbuf_space, bufsize);
|
2017-01-09 15:55:18 +00:00
|
|
|
}
|
2017-07-28 11:56:20 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2017-07-28 11:56:22 +00:00
|
|
|
void smc_sndbuf_sync_sg_for_device(struct smc_connection *conn)
|
|
|
|
{
|
2022-07-14 09:44:01 +00:00
|
|
|
if (!conn->sndbuf_desc->is_dma_need_sync)
|
|
|
|
return;
|
2022-01-13 08:36:41 +00:00
|
|
|
if (!smc_conn_lgr_valid(conn) || conn->lgr->is_smcd ||
|
|
|
|
!smc_link_active(conn->lnk))
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
return;
|
2020-04-29 15:10:40 +00:00
|
|
|
smc_ib_sync_sg_for_device(conn->lnk, conn->sndbuf_desc, DMA_TO_DEVICE);
|
2017-07-28 11:56:22 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
void smc_rmb_sync_sg_for_cpu(struct smc_connection *conn)
|
|
|
|
{
|
2020-04-29 15:10:41 +00:00
|
|
|
int i;
|
2017-07-28 11:56:22 +00:00
|
|
|
|
2022-07-14 09:44:01 +00:00
|
|
|
if (!conn->rmb_desc->is_dma_need_sync)
|
|
|
|
return;
|
2022-01-13 08:36:41 +00:00
|
|
|
if (!smc_conn_lgr_valid(conn) || conn->lgr->is_smcd)
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
return;
|
2020-04-29 15:10:41 +00:00
|
|
|
for (i = 0; i < SMC_LINKS_PER_LGR_MAX; i++) {
|
2020-07-18 13:06:16 +00:00
|
|
|
if (!smc_link_active(&conn->lgr->lnk[i]))
|
2020-04-29 15:10:41 +00:00
|
|
|
continue;
|
|
|
|
smc_ib_sync_sg_for_cpu(&conn->lgr->lnk[i], conn->rmb_desc,
|
|
|
|
DMA_FROM_DEVICE);
|
|
|
|
}
|
2017-07-28 11:56:22 +00:00
|
|
|
}
|
|
|
|
|
2017-07-28 11:56:20 +00:00
|
|
|
/* create the send and receive buffer for an SMC socket;
|
|
|
|
* receive buffers are called RMBs;
|
|
|
|
* (even though the SMC protocol allows more than one RMB-element per RMB,
|
|
|
|
* the Linux implementation uses just one RMB-element per RMB, i.e. uses an
|
|
|
|
* extra RMB for every connection in a link group
|
|
|
|
*/
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
int smc_buf_create(struct smc_sock *smc, bool is_smcd)
|
2017-07-28 11:56:20 +00:00
|
|
|
{
|
|
|
|
int rc;
|
|
|
|
|
|
|
|
/* create send buffer */
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
rc = __smc_buf_create(smc, is_smcd, false);
|
2017-07-28 11:56:20 +00:00
|
|
|
if (rc)
|
|
|
|
return rc;
|
|
|
|
/* create rmb */
|
net/smc: add base infrastructure for SMC-D and ISM
SMC supports two variants: SMC-R and SMC-D. For data transport, SMC-R
uses RDMA devices, SMC-D uses so-called Internal Shared Memory (ISM)
devices. An ISM device only allows shared memory communication between
SMC instances on the same machine. For example, this allows virtual
machines on the same host to communicate via SMC without RDMA devices.
This patch adds the base infrastructure for SMC-D and ISM devices to
the existing SMC code. It contains the following:
* ISM driver interface:
This interface allows an ISM driver to register ISM devices in SMC. In
the process, the driver provides a set of device ops for each device.
SMC uses these ops to execute SMC specific operations on or transfer
data over the device.
* Core SMC-D link group, connection, and buffer support:
Link groups, SMC connections and SMC buffers (in smc_core) are
extended to support SMC-D.
* SMC type checks:
Some type checks are added to prevent using SMC-R specific code for
SMC-D and vice versa.
To actually use SMC-D, additional changes to pnetid, CLC, CDC, etc. are
required. These are added in follow-up patches.
Signed-off-by: Hans Wippel <hwippel@linux.ibm.com>
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Suggested-by: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-06-28 17:05:07 +00:00
|
|
|
rc = __smc_buf_create(smc, is_smcd, true);
|
2020-07-18 13:06:17 +00:00
|
|
|
if (rc) {
|
2023-02-02 08:26:42 +00:00
|
|
|
down_write(&smc->conn.lgr->sndbufs_lock);
|
2020-07-18 13:06:17 +00:00
|
|
|
list_del(&smc->conn.sndbuf_desc->list);
|
2023-02-02 08:26:42 +00:00
|
|
|
up_write(&smc->conn.lgr->sndbufs_lock);
|
2018-05-18 07:34:17 +00:00
|
|
|
smc_buf_free(smc->conn.lgr, false, smc->conn.sndbuf_desc);
|
2020-09-03 19:53:17 +00:00
|
|
|
smc->conn.sndbuf_desc = NULL;
|
2020-07-18 13:06:17 +00:00
|
|
|
}
|
2017-07-28 11:56:20 +00:00
|
|
|
return rc;
|
2017-01-09 15:55:18 +00:00
|
|
|
}
|
2017-01-09 15:55:20 +00:00
|
|
|
|
|
|
|
static inline int smc_rmb_reserve_rtoken_idx(struct smc_link_group *lgr)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for_each_clear_bit(i, lgr->rtokens_used_mask, SMC_RMBS_PER_LGR_MAX) {
|
|
|
|
if (!test_and_set_bit(i, lgr->rtokens_used_mask))
|
|
|
|
return i;
|
|
|
|
}
|
|
|
|
return -ENOSPC;
|
|
|
|
}
|
|
|
|
|
2020-04-30 13:55:47 +00:00
|
|
|
static int smc_rtoken_find_by_link(struct smc_link_group *lgr, int lnk_idx,
|
|
|
|
u32 rkey)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < SMC_RMBS_PER_LGR_MAX; i++) {
|
|
|
|
if (test_bit(i, lgr->rtokens_used_mask) &&
|
|
|
|
lgr->rtokens[i][lnk_idx].rkey == rkey)
|
|
|
|
return i;
|
|
|
|
}
|
|
|
|
return -ENOENT;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* set rtoken for a new link to an existing rmb */
|
|
|
|
void smc_rtoken_set(struct smc_link_group *lgr, int link_idx, int link_idx_new,
|
|
|
|
__be32 nw_rkey_known, __be64 nw_vaddr, __be32 nw_rkey)
|
|
|
|
{
|
|
|
|
int rtok_idx;
|
|
|
|
|
|
|
|
rtok_idx = smc_rtoken_find_by_link(lgr, link_idx, ntohl(nw_rkey_known));
|
|
|
|
if (rtok_idx == -ENOENT)
|
|
|
|
return;
|
|
|
|
lgr->rtokens[rtok_idx][link_idx_new].rkey = ntohl(nw_rkey);
|
|
|
|
lgr->rtokens[rtok_idx][link_idx_new].dma_addr = be64_to_cpu(nw_vaddr);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* set rtoken for a new link whose link_id is given */
|
|
|
|
void smc_rtoken_set2(struct smc_link_group *lgr, int rtok_idx, int link_id,
|
|
|
|
__be64 nw_vaddr, __be32 nw_rkey)
|
|
|
|
{
|
|
|
|
u64 dma_addr = be64_to_cpu(nw_vaddr);
|
|
|
|
u32 rkey = ntohl(nw_rkey);
|
|
|
|
bool found = false;
|
|
|
|
int link_idx;
|
|
|
|
|
|
|
|
for (link_idx = 0; link_idx < SMC_LINKS_PER_LGR_MAX; link_idx++) {
|
|
|
|
if (lgr->lnk[link_idx].link_id == link_id) {
|
|
|
|
found = true;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
if (!found)
|
|
|
|
return;
|
|
|
|
lgr->rtokens[rtok_idx][link_idx].rkey = rkey;
|
|
|
|
lgr->rtokens[rtok_idx][link_idx].dma_addr = dma_addr;
|
|
|
|
}
|
|
|
|
|
2018-03-01 12:51:30 +00:00
|
|
|
/* add a new rtoken from peer */
|
2020-04-29 15:10:40 +00:00
|
|
|
int smc_rtoken_add(struct smc_link *lnk, __be64 nw_vaddr, __be32 nw_rkey)
|
2017-01-09 15:55:20 +00:00
|
|
|
{
|
2020-04-29 15:10:40 +00:00
|
|
|
struct smc_link_group *lgr = smc_get_lgr(lnk);
|
2018-03-01 12:51:30 +00:00
|
|
|
u64 dma_addr = be64_to_cpu(nw_vaddr);
|
|
|
|
u32 rkey = ntohl(nw_rkey);
|
2017-01-09 15:55:20 +00:00
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < SMC_RMBS_PER_LGR_MAX; i++) {
|
2020-04-29 15:10:40 +00:00
|
|
|
if (lgr->rtokens[i][lnk->link_idx].rkey == rkey &&
|
|
|
|
lgr->rtokens[i][lnk->link_idx].dma_addr == dma_addr &&
|
2017-01-09 15:55:20 +00:00
|
|
|
test_bit(i, lgr->rtokens_used_mask)) {
|
2018-03-01 12:51:30 +00:00
|
|
|
/* already in list */
|
|
|
|
return i;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
i = smc_rmb_reserve_rtoken_idx(lgr);
|
|
|
|
if (i < 0)
|
|
|
|
return i;
|
2020-04-29 15:10:40 +00:00
|
|
|
lgr->rtokens[i][lnk->link_idx].rkey = rkey;
|
|
|
|
lgr->rtokens[i][lnk->link_idx].dma_addr = dma_addr;
|
2018-03-01 12:51:30 +00:00
|
|
|
return i;
|
|
|
|
}
|
|
|
|
|
2020-04-29 15:10:42 +00:00
|
|
|
/* delete an rtoken from all links */
|
2020-04-29 15:10:40 +00:00
|
|
|
int smc_rtoken_delete(struct smc_link *lnk, __be32 nw_rkey)
|
2018-03-01 12:51:30 +00:00
|
|
|
{
|
2020-04-29 15:10:40 +00:00
|
|
|
struct smc_link_group *lgr = smc_get_lgr(lnk);
|
2018-03-01 12:51:30 +00:00
|
|
|
u32 rkey = ntohl(nw_rkey);
|
2020-04-29 15:10:42 +00:00
|
|
|
int i, j;
|
2018-03-01 12:51:30 +00:00
|
|
|
|
|
|
|
for (i = 0; i < SMC_RMBS_PER_LGR_MAX; i++) {
|
2020-04-29 15:10:40 +00:00
|
|
|
if (lgr->rtokens[i][lnk->link_idx].rkey == rkey &&
|
2018-03-01 12:51:30 +00:00
|
|
|
test_bit(i, lgr->rtokens_used_mask)) {
|
2020-04-29 15:10:42 +00:00
|
|
|
for (j = 0; j < SMC_LINKS_PER_LGR_MAX; j++) {
|
|
|
|
lgr->rtokens[i][j].rkey = 0;
|
|
|
|
lgr->rtokens[i][j].dma_addr = 0;
|
|
|
|
}
|
2018-03-01 12:51:30 +00:00
|
|
|
clear_bit(i, lgr->rtokens_used_mask);
|
2017-01-09 15:55:20 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
}
|
2018-03-01 12:51:30 +00:00
|
|
|
return -ENOENT;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* save rkey and dma_addr received from peer during clc handshake */
|
|
|
|
int smc_rmb_rtoken_handling(struct smc_connection *conn,
|
2020-04-29 15:10:42 +00:00
|
|
|
struct smc_link *lnk,
|
2018-03-01 12:51:30 +00:00
|
|
|
struct smc_clc_msg_accept_confirm *clc)
|
|
|
|
{
|
2020-09-10 16:48:23 +00:00
|
|
|
conn->rtoken_idx = smc_rtoken_add(lnk, clc->r0.rmb_dma_addr,
|
|
|
|
clc->r0.rmb_rkey);
|
2017-01-09 15:55:20 +00:00
|
|
|
if (conn->rtoken_idx < 0)
|
|
|
|
return conn->rtoken_idx;
|
|
|
|
return 0;
|
|
|
|
}
|
2018-05-18 07:34:11 +00:00
|
|
|
|
2019-10-09 08:07:46 +00:00
|
|
|
static void smc_core_going_away(void)
|
|
|
|
{
|
|
|
|
struct smc_ib_device *smcibdev;
|
|
|
|
struct smcd_dev *smcd;
|
|
|
|
|
2020-07-08 15:05:13 +00:00
|
|
|
mutex_lock(&smc_ib_devices.mutex);
|
2019-10-09 08:07:46 +00:00
|
|
|
list_for_each_entry(smcibdev, &smc_ib_devices.list, list) {
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < SMC_MAX_PORTS; i++)
|
|
|
|
set_bit(i, smcibdev->ports_going_away);
|
|
|
|
}
|
2020-07-08 15:05:13 +00:00
|
|
|
mutex_unlock(&smc_ib_devices.mutex);
|
2019-10-09 08:07:46 +00:00
|
|
|
|
2020-07-08 15:05:14 +00:00
|
|
|
mutex_lock(&smcd_dev_list.mutex);
|
2019-10-09 08:07:46 +00:00
|
|
|
list_for_each_entry(smcd, &smcd_dev_list.list, list) {
|
|
|
|
smcd->going_away = 1;
|
|
|
|
}
|
2020-07-08 15:05:14 +00:00
|
|
|
mutex_unlock(&smcd_dev_list.mutex);
|
2019-10-09 08:07:46 +00:00
|
|
|
}
|
|
|
|
|
2019-11-14 12:02:42 +00:00
|
|
|
/* Clean up all SMC link groups */
|
|
|
|
static void smc_lgrs_shutdown(void)
|
2018-05-18 07:34:11 +00:00
|
|
|
{
|
2019-10-09 08:07:43 +00:00
|
|
|
struct smcd_dev *smcd;
|
2018-05-18 07:34:11 +00:00
|
|
|
|
2019-10-09 08:07:46 +00:00
|
|
|
smc_core_going_away();
|
|
|
|
|
2019-11-14 12:02:47 +00:00
|
|
|
smc_smcr_terminate_all(NULL);
|
2019-10-09 08:07:43 +00:00
|
|
|
|
2020-07-08 15:05:14 +00:00
|
|
|
mutex_lock(&smcd_dev_list.mutex);
|
2019-10-09 08:07:43 +00:00
|
|
|
list_for_each_entry(smcd, &smcd_dev_list.list, list)
|
2019-11-14 12:02:42 +00:00
|
|
|
smc_smcd_terminate_all(smcd);
|
2020-07-08 15:05:14 +00:00
|
|
|
mutex_unlock(&smcd_dev_list.mutex);
|
2018-05-18 07:34:11 +00:00
|
|
|
}
|
2019-11-14 12:02:42 +00:00
|
|
|
|
2019-11-16 16:47:30 +00:00
|
|
|
static int smc_core_reboot_event(struct notifier_block *this,
|
|
|
|
unsigned long event, void *ptr)
|
|
|
|
{
|
|
|
|
smc_lgrs_shutdown();
|
2019-12-19 11:51:13 +00:00
|
|
|
smc_ib_unregister_client();
|
2023-01-23 18:17:49 +00:00
|
|
|
smc_ism_exit();
|
2019-11-16 16:47:30 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct notifier_block smc_reboot_notifier = {
|
|
|
|
.notifier_call = smc_core_reboot_event,
|
|
|
|
};
|
|
|
|
|
2019-11-16 16:47:29 +00:00
|
|
|
int __init smc_core_init(void)
|
|
|
|
{
|
2019-11-16 16:47:30 +00:00
|
|
|
return register_reboot_notifier(&smc_reboot_notifier);
|
2019-11-16 16:47:29 +00:00
|
|
|
}
|
|
|
|
|
2019-11-14 12:02:42 +00:00
|
|
|
/* Called (from smc_exit) when module is removed */
|
|
|
|
void smc_core_exit(void)
|
|
|
|
{
|
2019-11-16 16:47:30 +00:00
|
|
|
unregister_reboot_notifier(&smc_reboot_notifier);
|
2019-11-14 12:02:42 +00:00
|
|
|
smc_lgrs_shutdown();
|
|
|
|
}
|