Jens Axboe de36a15f9a io_uring/io-wq: reduce frequency of acct->lock acquisitions ... When we check if we have work to run, we grab the acct lock, check, drop it, and then return the result. If we do have work to run, then running the work will again grab acct->lock and get the work item. This causes us to grab acct->lock more frequently than we need to. If we have work to do, have io_acct_run_queue() return with the acct lock still acquired. io_worker_handle_work() is then always invoked with the acct lock already held. In a simple test cases that stats files (IORING_OP_STATX always hits io-wq), we see a nice reduction in locking overhead with this change: 19.32% -12.55% [kernel.kallsyms] [k] __cmpwait_case_32 20.90% -12.07% [kernel.kallsyms] [k] queued_spin_lock_slowpath Reviewed-by: Hao Xu <howeyxu@tencent.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>		2023-08-11 10:36:17 -06:00
..
advise.c	io_uring: always go async for unsupported fadvise flags	2023-01-29 15:18:26 -07:00
advise.h
alloc_cache.h	io_uring/rsrc: consolidate node caching	2023-04-12 12:09:41 -06:00
cancel.c	io_uring/cancel: wire up IORING_ASYNC_CANCEL_OP for sync cancel	2023-07-17 10:05:48 -06:00
cancel.h	io_uring/cancel: support opcode based lookup and cancelation	2023-07-17 10:05:48 -06:00
epoll.c	io_uring: undeprecate epoll_ctl support	2023-05-26 20:22:41 -06:00
epoll.h
fdinfo.c	io_uring/fdinfo: get rid of ref tryget	2023-08-10 10:24:19 -06:00
fdinfo.h
filetable.c	io_uring: add helpers to decode the fixed file file_ptr	2023-06-20 09:36:22 -06:00
filetable.h	io_uring: add helpers to decode the fixed file file_ptr	2023-06-20 09:36:22 -06:00
fs.c	io_uring: for requests that require async, force it	2023-01-29 15:18:26 -07:00
fs.h
io-wq.c	io_uring/io-wq: reduce frequency of acct->lock acquisitions	2023-08-11 10:36:17 -06:00
io-wq.h
io_uring.c	io_uring: remove unnecessary forward declaration	2023-08-10 15:01:58 -06:00
io_uring.h	io_uring: have io_file_put() take an io_kiocb rather than the file	2023-08-10 10:27:46 -06:00
kbuf.c	for-6.4/io_uring-2023-04-21	2023-04-26 12:40:31 -07:00
kbuf.h	io_uring: add support for user mapped provided buffer ring	2023-04-03 07:14:21 -06:00
Makefile
msg_ring.c	io_uring: use io_file_from_index in io_msg_grab_file	2023-06-20 09:36:22 -06:00
msg_ring.h	io_uring: get rid of double locking	2022-12-07 06:47:13 -07:00
net.c	io_uring-6.5-2023-07-03	2023-07-03 18:43:10 -07:00
net.h	io_uring: Add KASAN support for alloc_caches	2023-04-03 07:16:14 -06:00
nop.c
nop.h
notif.c	io_uring/notif: add constant for ubuf_info flags	2023-04-15 14:21:04 -06:00
notif.h	io_uring/notif: add constant for ubuf_info flags	2023-04-15 14:21:04 -06:00
opdef.c	io_uring: Pass whole sqe to commands	2023-05-04 08:19:05 -06:00
opdef.h	io_uring: Split io_issue_def struct	2023-01-29 15:17:41 -07:00
openclose.c	fsnotify: move fsnotify_open() hook into do_dentry_open()	2023-06-12 10:43:45 +02:00
openclose.h
poll.c	io_uring/cancel: support opcode based lookup and cancelation	2023-07-17 10:05:48 -06:00
poll.h	io_uring: avoid indirect function calls for the hottest task_work	2023-06-02 08:55:37 -06:00
refs.h
rsrc.c	- Yosry Ahmed brought back some cgroup v1 stats in OOM logs.	2023-06-28 10:28:11 -07:00
rsrc.h	io_uring/rsrc: Remove unused declaration io_rsrc_put_tw()	2023-08-09 10:46:46 -06:00
rw.c	io_uring: return REQ_F_ flags from io_file_get_flags	2023-06-20 09:36:22 -06:00
rw.h	io_uring: avoid indirect function calls for the hottest task_work	2023-06-02 08:55:37 -06:00
slist.h	io_uring: silence variable ‘prev’ set but not used warning	2023-03-09 10:10:58 -07:00
splice.c	io_uring/splice: use fput() directly	2023-08-10 10:24:25 -06:00
splice.h
sqpoll.c	io_uring: unlock sqd->lock before sq thread release CPU	2023-05-25 09:30:13 -06:00
sqpoll.h	io_uring: make io_sqpoll_wait_sq return void	2023-01-29 15:17:40 -07:00
statx.c	io_uring: for requests that require async, force it	2023-01-29 15:18:26 -07:00
statx.h
sync.c	io_uring: for requests that require async, force it	2023-01-29 15:18:26 -07:00
sync.h
tctx.c	io_uring: Add io_uring_setup flag to pre-register ring fd and never install it	2023-05-16 08:06:00 -06:00
tctx.h	io_uring: simplify __io_uring_add_tctx_node	2022-10-07 12:25:30 -06:00
timeout.c	io_uring: use cancelation match helper for poll and timeout requests	2023-07-17 10:05:48 -06:00
timeout.h	io_uring: remove unused return from io_disarm_next	2022-09-21 13:15:01 -06:00
uring_cmd.c	io_uring: Add io_uring command support for sockets	2023-08-09 10:46:15 -06:00
uring_cmd.h	io_uring: Remove unnecessary BUILD_BUG_ON	2023-05-04 08:19:05 -06:00
xattr.c	io_uring: for requests that require async, force it	2023-01-29 15:18:26 -07:00
xattr.h