mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
synced 2024-08-21 16:29:59 +00:00
![Ira Weiny](/assets/img/avatar_default.png)
Pach series "Add FOLL_LONGTERM to GUP fast and use it". HFI1, qib, and mthca, use get_user_pages_fast() due to its performance advantages. These pages can be held for a significant time. But get_user_pages_fast() does not protect against mapping FS DAX pages. Introduce FOLL_LONGTERM and use this flag in get_user_pages_fast() which retains the performance while also adding the FS DAX checks. XDP has also shown interest in using this functionality.[1] In addition we change get_user_pages() to use the new FOLL_LONGTERM flag and remove the specialized get_user_pages_longterm call. [1] https://lkml.org/lkml/2019/3/19/939 "longterm" is a relative thing and at this point is probably a misnomer. This is really flagging a pin which is going to be given to hardware and can't move. I've thought of a couple of alternative names but I think we have to settle on if we are going to use FL_LAYOUT or something else to solve the "longterm" problem. Then I think we can change the flag to a better name. Secondly, it depends on how often you are registering memory. I have spoken with some RDMA users who consider MR in the performance path... For the overall application performance. I don't have the numbers as the tests for HFI1 were done a long time ago. But there was a significant advantage. Some of which is probably due to the fact that you don't have to hold mmap_sem. Finally, architecturally I think it would be good for everyone to use *_fast. There are patches submitted to the RDMA list which would allow the use of *_fast (they reworking the use of mmap_sem) and as soon as they are accepted I'll submit a patch to convert the RDMA core as well. Also to this point others are looking to use *_fast. As an aside, Jasons pointed out in my previous submission that *_fast and *_unlocked look very much the same. I agree and I think further cleanup will be coming. But I'm focused on getting the final solution for DAX at the moment. This patch (of 7): This patch starts a series which aims to support FOLL_LONGTERM in get_user_pages_fast(). Some callers who would like to do a longterm (user controlled pin) of pages with the fast variant of GUP for performance purposes. Rather than have a separate get_user_pages_longterm() call, introduce FOLL_LONGTERM and change the longterm callers to use it. This patch does not change any functionality. In the short term "longterm" or user controlled pins are unsafe for Filesystems and FS DAX in particular has been blocked. However, callers of get_user_pages_fast() were not "protected". FOLL_LONGTERM can _only_ be supported with get_user_pages[_fast]() as it requires vmas to determine if DAX is in use. NOTE: In merging with the CMA changes we opt to change the get_user_pages() call in check_and_migrate_cma_pages() to a call of __get_user_pages_locked() on the newly migrated pages. This makes the code read better in that we are calling __get_user_pages_locked() on the pages before and after a potential migration. As a side affect some of the interfaces are cleaned up but this is not the primary purpose of the series. In review[1] it was asked: <quote> > This I don't get - if you do lock down long term mappings performance > of the actual get_user_pages call shouldn't matter to start with. > > What do I miss? A couple of points. First "longterm" is a relative thing and at this point is probably a misnomer. This is really flagging a pin which is going to be given to hardware and can't move. I've thought of a couple of alternative names but I think we have to settle on if we are going to use FL_LAYOUT or something else to solve the "longterm" problem. Then I think we can change the flag to a better name. Second, It depends on how often you are registering memory. I have spoken with some RDMA users who consider MR in the performance path... For the overall application performance. I don't have the numbers as the tests for HFI1 were done a long time ago. But there was a significant advantage. Some of which is probably due to the fact that you don't have to hold mmap_sem. Finally, architecturally I think it would be good for everyone to use *_fast. There are patches submitted to the RDMA list which would allow the use of *_fast (they reworking the use of mmap_sem) and as soon as they are accepted I'll submit a patch to convert the RDMA core as well. Also to this point others are looking to use *_fast. As an asside, Jasons pointed out in my previous submission that *_fast and *_unlocked look very much the same. I agree and I think further cleanup will be coming. But I'm focused on getting the final solution for DAX at the moment. </quote> [1] https://lore.kernel.org/lkml/20190220180255.GA12020@iweiny-DESK2.sc.intel.com/T/#md6abad2569f3bf6c1f03686c8097ab6563e94965 [ira.weiny@intel.com: v3] Link: http://lkml.kernel.org/r/20190328084422.29911-2-ira.weiny@intel.com Link: http://lkml.kernel.org/r/20190328084422.29911-2-ira.weiny@intel.com Link: http://lkml.kernel.org/r/20190317183438.2057-2-ira.weiny@intel.com Signed-off-by: Ira Weiny <ira.weiny@intel.com> Reviewed-by: Andrew Morton <akpm@linux-foundation.org> Cc: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com> Cc: Michal Hocko <mhocko@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> Cc: Paul Mackerras <paulus@samba.org> Cc: "David S. Miller" <davem@davemloft.net> Cc: Martin Schwidefsky <schwidefsky@de.ibm.com> Cc: Heiko Carstens <heiko.carstens@de.ibm.com> Cc: Rich Felker <dalias@libc.org> Cc: Yoshinori Sato <ysato@users.sourceforge.jp> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Ingo Molnar <mingo@redhat.com> Cc: Borislav Petkov <bp@alien8.de> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: James Hogan <jhogan@kernel.org> Cc: Dan Williams <dan.j.williams@intel.com> Cc: Mike Marshall <hubcap@omnibond.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
132 lines
2.7 KiB
C
132 lines
2.7 KiB
C
#include <linux/kernel.h>
|
|
#include <linux/mm.h>
|
|
#include <linux/slab.h>
|
|
#include <linux/uaccess.h>
|
|
#include <linux/ktime.h>
|
|
#include <linux/debugfs.h>
|
|
|
|
#define GUP_FAST_BENCHMARK _IOWR('g', 1, struct gup_benchmark)
|
|
#define GUP_LONGTERM_BENCHMARK _IOWR('g', 2, struct gup_benchmark)
|
|
#define GUP_BENCHMARK _IOWR('g', 3, struct gup_benchmark)
|
|
|
|
struct gup_benchmark {
|
|
__u64 get_delta_usec;
|
|
__u64 put_delta_usec;
|
|
__u64 addr;
|
|
__u64 size;
|
|
__u32 nr_pages_per_call;
|
|
__u32 flags;
|
|
__u64 expansion[10]; /* For future use */
|
|
};
|
|
|
|
static int __gup_benchmark_ioctl(unsigned int cmd,
|
|
struct gup_benchmark *gup)
|
|
{
|
|
ktime_t start_time, end_time;
|
|
unsigned long i, nr_pages, addr, next;
|
|
int nr;
|
|
struct page **pages;
|
|
|
|
if (gup->size > ULONG_MAX)
|
|
return -EINVAL;
|
|
|
|
nr_pages = gup->size / PAGE_SIZE;
|
|
pages = kvcalloc(nr_pages, sizeof(void *), GFP_KERNEL);
|
|
if (!pages)
|
|
return -ENOMEM;
|
|
|
|
i = 0;
|
|
nr = gup->nr_pages_per_call;
|
|
start_time = ktime_get();
|
|
for (addr = gup->addr; addr < gup->addr + gup->size; addr = next) {
|
|
if (nr != gup->nr_pages_per_call)
|
|
break;
|
|
|
|
next = addr + nr * PAGE_SIZE;
|
|
if (next > gup->addr + gup->size) {
|
|
next = gup->addr + gup->size;
|
|
nr = (next - addr) / PAGE_SIZE;
|
|
}
|
|
|
|
switch (cmd) {
|
|
case GUP_FAST_BENCHMARK:
|
|
nr = get_user_pages_fast(addr, nr, gup->flags & 1,
|
|
pages + i);
|
|
break;
|
|
case GUP_LONGTERM_BENCHMARK:
|
|
nr = get_user_pages(addr, nr,
|
|
(gup->flags & 1) | FOLL_LONGTERM,
|
|
pages + i, NULL);
|
|
break;
|
|
case GUP_BENCHMARK:
|
|
nr = get_user_pages(addr, nr, gup->flags & 1, pages + i,
|
|
NULL);
|
|
break;
|
|
default:
|
|
return -1;
|
|
}
|
|
|
|
if (nr <= 0)
|
|
break;
|
|
i += nr;
|
|
}
|
|
end_time = ktime_get();
|
|
|
|
gup->get_delta_usec = ktime_us_delta(end_time, start_time);
|
|
gup->size = addr - gup->addr;
|
|
|
|
start_time = ktime_get();
|
|
for (i = 0; i < nr_pages; i++) {
|
|
if (!pages[i])
|
|
break;
|
|
put_page(pages[i]);
|
|
}
|
|
end_time = ktime_get();
|
|
gup->put_delta_usec = ktime_us_delta(end_time, start_time);
|
|
|
|
kvfree(pages);
|
|
return 0;
|
|
}
|
|
|
|
static long gup_benchmark_ioctl(struct file *filep, unsigned int cmd,
|
|
unsigned long arg)
|
|
{
|
|
struct gup_benchmark gup;
|
|
int ret;
|
|
|
|
switch (cmd) {
|
|
case GUP_FAST_BENCHMARK:
|
|
case GUP_LONGTERM_BENCHMARK:
|
|
case GUP_BENCHMARK:
|
|
break;
|
|
default:
|
|
return -EINVAL;
|
|
}
|
|
|
|
if (copy_from_user(&gup, (void __user *)arg, sizeof(gup)))
|
|
return -EFAULT;
|
|
|
|
ret = __gup_benchmark_ioctl(cmd, &gup);
|
|
if (ret)
|
|
return ret;
|
|
|
|
if (copy_to_user((void __user *)arg, &gup, sizeof(gup)))
|
|
return -EFAULT;
|
|
|
|
return 0;
|
|
}
|
|
|
|
static const struct file_operations gup_benchmark_fops = {
|
|
.open = nonseekable_open,
|
|
.unlocked_ioctl = gup_benchmark_ioctl,
|
|
};
|
|
|
|
static int gup_benchmark_init(void)
|
|
{
|
|
debugfs_create_file_unsafe("gup_benchmark", 0600, NULL, NULL,
|
|
&gup_benchmark_fops);
|
|
|
|
return 0;
|
|
}
|
|
|
|
late_initcall(gup_benchmark_init);
|