mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
synced 2024-09-12 05:48:40 +00:00
![Mel Gorman](/assets/img/avatar_default.png)
When a user or administrator requires swap for their application, they create a swap partition and file, format it with mkswap and activate it with swapon. Swap over the network is considered as an option in diskless systems. The two likely scenarios are when blade servers are used as part of a cluster where the form factor or maintenance costs do not allow the use of disks and thin clients. The Linux Terminal Server Project recommends the use of the Network Block Device (NBD) for swap according to the manual at https://sourceforge.net/projects/ltsp/files/Docs-Admin-Guide/LTSPManual.pdf/download There is also documentation and tutorials on how to setup swap over NBD at places like https://help.ubuntu.com/community/UbuntuLTSP/EnableNBDSWAP The nbd-client also documents the use of NBD as swap. Despite this, the fact is that a machine using NBD for swap can deadlock within minutes if swap is used intensively. This patch series addresses the problem. The core issue is that network block devices do not use mempools like normal block devices do. As the host cannot control where they receive packets from, they cannot reliably work out in advance how much memory they might need. Some years ago, Peter Zijlstra developed a series of patches that supported swap over an NFS that at least one distribution is carrying within their kernels. This patch series borrows very heavily from Peter's work to support swapping over NBD as a pre-requisite to supporting swap-over-NFS. The bulk of the complexity is concerned with preserving memory that is allocated from the PFMEMALLOC reserves for use by the network layer which is needed for both NBD and NFS. Patch 1 adds knowledge of the PFMEMALLOC reserves to SLAB and SLUB to preserve access to pages allocated under low memory situations to callers that are freeing memory. Patch 2 optimises the SLUB fast path to avoid pfmemalloc checks Patch 3 introduces __GFP_MEMALLOC to allow access to the PFMEMALLOC reserves without setting PFMEMALLOC. Patch 4 opens the possibility for softirqs to use PFMEMALLOC reserves for later use by network packet processing. Patch 5 only sets page->pfmemalloc when ALLOC_NO_WATERMARKS was required Patch 6 ignores memory policies when ALLOC_NO_WATERMARKS is set. Patches 7-12 allows network processing to use PFMEMALLOC reserves when the socket has been marked as being used by the VM to clean pages. If packets are received and stored in pages that were allocated under low-memory situations and are unrelated to the VM, the packets are dropped. Patch 11 reintroduces __skb_alloc_page which the networking folk may object to but is needed in some cases to propogate pfmemalloc from a newly allocated page to an skb. If there is a strong objection, this patch can be dropped with the impact being that swap-over-network will be slower in some cases but it should not fail. Patch 13 is a micro-optimisation to avoid a function call in the common case. Patch 14 tags NBD sockets as being SOCK_MEMALLOC so they can use PFMEMALLOC if necessary. Patch 15 notes that it is still possible for the PFMEMALLOC reserve to be depleted. To prevent this, direct reclaimers get throttled on a waitqueue if 50% of the PFMEMALLOC reserves are depleted. It is expected that kswapd and the direct reclaimers already running will clean enough pages for the low watermark to be reached and the throttled processes are woken up. Patch 16 adds a statistic to track how often processes get throttled Some basic performance testing was run using kernel builds, netperf on loopback for UDP and TCP, hackbench (pipes and sockets), iozone and sysbench. Each of them were expected to use the sl*b allocators reasonably heavily but there did not appear to be significant performance variances. For testing swap-over-NBD, a machine was booted with 2G of RAM with a swapfile backed by NBD. 8*NUM_CPU processes were started that create anonymous memory mappings and read them linearly in a loop. The total size of the mappings were 4*PHYSICAL_MEMORY to use swap heavily under memory pressure. Without the patches and using SLUB, the machine locks up within minutes and runs to completion with them applied. With SLAB, the story is different as an unpatched kernel run to completion. However, the patched kernel completed the test 45% faster. MICRO 3.5.0-rc2 3.5.0-rc2 vanilla swapnbd Unrecognised test vmscan-anon-mmap-write MMTests Statistics: duration Sys Time Running Test (seconds) 197.80 173.07 User+Sys Time Running Test (seconds) 206.96 182.03 Total Elapsed Time (seconds) 3240.70 1762.09 This patch: mm: sl[au]b: add knowledge of PFMEMALLOC reserve pages Allocations of pages below the min watermark run a risk of the machine hanging due to a lack of memory. To prevent this, only callers who have PF_MEMALLOC or TIF_MEMDIE set and are not processing an interrupt are allowed to allocate with ALLOC_NO_WATERMARKS. Once they are allocated to a slab though, nothing prevents other callers consuming free objects within those slabs. This patch limits access to slab pages that were alloced from the PFMEMALLOC reserves. When this patch is applied, pages allocated from below the low watermark are returned with page->pfmemalloc set and it is up to the caller to determine how the page should be protected. SLAB restricts access to any page with page->pfmemalloc set to callers which are known to able to access the PFMEMALLOC reserve. If one is not available, an attempt is made to allocate a new page rather than use a reserve. SLUB is a bit more relaxed in that it only records if the current per-CPU page was allocated from PFMEMALLOC reserve and uses another partial slab if the caller does not have the necessary GFP or process flags. This was found to be sufficient in tests to avoid hangs due to SLUB generally maintaining smaller lists than SLAB. In low-memory conditions it does mean that !PFMEMALLOC allocators can fail a slab allocation even though free objects are available because they are being preserved for callers that are freeing pages. [a.p.zijlstra@chello.nl: Original implementation] [sebastian@breakpoint.cc: Correct order of page flag clearing] Signed-off-by: Mel Gorman <mgorman@suse.de> Cc: David Miller <davem@davemloft.net> Cc: Neil Brown <neilb@suse.de> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Mike Christie <michaelc@cs.wisc.edu> Cc: Eric B Munson <emunson@mgebm.net> Cc: Eric Dumazet <eric.dumazet@gmail.com> Cc: Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Cc: Mel Gorman <mgorman@suse.de> Cc: Christoph Lameter <cl@linux.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
532 lines
16 KiB
C
532 lines
16 KiB
C
/*
|
|
* Macros for manipulating and testing page->flags
|
|
*/
|
|
|
|
#ifndef PAGE_FLAGS_H
|
|
#define PAGE_FLAGS_H
|
|
|
|
#include <linux/types.h>
|
|
#include <linux/bug.h>
|
|
#include <linux/mmdebug.h>
|
|
#ifndef __GENERATING_BOUNDS_H
|
|
#include <linux/mm_types.h>
|
|
#include <generated/bounds.h>
|
|
#endif /* !__GENERATING_BOUNDS_H */
|
|
|
|
/*
|
|
* Various page->flags bits:
|
|
*
|
|
* PG_reserved is set for special pages, which can never be swapped out. Some
|
|
* of them might not even exist (eg empty_bad_page)...
|
|
*
|
|
* The PG_private bitflag is set on pagecache pages if they contain filesystem
|
|
* specific data (which is normally at page->private). It can be used by
|
|
* private allocations for its own usage.
|
|
*
|
|
* During initiation of disk I/O, PG_locked is set. This bit is set before I/O
|
|
* and cleared when writeback _starts_ or when read _completes_. PG_writeback
|
|
* is set before writeback starts and cleared when it finishes.
|
|
*
|
|
* PG_locked also pins a page in pagecache, and blocks truncation of the file
|
|
* while it is held.
|
|
*
|
|
* page_waitqueue(page) is a wait queue of all tasks waiting for the page
|
|
* to become unlocked.
|
|
*
|
|
* PG_uptodate tells whether the page's contents is valid. When a read
|
|
* completes, the page becomes uptodate, unless a disk I/O error happened.
|
|
*
|
|
* PG_referenced, PG_reclaim are used for page reclaim for anonymous and
|
|
* file-backed pagecache (see mm/vmscan.c).
|
|
*
|
|
* PG_error is set to indicate that an I/O error occurred on this page.
|
|
*
|
|
* PG_arch_1 is an architecture specific page state bit. The generic code
|
|
* guarantees that this bit is cleared for a page when it first is entered into
|
|
* the page cache.
|
|
*
|
|
* PG_highmem pages are not permanently mapped into the kernel virtual address
|
|
* space, they need to be kmapped separately for doing IO on the pages. The
|
|
* struct page (these bits with information) are always mapped into kernel
|
|
* address space...
|
|
*
|
|
* PG_hwpoison indicates that a page got corrupted in hardware and contains
|
|
* data with incorrect ECC bits that triggered a machine check. Accessing is
|
|
* not safe since it may cause another machine check. Don't touch!
|
|
*/
|
|
|
|
/*
|
|
* Don't use the *_dontuse flags. Use the macros. Otherwise you'll break
|
|
* locked- and dirty-page accounting.
|
|
*
|
|
* The page flags field is split into two parts, the main flags area
|
|
* which extends from the low bits upwards, and the fields area which
|
|
* extends from the high bits downwards.
|
|
*
|
|
* | FIELD | ... | FLAGS |
|
|
* N-1 ^ 0
|
|
* (NR_PAGEFLAGS)
|
|
*
|
|
* The fields area is reserved for fields mapping zone, node (for NUMA) and
|
|
* SPARSEMEM section (for variants of SPARSEMEM that require section ids like
|
|
* SPARSEMEM_EXTREME with !SPARSEMEM_VMEMMAP).
|
|
*/
|
|
enum pageflags {
|
|
PG_locked, /* Page is locked. Don't touch. */
|
|
PG_error,
|
|
PG_referenced,
|
|
PG_uptodate,
|
|
PG_dirty,
|
|
PG_lru,
|
|
PG_active,
|
|
PG_slab,
|
|
PG_owner_priv_1, /* Owner use. If pagecache, fs may use*/
|
|
PG_arch_1,
|
|
PG_reserved,
|
|
PG_private, /* If pagecache, has fs-private data */
|
|
PG_private_2, /* If pagecache, has fs aux data */
|
|
PG_writeback, /* Page is under writeback */
|
|
#ifdef CONFIG_PAGEFLAGS_EXTENDED
|
|
PG_head, /* A head page */
|
|
PG_tail, /* A tail page */
|
|
#else
|
|
PG_compound, /* A compound page */
|
|
#endif
|
|
PG_swapcache, /* Swap page: swp_entry_t in private */
|
|
PG_mappedtodisk, /* Has blocks allocated on-disk */
|
|
PG_reclaim, /* To be reclaimed asap */
|
|
PG_swapbacked, /* Page is backed by RAM/swap */
|
|
PG_unevictable, /* Page is "unevictable" */
|
|
#ifdef CONFIG_MMU
|
|
PG_mlocked, /* Page is vma mlocked */
|
|
#endif
|
|
#ifdef CONFIG_ARCH_USES_PG_UNCACHED
|
|
PG_uncached, /* Page has been mapped as uncached */
|
|
#endif
|
|
#ifdef CONFIG_MEMORY_FAILURE
|
|
PG_hwpoison, /* hardware poisoned page. Don't touch */
|
|
#endif
|
|
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
|
|
PG_compound_lock,
|
|
#endif
|
|
__NR_PAGEFLAGS,
|
|
|
|
/* Filesystems */
|
|
PG_checked = PG_owner_priv_1,
|
|
|
|
/* Two page bits are conscripted by FS-Cache to maintain local caching
|
|
* state. These bits are set on pages belonging to the netfs's inodes
|
|
* when those inodes are being locally cached.
|
|
*/
|
|
PG_fscache = PG_private_2, /* page backed by cache */
|
|
|
|
/* XEN */
|
|
PG_pinned = PG_owner_priv_1,
|
|
PG_savepinned = PG_dirty,
|
|
|
|
/* SLOB */
|
|
PG_slob_free = PG_private,
|
|
};
|
|
|
|
#ifndef __GENERATING_BOUNDS_H
|
|
|
|
/*
|
|
* Macros to create function definitions for page flags
|
|
*/
|
|
#define TESTPAGEFLAG(uname, lname) \
|
|
static inline int Page##uname(const struct page *page) \
|
|
{ return test_bit(PG_##lname, &page->flags); }
|
|
|
|
#define SETPAGEFLAG(uname, lname) \
|
|
static inline void SetPage##uname(struct page *page) \
|
|
{ set_bit(PG_##lname, &page->flags); }
|
|
|
|
#define CLEARPAGEFLAG(uname, lname) \
|
|
static inline void ClearPage##uname(struct page *page) \
|
|
{ clear_bit(PG_##lname, &page->flags); }
|
|
|
|
#define __SETPAGEFLAG(uname, lname) \
|
|
static inline void __SetPage##uname(struct page *page) \
|
|
{ __set_bit(PG_##lname, &page->flags); }
|
|
|
|
#define __CLEARPAGEFLAG(uname, lname) \
|
|
static inline void __ClearPage##uname(struct page *page) \
|
|
{ __clear_bit(PG_##lname, &page->flags); }
|
|
|
|
#define TESTSETFLAG(uname, lname) \
|
|
static inline int TestSetPage##uname(struct page *page) \
|
|
{ return test_and_set_bit(PG_##lname, &page->flags); }
|
|
|
|
#define TESTCLEARFLAG(uname, lname) \
|
|
static inline int TestClearPage##uname(struct page *page) \
|
|
{ return test_and_clear_bit(PG_##lname, &page->flags); }
|
|
|
|
#define __TESTCLEARFLAG(uname, lname) \
|
|
static inline int __TestClearPage##uname(struct page *page) \
|
|
{ return __test_and_clear_bit(PG_##lname, &page->flags); }
|
|
|
|
#define PAGEFLAG(uname, lname) TESTPAGEFLAG(uname, lname) \
|
|
SETPAGEFLAG(uname, lname) CLEARPAGEFLAG(uname, lname)
|
|
|
|
#define __PAGEFLAG(uname, lname) TESTPAGEFLAG(uname, lname) \
|
|
__SETPAGEFLAG(uname, lname) __CLEARPAGEFLAG(uname, lname)
|
|
|
|
#define PAGEFLAG_FALSE(uname) \
|
|
static inline int Page##uname(const struct page *page) \
|
|
{ return 0; }
|
|
|
|
#define TESTSCFLAG(uname, lname) \
|
|
TESTSETFLAG(uname, lname) TESTCLEARFLAG(uname, lname)
|
|
|
|
#define SETPAGEFLAG_NOOP(uname) \
|
|
static inline void SetPage##uname(struct page *page) { }
|
|
|
|
#define CLEARPAGEFLAG_NOOP(uname) \
|
|
static inline void ClearPage##uname(struct page *page) { }
|
|
|
|
#define __CLEARPAGEFLAG_NOOP(uname) \
|
|
static inline void __ClearPage##uname(struct page *page) { }
|
|
|
|
#define TESTCLEARFLAG_FALSE(uname) \
|
|
static inline int TestClearPage##uname(struct page *page) { return 0; }
|
|
|
|
#define __TESTCLEARFLAG_FALSE(uname) \
|
|
static inline int __TestClearPage##uname(struct page *page) { return 0; }
|
|
|
|
struct page; /* forward declaration */
|
|
|
|
TESTPAGEFLAG(Locked, locked)
|
|
PAGEFLAG(Error, error) TESTCLEARFLAG(Error, error)
|
|
PAGEFLAG(Referenced, referenced) TESTCLEARFLAG(Referenced, referenced)
|
|
PAGEFLAG(Dirty, dirty) TESTSCFLAG(Dirty, dirty) __CLEARPAGEFLAG(Dirty, dirty)
|
|
PAGEFLAG(LRU, lru) __CLEARPAGEFLAG(LRU, lru)
|
|
PAGEFLAG(Active, active) __CLEARPAGEFLAG(Active, active)
|
|
TESTCLEARFLAG(Active, active)
|
|
__PAGEFLAG(Slab, slab)
|
|
PAGEFLAG(Checked, checked) /* Used by some filesystems */
|
|
PAGEFLAG(Pinned, pinned) TESTSCFLAG(Pinned, pinned) /* Xen */
|
|
PAGEFLAG(SavePinned, savepinned); /* Xen */
|
|
PAGEFLAG(Reserved, reserved) __CLEARPAGEFLAG(Reserved, reserved)
|
|
PAGEFLAG(SwapBacked, swapbacked) __CLEARPAGEFLAG(SwapBacked, swapbacked)
|
|
|
|
__PAGEFLAG(SlobFree, slob_free)
|
|
|
|
/*
|
|
* Private page markings that may be used by the filesystem that owns the page
|
|
* for its own purposes.
|
|
* - PG_private and PG_private_2 cause releasepage() and co to be invoked
|
|
*/
|
|
PAGEFLAG(Private, private) __SETPAGEFLAG(Private, private)
|
|
__CLEARPAGEFLAG(Private, private)
|
|
PAGEFLAG(Private2, private_2) TESTSCFLAG(Private2, private_2)
|
|
PAGEFLAG(OwnerPriv1, owner_priv_1) TESTCLEARFLAG(OwnerPriv1, owner_priv_1)
|
|
|
|
/*
|
|
* Only test-and-set exist for PG_writeback. The unconditional operators are
|
|
* risky: they bypass page accounting.
|
|
*/
|
|
TESTPAGEFLAG(Writeback, writeback) TESTSCFLAG(Writeback, writeback)
|
|
PAGEFLAG(MappedToDisk, mappedtodisk)
|
|
|
|
/* PG_readahead is only used for file reads; PG_reclaim is only for writes */
|
|
PAGEFLAG(Reclaim, reclaim) TESTCLEARFLAG(Reclaim, reclaim)
|
|
PAGEFLAG(Readahead, reclaim) /* Reminder to do async read-ahead */
|
|
|
|
#ifdef CONFIG_HIGHMEM
|
|
/*
|
|
* Must use a macro here due to header dependency issues. page_zone() is not
|
|
* available at this point.
|
|
*/
|
|
#define PageHighMem(__p) is_highmem(page_zone(__p))
|
|
#else
|
|
PAGEFLAG_FALSE(HighMem)
|
|
#endif
|
|
|
|
#ifdef CONFIG_SWAP
|
|
PAGEFLAG(SwapCache, swapcache)
|
|
#else
|
|
PAGEFLAG_FALSE(SwapCache)
|
|
SETPAGEFLAG_NOOP(SwapCache) CLEARPAGEFLAG_NOOP(SwapCache)
|
|
#endif
|
|
|
|
PAGEFLAG(Unevictable, unevictable) __CLEARPAGEFLAG(Unevictable, unevictable)
|
|
TESTCLEARFLAG(Unevictable, unevictable)
|
|
|
|
#ifdef CONFIG_MMU
|
|
PAGEFLAG(Mlocked, mlocked) __CLEARPAGEFLAG(Mlocked, mlocked)
|
|
TESTSCFLAG(Mlocked, mlocked) __TESTCLEARFLAG(Mlocked, mlocked)
|
|
#else
|
|
PAGEFLAG_FALSE(Mlocked) SETPAGEFLAG_NOOP(Mlocked)
|
|
TESTCLEARFLAG_FALSE(Mlocked) __TESTCLEARFLAG_FALSE(Mlocked)
|
|
#endif
|
|
|
|
#ifdef CONFIG_ARCH_USES_PG_UNCACHED
|
|
PAGEFLAG(Uncached, uncached)
|
|
#else
|
|
PAGEFLAG_FALSE(Uncached)
|
|
#endif
|
|
|
|
#ifdef CONFIG_MEMORY_FAILURE
|
|
PAGEFLAG(HWPoison, hwpoison)
|
|
TESTSCFLAG(HWPoison, hwpoison)
|
|
#define __PG_HWPOISON (1UL << PG_hwpoison)
|
|
#else
|
|
PAGEFLAG_FALSE(HWPoison)
|
|
#define __PG_HWPOISON 0
|
|
#endif
|
|
|
|
u64 stable_page_flags(struct page *page);
|
|
|
|
static inline int PageUptodate(struct page *page)
|
|
{
|
|
int ret = test_bit(PG_uptodate, &(page)->flags);
|
|
|
|
/*
|
|
* Must ensure that the data we read out of the page is loaded
|
|
* _after_ we've loaded page->flags to check for PageUptodate.
|
|
* We can skip the barrier if the page is not uptodate, because
|
|
* we wouldn't be reading anything from it.
|
|
*
|
|
* See SetPageUptodate() for the other side of the story.
|
|
*/
|
|
if (ret)
|
|
smp_rmb();
|
|
|
|
return ret;
|
|
}
|
|
|
|
static inline void __SetPageUptodate(struct page *page)
|
|
{
|
|
smp_wmb();
|
|
__set_bit(PG_uptodate, &(page)->flags);
|
|
}
|
|
|
|
static inline void SetPageUptodate(struct page *page)
|
|
{
|
|
#ifdef CONFIG_S390
|
|
if (!test_and_set_bit(PG_uptodate, &page->flags))
|
|
page_set_storage_key(page_to_phys(page), PAGE_DEFAULT_KEY, 0);
|
|
#else
|
|
/*
|
|
* Memory barrier must be issued before setting the PG_uptodate bit,
|
|
* so that all previous stores issued in order to bring the page
|
|
* uptodate are actually visible before PageUptodate becomes true.
|
|
*
|
|
* s390 doesn't need an explicit smp_wmb here because the test and
|
|
* set bit already provides full barriers.
|
|
*/
|
|
smp_wmb();
|
|
set_bit(PG_uptodate, &(page)->flags);
|
|
#endif
|
|
}
|
|
|
|
CLEARPAGEFLAG(Uptodate, uptodate)
|
|
|
|
extern void cancel_dirty_page(struct page *page, unsigned int account_size);
|
|
|
|
int test_clear_page_writeback(struct page *page);
|
|
int test_set_page_writeback(struct page *page);
|
|
|
|
static inline void set_page_writeback(struct page *page)
|
|
{
|
|
test_set_page_writeback(page);
|
|
}
|
|
|
|
#ifdef CONFIG_PAGEFLAGS_EXTENDED
|
|
/*
|
|
* System with lots of page flags available. This allows separate
|
|
* flags for PageHead() and PageTail() checks of compound pages so that bit
|
|
* tests can be used in performance sensitive paths. PageCompound is
|
|
* generally not used in hot code paths.
|
|
*/
|
|
__PAGEFLAG(Head, head) CLEARPAGEFLAG(Head, head)
|
|
__PAGEFLAG(Tail, tail)
|
|
|
|
static inline int PageCompound(struct page *page)
|
|
{
|
|
return page->flags & ((1L << PG_head) | (1L << PG_tail));
|
|
|
|
}
|
|
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
|
|
static inline void ClearPageCompound(struct page *page)
|
|
{
|
|
BUG_ON(!PageHead(page));
|
|
ClearPageHead(page);
|
|
}
|
|
#endif
|
|
#else
|
|
/*
|
|
* Reduce page flag use as much as possible by overlapping
|
|
* compound page flags with the flags used for page cache pages. Possible
|
|
* because PageCompound is always set for compound pages and not for
|
|
* pages on the LRU and/or pagecache.
|
|
*/
|
|
TESTPAGEFLAG(Compound, compound)
|
|
__PAGEFLAG(Head, compound)
|
|
|
|
/*
|
|
* PG_reclaim is used in combination with PG_compound to mark the
|
|
* head and tail of a compound page. This saves one page flag
|
|
* but makes it impossible to use compound pages for the page cache.
|
|
* The PG_reclaim bit would have to be used for reclaim or readahead
|
|
* if compound pages enter the page cache.
|
|
*
|
|
* PG_compound & PG_reclaim => Tail page
|
|
* PG_compound & ~PG_reclaim => Head page
|
|
*/
|
|
#define PG_head_tail_mask ((1L << PG_compound) | (1L << PG_reclaim))
|
|
|
|
static inline int PageTail(struct page *page)
|
|
{
|
|
return ((page->flags & PG_head_tail_mask) == PG_head_tail_mask);
|
|
}
|
|
|
|
static inline void __SetPageTail(struct page *page)
|
|
{
|
|
page->flags |= PG_head_tail_mask;
|
|
}
|
|
|
|
static inline void __ClearPageTail(struct page *page)
|
|
{
|
|
page->flags &= ~PG_head_tail_mask;
|
|
}
|
|
|
|
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
|
|
static inline void ClearPageCompound(struct page *page)
|
|
{
|
|
BUG_ON((page->flags & PG_head_tail_mask) != (1 << PG_compound));
|
|
clear_bit(PG_compound, &page->flags);
|
|
}
|
|
#endif
|
|
|
|
#endif /* !PAGEFLAGS_EXTENDED */
|
|
|
|
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
|
|
/*
|
|
* PageHuge() only returns true for hugetlbfs pages, but not for
|
|
* normal or transparent huge pages.
|
|
*
|
|
* PageTransHuge() returns true for both transparent huge and
|
|
* hugetlbfs pages, but not normal pages. PageTransHuge() can only be
|
|
* called only in the core VM paths where hugetlbfs pages can't exist.
|
|
*/
|
|
static inline int PageTransHuge(struct page *page)
|
|
{
|
|
VM_BUG_ON(PageTail(page));
|
|
return PageHead(page);
|
|
}
|
|
|
|
/*
|
|
* PageTransCompound returns true for both transparent huge pages
|
|
* and hugetlbfs pages, so it should only be called when it's known
|
|
* that hugetlbfs pages aren't involved.
|
|
*/
|
|
static inline int PageTransCompound(struct page *page)
|
|
{
|
|
return PageCompound(page);
|
|
}
|
|
|
|
/*
|
|
* PageTransTail returns true for both transparent huge pages
|
|
* and hugetlbfs pages, so it should only be called when it's known
|
|
* that hugetlbfs pages aren't involved.
|
|
*/
|
|
static inline int PageTransTail(struct page *page)
|
|
{
|
|
return PageTail(page);
|
|
}
|
|
|
|
#else
|
|
|
|
static inline int PageTransHuge(struct page *page)
|
|
{
|
|
return 0;
|
|
}
|
|
|
|
static inline int PageTransCompound(struct page *page)
|
|
{
|
|
return 0;
|
|
}
|
|
|
|
static inline int PageTransTail(struct page *page)
|
|
{
|
|
return 0;
|
|
}
|
|
#endif
|
|
|
|
/*
|
|
* If network-based swap is enabled, sl*b must keep track of whether pages
|
|
* were allocated from pfmemalloc reserves.
|
|
*/
|
|
static inline int PageSlabPfmemalloc(struct page *page)
|
|
{
|
|
VM_BUG_ON(!PageSlab(page));
|
|
return PageActive(page);
|
|
}
|
|
|
|
static inline void SetPageSlabPfmemalloc(struct page *page)
|
|
{
|
|
VM_BUG_ON(!PageSlab(page));
|
|
SetPageActive(page);
|
|
}
|
|
|
|
static inline void __ClearPageSlabPfmemalloc(struct page *page)
|
|
{
|
|
VM_BUG_ON(!PageSlab(page));
|
|
__ClearPageActive(page);
|
|
}
|
|
|
|
static inline void ClearPageSlabPfmemalloc(struct page *page)
|
|
{
|
|
VM_BUG_ON(!PageSlab(page));
|
|
ClearPageActive(page);
|
|
}
|
|
|
|
#ifdef CONFIG_MMU
|
|
#define __PG_MLOCKED (1 << PG_mlocked)
|
|
#else
|
|
#define __PG_MLOCKED 0
|
|
#endif
|
|
|
|
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
|
|
#define __PG_COMPOUND_LOCK (1 << PG_compound_lock)
|
|
#else
|
|
#define __PG_COMPOUND_LOCK 0
|
|
#endif
|
|
|
|
/*
|
|
* Flags checked when a page is freed. Pages being freed should not have
|
|
* these flags set. It they are, there is a problem.
|
|
*/
|
|
#define PAGE_FLAGS_CHECK_AT_FREE \
|
|
(1 << PG_lru | 1 << PG_locked | \
|
|
1 << PG_private | 1 << PG_private_2 | \
|
|
1 << PG_writeback | 1 << PG_reserved | \
|
|
1 << PG_slab | 1 << PG_swapcache | 1 << PG_active | \
|
|
1 << PG_unevictable | __PG_MLOCKED | __PG_HWPOISON | \
|
|
__PG_COMPOUND_LOCK)
|
|
|
|
/*
|
|
* Flags checked when a page is prepped for return by the page allocator.
|
|
* Pages being prepped should not have any flags set. It they are set,
|
|
* there has been a kernel bug or struct page corruption.
|
|
*/
|
|
#define PAGE_FLAGS_CHECK_AT_PREP ((1 << NR_PAGEFLAGS) - 1)
|
|
|
|
#define PAGE_FLAGS_PRIVATE \
|
|
(1 << PG_private | 1 << PG_private_2)
|
|
/**
|
|
* page_has_private - Determine if page has private stuff
|
|
* @page: The page to be checked
|
|
*
|
|
* Determine if a page has private stuff, indicating that release routines
|
|
* should be invoked upon it.
|
|
*/
|
|
static inline int page_has_private(struct page *page)
|
|
{
|
|
return !!(page->flags & PAGE_FLAGS_PRIVATE);
|
|
}
|
|
|
|
#endif /* !__GENERATING_BOUNDS_H */
|
|
|
|
#endif /* PAGE_FLAGS_H */
|