Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 1 | .. _unevictable_lru: |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 2 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 3 | ============================== |
| 4 | Unevictable LRU Infrastructure |
| 5 | ============================== |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 6 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 7 | .. contents:: :local: |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 8 | |
| 9 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 10 | Introduction |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 11 | ============ |
| 12 | |
| 13 | This document describes the Linux memory manager's "Unevictable LRU" |
| 14 | infrastructure and the use of this to manage several types of "unevictable" |
| 15 | pages. |
| 16 | |
| 17 | The document attempts to provide the overall rationale behind this mechanism |
| 18 | and the rationale for some of the design decisions that drove the |
| 19 | implementation. The latter design rationale is discussed in the context of an |
| 20 | implementation description. Admittedly, one can obtain the implementation |
| 21 | details - the "what does it do?" - by reading the code. One hopes that the |
| 22 | descriptions below add value by provide the answer to "why does it do that?". |
| 23 | |
| 24 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 25 | |
| 26 | The Unevictable LRU |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 27 | =================== |
| 28 | |
| 29 | The Unevictable LRU facility adds an additional LRU list to track unevictable |
| 30 | pages and to hide these pages from vmscan. This mechanism is based on a patch |
| 31 | by Larry Woodman of Red Hat to address several scalability problems with page |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 32 | reclaim in Linux. The problems have been observed at customer sites on large |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 33 | memory x86_64 systems. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 34 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 35 | To illustrate this with an example, a non-NUMA x86_64 platform with 128GB of |
Hugh Dickins | 15b4473 | 2020-12-15 14:21:31 -0800 | [diff] [blame] | 36 | main memory will have over 32 million 4k pages in a single node. When a large |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 37 | fraction of these pages are not evictable for any reason [see below], vmscan |
| 38 | will spend a lot of time scanning the LRU lists looking for the small fraction |
| 39 | of pages that are evictable. This can result in a situation where all CPUs are |
| 40 | spending 100% of their time in vmscan for hours or days on end, with the system |
| 41 | completely unresponsive. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 42 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 43 | The unevictable list addresses the following classes of unevictable pages: |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 44 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 45 | * Those owned by ramfs. |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 46 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 47 | * Those mapped into SHM_LOCK'd shared memory regions. |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 48 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 49 | * Those mapped into VM_LOCKED [mlock()ed] VMAs. |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 50 | |
| 51 | The infrastructure may also be able to handle other conditions that make pages |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 52 | unevictable, either by definition or by circumstance, in the future. |
| 53 | |
| 54 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 55 | The Unevictable Page List |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 56 | ------------------------- |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 57 | |
Hugh Dickins | 15b4473 | 2020-12-15 14:21:31 -0800 | [diff] [blame] | 58 | The Unevictable LRU infrastructure consists of an additional, per-node, LRU list |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 59 | called the "unevictable" list and an associated page flag, PG_unevictable, to |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 60 | indicate that the page is being managed on the unevictable list. |
| 61 | |
| 62 | The PG_unevictable flag is analogous to, and mutually exclusive with, the |
| 63 | PG_active flag in that it indicates on which LRU list a page resides when |
Michal Hocko | e6e8dd5 | 2011-03-16 15:01:37 +0100 | [diff] [blame] | 64 | PG_lru is set. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 65 | |
| 66 | The Unevictable LRU infrastructure maintains unevictable pages on an additional |
| 67 | LRU list for a few reasons: |
| 68 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 69 | (1) We get to "treat unevictable pages just like we treat other pages in the |
| 70 | system - which means we get to use the same code to manipulate them, the |
| 71 | same code to isolate them (for migrate, etc.), the same code to keep track |
| 72 | of the statistics, etc..." [Rik van Riel] |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 73 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 74 | (2) We want to be able to migrate unevictable pages between nodes for memory |
| 75 | defragmentation, workload management and memory hotplug. The linux kernel |
| 76 | can only migrate pages that it can successfully isolate from the LRU |
| 77 | lists. If we were to maintain pages elsewhere than on an LRU-like list, |
| 78 | where they can be found by isolate_lru_page(), we would prevent their |
| 79 | migration, unless we reworked migration code to find the unevictable pages |
| 80 | itself. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 81 | |
| 82 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 83 | The unevictable list does not differentiate between file-backed and anonymous, |
| 84 | swap-backed pages. This differentiation is only important while the pages are, |
| 85 | in fact, evictable. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 86 | |
Hugh Dickins | 15b4473 | 2020-12-15 14:21:31 -0800 | [diff] [blame] | 87 | The unevictable list benefits from the "arrayification" of the per-node LRU |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 88 | lists and statistics originally proposed and posted by Christoph Lameter. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 89 | |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 90 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 91 | Memory Control Group Interaction |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 92 | -------------------------------- |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 93 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 94 | The unevictable LRU facility interacts with the memory control group [aka |
Mauro Carvalho Chehab | da82c92 | 2019-06-27 13:08:35 -0300 | [diff] [blame] | 95 | memory controller; see Documentation/admin-guide/cgroup-v1/memory.rst] by extending the |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 96 | lru_list enum. |
| 97 | |
Hugh Dickins | 15b4473 | 2020-12-15 14:21:31 -0800 | [diff] [blame] | 98 | The memory controller data structure automatically gets a per-node unevictable |
| 99 | list as a result of the "arrayification" of the per-node LRU lists (one per |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 100 | lru_list enum element). The memory controller tracks the movement of pages to |
| 101 | and from the unevictable list. |
| 102 | |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 103 | When a memory control group comes under memory pressure, the controller will |
| 104 | not attempt to reclaim pages on the unevictable list. This has a couple of |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 105 | effects: |
| 106 | |
| 107 | (1) Because the pages are "hidden" from reclaim on the unevictable list, the |
| 108 | reclaim process can be more efficient, dealing only with pages that have a |
| 109 | chance of being reclaimed. |
| 110 | |
| 111 | (2) On the other hand, if too many of the pages charged to the control group |
| 112 | are unevictable, the evictable portion of the working set of the tasks in |
| 113 | the control group may not fit into the available memory. This can cause |
| 114 | the control group to thrash or to OOM-kill tasks. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 115 | |
| 116 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 117 | .. _mark_addr_space_unevict: |
| 118 | |
| 119 | Marking Address Spaces Unevictable |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 120 | ---------------------------------- |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 121 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 122 | For facilities such as ramfs none of the pages attached to the address space |
| 123 | may be evicted. To prevent eviction of any such pages, the AS_UNEVICTABLE |
| 124 | address space flag is provided, and this can be manipulated by a filesystem |
| 125 | using a number of wrapper functions: |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 126 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 127 | * ``void mapping_set_unevictable(struct address_space *mapping);`` |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 128 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 129 | Mark the address space as being completely unevictable. |
| 130 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 131 | * ``void mapping_clear_unevictable(struct address_space *mapping);`` |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 132 | |
| 133 | Mark the address space as being evictable. |
| 134 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 135 | * ``int mapping_unevictable(struct address_space *mapping);`` |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 136 | |
| 137 | Query the address space, and return true if it is completely |
| 138 | unevictable. |
| 139 | |
Kuo-Hsin Yang | 64e3d12 | 2018-11-06 13:23:24 +0000 | [diff] [blame] | 140 | These are currently used in three places in the kernel: |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 141 | |
| 142 | (1) By ramfs to mark the address spaces of its inodes when they are created, |
| 143 | and this mark remains for the life of the inode. |
| 144 | |
| 145 | (2) By SYSV SHM to mark SHM_LOCK'd address spaces until SHM_UNLOCK is called. |
| 146 | |
| 147 | Note that SHM_LOCK is not required to page in the locked pages if they're |
| 148 | swapped out; the application must touch the pages manually if it wants to |
| 149 | ensure they're in memory. |
| 150 | |
Kuo-Hsin Yang | 64e3d12 | 2018-11-06 13:23:24 +0000 | [diff] [blame] | 151 | (3) By the i915 driver to mark pinned address space until it's unpinned. The |
| 152 | amount of unevictable memory marked by i915 driver is roughly the bounded |
| 153 | object size in debugfs/dri/0/i915_gem_objects. |
| 154 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 155 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 156 | Detecting Unevictable Pages |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 157 | --------------------------- |
| 158 | |
| 159 | The function page_evictable() in vmscan.c determines whether a page is |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 160 | evictable or not using the query function outlined above [see section |
| 161 | :ref:`Marking address spaces unevictable <mark_addr_space_unevict>`] |
| 162 | to check the AS_UNEVICTABLE flag. |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 163 | |
| 164 | For address spaces that are so marked after being populated (as SHM regions |
| 165 | might be), the lock action (eg: SHM_LOCK) can be lazy, and need not populate |
| 166 | the page tables for the region as does, for example, mlock(), nor need it make |
| 167 | any special effort to push any pages in the SHM_LOCK'd area to the unevictable |
| 168 | list. Instead, vmscan will do this if and when it encounters the pages during |
| 169 | a reclamation scan. |
| 170 | |
| 171 | On an unlock action (such as SHM_UNLOCK), the unlocker (eg: shmctl()) must scan |
| 172 | the pages in the region and "rescue" them from the unevictable list if no other |
| 173 | condition is keeping them unevictable. If an unevictable region is destroyed, |
| 174 | the pages are also "rescued" from the unevictable list in the process of |
| 175 | freeing them. |
| 176 | |
| 177 | page_evictable() also checks for mlocked pages by testing an additional page |
Hugh Dickins | 39b5f29 | 2012-10-08 16:33:18 -0700 | [diff] [blame] | 178 | flag, PG_mlocked (as wrapped by PageMlocked()), which is set when a page is |
| 179 | faulted into a VM_LOCKED vma, or found in a vma being VM_LOCKED. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 180 | |
| 181 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 182 | Vmscan's Handling of Unevictable Pages |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 183 | -------------------------------------- |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 184 | |
| 185 | If unevictable pages are culled in the fault path, or moved to the unevictable |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 186 | list at mlock() or mmap() time, vmscan will not encounter the pages until they |
| 187 | have become evictable again (via munlock() for example) and have been "rescued" |
| 188 | from the unevictable list. However, there may be situations where we decide, |
| 189 | for the sake of expediency, to leave a unevictable page on one of the regular |
| 190 | active/inactive LRU lists for vmscan to deal with. vmscan checks for such |
| 191 | pages in all of the shrink_{active|inactive|page}_list() functions and will |
| 192 | "cull" such pages that it encounters: that is, it diverts those pages to the |
Hugh Dickins | 15b4473 | 2020-12-15 14:21:31 -0800 | [diff] [blame] | 193 | unevictable list for the node being scanned. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 194 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 195 | There may be situations where a page is mapped into a VM_LOCKED VMA, but the |
| 196 | page is not marked as PG_mlocked. Such pages will make it all the way to |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 197 | shrink_page_list() where they will be detected when vmscan walks the reverse |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 198 | map in try_to_unmap(). If try_to_unmap() returns SWAP_MLOCK, |
| 199 | shrink_page_list() will cull the page at that point. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 200 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 201 | To "cull" an unevictable page, vmscan simply puts the page back on the LRU list |
| 202 | using putback_lru_page() - the inverse operation to isolate_lru_page() - after |
| 203 | dropping the page lock. Because the condition which makes the page unevictable |
| 204 | may change once the page is unlocked, putback_lru_page() will recheck the |
| 205 | unevictable state of a page that it places on the unevictable list. If the |
| 206 | page has become unevictable, putback_lru_page() removes it from the list and |
| 207 | retries, including the page_unevictable() test. Because such a race is a rare |
| 208 | event and movement of pages onto the unevictable list should be rare, these |
| 209 | extra evictabilty checks should not occur in the majority of calls to |
| 210 | putback_lru_page(). |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 211 | |
| 212 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 213 | MLOCKED Pages |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 214 | ============= |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 215 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 216 | The unevictable page list is also useful for mlock(), in addition to ramfs and |
| 217 | SYSV SHM. Note that mlock() is only available in CONFIG_MMU=y situations; in |
| 218 | NOMMU situations, all mappings are effectively mlocked. |
| 219 | |
| 220 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 221 | History |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 222 | ------- |
| 223 | |
| 224 | The "Unevictable mlocked Pages" infrastructure is based on work originally |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 225 | posted by Nick Piggin in an RFC patch entitled "mm: mlocked pages off LRU". |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 226 | Nick posted his patch as an alternative to a patch posted by Christoph Lameter |
| 227 | to achieve the same objective: hiding mlocked pages from vmscan. |
| 228 | |
| 229 | In Nick's patch, he used one of the struct page LRU list link fields as a count |
| 230 | of VM_LOCKED VMAs that map the page. This use of the link field for a count |
| 231 | prevented the management of the pages on an LRU list, and thus mlocked pages |
| 232 | were not migratable as isolate_lru_page() could not find them, and the LRU list |
| 233 | link field was not available to the migration subsystem. |
| 234 | |
| 235 | Nick resolved this by putting mlocked pages back on the lru list before |
| 236 | attempting to isolate them, thus abandoning the count of VM_LOCKED VMAs. When |
| 237 | Nick's patch was integrated with the Unevictable LRU work, the count was |
| 238 | replaced by walking the reverse map to determine whether any VM_LOCKED VMAs |
| 239 | mapped the page. More on this below. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 240 | |
| 241 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 242 | Basic Management |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 243 | ---------------- |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 244 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 245 | mlocked pages - pages mapped into a VM_LOCKED VMA - are a class of unevictable |
| 246 | pages. When such a page has been "noticed" by the memory management subsystem, |
| 247 | the page is marked with the PG_mlocked flag. This can be manipulated using the |
| 248 | PageMlocked() functions. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 249 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 250 | A PG_mlocked page will be placed on the unevictable list when it is added to |
| 251 | the LRU. Such pages can be "noticed" by memory management in several places: |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 252 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 253 | (1) in the mlock()/mlockall() system call handlers; |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 254 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 255 | (2) in the mmap() system call handler when mmapping a region with the |
| 256 | MAP_LOCKED flag; |
| 257 | |
| 258 | (3) mmapping a region in a task that has called mlockall() with the MCL_FUTURE |
| 259 | flag |
| 260 | |
| 261 | (4) in the fault path, if mlocked pages are "culled" in the fault path, |
| 262 | and when a VM_LOCKED stack segment is expanded; or |
| 263 | |
| 264 | (5) as mentioned above, in vmscan:shrink_page_list() when attempting to |
| 265 | reclaim a page in a VM_LOCKED VMA via try_to_unmap() |
| 266 | |
| 267 | all of which result in the VM_LOCKED flag being set for the VMA if it doesn't |
| 268 | already have it set. |
| 269 | |
| 270 | mlocked pages become unlocked and rescued from the unevictable list when: |
| 271 | |
| 272 | (1) mapped in a range unlocked via the munlock()/munlockall() system calls; |
| 273 | |
| 274 | (2) munmap()'d out of the last VM_LOCKED VMA that maps the page, including |
| 275 | unmapping at task exit; |
| 276 | |
| 277 | (3) when the page is truncated from the last VM_LOCKED VMA of an mmapped file; |
| 278 | or |
| 279 | |
| 280 | (4) before a page is COW'd in a VM_LOCKED VMA. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 281 | |
| 282 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 283 | mlock()/mlockall() System Call Handling |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 284 | --------------------------------------- |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 285 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 286 | Both [do\_]mlock() and [do\_]mlockall() system call handlers call mlock_fixup() |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 287 | for each VMA in the range specified by the call. In the case of mlockall(), |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 288 | this is the entire active address space of the task. Note that mlock_fixup() |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 289 | is used for both mlocking and munlocking a range of memory. A call to mlock() |
| 290 | an already VM_LOCKED VMA, or to munlock() a VMA that is not VM_LOCKED is |
| 291 | treated as a no-op, and mlock_fixup() simply returns. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 292 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 293 | If the VMA passes some filtering as described in "Filtering Special Vmas" |
| 294 | below, mlock_fixup() will attempt to merge the VMA with its neighbors or split |
| 295 | off a subset of the VMA if the range does not cover the entire VMA. Once the |
| 296 | VMA has been merged or split or neither, mlock_fixup() will call |
Kirill A. Shutemov | fc05f56 | 2015-04-14 15:44:39 -0700 | [diff] [blame] | 297 | populate_vma_page_range() to fault in the pages via get_user_pages() and to |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 298 | mark the pages as mlocked via mlock_vma_page(). |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 299 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 300 | Note that the VMA being mlocked might be mapped with PROT_NONE. In this case, |
| 301 | get_user_pages() will be unable to fault in the pages. That's okay. If pages |
| 302 | do end up getting faulted into this VM_LOCKED VMA, we'll handle them in the |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 303 | fault path or in vmscan. |
| 304 | |
| 305 | Also note that a page returned by get_user_pages() could be truncated or |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 306 | migrated out from under us, while we're trying to mlock it. To detect this, |
Kirill A. Shutemov | fc05f56 | 2015-04-14 15:44:39 -0700 | [diff] [blame] | 307 | populate_vma_page_range() checks page_mapping() after acquiring the page lock. |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 308 | If the page is still associated with its mapping, we'll go ahead and call |
| 309 | mlock_vma_page(). If the mapping is gone, we just unlock the page and move on. |
| 310 | In the worst case, this will result in a page mapped in a VM_LOCKED VMA |
| 311 | remaining on a normal LRU list without being PageMlocked(). Again, vmscan will |
| 312 | detect and cull such pages. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 313 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 314 | mlock_vma_page() will call TestSetPageMlocked() for each page returned by |
| 315 | get_user_pages(). We use TestSetPageMlocked() because the page might already |
| 316 | be mlocked by another task/VMA and we don't want to do extra work. We |
| 317 | especially do not want to count an mlocked page more than once in the |
| 318 | statistics. If the page was already mlocked, mlock_vma_page() need do nothing |
| 319 | more. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 320 | |
| 321 | If the page was NOT already mlocked, mlock_vma_page() attempts to isolate the |
| 322 | page from the LRU, as it is likely on the appropriate active or inactive list |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 323 | at that time. If the isolate_lru_page() succeeds, mlock_vma_page() will put |
| 324 | back the page - by calling putback_lru_page() - which will notice that the page |
Hugh Dickins | 15b4473 | 2020-12-15 14:21:31 -0800 | [diff] [blame] | 325 | is now mlocked and divert the page to the node's unevictable list. If |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 326 | mlock_vma_page() is unable to isolate the page from the LRU, vmscan will handle |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 327 | it later if and when it attempts to reclaim the page. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 328 | |
| 329 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 330 | Filtering Special VMAs |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 331 | ---------------------- |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 332 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 333 | mlock_fixup() filters several classes of "special" VMAs: |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 334 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 335 | 1) VMAs with VM_IO or VM_PFNMAP set are skipped entirely. The pages behind |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 336 | these mappings are inherently pinned, so we don't need to mark them as |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 337 | mlocked. In any case, most of the pages have no struct page in which to so |
| 338 | mark the page. Because of this, get_user_pages() will fail for these VMAs, |
| 339 | so there is no sense in attempting to visit them. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 340 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 341 | 2) VMAs mapping hugetlbfs page are already effectively pinned into memory. We |
| 342 | neither need nor want to mlock() these pages. However, to preserve the |
| 343 | prior behavior of mlock() - before the unevictable/mlock changes - |
| 344 | mlock_fixup() will call make_pages_present() in the hugetlbfs VMA range to |
| 345 | allocate the huge pages and populate the ptes. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 346 | |
Konstantin Khlebnikov | 314e51b | 2012-10-08 16:29:02 -0700 | [diff] [blame] | 347 | 3) VMAs with VM_DONTEXPAND are generally userspace mappings of kernel pages, |
| 348 | such as the VDSO page, relay channel pages, etc. These pages |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 349 | are inherently unevictable and are not managed on the LRU lists. |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 350 | mlock_fixup() treats these VMAs the same as hugetlbfs VMAs. It calls |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 351 | make_pages_present() to populate the ptes. |
| 352 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 353 | Note that for all of these special VMAs, mlock_fixup() does not set the |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 354 | VM_LOCKED flag. Therefore, we won't have to deal with them later during |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 355 | munlock(), munmap() or task exit. Neither does mlock_fixup() account these |
| 356 | VMAs against the task's "locked_vm". |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 357 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 358 | .. _munlock_munlockall_handling: |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 359 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 360 | munlock()/munlockall() System Call Handling |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 361 | ------------------------------------------- |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 362 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 363 | The munlock() and munlockall() system calls are handled by the same functions - |
| 364 | do_mlock[all]() - as the mlock() and mlockall() system calls with the unlock vs |
| 365 | lock operation indicated by an argument. So, these system calls are also |
| 366 | handled by mlock_fixup(). Again, if called for an already munlocked VMA, |
| 367 | mlock_fixup() simply returns. Because of the VMA filtering discussed above, |
| 368 | VM_LOCKED will not be set in any "special" VMAs. So, these VMAs will be |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 369 | ignored for munlock. |
| 370 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 371 | If the VMA is VM_LOCKED, mlock_fixup() again attempts to merge or split off the |
| 372 | specified range. The range is then munlocked via the function |
Kirill A. Shutemov | fc05f56 | 2015-04-14 15:44:39 -0700 | [diff] [blame] | 373 | populate_vma_page_range() - the same function used to mlock a VMA range - |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 374 | passing a flag to indicate that munlock() is being performed. |
| 375 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 376 | Because the VMA access protections could have been changed to PROT_NONE after |
Hugh Dickins | 63d6c5a | 2009-01-06 14:39:38 -0800 | [diff] [blame] | 377 | faulting in and mlocking pages, get_user_pages() was unreliable for visiting |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 378 | these pages for munlocking. Because we don't want to leave pages mlocked, |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 379 | get_user_pages() was enhanced to accept a flag to ignore the permissions when |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 380 | fetching the pages - all of which should be resident as a result of previous |
| 381 | mlocking. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 382 | |
Kirill A. Shutemov | fc05f56 | 2015-04-14 15:44:39 -0700 | [diff] [blame] | 383 | For munlock(), populate_vma_page_range() unlocks individual pages by calling |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 384 | munlock_vma_page(). munlock_vma_page() unconditionally clears the PG_mlocked |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 385 | flag using TestClearPageMlocked(). As with mlock_vma_page(), |
| 386 | munlock_vma_page() use the Test*PageMlocked() function to handle the case where |
| 387 | the page might have already been unlocked by another task. If the page was |
| 388 | mlocked, munlock_vma_page() updates that zone statistics for the number of |
| 389 | mlocked pages. Note, however, that at this point we haven't checked whether |
| 390 | the page is mapped by other VM_LOCKED VMAs. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 391 | |
Alistair Popple | cd62734 | 2021-06-30 18:54:12 -0700 | [diff] [blame] | 392 | We can't call page_mlock(), the function that walks the reverse map to |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 393 | check for other VM_LOCKED VMAs, without first isolating the page from the LRU. |
Alistair Popple | cd62734 | 2021-06-30 18:54:12 -0700 | [diff] [blame] | 394 | page_mlock() is a variant of try_to_unmap() and thus requires that the page |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 395 | not be on an LRU list [more on these below]. However, the call to |
Alistair Popple | cd62734 | 2021-06-30 18:54:12 -0700 | [diff] [blame] | 396 | isolate_lru_page() could fail, in which case we can't call page_mlock(). So, |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 397 | we go ahead and clear PG_mlocked up front, as this might be the only chance we |
Alistair Popple | cd62734 | 2021-06-30 18:54:12 -0700 | [diff] [blame] | 398 | have. If we can successfully isolate the page, we go ahead and call |
| 399 | page_mlock(), which will restore the PG_mlocked flag and update the zone |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 400 | page statistics if it finds another VMA holding the page mlocked. If we fail |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 401 | to isolate the page, we'll have left a potentially mlocked page on the LRU. |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 402 | This is fine, because we'll catch it later if and if vmscan tries to reclaim |
| 403 | the page. This should be relatively rare. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 404 | |
| 405 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 406 | Migrating MLOCKED Pages |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 407 | ----------------------- |
| 408 | |
| 409 | A page that is being migrated has been isolated from the LRU lists and is held |
| 410 | locked across unmapping of the page, updating the page's address space entry |
| 411 | and copying the contents and state, until the page table entry has been |
| 412 | replaced with an entry that refers to the new page. Linux supports migration |
| 413 | of mlocked pages and other unevictable pages. This involves simply moving the |
| 414 | PG_mlocked and PG_unevictable states from the old page to the new page. |
| 415 | |
| 416 | Note that page migration can race with mlocking or munlocking of the same page. |
| 417 | This has been discussed from the mlock/munlock perspective in the respective |
| 418 | sections above. Both processes (migration and m[un]locking) hold the page |
| 419 | locked. This provides the first level of synchronization. Page migration |
| 420 | zeros out the page_mapping of the old page before unlocking it, so m[un]lock |
| 421 | can skip these pages by testing the page mapping under page lock. |
| 422 | |
| 423 | To complete page migration, we place the new and old pages back onto the LRU |
| 424 | after dropping the page lock. The "unneeded" page - old page on success, new |
| 425 | page on failure - will be freed when the reference count held by the migration |
| 426 | process is released. To ensure that we don't strand pages on the unevictable |
| 427 | list because of a race between munlock and migration, page migration uses the |
| 428 | putback_lru_page() function to add migrated pages back to the LRU. |
| 429 | |
| 430 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 431 | Compacting MLOCKED Pages |
Eric B Munson | 922c055 | 2015-04-15 16:13:23 -0700 | [diff] [blame] | 432 | ------------------------ |
| 433 | |
| 434 | The unevictable LRU can be scanned for compactable regions and the default |
| 435 | behavior is to do so. /proc/sys/vm/compact_unevictable_allowed controls |
Mauro Carvalho Chehab | 5704324 | 2019-04-22 16:48:00 -0300 | [diff] [blame] | 436 | this behavior (see Documentation/admin-guide/sysctl/vm.rst). Once scanning of the |
Eric B Munson | 922c055 | 2015-04-15 16:13:23 -0700 | [diff] [blame] | 437 | unevictable LRU is enabled, the work of compaction is mostly handled by |
| 438 | the page migration code and the same work flow as described in MIGRATING |
| 439 | MLOCKED PAGES will apply. |
| 440 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 441 | MLOCKING Transparent Huge Pages |
Kirill A. Shutemov | 6fb8ddf | 2016-07-26 15:25:15 -0700 | [diff] [blame] | 442 | ------------------------------- |
| 443 | |
| 444 | A transparent huge page is represented by a single entry on an LRU list. |
| 445 | Therefore, we can only make unevictable an entire compound page, not |
| 446 | individual subpages. |
| 447 | |
| 448 | If a user tries to mlock() part of a huge page, we want the rest of the |
| 449 | page to be reclaimable. |
| 450 | |
| 451 | We cannot just split the page on partial mlock() as split_huge_page() can |
| 452 | fail and new intermittent failure mode for the syscall is undesirable. |
| 453 | |
| 454 | We handle this by keeping PTE-mapped huge pages on normal LRU lists: the |
| 455 | PMD on border of VM_LOCKED VMA will be split into PTE table. |
| 456 | |
| 457 | This way the huge page is accessible for vmscan. Under memory pressure the |
| 458 | page will be split, subpages which belong to VM_LOCKED VMAs will be moved |
| 459 | to unevictable LRU and the rest can be reclaimed. |
| 460 | |
| 461 | See also comment in follow_trans_huge_pmd(). |
Eric B Munson | 922c055 | 2015-04-15 16:13:23 -0700 | [diff] [blame] | 462 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 463 | mmap(MAP_LOCKED) System Call Handling |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 464 | ------------------------------------- |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 465 | |
Masanari Iida | df5cbb2 | 2014-03-21 10:04:30 +0900 | [diff] [blame] | 466 | In addition the mlock()/mlockall() system calls, an application can request |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 467 | that a region of memory be mlocked supplying the MAP_LOCKED flag to the mmap() |
Michal Hocko | 9b012a2 | 2015-06-24 16:57:50 -0700 | [diff] [blame] | 468 | call. There is one important and subtle difference here, though. mmap() + mlock() |
| 469 | will fail if the range cannot be faulted in (e.g. because mm_populate fails) |
| 470 | and returns with ENOMEM while mmap(MAP_LOCKED) will not fail. The mmaped |
| 471 | area will still have properties of the locked area - aka. pages will not get |
| 472 | swapped out - but major page faults to fault memory in might still happen. |
| 473 | |
| 474 | Furthermore, any mmap() call or brk() call that expands the heap by a |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 475 | task that has previously called mlockall() with the MCL_FUTURE flag will result |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 476 | in the newly mapped memory being mlocked. Before the unevictable/mlock |
| 477 | changes, the kernel simply called make_pages_present() to allocate pages and |
| 478 | populate the page table. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 479 | |
| 480 | To mlock a range of memory under the unevictable/mlock infrastructure, the |
| 481 | mmap() handler and task address space expansion functions call |
Kirill A. Shutemov | fc05f56 | 2015-04-14 15:44:39 -0700 | [diff] [blame] | 482 | populate_vma_page_range() specifying the vma and the address range to mlock. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 483 | |
Kirill A. Shutemov | fc05f56 | 2015-04-14 15:44:39 -0700 | [diff] [blame] | 484 | The callers of populate_vma_page_range() will have already added the memory range |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 485 | to be mlocked to the task's "locked_vm". To account for filtered VMAs, |
Kirill A. Shutemov | fc05f56 | 2015-04-14 15:44:39 -0700 | [diff] [blame] | 486 | populate_vma_page_range() returns the number of pages NOT mlocked. All of the |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 487 | callers then subtract a non-negative return value from the task's locked_vm. A |
| 488 | negative return value represent an error - for example, from get_user_pages() |
| 489 | attempting to fault in a VMA with PROT_NONE access. In this case, we leave the |
| 490 | memory range accounted as locked_vm, as the protections could be changed later |
| 491 | and pages allocated into that region. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 492 | |
| 493 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 494 | munmap()/exit()/exec() System Call Handling |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 495 | ------------------------------------------- |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 496 | |
| 497 | When unmapping an mlocked region of memory, whether by an explicit call to |
| 498 | munmap() or via an internal unmap from exit() or exec() processing, we must |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 499 | munlock the pages if we're removing the last VM_LOCKED VMA that maps the pages. |
Hugh Dickins | 63d6c5a | 2009-01-06 14:39:38 -0800 | [diff] [blame] | 500 | Before the unevictable/mlock changes, mlocking did not mark the pages in any |
| 501 | way, so unmapping them required no processing. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 502 | |
| 503 | To munlock a range of memory under the unevictable/mlock infrastructure, the |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 504 | munmap() handler and task address space call tear down function |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 505 | munlock_vma_pages_all(). The name reflects the observation that one always |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 506 | specifies the entire VMA range when munlock()ing during unmap of a region. |
| 507 | Because of the VMA filtering when mlocking() regions, only "normal" VMAs that |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 508 | actually contain mlocked pages will be passed to munlock_vma_pages_all(). |
| 509 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 510 | munlock_vma_pages_all() clears the VM_LOCKED VMA flag and, like mlock_fixup() |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 511 | for the munlock case, calls __munlock_vma_pages_range() to walk the page table |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 512 | for the VMA's memory range and munlock_vma_page() each resident page mapped by |
| 513 | the VMA. This effectively munlocks the page, only if this is the last |
| 514 | VM_LOCKED VMA that maps the page. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 515 | |
| 516 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 517 | try_to_unmap() |
| 518 | -------------- |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 519 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 520 | Pages can, of course, be mapped into multiple VMAs. Some of these VMAs may |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 521 | have VM_LOCKED flag set. It is possible for a page mapped into one or more |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 522 | VM_LOCKED VMAs not to have the PG_mlocked flag set and therefore reside on one |
| 523 | of the active or inactive LRU lists. This could happen if, for example, a task |
| 524 | in the process of munlocking the page could not isolate the page from the LRU. |
| 525 | As a result, vmscan/shrink_page_list() might encounter such a page as described |
| 526 | in section "vmscan's handling of unevictable pages". To handle this situation, |
| 527 | try_to_unmap() checks for VM_LOCKED VMAs while it is walking a page's reverse |
| 528 | map. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 529 | |
| 530 | try_to_unmap() is always called, by either vmscan for reclaim or for page |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 531 | migration, with the argument page locked and isolated from the LRU. Separate |
Hugh Dickins | b87537d9e | 2015-11-05 18:49:33 -0800 | [diff] [blame] | 532 | functions handle anonymous and mapped file and KSM pages, as these types of |
| 533 | pages have different reverse map lookup mechanisms, with different locking. |
| 534 | In each case, whether rmap_walk_anon() or rmap_walk_file() or rmap_walk_ksm(), |
| 535 | it will call try_to_unmap_one() for every VMA which might contain the page. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 536 | |
Hugh Dickins | b87537d9e | 2015-11-05 18:49:33 -0800 | [diff] [blame] | 537 | When trying to reclaim, if try_to_unmap_one() finds the page in a VM_LOCKED |
| 538 | VMA, it will then mlock the page via mlock_vma_page() instead of unmapping it, |
| 539 | and return SWAP_MLOCK to indicate that the page is unevictable: and the scan |
| 540 | stops there. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 541 | |
Hugh Dickins | b87537d9e | 2015-11-05 18:49:33 -0800 | [diff] [blame] | 542 | mlock_vma_page() is called while holding the page table's lock (in addition |
| 543 | to the page lock, and the rmap lock): to serialize against concurrent mlock or |
| 544 | munlock or munmap system calls, mm teardown (munlock_vma_pages_all), reclaim, |
| 545 | holepunching, and truncation of file pages and their anonymous COWed pages. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 546 | |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 547 | |
Alistair Popple | cd62734 | 2021-06-30 18:54:12 -0700 | [diff] [blame] | 548 | page_mlock() Reverse Map Scan |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 549 | --------------------------------- |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 550 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 551 | When munlock_vma_page() [see section :ref:`munlock()/munlockall() System Call |
| 552 | Handling <munlock_munlockall_handling>` above] tries to munlock a |
| 553 | page, it needs to determine whether or not the page is mapped by any |
| 554 | VM_LOCKED VMA without actually attempting to unmap all PTEs from the |
| 555 | page. For this purpose, the unevictable/mlock infrastructure |
Alistair Popple | cd62734 | 2021-06-30 18:54:12 -0700 | [diff] [blame] | 556 | introduced a variant of try_to_unmap() called page_mlock(). |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 557 | |
Alistair Popple | cd62734 | 2021-06-30 18:54:12 -0700 | [diff] [blame] | 558 | page_mlock() walks the respective reverse maps looking for VM_LOCKED VMAs. When |
| 559 | such a VMA is found the page is mlocked via mlock_vma_page(). This undoes the |
| 560 | pre-clearing of the page's PG_mlocked done by munlock_vma_page. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 561 | |
Alistair Popple | cd62734 | 2021-06-30 18:54:12 -0700 | [diff] [blame] | 562 | Note that page_mlock()'s reverse map walk must visit every VMA in a page's |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 563 | reverse map to determine that a page is NOT mapped into any VM_LOCKED VMA. |
Hugh Dickins | b87537d9e | 2015-11-05 18:49:33 -0800 | [diff] [blame] | 564 | However, the scan can terminate when it encounters a VM_LOCKED VMA. |
Alistair Popple | cd62734 | 2021-06-30 18:54:12 -0700 | [diff] [blame] | 565 | Although page_mlock() might be called a great many times when munlocking a |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 566 | large region or tearing down a large address space that has been mlocked via |
| 567 | mlockall(), overall this is a fairly rare event. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 568 | |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 569 | |
Mike Rapoport | a5e4da9 | 2018-03-21 21:22:42 +0200 | [diff] [blame] | 570 | Page Reclaim in shrink_*_list() |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 571 | ------------------------------- |
| 572 | |
| 573 | shrink_active_list() culls any obviously unevictable pages - i.e. |
Hugh Dickins | 39b5f29 | 2012-10-08 16:33:18 -0700 | [diff] [blame] | 574 | !page_evictable(page) - diverting these to the unevictable list. |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 575 | However, shrink_active_list() only sees unevictable pages that made it onto the |
| 576 | active/inactive lru lists. Note that these pages do not have PageUnevictable |
| 577 | set - otherwise they would be on the unevictable list and shrink_active_list |
| 578 | would never see them. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 579 | |
| 580 | Some examples of these unevictable pages on the LRU lists are: |
| 581 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 582 | (1) ramfs pages that have been placed on the LRU lists when first allocated. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 583 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 584 | (2) SHM_LOCK'd shared memory pages. shmctl(SHM_LOCK) does not attempt to |
| 585 | allocate or fault in the pages in the shared memory region. This happens |
| 586 | when an application accesses the page the first time after SHM_LOCK'ing |
| 587 | the segment. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 588 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 589 | (3) mlocked pages that could not be isolated from the LRU and moved to the |
| 590 | unevictable list in mlock_vma_page(). |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 591 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 592 | shrink_inactive_list() also diverts any unevictable pages that it finds on the |
Hugh Dickins | 15b4473 | 2020-12-15 14:21:31 -0800 | [diff] [blame] | 593 | inactive lists to the appropriate node's unevictable list. |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 594 | |
| 595 | shrink_inactive_list() should only see SHM_LOCK'd pages that became SHM_LOCK'd |
| 596 | after shrink_active_list() had moved them to the inactive list, or pages mapped |
| 597 | into VM_LOCKED VMAs that munlock_vma_page() couldn't isolate from the LRU to |
Alistair Popple | cd62734 | 2021-06-30 18:54:12 -0700 | [diff] [blame] | 598 | recheck via page_mlock(). shrink_inactive_list() won't notice the latter, |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 599 | but will pass on to shrink_page_list(). |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 600 | |
| 601 | shrink_page_list() again culls obviously unevictable pages that it could |
Hugh Dickins | 63d6c5a | 2009-01-06 14:39:38 -0800 | [diff] [blame] | 602 | encounter for similar reason to shrink_inactive_list(). Pages mapped into |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 603 | VM_LOCKED VMAs but without PG_mlocked set will make it all the way to |
Hugh Dickins | 63d6c5a | 2009-01-06 14:39:38 -0800 | [diff] [blame] | 604 | try_to_unmap(). shrink_page_list() will divert them to the unevictable list |
| 605 | when try_to_unmap() returns SWAP_MLOCK, as discussed above. |