High Memory Handling¶
By: Peter Zijlstra <a.p.zijlstra@chello.nl>
What Is High Memory?¶
High memory (highmem) is used when the size of physical memory approaches or exceeds the maximum size of virtual memory. At that point it becomes impossible for the kernel to keep all of the available physical memory mapped at all times. This means the kernel needs to start using temporary mappings of the pieces of physical memory that it wants to access.
The part of (physical) memory not covered by a permanent mapping is what we refer to as ‘highmem’. There are various architecture dependent constraints on where exactly that border lies.
In the i386 arch, for example, we choose to map the kernel into every process’s VM space so that we don’t have to pay the full TLB invalidation costs for kernel entry/exit. This means the available virtual memory space (4GiB on i386) has to be divided between user and kernel space.
The traditional split for architectures using this approach is 3:1, 3GiB for userspace and the top 1GiB for kernel space:
+--------+ 0xffffffff
| Kernel |
+--------+ 0xc0000000
| |
| User |
| |
+--------+ 0x00000000
This means that the kernel can at most map 1GiB of physical memory at any one time, but because we need virtual address space for other things - including temporary maps to access the rest of the physical memory - the actual direct map will typically be less (usually around ~896MiB).
Other architectures that have mm context tagged TLBs can have separate kernel and user maps. Some hardware (like some ARMs), however, have limited virtual space when they use mm context tags.
Temporary Virtual Mappings¶
The kernel contains several ways of creating temporary mappings. The following list shows them in order of preference of use.
kmap_local_page(). This function is used to require short term mappings. It can be invoked from any context (including interrupts) but the mappings can only be used in the context which acquired them.This function should be preferred, where feasible, over all the others.
These mappings are thread-local and CPU-local, meaning that the mapping can only be accessed from within this thread and the thread is bound the CPU while the mapping is active. Even if the thread is preempted (since preemption is never disabled by the function) the CPU can not be unplugged from the system via CPU-hotplug until the mapping is disposed.
It’s valid to take pagefaults in a local kmap region, unless the context in which the local mapping is acquired does not allow it for other reasons.
kmap_local_page()always returns a valid virtual address and it is assumed thatkunmap_local()will never fail.Nesting
kmap_local_page()andkmap_atomic()mappings is allowed to a certain extent (up to KMAP_TYPE_NR) but their invocations have to be strictly ordered because the map implementation is stack based. Seekmap_local_page()kdocs (included in the “Functions” section) for details on how to manage nested mappings.kmap_atomic(). This permits a very short duration mapping of a single page. Since the mapping is restricted to the CPU that issued it, it performs well, but the issuing task is therefore required to stay on that CPU until it has finished, lest some other task displace its mappings.kmap_atomic()may also be used by interrupt contexts, since it does not sleep and the callers too may not sleep until afterkunmap_atomic()is called.Each call of
kmap_atomic()in the kernel creates a non-preemptible section and disable pagefaults. This could be a source of unwanted latency. Therefore users should preferkmap_local_page()instead ofkmap_atomic().It is assumed that k[un]map_atomic() won’t fail.
kmap(). This should be used to make short duration mapping of a single page with no restrictions on preemption or migration. It comes with an overhead as mapping space is restricted and protected by a global lock for synchronization. When mapping is no longer needed, the address that the page was mapped to must be released withkunmap().Mapping changes must be propagated across all the CPUs.
kmap()also requires global TLB invalidation when the kmap’s pool wraps and it might block when the mapping space is fully utilized until a slot becomes available. Therefore,kmap()is only callable from preemptible context.All the above work is necessary if a mapping must last for a relatively long time but the bulk of high-memory mappings in the kernel are short-lived and only used in one place. This means that the cost of
kmap()is mostly wasted in such cases.kmap()was not intended for long term mappings but it has morphed in that direction and its use is strongly discouraged in newer code and the set of the preceding functions should be preferred.On 64-bit systems, calls to
kmap_local_page(),kmap_atomic()andkmap()have no real work to do because a 64-bit address space is more than sufficient to address all the physical memory whose pages are permanently mapped.vmap(). This can be used to make a long duration mapping of multiple physical pages into a contiguous virtual space. It needs global synchronization to unmap.
Cost of Temporary Mappings¶
The cost of creating temporary mappings can be quite high. The arch has to manipulate the kernel’s page tables, the data TLB and/or the MMU’s registers.
If CONFIG_HIGHMEM is not set, then the kernel will try and create a mapping simply with a bit of arithmetic that will convert the page struct address into a pointer to the page contents rather than juggling mappings about. In such a case, the unmap operation may be a null operation.
If CONFIG_MMU is not set, then there can be no temporary mappings and no highmem. In such a case, the arithmetic approach will also be used.
i386 PAE¶
The i386 arch, under some circumstances, will permit you to stick up to 64GiB of RAM into your 32-bit machine. This has a number of consequences:
- Linux needs a page-frame structure for each page in the system and the pageframes need to live in the permanent mapping, which means:
- you can have 896M/sizeof(struct page) page-frames at most; with struct page being 32-bytes that would end up being something in the order of 112G worth of pages; the kernel, however, needs to store more than just page-frames in that memory…
- PAE makes your page tables larger - which slows the system down as more data has to be accessed to traverse in TLB fills and the like. One advantage is that PAE has more PTE bits and can provide advanced features like NX and PAT.
The general recommendation is that you don’t use more than 8GiB on a 32-bit machine - although more might work for you and your workload, you’re pretty much on your own - don’t expect kernel developers to really care much if things come apart.
Functions¶
-
void *
kmap(struct page *page)¶ Map a page for long term usage
Parameters
struct page *page- Pointer to the page to be mapped
Return
The virtual address of the mapping
Description
Can only be invoked from preemptible task context because on 32bit systems with CONFIG_HIGHMEM enabled this function might sleep.
For systems with CONFIG_HIGHMEM=n and for pages in the low memory area this returns the virtual address of the direct kernel mapping.
The returned virtual address is globally visible and valid up to the
point where it is unmapped via kunmap(). The pointer can be handed to
other contexts.
For highmem pages on 32bit systems this can be slow as the mapping space
is limited and protected by a global lock. In case that there is no
mapping slot available the function blocks until a slot is released via
kunmap().
Parameters
struct page *page- Pointer to the page which was mapped by
kmap()
Description
Counterpart to kmap(). A NOOP for CONFIG_HIGHMEM=n and for mappings of
pages in the low memory area.
-
struct page *
kmap_to_page(void *addr)¶ Get the page for a kmap’ed address
Parameters
void *addr- The address to look up
Return
The page which is mapped to addr.
-
void
kmap_flush_unused(void)¶ Flush all unused kmap mappings in order to remove stray mappings
Parameters
void- no arguments
-
void *
kmap_local_page(struct page *page)¶ Map a page for temporary usage
Parameters
struct page *page- Pointer to the page to be mapped
Return
The virtual address of the mapping
Description
Can be invoked from any context.
Requires careful handling when nesting multiple mappings because the map management is stack based. The unmap has to be in the reverse order of the map operation:
addr1 = kmap_local_page(page1); addr2 = kmap_local_page(page2); … kunmap_local(addr2); kunmap_local(addr1);
Unmapping addr1 before addr2 is invalid and causes malfunction.
Contrary to kmap() mappings the mapping is only valid in the context of
the caller and cannot be handed to other contexts.
On CONFIG_HIGHMEM=n kernels and for low memory pages this returns the virtual address of the direct mapping. Only real highmem pages are temporarily mapped.
While it is significantly faster than kmap() for the higmem case it
comes with restrictions about the pointer validity. Only use when really
necessary.
On HIGHMEM enabled systems mapping a highmem page has the side effect of
disabling migration in order to keep the virtual address stable across
preemption. No caller of kmap_local_page() can rely on this side effect.
-
void *
kmap_local_folio(struct folio *folio, size_t offset)¶ Map a page in this folio for temporary usage
Parameters
struct folio *folio- The folio containing the page.
size_t offset- The byte offset within the folio which identifies the page.
Description
Requires careful handling when nesting multiple mappings because the map management is stack based. The unmap has to be in the reverse order of the map operation:
addr1 = kmap_local_folio(folio1, offset1);
addr2 = kmap_local_folio(folio2, offset2);
...
kunmap_local(addr2);
kunmap_local(addr1);
Unmapping addr1 before addr2 is invalid and causes malfunction.
Contrary to kmap() mappings the mapping is only valid in the context of
the caller and cannot be handed to other contexts.
On CONFIG_HIGHMEM=n kernels and for low memory pages this returns the virtual address of the direct mapping. Only real highmem pages are temporarily mapped.
While it is significantly faster than kmap() for the higmem case it
comes with restrictions about the pointer validity. Only use when really
necessary.
On HIGHMEM enabled systems mapping a highmem page has the side effect of
disabling migration in order to keep the virtual address stable across
preemption. No caller of kmap_local_folio() can rely on this side effect.
Context
Can be invoked from any context.
Return
The virtual address of offset.
-
void *
kmap_atomic(struct page *page)¶ Atomically map a page for temporary usage - Deprecated!
Parameters
struct page *page- Pointer to the page to be mapped
Return
The virtual address of the mapping
Description
In fact a wrapper around kmap_local_page() which also disables pagefaults
and, depending on PREEMPT_RT configuration, also CPU migration and
preemption. Therefore users should not count on the latter two side effects.
Mappings should always be released by kunmap_atomic().
Do not use in new code. Use kmap_local_page() instead.
It is used in atomic context when code wants to access the contents of a page that might be allocated from high memory (see __GFP_HIGHMEM), for example a page in the pagecache. The API has two functions, and they can be used in a manner similar to the following:
// Find the page of interest.
struct page *page = find_get_page(mapping, offset);
// Gain access to the contents of that page.
void *vaddr = kmap_atomic(page);
// Do something to the contents of that page.
memset(vaddr, 0, PAGE_SIZE);
// Unmap that page.
kunmap_atomic(vaddr);
Note that the kunmap_atomic() call takes the result of the kmap_atomic()
call, not the argument.
If you need to map two pages because you want to copy from one page to another you need to keep the kmap_atomic calls strictly nested, like:
vaddr1 = kmap_atomic(page1); vaddr2 = kmap_atomic(page2);
memcpy(vaddr1, vaddr2, PAGE_SIZE);
kunmap_atomic(vaddr2); kunmap_atomic(vaddr1);
-
struct page *
alloc_zeroed_user_highpage_movable(struct vm_area_struct *vma, unsigned long vaddr)¶ Allocate a zeroed HIGHMEM page for a VMA that the caller knows can move
Parameters
struct vm_area_struct *vma- The VMA the page is to be allocated for
unsigned long vaddr- The virtual address the page will be inserted into
Return
The allocated and zeroed HIGHMEM page
Description
This function will allocate a page for a VMA that the caller knows will be able to migrate in the future using move_pages() or reclaimed
An architecture may override this function by defining __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE_MOVABLE and providing their own implementation.
-
void
folio_zero_segments(struct folio *folio, size_t start1, size_t xend1, size_t start2, size_t xend2)¶ Zero two byte ranges in a folio.
Parameters
struct folio *folio- The folio to write to.
size_t start1- The first byte to zero.
size_t xend1- One more than the last byte in the first range.
size_t start2- The first byte to zero in the second range.
size_t xend2- One more than the last byte in the second range.
-
void
folio_zero_segment(struct folio *folio, size_t start, size_t xend)¶ Zero a byte range in a folio.
Parameters
struct folio *folio- The folio to write to.
size_t start- The first byte to zero.
size_t xend- One more than the last byte to zero.
-
void
folio_zero_range(struct folio *folio, size_t start, size_t length)¶ Zero a byte range in a folio.
Parameters
struct folio *folio- The folio to write to.
size_t start- The first byte to zero.
size_t length- The number of bytes to zero.
-
kunmap_atomic(__addr)¶ Unmap the virtual address mapped by
kmap_atomic()- deprecated!
Parameters
__addr- Virtual address to be unmapped
Description
Unmaps an address previously mapped by kmap_atomic() and re-enables
pagefaults. Depending on PREEMP_RT configuration, re-enables also
migration and preemption. Users should not count on these side effects.
Mappings should be unmapped in the reverse order that they were mapped.
See kmap_local_page() for details on nesting.
__addr can be any address within the mapped page, so there is no need
to subtract any offset that has been added. In contrast to kunmap(),
this function takes the address returned from kmap_atomic(), not the
page passed to it. The compiler will warn you if you pass the page.
-
kunmap_local(__addr)¶ Unmap a page mapped via
kmap_local_page().
Parameters
__addr- An address within the page mapped
Description
__addr can be any address within the mapped page. Commonly it is the
address return from kmap_local_page(), but it can also include offsets.
Unmapping should be done in the reverse order of the mapping. See
kmap_local_page() for details.