Concepts

Memory map

Which DRAM XNU gets, and the holes it must not touch.

LK leaves memory on the phone that does not belong to us: the trusted-OS and ATF regions, the SSPM and SCP, the modem’s memory and LK’s own framebuffer. Several of them are protected by the EMI memory protection unit, which faults an access from the application processor. XNU is told which DRAM it may use in two steps: a range in the payload and the ADT, and, with mp_mem4g, a list of holes inside it.

The two layouts

• The 2 GB layout. The payload’s own phys_base and mem_size are used, the ADT’s dram-base and dram-size. On spaced mkadt.py sets the range to 0xc0000000 up to 0x13e000000 (SPACED_PHYS_BASE, SPACED_MEM_SIZE: the largest free range, ending at the 32 MB boundary below the highest reservation, dramc-rk at 0x13ffff000) and takes the last 1 MiB off the top for the panic log (/pram), so dram-size is 0x7df00000. iOS saw 2015 MB in this layout.
• The 4 GB layout (mp_mem4g, on unless mp_mem4g=0). XNU’s RAM starts at the start of DRAM, 0x40000000, and ends where the payload’s range ends, 0x13df00000. What XNU must not use is a list of holes.

Safe boot adds mp_mem4g=0 to the payload’s line, so a safe boot is always the 2 GB layout.

How the shim makes the holes

mem_holes in shim/holes.c takes, from the tree LK passed:

1. every fixed reg of /reserved-memory, whether no-map or not (the TEE, ATF, SSPM, SCP, the modem, LK’s framebuffer: every EMI MPU region that limits the application processor lies inside one of them);
2. the payload and the tables the loader put after it (LK’s initrd range, which mtkloader extends over them);
3. LK’s device tree itself;
4. on spaced, the fixed range 0xbe000000-0xbff30000, where Android puts the Wi-Fi and CONNSYS pools (they are alloc-ranges in the tree, so not in LK’s list).

Each is rounded outward to 16 KB, clipped to DRAM, sorted and merged; at most 48 ranges (HOLES_MAX). The shim drops the 4 GB layout for the 2 GB one when the list is unusable: a hole below the end of the kernel and its ADT (the kernel would land in it), a kernel segment over the shim itself, or no list at all.

The list is left in the boot_args page at +0x2000: MPHOLES1, a count, then the [base, end) pairs.

What the kernel does with them

mapple_mem4g_init in arm_vm_init.c reads the list through the bootstrap map and checks it: ranges sorted, 16 KB aligned, inside RAM, 1 to 48 of them, and a panic otherwise. With all of DRAM and no list it panics (all of DRAM and no hole list). Holes are kept out of the free pages, out of pa_valid() (so they are mapped as I/O) and out of the physmap, so the CPU has no cacheable alias to prefetch through into memory the EMI MPU protects. Page numbers run by index from the first page, so the virtual memory walks through a hole and keeps its pages off the free list, instead of skipping them. The console prints each hole and the total (mp_mem4g: RAM 0x…-0x…, N holes of M MiB, K MiB for the VM).

On spaced

LK places 442 MB of fixed reservations (mblock-* in /reserved-memory): ATF at 0x4ce00000, ramoops at 0x4d010000, LK’s framebuffer at 0x771b0000 and the modem’s md_mem_usage. In the run that first used the 4 GB layout the kernel reported 13 holes; iOS saw 3573 MB, against 2015 MB in the 2 GB layout.

Android’s own pools come from /proc/iomem: Wi-Fi’s window is at 0xbf000000, and the Wi-Fi driver maps it cacheable and flushes it through its own mapping, because a flush by physical address skips holes and faults.

Source: shim/holes.c, shim/loader.c of mapple_loader at 2aa5786; osfmk/arm64/arm_vm_init.c and iokit/Kernel/arm/MT6631WiFi.cpp of the kernel at 76b737c; tools/mkadt.py of mapple_build at f7431bb; the bring-up’s run logs for the figures.