Hypercall Op-codes (hcalls)¶
Virtualization on 64-bit Power Book3S Platforms is based on the PAPR specification  which describes the run-time environment for a guest operating system and how it should interact with the hypervisor for privileged operations. Currently there are two PAPR compliant hypervisors:
- IBM PowerVM (PHYP): IBM’s proprietary hypervisor that supports AIX, IBM-i and Linux as supported guests (termed as Logical Partitions or LPARS). It supports the full PAPR specification.
- Qemu/KVM: Supports PPC64 linux guests running on a PPC64 linux host. Though it only implements a subset of PAPR specification called LoPAPR .
On PPC64 arch a guest kernel running on top of a PAPR hypervisor is called a pSeries guest. A pseries guest runs in a supervisor mode (HV=0) and must issue hypercalls to the hypervisor whenever it needs to perform an action that is hypervisor priviledged  or for other services managed by the hypervisor.
Hence a Hypercall (hcall) is essentially a request by the pseries guest asking hypervisor to perform a privileged operation on behalf of the guest. The guest issues a with necessary input operands. The hypervisor after performing the privilege operation returns a status code and output operands back to the guest.
The ABI specification for a hcall between a pseries guest and PAPR hypervisor is covered in section 14.5.3 of ref . Switch to the Hypervisor context is done via the instruction HVCS that expects the Opcode for hcall is set in r3 and any in-arguments for the hcall are provided in registers r4-r12. If values have to be passed through a memory buffer, the data stored in that buffer should be in Big-endian byte order.
Once control returns back to the guest after hypervisor has serviced the ‘HVCS’ instruction the return value of the hcall is available in r3 and any out values are returned in registers r4-r12. Again like in case of in-arguments, any out values stored in a memory buffer will be in Big-endian byte order.
Powerpc arch code provides convenient wrappers named plpar_hcall_xxx defined in a arch specific header  to issue hcalls from the linux kernel running as pseries guest.
Any hcall should follow same register convention as described in section 188.8.131.52 of “64-Bit ELF V2 ABI Specification: Power Architecture”. Table below summarizes these conventions:
|Register Range||Volatile (Y/N)||Purpose|
|r3||Y||hcall opcode/return value|
|r4-r10||Y||in and out values|
|r12||Y||Optional-usage/Function entry address at global entry point|
|XER||Y||Fixed-point exception register.|
|CR0-1||Y||Condition register fields.|
|CR2-4||N||Condition register fields.|
|CR5-7||Y||Condition register fields.|
DRC & DRC Indexes¶
DR1 Guest +--+ +------------+ +---------+ | | <----> | | | User | +--+ DRC1 | | DRC | Space | | PAPR | Index +---------+ DR2 | Hypervisor | | | +--+ | | <-----> | Kernel | | | <----> | | Hcall | | +--+ DRC2 +------------+ +---------+
PAPR hypervisor terms shared hardware resources like PCI devices, NVDIMMs etc available for use by LPARs as Dynamic Resource (DR). When a DR is allocated to an LPAR, PHYP creates a data-structure called Dynamic Resource Connector (DRC) to manage LPAR access. An LPAR refers to a DRC via an opaque 32-bit number called DRC-Index. The DRC-index value is provided to the LPAR via device-tree where its present as an attribute in the device tree node associated with the DR.
After servicing the hcall, hypervisor sets the return-value in r3 indicating success or failure of the hcall. In case of a failure an error code indicates the cause for error. These codes are defined and documented in arch specific header .
In some cases a hcall can potentially take a long time and need to be issued multiple times in order to be completely serviced. These hcalls will usually accept an opaque value continue-token within there argument list and a return value of H_CONTINUE indicates that hypervisor hasn’t still finished servicing the hcall yet.
To make such hcalls the guest need to set continue-token == 0 for the initial call and use the hypervisor returned value of continue-token for each subsequent hcall until hypervisor returns a non H_CONTINUE return value.
Below is a partial list of HCALLs that are supported by PHYP. For the corresponding opcode values please look into the arch specific header :
Given a DRC Index of an NVDIMM, read N-bytes from the metadata area associated with it, at a specified offset and copy it to provided buffer. The metadata area stores configuration information such as label information, bad-blocks etc. The metadata area is located out-of-band of NVDIMM storage area hence a separate access semantics is provided.
Given a DRC Index of an NVDIMM, write N-bytes to the metadata area associated with it, at the specified offset and from the provided buffer.
Given a DRC-Index of an NVDIMM, map a continuous SCM blocks range (startingScmBlockIndex, startingScmBlockIndex+numScmBlocksToBind) to the guest at targetLogicalMemoryAddress within guest physical address space. In case targetLogicalMemoryAddress == 0xFFFFFFFF_FFFFFFFF then hypervisor assigns a target address to the guest. The HCALL can fail if the Guest has an active PTE entry to the SCM block being bound.
H_SCM_UNBIND_MEM | Input: drcIndex, startingScmLogicalMemoryAddress, numScmBlocksToUnbind | Out: numScmBlocksUnbound | Return Value: H_Success, H_Parameter, H_P2, H_P3, H_In_Use, H_Overlap, | H_Busy, H_LongBusyOrder1mSec, H_LongBusyOrder10mSec
Given a DRC-Index of an NVDimm, unmap numScmBlocksToUnbind SCM blocks starting at startingScmLogicalMemoryAddress from guest physical address space. The HCALL can fail if the Guest has an active PTE entry to the SCM block being unbound.
Given a DRC-Index and an SCM Block index return the guest physical address to which the SCM block is mapped to.
Given a guest physical address return which DRC Index and SCM block is mapped to that address.
Depending on the Target scope unmap all SCM blocks belonging to all NVDIMMs or all SCM blocks belonging to a single NVDIMM identified by its drcIndex from the LPAR memory.
Given a DRC Index return the info on predictive failure and overall health of the PMEM device. The asserted bits in the health-bitmap indicate one or more states (described in table below) of the PMEM device and health-bit-valid-bitmap indicate which bits in health-bitmap are valid. The bits are reported in reverse bit ordering for example a value of 0xC400000000000000 indicates bits 0, 1, and 5 are valid.
Health Bitmap Flags:
|00||PMEM device is unable to persist memory contents. If the system is powered down, nothing will be saved.|
|01||PMEM device failed to persist memory contents. Either contents were not saved successfully on power down or were not restored properly on power up.|
|02||PMEM device contents are persisted from previous IPL. The data from the last boot were successfully restored.|
|03||PMEM device contents are not persisted from previous IPL. There was no data to restore from the last boot.|
|04||PMEM device memory life remaining is critically low|
|05||PMEM device will be garded off next IPL due to failure|
|06||PMEM device contents cannot persist due to current platform health status. A hardware failure may prevent data from being saved or restored.|
|07||PMEM device is unable to persist memory contents in certain conditions|
|08||PMEM device is encrypted|
|09||PMEM device has successfully completed a requested erase or secure erase procedure.|
|10:63||Reserved / Unused|
Given a DRC Index collect the performance statistics for NVDIMM and copy them to the resultBuffer.
Given a DRC Index Flush the data to backend NVDIMM device.
The hcall returns H_BUSY when the flush takes longer time and the hcall needs to be issued multiple times in order to be completely serviced. The continue-token from the output to be passed in the argument list of subsequent hcalls to the hypervisor until the hcall is completely serviced at which point H_SUCCESS or other error is returned by the hypervisor.
|||“Power Architecture Platform Reference” https://en.wikipedia.org/wiki/Power_Architecture_Platform_Reference|
|||(1, 2) “Linux on Power Architecture Platform Reference” https://members.openpowerfoundation.org/document/dl/469|
|||“Definitions and Notation” Book III-Section 14.5.3 https://openpowerfoundation.org/?resource_lib=power-isa-version-3-0|
|||(1, 2, 3) arch/powerpc/include/asm/hvcall.h|
|||“64-Bit ELF V2 ABI Specification: Power Architecture” https://openpowerfoundation.org/?resource_lib=64-bit-elf-v2-abi-specification-power-architecture|