Skip to content

Commit 7362fac

Browse files
Gregory Pricedavejiang
Gregory Price
authored and
davejiang
committed
Documentation/driver-api/cxl: device hotplug section
Describe cxl memory device hotplug implications, in particular how the platform CEDT CFMWS must be described to support successful hot-add of memory devices. Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com> Signed-off-by: Gregory Price <gourry@gourry.net> Reviewed-by: Dave Jiang <dave.jiang@intel.com> Reviewed-by: Alejandro Lucero Palau <alucerop@amd.com> Link: https://patch.msgid.link/20251219170538.1675743-3-gourry@gourry.net Signed-off-by: Dave Jiang <dave.jiang@intel.com>
1 parent 2489d83 commit 7362fac

3 files changed

Lines changed: 134 additions & 0 deletions

File tree

‎Documentation/driver-api/cxl/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ that have impacts on each other. The docs here break up configurations steps.
3030
platform/acpi
3131
platform/cdat
3232
platform/example-configs
33+
platform/device-hotplug
3334

3435
.. toctree::
3536
:maxdepth: 2

‎Documentation/driver-api/cxl/platform/bios-and-efi.rst‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,9 @@ up without requiring CXL driver support. These platform vendors should
4949
test their configurations with the existing CXL driver and provide driver
5050
support for their auto-configurations if features like RAS are required.
5151

52+
Platforms requiring boot-time programming and/or locking of CXL fabric
53+
components may prevent features, such as device hot-plug, from working.
54+
5255
UEFI Settings
5356
=============
5457
If your platform supports it, the :code:`uefisettings` command can be used to

‎Documentation/driver-api/cxl/platform/device-hotplug.rst‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
.. SPDX-License-Identifier: GPL-2.0
2+
3+
==================
4+
CXL Device Hotplug
5+
==================
6+
7+
Device hotplug refers to *physical* hotplug of a device (addition or removal
8+
of a physical device from the machine).
9+
10+
BIOS/EFI software is expected to configure sufficient resources **at boot
11+
time** to allow hotplugged devices to be configured by software (such as
12+
proximity domains, HPA regions, and host-bridge configurations).
13+
14+
BIOS/EFI is not expected (**nor suggested**) to configure hotplugged
15+
devices at hotplug time (i.e. HDM decoders should be left unprogrammed).
16+
17+
This document covers some examples of those resources, but should not
18+
be considered exhaustive.
19+
20+
Hot-Remove
21+
==========
22+
Hot removal of a device typically requires careful removal of software
23+
constructs (memory regions, associated drivers) which manage these devices.
24+
25+
Hard-removing a CXL.mem device without carefully tearing down driver stacks
26+
is likely to cause the system to machine-check (or at least SIGBUS if memory
27+
access is limited to user space).
28+
29+
Memory Device Hot-Add
30+
=====================
31+
A device present at boot may be associated with a CXL Fixed Memory Window
32+
reported in :doc:`CEDT<acpi/cedt>`. That CFMWS may match the size of the
33+
device, but the construction of the CEDT CFMWS is platform-defined.
34+
35+
Hot-adding a memory device requires this pre-defined, **static** CFMWS to
36+
have sufficient HPA space to describe that device.
37+
38+
There are a few common scenarios to consider.
39+
40+
Single-Endpoint Memory Device Present at Boot
41+
---------------------------------------------
42+
A device present at boot likely had its capacity reported in the
43+
:doc:`CEDT<acpi/cedt>`. If a device is removed and a new device hotplugged,
44+
the capacity of the new device will be limited to the original CFMWS capacity.
45+
46+
Adding capacity larger than the original device will cause memory region
47+
creation to fail if the region size is greater than the CFMWS size.
48+
49+
The CFMWS is **static** and cannot be adjusted. Platforms which may expect
50+
different sized devices to be hotplugged must allocate sufficient CFMWS space
51+
**at boot time** to cover all future expected devices.
52+
53+
Multi-Endpoint Memory Device Present at Boot
54+
--------------------------------------------
55+
Non-switch-based Multi-Endpoint devices are outside the scope of what the
56+
CXL specification describes, but they are technically possible. We describe
57+
them here for instructive reasons only - this does not imply Linux support.
58+
59+
A hot-plug capable CXL memory device, such as one which presents multiple
60+
expanders as a single large-capacity device, should report the **maximum
61+
possible capacity** for the device at boot. ::
62+
63+
HB0
64+
RP0
65+
|
66+
[Multi-Endpoint Memory Device]
67+
_____|_____
68+
| |
69+
[Endpoint0] [Empty]
70+
71+
72+
Limiting the size to the capacity preset at boot will limit hot-add support
73+
to replacing capacity that was present at boot.
74+
75+
No CXL Device Present at Boot
76+
-----------------------------
77+
When no CXL memory device is present on boot, some platforms omit the CFMWS
78+
in the :doc:`CEDT<acpi/cedt>`. When this occurs, hot-add is not possible.
79+
80+
This describes the base case for any given device not being present at boot.
81+
If a future possible device is not described in the CEDT at boot, hot-add
82+
of that device is either limited or not possible.
83+
84+
For a platform to support hot-add of a full memory device, it must allocate
85+
a CEDT CFMWS region with sufficient memory capacity to cover all future
86+
potentially added capacity (along with any relevant CEDT CHBS entry).
87+
88+
To support memory hotplug directly on the host bridge/root port, or on a switch
89+
downstream of the host bridge, a platform must construct a CEDT CFMWS at boot
90+
with sufficient resources to support the max possible (or expected) hotplug
91+
memory capacity. ::
92+
93+
HB0 HB1
94+
RP0 RP1 RP2
95+
| | |
96+
Empty Empty USP
97+
________|________
98+
| | | |
99+
DSP DSP DSP DSP
100+
| | | |
101+
All Empty
102+
103+
For example, a BIOS/EFI may expose an option to configure a CEDT CFMWS with
104+
a pre-configured amount of memory capacity (per host bridge, or host bridge
105+
interleave set), even if no device is attached to Root Ports or Downstream
106+
Ports at boot (as depicted in the figure above).
107+
108+
109+
Interleave Sets
110+
===============
111+
112+
Host Bridge Interleave
113+
----------------------
114+
Host-bridge interleaved memory regions are defined **statically** in the
115+
:doc:`CEDT<acpi/cedt>`. To apply cross-host-bridge interleave, a CFMWS entry
116+
describing that interleave must have been provided **at boot**. Hotplugged
117+
devices cannot add host-bridge interleave capabilities at hotplug time.
118+
119+
See the :doc:`Flexible CEDT Configuration<example-configurations/flexible>`
120+
example to see how a platform can provide this kind of flexibility regarding
121+
hotplugged memory devices. BIOS/EFI software should consider options to
122+
present flexible CEDT configurations with hotplug support.
123+
124+
HDM Interleave
125+
--------------
126+
Decoder-applied interleave can flexibly handle hotplugged devices, as decoders
127+
can be re-programmed after hotplug.
128+
129+
To add or remove a device to/from an existing HDM-applied interleaved region,
130+
that region must be torn down an re-created.

0 commit comments

Comments
 (0)