CXL on Linux: Boot To Bash

This documentation is intended as a companion to systems engineers (Platform, BIOS, EFI, OS, etc) trying to enable CXL devices on Linux.

CXL device configuration requires platform (Hardware, BIOS, EFI), OS (early boot, core kernel, driver), and user policy decisions that all impact each other. This doc breaks up these configurations into five main areas:

Overview

• CXL on Linux: Boot To Bash

Device Reference

• Devices and Protocols
  • Protocols
  • Device Types
  • Example Devices
• UEFI Data
  • Coherent Device Attribute Table (CDAT)

Platform Configuration

• BIOS/EFI Configuration
  • UEFI Settings
  • Physical Memory Map
  • Decoder Programming
• ACPI Tables
  • CEDT - CXL Early Discovery Table
  • SRAT - Static Resource Affinity Table
  • HMAT - Heterogeneous Memory Attribute Table
  • SLIT - System Locality Information Table
  • DSDT - Differentiated system Description Table
  • ACPI Debugging
• Example Platform Configurations
  • One Device per Host Bridge
  • Multiple Devices per Host Bridge
  • Cross-Host-Bridge Interleave
  • Flexible Presentation

Linux Kernel Configuration:

• Overview
• Linux Init (Early Boot)
  • BIOS, Build and Boot Options
  • Memory Map Creation
  • NUMA Node Reservation
  • Memory Tiers Creation
  • Contiguous Memory Allocation
• CXL Driver Operation
  • CXL-CLI Reference
  • Drivers
  • Driver Devices
    • Base Devices
    • Decoders
    • Regions
    • Mailbox Interfaces
  • Decoder Programming
    • Runtime Programming
    • Auto Decoders
    • Interleave
  • Example Configurations
    • Single Device
    • Inter-Host-Bridge Interleave
    • Intra-Host-Bridge Interleave
    • Multi-Level Interleave
• DAX Driver Operation
  • DAX Device
  • kmem conversion
• Memory Hotplug
  • Default Online Behavior
  • Hotplug Memory Block Size
  • Memory Map
  • Driver Managed Memory

Memory Allocation

• DAX Devices
• The Page Allocator
  • NUMA nodes and mempolicy
  • Memory Zones
  • Zone and Node Quirks
  • CGroups and CPUSets
• Reclaim
  • Demotion
  • ZSwap and Node Preference
  • Demotion with ZSwap
• Huge Pages
  • Contiguous Memory Allocator
  • HugeTLB
• Memory Tiering
  • Memory Tiers
  • Transparent Page Placement
  • Data Access MONitor

Use Cases

• Memory Expansion
  • As Page Cache
  • As DAX Device
• Dynamic Capacity
  • For Virtual Machines
  • For Workload Orchestration
  • For Shared Memory
• Virtual Machines
  • NUMA Passthrough
  • Flexible Shapes
  • Datacenter Efficiency
• Shared Memory
  • Coherence
  • Fabric Attached Memory FileSystem (FAMFS)

Contributors

• Gregory Price

• Joshua Hahn