Understanding NUMA
What Is NUMA?
NUMA stands for Non-Uniform Memory Access. It describes a computer architecture in which memory access time can vary depending on the relationship between a processor and the physical memory being accessed.
In a NUMA system, a processor generally accesses memory associated with its local NUMA node with lower latency than memory associated with another node. Accessing memory outside the local node can require communication across the processor interconnect, adding latency and consuming interconnect bandwidth.
Why Do Servers Use NUMA?
Modern enterprise servers can contain many processor cores and large amounts of memory. NUMA allows memory resources to be distributed closer to the processors that use them instead of forcing all processors to access memory through one centralized path.
More CPU Resources
NUMA architectures help large servers scale across multiple processors, cores and memory controllers.
Distributed Memory
Memory channels can be associated with specific processors or NUMA domains instead of one centralized memory controller.
Parallel Memory Access
Multiple processors can use their local memory resources simultaneously, helping increase aggregate system memory bandwidth.
Locality Matters
NUMA-aware software can try to keep processing and memory close together to reduce unnecessary remote memory access.
How a NUMA Server Is Organized
A simplified two-socket server can be viewed as two NUMA nodes, with each processor connected directly to its own memory resources and linked to the other processor through a high-speed interconnect.
This diagram is intentionally simplified. Actual NUMA topology varies by processor architecture, server platform, firmware configuration and operating system.
Local Memory vs Remote Memory
Local NUMA Memory
Local memory is memory associated with the NUMA node where a CPU core is executing. The processor can generally access this memory through its local memory controller with lower latency.
Remote NUMA Memory
Remote memory belongs to another NUMA node. Access may need to travel across a processor or node interconnect, which can increase latency compared with local memory access.
NUMA systems are designed to access memory across nodes. The performance consideration is that remote access can have different latency and bandwidth characteristics. NUMA-aware operating systems and applications attempt to place workloads and memory efficiently.
UMA vs NUMA
| Characteristic | UMA | NUMA |
|---|---|---|
| Full Name | Uniform Memory Access | Non-Uniform Memory Access |
| Memory Access | Designed around relatively uniform memory access characteristics | Access characteristics depend on memory location relative to the processor |
| Locality | Less dependent on processor-to-memory locality | Local vs remote memory placement can matter |
| Server Scaling | Simpler architecture | Well suited to scalable multiprocessor systems |
| Optimization | Memory placement is generally less topology-sensitive | Workload and memory placement can affect performance |
What Is a NUMA Node?
A NUMA node is a logical grouping of processor resources and memory that share relatively local access characteristics. In a simple dual-socket server, each processor may correspond to a NUMA node, but modern processor designs and firmware settings can create more complex topologies.
The operating system detects this topology and can use it when scheduling threads and allocating memory. Hypervisors can also expose virtual NUMA topology to virtual machines.
NUMA in Dual-Socket Servers
NUMA becomes especially important in servers containing multiple processor sockets because each processor typically has direct access to specific memory channels.
CPU Executes a Thread
An application thread executes on a core associated with a particular NUMA node.
Memory Is Requested
The application needs data located somewhere in the server's physical memory.
Local or Remote?
If the memory is local, the request can use the node's local memory path. Otherwise it may traverse the interconnect.
Performance Can Differ
Remote access can introduce additional latency and interconnect traffic compared with local access.
NUMA Makes Server Memory Population Important
Installing the correct total amount of RAM does not automatically mean the server has an optimal memory configuration. Memory should be populated according to the server manufacturer's channel, processor and DIMM population rules.
In a multi-socket server, unbalanced memory population can leave one processor with substantially different local memory resources than another. This can affect available memory bandwidth, capacity distribution and workload placement.
NUMA on Intel Xeon and AMD EPYC Servers
Intel Xeon
Multi-socket Intel Xeon server platforms use processor interconnects and distributed memory resources that create NUMA topology. Exact node organization varies by processor generation and server configuration.
Learn About Intel Xeon →AMD EPYC
AMD EPYC server platforms also expose NUMA topology. The exact relationship between sockets, memory controllers, processor dies and NUMA nodes depends on the EPYC generation and platform configuration.
Learn About AMD EPYC →NUMA and Virtualization
Hypervisors such as VMware ESXi, Microsoft Hyper-V and KVM-based virtualization platforms can use NUMA topology when scheduling virtual machines across physical CPU and memory resources.
What Is vNUMA?
vNUMA, or virtual NUMA, is a virtualized representation of NUMA topology presented to a virtual machine. It allows a NUMA-aware guest operating system and applications to understand that the VM's virtual CPUs and memory may span multiple topology domains.
vNUMA becomes particularly relevant with large VMs. Exact behavior and configuration recommendations depend on the hypervisor, processor platform and workload.
When Does NUMA Affect Server Performance?
NUMA Is Not Automatically a Performance Problem
NUMA is a normal architectural feature of many enterprise servers. The presence of multiple NUMA nodes does not mean a server is incorrectly configured or inherently slow.
Performance issues are more likely when workloads repeatedly access remote memory unnecessarily, memory is poorly balanced, applications are not NUMA-aware, or virtual machines are sized without considering the host's physical topology.
NUMA Configuration Checklist
NUMA FAQ
What does NUMA stand for?
NUMA stands for Non-Uniform Memory Access.
What is a NUMA node?
A NUMA node is a grouping of processor and memory resources that share relatively local memory access characteristics.
What is local NUMA memory?
Local memory is memory associated with the NUMA node where the processor core executing the workload resides.
What is remote NUMA memory?
Remote memory is memory associated with another NUMA node and may require access across a processor interconnect.
Is NUMA only used in dual-socket servers?
No. NUMA topology is not necessarily equivalent to socket count. Modern processor and server designs can expose topology that is more complex than one NUMA node per physical socket.
Does NUMA affect virtualization?
Yes. Hypervisors can use physical NUMA topology when scheduling virtual CPUs and allocating memory, especially for larger virtual machines.
What is vNUMA?
vNUMA is virtual NUMA topology presented to a virtual machine so a NUMA-aware guest can understand its virtual CPU and memory organization.
Does NUMA make a server slower?
Not inherently. NUMA enables scalable server architectures, but poorly placed workloads or excessive remote memory access can affect performance.
Does memory placement matter with NUMA?
Yes. Memory population and workload placement can influence memory locality, bandwidth and performance in NUMA systems.
Do Intel Xeon servers use NUMA?
Multi-socket Intel Xeon platforms commonly expose NUMA topology to the operating system.
Do AMD EPYC servers use NUMA?
Yes. AMD EPYC platforms can expose NUMA topology, with the exact arrangement depending on processor generation, server design and configuration.
Should NUMA be disabled?
NUMA-related firmware settings should not be changed simply because multiple nodes are visible. Configuration should follow the server manufacturer's guidance and the requirements of the operating system, hypervisor and workload.