A group of researchers from the University of Toronto has made public the details of a new one attacknamed GPUThor (via Bleeping Computer), capable of circumvent ECC protections (Error-Correcting Code) implemented on NVIDIA GPUspaving the way for both denial-of-service (DoS) scenarios and escalation techniques up to root privileges.
According to what is reported in the academic paper, GPUThor represents a significant leap forward compared to previous attack concepts developed by the same research team, such as GPUHammer, which had effectively been neutralized by the introduction of ECC protections on NVIDIA cards. The attack was Successfully demonstrated on NVIDIA workstation GPUs based on Ampere architecture and equipped with GDDR6 memoryincluding RTX A4000, RTX A4500, RTX A5000 and RTX A6000: models widely used in the AI and cloud infrastructure sectors.
The Rowhammer family of attacks exploits repeated and intensive access (“hammering”) to certain rows of memory to increase the probability that the bits in adjacent cells change state, going from 0 to 1 or vice versa. This phenomenon can generate data corruption and security vulnerabilities; In an AI model training context, which is heavily dependent on GPU computing power, a successful Rowhammer attack could substantially compromise the accuracy of the models.
To combat these types of threats, NVIDIA employs mitigations such as the SECDED ECC scheme, which can correct single-bit errors and detect (but not correct) double-bit errors within monitored memory blocks. Researchers have however GPUThor calibrated so that the hammering pattern follows a non-uniform pattern, calibrated to a frequency that does not activate the defense mechanisms Target Row Refresh (TRR) provided by the GDDR6 standard.
This was achieved by taking advantage of two undocumented GPU behaviors: how repeated memory requests are aggregated (coalescence) and how often TRR kicks in.
According to published data, GPUThor’s technique generates 6.6 times more attacker line activations than previous methodsreaching between 72,000 and 377,000 bit-flips per gigabyte on tested GPUs without ECC protection. These are values between 4,548 and 23,597 times higher than those obtained with GPUHammer, approaching the success rates typical of powerful CPU-side Rowhammer attacks such as Blacksmith. Consequentially, the time needed to spot an exploitable bit-flip is drastically reduced, going from around 21.9 hours with GPUHammer to just 1.1 minutes with GPUThor.
Even with ECC active, the attack is not completely neutralized: The researchers report generating 387 double-bit errors, which were detected but not correctable by the ECC system, and two triple-bit errors that the ECC incorrectly attempted to correct, still causing data corruption.
In tests conducted on an RTX A6000 with ECC active, GPUThor was able to induce a denial-of-service state, causing the GPU to reset every two hours and halt all running workloads. By repeating the attack on the same board, the device ends up signaling itself as requiring replacement.
The scenario considered most critical by researchers concerns however privilege escalation up to the root level: By corrupting GPU memory page tables, a CUDA program without elevated privileges could gain arbitrary memory access and open a root shell on the host system.
In addition to the four GPUs on which the attack was tested directly, the researchers point out that Ampere-based server-class GPUs, such as the A100, also remain vulnerable to privilege escalation as they still rely on SECDED-level ECC protections, despite some limitations that make them more resistant to DoS scenarios.
For some Blackwell architecture-based GPUs, the RAS Repair resilience feature makes the attack more time-consumingbut it doesn’t completely prevent it. According to what is indicated in the paper, published yesterday, even GPUs equipped with HBM3/HBM3E and GDDR7 memory with on-die ECC could be vulnerable if multiple bit-flips are triggered.
The researchers reported their findings to NVIDIA last April 29; on August 21st the company published an official advisory with mitigation indications. NVIDIA recommends enabling SYS-ECC and IOMMU/DMA isolation at the same time, continuously monitoring GPU error telemetry, and limiting the sharing or execution of untrusted workloads.
The company also specifies that the level of risk varies based on the DRAM device used, the memory technology, the platform design, the defenses integrated into the DRAM itself and the system configuration, highlighting how, with the same attack patterns, no bit-flips were observed on the tested GDDR6X or HBM2E GPUs.
For their part, researchers from the University of Toronto recommend avoiding, where possible, sharing GPUs between different tenantsactively monitor ECC error counters, and limit the execution of untrusted CUDA workloads. Truly comprehensive protection, they conclude, will likely require the adoption of more robust multi-bit ECC schemes and new hardware-level defenses in future generations of GPUs.
