Heap Overflow in Programming

Lecture



Heap overflow, heap overrun, or heap smashing is a type of buffer overflow that occurs in the heap data area. Heap overflows are exploited differently from stack-based overflows. Heap memory is allocated dynamically at run time and typically contains program data. Exploitation is performed by corrupting this data in specific ways so that the application overwrites internal structures such as linked list pointers. The canonical heap overflow technique overwrites dynamic memory allocation linkage (such as malloc metadata) and uses the resulting pointer exchange to overwrite a program function pointer.

For example, in older versions of Linux, two buffers located next to each other on the heap could cause the first buffer to overwrite the metadata of the second buffer. By setting the second buffer's in-use bit to zero and setting its length to a small negative value that allows null bytes to be copied, when the program calls free() on the first buffer, it will try to consolidate the two buffers into one. When this happens, the buffer that is supposed to be freed is expected to contain two pointers, FD and BK, in the first 8 bytes of the previously allocated buffer. BK is written to FD and can be used to overwrite a pointer.

How the heap is organized


Memory can be occupied (allocated) or free. The figure shows dynamic memory.
  • SSize is the size of the previous memory block, provided that it is free.
  • Size is the size of this memory block, to which 2 status bits are added.
  • Data is user data.
  • Fd is a pointer to the next free block.
  • Bk is a pointer to the previous free block.
  • Free is free memory.

Heap Overflow in Programming

Thus, no two free blocks can be neighbors. In addition, at the boundary between occupied and free system memory there is a specially handled free W-block.

Heap Overflow in Programming

Blocks are organized into lists (bins) as follows.

Heap Overflow in Programming

The unlink method is used to remove a free block from a list.
void unlink(S, BK, FD){
BK = S->bk;
FD = S->fd;
FD->bk=BK;
FD->fd=FD;
}

Allocating and freeing memory


Let us look at how mmap works. In the first step, the arrays (bins) of the required sizes (for example, 24 bytes) are checked. If a suitable block exists, it is detached using unlink.

Heap Overflow in Programming

In the second step, if the block is large enough, it is split into two parts. The first part is allocated, and the second is redistributed into another array.

Heap Overflow in Programming

In the third step, if no block of the required size was found, the W-block is checked. If it fits, the procedure from step two is performed on it. If the W-block turns out to be too small, sbrk() and mmap() are used to extend the available memory. The free method is the exact opposite of mmap.

Heap buffer overflow


A heap overflow is a type of buffer overflow that occurs in the heap data area. Heap memory is allocated dynamically by the application at run time and typically contains program data. Exploitation is performed by corrupting this data in a particular way so that the application overwrites internal structures such as linked list pointers. The canonical heap overflow technique overwrites the dynamic memory allocation linkage (such as malloc metadata) and uses pointer exchange to overwrite a pointer to a program function.

Consequences

An accidental overflow may result in data corruption or unexpected behavior in any process that accesses the affected memory area. On operating systems without memory protection, this could be any process on the system.

For example, the Microsoft JPEG GDI+ buffer overflow vulnerability could make remote code execution possible on a vulnerable machine.

Jailbreaking iOS often uses a heap overflow to execute arbitrary code.

Detection and prevention

As with buffer overflows, there are three main ways to protect against heap overflows. Some modern operating systems, such as Windows and Linux, provide some implementation of all three.

  • Prevent execution of the payload by separating code and data, usually with hardware features such as the NX bit
  • Introduce randomization so that the heap is not located at a fixed offset, usually with kernel features such as ASLR (address space layout randomization)
  • Introduce sanity checks into the heap manager

Starting with version 2.3.6, GNU libc includes protections that can detect heap overflows after the fact, for example by checking pointer consistency when unlink is called. However, it was shown almost immediately that these protections against earlier exploits can be circumvented as well. In addition, Linux has supported ASLR since 2005, although PaX introduced a better implementation several years earlier. Linux has also supported the NX bit since 2004.

Microsoft has included protections against heap buffer overflows since April 2003 in Windows Server 2003 and since August 2004 in Windows XP with Service Pack 2. These mitigations were safe unlinking and heap entry header cookies. Later versions of Windows, such as Vista, Server 2008, and Windows 7, include: removal of commonly used data structures, randomization of heap entry metadata, an expanded role for the heap header cookie, a randomized heap base address, function pointer encoding, termination on heap corruption, and algorithm variation. Ordinary Data Execution Prevention (DEP) and ASLR also help mitigate this attack.

See also

  • [[b8722]]
  • [[b8723]]
  • Heap spraying
  • Stack buffer overflow
  • Exploit
  • Shellcode

See also

created: 2021-11-27
updated: 2026-09-29
158



Was this answer useful?
Choose a quick rating so we can improve the next answer for you.
How satisfied are you?


Comments

To leave a comment

If you have any suggestion, idea, thanks or comment, feel free to write. We really value feedback and are glad to hear your opinion.
To reply

Lectures and tutorial on "Algorithmization and programming. Structural programming. C language"

Terms: Algorithmization and programming. Structural programming. C language