Binary Exploitation Basics All In One

Note

Obviously, this post is not finished yet, and is not going to be finished in the near future.
Please checkout this post instead:
https://archer-baiyi.github.io/2025/05/12/CTF/Pwn/Pwn%E5%85%A5%E9%97%A8/

Here is something basic for you, if you want to get into binary exploitation and maybe solving PWN challenges in CTF contests!

Note

We right now only look at x86_64 Linux binaries!

Preliminaries:

• Basic knowledge about computers
• Linux
• Basic C and assembly

Buffer Overflow

Most of the vulneralbilities are triggered by buffer overflows, which means we are reading / writing beyond what we are supposed to read and write.
Programs with user iteractions run on a linux system. The goal is to remotely execute code on the target system using the vulneralbility.
To understand this, a example of buffer overflow on the stack:

higher address
/     stack frame       /
|     return address    |
|     rbp               |
|     buffer            |     <- rsp
lower address

When writing the buffer, we write from lower address to higher address, if no size/bound check is happening, we can overflow rbp and return address of the stack frame.
return address control where rip will jump to, controlling rip means we can execute arbitrary code inside our memory space.

Shell Code Executing (ret2shell)

Shell code basically means when executed, a shell is spawned in the system.
If we overflow the return address and point return address to the stack, we can let the program (rip) jump to the stack and execute whatever code is loaded on the stack.
E.g. (addresses are simplified for readability):

0x7f00:   /   ...               /
0x7ef8:   |   return address    |
0x7ef0:   |   rbp               |
0x7ee8:   |   buffer            |

Assume we have a 16-byte shell code that can spawn a shell when executed. After writing the buffer, the memory layout should look like this:

0x7f00:   /   ...               /
0x7ef8:   |   0x7ee8            |
0x7ef0:   |   shell code        |
0x7ee8:   |   shell code        |

Then rip will jump to 0x7ee8 and CPU will execute the shell code to spawn a shell, where we can execute arbitrary command on the target system.