Principles of designing a standalone user-mode debugger
HyperDbg is a unique debugger. The intention of creating this tool was, of course, not just debugging. Regardless of debugging components, it also aims to bring a framework to use the innovative feature of processors for debugging and analyzing binaries. Here we discuss some of the principles of the user debugger.
In the user debugger, we aimed to make a lightweight debugger that integrates the different subsystems of the HyperDbg into a fast, reliable user-mode debugging tool.
There is no need to use kernel debugging facilities to debug a user-mode application when the entire debugging module runs on user mode. The user doesn't care what is running on the kernel side and just needs to debug the user mode side of the binary. That's why HyperDbg is armed with a user debugger.
Although, the user debugger is primarily designed to use in the user-mode application. Still, it can change the kernel mode and monitor kernel events.
HyperDbg didn't use any debugging API in the debugging process. Everything is handled at the hypervisor level. This brings a huge benefit as even Windows is unaware that the process is under debugging.
Moreover, all of the magical features of HyperDbg are integrated into the user debugger. You can debug a user-mode process with the capabilities of HyperDbg debugger.
Additionally, you can simultaneously debug multiple processes with kernel-mode debugging facilities on a single debugging session.
Whenever you start a new process or attach to the target process, HyperDbg starts the "Thread Interception" phase.
During this phase, HyperDbg will intercept every thread that tries to run an instruction from the user-mode (Not kernel mode). Thus, some threads might remain in kernel mode without returning to the user mode and be left unintercepted.
The "Thread Interception" phase continues as long as you run step-in or a step-over instruction or ultimately continue the process.
After that, the new threads or those threads that return from the kernel won't be caught by HyperDbg, and you need to pause the process again to intercept these threads.
It's recommended to keep interacting with the target process to make as many threads running to intercept them in the user debugger.
The user debugger can only step through the user-mode instructions. You can't step through the kernel-mode instructions.