Dynamic Reverse Engieerning#

After installing the Safe Exam Browser, I launched the application to observe the log of the application to briefly analyze the execution flow of the application. The execution flow was:

1. Initialize startup procedure
   • Load system language
2. Start communication host
   • Open a net pipe: net.pipe://localhost/safeexambrowser/runtime/[UUID]
3. Verify application and runtime integrity
4. Initialize new session configuration
   • Load Client-ID
   • Load Runtime-ID
   • Load Session-ID
5. Initialize application configuration
6. Validate version restrictions
7. Validate remote session policy
8. Validate session integrity
   • Disable sticky keys
   • Start cursor verification
   • Start ease of access verification
9. Validate virtual machine policy
10. Continue OR Exit

I launched Process Monitor (ProcMon) and examined the 2026-07-21_20h46m25s_Runtime.log file. The log entries indicated that the VM validation process began at 20:46:56.551 and ended at 20:46:58.012.

1
$ cat 2026-07-21_20h46m25s_Runtime.log | grep "virtual"
2
2026-07-21 20:46:56.551 [11] - INFO: Validating virtual machine policy...
3
2026-07-21 20:46:57.996 [11] - DEBUG: [VirtualMachineDetector] Computer 'THERUSTYMATE' appears to be a virtual machine (QEMU, 100%).
4
2026-07-21 20:46:58.012 [11] - ERROR: Detected virtual machine while SEB is not allowed to be run in a virtual machine! Aborting...

After exporting all ProcMon events to a CSV file, I used grep with an extended regular expression to filter events recorded during the VM validation process:

cat ProcMon_SEB.csv | grep ".531"

and

grep -E '^"?20:46:(56|57|58)\.' ProcMon_SEB.csv

1
$ cat ProcMon_SEB.csv | grep ".531"
2
"8:46:23.9355311 PM","SafeExamBrowser.exe","6960","Load Image","C:\Windows\System32\msvcp_win.dll","SUCCESS","Image Base: 0x7ffef0010000, Image Size: 0xa3000"

1
$ cat ProcMon_SEB.csv | grep -E "^......56|^......57|^......58"
2
...
3
"8:46:56.9634035 PM","SafeExamBrowser.exe","6960","RegOpenKey","HKLM\System\CurrentControlSet\Enum\SWD\MMDEVAPI\MICROSOFTGSWAVETABLESYNTH","SUCCESS","Desired Access: Query Value, Maximum Allowed"
4
"8:46:56.9634116 PM","SafeExamBrowser.exe","6960","RegQueryValue","HKLM\System\CurrentControlSet\Enum\SWD\MMDEVAPI\MicrosoftGSWavetableSynth\HardwareID","NAME NOT FOUND","Length: 4,108"
5
"8:46:56.9634185 PM","SafeExamBrowser.exe","6960","RegCloseKey","HKLM\System\CurrentControlSet\Enum\SWD\MMDEVAPI\MicrosoftGSWavetableSynth","SUCCESS",""
6
...

Then, I used a custom python script (procmon_csv_parser.py) to list unique items on selected columns within the ProcMon CSV file.

1
$ python3 procmon_csv_parser.py -c 5 -f ../research/dynamic/outputs/ProcMon_SEB_modified02.csv
2
...
3
HKLM\System\CurrentControlSet\Services\Tcpip\Parameters
4
HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\Domain
5
HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\Hostname
6
...

To further analyze the events on Linux, I used ProcMonParser, a tool I had previously developed to analyze ProcMon events recorded during SEB execution.

The stack trace suggested that the native DLL seb_x64.dll invoked the Windows API function RegGetValueW to retrieve registry values. Based on these observations, I suspected that seb_x64.dll might be involved in VM validation process of SEB.

Static Reverse Engineering#

To investigate this further, I began a static analysis of seb_x64.dll.

1
$ file seb_x64.dll
2
seb_x64.dll: PE32+ executable for MS Windows 6.00 (DLL), x86-64, 12 sections

Unexpectedly, I discovered that the DLL was protected by Themida Protector.

First, I expected the decompiled output to be unreliable due to this protection. However, I was still able to partially recover some functions listed in the Export Address Table (EAT).

Before proceeding with the reverse engineering of seb_x64.dll, I first analyzed the publicly available .NET component.

1
public bool IsVirtualMachine(out string manufacturer, out int probability)
2
{
3
    var isVirtualMachine = false;
4

5
    manufacturer = default;
6
    probability = default;
7

8
    try
9
    {
10
        isVirtualMachine = Native.IsVirtualMachine(out var bstr, out probability);
11

12
        if (bstr != IntPtr.Zero)
13
        {
14
            manufacturer = Marshal.PtrToStringBSTR(bstr);
15
            Marshal.FreeBSTR(bstr);
16
        }
17
    }
18
    catch (DllNotFoundException)
19
    {
20
        logger.Warn("Integrity module is not available!");
21
    }
22
    catch (Exception e)
23
    {
24
        logger.Error("Unexpected error while attempting to query virtual machine information!", e);
25
    }
26

27
    return isVirtualMachine;
28
}

Through this analysis, I identified the following function signature for IsVirtualMachine, which could help me interpret the decompiled code of seb_x64.dll:

1
extern "C" bool __cdecl IsVirtualMachine(
2
    BSTR* manufacturer,
3
    int* probability
4
);

Upon examining the decompiled code, I immediately noticed several hypervisor-related strings.

I began analyzing the code associated with these strings using a GPT-based LLM, focusing on StringConstruct, which I suspected was related to C++ string handling.

1
char IsVirtualMachine(BSTR *manufacturer,int *probability)
2

3
{
4
    char codeSignatureValid;
5
    byte vmProbability;
6
    undefined4 randomSeed;
7
    uint randomValue;
8
    int randomVendorValue;
9
    VmContext *vmContext;
10
    BSTR manufacturerBstr;
11
    VendorList *vendorAllocation;
12
    MsvcString *manufacturerStringA;
13
    WString32 *wideManufacturer;
14
    wchar_t *manufacturerWideBuffer;
15
    char isVirtualMachine;
16
    int probabilityValue;
17
    MsvcString *vendorList;
18
    MsvcString *selectedVendor;
19
    VmContext detectionContext;
20
    VmContext manufacturerContext;
21
    VmContext probabilityContext;
22
    WString32 manufacturerString;
23
    WString32 fallbackManufacturerString;
24
    MsvcString emptyVendor;
25
    MsvcString selectedVendorCopy;
26
    MsvcString manufacturerStorage;
27
    MsvcString movedVendor;
28

29
                    /* 0x128150  4  IsVirtualMachine */
30
    codeSignatureValid = VerifyCodeSignature();
31
    if (codeSignatureValid == '\0') {
32
        vendorAllocation = alloc(168);
33
        if (vendorAllocation == (VendorList *)0x0) {
34
            vendorList = (MsvcString *)0x0;
35
        }
36
        else {
37
            vendorAllocation->count = 5;
38
            StringConstruct(vendorAllocation->vendors,"Hyper-V");
39
            StringConstruct(vendorAllocation->vendors + 1,"KVM");
40
            StringConstruct(vendorAllocation->vendors + 2,"QEMU");
41
            StringConstruct(vendorAllocation->vendors + 3,"VMware");
42
            StringConstruct(vendorAllocation->vendors + 4,"VirtualBox");
43
            vendorList = vendorAllocation->vendors;
44
        }
45
        randomSeed = std_time(0);
46
        std_srand(randomSeed);
47
        randomValue = std_rand();
48
        isVirtualMachine = (randomValue & 1 ^ (int)randomValue >> 0x1f) - ((int)randomValue >> 0x1f) == 1;
49
        if (!(bool)isVirtualMachine) {
50
            selectedVendor = StringConstruct(&emptyVendor,"");
51
        } else {
52
            randomVendorValue = std_rand();
53
            selectedVendor = StringCopy(&selectedVendorCopy,vendorList + randomVendorValue % 5);
54
        }
55
        manufacturerStringA = StringMove(&movedVendor,selectedVendor);
56
        wideManufacturer = ConvertStringToWString(&fallbackManufacturerString,manufacturerStringA);
57
        manufacturerWideBuffer = WStringData(wideManufacturer);
58
        manufacturerBstr = (BSTR)(*(code *)0x8000000000000002)(manufacturerWideBuffer);
59
        *manufacturer = manufacturerBstr;
60
        WStringDestroy(&fallbackManufacturerString);
61
        if (!(bool)isVirtualMachine) {
62
            StringDestroy(&emptyVendor);
63
            probabilityValue = 0;
64
        }
65
        else {
66
            StringDestroy(&selectedVendorCopy);
67
            probabilityValue = std_rand();
68
            probabilityValue = probabilityValue % 0x65;
69
        }
70
        *probability = probabilityValue;
71
    }
72
    else {
73
        vmContext = BuildVmContext(&detectionContext);
74
        isVirtualMachine = IsVmDetected(vmContext);
75
        vmContext = BuildVmContext(&manufacturerContext);
76
        manufacturerStringA = GetVmManufacturerString(&manufacturerStorage,vmContext);
77
        wideManufacturer = ConvertStringToWString(&manufacturerString,manufacturerStringA);
78
        manufacturerWideBuffer = WStringData(wideManufacturer);
79
        manufacturerBstr = (BSTR)(*(code *)0x8000000000000002)(manufacturerWideBuffer);
80
        *manufacturer = manufacturerBstr;
81
        WStringDestroy(&manufacturerString);
82
        vmContext = BuildVmContext(&probabilityContext);
83
        vmProbability = GetVmProbability(vmContext);
84
        *probability = (uint)vmProbability;
85
    }
86
    return isVirtualMachine;
87
}

seb_types.h for Ghidra:

1
typedef struct MsvcString {
2
    union {
3
        char inlineBuffer[16];
4
        char *heapBuffer;
5
    } storage;
6

7
    unsigned long long size;
8
    unsigned long long capacity;
9
} MsvcString;
10

11
typedef struct WString32 {
12
    union {
13
        wchar_t inlineBuffer[8];
14
        wchar_t *heapBuffer;
15
    } storage;
16

17
    unsigned long long size;
18
    unsigned long long capacity;
19
} WString32;
20

21
typedef struct VendorList {
22
    unsigned long long count;
23
    MsvcString vendors[5];
24
} VendorList;
25

26
typedef struct VmContext {
27
    uint64_t bits[2];
28
} VmContext;

After completing the decompilation, I used a GPT-based LLM to transform the decompiled output into more human-readable C++ code.

1
bool IsVirtualMachine(BSTR* manufacturer, int* probability)
2
{
3
    if (!VerifyCodeSignature())
4
    {
5
        VendorList* vendors = alloc(168);
6

7
        if (vendors)
8
        {
9
            vendors->count = 5;
10
            vendors->vendors[0] = "Hyper-V";
11
            vendors->vendors[1] = "KVM";
12
            vendors->vendors[2] = "QEMU";
13
            vendors->vendors[3] = "VMware";
14
            vendors->vendors[4] = "VirtualBox";
15
        }
16

17
        std::srand(std::time(nullptr));
18

19
        bool isVirtualMachine = (std::rand() % 2) != 0;
20

21
        std::string vendor;
22

23
        if (isVirtualMachine)
24
        {
25
            vendor = vendors->vendors[std::rand() % 5];
26
        }
27

28
        std::wstring wideVendor(vendor.begin(), vendor.end());
29

30
        // Failed Recovery:
31
        // This can be string related function (different compare to the below Themida protected function.)
32
        *manufacturer =
33
            reinterpret_cast<BSTR (*)(const wchar_t*)>(
34
                0x8000000000000002
35
            )(wideVendor.c_str());
36

37
        *probability =
38
            isVirtualMachine ? std::rand() % 101 : 0;
39

40
        return isVirtualMachine;
41
    }
42

43
    VmContext detectionContext;
44
    VmContext manufacturerContext;
45
    VmContext probabilityContext;
46

47
    bool isVirtualMachine =
48
        IsVmDetected(
49
            BuildVmContext(&detectionContext)
50
        );
51

52
    std::string manufacturerString =
53
        GetVmManufacturerString(
54
            BuildVmContext(&manufacturerContext)
55
        );
56

57
    std::wstring wideManufacturer(
58
        manufacturerString.begin(),
59
        manufacturerString.end()
60
    );
61

62
    // Failed Recovery:
63
    // This function is likely the main VM detection function protected by Themida.
64
    *manufacturer =
65
        reinterpret_cast<BSTR (*)(const wchar_t*)>(
66
            0x8000000000000002
67
        )(wideManufacturer.c_str());
68

69
    *probability =
70
        GetVmProbability(
71
            BuildVmContext(&probabilityContext)
72
        );
73

74
    return isVirtualMachine;
75
}

Vulnerability Research#

Although the reconstructed code was not entirely reliable, I was still able to make several observations and draw tentative conclusions from the analysis.

Initially, I suspected that an attacker could bypass the VM detection mechanism by manipulating the random number generator (RNG), as the program appeared to return a randomized VM detection result when code signature verification failed.

I confirmed that the component returns a randomized result when integrity verification fails. I suspected that this might be an intentional deceptive behavior.

I confirmed that VM validation process had been successfully bypassed.

However, SEB still detected that it was running inside a virtual machine.

After further investigating this vulnerability, I identified two additional factors that affect the VM validation process.

• First, I discovered that the native component seb_x64.dll was not the only component involved in the VM validation process.
• Second, I found that, in my specific testing environment, bypassing remote session detection was also necessary to determine whether the vulnerability could actually be exploited.

1
...
2
public bool IsVirtualMachine()
3
{
4
    var isVm = false;
5

6
    isVm |= HasNoSystemHardware();
7
    isVm |= HasVirtualDevice();
8
    isVm |= HasVirtualMacAddress();
9
    isVm |= IsVirtualCpu();
10
    isVm |= IsVirtualRegistry();
11
    isVm |= IsVirtualSystem(systemInfo.BiosInfo, systemInfo.Manufacturer, systemInfo.Model);
12
    isVm |= integrityModule.IsVirtualMachine(out var manufacturer, out var probability);
13

14
    logger.Debug($"Computer '{systemInfo.Name}' appears {(isVm ? "" : "not ")}to be a virtual machine{(isVm ? $" ({manufacturer}, {probability}%)" : "")}.");
15

16
    return isVm;
17
}
18
...

Another bypass method I attempted involved manipulating the VM validation result using dnSpy and combining it with the DLL hijacking technique I had previously tested.

As a result, I successfully bypassed both the VM and remote session validation mechanisms, allowing me to access external resources while running SEB.

Using dnlib, I developed a PoC tool to automate the SEB patching process previously validated manually using dnSpy. Testing confirmed that the PoC tool also successfully bypasses VM validation.

The setup procedure is as follows:

1. Install virtio-win-0.1.285 drivers.
2. Use VirtIo video model type in KVM/QEMU.
3. Reboot.
4. Install SafeExamBrowser.
5. Apply EVirtPatch.
6. Manually apply seb_x64.dll DLL proxy.

Note: If the SEB displays internal/external display-related warnings, please modify the VM configuration to use VirtIO drivers for the video model.

Conclusion#

Using the SafeExamBrowser.Monitoring.dll patch, the virtual machine and remote session detection of the SafeExamBrowser was successfully bypassed. However, this patch triggers both the code and runtime integrity verification failure. Furthermore, the 7th layer of the virtual machine detection layers and the 2nd layer of the remote session detection layers can log both the virtual machine detection results and the remote session detection results.

Using the seb_x64.dll patch, the integrity verification results were manipulated, leading to a successful integrity verification bypass. The patch also manipulates the external virtual machine and remote session detections (7th layer of virtual machine detection layers & 2nd layer of remote session detection layers), leading to a complete detection bypass.

Fedora Linux Host#

Debian Linux Host#

Windows 11 Home Guest#

Windows 11 Pro Guest#

1. PE Validation#

A custom x64 Windows PE packer first validates IMAGE_DOS_HEADER, IMAGE_NT_HEADERS64, and the architecture (IMAGE_FILE_HEADER) to confirm the given raw data is in PE format.

Validating the IMAGE_DOS_HEADER:

1
#define IMAGE_DOS_SIGNATURE 0x5A4D
2

3
IMAGE_DOS_HEADER *dos = (IMAGE_DOS_HEADER *)data; // Type cast to a pointer of IMAGE_DOS_HEADER.
4
if (dos->e_magic != IMAGE_DOS_SIGNATURE) { // Verify magic bytes: 0x5A4D (MZ).
5
    return FALSE; // The given data does not contain a valid DOS header.
6
}
7
return TRUE; // DOS header is validated.

Validating the IMAGE_NT_HEADERS64:

1st: Identify NT header offset from the field, e_lfanew in DOS header:

1
// Base Address + dos->e_lfanew (NT header offset)
2
IMAGE_NT_HEADERS64 *nt = (IMAGE_NT_HEADERS64 *)(data + dos->e_lfanew);

2nd: Validate the header using the Signature field:

1
#define IMAGE_NT_SIGNATURE 0x00004550
2

3
// Verify NT header signature: 0x00004550
4
if (nt->Signature != IMAGE_NT_SIGNATURE) {
5
    // 1. The given data does not contain a valid NT header.
6
    // 2. The identified NT header offset is invalid.
7
    return FALSE;
8
}
9
return TRUE; // NT header is validated.

Validating the architecture (IMAGE_FILE_HEADER):

1st: Identify File header using the NT header:

1
IMAGE_FILE_HEADER *file = (IMAGE_FILE_HEADER *)&nt->FileHeader;

2nd: Validate the architecture using the field, Machine:

1
#define IMAGE_FILE_MACHINE_AMD64 0x8664
2

3
if (file->Machine != IMAGE_FILE_MACHINE_AMD64) {
4
    // 1. The file is not x64.
5
    // 2. The given file is not valid PE file.
6
    return FALSE;
7
}
8
return TRUE; // The architecture is x64.

2. Memory Allocation#

After validating the headers, it will allocate and load the image into an executable memory region. This region often has RWX or RX permissions. Allocating RWX (Read-Write-eXecutable) memory region often draws suspicion as common executable programs often do not allocate a memory region with RWX permission for the program’s safety. However, executable packers and shellcode loaders often do this.

On the other hand, changing the memory permission from RW to RX is similar to JIT compiler behaviours, mainly used by Java, C#, and JavaScript. This “can” be less suspicious as common programs written in Java and C# can show similar behaviours (RW to RX transition). The JIT compilation approach is better if I was planning to create a defense evasive packer (formally known as crypter/protector).

In this example, RWX permission is simpler and still acceptable as many non-commercial and commercial packers use embedded RWX sections. These sections are quite different as it doesn’t allocate RWX regions in memory at runtime, it has its own embedded RWX sections.

For example, Themida, a commercial PE protector creates a RWX section embedded within the PE file:

Another example, UPX creates a RWX section embedded within the PE file as well:

1
BYTE *image = (BYTE *)VirtualAlloc(
2
    NULL,                               // No designated memory address.
3
    nt->OptionalHeader.SizeOfImage,     // Allocation size = Image size.
4
    MEM_COMMIT | MEM_RESERVE,           // Allocate (commit) and reserve at the same time.
5
    PAGE_EXECUTE_READWRITE              // RWX (Read-Write-eXecutable).
6
);

3. Memory Mapping#

After allocating the memory, the packer needs to copy into the allocated memory:

1
memcpy(image, data, nt->OptionalHeader.SizeOfHeaders);

SizeOfImage is the amount of memory required for the image and SizeOfHeaders is the size of headers. the packer loads the headers first as the file layout is not identical to the memory layout.

The VirtualAddress and PointerToRawData store different data. VirtualAddress stores the section RVA and PointerToRawData stores the file offset. Therefore, mapping the file layout straight to the memory will result in an error.

RVA, or Relative Virtual Address is the offset of the address from the image base.

After mapping the headers, the packer then maps each section. As it was previously mentioned, VirtualAddress stores the “memory offset”. In order to map each section, each VirtualAddress must point to the corresponding section.

Therefore, each section must be copied from its file offset (PointerToRawData) to its memory offset (VirtualAddress, which represents the section RVA).

1
IMAGE_SECTION_HEADER *sections = IMAGE_FIRST_SECTION(nt);
2
for (WORD i = 0; i < nt->FileHeader.NumberOfSections; i++) {
3
    // Skip if the section is empty
4
    if (sections[i].SizeOfRawData == 0) continue;
5

6
    // Map the section at its RVA
7
    memcpy(
8
        image + sections[i].VirtualAddress,     // RVA in Mapped Image
9
        data + sections[i].PointerToRawData,    // File Offset
10
        sections[i].SizeOfRawData               // Size of Section
11
    );
12
}

4. Relocation#

When a program is linked, the linker records the locations of values that require base relocation if the image is loaded at a different base address. To adjust these addresses, “delta” value must be calculated:

1
// image is the pointer to the allocated memory (loaded image base)
2
BYTE *image = (BYTE *)VirtualAlloc(
3
    NULL,
4
    nt->OptionalHeader.SizeOfImage,
5
    MEM_COMMIT | MEM_RESERVE,
6
    PAGE_EXECUTE_READWRITE
7
);
8

9
// ...
10

11
// Delta = Loaded Image Base - Preferred Image Base
12
ULONG_PTR delta = (ULONG_PTR)image - nt->OptionalHeader.ImageBase;

The .reloc section stores base relocation entries that describe the locations of values that must be adjusted when the image is loaded at a different base address.

1
IMAGE_DATA_DIRECTORY reloc_dir = nt->OptionalHeader.DataDirectory[IMAGE_DIRECTORY_ENTRY_BASERELOC];

Each TypeOffset entry contains the relocation type in the high 4 bits and an offset within the 4 KB relocation page in the lower 12 bits. The high 4 bits specify the relocation type. In case of x64 images, this is commonly IMAGE_REL_BASED_DIR64. The low 12 bits are the offset within the 4 KB relocation page.

The VirtualAddress can be found in the relocation table. This VA is NOT a virtual address, it is a RVA (Relative Virtual Address) which means it is an offset.

Using this information, the packer is now able to adjust absolute addresses:

1
// First table.
2
IMAGE_BASE_RELOCATION *reloc = (IMAGE_BASE_RELOCATION *)(image + reloc_dir.VirtualAddress);
3

4
// Loop through each relocation tables
5
while (reloc->VirtualAddress > 0) {
6
    DWORD count = (reloc->SizeOfBlock - sizeof(IMAGE_BASE_RELOCATION)) / sizeof(WORD);
7
    WORD *list = (WORD *)(reloc + 1);
8

9
    for (DWORD i = 0; i < count; i++) {
10
        if (list[i] != 0) {
11
            DWORD type = list[i] >> 12;         // TypeOffset High 4 Bits (DIR64)
12
            DWORD offset = list[i] & 0x0FFF;    // TypeOffset Low 12 Bits (Offset)
13

14
            if (type == IMAGE_REL_BASED_DIR64) {
15
                // Calculate the address of the relocation target VA.
16
                ULONG_PTR *patch_addr = (ULONG_PTR *)(image + reloc->VirtualAddress + offset);
17

18
                // Adjust the address with "delta" value.
19
                *patch_addr += delta;
20
            }
21
        }
22
    }
23

24
    // Next relocation table.
25
    reloc = (IMAGE_BASE_RELOCATION *)((BYTE *)reloc + reloc->SizeOfBlock);
26
}

5. IAT Resolve#

The .idata section stores import-related data that describes the DLLs and functions that must be resolved at load time.

1
IMAGE_DATA_DIRECTORY import_dir = nt->OptionalHeader.DataDirectory[IMAGE_DIRECTORY_ENTRY_IMPORT];

the packer will first identify and read import descriptors.

1
// image is the pointer to the allocated memory (loaded image base)
2
BYTE *image = (BYTE *)VirtualAlloc(
3
    NULL,
4
    nt->OptionalHeader.SizeOfImage,
5
    MEM_COMMIT | MEM_RESERVE,
6
    PAGE_EXECUTE_READWRITE
7
);
8

9
// ...
10

11
// Import Descriptor Location = Image Base + Virtual Address
12
IMAGE_IMPORT_DESCRIPTOR *import_desc = (IMAGE_IMPORT_DESCRIPTOR *)(image + import_dir.VirtualAddress);

IMAGE_IMPORT_DESCRIPTOR is a table that describes information about imported functions of DLLs:

The Name field describes the name of DLL that needs to be imported. The packer then loads each imported DLL using LoadLibraryA():

1
for (; import_desc->Name; import_desc++) {
2
    // DLL Name from IMAGE_IMPORT_DESCRIPTOR.
3
    char *mod_name = (char *)(image + import_desc->Name);
4

5
    // Import the DLL.
6
    HMODULE h_mod = LoadLibraryA(mod_name);
7

8
    // ...
9
}

Now, the packer will loop through the imported functions for each DLL and resolve them. Before resolving the function, the packer needs to determine whether the function is imported by ordinal or by name.

1
#define IMAGE_SNAP_BY_ORDINAL64(Ordinal) ((Ordinal & IMAGE_ORDINAL_FLAG64)!=0)
2

3
for (; original_thunk->u1.AddressOfData; thunk++, original_thunk++) {
4
    if (IMAGE_SNAP_BY_ORDINAL64(original_thunk->u1.Ordinal)) {
5
        // ...
6
    }
7

8
    // ...
9
}

The packer then resolves each imported function and writes its address into the corresponding IAT “thunk”.

1
typedef struct _IMAGE_THUNK_DATA64 {
2
    union {
3
        ULONGLONG ForwarderString;
4
        ULONGLONG Function;
5
        ULONGLONG Ordinal;
6
        ULONGLONG AddressOfData;
7
    } u1;
8
} IMAGE_THUNK_DATA64;
9
typedef IMAGE_THUNK_DATA64 *PIMAGE_THUNK_DATA64;

IMAGE_THUNK_DATA64 represents an entry in the INT or IAT. In the INT, the entry describes how a function is imported, while in the IAT, the same-sized entry is eventually replaced with the resolved function address.

The most significant bit (bit 63) of a 64-bit thunk entry indicates whether the import is by ordinal. If the bit is set, the low 16 bits contain the ordinal value.

If the function is resolved using the name, the process is slightly more involved.

The AddressOfData points to the IMAGE_IMPORT_BY_NAME object that contains more detailed information:

1
typedef struct _IMAGE_IMPORT_BY_NAME {
2
    WORD Hint;
3
    char Name[1];
4
} IMAGE_IMPORT_BY_NAME,*PIMAGE_IMPORT_BY_NAME;

The Name field holds the name of the function in the DLL. The Hint field provides an index hint into the ENT (Export Name Table). Windows uses this Hint field to look up the function faster, however, packers often do not implement this.

1
#define IMAGE_ORDINAL64(Ordinal) (Ordinal & 0xffffull)
2

3
if (IMAGE_SNAP_BY_ORDINAL64(original_thunk->u1.Ordinal)) {
4
    // Low 16 bits = Ordinal.
5
    LPCSTR ordinal = (LPCSTR)IMAGE_ORDINAL64(original_thunk->u1.Ordinal);
6

7
    // Import the function using the ordinal number.
8
    thunk->u1.Function = (ULONG_PTR)GetProcAddress(h_mod, ordinal);
9
} else {
10
    //
11
    IMAGE_IMPORT_BY_NAME *iibn = (IMAGE_IMPORT_BY_NAME *)(image + original_thunk->u1.AddressOfData);
12

13
    // Import the function using the name
14
    thunk->u1.Function = (ULONG_PTR)GetProcAddress(h_mod, (LPCSTR)iibn->Name);
15
}

6. Calling the Entry Point#

Now, finally the packer will call the “original entry point” of the mapped program:

1
typedef void (*ORIGINAL_ENTRY_POINT)(void);
2

3
ORIGINAL_ENTRY_POINT oep = (ORIGINAL_ENTRY_POINT)(image + nt->OptionalHeader.AddressOfEntryPoint);
4
oep();

Out Of Scope#

• Packed or self-modifying binaries
• Kernel-mode drivers
• Heavy obfuscation frameworks (e.g., VM-based obfuscation)

Dev Environment#

Testing Environment#

1. Windows 10 (QEMU virtualized)
2. Ubuntu 24.04 LTS (Acer Laptop)

Compilers#

• Windows Compiler
  • x86_64-w64-mingw32-gcc (GCC) 13-win32
    • x86_64-w64-mingw32-gcc -O2 -o [OUTPUT] [C_CODE]
• Ubuntu Compiler
  • Ubuntu clang version 18.1.3 (1ubuntu1)
    • clang -O2 -s -fvisibility=hidden [C_CODE] -o [OUTPUT]

Fundamental Concepts & Definitions#

Fundamentally, commercial decompilers often statically reconstruct C or C-like pseudocode from assembly instructions within a binary. A decompilation process often looks like this:

• Disassembly
  • Translates machine language into assembly language
  • Recovers function boundaries
  • Identifies code sections and data sections
• Control Flow Analysis
  • Analyzes branches (if)
  • Analyzes loops (for/while)
  • Constructs CFG (Control Flow Graph)
• Data Flow Analysis
  • Abstracts variables from registers
  • Analyzes use of stack and heap memory
  • Transforms IR (Intermediate Representation) into SSA (Static Single Assignment) form
• Type Recovery
  • Identifies integers and pointers
  • Reconstructs structures (struct)
  • Infers function parameters and return types
• High-level Reconstruction
  • Converts condition branches (if/else) to C-like pseudocode
  • Converts loops (for/while) to C-like pseudocode
  • Converts function calls to C-like pseudocode
  • Removes or restructures low-level control flow (e.g., goto)
  • Simplifies expressions
• Symbol Recovery
  • Applies debug symbols (if present)
  • Matches known function names (e.g. libraries)

Control Flow Graph (CFG) is a graph representation of all possible execution paths (flows) within a function, based on branches, loops, and jump instructions.

Intermediate Representation (IR) is an abstract representation of code used between source code and machine language for program analysis and transformation.

Static Single Assignment (SSA) is a form of code representation in which each variable is assigned exactly once, enabling precise data flow analysis.

Techniques & Strategies#

Commercial decompilers often cannot precisely determine runtime-dependent values (e.g., return addresses, timestamps, or environment-dependent data). As decompilers heavily rely on the CFG, conditional branches that depend on runtime values can be exploited to distort CFG reconstruction, resulting in misleading or junk decompiled output.

Validation Method#

To demonstrate the effectiveness of the outcomes, simple samples written in C and C++ will be divided into two groups: those with BRKDEC applied and those without. For a detailed inspection, IDA Free, Ghidra (+ Cutter), and Binary Ninja will be used to directly compare and analyze the decompilation results.

I will compare identifiable functions within an obfuscated binary (at least functions whose execution flow can be verified or inferred from decompilation results) with functions inside a regular binary, and quantify the differences for validation.

Validation Standards#

• Pseudocode readability
• Variable recovery accuracy
• Function boundary accuracy

AND

• Loop count
• Conditional statement count
• flow change count

Validation Assumption#

• The compiled binary does not contain debug symbols
• The compiled binary is built with recommended optimizations (-O2)

IDA Free#

In the case of IDA Free, functions were not decompiled properly for either binary. The Decompiler Explorer was used to verify and compare some decompilation results. In the regular binary, the following numbers of branches were identified:

• 0 loops
• 19 conditional statements
• 14 flow changes

In the code after BRKDEC was applied, the following were identified:

• 0 loops
• 3 conditional statements
• 8 flow changes

Therefore, the impact was determined to be:

• 0% on loops
• 84% on conditional statements
• 42% on flow changes

Since IDA Free failed to decompile the baseline binary reliably, the post-transformation result should be interpreted as an additional degradation rather than an absolute decompilation failure rate. Based on the reduction in recovered conditionals and the near-complete loss of logic, the additional impact was assessed to be over 80%.

Ghidra#

In the case of Ghidra, most recovered functions degenerated into stubs centered around thrd_yield(). In the regular binary, the following numbers of branches were identified:

• 1 loop
• 17 conditional statements
• 4 flow changes

In the code after BRKDEC was applied, the following numbers of branches were identified:

• 0 loops
• 0 conditional statements
• 0 flow changes

Therefore, the impact was determined to be:

• 100% on loops
• 100% on conditional statements
• 100% on flow changes

Taking into account potential errors and disassembly, the overall impact was determined to be over 90%.

Binarny Ninja#

In the case of Binary Ninja, functions were relatively restored to a normal state, but some executable code was treated as data, leading to losses during code recovery. In the regular binary, the following numbers of branches were identified:

• 1 loop
• 19 conditional statements
• 5 flow changes

In the code after BRKDEC was applied, the following were identified:

• 0 loops
• 19 conditional statements
• 14 flow changes

Therefore, the impact was determined to be:

• 100% on loops
• 0% on conditional statements
• 180% on flow changes

Although the number of recovered conditional statements remained unchanged, the loss of loop recovery and the substantial increase in control-flow transfers significantly fragmented the reconstructed CFG. Because the overall algorithm remained partially interpretable, the net effectiveness was assessed at approximately 65%.

Windows#

Windows operates in the same way, but the specific figures may differ (in actual experimental data, IDA Free, Ghidra, and Binary Ninja all showed improved results). As a result of emulation using QEMU, the binary’s normal operation was tested 10 times, and all 10 tests confirmed normal functionality. The difficulty of reverse engineering, even after (inadvertently) applying symbols, showed approximately 5-10% greater impact compared to Linux binaries.