Nexus! _

General crackme: L5.0

This is the solution of a crackme that I wanted to tackle this weekend. It is on the harder side, as I wanted something challenging.

The main constraint I set was to avoid dynamic analysis; even with debugging protections, it is fairly simple to retrieve the flag.

Description

NEXUSCORE - VERY HARD CRACKME CHALLENGE

OBJECTIVE: Find the correct password to authenticate into NexusCore and retrieve the FLAG.

PROTECTIONS:

• Custom Virtual Machine - Password validation logic is executed inside an encrypted VM
• Anti-Debug - Multiple debugger detection mechanisms throughout execution
• Anti-Bruteforce - Limited attempts with exponential backoff delays
• Code Obfuscation - Opaque predicates, junk code, runtime transformations
• Timing Checks - Execution must complete within specific time constraints

WHAT YOU NEED TO DO:

1. Reverse engineer the custom VM implementation
2. Understand the bytecode encryption/decryption mechanism
3. Bypass or analyze the anti-debug protections
4. Either: a) Generate the correct password using a keygen (requires understanding the algorithm) b) Patch the binary to skip validation c) Emulate/instrument the VM to find the correct password
5. Enter the password and retrieve the flag: NEXUSCORE{VM_MASTERED}

Analysis of main @ [140001910]

Once I found the entry point, the main was pretty easy to locate because the file is compiled with MSVC. I tagged it and this is the first result:

int __fastcall main(int argc, const char **argv, const char **envp)
{
  FILE *v3; // rax
  FILE *v4; // rax
  __int64 v6; // [rsp+20h] [rbp-58h]
  char *v7; // [rsp+38h] [rbp-40h]
  char Buffer[32]; // [rsp+40h] [rbp-38h] BYREF

  print("========================================\n");
  print("         N E X U S C O R E >_<     \n");
  print("========================================\n");
  print("Enter password: ");
  v3 = _acrt_iob_func(1u);
  fflush(v3);
  v4 = _acrt_iob_func(0);
  if ( fgets(Buffer, 32, v4) )
  {
    v7 = Buffer;
    v6 = -1;
    do
      ++v6;
    while ( v7[v6] );
    if ( v6 && Buffer[v6 - 1] == 10 )
    {
      if ( (unsigned __int64)(v6 - 1) >= 0x20 )
        _canary_check();
      Buffer[v6 - 1] = 0;
    }
    if ( (unsigned int)check_user_pass(Buffer) )
    {
      print("\n[++] Access granted! Welcome to NexusCore.\n");
      print("[++] Flag: NEXUSCORE{VM_MASTERED}\n");
      return 0;
    }
    print("\n[-] Access denied.\n");
  }
  return 1;
}

Analysis of check_user_pass @ [140001780]

While analyzing the code I reconstructed a pseudo-struct that, at least for me, worked pretty well in understanding the algorithm:

struct VM_CTX
{
    uint32_t magic_1;
    uint32_t input_index;
    byte _pad1[24];
    VM_MEM vm_mem;
    uint32_t vsp;
    uint32_t vip;
    uint32_t do_jump_or_do_cmp;
    uint32_t flags;
    byte buff[16];
    uint64_t last_tick;
};
struct VM_MEM_LAYOUT
{
    uint8_t text[256];
    char user_input[32];
    uint8_t data_stack[3808];
};
union VM_MEM
{
    VM_MEM_LAYOUT layout;
    uint8_t raw[4096];
};

As we can see, using the struct in IDA gives a decent result:

_BOOL8 __fastcall check_user_pass(char *user_pass)
{
  unsigned __int64 i; // [rsp+20h] [rbp-10A8h]
  char curr_char; // [rsp+28h] [rbp-10A0h]
  unsigned __int64 user_input_len; // [rsp+30h] [rbp-1098h]
  BOOL is_flag_ok; // [rsp+38h] [rbp-1090h]
  ULONGLONG tick_now; // [rsp+40h] [rbp-1088h]
  VM_CTX vm_ctx; // [rsp+60h] [rbp-1068h] BYREF

  if ( (unsigned int)trials_linear_backoff() )
  {
    memset(&vm_ctx, 0, sizeof(vm_ctx));
    vm_ctx.magic_1 = 0xDEADBEEF;
    vm_ctx.flags = 0x1337BEEF;
    user_input_len = -1;
    do
      ++user_input_len;
    while ( user_pass[user_input_len] );
    for ( i = 0; i < 32; ++i )
    {
      vm_ctx.input_index = i;
      if ( i >= user_input_len )
        curr_char = 0;
      else
        curr_char = user_pass[i];
      vm_ctx.vm_mem.layout.user_input[i] = curr_char;// At 0x100 - 0x132 there is our input.
    }
    is_flag_ok = vm_entry(&vm_ctx, vm_ctx.vm_mem.layout.user_input);
    tick_now = GetTickCount64() - vm_ctx.last_tick;
    return tick_now <= 0x5DC && tick_now >= 0x320 && is_flag_ok;
  }
  else
  {
    print("Too many attempts. Wait...\n");
    return 0;
  }
}

Analysis of trials_linear_backoff @ [140001160]

This function is for anti-debug. LastGlobalTickCount and GlobalPasswordTries are global variables inside the .data region. Each time the program starts they are reset because the PE is loaded with a clean state, but this can prevent using a debugger to brute force passwords.

It can be trivially bypassed by NOP-ing the instructions in Ghidra/IDA (or any patching tool you prefer). After patching, the password can be cracked assuming the attacker has enough computational power and the password length is reasonable.

__int64 trials_linear_backoff()
{
  ULONGLONG TickCount64; // [rsp+20h] [rbp-18h]

  TickCount64 = GetTickCount64();
  if ( TickCount64 - LastGlobalTickCount < 0x1388 )
    return 0;
  if ( (unsigned int)++GlobalPasswordTries <= 0x32 )
  {
    LastGlobalTickCount = TickCount64;
    return 1;
  }
  else
  {
    Sleep(10000 * GlobalPasswordTries);
    return 0;
  }
}

Analysis of vm_entry @ [1400015a0] and final functions

_BOOL8 __fastcall vm_entry(VM_CTX *vm_ctx, char *input_flag)
{
  unsigned int op; // [rsp+30h] [rbp-18h]

  decode_bytecode(vm_ctx);
  vm_ctx->vip = 0;
  vm_ctx->vsp = 0xFFF;
  vm_ctx->last_tick = GetTickCount64();
  while ( vm_ctx->vip < bytecodeLen_ - 8 )
  {
    if ( (vm_ctx->vip & 0xF) == 0 )
    {
      if ( check_dbg() )
        return 0;
      if ( (unsigned int)check_dbg2() == 0xDEADBEEF )
        return 0;
    }
    op = decode_op(*(_DWORD *)&vm_ctx->vm_mem.layout.text[vm_ctx->vip]);
    vm_ctx->vip += 4;
    if ( op > 0xA1B2C3D4 )
    {
      if ( op == 0xB2C3D4E5 )
      {
        vm_push_imm8(vm_ctx);
      }
      else if ( op == 0xD4E5F607 )
      {
        vm_xor(vm_ctx);
      }
      else
      {
skip_instruction:
        vm_ctx->vip += 4;
      }
    }
    else
    {
      switch ( op )
      {
        case 0xA1B2C3D4:
          return vm_ctx->magic_1 == 0x4E455855;
        case 0x718293Au:
          vm_cmp(vm_ctx);
          break;
        case 0x18293A4Bu:
          vm_jz(vm_ctx);
          break;
        case 0x4B5C6D7Eu:
          vm_push_input_char(vm_ctx, input_flag);
          break;
        default:
          goto skip_instruction;
      }
    }
  }
  return 0;
}

In the code we can identify two functions that are interesting and can make our life harder when doing runtime analysis.

• The first is pretty simple and straightforward:

  BOOL check_dbg()
  {
    return IsDebuggerPresent();
  }
  

• The second is more subtle, as it checks whether the sleep is actually being run:

  __int64 check_dbg2()
  {
    DWORD TickCount; // [rsp+20h] [rbp-18h]
  
    TickCount = GetTickCount();
    Sleep(0xAu);
    if ( GetTickCount() - TickCount >= 5 )
      return TickCount;
    else
      return 3735928559LL;
  }
  

• Finally, we can see where the instructions are stored and how they are decrypted:

  //.data:0000000140005080 ; uint bytecodeLen_
  //.data:0000000140005080 bytecodeLen?    dd 48h
  //.data:0000000140005080
  //.data:0000000140005084                 align 10h
  //.data:0000000140005090 ; byte bytecode[80]
  //.data:0000000140005090 bytecode        db 12h, 34h, 56h, 78h, 9Ah, 0BCh, 0DEh, 0F0h, 12h, 34h
  //.data:000000014000509A                 db 56h, 78h, 0DEh, 0ADh, 0BEh, 0EFh, 1, 23h, 45h, 67h
  //.data:00000001400050A4                 db 89h, 0ABh, 0CDh, 0EFh, 0AAh, 0BBh, 0CCh, 0DDh, 0EEh
  //.data:00000001400050AD                 db 0FFh, 0, 11h, 22h, 33h, 44h, 55h, 66h, 77h, 88h, 99h
  //.data:00000001400050B8                 db 0A0h, 0B1h, 0C2h, 0D3h, 0E4h, 0F5h, 6, 17h, 28h, 39h
  //.data:00000001400050C2                 db 4Ah, 5Bh, 6Ch, 7Dh, 8Eh, 9Fh, 0A8h, 0B9h, 0CAh, 0DBh
  //.data:00000001400050CC                 db 0ECh, 0FDh, 0Eh, 1Fh, 20h, 31h, 42h, 53h, 64h, 75h
  //.data:00000001400050D6                 db 86h, 97h, 8 dup(0)
  
  uint32_t __fastcall decode_bytecode(VM_CTX *vm_ctx)
  {
  uint32_t result; // eax
  uint i; // [rsp+0h] [rbp-18h]
  uint32_t key; // [rsp+4h] [rbp-14h]
  
  key = vm_ctx->magic_1 ^ 0xDEADBEEF;
  for ( i = 0; ; ++i )
  {
      result = bytecodeLen_;
      if ( i >= bytecodeLen_ )
      break;
      vm_ctx->vm_mem.layout.text[i] = (0x37 * i) ^ key ^ bytecode[i];
      key = __ROL4__(key, 1) ^ 0xCAFEBABE;
  }
  return result;
  }
  

  The instructions are decrypted and placed inside vm_ctx->vm_mem.layout.text[i].

• The final interesting function is decode_op. As shown below, IDA did a good job eliminating unused code. This gives us the one and only interesting operation:

  __int64 __fastcall decode_op(uint32_t instruction)
  {
    __rdtsc();
    return instruction ^ 0xA1B2C3D4;
  }
  

At the beginning of this challenge I thought of using a tool to De-virtualize the program. This is not really necessary as the program is simple and straightforward. Instead I wrote some Python scripts to first decode the instructions and later on re-implement the VM:

• Bytecode decryption:

  def rol32(x: int, r: int) -> int:
      x &= 0xFFFFFFFF
      return ((x << r) | (x >> (32 - r))) & 0xFFFFFFFF
  
  def u32le(buf: List[int], off: int) -> int:
      return (buf[off]
              | (buf[off + 1] << 8)
              | (buf[off + 2] << 16)
              | (buf[off + 3] << 24)) & 0xFFFFFFFF
  
  def decode_bytecode(encoded: List[int]) -> List[int]:
      """
      key = (vm_ctx[0] ^ 0xDEADBEEF) and vm_ctx[0] is set to 0xDEADBEEF => key=0
      decoded[i] = encoded[i] ^ ((key ^ (i*0x1337)) & 0xFF)
      key = rol(key,1) ^ 0xCAFEBABE
      """
      key = 0
      decoded: List[int] = []
      for i, b in enumerate(encoded):
          mask_byte = (key ^ ((i * 0x1337) & 0xFFFFFFFF)) & 0xFF
          decoded.append(b ^ mask_byte)
          key = rol32(key, 1) ^ 0xCAFEBABE
      return decoded
  
  
  encoded_bytecode: List[int] = [
      0x12, 0x34, 0x56, 0x78, 0x9A, 0xBC, 0xDE, 0xF0, 0x12, 0x34, 0x56, 0x78, 0xDE, 0xAD, 0xBE, 0xEF,
      0x01, 0x23, 0x45, 0x67, 0x89, 0xAB, 0xCD, 0xEF, 0xAA, 0xBB, 0xCC, 0xDD, 0xEE, 0xFF, 0x00, 0x11,
      0x22, 0x33, 0x44, 0x55, 0x66, 0x77, 0x88, 0x99, 0xA0, 0xB1, 0xC2, 0xD3, 0xE4, 0xF5, 0x06, 0x17,
      0x28, 0x39, 0x4A, 0x5B, 0x6C, 0x7D, 0x8E, 0x9F, 0xA8, 0xB9, 0xCA, 0xDB, 0xEC, 0xFD, 0x0E, 0x1F,
      0x20, 0x31, 0x42, 0x53, 0x64, 0x75, 0x86, 0x97,
  ]
  
  decoded = decode_bytecode(encoded_bytecode)
  
  print("\n[*] Decoded (first 64 bytes):")
  hexdump(decoded, 0, 64)
  

  From this we obtain a bytecode that is not runnable on the VM.

  [*] Decoded (first 64 bytes):
  0000: 12 BD FB E5 88 8D 6F 38 86 3D 03 7D 45 C6 43 96
  0010: 4E 45 A7 B4 F6 F0 78 BF 7E A2 69 C9 E6 63 82 E7
  0020: C2 9A C9 E8 14 A6 59 B1 14 98 B7 B6 9F FE 1B 4E
  0030: 47 7F C8 68 73 C6 1B EF 5C C0 8F AF 04 41 AC C9
  

  In fact if you check this bytecode with the vm’s op codes they do not match:

  0xB2C3D4E5
  0xD4E5F607
  0xA1B2C3D4
  0x718293A
  0x18293A4B
  0x4B5C6D7E
  

This sent me into a long debugging session of my decoding script. After some hours I decided to give up and simply try to run the program with a debugger. To my surprise, my decoded bytecode was the same as the one decoded inside the VM.

I also spent a lot of time searching for runtime techniques that were catching my debugger and maybe not executing some important code path, but I found nothing.

The flag can be printed simply patching the instruction at the address 0x0140001A08:

- 74 1E 48 8D
+ 75 1E 48 8D

Running the binary now will print the flag:

========================================
         N E X U S C O R E >_<
========================================
Enter password: <password>

[++] Access granted! Welcome to NexusCore.
[++] Flag: NEXUSCORE{VM_MASTERED}