5

TASK:

I'm building a set of x86 assembly reverse engineering challenges, of which I have twenty or so already completed. They're just for fun / education.

The current challenge is one of the more advanced ones, and involves some trickery that makes it look like the EP is actually in the normal program, but it's actually packed away in another PE section.

Heres' the basic flow:

  • Starts out as if it were a normal MSVC++ application.
  • Injected a sneaky call away to a bunch of anti-debugger tricks.
  • If they pass, a DWORD in memory is set to 1.
  • Later in the program flow, it checks for that value being 1, and if it works it decrypts a small call table. If it fails, it sends them off on a wild goose chase of fake anti-debug tricks and eventually just crashes.
  • The call table points to the real decryption routines that decrypt the actual program code section.
  • The decryption routines are called, and they decrypt using a basic looped xor (C^k^n where C is ciphertext, k is a 32-bit key and n is the current data offset)
  • VirtualProtect is used to switch the section's protection flags from RW to RX.
  • Control flow is redirected to OEP, program runs.

The idea is that since they think they're in normal program flow, it makes them miss the anti-debug call and later checks. Anyway, that all works fine.

PROBLEM:

The current problem is that OllyDbg and a few other tools look at the packed section and see that it has high entropy, and throw up a warning that it's packed. The code section pointer in the PE header is correctly set, so it doesn't get this from having EP outside code - it's purely an entropy analysis thing.

QUESTION:

Is there an encryption method I can use that preserves low entropy, but is still easy to implement in x86 asm? I don't want to use a plain xor, since it's too easy, but I also don't want it to catch it as packed and give the game away.

I thought of something like a shuffler (somehow produce a keystream and use it to swap 4-byte blocks of code around), but I'm not sure that this is going to work, or even be simple.

Anyone got any ideas?

Polynomial
  • 27,674
  • 12
  • 80
  • 107
  • How about applying your normal encryption, then base64 encoding that to reduce entropy? – ninjalj Oct 29 '11 at 10:11
  • Most debuggers have extensive plugins to bypass anti-debug tricks. Are your sure yours are original, not widely known or modified? Otherwise it's easy to find the 'isBeingDebugged' flag that you set. (actually you don't need a dword to store 1) If the software relies on a simple flag that runs a decryption routine, I suggest you change your strategy. – 龚元程 Oct 29 '11 at 14:23
  • May I have some of those challenges when you finish them? I'm interested in this subject :) – BlackBear Oct 29 '11 at 15:50
  • @ninjalj That would just act as a giant red flag for anyone seeing it. – Polynomial Oct 29 '11 at 16:00
  • @龚元程 - Immunity Debugger bypasses the first two checks, but the rest of it is non-trivial and requires tracing program flow. This is just for fun challenge purposes, anyway. – Polynomial Oct 29 '11 at 16:02
  • @BlackBear - I'm working on them for a private forum, but I may post them publically at some point. If/when I do, I'll link them from here. Keep this question bookmarked just in case :) – Polynomial Oct 29 '11 at 16:03

4 Answers4

2

Actually, OllyDbg works like this pseudocode:

useful_bytes = number_of_bytes_in_section - count_bytes_with_values(0x00, 0x90, 0xCC)
warn about compression if useful_bytes > 0x2000 and count_bytes_with_values(0xFF, 0xE8, 0x8B, 0x89, 0x83) / useful_bytes < 0.075

So, the way to avoid that warning is to use enough bytes with the values 0xFF 0xE8 0x8B 0x89 0x83 in the compressed section.

Baffe Boyois
  • 2,120
  • 15
  • 15
2

Don't pack/encrypt your entire program code. Just encrypt a small percentage of bytes, randomly selected from your program code. If they're not decrypted, the program will soon crash if it tries to run the code anyway - and because the majority of the program is unchanged, entropy-based checks won't be set off.

bdonlan
  • 224,562
  • 31
  • 268
  • 324
  • This would be a good idea, but I'm trying to obscure the fact that the packed section even exists. Having two sections that both contain visible code is a dead giveaway. – Polynomial Oct 29 '11 at 16:07
1

What about simply reversing the bytes (from last to first)? Intel assembler instructions aren't fixed length, so this would shuffle them a little. Or you could simply rotate each byte by a fixed amount...

xanatos
  • 109,618
  • 12
  • 197
  • 280
  • Both of these seem a little simplistic. I don't want the decryption algorithm to be quite that trivial. Would rotating each dword by n%32 still produce the same result or would that trip the entropy scanner? – Polynomial Oct 29 '11 at 09:43
  • @Polynomial You'll have to check it. But I think it will rise the entropy, because the length of instructions isn't fixed. Remember that the easiest way to measure entropy is to try to zip the file :-) (the more it zips, the less entropy is present) – xanatos Oct 29 '11 at 09:45
1

EDIT: Wrong guess, this is not how Olly works. See my other answer. This still applies to tools other than OllyDbg that calculates entropy.

Expanding on ninjaljs comment:

While I haven't checked, the entropy value OllyDbg calculates is likely bytewise, without context. See How to calculate the entropy of a file? for a common algorithm for doing this.

This algorithm gives that the sequence 0 1 2 ... 254 255 have the maximum entropy possible, despite being completely predictable. A sequence of random bytes between 0 and 255 would get slightly lower entropy, since it won't have exactly the same number of each possible value.

Some quick checks on uncompressed executables with pefile tells me that uncompressed x86 code has entropy of about 6.3 to 6.6. Compressed code with entropy 8.0, encoded with base64, has entropy 6.0. Thus, base64 is easily enough to stop this algorithm from finding compressed code.

Community
  • 1
  • 1
Baffe Boyois
  • 2,120
  • 15
  • 15
  • You should remove whichever answer is "not how Olly works" (or merge into a single answer)... If you have it like this the mods will probably end up deleting one, but it's not entirely clear which one is correct so the wrong one might endu p being deleted... – bdonlan Oct 29 '11 at 14:19