Case study of a windows program security

Introduction

This is a remake (and a translation) of an old paper I wrote when I was younger.
The target is the small shareware Startclean that was used under Windows 95/98 in order to remove broken links in the Start menu. StartClean is a famous program in the cracking scene, many beginners was cuting their teeth on.

I am proposing here, a study of all the main methods to "crack" a limited application.

I. Serial fishing
II. Cracking
III. Reverse engineering
IV. Cryptanalysis
V. Keygening
VI. Writting a crack and a keygen

Preambule

First of all, let's review some statistics math notations. You probably know that already if you heard of statistics.

Let's declare E as a set. For exemple, we can fix E = {15.04, 72, 741.1, 3/2, -7}.
This is a mathematical set. As you can undestand, it contains some numbers in a some order.

Now, if we want to work with an ordered cluster of numbers, we are going to use a tuple.
Let's fix T = (15.04, 72, 741.1, 3/2, -7). Please note this difference: we are using parenthesis and not braces.

Note: {15.04, 72} = {72, 15.04} (order does not mind) but (15.04, 72) ≠ (72, 15.04).

The umpteenth member, the 4^th for exemple, is noted X₃ (the first being X₀). The index is subscripted (below, at the baseline). Here, T₃ = 3/2.
The last member is written X_n-1, the number of X elements is named the cardinality n, noted as n = C(X) or n = |X|.

The sum of all members of a set/tuple is noted Σ_{X_i}.
Symbol Σ is the greek letter Zigma (pronounced [sigma]).

We also can sum all image of a set/tuple's members by a function f, noted Σ_{f(X_i)}.

I. Serial Fishing

The simpliest way: we disassemble using w32dasm and then we look for String references to find this:

* Referenced by a (U)nconditional or (C)onditional Jump at Address: ; look at that |:004027A3(C) | :004027C1 6A00 push 00000000 :004027C3 6A00 push 00000000 * Possible StringData Ref from Data Obj ->"Incorrect code!" | :004027C5 68AC634000 push 004063AC ; the message :004027CA 56 push esi * Reference To: USER32.MessageBoxA, Ord:0188h | :004027CB FF1534934000 Call dword ptr [00409334] ; show dialog box :004027D1 B801000000 mov eax, 00000001 :004027D6 5E pop esi :004027D7 C21000 ret 0010

We couldn't get a better start. Using the button Code Location, we go to origin of jump, at 4027A3. Then, we see this part of assembler code:

:00402794 8B742408 mov esi, dword ptr [esp+08] :00402798 56 push esi :00402799 E8B2E9FFFF call 00401150 ; function checking serial key :0040279E 83C404 add esp, 00000004 :004027A1 85C0 test eax, eax :004027A3 741C je 004027C1 ; jump to error message

Just look three lines up, we have a function call. Let's see what this function look like, place the focus on and press the rigth arrow key.
I think we are going to find interresting things. The serial key checking is done in this function.

* Reference To: USER32.GetDlgItemTextA, Ord:00EDh | :004011AD 8B35D8924000 :004011B3 FFD6 :004011B5 8D442410 :004011B9 6800010000 :004011BE 50 * Possible Reference to Dialog: DialogID_0078, | :004011BF 6806040000 :004011C4 57 :004011C5 FFD6 :004011C7 6830604000 :004011CC 6830614000 :004011D1 E8AA000000 :004011D6 8D442418 :004011DA 83C408 :004011DD 50 :004011DE 6830604000 * Reference To: KERNEL32.lstrcmpA, Ord:0269h | :004011E3 FF1520924000 :004011E9 85C0 :004011EB 0F8580000000 pan> mov esi, dword ptr [004092D8] ; get entered serial key call esi lea eax, dword ptr [esp+10] push 00000100 push eax CONTROL_ID:0406, "" push 00000406 push edi call esi push 00406030 push 00406130 call 00401280 lea eax, dword ptr [esp+18] add esp, 00000008 push eax ; entered serial key push 00406030 ; good serial key Call dword ptr [00409220] ; compare test eax, eax ; test the result jne 00401271 ; jump if bad serial

So, we run the process in debugging mode and set a breakpoint on address 4011E3 : on api KERNEL32.lstrcmpA.
Try to register and when the debugger stop on breakpoint look on the stack. Then you will show the entered and the good serial key.

My picking is 3128-30326-2509-485 for marin.jb.free.fr.

II. Cracking

The last argument pushed on the stack was 00406030, this is the address of character string where the good serial key is stored.
Next the conditional jump at 004011EB, we notice that the program registers informations into Windows Registry, in the same function.

As we can see, when the program registers our informations, it uses the good serial key as reference and not the entered key (without program change, it should be the same).

:00401242 6800010000 push 00000100 :00401247 6830604000 push 00406030 ; key data :0040124C 6A01 push 00000001 :0040124E 6A00 push 00000000 * Possible StringData Ref from Data Obj ->"Code" | :00401250 6830624000 push 00406230 ; key name :00401255 51 push ecx :00401256 FFD6 call esi ; save in regisry :00401258 8B442408 mov eax, dword ptr [esp+08] :0040125C 50 push eax

The entered key value is used only to be compared with the correct serial key. If the program continue here (doesn't jump from 4011EB to 401271), it will register with the right serial key.

Therefore we can apply a patch in the executable file at offset 5EB to "nop" the jump (6 bytes, because of a conditional jump) : 909090909090.
A registration with any random value as key will then work.

III. Reverse engineering

Definition of Reverse engineering goes from studying a compiled program, to understand how it works, to modifying it, in order to add new or change existing features.
Here, we are about to do a minor change while some other experts would be able to completely change how the program works.

The right key is stored in a global variable, at address 406030. It could be fun to modify the bad serial key error text, by something else, for example by the good key.

* Referenced by a (U)nconditional or (C)onditional Jump at Address: |:004027A3(C) | :004027C1 6A00 push 00000000 :004027C3 6A00 push 00000000 * Possible StringData Ref from Data Obj ->"Incorrect code!" | :004027C5 68AC634000 push 004063AC ; error message :004027CA 56 push esi * Reference To: USER32.MessageBoxA, Ord:0188h | :004027CB FF1534934000 Call dword ptr [00409334] ; show message :004027D1 B801000000 mov eax, 00000001 :004027D6 5E pop esi :004027D7 C21000 ret 0010

We have to modify the pushed argument with the right serial.

* Referenced by a (U)nconditional or (C)onditional Jump at Address: |:004027A3(C) | :004027C1 6A00 push 00000000 :004027C3 6A00 push 00000000 :004027C5 6830604000 push 00406030 ; the code :004027CA 56 push esi * Reference To: USER32.MessageBoxA, Ord:0188h | :004027CB FF1534934000 Call dword ptr [00409334] :004027D1 B801000000 mov eax, 00000001 :004027D6 5E pop esi :004027D7 C21000 ret 0010

Modification can be done by applying a patch in the file at 1BC6 with these two bytes : 3060.

IV. Cryptanalysis

To go with this method, we need at start, a collection of couples name + valid key or a way to produce them quickly.
That could come from a modified program (reverse engineering method), or a validated method to do pickup serials (serial fishing method).

Well, you have to know that this way of finding algorithm basically... never works!
An exception can be done for very little/simpliest sharewares, weakly protected by the author, where serial keys are calculated using simple and reversible mathematical operations such as additions, multiplications, etc.
Also, generated key must be hardware independant, ie. not linked to the machine it runs onto.

For Startclean case, we have fortunately a specific simple example.

As start, we need to find out if the programmer wrote one (or more) translation table (alphabet) in the application. Here, nothing.
Often, progam use the traditionnal computer's table, the ascii table:

...	...
A	65
B	66
C	67
...	...

...	...
a	97
b	98
c	99
...	...

Some basic registration names will help us to understand key generation algorithm:

Name	Known serial key
""	106-106-106-106
"a"	300-2046-2135-417
"b"	302-2066-2157-421
"c"	304-2086-2179-425
"aa"	494-3986-2135-417
"bb"	498-4026-2157-421
"ab"	496-4006-2157-421
"ba"	496-4006-2135-417
"cb"	500-4046-2157-421
"bca"	694-5986-2135-417

Part 1

Let's start with the first part of serial. How do the number to get from 106 to 300 ? Or to 302 ? Or 304 ?
We notice that for each case, we get a pair number and rising number when the ascii code of character rise to.

300 - 106 = 194
194 / 2 = 97

97 is the character a in the ascii table. It also work for names "b" and "c".
We have got an entry point, now what about 2 or 3 characters registration names ?

First we notice that the result doesn't change with letter order:

"ab" = 496
"ba" = 496

Letter	Ascii code	2 x ascii code
a	97	194
b	98	196
c	99	198

496 - 106 = 390

390 is the sum of 194 and 196 !

194 + 196 = 390

So, for this part, the program would add all character's ascii code of registration name and multiply the result by 2, then it adds the first number: 106.

We verify our hypothesis with registration name "bca":

194 + 196 + 198 = 588
106 + 588 = 694

Bingo! The formula is: Key₁ = 106 + Σ_{2 × N_i}

Part 2

With following one-letter registration names ("a", "b", "c", etc.), we notice a difference of 20.

But you notice also that numbers are not multiples of 20, so we have a base number added to the final result. When the serial is empty, we got the value 106.

"a" = 2046
2046 - 106 = 1940
1940 / 20 = 97

"b" = 2066
2066 - 106 = 1960
1960 / 20 = 98

"c" = 2086
2086 - 106 = 1980
1980 / 20 = 99

As expected, the number was 106. Now what happens when the registration name have several characters?
May, the author did a sum like the first part of serial?

Let's check this theory:

"bca" = 5986
5986 - 106 = 5880
5880 / 20 = 294

or 98 + 99 + 97 = 294

Our hypothesis was again correct.

Formula: Key₂ = 106 + Σ_{20 × N_i}

Part 3

Here, being observant would help you a lot. Just observe that this part is the same for all names finishing by the same letter:

"XXXXa" = 2135
"XXXXb" = 2157
"XXXXc" = 2179

2157 - 2135 = 22
2179 - 2157 = 22

Difference between each following characters is 22.
Again, these numbers are not multiples of 22, we proceed backwards.

97 × 22 = 2134
98 × 22 = 2156
99 × 22 = 2179

This time, it's more simple, we just multiply ascii code of last character by 22 and add 1.

Formula: Key₃ = 1 + 22 × N_n-1

Part 4

By the same way we did in the Part 3, we see that the last number of the serial depends only on last character of registration name. That is not really clever because the second to last and last parts of the serial key will always be linked.

Here the difference between code ascii following characters is 4.

4 × 97 = 388
417 - 388 = 29

4 × 98 = 392
421 - 392 = 29

4 × 99 = 396
425 - 396 = 29

We notice a number of 29 that is added to the product.
The calculation is to multiply the character number by 4 and then to add 29.

Formula: Key₄ = 29 + 4 × N_n

Remark:

Due to the extreme simplicity of the key generation algorithm, previous analysis operations are similar to a simple mathematical regressions.. It's used often in physics or others experimental sciences but almost never in such use case.

V. Dead listing

This is the most popular method but it is combined with the Cryptanalysis while debugging also, the idea is to double check what we understood from the assembler code.

First, locate the routine in the disassembler. We get back in the function seen above:

* Reference To: USER32.GetDlgItemTextA, Ord:00EDh | :004011AD 8B35D8924000 mov esi, dword ptr [004092D8] :004011B3 FFD6 call esi ; get the first edit box content :004011B5 8D442410 lea eax, dword ptr [esp+10] :004011B9 6800010000 push 00000100 :004011BE 50 push eax * Possible Reference to Dialog: DialogID_0078, CONTROL_ID:0406, "" | :004011BF 6806040000 push 00000406 :004011C4 57 push edi :004011C5 FFD6 call esi ; get the second edit box content :004011C7 6830604000 push 00406030 ; result buffer :004011CC 6830614000 push 00406130 ; entered name :004011D1 E8AA000000 call 00401280 ; generate serial key :004011D6 8D442418 lea eax, dword ptr [esp+18] :004011DA 83C408 add esp, 00000008 :004011DD 50 push eax ; entered serial key :004011DE 6830604000 push 00406030 ; right key * Reference To: KERNEL32.lstrcmpA, Ord:0269h | :004011E3 FF1520924000 Call dword ptr [00409220] ; compare

So, our routine would be located in 401280.

It could seem quite long but it's not very long compared to other code generation assembler source.
We are going to work on in debbuging mode, with many breakpoints, part by part.

Part 1

* Referenced by a (U)nconditional or (C)onditional Jump at Address: |:004012DF(C) | :004012D2 0FBE08 movsx ecx, byte ptr [eax] ; copy next character in ecx :004012D5 50 push eax :004012D6 8D6C4D00 lea ebp, dword ptr [ebp+2*ecx] ; here, the LEA is equivalent to a MOV instruction :004012DA FFD7 call edi ; (ebp stays a pointer) :004012DC 803800 cmp byte ptr [eax], 00 ; continue to the end :004012DF 75F1 jne 004012D2 * Referenced by a (U)nconditional or (C)onditional Jump at Address: |:004012D0(C) | :004012E1 8D442410 lea eax, dword ptr [esp+10] :004012E5 55 push ebp ; our number * Possible StringData Ref from Data Obj ->"%d-" | :004012E6 6874624000 push 00406274 :004012EB 50 push eax * Reference To: USER32.wsprintfA, Ord:0249h | :004012EC FF15D4924000 Call dword ptr [004092D4] ; string conversion :004012F2 8D44241C lea eax, dword ptr [esp+1C] :004012F6 83C40C add esp, 0000000C :004012F9 50 push eax :004012FA 56 push esi * Reference To: KERNEL32.lstrcatA, Ord:0266h | :004012FB FF153C924000 Call dword ptr [0040923C] ; concatenation :00401301 8BC3 mov eax, ebx :00401303 803B00 cmp byte ptr [ebx], 00 :00401306 7412 je 0040131A

Here the solution is not difficult to find. Have a look on 004012A2, ebp is set to 106, and then a loop for each character where the character is copied in ecx register.

In the instruction lea ebp, dword ptr [ebp+2*ecx], LEA is followed by dword ptr [], so this is equivalent to a MOV.

As excepted, we get again: Key₁ = 106 + Σ_{2 × N_i}

Part 2

* Referenced by a (U)nconditional or (C)onditional Jump at Address: |:00401318(C) | :00401308 0FBE08 movsx ecx, byte ptr [eax] ; next character in ecx :0040130B 03C9 add ecx, ecx ; multiply ecx by 2 :0040130D 50 push eax :0040130E 8D14C9 lea edx, dword ptr [ecx+8*ecx] ; edx = 9 * ecx :00401311 03EA add ebp, edx :00401313 FFD7 call edi :00401315 803800 cmp byte ptr [eax], 00 ; continue to end of string :00401318 75EE jne 00401308 * Referenced by a (U)nconditional or (C)onditional Jump at Address: |:00401306(C) | :0040131A 8D442410 lea eax, dword ptr [esp+10] ; we don't mind :0040131E 55 push ebp ; our key * Possible StringData Ref from Data Obj ->"%d-" | :0040131F 6874624000 push 00406274 :00401324 50 push eax * Reference To: USER32.wsprintfA, Ord:0249h | :00401325 FF15D4924000 Call dword ptr [004092D4] ; string conversion :0040132B 8D44241C lea eax, dword ptr [esp+1C] :0040132F 83C40C add esp, 0000000C :00401332 50 push eax :00401333 56 push esi * Reference To: KERNEL32.lstrcatA, Ord:0266h | :00401334 FF153C924000 Call dword ptr [0040923C] ; concatenation :0040133A 8BC3 mov eax, ebx :0040133C 803B00 cmp byte ptr [ebx], 00 :0040133F 7418 je 00401359

ecx = 2 × ecx
ecx + 8 × ecx = 9 × ecx

⇒ edx = 2 × 9 × ecx
⇒ edx = 18 × ecx

And then edx is added to ebp.

ebp = ebp + Σ_{18 × N_i}

But look carefully, there is nothing between the first part calculation and the second one. So the program uses the first part of key already stored in ebp and add the second result to first part.

Key₁ = 106 + Σ_{2 × N_i}
Key₂ = code₁ + Σ_{18 × N_i}

Key₂ = 106 + Σ_{2 × N_i} + Σ_{18 × N_i}

Formula: Key₂ = 106 + Σ_{20 × N_i}

Part 3

* Referenced by a (U)nconditional or (C)onditional Jump at Address: |:00401357(C) | :00401341 0FBE08 movsx ecx, byte ptr [eax] ; next character into ecx :00401344 50 push eax :00401345 8D2C89 lea ebp, dword ptr [ecx+4*ecx] ; ebp = 5 * ecx :00401348 8D0C69 lea ecx, dword ptr [ecx+2*ebp] ; ecx = ecx + 2 * ebp :0040134B 8D2C4D01000000 lea ebp, dword ptr [2*ecx+00000001]; ebp = 2 * ecx + 1 :00401352 FFD7 call edi :00401354 803800 cmp byte ptr [eax], 00 ; continue to end of string :00401357 75E8 jne 00401341 * Referenced by a (U)nconditional or (C)onditional Jump at Address: |:0040133F(C) | :00401359 8D442410 lea eax, dword ptr [esp+10] :0040135D 55 push ebp * Possible StringData Ref from Data Obj ->"%d-" | :0040135E 6874624000 push 00406274 :00401363 50 push eax * Reference To: USER32.wsprintfA, Ord:0249h | :00401364 FF15D4924000 Call dword ptr [004092D4] ; string conversation :0040136A 8D44241C lea eax, dword ptr [esp+1C] :0040136E 83C40C add esp, 0000000C :00401371 50 push eax :00401372 56 push esi * Reference To: KERNEL32.lstrcatA, Ord:0266h | :00401373 FF153C924000 Call dword ptr [0040923C] ; concatenation :00401379 8BC3 mov eax, ebx :0040137B 803B00 cmp byte ptr [ebx], 00 ; continue to end of string :0040137E 7412 je 00401392

I think the programmer tried to complicate the algorithm but finally, it doesn't really succed: he forgot to perform a sum or other thing to link characters to each others. The calculation, in facts, is only based on the last character... Let me recapitulate chronogical operations:

ebp = 5 × ecx
ecx = ecx + 2 × ebp
ebp = 2 × ecx + 1

So, we got

ecx = ecx + 2 × 5 × ecx
ecx = ecx + 10 × ecx
ecx = 11 × ecx

and

ebp = 2 × ecx + 1
ebp = 2 × 11 × ecx + 1
ebp = 22 × ecx + 1

All this calculs are not usefull because there is not accumulation of these values.

Formula: Key₃ = 1 + 22 × N_n-1

Part 4

* Referenced by a (U)nconditional or (C)onditional Jump at Address: |:00401390(C) | :00401380 0FBE08 movsx ecx, byte ptr [eax] ; next character into ecx :00401383 50 push eax :00401384 8D2C8D1D000000 lea ebp, dword ptr [4*ecx+0000001D]; ebp = 4 * ecx + 29 :0040138B FFD7 call edi :0040138D 803800 cmp byte ptr [eax], 00 ; continue to end of string :00401390 75EE jne 00401380 * Referenced by a (U)nconditional or (C)onditional Jump at Address: |:0040137E(C) | :00401392 8D442410 lea eax, dword ptr [esp+10] :00401396 55 push ebp * Possible StringData Ref from Data Obj ->"%d" | :00401397 6870624000 push 00406270 :0040139C 50 push eax * Reference To: USER32.wsprintfA, Ord:0249h | :0040139D FF15D4924000 Call dword ptr [004092D4] ; string convertion :004013A3 8D44241C lea eax, dword ptr [esp+1C] :004013A7 83C40C add esp, 0000000C :004013AA 50 push eax :004013AB 56 push esi * Reference To: KERNEL32.lstrcatA, Ord:0266h | :004013AC FF153C924000 Call dword ptr [0040923C] ; concatenation

Same weaknesses here, the autor does not keep calculated values...
Only the last character is used to generated the key part.

With $1D = 29, we got the formula: Key₄ = 29 + 4 × N_n-1

VI. Writting a crack and a keygen

When I wrote this article, I gave sources in Pascal which is a easy-to-learn and demonstrative language for algorithm. For this new version, I decided to use C because I think, it's easier to find a C compiler today than a Pascal one.

a. Crack

source code

#include <stdio.h> #include <stdlib.h> #include <errno.h> typedef struct { unsigned long offset; size_t size; char data[]; } patch; size_t modify(FILE *f, const patch *p); // the filename const char filename[] = "startcln.exe"; // available patchs const patch jump_nop = {0x5eb, 6, {0x90, 0x90, 0x90, 0x90, 0x90, 0x90}}; const patch reverse = {0x1bc6, 2, {0x30, 0x60}}; // list here modifications to perform const patch *modif[] = {&jump_nop}; int main(int argc, char *argv[]) { FILE *f; int i; f = fopen(filename, "wb"); if (!f) { fprintf(stderr, "Unable to open '%s' for writting!\n\n", filename); return EXIT_FAILURE; } for(i=0; i < (sizeof(modif)/sizeof(patch*)); i++) { printf("Patching...step %i...", i); if (modify(f, modif[i]) == modif[i]->size) { printf("ok!\n"); } else { printf("error! (file may be corrupt)\n"); } } fclose(f); return EXIT_SUCCESS; } size_t modify(FILE *f, const patch *p) { fseek(f, p->offset, SEEK_SET); if (errno) { return 0; } else { return fwrite(p->data, 1, p->size, f); } }

b. Keygen

source code

#include <stdio.h> #include <stdlib.h> #include <string.h> void keygen(char *name); int main(int argc, char *argv[]) { int i; char buffer[1024]; if (argc > 1) { for(i=1; i < argc; i++) { keygen(argv[i]); } } else { printf("Enter a registration name:\n"); fgets(buffer, sizeof(buffer), stdin); i = strlen(buffer); if (i > 1) { buffer[i-1] = '\0'; keygen(buffer); } } return EXIT_SUCCESS; } void keygen(char *name) { unsigned long ebp = 106; unsigned long code[4]; unsigned int i, len; len = strlen(name); for(i=0; i < len; i++) { ebp += 2 * ((unsigned)name[i]); } code[0] = ebp; for(i=0; i < len; i++) { ebp += 18 * ((unsigned)name[i]); } code[1] = ebp; ebp = 1 + 22 * ((unsigned)name[len-1]); code[2] = ebp; ebp = 29 + 4 * ((unsigned)name[len-1]); code[3] = ebp; printf("Startclean registration code for '%s':\n", name); printf("%u-%u-%u-%u\n", code[0], code[1], code[2], code[3]); }