We had published a network protocol analysis challenge for free entry to our BlackHat 2012 Vegas training courses and received seven correct answers. We’d like to thank those who attempted this challenge and hope that they find it useful.

The winner, Peter Af Geijerstam managed to respond first, with the correct answer. As a result, he wins a free place on any of our Hacking By Numbers courses. Here is a brief solution for it:

If you start by running the client and server binaries provided in the challenge zip file, you’ll observe the following output from the client:

And we can see the same challenge (177) and 16-byte response values in the network traffic:

So, this indicates that the client and server are running a simple challenge-response authentication protocol with a 3-digit random number (R) as the challenge and a 16-byte response value which should be calculated using the R and the shared secret value. We need to figure out the response calculation formula in order to fully understand this simple authentication protocol.  This can be done by reading the provided source code. If the source code was not available then we’d have to use a disassembler, such as IDA Pro, in order to find out the formula.
The following code snippet shows a function call to HMAC(char *msg, char *key,byte *mac) to calculate the response value:

And inside HMAC function we can see calls to Windows CryptoAPI to calculate MD5 hash value of (msg+secret):

Now, we can summarise the authentication protocol as below and work out our attack strategy:

Client->Server : HELLO
Server->Client: R
Client->Server: RESP (MD5(R+secret))
Server->Client: OK/Incorrect Response

The attacker had both R and MD5(R+secret) values from the network traffic capture file and he also knew something about the shared secret format (7 alphanumeric excluding uppercase characters). Therefore, he can run a brute force attack on the 16-byte MD5 hash value with a narrowed charset and known message format which would be [448][abcdefghijklmnopqrstuvwxyz0123456789]. There are several public hash cracking tools which support raw md5 hashes, such as hashcat.  we can run hashcat with the following options:

cudaHashcat-plus32.exe –attack-mode 3 –custom-charset1 abcdefghijklmnopqrstuvwxyz0123456789 hash.txt 448?1?1?1?1?1?1?1

It would take about 43 minutes for a NVIDIA GeForce 405 graphic card to recover the shared secret:

And the shared secret value is: bm28lg1. In order to calculate the session key value (kc) we can simply set the R to 448 in authentication server source code instead of the random value and compile it. By running the client binary using the recovered secret key value (bm28lg1), we will get the session key:

And the session key value is : 07e0f7a7cbc2d8b3dba6b7d3b69c3236

I saw a similar solution (in Spanish) on the internet posted here . I also received a question not about the challenge itself, but the source code of the authentication client and why I’v set resp buffer size it to 128 bytes while the client response length is always 21 bytes (basically why I’ve wasted 107 bytes of 1MB default stack). The answer is that the server not only processes RESP messages from the client, but also need to receive and decrypt MSG messages (which is marked as not implemented in both source codes). MSG messages clearly have a bigger size than 21 bytes and in order to use the same RESP buffer for incoming data, I set its size to 128 bytes which is purely an arbitrary number in this case and should be changed to a more suitable size based on the encryption algorithm’s block sizes which are not implemented in the current code.

If you have questions or recommendations regarding this challenge (or similar ones), please drop me an email to the address inside the challenge file.