Write-up of AIVD Cyber challenge

Original page (currently offline): https://www.aivd.nl/@3269/ga-cyberchallenge/

In June of this year one of the Dutch intelligence agencies – the AIVD – held a cyber challenge just like back in 2012. In this blog post I want to give a walkthrough of the steps that could have been taken in solving this challenge.

At the beginning of this challenge a zip file was provided containing:
– A private key
– A LUA script to verify the password of the private key
– A network dump containing TLS encrypted traffic

Decrypting network traffic

When looking at the pcap file one could notice that an encrypted conversation takes place with an odd webserver. This webserver seems to use a self signed certificate generated specifically for this challenge. This session is likely the one that has to be decrypted.


In order to decrypt the network traffic we have to decrypt the provided private key.
The password of the private can be derived by looking at the Lua script that has been provided.
The main hashing/encoding routine can be rewritten in the following way in Java:

private void check(String input) {
  long t = 0x1000000 * input.charAt(0);
  t += 0x10000 * input.charAt(1);
  t += 0x100 * input.charAt(2);
  t += input.charAt(3);

  for (int i = 0; i < 10000000; i++) {
    int z = (int)(t % 4);
    switch(z) {
      case 0:
        t = (t + 3141592653l) % 4294967296l;
      case 1:
        t = (3*t + 1732050808l) % 4294967296l;
      case 2:
        t = (5*t + 2236067977l) % 4294967296l;
      case 3:
        t = (7*t + 2645751311l) % 4294967296l;

  String res = input + ":" + t;

The provided Lua script expects a password of 16 characters, but the password is checked in chunks of 4 chars. This means that we can brute force the password one chunk at the time.

Eventually I was able to brute force the password using a very primitive multi-threaded program written in Java which is not worth sharing. It took roughly 20 hours to retrieve all 4 char chunks.

The following chunks were recovered:

1. Grh7:2066590424
2. F1ma:4241186467
3. Ws9r:2486763883
4. 5Ty8:27430929

So, the password is: Grh7F1maWs9r5Ty8

The next step would be to decrypt the private key with the recovered password, something that can be easily done using openssl:

openssl rsa -in ./server.key -out ./server_dec.key

Decrypting the TLS traffic in Wireshark reveales an executable stored on an AWS server:


Reverse engineering executable

The provided executable is a 64-bit ELF binary that expects a certain password.
When looking at the decompilation it becomes clear that a password of 16 chars is expected.

When looking more in depth at the cleaned-up decompilation it can be noted that an encoding function is being called from the main function with as argument a password. The encoding function contains some obfuscation. After simplifying this function it becomes clear that this is just the equivalent of an XOR.

signed __int64 __fastcall main(int a1, __int64 argc) {
  signed __int64 result;
  signed __int64 len;
  __int64 v4;
  bool v5;

  if ( ptrace(0, 0LL, 1LL, 0LL) == -1 ) {
    result = 0xFFFFFFFFLL;
  } else if ( argc == 2 ) {
    len = -1LL;
    v4 = *(_QWORD *)(a2 + 8);

    do {
      if ( !v3 )
      v5 = *(_BYTE *)v4++ == 0;
    } while ( !v5 );

    if ( (unsigned int)encode(*(_QWORD *)(a2 + 8), len ) {
      exec(*(_QWORD *)(a2 + 8));
      result = 0LL;
    } else {
      result = 0xFFFFFFFFLL;
  } else {
    result = 0xFFFFFFFFLL;
  return result;

signed __int64 __fastcall encode(__int64 a1, int len) {
  signed __int64 result;
  char v3;
  char v4;
  unsigned __int8 v5;
  signed int i; 

  if ( len == 16 ) {
    for ( i = 0; i <= 15; ++i ) {
      v3 = ((a1 + i + 3) ^ (a1 + i));
      v4 = (v3 ^ (a1 + i + 5));
      v5 = (v4 ^ (a1 + i + 11));

      if ( byte_602060[(signed __int64)i] != (v5 ^ (a1 + i + 12)) )
        return 0LL;
    result = 1LL;
  } else {
    result = 0LL;
  return result;

int __fastcall exec(__int64 a) {
  int result;
  __int64 v2;
  char s;
  __int64 v4; 

  v4 = *MK_FP(__FS__, 40LL);
  snprintf(&s, 0x50uLL, "%s%s", 6300644LL, a1);
  result = system(&s);
  v2 = *MK_FP(__FS__, 40LL) ^ v4;
  return result;

The encoding function performs the following simplified loop which performs an XOR operation on five bytes of the byte_602060 byte sequence at the time.

for (i = 0; i <= 15; i++) {
  if (byte_602060[(signed __int64)i] != (((a1 + i + 3) ^ (a1 + i)) ^ (v3 ^ (a1 + i + 5)) ^ (v4 ^ (a1 + i + 11)) ^ (a1 + i + 12)))
    return 0LL;

The byte sequence byte_602060 contains the expected result of the operation.


The result is a system of linear equations that has to be solved. The Z3 theorem solver library written for python can be used to solve this system. The following python script is just one of the ways possible to recover the requested password:

from z3 import *

passwd = ""
# We have to use BitVecs: https://stackoverflow.com/questions/17380557/how-can-we-use-xor-operation-on-integers-in-z3-using-python
a,b,c,d,e,f,g,h = BitVecs('a b c d e f g h',8)
i,j,k,l,m,n,o,p = BitVecs('i j k l m n o p',8)

s2 = Solver()

# Mandatory check

# Get the model
model = s2.model()

intArr = [model[a], model[b], model[c], model[d], model[e], model[f], model[g], model[h], model[i], model[j], model[k], model[l], model[m], model[n], model[o], model[p]]

for v in intArr:
    passwd += chr(v.as_long())

print passwd

The recovered password (“xvFgr231Kif09Pbv”) can be used to download a zip file containing a PDF file and a bonus challenge.


At time of writing this blog the original file was already offline.

Analyzing the PDF

The PDF file is an interesting one, it indicates that we’re almost there, but not quite yet.


The following information regarding encrypted data can be found inside of the PDF:

/Author (e2l2OjRmY2NmZjlmMWM5OGExNGY3MWY0M2Q1NDY1NzQ3YmMwfQ==)
/Subject (e2FsZzpBRVNfT0ZCfQ==)
/Title (e2tleTo0YTViNmEwZGE3MzQ1ZTY2ODAyYzNhY2YxODA4MDg0MX0=)

This decodes as:


The only thing left to be recovered is the encrypted message.

I used the online PDF analyzer of malwaretracker.com to analyze the PDF. It turns out that the encrypted message is present inside of an embedded image.


Decrypting this message should be pretty trivial:

$iv = pack('H*', "4fccff9f1c98a14f71f43d5465747bc0");
$alg = "aes-128-ofb";
$key = pack('H*', "4a5b6a0da7345e66802c3acf18080841");
$enc_nr = pack('H*', "313aa9be110094c2bd4479c2a278d427");
print openssl_decrypt($enc_nr, $alg, $key, true, $iv) . "n";

Taking all things in to consideration I can say that this challenge was a quite interesting one. Only time will tell when a next challenge will be made available.