• Debugging
  • Vulnerability


  • Root Cause Analysis
  • Time-Travel-Debugging
  • Vulnerability

There are many ways to find vulnerabilities. One of the most scalable methods is fuzzing. In essence, fuzzing is a brute-forcing. In many cases, malformed input triggers a crash in the program. After you acquire a crash, the next step is understanding the root cause of the crash. Proper RCA (root cause analysis) is essential in understanding the nature of the bug. It helps to determine if the bug is actually an exploitable vulnerability and whether putting additional efforts to develop an exploit for the vulnerability makes sense or not. From the security engineer’s perspective, proper categorization of the vulnerabilities and understanding the nature of bugs is very helpful in establishing their mitigation strategy. Simply put, RCA is the starting point of exploit development and product defense strategy.

Time Travel Debugging

Time Travel Debugging is a tool from Microsoft that enables you to record the execution of your program and replay it later in an offline environment. It was developed to collect non-reproducible software bugs from Microsoft’s customers. Once it was reproduced in a customer environment with recordings enabled, the customer could submit the recordings to Microsoft so that engineers could analyze the issue and triage them.

Traditionally, understanding the nature of vulnerability has been a very painful and tedious process. If you have access to the source code of the software, you could recompile and debug the code to fully understand the context of the issue. But if you don’t, then it becomes more of trial, error and guessing game.

TTD can help with the RCA process because of its ability to record and replay program execution. TTD is built upon Nirvana and iDNA technology. Nirvana is a binary instrumentation technology. Program execution is recorded using iDNA Trace Writer and saved as a trace file. The trace file can be later run through iDNA Trace Reader. The concept is the same as Pin, but TTD has better usability by doing all the jobs for saving instruction execution logs and providing replay functionality integrated into the WinDbg.

An Adobe Acrobat Reader Vulnerability

There was a report about an Adobe Acrobat Reader vulnerability. With a short description, the article has a POC attached to it. It was described as a double free issue caused by malformed JP2 stream record.

Here is the overview of data and control flow for the crash caused by the POC. This insight was acquired through the use of TTD technology. With the following article, I am going to explain how I could acquire those insights in an efficient way.


Reproducing And Recording The Crash

First, you need to set up testing environment. I acquired older version of Adobe Acrobat Reader from official distribution site. After launching Acrobat Reader, you can attach a TTD session to the target process. The rendering process is an AcroRd32.exe process of the AcroRd32.exe parent process. You should use WinDbg Preview` to use TTD functionality.


The WinDbg should run with an Administrator privilege to perform TTD recording. Attach to the identified target process (AcroRd32.exe with pid: 2668).


After attaching TTD, you can reproduce the crash by opening the malformed PDF document downloaded from exploit-db.

I shared my TTD run files here so that you can follow our example analysis. The password for the archive is “DarunGrim”.

Crashing Point

When you get a command prompt from the WinDbg session after opening the TTD file, you can send “g” (go) command to go to the end of the recording, which will give you some idea where it crashed.

The following shows that it had an exception during execution of the program.

(2dc.13a0): Unknown exception - code c0000374 (first/second chance not available)
TTD: End of trace reached.
(2dc.13a0): Break instruction exception - code 80000003 (first/second chance not available)
Time Travel Position: 224FB2:1
eax=000d0004 ebx=00000000 ecx=ffffd8f0 edx=770d2330 esi=00003a98 edi=00000000
eip=67ce7001 esp=010cc25c ebp=010cc2a8 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
67ce7001 0970ce          or      dword ptr [eax-32h],esi ds:002b:000cffd2=????????

You can check the call stack to investigate where the exception is coming from. Apparently, ntdll!RtlFreeHeap called from MSVCR120!free called ntdll!RtlpLogHeapFailure function to report heap inconsistency. This means heap corruption happened and the heap manager detected it while it was freeing the memory location.

0:001> kp 10
 # ChildEBP RetAddr  
00 010cd970 7712b763 ntdll!RtlpReportHeapFailure
01 010cd980 770d16cf ntdll!RtlpHeapHandleError+0x1c
02 010cd9b0 770e23be ntdll!RtlpLogHeapFailure+0x9f
03 010cd9e4 6b1becfa ntdll!RtlFreeHeap+0x4abce
04 010cd9f8 6a18b2a7 MSVCR120!free(void * pBlock = 0x1fb1c850)+0x1a [f:\dd\vctools\crt\crtw32\heap\free.c @ 51] 
WARNING: Stack unwind information not available. Following frames may be wrong.
05 010cdb0c 6a17bc96 AcroRd32!CTJPEGTiledContentWriter::operator=+0x12c83
06 010cdcd4 6a179c26 AcroRd32!CTJPEGTiledContentWriter::operator=+0x3672
07 010cdd08 6a171033 AcroRd32!CTJPEGTiledContentWriter::operator=+0x1602
08 010cdd1c 6a1654a7 AcroRd32!AX_PDXlateToHostEx+0x271448
09 010cddc4 69c7c595 AcroRd32!AX_PDXlateToHostEx+0x2658bc
0a 010cdde0 69c7c4a9 AcroRd32!CTJPEGWriter::CTJPEGWriter+0x22d4d
0b 010cde00 69c119d7 AcroRd32!CTJPEGWriter::CTJPEGWriter+0x22c61
0c 010cde28 69c1198d AcroRd32!AcroWinBrowserMain+0x19eb3
0d 010cde3c 69cb0c16 AcroRd32!AcroWinBrowserMain+0x19e69
0e 010cde54 69d8d21a AcroRd32!CTJPEGWriter::CTJPEGWriter+0x573ce
0f 010cdea8 6a0ee398 AcroRd32!CTJPEGDecoderHasMoreTiles+0xf4a

You can set a breakpoint on ntdll!RtlpLogHeapFailure and run “g-“ (go backward) command to reach the point.

Time Travel Position: 222B02:4E9
eax=00000000 ebx=7715c908 ecx=00000002 edx=00000000 esi=00000002 edi=1fb1c848
eip=7712cfb0 esp=010cd974 ebp=010cd980 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
7712cfb0 8bff            mov     edi,edi

Heap Corruption

You can use “t-“ (step backward) commands to step back to identify what condition made the heap check failure. This is where the test happens.

Time Travel Position: 222B02:67
eax=6ae3fb4c ebx=1fb1c850 ecx=1fb1c850 edx=00000000 esi=1fb1c848 edi=01670000
eip=77097851 esp=010cd9c8 ebp=010cd9e4 iopl=0         ov up ei pl nz ac pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000a16
77097851 f646073f        test    byte ptr [esi+7],3Fh       ds:002b:1fb1c84f=80

The sanity check inside RtlFreeHeap function looks like following in a disassembled code using Ghidra.


A byte field at 0x1fb1c84f is corrupt and we want to identify what code modified it. You can use the following “ba” command at this point to identify the instruction.

ba w1 1fb1c84f

The following code shows the location where the byte at 0x1fb1c84f is modified.

Time Travel Position: 222B02:B
eax=0317022d ebx=1fb1c848 ecx=8317022d edx=0317022d esi=0317022d edi=078410c0
eip=77097953 esp=010cd990 ebp=010cd9bc iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
77097953 c6430780        mov     byte ptr [ebx+7],80h       ds:002b:1fb1c84f=88

Based on the call stack, it is caused by MSVCR120!free call upon memory 0x1fb1c850. So this is a double free issue upon memory 0x1fb1c850.

0:001> kp
 # ChildEBP RetAddr  
00 010cd9bc 7709787d ntdll!RtlpLowFragHeapFree+0x93
01 010cd9e4 6b1becfa ntdll!RtlFreeHeap+0x8d
Unable to load image C:\Program Files (x86)\Adobe\Acrobat Reader DC\Reader\AcroRd32.dll, Win32 error 0n2
02 010cd9f8 6a18b296 MSVCR120!free(void * pBlock = 0x1fb1c850)+0x1a [f:\dd\vctools\crt\crtw32\heap\free.c @ 51] 
WARNING: Stack unwind information not available. Following frames may be wrong.
03 010cdb0c 6a17bc96 AcroRd32!CTJPEGTiledContentWriter::operator=+0x12c72
04 010cdcd4 6a179c26 AcroRd32!CTJPEGTiledContentWriter::operator=+0x3672
05 010cdd08 6a171033 AcroRd32!CTJPEGTiledContentWriter::operator=+0x1602
06 010cdd1c 6a1654a7 AcroRd32!AX_PDXlateToHostEx+0x271448

The free calls upon memory at 0x1fb1c850 were performed twice. Now investigate whether this memory location was re-allocated after the first free operation or not. One of the ways to know is using the LINQ query upon the TTD object.

The following command will return all instances where MSVCR120!malloc returned 0x1fb1c850. There are four instances of the calls found.

0:000> dx -r1 @$cursession.TTD.Calls("MSVCR120!malloc").Where(c => c.ReturnValue == (void *)0x1fb1c850)
@$cursession.TTD.Calls("MSVCR120!malloc").Where(c => c.ReturnValue == (void *)0x1fb1c850)                

After investigation through the calls, it was discovered that the instance 0x14e29 is the last memory allocation that happened and it is before the two free operations. So definitely this vulnerability is caused by double free.

0:000> dx -r1 @$cursession.TTD.Calls("MSVCR120!malloc").Where(c => c.ReturnValue == (void *)0x1fb1c850)[0x14e29]
@$cursession.TTD.Calls("MSVCR120!malloc").Where(c => c.ReturnValue == (void *)0x1fb1c850)[0x14e29]                
    EventType        : 0x0
    ThreadId         : 0x13a0
    UniqueThreadId   : 0x3
    TimeStart        : 2201C5:62 [Time Travel]
    TimeEnd          : 2201C6:90 [Time Travel]
    Function         : MSVCR120!malloc
    FunctionAddress  : 0x6b1bed30
    ReturnAddress    : 0x6a17cd10
    ReturnValue      : 0x1fb1c850 [Type: void *]

Tracking Freed Memory

The code where double-free happens looks like this.

  • First free
6a18b286 8b8568ffffff    mov     eax,dword ptr [ebp-98h]
6a18b28c 85c0            test    eax,eax
6a18b28e 7407            je      AcroRd32!CTJPEGTiledContentWriter::operator=+0x12c73 (6a18b297)
6a18b290 50              push    eax
6a18b291 e821eea6ff      call    AcroRd32!AcroWinBrowserMain+0x2593 (69bfa0b7) <-- call to free
  • Second free
6a18b297 8b8570ffffff    mov     eax,dword ptr [ebp-90h]
6a18b29d 85c0            test    eax,eax
6a18b29f 7407            je      AcroRd32!CTJPEGTiledContentWriter::operator=+0x12c84 (6a18b2a8)
6a18b2a1 50              push    eax
6a18b2a2 e810eea6ff      call    AcroRd32!AcroWinBrowserMain+0x2593 (69bfa0b7) <-- call to free

With disassembled code, it is like the following.

void FreeJP2Resources(void)
    if (*(int *)(unaff_EBP + -0x98) != 0) {
      free(*(int *)(unaff_EBP + -0x98));
    if (*(int *)(unaff_EBP + -0x90) != 0) {
      free(*(int *)(unaff_EBP + -0x90));

The two memory location at 0x010cda74 (ebp-98h) and 0x010cda7c (ebp-90h) contains same pointer to 0x1fb1c850. Now I need to investigate why two memory values are the same.

0:001> dd 010cda74 
010cda74  1fb1c850 00000000 1fb1c850 00000018
010cda84  00000000 00000000 0000000d 07843b1c
010cda94  000034a0 11001001 00000000 00000000
010cdaa4  07843b1c 00000000 00000000 00000000
010cdab4  00000004 00000001 00000b20 000005ac
010cdac4  00000563 00ecc304 00000666 0000bbe6
010cdad4  1fb1c808 07798d0b 1fb1c898 1fb1c700
010cdae4  00000000 05000000 03030303 01000000

I traced it back to where these two memory locations were assigned, using “ba” (break on access) commands upon two memory addresses.

ba w4 010cda74
ba w4 010cda7c

These are two locations identified.

Time Travel Position: 222B00:1AD2
eax=1fb1c850 ebx=00000000 ecx=016afde0 edx=00000003 esi=1fb1c700 edi=00000000
eip=6a18ac72 esp=010cda04 ebp=010cdb0c iopl=0         nv up ei pl nz ac po nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000212
6a18ac72 898570ffffff    mov     dword ptr [ebp-90h],eax ss:002b:010cda7c=0000077f
Time Travel Position: 222B00:1AA3
eax=1fb1c850 ebx=00000000 ecx=016afde0 edx=00000003 esi=1fb1c700 edi=00000000
eip=6a18ac59 esp=010cd9cc ebp=010cdb0c iopl=0         nv up ei ng nz na pe cy
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000287
6a18ac59 898568ffffff    mov     dword ptr [ebp-98h],eax ss:002b:010cda74=1ff69498

GetMemoryBlock To Retrieve the Memory Block (0x1fb1c850)

Both of the memory locations were retrieved using the following function at 0x6a18b2e5 (GetMemoryBlock).

GetMemoryBlock(undefined4 memory_block_type,int *param_2,byte offsetVal1,byte offsetVal2,
              undefined4 offsetVal3,int param_6)
  undefined4 retVal;
  int memory_block_base;
  byte offset;
  code *pcVar1;
  if ((char)memory_block_type == '\x03') {
    if ((param_2 == (int *)0x0) || (param_6 == 0)) {
      memory_block_type = 0;
      pcVar1 = (code *)swi(3);
      retVal = (*pcVar1)();
      return retVal;
    *param_2 = *param_2 + 1;
    retVal = RetrieveMemoryBlock(param_6,*param_2 + -1);
  else {
    offset = offsetVal1;
    if (((char)memory_block_type != '\0') &&
       (offset = offsetVal2, (char)memory_block_type != '\x01')) {
      if ((char)memory_block_type != '\x02') goto exception;
      offset = (byte)offsetVal3;
    if (offset == 0) goto exception;
    memory_block_base = (*_TlsGetValueStub)(_dwTlsIndexForMemoryBlockBase);
    retVal = *(undefined4 *)(memory_block_base + 0x20 + (uint)offset * 4);
  return retVal;

memory_block_type parameter

Overall, based upon the memory_block_type parameter, it will use different offsets and memory_block_base addresses to retrieve pointers.

  • The first memory retrieval uses GetMemoryBlock with a memory_block_type value of 0x01000000.
.text:6A18AC3F push    esi
.text:6A18AC40 push    ebx
.text:6A18AC41 push    0Fh
.text:6A18AC49 lea     eax, [ebp-58h]
.text:6A18AC4C push    0Eh
.text:6A18AC4E push    eax
.text:6A18AC4F push    dword ptr [ebp-1Ch]
.text:6A18AC52 call    GetMemoryBlock <-- Getting memory location
.text:6A18AC59 mov     [ebp-98h], eax <-- 222B00:1AA3
0:001> dds esp L6
010cd9d4  01000000 <-- memory_block_type
010cd9d8  010cdab4
010cd9dc  0000000e
010cd9e0  0000000f
010cd9e4  00000000
010cd9e8  1fb1c700
  • The second memory retrieval uses GetMemoryBlock with a memory_block_type value of 0x01000000.
.text:6A18AC57 push    esi
.text:6A18AC58 push    ebx
.text:6A18AC5F push    0Fh
.text:6A18AC61 push    0Eh
.text:6A18AC63 lea     eax, [ebp-58h]
.text:6A18AC66 push    eax
.text:6A18AC67 push    dword ptr [ebp-1Bh]
.text:6A18AC6A call    GetMemoryBlock <-- Getting memory location (222B00:1AA9)
.text:6A18AC6F add     esp, 48h 
.text:6A18AC72 mov     [ebp-90h], eax <-- 222B00:1AD2
0:001> dds esp L6
010cd9bc  00010000 <-- memory_block_type
010cd9c0  010cdab4
010cd9c4  0000000e
010cd9c8  0000000f
010cd9cc  00000000
010cd9d0  1fb1c700

The first call passes 0x01000000 as memory_block_type and second call passes 0x00010000 as memory_block_type. But inside GetMemoryBlock call, memory_block_type is casted into “char” type. So both parameters will be interpreted as 0x00.

Both calls will retrieve memory block from the following code. And other parameters are the same for both calls. So the same memory_block_type values for two GetMemoryBlock calls makes the double free issue by assigning the same memory addresses to different fields.

    memory_block_base = (*_TlsGetValueStub)(_dwTlsIndexForMemoryBlockBase);
    retVal = *(undefined4 *)(memory_block_base + 0x20 + (uint)offset * 4);

Calculating memory_block_type 0x00010000 at CalcMemoryBlockType

The memory_block_type for the second call at 0x6A18AC6A pushed from the following instruction.

Time Travel Position: 222B00:1AA8
eax=010cdab4 ebx=00000000 ecx=016afde0 edx=00000003 esi=1fb1c700 edi=00000000
eip=6a18ac67 esp=010cd9c0 ebp=010cdb0c iopl=0         nv up ei ng nz na pe cy
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000287
6a18ac67 ff75e5          push    dword ptr [ebp-1Bh]  ss:002b:010cdaf1=00010000

The following instruction can be used to track where this memory value comes from. The address 0x010cdaf1 is the location of [ebp-1Bh] memory.

0:001> ba w1 010cdaf1
0:001> g-
Breakpoint 0 hit
Time Travel Position: 222B00:19AE
eax=00000000 ebx=1fbad818 ecx=00000002 edx=010cdb7c esi=010cdb54 edi=010cdaf4
eip=6a18ab57 esp=010cda04 ebp=010cdb0c iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
6a18ab57 f3a5            rep movs dword ptr es:[edi],dword ptr [esi]

The 4 bytes from 0x010cdb54 is copied to 0x010cdaf4 here. The byte in 0x010cdaf1 was copied from 0x010cdb51. After iterating “ba” commands combined with static code analysis, I could identify the code block where memory_block_type value is calculated inside CalcMemoryBlockType. At 0x6a17be56, “al” register holds memory_block_type value.

6a17be18 e809d5ffff     call    AcroRd32!CTJPEGTiledContentWriter::operator=+0xd02 (6a179326) <-- ReadInt
6a17be1d 0fb7c8         movzx   ecx, ax <-- 222AAB:7DD
6a17be4f 8bc1           mov     eax, ecx
6a17be51 c1e80a         shr     eax, 0Ah
6a17be54 22c3           and     al, bl
6a17be56 88467d         mov     byte ptr [esi+7Dh], al <--- 222AAB:7F4

The value from register ax at 0x6a17be1d is read from the following code inside ReadInt function that converts bytes to an integer using its own formula.

      ret_val = (uint)**buffer;
      currente_byte = *buffer + 1;
      *buffer = currente_byte;
      if (1 < size) {
        iVar2 = size - 1;
        do {
          ret_val = ret_val * 0x100 + (uint)*currente_byte;
          currente_byte = currente_byte + 1;
          *buffer = currente_byte;
          iVar2 = iVar2 + -1;
        } while (iVar2 != 0);

The bytes that are converted to integer are two bytes from 0x0763beac.

0:001> db 0763beac
0763beac  00 ff 00 00 05 63 20 00-77 65 55 23 00 00 00 00  .....c .weU#....

With the “ba” command, I could identify that the memory was copied from 0x0746ba24. But, 0x0746ba24 memory location is not written by any instruction before that position. TTD can’t track memory modification performed by kernel code. For now, the assumption is that the memory contents at 0x0746ba24 are copied inside kernel function, probably ReadFile. So to test my theory, I ran following the TTD query to find any ReadFile calls upon the target memory 0x0746ba24. The c.Parameters[1] holds the buffer address and c.Parameters[2] holds the size of the buffer.

0:001> dx -r1 @$cursession.TTD.Calls("kernel32!ReadFile").Where(c => c.Parameters[1] <= 0x0746ba24 && 0x0746ba24 <= c.Parameters[1] + c.Parameters[2])
@$cursession.TTD.Calls("kernel32!ReadFile").Where(c => c.Parameters[1] <= 0x0746ba24 && 0x0746ba24 <= c.Parameters[1] + c.Parameters[2])                

The command returned just one instance of 0x7b and more investigation confirmed that this is the ReadFile function that read contents from PDF file.

0:001> dx -r1 @$cursession.TTD.Calls("kernel32!ReadFile").Where(c => c.Parameters[1] <= 0x0746ba24 && 0x0746ba24 <= c.Parameters[1] + c.Parameters[2])[0x7b]
@$cursession.TTD.Calls("kernel32!ReadFile").Where(c => c.Parameters[1] <= 0x0746ba24 && 0x0746ba24 <= c.Parameters[1] + c.Parameters[2])[0x7b]                
    EventType        : 0x0
    ThreadId         : 0x13a0
    UniqueThreadId   : 0x3
    TimeStart        : 220141:4C [Time Travel]
    TimeEnd          : 220143:14 [Time Travel]
    Function         : UnknownOrMissingSymbols
    FunctionAddress  : 0x74db9c40
    ReturnAddress    : 0x69c11134
    ReturnValue      : 0x1

After this ReadFile call, the target memory buffer looks like following

0:001> db 07467850
07467850  00 00 00 0e 30 00 01 00-00 00 13 00 00 0b 20 00  ....0......... .
07467860  00 0e 44 00 00 2e 23 00-00 2e 23 00 8e 43 00 00  ..D...#...#..C..
07467870  00 0f 00 01 01 00 00 23-46 00 01 00 00 02 3f 00  .......#F.....?.
07467880  00 02 3f b8 7e 00 c0 20-70 04 08 07 e0 e9 a4 7f  ..?.~.. p.......
07467890  cf d8 ff ec 7c 43 f3 80-d9 3f 9f 9f ff c6 7f ff  ....|C...?......
074678a0  ff 3f c0 4f 69 1b 3e cb-cc 61 fd df 13 00 62 00  .?.Oi.>..a....b.
074678b0  08 08 2f 1d f8 00 e7 e3-ba 44 9c 96 7b bb be 0f  ../......D..{...
074678c0  e7 38 a0 08 1c 61 80 e7-67 f7 dd ff df 3b ff 7f  .8...a..g....;..

The contents were read from the file offset 0x130DF.


So, the memory bytes at 0x0746ba24 were copied from the following file location at 0x172B3 and this directly affects memory allocation behaviors.



The modified byte in the fuzzed document is the same location as I identified through memory tracking.


The original bytes are “00 1C” which will be converted into integer 1D by ReadInt call. The 1D value will be masked with 3 in the CalcMemoryBlockType function, which will become memory_block_type of 1, not 0 as the modified bytes of “00 ff” will produce. The duplicate memory blocks from the GetMemoryBlock is the root cause of the double free issue.


Now the following overview makes more sense. A fuzzed memory byte is causing duplicate memory usage by affecting memory type field. It looks like the fuzzed bytes can’t control the contents of the memory directly.


When a bug happens, the input data causing the bug passes through multiple stages of data copy operations. Some input data is copied over and over in multiple locations and goes through some arithmetic operations before manifesting a bug or a vulnerability. RCA is the reverse engineering technique to trace data and control flow to determine the cause of bug or vulnerability. The method shown in this article is mostly manual. But combined with other techniques like symbolic execution and some heuristics, it is possible to build a practical bug triaging system using binary instrumentation technology.