There are many ways to find vulnerabilities. One of the most scalable methods is fuzzing. In essence, fuzzing is a brute-forcing. In many cases, malformed input triggers a crash in the program. After you acquire a crash, the next step is understanding the root cause of the crash. Proper RCA (root cause analysis) is essential in understanding the nature of the bug. It helps to determine if the bug is actually an exploitable vulnerability and whether putting additional efforts to develop an exploit for the vulnerability makes sense or not. From the security engineer’s perspective, proper categorization of the vulnerabilities and understanding the nature of bugs is very helpful in establishing their mitigation strategy. Simply put, RCA is the starting point of exploit development and product defense strategy.
Time Travel Debugging
Time Travel Debugging is a tool from Microsoft that enables you to record the execution of your program and replay it later in an offline environment. It was developed to collect non-reproducible software bugs from Microsoft’s customers. Once it was reproduced in a customer environment with recordings enabled, the customer could submit the recordings to Microsoft so that engineers could analyze the issue and triage them.
Traditionally, understanding the nature of vulnerability has been a very painful and tedious process. If you have access to the source code of the software, you could recompile and debug the code to fully understand the context of the issue. But if you don’t, then it becomes more of trial, error and guessing game.
TTD can help with the RCA process because of its ability to record and replay program execution. TTD is built upon Nirvana and iDNA technology. Nirvana is a binary instrumentation technology. Program execution is recorded using iDNA Trace Writer and saved as a trace file. The trace file can be later run through iDNA Trace Reader. The concept is the same as Pin, but TTD has better usability by doing all the jobs for saving instruction execution logs and providing replay functionality integrated into the WinDbg.
An Adobe Acrobat Reader Vulnerability
There was a report about an Adobe Acrobat Reader vulnerability. With a short description, the article has a POC attached to it. It was described as a double free issue caused by malformed JP2 stream record.
Here is the overview of data and control flow for the crash caused by the POC. This insight was acquired through the use of TTD technology. With the following article, I am going to explain how I could acquire those insights in an efficient way.
Reproducing And Recording The Crash
First, you need to set up testing environment. I acquired older version of Adobe Acrobat Reader from official distribution site. After launching Acrobat Reader, you can attach a TTD session to the target process. The rendering process is an AcroRd32.exe process of the AcroRd32.exe parent process. You should use WinDbg Preview` to use TTD functionality.
The WinDbg should run with an Administrator privilege to perform TTD recording. Attach to the identified target process (AcroRd32.exe with pid: 2668).
After attaching TTD, you can reproduce the crash by opening the malformed PDF document downloaded from exploit-db.
I shared my TTD run files here so that you can follow our example analysis. The password for the archive is “DarunGrim”.
Crashing Point
When you get a command prompt from the WinDbg session after opening the TTD file, you can send “g” (go) command to go to the end of the recording, which will give you some idea where it crashed.
The following shows that it had an exception during execution of the program.
(2dc.13a0): Unknown exception - code c0000374 (first/second chance not available)
TTD: End of trace reached.
(2dc.13a0): Break instruction exception - code 80000003 (first/second chance not available)
Time Travel Position: 224FB2:1
eax=000d0004 ebx=00000000 ecx=ffffd8f0 edx=770d2330 esi=00003a98 edi=00000000
eip=67ce7001 esp=010cc25c ebp=010cc2a8 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
67ce7001 0970ce or dword ptr [eax-32h],esi ds:002b:000cffd2=????????
You can check the call stack to investigate where the exception is coming from. Apparently, ntdll!RtlFreeHeap called from MSVCR120!free called ntdll!RtlpLogHeapFailure function to report heap inconsistency. This means heap corruption happened and the heap manager detected it while it was freeing the memory location.
0:001> kp 10
# ChildEBP RetAddr
00 010cd970 7712b763 ntdll!RtlpReportHeapFailure
01 010cd980 770d16cf ntdll!RtlpHeapHandleError+0x1c
02 010cd9b0 770e23be ntdll!RtlpLogHeapFailure+0x9f
03 010cd9e4 6b1becfa ntdll!RtlFreeHeap+0x4abce
04 010cd9f8 6a18b2a7 MSVCR120!free(void * pBlock = 0x1fb1c850)+0x1a [f:\dd\vctools\crt\crtw32\heap\free.c @ 51]
WARNING: Stack unwind information not available. Following frames may be wrong.
05 010cdb0c 6a17bc96 AcroRd32!CTJPEGTiledContentWriter::operator=+0x12c83
06 010cdcd4 6a179c26 AcroRd32!CTJPEGTiledContentWriter::operator=+0x3672
07 010cdd08 6a171033 AcroRd32!CTJPEGTiledContentWriter::operator=+0x1602
08 010cdd1c 6a1654a7 AcroRd32!AX_PDXlateToHostEx+0x271448
09 010cddc4 69c7c595 AcroRd32!AX_PDXlateToHostEx+0x2658bc
0a 010cdde0 69c7c4a9 AcroRd32!CTJPEGWriter::CTJPEGWriter+0x22d4d
0b 010cde00 69c119d7 AcroRd32!CTJPEGWriter::CTJPEGWriter+0x22c61
0c 010cde28 69c1198d AcroRd32!AcroWinBrowserMain+0x19eb3
0d 010cde3c 69cb0c16 AcroRd32!AcroWinBrowserMain+0x19e69
0e 010cde54 69d8d21a AcroRd32!CTJPEGWriter::CTJPEGWriter+0x573ce
0f 010cdea8 6a0ee398 AcroRd32!CTJPEGDecoderHasMoreTiles+0xf4a
You can set a breakpoint on ntdll!RtlpLogHeapFailure and run “g-“ (go backward) command to reach the point.
Time Travel Position: 222B02:4E9
eax=00000000 ebx=7715c908 ecx=00000002 edx=00000000 esi=00000002 edi=1fb1c848
eip=7712cfb0 esp=010cd974 ebp=010cd980 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
ntdll!RtlpReportHeapFailure:
7712cfb0 8bff mov edi,edi
Heap Corruption
You can use “t-“ (step backward) commands to step back to identify what condition made the heap check failure. This is where the test happens.
Time Travel Position: 222B02:67
eax=6ae3fb4c ebx=1fb1c850 ecx=1fb1c850 edx=00000000 esi=1fb1c848 edi=01670000
eip=77097851 esp=010cd9c8 ebp=010cd9e4 iopl=0 ov up ei pl nz ac pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000a16
ntdll!RtlFreeHeap+0x61:
77097851 f646073f test byte ptr [esi+7],3Fh ds:002b:1fb1c84f=80
The sanity check inside RtlFreeHeap function looks like following in a disassembled code using Ghidra.
A byte field at 0x1fb1c84f is corrupt and we want to identify what code modified it. You can use the following “ba” command at this point to identify the instruction.
ba w1 1fb1c84f
g-
The following code shows the location where the byte at 0x1fb1c84f is modified.
Time Travel Position: 222B02:B
eax=0317022d ebx=1fb1c848 ecx=8317022d edx=0317022d esi=0317022d edi=078410c0
eip=77097953 esp=010cd990 ebp=010cd9bc iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
ntdll!RtlpLowFragHeapFree+0x93:
77097953 c6430780 mov byte ptr [ebx+7],80h ds:002b:1fb1c84f=88
Based on the call stack, it is caused by MSVCR120!free call upon memory 0x1fb1c850. So this is a double free issue upon memory 0x1fb1c850.
0:001> kp
# ChildEBP RetAddr
00 010cd9bc 7709787d ntdll!RtlpLowFragHeapFree+0x93
01 010cd9e4 6b1becfa ntdll!RtlFreeHeap+0x8d
Unable to load image C:\Program Files (x86)\Adobe\Acrobat Reader DC\Reader\AcroRd32.dll, Win32 error 0n2
02 010cd9f8 6a18b296 MSVCR120!free(void * pBlock = 0x1fb1c850)+0x1a [f:\dd\vctools\crt\crtw32\heap\free.c @ 51]
WARNING: Stack unwind information not available. Following frames may be wrong.
03 010cdb0c 6a17bc96 AcroRd32!CTJPEGTiledContentWriter::operator=+0x12c72
04 010cdcd4 6a179c26 AcroRd32!CTJPEGTiledContentWriter::operator=+0x3672
05 010cdd08 6a171033 AcroRd32!CTJPEGTiledContentWriter::operator=+0x1602
06 010cdd1c 6a1654a7 AcroRd32!AX_PDXlateToHostEx+0x271448
The free calls upon memory at 0x1fb1c850 were performed twice. Now investigate whether this memory location was re-allocated after the first free operation or not. One of the ways to know is using the LINQ query upon the TTD object.
The following command will return all instances where MSVCR120!malloc returned 0x1fb1c850. There are four instances of the calls found.
0:000> dx -r1 @$cursession.TTD.Calls("MSVCR120!malloc").Where(c => c.ReturnValue == (void *)0x1fb1c850)
@$cursession.TTD.Calls("MSVCR120!malloc").Where(c => c.ReturnValue == (void *)0x1fb1c850)
[0x14088]
[0x14e29]
[0x3d3dd]
[0x3d8b9]
After investigation through the calls, it was discovered that the instance 0x14e29 is the last memory allocation that happened and it is before the two free operations. So definitely this vulnerability is caused by double free.
0:000> dx -r1 @$cursession.TTD.Calls("MSVCR120!malloc").Where(c => c.ReturnValue == (void *)0x1fb1c850)[0x14e29]
@$cursession.TTD.Calls("MSVCR120!malloc").Where(c => c.ReturnValue == (void *)0x1fb1c850)[0x14e29]
EventType : 0x0
ThreadId : 0x13a0
UniqueThreadId : 0x3
TimeStart : 2201C5:62 [Time Travel]
TimeEnd : 2201C6:90 [Time Travel]
Function : MSVCR120!malloc
FunctionAddress : 0x6b1bed30
ReturnAddress : 0x6a17cd10
ReturnValue : 0x1fb1c850 [Type: void *]
Parameters
Tracking Freed Memory
The code where double-free happens looks like this.
- First free
6a18b286 8b8568ffffff mov eax,dword ptr [ebp-98h]
6a18b28c 85c0 test eax,eax
6a18b28e 7407 je AcroRd32!CTJPEGTiledContentWriter::operator=+0x12c73 (6a18b297)
6a18b290 50 push eax
6a18b291 e821eea6ff call AcroRd32!AcroWinBrowserMain+0x2593 (69bfa0b7) <-- call to free
- Second free
6a18b297 8b8570ffffff mov eax,dword ptr [ebp-90h]
6a18b29d 85c0 test eax,eax
6a18b29f 7407 je AcroRd32!CTJPEGTiledContentWriter::operator=+0x12c84 (6a18b2a8)
6a18b2a1 50 push eax
6a18b2a2 e810eea6ff call AcroRd32!AcroWinBrowserMain+0x2593 (69bfa0b7) <-- call to free
With disassembled code, it is like the following.
void FreeJP2Resources(void)
{
...
if (*(int *)(unaff_EBP + -0x98) != 0) {
free(*(int *)(unaff_EBP + -0x98));
}
if (*(int *)(unaff_EBP + -0x90) != 0) {
free(*(int *)(unaff_EBP + -0x90));
}
The two memory location at 0x010cda74 (ebp-98h) and 0x010cda7c (ebp-90h) contains same pointer to 0x1fb1c850. Now I need to investigate why two memory values are the same.
0:001> dd 010cda74
010cda74 1fb1c850 00000000 1fb1c850 00000018
010cda84 00000000 00000000 0000000d 07843b1c
010cda94 000034a0 11001001 00000000 00000000
010cdaa4 07843b1c 00000000 00000000 00000000
010cdab4 00000004 00000001 00000b20 000005ac
010cdac4 00000563 00ecc304 00000666 0000bbe6
010cdad4 1fb1c808 07798d0b 1fb1c898 1fb1c700
010cdae4 00000000 05000000 03030303 01000000
I traced it back to where these two memory locations were assigned, using “ba” (break on access) commands upon two memory addresses.
ba w4 010cda74
ba w4 010cda7c
g-
These are two locations identified.
Time Travel Position: 222B00:1AD2
eax=1fb1c850 ebx=00000000 ecx=016afde0 edx=00000003 esi=1fb1c700 edi=00000000
eip=6a18ac72 esp=010cda04 ebp=010cdb0c iopl=0 nv up ei pl nz ac po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000212
AcroRd32!CTJPEGTiledContentWriter::operator=+0x1264e:
6a18ac72 898570ffffff mov dword ptr [ebp-90h],eax ss:002b:010cda7c=0000077f
Time Travel Position: 222B00:1AA3
eax=1fb1c850 ebx=00000000 ecx=016afde0 edx=00000003 esi=1fb1c700 edi=00000000
eip=6a18ac59 esp=010cd9cc ebp=010cdb0c iopl=0 nv up ei ng nz na pe cy
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000287
AcroRd32!CTJPEGTiledContentWriter::operator=+0x12635:
6a18ac59 898568ffffff mov dword ptr [ebp-98h],eax ss:002b:010cda74=1ff69498
GetMemoryBlock To Retrieve the Memory Block (0x1fb1c850)
Both of the memory locations were retrieved using the following function at 0x6a18b2e5 (GetMemoryBlock).
undefined4
GetMemoryBlock(undefined4 memory_block_type,int *param_2,byte offsetVal1,byte offsetVal2,
undefined4 offsetVal3,int param_6)
{
undefined4 retVal;
int memory_block_base;
byte offset;
code *pcVar1;
if ((char)memory_block_type == '\x03') {
if ((param_2 == (int *)0x0) || (param_6 == 0)) {
exception:
CallThrowException(0x40000003,0);
memory_block_type = 0;
_CxxThrowException(&memory_block_type,0x7472e75c);
pcVar1 = (code *)swi(3);
retVal = (*pcVar1)();
return retVal;
}
*param_2 = *param_2 + 1;
retVal = RetrieveMemoryBlock(param_6,*param_2 + -1);
}
else {
offset = offsetVal1;
if (((char)memory_block_type != '\0') &&
(offset = offsetVal2, (char)memory_block_type != '\x01')) {
if ((char)memory_block_type != '\x02') goto exception;
offset = (byte)offsetVal3;
}
if (offset == 0) goto exception;
memory_block_base = (*_TlsGetValueStub)(_dwTlsIndexForMemoryBlockBase);
retVal = *(undefined4 *)(memory_block_base + 0x20 + (uint)offset * 4);
}
return retVal;
}
memory_block_type parameter
Overall, based upon the memory_block_type parameter, it will use different offsets and memory_block_base addresses to retrieve pointers.
- The first memory retrieval uses GetMemoryBlock with a memory_block_type value of 0x01000000.
.text:6A18AC3F push esi
.text:6A18AC40 push ebx
.text:6A18AC41 push 0Fh
.text:6A18AC49 lea eax, [ebp-58h]
.text:6A18AC4C push 0Eh
.text:6A18AC4E push eax
.text:6A18AC4F push dword ptr [ebp-1Ch]
.text:6A18AC52 call GetMemoryBlock <-- Getting memory location
.text:6A18AC59 mov [ebp-98h], eax <-- 222B00:1AA3
0:001> dds esp L6
010cd9d4 01000000 <-- memory_block_type
010cd9d8 010cdab4
010cd9dc 0000000e
010cd9e0 0000000f
010cd9e4 00000000
010cd9e8 1fb1c700
- The second memory retrieval uses GetMemoryBlock with a memory_block_type value of 0x01000000.
.text:6A18AC57 push esi
.text:6A18AC58 push ebx
.text:6A18AC5F push 0Fh
.text:6A18AC61 push 0Eh
.text:6A18AC63 lea eax, [ebp-58h]
.text:6A18AC66 push eax
.text:6A18AC67 push dword ptr [ebp-1Bh]
.text:6A18AC6A call GetMemoryBlock <-- Getting memory location (222B00:1AA9)
.text:6A18AC6F add esp, 48h
.text:6A18AC72 mov [ebp-90h], eax <-- 222B00:1AD2
0:001> dds esp L6
010cd9bc 00010000 <-- memory_block_type
010cd9c0 010cdab4
010cd9c4 0000000e
010cd9c8 0000000f
010cd9cc 00000000
010cd9d0 1fb1c700
The first call passes 0x01000000 as memory_block_type and second call passes 0x00010000 as memory_block_type. But inside GetMemoryBlock call, memory_block_type is casted into “char” type. So both parameters will be interpreted as 0x00.
Both calls will retrieve memory block from the following code. And other parameters are the same for both calls. So the same memory_block_type values for two GetMemoryBlock calls makes the double free issue by assigning the same memory addresses to different fields.
memory_block_base = (*_TlsGetValueStub)(_dwTlsIndexForMemoryBlockBase);
retVal = *(undefined4 *)(memory_block_base + 0x20 + (uint)offset * 4);
Calculating memory_block_type 0x00010000 at CalcMemoryBlockType
The memory_block_type for the second call at 0x6A18AC6A pushed from the following instruction.
Time Travel Position: 222B00:1AA8
eax=010cdab4 ebx=00000000 ecx=016afde0 edx=00000003 esi=1fb1c700 edi=00000000
eip=6a18ac67 esp=010cd9c0 ebp=010cdb0c iopl=0 nv up ei ng nz na pe cy
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000287
AcroRd32!CTJPEGTiledContentWriter::operator=+0x12643:
6a18ac67 ff75e5 push dword ptr [ebp-1Bh] ss:002b:010cdaf1=00010000
The following instruction can be used to track where this memory value comes from. The address 0x010cdaf1 is the location of [ebp-1Bh] memory.
0:001> ba w1 010cdaf1
0:001> g-
Breakpoint 0 hit
Time Travel Position: 222B00:19AE
eax=00000000 ebx=1fbad818 ecx=00000002 edx=010cdb7c esi=010cdb54 edi=010cdaf4
eip=6a18ab57 esp=010cda04 ebp=010cdb0c iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
AcroRd32!CTJPEGTiledContentWriter::operator=+0x12533:
6a18ab57 f3a5 rep movs dword ptr es:[edi],dword ptr [esi]
The 4 bytes from 0x010cdb54 is copied to 0x010cdaf4 here. The byte in 0x010cdaf1 was copied from 0x010cdb51. After iterating “ba” commands combined with static code analysis, I could identify the code block where memory_block_type value is calculated inside CalcMemoryBlockType. At 0x6a17be56, “al” register holds memory_block_type value.
6a17be18 e809d5ffff call AcroRd32!CTJPEGTiledContentWriter::operator=+0xd02 (6a179326) <-- ReadInt
6a17be1d 0fb7c8 movzx ecx, ax <-- 222AAB:7DD
6a17be4f 8bc1 mov eax, ecx
6a17be51 c1e80a shr eax, 0Ah
6a17be54 22c3 and al, bl
6a17be56 88467d mov byte ptr [esi+7Dh], al <--- 222AAB:7F4
The value from register ax at 0x6a17be1d is read from the following code inside ReadInt function that converts bytes to an integer using its own formula.
ret_val = (uint)**buffer;
currente_byte = *buffer + 1;
*buffer = currente_byte;
if (1 < size) {
iVar2 = size - 1;
do {
ret_val = ret_val * 0x100 + (uint)*currente_byte;
currente_byte = currente_byte + 1;
*buffer = currente_byte;
iVar2 = iVar2 + -1;
} while (iVar2 != 0);
The bytes that are converted to integer are two bytes from 0x0763beac.
0:001> db 0763beac
0763beac 00 ff 00 00 05 63 20 00-77 65 55 23 00 00 00 00 .....c .weU#....
With the “ba” command, I could identify that the memory was copied from 0x0746ba24. But, 0x0746ba24 memory location is not written by any instruction before that position. TTD can’t track memory modification performed by kernel code. For now, the assumption is that the memory contents at 0x0746ba24 are copied inside kernel function, probably ReadFile. So to test my theory, I ran following the TTD query to find any ReadFile calls upon the target memory 0x0746ba24. The c.Parameters[1] holds the buffer address and c.Parameters[2] holds the size of the buffer.
0:001> dx -r1 @$cursession.TTD.Calls("kernel32!ReadFile").Where(c => c.Parameters[1] <= 0x0746ba24 && 0x0746ba24 <= c.Parameters[1] + c.Parameters[2])
@$cursession.TTD.Calls("kernel32!ReadFile").Where(c => c.Parameters[1] <= 0x0746ba24 && 0x0746ba24 <= c.Parameters[1] + c.Parameters[2])
[0x7b]
The command returned just one instance of 0x7b and more investigation confirmed that this is the ReadFile function that read contents from PDF file.
0:001> dx -r1 @$cursession.TTD.Calls("kernel32!ReadFile").Where(c => c.Parameters[1] <= 0x0746ba24 && 0x0746ba24 <= c.Parameters[1] + c.Parameters[2])[0x7b]
@$cursession.TTD.Calls("kernel32!ReadFile").Where(c => c.Parameters[1] <= 0x0746ba24 && 0x0746ba24 <= c.Parameters[1] + c.Parameters[2])[0x7b]
EventType : 0x0
ThreadId : 0x13a0
UniqueThreadId : 0x3
TimeStart : 220141:4C [Time Travel]
TimeEnd : 220143:14 [Time Travel]
Function : UnknownOrMissingSymbols
FunctionAddress : 0x74db9c40
ReturnAddress : 0x69c11134
ReturnValue : 0x1
Parameters
After this ReadFile call, the target memory buffer looks like following
0:001> db 07467850
07467850 00 00 00 0e 30 00 01 00-00 00 13 00 00 0b 20 00 ....0......... .
07467860 00 0e 44 00 00 2e 23 00-00 2e 23 00 8e 43 00 00 ..D...#...#..C..
07467870 00 0f 00 01 01 00 00 23-46 00 01 00 00 02 3f 00 .......#F.....?.
07467880 00 02 3f b8 7e 00 c0 20-70 04 08 07 e0 e9 a4 7f ..?.~.. p.......
07467890 cf d8 ff ec 7c 43 f3 80-d9 3f 9f 9f ff c6 7f ff ....|C...?......
074678a0 ff 3f c0 4f 69 1b 3e cb-cc 61 fd df 13 00 62 00 .?.Oi.>..a....b.
074678b0 08 08 2f 1d f8 00 e7 e3-ba 44 9c 96 7b bb be 0f ../......D..{...
074678c0 e7 38 a0 08 1c 61 80 e7-67 f7 dd ff df 3b ff 7f .8...a..g....;..
The contents were read from the file offset 0x130DF.
So, the memory bytes at 0x0746ba24 were copied from the following file location at 0x172B3 and this directly affects memory allocation behaviors.
RCA
The modified byte in the fuzzed document is the same location as I identified through memory tracking.
The original bytes are “00 1C” which will be converted into integer 1D by ReadInt call. The 1D value will be masked with 3 in the CalcMemoryBlockType function, which will become memory_block_type of 1, not 0 as the modified bytes of “00 ff” will produce. The duplicate memory blocks from the GetMemoryBlock is the root cause of the double free issue.
Conclusion
Now the following overview makes more sense. A fuzzed memory byte is causing duplicate memory usage by affecting memory type field. It looks like the fuzzed bytes can’t control the contents of the memory directly.
When a bug happens, the input data causing the bug passes through multiple stages of data copy operations. Some input data is copied over and over in multiple locations and goes through some arithmetic operations before manifesting a bug or a vulnerability. RCA is the reverse engineering technique to trace data and control flow to determine the cause of bug or vulnerability. The method shown in this article is mostly manual. But combined with other techniques like symbolic execution and some heuristics, it is possible to build a practical bug triaging system using binary instrumentation technology.
- DarunGrim is a threat knowledge and intelligence company. Please check out our security trainings (Korean Training). Please contact jeongoh@darungrim.com for inquiries.