Windows Sysinternals Suite: "Process Monitor" tool

Process Monitor 

(I created these notes while learning about Process Monitor tool.) 

Process Monitor (or "ProcMon") is a combination of two older tools: Filemon and Regmon

Mark Russinovich wrote these old tools by doing something called as syscall hooking. Filemon and Regmon were one of the first tools written by him in the Sysinternals toolset.

ProcMon still uses some undocumented APIs in Windows and that's why it's not open source.

Introduction

Procmon helps to get to the root cause of often misleading error messages. And thus, Procmon has been proven as an indispensable troubleshooting tool.

There is a saying from David Solomon: "When in doubt, run Process Monitor".

• Process Monitor monitors syscall activity for files and registry.
• File and registry issues can be in the form of misleading error messages, application crashes, application hangs, silent process exits etc.
• Procmon can help determine the root cause for missing or corrupt files, missing or corrupt registry data, permission problems and wrong DLL versions.
• Other uses of procmon are: tuning I/O activity, understanding hard drive activity and understand file and registry usage of apps.
• Procmon captures a ton of live event data related to files and registry. It's like wireshark but for syscall activity related to files and registry!
• We can export the captured data and analyze it.

[The below explanation might be outdated but it applied to Filemon. Now it can be considered as the explanation to the file monitoring part of Procmon.]

How Filemon works?

• Filemon installs a filesystem "filter driver".
• First run of filemon requires "Load Driver" user right.
• After the filter driver is loaded, filemon offers a level of security to prevent low-privilege user from manipulating the filemon driver.
• Next, Filemon requires "Debug Programs" user right. This right allows the processes to debug any other process even if they are running in different accounts. Even though Filemon doesn't really require this right as it already has a driver to intercept all syscalls, Filemon checks if the user has the "Debug programs" right because the user will be able to see all the filesystem I/O events in Filemon even for the files they may not have access to. So Filemon checks for this right to make sure the user is privileged enough to get all that critical info. As administrators already have "debug programs" right by default, running filemon as administrator might be necessary.

Usage

We can pause/resume logging process by using Ctrl+E or using the capture icon in the toolbar.

We can clear all the logs and start from scratch using Ctrl+X or the icon in the toolbar.

We can search for any text thoughout all the columns using the search icon in the toolbar.

If we find anything interesting in the file activity and want to jump to that location in the explorer, we can right click on that entry and select Jump To to go to that location in the explorer.

History Depth option in (Options -> History Depth) can be useful if we're running ProcMon for a long period of time. ProcMon stores all the event data in the virtual memory. So running it for a long period of time can consume a lot of virtual memory or even cause ProcMon to raise an exception and crash because it ran out of memory. That's where History Depth option can be helpful. The default is 0 which means unlimited. Setting it to a fixed value will tell the Procmon to use only that much amount of memory and you can keep running it for however long you want.

We can add filters using the filter icon and even reset if we mess up anything and none of the events show up.

We can also set filters using the Filter options in the menu and set Highlighting feature as well. We also have an option to save the filter, import the previously exported filters etc.

Basic vs Advanced Mode

There is an option in Filters menu called Enable Advanced Output.

Things not seen in Basic mode:

• Raw I/O request names (basic mode displays a user friendly name for a few parameters).
• Internal filesystem ops
• Activity in System process (including the ops performed by NTFS itself)
• Procmon's own activity

Basically, Advanced mode has less Filters applied. You can see the change in the Filter list when you switch on the Advanced mode.

Using Regmon (or Registry monitoring part of Procmon)

Regmon is very similar to Filemon and it also creates a device driver to intercept Registry syscalls and checks for Debug Programs right of the user just like Filemon.

Note!!! The syscall hooking was done earlier before Windows XP. On Windows XP, Microsoft actually provides Registry IO interception hooks and Regmon uses that instead of syscall hooking. Mark discourages syscall hooking as it was being exploited by rootkits and various other types of malware. And Regmon was the first software to actually demonstrate syscall hooking in Windows and that led to all the bad guys taking advantage of this technique.

Starting Filemon/Regmon before Logon

If we want to capture file and registry logs even before logging in, we need to run Filemon and Regmon under the SYSTEM user. And because Filemon and Regmon don't belong to us, i.e., the current user, it keeps running before and after we login and logout respectively.

We can run Filemon and Regmon under the SYSTEM user using psexec (which is another Sysinternals tools).

The psexec command to do that is:

psexec -i -s <local path to filemon/regmon>

-i is to give filemon/regmon access to interactive desktop (for a GUI application and for us to see a window). -s is to run filemon/regmon under the SYSTEM account.

Running Filemon/Regmon as a Service to get traces from the Boot of the system

We need to run Filemon/Regmon as a service if we need to get traces right from the boot stage of the system.

We can actually do this by going to Options -> Enable Boot Logging. This will get us all the traces right from the beginning of the OS boot.

Enabling Boot logging will start procmon as a Boot Critical driver and captures boot traces.

Notes made from the video on Procmon by Sami Laiho

Source: https://www.youtube.com/watch?v=gVYZrUJdXqU&list=PL96F5PDvO1HHRbaGiDmcI0v92nb4yIcm3&index=5

We can view the Filter Driver installed by Procmon using the following command:

C:\Windows\system32>fltmc

Filter Name                     Num Instances    Altitude    Frame
------------------------------  -------------  ------------  -----
bindflt                                 1       409800         0
PROCMON24                               4       385200         0
WdFilter                                4       328010         0
storqosflt                              0       244000         0
wcifs                                   0       189900         0
CldFlt                                  1       180451         0
FileCrypt                               0       141100         0
luafv                                   1       135000         0
npsvctrig                               1        46000         0
Wof                                     3        40700         0
FileInfo                                4        40500         0

There might be cases where the traces might not reach the altitude of procmon filter driver. The syscall "packets" go from the highest altitude to the lowest altitude.

We can modify the altitude of the procmon driver in the Registry.

Sometimes, procmon can miss some traces from Defender or some security solution and in that case, we can higher or lower the value of the altitude in the Registry.

More on Filter Driver altitudes: https://docs.microsoft.com/en-us/windows-hardware/drivers/ifs/allocated-altitudes

Normally, filtering the logs will only filter them in GUI but all of the event logs will still be written to the page file and it keeps filling the memory.

To avoid filling up the memory and run procmon for a long time (like, for weeks), then we can choose Filter -> Drop Filtered Events option to truly drop events and only put the events that we are interested into the memory.

Windows Sysinternals Suite: "Process Explorer" tool

Process Explorer

(I created these notes while learning about Process Explorer tool.) 

Things to cover

• Process List
• Process Properties
• Process control
• Thread details
• Handle view & DLL view

Introduction

Process Explorer is like a "Super Task Manager".

It has a lot of general troubleshooting capabilities:

• DLL versioning problems
• Handle memory leaks and locked files
• Performance troubleshooting
• Hung processes

What is a process?

A process is an instance of a running program.

3 main components of a process:

• A private address space allocated to that particular process and is inaccessible by other process (so that other processes can't alter the data in the memory associated with this process)
• Open handles such as files or registry keys that the process might be accessing.
• Security token: username, the groups that the user is a member of, and the privilege list.

What is a thread?

Execution context within a process.

Threads run, not processes. A thread shares all the address space of the process, it shares the handle table and privileges. Every process starts with one thread.

Microsoft Task Manager

[The following description is outdated but is still informative. Video: https://www.youtube.com/watch?v=YGtsMa9wbjw] The 'Applications' tab shows the visible windows. Windows doesn't have any inherent concept of 'applications' or 'tasks' (it has the concept of task in terms of scheduled tasks that the user schedules in the scheduler) but it's all processes and threads.

The 'Status' column in Applications tab: 'Running' means waiting for the window messages (like user clicks, keyboard inputs etc.) 'Not responding' means it's doing something else in the background is currently not waiting for window messages or is not able to accept window messages.

Colors

Pink processes - service hosting processes. They are background tasks that run no matter who's logged in (generally).

Blue processes - processes running as me.

Cyan processes - the process is a Windows 8 application using the new APIs.

You can go to Options > Configure Color to see all sets of colors.

Process Controls

We can do the following on the Process in Process Explorer:

1. Set Priority
2. Kill Process (sends a kill signal to the process through the Win32 API). In the Options menu, we have Confirm Kill, which when unchecked, won't show us a dialog box to confirm whether we want to actually kill that process.
3. Kill Process Tree: Kills the entire tree including the parent process and it's children. The exception of this rule is, if process A starts process B and process B starts process C, and process B exits, then doing Kill Process Tree on process A won't kill process C as C is already an orphan at this point and there is no connection between A and C.
4. Restart: This option will kill the process and restart the process with the same command line switches that were used to start that process in the first place!
5. Suspend: suspends the threads in a process. It doesn't kill them but just puts them on pause and it can be resumed later.

The pink processes host services. We can see the services hosted by them in the Services tab of Process Properties that shows up after double-clicking on them.

Accounting for CPU Usage

Windows OS has an interrupt every 15 ms to check what's running on the system at that current moment. That 15ms interval might change depending on the system. There is a sysinternals tool to check that. It's called clockres.exe.

Some threads run in these 15ms time interval and quickly enter the Wait state during this heartbeat by Windows OS. And in that way, they never appear in the radar of CPU usage even though they are using the CPU.

So sometimes, even though the CPU usage might appear 0%, the system might be very slow and that might be because of these kinds of threads that are going below the radar.

How to catch these threads?

Windows has something called as a Context Switch counter, which is an integer assigned by the Windows kernel to the thread. Whenever a thread wants CPU time, the kernel increments this Context Switch integer.

Process Explorer takes advantage of this by comparing the Context Switch difference at each clock tick (~15ms). This is called Context Switch Delta in Process Explorer and it can be added as a column in Process Explorer.

We can compare Context Switch Delta column and the CPU column. If CPU is 0% but Context Switch Delta is higher for any process, then that process is trying to get under the radar of being accounted for using the CPU time!

(Source of the above explanation: https://youtu.be/YGtsMa9wbjw?t=3448 i.e., at around 57:28 timeline).

There is a pseudo-process created by Mark Russinovich in Process Explorer called Interrupts. This is not a real process running on the OS but just added by the developer of ProcExp to the process list. The Context Switch Delta for this "process" is actually the number of times the interrupt has occurred and not the number of times the thread has run. It's just to get an idea of how many times the hardware interrupts are being called. By moving the mouse very swiftly, we can observe that the Context Switch Delta increases significantly.

To know which device drivers are causing the interrupt, we need some other tools and that info cannot be viewed in ProcExp. (Some tools mentioned in the video are kernrate. An API was added in Windows XP SP2 which also helps in keep track of the interrupts. But turning on the tracing features slows down the PC as the kernel needs to write every CPU interrupt to the memory. Some other tools for doing this are tracelog.exe and tracerpt.exe).

Multi-component Processes

Also, please note that if you want to view the parent-child tree in process explorer, you need to click on the 'show process tree' icon in the top toolbar.

We can view the function call stack of each thread in threads tab. We need to view that list from bottom to top. If a process has a lot of threads doing various things, then we can view what each thread is doing and how much CPU each thread is consuming and by viewing the call stack, we can get an idea about why that thread is consuming CPU. And we can try to suspend or kill that thread.

When we click on the 'Stack' button, we'll see the function call stack:

Hung Processes

Again, to get to know why the process has hung, we need to look at the thread call stack. We can also create a memory dump of the process by right-clicking on the process and creating the dump and opening the dump file in Windbg.

Open Files & Handles

We can view the handle table of the process by selecting a process and opening the lower pane (select the process and click on the lower pane icon in the top toolbar of the ProcExp).

This lower pane window shows all kinds of open objects related to that table like open files, open sockets, and various other Windows objects.

We can even select more columns in this lower pane!

We can also search for Handles from the search option in the toolbar (the binocular icon) and if something is found, and by double-clicking on that handle in the search result, it opens the corresponding entry in the lower pane.

Why examine open handles?

• Solve file locked errors (for example, if you want to delete a file but you get an error saying a process has opened it. If you want to eject a flashdrive but get an error saying a process is using it etc. You can just search for the drive letter in ProcExp using binocular icon and get which process has opened which file on that flashdrive.)
• Understand app resources like files, registry keys etc.

DLL View

To get the DLL View in the lower pane, click on View -> Lower Pane View -> DLL View

It shows more than just loaded DLLs. 

It includes .exe and any "memory mapped files" (i.e., files on the harddisk opened by processes and those are mapped to the memory for high speed access of the file content).

Malware Detection

Identifying potentially malicious processes:

1. Check for company name and description
2. Image path
3. Verify signature (you can do that for all processes automatically by going to Options -> Verify Image Signatures)
4. Look at the parent process.

Investigating unknown processes:

1. Look at the Handle table for file or Registry Key handles
2. Packed Images (exe files that are compressed or encrypted) (shows with purple color in ProcExp)
3. Strings (can inspect strings in both the image and memory). (Here "image" is the exe file on disk). You can get the Strings in Process Properties (double-click on any process and you'll get that window).
   
4. Look at the DLLs and you can also do a signature verification on the DLLs
5. Look at the Autoruns sysinternals utility to find out the autoruns and if there are any malicious processes set to autorun on system startup or logon.

ProcExp doesn't tell us how the images have been configured to run on a system. A malware would like to have itself launched everytime we boot the system, logon to the system, launch certain applications etc. That's where we can use Autoruns.exe tool from sysinternals.

System Information

We can also view System information in ProcExp in View -> System Information option.

Image Hijack

When we select 'Replace Task Manager' in ProcExp (Options -> Replace Task Manager), ProcExp does something called as Image Hijacking where it edits the registry keys in such a way that whenever we wanna launch taskmgr.exe (i.e., the task manager exe), ProcExp executable will be launched.

We can view this info in Autoruns.exe in the Image Hijack tab to see what images have been hijacked.