I hope Microsoft is listening – I make a feature request at the end of the article. I recently was called in to help a client with an issue with a particularly bad application. They wanted to virtualize it using the latest version of App-V. In fact, they had called me in a couple of years back to help with the same application; it was ugly, it was nasty, it had more moving parts than a piano. But we got it to work. But now it didn’t work with the latest client and they could not figure out why. After listening to all they had done, I suggested we run Process Monitor (https://docs.microsoft.com/en-us/sysinternals/downloads/procmon) inside the virtual app to see what the app thought was going on.
Process Monitor is the successor to Regmon and Filemon from Sysinternals. After Microsoft purchased the company, they created this new tool that combines both file and registry access monitoring in a single view. I absolutely love this tool, but quite frankly it provides too much misleading clues that it keeps admins from easily finding the real problem. Version 2.8 was released just this month.
The Operating System operates in mysterious ways. Well, maybe not so mysterious if you learn how it works internally, but snarky nonetheless. The output of Process Monitor captures all of this (by default) and it easily confuses even the smartest IT Professional if they don’t live with this tool day-in and day-out.
When you use Process Monitor, you really need to learn how to use “filters”. Filters are a great way to clear out a lot of the clutter from your captured trace. Process Monitor works by registering itself with the Event Tracing for Windows (ETW, a feature of the OS kernel) to receive activity reports from both the file system and the windows registry. You certainly want to add a lot of filters to get rid of general system activity. A great way to know what to filter is to just start monitoring on an otherwise idle system. Process Monitor starts with a set of default filters, such as an exclusion for anything done by the Process Monitor task itself. You will probably want to exclude, by process name, your antivirus process, lsass.exe, and maybe svchost.exe (which hosts many standard windows services), for example.
One of my favorite things to exclude is an operation on “FAST IO DISALLOWED”. Fast IO indicators in a trace have to do with how the windows file cache works. Process Monitor provides a default filter that removes most of the Fast IO events, by doing an exclude on events that have an Operation starting with the string “FASTIO_”. This leaves “FAST IO DISALLOWED” events captured and displayed. You often see a “FAST IO DISALLOWED” entry on a file followed by the normal path attempt to open the file which succeeds. Adding a filter to exclude Operations start with “FAST IO” eliminates these red herrings.
But even with a good set of filters in place, there still is a lot of stuff to wade through in a trace. For example, when the application loads a dll, there may be several attempts to open the file (CreateFile) under different folders, until the file is found.
In our case, the application was a combination of apps written by several companies and integrated together. So pretty much every time a dll was loaded, we had a page and a half of NOT FOUND results as it kept trying different folders until eventually the file was found. The drawing above shows an example of this. Except, of course, our problem was the one case that the dll was not found anywhere! Even seasoned professionals have a hard time sifting through pages and pages of this stuff. Sure, there are shortcuts like stopping the trace as soon as you know the problem has happened and walking backwards through the trace, but still it remains hard.
So here is my request to Microsoft. If not Mark R. himself, maybe he has a minion to read this.
Please add a feature we can select, that in essence says “look for patterns where this dll search happens and glom them all into the single SUCCESS for display, with some sort of visual clue that there is more hidden detail”. Maybe we can even double click on the clue to expand just that one. With this feature in place, the most common case of troubleshooting will be so much easier!