A Feature Request for "Process Monitor"

I hope Microsoft is listening -I make a feature request at the end of the article. I recently was called in to help a client with an issue with a particularly bad application.

I hope Microsoft is listening – I make a feature request at the end of the article.  I recently was called in to help a client with an issue with a particularly bad application.  They wanted to virtualize it using the latest version of App-V.  In fact, they had called me in a couple of years back to help with the same application; it was ugly, it was nasty, it had more moving parts than a piano.  But we got it to work.  But now it didn’t work with the latest client and they could not figure out why. After listening to all they had done, I suggested we run Process Monitor (http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx) inside the virtual app to see what the app thought was going on. 

Process Monitor is the successor to Regmon and Filemon from Sysinternals.  After Microsoft purchased the company, they created this new tool that combines both file and registry access monitoring in a single view.  I absolutely love this tool, but quite frankly it provides too much misleading clues that it keeps admins from easily finding the real problem.  Version 2.8 was released just this month.

The Operating System operates in mysterious ways.  Well, maybe not so mysterious if you learn how it works internally, but snarky nonetheless.  The output of Process Monitor captures all of this (by default) and it easily confuses even the smartest IT Professional if they don’t live with this tool day-in and day-out.

When you use Process Monitor, you really need to learn how to use “filters”.  Filters are a great way to clear out a lot of the clutter from your captured trace.  Process Monitor works by registering itself with the Event Tracing for Windows (ETW, a feature of the OS kernel) to receive activity reports from both the file system and the windows registry.  You certainly want to add a lot of filters to get rid of general system activity.  A great way to know what to filter is to just start monitoring on an otherwise idle system.  Process Monitor starts with a set of default filters, such as an exclusion for anything done by the Process Monitor task itself.  You will probably want to exclude, by process name, your antivirus process, lsass.exe, and maybe svchost.exe (which hosts many standard windows services), for example.

One of my favorite things to exclude is an operation on “FAST IO DISALLOWED”.  Fast IO indicators in a trace have to do with how the windows file cache works.  Process Monitor provides a default filter that removes most of the Fast IO events, by doing an exclude on events that have an Operation starting with the string “FASTIO_”.  This leaves “FAST IO DISALLOWED” events captured and displayed.  You often see a “FAST IO DISALLOWED” entry on a file followed by the normal path attempt to open the file which succeeds.  Adding a filter to exclude Operations start with “FAST IO” eliminates these red herrings.

But even with a good set of filters in place, there still is a lot of stuff to wade through in a trace.  For example, when the application loads a dll, there may be several attempts to open the file (CreateFile) under different folders, until the file is found. 

In our case, the application was a combination of apps written by several companies and integrated together.  So pretty much every time a dll was loaded, we had a page and a half of NOT FOUND results as it kept trying different folders until eventually the file was found.  The drawing above shows an example of this.  Except, of course, our problem was the one case that the dll was not found anywhere!  Even seasoned professionals have a hard time sifting through pages and pages of this stuff.  Sure, there are shortcuts like stopping the trace as soon as you know the problem has happened and walking backwards through the trace, but still it remains hard.

       

So here is my request to Microsoft.  If not Mark R. himself, maybe he has a minion to read this.

Please add a feature we can select, that in essence says “look for patterns where this dll search happens and glom them all into the single SUCCESS for display, with some sort of visual clue that there is more hidden detail”.  Maybe we can even double click on the clue to expand just that one.  With this feature in place, the most common case of troubleshooting will be so much easier!

 

Join the conversation

6 comments

Send me notifications when other members comment.

By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Please create a username to comment.

I would love this feature to be even more powerful, imagine you are troubleshooting a certain error popup or crash. It would be nice to produce this crash for example 2 or 3 times in a row, and let Process Monitor automatically check for "patterns" in the logging. This way you can automatically filter a lot of unneeded log data.


Also, the ability to automatically load a certain filter settings file if it exists in the same directory would be very nice.


Cancel

Agreed, I don't use process monitor all that often, but if there's one feature I would add, it would be that one...


Cancel

Tim.


Having fun hunting are we?  I don't think the behavior that you're seeing is new to Windows 7.  Beginning with Windows XP SP2 and later Microsoft enabled something called SafeDLLSearchMode.  SafeDLLSearchMode does exactly what you're seeing, it causes the System32, System, and Windows folders to be searched before the Working Directory and before the SearchPath.  Prior to XP SP2 (or if you've disabled SafeDLLSearchMode) then the current directory is searched before the System32, System, and Windows directory are searched and before the System Path is walked.  In both cases, System32, System, and Windows are always injected prior to the System Path so even if they are taken out of the path, the critical OS DLLs, EXEs should be located properly (the same can't be said for middleware DLLs like Citrix though).  There's a really good article about SafeDLLSearchMode on MSDN here:  msdn.microsoft.com/.../ms682586%28VS.85%29.aspx


Shawn


Cancel

One of the projects that's sitting on my "if i ever get time..." list was the idea of writing a parser for Procmon logs to apply some pattern matching / rules based processing to try and hightlight actual failures vs. "expected" failures...


Cancel

this would be ideal!


i'd love it if microsoft could host some kind of service where you could upload your logs, then have all the logs mined looking for patterns - would take out the human guesswork... only question is.. how can convince someone to do this?


Cancel

I had a similar problem and I found that Process Monitor didn't help me.


I finally found this tool:


blog.nektra.com/.../deviare-api-interception-v1-0-1-released


I paid for some consulting, and then they found a solution that finally gave me:


LoadLIbrary doesn't contain information about the search path. There is an undocumented function that has that information. Intercepting LdrLoadDll in ntdll.dll with Deviare tool (it's free). This function has a parameter that contains the SearchPath that can be very helpful for what you need.


This tool has a great hook console in C# that has source code, so if you know to code C# you can even use GetModuleHandleEx function with the returned HINSTANCE of the LdrLoadDll and get the real path of the Dll that was loaded.


Hope it helps!


Cancel

-ADS BY GOOGLE

SearchVirtualDesktop

SearchEnterpriseDesktop

SearchServerVirtualization

SearchVMware

Close