Ask the Performance Team

Back when I was a Systems Administrator, one of my roles was to review the system configuration for a server prior to its implementation in the production environment.  We had a fairly streamlined process for server deployment – including scripts to verify the security hardening, installed software, disabled services etc.  Disabling “unnecessary” services on a system is sometimes a very subjective process.  If you were in the IT field in 2003, then you most likely remember the critical hotfix (MS03-043) that was released for a vulnerability in the Messenger service.  Since that patch, many (if not most) environments have made disabling that service a standard practice.  Other services are also turned off, based on the server’s role, whether it is in the DMZ or on the Internal network etc.  This is all part of good system configuration and maintenance – in terms of both reducing the attack surface and eliminating unnecessary overhead.

However … some caution should be exercised when deciding what services to turn off.  For example, when I walked into a new job four years ago, one of the first problems I was asked to investigate was why none of the servers were dynamically registering their DNS information.  Every single one of the desktop machines was registered, but not a single system in the server room was registered.  After about two minutes of looking at one of the problem systems, I realized that the DHCP client service was disabled.  A quick check on a couple of other systems confirmed that all of them had that service set to disabled via GPO.  The fix of course, was relatively simple – but of course the requisite (finger-pointing) post-mortem revealed that none of the admins really knew what the DHCP Client service did.  They had fallen into the common trap of thinking that DHCP was just for handing out IP Addresses to client machines.  Since the servers all had static IP addresses, they assumed that the DHCP Client service was “unnecessary” and disabled it (by the by, the Dynamic DNS failure due to the DHCP Client service not working is outlined in KB264539).

So, what was the point of that anecdote?  Recently we had a customer ask us “what services can safely be disabled”.  In this particular instance he was trying to create a standard configuration for his desktop systems and wanted to know if we could give him “the absolute minimum set” of services that needed to stay running so he could optimize the system performance of his client computers.  The short answer was that we don’t really test various service configuration combinations outside of what is enabled by default out of the box.  The reason for this is that services sometimes perform secondary functions.  A classic example of this is the Spooler Service.  We all know that the primary function of the spooler service is to handle print spooling and processing.  However – on a domain controller, the installation of the DC role adds a thread to the spooler service that is responsible for performing print pruning – removing the stale print queue objects from the Active Directory.  If the spooler service is not running on at least one DC in each site, then the AD has no means to remove old queues that no longer exist.

So the moral of the story is that while there are opportunities for tuning by turning off services to decrease the overhead – you have to know everything that the service does, lest your good intentions have dire consequences.  And with that, we’ve reached the end of our post.  Until next time …

- CC Hameed

Share this post :

That’s right – more WMI adventures.  Today we’re going to take a look at a problem that we recently encountered with a customer whose WMI script did not return any Event Log information – but the Event Logs themselves were functioning with no apparent issues.  In this particular instance, the customer was using a script to retrieve Event Log information – the Application, Security or System Logs were not returned, but the DNS, NTFRS, NTDS etc logs returned fine.  A conundrum indeed!  First, let’s take a look at a sample script that is very similar to what the customer was using:

if WScript.Arguments.Count > 0 then

    strComputer = WScript.Arguments(0)

else

    strComputer = "."

end if

Set objWMIService = GetObject("winmgmts:" & "{impersonationLevel=impersonate,(Security)}!\\" & strComputer & "\root\cimv2")

Set colLogFiles = objWMIService.ExecQuery("Select * from Win32_NTEventLogFile")

For Each objLogFile in colLogFiles

Wscript.Echo objLogFile.name

Next

All this script is designed to do is list out the different Event Logs.  When I run this on my Windows XP system, I get the following result:

There were no problems with the permissions on the WBEM folder, and changing the log file sizes didn’t resolve the issue either.  There was obviously something not quite right with WMI somewhere though.  Some elementary troubleshooting using WBEMTEST (steps below) revealed that there was a problem with the data in the repository when comparing a system that worked with one that did not.

STEPS TO CHECK THE REPOSITORY USING WBEMTEST

1. Launch WBEMTEST and connect to Root\CIMv2
2. Enumerate all classes recursively
3. Locate the Win32_NTEventLogFile class (referenced in the script above) and double-click it
4. Once the next window opens, click on "Instances" to see what data is in the repository
5. On the non-working system, the entries for the Application, System and Security logs were missing

And below is the screenshot of what the results look like (again, this is from a working machine):

What happens when the script is run is exactly what we did with WBEMTEST – query the provider and return the data.  But where does the provider get its data from?  In this specific instance, the provider opens the registry key for the Event Logs (HKLM\System\CurrentControlSet\Services\EventLog) and enumerates all the subkeys.  Each subkey represents a log.  So the first thing to check would be the permissions on these subkeys.  Assuming the subkeys are correct, the next thing to look at would be what information the provider needs from within the subkey.  For the Win32_NTEventLogFile provider, it reads the value named “File”.  Things to check here – does the value exist, and what is its type.  Based on the value returned in our example, it should be a REG_SZ or REG_EXPAND_SZ (string values).

Assuming that the subkey exists, and the value exists, the provider asks the CIM provider to populate the Win32_Datafile subset of properties using the “File” value as the path to the log file.  The obvious things to check here are whether file referenced in the registry value exist?  What are the permissions on the file?  Does the value contain references to environment variables that don’t exist (for example, is the variable %SYSTEM% as opposed to %SYSTEMROOT%?).

So as you can see, narrowing down why a particular WMI script, or portion of a script fails can be accomplished by understanding what the WMI provider does on initialization.  By following the initialization process we are able to identify where the problem lies.

That’s it for this post – thanks for stopping by!  Until next time …

- CC Hameed

Share this post :

Last year, we discussed using the WMI Diagnosis Utility in a post on Basic WMI Testing.  We’ve certainly noticed that more than a few of our customers are running the utility before contacting us, which certainly helps us in our troubleshooting.  However, in the last couple of months, we’ve had a few customers call in reporting that the utility is reporting several missing core WMI files on Windows Vista and Windows Server 2008 systems and this is creating concern for the administrators.  When reviewing the log file of the WMIDiag.vbs output, they see entries such as the ones below:

..167 07:09:07 (1) !! ERROR: WMI System file 'C:\WINDOWS\SYSTEM32\WBEM\FRAMEDYN.DLL' is MISSING or is access DENIED.
..168 07:09:07 (2) !! WARNING: WMI System file 'C:\WINDOWS\SYSTEM32\WBEM\FRAMEDYNOS.DLL' is MISSING or is access DENIED but it is an OPTIONAL component.
..177 07:09:07 (1) !! ERROR: WMI System file 'C:\WINDOWS\SYSTEM32\WBEM\PROVTHRD.DLL' is MISSING or is access DENIED
..193 07:09:07 (1) !! ERROR: WMI System file 'C:\WINDOWS\SYSTEM32\WBEM\WBEMCOMN.DLL' is MISSING or is access DENIED.

Naturally, when running any sort of diagnostic report on a system, you would get concerned about the fact that core files are reported as missing.  However, in this particular instance, this is not a problem with the OS – but rather, an issue with the current version of the WMI Diagnosis Utility.  Several WMI core files, such as the ones listed above (FRAMEDYN.DLL, FRAMEDYNOS.DLL, PROVTHRD.DLL and WBEMCOMN.DLL) have been moved from the C:\WINDOWS\SYSTEM32\WBEM folder into the C:\WINDOWS\SYSTEM32 folder.  However, the WMIDiag utility hasn’t been updated to reflect that, which is why you’ll see these errors in the WMI Log.

So what’s the way forward?  As with many tools, change sometimes comes slowly.  For those of you who have run into these problems with WMIDiag, rest assured that we are working on a new version of WMIDiag.

- CC Hameed

Share this post :

If you are in the midst of your IT migration, or just getting started with your planning but you’re not 100% sure what systems are in your environment, nor what applications have been deployed, then take heart – help is here!  This week, the Solution Accelerators Team announced the release of the Microsoft Assessment and Planning (MAP) Toolkit 3.2.  I’m sure that right now, there are two questions that many of you have:

1. What is a Solution Accelerator?
2. What is MAP?

Solution Accelerators are free, scenario-based guides and automations that are designed to help IT Professionals who are planning, deploying and operating IT systems using Microsoft products and technologies.  Security and Compliance, Management and Infrastructure and Communication and Collaboration are the different scenarios covered by the Solution Accelerators.  Now that we know what a Solution Accelerator is, what’s MAP?  MAP is a scalable and agent-less platform designed to assist you in adopting the latest Microsoft technologies.  MAP includes inventory, assessment and reporting features that do not require the installation of agent software on systems or devices.  MAP can not only discover all the systems within your Active Directory, but all the “unmanaged” workgroup machines as well!

So, what are some of the assessment areas are available in MAP 3.2?

• SQL Server Database Instance Discovery (new)
• Forefront Client Security / NAP Readiness Assessment (new)
• Power Savings Calculator (new)
• Virtualization Infrastructure Assessment (new)
• Windows Server 2008 Hardware Assessment
• Windows Vista and Microsoft Office 2007 Hardware Assessment

The inventory wizard uses an number of methods to collect inventory data – WMI, Remote Registry Service, Performance Counters, Active Directory Domain Services and SNMP.  The wizard must have the appropriate credentials.  There are some considerations to help ensure the effectiveness of the inventory and assessment.  Although the MAP toolkit is agent-less, it does require that all of the computers being inventoried are powered on and connected to the network.  There is no need to have a user logged on to the system, but the system itself should be network accessible.  If there are any network issues, or networking changes planned, those should be resolved prior to performing an inventory.  If you are working on discovering computers in workgroups or NT 4.0 domains, the Assessment Wizard has a Windows Networking Protocols computer discovery method that uses the Computer Browser service to discover systems.

The Remote Registry service is used to find the roles installed on a system.  It is also required for the Performance Metrics Wizard and completing the Windows Server 2008 role assessments.  The following should be taken into account in terms of the Remote Registry service:

• The Remote Registry Service must be started
• The Windows Firewall Remote Administration exception must be enabled
• You must authenticate using an account with Local Administrator-level privileges

So, as you can see, the MAP Toolkit has some powerful features that can ease some of the pains in planning and migrating an IT infrastructure.  More resources are listed below, including the MAP Team Blog.  That brings us to the end of this post – until next time …

Additional Resources:

• Download Microsoft Assessment and Planning Toolkit 3.2 (RTM Bits)
• TechNet: MAP home page
• MAP Team Blog
• Join the TechNet Forum Community for MAP
• Solution Accelerator Notifications

- CC Hameed

Share this post :

If you were one of the attendees at the Microsoft Professional Developer’s Conference (PDC) in Los Angeles last week, then you walked away with the Milestone 3 (M3) build of Windows 7.  The M3 build isn’t even a full Beta build – it’s pre-Beta!

Now, I can’t tell you when a public Beta version will be available.  Nor can I tell you when Windows 7 will be released (three years after the Windows Vista release is about the best I can do).  What I can tell you is that we’ll be blogging about Windows 7 in the very near future – just as we did prior to the release of Windows Vista and Windows Server 2008.

In the meantime, to whet your appetite, we wanted to share some Windows 7 links with you:

• Welcome to Windows 7 
• The Windows Blog – the October 29 post has some comments from some folks that are very critical of Windows about their impressions of Windows 7 following the PDC keynote
• The “Engineering Windows 7” Blog – we gave you this link recently, but for those that really want to see how Windows 7 is shaping up – this is the source!
• Windows 7 for the Enterprise – right now, there are some previews of the new features and key investments
• Windows 7 Developer Guide – get started building your Windows 7-ready applications now!
• PDC 2008 Site – watch recordings of the PDC sessions and the keynotes

Just a short post today to put you on the road to Windows 7.  Watch this space for more!

- CC Hameed

Hello AskPerf readers, it’s Don Geddes again and I’m back with that post that I promised you about that pesky dialog box that we all know so well. In this post, I’m going to walk through some common client/server scenarios and explain the technical details of what generates the “Unknown Publisher” dialog as well as what has to be done to hide it. I’ll try to make this as brief as possible without losing the important technical details.

To start, I want to describe the common scenarios and configurations that exist today so that you can better understand what your options are. Depending on your specific scenario, you may or may not have an option to hide the dialog, so it is important to know where your configuration fits in these scenarios.

Note: Server Operating System refers to the web server that is running either the Remote Desktop Web Connection in Windows Server 2003 or TS Web Access in Windows Server 2008.

Scenario 1:

Server Operating System: Windows Server 2003
Client Operating System: Windows XP or Windows Vista and you have NO control over the client machines

This scenario is one of the most common, and your only option for hiding the dialog in this scenario is to change your web page so that it sends a signed RDP file to the client. This scenario is also the one that creates the most confusion and frustration.

Scenario 2:

Server Operating System: Windows Server 2008
Client Operating System: Windows XP or Windows Vista and you have NO control over the client machines

This scenario will become more common as more servers are upgraded to Windows Server 2008, but the good news is that all you need to hide the dialog is built-in to Windows Server 2008.

Scenario 3:

Server Operating System: Windows Server 2008
Client Operating System: Windows XP or Windows Vista and you have direct administrative control over the client machines (a.k.a an Enterprise)

This scenario is similar to Scenario 2 but it adds the ability to use group policies to configure the client machines and make sure Vista users never see any dialog.

So now that I’ve defined the scenarios, let’s look at the technical details of what is happening with this prompt. First, I will start with a little history of how things used to work and then explain how the RDP 6 client changed things.

Then and Now

In the Remote Desktop Web Connection or TS Web Access scenarios described above, the RDP connection to the Terminal Server is started by a user clicking on a link in a web page. A user visited a web page, and that web page is now trying to open a connection to some other computer by launching the Terminal Server ActiveX control.

RDP version 5

In previous versions of the Remote Desktop client, the ActiveX control was not included and had to be downloaded from the web server. This scenario was handled by using a digitally signed msrdp.cab file that contained the ActiveX control and allowing the client to download it from the web server. Since the CAB file was digitally signed, Internet Explorer would extract the msrdp.ocx file contained within it and install it if the user clicked on “Install”. Once the ActiveX control was installed on the client, the user could make any connection they wanted to without any warning prompt because the user had to allow the ActiveX control to install in the first place. It might not be clear to most users what really happened but that is essentially what they did. It was implied that any connection made in this way would be safe because the user allowed the ActiveX control to install. The connection parameters were controlled by the code in the web page itself, so the web page was designed to send the connection parameters to the ActiveX control and allow it to make a connection. The user was not sent an RDP file with connection settings contained within it.

RDP version 6

With the RDP 6 client, the ActiveX control is now included with the Remote Desktop client installation package, and any previous versions that were installed are upgraded to the new version. When a user that has upgraded their Remote Desktop client visits a Remote Desktop Web Connection or TS Web Access web page, they no longer need to download the ActiveX control, it is already installed and should be enabled in Internet Explorer via the Manage Add-Ons user interface. If the control is not enabled then a message is displayed informing the user that they must enable the ActiveX control before they can make a connection. Enabling the control though does not imply that the connection itself is safe, especially when device redirection is involved. Every connection that is made to a Terminal Server must now be explicitly allowed by the user on a per-machine basis. So the prompt is designed to inform the user of two things:

1. The Remote Desktop Client cannot guarantee that the server they are attempting to connect to is who it claims to be

2. The remote computer may be allowed to access local resources

The user cannot control the identity verification of the server; this is something that the server administrator must configure. The user can however choose to not allow access to specific local resources, so the checkboxes under “Allow the remote computer to access the following resources on my computer:” are there to give the user a way to disable access to those resources. The user must then click the Connect button before the connection will be made.

Hiding the Prompt

In this section I will discuss how to remove the dialog based on the three scenarios mentioned previously. I’ll cover each scenario separately, but the key to understanding how to remove the prompt is to understand that in order to give the user the ability to hide the prompt, the RDP 6 client has the following requirements:

1. An RDP file must be sent from the web server to the client. Windows Server 2008 TS Web Access generates the RDP file on the fly and sends it to the user, whereas Windows Server 2003 Remote Desktop Web Connection does not send an RDP file at all.

2. The RDP file received by the client must be signed, but to be more specific, the RDP file itself must contain a signature blob within it that is generated when the RDP file is signed. This blob allows the client to verify the identity of the server that it is connecting to.

If requirements 1 and 2 are met, the user is allowed to check a box titled “Don’t ask me again for remote connections from this publisher”. Checking this box writes a key to the client registry that is based on the certificate that is used to sign the RDP file. The next time the user opens this same connection and is sent an RDP file with the same signature, the prompt will not display. The key is named PublisherBypassList and is written under:

HKEY_CURRENT_USER\Software\Microsoft\Terminal Server Client

When an RDP file is signed, two additional parameters are added to let the RDP client know that the connection is signed. You can see this by looking in the RDP file itself and looking for the signscope and signature fields:

redirectclipboard:i:1
redirectposdevices:i:0
redirectprinters:i:1
redirectcomports:i:1
redirectsmartcards:i:1
devicestoredirect:s:*
drivestoredirect:s:*
redirectdrives:i:1
session bpp:i:32
span monitors:i:1
prompt for credentials on client:i:1
remoteapplicationmode:i:1
server port:i:3389
allow font smoothing:i:1
promptcredentialonce:i:1
authentication level:i:0
gatewayusagemethod:i:2
gatewayprofileusagemethod:i:0
gatewaycredentialssource:i:0
full address:s:XXXXXX
alternate shell:s:||EXCEL
remoteapplicationprogram:s:||EXCEL
gatewayhostname:s:
remoteapplicationname:s:Microsoft Office Excel 2007
remoteapplicationcmdline:s:
signscope:s:Full Address,Server 
signature:s:

The “s” after signature designates a string value and this field would be populated with a hash that is generated from the signature.

So how do you implement RDP signing? In this next section I will give you the details for each of the previously mentioned scenarios.

Scenario 1

In this scenario, you cannot meet requirement 1 because the Remote Desktop Web Connection does not send the client an RDP file. Even if you modify your web page to send an RDP file to the client, you do not have a way to sign the RDP file and satisfy requirement 2. With no way to allow the client to verify the identity of the Terminal Server that it is connecting to, the user will never be allowed to click the check box and write the registry key that hides the prompt.

Also keep in mind that since you do not control the machine that is used to connect to your website, you cannot also control the RDP client that is being used. If a Windows XP machine that is still running the older 5.1 or 5.2 client connects, then they will not see the prompt.

The good news is that you are not alone in this scenario and there may be light at the end of the tunnel soon. Also, if you have access to a Windows Server 2008 server but cannot deploy it yet, there is a way to use rdpsign.exe to sign an RDP file that can be then served from a Windows Server 2003 web server. This will require that you modify your web page to send the user an RDP file, and if you use IIS you need to add a mime type value for application/x-rdp so that Internet Explorer knows what to do with the file that it is sent.

Scenario 2

If you’re running Windows Server 2008 and TS Web Access, all that you need to sign your RDP files is built-in. What you need to do is make sure you use the correct certificate to sign the RDP file and then use either the RemoteApp user interface or the command-line tool rdpsign.exe to sign the RDP file. The built-in help has information on how to sign your RDP file, here is an excerpt with the details:

If you use a digital certificate, the cryptographic signature on the connection file provides verifiable information about your identity as its publisher. This enables clients to recognize your organization as the source of the RemoteApp program or the remote desktop connection, and allows them to make more informed trust decisions about whether to start the connection. This helps protect against the use of .rdp files that were altered by a malicious user.

You can sign .rdp files that are used for RemoteApp connections by using a Server Authentication certificate [Secure Sockets Layer (SSL) certificate], a Code Signing certificate, or a specially defined Remote Desktop Protocol (RDP) Signing certificate. You can obtain SSL and Code Signing certificates from public certification authorities (CAs), or from an enterprise CA in your public key infrastructure hierarchy. Before you can use an RDP Signing certificate, you must configure a CA in your enterprise to issue RDP Signing certificates. For more information, see the Windows Server 2008 Terminal Services Deployment Guide (http://go.microsoft.com/fwlink/?LinkId=79603).

If you are already using an SSL certificate for terminal server or TS Gateway connections, you can use the same certificate to sign .rdp files. However, if users will connect to RemoteApp programs from public or home computers, you must use either of the following:

• A certificate from a public CA that participates in the Microsoft Root Certificate Program Members program (http://go.microsoft.com/fwlink/?LinkID=59547)
• If you are using an enterprise CA, your enterprise CA-issued certificate must be co-signed by a public CA that participates in the Microsoft Root Certification Program Members program.

Once you’ve signed your RDP files, users that connect to the TS Web Access site will download the signed RDP file and receive a prompt similar to the following:

The user then clicks the checkbox “Don’t ask me again for remote connections from this publisher” and the dialog will not display again for connections to this server.

Scenario 3

If you have Windows Server 2008 and you also control the client machines that are connecting, you can use group policy to configure Vista clients to always trust RDP files from specific publishers. Unfortunately this policy is only supported on Windows Vista but when configured properly, Vista clients would never see a prompt at all as long as the RDP file is signed.

To do this, configure the following policy:

Computer Configuration/Administrative Templates/Windows Components/Terminal Server/Remote Desktop Connection Client

Policy: Specify SHA1 thumbprints of certificates representing trusted .rdp publishers

You must ENABLE this policy and then specify the SHA1 thumbprint from the certificate(s) used to sign your RDP files.

And with that, we’ve reached the end of this post.  There is a lot of information here, hopefully this sheds some light on the issue and will help you work through some of the pain points.  This issue is one that we are keeping an eye on, so stay tuned to our blog for other potential workarounds for Scenario 1.

- Don Geddes

Share this post :

Hello AskPerf readers!  My name is Leena Nair, and I am a Support Engineer on the Performance team.  Over the last couple of months, I’ve had some very interesting discussions with customers regarding anti-virus software selection, and I thought it might be interesting to share some pieces of those discussions with you.  Almost anyone that owns, or works on a computer is aware of the dangers posed by malware and the need to install (and maintain) a reliable anti-virus program.  However, as we’ve noted in several posts in the past, the anti-virus software itself has been known to cause issues that impact system performance and reliability.  So, let’s start by taking a look at some things to consider when choosing an Anti-Virus package.

1. First and foremost, the AV package we choose has to be effective at both identifying and cleaning (or isolating) malware on the system.  If it can’t do either one effectively, then you probably want to be thinking about a different solution
2. If you are even slightly familiar with the security trends surrounding malware, then you know that virus and malware writers churn out variations on a theme at an astonishing rate.  A good AV solution provides frequent and timely updates to combat these variations. 
3. In addition, a good AV package is able to recognize the myriad of virus and malware programs that have been released.  In other words, quantity of viruses recognized and the quality of the actions that the AV program is able to take to address these virus issues are equally important.
4. Standard AV software examines individual files for known virus signatures.  This exact detection method is only as good as the program’s database of known virus definitions.  Thus, as virus variants are released, it becomes important to ensure that your virus definitions are up to date.  Given that, there is another method of virus detection, called heuristic detection, to consider.  Heuristic detection does not rely so much on detecting an exact virus signature to identify malware, but rather the pattern of behavior exhibited by the program.  Thus, malware programs that attempt to modify the registry or system files are detected by what they are trying to do as opposed to trying to match an exact fingerprint in a database.
5. The impact on system resources is a key consideration.  Remember that your AV software is an application – like Microsoft Outlook, Internet Explorer or Windows Live Messenger.  As with all of these applications, it requires system resources to perform its intended functions.  Actively scanning a system for viruses can impact the overall system performance, as the antivirus engine competes with other applications for system resources.  The degree to which an antivirus program detrimentally affects a system's performance varies depending on the task being performed.  For instance, most of the real time antivirus scan can affect both local and networking operations, can cause high CPU utilization etc.

Even though many of these things seem to be fairly obvious, we often work with customers who are running into issues caused by AV software not performing as expected – causing pool memory depletion, hangs etc.  And that will bring us to the end of this post.  Take care!

- Leena Nair

Share this post :

Hey folks, it’s Sakthi Ganesh again.  Today we’re going to talk a little bit about an interesting issue I recently worked on with one of our customers related to MMC Launch failures.  The problem itself had little to do with the MMC’s themselves and more to do with SideBySide Assemblies.  Before we dig into the issue though, let’s set the stage appropriately …

Most IT Administrators are familiar with the terms “DLL Hell” or “Dependency Hell”.  The terms refer to the different complications that can occur when dealing with multiple versions of the same Dynamic Link Library (DLL) file.  The whole point behind DLL's is that you can write a single container object that is shared by any number of other components that need that DLL’s functions.  This saves hard disk space as well as memory space since the DLL only has to map into memory once.  It also allows you to change functionality or even fix bugs in a single location.  This however opens you up to unintended consequences if a DLL is replaced with an older or sometimes even newer version. When this occurs, you might get errors like this:

• "The procedure entry point Foo couldn't be located in Bar.dll"
• "Ordinal 123 could not be located in the Dynamic Link Library Example.dll"

These errors simply mean that the app was calling an entry point that the DLL did not recognize because it moved or has been removed.  In some scenarios, a program may fail to check what version of a DLL (for example MFC42.DLL) is already installed on a system before it simply overwrites a newer version with an older one.  This creates all sorts of problems – the new program may work fine, but all the other programs on the system that relied on the updated functionality of the new DLL no longer work.  A well written installer should always check the version of shared files to as opposed to simply overwriting them.  Another scenario may arise when a newer version of a DLL has some functionality changes that result in compatibility issues.  In theory newer DLL’s should be fully backwards-compatible, but in actuality – issues may arise.  All of this is prior to the introduction of System File Protection and SideBySide Assemblies which by and large prevent an application from simply overwriting core files with whatever version it wants to.  System File Protection monitors a list of protected operating system files and makes sure they don't get replaced.  If an application tries to overwrite one of our protected files, we allow it to succeed, and then immediately come in behind it and replace it with the version stored in the DLLCACHE directory.  The application never even knows this happened and in the vast majority of instances, the new application will function with no issues.

However, if you are the developer of an application and you need to use your own version of one of our DLL's, that is fine too.  That is where DLL Redirection and SideBySide Assemblies come in.  DLL Redirection is actually a fairly simple concept.  When you install your application, place the DLL's you need in the same directory as the .EXE for the app, and then create a new empty file in that location with the .LOCAL extension.  For instance, you have an app named Test.exe, so you create a Test.exe.local file in the same location.  Then, when you run the Test.exe app, it will use the local directory as it's LoadLibrary search path, thus using the DLL's in that directory in preference of the ones stored by the system.

SideBySide Assemblies are Win32 assemblies described by manifests and created so that multiple versions can run at the same time without conflicts. SideBySide Assemblies store information about binding and COM activation in their manifests. Developers can use SideBySide assemblies provided by Microsoft, other publishers or themselves. This allows an app vendor to call the specific version of a DLL that they need to work properly. By calling a SideBySide assembly built into Windows for instance, a third-party application can take advantage of the visual styles included with Windows XP or Windows Vista without changing anything in their application. Much more information about DLL Redirection and SideBySide Assemblies is available at the MSDN pages listed below.

Now, let’s turn to that issue I was working on.  As I mentioned, the problem behavior was that no 32-bit MMC would launch on a 64-bit server.  In the Event Logs we found error messages such as the one below:

Event Type: Error
Event Source: SideBySide
Event Category: None
Event ID: 59
Date: (Date)
Time: (Time)
User: N/A
Computer: (Computer Name)
Description:
Generate Activation Context failed for C:\WINNT\SysWOW64\mmc.exe. Reference error message: The referenced assembly is not installed on your system.

Event Type: Error
Event Source: SideBySide
Event Category: None
Event ID: 59
Date: (Date)
Time: (Time)
User: N/A
Computer: (Computer Name)
Description:
Resolve Partial Assembly failed for Microsoft.Windows.Common-Controls. Reference error message: The referenced assembly is not installed on your system.

There are a number of different ways to troubleshoot a problem like this.  In this particular scenario, we elected to launch MMC.EXE from within the Windows Debugger on a working machine and see what was going on.  The debug output is below:

Microsoft (R) Windows Debugger Version 6.8.0004.0 X86
Copyright (c) Microsoft Corporation. All rights reserved. 

CommandLine: C:\Windows\SysWOW64\mmc.exe
Symbol search path is: srv*c:\symbols*\\symbols\symbols
Executable search path is: 
ModLoad: 00db0000 00f68000   mmc.exe 
ModLoad: 77b00000 77c50000   ntdll.dll
ModLoad: 77040000 77150000   C:\Windows\syswow64\kernel32.dll
ModLoad: 760d0000 7618f000   C:\Windows\syswow64\ADVAPI32.dll
ModLoad: 6bdd0000 6be35000   C:\Windows\SysWOW64\ODBC32.dll
ModLoad: 74480000 74506000   C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_5.82.6000.16386_none_87e0cb09378714f1\COMCTL32.dll
ModLoad: 764f0000 76fbe000   C:\Windows\syswow64\SHELL32.dll 

d: 74840000 7486c000   C:\Windows\SysWOW64\apphelp.dll
(1618.15f4): Break instruction exception - code 80000003 (first chance)
eax=00000000 ebx=00000000 ecx=c1400000 edx=00000000 esi=fffffffe edi=77be0094
eip=77b10004 esp=002af7e4 ebp=002af814 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ntdll!DbgBreakPoint:
77b10004 cc              int     3
0:000> g
ModLoad: 77390000 773f0000   C:\Windows\SysWOW64\IMM32.DLL
ModLoad: 76420000 764e7000   C:\Windows\syswow64\MSCTF.dll
ModLoad: 77ae0000 77ae9000   C:\Windows\syswow64\LPK.DLL
ModLoad: 75fc0000 7603d000   C:\Windows\syswow64\USP10.dll
ModLoad: 75720000 758b4000   C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.6000.16386_none_5d07289e07e1d100\comctl32.dll
ModLoad: 71e10000 71e48000   C:\Windows\SysWOW64\odbcint.dll
ModLoad: 773f0000 77474000   C:\Windows\syswow64\CLBCatQ.DLL
ModLoad: 6adc0000 6afd4000   C:\Windows\SysWOW64\mmcndmgr.dll

Based on the information in the Event Logs we surmised that there was a problem with the Common Control SideBySide Assemblies.  The two paths above (in red) are the only ones that refer to the WinSxS folder, so the first thing to do is to take a look to see if the referenced files are actually there (since the Event Log message indicates missing files).  In this particular instance, the files themselves were missing.  Now that we know exactly what the problem is, the resolution itself was relatively simple.  We replaced the relevant folders on the problem machine with folders from the working machine … and all was well again!

With that, it’s time to wrap up this post.  Thanks for stopping by!

Additional Resources:

• MSDN: Dynamic-Link Library Redirection
• MSDN: Isolated Applications and Side-by-side Assemblies
• MSDN: The End of DLL Hell

- Sakthi Ganesh

Share this post :

It occurred to me that with so many technical blogs out there, that it might be useful to bring you some posts from our fellow bloggers here at Microsoft that have been published recently …

If you haven’t been keeping up with our colleagues over at the Terminal Services blog, then you may have missed some extremely useful information that they’ve posted over the last couple of months, including a very cool troubleshooting walkthrough of an odd Terminal Services Gateway issue: A Funny Thing Happened on the Way to the Desktop

Other recent posts by the Terminal Services team:

• Developing Applications for the TS Platform
• How To: Extend the TS Session Broker to Support VDI (Part 1)
• TS Session Broker Extensibility (Part 2)

Even though Internet Explorer support has transitioned over to our colleagues in the Developer Support, that doesn’t mean that we still don’t keep an eye on what’s going on over in IE.  They’ve been extremely busy spreading the word about Internet Explorer 8 … check out some of what’s new:

• The IE8 IEAK
• Application Compatibility Logging in IE8
• Statistical Validation of the IE8 XSS Filter
• IE8 Tab Grouping
• IE8 Group Policy

Mark Russinovich needs no introduction – if you haven’t had a chance to read his blog, I strongly recommend spending some time going through his posts.  Mark’s blogging has had to take a back seat to his work on v5 of Windows Internals, but his last post is on a topic near and dear to our hearts (at least on the Performance team!): The Case of the Slooooow System.

Raymond Chen’s blog covers a multitude of topics – everything from debugging to email etiquette.  He is also one of the most prolific bloggers at Microsoft.  Here’s just a few of his recent posts:

• Psychic debugging: Why your thread is spending all its time processing meaningless thread timers
• Why does Task Manager let me kill critical system processes?
• The dangers of setting your double-click speed too short

And finally, there’s a new “must-read” blog out there – the Engineering Windows 7 blog which is hosted by the two senior engineering managers for the Windows 7 product, Jon DeVaan and Steven Sinofsky.  This is definitely one of those blogs that you want to start reading now if you want to understand the how and why of Windows 7.

That’s all for today folks!  Until next time …

- CC Hameed

Share this post :

Hey folks, it’s Yong Rhee once more.  Today’s post is a short, but an important one for Print Server administrators.  If you’ve ever administered a Print Server, the chances are that you’ve probably had to perform a print server migration as well.  Whether that migration is to a Windows Server 2003 or Windows Server 2008 machine, printer migrations always seem to somehow get very messy and complicated.  Printing issues are more of a helpdesk headache as well – more than logon or application issues.  So as an administrator, how can you migrate printers without causing a printing outage in your environment?  Let’s take a look …

Most Windows printer administrators have either heard of or used the Print Migrator tool (PrintMig).  PrintMig 3.0 was released back in May of 2002, and version 3.1 was released in September 2003.  By comparison, Windows Server 2003 was released in March of 2003, Windows Server 2003 SP1 was released in March 2005, and Windows Server 2003 SP2 was released in February 2007.  So, the operating system has been secured and updated, but the PrintMig tool has not been updated in over five years.  Hmmm, that makes you stop and think for a second, doesn’t it?  Here’s five other reasons that Print Migrator 3.1 is not as robust of a migration solution as it once was:

1. PrintMig was created before OEM printer manufacturers starting shipping printer drivers containing .cab files.  As a result, PrintMig 3.1 cannot restore printer drivers that have .cab files in them
2. PrintMig was created prior to Windows Server 2003 SP1 and SP2.  For clustered printing scenarios, the folder structure for the location of Print Processors changed, and thus PrintMig is unaware of these new locations
3. PrintMig only recognizes local or NetBIOS path names to separator pages (.sep files).  If you use a fully qualified name (FQDN) for the location of .sep files it will cause PrintMig to crash with an access violation
4. PrintMig cannot restore servers that include the word “Spool” in their network name, for example Spooler1 or PrintSpool
5. PrintMig does not work with 64-bit (x64 or IA64) print servers because x 64 servers were not widely deployed when PrintMig 3.1 was released

So, with PrintMig out of the picture, how should you migrate printers?  In Windows Vista and Windows Server 2008, the Print Migration Wizard within the new Print Management Console is one option.  Alternatively you could also use the PRINTBRM.EXE command line tool (which effectively replaces PrintMig 3.1).  I’ll be going over some of the benefits of both of these tools in future posts, so stay tuned!

Additional Resources:

• Microsoft KB 938923: How to back up & restore printers when upgrading from Windows Server 2003 to Windows Server 2008
• Technet: Migrating Print Servers

- Yong Rhee

Share this post :

As you can see, we revamped the look and feel of our blog last night.  As a result, the page load times should be improved.  In addition the new look presents a cleaner, more appealing appearance.  We hope you like the new layout.  As always, please send us your feedback.

Thanks!

- CC Hameed

Perhaps one of the most frustrating issues to troubleshoot is a system that sits at the “Applying Computer Settings …” screen for what seems like forever before (eventually) getting to the point that you can log in.  Interestingly, although several of these cases are routed to the Performance team as a perceived system hang, the fact of the matter is that the system really isn’t hung at all.  In the overwhelming majority of the cases, the system is trying to complete an operation and is waiting for a response.  With that having been said, the unfortunate reality is that there are literally hundreds of possible causes – but today, we’re going to go over a couple of them, and show you some quick troubleshooting shortcuts.  So without further ado …

Before we get too deep into what’s going on with “Applying Computer Settings”, let’s take a quick look at the boot sequence for a client machine joined to an Active Directory domain that receives a dynamic IP Address:

1. User powers on the machine which goes through a POST (Power-On Self-Test) sequence.  The POST sequence displays information about the BIOS, and the amount of physical memory installed.  The system will also attempt t initialize PCI cards.  Once the POST sequence has completed, you will normally see a basic table outlining the system configuration including disk information, and PCI card information.
2. The system goes through the process of acquiring a DHCP address, normally referred to as DORA:
   • DHCP (D)iscover
   • DHCP (O)ffer
   • DHCP (R)equest
   • DCHP (A)cknowledge
   If the system is able to successfully acquire an IP Address, then various networking configuration settings are populated, such as Default Gateway, DNS Servers, WINS servers and so on. If the system is not able to acquire an IP Address, then the system will time out / fail trying to process various networking functions - including the application of group policies and even possibly logging on to the client machine.
3. Assuming that the system received an IP address from the DHCP server, the DHCP Client service will try to register its IP Address with the DNS Server
4. The system attempts to query DNS for a SRV record for a Domain Controller.  If the system locates a domain controller it then begins the process of applying system-level domain and group policies.  If no DC can be located, then we are in another timeout situation where group policies will fail to apply.
5. Once the computer settings are applied, the user logon prompt is displayed and the user enters their credentials
6. If the system was able to contact a domain controller, then the logon is authenticated and user policies are applied.  If the system was not able to contact a domain controller, the system may log the user in with cached credentials.  Since no domain controller could be located, user GPO’s will not be applied.

Now, that’s the high level overview of the startup process.  So where does the “Applying Computer Settings” piece come into play?  Where you’re most likely to experience that issue is between steps 4 and 5.  The system has located a Domain Controller, and is querying it for a list of group policies that should be applied.

Now that we know where the problem is occurring in our process, how can we best troubleshoot it?  There are a couple of different approaches to take.  Believe it or not, one of the quickest ways to isolate if the problem is occurring locally or caused by something requiring network resources is simply to unplug the network cable.  By “breaking” the network connection, the network requests should time out nearly instantaneously and the logon process should continue.  You could also use the MSCONFIG.EXE utility to temporarily prevent non-Microsoft services from starting up the next time the system boots.  To do this, launch the System Configuration Utility, and on the Services tab, hide all non-Microsoft Services, and then select “Disable All”.  When you reboot the system, if the problem does not recur, then you now have a much smaller list of potential culprits to troubleshoot.