You Can’t Always Trust Perfmon

As a diligent IT professional, you decide you want to run perfmon to monitor the memory usage on a few of your servers.  Not being 100% familiar with what all the counters mean, you do a bit of research and decide that one of the counters you want to look at is Committed Bytes.  Good choice, this is an important counter which shows you how much memory Windows has committed to make available to processes if and when they need it.  Wanting to know what perfmon has to say about this counter, you open it up and see the following description:

Committed Bytes is the amount of committed virtual memory, in bytes. Committed memory is the physical memory which has space reserved on the disk paging file(s)

Unfortunately, this description (and the description for Commit Limit and % Committed Bytes In Use) is wrong.  Interestingly, if you go all the way back to Windows 2003 (and possibly further than that) and skip forward to Windows 10, the same incorrect description is there all the way through.  This is a good example of how difficult it is to truly understand the counters relating to memory management.  Let’s take a look at what is wrong with this description.

Spare Me The Details!

If you don’t care too much about the details, the easy way of showing that the description is wrong is to remove your page file and see if Committed Bytes is zero.  After all, the implication in the description is that committed memory has space reserved in the page file.  Therefore, without a page file you can’t reserve said space in the page file and therefore can’t have any Committed Bytes.  If you turn off your page file, you’ll see that you will still have a non-zero value for Committed Bytes just like you did with the page file still present.  Case closed!

Give Me The Details!

Ok, you asked for it – things are going to get technical now.  Let’s first look at what memory commitment is.  When a process starts up, it needs memory.  However, it doesn’t know exactly how much memory it’s going to need ahead of time.  It might need 100 MB or it might need 500 MB depending on load.  Let’s say that Windows has 800 MB of free memory when the process first starts up.  No problems here – even under full load the 500 MB requirement of our fictitious process can be met.  But what if it only needs the full 500 MB four days after starting up?  By that time, there may only be 100 MB of available memory in Windows at which point our process is going to be very unhappy when it asks for the full 500 MB and the allocation fails.  Avoiding this situation is what memory commitment is all about.  It allows a process to reserve a given amount of memory and Windows will promise to make the memory available should the process need it, even if this request comes weeks after the process first started.  They key, of course, is for Windows to ensure that it doesn’t agree to provide more memory to processes than it can physically provide.  We’ll look at how it does this in the next paragraph when we talk about the Commit Limit.

Needless to say, you’ve probably realized by this point that the concept of “reservation” from the built-in perfmon description is accurate.  What’s not accurate is where the reservation is held.  To see why this is the case, we first need to understand where this committed memory is being drawn from.  Windows has a value known as the Commit Limit.  This is the maximum amount of memory that Windows can promise to make available to processes and is composed (approximately) of the amount of physical RAM plus the size of the paging file(s).  For example, if you have 4 GB of RAM and a 2 GB Page File, you have a 6 GB Commit Limit.  Related to the Commit Limit, we have the Commit Charge.  The Commit Charge is how much memory Windows has promised to make available to processes if and when they need it and is what is represented by the Committed Bytes counter we’re discussing in this post. 

And now we get to the crux of the matter.  When Windows says, “I solemnly promise to reserve 500 MB of memory for you” (making the Committed Bytes counter go up by by 500 MB as well) it’s not reserving that space in the page file, contrary to what the description says.  What it is instead doing is setting aside 500 MB of memory for the process and then adding 500 MB to the Commit Charge.  If and when the process actually uses that memory, the page file may come into play or it may not, depending on what the Windows memory manager decides is best at that point in time.  If a request for memory from a process will make the Commit Charge exceed the Commit Limit, the request will fail (or the page file will need to be expanded).

So in summary – the built-in description for Committed Bytes states that committed memory is memory which has space reserved in the page file.  In this post we have seen that committed memory is a memory reservation, represented by Committed Bytes, which is charged against the Commit Limit and that there is no specific tie-in to where that memory reservation is being held.

Modifying an INI File

Occasionally, you might find yourself needing to modify a file which has an ini-type format, i.e. setting=value.  For example, you might be doing an unattended install of SQL using a script which needs to modify certain values in the configuration file.  Or perhaps you’re installing Micros-Retail’s XStore product which makes extensive use of files that use this format.

There are two methods I commonly use to do this type of thing, and which method I choose depends on how many values I’m modifying.  If it’s just one value, I’ll use the one-liner method but if I’m modifying many values I use a custom function I wrote to do this.  For the purposes of this post, let’s assume we’re customizing a SQL install and want to vary the instance name with each install.

The One-Liner Method

001 002 003 004

$iniFile = 'C:\Temp\SQLInstall.ini' (Get-Content $iniFile) | foreach { $_ -replace "INSTANCENAME=.+","INSTANCENAME=MYINSTANCE"} | Set-Content $iniFile

Pretty straightforward – any line matching “INSTANCENAME=” will be replaced with “INSTANCENAME=MYINSTANCE”.  While this approach works well for replacing one value, I like to move this work to a function when I’m replacing multiple values.  So let’s expand our scenario to include some other values that we want to modify and see how it looks with a function doing the work.

Using a Function

The basic approach with this function is to create a hashtable of the values we want to replace in the file and then pass the hashtable to a function which then does a search and replace and updates the file.  Let’s take a look at how it all works.

Since the function is expecting to receive a hashtable, let’s start by creating the input it’s expecting and then take a look at the function. 

001 002 003 004 005 006 007 008 009 010 011 012 013

$iniFile = 'C:\Temp\SQLInstall.ini' $sqlInstallParams = @{     "SQLSVCACCOUNT=" = "SQLServiceAccount"     "INSTANCENAME=" = "MYINSTANCE";     "SQLBACKUPDIR=" = "C:\Backups";     "SQLUSERDBDIR=" = "C:\SQL\DB";     "SQLUSERDBLOGDIR=" = "C:\SQL\DB\Logs";     "SQLTEMPDBDIR=" = "C:\SQL\TempDB"; } #Now call custom function to update the config file using the hashtable values Set-IniValue –targetFile $iniFile –hashTable$sqlInstallParams

Nothing much to this step – we just create a basic hashtable with the values we want to replace.  What I like about this approach is that it allows you to see what’s being modified in an easy-to-read format.  Now let’s take a look at the function we just called in Step 1.

001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030

function Set-IniValue ([string]$targetFile, [hashtable]$hashTable) {     # Get contents of current target file     $content = Get-Content $targetFile     # Loop through hash table of new values and compare each line of     # file with each key from hash table     # If no match is found, add line to text file     $hashTable.GetEnumerator() | ForEach-Object {         if (!(Select-String -Pattern $_.Key -Path $targetFile -Quiet))         {             $newText = $_.Key + $_.Value             Add-Content $targetFile $newText         }         else         {             foreach ($line in $content)             {                 #If line is found, replace it with key and new value                 if ($line -match $_.key)                 {                     $newText = $_.Key + $_.Value                     [IO.File]::ReadAllText($targetFile).Replace($line,$newText) `                       | Set-Content $targetFile -Force                 }             }         }     } }

The function starts off by checking if any of the entries we want to modify don’t exist in the ini file.  If any are found which don’t exist, they are added.  It then checks for any matches with the current key of the hashtable and modifies that line with the corresponding value from the hashtable, saving the ini file after each change.  And that’s pretty much it – I’ve used this function thousands of times over the years and it has never let me down!

Cleanup

Set-Content adds a blank line to the end of the file each time it is run so if you’re modifying a few values you can end up with a number of blank lines at the end of the file.  If you want to clean this up, you can add this line after the function call to remove these blank lines.

001	(Get-Content $iniFile) | where {$_.Trim() -ne "" } | Set-Content $iniFile