myShoggoth

Why Does Setup Make Windows Reboot So Much?

The short answer is: because memory mapped files are locked by the system and can't be changed while they are in use.

For the long answer, I want to go way back to how programs are run on a computer and begin at the beginning...

Running a Program

A computer program consists of a bunch of instructions to the CPU (for instance, the 1.3 GHz Pentium III) of the computer it is run on, and some additional data used by those instructions.  These are very detailed, taking the form of:

Take the number four and the value of address 4000 in RAM and add them together, put the result in address 5000 of RAM.

When you run your program, those instructions need to be read into RAM (the memory of the computer) so they can be available to be loaded into the CPU to tell it what to do.  There are a few options on how to load those instructions into RAM.  One of the most straightforward is to read the whole file (say, program.exe) in and keep it there, but it has a few drawbacks:

• It uses a lot of RAM.  Most programs use shared libraries (in Windows they're called Dynamic Link Libraries, or DLLs) to re-use functionality already in either the operating system or component pieces the program writers created themselves.  In other words, you not only have to load your program's file into memory, but also all the files of all the DLLs that the program uses.
• It is slow.  Reading from disk is one of the slower things you can do (relatively speaking) with modern computers.  Anything that can be done to minimize disk usage will make the overall system and the program you're running faster.
• • This is especially true if the program you're running, and/or one or more of the DLLs it is loading, is being accessed across a network instead of local to the computer.
• It is wasteful.  There are parts of programs that are not used most of the time.  Some users run Microsoft Word, as an example, and are very comfortable and skilled with a particular set of its functionality, but the Office suite in general has enormous functionality and very few people use a large portion of it.  Why take up a lot of RAM and make everything slow to load instructions for the CPU that will never be needed?

Another option, the one used by Microsoft Windows, is 'memory-mapped files'.

Memory-Mapped Files

Let's say we have our file program.exe, and it is 5,000 bytes long.  If we memory-map it, we associated a RAM address with the beginning of it, and the addresses from that start through the size of the file are now associated with the data in the file.  If RAM address 5000 was the start of program.exe's memory mapping, then RAM addresses 5000-10,000 would be the data in the file itself.

The operating system keeps track of this association, so if you access RAM address 6000, it knows to get the data from the file if it doesn't have it in memory already.  This is important, because now that there is this association, the operation system only needs to read from the disk when someone asks for a value. In practice it does this a page at a time, which means that if you ask for any information whose address is within the page, or predefined size chunk of memory, it will fault it in, which essentially means stop everything and read the data into memory.  Any requests for other information within the same page will find it already in memory, and no reading from the disk is necessary.

The operating system can be quite clever with this scheme, and keep track of which pages haven't been used in a long time, free up the space being used by it, and fault it back in later if someone wants information out of it again.

Some people reading this will said to themselves "Isn't this how virtual memory works?"  The answer is yes, virtual memory is implemented by a paging file which is just a memory-mapped file the system uses to store more information in the virtual RAM than there is physical RAM on the computer.  In fact, memory-mapped files are used for all sorts of things, including letting programs running on the same computer communicate with each other (inter process communication).

Sharing Memories

So far so good, but we quickly run into a snag with both of the options discussed above, what happens when more than one program wants to use the same information at the same time?

Let's say I've got a DLL called mydll.dll (believe it or not, such DLL naming conventions do exist), and it is being used by myprogram.exe and yourprogram.exe at the same time.  If you are going with the 'load the whole file into RAM' approach, you have to decide whether to load mydll.dll into memory a second time or share the already loaded version with the new process.  The former is wasteful and uses a lot of RAM as well as being slow (see the drawbacks listed above), but it cleanly takes care of the problem of what happens when mydll.dll has actually changed between when the first program started and when the second one did.

The drawbacks to having every program have its own version of the full copy :

• It uses a lot of RAM.  Look familiar?  I'm wasting RAM by keeping duplicate copies of the same information most of the time.  If I've faulted in the same page for the same DLL in two different programs, I would have those pages in physically different memory taking up twice the space they otherwise would.
• It is slow. Keep in mind that every time I fault in one of these pages I don't have to or read into memory a file I otherwise wouldn't, I'm accessing the hard drive (at best).  This slows down your computer in a huge way, don't underestimate it.  In addition, if by faulting in pages from a memory-mapped file I use up all the physical memory, I might have to page out stuff that is currently in memory, which means to write it out to disk and make more room in the computer's physical memory.  If that is information a program is going to need soon, it will have to be faulted in again soon, which creates a cycle of slowing down the computer. This kind of thing can lead to page thrashing, where a computer is spending so much time reading and writing pages to files that almost nothing useful is being done.
• It is wasteful. Except in very specific cases, most of the time for most programs the files we're talking about will not change between one program running and another running.  This means that the system would be optimizing for the exception, rather than the normal case.  Not good design if you want programs to run quickly and efficiently on a day to day basis.

The same basic choice exists with memory-mapped files as well.  Do I share the memory-mapping between the processes or keep them completely separate in order to deal with this versioning problem?  However, this doesn't work at all with memory-mapped files because the actual information is stored on the hard disk, not in memory.  If, halfway through running my program, the information gets changed and new pages of instructions and data from the new version of the file start getting faulted in, the program will begin executing instructions it never expected and doesn't know anything about.  This is bad.  In fact, the operating system needs to make sure this can't happen.

Locked Files

It does so by 'locking' the memory-mapped files.  The prevents anyone from changing the file out from underneath the programs using it, so we don't need to worry about getting unexpected changes to the instructions and data while running a program.  This is good, it makes the system stable, but it has one major drawback, so I'll try to bring everything I've said together into what I hope is a coherent whole:

1. For efficiency and performance, executable files and DLLs are accessed as memory-mapped files by programs running on Windows.
2. To allow multiple programs to share the memory-mapping of those files and to protect against the contents of those files changing underneath running programs, executables and DLLs are locked against changes while they are in use by any programs.
3. Windows itself is written as executable files and DLLs and runs as programs which lock those executables and DLLs.
4. Installation programs and programs that update Windows itself are written as executable files and DLLs that run as programs which lock executables and DLLs they use.
5. If the installation program needs, for whatever reason, to update a locked file, it cannot do so.
6. If the installation program for an update to Windows, for instance, needs to update a portion of Windows that it is itself using, it cannot possibly succeed.

Ut oh.  Through the joy of logic we've managed to paint ourselves into a corner and can't get out.  But of course most people who own or use a Windows machine have installed an update, a service pack, or a program that updates executables or DLLs that are in use (in fact, this is why most installers 'recommend highly' that you exit every program you possibly can before continuing installation, to reduce the number of locked files they're likely to encounter).  What's the solution to this problem?

Reboot

The solution is to put the new version of the file into a temporary place and add an instruction to the boot process for Windows to tell it that, next time it runs (next time the Windows system boots up), copy this new file over the old file.  The old file will not be locked because nothing except the Windows boot process is running, and it doesn't depend on anything but itself.  In other words, you break the deadlock by putting the computer into the only state it has where nothing can be locking a file, then copy the new version onto the old one.  Then encourage the user to reboot the computer, one way or another.

The Past

This was a much bigger problem in the past, and in previous versions of Windows (prior to Windows XP).  New programs would often install DLLs that they used but that weren't available in the earliest version of the operating system they supported, and would do so to the system directory.  Reboots have become quite a problem, however, so most companies now go to great lengths in order to avoid them by either having private versions of the DLLs in their own directories (which may waste hard disk space, but that's cheap enough these days) or by doing sophisticated detection during runtime to tell a user that, although their program is running fine, certain functionality won't be available until they reboot.

Having said that, the less adept developers may create programs that still cause this problem.  Types of installations that can still reasonably (more or less) require a reboot:

• Operating system updates.  Note that even here Microsoft goes to great lengths to try to avoid reboots in patches whenever possible.
• Driver updates.  Chances are if you're updating your video drivers the old ones are being used and their files are locked.
• ... Not much else.  If you install a normal piece of software and it tells you that you need to reboot for installation to complete, the odds are that blaming lazy developers is a winning bet.

The Present

There are a couple new operating system features that have been created to address these issues, and have the potential to eliminate most of the causes for reboots:

• System File Protection (SFP)  This is a feature to help security for the Windows user as well as help the reboot problem.  In sum, most of the files that Windows installs are locked down from the start, so nobody except Windows updates can change them.  This helps prevent malicious programs from copying themselves over a Windows executable, for instance, and also reduces the chance of program installations requiring a reboot since they can't even attempt to change those Windows system files.
• Side-by-Side Installation (WinSxS)  Up above I said "this doesn't work at all with memory-mapped files" when talking about having the two different versions of the same DLL coexist at the same time and be used by different programs, but of course there is a way.  Have two different DLLs.  Then you have to solve the problem that when a program says it needs functionality in mydll.dll the operating system needs to know which one is the right one.  Side-by-side does this problem my keeping a bunch of information about both the program, which versions of those DLLs it is dependent on, and the location of the different DLLs and which version they are.  When the program loads, it tells the operating system to load the DLL, and the OS knows the right one to load.  Since the different versions are physically different files, there's no problem.  If you install a new version, it is put into a new file, the old file remains, and there is no need for a reboot.

The Future

The ideal, of course, is for a reboot-free computing experience.  Expect more features like SPF and WinSxS in future versions of Windows, and even enhancements to those.  I'm not personally aware of what work is being done in this area, but I have no doubt there is.  This is a well understood although difficult problem area, and a lot of work has gone into and will no doubt continue going into making the experience better and hopefully eventually solving the issue once and for all.

The Joys of CSS

I don't know the exact name of the group of people who hold similar XHTML + CSS views as I do (other than 'snobs' or 'impractical'), but there's a certain amount of practical application  to it as well as style and simplicity.  This site's design is not so much a culmination of my CSS knowledge as an intersection between it plus my personal aesthetic.

The original intent of this site was to explore ideas around XHTML and CSS, and web sites created with those plus the necessary tools.  A while back my work project was heavily based on XML, HTML, CSS, XSLT, etc., but my current work has nothing to do with them in any meaningful way so I think about them much less these days.  Still, tipping my hat to the reasons I set up this site, I thought I'd talk about some CSS stuff.

This is not meant as a primer on CSS itself; for that I can refer to you to a friend of mine and the book he wrote for that purpose.  Instead I'll talk about how I set up a part of my pages and styles and why I made the decisions I did.

I like to have the styles defined in their own files linked to by the pages.  These styles should be related to each other either logically or functionally.  For instance, I have one CSS file for my site, and one for the logo for my site.  Why?  The site CSS is used by all the pages on it (naturally) and they all apply to the whole site.  Individual pages may not use pieces, but I don't expect big portions to change from section to section.

The logo CSS is separated out so if someone was to want to create a link to me use my "logo", they could do so by including a tiny CSS file and using:

<span class="myshoggothlogo"><strong>my</strong>Shoggoth</span>

And it would look like this:

myShoggoth

The version in the upper right corner of my pages (at least, that's where it is as of this writing) is modified by the div that wraps that:

<div id="logo">
<span class="myshoggothlogo"><strong>my</strong>Shoggoth</span>
</div>

That allows the 'logo' styling to be independent of the size and placement in a particular use.  Also notice that the fact that I use span and 'strong' tags make it very generic, so that even without any styling applied it still has some kind of look to it that is unusual.

The styling itself for the logo is:

.myshoggothlogo { font-family: Marigold,"Palatino Linotype",cursive; color: blue; font-style: italic; }
.myshoggothlogo strong { font-size: smaller; color: gray; text-decoration: overline; font-style: normal; }
.myshoggothlogo a { text-decoration: none; color: blue; }
.myshoggothlogo a:hover { text-transform: capitalize; }

I prefer the logo in the Marigold font, but understanding that it is somewhat rare I asked for Palatino Linotype to be tried second, otherwise the generic cursive font is fine.  The overall color is blue and the characters are italic.  For the strong part only (the 'my'), I want the font to be one size smaller (notice I don't use any font size here; this is important), gray, non-italic with an overline.

If it is a link to my site, I want to turn off the underline and do my own thing, which is why I explicitly set the <a> values.  Then, when you hover the cursor over the link it should capitalize the 'm'. Like this:

myShoggoth

I seem to have exposed an interesting IE bug with the above example.  This is not a rare thing when you play with CSS in depth.

What is important (at least to me) is that if any of this isn't supported by the browser or gets stripped off by browser settings or restrictions, it still looks good and should work fine.  The font size aspect (ha ha) is very important since it allows people who have selected a large default font size in their browser to see what they should, not what I said they should, so the 'my' is one size smaller than the font of the rest of the logo, not 9 points or something predefined.

I use this in my site CSS by defining styling for the logo div that surrounds the logo span.  I can have that logo span all over the place if I want, put it where it needs to be, and make it whatever size I want, since the logo styles are agnostic about it.  The user can define something fancier, such as:

/* Site logo */
#logo { text-align: right; font-size: 300%; margin: 0em 1em 0em 0em; }

Here I put the logo on the right, make it 3x the size of normal text for the browser (I still let the user determine what size they want it by defining the default font size), and I move it in from the right one 'em', which is a unit of measurement based on the font size.  This means when you change the browser's default font size, the logo will be positioned correctly according to the size of the font on the page.  Try changing the font size for your browser (View->Text Size menu in IE), you'll see why that makes the most sense.

RSS Feed Ahoy!

As I hoped and suspected, setting up an RSS feed in CityDesk wasn't too bad. The joys of a CMS, I guess. All it took was a quick check of the CityDesk discussion forum, one attempt to do it a broken way (sited at the top of the discussion), and then someone gave this link: http://www.henko.net/Computers/CreatingRSSfeedsusingCity.php

Now that's pretty.  Slick, uses the system as-is, and is very flexible.  Nice work, Henrik Jernevad.

Fun With Scripts

Computer usage can turn out to be an exercise in repetition, often defeating the purpose of having a machine capable of automating repetitive tasks available. When you're doing that repetitive tasks on the machine designed to get rid of them, we're talking real irony here (not the Alanis kind).

One day some friends of mine pointed out that the steps I was taking to launch Battlefield 1942 and connect to our GameVoice server were repetitive and could be scripted. When I stopped smacking myself in the forehead for not thinking of it, I took the script they had started and did my pedantic developer magic with it.

Since it is an interesting exercise in Windows scripting, I thought I'd share both the script and some details about how/why it works.

Start by opening up your favorite editor, notepad or maybe VIM or suchlike. We'll begin the script by launching GameVoice and telling it where to connect to. Those guys did a nice thing by registering a url namespace handler with urlmon, so we can simple execute the address of the GameVoice server by prepending gamevoice:, like you could ftp: or http:.

REM -- Start GameVoice --
start gamevoice://server.address.here:portnum

Fill in your GameVoice server's address there, as an example (you can use DNS names too):

start gamevoice://192.168.1.2:2350

The one thing that they screwed up is if you execute that URL twice they still prompt you if you want to leave the previous chat, instead of detecting that you're already in the same room and swallowing the notification. Oh well.

Since the game itself won't be in my path, and since many games want to be executed within their own directories, the next thing to do is figure out where that path is. You could be simplistic and hardcode your own path in, or code in the default installation location, but in this case I found out the way to figure out the real location even if the user changes where it is installed to during setup:

REM -- Start BF1942 --
REM -- Start with the default install directory --
set mybfdirectory="%ProgramFiles%\EA Games\Battlefield 1942"
REM -- Now grab the real install directory from the registry --
REM -- NOTE: The delims below is a single tab, this is crucial --
for /f "tokens=3 delims= " %%I in ('reg query "HKLM\Software\EA GAMES\Battlefield 1942" /v "GAMEDIR" ^| findstr "REG_SZ" ') do set mybfdirectory="%%I"

First I create a variable and initialize it to the default installation location, note I don't assume C:. Next I went to see if I could determine the real location from the one place that should have it right, the registry (BattleField's setup would write the value, in this case). Note that each game can put this info whereever it wants to, or not. Here I'm using the 'reg.exe' command to 'query' the registry in the location HKLM\Software\EA GAMES\Battlefield 1942.

HKLM -> HKEY Local Machine, or settings global to the computer
Software -> A 'subdirectory' from HKLM, this is where software installations normally put their settings
EA GAMES -> Normally software authors or publishers will create 'subdirectories' (they're not really subdirectories since this is a database of sorts, but you can think of them that way) in the registry to store the settings for all their stuff in one place.
Battlefield 1942 -> Settings related to this particular game

the /v option asks for the value of a specific key which is then named next (GAMEDIR), which in thise case hold sthe installation location for the game. It is safe to run this command, since you're only reading the registry and not writing to it, so if you have Battlefield 1942 installed go ahead and try:

reg query "HKLM\Software\EA GAMES\Battlefield 1942" /v "GAMEDIR"

You'll notice that the output isn't terribly clean, it has six or seven lines of output, blank lines, and the information we're after isn't on a line by itself. That's where the other fun stuff comes in. The 'for' loop iterates over the tokens based on a tab delimiter (that's why you can't see it after the delims= statement). The findstr makes sure I'm only acting on the line that has the string 'REG_SZ' in it, so it will iterate over the three tokens on that line. Those tokens are:

GAMEDIR
REG_SZ
C:\Program Files\EA GAMES\Battlefield 1942

The first is the registry key, the second is its type (string with a zero or null terminator) and the third is the prize we're after. Why does this work? Because it does the body of the loop for each of those items:

set mybfdirectory="%%I"

The last time through it sets it to what we want it to and then it is done. Now we have the correct path to the installation directory every time! Just that easy. Next, make that the current directory and start up the game:

pushd %mybfdirectory%
start bf1942.exe +game XPack1 +joinServer s1.201st.com:14567 +restart 1
popd

Why use pushd/popd instead of cd (change directory)? Because it remembers where you were and popd goes back there. It has a stack of directories, you can push them on and pop them off. Works across network shares and the lot.

The arguments to bf1942.exe are specific to that game. Finally, cleaning up the environment:

REM -- No environment variable turds --
set mybfdirectory=

Save the file to 'whateveryoulike.cmd' to make it a command file, and you can save it to your desktop or where you would prefer, and do it one double-click what you did in many steps before.

For more information on these commands and capabilities of cmd.exe in Windows, Start->Run "cmd.exe" and type:

help cmd
help for
help set
help start
reg /?
reg query /?

Software Installation Sucks

This has been a beef of mine recently, and installing CityDesk reaffirmed it. Software installation sucks.

This is a large topic, so I'll just talk about one issue: companies have done a poor job of helping users cope with their software. This has gotten a lot better with Add/Remove Programs on Windows, RPMs in the UNIX land, etc., but still has a long way to go.

Let's take the Programs menu in Windows as an example. Most software likes to install itself at the top level, in fact it is a revelation to most vendors (including Microsoft Office until recently) that their stuff isn't so important it should have each item at the top level. Granted, Office is important to many users, but that doesn't help them manage their software and their systems. In fact, with Windows XP's MRU on the new start menu, one could argue that any good reason that Office had for listing their apps at the top level was gone. Luckily, they seem to agree and their new version doesn't appear that it will continue in that tradition.

When a program puts a submenu/folder there instead (always at the top level, of course), it still doesn't really help the situation. The name of the menu is either the program's name, or the name of the company that writes or publishes it. This is very noticeable with games, in particular. There are some games I think about based on their publisher, but not many. Most importantly, they are games, and should be in that folder.

Programs should self-categorize when they get installed. All games should go in Start->All Programs->Games. Bonus points for those that can agree on further categorization, such as genre. I would kill for:

Start->All Programs->Games->First Person Shooter->Battlefield 1942

I would also enjoy expansion packs and such sorting themselves nicely underneath the folders of the applications they expand, since they require them to be present anyway. Why should Battlefield 1942's first expansion pack, Road to Rome, be a separate entry? It should, at worst, be a subdirectory under Battlefield 1942, but even better it could be a couple extra programs, links, and other files in that folder that already exists. Help your users.

What about CityDesk? Well, the default is:

Start->All Programs->Fog Creek Software->CityDesk

Where CityDesk is a program and is the only item in that folder. I don't like this approach for a few reasons:

1. If you only have one item in the folder, you have done nothing but create another level of indirection for no reason.
2. There may be some people who think about CityDesk by considering the company that makes it, but that's a company- rather than user-centric model.
3. I have no context about whether CityDesk is a game, a web site builder, or a new mp3 playing program except its name and the company it is apparently from.

Aurora, Once Again

I'm now going to try CityDesk from Fog Creek Software and see how it goes. I tried an older version a while back and was not impressed by a few aspects of it, but am giving v2.0 a shot because.

1. They reinvented the wheel with their templates/scripting language. My thought at the time was "Hey, XSL/XSLT exists, why reinvent the wheel?" After having fought usage and usabilities with them for a while, going a different direction isn't looking like such a bad idea.
2. A lot of the flexibility I assumed I wanted I have found over time I'm not as interested in. I'm more and more interested in getting something easy to use and powerful enough to be useful.
3. I'm a computer programmer, for a living no less. I can and have written programs that do this sort of thing. However, I'm paid to work on different software that tries to solve different problems. Why not get paid for what I'm working on and pay someone else (Joel etc.) for working on solving this one?
4. I watched the three demo videos they had, read their 'What's New?' list of features, and all in all I'm not only impressed by what they've done but even more in what they haven't done. It can can some serious intestinal fortitude to avoid doing stupid features sometimes.

So I'm writing up this entry in CityDesk, I'll see how accurately I can reproduce my site with it, and if it is good enough Mr. Spolsky will be getting some of my hard-earned money, deservedly so.