myShoggoth

Space Saving Ideas

Enlistments in the source for many of the projects around Microsoft can be very large, and it only gets worse when you need to work on multiple branches at the same time (see my article on Branches for good reasons why you should often be working in different branches simultaneously).

The nice thing about source code is that it is very compressible.  Go to the root directory for the enlistment, right click on the folder and select Properties.  Down in the Attributes section, click on the Advanced... button.  Choose the 'Compress contents to save disk space' checkbox.  When you hit okay or apply on the Properties dialog, it will ask you if you want just that folder or all subdirectories and files (you want the latter).  It will then take a bunch of time, based on the size of the folder, to compress everything down.

Use whatever the best method is for cleaning out objects and other build results to speed up compression.

Once you've set this option on a folder you no longer have to worry about it, new folders and files will automatically be compressed inside it.

Branching Theory

This is based on a conversation I had with a coworker in my former team, my own experience, and my thoughts stemming from that experience.

Basically, most people do branching structures in source depot wrong.

Trunk Branches Have More Churn

The root branch for the entire branching structure should be the work going for the most future version of the product being developed.  The reason this has the most churn is that all the work going into every other branch by definition should go into the next version.  This is why...

Changes Flow To The Trunk

Changes being made for a closer release than farthest out being worked on should be made in the farthest out, too.  In other words, let's say I've got a product with three versions:

• 1.0 - Released
• 1.1 - Schedule for release within a few months
• 2.0 - At least a year out

Since 2.0 is the farthest out being worked on, it should be the root branch (or main, or whatever you call it).  The 1.1 development is a branch off of it (I would call it 1.x), and 1.0 is a branch off of that (1.0.x).  Why this naming convention?  Let's take a few scenarios:

1. I need to make a patch for 1.0.  I've got a branch to do that in (1.0.x) that won't be destabilized by changes being made for other releases.  However, all the changes that are made there in almost all cases are wanted for all future versions of the product, so the changes should be made and the reverse integrated to the 1.x branch and then to the root branch.
2. A new release (1.2) is scheduled.  When this happens, a new branch should be created off of 1.x called 1.1.x.  Development of 1.2 continues in the 1.x branch, while stabilization and release for 1.1 moves to the 1.1.x branch.  Note that any patches needed for 1.1 after release now have a home, too.  All changes for 1.1.x are reverse integrated up to 1.x so that 1.2 is a superset of the functionality and bug fixes in 1.1, and all the 1.x changes are reverse integrated up to the root branch so 2.0 and all future versions are likewise supersets.
3. A fix is made into the 2.0 code (in the main branch) that should become a 1.0 patch.  This is the exception, in this case the change is propagated as a forward integrate into the 1.x and then the 1.0.x branches (also the 1.1.x branch assuming this is after the second example has happened).  This ensures that the fix exists in the entire hierarchy.
4. A 2.0 beta release is scheduled.  A branch should be created off the root (at this point I would normally get ahead of myself by creating 2.x as a branch off the root and then 2.0beta as a branch off of it), and the beta code stabilized in there.  Post-beta work continues in the root branch (or 2.x if you're getting ahead of yourself), beta changes and fixes are reverse integrated from the beta branch.

Leaf Branches Have Less Churn

Since reverse integrates are normal (and hopefully scheduled, the more often these happen the better), and forward integrates are rare, the farther you get from the root branch the less change that is happening.  The idea is to stabilize and fix farther away, then reverse integrate the changes to propagate them widely.

Branches vs. Labels

When to use a branch versus when to use a label depends on whether the contents are going to get modified in a meaningful way.  Take the case where you RTM a product, most groups want two different behaviors:

1. A snapshot in time of what exactly was release for reference purposes.  Because this will not change in a meaningful way (what was release is what was released), a label is correct.
2. A branch in which to do any patches against the version released.  This is a branch that will likely change, but it is different than the label.  The label exists as a reference point and a way to reconstruct the exact inputs that built the product that was RTMed, the branch exists for changes to be made in.

Stabilizing

There is often a big desire by product groups to be able to stabilize on a regular schedule and release self host quality builds for testing and dogfooding.  For instance, the end of coding milestones is a very common time for teams to push for stability and dogfoodability. The normal way this is handled, however, can bring all progress to a halt until the last part of the team is done.

The correct way is to make a branch off the version being developed for stabilization.  Stability changes can then be reverse integrated as the work progresses while unblocked work continues in the normal branch.

Build Scalability

As soon as a project gets to a certain size developers want to break it up to decrease their turnaround time for changes.  The common solution is to have a set of headers in a 'common' include directory that is separated from the source, the idea being that as long as you're not touching those headers you can build the subdirectories of source independently of each other without worrying about dependency issues.

Common Header Includes

On a smaller scale this works great, the developers are all enlisted in all the source and maybe only do a world build every now and then (when they sync everything), otherwise they're able to just build the portion of the source they're actively working in.  The only way this is broken is if people break the model and do a direct dependency between one of the subdirectories of source to another without using the common directory (that's bad).

Note that, given this model, you must rebuild the world if you sync anything in that depends on common header changes.  You also have to be careful about implied dependencies between different libraries or DLLs, such as a resource change in a localizable resource-only DLL that is required by code in a different file (maybe an EXE).  The correct way to declare this dependency is through header files (such as a resource.h equivalent in the common directory).  This can easily fall apart if there's an implied structure to the resource that changes in a way that doesn't impact the common files, however.

The larger the project gets, the more likely these subtle side effects will happen.  The more people working on the code, the more likely it will happen on any given day.  Since the time for a full build is usually large by this time, developers demand support for independently buildable portions of the source tree.  The common include directory solution doesn't hack it, anymore.

Pre-Built Binaries

The next step is to solve the binary file problem.  To be able to compile a portion of the tree independently of the rest, you must have what would be the results of the rest of the compilation available to you in some form or another.  Commonly this is done by having the group's build lab publish the results of their official builds.  The developer's build process then picks up the portions the developer is not enlisted in from this official build and compiles from source the portions they are enlisted in.

The usual model in this case is to have the common include files checked in to a central repository everyone is enlisted in, and they separately enlist in the portion of the overall tree that they're working in.  You now need to either be very careful about syncing to well known good build points or work to protect those common headers.  I'll go through both models.

Well Defined Sync Points

In this model the build supports a set of well defined timestamps or labels that mark points where the source was known to produce a good build (for many different values of 'good').  When you want to make changes, you sync everything to one of those labels and make your changes based on that build.

Problems start when you want to make changes that rely on changes made by someone else since that label.  Depending on the scope and dependencies of those changes, you may only need to sync the files they made the changes to and build.  At worst, however, you may need to enlist in everything and build the entire project.

Furthermore, any time you change one of the headers in the common includes you must enlist in and build the whole project to make sure you don't break anything that depends on it.  Same issue with the more subtle binary dependencies I talked about above.

Protected Common Directory

This is the 'publics' philosophy of the Windows build system.  There is a central location for the shared headers (and binaries), but only the build labs can check in changes to them.  This means that they will by definition always be at the sync point good state, since the build labs only check in their changes after a successful build (for many different values of 'successful').

What happens if you need to make a change to one of those files?  The copy that gets changed is in the normal sources, when you make changes to them they get 'published' to the central place during the build.  For things you're not building you get the known good copy, for things you are building you override with the copy you're building.  This applies for headers, libraries, and binary files.

The advantage is that you don't have to worry so much about what you sync to except for the state of the portions of the source you're enlisted in.  However, if someone (or you) makes a change that spans across multiple areas, changes common headers, etc. you're still in the same boat.

Future

The general form of this problem is:  I'm only making changes to this, that, and the other thing.  I want to only build what I have to and take advantage of official build lab work that has already been done as much as possible.  I only want to incrementally build what I have to, but get a stable, predictable, and usable result.

The Next Big Thing

Yup, I've figured out what the next big industry will be. Ready?

Robots.

Check out early, useful items such as the Roomba and the Robomower. Then look at the other end of the spectrum with the His and Hers Robots, attempting to take over the butler trade.

Why is it the next industry? A few reasons:

1. It is the next evolution up from appliances. Appliances are machines that are the evolution above tools. In other words, tools make you more efficient at doing tasks (or in some cases make tasks even possible), but require a lot of labor and skill. Appliances take away much (but not all) of the necessity for labor and skill, but still require time and some amount of effort. Robots as independent entities that perform the tasks instead of you take away the time and effort as well, and now the skill is programmable and upgradeable.<
2. The growth in the general purpose computing industry is running out of steam (it still has a long future in it, don't get me wrong, but the very early exploratory days are behind us).
3. There isn't a lot to be gained by having one robot that does everything vs. a robot per purpose. Most likely the eventual trend of the industry will have its version of the general purpose computer, but the way for companies to get early traction is through targeted items like vacuuming your house or mowing your lawn. Similar to having separate washers and driers for clothes, it is much easier (and more economical) to make them separate and better at one purpose than trying to combine them.
4. Productivity gains. The improvement between "we can make you take only an hour to vacuum your house" and "your house will just plain be vacuumed, don't worry about it as long as you make sure this robot gets some amount of maintenance" is enormous, and shouldn't be underestimated.

I still don't hold out much hope in the near future for flying cars, though. Oh well.

The Nature of the OS

[Note: I originally posted this 5/28/2003.  After spending time in the Core Windows Team I now believe more of this is well thought through than I did before, but there's still a lot in Windows and other OSes that have a very 'tacked on' feel that I mention here]

I bet someone is doing this, but I haven't found it yet.

When I think about how an Operating System is built, naturally I fall back to what I learned in my OS 101 class in college. Memory management, I/O, file systems, yadda yadda yadda.

The nature of what an OS is has changed, but we don't have a taxonomy of what the new landscape looks like. What is now the realm of the OS and what is the realm of the application? That distinction has blurred in a big way, and may go even farther.

OSes are taking on more application-like functionality, and a lot of the cool functionality that used to be put into traditional applications are now going into things such as plug-ins, shell extensions, and other value-adds to either the OS itself or applications that have been built on it.

This brings us back to what we at Microsoft consider to be one of our big strengths: integration. The days of applications as vertical silos of functionality are long gone, and this has been a weakness for the UNIX world for a long time.

The most interesting (and I say 'interesting' here in the sense of the old 'may you live during interesting times' saying) part of building a highly integrated component, whether it be more app-like or more of a shared API that people build on is what happens when more than one plug-in, 3rd party app, etc. want to use your stuff all at the same time? What's the right way to handle it? What's the right way to handle someone holding a COM reference to your application when the user wants you to exit?

These are all problems that need to be addressed fundamentally in the platform, in the OS, in order for things to work right across the board. We got relatively consistent UX for applications when enough of that interaction was put into the platform (/ OS) that each app didn't reinvent it themselves. To the extent they do reinvent the interaction models, the OS has failed and the platform will find it harder to succeed.

So I argue that we should create a new taxonomy for the Operating System, not only to take into account all the things it needs to do that modern ones do currently (catch up the model to realities of today), but be more forward looking to what things it will or does need to do.

Some things I think should be addressed in the taxonomy:

• Software distribution
• Schema-ed data storage/query/etc.
• Fundamental network interaction
• Power management
• State management
• Rethink IPC
• Authentication/ACLs

Many of these things are available (or will be), but how integrated are they? Do they feel as seamless as the memory manager, the file system? I don't think so, they feel like tack-ons.

My Version of the Blogging Tool Concept

Everyone seems to have their own, so I thought I'd talk about the code I've been working on in my "spare time", although I haven't made any progress for a long while due to an exponentially growing work load... [Note: this was posted originally 5/20/2003 and is no longer the way I do any other this]

First, the tool is about website design, not web page design, not blogging. I wanted a way to do photo albums, blogging, articles, and more types of content over time. I wanted to automatically generate things such as a breadcrumb bar, thumbnail pages, next/previous links for albums, etc.

The only thing I liked about one of the tools I played with was that idea of having the entire website contained in a single file, and since about a year and a half ago I investigated and later implemented a prototype C# version of the DIME format (I'd link to the spec, but it looks like it is MIA), I selected it for the file format. The first DIME record is an XML description of the site and its well-schema-ed contents. This XML can refer to binary content in later DIME records by GUID ID value. Any files that aren't XML and just need to be dropped are also located in later DIME records.

The build process itself is a two stage process:

1. Create subdirectories and drop all files that don't need processing.
2. Run through the interesting pieces of the site XML and do processing for the nodes that require it.

The processing involves a few steps:

1. Generate the list of pages that need to be generated for the node being processed. For example, in a blog there might be a default page with a certain number of entries, one page per entry and/or day, summary pages for days or months, archives with the next X many entries, etc. It also might generate non-HTML items such as RSS feeds that mirror all the pages, or whatever is desired.
2. For each of the items to be generated, build the XSLT arguments and determine the right XSLT to use.
3. Run the site XML through the XSLT passing in the arguments. This generates an XML file that will then be processed based on the target file type. It will be in terms of generic controls plus data (this is the next thing I plan to implement).
4. Then each of these XML files is run through the generic XSLT that turns the XML file into the target format.

Why did I go to all this trouble (besides being a total geek who can't do things the easy way)? I want a proper separate between the data to be stored (blog entries, photos, whatever) and the way they're displayed (my goal is strict XHTML plus CSS, I've gotten that working in the past, but having an RSS feed for the blog entries would be keen).

I've had this system running and generated what you see on http://www.mareimbrium.com/ using it, including the cool photo navigation. Then I broke everything (of course), and am in the process of rebuilding it. The previous version was a one-stage XSLT, now I want to move to the two-stage XSLT process. The DIME format, site XML, files in separate records, all of that stuff was working fine. One of the things I was missing was a decent way to edit content -- this is the main thing I like about JoeBlogger and some of the other tools I've seen.

Blogging Is Not The Answer

The title of this entry is a little misleading, but controversial for a reason. I see happening with blogging, but more RSS in particular, the same sort of oddness that happened with HTML: People latched onto a simple, targeted technology and tried to make it solve a huge class of problems.

For HTML, it was a simple markup language meant to allow researchers to easily reference each other's work in online versions of their papers. It has since been attempted to make it the next UI platform, with significantly mixed success. The process of going from what it was intended to do to what people tried to contort it to do has been a very painful road.

For RSS, the intent is clear and stated in the what RSS stands for: Really Simple Syndication. I believe people are trying to morph/evolve/turn it into a generalized solution to a more fundamental problem:

I have a bunch of information that I would like to make available. I want it to have a well understood data schema. I want to be able to easily customize the view I project to other people, but allow them control if they want their own way of looking at it. It should be easy to navigate and make sense of this data.

A lot of the fun investigation work around JoeBlogger, BlogX, BlogCandy, RSS Bandit - there are too many to count - revolves around the concept of authoring and publishing well schema-ed data, but specifically for the one schema (RSS). Once people are happy with the tools (if that is ever possible), they'll try to apply the same ones to a different aspect of the problem space. An example I've been working on is the photo album problem. After that, how about a list of books, or DVDs? Maybe a set of articles in DocBook format, with an RSS feed generated?

The meta-problem is defining data schemas, being able to sanely store the data, easy authoring, easy design of the way that I publish it (maybe as a web site, a small Avalon app, you name it), and then a way to publish the data in formats that are easily consumed so that others can take the raw schematized data and view it in their own custom way, too. This could include XML files ala RSS, web services, and more.

How Should Software Be Built?

My chief issues with the way that software is built (not the processes around designing it, figuring out feature sets, taking it to market, all of that -- rather, how it goes from lines in text files to machine instructions) is that it goes the wrong direction and the concepts are 20-30 years old.

Wrong Direction?

The way software has always been built is programmer-centric. This makes sense historically, but modern software is very much not coder-only, and the extent to which all the processes revolve around the developer is a major cause of coordination nightmares for releasing products. Think about all the groups involved in taking a substantial sized software product to market:

• Software Developers
• Testers
• Developers on the test team
• PMs
• Localizers
• UA
• Design
• Builders
• Legal
• PSS
• Beta program managers
• Partner and dependency teams
• Platform teams
• Marketing
• Branding
• Management
• Setup
• I know I'm missing some...

The Right Direction?

Start from the end result! This has normally been the realm of the Byzantine structure that is called the installer, but really should be the goal of the entire build system. This means what registry settings, what files, what settings, etc. The important thing for the files are what exports are required for DLLs, what typelibs are exported, essentially everything that is externally required to be in that file. Everything else should be flexible for performance, optimization, and flexibility reasons.

Old Tech

Why the blazes are we still editing text files to create computer programs? Why do we still have imperfect symbol resolution, imperfect dependency knowledge, weird build rules? Why are resources, which are such a fundamental aspect of production quality software development, still such a tack-on? I'll no doubt rant more about these issues in future entries, I think I've dragged this one along far enough.

The Next Few Entries

For a time I had an internal blog at Microsoft for posts that were either only really internally relevant or contained information that wasn't suitable for public consumption.  For assorted reasons I didn't keep it going, and the few poor people who were looking at it gave up, so when I moved groups most recently I decided not to try to put it back up.

I did name my main desktop computer 'myshoggoth' on the internal network, but that's a given.

Looking over the posts there are a few items that either as is or with some small amount of editing can be made publicly postable, so the next batch of articles will be ported posts from that internal blog.

Branding's Gaping Void

During my visit to Paris I attempted a meeting of opportunity to see if hugh at gapingvoid dot com would be available (since he recently relocated there for an underdetermined amount of time).  Beyond just a normal "hey, we're two bloggers in the same general area we should probably get together for some reason" sorta stuff, his thoughts on the demise of branding and how that might influence me, my project at work, and my company are very interesting.

Unfortunately it didn't pan out.

If customers don't know you and your personality isn't coming through to them directly, I'm convinced they will assign one to you.  It may not be flattering.  For many customers and anti-customers, the personality assigned to Microsoft has been far from flattering. That definitely seemed to be the state we were in when I joined Microsoft a little over four years ago.  And I was pretty anti-Windows, anti-Microsoft as well when I joined.

More recently with the community revolution here, that is changing.  We're letting individuals have more direct access both directions, where Microsofties can interact and let their personalities be known directly to customers and customers have more access to more Microsofties.  This means that for more of our customers, the personality (or personalities) they associate with the company are based on the conversations (as hugh would put it).

The alternative is the customers assigning us personalities based on random experiences, good or bad, with the software that have for whatever reason made a particular impression, or based on wacky corporate moves where it is pretty much impossible to have all the context without having been the decision maker themselves.

You may not be able to get as clear of a handle on my personality based on this blog as you can with Scoble based on his (for example), but it is infinitely more accurate than if you had no access to any of my thoughts at all.  I hope it helps, please drop me an email if there's ever something specific you'd like me to talk about or questions you'd like me to answer, I'll talk about what I can and answer questions as I have the expertise and time.

Oops, I burned the bacon I was cooking while writing this...