Microsoft Black Boxes

Ray Ozzie and Bill Gates


Happy Belated Birthday Microsoft (who turned 37 last week)…

Having worked at Microsoft for nearly half of Microsoft’s existence as well as more than half of my career it has provided one of my broadest pools of experiences including the areas I explore here. I was struck by Ray Ozzie’s (heir to Bill Gates as technical visionary of the company) memo (going on a little while ago now) that echoed so many of perspectives about ‘Black Box Complexity’ I have been investigating…

  • But as the PC client and PC-based server have grown from their simple roots over the past 25 years, the PC-centric / server-centric model has accreted simply immense complexity…. Complexity sucks the life out of users, developers and IT. Complexity makes products difficult to plan, build, test and use. Complexity introduces security challenges. Complexity causes administrator frustration. And as time goes on and as software products mature – even with the best of intent – complexity is inescapable…Complex interdependencies and any product’s inherent ‘quirks’ will virtually guarantee that broadly adopted systems won’t simply vanish overnight….But so long as customer or competitive requirements drive teams to build layers of new function on top of a complex core, ultimately a limit will be reached.”

It’s not just technical complexity that threatens to forge an inscrutable black box around the code base. But also organisational complexity can make the equally large and intricate matrix of decision making as inscrutable and opaque.

Microsoft 2.0

Microsoft 20 Mary Jo Foley

Speaking of technology earlier this month, my summer reading finally got around to probably the most insightful writer on my professional alma mater, Mary Jo Foley and her latest book on the subject, Microsoft 2.0. I couldn’t help gleaning an embracing failure gem…

“No matter how fault-tolerant and reliable systems are, downtime and outright system failure are unavoidable. Microsoft and other vendors seem to be talking less about 99.999 percent uptime guarantees these days. Instead, they’re focusing more on ‘graceful degradation’, ‘self-restoration’ and other realities. The Microsoft Research Eclipse project is all about designing distributed/fault-tolerant systems while taking performance realities into consideration.”

Microsoft’s Corner Office

Steve Ballmer Microsoft

The New York Times’ Adam Bryant interviews Microsoft CEO Steve Ballmer about leadership. Ballmer faces constant competitive pressures, new markets, new technologies, ever changing landscape, shareholder expectations and what does he say his biggest challenge is…

Q. What’s the most challenging part of your job?

A. Finding the right balance between optimism and realism. I’m an optimist by nature, and I start from the belief that you can always succeed if you have the right amount of focus combined with the right amount of hard work. So I can get frustrated when progress runs up against issues that should have been anticipated or that simply couldn’t have been foreseen. A realist knows that a certain amount of that is inevitable, but the optimist in me always struggles when progress doesn’t match my expectations.

Balancing upside and downside is one of the core executive issues requiring leadership (more upside) and management (less downside). 

Microsoft Senior Leadership

clip_image001                    clip_image003

Last month, I attended our annual global summit for Microsoft senior management to prepare for the upcoming fiscal year. The time is a chance to reflect on results, challenges and of course the leadership we provide to our respective parts of the company.

The host of the meeting is, like Allan Leighton featured last month, another Walmart executive alumnus, COO Kevin Turner who himself invests a lot of time, energy and thought on the subject of Leadership. At his keynote and later in an internal web-cast symposium he did on the topic, he shared a few choice words many of which focused on the role of adversity…

[Referring to the building of the Windows franchise] “They just refused to fail. Windows took ten years to be profitable.”

[Referring to Sam Walton’s description of looking for the downsides to address] “Divine discontent. No matter how well we did it yesterday, we can do it better today.”

[Referring to the Microsoft culture and values] “One of our corporate values is embracing self-criticism without getting de-motivated.”

[Referring to professional development] “Improvement always requires some degree of failure. Tough times don’t make you who you are, tough times show you who you are”

Kevin talked about building on one’s strengths versus fixing weaknesses. It might seem that not focusing fixing weakness would be out of step with ‘embracing failure’, but actually it is the other way around. Fixing a weakness is rejecting that shortcoming and investing sometimes disproportionate resources to overcome it. Embracing failure is accepting it and moving on. Of course, there are limits and contexts to the application of all of these tenets. Glaring or debilitating weaknesses certainly need attention. But, many times one can manage around the weakness typically through partnership. It’s really a variation of the management adage to ‘focus on core business’ (hopefully a strength).

Kevin does demonstrate characteristics of both the ‘Leader’ and ‘Manager’ persona as this blog defines it around upside and downside. His COO role is central to meeting the business commitments and ensuring the smooth operation of the enterprise (ie. a manager averting downside). But, when we talks about leadership, he focuses very keenly on the upside especially around people. His first principle of his leadership talk was about bringing “people from where they are to where they want to be.” He talked about a question he was asked by a manager and now he asks all of his reports when he first met them, “What are your dreams?”

Microsoft Global Exchange 2008

People Ready MGX FY08

Microsoft kicks off its fiscal year with a major internal conference for field staff each July where the big execs (Ballmer, Ozzie, Liddell, Raikes, and for his farewell tour appearance – Billg) outline their strategies and vision for the company to a crowd of 10,000 numbers-crazed and demo-overdosed sales and marketing folks (check out my colleague Georgina Mitcham’s TechNet blog for an overview of highlights). In the various rousing a keynotes a few comments emerged echoing some of the leadership themes of this blog.

Senior VP of Human Resources Lisa Brummel echoed the ‘great leaders are great energizers’ entry talking about Microsoft inimitable and indomitable energy and how “our job is to keep that energy going

Chief Operating Officer Kevin Turner, really the ‘host’ and force behind the event these days had a great comment articulating the Leadership/Management balance needed in great companies: “We need to have one eye on the horizon and one eye on today.” A bit of a mixed metaphor, but it still works.


Best Management Advice ever received by UK Microsoft Execs

Today I participated in a Management Excellence forum at Microsoft which covered all sorts of discussion about Management (and its link to ‘Leadership’) which in itself prompted several thoughts for upcoming entries on that part of my blog.  But one of the interesting parts were a number of perspectives on ‘Rising Each Time You Fall.’  The perspectives underscore a theme of this ‘Turning Adversity to Advantage’ blog that Microsoft is a company that very much ‘gets’ the concept of embracing failure as a potentially positive force in business.

The UK Board of Directors were asked in a roundtable Q&A, “What is the best piece of management advice you ever received?”  Several of the responses were fine articulations of rising each time you fall:

“A man who never made a mistake, never made anything at all.” – Chris Parker, Head of Law and Corporate Affairs

“You’re going to screw up, so get used to it.” – Terry Smith, Head of Public Sector

Attribute success to others and failure to yourself.” – Matt Bishop, Head of Developer and Platform Evangelism (who in turn attributed the advice to outgoing UK Managing Director Alistair Baker.)

You need to be a participant in your own rescue.” – Gordon Frazer, UK Managing Director

Decade of Failure

Bruce Lynn - Failure service NSUU
[“Embracing Failure” service at Northshore Unitarian Universalist Church 27 October 2002]

  • My philosophy is that it doesn’t pay to go to a conference unless you’re prepared to be vulnerable and meet people, and it doesn’t pay to go to a Q&A session unless you’re willing to sit in the front row. Reading blogs is great, writing one is even better.” – Seth Godin

Ten years ago this weekend I started blogging. At Microsoft, I had just hired a dynamic new marketer, Allister Frost, who was (and still is) years ahead of his time. He identified this new thing of ‘blogging’ as a great way to circumvent the onerous delays and rigid constraints on posting material to the corporate website and instead have a direct conversational connection with customers. It sounded intriguing and with my background in writing (eg. founder and editor of the school page in the local paper, year working as travel writer in Togo, West Africa), he was pushing on an open door for me to give it a go.

I first tried a few experimental posts on the hot (well, in technical circles) topic of “Interoperability”, but I soon twigged that to have an authentic voice I needed to choose a subject I had more personal conviction and curiosity about. I chose to write about “risk”. In particular, two dimensions – 1. “Leadership and Management” (leaders optimise upside opportunity, manager minimise downside risk), and 2. Embracing Failure (a quite popular topic these days, but much less so when I first delved into it). A few years later I added another blog on “Dynamic Work” (flexible work concepts) which was becoming an area of expertise for me and an area I did consulting in when I left Microsoft in 2009.

It was also in that year that I launched what was to be my biggest blog – Maldives Complete. It became so packed with great material that one of the most frequently asked questions I received was, “Why do I do it?” And of course, I answered the question with the blog. In fact, a series of blog posts which highlight a number of the motivations and benefits I get from the curious pastime.

One thing is for sure…there is no shortage of material on Leadership/Management and Embracing Failure. I have posted the equivalent of over 1000 pages (typed pages not web pages) so far. But I am always clipping and collecting material not to mention the countless pieces and bits I get sent by many friends and readers. As it happens, I have another 200+ pages of notes and drafts filed away for future posts at the apropos time.

So stay tuned for another ten years and more…

A Great Day to Make a Mistake


  • Science, my lad, has been built upon many errors; but they are errors which it was good to fall into, for they led to the truth.” – Jules Verne

Today is World Science Day for Peace and Development which seems like a particular great day to make a mistake. The kind Jennifer Gresham talks about in her article “Eureka! A Cure for Perfectionism” (thanks Katie)…

  • “It is estimated that 30-50% of all scientific discoveries are to some degree accidental. I worked as a scientist with some of the best scientists in the world for over 15 years, and I’d say that number is likely a conservative one. Here’s how it happens. You go into the lab, or one of your grad students goes into the lab to do an experiment you’ve done dozens of times. You make a mistake in the procedure, and all of a sudden the results are entirely unexpected. You can’t explain them with what you know today. So you dig a little deeper and then ‘Eureka!’. That’s how discovery often happens. These moments in science are hardly ever planned or anticipated. They start with doing things a little bit differently. In fact, they often start with a mistake. And I can’t help but wonder, why don’t we do the same thing with our own lives. Why don’t we approach life with more curiosity and tolerance for the unexpected?…Why? It’s a completely uninteresting life, a life devoid of surprise, and yet it’s one we increasingly choose for ourselves.

She relates the experience of parenting her own daughter and chastising her for her mistakes (like seeing if her duckie could swim in circles by trying to flush it down the toilet). So she instituted a morning ritual of saying to each other “It’s a great day to make a mistake!”

Have a great day…to make a mistake.

What’s New versus What Works

Gapingvoid - Innovation and fear


Yesterday was a high holiday on the technophile calendar with the latest Apple launch event.  New icons for the altar of the neophiles.

So many articles focus on “innovation” and “embracing failures” is often a theme in those examinations. But innovation for innovation’s sake is not really the objective. What we are really seeking are “positive outcomes” (on a micro, tactical level), and “progress” (on a macro, strategic level). Innovation is just a tool for that progress.

Curiously enough (since he normally weighs in heavily for Leadership over Management), Seth Godin makes a compelling appeal for the importance of Management (ie. averting downside by respecting “what works”) in the face of excessive pandering to new and shiny in his post Neophilia as a form of hiding :

  • “Every once in a while someone will say to me, ‘yeah, sure, I’ve heard that before… what do you have that’s new?’  In contemporary art or movies, it makes perfect sense to be focused on the bleeding edge, on the new idea that’s never been previously contemplated.  But when we’re discussing our goals, our passion and the way we interact with the culture, it seems to me that what works is significantly more important than what’s new. Racing to build your organization around the latest social network tool or graphics-rendering technology permits you to spend a lot of time learning the new system and skiing in the fresh powder of the unproven, but it might just distract you from the difficult work of telling the truth, looking people in the eye and making a difference.  ‘I can’t describe the value we deliver, I’m too busy integrating this new technology into my workflow!’  All too often, the ones who are aggressively seeking the theory of the day don’t have a lot to show for what they did yesterday.”

I definitely confronted this syndrome at Microsoft where senior executives were constantly wanting to hear about everyone’s colourful rain dance rather than the boring mechanics of the success achieved.

Leaders seek what’s new, Managers seek what works. Both together achieve progress.

Backup, Backup, Backup

Hard drive failure


If there was one word which captures the actionable advice behind the principle of embracing failure it would be…”backup”.  Not embracing failure means assuming that everything will go more or less to plan.  Embracing failure will be assuming and building into your plan – whether it be your business plan or your life plan – that things will go wrong.  Then you need a backup. A “Plan B”. If you have embraced failure with backup, then most failures in life will be minor bumps. Unfortunately, those bumps get a bit harsher when the backups themselves fail. Getting a flat tire is annoying. Getting a flat tire and finding that the spare in your boot is flat…is a big problem.

This tenet has been hammered home to me this past month with the failure that every digital citizen dreads – the hard disk failure.  In this day and age, computers have gotten so much more reliable; one could be seduced into thinking that they never completely fall apart.  The last hard-drive failure I had was in the 90s (though a reminder of hard-drive mortality hit me last year when an old machine I kept for testing had its drive fail). I do have a bit of a complicated computer environment.  I run a “virtual computer” on Windows (7.0) inside my MacBook Pro computer which doubles the operating environments each of which have their own backup protocols and tools.

Fortunately, practicing what I preach here, I was well backed up and now more or less back to normal.  I’m not sharing the story in some sort of sanctimonious ‘I told you so’, but rather as a wakeup call for just how down the downside can be of problems.  It turns out that not only did my hard disk fail, but a whole series of failures took place one after the other.  It was only the rigour of my backup practices (and a bit of extra money for some recovery assistance) that saved me. 

The litany of failures were…

  • Hard Disk Fail – My internal hard drive failed on my computer when I dropped it (okay, duh, but sh*t happens).
  • Disk Recovery Fail – I took the failed drive to a disk recovery specialist and the data on the drive was unrecoverable even with their clever clean room tools and tricks (these guys can sometimes do wonders…but not every time).
  • Mac Remote Drive Fail – The remote drive I used for Apple’s Time Machine back up program stopped working (fortunately just after I used it to restore my MacBook Pro…close call!)
  • Mac Backup Software Fail.  Apple’s backup program, Time Machine, actually worked impressively well and fast to restore my Mac environment to the clean replacement drive I bought…BUT it turns out that it doesn’t backup virtual machines (ie. my Windows environment) so that major part of my data was completely omitted. Fortunately, I had manually backed up the VM, but the backup was a bit older than my Time Machine one that had recently run.
  • Windows Remote Drive Fail – When I went to restore my Windows environment (under Parallels virtualization software), my remote drive chose to pack up and fail as well! As it turned out, it was just and chassis failure and the disk mechanism itself was fine. A superb service outfit in Las Vegas (Century 23) swapped the drive into a new chassis and it was fine.
  • Windows Backup Software Fail – After everything had been sorted out and restored recently, I resumed my backup of the Windows environment only to get a number of reported problems. The free backup process in Windows is convenient and value-priced, but it is not the greatest backup program (as I learned troubleshooting my way through various problems it threw at me).
  • Lori Backup Drive Fail – I used the occasion of all of the backup work to do a long overdue backup for my wife Lori’s computer. Lo and behold, I plugged in the Maxtor USB drive…and it too had failed.

If backups were spare tires, then my experience was the equivalent of having 7 different spares in the boot/trunk all flat!

In the final tally, I restored my machine and paid the deep price of the effort doing do, the stress of not really knowing what would and wouldn’t be restored, a productivity hit for a few days and about 4 hours of work lost from the time of my most recent backup before the crash. Not a disaster.

But what is more intriguing for here are the upsides to the whole mess. The incident and the aftermath actually and literally shook up my whole computing set up.  In the aftermath, I perversely feel that my computer is working better than ever and better than it would have been had I not suffered the blow.  I think that the productivity gains have quickly compensated for the hours lost in the lost work and recovery process.

Learnings – The first dividends of any failure are the lessons learned.

  • Hard drives are more likely to fail (in more ways) than you might think.
  • Apple Time Machine does not back up virtual machines.

Improving Backup – This bonus area is a combination of Learning and a more thorough embrace of failure. I thought that I was pretty well set on all of my backup protocols, but the “this is not a drill” crisis and the knock-on failures that followed (see above) pointed the way to a number of enhancements to my backing up now…

  • New Backup Devices – I’ve invested in two new Western Digital terabyte drives (one for the Windows environment and one for the Mac environment).
  • Wireless Backup – Part of my loss (4 hours of work lost) was from the frequency of my backing up. I thought that a weekly back up would be adequate. And it is for averting catastrophe. But not for averting the loss of hours of work you have done that week. I do more than 4 hours of work a week, but much of it is captured online. Since the incident I thought of doing of nightly backup setting it off before bed each night, but that will be easy to get slack on. The current state-of-the-art is wireless ongoing backup. A number of products on the market connect to your wireless router and their backup programs continually back up your work over the air (so you don’t have to think about instigating it).
  • VM Backup – Now that I understand better how Apple Time Machine deals with VMs, I am changing how I approach backing up the VM.
  • Organising Active from Archive Files – One of the things getting in the way of regular backups is the sheer volume. A good backup program should be able to do incremental backups on only the stuff you’ve changed, but in practice it needs to at least look at every file. And after 30 years of computing, I have a lot of files. And some programs (eg. DeepZoom Composer) produce not just hundreds or thousands, but tens of thousands as artefacts. Going through this pile slows the whole process down. The answer is to segregate your “Archive” material (files that are not like to change like archived email, photos, videos, old documents) and “Active” material (your current documents). When I did this, I found that only 4 GB of my 90 GB were in any way “active”. Once I segregated them, now even a brute force backup of my 4 GB active files just takes minutes (and I could put all of them on an SD card to have access to on any machine) compared to hours before.

Cleaning House – The new hard drive in my MacBook Pro meant I could set it up properly from scratch. I always remember in the early days of the PC that when there was a Windows upgrade, it was often a good idea (given the awkward complexities of the platform at the time) to simply back up all of the data and format the hard drive for a completely fresh start. You would have to go through the rigmarole of re-installing all your apps and restoring all your data, but it was a worthwhile investment to get everything running the most smoothly. Now upgrade processes are so refined that this approach rarely makes sense. But that situation does mean that people now go years with lots of mess slowly accruing on their machines that becomes hard to decipher and untangle.

  • Reconfigured New Environment – When I first got my MacBook Pro 3 years ago, I was a relative novice to the new environment (after my 15 year career at Microsoft). As such, I made a number of missteps along the way of setting up and building my environment over time (eg. I set up a partition I didn’t need, I got a clutter of Identities, I set up two user profiles when I could have done one). I learned the error of my ways, but in a number of cases it was easier to kludge a work-around than roll up my sleeves for a comprehensive fix.
  • Cleaned up MacBook Pro – The Genius Bar of the Apple Stores is about the best conceived and best delivered service concept that there is. With my machine checked in to install the new drive, they also took the opportunity to replace a missing screw and pad and clean out the dust that had accumulated inside with a specialist vacuum. I even go my cracked iPhone back fixed for a modest fee (didn’t know you could do that at the Apple store).
  • Enhanced Backup Setup – I set up an entirely new Windows backup using a Western Digital MyCloud wifi system. I also changed my Mac Backup to a new Western Digital My Passport USB drive for Mac.

The moral of the story: Embrace the failure of computer hard drives. A cautionary tale on the too often taken for granted world of backing up your computer. Belt, braces and duct tape. Backup, backup, backup.  And when the catastrophic failures do happen, you can look forward to the stronger situation you will get to in the aftermath.


Get every new post delivered to your Inbox.

Join 150 other followers