Sunarcher

Teamwork is hard; it demands perspective, talent, and commitment

2011-06-17T14:58:00.000-07:00

Another elusive challenge in managing software teams is that outstanding economic value derives from the effectiveness with which software developers themselves work together. For the manager, this requires managing the collection of direct reports such that the whole actually is greater than the sum of its parts. Teamwork brings superior performance and a bigger picture.

Collaboration is more often said than done. Teamwork requires teams, which are often more apparent than real. At its heart, this means the manager must manage the collaboration around the work -- who is doing which tasks, but not directly manage the people. This is especially elusive for inexperienced managers whose minds are more on “being boss” instead of “getting things done”. No good can come from such a situation.

Effective software teams are uncommon; high-performance ones rarer still. When it comes to exemplary teamwork, the fundamental problem for developers isn’t that they can’t or that they won’t; instead, it’s that they haven’t learned how. And more often than not, they’re managed in a manner that precludes effective teamwork.

To perform as a team, developers require common purpose and must act in concert with teammates, adapting to a singular and interdependent focus. Teamwork requires that their actions are coordinated in time, that they are attentive- and responsive- to the work of their teammates, that workloads amongst them are fairly balanced, and that their actions have purpose. Teamwork requires that people’s minds are on the task at hand and that they’re not distracted by worries or gambits.

Excellence requires attention, focus, and discipline. Working together requires clear purpose, accurate and timely communication, and intended, coordinated activity. An effective mechanism to bring a group of individuals into performing together as an outstanding team is via a choreography of occasional meetings and regular activities. Outstanding team work comes from a focus on the temporal matters of “what to do now” coupled with performance matters of “how to do this thing.”

Operationally, this means implementing a small set of activities and a smaller number of meetings, executed like clockwork in a most deliberate manner. It means that priorities of action always the drive workflow, that the idea of “sustainable system” is always at the forefront of activity, and that clear historical records of “who” did “what” and “when” -- especially project or product managers -- is kept current for all to see. Project and product managers are renown for their too-prescriptive approaches or their too short-sighted shifts of focus. Outstanding team performance requires that all these problems be tallied and regularly revealed to everyone who is watching. Visibility is the natural enemy of mischievous shenanigans. As US Supreme Court Justice Brandeis wrote: “Sunlight is the best disinfectant.”

As a cautionary tale, these approaches may be unfamiliar. They shape attention, and can be politically disadvantageous in the short term for those managers whose ambition supersedes their duty. In some situations, the organization itself operates to thwart all teamwork, irrespective of company propaganda. Teams deliver superior economic advantages. Team effectiveness requires astute and attentive management. It is a milieu where “we” trumps “me” and rewards openly endorse such. So. Now what?

Stay tuned; soon we’ll discuss backlogs, retrospectives, and cycles -- the elements of the choreography that yield a sustainable pace for today, tomorrow, and the days forthcoming.

Speaking truth: why “when we’ll be Done” is unknowable

2011-05-27T11:20:00.000-07:00

Another challenge in managing software teams to outstanding performance derives from the tawdry custom of decreeing the future date a software system will be “done” (which usually means a compete feature set delivered and operational). If only wishing could make it so. Prima facie: every target date is subject to myriad uncertainties, changing demands, and rapidly shifting forces. Each day does unfold a bit differently. Every future date is a statement of hope, not fact -- especially not in the realm of new product development.

In these situations the manager decides whether to speak in terms of facts or in terms of hopes. A guess must be seen for what it is -- a conjecture, a speculation, a bunch of B.S. In the milieu of some workplaces, dates (i.e., “when” speak) drowns out other topics. At times, “date-debate” becomes a cesspool of political stratagems. In truth, only the roughest and broadest date range can be guessed. Expecting guarantee of a firm date for a particularly scoped set of features is silly to be sure. It is economically damaging to be certain, especially as dependencies grow around those expectations. Efforts that will likely become wasted are initiated, all based on hope. You can be smart about this, but that takes backbone.

Big plans, long-term visions, and the next big thing, are much more hope than fact: dreams and schemes are more likely to remain wishes than to become a reality. What else could new product development be other than … uhm … new. New product development is exactly that: something that has never been done before. Expect surprises. Otherwise don’t expect anything new. It’s one or the other: new or old. The future isn’t predictable in any manner that matters for most software services/products in a place where projects shift directions, personnel change, the market shifts, new technologies appear while others become obsolete faster than planners anticipated.

It’s fascinating to see how often product managers want to be able to iterate on their designs, but don’t comprehend that in order to accommodate that, then developers likewise need time to adapt the code by iterating and refactoring. Product managers often attempt to fix a date on developers but not on themselves. Instead, the product manager should prioritize either a date or a feature set over the other -- it’s not possible for them to be equally important. Which takes priority? If it’s the date, then cut to fit. If it’s the scope, then keep everyone apprised of progress so they know when to leap into action.

It’s somewhat comical to watch how marketing and PR will be quick to talk about the constraints of their lead times -- which are typically small and predictable. Yet, they often want engineering to specify a fixed date far in the future. They’ve got it backwards. If marketing needs, say 6 weeks lead for printing and distribution, engineering can easily let them know 7 weeks in advance. What can’t be determined (as fact) is what things will look like 5-6 months from now.

Having an important business need doesn’t translate into software development becoming predictable on a scheduled timeline. Authority and command cannot make it so. All they can do is pressure someone to lie. To Speak The Future Date (in new product development) is to knowingly speak falsely. That’s the genesis of many problems, much stress, low morale, and employee turnover.

Others then come to depend upon that date as if it were real, expecting the complete feature set as a given (never mind it probably hasn’t yet been really defined, and PMs will want to tweak it as time goes on). Worse yet, personnel then face having their performance judged on whether they made the fiction come true -- which they can (once), of course, but at the cost of incurring substantial technical debt, delivering bug-infested and undocumented code, and thus sending huge costs downstream that vitiate an enterprise’s capacity for future innovation due to induced burdens such additional maintenance, support, and forced deprecation.

Spurred on by the quick half-life of software projects, of product shifts, and of staff changes, the codebase quickly becomes calcified and brittle. Acting only one step ahead leads to a “hackolution” instead of an “evolution” of the codebase. Continual piecemeal hacks optimize for the here and now, yet flexibility tomorrow means preparing for it today. In part, that means that recognizing that its less expensive to build in monitoring and testing today than it is to waste much time debugging and fixing in the future.

By nature, short-term actions prevail to the degree they’re rewarded. The short term can be good for the software author, for the product manager, for the marketing manager, but not as good for the company itself. For the longer horizons of the company, the manager is obliged to remain mindful of the economics of the codebase, thereby keeping a mental tally of its value, burden, and opportunity costs in light of development activities underway.

Balancing the short and long term is delicate and unique in every situation. In each case, the business needs, the available personnel, the project resources are different. Nonetheless, good engineering practices today makes for a better system tomorrow. Developers can easily become “too clever by half” to speed up code development -- something that can save half a day today may well cost two or three days to find and fix down the road in six to nine months or so. An unreasonable demand for a feature by a date can induce this behavior easily. Often, managers are the ones who actually create the downstream problems by lacking discipline at this point.

Haste today often precludes innovation tomorrow. Promising silly dates for feature-replete products yields premature dependence on fragile infrastructure. Burdensome downstream expenses accrue. Although an early release may be opportune for marketing purposes, it may also become crippling burdensome to the company over time. For a company that wants to last, or just not become a one-trick pony, prudence cautions managers to apprise every project’s performance in terms of both value today and opportunity for tomorrow.[1] Focus is served by ensuring everyone knows what matters most. A conscious choice is the responsible thing to do.

A real pernicious circumstance occurs from bandying about hopeful dates without recognizing actual capacity or necessary features/scope that can likely be ready by that date. The cost endured here is more subtle and more damaging than those mentioned above. Discussions of “dates” changes the milieu of the workplace -- thoughts shift from “what we are working on”, to who’s talking to “me” and when “I” am expected to be done with something that probably hasn’t yet been invented or defined. The predominant outcome from “date posturing” is to distract developers from their work. As a result, people become unsettled, and each developer’s thoughts naturally become drawn to conjuring up worrisome fictions, and fretting over the future. Like an evil Gresham’s Law of conversation, the milieu spawned from “when” chatter is combustible and predisposed to political shenanigans. Team performance becomes suppressed. Every person, for survival, optimizes for the self.

Commitments to future dates ask people to claim and then stand behind statements that are known to be speculative guesses. These are circumstances with capricious career impacts, either good or bad. No productive value is afforded by this focus, by this claim on attention. Whereas a plan of record is sensible, a date of future record is folly. How else could it be? Such talk of dates is the slippery slope that rapidly transforms a collaborative workplace milieu into one best described as “me first”. Heroes are bred here. Kiss teamwork goodbye. Recurring discussions of “when” foster rampant worry unless the discussion are accompanied by notions of “cut to fit”. That is, to have conversations that reflect pragmatism regarding what can actually be accomplished by that sacred date. In themselves, dates are fine -- it’s the unspoken baggage of feature completeness that is the hidden toxin.

Coding is an exacting, by-hand, attentive practice akin to the work of the finest artists and craft masters. Even tiny systems are built in pieces by different people, often with myriad small and interdependent parts. The economic value of that software is shaped by how well this blizzard of components and their relationships work together, and how costly it will be maintain or rewrite. Of the myriad uncertainties in software development, the human element clearly predominates. Such is ignored only by folly or snoozing or conniving.

But there a practical solution to this: pick which is more important -- the date or the feature set, then plan accordingly. It can’t be both.

US President Teddy Roosevelt offered plain-speaking guidance: “Do what you can, where you are, with what you’ve got.” What you as the manager can do is to speak in present tenses and keep everyone’s focus on “what” needs to be done. Shield the developers from the allied roles or executives clamoring for “when.” Stay alert enough to bring witness to shenanigans. Have the backbone to expose them to sunlight. You’ll get more done, and the developers will be a happier lot.

_______________________________
[1] From the vantage point of this author, “quickies” are necessary for startups and early stage companies, but notable deleterious to those orgs with greater resources, longer horizons, and larger workforces. Sadly, it seems at least, so many of this “next stage” orgs act according to the circumstances they used to be in instead of where they are and have evolved to.

Engaging with developers as the communicative beings they are

2011-05-19T12:51:00.000-07:00

Myriad challenges are encountered in leading software authoring teams to outstanding performance. After recruiting team players, come the difficulties of communication. And along with that, overcoming the way many authors have previously been managed.

Another challenge arises organically from the everyday, commonplace, without a second thought, mechanistic way that many software projects, and hence developers, are and have been managed. Mechanistic management still blossoms today replete with success for over a century in the industrialized factory. Yet such approaches are often an albatross on team performance. Today’s world is different from yesterday’s world. The mindset and methods of traditional management are revealed as irrelevant to achieving high performance of most software teams today. A mechanistic approach is as counterproductive as not managing the code authors at all, if not more so.

With software development -- that act of code authoring -- humans create the product, attended to by machines, whereas in the industrial age, machines made the product and were tended to by humans. That old world has now been turned upside down. The management instincts, common sense, and street wisdom that has been passed down from that work in those days are no longer applicable. Now, the environment of the minds, the factors that shape their attention, the conditions beneficial for high productivity are new and matter much.

Project managers, for example, all too often believe they can decompose a software development project into known, appropriate discrete steps, as if wishing would make it so. They then assign “resources” to them as if everybody’s skills are interchangeable and ramp-up time is non-existent. Such problems are further exacerbated by product managers who are oblivious to the debilitating costs of interruptions or directional shifts or late minute “just one small change”. What is being revealed here in each case is the mindset of someone whose expectations are based on how machines operate, not how people behave. Wishes for stability aside, numerous unpredictable unknowns await: staff may change, crises may distract, lawyercats may say no, dependent systems may go awry, a market urgency may arise, mandates may erupt from on-high, who knows what. It’s a good idea to not get overly tied to a plan for a future that most certainly isn’t going to arrive, at least not the realm of new product development.

Software teams and software projects are each complex systems. They’re variable; they adapt; they yield emergent phenomena, they go awry. That’s what complex systems do. Everything keeps changing; little of it predictable. Software is authored by unpredictable individuals. Developers, as thinking beings, pursue self-interest. They are embedded in an unpredictable workplace. Their employer sells into an unpredictable market. They’re surrounded by unpredictable coworkers and yet less predictable managers.

Amongst the group are authors at various life stages, with distinct concerns, personal ambitions, unique personalities, and particular interests. Some workplace milieus are maelstroms. As code quickly calcifies into legacy, expediency induces major downstream costs. Much code becomes no more than a collection of half-used/unused modules. Docs decay.

The intrinsic value of all the collective human effort becomes manifest in the software, the code. That software is something that is created by temperamental people, tossed into situations where they’re expected to work well with others, where everyone exists in a swirl of partial attention and regular distraction.

A blinding flash of the obvious is that the authored code comes from people’s minds; hence, it matters what the state of those minds are. The fingers just transcribe. The state of what is going on in the author’s head impacts the quality of the code and its rate of creation. Code authors, have -- yes -- moods and feelings, plus myriad reactions to work conditions, coworkers, and tasks. The adept manager wants those minds free of worry, free of apprehension and uncertainty, and instead to be pragmatically optimistic, and engaged with the tasks at hand.

Such a workplace milieu is not a “well-oiled machine” with its predictable rate of production, smooth progress, and trustworthy delivery dates. It simply can’t be; machines are not the factors of production for software. Humans author the code. Instead, it’s a crowd scene that challenges the manager to bring everyone together to get things done; things that are often ill-defined, poorly understood, and politically charged, each of which can raise and then compound a developer’s sense of nervousness, apprehension, frustration, or anger. Developers do quit jobs over architectural disputes. Developers also go where they’ll be well-treated. The workplace milieu matters.

It becomes the task of the manager to organize the work and guide -- actually choreograph or conduct -- these people towards a common goal. Inducing common purpose amongst this crowd is the key to getting things done, despite that it’s a situation where every individual is unpredictable and is acting in self-interest, who is moody, and not necessarily socially adept, Developers are not programmable robots; wishing won’t make it so. They need leadership; they rarely require direction. There is no machine here, only interpersonal chaos is present, yet teams can fly above this maelstrom.

The actions of this crowd are tantamount to a “jam session”. The energy source of a crowd is their communication. Workplace conversations shape people’s attention, thus influencing whether their thoughts are on their work, or more so on matters of career concern or personal ambition or bewilderment at what are perceived as blunders at the executive level (e.g., a poorly handled re-org, buying a company with soon-to-be obsolete technology, etc.).

Outstanding team performance only happens when every mind is engaged in the common goal. Personal thoughts that dwell on worry or become apprehensive of career risks draw precious attention away from the value-creating activities of authoring code. The optimum code creation environment is the relaxed mind.

Whether this crowded conversation becomes the cacophony of a mob or becomes more like the harmony of a choir is shaped by the manager’s handling of the workplace communication environment (its milieu). The attentive manager ensures that vital topics receive sufficient “air time”, that actual decisions are made and labeled as such, and that everyone is kept up to date.

Although companies spend lavishly on tutoring employees about supposedly how to communicate, much rarer still is any guidance on what to communicate about. The manager, to foster outstanding team performance, continually communicates in terms of “what” more so than “when” or “who”, thus guiding the team to aim it thoughts to the shared mission and “what” needs to become “done”.

A company milieu that dwells on “when” (i.e., mechanistically wishing for a predictable “done” date) distracts people from discussing the tactics and strategy most important to actually get things “done”. When topics are commonly about “when” more so than “what”, personal concerns detract from the team goal. Discussions about “when” inject into the body politic a strong sense of risk. (Bad idea.). Developers know any such date would be a fiction; after all, new product development is … uhm … new. Horizons may be clear but paths are unknown, hazards await. Developers wonder why others don’t see the obvious as well. They smell gamesmanship. Personal experience and observation provides evidence for such intuition.

Similarly, a conversational environment that is predominated by discussions of “who”, where communication spotlights “stars”, postures people into direct competition with peers. Where a workplace milieu dwells on celebrity, an environment of favor where power rests among a small clique's tribal circle of trust is spawned. Team objectives easily then devolve to lip service. Teams never gel. The result is a cast of “stars” and “bit players” -- very Hollywood, but economically debilitating for software development.

Jim Barksdale, Netscape’s CEO through the latter ‘90s, was renown for the mantra “the main thing is to keep the main thing the main thing.”[1] His deftness in the art of plain speaking and charismatic leadership is legendary. He continually presented a unifying voice to all employees encouraging them to focus on their “main thing.” It was up to the manager to ensure the “main thing” was something worthwhile and that it stayed top-of-mind for everyone. This clarity of focus was intrinsic to the business tempo that became known as “Internet time”, and continues to accelerate today. A key aspect of Barksdale’s genius was his deftness at keeping people’s minds focused on the task at hand, to be thinking about their work, to do their job.

Ensure that the “main thing” is phrased in terms of what instead of when or who. Do this by keeping an open ear to conversation, and regularly raise topics of “what” about the work and how it’s going. Redirect conversations about “dates for things” into conversations about “states of things”. Turn conversations about “stars” (or assholes) into conversations about “circumstances”.

The “right agenda” is essential to achieving outstanding team performance. Software authors are people, easily distracted to personal matters. Worrisome concerns draw aside their attention, leaving less cognitive capacity needed by their work. Guide their focus. Give them peace of mind. Make it easy for them get on with their main thing.
______________________
[1] “The main thing is to keep the main thing the main thing” is attributed to Stephen Covey.

Challenges in Fostering High Performance Software Teams

2011-05-13T10:45:00.000-07:00

Great teams, to quote legendary hockey coach Herb Brooks[1], are comprised of the “right” players, not necessarily the “best” players. The essence of “right”, in this case, is the capability for people to perform well with others -- as a team, a unit, or an ensemble.

Acquiring the “right” developers, and then managing them as a team is fraught with challenges. Experience and observation, again and again, have revealed these underlying challenges:

Recruiting a team from amidst a hotbed of celebrities and soloists
Engaging with developers as the communicative humans they are
Speaking truth: saying that “when we’ll be done” is unknowable for new product development
Working as a team is hard; it demands character, talent, and commitment
Raising developer’s individual sense of professionalism and excellence
Inducing a positive milieu exceeds the apperception of many technical managers

The manager must draw the “right” people together; this luxury doesn’t manifest quickly. In many workplaces staffs and work groups are just assigned. So this challenge comes to include mentoring people on how to act -- i.e., perform -- as team players. For the time being, think of people in a choir, symphony or soccer-team, etc. That is, they are performers. This requires guiding developers in ways of working together that foster outstanding performance.

The adept manager needs to have a mental model of how teamwork is performed. That is, the acts of the performers. This is wholly distinct from “what tasks need to be done”. Teamwork is induced by the manager, sustained by the group. The manager needs to maintain a public agenda of teamwork, and then communicate it clearly and frequently. It’s up to the manager to create this expectation.

Walter Wriston, while leading Citicorp, insightfully said about capital: “it goes where it’s wanted and stays where it’s well treated.” The same can be said about talent, and that means software developers. Recruiting the “right” people, and mentoring them to work together in a “right” manner to get things done is at the forefront of the manager’s responsibilities. And people tend to be set in their ways, with their own ambitions and resolve. It is attentive work. It’s hard to do well.

The Primary Challenge: Recruiting a team from amidst a hotbed of celebrities and soloists.

The primary challenge to achieving outstanding team performance is recruiting team players. This means drawing in people who in their essence prefer the rewards of teamwork more than a spotlight on themselves. For these people, team contribution trumps personal ambition. These are people who seek to participate, as the saying goes, in something that is “larger than themselves.” This challenge arises because the software development workplace is often a welcoming haven for wizards of the arcane and celebrities of crises -- two types of heroes who are inimical to teamwork. Of course, this is not surprising as many companies lucratively reward “stars”.

Building a team of the “right” players is where it all starts. Outstanding team performance expects personal excellence ahead of individual cleverness. It means that everybody must adjust their own individual pace in a manner such that the team, and consequently the project at hand, can step up its pulse. This approach blooms with ramifications -- it means not always putting the “best person for the job” on the job, for example, if the work is being managed to broaden the skills of other team members. It might mean that everyone on the team may be expected to use the same tools, instead of personal favorites. It all depends on the circumstances.

Perhaps this “religious war” is unknown away from the software development workplace milieu: emacs vs. vi. Each provides a specialized editor for authoring code. A third one is Eclipse (mainly for Java). Developers’ passions boil over regarding which editor to use. Their preference typically points to the editor most familiar to them personally. For the team’s sake, though, it may be better that only one editor is in use such that anyone can offer quick help to another and be working in a common environment. This is a deeply personal act; team players will be disappointed but not become argumentative. Solo artists will wail and whine endlessly. A manager can learn much about the personnel by raising the topic of “only one editor” at a staff meeting. Pay careful attention to who says what and keep a sharp eye for changes in body language and fidgeting. Make mental note of those who complain the most.

An outstanding term achieves success via well-rehearsed, collective finesse, not from the individual actions of a cast of stars. With a team, even when stars do fall or are just away for a day, success remains at hand. In teams, people work together towards a common goal. Here, they happily fly in formation. Finding people who are like this, the ones who excel personally and work well with others, is the fundamental challenge to achieving outstanding team performance. Recruit the right players, not the best players. Coach people to perform together, as a unit, as an ensemble

Playing well together trumps all; progress never stalls. Returns continue to accrue.

________________

[1] As immortalized in the classic show Miracle on Ice chronicling the 1980 USA Men’s Olympic Hockey team and their spectacular win, Herb Brooks, the coach, when putting the team together said: “I’m not looking for the best players; I’m looking for the right players." The rest, as they say, is history.

Teams Trump Groups When Creating Software

2011-05-06T17:23:00.000-07:00

Managing software development is my vocation. It has been for nearly three decades. I’ve been asked by various peers to write down some of my observations, theories, opinions, techniques, and suggestions. Thus begins various comments regarding managing software teams.

Software is written by people. It all starts there. Some people are better than others at working together. Some just don’t know how. It all ends there. In a nutshell, what I’ve found is this: manage the work, and let the people manage themselves. Ensure they do so as a team. Why?

Without software, contemporary civilization could not exist. Billions and billions of lines of computer code are necessary to keep our lives in working order -- code that is expensive and difficult to author, and then yet more challenging to maintain. Defects are common, sometimes glaring. Although software automates much of the world, and underlies all contemporary communications and financial transactions, the act of creating code is in itself a painstaking and individual activity. Building any software system, irrespective of whether for services, infrastructure, tools or products, has been repeatedly shown as frightfully expensive[1] and a high-risk undertaking in terms of both time and money. Economic returns are highly variable. Disappointment abounds. Nonetheless, this tiger can be tamed.

Creating and maintaining software is challenging and difficult work -- far more difficult than anyone who doesn’t do it can realistically imagine. The act of authoring software requires razor-sharp technical acumen, extensively refined knowledge, proficiency with algorithms and languages, adeptness with tools, and all while requiring an excruciating attention to detail as a clock is ticking and people are pushing. Nonetheless, the mental aspects of coding can be intoxicating, highly satisfying for those with an intellectual bent and those who truly enjoy the life of the mind.

Mastery of the art when it comes to coding does demand an attentive physical activity of dexterous keyboarding, coupled with creative mental focus, usually amidst noisy and distracting surroundings. For the developer, coding can be a combination of wonderful mind flights and wizardry skill. It’s what produces “that nifty code” that “does that shiny thing”. The work of the best of them is brilliantly beautiful. The act of authoring software can be a great solo experience. And the pay is really good.

Yet, in the interests of the employer, superior economic value, in a sustainable manner, cannot be based on the actions of individuals, of solo performers. Instead, enduring economic returns are founded on the collective actions of several or more people, preferable acting as a team where all members pitch-in to ensure that the tasks to be done get done in a timely and effective manner. It takes people who see to the mentoring of each other, and who keep systems running, and continually improve[2]: (1) how they work together; (2) the tools they work with; and (3) with whom they work.

With a team, forward progress on a project never stalls. In contrast, if a deliverable depends solely on the actions of an individual, progress on that task just freezes should that individual be sick tomorrow, go on vacation, or be called out of town. The underlying challenge for the manager is to reward team performance while recognizing individual excellence. It’s harder than one might think at first glance.

Many paths lead to success. The activities suggested in this series of posts are among those that experience and observation have taught me do indeed lead there. Every manager can take simple steps to foster outstanding performance among software developers.

The suggested activities described in this series of posts will be a breath of fresh air for many developers yet an anathema for some others. Why? Because the first requirement for having a software team is to expect, empower, and reward teamwork -- which is much more elusive and difficult than popular management literature reveals. And not everyone in an org wants to be part of a team; especially in celebrity milieus where individual results are rewarded disproportionally over team performance, irrespective of whether the true accomplishment was that of a team. The mindset and methods presented here, however, are team activities for sets of team players. Personal experience suggests that outstanding performance will result from practicing the approaches described in these posts.

Stay tuned!

____________________________________________

1. C.f., Robert Charette, "Why Software Fails", IEEE Spectrum 2005

2. Kaizen

Role of the Manager in Fostering Innovation

2009-11-23T20:28:00.000-08:00

Recently I was invited to address a group associated with a local university. I was asked to speak on what I've learned about the manager's role in fostering innovation. Here are the talking notes I used.

1. Pick what to manage

manage the work, not the people -- focus on the artifact, not the timecard or the appearance of being busy

2. Remain mindful of essential human behavior

circadian rhythms, personalities, learning styles, bursty vs. steady state output, hygiene

3. Keep ideas in motion: foster dialog

crowd people together and layout the space and normative expectations for quiet time too

4. "Don't let yesterday use up too much of today" (Will Rogers)

the prudent trumps the practical, stay aware of technical debt, be careful of what you mortgage, cut losses quickly

5. Decision making is performance art

frame decisions as such, be decisive, be public, be on the record

6. Relentlessly remind everyone that benefits sell, features don't

recognize that innovation != invention; don't mistake activity for progress

Google's #1 of ten things: "focus on the user and all else will follow"

7. The main thing is to keep the main thing the main thing (Covey, Barksdale)

focus!!! order can arise without anyone giving orders; continually be asking questions, in public earshot as well as privately; frame discourse to push at the edges

8. See to every employee's welfare

intellectual work is not factory work, minds are what matter and moods shape mental clarity -- seek to ensure everyone is in harmonious spirits; squelch rudeness; show backbone

9. Mission is value creation not cost containment

see the organization as system of roles as well as a system of persons; be savvy about what metrics mean, ensure they support the mission, not constrain it

10. Leadership

mission cheerleader; herding dog, not guard dog; maintain domain competence; keep eyes sharp for "doing the right thing"; let others have "the last word"; kaizen; stand back

The Magnetic "B-Field" of Leadership

2009-01-24T01:14:00.001-08:00

If you want to build a ship, don't drum up people to collect wood and don't assign them tasks and work, but rather teach them to long for the endless immensity of the sea.
-- Antoine de Saint-Exupery

"Leadership" is vital for organizations to operate effectively. And yes, folks commonly agree that leadership is in short supply or sadly missing in action. Much of what we hear about leadership's definitions, descriptions, and perceptions are gobbledygook at best, consultants' shills' hype at worst. Yet, leadership, to many people, reflects Supreme Court Justice Potter Stewart's famous remark about pornography: "I know it when I see it."

Leadership is that special magic I've seen that enables organizations, and especially those that develop software, to achieve high velocity and stay in their stride. And I've learned that leadership is like the wind -- it's invisible, but its effects are evident. Or like love -- it eludes definition, but is profound in shaping attention and sense of well-being. The closest analogy I've encountered in terms of "I know it is when I see it" regarding leadership is analogous to the B-Field of magnetism we learned in freshman physics. As a reminder, it's a vector field that exerts a force on electrical charges and magnetic dipoles.

The B-Field exerts an evident force; children learn about magnetism by seeing how iron filings become aligned and trace the "lines of force" on a sheet placed over a magnet. When leadership is effective it acts the same as that aligning force. When it's inept, then misguided and spasmodic management becomes the norm.

Leadership is an evident pattern. When this "B-Field" is present, people align their communication and activities; when absent, entropy rises and waste increases. Communication remains the essence: long term or short term? Innovation or speed-to-market? Whatever it is, if it's clear, people rally. Speech and actions become aligned towards such ends. Morale is often positive.

Likewise, when developers have a shared conception of their objective and the purpose of their activity, they tend to pursue it straight as an arrow. In contrast, when priorities shift, ambiguity reins, or false urgency becomes commonplace, the velocity of software development plummets. Things become chaotic; inefficiency becomes widespread. Morale often sucks. Misguided micro-management becomes the order of the day.

Nothing escapes the relentless decay of entropy. Organizations are fragile and feeble to begin with. Leadership is hard: it takes forethought, clear communication, and backbone. But leadership can be simple: it takes commitment, discipline, and empathy. The nature of leadership, especially for software development is -- as Jim Barksdale continually exhorted -- keep the main thing the main thing. It especially helps when people have some clue as to what the main thing is supposed to be. When leadership is evident, the main thing is likewise evident; when leadership is absent, there's no "main thing" (actually there are several, meaning there isn't one).

A strong B-Field, as evidenced from speaking about lean principles and acting in accord with agile practices, can increase velocity to the capacity of the project team. In a straight-forward manner, retrospectives, iterations, and prioritized backlogs, draw focus and choreograph activity in a way that naturally leads to improved velocity. To achieve this, leadership is necessary, and the B-Field may well stand for the "backbone" necessary to require prioritized backlogs, broad test coverage, real penalties for breaking the build, coding discipline (e.g., YAGNI, DRY, etc.), and the like.

There remains a clear pattern: In organizations without leadership, dead starts, re-starts, broken builds, side projects, and disconnected work all become commonplace. In organizations with evident leadership, people's activities align, and velocity increases to its capacity. Great code comes from happy developers. Order arises without anyone giving orders.

Do Less With More (So More Gets Done)

2008-05-16T00:41:00.000-07:00

Looking like things might be late? Did that Big Bad 'premature optimization is the root of all evil' thingy just nip you again? What's a poor soul to do? There is an answer . . . .

Okay, so there's a staff of developers busily being busy, with other folks awaiting new features. Bugs and distractions inhibit progress. Some sales guy snickered at a demo. Expectations were hyped. People are getting hyper. Things are becoming a flurry. When a contract looks like it might run late, a trade show looms, or market opportunities tick loudly, the situation can quickly get prickly. Used to be that everything else was a priority that interrupted this project; now it seems that this project is about to interrupt everyone else. What an exalted state!

The project's status and progress become the focus of attention. More and more conversations are about progress, people, and dates, driving out topics of performance, abstraction, or architecture. Such conversations tend towards the ineffectual, like idle chatter regarding schedules. That won't work, never does, except in an industry that's irrelevant to high-tech. Coercion and threats won't speed things up, most likely the developers are already working at their personal capacity. Plus, it's never a good idea to bully someone, especially in places like Silicon Valley, where any accomplished developer can find a new job in the time it takes to fill out a web form. Finally, or so it seems, Someone In Authority proclaims: "add more people to the project and get it done." Aside from what appears to be a Blinding Flash of the Obvious, adding additional staffing late in a project is renown for pushing releases even later.

Fred Brooks warned us more than thirty years ago about the calamitous burden of adding people late to a project. Nonetheless, many software managers ignore his sage advice. Business "as usual", driven by glacial forces of habit and ritual, is how employee activity typically becomes planned, then communicated. Often, it presumes people are as predictable as machines. We're not. We're not going to be in the foreseeable future. Nor can people just "switch on" as machines do. Too bad people can't download context and operating procedures as quickly as a computer which only needs to boot up, or a lathe that just needs to be switched on. But we can't: this is the workplace, not The Matrix.

Imagine a perfect world, where every company has a ready pool of developers just sitting on the bench just waiting to fill in. Add to that a magical circumstance where any developer can interchangeably step in for another and be up to speed in moments, disrupting nothing else underway. Reality delivers bad news on both fronts: projects are usually understaffed to begin with; and, developers are not fungible. Software developers have highly-honed, hard to acquire skills. It is a mindset of exactitude that few disciplines ever encounter. Developers are not interchangeable; moreover they're hard to replace. And even if a few devs with appropriate skills can be found, they may not have the personality or traits that would enable them to be effective on a software development team.

Software development is so complicated and detailed that even a brief vacation can leave a developer needing a few days to spin-up, after only being away for a week or two. After just a matter of days, important project details and techniques start fading to the background. And, over the intervening days, situations have likely changed in the company or project. Even in the best of circumstances, re-entry is a jarring situation. And hiring? Then there's the entire adaptation to the how the new company operates, the normative forces, to say nothing of the unique build and testing tools at the new employer. Vacations are bad enough; hiring promises large diversions of otherwise productive time for the project.

So, why do projects so often go awry?

Let's back up to the beginning. Somebody got an idea. Perhaps someone "blessed" with an overabundance of fast-twitch fibers got spooked by a competitor's FUD or became blinded by deal lust. Superiors called forth for The Plan. Ritual reigned as the manager scoped the work, wrote a project staffing plan, derived a schedule, and then, in the manner of many companies, wasted umpteen hours preparing a presentation, only to go forth and swear myriad oaths on broad altars promising to deliver a feature-complete, on-time, and within budget result.

Sadly, lurking within, is a noxious notion regarding "delivering on time" -- in other words, "by a date." On time means the work got done, which entailed productive time from developers. If developers are always available and fungible, then a simple quotient of total estimated days divided by count of available developers yields an expected date. By doing this, managers are presuming that the team's capacity for developing software for the project is constant, thus predictable. It is neither. Typically, projects start off understaffed, then people come and go, resource contention follows, progress slows down. People are added and the project slows yet more.

A project can only be "late" in reference to a date. Two evident factors are: (1) the progress of the project; and, (2) the correctness of that date. Conversation often focuses on the progress, but the likely culprit is more often the correctness of the date. Or, more precisely, the expectations, allocations, and, promises bound to that date -- which reflect an assumption of constant development capacity dedicated to the project. Irrespective of the cause, any change in project staffing has an immediate impact on any date, and recovery to a new equilibrium comes slowly, particularly if it requires hiring. Scope is hope; human factors are the reality.

Many managers think in expansive detail about the features to be built, their dependencies and commonalities. But any project's primary dependency is on the capacity and the timing of developer engagement. Effective developer staffing is the fundamental requirement for success for a project or for a company. Whenever a project staffing plan presumes development capacity is constant, and that recovery (if needed) would be quick, jeopardy is evident. Date and staff co-vary, though neither linearly nor directly. To just mindlessly pad the schedule is to miss the fundamental point -- software development is a complex, not linear, system based on the human vagaries of developers.

All staffing changes are disruptive. On a day to day basis, the actual time that developers are task-focused per project rises or falls as everyday events shape everyone's attendance and attention. Capacity is likewise reduced by other ever-present workplace duties, as well as other projects vying for attention. The development capacity available to all projects likewise rises and falls. This becomes especially problematic when key features or refactorings are underway by individuals.

If a feature only has one person working on it, it follows that when the person isn't working on it, then no progress is occurring. For the single-person feature, it is either getting worked on or it isn't. This isn't a slowdown, it's a stall. If a task is important, it's wise to staff it such that progress doesn't stall. Every task underway by an individual has a high risk of stalling. Thus, never have a critical task underway by a individual. Ensure that at least two developers are committed to the success of every key task.

Did the project actually go awry?

Against what measure does progress appear thwarted? Did expectations reflect a likely outcome or instead a hopeful outcome -- best guess or heart's desire? Developer time is not available at a constant rate. Simple summed estimates of "developer days" (regardless of padding) only suggest at best how long the tasks will take -- the sum of their time, but not when they will be done due to the influence of human factors. People get interrupted, become distracted, sometimes fall ill, or go on vacation. Sometimes someone new joins up. Developer capacity varies. Project allocations vary. Straight-line time is a fantasy. Any date derived in such a way is a heart's desire.

The generative point from which things often go awry is in planning the staffing pattern within the project. A problem arises from the practice of assigning critical chain tasks to individuals -- any individual -- then predicting dates based on constant capacity. Superstars are irrelevant if they're not available. If the show must go on, a backup, a substitute, an understudy, or the like must always be at the ready -- that's the cost of living in a situation where "the show must go on." This concept appears elusive to many software managers and many allied practitioners. Nonetheless, this solution has been in place across diverse professions ranging from the court to the classroom, from the theater to the baseball diamond, from the fire brigade to the emergency room. Where a human is required, at least two must be at the ready.

Software systems are arcane and intricate. They're among the most complex artifacts ever created. Complex systems give rise to emergent phenomena that aren't predictable. Developers participate inventively, today impacts tomorrow, and so on through a chain of circumstances. Full system comprehension is beyond the grasp of an individual, and its myriad parts are a chore to remember, let alone learn. Programmers are individuals, that's true. Yet software is done by groups, or better yet, teams. The whole can be greater than the sum of the parts, depending on how those "parts" are choreographed. When working together, progress continues irrespective of individual factors. If developer activities are constrained to collegial parallel-play with code, then every time any developer becomes unavailable, project progress slows because that developer's task just stalled.

Do less with more . . .

The solution is to ensure that no critical process depends upon any individual. For the manager, this suggests identifying the task, then ensuring at least two developers share the responsibility for its successful completion. Do less with more, thus ensuring continual forward progress on the critical chain. Staff smartly at the outset, being ever mindful of the surprising nature of human factors. People have lives and their futures are unforeseeable. If a task must be done, human vagaries require more than a single individual be committed to its completion. (XP's pair programming takes this in stride.) Individuals don't belong in the critical chain.

Common practice, though, is to assign tasks to individuals. If there's 10 small features and ten developers, often the accounting rules or company mores lead managers to implement a "single feature per single developer" assignment mechanism. Such assignments take on a life of their own as other developers quickly set filters that reduce prominence of anything for which they're not responsible. In our email maelstrom, it's easy for things to vanish. They don't get lost; but they do vanish. Things can be going awry for a while before drawing attention.

As a corollary, "one-per" reveals the belief that monitoring individuals trumps monitoring tasks: it is a mindset framed primarily by accounting for developer's time. Many managers assign a task to a person, invoking a premise that responsibility is best designated to an individual. In contrast, a progress mindset ensures tasks get done by staffing to avoid stalls. What's evident here are contrasting mindsets regarding the first principle management practice: accountability vs. progress.

And it's also true that many developers prefer to work alone for myriad personal and social factors. In any workplace, a plethora of mother-tongue challenges impedes communication, personal chemistry is a known operative force, and all developers have highly idiosyncratic ways of configuring their working environment. It's a natural characteristic of those who take to a career in software development to often wish to stay to themselves. Individuals are programmers, in a group they're software developers, but when they're a team magic can happen -- as many, many developers in Silicon Valley know in their bones. Good progress makes good teams, but this doesn't occur spontaneously. For starters, it means accepting that in effective software development, developers work together.

So, take the test of five things: if you have five features to be coded and five developers at hand, is it one each? On the other hand, perhaps as a team you'll list the features, factor that which is common, and prioritize the rest. Then ask the five developers to take it upon themselves the get first two features done before moving on to others. If someone becomes unavailable, progress continues. And peer review is a magical thing in itself.

The future of every software project is murky. A manager is wise to be ever mindful of capacity and flow, avoiding stalls, and thus avoiding adding people late to a project. It's easy to avoid, but it requires staffing in a more modern, less industrial-age, manner. Some suggestions:

Enable continual forward progress by disallowing individuals in the critical chain
Ensure everyone has someone to talk with so there's always someone else up to speed on the task
As a measure of progress, employ "features released" by staff, instead of accounting for developer hours, per individual. It's a useful proxy for progress, at least to begin with.

Then, tasks on the critical chain don't stall, things get done. Steep learning curves are flattened. Chaos is constrained. Stalls are avoided. Progress continues. The whole can be greater than the sum of the parts. Choreography counts. The buddy system has served us well over the ages.

Recalling Knuth, echoing Hoare: "Premature optimization is the root of all evil." Don't locally optimize for developers' time through accounting, optimize for progress through task completion. Don't be evil.

Interesting reading

The Goal and The Critical Chain by Eliyahu Goldratt, introducing the Theory of Constraints

Retrospectives: Informed, Continuous Team Improvement

2008-03-29T00:48:00.000-07:00

A retrospective, in simplest terms, is a facilitated, focused dialog. It is a time-tested and valuable method for improvement. Retrospectives draw immediate attention to ineffective situations and problematic circumstances. While related to notions like an "after action review" or a "postmortem", a retrospective explicitly examines current circumstances and looks to the future with an eye towards becoming better. Retrospectives are becoming ever more common as teams follow lean principles and agile practices to improve their performance as effective software development organizations.

Learning is key in the pursuit of excellence. Retrospectives reflect the human desire of attaining perfection, yet the recognition that it escapes our reach; hence, only by continually improving can we get closer. While retrospectives are useful in many types of situations, this essay is focused on the utility of retrospectives for improving team performance within on-going projects.

Retrospectives are learning situations. No matter how capable a set of developers may be as individuals, their overall productivity is constrained by the flow of their work. The timing and structure of their activities can either add lift or induce drag on that flow. Add to that the complexity of working together in teams, in a company awash with bureaucracy and politics, and it's a wonder that they can deliver any software at all. Yet we do. But, can we do better? Retrospectives are all about improving team performance and adding lift in that regard.

Retrospectives are situations where key issues are surfaced. They provide an opportunity for a team to take stock of their circumstances, assess the forces at work that influence them, and engage in focused brainstorming as to what to do next. It's a time to step back and look at the work and the flow. Typically, a retrospective identifies what's going well and what could be going better. With the overall situation out on the table, the team can identify one thing (one is enough) to improve that could make life a bit easier.

A retrospective doesn't seek to "fix something" and the totality that implies. The point isn't to come up with lofty visions, missions, or elaborate, phased implementation plans; it's not an intervention, and certainly not a silver bullet. Quite in contrast, a retrospective calls on people to see where they can simply "lighten the load" by improving their situation in one small way immediately. Retrospectives avoid blame-games and embarrassments, and because they happen frequently, problematic factors are brought to everyone's attention so problems can be addressed early on. The emphasis is not on "what went wrong", or who "did wrong", nor even "what when right" but, more importantly when improvement is the goal, "what to do better next time." With the cumulative nature of retrospectives, the benefits add up.

Is the code so jumbled that people wish for the good old days of spaghetti? Are vague requirements problematic? Are other parts of the organization creating a false sense of urgency for their own purposes, or behaving as if they are trying to "drive" the developers? Have deadlines upon deadlines and interruptions that interrupt interruptions sucked away all oxygen? Are there too many meetings? Is everything the #1 priority? Is the team limited by too few developers with relevant skills for the project? Are the tools broken? Is Mr X's lack of tests on Bar the reason that Ms Y's Foo code submission had to be rolled back? Is the geographic distribution of the team causing overhead and confusion? Are sloppy code and poor testing prevalent? Are there components that desperately need to be refactored? Are people mistaking haste for speed and creating a situation there never time to "do it right" but always time to "do it over"? Over time, the list of what's broken or needs improvement can get daunting. But through retrospectives, those circumstances that can be changed can be identified, and the most important ones can be taken on one at a time. Continuous improvement follows. Success builds.

A useful practice with retrospectives is to identify one single thing to improve (for now). This is for two reasons. First, if the team can't improve one thing they certainly can't make better two or more. Second, retrospectives are typically recurring events; after a while, fixing "one per" adds up. When a team is practicing iterations, say with 2 week intervals, that's a dozen or so things/circumstances the team consciously improves over this Quarter and next. Although the practice of retrospectives isn't necessarily tied to the practice of iterations, the two amplify each other in a virtuous circle.

So, the suggestion here is for every team to adopt the practice of retrospectives. Use them to identify circumstances that can be improved. If code reviews are a bottleneck, does there need to be a quick focus on having additional developers become skilled in a component and become a component owner? If yes, buddy up, see to it. Are necessary tests being put off? Discuss pairing up -- have one developer write a component while a teammate writes the tests for it; then switch. Code and tests will get written, knowledge will be distributed, and more things will actually "get done" quicker. Are there too many outside interruptions? Discuss a way to quickly triage them and, except for those truly urgent, schedule recurring "office hours" to handle them then, and so forth.

Retrospective Guidelines. Each retrospective is unique and the specifics must be tailored to fit the situation at hand. Nonetheless, practice in accord to some guiding principles is beneficial. Here are several suggestions, ranked in order of importance.

Facilitator. Trying to hold a retrospective without a facilitator is simply a waste of time. Make sure that every retrospective is facilitated by someone who manages the floor, moderates the conversation such that everyone gets to speak, and that no individual dominates. The facilitator is obligated to not comment, critique, or otherwise influence the content of the conversation. The job is to manage the flow. Stick to the question. Don't drift. Don't rush.
Neutral open-ended questions. Probes such as "what's going well" and "what could be going better" effectively reveal current circumstances and point to activities and situations to either foster or curtail. Avoid leading questions or questions that imply particular answers. More on questions below.
Scribe visible notes. For the facilitator, use the white board. Designate a scribe to take notes (the facilitator is too busy). When video conferencing, the scribe can display in presentation mode so the distributed participants see the notes as they grow. Archive notes in a persistent location for rapid search and retrieval.
Identify an improvement to put into practice. While the group is together, identify a single action to take that could make things better. Pick something tractable that is within the purview of the group holding the retrospective (e.g., making a commitment to refrain from checking in any code to the main trunk until tests are proven, finishing up small "niggling" things that seem to be getting in the way, establishing a continuous build process, and so on). Decide on one, put it into practice, review its effectiveness the next time around. If it works, pick an additional thing. If not, work on it until the underlying situation is alleviated.
Schedule sufficient time. The author's personal experience is that iteration retrospectives require about 50 minutes at first, but then level off at about 15-20 minutes. It becomes an enjoyable time of sharing points of view and figuring out ways to improve. In contrast, for retrospectives reflecting on events or engagements or such, personal experience is that 90 minutes are needed. Hour meetings tend to be ineffective because there's too much to cover, too many tales to be told, and people too antsy to have their say. When such events repeat occasionally, say every Quarter, they level off to about 45 minutes or so after the second or third time.
Repeat with a regular cadence. Decide a regular rhythm for retrospectives. They're an ideal fit for a team practicing iterative development, in that every iteration planning session can begin with a brief retrospective. In 15-20 minutes everyone gets a clear picture of how things are going and which improvement to next adopt. In this way, potentially problematic practices and circumstances are identified, called out, and action is taken to resolve them. It's also a good idea to hold a focused retrospective after all major events -- for example, a major feature release, a significant refactoring effort, a change in build methods, following an engagement with a vendor, or when interns return to school. Each of these situations has lessons to be learned.

Facilitator Guidelines. It is true that expert facilitation is a learned and disciplined art whose mastery requires years of diligent practice. Nonetheless, a few guidelines can get anyone started in the right direction. As a facilitator, be mindful of the following items and the retrospective will yield effective results.

Prepare the venue. Yes, as silly as it sounds, arrive prepared if you're the one to lead the show. If needed, have the VC ready and camera aimed. Have working dry-erase markers (and know the location of the nearest supply station). Have any visual aids, worksheets, estimating cards, or other work items in sufficient supply. Ensure there are enough chairs.
Ponder the topic questions. Give some thought to the likely interpretations and flow beforehand. Participants may be curious about preferred depth of details for responses, whether to use people's names, whether the proceedings are confidential, etc. Be ready to address such inquiries. If you are familiar with the participants, reflect on who might attempt to dominate the floor and who might need to be drawn out to comment. Anticipate.
Set ground rules. Call to order and ask everyone to agree to a set of ground rules. It's useful to have these written or displayed. Get agreement on the ground rules before proceeding. Some suggested ground rules: treat everyone with respect, no laptops (except for the scribe), don't interrupt whoever is speaking, take turns -- don't speak twice in succession, cell phones on stun, take calls out of the room. Make sure the scribe is taking minutes.
Stand, don't sit. Much of facilitation is executed through body language: proximity, pointing, and hand signals. One must be standing for these to be visible and effective. Most likely the facilitator will be writing on the white board. Be animated; keep the energy level up.
Control the floor and the clock. The job of the facilitator is like that of a moderator, although using prepared questions. Be vigilant regarding the agenda and mindful of the clock. Keep the conversation moving. It's rare that anyone has something that can't be said in 60 seconds or less. A minute is a long time for a thoughtful remark. Be ready to politely cut off someone who is enamored by soliloquy, and to do it as many times as necessary. Be attentive to body language such as raising of hands or fingers for a turn to speak, obvious excitement to say something, lack of attention due to staring at the mobile device. Be public. Call them on it.
Remain neutral. Be careful not to favor particular points of view, importance of problems, nor manner of solutions. It's important to curtail someone's enthusiasm for arguing a particular perspective; the retrospective is not the venue for that. Be careful of skilled rhetoricians who will ask the moderator leading questions such as "wouldn't you agree that" or "isn't it obvious that". The facilitator's duty is to enable the collective to reach its interpretations and decisions. It is a unique rhetorical vantage point. Stay above the situation, focus on mining the team's wisdom.

Starter Questions. Neutral questions are vital. A retrospective's purpose is to surface issues that are shared amongst the team, especially those that reflect positive practices that should be continued and negative ones that should be curtailed. It's a situation in which valuable insights arise. To go into a retrospective with a preconceived notion of what the results "should" be, is (as Mr. Boffo would say) being "unclear on the concept."

Depending on the retrospective situation, different questions are useful. The range of potential questions is boundless, and many precious hours can be whiled away striving to find "just the right ones" or some such. Don't do that. Start off using very simple ones. During the session, finish one question before moving on to the next. Capture the unique ideas, not the myriad ways that different people might be rephrasing the same notion in order to have their say. After facilitating about five or so retrospectives, the key questions will become evident given the team's capability, capacity, circumstances, and personalities. For starters, though, just pick a set below (tailored for iteration meetings) and get going.

Use the time-honored set of: (1) What's going well?, (2) What could be going better?, and (3) What one thing can we improve now, and how? Identify what will be changed and have everyone sign up for some part in it. Before you choose, decide -- collectively.
A parallel and humorous approach to this is to write three headings on a white board: "good", "bad", "ugly" and then go around the room having everyone make a remark under each heading. Pick something to improve right now, brainstorm what can be done. Have everyone acknowledge commitment to making the change.
The author's favorite -- after a brief reminder about drag and lift regarding flight, ask: (1) What's adding "lift"?, (2) What's adding "drag"? (3) Name one small thing we can do to reduce drag.
A different set: (1) What should we keep doing? (2) What should we stop doing? (3) What puzzles us? (4) What do we need to change now?
Yet another: (1) What did we expect to get done? (2) What is finished? (3) What caused the delta? (4) What can we do to improve it?

Retrospectives focused on events or particular circumstances can be served by questions of a different nuance. For example, if the purpose is to see what can be learned from a particular set of circumstances such as a previous project, a collaboration between two or more groups, experience with a vendor, and so on. Here are some starter questions for those retrospectives.

(1) Name a headline that best captures the experience. (2) Name a negative (but true) headline that reflects the worst of the situation. (3) What do you wish you knew then that you know now? Following that, identify what to do differently next time.
(1) Describe the situation with a cliche. (2) What practice or approach that was done do we need to remember to do again? (3) Say you had a "magic wand" and could have changed any one thing, what would that be?
(1) What policies/procedures added lift? (2) Which ones added drag? (3) What's the lesson learned? Identify something to improve immediately.
(1) What worked? (2) What broke? (3) Where can we do better?

A pragmatic note. Retrospectives, lean development, agile practices and the like are for teams. Teams are about collaborative activity, working together effectively, maintaining a shared perception of the situation, and a commitment to collective success. The mere perception of all this is beyond the ken of selfish or self-centric prima donnas. They can't see it; in a manner of speaking, they're tone-deaf to the distinction between a group and a team, and the "manner of being" in a team eludes them. Don't expect these techniques to work with them. But if you're actually with a team, the practice of retrospectives delivers high value for improving working conditions, enhancing how work gets done, and having shared pride in the work: something only true team members ever experience. Please, if you think that only 20%-30% of a retrospective meeting would be useful to you -- don't attend. Rest assured; you won't be missed. For everyone else, retrospectives are a simple, quick, and powerful practice to improve the team's overall situation.

Interesting Reading

http://www.retrospectives.com -- Norm Kerth's site. Kerth is the granddaddy of retrospectives.
Agile Retrospectives: Making Good Teams Great by Derby and Larsen -- numerous techniques for generating useful results from retrospectives. Also see their blog. They suggest this sequence: setting the stage, gathering data, generating insights, deciding what to do, closing the retrospective.
Collaboration Explained: Facilitation Skills for Software Project Leaders by Jean Tabaka -- perspectives, approaches, and practices for effective team facilitation.
The Art of Focused Conversation by Stanfield (editor).
A useful set of patterns (or proto-patterns) for retrospectives is posted on the XP123 site.

The Skipper & First Mate: A Pattern for Continual Progress

2008-03-21T23:03:00.000-07:00

How might a manager orchestrate the activities of developers such that progress is always being made? Software projects are fraught with interruptions, shifting priorities, and unforeseeable personnel events. Forward progress is often thwarted -- frequently in parts of a project, sometimes across whole projects -- when a developer in a critical chain becomes unavailable. Having a wizard developer is a wonderful thing; being in a situation where personal circumstances can lead to flow getting blocked is problematic, perhaps dangerous. In the spirit of Christopher Alexander, some observations of software development for over 20+ years are characterized here as the Skipper and First Mate pattern. It shifts attention to the feature being made and away from the people doing it.

Developing software is an exceptionally demanding intellectual task, one requiring exquisite attention to detail, crafting of architectures, inventing algorithms, and writing primo code (including effective tests). Whenever an individual developer is out of the workflow, a group's software development capability and capacity become yet more constrained. Each developer is unique. The knowledge and skills of each individual matters. The learning curve for taking on any new feature in mid-flight is daunting. The context switching is burdensome. And worst of all, some other important work would have to be shelved to free up someone who will have to drop what's underway and step in to pick up the task. Developers are not fungible, especially not in real time. The astute manager asks how such circumstances can be attenuated such that the work gets done and the critical chain remains unblocked. If addressed early, problematic situations won't spread too far or interrupt other work.

Sadly, management practice regarding software development is often blinded by an industrial-era mindset of now obsolete mass-production premises. Today, workers are no longer fungible. Developers aren't working to tend the machine as parts move through it; quite in contrast, the are creating "the machine" -- a very different thing altogether. Most of today's software managers are working in an environment where the technology, and the marketplace, are continually changing. Time is always of the essence -- every technology has a half-life, as does every market opportunity. Continual forward progress is a necessity of economic survival, a prerequisite of success. Having things delayed often means having things dropped, and the underlying investment becomes squandered. This starkly disappointing example of muda in product development is commonplace.

The cascade of illusion spawned by the mass production mindset is insidious and counter-productive when it comes to creating software. It is insidious in that it leads managers to break down work into "one person" tasks, often reflecting the tacit presumption that employees are fungible, and also separates key related elements (e.g., design and programming) into separate divisions of labor. It is counter-productive in that it doesn't recognize the risks and costs to the organization of reliance on individuals at critical chain nodes and the presumption that nothing will go wrong, and if it does, there's someone else at the ready to step in and get things done. If only it were that simple, but experience teaches us otherwise.

To the mass production mindset, the machine is the focus of human activity -- and humans are pretty much interchangeable in their roles. One person can do one job because should someone falter, another can step in a pick up where the other left off, nearly immediately. The critical chain is the machine, not the worker who is expected to execute known and often repetitive activities in a prescribed manner. With software, the critical chain goes through the human mind as the key factor in production. And that makes all the difference in the cosmos.

Things always go wrong -- people get sick, family members require attention, employees depart for greener pastures, and so on. An organization can only depend on individuals in critical chain processes when individuals are fungible -- one can quickly take over for another with no interruption of the work flow. Moreover, it implies that there's a pool of available developers, ready to step in, who themselves aren't engaged in other priority activities. Software isn't like that. Often, losing a developer, even for a while, means a feature or fix just got delayed or perhaps dropped. To the bean counters, and hopefully the managers, this means the entire investment has just been diluted, if not squandered. When such circumstances occur, expectations need to be reset -- uncomfortable conversations follow that never turn out well. This happens every day in organizations everywhere. There is an easy solution to this dilemma.

Many software managers practice, or have accepted the pressure, to assign single tasks to single individuals, and they leave it at that. Workplace rhetoric echoes with "directly responsible individuals" or of the importance that every task must have an "owner" if there's to be any hope of getting something done. Single individuals, of course, can complete even the most complex software tasks, but is that the best way for manager to ensure the feature gets done? Creating a feature, for instance, has diverse aspects (e.g., designing, coding, testing, etc.) any of which could be further decomposed into absurd detail. In the mass production mindset with its proliferation of organizational roles in the division of labor, it wouldn't be surprising to see each of these aspects handled by a different person in a different department. Role proliferation induces complex work flows, balkanizes knowledge, and creates numerous wait states and boundary transitions: all muda.

Accountability, of course, is a Wonderful Thing. But accountability doesn't go far enough when delivering a feature is the point of the activity . Having "one person responsible" shouldn't imply, as often seems to be the case, that only one individual gets to contribute to, and have responsibility for, getting something done. From a managerial standpoint, this raises a provocative question -- which is more important: having someone to blame or ensuring that the task gets done? To observe most organizations and most managers is to see that blame often trumps progress. Otherwise, managers would staff their projects to first and foremost ensure task completion. Too much conversation about responsibility drives out conversation about work and thereby diminishes the signal-noise level related to effective progress. A sticky web of human relations, violated expectations, and politics follows all of which detract from the objective of completing the work as originally intended.

The solution is actually simple: ensure that at least two people sign up for any feature (in a ranked manner described below). While two people share the tasks one is primarily responsible (a "skipper" so to speak) -- should things go awry, it becomes the other person's responsibility (the "first mate") to ensure the tasks get done. The focus here is on the completion of the feature. The pairing creates a circumstance of ensuring that knowledge is shared and that the work gets done. From a deployment standpoint, this means that at any one time, more people are assigned to fewer features, but those fewer features typically get done faster, with each avoiding the risk of total disruption due to factors arising from an individual's unforeseeable circumstances.

Yes, this means pairing, although not necessarily XP's keen insight of pair-programming. Just "pairing". It is an effective mechanism for enhancing team performance. In the simplest sense, pairing means two people working together -- not reflective of a pas-de-deux or duet, but more akin to a master-journeyman relationship or perhaps a bicinium. It means two people are responsible, one more so than the other, for finishing the work. Staff deployment for an iteration then has both a skipper and a first mate for each feature, aspect, or task being undertaken. With software teams, the skipper on one task also serves as the first mate on another, and vice versa. Given that every iteration has a prioritized activity list to be undertaken by the team (typically longer than can reasonably be completed) developers are deployed as skippers for the highest priority backlog items. Everyone is expected do serve in both roles.

Organizationally, the Skipper and First Mate is a powerfully productive pattern -- features get done at a reliable pace, personal circumstances of individuals don't thwart progress by disrupting a critical chain, and tacit knowledge diffuses throughout the team. The work gets done; progress continues apace.

Cooperation between the skipper and first mate takes many forms -- if the skipper is writing a feature, the first mate can be writing appropriate tests. If the skipper is authoring a design document, the first mate is available in a most timely manner for reading it closely for coherency and accuracy. If the skipper gets the flu, must return to the other side of Planet Earth, or is reorganized out of the workgroup, the first mate is obliged to step in and move the task forward until next iteration. The skipper and first mate allocations are revisited at every iteration planning session. This is a pattern of organizational adaptation to the changing circumstances inherent in having individual developers in a critical chain.

For every feature, developers are brought into on-going conversations with someone else about the design and code. They get a quick response for reviews, and get testing support right away. As a result of the interactions, more insights are generated, more risks are identified, and more solutions are surfaced. The key point here, however, is the feature typically gets done sooner than if an individual had done it alone, but more fundamentally, it's much more likely the feature will get done as a matter of pace in circumstances where the "directly responsible individual" becomes unavailable, distracted or reassigned. The critical chain is not disrupted and positive results are achieved.

The Skipper and First Mate is a pattern reflecting cooperation and readiness; it's about a shared objective and commitment. Someone with appropriate knowledge and skills is at the ready, but not waiting in reserve. No one sits idle, but the impact of any single individual's personal circumstances whether it be family emergency, temporary mind-block, whatever, doesn't thwart progress. The work gets done. In an environment of peer reviews, this pattern brings a shared sense of accomplishment, and respect for each other's contributions. The Organization benefits.

When bringing pairing to a team, the manager is faced with the delicate, but all so real, factor of personalities. Pairing is not a comfortable habitat for lone wolves. The extent that a manager caters to them (consciously or otherwise) does shape the nature of the team's activity and subsequent team performance. Inherent in this is the risk of diluting or squandering the to-date investment, and not meeting stakeholders' expectations.A manager needs to determine whether the work at hand requires a team or can be done by a set of individuals in critical chains. In a research group, for example, critical chains are few; in a product group they are common. Often, reliance on lone wolves brings about circumstances of critical chain disruption (and all the subsequent muda of adaptation and corrective action), although not necessarily intentionally. In an environment of peer reviews, this pattern brings the piercing eye of scrutiny. The Organization learns.

This pattern is offered in the spirit of Alexander's A Pattern Language, especially as brought to software engineering management through Coplien and Harrison's Organizational Patterns of Agile Software Development. Following the pattern definition of hillside.net, we have:

Name: Skipper and First Mate

Problem: An individual in the critical chain for completing a feature may at times becomes unavailable. The feature is necessary for an upcoming release and other teams are depending on it being ready for their own commitments. It is vital that the organization / department / team deliver the work in a timely manner.

Forces: Social pressure, economic understanding, and development habits lead managers to deploy staff in an ineffective manner. The work is on a critical chain for a project or feature needs to get done or is perhaps needed for an upcoming release or other teams are depending on it. Managers who have many responsibilities often believe that unless they assign one person per task they're not being efficient and this will reflect badly on them. These manager succumb to the mass production mindset of assigning one person per task. Problems can occur and the individual on the critical chain becomes unable to deliver in a timely manner. The work on the critical chain falters. Should the manager have placed two people on the task and the roles and expectations are unclear, efficiency is reduced as one person can remain idle and the situation of too much undone work becomes worse and the circumstances repeat themselves.

Solution: Sign-up two individuals, the "skipper" and the "first mate", with a clear order of responsibility and expectations of collaboration such as the skipper writing a feature while the first mate writes tests, the skipper writes a design doc whilst the first mate is at the ready to review it and so on. Should the skipper become unavailable; the first mate steps in to ensure the work is completed.

Resulting Context: The developer team becomes accustomed to working in pairs that are dynamic and often shift iteration by iteration. However, in organizations where the culture emphasizes single individual responsibility and individual accountability is more important than forward progress, a manager may be viewed negatively for implementing such a pattern leading the manager to return a single-task-per-person approach, thus reintroducing critical chain risks of reliance on single individuals and the cycle begins anew.

Rationale: Avoid depending on a single individual on a critical chain for development that is time critical.

Developers are bright folks. Many quickly recognize the personal advantages afforded by this pattern once they experience it. The skipper has someone to draw on, and the savvy ones will find advantage in that, especially for quick design, code reviews (or at least perusals), and testing. The first mate likewise knows that lending a helping had averts getting saddled with the work, and getting recognition for the contributions. Both recognize that influence of peer reviews on performance ratings. And the manager recognizes whether the feature got done. The organization continues progress apace.

The Skipper and First Mate pattern is broadly applicable. I've seen it work from advanced software development teams to university research projects, from the profit-focused corporations to non-profit service organizations. Most of all, people like to get things done, and many people like to work with others. This pattern is a simply way reducing risks along critical chains, ensuring necessary work gets done, and providing a convenient and comfortable opportunity for people to work together.

For more information on patterns, visit: c2.com, The Gang of Four, search Google books, or see pattern-related Amazon listings.

Timeless Truths Of Software Development

2007-12-29T22:12:00.000-08:00

Having been a software engineering manager for literally decades, a few truisms have become evident. These have emerged in interactions with other roles, notably clueless executives or, worse yet, folks who came into software management from some other discipline such mechanical or civil engineering. Or, heaven forbid, marketing or human resources.

Yes, software is different. In fact, nearly all software is unique in that the particular capabilities being developed have either (1) never been done before; or, (2) if not unique, are someone else's trade secrets and unavailable. Consequently, nearly all software requires more than a modicum of exploration and experimentation, of trial and error, of continual refinement and understanding.

Of course, convergence on workable code is accelerated by wisdom, experience, skill, and raw native intelligence. Nonetheless, it is not the same as looking up the load of a bridge pier in a book, or calculating frictional forces in a gear assembly. Such tasks have occurred untold thousands of times in the past. When someone says "I need a bridge from here to there" the intent is understandable. With software, the intent tends to be vague, open to interpretation, and time and time again there's "only one more thing" . . . .

Software, in contrast, is new, typically from the ground up. Moreover, the underpinnings of the software -- computational capabilities -- continue to advance in accord with Moore's Law, with no practical end in sight. This evolution in turn continually renders older software obsolete. Even if the software continues to exist, there in fact may be no machine to run it. Or if it could be made to run, the performance would be unacceptable given how everything else is likewise evolving. And this is to say nothing yet about security, wherein exploits continually require software patches or new applications and devices for protection, requiring the continual creation of yet more software that has never been done before.

These facts, blindingly obvious to everyone who has written code for a living, seem to elude certain allied roles. There are those clueless (sometimes brainless) people who demand a plan for software akin to a plan for a bridge or a printed circuit board. It would be nice were it possible, but it isn't aside from the trivial case. The journey from inspiration to working code is subtle, unique, arcane, and highly dependent on the brainage and personalities of the developers. The requirements keep changing; the hardware keeps changing; the security environment keeps changing. Obsolescence begins when the software (or it design) becomes frozen. Dylan said it better: Proves to warn that he not busy being born is busy dying. Here are several observations that continually recur, written in absolute form to be provocative.

People Who Don't Know How To Do Things Always Claim They're Easy To Do

Yes, Alexander Pope was right: A little learning is a dangerous thing; drink deep, or taste not the Pierian spring: there shallow draughts intoxicate the brain, and drinking largely sobers us again. Effective software development requires intricate algorithms and sophisticated coding practices, refined skills and the insight of experience, all brought together with social finesse. Those in allied roles who have a smattering of learning about software and its development often press their claims (e.g., "build this", "don't waste time on testing", etc.) by expressing how easy such a feature would be to build. Naturally, they have a friend, or so they say, who could do it in a day. Worse yet, is hearing them claim that were the developers to build it right testing wouldn't be needed. Of course, the inherent complexity of interactions amongst software modules in any application necessitate testing, especially to protect against the clown-code that "just one more thing" adds. There comes a point of foregoing arguing with others about the false perception of ease; instead recognizing the individual's basic lack of understanding of the circumstances at hand. Out of politeness, co-workers typically don't draw attention to this intellectual handicap when it becomes evident. The one truism that emerges from this is that such people who claim such things are easy, rarely -- if ever -- can do it themselves at all.

People Who Don't Have To Do Things Always Claim They're Quick To Do

Computers are magic to those who don't write code. I've noticed that those people who focus on schedules are rarely the people who actually have to produce the work. Closer inspection reveals they often they don't even know how the work actually gets done, let alone how to organize it. The vagaries of dependencies elude them; the complexity of software development is beyond their ken, the orneriness of tools and fragility of build systems and constraints of legacy code are outside their experience. Often, their day to day work is relatively simple, not requiring the daunting learning curve of application development. Hence, all these myriad things that cost time elude their understanding. The detours while developing software are natural to the task itself. To naively expect such detours won't occur is to mistake their conception of the destination for the duration of the journey to get there. Software takes as long as it takes. It could always use more work. To paraphrase the poet Paul Valery, software is never finished, only abandoned. Not understanding where the time goes, doesn't mean it doesn't go somewhere nor that detours could've been avoided.

People Who Don't Have To Decide Things Always Claim They Know What To Do

Software development is tricky involving many partially-useful ideas, numerous alternate algorithms, myriad approaches to coding, and a maelstrom of personalities. The skills of being a technical manager are distinct from those of being a people manager. The activities of tactics are distinct from the perspectives of strategy. It's easy, and myopic, for an observer equipped with only a smattering of facts from one of these skills to foresee a simple solution to the situation at hand. Those with wisdom of people and organizations don't necessarily have technical insight. Those with technical brilliance may be inept at working with others. Tactics can be blind, but strategy in itself doesn't get anything done. A decision's quality is constrained by the domains of its considerations. If it only sees technical situations, people problems create bedlam. If it only focuses on people, redo after redo from technical blunders are commonplace. In all of these situations, those with one or another myopic, selectively informed point of view, claim they know just what to do. As H.L. Mencken observed: There is always an easy solution every human problem -- neat, plausable, and wrong. Making a decision brings with it the responsibility to use all the facts at hand, to draw on wisdom from all available domains, and to do the right thing for all the people involved. Not having to weigh such competing notions and make unpleasant tradeoffs is easily avoided by those who don't have to make the decisions.

Decisions are hard. Software is hard. When others claim it's easy to do, think about it. When they claim something is quick to do, give it some thought. When those not responsible to make decisions claim they know what to do, think it through. What unanticipated consequences can you see right away?

Software development requires deep skill to understand, let alone to perform. A software development organization is a complex social situation of many specialties, diverse personalities, clashing agendas, and demanding exactitude. Those who can actually deliver software don't make claims such as those above.

(CC) Some Rights Reserved.

Code Is Fact; Design Is Hope

2007-11-25T16:45:00.001-08:00

Software is people's way of having machines do their bidding. Learning to command them is a thrill of the highest intellectual order. Developing software engages the mind in an exquisite artistry of creativity, consummate attention to detail, technical acumen, and new code written afresh with insight. Someplace between the spark of inspiration and the release of software, a design typically occurs: sometimes many. Designs are human creations and they are fun to make. They are ideas on a path to become realities. When we're lucky, the idea and reality match. Sometimes when we're more lucky, they don't.

Software is written by people who have banded together in an organization where the work gets done. Our work is intricate and hard. Developers on a payroll are expected to play well with others. In the workplace, the mind's eye shifts from imagining the individual artistry of a lone, pixel-lit wizard at midnight, to that of a work group immersed in their email, white boards, fluorescents, cubicles, meetings, budgets, plans, designs, and codebase. It's an environment of social dynamics: one permeated with ideas, conversations, misunderstandings, disagreements, personalities, politics, resource contention, celebrations, and so on. We inhabit a complicated and complex world. Entropy characterizes all human endeavors. Everything human churns with some level of chaos as our tasks and circumstances continually change. We thrash, in degrees unbeknownst to ourselves. Yet, we get a lot done -- software has become the foundation of modern society.

Whenever groups form from people coming together, chaos looms. Opportunities abound for control and anarchy, mindless management and turmoil, happy people and sad people, inspired teams and resigned individuals. The quality of the outcome is fundamentally enabled and constrained by the collaboration amongst the brainage available to tackle a task and the climate in which they engage. The organizations in which we develop software continually change, as do myriad external forces of technological advance, dynamic market conditions, and workforce globalization. Things change, not all for the better. Vigilance is vital; adaptability a necessity; and a clear understanding of the current circumstances a must.

Forethought is a Righteous Thing. It'd be silly to not carefully consider the situation, to ignore available skills, or to avoid what's been learned in similar circumstances. The future isn't completely uncertain -- some measure of planning would be prudent. Not all advantageous notions spring forth in an initial burst of insight. Writing code can offer a compelling sense of "process gratification" that arises from the simple act of doing it. This force can be so powerful that there's wisdom in holding off a bit and reflecting on what needs to be done. But how much design? How granular? From some points of view, a few sketches are sufficient. Other perspectives advocate that every anticipated activity should be planned and scheduled in precise detail. The notion of a "design" spans both these extremes, irrespective of the underlying presumptions of the future (dynamic or fixed, respectively).

Designs bring structure to the chaos. They can be illuminating intellectual conceptions. A design seeks to guide developers to create a particular software release formed from current ideas. It is an anticipated solution, often to a murky and ambiguous situation. It's a best guess as seen from the perspective of today. A design presents a hope about a future, a desired outcome as seen from long before. Sometimes designs are fluid, open to opportunity and serendipity. Other times they're not -- especially in a bureaucratic workplace. In such situations designs often include plans that specify what gets done when by whom -- a frozen design to be executed by worker-bees. Sometimes this plan reaches years into the future. Such circumstances are an evident absurdity to any but the most naive (or Machiavellian) observer. The latter is particularly pernicious in organizations whose culture rewards false precision over accuracy, posturing, and resource contention. It's the old human game of power vs. performance, all over again.

Designs are snapshots in time, an illusion of how things were (as soon as the notion occurred, it began to age). In such an unchanging world, nothing is learned along the way; insights can be simply dismissed. Neither optimizations nor new algorithms ever occur, so all can be safely ignored. A frozen design deems code to be written based solely on what is known today. What needs to be known is known; or is simple and straightforward to discern and recognize, or so some designers of the "expansive elaboration" persuasion would have us believe.

A frozen design becomes historic: ignorant of every tomorrow. All mismatch between design and outcome becomes a measure of waste, of muda. Irreplaceable time, priceless attention, and synergistic opportunities are all lost. Binding to a hope of tomorrow, instead of a fact of today, limits a design in equal measure. How dated is the design? That's a key question in assessing the value of a design, as are considerations of how fast things are changing. Circumstances will change, designs are anticipatory -- a planned solution. Designs are hope.

Designs can't be perfect. Complexity spawns the unexpected. Things break. Employee turnover occurs. People get chagrined, but not surprised, when requirements change. "As observed" differs from "as expected" (otherwise the Chi-square Test of 'goodness of fit' between 'observed' and 'expected' wouldn't have come about). When things change, learning opportunities arise, insights emerge. Better ways are invented.

Pursuing a presumed perfect solution, based on old methods and known faulty understanding, is a fool's errand, made worse in circumstances known to be uncertain. The search space for the "best" solution is immense and could take eons to traverse; judgment criteria for comparing alternatives are subtle and time-consuming. Sometimes designs leave things open; other times they guess. If the guess becomes cast in a plan as received wisdom, waste is generated.

Elegant designs reflect brilliance, discernment, and innovativeness. When developers architect a test, a feature, a system, et cetera, their designs make manifest a particular weltanschauung by specifying certain categories, constants, classes, and relations amongst them. APIs, event systems, and protocols are likewise determined. Use cases are identified. These concepts and nomenclature become the parlance of the task; it's how people talk about the code to be written. It is what lingers on white board diagrams. The design, hopefully, reflects the best educated guess about a future, presuming things remain the same, yet leaving flexibility because things are sure to change.

Software evolves into a tangled morass of classes and calls with myriad parts and pipes and protocols; developers naturally get boggled. Software is so complex that a full understanding is elusive and prohibitively time-consuming to attain. Moreover, all algorithms have fundamental and opaque trade-offs in all but the most trivial circumstances. Alternative methods typically abound, and every single one brings unanticipated side-effects and induces cryptic corner-cases. Often, a full appreciation of a typical system eludes even the keenest of available minds. Complexity yields emergent phenomena. Yes, there is a ghost in the machine.

A design often announces a schedule and roster and thereby creates its own reality; its own ambiance of how to work gets done and who does it when. Today's design becomes outmoded to the degree that tomorrow differs from today. When designs are a prescriptive script of what to do and when to do it, a steady-state world in being presumed -- unchanging, stagnant in all aspects. This postulates a world where no one learns, opportunity isn't recognized, serendipity is ignored. A design is a conjured illusion of a hoped-for future. Where's the limit of prudence regarding to what extent should a design be accepted as the final authority? Optical illusions in freshman psychology warned us away from being too quick to grab onto any particular perception as being The Truth.

Some designs focus on the goal to be accomplished. Others focus on detailing how the code expected to accomplish the goal shall be written, which makes it a plan. At any point along the way, the code -- not the plan -- is the true measure of finite progress. Most groups typically have at least one or two "flat-earthers" either obsessed by conformance to plan or inherent belief in a frozen design (typically, these aren't people who have written much code).

Code will only do what the code can do. It doesn't make promises; it's not an existential argument of what might be done, as a design may well be. What the code does today is reality. It is what is known. The code is fact. The code does what the code does. By the time the code is completed, circumstances will have changed. The more rapidly that markets and industries change, the quicker that today's design will lag tomorrow's solutions. The longer from today, the greater will be that deviation, causing ever more muda. Designs show what is hoped of the code. Tests reveal the fact of what the code does.

All computer science and software engineering students are cautioned by Knuth: "premature optimization is the root of all evil." A design is by definition a premature optimization. After all, it is the initial, best estimation of the circumstances. It provides the start values for the mental model of what/when/how to bring the software into being. Blockages, dependencies, staff turnover, shifting business priorities, legal surprises, technical breakthroughs, and all those other everyday, work-a-day world realities, haven't yet crept in. But they will. While designs are necessary, they are by no means sufficient. And to the extent that circumstances are dynamic, any manner of sufficiency degrades yet more, in much greater degree for the frozen ones.

The code is an expanding body of facts regarding the design -- what is found to be true and what has been shown to be muda. By focusing on the code -- the fact -- true progress can be judged. A question of balance emerges: plan or adapt? Circumstances often change. Plans seldom keep pace. Lean and agile emphasize deferring commitment as long as possible and to avoid premature decisions. Effective software development becomes an ongoing activity of planning, and using the facts revealed by the code for continual course-correction. The design reflects hope; the code reveals fact. Progress continues until closure is attained.

Saffo's adage "never mistake a clear view for a short distance" intertwines with Eisenhower's wisdom "Plans are nothing; planning is everything." Together, they illuminate an essential notion -- a design is static. It's a plan. Designing, though, is dynamic. Adaptation occurs, people shift attention. People continually scan the horizon; opportunity arises; deliberations ensue while progressing on what appears to a promising path. Hopefully, learning along the way. A design fills one with hope; the code reflects hard facts. Seeing this isn't rocket surgery.

(CC) Some rights reserved.

Lean Product Development

2007-03-19T20:20:00.000-07:00

The Toyota Product Development System was recently published by Morgan and Liker. It's a companion piece in some ways to The Machine That Changed The World which gave rise to the notion of "lean manufacturing." Toyota's product development system is characterized as thirteen principles which map exquisitely to agile software development. These principles span three mutually supportive and aligned system elements:

Process
Skilled people
Tools & Technology

These principles are are listed on page 18:

Establish customer-defined value to separate value-added from waste.
Front-load the product development process to explore thoroughly alternative solutions while there is maximum design space.
Create a leveled product development process flow.
Utilize rigorous standardization to reduce variation, and create flexibility and predictable outcomes

The People Subsystem
Develop a Chief Engineer system to integrate development from start to finish.
Organize to balance functional expertise and cross-functional integration.
Develop towering technical competence in all engineers.
Fully integrate suppliers into the product development system.
Build in learning and continuous improvement.
Build a culture to support excellence and relentless improvement.

The Tools and Technology Subsystem
Adapt technology to fit your people and process.
Align your organization through simple, visual communication.
Use powerful tools for standardization and organizational learning.

Google's Ten Things

2007-02-25T14:17:00.000-08:00

I'm feeling lucky. I have the opportunity to work everyday with world-class computer scientists and software engineers within one of the happiest organizations I've experienced. I work at Google.

The Google Organization has many positive facets, but one that always comes to mind when I'm walking my dogs and reflecting on life in general is the list: "ten things Google has found to be true". They are listed here:

Focus on the user and all else will follow.
It's best to do one thing really, really well.
Fast is better than slow.
Democracy on the web works.
You don't need to be at your desk to need an answer.
You can make money without doing evil.
There's always more information out there.
The need for information crosses all borders.
You can be serious without a suit.
Great just isn't good enough.

Plus, if you'd like some additional reading, check out the original IPO Founders Letter. Google is a special place at a special time.

Deming's System of Profound Knowledge

2007-02-04T15:25:00.000-08:00

W. Edwards Deming's work on quality, while widely misinterpreted and misapplied in the USA, was nonetheless a watershed that Japanese companies, especially Toyota, took to heart. Deming sees organizations as human systems and management as having the primary responsibility for effective results. Mistakes are most often the fault of the system and management's myopia in not seeing this -- not problems due to workers. For reference, see Chapter 2 of his Out of the Crisis. Below are Deming's 14 points accompanied by commentary related to software development.

1. Create constancy of purpose towards improvement of product and service, with the aim to become competitive and to stay in business, and to provide jobs.
Consider software development that is bogged down with heavy procedures and reporting, focusing more on plans than working software. In contrast, think about software development focused on maximizing the value to the customer in the minimum amount of time. A constancy of purpose calls for a focus on software, not on process, documentation or conformance to plan.

2. Adopt the new philosophy. We are in a new economic age. Western management must awaken to the challenge, must learn their responsibilities, and take on leadership for change.
Workers cannot adopt a philosophy of constant improvement if management doesn't do it as well. For software developers to be captive of rigorous project management processes that miss the fundamental dynamic that software is product development, not rote manufacturing, removes their opportunity to continually improve their means of working and creating great software. Management must come to recognize that what is needed are reliable means to generate software, not repeatable processes that sacrifice innovation and adaptability that results from a robotic and misguided religiosity of "conformance to plan".

3. Cease dependency on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place.
Design quality in, don't use inspection to find errors. Mistake proof the system.

4. End the practice of awarding business on the basis of price tag. Instead, minimize total cost. Move toward a single supplier for any one item, on a long-term relationship of loyalty and trust.
Recognize that outsourcing is often a consequence of accounting that focuses on human utilization, assuming that for software development labor costs are variable instead of fixed. A better measure is to focus on minimizing the time from concept to cash and having the appropriate personnel to do it. Focus on developing great teams that can deliver great software and the rest of the negotiations become minimized.

5. Improve constantly and forever the system of production and service, to improve quality and productivity, and thus constantly decrease costs.
Improvement is a constant process, it is not a big-bang perfect system. Avoid prescriptive systems as they are immutable and suppress improvement.

6. Institute training on the job.
Adequate training equips workers to make important decisions at the point of the work reducing needless variation and avoids slowing things down for management approval. Ensure that people are equipped to perform and then let them.

7. Institute leadership. The aim of supervision should be to help people and machines and gadgets to do a better job. Supervision of management is in need of overhaul, as well as supervision of production workers.
Distinguish between leadership that motivates people and engages them from quotas and schedules which are rudimentary management tasks.

8. Drive out fear, so that everyone may work effectively for the company
In a fearful environment, workers do not operate in the organization's best interest; instead their energies are by necessity focused on self-protection.

9. Break down barriers between departments. People in research, design, sales, and production must work as a team, to foresee problems of production and in use that may be encountered with the product or service.
It's necessary to recognize for effective organizational performance that departments and groups serve each other, not the hierarchy or the fiefdoms of local managers.

10. Eliminate slogans, exhortations, and targets for the work force asking for zero defects and new levels of productivity. Such exhortations only create adversarial relationships, as the bulk of the causes of low quality and low productivity belong to the system and thus lie beyond the power of the work force.

Eliminate work standards (quotas) on the factory floor. Substitute leadership.
Eliminate management by objective. Eliminate management by numbers, numerical goals. Substitute leadership.

Mistakes typically come from bad systems not bad workers. Don't exhort people to work harder or smarter; instead create a more intelligent workflow and system tailored tot he essential nature of software development as human collaboration (not just coordination) such that people can excel.

11. Remove barriers that rob the hourly worker of his right to pride of workmanship. The responsibility of supervisors must be changed from sheer numbers to quality.
Production quotas create incentives for poor quality goods. Objectives presume the world is stable and unchanging, thus squashing any hope for improvement in ongoing processes.

12. Remove barriers that rob people in management and in engineering of their right to pride of workmanship. This means, inter alia, abolishment of the annual or merit rating and of management by objective
Afford the opportunity for people to do excellent work and be adaptable to the situation at hand instead of being constrained by an abstract process of paint-by-the-numbers.

13. Institute a vigorous program of education and self-improvement.
Hopefully, this is self-explanatory.

14. Put everybody in the company to work to accomplish the transformation. The transformation is everybody's job.
Management must avoid the tendency of "do as I say, not as I do." Avoid having "change agents" as their efforts will be spotty at best. Instead, empower everyone to act to constantly improve the overall situation.

The Seven Principles of Lean Software Development

2007-01-02T21:11:00.000-08:00

#1 Eliminate Waste

Remove all non-value adding waste
Myth: Early specification reduces waste

#2 Build Quality In

Use inspection to prevent defects, not recognize them afterwards
Myth: The job of testing is to find defects

#3 Create Knowledge

Use feedback every step of the way; only then can complexity be tamed -- it's never available beforehand
Myth: Predictions create predictability

#4 Defer Commitment

Schedule irreversible decisions to occur at the last possible moment
Myth: Planning is commitment

#5 Deliver Fast

Minimize time from concept to cash by maximizing throughput
Myth: Haste makes waste

#6 Respect People

Use responsibility based planning and control; reward sphere of impact, not span of control
Myth: There is one best way

#7 Optimize the Whole

A lean organization optimizes the entire value stream; any other optimization is sub-optimal for the overall organization
Myth: Optimize by decomposition

From: Implementing Lean Software Development by Mary and Tom Poppendieck, Addison- Wesley, 2007

Power of the state in decline: Walter Wriston

2006-04-15T20:17:00.000-07:00

Q: Why is the power of the state in decline?

Walter Wriston: Money goes where it's wanted and stays where it is well treated, and that's all she wrote. This annoys governments to no end.

Stateless money functions as a plebiscite on your policy. There are 300,000 screens out there, lit up with all the news that traders need to make value judgements on how well you're running your economy.

Today, if the president goes into the Rose Garden and says something dumb, the cross rate of the dollar will change within 60 seconds.

The information standard is more draconian than the gold standard, because the government has lost control of the marketplace.

Technology has overwhelmed public policy.

Q: As the power of sovereign governments wanes, who will be left in charge?

Walter Wriston: Everybody.

A magazine clipping found while cleaning a closet this evening. It was from Wired, 1998. The implications of Wriston's message is one of the reasons I love Google. And the other search engines as well. And my browser. Having an open source browser is just so sweet! Welcome, home -- Everybody.

Programmable Web Marginalizes OS

2006-01-30T23:38:00.000-08:00

For many of us, mashups first appeared when we saw San Francisco Bay area apartment listings or Chicago crime sites superimposed over Google maps. Now, they're multiplying like Star Trek's Tribbles. See www.programmableweb.com/mashups and www.mashupfeed.com/. If you'd like to stop, kick back, and have a beer to think about all this, start here: beermapping.com/maps/sanfranbeer.html/.

For a big-picture, industry view see: blogs.zdnet.com/BTL/?p=2484 and poke around www.programmableweb.com/

With 'mashups' all the rage these days, taking a few moments to explore and ponder the API matrix (www.programmableweb.com/matrix) makes a few things clear. In simple terms, the matrix shows mashups as exemplars of combinations of various web service APIs, such as Google, Flickr, del.icio.us, etc. Web services are being interconnected via their own APIs via the Internet. This isn't the traditional notion of 'program' APIs communicating, but services out there in the ether dwelling on servers who knows where. A Web 2.0 blogging group posts at www.web20workgroup.com and innovations are distributed, mapped at www.fourio.com/web20map/.

These nerd bits are exciting! As innovators build out more services based on web APIs, the underlying platform's operating system shrinks in importance, asymptotically approaching irrelevance. The browser is the platform for web services. Netscape was right!

Communities are forming; developers are camping: www.mashupcamp.com/

It's happening here. It's happening now.
Watch the skies. And screens.

How You Gonna Keep 'Em Down On The Farm?

2006-01-30T01:20:00.000-08:00

True friends' messages encourage action. Between the Blue Oxen Catalyst event, and Engelbart's 81st birthday party today at Stanford, one message came through loud and clear from my friends and colleagues: "You've gotta get out more; you've become a hermit. And blog -- you've gotta blog."

In retrospect, of course they're right. The time demands over the last several years has been immense, but there's a richness to my personal life that happily lures my discretionary moments. And I write so much through my day job, that writing/blogging has no power to entice my hands when a synthesizer and guitar are in the same room.

So, OK. Time to start getting out more. And blogging, occassionally.

Thank you, friends. Camaradarie trumps solo pursuits. At least most times.

Year of the Dog

2006-01-30T00:52:00.000-08:00

All too often in "workshop" gatherings, governance issues emerge as a centroid of discourse. They still remain to learn that a more than 'few parts per million' of process and authority will spoil the soup.

It's the year of the Dog.
http://sunarcher.org/pix/whippets.jpg
Happy New Year.

Live Strong

2005-01-18T13:37:00.000-08:00

Lance Armstrong is an inspiration to many of us, and that's why I wear one of the Lance Armstrong Foundation's yellow bracelets. While it is a show of support for cancer survivors living strong, it's also a constant reminder of relentless dedication to achieving one's goals against the odds. Not the punditry odds pronounced by critics, mind you, who easily sway those people of faint heart lacking firm center. No, but the real odds presented by life threatening illness, athletic challenges of the grandest scale, and reaching a world-wide audience to "live strong."

A few weeks ago at a Los Altos City Architecture and Site Committee meeting I noticed one of the commissioners wearing a similar yellow bracelet. Of course, it could've been for show, but nonetheless that commissioner was the one to see the big picture in terms of our local codes and evolution of neighborhoods. The behavior fit the bracelet -- doing the right thing at the right time, irrespective of voiciferous critics, competing interests, and myopic expediency.

Thank you Lance. And thanks to everyone who echoes by example integrity and responsibility.

Gmail Is Wonderful

2005-01-10T17:54:00.000-08:00

Google's web mail "Gmail" has become my favorite mailer. I really like how Gmail works -- it makes my life easier, and breaks away from the file/folder hegemony we've suffered through since the '80s.

In essence, Gmail messages are categorized via labels, either by rules/filters or by manual action. From then on, messages are aggregated based on your personal labels instead of some physical location such as a folder. Moreover, a message can be denoted by multiple labels so it can be reached via alternate routes -- not just one place for ambiguous things (with traditional mailers, where do you file items related to "computer communications" ... with computers or with networks?). Plus, as you'd expect with Google, the searching is speedy and excellent. Want to find all messages about the "hydrogen economy" or instead about "whippets"? Just search for them. Want do find those people who communicate about "hydrogen economy" AND "basenjis" ... just search for it, and if that's in your messages, it'll become quickly evident.

Yes, I'm aware of the critisms regarding privacy, ad placement, big brother, perpetual text, and so on, but deep thinking reveals that's all myopic nonsense. While those things are mostly true, they're not unique to Google, so why single them out from Y!, $quish, AOL, or your own ISP? Privacy on the web is an oxymoron, save encrypting everything (Get my PGP key if needed from my personal page).

There's much more to Gmail, but that's a start. It took me a while to become accoustomed to working with labels and categories instead of folders and locations, and to find things by searching instead of index listings, but now that it's familiar, there's no way I'd want to give it up. Gmail is an out-of-the-park homerun in my book.

Our Canine Guardian Angels

2005-01-09T20:26:00.000-08:00

Dogs are characters, sometimes clowns, and can be great friends. And they're trainable to be more than a one trick pony. Tonight CBS 60 Minutes aired a segment about dogs that can smell bladder cancer -- work in the UK recently published in the BMJ. Dogs were trained to smell the chemical trace of a specific cancer and picked it out from a set of 6 dishes with over a 40% accuracy -- far beyond a chance occurrence. Curiously, the dogs kept going to one sample the hospital believed was not diseased which turned out that it was. Another life potentially saved by a furry friend that the physicians had missed.

This calls to mind a response to the 2005 question on the edge.org that asked notable authors about things they believed that they couldn't prove. Joseph Ledoux, a neuroscientist at NYU, chose to answer this question by saying he believes that animals have feelings and other states of consciousness. He then goes on to say that "we can't even prove that other people are conscious, much less other animals." Recently, my whippet underscored Dr. Ledoux's viewpoint.

My wacky whippet has an unnatural affection for toilet paper, preferably still on the role and mounted in its proper place. She knows this is unacceptable behavior, but like Roger Rabbit, she just can't help herself when the white fluffies are within her visual field. Last week I was at the sink shaving when she trotted into the bathroom, snatched the paper roll from its holder, and wheeled about ready to head out for delirious moments of frenzied shredding. But ... she saw me standing there in the doorway. She stared at me for about 10 seconds -- a long time -- then turned her head, purposefully dropped the roll into the open toilet, snorted, and pranced out of the room. Feelings? Other states of consciousness? That's as clear a "if I can't have it, you can't have it" as I've ever seen.

Life with a Whippet

2005-01-08T11:32:00.000-08:00

Wanna Play?

Thomas Barnett's Globalization "Rule Sets"

2005-01-08T09:21:00.000-08:00

The continual accumulation of experience (a polite way of saying "aging"), coupled with a voracious hunger to learn of fresh ideas and perspectives, does have a tendency to leave one a bit jaded as the years tick on. Every once in a while, though, a new view penetrates through the fog and calls all my gray matter to rapt attention. Thomas P. M. Barnett is one such.

During the holidays I was puttering around the house with C-SPAN on in the background, something I often do during the day. Out of the corner of my ear I heard Thomas Barnett talking about globalization, the "core" and the "gap", and his insightful perspective at making sense of the global situation, the stresses on world societies, and what he terms the "rule sets."

In short, don't wage war if you can't win peace. And he has a plan -- a remarkable one at that. Read about his ideas here: http://www.thomaspmbarnett.com/media/CFRBriefTranscript.htm

or stream the c-span video from this link: rtsp://video.c-span.org/project/ter/ter122004_barnett.rm?mode=compact. Enjoy -- it a fresh point of view that is remarkable for clarity, cogency, and importance.