Agile Business Analysts?

I have recently been fleshing out my thoughts on the role of business analysts in an Agile team. This is something I have historically addressed on a case-by-case basis with clients, but when thinking about it last week at a client site, I realized that I do have a general take on how the role and responsibilities of a business analyst change when a group moves to using an Agile process. Per the usual, I’ll be using Scrum as the example process where needed.

The Classical Roles

Historically the business analyst has had two primary roles. In a process where cross-functional communication and collaboration are minimized, these roles are essential. What little communication that takes place between the business and the software development teams purporting to serve them passes almost exclusively through one or more business analysts. I view the business analyst in a phased development approach as having two roles; Translator and Gatekeeper.

The Translator

The business analyst is primarily tasked with taking the needs of the business and translating them into a written document that is then handed to the software developers and testers. This role’s necessity is based upon some fundamental assumptions. First, programmers and testers are on the whole, too socially retarded to actually talk to the business persons and find out what they want. Second, the business is primarily viewed as being unable to focus its thought long enough to tell the programmers and testers what they need. Since a key value of an Agile approach to software development is to have individuals and their interactions be the driving force rather than processes an tools (see The Agile Manifesto), the obliteration of this role is a primary objective.

The Gatekeeper

Business analysts are also viewed the gatekeepers for requirements in classical, phased software process approaches. This is a somewhat unfair role, since they rarely have the authority to prevent business or technology changes, like a meter maid trying to stop an armed terrorist. It’s even more unnatural since change always happens, so it’s like the meter maid being assigned to an anti-terrorist unit with no additional training, authority, or weapons.

The Agile Roles

Some of the more extreme practitioners of Agile methodologies say that business analysts have no role in an Agile team. I disagree, provided the role of the business analyst is allowed to evolve. The roles I’ve seen them take on as valuable members of an Agile team are Facilitator, Historian, and Journalist.

The Facilitator

A high degree of collaboration and interaction take place between business and technology in Agile Software Development. The business analyst can serve a critical need in these interactions, facilitating communication between business and technology and making sure that critical areas are being covered in an interaction. This can be a more fulfilling experience for an analyst, since they often talk to one side, then go to other to pass on the information, all the while thinking, “Man, this would be simpler if you were both in the room with me at the same time.“

The Historian

The business analyst’s documentation skill is excellent for capturing significant information that often gets lost in a team of purely technical people. I have seen some great uses of business analysts in this area. One is to document the system that the team actually builds (not the big, up-front imagined system that is covered in a phased design stage). Another is capturing key technological and architectural decisions and the context in which they were made, so that when a group revisits certain items asking, “Why in the world did we decide to do that?”, they have the means to be reminded or informed why a particular path was chosen.

The Journalist

The business analyst can also be the team’s journalist, making sure the latest information makes it out to all interested parties. One creative approach I’ve seen at a client site is a business analyst who has a project blog. He posts entries after each meeting between the business and technology, documenting what new user stories were created (even scans in the story cards!), a summary of the discussions held, and documents any key decisions that might have been made. Oh, and he provides an excellent executive summary at the top that’s suitable for anyone’s review, right up to the CEO. Tell me you wouldn’t love to have that guy in your Agile team.

So I finally got around to checking out the TED site; I’ve quickly become a fan. One of the first talks I watched was Steven Levitt’s child carseats talk. Both the talk and the feedback comments on the TED site reminded me of things I see in software development. Here’s the video if you haven’t seen it.

Steven Levitt shows in his talk how 30 years of data on car crash fatalities imply that carseats do not outperform regular seatbelts for children ages 2 and up. Anyone who has a child or grandchildren will probably bristle at hearing that; as the father of a pre-schooler, it certainly gave me pause. We spent no small amount of time and decent chunk of money in selecting carseats for our child, thinking we had done our best to ensure our child’s safety. For that matter, it would be illegal for us not to have done so. To see a decent-sized data sample suggesting my child would be better off in a seatbelt at her age is rather unsettling.

By the end of the talk, I took away these observations:

• This great swath of an initiative was started by a very small yet vocal group of proponents whose assertions received little scrutiny.
• A solution tailored to the specific constraints of a portion of the audience has been put forth as the de facto solution for the entire audience.
• There is an ongoing assumption that the more complicated and costly solution must be superior to the simpler, existing solution.
• Cursory evaluation of a data set without rigorous attention to mitigating factors can lead to wildly inaccurate conclusions.
• The acceptability of a given solution is often tested against a narrow band of the overall criteria, and the solution is often optimized to pass that test.
• The likelihood that people will continue to choose the more expensive and complicated solution despite any data is high.

So, software professionals, does any of that sound familiar? It reminds me of numerous initiatives over the years that have led us inexorably to the software productivity morass we have slogged through for years now. I suppose the most heinous case study in my own experience would be J2EE and the insistence that it was the solution that any reasonable business application would choose for a platform. (Before you .Net folks jump on that one, DCOM and later the .Net enterprise stack was much the same.) And who hasn’t read a benchmark or white paper with seemingly incontrovertible data depicted in highly-polished graphics insisting that Product Y is the one solution to address them all?

I noticed that there were comments below the video on its page at the TED site, largely because the negative verbiage of the topmost comment jumped out at me. I took the time to read through them all (there were 36 at the time of this writing), and to my amusement, I saw parallels between them and how people react to questioning and examining our existing practices and means of developing software. See if any of these strike a chord:

• The idea that what we’re doing might be wrong unsettles some people, and their response is often ferocious and irrational defense of the status quo.
• Those who have a financial stake in the current mode of operation are less likely to be open to scrutinizing it.
• There will be some people whose primary objection to the scrutiny is that they didn’t think of doing it first; these people will sometimes attack the person questioning things on the grounds that it’s all a selfish, attention-mongering endeavor.
• There are some people who will also be open to examining why and how we’re doing things; whether they conclude the same thing or not, they are a welcome presence among the larger mass of those who ardently refuse to entertain the possibility of a need to change.

In recent years I’ve been greatly encouraged by the willingness of companies to question whether or not the heavyweight frameworks and technology stacks are what they should be using. Helping companies slough off high-ceremony processes in favor of right-sized process that focuses on delivering the right software in a timely manner has been some of the more rewarding work I’ve done the past several years. I think we have an encouraging number of people in the industry who are challenging the “more expensive and complicated always equals better” mantra.

For the brave individuals willing to put these questions to the community at large, I hope you find some comfort in knowing that the resistance and rejection you will encounter is a thing to be expected; you are not alone in that respect. Here’s hoping we continue to be open to self-examination, no matter what emotional responses it might provoke or what fear of the future it may stir up within us.

…Then they stopped, and Thorin muttered something about supper, “and where shall we get a dry patch to sleep on?”

Not until then did they notice Gandalf was missing. So far he had come all the way with them, never saying if he was in the adventure or merely keeping them company for a while. He had eaten most, talked most, and laughed most. But now he simply was not there at all!

“Just when a wizard would have been most useful, too,” groaned Dori and Nori (who shared the hobbit’s views about regular meals, plenty and often).

- J.R.R. Tokien, The Hobbit

I am very fond of the works of J.R.R. Tolkien; it would not surprise me to find that many of you recognize these words from the second chapter of The Hobbit titled “Roast Mutton”. It occurred to me recently that there are parallels between Gandalf’s role in The Hobbit and that of an Agile coach. Now, before my fellow Tolkien enthusiasts leap on their keyboards, bear with me on this. Know that I am not saying an Agile coach is on par with a wizard (OK, with one of the Istari sent by the Valar, but let’s table that so as not to scare off the normal folk, alright?); that should be enough to calm you down.

In the excerpt from The Hobbit at the top of the page, Bilbo and the dwarves have run into their first predicament. Note that it’s not a particularly difficult situation; they just need to find shelter and partake of some food. Fire would be nice, too, if it could be managed. (Mind you, it’s the first day of their journey; they set off mere hours ago fully provisioned and riding on ponies.)

It isn’t very long before the fledgling group finds itself not only without shelter and food, but in the hands of three rather nasty trolls who are deciding how best to eat the entire group. Gandalf returns once things have gotten out of control, and with some subtle adjustments to the situation, the crisis is averted.

Could Gandalf have come back sooner and spared them this entire experience? Perhaps, but in their struggle a few key things happened. First, the group had to work out how to assess tasks at hand and appropriately delegate. To their credit, that effort was partially successful. The most skilled firestarters were assigned to that task, one of the keen-eyed dwarves was assigned to be the lookout. Second, they gathered some field experience that led to the establishment of improved practices, i.e. don’t leave the ponies laden with packs when you make camp, particularly if it’s all your food. (They lost most of their food that night when the pony carrying it bolted and ran straight into a nearby river later that night.)

Third, Bilbo Baggins was called upon to perform his first task as burglar, albeit a fool’s errand that landed them in the troll predicament. While Bilbo was wildly under-equipped for his job, he managed to work through it. That experience began a developmental journey that would prepare him for the great things that would be expected of him later on.

This was not Gandalf’s first adventure, nor was it the first group of people he needed to equip and challenge in order to develop them to a point that they could accomplish their goal. At this point, he’s been in Middle-Earth just shy of 2,000 years. He would have been more than capable of walking them through their entire journey, but to what end?

Agile coaching is a discipline that aims to help teams develop their own use of methodologies like Scrum and cross-methodology practices like testing, user stories, etc. This means equipping teams with just enough information to strike out on their own for a bit, then letting them run with it rather than dazzle them with one’s own mastery so as to appear like the indispensable demigod. Until people struggle with the terse maxims of Agile Software Development, they cannot internalize them. And when the wizard is always around, why bother struggling?

One of my greatest frustrations as an Agile coach is how few companies are willing to take a coaching approach with their Agile adoption. They want you to come in and be the demigod as a full-time contractor. Sure, there are fiscal, political, and seemingly practical reasons that they will cite; I chalk most of them up to being excuses for a lack of willingness to embrace what it would take to face the hard task of nurturing what you have. It’s seemingly easier to just throw more money at more bodies and hope that somehow things will all work out, and surely if you can stumble upon some superstar to wrangle the mess, you’ll eventually be able to browbeat them into becoming a full-time employee.

Don’t get me wrong; in some ways, I benefit from this dysfunction. From a selfish business standpoint, having a single full-time client is certainly easier than juggling multiple concurrent clients and their schedules. As far as actually accomplishing the aim of my business, however, I think it hinders the mission.

One of my aims in working with companies is to be a coach despite being brought in as a contractor. It’s certainly possible, though it is more challenging. There’s not that natural separation of the coach from the team that forces them to take up the mantle on their own. Few things in my work are more rewarding than having a developer come to me and say, “I wasn’t really sure this Agile stuff could work, but now, after going through all this, I wouldn’t want to go back to the old way of doing things.”

Much later in the journey of the hobbit and his dwarf companions, Bilbo is again called upon for a challenging task. His response makes me think Gandalf’s approach has worked:

“…Perhaps I have begun to trust my luck more than I used to in the old days” – he meant last spring before he left his own house, but it seemed centuries ago – “but anyway I think I will go and have a peep at once and get it over. Now who is coming with me?”

- J.R.R. Tokien, The Hobbit

Here’s hoping more people will be willing to embrace the challenging, messy, and altogether beneficial task of equipping teams and allowing them to struggle when necessary, even if it means the occasional scuffle with trolls.

The Why is Agile hard? conversation I convened at this year’s Java Posse Roundup made its way into the Java Posse podcast feed. That was an enjoyable session with discussion of the challenges groups face when adopting Agile.

NOTE: The astute reader will notice that this entry has changed. My blog was compromised and the perpetrator substituted the original post with links to questionable material from faraway lands, and I don’t mean overripe strawberries from the west coast.

(This is part two in a series I have titled “Soft Skills in Software”, which came out of points I came up with for a panel discussion at CodeMash 2008.)

When introducing change in a technology practice, you can save yourself some frustration by being aware of the “sales cycle” for new ideas within your group and maintaining a perspective that always bears the long-term in mind.

As technologists, we often have great enthusiasm for new ideas that we perceive to hold value in terms of their (potential) positive effects on our work. The value is so apparent to us that we often see no reason not to put this new idea to work right away. However, the speed with which a new idea can be adopted is affected by a number of factors. Let’s start with these:

• The organization’s general disposition toward change
• The change agent’s standing as a thought leader within the technology practice
• The resultant impact of adopting the idea in question
• The temporal context in which the new idea is introduced

Depending up on the value of each of those variables (and more), the length of time it takes for a change to be adopted can be anywhere from a few weeks to months to years — or a special category I have, “not for the foreseeable future”. I use that category instead of “never”, because there have been times when I thought there was no way a given thing would ever happen, and suddenly a merger or regime change or some other major external change suddenly opened the door for the previously-infeasible.

I use the term “sales cycle” because it was in a previous career of mine where I first learned this lesson. I used to sell industrial packaging and marking systems, and when I first started, I was greatly frustrated at the seemingly low number of sales that I would close. I would follow up with some clients to the point of irritating them, convinced that “something should have happened by now”. I eventually learned that the different types of systems that I sold had different sales cycles. Because of a given system’s cost and the role it played in the manufacturing process at large, there was a minimum amount of time it took to get approval, move through a cusotmer’s procurement process, and finally issue me a purchase order. Some systems took 4-6 weeks to sell, while others were 6, 12, and even 24 months. In general, the longer the sales cycle, the bigger the sale. Once I understood this, I found myself far less frustrated, and there were fewer of those awkward interactions between my customers and I.

Introducing new ideas to your development group is quite similar. There is a direct relationship between the level of change and the time it takes for a team to agree and then adopt the idea. I have witnessed more than a few programmers who had good ideas that they presented and expected everyone to just approve right away. Their frustration with what they perceived as stalling and/or rejection typically resulted in undesireable outcomes. Even the most-valued technical person is only indulged a few temper tantrums before being labeled as “damaged goods”. Recovery from this sort of thing within an organization is discouragingly rare. The person is usually avoided in discussions about strategic direction, and their ideas are usually dismissed or stifled. In general, if letting person Y talk about changes has resulted in discomfort in the past, the typical solution is to just not let that person talk about changes. It’s warped, dysfunctional, and counterproductive, but that’s mainstream corporate culture for you.

One area where technical change differs from sales is the possibility of incremental change. It’s not always possible to sell only a piece of a large system; changes in software development, on the other hand, are well-suited to incremental change. Being able to adjust one’s expectation from demanding a single big win to being content with a series of methodical, steady, small victories is a valuable skill in software. For some big changes, that can be the only way to introduce them. For example, a company may not be open to switching to Agile Software Development overnight, but having daily stand-up meetings and developing in 2-week iterations certainly seems benign enough.

Having said all that, there have been times where something with a 12 or 24-month sales cycle was adopted immediately for me. Urgency, crisis, severe failure with the current situation, any number of things can result in exceptions to the rule. The art is knowing when to spot these factors and acting appropriately to leverage them. If I could sum all that up in a blog post, I’d probably be in a different line of work ;-).