Business Analyst Community & Resources | Modern Analyst

Agile Business Analysis: Interview with Scott Ambler

Featured

52835 Views

18 Comments

23 Likes

Whether you’ve never heard of Agile or you just finished your nth Agile project, you need to understand that Agile is here to stay! Are you, the Business Analyst, an extinct species in this new world? Is your career changing? Do you need new skills?

Agile guru and visionary Scott Ambler talked with Adrian Marchis, ModernAnalyst.com's Publishing Editor, and shared his vision on what’s next for Agile and his thoughts on the role of the business analyst in the Agile world.

Modern Analyst.com: First and foremost: What is Agile?

Scott Ambler: There are so many important features of agile:

It’s evolutionary in nature so it’s iterative and incremental.
Agile approaches are highly collaborative so we try to communicate as practically as possible.
We try to avoid documentation and prefer more effective means of communication such as face-to-face.
We’re self organizing, so that the people that do the work are the ones that are best suited to organize and plan it. That’s a bit different.
We do some documentation. We will hold some reviews but we’re really smart about it, and we focus on value.
Other important thing is the focus on quality. We are producing higher quality than we did in the traditional world. This is due to the focus on iterative approaches and factoring, testing and stuff like that.
We work closely with stakeholders resulting in higher stakeholder satisfaction.
And probably one of the more radical things, one of the more radical aspects of agile is that we welcome changing requirements. A change in requirement late in the lifecycle is a competitive advantage as long as you can act on it.

So, this is a bit different for a lot of traditionalists. We rethought things a bit and, as a result, we have higher levels of productivity and higher levels of success.

ModernAnalyst.com: One thing you mentioned that is common in agile is the focus on collaboration. Does agile work for projects that have distributed teams?

Scott Ambler: Yes it does actually. This is one of the things that we looked into in the Agile Adoption Survey.

We specifically asked about that and we explored the success rate in three different scenarios:

co-located,
near located, where you’re in the same building and you could get together on a daily basis if you had to; that might mean you need to drive but you could do it if you had to.
Then far located, it’s all plane to get there physically.

We found that co-located agile team had an 83% success rate. Near location team have a 72% success rate and far located had a 60% success rate. So, there is a risk premium the more distributed you become and the implication being that you want to throw better process at those approaches and better tooling. The real implication is that you don’t want to be distributed if you can avoid it.

And if you have to be distributed (if you have people working from home or on different floors in the same building), organizing an approach like that is significantly different than organizing a co-located one. So, this is something the project management community, really needs to pay attention to it because the risks are measurable and have been measured.

So, it shouldn’t be news to anybody. People are doing distributed agile.

ModernAnalyst.com: Another thing you mentioned is that agile is a very iterative process and that iterations are at a core of agile. So how does agile differ from RUP (Rational Unified Process)? Isn’t RUP also an iterative process?

Scott Ambler: RUP is iterative. RUP actually introduced and socialized a lot of the concepts that the agile community takes for granted like the focus on delivering working software on a regular basis, on iterations, on collaborations, on working iteratively, on delivering software incrementally, on testing throughout the lifecycle. So, it really sort of paved the way for the agile community. And RUP can be as agile as you want to make it.

There’re wonderful studies out there, case studies and books and stuff like that on how RUP can be agile. I was writing about how RUP could be agile long before I joined IBM. So, I think that’s an important thing to note. There’s a lot of knowledge out there on agile approaches to RUP.

And what I’m also seeing is that as the agile community is figuring out how to scale and do more complex things; that they are adopting a lot of the techniques that have been in RUP for a long time. It’s frustrating to watch them reinvent the wheel because they apparently don’t want to read about RUP. They like to beat up on RUP a lot. They beat up on RUP a lot but not realizing that there’re a lot of really great ideas … that they can take advantage of. But we’re an industry that loves to reinvent the wheel I guess, so I guess we have to do that yet again.

ModernAnalyst.com: How did you arrive at agile? Have you always worked on agile projects or did you start with more traditional approaches?

Scott Ambler: I’ve been all over the board so it’s interesting. I guess years ago, I was very agile not knowing it. Then I guess in the mid 90s, as I’ve gone to bigger and bigger projects, I started working in a more formal manner and got heavily into CMMI (CMM at the time) for a few years. But really sort of saw the formal approaches struggle. And beginning in the late 90s, began moving away from it. That’s actually one of the reasons why I became so interested in RUP… I was kind of looking at Scrum at the time, too. I was very heavily into process for almost two decades now, I guess. So, I was always looking for better ideas of how to work.

So, yeah, I’ve been on both sides of the fence I guess and can safely say that agile does appear to work. It does work better in practice than the more formal traditional stuff. And a lot of the stuff around traditional element isn’t really formal at all. It certainly isn’t disciplined. It’s more bureaucratic and I think there’s a lot of bureaucracy that gets touted as formalism and touted as discipline.

ModernAnalyst.com: So, you’re saying if I just use a template, it’s not necessarily disciplined, right?

Scott Ambler: Yeah, exactly. There’s actually a really great book out right now, that I don want to suggest to your readers… The book is called “Adrenaline Junkies and Template Zombies.” It’s a collection of patterns and anti-patterns for IT. So, one of the anti-patterns is template zombies; those are people who just fill in templates and making sure it’s filled out. Whether or not it makes sense, just fill it out. Whether or not it’s even the right template, but I have a template, I’m going to fill it out. It’s just a novel waste of time. A lot of wastage in general…. It’s a really interesting book and one of the really neat things about it … they present these patterns and anti-patterns just as a collection of one or two pages in length. … I suspect it will end up being one of those instant classics that we’ll be referring to 20 years from now.

ModernAnalyst.com: I noticed a couple of times you’ve mentioned something that a lot of the people in the traditional world would be surprised; and maybe even people from the agile world. Your survey respondents indicated that they actually used documentation?

Scott Ambler: Yeah. We ran this very detailed survey on how people are modeling and documenting in general. It wasn’t an agile specific thing. So, what we did was we started out with the usual, “Who are you” type questions. We also asked to choose a type of project that you’re going to describe throughout the rest of the survey. They’re given the choices of agile and traditional and iterative and ad hoc. The really neat thing is that it split out almost evenly. So, roughly a quarter, 25% plus or minus five we had in each of these four categories.

It was really interesting to see the habit when it comes to modeling and documentation. One of the things that the agile community is harping on for a few years now, is that agile is actually more disciplined than traditional. I have run the numbers exactly but just eyeballing things, it appears that many of the modeling and documentation practices you see on traditional projects, you also see on ad hoc projects.

One of the reasons why I run these surveys is I want to find out what’s actually happening because there’s a lot of religious discussion out there… And I use the term religion on purpose because people just have these religious beliefs over what works and what they should be doing. What is the proper way of working? There is no data to back it up, right? Particularly in a traditional community, there’s a phenomenal amount of theory out there, which has never really been backed up or if it was looked into, the numbers are basically show: oh-oh, that was a really bad idea to begin with. So, I’m trying to gather the numbers to find out what’s going on.

What was one thing that we found was that the agile folks are just as likely if not more likely than traditional people to write deliverable documentation like user manual, operations documentations, support documentation, system overview documentation. The true deliverable type stuff, which I thought was a bit surprising. So, I was thinking about this a bit; and what I think was going on is one of the things that we do know, is that Agilists are actually more likely to deliver than traditionalists. That type of documentation is something that you would write towards the end of the project as things stabilize. So, I think what’s going on is Agilists are more likely to get to the end, therefore, they are more likely to do the type of work that you would do at the end, which is finish up this documentation. That’s my theory at least. I can’t back that up with real numbers but that’s what the trend appears to…

I was surprised that Agilists were just as likely if not more likely to write documentation.

That’s not what I expected.

Article Pages

Page: 1 Of 4

First Previous 1 2 3 4 Next Last

Posted in: Career as a Business Systems Analyst, SDLC, Process, and Methodologies, Agile Methods

23 members liked this article

Featured

52835 Views

18 Comments

23 Likes

Why I am a proud Business Analyst

Product Owner vs. Business Analyst - Demystifying the Ambiguities

Strategy for Passing the ECBA Exam

The Secret is in the Wings

COMMENTS

ajmarkos posted on Monday, September 8, 2008 12:58 PM

I am all for "making decisions right now". Heck, once I have an adequate understanding of the as-is, that is a relatively easy thing to do. But I have to take the time required to learn the as-is; I lack divine intiution.

I may be hearing it wrong, but to me agile is about jumping right into the to-be - largely ignoring the as-is. Am I hearing correctly?

Tony

scottwambler posted on Tuesday, September 9, 2008 5:37 AM

If you visit www.agilemodeling.com and spend some time reading about the best practices, you'll see that it's common practice for agilists to invest a bit of time up front in requirements and architecture envisioning.

- Scott

ajmarkos posted on Tuesday, September 9, 2008 10:48 AM

Scott:

I have to admit, it has always been my experience that, especially on larger scale software projects, more than "a bit" of as-is modeling is required. The way I learned it and experieinced it is that coming up with an adequate as-is model is the major part of the required work (I was actually taught that the as-is model is ninty- eight percent (98%) of the required work ).

BA's often lead Business Process Re-engineering projects - very as-is-oriented efforts.

Tony

Tony

scottwambler posted on Tuesday, September 9, 2008 11:46 AM

In Agile Modeling, www.agilemodeling.com, we talk about doing just enough modeling for the situation at hand. So, just enough is situational. The larger, more complex the project the more modeling you're going to need to do. A good heuristic is that for every month of the release consider doing one day of up front modeling (i.e. for a 12 month project consider doing 12 days), with a cap of two weeks (10 days) for complex projects and one week for straighforward projects. Naturally logisitics and your organization's ability to focus on getting the job done and stay focused on high-value activities can stretch this out, but I would consider that a significant risk.

My experience is that the professional modelers among us tend to think that we need to do an order of magnitude more modeling than what is actually required. This is because they're often overly specialized on modeling hence overestimate its value, because they've been trained to think that way, and because they often haven't gotten their heads out of the theory of yesteryear. Having said that, the "extreme programmers" among us often think we need to do an order of magnitude less modeling than we actually need to do.

Bottom line is that if you have the skill to model something today, you will also have the skill to model it tomorrow when and if you actually need the details. The goal should be to get the value out of modeling which is to communicate and think things through without taking on the risks associated with too much detail too early.

In the November issue of Dr. Dobb's Journal I summarize the results of a recent survey that we did regarding the current state of modeling and documentation in IT. From the looks of it the agile community appears to be a bit more effective at modeling than the traditional community and a bit more likely to create deliverable documentation. I hope that the article will be an eye opener for a lot of people.

- Scott

ajmarkos posted on Wednesday, September 10, 2008 4:43 PM

Scott:

Thanks for the informative reponses.

Tony

Ivan posted on Friday, September 12, 2008 12:36 AM

Regardless of the name you apply to your approach to analysis, a characteristic of effective business analysis is clear communication. To that end, can you please clarify your fourth bullet point ("We’re self organizing, so that the people that do the work are the ones that are best suited to organize and plan it. That’s a bit different") which in its current form can be taken to mean either you advocate choosing the workers from the organizers OR that you advocate picking the organizers from amongst the workers. Thanks in advance

scottwambler posted on Friday, September 12, 2008 7:39 AM

Like I said, it's a bit different. The workers are the organizers. On agile teams there's a blurring of roles. We're moving away from overly specialized people, such as people who are just programmers, or just testers, or just BAs, to people who are generalizing specialists (see http://www.agilemodeling.com/essays/generalizingSpecialists.htm ) who have one or more specialities and a general knowlege of the process -- the sweet spot between being specialists and generalists. Generalizing specialists are more effective than specialists as they don't need as much supporting bureaucracy, they have the ability to apply the right technique to the situation to the right degree, and they can interact more effectively with others because they have a better understanding of the issues that they're trying to address. When you're just a specialist you're effectively someone who just has a hammer, everything looks like a nail.

So, instead of having someone do the planning for the team (i.e. the manager), the team itself does the planning. The manager may facilitate the effort, but the team does the majority of the planning work. When done on a JIT basis, this proves to be far more effective than traditional planning approaches.

You're correct that an important part of business analysis analysis is effective communication, arguably it's the most important part. But, we need to step back and observe that the traditional approach to analysis leaves significant room for improvement. The agile community has adopted several improvements, for the most part centered on reducing the feedback cycle and choosing more effective ways for people to share information (documentation, by the way, is the least effective method for people to share info). For several years now I've suggested that the business analysis community needs to point their skills back at themselves. I suspect that if they did that honestly that they'd start questioning the traditional strategies that they seem to prefer.

- Scott

Ivan posted on Sunday, September 14, 2008 9:43 PM

Thank you for directing me to the 'generalizing specialists article. Excising the circumlocution, your approach relies on an effective team. You pick (presumably expensive) resources who are skilled at more than one discipline. Ideal, that. I opine that such a team is by definition effective regardless of any particular method. In fact, left to their own devices, they may be relied upon to find and apply their own most effective methods, which may or may not be the methods you advocate. It is to improve the likelihood of success when the project team is less than ideal and/or operating in a less than ideal set of cirumstances that we codify and apply methods. The higher the risk, the more strictly we should enforce the methods which are, after all, insurance against failure. It strikes me as specious to promote Agile as a superior method if it requires a ‘better’ team than the alternatives, which themselves have evolved to maximize success even where the ideal team cannot be assembled, that is to say, almost always.

scottwambler posted on Monday, September 15, 2008 8:57 AM

My experience is that it's realistic to build teams from generalizing specialists, but you actually need to choose to do so. Just as a specialist team would be made up of some novices, some mid-level people and some experienced people so would an agile team made up of generalizing specialists. The novice GSs might just be specialists, the mid-level ones might be people with one specialty and a general knowledge, and the experts would have more than one specialty.

The deciding factor is that you need to build a team with a learning and collaborative culture going in, instead of building a Tayloristic team of specialists.

So, not having a group of experts on hand isn't a valid excuse for becoming more agile. It is, unfortunately, a common excuse for why organizations remain at lower levels of productivity. My advice is to choose to succeed.

Also, I've been pretty clear in my blog, and in other writings of course, that I certainly don't expect conditions to be ideal. Nor have I ever actually found myself in an ideal situation. So, not being in an ideal situation isn't a valid excuse to not become more agile either.

Strictly enforcing methods hasn't worked very well in practice over the years. Sounds good in theory though. You need to choose approaches which reflect human behavior, and as we know knowledge workers don't react well to being told what to do or to being strictly monitored. You might want to read the IBM whitepaper that I co-wrote with Per Kroll on Lean Development Governance which goes into this topic in more detail.

- Scott

Ivan posted on Monday, September 15, 2008 7:11 PM

No, thank you, I have read enough of the style, and in any case I will be busy searching for an example where “enforcing methods hasn't worked very well in practice over the years”. This could take some time.

scottwambler posted on Tuesday, September 16, 2008 5:15 AM

You're right, it's likely going to take some time because it's very rare for organizations to write up their failures and share them with the world. What's more common is that the problem gets ignored and then swept under the rug. Worse yet, many organizations will apply the same failed strategy over and over in the hope that somehow they'll get it right this time, being unable to observe the failure patterns that they're experiencing.

To see patterns of failure you need to work at a series of firms, something that consultants get to do but non-consultants generally don't. When you start to notice the same patterns over and over again you start to question what's going on. When you only see a one or two organizations struggle with a strategy, often several years apart, it's easy to assume that it's just bad luck that they both got it wrong.

You might find the results of the 2006 Data Management survey, http://www.ambysoft.com/surveys/dataManagementAugust2006.html , to be interesting because it found that two thirds of development go around their corporate data groups. Although this is a complex issue, it's clearly a sign that data management efforts are struggling in many organizations. Yet, when was the last time you read a case study about a struggling data management group?

Acadia posted on Tuesday, September 16, 2008 10:17 AM

Hi Scott,

I have been reading your articles for a while now and on Agile as well. I think Agile came in to place just to create a balance in a world that was relying too much on process and somehow forgetting the end user. I think Agile was a way to cut down the process drone and get some dynamic feedback mechanism into the picture. This is also clear in your article when you say one can choke oneself with all the modeling and documentation that no one will read.

Its also great to know you conducted a survey and you found out that Agile teams indeed focus and deliver modelling and documentation. Something that even you were surprised to learn.

And here is where I find the issue. Since politics is so much the talk these days, I can't help but say that throughout history individuals/ideologies comes to power in the hope of setting right and making better what existed, but end up doing some abject things themselves.

In the same light, I think the Agile community has to clearly advocate what it wants to do and there must be a absolute lack of ambiguity. Many people misunderstand Agile to mean no documentation. I think somewhere the founders did feel that way too and your "just enough" is not really a good enough term. Not having documentation is going the other extreme. Forget surveys and polls, I have found in my own company that after the product was delivered and shipped and the applause died down, people came back to fix something after months just to be utterly dumb founded and not knowing how to go about it. Or users in non-tech groups fumbling and cursing under their breath because no matter how good the usability Help does help many a times.

And when you talk of roles not being super specialized and planning being done by the teams, you have to have a precondition that it works in places where there is a good rapport and trust between the team. I cannot imagine Agile surviving in an environment where there is hostility backstabbing and an aggressive client. And there is no documentation to save your soul. So though your survey speaks clearly, I wish Agile community also understands the preconditions where Extreme Programming and Agile will work well to save a project and where it simply cannot. And I also wish that the Agile community mentions that documentation is really v important.

I strongly resent the fact that in life if you went about changing your mind, you would be booted, but in the software world, a client who changes his mind endlessly is accommodated. I need to understand yet how Agile even tags cost with all the changes and scope creeps with no change management and with documentation coming at the fag end?

scottwambler posted on Wednesday, September 17, 2008 7:31 AM

Some thoughts:
1. Software development is a complex and situational effort. As a result there is ambiguity in the agile message and justifiably so. One size does not fit all. Perhaps your need for definition is an indication that you may struggle to become more agile?
2. If people misunderstand agile to mean no documentation that is surely not the fault of the agile community, but instead the fault of those people to not do a bit of research. Just google the term "Agile documentation" and you'll find a wealth of articles on the subject. My Agile Modeling book, published in 2002, covered the topic in detail and the Agile Documentation book in greater detail in 2005. The topic has of course been addressed in other books too.
3. Just good enough is in the eye of the beholder. Just good enough artifacts are the most effective you can possibly create. Once again, software development is situational and one size does not fit all. I've described JBGE in detail at http://www.agilemodeling.com/essays/barelyGoodEnough.html
4. If your organization is having problems then you need to learn from your experiences and improve your process. If there is an actual need for more documentation then learn from that and invest effort in it on future projects. Agilists treat documentation just like any other requirement -- it's estimated, prioritized, and then implemented appropriately. Also, is the real problem lack of documentation or poor hand-off procedures? I've seen lots of orgs treat documentation like a band-aid over their release process. Instead of just handing a system off to a maintenance team, keep some of the developers around or bring some of the maintainers into the development effort. Also, were the teams writing high-quality code with a full regression test suite? If not, then I suspect that's your actual problem.
5. Help features, user manuals, ... should be treated like any other requirement (see above). If the stakeholders choose not to invest in these things then who is really at fault?
6. If you don't have good rapport and trust between the team you're in very serious trouble regardless of paradigm. With agile you'll likely fail quicker, which is a good thing. Papering over problems like this only extends the point at which you declare failure, increasing your actual costs as well as opportunity costs.
7. The agile community has been pretty clear about the preconditions for various agile methods. For example, the XP folks have been very blunt about this (once again, invest some time with Google) as have I with Agile Modeling (drop by www.agilemodeling.com and search the site).
8. People can and will change their minds. Get used to it, this is human nature. Furthermore, it's the customer's money, not yours, and if they choose to spend it in "questionable manners" that's their decision, not yours. If you resent things like this you either need to work through your issues or you need to find work in another profession more aligned with your value system.
9. The agile community is spectacularly clear about how to handle the cost of changed requirements, see http://www.agilemodeling.com/essays/changeManagement.htm for one of many write ups on this subject. We treat stakeholders like adults and ask that they be accountable for these sorts of decisions, it's their money after all. One of the advantages of agile, and one of the challenges I suppose, is that it provides greater control and visibility to stakeholders. The stakeholders can change the scope, control the budget, and control the schedule. When you deliver potentially shippable working software on a regular basis, that enables stakeholders to make concrete decisions on whether the system is ready to be shipped, whether they're getting what they want, and how much (if any) they should continue investing in this project. When you implement requirements in priority order you deliver the highest ROI possible, as defined by your stakeholders. When you deliver high quality working software, with a full regression test suite, on a regular basis you can accomodate change easily. Working in ths manner requires greater accountability on the part of stakeholders (clearly a challenge in many orgs) and greater discipline on the part of IT professionals (also a challenge in many orgs). We've upped the game in the agile world.
10. If you step back and observe what happens on traditional projects, most of the critical deliverable documentation is written towards the end of the lifecycle just like on agile projects. See http://www.agilemodeling.com/essays/documentLate.htm

Acadia posted on Wednesday, September 17, 2008 9:36 AM

Scott, thanks for your response and your time. I agree with your comments. The problem is when there is a lot of half-baked knowledge around and time is not invested in reading and researching. And I think practitioners and experts like yourself are doing a fabulous job of putting so much knowledge out there for people. There really is a need for advocates and enlighteners - who know the stuff to educate the people.

My worry is the misunderstanding of Agile must not undermine the role of a business analyst in the project sphere. Reducing our work to mere facilitation may not stand in the way of companies throwing out our role altogether.

Amar posted on Thursday, September 18, 2008 1:51 PM

It is a side question but can I ask anyone of you to shade some light on the type of questions asked on a BA skill evaluation test?

scottwambler posted on Friday, September 19, 2008 11:24 AM

Agile is real, it works well, it's here to stay, and it's growing. So, the BA community, and other communities for that matter, needs to accept this and find ways to add value on such projects. This will require them to be flexible enough to change. If not, they're going to struggle. BAs clearly have value to add, IMHO, but they need to choose to actually do so. The implication is that they will need to change the way that they approach their work.

jim4004 posted on Wednesday, November 12, 2008 3:25 PM

I think you've gone a step beyond the Agile Manifesto to say that Agile is against documentation. I don't see that at all, but rather that people and solutions are more valuable than documentation. There are many times that documentation IS part of the deliverable, and I don't see any conflict in that. Bureaucracy or not, if the business requires documentation, we should be delivering it, and why not deliver it in an Agile context? Many organizations believe that they cannot implement Agile because it's anti-documentation. They are legally bound to provide documentation. I'd much rather see the documentation as part of a sprint than a tired old traditional method.