Agile Documentation and Stake Holder Engagement

My continuing battle to heavily restrict documentation continues at my current place of work.

We are a very waterfall oriented business by the shear nature of the sector we operate in. We are not a software development company, we get oil and gas out of the ground. Exploration, building oil rigs, getting the resources out & selling them and cleaning up afterwards when there is no more resources left, by nature is very waterfall-ish. It requires big design up front.

Fortunately I work in the small project team where we have a reasonable amount of freedom for self governance. Since I have been involved (and most likely several months prior) the developers I work with and I have been pushing for an improved engagement and development process as the waterfall approaches has only ever failed to deliver. We have tried to implement more and more agile techniques with great (but isolated) success. TDD, DDD, CI and some aspects of scrum have been taken up primarily by the developers. Engaging the non technical staff has been problematic to say the least. This primarily revolves around Engagement and Documentation processes and the slow uptake and lack of interest of the non technical staff to process improvement.

Firstly is the personal disagree with the amount of documentation that is required for a project. I have no problem with this as I believe healthy conflict and the healthy resolution of those conflicts, usually results in a better working environment. My thoughts are: given we are a small projects team (I have been involved in 3 projects already, 2 of which are deployed, one is half done), I believe there should be minimal documentation and the code should simulate the vast majority of the detailed documentation.
The default documentation the developers have proposed is:

• A Vision Scope document: Define why we are even doing this project, the business outcomes and risk and very high level requirements
• Architectural design with design document if the design deviates from our standard web or smart client architecture. All integration points must be defined (i.e. SAP, JDE, Service buses, Web Services etc) and how they will be subscribed to or published to. The detail of this document is heavily reliant on the project itself. It could be as simple as a class diagram or a full blown very details design document.
• Use case/user stories. Definition of the business problem with the desired business functionality required to solve this problem. High level work item is probably broken down into several use cases. I personally don't care what format these are in; If a B.A. prefers one style over an other that is fine as long as all the necessary information is captured. One key aspect here is I do not want unnecessary technical information in this document. The person writing this document probably has a comparatively low technical comprehension when compared to the person delivering it. Don't tell me how to do my job!!!! If I ever see another proposed database table or stored procedure in a use case I will make the author eat it.

As far as documentation, that is it. The Vision and Scope is about 3 pages and if this can not be delivered then the PM/BA/Stake holder has no right to engage the team. Once these practices are agreed upon I would like to think that I wont even engage a project unless this fits our minimum templated requirements.

Architectural Design is done by the technical lead of the project. As we are lucky enough to have very skilled developers on our team (not that the company has acknowledge it yet) this is most likely done in a quick work shop with the PM, BA & SA. Other stakeholder may be invited, especially as we often work with other teams such as reporting. Their level of engagement is largely determined whether they are considered a technical owner or not. This work shop for 80% of our work will be done in about 30 minutes. Many of our applications are basic application with only a few integrations points that are well known. This document should be signed off by at least one other approved technical person.

From here use case and customer estimates can be done. I am still trying to push for an iterative approach, which is slowly sinking in. Typically we do 4 x 1 week sprints and release monthly. As the project moves on we may increase releases to fortnightly or weekly releases as functionality snowballs. This is a major benefit of a reusable architecture, reusable build and deployment scripts combined with Continuous Integration.

We are now at the point of Iteration zero. Our iteration zero should be about 1 day, including all of the interruptions we get. We have a custom software factory that allows us to have our infrastructure and application architecture standardised. This means with in about 5 minutes we can have a proven, architecturally sound application skeleton checked into source control and running off the build server ready for deployments. If only our non technical brothers were this organised... don't worry though, because we (the techies) have even written the templates for the Vision and Scope and Use Cases for them, all they have to do is fill in the blanks. To be honest I could probably write a Power Shell script to replace most of our non development staff...  ;p ^

AS for breaking down the actual work that a dev does, A use case will most likely be broken to multiple development tasks till each task has an estimate of less than 1 day, preferably .5 of a day. These tasks can be very briefly described eg

• create edit customer view- est 30 minutes
• create edit customer presenter logic & tests 45 minutes
• etc

Typically these will be described at the start of a sprint and added as sub tasks in the task tracker (eg TFS) by the developers so the PM has visibility of development progress. This is in no way an essential part of the documentation but I believe it aids in

• assigning responsibility (and therefore accountability),
• increases visibility of project progress
• Imp[roves estimation of what can be achieved in a sprint and
• increase ease of assigning bugs to people and to associated work items.
  

An excellent overview of agile documentation that is almost completely inline with my feelings of documentation is found here:
http://www.agilemodeling.com/essays/agileDocumentation.htm

Specially the following points:

  1.      The fundamental issue is communication, not documentation.
  3.      You should understand the total cost of ownership (TCO) for a document, and someone must explicitly choose to make that investment.
  7.        Documentation should be concise: overviews/roadmaps are generally preferred over detailed documentation.
  9.      With high quality source code and a test suite to back it up you need a lot less system documentation.
10.      Documentation should be just barely good enough.
12.      Comprehensive documentation does not ensure project success, in fact, it increases your chance of failure.
13.      Models are not necessarily documents, and documents are not necessarily models.
14.       Your team’s primary goal is to develop software, its secondary goal is to enable your next effort.
16.      The benefit of having documentation must be greater than the cost of creating and maintaining it.
17.      Each system has its own unique documentation needs, one size does not fit all.
19.      Ask whether you NEED the documentation, not whether you want it.
21.      Create documentation only when you need it at the appropriate point in the lifecycle.

We define our measure of success in term of production quality deployed software. For us as developers to move towards this we must provide a suitable engagement process for the non techies to follow. I believe the document outlined can be seen to be a bare minimum, but it is enough to deliver software. Any addition to this set of documents should be justified and be delivering an significant increase in business value; if not eliminate it. 

Too much documentation is a waste of time. Inaccurate or poorly maintained documentation is costly. Don't do it!

Recommend reading:

Software Requirements, Second Edition: Wiegers

Writing Effective Use Cases: Cockburn

Agile Project Management with Scrum: Schwaber

^ My disdain for the non technical people is not personal at all, I actually get on very well socially with them. I count myself lucky to work with a bunch of very nice people. What I don't like is the fact that these people are paid very well and I expect them to be not only competent, but experts in their field.  Unfortunately the developers are on a path on constant improvement; that passion is however not shared by our colleagues, which is a shame. We have very good developers running at about 30% efficiency as we spend too much time on non technical aspects of the SDLC.