Continuous Improvement - Bringing back UML

Continuous improvement is something we should all endeavour to pursue on a daily basis irrelevant of our chosen field. For me I am not a natural geek. I never built my own computer and never wrote my own gaming engine. I am much more interested in business than computers. What I love about computers is the way they can stream line our processes and, ironically, how unforgiving they are. They do what they are told, not you you meant. To me this poses a nice daily challenge and keeps my brain from turning to mush at work. Because i am not a natural geek I have to actively push myself to learn things with regard to IT.

When PowerShell came out a few years ago I looked at it in fear. I have never really been a shell guy. My 'nix mates were always playing on the green screens and I secretly knew there was a lot of power there but it was too much of a mind shift from the comfort of my drag-n-drop/intellisense laced world... But I knew it was a weakness and jumped in feet first.

Although i am still a complete n00b at PowerShell I can get what i need done with it. I am comfortable with the tools and have got real return on my learning investment. I got uncomfortable and it paid off in spades. I have even moved on to PSake and replaced most of my build and deployments steps to be completely PowerShell/PSake based, a feat I am pretty proud of. It has also helped immensely in me picking up Rails and Git, both of which are command line driven.

For me the next thing I think I really have to tackle is UML. I have dodged it for years.
I draw diagrams a lot. I use a white board every day and like to draw while I talk. I feel in mixed audiences it help get points across by using multiple forms of communication at the same time, voice body language and diagrams. However I am communicating I should be adhering to common practices where possible (as my last post pushed), when there is a common language, you should use it.
For diagrams in nerd world its all about UML.

My dislike for UML stems from my dislike of concrete documentation. I have inherited too many projects that have had documentation that did not marry up to the implementation, with UML typically being one of the worst offenders. It is incredibly frustrating and most importantly wasteful; wasted time, money and trees (paper has to come from somewhere)!

So I sit thinking that perhaps I was throwing the baby out with the bath water; UML is not bad, inaccurate documentation is bad. If I am to communicate with diagrams then I should use the common language. I also understand there is a time to be pragmatic and a time to be dogmatic. Most of the time a couple of boxes with some lines joining them will suffice but the ability to step up to the more accurate implementation is a worthwhile ability.
So my UML journey begins... or at least is reborn.

Common language and Patterns

Imagine if craftsman from other industries called everything basic patterns they used they did by different names. Instead of a butterfly stitch a doctor called it "the sticky skin healer" or a carpenter calling a dovetail join "the zig zag corner"... it just wouldn't fly.
Well unfortunately this is not common in our industry, it is the norm.
Firstly I want to define what "our industry" is:
It is not computer science.
It is, if anything, Information Systems.
We push data around, make it easy for users to persist and transmit data so systems or other users can make decisions, business decisions. All of this is intended to some how improve the bottom line of the client and/or the company you are working for. People forget this. Our job is 99.999% of the time about making money. The Wiki on cucumbers git hub site has a good blurb about defining features and asking "why?" with regard to implementing features, you should get to the following reasons:

• Protect revenue
• Increase revenue
• Manage cost
• Increase brand value
• Make the product remarkable
• Provide more value to your customers

Now this is clearly for a product; I have no problem with team defining what the underlying drivers for their project, department or business are but being able to see how a feature helps enhance that is important.

Anyway... back to language.
Language and the words we use are very underrated. I think it is possible that, in our industry, communication skills are so rare (hey, we are nerds!) that we trivialise the importance of the words we use. Evans makes it very clear that the ubiquitous language is a key aspect of DDD and the same can be said about whole industries and the language used by the individuals that make up that industry.
This is why patterns books are popular. If I want to have a discussion with a developer at another firm to bounce questions off them, which I frequently do, if we do not share the same language then we spend half the time getting our terms aligned. There is a way to avoid this: understand the common language!
How? Well, I'm sorry but you are going to have to pick up a book or two... OK that's a lie. You are going to have to read a crap load. Hey, we get paid well, I expect senior developers and consultants (again, in our industry) to have read the books I am about to describe.

This list is an ordered list of pattern books, it focuses on patterns i.e. a language based term to describe a common approach to solving a problem. The language is as important as the fix

• Refactoring - Fowler
• PEAA - Fowler
• Head First Design Patterns -or- Refactoring To Patterns
• APPP - Uncle Bob
• Refactoring Databases - Ambler
• GoF
• XUnit Test Patterns - Meszaros
• EIP - Hohpe
• DDD - Evans

There is also one I really cant wait for and that is the forthcoming Presentation Patterns by JDM; which should round out the whole stack.
To be honest this reading list would take the average developer that has a family and a life about a year to read. I'm sorry. Suck it up.

The order I have presented this list in is very deliberate. People will argue that GoF should be first. I heartily disagree. Gof is a hard book to read, with out having the understanding of refactoring and the softening blow of "Head First" the book is too much for the average nerd. I know this because it was my first patterns book. I have read it 4 times and only the last 2 time I have actually got anything out of it.

Refactoring
I feel this book is a minimum in a developers arsenal of pattern books. With this you at least can understand why resharper is doing what it is doing. It defines some basic terms that are low level enough that it is applicable to most developers irrelevant of the layers they work in. Some common terms like parameter object are introduced to the developers language

PEAA
Although he book references a metric tonne of other books, I think its aggregation of basics is its strength. The 2 book approach is great and the patterns introduced are very basic and high level enough that the are able to be picked up quickly. Read in conjunction with conversations with an experience developer that uses these patterns regularly and the developer should be very familiar with these patterns very quickly. This is a great book for Book Club type discussions too (hint hint @wolfbyte)

Head First Design Patterns -or- Refactoring To Patterns
These two books are a softer introduction to lower level patterns when compared to the godfather of patterns books, GoF. These are much easier to read and depending on the developer one may prefer one over the other. Ideally read both but read at least one of these prior to continuing down the stack. You may find it odd that the list starts with a low level pattern book (Refactoring) then a high level book (PEAA) then something in between. The pattern described in HFDP and RTP are more abstract patterns that require much more thought than the high level patterns in PEAA eg specific patterns like Table Gateway or ActiveRecord. I also find it easier to show what tool implements the PEAA pattern, eg Rails ActiveRecord or Castle Activerecord, NHibernate as DataMapper (all as examples of Lazy loading) etc

APPP
This book is great at helping developers step up to actually writing clean, readable, maintainable code by showing the patterns that you can use on a daily basis. S.O.L.D.I S.O.L.I.D is introduced to the reader. Unfortunately SOLID has just become another rote learnt acronym/pattern that is infrequently applied. I have found this does however allow code reviews to now have a common language and not just "um this code looks ugly" type code reviews, which i have been guilty of in the past.

Refactoring Databases
This book is in here by necessity. I doubt many .Net developers would get much from this. We have been force feed SQL drizzled in a Table sauce and Views jus and a side of Stored Procs for years. We know Databases, at times, it feels too well. However if you are from a non data base driven framework then this is a necessary read. I'm talking to you Ruby and Java kids. You lucky bastard get ORMs by default. In the twilight of the SQL age this book may seem a bit legacy but if you are truly and enterprise developer you will be dealing with SQL for years to come. I'm sorry, I really am ;)

GoF
Ahhh. The serotonin of pattern books. If you have insomnia this is for you. Its a great yet boring read, hmm what a paradox. This is just developer tax, its rite of passage, a necessity. Please note that just because a pattern exists it does not mean it must always be used, that it should ever be used or that it is not hard wired into you language. I'll let the reader guess/figure what I'm referring to.


If you have got through all of these books in my mind you have done you due diligence and are now allowed near an IDE.. OK I'm just kidding, but seriously I believe all of those previous books are minimum requirements for some one who leads a team of other software developers or consults. Below are the one you should continue on to if you want to be an above average developer

XUnit Test Patterns
By this stage you should be testing you software. The XUnit book, although not my favourite Testing book (that goes to The Art of Unit Testing) it is the best in describing the patterns of unit testing. Understanding these patterns will help you help other learn how to correctly add TDD to their skill set and firm up the teams language around key terms like doubles, fakes, stubs, mocks and spies; all things that are commonly misunderstood

EIP
This book cover various messaging patterns that relate to integrating system. Ideally you should read this book prior to using any ESB type framework like NServiceBus or MassTransit. If you have read this book and are using these patterns without having read or understood the previous books I fear you may have a big distributed mess on your hands. This is a great book but IMO this system you would tend to build using these patterns are most like system that should have been built on the patterns from the other books. If you do any integration you owe it to your self to read this, which means unless you are just doing the integration, reading all the other books first ;)

DDD
My favourite and I believe the most misunderstood book of the lot. This book is not just a code book it a process book. It raises the issue of language and introduces the notion of a ubiquitous language that the team including the stakeholders/SME/Users understand.
I believe this book has pushed what I can produce for a customer to the next level. The code feel more aligned with the business as opposed to just getting a task done. The closer code matches business processes and concepts the easier it can change with the business too.

So there it is. My list of pattern books I assume a Tech lead or Consultant has read, understands and implements. These are not the be all and end all of pattern books but I believe they cover the bases. Please note there are a suite of other books that are not pattern books that I believe are also "essential reads", but that is perhaps another blog post.

Would love to hear feedback,
cheers Rhys

NoSQL & Document databases

I am really loving the NOSQL movement at the moment, however there seems to be a lot of confusion as to when it is appropriate to use them.

Significant things to considered are:

• Atomic transactions are only for single operations. You can't really have long running or nested transactions (ACID rule a different).
• Joins don't make sense so don't expect to use them when designing the system
  
• Document databases are typically schema-less. This means adding new properties is much less of a hassle than in SQL-land, especially once you are in production.
• The notion of an aggregate root fits perfectly with a document so the idea of using a DocDB for DDD is appealing (assuming transactions are not required)
• DocDBs tend to scale horizontally very well unlike our SQL counterparts which tend to only scale vertically without huge headaches
• Read and write performance is possibly the opposite of what is expected with very fast writes (I understand) being the norm.
• Queries are done differently. MongoDB for example uses JavaScript as it query language (this does not mean it is used in the web tier!)

Several uses for document databases come to mind:

• High volume, low value writes; eg user data entry on social sites; this is not business critical but potentially requires easy scaling options; ie no one is going to die if you last Facebook update doesn't go through to all the DB servers instantaneously.
  
• Auditing; One area I'm keen on is command persistence. I like the idea of having a trail of all command sent to a component, it becomes a self documenting timeline of what users were trying to do to the system. When a command is handled by a component it can just write the whole serialized object to a DocDB, thereby capturing all the info without being bound to a schema (audit is version agnostic). The command can then be processed by the component. I will admit that an Object DB is also suitable for this.
  

Things not to use Document Database for is high value transaction heavy stuff i.e. banking transactions or thing that inherently require SQL... whatever use case that may be.

Hope this helps :)

Help Me Help You

We have been lucky here in Perth that we have a very active community, well run by people who have stepped up to the plate to provide us these events. However these events are not free. We need venues, we often get prizes and some sweet swag. I don't think some of our attendees quite understand that without that support we have no event.
We are also fortunate enough to have sponsors we actully like! For this reason when we make a plug it not just because they are a sponsor but because we either use the product or service (or want to) ourselves. As our audiene is largely made of business developers I thought they would understand these basic back scratching processes.
Anyway, below is a list companies or products that have made many live better by helping out a technical community I'm involved in AND I recommend professionally:
JetBrains-ReSharper, Teamcity
TekPub
Readify
Beacon
Excom
Redgate

Thanks. I like your stuff, recommend you and thank you for helping us out. Also want to thank Mitch Wheat and Mike @wolfbyte for organizing us presentations every month; you are appreciated!

Testing with Domian Driven Design

DDD is a hairy beast. It feels like everyone has a slightly different opinion of what it is. I personally thought I had a pretty good grasp on it until I caught up with Udi Dahan earlier this year where he (as per usual) completely turn my understanding on its head.
That being said I believe the way I am approaching the current project is in line with what I believe the majority of people believe DDD is, correctly or otherwise.
Key aspects in this project that help make it resemble a DDD domain are:

• The level of understanding the devs have and the ubiquitous language that is in use and they way it is constantly evolving amoungts all team memebers, including our SMEs/Users
  
• The general structure of the code - repositories, services, aggregate roots comprised of entities and value types etc etc
  

To be honest one could argue, that like most DDD projects, it is just clean OO coding. It is; that doesn't mean that there isn't something that others could glean of of the project, such as what we are doing to keep things clean and help with creating nice tests.

First and foremost follow TDD and BDD if possible
If you have a good understanding of the component you are building then writing explicit specifications should be easy. Do this with your BA and SME make sure the tests are not shallow low value tests. Investigate what BDD is and see if one of the many framework fit your teams needs. Personally I am still using xUnit frameworks and am creating elaborate contexts on which I make assertions. My fixture set ups can sometimes be complex but my test methods are very clear & clean and can often just be one line assert statements. I have not found a BDD framework that sits well with me in the .Net world like Cucumber does in my Rails development, so I will continue down this path till I find something I like better than explicit and somewhat elaborate xUnit styled tests.

The domain should remain as pure as humanly possibly
Anything that is public should have a good reason for being so. Unfortunately most developers use public as their default. This is bad practice as it produces a very non intuitive API. Use access to help show the other developers how you intended the API to be used.
Typically, IMO, this means protected virtual by default. (I'm an NH fanboi). Also restrict what you return. I personally don't like the idea of returning child entities from parent entities especially aggregate roots. What is the consumer going to do with these entities? Most of the time a value type or projection is more appropriate and helps keep the line of ownership clean.

Modifying the domain to be more testable should be frowned upon, especially if it confused the API
This ties into the first point. NB : There are concession that I make that I can live with (I will cover them soon) but I believe these do not negatively affect my domain and make life easier and my intentions more explicit to the next developer

Use subclass fakes to increase accessibility for testing
I often see people making fields, properties or methods public so tests can call them. Please don't do this. If these are private definitely do not expose them. This is an indication that you are not doing TDD. Private methods generally only come from refactoring once your test pass so testing private methods is a sure sign you are not doing TDD. Internals sometimes may warrant being tested. This is OK as we can make internals visible to the test project. I generally don't mind doing this as I put this in the AssemblyInfo file which is recreated in my deployment anyway so there is no dodgy test orientated code in my final deployed assembly. I must say that I don't really do this a lot and i think people use this as a crutch. True TDD generally will not warrant a lot of this, However i find myself doing it a bit in my domain, possibly out of fear more than any really rational reason.
If you really need to make something accessible to test and don't want to expose it fully then create a fake that inherits from the class and override the accessor in that class. Sometimes I do this to check things like IDs. Again question whether this is really needed, have faith in the TDD process!

Allow hooks in the domain for creating child objects
One thing I have used in the current project is protected factory method to create child entities. By opening up these one liners I can override the method in a fake sub class to create a fake subclass of the child: eg OrderFake will create and return an OrderLineFake in its CreateOrderLine method as opposed to the Order creating and returning an OrderLine in its CreateOrderLine.

When using fakes of real domain objects make sure you are not hiding any of the real object functionality. The fakes should be as plain as possible. Adding additional logic will surely corrupt your tests. One thing to watch for is to make sure your fakes implement the same ctors as their parent and call into those ctors. Failure to do this will create a big PITA :)

Next post I will talk about some issues and traps that we as a team have managed to fall into. Most of them have been things when consciously question know we shouldn't do but have managed to creep into the solution, hopefully we can help you avoid repeating our mistakes!

What is Regression Testing?

We are on a journey. The company I work for is a very large one and has suffered some lag in migrating to modern ways of software development, a situation many companies of this size often find themselves in. Fortunately the team I work with have come together to make a conscious and consistent effort to improve our area of influence. First and foremost was the introduction of a structured developer oriented testing plan. Because we are a development team we started with unit testing, integration testing and some build scripts to run these tests. From there we have improved they way we do those tests, streamlined our scripts and made CI a solid part of our process. Automated deployments have become easier too leading to automated staged deployments. We are now at the stage where we are adding in Fitness and UI testing to the equation.

Each of these step have been driven from the ground up. We have had to, carefully, maneuver these processes into our daily process and show the benefits to management. It has taken time but they have responded very well; so well that they are jumping on board! This is a great thing and has made all the hard work, and it was hard work, worthwhile.
However there are still political games to play. The language is fractured and management want to throw around terms that they don't really understand and force their idea of these terms onto the very people that introduced them to the department, sound familiar? ;)

"Unit Testing" and "Regression Testing" are two of the terms du jour. Unit testing "means" something a developer does and regression testing is an automated UI test that a non techie can watch as it magically clicks away at the screen. This, as you may know, is not correct... well not strictly correct; and in a place where the language is so commonly fractured it is hard to justify fighting for seemingly such trivial victories.
So instead I vent here.

The things that i felt i needed to express most importantly are:
Any automated test is a regression test
- and -
You do not write regression tests.
What i am referring to in that last statement is merely the fact that you write a repeatable test that tests a certain unit, module or scenario, that can be automated and then your check it in to source control. The very notion that it is part of your build process/continuous integration means it now acts as a regression test.
So our tests that

• Test a unit, i.e. an xUnit framework which may also uses mocks and stubs etc
  
• Testing seams or integration points i.e. DB tests, file systems tests, web service tests etc
  
• System smoke test
• High level acceptance tests using tools like cucumber, fitness etc
• UI testing - i.e. a button clicking framework like QTP

are all regression test because the are repeatable, automated tests.
Please don't make the mistake of thinking you will explicitly write a regression test. The test should test that the code produces a specific outcome e.g. enforces a business rule.
The automation and maintenance of a healthy CI environment makes those tests regression tests.

I guess the key here is really your CI process. If the tests can be dodged or accidentally not run somehow then this will need to be addressed.
Make the pit of success an easy one for you team to fall in to and let them know regression tests are a side effect of good daily habits.

InfoQ

It still surprises me that people are not aware of the great resource that is InfoQ. If you are reading this post from my web page then there is probably a big InfoQ link right in front of you. If you are like the majority then this is probably a feed and you cant see it! Anyway its a great site that I have as my home page that gives a running report of the happenings of our industry. It doesn't say anything about the latest Intel processors or have Dilbert cartoons its just enterprise development and architecture, specifically .Net/Java/Ruby, SOA and Agile; Basically stuff I am interested in on a professional level.

Go check it out : http://www.infoq.com