Tuesday, April 27, 2010

Begining with the end in mind

A long time ago, what feels like a previous life, I used to train bodybuilders. Yeah, weird, I know. I actually learnt a lot from this subculture: discipline, sacrifice and hard work are things you can not escape in that life. One huge lesson I picked up that many missed was "Begin with the end in mind". The art of bodybuilding is often confused with weight lifting. The number of competitors that complained that a physically weaker contestant beat them amazed me. Being able to bench more than your opposition counts for nothing.
In the sport of bodybuilding the winner is decided by a panel of judges; that's right, you are judged by humans. It is the image you present that they must judge you on. You may even have a better physique but if you do not display it better than the others you can lose. For this reason all of my athletes posed at every training session. In the middle of the gym, in front of mirrors down to their underwear, unashamedly posing as if they were in front of a panel of judges. We would critique, take photos, film it, change the lighting... I never saw any other body builders do this. I am sure that why, collectively, they won several, national and international championships.

Now ask yourself: Are you training for game day?
For us this is deploying to the production environment.
Are you practicing it everyday?
You should be.

If you are doing manual deployments you are a weight lifter in a bodybuilding show. You will lose.
To be honest we have it better than body builders. We have no competitors and we have predefined requirements. We can measure our performance, they cant, they can only contrast and compare.

So what do we need to do to prepare for the big day? Well first and foremost don't make it a big day. Make it just like every other day. Make deployments part of your daily routine and start deploying from day one. Personally I like to have automated deployments working prior to writing any business code. Infrastructure cruft should be done in iteration zero and deployments are infrastructure cruft.

A daily routine makes deployments so easy the are a non event. This usually mean defining everything you need to do to do a deployment then scripting it - Writing the best code in the world means nothing if the deployment is botched... and manual deployments get botched.

Deployments should also cover all the steps you will do on production deployment day, they do not cover just what needs to get done to make the application work on a developers box.

Define where you production environment is at and then work from there. If your testing environments are not the same, you should question: why? The more similar they are the less likely you are of having deployment issues. Where I currently work we have 5 environments

  1. Developers machine
  2. "Development" Environment - automated CI deployment
  3. Test
  4. UAT
  5. Production
The developer machine I consider an environment. I often get latest and (sometimes) hit F5 and run the application and I expect the application to work! This means I require a database that is in a valid state, external services to work, file paths to be correctly set up etc. For this reason I like developers to have local sandbox which includes local databases. Nothing pisses me off more than when I am running an app and some bugger has cleared the dev database for testing a script breaking my flow. Having your own database also forces you to properly script database changes in a sane manner. Checking in those changes and getting latest, running you build script should get you up and running every time. See Tarantino from Eric Hexter or  RoundHouse from the ChuckNorris framework for a simple way to get database migrations working cleanly in a .Net-SQL world.

The Development Environment is a name sake. No development is done on it, but it is owned by the developers. We may use this for debugging on a production-like environment if things go pear shaped, I just never have. Its main purpose is for, IMO, automated deployments from our build server. If anything breaks here the red light on the build server goes on and the check in for that build fails. The steps to do this include
  1. Cleaning the server - ie getting rid of the last deployment and backing it up
  2. Setting up the server including
    • creating apps pools/virtual and physical directories, 
    • ensuring dependencies are present eg MSMQ, DTC etc
    • ensuring the dependencies can run, ie queues are set up, services are running
    • setting up account privileges
  3. Deploying the packages to the server and installing it if applicable
  4. Running SQL scripts including
    • creating users roles and permissions
    • creating the database objects (tables, views, keys constraints, triggers etc)
    • creating required reference data
  5. Testing the deployment
    • creating test data
    • running high level acceptance and smoke tests
If you can get to this stage then is it not obvious that doing a test deployment is going to be next to trivial? Migrating to the test environment should be the same as migrating UAT and therefore the same as Production. Production deployments should therefore be just a matter of going through the motions.

This also means that you may need various scripts or at least functions with in these scripts to carry out these various steps. Obviously if the Production environment is already set up we do not need to do it again, and the deployment scripts should reflect that. Just like in normal code use pre-conditions and post-conditions to enforce a correct deployment. If certain set ups are not required log it and move on, just make sure it is part of the agreed process.

DBAs are involved and decide how you want to manage your deployments. Keep reminding the team that this should be streamlined.
One thing that often trip up teams is permission issues. Personally I prefer not having access to anything outside of the development environments (I'm pretty sure I am alone on this one). As far as I am concerned the testers can deploy there own stuff. It will be the same script that the SysAdmins and DBAs will run in UAT and production, why should I do it? I have code to write! They can have permission to run the scripts in their own environment making sure that no manual step has been introduced by any developer along the way. This separation I feel further reduces risk of failed deployments. If the deployment to Test does fail then they can raise a bug, roll back and tell the developers what they think of them. Sure this will be embarrassing and it does happen, but would you rather it done in the confines of the IT department or in full view of the customers?

This brings us back to what we are here for : to fix a customers problem. I assume this typically means delivering working software. Working software on your development machine has not fixed the customers problem, that's like being a weight lifter in a body building show. Don't be that guy, think about the end game and make sure that each day you are working towards that end goal of providing you customer with a business solution, in the cleanest possible way.

*Sorry for putting the images of most nude men in your mind, it (probably) wont happen again

No comments: