Sunday, July 8, 2012

Lessons Learnt - Building an SDP

I was recently asked what lessons had I learnt in regards to building an SDP and I thought it was an interesting question in many aspects and a question I would attempt explore in the form of a post on this blog.
Now to prefix this post I am obviously only a cog in the many wheels it takes to build a product like a Single Dealer Platform and these opinions are solely mine and many members of the team, past and present, will obviously ave differing opinions. The Alpha platform has had its ups and downs, as had the whole market, however I am pleased to say it is a successful and profitable multi asset platform and I am proud to have played a part in its success. That being said there are things that could have smoothed the sometime bumpy ride. However I dont feel it is fair to throw stones as many of these problems are industry wide and we have interesting issue of having two industries to deal with, finance and technology and to ease the thought process in this post I will take the question and rephrase to if you had to build an SDP what would focus on? 

Form Key Relationships

Business Buy-In

The platform should be supporting the business and not conflicting with exiting goals and other teams strategies. If you have sales teams or team members that feel the product is competition you need to get them on board or you risk having to work against their negative energy. Sales and traders can have significant sway in the bank so obviously having them as allies make for a much smoother project. The opposition to new products is not isolated to these profiles. Anyone who has developed software internal in a corporation will have come up against this, so it is a common problem we must over come.  Given the platform should enhance their daily dealings by being able to provide up and cross selling facilities as well as a portal-like experience into research and post trade facilities, the product should actually be a good thing for the sales desks.  


Backend Systems

Be prepared  - Front end systems like SDPs do not work without good service  from the various providers required to price and book trades. There is more to an SDP than this, however these systems are key. Work closely with these teams - the are your life blood. When delivery times are tight you may have to call in favours from these teams. At a business level these teams goals must have some alignment with that of the SDP or their work slate will continually be serving other priorities. To have a beautiful fornt end that is not integrated is an effort in futility and will quickly raise concerns from the bill payers.


Clear Vision

Firstly know your strengths and play to them. Every bank cant not be the best in FX. Although most SDPs will have a strong FX and Fixed income feature set, trying to have the best prices and lowest latency is a costly venture. If you are going to try to take on the big boys you had better have a strong financial backer. If you are not one of those players, acknowledge it and find where your strengths are and use them. A strong product owner will be key here. His role will be to clearly understand the direction of the bank and to know what will drive the platform. He should have a clear understanding of our income sources and our user base. Defining core users is also key here. This will provide direction in the type of platform we will technically deliver : Desktop, RIA, Web, mobile or a combination should be a business decision based on the users of the platform, not based on the latest technical craze? Lastly we must have clear success criteria for each deliverable and the platform as a whole. When this is clear the team can easily all move in the same direction.

Deliver the Goods

In my opinion: Delivery is king. Continuous delivery is a logical progression of a well oiled development team. To achieve continuous delivery you should address some key, common concerns early in the project. Authentication and authorisation, handling of entitlements, security concerns, auditing, logging and other such cross cutting concerns are all usually defined by the bank at a department or group level. IMO these should be addressed in the first iteration while other team members could possibly be working on PoCs realting to functional deliverables. This is also a good time to investigate some standard deployment concerns such as the handling of independent and aggregated modules, dependency management, versioning, forward and backward compatible migrations. From a higher level architectural stance the team will also need to define how they could deal with future scalability, reliability and regional requirements. Do we have the infrastructure to provide five 9s to a global audience with latency under 10ms? If not now is a good time to let the business know so they can lead the charge on making the necessary changes if required or so they can have their expectations realigned appropriately. The reason I prefer to address this early is because the cost is so much higher when done later in the project and delivery becomes an expectation and a core team value. It also raises the notion of non functional cost to the business; fives 9's up time is not free, nor are follow the sun deployments, the sooner the business are aware of this the better for everyone.

Justify your Costs

By understanding what brings value to the business we can more appropriately prioritise our work. One major issue that I have encountered on many projects is the confusion of the prioritisation of features that generating income versus those that prevent losses or reduce the cost of doing business. Both are of value.
Features that I can see SDPs typically spending too much on are technical "play things" (new frameworks etc), performance and low value "nice to haves". A good Product owner that has a good relation with his technical leads and architects should be able to address these issues easily.

Typical neglected features are resiliency, monitoring and work related to minimising maintenance cost. No one wants to pay for any of these but we often dont realise we want the consequence of not having them even less. The unfortunate thing is we often discover this far too late. IMO the key ways of reducing maintenance cost is clean well tested code (ie employing BDD and TDD).
Another key issue in resource allocation. It is easy to think due to the large mount of money involved in building these platforms that a large number of people should be involved. For any one who has read Fred Brooks' Mythical Man-Month or has managed very large software teams will know that the communication overhead of throwing more resources at the problem will most likely only exacerbate the problem. In addition to this hiring the wrong resources can also be catastrophic. Because of this I prefer, like many agile teams, to have small self managing teams comprised of exceptional people that are completely accountable for the delivery of their feature. The cost IMO of having unaccountable people or inappropriately skilled people on the delivery team is often just too high.


Summary

The person that originally asked the question would possibly be surprised that my answer wasn't more technical as I think he wanted to hear "Use product X". I believe the tools used should ideally be representative of the way the team works and the product that needs to be delivered.  Now this is not always feasible as there will be company wide mandates in place, however how that team works within those confines can often be quite impressive.

Like any project I believe a team led well, with a common clear direction and with members driven to deliver success will find a way.

A couple of books to me really resonate in this area, the Brooks classic as mentioned earlier and surprisingly to some The Art of Scalability. I originally bought this while working on the streaming price distribution components for Alpha but to be honest go more benefit in the non technical sections. The authors strongly believe a scalable system comes for a well oiled team that allows for scaling. If you are looking at starting a large scale system like an SDP would recommend this book for more than just the excellent technical read it is.





Back to the code

The last couple of years have been a whirlwind and in my time running a relatively large part of a relatively large project I was often unable to spend time keeping up to date in any technology other than that at hand at the present time.  I have finally been able to put my head up for some air so to speak and get my head back in the tech world that I feel I have turned my back on in the last 18 months. I may post more about this soon however for now it is updates on stuff I have been playing with lately.

Im quite keen to dive back into JavaScript, which would have sounded weird to me 10 years ago as it was a language that at the time I thought was just a hack fest. Having grown up somewhat and having more experience with the language it is obvious that it deserves a bit more respect than that. With HTML 5 and Windows 8 fully harnessing javascript and the acknowledgement of node.js as a legitimate backend technology, JavaScript is here to stay. As I started diving back into the JS world I realised that my knowledge of JS fundamentals was woefully lacking.

My first step back into the JS journey is getting my head around what JS actually is (vs what I thought it was). Being a prototype based language and not a class based language, it offers different ways of solving similar solutions to languages I'm more used to. This does not mean you cant use conventional OO based techniques, we know you can, but it means you may have to rethink how you could best solve those problems. Most of the change comes into the definition of the objects, not so much the use of them... well... kinda. For more good reading on JS for people with OO background have a look into this nice blog series on the topic.

One of the major steps forward in JS, as the language has not really changed, are the new APIs available.  I have the fortune of having some friends that are in web based start ups that I can tap into as a source of up-to-date trends and whats hot in the "valley". A few of those things that I have picked up lately are UnderScore.js, Backbone.js and in the testing realm Jasmine and Sinon for isolation/mocking.

UnderScore is a handy utility library that provides a bunch of assistance functions especially helpfully, amongst other things, in the handling of collections. As it is a utility library its super helpful and at 1000 lines in a single file, if you really want to know how it all works, you are only an hour or two away from figuring it all out. I would certainly read through the website as it quickly and succinctly describes all its functions. Being a .Net fan-boi, Linq is dear to my heart, so I was please to see that method chaining was supported albeit not as cleanly as something like linq.js, but hey you cant have everything :).

Backbone to me is much more interesting. I have many times bumbled my way through DOM manipulation handling various client side events using JS.  Backbone comes to the rescue by providing a Model-View binding approach that I am much more comfortable with (at least conceptually) and leads to clean SoC and a testable model, which is all good stuff.
Learning the basics of Backbone is not super obvious and the docs IMO are not super helpful either (I have been spoilt lately learning rails with a tonne of good docs). Fortunately the PeepCode series has been a fantastic resource with 3 video tutorials and the accompanying source code so you can follow along.

Now there is no point in saying Backbone is super testable without testing it and Jasmine with Sinon fit in nicely here. Jasmine and Sinon are covered in the PeepCode series and also in this nice blog series.
Backbone goes a long way in addressing my issues that I had with my previous "DOM manipulation" heavy approach, and Jasmine fits hand and hand with an improve testing approach. If you are used to BDD style frameworks like Cucumber or SpecFlow then Jasmine maybe for you, however if you prefer standard xUnit style testing there is always QUnit.

Sinon is also a nice fit with testing as Backbone has some convention based features that allow you to hit backend servers for model updates and persistence; Sinon steps in and saves you some pretty serious pain if you were serious testing your models over the wire.

Hopefully I will give some more feedback on these various APIs and approaches as I get a bit more experience in them, however for now it is just a link fest. Depending on how well my ventures into Backbone go I may or may not use Reactive Extensions for JS. Reactive Extension (RX) is something that is pervasive on the code base I currently work on so it is an obvious choice for me to experiment with. RX is a brilliant API for event based data streams and is something that I think will become common place in .Net in the very near future. For more info check out Intro To Rx

See y'all soon.

Links

https://peepcode.com/products/backbone-js/
https://peepcode.com/products/full-stack-nodejs-i

Monday, November 22, 2010

Agile Tools That Just Work

Here are some nice tools that I use and generally make life easier in an agile world.

Scrumy

http://www.scrumy.com
Free kanban board for managing projects (and life).

WebSequenceDiagrams

http://www.websequencediagrams.com/
Quick way to create sequence diagrams and the input is text meaning its easy to share with other that don't have tools like Visio or EA etc. I actually prefer this to visio and our scrum team used it a lot in our last delivery.

SpecFlow

http://specflow.org/
I like Fitnesse but there is too much friction (not a lot, but enough) in terms of the development cycle (e.g. its odd source control management). SpecFlow is close enough to cucumber for me (in the .Net world) and does not rely on Ruby to be installed. Ruby installs on windows are still a complete PITA as far as I'm concerned and I'll generally do anything to avoid it. SpecFlow also runs with your existing (x/n/mb)unit testing tool of choice. I must say I didn't like this tool at first but having come back to it, either the experience is cleaner or I have opened up a bit more.

Gist

https://gist.github.com/
Like Pastie but associated to my GitHub account. I'm starting to use it more and more to do brain dumps of simple code snippets (e.g. PowerShell functions etc) that can be handy to reuse, but not really worth its own repo.

If I was running an external commercial project (e.g. at home) I would seriously consider the use of LowDown. It fits my development style very well without the clutter and bollocks associated with other ALM/project management tools. Git integration makes it very interesting too.

Monday, October 18, 2010

More AutoTesting - Powershell extensions and addins

There is a nice library called PS-Eventing that makes the event handling a whole lot cleaner and generally a nicer experience. This post with the plugin is not a smaller and doesnt really do any else other than the last post except cleanly deregister the event, but is much more user friendly.
I think i will also post my code on gisthub as I am not and have not been at my real machine for so long... i miss it... sorry.
Heres the link : http://gist.github.com/632749

Wednesday, October 13, 2010

Auto test - Thanks Powershell

Im doing some funky Nunit stuff* at the moment and sometimes the ReSharper or TD.Net runner wont pick up my inner fixtures, the console runner however runs them just fine. Here is a PS script to have auto test running on build:


$Script:initialdir = "C:\myroot\"

$fsw = new-object system.io.filesystemwatcher
$fsw.Path = "c:\mysln\myproj\Bin\"
$fsw.EnableRaisingEvents = $true

$action = {
$filter = "*Tests*.dll"
$file = $Event.SourceEventArgs.FullPath
if($file -like $filter)
{
$nunit = "C:\3rdParty\NUnit-2.5.2\nunit-console"
$currentdate = get-date -format g
write-Host "Running Tests on $file $currentdate" -ForegroundColor Yellow
&"$nunit" /xml:$_.Name-testresult.xml $file
if($LastExitCode -ne 0)
{
WRITE-HOST "Test have failed for $file"-ForegroundColor RED
}
else
{
WRITE-HOST "Test have passed for $file"-ForegroundColor GREEN
}
#$null = Remove-Event $_.EventIdentifier
}
}

Register-ObjectEvent -InputObject $fsw -EventName Changed "FileChanged" -Action $action
Register-ObjectEvent -InputObject $fsw -EventName Created "FileCreated" -Action $action


Note that you will either have to explicit unregister your event handlers or kill powershell to get rid of the handlers. Cancel the watcher by hitting ctrl+c and then Unregister by runnning these lines

unregister-event "FileCreated"
unregister-event "FileChanged"



*The stuff we are doing is to do with inheritance and inner test fixtures to give us a BDD experience without the drama of the BDD frameworks, that personally I still dont like. We are still a fair way off the TDD experience of the Ruby kids.

Tuesday, September 7, 2010

Threading - I am on a Journey

Recently it has become more and more apparent that I had a crack in my understanding of a key aspect of not only the CLR but computing in general. No prizes for guessing that it was threading and the general parallel programming constructs that go along with it.
Now I am not a complete n00b in the threading world, I have written code that uses threads in client and server side but it was typically handing off a single task to a single thread and looking back it was probably pretty delicate and by now in need of some re-factoring love.
The first real kick in the pants was playing with my 8 core machine and never seeing all 8 core light up in the task manager. How wasteful... its like all the Aston Martins I see driving over Tower Bridge; what is the point?
Next major trigger for me was a computationally heavy project that I, up until recently, was working on. My colleagues and I knew that splitting up tasks could make or break this system, however given a distinct lack of human resources and not exactly compelling hardware, time was never allocated to fleshing out the best approach.
Well I am no longer at that job (its somewhat unfortunate as the project could have been epic had it not be butchered by red tape and ego's) so I have had time to scrub up on my threading knowledge... well I thought I would just be scrubbing up. Sometimes you just don't know how much you don't know.

So in an attempt to short cut any one else's learning curve on the wonderful world of threading and parallel programming in the .Net world here are some great places to get started.

As always C# in a Nutshell is a great place to start with anything C# related. I have both 3.0 and 4.0, Joe is a great author; IMO every C# developer should have the latest copy of this book on their desk.

Next up the deceptively good CLR via C#. This book looks like its going to be as much fun as having teeth pulled out however it is surprisingly good and dives deep in the areas you want and would expect. I read the 2nd edition however will be buying the 3rd as soon as I can find it.
Obviously these books are not completely dedicated to threading but they have some great chapters dedicated to it. If you are after a more in depth reference I have been recommend Joe Duffys book : Concurrent Programming On Windows

Next up is my favourite learning tool : Tekpub. There is a threading production under way, it only has 1 episode but it is great. As I have a yearly subscription it was a no-brainer. For me this was valuable as it has a bunch of code and a great visual comparison of auto and manual reset events, Semaphores and Mutexes. I highly recommend this video if these are confusing you at all.

For a more casual and much briefer talk on Threading in the pre .Net 4 days check out Jon Skeets articles . I have found the multiple angles these guys talk about the same topic good for bashing it into my head.

The piece that I am currently reading is the PnP new book Parallel Programming with Microsoft .NET that is also available as a hard copy book and eBook. Code snippets for the text are available here.

The last piece that is semi related is my reading of the Manning book Real-World Functional Programming. Although functional programming technically has no direct link to threading its basic tenants strongly support parallel programming and as such I have been endeavouring to improve my understanding and use of functional constructs where it is appropriate. So far the books is very good it is in F# and C# so it has bee a good (re)introduction to F# for me.

With all of the movement in not only hardware (hyper-threaded and multi-core processors) as well as architectures (Grid, Cloud, ESBs etc) and the software frameworks we use (.Net's TPL, Rx etc) and how we use them (i.e the movement towards functional programming in distributed/parallel systems) I think it is becoming very clear that as .Net developers we need to know how this stuff works, how we can use it and what it means for the systems we build and the users that use them.

As tempting as it may be to run off and just blindly use the new TPL I strongly urge you to get a good understanding of the history and the older APM practices, I am sure this will ensure you will make better choices when you write your next parallel task.
As for me I'm back to reading, I have a long way to go...

Sunday, July 25, 2010

Getting started in Asp.Net

I recently met a guy who was new to town and had picked up a job doing development for a place that appeared to be moving towards Microsoft focused solutions meaning he was likely to move away from his LAMP (Linux, Apache, MySQL, PHP) background to a .Net/Sql Server environment.

I realised after talking to him that there are a lot of ways you could go wrong coming into the .Net realm from a LAMP/Java/Python/Ruby background and this post, although intend for him, is generic enough that it may save other web developers valuable time to get up and running, and producing valuable software using the M$ stack

Asp.Net

Asp.net is a different beast to the many other frameworks out there and you could spend years (and many have) learning all the ins and outs of the beast.
Dont.
I would highly recommend skipping WebForms all together as IMO it is a dead framework. I have been doing MS MVC for the last year and did MonoRail (an open source ASP MVC framework) on the side prior to that and I have never looked back. Some of the significant benefits are :
  • Matches the way the web works (you explicitly work with http verbs)
  • clean html
  • easy to learn
  • pluggable
  • testable
  • simple
  • And finally, IMO, encourages better coding practices, which web forms simply does not.

I will assume you will trust me and accept that MVC is the way forward and I will go ahead and recommend how to start.

Asp.Net MVC in action
First port-of-call is to pick up this book. It will give you the no BS version of the truth and covers some third party tools that will help you along the way. There is a second edition on the way however if you are using VS 2008 then this books is for you anyway. Manning (the publisher) often have good discounts so register for emails or follow them on Twitter to know when promos are running.

TekPub : Build your own blog with MVC
Tekpub is one of my favourite resources at the moment. It is great value for money and video is an efficient way to learn new concepts. I had bought several series and now have the yearly membership and many of my colleagues have done the same. Go check out the BYOB which shows how Rob* built a blog in MVC. There are also MVC 2.0 and various other Asp.net series available. Buy one, buy a bunch or get a month or yearly subscription, go on, you know you want to ;)
*Rob was on the team that actually built Asp.net MVC

Asp.net
Yes the website is a good place to get lots of info too! There are lots of little articles and videos from from smart people who live and breathe this stuff. Just be aware that there is sometimes a very Microsoft slant to the information, not a bad thing but something to be aware of.

Next there is the Database

SQL Server

If you have a background in Oracle or MySql then picking up M$ Sql Server should be pretty easy. The standard practices apply e.g. dont run your web app as System Administrator. Nothing to really say here to be honest.


Joining the web and data world is your data access strategy

Linq2Sql or NHibernate

The 2 standout choices for new commers IMO are the light weight data acess strategy from Microsfot called Linq2Sql (pronounced "Link to Sequel", just dont ever write it like that) or the OpenSource and more heavy weight NHibernate. If you are from a Java background NH may be a good tool, if you are from a Rails background then Castle ActiveRecord (which sits on tope of NH) may even be a good idea. However for most Linq2Sql is probably the best bet when you are getting started and NH is the best if you are building up more complex domains. The caveat with NH is that there is a steep learning curve. If you are new to it then check out SharpArchitecture or one of the abstractions in the NHforge.com to help bootstrap yourself. If you have a simple domain or going to be building from the database up (i.e. create a database table and then the c# code to use it) Linq2Sql is a better fit anyway.

Tools

If you are playing with the M$ stack at home then you will most likely not want to shell out cash for the full blown enterprise versions of Visual studio and Sql Server. For this reason they have the express versions. I have only ever used the SQL express edition (which is great) so have no idea what VsExpress is like. In saying that if i did not have the full version of VS I probably would not be using .Net and just be doing rails/sinatra dev (but thats just me).
Assuming you are working with the full version at work then be sure to invest in the Resharper VS plug-in. Its not super cheap but the time it will save you will mean it will quickly pay for itself.

I terms of source control TFS is the M$ standard, but I personally don’t recommend it. I think tools like Git or even SVN are better tools that integrate better with other platforms/tools and are free. TFS however does more than source control, so if you are using make sure you use all the good ALM bug tracking bits etc

Extra links

Follow the team behind the framework, most importantly Scott Gu (all .net dev should read his blog) and Scott Hansleman. Mr Hanselman also has a good podcast called hanselminutes that is not Asp.Net specific but a good subscription nonetheless.
StackOverflow is a great QnA community place for lots of developer question. The are lots of .Net and Asp.Net people hanging out here so most of your initial question will most likely have already been answered. The site itslef is even built on top of Asp.Net MVC and Linq2Sql so you can see that this combination is a very real world and scalable solution.

If you are in Perth (like this person was) check out the Perth .Net Community of Practice and the Alt.Net lads that have monthly beers and discuss things that may or may not be from the Microsoft mother ship that affect .Net devs.