Udi Dahan's Advanced Distributed Systems Design with SOA & DDD

Over the last week I have been in Melbourne attending Udi Dahan's Distributed systems course and I thought now that I am home  will do a quick review on the course.

Firstly the course is not about how to build a 3 tier system on top of a Microsoft stack. Udi is a well known M$ MVP and has a well known open source .Net project however the course is an architectural course focusing extensively on architectural concerns that are largely technology agnostic. Most of the examples at in C# and use (an abstraction of)MSMQ however I guess near on 100% of the examples could work in Java (assuming tool support). The course is also not about how to build a Thomas Erl style web of web services.

To be honest leave all your preconceived ideas of architecture at home... it will just make life easier.

There were jokes of requiring a support group for attendees of the course, you will feel like Neo after he took the Red Pill; naked alone and very... well humbled.. or stupid, depends how diplomatic you want to be. Because people inherently don't like change there was the typical amount of resistance; given there would have been several hundred years of experience in building large systems in the small room and I would consider these guys some of the best minds in the country in terms of .Net, the red pill was pretty hard to swallow. However like Neo with Morpheus, we put our faith in Udi and let him show us the way.

Seriously though, the course was incredible, I think the overwhelming majority learnt more in those 5 days about designing large scale scalable enterprise architectures than they had over the 5 years of their career.

Just a side note about Udi: he would have to be the most tolerant presenter and teacher I have ever had. These concept were hard to grok all in one go, Udi allowed for lots of questions lots and ensured that the audience knew enough to progress. Sure a lot of the time I was thinking... "what the?" but typically the very next slide answered my question. In terms of raw ability, I have never met anyone like him. Yes he is an outstanding architect, truly outstanding, seriously never met anyone that can walk the walk like Udi in those terms. However to be a truly great architect you have to be a bit more than the guy who is drawing the UML. He has a fantastic mind that has a great understanding of business, organisational management, psychology and his technical understanding is better than all but the absolute best coders. This man is no Ivory Tower Architect. He would be just at home in the largest corporation  boardroom having discussions with CTOs as he would pair programming with the guy at the boiler plate writing the code. Possibly a genius.

Request Response is dead... well, its not but... well go on the course, its freaking awesome just be ready to start looking for a support group... you'll need it ;)

**********************

Course outline as it was for our course (from http://www.udidahan.com/training/ on 23 Jan 2010) :

The Rhys Campbell Course Rating: 10/10

Advanced Distributed Systems Design using SOA & DDD

Duration: 5 days

Introduction

Designing large-scale distributed systems is hard. New technologies make it easier to comply with today’s communications and security standards, but don’t auto-magically give you a robust and scalable system. Join Udi for a course packed with the wisdom of companies like SUN, Amazon, and EBay.

Tried-and-true theories and fallacies will be shown, keeping you from making those same costly mistakes today. Communications patterns like publish/subscribe and correlated one-way request/response will be used in conjunction with advanced object-oriented state management practices for long-running workflows. If you enjoy deep architectural discussion, if you are in charge of building a large-scale distributed system, if you want to know more about how the big guys run their systems, this is for you.

Audience

This workshop is targeted at team leads, application and solutions architects, as well as technologists who are involved in making decisions about the overall system design of software products and projects.

Course Topics

Module 1: Distributed Systems Theory
Decades of distributed systems development have taught us many lessons. In this module we’ll cover many historical mistakes as well as proven best practices for scalable and robust design. Topics include:

• 8 fallacies of distributed systems
• Transactions

Module 2: Coupling: Platform, Temporal, & Spatial
Loose coupling has become the watchword of complex systems development, yet few understand its multiple dimensions. In the module we’ll be covering the three different dimensions of coupling as well as patterns for dealing with them.

• Platform Coupling – XML/SOAP
• Temporal Coupling – Synchronous/Asynchronous
• Spatial Coupling – Endpoints/Topics

Module 3: Asynchronous Messaging Patterns
Although scalability is achieved through the use of asynchronous message passing, more advanced message exchange patterns are required to handle today’s complex integration scenarios. This module will cover the most commonly used patterns:

• One way
• Correlated Request/Response
• Publish/Subscribe

Module 4: Bus & Broker Architectural Styles
Enterprise Service Buses are all the rage these days. In this module we’ll be covering what’s the difference between the Bus architectural style, and the more well-known Broker, found commonly in many EAI projects. Topics will include:

• Architectural advantages and disadvantages
• Technological advantages and disadvantages

Module 5: SOA Building Blocks
One of the goals of SOA is to develop systems which are more closely aligned with Business. In this module we’ll be covering an analysis methodology from moving from the business domain to executable systems that comply with all the principles of loose-coupling.

• Business Services
• Business Components
• Autonomous components & Queues

Module 6: Scalability and Flexibility
In order to enable agility, services must be able to scale up, out, and down quickly. In this module we’ll see how autonomous components can be configured including transactional and durable aspects of message handling.

• Configuring autonomous components
• Scaling up and out

Module 7: Long running processes
The distributed communications patterns wouldn’t be complete without a discussion on orchestration. In this module we’ll see how to manage the state of long-running distributed communication flows as well as:

• Encapsulating process logic
• Advantages & disadvantages of orchestration

Module 8: Service / Autonomous Component Solutions
As developers go to implement autonomous components, guidance is required as to which concepts need to implemented in which project, what dependencies are there between projects, and how to bridge the worlds of messaging, business logic, and reporting. Topics include:

• Messages + Handlers
• Databases

Module 9: Service Layer – Domain Model Interaction
Logic-rich services require the use of advanced techniques for logic componentization. The Domain Model Pattern enforces a high level of Separation of Concerns, yet it must eventually be connected with Service Layer code that supports many concurrent users. In this module, the topics covered will include:

• Domain Model introduction
• Testing Domain Models
• Optimistic, Pessimistic, and Realistic Concurrency Models

Module 10: Creating High-Performance Domain Models
The strong separation between the Domain Model and the database which stores and retrieves its data may enable a high level of testability, yet often causes performance problems. In this module, we’ll see the various aspects impacting the performance of persistence:

• Transactions and Isolation Levels
• Lazy Loading, Eager Fetching
• Databases Tips & Tricks

Module 11: Web Services and User Interfaces
The ease of interacting with users over the web drives the need for service to UI interactions. Also, many integrations require exposing synchronous web services to customers. In this module, we’ll see what is required in both cases:

• ASP.NET 2.0 Asynchronous Tasks
• Rich Internet Applications and Services
• Web Services for integration

Module 12: Smart Client / Service Interaction
The publish/subscribe semantics with which services communicate require smart clients to perform a great deal of background work. Also, certain service contracts lead to more performant clients. In this module, we’ll cover the first part of these interactions:

• Multi-threaded client challenges
• Client-friendly Service Contracts
• Service Agents and Client Repositories

Module 13: Notifications & Smart Clients
After Message Handlers in the Service Layer create or update the relevant Model objects in the client Repository, Supervising Controllers are in charge of getting Views to show the updated data. In this module, we’ll describe the parts and interactions of these flows:

• Client-side Model Objects
• Supervising Controllers
• Views and their Interfaces

Module 14: Commands & Smart Clients
Capturing user intent and synchronization between views are at the core of smart clients. After describing solutions that use Events on the View Interfaces, the Command Pattern will be introduced to further decrease coupling between Supervising Controllers. In this module, we’ll describe the parts and interactions of these flows:

• View Interfaces, and how Entity Cloning affects them
• Supervising Controllers and clone reconciliation
• Commands, and Event-Based programming

Summary & Review
In order to make sure that attendees are able to put into practice all that they’ve learned throughout the course, here we strengthen the seams between the various topics. Q&A is also a core part of this final section.

Relearning WCF

Of late I have been playing with WCF again. We have some projects here at work that require some integration and we are desperately trying to move away from the old ASMX based services. Unfortunately because I have not touched WCF the whole time I have been here (12 months now, wow! That has gone fast!) and I have found myself at a point where I really need to relook at WCF again and basically relearn it... oh well.
Anyway here is a bunch of stuff that here, that at work we have found to be useful that you may not otherwise be able to do with ASMX or may not be aware you could do with WCF.

IOC and WCF

You can in fact use IoC with WCF, there are some good blog posts and accompanying videos to show what to do and if, like me, you just want one ready to that uses the CSL then The Code Junkie has done it for you!

Dynamic KnownType Resolution

This always erked me that I had to put into the data contract that I knew of other types, it was like really bad tight coupling*. There are a bunch of way to declare known types with the bottom example seeemingly a little known alternative : a provider mechanism

in config

<system.runtime.serialization> <dataContractSerializer> <declaredTypes>...

Data contract with attributes


[KnownType(typeof(PurchaseApprovalRequest))]
[DataContract]
public class ApprovalRequest
{...

Knowntype provider

The way I have just found out is by declaring a knowntype provider on the service contract:

[ServiceKnownType("GetKnownTypes", typeof(ApprovalRequestKnownTypesProvider))]
[ServiceContract]
public interface IApprovalService
{...

with the following class (change the implementation to suit yourself, this is from some of my demo code, it’s not recommended!)

internal static class ApprovalRequestKnownTypesProvider
{
public static IEnumerable<Type> GetKnownTypes(ICustomAttributeProvider provider)
{
// collect and pass back the list of known types
foreach (var module in Assembly.GetExecutingAssembly().GetLoadedModules())
{
foreach (var knownType in module.FindTypes(
(t, f) => ((Type)f).IsAssignableFrom(t), typeof(ApprovalRequest)))
{
yield return knownType;
}
}
}
}

With these two little nuggets I have been able to produce a pretty handy little broker service that act as a very basic content based router that keeps the client messages very clean and does not expose any implementation details (i.e. no passing of service or workflow names in the message header!)

*NB: to paraphrase Krzysztof: "Polymorphism is an OO term, not SOA term, so I don't use it, and make my contracts explicit wherever possible." be wary that you are using known types for the right reasons

Functional .Net : Tuple

Tuple's really don't have a lot to do with functional programming, they are a common concept in many language that for some reason are only making their way in the .Net on version 4.0. One could argue that you could have always easily constructed your own Tuple class but unnecessary duplication of such a simple type has obviously become apparent to the BCL team. This is good. Simple classes like this should be present in the framework. :)

So what is a Tuple? It is basically a container of a finite list of objects; a Point could be described as a list of 2 values Tuple<int, int> where the int values could be the X and The Y coordinates of a point. A Date could be described as Turple<int,int,int> with the values being Year, Month & Day. It is not a list in that your would typically enumerate through the items but you would reference them by index. You may be ask why is this different to an Array eg int[3]? Well with a Tuple the list length is set at compile time and the type of each index point is also set. Tuple<int,string,DateTime> forces you to always have the DateTime as the 3rd item. This is all pretty under-whelming to be honest, but like anything cool its the simplicity that is its strength and also why it pops up in functional styled programming a lot. Future demos are likely to include Tuples :)

Functional .Net : Currying

Currying is another functional technique that is possible to achieve with C#. The technique basically allows the rewriting of a function that takes in multiple arguments to one that takes in one argument and returns a function which may in turn take more arguments, the basic premise being able to build up composite function by splitting functions down and reducing the number of parameters dealt with. I will be honest and say that  I have found the language you use (C#, F#, Haskell etc) would be more influential to your predisposition  in using this technique, as it is with many of the function patterns and IMO C# does not lend itself nearly as well as F# for example. That being said it still can be done so lets look at a basic example. For starters currying is something that is not catered for explicitly out the box in C# but can easily be done using extension methods eg:

public static Func<TArg1, Func<TArg2, TResult>> Curry<TArg1, TArg2, TResult>(this Func<TArg1, TArg2, TResult> func)
{
    return a1 => a2 => func(a1, a2);
}
public static Func<TArg1, Action<TArg2>> Curry<TArg1, TArg2>(this Action<TArg1, TArg2> action)
{
    return a1 => a2 => action(a1, a2);
}

These extension method now allow you to take a 2 parameter delegate and split it into a one parameter argument return an Action or Func that take one argument. This can obviously be extended and helps facilitate the separation and composition of functions.

Now from my understanding of currying it is a specific form of partial application, being that currying splits its functions down to single argument delegates while partial application make no such claim, i.e. a three argument function be be reduced to a single argument function returning a 2 argument delegate. As this is purely academia I don't really care, the principle is the same.

A trivial example of currying (I'm lazy I stole it from Matt P and it is using the extension method above) :

Func<int, int, int> multiply = (x, y) => x * y;
var curriedMultiply = multiply.Curry();
var curriedMultiplyThree = curriedMultiply(3);
var curriedMultiplyResult = curriedMultiplyThree(15);
Console.WriteLine("Result of 3 * 15 = {0}", curriedMultiplyResult);

Unfortunately the verbosity of C# when approaching this style of coding very quickly begins to put me off.  The equivalent in F# is much more readable, but hey its what the language is strong at so it really should be a nicer experience. Either way its good to know the facilities are there if one day I do ever need to use it.

Basically the take away from this post is the extension methods at the top, with out these there will be no curry love.

Links forwarding (yeah this was a lazy post):

http://codebetter.com/blogs/matthew.podwysocki/archive/2009/04/26/functional-c-forward-functional-composition.aspx

http://mikehadlow.blogspot.com/2008/03/currying-in-c-with-oliver-sturm.html

http://stackoverflow.com/questions/411572/proper-currying-in-c

Functional .Net : Closures

One of the more commonly used functional techniques that can be used in C# is the use of Closures, a technique that if your are currently using lambdas, you may be using them inadvertently. My understanding of closure may be different to others as there seems to be so many subtlety different definitions especially when comparing languages. Anyway, in my mind the comments of Javascript closure best align with my understanding (http://www.jibbering.com/faq/faq_notes/closures.html)

A closure is a delegate that has references to a variable not passed to it and in an scope outside the delegates immediate scope.

Like any delegate its definition and execution are not the same thing. You can define a closure and never use it or just call it later

A simple closure example I can think of is:

static void Main(string[] args)
{
    var timesToRepeat = 100;
    //Declare the Action
    Action<string> print = text => //text (string) is the only parameter 
        {
            //using varb declared outside of Action
            for (int i = 0; i < timesToRepeat; i++)
            {
                Console.WriteLine(text);
            }
        };
    timesToRepeat = 3;//Lets modify the variable 
    print("Hello!");//Call the action/evaluate the expression
    //Prints:    
    //Hello!    
    //Hello!    
    //Hello!
}

Note that the timeToRepeat variable is declared outside of the declaration of the lambda statement. Think about this; the Action 'print' can be passed out side of this scope, it could be passed to another class which does not have visibility of the locally declared variable. The 'print' expression is bound to that variable declared outside of its scope. This obviously has ramification in terms of holding reference to that object. Please also note that the expression 'print', like all delegates is evaluated when it is called, not when it is declared; Stepping over the above code will not print when declaring the 'print' Action but at the last line when it is called. One last thing to note is that the variable timetoRepeat is modified after defining the print Action and this is carried when we call 'print' in the last line; "Hello!" is printed 3 times, not 100 times as the variable would imply when the closure was declared.

You may have been using closures with out knowing it. Javascript and the associated libraries like jQuery use this technique a lot, as do many open source library such as TopShelf, MassTransit etc.

Functional .Net : First Class Functions

One thing I notice in .Net is that many developers do not think of functions as first class citizens. I guess in the OO world classes or more appropriate the instance representations are the real hero's, however, in my mind, functions deserve much more appreciation than they perhaps get.

Since .Net 1.0 delegates have been around and I still think many developers do not fully understand how they work. I have previously made a post with regard to delegates showing how they can be used in a real world way to save code duplication here. I guess one of the first steps to being comfortable with functional programming is being comfortable with functions as first class citizens; The best way for a typical C# developer to do this is get comfortable with delegates. Before I continue on with my Functional programming journey I want others following with me to be on the same page. Please be sure you understand what a method & delegate are; I feel I describe them reasonably well on the previous mentioned post.

Functional .Net: The Beginning

Of late I have (along with a few colleagues and friends) started to make a bit more of a concerted effort to up skill in the area of functional programming. I admit that my knowledge is  of functional programming is high level (at best) although have in advertently been using several of the core concepts due to the language feature I am exposed by the C# language I use on a day to day basis.

What has spurred me on is the talks from Dr Erik Miejer on channel 9 (the first of 13 can be found here). The talks plan on tackling Functional programming by working through the benchmark Functional book: Programming in Haskell

I am keen to see how the series progresses, I am only up to episode 2 but am already seeing value, more in the "why" as opposed to the "how", which is fine for this early stage of my journey.

I also want a bit of commercial return on investment with relation to what I can do in my day to day job with functional programming. As I have mentioned C# actually handles several of the Functional paradigms (although perhaps not as elegantly as F# and the like) and thanks to .Net resident Functional voice-to-the-masses, a bunch of Functional programming samples in C# can be found here to download; Cheers Matt! Along with just raw C# code he has a bunch of Wiki links to highlight what each example is actually doing; you may be surprised that you are inadvertently using some of these techniques!

Any way I will keep you posted as to how I progress as I move forward on the beginning on what is hopefully a fruitful journey!