David Ross's Blog Random thoughts of a coder

In most large enterprises there will be hundreds or thousands of applications within their portfolio and with the incredible pace of change within the technology sector the vast majority of those applications will:

• have been developed using out dated techniques
• been built with languages or tool chains that are no longer supported
• have little or no documentation
• have a poor user experience
• have fundamental architectural flaws
• insecure
• poor transaction management
• don’t scale

Even though they might have faults or be technically impure they “Run the Business” and for large enterprises without their bespoke applications they would literally cease to exist. Hence every couple of years an architect or senior developer will be asked to do an architectural review to determine whether a particular application can hobble along for another couple of years as is, if some investment needs to be made to refactor or uplift the design or whether the organisation should explore decommissioning it entirely.

After partaking in many of these reviews I have developed a semi-formal process based around answering the following questions:

1. What enterprise application architecture was used to create the solution?
2. Did the architecture make sense when it was originally designed?
3. How does the architecture compare to a system designed today?
4. Are there any fundamental issues with the implementation?
5. Is it possible to refactor the code base into a more modern or clean one?

Over the next few posts I will be giving some background to each of the questions and then provide a set of review checklists.

Inheritance versus the strategy pattern

Broadly the patterns described in the “Gang of Four” patterns book (Design Patterns: Elements of Reusable Object-Oriented Software) can be broken into two groups. The first group provide techniques to manage an object’s lifecycle and the second group is about splitting an object’s responsibilities into a group of interconnected smaller objects which should simplify the design. Many of the lifecycle patterns (singletons, factories etc.) are found in modern Inversion of Control containers and no longer need to be implemented by hand. Meanwhile the second group are utilised in the design of a rich domain layer.

Consider a bank that has a range of savings account products with each having a slightly different method for charging fees. Some fees might be based on the amount of money in account, others based on the number of transactions and finally some premium based products might have a flat monthly fee. When developers first learn an object orientated programming language this type of problem is solved in one of two ways:

1. Implemented using procedural techniques
   • The Fee Calculation logic is placed into the Bank Account class and is implemented using cascading if-else or switch statements
   • Adding a new Fee Calculation involves modifying the Bank Account class
2. Implemented using inheritance
   • The developer realises that are five or six distinct bank account types
   • A base class is created where the common aspects of all of the account are stored
   • An abstract CalculateFees method is placed on the base class which the children override
   • Adding a new Fee Calculation involves creating a new leaf class meanwhile the base class is not modified

The inheritance based solution is far superior to the procedural alternative since new bank account products can be added to the solution within necessitating any changes to the base class.

Unfortunately the inheritance design has some issues:

• Hard to model bank accounts that have a blended fee structure without multiple inheritance
  • Retail Plus account combines aspects of the Premium account with the Basic Saver Account
  • Debit account is the same as a Retail saver but without overdraft facility
• Inheritance can’t be switched on and off its an integral part of the objects identity
  • Act as a Premium account until number of transactions is over 20 than act as a normal account

An alternative to inheritance is to utilise the strategy pattern and to move the Fee Calculation engine out of the Bank Account object and to place into a new group of objects that share an interface. The Bank Account then delegates to a collection of Fee Calculation objects instead.

This approach:

• Enables an account to have zero, one or many fees structures
  • Allows the creation of a rules engine that allows Fees Structure to be prioritised or combined easily
  • Scales to allow for any type of behaviour that is optional or time dependent to moved out of the object and into a strategy
• Tightens up the definition of a Bank Account
  • A collection of transactions that are associated with a customer
  • Business rules can be applied to the account that generate transactions which in turn allows interest to be paid and fees to be charged

After learning the strategy pattern developers will quickly find that they begin pushing inheritance out of their entity classes and into behavioural components instead. Thus the inheritance hierarchy (this includes the implementation of an interface) is used for allowing optional runtime polymorphic behaviour as opposed to modelling an object’s type.

UML with Colour

One of the difficulties when using UML is that it doesn’t validate the quality of the design or provide guidelines to aid in the modelling process. UML with Colour (described in Java Modeling in Color with UML) meanwhile overlays UML with a set of simple design rules for building out the logical domain model. The logical model that is created is similar to 3^rd or 4^th normal form in database modelling as the resulting object model is highly decoupled.

UML with Colour is based on the premise that domain objects are split into three groups.

Party, Place or Thing objects have Role objects which modify or create Moment-Interval or Transaction objects

Example:

• <People, Person or Thing> - <Role> - <Moment-Interval or Transaction>
• Manager – Approves – Timesheet
• Customer – Buys – Products
• Trader – Cancels - Trade

Party, Place or Thing

The Party, Place or Thing archetype is used to represent all of the system’s actors. UML with Colour ruthlessly removes many of the places where inheritance is used in modelling actors by stating that a Party, Place of Thing cannot manipulate or create Transactions directly and instead this work should be delegated to a Role. When modelling the users of a system instead of having a deep hierarchy that includes managers, team leads, customer service, administrators etc. there might only be a staff object and all of the different capabilities and responsibilities of a manager or administrator is modelled as Roles.

Typically Party, Place or Thing objects:

• Will have a unique identity across the application
• Can be an aggregate root

Role

Where UML with Colour differs from classical domain modelling is that it makes the strategy pattern a core part of the design process. Consider a requirement where a manager needs to approve timesheets in a HR application. Instead of putting an Approve timesheet method on a Manager object the design starts with a Person object that interacts with a Time Sheet Approver Role. Thus a Role is an object that creates or manipulates a Transaction on behalf of a Party, Place or Thing. As discussed removing this logic from the Party, Place or Thing objects has a number of benefits with the most important being that it allows for capabilities to become optional and combinable.

Typically Role objects:

• Will implement an interface
• Will often be a factory

Moment-Interval or Transaction

The Moment-Interval object represents an event or transaction.

Typically Moment-Interval objects:

• Include a timestamp, sequence number or time period as part of their identity
• Often have a collection of children such as a Purchase Order has Purchase Order Line Items

Description

There is a fourth archetype in UML with Colour called Description which is an object that stores metadata about another object. For example a car might have attributes including its registration number, amount of fuel in the tank and miles on the odometer. In addition there are attributes that are shared by all cars of the same model such as its manufacturing year, make, model and colour. These should be placed into a Description object to ensure that the data is shareable, not duplicated and most important recognises that the data is intrinsically immutable.

Typically Description objects:

• Are value objects
• Are similar to dimension tables in a data warehouse
• Are attached to Party, Place or Things and Moment-Intervals but not Roles

Colours?

The Colour part of UML with Colour is simply that within the UML diagram each of the classes in the diagram is shaded with a different colour representing each of the archetypes. The colourisation ensures that the basic design rule whereby a Person, Place or Thing can’t connect directly to a Moment-Interval is enforced.

The four colours have been purposely chosen so that they are available within a pack of coloured post-it notes. The design process will often consist of sticking post-it notes to a whiteboard and drawing arrows between them.

• Role is yellow
• Moment-Interval is pink
• Description is blue
• Party, Place or Thing is Green

Logical to Implemented Domain Model

Once the UML with Colour diagram is completed the developer is then able to decide how the model should be implemented.

The three choices are:

• Implement classes identical to those on the logical model
• Denormalise the design and utilise inheritance so that each Role of a similar type becomes an inheriting class off the Party, Place and Thing class
• Implement the capability of the Role directly in the Party, Place or Thing class

It might sounds strange after going through a process where we extract behaviour out of entities and into separate classes the design process then advocates that the developer has the opportunity to collapse all the responsibilities back into a single entity. However this follows the tradition of all refactoring processes in that they are by-directional. For example code can be pushed out of a function using the Extract Method refactoring and in turn if a function is no longer useful as an abstraction its contents can be pasted into its callers.

Examples:

• If there are many Roles that are optional leave the design as is
• If available Roles can change for the same Party, Place or Thing instance leave the design as is
  • Trades can be approved by all team members until midday after which time only a manager can approve the trades
• If a Party, Place or Thing class only has a single Role or a group of independent Roles that act on different Moment-Intervals that behaviour is not optional the Role should be merged back into the Party, Place or Thing
• If there are a set of Roles that provide similar functionality and are not optional consider removing the strategy and inherit off the Party, Place or Thing class.

Further as we have seen it is trivial to refactor from a de-normalised model into a normalised model that utilises separate Role classes i.e. it is possible to refactor from a design where everything resides in a single class, to having one with a base class with inheritors to one where the design relies of behavioural patterns like the strategy.

Limitations

• UML with Colour is focussed on domain objects and is a poor choice for modelling the interaction between layers or systems in an enterprise application.
• In initial discussions with the business it is preferable not to discuss concept of Roles
  • Business wants to discuss problem domain using titles such as manager and team member
  • In comparison calling managers, Staff with Approver or Proposer Roles can be initially confusing
• Not a common technique
• UML itself isn’t as popular as it once was
• UML with Colour is not widely known as it was introduced in a Java design book as opposed to be championed for all developers

Due to UMLs many faults for many years I had stopped using it as part of the design tools. A couple of years ago a colleague introduced me Colour-UML and his design approach which I now utilise on most of my projects.

UML should primarily be used as an exploratory design tool in a workshop environment with the intent that the artefacts created being quickly converted into code.

UML usage is limited to the domain entities and view models

• The team should be focussed on creating a rich domain model using a ubiquitous language
• Aggregate roots should be identified but in a post-LINQ world repositories are rarely required nor need to be designed
• Presentation Services/Controllers and Business Services are not modelled in UML – Instead only a single example of a presentation or architectural pattern needs to produced just to demonstrate the approach
• View Models are only modelled if there is a complex tree of View Models such as those used in Single Page Applications

UML diagrams are created on whiteboards, butches paper or Visio

• Don’t use UML to generate code
• Don’t generate UML diagrams from source code

Long lived UML artefacts are stored as pictures

• If a design has been completed on the whiteboard the design team can photograph it and upload into the Wiki
• UML diagrams are allowed to become “Out of Date” the source of truth is the source code
• Best diagrams are those stuck on the wall next to the development team

Only design objects relationships and core fields

• Diagram should show a Customer has an Address but shouldn’t detail that an Address contains a Postcode, Street, Suburb etc
• Agile principles in fields are only added to entities as they are required hence no need to model them “Up Front”
• Adorn the relationship with its multiplicity (0..1, 1, *)

Objects should be modelled using the following Colour-UML archetypes

• Person, Place or Thing
• Role
• Interval-Moment
• Description

UML

During the 80s and 90s there were around a dozen popular notations for graphically describing the relationship between objects. Booch’s notation mapped closely to C++. Coad’s notation was an extension to the modelling techniques he defined for databases. When Booch, Jacobson and Rumbaugh joined forces and merged their notations into the Unified Modelling Language it quickly the standard for Graphical Object Orientated Design and soon it emerged as a core tool for enterprise development.

However the last decade UML is gradually fallen out of favour as and its decline has been heavily influenced by the rise of Agile development and the popularity of dynamic languages such as Javascript. UML itself has ballooned in complexity as tool vendors have tried to move the language from a design to implementation tool.

Big Design Up Front

Waterfall practices in the 80s and 90s resulted in many enterprise environments devaluing the contribution of programmers in their development teams. Instead of the “coders” doing design work this role was performed by architecture teams who would deliver UML or Entity Relationship diagrams and developers were relegated to being low skilled implementers. These diagrams not only showed the relationships between entities but also fully detailed their attributes and would take many months of analysis to complete. Further since the source of truth was the diagrams and not the source code large amounts of effort were spent trying to keep the diagrams and source code aligned.

As Agile started to emerge across the industry developers were rightly given the freedom to implement requirements as they saw fit. Further refactoring techniques now mean that objects are continually being renamed, split, merged and deleted and trying to keep the UML diagrams aligned with code base is ultimately a fruitless exercise.

Feature driven development

Team that use a story or feature driven development approach naturally build up the systems capability in increments. This effectively means that only a small number of objects are being created during each iteration and thus the amount of design work required is far lower.

Rise of dynamic languages

Whether an object was created within a dynamic or static language its makeup is still defined by its fields, properties, methods and its relationship with other objects. Hence UML can still be valuable to model backend server side components in a node.js, Ruby or Python project it can also be a useful aid in modelling complex interactions within a tree of View Models on the front end.

However UML will always be relegated to a design tool only in a dynamic context as it is extremely difficult to provide a quality round tripping experience that will correctly push changes from source back into the graphical model.

UML complexity has increased

UML vendors have had the capability to round tripping UML graphical model and the source code for over a decade. By its very nature the graphical model will always be a high level abstraction of the underlying source code. To reduce the information lost in the model UML been continually expanded to provide lower level primitives. Eventually this has resulted in Model Drive Architecture where the UML model itself becomes an executable as it can fully describe the problem domain. For teams that aren’t using MDA and thus do to need the low level UML primitives UML has significantly grown in complexity without providing a tangible benefit. In comparison to the late 90s where UML diagrams were universally understood across the industry it is now fragmented.

UML doesn’t impart/impose good design principles

By design UML doesn’t impose design rules on the models that are being created. An overly coupled design with low cohesion is just as valid as one with strong interface boundaries and high cohesion. Some tools have added wizards that generate objects based on common design patterns but ultimately the designer needs to understand how and when these patterns should be used. Further most of the patterns are designed to reduce complexity in a growing code base and a developer will refactor towards the pattern as opposed to the design including the pattern from the start.

An Output to Input diagram traces the outputs of a system back to its inputs. It is similar to navigating with a map where you start at your destination and work back towards your starting point. The notation uses circles to represent nouns and the arcs are used to represent either:

· “Composed of” relationships

o

o Has

o Made from

o Combined with

· Or “Created by” relationships

o

o ETL

o Events

Let’s assume we need a birthday cake for a party and since I’m not a good cook I decide to buy one. So we start by with our output Birthday Cake and work backwards. The Birthday Cake is composed of a normal Cake, a birthday greeting written in icing and some candles. I bought the cake using a Purchase Order which included a Birthday Greeting that I wanted written on the cake in icing.

Figure 1 Buy cake

If I was a little more adventurous I could bake a cake. So once again starting with the Birthday Cake it is composed of a Layered Cake and some Candles. The Layered Cake was made by layering Icing, Crème and Jam and Cake. The Cake was baked from a Mixture of Flour, Egg Whites, Sugar and Milk. Finally the Egg Whites were extracted from Eggs.

Figure 2 Making a birthday cake

As you can imagine we can keep extending the diagram further and further to the right we will see how the factories process raw materials from into flour, sugar and milk.

Benefits

1) Promotes the concept that there can be many different solutions to get to the same output

2) Discussion and analysis isn’t constrained by the starting position

· If you start at a cake shop you will never come up with a design to make the cake

· If you start with ingredients you might miss the opportunity to buy a fancier cake than can be made outside of a bakery

3) Naturally breaks a problem into a data or process flow

General Principles

1) All arcs into a node should be of the same relationship type

· Node is Composed of its children

· Node is Created by/from process

2) As you move towards the right the level of granularity should be getting finer

· If the output is a Car has wheels and wheels have bolts

3) A single node in the middle of the diagram that everything is channelled through is probably the wrong

· Might be too granular and should be broken up

· Node possibly is trying to model a process and thus should instead be an arc

Arc Attributes

Once the diagram has been created and we are happy that the model is complete you can start tagging the arcs with descriptive text.

How to use

The diagrams are typically used for requirements gathering process brainstorming sessions with business users. I usually start with the Cake example on the whiteboard and let the users come up with their own Nouns to use.

Phrases to elicit discussion:

1) Can be broken into…

2) Consists of…

3) Once approved is…

4) Waits until…

5) Is combined with…

Once the diagram has been created I go back over all of the arcs and determine:

· Are they a “Composed of” or “Created by” relationship?

· Can the step be automated or is it manual?

You can then start grouping Nouns and Arcs and determine:

· Should they be part of the same IT solution?

o Since you can keep going right until the start of the Universe it likely that a diagram will cross multiple applications or business processes

· What do they nodes map to?

o Systems

o Objects

o Data flow

o ETL

o An aggregate root

Comparison to Input to Output diagrams

Inputs to Output diagrams (Sequence, flow charts etc) can:

· Produce a poor result if the wrong starting point is chosen

· Promote the idea that every process only has a single “starting” point

· Be poor at capturing all of the transitions involved in a process or system

· Make processes that could be performed in parallel appear serial

Limitations

1) Poor choice for looping or iterating processes

2) Notation purposely doesn’t differentiate between entities that are in different states (i.e. light is on or off) to entirely different entities. That decision making process is considered low level design.

I work in an agile environment where the majority of our design and requirement artefacts are photographs of whiteboard sessions with our users and colleagues. These uploaded into our requirement tracking tool and attached to the relevant user story.

The two diagrams that we use on a daily basis are:

· Output to Input diagrams which we utilise for requirements gathering

· Color-UML which overlays UML with design rules

Recently I have been training some business analysts and dev leads that have joined the team and I will be posting the training material here.

I have been an avid follower of Herb Sutter’s writings for years, from his magazine articles in the C++ Report and the C/C++ Users Journal, his books on C++ development and recently his series of posts on parallel programming and concurrency for Dr Dobbs.  Herb is the chair of the C++ standards committee where he and others are near to finalising C++0x standard.  In essence Herb knows his stuff.

Herb recently announced C++ AMP at the AMD Fusion conference and I was blown away.  Microsoft have modified their C++ compiler so that it can target both the CPU and a DirectX compatible graphics card within the same program.  This enables a developer to write code that utilises the hundreds of cores available on graphics cards, to perform blazingly fast parallel computations, without resorting to the current practice of writing code in a language designed for graphics processing (High Level Shader Language [HLSL] or GL Shading Language [GLSL]) or within a C like language (CUDA or OpenCL) where the OO benefits of C++ are missing.

Instead the keyword restrict is used to mark a method as being able to be executed on the graphics card.  The developer then uses a subset of the C++ language (enforced by the compiler) to implement the computation.  The complexity of moving data between the CPU, RAM and the GPU is also simplified by the introduction of a set of classes that automatically handle the marshalling of data between the devices.  The compiler converts the C++ code into HLSL code which is then embedded into the executable.  At runtime the HLSL code is sent to the DirectX driver which in turn converts it into the appropriate machine code (for a particular device) and executes it on the graphics card.

The example below (from Daniel Moth’s presentation - URL below) is a simple matrix by matrix multiplication.  Its clear that the outer for loop can be parallelised.  An interesting point is the complex indexing to pull out values from vA and vb which are both one dimensional arrays but actually store a two dimensional structure.

void MatrixMultiply( vector<float>& C,
const vector<float>& vA,
const vector<float>& vB, int M, int N, int W )
{
for (int y = 0; y < M; y++) {
for (int x = 0; x < N; x++) {
float sum = 0;
for(int i = 0; i < W; i++)
sum += vA[y * W + i] * vB[i * N + x];
vC[y * N + x] = sum;
}
}
}

void MatrixMultiply( vector<float>& vC,
const vector<float>& vA,
const vector<float>& vB, int M, int N, int W )
{
array_view<const float,2> a(M,W,vA),b(W,N,vB);
array_view<writeonly<float>,2> c(M,N,vC);
parallel_for_each(c.grid, [=](index<2> idx) restrict(direct3d) {
float sum = 0;
for(int i = 0; i < a.extent.x; i++)
sum += a(idx.y, i) * b(i, idx.x);
c[idx] = sum;
}
);
}

There are millions of lines of C++ code used in finance industry for modelling derivatives and many of the larger institutions are now looking at GPGPU technology to give them an edge in the competitive world of algorithmic trading.  CUDA is currently the default choice for GPGPU development it will be very interesting in twelve months to see if this trend reverses due to both the simplicity of parallelising a function using AMP C++.

Herb’s keynote.

Daniel Moth: Blazing-fast code using GPUs and more, with C++ AMP

NVidia will also support C++ AMP but points out CUDA is available for Linux and Mac as well as Windows

Finally Microsoft have indicated that they will take their C++ extensions to a standards board thus enabling other compiler vendors to implement the restrict keyword in their products.  Hopefully this will mean that in a couple of years it will be possible to have C++ code target other devices such as FPGAs, PS3’s cell processor and Intel’s Many Integrated Cores (MIC) daughter board.

Side Note: 

Eventually all good ideas come back again. When C with Classes was being created it had a readonly and a writeonly keyword and when C++ was created the C standard committee liked the concept so much they asked that the readonly keyword by renamed to const and added it to the language. Meanwhile writeonly was dropped entirely. Instead of adding writeonly as a keyword Microsoft have gone down the library root i.e. writeonly<T> instead of writeonly T. Personally I think a keyword would have been far cleaner but then again I don’t work on the standards committee and have no idea how complex it would be to add into the official ISO C++ standard…  But considering how long its taken to complete this version of the standard I cant say I am terribly surprised...

Your an IT manager at a small manufacturing company and have a staff of 5.  In two months time your team is slated to replace the internal HR web site that is currently running on Classic ASP. The Architect and lead developer have selected the following software stack:

• ASP.NET MVC
• Entity Framework
• .NET 4.0

The team is very experienced however for the last 2 years they have been maintaining the sales web site which runs on ASP.NET and .NET 2.0.  Due to production support constraints you are unable to send the team on an externally run training course… 

Over my career I have seen this situation played out far too many times.  Teams spend the first couple of weeks or months learning on the job and end up having to replace most of the code that was written at the project’s commencement.

A Coding Dojo is a meeting where a bunch of coders get together to work on a programming challenge. They are there have fun and to engage in DeliberatePractice in order to improve their skills. (ref Coding  Dojo)

What’s the best way to learn a new technology, pattern or programming language?  By playing, doing, prototyping, fiddling and experimenting with it.  What’s the best way for a team to learn a new technology, pattern or programming language? By doing it together!!

Coding Dojos rules:

• A challenge isn’t solved or challenge beaten without code i.e. A paper design doesn’t cut it
• Code doesn’t exists unless there are tests
• Work as a team/hook up with an expert

Formats:

• Prepared Kata – Presenter solves the problem in front of an audience but can/should/will change his solution based on ideas/enhancements/feedback from the audience
• Randori Kata – A pair of developers are selected from the audience.  The pair work together together to solve the problem and every 15 or 20 minutes one of the pair is swapped with another audience member.  Preferably everyone in the audience should have an opportunity to sit at the computer and bang out some code.

In a Coding Dojo “meet up” the Randori Kata can be intimidating for a new comer – No one wants to get a mental blank in front of a group of people or have a group commenting of their coding style.  This should be less of an issue in a work environment where the developers already know each other.  In fact the format ensures that everyone participates in the learning experience.

Workplace <Insert Technology here> Coding Dojo:

1. Prep work
   1. Find the most experienced person in the team who has already used the technology
   2. Ask them to create a 30 minute demo (PowerPoint is fine) where the technology is used
   3. Ask them to create a “Cheat sheet” that consists of 3 or 4 pages of generic code snippets that show how the technology is used for different situations
   4. Create a coding problem (don’t tell the team or the person that wrote the Cheat Sheet) that can be solved reasonably easily using the technology and ensure that the solution to the problem utilises items within the Cheat Sheet
2. Kata
   1. Have the team member present the 30 minute introduction demo
   2. Provide everyone with a copy of the “Cheat Sheet”
   3. Spend 10 minutes discussing the technology and the contents of the Cheat Sheet
   4. Explain the coding problem
   5. Randomly chose a developer to join the presenter – Coding in a pair is less stressful
   6. Have the presenter/colleague pair start solving the problem (make sure they write tests)
   7. Following Randori Kata rotate through the developers every 15 to 20 minutes
   8. After two hours of coding stop the clock
   9. Spend 10 minutes in a discussion
      1. What was hard?
      2. What was easy?
      3. How confident do people feel?
      4. What else do we need to learn?

During the week I presented an introduction to WPF & Silverlight using the Model View ViewModel pattern.  The audience was a group of developers that had been working on a .NET 2.0 based application for the last 5 years and were going to transition to .NET 4.0.  Hence the session material was designed to briefly touch on the main WPF concepts and allow them to drill into the topics in more detail at their own leisure.

Enjoy…

WPF & Silverlight Development using Model View ViewModel.pptx (164.07 kb)

After living in the UK since mid 2008 my wife and I have moved back to Sydney Australia.  We are also bringing back our baby boy Logan who is the reason I haven’t blogged in almost a year.  I’m looking forward to learning about the Open Source, Alt.NET and Agile community in Sydney.