David Ross's Blog Random thoughts of a coder

Although there has been a mind blowing amount of technological change since the beginning of the PC revolution when it comes to building enterprise applications the industry has moved through a small number of distinct architectural patterns.

• Compute power on the client has been continually increasing leading to a migration of processing activities from the mainframe, to the application server, to the desktop or mobile device
• Reducing deployment complexity by shifting from think clients, to web and recently to App Store delivery models
• The quality and depth of functionality available in libraries, frameworks and platforms has meant that smaller teams are able to deliver more complex systems

This post documents the main enterprise architectures that have been utilised over the last twenty years.  By placing the system under review into one or more of these categories will enable the reviewer to quickly hone in on possible architectural flaws and forms the foundation of the review process.

Client Server

Client server based applications such as those developed with Access, Power Builder and classic Visual Basic have client application directly accessing a shared resource such as a database. Unfortunately users would realise they could bypass the GUI and interact directly with underlying database leading to data corruption and/or security issues. While IT departments no longer create Client Server applications Excel spread sheets that source data from a database are effectively equivalent.

• Client software connects directly to database
• Without a layer of stored procedures between the client and the database the design by today’s standards is intrinsically insecure
• Stored procedures need to perform authorisation checks as well as requested business operations to be secure
• Difficult to refactor database as simple changes would affect multiple stored procedures
• Extremely difficult to scale since
• Hard to utilise a caching layer as clients aren’t channelled through a shared service
• Lots of separate connections to the database as connection pooling isn’t utilised

3 Tier

It quickly became apparent that having the client applications directly access the database was problematic. There were performance issues due to the slow client machines available at the time deployments suffered from DLL hell. Simple Line of Business applications starting moving to HTML and more complex GUI’s utilised COM+ to centralise database interactivity to the middleware servers.

• Client software connects to an application server
• Thick clients connect to COM+ or Web Services
• Browser loads HTML pages fully rendered on the server
• Application server is responsible for all connectivity to the database
• Application server is trusted
• User Authentication and Authorisation rules can be pulled out of the database and into the middle tier
• Database tables can be interrogated directly as opposed to utilising stored procedures and still be secure
• Interop issues
• COM+ based solutions were incompatible with Java
• Original Web Services frameworks generated poor quality code and system to system integration proved more difficult than industry expected

N-Tiered

As Web Pages became more complex having rendering code intermingled with database queries became unwieldy. Presentation Patterns were developed that would take a “model” and convert it into XML or HTML. Business Services would interrogate the database and return a “model”. This layering enabled multiple screens to utilise common business components.

• Simplifies development be splitting the software on the application server into logical layers
• Code to render the presentation layer is separate to the code that implements business rules
• Performance increased by adding new layers such as caching
• Often business components will “know” how to retrieve themselves from the database and how to save themselves
• Can’t test application logic without a database
• Can’t change persistence layer easily
• Some systems wrongly placed application layers on separate machines
• Web Server talks to an App Service talks to database
• Latency increases
• Transaction management extremely difficult

Rise of Object Relationship Mappers and Domain Driven Design

A problem with N-Tier architectures was that the Business components had a deep understanding of how they were persisted to the database. For example a request would come into a Web Service that would then call a Business Component, who would then call into a Data Access Layer, which was built using the Table Gateway Pattern. In the Table Gateway Pattern the developer would expose Select, Delete and Update methods for each table within the database and the classes in the domain would map almost exactly to a database table. Thus the business domain components were extremely database centric. Performance issues would arise when data straddled multiple tables as the Table Gateway Pattern struggles with this type of topology leading to either many views being created or the “SELECT N+1” anti-pattern. As Object Relationship Mappers and Domain Driven Design became widespread the shape of the data between the domain layer and the database started to diverge. Domains built with “Plain old Java/C# Objects” POJO/POCO removed the dependency on the database eventually leading to teams utilising NoSQL and Event Sourcing in their designs.

• POJO and POCO classes
• Domain objects are inflated externally by a repository or in the service layer
• Domain objects should be persistence layer agnostic
• Domain objects should not “know” how to connect to database
• Domain objects should only contain data, business rules and validation
• Enables unit tests to be simpler and not have database connectivity
• Once connection to database is broken developers start utilising other persistence stores such as No SQL, files, Web Services etc

Use of Open Source frameworks

Throughout the industry enterprise software development teams found that they were writing plumbing code to fill in gaps in the programming environments that they were using. Eventually Open Source projects have become so advanced and reliable that the entire development stack is now open source for many platforms. Legacy code can often be “modernised” by gradually replacing bespoke plumbing components with the Open Source equivalents.

• Teams no longer wrote cross functional libraries and instead use industry standards for:
• Logging
• Database connectivity
• View Engines
• Aspect Orientated Programming
• Inversion of Control

Continuous Integration and Deployment

While Continuous Integration is not an enterprise pattern the use of automated unit tests and deployments is a distinct stage within the progression of enterprise application architectures. Test Driven Development, Test First Development and Acceptance Test Driven Development all promote the creation of decoupled independent components. Due to the automated regression suite developers are able to make large systemic changes with relatively low risk leading to simpler more maintainable systems that will last longer in production. This has led to shorter development cycles and higher quality solutions.

• Automated builds
• Automated deployments
• Automated tests
• Developer created
• User created

HTML5 and mobile

While HTML5 can be described as the latest HTML standard where it provides richer access to the sensors on the device, higher speed graphics and the ability for applications to function in an occasionally connected environment for many development teams however it is the “line in the sand” where supporting older browsers is no longer necessary. Further Chromes decision to auto update has also contributed to all browser vendors to release far more rapidly and modern frameworks degrading gracefully when running on older browser are also contributing to making web development far easier.

The biggest failure of the Java desktop was the design choice of “Write Once Run Everywhere” which led to the decision to only support controls that were available on all of the platforms that run Java (UNIX, Windows and Mac). Unfortunately this inevitably led to a poor desktop experience on all platforms. Unfortunately teams are now going through the same dilemma today with mobile where HTML5 only provides a subset of the features available on Andriod, IOS and Windows Phone and has slower performance than the native frameworks. Further the look and feel of each of the platforms is significantly different and a web application can look out of place or be confusing for users. This has led to many teams building having to build multiple versions of their client application. This in turn further reduces the role of the Web Server to simply providing data to the client which is solely responsible for the rendering.

• Fast CPU and GPUs on clients means processing can be moved off servers and onto the clients
• Middle tier exposes data and services as opposed to rendering screen layouts
• Model View Controller (MVC) and Model View ViewModel (MVVM) frameworks with two way data binding reduces hand crafted client side code
• Front end code complexity increases to level where unit test of client becomes a necessity
• Since mobile platform’s native presentation framework provides a richer and more integrated user experience than the HTML equivalent teams are forced to create multiple clients for their publicly facing applications.

Big Data

Big data is becoming a significant new stage in enterprise application development. Due to the huge volumes of data streaming into an organisation retrieving and interacting with raw items of data is meaningless. Instead the data needs to be transformed, aggregated and grouped before it can be analysed and many designs include multiple persistence stores.

• Data quantities are so large that classical data processing techniques cannot be used
• Data is continually pre-processed and aggregated
• Clients rarely interact with raw data but utilise aggregates
• Raw data is not thrown away but is available forever as each different analysis tasks will need the data to be projected using a different manner
• Data schema is less relevant as the analysis process overlays the data with a custom schema necessary for the problem that is being solved
• Multiple persistence stores each solving a different problem are utilised on a single project

In most large enterprises there will be hundreds or thousands of applications within their portfolio and with the incredible pace of change within the technology sector the vast majority of those applications will:

• have been developed using out dated techniques
• been built with languages or tool chains that are no longer supported
• have little or no documentation
• have a poor user experience
• have fundamental architectural flaws
• insecure
• poor transaction management
• don’t scale

Even though they might have faults or be technically impure they “Run the Business” and for large enterprises without their bespoke applications they would literally cease to exist. Hence every couple of years an architect or senior developer will be asked to do an architectural review to determine whether a particular application can hobble along for another couple of years as is, if some investment needs to be made to refactor or uplift the design or whether the organisation should explore decommissioning it entirely.

After partaking in many of these reviews I have developed a semi-formal process based around answering the following questions:

1. What enterprise application architecture was used to create the solution?
2. Did the architecture make sense when it was originally designed?
3. How does the architecture compare to a system designed today?
4. Are there any fundamental issues with the implementation?
5. Is it possible to refactor the code base into a more modern or clean one?

Over the next few posts I will be giving some background to each of the questions and then provide a set of review checklists.

Inheritance versus the strategy pattern

Broadly the patterns described in the “Gang of Four” patterns book (Design Patterns: Elements of Reusable Object-Oriented Software) can be broken into two groups. The first group provide techniques to manage an object’s lifecycle and the second group is about splitting an object’s responsibilities into a group of interconnected smaller objects which should simplify the design. Many of the lifecycle patterns (singletons, factories etc.) are found in modern Inversion of Control containers and no longer need to be implemented by hand. Meanwhile the second group are utilised in the design of a rich domain layer.

Consider a bank that has a range of savings account products with each having a slightly different method for charging fees. Some fees might be based on the amount of money in account, others based on the number of transactions and finally some premium based products might have a flat monthly fee. When developers first learn an object orientated programming language this type of problem is solved in one of two ways:

1. Implemented using procedural techniques
   • The Fee Calculation logic is placed into the Bank Account class and is implemented using cascading if-else or switch statements
   • Adding a new Fee Calculation involves modifying the Bank Account class
2. Implemented using inheritance
   • The developer realises that are five or six distinct bank account types
   • A base class is created where the common aspects of all of the account are stored
   • An abstract CalculateFees method is placed on the base class which the children override
   • Adding a new Fee Calculation involves creating a new leaf class meanwhile the base class is not modified

The inheritance based solution is far superior to the procedural alternative since new bank account products can be added to the solution within necessitating any changes to the base class.

Unfortunately the inheritance design has some issues:

• Hard to model bank accounts that have a blended fee structure without multiple inheritance
  • Retail Plus account combines aspects of the Premium account with the Basic Saver Account
  • Debit account is the same as a Retail saver but without overdraft facility
• Inheritance can’t be switched on and off its an integral part of the objects identity
  • Act as a Premium account until number of transactions is over 20 than act as a normal account

An alternative to inheritance is to utilise the strategy pattern and to move the Fee Calculation engine out of the Bank Account object and to place into a new group of objects that share an interface. The Bank Account then delegates to a collection of Fee Calculation objects instead.

This approach:

• Enables an account to have zero, one or many fees structures
  • Allows the creation of a rules engine that allows Fees Structure to be prioritised or combined easily
  • Scales to allow for any type of behaviour that is optional or time dependent to moved out of the object and into a strategy
• Tightens up the definition of a Bank Account
  • A collection of transactions that are associated with a customer
  • Business rules can be applied to the account that generate transactions which in turn allows interest to be paid and fees to be charged

After learning the strategy pattern developers will quickly find that they begin pushing inheritance out of their entity classes and into behavioural components instead. Thus the inheritance hierarchy (this includes the implementation of an interface) is used for allowing optional runtime polymorphic behaviour as opposed to modelling an object’s type.

UML with Colour

One of the difficulties when using UML is that it doesn’t validate the quality of the design or provide guidelines to aid in the modelling process. UML with Colour (described in Java Modeling in Color with UML) meanwhile overlays UML with a set of simple design rules for building out the logical domain model. The logical model that is created is similar to 3^rd or 4^th normal form in database modelling as the resulting object model is highly decoupled.

UML with Colour is based on the premise that domain objects are split into three groups.

Party, Place or Thing objects have Role objects which modify or create Moment-Interval or Transaction objects

Example:

• <People, Person or Thing> - <Role> - <Moment-Interval or Transaction>
• Manager – Approves – Timesheet
• Customer – Buys – Products
• Trader – Cancels - Trade

Party, Place or Thing

The Party, Place or Thing archetype is used to represent all of the system’s actors. UML with Colour ruthlessly removes many of the places where inheritance is used in modelling actors by stating that a Party, Place of Thing cannot manipulate or create Transactions directly and instead this work should be delegated to a Role. When modelling the users of a system instead of having a deep hierarchy that includes managers, team leads, customer service, administrators etc. there might only be a staff object and all of the different capabilities and responsibilities of a manager or administrator is modelled as Roles.

Typically Party, Place or Thing objects:

• Will have a unique identity across the application
• Can be an aggregate root

Role

Where UML with Colour differs from classical domain modelling is that it makes the strategy pattern a core part of the design process. Consider a requirement where a manager needs to approve timesheets in a HR application. Instead of putting an Approve timesheet method on a Manager object the design starts with a Person object that interacts with a Time Sheet Approver Role. Thus a Role is an object that creates or manipulates a Transaction on behalf of a Party, Place or Thing. As discussed removing this logic from the Party, Place or Thing objects has a number of benefits with the most important being that it allows for capabilities to become optional and combinable.

Typically Role objects:

• Will implement an interface
• Will often be a factory

Moment-Interval or Transaction

The Moment-Interval object represents an event or transaction.

Typically Moment-Interval objects:

• Include a timestamp, sequence number or time period as part of their identity
• Often have a collection of children such as a Purchase Order has Purchase Order Line Items

Description

There is a fourth archetype in UML with Colour called Description which is an object that stores metadata about another object. For example a car might have attributes including its registration number, amount of fuel in the tank and miles on the odometer. In addition there are attributes that are shared by all cars of the same model such as its manufacturing year, make, model and colour. These should be placed into a Description object to ensure that the data is shareable, not duplicated and most important recognises that the data is intrinsically immutable.

Typically Description objects:

• Are value objects
• Are similar to dimension tables in a data warehouse
• Are attached to Party, Place or Things and Moment-Intervals but not Roles

Colours?

The Colour part of UML with Colour is simply that within the UML diagram each of the classes in the diagram is shaded with a different colour representing each of the archetypes. The colourisation ensures that the basic design rule whereby a Person, Place or Thing can’t connect directly to a Moment-Interval is enforced.

The four colours have been purposely chosen so that they are available within a pack of coloured post-it notes. The design process will often consist of sticking post-it notes to a whiteboard and drawing arrows between them.

• Role is yellow
• Moment-Interval is pink
• Description is blue
• Party, Place or Thing is Green

Logical to Implemented Domain Model

Once the UML with Colour diagram is completed the developer is then able to decide how the model should be implemented.

The three choices are:

• Implement classes identical to those on the logical model
• Denormalise the design and utilise inheritance so that each Role of a similar type becomes an inheriting class off the Party, Place and Thing class
• Implement the capability of the Role directly in the Party, Place or Thing class

It might sounds strange after going through a process where we extract behaviour out of entities and into separate classes the design process then advocates that the developer has the opportunity to collapse all the responsibilities back into a single entity. However this follows the tradition of all refactoring processes in that they are by-directional. For example code can be pushed out of a function using the Extract Method refactoring and in turn if a function is no longer useful as an abstraction its contents can be pasted into its callers.

Examples:

• If there are many Roles that are optional leave the design as is
• If available Roles can change for the same Party, Place or Thing instance leave the design as is
  • Trades can be approved by all team members until midday after which time only a manager can approve the trades
• If a Party, Place or Thing class only has a single Role or a group of independent Roles that act on different Moment-Intervals that behaviour is not optional the Role should be merged back into the Party, Place or Thing
• If there are a set of Roles that provide similar functionality and are not optional consider removing the strategy and inherit off the Party, Place or Thing class.

Further as we have seen it is trivial to refactor from a de-normalised model into a normalised model that utilises separate Role classes i.e. it is possible to refactor from a design where everything resides in a single class, to having one with a base class with inheritors to one where the design relies of behavioural patterns like the strategy.

Limitations

• UML with Colour is focussed on domain objects and is a poor choice for modelling the interaction between layers or systems in an enterprise application.
• In initial discussions with the business it is preferable not to discuss concept of Roles
  • Business wants to discuss problem domain using titles such as manager and team member
  • In comparison calling managers, Staff with Approver or Proposer Roles can be initially confusing
• Not a common technique
• UML itself isn’t as popular as it once was
• UML with Colour is not widely known as it was introduced in a Java design book as opposed to be championed for all developers

Due to UMLs many faults for many years I had stopped using it as part of the design tools. A couple of years ago a colleague introduced me Colour-UML and his design approach which I now utilise on most of my projects.

UML should primarily be used as an exploratory design tool in a workshop environment with the intent that the artefacts created being quickly converted into code.

UML usage is limited to the domain entities and view models

• The team should be focussed on creating a rich domain model using a ubiquitous language
• Aggregate roots should be identified but in a post-LINQ world repositories are rarely required nor need to be designed
• Presentation Services/Controllers and Business Services are not modelled in UML – Instead only a single example of a presentation or architectural pattern needs to produced just to demonstrate the approach
• View Models are only modelled if there is a complex tree of View Models such as those used in Single Page Applications

UML diagrams are created on whiteboards, butches paper or Visio

• Don’t use UML to generate code
• Don’t generate UML diagrams from source code

Long lived UML artefacts are stored as pictures

• If a design has been completed on the whiteboard the design team can photograph it and upload into the Wiki
• UML diagrams are allowed to become “Out of Date” the source of truth is the source code
• Best diagrams are those stuck on the wall next to the development team

Only design objects relationships and core fields

• Diagram should show a Customer has an Address but shouldn’t detail that an Address contains a Postcode, Street, Suburb etc
• Agile principles in fields are only added to entities as they are required hence no need to model them “Up Front”
• Adorn the relationship with its multiplicity (0..1, 1, *)

Objects should be modelled using the following Colour-UML archetypes

• Person, Place or Thing
• Role
• Interval-Moment
• Description

UML

During the 80s and 90s there were around a dozen popular notations for graphically describing the relationship between objects. Booch’s notation mapped closely to C++. Coad’s notation was an extension to the modelling techniques he defined for databases. When Booch, Jacobson and Rumbaugh joined forces and merged their notations into the Unified Modelling Language it quickly the standard for Graphical Object Orientated Design and soon it emerged as a core tool for enterprise development.

However the last decade UML is gradually fallen out of favour as and its decline has been heavily influenced by the rise of Agile development and the popularity of dynamic languages such as Javascript. UML itself has ballooned in complexity as tool vendors have tried to move the language from a design to implementation tool.

Big Design Up Front

Waterfall practices in the 80s and 90s resulted in many enterprise environments devaluing the contribution of programmers in their development teams. Instead of the “coders” doing design work this role was performed by architecture teams who would deliver UML or Entity Relationship diagrams and developers were relegated to being low skilled implementers. These diagrams not only showed the relationships between entities but also fully detailed their attributes and would take many months of analysis to complete. Further since the source of truth was the diagrams and not the source code large amounts of effort were spent trying to keep the diagrams and source code aligned.

As Agile started to emerge across the industry developers were rightly given the freedom to implement requirements as they saw fit. Further refactoring techniques now mean that objects are continually being renamed, split, merged and deleted and trying to keep the UML diagrams aligned with code base is ultimately a fruitless exercise.

Feature driven development

Team that use a story or feature driven development approach naturally build up the systems capability in increments. This effectively means that only a small number of objects are being created during each iteration and thus the amount of design work required is far lower.

Rise of dynamic languages

Whether an object was created within a dynamic or static language its makeup is still defined by its fields, properties, methods and its relationship with other objects. Hence UML can still be valuable to model backend server side components in a node.js, Ruby or Python project it can also be a useful aid in modelling complex interactions within a tree of View Models on the front end.

However UML will always be relegated to a design tool only in a dynamic context as it is extremely difficult to provide a quality round tripping experience that will correctly push changes from source back into the graphical model.

UML complexity has increased

UML vendors have had the capability to round tripping UML graphical model and the source code for over a decade. By its very nature the graphical model will always be a high level abstraction of the underlying source code. To reduce the information lost in the model UML been continually expanded to provide lower level primitives. Eventually this has resulted in Model Drive Architecture where the UML model itself becomes an executable as it can fully describe the problem domain. For teams that aren’t using MDA and thus do to need the low level UML primitives UML has significantly grown in complexity without providing a tangible benefit. In comparison to the late 90s where UML diagrams were universally understood across the industry it is now fragmented.

UML doesn’t impart/impose good design principles

By design UML doesn’t impose design rules on the models that are being created. An overly coupled design with low cohesion is just as valid as one with strong interface boundaries and high cohesion. Some tools have added wizards that generate objects based on common design patterns but ultimately the designer needs to understand how and when these patterns should be used. Further most of the patterns are designed to reduce complexity in a growing code base and a developer will refactor towards the pattern as opposed to the design including the pattern from the start.

An Output to Input diagram traces the outputs of a system back to its inputs. It is similar to navigating with a map where you start at your destination and work back towards your starting point. The notation uses circles to represent nouns and the arcs are used to represent either:

· “Composed of” relationships

o

o Has

o Made from

o Combined with

· Or “Created by” relationships

o

o ETL

o Events

Let’s assume we need a birthday cake for a party and since I’m not a good cook I decide to buy one. So we start by with our output Birthday Cake and work backwards. The Birthday Cake is composed of a normal Cake, a birthday greeting written in icing and some candles. I bought the cake using a Purchase Order which included a Birthday Greeting that I wanted written on the cake in icing.

Figure 1 Buy cake

If I was a little more adventurous I could bake a cake. So once again starting with the Birthday Cake it is composed of a Layered Cake and some Candles. The Layered Cake was made by layering Icing, Crème and Jam and Cake. The Cake was baked from a Mixture of Flour, Egg Whites, Sugar and Milk. Finally the Egg Whites were extracted from Eggs.

Figure 2 Making a birthday cake

As you can imagine we can keep extending the diagram further and further to the right we will see how the factories process raw materials from into flour, sugar and milk.

Benefits

1) Promotes the concept that there can be many different solutions to get to the same output

2) Discussion and analysis isn’t constrained by the starting position

· If you start at a cake shop you will never come up with a design to make the cake

· If you start with ingredients you might miss the opportunity to buy a fancier cake than can be made outside of a bakery

3) Naturally breaks a problem into a data or process flow

General Principles

1) All arcs into a node should be of the same relationship type

· Node is Composed of its children

· Node is Created by/from process

2) As you move towards the right the level of granularity should be getting finer

· If the output is a Car has wheels and wheels have bolts

3) A single node in the middle of the diagram that everything is channelled through is probably the wrong

· Might be too granular and should be broken up

· Node possibly is trying to model a process and thus should instead be an arc

Arc Attributes

Once the diagram has been created and we are happy that the model is complete you can start tagging the arcs with descriptive text.

How to use

The diagrams are typically used for requirements gathering process brainstorming sessions with business users. I usually start with the Cake example on the whiteboard and let the users come up with their own Nouns to use.

Phrases to elicit discussion:

1) Can be broken into…

2) Consists of…

3) Once approved is…

4) Waits until…

5) Is combined with…

Once the diagram has been created I go back over all of the arcs and determine:

· Are they a “Composed of” or “Created by” relationship?

· Can the step be automated or is it manual?

You can then start grouping Nouns and Arcs and determine:

· Should they be part of the same IT solution?

o Since you can keep going right until the start of the Universe it likely that a diagram will cross multiple applications or business processes

· What do they nodes map to?

o Systems

o Objects

o Data flow

o ETL

o An aggregate root

Comparison to Input to Output diagrams

Inputs to Output diagrams (Sequence, flow charts etc) can:

· Produce a poor result if the wrong starting point is chosen

· Promote the idea that every process only has a single “starting” point

· Be poor at capturing all of the transitions involved in a process or system

· Make processes that could be performed in parallel appear serial

Limitations

1) Poor choice for looping or iterating processes

2) Notation purposely doesn’t differentiate between entities that are in different states (i.e. light is on or off) to entirely different entities. That decision making process is considered low level design.

I work in an agile environment where the majority of our design and requirement artefacts are photographs of whiteboard sessions with our users and colleagues. These uploaded into our requirement tracking tool and attached to the relevant user story.

The two diagrams that we use on a daily basis are:

· Output to Input diagrams which we utilise for requirements gathering

· Color-UML which overlays UML with design rules

Recently I have been training some business analysts and dev leads that have joined the team and I will be posting the training material here.