Architecture essays
The perspective
IT systems interact and therefore integrate for the same reasons as people.
- When we need some information from others for our work
- When we need someone to do something for us
- When we want to socialise with someone
The corresponding IT examples are an API call for a query, an operation, and a keep-alive ping/health check (with AI, true socialising between IT systems may also appear). They are also the most to the least common types of interaction.
In the enterprise, we are concerned with the interaction between technological systems and the humans involved in business processes. The latter can be customers, staff, and partners. Not everything can be automated, and there are humans at the ends or within sequences of events, and we need to make them equally a part of the solution.
Integration of systems is expensive and often the point of failure. Like with people, the following principles make the interaction between systems fast, efficient, and capacious.
- The specialisation of responsibilities (cohesion)
- The unimportance of knowing how something works or provides information (loose coupling)
- A commonality of language (standardised protocols)
How to think of integration
Integration is not just HTTP, REST, and some old FTP or SQLNet in the background. It is an interesting and complex architectural area and discipline. There is no single way of looking at it. I look at it from six viewpoints that allow us to create sound integration architecture requirements, problem statements, and solutions.
These integration viewpoints, from the broadest to the most detailed, are
Scope of enterprise integration → Hierarchy of enterprise integration patterns → Application integration patterns → Horizontal integration of systems’ layers → Stateless vs stateful integration → Integration security
(We will use architecturally relevant terminology from Domain Driven Design, especially the DDD-specific terms for layers — UI, Application, Domain, and Infrastructure. Other DDD terms are italicised. Please refer to my articles on Domain Driven Architecture Part I and II.)
Once you understand integration from the viewpoints below, you can define and solve integration problems holistically, efficiently and effectively.
(All the images used are in this PPT that you can download for better viewing. You may use them with attribution.)
Viewpoint 1: The scope of enterprise integration
It’s useful to start with a view of where the interactions are happening as it influences their nature and solutions.
Intra-Domain
Integrations within a business domain are the most common. They connect related systems having similar architecture and design models (assuming Domain Driven Design). This allows for simpler context maps between bounded contexts and more direct integrations with fewer capacity and scaling concerns: e.g., integration between an Inventory Management system and a Procurement System.
Inter-Domain
Integrations between business domains are fewer but have more challenges. The Ubiquitous Language is likely to differ in each domain, and data payload and response objects often need to be transformed in interactions. The infrastructure may also be separated a little or a lot, introducing the need for several supporting functions (see viewpoint #2 below) for successful integration, e.g., integration between an Order Management System in the sales domain with a Customer Relationship Management system in the customer service domain.
Inter-Enterprise
Integration between partner enterprises in B2B and B2B2C scenarios is increasing exponentially with modern web and digital services. Fortunately, service integration standards and cloud services have kept up with the emerging web, mobile and digital architectures, addressing the needs for reliable, rapidly scaling, secure and agile developing integrated services that span two to scores of companies in ecosystems such as eCommerce, logistics, travel, automotive, manufacturing, airline, banking and financial services.
The picture below illustrates this viewpoint.
Guidelines
Consider each type of integration above separately for its characteristics and the five viewpoints below to tailor the integration architectures.
Viewpoint 2: Hierarchy of enterprise integration patterns
Business function hierarchy
BPM — Business process management coordinates people, systems, information and things to produce business outcomes for the domains and sub-domains of the enterprise. Examples are product design, manufacturing, marketing, customer acquisition, sales, etc. BPM architecture automates the dynamic integration of business processes to the maximum extent possible to provide well-defined, repeatable, efficient and reliable outcomes.
Workflow — A workflow is a defined sequence of tasks carried out by systems and humans working together to deliver a business task. It can be considered a subset of a business process. An example is the approval workflow of a business loan. Workflow architecture integrates IT systems and humans for the initiation, routing, analytical decision-support, manual steps, automated tasks, coordination, and monitoring of workflows.
Systems integration — In a well-designed IT landscape, functionality is divided into cohesive systems mapping to business domain and sub-domain capabilities. These systems must work together to deliver useful outcomes for workflows and business processes. System integration automates their dynamic cooperation. (Read this article for more about the types of systems that need to be integrated → A Practical Abstraction of Functional IT Systems.)
Component integration — All IT systems are internally segregated into cohesive and loosely coupled components and sub-components. These are integrated using standard patterns (see viewpoint #4 below) to deliver the externally useful functions of the system.
The picture below illustrates this framework.
Technical function hierarchy
Integration is expensive and should be as simple as possible. How difficult and trouble-prone it will be depends on the maturity of IT systems (aka applications). When client and server systems/services talk to each other as part of a business process, some or all of the following functions may be required.
- Retrying — the server does not respond on the first attempt
- Messaging — the server cannot be contacted directly
- Load management (queuing, balancing, connection pooling) — the server can respond to a limited number of requests per time unit
- Publish-subscribe — the server has information one or more consumers can take while it is fresh
- Object transformation — the client and server work with sufficient yet differing data packages
- Protocol transformation — the client and server do not use the same communication protocol
- Synch-async conversion — the client expects an online response while the server is designed for an offline response or expects an offline response while the server is designed for an online response.
- Merging of functions or data — the client has to be served from a combination of more than one operation or information repository.
- Routing — the correct server has to be reached based on rules
- Orchestration — a sequence of operations needs to be performed for the response to the client and the logic is in one place. (Choreography is a related concept, where the action and sequence logic are distributed in the participating services; however it’s functional capability is limited, and it usually turns out there is a centralised orchestrator of some form, even if it is only a passive reference ruleset.)
- Version management — different clients need different versions of the same operation
- Security (see viewpoint #6 below) — the client and server need to be secured in one or more ways
The more of these needs the client and server systems cater for internally, the less the integration architect needs to provide for externally. In other words, the application and information architecture significantly influence the integration architecture’s complexity and cost to build and operate.
Usually, there will be some external integration functions that we need to cater for. But the more direct and straightforward the integration, the better.
Intermediary integration solution patterns and platforms
In increasing order of scope, the external integration solution patterns are as below. Strictly speaking, they are not architectural patterns, and some overlap functionally. So grasp the difference in scope of each and use the terms with care.
- RPC — a remote procedure call (RPC) is when a computer program causes a procedure (subroutine) to execute in a different address space (commonly on another computer on a shared network), which is coded as if it were a regular (local) procedure call, to simplify programming and not have to deal with network, protocol, and OS details, etc.
- Intra-app integration — interactions between a system’s UI, application, domain and repository layers (DDD terms; or UI, business logic and database layers in older terminology), e.g., between the UI and business logic layer (using HTTP(s) protocols) and between the business logic and database layer (using TNS/TTC over TCP/IP protocols for Oracle DB).
- Read API (DAL) — a Read API or Data Access Layer is a common integration layer on Operational Data Sources (ODS), Management Information Systems Databases (MIS DBs) and BI business access layers that are accessed by OLTP and OLAP read transactions.
- API —an Application Programming Interface is a software interface for communication between IT systems. An API usually supports multiple operations. The design is guided by the standardisation of the function signatures and network protocols, and an API specification document is published. APIs often have various versions available for different clients. Dynamic discovery and binding of APIs and operations by clients during run time may also be supported.
- API GW — an API Gateway realises a combination of architectural patterns such as facade, adapter, mediator, and (reverse) proxy. It receives client calls and routes them to the services behind it while providing functions such as routing, rate limiting, protocol transformation, aggregation, security, versioning, logging, etc.
- Messaging Systems — the integration pattern of messaging allows systems to be loosely coupled by communicating asynchronously, making communication more reliable because the two systems do not have to run simultaneously. The messaging system is responsible for transferring data from one system to the other, so they can focus on the information they need to share and not manage the interaction actively. It’s like a letter postal service.
- (Message) Queuing Systems — a queuing pattern extends the asynchronous communication pattern of Messaging by providing a pipeline where multiple messages can be stored sequentially by client or server systems until they are consumed, usually on a First In, First Out (FIFO) basis, or become stale. Request messages are put into queues by client systems and picked up and serviced with messages on response queues by serving systems. Queues increase reliability and net throughput and reduce data loss. This loose coupling allows independent and agile development of systems. Managing the queues and their contents, ageing, scaling, logging, etc., are functions provided by the message queueing platform. It’s like queuing up to buy a ticket for a movie.
- Publish Subscribe Systems — a publish pattern is an asynchronous and loosely coupled communication pattern for a serving system to make information or functions available that interested clients can subscribe to. It’s like newspaper publishing and subscription by interested readers.
- ESBs — an Enterprise Service Bus combines multiple integration functions, such as messaging, queuing, publish-subscribe, routing, protocol transformation, object transformation, etc., in one software platform. They are one of the most common EAI platforms (see below).
- EAI Platforms — broadly, Enterprise Application Integration platforms can provide all the 12 integration functions covered in the Technical Functions Hierarchy section above and solution types 2 to 9 above.
- Workflow Managers — Workflow platforms integrate the systems and humans for the initiation, routing, decision-support, manual actions, automated jobs, coordination, and monitoring of the sequence of steps carried out by systems and humans together for a business task.
- Business Process Managers — BPM platforms automate the dynamic integration of business processes to provide well-defined, repeatable, efficient and reliable outcomes. They often cover discovery, analysis and optimisation services.
Guidelines
What is required for integration emerges organically from the application and information architecture and the business processes and workflows they service. But sometimes, there is a decision to be made whether we should refactor applications for better integration or use an external platform.
The litmus test pictured below helps us to score the need for external support. If the score is low, refactoring the applications is advised.
Viewpoint 3: App integration patterns
From the very outset, application design patterns have included integration patterns. Some of them are sound for architectural thinking, and we adapt and adopt them. The ‘Gang of Four’ book on Application Design Patterns covers the following architecturally relevant ones that are briefly described. As an application and integration architect, learn and use these patterns.
Facade — Provides a unified interface to a set of interfaces in a subsystem. Facade defines a higher-level interface that makes the subsystem easier to use, e.g., an API facade for Order collates the interfaces of multiple underlying sub-systems such as ManageStock, CheckPayment, FulfilOrder, etc.
Adapter — Converts the interface of a class into another interface clients expect. The adapter lets classes work together that couldn’t otherwise because of incompatible interfaces. In architectural terms, think system, application, component or sub-component instead of class, e.g., a PaymentGWAdapter fills in a mandatory clienttype field required by a payment system interface with a default value for a client that does not supply it.
Proxy — Provides a surrogate or placeholder for another object to control access to it. Functionally, they can be further divided into Remote Proxy (local representation of a remote API), Virtual Proxy (provide caching and on-demand backend access), Protection Proxy (control access based on roles and rules), and Helper Proxy (provide additional services such as counting, logging, etc.), e.g., a CDN (content delivery network) uses a reverse proxy for security, load balancing, and scaling.
Observer (aka Publish-Subscribe) — Defines a one-to-many dependency between objects so that when one object changes state, all its dependents are notified and updated automatically. (The Event Driven Architecture pattern is a case of observer or publish-subscribe application integration pattern.) E.g., if an airline (subject) flight schedule changes and several travel portals (observers) need to know about it, an Observer or Publish-Subscribe pattern integrates them using notifications, requests, etc.
Mediator — Defines an object that encapsulates how a set of objects interact. Mediator promotes loose coupling by keeping objects from explicitly referring to each other and letting you vary their interaction independently, e.g., An eCommerce platform talks to several payment gateways through a mediator so they can all change independently of each other.
Anti-corruption Layer (this pattern comes from DDD) — When systems based on different models are combined, the need for the new system to adapt to the semantics of the other system can lead to a corruption of the new system’s model. Create an isolating layer to provide clients with functionality in terms of their domain model. The layer talks to the other system through its existing interface, requiring little or no modification to the other system. Internally, the layer translates in both directions as necessary between the two models, e.g., A mobile banking application talks to a legacy core banking system through an ACL that separates credit checking from orders that are combined in one function in the legacy system.
Guidelines
Adopt Domain Driven Architecture Design as it naturally leads to the bounded contexts, sub-domain system architectures and context maps that guide the choice of the intra- and inter-system integration patterns.
Viewpoint 4: Horizontal integration of systems’ layers
UI layer horizontal integration
Horizontal integrations in the UI layer are increasing, driven by the trend of ‘assembling’ applications by combining ready-to-use services such as product searches, e-commerce portals, payment gateways, etc. Let’s consider this in the three typical types of UIs.
- Static Single Page Applications (SPAs) — These are simpler apps where the functionality is all contained in one page, fully served by one or more backend systems. E.g., Gmail, Google Maps, Twitter, Facebook, etc. Horizontal UI integration is less common in SPAs.
- Static Multi Page Applications (SMPAs) — Each page in such UIs provides a cohesive set of user functionality, e.g., an eCommerce site. There are different pages for user accounts, orders, support, etc. Each page can be quite independent of the others and connected to a separate backend server component or ‘white label’ partner systems. They are loosely integrated into one web portal or mobile app, but the look and feel is consistent across the pages.
- Dynamic Multi Page Applications (DMPAs) — In such applications, there is a core set of pages connected to the primary backend components, but the user tasks also traverse pages populated by systems in other domains of the same enterprise or by partners’ systems, all of which may not be ‘white label’. In such cases, the look and feel of the external pages can be a little or a lot different and change with time. It may also vary between web and mobile versions of the same task.
Most of the UI layer horizontal integration is effected by the (DDD) application layer horizontal integration as below, including such designs as portals with a collection of portlets from different systems.
Application layer horizontal integration
Eric Evans defines the application layer for DDD as ‘Defines the jobs the software is supposed to do and expressive domain objects to work out problems. The tasks this layer is responsible for are meaningful to the business or necessary for the interaction with the application layers of other systems. This layer is kept thin. It does not contain business rules or knowledge but only coordinates tasks and delegates work to collaborations of domain objects in the next layer down. It does not have a state reflecting the business situation, but it can have a state that reflects the progress of a task for the user or the program.’
By its nature, the application layer can integrate with external systems horizontally, as long as there is little or no need to prepare the data for the presentation layer or persist data as per business rules in the backend. This type of horizontal integration from the application layer (e.g., Javascript running in the browser) is increasing, driven by the trend of ‘assembling’ applications by combining ready-to-use services such as product and service catalogues, eKYC, credit checks, payment APIs, etc. An example standard for this is CORS, Cross-Origin Resource Sharing.
Domain layer horizontal integration
This is still the most common horizontal integration pattern. Domain layer objects can be specifically created that primarily call out to external systems, prepare the response for the application and UI layers, and update the business state in the persistent store in the infrastructure layer. Domain-focused business objects may also need to make external calls as a part of their processing.
(Infrastructure) Persistence layer horizontal integration
Horizontal integration of persistent data repositories in the infrastructure layer is required and unavoidable. Please see this article by me to understand the motivations for data integrations → Enterprise Data Repository Patterns and Progression.
The integration architect should consider the following aspects to optimise the integration design and select the appropriate methods, protocols and technologies.
- Data can move in raw, modified or enriched form between systems
- It can be a subset (usually) or all of the original data
- Modifications or transformations can be done at the source, on the way, or in the receiving system
- It can move in real-time, near real-time, or with a delay (we may also speak of this as online vs offline; or real-time vs batch)
- It can be the unit or aggregated data (the latter by size or time)
- Encryption, compression and security requirements
The picture below illustrates the horizontal viewpoint.
Guidelines
The integration architect should carefully consider making the user experience intuitive and seamless through the solution patterns in viewpoints #2 and #5, along with capacity and performance management.
Viewpoint 5: Stateless integration patterns for web & digital applications
In the web and digital services world, being ‘stateless’ is touted as a silver bullet for components not to persist data or care about the past and future and magically get high performance, scaling and agile development. But no useful business system or task can be stateless. It would be formless or stuck in time and not of much use. (For more, see this excellent article.)
But the idea behind the misnomer of ‘statelessness’ is helpful when we disambiguate it. It enables architectures to rapidly deliver digital business services by assembling web services. And it takes advantage of the increasingly egalitarian distribution of processing power between servers and user devices.
A prominent model for stateless architectures is REST — Representational State Transfer. It works well with the layered architecture model of DDD. (Please see Vernon’s book ‘Implementing DDD’ for more on this.)
Let’s look at the rationale behind statelessness in terms of the three typical architectural decisions (please see →The 3 Most Important Architectural Decisions)
1. Function placement Architecture Decision
Problem: For the digital, web and mobile world, how can we build backend and partner services as swarms of identical server units that grow smoothly (from a handful to scores) with the number of clients (thousands to hundreds of millions)?
Architectural Thinking: If the serving systems maintain the real-time state of the user and task, the front end has to stick to the same backend until the task is completed. This wastes resources throughout the ecosystem and limits the number of clients that can be served, as well as rapid scaling up and down with the number of active clients.
What if we move this real-time user and task state knowledge, utilisation and management (e.g., logged in, checking out, etc.) responsibility to the UI and Application layers (DDD terminology), relieve the domain layer in primary and partner systems from the need to have state, and let them manage only the non-real-time and offline business state (e.g., user role, inventory status, etc.) and persist it in the infrastructure layer.
When we differentiate the user/task state from the business state and their real-time vs non-real-time nature in this manner, all the goodness of statelessness falls into place.
Decision: Make the UI client instance the fixed or ‘sticky’ component in the ecosystem for maintaining the session and state context. Relieve the server components and supporting ecosystem applications/services from the need to persist or be sticky through knowing or having a real-time state.
2. Information repository Architecture Decision
There are three types of information involved.
- Real-time State information — User and task state that changes rapidly as the use case progresses, e.g., logged in, checking out, payment made, logged out, etc. This is temporary and transient information. Keep it in the client application layer (DDD term) close in memory.
- Real-time Context information — Environmental and user information that changes less quickly with the use case, e.g., device type, OS, geolocation, anonymous or identified user, selected airline, etc. Keep this in the client application layer (DDD term) in local semi-permanent stores like cookies, etc.
- Non-real-time & Long-term Business State information — Business information that needs to be kept for minutes to years, e.g., users, accounts, profiles, orders, inventory, payments, service history, media, etc. Keep it in server-side caches, RDBMS, files, Hadoop and other appropriate storage repositories.
So we see that the need remains for information and its storage in stateless focused architectures. Clients and caching systems write temporary data in memory or on local disk storage.
And most backend and partner services store long-term data in central databases. These central databases can still be a point of availability and performance issues. Yet, the tradeoff is acceptable for its cost-effectiveness as long-term data is voluminous and cannot be replicated easily like the business logic components of the domain layer.
3. Integration method Architecture Decision
Problem: (1) How should stateful clients interact with stateless backends and partner systems? (2) How should stateless backend systems interact with other stateless systems?
Stateful interaction
These interactions happen between the client application layer and backend stateless components. State-indicating information required by the stateless domain layer backend or partner components is included in the stateful API call by the client. For example, in the call below asking for an authorisation token, a valid authentication token is necessary and included in the form of the client_id value.
https://MyDomainName.my.salesforce.com/services/oauth2/authorize?
response_type=token&
client_id=3MVG9lKcPoNINVBKV6EgVJiF.snSDwh6_2wSS7BrOhHGEJkC_&
redirect_uri=http%3A%2F%2F2www.example.org%2Fqa%2Fsecurity%2Foauth%2Fuseragent_flow_callback.jsp&
scope=api%20id%20web
Stateless interaction
In the DDD domain layer, components interact statelessly with components in other domains and external systems. Based on requests from the client application layer, information is acquired by client components from server components in the form of HyperMedia resources that comprise information and hyperlinks for further action choices. The information and choices are then returned to the client (DDD application and UI layers) for presentation and next steps.
Here is an example of a request and a stateless RESTful response with information and valid next-state action URI options. It shows the essence of the ‘representation’ and ‘state-transfer’ information that becomes information and action choices in the UI through the application layer. This is the REST architecture pattern’s Hypermedia as the Engine of Application State (HATEOAS) design.
GET /accounts/12345 HTTP/1.1
Host: bank.example.com
HTTP/1.1 200 OK
{
"account": {
"account_number": 12345,
"balance": {
"currency": "usd",
"value": 100.00
},
"links": {
"deposits": "/accounts/12345/deposits",
"withdrawals": "/accounts/12345/withdrawals",
"transfers": "/accounts/12345/transfers",
"close-requests": "/accounts/12345/close-requests"
}
}
}
See this picture for an illustration of this typical overall web integration architecture.
Event Driven Architecture’s integration aspect is essentially a case of the observer or publish-subscribe application integration patterns that support the asynchronous use of the REST model, which is essential in most practical solutions.
Guidelines
The most appropriate digital/web application and information architecture should be decided for the stateful and stateless components and data repositories. Then the integration architecture decisions will follow naturally. RESTful interactions would be the norm, using JSON, XML, and HTML payloads.
DDD REST System Architecture — an overview
The combination of DDD and REST architectures results in the interesting practical application and integration architecture model illustrated below. Study it carefully and consider extant digital web, mobile architecture, and software development frameworks.
Viewpoint 6: Integration Security Patterns
Broadly there are three areas an enterprise needs to secure:
- Business information in stores
- User information in stores
- Either type of information in flight
The third is of particular concern for the integration architect. When people or systems communicate, we need to assure them of five things:
- They cannot be overheard or spied on (privacy, provided by private networks and encryption)
- They are talking to a person or system they know (authentication, provided by 1/2/3 factors)
- The person or system has a right to the information or action requested (authorisation, provided by roles)
- If something goes wrong, it can be investigated (non-repudiation and other details provided by accounting)
- Attacks are prevented or withstood while communicating (protection against various types of threats)
Integration mechanisms must cater to the security needs above, which we can abbreviate as PAAAP.
These needs appear in human-to-system and system-to-system interaction channels as below.
- Human to machine — keyboard, mouse, stylus, touch, speech, and biometrics.
- Machine to machine — The seven OSI network layers, from top to bottom: Application, Presentation, Session, Transport, Network, Data Link, and Physical.
Being an architecture essay, we will not discuss the mechanisms and technologies of PAAAP. The interested reader will easily find the information elsewhere. The table below summarises the typical security solution patterns to deliver PAAAP at each layer.
(Please note that the OSI layers and security solutions are not architectural or immutable. Use the table only for aspects and principles thinking.)
Guidelines
Consider the organisation’s security standards, industry security frameworks such as SABSA, security reference architectures (e.g., AWS, Cisco), and the particular need of every interface in the solution scope to decide the security architecture, design and technology.
End Note
These integration architecture patterns and viewpoints will stand the test of time. I like to study and describe stable ideas for their practical value. See the fundamental ways things interact; everything flows from there.
Go forth and architect, architect.
Connect with me: 
