Integration Architecture

Edit me

#Overview 

##What is Integration Architecture 

Integration architecture is evolving to meet increasing stakeholder demands and accompanying advances in information technology. There is a growing need to new types of integrations. Newer styles of integration are displacing legacy approaches. The integration architecture is often a messaging architecture1 and generally adheres to defined principles that are described below.  To summarize: 

• Definition: Integration architecture is the software architecture that describes how to connect Information Technology components to create systems. 

• What matters: Business value or constituent value is created by connecting components together, yielding a new emergent system. The architecturally significant requirements and the quality attributes drive the design of the architecture. Often these integrations focus on the end-points and looks at the component Application Programming Interface API.    

• What Does not matter:  Component implementations are less important than the way they are connected.  

• Measures: With any business system, value measurement is important.  As an architected system, we want to measure the emergent properties of the overall system, the integration architecture, and the measures of the component interactions as defined by quality attributes.  

• Key Benefits:  the key benefits of an integration architecture are often explicitly documented to demonstrate that the emergent system creates value. Usually, the integration can provide emergent benefits and value through the emergent integration architecture style.   

Integration architecture facilitates component communication. This communication works across the layers of software abstraction: data/information, application, and business. Although there are many styles of integration, integration messaging is the basis for many types of modern integration.   Integration messaging itself is usually viewed at application layer and can be described according to:  

• Components: The components of an integration architecture are the building block applications. These components often require greater attention depending on the context where they are used.  For example, endpoint application management with APIs, may require specialized tools such as Application Programming Interface (API) tools made specifically to support integrations.  These integration components are based on the key elements of the integration (see messaging).  

• Integration Architecture Principles:  These principles guide the creation of robust integration architectures and drive the creation of patterns, reference architectures and final integration architectures.   

• Integration Architecture Landscape: The set of integration architectures is the landscape.  This landscape includes reference architectures for digital, content delivery, internet of things and a variety of other services.  Each integration architecture consists of the integration of messaging principles and patterns.  The reference architectures that are part of this landscape are continually evolving.  

• Integration Messaging: Integration messages consist of the components, endpoints, the channel, and the message. The messaging patterns are a superset of the Integration messaging patterns2.   Message implementation may also require tools made specifically for handling messaging distribution between endpoints across a message channel. Even though messaging is an architecture that is mainly driven by the application layer (and supported through networking), it also needs to be able to share data/information to create some changes at the business layer. 

• Integration Messaging Pattern Taxonomy: Integration messaging patterns may be classified.  This article uses a taxonomy based on the business problem’s social complexity3, the integration’s intention, and the messaging style (interaction mechanics of the underlying messaging approach).  Each pattern represents an approach to deal with specific messaging situations.  The messaging style captures the implementation aspects of the specific messaging pattern.    The intention is closely related to the business needs but more fully captures how the messaging approach meets the business demands. 

• Integration Evolution: The demand to solve new problems is increasing along with the ability of technology to support those new demands. However, there are still significant issues with the widespread adoption of existing integration architectures, and the landscape will continue to evolve for the foreseeable future. 

• Stakeholders: Are those individuals that are concerned with the overall business system and the integrations and components that build that system.  

Architecting Integration 

Architecting an integration brings together key stakeholders.  The architecturally significant requirements and overall quality attributes define the eventual architecture. Some of the critical stakeholders for the whole business system and underlying subsystem includes those for the integration architecture.   

• Who Owns It: Each overall business system, integration platform, and its components have owners – the person who is responsible for it being able to deliver its services. 

• Who is the Customer:  Customers are consumers of the products and services.  Components may have consumers in the digital ecosystem, and the overall business system may also have customers.  “Customers see enterprise as a whole, want to execute business functions that span multiple applications” – Gregor Hohpe One of the unexpected consequences of integration architectures such as digital, is the reuse of components in novel ways. 

• What is Success: Successful integration architectures are measured by the business value created.  An architecture services customer, and the owners by meeting the Architecturally Significant Requirements, and all other essential Quality attributes.   

• Inputs: Architecturally Significant requirements, the business requirements and identified quality attributes are the primary inputs.  Reference architectures, patterns and principles are also required.  

• Outputs: An integration architecture that supports the implementation of all critical components that are existing or net new.  A set of measures (KPI) and (objectives and key results) OKRs that satisfy business value demand.  A set of both leading and lagging indicators is essential so that we can identify early failures and long-term success. An integration architecture provides traceability across business value, business objectives, the integration, and the integrated components. 

Integration of Architecture Principles 

The following principles provide integration architecture guidance. 

• Abstraction Logic is contained or encapsulated within interface logic components defined by the system boundary. These components include those that provide services and others that support the integration. The component functionality is exposed at the end points usually through some sort of an API. 

• Loose Coupling Minimize dependencies across interfaces. The APIs need to be independent, and we want to ensure that component calls are decoupled from the integration of those components. 

• Reusability Architect for reuse where possible.   First, use a defined integration architecture reference model where possible. Second, group components logic into functional interface groupings - with the intent of maximizing reuse. 

• Interoperability Use information and technology standards that allow broader subscriber inclusion to help ensure future interoperability. 

• Composable architectures break down complex problems into smaller problems.  E.g. APIs, database views, web services are combined to create composable interfaces.   

• Design for Integration Provide overall and sub-views of the business system and its components. 

• Legacy integration Don’t ignore legacy integration and its accompanying issues. 

Understanding Integration Architectures 

Classifying Integration Architectures 

Social complexity, the properties of the reference architecture, and the integration style are the three dimensions used to classify integration architectures.  Social complexity is a concept that helps to separate the business motivation of integration from the way its realization.  The Architecture Landscape is just the set of architectures including the integration reference architecture. These architectures will take different approaches or styles to implement the messaging.  We can also view these messaging patterns that are used to build different types of architectures and arrange them according to a taxonomy.  

•  Social Complexity:  Social complexity is the overall context of the integration architecture.   It is a social perspective that describes the integration without making assumptions about how its implementation.  It contains the whole description and the cardinality /directionality of the integration.  For example,” point to point” describes a simple two-person conversation.  It also represents an integration between two components.  The auction house, publishing house, brokerage, and information portal are business descriptions that have overloaded meanings4 but also refer to commonly used message patterns.     

• Reference Architecture / Landscape:  the integration landscape is the set of integration reference architectures classified according to the overall type of solution.  These architectures are solutions for problems with a wide variation of complexity.  For example, file transfer is a straightforward way to share messages, whereas digital is much more complicated.  In modern environments, digital, content delivery and internet of things are three examples of related messaging architectures that have different solution nuances that we will describe below. 

• Integration Style (messaging) the interaction mechanisms need to address how information is moved between components at each layer of software abstractions: data, information, application, and business. This dynamic will identify different types of solutions for synchronous versus asynchronous messaging.  For example, RESTful interfaces commonly used crucial for internal synchronous communications (i.e. within a program). Of course, RPC also implements synchronous messaging between application components. Here we deal with many possible integration solutions that target different problems.  

Figure 1 Social Complexity 

 

Figure 1 describes social complexity (the cardinality, the directionality and description).  The example provides a reference to a common type of architectural realization.  

Messaging 

Generally, many integration architectures use messaging systems to link components.  This discussion focuses on messaging even though there are other styles of integration (such as file transfer, data-based transfer, point to point, remote procedure calls and many others). The messaging form describes the interaction mechanics of the message and is typically able to provide many of the benefits of asynchronous communication. Different messaging patterns are useful for handling various messaging problems. 5 S     point to point, file transfer, and other approaches that support build on a services / micro-services approach each capture the unique intent of the messaging architecture.  

Messaging makes applications loosely couples through asynchronous interaction mechanics.   

• Message:  A message is a finite packet of data transmitted across a channel. 

• Channel The virtual pipe that connects message senders and receivers is called the channel. 

• A component represents the applications that need to communicate.  

• Endpoints. Endpoints enable the underlying component applications to send and receive messages and act as the application interface points. APIs are a vital part of the endpoint description. 

Figure 2 Components of a Messaging System 

 

Reference Architectures 

As described, there are a variety of integration architectures (that are each part of a landscape of architectures) that each describe how to solve a problem. Each of these architectures is also built on established patterns for messaging, data sharing and other items. This section provides a tour of some of the current and popular Integration messaging reference architectures. The entire set of Reference architectures is the landscape. Let us focus on some of the more common microservice-based reference architectures.  

Microservices 

Microservice based approaches is not a standalone architecture as much as it is a typically high-level pattern or architectural approach that is fundamental to some of the reference architectures described below.   

The microservices architecture extends the service-oriented architecture but at a finer granularity.  In many ways, the invention of microservices enables digital and other emergent integration styles but also makes organizational structure redeployment much easier – around digital APIs, for instance. 

Each of the example messaging architectures below can be a variation of a microservices architecture.  The figure below describes some of the critical elements in a microservices-based approach. Requests for microservices come through well-defined API endpoints.  Microservices, in turn, access information/data or possibly other microservices. 

Figure 3 Key Elements of the Modern Digital Reference Architecture 

 

 

 

 

 

 

 

 

  

Let’s now examine several types of API and microservice-based architectures, beginning with the most general model: “digital.”  These are types of architectures, in the landscape. each modified to service specific needs. 

Digital Messaging Reference 

Modern digital architectures deploy products built from APIs (components). In many cases, a component could also bridge a legacy implementation.  

APIs make digital society and digital business work. The focus in a digital architecture is how to create, manage and change the APIs (components).  There is a twofold focus: the product and the component.  By using an API based approach, product development is better able to deliver digital business models.  

The distinction between internal, external, and remote offerings is blurred.  This field is evolving to a more general type of distributed digital computing. In a digital product architecture, the focus from product consumption and subsequent API consumption is normally transactional and based on a wide variety of business models.  

Content Delivery Reference 

Integration for content distribution is different than for digital environments.  Responsiveness and transactional support are not as important as availability. This architecture provides streaming content to clients. The demand for high throughput has many providers focusing on microservice-based event architecture.  Service that accelerates content delivery has many characteristics in common with event-based architectures but also has significant differences. 

With content delivery streaming throughput, in a highly available network, becomes the focus with less emphasis on a transactional component. Some of the content delivery characteristics include: 

• High-performance streaming throughput 

• Performance and availability in a global environment that may employ many proxy servers for instance, 

• Global user base (a worldwide server distribution network is required), 

• Unique security needs such as protection from Denial of Service attacks 

• Minimize network latency  

• Maximize bandwidth. 

 Overall, a content delivery network reduces the effective network distance from content to consumer. Reduce customer traffic to the central server.  This example illustrates how the underlying network implications become significant and alter the primary event messaging architecture. 

Internet of Things Reference 

The integration architectures that are needed to support a connected network that is used collaboratively with a high level of sensory feedback.   Connecting devices need to ensure real-time event architecture that collects data from either local or remote devices.  Often this type of architecture will deploy data collection and dissemination points called gateways to interface IoT devices. For performance reasons, this type of network usually leverages an event-based microservices architecture. While it shares some similarities with content delivery, the focus is to ensure control data and sensory inputs are not lost.  Some of the characteristics of IoT include: 

• High-performance streaming input (and possibly control output), 

• Integration of a wide variety of devices, often using points called gateways to condense data for internet transmission. 

• It could be the internet, or hybrid (on-premises and internet). 

• Varying locales depending on unique characteristics 

• Higher levels of customization to meet specific problem requirements. 

Open Issues 

Digital architecture and many other types in the architecture landscape are evolving. However, legacy systems still need to connect with older systems – technology debt describes the choices we made knowingly not putting something in the current environment, thinking we could come back and revisit.  Integration must connect legacy systems and systems of the record too. As architects, we need to deal with several issues.  How do you include legacy systems?  How do we ensure syntactic and semantic interoperability in the future adoption of our schemes?   

Platforms may be one way to keep ahead of this curve, and they pretty much work until the technology changes or the integration needs to be used in new or unexpected ways. 

One of the most significant open concerns is around syntactic consistency and building semantically interoperable systems. Even if the data and functions seem to match, we need to be on the lookout for impedance mismatches.   As we move forward, one of the significant open issues that remain is how to manage semantic incompatibility. 

References 

Enterprise Integration Patterns by Gregor Hohpe and Bobby Wolf, 2004. 

Information Technology Architecture Body of Knowledge and core principles presented on the https://iasaglobal.org/ website, by IASA Global  

Cloud Design Patterns by Alex Homer et al. describes some of the practical cloud design patterns (that cloud and hybrid base integration build on), 2014. 

Continuous API Management by Mehdi Medjaoui et al. drives down into some of the critical agile architecture issues and API architecture and management, 2018. 

Integrating Serverless Architecture by Rami Vemula provides an excellent overview of serverless integration using Azure functions, 2019.