Real-time Data Governance, Risks and Security Challenges – Part 2: Impact of major shifts in application architecture and platformsSubmitted by admin on Mon, 2017-12-25 00:01
Digital business about business agility (or event thinking), i.e., always sensing, always ready, always changing and always learning. There is push for investing in technology and design skills for event-orientation in business and technology initiatives.
There can be no large-scale success without event thinking.
By 2020, achieving broad competence in event-driven IT will be a top-three priority for the majority of global enterprise CIOs.7
IT leaders are championing event thinking because it paves the way to a digital business and powers digital business agility. Events are being used to empower decisions in real-time. Event-driven architectures typically include: timely reaction to unexpected, unplanned business events (Business Moments), decision makers’ instant business-state understanding (Situational Awareness), real-time, self-service, unconstrained analytics (Real-Time Insights), and absorbing large volumes of data “on the move” (Stream Analytics).
Enterprises are finding event thinking challenging because of unfamiliar technology, complex architecture, limited standards adoption and tools market, requiring massive applications redesign and / or limited industry experience / skills.
By 2020, event-sourced, real-time situational awareness will be a required characteristic for 80% of digital business solutions.
There is a shift from Request-Response architectures to Event-Driven Architectures (EDA). This involves a significant change in perspective as Request-Response architectures are typically client-driven, inherently synchronous, one-to-one, and rely on service availability and responsiveness whereas Event-Driven architectures are client- or server-driven, inherently asynchronous, many-to-many, and allow for events buffering.
Similarly, the shift from data-driven architectures where the source of truth is the data store and the first priority of the system is to preserve the data to Event-Driven architectures where the source of truth is the log of events and the first priority of the system is to react or respond to events. There is an associated shift in IT perspective as well from being largely transaction-centric (primarily dealing with Data-at-Rest) to being event-centric (primarily dealing with Data-in-Flight).
These new architectures do not comprise of just loosely-coupled components but also minimally-coupled components with a separation between declaration of interest from event occurrence, event capture from event processing, and event publisher(s) from event subscribers(s). Additionally, eventual consistency has increasingly become the norm in EDA.
Event-Driven architectures typically employ several communication patterns such as fire-and-forget, publish-subscribe, and point-to-point. And Event-Driven APIs have two distinct interfaces - the Channel Interface and the Functional Interface.
Channel interfaces are characterized by subscribe / unsubscribe, publish events / retrieve events and request/response interactions. Channel interfaces are often synchronous RESTful interfaces.. The functional interfaces define event types and payloads, service-side and consumer-side events, and topic structures or equivalent. They typically use Pub/Sub conversations.
Event-Driven APIs support external publishers or subscribers, and extend REST interfaces to avoid polling. Common use cases for such APIs include application integration, reactive mobile, web and device clients, IoT device data ingestion and notification. However, note that your architecture will not be entirely event-driven as useful APIs are not purely event-driven.
It is complicated, expensive, and time consuming to change, scale, and introduce new features in a monolithic app. Such applications just do not fit very well in the real-time, continuous business environments. We want to significantly reduce application delivery timelines from weeks to minutes. Shifting to a microservices architecture can help mitigate these challenges. Especially, instances where microservices leverage cloud-native services, the delivery times can be accelerated, significantly.
Microservices are typically characterized by fine granularity, independence, and immutability. They have a single well-defined responsibility and can be developed, tested, deployed and replaced in isolation. Additionally, they can be independently scaled, have explicitly loosely-coupled dependencies (event-driven), and operate in shared nothing environments. They are also typically disposable and / or replaceable.
The main benefits of a microservices architecture includes continuous delivery, deployment flexibility, hyperscalability, resiliency, team autonomy and polyglot support. It enables a significant shift from managing complex dependencies, scheduled release cycles and coarse-grained application-level scalability to a set of minimally dependent or isolated features, feature level releases, and fine-grained component level scalability. A microservices-based architecture is best served by agile development and automated DevOps processes. Highly-automated continuous integration / delivery (CI / CD) pipelines that automate builds, tests and deployments help reduce time and effort significantly.
Additionally, there is a burgeoning microservices infrastructure market that support various frameworks, automation, telemetry and other infrastructural components such as NoSQL databases, caching services, and so on.
However, microservices is not applicable in all circumstances and careful analysis of readiness and costs / benefits are required before adopting microservices.
There is an emerging demand for low / no code solutions for achieving higher productivity. For example, increasing adoption platforms such as Mobile Application Development Platforms (MADPs) to help drive innovation, digital transformation, reduce time to market (for mobile apps), improve business process agility and improve business process outcomes. Additionally, MADPs are increasingly, being used to develop and deploy mobile apps at high volume.
Function PaaS (fPaaS) provides an increasingly popular serverless platform where you can upload your code (no need to pre-allocate resources) and configure a trigger (API, DB write, event stream, notification, etc.). fPaaS runs your code when triggered. Additionally, it scales automatically and you pay just for the compute time you use.
Lambda and Kappa Architectures
Stream processing represents a great stride toward achieving real-time insights and driving real-time operations. Two typical objectives for implementing stream processing are: reducing latency of the existing systems and building new, fast systems from scratch. Both can be achieved by incorporating data streaming as part of Lambda or Kappa architecture.
The idea of Lambda architecture is in combining a batch layer and a speed layer for the up-to-the-moment insights. The batch layer (such as Hadoop/MapReduce) periodically computes views on all the data from the beginning of time — a slow process. When the batch layer already exists, the new speed layer adds on the data that is fresher than the data available in the batch layer. The incoming data then is dispatched to both layers.
The speed layer provides the view on a relatively small set of data, to compensate the difference between the last pre-computed batch view and now. The speed layer is fast; it is thinner compared with the batch. Batch views are periodically refreshed, for example, doing incremental model updates. Then, computations of the real-time views restart from the last batch view recalculation. Streaming the data in the speed layer starts afresh after re-computation of the batch views.
The presentation layer combines batch and real-time views to make the unified data available for immediate consumption by users, applications and analytics tools.
Compared to Lambda, Kappa architecture eliminates the two separate layers and takes advantage of stream processing for batch (and not only for real time). Kappa architectures are being used for streaming data applications such as real-time machine learning, analyzing sensor data in IoT for real- time alerts, log analysis, and so on.
In this blog, we explored some of the emerging application architectures and platforms that need to be understood now for developing strategies for addressing the associated governance, risks and security requirements, comprehensively.
In subsequent blogs we will explore the impact of shifts in operating environments, and emerging integration solutions.
Anne Thomas, Microservices: The future of SOA, Gartner Application Strategies & Solutions Summit, 2017
Yefim V. Natis, How event thinking paves the way to digital business, Gartner Application Strategies & Solutions Summit, 2017
Svetlana Sicular, Harness streaming data for Real-Time Analytics, Gartner, 2016
Jason Wong, Magic Quadrant: Mobile App Development Platforms, Gartner Application Strategies & Solutions Summit, 2017
Anne Thomas, Tutorial: Event-Driven architecture, Gartner Application Strategies & Solutions Summit, 2017
Olive Huang, The state of customer experience technologies and their impact to your application strategy, Gartner Application Strategies & Solutions Summit, 2017
Nadine LeBlanc, To the point: The future of CRM is event-driven, Gartner Application Strategies & Solutions Summit, 2017