Streaming Data Security and Governance for Enterprise Operational Intelligence

header image

Streaming Data Security and Governance for Enterprise Operational Intelligence

Businesses are increasingly operating in a real-time environment. Most data is produced continuously from various sources such as Mobile Apps, Web Clickstreams, Application Logs, IoT Sensors, and so on. The value of streaming data used in real-time decision-making diminishes or perishes, very quickly. With briefest of passage of time such data quickly becomes historical data that is generally suitable to be processed by traditional batch jobs for business intelligence type applications. For governance, risk and security function to be successful in “real-time” enterprises, we increasingly find that applying streaming big data and machine learning technology-based solutions are required - to effectively manage high-speed data at scale.

In the security domain, as the severity of attacks continue to increase the damages and costs are spiraling out of control. The damage to brand, and the years lost in rebuilding the image and reputation, and recovering from the financial losses present enormous challenges to many businesses. Additionally, given the evolving nature of security threats and the massive inventory of credentials up for sale on the black market, most attacks involve the illegitimate use of a legitimate account. The eroding organizational boundaries and the lack of governance mechanisms for streaming data are leading to a situation where there is increasing distrust (the fourth V – veracity) in using such data in decision-making (leading to further losses).

Architectures and development processes are also evolving in enterprises to include a mix of in-premise and cloud infrastructure, internal and external data sources, big data storage and application frameworks, and legacy and new-age applications such as streaming analytics, machine learning applications, neural networks, big data visualization tools, and agile process-driven environments. In this scenario, implementing security and governance measures becomes a much more challenging and complex task.

Though, integration of varied data sources, systems, big data technologies, advanced analytics, and ML / DL driven applications have been the mantra in the business applications space for a while now, there’s been little progress towards integrating disparate point solutions used in security operations or using integrated big data security analytics security and risk management. There are some big data based security and governance solutions however very few that can be used for meeting real-time security, risk and governance of high-speed streaming data.

Some of the emerging threats and complexity of security / governance problems include:

  • Security threats to machine learning and deep learning models being deployed by businesses

  • Understanding newer technologies and methods being used by businesses. Furthermore, understanding the latest techniques being used by the adversaries is perhaps in even more important.

  • Protecting privacy in streaming structured, semi-structured and unstructured data across data sources, applications and users is complex.

  • Continuous monitoring of an organization’s employees’ digital activities.

  • Predicting security problems with constantly changing attack vectors

  • Timely interventions in online learning systems under attack.

  • Minimizing false positive rates and automating prioritization of incidents to be investigated further.

The security and governance function needs to evolve in response to these emerging threats and problems. Protecting against Advanced Persistent Threats (APTs) where adversaries are using attacks that exploit zero-day vulnerabilities. Such adversaries can easily set up labs to carefully design attacks with a very specific goal to collect and steal information (Confidentiality), or make the victim’s systems unavailable (Availability), and / or modify the behavior of applications (so that they malfunction or produce incorrect results), or change system data (Integrity).

Security teams need to be equipped with the knowledge, tools, analytics, visualization, and infrastructure to effectively counter the increasing risks to the business while improving the truthfulness of the data and systems being used by the business. Additionally, the processes and skills of security teams need to be augmented as well. For example, the security teams now need to include big data and ML / AI specialists to not only protect the organization but also advise the development teams on security practices required for developing more robust and secure systems.

There are profound shifts going on with security today – what we’ve done historically is not working. We need to integrate the security infrastructure and systems, use big data analytics and machine intelligence driven to assist us in our security work. This evolution is obvious but it will be difficult and CISOs will need a lot of help. Educating your customers and prospects, and having the resources, knowledge, services, and infrastructure will be key to success. We need think about security analytics, visualization, automation, centralized C&C, and machine intelligence to deliver on Governance, Risk and Compliance (GRC) requirements, security management, fraud detection, forensics support, and other initiatives at scale. That is the future and we need to start preparing for it today.

Challenges in big data security analytics, incident detection & mitigation

Below, we list some of key challenges and problems in security management:

  • Not enough skilled security people available

  • Too many false positives (tools not doing a good job of filtering out the noise)

  • Depends upon too many manual processes (running around putting out fires is not scalable)

  • Depends upon too many independent tools that aren’t integrated together

  • Sophisticated security events have become too hard to detect

  • Lack of adequate data collection & monitoring

  • In some cases, once the attack goal is met, the malicious code immediately disappears without a trace.

  • The collected data is stored in giant repositories. Connecting the dots for an attack is challenging.

  • Slow speed of response

  • Variability of threats

  • Adaptive adversaries

  • Security is borderless and extends well beyond the traditional perimeter

  • Nature of streaming data: Noisy, complex schemas, cyclical volumes

  • New Architecture Paradigms – large-scale distributed processing and storage framworks

  • Large and diverse user organizational models

  • More data means, even more policies and rules to be created and maintained

  • Evolving development models with several production releases per day

Using Veristorm Real-Time Platform for Streaming Data Security and Governance

Many organizations are looking for solutions that allow them to monitor, and filter, correlate, connect the dots, analyze, report, detect, and predict security incidents and governance issues to raise appropriate alarms and alert people responsible for them. What is needed is a flexible real-time platform that can integrate existing security solutions and enterprise data, ability to deploy domain-specific ML / DL models and algorithms, automate security processes and responses, visualize security and governance related data points at scale, continuously monitor across multiple data sources, users, network traffic, data flows, and deviations.

Veristorm Platform can gather, store, visualize, learn, analyze, report, and act on data in real time. It can continuously track data movement and usage, securely log such information, and provide analytical tools to eliminate the noise and help you zoom-in on points of interest to investigate, at the finest level. Additionally, it provides you with the capabilities to integrate intelligent algorithms to predict issues before they snowball into real crises.

One of primary goals of the Veristorm platform is the focus on intelligence and automation in support of the data governance and security related tasks in the organization. Towards that goal there are data driven decisions made starting from ingestion itself. Execution of automated data validation, metadata generation, etc. can eliminate scattered manual processes throughout the enterprise.

  1. Real-time: Analyze data in real time. Provide analytics on real time streaming data from a wide variety of internal and external data sources to identify and predict specific problems and their impact. So, instead of storing and analyzing over a period of time, our platform starts analyzing data as it is streaming into the organization.

  2. Centralized: Monitoring and visualizing security in multiple places using multiple tools and analytics is too hard. Integrate best-of-breed, point solutions into our centralized pipeline to feed real-time security dashboards. Implement centralized real-time data monitoring and visualization of streaming data in one place. Provides a non-intrusive way to gather and analyze streaming data for real-time security and governance purposes without impacting your core business applications.

  3. Automation and productivity: Collects and analyzes data in an effort to identify targets for automation of security tasks.

  4. Big data: data collection and data analysis at scale to better understand the security status across the enterprise.

  5. Extensibility: A platform that let’s you extend its functionality by plugging in your specific custom applications or algorithms.