In today's world, "information" is the key. With IT enabled world, we collect large amount of data from various sources but how effectively we can use such large amount of data is always a challenge. Different domains such as Financial, Sales, Security, Retail, Energy, etc. initiate and collect large amount of data from various source for their day-to-day IT operations and management. For e.g. in our day-to-day operation of using e-services, we perform various activities and transactions and one such example is e-banking. Do we ever realize when we do an online transaction how much data is collected to ensure safety of the transaction? Now, when each and every online activity is monitored, extracting meaning out of billions of activities is a big challenge in IT enabled domains. When billions of transactions are happening per day, how do we find out which one of those is a suspicious activity. With the world becoming more and more IT enabled, IT security becomes extremely important in every human's life. Not collection, rather processing huge amount of data and transactions near real-time and finding out which one we need to know is one of the challenges that IT enabled businesses are facing today and this becomes more critical if it is related to "security". Here, I am going to talk about how a key technology in BI space called "Complex Event Processing" can be leveraged to address some of the challenges in IT security space. So, the success is to process data the data in motion, as it flows to generate meaning out of it...
Challenges in Security Domain
Background
Impact
Challenges in Security Domain
- Growing Information Technology and Information Security spaces have
- Critical applications and systems
- Simultaneous and complex operations
- Sensors everywhere and real-time collection of security intelligence
- Massive data volume
- Need to ensure reliability and quicker turnaround
- New paradigm is the move to data-driven decisions
- High security event data rates, continuous queries, and millisecond latency requirements that make it impractical to persist the data in a relational database for processing.
Major challenges are -
How to manage all this data and monitor critical applications to rapidly diagnose problems and maximize uptime?
How to analyze data in motion to gain insight and lower risk?
How to ensure that the right data is available in the right place at the right time enabling the best decisions?
How to manage large and simultaneous streams of data and analyze it in real time to reduce decision time and increase operational efficiency.
- Millions/Billions of Security Events are generated every day
- There is a need to process to data in near real time in Security Intelligence space to make meaning of huge amount of data
- Faster processing to improve business decisions
- Data processing in-motion than post storage to deliver real-time analytics
- Each BI vendor provides technology leveraging Complext Event Processing concept and one of them is Microsoft StreamInsight
- Microsoft StreamInsight, a powerful, cost-effective Complex Event Processing platform
- StreamInsight™ provides high-throughput stream processing architecture to process events in thousands per second
- StreamInsight™ speeds the transformation of data to actionable business information.
- It accomplishes this by reducing the latency associated with the traditional BI approach of acquisition, filtering/compression, indexing, storing and then analyzing.
- StreamInsight™ based solution for huge data-in-motion processing in Security Intelligence space
- Unified solution for data-in-motion as well as post-storage to deliver real-time-analytics
- Processing huge data-in-motion regardless of data source, relationship and destination
- Open and independent development platform based upon an event-driven architecture where data from multiple, heterogeneous sources can be intelligently analyzed in real time.
Impact
- Continuous and incremental processing of never-ending sequences of events
- Lightweight streaming architecture that supports highly parallel execution of continuous queries over high-speed data.
- In-memory caches and incremental result computation provide excellent performance with high throughput and low latency.
- Low latency because of events are processed without costly data storage in the critical processing path.
- All processing is automatically triggered by incoming events
- Historical data can be accessed and included in the low-latency analysis
- Extensive set of Input and Output Adapters
- Unified Data Processing architecture for both data-in-motion and data-in-storage
- Data-in-motion Processing Model for
- Real-time operational dashboard and analytics
- Predictive Analysis in predictive future security posture looking at the current events
- Generating and delivering Real-time Alerts and Notifications
- Push based data delivery using StreamInsight Output Adapters
- Data-in-storage Processing Model
- Based on OLAP model and has different cubes for current and historical data analysis
- Analytics Dashboard that doesn’t need real time data
- For what-if-analysis and data mining on historical data
- For Static Reports
Benefits
- In Information Security space, there is a need to process and analyze huge security events for quicker business decisions. The proposed solution provided a unified architecture to process both data-in-motion and data-in-storage.
- The proposed solution will help to
- Process large volumes of Security Events across multiple data streams while data-in-motion
- Gain insights from critical information in near real time by monitoring, analyzing, and acting on data in motion
- Reduces storage cost
- Low deployment and development costs by utilizing existing technology and skill sets.
- Gain continuous insight through historical data mining.
- Take quicker business decisions
- Improved Time-to-Market
- Embedded Options and Regional Hubs
- Embedded-options to pre-process on the edge (for example, sensors and other devices).
- Regional hubs that provide local processing of event streams from embedded engines for aggregation and correlation.
- Run complex analytics and mine insights with centralized processing.
Comments
Post a Comment