Category Archives: Outlier Detection

Time Series Sequence Anomaly Detection with Markov Chain on Spark

Posted on July 25, 2019 by

There are many techniques for time series anomaly detection. In this post, the focus is on sequence based anomaly detection of time series data with Markov Chain. The technique will be elucidated with a use case involving data from a … Continue reading →

Posted in Anomaly Detection, Big Data, Data Science, Machine Learning, Outlier Detection, Scala, Spark | Tagged , , , , | 1 Comment

Learning Alarm Threshold from User Feedback using Decision Tree on Spark

Posted on October 15, 2018 by

Alarm fatigue is a phenomena where some one is exposed to large number of alarms, become desensitized to them and start ignoring them. It’s been reported that security professionals ignore 32% of alarms because they are thought to be false. … Continue reading →

Posted in Anomaly Detection, Big Data, Data Science, Outlier Detection, Spark | Tagged , , , , | 1 Comment

Data Quality Control With Outlier Detection

Posted on June 8, 2015 by

For many Big Data projects, it has been reported  that significant part of the time, sometimes up to 70-80% of time,  is spent in data cleaning and preparation. Typically, in most ETL tools,  you define constraints and rules statically for … Continue reading →

Posted in Big Data, Data Science, ETL, Hadoop and Map Reduce, Internet of Things, Outlier Detection, Statistics | Tagged , , , , | 1 Comment

Real Time Detection of Outliers in Sensor Data using Spark Streaming

Posted on February 19, 2015 by

As far as analytic of sensor generated data is concerned, in Internet of Things (IoT) and in a connected everything world, it’s mostly about real time analytic of time series data. In this post, I will be addressing an use … Continue reading →

Posted in Big Data, Data Science, Internet of Things, Outlier Detection, Real Time Processing, Spark, Time Series Analytic | Tagged , , | 2 Comments