Category Archives: Outlier Detection

Out of Distribution Data Detection in Deployed Machine Learning Models

Posted on February 25, 2022 by Pranab

If a deployed machine learning model encounters an out of distribution data, it should either reject it or delegate it to a human reviewer for further investigation and decision making. A sample is out of distribution (OOD) when it is … Continue reading →

Posted in AI, Machine Learning, mlops, Outlier Detection, Python, PyTorch | Tagged , , | Leave a comment

Customer Service Quality Monitoring with AutoEncoder based Anomalous Case Detection

Posted on January 20, 2021 by Pranab

Most companies put lot of effort ensuring superb customer service. They want to resolve customer issues as quickly as possible leaving a positive experience with customers. It’s been said that one negative experience with customer service an obliterate loyalty to … Continue reading →

Posted in Data Science, Machine Learning, Outlier Detection, Python, PyTorch | Tagged , , | Leave a comment

Time Series Sequence Anomaly Detection with Markov Chain on Spark

Posted on July 25, 2019 by Pranab

There are many techniques for time series anomaly detection. In this post, the focus is on sequence based anomaly detection of time series data with Markov Chain. The technique will be elucidated with a use case involving data from a … Continue reading →

Posted in Anomaly Detection, Big Data, Data Science, Machine Learning, Outlier Detection, Scala, Spark | Tagged , , , , | 1 Comment

Learning Alarm Threshold from User Feedback using Decision Tree on Spark

Posted on October 15, 2018 by Pranab

Alarm fatigue is a phenomena where some one is exposed to large number of alarms, become desensitized to them and start ignoring them. It’s been reported that security professionals ignore 32% of alarms because they are thought to be false. … Continue reading →

Posted in Anomaly Detection, Big Data, Data Science, Outlier Detection, Spark | Tagged , , , , | 1 Comment

Data Quality Control With Outlier Detection

Posted on June 8, 2015 by Pranab

For many Big Data projects, it has been reported  that significant part of the time, sometimes up to 70-80% of time,  is spent in data cleaning and preparation. Typically, in most ETL tools,  you define constraints and rules statically for … Continue reading →

Posted in Big Data, Data Science, ETL, Hadoop and Map Reduce, Internet of Things, Outlier Detection, Statistics | Tagged , , , , | 1 Comment

Real Time Detection of Outliers in Sensor Data using Spark Streaming

Posted on February 19, 2015 by Pranab

As far as analytic of sensor generated data is concerned, in Internet of Things (IoT) and in a connected everything world, it’s mostly about real time analytic of time series data. In this post, I will be addressing an use … Continue reading →

Posted in Big Data, Data Science, Internet of Things, Outlier Detection, Real Time Processing, Spark, Time Series Analytic | Tagged , , | 2 Comments