Category Archives: Spark

Detecting Quarantine Violation from Mobile Phone Location Anomaly on Spark

Posted on April 20, 2020 by

With the world under siege with Corona virus, you might find this topic timely. There are two main aspects of any epidemic breakout, epidemic spread and containment. There are various strategies for containing epidemic spread. One of them is to … Continue reading →

Posted in Anomaly Detection, Big Data, Data Science, Scala, Spark | Tagged , , | Leave a comment

Model Drift Detection with Kolmogorov Smirnov Statistic on Spark

Posted on February 24, 2020 by

In retail business, you may be using various business solutions based on product demand data e.g inventory management or how a newly introduced product may be performing with time. The buying behavior model may change with time rendering the those … Continue reading →

Posted in Data Science, Machine Learning, Spark, Statistics | Tagged | Leave a comment

Contextual Data Completeness Metric Computation on Spark

Posted on December 18, 2019 by

Data quality is critical for the healthy operation of any data driven enterprise. There are various kinds of data quality metrics. In this post, the focus will be on the completeness of data. Data quality from a completeness point of … Continue reading →

Posted in Big Data, Data Science, ETL, Spark | Tagged , | Leave a comment

Time Series Trend and Seasonality Component Decomposition with STL on Spark

Posted on September 24, 2019 by

You may be interested in decomposing a time series into level, trend, seasonality and remainder components to gain more insight into your time series. You may also be interested in decomposition to separate out the remainder component for anomaly detection. … Continue reading →

Posted in Anomaly Detection, Big Data, Data Science, ETL, Spark, Time Series Analytic | Tagged , , , | Leave a comment

Encoding High Cardinality Categorical Variables with Feature Hashing on Spark

Posted on August 7, 2019 by

Categorical variables are ubiquitous in data. They pose a serious problem in many Data Science analysis processes. For example, many supervised Machine Learning algorithms work only with numerical data. With high cardinality categorical variables, popular encoding solutions like One Hot … Continue reading →

Posted in Big Data, Data Science, ETL, Scala, Spark | Tagged , , | Leave a comment

Time Series Sequence Anomaly Detection with Markov Chain on Spark

Posted on July 25, 2019 by

There are many techniques for time series anomaly detection. In this post, the focus is on sequence based anomaly detection of time series data with Markov Chain. The technique will be elucidated with a use case involving data from a … Continue reading →

Posted in Anomaly Detection, Big Data, Data Science, Machine Learning, Outlier Detection, Scala, Spark | Tagged , , , , | 1 Comment

Elastic Search or Solr Search Result Quality Evaluation with NCDG Metric on Spark

Posted on April 24, 2019 by

You have built an enterprise search engine with Elastic Search or Solr. You have tweaked all the knobs in the search engine to get the best possible quality for the search results. But how do you know how well your … Continue reading →

Posted in Big Data, Data Science, elastic search, Log Analysis, Scala, Search Analytic, Solr, Spark | Tagged , , , | Leave a comment

Plugin Framework Based Data Transformation on Spark

Posted on March 21, 2019 by

Data transformation is one of the key components in most ETL process. It is well known, that in most data projects, more than 50% of the time in spent in data pre processing. In my earlier blog, a Hadoop based … Continue reading →

Posted in Big Data, Data Science, ETL, Scala, Spark | Tagged , | 2 Comments

Normal Distribution Fitness Test with Chi Square on Spark

Posted on January 30, 2019 by

Many Machine Learning models is based on certain assumptions made about the data. For example, in ZScore based  anomaly detection, it is  assumed that the data has normal distribution. Your Machine Learning model will be as good as how those … Continue reading →

Posted in Anomaly Detection, Big Data, Data Science, Spark, Statistics | Tagged , | Leave a comment

Time Series Seasonal Cycle Detection with Auto Correlation on Spark

Posted on December 23, 2018 by

There are may benefits of auto correlation analysis on time series data, as we will be alluding to in detail later. It allows us to gain important insights on the nature of the time series data. Cycle detection is one … Continue reading →

Posted in Big Data, Correlation, Spark, Statistics, Time Series Analytic | Tagged , , | 3 Comments