Route Planning with DynaQ Reinforcement Learning

The purpose of a simple route planning is to find the optimum route from one location to another that minimizes travel distance or time. Generally various heuristic optimization techniques are used to find optimum route. In this post we will use a Reinforcement Learning (RL) algorithm called DynaQ to solve the routing problem. DynaQ belongs to the Temporal Difference (TD) learning family of RL algorithms. The DynaQ implementation is available in my RL Python package qinisa. The code is in my GitHub repo whakapai.

Time Series Feature Engineering with Histogram

Typically time series data requires manual feature engineering unless you are using Deep Learning. Deep Learning alleviates you from this task but there is no guarantee. There are many techniques for feature engineering time series. In this post will use a simple feature engineering technique based on histogram. In this post, we will be using this technique for anomaly detection in machinary vibration data. The implementation is in my GitHub repo. It’s also available as a time series Python package zaman.

AI Chatbots with Human in the Loop

LLM based AI Chatbots on auto pilot is not very wise and risky. Even when manually composing text or getting an answer to a question through AI chatbots, a human tries different prompts and evaluates responses. So there is human in a try and evaluate loop. An AI chatbot in autonomous mode without human intervention is a ticking time bomb. There have been reports of direct customer facing chatbots malfunctioning, producing wrong or poor quality responses. A user can also abuse such chatbots with prompt hacking to elicit inappropriate responses.

In this post, we will go through three examples of human in loop Ai Chatbot applications as it’s been reported in recent papers. Two of them are from the industry and one from a university.

Time Series Classification with Neural Network using Random Sub Sequence Statistics as Features

Deep Learning models generally extract features in the early layers of the network. It does away with the need for manual feature engineering making it attractive ML technique. However, this not always the case, time series clustering and classification being some examples, unless you use complex solution involving RNN and MLP. Time series forecasting is easier. You can feed raw time series data to auto regressive DL models like Transformer or LSTM. In this post we will use a feature engineering technique called interval statistics to engineer features for a time series to be used in a classification model based on MLP with one hidden layer.

The feature extraction implementation is my Python package for time series called zaman. For the MLP I have used my PyTorch no code framework in a Python package called torvik. The code is in my Github repo whakapai.

Time Series Forecasting with Decomposition and Two Linear Networks

Many solutions for time series forecasting are available, starting from classic ARIMA to Deep Learning Transformer based. Results are varied. There doesn’t appear to be any consensus for the best performing forecasting model overall. In this post, we will go through a simple decomposition and linear MLP based solution. It’s been claimed that this embarrassingly simple model works as good a complex Transformer based solutions. The solution is available in my Python package zaman. The code is is in my GitHub repo whakapai.

Unreasonable Claim of Reasoning Ability of LLM

Reasoning is critical for problem solving, decision making and in general for human intelligence. There have been various claims of reasoning abilities of LLM. Typically these claims are based on few anecdotal examples, followed by by some broad brush conclusions. There have several papers debunking such claims, demonstrating how LLM fails for non trivial reasoning tasks. I will review two of those papers showing how the so called reasoning ability is an illusion. I will also show how the success for simple reasoning tasks can be explained by co occurrence pattern learning and In Context Learning in the GPT Transformer of LLM.

Although the focus in this post is on the reasoning ability of LLM, the arguments presented are equally applicable for any claimed problem solving ability of LLM.

Customization of LLM ChatBots with Retrieval Augmented Generation

For many domain specific business application the pre trained LLM is not enough. Customization is necessary. Out of the box pre trained black box models, even with prompt tuning may be inadequate. There are two ways to customize LLM with recent or private data. The solution is either Fine Tuning(FT) or Retrieval Augmented Search (RAG). For various reasons Fine Tuning is often not viable. In this post we will review RAG, including the technique, pros, cons and it’s inner workings.

Time Series Data Exploration with Wavelet Transform

Time series data provides time domain information and Fast Fourier Transform provides frequency domain information only. What if you wanted both, for example you may be interested in frequency domain information only within a specified time range. Wavelet transform provides time and frequency domain information simultaneously. In this post we will explore machinery vibration data using Wavelet transform. The implementation is in my time series Python package called zaman. Here is the GitHub repo.

Store Inventory Replenishment with Q Value Reinforcement Learning

Subscribe to continue reading

Subscribe to get access to the rest of this post and other subscriber-only content.

Already a subscriber?

Online Advertisement Placement with Contextual Bandit.

There is a class of algorithms called Multi Arm Bandit (MAB) applicable for Reinforcement Leaning problems without state. Sometimes there is side information available for each action. MAB algorithms that take this side information aka context into account are called Contextual Bandit. The decision made by a Contextual Bandit at any given time is a function of the context in addition to past rewards. In this post we will use a Contextual Bandit algorithms called Linear Payoff Upper Confidence Bound (LinUCB) for making online ad placement decisions.

The implementation is available in my Reinforcement learning Python package called qinsia. Here is the GitHub repo.