fbpx
Anomaly Detection for CRM Data: A Step-By-Step Guide Anomaly Detection for CRM Data: A Step-By-Step Guide
Editor’s note: Geeta Shankar and Tuli Nivas are speakers for ODSC West 2023 this October 30th to November 2nd. Be sure... Anomaly Detection for CRM Data: A Step-By-Step Guide

Editor’s note: Geeta Shankar and Tuli Nivas are speakers for ODSC West 2023 this October 30th to November 2nd. Be sure to check out their talk, “Anomaly Detection for CRM Production Data,” there!

Anomaly detection for CRM (Customer Relationship Management) data is gaining increasing importance. It is crucial to monitor any unusual behavior in production data and proactively identify the root causes. Regression models can be employed to detect anomalies when two data measures exhibit a high correlation (R2 value). For metrics that may not correlate with any other variables, we can attempt to characterize the behavior over time using forecasting algorithms. One state-of-the-art forecasting algorithm is Prophet, developed by Meta. This is particularly useful for metrics related to response times, such as runtime or compile time. In this blog, we aim to provide a comprehensive guide for building your first anomaly detection models based on production data metrics such as runtime, app CPU time, and database time.

Understanding Anomaly Detection

What are anomalies in CRM data? Anomalies can manifest as outliers in customer behavior or unusual patterns in production systems. It’s imperative to promptly determine whether an anomaly is a result of internal factors, such as capacity constraints or excessive resource utilization, or if it is simply an atypical customer behavior. Occasionally, we encounter seasonal or cyclical anomalies, necessitating the analysis of historical data to discern recurring patterns, like spikes in sales during the holidays or drops in customer engagement during specific months. Therefore, constructing anomaly detection models for critical metrics, such as runtime, app CPU time, and database time, is pivotal for gaining insights into CRM production data.

Getting Started with Your Anomaly Detection Model

The anomaly detection models demonstrated here are implemented in Python, utilizing sklearn for regression modeling, Prophet for forecasting. To begin, you need to set up a working environment with Python and these libraries installed. We recommend using Jupyter Notebook as an excellent environment for experimentation, allowing you to run code snippets and test functionality. Here’s a screenshot of the initial Jupyter Notebook cell with all the necessary imports:

pip install sklearn
pip install prophet
import pandas as pd
import numpy as np
from sklearn.linear_model import LinearRegression
from prophet import Prophet
import plotly.express as px
import plotly.graph_objects as go

In-Person and Virtual Conference

April 23rd to 25th, 2024

Join us for a deep dive into the latest data science and AI trends, tools, and techniques, from LLMs to data analytics and from machine learning to responsible AI.

Preparing Your Data

First, you’ll need to access and load your CRM production data. We’ve created synthetic data that closely resembles the metrics collected from a production pod with some of our customers. You can download our synthetic data from here.  

Regression Modeling: Detecting Anomalies in Resource Utilization

For detecting anomalies in resource utilization, we will use regression modeling techniques. This example establishes a linear regression model that correlates the number of transactions (count) of previous release data to a metric of high correlation in the previous release data i.e. cumulative app CPU time or cumulative database time. The linear regression model is then applied to theccccccvrcbdbjbujrvdvvchnrtekgjnuefbknhknjrtn

 number of transactions processed from current release data to yield predictions of the metric of interest on current release data. Finally, the z-scores are calculated using the linear regression applied on the current release data divided by the standard deviation of the linear regression applied on the previous release data. 

lm = LinearRegression()
lm.fit(np.array(prev_release['count']).reshape(-1,1),np.array(prev_release[metric]))
predicted_current_release_metric = lm.predict(np.array(current_release['count']).reshape(-1,1))
predicted_previous_release_metric = lm.predict(np.array(prev_release['count']).reshape(-1,1))
zscores = (current_release[metric]-predicted_current_release_metric)/np.std(prev_release[metric]-predicted_previous_release_metric)

Regression Modeling: Detecting Anomalies in Response Time

To identify anomalies in response time, we will employ the Prophet model. This example shows a Prophet model created with a confidence interval of 99% and yearly and weekly seasonality parameters. We then fit our response data, which we have provided a CSV of P95 runtime for, with hourly granularity. A forecast is created by using the Prophet model to predict on the data which creates a unique confidence interval for each data point. If the actual data point is outside the predicted confidence interval, it is anomalous. The bottom code is a sample for coloring the data green if they are within the confidence interval and red if outside the confidence interval, indicating that they are anomalous. 

model = Prophet(interval_width=0.99, yearly_seasonality=True, weekly_seasonality=True)
model.fit(data)
forecast = model.predict(data)
performance = pd.merge(data, forecast[['ds', 'yhat', 'yhat_lower', 'yhat_upper']], on='ds')
performance['anomaly'] = performance.apply(lambda rows: 1 if ((rows.y<rows.yhat_lower)|(rows.y>rows.yhat_upper)) else 0, axis = 1)
anomalies = performance[performance['anomaly']==1].sort_values(by='ds')
performance['color'] = np.where(performance['anomaly']== 1, 'red', 'green')
performance['name'] = np.where(performance['anomaly'] == 1, 'Anomaly','Within Confidence Interval')

Want to Learn More?

If you are interested in putting all these ideas for anomaly detection together, please attend our ODSC West 2023 Workshop where you will find out more on how to transform the data to fit into these anomaly detection models as well as create impactful visualizations for them.

About the Authors/ODSC West 2023 Speakers

Geeta Shankar is a software engineer who specializes in leveraging data for business success. With expertise in computer science, data science, machine learning, and artificial intelligence, she stays updated with the latest data-driven innovations. Her Indian classical music background has taught her the value of sharp thinking, spontaneity, and connecting with diverse individuals. Geeta uses these skills to translate complex data into meaningful insights that enhance performance and customer experiences.

 

 

Tuli Nivas is a Software Engineering Architect at Salesforce with extensive experience in design and implementation of test automation and monitoring frameworks. Her interests lie in software testing, cloud computing, big data analytics, systems engineering, and architecture. Tuli holds a PhD in computer science with a focus on building processes to set up robust and fault-tolerant performance engineering systems. Her recent area of expertise has been around machine learning and building data analytics for better and faster troubleshooting of performance problems and anomaly detection in production.

ODSC Community

The Open Data Science community is passionate and diverse, and we always welcome contributions from data science professionals! All of the articles under this profile are from our community, with individual authors mentioned in the text itself.

1