towardsai.net
The Complexities of Trend Detection in Time Series Data 0 like February 12, 2025Share this postAuthor(s): Lily Chen Originally published on Towards AI. One common use case of working with time series data is trend detection, or trend analysis. A real life example is looking at memory data to see if a service is experiencing a memory leak.Some time ago, my team at Datadog released a Memory Leak Wizard, whose aim was to help users solve memory leak issues. When the feature was first released, we asked the users to manually answer a set of questions before bringing them to what they need to solve their potential memory leak issue.We first ask if there is a goroutine leak.If there isnt (as in the example above), we ask about Live Go Memory.In this case, there is growth, and after users selects Yes, we show the relevant Comparison Flame Graph that they can use to investigate where the growth in memory is from.Of course, in this modern age of AI, you might be wondering why does a human need to answer this? And you are correct, this could very much be automated.Fast forward a few months, we released the automated version of the Memory Leak workflow.Automatic trend analysisHowever, the process of capturing an upward trend in a time series data has some complexities involved. Here are some pitfalls I ran into and lessons I learned that might help others or help my future self if she were to do this kind of work again.First of all, is this even an LLM problem? My conclusion after working on this project is no. This is not a problem best solved by the current generation of LLMs like ChatGPT-4o. But before we get to why thats the case, lets delve first into some of the complexities of working with time series data.The necessity of scaling the dataThe way to identify linear growth in data is by performing linear regression. Linear regression finds the slope of the line of best fit, and the R-Squared value. Typically, to assess whether a dataset has a linear upward trend, we first define a threshold for slope and a threshold for the R-Squared value. A reasonable threshold for the slope could be 0.1. If both values are greater or equal to the threshold, we say the dataset has a linear growth.Linear regression, however, is sensitive to the magnitude of the data. When performing time series analysis, it is important to take that into consideration.For all examples below, the x-axis values are in milliseconds since epoch, or what Date.now() would return in Javascript.Lets look at a simple dataset with 2 points. In both examples below, the y doubled. But in the first case, a linear regression will return a slope thats pretty much 0. A slope of zero (or below) means no upward trend.Case 1: y axis values go from 10 to 20If you try to fit a linear regression line through this data for an hour time frame, you will get a slope close to zero, i.e. no growth, even though the data doubled.Case 2: y axis value go from 1 million to 2 millionWhen looking at memory usage in bytes, the y-axis values could be in the order of millions or billions. In this case, you might not want to scale the x-axis, depending on the actual time frame and memory values are.The caveat with scaling dataIf you scale the x-axis, the value of the magnitude of the slope kind of becomes meaningless on their own, and not to be relied on alone.Lets look at this following example. Lets say the threshold for positive growth is 0.1, and you have 2 points where the y-values go from 100 to 101. Lets say theyre taken 1 minute apart so the deltaX (without scaling) is 60 * 1000.Without scaling, the effective slope is 1 / 60,000 or 0.000017 (i.e. very close to zero), so it would not pass check for linear growth. Good.But lets say we scale the x-axis. Lets set t1 to be 0, and t2 to be t1 + 60000. Or in other words, if t2 is Date.now(), t1 would be t2 - 60000.To scale the timestamp, lets divide the timestamps by 6000. The delta X after scaling would be 10.In this case, the effective slope is 1 / 10, or 0.1, and would pass a threshold of >=0.1 for linear growth.Thus, slope alone is not reliable when scaling of the axis is involved.To get around this, we also need to look at the min(y) (minimum of the y-values) and max(y) (maximum of the y-values) of the entire series. If the max(y) is not a certain threshold greater than min(y), we do not label the graph as trending upward. For example, if max(y) <= min(y) * 1.05, i.e. if the data has not increased by 5% between the min and max, we can call it not increasing in a significant manner, even if linear regression after scaling gives us a large slope with high confidence.This is not foolproof, because what if max(y) is actually before min(y) in the time series? Were using the ideal scenario that max(y) is towards the end and min(y) towards the beginning to filter out non significant growth. Hopefully it is enough in the context of what were doing at Datadog in the context of the Memory Leak Workflow. However, its something to keep an eye out on, and well deal with this problem if we ever cross the bridge.So far Ive only talked about linear regression, which is good at detecting linear data. However, there could be many shapes of growth in the context of a memory leak, such as quadratic or exponential growth.Ill write in a later blog post the challenges of working with other regressions in the context of real world trend analysis, but first, the big question you might be wondering:Can LLMs do better?What would happen if we give the data to LLMs like ChatGPT-4o and ask it to tell us if the graph is increasing?I fed the model some goroutines data and asked if theres an upward trend, and it answered:While the trend is technically increasing, the low R-squared value suggests a lot of variability in the data, meaning the upward movement isnt strong or consistentTo us humans, this is clearly an increase not just linear, but what appears to be exponential. Yet, ChatGPT-4o labeled this as an upward movement that isnt strong or consistent. Fail.You can tell from the image ChatGPT generated that it did a linear regression on the dataset. If you expand to view more, you can see it using scipy.stats.import pandas as pdimport matplotlib.pyplot as pltfrom scipy.stats import linregress# makes pandas DataFrame from the given data...# Perform linear regression to determine trendslope, intercept, r_value, p_value, std_err = linregress(df['timestamp'], df['value'])The aforementioned goroutines data needs scaling; notice how the slope is almost zero? The data would also much better fit an exponential or quadratic curve. Yet ChatGPT-4o isnt smart enough to know those things on its own.We learned 2 things from just this one example:LLMs arent magic, and in this case, it used statistics under the hood.LLMs dont solve this type of problem well (at least not without prompt engineering).Ive written previously about how the current generation of LLMs dont handle context-dependent problems well. When it comes to working with time series data in the real world, its all about the context.Perhaps we could train an LLM to do this through prompt engineering, telling it to run exponential and quadratic regressions if linear regression fails to pass the threshold. But is doing that really better than coding it up? Id have to know exactly what to do to detect an upward trend, whereas in the ideal world, the LLM can do it automatically, the LLM knows all the pitfalls and edge cases to look out for.I have to admit that when I undertook the task of automating the Datadog Memory Leak Workflow, my first instinct was surely ChatGPT can do this. But now that I know ChatGPT merely do regressions under the hood, without even taking into account the intricacies of working with time series data, I know it cant be trusted to solve this kind of problem.It seems nowadays, many people treat any problem as an LLM problem. I often see on LinkedIn posts from people that are along the line of we wont need to hire software engineers soon because I had [insert LLM model] code up some [insert prototype functionality] for me.While Im certain optimistic about the accelerating improvements were making to LLMs, weve observed here that to blindly use LLMs to solve any problem can be dangerous. I know this is true when it comes to code generation, as of Feb 2025. If you trust these models blindly, youd either get lucky or get it wrong.Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming asponsor. Published via Towards AITowards AI - Medium Share this post
