IBM Watson AutoAI Time Series Forecasting Bakeoff

Warning: this content is older than 365 days. It may be out of date and no longer relevant.

IBM Watson AutoAI Time Series Forecasting Bakeoff

Today, let’s take a look at a new offering from IBM, the new AutoAI Time Series Forecasting module. Before we begin, let’s define a few things.

What is Time Series Forecasting?

Time series forecasting is predicting one or more data variables over some future time, based on previous data.

Why is Time Series Forecasting Valuable?

Because you want to know what’s likely to happen.

When is Time Series Forecasting Not Valuable?

When you’re trying to forecast something that hasn’t ever happened, or you’re trying to forecast something that cannot be forecasted because there are too many inputs.

For example, let’s say you were forecasting your restaurant business, and it’s February 2020. All your forecasts are based on previous data. No software in the world, no matter how good, would have successfully predicted the global pandemic.

The Bakeoff

We’ll compare two different methods for time series forecasting. First, we’ll use the traditional, code-based method in the R programming language. Then we’ll compare it with IBM Watson AutoAI’s Time Series Forecasting.

We start with good ol’ Google Analytics data. For this exercise, we’ll keep it simple. Based on the last few years of data from my website, can I forecast ahead what my likely traffic is going to be?

Google Analytics data

This is nearly ideal data for time series forecasting – lots of back data to work with, well-structured, nothing missing. If one or more of these conditions weren’t true, we’d have a hard time doing any kind of forecasting.

The Highly Technical Forecast with R

Using the R programming language, we feed our data into 6 different algorithms: ARIMA (autoregressive integrated moving averages), ETS (error trend and seasonality), Facebook’s Prophet algorithm, with and without XGBoost (extreme gradient boosting), GLMNet (generalized linear models with elastic net model paths), and random forests. These are all tried and true time series forecasting techniques with varying levels of technical sophistication.

We look at the outputs from our backtesting first to see which model gives us the greatest accuracy levels:

Accuracy table

Gold stars to the Prophet models for having the lowest SMAPE (symmetric mean absolute percentage error) rates; this indicates that these models performed the best for time series forecasting. That said, R squared is a little on the low side for all of them, which means the models may not necessarily have done a great job of predicting what’s likely to happen.

Next, let’s look at the forecasts themselves. What do the models look like?

Forecast plot

We see some models, like Prophet (lime green), really get a decent model of the ebbs and flows of website traffic. Others, like ARIMA (red), literally just project a straight line. That’s super not helpful.

Okay, so we’ve got a time series forecast (or set of forecasts, really) from R. Quick list of pros and cons.

Pros of R forecasting

Free, you run it on your desktop
Goes as fast as your computer goes (don’t try this on a weak machine)
Highly configurable, you write the code to do what you want

Cons of R forecasting

You have to know how to code
You also have to have deep statistical knowledge to understand the performance of your models
Need addons (also free) to make the process easier

The IBM Watson AutoAI Forecast

Next, let’s follow the same process with IBM Watson. We load our data in:

Watson Data Load

We see what it’s going to test, which is handy. Right now Watson can only predict 60 days ahead. Below, we see its backtesting process, which will check its predictions against known historical data:

Backtest diagram

Then, as we did manually, it automatically tests dozens of different combinations of algorithms and models. Note that I don’t have to do any of this as the user. It does it for me.

First run

And it settled on BATS (Bayesian Analysis of Time Series):

Final output

We see that the SMAPE error for Watson’s choice of BATS is not only lower than the algorithms we tested in R, but the R squared error metric is MUCH better, almost 3x better.

When we look at the forecasted line, the dotted green line above, we see that as well. The series looks like a much more reasonable growth model. In the R forecast, the Prophet models looked good but they climbed ever higher; the Watson model increases, but seems to do so at a much more reasonable rate.

So, let’s look at the pros and cons of Watson AutoAI time series forecasting.

Pros of Watson AutoAI Forecasting

More accurate out of the box than our custom built model
No coding!
Many more models and techniques tested out of the box

Cons of Watson AutoAI Forecasting

No data export of the forecast itself
Limited to a 60 day forecast window
Not free

What’s the Best Time Series Forecasting Solution?

The best choice for time series forecasting is going to depend on your needs and use case. IBM Watson AutoAI delivers very high quality forecasts in a no-code environment, suitable for offering short-term (60 day) prediction windows. It’s not free, but it’s substantially easier to use than the open-source option.

R delivers high quality forecasts that are free and can extend out very far into the future (within statistical limits), but require substantial technical and statistical skills to implement.

Which you choose depends on what you need more, but both are good solutions for some organizations, not as good a fit for others, and both will continue to evolve and improve over time.

Important Compliance Disclosures

I am an IBM Champion, for which I receive non-monetary compensation from IBM (mostly apparel). IBM did not solicit this review.

My company, Trust Insights, is an IBM Registered Business Partner. Should you purchase something from IBM through Trust Insights, I would receive indirect financial benefit.