Current Issue

ARIMA is a forecasting model for univariate data and is limited for this study dataset which is a multivariate type. Moreover, another model can account for more than just past prices or residuals, named the "MAX" model. The name ARIMAX derives from an obtainment of the ARIMA, respectively. The X added to the end stands for "exogenous." The model suggests adding various variables to assist measurement of the endogenous variable[5]. In time series, the exogenous variable is a parallel time series that are not modeled directly but is used as a weighted input to the model. Therefore, this model fits univariate time series with trend and seasonal components and exogenous variables.

Developed by Facebook, the model is known for its simplicity for the modular regression with the critical components of trends, seasonality, and holidays. There are a variety of functional benefits of this approach. Due to weekly and annual seasonality, the seasonal feature provides a flexible model of periodic changes[6].

LightGBM[7] utilizes the boosting method to combine multiple weak learners and develop a strong learner. This robust and expressive model can be utilized for most regression problems. Among the general regression tree algorithm models, LightGBM has the superiorities of preventing over-fitting, outstanding generalization ability, faster speed, and lower memory consumption.

XGBoost is a set of machine learning techniques with a scalable boost tree. XGBoost calculates the results of K trees as the final predicted value. XGBoost also shows shrinkage and column subsampling. Besides, XGBoost uses acquisitive and approximate algorithms to find the most acceptable split point [8].

GRU architecture was introduced[10] and is regarded as a deviation of Long Short-Term Memory (LSTM) as it uses identical functions but is organized differently. To solve the disappearing gradient problem of a standard RNN, GRU employs the so-called update gate and reset gate. These two vectors determine what information should be transferred to the output. Their phenomenal specialty is that they can be trained to keep information from long ago without removing it through time or withdrawing information unrelated to the prediction.

Bi-LSTM just put two independent RNNs together. This structure enables the networks to hold backward and forward information about the sequence at every step. Furthermore, Bi-LSTM operates the inputs in two ways, one from past to future and one from future to past, which differs from unidirectional LSTM.

As shown in Fig. 2, all three particle types contain outliers. Then, we would like to see the total amount by each day of the week and by the time of the day. Fig. 3 shows that all three particles obtain the most on Thursday; meanwhile, the highest peak is from 2 pm to 3 pm during the day.

Fig. 2. 
Outliers present in zone 100

Fig. 3. 
Histogram grouped the particles by days and hours

Since the data is not normally distributed and contains outliers, the Spearman Ranking analysis was conducted to check the correlation coefficient between the variables. As shown in Fig. 4, only particle05 is strongly correlated with particle03. There is no other significant relationship between particle variables with other variables. Therefore, we will focus on particle05 forecasting from here.

Fig. 4. 
Correlation analysis results using spearman

⦁ Stationary Test: The time series need to quality check on the stationary, self-correlation, trends, and seasonality. Therefore, the study used the Augmented Dickey-Fuller test(ADF) to check whether the data were stationary. Fortunately, the result shows that the data obtained has no unit root and is stationary, meaning differentiation is unnecessary.

⦁ Decompose Time Series: Then, the study decomposes the time series to check its trends and seasonality. As shown in Fig. 5, the data is seasonal, and there is always a peak around 2 pm, like the histogram in the EDA section mentioned earlier.

Fig. 5. 
Seasonality test of particle 05

With the time-series analysis, it is relatively more complicated than simple datasets. The study converted the moving averages and lagging with seven days, 15 days, and 30 days respectively, and included them as the exogenous variables with other existing features to define the models. After converting those features, the datasets with 40,564 rows and 96 columns are ready to be split for the model. After separating, the train data contains 32,500 rows, while validation data contain 8,064 rows.

The research used the desktop with Intel(R) Core(TM) i5-8500 CPU @ 3.00GHz as a processor and 16.0 GB of physical memory to train all the models,

With the inclusion of exogenous variables, four models use the engineered data to train and show remarkable results.

⦁ SARIMAX: since it is seasonal data, SARIMAX was defined. The study used the existing library that automatically performs the stepwise feature selection to find the least Akaike Information Criterion(AIC), a mathematical method to evaluate the goodness of fit. The best model found at SARIMAX(5,0,2)(0,0,0)[0] intercept with the AIC value = 448901.578.

⦁ Facebook Prophet: the model performs the decomposing, including trend, weekly, and daily pattern analysis. The result showed the same pattern as the histogram in EDA.

⦁ LightGBM: the model automatically performed the feature selection. Features used in the model are 78.

⦁ XGBOOST: Time series features such as data, hour, min, day of the week, quarter, month, year, day of the year, day of the month, and week of year were added to the data to enhance the performance of the model.

Fig. 6 and the table1 below show the results of the four models. As highlighted in bold in Table 1, LightGBM outperforms the other three models with the least 93.42 RMSE score, 56.54 MAE score, and 0.3 seconds performing time.

Fig. 6. 
Comparison of four models

By using the same amount of training and validation data, GRU and BiLSTM models were created. These two models were developed with the same hypermeters described in table 2.

As shown in Fig. 7 and Fig. 8, GRU shows slightly better results comparing BiLSTM. With all the six models' comparison, the result indicates that GRU outperformed the other models with the lowest RMSE(88.63) and MAE(54.58), while LightGBM took the shortest(0.3 seconds) computing time, as shown in Table 3. Since GRU is a deep learning model, the performing time is longer than machine learning models; however, this short computation time can be considered efficient since the model was trained using a CPU. The performance would be better by using the higher capability processers.

Fig. 7. 
BiLSTM prediction result

Fig. 8. 
BiLSTM prediction results