Statistical Performance Evaluation of the Model Statistical Performa ma

Masoud 2014,... The performance evaluation of the model can be described below:

3.1 Statistical Performance Evaluation of the Model

In order to estimate the forecasting statistical performance of some methods or to compare several methods we should define error functions. Many previous research works had applied some of the following forecast accuracy measures: Mean Error ME, Mean Absolute Error MAE, Mean Squared Error MSE, Root Mean Squared Error RMSE, Standard Deviation of Errors SDE, Mean Percent Error MPE and Mean Absolute Per cent Error MAPE, etc. In our study we use four performance criteria namely mean absolute error MAE, root mean square error RMSE, mean absolute percentage error MAPE and goodness of fit R 2 . The back- propagation learning algorithm was used to train the three-layered feed- forward ANN structure in this study were the most used error functions is as following: The mean absolute error is an average of the absolute errors E = P i - , where P i and are the actual or observed value and predicted value, respectively. Lesser values of these measures show more correctly predicted outputs. This follows a long-standing tradition of using the “ex-post facto” perspective in examining forecast error, where the error of a forecast is evaluated relative to what was subsequently observed, typically a census based benchmark Poon 2005. The most commonly used scale-dependent

3.1 Statistical Performa ma

n nce Evaluation of the Mo M del In order er t to estimate the for or ec c as s ti t ng ng statist t ic i al performan ce ce of some methods or or to compare e se e ve ve ra r l meth h ods we sho ho ul ul d d de defi fi n ne error fun nct c ions. Many previo io us us r r e esearch h w work s ha d ap plied so o me me of th he e fo fo ll ll owing forecast ac accu cura ra cy m m ea sure s: Mean Erro r M E, Mean Abs ol ut ute e Erro ro r M MAE AE , M Mean Sq S uare re d Error MSE, Root Me an Squar ed Error R RMSE E , , S Standar rd De e v vi ation of Errors S DE , Mea n Percen t Er ror MPE and M Mean n A A bs bs ol o ute P Per cent E rr or MAPE , etc . In o ur s tu dy we us e fo ur perfo rm m ance c criteri ria a n na mely mean absolu te err or MAE, root mea n square error RM MSE, m mean n a ab so lute perce nt ag ag e e er er ro r r MAPE and g g oo oo dn dn ess of fit R 2 . T T h he bac ac k- k- propagation learning algorit ithm hm w wa as used to train the three-layered f fee ee d d- forward ANN structure in this study were the most used error fun n ct ct i ion ns i is s as fo fo ll ll ow ow in in g: g: Th Th e me mean absolute error is a a n averag age of the absolute er er ro ro rs rs E E = P i - , where P i and are the e actual or observed value and predicted value, d respectively. Lesser values o of these m measures show more correctly predicted outputs. This follows a long-st tand ding tradition of using the “ex-post facto” summary measures of forecast accuracy are based on the distributions of absolute errors |E| or squared errors E 2 observations n is the sample volume. The mean absolute error is given by: Mean Absolute Error MAE = ¸ ¹ · ¨ © § ¦ n i n E 1 i = 1, 2,…n 6 The MAE is often abbreviated as the MAD “D” for “deviation”. Both MSE and RMSE are integral components in statistical models e.g., regression. As such, they are natural measures to use in many forecast error evaluations that use regression-based and statistical. The square root of the mean squared error as follows: Mean Square Error MSE = ¸ ¹ · ¨ © § ¦ n i n E 1 2 i = 1, 2,…n Root Mean Square Error RMSE = ¸ ¹ · ¨ © § ¦ n i n E Sqrt 1 2 i = 1, 2,…n 7 If the above RMSE is very less significant, the prediction accuracy of the ANN model is very close to 100. Since percentage errors are not scale- independent, they are used to compare forecast performance across different data sets of the area using absolute percentage error given by APE = P i - 100. Like the scale dependent measures, a positive value of APE is derived by taking its absolute value | APE | observations n. This measure includes: MAPE = ¸ ¹ · ¨ © § ¦ n i n APE 1 i = 1, 2,…n 8 volume. The mean absol ol ut ute error is g iv iv en en by: Mean A A bs olute Error M M AE = ¸ ¹ ¸¸·¸¸ ¨ © © ¨¨§¨¨ ¦ n i i n 1 1 i i = 1, 2,…n 6 T The MAE is of fte ten n ab ab breviated as the E MAD “ “ D” D” f f or “de de viation” ” . . Both MSE and RM RM SE SE are i i nt nt egral compon en ts in statis ti i ca cal l mode d ls ls e. e g. g , regre ession. E As As s such, , t t h he y are na tural measur es to use in m an y fo re ca ca st e rr rr or or e ev valuat io io ns that u use regressio n- based and st atistical. T he square ro o ot o o o f f th the e mean n sq q u ua red er ror as follows : Mean Square Erro r M SE = ¸ ¹ ¸¸ · ¸¸ ¨ © ¨¨ §¨¨ ¦ n i n 1 i = 1, 2, … …n Root Mean Square Error r RM RMSE E = = ¸ ¹ ¸¸ · · ¸ ¸ ¸ ¸ © § § ¨ ¨ ¨ ¨ ¦ ¦ n i n E S Sqrt ¨ © ¨¨§¨¨ ¦ 1 i = 1, 2,…n 7 7 If If t t he h above RMSE is very less significant, , the pr p ediction a a cc cc u urac c y y of of t the ANN model l is is v v er y y close to to 1 100 00. . Since p p erce cent ntag ag e errors are e n not ot s cale- in in de de pe pe nd nd en en t t, they are used t t o o compar are forecast perform m an an ce ce a a cr cr os s different data sets of the area using a absolute per rcentage error given by APE = P i - 100. Like the scale dep pendent m measures, a positive value of APE is derived by taking its absolute v value ue | APE | observations n. This measure E i l d i = 1,2,...n 10 The use of absolute values or squared values prevents negative and positive errors from offsetting each other. All these features and more make MATLAB an indispensable tool for use in this work. Goodness of Fit R 2 = ¸ ¹ · ¨ © § ¦ n i e E 1 2 2 i= 1, 2,…n 9 where e i = p i - p i , is the forecast error values. p i , the actual values and p i , denote the predicted values. The more R 2 correlation coefficient gets closer to one, the more the two data sets are correlated perfectly. As the aim of all of the prediction system models proposed in this study is to predict the direction of the stock price index forecasting, the correlation between the outputs do not directly reflect the overall performance of the network.

3.2 Financial Performance Evaluation of the Model