Michael Baker - Thesis - Problems in Longterm Forecasting and Planning
About | Contents
Previous: 4. Review of Lessons learnt | This: 5. Forecasting | Next: 6. Criticisms of Forecasting
In this chapter I shall explain how forecasts are made [1]. The chapter starts with a review of forecasting which gives definitions of some of the terms I use, and an outline of the processes used. Following this I shall describe how models are used in forecasting, the inputs to them and how they are constructed. I shall conclude the chapter with a description of the general assumptions involved in forecasting. In the next chapter I shall deal with some of my criticisms of forecasting.
In the forecasting field there appear to be as many definitions of commonly used terms as there are writers on the subject. The definitions they give are not always consistent with one another. Many authors (particularly those who make forecasts as opposed to those who write about forecasting) have no clear idea as to the difference between such terms as:
Rather than make a long review of definitions of terms I shall just quote some of those used by Jantch (1967).
"The following terms have been adopted because (a) they are simple and comprehensive, (b) they correspond to the actual pattern existing today at the operational level and, therefore, (c) best serve the purpose of this report. They are not intended as rigorous definitions and no claim is made for their universal applicability....
A forecast is a probabilistic statement [2], on a relatively high confidence level, about the future. A prediction is an apodictic (non-probabilistic) statement, on an absolute confidence level, about the future. An anticipation is a logical constructed model of a possible future, on a confidence level as yet undefined.... 'The future' referred to in these notions includes situations, events, attitudes, etc." (Jantch 1967)
It is upon the above which I base the following definitions which I shall use in this thesis.
A forecast is a statement about the future which its author puts forward as having a high probability of being true.
A prediction is a statement about the future which is author is certain will be true.
A scenario is a logical model of a possible future with no specified probability.
An extrapolation is the result of continuing a past trend into the future.
A projection is much the same but can also refer to a wider view of the future, such as a scenario which was constructed from a consideration of past and current events. I shall only use the term projection where it is clear from the context the meaning which I attach to it.
One of the difficulties I have in selecting words to describe what I mean is to find a word which describes any process of making statements about the future. The word which seems most appropriate is forecasting. However this can be confused with the narrower definition which I have given above. Where necessary I shall make clear which of these two meanings I attach to the word forecast.
Before describing how forecasts are made I shall first give my views on the accuracy and reliability of forecasts.
After the event forecasts can be taken as predictions and compared with the actual outcome. Accuracy is then a measure of the closeness of the forecast to the outcome. Before the event it is not possible to say anything objective about an individual forecast. However it may be possible to say something about the process or method used.
The accuracy of a forecasting model can be judged by how well it explains past behaviour and that of a forecasting process can be judged by the accuracy of past forecasts (of events which have since occurred) made using the given process. It is possible to make judgements of particular forecasts or models or processes in terms of subjective prior probabilities. However there is no necessity for my perception of what is accurate to be the same as anyone else's.
My view on reliability (of a set of forecasts) is that it is a measure of the accuracy of a set of forecasts. For example, a set of forecasts in which some forecasts were very inaccurate while others were not, would be less reliable than one in which all forecasts were reasonably accurate. Reliability applies to a set of forecasts, or to a forecasting method.
I shall make no attempt to quantify either accuracy or reliability. However if I were to do so, I would start from the basis that they were related in the following way.
R = sum from i=1 to n of (1/ei)2/n
Where R is the reliability of a set of n forecasts whose errors were ei.
Reliability and accuracy can only be measured and talked about "objectively" after the event(s) forecast have occurred. Before this time any use of these terms must be subjective, that is they are the opinion of the person making the comment.
Forecasting errors can only be measured objectively after the forecast event has occurred. Error is the difference between forecast and outcome. It can only be numerical if the forecast is numerical. If the forecast is that an event will/won't happen then there can only be error/non-error.
In general all types of forecasting, prediction etc. use the same basic process. This can be summarised as a model (of the system in question) into which data is input, and from which a result is output (see Figure 5.1).
Figure 5.1 Basics of a forecasting model
The model may be no more than an observed correlation between two variables (such as primary energy demand and gross domestic product), or it may contain relationships between many variables some of which are input (exogenous) and some of which are determined by the model (endogenous). An example of the latter type of model is that of Energy supply and Demand used by the Department of Energy (1978). There are also a whole variety of models which lie between these two types, of varying degrees of complexity.
The data which is input to the model is of two basic types. These are data used for calibration and that used for projection. For example in the case of the energy demand/GDP model past data on primary energy demand and of gross domestic product for a number of years are used to calibrate the equation used in the model, then projections of GDP are used in the model to get the corresponding projections of energy demand.
To make a forecast it is necessary to have a model which covers all of the "relevant" variables. In this context I take "relevant" to mean all the variables which have a significant effect on the variables being forecast. However due to limitations in mans' ability to comprehend complete systems, any model will be an abstraction of the salient supposedly features of the system of interest.
All models exclude factors which could affect the result/output. However the model builder assumes that all factors which he excludes from his model have negligible effect. That is he assumes that such factors can be neglected without loss of accuracy. He must also assume that the effect of such factors will remain negligible.
There are several ways in which models vary. Some of these are due to:
The sections below cover these aspects.
The example given above of an energy demand/GDP relationship used in a forecasting model contains no disaggregation. To improve the "accuracy" [3] of the model, energy demand could be disaggregated by sector (such as Domestic, Industry, Transport and Other) and the demand could be related to activity in each of these sectors. A further disaggregation could be made by splitting the demand for energy in each sector by the type of fuel used (such as Coal, Oil, Gas, Electricity) and relating the demand for energy to useful demands (such as heat, light and work) rather than primary energy. A slightly higher level of disaggregation than this is used in the Department of Energy's (1978) forecasting model, and by Chapman et al (1976).
Disaggregation may improve the accuracy with which the model matches historical data, however it does require consideration, and usually projection, of many more variables to arrive at the desired forecast. In the above example of a disaggregated energy demand model instead of making a projection of one variable (GDP) projections of maybe 30 to 40 variables (activity levels in 4 sectors, 4-6 primary fuel conversion efficiencies, and about 10 fuel to useful energy conversion efficiencies, plus the market share of the four fuels in the four sectors) would have to be made.
Another way in which models vary is whether the relationships in the model are causal relationships or not. For example the relationship between primary energy demand and GDP is not a causal relationship whereas a relationship between the heating requirements of a building and the inside and outside temperature is causal. The difference is that the first relationship is only based on historic observation whereas the second is also based on theory [4]. (In this case the theory is Newton's Law of Cooling which states that the rate of heat loss is proportional to the temperature difference between an object and its surroundings).
These two examples lie near the two ends of a continuum. There are also cases where the causes of a relationship are partially known. For example a relationship between useful energy demand and a sector's level of activity can be considered to be mainly causal. However there are, or may be, other factors which could also affect the level of useful energy demand. These include the mix of outputs of the sector and the mix of processes used. In general a single process will have a fixed relationship between useful energy demand and output, so the extent to which demand and output are causally related in a sector will depend upon how similar this relationship is between the different processes used and its mix of such processes.
In general the more disaggregated a model is the more likely it is that the relationships will be causal. Consequently a more aggregated model is less likely to have causal relationships. There are advantages of having causal relationships in a model in that it is easier to see if the relationships, and so the validity of the model, are likely to persist over time. However there is often the disadvantage noted above of increased complexity caused by disaggregation.
Cross (1975) following the presentation of Wilson (1968) identified two types of differences between models. These are whether the model is static equilibrium or dynamic and whether it is deterministic or probabilistic. This gives rise to four different types of model.
- Deterministic static equilibrium model. In this model the system is described at a particular time by sets of equations which can adequately represent the existing system. It is deterministic in that for a given set of input variables a unique model is defined. It is a static equilibrium model in that there is no provision for making the model evolve with time. However, input variables appropriate to a future epoch may be used to define a future state assuming the same equilibrium model remains valid.
- Probabilistic static equilibrium model. This model is similar in structure to (a) but now the relations are acknowledged to have a range of uncertainty. For a given set of input parameters, the model predicts a range of values for each output variable. To each value a relative probability can be assigned.
- Deterministic Dynamic Model. This type of model explicitly includes time as one of the variables. Thus, in contrast to (a) where it is assumed that the system adjusts itself rapidly to the equilibrium situation, here the dynamic behaviour of the individual components of the system are explicitly included. The model is again deterministic in the sense of (a).
- Stochastic Dynamic Model. This is similar to (c) but now uncertainties are explicitly included in the input relations and their dynamical behaviour. A range of models of the future systems are produced with assigned probabilities consistent with uncertainties in the model of the system. (Cross 1975, p28)
Most forecasting models are deterministic and most are also static equilibrium so that the large majority of models fall into category (a) above.
There are two general stages in the construction of a Forecasting model. The first is the composition of its structure and the second is its calibration which may or may not include verification.
There would appear to be no hard and fast rules about the construction of a forecasting model. Having said that though there are usually two stages involved. In the first stage components of the system to be modelled are identified and in the second the relationship between the parts or variables are determined.
The number of variables involved in the model and the level of disaggregation will depend on several factors. These include the resources available to construct and use the model, the required "level of accuracy" of the forecasts, the level of understanding of the model builder of the systems he is modelling, and the desired comprehensibility of the model. More resources, higher "accuracy" and more understanding will (in most cases) tend to lead to the use of more variables and greater disaggregation. However this also tends to lead to a less comprehensible model.
There is a continuum of possible sources for the relationships in models from well established laws through theories, hypotheses, to empirical studies. Beyond empirical studies there is more or less well informed speculation. The use of empirical studies covers a wide field which includes time trend analysis (using such techniques as those developed by Box and Jenkins (1970)), multiple linear regression, etc.
Some relationships may be omitted from a model on the assumption that the relationships will not change over the forecast period and so will have no effect on the forecast.
There are three broad areas in which assumptions are made when constructing a forecasting model. These are about the scope of the model, the continuity of relationships over time and the constancy of social "values" and "preferences" over time.
It is generally assumed that all of the variables or factors which might have a significant effect on the variable being forecast have been included in the model. As mentioned above it is possible that a relationship between variables is not included on the assumption that it will not change over the forecast period.
The second area in which general assumptions are made is that of continuity of relationships. The relationships in the model are either assumed not to change over the forecast period, or it is assumed that if they do change it will be in known ways which are built into the model.
Many of the relationships in forecasting models are dependent upon social "values" and "preferences". For example there may be a preference for foreign holidays in the sun rather than holidays in the UK. This preference will have an effect on holiday traffic forecasts both on destinations, inside or outside the UK, and mode of travel, car or aircraft. In general it is assumed that changes in "values" and "preferences" can be left out of forecasting models.
As illustrated in Figure 5.2 there are three types of input to a forecasting model.
Figure 5.2 The three types of input to a forecasting model
These are:
When a relationship is included in a model various assumptions are made. A relationship in a model may be derived from a well established theory, or from past observation, or from an unestablished hypothesis. In each of these cases the assumption is made that the relationship does hold and that it will continue to hold in the future. Another assumption that is made is that all of the relevant variables have been included in the relationship.
Having determined the structure of a model and the relationships in it, it is necessary to calibrate the relationships. That is to find numerical values for any constants in the relationships.
The final input to a forecasting model is of projected future values for exogenous variables. Specifically these are required for static equilibrium models. For example to obtain a forecast of primary energy demand using a simple energy demand/GDP model it is necessary to make projections of GDP over the period for which the forecast is required.
I shall say more on the possible sources of projected data below.
Each of the inputs to a forecasting model, as described above, can be classified as being either historic, or an extrapolation, or target, or from another forecast. This classification is illustrated in Figure 5.3 and is described below.
Figure 5.3 Origins of the inputs to a forecasting model
In the case of calibration data the only source is historic data. Due to the finite time it takes to collect data on what is happening now, it is not possible to know what is happening now, but only what has happened in the past (up to some point near the present). Consequently the only data available to calibrate a model is historic data.
There are three possible sources for projected data input to models. These are extrapolations of past trends, targets or other forecasts. If an input to a model is projected on the basis of a relationship with other variables, either this relationship can be considered to be a part of the model, in which case it is the "other" variables which must be projected for input to the model, or the projection of the variable on the basis of the relationship can be considered to be a forecast its self, in which case the projected input is the result of another forecast. In either case the ultimate inputs are made upon the extrapolation of past trends. That is, at some point recourse is made to the argument that, "for this variable we shall assume that its observed past behaviour will continue into the future". This is often made without any underlying reason. In many cases the past behaviour of the variable is that is has remained constant and it is assumed that it will continue to be constant in the future.
The second possible source of projected data for input to models is targets. For example one of the inputs to a model might be GDP, and rather than making an extrapolation of GDP from past trends or using a projection from another forecast, the values input may be targets. To the extent that targets are used as the inputs to a model the resultant forecast is normative rather than positive. That is, it is made on the basis of what for forecaster (or those he is forecasting for) wants to happen rather than on the basis of what he thinks will happen.
The use of a target [5] as an input might be to determine the effects of reaching a target, or it might be to determine the necessary prerequisites of reaching a target, or it might be used on the assumption that the target will be reached. In the latter case the target is effectively being used as another forecast.
The third source of projected data for input to models is from other forecasts. More is said, about this source below.
Like projected data there are also the same three sources for relationships in the model. In most cases the relationships in a model are extrapolations of relationships from the past. This extrapolation is usually that the relationship will be the same in the future as in the past. However some times a relationship has been changing in the recent past and this change in the relationship is projected into the future. The use of targets or other forecasts as the source of relationships is less common. It happens most often in dynamic models where the only way inn which the results of the model can be changed is by changing the relationships used.
The three sources of input for relationships and projected data are illustrated in Figure 5.4.
Figure 5.4 Basic inputs to a forecasting model
However since the inputs to all other forecasts also consist of these three sources there are only two independent inputs to the forecasting process in general. These are extrapolations of past trends and targets, as illustrated in Figure 5.5.
Figure 5.5 Exogenous inputs to forecasting
In this chapter I have covered some of the terms used in the literature on forecasting as well as making a very brief review of the literature. I then went on to describe the way in which models are used in, and constructed for, the forecasting process. In the next chapter I shall describe some of my criticisms of the forecasting process.
[1] My description is of the general processes used. It is not concerned with detailed mathematical techniques such as those developed by Box and Jenkins (1970).
[2]
"European readers, by association with 'weather forecast,' sometimes
think of a forecast in the same sense as has been adopted here for a
prediction.
A weather forecast is usually given in the form of a prediction in Europe,
but as a probabilistic forecast-'80 percent probability of rain'-in
the United States and Canada."
(Jantch 1967)
[3] The degree to which the model represents reality. That is how well it could (in the view of the model builder) give a true reflection of the effects of different conditions/possible futures.
[4] My model of the development of "Scientific Knowledge" goes something like:
So the difference between "historic observation" and "theory" is one of how far the above process has progressed. However for any events which are not repeatable the process can not be taken beyond stage (b).
[5] Within the model the target will be treated as a constraint.
About | Contents
Previous: 4. Review of Lessons learnt | This: 5. Forecasting | Next: 6. Criticisms of Forecasting
Copyright © Michael Baker 1981,2005. All Rights Reserved.
