Hi Stuart here,
In my last blog post on this subject, I mentioned a very intensive study we undertook in order to test the accuracy and power of TVmap’s ratings forecasting, and outlined the methodologies which we compared. If you haven’t read it already, you can find it here
. In this post I want to expand on that in greater detail, and share with you the actual figures. As you will see, the results were pretty outstanding.
Forecasting ratings is a combination of art and science. There is never going to be one method which is exactly right in all situations. My view is that the best possible forecast is a judgement call made by a skilled media person who bases their decision on all the relevant facts. Such people are expensive resources and the challenge faced by media buying companies is one of using their skilled people in a cost effective manner.
TVmap is a versatile product in that it automatically produces robust seasonally adjusted rating forecasts and these can be used to quickly create or review a media ‘buy’. On the flip side, TVmap also enhances the function of skilled media buyers by providing all the relevant information in a convenient format to assist in making an informed judgement about future ratings. Ultimately it’s the user’s choice as to whether they accept all rating forecasts, or if they spend time reviewing, and if necessary, overriding the automatically generated ratings.
TVmap forecasts both zone averages and programme averages depending upon the context in which the forecast is to be used. This study focuses on evaluating the effectiveness of the zone level forecasting.
Whilst the seasonal adjustment is performed in a similar manner, additional ‘smarts’ are used to avoid the pitfalls encountered by traditional buying systems when forecasting programmes. This results in TVmap delivering higher accuracy in forecasting media buys than alternative methods.
A process of continuous improvement
We are constantly reviewing our forecasting and refining it when we identify a significant improvement. To give one example of how we assess if the TVmap predictions are still at the leading edge of accuracy, we built a simulation framework which tried a variety of methodologies and compared them with traditional methods of forecasting.
This is the largest study of this type that we are aware of in New Zealand. It involved calculating forecasts for 157,080 individual points using six pre-agreed methodologies over three different forecast lead times and comparing these with the actual ratings achieved. Each individual point required 185 contributing rating calculations as part of the forecasting, meaning that in total nearly 30 million rating calculations were required!
The scope of this study has been made possible by a number of unique factors available to us through TVmap:
• The flexibility and speed offered by the latest version of TVmap in Excel allows very sophisticated analysis over a large number of data points with the use of advanced Excel skills rather than the historic requirement to use a programming language or a statistical analysis package.
• Using our new Excel team whose primary function is to assist our clients produce custom analytical and reporting work. This resulted in the benefit of a robust and flexible development platform, from which we can do further studies if additional hypotheses are to be tested.
What we found was that overall, TVmap’s forecasting engine is the most accurate of the computerised methods of forecasting.
Of the 38 scenarios analysed in this study:
• TVmap outperformed ‘Same Time Last Year’ by 24.4%, winning in 37 scenarios and was equal in performance in the remaining scenario.
• TVmap outperformed ‘Recent weeks without seasonal adjustment’ by 35.8%, winning in 37 scenarios and was equal in the remaining scenario.
In terms of experiments to improve the current TVmap methodology:
• Upweighting recent years when calculating seasonal factors offered a marginal improvement in forecast accuracy, but this was not significant. There is scope for further experimentation with different weighting factors in this area.
• Using individual days of the week when calculating seasonal factors (instead of using entire weeks) did not improve the accuracy.
• Introducing a seasonal share factor did not improve the accuracy.
Thanks for taking the time to read this, and if you have any questions or comments, please get in touch with me.
+64 9 304 0762