On the Limitation of Predictive Models, Even When They Don't Suck
Lordy, but I want to talk about this post I found over at Watts Up With That?
I want to talk about it because... aw heck... let me deliver the punchline first before anyone thinks I'm about to go on some kind of stimulus/bailout rant...
Just to sum up for those of you who skim over multi-paragraph excerpts... Modeling - any modeling - is intrinsically error prone. You've got to be veeeery skeptical about the answer a model gives you, even when it's built by the very smartest people you know, given all the funding they ask for, and trusted with the most sensitive information.
And that's why the financial meltdown is instructive about anyone too certain about the models driving the current "climate change" panic.
We haven't actually observed "Global Warming", despite the alarmist reporting of every unpleasant weather event as "proof" of it. According to even the most alarmist models, the really bad effects of the warming are years, even decades, away. Climate change models are such that "you can’t run experiments with it," as the man said above. Lacking the opportunity for direct evidence, we must rely on models.
The problem is almost no one is recognizing some well known problems related to relying on models in any predictive sphere, and maybe the current problems in the financial sphere can wake a few people up in regard to that.
If you have never personally built a predictive model AND had to see its results compare to reality, you probably don't understand the intrinsic problem with modeling in general. At least not as deeply as you ought to.
As a classroom exercise, modeling is a blast. It makes you feel almost omniscient. You get to define the parameters. You know what you want the model to do. You set the rules. You define the data. You craft it until the thing virtually hums to the tune you play for it (as long as the "tune" can be translated into "data" anyway) It's beautiful.
In the real world we call this the "development environment."
I would imagine that if you live in academia this environment is more or less the primary place you live in and write about. Especially when you're modeling something you can't actually compare to any real world results.
My own experience is a little different. I'm used to testing models against the real world, where people intend to USE the model's predictions to make decisions affecting dollars, careers, and industries. That one extra step is... eye opening.
The first thing you learn is this: You can NEVER account for ALL the events that might impact the data in your model. That means you can NEVER be 100% certain that your model's predictions are correct.
Unless you've tried this, you probably don't believe me. All I can do is recommend that you try it and see for yourself. I don't know a single person who has actually done predictive modeling in the real world who would disagree.
The more complex the system you're modeling the less likely it is you have even considered all the events that might affect your prediction. If that sounds like an exaggeration to you, your inexperience is showing. As Donnie Rumsfeld once said, you've got your unknown unknowns out there.
Compare this basic truth about modeling to this criticism of the modeling work of Climate Alarmist in Chief James Hansen by former NASA scientist John Theon:
If you're someone unfamiliar with modeling, this comment might seem like nit-picking. If you have ever DONE modeling, you recognize this as a statement attacking the fundamental basis of trusting these climate models at all. He's saying, "Those models' predictions are crap." What's more, he tells anyone smart enough to grasp his point how to test whether his statement is correct. Ask the modelers to show how their models represent the "sub-scale processes" to which Theon refers. Compare that to how we know those processes actually work.
You're not going to see the eco-journalists ask to see that because they're either sufficiently propagandized or scientifically illiterate. Also because, in bald defiance of the scientific method, the IPCC climate models are somehow deemed "too important" to make available to outside scientists for independent verification.
However, the careful skeptic, even one ignorant of both the economy and the climate, should note that the best models available couldn't predict a chaotic system like the economy. But they continue to assume, as fundamental truth, that their models predicting the chaotic system of our climate cannot be wrong.
The deal here is... You can have very GOOD model, and still not accurately predict the future. That's not a bad model. That is a model showing little more than the limitations of modeling itself. A predictive model can only predict what it's designed to predict. And the designers, being less than omniscient, can't possibly build everything into the design. Therefore the models cannot account for reality. They can, at best, account for something like reality might become given certain conditions making certain assumptions. That's not the same thing as "reality."
The financial sector is learning this good and hard. When mankind's inability to predict the climate is realized we're going to have another hard realization... harder the further down the path of trusting the climate models we go.
Amid all the hand-wringing about financial systems in meltdown mode, the subject of modeling hasn’t gotten a lot of notice. Banks and other financial institutions employed legions of Ph.D. mathematicians and statistics specialists to model the risks those firms were assuming under a variety of scenarios. The point was to avoid taking on obligations that could put the company under.
Judging by the calamity we are now living through, one would have to say those models failed miserably. They did so despite the best efforts of numerous professionals, all highly paid and with a lot of intellectual horsepower, employed specifically to head off such catastrophes.
What went wrong with the modeling?
I want to talk about it because... aw heck... let me deliver the punchline first before anyone thinks I'm about to go on some kind of stimulus/bailout rant...
Interestingly enough, Groenendaal suggests skepticism is also in order for an equally controversial area of modeling: climate change.
“Climate change is similar to financial markets in that you can’t run experiments with it as you might when you are formulating theories in physics. That means your skepticism should go up,” he says.
Just to sum up for those of you who skim over multi-paragraph excerpts... Modeling - any modeling - is intrinsically error prone. You've got to be veeeery skeptical about the answer a model gives you, even when it's built by the very smartest people you know, given all the funding they ask for, and trusted with the most sensitive information.
And that's why the financial meltdown is instructive about anyone too certain about the models driving the current "climate change" panic.
We haven't actually observed "Global Warming", despite the alarmist reporting of every unpleasant weather event as "proof" of it. According to even the most alarmist models, the really bad effects of the warming are years, even decades, away. Climate change models are such that "you can’t run experiments with it," as the man said above. Lacking the opportunity for direct evidence, we must rely on models.
The problem is almost no one is recognizing some well known problems related to relying on models in any predictive sphere, and maybe the current problems in the financial sphere can wake a few people up in regard to that.
If you have never personally built a predictive model AND had to see its results compare to reality, you probably don't understand the intrinsic problem with modeling in general. At least not as deeply as you ought to.
As a classroom exercise, modeling is a blast. It makes you feel almost omniscient. You get to define the parameters. You know what you want the model to do. You set the rules. You define the data. You craft it until the thing virtually hums to the tune you play for it (as long as the "tune" can be translated into "data" anyway) It's beautiful.
In the real world we call this the "development environment."
I would imagine that if you live in academia this environment is more or less the primary place you live in and write about. Especially when you're modeling something you can't actually compare to any real world results.
My own experience is a little different. I'm used to testing models against the real world, where people intend to USE the model's predictions to make decisions affecting dollars, careers, and industries. That one extra step is... eye opening.
The first thing you learn is this: You can NEVER account for ALL the events that might impact the data in your model. That means you can NEVER be 100% certain that your model's predictions are correct.
Unless you've tried this, you probably don't believe me. All I can do is recommend that you try it and see for yourself. I don't know a single person who has actually done predictive modeling in the real world who would disagree.
The more complex the system you're modeling the less likely it is you have even considered all the events that might affect your prediction. If that sounds like an exaggeration to you, your inexperience is showing. As Donnie Rumsfeld once said, you've got your unknown unknowns out there.
Compare this basic truth about modeling to this criticism of the modeling work of Climate Alarmist in Chief James Hansen by former NASA scientist John Theon:
“My own belief concerning anthropogenic climate change is that the models do not realistically simulate the climate system because there are many very important sub-grid scale processes that the models either replicate poorly or completely omit.”
If you're someone unfamiliar with modeling, this comment might seem like nit-picking. If you have ever DONE modeling, you recognize this as a statement attacking the fundamental basis of trusting these climate models at all. He's saying, "Those models' predictions are crap." What's more, he tells anyone smart enough to grasp his point how to test whether his statement is correct. Ask the modelers to show how their models represent the "sub-scale processes" to which Theon refers. Compare that to how we know those processes actually work.
You're not going to see the eco-journalists ask to see that because they're either sufficiently propagandized or scientifically illiterate. Also because, in bald defiance of the scientific method, the IPCC climate models are somehow deemed "too important" to make available to outside scientists for independent verification.
However, the careful skeptic, even one ignorant of both the economy and the climate, should note that the best models available couldn't predict a chaotic system like the economy. But they continue to assume, as fundamental truth, that their models predicting the chaotic system of our climate cannot be wrong.
The deal here is... You can have very GOOD model, and still not accurately predict the future. That's not a bad model. That is a model showing little more than the limitations of modeling itself. A predictive model can only predict what it's designed to predict. And the designers, being less than omniscient, can't possibly build everything into the design. Therefore the models cannot account for reality. They can, at best, account for something like reality might become given certain conditions making certain assumptions. That's not the same thing as "reality."
The financial sector is learning this good and hard. When mankind's inability to predict the climate is realized we're going to have another hard realization... harder the further down the path of trusting the climate models we go.
Posted by Doug Williams on
Thursday February 19, 2009 at 11:25pm
The models I worked on could be easily tested where, as you say, long-range climate models cannot. But we have short-range climate models now, and they tend to be accurate for about as long as Granpaw was looking out the window. The Farmer's Almanac wasn't that smart, either. None of these models, to my knowledge, can even successfully predict the PAST, and correcting that with fudge factors, as some climate modellers have done, does not guarantee that they remain accurate for future predictions, because the corrections cannot be explained in terms of the physical science, meaning the underlying equations and assumptions are simply wrong.
We've got the evidence that manmade CO2 does not cause warming. We've got the evidence that the increasing CO2 of the last ten years is denied by the cooling of the last ten years. At its most fundamental level, these climate models are flawed even as scientific playthings. They certainly should have no place in promoting a catastrophic public policy.
AGW is religion masquerading as science. I'd compare it to Scientology, except that would be unfair to the Scientologists.