Cary Oler - Beitrag zu Nöten der Vorhersage

[Herwig, Rostock]

muss selbst erst mal lesen, nicht ganz ohne dictionary.

Name: Cary
Date: Fri, 9 Nov 2001 21:45:25 -0700
Subject: (Mike Branick) Re: Rating 'skill' in CME predictions

Take a deep breath... pull out your pillows... get comfortable (but not TOO comfortable)... and try not to sleep while I drone on...

Undoubtably many people may try to use the predictions that we issue as a means of testing our skill in producing them. But to do so without a knowledge of the limitations of the predictions would be unfair. Take heed of the note that we affix to each prediction:

"These predictions may be based on preliminary data and may be revised without warning. The predictions should not be used as a definitive indication of CME impact times or strengths and may frequently be in error. The proprietary methods used to estimate shock impact times are under continual development. Caution is advised."

Forecast accuracy is determined in-part by the quality of the data we receive. For example, what happens if a radio observatory such as Culgoora fails to accurately interpret a radio spectrogram indicating (for example) the presence of a Type II shock in the solar corona? What happens if they (for whatever reason) make a mistake in computing the estimated shock velocity of a Type II radio sweep (ex. if they mistakenly assume a 'noise' feature of the spectrogram represents the harmonic trace instead of the fundamental)? If they don't catch the problem, and if the forecaster has no reason to refute their claim (they often don't because radio observatories are supposed to have skilled observers), then the Type II shock velocity measurement will be taken into consideration when the prediction is produced. Right away, an error may be introduced into the prediction that has nothing to do with that particular forecasters' skill.

Similarly, what happens if the SOHO science team releases an estimated plane of sky CME velocity measurement that proves to be in error? It's not difficult for unintentional (instrument-related or perspective type limitational) errors to crop into the system considering we're looking at the Sun without stereoscopic vision.

So it is not only OUR skill that is put to the test, but the skill of ALL of the organizations that supply data that is used in the formation of predictions.

And then there are the inherent assumptions that must be made because there are no known ways (yet) to quantitatively determine specifics. For example, if a CME is directed Earthward, many assumptions have to be made that can affect the end-result prediction: what is the true "cone angle" of the CME? Is the velocity of that portion of the CME that is directed Earthward comparable to the plane of sky velocity? Is the expansion component of the CME constant, increasing, or decreasing? Will the plasma cloud be deflected toward or away from the Earth more than usual by external magnetic fields? Is the Type II shock velocity in error because the electron density profile of the corona is much different than what researchers have modelled it to be? It's impossible to determine answers to these types of questions quantitatively with the instrumentation and technology presently at our disposal. In most cases, proxies and statistical assumptions are used. But that's not good enough. Nevertheless, in many cases it's all we have at our disposal.

Finally, the methods used to determine impact times are still in their relative infancy. The worlds most elaborate models (ex. three dimensional magnetohydrodynamic models) capable of modelling stream-stream interactions and how multiple interacting shocks might modify each other are still too simplistic and course to accurately model what really happens in all cases. We have a long way to go.

What we are doing by providing these CME predictions is supplying a very rough, preliminary and probably somewhat error-prone prediction of CME impact times and strengths. I marvel that the techniques used are as accurate as they often are (and often are not).

My point is this: it isn't OUR (STDs) forecast skills alone that should be scrutinized by publicizing these values. Rather, it is the skills of a whole host of people in different organizations and in different fields that are being tested. Since the predictions are based on VERY PRELIMINARY (often "first-look" estimated) data, it should be understood that forecast accuracy will vary. Some may occassionally think that a monkey with a wet finger could better predict conditions. Sometimes that may be true. But most of the time, we should at least be in the ball park.

We received numerous e-mails after this large storm on 06 November. One thing that surprised me (but shouldn't have surprised me, I guess) is how many people thought that our original predicted time of arrival of 06:00 UTC on 06 November (impact was actually at 01:53 UTC) was WAY off because they saw the northern lights 4 hours earlier. They expected to begin seeing activity at exactly 06:00 UTC on 06 November!

Many terrestrial weather forecasters have a hard time predicting the arrival of a cold front with an accuracy of +/- 4 hours - and they have IN-SITU measurements and real-time indications of the progress of the front to work with!

On the other hand, we (as naive as we are) try to predict the arrival of a disturbance that is launched at a distance of 149 million kilometers and travels toward the Earth with an uncertain ever-changing velocity, an uncertain and often changing trajectory, an uncertain shock-front shape, that impacts uncertain obstacles along the way which could further distort the shape, velocity and strength of the disturbance, with one eye shut (no stereoscopic vision to help discern trajectory, proximity, velocity, etc), without the aid of in-situ measurements (except near the Earth - less than an hour before the disturbance impacts the Earth) and without a full understanding of the physics involved.

It's tough! Real tough! So, please keep in mind the caveat that is affixed to each prediction. Don't assume these predictions will always be right. They won't be. We hope that we (and other forecast organizations) won't be judged too harshly when predictions fail. It's perhaps something of a miracle that forecasters are able to do what they do, as well as they do it.

I definitely appreciate the feedback we have received concerning making the predictions available on the internet. There aren't many resources like it, and we are perhaps inviting people who don't understand the intricate nature of space weather forecasting to begin rating our 'skill' based on these predictions. I just hope those that do attempt to rate us are educated well enough to be able to separate 'skill' from 'errors due to uncertainty.'

[Ulrich Rieth]

...habs aber vergessen.
Ich würde nur mal gerne wissen, was für Leute dem STD böse Mails schreiben, nur weil der Sturm vom 06.11. nicht um 6 UT sondern schon um 2 UT angefangen hat.
Gruß

Ulrich

[Herwig, Rostock]

> ...habs aber vergessen.
> Ich würde nur mal gerne wissen, was für Leute dem STD böse Mails
> schreiben, nur weil der Sturm vom 06.11. nicht um 6 UT sondern
> schon um 2 UT angefangen hat.

...vielleicht phantasielose Konsumenten, die ein Polarlicht gemäß Fersehzeitung haben wollten. Immerhin haben wir diesen Nasen einen starken Beitrag von Cary zu verdanken.

Gruß Herwig

[Ulrich Rieth]

...aber wer ab und zu mal im IRC ist, der bekommt sowas auch häppchenweise und mit der Möglickeit der Nachfrage serviert
Gruß

Ulrich

[Lutz Schenk]

Einfach nur stark dieses posting.

Es werden wahrscheinlich auch noch so viele "disclaimers", also Warnungen vor den Ungenauigkeiten der Vorhersasge, nicht helfen:

Es wird immer wieder Leute geben die sich halt nicht ganz so intensiv mit der Materie beschäftigen, und diese Vorhersagen für zu wörtlich nehmen.

Ich finde diese CME-Prediction-Seite des STD schon sehr mutig und sehr gut (halt um die Schwierigkeit der Vorhersage-Parameter wissend).

Auch die Art und Weise wie sich Cary mit dem Thema auseinandersetzt, und die Dinge versucht rüberzubringen, meinen allergrößten Respekt.

Ich denke, das ich dieses Gesagte nicht so gut in englisch rüberbringen könnte.

Falls jemand Cary etwas dazu sagen möchte, kann er sich auch gerne auf mich berufen.

Gruß Lutz

[Ulrich Rieth]

...mit Dank aus dem deutschen Forum ge-e-mailt.
Ich denke, dass kann jeder hier unterstützen und daher war ich so frei, den Dank auf alle auszudehnen.
Gruß

Ulrich

[Herwig, Rostock]

Hallo,
eine Beitrag der Einfachheit halber noch:
Wetterfrosch-Antwort Mike Branick auf Cary Oler auch lesenswert.

Name: Mike Branick
Date: Sat, 10 Nov 2001 18:27:05 -0700
Subject: Predictions - Accuracy and Uncertainties

I hope everyone on this forum takes the time to thoroughly read Cary's excellent response to my earlier comment regarding verification/skill assessment in CME impact predictions. I wish there was a place to post it where it won't fall off the queue.

As an operational meteorologist for over 20 years, I think I can safely say that I know something about limitations and uncertainty when it comes to forecasting natural phenomena. So can anyone who's had to shovel a foot of "partly cloudy" off their driveway, or sat in a drenching cloudburst while hearing a forecast of "20 percent chance of rain," or stood in freezing cold despite hearing a forecast of "high near 80" (F)! Indeed, sometimes the cold front gets here faster than we thought it would!

So I reaffirm Cary's admonition regarding the note attached to each prediction, to the effect that there are many, many uncertainties inherent in the process of formulating the predictions. They're not going to be right all the time, and so no one should expect them to be.

Does that mean they're without value? Certainly not! They can be very useful! So use them, but as Cary said, use them with appropriate caution.

As a scientist, what I was trying to get at was a most-fundamental concept of the Scientific Method: The _Testable_ Hypothesis. In this case, the hypothesis is that certain aspects of CME impacts (arrival time, intensity, geomagnetic influences, etc) are predictable with some degree of skill (over, say, climatology or random chance). The hypothesis here is implied by the existence of such forecasts. If the hypothesis is indeed testable, then the forecasts must be _verifiable_ - that is, in a form such that they can be compared with observation to allow for assessment of skill. The hypothesis typically is tested by comparing skill of the forecast to skill of some "control" forecast, such as one based on climatology.

My point is that these predictions are good, solid science; skill _can_ be assessed in these forecasts. They are verifiable, therefore the hypothesis is testable, and so the whole thing falls beautifully within the rigors of the Scientific Method.

Now back to those uncertainties and limitations. They're out there in virtually any scientific discipline (some more than others) and as such must be dealt with in any effort to predict nature. Predictions thus must have a way to express the level of uncertainty due to the inherent unknowns (which, in the case of CME prediction, Cary described so well). In weather forecasts, that's why we usually give ranges of temperatures ("low to mid 20s", e.g.; a specific number like 24 doesn't allow us to reflect uncertainty at all) and probabilities of precipitation. That's why there is, and should continue to be, a time _range_ for forecast CME arrivals. Uncertainty can be expressed in the length of the range; narrower range equals less uncertainty.

Take next Sunday's Leonids. Will there be clear skies? I don't know. Too many uncertainties. That's a week from now; we have a hard enough time figuring out tomorrow. But ask me again next Saturday, and I might be able to say yes or no with high confidence. Or I might not. Or if I do, I might be wrong.

Those uncertainties will be there. They'll _always_ be there.

The meteor shower is an interesting case in itself. Current predictions from independent researchers on the Sunday morning peak range from 800/hr to 4200/hr. Peak times range from 0955 to 1032UTC. Can't all be right, but taken together with appropriate caveats, one can be reasonably sure that there will be a decent show somewhere between about 4 and 6 AM EST. If the clouds stay away...

The nature of humans is that we tend to want objective, black-and-white, yes-or-no answers. But the nature of science is that there usually aren't any.

Sorry about the length,
Mike B.

---------------------------------------------------------------------------