Predicting the future of a random process often relies on the assumption of stationarity, where statistics remain constant over time and frequencies never interact. When processes drift, classical tools like Wold decomposition and Kolmogorov-Wiener prediction encounter serious mathematical limits. Research by Marc Mehlman, a retired member of the Mathematics and Physics Department at the University of New Haven, explores the broader domain of harmonizable processes, where frequency coupling is permitted on an off-diagonal spectral grid. Drawing on M. M. Rao's dilation theorem, Mehlman generalised classical moving averages into fundamental moving average representations for multidimensional harmonizable systems. This framework yields explicit least-squares prediction formulas while uncovering crucial counterexamples that reveal how standard stationary assumptions break down in drifting environments.
Episode 6292 manual-script
Predicting Random Signals When Statistics Drift
A harmonizable process is mathematically identical to an orthogonal projection of a stationary process living in a larger space.
24:39 Transcript
Episode Details
- Episode ID
- MWP-6494
- Published
- Duration
- 24:39
- Audio
- Direct link
- Pipeline
- V5.22
- TTS Engine
-
chatterbox-regular - Script Writing Agent
- Claude Sonnet 5.5
- Topics
- mathematicsstatisticsphysics
AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.
The prompt
The prompt that started this episode, as sent in (production notes for the hosts removed).
Herman's latest fascination: the mathematical work of Marc Mehlman, retired University of New Haven professor, on moving average representations and least-squares prediction for multidimensional harmonizable processes, connected to the show's recent forecasting episodes.
Mentions
- Hughes Aircraft Defense contractor, image-processing target recognizer
- Journal of Multivariate Analysis Published Mehlman's 1992 prediction paper
- Korea University Mehlman's adjunct institute affiliation
- Lawrence Livermore National lab, early signal processing work
- M. M. Rao Probability theorist, Mehlman's PhD advisor
- Marc Mehlman Retired mathematician, harmonizable process researcher
- Stochastic Analysis and Applications Journal for 1991 harmonizable processes paper
- UC Riverside Where Mehlman earned his PhD
- University of New Haven Mehlman's former mathematics department
- University of Sydney His undergraduate alma mater in Australia
Downloads
Transcript (TXT)
Plain text transcript file
Episode Book (PDF)
The episode's record — date, duration, models, sources — with the full transcript
You Might Also Like
Never miss an episode
New episodes drop daily — subscribe on your favorite platform
New to the show? Start here#6292: Predicting Random Signals When Statistics Drift
Marc Mehlman is my newest fascination, and I'll admit up front that I only found him recently. We've done a lot of forecasting on this show lately. The final odds on Israel's vote, the Part Four forecast, and further back, how weather prediction became a science. Somewhere in the middle of all that I started wondering about the mathematics underneath. How do you predict a process from its own past when its statistics won't sit still? Every method in my head assumed they do. A trail of citations led me to a mathematician in Connecticut.
A retired mathematician in Connecticut. Herman, nobody finds those by accident. I'm only asking whether "retired" is the cover story.
His own homepage calls him a retired member of the Mathematics and Physics Department at the University of New Haven.
That is what it would say.
I'm about to talk about covariance, Corn, and you're going to let me.
Slowly. It's what I'm best at.
His output is small and it's all one subject. Four research papers between nineteen ninety-one and two thousand four, every one of them about a family of random processes called harmonizable processes, and about predicting them. He did his PhD at UC Riverside in nineteen ninety under M. M. Rao, who is the central figure in that corner of probability. In the footnote of his two thousand four chapter, which is in a volume honoring Rao, he writes that one could not ask for a better mentor. I like a person who says that in a footnote.
Four papers, all on one subject. That's focus.
Four papers where each one answers a question the last one opened. I'd take that over forty. But to see what he did, we need the classical picture first, because he's rebuilding it on looser ground.
Go on. Start with the ground that's there.
A stationary process is one whose statistics don't change with time. The mean is the same, and the relationship between now and one second ago is the same whether you check at noon or at midnight. Picture the steady hum of a refrigerator. Corn, is that stationary?
Until the compressor kicks in. Then it's a refrigerator with opinions.
Hold onto that, it's the problem. For the idealised steady hum, though, there's a beautiful theory, and it begins with Wold. His decomposition says that any stationary process splits cleanly into two parts. One part is perfectly predictable from its own past. The other is built entirely out of what mathematicians call innovations.
Innovations sounds like a startup.
It's the new information that arrives at each moment, the part of today's value you could not have known from everything before it. Surprise, basically, in units you can add up. The surprise at each step is a fresh piece of randomness, and the claim is that the whole unpredictable half of the process is just those surprises, weighted and stacked.
Weighted how?
Each moment's value is a sum. Today's surprise gets a weight, yesterday's surprise gets a weight, the day before gets another, going back as far as the process has a past. That's a moving average. Not the stock chart kind where you smooth the last thirty days. It's a recipe that says the signal is the past surprises, mixed with fixed coefficients.
So the signal is a memory of everything that startled it.
Close enough. Now the word that matters for Mehlman. A fundamental moving average is one where the surprises you use are exactly the innovations. Not extra noise, not a bigger collection that happens to contain them. Precisely the new information, no more and no less. And Rozanov showed that a stationary process has one of those if and only if it's what's called purely nondeterministic, meaning there's no part of it you could predict perfectly from the past.
Fine. And why do we care that it's fundamental?
Because that's what makes prediction easy. This is the Kolmogorov and Wiener story. You want the best guess for a value some steps ahead, best in the sense of the smallest average squared error. Write the process as its fundamental moving average. The future surprises haven't happened, you can't know them, and their average is nothing. So you keep the sum, throw away every term that involves a future surprise, and what's left is the best predictor. You've literally cut off the part of the world that hasn't happened yet.
It's a tidy trick. Cut off what you don't know and call the rest a forecast.
It's optimal, though. That's what makes it more than a trick.
Hm. So where does it break?
When the statistics move. Real signals drift. The loudness of the hum changes slowly, or a market's volatility swells before an event, or a weather system's character shifts with the season. Strict stationarity is a very strong demand to make of the world. And here's the part I find lovely. In the frequency picture, a stationary process is one where different frequencies never interact. Each frequency carries its own energy and has nothing to say to its neighbors. Mathematically, all the spectral mass sits on a diagonal.
A diagonal of what?
Imagine a grid with frequency on both axes. The energy at one frequency paired with itself sits on the diagonal. Energy that links one frequency with a different one sits off it. Stationary means the off-diagonal is empty. Harmonizable, which is Loève's idea, means the off-diagonal is allowed to be nonempty. Frequencies can talk to each other, but in a controlled way.
Give me the audio version. I'm a sloth. I need to hear it.
Take a pure steady tone and slowly turn the volume up and down. Swelling and fading. That's not stationary, because the loudness depends on when you look. Spectrally, the swelling creates sidebands, little copies of the tone shifted slightly up and down in frequency, and those sidebands are correlated with the original. The frequencies are coupled. That's the off-diagonal. Harmonizable processes are the class where that kind of coupling is organised enough to still do mathematics on.
Is there a trap? Whenever someone says "controlled", I look for the trap.
There's a split, and Mehlman is careful about it. In his own notation, the covariance of the process is built from a function that lives on that grid. If that function is a proper measure, the process is strongly harmonizable. If it's only what's called a bimeasure, it's weakly harmonizable. Stationary is the special case where everything sits on the diagonal. Strong is the better-behaved class, and a lot of his sharpest results are for strong.
So weak is the wilder one.
Yes. Strong implies weak, so everything strongly harmonizable is also weakly harmonizable, and then you get the question of what survives in each.
Let me ask the thing I actually want to know. Is any of this a trick for faking stationarity, or is it a different kind of object?
I think the answer is both. Rao has a dilation theorem that Mehlman leans on heavily. A process is harmonizable exactly when it's the orthogonal projection of a stationary process living in a bigger space. A shadow, if you like. The drifting thing you can see is a shadow cast by a perfectly well-behaved stationary thing you can't see directly. So it's a different object, but one with a stationary parent.
What do you think, Herman, is that comforting or alarming?
Comforting to me. Alarming for the stationary parent, which has to be carried around in a bigger space. But I'm biased.
You are. You've been carrying around a bigger space since you got here.
Fair. The first is from nineteen ninety-one, in Stochastic Analysis and Applications, titled Structure and Moving Average Representation for Multidimensional Strongly Harmonizable Processes. Multidimensional means the process is a vector, several signals at once, say several microphones, rather than one. The abstract says the classical moving average representation gets generalised, with necessary and sufficient conditions on the covariance, and that the structure theory is given in considerable detail. Inside, per the abstract, there's a new proof that time and frequency mirror each other as Hilbert spaces, and a family he calls splitting processes, whose covariance factors into two matching halves. Those are building blocks. He discusses how densely they sit among the harmonizable ones. I'm going on the abstract there, I haven't worked through the details.
Splitting processes. It sounds like an ice dance.
Doesn't it. Now the career line, because it's interesting. Before the PhD he was working with computers and signal processing. Fast Fourier transforms and space-scale filtering at Lawrence Livermore from nineteen eighty-one, a spell at Hughes Aircraft designing and analysing an image-processing experiment for a target recognizer, and he also appears on a paper about simulating shock waves.
Livermore. Then Hughes Aircraft. A target recognizer. Then he retires to Connecticut and writes about harmonizable processes.
That's a mathematician's career.
It's also exactly what a person would put on a résumé if they wanted me to stop asking. And his site has travel journals, doesn't it?
It does. An index, anyway. I haven't read the text of any of them.
Go on, read me the index.
Fine. Far East in nineteen ninety-nine, China, Singapore, Australia, New Zealand. Spain, Portugal and France in two thousand.
Nineteen ninety-nine. China, Singapore, Australia, New Zealand in one trip. That's not a vacation, that's a route.
It's a man who enjoys travelling and wrote it up for his friends. Back to the second paper, which is the one I find most beautiful.
You have a tell, Herman. You change the subject by saying "beautiful".
It worked, didn't it? The second paper is from nineteen ninety-two, in the Journal of Multivariate Analysis, Prediction and Fundamental Moving Averages for Discrete Multidimensional Harmonizable Processes. The abstract from his own preprint says the moving average representations of discrete multidimensional stationary processes get generalised to fundamental moving average representations of weakly harmonizable processes. For strongly harmonizable processes he obtains necessary and sufficient conditions on the covariance for such a representation to exist, and uses them to get least squares prediction formulae.
That's a lot of words to say "the Wold trick still works".
Sometimes. That's the interesting thing. It sort of works, with fine print, and the fine print is the paper. Start with his definition, which is the one that matters. A fundamental moving average of a process at time t is a sum over all earlier times j, up to t, of a coefficient times a random variable xi sub j, where those xi's span exactly the innovation space at time j. The innovation space is the new information arriving at that moment. Only purely nondeterministic processes can have one. And then the prediction formula looks just like the classical one. The best predictor some steps ahead keeps the same sum, with shifted coefficients, over the innovations already seen.
The same cut-off-the-future move.
The same move. And he shows that every fundamental moving average is weakly harmonizable, and that its predictor is strongly harmonizable if the moving average is. So the class is closed under predicting, which you'd want. The predictor doesn't escape the family it came from.
Okay, but you said fine print. I want the part where it breaks. That's usually where the person is clever.
That's his counterexamples, and I think they're the story. They show things that are automatic for stationary processes and quietly false for harmonizable ones. First, in a stationary process the amount of genuinely new information per step is the same at every moment. Call it the dimension of the innovation space. For harmonizable processes it needn't be. His example is almost insultingly simple. Take ordinary white noise, independent surprises, and just set the value at time zero to zero. That process is strongly harmonizable, and at time zero there are no innovations at all. Zero-dimensional surprise, in the middle of a process that is otherwise full of it.
He zeroed one point and broke the theory.
He zeroed one point and showed the theory has to be stated more carefully than the stationary one did. The time-invariance you'd been quietly relying on was never free.
I like a counterexample you can say in one breath.
The second one is subtler. Take a purely nondeterministic stationary process, vector-valued, and project it onto one direction. You'd expect that to stay unpredictable. But the projection can be a harmonizable process that is not purely nondeterministic. Part of it becomes predictable.
Looking at less of the signal made it easier to forecast.
Which is odd, until you remember the shadow picture. Shadows can lose information in surprising places.
Herman, which of these would have caught you out? Would you have guessed that projection keeps the unpredictability?
Embarrassingly, yes. I'd have said projecting can only add predictability if it throws away independent pieces, and it turns out a projection can leave behind something with structure. I walked in assuming the stationary intuition carried over. That's what the examples are for.
And the third?
The third is the one I'd put on a poster. In the stationary theory there's a test due to Kolmogorov and Szegő. Roughly, you look at the spectral density, take its logarithm, and ask whether that integrates to something finite. If it does, the process has genuine room for surprise. It isn't perfectly predictable. For harmonizable processes, per his preprint, that test doesn't carry over. His example is a process built from a single random variable Y, where the value at time t is just a fixed coefficient depending on t, times Y.
One coin flip, rescaled forever.
One coin flip, rescaled forever. The moment you've seen Y, you know the entire future. The process is completely deterministic. And yet the log-integral condition holds, so the stationary test would have told you there's room for surprise. The old test says the future is open, and it's closed.
So the stationary checklist is a decent guess, and he shows exactly where it stops being one.
Not even a guess, a checklist whose boxes don't all mean the same thing off the diagonal. He also proves a positive result that holds things together: every purely nondeterministic harmonizable process has a purely nondeterministic stationary dilation. So the shadow of an unpredictable thing comes from an unpredictable thing. The bridge to the stationary world survives, even when the intuitions don't.
I notice he's careful to show what's lost and what's kept. A lot of people just announce the generalisation.
He says in that paper, and I'm paraphrasing, that for strongly harmonizable processes there's as yet no known method of getting the standard Cramér form straight from the covariance. That's an honest sentence to put in a paper. It tells you where the edge is.
Does anyone use this? Aircraft, radar, the target recognizer?
His abstracts talk about prediction and filtering. I haven't seen a specific deployment confirmed, so I'll just say the setting is the right one for systems whose statistics drift. A loudness that wanders, a seasonal swell. That's the kind of thing this class is built for.
Before the next paper, we have a guest. A specialist in these matters, Tormund, holder of the honorary chair in Applied Stochastic Divination at the Institute for Advanced Moss Studies, and author of the bestselling Mean Squared Mysteries. So, Tormund, as our guest expert, what can you tell us about Mehlman's moving average representations?
Ah, Marc H. Mehlman. I have not read his work, but the prophecy speaks of him. It says that in the year of our Lord two thousand and one, a man would arise who would smooth the jagged waters of the stochastic sea.
His first paper was nineteen ninety-one, Tormund, and it was about the moving average representation itself. The sum of past innovations. Can you walk us through it?
The moving average representation, you ask? It is foretold that one must take the household object known as the colander and hold it aloft during a full moon. That is the ritual. Anything that goes wrong is the moon's doing, mark my words.
Mm. Least squares prediction?
Least squares prediction? The prophecy is clear. You must whisper the numbers to a toad at midnight. I am a toad, and I am also a wizard, so I should know. But I cannot tell you the numbers. That is the mystery.
Perhaps the multidimensional harmonizable case, then, where there are several signals at once?
Multidimensional harmonizable processes? Ah, yes. The prophecy mentions them only in passing, something about seven veils and a very confused donkey. I am afraid I must leave you now. I have another booking in a swamp.
He was very sure of the toad part.
The two thousand two chapter, in a volume edited by Balakrishnan, is called Some Properties of Harmonizable Processes. Per its abstract, increments of harmonizable processes are harmonizable, and so are derivatives and definite integrals. That matters, because calculus on a class of processes is useless if differentiating takes you out of the class. He also introduces the nth moment of a vector measure, and suggests it could link harmonizable processes to their vector measures the way characteristic functions link ordinary random variables to their probability measures.
Which means what, to a sloth?
A fingerprint you can compute with, instead of reasoning about the bare measure. He's suggesting it, to be clear, not claiming the theory is finished. Then the two thousand four chapter, in the volume honoring Rao, takes up the full question: when does a purely nondeterministic multidimensional harmonizable process have a fundamental moving average representation at all?
A decade on one question.
Thirteen years, counting from the first paper. A long conversation with one problem. I think there's a dignity in that.
For a man who retired, he stays very consistent. He gave talks too?
Mathematics meetings in Tampa in nineteen ninety-one and DeKalb in nineteen ninety-three, invited sessions of the International Indian Statistical Association in Hamilton, Ontario and later DeKalb again, a colloquium at UConn in two thousand four, and a colloquium at the University of Sydney, which his resume puts on the twenty-first of May two thousand nine, though the Sydney page says the twenty-second.
Sydney, two thousand nine. China the same year. Taiwan the year before. Korea in two thousand ten, after Korea University in two thousand seven.
Adjunct at Korea University's Institute of International Education, yes. Japan and Korea in two thousand seven.
Thailand and Cambodia, two thousand eleven. Bangkok, Chiang Mai, Siem Reap. Israel in twenty nineteen, and again in twenty twenty-three.
You've memorised the index.
Somebody had to. If I were running a quiet network of mathematicians who occasionally need to be in Sydney, this is what the file would look like.
The file says a man who likes airports and writes up his trips. I'll add that the two Israel entries are the ones that land closest to home for us, and I'm happy to read them as a man who enjoys the place.
Noted. I'm writing nothing down.
Corn, you've watched me fall for every elegant mechanism I've ever met. Which would you rather forecast with? Stationary, where the rules never change, or harmonizable, where the rules drift but the drift has a shadow?
The second. The first is a story about a world nobody lives in. The second at least admits the volume knob exists. My complaint is that it's harder to say out loud.
It is harder. I'm still working out how to say the two thousand four conditions at all. That chapter runs twenty-odd pages, and nearly every line is a condition on the covariance.
Twelve. Two sixty-five to two seventy-six. I have eleven days of Mr Pennywhistle's kitchen on a clipboard, which is awkward with claws. He says his refrigerator hums one steady note and always has. He pays me in jars. Pickled walnuts. Forty-one so far. It is louder after eight in the evening, Herman. Every night. He says that is the neighbour. Tuesday he says does not count, because he was out and the fridge was lonely.
A stopwatch on the compressor would give you the on and off timing, even without frequency data. Tuesday's gap would show up in it.
It shows. Fourteen minutes on, six off, most nights. The seventh night it was nineteen on. I priced a used spectrum analyser at three hundred and ten dollars and walked past it. You kept saying shadow earlier, by the way. I counted four.
Nineteen on is still a drift, whatever the neighbour does.
He says it is the fridge thinking. I will be back Thursday with the same pencil.
If the fourteen and six hold on the other evenings, you've got a baseline, and the seventh night becomes something you can actually talk about.
Saturday. He cancelled Thursday.
What I take from Mehlman is that the careful version of a result is worth more than the broad one. I came to him wanting a way to predict a drifting process from its past, and what I got is a map of which parts of the classical machinery survive the drift and which quietly don't. The fundamental moving average survives, with a tighter definition. The recipe of cutting off future surprises survives. The tests and the dimension counts need rechecking every time.
What stays with me is how many things we call steady that nobody has watched for eleven days. And the shadow idea, that something messy might have a tidy parent we haven't met.
The weather series is where I'd take that next.
Thanks to our producer, Hilbert Flumingtop, for keeping the desk running. This has been My Weird Prompts. If you want the archive, it's at my weird prompts dot com.
Thanks for listening. We'll be back soon, and I'm sure Mehlman really is just retired.
Mm. That's what the file would say.
This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities. It may contain errors: disclaimer · report an error