I had had recent phone and email conversations with senior economists at the New York Fed and Atlanta Fed regarding their GDP forecasts.
The Atlanta Fed calls its model GDPNow and the New York Fed calls its model the FRBNY Staff Nowcast (simply referred to as “Nowcast” below).
The Fed offices are not in competition with each other, although quarter-to-quarter “bragging rights” may be in order. That is my subjective hypothesis, not based on any economist’s statement.
Let’s take a look at the theory as well as possible flaws in the models.
I know most about the New York Fed model, so let’s start there. Some of the discussion is mathematical, but I will explain in easy to understand terms.
Nowcast is a “single factor model” based on Kalman Common Filtering Techniques.
The explanation is easier than the math.
- The model starts from a presumed known point such as the last GDP estimate, then takes into consideration new data as it comes in.
- The forecast is derived on the basis of past correlations, continually revised by incoming data. Current data is more important than prior data.
- The difference between the model forecast and incoming new data determines whether or not estimate rises and falls.
The “single factor” is the latent state of the economy, a pervasive source of common fluctuations in all the data that enter the model. Think of it as a business cycle factor that summarizes the state of the system. The modeling approach is based on the insight that business cycle fluctuations are pervasive and hence there is a lot of co-movement in the economy.
The model does not attempt to mimic the BEA. Rather, the model attempts to predict what the BEA will report based on incoming data.
In short, one data point affects estimates of related data points that follow.
Model Explanations and Examples
- The BEA GDP report does not take into consideration any regional manufacturing reports, jobs, or any of the factors in the model. In contrast, and based on past experience, there is an established relationship between regional Fed reports (such as the Philly Fed manufacturing report) and the next GDP report. This relationship goes into the Nowcast model.
- In advance, the model predicts the next result of the Philly Fed report (based on all the previous data points, not just recent Philly Fed reports). For example, a change in the Empire State manufacturing report may change the previous assumption about the Philly Fed manufacturing report.
- One cannot assume a good data report will necessarily cause the model forecast to rise. A recent example of this is construction spending. A rise in construction spending may cause the model estimate for housing starts to go up. If the actual housing starts report is less than the model predicted, the GDP model estimate will drop.
- Thus, a seemingly good report might lead to a lower forecast if the model predicted a better outcome. The converse is true as well. A seemingly bad data report could actually add to the GDP model estimate.
The BEA states its accuracy in the range of +-1 standard deviation, approximately 1 percentage point in GDP.
My Nowcast economist contact made this statement “We will release the estimates of uncertainty based on the historical errors of the model in real-time.”
Patrick Higgins writes …
The first version of GDPNow was completed in 2011q3 and we’ve been maintaining real-time forecasts of the model since then. The Forecast Errors for the model’s final forecasts over the 2011q3 – 2016q1 period are available in the tab “TrackRecord” of the Excel sheet. Over this period, the average absolute forecast error of the annualized real GDP growth rate was 0.61 percentage points and the root mean square error was 0.83 percentage points. An analysis of the forecast errors and a comparison of the model’s performance with professional forecasters is available in the May 16, 2016 Macroblog GDPNow and Then and also the May 23, 2016 Macroblog Can Two Wrongs Make a Right? The tab “TrackRecord” has a chart showing an estimate of the historical average absolute forecast error as a function of how far away the advance release data is. The spreadsheet linked above also has historical forecasts of the model if you ever want to do your own analysis. See the tab “ReadMe” for a description of what tabs contain what data.
That table reflects the Nowcast as of June 3. I highlighted the key changes from the May 27 report.
For the month, I count 35 items in a grouping of 8 general buckets. The two regional manufacturing reports are the Philly Fed and New York Fed regions.
Why just those two regional reports? Why not Dallas, Kansas City, Richmond, etc.? Where does one stop? Why not other variables?
The answer is complexity.
Forecasting GDP with a Dynamic Factor Model
Please consider a snip from Forecasting GDP with a Dynamic Factor Model, an article by the Spanish Ministry of Economy and Finance, on how Spain derives its GDP forecasts.
At the Ministry of Economy and Finance we have developed a dynamic factor model to estimate and forecast the rate of growth of the Spanish economy in the very short term. This model uses a coincident indicator, or estimated common factor, to forecast GDP by means of a transfer function. The model estimates a common factor underlying 31 economic indicators spanning domestic production, the labor market, and domestic trade flows. It enables us to forecast GDP several times a week, providing a virtually real-time complement to the four quarterly GDP official reports issued each year.
With 31 indicators, our model avoids the disadvantages inherent in both larger and smaller models. Models with more than 80 indicators are difficult to interpret, require more work to maintain, and suffer from group effects that can distort the estimation of the common factor. On the other hand, models with fewer than 12 indicators can lack sufficient coverage, omit useful information from excluded indicators, and be more susceptible to the abnormal behavior of a single indicator.
The indicators we use cover a wide range of domestic economic activity, including industrial production, air traffic, cement usage, tourism, energy consumption, transport by railway, housing starts, and employee compensation. We supplement these domestic indicators with external ones, such as imports and exports of goods and services, as well as social security contributors (employment) and other labor-market indicators.
In addition, our model includes qualitative indicators reflecting the level of confidence Spaniards have in their business and household finances. Such indicators are less common in dynamic factor models because they are difficult to integrate with hard data. These so-called soft indicators, however, provide a useful alternative perspective on the economy, and they are promptly available, making them valuable for short-term forecasting.
Atlanta Fed economist Patrick Higgins explains GDPNow: A Model for GDP “Nowcasting”.
The GDPNow model forecasts GDP growth by aggregating 13 subcomponents that make up GDP with the chain-weighting methodology used by the U.S. Bureau of Economic Analysis.
Using real-time data since the second-half of 2011, GDPNow model forecasts are found to be only slightly inferior to consensus near-term GDP forecasts from Blue Chip Economic Indicators. The forecast error variance of GDP growth for each of the GDPNow model, Blue Chip, and the Federal Reserve staff’s Green Book is decomposed as the sum of the forecast error covariances for the contributions to growth of the subcomponents of GDP. The decompositions show that “net exports” and “change in private inventories” are particularly difficult subcomponents to nowcast.
The advantage of GDPNow over Blue Chip is the GDPNow forecast is continual, following key economic reports, not twice a month as with the Blue Chip estimates.
One can download the Full Text of the GDPNow Model but I caution the document might only be understood by a mathematician.
GDPNow Construction Spending
On May 26, I noted GDPNow Bounces to 2.9% Following Durable Goods Report.
There was nothing remarkable in the report at first glance. GDP rose almost entirely due to the durable goods report. However, one day later I reported New York Fed 2nd Quarter “Nowcast” Rises to 2.2% Following Housing, Durable Goods Reports
Housing, durable goods, and revisions added 0.254, 0.099, and 0.074 percentage points to the New York Fed model forecast.
On May 24, there was a Massive Jump in New Home Sales but that did not factor into the GDPNow model at all.
Higgins explained in an email.
Both new and existing single-family home sales are ingredients used to estimate brokerage commissions, which comprise 23% of residential investment in nominal terms. Since new-single family home sales are only about 1/10th the size of existing-single family home sales; the latter are generally more important for this subcomponent.
Is that a big model difference between GDPNow and Nowcast?
What Can Go Wrong?
The answer is plenty. GDP is heavily revised. So are construction spending, durable goods, housing, and jobs reports.
I also question at least one Nowcast model assumption as noted in New York Fed Nowcast Up to 2.4% (I’ll Take “The Under”); Modeling Error on Unemployment Rate?
Positive and Negative Factors
The biggest positive factors since the May 27 report were personal consumption expenditures, the manufacturing ISM, and the unemployment rate.
The largest negative factor since the May 27 report is construction.
Modeling Error on Unemployment Rate?
I suspect the New York Fed has a small modeling error in the civilian unemployment rate.
In general, it’s logical to presume a decline in unemployment likely is a positive factor for growth, but I also propose one needs to look at why the rate declined.
Today, the unemployment rate dropped because close to half a million people dropped out of the labor force.
Moreover, there was a massive jump in involuntary part-time work of 468,000. Neither of those can remotely be considered positive for spending or GDP.
Take a look at the top line in the above Nowcast detail. It’s on JOLTS (Job Openings and Labor Turnover). It added 0.04 percentage points to the Nowcast model.
I believe job openings data should have subtracted from the model.
I discussed why in BLS Says Jobs Openings Up; Actually, Openings Falling Fast!
First, let’s consider what the BLS says, then let’s look at an alternative viewpoint.
BLS: Job Openings vs. Hires
Job Openings – Real Time Macroeconomics
Jon Hartley, Researcher and Policy Analyst for Real Time Macroeconomics sees things this way (anecdotes Mish).
Clearly someone is wrong. Who is it?
If I am correct on both counts, the Nowcast model should have subtracted instead of added for both the unemployment rate and job openings.
GDPNow does not consider either the unemployment rate or job openings, but it does include some related variables.
Higgins writes …
The civilian unemployment rate is not a direct input into the model, but GDPNow does include some variables related to the unemployment rate in its dynamic factor model.
- Civilian Employment: Nonagricultural Industries: 16 yr + (SA, Thous)
- Civilian Employment: Sixteen Years & Over (SA, Thous)
- Civilian Unemployment Rate: Men, 25-54 Years (SA, %)
- Median Duration of Unemployment (SA, Weeks)
But you’re right; the civilian unemployment rate is not a direct input into the model.
Employment and unemployment are related in ways, but unemployment can rise of fall independent of employment.
Unemployment is a factor of labor force participation compared to employment, not employment directly.
A decline in unemployment because people dropped out of the labor force is a lot different than a decline in unemployment because more people are working.
In the last jobs report, the civilian Labor Force declined by 468,000. Household survey employment rose by only 26,000. Last month, household employment declined by 316,000 but the unemployment rate was steady.
If there is another bad jobs report, it’s quite likely there is a significant trend change. Regardless, I suggest the Nowcast model needs to incorporate “why” the unemployment rate went up or down.
Second Quarter GDP Estimates
- New York Fed Nowcast June 3: 2.4% New York Fed Nowcast Up to 2.4% (I’ll Take “The Under”); Modeling Error on Unemployment Rate?
- Atlanta Fed GDPNow June 14: 2.8% GDPNow Forecast Rises to 2.8% Following Retail Sales Report
- Markit June 3: 0.7% to 0.8% Composite PMI Flirts With Contraction; Markit Chief Economist Estimates GDP 0.7-0.8%
- ISM June 3: 1.6% Non-Manufacturing ISM Much Weaker Than Expected
GDP models are only as good as the data going into them. In general, it’s highly likely that small errors even out in the GDPNow and Nowcast models.
However, at economic turning points, models forecasts are likely to be way off. Are we at a turning point?
Here’s two likely clues, and something that none of the above models considers.
Percent Change in Temporary Help Services Employment
US Federal Personal Witholding Tax Receipts
For tax survey details, please see US Tax Receipts Signaling Recession?
If we are at an economic turning point, the model forecasts are likely significantly wrong, and downward revisions will follow.
Everything above has been reviewed by the New York and Atlanta Fed economists. They did not comment on my estimation of modeling errors or turnings points, only their models.
I asked each of them one final question: “How far back does back-testing of your model go?”
I received an answer from Pat Higgins at GDPNow: “The forecast errors are constructed by comparing with the BEA’s as reported first-release GDP data. Using revised current-vintage data, we constructed forecasts back to 2000q1. These data are described in rows 28-31 of the ReadMe Tab of the Excel spreadsheet GDP Tracking Model Data and the historical forecasts can be found in the linked tabs in column A.”
The New York Fed economist was not available to comment on the short notice I gave him.
Back-testing does bring up another issue in regards to modeling. Each recession will be different. We have already seen manufacturing data so bad that in every historical precedent, the economy was long in recession. There are normally yield curve inversions prior to recessions. I am confident we will not see one this time. Might there be a “job gain recession, or one in which minimal jobs are lost?” Sure, why not?
None of this detracts from the respective models. There is no such thing as a perfect model. I hope my comments go into making a better model, and I thank both the Atlanta Fed and New York Fed economists to taking the time to answer my questions.
Mike “Mish” Shedlock
Why in the hell are you looking at this BS from the Fed. This is all a game cooked up by these Zionists to provide a smokescreen and keep the country under their control. You even addressing it is a shame.
Old Guy said:
Good read Mish,
Tapp at some point in time you have to examine the numbers yourself and come to a conclusion. Absolutely agree numbers can be used to get the outcome of the stats. Mish points that out with some of his points about the formulas used. Someone has to look at numbers Tapp and I appreciate that Mish does.
It would not matter who the sitting president is repub or dem they both do the same thing to make the sitting president look good..
No, it’s more about making a SWAG look like documented research. By SWAG I mean scientific-like guesswork via the use of math to make it look precise.
Look, if I worked as a Fed governor, I would relent and take the best guesswork I could find since there is no such thing as actual scientific forensic fortune telling regarding the state of the economy given any interest rate policy … unless it’s based on tortured logic .. in which case it’s probably devoted to justifying low rates and QE or doing nothing. If nothing else, I would be looking busy and productive and so would the former math econ majors putting the models together. If they were right, I, as a Fed Governor, would look good. If they were wrong, which is the likely result, I could blame them or society at large for being a mess that ruins out ability to do the job we were hired to do.
If Fed Bank A says ‘This way’ and Fed Bank B says ‘No, this way’ then this is implicit proof that neither knows anything more than how to make a guess that look scientific. Since these people tell the FOMC how to evaluate the future then, by inference, they are all just tossing Hail Mary’s and hoping for the best … at best. In reality, the FOMC is just doing what their profiler’s put into their heads. They are sheep who listen to whispers.
These numbers are only as good as the collection process. Thus instead of being in such a hurry to publish numbers they should simply wait a while longer and get the correct information instead of constantly revising them after the fact. Imagine if your doctor gave you a prognosis and then more results come in later and he changed it. Before long you would not believe a thing he said.
As a retired State Budget Director, a dramatic down turn in income tax withholding was always a sign that it was going to be a challenging two or three years. The chart you show made me age very quickly during the 2000 to 2010 period. Good luck to current practioners of the Budget Game.
Retirement is good!
Jon Sellers said:
The economy is purely the exchange of goods. Sales tax collections should be a good indicator of the current state of the economy. One would think changes in individual income tax collections should forecast changes in sales taxes. And one would think the rate of decline of income tax collections over some short period of time ought to be a good indicator of a looming recession.
However, I’d assume the timeliness of how individual states monitor and report their collections may make that data an unreliable predictor.
If prices halve but activity stays the same it will look negative, if prices increase by double but activity halves all will read as well . The game of state is to try to maintain its level of tax revenue, due to debt amd commitments. In effect it sets about rigging and taking over the economy to suit its own model, or those it indirectly sponsors and is sponsored by.
‘If you are lucky there is a place for you with us’ should be its motto.
I wonder how much the constantly increasing sales tax rates and schemes for new taxes affect the historical tax collection comparisons. I have not noticed that those factors are taken into account.
A nice exercise in mental gymnastics and I hear that the Fed has 750 PhD’s on staff to do such work. Come in by 9, break for lunch, home by 5:30. Newspaper, drink, dinner, bed. Maybe a quick review of the day’s market news. Salary probably in the $150 range, nice benefit package and defined benefit retirement plan. No worries about job security, since there are always new numbers to crunch. And you can bunker down for your 30 years then hit the links. Maybe write a book. Take the wife on that cruise or buy an RV.
For those of us out here in real investment land who have to face the millions (trillions?) of decisions against or for our positions every day, these forecasts are somewhat meaningless, since they are not accompanied by a set of if/when investment instructions.
No, we focus on the only thing that matters. Price.
Just make the money meaningless and GDP growth will almost always be positive. The only exception I can think of is 1973-1974.
Not a single economist saw that recession coming that I am aware of.
Equity prices must stay high and bid or interest rates will continue to soar not just for private borrowers (as they already have) but for the public sector as well.
There is a time limit for how long these equity prices can stay bid though…I think the Fed is out of bullets and now what happened to Detroit and now Puerto Rico will spread like wildfire.
US Treasuries are the short of the Century. Once that debt sells off any all debt globally will be seen as worthless leaving only gold as your capital and silver as your currency.
Jon Sellers said:
I don’t know. Money doesn’t disappear except in bankruptcy. Money just flows around to different folks with different interests. The U.S. government can’t go bankrupt and can always meet its obligations. So I tend to think when things go bad treasuries will just go higher. It is the one place you can guarantee the return of capital.
Tony Bennett said:
“US Treasuries are the short of the Century. Once that debt sells off any all debt globally will be seen as worthless leaving only gold as your capital and silver as your currency.”
Haha. Been hearing that for years. One day … far into the future you may be right.
But for now, deflation on tap and Treasuries will do VERY well.
If the model is based on domestic data mining back to Y2K, it won’t contain data on how printing leads to banana republics in the long run.
Excellent discussion of modeling. I spent my career in the field of air and ship launched tactical guided missiles, and modeling was a critical tool in developing an effective self defense weapon. The key issues in modeling are accuracy of the model and completeness of the model.
The model is used to simulate thousands of missile launches over a multitude of geometries and electronic countermeasure conditions, and then to measure if the missile would destroy the target. A number of missile launches are performed to measure the actual flight of the missile versus the model and the model is then adjusted to validate the model. The model is then used to predict the results of a real launch before the actual launch, which is similar to the Fed’s models striving to predict what the economy measurement will be when released.
The somewhat insurmountable problem with the Fed model is the complexity of the system they are trying to model, which to be accurate is modeling the US and to a degree the global economy.
Model inaccuracies that dominate the problem are the interactions within the system and whether or not the model contains all the key subsystems that materially affect the results. Over the many years in the missile industry, we learned that it was often what was not in the model that messed up the model, or the detail of the model of a part of the system was too crude.
You discussed the use of a Kalman filter, which is a fairly sophisticated mathematical tool. For a missile, the missile is trying to guide itself to lead the target so a collision occurs. The enemy target has radar warning receivers and warns the pilot that a missile is guiding and the pilot takes evasive action and maneuvers to avoid the intercept. The model inside the missile uses a Kalman filter to predict the future target motion due to the target turning at a constant G level and the missile recomputes and maneuvers accordingly.
The Kalman filter deals with noise in the measurements (just like the Fed trying to measure job or manufacturing growth) and minimizes the least squares error.
The problem is that the target may maneuver differently than a constant G level, so the prediction is off, or electronic countermeasures from the target may distort the measurements, and the missile will miss.
The same problem of completeness of the model and accuracy exists in spades in the Fed modeling. The economy is composed of hundreds of millions of interactions of individuals and companies and the government and other countries peoples and companies and governments interacting financially.
The true test of a model is the consistency with which it accurately predicts a future result within a certain accuracy.
Unfortunately, in my opinion, the state of economic modeling at the macro level is cruder than Fred Flintstone, and not yet useful. That said, given time, models may evolve that can actually be used to make key economic decisions better than we do now. Policy makers and the Fed could use a good, validated model, but at age 68, I am not sure I will live to see it.
Just a few thoughts from the aerospace world, where modeling is one or our key bread and butter tools.
Many thanks for all you do to keep us informed.
Thanks for that excellent set of comments Bob.
Jon Sellers said:
Great post. I think an even greater disconnect is that the Fed is attempting, in the end, to model human behavior and beliefs and the resulting spending patterns.
The new generation Uber driver is going to spend very differently than a unionized steelworker of the previous generation who had guaranteed benefits and a pension. A generation raised to believe property values only go up is going to spend differently than a young man who went through the bust.
But I didn’t see modeling changes in psychology in the discussion anywhere.
I admire your detail and attention span for something like the concept of the accuracy and validity of Fed modeling, or any economic modeling for that matter.
Me, I think even accepting the premise of ‘Fed Modeling’ or ‘Economic Modeling’ is like letting the salesman put a foot in the door and being too polite and / or gullible to go about my business uninterrupted. Even listening is buying something. Common sense is enough to reveal them as salesman or front men.
I have a hard time saying ‘Thank you for your analysis that is never right or only right sometimes in small part if you look at it in a certain light.’ I expect more from ‘experts’. The acceptance of their analysis as even meriting debate is an insult to taxpayers and everyone who is impacted by the decisions made from their mathy guesswork.
Tony Bennett said:
“The decompositions show that “net exports” and “change in private inventories” are particularly difficult subcomponents to nowcast.”
GDP growth is calculated quarter over quarter. Yesterdays “beat” on retail sales (for May) not the story, but the significant downward revisions to February and March business inventories. Which will be a downward revision to Q1 GDP … but boosting Q2’s by lowering the hurdle. A lot of moving parts with the plethora of data points and revisions.
Current Q1 real GDP +0.8% … I’m (fairly) confident at some point (end of this month? July? next year?) it will be revised negative.
Tony Bennett said:
“Might there be a “job gain recession, or one in which minimal jobs are lost?” Sure, why not?”
Possibly, if recession not deep. But need to dig deeper than jobs lost/gain … need to look at totality of hours worked, wages and participation. For someone working 30 hours a week … and see them cut to 20 … it will be painful, but won’t show up statistically as job lost.
Bob Henry said:
PS One more modeling thought. Systems can be linear, nonlinear, or chaotic or some combination. Possibly the most difficult piece of economic modeling is the human emotion modeling, which can involve fear and panic and create extreme discontinuities in times of panic. As a user of modern portfolio theory in my personal investing, I found that the theory works well until a time of financial panic, such as 2008 and 2009. In times of panic nearly all the somewhat decorrelated asset classes suddenly correlated, and tanked together resulting in a large hit to even a well diversified portfolio. Even gold struggled for a while, which I had thought would do well in time of panic. Of course, my mental model of gold was too simplistic and did not take into account liquidity, and that as assets tanked and there were margin calls, investors had to liquidate their gold.
I hope you keep up your quest to understand and better evolve the economic models, so over time better tools will be developed to fight stupidity.
Mathew Marty said:
I think a better model could be created using a dart board.
Throw a dart to start and then then adjustment darts as we go. I things are bad the adjustment darts subtract, f they look good then add.
What a colossal waste of money it is paying these people to forecast!!
Bob Henry said:
PPS Of course, my comments on modeling are completely swamped by politicians and Federal Reserve officials wanting to cook the books and use the models to justify doing what they want to enrich themselves and retain control.
Figures don’t lie, but liers figure.
I remember one really sharp statistician I worked with that finally figured out to ask his boss a key question before he did all of the complicated analysis. The key question he asked his boss was: What answer do you want? Fortunately the boss had the right motives of seeking the truth, which may not be true of the sponsors of the models, rendering the models worthless.
Indeed this is a rape of the west. Imagine if u had a slice of every dollar, could set interest rates, could manipulate the market, and do this all without any intervention? Any you fools look at analyzing the BS that put out. Fools indeed. They know what they are doing.