Methodology for Producing and Ranking Estimates of Net Worth and Asset Poverty

Producing estimates and margin of error data from the SIPP

Data on net worth, asset poverty, liquid asset poverty and extreme asset poverty in the Assets & Opportunity Scorecard are produced from the U.S. Census Bureau’s Survey of Income and Program Participation (SIPP). The SIPP is a national panel survey with a sample that includes over 40,000 households and is designed to be representative at the state level. The SIPP’s relatively large sample size makes it the only data source that is large enough to provide data on wealth and assets at the state level. However, the sample size for any given state will be much smaller than 40,000 households. When sample sizes are small, it means there is less certainty that an estimate from the sample accurately reflects the population that the sample is supposed to represent. To account for this, statisticians calculate what is known as a margin of error to help us understand how precise or reliable the sample data is as a proxy for what is true for the entire population. Most people have seen election polls reported in the news that declare, for example, that a candidate holds a 7% lead over his or her opponent with a margin of error of +/-2%.

In the case of the SIPP data used in the Scorecard, a state with a small sample (or a small sample of a specific type of household when we are doing sub-state analysis by race or family structure) could have an estimate for household wealth that might not accurately represent the true amount of wealth for all households in the population.

Data in the Scorecard calculated from the SIPP • Asset poverty • Asset poverty by race • Asset poverty by gender • Asset poverty by family structure • Liquid asset poverty • Liquid asset poverty by race • Liquid asset poverty by gender • Liquid asset poverty by family structure • Extreme asset poverty • Net worth • Net worth by race • Net worth by income • Net worth by gender • Net worth by family structure

In an effort to improve the precision and transparency of the data calculated from the SIPP in the Scorecard, CFED has commissioned further statistical analysis to examine the margin of error for all outcome measures that are produced from the SIPP. The margin of error provides a range within which we are 90% confident that the true value lies. For example, if we estimate that the asset poverty rate for a particular state is 30%, and the margin of error is +/-4%, we are 90% confident that the asset poverty rate in that state for the entire population is between 26% and 34%.

To determine how large the margin of error is relative to the state’s estimate, CFED calculated the relative margin of error (RMOE) for the estimates for each state. To calculate relative margin of error, the absolute margin of error is simply divided by the sample’s point estimate (so, 4% divided by 30% equals a 13% RMOE). The larger the margin of error or RMOE, the less accurate the sample’s estimate is as a representation of the true household wealth of the population.

You can download margin of error data for each Scorecard measure produced from the SIPP by clicking here. We also calculated the margin of error for the SIPP measures in the previous Scorecard, and you can download the 2012 margin of error data here.

Criteria for publishing and ranking estimates from the SIPP in the Scorecard

Once the margin of error and relative margin of error are calculated for each of the 14 outcomes measures in the Scorecard that are derived from the SIPP, we use the following criteria to determine if and how to include the estimates for the outcome measures that are included in the Scorecard:

1. Estimates are only published if the RMOE is 50% or less. For example, if asset poverty is estimated to be 30%, +/- 18% (RMOE of 60%), we are 90% confident that asset poverty in that state is between 12% and 48%. A margin of error of that magnitude indicates that the point estimate does not provide a meaningful assessment of asset poverty in that state.
2. Estimates with a RMOE between 50% and 25% are published, but not ranked. For example, if a state’s asset poverty rate is estimated to be 30%, +/- 9% (RMOE of 30%), we are 90% confident that asset poverty in that state is between 21% and 39%. While these estimates are relatively imprecise, we believe they provide useful information about the financial state of households that would otherwise be unavailable. However, caution should be used when interpreting these data. We do not rank these estimates because the data is too imprecise to say with confidence how the state compares to other states.
3. Estimates with a RMOE of ≤ 25% are eligible for ranking in the 2012 Scorecard if they meet two additional criteria:
   • Estimates in the Scorecard are only ranked if estimates have an RMOE of ≤ 25% and if we are confident of the stability of the rank vis-a-vis other states. In order to be ranked, we need to be 90% confident that the state’s true rank would not move more than 10 places in absolute terms. In other words, if a state’s rank could move more than 10 positions in either direction when compared to other states with estimates in its confidence interval, we do not rank those estimates. For example, if a state’s asset poverty rate is estimated to be between 25% and 30%, a state’s rank could also range from 18 to 40 if many other states also have estimates in the range of 25% to 30%, and we would not rank that state for their asset poverty estimate.
   • Estimates that meet precision criteria for both the point estimate and rank stability are only ranked if data is available for at least 35 states. We only rank states on data measures for data can be ranked for at least 35 states. Measures that are not ranked are not included in the calculations of the issue area grades or ranks or the overall rank.

Applying the criteria outlined above, of the 14 measures included in the Scorecard, CFED is only ranking liquid asset poverty in the 2012 edition of the Scorecard.

Acknowledgements

CFED extends warm appreciation to Caroline Ratcliffe and Doug Wissoker of the Urban Institute and Jon Haveman of the Bay Area Economic Institute for assistance in evaluating and developing the criteria around the bias and precision of the SIPP estimates. We also thank the Frank DiGiovanni and the Ford Foundation for support that enabled us to deepen our analysis and refine our methodology for utilizing SIPP data in the Scorecard.