CFED Scorecard

Methodology

The Scorecard Framework

Methodology for Outcome Data
    Data Collection
    Data Sources
    Precision of Estimates
    Ranking Outcome Measures
    How States are Ranked and Graded
    Timeliness of the Data
    Calculating Estimated Impacts

Methodology for Policy Data
    Data Collection
    Data Sources
    Policy Ratings
    Scorecard Methodology Prior to 2014
   
New in the 2015 Scorecard
    Liquid Asset Poverty Calculator

   

The Scorecard Framework

The 2015 Assets & Opportunity Scorecard provides a picture, from a household financial security perspective, of both how families in each state fare and the policies in place to improve outcomes. In developing the Scorecard's framework and measures, CFED draws upon its own hands-on experience in technical assistance and strategic policy design as well as the expertise of external advisors.

To present a revealing portrait of each state and the District of Columbia, CFED compiled 67 outcome measures and 68 policy measures organized into a five-issue area framework: Financial Assets & Income, Businesses & Jobs, Housing & Homeownership, Health Care and Education.

The 68 policy measures are identified by CFED as critical for promoting financial security and opportunity in a state. Taken together, these policies provide a comprehensive view of what states can do to help residents build and protect assets. They are, however, by no means the only policies that states could or should adopt to expand economic opportunity. These policies provide a starting point for individual states to consider in the context of the environment in their particular states.

Return to top

Methodology for Outcome Data

The Scorecard assesses states on the financial security and economic opportunity of households on 67 outcome measures. For each outcome measure, states are compared to each other and ranked. Measures in each issue area are averaged, ranked and graded.

Data Collection

The Scorecard draws on a wide range of data sources to produce a comprehensive picture of financial security and economic opportunity. Data collection for outcome measures included in the 2015 Scorecard took place between September and December in 2014, and most of the measures reference outcomes from 2012 to 2014.

Data Sources

Data on the 67 outcome measures primarily comes from publically available data sources, but data are also purchased and commissioned from private sources when data are not publically available. The U.S. Census Bureau is the largest source of data for the Scorecard – predominantly the American Community Survey (ACS) and the Survey of Income and Program Participation (SIPP) – but we also collect data from other federal agencies such as the Bureau of Labor Statistics. Private data purchased for the Scorecard include data on foreclosed and delinquent mortgages from the Mortgage Bankers Association and data on credit and debt from TransUnion.

All Scorecard measures show data at the state level; however, for 12 outcome measures, data and rankings are not available for every state due to issues related to sample size.

Precision of Estimates

A key data source for the Scorecard is the Census Bureau’s SIPP, a national survey that collects data on household wealth, assets and liabilities. While the SIPP is the only available data source that is large enough to produce estimates of net worth and asset poverty at the state level, the sample size for the SIPP is relatively small, particularly for smaller states and for sub-populations – such as households of color or single-parent households. A small sample size means that estimates produced from the data are more likely to be imprecise, which means that an estimate of household wealth may not accurately represent the true wealth that households in that state or population hold. More information concerning the precision of Scorecard data generated from the SIPP can be found here.

Ranking Outcome Measures

Each state is individually ranked for 55 of the 67 outcome measures. The state with the most desirable outcome is ranked 1st, and the least desirable is ranked 51st. For example, the state with the highest homeownership rate is ranked 1st. Similarly, because a low level of credit card debt is desirable, the state with the lowest average credit card debt is ranked 1st.

For measures where data are not available for every state, states are not ranked out of 51; instead, they are ranked out of the total number of states for which data are available. For example, data on liquid asset poverty is available for only 39 states; thus, the state with the least desirable outcome is ranked 39th.

Although data are rounded for presentation purposes, states are ranked based on the actual, unrounded values when available. For example, two states might have different ranks even though the measures appear to be the same. There are a few measures for which unrounded values are not available from the data source, and in those instances, states with the same value share the same rank. The next-best performing state is ranked as if the tie had not occurred. For example, if two states have a rate of college graduates with student loan debt of 69%, each is ranked 43rd, and the next state is ranked 45th.

States are not ranked on 12 outcome measures in the 2015 Scorecard. There are two reasons a measure is unranked:

  1. Insufficient Data: Data for specific states may be unavailable, either due to a small sample size, or because the margin of error for the data estimate is too large to publish or rank that outcome measure (see here for more information on estimates calculated from the SIPP). Using this methodology, if fewer than 35 states can be ranked for a given outcome measure, that outcome measure is not ranked.
  2. Clustering of Data: Two measures in the 2015 Scorecard are unranked because there is too little variation in the data between states to meaningfully compare outcomes between states. These two measures are Homeownership by Gender and Four-Year Degree by Gender, which measure the disparity between men and women. Because the difference between the outcomes of the best and worst states for both measures is relatively small, CFED chose not to rank states on these measures.

How States Are Ranked and Graded

To calculate a state’s overall outcome rank, the state’s ranks for each individual outcome are averaged to generate an overall score. The lower the overall score, the better the state's overall performance in the Scorecard. The overall score for the states is then ranked from 1 to 51.

In prior iterations of the Scorecard, overall outcome ranks were calculated by averaging each state’s issue area ranks. This ranking methodology was designed to weight each issue area equally, but in the new methodology, all outcome measures are now weighted equally. The methodology has been updated in an effort to align the overall policy and outcome rankings.

Individual issue area outcome ranks and grades are calculated using roughly the same methodology as overall ranks – individual measure ranks are summed and averaged within the issue area to generate an score for that issue area, upon which the states are ranked. Issue area grades are assigned on a curve: states that rank from 1 to 10 earn an A; from 11 to 20 earn a B; from 21 to 36 earn a C; from 37 to 46 earn a D; and from 47 to 51 earn an F.

Notes on the Ranks:

  • A state is not penalized in the issue area ranks if data is missing for that state. A state’s issue area rank is calculated by taking the average rank of every outcome measure available for a state in that issue area. For example, North Dakota only has data for ten out of twelve outcome measures in the Education issue area, and as a result, their issue area rank is calculated using the ten available measures instead of twelve.
  • When two states receive the same overall score or average issue area rank, each state receives the same rank, and the next-best performing state is ranked as if the tie had not occurred. For example, if two states have the best score, each is ranked 1st, and the next state is ranked 3rd.

Timeliness of the Data

Some outcome measures in the Scorecard are collected from sources that are updated quarterly or annually (such as foreclosure and unemployment), while other measures are only available every two to five years (such as unbanked households and business ownership by race and gender). Moreover, even recently released data can reference a time period that is several years in the past. Because of this lag, even though the Scorecard draws on the most recent data available at the time of production, it is inevitable for some data to become out-of-date rather quickly. These inherent data issues limit the Scorecard's ability to reflect recent changes that have taken place in the economy. However, such issues do not diminish ability of the Scorecard to tell a story of relative performance of states when compared to each other during the same time period. here.

Calculating Estimated Impacts

As a feature of the Scorecard, we calculate the estimated impact of an improvement in a state’s performance to match the performance of the top ranking state on a select number of measures. For example: If Ohio improved its homeownership rate (66%) to match the homeownership rate in the best performing state, West Virginia (72%), how many more homeowners would there be in Ohio?

The estimates were calculated as follows: for each indicator, the rate of the best performing state was multiplied by the appropriate population (households, adults over 25, labor force, etc.) in the remaining states. The difference between the states’ improved performance and actual performance was then calculated. There are 3,015,795 homeowners in Ohio. If the state’s homeownership rate improved to West Virginia’s, there would be 3,301,669 homeowners – an estimated impact of 285,874 additional homeowners.

Return to top

Methodology for Policy Data

The 2015 Scorecard assesses states on 68 policies. Each of the 68 policies is drawn from the five issue areas in the Scorecard: Financial Assets & Income, Businesses & Jobs, Housing & Homeownership, Health Care and Education. None of these policies alone is the silver bullet solution, but each contributes to building the financial security of residents. The Scorecard presents a menu of policy options for what state policymakers can do to provide financial security and opportunity for residents. States can consider these 68 policies as the starting point for a proactive asset policy agenda and pursue those policies that reflect the particular needs of their state. 

For presentation purposes, many policies are grouped together for ease of understanding—e.g., all the policies related to a state Earned Income Tax Credit are displayed on a single web page. Regardless of how the policies are grouped, each policy is given equal weight—i.e., one point if the state has adopted the policy and zero points if the state has not adopted the policy.

Data Collection

Data collection for the policy priorities in the Scorecard took place between June and December, 2014. With a few exceptions, the 68 policy priorities reflect policies adopted as of the end of September 2014. 

Data Sources

Policy data for the Scorecard are drawn from research and resources created mainly by policy organizations, academic institutions and think tanks with expertise in the specific issue areas covered in the Scorecard. The policies selected for inclusion in the Scorecard and the criteria for assessing the strength of those policies were identified via one-on-one phone conversations with national intermediaries, funders, issue area experts and researchers; conference calls with a Scorecard Policy Advisory Group made up of nine Assets & Opportunity Network Leaders; and data reported from 67 Lead Organizations about their policy priorities. The policy measures span the five issue areas and are either promising or proven in helping families build and protect assets. Where information on the state-by-state status of policies was not already documented by an external organization, CFED consulted with experts and conducted original research, which included state-by-state surveys and interviews.

Policy Ratings

Unlike the outcome measure rankings, which compare states to one another within each measure, individual policy measures assess states on a binary scale – yes or no. As such, there are no rankings within policy measures. Instead, states are ranked on the number of policies each has adopted, relative to the other 50 states, within each of the five issue areas and overall. Both the overall and issue area ranks are derived by calculating the sum of all policies each state has adopted as a percentage of all policies for which data was available. As a result, states are not penalized for a lack of available data, and are instead assessed on a relative scale. The state with the highest percentage of policies adopted is ranked 1st; the state with the lowest percentage is ranked 51st. Like the outcome measure rankings, when two states receive the same overall score or average issue area rank, each state receives the same rank. The next-best performing state is then ranked as if the tie had not occurred.

Scorecard Methodology Prior to 2014

Prior to 2014, CFED assessed states on the strength of 12 “policy priorities,” each of which had four sub-criteria. The Scorecard reported on other policies, which were called “additional policies,” but it did not judge whether these policies met a particular threshold. In 2014, the Scorecard began treating all policies equally. Most of the 48 “criteria” became separate policies and many of the “additional policies” were evaluated, rather than just reported. Several new policies were also added to the Scorecard based on input from issue-area experts and Lead Organizations in the Assets & Opportunity Network. By expanding the breadth of the policy measures, the Scorecard provides a clearer picture of the overall strength of each state's policies. 

Return to top

New in the 2015 Scorecard

The 2015 Scorecard now includes two new outcome measures and five new policy measures. The two new outcome measures are disconnected youth (i.e., percentage of population aged 16-24 that are neither in school nor employed) and income inequality. The five new policy measures assess whether states have eliminated asset limits from the Low-Income Home Energy Assistance Program; streamline enrollment into Medicaid for adults; protect against abusive debt-buying practices; run an auto-Individual Retirement Account program; and provide residents a universal incentive to save for college.

In 2015, one outcome measure (uninsured low-income parents) and four policy measures were removed (expanded COBRA coverage; preservation of affordable rental housing; mortgage credit certificates for first-time homebuyers; and targeted education spending in low-poverty vs. high-poverty districts).

Additionally, the criteria for eight policy measures were modified in 2015: state tax fairness; CDBG support for microbusiness; WIA support for microbusiness; minimum wage; eligibility for unemployment benefits; titling and zoning of manufactured homes; college savings incentives; and student protections from deceptive for-profit schools. 

Liquid Asset Poverty Calculator

New to the 2015 Scorecard is the Liquid Asset Poverty Calculator, which allows you to determine exactly how much in liquid savings (i.e., cash, retirement savings, checking account, etc.) your household would need to avoid falling into liquid asset poverty. This liquid asset poverty threshold is based upon the most recent poverty guidelines published by the United States Department of Health and Human Services (HHS). The HHS poverty guidelines vary by household size. The three-month liquid asset poverty threshold is calculated by dividing the annual poverty guideline for each household size by four.

Return to top

Copyright © 2015 CFED — Corporation for Enterprise Development 1200 G Street, NW Suite 400 Washington, DC 20005 202.408.9788