Methodology

The Scorecard Framework

Methodology for Outcome Data
    Data Collection
    Data Sources
    Precision of Estimates
    Ranking Outcome Measures
    How States are Ranked and Graded
    Timeliness of the Data

Methodology for Policy Data
    Policy Priorities
    Data Collection
    Data Sources
    Policy Ratings
    Additional Policies

New in the 2013 Scorecard

    

The Scorecard Framework

The 2013 Assets & Opportunity Scorecard provides a picture, from a household financial security perspective, of both how families in each state fare and the policies in place to improve outcomes. In developing the Scorecard's framework and measures, CFED draws upon its own hands-on experience in technical assistance and strategic policy design as well as the expertise of external advisors.

To present a revealing portrait of each state and the District of Columbia, CFED compiled 69 outcome measures and 33 policy measures organized into a five-issue area framework: Financial Assets & Income, Businesses & Jobs, Housing & Homeownership, Health Care and Education.

The 33 policy measures include 12 policy priorities and 21 additional policies. The policy priorities are identified by CFED as critical for promoting financial security and opportunity in a state. Taken together, these policies provide a comprehensive view of what states can do to help residents build and protect assets. They are, however, by no means the only policies that states could or should adopt to expand economic opportunity. These policies provide a starting point for individual states to consider in the context of the environment in their particular states.

Return to top

Methodology for Outcome Data

The Scorecard assesses states on the financial security and economic opportunity of households on 69 outcome measures. For each outcome measure, states are compared to each other and ranked. Measures in each issue area are averaged, ranked and graded. To download a spreadsheet of all outcome measures and ranks published in the 2013 Scorecard, click here.

Data Collection

The Scorecard draws on a wide range of data sources to produce a comprehensive picture of financial security and economic opportunity. Data collection for outcome measures included in the 2013 Scorecard took place between June and December in 2012, and most of the data reference outcomes from 2010 to 2012.

Data Sources

Data on the 69 outcome measures primarily comes from publically available data sources, but data are also purchased and commissioned from private sources when data are not publically available. The U.S. Census Bureau is the largest source of data for the Scorecard – predominantly the American Community Survey (ACS) and the Survey of Income and Program Participation (SIPP) – but we also collect data from other federal agencies such as the Bureau of Labor Statistics. Private data purchased for the Scorecard include data on foreclosed and delinquent mortgages from the Mortgage Bankers Association and data on credit and debt from TransUnion.

All Scorecard measures show data at the state level; however, for 16 measures data are not available for every state due to issues related to sample size.

Precision of Estimates

A key data source for the Scorecard is the Census Bureau’s SIPP, a national survey that collects data on household wealth, assets and liabilities. While the SIPP is the only available data source that is large enough to produce estimates of net worth and asset poverty at the state level, the sample size for the SIPP is relatively small, particularly for smaller states and for sub-populations – such as households of color or single-parent households. A small sample size means that estimates produced from the data are more likely to be imprecise, which means that an estimate of household wealth may not accurately represent the true wealth that households hold. More information concerning the precision of Scorecard data generated from the SIPP can be found here.

Ranking Outcome Measures

Each state is individually ranked for 53 of the 69 outcome measures. The state with the most desirable outcome is ranked 1^st, and the least desirable is ranked 51^st. For example, the state with the highest net worth is ranked 1^st. Similarly, because a low level of credit card debt is desirable, the state with the lowest average credit card debt is ranked 1^st.

For measures where data are not available for every state, states are not ranked out of 51; instead, they are ranked out of the total number of states for which data are available. For example, data on liquid asset poverty is available for only 41 states; thus, the state with the least desirable outcome is ranked 41^st.

Although data are rounded for presentation purposes, states are ranked based on the actual, unrounded values. For example, two states might have different ranks even though the measures appear to be the same.

States are not ranked on 16 outcome measures in the 2013 Scorecard. There are two reasons a measure is unranked:

1. Insufficient Data: Data for specific states may be unavailable, either due to a small sample size, or because the margin of error for the data estimate is too large to publish or rank that outcome measure (see here for more information on estimates calculated from the SIPP). Using this methodology, if fewer than 35 states can be ranked for a given outcome measure, that outcome measure is not ranked.
2. Clustering of Data: Two measures in the 2013 Scorecard are unranked because there is too little variation in the data between states to meaningfully compare outcomes between states. These two measures are Homeownership by Gender and Four-Year Degree by Gender, which measure the disparity between men and women. Because the difference between the outcomes of the best and worst states for both measures is relatively small, CFED chose not to rank states on these measures.

How States Are Ranked and Graded

To calculate a state’s rank for each issue area, the state’s ranks for each outcome measure in an issue area are averaged (all measures are weighted equally). The lower the average, the better the state's overall performance for that issue area. Using a state’s issue area rank, a grade is assigned to each state. Grades are assigned on a curve: states that rank from 1 to 10 earn an A; from 11 to 20 earn a B; from 21 to 36 earn a C; from 37 to 46 earn a D; and from 47 to 51 earn an F.

The overall outcome rank is calculated by averaging a state’s issue area ranks to generate an overall score (all issue areas are weighted equally). The lower the overall score, the better the state’s overall performance in the Scorecard. The overall score for the states is then ranked from 1 to 51.

Notes on the Ranks:

• A state is not penalized in the issue area ranks if outcome data is missing for that state. A state’s issue area rank is calculated by taking the average rank of every outcome measure available for a state in that issue area. For example, New Mexico only has data for ten out of twelve outcome measures in the Education issue area, and as a result, their issue area rank is calculated using the ten available measures instead of twelve.
• When two states receive the same overall score or average issue area rank, each state receives the same rank, and the next-best performing state is ranked as if the tie had not occurred. For example, if two states have the best score, each is ranked 1st, and the next state is ranked 3rd.

Timeliness of the Data

Some outcome measures in the Scorecard are collected from sources that are updated quarterly or annually (such as foreclosure and unemployment), while other measures are only available every two to five years (such as net worth and business ownership by race and gender). Moreover, even recently released data can reference a time period that is several years in the past. Because of this lag, even though the Scorecard draws on the most recent data available at the time of production, it is inevitable for some data to become out-of-date rather quickly. These inherent data issues limit the Scorecard's ability to reflect recent changes that have taken place in the economy. However, such issues do not diminish ability of the Scorecard to tell a story of relative performance of states when compared to each other during the same time period.

Return to top

Methodology for Policy Data

The Scorecard includes two types of policies – policy priorities and additional policies – with separate methodologies, described below.

Policy Priorities

The Scorecard selects a dozen policies to highlight. For these 12, CFED conducts extensive research to assess the strength of these policies in all 50 states and the District of Columbia. By highlighting 12 of the 33 policies as policy priorities, the Scorecard lays out a model agenda for what state policymakers can do to provide financial security and opportunity for residents. States can consider these 12 policies as the starting point for a proactive asset policy agenda and pursue those policies that reflect the particular needs of their state. The Scorecard evaluates each state on the strength of its policy priorities, utilizing easy-to-follow policy icons.

The 12 policies are drawn from the five issue areas in the Scorecard: Financial Assets & Income, Businesses & Jobs, Housing & Homeownership, Health Care, and Education. None of these policies alone is the silver bullet solution, but each is a piece of the assets puzzle. There is no overall policy priority rating for the states, but rather, each policy priority receives its own individual rating.

Data Collection

Data collection for the policy priorities in the Scorecard took place between April and September in 2012. With a few exceptions, the 12 policy priorities reflect policies adopted as of the end of September 2012.

Data Sources

Policy data for the Scorecard are drawn from research and resources created mainly by policy organizations, academic institutions and think tanks with expertise in the specific issue areas covered in the Scorecard. The policy priorities selected for inclusion in the Scorecard and the criteria for assessing the strength of those policies were identified and vetted with the Assets & Opportunity Network Steering Committee and with other policy experts. The policy priorities span the five issue areas and are either promising or proven in helping families build and protect assets. Where information on the state-by-state status of policies was not already documented by an external organization, CFED consulted with experts and conducted original research, which included state-by-state surveys and interviews and using LexisNexis’ state legislative tracking service.

Policy Ratings

Unlike the outcome measure rankings, which compare states to one another, for the policy priority measures, states are assessed against an absolute standard for what constitutes a strong policy. For each of the 12 policy priorities, CFED gathered the best available research and guidance from the foremost policy experts in the field to develop four criteria for what constitutes a strong policy. CFED then evaluated the strength of each state’s policies against the four criteria for each policy priority. Criteria used to rate state policies vary among the 12 policy priorities and are specific to individual policies. Policy ratings are based on the following scale:

Additional Policies

The Scorecard provides information on 21 policies in addition to the 12 policy priorities. Data for the 21 additional policy measures in the Scorecard were collected from external policy experts and organizations between June and December in 2012 and reflect the most recent data available from these sources. While the strength of states' additional policies are not evaluated and rated, the Scorecard includes this data to provide a comprehensive picture of state policies that support a positive asset-building and asset-protection environment.

Return to top

New in the 2013 Scorecard

Aside from several design updates – including a custom reporting center and in-page graphics for all outcome measures – the 2013 Scorecard now includes two new measures that were not included in the 2012 edition: Households with Savings Accounts and Student Loan Default Rate. One measure – Employee Ownership Rate – was removed, as our data source is unlikely to update the data in the foreseeable future.

Return to top