Differences in college and post-graduate degree attainment alone explain less than half of Black-White and Hispanic-White wealth gaps in a standard wealth regression. Differences in family structure and measures of luck such as income windfalls and inheritances explain even less. Measures of financial decisionmaking, such as the share of housing in total assets and debt ratios, are much more important. Controlling for differences in all of these observable factors simultaneously and adjusting for life-cycle effects, we can explain about 70 percent of the Black-White wealth gap and virtually all of the Hispanic-White and Asian-White wealth gaps, consistent with earlier research. However, this model assumes equal wealth returns to education levels across racial and ethnic groups. Relaxing this assumption, we find significantly different wealth outcomes across racial and ethnic groups within the same education level. This, in turn, weakens the conclusion that racial and ethnic wealth gaps are largely explainable with observable factors. More importantly, the standard approach assumes all families face the same choice and opportunity sets. We investigate an alternative theoretical framework that attributes racial or ethnic group mean differences in education, family structure, financial decisions, and luck not to individual choice or effort in the face of equal opportunities but instead to systemic or structural factors in the past and present. In this model, the share of wealth gaps that can be explained by observables falls below one fifth for Blacks and Hispanics and to about three quarters for Asians. In other words, a structural determinants framework suggests the vast majority of the Black-White and Hispanic-White wealth gaps may lie beyond the scope of individual actions or marginal policy changes directed at educational attainment, family structure, financial decisionmaking, or even wealth redistribution. Instead, the gaps appear to be deeply rooted in unobservable factors that may include discrimination or other long-lasting disadvantages.