Large cities in the US are the most expensive places to live. Paradoxically, this cost is paid disproportionately by workers who could work remotely, and live anywhere. The greater potential for remote work in large cities is mostly accounted for by their specialization in skill- and information-intensive service industries. We highlight that this specialization makes these cities vulnerable to remote work shocks. When high-skill workers begin to work from home or leave the city altogether, they withdraw spending from local consumer service industries that rely heavily on their demand. As a result, low-skill service workers in big cities bore most of the recent pandemic’s economic impact.