To understand how best to combat COVID-19, we must understand how deadly is the disease. There is a substantial debate in the epidemiological literature as to whether the fatality rate is 1% or 0.1% or somewhere in between. In this note, I use an SIR model to examine why it is difficult to estimate the fatality rate from the disease and how long we might have to wait to resolve this question absent a large-scale randomized testing program. I focus on uncertainty over the joint distribution of the fatality rate and the initial number of active cases at the start of the epidemic around January 15, 2020. I show how the model with a high initial number of active cases and a low fatality rate gives the same predictions for the evolution of the number of deaths in the early stages of the pandemic as the same model with a low initial number of active cases and a high fatality rate. The problem of distinguishing these two parameterizations of the model becomes more severe in the presence of effective mitigation measures. As discussed by many, this uncertainty could be resolved now with large-scale randomized testing.