Physical distancing reduces transmission risks and slows the spread of COVID-19. Yet compliance with shelter-in-place policies issued by local and regional governments in the United States is uneven and may be influenced by science skepticism and attitudes towards topics of scientific consensus. Using county-day measures of physical distancing derived from cellphone location data, we demonstrate that the proportion of people who stay at home after shelter-in-place policies go into effect is significantly lower in counties with a high concentration of science skeptics. These results are robust to controlling for other potential drivers of differential physical distancing, such as political partisanship, income, education and COVID severity. Our findings suggest public health interventions that take local attitudes toward science into account in their messaging may be more effective.