The COVID-19 pandemic caused unprecedented disruptions to education, with over 1.5 billion students worldwide experiencing learning interruptions. To address the challenges posed by the pandemic, government agencies and parliamentary committees around the globe published over 2,500 policy documents citing 15,000 scholarly papers. In this study, we aim to discover if policy makers used the best available academic research, studying the difference in cited papers across geography and academic discipline. Using data from the International Centre for the Study of Research (ICSR) Lab and Overton platforms, we analysed the influence of academic citations, journal CiteScore, author h-index, open access, and media citations on policy impact. Our findings revealed that policy makers relied on a narrow range of scholarly evidence. Although there were over 400,000 scholarly papers published during the COVID-19 pandemic, that vast majority of these papers were not cited in policy. Over 70% of cited papers in education policy were published before 2020. There is also a high discrepancy across discipline, with policy makers citing modern, medical research, but relying on dated education research. Finally, we found a weak relationship between research excellence and policy impact, indicating that other, non-academic factors may play a larger role in determining which research is cited in policy.