The arrival of deep learning techniques able to infer patterns from large datasets has dramati-
cally improved the performance of Artificial Intelligence (AI) systems. Deep learning’s rapid
development and adoption, in great part led by large technology companies, has however cre-
ated concerns about a premature narrowing in the technological trajectory of AI research de-
spite its weaknesses, which include lack of robustness, high environmental costs, and poten-
tially unfair outcomes. We seek to improve the evidence base with a semantic analysis of AI
research in arXiv, a popular pre-prints database. We study the evolution of the thematic diver-
sity of AI research, compare the thematic diversity of AI research in academia and the private
sector and measure the influence of private companies in AI research through the citations they
receive and their collaborations with other institutions. Our results suggest that diversity in AI
research has stagnated in recent years, and that AI research involving the private sector tends to
be less diverse and more influential than research in academia. We also find that private sector
AI researchers tend to specialise in data-hungry and computationally intensive deep learning
methods at the expense of research involving other AI methods, research that considers the so-
cietal and ethical implications of AI, and applications in sectors like health. Our results provide
a rationale for policy action to prevent a premature narrowing of AI research that could con-
strain its societal benefits, but we note the informational, incentive and scale hurdles standing
in the way of such interventions