Anyone who seriously studies algebraic graph theory or finite permutation groups will, sooner or later, come across the Paley graphs and their automorphism groups. The most frequently cited sources for these are respectively Paley's 1933 paper for their discovery, and Carlitz's 1960 paper for their automorphism groups. It is remarkable that neither of those papers uses the concepts of graphs, groups or automorphisms. Indeed, one cannot find these three terms, or any synonyms for them, in those papers: Paley's paper is entirely about the construction of what are now called Hadamard matrices, while Carlitz's is entirely about permutations of finite fields. The aim of this talk is to explain how this strange situation came about, by describing the background to these two papers and how they became associated with the Paley graphs. This involves links with other branches of mathematics, such as matrix theory, number theory, design theory, coding theory, finite geometry, polytope theory and group theory, reaching back to 1625. I will summarise the life and work of these two great mathematicians, together with important contributions from Coxeter and Todd, Sachs, and Erd\H os and R\'enyi. I will also briefly cover some recent developments concerning generalised Paley graphs. A preprint is available at arXiv:1702.00285.