He said this: I have never done anything "useful". No discovery of mine has made, or is likely to make, directly or indirectly, for good or ill, the least difference to the amenity of the world.

That's in https://en.m.wikipedia.org/wiki/A_Mathematician%27s_Apology, the summary of which matches what you said:

One of the main themes of the book is the beauty that mathematics possesses, which Hardy compares to painting and poetry.[5] For Hardy, the most beautiful mathematics was that which had no practical applications in the outside world (pure mathematics) and, in particular, his own special field of number theory. Hardy contends that if useful knowledge is defined as knowledge which is likely to contribute to the material comfort of mankind in the near future (if not right now), so that mere intellectual satisfaction is irrelevant, then the great bulk of higher mathematics is useless. He justifies the pursuit of pure mathematics with the argument that its very "uselessness" on the whole meant that it could not be misused to cause harm. On the other hand, Hardy denigrates much of the applied mathematics as either being "trivial", "ugly", or "dull", and contrasts it with "real mathematics", which is how he ranks the higher, pure mathematics.