It can refer to a person when you treat the adjective as a noun, but it is really just the color. Like you can say a black person. That does not mean that black refers to a person, it means that, in this particular context, you are modifying the person noun.

I'm not sure if in English you can do this, but in Spanish (my language), you can say something like "Pass me the knife" "Which knife?" "The black" (I think English needs a "one" added, as in, "the black one")

In Spanish, it would just be "Pasame el cuchillo" "¿Qué cuchillo?" "El negro".

So, negro is just the color black, but it has come to refer as well to black people. But it still is the colour. https://dle.rae.es/negro

You can see that 4 and 5 refer to black people and their related culture, but still, it's because of the colour.