I don't know what type of pool you have, but vinyl liner could be mold underneath. Plaster pool could be black algae. Neither one of those will just brush up. Black algae looks likesmall black spots and can be treated with copper algaecide. I dot know what to suggest if it's mold under a liner.