You’re describing perfectly some fine sediment that collected in bunches on our vinyl floor. Gritty and not slimy. In the end I deemed it to be very fine inorganic soil components. Probably silt. Our pool collected a lot of grass and dirt from the yard. Maybe yours is the same.
But we could...