Ok, I am a noob pool owner. Been lurking here for awhile now.
I live in a rural location with crop fields, pine trees and other trees all around. I get this brownish silt daily and I have vacuumed to waste, super chlorinated for 5 days a few times and it starts to come back in a few hours. I think it is just in the air and deposited freely in the pool all the time. I had the pool installed last Oct. and opened it in May so it has only been open a month and a half. Hopefully this is a spring, early summer thing but out here I would not be surprised if the stuff doesn't fly around here all the time though.