More often than not, what many folks see as "brownish dirt" is dead algae. You may have clear water but invisible algae in your pool and when you add chlorine, some (but not all) of that algae is killed and becomes debris on the bottom of your pool floor. If you vacuum it up, it often returns the next day in the same spots.
Invariably, these pools have inadequate chlorine and the solution is a SLAM. The SLAM usually goes very quickly and it rids your pool of ALL the algae and the "brownish dust" disappears.
Most people search for answers other than that (filtration, nearby fields, pollen, etc.) but time and again we see inadequate chlorine as the culprit and the SLAM as the cure.
If you decide you are not in the mood for a full blown SLAM or that your "dust" problem is VERY minimal, then elevate your chlorine to twice it's normal level for 48 hours and see if that doesn't cure it. It's the same idea but with a lot less formality and rules to follow but, of course, it is not as effective. I would hazard a guess and say that chlorine elevation (x2) would probably work 50% of the time.
