If it's clean when you finish vacuuming, it is most likely one of two things...
1. Dead algae settling to the pool floor. It brushes up into a cloud with a pool brush
2. Airborne dirt and dust which resettles VERY quickly if you brush it up.
If it's #1, you need to post a full set of test results and be prepared to start the shock process.
If it's #2, you just have to live with it.