You could tie skimmer sock over the returns, basically to re-filter the water before they come into the pool.
I had a similar issue where sand/dirt/algae just keeps showing up even a few hours after I vacuumed it up. And it always accumulates at the same spot—-later I figured out that it has to do with where the return is and how water circulates in the pool.
I have a cartridge filter, so I know it’s not from crack in the filter. But I did open it up and see a pile of sand, god knows from where. I cleaned it up and carefully vacuumed the floor once more. It doesn’t seem to return now.
I was googling a lot because of my experience. There was a very old post calling this “pool dust”. The skimmer sock over return thing is their idea and a lot of people there said it worked.
Not saying your problem is the same, but it is an easy and cheap thing to try.