Wind dirt, nearby vegetation, rain, people, dogs....
When vacuuming, make sure the pump is on high, filter reasonably clean - usually does not have to be freshly backwashed. All other skimmers/deep ends are closed (may have to keep one or more open a bit to avoid air in the pump). If your head has removable brushes, leave one in the front off to provide an easy path for the water to flow to the suction hole. Vacuum slowly to avoid kicking up dirt, and allowing it time to get sucked up. Do small patches at a time. Overlap strokes - one pass out, move the head 1/2 of its width for the stroke back, then over 1/2 width again for the next. You may have to linger over piles and jiggle the head back and forth over it to get it all sucked up.
I like the clear triangular heads for those reasons - add or subtract brushes to aid in suction (or scrubbing) as needed. Clear so you can see what is getting sucked in vs just being kicked up into suspension.
It may take me an hour or more to vac my 18x36 pool.
With TFP methods, moderate weather, and luck, that may be only every few weeks. Brushing much more often, of course. Let the amount of accumulation and how well it is being sucked up be your guide.