I just joined and am new to being a pool owner, but my pool is also crystal clear and I can get silt in the bottom of the pool. For me, though, the source is obvious. I live in Pflugerville, TX just outside Austin. It can get windy often here. When the wind picks up debris from trees and what I don't even really notice directly (sand and dirt) get dumped into the pool.
My filter and pool cleaner (Polaris 360) cleans it up almost perfectly ( a few corners by the steps have to be manually stirred up and now and then). My pool cleaner just died and I am waiting for my rebuild kit to arrive. I had no idea just how effective it can be until now. I have an obvious layer of silt/sand building up on the bottom in only 2-3 days. The type of bag I use on my cleaner also catches sand (finer mesh) and it gathers anywhere from nothing to 2-3 cups per week depending on wind and what the wind picks up. I sat one day and watched the dirt on my neighbors roof get scoured up (and its not all that close) and a stream of wind perfectly directed the dirt collected on the roof into my pool.
Also note: I use a DE filter and it does a tremendous job of removing fine particles from the water. It is clearly far more effective with the robot tail constantly stirring up the water. Without the robot, my water is still crystal clear, but the bottom is collecting a lot, in days.