I have the same dirt or sand in the bottom of my pool as well.... Changed laterals, new sand... Dirt still on bottom... Used robot to clean and have same dirt in my robot... Looks nothing like the filter sand.
Could an underground pipe be broke and pull and in?