bottom track

Working on leveling bottom track. Have a few pavers in. all level and set at the same height but there are gaps under the track, causing the actual track to be out of level. Do I just pack dirt underneath the track to bring it up to level or will this cause issues later on? Also considered...