Welcome to the forum :lol:
Reluctantly, I must agree with JohnT and bk406. I absolutelyhated the expense of the extra gravel to back-fill but I bit the bullet (also a concrete block wall, incidentally) and did it. Five years later I have yet to have a crack in over 2000 sq. ft. of decking.
However, I will say that the truest answer is that "It depends". The freezing level in your area is important. Perhaps even more so is the nature of your soil. Given a sandy soil in a Southern state, I believe it may compact safely in 2-3 months for pavers.
A clay soil in Canada, however, may never stop moving and settling completely because of the frost heave.