Dig 6 inches under where paver will go, put a compactablele base ( crusher run or what ever its called out there) put 2 inch layer at a time,slightly dampen it, and compact it with a hand tamper. Repeat until flush with ground and top it with the paver. Should solve ur issue