Hi, welcome to TFP! What would worry me is that pea gravel never really compacts. I have never seen the foam panels that people put under the liner, I have always used sand or sand vermiculite mixture. I guess the only way I would even consider leaving the gravel is cover it with a good layer...