I would not unless it was a concrete pad. Rule #1 is do not build up to level, dig down onto solid ground. I have seen retaining walls collpase over years due to erosion, poorly constructed, heavy rains, ground shifting, etc. At least a concrete pad would reinforce it.
Saying that however, if the pad was several years old, and heavily compacted that may be a different story.
I would consider using the area in front of the second wall, and turning that newly done section into your patio/pool/party area! You would almost have a walk in pool due to the height differences, and have an awesome size pool party area.