I don't think you are wrong for looking to be safe. An overabundance of safety can only drive you nuts haha
Last year, I was a little off on one side. I agonized over getting it level before I started but decided to leave it as it wasn't "that far off." No issues.
This year, I spent 2 days digging out parts of the ground to fix that slight imbalance and also because the new pool was 2 feet larger than our other pool. Sure enough, as I'm filling, I notice the ground is not level. I was P-O'd...lol I'm going to see how it settles once filled but also thought about jacking up 2 of the posts on the lower side but was not sure if that would put too much stress on the poles and railings. If it looks that far off at the end I'll re-do things but my back is still screaming from trying to level it so I said f it lol