There are industry standards (depending on where you live), that determine acceptable workmanship. For example, when setting tile, their height can vary only by x amount. I think it's the thickness of a dime or something like that. There will likely be some sort of similar dimension for pavers, though, sorry, I'm not sure what the resource is for such things. Someone else here might know. Is there a building inspector involved in your project? Or maybe google "acceptable paver installation specs" or something like that.
If the industry standards exist for pavers, then that will be your guide in approaching the installer about this issue...
That said, pavers won't be perfect, and even if you get them that way somehow, they can move over time. An alternate solution is to have the guy fix a few more, watch and learn, and fix them yourself, now, and when they go out again sometime in the future. That way, they can always be as perfect as you need them to be, regardless of standards.