The nut in impeller can sometimes come loose and spin inside the impeller but usually it is not intermittent and would have the same issue after priming.
A clogged impeller can also reduce the priming ability of the pump so either it won't prime or it takes a long time to prime.
But per other recommendations, have you actually watched the weir/skimmer when the pump is starting to prime? If the water level is too low or the weir is stuck, the skimmer can suck in air to prevent priming but after it is primed, there may not be an issue. So just because there is no air entering the system after prime does not mean there isn't an issue at the skimmer. Just make sure.
Also, once primed, is the filter pressure above or below normal operating pressure?