It was the cam. Not broken, but grooved and mangled. Too bad I didn't check the thread earlier, I already ordered a repair kit. The cam and lifter are pretty worn, and maybe could be reused, but it didn't cost much more for the kit than the cam alone.
Thanks for the help!