I always test mine in the afternoon on a sunny day. I will sit in a shady spot to take my reading. Because of these types of variances and the subjective nature of the test, testing the same way each time works out best.
Apparently, the intensity of ambient light has the potential to skew the test. (See the quote below)
With significant amounts of practice, and proper outdoor lighting, most people can get repeatable results that give the same answer +-5. But the test it's self isn't really that precise. The CYA test is about +-15 when you are doing it correctly. Novices often stop early, when the black dot is kind of mostly obscured, which can give results that are 20 or 30 ppm higher than what they would get if they were doing it correctly. Reading the test indoors seems to result in numbers that are 10 or 20 ppm higher than otherwise.
But for me, I believe I get the best results from reading the test indoors, in front of a west facing full length window, in the afternoon. This way I get the benefit of strong indirect light, but I avoid the glare that I get when I try to read the test outdoors.