Yellowing on new plaster

I am thinking that the yellow areas are from some calcium build up due to the plaster "dust" hardening in those areas that didn't get brushed during the first two weeks. If so, that calcium deposit can be easily removed by sanding with 80 grit wet&dry sandpaper. It does NOT harm the plaster and will actually make the sanded areas smoother than the other non-yellow areas. A wire brush is not as effective as using sandpaper.
Just an observation.

Thanks for the response. Its something I can try. The area with more blue quartz seems a bit rougher compared to the rest of the pool. I think this seems to align with what you're saying in the sense that the blue spec area was consistently getting 'hit' with the return jet.

My biggest worry was that it was due to a poor mixture of the plaster itself, resulting in an inconsistent spread of the quartz.