The organic material in your filter is treated the same way as anywhere else in your pool, by contact with chlorinated water. It's all part of the same system. Let's say there was something in your filter that was consuming chlorine, it would get consumed in the same way whether it went 'straight to the filter' via the skimmer, or more diluted via the rest of the water. Hope that makes sense.
Is there any reason you add it to the skimmer? My understanding is that the reason this isn't recommended is only because then it exposes your equipment to a higher concentration of chlorine than if it had been diluted in the pool first. Maybe someone else can expand on that, but that's my understanding.
As mentioned, the average pool looses 2 to 4 ppm per day, depending on weather and bather load, so it sounds like you're right in line there (especially since you say it's consistent, not a sudden jump in consumption).
What is it about that amount of demand that concerns you?