You are fine.
You are correct in thinking that if you can't hold your FC easily that there is something going on in your pool causing it to work which breaks down the chlorine into CC.
I would SLAM and look for any nooks and crannies where any dirt or algae could be hiding. Steps, lights...