As FPM noted, if you don't use any CYA at all and use an FC level of 2-3 ppm as you suggest, then compared to the 4 ppm FC with 20 ppm CYA that we recommend you've got 10-20 times as much active chlorine (hypochlorous acid) in the pool which will oxidize swimsuits, skin and hair up to 10-20 times faster and also likely produce about 10-20 times as much irritating and volatile nitrogen trichloride. The significant reduction in active chlorine when CYA is present is not a bad thing unless overdone. Essentially, pools with no CYA are over-chlorinated unless they have very low FC levels which is what the German DIN 19643 standard tries to achieve (but such low FC levels could "run out" of chlorine under local bather load conditions).
The downside to the lower active chlorine level is that it takes longer to oxidize bather waste and when there is no UV in sunlight to help you it's possible to build up more CC, at least temporarily. Indoor pools are a challenge and often need supplemental oxidation (UV or ozone) unless the bather loads are very light and this is true even when no CYA is used.