I've had many over the years, some more successful than others. Essentially, you have to treat it like a garden, and constantly tend to it
The Nitrogen cycle is fairly straight forward. Fish Eat, waste is produced, and breaks down into Ammonia. The ammonia in turn is eaten by nitrifying bacteria, which excrete Nitrites (NO2), and then by other bacteria in to Nitrates (NO3). Plants and Algae like NO3, and consume it in to grow. Most people experience this as Algae...your tank goes green, because you're not getting rid of the Nitrates...which is why you are encouraged to clean the tank, vacuum up the poop, and change the water regularly (thereby diluting the nitrates again).
Generally, the biggest mistake made is too many fish, and not enough plants. You need a LOT of plants, and a lot of light for them to grow, which will encourage them to consume the Nitrates being produced by even a few fish. A good biological filter to encourage nitrification from Ammonia to Nitrate is important, and then a good substrate to supplement the plants. Pennywort, Hornwort and Wisteria, all work well to consume the Nitrates. Some species of hairgrass can grow an inch or more in a day. Add in a few 'turners' like Corys and Loaches, even some species of shrimp...
You can always ADD nitrates by adding a little extra food or organic material to break down if your plants seem to be slowing in growth..it's harder to make the plants grow faster to consume more Nitrate
Of course, the next issue is then Fish like to eat the plants...
One thing to consider is a 'refugium' setup, where you have a tank under the main 'display' tank, but lit on the opposite schedule, and that tank has plants in it that are regularly harvested, but are 'isolated' away from your fish so that they don't all just get eaten and turned back into poop

Stuff it full of Hairgrass and Moneywort/Hornwort, and then use Hornwort/Java Fern in the main tank, and you have a good start.
I found some pictures of when I last did this - I don't have any with it full of plants, but it filtered my reef tank for a couple years very effectively.

OMG I want a tank again...NOOOOOooooooooo !!!