Let me explain.
For example, you have a site, site.com, and on the domain provider service, you decide you want to have your own nameservers, and enter the new addresses.
You decide to let these nameservers have domain names, ns.site.com.
Now what happens if a client wants to resolve my.site.com, as far as i understand it, it (client and/or recursive DNS servers) will ask for authority for the second level domain record of site.com, pick the NS name from it, and resolve that to contact it.
Now we have set the nameserver to ns.site.com, which lays in it's own domain... wait, that's recursive.
This is exactly what I think could be a problem, how is it solved in the real world? Does/did this ever happen?