Yes, ForumNG does this. There were two approaches:
- Prior to 2.3, I used some small (but ugly) JavaScript hacks to include it using AJAX, like you're trying to do.
- In 2.3 I came to my senses and changed it to use an iframe.
Getting this to work in AJAX is really tricky - it's vital to remember it's not just the actual editor but also the filemanager for the editor that has to work. This is basically a nightmare and requires core code changes or horrible hacks that will be fragile and break with each release.
For ForumNG, making it work in an iframe was simple because this is done for the 'Reply' feature which is basically a self-contained form anyway. I just made a script output a page with just the form in (and suitable styling) and added this in an iframe. There's then a JavaScript event handler that runs after the form (in the iframe) is submitted, to process the results and remove the iframe again.
Because of the way the file manager works you currently have to make a round-trip php request to server when you init the editor anyway, so it doesn't even make it too much slower to do it the iframe way.
The iframe approach might not work so well when it's part of a form rather than a self-contained form that you just want to appear via AJAX, but it might well still be possible to use it that way and I would recommend attempting that rather than trying to subvert the filemanager and editor JS to actually load it in AJAX. (Of course, if there is an official way to do it, this might become feasible.)
--sam