Are custom DSLs in JavaScript still viable, Specifically ones written with Antler, Boo, or the former Microsoft Oslo?
With the proliferation of open source JavaScript/ECMAScript engines (V8, IronJS, etc.), does it make more sense to write the functionality needed into one of those instead of writing an entirely new language?
If you mean DSL in strict terms, as in a programming language spec restricted to only one domain, then yes. The proliferation of the JavaScript/ECMAScript engines actually increases their viability - given a common virtual machine which is as flexible as the ECMAScript ones tend to be, DSL are both much easier to create, and likelier to be encountered.
What you won't see as much of, though, are languages with their own dedicated environments. At the high end, you'll see compilers to the existing languages (ala the Java to Javascript compiler that Google put out). Much more commonly, you'll see a library of supporting functions coupled with a programming language/style enforced by api restrictions and convention (the api to the jQuery library could be called a DSL).
Custom DSL's are still viable, but you have other options which could be more or less complex depending on the problem you are trying to solve. For example, you can implement DSLs using dynamic languages. (DSLs using Groovy is one example that comes to mind) Which can then be compiled or translated it yet other languages. This can have a lower upfront cost, but may you might find that it is not the right fit in your case.
Perhaps you should start with defining the domain in question, and put together a list of things you would need from a DSL.
DSLs are rarely built as "entirely new" languages. Most of the practical DSLs are built on top of the existing languages - e.g., jQuery is a DSL on top of Javascript. And those DSLs are not necessarily embedded DSLs - they can be standalone as well, but compiled into a combination of the existing languages instead of going all the way down to the low level.
