I'm evaluating the Nexenta platform to hopefully one day replace our legacy file servers. I would primarily use it as a CIFS server in an Active Directory environment.
I can't comment on Nexenta's AD/CIFS implementation as I've just started using that feature, but we have been using it as our primary storage for just over a year now, with an appliance here in the office and one in the datacenter. I assume you're talking about the NexentaStor appliance platform, and not the Nexenta distribution.
1 - It has been great to use, though we are primarily using it for NFS. Depending on the system configuration, it is very fast and reliable. In our (crappy) office, we have had quite a few power outages, and it never skips a step. We use it with both OracleVM (Xen) and VMware esx, it has been great for both. The only caveat, at least with NFS, is to be sure to have dns setup and fully populated with all your servers.
2 - I would recommend going through a vendor, having both built our own (in the office) and purchased from a reseller (PogoLinux appliance in the datacenter). Everything works fine on the one we spec'ed out ourselves, but performance could be better, and we forgot a couple things up front. We didn't spec out OOBM on our home-built one, so that has bit us a couple times and necessitated a couple late night drives to the office. We also chose a storage controller (sata JBOD) that I've since heard is very slow compared to other stuff out there. Our experience with PogoLinux has been great, the solution we created them has been top-notch performance wise, and their support is a great addition to normal nexenta support as they have a lot of hands on experience with the hardware, software, and storage implementations in general. Price-wise, it was very close to building it ourselves, so the extra 2% or so we spent with PogoLinux has been well worth it.
Sadly there is a massive stack of Kerberos and AD bugs in opensolaris (and nexenta) that are plaguing MS AD users. I've been cataloging mine since Dec 09. I wouldn't count on using CIFS reliably with AD at this point.
I was quite impressed with how well nexanta fulfilled it's primary purpose, pulling in data from other systems and snapshotting it. I have gotten extremely high dedup/compress rate, 3.67x with .5T of data. Any linux/bsd system will lag behind on zfs fixes compared to a opensolaris version.
I for one without knowing much about the cifs in opensolaris would trust it as a samba server.
Of course one way to get around this is to use a separate device for the L2ARC. As for stability and production readiness - http://www.informationweek.com/news/software/open_source/showArticle.jhtml?articleID=227600191&subSection=Hosted+Software - ultimately will be 1PB - mission critical stuff.
Both Illumos and OpenIndiana will provide a solid base for the future of OpenSolaris I think. It maybe that they merge at some point to pool their resources.
I know Nexenta are hiring CIFS chaps from Redmond to work on getting that side of things tip-top though it works well already. Personal experience here.
I tried it in a VM and it seemed ok, however do be aware that as Oracle (Sun) have now (essentially) killed OpenSolaris this may spell the end for Nexenta.
An alternate ZFS serving platform would be FreeBSD.
This is old but I don't think it has been adequately answered. The CIFS integration with ZFS is terrible. I know of no one using it in a production environment, it is just broken. ACLs get horribly mangled and it is impossible to keep them managed.
Samba, OTOH, is fine - or at least, far more mature.
I'm not pumping my own blog (I get about one visitor per month and don't care), but I already went into some more detail about it:
You might want to consider the 3+ years bug with ZFS ARC that still persists before jumping in too deep with ZFS...
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6522017
(This one is nasty as it will also go out-of-bounds from the VM limits of a hypervisor!)
