I've not worked with OS X Server yet, but I believe it uses the same kernel as the desktop version.

Mac OS X does support IPSec. Because of its BSD roots, it inherits the KAME project. However, it may not have a nice GUI to configure it and you'll have to do it from the command line like all the rest of the *BSDs

Check out the articles that start with "Flying Racoons:" at http://www.afp548.com/Articles/