Mike Sax mentions the ridiculous fees for setting up Passport to use for authentication into your site. We looked into this for a customer at one point and found it pretty much prohibitive. It would have fit quite well for this customers target market but it just was not feasible for this company who had a few hundred customers/trading partners. Sure helps to understand why you see so few passport enabled sites.