Since everyone has highlighted a lot of the good stuff, I'll play devil's advocate here. I've spent a year developing Facebook apps, so I know a little about this stuff.
Fast Pace
Facebook moves very fast when it comes to making changes and obsoleting things. What works today might not work tomorrow, and most certainly won't work in a year. So, unless you keep tabs on what they're doing and what has changed recently, your FB Connect login may break silently and new users won't be able to sign up. It's not impossible to keep up, but it's yet another thing you have to check every now and then to make sure it's still working. They might be more careful with FB Connect than they are with the app platform, but who knows?
No True Emails
Someone please correct me if I'm wrong, but I don't believe you get a true email address for FB Connect users. Instead, you get an FB proxy email. This means that Facebook will always sit between you and that user when you try to communicate. Having such a giant, uncaring monolith between me and my users makes me uncomfortable.
TOS Quicksand
Facebook has no fear of modifying their terms of service, sometimes drastically. So, even if you read all the FB Connect fine print, there's no guarantee that it will still apply in a week or a month. You could sign up ten thousand FB users, then have FB change their TOS in some way that makes your entire site a violation, then they shut off your access to FB Connect, stranding you. Granted, it probably won't be this drastic, but just look at the Beacon debacle. A lot of 3rd party code immediately stopped working when they (thankfully) shut Beacon off.
Now I'm not trying to say that you shouldn't use FB Connect. I'm just trying to emphasize that FB Connect is not all kittens and unicorns. Remember: Facebook has their own agenda and will continue to pursue it, regardless of the positive or negative impact it has on your particular site.