I believe you should only invite people you want to socialize with. Regardless of religion, race, or sexual preference, invite people you want to hang out with. It will make everyone’s life easier. Don’t feel bad about it. It is your property after all.