Exactly. But sadly you're in a tiny minority of people thinking about the security implications. Most developers don't: which is why we're having countless OAuth exploits and whatnots.

Schneier wrote a long time that anything too complex cannot ever be secure. That's the case of OAuth and of many federated/unique/single sign-on logins.