Out of Character (OOC) is when a person talks on behalf of himself, and not is character.
Generally, all forms of communication aside from /say (regular speech) and /yell (shouting to an extended distance) are judged as OOC unless specified otherwise. This is due to the fact that other methods such as chat channels and whispering are unrealistic and can be used across any distances without the precense of characters by each other.
Chat channels are subject to different rules relative to the RP Realm Policy, which basically states that all things spoken about in General chat should be related to the game. This is the case most of the time as it is, as topics not pertaining to the game in any way rarely arise in this channel.
Whispering and guild chat are seemingly uncontrolled forms of communication and thus remain OOC for the most part. Whispering is OOC most of the time, unless a character decides to whisper something to another. In this case, it is more logical to add an "IC:" prefix rather than adding "OOC" or "(())" to every OOC phrase in /w communication.
Different guild masters control their guild chats differently. For example, some have regular (/g) guild chat channel activity pseudo-in-character (RP conversations between characters without their presence near each other) and other guild channels such as the officers' channel (/o) for OOC discussions. However, this is completely unique to each guild and just like most of the above stated concepts, remains a mere suggestion rather than any policy-driven strict rule.
Party chat is strictly limited to in-character interaction by the RP Realm Policy as well, however it is most common practice to use party chat as out of character. This is indeed against the rules but serves a purpose of its own and is easily explainable for being OOC for the same reason as above: lack of realism (no others around can hear members of the party speaking to each other, it can be used across any distances, etc.).