Protocol is just specification about networking between several nodes. I usually make protocol over HTTP. Maybe many developers working on web do this everyday. However, it's hard to make good documentation about protocol. For the program source code, it's easy because there are so many commentary documentation tools, and the program structure itself is very good documentation framework. However how should I approach about protocols? There is no base language.
How do you document your protocol?
Have a document specifying what are the acceptable messages, and which fields within the messages may vary.
Then, for each message, have an example reply, also indicating the structure of the returned message, and the portions of the message that my vary. If more than one reply for a message is possible, indicate all the acceptable replies and the conditions under which each one can be expected to be received.
In each "Varying" section of the message, indicate whether it can be returned blank, with one "answer", or with multiple "answers". If you use a "special" character to separate out multiple answers, specify how a message would be able to "embed" that special character into one answer (without it trigging code that would split the "one" answer into multiple answers).
If your messages indicate a start and stop sequence, prove in your protocol that the start and stop sequence cannot be part of the transmitted "body" of the message. This is typically done by "escaping" the start / stop sequences in such a manner that they can be transmitted safely without prematurely restarting / terminating the message.
It's a slow process. If your messages are text, then document the items as acceptable text formats. If the messages are binary, then document the items as acceptable orderings of bytes (as bytes are the smallest element that you'll likely transmit).