A protocol is a set of code-based rules and procedures that define how a blockchain works.
For cryptocurrencies, the protocol layer is extremely important because the protocol defines how the blockchain manages, validates, and confirms data. Additionally — and this is the novel part of crypto-specific protocols — they dictate how the underlying economics of the system including how much cryptocurrency will be produced and for how long.
The word protocol is used frequently to describe the base layer of a crypto-based system: Bitcoin is a protocol, and Ethereum is a protocol, etc. New protocols are usually defined by a white paper, which is a document outlining how the system will work.
Since crypto protocols are open source and designed to be decentralized, often developers will fork or take the existing protocol code and modify it to behave differently, creating a new crypto protocol in the process. Litecoin, for example, is a fork of Bitcoin.
While protocol is a term thrown around frequently in crypto, it's actually not that unfamiliar. In fact, the entire internet is built on a series of protocols that outline rules for how data is handled and found on the internet. Email too, is based on protocols.