The next sentence in the paper says: "Just as a bottle can contain between zero and one pint, so a binary digit (when averaged over both of its possible states) can convey between zero and one bit of information."

These definitions are for information theory.

A "binary digit" is either a zero or a one. It's a specific value.

A "bit" is an amount of information: a container that can hold a binary digit.

That may very well be the definition. People certainly do say e.g. "... a 0 bit followed by a 1 bit ...", however.