> author decided it was a good idea to prepend the message with the message length encoded as a varint.
> WHY? Oh, why?!
Uh oh. Is this my HN moment?
This is exactly how I implemented it at my company. We had to write many protobuf messages to one file in bulk (in parallel). I did a fair amount of research before designing this and didn’t find any standard for separating protobuf messages (in fact, found that there explicitly isn’t a standard in that protobuf doesn’t care). So I thought rather than using some “special” control character, like a null byte, which would inevitably be not-so-special and collide with somebody else’s (like Schema Registry’s “magic byte”), I’d use something meaningful like the number of bytes the following record is.
As for why I chose varint instead of just picking an interger size, well for one I got nerd-sniped by varint encoding and thought it would be cool to try and implement it in Scala. Secondly, I thought if I chose a fixed size integer, no matter what size I pick, my users will always surprise me and exceed it at least once, and when that happens, kaboom! I wanted to future proof this without wasting 64 goddamn bytes in front of each message, and also I got nerd-sniped, OK?!?
Someone on my team recently shared one of these files outside the company and so I really hope she’s not talking about me but that’s a crazy coincidence if not!
- Riegeli is "a file format for storing a sequence of string records, typically serialized protocol buffers. It supports dense compression, fast decoding, seeking, detection and optional skipping of data corruption, filtering of proto message fields for even faster decoding, and parallel encoding": https://github.com/google/riegeli
Create a MessagePack message containing a byte array, put the protobuf in that. Of course, the enterprise solution would be to hex encode it and put it in a CDATA section in an XML document.
You got me. What's that format / magic number? At first I thought it was the zlib header for incompressible data, but that's not right. Is that just the MAC address of some NIC you own?
> “I wanted to future proof this without wasting 64 goddamn bytes in front of each message”
64 bytes would be a 512-bit integer. That seems like excessive future proofing for the length of any message that would be transmitted before the Sun runs out of fuel.
I wonder whether there’s an upper bound on the largest number that can be expressed in the observable universe.
All digital representations rely on discrete states in hardware, and there’s a finite number of those in the observable universe, so there should a finite maximum number for computers.
I'll have to think about this. I want a very physical meaning of the term using values rather than references. One where the largest number that can be expressed using up/down fingers with two hands is 1024, not a sign language reference to a googolplex.
I wonder if it would be helpful to restrict the question a bit, maybe something like: what is the largest number for which the number, and also all smaller magnitude integers, can be expressed.
Just as a quick example of why it's a bit absurd - I name that number you just defined $zeta$. Now I make $zeta'$ = zeta^zeta. Or whatever manipulation you like.
Adding constraints is addressed in the link.
And zeta' can not be expressed by any state of the visible Universe.
The GP question was not about encoding, and thus is not subject to compression. The largest number we can measure of anything is a pretty well defined concept.
I suppose pi has a cheat code: if you made the largest possible circle in the universe you only need enough digits to distinguish points on its circumference that are a Planck distance apart. Then you can either ignore any digits beyond that or even simply make them up, as there would be no way to measure the difference.
A related point of fact is protobufs are limited to 2G anyway. You wouldn’t need 64 bits. Anything bigger than about 100MB is dangerously large anyway.
You can also define a wrapper message with a single repeated field. The resulting encoding would be `varint(field id * 8 + 1) varint(length)` plus the actual message, so it can be also easily generated from and parsed back into a raw byte sequence without protoc.
This approach is problematic because, as far as I know, all implementations expect to deserialize from a complete buffer -- at least all official implementations. That means the entire "stream" would have to be buffered in memory.
You can, of course, simply read field by field, but few libraries expose the ability to do that simply. And the naive operation then becomes quite problematic indeed.
This is the same as it's implemented in ClickHouse. In fact, it has two formats: `Protobuf` (with length-delimited messages) and `ProtobufSingle` (no delimiters, but only a single message can be read/written). And it is fairly common:
Conforming implementations will fail to interoperate with their “protobuf” serialization; it’s absolutely incorrect for them to call their length-prefixed framing protobuf.
Using a delimiter to mark end of packet. For text protocols we usually use "\n" to be the delimiter. This usually requires some escaping in the packet to make it unambiguous. Two standard protocols for this are SLIP and HDLC.
Length encoding - like what you did.
The downside of the delimiter approach is that it changes the length of the packet - when you are escaping one byte becomes two and that's sometimes a pain if you're doing it in place in memory (less of a problem if you're streaming byte by byte). The big advantage is that it allows for resynchronisation - if you lose a single byte from your stream, or your length byte ever gets corrupted then you're permanently out of sync - the receiver will never again know where the start or end of a packet is. With the delimiter approach, you just lose one packet. So if you're ever doing this for a UART or network stream or something, always do the delimiter approach!
> So if you're ever doing this for a UART or network stream or something, always do the delimiter approach!
I'm confused. I mean for UART, sure. But network streams is usually sent over a protocol that recovers lost data. Am I reckless for sending length-prefixed data chunks over TCP?
No, you're right that this approach is ok for TCP streams. I shouldn't have thrown that in there without clarification that I meant something more generic.
I actually use SLIP for packetisation over TCP anyway, because then syncing for any other logging or diagnostic thing that joins halfway through is easy. Basically I find it to just be a more robust system.
TCP is a stream with no packet boundaries. If you want to separate that stream into a series of messages then you need to do packetisation. The SLIP delimiter approach is a pretty good one.
You're right that it was originally for segmenting a serial link to send separate IP packets (so you could theoretically have SLIP over TCP over IP over SLIP), but it works just as well in other contexts.
Oh who hasn’t written some LengthPrefixedProtobufRecord class. I get why Rachel would be annoyed if someone claimed it was protobuf, but even then it’s like, 8 seconds in xxd to see what’s going on.
I shudder to think how well shit must be going for this to merit a Rachel post.
Why not just define a higher level proto that contains all possible (maybe repeated) protos you might want to include? Then if one of the included protos is not present, the higher level proto will efficiently encode that, and nothing gets broken.
If you want to future proof it you need to version it. It sounds like you're trying to pack many things into a single file, so having the first few bits of the file represent a version allows you to use fixed length integers without fear of them being too small (in the future). You can reserve the "last" version for varint if you truly need it.
In general, I find adding versions to things allows for much more graceful future redesigns and that is, IMO, invaluable if you're concerned about longevity and are not confident in your ability to perfectly design something for the indefinite future.
I don't see the point. As soon as you bump the version number, old versions of the software will refuse to read newer files. So you might as well use a new format and file extension for the newer files. Of course the new software will read both formats.
The way protobufs handles versioning (old software ignores unknown fields) is far superior and realistically everyone uses 64 bit fixed length sizes everywhere
If you prefer to ship fragile binary file formats for no reason other than finding it too onerous to define something as trivial as an eight-byte file header, that’s silly, but nobody is going to stop you.
If you’re going to use text-based serialization formats as your justification for the decision, however, I’d suggest you look into all the fun bugs, security issues, and weird edges cases that arise from parsers having to make a best guess at character encoding and file format when all you have to work with is the file extension, maybe a byte order mark, and heuristics over the file contents.
Your length-prefixed message frames represent a distinct serialization scheme that provides external framing for embedded messages; it’s a standalone format, and you can use whatever frame header you want without worrying about collision.
The only issue is if you were to ship a “protobuf” library that emits/consumes your (very much not protobuf) framing format.
Also, a 64 bit frame length would only be 8 bytes, not 64 :-)
> WHY? Oh, why?!
Uh oh. Is this my HN moment?
This is exactly how I implemented it at my company. We had to write many protobuf messages to one file in bulk (in parallel). I did a fair amount of research before designing this and didn’t find any standard for separating protobuf messages (in fact, found that there explicitly isn’t a standard in that protobuf doesn’t care). So I thought rather than using some “special” control character, like a null byte, which would inevitably be not-so-special and collide with somebody else’s (like Schema Registry’s “magic byte”), I’d use something meaningful like the number of bytes the following record is.
As for why I chose varint instead of just picking an interger size, well for one I got nerd-sniped by varint encoding and thought it would be cool to try and implement it in Scala. Secondly, I thought if I chose a fixed size integer, no matter what size I pick, my users will always surprise me and exceed it at least once, and when that happens, kaboom! I wanted to future proof this without wasting 64 goddamn bytes in front of each message, and also I got nerd-sniped, OK?!?
Someone on my team recently shared one of these files outside the company and so I really hope she’s not talking about me but that’s a crazy coincidence if not!