I had a play with something similar and I noticed that -- at least for my use case -- you can get ~90% of the gains through just a couple of simple tricks:
1) Identify medium-scale similarity boundaries in the data structures. E.g.: a sequence of messages in a protocol, such as a C "struct" with a bunch of fields.
2) Compute the binary difference between these structures so that most of the subsequent bytes after the first message are either zeroes or small numbers. Both the sender and receiver have to keep the previous message in a buffer to allow this.
3) Use a high-performance compression algorithm that supports "user provided dictionaries", such as Zstandard. Train it with sample data.
This above is surprisingly straightforward because it doesn't require complex changes to the underlying data structures. You don't even necessarily need to be able to parse it at all, as long as it has large-scale repeating structures that you can identify.
Agreed. It really blows generic compression algos out of the water. Didn't know about Zstandard dictionary encoding. This might just be what I'm after. Thanks
1) Identify medium-scale similarity boundaries in the data structures. E.g.: a sequence of messages in a protocol, such as a C "struct" with a bunch of fields.
2) Compute the binary difference between these structures so that most of the subsequent bytes after the first message are either zeroes or small numbers. Both the sender and receiver have to keep the previous message in a buffer to allow this.
3) Use a high-performance compression algorithm that supports "user provided dictionaries", such as Zstandard. Train it with sample data.
This above is surprisingly straightforward because it doesn't require complex changes to the underlying data structures. You don't even necessarily need to be able to parse it at all, as long as it has large-scale repeating structures that you can identify.