I’ve been working on a compression algorithm for fast random access to individual strings in large collections.The problem came up when working with large in-memory database columns (emails, URLs, product titles, etc.), where low-latency point queries are essential. With short strings, LZ77-based compressors don’t perform well. Block compression helps, but block size forces a trade-off between ratio and access speed.Some existing options:- BPE: good ratios, but slow and memory-heavy- FSST (discussed here: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=41489047">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=41489047</a>): very fast, but weaker compressionThis solution provides an interesting balance (more details in the paper):- Compression ratio: similar to BPE- Compression speed: 100–200 MiB&#x2F;s- Decompression speed: 6–7 GiB&#x2F;sI’d love to hear your thoughts — whether it’s workloads you think this could help with, ideas for API improvements, or just general discussion. Always happy to chat here on HN or by email.---Resources:- Paper: <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2508.02280" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2508.02280</a>- Rust: <a href="https:&#x2F;&#x2F;github.com&#x2F;gargiulofrancesco&#x2F;onpair_rs" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;gargiulofrancesco&#x2F;onpair_rs</a>- C++: <a href="https:&#x2F;&#x2F;github.com&#x2F;gargiulofrancesco&#x2F;onpair_cpp" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;gargiulofrancesco&#x2F;onpair_cpp</a>

Show HN: OnPair – String compression with fast random access (Rust, C++)