mirror of
https://github.com/tailscale/tailscale.git
synced 2026-09-14 06:48:32 -04:00
Joe Tsai
d2c5166298
Package cobs implements Consistent Overhead Byte Stuffing (COBS), a technique for reliable packet framing over serial byte streams. This has future utility for storing a sequence of arbitrary log entries on disk without needing to depend on intrinsic framing within the log entries themselves (e.g., JSON or CBOR). While more complicated, COBS is superior to offset-based framing mechanisms as the null byte can be trivially used to demarcate the boundaries of a frame. This makes COBS more resistant against bit-corruption where a single corrupted offset can make everything else in the file unreadable. COBS makes it possible to resynchronize framing after a corrupted section by simply searching for the next null. Performance: Benchmark/EncodeForward/Zeros-32 16341 76312 ns/op 13740.68 MB/s 0 B/op 0 allocs/op Benchmark/EncodeReverse/Zeros-32 6326 188261 ns/op 5569.79 MB/s 0 B/op 0 allocs/op Benchmark/DecodeForward/Zeros-32 16461 72140 ns/op 14535.28 MB/s 0 B/op 0 allocs/op Benchmark/EncodeForward/NonZeros-32 41797 29155 ns/op 35965.56 MB/s 0 B/op 0 allocs/op Benchmark/EncodeReverse/NonZeros-32 4792 248788 ns/op 4214.74 MB/s 0 B/op 0 allocs/op Benchmark/DecodeForward/NonZeros-32 35790 34584 ns/op 30319.92 MB/s 0 B/op 0 allocs/op Benchmark/EncodeForward/Random-32 23042 53727 ns/op 19516.64 MB/s 0 B/op 0 allocs/op Benchmark/EncodeReverse/Random-32 3164 374590 ns/op 2799.26 MB/s 0 B/op 0 allocs/op Benchmark/DecodeForward/Random-32 27241 58506 ns/op 17922.41 MB/s 0 B/op 0 allocs/op EncodeReverse performance is notably slower than EncodeForward because modern CPU architectures are not as optimized for reading from memory in reverse. However, reverse encoding is necessary if appending into a dst buffer that is identical to the src buffer. In such a case, the CPU performance hit is worth the benefit of avoiding an intermediate allocation. Speeds of GB/s is still plenty fast enough and magnitudes faster than JSON or CBOR encoding. Updates #17242 Updates tailscale/corp#21363 Signed-off-by: Joe Tsai <joetsai@digital-static.net>
63 lines
1.7 KiB
Go
63 lines
1.7 KiB
Go
// Copyright (c) Tailscale Inc & contributors
|
|
// SPDX-License-Identifier: BSD-3-Clause
|
|
|
|
package cobs
|
|
|
|
import (
|
|
"bytes"
|
|
"fmt"
|
|
|
|
"tailscale.com/util/must"
|
|
)
|
|
|
|
func Example() {
|
|
// Frames is a list of frames that may contain null bytes.
|
|
// Trivially joining the frames with a null byte would lead to
|
|
// ambiguity when parsing as null bytes within the frame itself
|
|
// cannot be distinguished from nulls used to mark frame boundaries.
|
|
frames := [][]byte{
|
|
[]byte("Hello world!"),
|
|
[]byte(""),
|
|
[]byte("\x00\x00\x00"),
|
|
[]byte("Fizz\x00Buzz"),
|
|
}
|
|
|
|
// COBS encoding ensures that each frame never contains null bytes.
|
|
for i, frame := range frames {
|
|
frames[i] = AppendEncode(frame[:0], frame)
|
|
}
|
|
|
|
// Since the COBS-encoded frame lacks null bytes,
|
|
// we can trivially join the frames together using a null byte.
|
|
stream := bytes.Join(frames, []byte("\x00"))
|
|
|
|
// Print out the COBS-encoded stream.
|
|
fmt.Printf("COBS-encoded stream:\n\t%q\n\n", stream)
|
|
|
|
// When decoding, the frame boundaries can be trivially detected
|
|
// by splitting upon the null byte.
|
|
frames = bytes.Split(stream, []byte("\x00"))
|
|
|
|
// However, each individual frame is still COBS-encoded,
|
|
// so we need to decode each one back to the original frame payload.
|
|
for i, frame := range frames {
|
|
frames[i] = must.Get(AppendDecode(frame[:0], frame))
|
|
}
|
|
|
|
// Print out each COBS-decoded frame to verify that it matches.
|
|
fmt.Println("COBS-decoded frames:")
|
|
for _, frame := range frames {
|
|
fmt.Printf("\t%q\n", frame)
|
|
}
|
|
|
|
// Output:
|
|
// COBS-encoded stream:
|
|
// "\rHello world!\x00\x01\x00\x01\x01\x01\x01\x00\x05Fizz\x05Buzz"
|
|
//
|
|
// COBS-decoded frames:
|
|
// "Hello world!"
|
|
// ""
|
|
// "\x00\x00\x00"
|
|
// "Fizz\x00Buzz"
|
|
}
|