msgpck

package module
v0.1.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 20, 2026 License: MIT Imports: 8 Imported by: 3

README

msgpck

CI Coverage Status Go Report Card Go Reference Quality Gate Status Vulnerabilities

A high-performance msgpack library for Go, optimized for database and serialization-heavy workloads. Zero external dependencies.

Why Another Msgpack Library?

Gives how standards proliferate vibes. s/standards/msgpack libraries/g

I built msgpck for tinykvs, a key-value store where msgpack encoding/decoding is on the hot path. I found issues with fuzz testing where vmihailenco would allocate excessively on big maps/arrays, and decided to check out the alternatives: shamaton/msgpack, hashicorp/go-msgpack, and tinylib/msgp. I didn't want code generation so tinylib/msgp was out. hashicorp/go-msgpack was slow. shamaton/msgpack was better but didn't compete with vmihailenco/msgpack on map[string]any performance. So I built msgpck focused on the common case: decoding known struct types and map[string]any with minimal allocations.

Performance

Benchmarks vs vmihailenco/msgpack (Apple M3 Max):

Struct Operations (using cached GetStructEncoder/GetStructDecoder)
Operation vmihailenco msgpck Speedup
SmallStruct Encode 137 ns, 3 allocs 32 ns, 0 allocs 4.3x
MediumStruct Encode 431 ns, 5 allocs 157 ns, 0 allocs 2.7x
SmallStruct Decode 167 ns, 3 allocs 40 ns, 1 alloc 4.2x
SmallStruct Decode (zeroCopy: true) - 29 ns, 0 allocs 5.8x
MediumStruct Decode 671 ns, 14 allocs 324 ns, 12 allocs 2.1x
MediumStruct Decode (zeroCopy: true) - 230 ns, 3 allocs 2.9x
Map Operations
Operation vmihailenco msgpck Speedup
SmallMap Encode (Marshal) 127 ns, 2 allocs 62 ns, 0 allocs 2.0x
MediumMap Encode (Marshal) 491 ns, 4 allocs 193 ns, 0 allocs 2.5x
SmallMap Decode (UnmarshalMapStringAny) 201 ns, 8 allocs 107 ns, 3 allocs 1.9x
MediumMap Decode (UnmarshalMapStringAny) 810 ns, 34 allocs 392 ns, 15 allocs 2.1x
StringMap Decode (UnmarshalMapStringString) 305 ns, 12 allocs 114 ns, 2 allocs 2.7x

Run benchmarks yourself:

go test -bench=. -benchmem

Quick Start

import "github.com/freeeve/msgpck"

// Encode any value
data, _ := msgpck.MarshalCopy(map[string]any{"name": "Alice", "age": 30})

// Decode to map
m, _ := msgpck.UnmarshalMapStringAny(data, false)

// Decode to struct
var user User
msgpck.UnmarshalStruct(data, &user)

Key Features

Cached Struct Codecs

For hot paths, use cached codecs that avoid reflection on every call:

// Get cached encoder/decoder (created on first use, reused forever)
enc := msgpck.GetStructEncoder[User]()
dec := msgpck.GetStructDecoder[User](false)

// Encode - 0 allocations with pooled buffer
data, _ := enc.Encode(&user)

// Decode - minimal allocations
var user User
dec.Decode(data, &user)
Zero-Copy Mode

When your input buffer outlives the decoded result (common in databases), skip string allocations entirely:

// Get cached zero-copy decoder
dec := msgpck.GetStructDecoder[User](true)

// Strings point directly into 'data' - no copies
dec.Decode(data, &user)

Warning: Zero-copy strings are only valid while the input buffer exists. Use the callback API when buffer lifetime is uncertain:

// Safe zero-copy: strings valid only within callback
msgpck.DecodeStructFunc(data, func(user *User) error {
    // Use user.Name here - it points into data
    processUser(user)
    return nil
})
// After callback returns, data can be reused safely
Typed Map Decoding

When you know your map types, avoid any boxing overhead:

// map[string]any - general purpose
m, _ := msgpck.UnmarshalMapStringAny(data, false)

// map[string]string - faster when you know the type
m, _ := msgpck.UnmarshalMapStringString(data, false)

// Zero-copy variants (strings point into input buffer)
m, _ := msgpck.UnmarshalMapStringAny(data, true)
m, _ := msgpck.UnmarshalMapStringString(data, true)

API Reference

Encoding
// Encode any Go value to msgpack
msgpck.Marshal(v any) ([]byte, error)       // pooled buffer - don't retain
msgpck.MarshalCopy(v any) ([]byte, error)   // safe to retain

// For hot paths: cached struct encoder
enc := msgpck.GetStructEncoder[MyStruct]()
enc.Encode(&src)      // pooled buffer
enc.EncodeCopy(&src)  // safe to retain
Decoding
// General decoding
msgpck.Unmarshal(data []byte) (any, error)

// Map decoding
msgpck.UnmarshalMapStringAny(data []byte, zeroCopy bool) (map[string]any, error)
msgpck.UnmarshalMapStringString(data []byte, zeroCopy bool) (map[string]string, error)

// Struct decoding (reflection-based, works with any struct)
msgpck.UnmarshalStruct(data []byte, dst any) error

// For hot paths: cached struct decoder
dec := msgpck.GetStructDecoder[MyStruct](false)
dec.Decode(data, &dst)

// Zero-copy cached decoder
dec := msgpck.GetStructDecoder[MyStruct](true)
dec.Decode(data, &dst)
Callback APIs (Safe Zero-Copy)
// Buffer guaranteed valid during callback - safest for zero-copy
msgpck.DecodeStructFunc(data, func(v *MyStruct) error { ... })
msgpck.DecodeMapFunc(data, func(m map[string]any) error { ... })
msgpck.DecodeStringMapFunc(data, func(m map[string]string) error { ... })
Timestamps

msgpck supports the msgpack timestamp extension type (-1). Times are encoded using the most compact format and decoded to UTC:

// Encode a time.Time
data := msgpck.MarshalTimestamp(time.Now())

// Decode back to time.Time (UTC)
t, _ := msgpck.UnmarshalTimestamp(data)

// Streaming API
enc := msgpck.NewEncoder(nil)
enc.EncodeTimestamp(time.Now())

dec := msgpck.NewDecoder(data)
t, _ := dec.DecodeTimestamp()

// Convert extension values
ext, _ := dec.DecodeExt()
if msgpck.IsTimestamp(ext) {
    t, _ := msgpck.ExtToTimestamp(ext)
}

Concurrency

All public APIs are concurrent-safe:

  • Marshal, MarshalCopy, Unmarshal* functions use internal pools
  • GetStructEncoder[T](), GetStructDecoder[T](zeroCopy) return cached, thread-safe codecs
  • StructEncoder and StructDecoder instances are safe to use from multiple goroutines

When to Use msgpck vs vmihailenco/msgpack

Use msgpck when:

  • Encoding/decoding is on your hot path
  • You decode the same struct types repeatedly
  • You can benefit from zero-copy (database, network buffers)
  • You need minimal allocations

Use vmihailenco/msgpack when:

  • You need custom encoders/decoders for complex types
  • You're decoding unknown/dynamic schemas
  • Convenience matters more than raw speed

Compatibility

msgpck produces standard msgpack bytes. Data encoded with vmihailenco/msgpack decodes correctly with msgpck and vice versa.

License

MIT

Documentation

Index

Constants

View Source 
const (
	DefaultMaxStringLen = math.MaxInt32 // ~2GB, spec allows 2^32-1
	DefaultMaxBinaryLen = math.MaxInt32 // ~2GB, spec allows 2^32-1
	DefaultMaxArrayLen  = math.MaxInt32 // ~2B elements, spec allows 2^32-1
	DefaultMaxMapLen    = math.MaxInt32 // ~2B pairs, spec allows 2^32-1
	DefaultMaxExtLen    = math.MaxInt32 // ~2GB, spec allows 2^32-1
	DefaultMaxDepth     = 10000         // practical limit for nested structures
)

Default limits - aligned with msgpack spec (max 2^32-1) We use MaxInt32 for cross-platform safety (32-bit systems)

View Source 
const TimestampExtType int8 = -1

TimestampExtType is the extension type for msgpack timestamps.

Variables

View Source 
var (
	// ErrUnexpectedEOF is returned when the input ends before a complete value
	ErrUnexpectedEOF = errors.New("msgpck: unexpected end of input")

	// ErrInvalidFormat is returned when an invalid format byte is encountered
	ErrInvalidFormat = errors.New("msgpck: invalid format byte")

	// ErrStringTooLong is returned when a string exceeds MaxStringLen
	ErrStringTooLong = errors.New("msgpck: string exceeds maximum length")

	// ErrBinaryTooLong is returned when binary data exceeds MaxBinaryLen
	ErrBinaryTooLong = errors.New("msgpck: binary data exceeds maximum length")

	// ErrArrayTooLong is returned when an array exceeds MaxArrayLen
	ErrArrayTooLong = errors.New("msgpck: array exceeds maximum length")

	// ErrMapTooLong is returned when a map exceeds MaxMapLen
	ErrMapTooLong = errors.New("msgpck: map exceeds maximum length")

	// ErrExtTooLong is returned when ext data exceeds MaxExtLen
	ErrExtTooLong = errors.New("msgpck: ext data exceeds maximum length")

	// ErrMaxDepthExceeded is returned when nesting exceeds MaxDepth
	ErrMaxDepthExceeded = errors.New("msgpck: maximum nesting depth exceeded")

	// ErrTypeMismatch is returned when decoding into an incompatible type
	ErrTypeMismatch = errors.New("msgpck: type mismatch")

	// ErrNotPointer is returned when DecodeStruct is called with a non-pointer
	ErrNotPointer = errors.New("msgpck: decode target must be a pointer")

	// ErrNotStruct is returned when DecodeStruct is called with a non-struct pointer
	ErrNotStruct = errors.New("msgpck: decode target must be a pointer to struct")

	// ErrUnsupportedType is returned when encoding an unsupported Go type
	ErrUnsupportedType = errors.New("msgpck: unsupported type")
)

Functions

func DecodeMapFunc 

func DecodeMapFunc(data []byte, fn func(m map[string]any) error) error

DecodeMapFunc decodes msgpack into map[string]any and calls fn. Zero-copy strings: valid only within the callback. This is the safest way to use zero-copy when the buffer may be reused.

func DecodeStringMapFunc 

func DecodeStringMapFunc(data []byte, fn func(m map[string]string) error) error

DecodeStringMapFunc decodes msgpack into map[string]string and calls fn. Zero-copy strings: valid only within the callback.

func DecodeStructFunc 

func DecodeStructFunc[T any](data []byte, fn func(v *T) error) error

DecodeStructFunc decodes msgpack into struct T and calls fn. Zero-copy strings: valid only within the callback. Uses cached zero-copy decoder for repeated types.

func ExtToTimestamp 

func ExtToTimestamp(ext Ext) (time.Time, error)

ExtToTimestamp converts an Ext value to time.Time. Returns ErrTypeMismatch if the extension type is not a timestamp. Returns the time in UTC.

func IsTimestamp 

func IsTimestamp(ext Ext) bool

IsTimestamp checks if an Ext value is a timestamp.

func Marshal 

func Marshal(v any) ([]byte, error)

Marshal encodes a Go value to msgpack bytes. WARNING: The returned slice is only valid until the next Marshal call from the same goroutine. Copy the bytes if you need to retain them, or use MarshalCopy for a safe copy. This is zero-allocation for map[string]any and primitive types.

func MarshalAppend 

func MarshalAppend(dst []byte, v any) ([]byte, error)

MarshalAppend encodes a Go value to msgpack, appending to dst. The returned slice is safe to retain (it's the caller's buffer).

func MarshalCopy 

func MarshalCopy(v any) ([]byte, error)

MarshalCopy encodes a Go value to msgpack bytes. The returned bytes are a copy and safe to retain.

func MarshalTimestamp 

func MarshalTimestamp(t time.Time) []byte

MarshalTimestamp encodes a time.Time to msgpack timestamp bytes. Returns a copy of the encoded bytes (safe to retain). The time is converted to UTC before encoding.

func Unmarshal 

func Unmarshal(data []byte) (any, error)

Unmarshal decodes msgpack data into any. Uses pooled decoder for zero decoder allocation.

func UnmarshalMapStringAny 

func UnmarshalMapStringAny(data []byte, zeroCopy bool) (map[string]any, error)

UnmarshalMapStringAny decodes msgpack into map[string]any. Zero-copy strings when zeroCopy is true.

func UnmarshalMapStringString 

func UnmarshalMapStringString(data []byte, zeroCopy bool) (map[string]string, error)

UnmarshalMapStringString decodes msgpack into map[string]string. Much faster than map[string]any when you know the type.

func UnmarshalStruct 

func UnmarshalStruct(data []byte, v any) error

UnmarshalStruct is a convenience function that decodes msgpack data into a struct.

func UnmarshalStructWithConfig 

func UnmarshalStructWithConfig(data []byte, v any, cfg Config) error

UnmarshalStructWithConfig decodes msgpack data into a struct with custom config.

func UnmarshalTimestamp 

func UnmarshalTimestamp(data []byte) (time.Time, error)

UnmarshalTimestamp decodes msgpack timestamp bytes to time.Time. Returns the time in UTC. If the input has no timezone info, it is treated as UTC.

func UnmarshalWithConfig 

func UnmarshalWithConfig(data []byte, cfg Config) (any, error)

UnmarshalWithConfig decodes msgpack data with custom config.

Types

type Config 

type Config struct {
	// MaxStringLen is the maximum allowed string length in bytes
	MaxStringLen int

	// MaxBinaryLen is the maximum allowed binary data length in bytes
	MaxBinaryLen int

	// MaxArrayLen is the maximum allowed array length (number of elements)
	MaxArrayLen int

	// MaxMapLen is the maximum allowed map length (number of key-value pairs)
	MaxMapLen int

	// MaxExtLen is the maximum allowed ext data length in bytes
	MaxExtLen int

	// MaxDepth is the maximum allowed nesting depth for arrays and maps
	MaxDepth int
}

Config controls decoder/encoder behavior and security limits

func DefaultConfig 

func DefaultConfig() Config

DefaultConfig returns a Config with sensible defaults

func (Config) WithMaxArrayLen 

func (c Config) WithMaxArrayLen(n int) Config

WithMaxArrayLen returns a new Config with the specified MaxArrayLen

func (Config) WithMaxBinaryLen 

func (c Config) WithMaxBinaryLen(n int) Config

WithMaxBinaryLen returns a new Config with the specified MaxBinaryLen

func (Config) WithMaxDepth 

func (c Config) WithMaxDepth(n int) Config

WithMaxDepth returns a new Config with the specified MaxDepth

func (Config) WithMaxExtLen 

func (c Config) WithMaxExtLen(n int) Config

WithMaxExtLen returns a new Config with the specified MaxExtLen

func (Config) WithMaxMapLen 

func (c Config) WithMaxMapLen(n int) Config

WithMaxMapLen returns a new Config with the specified MaxMapLen

func (Config) WithMaxStringLen 

func (c Config) WithMaxStringLen(n int) Config

WithMaxStringLen returns a new Config with the specified MaxStringLen

type Decoder 

type Decoder struct {
	// contains filtered or unexported fields
}

Decoder reads MessagePack data from a byte slice. It provides zero-copy decoding where possible.

func NewDecoder 

func NewDecoder(data []byte) *Decoder

NewDecoder creates a new Decoder for the given data with default config

func NewDecoderWithConfig 

func NewDecoderWithConfig(data []byte, cfg Config) *Decoder

NewDecoderWithConfig creates a new Decoder with custom config

func (*Decoder) Decode 

func (d *Decoder) Decode() (Value, error)

Decode decodes a single MessagePack value. Returns a Value with zero-copy references to strings/binary data in the source.

func (*Decoder) DecodeAny 

func (d *Decoder) DecodeAny() (any, error)

DecodeAny decodes a MessagePack value directly into a Go native type. Returns: nil, bool, int64, uint64, float64, string, []byte, []any, map[string]any This is optimized to avoid intermediate Value structs.

func (*Decoder) DecodeStruct 

func (d *Decoder) DecodeStruct(v any) error

DecodeStruct decodes a msgpack map into a struct. v must be a pointer to a struct.

func (*Decoder) DecodeTimestamp 

func (d *Decoder) DecodeTimestamp() (time.Time, error)

DecodeTimestamp decodes a msgpack timestamp extension to time.Time. Returns the time in UTC. Returns ErrTypeMismatch if the value is not a timestamp extension.

func (*Decoder) Position 

func (d *Decoder) Position() int

Position returns the current position in the data

func (*Decoder) Remaining 

func (d *Decoder) Remaining() int

Remaining returns the number of unread bytes

func (*Decoder) Reset 

func (d *Decoder) Reset(data []byte)

Reset resets the decoder to decode new data

type Encoder 

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder writes MessagePack data to a byte buffer.

func NewEncoder 

func NewEncoder(capacity int) *Encoder

NewEncoder creates a new Encoder with the given initial capacity.

func NewEncoderBuffer 

func NewEncoderBuffer(buf []byte) *Encoder

NewEncoderBuffer creates an Encoder that writes to an existing buffer. The buffer will be grown as needed.

func (*Encoder) Bytes 

func (e *Encoder) Bytes() []byte

Bytes returns the encoded bytes.

func (*Encoder) Encode 

func (e *Encoder) Encode(v any) error

Encode encodes any Go value to msgpack.

func (*Encoder) EncodeArrayHeader 

func (e *Encoder) EncodeArrayHeader(length int)

EncodeArrayHeader writes the header for an array of the given length. Call this, then encode each element.

func (*Encoder) EncodeBinary 

func (e *Encoder) EncodeBinary(v []byte)

EncodeBinary writes binary data

func (*Encoder) EncodeBool 

func (e *Encoder) EncodeBool(v bool)

EncodeBool writes a boolean value

func (*Encoder) EncodeExt 

func (e *Encoder) EncodeExt(extType int8, data []byte)

EncodeExt writes extension data

func (*Encoder) EncodeFloat32 

func (e *Encoder) EncodeFloat32(v float32)

EncodeFloat32 writes a float32 value

func (*Encoder) EncodeFloat64 

func (e *Encoder) EncodeFloat64(v float64)

EncodeFloat64 writes a float64 value

func (*Encoder) EncodeInt 

func (e *Encoder) EncodeInt(v int64)

EncodeInt writes an int64 using the most compact format

func (*Encoder) EncodeMapHeader 

func (e *Encoder) EncodeMapHeader(length int)

EncodeMapHeader writes the header for a map of the given length. Call this, then encode each key-value pair.

func (*Encoder) EncodeNil 

func (e *Encoder) EncodeNil()

EncodeNil writes a nil value

func (*Encoder) EncodeString 

func (e *Encoder) EncodeString(v string)

EncodeString writes a string value (zero-copy using unsafe)

func (*Encoder) EncodeStringBytes 

func (e *Encoder) EncodeStringBytes(v []byte)

EncodeStringBytes writes a string from bytes

func (*Encoder) EncodeTimestamp 

func (e *Encoder) EncodeTimestamp(t time.Time)

EncodeTimestamp encodes a time.Time as a msgpack timestamp extension. Uses the most compact format that can represent the timestamp:

  • Timestamp 32: seconds only (no nanoseconds), fits in uint32
  • Timestamp 64: nanoseconds + seconds fits in 34-bit unsigned
  • Timestamp 96: full range with 64-bit signed seconds

The time is converted to UTC before encoding.

func (*Encoder) EncodeUint 

func (e *Encoder) EncodeUint(v uint64)

EncodeUint writes a uint64 using the most compact format

func (*Encoder) EncodeValue 

func (e *Encoder) EncodeValue(v *Value)

EncodeValue writes a Value

func (*Encoder) Len 

func (e *Encoder) Len() int

Len returns the length of encoded data.

func (*Encoder) Reset 

func (e *Encoder) Reset()

Reset resets the encoder for reuse.

type Ext 

type Ext struct {
	Type int8
	Data []byte // points into source
}

Ext represents MessagePack extension data

type KV 

type KV struct {
	Key   []byte // points into source
	Value Value
}

KV represents a key-value pair in a map. Key is a []byte pointing into the source buffer for zero-copy.

type StructDecoder 

type StructDecoder[T any] struct {
	// contains filtered or unexported fields
}

StructDecoder is a pre-registered decoder for a specific struct type. Create once, reuse many times for best performance.

func GetStructDecoder 

func GetStructDecoder[T any](zeroCopy bool) *StructDecoder[T]

GetStructDecoder returns a cached decoder for type T. Creates one if it doesn't exist. If zeroCopy is true, strings point directly into the input buffer and are only valid while the buffer exists.

func (*StructDecoder[T]) Decode 

func (sd *StructDecoder[T]) Decode(data []byte, dst *T) error

Decode decodes msgpack data into the struct. This is the fast path - single pass, no intermediate allocations.

func (*StructDecoder[T]) ZeroCopy 

func (sd *StructDecoder[T]) ZeroCopy() *StructDecoder[T]

ZeroCopy returns a zero-copy version of this decoder. Strings will point directly into the input buffer - no allocations! WARNING: Decoded strings are only valid while the input buffer exists. Use this when the input buffer outlives the decoded struct (e.g., from database).

type StructEncoder 

type StructEncoder[T any] struct {
	// contains filtered or unexported fields
}

StructEncoder is a pre-registered encoder for a specific struct type. Create once, reuse many times for best performance.

func GetStructEncoder 

func GetStructEncoder[T any]() *StructEncoder[T]

GetStructEncoder returns a cached encoder for type T.

func (*StructEncoder[T]) Encode 

func (se *StructEncoder[T]) Encode(src *T) ([]byte, error)

Encode encodes the struct to msgpack bytes. Returns bytes from a pooled buffer - copy if you need to retain.

func (*StructEncoder[T]) EncodeAppend 

func (se *StructEncoder[T]) EncodeAppend(dst []byte, src *T) ([]byte, error)

EncodeAppend encodes and appends to dst.

func (*StructEncoder[T]) EncodeCopy 

func (se *StructEncoder[T]) EncodeCopy(src *T) ([]byte, error)

EncodeCopy encodes and returns a copy (safe to retain).

type Type 

type Type uint8

Type represents the type of a decoded MessagePack value 

const (
	TypeNil Type = iota
	TypeBool
	TypeInt
	TypeUint
	TypeFloat32
	TypeFloat64
	TypeString
	TypeBinary
	TypeArray
	TypeMap
	TypeExt
)

func (Type) String 

func (t Type) String() string

String returns the string representation of the type

type Value 

type Value struct {
	Type    Type
	Bool    bool
	Int     int64
	Uint    uint64
	Float32 float32
	Float64 float64
	Bytes   []byte  // string/binary - points into source
	Array   []Value // array elements
	Map     []KV    // map key-value pairs
	Ext     Ext     // extension data
}

Value represents a decoded MessagePack value. For strings and binary data, Bytes points directly into the source buffer (zero-copy). The caller must not modify the source buffer while Value is in use.

func (*Value) AsBool 

func (v *Value) AsBool() bool

AsBool returns the value as bool. Panics if Type != TypeBool.

func (*Value) AsBytes 

func (v *Value) AsBytes() []byte

AsBytes returns Bytes directly (zero-copy reference to source).

func (*Value) AsFloat64 

func (v *Value) AsFloat64() float64

AsFloat64 returns the value as float64. Works for float32/64 and int/uint.

func (*Value) AsInt 

func (v *Value) AsInt() int64

AsInt returns the value as int64. Works for TypeInt and TypeUint.

func (*Value) AsString 

func (v *Value) AsString() string

AsString returns Bytes as a string. This allocates.

func (*Value) AsUint 

func (v *Value) AsUint() uint64

AsUint returns the value as uint64. Works for TypeInt and TypeUint.

func (*Value) Get 

func (v *Value) Get(key []byte) *Value

Get returns the value for a key in a map (linear search). Returns nil if not found or not a map.

func (*Value) GetString 

func (v *Value) GetString(key string) *Value

GetString is a convenience for Get with a string key.

func (*Value) Index 

func (v *Value) Index(i int) *Value

Index returns the i-th element of an array. Panics if out of bounds or not an array.

func (*Value) IsNil 

func (v *Value) IsNil() bool

IsNil returns true if the value is nil

func (*Value) Len 

func (v *Value) Len() int

Len returns the length of array/map/string/binary.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.