README
¶
recordio
Package recordio implements a file format for a sequence of variable-length data.
Benchmarks
On Intel(R) Core(TM) i5-8259U CPU @ 2.30GHz
goos: darwin
goarch: amd64
pkg: github.com/jhahn21/recordio
BenchmarkRead/nocompression_100 11585869 99.5 ns/op 32 B/op 1 allocs/op
BenchmarkRead/nocompression_1000 10453348 111 ns/op 32 B/op 1 allocs/op
BenchmarkRead/nocompression_10000 6116316 195 ns/op 32 B/op 1 allocs/op
BenchmarkRead/snappy_100 4472151 267 ns/op 32 B/op 1 allocs/op
BenchmarkRead/snappy_1000 3407258 357 ns/op 32 B/op 1 allocs/op
BenchmarkRead/snappy_10000 1000000 1019 ns/op 32 B/op 1 allocs/op
BenchmarkRead/zlib_100 834781 1426 ns/op 4228 B/op 4 allocs/op
BenchmarkRead/zlib_1000 571678 1857 ns/op 4228 B/op 4 allocs/op
BenchmarkRead/zlib_10000 193426 5881 ns/op 4228 B/op 4 allocs/op
BenchmarkWrite/nocompression_100 73785631 14.2 ns/op 0 B/op 0 allocs/op
BenchmarkWrite/nocompression_1000 83530110 14.1 ns/op 0 B/op 0 allocs/op
BenchmarkWrite/nocompression_10000 82618550 14.1 ns/op 0 B/op 0 allocs/op
BenchmarkWrite/snappy_100 6451332 183 ns/op 0 B/op 0 allocs/op
BenchmarkWrite/snappy_1000 2747001 438 ns/op 0 B/op 0 allocs/op
BenchmarkWrite/snappy_10000 465871 2446 ns/op 0 B/op 0 allocs/op
BenchmarkWrite/zlib_100 37065 32426 ns/op 21 B/op 0 allocs/op
BenchmarkWrite/zlib_1000 14058 82467 ns/op 57 B/op 0 allocs/op
BenchmarkWrite/zlib_10000 5492 193872 ns/op 150 B/op 0 allocs/op
PASS
ok github.com/jhahn21/recordio 24.716s
How to contribute
We use Gerrit for reviewing proposed changes.
$ git clone https://github.com/jhahn21/recordio.git
$ cd recordio
$ curl -Lo .git/hooks/commit-msg https://review.gerrithub.io/tools/hooks/commit-msg
$ chmod +x .git/hooks/commit-msg
$ git remote add gerrit https://review.gerrithub.io/jhahn21/recordio
...
// Push for code review
...
$ git commit
$ git push gerrit HEAD:refs/for/master
...
// Push a patch Set
...
$ git commit --amend
$ git push gerrit HEAD:refs/for/master
...
You can find more details at Gerrit Code Review Workflow.
Documentation
¶
Overview ¶
Package recordio implements a file format for a sequence of variable-length data.
The layout of each record is
+-----------------------------------------------------------| | 4-bytes Magic number (including compression mode) | +-----------------------------------------------------------| | 4-byte Uncompressed record size (uint32 in little-endian) | +-----------------------------------------------------------| | 4-byte Compressed record size (uint32 in little-endian) | +-----------------------------------------------------------| | Record data (maximum 1024MB) | +-----------------------------------------------------------|
Example:
// Writing records.
f, _ := os.Create("data.rec")
defer f.Close()
w, _ := recordio.NewWriter(f, recordio.Snappy)
w.Write([]byte("this is a record"))
w.Write([]byte("this is a second record"))
...
// Reading records.
f, _ := os.Open("data.rec")
defer f.Close()
r, _ := recordio.NewReader(f)
for {
rec, err := r.Read()
if err == io.EOF {
break
}
...
}
Index ¶
Examples ¶
Constants ¶
View Source
const ( // NoCompression does not compress records. NoCompression = modeBase + iota // Snappy compresses records with Snappy. Snappy // Zlib compresses records with zlib. Zlib // EndOfMode is the end of mode. EndOfMode )
Variables ¶
View Source
var ( // ErrInvalidHeader means that a record header is not valid. ErrInvalidHeader = errors.New("invalid record header") // ErrInvalidMode means that a (compression) mode is not valid. ErrInvalidMode = errors.New("invalid mode") // ErrRecordCorrupted means that a record is corrupted. ErrRecordCorrupted = errors.New("record corrupted") // ErrRecordTooLarge means that a record is too large. ErrRecordTooLarge = errors.New("record too large") )
Functions ¶
This section is empty.
Types ¶
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader implements sequential reading of variable-length data.
func NewReader ¶
NewReader returns a new reader that reads records from r.
Example ¶
package main
import (
"bytes"
"fmt"
"io"
"github.com/jhahn21/recordio"
)
func main() {
b := bytes.NewReader([]byte{210, 159, 49, 224, 5, 0, 0, 0, 23, 0, 0, 0, 255, 6, 0, 0, 115, 78, 97, 80, 112, 89, 1, 9, 0, 0, 85, 255, 35, 229, 102, 105, 114, 115, 116, 210, 159, 49, 224, 6, 0, 0, 0, 24, 0, 0, 0, 255, 6, 0, 0, 115, 78, 97, 80, 112, 89, 1, 10, 0, 0, 211, 224, 211, 202, 115, 101, 99, 111, 110, 100})
r, err := recordio.NewReader(b)
if err != nil {
panic(err)
}
for {
d, err := r.Read()
if err == io.EOF {
break
}
fmt.Println(string(d))
}
}
Output: first second
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer implements sequential writing of variable-length data.
func NewWriter ¶
NewWriter returns a new writer that writes records to w.
Example ¶
package main
import (
"bytes"
"fmt"
"github.com/jhahn21/recordio"
)
func main() {
var b bytes.Buffer
w, err := recordio.NewWriter(&b, recordio.Snappy)
if err != nil {
panic(err)
}
w.Write([]byte("first"))
w.Write([]byte("second"))
fmt.Println(b.Bytes())
}
Output: [210 159 49 224 5 0 0 0 23 0 0 0 255 6 0 0 115 78 97 80 112 89 1 9 0 0 85 255 35 229 102 105 114 115 116 210 159 49 224 6 0 0 0 24 0 0 0 255 6 0 0 115 78 97 80 112 89 1 10 0 0 211 224 211 202 115 101 99 111 110 100]
Source Files
Click to show internal directories.
Click to hide internal directories.