seltabl

package module
v0.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 24, 2024 License: MIT Imports: 7 Imported by: 0

README

seltabl

seltabl logo

go.dev Build Status Go Report Card

A golang library with accompanying cli and language server for configurably parsing html sequences into stucts originally built for html tables, but can be used for any html sequence.

Enables data binding to structs and provides a simple, but dynamic way to define a table schema.

Installation

Install the package in a project with:

go get github.com/conneroisu/seltabl

Install the cli containing the language server operating over the lsp protocol and package command line utilities with:

go install github.com/conneroisu/seltabl/tools/seltabls@latest
Recording of Language Server

Usage

package main

import (
	"fmt"
	"github.com/conneroisu/seltabl"
	"github.com/conneroisu/seltabl/testdata"
)

type TableStruct struct {
	A string `json:"a" hSel:"tr:nth-child(1) td:nth-child(1)" dSel:"tr td:nth-child(1)" ctl:"text"`
	B string `json:"b" hSel:"tr:nth-child(1) td:nth-child(2)" dSel:"tr td:nth-child(2)" ctl:"text"`
}

var fixture = `
<table>
	<tr>
		<td>a</td>
		<td>b</td>
	</tr>
	<tr>
		<td>1</td>
		<td>2</td>
	</tr>
	<tr>
		<td>3</td>
		<td>4</td>
	</tr>
	<tr>
		<td>5</td>
		<td>6</td>
	</tr>
	<tr>
		<td>7</td>
		<td>8</td>
	</tr>
</table>
`

func main() {
	fss, err := seltabl.NewFromString[TableStruct](fixture)
	if err != nil {
		panic(fmt.Errorf("failed to parse html: %w", err))
	}
	for _, fs := range fss {
		fmt.Printf("%+v\n", fs)
	}
}

Output:

{A:1 B:2}
{A:3 B:4}
{A:5 B:6}
{A:7 B:8}

Development

A makefile at the root of the project is provided to help with development.

Testing

One can run the tests with:

make test
Linting

One can run the linter with:

make lint
Formatting

One can run the formatter with:

make fmt
Generating documentation

One can run the documentation generator with:

make doc

License

MIT

Documentation

Overview

Package seltabl provides a simple way to parse html tables into structs.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func New

func New[T any](doc *goquery.Document) ([]T, error)

New parses a goquery doc into a slice of structs.

The struct given as an argument must have a field with the tag seltabl, a header selector with the tag hSel, and a data selector with the tag dSel.

The selectors responsibilities:

  • header selector (hSel): used to find the header row and column for the field in the given struct.
  • data selector (dSel): used to find the data column for the field in the given struct.
  • query selector (qSel): used to query for the inner text or attribute of the cell.
  • control selector (cSel): used to control what to query for the inner text or attribute of the cell.

Example:

package main

var fixture = `
<table>
     <tr> <td>a</td> <td>b</td> </tr>
     <tr> <td>1</td> <td>2</td> </tr>
     <tr> <td>3</td> <td>4</td> </tr>
     <tr> <td>5</td> <td>6</td> </tr>
     <tr> <td>7</td> <td>8</td> </tr>
</table>
`

type FixtureStruct struct {
        A string `json:"a" hSel:"tr:nth-child(1)" dSel:"table tr:not(:first-child) td:nth-child(1)" cSel:"$text"`
        B string `json:"b" hSel:"tr:nth-child(1)" dSel:"table tr:not(:first-child) td:nth-child(2)" cSel:"$text"`
}

func main() {
	p, err := seltabl.New[fixtureStruct](fixture)
	if err != nil {
		panic(err)
	}
	for _, pp := range p {
		fmt.Printf("pp %+v\n", pp)
	}
}

func NewFromReader

func NewFromReader[T any](r io.Reader) ([]T, error)

NewFromReader parses a reader into a slice of structs.

The reader must be a valid html page with a single table.

The passed in generic type must be a struct with valid selectors for the table and data (hSel, dSel, cSel).

The selectors responsibilities:

  • header selector (hSel): used to find the header row and column for the field in the given struct.
  • data selector (dSel): used to find the data column for the field in the given struct.
  • query selector (qSel): used to query for the inner text or attribute of the cell.
  • control selector (cSel): used to control what to query for the inner text or attribute of the cell.

Example:

package main

import (
	"fmt"
	"github.com/conneroisu/seltabl"
)

type TableStruct struct {
	A string `json:"a" hSel:"tr:nth-child(1) td:nth-child(1)" dSel:"tr td:nth-child(1)" cSel:"$text"`
	B string `json:"b" hSel:"tr:nth-child(1) td:nth-child(2)" dSel:"tr td:nth-child(2)" cSel:"$text"`
}

func main() {
	p, err := seltabl.NewFromReader[TableStruct](strings.NewReader(`
	<table>
		<tr> <td>a</td> <td>b</td> </tr>
		<tr> <td>1</td> <td>2</td> </tr>
		<tr> <td>3</td> <td>4</td> </tr>
		<tr> <td>5</td> <td>6</td> </tr>
		<tr> <td>7</td> <td>8</td> </tr>
	</table>
	`))
	if err != nil {
		panic(err)
	}
	for _, pp := range p {
		fmt.Printf("pp %+v\n", pp)
	}
}

func NewFromString

func NewFromString[T any](htmlInput string) ([]T, error)

NewFromString parses a string into a slice of structs.

The struct must have a field with the tag seltabl, a header selector with the tag hSel, and a data selector with the tag dSel.

The selectors responsibilities:

  • header selector (hSel): used to find the header row and column for the field in the given struct.
  • data selector (dSel): used to find the data column for the field in the given struct.
  • query selector (qSel): used to query for the inner text or attribute of the cell.
  • control selector (cSel): used to control what to query for the inner text or attribute of the cell.

Example:

package main

import (
	"fmt"
	"github.com/conneroisu/seltabl"
)

type FixtureStruct struct {
        A string `json:"a" hSel:"tr:nth-child(1)" dSel:"table tr:not(:first-child) td:nth-child(1)" cSel:"$text"`
        B string `json:"b" hSel:"tr:nth-child(1)" dSel:"table tr:not(:first-child) td:nth-child(2)" cSel:"$text"`
}

func main() {
	p, err := seltabl.NewFromString[TableStruct](`
	<table>
		<tr> <td>a</td> <td>b</td> </tr>
		<tr> <td>1</td> <td>2</td> </tr>
		<tr> <td>3</td> <td>4</td> </tr>
		<tr> <td>5</td> <td>6</td> </tr>
		<tr> <td>7</td> <td>8</td> </tr>
	</table>
	`)
	if err != nil {
		panic(err)
	}
	for _, pp := range p {
		fmt.Printf("pp %+v\n", pp)
	}
}

func NewFromURL

func NewFromURL[T any](url string) ([]T, error)

NewFromURL parses a given URL's html into a slice of structs adhering to the given generic type.

The URL must be a valid html page with a single table.

The passed in generic type must be a struct with valid selectors for the table and data (hSel, dSel, cSel).

The selectors responsibilities:

  • header selector (hSel): used to find the header row and column for the field in the given struct.
  • data selector (dSel): used to find the data column for the field in the given struct.
  • query selector (qSel): used to query for the inner text or attribute of the cell.
  • control selector (cSel): used to control what to query for the inner text or attribute of the cell.

Example:

package main

import (
	"fmt"
	"github.com/conneroisu/seltabl"
)

type FixtureStruct struct {
        A string `json:"a" hSel:"tr:nth-child(1)" dSel:"table tr:not(:first-child) td:nth-child(1)" cSel:"$text"`
        B string `json:"b" hSel:"tr:nth-child(1)" dSel:"table tr:not(:first-child) td:nth-child(2)" cSel:"$text"`
}

func main() {
	p, err := seltabl.NewFromURL[TableStruct]("https://github.com/conneroisu/seltabl/blob/main/testdata/ab_num_table.html")
	if err != nil {
		panic(err)
	}
	for _, pp := range p {
		fmt.Printf("pp %+v\n", pp)
	}
}

func SetStructField

func SetStructField[T any](
	structPtr *T,
	fieldName string,
	cellValue *goquery.Selection,
	selector SelectorI,
) error

SetStructField sets a struct field to a value. It uses generics to specify the type of the struct and the field name. It also uses the selector interface to find the value and uses the type of the selector to parse and set the value.

It is used by the NewFromString function.

Types

type Decoder

type Decoder[T any] struct {
	// contains filtered or unexported fields
}

Decoder is a struct for decoding a reader into a slice of structs.

It is used by the NewDecoder function.

It is not intended to be used directly.

Example:

type TableStruct struct {
	A string `json:"a" seltabl:"a" hSel:"tr:nth-child(1) td:nth-child(1)" dSel:"tr td:nth-child(1)" cSel:"$text"`
	B string `json:"b" seltabl:"b" hSel:"tr:nth-child(1) td:nth-child(2)" dSel:"tr td:nth-child(2)" cSel:"$text"`
}

func main() {
	r := strings.NewReader(`
	<table>
		<tr>
			<td>a</td>
			<td>b</td>
		</tr>
		<tr>
			<td> 1 </td>
			<td>2</td>
		</tr>
		<tr>
			<td>3 </td>
			<td> 4</td>
		</tr>
		<tr>
			<td> 5 </td>
			<td> 6</td>
		</tr>
		<tr>
			<td>7 </td>
			<td> 8</td>
		</tr>
	</table>
	`)
	p, err := seltabl.NewDecoder[TableStruct](r)
	if err != nil {
		panic(err)
	}
	for _, pp := range p {
		fmt.Printf("pp %+v\n", pp)
	}
}

func NewDecoder

func NewDecoder[T any](r io.ReadCloser) *Decoder[T]

NewDecoder parses a reader into a slice of structs.

It is used by the NewFromReader function.

This allows for decoding a reader into a slice of structs.

Similar to the json.Decoder for brevity.

func (*Decoder[T]) Decode

func (d *Decoder[T]) Decode() ([]T, error)

Decode parses a reader into a slice of structs.

It is used by the Decoder.Decode function.

This allows for decoding a reader into a slice of structs.

Similar to the json.Decoder for brevity.

type ErrMissingMustBePresent added in v0.3.1

type ErrMissingMustBePresent struct {
	Field reflect.StructField
	Cfg   *SelectorConfig
}

ErrMissingMustBePresent is an error for when a must be present selector is not found

func (*ErrMissingMustBePresent) Error added in v0.3.1

func (e *ErrMissingMustBePresent) Error() string

Error implements the error interface

type ErrNoDataFound added in v0.5.1

type ErrNoDataFound struct {
	Typ   reflect.Type
	Field reflect.StructField
	Cfg   *SelectorConfig
	Doc   *goquery.Document
}

ErrNoDataFound is an error for when no data is found for a selector

func (*ErrNoDataFound) Error added in v0.5.1

func (e *ErrNoDataFound) Error() string

Error implements the error interface for ErrNoDataFound

type ErrSelectorNotFound added in v0.5.1

type ErrSelectorNotFound struct {
	Typ   reflect.Type        // type of the struct
	Field reflect.StructField // field of the struct
	Cfg   *SelectorConfig     // selector config
	Doc   *goquery.Document   // goquery document
}

ErrSelectorNotFound is an error for when a selector is not found

func (*ErrSelectorNotFound) Error added in v0.5.1

func (e *ErrSelectorNotFound) Error() string

Error implements the error interface for ErrSelectorNotFound

type SelectorConfig added in v0.2.9

type SelectorConfig struct {
	DataSelector  string // selector for the data cell
	HeadSelector  string // selector for the header cell
	QuerySelector string // selector for the data cell
	ControlTag    string // tag used to signify selecting aspects of a cell
	MustBePresent string // text that must be present in a given content
}

SelectorConfig is a struct for configuring a selector

func NewSelectorConfig added in v0.2.9

func NewSelectorConfig(tag reflect.StructTag) *SelectorConfig

NewSelectorConfig parses a struct tag and returns a SelectorConfig

type SelectorI added in v0.5.1

type SelectorI interface {
	Select(cellValue *goquery.Selection) (string, error)
}

SelectorI is an interface for running a goquery selector on a cellValue

It is an interface that defines a Select method that takes a cellValue (goquery.Selection) and returns a string of the applied selection and an error.

Directories

Path Synopsis
examples
huggingface-leader-board command
Package main shows how to use the seltabl package to scrape a table from a given url.
Package main shows how to use the seltabl package to scrape a table from a given url.
ncaa command
Package main is the an example of how to use the seltabl package.
Package main is the an example of how to use the seltabl package.
penguins-wikipedia command
Package main is the an example of how to use the seltabl package.
Package main is the an example of how to use the seltabl package.
tools
seltabl-lsp module
seltabls module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL