seltabl

package module
v0.3.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 9, 2024 License: MIT Imports: 7 Imported by: 0

README

seltabl

go.dev Build Status Go Report Card

A golang library for configurably parsing html sequences into stucts originally built for html tables, but can be used for any html sequence.

Enables data binding to structs and provides a simple, but dynamic way to define a table schema.

Installation

Install the package with:

go get github.com/conneroisu/seltabl

Usage

package main

import (
	"fmt"
	"github.com/conneroisu/seltabl"
	"github.com/conneroisu/seltabl/testdata"
)

type TableStruct struct {
	A string `json:"a" seltabl:"a" hSel:"tr:nth-child(1) td:nth-child(1)" dSel:"tr td:nth-child(1)" cSel:"$text"`
	B string `json:"b" seltabl:"b" hSel:"tr:nth-child(1) td:nth-child(2)" dSel:"tr td:nth-child(2)" cSel:"$text"`
}

var fixture = `
<table>
	<tr>
		<td>a</td>
		<td>b</td>
	</tr>
	<tr>
		<td>1</td>
		<td>2</td>
	</tr>
	<tr>
		<td>3</td>
		<td>4</td>
	</tr>
	<tr>
		<td>5</td>
		<td>6</td>
	</tr>
	<tr>
		<td>7</td>
		<td>8</td>
	</tr>
</table>
`

func main() {
	fss, err := seltabl.NewFromString[TableStruct](fixture)
	if err != nil {
		panic(fmt.Errorf("failed to parse html: %w", err))
	}
	for _, fs := range fss {
		fmt.Printf("%+v\n", fs)
	}
}

Output:

{A:1 B:2}
{A:3 B:4}
{A:5 B:6}
{A:7 B:8}

Development

A makefile at the root of the project is provided to help with development.

Testing

One can run the tests with:

make test
Linting

One can run the linter with:

make lint
Formatting

One can run the formatter with:

make fmt
Generating documentation

One can run the documentation generator with:

make doc

License

MIT

Documentation

Overview

Package seltabl provides a simple way to parse html tables into structs.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func New 

func New[T any](doc *goquery.Document) ([]T, error)

New parses a goquery doc into a slice of structs.

The struct given as an argument must have a field with the tag seltabl, a header selector with the tag hSel, and a data selector with the tag dSel.

The selectors responsibilities:

  • header selector (hSel): used to find the header row and column for the field in the given struct.
  • data selector (dSel): used to find the data column for the field in the given struct.
  • query selector (qSel): used to query for the inner text or attribute of the cell.
  • control selector (cSel): used to control what to query for the inner text or attribute of the cell.

Example: 

package main

var fixture = `
<table>

     <tr>
     	<td>a</td>
     	<td>b</td>
     </tr>
     <tr>
     	<td>1</td>
     	<td>2</td>
     </tr>
     <tr>
     	<td>3</td>
     	<td>4</td>
     </tr>
     <tr>
     	<td>5</td>
     	<td>6</td>
     </tr>
     <tr>
     	<td>7</td>
     	<td>8</td>
     </tr>

</table>
`

type fixtureStruct struct {
	A string `json:"a" seltabl:"a" hSel:"tr:nth-child(1) td:nth-child(1)" dSel:"tr td:nth-child(1)" cSel:"$text"`
	B string `json:"b" seltabl:"b" hSel:"tr:nth-child(1) td:nth-child(2)" dSel:"tr td:nth-child(2)" cSel:"$text"`
}

func main() {
	p, err := seltabl.New[fixtureStruct](fixture)
	if err != nil {
		panic(err)
	}
	for _, pp := range p {
		fmt.Printf("pp %+v\n", pp)
	}
}

func NewFromReader 

func NewFromReader[T any](r io.Reader) ([]T, error)

NewFromReader parses a reader into a slice of structs.

The reader must be a valid html page with a single table.

The passed in generic type must be a struct with valid selectors for the table and data (hSel, dSel, cSel).

The selectors responsibilities:

  • header selector (hSel): used to find the header row and column for the field in the given struct.
  • data selector (dSel): used to find the data column for the field in the given struct.
  • query selector (qSel): used to query for the inner text or attribute of the cell.
  • control selector (cSel): used to control what to query for the inner text or attribute of the cell.

Example: 

package main

import (
	"fmt"
	"github.com/conneroisu/seltabl"
)

type TableStruct struct {
	A string `json:"a" seltabl:"a" hSel:"tr:nth-child(1) td:nth-child(1)" dSel:"tr td:nth-child(1)" cSel:"$text"`
	B string `json:"b" seltabl:"b" hSel:"tr:nth-child(1) td:nth-child(2)" dSel:"tr td:nth-child(2)" cSel:"$text"`
}

func main() {
	p, err := seltabl.NewFromReader[TableStruct](strings.NewReader(`
	<table>
		<tr>
			<td>a</td>
			<td>b</td>
		</tr>
		<tr>
			<td>1</td>
			<td>2</td>
		</tr>
		<tr>
			<td>3</td>
			<td>4</td>
		</tr>
		<tr>
			<td>5</td>
			<td>6</td>
		</tr>
		<tr>
			<td>7</td>
			<td>8</td>
		</tr>
	</table>
	`))
	if err != nil {
		panic(err)
	}
	for _, pp := range p {
		fmt.Printf("pp %+v\n", pp)
	}
}

func NewFromString 

func NewFromString[T any](htmlInput string) ([]T, error)

NewFromString parses a string into a slice of structs.

The struct must have a field with the tag seltabl, a header selector with the tag hSel, and a data selector with the tag dSel.

The selectors responsibilities:

  • header selector (hSel): used to find the header row and column for the field in the given struct.
  • data selector (dSel): used to find the data column for the field in the given struct.
  • query selector (qSel): used to query for the inner text or attribute of the cell.
  • control selector (cSel): used to control what to query for the inner text or attribute of the cell.

Example: 

package main

import (
	"fmt"
	"github.com/conneroisu/seltabl"
)

type TableStruct struct {
	A string `json:"a" seltabl:"a" hSel:"tr:nth-child(1) td:nth-child(1)" dSel:"tr td:nth-child(1)" cSel:"$text"`
	B string `json:"b" seltabl:"b" hSel:"tr:nth-child(1) td:nth-child(2)" dSel:"tr td:nth-child(2)" cSel:"$text"`
}

func main() {
	p, err := seltabl.NewFromString[TableStruct](`
	<table>
		<tr>
			<td>a</td>
			<td>b</td>
		</tr>
		<tr>
			<td>1</td>
			<td>2</td>
		</tr>
		<tr>
			<td>3</td>
			<td>4</td>
		</tr>
		<tr>
			<td>5</td>
			<td>6</td>
		</tr>
		<tr>
			<td>7</td>
			<td>8</td>
		</tr>
	</table>
	`)
	if err != nil {
		panic(err)
	}
	for _, pp := range p {
		fmt.Printf("pp %+v\n", pp)
	}
}

func NewFromURL 

func NewFromURL[T any](url string) ([]T, error)

NewFromURL parses a given URL's html into a slice of structs adhering to the given generic type.

The URL must be a valid html page with a single table.

The passed in generic type must be a struct with valid selectors for the table and data (hSel, dSel, cSel).

The selectors responsibilities:

  • header selector (hSel): used to find the header row and column for the field in the given struct.
  • data selector (dSel): used to find the data column for the field in the given struct.
  • query selector (qSel): used to query for the inner text or attribute of the cell.
  • control selector (cSel): used to control what to query for the inner text or attribute of the cell.

Example: 

package main

import (
	"fmt"
	"github.com/conneroisu/seltabl"
)

type TableStruct struct {
	A string `json:"a" seltabl:"a" hSel:"tr:nth-child(1) td:nth-child(1)" dSel:"tr td:nth-child(1)" cSel:"$text"`
	B string `json:"b" seltabl:"b" hSel:"tr:nth-child(1) td:nth-child(2)" dSel:"tr td:nth-child(2)" cSel:"$text"`
}

func main() {
	p, err := seltabl.NewFromURL[TableStruct]("https://github.com/conneroisu/seltabl/blob/main/testdata/ab_num_table.html")
	if err != nil {
		panic(err)
	}
	for _, pp := range p {
		fmt.Printf("pp %+v\n", pp)
	}
}

func SetStructField 

func SetStructField[T any](
	structPtr *T,
	fieldName string,
	cellValue *goquery.Selection,
	selector SelectorInferface,
) error

SetStructField sets a struct field to a value It uses generics to specify the type of the struct and the field name.

It is used by the NewFromString function.

Types

type Decoder 

type Decoder[T any] struct {
	// contains filtered or unexported fields
}

Decoder is a struct for decoding a reader into a slice of structs.

It is used by the NewDecoder function.

It is not intended to be used directly.

Example: 

type TableStruct struct {
	A string `json:"a" seltabl:"a" hSel:"tr:nth-child(1) td:nth-child(1)" dSel:"tr td:nth-child(1)" cSel:"$text"`
	B string `json:"b" seltabl:"b" hSel:"tr:nth-child(1) td:nth-child(2)" dSel:"tr td:nth-child(2)" cSel:"$text"`
}

func main() {
	r := strings.NewReader(`
	<table>
		<tr>
			<td>a</td>
			<td>b</td>
		</tr>
		<tr>
			<td> 1 </td>
			<td>2</td>
		</tr>
		<tr>
			<td>3 </td>
			<td> 4</td>
		</tr>
		<tr>
			<td> 5 </td>
			<td> 6</td>
		</tr>
		<tr>
			<td>7 </td>
			<td> 8</td>
		</tr>
	</table>
	`)
	p, err := seltabl.NewDecoder[TableStruct](r)
	if err != nil {
		panic(err)
	}
	for _, pp := range p {
		fmt.Printf("pp %+v\n", pp)
	}
}

func NewDecoder 

func NewDecoder[T any](r io.ReadCloser) *Decoder[T]

NewDecoder parses a reader into a slice of structs.

It is used by the NewFromReader function.

This allows for decoding a reader into a slice of structs.

Similar to the json.Decoder for brevity.

func (*Decoder[T]) Decode 

func (d *Decoder[T]) Decode() ([]T, error)

Decode parses a reader into a slice of structs.

It is used by the Decoder.Decode function.

This allows for decoding a reader into a slice of structs.

Similar to the json.Decoder for brevity.

type ErrMissingMustBePresent added in v0.3.1 

type ErrMissingMustBePresent struct {
	Field         reflect.StructField
	MustBePresent string
}

ErrMissingMustBePresent is an error for when a must be present selector is not found

func (*ErrMissingMustBePresent) Error added in v0.3.1 

func (e *ErrMissingMustBePresent) Error() string

Error implements the error interface

type SelectorConfig added in v0.2.9 

type SelectorConfig struct {
	HeadName      string
	DataSelector  string
	HeadSelector  string
	QuerySelector string
	ControlTag    string
	MustBePresent string
}

SelectorConfig is a struct for configuring a selector

func NewSelectorConfig added in v0.2.9 

func NewSelectorConfig(tag reflect.StructTag) *SelectorConfig

NewSelectorConfig parses a struct tag and returns a SelectorConfig

type SelectorInferface added in v0.2.0 

type SelectorInferface interface {
	Select(cellValue *goquery.Selection) (*string, error)
}

SelectorInferface is an interface for running a goquery selector on a cellValue

Source Files

View all Source files

Directories

Path Synopsis
examples
huggingface-leader-board command
Package main shows how to use the seltabl package to scrape a table from a given url.
 Package main shows how to use the seltabl package to scrape a table from a given url.
ncaa command
Package main is the an example of how to use the seltabl package.
 Package main is the an example of how to use the seltabl package.
penguins-wikipedia command
Package main is the an example of how to use the seltabl package.
 Package main is the an example of how to use the seltabl package.
tools
seltabl-lsp module
seltabls module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.