README
¶
feedtr - A Go Language Feed Transformer
feedtr fetches feed sources (Atom, RSS or anything using XML), transforms them using XSLT and saves the outputs.
feedtr is implemented in Go as a library with a command line front end.
Command Line
The source code for the command line interface can be found in
cmd/feedtr/main.go.
After building, run with feedtr --config-file=config.json or
feedtr < config.json.
The config file specifies a number of outputs to be produced. Each output is created by fetching a source and transforming it.
For example, fetching the GitHub public timeline and filtering it to only include fork events:
config.json:
{
"outputs": [
{
"name": "github-fork-events.atom",
"source": "https://github.com/timeline",
"transforms": ["github-fork-events.xslt"]
}
]
}
If using the --config-file option, the files referenced in "transforms"
should be placed in a directory named transforms located in the same directory
as the config file. If loading the config from standard input, the transforms
directory should be in the current directory.
transforms/github-fork-events.xslt:
<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:atom="http://www.w3.org/2005/Atom">
<!-- Copy everything unless explicitly handled below -->
<xsl:template match="@*|*|processing-instruction()|comment()">
<xsl:copy>
<xsl:apply-templates select="*|@*|text()|processing-instruction()|comment()"/>
</xsl:copy>
</xsl:template>
<!-- Adjust the title -->
<xsl:template match="//atom:feed/atom:title">
<atom:title>GitHub Public Timeline Fork Events Feed</atom:title>
</xsl:template>
<!-- Exclude any <entry>s that don't relate to ForkEvents -->
<xsl:template match="atom:entry[not(contains(atom:id, ':ForkEvent/'))]"/>
</xsl:stylesheet>
When feedtr runs, it will create a cache directory to store feeds that have
been fetched and a outputs directory to store the transformed outputs.
In the above example, a file named github-fork-events.atom will be created in
the outputs directory. This will contain the result of running the XSLT
transform.
Transforms will only be run if the source has changed (based on the date returned over HTTP) since the last output file was written. If the configuration or transforms are changed, the relevant outputs should be manually deleted.
Library
Please refer to the code documentation along with the cmd/feedtr/main.go
source code for example usage.
Documentation
¶
Overview ¶
The feedtr package allows feeds to be fetched and transformed to create outputs. Please refer to the feedtr command line program for example usage.
Index ¶
- func FetchSources(config *Config, cache Cache) []error
- func Process(config *Config, cache Cache, transforms Transforms, outputs Outputs) []error
- type Cache
- type CacheEntry
- type Config
- type FileCache
- type FileCacheEntry
- type FileOutputEntry
- type FileOutputs
- type FileTransforms
- type Output
- type OutputEntry
- type Outputs
- type Transform
- type Transforms
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func FetchSources ¶
Fetches all the sources specified by config, storing them in the given cache.
Types ¶
type Cache ¶
type Cache interface {
// Returns a CacheEntry for a given feed source URL. A result will always be
// returned regardless of whether the entry exists.
Entry(url string) CacheEntry
}
A cache used to store fetched feed sources.
type CacheEntry ¶
type CacheEntry interface {
// Writes content from the given Reader to the cache entry. The given
// lastModified time is recorded along with the content.
Write(reader io.Reader, lastModified time.Time) error
// Returns the last modified Time of the entry or nil if nothing has been
// stored for the entry.
LastModified() *time.Time
// Returns the cached content for the cache entry as a byte slice.
Read() ([]byte, error)
}
A entry for a feed source in a Cache.
type Config ¶
type Config struct {
// A list of the outputs to be produced, each specified as an Output.
Outputs []Output
}
Specifies the outputs to produce.
type FileCache ¶
type FileCache struct {
// contains filtered or unexported fields
}
A implementation of the Cache interface using the file system to store entries.
func NewFileCache ¶
Creates and returns a new FileCache to write cache entries to the given path. The directory specified by the path is created if it does not exist.
func (*FileCache) Entry ¶
func (cache *FileCache) Entry(url string) CacheEntry
Returns a CacheEntry for a given feed source URL. A result will always be returned regardless of whether the entry exists.
type FileCacheEntry ¶
type FileCacheEntry struct {
// contains filtered or unexported fields
}
A implementation of the CacheEntry interface referencing a file on the file system.
func (*FileCacheEntry) LastModified ¶
func (entry *FileCacheEntry) LastModified() *time.Time
Returns the last modified Time of the entry or nil if nothing has been stored for the entry.
func (*FileCacheEntry) Read ¶
func (entry *FileCacheEntry) Read() ([]byte, error)
Returns the cached content for the cache entry as a byte slice.
type FileOutputEntry ¶
type FileOutputEntry struct {
// contains filtered or unexported fields
}
An implementation of the OutputEntry interface using the file system for storage.
func (*FileOutputEntry) LastModified ¶
func (entry *FileOutputEntry) LastModified() *time.Time
Returns the last modified time of the output or nil if the entry has not yet been written.
func (*FileOutputEntry) Save ¶
func (entry *FileOutputEntry) Save(content []byte) error
Stores the given content.
type FileOutputs ¶
type FileOutputs struct {
// contains filtered or unexported fields
}
An implementation of the Outputs interface using the file system for storage.
func NewFileOutputs ¶
func NewFileOutputs(path string) (*FileOutputs, error)
Returns a new FileOutputs instance to write outputs to the given path. The directory referenced by the path is created if it does not exist.
func (*FileOutputs) Entry ¶
func (outputs *FileOutputs) Entry(output Output) OutputEntry
Returns an entry for an individual configured transformed feed output.
type FileTransforms ¶
type FileTransforms struct {
// contains filtered or unexported fields
}
An implementation of the Transforms instance using the file system to store transformation files.
func NewFileTransforms ¶
func NewFileTransforms(path string) *FileTransforms
Creates an instance of FileTransforms, to load transformation files from the given path.
type Output ¶
type Output struct {
// Name of the output to be written (typically a file name).
Name string
// URL of the source feed to be fetched an transformed.
Source string
// List of transforms to be applied to the feed (i.e. file names of the
// transform files).
Transforms []string
}
An configured output to produce.
type OutputEntry ¶
type OutputEntry interface {
// Stores the given content.
Save(content []byte) error
// Returns the last modified time of the output or nil if the entry has not
// yet been written.
LastModified() *time.Time
}
An entry for a transformed feed output.
type Outputs ¶
type Outputs interface {
// Returns an entry for an individual configured transformed feed output.
Entry(output Output) OutputEntry
}
The output location for transformed feeds.
type Transform ¶
type Transform interface {
// Runs the transformation on the given input, returning an output as a byte
// slice.
Process(input []byte) ([]byte, error)
// Closes the Transform. Must be called after the instance is no longer
// needed.
Close()
}
A transformation that can be run on a feed.
type Transforms ¶
A source that can be used to obtain Transform instances.
Source Files
Directories