README
¶
scholar
scholar is a WiP Go module that implements a querier and parser for Google Scholar's output. Its classes can be used independently, but it can also be invoked as a command-line tool.
This tool is inspired by scholar.py
Usage
import "github.com/compscidr/scholar"
sch := scholar.New("profiles.json", "articles.json")
articles := sch.QueryProfile("SbUmSEAAAAAJ", 1)
for _, article := range articles {
// do something with the article
}
Features
Working:
- Queries and parses a user profile by user id to get basic publication data
- Queries each of the articles listed (up to 80) and parses the results for extra information
- Caches the profile for a day, and articles for a week (need to confirm this is working)
- This is in memory, so if the program is restarted, the cache is lost
- Configurable limit to number of articles to query in one go
- On-disk caching of the profile and articles to avoid hitting the rate limit
TODO:
- Pagination of articles
- Add throttling to avoid hitting the rate limit (figure out what the limit is)
Possible throttle info:
https://stackoverflow.com/questions/60271587/how-long-is-the-error-429-toomanyrequests-cooldown
Documentation
¶
Index ¶
- Constants
- type Article
- type Profile
- type Scholar
- func (sch *Scholar) QueryArticle(url string, article *Article, dumpResponse bool) (*Article, error)
- func (sch *Scholar) QueryProfile(user string, limit int) ([]*Article, error)
- func (sch *Scholar) QueryProfileDumpResponse(user string, queryArticles bool, limit int, dumpResponse bool) ([]*Article, error)
- func (sch *Scholar) QueryProfileWithMemoryCache(user string, limit int) ([]*Article, error)
- func (sch *Scholar) SaveCache(profileCache string, articleCache string)
Constants ¶
View Source
const AGENT = "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/81.0"
View Source
const BaseURL = "https://scholar.google.com"
View Source
const MAX_TIME_ARTICLE = time.Second * 3600 * 24 * 7 // 1 week
View Source
const MAX_TIME_PROFILE = time.Second * 3600 * 24 // 1 Day
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Article ¶
type Article struct {
Title string
Authors string
ScholarURL string
Year int
Month int
Day int
NumCitations int
Articles int // if there are more than one article within this publication (it will also tell how big the arrays below are)
Description string
PdfURL string
Journal string
Volume string
Pages string
Publisher string
ScholarCitedByURLs []string
ScholarVersionsURLs []string
ScholarRelatedURLs []string
LastRetrieved time.Time
}
type Scholar ¶
type Scholar struct {
// contains filtered or unexported fields
}
func (*Scholar) QueryArticle ¶
func (*Scholar) QueryProfile ¶
func (*Scholar) QueryProfileDumpResponse ¶
func (sch *Scholar) QueryProfileDumpResponse(user string, queryArticles bool, limit int, dumpResponse bool) ([]*Article, error)
QueryProfileDumpResponse queries the profile of a User and returns a list of Articles if queryArticles is true, it will also query the Articles for extra information which isn't present on the profile page
we may wish to set this to false if we are only interested in some article info, or we have a cache hit and we just want to get updated information from the profile page only to save requests
if dumpResponse is true, it will print the response to stdout (useful for debugging)
func (*Scholar) QueryProfileWithMemoryCache ¶ added in v1.0.5
Source Files
Directories
Click to show internal directories.
Click to hide internal directories.