veryscrape.scrapers package¶
Submodules¶
veryscrape.scrapers.google module¶
-
class
veryscrape.scrapers.google.
Google
(*args, proxy_pool=None, **kwargs)[source]¶ Bases:
veryscrape.scrape.SearchEngineScraper
-
item_gen
¶ alias of
ArticleGen
-
session_class
¶ alias of
GoogleSession
-
source
= 'article'¶
-
-
class
veryscrape.scrapers.google.
GoogleSession
(*args, proxy_pool=None, **kwargs)[source]¶ Bases:
veryscrape.session.Session
-
error_on_failure
= False¶
-
retries_to_error
= 2¶
-
veryscrape.scrapers.reddit module¶
-
class
veryscrape.scrapers.reddit.
CommentGen
(q, topic='', source='')[source]¶ Bases:
veryscrape.items.ItemGenerator
-
removed_comments
= {'[removed]', '[deleted]'}¶
-
-
class
veryscrape.scrapers.reddit.
Reddit
(key, secret, *, proxy_pool=None)[source]¶ Bases:
veryscrape.scrape.Scraper
-
item_gen
¶ alias of
CommentGen
-
scrape_every
= 600¶
-
session_class
¶ alias of
RedditSession
-
source
= 'reddit'¶
-
veryscrape.scrapers.twingly module¶
veryscrape.scrapers.twitter module¶
-
class
veryscrape.scrapers.twitter.
TweetGen
(q, topic='', source='')[source]¶ Bases:
veryscrape.items.ItemGenerator
-
last_item
= None¶
-
-
class
veryscrape.scrapers.twitter.
Twitter
(key, secret, token, token_secret, *, proxy_pool=None)[source]¶ Bases:
veryscrape.scrape.Scraper
-
session_class
¶ alias of
TwitterSession
-
source
= 'twitter'¶
-
-
class
veryscrape.scrapers.twitter.
TwitterSession
(*args, **kwargs)[source]¶ Bases:
veryscrape.session.OAuth1Session
-
base_url
= 'https://stream.twitter.com/1.1/'¶
-
Module contents¶
-
class
veryscrape.scrapers.
Twitter
(key, secret, token, token_secret, *, proxy_pool=None)[source]¶ Bases:
veryscrape.scrape.Scraper
-
item_gen
¶ alias of
TweetGen
-
session_class
¶ alias of
TwitterSession
-
source
= 'twitter'¶
-
-
class
veryscrape.scrapers.
Reddit
(key, secret, *, proxy_pool=None)[source]¶ Bases:
veryscrape.scrape.Scraper
-
item_gen
¶ alias of
CommentGen
-
scrape_every
= 600¶
-
session_class
¶ alias of
RedditSession
-
source
= 'reddit'¶
-
-
class
veryscrape.scrapers.
Google
(*args, proxy_pool=None, **kwargs)[source]¶ Bases:
veryscrape.scrape.SearchEngineScraper
-
item_gen
¶ alias of
ArticleGen
-
session_class
¶ alias of
GoogleSession
-
source
= 'article'¶
-
-
class
veryscrape.scrapers.
Twingly
(api_key, *, proxy_pool=None)[source]¶ Bases:
veryscrape.scrape.SearchEngineScraper
-
item_gen
¶ alias of
BlogGen
-
source
= 'blog'¶
-