Skip to content

Scrape all the posts and comments of posts from one user, and store them in MongoDB.

Notifications You must be signed in to change notification settings

avispeng/weibo_crawler

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

8 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

weibo_crawler

Updated on 2018-5-12

The program can be used to scrape all the posts and comments of the posts(only text) from one user and store them in MongoDB.

Python version: 3.6

Libraries needed: requests, BeautifulSoup, pymongo, fake-useragent

An IP proxies pool and a cookies pool are used in this project

About

Scrape all the posts and comments of posts from one user, and store them in MongoDB.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages