Skip to content

Latest commit

 

History

History
12 lines (6 loc) · 488 Bytes

README.md

File metadata and controls

12 lines (6 loc) · 488 Bytes

weibo_crawler

Updated on 2018-5-12

The program can be used to scrape all the posts and comments of the posts(only text) from one user and store them in MongoDB.

Python version: 3.6

Libraries needed: requests, BeautifulSoup, pymongo, fake-useragent

An IP proxies pool and a cookies pool are used in this project