TKU-homework: Text Analytics

Environment

• System : Windows 10
• Python : 3.6.0
• Java jre : 1.8.0_161
• Zookeeper : 3.4.11
• Kafka : 2.11-1.1.0
• Chromedriver

Flow

• Install chromedriver
• Refert to 如何在 Windows OS 安裝 Apache Kafka
• Install Java jre
• Install Kafka
• Install Zookeeper

1. Start Zookeeper and Kafka
2. Create your topics

\kafka\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic YOUR_TOPIC

3. Write the Python web crawler with pykafka
4. Build TF-IDF Model for each article
5. pipe to the python text analytics

START

• Get the news form 政治/娛樂 with web crawler

python newsCrawler.py p
python newsCrawler.py e

• Analytics

python analytics.py p
python analytics.py e