要求：
选一个自己感兴趣的主题。用python 编写爬虫程序，从网络上爬取相关主题的数据。对爬了的数据进行文本分析，生成词云。对文本分析结果进行解释说明。写一篇完整的博客，描述上述实现过程、遇到的问题及解决办法、数据分析思想及结论。最后提交爬取的全部数据、爬虫及数据分析源代码。
在此次作业中，我通过爬取网站‘句子迷’中方文山的歌词片段来看其作词中词频以及网友较为喜欢方老师那些句子。
在爬取的过程中主要遇到的问题是该网站对于请求过来的headers有做检查，所以需要加入headers参数，声明UA。
代码如下：
import jiebaimport requestsfrom bs4 import BeautifulSouplyrics = ''headers = {'User-Agent': 'User-Agent:*/*'}resp = requests.get('/writer/%E6%96%B9%E6%96%87%E5%B1%B1', headers=headers)resp.encoding = 'UTF-8'print(resp.status_code)soup = BeautifulSoup(resp.text, 'html.parser')page_url = '/writer/%E6%96%B9%E6%96%87%E5%B1%B1?page={}'page_last = soup.select('.pager-last')if len(page_last) > 0:page_last = page_last[0].textfor i in range(0, int(page_last)):print(i)resp = requests.get(page_url.format(i), headers=headers)resp.encoding = 'UTF-8'soup = BeautifulSoup(resp.text, 'html.parser')for a in soup.select('.xlistju'):lyrics += a.text + ' '# 保留爬取的句子with open('lyrics.txt', 'a+', encoding='UTF-8') as lyricFile:lyricFile.write(lyrics)# 加载标点符号并去除歌词中的标点with open('punctuation.txt', 'r', encoding='UTF-8') as punctuationFile:for punctuation in punctuationFile.readlines():lyrics = lyrics.replace(punctuation[0], ' ')# 加载无意义词汇with open('meaningless.txt', 'r', encoding='UTF-8') as meaninglessFile:mLessSet = set(meaninglessFile.read().split('\n'))mLessSet.add(' ')# 加载保留字with open('reservedWord.txt', 'r', encoding='UTF-8') as reservedWordFile:reservedWordSet = set(reservedWordFile.read().split('\n'))for reservedWord in reservedWordSet:jieba.add_word(reservedWord)keywordList = list(jieba.cut(lyrics))keywordSet = set(keywordList) - mLessSet # 将无意义词从词语集合中删除keywordDict = {}# 统计出词频字典for word in keywordSet:keywordDict[word] = keywordList.count(word)# 对词频进行排序keywordListSorted = list(keywordDict.items())keywordListSorted.sort(key=lambda e: e[1], reverse=True)# 将所有词频写出到txtfor topWordTup in keywordListSorted:print(topWordTup)with open('word.txt', 'a+', encoding='UTF-8') as wordFile:for i in range(0, topWordTup[1]):wordFile.write(topWordTup[0]+'\n')
上面的代码生成的word.txt中，将词汇复制到网站/做词云生成，生成后的词云图如下：
以上代码可在此下载
