一、安装Spark
1.检查基础环境hadoop,jdk
2.配置文件
vim /usr/local/spark/conf/spark-env.sh
3.环境配置
gedit ~/.bashrc
4.试运行Python代码
二、Python编程练习:英文文本的词频统计
1.准备文本文件
2.读文件
f=open("mystory.txt","r") ljjstory=f.read()
3.预处理:大小写,标点符号,停用词
#所有字母变为小写 ljjstory.lower() #将各种特殊字符和标点符号替换为空格 for ch in string.punctuation: ljjstory=ljjstory.replace(ch," ") #将停用词去除 stop_words = ['an','a','as','so','out','all','for','of','to','on','in','if','by','under','it','at','into','with','about'] lenwords=len(words) newstory=[] for i in range(lenwords): a=1 for j in range(len(stop_words)): if words[i]==stop_words[j]: continue else: if a==len(stop_words): newstory.append(words[i]) break a=a+1 continue
4.分词
words=ljjstory.split()
5.统计每个单词出现的次数
counts = {} #创建空词典存放统计结果 for word in newstory: counts[word] = counts.get(word,0) + 1 items = list(counts.items()) items.sort(key=lambda x:x[1],reverse=True)
6.按词频大小排序
while i<=len(items): word,count = items[i-1] print("{0:<20}{1}".format(word,count)) i=i+17.结果写文件
txt= open("newstory.txt", "w",encoding='UTF-8') txt.write(str(items)) print("文件写入成功") pass print(items)
运行截图: