你的分享就是我们的动力 ---﹥

【原】Python NLP实战之一:环境准备

时间:2012-05-08 12:33来源:www.chengxuyuans.com 点击:
最近正在学习Python,看了几本关于Python入门的书和用Python进行自然语言处理的书,如《Python编程实践》、《Python基础教程》(第2版)、《Python自然语言处理》(影印版)。因为以前是学Java的,有着良好的面向对象的思维方式,所以刚开始看Python的语法,觉得Pyhon太随意了,很别扭,有不正规之嫌。而且,Python自己也正在向面向对象(OO)靠拢。但是,后来看到Python有强大的类库,尤其在自然语言处理方面有着强大的NLTK支持,我逐渐改变了对它的看法。不得不承认,Python非常简洁和清晰,很容易上手,对于有编程经验的人来说,可以快速编写程序来实现某个应用。下面是本人学习中的一些心得,与大家分享。

Python NLP实战之一:环境准备

要下载和安装的软件和资源有:
  • Python
  • PyYAML
  • NLTK
  • NLTK-Data
  • NumPy
  • Matplotlib

(一)下载地址和版本:
  • Python:http://www.python.org/getit/releases/2.7.2/ 版本:Python 2.7.2 (注:现在是2.7.3。Python已经发布3.3版了,之所以下载2.7,是因为2.x比较稳定,兼容的第三方软件多。Python官网提示:如果你不知道用哪个版本的话,就从2.7开始吧!)
  • PyYAML:http://pypi.python.org/pypi/PyYAML/ 版本:PyYAML 3.10 功能:YAML的解析工具
  • NLTK: http://www.nltk.org 版本:nltk-2.0.1 功能:自然语言工具包
  • NumPy: http://pypi.python.org/pypi/numpy 版本:numpy 1.6.1 功能:支持多维数组和线性代数
  • Matplotlib: http://sourceforge.net/projects/matplotlib/files/matplotlib/matplotlib-1.1.0/ 版本:matplotlib-1.1.0 功能:用于数据可视化的二维图库
安装都很简单,我是在Window下安装的。

(二)运行Python IDLE
Python安装完成后,运行Python集成开发环境IDLE:开始->所有程序->Python 2.7 ->IDLE (Python GUI),打开一个新的窗口,显示如下信息,表明安装成功。
Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>>

(三)下载NLTK数据包
接下来,导入NLTK工具包,然后,下载NLTK数据源。
>>> import nltk
>>> nltk.download()


注意:在导入MLTK工具包时,如果显示如下信息,表明没有安装PyYAML。
>>> import nltk
Traceback (most recent call last):
  File "<pyshell#0>", line 1, in <module>
    import nltk
  File "C:\Python27\lib\site-packages\nltk\__init__.py", line 107, in <module>
    from yamltags import *
  File "C:\Python27\lib\site-packages\nltk\yamltags.py", line 10, in <module>
    import yaml
ImportError: No module named yaml


按照(一)所列的地址下载、安装完PyYAML后,再打开Python IDLE,导入NLTK,执行nltk.download(),我的界面出现的是文字提示,书上和网上有同学说是图形界面,两者都可以吧。

Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> import nltk
>>> nltk.download()
NLTK Downloader
---------------------------------------------------------------------------
    d) Download   l) List    u) Update   c) Config   h) Help   q) Quit
---------------------------------------------------------------------------
Downloader>

选择d) Download,敲入d,再敲入l,然后按提示敲几次回车,显示的是将要下载的各种不同的数据包。
Downloader> d
Download which package (l=list; x=cancel)?
  Identifier> l
Packages:
  [ ] maxent_ne_chunker... ACE Named Entity Chunker (Maximum entropy)
  [ ] abc................. Australian Broadcasting Commission 2006
  [ ] alpino.............. Alpino Dutch Treebank
  [ ] biocreative_ppi..... BioCreAtIvE (Critical Assessment of Information
                           Extraction Systems in Biology)
  [ ] brown_tei........... Brown Corpus (TEI XML Version)
  [ ] cess_esp............ CESS-ESP Treebank
  [ ] chat80.............. Chat-80 Data Files
  [ ] brown............... Brown Corpus
  [ ] cmudict............. The Carnegie Mellon Pronouncing Dictionary (0.6)
  [ ] city_database....... City Database
  [ ] cess_cat............ CESS-CAT Treebank
  [ ] comtrans............ ComTrans Corpus Sample
  [ ] conll2002........... CONLL 2002 Named Entity Recognition Corpus
  [ ] conll2007........... Dependency Treebanks from CoNLL 2007 (Catalan
                           and Basque Subset)
  [ ] europarl_raw........ Sample European Parliament Proceedings Parallel
                           Corpus
  [ ] dependency_treebank. Dependency Parsed Treebank
  [ ] conll2000........... CONLL 2000 Chunking Corpus
Hit Enter to continue: 
  [ ] floresta............ Portuguese Treebank
  [ ] names............... Names Corpus, Version 1.3 (1994-03-29)
  [ ] gazetteers.......... Gazeteer Lists
  [ ] genesis............. Genesis Corpus
  [ ] gutenberg........... Project Gutenberg Selections
  [ ] inaugural........... C-Span Inaugural Address Corpus
  [ ] jeita............... JEITA Public Morphologically Tagged Corpus (in
                           ChaSen format)
  [ ] movie_reviews....... Sentiment Polarity Dataset Version 2.0
  [ ] ieer................ NIST IE-ER DATA SAMPLE
  [ ] nombank.1.0......... NomBank Corpus 1.0
  [ ] indian.............. Indian Language POS-Tagged Corpus
  [ ] paradigms........... Paradigm Corpus
  [ ] kimmo............... PC-KIMMO Data Files
  [ ] knbc................ KNB Corpus (Annotated blog corpus)
  [ ] langid.............. Language Id Corpus
  [ ] mac_morpho.......... MAC-MORPHO: Brazilian Portuguese news text with
                           part-of-speech tags
  [ ] machado............. Machado de Assis -- Obra Completa
  [ ] pe08................ Cross-Framework and Cross-Domain Parser
                           Evaluation Shared Task
Hit Enter to continue: 
  [ ] pl196x.............. Polish language of the XX century sixties
  [ ] pil................. The Patient Information Leaflet (PIL) Corpus
  [ ] nps_chat............ NPS Chat
  [ ] reuters............. The Reuters-21578 benchmark corpus, ApteMod
                           version
  [ ] qc.................. Experimental Data for Question Classification
  [ ] rte................. PASCAL RTE Challenges 1, 2, and 3
  [ ] ppattach............ Prepositional Phrase Attachment Corpus
  [ ] propbank............ Proposition Bank Corpus 1.0
  [ ] problem_reports..... Problem Report Corpus
  [ ] sinica_treebank..... Sinica Treebank Corpus Sample
  [ ] verbnet............. VerbNet Lexicon, Version 2.1
  [ ] state_union......... C-Span State of the Union Address Corpus
  [ ] semcor.............. SemCor 3.0
  [ ] senseval............ SENSEVAL 2 Corpus: Sense Tagged Text
  [ ] smultron............ SMULTRON Corpus Sample
  [ ] shakespeare......... Shakespeare XML Corpus Sample
  [ ] stopwords........... Stopwords Corpus
  [ ] swadesh............. Swadesh Wordlists
  [ ] switchboard......... Switchboard Corpus Sample
  [ ] toolbox............. Toolbox Sample Files
Hit Enter to continue: 
  [ ] unicode_samples..... Unicode Samples
  [ ] webtext............. Web Text Corpus
  [ ] timit............... TIMIT Corpus Sample
  [ ] ycoe................ York-Toronto-Helsinki Parsed Corpus of Old
                           English Prose
  [ ] treebank............ Penn Treebank Sample
  [ ] udhr................ Universal Declaration of Human Rights Corpus
  [ ] sample_grammars..... Sample Grammars
  [ ] book_grammars....... Grammars from NLTK Book
  [ ] spanish_grammars.... Grammars for Spanish
  [ ] wordnet............. WordNet
  [ ] wordnet_ic.......... WordNet-InfoContent
  [ ] words............... Word Lists
  [ ] tagsets............. Help on Tagsets
  [ ] basque_grammars..... Grammars for Basque
  [ ] large_grammars...... Large context-free and feature-based grammars
                           for parser comparison
  [ ] maxent_treebank_pos_tagger Treebank Part of Speech Tagger (Maximum entropy)
  [ ] rslp................ RSLP Stemmer (Removedor de Sufixos da Lingua
                           Portuguesa)
  [ ] hmm_treebank_pos_tagger Treebank Part of Speech Tagger (HMM)
Hit Enter to continue: 
  [ ] punkt............... Punkt Tokenizer Models

Collections:
  [ ] all-corpora......... All the corpora
  [ ] all................. All packages
  [ ] book................ Everything used in the NLTK Book

([*] marks installed packages)


你可以选择敲入 all-corpora,或all,或book,我选的是all。保持网络畅通,下载可能需要一段时间。显示信息如下:
Download which package (l=list; x=cancel)?
  Identifier> all
    Downloading collection 'all'
       | 
       | Downloading package 'abc' to C:\Documents and
       |     Settings\lenovo\Application Data\nltk_data...
       |   Unzipping corpora\abc.zip.
       | Downloading package 'alpino' to C:\Documents and
       |     Settings\lenovo\Application Data\nltk_data...
       |   Unzipping corpora\alpino.zip.
       | Downloading package 'biocreative_ppi' to C:\Documents and
       |     Settings\lenovo\Application Data\nltk_data...
       |   Unzipping corpora\biocreative_ppi.zip.
       | Downloading package 'brown' to C:\Documents and
       |     Settings\lenovo\Application Data\nltk_data...
       |   Unzipping corpora\brown.zip.
       | Downloading package 'brown_tei' to C:\Documents and
       |     Settings\lenovo\Application Data\nltk_data...
       |   Unzipping corpora\brown_tei.zip.
       | Downloading package 'cess_cat' to C:\Documents and
       |     Settings\lenovo\Application Data\nltk_data...
       |   Unzipping corpora\cess_cat.zip.
       | Downloading package 'cess_esp' to C:\Documents and
       |     Settings\lenovo\Application Data\nltk_data...
       |   Unzipping corpora\cess_esp.zip.
       | Downloading package 'chat80' to C:\Documents and
       ...
       ...
       ...

       | Downloading package 'book_grammars' to C:\Documents and
       |     Settings\lenovo\Application Data\nltk_data...
       |   Unzipping grammars\book_grammars.zip.
       | Downloading package 'sample_grammars' to C:\Documents and
       |     Settings\lenovo\Application Data\nltk_data...
       |   Unzipping grammars\sample_grammars.zip.
       | Downloading package 'spanish_grammars' to C:\Documents and
       |     Settings\lenovo\Application Data\nltk_data...
       |   Unzipping grammars\spanish_grammars.zip.
       | Downloading package 'basque_grammars' to C:\Documents and
       |     Settings\lenovo\Application Data\nltk_data...
       |   Unzipping grammars\basque_grammars.zip.
       | Downloading package 'large_grammars' to C:\Documents and
       |     Settings\lenovo\Application Data\nltk_data...
       |   Unzipping grammars\large_grammars.zip.
       | Downloading package 'tagsets' to C:\Documents and
       |     Settings\lenovo\Application Data\nltk_data...
       | 
     Done downloading collection 'all'
---------------------------------------------------------------------------
    d) Download   l) List    u) Update   c) Config   h) Help   q) Quit
---------------------------------------------------------------------------
Downloader>

选择q,退出下载。
Downloader> q
True

注1:在这个过程中,可以获取帮助,用h) Help,敲入:h。显示如下:
Downloader> h

Commands:
  d) Download a package or collection     u) Update out of date packages
  l) List packages & collections          h) Help
  c) View & Modify Configuration          q) Quit

---------------------------------------------------------------------------
    d) Download   l) List    u) Update   c) Config   h) Help   q) Quit
---------------------------------------------------------------------------
Downloader>

注2:下载之前,可以选择下载到本地的路径,选择c) Config,敲入:c,进入Config环境。显示默认的下载路径,如下:

Downloader> c

Data Server:
  - URL: <http://nltk.googlecode.com/svn/trunk/nltk_data/index.xml>
  - 3 Package Collections Available
  - 74 Individual Packages Available

Local Machine:
  - Data directory: C:\Documents and Settings\lenovo\Application Data\nltk_data

---------------------------------------------------------------------------
    s) Show Config   u) Set Server URL   d) Set Data Dir   m) Main Menu
---------------------------------------------------------------------------
Config>

选择d) Set Data Dir,敲入:d,键入新的下载路径:
Config> d
  New Directory> D:\nltk_data

---------------------------------------------------------------------------
    s) Show Config   u) Set Server URL   d) Set Data Dir   m) Main Menu
---------------------------------------------------------------------------
Config>

如果不改变下载路径,直接退出设置环境。
New Directory> q
  Cancelled!

---------------------------------------------------------------------------
    s) Show Config   u) Set Server URL   d) Set Data Dir   m) Main Menu
---------------------------------------------------------------------------
Config>

返回到主菜单:
Config> m

---------------------------------------------------------------------------
    d) Download   l) List    u) Update   c) Config   h) Help   q) Quit
---------------------------------------------------------------------------
Downloader>

退出下载环境:
Downloader> q
True

注3:如果上述不成功的话,你可以直接到 http://nltk.googlecode.com/svn/trunk/nltk_data/index.xml 去下载数据包,放到下载路径的目录下即可。

(四)测试NLTK数据包
导入nltk.book包中所有的东西:
>>> from nltk.book import *

显示如下,表明NLTK数据成功装载。
>>> from nltk.book import *
*** Introductory Examples for the NLTK Book ***
Loading text1, ..., text9 and sent1, ..., sent9
Type the name of the text or sentence to view it.
Type: 'texts()' or 'sents()' to list the materials.
text1: Moby Dick by Herman Melville 1851
text2: Sense and Sensibility by Jane Austen 1811
text3: The Book of Genesis
text4: Inaugural Address Corpus
text5: Chat Corpus
text6: Monty Python and the Holy Grail
text7: Wall Street Journal
text8: Personals Corpus
text9: The Man Who Was Thursday by G . K . Chesterton 1908
>>> 

(五)开始NLP工作
运行《Python自然语言处理》(影印版)中的例子,检索含“monstrous”的句子,查询词居中显示:
>>> text1.concordance('monstrous')
Building index...
Displaying 11 of 11 matches:
ong the former , one was of a most monstrous size . ... This came towards us , 
ON OF THE PSALMS . " Touching that monstrous bulk of the whale or ork we have r
ll over with a heathenish array of monstrous clubs and spears . Some were thick
d as you gazed , and wondered what monstrous cannibal and savage could ever hav
that has survived the flood ; most monstrous and most mountainous ! That Himmal
they might scout at Moby Dick as a monstrous fable , or still worse and more de
th of Radney .'" CHAPTER 55 Of the Monstrous Pictures of Whales . I shall ere l
ing Scenes . In connexion with the monstrous pictures of whales , I am strongly
ere to enter upon those still more monstrous stories of them which are to be fo
ght have been rummaged out of this monstrous cabinet there is no telling . But 
of Whale - Bones ; for Whales of a monstrous size are oftentimes cast up dead u
>>>

再看几个例子。
查看语料库中的文本信息,直接敲它的名字:
>>> text1
<Text: Moby Dick by Herman Melville 1851>

>>> text2
<Text: Sense and Sensibility by Jane Austen 1811>

查看与检索词类似的词语:
>>> text1.similar('monstrous')
Building word-context index...
abundant candid careful christian contemptible curious delightfully
determined doleful domineering exasperate fearless few gamesome
horrible impalpable imperial lamentable lazy loving

>>> text2.similar('monstrous')
Building word-context index...
very exceedingly heartily so a amazingly as extremely good great
remarkably sweet vast

查看词语的分散度图:
>>> text4.dispersion_plot(['citizens','democracy','freedom','duties','America'])

Traceback (most recent call last):
  File "<pyshell#12>", line 1, in <module>
    text4.dispersion_plot(['citizens','democracy','freedom','duties','America'])
  File "C:\Python27\lib\site-packages\nltk\text.py", line 454, in dispersion_plot
    dispersion_plot(self, words)
  File "C:\Python27\lib\site-packages\nltk\draw\dispersion.py", line 25, in 

dispersion_plot
    raise ValueError('The plot function requires the matplotlib package (aka pylab).'
ValueError: The plot function requires the matplotlib package (aka pylab).See 

http://matplotlib.sourceforge.net/
>>>

注意到这里出错了,是因为找不到画图的工具包。按照提示,从(一)中所列的网站上下载、安装Matplotlib即可。显示如下图:

文本生成的例子:
>>> text3.generate()
Building ngram index...
In the six hundredth year of Noah , and Epher , and I put the stone
from the field , And Ophir , and laid him on the morrow , that thou
dost overtake them , and herb yielding seed after him that curseth
thee , of a tree yielding fruit after his kind , cattle , and spread
his tent , and Abimael , and Lot went out . The LORD God had taken
from our father is in the inn , he gathered up his hand . And say ye
moreover , Behold , I know that my

查看文本中的词例(token)数:
>>> len(text3)
44764

查看文本中的词型(type)数,并按字母升序列出:
>>> sorted(set(text3))
['!', "'", '(', ')', ',', ',)', '.', '.)', ':', ';', ';)', '?', '?)', 'A', 'Abel', 'Abelmizraim', 'Abidah', 'Abide', 'Abimael', 'Abimelech', 'Abr', 'Abrah', 'Abraham', 'Abram', 'Accad', 'Achbor', 'Adah', 'Adam', 'Adbeel', 'Admah', 'Adullamite', 'After', 'Aholibamah', 'Ahuzzath', 'Ajah', 'Akan', 'All', 'Allonbachuth', 'Almighty', 'Almodad', 'Also', 'Alvah', 'Alvan', 'Am', 'Amal', 'Amalek', 'Amalekites', 'Ammon', 'Amorite',...]

好了,准备工作已经做完了,以后就可以开始进行各种各样的NLP工作了,特别是对于中文(汉语)的自然语言处理工作,《Python自然语言处理》(影印版)并没有给出特别的篇幅。看来,这些工作还得我们自己想办法解决啊。

转载注明地址:http://www.chengxuyuans.com/Python/40208.html

推荐文章