Twisted
抓網(wǎng)頁，它有優(yōu)秀的非同步事件驅(qū)動的架構(gòu)，常見的協(xié)定都已經(jīng)有實(shí)做，包括HTTP、SMTP等等
getPage("http://www.google.com").addCallback(printPage)
一行就可以抓網(wǎng)頁
lxml
效率高，支持xpath
def getNextPageLink(self, tree):
"""Get next page link
@param tree: tree to get link
@return: Return url of next page, if there is no next page, return None
"""
paging = tree.xpath("http://span[@class='paging']")
if paging:
links = paging[0].xpath("./a[(text(), '%s')]" % self.localText['next'])
if links:
return str(links[0].get('href'))
return None
listPrice = tree.xpath("http://*[@class='priceBlockLabel']/following-sibling::*")
if listPrice:
detail['listPrice'] = self.stripMoney(listPrice[0].text)
使用的工具
FireFox的插件，XPath checker等xpath的工具，可以先用它來確定抓到的元素是正確的，然后FireBug在檢視網(wǎng)頁結(jié)構(gòu)