爬页面频率一高，服务器就返回 403，求破

imtinge — Mon, 18 Sep 2017 10:09:19 +0800

帮同学爬本地的一个新闻网站，需要获取标题、内容，发布日期和点击量

点击量是通过 js 生成的，

<script src="/count/index?id=2838670&amp;siteid=199"></script>
次

全地址

http://www.trs.gov.cn/count/index?id=2838670&siteid=199

通过连接地址，可以取到内容

document.write("195")

使用如下脚本也可以取到

pubdate, count_id = post.scan(/t(\d{8}_\d+)\./).first.first.split('_')

count_link = "http://www.xxx.cn/count/index?id=#{count_id}&siteid=#{site_id}"
begin
  html = Nokogiri::HTML(open(count_link, 'User-Agent' => 'ruby'))
  hits = html.text.scan(/\d+/).first
rescue => e
  puts "#{count_link}, #{e.message}"
  hits = 0
end

但一旦使用多线程访问 count_link 就会返回 403 错误，减少线程量，添加 sleep 可以降低出现 403 的几率，一旦出现 403 使用浏览器访问也看不到点击量（之前使用 phantomjs 也出现过抓不到点击量的情况），我让外地的朋友帮我试了也看不到，看来并不是封了我的 IP，这个 403 错误，会持续一段时间，时间到了以后，又可以正常的访问了

本来是爬数据，当然希望越快越好，但是一快就报 403，心碎，求破先谢谢大家了。

watir 报错 “`rbuf_fill': Net::ReadTimeout (Net::ReadTimeout)” 求助

imtinge — Wed, 13 Sep 2017 16:45:37 +0800

新学 watir 的小白，正常运行一段时间后，就会报错，不用 thread 也一样，请各位大神指导一下

代码如下：

def get_post_link_by_slice(page_slice)
  b = Watir::Browser.new :phantomjs

  page_slice.each do |page|
  ┆ b.goto page

  ┆ links = b.ul(class: "text-list").lis
  ┆ links.each do |li|
  ┆ ┆ link = li.a.href
  ┆ ┆ if link.end_with?('html') && link.include?('trsyw')
  ┆ ┆ ┆ puts "#{page} #{link}"
  ┆ ┆ end
  ┆ end
  end

  b.close
end

pages = get_pages()

threads = []
pages.each_slice(300) do |s|
  threads << Thread.new do
  ┆ get_post_link_by_slice(s)
  end
end

threads.each { |t| t.join }

报错如下：

/Users/tanagisa/.rbenv/versions/2.4.1/lib/ruby/2.4.0/net/protocol.rb:176:in `rbuf_fill': Net::ReadTimeout (Net::ReadTimeout)
    from /Users/tanagisa/.rbenv/versions/2.4.1/lib/ruby/2.4.0/net/protocol.rb:154:in `readuntil'
    from /Users/tanagisa/.rbenv/versions/2.4.1/lib/ruby/2.4.0/net/protocol.rb:164:in `readline'
    from /Users/tanagisa/.rbenv/versions/2.4.1/lib/ruby/2.4.0/net/http/response.rb:40:in `read_status_line'
    from /Users/tanagisa/.rbenv/versions/2.4.1/lib/ruby/2.4.0/net/http/response.rb:29:in `read_new'
    from /Users/tanagisa/.rbenv/versions/2.4.1/lib/ruby/2.4.0/net/http.rb:1446:in `block in transport_request'
    from /Users/tanagisa/.rbenv/versions/2.4.1/lib/ruby/2.4.0/net/http.rb:1443:in `catch'
    from /Users/tanagisa/.rbenv/versions/2.4.1/lib/ruby/2.4.0/net/http.rb:1443:in `transport_request'
    from /Users/tanagisa/.rbenv/versions/2.4.1/lib/ruby/2.4.0/net/http.rb:1416:in `request'
    from /Users/tanagisa/.rbenv/versions/2.4.1/lib/ruby/2.4.0/net/http.rb:1409:in `block in request'
    from /Users/tanagisa/.rbenv/versions/2.4.1/lib/ruby/2.4.0/net/http.rb:877:in `start'
    from /Users/tanagisa/.rbenv/versions/2.4.1/lib/ruby/2.4.0/net/http.rb:1407:in `request'
    from /Users/tanagisa/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/remote/http/default.rb:124:in `response_for'
    from /Users/tanagisa/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/remote/http/default.rb:78:in `request'
    from /Users/tanagisa/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/remote/http/common.rb:61:in `call'
    from /Users/tanagisa/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/remote/bridge.rb:170:in `execute'
    from /Users/tanagisa/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/remote/oss/bridge.rb:581:in `execute'
    from /Users/tanagisa/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/remote/oss/bridge.rb:52:in `get'
    from /Users/tanagisa/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/common/navigation.rb:32:in `to'
    from /Users/tanagisa/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/watir-6.8.4/lib/watir/browser.rb:82:in `goto'
    from test.rb:41:in `block in get_post_link_by_slice'
    from test.rb:40:in `each'
    from test.rb:40:in `get_post_link_by_slice'
    from test.rb:60:in `block in <main>'
    from test.rb:58:in `each_slice'
    from test.rb:58:in `with_index'
    from test.rb:58:in `<main>'

imtinge (imtinge)

爬页面频率一高，服务器就返回 403，求破

watir 报错 “`rbuf_fill': Net::ReadTimeout (Net::ReadTimeout)” 求助