cym2015 (蔡源茂)

用 Ruby 写一个没有什么用的爬虫

cym2015 — Sun, 07 Jan 2018 17:36:12 +0800

背景

前两天面试一家公司，被问道，如何获取一个网页，找出其中的所有链接，并且得到哪些链接是可以访问的，哪些不可以？

最初，我以为这是一个比较清晰简单的题目，构想了以下步骤：

获取整个 html 文档
解析整个 html 文档，找出里面的所有链接
依次访问链接中的地址，

今天想着做一个试试，却发现这三个步骤做起来，要考虑的东西还是蛮多的。

如何获取到整个 html 页面。
- 大部分网页都是由 js 动态生成的，仅通过 net/http 不能获取到完整的 html 页面。这步骤也是最麻烦的，当然，解决方案也有很多，这个帖子汇总的比较全面：https://ruby-china.org/topics/31784
如何解析 html 文档。
- 这个就相对的要单纯一点，nokogiri 是一个很容易找到的 gem 包
http 请求超时问题。
- Timeout可以搞定
多链接请求带来的并发问题。
- 线程是个好东西

未考虑到的细节，欢迎大家补充

代码

贴一段我尝试实现的代码

#ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux]
require 'timeout'
require 'net/http'
require 'nokogiri'  #gem install nokogiri 

class Web_crawler

    def initialize(url)
        @url = url

        #存储检索页面各项信息数据
        @links = []  #页面链接总数
        @success_links = [] 
        @failed_links = [] 
        @timeout_links = [] 
        @invalid_links = [] 
        @forbidden_links = [] 
        @exception_links = [] 

        #记录程序启动时间
        @begin_time = Time.now
        @timeout = 10 #访问超时设定，单位秒

    end

    def run

        res = open_url(@url)

        analysis(res)

        print_result()
    end

    private 

        #采用线程并发检查各个链接的可达性，并限制线程并发量和链接访问超时时间
        def analysis(response)
            thread = []
            html_doc = Nokogiri::HTML(response.body)
            a_lable = html_doc.xpath("//a") #找出所有的<a>标签
            a_lable.each{|each_lable|
                while true 
                    break if  Thread.list.size <= 20 #线程并发量
                    sleep 1
                end

                href1 = each_lable.attributes["href"].value
                thread << Thread.new(href1) do |href|
                    @links << href

                    #本地链接时，补齐url地址
                    if !(href.include?("http://") || href.include?("https://"))
                        href = @url + href
                    end

                    open_check(href)
                end
            }
            #两倍超时设定，保证所有链接均正常检查完成
            Timeout::timeout(@timeout*2){ 
                while true 
                    break if  (@success_links+@failed_links).size >= a_lable.size
                    sleep 1
                end
            } rescue "遍历异常结束，请检查！"
            thread.each{|t| t.kill }
        end 


        def open_check(href)
            begin
                res = open_url(href)
                case res
                when Net::HTTPSuccess
                    ##<Net::HTTPOK 200 OK readbody=true>
                    @success_links << href
                when Net::HTTPForbidden
                    #<Net::HTTPForbidden 403 Forbidden readbody=true>
                    @forbidden_links << href
                    @failed_links << href
                when Net::HTTPMovedPermanently , Net::HTTPFound
                    #<Net::HTTPMovedPermanently 301 Moved Permanently readbody=true>
                    #<Net::HTTPFound 302 Moved Temporarily readbody=true>
                    @invalid_links << href
                    @failed_links << href
                else
                    @failed_links << href
                end

            rescue URI::InvalidURIError
                @invalid_links << href
                @failed_links << href
            rescue Timeout::Error
                @timeout_links << href
                @failed_links << href
            rescue Exception => ex 
                @exception_links << href
                @failed_links << href
            end
        end

        def open_url(url)
            status = Timeout::timeout(@timeout) {
                response = Net::HTTP.get_response(URI(url))
            }
        end

        def print_result
            puts "网址 #{@url} 分析结果:"
            puts "共发现链接 #{@links.size} 个."
            puts "能成功访问 #{@success_links.size} 个."
            printf "访问失败 #{@failed_links.size} 个"
            if (@forbidden_links + @timeout_links + @invalid_links + @exception_links ).size > 0
                rs = []
                rs << "拒绝访问 #{@forbidden_links.size} 个" if  @forbidden_links !=[]
                rs << "访问超时 #{@timeout_links.size} 个" if  @timeout_links !=[]
                rs << "无效链接 #{@invalid_links.size} 个" if  @invalid_links !=[]
                rs << "访问异常 #{@exception_links.size} 个" if  @exception_links !=[]
                printf "(#{rs.join(",")})"
            end 
            puts "."
            run_time()
        end

        def run_time
            time = Time.now - @begin_time
            puts "耗时：#{time.to_i/60} 分 #{time.to_i-time.to_i/60*60} 秒 #{((time - time.to_i)*1000).to_i}毫秒"
        end
end

begin 
    web = Web_crawler.new('http://www.sangfor.com.cn')
    web.run()
rescue Exception => ex
    puts "出现未知异常，程序异常终止！"
    puts ex
    puts ex.backtrace.join("\n")
end

执行结果

部署 Rails 访问 demo 实例，总是提示 ‘Incomplete response received from application’

cym2015 — Mon, 20 Mar 2017 00:09:56 +0800

本人新手，第一次在 linux 上部署 rails，由于周围没有老前辈指引，虽已尝试部署多日，但是总是不成。无奈，来社区发帖，希望前辈们指点一二，小弟万分感激。

部署上是参照社区 wiki：https://ruby-china.org/wiki/rvm-guide 示例，在 ubuntu 上操作，除最后一步安装 nginx 是采用的自行下载源码，通过 rvmsudo passenger-install-nginx-module 外，其他均与示例中保持一致操作。

我本地的 demo 实例，rails s 启动后，是可以通过http://localhost:3000/say/hello,正常访问，但是，通过 sudo /opt/nginx/sbin/nginx 启动 nginx 后，本地访问http://localhost/say/helloIncomplete时，总是提示‘response received from application’

以下是 nginx 的配置文件


#user  nobody;
user caiyuanmao caiyuanmao;
worker_processes  1;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {
    worker_connections  1024;
}


http {
    passenger_root /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/passenger-5.1.2;
    passenger_ruby /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/wrappers/ruby;

    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

    server {
        listen       80;
        server_name  localhost;

        #charset koi8-r;

        #access_log  logs/host.access.log  main;
    root   /home/caiyuanmao/demo/public;
    passenger_enabled on;

        #location / {
        #alias /websites/secondapp/public$1;  # <-- be sure to point to 'public'!
            #passenger_base_uri /subpath;
            #passenger_app_root /websites/secondapp;
            #passenger_document_root /websites/secondapp/public;
            #passenger_enabled on;
            #root   html;
            #index  index.html index.htm;
        #}

        #error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

        # proxy the PHP scripts to Apache listening on 127.0.0.1:80
        #
        #location ~ \.php$ {
        #    proxy_pass   http://127.0.0.1;
        #}

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        #
        #location ~ \.php$ {
        #    root           html;
        #    fastcgi_pass   127.0.0.1:9000;
        #    fastcgi_index  index.php;
        #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
        #    include        fastcgi_params;
        #}

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
        #    deny  all;
        #}
    }


    # another virtual host using mix of IP-, name-, and port-based configuration
    #
    #server {
    #    listen       8000;
    #    listen       somename:8080;
    #    server_name  somename  alias  another.alias;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}


    # HTTPS server
    #
    #server {
    #    listen       443 ssl;
    #    server_name  localhost;

    #    ssl_certificate      cert.pem;
    #    ssl_certificate_key  cert.key;

    #    ssl_session_cache    shared:SSL:1m;
    #    ssl_session_timeout  5m;

    #    ssl_ciphers  HIGH:!aNULL:!MD5;
    #    ssl_prefer_server_ciphers  on;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}

}

nginx 的 error.log 如下：

[ 2017-03-19 23:47:31.0842 25983/b6810700 age/Wat/WatchdogMain.cpp:1281 ]: Starting Passenger watchdog...
[ 2017-03-19 23:47:31.1052 25986/b6778700 age/Cor/CoreMain.cpp:1070 ]: Starting Passenger core...
[ 2017-03-19 23:47:31.1054 25986/b6778700 age/Cor/CoreMain.cpp:245 ]: Passenger core running in multi-application mode.
[ 2017-03-19 23:47:31.1255 25986/b6778700 age/Cor/CoreMain.cpp:820 ]: Passenger core online, PID 25986
[ 2017-03-19 23:47:31.1467 25999/b677c700 age/Ust/UstRouterMain.cpp:529 ]: Starting Passenger UstRouter...
[ 2017-03-19 23:47:31.1481 25999/b677c700 age/Ust/UstRouterMain.cpp:342 ]: Passenger UstRouter online, PID 25999
[ 2017-03-19 23:47:34.2566 25986/b64ffb40 age/Cor/SecurityUpdateChecker.h:464 ]: Security update check: no update found (next check in 24 hours)
App 26034 stdout: 
App 26113 stdout: 
[ 2017-03-19 23:47:48.4542 25986/b59feb40 age/Cor/Con/InternalUtils.cpp:112 ]: [Client 1-1] Sending 502 response: application did not send a complete response
App 26034 stderr: [ 2017-03-19 23:47:48.4538 26113/0x84e5bbc(Worker 1) utils.rb:87 ]: *** Exception RuntimeError in Rack application object (Missing `secret_key_base` for 'production' environment, set this value in `config/secrets.yml`) (process 26113, thread 0x84e5bbc(Worker 1)):
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/railties-5.0.0.1/lib/rails/application.rb:513:in `validate_secret_key_config!'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/railties-5.0.0.1/lib/rails/application.rb:246:in `env_config'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/railties-5.0.0.1/lib/rails/engine.rb:693:in `build_request'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/railties-5.0.0.1/lib/rails/application.rb:521:in `build_request'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/railties-5.0.0.1/lib/rails/engine.rb:521:in `call'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/passenger-5.1.2/src/ruby_supportlib/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/passenger-5.1.2/src/ruby_supportlib/phusion_passenger/request_handler/thread_handler.rb:152:in `accept_and_process_next_request'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/passenger-5.1.2/src/ruby_supportlib/phusion_passenger/request_handler/thread_handler.rb:113:in `main_loop'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/passenger-5.1.2/src/ruby_supportlib/phusion_passenger/request_handler.rb:416:in `block (3 levels) in start_threads'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/passenger-5.1.2/src/ruby_supportlib/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'
App 26034 stderr: [ 2017-03-19 23:49:39.9406 26113/0x84e5bbc(Worker 1) utils.rb:87 ]: *** Exception RuntimeError in Rack application object (Missing `secret_key_base` for 'production' environment, set this value in `config/secrets.yml`) (process 26113, thread 0x84e5bbc(Worker 1)):
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/railties-5.0.0.1/lib/rails/application.rb:513:in `validate_secret_key_config!'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/railties-5.0.0.1/lib/rails/application.rb:246:in `env_config'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/railties-5.0.0.1/lib/rails/engine.rb:693:in `build_request'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/railties-5.0.0.1/lib/rails/application.rb:521:in `build_request'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/railties-5.0.0.1/lib/rails/engine.rb:521:in `call'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/passenger-5.1.2/src/ruby_supportlib/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/passenger-5.1.2/src/ruby_supportlib/phusion_passenger/request_handler/thread_handler.rb:152:in `accept_and_process_next_request'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/passenger-5.1.2/src/ruby_supportlib/phusion_passenger/request_handler/thread_handler.rb:113:in `main_loop'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/passenger-5.1.2/src/ruby_supportlib/phusion_passenger/request_handler.rb:416:in `block (3 levels) in start_threads'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/passenger-5.1.2/src/ruby_supportlib/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'
[ 2017-03-19 23:49:39.9408 25986/b517cb40 age/Cor/Con/InternalUtils.cpp:112 ]: [Client 2-1] Sending 502 response: application did not send a complete response
App 26034 stderr: [ 2017-03-19 23:53:01.8904 26113/0x84e5bbc(Worker 1) utils.rb:87 ]: *** Exception RuntimeError in Rack application object (Missing `secret_key_base` for 'production' environment, set this value in `config/secrets.yml`) (process 26113, thread 0x84e5bbc(Worker 1)):
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/railties-5.0.0.1/lib/rails/application.rb:513:in `validate_secret_key_config!'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/railties-5.0.0.1/lib/rails/application.rb:246:in `env_config'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/railties-5.0.0.1/lib/rails/engine.rb:693:in `build_request'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/railties-5.0.0.1/lib/rails/application.rb:521:in `build_request'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/railties-5.0.0.1/lib/rails/engine.rb:521:in `call'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/passenger-5.1.2/src/ruby_supportlib/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request'
[ 2017-03-19 23:53:01.8906 25986/b59feb40 age/Cor/Con/InternalUtils.cpp:112 ]: [Client 1-2] Sending 502 response: application did not send a complete response
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/passenger-5.1.2/src/ruby_supportlib/phusion_passenger/request_handler/thread_handler.rb:152:in `accept_and_process_next_request'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/passenger-5.1.2/src/ruby_supportlib/phusion_passenger/request_handler/thread_handler.rb:113:in `main_loop'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/passenger-5.1.2/src/ruby_supportlib/phusion_passenger/request_handler.rb:416:in `block (3 levels) in start_threads'
App 26034 stderr:   from /home/caiyuanmao/.rvm/gems/ruby-2.3.0@rails5/gems/passenger-5.1.2/src/ruby_supportlib/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'