部署 请教服务器迁移到国内后 twitter oauth 的 proxy 问题

jasl · 发布于 2013年6月28日 · 最后由 jasl 回复于 2013年7月06日 · 5058 次阅读
1107

如题,过去网站在linode上,现准备迁移到国内的UCloud,网站支持twitter oauth方式登录并且需要使用一些api,所以为了保证迁回国内后这部分功能可用,需要架设twitter api proxy 环境:debian 7 + ruby 1.9.3 + openssl 1.0.1e,已关闭防火墙 我尝试过了两个方案 1 利用GoAgent 3.0.1,然后设置omniauth的proxy到GoAgent端口,开发机osx 10.8正常,但是在ucloud上GoAgent日志

INFO - [Jun 28 02:48:04] 127.0.0.1:45539 "AGENT CONNECT api.twitter.com:443 HTTP/1.1" - -
ERROR - [Jun 28 02:48:04] ssl.wrap_socket(self.connection=<socket at 0x1c90210 fileno=7 sock=127.0.0.1:8087>) failed: [Errno 1] _ssl.c:504: error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca

rails日志

(twitter) Request phase initiated.
(twitter) Authentication failure! service_unavailable: OpenSSL::SSL::SSLError, SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

GoAgent开启关闭https mode结果均一样 使用curl测试curl https://api.twitter.com -x 127.0.0.1:8087测试,同样得到unknown ca的错误 但是curl https://api.twitter.com -x 127.0.0.1:8087 --cacert /usr/local/goagent/local/CA.crt 则能获取到html 但是配置GoAgent的时候本身就会自动导入ca,经检查确实已经导入了证书到/etc/ssl/certs,并且开发机不需要指定ca也能正常得到结果 测试数次,结果比较稳定

2 在linode的机器上用nginx来转发api实现proxy,nginx配置文件如下

server {
    listen          80;
    server_name     kexue.knewone.com;

    access_log      /var/log/nginx/twitter.access_log;

    location / {
        proxy_pass              https://api.twitter.com/;
        proxy_redirect          off;
        proxy_buffering off;
        proxy_cache off;
        #proxy_set_header        X-Real-IP       $remote_addr;
        #proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        #proxy_set_header       cookie          $http_cookie;
        proxy_pass_header       User-Agent;
        proxy_pass_header       Authorization;
        proxy_pass_header       Content-Type;
        proxy_pass_header       WWW-Authenticate;
        proxy_pass_header       Host;
    }
}

可以确定的是,请求内容完整的发送到proxy上了,但返回401 unauthorized rails日志

Started GET "/users/auth/twitter" for 127.0.0.1 at 2013-06-28 03:17:32 +0800

OAuth::Unauthorized (401 Unauthorized):
  oauth (0.4.7) lib/oauth/consumer.rb:216:in `token_request'
  oauth (0.4.7) lib/oauth/consumer.rb:136:in `get_request_token'
  omniauth-oauth (1.0.1) lib/omniauth/strategies/oauth.rb:29:in `request_phase'
  omniauth-twitter (1.0.0) lib/omniauth/strategies/twitter.rb:63:in `request_phase'
  omniauth (1.1.4) lib/omniauth/strategy.rb:214:in `request_call'
  omniauth (1.1.4) lib/omniauth/strategy.rb:181:in `call!'
  omniauth (1.1.4) lib/omniauth/strategy.rb:164:in `call'
  omniauth (1.1.4) lib/omniauth/strategy.rb:184:in `call!'
  omniauth (1.1.4) lib/omniauth/strategy.rb:164:in `call'
  newrelic_rpm (3.6.4.122) lib/new_relic/rack/error_collector.rb:12:in `call'
  newrelic_rpm (3.6.4.122) lib/new_relic/rack/agent_hooks.rb:22:in `call'
  newrelic_rpm (3.6.4.122) lib/new_relic/rack/browser_monitoring.rb:16:in `call'
  newrelic_rpm (3.6.4.122) lib/new_relic/rack/developer_mode.rb:28:in `call'
  mongoid (3.1.4) lib/rack/mongoid/middleware/identity_map.rb:34:in `block in call'
  mongoid (3.1.4) lib/mongoid/unit_of_work.rb:39:in `unit_of_work'
  mongoid (3.1.4) lib/rack/mongoid/middleware/identity_map.rb:34:in `call'
  warden (1.2.1) lib/warden/manager.rb:35:in `block in call'
  warden (1.2.1) lib/warden/manager.rb:34:in `catch'
  warden (1.2.1) lib/warden/manager.rb:34:in `call'
  actionpack (3.2.13) lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
  rack (1.4.5) lib/rack/etag.rb:23:in `call'
  rack (1.4.5) lib/rack/conditionalget.rb:25:in `call'
  actionpack (3.2.13) lib/action_dispatch/middleware/head.rb:14:in `call'
  actionpack (3.2.13) lib/action_dispatch/middleware/params_parser.rb:21:in `call'
  actionpack (3.2.13) lib/action_dispatch/middleware/flash.rb:242:in `call'
  rack (1.4.5) lib/rack/session/abstract/id.rb:210:in `context'
  rack (1.4.5) lib/rack/session/abstract/id.rb:205:in `call'
  actionpack (3.2.13) lib/action_dispatch/middleware/cookies.rb:341:in `call'
  actionpack (3.2.13) lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
  activesupport (3.2.13) lib/active_support/callbacks.rb:405:in `_run__53745951561906208__call__2319728286447542957__callbacks'
  activesupport (3.2.13) lib/active_support/callbacks.rb:405:in `__run_callback'
  activesupport (3.2.13) lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
  activesupport (3.2.13) lib/active_support/callbacks.rb:81:in `run_callbacks'
  actionpack (3.2.13) lib/action_dispatch/middleware/callbacks.rb:27:in `call'
  actionpack (3.2.13) lib/action_dispatch/middleware/reloader.rb:65:in `call'
  actionpack (3.2.13) lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
  airbrake (3.1.12) lib/airbrake/rails/middleware.rb:13:in `call'
  actionpack (3.2.13) lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
  actionpack (3.2.13) lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
  railties (3.2.13) lib/rails/rack/logger.rb:32:in `call_app'
  railties (3.2.13) lib/rails/rack/logger.rb:16:in `block in call'
  activesupport (3.2.13) lib/active_support/tagged_logging.rb:22:in `tagged'
  railties (3.2.13) lib/rails/rack/logger.rb:16:in `call'
  quiet_assets (1.0.2) lib/quiet_assets.rb:18:in `call_with_quiet_assets'
  actionpack (3.2.13) lib/action_dispatch/middleware/request_id.rb:22:in `call'
  rack (1.4.5) lib/rack/methodoverride.rb:21:in `call'
  rack (1.4.5) lib/rack/runtime.rb:17:in `call'
  activesupport (3.2.13) lib/active_support/cache/strategy/local_cache.rb:72:in `call'
  rack (1.4.5) lib/rack/lock.rb:15:in `call'
  actionpack (3.2.13) lib/action_dispatch/middleware/static.rb:63:in `call'
  airbrake (3.1.12) lib/airbrake/user_informer.rb:16:in `_call'
  airbrake (3.1.12) lib/airbrake/user_informer.rb:12:in `call'
  railties (3.2.13) lib/rails/engine.rb:479:in `call'
  railties (3.2.13) lib/rails/application.rb:223:in `call'
  railties (3.2.13) lib/rails/railtie/configurable.rb:30:in `method_missing'
  /Users/jasl/Library/Application Support/Pow/Versions/0.4.1/node_modules/nack/lib/nack/server.rb:145:in `handle'
  /Users/jasl/Library/Application Support/Pow/Versions/0.4.1/node_modules/nack/lib/nack/server.rb:99:in `rescue in block (2 levels) in start'
  /Users/jasl/Library/Application Support/Pow/Versions/0.4.1/node_modules/nack/lib/nack/server.rb:96:in `block (2 levels) in start'
  /Users/jasl/Library/Application Support/Pow/Versions/0.4.1/node_modules/nack/lib/nack/server.rb:86:in `each'
  /Users/jasl/Library/Application Support/Pow/Versions/0.4.1/node_modules/nack/lib/nack/server.rb:86:in `block in start'
  /Users/jasl/Library/Application Support/Pow/Versions/0.4.1/node_modules/nack/lib/nack/server.rb:66:in `loop'
  /Users/jasl/Library/Application Support/Pow/Versions/0.4.1/node_modules/nack/lib/nack/server.rb:66:in `start'
  /Users/jasl/Library/Application Support/Pow/Versions/0.4.1/node_modules/nack/lib/nack/server.rb:13:in `run'
  /Users/jasl/Library/Application Support/Pow/Versions/0.4.1/node_modules/nack/bin/nack_worker:4:in `<main>'

另外有篇 文章 里的提示尝试使用OAuth 2方式认证,nginx配置如下

server {
  access_log      /var/log/nginx/proxy.access_log;

  # If your want to secure your proxy with SSL, replace with the appropriate SSL configuration.
  listen 80;

  # Replace this with the name of the domain you wish to run your proxy on.
  server_name kexue.knewone.com;

  # The Twitter proxy code!
  location / {
    proxy_buffering off;
    proxy_cache off;
    proxy_ignore_headers X-Accel-Expires Expires Cache-Control Set-Cookie;

    # Hide Twitter's own caching headers - we're applying our own.
    proxy_hide_header X-Accel-Expires;
    proxy_hide_header Expires;
    proxy_hide_header Cache-Control;
    proxy_hide_header pragma;
    proxy_hide_header set-cookie;

    proxy_pass_header Content-type;
    proxy_pass_header WWW-Authenticate;

    # Set the correct host name to connect to the Twitter API.
    proxy_set_header Host api.twitter.com;

    # Add authentication headers - edit and add in your own bearer token.
    proxy_set_header Authorization "Bearer 哔~~~";
    # Actually proxy the request to Twitter API!
    proxy_pass https://api.twitter.com;
  }

返回403 Forbidden rails日志

Started GET "/users/auth/twitter" for 127.0.0.1 at 2013-06-28 03:14:45 +0800

OAuth::Unauthorized (403 Forbidden):
  oauth (0.4.7) lib/oauth/consumer.rb:216:in `token_request'
  oauth (0.4.7) lib/oauth/consumer.rb:136:in `get_request_token'
  omniauth-oauth (1.0.1) lib/omniauth/strategies/oauth.rb:29:in `request_phase'
  omniauth-twitter (1.0.0) lib/omniauth/strategies/twitter.rb:63:in `request_phase'
  omniauth (1.1.4) lib/omniauth/strategy.rb:214:in `request_call'
  omniauth (1.1.4) lib/omniauth/strategy.rb:181:in `call!'
  omniauth (1.1.4) lib/omniauth/strategy.rb:164:in `call'
  omniauth (1.1.4) lib/omniauth/strategy.rb:184:in `call!'
  omniauth (1.1.4) lib/omniauth/strategy.rb:164:in `call'
  newrelic_rpm (3.6.4.122) lib/new_relic/rack/error_collector.rb:12:in `call'
  newrelic_rpm (3.6.4.122) lib/new_relic/rack/agent_hooks.rb:22:in `call'
  newrelic_rpm (3.6.4.122) lib/new_relic/rack/browser_monitoring.rb:16:in `call'
  newrelic_rpm (3.6.4.122) lib/new_relic/rack/developer_mode.rb:28:in `call'
  mongoid (3.1.4) lib/rack/mongoid/middleware/identity_map.rb:34:in `block in call'
  mongoid (3.1.4) lib/mongoid/unit_of_work.rb:39:in `unit_of_work'
  mongoid (3.1.4) lib/rack/mongoid/middleware/identity_map.rb:34:in `call'
  warden (1.2.1) lib/warden/manager.rb:35:in `block in call'
  warden (1.2.1) lib/warden/manager.rb:34:in `catch'
  warden (1.2.1) lib/warden/manager.rb:34:in `call'
  actionpack (3.2.13) lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
  rack (1.4.5) lib/rack/etag.rb:23:in `call'
  rack (1.4.5) lib/rack/conditionalget.rb:25:in `call'
  actionpack (3.2.13) lib/action_dispatch/middleware/head.rb:14:in `call'
  actionpack (3.2.13) lib/action_dispatch/middleware/params_parser.rb:21:in `call'
  actionpack (3.2.13) lib/action_dispatch/middleware/flash.rb:242:in `call'
  rack (1.4.5) lib/rack/session/abstract/id.rb:210:in `context'
  rack (1.4.5) lib/rack/session/abstract/id.rb:205:in `call'
  actionpack (3.2.13) lib/action_dispatch/middleware/cookies.rb:341:in `call'
  actionpack (3.2.13) lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
  activesupport (3.2.13) lib/active_support/callbacks.rb:405:in `_run__53745951561906208__call__2319728286447542957__callbacks'
  activesupport (3.2.13) lib/active_support/callbacks.rb:405:in `__run_callback'
  activesupport (3.2.13) lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
  activesupport (3.2.13) lib/active_support/callbacks.rb:81:in `run_callbacks'
  actionpack (3.2.13) lib/action_dispatch/middleware/callbacks.rb:27:in `call'
  actionpack (3.2.13) lib/action_dispatch/middleware/reloader.rb:65:in `call'
  actionpack (3.2.13) lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
  airbrake (3.1.12) lib/airbrake/rails/middleware.rb:13:in `call'
  actionpack (3.2.13) lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
  actionpack (3.2.13) lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
  railties (3.2.13) lib/rails/rack/logger.rb:32:in `call_app'
  railties (3.2.13) lib/rails/rack/logger.rb:16:in `block in call'
  activesupport (3.2.13) lib/active_support/tagged_logging.rb:22:in `tagged'
  railties (3.2.13) lib/rails/rack/logger.rb:16:in `call'
  quiet_assets (1.0.2) lib/quiet_assets.rb:18:in `call_with_quiet_assets'
  actionpack (3.2.13) lib/action_dispatch/middleware/request_id.rb:22:in `call'
  rack (1.4.5) lib/rack/methodoverride.rb:21:in `call'
  rack (1.4.5) lib/rack/runtime.rb:17:in `call'
  activesupport (3.2.13) lib/active_support/cache/strategy/local_cache.rb:72:in `call'
  rack (1.4.5) lib/rack/lock.rb:15:in `call'
  actionpack (3.2.13) lib/action_dispatch/middleware/static.rb:63:in `call'
  airbrake (3.1.12) lib/airbrake/user_informer.rb:16:in `_call'
  airbrake (3.1.12) lib/airbrake/user_informer.rb:12:in `call'
  railties (3.2.13) lib/rails/engine.rb:479:in `call'
  railties (3.2.13) lib/rails/application.rb:223:in `call'
  railties (3.2.13) lib/rails/railtie/configurable.rb:30:in `method_missing'
  /Users/jasl/Library/Application Support/Pow/Versions/0.4.1/node_modules/nack/lib/nack/server.rb:145:in `handle'
  /Users/jasl/Library/Application Support/Pow/Versions/0.4.1/node_modules/nack/lib/nack/server.rb:99:in `rescue in block (2 levels) in start'
  /Users/jasl/Library/Application Support/Pow/Versions/0.4.1/node_modules/nack/lib/nack/server.rb:96:in `block (2 levels) in start'
  /Users/jasl/Library/Application Support/Pow/Versions/0.4.1/node_modules/nack/lib/nack/server.rb:86:in `each'
  /Users/jasl/Library/Application Support/Pow/Versions/0.4.1/node_modules/nack/lib/nack/server.rb:86:in `block in start'
  /Users/jasl/Library/Application Support/Pow/Versions/0.4.1/node_modules/nack/lib/nack/server.rb:66:in `loop'
  /Users/jasl/Library/Application Support/Pow/Versions/0.4.1/node_modules/nack/lib/nack/server.rb:66:in `start'
  /Users/jasl/Library/Application Support/Pow/Versions/0.4.1/node_modules/nack/lib/nack/server.rb:13:in `run'
  /Users/jasl/Library/Application Support/Pow/Versions/0.4.1/node_modules/nack/bin/nack_worker:4:in `<main>'

另外,我大概看过一些twiiter api proxy工具,都比较老,还在使用即将作废的1.0版api,所以就不考虑了

请教:

  • 有什么好的国内主机使用twitter oauth api的方案?
  • 我感觉我想到的两个方案理论上都是可行的,但哪里出问题或者我没考虑到导致失败呢?
  • 有什么办法截取代理服务器出去的包?tcpdump nc都是针对socket的,对于只想观察request来说 非常重,而且难用。。。

感谢!

@luikore @hooopo @yedingding @quakewang @bhuztez @lgn21st

共收到 17 条回复
2880

http proxy 是明文的, 和直接访问没区别, 必墙...

goagent 原理应该和 gapproxy 差不多, 就算加了证书, 由于服务器是知道连接内容的, 改不了中间人角色, 必然 ssl 验证不通过.


socks 隧道型的就没这个问题. 本地开个 shadowsocks / stochastic-socks 客户端, 然后用本地的 socks5 代理就可以了. 防火墙不用关.

如果不需要跑国内请求, 可以用 socksify-ruby 猴子补丁掉 Net::HTTP, 就不用改任何代码了.

如果熟悉 oauth 原理, 用 curl 跑 oauth 也很简单... curl --socks5-hostname localhost:6789 api.twitter.com...


还有一个办法是配置 vpn

1107

#1楼 @luikore 忘记了...没用socks主要是担心长期保持连接会封掉linode的ip,坛子里貌似有一些先例吧?我也说不好...现在有点迷糊

2880

#2楼 @jasl ssh 隧道是有这个问题, 只有一个连接 hang 住, 而且并发时很卡, 不开混淆的话特征很容易学习, 但 s*socks 是每请求一个连接非阻塞的, 特征和 ssh 不同, 暂时好像还没出问题 (将来咋样我也说不好...)

1107

#3楼 @luikore 唔 那我就尝试一下

1107

#3楼 @luikore 吕哥一席话,胜读十年书啊!

96

对于这种情况,我是死活都不会让老板把服务器移到国内的,太折腾了

162

方法1, 是因为中间人证书警告?设置忽略试试看: OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE

1107

#1楼 @luikore 睡醒之后 表示。。。没理解 - - 你能详细的讲讲么,如果是这样的话,goagent这类工具是怎么达到翻墙目的的?

1107

#7楼 @quakewang cool...简单粗暴。。

162

#9楼 @jasl 昨天没仔细看贴,嫌太粗暴的话,你试试看设置ca file应该也可以。 config.omniauth :twitter, "xxx", "xxx", {:scope => 'xxx,xxx', :client_options => {:ssl => {:ca_file => '/usr/local/goagent/local/CA.crt'}}}

2880

#8楼 @jasl 因为 goagent 部署在 appengine 的限制, 远端是只能用 http client 连接而不能直接开 TCP socket, 而 http client 必须知道连接的内容 (header, path 之类的), 所以不算加密传输层... 隧道是服务器不知道连接的内容, 收到什么就发什么

@quakewang 不忽略证书的话, 可能要自己生成域名为 twitter.com 的假证书?

1107

#11楼 @luikore goagent似乎包含twitter假证书的,我好像翻过证书文件

1107

#6楼 @yggg 有些东西必须国内才能做的,我也不喜欢国内的主机

1107

@luikore @quakewang 解决ssl证书之后还有一个坑,有时候Ruby 2.0 的net库在做inflate的时候会触发Zlib::DataError: incorrect header check,omniauth走proxy进行oauth的时候刚好会触发,不太懂原理,能讲讲么?

http://www.civet.ws/2013/06/adventures-with-ruby-2-0-0-and-zlib/ https://gist.github.com/geopet/5782836 这是两篇讨论

查了些资料,rest-client 做过处理,免疫这个问题。 于是效法他,我写了个很脏的猴子补丁https://gist.github.com/jasl/5939460

2880

#14楼 @jasl omniauth 走的什么类型的 proxy? http?

如果是 goagent, 可能是它没处理好 header 的原因

如果原本的响应是 transfer-encoding: chunked, 然后 goagent 把 body decode 出来, 加上 content-length 返回给你, 但没有删除 chunked 项, 客户端就按照 chunked 去 decode 已经 decode 过的内容...

2880

响应 header 中包含 transfer-encoding: chunked 的时候, 客户端就不去读 content-length, 而是一段一段的读内容, 往往适用于发 header 的时候还不知道 body 有多长的情况. chunked encoding 的 body 里, 每段内容以 16 进制数字标识这段内容的长度, 接着是对应长度的数据, 后面再接上 "\r\n" 做分隔符. 当碰到 "\0\r\n\r\n" 的时候, body 就结束了.

content-encoding: gzip 是在处理完 transfer-encoding 后处理的, 就是把内容扔给 gzip 去解压.

Net::HTTP 对两种 header 都做了考虑. 除非 header 被 http 代理搞乱掉 (解压了却没删掉 content-encoding, 或者解码了 chunked 缺忘记改 transfer-encoding), 是不会出问题的.

隧道代理就完全不会出 http 代理的这种问题.

1107

#16楼 @luikore 涨姿势!

需要 登录 后方可回复, 如果你还没有账号请点击这里 注册