Erlang/Elixir Elixir Phoenix 如何用 10 分钟 50 行代码快速撸一个 http 透传通知服务

edwardzhou · August 07, 2017 · Last by edwardzhou replied at August 25, 2017 · 8775 hits

我们购买 Github 服务来托管源代码,用 Jenkins 来做持续集成,因此需要让 github 能通知 Jenkins 各种通知事件,触发持续集成构建。

公司网络环境为 Internet <------ 光猫 <------ 路由器/交换机 <------ 内网 光猫是是电信提供的 100M 光纤,自动拨号上网,网段 192.168.1.x,不是路由器控制拨号上网;路由器与内网为 192.168.2.x。

Jenkins 部署在内网上的 CentOS 主机,IP 为 192.168.2.231。

由于光猫是被电信阉割了的,许多功能(如端口转发)都被屏蔽了,而路由器在上设置端口转发也起不到作用。

即在 CentOS 上使用花生壳之类的,也无法让 Github 直接连通内网的 Jenkins。

我们只能通过让 Jenkins 去定时轮询 Github 感知代码合并请求。这种方式有两个问题:

  1. 及时性不够;
  2. 代码合并事件无法感知;

为了弥补这个问题,我们的 CI 半手工触发,有些蛋疼。

最近正好在看 Elixir 和 Phoenix, 两者分别与 Ruby 和 RubyOnRails 相对应。

elixir_logo.png

传送门: http://elixir-lang.org/ http://www.phoenixframework.org/

有一句话说得很好,当你手里拿着一把锤子,但看到什么都是钉子。😂

因此,打算自己快速撸一个透传通知服务给 github -> Jenkins 用。

方案比较简单,用 Elixir Phoenix 框架打一个应用 jenkins_hook_proxy,提供 web 服务与 websocket 服务,运行在公网云主机上; 内网 forwarder 用 websocket 连接 jenkins_hook_proxy, 1) Jenkins_hook_proxy 收到 GitHub 的通知时,将请求打包由 websocket 通知会内网, 2) 内网 forwarder 收到 websocket 通知后,拆解请求后转给 Jenkins 3) Jenkins 完成构建后,将构建结果通知 GitHub

GitHub ---http post--> [jenkins_hook_proxy] <---websocket---> [forwarder] ----http post----> Jenkins ----(CI 构建结构) http post----> GitHub

由于 Phoenix 的 channel 为提供了非常方便且健壮的 web 前端浏览器 js 脚本,为了能直接利用它的 js 脚本,forwarder 就用 nodejs 编写。

整个过程下来,发现 jenkins_hook_proxy 和 forwarder 我们写的代码也就 20 句左右。

环境准备

Erlang 安装

CentOS 上安装 Erlang 麻烦些,需要下载 pre-built binary package 安装 进入 https://www.erlang-solutions.com/resources/download.html 下载 19.3 的 rpm 包,然后用 root 用户安装 (sudo rpm -i eslxxx.rpm), 根据提示安装所需的依赖包 (eg. wxBase, wxGTK 等)。

Elixir

下载 elixir 源码 https://github.com/elixir-lang/elixir/archive/v1.4.2.zip 解压后,make && sudo make install 安装。 安装完运行 iex --version 看看是否正常。

安装 Phoenix framework

http://www.phoenixframework.org/docs/installation

$ mix local.hex 由于安装的是 1.3rc 版,包为 phx_new.ex。phoenix_new.ez 是 1.2.x 版本。 $ mix archive.install https://github.com/phoenixframework/archives/raw/master/phx_new.ez

安装 NodeJS

phoenix 要求安装的有 NodeJS 用于编译 assets。 $ sudo yum install node

创建运行在外网的项目 Jenkins_hook_proxy

$ mix phx.new jenkins_hook_proxy --database mysql

* creating jenkins_hook_proxy/config/config.exs
* creating jenkins_hook_proxy/config/dev.exs
* creating jenkins_hook_proxy/config/prod.exs
* creating jenkins_hook_proxy/config/prod.secret.exs
* creating jenkins_hook_proxy/config/test.exs
* creating jenkins_hook_proxy/lib/jenkins_hook_proxy/application.ex
* creating jenkins_hook_proxy/lib/jenkins_hook_proxy/web/channels/user_socket.ex
* creating jenkins_hook_proxy/lib/jenkins_hook_proxy/web/views/error_helpers.ex
* creating jenkins_hook_proxy/lib/jenkins_hook_proxy/web/views/error_view.ex
* creating jenkins_hook_proxy/lib/jenkins_hook_proxy/web/endpoint.ex
* creating jenkins_hook_proxy/lib/jenkins_hook_proxy/web/router.ex
* creating jenkins_hook_proxy/lib/jenkins_hook_proxy/web/web.ex
* creating jenkins_hook_proxy/mix.exs
* creating jenkins_hook_proxy/README.md
* creating jenkins_hook_proxy/test/support/channel_case.ex
* creating jenkins_hook_proxy/test/support/conn_case.ex
* creating jenkins_hook_proxy/test/test_helper.exs
* creating jenkins_hook_proxy/test/web/views/error_view_test.exs
* creating jenkins_hook_proxy/lib/jenkins_hook_proxy/web/gettext.ex
* creating jenkins_hook_proxy/priv/gettext/en/LC_MESSAGES/errors.po
* creating jenkins_hook_proxy/priv/gettext/errors.pot
* creating jenkins_hook_proxy/lib/jenkins_hook_proxy/repo.ex
* creating jenkins_hook_proxy/priv/repo/seeds.exs
* creating jenkins_hook_proxy/test/support/data_case.ex
* creating jenkins_hook_proxy/lib/jenkins_hook_proxy/web/controllers/page_controller.ex
* creating jenkins_hook_proxy/lib/jenkins_hook_proxy/web/templates/layout/app.html.eex
* creating jenkins_hook_proxy/lib/jenkins_hook_proxy/web/templates/page/index.html.eex
* creating jenkins_hook_proxy/lib/jenkins_hook_proxy/web/views/layout_view.ex
* creating jenkins_hook_proxy/lib/jenkins_hook_proxy/web/views/page_view.ex
* creating jenkins_hook_proxy/test/web/controllers/page_controller_test.exs
* creating jenkins_hook_proxy/test/web/views/layout_view_test.exs
* creating jenkins_hook_proxy/test/web/views/page_view_test.exs
* creating jenkins_hook_proxy/.gitignore
* creating jenkins_hook_proxy/assets/brunch-config.js
* creating jenkins_hook_proxy/assets/css/app.css
* creating jenkins_hook_proxy/assets/css/phoenix.css
* creating jenkins_hook_proxy/assets/js/app.js
* creating jenkins_hook_proxy/assets/js/socket.js
* creating jenkins_hook_proxy/assets/package.json
* creating jenkins_hook_proxy/assets/static/robots.txt
* creating jenkins_hook_proxy/assets/static/images/phoenix.png
* creating jenkins_hook_proxy/assets/static/favicon.ico

Fetch and install dependencies? [Yn] 

敲回车安装依赖包。

* running mix deps.get
* running mix deps.compile
* running cd assets && npm install && node node_modules/brunch/bin/brunch build

We are all set! Run your Phoenix application:

    $ cd jenkins_hook_proxy
    $ mix phx.server

You can also run your app inside IEx (Interactive Elixir) as:

    $ iex -S mix phx.server

Before moving on, configure your database in config/dev.exs and run:

    $ mix ecto.create

创建 Channel (websocket 服务) JenkinsChannel. JenkinsChannel 仅作为 websocket 通道用于回传信息给内网,使用默认生成的代码就足够了。不是我们自己动手写的,不纳入行数计算 😜


$ cd Jenkins_hook_proxy
$ mix phx.gen.channel Jenkins
* creating lib/jenkins_hook_proxy/web/channels/jenkins_channel.ex
* creating test/web/channels/jenkins_channel_test.exs

Add the channel to your `lib/jenkins_hook_proxy/web/channels/user_socket.ex` handler, for example:

    channel "jenkins:lobby", JenkinsHookProxy.Web.JenkinsChannel


编辑 user_socket.ex , 把上面那句话加进去。 这算一行代码吧 😄

$ vi lib/jenkins_hook_proxy/web/channels/user_socket.ex

defmodule JenkinsHookProxy.Web.UserSocket do
  use Phoenix.Socket

  ## Channels
  channel "jenkins:lobby", JenkinsHookProxy.Web.JenkinsChannel

  ## Transports
  transport :websocket, Phoenix.Transports.WebSocket
  # transport :longpoll, Phoenix.Transports.LongPoll

  # Socket params are passed from the client and can
  # be used to verify and authenticate a user. After
  # verification, you can put default assigns into
  # the socket that will be set for all channels, ie
  #
  #     {:ok, assign(socket, :user_id, verified_user_id)}
  #
  # To deny connection, return `:error`.
  #
  # See `Phoenix.Token` documentation for examples in
  # performing token verification on connect.
  def connect(_params, socket) do
    {:ok, socket}
  end

  # Socket id's are topics that allow you to identify all sockets for a given user:
  #
  #     def id(socket), do: "user_socket:#{socket.assigns.user_id}"
  #
  # Would allow you to broadcast a "disconnect" event and terminate
  # all active sockets and channels for a given user:
  #
  #     JenkinsHookProxy.Web.Endpoint.broadcast("user_socket:#{user.id}", "disconnect", %{})
  #
  # Returning `nil` makes this socket anonymous.
  def id(_socket), do: nil
end

我们打算支持内网有多个 jenkins 的情况,同时,为了一点点安全考虑,每个 jinkins 我们分配一个随机字符串 (如 uuid) 作为 token, 至少一定程度上避免一些无效调用。用数据库来保存这个配置信息。

不需要 HTML,因此直接用 json 创建整个脚手架 (controller, model ...)

$ mix phx.gen.json Jenkins Callback callbacks host_id:string token:string callback_url:string

* creating lib/jenkins_hook_proxy/web/controllers/callback_controller.ex
* creating lib/jenkins_hook_proxy/web/views/callback_view.ex
* creating test/web/controllers/callback_controller_test.exs
* creating lib/jenkins_hook_proxy/web/views/changeset_view.ex
* creating lib/jenkins_hook_proxy/web/controllers/fallback_controller.ex
* creating lib/jenkins_hook_proxy/jenkins/callback.ex
* creating priv/repo/migrations/20170330015124_create_jenkins_callback.exs
* creating lib/jenkins_hook_proxy/jenkins/jenkins.ex
* creating test/jenkins_test.exs

Add the resource to your api scope in lib/jenkins_hook_proxy/web/router.ex:

    resources "/callbacks", CallbackController, except: [:new, :edit]


Remember to update your repository by running migrations:

    $ mix ecto.migrate


创建数据库并执行迁移脚本创建 callbacks 表

$ mix ecto.create
$ mix ecto.migrate

编辑 lib/jenkins_hook_proxy/jenkins/jenkins.ex 在 get_callback 方法后面,添加 get_callback_by_host_id

$ vi  lib/jenkins_hook_proxy/jenkins/jenkins.ex

  @doc """
  Gets a single callback by host_id.

  Raises `Ecto.NoResultsError` if the Callback does not exists.

  """
  def get_callback_by_host_id!(host_id), do: Repo.get_by!(Callback, host_id: host_id)

编辑 callback_controller, 把现有的方法 index create show update delete 统统删掉,然后重写 create 方法,如下:

$ vi lib/jenkins_hook_proxy/web/controllers/callback_controller.ex

defmodule JenkinsHookProxy.Web.CallbackController do
  use JenkinsHookProxy.Web, :controller

  alias JenkinsHookProxy.Jenkins
  alias JenkinsHookProxy.Jenkins.Callback

  action_fallback JenkinsHookProxy.Web.FallbackController

  def create(conn, %{"host_id" => host_id, "token" => token} = params) do
    with %Callback{} = callback <- Jenkins.get_callback_by_host_id!(host_id) do
      case callback.token do
        ^token ->
          # token与数据库的token匹配
          payload = %{
            # 提取github请求头(x- 开头 和 user-agent), 用于内网转发
            github_headers: conn.req_headers
                            |> Enum.filter(fn({k, v}) -> to_string(k) |> String.starts_with?(["x-", "user-agent"]) end)
                            |> Enum.into(%{}),
            jenkins: %{
              # 内网Jenkins Hook地址
              "callback_url": callback.callback_url
            },
            # 原生请求头, 无实际用途,方便调试查看用
            raw_headers: conn.req_headers |> Enum.into(%{}),
            # github请求参数,剔除 host_id 与 token
            body: params |> Map.drop(["token", "host_id"])
          }
          # websocket广播,向通道 jenkins:lobby 回传 jenkins_msg 事件
          JenkinsHookProxy.Web.Endpoint.broadcast! "jenkins:lobby", "jenkins_msg", payload

          # 返回正常给github
          conn
          |> json(%{result: "ok"})

       _ ->
        # token无效,返回403给github
        conn
        |> put_status(:forbidden)
        |> json(%{error: "token not matched."})
      end
    end
  end
end

这些代码中,真正发挥作用的是

# websocket广播,向通道 jenkins:lobby 回传 jenkins_msg 事件
JenkinsHookProxy.Web.Endpoint.broadcast! "jenkins:lobby", "jenkins_msg", payload 

由于 Jenkins.get_callback_by_host_id!(host_id) 当数据没有相应 host_id 记录时,会抛出异常 Ecto.NoResultsError。

action_fallback 机制是 Phoenix 1.3 的一个非常不错的特性,controller 中我们可以只写正常业务逻辑代码,错误处理移交给 fallback_controller 去集中处理,这一点对 API 类应用尤其实用。

修改 fallback_controller.ex,加入 对 Ecto.NoResultsError 返回 404

$ vi lib/jenkins_hook_proxy/web/controllers/fallback_controller.ex


  def call(conn, {:error, %Ecto.NoResultsError{} = _error}) do
    conn
    |> put_status(:not_found)
    |> json(%{error: "Jenkins not found"})
  end

编辑路由,把 router.ex 中最后的 scope "api" 段打开并加入 callback

$ vi lib/jenkins_hook_proxy/web/router.ex

  # Other scopes may use custom stacks.
  scope "/api", JenkinsHookProxy.Web do
    pipe_through :api
    post "/callbacks/:host_id/:token", CallbackController, :create
  end

可以运行 mix phx.routes 检查一下路由

$ mix phx.routes


    page_path  GET   /                               JenkinsHookProxy.Web.PageController :index
callback_path  POST  /api/callbacks/:host_id/:token  JenkinsHookProxy.Web.CallbackController :create

jenkins_hook_proxy 的功能已经全部完成。

我们手动编辑添加的代码,去掉空行、注释行后,代码也就 50 行左右,如果按代码语句 (为了阅读看起来舒服,一条代码语句会分成多行来写),都不足 20 条。

像这种,就是一条代码语句,却占了 4 行

# token无效,返回403给github
conn
|> put_status(:forbidden)
|> json(%{error: "token not matched."})

运行 Jenkins_hook_proxy 测试

用命令行交互方式启动,添加一条配置数据 host_id: test token: abc123 callback_url: http://192.168.2.231:8020/github-webhook

JenkinsHookProxy.Jenkins.create_callback(%{host_id: "test", token: "abc123", callback_url: "http://192.168.2.231:8020/github-webhook"})

$ iex -S mix phx.server
Erlang/OTP 19 [erts-8.3] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

[info] Running JenkinsHookProxy.Web.Endpoint with Cowboy using http://0.0.0.0:4000
Interactive Elixir (1.4.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> 16:37:04 - info: compiled 6 files into 2 files, copied 3 in 811 ms

nil
iex(2)> JenkinsHookProxy.Jenkins.create_callback(%{host_id: "test", token: "abc123", callback_url: "http://192.168.2.231:8020/github-webhook"})
[debug] QUERY OK db=5.2ms
INSERT INTO `jenkins_callbacks` (`callback_url`,`host_id`,`token`,`inserted_at`,`updated_at`) VALUES (?,?,?,?,?) ["http://192.168.2.231:8020/github-webhook", "test", "abc123", {{2017, 3, 30}, {8, 46, 9, 810223}}, {{2017, 3, 30}, {8, 46, 9, 816571}}]
{:ok,
 %JenkinsHookProxy.Jenkins.Callback{__meta__: #Ecto.Schema.Metadata<:loaded, "jenkins_callbacks">,
  callback_url: "http://192.168.2.231:8020/github-webhook", host_id: "test",
  id: 3, inserted_at: ~N[2017-03-30 08:46:09.810223], token: "abc123",
  updated_at: ~N[2017-03-30 08:46:09.816571]}}
iex(3)> 


创建运行在内网的项目 Forwarder (nodejs)

启动另外一个命令行窗口。

创建项目目录

$ mkdir forwarder
$ cd forwarder

创建 package.json

$ vi package

{
  "name": "forwarder",
  "version": "0.0.1",
  "private": false
}

添加依赖项

phoneix 的 js 是用 ES6 风格写的,因此需要引入 babel 转换。

$ npm install --save babel-cli 
$ npm install --save babel-preset-env 
$ npm install --save babel-preset-es2015
$ npm install --save babel-preset-es2016
$ npm install --save ws
$ npm install --save node-fetch

jenkins_hook_proxy 中复制需要的 js

$ cp <jenkins_hook_proxy>/assets/js/socket.js .
$ cp < jenkins_hook_proxy>/deps/phoenix/assets/js/phoenix.js .

修改 phoenix.js

phoenix.js 本身是运行在浏览器中的,nodejs 并没有 window 对象,也没有 window.WebSocket。 因此我们需要修改 phoenix.js 注入 window 及 window.WebSocket。

把下面这句话插入到 192 行 class Push { 之前。

let window = {WebSocket: require('ws')};

class Push {

编辑 socket.js

在 socket.js 中实现 channel 订阅与对服务器传回的 Jenkins_msg 事件进行处理。

$ vi socket.js

// NOTE: The contents of this file will only be executed if
// you uncomment its entry in "web/static/js/app.js".

// To use Phoenix channels, the first step is to import Socket
// and connect at the socket path in "lib/my_app/endpoint.ex":
import {Socket} from "./phoenix";

import fetch from "node-fetch";

let socket = new Socket("ws://localhost:4000/socket", {params: {token: ''}})

// When you connect, you'll often need to authenticate the client.
// For example, imagine you have an authentication plug, `MyAuth`,
// which authenticates the session and assigns a `:current_user`.
// If the current user exists you can assign the user's token in
// the connection for use in the layout.
//
// In your "web/router.ex":
//
//     pipeline :browser do
//       ...
//       plug MyAuth
//       plug :put_user_token
//     end
//
//     defp put_user_token(conn, _) do
//       if current_user = conn.assigns[:current_user] do
//         token = Phoenix.Token.sign(conn, "user socket", current_user.id)
//         assign(conn, :user_token, token)
//       else
//         conn
//       end
//     end
//
// Now you need to pass this token to JavaScript. You can do so
// inside a script tag in "web/templates/layout/app.html.eex":
//
//     <script>window.userToken = "<%= assigns[:user_token] %>";</script>
//
// You will need to verify the user token in the "connect/2" function
// in "web/channels/user_socket.ex":
//
//     def connect(%{"token" => token}, socket) do
//       # max_age: 1209600 is equivalent to two weeks in seconds
//       case Phoenix.Token.verify(socket, "user socket", token, max_age: 1209600) do
//         {:ok, user_id} ->
//           {:ok, assign(socket, :user, user_id)}
//         {:error, reason} ->
//           :error
//       end
//     end
//
// Finally, pass the token on connect as below. Or remove it
// from connect if you don't care about authentication.

socket.connect()

// Now that you are connected, you can join channels with a topic:
let channel = socket.channel("jenkins:lobby", {})

channel.on("jenkins_msg", payload => {
  channel.lastPayload = payload;
  let url = payload.jenkins.callback_url;
  let headers = Object.assign({}, {"content-type": "application/json"}, payload.github_headers);

  console.log(`[${Date()}]\n ${JSON.stringify(payload, null, "  ")}`);
  fetch(url, {
    method: "POST",
    headers: headers,
    body: JSON.stringify(payload.body)
  })
    .then( (e) => console.log(e) );
});

channel.join()
  .receive("ok", resp => { console.log("Joined successfully", resp) })
  .receive("error", resp => { console.log("Unable to join", resp) })

export {socket, channel};

创建 app.js

$ vi app.js

import socket from "./socket"

添加.babelrc

$ vi .babelrc

{
  "presets": ["env", "es2015"]
}

运行 forwarder 测试

ES6 语法,需要用 babel-node.js app.js 来转译执行,并看到结果。

$ ./node_modules/babel-cli/bin/babel-node.js app.js
Joined successfully {}

看到上面的 Joined successfully {} ,说明已经成功连接上 Jenkins_hook_proxy。

在 jenkins_hook_proxy 的运行控制台中,也能看到相应的输出:

iex(3)> [info] JOIN "jenkins:lobby" to JenkinsHookProxy.Web.JenkinsChannel
  Transport:  Phoenix.Transports.WebSocket
  Parameters: %{}
[info] Replied jenkins:lobby :ok

再开一个命令行窗口,模拟 github 发起请求

$ curl -H "Content-Type: application/json" -d '{"foo": "bar"}' http://localhost:4000/api/callbacks/test/abc123
{"result":"ok"}                                                                                                         

jenkins_hook_proxy 运行输出可以看到日志信息:

iex(12)> [info] POST /api/callbacks/test/abc123
[debug] Processing with JenkinsHookProxy.Web.CallbackController.create/2
  Parameters: %{"foo" => "bar", "host_id" => "test", "token" => "abc123"}
  Pipelines: [:api]
[debug] QUERY OK source="jenkins_callbacks" db=0.5ms
SELECT j0.`id`, j0.`callback_url`, j0.`host_id`, j0.`token`, j0.`inserted_at`, j0.`updated_at` FROM `jenkins_callbacks` AS j0 WHERE (j0.`host_id` = ?) ["test"]
[info] Sent 200 in 998µs

forwarder 的运行输出也能看到日志信息:

[Thu Mar 30 2017 17:10:25 GMT+0800 (CST)]
 {
  "raw_headers": {
    "user-agent": "curl/7.51.0",
    "host": "localhost:4000",
    "content-type": "application/json",
    "content-length": "14",
    "accept": "*/*"
  },
  "jenkins": {
    "callback_url": "http://192.168.2.231:8020/github-webhook"
  },
  "github_headers": {
    "user-agent": "curl/7.51.0"
  },
  "body": {
    "foo": "bar"
  }
}
(node:47869) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 2): FetchError: request to http://192.168.2.231:8020/github-webhook failed, reason: connect ECONNREFUSED 192.168.2.231:8020

最后这个错误,是因为 http://192.168.2.231:8020/github-webhook jenkins 服务器没打开。 但已经能说明,整条链路是正常的了。

测试无效 token

$ curl -H "Content-Type: application/json" -d '{"foo": "bar"}' http://localhost:4000/api/callbacks/test/abc   
{"error":"token not matched."}                                                                                      

总结

真实环境部署 jenkins_hook_proxy 运行在外网云主机上;

forwarder 运行在内网主机上,可以直接在 jenkins 的机器上跑。

反正我们已经是使用这么使用了。

代码可以上 github 获取 https://github.com/edwardzhou/jenkins_hook_proxy https://github.com/edwardzhou/jenkins-forwarder

#注意 上面的代码是十几分钟撸出来的玩具,帮助我们偿还持续集成的技术债务而已。 对于其他问题和副作用概不承担责任 😜😜😜

如果能用 ssh 的话,直接 ssh -R 就可以把内网端口 forward 出去。 我正在拿 elixir 写一个 websocket tunnel 替换 ssh tunnel 用,以应对恶劣的网络环境,等测试完狗粮可以扔出来晒晒

Reply to hemslo

期待 :)

由于Phoenix的channel为提供了非常方便且健壮的web前端浏览器js脚本,为了能直接利用它的js脚本,forwarder 就用nodejs编写。 Phoenix Channel 也有 Elixir 的客户端

10 分钟好高效啊

为啥不撸一个 jenkins ... 感觉更有用一点

Reply to fuyang

需要自己动手的代码也就那么几十简单代码行(十来条语句),谈不上高效。

Reply to luikore

我的工作重心与兴趣是 scrum 敏捷过程,TDD CI CD。

为什么不用ngrok,为什么不用 frp

如果按照减少技术负债的想法,首先最浪费钱的应该是程序员的时间。

Reply to gyorou

ngrok 花生壳等 优先都试过,不稳定。也看了 frp,没选 frp(go)的原因,是我们在探索 Elixir + Phoenix, 希望能把我们在 Rails 的成功方案转移到 Elixir Phoenix 上面,自己撸一个,一举多得。

You need to Sign in before reply, if you don't have an account, please Sign up first.