Ruby China

Sinatra 想知道 Ruby 对 Redis 中文数据的编码处理

yiluo · 2016年08月20日 · 最后由 yiluo 回复于 2016年09月10日 · 7779 次阅读

小白在写一个小博客遇上了点麻烦，我将博客的错误部分用下面的代码复现了一下。

require 'sinatra'
require 'redis'

$redis = Redis.new
include ERB::Util

get '/' do
 erb :index,:locals => {hash: $redis.get('content:1')}
end

Redis 那边是这样的

user@domain:~# redis-cli --raw
127.0.0.1:6379> get content:1
哈哈
127.0.0.1:6379>

views 下 erb 文件长这样

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <meta http-equiv="X-UA-Compatible" content="ie=edge">
  <title>Document</title>
</head>
<body>
  <%= hash %>
  </body>
  </html>

运行 ruby，浏览器打开 '/' 提示错误

2016-08-20 09:02:07 - Encoding::CompatibilityError - incompatible character encodings: UTF-8 and US-ASCII:
    /root/test/views/index.erb:12:in `block in singleton class'
    /root/test/views/index.erb:-6:in `instance_eval'
    /root/test/views/index.erb:-6:in `singleton class'
    /root/test/views/index.erb:-8:in `__tilt_11968560'
    /usr/local/rvm/gems/ruby-2.3.1/gems/tilt-2.0.5/lib/tilt/template.rb:167:in `call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/tilt-2.0.5/lib/tilt/template.rb:167:in `evaluate'
    /usr/local/rvm/gems/ruby-2.3.1/gems/tilt-2.0.5/lib/tilt/template.rb:102:in `render'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:823:in `render'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:667:in `erb'
    app.rb:13:in `block in <main>'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in `call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in `block in compile!'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `block (3 levels) in route!'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:994:in `route_eval'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `block (2 levels) in route!'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1015:in `block in process_route'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:in `catch'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:in `process_route'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:973:in `block in route!'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:in `each'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:in `route!'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1085:in `block in dispatch!'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `block in invoke'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `catch'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `invoke'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1082:in `dispatch!'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in `block in call!'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `block in invoke'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `catch'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `invoke'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in `call!'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:895:in `call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/logger.rb:15:in `call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/commonlogger.rb:33:in `call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:219:in `call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:212:in `call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/methodoverride.rb:22:in `call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/show_exceptions.rb:25:in `call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:182:in `call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:2013:in `call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in `block in call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1787:in `synchronize'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in `call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/thin-1.7.0/lib/thin/connection.rb:86:in `block in pre_process'
    /usr/local/rvm/gems/ruby-2.3.1/gems/thin-1.7.0/lib/thin/connection.rb:84:in `catch'
    /usr/local/rvm/gems/ruby-2.3.1/gems/thin-1.7.0/lib/thin/connection.rb:84:in `pre_process'
    /usr/local/rvm/gems/ruby-2.3.1/gems/thin-1.7.0/lib/thin/connection.rb:50:in `block in process'
    /usr/local/rvm/gems/ruby-2.3.1/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:1076:in `block in spawn_threadpool'
Unexpected error while processing request: invalid byte sequence in US-ASCII
    /usr/local/rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/utils.rb:249:in `gsub'
    /usr/local/rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/utils.rb:249:in `escape_html'
    /usr/local/rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/showexceptions.rb:103:in `h'
    (erb):169:in `block (2 levels) in pretty'
    (erb):168:in `each'
    (erb):168:in `block in pretty'
    (erb):155:in `each'
    (erb):155:in `pretty'
    /usr/local/rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/erb.rb:864:in `eval'
    /usr/local/rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/erb.rb:864:in `result'
    /usr/local/rvm/gems/ruby-2.3.1/gems/rack-1.6.4/lib/rack/showexceptions.rb:97:in `pretty'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/show_exceptions.rb:34:in `rescue in call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/show_exceptions.rb:25:in `call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:182:in `call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:2013:in `call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in `block in call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1787:in `synchronize'
    /usr/local/rvm/gems/ruby-2.3.1/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in `call'
    /usr/local/rvm/gems/ruby-2.3.1/gems/thin-1.7.0/lib/thin/connection.rb:86:in `block in pre_process'
    /usr/local/rvm/gems/ruby-2.3.1/gems/thin-1.7.0/lib/thin/connection.rb:84:in `catch'
    /usr/local/rvm/gems/ruby-2.3.1/gems/thin-1.7.0/lib/thin/connection.rb:84:in `pre_process'
    /usr/local/rvm/gems/ruby-2.3.1/gems/thin-1.7.0/lib/thin/connection.rb:50:in `block in process'
    /usr/local/rvm/gems/ruby-2.3.1/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:1076:in `block in spawn_threadpool'

**6.*29.*7.*** - - [20/Aug/2016:09:02:08 -0400] "GET /favicon.ico HTTP/1.1" 404 471 0.0114

检查发现 hash 的编码格式是 US-ASCII。目前的解决方法是在 erb 文件中用.force_encoding('utf-8') 的方法将变量 hash 改回 utf-8，显示正常。但我还有疑点没有解决。

1.是什么导致了这个问题？ 2.为什么同样的程序在 Debian GNU/Linux 7.11 下跑出现问题，在我本地 OS X10.11.5 上跑一点事都没有。 3.翻过论坛帖子，知道 Ruby 2.0 默认 Ruby 源文件的编码为 UTF-8，为什么它还会将中文解成 us-ascii 4.方法.force_encoding('utf-8') 真的是唯一解了吗？这只是一个复现小程序，放到更大的项目上每个变量都要转一下？

———————————————————————————————————————————————————————— 尝试过的解决方案 1.在程序代码第一行加上#encoding:UTF-8 2.虽然我知道 sinatra 的 default_encoding 是 utf-8，但是我还是写了一段

configure do
    set :default_encoding,'utf-8'
end

3.将 redis 升级到 3.2.3, Ruby 升级到 ruby-2.3.1

谢谢指点

yanhao #1 2016年08月22日

在 Linux 上用locale命令检查一下系统编码：

 > locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE=en_US.UTF-8
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

很可能系统编码不是 UTF-8，设置一下应该就好了。

1 个赞
lululau #2 2016年08月23日

Socket 对象的外部编码默认是 ASCII-8BIT，你可以查下 Redis 类有没有可以指定外部编码的选项

在 OS X 上没问题这点我也想不通了，难道你 OS X 上用的是 Ruby 1.9.x？

yiluo #3 2016年09月10日

十分感谢！确实是系统编码的问题！

*@*:~/test# locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE=UTF-8
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

LC_CTYPE 的值更改为 en_US.UTF-8 之后我的程序就能正常运行了！

yiluo #4 2016年09月10日

#2 楼 @lululau 本地 OS X 用的是 2.x.x 的版本。这应该和 ruby 的版本没有关系

需要 登录 后方可回复, 如果你还没有账号请 注册新账号