ynopeeb

Ruby 对 csv 文件进行统计

ynopeeb — Mon, 30 Jul 2018 13:21:11 +0800

需求：有一个 test.csv 文件，格式如下：

关键字   问题ID    概要  优先级
ONLINE-30763t43 2139480 查询  重要
ONLINE-38784480 2149335 失败  重要
ONLINE-33765689 2141691 消费者 普通
ONLINE-38127341 2154990 无法  重要
ONLINE-31797797 2154503 订单  普通
ONLINE-37566180 2139727 选择  紧急
ONLINE-39804474 2156819 查询  重要

需求统计优先级中重要，紧急，普通的数量

查看了文档 : https://ruby-doc.org/stdlib-2.5.1/libdoc/csv/rdoc/CSV.html#method-i-converters ，尝试如下：

CSV.foreach("test.csv", headers: true) do | row |
## how to count
end

这种每个帖子都发广告用户，是不是要踢了

ynopeeb — Thu, 16 Mar 2017 10:15:51 +0800

这段蹩脚的代码该如何优化？

ynopeeb — Tue, 21 Feb 2017 23:10:59 +0800

要使用 Ruby 处理一份日志，提取出来里面的 duration 字段的值，并且相加，得到总的数值。日志需要先处理，找出里面的 kafka 字段，取出其中的 json 串，再做处理。作为新手，跌跌撞撞的使用 Ruby 完成了一部分功能，越看越别扭，希望能指出其中的问题，是否还有其他更简洁优雅的实现方式，多谢。

日志文件为 test.log，内容已经做了处理，参考如下：

/disk/sata13/xxx-platform/2017_01_03/0.0.0.0/platform.07_30.log.0.0.0.0.2017_01_03.gz: /naga/v0.5.3-small: {"log":"[Naga 0.5.3 I 17-01-03 07:19:43] ddc7c32422a7a2e413346258034c39e6 - kafka:{\"media_uris\": [\"\/uploads\/audio\/120\/14833991837592.amr\"], \"status\": 200, \"tasks\": [\"transcode\"], \"request_time\": 1483399183, \"task_id\": \"ddc7c32422a7a2e413346258034cxxx39e6\", \"timestamp\": 1483399183, \"bucket\": \"xxx\", \"convtime\": 0.18000006675720215, \"notify_url\": \"http:\/\/www.xxx.net\/api\/upload\/xxx\", \"input_info\": {\"streams\": [{\"index\": 0, \"audio_channels\": 1, \"audio_samplerate\": 8000.0, \"bitrate\": 8800, \"codec_desc\": \"AMR-NB (Adaptive Multi-Rate NarrowBand)\", \"codec\": \"amrnb\", \"duration\": 0.029125000000000002, \"type\": \"audio\"}], \"format\": {\"duration\": 0.029125000000000002, \"fullname\": \"3GPP AMR\", \"bitrate\": 10437.0, \"filesize\": 38.0, \"format\": \"amr\"}}, \"uploads\": [\"\/uploads\/audio\/120\/14833991837592.mp3\"], \"starttime\": 1483399183, \"async\": true, \"media_info\": {\"streams\": [{\"index\": 0, \"audio_channels\": 1, \"audio_samplerate\": 8000.0, \"bitrate\": 64000, \"codec_desc\": \"MP3 (MPEG audio layer 3)\", \"codec\": \"mp3\", \"duration\": 0.216, \"type\": \"audio\"}], \"format\": {\"duration\": 0.216, \"fullname\": \"MP2\/3 (MPEG audio layer 2\/3)\", \"bitrate\": 87000.0, \"filesize\": 2349.0, \"format\": \"mp3\"}}, \"type\": \"audio\", \"options\": {\"ab\": \"64\", \"f\": \"mp3\"}, \"errmsg\": \"\"}","time":"2017\/01\/02 23:19:43 UTC","hostname":"bd39a20ecef9@xxx"}
/disk/sata13/xxxx-platform/2017_01_03/0.0.0.0/platform.07_30.log.0.0.0.0.2017_01_03.gz: /naga/v0.5.3-small: {"log":"[Naga 0.5.3 I 17-01-03 07:19:43] ddc7c32422a7a2e413346258034c39e6 - notify:message:{'bucket': 'xxx', 'notify_url': 'http:\/\/www.xxx.net\/api\/upload\/xxx', 'content_type': 'application\/x-www-form-urlencoded', 'operator': 'xxx', 'password': 'xxx', 'data': 'xxx'}","time":"2017\/01\/02 23:19:43 UTC","hostname":"bd39a20ecef9@xxx"}
/disk/sata13/xxx-platform/2017_01_03/0.0.0.0/platform.07_30.log.0.0.0.0.2017_01_03.gz: /naga/v0.5.3-small: {"log":"[Naga 0.5.3 I 17-01-03 07:19:43] ddc7c32422a7a2e413346258034c39e6 - kafka:{\"media_uris\": [\"\/uploads\/audio\/120\/14833991837592.amr\"], \"status\": 200, \"tasks\": [\"transcode\"], \"request_time\": 1483399183, \"task_id\": \"ddc7c32422a7a2e413346258034cxxx39e6\", \"timestamp\": 1483399183, \"bucket\": \"xxx\", \"convtime\": 0.18000006675720215, \"notify_url\": \"http:\/\/www.xxx.net\/api\/upload\/xxx\", \"input_info\": {\"streams\": [{\"index\": 0, \"audio_channels\": 1, \"audio_samplerate\": 8000.0, \"bitrate\": 8800, \"codec_desc\": \"AMR-NB (Adaptive Multi-Rate NarrowBand)\", \"codec\": \"amrnb\", \"duration\": 0.029125000000000002, \"type\": \"audio\"}], \"format\": {\"duration\": 0.029125000000000002, \"fullname\": \"3GPP AMR\", \"bitrate\": 10437.0, \"filesize\": 38.0, \"format\": \"amr\"}}, \"uploads\": [\"\/uploads\/audio\/120\/14833991837592.mp3\"], \"starttime\": 1483399183, \"async\": true, \"media_info\": {\"streams\": [{\"index\": 0, \"audio_channels\": 1, \"audio_samplerate\": 8000.0, \"bitrate\": 64000, \"codec_desc\": \"MP3 (MPEG audio layer 3)\", \"codec\": \"mp3\", \"duration\": 0.216, \"type\": \"audio\"}], \"format\": {\"duration\": 0.216, \"fullname\": \"MP2\/3 (MPEG audio layer 2\/3)\", \"bitrate\": 87000.0, \"filesize\": 2349.0, \"format\": \"mp3\"}}, \"type\": \"audio\", \"options\": {\"ab\": \"64\", \"f\": \"mp3\"}, \"errmsg\": \"\"}","time":"2017\/01\/02 23:19:43 UTC","hostname":"bd39a20ecef9@xxx"}
/disk/sata13/docker-platform/2017_01_03/0.0.0.0/platform.07_30.log.0.0.0.0.2017_01_03.gz: /naga/v0.5.3-small: {"log":"[Naga 0.5.3 I 17-01-03 07:19:43] ddc7c32422a7a2e413346258034c39e6 - notify:message:{'bucket': 'xxx', 'notify_url': 'http:\/\/www.xxx.net\/api\/upload\/xxx', 'content_type': 'application\/x-www-form-urlencoded', 'operator': 'xxx', 'password': 'xxx', 'data': 'xxx'}","time":"2017\/01\/02 23:19:43 UTC","hostname":"bd39a20ecef9@xxx"}
/disk/sata13/docker-platform/2017_01_03/0.0.0.0/platform.07_30.log.0.0.0.0.2017_01_03.gz: /naga/v0.5.3-small: {"log":"[Naga 0.5.3 I 17-01-03 07:19:43] ddc7c32422a7a2e413346258034c39e6 - notify:message:{'bucket': 'xxx', 'notify_url': 'http:\/\/www.xxx.net\/api\/upload\/xxx', 'content_type': 'application\/x-www-form-urlencoded', 'operator': 'xxx', 'password': 'xxx', 'data': 'xxx'}","time":"2017\/01\/02 23:19:43 UTC","hostname":"bd39a20ecef9@xxx"}
/disk/sata13/xxx-platform/2017_01_03/0.0.0.0/platform.07_30.log.0.0.0.0.2017_01_03.gz: /naga/v0.5.3-small: {"log":"[Naga 0.5.3 I 17-01-03 07:19:43] ddc7c32422a7a2e413346258034c39e6 - kafka:{\"media_uris\": [\"\/uploads\/audio\/120\/14833991837592.amr\"], \"status\": 200, \"tasks\": [\"transcode\"], \"request_time\": 1483399183, \"task_id\": \"ddc7c32422a7a2e413346258034cxxx39e6\", \"timestamp\": 1483399183, \"bucket\": \"xxx\", \"convtime\": 0.18000006675720215, \"notify_url\": \"http:\/\/www.xxx.net\/api\/upload\/xxx\", \"input_info\": {\"streams\": [{\"index\": 0, \"audio_channels\": 1, \"audio_samplerate\": 8000.0, \"bitrate\": 8800, \"codec_desc\": \"AMR-NB (Adaptive Multi-Rate NarrowBand)\", \"codec\": \"amrnb\", \"duration\": 0.029125000000000002, \"type\": \"audio\"}], \"format\": {\"duration\": 0.029125000000000002, \"fullname\": \"3GPP AMR\", \"bitrate\": 10437.0, \"filesize\": 38.0, \"format\": \"amr\"}}, \"uploads\": [\"\/uploads\/audio\/120\/14833991837592.mp3\"], \"starttime\": 1483399183, \"async\": true, \"media_info\": {\"streams\": [{\"index\": 0, \"audio_channels\": 1, \"audio_samplerate\": 8000.0, \"bitrate\": 64000, \"codec_desc\": \"MP3 (MPEG audio layer 3)\", \"codec\": \"mp3\", \"duration\": 0.216, \"type\": \"audio\"}], \"format\": {\"duration\": 0.216, \"fullname\": \"MP2\/3 (MPEG audio layer 2\/3)\", \"bitrate\": 87000.0, \"filesize\": 2349.0, \"format\": \"mp3\"}}, \"type\": \"audio\", \"options\": {\"ab\": \"64\", \"f\": \"mp3\"}, \"errmsg\": \"\"}","time":"2017\/01\/02 23:19:43 UTC","hostname":"bd39a20ecef9@xxx"}
/disk/sata13/docker-platform/2017_01_03/0.0.0.0/platform.07_30.log.0.0.0.0.2017_01_03.gz: /naga/v0.5.3-small: {"log":"[Naga 0.5.3 I 17-01-03 07:19:43] ddc7c32422a7a2e413346258034c39e6 - notify:message:{'bucket': 'xxx', 'notify_url': 'http:\/\/www.xxx.net\/api\/upload\/xxx', 'content_type': 'application\/x-www-form-urlencoded', 'operator': 'xxx', 'password': 'xxx', 'data': 'xxx'}","time":"2017\/01\/02 23:19:43 UTC","hostname":"bd39a20ecef9@xxx"}

代码如下：

#! /usr/bin/ruby
# something
require 'json'

File.open('test.log', 'r') do |f|
    f.each do |line|
        re = /(kafka:)({.+}(?=",))+/
        match_data = re.match(line)
        data = $2.gsub('\\', '') if match_data.class == MatchData
        data1 = JSON.parse(data,{ symbolize_names: true }) if data.class == String
        if data1.is_a? Hash
            puts data1[:media_info][:format][:duration].to_f 
        end
    end
end

为何调用的方法没生效？

ynopeeb — Wed, 14 Oct 2015 11:29:33 +0800

场景：将一个包含很多二级域名的文件，解析出来第一条记录。以谷歌地图为例，格式化的输出 map.google.com maps。

~ dig map.google.com

;; ANSWER SECTION:
map.google.com.     2400    IN  CNAME   maps.google.com.
maps.google.com.    299 IN  A   216.58.221.110

代码：

# /usr/bin/ruby -w
# coding:utf-8
# dig the domain banding in UPYUN
# https://github.com/bluemonk/net-dns

require 'net/dns'

# 将给定的 url dig 出来 answer 里的第一条解析记录
def dig_dns(url)
    packet = Net::DNS::Resolver.start("#{url}")
    r = packet.answer[0].to_s.split(" ")
    print "#{r[0].to_s.chop}  " "#{r[4].to_s.split(".")[0]}\n"
end

# 将泛域名里面的 * 替换成 test，以符合域名解析规则
def subt(str)
    if str.include? '*'
        puts str.sub('*','upyun')
    else
        puts str
    end
end

# 包含域名的文件作为参数输入
filename = ARGV[0]

# 遍历文件中的 url，然后格式化的输出
File.open(filename,'r+') do |f|
  f.each do |line|
    line.split(" ").each {|s| 
        dig_dns(subt(s))
    }
  end
end

疑问：在最后一个部分 dig_dns(subt(s)) 中 dig_dns 方法没有生效，结果输出的还是单个 url，请问问题出在哪里呢？

为何提示该方法未定义

ynopeeb — Fri, 09 Oct 2015 21:40:47 +0800

我想实现一个简单的小程序，读取文件中的域名，打印出来第一条解析记录 (CNAME 或者 A 记录)。例如，有一个 test.txt 文件，里面有如下域名 baidu.com google.com

希望输出 => baidu.com. 30 IN A 220.181.57.217 => google.com. 199 IN A 64.233.189.101

我的代码如下：

#!/usr/bin/ruby
# -*- coding: UTF-8 -*-

require 'net/dns'
filename = ARGV[0]
file = File.open(filename)

file.each_line do |line|
    dig_dns("#{line}")[0]
end
   file.close

def dig_dns(url)
    packet = Net::DNS::Resolver.start(url)
    packet.answer
end

执行之后，总是提示方法未定义，求解～

chongfu.rb:11:in `block in <main>': undefined method `dig_dns' for main:Object (NoMethodError)
    from chongfu.rb:10:in `each_line'
    from chongfu.rb:10:in `<main>'