Ruby China
  • Topics
  • 招聘
  • Wiki
  • 酷站
  • Gems
  • Sign Up
  • Sign In
Hooopo
@hooopo
Admin
NO. 8 / 2011-10-28

[email protected]
nil
北京
160 Topics / 3013 Replies
360 Followers
0 Following
74 Favorites
聪明的妖怪录下了唐僧的紧箍咒
Reward
GitHub Public Repos
  • oh-my-github-circles 47

    GitHub User Circle Generator Using GitHub Actions

  • hackernews-insight 21

    Hackernews Insight using TiDB Cloud

  • repo-track-pipeline 6

    🔄 A flexible open-source data pipeline for seamlessly syncing data from any repository to your da...

  • oh-my-github-pipeline 6

    🔄 A flexible open-source data pipeline for seamlessly syncing data from any github user to your d...

  • chatgpt-xiaoai 3

    小爱音箱集成LLM,SaaS 服务

  • repo-contributor-circles 1

    GitHub repo contributor circles generator.

  • ossinsight-x 1

    Automatically post trending repos to Twitter every day.

  • mi-service 1

    XiaoMi Cloud Service for mi.com

  • hooopo 0

  • streamlit-echarts-demo 0

    Demo for Streamlit ECharts component

More on GitHub
  • Overview
  • Topics
  • Replies
  • Favorites
  • Following
  • Followers
  • Buffer Queue for Ruby at March 25, 2021

    👍

  • Rails 因為 mimemagic 炸了 at March 25, 2021

    昨天镜像 build 炸了 就是因为这个

  • Hypercable Analytics benchmark at March 25, 2021

    我宁愿迁移 crystal

  • Crystal 1.0.0 Released! at March 24, 2021

    6

  • Buffer Queue for Ruby at March 23, 2021

  • Buffer Queue for Ruby at March 23, 2021

    是的,没有实现 worker,但我现在的用法是和 sidekiq 一起,利用 sidekiq 的 worker 来实现,可以看上面新提交的代码

    if Sidekiq.server?
      Sidekiq.on(:startup) do
        BQ = BufferQueue.new(max_batch_size: 10, execution_interval: 100) do |batch|
          puts "bulk insert #{batch.size} records"
          Hyper::Event.import(
            EventJob::COLUMN_NAMES,
            batch.flatten.map { |attr| Hyper::Event.new(attr) },
            validate: false,
            timestamps: false
          ) unless batch.empty?
        end
      end
    end
    

    in sidekiq job:

    class EventJob
      include Sidekiq::Worker
    
      COLUMN_NAMES = Hyper::Event.column_names
    
      def perform(*args)
          result = a_lot_of_cal
          BQ.push result
        end
      end
    end
    

    效果就是先利用 sidekiq 的多线程并行处理一些计算任务,产生的结果批量插入数据库; 相比之前 sidekiq job 里单条插入会快很多。并且可以利用 activerecord import 把多条记录改成 insert into values 的形式,插入更快。

  • Buffer Queue for Ruby at March 23, 2021

    ruby 的内置 queue 不太好用:https://bugs.ruby-lang.org/issues/9145

  • Buffer Queue for Ruby at March 23, 2021

    效果

    [1] pry(main)> bq = BufferQueue.new(max_batch_size: 10, execution_interval: 5) do |batch|
      p batch
    end
    25.times do |i|
      bq.push(i)
    end
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
    => 25
    [2] pry(main)> [20, 21, 22, 23, 24]
    
    [2] pry(main)> quit
    shutdown ...
    
  • Buffer Queue for Ruby at March 23, 2021

    搞了一个简单版 https://github.com/HyperCable/hypercable/pull/49/files

    # frozen_string_literal: true
    
    class BufferQueue
      attr_reader :max_batch_size, :execution_interval, :timeout_interval, :callback
      def initialize(max_batch_size: 100, execution_interval: 60, timeout_interval: 60, &callback)
        @max_batch_size = max_batch_size
        @execution_interval = execution_interval
        @timeout_interval = timeout_interval
        @queue = Queue.new
        @timer = Concurrent::TimerTask.new(execution_interval: execution_interval, timeout_interval: timeout_interval) do
          flush
        end
        @timer.execute
        @callback = callback
        at_exit { shutdown }
      end
    
      def flush
        batch = []
        max_batch_size.times do
          if not @queue.empty?
            begin
              batch << @queue.pop(true)
            rescue ThreadError
              puts "queue is empty"
              break
            end
          else
            break
          end
        end
        callback.call(batch) unless batch.empty?
      end
    
      def push(item)
        @queue << item
        if @queue.size >= max_batch_size
          flush
        end
        item
      end
    
      def shutdown
        puts "shutdown ..."
        @timer.shutdown
        flush
      end
    end
    
    
  • 东京创业团队寻找高级工程师 (ror) or 技术合伙人 (CTO) at March 22, 2021

    这种帖子底下需要一个 warning 功能

  • Buffer Queue for Ruby at March 22, 2021

    看起来不错

  • Buffer Queue for Ruby at March 22, 2021

    那个是用的 client middleware,要用的话,我要改成 server middleware。不一定基于 sidekiq,感觉 Ractor 可以做这东西。

  • Headless Analytics stack? at March 21, 2021

    😝

  • Headless Analytics stack? at March 20, 2021

    相似度很高

  • 配合 Rails 和 PostgreSQL 实现站点对象存储服务的更换 at March 15, 2021

    可以把爬虫也替换了 https://github.com/pramsey/pgsql-http :😂

  • Hypercable Analytics open sourced at March 08, 2021

    居然还有人在

  • Hypercable Analytics open sourced at March 08, 2021

    还没 production ready 就被官方发现了

  • Google Analytics 存在的问题 at March 03, 2021

    sdk 都是可以得到源码的,但经过混淆了。

    也不需要开源,目前 hypercable analytics 的使用方式:

    <script async src="https://www.googletagmanager.com/gtag/js?id=G-PJBVFLBFSB"></script>
    <script>
      window.dataLayer = window.dataLayer || [];
      function gtag(){dataLayer.push(arguments);}
      gtag('js', new Date());
      gtag('config', 'G-PJBVFLBFSB', {transport_url: 'https://learnsql.io/c7f4edce-58c3-4917-8f18-a2ea6c1b93dc'});
    </script>
    
  • Google Analytics 存在的问题 at March 03, 2021

    新建的 ga 都是 google Analytic v4 的;

    这里有介绍:https://support.google.com/analytics/answer/10089681

  • Hypercable Analytics open sourced at March 02, 2021

    是 数据自有的需求很大 有了这些行为数据之后,做 crm 广告系统 推荐系统 也就很容易了

  • Hypercable Analytics open sourced at March 02, 2021

    还有一个计划 就是 saas 版也可以直接暴露一个只读的数据库链接,直接使用或拉取原始数据到自己服务器。私有部署版当然就更灵活了

  • Hypercable Analytics open sourced at March 02, 2021

    会的 该有的都可以有 只是目前需要做的太多了

  • Hypercable Analytics open sourced at March 01, 2021

    目前好多 broken link,下个版本一起修。。

  • Hypercable Analytics open sourced at March 01, 2021

    [email protected] 111111

  • Hypercable Analytics open sourced at March 01, 2021

    还有很多更酷的特性等待发布

  • PostgreSQL 大小写敏感问题 at February 01, 2021

    http://www.postgres.cn/docs/9.4/citext.html

  • 我的屋子里有一只蝙蝠,不知道它冬天吃什么,会不会被饿死。 at February 01, 2021

    。。。

  • 如何实现一个信息架构友好的标签系统 at January 31, 2021

    admin 里

  • 如何实现一个信息架构友好的标签系统 at January 30, 2021

    👌

  • Prev
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • …
  • 108
  • 109
  • Next
关于 / RubyConf / Ruby 镜像 / RubyGems 镜像 / 活跃会员 / 组织 / API / 贡献者
由众多爱好者共同维护的 Ruby 中文社区,本站使用 Homeland 构建,并采用 Docker 部署。
服务器由 赞助 CDN 由 赞助
iOS 客户端 / Android 客户端 简体中文 / English