@suxiaohun 分页条件,过大的 offse 提供了一种解决方式,可以试一试
测试了一下,总共 5w 条数据。效果还是很明显的
def test1
client = Mysql2::Client.new(:host => "localhost",
:username => "root",
:password => "",
:database => "happy_for_ni"
)
result = client.query('SELECT * FROM videos', :stream => true)
result.each do |row|
p row
end
end
def test2
Videos.find_each do |row|
p row
end
end
Benchmark.bm do |x|
x.report { test2 }
x.report { test1 }
end
[#<Benchmark::Tms:0x0000010cdc9788 @cstime=0.0, @cutime=0.0, @label="", @real=18.736821, @stime=0.8600000000000003, @total=11.649999999999999, @utime=10.79>,
#<Benchmark::Tms:0x000001084c0d70 @cstime=0.0, @cutime=0.0, @label="", @real=14.197525, @stime=0.79, @total=6.850000000000002, @utime=6.060000000000002>]
👍
语言仅仅是个工具而已,重要的是使用这个语言的人。
@jerrym 60W 的数据一点都不多,我们的表经常是上百万,千万的,合适的表结构,恰当的索引,都会大大减少查询的时间。
@MrPasserby 产品是开发的死对头。一般遇到这种问题,都会详细的了解需求,确定是否要该表结构。 一般遇到这种频繁修改表结构的需求,至少都会在我这边自动屏蔽。
SO,一个好的产品,很关键。
1、在设计之初,考虑数据增长趋势,建立合适的索引。 2、表关联的外键建立索引。 3、建立一些冗余数据,来避免 join 查询
和 @peter 所说的一样
比如两个表 deals
(tb_shop_id) 和 tb_shops
(wangwang, name.....) 。经常有这种需求,根据 wangwang
去查询deals
数据
一般写法
SELECT deals.*
FROM deals INNER JOIN tb_shops ON deals.tb_shop_id = tb_shops.id
WHERE tb_shops.wangwang='xxxxxx'
但是,如果在设计之初,考虑到这种需求,那么在表 deals
上记录 tb_shop_id 和 wangwang,那么就不需要 join 表查询了。
只有在 tb_shops.wangwang 修改的时候同步 deals
表
如下:
SELECT deals.*
FROM deals
WHERE deals.wangwang = 'xxxxxx'
4、数据库良好的设计,一定要,所见及所得,避免复杂的查询。 比如日志表,xxxx_logs 可能会有百万或千万的数据。那么避免对这个表做任何查询。可以根据需求建立几个小的统计表。 比如,商家每天的消费情况,商家的总消费情况,所有的数据在存入 xxxx_logs 的时候,对应把数据同步到这几个小表中。
商家每日消费情况表tb_shop_day_details
,商家总消费情况tb_shops_total_details
比如查看所有商家的总消费情况,只需要查询 tb_shops_total_details
不再需要从 xxxx_logs
临时计算数据,返回。
比如,要查看商家2014-11-11那一天消费的情况,那么也只需要
SELECT *
FROM tb_shop_day_details
WHERE st_date = '2014-11-11'
所有数据所见及所得,那么这种情况就不需要 JOIN 表操作。
5、将多表的数据查询,提前汇总到一个表。在真正需要查询的时候,从该汇总表中查询。
6、读写分离。在主库写,从库读。
7、将多表 join 查询,改成几条简单的 SQL 查询,最后在组合数据。
8、如果觉得上述的太麻烦,使用 solr 吧,但是原理是一样的,将预计要查询的汇总在一个表。
其实多表查询慢,第一、没有建立合适的索引,第二,存在不良的表结构设计。 在系统初期,没有做好设计。导致数据增长不可控,不良的表结构设计,肯定会导致多表查询慢。 要解决多表查询缓慢的问题,从修改表结构开始,一步一步优化,肯定能达到预期的效果。
@hbin 真真的 -:
使用 ID 最好。
不过在做这个 API 之前,验证请求的访问者。
不建议使用A-1
这种方式,这种情况加大了程序的复杂度。违背单一职责原则。如果 PC,无线,其他平台都需要使用你这个接口,那么是不是,这几个平台都需要重写一下解析A-1
这段代码呢?
本来很简单的东西,考虑的情况太多,反而影响自己的判断。不论任何设计,都应该以简单,松耦合,单一,可扩展为基础准则。
直接操作 redis,就能实现共享。写入 redis 的数据,消息队列就可以直接消耗了呢。
给你个例子参考下
插入队列的名字:zzzz:queue:cpc_for_deals
Redis.current.lpush("zzzz:queue:cpc_for_deals", example_record.to_json)
务必为左入队列!!!
example_record = { :retry=>5, :queue=>"cpc_for_deals", :class => "CpcLogsWorker", :args => [deal_id, added_count], :jid => "4f54290c1224e44007a02f49", :enqueued_at => 1427009697.350305 }
其中:
retry 表示 重试次数
queque 表示 入哪个队列
class 表示 sidekiq 处理的 Worker 的名字
args 表示 需要处理的数据。第一个参数为 deal_id , 第二个参数为 该 deal 增加的点击数
jid 表示 24 位唯一随机数 Ruby 使用 SecureRandom.hex(12) 实现
enqueued_at 表示 从 1970.01.01 00:00:00 到现在的毫秒数,注意是带小数点的
Ruby 中的表示方式为 Time.now.to_f
Redis.current.lpush("zzzz:queue:cpc_for_deals", example_record.to_json)
@hxh1246996371 MySQL 的关键字也需要注意,戳这里,MySQL 关键字带来的坑
七牛,虽然之前出过几次大问题,修复的还算及时。之前测试过,速度很快滴。
@dandananddada 别把馒头不当干粮。:)
数据结构,操作系统,算法,计算机网络。这些该学的一定要学好。要做到架构师级别,这些东西是必须要精通的。语言这些可以后天锻炼的,别着急。
曾经教我们计算机网络的老师,和我这么说,栓个馒头,狗都能编程。。。当时我一脸黑线
工作这么久了,发现,越深入,还得用到大学学到的东西。
@sun1752709589 正解,多列索引,第一列正序排列,第二列次之,。。。第二列之后的数据组织,并不是有序的,和 id 没有关系。如果没有索引的话,数据按照主键 id 正序排列,这个是聚簇索引。我们大多数认为的情况是这种聚簇索引。
@foghost very good! 👍
实践是检验整理的唯一方式。
SELECT SQL_NO_CACHE id, account_id, log_type FROM `pay_account_logs` WHERE `pay_account_logs`.`account_id` = 408;
+---------+------------+----------+
| id | account_id | log_type |
+---------+------------+----------+
| 59 | 408 | 1 |
| 45 | 408 | 2 |
| 43 | 408 | 4 |
| 55 | 408 | 4 |
+---------+------------+----------+
EXPLAIN SELECT SQL_NO_CACHE `pay_account_logs`.* FROM `pay_account_logs` WHERE `pay_account_logs`.`account_id` = 408\G;
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pay_account_logs
type: ref
possible_keys: idx_of_account_id_log_type
key: idx_of_account_id_log_type
key_len: 4
ref: const
rows: 4
Extra:
1 row in set (0.01 sec)
SELECT SQL_NO_CACHE id, account_id, log_type FROM `pay_account_logs` FORCE KEY(PRIMARY) WHERE `pay_account_logs`.`account_id` = 408;
+---------+------------+----------+
| id | account_id | log_type |
+---------+------------+----------+
| 43 | 408 | 4 |
| 45 | 408 | 2 |
| 55 | 408 | 4 |
| 59 | 408 | 1 |
+---------+------------+----------+
EXPLAIN SELECT SQL_NO_CACHE `pay_account_logs`.* FROM `pay_account_logs` FORCE KEY(PRIMARY) WHERE `pay_account_logs`.`account_id` = 408\G;
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pay_account_logs
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 1684051
Extra: Using where
1 row in set (0.00 sec)
可以看到,如果在 有 (account_id, log_type, id) 的情况下,强制按照主键排序,会用不到索引。 再次证明了,如果在这种多列索引中,id 就不一定是升序排列的。
3Q for @hooopo
@trenttian 本来想去成都的,临时变化了。人在帝都,心在蓉城。
我总结了下,在下面这个情况下,id 不是升序排列的情况。
表结构索引如下:
CREATE TABLE `happy_for_ni_logs` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`account_id` int(11) NOT NULL DEFAULT '0' ,
`b_amount` int(11) NOT NULL DEFAULT '0' ,
`r_amount` int(11) NOT NULL DEFAULT '0' ,
`amount` int(11) NOT NULL DEFAULT '0' ,
`log_type` int(11) NOT NULL DEFAULT '0' ,
`created_at` datetime NOT NULL,
`updated_at` datetime NOT NULL,
PRIMARY KEY (`id`),
KEY `idx_created_at` (`created_at`,`id`),
KEY `idx_account_type` (`account_id`,`log_type`,`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
三个 column 的索引
mysql> explain SELECT SQL_NO_CACHE `happy_for_ni_logs`.* FROM `happy_for_ni_logs` WHERE `happy_for_ni_logs`.`account_id` = 408 AND (created_at >= '2015-08-23 00:00:00');
+----+-------------+------------------+------+---------------------------------+------------------+---------+-------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------------+------+---------------------------------+------------------+---------+-------+------+-------------+
| 1 | SIMPLE | happy_for_ni_logs | ref | idx_created_at,idx_account_type | idx_account_type | 4 | const | 4 | Using where |
+----+-------------+------------------+------+---------------------------------+------------------+---------+-------+------+-------------+
1 row in set (0.01 sec)
mysql> SELECT SQL_NO_CACHE `happy_for_ni_logs`.* FROM `happy_for_ni_logs` WHERE `happy_for_ni_logs`.`account_id` = 408;
+---------+------------+----------------+---------------+----------+----------+---------------------+---------------------+
| id | account_id | b_amount | r_amount | amount | log_type | created_at | updated_at |
+---------+------------+----------------+---------------+----------+----------+---------------------+---------------------+
| 59 | 408 | 0 | 0 | -210000 | 1 | 2015-08-23 13:18:43 | 2015-08-23 13:18:43 |
| 45 | 408 | 0 | 0 | -1000000 | 2 | 2015-08-23 13:14:44 | 2015-08-23 13:14:44 |
| 43 | 408 | 1000000 | 0 | 1000000 | 4 | 2015-08-23 13:13:46 | 2015-08-23 13:13:46 |
| 55 | 408 | 210000 | 0 | 210000 | 4 | 2015-08-23 13:18:17 | 2015-08-23 13:18:17 |
+---------+------------+----------------+---------------+----------+----------+---------------------+---------------------+
结果并不是按照id正序排列的
explain SELECT SQL_NO_CACHE `happy_for_ni_logs`.* FROM `happy_for_ni_logs` WHERE `happy_for_ni_logs`.`account_id` = 408 order by id desc;
+----+-------------+------------------+------+----------------------------+----------------------------+---------+-------+------+-----------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------------+------+----------------------------+----------------------------+---------+-------+------+-----------------------------+
| 1 | SIMPLE | happy_for_ni_logs | ref | idx_of_account_id_log_type | idx_of_account_id_log_type | 4 | const | 4 | Using where; Using filesort |
+----+-------------+------------------+------+----------------------------+----------------------------+---------+-------+------+-----------------------------+
1 row in set (0.01 sec)
explain SELECT SQL_NO_CACHE `happy_for_ni_logs`.* FROM `happy_for_ni_logs` WHERE `happy_for_ni_logs`.`account_id` = 408 order by log_type desc, id desc;
+----+-------------+------------------+------+----------------------------+----------------------------+---------+-------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------------+------+----------------------------+----------------------------+---------+-------+------+-------------+
| 1 | SIMPLE | happy_for_ni_logs | ref | idx_of_account_id_log_type | idx_of_account_id_log_type | 4 | const | 4 | Using where |
+----+-------------+------------------+------+----------------------------+----------------------------+---------+-------+------+-------------+
1 row in set (0.01 sec)
explain SELECT SQL_NO_CACHE `happy_for_ni_logs`.* FROM `happy_for_ni_logs` WHERE `happy_for_ni_logs`.`account_id` = 408 order by log_type asc, id desc;
+----+-------------+------------------+------+----------------------------+----------------------------+---------+-------+------+-----------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------------+------+----------------------------+----------------------------+---------+-------+------+-----------------------------+
| 1 | SIMPLE | happy_for_ni_logs | ref | idx_of_account_id_log_type | idx_of_account_id_log_type | 4 | const | 4 | Using where; Using filesort |
+----+-------------+------------------+------+----------------------------+----------------------------+---------+-------+------+-----------------------------+
1 row in set (0.01 sec)
explain SELECT SQL_NO_CACHE `happy_for_ni_logs`.* FROM `happy_for_ni_logs` WHERE `happy_for_ni_logs`.`account_id` = 408 order by log_type desc;
+----+-------------+------------------+------+----------------------------+----------------------------+---------+-------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------------+------+----------------------------+----------------------------+---------+-------+------+-------------+
| 1 | SIMPLE | happy_for_ni_logs | ref | idx_of_account_id_log_type | idx_of_account_id_log_type | 4 | const | 4 | Using where |
+----+-------------+------------------+------+----------------------------+----------------------------+---------+-------+------+-------------+
1 row in set (0.01 sec)
数据按照 account_id
,log_type
,id
升序排序
两个 column 的情况
如果索引是如下方式创建
mysql> alter table happy_for_ni_logs add index `idx_of_account_id` (`account_id`, `id`);
mysql> SELECT SQL_NO_CACHE `happy_for_ni_logs`.* FROM `happy_for_ni_logs` WHERE `happy_for_ni_logs`.`account_id` = 408;
+---------+------------+----------------+---------------+----------+----------+---------------------+---------------------+
| id | account_id | b_amount | r_amount | amount | log_type | created_at | updated_at |
+---------+------------+----------------+---------------+----------+----------+---------------------+---------------------+
| 43 | 408 | 1000000 | 0 | 1000000 | 4 | 2015-08-23 13:13:46 | 2015-08-23 13:13:46 |
| 45 | 408 | 0 | 0 | -1000000 | 2 | 2015-08-23 13:14:44 | 2015-08-23 13:14:44 |
| 55 | 408 | 210000 | 0 | 210000 | 4 | 2015-08-23 13:18:17 | 2015-08-23 13:18:17 |
| 59 | 408 | 0 | 0 | -210000 | 1 | 2015-08-23 13:18:43 | 2015-08-23 13:18:43 |
+---------+------------+----------------+---------------+----------+----------+---------------------+---------------------+
4 rows in set (0.00 sec)
mysql> explain SELECT SQL_NO_CACHE `happy_for_ni_logs`.* FROM `happy_for_ni_logs` WHERE `happy_for_ni_logs`.`account_id` = 408;
+----+-------------+------------------+------+-------------------+-------------------+---------+-------+------+-------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------------+------+-------------------+-------------------+---------+-------+------+-------+
| 1 | SIMPLE | happy_for_ni_logs | ref | idx_of_account_id | idx_of_account_id | 4 | const | 4 | |
+----+-------------+------------------+------+-------------------+-------------------+---------+-------+------+-------+
1 row in set (0.01 sec)
以上是按照id正序排列的
EXPLAIN SELECT SQL_NO_CACHE `happy_for_ni_logs`.* FROM `happy_for_ni_logs` WHERE `happy_for_ni_logs`.`account_id` = 408 ORDER BY id desc;
+----+-------------+------------------+------+-------------------+-------------------+---------+-------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------------+------+-------------------+-------------------+---------+-------+------+-------------+
| 1 | SIMPLE | happy_for_ni_logs | ref | idx_of_account_id | idx_of_account_id | 4 | const | 4 | Using where |
+----+-------------+------------------+------+-------------------+-------------------+---------+-------+------+-------------+
1 row in set (9.51 sec)
以上情况并没有出现 Using filesort 的情况
数据按照account_id
,id
升序排列
在没有索引的情况
mysql> explain SELECT SQL_NO_CACHE `happy_for_ni_logs`.* FROM `happy_for_ni_logs` WHERE `happy_for_ni_logs`.`account_id` = 408;
+----+-------------+------------------+------+---------------+------+---------+------+---------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------------+------+---------------+------+---------+------+---------+-------------+
| 1 | SIMPLE | happy_for_ni_logs | ALL | NULL | NULL | NULL | NULL | 1684032 | Using where |
+----+-------------+------------------+------+---------------+------+---------+------+---------+-------------+
1 row in set (0.00 sec)
mysql> SELECT SQL_NO_CACHE `happy_for_ni_logs`.* FROM `happy_for_ni_logs` WHERE `happy_for_ni_logs`.`account_id` = 408;
+---------+------------+----------------+---------------+----------+----------+---------------------+---------------------+
| id | account_id | balance_amount | refund_amount | amount | log_type | created_at | updated_at |
+---------+------------+----------------+---------------+----------+----------+---------------------+---------------------+
| 43 | 408 | 1000000 | 0 | 1000000 | 4 | 2015-08-23 13:13:46 | 2015-08-23 13:13:46 |
| 45 | 408 | 0 | 0 | -1000000 | 2 | 2015-08-23 13:14:44 | 2015-08-23 13:14:44 |
| 55 | 408 | 210000 | 0 | 210000 | 4 | 2015-08-23 13:18:17 | 2015-08-23 13:18:17 |
| 59 | 408 | 0 | 0 | -210000 | 1 | 2015-08-23 13:18:43 | 2015-08-23 13:18:43 |
+---------+------------+----------------+---------------+----------+----------+---------------------+---------------------+
4 rows in set (2.47 sec)
数据按照主键 id 升序排列。
@oxffff @msg7086 谢两位仁兄,我参照,https://dev.mysql.com/doc/ 详细研究下。
👍,javaeye 真的好怀念
@foghost Bingo 所以慢查询优化的时候,尽量避免那些偏门的的技巧。数据库升级会带来不小的隐患。
@msg7086 我没有说,认为日常的请求都是和 count(1) 相近 呀。文章中我只是表明了一种情况。为了避免这个疑惑,我还是将文章中的 平均查询时间 改成 查询时间。
这里指明 存储引擎为 InnoDB。如果使用 MyISAM 就不会保证,记录是按照 主键 Id 排序的
mysql> CREATE TABLE `foo` (
-> `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
-> `bar` char(10) DEFAULT NULL,
-> PRIMARY KEY (`id`),
-> UNIQUE KEY `id` (`id`)
-> ) ENGINE=MyISAM;
Query OK, 0 rows affected (0.08 sec)
mysql> INSERT INTO foo (bar) VALUES
-> ('testing'), ('testing'), ('testing'), ('testing'), ('testing'),
-> ('testing'), ('testing'), ('testing'), ('testing'), ('testing');
Query OK, 10 rows affected (0.01 sec)
Records: 10 Duplicates: 0 Warnings: 0
mysql> DELETE FROM foo WHERE id BETWEEN 4 AND 7;
Query OK, 4 rows affected (0.01 sec)
mysql> SELECT * FROM foo;
+----+---------+
| id | bar |
+----+---------+
| 1 | testing |
| 2 | testing |
| 3 | testing |
| 8 | testing |
| 9 | testing |
| 10 | testing |
+----+---------+
6 rows in set (0.01 sec)
mysql> INSERT INTO foo (bar) VALUES
-> ('testing'), ('testing'), ('testing'), ('testing'), ('testing'),
-> ('testing'), ('testing'), ('testing'), ('testing'), ('testing');
Query OK, 10 rows affected (0.00 sec)
Records: 10 Duplicates: 0 Warnings: 0
mysql> SELECT * FROM foo;
+----+---------+
| id | bar |
+----+---------+
| 1 | testing |
| 2 | testing |
| 3 | testing |
| 14 | testing |
| 13 | testing |
| 12 | testing |
| 11 | testing |
| 8 | testing |
| 9 | testing |
| 10 | testing |
| 15 | testing |
| 16 | testing |
| 17 | testing |
| 18 | testing |
| 19 | testing |
| 20 | testing |
+----+---------+
16 rows in set (0.01 sec)
@oxffff 第一个问题,平均查询时间就是个近似值,一个查询肯定是不能代表的。不过 MySQL 同一个查询,消耗的时间很相近。 第二个查询,请戳这里 http://segmentfault.com/q/1010000000342176 ,这里的疑问是,理论上用不到索引,但为啥这里用到了索引?难道是 MySQL 做了什么优化处理么?
显示执行计划
mysql> EXPLAIN SELECT `artwork_transactions`.`id`,
-> `artwork_transactions`.`artwork_id`,
-> `artwork_transactions`.`transaction_date`,
-> `artwork_transactions`.`trans_type`,
-> `artwork_transactions`.`display_price`,
-> `artwork_transactions`.`icon_img_url`
-> FROM `artwork_transactions`
-> INNER JOIN `artworks`
-> ON `artwork_transactions`.`artwork_id` = `artworks`.`id`
-> WHERE `artworks`.`artist_id` = 28896
-> ORDER BY `artwork_transactions`.`transaction_date` ASC
-> LIMIT 25
-> OFFSET 0\G;
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: artwork_transactions
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 11083 <- 返回条数太多
Extra: Using filesort <- 使用排序不好
*************************** 2. row ***************************
id: 1
select_type: SIMPLE
table: artworks
type: eq_ref
possible_keys: PRIMARY,_ArtistID
key: PRIMARY
key_len: 4
ref: tao800.artwork_transactions.artwork_id
rows: 1
Extra: Using where
2 rows in set (0.00 sec)
使用如下方式进行优化
第一步
修改column属性
alter table artwork_transactions modify column transaction_date timestamp;
添加索引
alter table artwork_transactions add index `index_of_transaction_date` (`transaction_date`, `id`);
alter table artwork_transactions add index `idx_of_artwork_id` (`artwork_id`, `id`);
第二步
mysql> EXPLAIN SELECT `artwork_transactions`.`id`,
-> `artwork_transactions`.`artwork_id`,
-> `artwork_transactions`.`transaction_date`,
-> `artwork_transactions`.`trans_type`,
-> `artwork_transactions`.`display_price`,
-> `artwork_transactions`.`icon_img_url`
-> FROM `artwork_transactions`
-> STRAIGHT_JOIN `artworks`
-> ON `artwork_transactions`.`artwork_id` = `artworks`.`id`
-> WHERE `artworks`.`artist_id` = 28896
-> ORDER BY `artwork_transactions`.`transaction_date` ASC
-> LIMIT 25
-> OFFSET 0\G;
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: artwork_transactions
type: index
possible_keys: idx_of_artwork_id
key: index_of_transaction_date
key_len: 12
ref: NULL
rows: 25 <- 返回行数大大降低
Extra: <- 并且未使用排序
*************************** 2. row ***************************
id: 1
select_type: SIMPLE
table: artworks
type: eq_ref
possible_keys: PRIMARY,_ArtistID
key: PRIMARY
key_len: 4
ref: artwork_transactions.artwork_id
rows: 1
Extra: Using where
@sun1752709589 有道理,extra 显示很明显使用到了索引。那就推翻了之前说到的,索引 (a,b) 其中 b 是用不到索引的这种说法。这种理论网上一搜一大把这种。