0
点赞
收藏
分享

微信扫一扫

postgresql/lightdb vacuum对性能的影响及彻底理解表膨胀

pipu 2022-12-10 阅读 62
执行时间postgresql子节点Spark大数据
加载失败

先来看测试结果
zjh@postgres=# create table big_table(id int,v text);
CREATE TABLE
zjh@postgres=# ALTER TABLE big_table SET (autovacuum_enabled = off);     # 关闭自动vacuum
ALTER TABLE
zjh@postgres=# insert into big_table select id, rpad('x',64,'x') from generate_series(1,1000000) id;
INSERT 0 1000000
zjh@postgres=# 
zjh@postgres=# \timing on
Timing is on.
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 76.603 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 60.682 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 60.963 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 60.632 ms           ### 稳定的执行时间
zjh@postgres=# update big_table set v = v, id = id;             #生成50%死元祖
UPDATE 1000000
Time: 1006.034 ms (00:01.006)
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 307.994 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 75.222 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 115.800 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 109.309 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 76.994 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 76.219 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 75.804 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 75.834 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 76.684 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 76.299 ms         ## 稳定后带死元祖的执行时间，25%以上的额外时延
zjh@postgres=# update big_table set v = v, id = id;
UPDATE 1000000
Time: 1923.425 ms (00:01.923)
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 352.238 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 103.585 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 211.861 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 144.573 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 99.129 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 100.284 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 99.148 ms     ## 两次update后，额外负载65%
zjh@postgres=# vacuum big_table ;
VACUUM
Time: 214.800 ms
zjh@postgres=# 
zjh@postgres=# vacuum big_table ;
VACUUM
Time: 11.348 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 88.478 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 85.893 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 87.403 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 85.340 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 85.990 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 86.514 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 85.684 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 84.336 ms    # vacuum之后，仍然还有40%的额外负载
zjh@postgres=# vacuum full big_table ;    # full vacuum
VACUUM
Time: 416.220 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 66.535 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 63.514 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 63.039 ms    ## 重组织后，负载基本消除，虽然也有一点点，但基本可以忽略不计了。
zjh@postgres=# update big_table set v = v, id = id;
UPDATE 1000000
Time: 1116.814 ms (00:01.117)
zjh@postgres=# update big_table set v = v, id = id;
UPDATE 1000000
Time: 2250.193 ms (00:02.250)
zjh@postgres=# update big_table set v = v, id = id;
UPDATE 1000000
Time: 1264.835 ms (00:01.265)
zjh@postgres=# update big_table set v = v, id = id;
UPDATE 1000000
Time: 1266.069 ms (00:01.266)
zjh@postgres=# update big_table set v = v, id = id;
UPDATE 1000000
Time: 2000.205 ms (00:02.000)
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 384.475 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 174.367 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 364.749 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 178.216 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 176.623 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 170.568 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 170.408 ms
zjh@postgres=# 
zjh@postgres=# 
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 172.460 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 171.467 ms    ## 五次update之后
zjh@postgres=# update big_table set v = v, id = id;
UPDATE 1000000
Time: 1114.980 ms (00:01.115)
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 455.640 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 197.581 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 335.761 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 212.413 ms
zjh@postgres=# select count(1) from big_table ;
  count  
---------
 1000000
(1 row)

Time: 195.236 ms    ## 六次update之后

zjh@postgres=# select pg_size_pretty(pg_relation_size('big_table'));
pg_size_pretty
----------------
675 MB    -- 可见膨胀是非常厉害的。
(1 row)

zjh@postgres=# vacuum big_table ;
VACUUM
Time: 662.148 ms
zjh@postgres=# select pg_size_pretty(pg_relation_size('big_table'));
 pg_size_pretty 
----------------
 675 MB     #  不回收，但是新的DML可以复用
(1 row)

Time: 0.406 ms
zjh@postgres=# vacuum full big_table ;
VACUUM
Time: 568.773 ms
zjh@postgres=# select pg_size_pretty(pg_relation_size('big_table'));
 pg_size_pretty 
----------------
 96 MB            # 回收，其实就是重组织
(1 row)

zjh@postgres=# vacuum big_table ;
VACUUM
Time: 40.442 ms
zjh@postgres=# 
zjh@postgres=# select pg_size_pretty(pg_relation_size('big_table'));
 pg_size_pretty 
----------------
 204 MB
(1 row)

Time: 0.331 ms
zjh@postgres=# update big_table set v = v, id = id limit 100000;
UPDATE 100000
Time: 98.444 ms
zjh@postgres=# update big_table set v = v, id = id limit 100000;
UPDATE 100000
Time: 105.852 ms
zjh@postgres=# update big_table set v = v, id = id limit 100000;
UPDATE 100000
Time: 104.016 ms
zjh@postgres=# select pg_size_pretty(pg_relation_size('big_table'));
 pg_size_pretty 
----------------
 204 MB
(1 row)

Time: 0.273 ms
zjh@postgres=# update big_table set v = v, id = id limit 100000;
UPDATE 100000
Time: 106.985 ms
zjh@postgres=# update big_table set v = v, id = id limit 100000;
UPDATE 100000
Time: 106.612 ms
zjh@postgres=# update big_table set v = v, id = id limit 100000;
UPDATE 100000
Time: 106.837 ms
zjh@postgres=# update big_table set v = v, id = id limit 100000;
UPDATE 100000
Time: 107.296 ms
zjh@postgres=# update big_table set v = v, id = id limit 100000;
UPDATE 100000
Time: 105.588 ms
zjh@postgres=# select pg_size_pretty(pg_relation_size('big_table'));
 pg_size_pretty 
----------------
 204 MB
(1 row)

Time: 0.347 ms
zjh@postgres=# update big_table set v = v, id = id limit 100000;
UPDATE 100000
Time: 103.695 ms
zjh@postgres=# update big_table set v = v, id = id limit 100000;
UPDATE 100000
Time: 104.665 ms
zjh@postgres=# update big_table set v = v, id = id limit 100000;
UPDATE 100000
Time: 106.016 ms
zjh@postgres=# select pg_size_pretty(pg_relation_size('big_table'));
 pg_size_pretty 
----------------
 204 MB
(1 row)

Time: 0.310 ms
zjh@postgres=# update big_table set v = v, id = id limit 100000;
UPDATE 100000
Time: 104.338 ms    # 此时所有死元祖回收的空间已经用完。
zjh@postgres=# select pg_size_pretty(pg_relation_size('big_table'));
 pg_size_pretty 
----------------
 212 MB    # 即使大量的死元祖，也会导致空间膨胀
(1 row)

Time: 0.446 ms
zjh@postgres=# update big_table set v = v, id = id limit 100000;
UPDATE 100000
Time: 109.384 ms
zjh@postgres=# select pg_size_pretty(pg_relation_size('big_table'));
 pg_size_pretty 
----------------
 222 MB    # 即使大量的死元祖，也会导致空间膨胀

(1 row) Time: 0.266 ms

　　从上可知，因为未清理的死元祖会占用额外空间、也会导致vm未被置为可见，所以不仅I/O更高、占用的buffer更多、也需要额外判断mvcc是否满足，进而对性能的影响还是比较大。只读特性确保避免意外操作导致表被更新，进而产生额外不必要的死元祖、vaccum活动以及vm被误干扰。

　　所以，对于大表做大量的update/delete（能truncate或insert select，就不要update/delete）之后，最好执行一次vacuum full xxx，释放空间。

postgresql中更新操作的物理实现

　　8.2版本引入了HOT特性，极大地提高了索引更新的性能。

 

postgresql/lightdb vacuum对性能的影响及彻底理解表膨胀_子节点

 

　　 8.2及之后，支持HOT，索引条目未更新不会插入新的，如下：

　　

postgresql/lightdb vacuum对性能的影响及彻底理解表膨胀_执行时间_02

 

 　　同时也会不定期的做碎片清理，如下：

　　

postgresql/lightdb vacuum对性能的影响及彻底理解表膨胀_子节点_03

 

 　　所以在10+新版中，CPU最多多两次指针访问+空间浪费，一段时间后，只会多一次指针访问+0空间浪费。相比这一次指针访问，索引叶子节点中定位索引键就好几次比较了，所以几乎可以忽略不计。

　　最后，vacuum除了能够解决清理死元组消除表膨胀外，冻结事务ID也是其职责，这影响到事务回卷，可参见 ​​postgresql中的事务回卷原理及预防措施​​。

　　具体可参考下​​https://zhuanlan.zhihu.com/p/379706959​​。



举报

相关推荐

提升MySQL性能的秘密：了解表压缩功能的使用及其对性能的影响。

数据库MySQL表空间yyds干货盘点

PostgreSQL 2023 Cc 大会 美女讲 index 维护,膨胀 与 vacuum

元组数据数据库Python后端开发

postgresql 表膨胀及优化

元组数据PostgreSQLNoSQL数据库

lightdb/postgresql性能分析之ltcenter

githubperlSQL虚拟化云计算

postgresql/lightdb中的collation实现、对order by的影响及与oracle/mysql异同点，c实现库参考

postgresqlhtmlicoPython后端开发

PostgreSQL的vacuum流程

postgresql数据库database

lightdb/postgresql中的事务回卷原理解析及避免

PostgreSQL数据库重启Spark大数据

带宽对网站性能的影响

ip地址javascript系统/运维

索引对MYSQL性能的影响

phpmysqlsql虚拟化云计算

PostgreSQL 16 关于vacuum freezing 性能提升的来由 与 之前PostgreSQL 的缺陷讨论

postgresql数据库gitPostgreSQLPython后端开发
0 条评论
加载失败