博客ob425黑屏搭建集群以及三种方式搭建备库（包含主备切换）

ob425黑屏搭建集群以及三种方式搭建备库（包含主备切换）-c7电子娱乐

2024-11-22用户实践

本文是个超长篇的实操笔记，具体的命令和详情c7电子娱乐官网也都有，汇总和记录下。

黑屏创建ob集群

1.所需软件包

oceanbase-4.2.5.0-100000082024102022.el7.x86_64.rpm --observer软件包 obproxy-4.3.2.0-432024100811.el7.x86_64.rpm --obproxy软件包 tsar-master.zip --tsar软件包 t-oceanbase-antman-1.3.9-1932687.alios7.x86_64.rpm ---t-antman软件包 obclient-2.2.5-20231227133752.el7.x86_64.rpm ---客户端软件包

2.环境准备

环境按照observer准备初始化，这也是我使用antman的原因。

3.单节点搭建observer

安装前置准备

####安装observer软件包#########
rpm -ivh  oceanbase-4.2.5.0-100000082024102022.el7.x86_64.rpm
cd  /home/admin
chown -r admin:admin oceanbase
#####创建所需目录#########
su - admin
mkdir -p  /home/admin/oceanbase/store/obzry/
mkdir -p /data/1/obzry/{etc3,sstable,slog}
mkdir -p /data/log1/obzry/{clog,etc2}
#######创建软连接###########
for t in {etc3,sstable,slog};
do
ln -s /data/1/obzry/$t /home/admin/oceanbase/store/obzry/$t;
done
for t in {clog,etc2};
do
ln -s /data/log1/obzry/$t /home/admin/oceanbase/store/obzry/$t;
done
#########检查目录##########
cd  /home/admin/oceanbase
[admin@obzrycs01 oceanbase]$ tree store
store
└── obzry
    ├── clog -> /data/log1/obzry/clog
    ├── etc2 -> /data/log1/obzry/etc2
    ├── etc3 -> /data/1/obzry/etc3
    ├── slog -> /data/1/obzry/slog
    └── sstable -> /data/1/obzry/sstable

启动进程

su - admin
cd  /home/admin/oceanbase && /home/admin/oceanbase/bin/observer -i 10.10.10.111 -p 2881 -p 2882 -z zone1 -d /home/admin/oceanbase/store/obzry -r '10.10.10.111:2882:2881' -c 1 -n obzry -o "config_additional_dir=/data/1/obzry/etc3;/data/log1/obzry/etc2"
#########检查#########
[admin@obzrycs01 oceanbase]$ ps -ef|grep observer
admin      65306       1 46 15:26 ?        00:00:07 /home/admin/oceanbase/bin/observer -i 10.10.10.111 -p 2881 -p 2882 -z zone1 -d /home/admin/oceanbase/store/obzry -r 10.10.10.111:2882:2881 -c 1 -n obzry -o config_additional_dir=/data/1/obzry/etc3;/data/log1/obzry/etc2
admin      65993   64118  0 15:26 pts/0    00:00:00 grep --color=auto observer
[admin@obzrycs01 oceanbase]$ netstat -ntlp
(not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
active internet connections (only servers)
proto recv-q send-q local address           foreign address         state       pid/program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               listen      -                   
tcp        0      0 0.0.0.0:2881            0.0.0.0:*               listen      65306/observer      
tcp        0      0 0.0.0.0:2882            0.0.0.0:*               listen      65306/observer      
tcp        0      0 127.0.0.1:199           0.0.0.0:*               listen      -                   
tcp6       0      0 :::22                   :::*                    listen      -                   
tcp6       0      0 :::2882                 :::*                    listen      65306/observer

集群bootstrap

obclient -h127.0.0.1 -uroot -p2881
set session ob_query_timeout=720000000000;
alter system bootstrap zone 'zone1' server '10.10.10.111:2882';
obclient [(none)]> show databases;
 -------------------- 
| database           |
 -------------------- 
| information_schema |
| lbacsys            |
| mysql              |
| oceanbase          |
| oraauditor         |
| sys                |
| sys_external_tbs   |
| test               |
 -------------------- 
8 rows in set (0.003 sec)
####设置密码###########
obclient [(none)]> alter user root identified by 'weddvfd324@@@';
query ok, 0 rows affected (0.036 sec)
########创建proxyro用户##########
create  user proxyro identified by "oxxxxxbxxx##123";
grant select  on *.* to proxyro;
###########查询sha1转化的密码（启动proxy的时候用）#############
select sha1('oxxxxxbxxx##123')
 ------------------------------------------ 
| sha1('oxxxxxbxxx##123')                  |
 ------------------------------------------ 
| 5b5b18b0f24996458bbcfd60575e0252bc0552bc |
 ------------------------------------------ 
obclient [(none)]> select sha1('oxxxxxbxxx##321');
 ------------------------------------------ 
| sha1('oxxxxxbxxx##321')                  |
 ------------------------------------------ 
| 40f912e7f41137f46136894d7daed35933b8fbf8 |
 ------------------------------------------ 
1 row in set (0.000 sec)

4.安装proxy

软件安装

####修改系统文件########
chmod  u w /etc/sudoers
echo 'admin all=(all) nopasswd:all'  >> /etc/sudoers
chmod  u-w /etc/sudoers
#########解压软件############
rpm -ivh  obproxy-4.3.2.0-432024100811.el7.x86_64.rpm
#######创建软连接##########
[root@obzrycs01 home]# cd /opt/taobao/install/
[root@obzrycs01 install]# ls
obproxy-4.3.2.0
[root@obzrycs01 install]# ln -s obproxy-4.3.2.0 obproxy
chown -r admin:admin obproxy

启动proxy

su - admin
 cd /opt/taobao/install/obproxy 
./bin/obproxy -p 2883 -s 2885 -r '10.10.10.111:2881' -n obproxy_zry -o observer_sys_password=5b5b18b0f24996458bbcfd60575e0252bc0552bc,obproxy_sys_password=40f912e7f41137f46136894d7daed35933b8fbf8,enable_cluster_checkout=false,proxy_id=1,client_session_id_version=2 -c obzry

备集群搭建

1.环境准备

环境准备与上面操作一致，集群名称可以任意取

2.主库创建主租户和测试数据


create resource unit s1_unit_config
memory_size = '100g',
max_cpu = 40, min_cpu = 40,
log_disk_size = '200g',
max_iops = 100000, min_iops = 100000, iops_weight=1;
create resource pool mq_pool_01 
unit='s1_unit_config', 
unit_num=1, 
zone_list=('zone1'); 
create tenant if not exists oracle_zry1 
primary_zone='zone1', 
resource_pool_list=('mq_pool_01')
set ob_tcp_invited_nodes='%',
ob_compatibility_mode='oracle';
obclient -h10.10.10.111 -usys@oracle_zry1#obzry -p2883
alter user sys identified by "weddvfd324@@@";      
obclient [sys]> create table  zrycs (a varchar2(20),b int);
query ok, 0 rows affected (0.132 sec)
obclient [sys]> insert into  zrycs values ('张瑞远',150);
query ok, 1 row affected (0.033 sec)
obclient [sys]> 
obclient [sys]> 
obclient [sys]> commit;
query ok, 0 rows affected (0.001 sec)

3.备集群搭建

（一）空白租户创建备集群

创建空备租户适用于主租户为新创建，或您可以确认该主租户当前拥有租户自创建后的完整日志的场景。

1.主库查看主租户上的日志流对比信息，sys租户执行。

obclient [oceanbase]> (select ls_id from oceanbase.cdb_ob_ls_history where tenant_id = 1001) except (select ls_id from oceanbase.cdb_ob_ls where tenant_id = 1002);
empty set (0.017 sec)
obclient [oceanbase]> 
obclient [oceanbase]> select ls_id, begin_lsn from oceanbase.gv$ob_log_stat where tenant_id = 1002 and role = 'leader' ;
 ------- ----------- 
| ls_id | begin_lsn |
 ------- ----------- 
|     1 |         0 |
|  1001 |         0 |
 ------- ----------- 
2 rows in set (0.006 sec)

begin_lsn 表示当前日志流副本保存的最早的日志 lsn（log sequence number），如果 begin_lsn的值为 0，则表示当前日志流副本拥有自创建以来完整的日志。根据查询结果，日志流副本所对应的 begin_lsn 的值为 0，表示当前日志流副本拥有自创建以来完整的日志；如果该租户的所有日志流副本对应的 begin_lsn 的值均为 0，则表示该租户的所有日志流均拥有完整日志，可以通过 create standby tenant 语句创建空备租户

计算备租户资源（不必完全按照）


obclient [oceanbase]> call dbms_ob_limit_calculator.calculate_min_phy_res_needed_by_standby_tenant(primary_tenant_id => 1002, standby_tenant_unit_num => 1);
 ------------------------ ------------ 
| physical_resource_name | min_value  |
 ------------------------ ------------ 
| memstore               |          0 |
| memory                 | 4294967296 |
| data_disk              |          0 |
| clog_disk              | 2147483648 |
| cpu                    |          0 |
 ------------------------ ------------ 
5 rows in set (0.007 sec)
query ok, 0 rows affected (0.007 sec)

创建同步账号


obclient -h10.10.10.111 -usys@oracle_zry1#obzry -p2883 -p'weddvfd324@@@'
create user rep_user identified by "rep_user123";
grant standby_replication to rep_user;

创建空白备租户资源

obclient -h10.10.10.222 -uroot@sys#obzry_bak -p2883 -p'weddvfd324@@@'
create resource unit s1_unit_config
                memory_size = '100g',
                max_cpu = 40, min_cpu = 40,
                log_disk_size = '200g',
                max_iops = 100000, min_iops = 100000, iops_weight=1;
create resource pool mq_pool_01 
                unit='s1_unit_config', 
                unit_num=1, 
                zone_list=('zone1');

主库查询备库创建日志源需要的信息

obclient [sys]> select  * from dba_ob_access_point;
 ----------- ------------- --------------- ---------- 
| tenant_id | tenant_name | svr_ip        | sql_port |
 ----------- ------------- --------------- ---------- 
|      1002 | oracle_zry1 | 10.10.10.111 |     2881 |
 ----------- ------------- --------------- ----------

备库执行创建备库操作


obclient [oceanbase]> create standby tenant if not exists oracle_zry1 log_restore_source = "service=10.10.10.111:2881 user=rep_user@oracle_zry1 password=rep_user123" resource_pool_list=('mq_pool_01');
query ok, 0 rows affected (14.958 sec)

检查租户状态

obclient [oceanbase]> select tenant_name, tenant_type, create_time, status, tenant_role,scn_to_timestamp(sync_scn) from oceanbase.dba_ob_tenants where tenant_name = 'oracle_zry1';
 ------------- ------------- ---------------------------- -------- ------------- ---------------------------- 
| tenant_name | tenant_type | create_time                | status | tenant_role | scn_to_timestamp(sync_scn) |
 ------------- ------------- ---------------------------- -------- ------------- ---------------------------- 
| oracle_zry1 | user        | 2024-11-20 16:44:48.016641 | normal | standby     | 2024-11-20 16:46:57.541396 |
 ------------- ------------- ---------------------------- -------- ------------- ---------------------------- 
1 row in set (0.010 sec)

根据查询结果，tenant_role的值为 standby, 且 status 的值为 normal，则表示备租户状态正常，备租户已创建成功。

验证备租户

obclient -h10.10.10.222 -usys@oracle_zry1#obzry_bak -p2883 -p'weddvfd324@@@'
obclient [sys]> select  * from  zrycs;
 ----------- ------ 
| a         | b    |
 ----------- ------ 
| 张瑞远    |  150 |
 ----------- ------ 
1 row in set (0.041 sec)
obclient [sys]> insert into  zrycs values ('张正赟',120);
ora-16000: standby tenant is read only

主库执行

insert into  zrycs values ('张正赟',120);
commit

备租户查询

obclient [sys]> select  * from  zrycs;
 ----------- ------ 
| a         | b    |
 ----------- ------ 
| 张瑞远    |  150 |
| 张正赟    |  120 |
 ----------- ------ 
2 rows in set (0.000 sec)

（二）使用备份恢复（带日志）功能创建备租户

归档介质使用 nfs，归档目的端路径为 /nfs_zry/zry425/sh_archive/

主租户下执行，或者主sys租户指定tenant

alter system set log_archive_dest = 'location=file:///nfs_zry/zry425/sh_archive';
alter system archivelog;
########alter system archivelog tenant = mysql;
#####select dest_id, round_id, dest_no, status, checkpoint_scn, checkpoint_scn_display, path from oceanbase.cdb_ob_archivelog;
obclient [sys]> select dest_id, round_id, dest_no, status, checkpoint_scn, checkpoint_scn_display, path from sys.dba_ob_archivelog;
 --------- ---------- --------- ----------- --------------------- --------------------------------- -------------------------------- 
| dest_id | round_id | dest_no | status    | checkpoint_scn      | checkpoint_scn_display          | path                           |
 --------- ---------- --------- ----------- --------------------- --------------------------------- -------------------------------- 
|    1001 |        1 |       0 | beginning | 1732151691642371000 | 21-nov-24 09.14.51.642371000 am | file:///nfs_zry/zry425/sh_archive |
 --------- ---------- --------- ----------- --------------------- --------------------------------- -------------------------------- 
1 row in set (0.007 sec)

主租户查看主租户的归档进度

## select * from oceanbase.cdb_ob_archivelog\g 
obclient [sys]> select * from sys.dba_ob_archivelog\g
*************************** 1. row ***************************
dest_id: 1001
round_id: 1
incarnation: 1
dest_no: 0
status: doing
start_scn: 1732151691642371000
start_scn_display: 21-nov-24 09.14.51.642371000 am
checkpoint_scn: 1732151691642371001
checkpoint_scn_display: 21-nov-24 09.14.51.642371001 am
compatible: 1
base_piece_id: 1
used_piece_id: 1
piece_switch_interval: 86400000000
unit_size: 1
compression: none
input_bytes: 59202014
input_bytes_display: 56.46mb
output_bytes: 59202014
output_bytes_display: 56.46mb
compression_ratio: 1
deleted_input_bytes: 0
deleted_input_bytes_display: 0mb
deleted_output_bytes: 0
deleted_output_bytes_display: 0mb
comment: null
path: file:///nfs_zry/zry425/sh_archive
1 row in set (0.033 sec)

设置主租户的备份目的端

###alter system set data_backup_dest = 'file:///data/1/sh_databackup' tenant = mysql;
alter system set data_backup_dest = 'file:///nfs_zry/zry425/sh_databackup';
备份
###alter system backup tenant = mysql;
主租户备份
alter system backup database;
--新增测试数据
insert into  zrycs values ('张芮遠',100);
commit;
确认数据备份状态
##select * from oceanbase.cdb_ob_backup_jobs;
oracle:
select * from sys.dba_ob_backup_jobs;
历史
##select * from oceanbase.cdb_ob_backup_job_history;
obclient [sys]> select * from sys.dba_ob_backup_job_history;
 -------- ------------- --------------- --------------------- ------------------ -------------------- ----------------- ------------- ------------- ----------------- -------- --------------------- --------------------- ----------- -------- --------- ------------- ----------------------------------- 
| job_id | incarnation | backup_set_id | initiator_tenant_id | initiator_job_id | executor_tenant_id | plus_archivelog | backup_type | job_level   | encryption_mode | passwd | start_timestamp     | end_timestamp       | status    | result | comment | description | path                              |
 -------- ------------- --------------- --------------------- ------------------ -------------------- ----------------- ------------- ------------- ----------------- -------- --------------------- --------------------- ----------- -------- --------- ------------- ----------------------------------- 
|      1 |           1 |             1 |                1002 |                0 | 1002               | off             | full        | user_tenant | none            | null   | 2024-11-21 09:43:37 | 2024-11-21 09:45:26 | completed |      0 | null    | null        | file:///nfs_zry/zry425/sh_databackup |
 -------- ------------- --------------- --------------------- ------------------ -------------------- ----------------- ------------- ------------- ----------------- -------- --------------------- --------------------- ----------- -------- --------- ------------- ----------------------------------- 
1 row in set (0.034 sec)

--（可选）主租户关闭归档模式

创建新备租户资源

obclient -h10.10.10.222 -uroot@sys#obzry_bak -p2883 -p'weddvfd324@@@'
create resource unit s2_unit_config
                memory_size = '100g',
                max_cpu = 40, min_cpu = 40,
                log_disk_size = '200g',
                max_iops = 100000, min_iops = 100000, iops_weight=1;
create resource pool zry_pool_01 
                unit='s2_unit_config', 
                unit_num=1, 
                zone_list=('zone1');

主库sys租户查询


obclient [oceanbase]> select * from oceanbase.cdb_ob_archivelog\g
*************************** 1. row ***************************
tenant_id: 1002
dest_id: 1001
round_id: 1
incarnation: 1
dest_no: 0
status: doing
start_scn: 1732151691642371000
start_scn_display: 2024-11-21 09:14:51.642371
checkpoint_scn: 1732154211467090004
checkpoint_scn_display: 2024-11-21 09:56:51.467090
compatible: 1
base_piece_id: 1
used_piece_id: 1
piece_switch_interval: 86400000000
unit_size: 1
compression: none
input_bytes: 65875798
input_bytes_display: 62.82mb
output_bytes: 65875798
output_bytes_display: 62.82mb
compression_ratio: 1.00
deleted_input_bytes: 0
deleted_input_bytes_display: 0.00mb
deleted_output_bytes: 0
deleted_output_bytes_display: 0.00mb
comment: 
path: file:///nfs_zry/zry425/sh_archive
1 row in set (0.017 sec)

从主租户的带有归档日志的完整数据集中恢复出备租户

alter system restore oracle_zry3 from 'file:///nfs_zry/zry425/sh_databackup,file:///nfs_zry/zry425/sh_archive' until time='2024-11-21 09:56:51.467090' with 'pool_list=zry_pool_01';

确认恢复是否完成


obclient [(none)]> select tenant_name, tenant_type, create_time, status, tenant_role,scn_to_timestamp(sync_scn)  from oceanbase.dba_ob_tenants where tenant_name = 'oracle_zry3';
 ------------- ------------- ---------------------------- -------- ------------- ---------------------------- 
| tenant_name | tenant_type | create_time                | status | tenant_role | scn_to_timestamp(sync_scn) |
 ------------- ------------- ---------------------------- -------- ------------- ---------------------------- 
| oracle_zry3 | user        | 2024-11-21 10:00:34.100851 | normal | standby     | 2024-11-21 09:56:51.467090 |
 ------------- ------------- ---------------------------- -------- ------------- ---------------------------- 
1 row in set (0.025 sec)

设置日志恢复源

alter system set log_restore_source = 'service=10.10.10.111:2881 user=rep_user@oracle_zry1 password=rep_user123' tenant = oracle_zry3;

设置成功后，系统租户（sys租户）或用户租户可以分别通过cdb_ob_log_restore_source 视图和 dba_ob_log_restore_source 视图，确认日志恢复源是否修改成功

obclient [oceanbase]> select  * from  oceanbase.cdb_ob_log_restore_source;
 ----------- ------ --------- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- --------------------- 
| tenant_id | id   | type    | value                                                                                                                                                                                                  | recovery_until_scn  |
 ----------- ------ --------- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- --------------------- 
|      1008 |    1 | service | ip_list=10.10.10.111:2881,user=rep_user@oracle_zry1,password=85aa3f6d6e11152bf7624ad553a454e209221d105d6db782a787b9ef0af865cd,tenant_id=1002,cluster_id=1,compatibility_mode=oracle,is_encrypted=true | 4611686018427387903 |
|      1010 |    1 | service | ip_list=10.10.10.111:2881,user=rep_user@oracle_zry1,password=436d7278f9d06984854d2c9d178c8aa88225de21a7088668f6c7db9bebc36fe0,tenant_id=1002,cluster_id=1,compatibility_mode=oracle,is_encrypted=true | 1732154211467090000 |
 ----------- ------ --------- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- --------------------- 
2 rows in set (0.014 sec)

日志链（主库查询）

select * from oceanbase.gv$ob_log_transport_dest_stat;

验证同步

obclient -h10.10.10.222 -usys@oracle_zry3#obzry_bak -p2883 -p'weddvfd324@@@'
obclient [sys]>  select  * from  zrycs ;      
 ----------- ------ 
| a         | b    |
 ----------- ------ 
| 张瑞远    |  150 |
| 张正赟    |  120 |
| 张芮遠    |  100 |
 ----------- ------ 
3 rows in set (0.015 sec)
主库：
obclient -h10.10.10.111 -usys@oracle_zry1#obzry -p2883 -p'weddvfd324@@@'
insert into zrycs values('张锐源','60');
commit;
备库：
obclient [sys]> select  * from  zrycs ;  
 ----------- ------ 
| a         | b    |
 ----------- ------ 
| 张瑞远    |  150 |
| 张正赟    |  120 |
| 张芮遠    |  100 |
 ----------- ------ 
3 rows in set (0.001 sec)

设置日志恢复终点

##alter system recover standby [tenant = tenant_name] until unlimited;
obclient [sys]> alter system recover standby  until unlimited;
query ok, 0 rows affected (0.006 sec)
obclient [sys]> select  * from  zrycs ;  
 ----------- ------ 
| a         | b    |
 ----------- ------ 
| 张瑞远    |  150 |
| 张正赟    |  120 |
| 张芮遠    |  100 |
| 张锐源    |   60 |
 ----------- ------ 
4 rows in set (0.001 sec)

（三）使用 backup database plus archivelog 功能创建备租户

主租户开启归档模式并执行数据备份 --因上一步已经开启归档了就不重复了

主租户使用 backup database plus archivelog 功能执行数据备份

改下备份目录

##alter system set data_backup_dest = 'file:///data/1/sh_databackup1' tenant = mysql;
##alter system set data_backup_dest = 'file:///nfs_zry/zry425/sh_databackup1';
alter system set data_backup_dest = 'file:///data/1/sh_databackup1';
##alter system backup tenant = oracle_zry1 plus archivelog; 
obclient [sys]> alter system backup database plus archivelog;
query ok, 0 rows affected (0.009 sec)

确认数据备份状态

obclient [(none)]> select * from oceanbase.cdb_ob_backup_jobs;
oracle:
obclient [sys]> select * from sys.dba_ob_backup_jobs;
 -------- ------------- --------------- --------------------- ------------------ -------------------- ----------------- ------------- ------------- ----------------- -------- --------------------- --------------- -------- -------- --------- ------------- ------------------------------- 
| job_id | incarnation | backup_set_id | initiator_tenant_id | initiator_job_id | executor_tenant_id | plus_archivelog | backup_type | job_level   | encryption_mode | passwd | start_timestamp     | end_timestamp | status | result | comment | description | path                          |
 -------- ------------- --------------- --------------------- ------------------ -------------------- ----------------- ------------- ------------- ----------------- -------- --------------------- --------------- -------- -------- --------- ------------- ------------------------------- 
|      3 |           1 |             3 |                1002 |                0 | 1002               | on              | full        | user_tenant | none            | null   | 2024-11-21 10:38:10 | null          | doing  |      0 | null    | null        | file:///data/1/sh_databackup1 |
 -------- ------------- --------------- --------------------- ------------------ -------------------- ----------------- ------------- ------------- ----------------- -------- --------------------- --------------- -------- -------- --------- ------------- ------------------------------- 
1 row in set (0.007 sec)

查询备份历史

select * from oceanbase.cdb_ob_backup_job_history;
*************************** 3. row ***************************
job_id: 3
incarnation: 1
backup_set_id: 3
initiator_tenant_id: 1002
initiator_job_id: 0
executor_tenant_id: 1002
plus_archivelog: on
backup_type: full
job_level: user_tenant
encryption_mode: none
passwd: null
start_timestamp: 2024-11-21 10:38:10
end_timestamp: 2024-11-21 10:42:02
status: completed
result: 0
comment: null
description: null
path: file:///data/1/sh_databackup1
3 rows in set (0.007 sec)

文件打包传输到备集群的机器

su - admin
cd /data/1/
tar -czvf sh_databackup1.tar.gz sh_databackup1
传输到备集群解压
tar -xzvf sh_databackup1.tar.gz

--（可选）主租户关闭归档模式

备库创建资源

obclient -h10.10.10.222 -uroot@sys#obzry_bak -p2883 -p'weddvfd324@@@'
create resource unit s3_unit_config
                memory_size = '100g',
                max_cpu = 40, min_cpu = 40,
                log_disk_size = '200g',
                max_iops = 100000, min_iops = 100000, iops_weight=1;
create resource pool zry_pool_02 
                unit='s3_unit_config', 
                unit_num=1, 
                zone_list=('zone1');

执行恢复

alter system restore oracle_zry4 from 'file:///data/1/sh_databackup1' with 'pool_list=zry_pool_02';

确认恢复是否完成


obclient [oceanbase]>  select tenant_name, tenant_type, create_time, status, tenant_role,scn_to_timestamp(sync_scn)  from oceanbase.dba_ob_tenants where tenant_name = 'oracle_zry4';
 ------------- ------------- ---------------------------- --------- ------------- ---------------------------- 
| tenant_name | tenant_type | create_time                | status  | tenant_role | scn_to_timestamp(sync_scn) |
 ------------- ------------- ---------------------------- --------- ------------- ---------------------------- 
| oracle_zry4 | user        | 2024-11-21 11:09:44.383241 | restore | restore     | 1970-01-01 08:00:00.000000 |
 ------------- ------------- ---------------------------- --------- ------------- ---------------------------- 
1 row in set (0.022 sec)
obclient [oceanbase]>  select tenant_name, tenant_type, create_time, status, tenant_role,scn_to_timestamp(sync_scn)  from oceanbase.dba_ob_tenants where tenant_name = 'oracle_zry4';
 ------------- ------------- ---------------------------- -------- ------------- ---------------------------- 
| tenant_name | tenant_type | create_time                | status | tenant_role | scn_to_timestamp(sync_scn) |
 ------------- ------------- ---------------------------- -------- ------------- ---------------------------- 
| oracle_zry4 | user        | 2024-11-21 11:09:44.383241 | normal | standby     | 2024-11-21 11:01:48.133777 |
 ------------- ------------- ---------------------------- -------- ------------- ---------------------------- 
1 row in set (0.013 sec)

设置日志恢复源


alter system set log_restore_source = 'service=10.10.10.111:2881 user=rep_user@oracle_zry1 password=rep_user123' tenant = oracle_zry4;

设置成功后，系统租户（sys 租户）或用户租户可以分别通过 cdb_ob_log_restore_source 视图和 dba_ob_log_restore_source 视图确认日志恢复源是否修改成功

obclient [oceanbase]> select  * from  oceanbase.cdb_ob_log_restore_source;
 ----------- ------ --------- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- --------------------- 
| tenant_id | id   | type    | value                                                                                                                                                                                                  | recovery_until_scn  |
 ----------- ------ --------- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- --------------------- 
|      1008 |    1 | service | ip_list=10.10.10.111:2881,user=rep_user@oracle_zry1,password=85aa3f6d6e11152bf7624ad553a454e209221d105d6db782a787b9ef0af865cd,tenant_id=1002,cluster_id=1,compatibility_mode=oracle,is_encrypted=true | 4611686018427387903 |
|      1010 |    1 | service | ip_list=10.10.10.111:2881,user=rep_user@oracle_zry1,password=436d7278f9d06984854d2c9d178c8aa88225de21a7088668f6c7db9bebc36fe0,tenant_id=1002,cluster_id=1,compatibility_mode=oracle,is_encrypted=true | 4611686018427387903 |
|      1012 |    1 | service | ip_list=10.10.10.111:2881,user=rep_user@oracle_zry1,password=8c0c773131341e21819efba6b94dcc8d7384815f2cdd1d977507eb458382fbf6,tenant_id=1002,cluster_id=1,compatibility_mode=oracle,is_encrypted=true | 1732158108133777000 |
 ----------- ------ --------- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------

日志链主库查询


obclient [(none)]> select * from oceanbase.gv$ob_log_transport_dest_stat;
 ----------- --------------- ---------- ------- --------------- ------------ ------------------ ------------- ---------------------------- ---------------------------- ------------------ -------------------------- ---------------------- ---------------------- --------------------- -------------------------- ------------------------ --------------------------- --------------------------- ----------------------------- 
| tenant_id | svr_ip        | svr_port | ls_id | client_ip     | client_pid | client_tenant_id | client_type | start_serve_time           | last_serve_time            | last_read_source | last_request_type        | last_request_log_lsn | last_request_log_scn | last_failed_request | avg_request_process_time | avg_request_queue_time | avg_request_read_log_time | avg_request_read_log_size | avg_log_transport_bandwidth |
 ----------- --------------- ---------- ------- --------------- ------------ ------------------ ------------- ---------------------------- ---------------------------- ------------------ -------------------------- ---------------------- ---------------------- --------------------- -------------------------- ------------------------ --------------------------- --------------------------- ----------------------------- 
|      1002 | 10.10.10.111 |     2882 |     1 | 10.10.10.222 |      39515 |             1010 | standby     | 2024-11-21 10:12:48.841463 | 2024-11-21 11:14:51.480241 | online           | sequential_read_parallel |            127647744 |  1732158891331754000 | null                |                        5 |                    104 |                         4 |                      1904 | 36.07kb/s                   |
|      1002 | 10.10.10.111 |     2882 |     1 | 10.10.10.222 |      39515 |             1008 | standby     | 2024-11-20 16:55:28.110781 | 2024-11-21 11:14:51.480395 | online           | sequential_read_parallel |            127647744 |  1732158891331754000 | null                |                        4 |                     97 |                         3 |                      1904 | 36.07kb/s                   |
|      1002 | 10.10.10.111 |     2882 |  1001 | 10.10.10.222 |      39515 |             1008 | standby     | 2024-11-20 16:55:33.277167 | 2024-11-21 11:14:51.480119 | online           | sequential_read_parallel |             84992000 |  1732158891256454000 | null                |                        4 |                     89 |                         3 |                      2085 | 39.51kb/s                   |
|      1002 | 10.10.10.111 |     2882 |  1001 | 10.10.10.222 |      39515 |             1010 | standby     | 2024-11-21 10:12:48.842884 | 2024-11-21 11:14:51.480332 | online           | sequential_read_parallel |             84992000 |  1732158891256454000 | null                |                        4 |                    102 |                         4 |                      2085 | 39.51kb/s                   |
 ----------- --------------- ---------- ------- --------------- ------------ ------------------ ------------- ---------------------------- ---------------------------- ------------------ -------------------------- ---------------------- ---------------------- --------------------- -------------------------- ------------------------ --------------------------- --------------------------- -----------------------------

验证主备同步


obclient -h10.10.10.222 -usys@oracle_zry4#obzry_bak -p2883 -p'weddvfd324@@@'
obclient [sys]>  select  * from  zrycs ; 
 ----------- ------ 
| a         | b    |
 ----------- ------ 
| 张瑞远    |  150 |
| 张正赟    |  120 |
| 张芮遠    |  100 |
| 张锐源    |   60 |
 ----------- ------ 
4 rows in set (0.030 sec)
主库：
obclient -h10.10.10.111 -usys@oracle_zry1#obzry -p2883 -p'weddvfd324@@@'
insert into zrycs values('张蕊缘','240');
commit;
备库：
obclient [sys]> select  * from  zrycs ;  
 ----------- ------ 
| a         | b    |
 ----------- ------ 
| 张瑞远    |  150 |
| 张正赟    |  120 |
| 张芮遠    |  100 |
| 张锐源    |   60 |
 ----------- ------ 
4 rows in set (0.001 sec)

设置日志恢复终点


##alter system recover standby [tenant = tenant_name] until unlimited;
obclient [sys]> alter system recover standby  until unlimited;
query ok, 0 rows affected (0.005 sec)
obclient [sys]> select  * from  zrycs ;                       
 ----------- ------ 
| a         | b    |
 ----------- ------ 
| 张瑞远    |  150 |
| 张正赟    |  120 |
| 张芮遠    |  100 |
| 张锐源    |   60 |
| 张蕊缘    |  240 |
 ----------- ------ 
5 rows in set (0.001 sec)

主备切换

（一）switchover

1. 基于网络传输的切换

>主切成备

登录主租户或主租户所在集群的 sys 租户

执行 switchover to standby verify 命令，验证 switchover 命令是否可以执行成功。
###obclient> alter system switchover to standby tenant = mysql verify;
obclient [sys]> alter system switchover to standby verify;
query ok, 0 rows affected (0.005 sec)

验证通过后，执行主切备命令，将主租户切换为备租户。

主租户所在集群的 sys 租户将主租户切换为备租户

###alter system switchover to standby tenant = mysql;
alter system switchover to standby;

查询 dba_ob_tenants 视图，确认主租户是否已切换为备租户


obclient [(none)]> select tenant_name, tenant_type, tenant_role, switchover_status from oceanbase.dba_ob_tenants;
 ------------- ------------- ------------- ------------------- 
| tenant_name | tenant_type | tenant_role | switchover_status |
 ------------- ------------- ------------- ------------------- 
| sys         | sys         | primary     | normal            |
| meta$1002   | meta        | primary     | normal            |
| oracle_zry1 | user        | standby     | normal            |
 ------------- ------------- ------------- ------------------- 
3 rows in set (0.014 sec)

根据查询结果，tenant_role为 standby，且 switchover_status 为 normal，则表示主切备成功。

>将备租户切换为主租户。（基于网络）


obclient [oceanbase]> alter system switchover to primary tenant = oracle_zry2 verify;
query ok, 0 rows affected (0.046 sec)
###obclient> alter system switchover to primary verify;

验证通过后，执行备切主命令，将备租户切换为主租户

备租户所在集群的 sys 租户将备租户切换为主租户


obclient [oceanbase]> alter system switchover to primary tenant = oracle_zry2;
query ok, 0 rows affected (0.172 sec)
########alter system switchover to primary;

查询dba_ob_tenants 视图，确认备租户是否已切换为主租户

备租户所在集群的 sys 租户查询视图

###########select tenant_name, tenant_type, tenant_role, switchover_status from sys.dba_ob_tenants;
obclient [oceanbase]> select tenant_name, tenant_type, tenant_role, switchover_status from oceanbase.dba_ob_tenants;
 ------------- ------------- ------------- ------------------- 
| tenant_name | tenant_type | tenant_role | switchover_status |
 ------------- ------------- ------------- ------------------- 
| sys         | sys         | primary     | normal            |
| meta$1008   | meta        | primary     | normal            |
| oracle_zry2 | user        | primary     | normal            |
| meta$1010   | meta        | primary     | normal            |
| oracle_zry3 | user        | standby     | normal            |
| meta$1012   | meta        | primary     | normal            |
| oracle_zry4 | user        | standby     | normal            |
 ------------- ------------- ------------- ------------------- 
7 rows in set (0.013 sec)

根据查询结果可知，主租户的tenant_role 已变为 primary，且 switchover_status 变为 normal，则表示备切主成功。

为原主租户设置日志恢复源

设置日志恢复源的操作可以在主备切换操作之前执行，也可以在主备切换操作之后执行。

原主租户所在集群的 sys 租户设置原主租户的日志恢复源

obclient [oceanbase]> alter system set log_restore_source = 'service=10.10.10.222:2881 user=rep_user@oracle_zry2 password=rep_user123' tenant = oracle_zry1;
query ok, 0 rows affected (0.083 sec)

设置日志恢复终点


##alter system recover standby [tenant = tenant_name] until unlimited;
obclient [sys]> alter system recover standby  until unlimited;
query ok, 0 rows affected (0.004 sec)

验证主备同步

###新主
obclient -h10.10.10.222 -usys@oracle_zry2#obzry_bak -p2883 -p'weddvfd324@@@' 
obclient [sys]> select * from  zry;
empty set (0.008 sec)
obclient [sys]> insert into zry values('张瑞远','150');
query ok, 1 row affected (0.002 sec)
obclient [sys]> commit;
query ok, 0 rows affected (0.002 sec)
####原主
obclient -h10.10.10.111 -usys@oracle_zry1#obzry -p2883 -p'weddvfd324@@@' 
obclient [sys]> select  * from  zry;
 ----------- ------ 
| a         | b    |
 ----------- ------ 
| 张瑞远    |  150 |
 ----------- ------ 
1 row in set (0.009 sec)
###其他备
obclient -h10.10.10.222 -usys@oracle_zry3#obzry_bak -p2883 -p'weddvfd324@@@' 
obclient [sys]> select  * from  zry;
 ----------- ------ 
| a         | b    |
 ----------- ------ 
| 张瑞远    |  150 |
 ----------- ------ 
1 row in set (0.008 sec)

2. 基于日志的主备切换

这部分没有单独测试（在下一部分级联环境切换中有相关步骤）步骤整理如下

备租户开启日志归档模式--> 将主租户切换为备租户--> 将备租户切换为主租户1.备租户开启归档模式

alter system set log_archive_dest='location=file:///data/1/standby2_archive2/ binding=optional' tenant = standby2;

alter system archivelog tenant = standby2;

备租户所在集群的 sys 租户查询视图

select dest_id, round_id, dest_no, status, checkpoint_scn, checkpoint_scn_display, path from oceanbase.cdb_ob_archivelog;

根据查询结果，status 的值为 doing，表示归档处于正常工作状态，checkpoint_scn 和 checkpoint_scn_display 为当前的归档进度信息。

2.将主租户切换为备租户

alter system switchover to standby verify;alter system switchover to standby;

查询dba_ob_tenants 视图，确认主租户是否已切换为备租户。

主租户所在集群的 sys 租户查询视图

select tenant_id, tenant_name,tenant_type,tenant_role,switchover_statusfrom oceanbase.dba_ob_tenantswhere tenant_name='mysql2'

主租户的tenant_role 已变为 standby，且 switchover_status 变为 normal，则表示主切备成功。

查询v$ob_archive_dest_status 视图中的 synchronized 字段，等待主租户归档完成。

基于日志归档的物理备库场景中，备租户在切换为主租户之前，需要从原主租户的日志归档上读取完整的日志。

由于日志归档为异步同步模式，因此在原主租户切换为备租户后，需要检查原主租户上的日志归档是否完整。

select * from sys.v$ob_archive_dest_status where tenant_id = 1004;

根据查询结果，如果synchronized 字段显示为 yes，则表示主租户已归档完成。

3.将备租户切换为主租户

alter system switchover to primary verify;alter system switchover to primary;

查询dba_ob_tenants 视图，确认备租户是否已切换为主租户。

备租户所在集群的 sys 租户查询视图

obclient> select tenant_id, tenant_name,tenant_type,tenant_role,switchover_statusfrom oceanbase.dba_ob_tenantswhere tenant_name='standby2';

设置新备租户（原主租户）的恢复源，接收新主租户的归档日志。

如果原主租户在执行切换操作前，未设置过日志恢复源，则其日志同步位点会停留在执行切换操作的时间点。因此，需要为原主租户设置日志恢复源，且日志恢复源指向新的主租户。

新备租户设置本租户的恢复源

alter system set log_restore_source ='location=file:///data/1/standby2/archive2' tenant = mysql2;

原主租户所在集群的 sys 租户查询视图

select tenant_name, tenant_type, tenant_role, switchover_status, scn_to_timestamp(sync_scn), recovery_until_scn from oceanbase.dba_ob_tenants;

（二）faildover

注意事项：

执行 failover 时，要求备租户所有日志流的副本均在线，否则需要等待对应的副本永久下线。
备租户或备租户所在集群的sys 租户可以分别通过dba_ob_ls 视图或 cdb_ob_ls 视图查询所有日志流副本是否在线，有关日志流副本的详细说明信息
执行 failover 时，不会检查对应主租户的状态，因此除主租户故障的容灾需求外，failover 操作也可以用于独立的一个主租户在某一个时间点的快照并用于后续业务验证的需求场景。
不允许在通过服务名建立的 session 中执行 failover 命令。
failover 操作只会对日志文件做修改，不会对数据文件做修改。由于 oceanbase 数据库的每个租户均有多个日志流，而 failover 操作需要在操作执行完成后达到数据一致的状态，故系统会选择所有日志流的同步位点中 scn 最小的值作为failover 的执行位点。执行failover操作后，租户下的所有日志流都会统一回退到该位点。

操作前测试

obclient -h10.10.10.222 -usys@oracle_zry3#obzry_bak -p2883 -p'weddvfd324@@@'
obclient [sys]> insert into zry values('张惢渊','120');
ora-16000: standby tenant is read only

执行 activate standby verify 命令，验证 activate standby 命令可以执行成功后执行failedover命令。


obclient [oceanbase]> alter system activate standby tenant = oracle_zry3 verify;
query ok, 0 rows affected (0.005 sec)
obclient [oceanbase]> alter system activate standby tenant = oracle_zry3;
query ok, 0 rows affected (0.131 sec)

检查

备租户查询视图

obclient [oceanbase]> select tenant_name, tenant_type, tenant_role, switchover_status from oceanbase.dba_ob_tenants;
 ------------- ------------- ------------- ------------------- 
| tenant_name | tenant_type | tenant_role | switchover_status |
 ------------- ------------- ------------- ------------------- 
| sys         | sys         | primary     | normal            |
| meta$1008   | meta        | primary     | normal            |
| oracle_zry2 | user        | standby     | normal            |
| meta$1010   | meta        | primary     | normal            |
| oracle_zry3 | user        | primary     | normal            |
| meta$1012   | meta        | primary     | normal            |
| oracle_zry4 | user        | primary     | normal            |
 ------------- ------------- ------------- ------------------- 
7 rows in set (0.014 sec)
#####验证
obclient [sys]> insert into zry values('张惢渊','100');
query ok, 1 row affected (0.001 sec)

failover 后的注意事项

由于 failover 操作会将所有日志流已同步的数据恢复到一致性位点，保证该位点之前的所有日志流的数据是完整的。故执行 failover 操作后不支持原主租户降备后接入成为新主租户的备租户。
对于第三方备租户，取决于备租户同步的日志，如果同步的日志超过一致性位点，则不支持接入作为新主租户的备租户。为了便于管理，不建议第三方备租户接入作为新主租户的备租户。

（三）特殊的主备切换方案

主要测试下网络和日志的混合级联备库的切换

现在架构如下

oracle_zry2#obzry_bak(主) -> oracle_zry1#obzry(网络传输) -> oracle_zry4#obzry_bak(日志传输)

目标是切换oracle_zry4#obzry_bak为新主 oracle_zry2#obzry_bak为备，从oracle_zry1#obzry网络传输

完成切换后的架构如下

oracle_zry4#obzry_bak(新主) -> oracle_zry1#obzry(日志传输) -> oracle_zry2#obzry_bak(网络传输)

需要按照如下步骤变更

1.oracle_zry4#obzry_bak（二级备）开启归档

alter system set log_archive_dest = 'location=file:///nfs_zry/zry425/cs_archive' tenant = oracle_zry4;
alter system archivelog tenant = oracle_zry4;
obclient [oceanbase]> select dest_id, round_id, dest_no, status, checkpoint_scn, checkpoint_scn_display, path from oceanbase.cdb_ob_archivelog;
 --------- ---------- --------- --------- ---------------- ------------------------ -------------------------------- 
| dest_id | round_id | dest_no | status  | checkpoint_scn | checkpoint_scn_display | path                           |
 --------- ---------- --------- --------- ---------------- ------------------------ -------------------------------- 
|    1001 |        1 |       0 | prepare |              0 | null                   | file:///nfs_zry/zry425/cs_archive |
 --------- ---------- --------- --------- ---------------- ------------------------ -------------------------------- 
1 row in set (0.012 sec)

2.oracle_zry2#obzry_bak （主）切换备

alter system switchover to standby tenant = oracle_zry2 verify;
query ok, 0 rows affected (0.003 sec)
alter system switchover to standby tenant = oracle_zry2;
obclient [oceanbase]> select tenant_name, tenant_type, tenant_role, switchover_status from oceanbase.dba_ob_tenants;
 ------------- ------------- ------------- ------------------- 
| tenant_name | tenant_type | tenant_role | switchover_status |
 ------------- ------------- ------------- ------------------- 
| sys         | sys         | primary     | normal            |
| meta$1008   | meta        | primary     | normal            |
| oracle_zry2 | user        | standby     | normal            |
| meta$1010   | meta        | primary     | normal            |
| oracle_zry3 | user        | standby     | normal            |
| meta$1012   | meta        | primary     | normal            |
| oracle_zry4 | user        | standby     | normal            |
 ------------- ------------- ------------- ------------------- 
7 rows in set (0.014 sec)

3.oracle_zry1#obzry（一级备）确认日志完整

obclient [sys]> select * from sys.v$ob_archive_dest_status where tenant_id = 1002;
 ----------- --------- -------------------------------- -------- --------------------- -------------- --------- 
| tenant_id | dest_id | path                           | status | checkpoint_scn      | synchronized | comment |
 ----------- --------- -------------------------------- -------- --------------------- -------------- --------- 
|      1002 |    1001 | file:///nfs_zry/zry425/sh_archive | doing  | 1732180488893078000 | no           | null    |
 ----------- --------- -------------------------------- -------- --------------------- -------------- --------- 
1 row in set (0.039 sec)

如果 synchronized 字段显示为 yes，则表示主租户已归档完成


obclient [sys]> select * from sys.v$ob_archive_dest_status where tenant_id = 1002;
 ----------- --------- -------------------------------- -------- --------------------- -------------- --------- 
| tenant_id | dest_id | path                           | status | checkpoint_scn      | synchronized | comment |
 ----------- --------- -------------------------------- -------- --------------------- -------------- --------- 
|      1002 |    1001 | file:///nfs_zry/zry425/sh_archive | doing  | 1732180520476199000 | yes          | null    |
 ----------- --------- -------------------------------- -------- --------------------- -------------- --------- 
1 row in set (0.026 sec)

4.oracle_zry1#obzry（一级备）修改日志恢复源到 oracle_zry4#obzry_bak（二级备）

alter system set log_restore_source = 'location=file:///nfs_zry/zry425/cs_archive' tenant = oracle_zry1;
alter system recover standby tenant = oracle_zry1 until unlimited;

5.oracle_zry2#obzry_bak(主) 修改日志恢复源到oracle_zry1#obzry（一级备）


alter system set log_restore_source = 'service=10.10.10.111:2881 user=rep_user@oracle_zry1 password=rep_user123' tenant = oracle_zry2;
alter system recover standby tenant = oracle_zry2 until unlimited;

6.oracle_zry4#obzry_bak（二级备）切换主

obclient [oceanbase]> alter system switchover to primary tenant = oracle_zry4 verify;
query ok, 0 rows affected (0.109 sec)
obclient [oceanbase]> alter system switchover to primary tenant = oracle_zry4;
query ok, 0 rows affected (0.440 sec)
obclient [oceanbase]> select tenant_name, tenant_type, tenant_role, switchover_status from oceanbase.dba_ob_tenants;
 ------------- ------------- ------------- ------------------- 
| tenant_name | tenant_type | tenant_role | switchover_status |
 ------------- ------------- ------------- ------------------- 
| sys         | sys         | primary     | normal            |
| meta$1008   | meta        | primary     | normal            |
| oracle_zry2 | user        | standby     | normal            |
| meta$1010   | meta        | primary     | normal            |
| oracle_zry3 | user        | standby     | normal            |
| meta$1012   | meta        | primary     | normal            |
| oracle_zry4 | user        | primary     | normal            |
 ------------- ------------- ------------- ------------------- 
7 rows in set (0.014 sec)

7.验证

####（新主）
obclient -h10.10.10.222 -usys@oracle_zry4#obzry_bak -p2883 -p'weddvfd324@@@' 
obclient [sys]>  insert into zry values('张正赟','120');
query ok, 1 row affected (0.001 sec)
obclient [sys]> commit;
query ok, 0 rows affected (0.002 sec)
#####(新一级备)
--确认位点更新
select  tenant_name,tenant_id,tenant_role,scn_to_timestamp(sync_scn) from  oceanbase.dba_ob_tenants where tenant_name like 'oracle_zry%'; 
 ------------- ----------- ------------- ---------------------------- 
| tenant_name | tenant_id | tenant_role | scn_to_timestamp(sync_scn) |
 ------------- ----------- ------------- ---------------------------- 
| oracle_zry1 |      1002 | standby     | 2024-11-21 17:24:55.577904 |
 ------------- ----------- ------------- ---------------------------- 
1 row in set (0.014 sec)
obclient -h10.10.10.111 -usys@oracle_zry1#obzry -p2883 -p'weddvfd324@@@' 
obclient [sys]> select  * from  zry;
 ----------- ------ 
| a         | b    |
 ----------- ------ 
| 张正赟    |  120 |
| 张瑞远    |  150 |
 ----------- ------ 
2 rows in set (0.001 sec)
#####(新二级备)
obclient -h10.10.10.222 -usys@oracle_zry2#obzry_bak -p2883 -p'weddvfd324@@@' 
obclient [sys]> select  * from  zry;
 ----------- ------ 
| a         | b    |
 ----------- ------ 
| 张正赟    |  120 |
| 张瑞远    |  150 |
 ----------- ------ 
2 rows in set (0.002 sec)