热门搜索
分布式
数据库一体机
云数据库
建设银行
保险核心
密集交易
登录
注册
达梦主备集群切换实战
技术分享/
文章详情
/
达梦主备集群切换实战
肖庚
2026/08/14
77
0
0
实验环境
本地的虚拟机环境，2个节点。
初始环境：节点1 是主库，节点2是备库。monitor在节点2上。
本实验中模拟了4个场景
场景一：monitor 和 备库服务器同时挂掉
场景二：monitor 存活，备库服务器挂掉
场景三：monitor 和 主库服务器挂掉
场景四：monitor存活，主库服务器挂掉
场景一：monitor 和 备库服务器同时挂掉
当前情况 节点1 主库 ， 节点2 备库。 monitor 在节点2上。直接停掉备库服务器。模拟 monitor和备库同时挂掉。
这时，节点1 的主库 状态会变为 SUSPEND PRIMARY。主库可以查询，但是插入数据时会出现卡住。并且也查询不到插入的数据。
SQL> select status$ from v$instance;
Server[LOCALHOST:5236]:mode is primary, state is suspend
connected
LINEID STATUS$
---------- -------
1 SUSPEND
SQL> select * from test_table;
LINEID ID
---------- -----------
1 1
used time: 3.411(ms). Execute id is 13700.
SQL> insert into test_table values (5);
affect rows 1
used time: 3.404(ms). Execute id is 13701.
SQL> commit;
^[[3~^H^C
Server[LOCALHOST:5236]:mode is primary, state is suspend
connected
executed successfully
used time: 1.874(ms). Execute id is 13800.
SQL> SQL> SQL> SQL>
SQL>
SQL> select * from test_table
2 ;
LINEID ID
---------- -----------
1 1
在节点1 中 使用 disql 执行
SQL> sp_set_para_value(1,'ALTER_MODE_STATUS',1);
SQL> alter database open force;
但是 节点1 的主库状态没有变化，并不能强制打开主库。
解决方案：
需要启动一个非确认监视器，手动的将节点1 主库变为open状态
dmmonitor /dm8/dmdbms/bin/dmmonitor_manual.ini
login
open database grp1.grp1_rt_01
强制 节点1 的数据库。这时 节点1 的数据库状态变为 open。
并且 之前在 SUSPEND 状态中 提交的数据，可以再open状态下查询到。
然后 正常启动节点2 的 DmWatcherServiceWatcher 和 DmServiceGRP1_RT_02 。
通过monitor 可以看到
[monitor] 2026-08-04 01:24:05:627: Received message from(GRP1_RT_02)
WTIME WSTATUS INST_OK INAME ISTATUS IMODE RSTAT N_OPEN FLSN CLSN
2026-08-04 01:24:05 OPEN OK GRP1_RT_02 OPEN STANDBY NULL 10 60373 60373
[monitor] 2026-08-04 01:24:05:628: Received message from(GRP1_RT_01)
WTIME WSTATUS INST_OK INAME ISTATUS IMODE RSTAT N_OPEN FLSN CLSN
2026-08-04 01:24:05 OPEN OK GRP1_RT_01 OPEN PRIMARY VALID 10 60373 60373
2个节点的状态都是正常了，主备集群的同步也是正常的。之前主库 SUSPEND 这个状态下插入的数据。在备库都能看到
SQL> select * from test_table;
LINEID ID
---------- -----------
1 1
2 2
3 5
场景二：monitor 存活，备库服务器挂掉
节点1 主库， 节点2备库，monitor在节点2上，
模拟故障情况，直接 杀掉 节点2 的 watch 和 dms 进程，观察现象。
[dmdba@dm8ha2 ~]$ ps -ef | grep dmw
dmdba 2099 1 0 01:21 pts/0 00:00:01 /dm8/dmdbms/bin/dmwatcher path=/dm8/dmdbms/data/DAMENG/dmwatcher.ini -noconsole
dmdba 2332 2005 0 01:41 pts/0 00:00:00 grep --color=auto dmw
[dmdba@dm8ha2 ~]$ ps -ef | grep dms
dmdba 2124 1 0 01:21 ? 00:00:05 /dm8/dmdbms/bin/dmserver /dm8/dmdbms/data/DAMENG/dm.ini mount
dmdba 2334 2005 0 01:41 pts/0 00:00:00 grep --color=auto dms
kill -9 2099 2124
通过monitor观察
show
2026-08-04 01:43:35
#================================================================================#
GROUP OGUID MON_CONFIRM MODE MPP_FLAG
GRP1 45331 FALSE AUTO FALSE
<<DATABASE GLOBAL INFO:>>
DW_IP MAL_DW_PORT WTIME WTYPE WCTLSTAT WSTATUS INAME INST_OK N_EP N_OK ISTATUS IMODE DSC_STATUS RTYPE RSTAT DETACHED AUTO_MODE_CHANGE
192.168.88.101 5436 2026-08-04 01:43:34 GLOBAL VALID OPEN GRP1_RT_01 OK 1 1 OPEN PRIMARY DSC_OPEN REALTIME VALID FALSE OFF
EP INFO:
INST_IP INST_PORT INST_OK INAME ISTATUS IMODE DSC_SEQNO DSC_CTL_NODE RTYPE RSTAT FSEQ FLSN CSEQ CLSN DW_STAT_FLAG
192.168.182.101 5236 OK GRP1_RT_01 OPEN PRIMARY 0 0 REALTIME VALID 3503 60380 3503 60380 NONE
#================================================================================#
[monitor] 2026-08-04 01:44:01:615: Dmwatcher process GRP1_RT_01 status switching [OPEN-->STARTUP]
WTIME WSTATUS INST_OK INAME ISTATUS IMODE RSTAT N_OPEN FLSN CLSN
2026-08-04 01:44:01 STARTUP OK GRP1_RT_01 SUSPEND PRIMARY VALID 10 60380 60382
[monitor] 2026-08-04 01:44:02:515: Dmwatcher process GRP1_RT_01 status switching [STARTUP-->MON CONFIRM]
WTIME WSTATUS INST_OK INAME ISTATUS IMODE RSTAT N_OPEN FLSN CLSN
2026-08-04 01:44:02 MON CONFIRM OK GRP1_RT_01 SUSPEND PRIMARY VALID 10 60380 60384
[monitor] 2026-08-04 01:44:03:239: Dmwatcher process GRP1_RT_01 status switching [MON CONFIRM-->FAILOVER]
WTIME WSTATUS INST_OK INAME ISTATUS IMODE RSTAT N_OPEN FLSN CLSN
2026-08-04 01:44:03 FAILOVER OK GRP1_RT_01 SUSPEND PRIMARY VALID 10 60380 60384
[monitor] 2026-08-04 01:44:05:519: Dmwatcher process GRP1_RT_01 status switching [FAILOVER-->OPEN]
WTIME WSTATUS INST_OK INAME ISTATUS IMODE RSTAT N_OPEN FLSN CLSN
2026-08-04 01:44:05 OPEN OK GRP1_RT_01 OPEN PRIMARY VALID 10 60385 60385
show
2026-08-04 01:44:14
#================================================================================#
GROUP OGUID MON_CONFIRM MODE MPP_FLAG
GRP1 45331 FALSE AUTO FALSE
<<DATABASE GLOBAL INFO:>>
DW_IP MAL_DW_PORT WTIME WTYPE WCTLSTAT WSTATUS INAME INST_OK N_EP N_OK ISTATUS IMODE DSC_STATUS RTYPE RSTAT DETACHED AUTO_MODE_CHANGE
192.168.88.101 5436 2026-08-04 01:44:14 GLOBAL VALID OPEN GRP1_RT_01 OK 1 1 OPEN PRIMARY DSC_OPEN REALTIME VALID FALSE OFF
EP INFO:
INST_IP INST_PORT INST_OK INAME ISTATUS IMODE DSC_SEQNO DSC_CTL_NODE RTYPE RSTAT FSEQ FLSN CSEQ CLSN DW_STAT_FLAG
192.168.182.101 5236 OK GRP1_RT_01 OPEN PRIMARY 0 0 REALTIME VALID 3506 60385 3506 60385 NONE
#================================================================================#
刚kill掉进程后
（2026-08-04 01:43:34）节点1 的主库是正常open状态
2026-08-04 01:44:01:615 节点1 [OPEN–>STARTUP] 启动到 SUSPEND 状态
2026-08-04 01:44:02:515 节点1[STARTUP–>MON CONFIRM] SUSPEND 状态
2026-08-04 01:44:03:239 节点1[MON CONFIRM–>FAILOVER] SUSPEND 状态
2026-08-04 01:44:05:519 节点1[FAILOVER–>OPEN] open 状态
说明在monitor存活的状态，可以自动的启动节点1 数据库从 suspend状态到open状态。
在节点1 上 执行 disql 也可以看到，插入数据时 会有短暂的卡顿，之后就正常commit；提交了。
然后在节点2 上 启动 DmWatcherServiceWatcher ，节点2 的备库 直接加入了 集群，主备正常了
#-----------------------------------------------------------------------------------------------#
[monitor] 2026-08-04 01:55:56:012: Received message from(GRP1_RT_02)
WTIME WSTATUS INST_OK INAME ISTATUS IMODE RSTAT N_OPEN FLSN CLSN
2026-08-04 01:55:56 OPEN OK GRP1_RT_02 OPEN STANDBY NULL 10 60385 60385
[monitor] 2026-08-04 01:55:56:013: Received message from(GRP1_RT_01)
WTIME WSTATUS INST_OK INAME ISTATUS IMODE RSTAT N_OPEN FLSN CLSN
2026-08-04 01:55:55 OPEN OK GRP1_RT_01 OPEN PRIMARY VALID 10 60385 60385
在备库上查看数据，故障期间插入的数据同步过来了。
SQL> select * from test_table;
LINEID ID
---------- -----------
1 1
2 2
3 5
4 7
场景三：monitor 和 主库服务器挂掉
先对 主备库做 switchover ，主库和 monitor 都在节点2上，备库在节点1 上。
做switchover时，连接主库的disql会话没有关闭。观察到。switchover后disql的会话没有报错和退出，如果再次执行命令，它是自动的重连了数据库，以standby身份登入了数据库做查询。
模拟故障 同时停掉 monitor 、watcher、dbserver进程
[dmdba@dm8ha2 ~]$ ps -ef | grep dms
dmdba 2404 1 0 01:55 ? 00:00:13 /dm8/dmdbms/bin/dmserver /dm8/dmdbms/data/DAMENG/dm.ini mount
dmdba 2626 2005 0 02:51 pts/0 00:00:00 grep --color=auto dms
[dmdba@dm8ha2 ~]$ ps -ef | grep dmw
dmdba 2379 1 0 01:55 pts/0 00:00:03 /dm8/dmdbms/bin/dmwatcher path=/dm8/dmdbms/data/DAMENG/dmwatcher.ini -noconsole
dmdba 2628 2005 0 02:51 pts/0 00:00:00 grep --color=auto dmw
[dmdba@dm8ha2 ~]$ ps -ef | grep dmm
dmdba 973 1 0 01:02 ? 00:00:06 /dm8/dmdbms/bin/dmmonitor path=/dm8/dmdbms/bin/dmmonitor.ini
dmdba 2630 2005 0 02:51 pts/0 00:00:00 grep --color=auto dmm
kill -9 973 2404 2379
观察备库和非确认monitor状态
备库会保持在 STANDBY 状态，非确认监控器可以看到 备库是正常的。
#-----------------------------------------------------------------------------------------------#
GET MONITOR CONNECT INFO FROM DMWATCHER(GRP1_RT_01), THE FIRST LINE IS SELF INFO.
DW_CONN_TIME MON_CONFIRM MID MON_IP MON_VERSION
2026-08-04 02:53:33 FALSE 1777843354 ::ffff:192.168.88.102 DMMONITOR[4.0] V8
#-----------------------------------------------------------------------------------------------#
[monitor] 2026-08-04 02:53:33:332: Received message from(GRP1_RT_01)
WTIME WSTATUS INST_OK INAME ISTATUS IMODE RSTAT N_OPEN FLSN CLSN
2026-08-04 02:53:33 OPEN OK GRP1_RT_01 OPEN STANDBY NULL 11 60438 60438
但备库不会自动变为主库。
在非确认监视器上执行 takeover是不起作用的。需要使用 takeover force。
choose takeover
no database can do takeover!
takeover force
Be careful to do so, this operation may cause data split, continue to do TAKEOVER FORCE with GRP1.GRP1_RT_01 or not(YES/NO/Y/N)?
y
[monitor] 2026-08-04 03:05:10:163: Start to takeover use instance GRP1_RT_01
[monitor] 2026-08-04 03:05:10:164: Notify dmwatcher(GRP1_RT_01) switch to TAKEOVER status
[monitor] 2026-08-04 03:05:10:166: Dmwatcher process GRP1_RT_01 status switching [OPEN-->TAKEOVER]
[monitor] 2026-08-04 03:05:10:267: Switch dmwatcher GRP1_RT_01 to TAKEOVER status success
[monitor] 2026-08-04 03:05:10:270: Instance GRP1_RT_01 start to execute sql SP_SET_GLOBAL_DW_STATUS(0, 7)
[monitor] 2026-08-04 03:05:10:321: Instance GRP1_RT_01 execute sql SP_SET_GLOBAL_DW_STATUS(0, 7) success
[monitor] 2026-08-04 03:05:10:322: Instance GRP1_RT_01 start to execute sql SP_APPLY_KEEP_PKG()
[monitor] 2026-08-04 03:05:10:373: Instance GRP1_RT_01 execute sql SP_APPLY_KEEP_PKG() success
[monitor] 2026-08-04 03:05:10:374: Instance GRP1_RT_01 start to execute sql ALTER DATABASE MOUNT
[monitor] 2026-08-04 03:05:10:427: Instance GRP1_RT_01 execute sql ALTER DATABASE MOUNT success
[monitor] 2026-08-04 03:05:10:427: Instance GRP1_RT_01 start to execute sql ALTER DATABASE PRIMARY
[monitor] 2026-08-04 03:05:10:478: Instance GRP1_RT_01 execute sql ALTER DATABASE PRIMARY success
[monitor] 2026-08-04 03:05:10:478: Notify instance GRP1_RT_01 to change all arch status to be invalid
[monitor] 2026-08-04 03:05:10:530: Succeed to change all instances arch status to be invalid
[monitor] 2026-08-04 03:05:10:531: Instance GRP1_RT_01 start to execute sql ALTER DATABASE OPEN FORCE
[monitor] 2026-08-04 03:05:10:682: Instance GRP1_RT_01 execute sql ALTER DATABASE OPEN FORCE success
[monitor] 2026-08-04 03:05:10:683: Instance GRP1_RT_01 start to execute sql SP_SET_GLOBAL_DW_STATUS(7, 0)
[monitor] 2026-08-04 03:05:10:734: Instance GRP1_RT_01 execute sql SP_SET_GLOBAL_DW_STATUS(7, 0) success
[monitor] 2026-08-04 03:05:10:735: Notify dmwatcher(GRP1_RT_01) switch to OPEN status
[monitor] 2026-08-04 03:05:10:737: Dmwatcher process GRP1_RT_01 status switching [TAKEOVER-->OPEN]
[monitor] 2026-08-04 03:05:10:837: Switch dmwatcher GRP1_RT_01 to OPEN status success
[monitor] 2026-08-04 03:05:10:838: Notify group(GRP1)'s dmwatcher to do clear
[monitor] 2026-08-04 03:05:10:892: Clean request of dmwatcher processer GRP1_RT_01 success
[monitor] 2026-08-04 03:05:10:892: Success to takeover use instance GRP1_RT_01
2026-08-04 03:05:10
#================================================================================#
GROUP OGUID MON_CONFIRM MODE MPP_FLAG
GRP1 45331 FALSE AUTO FALSE
<<DATABASE GLOBAL INFO:>>
DW_IP MAL_DW_PORT WTIME WTYPE WCTLSTAT WSTATUS INAME INST_OK N_EP N_OK ISTATUS IMODE DSC_STATUS RTYPE RSTAT DETACHED AUTO_MODE_CHANGE
192.168.88.101 5436 2026-08-04 03:05:10 GLOBAL VALID OPEN GRP1_RT_01 OK 1 1 OPEN PRIMARY DSC_OPEN REALTIME VALID FALSE OFF
EP INFO:
INST_IP INST_PORT INST_OK INAME ISTATUS IMODE DSC_SEQNO DSC_CTL_NODE RTYPE RSTAT FSEQ FLSN CSEQ CLSN DW_STAT_FLAG
192.168.182.101 5236 OK GRP1_RT_01 OPEN PRIMARY 0 0 REALTIME VALID 3513 60480 3513 60480 NONE
#===============================================================================#
这时 原来的备库 变为主库，插入了一条数据，看下 原主库恢复后能否同步数据。
原主库的 monitor 、watch、和数据库服务 启动后，原主库直接变为了备库。
#-----------------------------------------------------------------------------------------------#
[monitor] 2026-08-04 03:10:02:294: Received message from(GRP1_RT_02)
WTIME WSTATUS INST_OK INAME ISTATUS IMODE RSTAT N_OPEN FLSN CLSN
2026-08-04 03:10:02 OPEN OK GRP1_RT_02 OPEN STANDBY NULL 12 60488 60488
[monitor] 2026-08-04 03:10:02:295: Received message from(GRP1_RT_01)
WTIME WSTATUS INST_OK INAME ISTATUS IMODE RSTAT N_OPEN FLSN CLSN
2026-08-04 03:10:02 OPEN OK GRP1_RT_01 OPEN PRIMARY VALID 12 60488 60488
再查看测试表中的数据，可以看到 故障期间 在新的主库中插入的数据同步到了 新的备库。
SQL> select * from test_table;
LINEID ID
---------- -----------
1 1
2 2
3 5
4 7
5 9
场景四：monitor存活，主库服务器挂掉
主库在节点1 上， 备库和monitor在节点2上。
直接把节点1 服务器shutdown，通过monitor产看备库的状态。
#================================================================================#
[monitor] 2026-08-04 06:15:02:546: Dmwatcher process GRP1_RT_02 status switching [OPEN-->TAKEOVER]
WTIME WSTATUS INST_OK INAME ISTATUS IMODE RSTAT N_OPEN FLSN CLSN
2026-08-04 06:15:02 TAKEOVER OK GRP1_RT_02 OPEN STANDBY VALID 12 60502 60502
[monitor] 2026-08-04 06:15:03:056: [!!! Instance GRP1_RT_01 dmwatcher's DW_MODE is configured as AUTO, but this dmmonitor is not in CONFIRM mode, cannot takeover instance GRP1_RT_01 automatically !!!]
[monitor] 2026-08-04 06:15:03:057: Received message timeout from(GRP1_RT_01)
WTIME WSTATUS INST_OK INAME ISTATUS IMODE RSTAT N_OPEN FLSN CLSN
2026-08-04 06:14:41 ERROR OK GRP1_RT_01 OPEN PRIMARY VALID 12 60502 60502
[monitor] 2026-08-04 06:15:04:170: Dmwatcher process GRP1_RT_02 status switching [TAKEOVER-->OPEN]
WTIME WSTATUS INST_OK INAME ISTATUS IMODE RSTAT N_OPEN FLSN CLSN
2026-08-04 06:15:04 OPEN OK GRP1_RT_02 OPEN PRIMARY VALID 13 60538 60557
可以看到 备库大概在 5秒之后 成功自动的变为了主库。
2026-08-04 06:15:02:546 备库 [OPEN–>TAKEOVER]
2026-08-04 06:15:04:170 备库 [TAKEOVER–>OPEN]
备库变为主库
https://eco.dameng.com
评论
后发表回复
作者
文章
阅读量
获赞
扫一扫
联系客服
联系客服
鄂Copyright © 达梦国产化应用创新实验室
达梦在线服务平台社区用户管理规范
鄂ICP备18017926号-1
达梦在线服务平台社区用户管理规范
鄂ICP备18017926号-1