热门搜索
分布式
数据库一体机
云数据库
建设银行
保险核心
密集交易
注册
DM8 DW集群备库重建
专栏/技术分享/ 文章详情 /

DM8 DW集群备库重建

喜东东 2025/09/12 228 0 0
摘要

1. 概述

本文档介绍了在DM8数据守护（Data Watch，简称DW）集群环境中，当备库服务器故障且主库归档日志不全导致主备数据不一致且数据量较大时，如何通过增量备份重建的方式恢复备库并重新加入集群。内容包括同步机制简介、实时归档机制、模拟故障重建步骤详解及相关验证方法

2. 同步机制

DM8 DW集群通过实时归档将主库产生的 Redo 日志传输到备库，备库接收并重新应用 Redo 日志，从而实现备库与主库的数据同步。其核心组件包括：

  • Redo 日志传输：
    主备库之间的 Redo 日志传输，以日志包 RLOG_PKG 为单位，主库通过 MAL 系统发送 Redo 日志到备库。各种不同数据守护类型的区别，就在于主库日志包 RLOG_PKG 的发送时机，以及备库收到 Redo 日志后的处理策略

  • Redo 日志重演：
    Redo 日志重演的过程，就是备库收到主库发送的 Redo 日志后，在物理数据页上，重新修改数据的过程。Redo 日志重演由专门的 Redo 日志重演服务完成，重演服务严格按照 Redo 日志产生的先后顺序，解析 Redo 日志、修改相应的物理数据页，并且重演过程中备库会生成自身的 Redo 日志写入本地联机日志文件

  • 守护进程（dmwatcher） ：监控数据库实例的运行状态和主备库数据同步情况，接收数据库实例、其他守护进程、以及监视器发送的各种消息以及消息转发

    • 守护进程故障处理：

      故障自动切换模式下，备库故障后，如果主备库之间的归档状态仍然有效，主库的守护进程会先切换为 Confirm 状态，等待确认监视器的确认消息，如果确认为符合故障处理条件，主库守护进程再切换至 Failover 状态，将故障备库的归档失效

  • 监视器（dmmonitor） ：用来监控守护系统内守护进程、数据库实例信息，执行用户输
    入命令、监控实例故障、实现自动切换等

3. 实时归档

实时归档模式下，主库的redolog发送与备库对其重演流程如下：
image20250814144552gglkf60.png

说明：

  • 高性能模式： 备机收到重演日志，加到重演线程队列后响应主机，主机不需要等待备机重演完成后在响应主库
  • 事务一致模式： 备机收到重演日志，重演完成后响应主机，主机需要等待备机重演完成后响应主库（主要在读写分离场景下使用）

4. 模拟测试

4.1 模拟DW02备库服务器故障

  • 集群状态查看

    此时DW集群状态正常，且归档状态正常

    image20250910173517eonejyf.png

  • 停止备库守护进程以及数据库

    [dmdba@dmdbtest2:/home/dmdba]$ /home/dmdba/dmdbms/bin/DmWatcherServiceDMW stop
Stopping DmWatcherServiceDMW: 
                                                           [ OK ]
[dmdba@dmdbtest2:/home/dmdba]$ 
[dmdba@dmdbtest2:/home/dmdba]$ /home/dmdba/dmdbms/bin/DmServiceDMS stop
Stopping DmServiceDMS: 
                                                           [ OK ]

  • 删除归档

    主库删除归档，模拟主库归档意外丢失

    --查看归档信息
10:30:23 SYSDBA@DAMENG:5236 SQL> select arch_dest from V$DM_ARCH_INI;

ARCH_DEST 
----------
/db/dmarch
DW02
--删除归档
rm -f  /db/dmarch/*

  • 集群状态查看

    通过截图信息能够看到，守护进程状态异常，主备同步链路正常，且备库CLSN和FLSN与主库一致

    说明：

    • WSTATUS：守护进程状态

    • RSTAT：主库到备库的归档状态

    • CLSN：节点实例的系统当前 LSN，当前节点数据库最新产生的 LSN 值

    • FLSN：备库已经写入联机日志文件的最大 LSN 值

    image20250911145004hfj6h16.png

4.1 创建测试数据

  • 主库创建测试数据

    --创建测试表
CREATE TABLE CLUSTER_RECOVERY_TEST (
    ID INT PRIMARY KEY,                  -- 主键
    TRANSACTION_ID VARCHAR(36),          -- 事务ID
    AMOUNT DECIMAL(18,2),                -- 交易金额
    STATUS CHAR(1) DEFAULT 'A',          -- 状态 (A-有效, D-删除)
    CREATE_TIME TIMESTAMP DEFAULT CURRENT_TIMESTAMP,  -- 创建时间
    UPDATE_TIME TIMESTAMP                 
);

COMMENT ON TABLE CLUSTER_RECOVERY_TEST IS '集群故障恢复测试表';
COMMENT ON COLUMN CLUSTER_RECOVERY_TEST.TRANSACTION_ID IS '模拟业务流水号';
COMMENT ON COLUMN CLUSTER_RECOVERY_TEST.AMOUNT IS '交易金额';
COMMENT ON COLUMN CLUSTER_RECOVERY_TEST.STATUS IS '数据状态：A-活跃, D-逻辑删除';

--创建更新时间触发器
CREATE OR REPLACE TRIGGER TRG_UPDATE_TIME
BEFORE UPDATE ON CLUSTER_RECOVERY_TEST
FOR EACH ROW
BEGIN
    :NEW.UPDATE_TIME = CURRENT_TIMESTAMP;
END;

--生成10万条测试数据（含时间分布）
BEGIN
  FOR i IN 1..10000 LOOP
    INSERT INTO CLUSTER_RECOVERY_TEST (
      ID,
      TRANSACTION_ID,
      AMOUNT,
      STATUS
    ) VALUES (
      i,
      'TX' || LPAD(i, 8, '0') || TO_CHAR(SYSDATE - MOD(i,30), 'YYYYMMDD'), -- 带日期的业务ID
      ROUND(DBMS_RANDOM.VALUE(1, 100000), 2),  -- 随机金额(1~100000)
      CASE WHEN MOD(i,100) = 0 THEN 'D' ELSE 'A' END  -- 每100条标记1条删除
    );

    -- 每1000条提交一次（模拟真实事务）
    IF MOD(i, 1000) = 0 THEN
      COMMIT;
    END IF;
  END LOOP;
  COMMIT;
END;

--验证数据分布
09:20:04 SYSDBA@DAMENG:5236 SQL> SELECT 
  STATUS AS 状态,
  COUNT(*) AS 数据量,
  MIN(CREATE_TIME) AS 最早创建时间,
  MAX(UPDATE_TIME) AS 最近更新时间
FROM CLUSTER_RECOVERY_TEST
GROUP BY STATUS;2   3   4   5   6   7   

状态 数据量            最早创建时间         最近更新时间
------ -------------------- -------------------------- ------------------
A      9900                 2025-09-10 17:38:47.039096 NULL
D      100                  2025-09-10 17:38:47.041879 NULL

  • 查看集群信息

    通过截图信息能够看到，主备同步链路已经失效，且备库CLSN和FLSN均小于主库

    image202509101740287u49tx3.png

  • 查看主库数据库日志

    停掉备库守护进程和数据库后，主库日志报错，连接备库的5336（MAL系统监听端口）端口失败，导致实时归档失败

    2025-09-10 17:38:03.914 [ERROR] database P0000006938 T0000000000000007025  Can't connect to DM server on 'xxx.xx.110.187' port(5336) errno(111)
2025-09-10 17:38:08.560 [ERROR] database P0000006938 T0000000000000007038  [realtime arch] mal send to site(DW02, mal_seqno:1) failed, begin lsn:664692, end lsn:664702, code:-6010

    实时归档失败后，修改归档send type，修改数据库状态为suspend，修改dw状态为DW_FAILOVER，并将DW02的归档状态设置为INVALID，最后强制open主库

    2025-09-10 17:38:08.560 [INFO] database P0000006938 T0000000000000007038  Local arch send type to DW02 change from 1 to 0, clear recent arch send info!
2025-09-10 17:38:08.560 [ERROR] database P0000006938 T0000000000000007038  send realtime archive to instance[DW02] failed, code = -6010, begin_lsn = 664692, end_lsn = 664702!
2025-09-10 17:38:08.560 [INFO] database P0000006938 T0000000000000007038  rlog4_process_arch_failed, need_suspend:1
2025-09-10 17:38:08.560 [INFO] database P0000006938 T0000000000000007038  rlog4_process_arch_failed, reset req_ep_arr and res_ep_arr.
2025-09-10 17:38:08.560 [INFO] database P0000006938 T0000000000000007038  Send archive log to remote instance failed, switch all ep to SUSPEND status success!
2025-09-10 17:38:11.453 [INFO] database P0000006938 T0000000000000007169  utsk_cmd_add, cmd info: cmd=217, dseq=1757474178, name_in=, begin_lsn=-1!
2025-09-10 17:38:11.454 [INFO] database P0000006938 T0000000000000007169  utsk_set_global_dw_stat, begin, msg_dseq:1757474178
2025-09-10 17:38:11.454 [INFO] database P0000006938 T0000000000000007169  set g_dw_stat from NONE to DW_FAILOVER success, g_dw_recover_stop is 0
2025-09-10 17:38:11.454 [INFO] database P0000006938 T0000000000000007169  utsk_set_global_dw_stat, finished, msg_dseq:1757474178, set code:0
2025-09-10 17:38:12.454 [INFO] database P0000006938 T0000000000000007169  utsk_cmd_add, cmd info: cmd=214, dseq=1757474179, name_in=, begin_lsn=-1!
2025-09-10 17:38:12.454 [INFO] database P0000006938 T0000000000000007169  Change DW02 arch status from VALID to INVALID, arch_type[REALTIME]
2025-09-10 17:38:12.454 [INFO] database P0000006938 T0000000000000007169  utsk_cmd_exec->utsk_set_arch_fail_invalid, dseq[1757474179], code[0] sys_status:SUSPEND!
2025-09-10 17:38:12.455 [INFO] database P0000006938 T0000000000000007169  utsk_cmd_add, received sql exec cmd:1, dseq:1757474180, sql:ALTER DATABASE OPEN FORCE

  • 查看守护进程日志

    在备库守护进程关闭后，监听不到备库DW02的守护进程监听端口

    2025-09-10 17:37:19.881 [INFO] dmwatcher P0000007152 T0000000000000004382  dw2_group_tcp_recv_thread, receive tcp msg failed, close tcp port, vio:4, mid:-1, from_flag:0, from_name:DW02, dw_closed:0, ip:xxx.xx.110.187, port:5436, errno:0, error:Failure occurs in data_recv_inet_once, code(0) len(8128, 0), code:-6007
2025-09-10 17:37:19.881 [WARNING] dmwatcher P0000007152 T0000000000000004382  Tcp port from dmwatcher vio(4) close, ip:xxx.xx.110.187, port:5436, mid(-1), is confirm(FALSE), from name:DW02
2025-09-10 17:37:19.881 [INFO] dmwatcher P0000007152 T0000000000000004382  dw2_tcp_port_close_low, reset report_err to TRUE, ip:xxx.xx.110.187, port:5436!
2025-09-10 17:37:19.881 [INFO] dmwatcher P0000007152 T0000000000000004381  dw2_tcp_port_close_low, reset report_err to TRUE, ip:xxx.xx.110.187, port:5436!
2025-09-10 17:37:19.967 [ERROR] dmwatcher P0000007152 T0000000000000007158  Can't connect to DM server on 'xxx.xx.110.187' port(5436) errno(111)
2025-09-10 17:37:20.443 [INFO] dmwatcher P0000007152 T0000000000000007156  没有收到远程守护进程(DW02)消息，原状态为(OPEN)，距上次接收消息时间间隔(1)s, 设置远程>守护进程为ERROR状态

  • 启动备库守护进程以及数据库

    [dmdba@dmdbtest2:/home/dmdba]$ 
[dmdba@dmdbtest2:/home/dmdba]$ /home/dmdba/dmdbms/bin/DmServiceDMS start
Stopping DmServiceDMS: 
                                                           [ OK ]
[dmdba@dmdbtest2:/home/dmdba]$ /home/dmdba/dmdbms/bin/DmWatcherServiceDMW start
Stopping DmWatcherServiceDMW: 
                                                           [ OK ]

  • 查看主库守护进程日志

    启动DW02备库守护进程以及数据库服务后，DW01守护进程对备库状态进行检查，并依次修改远程实例的模式、状态、守护进程状态以及归档状态

    2025-09-11 10:01:26.813 [INFO] dmwatcher P0000007152 T0000000000000014455  远程实例的模式、状态或者归档状态发生变化，原状态是：
2025-09-11 10:01:26.813 [INFO] dmwatcher P0000007152 T0000000000000014455  Instance: 守护进程状态(ERROR) 实例状态(OK) 实例名(DW02) 模式(STANDBY) 实例状态(OPEN) 归档状态(UNKNOWN) POCNT(5) FLSN(664691) CLSN(664691) SLSN(664691) SSLSN(664691)
2025-09-11 10:01:26.813 [INFO] dmwatcher P0000007152 T0000000000000014455  远程实例的模式、状态或者归档状态发生变化，新状态是：
2025-09-11 10:01:26.989 [INFO] dmwatcher P0000007152 T0000000000000014455  ohis_inst_info_copy_low, inst(DW02) apply info changed, old info[p_db_magic:0, n_apply_ep:0], new info to set[p_db_magic:1394758080, n_apply_ep:1]!
2025-09-11 10:01:26.989 [INFO] dmwatcher P0000007152 T0000000000000014455  远程实例的模式、状态或者归档状态发生变化，原状态是：
2025-09-11 10:01:26.989 [INFO] dmwatcher P0000007152 T0000000000000014455  Instance: 守护进程状态(STARTUP) 实例状态(ERROR) 实例名(DW02) 模式(UNKNOWN) 实例状态(SHUTDOWN) 归档状态(UNKNOWN) POCNT(0) FLSN(0) CLSN(0) SLSN(0) SSLSN(0)
2025-09-11 10:01:27.040 [INFO] dmwatcher P0000007152 T0000000000000014455  远程实例的模式、状态或者归档状态发生变化，新状态是：
2025-09-11 10:01:27.040 [INFO] dmwatcher P0000007152 T0000000000000014455  Instance: 守护进程状态(UNIFY EP) 实例状态(OK) 实例名(DW02) 模式(STANDBY) 实例状态(MOUNT) 归档状态(UNKNOWN) POCNT(5) FLSN(664691) CLSN(664691) SLSN(664691) SSLSN(664691)
2025-09-11 10:01:27.040 [INFO] dmwatcher P0000007152 T0000000000000014455  远程实例的模式、状态或者归档状态发生变化，原状态是：
2025-09-11 10:01:27.040 [INFO] dmwatcher P0000007152 T0000000000000014455  Instance: 守护进程状态(UNIFY EP) 实例状态(OK) 实例名(DW02) 模式(STANDBY) 实例状态(MOUNT) 归档状态(UNKNOWN) POCNT(5) FLSN(664691) CLSN(664691) SLSN(664691) SSLSN(664691)
2025-09-11 10:01:27.040 [INFO] dmwatcher P0000007152 T0000000000000014455  远程实例的模式、状态或者归档状态发生变化，新状态是：
2025-09-11 10:01:27.040 [INFO] dmwatcher P0000007152 T0000000000000014455  Instance: 守护进程状态(STARTUP) 实例状态(OK) 实例名(DW02) 模式(STANDBY) 实例状态(OPEN) 归档状态(UNKNOWN) POCNT(5) FLSN(664691) CLSN(664691) SLSN(664691) SSLSN(664691)
2025-09-11 10:01:27.042 [INFO] dmwatcher P0000007152 T0000000000000014455  远程实例的模式、状态或者归档状态发生变化，原状态是：
2025-09-11 10:01:27.042 [INFO] dmwatcher P0000007152 T0000000000000014455  Instance: 守护进程状态(STARTUP) 实例状态(OK) 实例名(DW02) 模式(STANDBY) 实例状态(OPEN) 归档状态(UNKNOWN) POCNT(5) FLSN(664691) CLSN(664691) SLSN(664691) SSLSN(664691)

    在修改对应状态时，ohis_check_can_recover线程会一直去对比主备库的RLOG_PKG 包序号数组，并在主库归档日志中查找对应的的lsn做对比，当在主库上找不到需要的归档日志则将备库从恢复列表中删除，此时需要手动重建备库

    2025-09-11 10:01:27.096 [INFO] dmwatcher P0000007152 T0000000000000007218  [ohis_check_can_recover, p_iname:DW01, n_p_apply=0, p_apply_db_magic=1394758080, p_apply_seqno_arr=[3716, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], p_apply_lsn_arr=[685201, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]
2025-09-11 10:01:27.096 [INFO] dmwatcher P0000007152 T0000000000000007218  [ohis_check_can_recover, s_iname:DW02, n_s_apply=1, s_apply_db_magic=1394758080, s_apply_seqno_arr=[3650, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], s_apply_lsn_arr=[664691, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]

2025-09-11 10:01:27.101 [INFO] dmwatcher P0000007152 T0000000000000007156  [主库上未找到被恢复备库(DW02)需要的归档日志,code=601]
2025-09-11 10:01:27.101 [INFO] dmwatcher P0000007152 T0000000000000007156  dw2_set_recover_info, instance:DW02, recover flag:FALSE, from monitor:FALSE, last_recv_time:1757556087, last_recv_tc:674898, recover retry time:1800
2025-09-11 10:01:27.101 [INFO] dmwatcher P0000007152 T0000000000000007156  将实例(DW02)从恢复列表中删除
2025-09-11 10:01:28.096 [INFO] dmwatcher P0000007152 T0000000000000007156  不存在可恢复备库
2025-09-11 10:01:28.096 [INFO] dmwatcher P0000007152 T0000000000000007156  dw2_clear_ep_cmd_info_low, clear ep(DW01) cmd info, and reset curr_ep to NULL.
2025-09-11 10:01:28.096 [INFO] dmwatcher P0000007152 T0000000000000007156  设置GDW1守护进程子状态为SUB_STATE_CLEAR状态
2025-09-11 10:01:28.096 [INFO] dmwatcher P0000007152 T0000000000000007156  Clear all ep dw_stat value!
2025-09-11 10:01:28.101 [INFO] dmwatcher P0000007152 T0000000000000007156  dw2_notify_set_dw_stat, dseq = 1757474184, from_dw_stat: DW_RECOVERY, to_dw_stat: NONE

  • 查看此时备库日志发送信息

    此处获取信息为此处获取信息为lsn号472221-473480redo日志包发送失败，且备库归档状态失效

    show arch send info GDW1.DW02

#----DMSERVER(DW01) MAL/ARCH STATUS TO (DW02)----#
MPP_FLAG               = FALSE
ARCH TYPE              = REALTIME
MAL STATUS             = CONNECTED
ARCH STATUS            = INVALID

#----DMSERVER(DW01) LAST ARCH SEND INFO TO (DW02)----#
SEND TYPE              = FOR REALTIME SEND
SEND START TIME        = 2025-09-10 17:38:08
SEND END TIME          = 2025-09-10 17:38:08
SEND TIME USED         = 4(us)
SEND START LSN         = 664692
SEND END LSN           = 664702
SEND LOG LEN           = 4100(bytes)
SEND PTX COUNT         = 11
SEND CODE              = -6010
SEND DESC INFO         = [realtime arch] mal send to site(DW02, mal_seqno:1) failed, begin lsn:664692, end lsn:664702, code:-6010

    从发送信息中能看出，归档状态异常，发送错误号为-6010表示连接丢失

    SQL>  select * from SYS.V$ERR_INFO a where a.code='-6010';

行号     CODE        ERRINFO     
---------- ----------- ------------
1          -6010       连接丢失

已用时间: 2.749(毫秒). 执行号:86301.

4.3 DW01主库增量备份

进入DW01主库基于DW02备库已经应用的lsn（为集群状态中备库的FLSN号或者为send start lsn-1，此处为46691）进行增量备份

注： 在DW01主库进行归档切换，不切换会报错

SQL> checkpoint(100);
DMSQL 过程已成功完成
已用时间: 44.662(毫秒). 执行号:15330.
SQL> alter database archivelog current;
操作已执行
已用时间: 74.845(毫秒). 执行号:0.
SQL> backup database increment from lsn 664691 backupset '/db/dmbak/bfcq_test';
操作已执行
已用时间: 00:00:05.902. 执行号:1707.

4.4 DW02备库恢复备份

通过dmrman工具进行脱机恢复

  • 备份集拷贝至备份库

    scp /db/dmbak/bfcq_test DW02:/db/dmbak

  • 停机恢复增量备份

    • 停止备库

      [dmdba@dmdbtest2:/home/dmdba]$ /home/dmdba/dmdbms/bin/DmWatcherServiceDMW stop
Stopping DmWatcherServiceDMW: 
                                                           [ OK ]
[dmdba@dmdbtest2:/home/dmdba]$ 
[dmdba@dmdbtest2:/home/dmdba]$ /home/dmdba/dmdbms/bin/DmServiceDMS stop
Stopping DmServiceDMS: 
                                                           [ OK ]

    • 恢复备库

      merge database '/db/dmdata/DAMENG/dm.ini' from backupset '/db/dmbak/bfcq_test';
recover database '/db/dmdata/DAMENG/dm.ini' from backupset '/db/dmbak/bfcq_test';
recover database '/db/dmdata/DAMENG/dm.ini' update db_magic;

RMAN> merge database '/db/dmdata/DAMENG/dm.ini' from backupset '/db/dmbak/bfcq_test';
merge database '/db/dmdata/DAMENG/dm.ini' from backupset '/dmbak/bfcq_test';
[Percent:100.00%][Speed:0.00M/s][Cost:00:00:02][Remaining:00:00:00]                                 
merge successfully.
time used: 00:00:02.704
RMAN> 
RMAN> 
RMAN> 
RMAN> recover database '/db/dmdata/DAMENG/dm.ini' from backupset '/dmbak/bfcq_test';
recover database '/db/dmdata/DAMENG/dm.ini' from backupset '/dmbak/bfcq_test';
[Percent:100.00%][Speed:0.00PKG/s][Cost:00:00:00][Remaining:00:00:00]                               
recover successfully!
time used: 00:00:04.228
RMAN> recover database '/db/dmdata/DAMENG/dm.ini' update db_magic;
recover database '/db/dmdata/DAMENG/dm.ini' update db_magic;
recover successfully!
time used: 00:00:01.229

4.5 相关信息检查

  • 启动备库以及守护进程

    [dmdba@dmdbtest2:/home/dmdba]$ 
[dmdba@dmdbtest2:/home/dmdba]$ /home/dmdba/dmdbms/bin/DmServiceDMS start
Stopping DmServiceDMS: 
                                                           [ OK ]
[dmdba@dmdbtest2:/home/dmdba]$ /home/dmdba/dmdbms/bin/DmWatcherServiceDMW start
Stopping DmWatcherServiceDMW: 
                                                           [ OK ]

  • 查看主库守护进程日志

    备库守护进程及数据库启动后守护进程的子线程ohis_check_can_recover会进行lsn对比，此处不一致为在备份过程以及备份集传输过程中主库lsn发生变化，所以恢复后的备库lsn与主库lsn有细微差距

    2025-09-11 14:09:29.981 [INFO] dmwatcher P0000007152 T0000000000000007156  Send tcp msg to local ep DW01, hpc_seqno:0, code:0
2025-09-11 14:09:29.981 [INFO] dmwatcher P0000007152 T0000000000000007156  [ohis_check_can_recover, p_iname:DW01, n_p_apply=0, p_apply_db_magic=1394758080, p_apply_seqno_arr=[3745, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], p_apply_lsn_arr=[685364, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]
2025-09-11 14:09:29.981 [INFO] dmwatcher P0000007152 T0000000000000007156  [ohis_check_can_recover, s_iname:DW02, n_s_apply=1, s_apply_db_magic=1394758080, s_apply_seqno_arr=[3738, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], s_apply_lsn_arr=[685321, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]

    当lsn不一致后，会从主库发送归档日志，归档发送完毕并在备库应用完成后，恢复主备同步链路，备库归档状态恢复为VALID，并将DW02从从恢复列表中删除；

    在确认主库归档存在后，主库守护进程状态变化为open–>recover

    备库应用完新发送归档日志后，主库守护进程状态变化为recover–>open

    2025-09-11 14:09:30.135 [INFO] dmwatcher P0000007152 T0000000000000007156  转入suspend状态后，再次发送归档日志
2025-09-11 14:09:30.135 [INFO] dmwatcher P0000007152 T0000000000000007156  dw2_rarch_send to DW02[seqno: 0], dseq = 1757474231
2025-09-11 14:09:30.135 [INFO] dmwatcher P0000007152 T0000000000000007156  Send tcp msg to local ep DW01, hpc_seqno:0, code:0
2025-09-11 14:09:30.135 [INFO] dmwatcher P0000007152 T0000000000000007156  设置GDW1守护进程子状态为WAIT_SEND_ALL_ARCH状态
2025-09-11 14:09:30.135 [INFO] dmwatcher P0000007152 T0000000000000007156  dw2_group_get_curr_ep_retcode, ep(DW01) cmd_ret:cmd=210, dseq=1757474231, code=100
2025-09-11 14:09:30.161 [INFO] dmwatcher P0000007152 T0000000000000007156  dw2_group_get_curr_ep_retcode, ep(DW01) cmd_ret:cmd=210, dseq=1757474231, code=0
2025-09-11 14:09:30.161 [INFO] dmwatcher P0000007152 T0000000000000007156  发送归档完毕，设置实例(DW02)归档有效
2025-09-11 14:09:30.161 [INFO] dmwatcher P0000007152 T0000000000000007156  dw2_notify_chg_arch_status, dseq = 1757474232, rstat = 0
2025-09-11 14:09:30.161 [INFO] dmwatcher P0000007152 T0000000000000007156  Send tcp msg to local ep DW01, hpc_seqno:0, code:0
2025-09-11 14:09:30.161 [INFO] dmwatcher P0000007152 T0000000000000007156  设置GDW1守护进程子状态为WAIT_SET_ARCH状态
2025-09-11 14:09:30.161 [INFO] dmwatcher P0000007152 T0000000000000007156  dw2_group_get_curr_ep_retcode, ep(DW01) cmd_ret:cmd=100, dseq=1757474232, code=100
2025-09-11 14:09:30.163 [INFO] dmwatcher P0000007152 T0000000000000007170  实例(DW02)归档状态发生变化：INVALID --> VALID
2025-09-11 14:09:30.163 [INFO] dmwatcher P0000007152 T0000000000000007156  dw2_group_get_curr_ep_retcode, ep(DW01) cmd_ret:cmd=100, dseq=1757474232, code=0
2025-09-11 14:09:30.163 [INFO] dmwatcher P0000007152 T0000000000000007156  dw2_clear_ep_cmd_info_with_recv_inst_low, clear ep(DW01) cmd info, and reset curr_ep to NULL.
2025-09-11 14:09:30.163 [INFO] dmwatcher P0000007152 T0000000000000007156  设置实例(DW02)归档有效成功，通知实例(DW01)OPEN
2025-09-11 14:09:30.163 [INFO] dmwatcher P0000007152 T0000000000000007156  dw2_notify_sql_exec, dseq = 1757474233, sql: ALTER DATABASE OPEN FORCE
2025-09-11 14:09:30.163 [INFO] dmwatcher P0000007152 T0000000000000007156  Send tcp msg to local ep DW01, hpc_seqno:0, code:0
2025-09-11 14:09:30.163 [INFO] dmwatcher P0000007152 T0000000000000007156  设置GDW1守护进程子状态为WAIT_TO_OPEN状态
2025-09-11 14:09:30.163 [INFO] dmwatcher P0000007152 T0000000000000007156  dw2_group_get_curr_ep_retcode, ep(DW01) cmd_ret:cmd=1, dseq=1757474233, code=100
2025-09-11 14:09:30.163 [INFO] dmwatcher P0000007152 T0000000000000007156  dw2_group_get_curr_ep_retcode, ep(DW01) cmd_ret:cmd=1, dseq=1757474233, code=0
2025-09-11 14:09:30.163 [INFO] dmwatcher P0000007152 T0000000000000007156  dw2_clear_ep_cmd_info_with_recv_inst_low, clear ep(DW01) cmd info, and reset curr_ep to NULL.
2025-09-11 14:09:30.163 [INFO] dmwatcher P0000007152 T0000000000000007156  dw2_set_recover_info, instance:DW02, recover flag:TRUE, from monitor:FALSE, last_recv_time:1757570970, last_recv_tc:689781, recover retry time:60
2025-09-11 14:09:30.163 [INFO] dmwatcher P0000007152 T0000000000000007156  本地守护进程为RECOVERY状态，本机实例为PRIMARY & OPEN，实例(DW02)故障恢复完成
2025-09-11 14:09:30.163 [INFO] dmwatcher P0000007152 T0000000000000007156  将实例(DW02)从恢复列表中删除
2025-09-11 14:09:30.744 [INFO] dmwatcher P0000007152 T0000000000000007156  不存在可恢复备库
2025-09-11 14:09:30.744 [INFO] dmwatcher P0000007152 T0000000000000007156  设置GDW1守护进程子状态为SUB_STATE_CLEAR状态
2025-09-11 14:09:30.744 [INFO] dmwatcher P0000007152 T0000000000000007156  Clear all ep dw_stat value!

  • 查看集群状态

    RSTAT已经恢复为VAILA且FLSN和CLSN主备均已经保持一致，即同步状态恢复正常

    image202509111427018kvym8m.png

  • 查看实时归档信息

    主库最DW01最近发送到备库DW02的归档信息

    #----DMSERVER(DW01) MAL/ARCH STATUS TO (DW02)----#
MPP_FLAG               = FALSE
ARCH TYPE              = REALTIME
MAL STATUS             = CONNECTED
ARCH STATUS            = VALID

#----DMSERVER(DW01) LAST ARCH SEND INFO TO (DW02)----#
SEND TYPE              = FOR REALTIME SEND
SEND START TIME        = 2025-09-11 14:18:56
SEND END TIME          = 2025-09-11 14:18:56
SEND TIME USED         = 1489(us)
SEND START LSN         = 685381
SEND END LSN           = 685381
SEND LOG LEN           = 4100(bytes)
SEND PTX COUNT         = 1
SEND CODE              = 0
SEND DESC INFO         = send arch to site(DW02) success, begin lsn:685381, end lsn:685381

  • 查看备库数据

    主备同步已经恢复，主库数据正常同步至备库

    SQL> SELECT 
  STATUS AS 状态,
  COUNT(*) AS 数据量,
  MIN(CREATE_TIME) AS 最早创建时间,
  MAX(UPDATE_TIME) AS 最近更新时间
FROM CLUSTER_RECOVERY_TEST
GROUP BY STATUS;2   3   4   5   6   7   

行号     状态 数据量            最早创建时间         最近更新时间
---------- ------ -------------------- -------------------------- ------------------
1          A      9900                 2025-09-10 17:38:47.039096 NULL
2          D      100                  2025-09-10 17:38:47.041879 NULL

已用时间: 4.222(毫秒). 执行号:1.

5.结论

以上就是当我们DW集群备库故障且主库归档日志被误删除或者硬件原因导致归档日志丢失导致主库归档不连续，以致于主备同步链路失效备库无法加入集群时的快速恢复备库的方法；当数据量比较小时可以考虑全量备份重做备库，数据量较大时，在主库进行基于备库已经apply的lsn号的增量备份在备库进行恢复为最快的方法

‍

‍

上一篇：
达梦数据库 DCA考试心得
评论
后发表回复

作者

文章

阅读量

获赞

扫一扫
联系客服
logo
logo
logo
鄂Copyright © 达梦国产化应用创新实验室
达梦在线服务平台社区用户管理规范

鄂ICP备18017926号-1