注册
记录一次dmfldr测试
专栏/技术分享/ 文章详情 /

记录一次dmfldr测试

dm-9527 2026/09/30 174 0 0
摘要

记录一次dmfldr测试

一、需求:

数据库中根据日期

每日生成一个表,将本地今日生成的文件通过dmfldr导入到今日的数据表中。

二、模拟流程:

每天生成一个以时间命名的.txt文件。通过dm_test.sh 脚本传入时间参数,并导入对应的.txt文件至数据库里边
数据文件                        目标表
─────────────────              ──────────────
test20260830.txt       →       TABLE_20260830
test20260831.txt       →       TABLE_20260831
test20260901.txt       →       TABLE_20260901

* 文件名中带日期
* 表名中也带日期
* 表名前缀固定
* 日期动态变化
* Shell 自动从文件名提取日期
* 自动拼接目标表名
* 再调用 dmfldr

文件要求:

第一天文件名:test20260830.txt
目标表:HR.TABLE_TEMP_20260830
第二天文件名:test20260831.txt
目标表:HR.TABLE_TEMP_20260831

前提条件:

生产环境已经提前创建对应数据表:
TABLE_TEMP_20260830
TABLE_TEMP_20260831
TABLE_TEMP_20260901

三、测试步骤

1.创建目录

mkdir -p /tmp/dmfldr_date_test
cd /tmp/dmfldr_date_test

2.创建按日期划分的测试表

# 20260830
CREATE TABLE HR.TABLE_TEMP_20260830
(
    DATA_DATE VARCHAR(8),
    END_DATE  VARCHAR(8),
    CITY_CODE VARCHAR(20),
    NUM1      INT,
    NUM2      INT,
    AMOUNT    DECIMAL(20,2)
);

# 20260831
CREATE TABLE HR.TABLE_TEMP_20260831
(
    DATA_DATE VARCHAR(8),
    END_DATE  VARCHAR(8),
    CITY_CODE VARCHAR(20),
    NUM1      INT,
    NUM2      INT,
    AMOUNT    DECIMAL(20,2)
);

3.验证

SELECT TABLE_NAME FROM DBA_TABLES WHERE OWNER='HR' and TABLE_NAME LIKE 'TABLE_TEMP%';

行号        TABLE_NAME                     
---------- -------------------------------
1          TABLE_TEMP_20260830
2          TABLE_TEMP_20260831

4.准备两个日期文件

/tmp/dmfldr_date_test/
├── test20260830.txt
└── test20260831.txt

cat > test20260830.txt <<EOF
20260830,20260905,UCR,421,12,1209.0
20260830,20260912,QDK,609,2,506.0
20260830,20260903,QDK,684,0,0.0
20260830,20260903,XAY,0,1,419.5
20260830,20260902,JNK,118,5,88.5
EOF

cat > test20260831.txt <<EOF
20260831,20260906,UCR,500,10,1500.0
20260831,20260913,QDK,700,3,600.0
20260831,20260904,XAY,100,2,300.5
EOF

5.脚本:根据文件名自动匹配日期表

vi dm_test.sh

#!/bin/bash
# ========================================
# 参数检查
# ========================================
if [ $# -ne 1 ]; then
    echo "Usage: $0 <data_file>"
    exit 1
fi
# ========================================
# 数据文件
# ========================================
DATA_FILE=$1
if [ ! -f "$DATA_FILE" ]; then
    echo "ERROR: 文件不存在: $DATA_FILE"
    exit 1
fi
# ========================================
# 获取文件名
# ========================================
FILE_NAME=$(basename "$DATA_FILE")
echo "DATA FILE: $FILE_NAME"
# ========================================
# 提取8位日期
# ========================================
DATE_STR=$(echo "$FILE_NAME" | grep -oE '[0-9]{8}' | head -1)
if [ -z "$DATE_STR" ]; then
    echo "ERROR: 无法从文件名提取日期"
    exit 1
fi
echo "DATE: $DATE_STR"
# ========================================
# 拼接目标表名
# ========================================
TABLE_NAME="TABLE_TEMP_${DATE_STR}"
echo "TARGET TABLE: HR.$TABLE_NAME"
# ========================================
# 动态生成 CONTROL 文件
# ========================================
CTL_FILE="/tmp/dmfldr_date_test/${TABLE_NAME}.ctl"
cat > "$CTL_FILE" <<EOF
LOAD DATA
INFILE '$DATA_FILE'
INTO TABLE HR.$TABLE_NAME
FIELDS TERMINATED BY ','
(
    DATA_DATE,
    END_DATE,
    CITY_CODE,
    NUM1,
    NUM2,
    AMOUNT
)
EOF
# ========================================
# 调用 dmfldr
# ========================================
dmfldr \
USERID=HR/Dameng123@192.168.64.30:5237 \
MODE=\'IN\' \
CONTROL=\'$CTL_FILE\'
RET=$?
# ========================================
# 删除临时 ctl
# ========================================
rm -f "$CTL_FILE"
# ========================================
# 返回结果
# ========================================
if [ $RET -eq 0 ]; then
    echo "SUCCESS!"
    echo "$DATA_FILE -> HR.$TABLE_NAME"
else
    echo "FAILED!"
fi
exit $RET

6.赋权

chmod +x dm_test.sh

7.测试第一个日期文件

执行:
./dm_test.sh test20260830.txt
脚本自动处理:
test20260830.txt
       ↓
提取日期
       ↓
20260830
       ↓
拼接表名
       ↓
TABLE_TEMP_20260830
       ↓
dmfldr 导入

验证:
SELECT * FROM HR.TABLE_TEMP_20260830;
行号        DATA_DATE END_DATE CITY_CODE NUM1        NUM2        AMOUNT
---------- --------- -------- --------- ----------- -----       ------ 
1          20260830  20260905 UCR       421         12          1209
2          20260830  20260912 QDK       609         2           506
3          20260830  20260903 QDK       684         0           0
4          20260830  20260903 XAY       0           1           419.5
5          20260830  20260902 JNK       118         5           88.5

8.测试第二个日期文件

执行:
./dm_test.sh test20260831.txt

自动:
test20260831.txt
       ↓
20260831
       ↓
TABLE_TEMP_20260831
       ↓
dmfldr 导入

验证:
SELECT * FROM HR.TABLE_TEMP_20260831;
行号     DATA_DATE END_DATE CITY_CODE NUM1        NUM2        AMOUNT
---------- --------- -------- --------- ----------- ----------- 
1          20260831  20260906 UCR       500         10          1500
2          20260831  20260913 QDK       700         3           600
3          20260831  20260904 XAY       100         2           300.5

已用时间: 8.078(毫秒). 执行号:3305.
评论
后发表回复

作者

文章

阅读量

获赞

扫一扫
联系客服