perf(log): 日志表按天分区,只保留 10 天
callback_logs / api_outbound_logs / request_logs 只增不减,DELETE 只标记删除、 磁盘不还。改为按 TO_DAYS(created_at) 做天分区,用 DROP PARTITION 物理删除。 - 迁移 20261107080000:三张日志表主键改为 (id, created_at)(MySQL 要求分区键 包含在主键里),分区布局 p_old + 最近 10 天 + 未来 2 天 + pmax - 新增 logRetention service:补未来分区(REORGANIZE pmax)→ DROP 过期分区; 表若未分区则退化为分批 DELETE + OPTIMIZE 兜底 - 新增每日任务 purgeApiLogs(03:30):clean_request_logs 改为 disable, 避免两条清理逻辑抢同一张表 - 保留天数默认 10 天,可用 LOG_RETENTION_DAYS 覆盖 - 文档补分区口径、为什么要 DROP PARTITION、上线注意 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
parent
1d29e03ba2
commit
6facd9c6f3
@ -1009,4 +1009,43 @@ WMS 按本次需求实现为**按店铺分组、组内共用一份 `params`**(
|
||||
生产 `wms_order_express` 当前约 79.5 万行 / ibd 14.6GB,生产 MySQL 5.7.44
|
||||
(`ADD COLUMN` 需重建表,`/www` 剩余 24GB)。开发库已执行并通过校验,生产尚未执行。
|
||||
|
||||
wms/
|
||||
## 回调解可查 + 日志按天分区只留 10 天(2026-11-07)
|
||||
|
||||
### 1. 背景
|
||||
|
||||
`callback_logs`(外部系统 → OMS `POST /openapi`)一直在写,但**没有查询入口**,
|
||||
WMS 侧排查「回调到底有没有到、报了什么」只能翻服务器日志。
|
||||
同时 `callback_logs` / `api_outbound_logs` / `request_logs` 只增不减,
|
||||
`DELETE` 只标记删除、`ibd` 不还盘,硬盘只会涨。
|
||||
|
||||
### 2. 本次改动
|
||||
|
||||
| 项 | 说明 |
|
||||
|---|---|
|
||||
| 查询接口 | `GET /api/callback-logs`([callbackLog.js](backend/app/controller/callbackLog.js)):按主体隔离,支持 `warehouse_id` / `warehouse_api_id` / `method` / `app_key` / `is_success` / `ip` / `keyword`(匹配 message、error)/ `start_time` / `end_time` + `page`/`pageSize` |
|
||||
| 前端 | 「外部接口 → 回调日志」tab([Index.vue](frontend/src/views/ExternalApi/Index.vue)),可在列表里直接看请求/返回报文、HTTP 状态、业务码、耗时 |
|
||||
| 分区迁移 | `20261107080000-partition-log-tables-by-day.js`:三张日志表主键改为 `(id, created_at)`,按 `TO_DAYS(created_at)` 做天分区 |
|
||||
| 保留任务 | [logRetention.js](backend/app/service/logRetention.js) + [purgeApiLogs.js](backend/app/schedule/purgeApiLogs.js),每天 03:30 跑:补未来分区 → `DROP PARTITION` 删过期分区 |
|
||||
| 旧任务 | `clear_request_logs` 已 `disable`,避免两条清理逻辑抢同一张表 |
|
||||
|
||||
### 3. 分区口径
|
||||
|
||||
- 分区名 `pYYYYMMDD` 表示「`<` 次日 0 点」,即 `p20261107` 装的是 11-07 当天的数据;
|
||||
- `p_old` 装保留期之前的历史数据,**首次运行会直接 DROP**(老分区表初始化时用它兜住存量);
|
||||
- `pmax` 是 `MAXVALUE` 兜底分区,每天用 `REORGANIZE PARTITION pmax` 拆出新的日分区;
|
||||
- 保留期默认 10 天(今天 + 前 9 天),可用环境变量 `LOG_RETENTION_DAYS` 调整;
|
||||
迁移与服务读同一个变量,改的时候两边一致(服务侧每次运行都读最新值,无需改迁移)。
|
||||
|
||||
### 4. 为什么用 DROP PARTITION
|
||||
|
||||
`DROP PARTITION` 是元数据操作 + 直接删 `ibd` 段,**毫秒级完成并立刻还盘**;
|
||||
`DELETE FROM ... WHERE created_at < ?` 在大表上既慢又要等 purge 线程才还盘,
|
||||
`OPTIMIZE TABLE` 还会重建整表(锁表 + 双倍空间)。
|
||||
因此只要表是分区的就走 `DROP PARTITION`,只有出现「表被还原成非分区」的情况才退化为分批 `DELETE` + `OPTIMIZE` 兜底。
|
||||
|
||||
### 5. 上线注意
|
||||
|
||||
1. 迁移 `20261107080000` 会改主键(`(id)` → `(id, created_at)`)并重建表分区,
|
||||
**日志表大的话要选低峰执行**(重建期间该表不可写);
|
||||
2. 只保留 10 天意味着**历史日志会被物理删除**,需要留存的话先备份/导出;
|
||||
3. 任务失败不会重试,失败原因写在 `logs/oms/oms-error.log` 的 `[log-retention]` 前缀里。
|
||||
|
||||
@ -1,13 +1,17 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* 定时清理请求日志:每日凌晨 1 点执行,删除超过保留天数(默认 60 天)的记录
|
||||
* 【已停用】请求日志清理:改由 `schedule/purgeApiLogs.js` 按天分区回收。
|
||||
*
|
||||
* 原来这里用 `DELETE WHERE created_at < 阈值`,只标记删除、磁盘不还;
|
||||
* 现在 request_logs 已按天分区,`DROP PARTITION` 才是真正的物理删除(立刻释放空间),
|
||||
* 且保留天数统一收敛到 `LOG_RETENTION_DAYS`(默认 10 天)。
|
||||
*/
|
||||
module.exports = {
|
||||
schedule: {
|
||||
cron: '0 0 1 * * *',
|
||||
type: 'worker',
|
||||
disable: false,
|
||||
disable: true,
|
||||
},
|
||||
|
||||
/**
|
||||
|
||||
40
backend/app/schedule/purgeApiLogs.js
Normal file
40
backend/app/schedule/purgeApiLogs.js
Normal file
@ -0,0 +1,40 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* 每日清理日志表(callback_logs / api_outbound_logs / request_logs)。
|
||||
*
|
||||
* 保留天数默认 10 天(`LOG_RETENTION_DAYS` 可覆盖)。表按天分区,
|
||||
* 清理方式是 `DROP PARTITION`(瞬时物理删除、立刻释放磁盘),不是 DELETE:
|
||||
* DELETE 只标记删除,磁盘空间不会还给系统。
|
||||
*
|
||||
* 顺带补建未来 2 天的分区,避免跨天后新数据落到 pmax 分区导致不能按天回收。
|
||||
* 若某张表还没分区(例如老库未跑迁移),退化为「分批 DELETE + OPTIMIZE TABLE」。
|
||||
*/
|
||||
let running = false;
|
||||
|
||||
module.exports = {
|
||||
schedule: {
|
||||
// 每天 03:30 执行(低峰)
|
||||
cron: '0 30 3 * * *',
|
||||
type: 'worker',
|
||||
disable: false,
|
||||
},
|
||||
|
||||
/**
|
||||
* 执行清理
|
||||
* @param {Egg.Context} ctx egg context
|
||||
* @return {Promise<Array<Object>>} 各表清理结果
|
||||
*/
|
||||
async task(ctx) {
|
||||
if (running) {
|
||||
ctx.logger.info('[log-retention] 上一轮仍在执行,跳过本次');
|
||||
return { skipped: true };
|
||||
}
|
||||
running = true;
|
||||
try {
|
||||
return await ctx.service.logRetention.runAll();
|
||||
} finally {
|
||||
running = false;
|
||||
}
|
||||
},
|
||||
};
|
||||
236
backend/app/service/logRetention.js
Normal file
236
backend/app/service/logRetention.js
Normal file
@ -0,0 +1,236 @@
|
||||
'use strict';
|
||||
|
||||
const Service = require('egg').Service;
|
||||
|
||||
/**
|
||||
* 日志表保留策略(按天分区 + 每日滚动清理)
|
||||
*
|
||||
* 背景:日志表长期增长,MySQL 的 `DELETE` 只标记删除、磁盘不还;
|
||||
* 按天分区后 `ALTER TABLE ... DROP PARTITION` 是瞬时物理删除,立刻释放空间。
|
||||
*
|
||||
* 约定(见迁移 20261107080000-partition-log-tables-by-day.js):
|
||||
* - 分区键 `TO_DAYS(created_at)`,分区名 `pYYYYMMDD` 表示「< 次日 0 点」;
|
||||
* - `p_old` 装保留期之前的历史数据,直接 DROP;
|
||||
* - `pmax` 是兜底分区,每天用 `REORGANIZE PARTITION pmax` 补出新的日分区。
|
||||
*
|
||||
* 保留天数默认 10 天,可用 `LOG_RETENTION_DAYS` 覆盖。
|
||||
*/
|
||||
|
||||
/** 需要滚动清理的日志表 */
|
||||
const LOG_TABLES = [ 'callback_logs', 'api_outbound_logs', 'request_logs' ];
|
||||
|
||||
/** 预建未来分区天数 */
|
||||
const AHEAD_DAYS = 2;
|
||||
|
||||
/** 非分区表的兜底清理:每批删除行数 */
|
||||
const PURGE_BATCH = 5000;
|
||||
|
||||
/** 分区名:pYYYYMMDD / p_old / pmax */
|
||||
const DAY_PARTITION_RE = /^p(\d{4})(\d{2})(\d{2})$/;
|
||||
|
||||
/**
|
||||
* 日期格式(YYYY-MM-DD)
|
||||
* @param {Date} date 日期
|
||||
* @return {string} 形如 2026-11-07
|
||||
*/
|
||||
function fmtDate(date) {
|
||||
const pad = value => String(value).padStart(2, '0');
|
||||
return `${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* 分区名里的日期格式(YYYYMMDD)
|
||||
* @param {Date} date 日期
|
||||
* @return {string} 形如 20261107
|
||||
*/
|
||||
function fmtCompact(date) {
|
||||
return fmtDate(date).replace(/-/g, '');
|
||||
}
|
||||
|
||||
/**
|
||||
* 当天 0 点
|
||||
* @param {Date} date 日期
|
||||
* @return {Date} 0 点
|
||||
*/
|
||||
function startOfDay(date) {
|
||||
const copy = new Date(date);
|
||||
copy.setHours(0, 0, 0, 0);
|
||||
return copy;
|
||||
}
|
||||
|
||||
/**
|
||||
* 加减天数
|
||||
* @param {Date} date 日期
|
||||
* @param {number} days 天数(可为负)
|
||||
* @return {Date} 结果
|
||||
*/
|
||||
function addDays(date, days) {
|
||||
const copy = new Date(date);
|
||||
copy.setDate(copy.getDate() + days);
|
||||
return copy;
|
||||
}
|
||||
|
||||
class LogRetentionService extends Service {
|
||||
/** 保留天数(默认 10 天) */
|
||||
get keepDays() {
|
||||
return Math.max(1, Number(process.env.LOG_RETENTION_DAYS) || 10);
|
||||
}
|
||||
|
||||
/** 需要处理的日志表 */
|
||||
get tables() {
|
||||
return LOG_TABLES;
|
||||
}
|
||||
|
||||
/**
|
||||
* 列出某张表的分区
|
||||
* @param {string} table 表名
|
||||
* @return {Promise<Array<{name: string, rows: number}>>} 分区清单(按顺序)
|
||||
*/
|
||||
async listPartitions(table) {
|
||||
const rows = await this.ctx.model.query(
|
||||
`SELECT PARTITION_NAME AS name, TABLE_ROWS AS row_count
|
||||
FROM information_schema.PARTITIONS
|
||||
WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = ? AND PARTITION_NAME IS NOT NULL
|
||||
ORDER BY PARTITION_ORDINAL_POSITION`,
|
||||
{ replacements: [ table ], type: this.ctx.model.QueryTypes.SELECT }
|
||||
);
|
||||
return rows.map(row => ({ name: String(row.name), rows: Number(row.row_count) || 0 }));
|
||||
}
|
||||
|
||||
/**
|
||||
* 补建未来分区(把 pmax 拆出 pYYYYMMDD)
|
||||
* @param {string} table 表名
|
||||
* @param {string[]} existing 现有分区名
|
||||
* @return {Promise<string[]>} 本次新建的分区名
|
||||
*/
|
||||
async ensureFuturePartitions(table, existing) {
|
||||
const has = new Set(existing);
|
||||
const created = [];
|
||||
const today = startOfDay(new Date());
|
||||
for (let i = 0; i <= AHEAD_DAYS; i += 1) {
|
||||
const day = addDays(today, i);
|
||||
const name = `p${fmtCompact(day)}`;
|
||||
if (has.has(name)) continue;
|
||||
// 已有更晚的日分区时跳过(避免上界冲突)
|
||||
const laterExists = Array.from(has).some(item => {
|
||||
const matched = DAY_PARTITION_RE.exec(item);
|
||||
if (!matched) return false;
|
||||
return `${matched[1]}${matched[2]}${matched[3]}` > fmtCompact(day);
|
||||
});
|
||||
if (laterExists) continue;
|
||||
const upper = fmtDate(addDays(day, 1));
|
||||
await this.ctx.model.query(
|
||||
`ALTER TABLE \`${table}\` REORGANIZE PARTITION pmax INTO (
|
||||
PARTITION ${name} VALUES LESS THAN (TO_DAYS('${upper}')),
|
||||
PARTITION pmax VALUES LESS THAN MAXVALUE
|
||||
)`
|
||||
);
|
||||
has.add(name);
|
||||
created.push(name);
|
||||
}
|
||||
return created;
|
||||
}
|
||||
|
||||
/**
|
||||
* 删除过期分区(瞬时物理删除,释放磁盘)
|
||||
* @param {string} table 表名
|
||||
* @param {string[]} partitions 现有分区名
|
||||
* @return {Promise<{dropped: string[], rows: number}>} 删除结果
|
||||
*/
|
||||
async dropExpiredPartitions(table, partitions) {
|
||||
const today = startOfDay(new Date());
|
||||
// 保留期第一天:早于它的日分区全部删除
|
||||
const cutoff = addDays(today, -(this.keepDays - 1));
|
||||
const cutoffCompact = fmtCompact(cutoff);
|
||||
const dropped = [];
|
||||
let rows = 0;
|
||||
|
||||
const listRows = await this.listPartitions(table);
|
||||
const rowsByName = new Map(listRows.map(item => [ item.name, item.rows ]));
|
||||
|
||||
for (const name of partitions) {
|
||||
let expired = false;
|
||||
if (name === 'p_old') {
|
||||
expired = true;
|
||||
} else {
|
||||
const matched = DAY_PARTITION_RE.exec(name);
|
||||
if (matched) {
|
||||
const compact = `${matched[1]}${matched[2]}${matched[3]}`;
|
||||
expired = compact < cutoffCompact;
|
||||
}
|
||||
}
|
||||
if (!expired) continue;
|
||||
await this.ctx.model.query(`ALTER TABLE \`${table}\` DROP PARTITION ${name}`);
|
||||
dropped.push(name);
|
||||
rows += rowsByName.get(name) || 0;
|
||||
}
|
||||
return { dropped, rows };
|
||||
}
|
||||
|
||||
/**
|
||||
* 非分区表兜底清理:分批删除过期行后 OPTIMIZE 回收空间
|
||||
* @param {string} table 表名
|
||||
* @return {Promise<number>} 删除行数
|
||||
*/
|
||||
async purgeFlatTable(table) {
|
||||
const cutoff = startOfDay(addDays(new Date(), -(this.keepDays - 1)));
|
||||
let deleted = 0;
|
||||
for (;;) {
|
||||
const [ result ] = await this.ctx.model.query(
|
||||
`DELETE FROM \`${table}\` WHERE created_at < ? ORDER BY id LIMIT ${PURGE_BATCH}`,
|
||||
{ replacements: [ cutoff ] }
|
||||
);
|
||||
const affected = Number(result && result.affectedRows) || 0;
|
||||
deleted += affected;
|
||||
if (affected < PURGE_BATCH) break;
|
||||
}
|
||||
// DELETE 不还盘,OPTIMIZE 才能真正释放(表大时会重建表,放在每日低峰执行)
|
||||
await this.ctx.model.query(`OPTIMIZE TABLE \`${table}\``);
|
||||
return deleted;
|
||||
}
|
||||
|
||||
/**
|
||||
* 处理单张日志表:补分区 → 删过期分区(未分区则走兜底删除)
|
||||
* @param {string} table 表名
|
||||
* @return {Promise<Object>} 执行结果
|
||||
*/
|
||||
async runTable(table) {
|
||||
const { ctx } = this;
|
||||
try {
|
||||
const partitions = await this.listPartitions(table);
|
||||
if (!partitions.length) {
|
||||
const deleted = await this.purgeFlatTable(table);
|
||||
ctx.logger.info('[log-retention] %s 未分区,兜底删除 %d 行(保留 %d 天)', table, deleted, this.keepDays);
|
||||
return { table, partitioned: false, deleted };
|
||||
}
|
||||
const created = await this.ensureFuturePartitions(table, partitions.map(item => item.name));
|
||||
const { dropped, rows } = await this.dropExpiredPartitions(table, partitions.map(item => item.name));
|
||||
ctx.logger.info(
|
||||
'[log-retention] %s 新建分区 %s;删除分区 %s(约 %d 行,保留 %d 天)',
|
||||
table,
|
||||
created.join(',') || '-',
|
||||
dropped.join(',') || '-',
|
||||
rows,
|
||||
this.keepDays
|
||||
);
|
||||
return { table, partitioned: true, created, dropped, rows };
|
||||
} catch (err) {
|
||||
ctx.logger.error('[log-retention] %s 处理失败:%s', table, err.message);
|
||||
return { table, error: err.message };
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 处理全部日志表
|
||||
* @return {Promise<Array<Object>>} 各表结果
|
||||
*/
|
||||
async runAll() {
|
||||
const results = [];
|
||||
for (const table of this.tables) {
|
||||
results.push(await this.runTable(table));
|
||||
}
|
||||
return results;
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = LogRetentionService;
|
||||
@ -0,0 +1,111 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* 日志表按天分区(callback_logs / api_outbound_logs / request_logs)。
|
||||
*
|
||||
* 目的:日志量大时 `DELETE` 只标记删除、不还盘;按天分区后 `DROP PARTITION` 是瞬时物理删除,
|
||||
* 能立刻释放磁盘。分区键用 `TO_DAYS(created_at)`,分区名 `pYYYYMMDD`(表示「< 次日 0 点」)。
|
||||
*
|
||||
* MySQL 要求分区键必须包含在主键里,因此主键从 `(id)` 改为 `(id, created_at)`(id 仍自增)。
|
||||
* 分区布局:`p_old`(保留期外的历史数据,由每日任务直接 DROP)+ 最近 10 天 + 未来 2 天 + `pmax`。
|
||||
*/
|
||||
|
||||
/** 需要分区(=按天滚动的日志表) */
|
||||
const TABLES = [ 'callback_logs', 'api_outbound_logs', 'request_logs' ];
|
||||
|
||||
/** 保留天数(与 app/service/logRetention.js 保持一致,可用 LOG_RETENTION_DAYS 覆盖) */
|
||||
const KEEP_DAYS = Math.max(1, Number(process.env.LOG_RETENTION_DAYS) || 10);
|
||||
|
||||
/** 预建未来分区天数(留出余量,避免跨天时写入落到 pmax 后不好回收) */
|
||||
const AHEAD_DAYS = 2;
|
||||
|
||||
/**
|
||||
* 分区名里的日期格式(YYYYMMDD)
|
||||
* @param {Date} date 日期
|
||||
* @return {string} 形如 20261107
|
||||
*/
|
||||
function fmtCompact(date) {
|
||||
const pad = value => String(value).padStart(2, '0');
|
||||
return `${date.getFullYear()}${pad(date.getMonth() + 1)}${pad(date.getDate())}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* 日期格式(YYYY-MM-DD,用于 SQL 字面量)
|
||||
* @param {Date} date 日期
|
||||
* @return {string} 形如 2026-11-07
|
||||
*/
|
||||
function fmtDate(date) {
|
||||
const pad = value => String(value).padStart(2, '0');
|
||||
return `${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* 当天 0 点
|
||||
* @param {Date} date 日期
|
||||
* @return {Date} 0 点
|
||||
*/
|
||||
function startOfDay(date) {
|
||||
const copy = new Date(date);
|
||||
copy.setHours(0, 0, 0, 0);
|
||||
return copy;
|
||||
}
|
||||
|
||||
/**
|
||||
* 加减天数
|
||||
* @param {Date} date 基准日期
|
||||
* @param {number} days 天数(可为负)
|
||||
* @return {Date} 结果
|
||||
*/
|
||||
function addDays(date, days) {
|
||||
const copy = new Date(date);
|
||||
copy.setDate(copy.getDate() + days);
|
||||
return copy;
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
async up(queryInterface) {
|
||||
const sequelize = queryInterface.sequelize;
|
||||
const today = startOfDay(new Date());
|
||||
const firstKeep = addDays(today, -(KEEP_DAYS - 1));
|
||||
|
||||
for (const table of TABLES) {
|
||||
const [ partitioned ] = await sequelize.query(
|
||||
`SELECT COUNT(*) AS cnt FROM information_schema.PARTITIONS
|
||||
WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = '${table}' AND PARTITION_NAME IS NOT NULL`,
|
||||
{ type: sequelize.QueryTypes.SELECT }
|
||||
);
|
||||
if (Number(partitioned && partitioned.cnt) > 0) {
|
||||
// 已分区(重复执行/手工改过)则不重复处理
|
||||
continue;
|
||||
}
|
||||
|
||||
// 分区键必须包含在主键里
|
||||
await sequelize.query(`ALTER TABLE \`${table}\` DROP PRIMARY KEY, ADD PRIMARY KEY (id, created_at)`);
|
||||
|
||||
const parts = [ `PARTITION p_old VALUES LESS THAN (TO_DAYS('${fmtDate(firstKeep)}'))` ];
|
||||
for (let i = 0; i <= KEEP_DAYS - 1 + AHEAD_DAYS; i += 1) {
|
||||
const day = addDays(firstKeep, i);
|
||||
parts.push(`PARTITION p${fmtCompact(day)} VALUES LESS THAN (TO_DAYS('${fmtDate(addDays(day, 1))}'))`);
|
||||
}
|
||||
parts.push('PARTITION pmax VALUES LESS THAN MAXVALUE');
|
||||
|
||||
await sequelize.query(
|
||||
`ALTER TABLE \`${table}\` PARTITION BY RANGE (TO_DAYS(created_at)) (${parts.join(', ')})`
|
||||
);
|
||||
}
|
||||
},
|
||||
|
||||
async down(queryInterface) {
|
||||
const sequelize = queryInterface.sequelize;
|
||||
for (const table of TABLES) {
|
||||
const [ partitioned ] = await sequelize.query(
|
||||
`SELECT COUNT(*) AS cnt FROM information_schema.PARTITIONS
|
||||
WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = '${table}' AND PARTITION_NAME IS NOT NULL`,
|
||||
{ type: sequelize.QueryTypes.SELECT }
|
||||
);
|
||||
if (!Number(partitioned && partitioned.cnt)) continue;
|
||||
await sequelize.query(`ALTER TABLE \`${table}\` REMOVE PARTITIONING`);
|
||||
await sequelize.query(`ALTER TABLE \`${table}\` DROP PRIMARY KEY, ADD PRIMARY KEY (id)`);
|
||||
}
|
||||
},
|
||||
};
|
||||
Loading…
x
Reference in New Issue
Block a user