refactor(log): 接口流水改用 Egg 日志落盘,整表下线 wms_system_log
背景:接口流水原本逐条写 wms_system_log 单表(生产 687,030 行、.ibd 28.8 GB、DATA_FREE 仅 5 MB),
删行无法把空间还给操作系统;且每个 needLog 请求都要多一次 INSERT。改为经 Egg 自定义 logger 平铺落盘,
整表 DROP 回收磁盘(生产实测释放 28.8 GB,/www 可用 20G→49G);日志目录同时由 /root/logs/wms
迁到 /www/wwwlogs/wms,不再占用系统盘。
新增 app/lib/api_log_sink.js:
- 一行一条 `HH:mm:ss {JSON}`:前缀是本地时区时分秒,只给人眼定位(tail/grep 的时间锚点),与行内
created_at(UTC ISO 8601)是同一时刻;机器解析请用 created_at。
- 时间前缀刻意放在代码里而不是 config 的 formatter —— 生产 config 不参与 deploy、由人工维护,
放代码才能随发布自动生效(此前已因「代码上线、配套配置没上」出现过接口日志静默丢失)。
- 对象/数组一律 JSON.stringify,避免调用方漏 stringify 时落盘成 "[object Object]"。
- customLogger.apiLog 缺失时每个 worker 显式告警一次:egg 的 getLogger(name) 对未声明的 name 返回
null 而不抛错,不处理就会「既不落盘、也无任何报错」。
新增迁移 database/migrations/20260927090000-offline-wms-system-log.js:
- up 用 information_schema 探测后 DROP TABLE(幂等);该表对 wms_store 是出边外键,无需关外键检查。
- down 用原生 DDL 重建:queryInterface.createTable 在 MySQL 下默认字符集是 utf8mb3,且不会生成原始
DDL 里的 created_at ON UPDATE CURRENT_TIMESTAMP,只有原生 DDL 才能忠实还原。
- 已在开发库跑通 up → down → up:重建后 14 列、主键、store_id 索引、外键、utf8mb4 均与下线前一致。
下线代码:删除 app/model/system_log.js、app/service/systems/system_log.js(60 天保留期清理服务)、
app/schedule/system_log_cleanup.js(每日 03:00 清理任务);移除 config.systemLog(writeToDb 回滚开关)
与 config.systemLogCleanup;api_log_sink 不再保留任何入库分支。
文档与脚本:AI_METHOD_ROUTING.md 第 6 节清单移除已下线的 systems/system_log.js(文件数 220→219);
script/method-output-regression.cjs 的回放样本来源由 `SELECT ... FROM wms_system_log` 改为读
wms-api.log(含按天轮转文件),并改为先整体 parse、失败再截到第一个 `{` 重试,兼容带/不带时间前缀
两种行(在真实 EggLogger 上验证:FileBufferTransport 每 1000ms flush,需等过 1s 才能读到)。
注:按日切割与 31 天清理由 egg 内置 egg-logrotator 自动完成(walkLoggerFile 覆盖 customLogger),
无需额外配 cron。
This commit is contained in:
parent
aca6867d57
commit
d129103d0b
@ -321,9 +321,10 @@ rg "class .*Service|module.exports" app/service/qimenMethods
|
||||
<!-- SERVICE_METHOD_INVENTORY:BEGIN -->
|
||||
自动生成时间: 2026-09-24T13:58:51.474Z
|
||||
(2026-09-25 手动补录:systemMethods/index.js 新增 uploadClientErrorLog、新增 systems/client_error_log.js;生成脚本 tmp-gen-service-method-inventory.cjs 不在仓库,待补齐脚本后重扫)
|
||||
(2026-09-27 手动下线:删除 systems/system_log.js(findCleanupBoundaryId / deleteSystemLogsBatch / cleanupExpiredSystemLogs)—— wms_system_log 表已随迁移 20260927090000-offline-wms-system-log.js 整表下线,文件数 220 -> 219)
|
||||
|
||||
扫描范围: app/service/**/*.js
|
||||
文件数: 220
|
||||
文件数: 219
|
||||
|
||||
- app/service/api.js (1)
|
||||
- callMethod
|
||||
@ -741,8 +742,6 @@ rg "class .*Service|module.exports" app/service/qimenMethods
|
||||
- createError, getCurrentStoreId, getCurrentAccountId, toPositiveNumber, normalizePage, normalizeStatusFlag, normalizeAuthStatus, normalizeNonNegativeInteger, normalizeInteger, normalizeText, normalizeIntegerRange, normalizeEmail, getPolicyConfigFieldList, getPolicyConfigFieldMetaList, normalizePolicyConfig, getPolicyConfigCacheKey, hasPolicyConfigColumn, loadPolicyConfigByContractId, savePolicyConfigByContractId, getPolicyConfigDefaults, mergePolicyConfigWithDefaults, getEffectivePolicyConfigByContractId, ensureWarehouseSubject, getNoticeService, sendPrintQuantityAdjustNotice, resolveRelationId, normalizeDate, normalizeSignAction, normalizeTempCode, getTempCodeCacheKey, createTempCode, pickRemaskField, resolveRemaskValue, buildCurrentRemaskPatch, requireCurrentRemaskPatch, saveTempSignPayload, getTempSignPayload, clearTempSignPayload, buildTempSignView, getCurrentSubject, getCounterparty, buildListWhereByStoreId, buildListWhere, getRelationIncludes, getQimenContractCacheKey, clearQimenContractCache, getApiInterfaceType, getQimenInterfaceType, resolveQimenCustomerIdByAppKey, getQimenErpNameMapByAppKeys, serializeQimenConfig, getQimenConfigMap, getSingleQimenConfig, resolvePairIds, serializeRelation, findScopedRelation, buildListForStoreWhere, listForStore, exportListForStore, listForWarehouse, list, sign, trimContract, confirmSign, rename, adjustPrintQuantity, queryPrintQuantityLogs, exportPrintQuantityLogs, getPolicyConfigFields, getPolicyConfig, savePolicyConfig, listApiConfig, saveApiConfig, deleteQimenConfig, unbind, getContractConfigCacheKey, invalidateContractConfigCache, getContractConfig, openAccount, createUser
|
||||
- app/service/systems/store.js (22)
|
||||
- normalizeSubjectConfigInput, validateStoreData, checkCompanyNameExists, findByAccountId, findAllEnabled, findAllWarehouses, getStoreName, getStoreRemaskName, createStore, createWarehouse, joinStore, joinWarehouse, ensureEmployeeRoleBinding, assignAdminRoleToCreator, generateJoinToken, sendNotification, sendNotificationToAdmins, getStoreRoles, auditApply, loadStores, loadWarehouses, bindCode
|
||||
- app/service/systems/system_log.js (3)
|
||||
- findCleanupBoundaryId, deleteSystemLogsBatch, cleanupExpiredSystemLogs
|
||||
- app/service/systems/upload.js (19)
|
||||
- getTencentCloudConfig, getCosConfig, initCOS, getUploadParams, hash_hmac, generatePolicyAndSignature, resolveUploadMaxSizeBytes, validateFileInfo, generateFileName, parseSizeToBytes, formatFileSize, getFrontendConfig, uploadFile, uploadBuffer, validateFile, uploadSuccessNotify, getMimeType, createFileReference, handleUploadCallback
|
||||
- app/service/systems/video_record_request.js (15)
|
||||
@ -819,6 +818,10 @@ rg "class .*Service|module.exports" app/service/qimenMethods
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
190
app/lib/api_log_sink.js
Normal file
190
app/lib/api_log_sink.js
Normal file
@ -0,0 +1,190 @@
|
||||
// app/lib/api_log_sink.js
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* 接口日志落盘器:接口流水经 Egg 日志写出(`wms_system_log` 表已下线删除)。
|
||||
*
|
||||
* 设计要点:
|
||||
* 1. 只使用 Egg 自带 logger(`app.getLogger('apiLog')`),不自行操作 fs ——
|
||||
* egg-logger 的 FileTransport 以 O_APPEND 追加写、多 worker 共写同一文件、
|
||||
* 自动创建目录并按天轮转,避免自建写流带来的「轮转后句柄失效 / ENOSPC 崩进程 /
|
||||
* 每请求建流导致 fd 泄漏 / 退出前未 flush」等问题。
|
||||
* 2. 一行一条记录,形如 `HH:mm:ss {JSON}`:
|
||||
* 前缀是**本地时区的时分秒**,只给人眼定位用(tail/grep 时有个时间锚点),
|
||||
* 机器解析请用行内的 `created_at`(ISO 8601 UTC,无时区歧义,与前缀为同一时刻);
|
||||
* 前缀之后是沿用旧 `wms_system_log` 表结构的平铺 JSON(不嵌套,首字符是 `{`),
|
||||
* 可直接用 grep 搜索;用 jq 时先剥前缀:`jq -R 'sub("^[0-9:]+ "; "") | fromjson'`。
|
||||
* 3. 写入失败绝不影响接口返回:异常由调用方兜底,只记 app logger。
|
||||
*/
|
||||
|
||||
/** body / result 截断长度:沿用旧 wms_system_log 入库时的长文本上限(65535) */
|
||||
const TEXT_LIMIT = 65535;
|
||||
|
||||
/** customLogger.apiLog 缺失的告警只打一次,避免每请求刷屏 */
|
||||
let warnedApiLogMissing = false;
|
||||
|
||||
/**
|
||||
* 归一化为字符串,必要时截断。
|
||||
* @param {*} value 原始值。
|
||||
* @param {number} [limit] 截断长度(正数才生效),默认不截断。
|
||||
* @return {string} 字符串结果。
|
||||
*/
|
||||
function toText(value, limit) {
|
||||
if (value === undefined || value === null) {
|
||||
return '';
|
||||
}
|
||||
// 对象/数组一律 JSON 序列化:调用方若漏了 JSON.stringify(例如 controller 直接把
|
||||
// 响应对象塞进 logData.result),String() 会得到 "[object Object]" 而丢掉内容。
|
||||
let text;
|
||||
if (typeof value === 'string') {
|
||||
text = value;
|
||||
} else if (typeof value === 'object') {
|
||||
try {
|
||||
text = JSON.stringify(value);
|
||||
} catch (e) {
|
||||
text = String(value);
|
||||
}
|
||||
} else {
|
||||
text = String(value);
|
||||
}
|
||||
if (Number(limit) > 0 && text.length > Number(limit)) {
|
||||
return text.substring(0, Number(limit));
|
||||
}
|
||||
return text;
|
||||
}
|
||||
|
||||
/**
|
||||
* 构造一行平铺记录。
|
||||
*
|
||||
* 字段与顺序沿用旧 `wms_system_log` 入库时的入参:
|
||||
* method / v / url / headers / auth / created_at / body / result / use_time / store_id / device。
|
||||
* `created_at` 统一写 ISO 8601(UTC,带 Z):与原入库是同一时刻,但没有时区歧义。
|
||||
*
|
||||
* @param {object} logData 原始日志数据。
|
||||
* @param {object} [options] 选项。
|
||||
* @param {number} [options.startTime] 请求开始时间戳(ms),用于计算 use_time。
|
||||
* @param {number} [options.useTime] 直接指定耗时(ms),优先于 startTime。
|
||||
* @param {string} [options.device] 设备标识。
|
||||
* @param {Date} [options.createdAt] 记录时间,默认当前时间。
|
||||
* @return {object} 平铺记录对象。
|
||||
*/
|
||||
function buildRecord(logData, options = {}) {
|
||||
const source = logData && typeof logData === 'object' ? logData : {};
|
||||
const createdAt = options.createdAt instanceof Date ? options.createdAt : new Date();
|
||||
|
||||
let useTime = 0;
|
||||
if (options.useTime !== undefined) {
|
||||
useTime = Number(options.useTime);
|
||||
} else if (Number(options.startTime) > 0) {
|
||||
useTime = Date.now() - Number(options.startTime);
|
||||
}
|
||||
if (!Number.isFinite(useTime) || useTime < 0) {
|
||||
useTime = 0;
|
||||
}
|
||||
|
||||
return {
|
||||
method: toText(source.method),
|
||||
v: toText(source.v),
|
||||
url: toText(source.url),
|
||||
headers: toText(source.headers),
|
||||
auth: toText(source.auth),
|
||||
created_at: createdAt.toISOString(),
|
||||
body: toText(source.body, TEXT_LIMIT),
|
||||
result: toText(source.result, TEXT_LIMIT),
|
||||
use_time: useTime,
|
||||
store_id: source.store_id || null,
|
||||
device: toText(options.device),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* customLogger.apiLog 缺失时的显式告警(每个 worker 只打一次)。
|
||||
*
|
||||
* egg 的 `app.getLogger(name)` 对未声明的 name 返回 null 而不抛错,
|
||||
* 若不处理就会「日志静默消失、且无任何报错」。
|
||||
* 生产 config 不参与 deploy,「代码已上线、配套配置没上」极易发生,故必须显式报出来。
|
||||
*
|
||||
* @param {Egg.Application} app Egg 应用实例。
|
||||
* @return {void}
|
||||
*/
|
||||
function warnApiLogMissing(app) {
|
||||
if (warnedApiLogMissing) {
|
||||
return;
|
||||
}
|
||||
warnedApiLogMissing = true;
|
||||
const logger = app && typeof app.getLogger === 'function' ? app.getLogger('logger') : null;
|
||||
if (logger && typeof logger.error === 'function') {
|
||||
logger.error(
|
||||
'[api_log_sink] config.customLogger.apiLog 未配置,接口日志不会落盘!' +
|
||||
'请在 config.default.js 补上 customLogger.apiLog' +
|
||||
'(生产 config 不参与 deploy,需手工同步到线上)。'
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 两位补零。
|
||||
* @param {number} value 待补零的数字。
|
||||
* @return {string} 补零后的字符串。
|
||||
*/
|
||||
function pad2(value) {
|
||||
return String(value).padStart(2, '0');
|
||||
}
|
||||
|
||||
/**
|
||||
* 生成日志行的时间前缀(本地时区 `HH:mm:ss`)。
|
||||
*
|
||||
* 整行内容本身是完整 JSON(首字符是 `{`),直接 tail/grep 时没有时间锚点,
|
||||
* 故在行首加一个本地时间的时分秒前缀。前缀只给人看:
|
||||
* 它与行内 `created_at` 取同一时刻,仅表达方式不同(本地 `HH:mm:ss` vs UTC ISO 8601)。
|
||||
* 日期无需前缀 —— 日志按天轮转为 `wms-api.log.YYYY-MM-DD`,日期由文件名提供。
|
||||
*
|
||||
* @param {Date|string} [value] 记录时间;缺省或非法时回退当前时间。
|
||||
* @return {string} `HH:mm:ss` 形式的前缀。
|
||||
*/
|
||||
function formatTimePrefix(value) {
|
||||
const parsed = value instanceof Date ? value : new Date(value || Date.now());
|
||||
const date = Number.isNaN(parsed.getTime()) ? new Date() : parsed;
|
||||
return `${pad2(date.getHours())}:${pad2(date.getMinutes())}:${pad2(date.getSeconds())}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* 写出接口日志。
|
||||
*
|
||||
* 只落 Egg 日志文件(`config.customLogger.apiLog`);wms_system_log 表已下线,
|
||||
* 不再存在任何入库分支。
|
||||
*
|
||||
* @param {Egg.Application} app Egg 应用实例。
|
||||
* @param {object} payload 负载。
|
||||
* @param {object} payload.logData 原始日志数据。
|
||||
* @param {number} [payload.startTime] 请求开始时间戳(ms)。
|
||||
* @param {number} [payload.useTime] 直接指定耗时(ms)。
|
||||
* @param {string} [payload.device] 设备标识。
|
||||
* @param {Date} [payload.createdAt] 记录时间。
|
||||
* @return {Promise<object>} 实际写出的平铺记录。
|
||||
*/
|
||||
async function write(app, payload = {}) {
|
||||
const record = buildRecord(payload.logData, {
|
||||
startTime: payload.startTime,
|
||||
useTime: payload.useTime,
|
||||
device: payload.device,
|
||||
createdAt: payload.createdAt,
|
||||
});
|
||||
|
||||
const logger = app && typeof app.getLogger === 'function' ? app.getLogger('apiLog') : null;
|
||||
if (logger) {
|
||||
// 一行一条:`HH:mm:ss {完整 JSON}`,前缀与行内 created_at 是同一时刻
|
||||
logger.info(`${formatTimePrefix(record.created_at)} ${JSON.stringify(record)}`);
|
||||
} else {
|
||||
warnApiLogMissing(app);
|
||||
}
|
||||
|
||||
return record;
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
TEXT_LIMIT,
|
||||
buildRecord,
|
||||
formatTimePrefix,
|
||||
write,
|
||||
};
|
||||
@ -1,94 +0,0 @@
|
||||
// app/model/system_log.js
|
||||
'use strict';
|
||||
|
||||
module.exports = app => {
|
||||
const { BIGINT, STRING, TEXT, DATE, INTEGER } = app.Sequelize;
|
||||
|
||||
const SystemLog = app.model.define('wms_system_log', {
|
||||
id: {
|
||||
type: BIGINT.UNSIGNED,
|
||||
primaryKey: true,
|
||||
autoIncrement: true,
|
||||
allowNull: false,
|
||||
comment: 'ID',
|
||||
},
|
||||
method: {
|
||||
type: STRING(100),
|
||||
allowNull: true,
|
||||
comment: '方法名',
|
||||
},
|
||||
v: {
|
||||
type: STRING(50),
|
||||
allowNull: true,
|
||||
comment: '版本',
|
||||
},
|
||||
url: {
|
||||
type: TEXT,
|
||||
allowNull: true,
|
||||
comment: '请求URL',
|
||||
},
|
||||
created_at: {
|
||||
type: DATE,
|
||||
allowNull: true,
|
||||
comment: '创建时间',
|
||||
},
|
||||
body: {
|
||||
type: TEXT('long'),
|
||||
allowNull: true,
|
||||
comment: '请求体',
|
||||
},
|
||||
result: {
|
||||
type: TEXT('long'),
|
||||
allowNull: true,
|
||||
comment: '响应体',
|
||||
},
|
||||
use_time: {
|
||||
type: INTEGER,
|
||||
allowNull: true,
|
||||
comment: '耗时',
|
||||
},
|
||||
updated_at: {
|
||||
type: DATE,
|
||||
allowNull: true,
|
||||
comment: '更新时间',
|
||||
},
|
||||
delete_at: {
|
||||
type: DATE,
|
||||
allowNull: true,
|
||||
comment: '删除时间',
|
||||
},
|
||||
headers: {
|
||||
type: TEXT,
|
||||
allowNull: true,
|
||||
comment: '请求头',
|
||||
},
|
||||
auth: {
|
||||
type: TEXT,
|
||||
allowNull: true,
|
||||
comment: '认证信息',
|
||||
},
|
||||
store_id: {
|
||||
type: BIGINT.UNSIGNED,
|
||||
allowNull: true,
|
||||
comment: '门店ID',
|
||||
},
|
||||
device: {
|
||||
type: STRING(100),
|
||||
allowNull: true,
|
||||
comment: '设备',
|
||||
},
|
||||
}, {
|
||||
tableName: 'wms_system_log',
|
||||
timestamps: false,
|
||||
underscored: false,
|
||||
comment: '系统日志表',
|
||||
indexes: [
|
||||
{
|
||||
name: 'store_id',
|
||||
fields: [ 'store_id' ],
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
return SystemLog;
|
||||
};
|
||||
@ -1,88 +0,0 @@
|
||||
// app/schedule/system_log_cleanup.js
|
||||
'use strict';
|
||||
|
||||
/** @type {number} 默认单批硬删除行数 */
|
||||
const DEFAULT_BATCH_SIZE = 2000;
|
||||
/** @type {number} 默认批间休眠最小毫秒数 */
|
||||
const DEFAULT_SLEEP_MIN_MS = 100;
|
||||
/** @type {number} 默认批间休眠最大毫秒数 */
|
||||
const DEFAULT_SLEEP_MAX_MS = 500;
|
||||
|
||||
/**
|
||||
* 系统日志超期清理定时任务。
|
||||
*
|
||||
* 默认每日凌晨 03:00:00(cron:秒 分 时 日 月 周)在单 worker 中执行一次,
|
||||
* 硬删除 wms_system_log 超过保留天数(默认 60 天)的日志以释放磁盘空间,避开 08:00–23:00 业务高峰。
|
||||
* cron 从 config.systemLogCleanup.cron 读取(config.default.js 提供默认值),
|
||||
* 本地环境可在 config.local.js 覆盖为短间隔以便快速验证,不影响生产 03:00 触发时间。
|
||||
* 保留天数由 service.systems.systemLog 读取 config.systemLogCleanup.retainDays(默认 60,非法值回退 60);
|
||||
* 由 config.systemLogCleanup 控制:enable=false 时不注册本任务;
|
||||
* 实际清理逻辑收敛在 service.systems.systemLog.cleanupExpiredSystemLogs。
|
||||
*
|
||||
* @param {Object} app Egg 应用实例。
|
||||
* @return {Object} 定时任务定义(含 schedule 与 task)。
|
||||
*/
|
||||
module.exports = app => {
|
||||
const cleanupConfig = (app.config && app.config.systemLogCleanup) || {};
|
||||
// cron 从配置读取,缺省时回退为每日凌晨 03:00(cron 格式:秒 分 时 日 月 周)
|
||||
const cleanupCron = typeof cleanupConfig.cron === 'string' && cleanupConfig.cron
|
||||
? cleanupConfig.cron
|
||||
: '0 0 3 * * *';
|
||||
|
||||
return {
|
||||
schedule: {
|
||||
cron: cleanupCron,
|
||||
type: 'worker',
|
||||
immediate: false,
|
||||
disable: cleanupConfig.enable === false,
|
||||
},
|
||||
|
||||
/**
|
||||
* 执行系统日志超期清理。
|
||||
*
|
||||
* 采用单 worker 防重入:上一轮未结束时跳过本轮;
|
||||
* 任务失败只记录错误日志,不向上抛出,避免导致 schedule 崩溃。
|
||||
*
|
||||
* @param {Object} ctx Egg 上下文。
|
||||
* @return {Promise<void>} 无返回值。
|
||||
*/
|
||||
async task(ctx) {
|
||||
const scheduleName = 'system_log_cleanup';
|
||||
|
||||
if (ctx.app._systemLogCleanupRunning) {
|
||||
ctx.logger.info(`[${scheduleName}] 本轮任务跳过,上一轮尚未结束`);
|
||||
return;
|
||||
}
|
||||
|
||||
ctx.app._systemLogCleanupRunning = true;
|
||||
const startedAt = Date.now();
|
||||
try {
|
||||
const config = ctx.app.config.systemLogCleanup || {};
|
||||
// 保留天数(retainDays)由 service 读取 config.systemLogCleanup.retainDays 自行解析,此处不再透传
|
||||
const options = {
|
||||
batchSize: Number(config.batchSize) > 0
|
||||
? Math.floor(Number(config.batchSize))
|
||||
: DEFAULT_BATCH_SIZE,
|
||||
sleepMinMs: Number(config.sleepMinMs) >= 0
|
||||
? Math.floor(Number(config.sleepMinMs))
|
||||
: DEFAULT_SLEEP_MIN_MS,
|
||||
sleepMaxMs: Number(config.sleepMaxMs) >= 0
|
||||
? Math.floor(Number(config.sleepMaxMs))
|
||||
: DEFAULT_SLEEP_MAX_MS,
|
||||
};
|
||||
|
||||
const summary = await ctx.service.systems.systemLog.cleanupExpiredSystemLogs(options);
|
||||
|
||||
ctx.logger.info(
|
||||
`[${scheduleName}] 本轮完成 skipped=${summary.skipped} boundary_id=${summary.boundary_id}`
|
||||
+ ` batches=${summary.batch_count} deleted=${summary.deleted_rows}`
|
||||
+ ` duration=${Date.now() - startedAt}ms`
|
||||
);
|
||||
} catch (error) {
|
||||
ctx.logger.error(`[${scheduleName}] 执行失败: ${error.message}`);
|
||||
} finally {
|
||||
ctx.app._systemLogCleanupRunning = false;
|
||||
}
|
||||
},
|
||||
};
|
||||
};
|
||||
@ -2,6 +2,8 @@
|
||||
|
||||
const Service = require('egg').Service;
|
||||
|
||||
const apiLogSink = require('../lib/api_log_sink');
|
||||
|
||||
class pdaService extends Service {
|
||||
async callMethod(className, actionName, requestData, authHeader) {
|
||||
try {
|
||||
@ -62,10 +64,13 @@ class pdaService extends Service {
|
||||
}
|
||||
|
||||
/**
|
||||
* 保存接口日志到系统日志模型
|
||||
* 保存接口日志。
|
||||
*
|
||||
* 经 Egg logger(apiLog)平铺写出到日志文件;wms_system_log 表已下线,不再有任何入库路径。
|
||||
*
|
||||
* @param {Object} logData 日志数据
|
||||
* @param {number} startTime 开始时间戳
|
||||
* @return {Promise<Object|undefined>} 写入结果
|
||||
* @return {Promise<Object|undefined>} 写出的平铺记录
|
||||
*/
|
||||
async saveLog(logData, startTime) {
|
||||
const {
|
||||
@ -73,38 +78,14 @@ class pdaService extends Service {
|
||||
} = this;
|
||||
|
||||
try {
|
||||
// 限制日志内容长度
|
||||
if (logData.body && logData.body.length > 65535) {
|
||||
logData.body = logData.body.substring(0, 65535);
|
||||
}
|
||||
|
||||
if (logData.result && logData.result.length > 65535) {
|
||||
logData.result = logData.result.substring(0, 65535);
|
||||
}
|
||||
|
||||
logData.use_time = Date.now() - startTime;
|
||||
|
||||
// 使用 SystemLog model 保存日志到数据库
|
||||
if (!ctx.model || !ctx.model.SystemLog) {
|
||||
throw new Error('SystemLog model 不存在,无法写入系统日志');
|
||||
}
|
||||
const result = await ctx.model.SystemLog.create({
|
||||
method: logData.method || '',
|
||||
v: logData.v || '',
|
||||
url: logData.url || '',
|
||||
headers: logData.headers || '',
|
||||
auth: logData.auth || '',
|
||||
created_at: new Date(),
|
||||
body: logData.body || '',
|
||||
result: logData.result || '',
|
||||
use_time: logData.use_time || 0,
|
||||
store_id: logData.store_id || null,
|
||||
return await apiLogSink.write(ctx.app, {
|
||||
logData,
|
||||
startTime,
|
||||
device: 'pda',
|
||||
});
|
||||
return result;
|
||||
} catch (error) {
|
||||
ctx.logger.error('保存日志失败: ' + error.message);
|
||||
// 可以选择记录到文件日志作为备用
|
||||
// 兜底:至少把原始数据落到 app 日志,避免整条丢失
|
||||
ctx.logger.info('日志数据: ', JSON.stringify(logData));
|
||||
}
|
||||
}
|
||||
|
||||
@ -2,6 +2,8 @@
|
||||
|
||||
const Service = require('egg').Service;
|
||||
|
||||
const apiLogSink = require('../lib/api_log_sink');
|
||||
|
||||
const ALLOWED_PLATFORMS = new Set([ 'web', 'pda', 'app', 'wx', 'miniprogram', 'other', 'workerstation', 'workstation' ]);
|
||||
// project: all=全部, warehouse=仓库, store=货主, tms=物流
|
||||
const ALLOWED_PROJECTS = new Set([ 'all', 'warehouse', 'store', 'tms', 'wms', 'scm', 'other' ]);
|
||||
@ -159,10 +161,13 @@ class systemService extends Service {
|
||||
}
|
||||
|
||||
/**
|
||||
* 保存接口日志到系统日志模型
|
||||
* 保存接口日志。
|
||||
*
|
||||
* 经 Egg logger(apiLog)平铺写出到日志文件;wms_system_log 表已下线,不再有任何入库路径。
|
||||
*
|
||||
* @param {Object} logData 日志数据
|
||||
* @param {number} startTime 开始时间戳
|
||||
* @return {Promise<Object|undefined>} 写入结果
|
||||
* @return {Promise<Object|undefined>} 写出的平铺记录
|
||||
*/
|
||||
async saveLog(logData, startTime) {
|
||||
const {
|
||||
@ -170,38 +175,14 @@ class systemService extends Service {
|
||||
} = this;
|
||||
|
||||
try {
|
||||
// 限制日志内容长度
|
||||
if (logData.body && logData.body.length > 65535) {
|
||||
logData.body = logData.body.substring(0, 65535);
|
||||
}
|
||||
|
||||
if (logData.result && logData.result.length > 65535) {
|
||||
logData.result = logData.result.substring(0, 65535);
|
||||
}
|
||||
|
||||
logData.use_time = Date.now() - startTime;
|
||||
|
||||
// 使用 SystemLog model 保存日志到数据库
|
||||
if (!ctx.model || !ctx.model.SystemLog) {
|
||||
throw new Error('SystemLog model 不存在,无法写入系统日志');
|
||||
}
|
||||
const result = await ctx.model.SystemLog.create({
|
||||
method: logData.method || '',
|
||||
v: logData.v || '',
|
||||
url: logData.url || '',
|
||||
headers: logData.headers || '',
|
||||
auth: logData.auth || '',
|
||||
created_at: new Date(),
|
||||
body: logData.body || '',
|
||||
result: logData.result || '',
|
||||
use_time: logData.use_time || 0,
|
||||
store_id: logData.store_id || null,
|
||||
return await apiLogSink.write(ctx.app, {
|
||||
logData,
|
||||
startTime,
|
||||
device: ctx.header.platform,
|
||||
});
|
||||
return result;
|
||||
} catch (error) {
|
||||
ctx.logger.error('保存奇门接口日志失败: ' + error.message);
|
||||
// 可以选择记录到文件日志作为备用
|
||||
// 兜底:至少把原始数据落到 app 日志,避免整条丢失
|
||||
ctx.logger.info('奇门接口日志数据: ', JSON.stringify(logData));
|
||||
}
|
||||
}
|
||||
|
||||
@ -1,180 +0,0 @@
|
||||
// app/service/systems/system_log.js
|
||||
'use strict';
|
||||
|
||||
const MainService = require('../main');
|
||||
|
||||
/** @type {string} 系统日志表名 */
|
||||
const SYSTEM_LOG_TABLE = 'wms_system_log';
|
||||
/** @type {number} 默认日志保留天数:删除创建时间早于该天数的日志 */
|
||||
const DEFAULT_RETAIN_DAYS = 60;
|
||||
/** @type {number} 默认单批硬删除行数 */
|
||||
const DEFAULT_BATCH_SIZE = 2000;
|
||||
/** @type {number} 默认批间休眠最小毫秒数 */
|
||||
const DEFAULT_SLEEP_MIN_MS = 100;
|
||||
/** @type {number} 默认批间休眠最大毫秒数 */
|
||||
const DEFAULT_SLEEP_MAX_MS = 500;
|
||||
|
||||
/**
|
||||
* 休眠指定毫秒数。
|
||||
*
|
||||
* @param {number} ms 休眠毫秒数。
|
||||
* @return {Promise<void>} 无返回值。
|
||||
*/
|
||||
function sleep(ms) {
|
||||
return new Promise(resolve => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
/**
|
||||
* 将保留天数归一化为正整数。
|
||||
*
|
||||
* 仅接受正整数(含数字字符串),其余非法值(负数、0、小数、非数字)一律回退默认保留天数;
|
||||
* 归一化后的值直接拼入 SQL 模板的 INTERVAL 表达式,保证无 SQL 注入风险。
|
||||
*
|
||||
* @param {*} value 待归一化的保留天数。
|
||||
* @return {number} 归一化后的正整数保留天数。
|
||||
*/
|
||||
function normalizeRetainDays(value) {
|
||||
const normalized = Number(value);
|
||||
return Number.isInteger(normalized) && normalized > 0 ? normalized : DEFAULT_RETAIN_DAYS;
|
||||
}
|
||||
|
||||
/**
|
||||
* 系统日志清理服务。
|
||||
*
|
||||
* 背景:wms_system_log 单表无分表无分区,日志长期增长会占用大量磁盘空间。
|
||||
* created_at 由应用写入 new Date(),与自增主键 id 严格单调递增,且 created_at 无索引,
|
||||
* 因此按时间条件过滤会全表扫描,必须按 id 边界走主键索引删除;
|
||||
* 同时必须使用原生 SQL 硬删除,禁止 SystemLog.destroy(全局 paranoid 软删除会增大行体积且语义不符)。
|
||||
*/
|
||||
class SystemLogService extends MainService {
|
||||
/**
|
||||
* 定位删除边界 id:创建时间早于保留天数截止时间的最大 id。
|
||||
*
|
||||
* 由于 created_at 与 id 严格单调递增,id 不大于该边界的行即全部过期行,
|
||||
* 等价于按 created_at 时间边界删除且走主键索引;查不到时返回 null 表示无过期日志。
|
||||
*
|
||||
* @param {number} retainDays 日志保留天数(正整数;非法值回退默认 60)。
|
||||
* @return {Promise<number|null>} 边界 id;无过期日志时返回 null。
|
||||
*/
|
||||
async findCleanupBoundaryId(retainDays) {
|
||||
const { ctx } = this;
|
||||
const safeRetainDays = normalizeRetainDays(retainDays);
|
||||
|
||||
const rows = await ctx.model.query(
|
||||
`SELECT id FROM ${SYSTEM_LOG_TABLE}`
|
||||
+ ` WHERE created_at < DATE_SUB(NOW(), INTERVAL ${safeRetainDays} DAY)`
|
||||
+ ' ORDER BY id DESC LIMIT 1',
|
||||
{ type: ctx.model.QueryTypes.SELECT }
|
||||
);
|
||||
|
||||
const firstRow = Array.isArray(rows) ? rows[0] : null;
|
||||
const boundaryId = firstRow ? Number(firstRow.id) : null;
|
||||
return Number.isInteger(boundaryId) && boundaryId > 0 ? boundaryId : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* 单批硬删除 id 不大于边界值的系统日志(每批最多 batchSize 行)。
|
||||
*
|
||||
* @param {number} boundaryId 删除边界 id(含边界行本身)。
|
||||
* @param {number} batchSize 单批最大删除行数。
|
||||
* @return {Promise<number>} 本批实际删除行数。
|
||||
*/
|
||||
async deleteSystemLogsBatch(boundaryId, batchSize) {
|
||||
const { ctx } = this;
|
||||
const safeBoundaryId = Number(boundaryId) > 0 ? Math.floor(Number(boundaryId)) : 0;
|
||||
const safeBatchSize = Number(batchSize) > 0 ? Math.floor(Number(batchSize)) : DEFAULT_BATCH_SIZE;
|
||||
if (safeBoundaryId <= 0) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
// 不传 type:默认走 RAW 查询,mysql2 对 DELETE 返回 ResultSetHeader,
|
||||
// 取 [result, data] 结构中第一项的 affectedRows 作为本批删除行数
|
||||
const rawResult = await ctx.model.query(
|
||||
`DELETE FROM ${SYSTEM_LOG_TABLE}`
|
||||
+ ` WHERE id <= ${safeBoundaryId}`
|
||||
+ ' ORDER BY id'
|
||||
+ ` LIMIT ${safeBatchSize}`
|
||||
);
|
||||
const header = Array.isArray(rawResult) ? rawResult[0] : rawResult;
|
||||
const affectedRows = Number(header && header.affectedRows);
|
||||
return Number.isFinite(affectedRows) && affectedRows > 0 ? affectedRows : 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* 清理超过保留天数的系统日志(定时任务入口)。
|
||||
*
|
||||
* 流程:先定位边界 id,再按 id 升序分批硬删除,循环到单批受影响行数为 0 为止,
|
||||
* 批间随机休眠 100~500ms 降低数据库压力;不执行 OPTIMIZE TABLE,
|
||||
* 由 InnoDB 自行复用已删除数据页,避免重建大表占满磁盘。
|
||||
*
|
||||
* @param {Object} [options={}] 清理参数。
|
||||
* @param {number} [options.retainDays] 日志保留天数(正整数);缺省时读取
|
||||
* config.systemLogCleanup.retainDays,仍非法或未配置时回退 60。
|
||||
* @param {number} [options.batchSize=2000] 单批硬删除行数。
|
||||
* @param {number} [options.sleepMinMs=100] 批间休眠最小毫秒数。
|
||||
* @param {number} [options.sleepMaxMs=500] 批间休眠最大毫秒数。
|
||||
* @return {Promise<Object>} 清理结果摘要。
|
||||
*/
|
||||
async cleanupExpiredSystemLogs(options = {}) {
|
||||
const { ctx } = this;
|
||||
const startedAt = Date.now();
|
||||
const safeOptions = options && typeof options === 'object' ? options : {};
|
||||
|
||||
const cleanupConfig = (ctx.app.config && ctx.app.config.systemLogCleanup) || {};
|
||||
const rawRetainDays = safeOptions.retainDays !== undefined && safeOptions.retainDays !== null
|
||||
? safeOptions.retainDays
|
||||
: cleanupConfig.retainDays;
|
||||
const retainDays = normalizeRetainDays(rawRetainDays);
|
||||
const batchSize = Number(safeOptions.batchSize) > 0
|
||||
? Math.floor(Number(safeOptions.batchSize))
|
||||
: DEFAULT_BATCH_SIZE;
|
||||
const sleepMinMs = Number(safeOptions.sleepMinMs) >= 0
|
||||
? Math.floor(Number(safeOptions.sleepMinMs))
|
||||
: DEFAULT_SLEEP_MIN_MS;
|
||||
let sleepMaxMs = Number(safeOptions.sleepMaxMs) >= 0
|
||||
? Math.floor(Number(safeOptions.sleepMaxMs))
|
||||
: DEFAULT_SLEEP_MAX_MS;
|
||||
if (sleepMaxMs < sleepMinMs) {
|
||||
sleepMaxMs = sleepMinMs;
|
||||
}
|
||||
|
||||
const boundaryId = await this.findCleanupBoundaryId(retainDays);
|
||||
if (boundaryId === null) {
|
||||
ctx.logger.info(`[system-log-cleanup] 无超过 ${retainDays} 天的系统日志,跳过本次清理`);
|
||||
return {
|
||||
skipped: true,
|
||||
boundary_id: null,
|
||||
batch_count: 0,
|
||||
deleted_rows: 0,
|
||||
duration_ms: Date.now() - startedAt,
|
||||
};
|
||||
}
|
||||
|
||||
let batchCount = 0;
|
||||
let totalDeleted = 0;
|
||||
let affectedRows = 1;
|
||||
|
||||
while (affectedRows > 0) {
|
||||
if (batchCount > 0) {
|
||||
const sleepMs = Math.floor(sleepMinMs + Math.random() * (sleepMaxMs - sleepMinMs + 1));
|
||||
await sleep(sleepMs);
|
||||
}
|
||||
|
||||
affectedRows = await this.deleteSystemLogsBatch(boundaryId, batchSize);
|
||||
if (affectedRows > 0) {
|
||||
batchCount += 1;
|
||||
totalDeleted += affectedRows;
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
skipped: false,
|
||||
boundary_id: boundaryId,
|
||||
batch_count: batchCount,
|
||||
deleted_rows: totalDeleted,
|
||||
duration_ms: Date.now() - startedAt,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = SystemLogService;
|
||||
@ -17,7 +17,6 @@ module.exports = appInfo => {
|
||||
|
||||
// add your middleware config here
|
||||
config.middleware = [];
|
||||
|
||||
// add your user config here
|
||||
const userConfig = {
|
||||
// myAppName: 'egg',
|
||||
@ -122,6 +121,33 @@ module.exports = appInfo => {
|
||||
fileExtension: '.log',
|
||||
progressExtension: '.progress',
|
||||
};
|
||||
// 日志落盘目录:显式声明,避免依赖 $HOME 的隐式默认值
|
||||
// (生产 HOME=/root 时 egg 默认会落到 /root/logs/wms;此处固定下来便于查询与清理)
|
||||
config.logPaths = {
|
||||
base: process.env.WMS_LOG_DIR || (process.platform === 'win32'
|
||||
? path.join(appInfo.baseDir, 'logs')
|
||||
: '/root/logs/wms'),
|
||||
};
|
||||
// 文件日志只保留 WARN/ERROR:INFO(含 egg-sequelize 逐条 SQL 日志)不再落盘,
|
||||
// 避免 wms-web.log 每天增长约 2GB 撑爆根分区(/ 仅 40G)。
|
||||
// consoleLevel 仍为 INFO,本地开发在终端照常看到 INFO,不受影响。
|
||||
// 与线上 /www/wwwroot/qimen/v1、v2 的手工配置保持一致(生产 config 不参与 deploy,由人工维护)。
|
||||
config.logger = {
|
||||
dir: config.logPaths.base,
|
||||
level: 'WARN',
|
||||
consoleLevel: 'INFO',
|
||||
};
|
||||
// 接口流水日志(已替代 wms_system_log 表,该表已下线):一行一条,字段沿用旧表结构并平铺。
|
||||
// formatter 直接返回 message,不改写内容;整行形如 `HH:mm:ss {JSON}`
|
||||
// (行首时分秒前缀由 app/lib/api_log_sink.js 加上,行尾 eol 由 egg-logger 追加)。
|
||||
config.customLogger = {
|
||||
apiLog: {
|
||||
file: path.join(config.logPaths.base, 'wms-api.log'),
|
||||
level: 'INFO',
|
||||
consoleLevel: 'NONE',
|
||||
formatter: meta => meta.message,
|
||||
},
|
||||
};
|
||||
config.fileTypes = {
|
||||
// 图片类型
|
||||
image: 'image',
|
||||
@ -267,19 +293,6 @@ module.exports = appInfo => {
|
||||
// 单批硬删除行数(按 id 分批,走主键索引)
|
||||
batchSize: 500,
|
||||
};
|
||||
// 系统日志超期清理定时任务配置(每日凌晨 03:00 执行,硬删除 wms_system_log 超期数据释放磁盘空间)
|
||||
config.systemLogCleanup = {
|
||||
enable: true,
|
||||
// 触发 cron(秒 分 时 日 月 周):默认每日凌晨 03:00;本地环境可在 config.local.js 覆盖为短间隔
|
||||
cron: '0 0 3 * * *',
|
||||
// 日志保留天数:删除 created_at 早于该天数的日志(默认 60 天,仅接受正整数,非法值回退 60)
|
||||
retainDays: 60,
|
||||
// 单批硬删除行数(按 id 升序分批,走主键索引)
|
||||
batchSize: 2000,
|
||||
// 批间休眠毫秒数下限/上限:在区间内随机取值,降低数据库瞬时压力
|
||||
sleepMinMs: 100,
|
||||
sleepMaxMs: 500,
|
||||
};
|
||||
// 配送表(dispatch) → 历史表(history) 归档定时任务配置
|
||||
// 每日凌晨 06:00 把配送表超期订单(连同商品明细)迁入历史表,并更新订单索引表位置指针
|
||||
config.orderDeliveryArchive = {
|
||||
|
||||
106
database/migrations/20260927090000-offline-wms-system-log.js
Normal file
106
database/migrations/20260927090000-offline-wms-system-log.js
Normal file
@ -0,0 +1,106 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* 下线 wms_system_log(接口流水表)。
|
||||
*
|
||||
* 背景:接口流水已改为经 Egg 自定义 logger(`config.customLogger.apiLog`)平铺写入
|
||||
* `/root/logs/wms/wms-api.log`,应用侧不再有任何写入路径;生产库该表已膨胀到
|
||||
* 687,030 行 / DATA_LENGTH 19,357 MB(约 18.9 GB),而 DATA_FREE 仅 5 MB ——
|
||||
* 单纯按保留期删行不会把空间还给操作系统,必须整表下线才能回收磁盘。
|
||||
*
|
||||
* ⚠️ 本迁移会永久丢弃表内历史数据。如需留档,请在执行前先自行导出:
|
||||
* mysqldump -h127.0.0.1 -uwms -p wms wms_system_log | gzip > /root/wms_system_log-$(date +%F).sql.gz
|
||||
*
|
||||
* 幂等:up 用 DROP TABLE IF EXISTS;down 先探测表存在性再重建。
|
||||
*
|
||||
* 同批次一起下线的代码(见同批改动):
|
||||
* - app/model/system_log.js (已删除)
|
||||
* - app/service/systems/system_log.js (已删除,60 天保留期清理服务)
|
||||
* - app/schedule/system_log_cleanup.js (已删除,每日 03:00 清理定时任务)
|
||||
* - app/lib/api_log_sink.js (移除 writeToDb 回滚入库分支)
|
||||
* - config/config.default.js (移除 systemLog / systemLogCleanup)
|
||||
*
|
||||
* @param {import('sequelize').QueryInterface} queryInterface 查询接口
|
||||
* @param {import('sequelize')} Sequelize Sequelize 实例
|
||||
* @return {Promise<void>} 无返回值
|
||||
*/
|
||||
module.exports = {
|
||||
/**
|
||||
* 执行迁移:删除 wms_system_log 表(幂等)。
|
||||
*
|
||||
* 说明:该表对 wms_store 有外键(wms_system_log.store_id → wms_store.id),
|
||||
* 属"出边"外键,删除子表不影响父表,无需先关闭 FOREIGN_KEY_CHECKS;
|
||||
* 已确认库内没有指向 wms_system_log 的入边外键、视图或触发器。
|
||||
*
|
||||
* @param {import('sequelize').QueryInterface} queryInterface 查询接口
|
||||
* @return {Promise<void>} 无返回值
|
||||
*/
|
||||
async up(queryInterface) {
|
||||
const tableName = 'wms_system_log';
|
||||
// 用 information_schema 探测而非依赖异常:MySQL 5.7 / 8.0 的 IF EXISTS 行为一致但便于记录日志
|
||||
const [[ probe ]] = await queryInterface.sequelize.query(
|
||||
'SELECT COUNT(*) AS cnt FROM information_schema.TABLES'
|
||||
+ ' WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = :tableName',
|
||||
{ replacements: { tableName } }
|
||||
);
|
||||
if (!Number(probe && probe.cnt)) {
|
||||
return;
|
||||
}
|
||||
|
||||
const [[ size ]] = await queryInterface.sequelize.query(
|
||||
'SELECT ROUND(DATA_LENGTH / 1024 / 1024) AS data_mb, ROUND(INDEX_LENGTH / 1024 / 1024) AS idx_mb, TABLE_ROWS AS est_rows'
|
||||
+ ' FROM information_schema.TABLES WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = :tableName',
|
||||
{ replacements: { tableName } }
|
||||
);
|
||||
await queryInterface.sequelize.query(`DROP TABLE IF EXISTS \`${tableName}\``);
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(
|
||||
`[offline-wms-system-log] 已删除 ${tableName}(原估算行数=${size && size.est_rows}`
|
||||
+ ` 数据=${size && size.data_mb}MB 索引=${size && size.idx_mb}MB)`
|
||||
);
|
||||
},
|
||||
|
||||
/**
|
||||
* 回滚迁移:按原结构重建空的 wms_system_log 表(幂等,不恢复数据)。
|
||||
*
|
||||
* 直接用原生 DDL 重建,而不是 queryInterface.createTable —— 后者在 MySQL 下
|
||||
* 默认字符集是 utf8(即 utf8mb3),也不会生成原始的
|
||||
* `created_at datetime DEFAULT NULL ON UPDATE CURRENT_TIMESTAMP`;
|
||||
* 只有原生 DDL 才能让回滚后的结构与下线前完全一致。
|
||||
*
|
||||
* ⚠️ 表内数据不会恢复;需要留档请在 up 之前自行 mysqldump。
|
||||
*
|
||||
* @param {import('sequelize').QueryInterface} queryInterface 查询接口
|
||||
* @return {Promise<void>} 无返回值
|
||||
*/
|
||||
async down(queryInterface) {
|
||||
const tableName = 'wms_system_log';
|
||||
const tables = await queryInterface.showAllTables();
|
||||
if (tables.includes(tableName)) {
|
||||
return;
|
||||
}
|
||||
|
||||
await queryInterface.sequelize.query(
|
||||
`CREATE TABLE \`${tableName}\` (`
|
||||
+ ' `id` bigint UNSIGNED NOT NULL AUTO_INCREMENT,'
|
||||
+ ' `method` varchar(100) DEFAULT NULL,'
|
||||
+ ' `v` varchar(50) DEFAULT NULL,'
|
||||
+ ' `url` text,'
|
||||
+ ' `created_at` datetime DEFAULT NULL ON UPDATE CURRENT_TIMESTAMP,'
|
||||
+ ' `body` longtext,'
|
||||
+ ' `result` longtext,'
|
||||
+ ' `use_time` int DEFAULT NULL,'
|
||||
+ ' `updated_at` datetime DEFAULT NULL,'
|
||||
+ ' `delete_at` datetime DEFAULT NULL,'
|
||||
+ ' `headers` text,'
|
||||
+ ' `auth` text,'
|
||||
+ ' `store_id` bigint UNSIGNED DEFAULT NULL,'
|
||||
+ ' `device` varchar(100) DEFAULT NULL COMMENT \'设备\','
|
||||
+ ' PRIMARY KEY (`id`),'
|
||||
+ ' KEY `store_id` (`store_id`),'
|
||||
+ ' CONSTRAINT `wms_system_log_ibfk_1` FOREIGN KEY (`store_id`)'
|
||||
+ ' REFERENCES `wms_store` (`id`) ON DELETE CASCADE ON UPDATE NO ACTION'
|
||||
+ ') ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci'
|
||||
);
|
||||
},
|
||||
};
|
||||
@ -1,6 +1,5 @@
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const mysql = require('mysql2/promise');
|
||||
|
||||
const ROOT = process.cwd();
|
||||
const OUTPUT_DIR = path.join(ROOT, 'temp', 'method-output');
|
||||
@ -128,28 +127,76 @@ function pickProject(headers) {
|
||||
return project;
|
||||
}
|
||||
|
||||
async function createDbConnection(cfg) {
|
||||
const db = cfg.mysql.clients.wms;
|
||||
return mysql.createConnection({
|
||||
host: db.host,
|
||||
port: Number(db.port),
|
||||
user: db.user,
|
||||
password: db.password,
|
||||
database: db.database,
|
||||
});
|
||||
/** 接口流水日志目录:优先取环境变量,其次为本地开发目录 <repo>/logs(与 config.logPaths.base 一致) */
|
||||
function resolveApiLogDir() {
|
||||
if (process.env.WMS_LOG_DIR) return process.env.WMS_LOG_DIR;
|
||||
return path.join(ROOT, 'logs');
|
||||
}
|
||||
|
||||
async function loadRecentSystemLogCases(conn, methodDocs, limit, username) {
|
||||
const [rows] = await conn.query(`
|
||||
SELECT id, method, body, headers, auth, device, result, created_at
|
||||
FROM wms_system_log
|
||||
WHERE method IS NOT NULL
|
||||
AND method <> ''
|
||||
AND body IS NOT NULL
|
||||
AND body <> ''
|
||||
ORDER BY id DESC
|
||||
LIMIT 1200
|
||||
`);
|
||||
/**
|
||||
* 解析一行接口日志。
|
||||
*
|
||||
* 行格式为 `HH:mm:ss {JSON}`(行首时分秒前缀由 app/lib/api_log_sink.js 写入)。
|
||||
* 为兼容不带前缀的历史行,先尝试整体 parse,失败再截到第一个 `{` 重试。
|
||||
*
|
||||
* @param {string} line 原始行。
|
||||
* @return {Object|null} 解析出的记录;不是有效对象时返回 null。
|
||||
*/
|
||||
function parseApiLogLine(line) {
|
||||
const trimmed = String(line || '').trim();
|
||||
if (!trimmed) return null;
|
||||
const braceIndex = trimmed.indexOf('{');
|
||||
const candidates = braceIndex > 0 ? [ trimmed, trimmed.slice(braceIndex) ] : [ trimmed ];
|
||||
for (const text of candidates) {
|
||||
const parsed = safeJsonParse(text, null);
|
||||
if (parsed && typeof parsed === 'object' && !Array.isArray(parsed)) {
|
||||
return parsed;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* 读取 wms-api.log 及其按天轮转的历史文件,按“最近优先”返回记录。
|
||||
*
|
||||
* 背景:接口流水原写入 wms_system_log 表,该表已下线
|
||||
* (见 database/migrations/20260927090000-offline-wms-system-log.js),
|
||||
* 现由 Egg 自定义 logger 落盘为一行一条的平铺 JSON,字段与旧表一一对应,
|
||||
* 因此回放样本改为从日志文件读取;返回结构保持与旧 SQL 结果同形
|
||||
* (method / body / headers / auth / device / result / created_at,并补 id 作为来源标识)。
|
||||
*
|
||||
* @return {Array<Object>} 记录列表,最近优先;目录/文件缺失时返回空数组。
|
||||
*/
|
||||
function loadRecentApiLogRows() {
|
||||
const dir = resolveApiLogDir();
|
||||
if (!fs.existsSync(dir)) return [];
|
||||
|
||||
const files = fs.readdirSync(dir)
|
||||
.filter(name => name === 'wms-api.log' || name.startsWith('wms-api.log.'))
|
||||
.map(name => ({
|
||||
full: path.join(dir, name),
|
||||
mtime: fs.statSync(path.join(dir, name)).mtimeMs,
|
||||
}))
|
||||
.sort((a, b) => a.mtime - b.mtime);
|
||||
|
||||
const rows = [];
|
||||
for (const file of files) {
|
||||
const text = fs.readFileSync(file.full, 'utf8');
|
||||
for (const line of text.split('\n')) {
|
||||
const row = parseApiLogLine(line);
|
||||
// 日志行没有自增 id,用 created_at 充当来源标识
|
||||
if (row && row.method) {
|
||||
rows.push(Object.assign({ id: row.created_at }, row));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 日志为追加写,倒序即“最近优先”,与原 SQL 的 ORDER BY id DESC 语义一致
|
||||
return rows.reverse();
|
||||
}
|
||||
|
||||
function loadRecentApiLogCases(methodDocs, limit, username) {
|
||||
const rows = loadRecentApiLogRows();
|
||||
|
||||
const preferred = [];
|
||||
const fallback = [];
|
||||
@ -173,7 +220,7 @@ async function loadRecentSystemLogCases(conn, methodDocs, limit, username) {
|
||||
const historySuccess = typeof row.result === 'string' && row.result.includes('"status":200');
|
||||
|
||||
const item = {
|
||||
source: 'wms_system_log',
|
||||
source: 'wms-api.log',
|
||||
sourceId: row.id,
|
||||
sourceCreatedAt: row.created_at,
|
||||
method: row.method,
|
||||
@ -278,65 +325,63 @@ async function runBaseline() {
|
||||
const cfg = readLocalConfig();
|
||||
const baseUrl = buildBaseUrl(cfg);
|
||||
const methodDocs = loadSystemMethodDocs();
|
||||
const conn = await createDbConnection(cfg);
|
||||
|
||||
try {
|
||||
const selected = await loadRecentSystemLogCases(conn, methodDocs, LIMIT, TEST_USERNAME);
|
||||
if (!selected.length) {
|
||||
throw new Error('未从 wms_system_log 找到可回放的只读 method 样本');
|
||||
}
|
||||
|
||||
const login = await callSystemMethod(
|
||||
baseUrl,
|
||||
'user.login',
|
||||
{ username: TEST_USERNAME, password: TEST_PASSWORD },
|
||||
{ platform: 'web', project: 'wms' },
|
||||
''
|
||||
const selected = loadRecentApiLogCases(methodDocs, LIMIT, TEST_USERNAME);
|
||||
if (!selected.length) {
|
||||
throw new Error(
|
||||
`未从 ${path.join(resolveApiLogDir(), 'wms-api.log')} 找到可回放的只读 method 样本`
|
||||
+ '(需先用真实账号跑一批只读接口,让日志里先有样本)'
|
||||
);
|
||||
const token = extractToken(login);
|
||||
if (!token) {
|
||||
throw new Error('登录成功但未拿到 token,请检查 user.login 返回结构');
|
||||
}
|
||||
|
||||
const cases = [];
|
||||
for (const item of selected) {
|
||||
const call = await callSystemMethod(baseUrl, item.method, item.body, item.headers, token);
|
||||
cases.push({
|
||||
method: item.method,
|
||||
source: item.source,
|
||||
sourceId: item.sourceId,
|
||||
sourceCreatedAt: item.sourceCreatedAt,
|
||||
headers: item.headers,
|
||||
requestBody: item.body,
|
||||
doc: methodDocs.get(item.method) || null,
|
||||
httpStatus: call.httpStatus,
|
||||
response: call.response,
|
||||
normalized: call.normalized,
|
||||
});
|
||||
}
|
||||
|
||||
const baseline = {
|
||||
generatedAt: new Date().toISOString(),
|
||||
baseUrl,
|
||||
user: TEST_USERNAME,
|
||||
note: 'Before-change baseline generated from recent system logs and methods/system.json',
|
||||
login: {
|
||||
httpStatus: login.httpStatus,
|
||||
normalized: login.normalized,
|
||||
},
|
||||
cases,
|
||||
};
|
||||
|
||||
ensureOutputDir();
|
||||
fs.writeFileSync(BASELINE_FILE, JSON.stringify(baseline, null, 2), 'utf8');
|
||||
fs.writeFileSync(BASELINE_SUMMARY, renderBaselineMarkdown(baseline), 'utf8');
|
||||
|
||||
console.log(`BASELINE_OK cases=${cases.length}`);
|
||||
console.log(`BASELINE_FILE ${BASELINE_FILE}`);
|
||||
console.log(`BASELINE_SUMMARY ${BASELINE_SUMMARY}`);
|
||||
} finally {
|
||||
await conn.end();
|
||||
}
|
||||
|
||||
const login = await callSystemMethod(
|
||||
baseUrl,
|
||||
'user.login',
|
||||
{ username: TEST_USERNAME, password: TEST_PASSWORD },
|
||||
{ platform: 'web', project: 'wms' },
|
||||
''
|
||||
);
|
||||
const token = extractToken(login);
|
||||
if (!token) {
|
||||
throw new Error('登录成功但未拿到 token,请检查 user.login 返回结构');
|
||||
}
|
||||
|
||||
const cases = [];
|
||||
for (const item of selected) {
|
||||
const call = await callSystemMethod(baseUrl, item.method, item.body, item.headers, token);
|
||||
cases.push({
|
||||
method: item.method,
|
||||
source: item.source,
|
||||
sourceId: item.sourceId,
|
||||
sourceCreatedAt: item.sourceCreatedAt,
|
||||
headers: item.headers,
|
||||
requestBody: item.body,
|
||||
doc: methodDocs.get(item.method) || null,
|
||||
httpStatus: call.httpStatus,
|
||||
response: call.response,
|
||||
normalized: call.normalized,
|
||||
});
|
||||
}
|
||||
|
||||
const baseline = {
|
||||
generatedAt: new Date().toISOString(),
|
||||
baseUrl,
|
||||
user: TEST_USERNAME,
|
||||
note: 'Before-change baseline generated from recent wms-api.log samples and methods/system.json',
|
||||
login: {
|
||||
httpStatus: login.httpStatus,
|
||||
normalized: login.normalized,
|
||||
},
|
||||
cases,
|
||||
};
|
||||
|
||||
ensureOutputDir();
|
||||
fs.writeFileSync(BASELINE_FILE, JSON.stringify(baseline, null, 2), 'utf8');
|
||||
fs.writeFileSync(BASELINE_SUMMARY, renderBaselineMarkdown(baseline), 'utf8');
|
||||
|
||||
console.log(`BASELINE_OK cases=${cases.length}`);
|
||||
console.log(`BASELINE_FILE ${BASELINE_FILE}`);
|
||||
console.log(`BASELINE_SUMMARY ${BASELINE_SUMMARY}`);
|
||||
}
|
||||
|
||||
async function runCompare() {
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user