FDE & ENGINEERING · 2026.10

Convo AgentHarness.

让业务规则与优化方法，在大规模真实数据上持续执行、比较和复用Run, compare and reuse business rules and tuning methods across real conversations at scale

让每次迭代都有数据依据Evidence for every iteration

增长逻辑Growth drivers

上量，更要提升转化率Scale volume. Improve conversion.

毛利 = 用量 × 转化率 × 单价 − 直接成本Gross profit
= Volume × Conversion rate × Price − Direct costs

简化模型：按成功转化计费，单价为每次转化收入。Simplified model: billing per successful conversion; price is revenue per conversion.

27 天 · +73%全场景日外呼 515 万 → 890 万27 days · +73%All-scenario daily calls: 5.15M → 8.9M

日外呼量DAILY OUTBOUND CALLS

全场景（当日同步会数）All scenarios (daily-sync figure)仅汽车Automotive only各点为日会快照，非连续序列；虚线仅示意走向Each point is a daily-sync snapshot, not a continuous series; dotted lines only show the direction

7–9 月 OKR：单日外呼从 0 突破 350 万+，落地场景 10+Jul–Sep OKR: daily outbound grew from 0 to over 3.5M, with 10+ scenarios live

基础通话体验Basic call experience

延迟Latency端到端响应时间end-to-end response time
打断Barge-in用户插话时及时停下stop promptly when the user cuts in
WERWER语音识别词错误率speech-recognition word error rate
读法Spoken rendering数字、日期等转为口语读法numbers, dates etc. in spoken form

优化目标Optimization goal减少等待与抢话，降低沟通阻力Reduce waiting and talk-over to keep calls flowing

话术遵循Script adherence

红线Red lines不说禁止的内容never say prohibited content
流程遵循Flow adherence按既定流程推进follow the defined flow
卡话术Script loops / stalls重复或停在同一句话术repeating or stalling on one line
关键信息提取Key-info extraction准确提取用户给出的信息capture what the user provides

优化目标Optimization goal稳定执行业务流程，减少跑题、循环与违规Follow the business flow without drift, loops or violations

真人感Human-likeness

主动附和Backchanneling对话中适时回应timely acknowledgements
主动打断Proactive interruption必要时由 Agent 主动插话the Agent cuts in when needed
垫词Fillers语气词与停顿filler words and pauses
情绪识别与表达Emotion识别用户情绪，以合适语气回应recognize emotion and respond with an appropriate tone

优化目标Optimization goal让回应自然连贯，提高持续沟通意愿Make responses natural and coherent to sustain engagement

目标达成Goal achievement

采集目标Collection goals拿到业务需要的信息或结果obtain the information or outcome the business needs
采集顺序Collection order按要求的顺序完成采集collect in the required order
信息抽取准确率Extraction accuracy抽取字段与用户真实表达一致的比例share of extracted fields matching what the user said

优化目标Optimization goal准确采集有效信息，推动业务转化Capture accurate, useful information to drive conversion

Harness

数据驱动Data-driven用量与问题都来自线上通话数据volume and issues both come from live call data

02 / 15

本页调整：先说明增长与转化，再介绍 Harness 调优闭环。公式是按成功转化计费的简化业务模型，扣除直接成本后才是毛利；并非声称当前平台按此方式计费。图表横轴按真实日期等距绘制；全场景 9/30 至 10/27，27 天从 515 万增至 890 万，增幅约 72.8%。汽车与全场景口径不同，不合并计算增速。四卡底部为优化目标，以下项目证据仅保留在讲稿，不代表统一验收阈值。

This slide now precedes the Harness loop. The formula is a simplified model for billing per successful conversion, not a claim about current platform billing. Gross profit deducts direct costs. The time axis uses calendar dates; all-scenario calls rose from 5.15M on September 30 to 8.9M on October 27: about 72.8% in 27 days. Automotive and all-scenario series have different scopes and are not combined for growth calculations. Card footers describe optimization goals; historical examples below are speaker context, not universal acceptance thresholds.

数据来源（会议纪要 / 文档，整理于 2026-10-28）：
· 全场景日外呼（图中标注 9/30 515 万、10/13 690 万、10/27 890 万）：9/30 约 515 万（GKAI-G项目每日同步会-2026-09-30）；10/08 约 510 万（GKAI-G项目每日同步会-2026-10-08；Q4 规划 q4plan.txt 同口径）；10/09 约 510 万（同会 10-09）；10/10 约 540 万（同会 10-10）；10/12 约 610 万、10/13 约 690 万（知识库，GKAI-G项目每日同步会对应日期）；10/15 约 665 万（同会 10-15）；10/19 约 620 万+（同会 10-19）；10/20 约 663 万（同会 10-20）；10/27 约 890 万（GKAI-sync-2026-10-27）。
· 仅汽车（图中标注 9/16 280 万）：9/15 近 260 万（GKAI-sync-09-15）；9/16 约 280 万（GKAI-sync-09-16）；9/21 约 290 万（GKAI-sync-09-21）；9/22 近 300 万（GKAI-sync-09-22）；9/28 约 440 万（GKAI-sync-09-28）。7–9 月 OKR（Confluence 2033059005）。
· 卡片证据：光潽 DeepSeek 2.2–2.4 秒（GPCZ-光潽站会-2026-10-21）；规划端到端 2500ms 内（Confluence 1911160966、1891926708、1852637576）；打断 100ms（GPCZ-光潽站会-2026-10-13）；千问 3.6 / Plus 话术遵循性差（GKAI-sync-09-14、09-15）；汽车语气 / 停顿（GKAI-sync-09-28）；光潽主动打断 demo（GPCZ-光潽站会-2026-10-13、10-20）；上海 NLP（GKAI-G项目每日同步会-2026-10-22）；Entel 漏斗（ENTEL-CAMPAIGN-2026-10-16-ANALYSIS）。
· 未上卡片的案例：早期端到端约 2.8 秒、长期目标 2 秒内（同上 Confluence）；光潽认为 1.6 秒内合理（GPCZ-光潽站会-2026-09-15）；武汉运营商来回抢话，疑与约 2 秒多的模型延迟有关（GKAI-sync-2026-10-27）；拟人化与语速控制调研中，目标 11-10 前可用（GPCZ-10-27）；各客户目标口径不同（A 单、采集成功率、转接），暂无统一的目标达成率（知识库）。
· WER、Text Normalization、红线、流程遵循、卡话术、关键信息提取、主动附和、情绪识别、情绪表达、采集目标、采集顺序：素材中暂无对应案例，卡片只列名称与定义。
· 未上屏：收入与单分钟成本（GKAI-sync-2026-10-27；GKCHENGBEN-成本周会-2026-10-21）属内部商务数据；俊博 100–120 万 / 日为潜在客户人工团队量，非我方平台量，不使用。

Sources (meeting minutes / documents, compiled 2026-10-28):
· All-scenario daily outbound (labelled on the chart: 9/30 5.15M, 10/13 6.9M, 10/27 8.9M): 9/30 ~5.15M (GKAI-G daily sync 2026-09-30); 10/08 ~5.1M (GKAI-G daily sync 2026-10-08; same figure in the Q4 plan q4plan.txt); 10/09 ~5.1M (same meeting, 10-09); 10/10 ~5.4M (10-10); 10/12 ~6.1M and 10/13 ~6.9M (knowledge base, GKAI-G daily sync of those dates); 10/15 ~6.65M (10-15); 10/19 6.2M+ (10-19); 10/20 ~6.63M (10-20); 10/27 ~8.9M (GKAI-sync-2026-10-27).
· Automotive only (labelled on the chart: 9/16 2.8M): 9/15 nearly 2.6M (GKAI-sync-09-15); 9/16 ~2.8M (GKAI-sync-09-16); 9/21 ~2.9M (GKAI-sync-09-21); 9/22 nearly 3M (GKAI-sync-09-22); 9/28 ~4.4M (GKAI-sync-09-28). Jul–Sep OKR (Confluence 2033059005).
· Card evidence: Guangpu DeepSeek 2.2–2.4 s (GPCZ Guangpu stand-up 2026-10-21); planning target within 2,500 ms end to end (Confluence 1911160966, 1891926708, 1852637576); barge-in 100 ms (GPCZ 2026-10-13); Qwen 3.6 / Plus poor script adherence (GKAI-sync-09-14, 09-15); automotive fillers / pauses (GKAI-sync-09-28); Guangpu proactive-interruption demo (GPCZ 2026-10-13, 10-20); Shanghai NLP (GKAI-G daily sync 2026-10-22); Entel funnel (ENTEL-CAMPAIGN-2026-10-16-ANALYSIS).
· Examples not on the cards: about 2.8 s end to end early on, under 2 s long term (same Confluence pages); Guangpu considers 1.6 s acceptable (GPCZ 2026-09-15); repeated talk-over in the Wuhan carrier project, suspected to relate to model latency of just over 2 s (GKAI-sync-2026-10-27); anthropomorphism and speech-rate control under research, targeted before 11-10 (GPCZ-10-27); each customer defines its goal differently (A-grade leads, collection success, transfers), so there is no unified goal-achievement rate (knowledge base).
· WER, Text Normalization, red lines, flow adherence, script loops / stalls, key-info extraction, backchanneling, emotion recognition, emotion expression, collection goals and collection order: no matching example in the sources, so the cards list only the name and definition.
· Not on the slide: revenue and per-minute cost (GKAI-sync-2026-10-27; GKCHENGBEN cost weekly 2026-10-21) are internal commercial data; Junbo's 1.0–1.2M/day is a prospect's human team, not our platform volume, and is not used.

目标流程 · 含规划TARGET FLOW · INCLUDES PLANS

Harness 总览Harness overview

从线上数据到持续调优From Live Data to Continuous Improvement

Harness

数据驱动Data-driven闭环的起点和终点都是线上数据the loop starts and ends with live data

03 / 15

自身迭代设计（2026-10-28）：本轮按用户要求融入总览图与仿真、测评自身页面，不新增幻灯片。仿真闭环验证模拟用户与场景是否符合事实、人物、行为及流程约束；测评闭环验证判定是否有规则与会话证据支持。Agent 辅助复核不是事实金标，模型间一致也不等于正确；人工确认样本应保留独立的校准/回归集合，避免按待测结果修改标准。异常归因保留“待确认”，不把模拟或测评器错误直接算作业务 Agent 失败。先区分修订对象，再修正、确认、回归并启用版本，保留原会话、运行时配置、原判定与复核记录。总览新增回环表示目标流程，不代表功能已实现或允许自动发布。

Quality-loop design (2026-10-28): integrated into the overview and the simulation/evaluation pages without adding a slide. Review simulator fidelity against facts, persona, behavior and flow constraints; review evaluation verdicts against runtime rules and conversation evidence. Agent-assisted review and model agreement are not ground truth. Keep independently human-confirmed calibration/regression cases; do not change standards merely to fit tested outputs. Preserve uncertain attribution and distinguish simulator/evaluator faults from business-Agent defects. Identify the revision target, correct, confirm, regress and activate a version while retaining original evidence. Loops are target workflows, not shipped-feature claims or automatic release authorization.

本轮融合：测评输出判定与证据，可以写入问题标签；标签本身保持独立分类语义，可由人工或其他自动流程产生。流程统一是产品方向，不表示当前自动打标与测评 API 已合并。用户 Agent 驱动的验证资产生成与调优闭环属于 Q4 目标，自主改进为 2027 探索。Integration: evaluation yields verdicts and evidence and may write issue labels. Labels retain independent classification semantics and may be created manually or by other automation. A unified product workflow does not assert that current labeling and evaluation APIs are merged. User-Agent-driven asset generation and improvement are Q4 goals; autonomous improvement is a 2027 exploration.

主环：生产数据自动导入 → 测评 → 候选验证后的推送、AB 与上线 → 线上反馈 → 新数据。自动评测由新数据到达触发；手工测评由用户主动启动，仍由系统执行。两者共用测评规则，触发方式不同。自动打标负责识别并标记同类问题，不能与自动评测混为一谈。Bad Smell 来自测评或数据探索，可经自动打标形成标签，也可由人工标注；标签分别沉淀为测评规则/回归检查和仿真场景/用户行为/流程。测评固化反馈到测评执行，仿真固化反馈到仿真。主环之外的 Agent 配置支持克隆、微调形成候选，仿真后主动测评，再按反馈迭代；图中自进化为规划，不代表已实现无人值守执行或自动上线。主环和小环均无步骤编号。虚线表示跨环反馈与规划衔接。中文使用“自动评测 / 自动打标 / 手工测评 / 标签”，英文对应“自动评测 / Auto Label / Manual evaluation / Labels”；Bad Smell 保留原术语。

Main loop: automatic data import → evaluation → push, AB and launch after candidate validation → live feedback → new data. Auto Eval is triggered by arriving data; Manual evaluation is actively started by a user and still executed by the system. Both share evaluation rules. Auto Label identifies and labels recurring patterns; it is distinct from Auto Eval. Bad Smell can be identified by evaluation or exploration, then labeled automatically or manually. Labels produce both evaluation rules/regression checks and simulation scenarios/behavior/flow. Codified evaluation feeds evaluation execution; codified simulation feeds simulation. Agent configurations live outside the production loop: clone and tune, simulate a candidate, start evaluation and iterate from feedback. Self-evolution is planned and does not assert unattended execution or automatic release. There are no numbered steps; dashed curves indicate cross-loop feedback and planned handoffs.

上线与迭代Launch and iteration

缩短上线与调优周期Shorten launch and tuning cycles

首版上线First launch

把客户素材变成可验证的首版Turn customer material into a testable first version

建立可复用的验证基线Build a reusable validation baseline

从客户录音、业务规则与历史问题中提炼流程和成功标准。Derive flows and success criteria from recordings, business rules and past issues.

准备测评集与仿真场景，覆盖正常流程、异常输入和边界情况。Prepare evaluation cases and simulation scenarios for normal, abnormal and boundary conditions.

形成可运行的首版配置Prepare a runnable first configuration

基于模板克隆并微调模型、Prompt、变量和工具配置。Clone a template and tune the model, prompt, variables and tools.

固定配置版本与执行绑定，跑通首轮仿真和测评，定位失败环节。Pin the configuration and execution binding; simulate and evaluate to locate failures.

完成接入与上线准备Prepare integration and launch

对齐 APP ID、号码、权限、并发与预热等依赖，减少反复协调。Align APP ID, numbers, permissions, concurrency and warm-up dependencies.

确认目标环境与配置一致，补齐音频体验验证、发布检查和回退方案。Verify configuration consistency, audio experience, release checks and a rollback plan.

预期产出Expected output可运行配置 · 验证基线 · 首轮报告 · 上线清单Runnable configuration · Test baseline · Initial report · Launch checklist

持续迭代Continuous iteration

把问题信号变成可衡量的改进Turn issue signals into measurable improvements

发现并沉淀 Bad SmellDiscover and codify Bad Smell

来源包括听录音、客户报障，以及转化率、挂断率等统计异常。Signals come from listening, customer reports and anomalies in conversion or hang-up rates.

回到 Session 与 Turn 核实原因，归类打标，并固化为测评或仿真场景。Inspect Sessions and Turns, verify causes, label patterns and codify evaluation or simulation cases.

量化每次改动的影响Measure the impact of every change

换模型、换 Prompt、换配置：是否有影响，影响哪些场景，幅度多大？Change a model, prompt or configuration: what changes, where, and by how much?

固定同一验证集对比新旧版本，报告质量、延迟、成本差异与回退风险。Compare versions on a fixed test set; report quality, latency, cost and regression risk.

用线上 AB 验证收益Validate gains with live AB tests

离线验证通过后，再比较线上转化率、挂断率、延迟与成本。After offline validation, compare live conversion, hang-ups, latency and cost.

明确实验组、流量与观察周期，让结果决定放量、继续调优或回退。Define groups, traffic and observation windows; use results to scale, tune or roll back.

预期产出Expected output问题标签 · 版本差异报告 · 回归用例 · 放量依据Issue labels · Version comparison · Regression cases · Rollout evidence

Harness

数据驱动Data-driven同一基线验证改动，线上数据检验效果compare changes on a shared baseline; validate impact with live data

04 / 15

本页描述上线与迭代需要完成的工作和预期产出，不声称这些流程已全部自动化。自动生成验证材料、配置回写和 AB 集成仍以旅程页和规划页的状态标记为准。Bad Smell 是待核实的问题信号，听录音、客户报障或统计异常均可触发排查，不直接等同于已确认缺陷。

This page describes work and expected outputs, not a fully automated workflow. Automatic test generation, configuration write-back and AB integration retain the status shown on the workflow and roadmap slides. Bad Smell is a signal to investigate, not a confirmed defect.

数据来源（整理于 2026-10-28）：
· 快速上线：测评集与 Agent配置生成靠人工（Q4 规划 q4plan.txt / convo-agent-q4-plan*.html，约 10/09 现状）；冷启动约 3 天、模板提示词一周内交付为口头节奏（同上；另见 AIBUSINESS-2026-09-29、GKAI-G项目每日同步会-2026-10-08 待办）；新项目多人人工开通（CALL-HLD评审-2026-10-27）；平均接入天数无统一统计（两份素材均注明查不到）。Agent配置同步一项素材中暂无对应实例。
· 快速迭代：约 264 条 Bad Smell 人工听音（GKAI-sync-2026-09-28；Q4 规划同口径，素材原文写“苏州口腔”，Chang 已确认统一写“口腔”）；TO 每天 ≥10 个录音（CONVOTO-TO预备队项目周会-2026-10-10）；光潽每天一版 prompt（GPCZ-光潽站会-2026-10-13）；931 条测试集、对比约 40 分钟–1 小时（POC-POC周会-2026-10-09；Q4 规划同口径）；1 FDE + 1 TO（GKAI-G项目每日同步会-2026-10-19）；人天无统一统计（两份素材均注明查不到）。
· 驱动 AB / 提升转化：DeepSeek vs Qwen Plus（GKAI-G项目每日同步会-2026-10-09）；运营商开场白 1.49%（GKAI-G项目每日同步会-2026-10-22）；口腔开场白 4–7 个百分点（GKAI-sync-2026-09-16）。
· 另有语音信箱检测 AB：A 6.32% vs B 4.98%，延迟 +10–20ms（GKAI-sync-2026-09-16），指标含义未注明，未上屏。
· “提升转化”归入“快速迭代”（Chang）。
· 问题名称 = Chang 的定义（2026-10-28）：“建立测评集与仿真集”；“Bad Smell 识别、修复并固化为测评”。

Sources (compiled 2026-10-28):
· Fast launch: Evaluation Suites and Configuration generation are manual (Q4 plan q4plan.txt / convo-agent-q4-plan*.html, status around 10/09); ~3-day cold start and template prompts within a week are a verbal cadence (same; also AIBUSINESS-2026-09-29 and the GKAI-G daily sync 2026-10-08 to-do list); new projects set up by several people by hand (CALL HLD review 2026-10-27); no unified onboarding-days statistic (both sources say none was found). Configuration sync has no matching example in the sources.
· Fast iteration: ~264 Bad Smell by manual listening (GKAI-sync-2026-09-28; same in the Q4 plan; the source says “Suzhou Dental”, Chang confirmed the name “Dental”); TO ≥10 recordings a day (CONVOTO TO reserve weekly 2026-10-10); Guangpu one prompt version a day (GPCZ Guangpu stand-up 2026-10-13); 931-case test set compared in ~40 min–1 h (POC weekly 2026-10-09; same in the Q4 plan); 1 FDE + 1 TO (GKAI-G daily sync 2026-10-19); no unified person-day statistic (both sources say none was found).
· Driving AB / conversion: DeepSeek vs Qwen Plus (GKAI-G daily sync 2026-10-09); carrier opening line 1.49% (GKAI-G daily sync 2026-10-22); dental opening line 4–7 points (GKAI-sync-2026-09-16).
· Also: a voicemail-detection AB, A 6.32% vs B 4.98%, +10–20 ms latency (GKAI-sync-2026-09-16); the metric is not defined in the source, so it is not on the slide.
· Raising conversion belongs under fast iteration (Chang).
· Problem names = Chang's definitions (2026-10-28): “Build the Evaluation Suite and Simulation Suite”; “Identify and fix Bad Smell, and turn them into Evaluations”.

历史项目背景（移入讲稿）：测评集目前靠人工编写；新环境冷启动约 3 天，模板提示词一周内交付（口头节奏）；新项目每开一个 APP ID 和号码，需多人分别处理池化、加白、并发和预热，全部人工完成；暂无新客户平均接入天数的统一统计；口腔项目约 264 条 Bad Smell 依靠人工听音发现；TO 每天至少随机听 10 个录音查找 Bad Smell；光潽每天发布一版 prompt；光潽 931 条测试集，新旧版本对比约 40 分钟–1 小时（10/09）；模型 AB：DeepSeek 比 Qwen Plus 转化高约 0.073（相对约 8–10%），延迟多 400 多毫秒；运营商开场白优化后当天转化 1.49%（目标 1%）；口腔开场白由 12 秒缩短到 7–8 秒，约 10 秒挂断率下降约 4–7 个百分点；每个项目固定分配 1 名 FDE + 1 名 TO；暂无“一轮调优所需人天”的统一统计

Historical project context (moved to speaker notes): Evaluation Suites are written by hand today；A new environment takes about 3 days to cold-start; template prompts ship within a week (verbal cadence)；Each new APP ID and number needs several people to handle pooling, whitelisting, concurrency and warm-up separately, all by hand；There is no unified statistic on average onboarding days；About 264 Bad Smell in the Dental project were found by manual listening；TOs listen to at least 10 random recordings a day to find Bad Smell；Guangpu ships a new prompt version every day；Guangpu's 931-case test set: old vs new version compared in about 40 minutes to 1 hour (10/09)；Model AB: DeepSeek converts about 0.073 higher than Qwen Plus (about 8–10% relative), with 400+ ms more latency；After the carrier opening-line change, conversion was 1.49% that day (target 1%); shortening the dental opening line from 12 s to 7–8 s cut the ~10-second hang-up rate by about 4–7 points；Each project is staffed with 1 FDE + 1 TO；There is no unified statistic on person-days per tuning round

设计图 · 回写 / AB 为规划DESIGN · WRITE-BACK / AB PLANNED

系统架构System architecture

数据、执行与平台集成Data, execution and platform integration

Convo Agent Harness 由 Harness 前端 和 Harness 后端 组成；数据存 DB，录音存 OSSConvo Agent Harness consists of the Harness frontend and the Harness backend; data is stored in the DB and recordings in OSS
外部平台的通话经 ConvoAI Importer 单向导入 Harness：Studio（海外 Agent 平台，外部）、Call Center（国内 Agent 平台）、ConvoAI 后台Calls from external platforms flow one way into Harness through the ConvoAI Importer: Studio (an external overseas Agent platform), Call Center (domestic Agent platform) and the ConvoAI backend
Worker 是仿真与测评的实际执行位置，经私有 API 与后端通信，不直连 DB；目标 Agent 的模型请求经 Worker 内的 Interceptor 路由到 LLM 提供方Workers are where Simulation and Evaluation actually run; they talk to the backend over a private API and never connect to the DB directly; the target Agent's model requests route through the Interceptor inside the Worker to the LLM provider
仿真测评定义由 Harness 后端同步Simulation & Evaluation definitions are synced by the Harness backend
Harness 后端向 AB 平台发起 AB，AB 平台作用于 ConvoAI 后台The Harness backend starts AB tests on the AB platform, which acts on the ConvoAI backend
推送新版本 Agent配置：由 Harness 后端直接推送到 Studio 和 Call CenterPush new Configuration version: the Harness backend pushes directly to Studio and Call Center

Harness

数据驱动Data-driven数据存于 DB 与 OSSdata is stored in DB and OSS

05 / 15

来源 = architecture.md §1–§4，按 Chang 的要求简化为前端 / 后端两部分，存储只写 DB 与 OSS。Worker 经私有 API（machine-token）回连后端，不直连 DB；Interceptor 是 Worker 内的伴随进程，目标 Agent 的模型请求经它路由（concepts.md §5）。
外部来源与 AB 平台按 Chang 给定的结构绘制：Studio 指海外的外部 Agent 平台（不是本产品），Call Center 为国内 Agent 平台，ConvoAI 后台为第三个来源；三者都是单向进入 ConvoAI Importer。
本页为设计图：AB 平台（Harness 后端 → AB 平台“发起 AB”，AB 平台 → ConvoAI 后台）与“推送新版本 Agent配置”（Harness 后端 → Studio 和 Call Center）两条连线按 Chang 的设计绘制。
仿真测评定义的来源仓库不在页面展示。

Source = architecture.md §1–§4, simplified to a frontend and a backend as Chang asked, with storage shown only as DB and OSS. Workers call back to the backend over a private API (machine token) and never connect to the DB directly; the Interceptor is a companion process inside the Worker that routes the target Agent's model requests (concepts.md §5).
External sources and the AB platform follow the structure Chang specified: Studio is an external overseas Agent platform (not this product), Call Center is a domestic Agent platform, and the ConvoAI backend is the third source; all three flow one way into the ConvoAI Importer.
This is a design diagram: the AB platform links (Harness backend → AB platform “Start AB”, AB platform → ConvoAI backend) and “Push new Configuration version” (Harness backend → Studio and Call Center) follow Chang's design.
The repository behind the Simulation & Evaluation definitions is not shown on the slide.

概念视图 · 非完整数据模型CONCEPT VIEW · NOT A FULL SCHEMA

数据模型Data model

Agent 配置、Session 与标签Agent configurations, Sessions and labels

Agent配置是核心，本质对应 ConvoAI 的 Pipeline ID + Version，支持创建临时版本和 Fork；同时关联仿真任务与生产导入Configuration is the core; it essentially corresponds to ConvoAI's Pipeline ID + Version and supports creating temporary versions and Fork; it is linked to both Simulation runs and production imports
Session 有三个来源：仿真任务（关联 Agent配置与仿真定义）产出机机数据；生产导入产出人机数据；手工导入产出人人数据等离线数据Sessions have three sources: Simulation runs (linked to a Configuration and a Simulation definition) produce Agent–Agent data; production imports produce Human–Agent data; manual imports bring in Human–Human and other offline data
人机数据包括线上实际数据和调试数据Human–Agent data includes real online data and debug data
元数据主要来自数据生产过程，例如人机数据会记录 pipeline id、llm_config、是否来自仿真、结构化抽取结果；录音链接也属于元数据Metadata comes mostly from the data production process; for example, Human–Agent data records the pipeline id, llm_config, whether it came from Simulation, and structured-extraction results; recording links are also Metadata
标签可由 AI 打标，也可来自人工标注，内容不限，默认无Annotations can be labelled by AI or come from human annotation; they can be any content and are empty by default
元数据和标签都可以附加在 Session 本身，也可以附加在 Session 的某个 Turn 上Both Metadata and Annotations can attach to the Session itself or to a single Turn in the Session

Harness

数据驱动Data-driven数据最终都落入 Sessionall data ultimately lands in Sessions

06 / 15

本轮融合：测评输出判定与证据，可以写入问题标签；标签本身保持独立分类语义，可由人工或其他自动流程产生。流程统一是产品方向，不表示当前自动打标与测评 API 已合并。用户 Agent 驱动的验证资产生成与调优闭环属于 Q4 目标，自主改进为 2027 探索。Integration: evaluation yields verdicts and evidence and may write issue labels. Labels retain independent classification semantics and may be created manually or by other automation. A unified product workflow does not assert that current labeling and evaluation APIs are merged. User-Agent-driven asset generation and improvement are Q4 goals; autonomous improvement is a 2027 exploration.

来源 = Chang 对实体模型的说明（2026-10-28）：Agent配置为核心，本质对应 ConvoAI 的 Pipeline ID + Version，支持临时版本和 Fork；仿真本身是一个任务，关联仿真定义，最终产出 Session；Session 还会来自生产导入或手工导入，生产导入也关联 Agent配置；Session 有三类数据；元数据与标签均可附加在 Session 或单个 Turn 上。
来源与数据类型的对应按 Chang 此前的定义：人人数据 = 直接离线导入，对应手工导入；人机数据 = 线上实际数据和调试数据，对应生产导入（调试数据也属于人机数据，未单独画箭头）；机机数据 = 来自仿真，对应仿真任务。
与已核实事实的对应：三类数据在数据中的取值为 human_human、human_ai、ai_ai，Session 与 Turn 都可带标签（concepts.md §4；entities.md）。
按 Chang 的要求，本页不画实体之间的引用关系，也不包含运行时相关的实体与流程；测评集、测评、报告等实体在本页省略。

Source = Chang's description of the entity model (2026-10-28): Configuration is the core and essentially corresponds to ConvoAI's Pipeline ID + Version, with temporary versions and Fork; Simulation is itself a run linked to a Simulation definition, and its output lands in Sessions; Sessions also come from production imports or manual imports, and production imports are also linked to a Configuration; Sessions have three data types; Metadata and Annotations can attach to the Session or to a single Turn.
The source-to-data-type mapping follows Chang's earlier definitions: Human–Human = imported offline directly, so manual import; Human–Agent = real online data and debug data, so production import (debug data is also Human–Agent and has no separate arrow); Agent–Agent = from Simulation, so Simulation runs.
Match with verified facts: the three data types are stored as human_human, human_ai and ai_ai, and both Sessions and Turns can carry annotations (concepts.md §4; entities.md).
As Chang asked, this slide does not draw reference relationships between entities and leaves out runtime entities and flows; Evaluation Suites, Evaluations and Reports are omitted here.

2026-10-27 数据快照DATA SNAPSHOT · 2026-10-27

数据规模Data at a glance

Harness 现有数据量Current data in Harness

约 98 万~980K条 Session（生产库只读计数，2026-10-27）Sessions (read-only production count, 2026-10-27)

按数据类型BY DATA TYPE

人机数据Human–Agent891,04890.9%

机机数据Agent–Agent72,8547.4%

人人数据Human–Human16,4611.7%

各业务累计数据 · 含全部数据类型Cumulative data by business · All data types

冠客汽车Guanke Auto

722,29873.7%

口腔Dental

75,4497.7%

光潽Guangpu

63,0276.4%

EntelEntel

58,0025.9%

台湾民调Taiwan poll

31,6463.2%

其他Others

29,9413.1%

全量接入后的日增预期Expected daily growth at full import全场景 · 估算All scenarios · Estimate

约 90 万~900K条 Session / 日Sessions / day

900 万次呼叫 / 日 × 10% 接通率9M calls / day × 10% connection rate

按一次接通对应一条 Session 估算，目标是全量导入。Assumes one Session per connected call, with all live Sessions imported.

现有约 98 万条是累计存量；90 万条是日增预期，不代表实际覆盖率。~980K is cumulative inventory; ~900K is expected daily growth, not measured coverage.

每周新增 Session（按数据类型，按通话时间）· 峰值 24.0 万（10-05 周）New Sessions per week (by data type, by call time) · peak 240K (week of 10-05)

人机数据机机数据人人数据

Human–AgentAgent–AgentHuman–Human

任务与标签（生产库只读计数，2026-10-27）RUNS AND ANNOTATIONS (read-only production count, 2026-10-27)

50,078仿真任务Simulation runs

68,346测评任务Evaluation runs

93,134测评结果Evaluation results

121自动打标批次Auto-label batches

640标签词表条目Annotation vocabulary entries

注：人机数据含线上与调试数据，不等于线上导入量。周图按通话时间，覆盖 95.5%；10-26 周仅截至 10-27，不能据此判断业务量下降。Note: Human–Agent data includes live and debug Sessions, so it is not the live-import count. Weekly data covers 95.5%; the week of Oct 26 ends on Oct 27 and does not establish a decline in usage.

Harness

数据驱动Data-drivenSession、任务与标签均来自生产库只读计数Sessions, runs and annotations all come from read-only production counts

07 / 15

本页采用用户确认的规划估算：全场景每日 900 万次呼叫，按 10% 接通率、每次接通对应 1 条源 Session，预计每日产生约 90 万条 Session，目标全量导入。900 万与 10% 是估算输入，不是从源系统核实的同期 Session 计数。现有 980,363 条为累计全类型数据，其中人机数据也含调试；不能直接除以每日 900,000 条来算覆盖率。实际覆盖率若后续补充，应使用同日、同范围、按源 Session 去重的已导入线上量与线上实际量。数据类型用环形图，项目条形图表示各项目占总量的比例。

Planning estimate confirmed by the user: 9M calls per day across all scenarios, 10% connection rate and one source Session per connected call imply about 900K new Sessions per day, targeting full import. These are planning inputs, not verified source counts. The existing 980,363 Sessions are cumulative and include all data types; Human–Agent also includes debug Sessions. Do not divide this stock by the daily estimate to claim coverage. Any future measured coverage must match day, scope and deduplicated source Sessions. The donut shows type composition; project bars show shares of total inventory.

来源 = stats.md（生产 DB 只读计数，2026-10-27）：
· Session：按数据类型 human_ai 891,048 / ai_ai 72,854 / human_human 16,461，合计 980,363（key 总数 980,368，差 5 条为两次查询间写入，页面写“约 98 万”）；按项目 gk-vehicle 722,298、gk-dentist 75,449、gp 63,027、entel 58,002、tw-survey 31,646，其余项目合计 29,941（stats.md §2）。
· 任务与标签：SimulationRun 50,078（仿真任务）；EvaluationRun 68,346（测评任务）；Evaluation 93,134（单个测评结果文档）；auto-label batch 121（自动打标批次）；Tag 640 为 project + scope 级的标签词表条目，不是被打标次数（stats.md §1）。
· 周图来源 = trend.md、charts/trend_summary.json；按通话时间 metadata.call_time 统计。
项目 ID → 名称（仅讲稿）：gk-vehicle = 冠客汽车；gk-dentist = 口腔；gp = 光潽；entel = Entel（秘鲁）；tw-survey = 台湾民调。
本页不放项目数与 token 数。

Source = stats.md (read-only production DB counts, 2026-10-27):
· Sessions: by data type human_ai 891,048 / ai_ai 72,854 / human_human 16,461, total 980,363 (980,368 keys; the 5-row gap is writes between queries, so the slide says “~980K”); by project gk-vehicle 722,298, gk-dentist 75,449, gp 63,027, entel 58,002, tw-survey 31,646, all other projects 29,941 combined (stats.md §2).
· Runs and annotations: SimulationRun 50,078 (Simulation runs); EvaluationRun 68,346 (Evaluation runs); Evaluation 93,134 (individual Evaluation result documents); auto-label batch 121; Tag 640 is the project- and scope-level annotation vocabulary, not the number of annotations applied (stats.md §1).
· Weekly chart source = trend.md, charts/trend_summary.json, by call time (metadata.call_time).
Project ID → name (notes only): gk-vehicle = Guanke Auto; gk-dentist = Dental; gp = Guangpu; entel = Entel (Peru); tw-survey = Taiwan poll.
No project count or token figures on this slide.

文本起步 · 音频与故障注入规划中TEXT FIRST · AUDIO & FAILURE INJECTION PLANNED

核心概念 · 仿真Core concepts · Simulation

仿真：覆盖真实对话的变化Simulation: cover real conversation variability

定义用户、场景与交互条件，观察 Agent 如何应对Define the user, scenario and interaction conditions以下为覆盖设计，逐步扩展能力Coverage design for staged expansion

人物与事实边界Persona and factual boundaries

用户设定Persona身份、诉求、知识水平与情绪语气Identity, intent, knowledge and emotional tone
事实边界Fact boundaries固定事实、可变背景与未知信息分别定义Separate fixed facts, variable context and unknowns
披露节奏Disclosure按提问逐步透露，不提前泄露全部答案Reveal facts as asked; never disclose all answers upfront

状态与分支覆盖States and branch coverage

触发条件Branch triggers正常路径与红线分支；未触发不强行补演Cover normal and red-line paths; do not force untriggered branches
状态变化State changes配合后拒绝、授权撤回、换人接听与情绪变化Refusal, withdrawn consent, a new speaker and emotional shifts
结束与重入End and re-entry明确结束条件；覆盖结束后再提问与恢复Define stopping rules; test questions after closure and resumption

意向、授权与表达Intent, consent and expression

意向与授权Intent vs. consent愿意了解、同意联系、要求转接分别设场景Separate interest, permission to contact and transfer requests
表达变化Language variation同义改写、多语言、数字日期与分段补充Paraphrases, languages, numbers, dates and partial answers
纠错与冲突Corrections and conflicts改口、前后矛盾、未知回答与越界要求Corrections, contradictions, unknown answers and rule conflicts

等待与轮次交互Timing and turn-taking

时序 · 规划Timing · Planned

等待与停顿Waits and pauses应答延迟、句中停顿、静默与静态等待语Response delays, pauses, silence and fixed waiting messages
插话与重叠Overlap主动插话、短附和、同时说话与被截断的残句Interruptions, backchannels, overlap and cut-off utterances
耐心与恢复Patience and recovery连续未听清、催促／挂机；打断后续说或改口Repeated mishearing, impatience or hang-up; resume or revise

声音与声学环境Voice and acoustic conditions

音频 · 规划Audio · Planned

说话方式Voice variation口音、语速、音量、情绪语气与数字读法Accent, pace, loudness, emotional tone and spoken numbers
环境干扰Acoustic noise背景人声、音乐、噪声、回声与低音量Background speech, music, noise, echo and quiet speech
识别偏差Recognition errors漏字、错字与截断；区分文本注入和真实音频Dropped words, substitutions and clipping; text vs. audio tests

工具与链路异常Tool and connection failures

注入 · 规划Injection · Planned

工具返回Tool responses超时、失败、空结果、慢返回与重复请求Timeouts, failures, empty or slow results and repeated calls
传输条件Transport网络延迟、抖动、丢包、断连与恢复Delay, jitter, packet loss, disconnection and recovery
通话状态Call states语音信箱、中途挂断、转接失败与结束确认Voicemail, hang-ups, transfer failures and end confirmation

仿真自身迭代Improve simulation itself目标流程Target workflow

生成并执行Generate & run检查模拟用户Review simulated user修正场景Refine scenarios回归验证Regression test

检查提前泄露、被 Agent 带偏、分支未触发与结束后续演；区分模拟偏差和 Agent 问题。确认修订后回归，保留会话、证据与版本。Check early disclosure, Agent-led drift, missed branches and continuation after closure. Separate simulator faults from Agent issues; confirm, retest and retain evidence and versions.

固化、覆盖与复现Codification, coverage and reproducibility

有效对话与已核实 Bad Smell → 仿真固化／测评固化。先检查覆盖，再复用、扩充或新增；固定配置、场景版本与随机种子，重复运行观察波动。Good conversations and verified Bad Smell → simulation/evaluation codification. Check coverage before reuse, expansion or creation; pin configuration, scenario version and seed, then repeat to assess variation.

Harness

数据驱动Data-driven场景定义驱动仿真，产出 Session 与可复现证据scenarios drive simulation, producing Sessions and reproducible evidence

08 / 15

Eval Repo 覆盖补充（2026-10-28，master c588e91）：参考汽车、口腔、光谱与 Entel 的场景及测评定义。规则存在不等于任务已执行或效果已验证。分级、不适用与证据不足属于规则级语义，不宣称新增统一 API verdict；现有 pass / fail / unknown 契约保持不变。逐轮判定只使用截至该轮的证据；会话级结果另按完整会话与工具事件判断。音频与精确时序仍为扩展规划。Eval Repo coverage additions (2026-10-28, master c588e91): informed by automotive, dental, GP and Entel definitions. Defined rules do not prove execution or validated results. Grades, non-applicability and insufficient evidence are rule-level semantics, not new unified API verdicts; pass / fail / unknown remains unchanged. Turn-level judgments use evidence only up to that turn; session outcomes use the full conversation and tool events. Audio and precise timing remain planned.

自身迭代设计（2026-10-28）：本轮按用户要求融入总览图与仿真、测评自身页面，不新增幻灯片。仿真闭环验证模拟用户与场景是否符合事实、人物、行为及流程约束；测评闭环验证判定是否有规则与会话证据支持。Agent 辅助复核不是事实金标，模型间一致也不等于正确；人工确认样本应保留独立的校准/回归集合，避免按待测结果修改标准。异常归因保留“待确认”，不把模拟或测评器错误直接算作业务 Agent 失败。先区分修订对象，再修正、确认、回归并启用版本，保留原会话、运行时配置、原判定与复核记录。总览新增回环表示目标流程，不代表功能已实现或允许自动发布。

Quality-loop design (2026-10-28): integrated into the overview and the simulation/evaluation pages without adding a slide. Review simulator fidelity against facts, persona, behavior and flow constraints; review evaluation verdicts against runtime rules and conversation evidence. Agent-assisted review and model agreement are not ground truth. Keep independently human-confirmed calibration/regression cases; do not change standards merely to fit tested outputs. Preserve uncertain attribution and distinguish simulator/evaluator faults from business-Agent defects. Identify the revision target, correct, confirm, regress and activate a version while retaining original evidence. Loops are target workflows, not shipped-feature claims or automatic release authorization.

术语统一：Auto Eval 的中文为“自动评测”。Case 转仿真／测评资产，就是总览图中的“仿真固化与测评固化”，不是另外一项平行能力；产出是可执行、可复用的仿真场景与测评规则。

Terminology: Auto Eval is “自动评测” in Chinese. Turning cases into simulation/evaluation assets is the same simulation and evaluation codification shown in the overview, not a separate capability.

本轮融合：测评输出判定与证据，可以写入问题标签；标签本身保持独立分类语义，可由人工或其他自动流程产生。流程统一是产品方向，不表示当前自动打标与测评 API 已合并。用户 Agent 驱动的验证资产生成与调优闭环属于 Q4 目标，自主改进为 2027 探索。Integration: evaluation yields verdicts and evidence and may write issue labels. Labels retain independent classification semantics and may be created manually or by other automation. A unified product workflow does not assert that current labeling and evaluation APIs are merged. User-Agent-driven asset generation and improvement are Q4 goals; autonomous improvement is a 2027 exploration.

本轮扩展（2026-10-28）：本页是产品覆盖设计，不是已上线能力清单。当前演示仍以文本仿真与文本测评为起点；音频端到端、精确时序控制、故障注入及完整运行指标是扩展方向。用户明确要求加入等待延迟、情绪与互相抢话，其余维度是设计建议。各维度是否已实现需逐项核对后端与执行器，不由本页作实现承诺。参考资料：OpenAI · Voice activity detection；LiveKit · Data hooks and metrics；LiveKit · Agent insights。

Expansion on 2026-10-28: this is a product coverage design, not a shipped-feature inventory. The current demo starts with text simulation and evaluation; end-to-end audio, precise timing control, failure injection and full runtime metrics are expansion areas. Waiting, emotion and talk-over were requested by the user; other dimensions are design recommendations. Implementation must be verified per backend/executor capability. References: OpenAI · Voice activity detection；LiveKit · Data hooks and metrics；LiveKit · Agent insights.

来源 = Chang 对仿真的说明（2026-10-28）：本质上是如何模拟 User；行为上包括 User 说什么、是否静默、是否打断 Agent、是否因 ASR 问题导致识别偏差；目标上先保证 Happy Path 的模拟，再保证 Bad Smell 的覆盖，持续补充高风险 Corner Case；不能只是 Persona，还要兼顾流程（“当说到什么时候，应该说什么”）。
“说到 X 时 → 说 Y”为规则形式的示意，不是具体客户对话。
与已核实事实的对应：仿真中由 simulated_user 扮演客户（concepts.md §5）；每个 case 产出一个 ai_ai Session（同上）。

Source = Chang's description of Simulation (2026-10-28): at its core it is about how to simulate the User; in behaviour, what the User says, whether the User goes silent, whether the User interrupts the Agent, and whether ASR problems cause the User's speech to be misrecognised; in goals, first make sure the Happy Path is simulated, then cover Bad Smell, and continuously add high-risk Corner Cases; it must not be Persona alone but must also take the flow into account (“when the conversation reaches a certain point, what the User should say”).
“When the conversation reaches X → the User says Y” illustrates the form of a rule; it is not real customer dialogue.
Match with verified facts: in Simulation, simulated_user plays the customer (concepts.md §5); each case produces one ai_ai Session (same source).

覆盖设计 · 音频与运行指标逐步扩展COVERAGE DESIGN · AUDIO & METRICS EXPANSION

核心概念 · 测评Core concepts · Evaluation

测评：从回答质量到完整通话体验Evaluation: from answers to the full call experience

按 Turn、Session 与事件时间线判断，保留证据Evaluate Turns, Sessions and event timelines with evidence

passfailunknown

语义与信息准确性Meaning and information

事实与条件Facts and conditions不编造事实、不丢失前提、不把未知说成确定No invented facts, omitted conditions or false certainty
信息抽取Extraction字段准确与完整；改口后更新，不沿用旧信息Accurate, complete fields; update corrections, not stale values
逐轮证据Turn-local evidence引用当前轮及之前的证据，不用后文倒推Cite evidence up to the current turn, never later information

流程与规则遵循Flow and rule adherence

流程与状态Flow and state按最新意向推进；撤回后停止，不强推或误重启Honor current intent and withdrawal; no pressure or false restart
约束与红线Constraints禁止内容、隐私边界、越权承诺与指令冲突Prohibited content, privacy, unauthorized promises and conflicts
规则适用性Applicability区分不适用、证据不足与失败；不适用不入分母Separate not applicable, insufficient evidence and failure; exclude N/A

任务完成与工具使用Task completion and tools

意向不等于授权Intent is not consent不把愿意了解当作同意联系或转接Interest alone does not authorize contact or transfer
承诺不等于完成A promise is not success以工具／业务结果确认预约、转接等是否完成Verify bookings and transfers with tool or business outcomes
调用与失败处理Tool use and recovery时机、参数与重复副作用；报错后重试或降级Timing, arguments, duplicate effects; retry or fall back on error

轮次、延迟与抢话Turns, latency and overlap

音频／事件Audio / Events

响应时延Response latency用户说完到首音；中位数、长尾与冷场时长Speech end to first audio; median, tail and silent gaps
互相抢话Talk-over重叠次数／时长、停播与恢复；短附和不误判Overlap, stop latency and recovery; allow valid backchannels
中断归因Interruption attribution区分生成稿、实播与事件；不凭残句判不完整Separate generated text, played audio and events; fragments alone prove little

声音、读法与情绪Voice, rendering and emotion

音频 · 规划Audio · Planned

可懂度与读法Intelligibility吞字、错读、数字日期读法；ASR 错误另计Clipping, mispronunciation and numbers; separate ASR errors
自然度Naturalness语速、停顿、音量、音色稳定与合成伪影Pace, pauses, loudness, voice consistency and artifacts
情绪回应Emotional response是否理解用户情绪，语气是否合适、不过度Appropriate, proportionate responses to the user’s emotion

稳定性与运行成本Reliability and cost

事件／用量Events / Usage

执行健康Execution health超时、异常、重试与输出缺失，单独报告Report timeouts, errors, retries and missing outputs separately
性能与成本Performance and cost阶段耗时、资源用量与每通成本，比较质量取舍Stage latency, usage and per-call cost alongside quality
线上结果Live outcomes转化、挂断与转接；需真实线上数据／AB 验证Conversion, hang-ups and transfers require live data or AB

测评自身迭代Improve evaluation itself目标流程Target workflow

执行测评Evaluate证据复核Review evidence人工金标校准Calibrate to gold labels修订后回归Revise & retest

用人工金标检查误报、漏报、证据错配与适用性；区分 Agent、规则、测评器和环境问题。保留待确认，原判定与复核结果分别留存。Use human gold labels to check false positives, misses, evidence and applicability. Separate Agent, rule, evaluator and environment issues; retain uncertainty and both verdicts.

判定、分级与比较Verdicts, grading and comparison

红线判定与质量分级并行，按规则报告严重度／待改进，避免重复扣分。固定验证集与版本，报告有效分母、覆盖率与波动。手工测评主动启动；自动评测随数据到达（目标）。Combine hard-rule verdicts with quality grades and severity; avoid duplicate penalties. Fix cases and versions; report applicable denominators, coverage and variation. Manual runs are user-started; Auto Eval runs on arrival (target).

Harness

数据驱动Data-driven文本判断语义，音频与事件判断时序，线上数据验证转化text for meaning; audio and events for timing; live data for conversion

09 / 15

Eval Repo 覆盖补充（2026-10-28，master c588e91）：参考汽车、口腔、光谱与 Entel 的场景及测评定义。规则存在不等于任务已执行或效果已验证。分级、不适用与证据不足属于规则级语义，不宣称新增统一 API verdict；现有 pass / fail / unknown 契约保持不变。逐轮判定只使用截至该轮的证据；会话级结果另按完整会话与工具事件判断。音频与精确时序仍为扩展规划。Eval Repo coverage additions (2026-10-28, master c588e91): informed by automotive, dental, GP and Entel definitions. Defined rules do not prove execution or validated results. Grades, non-applicability and insufficient evidence are rule-level semantics, not new unified API verdicts; pass / fail / unknown remains unchanged. Turn-level judgments use evidence only up to that turn; session outcomes use the full conversation and tool events. Audio and precise timing remain planned.

自身迭代设计（2026-10-28）：本轮按用户要求融入总览图与仿真、测评自身页面，不新增幻灯片。仿真闭环验证模拟用户与场景是否符合事实、人物、行为及流程约束；测评闭环验证判定是否有规则与会话证据支持。Agent 辅助复核不是事实金标，模型间一致也不等于正确；人工确认样本应保留独立的校准/回归集合，避免按待测结果修改标准。异常归因保留“待确认”，不把模拟或测评器错误直接算作业务 Agent 失败。先区分修订对象，再修正、确认、回归并启用版本，保留原会话、运行时配置、原判定与复核记录。总览新增回环表示目标流程，不代表功能已实现或允许自动发布。

Quality-loop design (2026-10-28): integrated into the overview and the simulation/evaluation pages without adding a slide. Review simulator fidelity against facts, persona, behavior and flow constraints; review evaluation verdicts against runtime rules and conversation evidence. Agent-assisted review and model agreement are not ground truth. Keep independently human-confirmed calibration/regression cases; do not change standards merely to fit tested outputs. Preserve uncertain attribution and distinguish simulator/evaluator faults from business-Agent defects. Identify the revision target, correct, confirm, regress and activate a version while retaining original evidence. Loops are target workflows, not shipped-feature claims or automatic release authorization.

本轮融合：测评输出判定与证据，可以写入问题标签；标签本身保持独立分类语义，可由人工或其他自动流程产生。流程统一是产品方向，不表示当前自动打标与测评 API 已合并。用户 Agent 驱动的验证资产生成与调优闭环属于 Q4 目标，自主改进为 2027 探索。Integration: evaluation yields verdicts and evidence and may write issue labels. Labels retain independent classification semantics and may be created manually or by other automation. A unified product workflow does not assert that current labeling and evaluation APIs are merged. User-Agent-driven asset generation and improvement are Q4 goals; autonomous improvement is a 2027 exploration.

测量口径：响应时延从真实用户语音结束到 Agent 首个可听音频；抢话需双方音轨或说话区间与播放事件，不能单凭转写判定，也不能将所有重叠（如附和）都判为失败。ASR WER 需要参考转写；任务完成需工具或业务结果证据；真实转化需线上业务定义与实验口径。情绪只评价可观察的表达与回应，不推断用户内心状态。规则阈值按场景确定，保留未知，执行异常不等同于质量失败。多次运行及固定种子不能消除全部外部系统或模型波动；必要时报告区间与不同条件下的分布。

Measurement: response latency runs from actual user speech end to first audible Agent output. Talk-over needs speaker intervals/audio and playback events; transcripts alone are insufficient, and valid backchannels are not failures. ASR WER needs reference transcripts. Task success needs tool/business evidence; conversion needs live business definitions and experiment scope. Emotion evaluation concerns observable expression and responses, not private mental states. Thresholds are scenario-specific; preserve unknowns and separate execution errors from quality failures. Repetition and fixed seeds do not remove all model or external-system variation.

本轮扩展（2026-10-28）：本页是产品覆盖设计，不是已上线能力清单。当前演示仍以文本仿真与文本测评为起点；音频端到端、精确时序控制、故障注入及完整运行指标是扩展方向。用户明确要求加入等待延迟、情绪与互相抢话，其余维度是设计建议。各维度是否已实现需逐项核对后端与执行器，不由本页作实现承诺。参考资料：OpenAI · Voice activity detection；LiveKit · Data hooks and metrics；LiveKit · Agent insights。

Expansion on 2026-10-28: this is a product coverage design, not a shipped-feature inventory. The current demo starts with text simulation and evaluation; end-to-end audio, precise timing control, failure injection and full runtime metrics are expansion areas. Waiting, emotion and talk-over were requested by the user; other dimensions are design recommendations. Implementation must be verified per backend/executor capability. References: OpenAI · Voice activity detection；LiveKit · Data hooks and metrics；LiveKit · Agent insights.

本次评审补充：已按引用代码版本的 verdict 契约补齐 unknown；执行状态与质量判定分开。统计方法、覆盖率呈现为评审建议，不代表新增 API。Review clarification: The verdict contract includes unknown; execution state and quality verdict are distinct. Statistical guidance is a review recommendation, not a new API.

来源 = Chang 对测评的说明（2026-10-28）：本质上是评估 Agent，手段是判断某一个 Turn 的回答是否满足预期；短期内是 0/1 问题，通过多次仿真提供有统计意义的结果；效果上测评流程是否遵守、红线是否遵守；流程上不仅用于仿真结果的测评，还应扩展到线上实时导入数据的自动评测和 AB 效果的自动评测；来源上从 Agent 的定义完成初版测评，不断补充 Bad Smell 的测评，不断扩容能力。
图中的 0/1 标记只示意汇总方式，不代表实际数据，页面不给通过率数值。

Source = Chang's description of Evaluation (2026-10-28): at its core it evaluates the Agent, and the means is judging whether the reply in a given Turn meets expectations; in the near term it is a 0/1 question, and many Simulations give statistically meaningful results; on effect, it evaluates whether the flow is followed and whether the red lines are respected; on process, it is not only for Simulation results but should extend to automatic Evaluation of data imported in real time from production and automatic Evaluation of AB results; on source, it starts with an initial Evaluation built from the Agent's definition, keeps adding Evaluations for Bad Smell, and keeps expanding capability.
The 0/1 marks only illustrate the aggregation; they are not real data, and the slide gives no pass-rate value.

接口基线 · 3c9b914API BASELINE · 3c9b914

API 接入API onboarding

通过 Coding Agent 接入 HarnessConnect to Harness with a Coding Agent

01 / ACCESS

申请项目权限Obtain project access

由具备 admin:api_keys 权限的人员在管理控制台创建 Key，限定项目与所需权限。A holder of admin:api_keys creates a key in the admin console, scoped to the project and required operations.

sessions:readauto_labels:read / writedispatcher:simulationdispatcher:evaluationdebug:run

个人自助创建 Key：Q4 规划。按需授予权限，不需要一次勾选全部能力。Personal self-service keys are planned for Q4. Grant only the capabilities needed.

02 / CONNECT

让 Coding Agent 接入Connect your Coding Agent

先通过本地环境或凭据配置保存 Key，再提供地址、项目与任务。Configure the key locally through the environment or credential settings, then provide the address, project and task.

可交给 Coding Agent 的提示Prompt for your Coding Agent

读取 https://rong.agoralab.co/api/skills 的 skill。使用本地环境变量 STUDIO_API_KEY，不显示其值。先确认口腔项目权限，再读取最近一批 Session，列出可执行的仿真与测评。Read the skills at https://rong.agoralab.co/api/skills. Use the local STUDIO_API_KEY environment variable without displaying its value. Verify access to the dental project, then read a recent Session cohort and identify available Simulations and Evaluations.

X-API-Key ← STUDIO_API_KEY

03 / FIRST RUN

从一次只读探索开始Start with a read-only query

确认项目和访问范围Confirm the project and access scope
读取一小批 Session 与元数据Read a small cohort of Sessions and metadata
核对配置、Suite 和执行依赖Check the configuration, Suite and execution prerequisites
再选择打标、仿真或测评Then choose labeling, Simulation or Evaluation

API 存在不等于专用 skill 齐备；导入等旅程需另外核对权限。An API may exist before a dedicated skill is ready. Check additional permissions for journeys such as import.

Harness

数据驱动Data-drivenCoding Agent 凭 Key 直接读写 Harness 数据Coding Agents read and write Harness data directly with a key

10 / 15

本次评审补充：主页面已修正个人自助 Key 与当前管理权限入口的差异。Key 仅通过本地环境变量引用；新增导入能力需 imports:preview / stage / apply 等对应权限。Review clarification: The slide now distinguishes planned self-service keys from the current admin permission. Secrets are referenced through a local environment variable. Import needs the corresponding imports permissions.

思路 = Chang 的用户旅程（2026-10-28）：每个用户在自己的控制台创建 API Key，勾选自己可以授予的权限；把 API Key 与 skill 根路径交给 Coding Agent，即可开始工作；工作由五项原子 API 能力组成：探索 Session、打标签、执行仿真、执行测评、创建临时 Agent配置。
代码事实（voice-agent-studio origin/main 3c9b914）：
· 创建 Key：页面 /admin/api-keys，接口 POST /api/admin/api-keys，需要 admin:api_keys 权限（admin.go:573–574）；创建者只能授予自身权限与项目的子集（checkOperatorGrant，admin.go:655–657；admin_role.go:22–45）。
· skill：公开提供，路径 /api/skills、/api/skills/{name}、/api/skills/{name}/archive（skills.go:26–50）；skill 脚本接受 --base-url 与 --api-key-env 参数（例如 STUDIO_API_KEY）；请求以 X-API-Key 请求头携带 Key。
· 第二步的提示样例：按 Chang 给的样例整理；Key 已打码，只保留真实前缀 vask_（admin.go），不展示任何真实 Key；https://rong.agoralab.co 取自 skill 文档中 --base-url 的示例地址，/api/skills 为公开的 skill 目录（无需 Key）；第三行任务只用已核实的能力（探索 Session、仿真、测评）。
· 能力与 skill 的对应：打标签由 auto-label skill 覆盖；执行仿真与执行测评由 simulation-evaluation skill 覆盖；探索 Session 即 GET /api/sessions（sessions:read）。
· 创建临时 Agent配置：经 Debug Session → pipeline-versions:promote 或 temporary 集合，需要 debug:run（debug_sessions.go；pipeline_versions.go:28–32；debug_collection.go:227–246）；目前只出现在 simulation-evaluation skill 的自我演进前言中。
差距：
· 目前创建 Key 的页面是管理控制台，需要 admin:api_keys；还没有每个账号都可用的个人控制台。
· 探索 Session 与创建临时 Agent配置还没有专门的 skill。

Idea = Chang's user journey (2026-10-28): each user creates an API key in their own console and ticks the permissions they can grant; they give the API key and the skill root path to their Coding Agent, which can then start working; the work consists of five atomic API capabilities: explore Sessions, label, run Simulations, run Evaluations, create a temporary Configuration.
Code facts (voice-agent-studio origin/main 3c9b914):
· Creating a key: page /admin/api-keys, endpoint POST /api/admin/api-keys, which requires admin:api_keys (admin.go:573–574); the creator can only grant a subset of their own permissions and projects (checkOperatorGrant, admin.go:655–657; admin_role.go:22–45).
· Skills: served publicly at /api/skills, /api/skills/{name} and /api/skills/{name}/archive (skills.go:26–50); skill scripts take --base-url and --api-key-env (e.g. STUDIO_API_KEY); requests carry the key in the X-API-Key header.
· Step 2 prompt sample: based on Chang's example; the key is masked, keeping only the real prefix vask_ (admin.go), and no real key is shown; https://rong.agoralab.co is the host from the --base-url example in the skill docs, and /api/skills is the public skill catalog (no key needed); the third line uses only verified capabilities (explore Sessions, Simulation, Evaluation).
· Capabilities and skills: labelling is covered by the auto-label skill; Simulation and Evaluation by the simulation-evaluation skill; exploring Sessions is GET /api/sessions (sessions:read).
· Creating a temporary Configuration: Debug Session → pipeline-versions:promote or the temporary collection, which needs debug:run (debug_sessions.go; pipeline_versions.go:28–32; debug_collection.go:227–246); it appears only in the simulation-evaluation skill's self-evolution preamble.
Gaps:
· Today the key page is the admin console and needs admin:api_keys; there is no per-user console for every account.
· There is no dedicated skill for exploring Sessions or for creating a temporary Configuration.

AB 集成属于规划AB INTEGRATION PLANNED

演示路径 · 需预检DEMO PATH · PREFLIGHT REQUIRED

使用旅程 · 上量Workflow · Scaling

用测评验证改动，用 AB 验证效果Evaluate changes. Validate impact with AB.

目标GOAL发现典型对话 → 仿真／测评固化 → 修改配置 → 离线对比 → 线上 AB 验证Find representative cases → codify simulations/evaluations → change configuration → compare offline → validate with live AB

1「对 Coding Agent 说」Tell your Coding Agent探索 SessionExplore Sessions

>在口腔项目中读取线上 Session，筛选有效对话与疑似 Bad Smell，保留规则依据和原始证据。Read live Dental Sessions, identify effective conversations and suspected Bad Smell, and retain the criteria and source evidence.

2「对 Coding Agent 说」Tell your Coding Agent归类 · 仿真／测评固化Classify · Codify simulations/evaluations

>按规则为典型对话打标，核实问题；整理可复现的仿真场景与测评规则草案，人工确认后纳入验证集。Label representative conversations and verify issues. Draft reproducible simulation scenarios and evaluation rules; confirm them before adding them to the test set.

3「对 Coding Agent 说」Tell your Coding Agent创建临时 Agent配置Create a temporary Configuration

>针对发现的问题修改 Prompt，通过 Debug Session 创建一个临时 Agent配置版本。Change the Prompt for the issues found, and create a temporary Configuration version through a Debug Session.

4「对 Coding Agent 说」Tell your Coding Agent执行仿真 · 执行测评Run Simulations · Run Evaluations

>固定同一测评集、规则与 case/seed，用当前版本和候选版本分别仿真、测评。Freeze the same Suite, rules and case/seed pairs; simulate and evaluate the current and candidate versions.

5「对 Coding Agent 说」Tell your Coding Agent报告Reports

>生成 baseline / candidate 报告，按同一 case 配对，列出差异及 unknown / 执行异常。Create a baseline / candidate report paired by case; include differences, unknowns and execution errors.

规划衔接：配置回写 → AB 平台 → 线上效果验证Planned handoff: configuration write-back → AB platform → live validation

Harness

数据驱动Data-driven目标衔接：仿真／测评固化 → 配置回写 → AB → 线上效果Target flow: simulation/evaluation codification → write-back → AB → live impact

11 / 15

本轮融合：测评输出判定与证据，可以写入问题标签；标签本身保持独立分类语义，可由人工或其他自动流程产生。流程统一是产品方向，不表示当前自动打标与测评 API 已合并。用户 Agent 驱动的验证资产生成与调优闭环属于 Q4 目标，自主改进为 2027 探索。Integration: evaluation yields verdicts and evidence and may write issue labels. Labels retain independent classification semantics and may be created manually or by other automation. A unified product workflow does not assert that current labeling and evaluation APIs are merged. User-Agent-driven asset generation and improvement are Q4 goals; autonomous improvement is a 2027 exploration.

本次评审补充：演示步骤是讲解路径，尚未在本次评审中实跑。单组/线上 cohort 由 Coding Agent 汇总；/reports 需要双版本、冻结 Suite、共同 seed manifest 与配置差异，不能直接当作通用汇总接口。Review clarification: This is an explanatory demo path, not a live-tested walkthrough. A single arm or online cohort is summarized by the Coding Agent. /reports requires two versions, a frozen Suite, a shared seed manifest and a configuration diff; it is not a generic summary endpoint.

思路 = Chang：旅程的核心是怎么探索；每张卡片是一句可直接交给 Coding Agent 的提示。目标与“子迭代 / AB 验证”口径 = Chang 的 epoch loop 说明。口腔为上量的主要使用项目之一（stats.md §4 任务分布）。
提示与接口（voice-agent-studio origin/main 3c9b914）：
1. 探索 Session：GET /api/sessions（sessions:read）。
2. 打标签：/api/auto-label-batches（auto_labels:write）；写回标签为 PATCH /api/sessions 的标签（annotations:write）。
3. 创建临时 Agent配置：Debug Session → pipeline-versions:promote 或 temporary 集合（debug:run）。
4. 执行仿真：/api/projects/{id}/simulation-runs（dispatcher:simulation）；执行测评：/api/projects/{id}/evaluation-runs（dispatcher:evaluation）。
5. 报告：/api/v2/projects/{id}/reports，baseline / candidate 对比（读取需 sessions:read，创建需 dispatcher:simulation）。
AB：只作为“推送到 AB 平台”一步，Coding Agent 不调用 AB；代码中没有 AB 的 API。
演示路径 · 需预检：按页面上的提示逐条交给 Coding Agent（口腔项目），现场展示每一步的返回结果和最终的 baseline / candidate 报告。

Idea = Chang: the core of each journey is how to explore; each card is one prompt to hand to a Coding Agent. The goal and the sub-iteration / AB-validation wording = Chang's epoch-loop description. Dental is one of the main Scale-up projects (stats.md §4 run distribution).
Prompts and APIs (voice-agent-studio origin/main 3c9b914):
1. Explore Sessions: GET /api/sessions (sessions:read).
2. Annotate: /api/auto-label-batches (auto_labels:write); writing annotations back is PATCH on /api/sessions (annotations:write).
3. Create a temporary Configuration: Debug Session → pipeline-versions:promote or the temporary collection (debug:run).
4. Run Simulations: /api/projects/{id}/simulation-runs (dispatcher:simulation); run Evaluations: /api/projects/{id}/evaluation-runs (dispatcher:evaluation).
5. Reports: /api/v2/projects/{id}/reports, baseline / candidate comparison (read needs sessions:read, create needs dispatcher:simulation).
AB: only the step “push to the AB platform”; the Coding Agent does not call AB, and the code has no AB API.
DEMO PATH · PREFLIGHT REQUIRED: hand each prompt on the slide to the Coding Agent in turn (Dental project) and show each step's result and the final baseline / candidate report.

自动触发需另行接入AUTO-TRIGGER NEEDS INTEGRATION

演示路径 · 需预检DEMO PATH · PREFLIGHT REQUIRED

使用旅程 · 持续测评Workflow · Continuous evaluation

从线上数据发现并追踪问题Find and track issues in live data

目标GOAL持续识别有效行为与问题，追踪变化，并沉淀可复用的验证资产Continuously identify effective behavior and issues, track changes and build reusable validation assets

1「对 Coding Agent 说」Tell your Coding Agent探索 SessionExplore Sessions

>在冠客汽车项目中筛选线上 Session，按业务规则找出有效对话与疑似 Bad Smell，保留证据供核实。Filter live Guanke Auto Sessions; use business rules to identify effective conversations and suspected Bad Smell with evidence for review.

2「对 Coding Agent 说」Tell your Coding Agent打标签Annotate

>按归纳出的规则创建自动打标批次，找出同类问题的 Session 并打上标签。Create an auto-label batch from the summarized rules to find Sessions with the same issue and annotate them.

3「对 Coding Agent 说」Tell your Coding Agent探索 Session · 执行测评Explore Sessions · Run Evaluations

>对带该标签的 Session 执行测评，判断每个 Turn 的回答是否满足预期。Run an Evaluation on the Sessions with that annotation, judging whether each Turn's reply meets expectations.

4「对 Coding Agent 说」Tell your Coding Agent执行测评 · 报告Run Evaluations · Reports

>对新导入的线上 Session 执行同一测评，由 Coding Agent 汇总结果与异常。Run the same Evaluation on newly imported live Sessions; have the Coding Agent summarize results and errors.

规划衔接：将测评结论关联到 AB 实验及线上指标Planned handoff: link Evaluation conclusions to AB experiments and live metrics

Harness

数据驱动Data-driven目标：全量自动评测 → 有效行为与问题识别 → 仿真／测评固化Target: full-data Auto Eval → useful patterns and issues → simulation/evaluation codification

12 / 15

本轮融合：测评输出判定与证据，可以写入问题标签；标签本身保持独立分类语义，可由人工或其他自动流程产生。流程统一是产品方向，不表示当前自动打标与测评 API 已合并。用户 Agent 驱动的验证资产生成与调优闭环属于 Q4 目标，自主改进为 2027 探索。Integration: evaluation yields verdicts and evidence and may write issue labels. Labels retain independent classification semantics and may be created manually or by other automation. A unified product workflow does not assert that current labeling and evaluation APIs are merged. User-Agent-driven asset generation and improvement are Q4 goals; autonomous improvement is a 2027 exploration.

本次评审补充：演示步骤是讲解路径，尚未在本次评审中实跑。单组/线上 cohort 由 Coding Agent 汇总；/reports 需要双版本、冻结 Suite、共同 seed manifest 与配置差异，不能直接当作通用汇总接口。Review clarification: This is an explanatory demo path, not a live-tested walkthrough. A single arm or online cohort is summarized by the Coding Agent. /reports requires two versions, a frozen Suite, a shared seed manifest and a configuration diff; it is not a generic summary endpoint.

思路 = Chang：旅程的核心是怎么探索；每张卡片是一句可直接交给 Coding Agent 的提示。目标与人群 = Chang 的说明（TO 线偏数据运营与数据挖掘，使用人群为技术、运营、TO 以及有数据 sense 的产品）；冠客汽车为线上 Session 最多的项目（stats.md §2）。“人工找 Bad Smell → AI 泛化成规则 → 机标”流程来自团队内部资料，未在代码中核对。
提示与接口（voice-agent-studio origin/main 3c9b914）：
1. 探索 Session：GET /api/sessions（sessions:read）。
2. 打标签：/api/auto-label-batches（auto_labels:write）。
3. 执行测评（Session cohort）：/api/projects/{id}/evaluation-runs（dispatcher:evaluation），筛选 Session 需 sessions:read。
4. 执行测评 + 报告：/api/projects/{id}/evaluation-runs（dispatcher:evaluation）；/api/v2/projects/{id}/reports（读取需 sessions:read，创建需 dispatcher:simulation）。新数据持续进入依赖 ConvoAI Importer 的持续导入。
AB：只作为“推送到 AB 平台”一步，Coding Agent 不调用 AB；代码中没有 AB 的 API。
演示路径 · 需预检：按页面上的提示逐条交给 Coding Agent（冠客汽车项目），现场展示归纳出的 Bad Smell 类型、打标结果与测评报告。

Idea = Chang: the core of each journey is how to explore; each card is one prompt to hand to a Coding Agent. Goal and audience = Chang's description (the TO line is data operations and mining, used by tech, ops, TO and product people with data sense); Guanke Auto is the project with the most live Sessions (stats.md §2). The “manual Bad Smell finding → AI generalizes into rules → machine labels” flow comes from internal team material and was not verified in code.
Prompts and APIs (voice-agent-studio origin/main 3c9b914):
1. Explore Sessions: GET /api/sessions (sessions:read).
2. Annotate: /api/auto-label-batches (auto_labels:write).
3. Run Evaluations (Session cohort): /api/projects/{id}/evaluation-runs (dispatcher:evaluation); filtering Sessions needs sessions:read.
4. Run Evaluations + report: /api/projects/{id}/evaluation-runs (dispatcher:evaluation); /api/v2/projects/{id}/reports (read needs sessions:read, create needs dispatcher:simulation). New data keeps arriving through the ConvoAI Importer's continuous import.
AB: only the step “push to the AB platform”; the Coding Agent does not call AB, and the code has no AB API.
DEMO PATH · PREFLIGHT REQUIRED: hand each prompt on the slide to the Coding Agent in turn (Guanke Auto project) and show the summarized Bad Smell types, the annotation results and the Evaluation report.

验证资产自动生成为目标VALIDATION ASSET GENERATION PLANNED

当前路径 + 目标能力CURRENT PATH + TARGET CAPABILITIES

使用旅程 · 0 → 1Workflow · First launch

从客户素材到首轮验证From customer material to initial validation

目标GOAL客户素材与业务规则 → 用户 Agent 生成仿真场景、测评定义与配置 → 首轮验证Customer material and business rules → user Agent drafts scenarios, evaluation definitions and configuration → initial validation

1「对 Coding Agent 说」Tell your Coding Agent导入Import

>通过 Import Job 导入客户的录音与通话，先预览，再确认写入。Import the customer's recordings and calls with an Import Job: preview first, then confirm the write.

2目标流程 · 当前需人工准备TARGET WORKFLOW · MANUAL PREPARATION TODAY生成验证资产 · 目标Generate validation assets · Planned

>目标：由用户 Agent 将业务规则、有效对话和问题生成仿真场景与测评定义；当前先人工整理，确认后准备已发布 Suite。Target: a user Agent generates simulation scenarios and evaluation definitions from rules, good conversations and issues. For now, prepare them manually, confirm and publish a Suite.

3「对 Coding Agent 说」Tell your Coding Agent创建临时 Agent配置Create a temporary Configuration

>确认可用配置基线与执行绑定；依据客户素材写出 Prompt，再准备可运行候选版本。Confirm a usable configuration baseline and execution bindings; draft the Prompt and prepare a runnable candidate.

4「对 Coding Agent 说」Tell your Coding Agent执行仿真 · 执行测评Run Simulations · Run Evaluations

>对该版本执行首轮仿真（纯文本），并对仿真结果执行测评。Run the first round of Simulation (text-only) on that version, then run an Evaluation on the results.

5「对 Coding Agent 说」Tell your Coding Agent报告 · 资产沉淀Report · Retain assets

>汇总首轮结果与异常，保留场景、规则和配置版本；将可通用部分整理为参数化模板，经权限与数据处理后跨项目复用。Summarize results and errors; retain scenario, rule and configuration versions. Parameterize reusable templates for sharing with appropriate permissions and data handling.

Harness

数据驱动Data-driven生成的是可验证的场景与规则草案，确认后执行和复用generate testable scenario and rule drafts; confirm, execute and reuse

13 / 15

本轮融合：测评输出判定与证据，可以写入问题标签；标签本身保持独立分类语义，可由人工或其他自动流程产生。流程统一是产品方向，不表示当前自动打标与测评 API 已合并。用户 Agent 驱动的验证资产生成与调优闭环属于 Q4 目标，自主改进为 2027 探索。Integration: evaluation yields verdicts and evidence and may write issue labels. Labels retain independent classification semantics and may be created manually or by other automation. A unified product workflow does not assert that current labeling and evaluation APIs are merged. User-Agent-driven asset generation and improvement are Q4 goals; autonomous improvement is a 2027 exploration.

本次评审补充：演示步骤是讲解路径，尚未在本次评审中实跑。单组/线上 cohort 由 Coding Agent 汇总；/reports 需要双版本、冻结 Suite、共同 seed manifest 与配置差异，不能直接当作通用汇总接口。Review clarification: This is an explanatory demo path, not a live-tested walkthrough. A single arm or online cohort is summarized by the Coding Agent. /reports requires two versions, a frozen Suite, a shared seed manifest and a configuration diff; it is not a generic summary endpoint.

思路 = Chang：旅程的核心是怎么探索；每张卡片是一句可直接交给 Coding Agent 的提示。目标 = Chang 确认的方向（核心是自动生成第一版测评集和仿真，Agent配置（含 Prompt）生成包含其中）；当前自动生成仍为目标，页面单独标为目标流程；执行前仍需人工确认与准备。
提示与接口（voice-agent-studio origin/main 3c9b914）：
1. 导入：/api/import-jobs（preview / stage / apply；imports:*）。
2. 探索 Session：GET /api/sessions（sessions:read）。
3. 创建临时 Agent配置：Debug Session → pipeline-versions:promote 或 temporary 集合（debug:run）。
4. 执行仿真：/api/projects/{id}/simulation-runs（dispatcher:simulation）；执行测评：/api/projects/{id}/evaluation-runs（dispatcher:evaluation）。
5. 报告：/api/v2/projects/{id}/reports（读取需 sessions:read，创建需 dispatcher:simulation）。
演示路径 · 需预检：按页面上的提示逐条交给 Coding Agent，从导入一批客户录音开始，现场展示首轮仿真、测评与报告。

Idea = Chang: the core of each journey is how to explore; each card is one prompt to hand to a Coding Agent. The goal = the direction Chang confirmed (the core is auto-generating the first Evaluation Suite and Simulation, with Configuration (incl. Prompt) generation included); automatic generation remains a target and is shown as a planned step; confirmation and preparation are still required.
Prompts and APIs (voice-agent-studio origin/main 3c9b914):
1. Import: /api/import-jobs (preview / stage / apply; imports:*).
2. Explore Sessions: GET /api/sessions (sessions:read).
3. Create a temporary Configuration: Debug Session → pipeline-versions:promote or the temporary collection (debug:run).
4. Run Simulations: /api/projects/{id}/simulation-runs (dispatcher:simulation); run Evaluations: /api/projects/{id}/evaluation-runs (dispatcher:evaluation).
5. Reports: /api/v2/projects/{id}/reports (read needs sessions:read, create needs dispatcher:simulation).
DEMO PATH · PREFLIGHT REQUIRED: hand each prompt on the slide to the Coding Agent in turn, starting from importing a batch of customer recordings, and show the first Simulation, Evaluation and report.

规划方向 · 非交付承诺PLANNED DIRECTION · NOT A COMMITMENT

能力规划Roadmap

Q4：规模化验证与调优Q4: validation and improvement at scale

把已有业务规则与优化方法，变成可持续执行、比较和复用的能力Make existing business rules and tuning methods continuously executable, comparable and reusable

规模与资产SCALE & ASSETS

大规模数据下的稳定验证与改进Reliable validation and improvement at scale

全量对话自动评测Auto Eval across live conversations
数据到达即进入测评，按场景与风险分层执行规则，识别有效行为与问题。Evaluate arriving data with rules selected by scenario and risk; identify effective behavior and issues.
仿真固化与测评固化Simulation and evaluation codification
有效对话与已核实 Bad Smell → 仿真场景与测评规则，即仿真固化与测评固化，持续扩展覆盖与回归。Generate scenarios and rules from good conversations and verified Bad Smell; expand coverage and regression tests.
通用验证资产Reusable validation assets
积累带版本的场景、规则与模板，参数化后按权限跨项目共享。Build versioned scenarios, rules and templates; parameterize and share across projects with access controls.
执行与数据底座Execution and data foundation
提升吞吐、稳定性和成本效率，支持大规模存储与冷热分层。Improve throughput, reliability and cost efficiency, with large-scale storage and hot/cold tiers.

让每批真实数据都能进入验证流程Bring every batch of real data into validation

用户 Agent 闭环USER-AGENT LOOP

打通线上与离线调优Connect offline and online improvement

API 与 Skill 补齐Complete APIs and Skills
补齐探索、配置、资产生成与执行能力，完善自助 API Key。Complete exploration, configuration, asset generation and execution capabilities; improve self-service API keys.
Agent 自助生成验证资产Agent-generated validation assets
让用户 Agent 依据业务规则和方法生成仿真 / 测评草案，确认后执行。Let user Agents draft simulation and evaluation assets from business rules and methods; confirm before execution.
离线验证与版本比较Offline validation and comparison
固定验证条件，比较模型、Prompt 与配置修改的收益和回退风险。Compare model, prompt and configuration changes under fixed conditions for gains and regressions.
线上调试与效果验证Live debugging and validation
配置回写、线上 Debug、AB 分组与对比，接回真实业务效果。Connect configuration write-back, live debugging, AB groups and comparisons to real outcomes.

由用户 Agent 驱动，Harness 提供执行与证据User Agents drive the loop; Harness supplies execution and evidence

能力扩展CAPABILITY EXPANSION

扩展仿真与测评的范围Expand simulation and evaluation

音频端到端End-to-end audio
覆盖等待、打断、互相抢话、噪声、读法和情绪表达，结合音频与事件证据。Cover waits, interruptions, overlap, noise, spoken rendering and emotion with audio and event evidence.
自定义脚本测评Custom scripted evaluation
在 Prompt 判定之外执行自定义规则与指标，保留版本、判定依据及运行边界。Run custom rules and metrics alongside prompt-based judgments, with versions, evidence and execution limits.

保持统一的结果、证据与比较方式Retain consistent results, evidence and comparison methods

2027 · 探索自主改进闭环2027 · Explore self-improvement loops

探索自主发现问题 → 提出修改 → 生成验证 → 比较收益 → 持续迭代；以验收门槛与发布控制为边界，不等同于自动上线。Explore autonomous discovery → changes → generated validation → measured gains → iteration, bounded by acceptance and release controls.

Harness14 / 15

术语统一：Auto Eval 的中文为“自动评测”。Case 转仿真／测评资产，就是总览图中的“仿真固化与测评固化”，不是另外一项平行能力；产出是可执行、可复用的仿真场景与测评规则。

Terminology: Auto Eval is “自动评测” in Chinese. Turning cases into simulation/evaluation assets is the same simulation and evaluation codification shown in the overview, not a separate capability.

2026-10-28 用户接受 review 的第 1、3、4 点并要求融入演示，不加入角色划分。Q4 从原七项能力重组为三个目标，保留音频、API/Skill、自助 Key、配置回写、AB、数据治理与线上调试、脚本测评；新增全量自动评测、Case 转仿真/测评、通用资产复用及用户 Agent 生成验证资产的目标表达。全量表示所有符合来源范围的对话进入测评流程，不意味着每条对话无差别运行所有规则。跨项目复用以参数化场景/规则/模板为单位，不默认共享客户原始会话。2027 自主改进为探索，不是 Q4 已实现能力或自动生产发布承诺。未据此认定任何具体 K2 案例具有普遍性，也未新增交付日期或负责人。

On 2026-10-28 the user accepted review points 1, 3 and 4 without role-based messaging. Three Q4 outcomes reorganize and retain the original audio, API/Skill, self-service key, write-back, AB, governance, debugging and scripted evaluation goals. Added goals cover full-data Auto Eval, case-to-validation assets, reusable templates and user-Agent generation. Full-data means all in-scope conversations enter evaluation, not every rule on every conversation. Share parameterized assets with permissions, not raw customer conversations by default. Autonomous improvement is a 2027 exploration, not shipped Q4 behavior or automatic production release. No K2 generality, owner or delivery date is asserted.

致谢Thank you

开始探索你的 Convo Agent 吧Start exploring your Convo Agent