diff --git a/CLAUDE.md b/CLAUDE.md index 4cc6e6b1..48ab9570 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -738,6 +738,272 @@ Android/iOS 两端同改,改动都在各自的 `AliyunBailianE2EHelper` 的 `h ⚠️ 官方标称 2.8s 同传延迟,那是模型为语义完整做的等待,配不掉。文本接上流式之后 提前的是「开始出字」的时间,音频总延迟不变。 +### 通话状态:`BB 8E` 是权威,`BB 02/03/04` 是噪声(2026-09-21 实测,**尚未据此改代码**) + +真机抓到的通话状态帧里,只有 `BB 8E` 是干净的,整份日志只有两种形态、末字节即状态: + +``` +bb 8e 07 00 02 01 02 → 通话中 (29 次) +bb 8e 07 00 02 01 03 → 已结束 (18 次) +``` + +它是 `queryCallState` 的应答,我们问一次它答一次,与真实状态始终一致。 + +而 `BB 02` / `BB 03` 会**成对、高频**地刷(间隔 68~260ms),带的是同一个电话号码 +(`bb 02 16 0f 01 2b 38 36 31 33…` 与 `bb 03 13 0f 01 2b 38 36 31 33…`,`2b 38 36…` 是 +ASCII `+8613…`)。更像是**通话列表的周期性上报**(类似 HFP 的 `+CLCC` 轮询), +而原生 `case 0x02 / 0x03` 把它们当成了通话起止事件。 + +实测后果:一次挂断之后 30 秒内 `DeviceHub.inCall` 翻转 10 多次,主界面那道 +「通话中禁用同传/面对面/音视频」的闸门跟着反复开关,用户点的时候正好落在某个 true +的瞬间就被拦,表现是「电话挂了,同传要等好几秒、点好几次才进得去」。 + +⚠️ **对照时间线可以确认锅在 `BB 02/03` 而不在 `BB 8E`**:22:21:37~22:22:52 与 +22:23:54~22:24:27 两段震荡期间,`BB 8E` 一直报「通话中」,直到 22:22:52 / 22:24:27 +才报结束——也就是说电话确实还没挂断完,是 `BB 02/03` 在乱报。 + +⚠️ 还有一个**不带号码**的短帧 `bb 02 08 01 01 02 03 02`,同样被判成 `startMakeCall`, +总是紧跟在挂断序列(`bb 04 03` / `bb d2` 的 stopCall ACK)之后冒出来。 + +**想改的话**(2026-09-21 讨论后决定暂不动,先留记录):通话状态只认 `BB 8E`; +`BB 02` 可以立即置 true(宁可早开门,通话翻译本就只在通话中用), +`BB 03/04` **不直接置 false**、只触发一次 `queryCallState()` 由 `BB 8E` 定论, +再加 3 秒兜底防止哪天 `BB 8E` 也换了命令号(见上文「命令号是 (产品, 固件版本) 的函数」)。 +⚠️ 这条链路是通话翻译的**准入判据**,判严的后果(通话中进不去)比现在的 +「挂断后要多等几秒」严重得多,动之前先想清楚失败方向。 + +### 排查这条链路时日志怎么拿(2026-09-21) + +⚠️ **`idevicesyslog` 会自己断**,而且断了之后进程还在、只是不再有输出(今晚断过两次), +`idevice_id -l` 返回空但 `devicectl` 仍能看到进程 —— 那是 libimobiledevice 与 +macOS 26+ 的 usbmuxd 不对付,杀掉残留的 `iproxy` 也救不回来。 + +**更可靠的路子是直接拉 App 自己写的日志文件**,完全不依赖 usbmuxd: + +```bash +# 先列出来看哪个文件在更新(Logger.initFileLog 写在 Documents/logs/) +xcrun devicectl device info files --device \ + --domain-type appDataContainer --domain-identifier com.yimai.eaimar | grep logs + +xcrun devicectl device copy from --device \ + --domain-type appDataContainer --domain-identifier com.yimai.eaimar \ + --source Documents/logs/app_2026-09-21_221314.log --destination /tmp/latest.log +``` + +⚠️ `copy from` **没有 `--username` 参数**(`info files` 有),带上会直接报 Unknown option。 +文件日志的级别门槛与 syslog 不同,`Logger.i` 也会落盘,比 syslog 那条路看得更全。 + +### iOS 上恒玄走的是 **GATT over BR/EDR**,不是 BLE(2026-09-21 真机定性) + +⚠️ **先记住这条,否则后面每一步推理都会跑偏。** iOS 侧 `bluetooth_manager` 用 CoreBluetooth +的 API(`CBCentralManager`/`CBPeripheral`),但底层链路是**经典蓝牙 BR/EDR 上的 GATT**: +连接靠 `registerForConnectionEvents` + `.peerConnected` 接管系统已配对的设备, +`retrieveConnectedPeripherals` 枚举的也是 BR/EDR GATT 设备。代码里 `enableBREDRConnectionEvents` +这个开关就是为它存在的。 + +**别把 BLE 那套模型套上来**,两处结论会直接错: + +| BLE 的说法 | 在 BR/EDR GATT 上 | +|---|---| +| connection interval 15~30ms,一个 interval 只发一包 | 不适用。BR/EDR 是 625µs 时隙轮询 | +| 84B × 50 包/秒 ≈ 34 kbps 已接近链路上限 | 差得远。ACL 带宽几百 kbps ~ 2Mbps | + +下行 84 字节 / 16~20ms 一包(50~62 包/秒)这个量级,**对这条链路完全不构成压力**—— +2026-09-21 压测实测 10164 包、`dropped: 0`。排查「音频不流畅」时不要往带宽方向猜。 + +### GATT 写入:控制命令与实时音频的三条铁律(2026-09-21 真机定性) + +通话翻译下行全部经 `BluetoothManager.write` 递给 CoreBluetooth,这条路上踩全了三个坑, +**每一个都不报错**: + +1. **写类型必须优先 `.withoutResponse`。** 原实现是 `contains(.write) ? .withResponse : .withoutResponse`, + 而特征通常两种写都支持(真机实测 `props: write=Y writeNoResp=Y`),于是恒选带应答。 + **ATT 是请求-响应协议**(这一点与底层是 BLE 还是 BR/EDR 无关),带应答写每包都要等对端 + ACK 才能发下一包,50+ 包/秒的实时流必然追不上,表现是音频越听越滞后、断断续续。 +2. **`canSendWriteWithoutResponse` 只能作观测,绝不能作门槛。** 它是 CoreBluetooth + **为 BLE 设计**的流控提示,在 BR/EDR GATT 上不对应真实链路能力——压测实测它 + **100% 为 false,同时 10164 包全部发出、零丢弃**。 + ⚠️ 拿它当闸门会**死锁**:为 false 时我们一个字节都不写,而 + `peripheralIsReady(toSendWriteWithoutResponse:)` 的触发前提恰恰是「你写了、被限流了」—— + 完全不写,回调就永远不来,canSend 再也翻不回 true。真机表现:整场通话 `sent` 恒定停在 10、 + `blocked` 停在 1(队列一次都没排空过),548 包只出去 11 个,**远端完全听不到译文**。 + 同一版里走无条件直发的控制命令(`AA 06`/`AA 09`/`queryCallState`)却全部成功——这组对比就是判据。 +3. **实时音频不需要在这一层再做队列。** 下行本来就是 `CallTranslationDownlink` 用 20ms 定时器 + **匀速**产出的,不存在突发。直接 `writeValue` 交给 CoreBluetooth 即可。 + +⚠️ **控制命令绝不能排队**。`AA 09`(查电量)/`AA 06`(查版本)/`queryCallState` 一旦排到积压的音频 +后面就再也发不出去,表象是「耳机连着但电量、版本全查不到」「耳机在通话中但 App 里通话翻译 +不让点」,**极易误判成固件换了命令号**(上文那条警告就是这么来的)。 +现在两个入口都是无条件直发,`writeRealtimeAudio` 仅用于区分统计口径。 +另外 `CallTranslationDownlink.stop()` 只清它自己那两条腿的队列,必要时还得调 +`BluetoothManager.flushRealtimeQueue()` 清 GATT 层的遗留积压。 + +⚠️ **排查这条链路时 `NSLog` 一个字都看不到**:release / profile 包里 `idevicesyslog` 只收得到 +`os_log`。`BesCallDownlink` / `BesCallPcmBridge` 原来全用 NSLog,于是下行是整条链路唯一的盲区, +只能靠猜。现在关键计数都走 `os_log`(`com.eaimar.bluetooth` 的 `write`/`downlink` 两个 category), +每 100 包一条队列长度 + 定时器真实间隔,会话结束一条收尾总账。 + +⚠️ **别去碰标准电池服务 `0x180F`**:Echo-one 上它是空壳(值恒为 `0%`,同一时刻私有 `BB 0A` +解出的是 L=90 R=86),而对它 `discoverCharacteristics` + `readValue` + `setNotifyValue` 之后, +**ATT 事务串行排队会把整条通道堵死**——实测下行 548 包只发出去 11 个,远端完全听不到译文。 +这台耳机的完整服务清单(2026-09-21 一次性枚举,常态下只发现私有服务那一个): +`180A`(设备信息)、`180F`(空壳电池)、`00001100-D102-11E1-9B23-00025B00AFAE`(私有,在用)、 +`66666666-…`、`86868686-…`。 + +### 通话翻译的 PCM 已下沉到原生:bluetooth_manager ⇄ azure_speech 直连(2026-09-20) + +苹果测试反馈「WiFi 下断断续续、长句基本不行」正是此前记录的技术债触发条件(Dart 链路每秒约 200 次跨界, +event loop 一被拖住原生就报 `AudioSendSlow 1011`)。现在通话翻译的双路 PCM **不再经过 Dart**: + +| 端 | 落点 | 传输 | +|---|---|---| +| 耳机插件 → 翻译插件(上行 A=本端麦 / B=对端) | `bluetooth_manager` 的 `CallPcmBridge.swift` / `BluetoothManager.callPcmTap`(kt) → `azure_speech` 的 `BesCallPcmBridge.swift/.kt` | iOS `NotificationCenter`(两插件一个是 CocoaPods 一个是 SwiftPM,**不能互相 import**);Android `compileOnly project(":bluetooth_manager")` 直接赋值静态槽 | +| 翻译插件 → 耳机插件(下行译文 PCM) | `pushAstTtsFrame`/`emitAstTtsFrame` 直接喂 `CallTranslationDownlink` | 同上 | + +- **按会话开关**:`TranslationController._tryEnableNativeCallBridge` 先开翻译端(`AstService.setNativeCallBridge`) + 再开设备端(`DeviceFeatures.callPcmNativeBridge`),任一端不支持就退回原来的 Dart 流路径(`openMic`/`pushExternalAudio`), + 功能不受影响。关的顺序相反。桥开着时原生不再把 micData/spkData 推给 Dart 的 audio EventChannel, + TTS 也只回 `final` 标记;`[CALL-BRIDGE] 原生直连桥统计` 这行 warn 日志在每次会话结束时打, + `dropped>0` 表示上行拥塞(原生那边 2s 积压即丢帧,取代原来整个会话被 1011 掀掉)。 +- **B 路说话人门限 `PeerSpeechGate` 已移植成 Swift/Kotlin**(`azure_speech` 内),阈值三处(Dart/Swift/Kotlin)相同,调要同调; + 门限开启时原生发 `peerGateOpened` 事件,Dart 以 warn 打 `[CALL-BRIDGE] B 路门限开启(原生)`——真机调阈值仍看这行。 +- **重连期间不丢音频**:AST helper 重建时帧留在桥的环形缓冲(A 路 3s,B 路因门限复位丢弃),`markReady` 后整段补推。 + ⚠️ iOS 通知名/键、Android 槽名两边各写一份,**改一处必须同改另一处**,改漏了不报错、只表现为「桥开着但一帧都到不了」。 +- 顺带改的:AST 重连预算由「3 次」改成「首次掉线起 60s 窗口内不限次、退避封顶 8s」(`_astReconnectWindow`), + 终止时按错误原文分类弹「网络连接失败」或「服务连接失败」(`_classifyAstFailure`),通话翻译页顶栏多了链路状态灯(`astLinkState`)。 + +### 音频链路继续下沉:录音落盘与 EMAI 会话(2026-09-21) + +**判据不是「业务逻辑 vs 展示逻辑」,是「这条链路有没有实时截止期」。** Dart 侧的 CPU +开销基本不是瓶颈(一帧 640 字节算个 RMS 是微秒级);真正的代价有三条,只在特定链路上成立: +平台通道每条消息要序列化 + 跳一次主线程;Dart 只有一个 isolate,音频回调与页面布局、GC +排同一个 event loop(`AudioSendSlow 1011` 就是这么来的);以及后台存活要求。 +没有实时截止期的状态机、接口编排、列表渲染搬原生只会把一份代码写成 Swift + Kotlin 两份。 + +⚠️ **「搬到原生就能扛住 iOS 锁屏」是错的**:iOS 挂起的是**整个进程**,原生和 Dart 一起停。 +锁屏后能不能活着只取决于 `UIBackgroundModes=audio` + 确实有音频在播 +(`BackgroundKeepalive` 干的事),与代码在哪一层无关。排查「锁屏断连」别往下沉的方向查。 + +#### 设备录音直接在原生落盘(`PcmWavRecorder`) + +| | 之前 | 现在 | +|---|---|---| +| 路径 | 原生解码 → EventChannel → Dart → `WavWriter`(RandomAccessFile 排队) | 原生解码完**就地写 WAV** | +| 跨界 | 单路 50 帧/s、通话双路 100 帧/s | 只剩 100ms 一条 `{bytes, level}` | +| 入口 | `session.openMic(...)` + 自己写盘 | `session.openFileRecorder(path:, routes:)` | + +- 契约在 `device_plugin_interface`:`DeviceFileRecorder` / `DeviceRecordStats`, + `DeviceSession.openFileRecorder` **默认返回 null**——其它厂商零改动,老原生自动退回 + Dart 落盘那条路(三个服务都保留了完整的退路分支)。 +- 实现是 `bluetooth_manager` 的 `PcmWavRecorder.swift` / `.kt`:**实例化,不是单例**。 + azure_speech 里那个 `RecordFile` 是单例且已被通话翻译的立体声录音占着,复用会互相踩。 +- **录音是额外的 sink,不拦截原有回调**(与 `CallPcmBridge` 不同):通话翻译开着桥时 + micData/spkData 不再上抛 Dart,但用户可能同时在录这通电话,拦了就录出空文件。 +- `StereoPairer` 的配对逻辑三处(Dart / Swift / Kotlin)各一份,金测试 + `test/stereo_pairer_test.dart` 定的是同一个行为。**写成「只留最后一帧对端」会把右声道 + 断续抽掉,转写出来只剩自己这一方**。 +- 顺带修掉现场录音的**相位问题**:Dart 原来用 `_alternate(audioStream)` 隔帧取主麦, + 靠的是「订阅时恰好对上相位」,中途订阅就永久取到前馈麦。原生早把两路标成了 + `mainMicData` / `ffMicData`,按类型取即可。映射由 `test/devices/bes_recorder_sources_test.dart` + 钉死——那几个串是跨语言契约,写错不报任何错,只表现为「录音起来了但一个字节都没写进去」。 +- 断连兜底:设备断开时原生自己收尾回填 WAV 头。不做这一步、进程随后被杀,就留下一个 + `data` 长度为 0 的文件——有几十 MB,播放器却认为它是空的。 + +#### EMAI 耳机唤醒整条会话下沉(`BailianDialogClient`) + +耳机按键唤醒那条链路的百炼 WebSocket 与音频收发都搬到了原生,**Dart 只收文本与状态**: + +``` +bluetooth_manager(AiPcmBridge) → azure_speech(BesAiPcmBridge 攒 100ms 包) + → BailianDialogClient(WS) → 合成 PCM 原路回 AiAudioDownlink +``` + +- 按会话开关,**两端都要开**:`BailianNativeDialog.start()`(azure_speech 端)+ + `DeviceFeatures.aiPcmNativeBridge`(bluetooth_manager 端)。任一端不支持就整条退回 + Dart 的 `BailianMultimodalService`。**不能只开一半**:只开翻译端一帧都收不到, + 只开设备端 PCM 被摘走没人接,两种都表现为「唤醒了但 AI 不说话」。 +- **手机麦那条链路不动**(EMAI 页按住说话),音频本来就在 Dart 侧采集,没有下沉的意义。 +- **指令解析刻意留在 Dart**:`extra_info.tool_calls` 那些坑有一批真实 payload 的金测试守着 + (`test/assistant_directive_test.dart`),解析结果又要落进 GetStorage 和「代办」页。 + 原生把 `extra_info` 原样转成字符串带上来,走同一个解析器。 +- **门控(AI 播报时把上行换成等长静音)只有逐帧替换在原生**,什么时候开关门仍在 + `DeviceAiSessionService`:那套有兜底计时、非致命错误要关门、纯指令帧收尾也要关门的分支 + 是来之不易的(门卡开着 = 会话活着但是聋的)。 +- Dart 拿不到下行 PCM,但它要靠「首包到达」开门控、靠累计字节数估播完时刻, + 所以原生按 100ms 报一次累计 `bytes`;设备上行有声按 500ms 报一次 `micActivity` 续空闲计时。 + ⚠️ `micActivity` **不能当成 `speechStarted`**:那个事件在会话服务里会丢弃待播队列 + (「真人插话」的语义),混用会让 AI 每说一句就被自己的回声打断一次。 +- ⚠️ **`AiAudioDownlink.flush()` 的语义是「清空队列」不是「把尾包发出去」** + (`outQueue.clear()`)。名字容易读反,在「本轮播报结束」处调它等于把每句回复的尾音切掉。 + 真正要丢弃待播的只有真人插话,走 `_spk?.discardPending()`。 +- 握手参数、事件名映射、心跳间隔在 Dart / Swift / Kotlin 三份实现里逐条对齐; + 配置解析(apiKey / appId / workspaceId)与 MCP 工具名**只在 Dart 维护一份**, + 由 `start` 的参数传给原生(`test/bailian_native_parity_test.dart` 守着)。 + 对不上不报任何错,只表现为「用耳机问和在 App 里问,助手的能力不一样」。 + +#### EMAI 下行播放两端都走原生(`PcmSpeakerPlayer`) + +Dart 的 `PcmStreamPlayer` 现在**默认走原生播放器**(`azure_speech/pcm_player`, +iOS `AVAudioEngine` / Android `AudioTrack`,方法名 start/feed/end/stop/dispose 逐字一致), +只有老包没有这条通道时才退回 just_audio。 + +- iOS 早就是原生的,那边是**被迫**:AVPlayer 放不了 just_audio 代理服务给的 + 「长度未知的 chunked WAV」,只发一次 `Range: bytes=0-1` 的嗅探就判定不可播、 + 回 `-11850 Operation Stopped`,一次都没真正起播。 +- Android 是**为了少绕一圈**:ExoPlayer 不发那个嗅探请求,所以老路一直能用,但一条 + 实时语音流要先经过 just_audio 内部起的**本地代理 HTTP 服务**(chunked 编码 → + localhost TCP → ExoPlayer)。直接喂 AudioTrack 更短、起播更快。 +- ⚠️ **播完的判定不能是「队列排空」**:Android 侧按 `playbackHeadPosition` 对比已写入 + 帧数(iOS 用 `.dataPlayedBack` 而不是 `.dataRendered`,同一个道理)。按排空报完成, + 最后一两百毫秒还在缓冲里没出声,调用方据此回 `LocalRespondingEnded`,尾音被切掉。 +- ⚠️ AudioTrack 的 `AudioAttributes` 与 `emai_audio_session.dart` 里声明的 + `AndroidAudioAttributes` 对齐(`USAGE_MEDIA` + `CONTENT_TYPE_SPEECH`)。 + 改这两行会改变输出路由(比如 A2DP 还走不走)。 +- 原生路径**只支持单声道**(两端都是 mono 链路),`channels != 1` 时自动退回 just_audio, + 而不是把交织 PCM 当单声道播成两倍语速。当前两个调用点(EMAI 24k、车载香薰 16k)都是单声道。 + +#### 顺带清掉的每帧 Rx + +电平原来是**每帧** `level.value = rms(pcm)`(50~100 次/s),而读它的只有 100ms 一次的 +采样定时器——每 5 次里有 4 次是唤醒了 Obx 又被下一帧覆盖。现在一律:每帧只更新**纯字段的 +窗口峰值**,由采样定时器发布。涉及 `device_recording_service` / `device_call_recording_service` / +`pcm_stream_player` / `mic_pcm_streamer` / `meeting_record_controller`。 +`meeting_record_controller.rms` 在 meeting 模块里**一个 widget 消费方都没有**。 + +⚠️ `WavWriter.write` 的 `onWritten` 回调**已删除**:它把 UI 挂在文件写入队列的尾巴上, +队列一堵界面跟着停、写失败波形整段消失。电平在调用方直接从那块 PCM 上算,与落盘无关。 + +⚠️ **CocoaPods 的 `source_files` 是在 `pod install` 时展开通配的**:给 +`bluetooth_manager`(CocoaPods 包)新增 .swift 文件后必须 `cd ios && pod install`, +否则报 `Cannot find 'XXX' in scope`,看着像 import 问题。azure_speech 是 SwiftPM 包,不受影响。 + +### 苹果测试那一轮一并修掉的四处(2026-09-20) + +- **iOS 字幕小窗被别的 App 的画中画顶掉后再也不回来**:iOS 全系统只允许一个画中画。`SubtitlePipController` 现在区分 + 「我们主动停」和「被系统停」,后者退避重试(3/6/12s,5 次)并在 App 下次 `willResignActive` 时再拉一次; + 状态经 `native_plugin` 通道的 `subtitlePipStateChanged` 回传(`started`/`stopped`/`stoppedBySystem`/`failed`), + `isFloatingWindowEnabled` 以此为准(原来只在进页面查一次,被顶掉后点开关是「关」,要点两下)。 + + ⚠️ **两条真机实测出来的前置条件,都会让画中画"静默失败"**(2026-09-21,`SubtitlePip.diagnostics()` 是为此加的, + 起不来时先问它,别靠猜): + 1. **音频会话类别必须支持后台音频**(`.playback` / `.playAndRecord` / `.multiRoute`)。类别是 `.ambient` / `.soloAmbient` + 时 `startPictureInPicture()` **连 `failedToStartPictureInPicture` 都不回调**,表现就是"点了什么都没发生"。 + 所以 `subtitlePipStart` 仍要配会话,但**只在当前类别撑不起画中画时才动**(`prepareAudioSessionForPip`)—— + 无条件改成 `.playback` 会把通话/多媒体翻译正在跑的 `.playAndRecord` 录音链路踩掉,而 `.playAndRecord` 本身就满足要求。 + 2. **App 在前台时不会真正进小窗**:`possible=true` 但 `active` 恒为 false,并回调一次 `failed`。 + 要等 App 切到后台才由 `canStartPictureInPictureAutomaticallyFromInline` / 我们的 `willResignActive` 重试拉起来。 + 这是 sample-buffer 画中画的正常行为,**不是 bug**;真机实测:前台 `failed` → 切后台 `started` → 小窗浮在别的 App 上。 +- **语音纪要「列表显示总结完成、第一次点进去是空的」**:`OverviewTab` 原来在 `initState` 快照一次,`SummaryTab` 只听 `personnelNumberModifications`; + 现在两个 tab 都跟着 `meetingDetails` / `summaryNumberModifications` 重画。`MeetingDetailsController` 的 `ever(tasktype)` 只取 + service 里**非空**的值(原来会用空值把刚拉回来的总结抹掉),`_everType` 门槛删除;进行中状态每 6s 自己拉一次 + `refreshFromServer()`、切后台回来补拉一次、占位图上有「刷新」按钮——不再依赖那个会被首页退出清空的全局轮询。 + `MeetingTaskService.addTask` 改成返回是否受理,没受理不再把页面切成「等待转写」。 +- **Ask AI 抽屉的 X 点不到**:`SafeArea` 挪到受约束的 Container 里面、顶部留 `viewPadding.top`、不再重复减 `viewInsets.bottom` + (GetX 已经减过一次)。原来抽屉比屏幕只矮 18pt,标题行整个埋在状态栏里。 +- **EMAI 耳机对话锁屏易断、蜂窝易断连**:`DeviceAiSessionService` 的空闲超时 5s→15s 且设备上行有声(RMS≥600)也算活动; + 百炼 WS 掉线先重连 2 次(1s/3s)再拆会话;设备断连给 3s 观察期;iOS 会话期间用 `BackgroundKeepalive` + (just_audio 循环 `assets/audio/silence_1s.wav`,`playback+mixWithOthers`)撑住 `audio` 后台模式——Android 有前台服务,iOS 原来什么都没有。 + ### 面对面 / 同声翻译:延迟构成与已做的优化(2026-09-03) 这两个模式**不走端到端**(端到端只用在 call 模式),是 ASR → HTTP 机器翻译 → TTS @@ -858,18 +1124,33 @@ Android/iOS 两端同改,改动都在各自的 `AliyunBailianE2EHelper` 的 `h `test/version_compare_test.dart` **在当前工作区里并不存在**(version_update_service.dart 有大量未提交改动),所以 OTA 这条没有复用它,版本比较是本地实现的。 -### ⚠️ 私有协议命令号会随固件版本变(2026-09-02 实测) +### ⚠️ 私有协议命令号是 (产品, 固件版本) 的函数,不是全局常量 -**下面那张命令表是在 Echo-one 固件 0.0.2 上扫出来的,固件 0.0.3 上 `AA 09` 已经不回 -`BB 0A` 了**——连 `BB 89` ack 都没有,静默无回包,表现就是「电量那块不显示」。 -同一次连接里 `AA 06`(固件版本)仍然正常回 `BB 86`,所以不是链路问题,是命令号变了。 +**每个产品有自己的固件线,指令集也各是各的**,下面那张表只属于 **Echo-one**。 +换产品要重新拿厂商文档 + 实测,换固件版本要重扫;任何情况下都别把这张表 +当成「恒玄耳机通用协议」跨产品套用。 -结论:**换固件就要重扫命令**,别把这张表当稳定契约。重扫用 -`BesBluetoothService.to.probeCommands(from: 0x01, to: 0x20)`(当初 `AA 09` 就是在这段扫出来的; -`0x21~0x7F` 已确认整段无回包,2026-09-01 扫过 69 个命令,不用再浪费 28 秒)。 +同一个 `AA 09`(查电量)在 Echo-one 自己的固件线上就反复变过: + +| Echo-one 固件 | `AA 09` → `BB 0A` | 实测日期 | +|---|---|---| +| 0.0.2 | ✅ 有效(这张表就是在它上面扫出来的) | 2026-09-01 | +| 0.0.3 | ❌ 静默无回包,连 `BB 89` ack 都没有 | 2026-09-02 | +| 0.0.8 | ✅ 又有效了,解出 L=90 R=86 | 2026-09-21 | + +重扫用 `BesBluetoothService.to.probeCommands(from: 0x01, to: 0x20)`(当初 `AA 09` 就是在这段 +扫出来的;`0x21~0x7F` 已确认整段无回包,2026-09-01 扫过 69 个命令,不用再浪费 28 秒)。 `0x5x`/`0x6x` 会切工作模式(进通话/录音/AI),任何情况下都别扫。 -### 恒玄耳机电量:`AA 09` 请求 / `BB 0A` 上报(2026-09-01 真机实测定稿,**仅适用固件 0.0.2**) +⚠️ **「`AA 06` 成功、`AA 09` 无回包」不足以判定命令号失效**——2026-09-21 就这么误判过一次。 +握手里这两条是**紧挨着连发**的(中间没有延时),而当时 iOS 的 BLE 写入刚加了无应答写的 +流控队列:第一条赶上 `canSendWriteWithoutResponse` 为真、直发成功;第二条紧随其后, +CoreBluetooth 刚吃下一个写请求、`canSend` 转假,于是进了队列等 `peripheralIsReady` 回调—— +而那个回调在队列满住后不再触发(见下文「控制命令不能和实时音频共用发送队列」), +`AA 09` 就永远卡在队列里。**表象与「固件不认这个命令号」一模一样。** +判定命令号之前,先确认同一条链路上**后发**的命令也能出去。 + +### 恒玄耳机电量:`AA 09` 请求 / `BB 0A` 上报(**仅适用 Echo-one**,2026-09-01 定稿于固件 0.0.2,2026-09-21 在 0.0.8 复测仍成立) 厂商协议 3.1.10「耳机响应 APP 获取状态(响应命令 20)或者耳机状态变化主动上报」。 响应帧 `BB 0A` 与文档一致,但**请求码不是文档写的那个**——实测是 **`AA 09`**。 diff --git a/apps/client/assets/audio/silence_1s.wav b/apps/client/assets/audio/silence_1s.wav new file mode 100644 index 00000000..3f1a79e7 Binary files /dev/null and b/apps/client/assets/audio/silence_1s.wav differ diff --git a/apps/client/ios/Runner.xcodeproj/project.pbxproj b/apps/client/ios/Runner.xcodeproj/project.pbxproj index cbbc4b21..5062f484 100644 --- a/apps/client/ios/Runner.xcodeproj/project.pbxproj +++ b/apps/client/ios/Runner.xcodeproj/project.pbxproj @@ -3,7 +3,7 @@ archiveVersion = 1; classes = { }; - objectVersion = 60; + objectVersion = 54; objects = { /* Begin PBXBuildFile section */ @@ -338,7 +338,7 @@ ); mainGroup = 97C146E51CF9000F007C117D; packageReferences = ( - 781AD8BC2B33823900A9FFBB /* XCLocalSwiftPackageReference "Flutter/ephemeral/Packages/FlutterGeneratedPluginSwiftPackage" */, + 781AD8BC2B33823900A9FFBB /* XCLocalSwiftPackageReference "FlutterGeneratedPluginSwiftPackage" */, ); productRefGroup = 97C146EF1CF9000F007C117D /* Products */; projectDirPath = ""; @@ -389,14 +389,10 @@ inputFileListPaths = ( "${PODS_ROOT}/Target Support Files/Pods-Runner/Pods-Runner-resources-${CONFIGURATION}-input-files.xcfilelist", ); - inputPaths = ( - ); name = "[CP] Copy Pods Resources"; outputFileListPaths = ( "${PODS_ROOT}/Target Support Files/Pods-Runner/Pods-Runner-resources-${CONFIGURATION}-output-files.xcfilelist", ); - outputPaths = ( - ); runOnlyForDeploymentPostprocessing = 0; shellPath = /bin/sh; shellScript = "\"${PODS_ROOT}/Target Support Files/Pods-Runner/Pods-Runner-resources.sh\"\n"; @@ -441,14 +437,10 @@ inputFileListPaths = ( "${PODS_ROOT}/Target Support Files/Pods-Runner/Pods-Runner-frameworks-${CONFIGURATION}-input-files.xcfilelist", ); - inputPaths = ( - ); name = "[CP] Embed Pods Frameworks"; outputFileListPaths = ( "${PODS_ROOT}/Target Support Files/Pods-Runner/Pods-Runner-frameworks-${CONFIGURATION}-output-files.xcfilelist", ); - outputPaths = ( - ); runOnlyForDeploymentPostprocessing = 0; shellPath = /bin/sh; shellScript = "\"${PODS_ROOT}/Target Support Files/Pods-Runner/Pods-Runner-frameworks.sh\"\n"; @@ -649,7 +641,7 @@ CODE_SIGN_ENTITLEMENTS = Runner/RunnerProfile.entitlements; CODE_SIGN_IDENTITY = "Apple Development"; CODE_SIGN_STYLE = Automatic; - CURRENT_PROJECT_VERSION = 15; + CURRENT_PROJECT_VERSION = 16; DEVELOPMENT_TEAM = B8AJQ7ACU7; ENABLE_BITCODE = NO; INFOPLIST_FILE = Runner/Info.plist; @@ -843,7 +835,7 @@ CODE_SIGN_ENTITLEMENTS = Runner/Runner.entitlements; CODE_SIGN_IDENTITY = "Apple Development"; CODE_SIGN_STYLE = Automatic; - CURRENT_PROJECT_VERSION = 15; + CURRENT_PROJECT_VERSION = 16; DEVELOPMENT_TEAM = B8AJQ7ACU7; ENABLE_BITCODE = NO; INFOPLIST_FILE = Runner/Info.plist; @@ -877,7 +869,7 @@ CODE_SIGN_ENTITLEMENTS = Runner/Runner.entitlements; CODE_SIGN_IDENTITY = "Apple Development"; CODE_SIGN_STYLE = Automatic; - CURRENT_PROJECT_VERSION = 15; + CURRENT_PROJECT_VERSION = 16; DEVELOPMENT_TEAM = B8AJQ7ACU7; ENABLE_BITCODE = NO; INFOPLIST_FILE = Runner/Info.plist; @@ -996,7 +988,7 @@ /* End XCConfigurationList section */ /* Begin XCLocalSwiftPackageReference section */ - 781AD8BC2B33823900A9FFBB /* XCLocalSwiftPackageReference "Flutter/ephemeral/Packages/FlutterGeneratedPluginSwiftPackage" */ = { + 781AD8BC2B33823900A9FFBB /* XCLocalSwiftPackageReference "FlutterGeneratedPluginSwiftPackage" */ = { isa = XCLocalSwiftPackageReference; relativePath = Flutter/ephemeral/Packages/FlutterGeneratedPluginSwiftPackage; }; @@ -1005,7 +997,7 @@ /* Begin XCSwiftPackageProductDependency section */ 78A3181F2AECB46A00862997 /* FlutterGeneratedPluginSwiftPackage */ = { isa = XCSwiftPackageProductDependency; - package = 781AD8BC2B33823900A9FFBB /* XCLocalSwiftPackageReference "Flutter/ephemeral/Packages/FlutterGeneratedPluginSwiftPackage" */; + package = 781AD8BC2B33823900A9FFBB /* XCLocalSwiftPackageReference "FlutterGeneratedPluginSwiftPackage" */; productName = FlutterGeneratedPluginSwiftPackage; }; /* End XCSwiftPackageProductDependency section */ diff --git a/apps/client/lib/core/translations/language/ar_sa.dart b/apps/client/lib/core/translations/language/ar_sa.dart index 20f8bce3..ec53db6c 100644 --- a/apps/client/lib/core/translations/language/ar_sa.dart +++ b/apps/client/lib/core/translations/language/ar_sa.dart @@ -2074,4 +2074,6 @@ const Map arSA = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'الترجمة قيد التشغيل. أوقفها قبل تغيير اللغة.', + 'meetingRefreshNow': 'تحديث', + 'pipTakenOverByOtherApp': 'تم استبدال نافذة الترجمة بصورة داخل صورة من تطبيق آخر. ارجع إلى هذا التطبيق ثم غادره مرة أخرى لاستعادتها تلقائيًا', }; diff --git a/apps/client/lib/core/translations/language/bg_bg.dart b/apps/client/lib/core/translations/language/bg_bg.dart index 9e2deb5b..55baf8c9 100644 --- a/apps/client/lib/core/translations/language/bg_bg.dart +++ b/apps/client/lib/core/translations/language/bg_bg.dart @@ -2137,4 +2137,6 @@ const Map bgBG = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Преводът е в ход. Спрете го, преди да смените езика.', + 'meetingRefreshNow': 'Опресни', + 'pipTakenOverByOtherApp': 'Прозорецът със субтитри беше изместен от картина в картина на друго приложение. Върнете се в това приложение и излезте отново, за да се възстанови автоматично', }; diff --git a/apps/client/lib/core/translations/language/cs_cz.dart b/apps/client/lib/core/translations/language/cs_cz.dart index 0a67417a..fa5c339f 100644 --- a/apps/client/lib/core/translations/language/cs_cz.dart +++ b/apps/client/lib/core/translations/language/cs_cz.dart @@ -2105,4 +2105,6 @@ const Map csCZ = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Překlad běží. Před změnou jazyka jej zastavte.', + 'meetingRefreshNow': 'Obnovit', + 'pipTakenOverByOtherApp': 'Okno titulků bylo nahrazeno obrazem v obraze jiné aplikace. Vraťte se do této aplikace a znovu ji opusťte, obnoví se automaticky', }; diff --git a/apps/client/lib/core/translations/language/da_dk.dart b/apps/client/lib/core/translations/language/da_dk.dart index b7e47976..c7ff446c 100644 --- a/apps/client/lib/core/translations/language/da_dk.dart +++ b/apps/client/lib/core/translations/language/da_dk.dart @@ -2107,4 +2107,6 @@ const Map daDK = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Oversættelse kører. Stop den, før du skifter sprog.', + 'meetingRefreshNow': 'Opdater', + 'pipTakenOverByOtherApp': 'Undertekstvinduet blev overtaget af en anden apps billede-i-billede. Gå tilbage til denne app og forlad den igen, så gendannes det automatisk', }; diff --git a/apps/client/lib/core/translations/language/de_de.dart b/apps/client/lib/core/translations/language/de_de.dart index 0f817a4d..49ff88f0 100644 --- a/apps/client/lib/core/translations/language/de_de.dart +++ b/apps/client/lib/core/translations/language/de_de.dart @@ -2174,4 +2174,6 @@ const Map deDE = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Übersetzung läuft. Stoppen Sie sie, bevor Sie die Sprache wechseln.', + 'meetingRefreshNow': 'Aktualisieren', + 'pipTakenOverByOtherApp': 'Das Untertitelfenster wurde vom Bild-im-Bild einer anderen App verdrängt. Kehren Sie zu dieser App zurück und verlassen Sie sie erneut, dann wird es automatisch wiederhergestellt', }; diff --git a/apps/client/lib/core/translations/language/el_gr.dart b/apps/client/lib/core/translations/language/el_gr.dart index 6eb53c52..5fc79f7e 100644 --- a/apps/client/lib/core/translations/language/el_gr.dart +++ b/apps/client/lib/core/translations/language/el_gr.dart @@ -2141,4 +2141,6 @@ const Map elGR = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Η μετάφραση εκτελείται. Σταματήστε την πριν αλλάξετε γλώσσα.', + 'meetingRefreshNow': 'Ανανέωση', + 'pipTakenOverByOtherApp': 'Το παράθυρο υποτίτλων αντικαταστάθηκε από το εικόνα-σε-εικόνα άλλης εφαρμογής. Επιστρέψτε σε αυτήν την εφαρμογή και βγείτε ξανά για αυτόματη επαναφορά', }; diff --git a/apps/client/lib/core/translations/language/en_us.dart b/apps/client/lib/core/translations/language/en_us.dart index 2290d9ff..6358b714 100644 --- a/apps/client/lib/core/translations/language/en_us.dart +++ b/apps/client/lib/core/translations/language/en_us.dart @@ -2130,4 +2130,6 @@ const Map enUS = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Translation is running. Stop it before changing languages.', + 'meetingRefreshNow': 'Refresh', + 'pipTakenOverByOtherApp': 'The subtitle window was taken over by another app\'s Picture in Picture. Return to this app and leave it again to restore it automatically', }; diff --git a/apps/client/lib/core/translations/language/es_es.dart b/apps/client/lib/core/translations/language/es_es.dart index 89d59672..9fba8acf 100644 --- a/apps/client/lib/core/translations/language/es_es.dart +++ b/apps/client/lib/core/translations/language/es_es.dart @@ -2168,4 +2168,6 @@ const Map esES = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'La traducción está en curso. Deténla antes de cambiar de idioma.', + 'meetingRefreshNow': 'Actualizar', + 'pipTakenOverByOtherApp': 'La ventana de subtítulos fue reemplazada por la imagen en imagen de otra app. Vuelve a esta app y sal de nuevo para restaurarla automáticamente', }; diff --git a/apps/client/lib/core/translations/language/et_ee.dart b/apps/client/lib/core/translations/language/et_ee.dart index 39acc8ee..0c38c555 100644 --- a/apps/client/lib/core/translations/language/et_ee.dart +++ b/apps/client/lib/core/translations/language/et_ee.dart @@ -2101,4 +2101,6 @@ const Map etEE = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Tõlkimine käib. Peata see enne keele vahetamist.', + 'meetingRefreshNow': 'Värskenda', + 'pipTakenOverByOtherApp': 'Subtiitriakna võttis üle teise rakenduse pilt-pildis. Naase sellesse rakendusse ja lahku uuesti, see taastatakse automaatselt', }; diff --git a/apps/client/lib/core/translations/language/fa_ir.dart b/apps/client/lib/core/translations/language/fa_ir.dart index 25ce6131..1322f779 100644 --- a/apps/client/lib/core/translations/language/fa_ir.dart +++ b/apps/client/lib/core/translations/language/fa_ir.dart @@ -2092,4 +2092,6 @@ const Map faIR = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'ترجمه در حال اجراست. قبل از تغییر زبان آن را متوقف کنید.', + 'meetingRefreshNow': 'تازه‌سازی', + 'pipTakenOverByOtherApp': 'پنجره زیرنویس توسط تصویر در تصویر برنامه دیگری جایگزین شد. به این برنامه برگردید و دوباره خارج شوید تا خودکار بازیابی شود', }; diff --git a/apps/client/lib/core/translations/language/fi_fi.dart b/apps/client/lib/core/translations/language/fi_fi.dart index 13471d58..0adacc6b 100644 --- a/apps/client/lib/core/translations/language/fi_fi.dart +++ b/apps/client/lib/core/translations/language/fi_fi.dart @@ -2111,4 +2111,6 @@ const Map fiFI = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Käännös on käynnissä. Pysäytä se ennen kielen vaihtamista.', + 'meetingRefreshNow': 'Päivitä', + 'pipTakenOverByOtherApp': 'Toinen sovellus otti tekstitysikkunan kuva kuvassa -tilan haltuunsa. Palaa tähän sovellukseen ja poistu uudelleen, niin se palautuu automaattisesti', }; diff --git a/apps/client/lib/core/translations/language/fr_fr.dart b/apps/client/lib/core/translations/language/fr_fr.dart index 8d9c5f36..6c9e7851 100644 --- a/apps/client/lib/core/translations/language/fr_fr.dart +++ b/apps/client/lib/core/translations/language/fr_fr.dart @@ -2187,4 +2187,6 @@ const Map frFR = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Traduction en cours. Arrêtez-la avant de changer de langue.', + 'meetingRefreshNow': 'Actualiser', + 'pipTakenOverByOtherApp': 'La fenêtre de sous-titres a été remplacée par l\'image dans l\'image d\'une autre app. Revenez dans cette app puis quittez-la à nouveau pour la restaurer automatiquement', }; diff --git a/apps/client/lib/core/translations/language/hi_in.dart b/apps/client/lib/core/translations/language/hi_in.dart index 963feac2..59395e8b 100644 --- a/apps/client/lib/core/translations/language/hi_in.dart +++ b/apps/client/lib/core/translations/language/hi_in.dart @@ -2110,4 +2110,6 @@ const Map hiIN = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'अनुवाद चल रहा है। भाषा बदलने से पहले इसे रोकें।', + 'meetingRefreshNow': 'रीफ़्रेश', + 'pipTakenOverByOtherApp': 'सबटाइटल विंडो को किसी अन्य ऐप के पिक्चर-इन-पिक्चर ने हटा दिया। इस ऐप में वापस आएँ और फिर बाहर जाएँ, यह अपने आप वापस आ जाएगी', }; diff --git a/apps/client/lib/core/translations/language/hr_hr.dart b/apps/client/lib/core/translations/language/hr_hr.dart index a0abb9ca..cf29f4b2 100644 --- a/apps/client/lib/core/translations/language/hr_hr.dart +++ b/apps/client/lib/core/translations/language/hr_hr.dart @@ -2126,4 +2126,6 @@ const Map hrHR = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Prijevod je u tijeku. Zaustavite ga prije promjene jezika.', + 'meetingRefreshNow': 'Osvježi', + 'pipTakenOverByOtherApp': 'Prozor s titlovima preuzela je slika u slici druge aplikacije. Vratite se u ovu aplikaciju i ponovno je napustite kako bi se automatski vratio', }; diff --git a/apps/client/lib/core/translations/language/hu_hu.dart b/apps/client/lib/core/translations/language/hu_hu.dart index a2e65875..c4a55d63 100644 --- a/apps/client/lib/core/translations/language/hu_hu.dart +++ b/apps/client/lib/core/translations/language/hu_hu.dart @@ -2129,4 +2129,6 @@ const Map huHU = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'A fordítás fut. Nyelvváltás előtt állítsa le.', + 'meetingRefreshNow': 'Frissítés', + 'pipTakenOverByOtherApp': 'A feliratablakot egy másik alkalmazás kép a képben funkciója kiszorította. Térjen vissza ebbe az alkalmazásba, majd lépjen ki újra, és automatikusan helyreáll', }; diff --git a/apps/client/lib/core/translations/language/id_id.dart b/apps/client/lib/core/translations/language/id_id.dart index 21b4a0c3..6f2d6eb9 100644 --- a/apps/client/lib/core/translations/language/id_id.dart +++ b/apps/client/lib/core/translations/language/id_id.dart @@ -2110,4 +2110,6 @@ const Map idID = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Terjemahan sedang berjalan. Hentikan dulu sebelum mengganti bahasa.', + 'meetingRefreshNow': 'Segarkan', + 'pipTakenOverByOtherApp': 'Jendela subtitle diambil alih oleh gambar-dalam-gambar aplikasi lain. Kembali ke aplikasi ini lalu keluar lagi agar pulih otomatis', }; diff --git a/apps/client/lib/core/translations/language/is_is.dart b/apps/client/lib/core/translations/language/is_is.dart index 37d9a57f..f6330ac5 100644 --- a/apps/client/lib/core/translations/language/is_is.dart +++ b/apps/client/lib/core/translations/language/is_is.dart @@ -2107,4 +2107,6 @@ const Map isIS = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Þýðing er í gangi. Stöðvaðu hana áður en þú skiptir um tungumál.', + 'meetingRefreshNow': 'Endurnýja', + 'pipTakenOverByOtherApp': 'Textagluggi var tekinn yfir af mynd í mynd í öðru forriti. Farðu aftur í þetta forrit og út aftur til að endurheimta hann sjálfkrafa', }; diff --git a/apps/client/lib/core/translations/language/it_it.dart b/apps/client/lib/core/translations/language/it_it.dart index a9abc503..ddfcd140 100644 --- a/apps/client/lib/core/translations/language/it_it.dart +++ b/apps/client/lib/core/translations/language/it_it.dart @@ -2148,4 +2148,6 @@ const Map itIT = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Traduzione in corso. Interrompila prima di cambiare lingua.', + 'meetingRefreshNow': 'Aggiorna', + 'pipTakenOverByOtherApp': 'La finestra dei sottotitoli è stata sostituita dal Picture in Picture di un\'altra app. Torna in questa app ed esci di nuovo per ripristinarla automaticamente', }; diff --git a/apps/client/lib/core/translations/language/ja_jp.dart b/apps/client/lib/core/translations/language/ja_jp.dart index d337373d..64c20ca5 100644 --- a/apps/client/lib/core/translations/language/ja_jp.dart +++ b/apps/client/lib/core/translations/language/ja_jp.dart @@ -1969,4 +1969,6 @@ const Map jaJP = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': '翻訳中です。言語を切り替える前に翻訳を停止してください。', + 'meetingRefreshNow': '更新', + 'pipTakenOverByOtherApp': '字幕ウィンドウが他のアプリのピクチャ・イン・ピクチャに置き換えられました。このアプリに戻ってから再度切り替えると自動的に復元されます', }; diff --git a/apps/client/lib/core/translations/language/ko_kr.dart b/apps/client/lib/core/translations/language/ko_kr.dart index f53ccf96..dfff6250 100644 --- a/apps/client/lib/core/translations/language/ko_kr.dart +++ b/apps/client/lib/core/translations/language/ko_kr.dart @@ -1962,4 +1962,6 @@ const Map koKR = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': '번역이 진행 중입니다. 언어를 바꾸려면 먼저 번역을 중지하세요.', + 'meetingRefreshNow': '새로고침', + 'pipTakenOverByOtherApp': '자막 창이 다른 앱의 PIP에 의해 대체되었습니다. 이 앱으로 돌아왔다가 다시 나가면 자동으로 복원됩니다', }; diff --git a/apps/client/lib/core/translations/language/lt_lt.dart b/apps/client/lib/core/translations/language/lt_lt.dart index 179c1822..f7a0b63b 100644 --- a/apps/client/lib/core/translations/language/lt_lt.dart +++ b/apps/client/lib/core/translations/language/lt_lt.dart @@ -2125,4 +2125,6 @@ const Map ltLT = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Vyksta vertimas. Sustabdykite jį prieš keisdami kalbą.', + 'meetingRefreshNow': 'Atnaujinti', + 'pipTakenOverByOtherApp': 'Subtitrų langą perėmė kitos programos vaizdas vaizde. Grįžkite į šią programą ir vėl išeikite – jis bus atkurtas automatiškai', }; diff --git a/apps/client/lib/core/translations/language/lv_lv.dart b/apps/client/lib/core/translations/language/lv_lv.dart index 37420ac0..9dd9d9d6 100644 --- a/apps/client/lib/core/translations/language/lv_lv.dart +++ b/apps/client/lib/core/translations/language/lv_lv.dart @@ -2130,4 +2130,6 @@ const Map lvLV = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Notiek tulkošana. Apturiet to pirms valodas maiņas.', + 'meetingRefreshNow': 'Atsvaidzināt', + 'pipTakenOverByOtherApp': 'Subtitru logu pārņēma citas lietotnes attēls attēlā. Atgriezieties šajā lietotnē un izejiet vēlreiz, lai to atjaunotu automātiski', }; diff --git a/apps/client/lib/core/translations/language/ms_my.dart b/apps/client/lib/core/translations/language/ms_my.dart index 731f5cc8..0d603fa5 100644 --- a/apps/client/lib/core/translations/language/ms_my.dart +++ b/apps/client/lib/core/translations/language/ms_my.dart @@ -2114,4 +2114,6 @@ const Map msMY = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Terjemahan sedang berjalan. Hentikan dahulu sebelum menukar bahasa.', + 'meetingRefreshNow': 'Segar semula', + 'pipTakenOverByOtherApp': 'Tetingkap sari kata diambil alih oleh gambar-dalam-gambar aplikasi lain. Kembali ke aplikasi ini dan keluar semula untuk memulihkannya secara automatik', }; diff --git a/apps/client/lib/core/translations/language/nb_no.dart b/apps/client/lib/core/translations/language/nb_no.dart index 340864d8..eae05da5 100644 --- a/apps/client/lib/core/translations/language/nb_no.dart +++ b/apps/client/lib/core/translations/language/nb_no.dart @@ -2108,4 +2108,6 @@ const Map nbNO = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Oversettelse pågår. Stopp den før du bytter språk.', + 'meetingRefreshNow': 'Oppdater', + 'pipTakenOverByOtherApp': 'Undertekstvinduet ble overtatt av en annen apps bilde-i-bilde. Gå tilbake til denne appen og forlat den igjen, så gjenopprettes det automatisk', }; diff --git a/apps/client/lib/core/translations/language/nl_nl.dart b/apps/client/lib/core/translations/language/nl_nl.dart index 95bb289f..6f738066 100644 --- a/apps/client/lib/core/translations/language/nl_nl.dart +++ b/apps/client/lib/core/translations/language/nl_nl.dart @@ -2162,4 +2162,6 @@ const Map nlNL = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Vertaling loopt. Stop deze voordat u van taal wisselt.', + 'meetingRefreshNow': 'Vernieuwen', + 'pipTakenOverByOtherApp': 'Het ondertitelvenster is overgenomen door beeld-in-beeld van een andere app. Ga terug naar deze app en verlaat hem opnieuw om het automatisch te herstellen', }; diff --git a/apps/client/lib/core/translations/language/pl_pl.dart b/apps/client/lib/core/translations/language/pl_pl.dart index aacffca7..a5c5adbd 100644 --- a/apps/client/lib/core/translations/language/pl_pl.dart +++ b/apps/client/lib/core/translations/language/pl_pl.dart @@ -2136,4 +2136,6 @@ const Map plPL = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Tłumaczenie trwa. Zatrzymaj je przed zmianą języka.', + 'meetingRefreshNow': 'Odśwież', + 'pipTakenOverByOtherApp': 'Okno napisów zostało zastąpione przez obraz w obrazie innej aplikacji. Wróć do tej aplikacji i opuść ją ponownie, aby przywrócić je automatycznie', }; diff --git a/apps/client/lib/core/translations/language/pt_pt.dart b/apps/client/lib/core/translations/language/pt_pt.dart index 6bdbca2e..af5b61fb 100644 --- a/apps/client/lib/core/translations/language/pt_pt.dart +++ b/apps/client/lib/core/translations/language/pt_pt.dart @@ -2174,4 +2174,6 @@ const Map ptPT = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'A tradução está em curso. Pare-a antes de mudar de idioma.', + 'meetingRefreshNow': 'Atualizar', + 'pipTakenOverByOtherApp': 'A janela de legendas foi substituída pelo picture-in-picture de outra app. Volte a esta app e saia novamente para a restaurar automaticamente', }; diff --git a/apps/client/lib/core/translations/language/ro_ro.dart b/apps/client/lib/core/translations/language/ro_ro.dart index 5d00788c..abf93fa6 100644 --- a/apps/client/lib/core/translations/language/ro_ro.dart +++ b/apps/client/lib/core/translations/language/ro_ro.dart @@ -2138,4 +2138,6 @@ const Map roRO = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Traducerea este în curs. Opriți-o înainte de a schimba limba.', + 'meetingRefreshNow': 'Reîmprospătează', + 'pipTakenOverByOtherApp': 'Fereastra de subtitrări a fost preluată de imaginea în imagine a altei aplicații. Reveniți în această aplicație și ieșiți din nou pentru a o restaura automat', }; diff --git a/apps/client/lib/core/translations/language/ru_ru.dart b/apps/client/lib/core/translations/language/ru_ru.dart index 3da0b223..445069ac 100644 --- a/apps/client/lib/core/translations/language/ru_ru.dart +++ b/apps/client/lib/core/translations/language/ru_ru.dart @@ -2149,4 +2149,6 @@ const Map ruRU = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Идёт перевод. Остановите его, прежде чем менять язык.', + 'meetingRefreshNow': 'Обновить', + 'pipTakenOverByOtherApp': 'Окно субтитров было вытеснено режимом «картинка в картинке» другого приложения. Вернитесь в это приложение и снова выйдите — оно восстановится автоматически', }; diff --git a/apps/client/lib/core/translations/language/sk_sk.dart b/apps/client/lib/core/translations/language/sk_sk.dart index 7addfeb6..1e236e74 100644 --- a/apps/client/lib/core/translations/language/sk_sk.dart +++ b/apps/client/lib/core/translations/language/sk_sk.dart @@ -2119,4 +2119,6 @@ const Map skSK = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Preklad beží. Pred zmenou jazyka ho zastavte.', + 'meetingRefreshNow': 'Obnoviť', + 'pipTakenOverByOtherApp': 'Okno titulkov nahradil obraz v obraze inej aplikácie. Vráťte sa do tejto aplikácie a znova ju opustite, obnoví sa automaticky', }; diff --git a/apps/client/lib/core/translations/language/sl_si.dart b/apps/client/lib/core/translations/language/sl_si.dart index 1ae9f208..99792b9c 100644 --- a/apps/client/lib/core/translations/language/sl_si.dart +++ b/apps/client/lib/core/translations/language/sl_si.dart @@ -2122,4 +2122,6 @@ const Map slSI = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Prevajanje poteka. Ustavite ga, preden zamenjate jezik.', + 'meetingRefreshNow': 'Osveži', + 'pipTakenOverByOtherApp': 'Okno s podnapisi je prevzela slika v sliki druge aplikacije. Vrnite se v to aplikacijo in jo znova zapustite, da se samodejno obnovi', }; diff --git a/apps/client/lib/core/translations/language/sr_rs.dart b/apps/client/lib/core/translations/language/sr_rs.dart index 729c5bdb..b357c2de 100644 --- a/apps/client/lib/core/translations/language/sr_rs.dart +++ b/apps/client/lib/core/translations/language/sr_rs.dart @@ -2118,4 +2118,6 @@ const Map srRS = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Prevod je u toku. Zaustavite ga pre promene jezika.', + 'meetingRefreshNow': 'Освежи', + 'pipTakenOverByOtherApp': 'Прозор са титловима је преузела слика у слици друге апликације. Вратите се у ову апликацију и поново је напустите да би се аутоматски вратио', }; diff --git a/apps/client/lib/core/translations/language/sv_se.dart b/apps/client/lib/core/translations/language/sv_se.dart index e88e88b2..7b4b94fb 100644 --- a/apps/client/lib/core/translations/language/sv_se.dart +++ b/apps/client/lib/core/translations/language/sv_se.dart @@ -2114,4 +2114,6 @@ const Map svSE = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Översättning pågår. Stoppa den innan du byter språk.', + 'meetingRefreshNow': 'Uppdatera', + 'pipTakenOverByOtherApp': 'Undertextfönstret togs över av en annan apps bild-i-bild. Gå tillbaka till den här appen och lämna den igen så återställs det automatiskt', }; diff --git a/apps/client/lib/core/translations/language/th_th.dart b/apps/client/lib/core/translations/language/th_th.dart index f3ac8bcd..40396647 100644 --- a/apps/client/lib/core/translations/language/th_th.dart +++ b/apps/client/lib/core/translations/language/th_th.dart @@ -2067,4 +2067,6 @@ const Map thTH = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'กำลังแปลอยู่ โปรดหยุดการแปลก่อนเปลี่ยนภาษา', + 'meetingRefreshNow': 'รีเฟรช', + 'pipTakenOverByOtherApp': 'หน้าต่างคำบรรยายถูกแทนที่ด้วยภาพซ้อนภาพของแอปอื่น กลับมาที่แอปนี้แล้วออกอีกครั้งเพื่อกู้คืนโดยอัตโนมัติ', }; diff --git a/apps/client/lib/core/translations/language/tl_ph.dart b/apps/client/lib/core/translations/language/tl_ph.dart index 524d6dad..584ab985 100644 --- a/apps/client/lib/core/translations/language/tl_ph.dart +++ b/apps/client/lib/core/translations/language/tl_ph.dart @@ -2152,4 +2152,6 @@ const Map tlPH = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Tumatakbo ang pagsasalin. Ihinto muna ito bago magpalit ng wika.', + 'meetingRefreshNow': 'I-refresh', + 'pipTakenOverByOtherApp': 'Napalitan ang subtitle window ng picture-in-picture ng ibang app. Bumalik sa app na ito at lumabas muli para awtomatiko itong maibalik', }; diff --git a/apps/client/lib/core/translations/language/tr_tr.dart b/apps/client/lib/core/translations/language/tr_tr.dart index b3ebf80e..54297ccb 100644 --- a/apps/client/lib/core/translations/language/tr_tr.dart +++ b/apps/client/lib/core/translations/language/tr_tr.dart @@ -2108,4 +2108,6 @@ const Map trTR = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Çeviri devam ediyor. Dili değiştirmeden önce çeviriyi durdurun.', + 'meetingRefreshNow': 'Yenile', + 'pipTakenOverByOtherApp': 'Altyazı penceresi başka bir uygulamanın resim içinde resim özelliği tarafından devralındı. Bu uygulamaya dönüp tekrar çıkarsanız otomatik olarak geri gelir', }; diff --git a/apps/client/lib/core/translations/language/uk_ua.dart b/apps/client/lib/core/translations/language/uk_ua.dart index eea573c1..ab096e44 100644 --- a/apps/client/lib/core/translations/language/uk_ua.dart +++ b/apps/client/lib/core/translations/language/uk_ua.dart @@ -2126,4 +2126,6 @@ const Map ukUA = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Триває переклад. Зупиніть його, перш ніж змінювати мову.', + 'meetingRefreshNow': 'Оновити', + 'pipTakenOverByOtherApp': 'Вікно субтитрів витіснив режим «картинка в картинці» іншого застосунку. Поверніться в цей застосунок і знову вийдіть — воно відновиться автоматично', }; diff --git a/apps/client/lib/core/translations/language/vi_vn.dart b/apps/client/lib/core/translations/language/vi_vn.dart index 880bf858..0ea0b2bd 100644 --- a/apps/client/lib/core/translations/language/vi_vn.dart +++ b/apps/client/lib/core/translations/language/vi_vn.dart @@ -2086,4 +2086,6 @@ const Map viVN = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': 'Đang dịch. Hãy dừng dịch trước khi đổi ngôn ngữ.', + 'meetingRefreshNow': 'Làm mới', + 'pipTakenOverByOtherApp': 'Cửa sổ phụ đề đã bị ảnh-trong-ảnh của ứng dụng khác chiếm chỗ. Quay lại ứng dụng này rồi thoát ra lần nữa để tự động khôi phục', }; diff --git a/apps/client/lib/core/translations/language/zh_cn.dart b/apps/client/lib/core/translations/language/zh_cn.dart index 27f9dc5c..37feebee 100644 --- a/apps/client/lib/core/translations/language/zh_cn.dart +++ b/apps/client/lib/core/translations/language/zh_cn.dart @@ -1942,4 +1942,6 @@ const Map zhCN = { 'vipExpiringSoon': '会员还有 @days 天到期,续费后可继续使用翻译与会议纪要', 'computeLow': '算力不多了,用完后将无法继续使用', 'stopTranslationBeforeLanguageChange': '翻译进行中,请先停止翻译再切换语言', + 'meetingRefreshNow': '刷新', + 'pipTakenOverByOtherApp': '字幕小窗被其它应用的画中画占用了,回到本应用再切出去会自动恢复', }; diff --git a/apps/client/lib/core/translations/language/zh_hk.dart b/apps/client/lib/core/translations/language/zh_hk.dart index 013ab86f..587f7d0b 100644 --- a/apps/client/lib/core/translations/language/zh_hk.dart +++ b/apps/client/lib/core/translations/language/zh_hk.dart @@ -1939,4 +1939,6 @@ const Map zhHK = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': '翻譯進行中,請先停止翻譯再切換語言', + 'meetingRefreshNow': '重新整理', + 'pipTakenOverByOtherApp': '字幕小窗被其他應用程式的畫中畫佔用了,返回本應用程式再切出去會自動恢復', }; diff --git a/apps/client/lib/core/translations/language/zh_tw.dart b/apps/client/lib/core/translations/language/zh_tw.dart index e9171034..9ad08901 100644 --- a/apps/client/lib/core/translations/language/zh_tw.dart +++ b/apps/client/lib/core/translations/language/zh_tw.dart @@ -1939,4 +1939,6 @@ const Map zhTW = { 'vipExpiringSoon': 'Membership expires in @days day(s). Renew to keep using translation and meeting notes.', 'computeLow': 'Running low on compute. You won\'t be able to continue once it runs out.', 'stopTranslationBeforeLanguageChange': '翻譯進行中,請先停止翻譯再切換語言', + 'meetingRefreshNow': '重新整理', + 'pipTakenOverByOtherApp': '字幕小窗被其他應用程式的子母畫面佔用了,返回本應用程式再切出去會自動恢復', }; diff --git a/apps/client/lib/data/services/ast_service.dart b/apps/client/lib/data/services/ast_service.dart index b406b176..28b182d1 100644 --- a/apps/client/lib/data/services/ast_service.dart +++ b/apps/client/lib/data/services/ast_service.dart @@ -56,6 +56,15 @@ abstract class AstService { /// [pcm] 16k mono 16bit PCM little-endian Future pushExternalAudio({required String leg, required Uint8List pcm}); + /// 通话翻译 PCM **原生直连桥**(翻译插件这一端)。开着时设备双路 PCM 在原生层 + /// 直接进 AST,译文 PCM 直接回设备;[pushExternalAudio] 与 [astTtsFrames] 的 PCM + /// 载荷都不再走 Dart([astTtsFrames] 只剩 final 标记)。 + /// 原生不支持返回 false,调用方退回 Dart 路径。 + Future setNativeCallBridge(bool enabled); + + /// 桥的计数(两路帧数 / 丢帧 / 门限状态),会话结束时打进日志用。 + Future> nativeCallBridgeStats(); + /// AST 流水线 / E2E 服务合成的 TTS PCM 帧流。每帧包含 leg('A'|'B') + pcm。 /// 调用方负责把它转发给设备(如 [Jielihome.feedTranslatedAudio])。 Stream get astTtsFrames; diff --git a/apps/client/lib/data/services/bailian_multimodal_service.dart b/apps/client/lib/data/services/bailian_multimodal_service.dart index 64c8246e..61cf74e9 100644 --- a/apps/client/lib/data/services/bailian_multimodal_service.dart +++ b/apps/client/lib/data/services/bailian_multimodal_service.dart @@ -148,22 +148,30 @@ class BailianMultimodalService extends GetxService { final Rx state = BailianDialogState.idle.obs; final RxBool isConnected = false.obs; - String get _apiKey => AppConfig.cred('ast_alibaba', 'app_key', 'ALIBABA_OPENSPEECH_APP_KEY') ?? ''; - - String get _workspaceId { - final o = _workspaceIdOverride?.trim() ?? ''; + /// 下面三个解析是 **static**,因为原生客户端(BailianNativeDialog)要用同一套取值: + /// 握手参数在两份实现里必须一模一样,否则「用耳机唤醒」和「在 App 里按住说话」 + /// 会连到不同的百炼应用上,而且两边都不报错。 + static String resolveApiKey() => + AppConfig.cred('ast_alibaba', 'app_key', 'ALIBABA_OPENSPEECH_APP_KEY') ?? ''; + + static String resolveWorkspaceId([String? override]) { + final o = override?.trim() ?? ''; if (o.isNotEmpty) return o; final v = (AppConfig.cred('sts_bailian', 'workspace_id', 'ALIBABA_BAILIAN_WORKSPACE_ID') ?? '').trim(); return v.isEmpty ? _defaultWorkspaceId : v; } - String get _appId { - final o = _appIdOverride?.trim() ?? ''; + static String resolveAppId([String? override]) { + final o = override?.trim() ?? ''; if (o.isNotEmpty) return o; final v = (AppConfig.cred('sts_bailian', 'app_id', 'ALIBABA_BAILIAN_APP_ID') ?? '').trim(); return v.isEmpty ? _defaultAppId : v; } + String get _apiKey => resolveApiKey(); + String get _workspaceId => resolveWorkspaceId(_workspaceIdOverride); + String get _appId => resolveAppId(_appIdOverride); + /// 只有 API Key 是硬要求:workspace / app id 有兜底默认值。 /// Key 缺失时不要尝试连接——服务端会先回 task-started 再立刻 task-failed, /// 日志上看起来像"连上了又掉",容易误判成网络问题。 @@ -293,7 +301,7 @@ class BailianMultimodalService extends GetxService { // ⚠️ 放的是 user_getmcptoken 签发的**短时效专用令牌**, // 绝不能换成登录 JWT —— 这条路要经过百炼(第三方)。 if (_hasMcpToken(mcpToken)) - for (final tool in _mcpAuthTools) + for (final tool in mcpAuthTools) tool: {'auth_token': mcpToken}, if (enableWebSearch) 'extra_config': {'enable_web_search': true}, @@ -580,7 +588,7 @@ class BailianMultimodalService extends GetxService { /// ⚠️ 服务端新增需要身份的工具时,这里要同步加一行,否则那个工具永远 /// 拿不到令牌、永远返回「身份校验失败」,而且客户端不会有任何报错。 /// 服务端工具清单见 apps/services/modules/mcp/tool_*.go。 - static const List _mcpAuthTools = [ + static const List mcpAuthTools = [ 'search_meeting_notes', 'get_memory_items', 'get_memory_stats', diff --git a/apps/client/lib/data/services/bailian_native_dialog.dart b/apps/client/lib/data/services/bailian_native_dialog.dart new file mode 100644 index 00000000..1333b8f2 --- /dev/null +++ b/apps/client/lib/data/services/bailian_native_dialog.dart @@ -0,0 +1,324 @@ +import 'dart:async'; +import 'dart:convert'; +import 'dart:io' show Platform; + +import 'package:flutter/foundation.dart' show kIsWeb; +import 'package:flutter/services.dart'; + +import '../../core/utils/logger.dart'; +import '../models/assistant_directive.dart'; +import 'bailian_multimodal_service.dart'; + +/// 耳机唤醒那条 EMAI 链路的**原生**百炼客户端句柄。 +/// +/// 真正的 WebSocket 与音频收发在 `azure_speech` 插件里 +/// (`BailianDialogClient.swift` / `.kt`);上行 PCM 由 `bluetooth_manager` 直接 +/// 交过去,合成音频原路回耳机,**一个字节都不经过 Dart**。这里只做三件事: +/// 起停、把原生事件翻译成 [BailianEvent]、把门控开关推下去。 +/// +/// ## 为什么保留 Dart 版 +/// [BailianMultimodalService] 仍然服务两种情况: +/// 1. **手机麦那条链路**(EMAI 页按住说话)——音频本来就在 Dart 侧采集,没有下沉的意义; +/// 2. **退路**——插件是老版本(`MissingPluginException`)、或设备不是恒玄(AI PCM 桥 +/// 只存在于 bluetooth_manager)时,整条链路退回 Dart,功能不受影响。 +/// +/// ## ⚠️ 这不修「锁屏断连」 +/// iOS 挂起的是整个进程,原生和 Dart 一起停。锁屏能不能活着取决于 +/// `UIBackgroundModes=audio` + 确实有音频在播(`BackgroundKeepalive` 干的事), +/// 与代码在哪一层无关。本类解决的是 event loop 抖动与跨界开销。 +class BailianNativeDialog { + BailianNativeDialog({String? appId, String? workspaceId}) + : _appIdOverride = appId, + _workspaceIdOverride = workspaceId; + + static const String _tag = 'BailianNative'; + + static const MethodChannel _channel = + MethodChannel('azure_speech/bailian_dialog'); + static const EventChannel _events = + EventChannel('azure_speech/bailian_dialog_events'); + + final String? _appIdOverride; + final String? _workspaceIdOverride; + + final _eventCtl = StreamController.broadcast(); + final _downBytesCtl = StreamController.broadcast(); + final _micActivityCtl = StreamController.broadcast(); + + StreamSubscription? _sub; + bool _running = false; + + /// 本次会话解出来的指令记到哪个来源名下(与 Dart 版同义) + String directiveSource = 'earphone'; + + /// 业务事件流。与 [BailianMultimodalService.events] 同一套 [BailianEvent], + /// 所以 `DeviceAiSessionService` 的事件处理一行都不用改。 + Stream get events => _eventCtl.stream; + + /// 下行已写入的**累计**字节数(约 100ms 一条)。 + /// + /// 原生直连之后 Dart 拿不到下行 PCM,但它要靠「首包到达」开门控、靠累计字节数 + /// 估算播完时刻。这两件事都只依赖长度,所以只报数字。 + Stream get downstreamBytes => _downBytesCtl.stream; + + /// 设备上行有声(原生按 500ms 节流上报)。 + /// + /// ⚠️ **不能把它当成 `speechStarted`**:那个事件在会话服务里会丢弃待播队列、 + /// 取消播完计时(「真人插话」的语义)。这里只是给空闲计时续命, + /// 混在一起会让 AI 每说一句就被自己的回声打断一次。 + Stream get micActivity => _micActivityCtl.stream; + + bool get isRunning => _running; + + bool get isConfigured => + BailianMultimodalService.resolveApiKey().trim().isNotEmpty; + + String get configHint => isConfigured ? '' : '未配置 ALIBABA_OPENSPEECH_APP_KEY'; + + /// 这台机器/这个包能不能用原生链路。 + /// + /// 老包没有这个方法通道 → `MissingPluginException` → 返回 false,调用方退回 Dart。 + /// Web 直接排除(压根没有原生插件)。 + static Future isAvailable() async { + if (kIsWeb || !(Platform.isIOS || Platform.isAndroid)) return false; + try { + await _channel.invokeMethod('stats'); + return true; + } on MissingPluginException { + return false; + } catch (e) { + Logger.w(_tag, '原生百炼客户端探测失败(退回 Dart): $e'); + return false; + } + } + + /// 建立会话。参数与 [BailianMultimodalService.start] 一一对应。 + Future start({ + String mode = 'duplex', + String? voice, + String? userId, + bool enableWebSearch = true, + int downstreamRate = 16000, + String source = 'earphone', + String? mcpToken, + }) async { + if (!isConfigured) { + Logger.e(_tag, '配置不全,无法启动会话:$configHint'); + _eventCtl.add(BailianEvent(BailianEventType.error, + text: configHint, code: 'NOT_CONFIGURED', fatal: true)); + return false; + } + await stop(); + directiveSource = source; + _sub = _events.receiveBroadcastStream().listen( + _onNativeEvent, + onError: (e) { + Logger.e(_tag, '原生事件流错误: $e'); + _eventCtl.add( + BailianEvent(BailianEventType.error, text: '$e', fatal: true)); + }, + ); + try { + final ok = await _channel.invokeMethod('start', { + 'apiKey': BailianMultimodalService.resolveApiKey(), + 'appId': BailianMultimodalService.resolveAppId(_appIdOverride), + 'workspaceId': + BailianMultimodalService.resolveWorkspaceId(_workspaceIdOverride), + 'mode': mode, + if (voice != null && voice.trim().isNotEmpty) 'voice': voice, + 'userId': userId ?? '', + 'enableWebSearch': enableWebSearch, + 'downstreamRate': downstreamRate, + if (mcpToken != null && mcpToken.trim().isNotEmpty) 'mcpToken': mcpToken, + // 工具名只在 Dart 维护一份:服务端加工具时改那一处就够了 + 'mcpAuthTools': BailianMultimodalService.mcpAuthTools, + }); + _running = ok == true; + if (!_running) await _cancelSub(); + return _running; + } on MissingPluginException { + Logger.w(_tag, '原生不支持,退回 Dart 链路'); + await _cancelSub(); + return false; + } catch (e) { + Logger.e(_tag, '启动原生会话失败: $e'); + await _cancelSub(); + return false; + } + } + + Future stop() async { + if (!_running) { + await _cancelSub(); + return; + } + _running = false; + try { + await _channel.invokeMethod('stop'); + } catch (e) { + Logger.w(_tag, '停止原生会话失败: $e'); + } + await _cancelSub(); + } + + /// AI 是否正在耳机里播报。播报期间原生把上行换成等长静音。 + /// + /// **什么时候开关门由 Dart 决定**(`DeviceAiSessionService` 里那套带兜底计时、 + /// 非致命错误也要关门的逻辑),这里只是把结论推下去。 + Future setGated(bool gated) => _invoke('setGated', {'gated': gated}); + + Future notifyLocalRespondingStarted() => + _invoke('localRespondingStarted'); + + Future notifyLocalRespondingEnded() => _invoke('localRespondingEnded'); + + Future interrupt() => _invoke('interrupt'); + + Future beginSpeech() => _invoke('beginSpeech'); + + Future endSpeech() => _invoke('endSpeech'); + + Future cancelSpeech() => _invoke('cancelSpeech'); + + Future sendText(String text, {bool speakDirectly = false}) => + _invoke('sendText', {'text': text, 'speakDirectly': speakDirectly}); + + /// 诊断:上/下行字节、出包、丢包、门控状态。桥开着却没声音时先问它。 + Future> stats() async { + try { + final raw = await _channel.invokeMethod('stats'); + return raw == null ? const {} : Map.from(raw); + } catch (e) { + return {'error': '$e'}; + } + } + + Future _invoke(String method, [Map? args]) async { + if (!_running) return; + try { + await _channel.invokeMethod(method, args); + } catch (e) { + Logger.w(_tag, '$method 失败: $e'); + } + } + + Future _cancelSub() async { + await _sub?.cancel(); + _sub = null; + } + + // ---------- 事件翻译 ---------- + + void _onNativeEvent(dynamic raw) { + if (raw is! Map) return; + final name = raw['event']?.toString() ?? ''; + final text = raw['text']?.toString() ?? ''; + final isFinal = raw['isFinal'] == true; + + switch (name) { + case 'started': + _emit(const BailianEvent(BailianEventType.started)); + break; + case 'stateChanged': + final s = raw['state']?.toString() ?? ''; + _emit(BailianEvent(BailianEventType.stateChanged, state: _mapState(s))); + break; + case 'speechStarted': + _emit(const BailianEvent(BailianEventType.speechStarted)); + break; + case 'speechContent': + _emit(BailianEvent(BailianEventType.speechContent, + text: text, isFinal: isFinal)); + break; + case 'speechEnded': + _emit(BailianEvent(BailianEventType.speechEnded, + text: text, isFinal: true)); + break; + case 'requestAccepted': + _emit(const BailianEvent(BailianEventType.requestAccepted)); + break; + case 'respondingStarted': + _emit(const BailianEvent(BailianEventType.respondingStarted)); + break; + case 'respondingContent': + _emit(BailianEvent( + BailianEventType.respondingContent, + text: text, + isFinal: isFinal, + finishReason: raw['finishReason']?.toString() ?? '', + directives: _parseDirectives(raw['extraInfo']?.toString(), text), + )); + break; + case 'respondingEnded': + _emit(BailianEvent(BailianEventType.respondingEnded, + text: text, isFinal: true)); + break; + case 'error': + _emit(BailianEvent(BailianEventType.error, + text: text, + code: raw['code']?.toString(), + fatal: raw['fatal'] == true)); + break; + case 'closed': + _running = false; + _emit(const BailianEvent(BailianEventType.closed)); + break; + case 'downlink': + final n = (raw['bytes'] as num?)?.toInt() ?? 0; + if (!_downBytesCtl.isClosed) _downBytesCtl.add(n); + break; + case 'micActivity': + // 只走自己的流,不冒充任何协议事件(见 micActivity 的说明) + if (!_micActivityCtl.isClosed) _micActivityCtl.add(null); + break; + case 'overflow': + // 上行拥塞丢包。warn 级:这是「说了没反应」时唯一的证据 + Logger.w(_tag, '[AI-BRIDGE] 上行拥塞,原生累计丢包=${raw['dropped']}'); + break; + default: + Logger.d(_tag, '未处理的原生事件: $name'); + } + } + + /// 指令解析**仍然走 Dart 这一份**:`extra_info.tool_calls` 的那些坑 + /// (参数是字符串套 JSON、值恒为中文、下午时间要校正)有一批真实 payload 的 + /// 金测试守着,原生再写一遍只会让那批测试失去意义。原生把 `extra_info` + /// 原样转成字符串带上来,这里拼回 output 的形状交给同一个解析器。 + List _parseDirectives(String? extraInfoJson, String text) { + if (extraInfoJson == null || extraInfoJson.isEmpty) return const []; + try { + final extra = json.decode(extraInfoJson); + final parsed = AssistantDirective.parseFromOutput( + {'extra_info': extra, 'text': text}, + source: directiveSource, + ); + if (parsed.isNotEmpty) { + // warn 级、且只在真的带了指令时打——「闹钟没定上」只有两种可能 + // (百炼没下发 / 端侧没落地),这一行是把两者分开的证据 + Logger.w(_tag, '[指令帧] 原生链路解析出 ${parsed.length} 条'); + } + return parsed; + } catch (e) { + Logger.w(_tag, 'extra_info 解析失败: $e'); + return const []; + } + } + + static BailianDialogState _mapState(String s) => switch (s) { + 'listening' => BailianDialogState.listening, + 'thinking' => BailianDialogState.thinking, + 'responding' => BailianDialogState.responding, + _ => BailianDialogState.idle, + }; + + void _emit(BailianEvent e) { + if (!_eventCtl.isClosed) _eventCtl.add(e); + } + + Future dispose() async { + await stop(); + await _eventCtl.close(); + await _downBytesCtl.close(); + await _micActivityCtl.close(); + } +} diff --git a/apps/client/lib/data/services/device_ai_session_service.dart b/apps/client/lib/data/services/device_ai_session_service.dart index e2b3365a..a9b3dbe4 100644 --- a/apps/client/lib/data/services/device_ai_session_service.dart +++ b/apps/client/lib/data/services/device_ai_session_service.dart @@ -9,10 +9,13 @@ import '../../devices/device_hub.dart'; import '../models/agent_module.dart'; import '../models/emai_message.dart'; import '../models/user_Info.dart' as user_info; +import '../utils/background_keepalive.dart'; import '../utils/cue_tone.dart'; +import '../utils/peer_speech_gate.dart' show rmsOfPcm16; import 'agent_module_defaults.dart'; import 'assistant_directive_service.dart'; import 'bailian_multimodal_service.dart'; +import 'bailian_native_dialog.dart'; import 'emai_chat_store.dart'; import 'mcp_token_service.dart'; @@ -44,6 +47,22 @@ class DeviceAiSessionService extends GetxService { /// 「Smartcar 车载助手」(各自的 app_id 不同)。所以这里不能是 final 单例, /// 由 [_ensureService] 按厂商换实例;app_id 没变就复用,不白开 WebSocket。 BailianMultimodalService _svc = BailianMultimodalService(); + + /// 原生百炼客户端。非空 = 本次会话走原生直连:耳机麦 PCM 与合成音频都不经过 + /// Dart,这边只收文本、状态、下行字节数。为 null 时整条链路退回 [_svc]。 + /// + /// ⚠️ 两条路**不会同时开**:[_useNative] 是本次会话的唯一开关, + /// 两边都起来的话上行会被推两份,服务端 VAD 会把它当成一直有人在说话。 + BailianNativeDialog? _native; + bool get _useNative => _native != null; + StreamSubscription? _nativeDownSub; + StreamSubscription? _nativeMicSub; + + /// 原生上报的是**累计**下行字节数,这里存上一次的值好算增量 + int _nativeDownSeen = 0; + + /// 下次开原生会话要用的百炼应用(跟着 [_ensureService] 换) + String? _wantNativeAppId; String? _svcAppId; /// 厂商 → 该接哪个 agent 的 app_id。走的是「厂商 → 品类 → 默认编排第一个模块 @@ -68,6 +87,10 @@ class DeviceAiSessionService extends GetxService { // 旧实例可能还挂着 WebSocket,先停掉再丢 unawaited(_svc.stop()); _svc = BailianMultimodalService(appId: want.isEmpty ? null : want); + // 原生那份也跟着换:两个 agent 用的是百炼里不同的应用,共用会串会话 + unawaited(_native?.dispose()); + _native = null; + _wantNativeAppId = want.isEmpty ? null : want; _svcAppId = want; } @@ -124,8 +147,36 @@ class DeviceAiSessionService extends GetxService { static const Duration _gateMax = Duration(seconds: 90); bool _tearingDown = false; static const Duration _gateTail = Duration(milliseconds: 500); - static const Duration _idleTimeout = Duration(seconds: 5); + /// 双方多久没动静就自动结束。 + /// + /// ⚠️ 原来是 5s,而且只有**百炼事件**才刷新它——设备上行 PCM 到了不算:AI 说完、 + /// 门控解除之后用户必须在 5 秒内让服务端 VAD 判到说话,否则会话被判「双方无声」 + /// 直接结束。锁屏 + 蓝牙上行延迟 + 蜂窝 RTT,这 5 秒非常容易超, + /// 这就是「锁屏后用耳机对话很容易中断」的直接原因(2026-09-20 苹果测试)。 + /// 现在放到 15s,并且设备上行里**有声音**(RMS 过阈值)也算活动。 + static const Duration _idleTimeout = Duration(seconds: 15); Timer? _idleTimer; + + /// 设备上行判「有人在说」的 RMS 阈值(与 PeerSpeechGate 同一档:静音 0~50, + /// 正常说话 1000~8000),以及刷新空闲计时的节流间隔。 + static const int _micActivityRms = 600; + static const Duration _micActivityThrottle = Duration(milliseconds: 500); + DateTime? _lastMicActivityAt; + + /// 百炼 WebSocket 掉了以后的自动重连(蜂窝网切换、弱网抖动)。 + /// 原来 `closed` 一到就整个会话拆掉——用户还戴着耳机在说话。 + static const int _wsReconnectMax = 2; + static const List _wsReconnectDelays = [ + Duration(seconds: 1), + Duration(seconds: 3), + ]; + int _wsReconnectAttempts = 0; + Timer? _wsReconnectTimer; + + /// 设备断连后的观察期:iOS 上 Echo-one 的系统级连接本来就会自行断续, + /// 一抖就拆会话太激进;期间设备又回到 ready 就继续,到期没回来才拆。 + static const Duration _deviceDropGrace = Duration(seconds: 3); + Timer? _deviceDropTimer; DateTime? _firstAudioAt; int _micPackets = 0; int _downPackets = 0; @@ -142,12 +193,27 @@ class DeviceAiSessionService extends GetxService { Logger.e(_tag, '唤醒事件流错误: $e'); }); _hubSub = hub.sessionEvents.listen((e) { - if (e.type == DeviceSessionEventType.connectionStateChanged && - e.connectionState == DeviceConnectionState.disconnected && - isActive.value && - _session?.deviceId == e.deviceId) { - Logger.w(_tag, '设备断连,结束 AI 会话'); - _teardown(notifyDevice: false, playEndTone: false); + if (e.type != DeviceSessionEventType.connectionStateChanged) return; + if (!isActive.value || _session?.deviceId != e.deviceId) return; + if (e.connectionState == DeviceConnectionState.disconnected) { + if (_deviceDropTimer != null) return; + Logger.w(_tag, '设备断连,观察 ${_deviceDropGrace.inSeconds}s 看它回不回来'); + _deviceDropTimer = Timer(_deviceDropGrace, () { + _deviceDropTimer = null; + if (!isActive.value) return; + Logger.w(_tag, '设备 ${_deviceDropGrace.inSeconds}s 内没回来,结束 AI 会话'); + _teardown(notifyDevice: false, playEndTone: false); + }); + } else if (e.connectionState == DeviceConnectionState.ready && + _deviceDropTimer != null) { + _deviceDropTimer!.cancel(); + _deviceDropTimer = null; + // 新的 session 对象(插件断开重连会重建),把引用换过来,之后的 ai.state / + // ai.exit 才发得出去;上下行流是插件级的,不用重开。 + final fresh = hub.sessions.firstWhereOrNull((s) => s.deviceId == e.deviceId); + if (fresh != null) _session = fresh; + Logger.w(_tag, '设备在观察期内回来了,AI 会话继续'); + _touchActivity(); } }); Logger.i(_tag, '设备 AI 会话服务已就绪,等待按键唤醒'); @@ -178,9 +244,92 @@ class DeviceAiSessionService extends GetxService { } } + /// 起本次对话。两条路径的参数**必须一致**,所以只在这一处写。 + Future _startDialog() async { + final token = await McpTokenService.get(); + if (_useNative) { + return _native!.start( + mode: 'duplex', + downstreamRate: _deviceRate, + userId: _currentUserId(), + source: 'earphone', + mcpToken: token, + ); + } + return _svc.start( + mode: 'duplex', + downstreamRate: _deviceRate, + userId: _currentUserId(), + source: 'earphone', + mcpToken: token, + ); + } + + /// 试着开原生直连。两端都要开: + /// - azure_speech 那端([BailianNativeDialog])负责 WS 与攒包; + /// - bluetooth_manager 那端(`DeviceFeatures.aiPcmNativeBridge`)负责把耳机麦 + /// PCM 从 Dart 的 EventChannel 上摘下来、并接住回灌的合成音频。 + /// + /// 任一端不支持就整条退回 Dart,**不能只开一半**:只开翻译端的话上行一帧都收不到, + /// 只开设备端的话 PCM 被摘走却没人接,两种都表现为「唤醒了但 AI 不说话」。 + Future _tryOpenNativeDialog(DeviceSession session) async { + if (!await BailianNativeDialog.isAvailable()) { + Logger.i(_tag, '原生百炼客户端不可用,走 Dart 链路'); + return null; + } + try { + final r = await session.invokeFeature( + DeviceFeatures.aiPcmNativeBridge, {'enabled': true}); + if (r['enabled'] != true) { + Logger.i(_tag, '设备端不支持 AI 原生桥,走 Dart 链路'); + return null; + } + } on DeviceException catch (e) { + Logger.i(_tag, '设备端不支持 AI 原生桥(${e.code}),走 Dart 链路'); + return null; + } catch (e) { + Logger.w(_tag, '开设备端 AI 原生桥失败,走 Dart 链路: $e'); + return null; + } + Logger.i(_tag, 'EMAI 走原生直连(耳机 PCM 不经过 Dart)'); + return BailianNativeDialog(appId: _wantNativeAppId); + } + + /// 告诉服务端「这一轮本地已经播完了」,它据此把对话状态推回 Listening/Idle。 + /// + /// ⚠️ 这里**不要**去动耳机的下行队列。原生 `AiAudioDownlink.flush()` 的语义是 + /// **清空**(`outQueue.clear()`),不是"把尾包发出去"——名字容易读反, + /// 在这里调它等于把每一句回复的尾音切掉。真正要丢弃待播的地方只有 + /// 「真人插话」,那条走 `_spk?.discardPending()`,两种链路都有效 + /// (下行发送器是同一个原生对象,跟谁喂的 PCM 无关)。 + void _notifyLocalRespondingEnded() { + if (_useNative) { + unawaited(_native!.notifyLocalRespondingEnded()); + } else { + _svc.notifyLocalRespondingEnded(); + } + } + + /// 原生只报**累计**下行字节数(约 100ms 一条):PCM 已经直接回了耳机, + /// Dart 这边要做的只有开门控、计时、计数,三件事都只依赖长度。 + void _onNativeDownstreamTotal(int total) { + final delta = total - _nativeDownSeen; + _nativeDownSeen = total; + if (delta > 0) _noteDownstream(delta); + } + void _onMicPcm(Uint8List pcm) { if (!isActive.value || pcm.isEmpty) return; _micPackets++; + // 用户在说话 = 有活动。只看能量、带节流,静音帧不续命 + if (!_uplinkGated && rmsOfPcm16(pcm) >= _micActivityRms) { + final now = DateTime.now(); + if (_lastMicActivityAt == null || + now.difference(_lastMicActivityAt!) >= _micActivityThrottle) { + _lastMicActivityAt = now; + _touchActivity(); + } + } if (_micPackets == 1 || _micPackets % 50 == 0) { Logger.i(_tag, '设备上行 #$_micPackets (${pcm.length}B/包)${_uplinkGated ? ' [门控中]' : ''}'); @@ -232,6 +381,7 @@ class DeviceAiSessionService extends GetxService { return; } isActive.value = true; + _wsReconnectAttempts = 0; // 换了设备就重新试一次 ai.state(上一台不支持不代表这台也不支持) _aiStateUnsupported = false; _quitAfterPlayout = false; @@ -241,13 +391,29 @@ class DeviceAiSessionService extends GetxService { _micAccum.clear(); await _yieldPhonePage(); + // iOS:会话期间保持一个静音的后台音频播放,否则锁屏几秒后进程被挂起, + // BLE 上行与 WebSocket 一起停摆(见 BackgroundKeepalive)。Android 空操作。 + await BackgroundKeepalive.acquire(_tag); + // ★ 先试原生直连:上行 PCM 与合成音频都不经过 Dart。 + // 插件老 / 设备不是恒玄 → 返回 null,整条链路退回下面的 Dart 路径。 + _native = await _tryOpenNativeDialog(session); try { - _mic = await session.openMic(route: AudioRoute.ai); - _micSub = _mic!.pcm.listen(_onMicPcm); - _eventSub = _svc.events.listen(_onBailianEvent); - // 下行 16k:G.722 编码器就是 16k,让服务端直接出 16k 省掉一次重采样 - _audioSub = _svc.audioOut.listen(_onDownstreamPcm); + if (_useNative) { + _eventSub = _native!.events.listen(_onBailianEvent); + _nativeDownSub = _native!.downstreamBytes.listen(_onNativeDownstreamTotal); + // 设备上行有声只用来续空闲计时,不能当 speechStarted(那会丢弃待播队列) + _nativeMicSub = _native!.micActivity.listen((_) => _touchActivity()); + } else { + _mic = await session.openMic(route: AudioRoute.ai); + _micSub = _mic!.pcm.listen(_onMicPcm); + _eventSub = _svc.events.listen(_onBailianEvent); + // 下行 16k:G.722 编码器就是 16k,让服务端直接出 16k 省掉一次重采样 + _audioSub = _svc.audioOut.listen(_onDownstreamPcm); + } + // 两条路都要开扬声器:它会启动耳机的 AI 下行发送器(G.722 编码与节流)。 + // 原生直连时我们不往里写 PCM,但那个发送器必须在跑,否则原生推下去的 + // 合成音频没人发。提示音(叮/嘟)两种模式下都还是由 Dart 写。 _spk = await session.openSpeaker(route: AudioRoute.ai); } catch (e) { Logger.e(_tag, '打开设备音频通路失败: $e'); @@ -258,13 +424,7 @@ class DeviceAiSessionService extends GetxService { await _playCue(CueTone.ding(sampleRate: _deviceRate)); _touchActivity(); - final ok = await _svc.start( - mode: 'duplex', - downstreamRate: _deviceRate, - userId: _currentUserId(), - source: 'earphone', - mcpToken: await McpTokenService.get(), - ); + final ok = await _startDialog(); final elapsed = DateTime.now().difference(ackAt); if (elapsed > _ackDeadline) { Logger.w( @@ -360,6 +520,14 @@ class DeviceAiSessionService extends GetxService { case BailianEventType.error: Logger.e(_tag, '会话错误: ${e.code} ${e.text} fatal=${e.fatal}'); + // 网络类的致命错误(socket/连接被掐)先当掉线重连,不当「功能结束」 + if (e.fatal && _looksLikeNetworkError(e.text) && _scheduleWsReconnect(e.text)) { + _store.dropIfEmpty(_streamingUser); + _streamingUser = null; + _store.dropIfEmpty(_streamingAssistant); + _streamingAssistant = null; + break; + } _store.dropIfEmpty(_streamingUser); _streamingUser = null; _store.dropIfEmpty(_streamingAssistant); @@ -402,7 +570,8 @@ class DeviceAiSessionService extends GetxService { break; case BailianEventType.closed: - if (isActive.value) { + if (isActive.value && !_tearingDown) { + if (_scheduleWsReconnect('closed')) break; Logger.w(_tag, '百炼会话关闭,结束设备 AI'); _teardown(notifyDevice: true); } @@ -413,18 +582,27 @@ class DeviceAiSessionService extends GetxService { } } + /// 退回 Dart 链路时才会跑(原生直连时 PCM 不经过这里) void _onDownstreamPcm(Uint8List pcm) { - if (!isActive.value || pcm.isEmpty) return; + if (pcm.isEmpty) return; + _noteDownstream(pcm.length); + _spk?.write(pcm); + } + + /// 收到 [n] 字节下行音频时要做的记账。两条路径共用: + /// 走 Dart 时由 [_onDownstreamPcm] 调,走原生时由 [_onNativeDownstreamTotal] 调。 + /// 开门控、估播完时长、计数——都只依赖字节数,不需要 PCM 本身。 + void _noteDownstream(int n) { + if (!isActive.value || n <= 0) return; if (_firstAudioAt == null) { _firstAudioAt = DateTime.now(); _openGate(); } - _respondingBytes += pcm.length; + _respondingBytes += n; _downPackets++; if (_downPackets == 1 || _downPackets % 50 == 0) { - Logger.i(_tag, '下行给设备 #$_downPackets (${pcm.length}B)'); + Logger.i(_tag, '下行给设备 #$_downPackets (${n}B)'); } - _spk?.write(pcm); } /// 有动静就把静默计时重新拨回 5 秒。播报期间不计时。 @@ -459,6 +637,47 @@ class DeviceAiSessionService extends GetxService { ); } + static bool _looksLikeNetworkError(String text) { + final t = text.toLowerCase(); + const markers = [ + 'socket', 'connection', 'network', 'timed out', 'timeout', 'reset', + 'broken pipe', 'unreachable', 'host lookup', 'websocket', 'closed', + ]; + return markers.any(t.contains); + } + + /// 百炼 WS 掉了:会话还活着、设备还连着,就换条连接接着聊。 + /// 返回 false 表示预算用完,调用方按原来的方式结束会话。 + bool _scheduleWsReconnect(String reason) { + if (!isActive.value || _tearingDown) return false; + if (_wsReconnectAttempts >= _wsReconnectMax) { + Logger.w(_tag, '百炼 WS 重连 $_wsReconnectMax 次仍失败($reason),放弃'); + return false; + } + if (_wsReconnectTimer != null) return true; // 已在排队 + final delay = _wsReconnectDelays[_wsReconnectAttempts]; + _wsReconnectAttempts++; + Logger.w(_tag, '百炼 WS 掉线($reason),${delay.inSeconds}s 后第 $_wsReconnectAttempts 次重连'); + // 重连期间别让空闲计时把会话判死 + _idleTimer?.cancel(); + _idleTimer = null; + _wsReconnectTimer = Timer(delay, () async { + _wsReconnectTimer = null; + if (!isActive.value || _tearingDown) return; + _cancelPlayoutTimer(); + _closeGate(); + final ok = await _startDialog(); + if (!isActive.value) return; + if (ok) { + Logger.w(_tag, '百炼 WS 第 $_wsReconnectAttempts 次重连成功'); + _touchActivity(); + } else if (!_scheduleWsReconnect('start failed')) { + _teardown(notifyDevice: true); + } + }); + return true; + } + void _touchActivity() { _idleTimer?.cancel(); _idleTimer = null; @@ -494,11 +713,13 @@ class DeviceAiSessionService extends GetxService { if (!_uplinkGated) return; Logger.w(_tag, '门控开了 ${_gateMax.inSeconds}s 还没等到播完信号,强制解除'); _cancelPlayoutTimer(); - _svc.notifyLocalRespondingEnded(); + _notifyLocalRespondingEnded(); _closeGate(); }); if (_uplinkGated) return; _uplinkGated = true; + // 原生直连时逐帧替换静音在原生做,这里只把结论推下去 + unawaited(_native?.setGated(true)); _idleTimer?.cancel(); _idleTimer = null; Logger.i(_tag, '上行门控开启(AI 播报中)'); @@ -511,6 +732,7 @@ class DeviceAiSessionService extends GetxService { _gateSafety = null; if (!_uplinkGated) return; _uplinkGated = false; + unawaited(_native?.setGated(false)); Logger.i(_tag, '上行门控解除'); _touchActivity(); } @@ -537,7 +759,7 @@ class DeviceAiSessionService extends GetxService { Logger.i(_tag, '预计还需 ${remainMs}ms 播完(已推 ${totalMs}ms 音频)'); _playoutTimer = Timer(Duration(milliseconds: remainMs), () { if (!isActive.value) return; - _svc.notifyLocalRespondingEnded(); + _notifyLocalRespondingEnded(); _gateTimer = Timer(_gateTail, _closeGate); if (_quitAfterPlayout) { _quitAfterPlayout = false; @@ -602,6 +824,12 @@ class DeviceAiSessionService extends GetxService { _idleTimer = null; _gateSafety?.cancel(); _gateSafety = null; + _wsReconnectTimer?.cancel(); + _wsReconnectTimer = null; + _wsReconnectAttempts = 0; + _deviceDropTimer?.cancel(); + _deviceDropTimer = null; + _lastMicActivityAt = null; _cancelPlayoutTimer(); _micAccum.clear(); @@ -627,16 +855,37 @@ class DeviceAiSessionService extends GetxService { _audioSub = null; await _micSub?.cancel(); _micSub = null; + await _nativeDownSub?.cancel(); + _nativeDownSub = null; + await _nativeMicSub?.cancel(); + _nativeMicSub = null; + _nativeDownSeen = 0; await _spk?.close(); _spk = null; await _mic?.close(); _mic = null; + // 原生直连时两端都要关:只关一端会把耳机麦的 PCM 一直摘在原生里没人收 + final nat = _native; + _native = null; + if (nat != null) { + await nat.stop(); + final s0 = _session; + if (s0 != null) { + try { + await s0.invokeFeature( + DeviceFeatures.aiPcmNativeBridge, {'enabled': false}); + } catch (e) { + Logger.w(_tag, '关设备端 AI 原生桥失败: $e'); + } + } + } await _svc.stop(); final s = _session; if (wasActive && notifyDevice && s != null) await _sendExit(s); _session = null; + await BackgroundKeepalive.release(_tag); if (wasActive) { Logger.i(_tag, '设备 AI 会话已结束 (上行 $_micPackets 包 / 下行 $_downPackets 包)'); await _resumePhonePage(); diff --git a/apps/client/lib/data/services/device_call_recording_service.dart b/apps/client/lib/data/services/device_call_recording_service.dart index cf78ddd2..b9ac0206 100644 --- a/apps/client/lib/data/services/device_call_recording_service.dart +++ b/apps/client/lib/data/services/device_call_recording_service.dart @@ -49,6 +49,20 @@ class DeviceCallRecordingService extends GetxService { final Stopwatch _stopwatch = Stopwatch(); bool _busy = false; + /// 本采样窗口内的电平峰值。纯字段、不是 Rx——每成一对帧更新它, + /// 由 [_sample] 每 [sampleIntervalMs] 发布进 [level]。 + /// + /// 只在**退回 Dart 落盘**那条路上用;走原生录音器时电平由原生算好送上来。 + double _levelPeak = 0; + + /// 原生落盘录音器。非空 = 双路 PCM 一个字节都不经过 Dart + /// (通话是双路 50 帧/s,这条路径原先每秒要跨界 100 次)。 + DeviceFileRecorder? _rec; + StreamSubscription? _recSub; + + /// 已写入的音频字节数(两条路径共用) + int get _dataBytes => _rec?.bytesWritten ?? _wav?.dataBytes ?? 0; + /// 本次录音是不是设备按键发起的。设备侧处于录音态时,停止要额外补一条 /// `record.stop` 让它自己也退出来,只发 `call.stop` 不足以清掉设备端的录音状态。 bool _deviceRecordingActive = false; @@ -132,14 +146,26 @@ class DeviceCallRecordingService extends GetxService { if (!await dir.exists()) await dir.create(recursive: true); final ts = DateFormat('yyyyMMdd_HHmmss').format(DateTime.now()); _path = '${dir.path}/call_$ts.wav'; - _wav = await WavWriter.open(_path!, channels: 2); _pairer.clear(); _frameCount = 0; - _peer = await session.openMic(route: AudioRoute.callPeer); - _local = await session.openMic(route: AudioRoute.callLocal); - _peerSub = _peer!.pcm.listen(_onPeer); - _localSub = _local!.pcm.listen(_onLocal); + // ★ 优先让原生直接落盘:左=本端麦、右=对端,交织与配对都在原生做。 + // 双路每秒 100 帧的跨界连同 Dart 侧的写盘队列一起消失。 + _rec = await session.openFileRecorder( + path: _path!, + routes: const [AudioRoute.callLocal, AudioRoute.callPeer], + ); + if (_rec != null) { + _recSub = _rec!.stats.listen(_onRecStats); + Logger.i(_tag, '通话录音走原生直接落盘(双路 PCM 不经过 Dart)'); + } else { + _wav = await WavWriter.open(_path!, channels: 2); + _peer = await session.openMic(route: AudioRoute.callPeer); + _local = await session.openMic(route: AudioRoute.callLocal); + _peerSub = _peer!.pcm.listen(_onPeer); + _localSub = _local!.pcm.listen(_onLocal); + Logger.i(_tag, '插件不支持原生落盘,退回 Dart 写盘'); + } // 进通话模式,设备才会双路上行。不管是不是 App 主动发起的都补发一次—— // 只靠设备自己进通话模式时,它未必会把音频回传给 App。 @@ -165,6 +191,7 @@ class DeviceCallRecordingService extends GetxService { isRecording.value = true; elapsedMs.value = 0; level.value = 0; + _levelPeak = 0; waveform.clear(); _stopwatch ..reset() @@ -197,7 +224,8 @@ class DeviceCallRecordingService extends GetxService { Future _stop(bool sendCommand) async { _silenceCheck?.cancel(); - Logger.i(_tag, '停止通话录音: 收到 $_frameCount 帧,写入 ${_wav?.dataBytes} 字节'); + Logger.i(_tag, + '停止通话录音: ${_rec != null ? "原生落盘" : "收到 $_frameCount 帧"},写入 $_dataBytes 字节'); _tick?.cancel(); _sampleTimer?.cancel(); _stopwatch.stop(); @@ -223,12 +251,13 @@ class DeviceCallRecordingService extends GetxService { } _deviceRecordingActive = false; - final dataBytes = _wav?.dataBytes ?? 0; + final dataBytes = _dataBytes; await _closeFile(save: true); unawaited(AudioForegroundService.stop()); isRecording.value = false; elapsedMs.value = 0; level.value = 0; + _levelPeak = 0; waveform.clear(); _session = null; @@ -256,6 +285,17 @@ class DeviceCallRecordingService extends GetxService { /// 结果右声道(对端)被断续抽掉,转写出来只剩自己这一方。 final StereoPairer _pairer = StereoPairer(); + /// 原生录音器的统计(约 100ms 一条)。[_frameCount] 也在这里推进—— + /// 它驱动「2s 内没收到任何音频就重发 call.start」那条自愈, + /// 走原生路径时 Dart 收不到帧,不接上的话会每次都误判成没上行。 + void _onRecStats(DeviceRecordStats s) { + // 双声道交织后 20ms = 1280 字节(单路 640 × 2 声道)。 + // 除以 640 会把帧数报成两倍,那条日志就对不上真机抓到的帧数了。 + if (s.bytes > 0) _frameCount = s.bytes ~/ 1280; + if (s.level > _levelPeak) _levelPeak = s.level; + } + + /// 退回 Dart 落盘时才会跑(插件不支持原生录音器)。 void _onPeer(Uint8List pcm) { if (_wav == null || pcm.isEmpty) return; _frameCount++; @@ -276,12 +316,19 @@ class DeviceCallRecordingService extends GetxService { void _flushPairs() { final pair = _pairer.take(); if (pair == null) return; - _wav!.write(WavWriter.interleave(pair.left, pair.right), onWritten: () { - level.value = WavWriter.rmsLevel(pair.left); - }); + // 电平只更新窗口峰值(纯字段,不碰 Rx),由 _sample() 每 + // [sampleIntervalMs] 发布一次。双路各 50 帧/s,原先每成一对就 + // `level.value =` 一次 = 50 次/s 的 Rx 通知,而读它的只有 100ms + // 一次的 _sample()。也不再挂在 onWritten 上:显示不该排在写盘队列后面。 + final l = WavWriter.rmsLevel(pair.left); + if (l > _levelPeak) _levelPeak = l; + _wav!.write(WavWriter.interleave(pair.left, pair.right)); } void _sample() { + // 发布这一窗口的峰值并清零:这里是 level 唯一的写入点 + level.value = _levelPeak; + _levelPeak = 0; final list = waveform.toList(); list.add(level.value); if (list.length > _maxSamples) { @@ -299,6 +346,15 @@ class DeviceCallRecordingService extends GetxService { await _peer?.close(); _local = null; _peer = null; + await _recSub?.cancel(); + _recSub = null; + final rec = _rec; + _rec = null; + if (rec != null) { + // 原生停下来时才回填 WAV 头,所以这一步不能跳 + await rec.stop(save: save); + return; + } final w = _wav; _wav = null; if (w == null) return; diff --git a/apps/client/lib/data/services/device_recording_service.dart b/apps/client/lib/data/services/device_recording_service.dart index 11534350..7efe48e6 100644 --- a/apps/client/lib/data/services/device_recording_service.dart +++ b/apps/client/lib/data/services/device_recording_service.dart @@ -51,6 +51,19 @@ class DeviceRecordingService extends GetxService { DateTime? _cmdSentAt; bool _busy = false; + /// 本采样窗口内的电平峰值。纯字段、不是 Rx——每帧更新它, + /// 由 [_sample] 每 [sampleIntervalMs] 发布进 [level]。 + /// + /// 只在**退回 Dart 落盘**那条路上用;走原生录音器时电平由原生算好送上来。 + double _levelPeak = 0; + + /// 原生落盘录音器。非空 = PCM 一个字节都不经过 Dart。 + DeviceFileRecorder? _rec; + StreamSubscription? _recSub; + + /// 已写入的音频字节数(两条路径共用,用来算时长) + int get _dataBytes => _rec?.bytesWritten ?? _wav?.dataBytes ?? 0; + @override void onInit() { super.onInit(); @@ -121,7 +134,6 @@ class DeviceRecordingService extends GetxService { if (!await dir.exists()) await dir.create(recursive: true); final ts = DateFormat('yyyyMMdd_HHmmss').format(DateTime.now()); _path = '${dir.path}/${session.vendor}_$ts.wav'; - _wav = await WavWriter.open(_path!, channels: 1); _firstPcmLogged = false; _cmdSentAt = null; @@ -140,19 +152,37 @@ class DeviceRecordingService extends GetxService { '已下发 record.start,写入耗时 ${_cmdSentAt!.difference(t0).inMilliseconds}ms'); } - _src = await session.openMic(route: AudioRoute.mic); - _audioSub = _src!.pcm.listen(_onPcm); + // ★ 优先让原生直接落盘:PCM 不进 Dart,每秒 50 次跨界与 Dart 侧的 + // 文件写入队列一起消失。插件不支持就返回 null,退回下面的老路径。 + _rec = await session.openFileRecorder( + path: _path!, + routes: const [AudioRoute.mic], + ); + final int bytesPerMs; + if (_rec != null) { + // 原生每 100ms 报一次 {bytes, level},正好是波形的节奏, + // 不需要再在 Dart 里起采样定时器算电平。 + _recSub = _rec!.stats.listen(_onRecStats); + bytesPerMs = 32; // 16k/16bit/mono + Logger.i(_tag, '录音走原生直接落盘(PCM 不经过 Dart)'); + } else { + _wav = await WavWriter.open(_path!, channels: 1); + _src = await session.openMic(route: AudioRoute.mic); + _audioSub = _src!.pcm.listen(_onPcm); + bytesPerMs = _src!.format.bytesPerMs; + Logger.i(_tag, '插件不支持原生落盘,退回 Dart 写盘'); + } isRecording.value = true; elapsedMs.value = 0; level.value = 0; + _levelPeak = 0; waveform.clear(); _stopwatch.reset(); // 时长按**已写入的数据量**算,不用墙钟:显示值与文件实际播放长度永远一致 // (恒玄收到 AA 59 后要 1.8~6.2s 才开始上行,还会把等待期缓存的音频补推)。 - final bytesPerMs = _src!.format.bytesPerMs; _tick = Timer.periodic(const Duration(milliseconds: 200), (_) { - elapsedMs.value = _wav == null ? 0 : _wav!.dataBytes ~/ bytesPerMs; + elapsedMs.value = _dataBytes ~/ bytesPerMs; }); _sampleTimer = Timer.periodic( const Duration(milliseconds: sampleIntervalMs), @@ -190,7 +220,7 @@ class DeviceRecordingService extends GetxService { // 等队列写完再关文件,否则最后几帧会丢、WAV 头长度也对不上 await _wav?.drain(); - final int dataBytes = _wav?.dataBytes ?? 0; + final int dataBytes = _dataBytes; final int seconds = elapsedMs.value ~/ 1000; final String? path = _path; final wallMs = _stopwatch.elapsedMilliseconds; @@ -198,7 +228,7 @@ class DeviceRecordingService extends GetxService { _tag, '录音统计: 音频${(dataBytes / 32000).toStringAsFixed(2)}s 数据${dataBytes}B ' '墙钟${(wallMs / 1000).toStringAsFixed(2)}s ' - '写入${_wav?.writtenFrames}帧 丢${_wav?.droppedFrames}帧'); + '${_rec != null ? "原生落盘" : "写入${_wav?.writtenFrames}帧 丢${_wav?.droppedFrames}帧"}'); final session = _session; if (sendCommand && @@ -218,6 +248,7 @@ class DeviceRecordingService extends GetxService { isRecording.value = false; elapsedMs.value = 0; level.value = 0; + _levelPeak = 0; waveform.clear(); _session = null; @@ -237,6 +268,23 @@ class DeviceRecordingService extends GetxService { _path = null; } + /// 原生录音器的统计(约 100ms 一条)。电平已经在原生按窗口取过峰值, + /// 这里直接放进 [_levelPeak],由 [_sample] 与 Dart 落盘那条路统一发布。 + void _onRecStats(DeviceRecordStats s) { + if (!_firstPcmLogged && s.bytes > 0) { + _firstPcmLogged = true; + if (!_stopwatch.isRunning) _stopwatch.start(); + final sent = _cmdSentAt; + Logger.i( + _tag, + '首帧音频到达(原生落盘)' + '${sent != null ? ",距 record.start ${DateTime.now().difference(sent).inMilliseconds}ms" : ""}' + ',计时从此刻开始'); + } + if (s.level > _levelPeak) _levelPeak = s.level; + } + + /// 退回 Dart 落盘时才会跑(插件不支持原生录音器)。 void _onPcm(Uint8List pcm) { final w = _wav; if (w == null || pcm.isEmpty) return; @@ -250,15 +298,25 @@ class DeviceRecordingService extends GetxService { '${sent != null ? ",距 record.start ${DateTime.now().difference(sent).inMilliseconds}ms" : ""}' ',计时从此刻开始'); } - // 串行写,绝不在回调里直接 await 文件(见 WavWriter);电平只是显示,不能挡写入 - w.write(pcm, onWritten: () { - try { - _updateLevel(pcm); - } catch (_) {} - }); + // 电平只更新窗口峰值(纯字段,不碰 Rx),由 _sample() 每 + // [sampleIntervalMs] 发布一次。 + // + // ⚠️ 原先是每帧 `level.value = rms(pcm)`:设备 20ms 一帧 = 50 次/s 的 Rx + // 通知,而读它的只有 100ms 一次的 _sample(),也就是说每 5 次里有 4 次是 + // 唤醒了 Obx 又立刻被下一帧覆盖。峰值比"最后一帧"更适合做音浪,顺带 + // 把短促的说话尖峰留住。 + // + // 也不再挂在 onWritten 上:电平是显示,不该排在文件写入队列后面等。 + final l = WavWriter.rmsLevel(pcm); + if (l > _levelPeak) _levelPeak = l; + // 串行写,绝不在回调里直接 await 文件(见 WavWriter) + w.write(pcm); } void _sample() { + // 发布这一窗口的峰值并清零:这里是 level 唯一的写入点 + level.value = _levelPeak; + _levelPeak = 0; final list = waveform.toList(); list.add(level.value); if (list.length > _maxSamples) { @@ -267,15 +325,20 @@ class DeviceRecordingService extends GetxService { waveform.assignAll(list); } - void _updateLevel(Uint8List pcm) { - level.value = WavWriter.rmsLevel(pcm); - } - Future _closeFile({required bool save}) async { await _audioSub?.cancel(); _audioSub = null; await _src?.close(); _src = null; + await _recSub?.cancel(); + _recSub = null; + final rec = _rec; + _rec = null; + if (rec != null) { + // 原生停下来时才回填 WAV 头,所以这一步不能跳 + await rec.stop(save: save); + return; + } final w = _wav; _wav = null; if (w == null) return; diff --git a/apps/client/lib/data/services/device_stereo_recorder.dart b/apps/client/lib/data/services/device_stereo_recorder.dart index 76f4e5a2..e6ce53e6 100644 --- a/apps/client/lib/data/services/device_stereo_recorder.dart +++ b/apps/client/lib/data/services/device_stereo_recorder.dart @@ -14,24 +14,35 @@ import 'wav_writer.dart'; /// 文件放哪、什么时候停都由调用方决定,不归档。 /// 左声道=本端麦克风,右声道=对端声音。 class DeviceStereoRecorder { - DeviceStereoRecorder({this.onWavePcm}); + DeviceStereoRecorder({this.onWavePcm, this.onLevel}); static const String _tag = 'DeviceStereoRecorder'; - /// 每写一对帧回调一次本端 PCM(驱动波形) + /// 每写一对帧回调一次本端 PCM(驱动波形)。 + /// **只在退回 Dart 落盘那条路上触发**——走原生录音器时 PCM 不进 Dart, + /// 波形改由 [onLevel] 驱动。 final void Function(Uint8List pcm)? onWavePcm; + /// 电平回调,0~1 的**线性 RMS**(与 onWavePcm 那条路上调用方自己算出来的 + /// 口径一致,不带开根的视觉增强)。原生路径约 100ms 一次。 + final void Function(double level)? onLevel; + DeviceSession? _session; DeviceAudioSource? _local; DeviceAudioSource? _peer; StreamSubscription? _localSub; StreamSubscription? _peerSub; WavWriter? _wav; + + /// 原生落盘录音器。非空 = 双路 PCM 一个字节都不经过 Dart。 + DeviceFileRecorder? _rec; + StreamSubscription? _recSub; + /// 两路各自排队、按字节对齐后成对写入,见 [StereoPairer] final StereoPairer _pairer = StereoPairer(); bool _paused = false; - bool get isRunning => _wav != null; + bool get isRunning => _wav != null || _rec != null; /// 开始。没有具备通话双路抓音能力的设备返回 false。 Future start(String filePath) async { @@ -43,21 +54,38 @@ class DeviceStereoRecorder { } try { _session = session; - _wav = await WavWriter.open(filePath, channels: 2); _pairer.clear(); _paused = false; - _peer = await session.openMic(route: AudioRoute.callPeer); - _local = await session.openMic(route: AudioRoute.callLocal); - _peerSub = _peer!.pcm.listen((pcm) { - if (_paused || pcm.isEmpty) return; - _pairer.pushRight(pcm); - _flushPairs(); - }); - _localSub = _local!.pcm.listen((pcm) { - if (_paused || pcm.isEmpty) return; - _pairer.pushLeft(pcm); - _flushPairs(); - }); + // ★ 优先让原生直接落盘(左=本端、右=对端),双路每秒 100 帧的跨界消失 + _rec = await session.openFileRecorder( + path: filePath, + routes: const [AudioRoute.callLocal, AudioRoute.callPeer], + ); + if (_rec != null) { + _recSub = _rec!.stats.listen((s) { + if (_paused) return; + // 原生给的 level 是开过根的视觉值,这里平方还原成线性 RMS, + // 与 onWavePcm 那条路上调用方自己算出来的口径对齐—— + // 不还原的话波形会被再开一次根,整条线贴着顶跑 + onLevel?.call(s.level * s.level); + }); + Logger.i(_tag, '走原生直接落盘(双路 PCM 不经过 Dart)'); + } else { + _wav = await WavWriter.open(filePath, channels: 2); + _peer = await session.openMic(route: AudioRoute.callPeer); + _local = await session.openMic(route: AudioRoute.callLocal); + _peerSub = _peer!.pcm.listen((pcm) { + if (_paused || pcm.isEmpty) return; + _pairer.pushRight(pcm); + _flushPairs(); + }); + _localSub = _local!.pcm.listen((pcm) { + if (_paused || pcm.isEmpty) return; + _pairer.pushLeft(pcm); + _flushPairs(); + }); + Logger.i(_tag, '插件不支持原生落盘,退回 Dart 写盘'); + } await session.invokeFeature(DeviceFeatures.callStart); return true; } catch (e) { @@ -70,8 +98,9 @@ class DeviceStereoRecorder { void _flushPairs() { final pair = _pairer.take(); if (pair == null) return; - _wav?.write(WavWriter.interleave(pair.left, pair.right), - onWritten: () => onWavePcm?.call(pair.left)); + _wav?.write(WavWriter.interleave(pair.left, pair.right)); + // 波形是显示,不该排在写盘队列后面等落盘;写失败时也照样要动 + onWavePcm?.call(pair.left); } void pause() => _paused = true; @@ -102,6 +131,15 @@ class DeviceStereoRecorder { Logger.w(_tag, 'call.stop 失败: $e'); } } + await _recSub?.cancel(); + _recSub = null; + final rec = _rec; + _rec = null; + if (rec != null) { + // 原生停下来时才回填 WAV 头,所以这一步不能跳 + await rec.stop(save: saveFile); + return; + } final w = _wav; _wav = null; await w?.close(save: saveFile); diff --git a/apps/client/lib/data/services/meeting/meeting_task_service.dart b/apps/client/lib/data/services/meeting/meeting_task_service.dart index 6fc2cb49..29829fb3 100644 --- a/apps/client/lib/data/services/meeting/meeting_task_service.dart +++ b/apps/client/lib/data/services/meeting/meeting_task_service.dart @@ -54,7 +54,12 @@ class MeetingTaskService extends GetxService { localImportState.value++; } - Future addTask( + /// 返回任务是否**已受理或已排队**。 + /// + /// 原来返回 void,调用方(详情页「生成」)不管结果一律把页面切成「等待转写」; + /// 而 `audiourl` 为空、上传队列里又找不到这条(上传项在离开详情页时会被摘掉)时, + /// 这里什么都没做——页面就永远停在占位图上等一个不存在的任务。 + Future addTask( int id, String audiourl, String formlanguage, @@ -68,27 +73,20 @@ class MeetingTaskService extends GetxService { (element) => element['id'] == id, orElse: () => {}, ); - if (uploadData.isNotEmpty) { - if (uploadData['audiourl'].isEmpty) { - uploadData['isTask'] = true; - uploadData['formlanguage'] = formlanguage; - uploadData['tolanguage'] = tolanguage; - uploadData['isdistinguishspeaker'] = isdistinguishspeaker; - uploadData['templateid'] = templateid; - uploadData['tid'] = tid; - } else { - await submitTask( - id, - formlanguage, - tolanguage, - isdistinguishspeaker, - templateid, - tid, - ); - } + if (uploadData.isEmpty) { + Logger.w('MeetingTask', '记录 $id 没有音频 URL、上传队列里也没有它,无法提交任务'); + return false; + } + if (uploadData['audiourl'].isEmpty) { + uploadData['isTask'] = true; + uploadData['formlanguage'] = formlanguage; + uploadData['tolanguage'] = tolanguage; + uploadData['isdistinguishspeaker'] = isdistinguishspeaker; + uploadData['templateid'] = templateid; + uploadData['tid'] = tid; + return true; // 上传完成时由 MeetingUploadService 提交 } - } else { - await submitTask( + return submitTask( id, formlanguage, tolanguage, @@ -97,6 +95,14 @@ class MeetingTaskService extends GetxService { tid, ); } + return submitTask( + id, + formlanguage, + tolanguage, + isdistinguishspeaker, + templateid, + tid, + ); } /// 智能总结模板的 tid(后台「Autopilot智能总结」,41 种语言各有一版,按 tid 反查)。 diff --git a/apps/client/lib/data/services/speech_impl/azure_ast_service.dart b/apps/client/lib/data/services/speech_impl/azure_ast_service.dart index 4774f5b3..306c8396 100644 --- a/apps/client/lib/data/services/speech_impl/azure_ast_service.dart +++ b/apps/client/lib/data/services/speech_impl/azure_ast_service.dart @@ -224,6 +224,17 @@ class AzureAstService extends GetxService implements AstService { Logger.info('[STS] 服务初始化: ${eventMap['serviceId']} ${eventMap['direction']}'); break; + // 原生直连桥的诊断事件。warn 级:release 的日志门槛是 warning, + // 这两行是真机调门限阈值 / 判断上行有没有拥塞的唯一依据。 + case 'peerGateOpened': + Logger.w('AzureAST', + '[CALL-BRIDGE] B 路门限开启(原生):${eventMap['fallback'] == true ? "兜底强开(阈值可能偏高/全程只有串音)" : "检测到对端语音"} rms=${eventMap['rms']}'); + break; + case 'nativeBridgeOverflow': + Logger.w('AzureAST', + '[CALL-BRIDGE] 原生上行积压丢帧,累计 ${eventMap['dropped']}(网络/服务端跟不上耳机实时速率)'); + break; + default: Logger.info('[STS] 未处理事件: $eventType'); break; @@ -505,6 +516,31 @@ class AzureAstService extends GetxService implements AstService { } } + @override + Future setNativeCallBridge(bool enabled) async { + try { + final ok = await _channel + .invokeMethod('setNativeCallBridge', {'enabled': enabled}); + Logger.w('AzureAST', '[STS] 原生直连桥 ${enabled ? "开启" : "关闭"}: ${ok ?? false}'); + return ok ?? false; + } on MissingPluginException { + return false; + } catch (e) { + Logger.error('setNativeCallBridge($enabled) 失败: $e'); + return false; + } + } + + @override + Future> nativeCallBridgeStats() async { + try { + final raw = await _channel.invokeMethod('getNativeCallBridgeStats'); + return raw == null ? const {} : Map.from(raw); + } catch (_) { + return const {}; + } + } + @override Future pushExternalAudio({ required String leg, diff --git a/apps/client/lib/data/services/wav_writer.dart b/apps/client/lib/data/services/wav_writer.dart index 8de2a964..b20a9142 100644 --- a/apps/client/lib/data/services/wav_writer.dart +++ b/apps/client/lib/data/services/wav_writer.dart @@ -36,8 +36,12 @@ class WavWriter { return w; } - /// 入队写一块 PCM。[onWritten] 在这一块真正落盘后回调(电平等只读用途)。 - void write(Uint8List pcm, {void Function()? onWritten}) { + /// 入队写一块 PCM。 + /// + /// ⚠️ 这里刻意**没有**"写完回调"。它原本是给电平/波形用的,等于把 UI 挂在 + /// 文件写入队列的尾巴上:队列一堵,界面跟着停;写失败时波形还会整段消失。 + /// 显示所需的电平在调用方直接从这块 PCM 上算,与落盘无关。 + void write(Uint8List pcm) { if (_closed) return; _chain = _chain.then((_) async { if (_closed) return; @@ -45,7 +49,6 @@ class WavWriter { await _raf.writeFrom(pcm); dataBytes += pcm.length; writtenFrames++; - onWritten?.call(); } catch (e) { droppedFrames++; Logger.e(_tag, '写入 PCM 失败(已丢帧 $droppedFrames): $e'); diff --git a/apps/client/lib/data/utils/background_keepalive.dart b/apps/client/lib/data/utils/background_keepalive.dart new file mode 100644 index 00000000..423365ad --- /dev/null +++ b/apps/client/lib/data/utils/background_keepalive.dart @@ -0,0 +1,92 @@ +import 'dart:async' show unawaited; +import 'dart:io' show Platform; + +import 'package:audio_session/audio_session.dart'; +import 'package:flutter/foundation.dart' show kIsWeb; +import 'package:just_audio/just_audio.dart'; + +import '../../core/utils/logger.dart'; + +/// iOS 后台保活:循环播一段**静音**,让 `UIBackgroundModes=audio` 有个成立的理由。 +/// +/// ## 为什么需要它 +/// 设备按键唤醒的 AI 会话,音频全走耳机(BLE),手机侧一个 AVAudioSession 都没激活; +/// 而 Info.plist 里只有 `bluetooth-central / audio / location`——光靠 bluetooth-central, +/// 锁屏后系统只给短暂的唤醒窗口,撑不住一段几十秒、每秒 50 包上行 + 一条 WebSocket +/// 的会话。Android 有 `AudioForegroundService`,iOS 没有等价物,这是 iOS 上让进程 +/// 在锁屏后活下去的标准做法(2026-09-20 苹果测试「锁屏后耳机对话很容易中断」)。 +/// +/// - 用引用计数:多条链路同时要保活时,最后一个 release 才真的停; +/// - 类别 `playback` + `mixWithOthers`:不打断别的 App 的声音,也不抢路由; +/// - 音量 0 + 文件本身就是静音,用户听不到任何东西; +/// - Android / Web 全是空操作。 +class BackgroundKeepalive { + BackgroundKeepalive._(); + + static const String _tag = 'BackgroundKeepalive'; + static const String _asset = 'assets/audio/silence_1s.wav'; + + static AudioPlayer? _player; + static int _refs = 0; + + static bool get _apply => !kIsWeb && Platform.isIOS; + + static Future acquire(String owner) async { + if (!_apply) return; + _refs++; + if (_refs > 1) return; + try { + final session = await AudioSession.instance; + // ⚠️ **只在当前类别撑不住后台音频时才改它**。 + // + // 无条件 configure 成 .playback 会把通话翻译/多媒体翻译正在跑的 + // .playAndRecord 录音链路踩掉(同 SubtitlePip 的 prepareAudioSessionForPip, + // 那边栽过一次)。而 .playAndRecord / .multiRoute 本身就满足 + // UIBackgroundModes=audio 的要求,不需要降级——这时只要把静音循环播起来, + // 让「确实有音频在播」这个条件成立就够了。 + final cur = session.configuration?.avAudioSessionCategory; + final alreadyOk = cur == AVAudioSessionCategory.playback || + cur == AVAudioSessionCategory.playAndRecord || + cur == AVAudioSessionCategory.multiRoute; + if (!alreadyOk) { + await session.configure(const AudioSessionConfiguration( + avAudioSessionCategory: AVAudioSessionCategory.playback, + avAudioSessionCategoryOptions: + AVAudioSessionCategoryOptions.mixWithOthers, + avAudioSessionMode: AVAudioSessionMode.defaultMode, + )); + } else { + Logger.i(_tag, '当前音频类别 $cur 已支持后台音频,不改类别'); + } + await session.setActive(true); + final p = AudioPlayer(); + await p.setAsset(_asset); + await p.setLoopMode(LoopMode.one); + await p.setVolume(0); + unawaited(p.play()); + _player = p; + Logger.i(_tag, '后台保活已开启($owner)'); + } catch (e) { + // 起不来不阻断会话本身:只是锁屏后可能被挂起,前台使用不受影响 + Logger.w(_tag, '后台保活开启失败($owner): $e'); + _refs--; + } + } + + static Future release(String owner) async { + if (!_apply) return; + if (_refs == 0) return; + _refs--; + if (_refs > 0) return; + final p = _player; + _player = null; + if (p == null) return; + try { + await p.stop(); + await p.dispose(); + Logger.i(_tag, '后台保活已关闭($owner)'); + } catch (e) { + Logger.w(_tag, '后台保活关闭失败: $e'); + } + } +} diff --git a/apps/client/lib/data/utils/mic_pcm_streamer.dart b/apps/client/lib/data/utils/mic_pcm_streamer.dart index 3664626b..93d29758 100644 --- a/apps/client/lib/data/utils/mic_pcm_streamer.dart +++ b/apps/client/lib/data/utils/mic_pcm_streamer.dart @@ -168,10 +168,30 @@ class MicPcmStreamer { await start(); } + /// 音浪发布间隔与窗口峰值。原始块由平台决定大小(Android 几百字节~2KB + /// = 15~60ms 一块),块块都发就是每秒几十次 Rx 通知,而屏幕最多 60Hz、 + /// 音浪控件也没那么灵敏。取窗口峰值而不是最后一块:说话的尖峰常常只占 + /// 一两块,按最后一块取样会把音浪削平。 + static const Duration _levelEmitInterval = Duration(milliseconds: 100); + double _levelPeak = 0; + DateTime? _levelEmitAt; + + void _emitLevel(Uint8List chunk) { + if (_levelCtl.isClosed) return; + final v = PcmStreamPlayer.rmsLevel(chunk); + if (v > _levelPeak) _levelPeak = v; + final now = DateTime.now(); + final last = _levelEmitAt; + if (last != null && now.difference(last) < _levelEmitInterval) return; + _levelEmitAt = now; + _levelCtl.add(_levelPeak); + _levelPeak = 0; + } + void _onChunk(Uint8List chunk) { if (chunk.isEmpty) return; _lastChunkAt = DateTime.now(); - if (!_levelCtl.isClosed) _levelCtl.add(PcmStreamPlayer.rmsLevel(chunk)); + _emitLevel(chunk); _buffer.add(chunk); while (_buffer.length >= _packetBytes) { @@ -211,6 +231,9 @@ class MicPcmStreamer { _emit(_buffer.takeBytes()); } _buffer.clear(); + // 下一轮从零开始计峰值,并让首块立刻出一次电平 + _levelPeak = 0; + _levelEmitAt = null; if (!_levelCtl.isClosed) _levelCtl.add(0); } diff --git a/apps/client/lib/data/utils/pcm_stream_player.dart b/apps/client/lib/data/utils/pcm_stream_player.dart index 920eef46..42a157c4 100644 --- a/apps/client/lib/data/utils/pcm_stream_player.dart +++ b/apps/client/lib/data/utils/pcm_stream_player.dart @@ -12,17 +12,21 @@ import 'emai_audio_session.dart'; /// 流式 PCM 播放器:一边收字节一边出声,用于百炼下行的 TTS 音频。 /// -/// 实现方式是给 just_audio 喂一个「长度未知」的 [StreamAudioSource]: -/// just_audio 内部会起一个本地代理 HTTP 服务,`contentLength == null` 时 -/// 它按 chunked 传输回给播放器,ExoPlayer / AVPlayer 都当直播流处理, -/// 于是首包到达就能出声,不用等整段合成完。 +/// **默认走原生播放器**(`azure_speech/pcm_player`,iOS `AVAudioEngine` / +/// Android `AudioTrack`,两端方法名逐字一致),PCM 直接进音频硬件。 +/// 只有原生不可用(老包没有这条通道)时才退回下面那套 just_audio 的路径。 +/// 两条路的取舍见 [_useNative]。 /// -/// WAV 头里的 RIFF/data 长度写 [_unknownSize](0xFFFFFFFF)—— -/// 这是流式 WAV 的通行写法,表示"长度未知,读到流结束为止"。 -/// 写真实长度是不可能的:合成还没结束,长度就还不知道。 +/// ## 退路:just_audio 的「长度未知」流 +/// 给 just_audio 喂一个 [StreamAudioSource],它内部起一个**本地代理 HTTP 服务**, +/// `contentLength == null` 时按 chunked 传输回给播放器,ExoPlayer 当直播流处理, +/// 于是首包到达就能出声。WAV 头里的 RIFF/data 长度写 [_unknownSize](0xFFFFFFFF)—— +/// 流式 WAV 的通行写法,表示"长度未知,读到流结束为止";写真实长度是不可能的, +/// 合成还没结束长度就还不知道。 /// -/// 一轮回复对应一个 source:[begin] → 若干次 [feed] → [end]。 -/// [end] 只是关掉输入流,播放器会把已经收到的字节播完再报 [onCompleted]。 +/// 一轮回复对应一轮播放:[begin] → 若干次 [feed] → [end]。 +/// [end] 只是告诉播放器没有更多数据了,它会把已经收到的字节**播完**再报 +/// [onCompleted](原生侧按已渲染帧数判定,不是队列排空就算完,否则尾音会被切)。 /// 打断走 [stopNow],立即断流并停止播放。 class PcmStreamPlayer { PcmStreamPlayer({ @@ -46,12 +50,33 @@ class PcmStreamPlayer { /// 必须 playAndRecord,不能改。 final bool playbackOnly; - /// iOS 专用的原生 PCM 播放通道,见 [_iosNative] 的说明 - static const MethodChannel _iosChannel = + /// 原生 PCM 播放通道(两端同名同方法,见 [_useNative] 的说明) + static const MethodChannel _nativeChannel = MethodChannel('azure_speech/pcm_player'); - /// iOS 上不能用 just_audio 播这条流(原因见 [_iosNative]) - bool get _iosNative => !kIsWeb && Platform.isIOS; + /// 这个包里有没有原生播放器。老包没有这条通道 → `MissingPluginException`, + /// 置 false 之后本进程一律退回 just_audio,功能不受影响。 + static bool _nativeUnavailable = false; + + /// 走原生播放器而不是 just_audio。 + /// + /// - **iOS 上是被迫的**:AVPlayer 放不了 just_audio 代理服务给的那种 + /// 「长度未知的 chunked WAV」,只发一次 `Range: bytes=0-1` 的嗅探请求就判定 + /// 资源不可播、回 `-11850 Operation Stopped`,一次都没真正起播。 + /// - **Android 上是为了少绕一圈**:ExoPlayer 不发那个嗅探请求,所以老路能用, + /// 但一条实时语音流要先经过 just_audio 内部起的**本地代理 HTTP 服务** + /// (chunked 编码 → localhost TCP → ExoPlayer)。白白多一次拷贝、一次 HTTP + /// 分帧和 ExoPlayer 自己的起播缓冲。直接喂 AudioTrack 更短也更快出声。 + /// + /// ⚠️ 只支持**单声道**:两端的原生播放器都是按 mono 建的链路 + /// (iOS 的 AVAudioFormat、Android 的 CHANNEL_OUT_MONO)。真出现双声道用法时 + /// 退回 just_audio,而不是把交织 PCM 当单声道播成两倍语速。 + /// 当前两个调用点(EMAI 24k、车载香薰 16k)都是单声道。 + bool get _useNative => + !kIsWeb && + !_nativeUnavailable && + channels == 1 && + (Platform.isIOS || Platform.isAndroid); final AudioPlayer _player = AudioPlayer(); _LivePcmSource? _source; @@ -68,7 +93,7 @@ class PcmStreamPlayer { /// begin 期间到达的音频,等 source 建好再补进去,避免开头被吞 final _pending = BytesBuilder(copy: false); - bool get isPlaying => _iosNative ? _active : _player.playing; + bool get isPlaying => _useNative ? _active : _player.playing; final _levelCtl = StreamController.broadcast(); @@ -116,19 +141,13 @@ class PcmStreamPlayer { if (_active) await _teardown(); await _ensureSession(); - if (_iosNative) { - _iosChannel.setMethodCallHandler(_onIosCall); - final ok = await _iosChannel - .invokeMethod('start', {'sampleRate': sampleRate}); - if (ok != true) { - Logger.e(_tag, 'iOS 原生播放器启动失败'); - return; - } - _active = true; - if (_pending.length > 0) { - await _iosChannel.invokeMethod('feed', {'data': _pending.takeBytes()}); - } - return; + if (_useNative) { + final started = await _startNative(); + if (started) return; + // 起不来的原因有两种:老包没有这条通道(已置 _nativeUnavailable, + // 下面直接走 just_audio),或者原生自己失败了(那就这一轮没声音)。 + if (!_nativeUnavailable) return; + Logger.w(_tag, '原生播放器不可用,本进程退回 just_audio'); } final src = _LivePcmSource(_wavHeader()); @@ -166,9 +185,9 @@ class PcmStreamPlayer { if (pcm.isEmpty) return; _emitLevel(pcm); - if (_iosNative) { + if (_useNative) { if (_active) { - _iosChannel.invokeMethod('feed', {'data': pcm}); + _nativeChannel.invokeMethod('feed', {'data': pcm}); } else { _pending.add(pcm); if (!_starting) unawaited(begin()); @@ -189,7 +208,7 @@ class PcmStreamPlayer { /// 本轮下行结束:关掉输入流,剩余缓冲播完后会触发 [onCompleted]。 /// 这一轮一个音频字节都没有时直接报完成——不然调用方会一直等一个不会来的回调。 Future end() async { - if (_iosNative) { + if (_useNative) { // 起播还在路上就先等它:不等的话短回复会被判成「没有音频」 if (_starting) { try { @@ -201,7 +220,7 @@ class PcmStreamPlayer { _emitCompleted(); return; } - await _iosChannel.invokeMethod('end'); + await _nativeChannel.invokeMethod('end'); return; } final src = _source; @@ -213,8 +232,33 @@ class PcmStreamPlayer { await src.close(); } + /// 起原生播放器。返回 false 且 [_nativeUnavailable] 为 true 时表示这个包 + /// 压根没有原生播放器,调用方应改走 just_audio。 + Future _startNative() async { + try { + _nativeChannel.setMethodCallHandler(_onNativeCall); + final ok = + await _nativeChannel.invokeMethod('start', {'sampleRate': sampleRate}); + if (ok != true) { + Logger.e(_tag, '原生播放器启动失败'); + return false; + } + _active = true; + if (_pending.length > 0) { + await _nativeChannel.invokeMethod('feed', {'data': _pending.takeBytes()}); + } + return true; + } on MissingPluginException { + _nativeUnavailable = true; + return false; + } catch (e) { + Logger.e(_tag, '原生播放器启动异常: $e'); + return false; + } + } + /// 原生播放器回调:本轮排队的 PCM 全部播完 - Future _onIosCall(MethodCall call) async { + Future _onNativeCall(MethodCall call) async { if (call.method == 'onCompleted') _finish(); return null; } @@ -222,6 +266,7 @@ class PcmStreamPlayer { /// 打断:立刻断流停声,不报完成 Future stopNow() async { await _teardown(); + _resetLevelWindow(); if (!_levelCtl.isClosed) _levelCtl.add(0); } @@ -229,9 +274,9 @@ class PcmStreamPlayer { await _teardown(); await _levelCtl.close(); await _completedCtl.close(); - if (_iosNative) { + if (_useNative) { try { - await _iosChannel.invokeMethod('dispose'); + await _nativeChannel.invokeMethod('dispose'); } catch (_) {} return; } @@ -241,9 +286,9 @@ class PcmStreamPlayer { Future _teardown() async { _active = false; _pending.clear(); - if (_iosNative) { + if (_useNative) { try { - await _iosChannel.invokeMethod('stop'); + await _nativeChannel.invokeMethod('stop'); } catch (_) {} return; } @@ -268,6 +313,7 @@ class PcmStreamPlayer { _errSub = null; _source = null; _emitCompleted(); + _resetLevelWindow(); if (!_levelCtl.isClosed) _levelCtl.add(0); } @@ -275,9 +321,32 @@ class PcmStreamPlayer { if (!_completedCtl.isClosed) _completedCtl.add(null); } + /// 音浪发布间隔。下行包到得比屏幕刷新还密,包包都发等于让 + /// 界面在两帧画面之间白白重建好几次。 + static const Duration _levelEmitInterval = Duration(milliseconds: 100); + + /// 本窗口内的峰值 / 上次发布时刻 + double _levelPeak = 0; + DateTime? _levelEmitAt; + + /// 取窗口峰值而不是"最后一包":说话的尖峰往往只占一两包, + /// 按最后一包取样会把音浪削平。 void _emitLevel(Uint8List pcm) { if (_levelCtl.isClosed) return; - _levelCtl.add(rmsLevel(pcm)); + final v = rmsLevel(pcm); + if (v > _levelPeak) _levelPeak = v; + final now = DateTime.now(); + final last = _levelEmitAt; + if (last != null && now.difference(last) < _levelEmitInterval) return; + _levelEmitAt = now; + _levelCtl.add(_levelPeak); + _levelPeak = 0; + } + + /// 一轮结束/打断时把窗口清掉,下一轮从零开始计峰值 + void _resetLevelWindow() { + _levelPeak = 0; + _levelEmitAt = null; } /// 44 字节标准 WAV 头,长度字段填"未知" diff --git a/apps/client/lib/devices/bes/bes_bluetooth_service.dart b/apps/client/lib/devices/bes/bes_bluetooth_service.dart index c0363bb6..a61864d1 100644 --- a/apps/client/lib/devices/bes/bes_bluetooth_service.dart +++ b/apps/client/lib/devices/bes/bes_bluetooth_service.dart @@ -4,6 +4,7 @@ import 'dart:typed_data'; import 'package:bluetooth_manager/bluetooth_manager.dart' as bes; +import 'package:flutter/services.dart' show MissingPluginException; import 'package:get/get.dart'; import 'package:get_storage/get_storage.dart'; @@ -306,6 +307,10 @@ class BesBluetoothService extends GetxService { // 把用户挡在门外,而电话其实一直在通着;要等耳机下一次推真实状态帧 // (0x02/0x8E)才恢复,表现就是「关掉再点提示未在通话中,再点一次才进得去」。 case 'startMakeCall': + // 挂断后被这一帧翻回 true 是已知可疑场景,把上一个状态一起打出来便于对账 + if (!isCallOngoing.value) { + Logger.w(_tag, '[通话状态] false → true,触发帧 raw=[$hex]'); + } // ⚠️ 用 info 而不是 debug:这是排查「刚连上耳机就被判成通话中、 // 同传点不动」的关键线索——原生侧 `case 0x02` 对任何第二字节为 0x02 // 的帧都会上报 startMakeCall,且不做任何长度/上下文校验, @@ -317,6 +322,11 @@ class BesBluetoothService extends GetxService { Logger.i(_tag, 'isCallOngoing → false (type=$type, raw=[$hex])'); isCallOngoing.value = false; break; + // ⚠️ 观测点:挂断之后紧跟着又来一帧把状态翻回 true 的情况,真机上反复出现 + // (2026-09-21:`bb 03`/`bb 04` 之后同一秒内来 `bb 02 16 0f 01 2b 38 36 …`, + // 末尾那串是 ASCII 电话号码)。原生 `case 0x02` 对任何第二字节为 0x02 的帧 + // 都报 startMakeCall、不做长度与上下文校验,所以这帧很可能是**通话记录/号码 + // 上报**而不是「去电开始」。先只记录不改判定——改错了会让通话翻译进不去。 case 'startCall': case 'stopCall': // 我们自己发的进/退通话模式指令的 ACK,只记一笔,**不动通话状态**(见上) @@ -718,6 +728,85 @@ class BesBluetoothService extends GetxService { Future pushTranslationTtsPcm(String leg, Uint8List pcm) => _manager.pushTtsPcm(leg, pcm); + /// EMAI 会话 PCM 原生直连桥(bluetooth_manager 这一端的开关)。 + /// 开着时 AI 模式的耳机麦 PCM 不再进 Dart,直接去 azure_speech 的百炼客户端; + /// 合成音频也直接回到 AI 下行发送器。老原生(没有这个方法)返回 false。 + Future setNativeAiPcmBridge(bool enabled) async { + try { + return await _manager.setAiPcmBridge(enabled); + } on MissingPluginException { + return false; + } catch (e) { + Logger.e(_tag, '设置 AI 原生桥失败: $e'); + return false; + } + } + + /// 设备音频直接落盘(原生 `PcmWavRecorder`):PCM 在原生解码完就地写 WAV, + /// 不再逐帧经 EventChannel 进 Dart。老原生没有这个方法 → 返回 false, + /// 调用方退回 [audioStream] / [micPcmStream] + Dart 落盘。 + Future startNativeFileRecorder({ + required String id, + required String path, + required int channels, + String? source, + String? left, + String? right, + }) async { + try { + return await _manager.startFileRecorder( + id: id, + path: path, + channels: channels, + source: source, + left: left, + right: right, + ); + } on MissingPluginException { + return false; + } catch (e) { + Logger.e(_tag, '启动原生录音失败: $e'); + return false; + } + } + + /// 停止原生录音并回填 WAV 头。null = 这个 id 已经不在了(断连时原生自己收尾过)。 + Future?> stopNativeFileRecorder(String id, + {bool save = true}) async { + try { + return await _manager.stopFileRecorder(id, save: save); + } catch (e) { + Logger.e(_tag, '停止原生录音失败: $e'); + return null; + } + } + + /// 原生录音统计流:`{'id', 'bytes', 'level'}`,约 100ms 一条 + Stream get nativeRecorderEvents => _manager.onRecorderEvent(); + + /// 通话翻译 PCM 原生直连桥(bluetooth_manager 这一端的开关)。 + /// 开着时 [micPcmStream] / [spkPcmStream] **不会再有帧**——它们在原生层直接 + /// 去了翻译插件;译文 PCM 也不再需要 [pushTranslationTtsPcm]。 + /// 老原生(没有这个方法)返回 false,调用方据此退回 Dart 流。 + Future setNativeCallPcmBridge(bool enabled) async { + try { + return await _manager.setCallPcmBridge(enabled); + } on MissingPluginException { + return false; + } catch (e) { + Logger.w(_tag, 'setCallPcmBridge($enabled) 失败: $e'); + return false; + } + } + + Future> nativeCallPcmBridgeStats() async { + try { + return await _manager.getCallPcmBridgeStats(); + } catch (_) { + return const {}; + } + } + /// 进入通话模式(`AA 51`):耳机开始双路上行(对端 + 本端), /// mode=0x03 交织,原生解码后走 [audioStream]。通话录音/通话翻译都用它。 Future startCallMode() => sendData(BesCmd.frame(BesCmd.callModeStart)); diff --git a/apps/client/lib/devices/bes/bes_device_plugin.dart b/apps/client/lib/devices/bes/bes_device_plugin.dart index 2cf27a67..b1ba95b6 100644 --- a/apps/client/lib/devices/bes/bes_device_plugin.dart +++ b/apps/client/lib/devices/bes/bes_device_plugin.dart @@ -271,6 +271,10 @@ class BesDeviceSession extends DeviceSession { })); _workers.add(ever(_link.isCallOngoing, (on) { _feature(on ? DeviceFeatureKeys.callStarted : DeviceFeatureKeys.callEnded); + // 同时刷一次快照:DeviceHub 把 info.metadata['inCall'] 当作 inCall 的兜底真相 + // (事件会随 worker 一起被 dispose 掉,快照不会)。不发这一条,兜底就只能 + // 等下次电量/版本变化时顺带对齐,慢一拍。 + _infoUpdated(); })); void infoChanged(dynamic _) => _infoUpdated(); _workers.add(ever(_link.batteryLevel, (_) { @@ -306,6 +310,16 @@ class BesDeviceSession extends DeviceSession { }, onError: (e) => Logger.e(_tag, 'AI 事件流错误: $e')); } + @override + Future> diagnostics() async { + try { + final dl = await _link.manager.getCallDownlinkStats(); + return {'downlink': dl}; + } catch (e) { + return {'downlink': 'err:$e'}; + } + } + void _feature(String key, [Map data = const {}]) { if (_events.isClosed) return; _events.add(DeviceSessionEvent( @@ -458,6 +472,53 @@ class BesDeviceSession extends DeviceSession { } } + /// 原生落盘录音。route → 原生的音频类型串: + /// + /// | route | 类型 | 说明 | + /// |---|---|---| + /// | mic | `mainMicData` | 现场录音的主麦 | + /// | callLocal | `micData` | 通话本端麦 | + /// | callPeer | `spkData` | 通话对端声 | + /// + /// ⚠️ `mic` 这里取的是**带类型的主麦帧**,而不是 [openMic] 那条 + /// `_alternate(audioStream)`(裸流隔帧取)。隔帧取靠的是"订阅时恰好对上相位", + /// 中途开始订阅就会永久取到前馈麦——原生早就把两路标好了 mainMicData / + /// ffMicData,按类型取顺带把这类错相消掉。 + /// 金测试 `test/devices/bes_recorder_sources_test.dart` 逐条钉着这张表: + /// 串要与原生 `BluetoothManager` 发布的类型名逐字一致,改错不报任何错, + /// 只表现为「录音起来了但一个字节都没写进去」。 + static const Map recorderSources = { + AudioRoute.mic: 'mainMicData', + AudioRoute.callLocal: 'micData', + AudioRoute.callPeer: 'spkData', + }; + + @override + Future openFileRecorder({ + required String path, + required List routes, + }) async { + _requireReady(); + if (routes.isEmpty || routes.length > 2) return null; + final sources = []; + for (final r in routes) { + final s = recorderSources[r]; + if (s == null) return null; // 这一路没有对应的原生类型,让调用方退回 Dart 落盘 + sources.add(s); + } + final id = 'bes_${DateTime.now().microsecondsSinceEpoch}'; + final ok = await _link.startNativeFileRecorder( + id: id, + path: path, + channels: sources.length, + source: sources.length == 1 ? sources.first : null, + left: sources.length == 2 ? sources[0] : null, + right: sources.length == 2 ? sources[1] : null, + ); + if (!ok) return null; + return _BesFileRecorder(id, path, _link); + } + static Stream _alternate(Stream raw) { var take = true; return raw.where((_) { @@ -524,6 +585,24 @@ class BesDeviceSession extends DeviceSession { case DeviceFeatures.stateRefresh: await _link.refreshDeviceState(); return const {}; + case DeviceFeatures.callPcmNativeBridge: + // 不是发给耳机的命令,不进 featureCommands(金测试只管字节表)。 + // 原生没这个方法(老包)会返回 false → 业务层退回 Dart 流。 + final enabled = args['enabled'] == true; + final ok = await _link.setNativeCallPcmBridge(enabled); + if (!ok && enabled) { + throw DeviceException(DeviceErrorCode.notSupported, featureKey); + } + return {'enabled': ok && enabled}; + case DeviceFeatures.aiPcmNativeBridge: + // 同 callPcmNativeBridge:不是发给耳机的命令,不进 featureCommands。 + // 原生没这个方法(老包)会返回 false → 业务层退回 Dart 链路。 + final aiOn = args['enabled'] == true; + final aiOk = await _link.setNativeAiPcmBridge(aiOn); + if (!aiOk && aiOn) { + throw DeviceException(DeviceErrorCode.notSupported, featureKey); + } + return {'enabled': aiOk && aiOn}; case 'bes.probe': await _link.probeCommands( from: (args['from'] as int?) ?? 0x21, @@ -635,3 +714,53 @@ class _BesCallSink implements DeviceAudioSink { await _onClose(); } } + +/// [DeviceFileRecorder] 的恒玄实现:只是原生录制器的一个句柄。 +/// 音频一个字节都不经过这里——它只转发统计、并在结束时让原生回填 WAV 头。 +class _BesFileRecorder implements DeviceFileRecorder { + _BesFileRecorder(this._id, this.path, this._link) { + _sub = _link.nativeRecorderEvents + .where((e) => e['id'] == _id) + .listen((e) { + final b = (e['bytes'] as num?)?.toInt() ?? _bytes; + _bytes = b; + if (!_ctrl.isClosed) { + _ctrl.add(DeviceRecordStats( + bytes: b, + level: (e['level'] as num?)?.toDouble() ?? 0, + )); + } + }); + } + + final String _id; + @override + final String path; + final BesBluetoothService _link; + + StreamSubscription? _sub; + final StreamController _ctrl = + StreamController.broadcast(); + int _bytes = 0; + bool _stopped = false; + + @override + Stream get stats => _ctrl.stream; + + @override + int get bytesWritten => _bytes; + + @override + Future stop({bool save = true}) async { + if (_stopped) return; + _stopped = true; + await _sub?.cancel(); + _sub = null; + final r = await _link.stopNativeFileRecorder(_id, save: save); + // null = 原生已经自己收尾过(设备断连那条路径),这时最后一次统计里的 + // 字节数就是最终值,别把它清零 + final b = (r?['bytes'] as num?)?.toInt(); + if (b != null) _bytes = b; + await _ctrl.close(); + } +} diff --git a/apps/client/lib/devices/device_hub.dart b/apps/client/lib/devices/device_hub.dart index a682dec5..ea099cf8 100644 --- a/apps/client/lib/devices/device_hub.dart +++ b/apps/client/lib/devices/device_hub.dart @@ -244,14 +244,27 @@ class DeviceHub extends GetxService { break; case DeviceSessionEventType.feature: final key = e.feature?.key; - if (key == DeviceFeatureKeys.callStarted) inCall.value = true; - if (key == DeviceFeatureKeys.callEnded) inCall.value = false; + if (key == DeviceFeatureKeys.callStarted) { + _setInCall(true, 'callStarted/${s.vendor}'); + } + if (key == DeviceFeatureKeys.callEnded) { + _setInCall(false, 'callEnded/${s.vendor}'); + } break; default: break; } } + /// inCall 的唯一写入口。这个值决定主界面那道「通话中禁用同传/面对面/音视频」 + /// 的闸门(DeviceConnection.isBlockedByCall),卡住了用户就进不去, + /// 所以每次翻转都留一条 warn —— 之前它一条日志都没有,排查时是个纯盲区。 + void _setInCall(bool v, String why) { + if (inCall.value == v) return; + Logger.w(_tag, 'inCall ${inCall.value} → $v ($why)'); + inCall.value = v; + } + void _recompute() { anyConnected.value = sessions.isNotEmpty; final caps = {}; @@ -262,7 +275,18 @@ class DeviceHub extends GetxService { final p = primary; primaryInfo.value = p?.info; primaryName.value = p?.info.name ?? ''; - if (p == null) inCall.value = false; + // 通话状态以**会话快照**为准,feature 事件只是加速通道。 + // + // ⚠️ 只靠事件会漏:worker 在 _markDisconnected() 里被整批 dispose,断连那一刻 + // 之后的状态变化发不出来;而重连建的新 session 里 `ever` 注册时又**不会回调当前值**, + // 于是 inCall 停在上一轮的旧值。偏偏之后耳机每次刷新状态都是重复赋同一个值, + // RxBool 去重 → ever 再也不触发 → **永远不会自愈**,表现就是 + // 「电话早挂了,同声翻译还一直点不进去」。快照每次 _recompute 都会重新对齐。 + if (p == null) { + _setInCall(false, 'no-session'); + } else { + _setInCall(p.info.metadata['inCall'] == true, 'snapshot/${p.vendor}'); + } revision.value++; } diff --git a/apps/client/lib/devices/smartcar/smartcar_device_plugin.dart b/apps/client/lib/devices/smartcar/smartcar_device_plugin.dart index 0e965084..f25e434d 100644 --- a/apps/client/lib/devices/smartcar/smartcar_device_plugin.dart +++ b/apps/client/lib/devices/smartcar/smartcar_device_plugin.dart @@ -658,6 +658,22 @@ class SmartcarSession implements DeviceSession { @override Stream get wakeEvents => _wake.stream; + @override + Future openFileRecorder({ + required String path, + required List routes, + }) async => + // 香薰没有录音场景(上行只在唤醒后送给助手),也没有原生落盘器。 + // 返回 null 让调用方退回 openMic + Dart 写盘,行为与改动前一致。 + null; + + @override + // 香薰这条链路没有通话翻译下行,没有可诊断的计数。 + // ⚠️ 这个类是 `implements DeviceSession` 而不是 extends,接口上的默认实现 + // 对它不生效,契约加成员时必须同步补一份(否则编译期直接报 + // 「missing implementations」)。 + Future> diagnostics() async => const {}; + @override Future openMic({AudioRoute route = AudioRoute.mic}) async { _requireReady(); diff --git a/apps/client/lib/modules/home/controllers/home_controller.dart b/apps/client/lib/modules/home/controllers/home_controller.dart index c3e705e1..44c77e35 100644 --- a/apps/client/lib/modules/home/controllers/home_controller.dart +++ b/apps/client/lib/modules/home/controllers/home_controller.dart @@ -145,6 +145,12 @@ class HomeController extends GetxController { return; } if (DeviceConnection.isBlockedByCall(translationId)) { + // ⚠️ 这条闸门卡住过用户好几次(「电话早挂了,同传还是点不进去」), + // 而 inCall 的来路很长:耳机帧 → isCallOngoing → feature 事件 → DeviceHub.inCall。 + // 被拦时把当时的判据原样打出来,省得再靠推演。 + Logger.w('HomeController', + '[闸门] $translationId 被拦: inCall=${DeviceConnection.inCall} ' + 'anyDevice=${DeviceConnection.any} device=${DeviceConnection.connectedName}'); ToastThrottle.show(ToastThrottle.keyInCallBusy, title: 'inCallBusyTitle'.tr, message: 'inCallBusyDesc'.tr, diff --git a/apps/client/lib/modules/meeting/controllers/meeting_details_controller.dart b/apps/client/lib/modules/meeting/controllers/meeting_details_controller.dart index a7f8e9ab..ba93dc61 100644 --- a/apps/client/lib/modules/meeting/controllers/meeting_details_controller.dart +++ b/apps/client/lib/modules/meeting/controllers/meeting_details_controller.dart @@ -22,7 +22,7 @@ import '../model/meeting_model.dart'; /// 会议详情控制器 class MeetingDetailsController extends GetxController - with GetTickerProviderStateMixin { + with GetTickerProviderStateMixin, WidgetsBindingObserver { late TabController tabController; final RxInt currentIndex = 0.obs; @@ -62,7 +62,6 @@ class MeetingDetailsController extends GetxController RxInt editSpeakerTextId = 0.obs; String editSpeakerTextChange = ''; - int _everType = 99999; RxInt summaryNumberModifications = 0.obs; RxInt personnelNumberModifications = 0.obs; @@ -94,15 +93,21 @@ class MeetingDetailsController extends GetxController final Map args = Get.arguments ?? {}; id = args['id']; _meetingTaskService.meetingId = id; + WidgetsBinding.instance.addObserver(this); _getDataDetails(); _loadJS(); _tasktypeListener = ever(_meetingTaskService.tasktype, (value) { meetingDetails.value.tasktype = value; - if (_everType <= value) { - if (value >= 3 && value < 10000) { - textList.value = _meetingTaskService.textList; + // ⚠️ 只拿**有内容**的值。MeetingTaskService 那几个字段是「上一次轮询到 + // 本条记录时」填的,没轮询到过就是空的;原来无条件覆盖,refreshFromServer + // 刚拉回来的总结/转写会被这里的空值当场抹掉(本地库是对的,所以退出重进 + // 又有了)。原先那个 _everType 门槛也一并去掉:它是按「进页面时的状态」定的, + // 服务端跑得比本地快一步就把数据分支整个跳过。 + if (_meetingTaskService.meetingId == id) { + if (value >= 3 && value < 10000 && _meetingTaskService.textList.isNotEmpty) { + textList.value = List.from(_meetingTaskService.textList); } - if (value >= 5 && value < 10000) { + if (value >= 5 && value < 10000 && _meetingTaskService.summary.isNotEmpty) { meetingDetails.value.personnel = _meetingTaskService.personnel; meetingDetails.value.summary = _meetingTaskService.summary; meetingDetails.value.overview = _meetingTaskService.overview; @@ -110,11 +115,21 @@ class MeetingDetailsController extends GetxController } } meetingDetails.refresh(); + _syncProgressPolling(); }); } + @override + void didChangeAppLifecycleState(AppLifecycleState state) { + // 切后台回来补拉一次:全局轮询可能已被冻结/清空,这是最常见的「卡在转写中」场景 + if (state == AppLifecycleState.resumed) unawaited(refreshFromServer()); + } + @override void onClose() { + WidgetsBinding.instance.removeObserver(this); + _refreshTimer?.cancel(); + _refreshTimer = null; _tasktypeListener.dispose(); _currentDurationSubscription?.cancel(); _playerStateSubscription?.cancel(); @@ -139,7 +154,6 @@ class MeetingDetailsController extends GetxController // 只有里程碑才更新,存量记录可能还停在 0。不取这一份的话, // 一条正在转写的记录点进来会既没有等待态、也没有内容,只剩空白。 meetingDetails.value.tasktype = meetingData.value.tasktype; - _everType = meetingDetails.value.tasktype + 1; _meetingTaskService.tasktype.value = meetingDetails.value.tasktype; if (meetingDetails.value.tasktype == 5) { await _meetingTaskService.readTask(id); @@ -158,15 +172,54 @@ class MeetingDetailsController extends GetxController textList.value = List.from(speakerList); } isLoading.value = false; + _syncProgressPolling(); // 本地先出,服务端整条再覆盖(不 await:页面已经能用了) - unawaited(_refreshFromServer()); + unawaited(refreshFromServer()); + } + + // ---------- 进行中状态的自我刷新 ---------- + // + // 本页原来一辈子只向服务端拉一次数据,之后完全依赖 MeetingTaskService 那个它既不 + // 拥有、也无法重启的全局轮询:轮询队列会被首页退出 (stopTask) 清空、被 Synchrodata + // 整体替换、进程被冻结时定时器也不走——任何一条命中,页面就永远停在「正在转写…」, + // 退出重进才好(2026-09-20 苹果测试原话)。现在进行中就自己每 6s 拉一次, + // 到终态/失败态即停;切后台回来也补拉一次;占位图上还给了一个手动刷新。 + Timer? _refreshTimer; + bool _refreshing = false; + static const Duration _progressPollInterval = Duration(seconds: 6); + + bool get _inProgress { + final t = meetingDetails.value.tasktype; + return t > 0 && t < 5; + } + + void _syncProgressPolling() { + if (isClosed) return; + if (_inProgress) { + _refreshTimer ??= Timer.periodic(_progressPollInterval, (_) { + unawaited(refreshFromServer()); + }); + } else { + _refreshTimer?.cancel(); + _refreshTimer = null; + } } /// 用服务端整条记录刷新本地缓存与页面。 /// /// 列表接口只回骨架(见 Synchrodata),转写和总结**只从这里进本地库**: /// 每次进详情拉一次 `echomeet_getrecord`,服务端是真源,别的设备重新生成的总结、 - /// 改过的标题都在这一步跟过来。拉不到就沿用本地,不报错。 + /// 改过的标题都在这一步跟过来。拉不到就沿用本地,不报错。并发调用直接返回。 + Future refreshFromServer() async { + if (_refreshing || isClosed) return; + _refreshing = true; + try { + await _refreshFromServer(); + } finally { + _refreshing = false; + } + } + Future _refreshFromServer() async { dynamic resp; try { @@ -227,10 +280,24 @@ class MeetingDetailsController extends GetxController summaryNumberModifications.value++; } if (meetingDetails.value.tasktype != state) { + // 先把服务端拿到的内容喂给 MeetingTaskService,再推状态: + // 推状态会触发 onInit 里那个 ever 监听,它从 service 里取内容—— + // 不先喂的话它取到的是空值(见那里的说明)。 + if (_meetingTaskService.meetingId == id) { + if (state >= 3 && state < 10000 && textList.isNotEmpty) { + _meetingTaskService.textList = List.from(textList); + } + if (state >= 5 && state < 10000) { + _meetingTaskService.personnel = personnel; + _meetingTaskService.summary = summary; + _meetingTaskService.overview = overview; + } + } meetingDetails.value.tasktype = state; _meetingTaskService.tasktype.value = state; } meetingDetails.refresh(); + _syncProgressPolling(); } List>? _decodeSpeakers(dynamic raw) { @@ -500,9 +567,8 @@ class MeetingDetailsController extends GetxController String tid, ) async { EasyLoading.show(); - _everType = 2; final String audiourl = meetingData.value.audiourl; - await _meetingTaskService.addTask( + final bool accepted = await _meetingTaskService.addTask( id, audiourl, formlanguage, @@ -511,17 +577,26 @@ class MeetingDetailsController extends GetxController templateid, tid, ); + EasyLoading.dismiss(); + // ⚠️ 没受理(服务端拒绝 / 上传队列里找不到这条)就别把页面切成「等待转写」: + // 那样页面会永远停在占位图上等一个根本不存在的任务。 + if (!accepted) { + Logger.w('MeetingDetails', '生成任务未受理 id=$id,页面保持未处理'); + Get.snackbar('error'.tr, 'networkErrorRetry'.tr, + snackPosition: SnackPosition.TOP); + return; + } meetingDetails.value.tasktype = 1; meetingDetails.refresh(); - EasyLoading.dismiss(); + _syncProgressPolling(); } void refreshSummary(int templateid, String tid, String tolanguage) async { EasyLoading.show(); - _everType = 4; await _meetingTaskService.refreshTask(id, templateid, tid, tolanguage); meetingDetails.value.tasktype = 3; meetingDetails.refresh(); + _syncProgressPolling(); EasyLoading.dismiss(); } } diff --git a/apps/client/lib/modules/meeting/controllers/meeting_record_controller.dart b/apps/client/lib/modules/meeting/controllers/meeting_record_controller.dart index b563f692..24d27934 100644 --- a/apps/client/lib/modules/meeting/controllers/meeting_record_controller.dart +++ b/apps/client/lib/modules/meeting/controllers/meeting_record_controller.dart @@ -616,7 +616,8 @@ class MeetingRecordController extends GetxController // 通话录音:走设备的通话双路通路(需要 callAudioTap 能力), // 双声道交织 PCM(L=本端 / R=对端)直写 stereo WAV。 _jieliCallRecorder = DeviceStereoRecorder( - onWavePcm: _processAudioData, + onWavePcm: _processAudioData, // 退回 Dart 落盘时才会触发 + onLevel: _applyAmplitude, // 原生落盘时的波形来源 ); final ok = await _jieliCallRecorder!.start(filePath); if (!ok) { @@ -721,13 +722,21 @@ class MeetingRecordController extends GetxController /// 处理原始Uint8List音频数据 void _processAudioData(Uint8List audioData) { if (!isRecording.value) return; - final next = _calculateAmplitude(audioData); + _applyAmplitude(_calculateAmplitude(audioData)); + } + + /// 设备通话录音走原生落盘时 PCM 不进 Dart,电平由原生算好送来, + /// 从这里进入同一条平滑/采样链路。[next] 是 0~1 的线性 RMS。 + void _applyAmplitude(double next) { + if (!isRecording.value) return; if (_smoothedRms == 0.0) { _smoothedRms = next; } else { _smoothedRms = _smoothedRms * 0.8 + next * 0.2; } - rms.value = _smoothedRms; + // ⚠️ 这里只更新纯字段,不碰 Rx。原先每帧 `rms.value = _smoothedRms`, + // 设备通话双路是 50 次/s,而真正读它的 _appendWaveSample 由 + // [_waveformTick](40ms)驱动,中间那一半通知没有任何人用。 _latestRms = _smoothedRms; } @@ -751,6 +760,8 @@ class MeetingRecordController extends GetxController } void _appendWaveSample(double value) { + // rms 的唯一发布点,跟着波形同频(见 _processAudioData 的说明) + rms.value = value; final normalized = value.clamp(0.0, 1.0); final eased = sqrt(normalized); final data = waveformController.waveData; diff --git a/apps/client/lib/modules/meeting/views/bottomSheet/ai_bottom_sheet.dart b/apps/client/lib/modules/meeting/views/bottomSheet/ai_bottom_sheet.dart index 32c6134b..0e4401ba 100644 --- a/apps/client/lib/modules/meeting/views/bottomSheet/ai_bottom_sheet.dart +++ b/apps/client/lib/modules/meeting/views/bottomSheet/ai_bottom_sheet.dart @@ -91,15 +91,35 @@ class _AIBottomSheetState extends State { @override Widget build(BuildContext context) { final isDarkMode = Theme.of(context).brightness == Brightness.dark; - return SafeArea( - top: false, - child: Container( - padding: EdgeInsets.symmetric(horizontal: 12.w), - constraints: BoxConstraints( - maxHeight: 1.sh - 50.h - MediaQuery.of(context).viewInsets.bottom, - ), + // 高度算法(2026-09-20 修): + // - GetX 的 bottomSheet 已经在外层按键盘高度 Padding 过了,这里**不能再减** + // viewInsets.bottom,否则键盘一弹起抽屉就少了两个键盘那么高; + // - SafeArea 要放在受约束的 Container **里面**:放外面它的底部安全区是"额外加" + // 在 maxHeight 之上的,抽屉比屏幕只矮 18pt,标题行连同右上角的 X 整个埋进 + // 状态栏/灵动岛里——"X 点不了、只能往下拖"就是这么来的; + // - 顶部留出状态栏高度再加一点空隙,标题行永远在状态栏下面。 + // ⚠️ 不能从 MediaQuery.of(context) 读:GetX 的 bottomSheet 用 removePadding(removeTop) + // 把 padding.top **和 viewPadding.top 一起**抹成了 0(2026-09-20 真机截图:抽屉顶到 + // 屏幕最上沿,X 压在状态栏里)。要读就读窗口本身的安全区。 + final double topGap = + MediaQueryData.fromView(View.of(context)).padding.top + 8.h; + return Container( + constraints: BoxConstraints(maxHeight: 1.sh - topGap), + padding: EdgeInsets.symmetric(horizontal: 12.w), + child: SafeArea( + top: false, child: Column( children: [ + // 拖拽把手:明示"这是可以往下拖关掉的抽屉" + Container( + width: 36.w, + height: 4.w, + margin: EdgeInsets.only(top: 8.w), + decoration: BoxDecoration( + color: isDarkMode ? Colors.grey[600] : Colors.grey[300], + borderRadius: BorderRadius.circular(2.w), + ), + ), Container( padding: EdgeInsets.symmetric(vertical: 10.w), child: Row( diff --git a/apps/client/lib/modules/meeting/views/meeting_details_view.dart b/apps/client/lib/modules/meeting/views/meeting_details_view.dart index 1e794513..063d3d38 100644 --- a/apps/client/lib/modules/meeting/views/meeting_details_view.dart +++ b/apps/client/lib/modules/meeting/views/meeting_details_view.dart @@ -357,7 +357,7 @@ class MeetingDetailsView extends GetView { ? const SizedBox() : controller.meetingDetails.value.tasktype > 0 && controller.meetingDetails.value.tasktype < 5 - ? TranscribingPlaceholder( + ? TranscribingPlaceholder(onRefresh: controller.refreshFromServer, controller.meetingDetails.value.tasktype > 0 && controller.meetingDetails.value.tasktype < 3 @@ -368,13 +368,13 @@ class MeetingDetailsView extends GetView { Obx( () => controller.meetingDetails.value.tasktype > 0 && controller.meetingDetails.value.tasktype < 3 - ? TranscribingPlaceholder('transcribing'.tr) // 对应中文:正在转写... + ? TranscribingPlaceholder(onRefresh: controller.refreshFromServer, 'transcribing'.tr) // 对应中文:正在转写... : const SpeechTab(), ), Obx( () => controller.meetingDetails.value.tasktype > 0 && controller.meetingDetails.value.tasktype < 5 - ? TranscribingPlaceholder( + ? TranscribingPlaceholder(onRefresh: controller.refreshFromServer, controller.meetingDetails.value.tasktype > 0 && controller.meetingDetails.value.tasktype < 3 ? 'transcribing'.tr // 对应中文:正在转写... @@ -385,7 +385,7 @@ class MeetingDetailsView extends GetView { ? const MindMapTab() : controller.meetingDetails.value.tasktype == 0 ? const SizedBox() - : TranscribingPlaceholder(controller + : TranscribingPlaceholder(onRefresh: controller.refreshFromServer, controller .meetingDetails.value.tasktype > 0 && controller.meetingDetails.value.tasktype < 3 diff --git a/apps/client/lib/modules/meeting/views/tabs/overview_tab.dart b/apps/client/lib/modules/meeting/views/tabs/overview_tab.dart index 87e43f45..85d45f8e 100644 --- a/apps/client/lib/modules/meeting/views/tabs/overview_tab.dart +++ b/apps/client/lib/modules/meeting/views/tabs/overview_tab.dart @@ -18,22 +18,20 @@ class OverviewTab extends StatefulWidget { class _OverviewTabState extends State { final _controller = Get.find(); - List _dataList = []; - - @override - void initState() { - super.initState(); - if (_controller.meetingDetails.value.overview.isNotEmpty) { - _dataList = _controller.meetingDetails.value.overview.split('\n'); - } - } - @override Widget build(BuildContext context) { final isDarkMode = Theme.of(context).brightness == Brightness.dark; return Stack( children: [ - _dataList.isNotEmpty + // 概述从 meetingDetails **现读**:服务端刷新(refreshFromServer)、总结重跑、 + // 别的设备改过的内容都会随之重画。原来在 initState 里快照进一个 List, + // 之后永不更新——列表里显示「总结完成」、点进去却是空的、退出重进才有, + // 就是这一处(2026-09-20 修)。 + Obx(() { + final String overview = _controller.meetingDetails.value.overview; + final List dataList = + overview.isNotEmpty ? overview.split('\n') : const []; + return dataList.isNotEmpty ? Column( crossAxisAlignment: CrossAxisAlignment.start, children: [ @@ -72,7 +70,7 @@ class _OverviewTabState extends State { padding: EdgeInsets.symmetric(vertical: 10.w), child: Text( 'keyEventsAnalyzed'.tr.replaceAll('{count}', - '${_dataList.length}'), // 对应中文:分析出{count}个关键事件。 + '${dataList.length}'), // 对应中文:分析出{count}个关键事件。 style: TextStyle( color: isDarkMode ? Colors.grey[400] : Colors.grey, fontSize: 12.sp, @@ -81,16 +79,17 @@ class _OverviewTabState extends State { ), Expanded( child: ListView.builder( - itemCount: _dataList.length, + itemCount: dataList.length, padding: EdgeInsets.fromLTRB(10.w, 10.w, 10.w, 80.w), itemBuilder: (BuildContext context, int index) { - return _item(_dataList[index]); + return _item(dataList[index]); }, ), ), ], ) - : _empty(), + : _empty(); + }), Obx( () => Positioned( bottom: MediaQuery.of(context).padding.bottom + 20.w, diff --git a/apps/client/lib/modules/meeting/views/tabs/summary_tab.dart b/apps/client/lib/modules/meeting/views/tabs/summary_tab.dart index 066b589b..63d72625 100644 --- a/apps/client/lib/modules/meeting/views/tabs/summary_tab.dart +++ b/apps/client/lib/modules/meeting/views/tabs/summary_tab.dart @@ -25,6 +25,10 @@ class _SummaryTabState extends State { final _controller = Get.find(); late final Worker _personnelListener; + late final Worker _summaryListener; + + /// 上一次装进编辑器的总结原文;服务端刷新回来的内容和它一样就不重建编辑器。 + String _lastLoadedSummary = ''; final String _addressPrefix = '${'location'.tr}:'; // 对应中文:地点 String _addressText = ''; @@ -54,6 +58,15 @@ class _SummaryTabState extends State { ever(_controller.personnelNumberModifications, (value) { _initController(); }); + // 总结内容变了(服务端刷新 / 总结重跑 / 别的设备改过)就重装编辑器。 + // 原来只听与会人员那个信号,summaryNumberModifications 全项目只有思维导图在听, + // 于是第一次进详情、服务端把总结拉回来之后,本页还是空的(2026-09-20 修)。 + // 用户正在编辑(_summaryDirty)时不动它,别把没保存的改动冲掉。 + _summaryListener = ever(_controller.summaryNumberModifications, (value) { + if (_summaryDirty) return; + if (_controller.meetingDetails.value.summary == _lastLoadedSummary) return; + _initController(); + }); } void _initController() { @@ -65,6 +78,7 @@ class _SummaryTabState extends State { (_controller.meetingDetails.value.personnel.isNotEmpty ? _controller.meetingDetails.value.personnel : '[${'inputAttendees'.tr}]'); // 对应中文:输入与会人员 + _lastLoadedSummary = _controller.meetingDetails.value.summary; final markdown = const ParchmentMarkdownCodec() .decode(_controller.meetingDetails.value.summary); final fc = FleatherController(document: markdown); @@ -179,6 +193,7 @@ class _SummaryTabState extends State { @override void dispose() { _personnelListener.dispose(); + _summaryListener.dispose(); _saveSummary(); _summaryChangeSub?.cancel(); _fleatherController?.dispose(); diff --git a/apps/client/lib/modules/meeting/views/widgets/transcribing_placeholder.dart b/apps/client/lib/modules/meeting/views/widgets/transcribing_placeholder.dart index cd8a436f..7210eabf 100644 --- a/apps/client/lib/modules/meeting/views/widgets/transcribing_placeholder.dart +++ b/apps/client/lib/modules/meeting/views/widgets/transcribing_placeholder.dart @@ -2,6 +2,7 @@ import 'dart:math' as math; import 'package:flutter/material.dart'; import 'package:flutter_screenutil/flutter_screenutil.dart'; +import 'package:get/get.dart'; import '../../../../core/theme/app_palette.dart'; @@ -17,11 +18,15 @@ import '../../../../core/theme/app_palette.dart'; /// ② 颜色全部走 [AppPalette],明暗主题自动跟随,插画做不到; /// ③ 四个 tab(概览/发言/全文/思维导图)共用同一个等待态,自绘好调。 class TranscribingPlaceholder extends StatefulWidget { - const TranscribingPlaceholder(this.title, {super.key}); + const TranscribingPlaceholder(this.title, {super.key, this.onRefresh}); /// 状态文案,已含省略号(`transcribing` / `analyzing`),这里不要再补点。 final String title; + /// 手动刷新。给用户一个不退出页面也能自救的出口——全局轮询死了、 + /// 进程被冻结过,都不用再「退出重进」。null 时不显示按钮。 + final Future Function()? onRefresh; + @override State createState() => _TranscribingPlaceholderState(); @@ -74,12 +79,40 @@ class _TranscribingPlaceholderState extends State color: AppPalette.textSecondary(isDark), ), ), + if (widget.onRefresh != null) ...[ + SizedBox(height: 16.h), + TextButton.icon( + onPressed: _refreshing ? null : _onRefreshTap, + icon: _refreshing + ? SizedBox( + width: 14.w, + height: 14.w, + child: const CircularProgressIndicator(strokeWidth: 2), + ) + : Icon(Icons.refresh, size: 18.sp), + label: Text('meetingRefreshNow'.tr, // 刷新 + style: TextStyle(fontSize: 13.sp)), + ), + ], ], ), ), ); } + bool _refreshing = false; + + Future _onRefreshTap() async { + final cb = widget.onRefresh; + if (cb == null) return; + setState(() => _refreshing = true); + try { + await cb(); + } finally { + if (mounted) setState(() => _refreshing = false); + } + } + /// 声波:每根柱子给一个固定相位差,整排看起来像一道行进的波而不是随机跳动。 Widget _waveform(bool isDark) { return SizedBox( diff --git a/apps/client/lib/modules/translation/controllers/translation_controller.dart b/apps/client/lib/modules/translation/controllers/translation_controller.dart index 2b07f835..aac0a3da 100644 --- a/apps/client/lib/modules/translation/controllers/translation_controller.dart +++ b/apps/client/lib/modules/translation/controllers/translation_controller.dart @@ -8,6 +8,7 @@ import 'package:get/get.dart'; import 'package:get_storage/get_storage.dart'; import '../../../core/utils/audio_foreground_service.dart'; import '../../../data/utils/peer_speech_gate.dart'; +import '../../../data/utils/background_keepalive.dart'; import '../../../core/utils/recent_languages.dart'; import 'package:intl/intl.dart'; import 'package:floating_ui_plugin/native_plugin.dart'; @@ -99,6 +100,20 @@ class TranslationController extends GetxController with WidgetsBindingObserver { // 那不是「功能该结束了」,用户还在通话里。见 _handleAstDrop。 int _astReconnectAttempts = 0; bool _astReconnecting = false; + + /// 重连预算改成**时间窗**而不是次数(2026-09-20):原来固定 3 次、退避 1/2/4s, + /// 蜂窝网抖动频繁时 7 秒就烧光,然后整个通话翻译被终止——用户还在电话里。 + /// 现在:从第一次掉线起 [_astReconnectWindow] 内不限次数重连(退避封顶 + /// [_astReconnectMaxDelaySec]),收到终态结果就把窗口清零;超过窗口才真的终止。 + DateTime? _astDropWindowStart; + + /// 给界面看的链路状态(通话翻译页顶栏那颗小灯)。 + /// 原来用户只能靠「有没有字幕」猜链路是死是活,重连中也没有任何提示。 + final Rx astLinkState = AstLinkState.idle.obs; + + /// 通话翻译 PCM 是否走了原生直连桥(见 [_attachBesCallTranslationBridge])。 + /// true 时 Dart 不再搬任何一帧音频,[_peerGate] 也不参与(门限在原生)。 + bool _callBridgeNative = false; /// 对端(B 路)说话人门限,保证阿里刻到的第一段音频来自对端本人。 /// 见 [PeerSpeechGate];**每次 AST 会话重建都要 reset**,否则重连后照样刻到串音。 final PeerSpeechGate _peerGate = PeerSpeechGate(); @@ -107,7 +122,8 @@ class TranslationController extends GetxController with WidgetsBindingObserver { /// [didChangeAppLifecycleState] 靠它决定回到前台要不要补一次重连。 bool _astHealthy = false; Timer? _astReconnectTimer; - static const int _astMaxReconnects = 3; + static const Duration _astReconnectWindow = Duration(seconds: 60); + static const int _astReconnectMaxDelaySec = 8; final GetStorage _storage = GetStorage(); final _musiceManager = Get.find(); final _qqMusicManager = Get.find(); @@ -288,9 +304,49 @@ class TranslationController extends GetxController with WidgetsBindingObserver { // 小窗里只有一块黑的。现在改成自绘:把字幕画进 CVPixelBuffer 逐帧喂给 // AVSampleBufferDisplayLayer,小窗显示的就是它(见原生 SubtitlePipController)。 _restorePipStyle(); + SubtitlePip.onStateChanged = _onSubtitlePipStateChanged; isFloatingWindowEnabled.value = await SubtitlePip.isActive(); } + /// iOS 字幕小窗的真实状态(原生回传)。 + /// + /// iOS 全系统同时只允许一个画中画:用户在视频 App 里开了它自己的小窗,我们的就被 + /// 系统停掉。原来这里没有任何回调,标志位停在 stale-true——用户点开关走的是"关", + /// 要点两下才回来;而且没人再把小窗拉起来,"弹幕框就丢了"(2026-09-20 苹果测试)。 + /// 原生现在会自愈重试、并在下次退后台时再拉一次;这里只负责把标志校准、并在被 + /// 顶掉时告诉用户怎么恢复。 + void _onSubtitlePipStateChanged(String state) { + switch (state) { + case 'started': + isFloatingWindowEnabled.value = true; + break; + case 'stopped': + // 只有这一种是"我们自己关的"——原生只在 stop() 里发它。 + // 其余两种 wanted 仍为 true,开关不能跟着灭(见下面的说明)。 + isFloatingWindowEnabled.value = false; + break; + case 'failed': + // ⚠️ **不要把开关置成关**。前台调 start 本来就不会真正进小窗 + // (possible=true / active=false,并回调一次 failed,sample-buffer + // 画中画的正常行为),真正呈现是在 App 退到后台那一刻由系统拉起。 + // 置成关的后果是:默认开启之后用户立刻看到开关是灭的,以为没生效, + // 去点一下反而把原生的 wanted 关掉,自愈也跟着没了。 + Logger.i('Translation', '字幕小窗这次没拉起来(前台或被占用),保持"已开启"等待系统拉起'); + break; + case 'stoppedBySystem': + // 被别的 App 的画中画顶掉。原生的 wanted 仍是 true、还在退避重试, + // 并且会在本 App 下次退到后台时再拉一次——**用户什么都不用点**。 + // 所以开关保持"开",只给一句提示说明它为什么暂时不见了。 + if ((isRecognizing.value || isPreparing.value) && + _isFloatingWindowSupportedMode()) { + Get.snackbar('tip'.tr, 'pipTakenOverByOtherApp'.tr, + snackPosition: SnackPosition.TOP, + duration: const Duration(seconds: 5)); + } + break; + } + } + /// 切换悬浮窗功能 Future toggleFloatingWindow() async { if (isFloatingWindowEnabled.value) { @@ -452,6 +508,7 @@ class TranslationController extends GetxController with WidgetsBindingObserver { /// 异常:可能抛出底层服务调用异常 Future changeTranslationMode(String mode) async { currentMode.value = mode; + astLinkState.value = AstLinkState.idle; // 每次进/出面对面翻译都回到客户端触发;设备端触发由耳机按键在 // [setActiveSpeaker] 里重新标记,避免上一轮的设备状态残留下来 // 导致这一轮点开始按钮又不响应。 @@ -1090,6 +1147,9 @@ class TranslationController extends GetxController with WidgetsBindingObserver { // _astService.dispose(); unawaited(_disableFloatingWindow()); + if (SubtitlePip.onStateChanged == _onSubtitlePipStateChanged) { + SubtitlePip.onStateChanged = null; + } // 释放管理器资源 _timerManager.dispose(); @@ -1105,6 +1165,13 @@ class TranslationController extends GetxController with WidgetsBindingObserver { /// 当应用进入后台、失去焦点或分离时,兜底将 activeSpeaker 重置为 0 @override void didChangeAppLifecycleState(AppLifecycleState state) { + // 观测点:切后台会不会让通话翻译卡顿,光看识别/翻译日志分不出来—— + // 那两条在后台照样按时出结果,真正被 iOS 节流的是下行那条 20ms 定时器。 + // 所以每次切换都把当时的桥统计打一份,前后一对比就知道帧是在哪一段丢的。 + if (currentMode.value == 'call' && isRecognizing.value) { + Logger.w('Translation', '[生命周期] → $state(通话翻译进行中)'); + unawaited(_dumpCallBridgeStats('lifecycle:$state')); + } if (state == AppLifecycleState.inactive || state == AppLifecycleState.paused || state == AppLifecycleState.detached) { @@ -1129,9 +1196,35 @@ class TranslationController extends GetxController with WidgetsBindingObserver { _astReconnectTimer = null; _astReconnecting = false; _astReconnectAttempts = 0; // 用户回来了,重连预算给满 + _astDropWindowStart = null; unawaited(_handleAstDrop('回到前台后恢复')); } + /// 拉一次原生直连桥的实时统计并落日志。 + /// + /// 与会话结束时那条 `[CALL-BRIDGE] 原生直连桥统计` 同源,区别是可以**在会话中途** + /// 随时取样。framesA/framesB 的增量除以两次取样的间隔,就是上行的实际帧率 + /// (正常 50 帧/秒),掉下来就说明进程被挂起或节流了。 + Future _dumpCallBridgeStats(String why) async { + try { + final st = await _astService.nativeCallBridgeStats(); + if (st.isNotEmpty) { + Logger.w('Translation', '[CALL-BRIDGE] 上行取样($why): $st'); + } + // 下行(译文回灌耳机)单独取一份:远端听到的卡顿看的是这一组。 + // gapAvgMs 明显大于 intervalMs = iOS 把 20ms 定时器节流了; + // queueA 一路上涨 = 发得比产出慢。两者都只在这里看得到。 + // ⚠️ 走 DeviceSession.diagnostics() 而不是直接 import bluetooth_manager: + // lib/modules 不允许依赖厂商插件,device_hub_boundary_test 会 fail。 + final dl = await _callSession?.diagnostics() ?? const {}; + if (dl.isNotEmpty) { + Logger.w('Translation', '[DOWNLINK] 取样($why): $dl'); + } + } catch (e) { + Logger.w('Translation', '[CALL-BRIDGE] 取样($why) 失败: $e'); + } + } + /// 切换TTS功能开关 void toggleTts() => isTtsEnabled.toggle(); @@ -1387,6 +1480,21 @@ class TranslationController extends GetxController with WidgetsBindingObserver { } } + // iOS 音视频翻译:同样默认把字幕小窗开起来,不用用户再点一次开关。 + // 这个模式的使用场景本来就是"切到别的 App 看视频",小窗是唯一能看到 + // 译文的地方,默认关着等于默认不可用。 + // + // ⚠️ 与 Android 那条不同,**起不来不中断翻译**: + // 1) 前台调 start 本来就不会真正进小窗(见 _onSubtitlePipStateChanged + // 的 failed 分支),拿到 false 不代表功能坏了; + // 2) 别的 App 的画中画正占着时也会失败,而那时翻译本身完全正常。 + // Android 之所以 return,是因为没有系统悬浮窗就真的什么都看不见。 + if (Platform.isIOS && + currentMode.value == 'audioVideo' && + !isFloatingWindowEnabled.value) { + await _startPip(); + } + await _configureAudioMode(); // 设备端触发的面对面翻译不从这里启动——那是「按住说话」,由耳机左右耳 // 按键经 [setActiveSpeaker] 逐句拉起识别。客户端触发(App 上的开始按钮) @@ -1715,6 +1823,42 @@ class TranslationController extends GetxController with WidgetsBindingObserver { await _detachBesCallTranslationBridge(closeSinks: false); final session = _callSession; if (session == null) return; + + // ---- 首选:原生直连桥(两个原生插件之间直接传 PCM,Dart 一帧都不碰)---- + // + // 原来这条链路是 `恒玄插件 → EventChannel → Dart → MethodChannel → azure_speech`, + // 20ms 一帧、双路,每秒约 200 次跨界。长句一多 event loop 就被拖住,原生那边 + // 报 AudioSendSlow 1011——2026-09-20 苹果测试「WiFi 下断断续续、长句基本不行」 + // 就是它。桥开着时门限(PeerSpeechGate)也在原生做,重连期间的帧由原生缓存补推。 + // 任一端不支持(老包 / 别的厂商)就退回下面的 Dart 流路径,功能不受影响。 + if (await _tryEnableNativeCallBridge(session)) { + _callBridgeNative = true; + // iOS 后台保活。通话翻译的两路音频**全走耳机 GATT**,手机侧没有任何 + // AVAudioSession 的实际 I/O(译文也是发给耳机、不经扬声器),于是 + // UIBackgroundModes=audio 没有成立的理由,切后台几十秒后上行就被系统掐掉: + // WebSocket 还连着(所以译文照出),但服务端收不到可识别的语音,只能靠 + // 默认 VAD 每 30 秒强切一句,吐出空译文或幻觉 —— 真机表现就是 + // 「切后台一会儿,只剩译文、原文全没了」(2026-09-21 实测,23:02:33 起 + // 连流式的「识别中」都消失,之后每 30.0/29.5/30.4/31.0 秒一条空翻译)。 + // ⚠️ acquire 内部只在当前类别撑不住后台音频时才改类别,不会踩掉 .playAndRecord。 + unawaited(BackgroundKeepalive.acquire('callTranslation')); + int finals = 0; + _besCallAstTtsSub = _astService.astTtsFrames.listen( + (frame) { + // 桥模式下原生只回 final 标记(PCM 已直接下发耳机),这里只记账 + if (frame.isFinal) { + finals++; + Logger.info('[CALL-BRIDGE] TTS 段尾 leg=${frame.leg} (#$finals, 原生直发)'); + } + }, + onError: (e) => Logger.error('[CALL-BRIDGE] ast tts stream error: $e'), + ); + Logger.w('Translation', '[CALL-BRIDGE] 已连通 设备双路 ↔ AST(原生直连,不经 Dart)'); + return; + } + + // ---- 兜底:Dart 流路径(原样保留)---- + Logger.w('Translation', '[CALL-BRIDGE] 原生直连桥不可用,退回 Dart 流路径'); _callLocalSrc = await session.openMic(route: AudioRoute.callLocal); _callPeerSrc = await session.openMic(route: AudioRoute.callPeer); _besCallMicSub = _callLocalSrc!.pcm.listen( @@ -1759,7 +1903,79 @@ class TranslationController extends GetxController with WidgetsBindingObserver { Logger.info('[CALL-BRIDGE] 已连通 设备双路 ↔ AST'); } + /// 试着把两端的原生直连桥都打开;任一端失败就把已开的那端关回去,返回 false。 + /// + /// 顺序:先开翻译插件那端(它要先挂上收帧的槽/观察者),再开设备那端 + /// (它一开就开始往外推帧)。反过来会有帧无人接收。 + Future _tryEnableNativeCallBridge(DeviceSession session) async { + bool astOn = false; + try { + astOn = await _astService.setNativeCallBridge(true); + if (!astOn) return false; + final r = await session + .invokeFeature(DeviceFeatures.callPcmNativeBridge, {'enabled': true}); + if (r['enabled'] == true) return true; + } on DeviceException catch (e) { + Logger.w('Translation', '[CALL-BRIDGE] 设备端不支持原生直连桥: ${e.code}'); + } catch (e) { + Logger.w('Translation', '[CALL-BRIDGE] 开启原生直连桥失败: $e'); + } + if (astOn) { + try { + await _astService.setNativeCallBridge(false); + } catch (_) {} + } + return false; + } + + /// 收到终态结果:链路健康,重连预算与时间窗都归零。 + void _markAstLinkHealthy() { + _astReconnectAttempts = 0; + _astDropWindowStart = null; + if (currentMode.value == 'call' && isRecognizing.value) { + astLinkState.value = AstLinkState.connected; + } + } + + /// 把原生/底层的错误原文分成用户能看懂的两类:网络断了 vs 服务出错。 + /// 别把 `[NSPOSIXErrorDomain:53]`、`Software caused connection abort` 这种原样甩给用户。 + String _classifyAstFailure(String reason) { + final r = reason.toLowerCase(); + const netMarkers = [ + '1011', '1012', 'nsposixerrordomain', 'connection abort', 'econnaborted', + 'socket', 'network', 'timed out', 'timeout', 'host lookup', 'offline', + 'connection failed', 'urlsession', 'unreachable', + ]; + if (netMarkers.any(r.contains)) { + return 'networkConnectionFailed'.tr; // 网络连接失败,请检查网络设置后重试 + } + return 'asrConnectionFailed'.tr; // 语音识别服务连接失败,请检查网络或稍后重试 + } + Future _detachBesCallTranslationBridge({bool closeSinks = true}) async { + if (_callBridgeNative) { + _callBridgeNative = false; + // 先关翻译插件那端(停止收 tap/通知),再关耳机插件那端(恢复 Dart 路径); + // 反过来会有一小段「耳机在推、没人收」的空窗。统计打 warn:release 能看到。 + try { + final stats = await _astService.nativeCallBridgeStats(); + Logger.w('Translation', '[CALL-BRIDGE] 原生直连桥统计: $stats'); + } catch (_) {} + try { + await _astService.setNativeCallBridge(false); + } catch (e) { + Logger.error('[CALL-BRIDGE] 关闭翻译端原生桥失败(忽略): $e'); + } + final session = _callSession; + if (session != null && session.state == DeviceConnectionState.ready) { + try { + await session.invokeFeature( + DeviceFeatures.callPcmNativeBridge, {'enabled': false}); + } catch (e) { + Logger.error('[CALL-BRIDGE] 关闭设备端原生桥失败(忽略): $e'); + } + } + } _besCallMicSub?.cancel(); _besCallMicSub = null; _besCallSpkSub?.cancel(); @@ -1937,6 +2153,10 @@ class TranslationController extends GetxController with WidgetsBindingObserver { // 有事件进来就说明链路是活的(错误/取消除外,它们在下面各自置回 false) if (event.type != ASTEventType.error && event.type != ASTEventType.canceled) { _astHealthy = true; + if (currentMode.value == 'call' && isRecognizing.value && + astLinkState.value != AstLinkState.connected) { + astLinkState.value = AstLinkState.connected; + } } // 仅对终态事件打 info 日志,避免中间事件高频写 I/O 阻塞 event loop(会导致 AudioSendSlow 1011) if (event.type == ASTEventType.finalResult || @@ -1988,8 +2208,8 @@ class TranslationController extends GetxController with WidgetsBindingObserver { case ASTEventType.finalResult: if (event.text.isEmpty) break; // 有终态结果回来 = 链路是通的,把重连预算还回去(否则一通长电话里 - // 攒够 3 次零星抖动就再也不重连了) - _astReconnectAttempts = 0; + // 攒够几次零星抖动就再也不重连了) + _markAstLinkHealthy(); Logger.i('Translation', 'AST 识别完成 [${event.serviceId}]: ${event.text}'); currentSessionId ??= DateTime.now().millisecondsSinceEpoch.toString(); @@ -2139,6 +2359,11 @@ class TranslationController extends GetxController with WidgetsBindingObserver { /// 停止语音识别 Future stopRecognition() async { if (!isRecognizing.value) return; + // 正常结束:链路灯归位(错误终止那条路在 _terminateCallOnError 里已置成 disconnected) + if (currentMode.value == 'call' && + astLinkState.value != AstLinkState.disconnected) { + astLinkState.value = AstLinkState.idle; + } // ⚠️ 先翻状态再清理,别放到最后。 // 底下这串(停录音、关 ASR 的 WebSocket、归档)随时可能要好几秒,而 @@ -2155,7 +2380,7 @@ class TranslationController extends GetxController with WidgetsBindingObserver { _astReconnectTimer?.cancel(); _astReconnectTimer = null; _astReconnecting = false; - _astReconnectAttempts = 0; + _markAstLinkHealthy(); await _astService.stopContinuousTranslation(); _astAutoStartPending = false; } @@ -2665,14 +2890,22 @@ class TranslationController extends GetxController with WidgetsBindingObserver { Logger.i('Translation', '[STS] 已在重连中,忽略重复上报: $reason'); return; } - if (_astReconnectAttempts >= _astMaxReconnects) { - Logger.e('Translation', '[STS] 重连 $_astMaxReconnects 次仍失败,终止通话翻译: $reason'); + final now = DateTime.now(); + _astDropWindowStart ??= now; + final sinceFirstDrop = now.difference(_astDropWindowStart!); + if (sinceFirstDrop > _astReconnectWindow) { + Logger.e('Translation', + '[STS] 掉线后 ${sinceFirstDrop.inSeconds}s 内重连 $_astReconnectAttempts 次仍未恢复,终止通话翻译: $reason'); await _terminateCallOnError(reason); return; } _astReconnecting = true; _astReconnectAttempts++; - final delay = Duration(seconds: 1 << (_astReconnectAttempts - 1)); // 1s / 2s / 4s + astLinkState.value = AstLinkState.reconnecting; + // 1s / 2s / 4s / 8s / 8s …:窗口内不限次数,退避封顶 + final delay = Duration( + seconds: (1 << (_astReconnectAttempts - 1).clamp(0, 3)) + .clamp(1, _astReconnectMaxDelaySec)); Logger.w('Translation', '[STS] AST 掉线($reason),第 $_astReconnectAttempts 次重连将在 ${delay.inSeconds}s 后开始'); _astReconnectTimer?.cancel(); @@ -2701,8 +2934,19 @@ class TranslationController extends GetxController with WidgetsBindingObserver { _astReconnectTimer = null; _astReconnecting = false; _astReconnectAttempts = 0; + _astDropWindowStart = null; + astLinkState.value = AstLinkState.disconnected; _astEventSubscription?.cancel(); _astEventSubscription = null; + // 终止是用户可感知的:给一句**分得清是网断了还是服务挂了**的提示。 + // 原来不弹任何东西,用户只看到字幕不再更新;再往前是弹 Dio 的英文原文。 + if (reason.isNotEmpty && isRecognizing.value) { + Get.snackbar('error'.tr, _classifyAstFailure(reason), + snackPosition: SnackPosition.TOP, + backgroundColor: Colors.red.withOpacity(0.1), + colorText: Colors.red, + duration: const Duration(seconds: 4)); + } try { await _astService.stopContinuousTranslation(); } catch (_) {} @@ -2784,6 +3028,7 @@ class TranslationController extends GetxController with WidgetsBindingObserver { try { if (alive) await session.invokeFeature(DeviceFeatures.asrStopped); } catch (_) {} + unawaited(BackgroundKeepalive.release('callTranslation')); await _detachBesCallTranslationBridge(); // 关掉两路 sink = 停译文下行 if (!alive) return; try { @@ -3248,3 +3493,6 @@ class TranslationController extends GetxController with WidgetsBindingObserver { } } } + +/// 通话翻译(AST)链路状态,给顶栏指示灯用。 +enum AstLinkState { idle, connected, reconnecting, disconnected } diff --git a/apps/client/lib/modules/translation/views/translation_view.dart b/apps/client/lib/modules/translation/views/translation_view.dart index 35245f11..bd24b4cc 100644 --- a/apps/client/lib/modules/translation/views/translation_view.dart +++ b/apps/client/lib/modules/translation/views/translation_view.dart @@ -357,6 +357,52 @@ class TranslationView extends GetView { // 未录音:显示红色录音按钮 + 文件夹按钮 + 悬浮窗按钮 return Row( children: [ + // 通话翻译链路状态灯:已连接 / 重连中 / 已断开。 + // 原来用户只能靠「字幕还在不在更新」猜链路是不是活着,重连中也没有任何提示。 + Obx(() { + final mode = controller.currentMode.value; + final st = controller.astLinkState.value; + if (mode != 'call' || st == AstLinkState.idle) { + return const SizedBox.shrink(); + } + final Color color; + final String label; + switch (st) { + case AstLinkState.connected: + color = Colors.green; + label = 'connected'.tr; + break; + case AstLinkState.reconnecting: + color = Colors.orange; + label = 'connecting'.tr; + break; + default: + color = Colors.red; + label = 'unconnected'.tr; + } + return Padding( + padding: EdgeInsets.only(right: 4.w), + child: Row( + mainAxisSize: MainAxisSize.min, + children: [ + Container( + width: 8.w, + height: 8.w, + decoration: + BoxDecoration(color: color, shape: BoxShape.circle), + ), + SizedBox(width: 4.w), + Text( + label, + style: TextStyle( + fontSize: 11.sp, + color: isDarkMode ? Colors.white70 : Colors.black54, + ), + ), + ], + ), + ); + }), // 悬浮窗切换按钮(仅在支持的模式下显示) Obx(() { if (controller.currentMode.value == 'call' || diff --git a/apps/client/local_plugins/azure_speech/android/build.gradle.kts b/apps/client/local_plugins/azure_speech/android/build.gradle.kts index 1f0d19d0..57790d64 100644 --- a/apps/client/local_plugins/azure_speech/android/build.gradle.kts +++ b/apps/client/local_plugins/azure_speech/android/build.gradle.kts @@ -62,6 +62,8 @@ dependencies { implementation("com.squareup.okhttp3:okhttp:4.12.0") implementation("com.google.protobuf:protobuf-javalite:3.25.1") add("compileOnly", project(":ble_service")) + // 通话翻译 PCM 原生直连桥:直接给 bluetooth_manager 的 tap 槽赋值 / 直喂它的下行发送器(见 BesCallPcmBridge.kt) + add("compileOnly", project(":bluetooth_manager")) // gRPC val grpcVersion = "1.57.2" diff --git a/apps/client/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/AzureSpeechPlugin.kt b/apps/client/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/AzureSpeechPlugin.kt index e365fb9a..8d820ff5 100644 --- a/apps/client/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/AzureSpeechPlugin.kt +++ b/apps/client/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/AzureSpeechPlugin.kt @@ -97,6 +97,12 @@ class AzureSpeechPlugin : BleService.Callback, FlutterPlugin, ActivityAware, private val astStereoRecorder = AstStereoRecorder() private fun emitAstTtsFrame(leg: String, data: ByteArray, isFinal: Boolean = false) { + // 原生直连桥开着:PCM 直接喂耳机下行发送器,Dart 只拿 final 标记(不再逐块过通道) + val bridge = callPcmBridge + if (bridge != null && bridge.isEnabled) { + bridge.postDownlink(leg, data) + if (!isFinal) return + } val sink = astTtsEventSink ?: return // EventChannel 在主线程 send,确保有效;TTS 段一段一段过来,量级在 KB,主线程开销可忽略 mainHandler.post { @@ -154,6 +160,159 @@ class AzureSpeechPlugin : BleService.Callback, FlutterPlugin, ActivityAware, private var astWsConfig: Config? = null private var currentAstProvider: String = "azure" + /** 通话翻译 PCM 原生直连桥,按会话开关(见 setNativeCallBridge) */ + private var callPcmBridge: BesCallPcmBridge? = null + private fun ensureCallPcmBridge(): BesCallPcmBridge { + callPcmBridge?.let { return it } + val b = BesCallPcmBridge( + onLegA = { pushAstAudioToA(it) }, + onLegB = { pushAstAudioToB(it) }, + onGateOpened = { rms, fallback -> + sendAstEvent(mapOf("type" to "peerGateOpened", "rms" to rms, "fallback" to fallback)) + }, + onOverflow = { dropped -> sendAstEvent(mapOf("type" to "nativeBridgeOverflow", "dropped" to dropped)) }, + ) + callPcmBridge = b + return b + } + // ---------- EMAI 会话(百炼多模态)原生客户端 ---------- + // + // 耳机唤醒那条链路的 WS 与音频收发都在原生:上行 PCM 由 bluetooth_manager 经 + // aiPcmTap 直达 [aiPcmBridge],攒成 100ms 一包推给 [bailianClient];合成回来的 PCM + // 原路回到耳机下行发送器。Dart 只收文本与状态。 + // 手机麦那条链路仍走 Dart 的 BailianMultimodalService,两者互不影响。 + /** + * EMAI 下行 PCM 播放。与 iOS 的 `azure_speech/pcm_player` 同名同方法: + * Dart 侧两端共用一条代码路径,不再走 just_audio 的本地 HTTP 代理。 + */ + private lateinit var pcmPlayerChannel: MethodChannel + private val pcmSpeakerPlayer by lazy { + PcmSpeakerPlayer(onCompleted = { + mainHandler.post { pcmPlayerChannel.invokeMethod("onCompleted", null) } + }) + } + + private lateinit var bailianDialogChannel: MethodChannel + private lateinit var bailianDialogEventChannel: EventChannel + private var bailianDialogEventSink: EventChannel.EventSink? = null + private var bailianClient: BailianDialogClient? = null + private var aiPcmBridge: BesAiPcmBridge? = null + + /** + * EMAI 下行 PCM 播放的方法分发。**方法名与 iOS 的 handlePcmPlayerMethodCall 逐字一致**, + * Dart 那边不分平台。 + * + * `inputFormat` 是 iOS 独有的探路(record 插件会拿输入格式去 installTap,0Hz 会崩), + * Dart 侧 `MicPcmStreamer._waitIosInputReady` 在非 iOS 上直接放行,走不到这里。 + */ + private fun handlePcmPlayerCall(call: MethodCall, result: MethodChannel.Result) { + when (call.method) { + "start" -> { + val rate = (call.argument("sampleRate"))?.toInt() ?: 24000 + result.success(pcmSpeakerPlayer.start(rate)) + } + "feed" -> { + call.argument("data")?.let { pcmSpeakerPlayer.feed(it) } + result.success(null) + } + "end" -> { pcmSpeakerPlayer.end(); result.success(null) } + "stop" -> { pcmSpeakerPlayer.stop(); result.success(null) } + "dispose" -> { pcmSpeakerPlayer.dispose(); result.success(null) } + else -> result.notImplemented() + } + } + + /** + * EMAI 会话(百炼多模态)的方法分发。 + * + * 会话的**编排**仍在 Dart(DeviceAiSessionService):什么时候起、什么时候收、 + * 门控什么时候开关、指令怎么落地,都在那边。这里只负责协议与音频搬运。 + * ⚠️ 方法名与 iOS 的 handleBailianDialogCall 逐条对齐。 + */ + private fun handleBailianDialogCall(call: MethodCall, result: MethodChannel.Result) { + when (call.method) { + "start" -> { + val bridge = ensureAiPcmBridge() + val client = ensureBailianClient() + val cfg = BailianDialogClient.Config( + apiKey = call.argument("apiKey") ?: "", + appId = call.argument("appId") ?: "", + workspaceId = call.argument("workspaceId") ?: "", + mode = call.argument("mode") ?: "duplex", + voice = call.argument("voice"), + userId = call.argument("userId") ?: "", + enableWebSearch = call.argument("enableWebSearch") ?: true, + downstreamRate = call.argument("downstreamRate") ?: 16000, + mcpToken = call.argument("mcpToken"), + mcpAuthTools = call.argument>("mcpAuthTools") ?: emptyList(), + ) + val ok = client.start(cfg) + if (ok) bridge.enable() + result.success(ok) + } + "stop" -> { + aiPcmBridge?.disable() + bailianClient?.stop() + result.success(true) + } + "sendText" -> { + bailianClient?.sendText( + call.argument("text") ?: "", + call.argument("speakDirectly") ?: false, + ) + result.success(true) + } + "beginSpeech" -> { bailianClient?.beginSpeech(); result.success(true) } + "endSpeech" -> { bailianClient?.endSpeech(); result.success(true) } + "cancelSpeech" -> { bailianClient?.cancelSpeech(); result.success(true) } + "interrupt" -> { bailianClient?.interrupt(); result.success(true) } + "localRespondingStarted" -> { bailianClient?.notifyLocalRespondingStarted(); result.success(true) } + "localRespondingEnded" -> { bailianClient?.notifyLocalRespondingEnded(); result.success(true) } + // 播报期间把上行换成等长静音。开关时机由 Dart 的状态机决定(见 BesAiPcmBridge) + "setGated" -> { + aiPcmBridge?.setGated(call.argument("gated") ?: false) + result.success(true) + } + "stats" -> { + val m = HashMap() + bailianClient?.stats()?.let { m.putAll(it) } + aiPcmBridge?.stats()?.let { b -> b.forEach { (k, v) -> m.putIfAbsent(k, v) } } + if (m.isEmpty()) m["running"] = false + result.success(m) + } + else -> result.notImplemented() + } + } + + private fun ensureBailianClient(): BailianDialogClient { + bailianClient?.let { return it } + val c = BailianDialogClient( + onEvent = { e -> bailianDialogEventSink?.success(e) }, + // 合成音频原路回耳机:不经过 Dart,也不经过手机扬声器 + onAudio = { pcm -> aiPcmBridge?.postDownlink(pcm) }, + ) + bailianClient = c + return c + } + + private fun ensureAiPcmBridge(): BesAiPcmBridge { + aiPcmBridge?.let { return it } + val b = BesAiPcmBridge( + onPacket = { pkt -> bailianClient?.pushAudio(pkt) }, + // 只是给 Dart 的空闲计时续命,已按 500ms 节流 + onMicActivity = { + mainHandler.post { bailianDialogEventSink?.success(mapOf("event" to "micActivity")) } + }, + onOverflow = { dropped -> + mainHandler.post { + bailianDialogEventSink?.success(mapOf("event" to "overflow", "dropped" to dropped)) + } + }, + ) + aiPcmBridge = b + return b + } + // AST 初始化世代号:每次 dispose / initialize 时自增,被 dispatch 出去的 Main 协程 // 在执行前先比对,发现已经过时就直接 return,避免 1.2s delay 后落到旧 provider 上。 private val astInitGeneration = AtomicInteger(0) @@ -424,6 +583,31 @@ class AzureSpeechPlugin : BleService.Callback, FlutterPlugin, ActivityAware, } }) + // EMAI 下行 PCM 播放:只有方法通道,完成回调用 invokeMethod 回传(与 iOS 一致) + pcmPlayerChannel = + MethodChannel(flutterPluginBinding.binaryMessenger, "azure_speech/pcm_player") + pcmPlayerChannel.setMethodCallHandler { call, result -> + handlePcmPlayerCall(call, result) + } + + // EMAI 会话(百炼多模态):方法通道 + 事件通道。音频不经过这两条。 + bailianDialogEventChannel = + EventChannel(flutterPluginBinding.binaryMessenger, "azure_speech/bailian_dialog_events") + bailianDialogEventChannel.setStreamHandler(object : EventChannel.StreamHandler { + override fun onListen(arguments: Any?, events: EventChannel.EventSink?) { + bailianDialogEventSink = events + } + + override fun onCancel(arguments: Any?) { + bailianDialogEventSink = null + } + }) + bailianDialogChannel = + MethodChannel(flutterPluginBinding.binaryMessenger, "azure_speech/bailian_dialog") + bailianDialogChannel.setMethodCallHandler { call, result -> + handleBailianDialogCall(call, result) + } + // 初始化翻译事件通道 translationEventChannel = EventChannel(flutterPluginBinding.binaryMessenger, "azure_speech/translation_events") @@ -1289,6 +1473,7 @@ class AzureSpeechPlugin : BleService.Callback, FlutterPlugin, ActivityAware, } "dispose" -> { + callPcmBridge?.onSessionDisposed() try { // 让所有等待 delay(1200L) 的旧 init 协程在恢复时直接 return,避免落到已 dispose 的 helper 上 nextAstInitGeneration() @@ -1399,6 +1584,19 @@ class AzureSpeechPlugin : BleService.Callback, FlutterPlugin, ActivityAware, * leg = 'B' / 'downlink' = 对端通话音 * pcm 16k mono PCM16 little-endian */ + // 通话翻译 PCM 原生直连桥(见 BesCallPcmBridge.kt;对端是 bluetooth_manager 的 callPcmTap)。 + // 开着时耳机双路 PCM 不经 Dart 直达 AST,译文 PCM 直喂耳机下行发送器; + // Dart 只收 TTS 的 final 标记与门限/丢帧的诊断事件。 + "setNativeCallBridge" -> { + val enabled = call.argument("enabled") ?: false + if (enabled) ensureCallPcmBridge().enable() else callPcmBridge?.disable() + result.success(true) + } + + "getNativeCallBridgeStats" -> { + result.success(callPcmBridge?.stats() ?: mapOf("enabled" to false)) + } + "pushExternalAudio" -> { val leg = call.argument("leg")?.uppercase() ?: "" val pcm = call.argument("pcm") @@ -1429,6 +1627,8 @@ class AzureSpeechPlugin : BleService.Callback, FlutterPlugin, ActivityAware, val ttsLang1 = supportedLanguages.getOrElse(5) { "en-US-AriaNeural" } Log.d(tag, "initializeIntegrated: supportedLanguages=$supportedLanguages") currentAstProvider = provider + // 原生直连桥:新会话重新刻音色 → 门限复位;helper 起来之前的帧先攒着(见 BesCallPcmBridge) + callPcmBridge?.onSessionInit() val subscriptionKey = call.argument("subscriptionKey") ?: "" val region = call.argument("region") ?: "" @@ -1531,6 +1731,7 @@ class AzureSpeechPlugin : BleService.Callback, FlutterPlugin, ActivityAware, callback = callbackB ) ?: false + callPcmBridge?.markReady() result.success(initA && initB) } } else if (currentAstProvider == "azure" ) { @@ -1591,6 +1792,7 @@ class AzureSpeechPlugin : BleService.Callback, FlutterPlugin, ActivityAware, serviceConfig = serviceConfigB, callback = callbackB ) + callPcmBridge?.markReady() result.success(true) } } else if (currentAstProvider == "volcano" ) { @@ -1636,6 +1838,7 @@ class AzureSpeechPlugin : BleService.Callback, FlutterPlugin, ActivityAware, result.success(false); return@launch } doubaoAstHelperB?.initialize(cfgB, callbackB) + callPcmBridge?.markReady() result.success(true) } } else if (currentAstProvider == "alibaba" ) { @@ -1684,6 +1887,7 @@ class AzureSpeechPlugin : BleService.Callback, FlutterPlugin, ActivityAware, result.success(false); return@launch } val initB = bailianAstHelperB?.initialize(bailianConfigB, callbackB) ?: false + callPcmBridge?.markReady() result.success(initA && initB) } } diff --git a/apps/client/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/BailianDialogClient.kt b/apps/client/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/BailianDialogClient.kt new file mode 100644 index 00000000..949036f3 --- /dev/null +++ b/apps/client/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/BailianDialogClient.kt @@ -0,0 +1,442 @@ +package com.yunqiinnovation.azure_speech + +import android.os.Handler +import android.os.Looper +import android.util.Log +import okhttp3.OkHttpClient +import okhttp3.Request +import okhttp3.Response +import okhttp3.WebSocket +import okhttp3.WebSocketListener +import okio.ByteString +import org.json.JSONArray +import org.json.JSONObject +import java.util.UUID +import java.util.concurrent.Executors +import java.util.concurrent.TimeUnit + +/** + * 阿里云百炼「多模态交互协议」的**原生**客户端(EMAI 助手)。 + * + * 协议文档:https://help.aliyun.com/zh/model-studio/multimodal-interaction-protocol + * 端点:`wss://dashscope.aliyuncs.com/api-ws/v1/inference` + * + * ## 为什么有原生这一份 + * Dart 侧的 `BailianMultimodalService` 仍在(手机麦那条链路、以及本类起不来时的退路)。 + * 耳机按键唤醒那条链路原先是 + * `bluetooth_manager → EventChannel → Dart → WS(Dart) → Dart → MethodChannel → bluetooth_manager`, + * 上行 20ms 一帧、下行每个音频块一次,全压在 Dart 的单 isolate 上。本类把 WS 连同音频 + * 收发搬到原生:PCM 经 [BesAiPcmBridge] 在原生之间直连,**Dart 只收文本与状态事件**。 + * + * ## ⚠️ 这不修「锁屏断连」 + * 进程被系统挂起时原生和 Dart 一起停。锁屏能不能活着取决于前台服务 / 后台音频, + * 与代码在哪一层无关。本类解决的是 event loop 抖动与跨界开销。 + * + * ## ⚠️ 与 Dart 版、iOS 版**逐条对齐** + * 握手参数、事件名映射、心跳间隔照 `lib/data/services/bailian_multimodal_service.dart` + * 与 `BailianDialogClient.swift` 写。改一处必须同改另两处,否则「用耳机唤醒」和 + * 「在 App 里按住说话」会变成两种行为,而且都不报错。 + * + * ## 指令解析刻意**留在 Dart** + * `extra_info.tool_calls` 的解析有一批真实 payload 的金测试守着 + * (`test/assistant_directive_test.dart`),解析结果又要落进 GetStorage 和「代办」页。 + * 在原生解析再序列化回 Dart 毫无收益,只会把那批测试作废。这里把 `extra_info` + * 原样转成字符串带上去,Dart 照旧解析。 + */ +class BailianDialogClient( + private val onEvent: (Map) -> Unit, + private val onAudio: (ByteArray) -> Unit, +) { + companion object { + private const val TAG = "BailianDialog" + private const val ENDPOINT = "wss://dashscope.aliyuncs.com/api-ws/v1/inference" + + /** 上行音频:16k / 16bit / 单声道 / 小端 */ + private const val UPSTREAM_SAMPLE_RATE = 16000 + + /** 服务端要求任意连续 60s 内必须有消息往来;文档建议 50s 一次心跳 */ + private const val HEARTBEAT_INTERVAL_MS = 50_000L + } + + data class Config( + val apiKey: String, + val appId: String, + val workspaceId: String, + val mode: String, + val voice: String?, + val userId: String, + val enableWebSearch: Boolean, + val downstreamRate: Int, + val mcpToken: String?, + /** + * 需要用户身份的 MCP 工具名。**由 Dart 传下来**,不在原生再抄一份: + * 服务端加工具时只改 Dart 那一处,漏改的表现是那个工具永远「身份校验失败」。 + */ + val mcpAuthTools: List, + ) + + private val exec = Executors.newSingleThreadExecutor { r -> Thread(r, "azure_speech.bailian") } + private val main = Handler(Looper.getMainLooper()) + + private var http: OkHttpClient? = null + private var ws: WebSocket? = null + private var config: Config? = null + private var taskId = "" + private var dialogId = "" + + @Volatile private var running = false + private var heartbeat: Runnable? = null + + private var uplinkBytes = 0L + private var downlinkBytes = 0L + + /** + * 下行字节数的上报节流。Dart 侧不再拿到 PCM,但它要靠「首包到达」开门控、 + * 靠累计字节数估算播完时刻(门控的收尾计时),所以这两件事必须报上去。 + * 按 100ms 汇总一次,别每块都报——那就把省下来的跨界又加回去了。 + */ + private var lastDownlinkReportAt = 0L + + val isRunning: Boolean get() = running + + // ---------- 生命周期 ---------- + + /** 建立会话。返回 false 表示配置不全。连接是异步的,失败经 onEvent 上报。 */ + fun start(c: Config): Boolean { + if (c.apiKey.isBlank()) { + Log.e(TAG, "未配置 API Key,不尝试连接") + // Key 缺失时不要尝试连接——服务端会先回 task-started 再立刻 task-failed, + // 日志上看起来像"连上了又掉",容易误判成网络问题 + emit(mapOf("event" to "error", "text" to "未配置 ALIBABA_OPENSPEECH_APP_KEY", + "code" to "NOT_CONFIGURED", "fatal" to true)) + return false + } + stop() + config = c + taskId = UUID.randomUUID().toString() + dialogId = "" + uplinkBytes = 0L + downlinkBytes = 0L + lastDownlinkReportAt = 0L + + val client = OkHttpClient.Builder() + .pingInterval(20, TimeUnit.SECONDS) // 与 Dart 的 pingInterval 对齐 + .readTimeout(0, TimeUnit.MILLISECONDS) + .build() + http = client + val req = Request.Builder() + .url(ENDPOINT) + .addHeader("Authorization", "Bearer ${c.apiKey}") + .addHeader("X-DashScope-DataInspection", "enable") + .build() + running = true + ws = client.newWebSocket(req, Listener()) + sendJson(startPayload(c)) + startHeartbeat() + Log.i(TAG, "会话启动 app_id=${c.appId} workspace=${c.workspaceId} mode=${c.mode} downstream=${c.downstreamRate}Hz") + return true + } + + /** 结束会话:先发 Stop 让服务端收尾,再 finish-task 结束整个 WS 任务 */ + fun stop() { + if (!running) { + cleanup() + return + } + try { + sendDirective("Stop") + sendJson( + JSONObject() + .put("header", JSONObject() + .put("action", "finish-task").put("task_id", taskId).put("streaming", "duplex")) + .put("payload", JSONObject().put("input", JSONObject())) + ) + } catch (_: Exception) { + } + Log.i(TAG, "会话结束 上行=$uplinkBytes 字节 下行=$downlinkBytes 字节") + try { + ws?.close(1000, null) + } catch (_: Exception) { + } + cleanup() + } + + // ---------- 上行 ---------- + + /** 推一包上行音频(PCM 16k/16bit/mono)。由 [BesAiPcmBridge] 的串行执行器调用。 */ + fun pushAudio(pcm: ByteArray) { + if (pcm.isEmpty() || !running) return + exec.execute { + val w = ws ?: return@execute + uplinkBytes += pcm.size + if (!w.send(ByteString.of(*pcm))) { + Log.w(TAG, "推送音频失败(发送队列已满)") + } + } + } + + fun beginSpeech() = sendDirective("SendSpeech") + fun endSpeech() = sendDirective("StopSpeech") + fun cancelSpeech() = sendDirective("CancelSpeech") + fun interrupt() = sendDirective("RequestToSpeak") + fun notifyLocalRespondingStarted() = sendDirective("LocalRespondingStarted") + fun notifyLocalRespondingEnded() = sendDirective("LocalRespondingEnded") + + /** 文本对话:交给应用的 LLM 处理(prompt),或让它直接朗读(transcript) */ + fun sendText(text: String, speakDirectly: Boolean) { + val t = text.trim() + if (t.isEmpty()) return + sendDirective("RequestToRespond", JSONObject() + .put("type", if (speakDirectly) "transcript" else "prompt") + .put("text", t)) + } + + fun stats(): Map = mapOf( + "running" to running, + "uplinkBytes" to uplinkBytes, + "downlinkBytes" to downlinkBytes, + "dialogId" to dialogId, + ) + + // ---------- 协议帧 ---------- + + private fun startPayload(c: Config): JSONObject { + val downstream = JSONObject() + .put("audio_format", "pcm") + .put("sample_rate", c.downstreamRate) + // transcript:中间态只回朗读文本,跟 TTS 出声一致, + // 免得 UI 上先闪一段 dialog 思考态文本再被覆盖 + .put("intermediate_text", "transcript") + // 显式钉死"每帧回全文":Dart 收到 RespondingContent 是整体替换气泡内容的 + .put("incremental_response", false) + // 不传 voice:音色属于百炼应用的配置,硬编码只会跟应用实际用的 TTS 模型对不上 + if (!c.voice.isNullOrBlank()) downstream.put("voice", c.voice) + + val parameters = JSONObject() + .put("upstream", JSONObject() + .put("type", "AudioOnly") + .put("mode", c.mode) + .put("audio_format", "pcm") + .put("sample_rate", UPSTREAM_SAMPLE_RATE)) + .put("downstream", downstream) + .put("client_info", JSONObject().put("user_id", safeUserId(c.userId))) + + // biz_params 是可选块,两件事都没有时整块不发——万一应用侧没开这些能力, + // 多带一个字段可能让 Start 直接被拒,那样整个功能都起不来 + val hasToken = !c.mcpToken.isNullOrBlank() + if (c.enableWebSearch || hasToken) { + val userDefined = JSONObject() + if (hasToken) { + // ⚠️ **必须按工具名嵌套**,平铺写 auth_token 会被静默丢弃 + for (tool in c.mcpAuthTools) { + userDefined.put(tool, JSONObject().put("auth_token", c.mcpToken)) + } + } + if (c.enableWebSearch) { + userDefined.put("extra_config", JSONObject().put("enable_web_search", true)) + } + parameters.put("biz_params", JSONObject().put("user_defined_params", userDefined)) + } + + return JSONObject() + .put("header", JSONObject() + .put("action", "run-task").put("task_id", taskId).put("streaming", "duplex")) + .put("payload", JSONObject() + .put("task_group", "aigc") + .put("task", "multimodal-generation") + .put("function", "generation") + .put("model", "multimodal-dialog") + .put("input", JSONObject() + .put("directive", "Start") + .put("workspace_id", c.workspaceId) + .put("app_id", c.appId)) + .put("parameters", parameters)) + } + + private fun sendDirective(directive: String, extra: JSONObject? = null) { + val c = config ?: return + if (!running) return + val input = JSONObject() + .put("directive", directive) + .put("workspace_id", c.workspaceId) + .put("app_id", c.appId) + if (dialogId.isNotEmpty()) input.put("dialog_id", dialogId) + extra?.keys()?.forEach { k -> input.put(k, extra.get(k)) } + sendJson(JSONObject() + .put("header", JSONObject() + .put("action", "continue-task").put("task_id", taskId).put("streaming", "duplex")) + .put("payload", JSONObject() + .put("task_group", "aigc") + .put("task", "multimodal-generation") + .put("function", "generation") + .put("model", "multimodal-dialog") + .put("input", input))) + } + + private fun sendJson(msg: JSONObject) { + exec.execute { + val w = ws ?: return@execute + try { + w.send(msg.toString()) + } catch (e: Exception) { + Log.w(TAG, "发送失败: ${e.message}") + } + } + } + + // ---------- 下行 ---------- + + private inner class Listener : WebSocketListener() { + override fun onMessage(webSocket: WebSocket, bytes: ByteString) { + // 二进制帧就是下行合成音频 + val b = bytes.toByteArray() + downlinkBytes += b.size + val now = System.currentTimeMillis() + // 首块立刻报(Dart 要用它开门控),之后按间隔汇总 + if (lastDownlinkReportAt == 0L || now - lastDownlinkReportAt >= 100L) { + lastDownlinkReportAt = now + emit(mapOf("event" to "downlink", "bytes" to downlinkBytes)) + } + if (b.isNotEmpty()) onAudio(b) + } + + override fun onMessage(webSocket: WebSocket, text: String) { + handleText(text) + } + + override fun onFailure(webSocket: WebSocket, t: Throwable, response: Response?) { + if (!running) return + Log.e(TAG, "WebSocket 错误: ${t.message}") + emit(mapOf("event" to "error", "text" to (t.message ?: "WebSocket error"), "fatal" to true)) + cleanup() + } + + override fun onClosed(webSocket: WebSocket, code: Int, reason: String) { + Log.i(TAG, "WebSocket 关闭 code=$code") + emit(mapOf("event" to "closed")) + cleanup() + } + } + + private fun handleText(raw: String) { + val obj = try { + JSONObject(raw) + } catch (e: Exception) { + Log.w(TAG, "无法解析的文本帧: ${e.message}") + return + } + val header = obj.optJSONObject("header") ?: JSONObject() + val payload = obj.optJSONObject("payload") ?: JSONObject() + when (val event = header.optString("event", "")) { + "task-started" -> Log.i(TAG, "task-started") + "task-failed" -> { + val code = header.optString("error_code", "") + val msg = header.optString("error_message", "") + Log.e(TAG, "task-failed: $code $msg") + // header 级 = 整个 task 结束,必须重连 + emit(mapOf("event" to "error", "text" to msg, "code" to code, "fatal" to true)) + } + "task-finished" -> { + Log.i(TAG, "task-finished") + emit(mapOf("event" to "closed")) + } + else -> handleDirectiveEvent(event, payload) + } + } + + /** 业务事件都在 payload.output.event 里 */ + private fun handleDirectiveEvent(outerEvent: String, payload: JSONObject) { + val output = payload.optJSONObject("output") ?: JSONObject() + val name = output.optString("event", outerEvent) + val text = output.optString("text", "") + val finished = output.optBoolean("finished", false) + + when (name) { + "Started" -> { + dialogId = output.optString("dialog_id", "") + Log.i(TAG, "Started dialog_id=$dialogId") + emit(mapOf("event" to "started")) + } + "DialogStateChanged" -> { + // 协议里是首字母大写(Listening/Thinking/Responding),统一小写再交给 Dart + emit(mapOf("event" to "stateChanged", + "state" to output.optString("state", "").lowercase())) + } + "SpeechStarted" -> emit(mapOf("event" to "speechStarted")) + "SpeechContent" -> emit(mapOf("event" to "speechContent", "text" to text, "isFinal" to finished)) + "SpeechEnded" -> emit(mapOf("event" to "speechEnded", "text" to text, "isFinal" to true)) + "RequestAccepted" -> emit(mapOf("event" to "requestAccepted")) + "RespondingStarted" -> emit(mapOf("event" to "respondingStarted")) + "RespondingContent" -> { + val reason = output.optString("finish_reason", "") + // 指令解析留在 Dart(见类注释):把 extra_info 原样带上去。 + // 它只在整轮收尾那一帧出现,不是每个流式分片都有,频率可控。 + val e = mutableMapOf( + "event" to "respondingContent", + "text" to text, + "isFinal" to finished, + "finishReason" to reason, + ) + val extra = output.opt("extra_info") + if (extra is JSONObject || extra is JSONArray) e["extraInfo"] = extra.toString() + emit(e) + } + "RespondingEnded" -> emit(mapOf("event" to "respondingEnded", "text" to text, "isFinal" to true)) + "Error" -> { + val code = output.optString("error_code", "") + val msg = output.optString("error_message", "") + val name0 = output.optString("error_name", "") + Log.e(TAG, "Error: $code/$name0 $msg") + // payload 级 Error 是"这一轮不行",连接通常还在,不标 fatal + val shown = if (msg.isNotEmpty()) msg else if (text.isNotEmpty()) text else name0 + emit(mapOf("event" to "error", "text" to shown, "code" to code, "fatal" to false)) + } + "Stopped" -> emit(mapOf("event" to "closed")) + "HeartBeat" -> Unit + else -> Log.d(TAG, "未处理事件: $name") + } + } + + // ---------- 其它 ---------- + + private fun startHeartbeat() { + stopHeartbeat() + val r = object : Runnable { + override fun run() { + if (!running) return + sendDirective("HeartBeat") + main.postDelayed(this, HEARTBEAT_INTERVAL_MS) + } + } + heartbeat = r + main.postDelayed(r, HEARTBEAT_INTERVAL_MS) + } + + private fun stopHeartbeat() { + heartbeat?.let { main.removeCallbacks(it) } + heartbeat = null + } + + private fun cleanup() { + stopHeartbeat() + running = false + ws = null + dialogId = "" + try { + http?.dispatcher?.executorService?.shutdown() + } catch (_: Exception) { + } + http = null + } + + private fun emit(e: Map) { + main.post { onEvent(e) } + } + + /** 协议限制 user_id 不超过 36 字符 */ + private fun safeUserId(raw: String): String { + val v = raw.trim() + if (v.isEmpty()) return "eaimar-guest" + return if (v.length <= 36) v else v.substring(0, 36) + } +} diff --git a/apps/client/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/BesAiPcmBridge.kt b/apps/client/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/BesAiPcmBridge.kt new file mode 100644 index 00000000..7f38bd57 --- /dev/null +++ b/apps/client/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/BesAiPcmBridge.kt @@ -0,0 +1,182 @@ +package com.yunqiinnovation.azure_speech + +import android.util.Log +import com.example.bluetooth_manager.AiAudioDownlink +import com.example.bluetooth_manager.BluetoothManager +import java.util.concurrent.Executors +import java.util.concurrent.atomic.AtomicInteger +import kotlin.math.sqrt + +/** + * EMAI 会话 PCM 的**原生直连桥**(azure_speech 这一端)。 + * + * 对端是 bluetooth_manager 的 `BluetoothManager.aiPcmTap`:耳机解码出来的 AI 上行 PCM 直接 + * 到这里(不再经过 Dart),攒成 100ms 一包推给 [BailianDialogClient];百炼合成回来的 PCM 由 + * [postDownlink] 直接喂给 [AiAudioDownlink]。 + * + * 本模块 `compileOnly project(":bluetooth_manager")`,运行时两者都在 APK 里。 + * **与 iOS 端 BesAiPcmBridge.swift 语义逐条对齐**,改一端要同改另一端。 + * + * ## 门控为什么是从 Dart 推下来的布尔量 + * 「AI 在耳机里播报时把上行换成等长静音」这件事本身必须在这里做(逐帧);但**什么时候 + * 开关门**留在 Dart 的 `DeviceAiSessionService`:那套逻辑有兜底计时、非致命错误要关门、 + * 纯指令帧收尾也要关门等一串来之不易的分支(门卡开着的后果是「会话活着但是聋的」)。 + * 门一轮只切换几次,一次 MethodChannel 调用的开销可以忽略。 + * + * ## 线程与背压 + * tap 在耳机解码线程上同步到达,这里**立刻转到自己的单线程执行器**,不让 WS 发送拖住解码线程。 + * 排队超过 [MAX_PENDING_PACKETS](约 2s)就丢新包并计数。 + */ +class BesAiPcmBridge( + private val onPacket: (ByteArray) -> Unit, + private val onMicActivity: () -> Unit, + private val onOverflow: (dropped: Long) -> Unit, +) { + companion object { + private const val TAG = "BesAiPcmBridge" + + /** + * 上行攒包大小:16k/16bit/mono 的 100ms = 3200 字节。 + * ⚠️ 与 Dart `DeviceAiSessionService._micBatchBytes`、iOS 的 batchBytes 相同,三处同改。 + */ + private const val BATCH_BYTES = 3200 + + /** + * 说话人能量门限与上报节流。 + * ⚠️ 与 Dart 的 `_micActivityRms` / `_micActivityThrottle`、iOS 版相同,三处同改。 + * 只用来给 Dart 续空闲计时,不参与切句。 + */ + private const val ACTIVITY_RMS = 600.0 + private const val ACTIVITY_THROTTLE_MS = 500L + + /** 约 2 秒的积压上限(100ms 一包) */ + private const val MAX_PENDING_PACKETS = 20 + } + + private val ingress = Executors.newSingleThreadExecutor { r -> Thread(r, "azure_speech.aipcm.ingress") } + private val pending = AtomicInteger(0) + + @Volatile private var enabled = false + @Volatile private var gated = false + + private var accum = ByteArray(0) + private var lastActivityAt = 0L + + private var framesIn = 0L + private var packetsOut = 0L + private var dropped = 0L + private var downlinkFrames = 0L + + val isEnabled: Boolean get() = enabled + + fun enable() { + if (enabled) return + enabled = true + ingress.execute { + accum = ByteArray(0) + gated = false + lastActivityAt = 0L + framesIn = 0L + packetsOut = 0L + dropped = 0L + downlinkFrames = 0L + } + pending.set(0) + BluetoothManager.aiPcmTap = { pcm -> ingest(pcm) } + BluetoothManager.setAiPcmBridge(true) + Log.d(TAG, "AI 上行直连已开启") + } + + fun disable() { + if (!enabled) return + enabled = false + BluetoothManager.setAiPcmBridge(false) + BluetoothManager.aiPcmTap = null + ingress.execute { + accum = ByteArray(0) + Log.d(TAG, "AI 上行直连已关闭 上行=$framesIn 帧 出包=$packetsOut 丢包=$dropped 下行=$downlinkFrames 帧") + } + } + + /** + * AI 是否正在耳机里播报。播报期间上行换成**等长静音**而不是断流: + * 上行保持连续,服务端 VAD 收到的是"真的没人说话",而不是一段数据缺口。 + */ + fun setGated(v: Boolean) { + ingress.execute { + gated = v + // 关门时把攒了一半的静音丢掉,下一包从真实音频开始, + // 免得用户开口的头 100ms 里混着上一轮的静音 + if (!v) accum = ByteArray(0) + } + } + + // ---------- 上行 ---------- + + private fun ingest(pcm: ByteArray) { + if (pcm.isEmpty() || !enabled) return + // 背压:排队太多就丢新包。判断放在入队前,不然队列本身就是那个积压 + if (pending.get() >= MAX_PENDING_PACKETS) { + ingress.execute { + dropped++ + if (dropped == 1L || dropped % 50L == 0L) Log.w(TAG, "上行拥塞,已丢 $dropped 包") + onOverflow(dropped) + } + return + } + pending.incrementAndGet() + ingress.execute { + pending.decrementAndGet() + framesIn++ + accum = if (gated) { + accum + ByteArray(pcm.size) // 等长静音 + } else { + if (rms(pcm) >= ACTIVITY_RMS) { + val now = System.currentTimeMillis() + if (now - lastActivityAt >= ACTIVITY_THROTTLE_MS) { + lastActivityAt = now + onMicActivity() + } + } + accum + pcm + } + while (accum.size >= BATCH_BYTES) { + val pkt = accum.copyOfRange(0, BATCH_BYTES) + accum = accum.copyOfRange(BATCH_BYTES, accum.size) + packetsOut++ + onPacket(pkt) + } + } + } + + // ---------- 下行 ---------- + + /** 把一块合成音频交回耳机下行发送器(G.722 编码与节流在 bluetooth_manager 那边做) */ + fun postDownlink(pcm: ByteArray) { + if (pcm.isEmpty()) return + AiAudioDownlink.pushPcm(pcm) + ingress.execute { downlinkFrames++ } + } + + fun stats(): Map = mapOf( + "enabled" to enabled, + "framesIn" to framesIn, + "packetsOut" to packetsOut, + "dropped" to dropped, + "downlinkFrames" to downlinkFrames, + "gated" to gated, + ) + + /** 16bit 小端 PCM 的 RMS(0~32767 量纲,与 Dart 的 `rmsOfPcm16` 同口径) */ + private fun rms(pcm: ByteArray): Double { + val count = pcm.size / 2 + if (count == 0) return 0.0 + var sum = 0.0 + for (i in 0 until count) { + // 高字节 toInt() 自带符号扩展,or 上无符号低字节即为 -32768..32767 + val v = (pcm[i * 2 + 1].toInt() shl 8) or (pcm[i * 2].toInt() and 0xFF) + sum += v.toDouble() * v.toDouble() + } + return sqrt(sum / count) + } +} diff --git a/apps/client/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/BesCallPcmBridge.kt b/apps/client/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/BesCallPcmBridge.kt new file mode 100644 index 00000000..40265bea --- /dev/null +++ b/apps/client/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/BesCallPcmBridge.kt @@ -0,0 +1,176 @@ +package com.yunqiinnovation.azure_speech + +import android.util.Log +import com.example.bluetooth_manager.BluetoothManager +import com.example.bluetooth_manager.CallTranslationDownlink +import java.util.concurrent.Executors +import java.util.concurrent.atomic.AtomicInteger + +/** + * 通话翻译 PCM 的**原生直连桥**(azure_speech 这一端)。 + * + * 对端是 bluetooth_manager 的 `BluetoothManager.callPcmTap`:耳机解码出来的双路 PCM 直接到这里 + * (不再经过 Dart),这里过 B 路说话人门限后推给 AST helper;AST 合成出来的译文 PCM 由 + * [postDownlink] 直接喂给 [CallTranslationDownlink]。 + * + * 本模块 `compileOnly project(":bluetooth_manager")`(与 agent_service→azure_speech 同一种先例), + * 运行时两者都在 APK 里。**与 iOS 端 BesCallPcmBridge.swift 语义逐条对齐**,改一端要同改另一端。 + * + * ## 线程与背压 + * tap 在耳机解码线程上同步到达,这里**立刻转到自己的单线程执行器** [ingress],不让 WS 发送或 + * base64 编码拖住解码线程。排队超过 [MAX_PENDING_FRAMES](2s)就丢新帧并计数——这就是原来 + * Dart 链路上「AudioSendSlow 1011」的替代品:同样的拥塞,现在有数字、有日志,丢的是帧而不是整个会话。 + * + * ## 重连期间不丢音频 + * AST helper 在 initialize 期间会被重建,它自己的缓冲随之清空。「未就绪」期间的帧留在**这里** + * 的环形缓冲(每路 [HOLD_FRAMES],3s),[markReady] 时整段补推——弱网重连的那 1~4 秒不再是纯丢音频。 + */ +class BesCallPcmBridge( + private val onLegA: (ByteArray) -> Unit, + private val onLegB: (ByteArray) -> Unit, + private val onGateOpened: (rms: Int, fallback: Boolean) -> Unit, + private val onOverflow: (dropped: Long) -> Unit, +) { + companion object { + private const val TAG = "BesCallPcmBridge" + private const val MAX_PENDING_FRAMES = 100 + private const val HOLD_FRAMES = 150 + } + + private val ingress = Executors.newSingleThreadExecutor { r -> Thread(r, "azure_speech.callpcm.ingress") } + private val pending = AtomicInteger(0) + + @Volatile var isEnabled: Boolean = false + private set + + // 以下只在 ingress 线程上读写 + private var ready = false + private val gate = PeerSpeechGate() + private val holdA = ArrayDeque() + private val holdB = ArrayDeque() + private var framesA = 0L + private var framesB = 0L + @Volatile private var dropped = 0L + private var gatedB = 0L + @Volatile private var downlinkFrames = 0L + @Volatile private var maxPendingSeen = 0 + + fun enable() { + if (isEnabled) return + isEnabled = true + ingress.execute { + framesA = 0; framesB = 0; dropped = 0; gatedB = 0; downlinkFrames = 0; maxPendingSeen = 0 + } + BluetoothManager.callPcmTap = { leg, pcm -> enqueue(leg, pcm) } + BluetoothManager.setCallPcmBridge(true) + Log.d(TAG, "已开启:上行经 tap 直达,下行直喂发送器") + } + + fun disable() { + if (!isEnabled) return + isEnabled = false + BluetoothManager.setCallPcmBridge(false) + BluetoothManager.callPcmTap = null + ingress.execute { + Log.d(TAG, "已关闭 A=$framesA B=$framesB 丢帧=$dropped B门限前=$gatedB 下行=$downlinkFrames 峰值积压=$maxPendingSeen") + holdA.clear(); holdB.clear() + ready = false + } + } + + /** + * AST 会话(重)建立开始:新会话要重新刻音色,门限重新关上;帧先攒着等 [markReady]。 + * + * ⚠️ 只清 B 路的缓存:那些帧是在**旧**门限状态下放行的,门限复位后必须由现场音频 + * 重新判定「对端开口了没有」,否则重连后照样可能刻到串音。A 路是机主自己的麦, + * 没有门限,攒下的整段都要补推——这正是「重连那几秒不丢音频」的价值所在。 + */ + fun onSessionInit() { + ingress.execute { + ready = false + gate.reset() + holdB.clear() + } + } + + /** AST 会话销毁:帧继续攒着(重连场景),直到下一次 markReady。 */ + fun onSessionDisposed() { + ingress.execute { ready = false } + } + + /** A/B 两路 helper 都起来了:把攒下的整段补推出去,之后直通。 */ + fun markReady() { + ingress.execute { + val a = holdA.toList(); val b = holdB.toList() + holdA.clear(); holdB.clear() + ready = true + if (a.isNotEmpty() || b.isNotEmpty()) Log.d(TAG, "就绪,补推积攒的 A=${a.size} B=${b.size} 帧") + a.forEach(onLegA) + b.forEach(onLegB) + } + } + + /** 由 AST 的 TTS 输出调用:译文 PCM 直接喂给耳机下行发送器。 */ + fun postDownlink(leg: String, pcm: ByteArray) { + if (pcm.isEmpty()) return + CallTranslationDownlink.pushPcm(leg, pcm) + downlinkFrames++ + } + + fun stats(): Map = mapOf( + "enabled" to isEnabled, "ready" to ready, + "framesA" to framesA, "framesB" to framesB, + "dropped" to dropped, "gatedB" to gatedB, + "downlinkFrames" to downlinkFrames, "maxPending" to maxPendingSeen, + "gateOpen" to gate.isOpen, "gateFallback" to gate.openedByFallback, + ) + + private fun enqueue(leg: String, pcm: ByteArray) { + if (pcm.isEmpty()) return + val p = pending.incrementAndGet() + if (p > MAX_PENDING_FRAMES) { + pending.decrementAndGet() + val d = ++dropped + if (d == 1L || d % 100L == 0L) { + Log.w(TAG, "上行积压超过 $MAX_PENDING_FRAMES 帧,丢帧累计 $d") + onOverflow(d) + } + return + } + if (p > maxPendingSeen) maxPendingSeen = p + ingress.execute { + pending.decrementAndGet() + handle(leg, pcm) + } + } + + /** 只在 ingress 线程上调用 */ + private fun handle(leg: String, pcm: ByteArray) { + if (leg == "A") { + framesA++ + deliverA(pcm) + return + } + framesB++ + val wasOpen = gate.isOpen + val frames = gate.accept(pcm) + if (frames.isEmpty()) gatedB++ + if (!wasOpen && gate.isOpen) { + Log.w(TAG, "B 路门限开启 rms=${gate.openRms} fallback=${gate.openedByFallback}") + onGateOpened(gate.openRms, gate.openedByFallback) + } + frames.forEach { deliverB(it) } + } + + private fun deliverA(pcm: ByteArray) { + if (ready) { onLegA(pcm); return } + holdA.addLast(pcm) + if (holdA.size > HOLD_FRAMES) holdA.removeFirst() + } + + private fun deliverB(pcm: ByteArray) { + if (ready) { onLegB(pcm); return } + holdB.addLast(pcm) + if (holdB.size > HOLD_FRAMES) holdB.removeFirst() + } +} diff --git a/apps/client/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/PcmSpeakerPlayer.kt b/apps/client/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/PcmSpeakerPlayer.kt new file mode 100644 index 00000000..b7ef052d --- /dev/null +++ b/apps/client/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/PcmSpeakerPlayer.kt @@ -0,0 +1,320 @@ +package com.yunqiinnovation.azure_speech + +import android.media.AudioAttributes +import android.media.AudioFormat +import android.media.AudioManager +import android.media.AudioTrack +import android.os.Build +import android.util.Log +import java.util.concurrent.LinkedBlockingQueue +import java.util.concurrent.TimeUnit + +/** + * 流式 PCM 外放播放器(16bit 小端、单声道),Android 侧。 + * + * ## 为什么不用 just_audio + * Dart 的 [PcmStreamPlayer] 在 Android 上原来是给 just_audio 喂一个「长度未知的 + * StreamAudioSource」,而 just_audio 内部**会起一个本地代理 HTTP 服务**,把 PCM + * 包成 chunked WAV 再交给 ExoPlayer。也就是说一条实时语音流要先走一遍 localhost + * TCP + HTTP 分块编码,再进解码器。能用,但这条路上白白多了一次内存拷贝、一次 + * HTTP 分帧和 ExoPlayer 自己的起播缓冲,首字延迟比直接喂 AudioTrack 高一截。 + * + * iOS 那边早就绕开了(`PcmSpeakerPlayer.swift`,那边是被迫的:AVPlayer 压根放不了 + * 长度未知的直播流)。这个类把 Android 也对齐过来,**方法名与 iOS 逐字一致** + * (start / feed / end / stop / dispose + 回调 onCompleted),Dart 侧两端共用一条 + * 代码路径。 + * + * ## ⚠️ 不碰 AudioManager 的焦点与模式 + * 音频会话由 Dart 侧的 audio_session 统一配置(`EmaiAudioSession`)。这里只按它 + * 声明的属性建 AudioTrack(`USAGE_MEDIA` + `CONTENT_TYPE_SPEECH`,与 + * `emai_audio_session.dart` 里那份 `AndroidAudioAttributes` 对齐)。 + * 自己再去 requestAudioFocus / setMode 会和录音链路互相掀桌子。 + * + * ## 播完的判定 + * AudioTrack 没有 iOS 那种逐 buffer 的完成回调,所以用「写进去的帧数」对比 + * `playbackHeadPosition`(已经真正推给硬件的帧数)。**不能在队列排空时就报完成**: + * 那时最后一两百毫秒还在 AudioTrack 的缓冲里没出声,调用方会据此给服务端回 + * `LocalRespondingEnded`,尾音被切掉。 + */ +class PcmSpeakerPlayer(private val onCompleted: () -> Unit) { + + companion object { + private const val TAG = "PcmSpeakerPlayer" + + /** 队列轮询间隔:既是写线程的空转节奏,也是播完判定的精度 */ + private const val POLL_MS = 20L + + /** + * 播完判定的兜底上限。正常情况下缓冲里最多就几百毫秒, + * 走到这个上限说明 AudioTrack 卡住了(拔耳机、路由切换), + * 这时也必须报完成,否则服务端会一直停在 Responding。 + */ + private const val DRAIN_TIMEOUT_MS = 5_000L + } + + private val lock = Any() + private val queue = LinkedBlockingQueue() + + private var track: AudioTrack? = null + private var trackRate = 0 + private var writer: Thread? = null + + @Volatile private var running = false + @Volatile private var ended = false + + /** 本轮代际。stop() 之后迟到的写线程按代际自行退出 */ + @Volatile private var generation = 0 + + /** 已写进 AudioTrack 的帧数,只在写线程上累加 */ + private var framesWritten = 0 + + // ---------- 对外接口(与 iOS 逐字对齐)---------- + + fun start(sampleRate: Int): Boolean { + synchronized(lock) { + generation++ + val gen = generation + stopWriterLocked() + queue.clear() + ended = false + framesWritten = 0 + + if (!ensureTrackLocked(sampleRate)) return false + val t = track ?: return false + try { + t.play() + } catch (e: Exception) { + Log.e(TAG, "AudioTrack.play 失败: ${e.message}") + return false + } + running = true + val th = Thread({ writeLoop(gen) }, "azure_speech.pcm_speaker") + th.isDaemon = true + writer = th + th.start() + return true + } + } + + fun feed(pcm: ByteArray) { + if (!running || pcm.isEmpty()) return + queue.offer(pcm) + } + + /** 上层不再喂数据了:已排队的播完就报完成 */ + fun end() { + ended = true + } + + /** 打断:立刻停声,不报完成 */ + fun stop() { + synchronized(lock) { + generation++ + running = false + ended = false + queue.clear() + stopWriterLocked() + val t = track + if (t != null) { + try { + // pause + flush 才是"立刻不出声";stop() 会把缓冲里的先放完 + t.pause() + t.flush() + } catch (e: Exception) { + Log.w(TAG, "停止失败: ${e.message}") + } + } + } + } + + fun dispose() { + synchronized(lock) { + generation++ + running = false + queue.clear() + stopWriterLocked() + val t = track + track = null + trackRate = 0 + try { + t?.pause() + t?.flush() + t?.release() + } catch (e: Exception) { + Log.w(TAG, "释放失败: ${e.message}") + } + } + } + + // ---------- 内部 ---------- + + private fun writeLoop(gen: Int) { + try { + while (running && gen == generation) { + val chunk = queue.poll(POLL_MS, TimeUnit.MILLISECONDS) + if (chunk != null) { + writeFully(chunk, gen) + continue + } + // 队列空了。没收到 end() 就继续等下一块(服务端还在合成) + if (!ended) continue + if (awaitDrained(gen)) { + finish(gen) + return + } + } + } catch (e: InterruptedException) { + Thread.currentThread().interrupt() + } catch (e: Exception) { + Log.e(TAG, "写线程异常: ${e.message}") + // 异常也要收尾,否则调用方永远等不到 onCompleted, + // 服务端会一直停在 Responding + finish(gen) + } + } + + private fun writeFully(chunk: ByteArray, gen: Int) { + val t = track ?: return + var off = 0 + while (off < chunk.size && running && gen == generation) { + val n = try { + t.write(chunk, off, chunk.size - off) + } catch (e: Exception) { + Log.e(TAG, "写入失败: ${e.message}") + return + } + if (n <= 0) return // ERROR_INVALID_OPERATION / ERROR_DEAD_OBJECT + off += n + } + framesWritten += chunk.size / 2 + } + + /** + * 等已写入的帧真正播完。返回 false 表示这一轮被 stop() 掀掉了。 + * + * `playbackHeadPosition` 是 Int 且会回绕,但一轮回复远到不了 2^31 帧 + * (16k 下约 37 小时),直接比就行。 + */ + private fun awaitDrained(gen: Int): Boolean { + val t = track ?: return true + val deadline = System.currentTimeMillis() + DRAIN_TIMEOUT_MS + while (running && gen == generation) { + // 期间又来了新数据(服务端接着说):回到主循环继续写 + if (queue.isNotEmpty()) return false + val head = try { + t.playbackHeadPosition + } catch (e: Exception) { + return true + } + if (head >= framesWritten) return true + if (System.currentTimeMillis() > deadline) { + Log.w(TAG, "等待播完超时(已写 $framesWritten 帧,已播 $head 帧),按播完处理") + return true + } + try { + Thread.sleep(POLL_MS) + } catch (e: InterruptedException) { + Thread.currentThread().interrupt() + return false + } + } + return false + } + + private fun finish(gen: Int) { + synchronized(lock) { + if (gen != generation) return + running = false + ended = false + framesWritten = 0 + try { + // 一轮播完就把 track 停掉,别让它空跑着占住输出。 + // 这时缓冲已经排空(awaitDrained 保证),stop() 不会切掉任何声音。 + track?.stop() + } catch (e: Exception) { + Log.w(TAG, "停止 track 失败: ${e.message}") + } + } + onCompleted() + } + + private fun stopWriterLocked() { + val th = writer + writer = null + th?.interrupt() + } + + /** 采样率变了要重建:AudioTrack 的采样率是构造时定死的 */ + private fun ensureTrackLocked(sampleRate: Int): Boolean { + val t = track + if (t != null && trackRate == sampleRate && + t.state == AudioTrack.STATE_INITIALIZED + ) { + try { + t.pause() + t.flush() // 上一轮的残留不能带到这一轮 + } catch (_: Exception) { + } + return true + } + try { + t?.release() + } catch (_: Exception) { + } + track = null + trackRate = 0 + + val minBuf = AudioTrack.getMinBufferSize( + sampleRate, AudioFormat.CHANNEL_OUT_MONO, AudioFormat.ENCODING_PCM_16BIT + ) + if (minBuf <= 0) { + Log.e(TAG, "不支持的采样率 $sampleRate(getMinBufferSize=$minBuf)") + return false + } + // 给到 ~200ms,留足抖动余量。太小会在弱网分块到达不均时出现断音。 + val bufSize = maxOf(minBuf, sampleRate * 2 / 5) + + val attrs = AudioAttributes.Builder() + // 与 emai_audio_session.dart 里声明的 AndroidAudioAttributes 对齐, + // 改这两行会改变输出路由(比如 A2DP 还走不走) + .setUsage(AudioAttributes.USAGE_MEDIA) + .setContentType(AudioAttributes.CONTENT_TYPE_SPEECH) + .build() + val fmt = AudioFormat.Builder() + .setEncoding(AudioFormat.ENCODING_PCM_16BIT) + .setSampleRate(sampleRate) + .setChannelMask(AudioFormat.CHANNEL_OUT_MONO) + .build() + + val newTrack = try { + if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.M) { + AudioTrack.Builder() + .setAudioAttributes(attrs) + .setAudioFormat(fmt) + .setBufferSizeInBytes(bufSize) + .setTransferMode(AudioTrack.MODE_STREAM) + .build() + } else { + @Suppress("DEPRECATION") + AudioTrack( + AudioManager.STREAM_MUSIC, sampleRate, AudioFormat.CHANNEL_OUT_MONO, + AudioFormat.ENCODING_PCM_16BIT, bufSize, AudioTrack.MODE_STREAM + ) + } + } catch (e: Exception) { + Log.e(TAG, "创建 AudioTrack 失败: ${e.message}") + return false + } + if (newTrack.state != AudioTrack.STATE_INITIALIZED) { + Log.e(TAG, "AudioTrack 未初始化成功 state=${newTrack.state}") + try { + newTrack.release() + } catch (_: Exception) { + } + return false + } + track = newTrack + trackRate = sampleRate + return true + } +} diff --git a/apps/client/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/PeerSpeechGate.kt b/apps/client/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/PeerSpeechGate.kt new file mode 100644 index 00000000..21243993 --- /dev/null +++ b/apps/client/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/PeerSpeechGate.kt @@ -0,0 +1,81 @@ +package com.yunqiinnovation.azure_speech + +import kotlin.math.roundToInt +import kotlin.math.sqrt + +/** + * 对端(B 路)说话人门限——Dart 版 `lib/data/utils/peer_speech_gate.dart` 的逐条移植。 + * + * **在对端真的开口之前,不要把音频推给端到端翻译。** 阿里的实时声音复刻是 + * `frequency = "once"`,只在会话开头刻一次音色;此时对端往往还没开口,B 路先收到的是 + * 机主声音的串音,刻错之后用户听到的对端译文用的是自己的声音。 + * + * - 未开门期间音频进环形缓冲([prerollFrames]),开门瞬间整段补发——对端先开口也不吃掉开头; + * - 连续 [openFrames] 帧 RMS ≥ [rmsThreshold] 才开门,断续的串音尖峰不算; + * - 收满 [fallbackFrames] 仍未开门就**强行开门**(门限太严=整通电话不翻译,比刻错音色严重得多); + * - 每次会话重建(含断线重连)必须 [reset]。 + * + * ⚠️ 参数默认值与 Dart 版、Swift 版三处相同,调阈值要三处同改。 + * 线程:调用方保证串行(BesCallPcmBridge 的 ingress 线程)。 + */ +class PeerSpeechGate( + val rmsThreshold: Int = 600, + val openFrames: Int = 8, + val prerollFrames: Int = 25, + val fallbackFrames: Int = 500, +) { + var isOpen = false + private set + var openRms = 0 + private set + var openedByFallback = false + private set + private var run = 0 + private var seen = 0 + private val preroll = ArrayDeque() + + /** 喂一帧,返回应当推给翻译服务的帧(0 帧、1 帧,或开门瞬间的一整串)。 */ + fun accept(frame: ByteArray): List { + if (isOpen) return listOf(frame) + seen++ + val rms = rmsOfPcm16(frame) + if (rms >= rmsThreshold) run++ else run = 0 + preroll.addLast(frame) + while (preroll.size > prerollFrames) preroll.removeFirst() + val hitThreshold = run >= openFrames + val hitFallback = seen >= fallbackFrames + if (!hitThreshold && !hitFallback) return emptyList() + isOpen = true + openRms = rms + openedByFallback = !hitThreshold + val out = preroll.toList() + preroll.clear() + return out + } + + fun reset() { + isOpen = false + run = 0 + seen = 0 + openRms = 0 + openedByFallback = false + preroll.clear() + } + + companion object { + /** PCM16 小端单声道的 RMS;字节数为奇数时忽略末尾半个样本。 */ + fun rmsOfPcm16(frame: ByteArray): Int { + val n = frame.size / 2 + if (n == 0) return 0 + var sum = 0.0 + for (i in 0 until n) { + val lo = frame[i * 2].toInt() and 0xFF + val hi = frame[i * 2 + 1].toInt() + val s = ((hi shl 8) or lo).toDouble() + sum += s * s + } + val mean = sum / n + return if (mean.isFinite()) sqrt(mean).roundToInt() else 0 + } + } +} diff --git a/apps/client/local_plugins/azure_speech/ios/azure_speech/Sources/azure_speech/AzureSpeechPlugin.swift b/apps/client/local_plugins/azure_speech/ios/azure_speech/Sources/azure_speech/AzureSpeechPlugin.swift index e33b5da2..645d2d4c 100644 --- a/apps/client/local_plugins/azure_speech/ios/azure_speech/Sources/azure_speech/AzureSpeechPlugin.swift +++ b/apps/client/local_plugins/azure_speech/ios/azure_speech/Sources/azure_speech/AzureSpeechPlugin.swift @@ -63,6 +63,9 @@ import ble_service // 通话翻译下行:A/B 两路 TTS PCM 以 {leg, pcm, final} 抩回 Dart, // 由 translation_controller 转发给 device_jieli 回灌耳机。 case astTts + // EMAI 会话(百炼多模态)的文本与状态事件。音频不走这里—— + // 上行/下行 PCM 都在原生之间直连(见 BesAiPcmBridge)。 + case bailianDialog } /** * 通道配置结构体 @@ -125,6 +128,50 @@ import ble_service private var bailianAstHelperA = AliyunBailianE2EHelper() private var bailianAstHelperB = AliyunBailianE2EHelper() private var currentAstProvider: String = "azure" + /// 通话翻译 PCM 原生直连桥,按会话开关(见 setNativeCallBridge) + private var callPcmBridge: BesCallPcmBridge? + + /// AST 两路 helper 是否已就绪。 + /// + /// ⚠️ **不能只靠 `callPcmBridge?.markReady()` 那一句**:建桥(setCallPcmBridge)和 + /// AST 的异步初始化是两条独立路径,谁先完成不确定。初始化先完成时,那句 optional + /// chaining 打在 nil 上、**静默丢失**;之后桥建起来 `ready` 恒为 false,上行帧全部 + /// 积在环形缓冲里一帧都不推给 AST —— 真机表现就是「打开通话翻译没反应,关掉再开 + /// 一次又好了」(第二次恰好是桥先建好、初始化后到)。2026-09-21 真机实测:连续三次 + /// 会话 `ready:false`、`downlinkFrames:0`,第四次才 `ready:true`。 + /// 所以就绪状态记在插件上,建桥时继承。 + private var astSessionReady = false + private let astReadyLock = NSLock() + + private func setAstSessionReady(_ v: Bool) { + astReadyLock.lock(); astSessionReady = v; astReadyLock.unlock() + } + private var isAstSessionReady: Bool { + astReadyLock.lock(); defer { astReadyLock.unlock() } + return astSessionReady + } + /// AST 就绪:置标志 + 通知桥。桥还没建也没关系,建的时候会自己补上。 + private func markAstBridgeReady() { + setAstSessionReady(true) + callPcmBridge?.markReady() + } + + private func ensureCallPcmBridge() -> BesCallPcmBridge { + if let b = callPcmBridge { return b } + let b = BesCallPcmBridge() + b.onLegA = { [weak self] pcm in self?.pushAstAudioToA(pcm) } + b.onLegB = { [weak self] pcm in self?.pushAstAudioToB(pcm) } + b.onGateOpened = { [weak self] rms, fallback in + self?.sendAstEvent(["type": "peerGateOpened", "rms": rms, "fallback": fallback]) + } + b.onOverflow = { [weak self] dropped in + self?.sendAstEvent(["type": "nativeBridgeOverflow", "dropped": dropped]) + } + callPcmBridge = b + // 桥比 AST 初始化晚建:把错过的那次 markReady 补回来,否则这一整场会话都不会 ready + if isAstSessionReady { b.markReady() } + return b + } // AST 初始化世代号:每次 dispose / initialize 自增;异步任务在恢复时比对,过期就 return, // 避免新 provider 已切换后旧 init 的延迟代码再落到当前 helper 上。 private var astInitGeneration: Int = 0 @@ -238,6 +285,18 @@ import ble_service return data.prefix(take).map { String(format: "%02x", $0) }.joined() } + // ---------- EMAI 会话(百炼多模态)原生客户端 ---------- + // + // 耳机唤醒那条链路的 WS 与音频收发都在原生:上行 PCM 由 bluetooth_manager 经 + // AiPcmBridge 直达 [aiPcmBridge],攒成 100ms 一包推给 [bailianClient]; + // 合成回来的 PCM 原路回到耳机下行发送器。Dart 只收文本与状态。 + // 手机麦那条链路仍走 Dart 的 BailianMultimodalService,两者互不影响。 + private var bailianDialogEventChannel: FlutterEventChannel? + private var bailianDialogChannel: FlutterMethodChannel? + internal var bailianDialogEventSink: FlutterEventSink? + private var bailianClient: BailianDialogClient? + private var aiPcmBridge: BesAiPcmBridge? + // 音频数据相关 private var audioDataEventChannel: FlutterEventChannel? @@ -297,8 +356,8 @@ import ble_service case .ast: self.astChannel = methodChannel self.astEventChannel = eventChannel - case .astTts: - // 不走 setupChannel 路径(在 register 里单独注册),这里仅为了详尽性。 + case .astTts, .bailianDialog: + // 都不走 setupChannel 路径(在 register 里单独注册),这里仅为了详尽性。 break } } @@ -328,6 +387,10 @@ private func sendEvent(_ event: [String: Any], type: EventType) { // .astTts 不走 sendEvent 通用路径(单独走 pushAstTtsFrame),这里仅为了详尽性。 eventSink = astTtsEventSink logMessage = "无法发送AST-TTS事件:事件通道未准备好" + case .bailianDialog: + // 同上:百炼会话事件由 BailianDialogClient 的回调直接投 sink,这里仅为了详尽性。 + eventSink = bailianDialogEventSink + logMessage = "无法发送百炼会话事件:事件通道未准备好" } guard eventSink != nil else { @@ -382,6 +445,19 @@ private func sendEvent(_ event: [String: Any], type: EventType) { astTtsEventChannel.setStreamHandler(BaseEventStreamHandler(instance: instance, eventType: .astTts)) instance.astTtsEventChannel = astTtsEventChannel + // EMAI 会话(百炼多模态):方法通道 + 事件通道。音频不经过这两条。 + let bailianDialogEventChannel = FlutterEventChannel( + name: "azure_speech/bailian_dialog_events", binaryMessenger: registrar.messenger()) + bailianDialogEventChannel.setStreamHandler( + BaseEventStreamHandler(instance: instance, eventType: .bailianDialog)) + instance.bailianDialogEventChannel = bailianDialogEventChannel + let bailianDialogChannel = FlutterMethodChannel( + name: "azure_speech/bailian_dialog", binaryMessenger: registrar.messenger()) + bailianDialogChannel.setMethodCallHandler { [weak instance] call, result in + instance?.handleBailianDialogCall(call, result: result) + } + instance.bailianDialogChannel = bailianDialogChannel + // EMAI 下行 PCM 播放(iOS 专用):just_audio 在 iOS 放不了直播 PCM 流, // 详见 PcmSpeakerPlayer。只有方法通道,完成回调用 invokeMethod 回传。 let pcmPlayerChannel = FlutterMethodChannel( @@ -400,6 +476,96 @@ private func sendEvent(_ event: [String: Any], type: EventType) { } } +// ===== EMAI 会话(百炼多模态)的方法分发 ===== +// +// 会话的**编排**仍在 Dart(DeviceAiSessionService):什么时候起、什么时候收、 +// 门控什么时候开关、指令怎么落地,都在那边。这里只负责协议与音频搬运。 +internal func handleBailianDialogCall(_ call: FlutterMethodCall, result: @escaping FlutterResult) { + let args = call.arguments as? [String: Any] + switch call.method { + case "start": + guard let a = args else { result(false); return } + let bridge = ensureAiPcmBridge() + let client = ensureBailianClient() + let cfg = BailianDialogClient.Config( + apiKey: (a["apiKey"] as? String) ?? "", + appId: (a["appId"] as? String) ?? "", + workspaceId: (a["workspaceId"] as? String) ?? "", + mode: (a["mode"] as? String) ?? "duplex", + voice: a["voice"] as? String, + userId: (a["userId"] as? String) ?? "", + enableWebSearch: (a["enableWebSearch"] as? Bool) ?? true, + downstreamRate: (a["downstreamRate"] as? Int) ?? 16000, + mcpToken: a["mcpToken"] as? String, + mcpAuthTools: (a["mcpAuthTools"] as? [String]) ?? [] + ) + let ok = client.start(config: cfg) + if ok { bridge.enable() } + result(ok) + + case "stop": + aiPcmBridge?.disable() + bailianClient?.stop() + result(true) + + case "sendText": + bailianClient?.sendText((args?["text"] as? String) ?? "", + speakDirectly: (args?["speakDirectly"] as? Bool) ?? false) + result(true) + + case "beginSpeech": bailianClient?.beginSpeech(); result(true) + case "endSpeech": bailianClient?.endSpeech(); result(true) + case "cancelSpeech": bailianClient?.cancelSpeech(); result(true) + case "interrupt": bailianClient?.interrupt(); result(true) + case "localRespondingStarted": bailianClient?.notifyLocalRespondingStarted(); result(true) + case "localRespondingEnded": bailianClient?.notifyLocalRespondingEnded(); result(true) + + // 播报期间把上行换成等长静音。开关时机由 Dart 的状态机决定(见 BesAiPcmBridge 注释) + case "setGated": + aiPcmBridge?.setGated((args?["gated"] as? Bool) ?? false) + result(true) + + case "stats": + var m = bailianClient?.stats() ?? ["running": false] + if let b = aiPcmBridge?.stats() { m.merge(b) { a, _ in a } } + result(m) + + default: + result(FlutterMethodNotImplemented) + } +} + +private func ensureBailianClient() -> BailianDialogClient { + if let c = bailianClient { return c } + let c = BailianDialogClient() + c.onEvent = { [weak self] e in + self?.bailianDialogEventSink?(e) + } + c.onAudio = { [weak self] pcm in + // 合成音频原路回耳机:不经过 Dart,也不经过手机扬声器 + self?.aiPcmBridge?.postDownlink(pcm) + } + bailianClient = c + return c +} + +private func ensureAiPcmBridge() -> BesAiPcmBridge { + if let b = aiPcmBridge { return b } + let b = BesAiPcmBridge() + b.onPacket = { [weak self] pkt in self?.bailianClient?.pushAudio(pkt) } + b.onMicActivity = { [weak self] in + // 只是给 Dart 的空闲计时续命,已按 500ms 节流 + DispatchQueue.main.async { self?.bailianDialogEventSink?(["event": "micActivity"]) } + } + b.onOverflow = { [weak self] dropped in + DispatchQueue.main.async { + self?.bailianDialogEventSink?(["event": "overflow", "dropped": dropped]) + } + } + aiPcmBridge = b + return b +} + // EMAI 下行 PCM 播放的方法分发 internal func handlePcmPlayerMethodCall(_ call: FlutterMethodCall, result: @escaping FlutterResult) { switch call.method { @@ -1143,9 +1309,27 @@ private func sendAudioDataEvent(_ event: [String: Any]) { } result(true) + // 通话翻译 PCM 原生直连桥(见 BesCallPcmBridge.swift;对端是 bluetooth_manager 的 CallPcmBridge.swift)。 + // 开着时耳机双路 PCM 不经 Dart 直达 AST,译文 PCM 直回耳机下行发送器; + // Dart 只收 TTS 的 final 标记与门限/丢帧的诊断事件。 + case "setNativeCallBridge": + let args = call.arguments as? [String: Any] + let enabled = (args?["enabled"] as? Bool) ?? false + if enabled { + ensureCallPcmBridge().enable() + } else { + callPcmBridge?.disable() + } + result(true) + + case "getNativeCallBridgeStats": + result(callPcmBridge?.stats() ?? ["enabled": false]) + case "dispose": // 让 init 的 Task 在 sleep 之后比对 generation 时直接 return,避免落到已 dispose 的 helper 上 _ = nextAstInitGeneration() + setAstSessionReady(false) + callPcmBridge?.onSessionDisposed() os_log("[AST-DISPOSE] Releasing all AST resources, provider=%{public}@", log: ctLog, type: .info, currentAstProvider) azureAstHelperA.dispose() azureAstHelperB.dispose() @@ -1188,6 +1372,9 @@ private func sendAudioDataEvent(_ event: [String: Any]) { let ttsLang0 = supported.count > 4 ? supported[4] : "zh-CN-XiaoxiaoNeural" let ttsLang1 = supported.count > 5 ? supported[5] : "en-US-AriaNeural" currentAstProvider = provider + // 原生直连桥:新会话重新刻音色 → 门限复位;helper 起来之前的帧先攒着(见 BesCallPcmBridge) + setAstSessionReady(false) + callPcmBridge?.onSessionInit() let subscriptionKeyI = (args["subscriptionKey"] as? String) ?? "" let regionI = (args["region"] as? String) ?? "" @@ -1269,6 +1456,7 @@ private func sendAudioDataEvent(_ event: [String: Any]) { os_log("[AST-INIT] Azure: initializing B service...", log: self.ctLog, type: .info) await azureAstHelperB.initialize(azureConfig: azureConfigI, translationConfig: translationConfigI, serviceConfig: svcB, callback: callbackB) os_log("[AST-INIT] Azure: A+B both initialized", log: self.ctLog, type: .info) + self.markAstBridgeReady() DispatchQueue.main.async { result(true) } } } else if currentAstProvider == "volcano" { @@ -1304,6 +1492,7 @@ private func sendAudioDataEvent(_ event: [String: Any]) { os_log("[AST-INIT] Volcano: initializing B service...", log: self.ctLog, type: .info) let rB = await doubaoAstHelperB.initialize(config: cfgB, cb: cbB) os_log("[AST-INIT] Volcano: B service result=%d. Both services ready.", log: self.ctLog, type: .info, rB ? 1 : 0) + self.markAstBridgeReady() DispatchQueue.main.async { result(true) } } } else if currentAstProvider == "alibaba" { @@ -1355,9 +1544,11 @@ private func sendAudioDataEvent(_ event: [String: Any]) { } let rB = await bailianAstHelperB.initialize(config: cfgB, cb: cbB) os_log("[AST-INIT] Alibaba: B service result=%d. Both services ready.", log: self.ctLog, type: .info, rB ? 1 : 0) + self.markAstBridgeReady() DispatchQueue.main.async { result(true) } } } else { + markAstBridgeReady() result(true) } @@ -1376,6 +1567,11 @@ private func sendAudioDataEvent(_ event: [String: Any]) { if leg == "B", !pcm.isEmpty, astStereoRecorder.isActive { astStereoRecorder.enqueueRightChannel(pcm) } + // 原生直连桥开着:PCM 直接回耳机下行发送器,Dart 只拿 final 标记(不再逐块过通道) + if let bridge = callPcmBridge, bridge.isEnabled { + bridge.postDownlink(leg: leg, pcm: pcm) + if !isFinal { return } + } guard let sink = astTtsEventSink else { // 调试:报警一次,避免刷屏 astTtsEmitCount &+= 1 @@ -1764,6 +1960,9 @@ private class BaseEventStreamHandler: NSObject, FlutterStreamHandler { case .astTts: instance?.astTtsEventSink = events print("[AST-TTS] event channel ready (azure_speech/ast_tts_events)") + case .bailianDialog: + instance?.bailianDialogEventSink = events + print("[BailianDialog] event channel ready (azure_speech/bailian_dialog_events)") } return nil } @@ -1782,6 +1981,9 @@ private class BaseEventStreamHandler: NSObject, FlutterStreamHandler { case .astTts: instance?.astTtsEventSink = nil print("[AST-TTS] event channel cancelled") + case .bailianDialog: + instance?.bailianDialogEventSink = nil + print("[BailianDialog] event channel cancelled") } return nil } diff --git a/apps/client/local_plugins/azure_speech/ios/azure_speech/Sources/azure_speech/BailianDialogClient.swift b/apps/client/local_plugins/azure_speech/ios/azure_speech/Sources/azure_speech/BailianDialogClient.swift new file mode 100644 index 00000000..df26cb3b --- /dev/null +++ b/apps/client/local_plugins/azure_speech/ios/azure_speech/Sources/azure_speech/BailianDialogClient.swift @@ -0,0 +1,487 @@ +import Foundation + +/// 阿里云百炼「多模态交互协议」的**原生**客户端(EMAI 助手)。 +/// +/// 协议文档:https://help.aliyun.com/zh/model-studio/multimodal-interaction-protocol +/// 端点:`wss://dashscope.aliyuncs.com/api-ws/v1/inference` +/// +/// ## 为什么有原生这一份 +/// Dart 侧的 `BailianMultimodalService` 仍在(手机麦那条链路、以及本类起不来时的退路)。 +/// 耳机按键唤醒那条链路原先是 +/// `bluetooth_manager → EventChannel → Dart → WS(Dart) → Dart → MethodChannel → bluetooth_manager`, +/// 上行 20ms 一帧、下行每个音频块一次,全压在 Dart 的单 isolate 上,和页面布局、GC +/// 排同一个 event loop。本类把 WS 连同音频收发搬到原生:PCM 经 `BesAiPcmBridge` +/// 在原生之间直连,**Dart 只收文本与状态事件**。 +/// +/// ## ⚠️ 这不修「锁屏断连」 +/// iOS 挂起的是整个进程,原生和 Dart 一起停。锁屏后能不能活着取决于 +/// `UIBackgroundModes=audio` + 确实有音频在播(Dart 侧 `BackgroundKeepalive` 干的事), +/// 与代码在哪一层无关。本类解决的是 event loop 抖动与跨界开销,别把它当保活手段。 +/// +/// ## ⚠️ 与 Dart 版**逐条对齐** +/// 握手参数(upstream/downstream/client_info/biz_params)、事件名映射、心跳间隔都照 +/// `lib/data/services/bailian_multimodal_service.dart` 写。改一边必须同改另一边, +/// 否则「用耳机唤醒」和「在 App 里按住说话」会变成两种行为,而且两边都不报错。 +/// +/// ## 指令解析刻意**留在 Dart** +/// `extra_info.tool_calls` 的解析(参数是字符串套 JSON、值恒为中文、下午时间要校正…) +/// 有一批真实 payload 的金测试守着(`test/assistant_directive_test.dart`), +/// 而解析结果最终要落进 GetStorage 和「代办」页——在原生解析再序列化回 Dart 毫无收益, +/// 只会把那批测试作废。这里把整个 `output` 原样上抛,Dart 照旧解析。 +final class BailianDialogClient: NSObject { + + // MARK: - 配置 + + struct Config { + let apiKey: String + let appId: String + let workspaceId: String + let mode: String // push2talk / tap2talk / duplex + let voice: String? // 留空用百炼应用自己配置的音色 + let userId: String + let enableWebSearch: Bool + let downstreamRate: Int + let mcpToken: String? + /// 需要用户身份的 MCP 工具名。**由 Dart 传下来**,不在原生再抄一份: + /// 服务端加工具时只改 Dart 那一处,漏改的表现是那个工具永远「身份校验失败」。 + let mcpAuthTools: [String] + } + + private static let endpoint = "wss://dashscope.aliyuncs.com/api-ws/v1/inference" + + /// 上行音频:16k / 16bit / 单声道 / 小端 + private static let upstreamSampleRate = 16000 + + /// 服务端要求任意连续 60s 内必须有消息往来;文档建议 50s 一次心跳 + private static let heartbeatInterval: TimeInterval = 50 + + /// WS 层 ping,与业务心跳是两回事(对齐 Dart 的 pingInterval: 20s) + private static let pingInterval: TimeInterval = 20 + + // MARK: - 回调 + + /// 业务事件(已拍平成 Dart 能直接用的字典),在主队列上回调 + var onEvent: (([String: Any]) -> Void)? + + /// 下行合成音频(PCM,采样率 = config.downstreamRate),在接收队列上回调 + var onAudio: ((Data) -> Void)? + + // MARK: - 状态 + + private let queue = DispatchQueue(label: "azure_speech.bailian.dialog") + private var session: URLSession? + private var task: URLSessionWebSocketTask? + private var heartbeatTimer: DispatchSourceTimer? + private var pingTimer: DispatchSourceTimer? + private var config: Config? + private var taskId = "" + private var dialogId = "" + private var running = false + + private var uplinkBytes: Int64 = 0 + private var downlinkBytes: Int64 = 0 + /// 下行字节数的上报节流。Dart 侧不再拿到 PCM,但它要靠「首包到达」开门控、 + /// 靠累计字节数估算播完时刻(门控的收尾计时),所以这两件事必须报上去。 + /// 按 100ms 汇总一次,别每块都报——那就把省下来的跨界又加回去了。 + private var lastDownlinkReportAt: TimeInterval = 0 + private static let downlinkReportInterval: TimeInterval = 0.1 + + var isRunning: Bool { + queue.sync { running } + } + + // MARK: - 生命周期 + + /// 建立会话。返回 false 表示配置不全或连接没起来。 + func start(config: Config) -> Bool { + guard !config.apiKey.trimmingCharacters(in: .whitespaces).isEmpty else { + NSLog("[BailianDialog] 未配置 API Key,不尝试连接") + // Key 缺失时不要尝试连接——服务端会先回 task-started 再立刻 task-failed, + // 日志上看起来像"连上了又掉",容易误判成网络问题 + emit(["event": "error", "text": "未配置 ALIBABA_OPENSPEECH_APP_KEY", + "code": "NOT_CONFIGURED", "fatal": true]) + return false + } + stop() + return queue.sync { + self.config = config + taskId = Self.uuid() + dialogId = "" + uplinkBytes = 0 + downlinkBytes = 0 + lastDownlinkReportAt = 0 + + guard let url = URL(string: Self.endpoint) else { return false } + var req = URLRequest(url: url) + req.setValue("Bearer \(config.apiKey)", forHTTPHeaderField: "Authorization") + req.setValue("enable", forHTTPHeaderField: "X-DashScope-DataInspection") + + let cfg = URLSessionConfiguration.default + let s = URLSession(configuration: cfg) + let t = s.webSocketTask(with: req) + session = s + task = t + running = true + t.resume() + receiveLoop() + sendJson(startPayload(config)) + startTimers() + NSLog("[BailianDialog] 会话启动 app_id=%@ workspace=%@ mode=%@ downstream=%dHz", + config.appId, config.workspaceId, config.mode, config.downstreamRate) + return true + } + } + + /// 结束会话:先发 Stop 让服务端收尾,再 finish-task 结束整个 WS 任务 + func stop() { + queue.sync { + guard running, let t = task else { + cleanupLocked() + return + } + sendDirectiveLocked("Stop") + sendJsonLocked([ + "header": ["action": "finish-task", "task_id": taskId, "streaming": "duplex"], + "payload": ["input": [String: Any]()], + ]) + t.cancel(with: .normalClosure, reason: nil) + NSLog("[BailianDialog] 会话结束 上行=%lld 字节 下行=%lld 字节", uplinkBytes, downlinkBytes) + cleanupLocked() + } + } + + // MARK: - 上行 + + /// 推一包上行音频(PCM 16k/16bit/mono)。由 `BesAiPcmBridge` 的串行队列调用。 + func pushAudio(_ pcm: Data) { + guard !pcm.isEmpty else { return } + queue.async { + guard self.running, let t = self.task else { return } + self.uplinkBytes &+= Int64(pcm.count) + t.send(.data(pcm)) { err in + if let e = err { NSLog("[BailianDialog] 推送音频失败: %@", e.localizedDescription) } + } + } + } + + /// push2talk:开始/结束/放弃一段说话 + func beginSpeech() { sendDirective("SendSpeech") } + func endSpeech() { sendDirective("StopSpeech") } + func cancelSpeech() { sendDirective("CancelSpeech") } + + /// 打断当前回复并请求发言权 + func interrupt() { sendDirective("RequestToSpeak") } + + /// 本地播报开始/结束(服务端据此推进对话状态) + func notifyLocalRespondingStarted() { sendDirective("LocalRespondingStarted") } + func notifyLocalRespondingEnded() { sendDirective("LocalRespondingEnded") } + + /// 文本对话:交给应用的 LLM 处理(prompt),或让它直接朗读(transcript) + func sendText(_ text: String, speakDirectly: Bool) { + let t = text.trimmingCharacters(in: .whitespacesAndNewlines) + guard !t.isEmpty else { return } + sendDirective("RequestToRespond", extra: [ + "type": speakDirectly ? "transcript" : "prompt", + "text": t, + ]) + } + + func stats() -> [String: Any] { + queue.sync { + ["running": running, "uplinkBytes": uplinkBytes, + "downlinkBytes": downlinkBytes, "dialogId": dialogId] + } + } + + // MARK: - 协议帧 + + private func startPayload(_ c: Config) -> [String: Any] { + var downstream: [String: Any] = [ + "audio_format": "pcm", + "sample_rate": c.downstreamRate, + // transcript:中间态只回朗读文本,跟 TTS 出声一致, + // 免得 UI 上先闪一段 dialog 思考态文本再被覆盖 + "intermediate_text": "transcript", + // 显式钉死"每帧回全文":Dart 收到 RespondingContent 是整体替换气泡内容的, + // 改成增量会导致只显示最后一个片段 + "incremental_response": false, + ] + // 不传 voice:音色属于百炼应用的配置,客户端硬编码一个名字只会跟应用实际用的 + // TTS 模型对不上(实测传 longxiaochun_v2 会被回 "tts voice error") + if let v = c.voice?.trimmingCharacters(in: .whitespaces), !v.isEmpty { + downstream["voice"] = v + } + + var parameters: [String: Any] = [ + "upstream": [ + "type": "AudioOnly", + "mode": c.mode, + "audio_format": "pcm", + "sample_rate": Self.upstreamSampleRate, + ], + "downstream": downstream, + "client_info": ["user_id": Self.safeUserId(c.userId)], + ] + + // biz_params 是可选块,两件事都没有时整块不发——万一应用侧没开这些能力, + // 多带一个字段可能让 Start 直接被拒,那样整个功能都起不来 + let hasToken = !(c.mcpToken ?? "").trimmingCharacters(in: .whitespaces).isEmpty + if c.enableWebSearch || hasToken { + var userDefined: [String: Any] = [:] + if hasToken, let tok = c.mcpToken { + // ⚠️ **必须按工具名嵌套**,平铺写 `auth_token` 会被静默丢弃 + for tool in c.mcpAuthTools { + userDefined[tool] = ["auth_token": tok] + } + } + if c.enableWebSearch { + userDefined["extra_config"] = ["enable_web_search": true] + } + parameters["biz_params"] = ["user_defined_params": userDefined] + } + + return [ + "header": ["action": "run-task", "task_id": taskId, "streaming": "duplex"], + "payload": [ + "task_group": "aigc", + "task": "multimodal-generation", + "function": "generation", + "model": "multimodal-dialog", + "input": [ + "directive": "Start", + "workspace_id": c.workspaceId, + "app_id": c.appId, + ], + "parameters": parameters, + ], + ] + } + + private func sendDirective(_ directive: String, extra: [String: Any]? = nil) { + queue.async { self.sendDirectiveLocked(directive, extra: extra) } + } + + private func sendDirectiveLocked(_ directive: String, extra: [String: Any]? = nil) { + guard running, let c = config else { return } + var input: [String: Any] = [ + "directive": directive, + "workspace_id": c.workspaceId, + "app_id": c.appId, + ] + if !dialogId.isEmpty { input["dialog_id"] = dialogId } + if let e = extra { input.merge(e) { _, new in new } } + sendJsonLocked([ + "header": ["action": "continue-task", "task_id": taskId, "streaming": "duplex"], + "payload": [ + "task_group": "aigc", + "task": "multimodal-generation", + "function": "generation", + "model": "multimodal-dialog", + "input": input, + ], + ]) + } + + private func sendJson(_ msg: [String: Any]) { + queue.async { self.sendJsonLocked(msg) } + } + + private func sendJsonLocked(_ msg: [String: Any]) { + guard let t = task else { return } + guard let data = try? JSONSerialization.data(withJSONObject: msg), + let s = String(data: data, encoding: .utf8) else { + NSLog("[BailianDialog] 序列化失败") + return + } + t.send(.string(s)) { err in + if let e = err { NSLog("[BailianDialog] 发送失败: %@", e.localizedDescription) } + } + } + + // MARK: - 下行 + + private func receiveLoop() { + guard let t = task else { return } + t.receive { [weak self] result in + guard let self = self else { return } + switch result { + case .failure(let err): + self.queue.async { + guard self.running else { return } + NSLog("[BailianDialog] WebSocket 错误: %@", err.localizedDescription) + self.emit(["event": "error", "text": err.localizedDescription, "fatal": true]) + self.cleanupLocked() + } + case .success(let msg): + switch msg { + case .data(let d): + // 二进制帧就是下行合成音频 + self.queue.async { + self.downlinkBytes &+= Int64(d.count) + let now = Date().timeIntervalSince1970 + // 首块立刻报(Dart 要用它开门控),之后按间隔汇总 + if self.lastDownlinkReportAt == 0 + || now - self.lastDownlinkReportAt >= Self.downlinkReportInterval { + self.lastDownlinkReportAt = now + self.emit(["event": "downlink", "bytes": self.downlinkBytes]) + } + } + if !d.isEmpty { self.onAudio?(d) } + case .string(let s): + self.handleText(s) + @unknown default: + break + } + self.receiveLoop() + } + } + } + + private func handleText(_ raw: String) { + guard let data = raw.data(using: .utf8), + let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any] else { + NSLog("[BailianDialog] 无法解析的文本帧") + return + } + let header = obj["header"] as? [String: Any] ?? [:] + let payload = obj["payload"] as? [String: Any] ?? [:] + let event = (header["event"] as? String) ?? "" + + switch event { + case "task-started": + NSLog("[BailianDialog] task-started") + case "task-failed": + let code = (header["error_code"] as? String) ?? "" + let msg = (header["error_message"] as? String) ?? "" + NSLog("[BailianDialog] task-failed: %@ %@", code, msg) + // header 级 = 整个 task 结束,必须重连 + emit(["event": "error", "text": msg, "code": code, "fatal": true]) + case "task-finished": + NSLog("[BailianDialog] task-finished") + emit(["event": "closed"]) + default: + handleDirectiveEvent(event, payload) + } + } + + /// 业务事件都在 payload.output.event 里 + private func handleDirectiveEvent(_ outerEvent: String, _ payload: [String: Any]) { + let output = payload["output"] as? [String: Any] ?? [:] + let name = (output["event"] as? String) ?? outerEvent + let text = (output["text"] as? String) ?? "" + let finished = (output["finished"] as? Bool) ?? false + + switch name { + case "Started": + let did = (output["dialog_id"] as? String) ?? "" + queue.async { self.dialogId = did } + NSLog("[BailianDialog] Started dialog_id=%@", did) + emit(["event": "started"]) + case "DialogStateChanged": + // 协议里是首字母大写(Listening/Thinking/Responding),统一小写再交给 Dart + let s = ((output["state"] as? String) ?? "").lowercased() + emit(["event": "stateChanged", "state": s]) + case "SpeechStarted": + emit(["event": "speechStarted"]) + case "SpeechContent": + emit(["event": "speechContent", "text": text, "isFinal": finished]) + case "SpeechEnded": + emit(["event": "speechEnded", "text": text, "isFinal": true]) + case "RequestAccepted": + emit(["event": "requestAccepted"]) + case "RespondingStarted": + emit(["event": "respondingStarted"]) + case "RespondingContent": + let reason = (output["finish_reason"] as? String) ?? "" + // 指令解析留在 Dart(见类注释):这里把 output 原样带上去。 + // extra_info 只在整轮收尾那一帧出现,不是每个流式分片都有,频率可控。 + var e: [String: Any] = [ + "event": "respondingContent", + "text": text, + "isFinal": finished, + "finishReason": reason, + ] + if let extra = output["extra_info"] { + e["extraInfo"] = Self.jsonString(extra) + } + emit(e) + case "RespondingEnded": + emit(["event": "respondingEnded", "text": text, "isFinal": true]) + case "Error": + let code = (output["error_code"] as? String) ?? "" + let msg = (output["error_message"] as? String) ?? "" + let name0 = (output["error_name"] as? String) ?? "" + NSLog("[BailianDialog] Error: %@/%@ %@", code, name0, msg) + // payload 级 Error 是"这一轮不行",连接通常还在,不标 fatal + let shown = !msg.isEmpty ? msg : (!text.isEmpty ? text : name0) + emit(["event": "error", "text": shown, "code": code, "fatal": false]) + case "Stopped": + emit(["event": "closed"]) + case "HeartBeat": + break + default: + NSLog("[BailianDialog] 未处理事件: %@", name) + } + } + + // MARK: - 定时器 + + private func startTimers() { + let hb = DispatchSource.makeTimerSource(queue: queue) + hb.schedule(deadline: .now() + Self.heartbeatInterval, repeating: Self.heartbeatInterval) + hb.setEventHandler { [weak self] in self?.sendDirectiveLocked("HeartBeat") } + hb.resume() + heartbeatTimer = hb + + let pg = DispatchSource.makeTimerSource(queue: queue) + pg.schedule(deadline: .now() + Self.pingInterval, repeating: Self.pingInterval) + pg.setEventHandler { [weak self] in + self?.task?.sendPing { err in + if let e = err { NSLog("[BailianDialog] ping 失败: %@", e.localizedDescription) } + } + } + pg.resume() + pingTimer = pg + } + + private func cleanupLocked() { + heartbeatTimer?.cancel() + heartbeatTimer = nil + pingTimer?.cancel() + pingTimer = nil + task = nil + session?.invalidateAndCancel() + session = nil + dialogId = "" + running = false + } + + // MARK: - 工具 + + private func emit(_ e: [String: Any]) { + let cb = onEvent + DispatchQueue.main.async { cb?(e) } + } + + /// 把任意 JSON 结构转回字符串交给 Dart 解析(保持与 Dart 版同一份解析代码) + private static func jsonString(_ v: Any) -> String { + guard JSONSerialization.isValidJSONObject(v), + let d = try? JSONSerialization.data(withJSONObject: v), + let s = String(data: d, encoding: .utf8) else { return "" } + return s + } + + /// 协议限制 user_id 不超过 36 字符 + private static func safeUserId(_ raw: String) -> String { + let v = raw.trimmingCharacters(in: .whitespaces) + if v.isEmpty { return "eaimar-guest" } + return v.count <= 36 ? v : String(v.prefix(36)) + } + + private static func uuid() -> String { + func hex(_ n: Int) -> String { + (0.. Void)? + + /// 用户在说话(能量过门限、且不在门控中),已按 [activityThrottle] 节流。 + /// Dart 用它给空闲计时续命——设备上行有声也算活动,不然锁屏 + 蓝牙延迟下 + /// 很容易被判「双方无声」而提前收尾。 + var onMicActivity: (() -> Void)? + + /// 背压丢包时上报一次(累计丢包数) + var onOverflow: ((Int64) -> Void)? + + var isEnabled: Bool { + lock.lock(); defer { lock.unlock() } + return enabled + } + + func enable() { + lock.lock() + if enabled { lock.unlock(); return } + enabled = true + lock.unlock() + ingress.async { + self.accum.removeAll(keepingCapacity: true) + self.pending = 0 + self.gated = false + self.lastActivityAt = 0 + self.framesIn = 0 + self.packetsOut = 0 + self.dropped = 0 + self.downlinkFrames = 0 + } + observer = NotificationCenter.default.addObserver( + forName: Self.uplinkNotification, object: nil, queue: nil + ) { [weak self] note in + guard let self = self, + let pcm = note.userInfo?[Self.pcmKey] as? Data, !pcm.isEmpty else { return } + self.ingest(pcm) + } + NSLog("[BesAiPcmBridge] AI 上行直连已开启") + } + + func disable() { + lock.lock() + if !enabled { lock.unlock(); return } + enabled = false + lock.unlock() + if let o = observer { + NotificationCenter.default.removeObserver(o) + observer = nil + } + ingress.async { + self.accum.removeAll(keepingCapacity: false) + NSLog("[BesAiPcmBridge] AI 上行直连已关闭 上行=%lld 帧 出包=%lld 丢包=%lld 下行=%lld 帧", + self.framesIn, self.packetsOut, self.dropped, self.downlinkFrames) + } + } + + /// AI 是否正在耳机里播报。播报期间上行换成**等长静音**而不是断流: + /// 上行保持连续,服务端 VAD 收到的是"真的没人说话",而不是一段数据缺口。 + func setGated(_ v: Bool) { + ingress.async { + self.gated = v + // 关门时把攒了一半的静音丢掉,下一包从真实音频开始, + // 免得用户开口的头 100ms 里混着上一轮的静音 + if !v { self.accum.removeAll(keepingCapacity: true) } + } + } + + // MARK: - 上行 + + private func ingest(_ pcm: Data) { + // 背压:排队太多就丢新包。判断放在入队前,不然队列本身就是那个积压 + lock.lock() + let p = pending + lock.unlock() + if p >= Self.maxPendingPackets { + ingress.async { + self.dropped &+= 1 + let n = self.dropped + if n == 1 || n % 50 == 0 { + NSLog("[BesAiPcmBridge] 上行拥塞,已丢 %lld 包", n) + } + self.onOverflow?(n) + } + return + } + lock.lock(); pending += 1; lock.unlock() + ingress.async { + self.lock.lock(); self.pending -= 1; self.lock.unlock() + self.framesIn &+= 1 + if self.gated { + // 等长静音 + self.accum.append(Data(count: pcm.count)) + } else { + if Self.rms(pcm) >= Self.activityRms { + let now = Date().timeIntervalSince1970 + if now - self.lastActivityAt >= Self.activityThrottle { + self.lastActivityAt = now + self.onMicActivity?() + } + } + self.accum.append(pcm) + } + while self.accum.count >= Self.batchBytes { + let pkt = self.accum.prefix(Self.batchBytes) + self.accum = Data(self.accum.dropFirst(Self.batchBytes)) + self.packetsOut &+= 1 + self.onPacket?(Data(pkt)) + } + } + } + + // MARK: - 下行 + + /// 把一块合成音频交回 bluetooth_manager 的下行发送器(G.722 编码与节流在那边做) + func postDownlink(_ pcm: Data) { + guard !pcm.isEmpty else { return } + NotificationCenter.default.post(name: Self.downlinkNotification, object: nil, + userInfo: [Self.pcmKey: pcm]) + ingress.async { self.downlinkFrames &+= 1 } + } + + func stats() -> [String: Any] { + ingress.sync { + ["enabled": isEnabled, "framesIn": framesIn, "packetsOut": packetsOut, + "dropped": dropped, "downlinkFrames": downlinkFrames, "gated": gated] + } + } + + /// 16bit 小端 PCM 的 RMS(0~32767 量纲,与 Dart 的 `rmsOfPcm16` 同口径) + private static func rms(_ pcm: Data) -> Double { + let count = pcm.count / 2 + guard count > 0 else { return 0 } + var sum = 0.0 + pcm.withUnsafeBytes { raw in + guard let base = raw.baseAddress?.assumingMemoryBound(to: UInt8.self) else { return } + for i in 0..= 0x8000 { v -= 0x10000 } + sum += Double(v) * Double(v) + } + } + return (sum / Double(count)).squareRoot() + } +} diff --git a/apps/client/local_plugins/azure_speech/ios/azure_speech/Sources/azure_speech/BesCallPcmBridge.swift b/apps/client/local_plugins/azure_speech/ios/azure_speech/Sources/azure_speech/BesCallPcmBridge.swift new file mode 100644 index 00000000..0dc6bfd1 --- /dev/null +++ b/apps/client/local_plugins/azure_speech/ios/azure_speech/Sources/azure_speech/BesCallPcmBridge.swift @@ -0,0 +1,208 @@ +import Foundation + +/// 通话翻译 PCM 的**原生直连桥**(azure_speech 这一端)。 +/// +/// 对端是 bluetooth_manager 的 `CallPcmBridge.swift`:耳机解码出来的双路 PCM 经 +/// NotificationCenter 直接到这里(不再经过 Dart),这里过 B 路说话人门限后推给 AST helper; +/// AST 合成出来的译文 PCM 由 [postDownlink] 直接回给 bluetooth_manager 的下行发送器。 +/// +/// ## ⚠️ 与 bluetooth_manager 手工同步的契约(通知名 / userInfo 键) +/// - 上行 `com.eaimar.bes.callpcm.uplink` `leg`("A"=本端麦 / "B"=对端通话音), `pcm`(Data) +/// - 下行 `com.eaimar.bes.callpcm.downlink` `leg`("A"=对端听 / "B"=己方听), `pcm`(Data) +/// 改一处必须同改另一处;改漏了不会编译报错,只会「桥开着但一帧都到不了」。 +/// +/// ## 线程与背压 +/// 通知在耳机解码线程上同步到达,这里**立刻转到自己的串行队列** [ingress],不让 WS 发送 +/// 或 base64 编码拖住解码线程。队列里排队超过 [maxPendingFrames](2s)就丢新帧并计数: +/// 这就是原来 Dart 链路上「AudioSendSlow 1011」的替代品——同样的拥塞,现在有数字、有日志、 +/// 而且丢的是帧而不是整个会话。 +/// +/// ## 重连期间不丢音频 +/// AST helper 在 initialize 期间会被 dispose 重建,它自己的 pending 缓冲随之清空。 +/// 所以「未就绪」期间的帧留在**这里**的环形缓冲(每路 [holdFrames],3s), +/// [markReady] 时整段补推——弱网重连的那 1~4 秒不再是纯丢音频。 +final class BesCallPcmBridge { + static let uplinkNotification = Notification.Name("com.eaimar.bes.callpcm.uplink") + static let downlinkNotification = Notification.Name("com.eaimar.bes.callpcm.downlink") + static let legKey = "leg" + static let pcmKey = "pcm" + + private static let maxPendingFrames = 100 + private static let holdFrames = 150 + + private let ingress = DispatchQueue(label: "azure_speech.callpcm.ingress", qos: .userInitiated) + private let lock = NSLock() + + private var enabled = false + private var ready = false + private var observer: NSObjectProtocol? + private var pending = 0 + private let gate = PeerSpeechGate() + private var holdA: [Data] = [] + private var holdB: [Data] = [] + + // 统计(只在 ingress 队列上写;读走 stats()) + private var framesA: Int64 = 0 + private var framesB: Int64 = 0 + private var dropped: Int64 = 0 + private var gatedB: Int64 = 0 + private var downlinkFrames: Int64 = 0 + private var maxPendingSeen = 0 + + /// 上行落点:A = 本端麦 → A 路 AST;B = 对端通话音 → B 路 AST(已过门限) + var onLegA: ((Data) -> Void)? + var onLegB: ((Data) -> Void)? + /// 门限开启(rms, 是否兜底强开),供上抛给 Dart 记日志——真机调阈值的唯一依据 + var onGateOpened: ((Int, Bool) -> Void)? + /// 背压丢帧时上报一次(累计丢帧数) + var onOverflow: ((Int64) -> Void)? + + var isEnabled: Bool { + lock.lock(); defer { lock.unlock() } + return enabled + } + + func enable() { + lock.lock() + if enabled { lock.unlock(); return } + enabled = true + lock.unlock() + ingress.async { self.resetCountersLocked() } + observer = NotificationCenter.default.addObserver( + forName: Self.uplinkNotification, object: nil, queue: nil + ) { [weak self] note in + guard let self = self, + let leg = note.userInfo?[Self.legKey] as? String, + let pcm = note.userInfo?[Self.pcmKey] as? Data, !pcm.isEmpty else { return } + self.enqueue(leg: leg, pcm: pcm) + } + NSLog("[BesCallPcmBridge] 已开启:上行经通知直达,下行直回发送器") + } + + func disable() { + lock.lock() + if !enabled { lock.unlock(); return } + enabled = false + lock.unlock() + if let o = observer { + NotificationCenter.default.removeObserver(o) + observer = nil + } + ingress.async { + NSLog("[BesCallPcmBridge] 已关闭 A=\(self.framesA) B=\(self.framesB) 丢帧=\(self.dropped) B门限前=\(self.gatedB) 下行=\(self.downlinkFrames) 峰值积压=\(self.maxPendingSeen)") + self.holdA.removeAll(); self.holdB.removeAll() + self.ready = false + } + } + + /// AST 会话(重)建立开始:新会话要重新刻音色,门限重新关上;帧先攒着等 [markReady]。 + /// + /// ⚠️ 只清 B 路的缓存:那些帧是在**旧**门限状态下放行的,门限复位后必须由现场音频 + /// 重新判定「对端开口了没有」,否则重连后照样可能刻到串音。A 路是机主自己的麦, + /// 没有门限,攒下的整段都要补推——这正是「重连那几秒不丢音频」的价值所在。 + func onSessionInit() { + ingress.async { + self.ready = false + self.gate.reset() + self.holdB.removeAll() + } + } + + /// AST 会话销毁:帧继续攒着(重连场景),直到下一次 markReady。 + func onSessionDisposed() { + ingress.async { self.ready = false } + } + + /// A/B 两路 helper 都起来了:把攒下的整段补推出去,之后直通。 + func markReady() { + ingress.async { + let a = self.holdA, b = self.holdB + self.holdA.removeAll(); self.holdB.removeAll() + self.ready = true + if !a.isEmpty || !b.isEmpty { + NSLog("[BesCallPcmBridge] 就绪,补推积攒的 A=\(a.count) B=\(b.count) 帧") + } + for f in a { self.onLegA?(f) } + for f in b { self.onLegB?(f) } + } + } + + /// 由 AST 的 TTS 输出调用:译文 PCM 直接回给 bluetooth_manager 的下行发送器。 + func postDownlink(leg: String, pcm: Data) { + guard !pcm.isEmpty else { return } + NotificationCenter.default.post(name: Self.downlinkNotification, object: nil, + userInfo: [Self.legKey: leg, Self.pcmKey: pcm]) + ingress.async { self.downlinkFrames &+= 1 } + } + + func stats() -> [String: Any] { + var out: [String: Any] = [:] + ingress.sync { + out = [ + "enabled": self.enabled, "ready": self.ready, + "framesA": self.framesA, "framesB": self.framesB, + "dropped": self.dropped, "gatedB": self.gatedB, + "downlinkFrames": self.downlinkFrames, "maxPending": self.maxPendingSeen, + "gateOpen": self.gate.isOpen, "gateFallback": self.gate.openedByFallback, + ] + } + return out + } + + // MARK: - 私有 + + private func resetCountersLocked() { + framesA = 0; framesB = 0; dropped = 0; gatedB = 0; downlinkFrames = 0; maxPendingSeen = 0 + } + + private func enqueue(leg: String, pcm: Data) { + lock.lock() + if pending >= Self.maxPendingFrames { + dropped &+= 1 + let d = dropped + lock.unlock() + if d == 1 || d % 100 == 0 { + NSLog("[BesCallPcmBridge] 上行积压超过 \(Self.maxPendingFrames) 帧,丢帧累计 \(d)") + onOverflow?(d) + } + return + } + pending += 1 + if pending > maxPendingSeen { maxPendingSeen = pending } + lock.unlock() + ingress.async { [self] in + lock.lock(); pending -= 1; lock.unlock() + handle(leg: leg, pcm: pcm) + } + } + + /// 只在 ingress 队列上调用 + private func handle(leg: String, pcm: Data) { + if leg == "A" { + framesA &+= 1 + deliver(a: pcm) + return + } + framesB &+= 1 + let wasOpen = gate.isOpen + let frames = gate.accept(pcm) + if frames.isEmpty { gatedB &+= 1 } + if !wasOpen && gate.isOpen { + NSLog("[BesCallPcmBridge] B 路门限开启 rms=\(gate.openRms) fallback=\(gate.openedByFallback)") + onGateOpened?(gate.openRms, gate.openedByFallback) + } + for f in frames { deliver(b: f) } + } + + private func deliver(a pcm: Data) { + if ready { onLegA?(pcm); return } + holdA.append(pcm) + if holdA.count > Self.holdFrames { holdA.removeFirst() } + } + + private func deliver(b pcm: Data) { + if ready { onLegB?(pcm); return } + holdB.append(pcm) + if holdB.count > Self.holdFrames { holdB.removeFirst() } + } +} diff --git a/apps/client/local_plugins/azure_speech/ios/azure_speech/Sources/azure_speech/PeerSpeechGate.swift b/apps/client/local_plugins/azure_speech/ios/azure_speech/Sources/azure_speech/PeerSpeechGate.swift new file mode 100644 index 00000000..46d9e728 --- /dev/null +++ b/apps/client/local_plugins/azure_speech/ios/azure_speech/Sources/azure_speech/PeerSpeechGate.swift @@ -0,0 +1,79 @@ +import Foundation + +/// 对端(B 路)说话人门限——Dart 版 `lib/data/utils/peer_speech_gate.dart` 的逐条移植。 +/// +/// **在对端真的开口之前,不要把音频推给端到端翻译。** 阿里的实时声音复刻是 +/// `frequency = "once"`,只在会话开头刻一次音色;此时对端往往还没开口,B 路先收到的是 +/// 机主声音的串音,刻错之后用户听到的对端译文用的是自己的声音。 +/// +/// - 未开门期间音频进环形缓冲([prerollFrames]),开门瞬间整段补发——对端先开口也不吃掉开头; +/// - 连续 [openFrames] 帧 RMS ≥ [rmsThreshold] 才开门,断续的串音尖峰不算; +/// - 收满 [fallbackFrames] 仍未开门就**强行开门**(门限太严=整通电话不翻译,比刻错音色严重得多); +/// - 每次会话重建(含断线重连)必须 [reset]。 +/// +/// ⚠️ 参数默认值与 Dart 版、Kotlin 版三处相同,调阈值要三处同改。 +/// 线程:调用方保证串行(BesCallPcmBridge 的 ingress 队列)。 +final class PeerSpeechGate { + let rmsThreshold: Int + let openFrames: Int + let prerollFrames: Int + let fallbackFrames: Int + + private(set) var isOpen = false + private(set) var openRms = 0 + private(set) var openedByFallback = false + private var run = 0 + private var seen = 0 + private var preroll: [Data] = [] + + init(rmsThreshold: Int = 600, openFrames: Int = 8, prerollFrames: Int = 25, fallbackFrames: Int = 500) { + self.rmsThreshold = rmsThreshold + self.openFrames = openFrames + self.prerollFrames = prerollFrames + self.fallbackFrames = fallbackFrames + } + + /// 喂一帧,返回应当推给翻译服务的帧(0 帧、1 帧,或开门瞬间的一整串)。 + func accept(_ frame: Data) -> [Data] { + if isOpen { return [frame] } + seen += 1 + let rms = PeerSpeechGate.rms(ofPcm16: frame) + if rms >= rmsThreshold { run += 1 } else { run = 0 } + preroll.append(frame) + while preroll.count > prerollFrames { preroll.removeFirst() } + let hitThreshold = run >= openFrames + let hitFallback = seen >= fallbackFrames + if !hitThreshold && !hitFallback { return [] } + isOpen = true + openRms = rms + openedByFallback = !hitThreshold + let out = preroll + preroll.removeAll() + return out + } + + func reset() { + isOpen = false + run = 0 + seen = 0 + openRms = 0 + openedByFallback = false + preroll.removeAll() + } + + /// PCM16 小端单声道的 RMS;字节数为奇数时忽略末尾半个样本。 + static func rms(ofPcm16 frame: Data) -> Int { + let n = frame.count / 2 + if n == 0 { return 0 } + var sum: Double = 0 + frame.withUnsafeBytes { (raw: UnsafeRawBufferPointer) in + let base = raw.baseAddress! + for i in 0..) -> Unit)? = null + // ---------- 通话翻译 PCM 原生直连桥(bluetooth_manager 这一端) ---------- + // + // 通话翻译的音频原来是 `本插件 → EventChannel → Dart → MethodChannel → azure_speech`, + // 20ms 一帧、双路,每秒约 200 次跨界;2026-09-20 真机复现「WiFi 下断断续续、长句基本不行」 + // 之后下沉到原生:桥开着时 micData/spkData **不再进 Dart**,直接交给 [callPcmTap] + // (由 azure_speech 的 BesCallPcmBridge 在开桥时赋值,它 compileOnly 依赖本模块)。 + // 其它事件(startCall/stopCall/speedAdjustment…)照旧上抛,Dart 侧状态机不受影响。 + // + // ⚠️ 与 iOS 端 CallPcmBridge.swift 语义逐条对齐(那边因 CocoaPods/SwiftPM 不能互相 import, + // 传输走 NotificationCenter),改一端要同改另一端。 + @Volatile + var callPcmBridgeEnabled: Boolean = false + + /** (leg, pcm):leg "A" = 本端麦(micData);"B" = 对端通话音(spkData)。在解码线程上调用。 */ + @Volatile + var callPcmTap: ((String, ByteArray) -> Unit)? = null + + @Volatile + private var callPcmBridgedFrames: Long = 0L + + fun callPcmBridgeStats(): Map = mapOf( + "enabled" to callPcmBridgeEnabled, + "uplinkFrames" to callPcmBridgedFrames, + ) + + // ---------- EMAI 会话 PCM 原生直连桥(bluetooth_manager 这一端) ---------- + // + // 耳机按键唤醒的 AI 会话,音频原来是 + // `本插件 → EventChannel → Dart → 百炼 WebSocket(Dart) → Dart → MethodChannel → 本插件`, + // 上行 20ms 一帧(50 次/s)、下行每个 WS 音频块一次,全压在 Dart 的单 isolate 上。 + // 桥开着时 micData 直接交给 [aiPcmTap](azure_speech 的 BesAiPcmBridge 开桥时赋值), + // 合成回来的 PCM 由它直接喂 [AiAudioDownlink],**Dart 只收文本与状态事件**。 + // startAI / stopAI / 调速这些控制事件照旧上抛——会话起停仍由 Dart 编排。 + // + // ⚠️ 与 iOS 端 AiPcmBridge.swift 语义逐条对齐,改一端要同改另一端。 + // ⚠️ 刻意与通话翻译那条桥分开:那条是已冻结的链路,leg A/B 有明确语义(本端/对端), + // 混用会让 leg 多出一个不属于它的取值,排查时分不清一帧属于哪条链路。 + @Volatile + var aiPcmBridgeEnabled: Boolean = false + + /** AI 模式下耳机麦的一帧 PCM(16k/16bit/mono)。在解码线程上调用。 */ + @Volatile + var aiPcmTap: ((ByteArray) -> Unit)? = null + + @Volatile + private var aiPcmBridgedFrames: Long = 0L + + fun aiPcmBridgeStats(): Map = mapOf( + "enabled" to aiPcmBridgeEnabled, + "uplinkFrames" to aiPcmBridgedFrames, + ) + + /** 开/关 EMAI PCM 原生直连桥;关掉即恢复原来的 EventChannel 路径。 */ + fun setAiPcmBridge(enabled: Boolean) { + if (aiPcmBridgeEnabled == enabled) return + aiPcmBridgeEnabled = enabled + if (enabled) aiPcmBridgedFrames = 0L + Log.d("AiPcmBridge", if (enabled) "AI 原生直连桥已开启" else "AI 原生直连桥已关闭 上行=$aiPcmBridgedFrames 帧") + } + /** GATT/RFCOMM 已连接(与 deviceStatus 2 对齐)。 */ val isConnected: Boolean get() = deviceStatus == 2 @@ -582,6 +642,10 @@ object BluetoothManager : PluginRegistry.RequestPermissionsResultListener { if (status == 0) { // 链路断开立即通知 AI 会话停止。此路径可能从任意线程、重复触发,接收方须幂等 aiCallback?.invoke(mapOf("type" to "deviceDisconnected")) + // 设备断了就不会再有帧:把还开着的录音收尾并回填 WAV 头。Dart 侧的服务也会 + // 收到断连并调 recorderStop,那次拿到 null(已注销),调用方用最后一次统计 + // 里的字节数兜底。不做这一步、进程随后被杀,就留下一个 data 长度为 0 的 WAV。 + PcmRecorderRegistry.stopAll(true) } val handler = mainHandler if (handler != null) { @@ -596,9 +660,33 @@ object BluetoothManager : PluginRegistry.RequestPermissionsResultListener { if (type == "stopCall" || type == "stopMicRecording" || type == "stopFaceToFace") { releaseCodecs() } + // 录音是**额外的 sink**,放在桥的判断之前:通话翻译开着桥时 micData/spkData + // 不再上抛 Dart,但用户可能同时在录这通电话,放在 return 之后会录出空文件。 + if (!PcmRecorderRegistry.isEmpty && type is String) { + (data["data"] as? ByteArray)?.let { PcmRecorderRegistry.feed(type, it) } + } + // 原生直连桥:双路 PCM 直接交给 azure_speech,不再经过 Dart(见 callPcmTap 的说明) + if (callPcmBridgeEnabled && (type == "micData" || type == "spkData")) { + val tap = callPcmTap + val pcm = data["data"] as? ByteArray + if (tap != null && pcm != null) { + tap.invoke(if (type == "micData") "A" else "B", pcm) + val n = ++callPcmBridgedFrames + if (n == 1L || n % 1000L == 0L) Log.d("CallPcmBridge", "上行已直连转发 $n 帧") + return + } + } translationCallback?.invoke(data) } + /** 开/关通话翻译 PCM 原生直连桥;关掉即恢复原来的 EventChannel 路径。 */ + fun setCallPcmBridge(enabled: Boolean) { + if (callPcmBridgeEnabled == enabled) return + callPcmBridgeEnabled = enabled + if (enabled) callPcmBridgedFrames = 0L + Log.d("CallPcmBridge", if (enabled) "原生直连桥已开启" else "原生直连桥已关闭 上行=$callPcmBridgedFrames 帧") + } + private fun publishAIData(data: Map) { // 与 publishTranslationData 对称:AI 会话结束时释放共享的 g722/msbc 解码 handle, // 让下一次会话(AI 或翻译)以干净的解码器状态开始,避免复用脏状态导致开头解码瑕疵。 @@ -608,6 +696,17 @@ object BluetoothManager : PluginRegistry.RequestPermissionsResultListener { if (data["type"] == "stopAI") { releaseCodecs() } + // AI 原生直连桥:唤醒后的耳机麦 PCM 直接交给 azure_speech 的百炼客户端,不经过 Dart + if (aiPcmBridgeEnabled && data["type"] == "micData") { + val tap = aiPcmTap + val pcm = data["data"] as? ByteArray + if (tap != null && pcm != null) { + tap.invoke(pcm) + val n = ++aiPcmBridgedFrames + if (n == 1L || n % 1000L == 0L) Log.d("AiPcmBridge", "上行已直连转发 $n 帧") + return + } + } aiCallback?.invoke(data) } diff --git a/apps/client/local_plugins/bluetooth_manager/android/src/main/kotlin/com/example/bluetooth_manager/BluetoothManagerPlugin.kt b/apps/client/local_plugins/bluetooth_manager/android/src/main/kotlin/com/example/bluetooth_manager/BluetoothManagerPlugin.kt index ecca6c00..562b1392 100644 --- a/apps/client/local_plugins/bluetooth_manager/android/src/main/kotlin/com/example/bluetooth_manager/BluetoothManagerPlugin.kt +++ b/apps/client/local_plugins/bluetooth_manager/android/src/main/kotlin/com/example/bluetooth_manager/BluetoothManagerPlugin.kt @@ -38,6 +38,11 @@ class BluetoothManagerPlugin : FlutterPlugin, MethodCallHandler, ActivityAware { private lateinit var bluetoothAiChannel: EventChannel private var bluetoothAiChannelSink: EventSink? = null + // 录音统计通道:原生每 100ms 报一次 {id, bytes, level}。PCM 本身不再过 Dart, + // 时长由 bytes 算(与文件实际长度永远一致),电平驱动波形。见 PcmWavRecorder.kt。 + private lateinit var bluetoothRecorderChannel: EventChannel + private var bluetoothRecorderChannelSink: EventSink? = null + override fun onAttachedToEngine(flutterPluginBinding: FlutterPlugin.FlutterPluginBinding) { channel = MethodChannel(flutterPluginBinding.binaryMessenger, "bluetooth_manager") bluetoothReadChannel = @@ -50,6 +55,17 @@ class BluetoothManagerPlugin : FlutterPlugin, MethodCallHandler, ActivityAware { EventChannel(flutterPluginBinding.binaryMessenger, "bluetooth_manager/audio") bluetoothAiChannel = EventChannel(flutterPluginBinding.binaryMessenger, "bluetooth_manager/ai") + bluetoothRecorderChannel = + EventChannel(flutterPluginBinding.binaryMessenger, "bluetooth_manager/recorder") + bluetoothRecorderChannel.setStreamHandler(object : StreamHandler { + override fun onCancel(arguments: Any?) { + bluetoothRecorderChannelSink = null + } + + override fun onListen(arguments: Any?, events: EventSink?) { + bluetoothRecorderChannelSink = events + } + }) channel.setMethodCallHandler(this) BluetoothManager.setup(flutterPluginBinding.applicationContext) bluetoothManagerListen() @@ -123,6 +139,13 @@ class BluetoothManagerPlugin : FlutterPlugin, MethodCallHandler, ActivityAware { CallTranslationDownlink.start() result.success(true) } + // 通话翻译 PCM 原生直连桥(见 BluetoothManager.callPcmTap)。 + "setCallPcmBridge" -> { + val enabled = call.argument("enabled") ?: false + BluetoothManager.setCallPcmBridge(enabled) + result.success(true) + } + "getCallPcmBridgeStats" -> result.success(BluetoothManager.callPcmBridgeStats()) "stopCallDownlink" -> { CallTranslationDownlink.stop() result.success(true) @@ -154,11 +177,62 @@ class BluetoothManagerPlugin : FlutterPlugin, MethodCallHandler, ActivityAware { AiAudioDownlink.flush() result.success(true) } + "setAiPcmBridge" -> { + val enabled = call.argument("enabled") ?: false + BluetoothManager.setAiPcmBridge(enabled) + result.success(true) + } + "getAiPcmBridgeStats" -> { + result.success(BluetoothManager.aiPcmBridgeStats()) + } "getConnectedDevice" -> { // 自动回连(ACL 广播触发的 reconnectWithRetry)只 publish status, // 不会走 connectToDevice,Dart 侧需要靠这个回退查询补齐 id/name。 result.success(BluetoothManager.getConnectedDeviceInfo()) } + // ===== 设备音频直接落盘(见 PcmWavRecorder.kt)===== + // 帧在原生解码完就地写 WAV,不再每帧过一次平台通道给 Dart 写盘。 + "recorderStart" -> { + val id = call.argument("id") + val path = call.argument("path") + if (id == null || path == null) { + result.success(false) + } else { + val channels = call.argument("channels") ?: 1 + val sampleRate = call.argument("sampleRate") ?: 16000 + val rec = PcmWavRecorder(id, path, channels, sampleRate) + rec.onStats = { rid, bytes, level -> + mainHandler.post { + bluetoothRecorderChannelSink?.success( + mapOf("id" to rid, "bytes" to bytes, "level" to level) + ) + } + } + if (!rec.start()) { + result.success(false) + } else { + if (channels >= 2) { + PcmRecorderRegistry.register( + rec, + null, + call.argument("left") ?: "micData", + call.argument("right") ?: "spkData", + ) + } else { + PcmRecorderRegistry.register( + rec, call.argument("source") ?: "mainMicData", null, null + ) + } + result.success(true) + } + } + } + "recorderStop" -> { + val id = call.argument("id") + val save = call.argument("save") ?: true + val rec = if (id == null) null else PcmRecorderRegistry.unregister(id) + result.success(rec?.stop(save)) + } else -> result.notImplemented() } } @@ -243,6 +317,13 @@ class BluetoothManagerPlugin : FlutterPlugin, MethodCallHandler, ActivityAware { bluetoothAudioChannel.setStreamHandler(null) CallTranslationDownlink.stop() AiAudioDownlink.stop() + BluetoothManager.setCallPcmBridge(false) + BluetoothManager.setAiPcmBridge(false) + // 引擎都拆了就不会再有帧:把还开着的录音收尾并回填 WAV 头, + // 否则留下的是一个 data 长度为 0 的文件——有几十 MB,播放器却认为它是空的 + PcmRecorderRegistry.stopAll(true) + bluetoothRecorderChannelSink = null + bluetoothRecorderChannel.setStreamHandler(null) bluetoothAiChannelSink = null bluetoothAiChannel.setStreamHandler(null) BluetoothManager.translationCallback = null diff --git a/apps/client/local_plugins/bluetooth_manager/android/src/main/kotlin/com/example/bluetooth_manager/PcmWavRecorder.kt b/apps/client/local_plugins/bluetooth_manager/android/src/main/kotlin/com/example/bluetooth_manager/PcmWavRecorder.kt new file mode 100644 index 00000000..c45b0f76 --- /dev/null +++ b/apps/client/local_plugins/bluetooth_manager/android/src/main/kotlin/com/example/bluetooth_manager/PcmWavRecorder.kt @@ -0,0 +1,335 @@ +package com.example.bluetooth_manager + +import android.util.Log +import java.io.File +import java.io.RandomAccessFile +import java.util.concurrent.ConcurrentHashMap +import java.util.concurrent.Executors +import kotlin.math.min +import kotlin.math.sqrt + +/** + * 设备音频直接落盘的 WAV 录制器(16bit 小端,单声道或双声道交织)。 + * + * **为什么在原生**:此前是每帧 PCM 经 EventChannel 到 Dart、由 `WavWriter` 用 + * `RandomAccessFile` 排队写盘。恒玄 20ms 一帧 = 单路 50 帧/s、通话双路 100 帧/s, + * 每帧都要跨一次平台通道并在 Dart 的单 isolate 上排队——event loop 一被页面布局 + * 或 GC 占住,写入队列就堆积。`WavWriter` 的注释里记着这条链路出过的事故: + * 写入重叠导致丢帧,**等效采样率只剩 1587Hz**。现在帧在原生解码完就地写文件。 + * + * 与 iOS 的 `PcmWavRecorder.swift` 逐条对齐,改一边必须同改另一边。 + */ +class PcmWavRecorder( + val id: String, + private val path: String, + channels: Int, + private val sampleRate: Int, +) { + companion object { + private const val TAG = "PcmWavRecorder" + + /** 统计上报间隔,与 Dart 侧波形采样同频 */ + private const val STATS_INTERVAL_MS = 100L + + /** 一路长期缺席时另一路的最大堆积:10 秒 @16k/16bit */ + private const val MAX_PENDING = 16000 * 2 * 10 + + /** + * 与 Dart `WavWriter.rmsLevel` 同口径:归一化 RMS 再开一次根 + * (语音 RMS 常年在 0.02~0.3,不拉一下动态范围界面几乎没动静)。 + */ + fun rmsLevel(pcm: ByteArray, length: Int = pcm.size): Double { + val count = length / 2 + if (count == 0) return 0.0 + var sum = 0.0 + for (i in 0 until count) { + // 高字节 toInt() 本身就带符号扩展,`shl 8` 再 or 上无符号低字节, + // 结果直接落在 -32768..32767,不需要再减 0x10000(Dart 那边因为 + // Uint8List 的元素是无符号的,才要手工补那一步) + val v = (pcm[i * 2 + 1].toInt() shl 8) or (pcm[i * 2].toInt() and 0xFF) + val n = v / 32768.0 + sum += n * n + } + val rms = sqrt(sum / count) + return min(1.0, sqrt(rms)) + } + + /** 两路单声道按样本交织成双声道(左在前) */ + fun interleave(left: ByteArray, leftLen: Int, right: ByteArray, rightLen: Int): ByteArray { + val samples = min(leftLen, rightLen) / 2 + val out = ByteArray(samples * 4) + for (i in 0 until samples) { + val si = i * 2 + val di = i * 4 + out[di] = left[si] + out[di + 1] = left[si + 1] + out[di + 2] = right[si] + out[di + 3] = right[si + 1] + } + return out + } + + /** 44 字节标准 WAV 头 */ + fun wavHeader(dataLength: Int, channels: Int, sampleRate: Int): ByteArray { + val bits = 16 + val byteRate = sampleRate * channels * bits / 8 + val blockAlign = channels * bits / 8 + val h = ByteArray(44) + var p = 0 + fun ascii(s: String) { for (c in s) h[p++] = c.code.toByte() } + fun u32(v: Int) { + h[p++] = (v and 0xFF).toByte() + h[p++] = ((v shr 8) and 0xFF).toByte() + h[p++] = ((v shr 16) and 0xFF).toByte() + h[p++] = ((v shr 24) and 0xFF).toByte() + } + fun u16(v: Int) { + h[p++] = (v and 0xFF).toByte() + h[p++] = ((v shr 8) and 0xFF).toByte() + } + ascii("RIFF"); u32(36 + dataLength); ascii("WAVE") + ascii("fmt "); u32(16); u16(1); u16(channels) + u32(sampleRate); u32(byteRate); u16(blockAlign); u16(bits) + ascii("data"); u32(dataLength) + return h + } + } + + val channels: Int = channels.coerceIn(1, 2) + + /** 串行写线程。文件句柄只在这上面碰。 */ + private val exec = Executors.newSingleThreadExecutor { r -> Thread(r, "bes-pcmwav-$id") } + + private var raf: RandomAccessFile? = null + + @Volatile var dataBytes: Int = 0; private set + @Volatile var writtenFrames: Int = 0; private set + @Volatile var droppedFrames: Int = 0; private set + @Volatile private var closed = false + + /** 双声道的左右暂存(见 flushPairs 的移植说明) */ + private var leftBuf = ByteArray(0) + private var rightBuf = ByteArray(0) + + private var levelPeak = 0.0 + private var lastStatsAt = 0L + + /** 统计回调(在写线程上调用,接收方自己派发到主线程) */ + var onStats: ((String, Int, Double) -> Unit)? = null + + // ---------- 生命周期 ---------- + + /** 建文件并写占位头。返回 false 表示建不出来(目录不可写等)。 */ + fun start(): Boolean { + return try { + val f = File(path) + f.parentFile?.mkdirs() + val r = RandomAccessFile(f, "rw") + r.setLength(0) + r.write(wavHeader(0, channels, sampleRate)) + raf = r + dataBytes = 0 + writtenFrames = 0 + droppedFrames = 0 + lastStatsAt = System.currentTimeMillis() + true + } catch (e: Exception) { + Log.e(TAG, "建文件失败: ${e.message}") + false + } + } + + /** + * 停止。[save] = false 时删除文件(没收到任何音频那种)。 + * 会等写队列排空再回填 WAV 头,否则最后几帧会丢、头里的长度也对不上。 + */ + fun stop(save: Boolean): Map { + if (closed) { + return mapOf("bytes" to dataBytes, "written" to writtenFrames, "dropped" to droppedFrames) + } + closed = true + // 排空:submit 一个空任务并等它跑完,前面排队的写入就都落盘了 + try { + exec.submit { }.get() + } catch (_: Exception) { + } + try { + val r = raf + raf = null + if (r != null) { + if (save) { + try { + r.seek(0) + r.write(wavHeader(dataBytes, channels, sampleRate)) + } catch (e: Exception) { + Log.e(TAG, "回填 WAV 头失败: ${e.message}") + } + } + r.close() + } + if (!save) File(path).delete() + } catch (e: Exception) { + Log.e(TAG, "关闭失败: ${e.message}") + } finally { + exec.shutdown() + } + return mapOf("bytes" to dataBytes, "written" to writtenFrames, "dropped" to droppedFrames) + } + + // ---------- 进料 ---------- + + /** 单声道:直接落盘 */ + fun append(pcm: ByteArray) { + if (pcm.isEmpty() || closed) return + submit { + trackLevel(pcm, pcm.size) + write(pcm) + emitStatsIfDue() + } + } + + /** + * stop() 里 shutdown 之后仍可能有在途帧进来(解码线程与调用方不同线程), + * 那时 execute 会抛 RejectedExecutionException。这是正常收尾,不是错误。 + */ + private fun submit(task: () -> Unit) { + try { + exec.execute { + if (!closed) task() + } + } catch (_: Exception) { + } + } + + /** 双声道左路(本端麦) */ + fun appendLeft(pcm: ByteArray) = appendStereo(pcm, true) + + /** 双声道右路(对端) */ + fun appendRight(pcm: ByteArray) = appendStereo(pcm, false) + + private fun appendStereo(pcm: ByteArray, isLeft: Boolean) { + if (pcm.isEmpty() || closed) return + submit { + if (isLeft) { + leftBuf = pushCapped(leftBuf, pcm) + } else { + rightBuf = pushCapped(rightBuf, pcm) + } + flushPairs() + emitStatsIfDue() + } + } + + private fun pushCapped(buf: ByteArray, add: ByteArray): ByteArray { + val merged = buf + add + return if (merged.size > MAX_PENDING) merged.copyOfRange(merged.size - MAX_PENDING, merged.size) + else merged + } + + /** + * StereoPairer 的移植:两路都排队,谁都不丢,每次取出两边都已有的那一段 + * (长度取较短者、对齐到整样本)。 + * + * ⚠️ 不能写成「只留最后一帧对端、本端来一帧配一帧」:两路在 BLE 上是同一个帧里 + * 交替的 40 字节块,但到达这里已经是两次独立回调,顺序不保证严格交替。按 + * 「最后一帧」配对会把右声道断续抽掉,转写出来只剩自己这一方。Dart 侧的金测试 + * `test/stereo_pairer_test.dart` 定的就是这个行为,三处(Dart/Swift/Kotlin)同改。 + */ + private fun flushPairs() { + val n = min(leftBuf.size, rightBuf.size) and 1.inv() + if (n <= 0) return + val l = leftBuf.copyOfRange(0, n) + val r = rightBuf.copyOfRange(0, n) + leftBuf = leftBuf.copyOfRange(n, leftBuf.size) + rightBuf = rightBuf.copyOfRange(n, rightBuf.size) + trackLevel(l, n) // 电平按本端(左声道)算,与 Dart 原行为一致 + write(interleave(l, n, r, n)) + } + + private fun write(data: ByteArray) { + val r = raf ?: return + try { + r.write(data) + dataBytes += data.size + writtenFrames++ + } catch (e: Exception) { + droppedFrames++ + Log.e(TAG, "写入失败(已丢 $droppedFrames): ${e.message}") + } + } + + // ---------- 电平与统计 ---------- + + private fun trackLevel(pcm: ByteArray, length: Int) { + val v = rmsLevel(pcm, length) + if (v > levelPeak) levelPeak = v + } + + private fun emitStatsIfDue() { + val now = System.currentTimeMillis() + if (now - lastStatsAt < STATS_INTERVAL_MS) return + lastStatsAt = now + val peak = levelPeak + levelPeak = 0.0 + onStats?.invoke(id, dataBytes, peak) + } +} + +/** + * 活跃录制器登记处。 + * + * `BluetoothManager` 解码完一帧就按类型喂给这里的录制器——**只是多挂一个 sink, + * 不拦截原有的 EventChannel 回调**。这点和原生直连桥(callPcmBridgeEnabled)不同: + * 那条是「桥开着就不再上抛」,因为通话翻译的 PCM 有唯一消费者;录音则可能与别的 + * 订阅方(比如正在跑的通话翻译)并存,拦了会把对方饿死。 + */ +object PcmRecorderRegistry { + private val byId = ConcurrentHashMap() + + /** + * 音频类型 → 录制器。类型串与 `BluetoothManager` 里发的一致: + * `mainMicData`(现场录音主麦)/ `micData`(通话本端)/ `spkData`(通话对端)。 + * value 的 Boolean:null=单声道,true=左路,false=右路。 + */ + private val sinks = ConcurrentHashMap>>() + + val isEmpty: Boolean get() = byId.isEmpty() + + fun register(rec: PcmWavRecorder, monoSource: String?, left: String?, right: String?) { + byId[rec.id] = rec + monoSource?.let { sinks.getOrPut(it) { mutableListOf() }.add(rec to null) } + left?.let { sinks.getOrPut(it) { mutableListOf() }.add(rec to true) } + right?.let { sinks.getOrPut(it) { mutableListOf() }.add(rec to false) } + } + + fun unregister(id: String): PcmWavRecorder? { + val rec = byId.remove(id) ?: return null + val empties = mutableListOf() + for ((k, v) in sinks) { + v.removeAll { it.first === rec } + if (v.isEmpty()) empties.add(k) + } + for (k in empties) sinks.remove(k) + return rec + } + + /** 由 `BluetoothManager` 在解码线程上调用。没有录制器时立刻返回。 */ + fun feed(type: String, pcm: ByteArray) { + val list = sinks[type] ?: return + for ((rec, side) in list) { + when (side) { + null -> rec.append(pcm) + true -> rec.appendLeft(pcm) + false -> rec.appendRight(pcm) + } + } + } + + /** 断连/停录时把还开着的全停掉,避免留下没回填头的半截文件 */ + fun stopAll(save: Boolean) { + val all = byId.values.toList() + byId.clear() + sinks.clear() + for (r in all) r.stop(save) + } +} diff --git a/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/AiAudioDownlink.swift b/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/AiAudioDownlink.swift index 4049bfef..e9d92906 100644 --- a/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/AiAudioDownlink.swift +++ b/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/AiAudioDownlink.swift @@ -174,7 +174,7 @@ public final class AiAudioDownlink { // 整包一帧真实数据都没有就不发 guard hasData else { return } - BluetoothManager.writeBytes(Data(packet)) + BluetoothManager.writeRealtimeAudio(Data(packet)) sentPackets += 1 if sentPackets == 1 || sentPackets % 100 == 0 { NSLog("[BesAiDownlink] AI 已下发 \(sentPackets) 包 队列=\(outQueue.count) 间隔=\(intervalMs)ms") diff --git a/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/AiPcmBridge.swift b/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/AiPcmBridge.swift new file mode 100644 index 00000000..2709b4ac --- /dev/null +++ b/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/AiPcmBridge.swift @@ -0,0 +1,94 @@ +import Foundation + +/// EMAI 会话 PCM 的**原生直连桥**(bluetooth_manager 这一端)。 +/// +/// ## 为什么有这个 +/// 耳机按键唤醒的 AI 会话,音频原来是 +/// `本插件 → EventChannel → Dart → 百炼 WebSocket(Dart)→ Dart → MethodChannel → 本插件`。 +/// 上行 20ms 一帧(50 次/s),下行每个 WS 音频块一次,全都压在 Dart 的单 isolate 上, +/// 和页面布局、GC 排同一个 event loop。桥开着时上行 PCM 直接交给 azure_speech 里的 +/// 原生百炼客户端,合成回来的 PCM 也直接回到 [AiAudioDownlink],**Dart 只收文本与状态事件**。 +/// +/// ## 与 [CallPcmBridge] 的关系 +/// 结构逐条对称,但**刻意分成两个文件、两组通知名**:通话翻译那条是已量产、已冻结的 +/// 链路,它的 leg A/B 有明确语义(本端/对端);AI 只有一路。混用会让通话的 leg 语义 +/// 多出一个不属于它的取值,排查时分不清一帧到底属于哪条链路。 +/// +/// ## ⚠️ 传输为什么是 NotificationCenter +/// 本插件是 CocoaPods 包、azure_speech 是 SwiftPM 包,两边**不能互相 import** +/// (Android 那边是 Gradle 子工程,可以直接依赖,走的是类型化的静态槽)。 +/// NotificationCenter 的 post 是**同步、在发帖线程上**执行的,没有序列化、没有跨线程。 +/// +/// ## ⚠️ 两边手工同步的契约 +/// 通知名与 userInfo 键在 azure_speech 的 `BesAiPcmBridge.swift` 里各写了一份, +/// **改一处必须同改另一处**,改漏了不会编译报错,只会表现为「桥开着但一帧都到不了」。 +/// 契约: +/// - 上行 `com.eaimar.bes.aipcm.uplink` userInfo: `pcm`(Data, 16k/16bit/mono) +/// - 下行 `com.eaimar.bes.aipcm.downlink` userInfo: `pcm`(Data, 16k/16bit/mono) +public enum AiPcmBridge { + public static let uplinkNotification = Notification.Name("com.eaimar.bes.aipcm.uplink") + public static let downlinkNotification = Notification.Name("com.eaimar.bes.aipcm.downlink") + public static let pcmKey = "pcm" + + private static let lock = NSLock() + private static var enabled = false + private static var downlinkObserver: NSObjectProtocol? + private static var uplinkFrames: Int64 = 0 + private static var downlinkFrames: Int64 = 0 + + /// 桥是否开着。解码线程每帧都读它。 + public static var isEnabled: Bool { + lock.lock(); defer { lock.unlock() } + return enabled + } + + /// 开桥:之后 AI 模式的 micData 只走通知,不再进 Dart 的 ai EventChannel; + /// 同时开始收 azure_speech 发来的合成 PCM 喂给下行发送器。 + public static func enable() { + lock.lock() + if enabled { lock.unlock(); return } + enabled = true + uplinkFrames = 0 + downlinkFrames = 0 + lock.unlock() + downlinkObserver = NotificationCenter.default.addObserver( + forName: downlinkNotification, object: nil, queue: nil + ) { note in + guard let pcm = note.userInfo?[pcmKey] as? Data, !pcm.isEmpty else { return } + AiAudioDownlink.shared.pushPcm(pcm) + lock.lock(); downlinkFrames &+= 1; lock.unlock() + } + NSLog("[AiPcmBridge] AI 原生直连桥已开启(上行走通知,下行直喂发送器)") + } + + public static func disable() { + lock.lock() + if !enabled { lock.unlock(); return } + enabled = false + let up = uplinkFrames, down = downlinkFrames + lock.unlock() + if let o = downlinkObserver { + NotificationCenter.default.removeObserver(o) + downlinkObserver = nil + } + NSLog("[AiPcmBridge] AI 原生直连桥已关闭 上行=\(up) 帧 下行=\(down) 帧") + } + + /// 由 BluetoothManager 的解码线程调用:把一帧上行 PCM 交给 azure_speech。 + static func postUplink(pcm: Data) { + NotificationCenter.default.post(name: uplinkNotification, object: nil, + userInfo: [pcmKey: pcm]) + lock.lock() + uplinkFrames &+= 1 + let n = uplinkFrames + lock.unlock() + if n == 1 || n % 1000 == 0 { + NSLog("[AiPcmBridge] 上行已直连转发 \(n) 帧") + } + } + + public static func stats() -> [String: Any] { + lock.lock(); defer { lock.unlock() } + return ["enabled": enabled, "uplinkFrames": uplinkFrames, "downlinkFrames": downlinkFrames] + } +} diff --git a/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/BluetoothManager.swift b/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/BluetoothManager.swift index 1506b6c7..2464f58d 100644 --- a/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/BluetoothManager.swift +++ b/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/BluetoothManager.swift @@ -1,5 +1,6 @@ import Foundation import CoreBluetooth +import os.log public final class BluetoothManager: NSObject { @@ -19,6 +20,12 @@ public final class BluetoothManager: NSObject { sharedInstance?.write(data) } + /// 实时音频专用写入口(通话翻译下行)。与 [writeBytes] 的差别只在统计口径, + /// 两者现在都是无条件直发 —— 见 enqueueOrSendNoResp 里那段死锁说明。 + public static func writeRealtimeAudio(_ data: Data) { + sharedInstance?.write(data, realtime: true) + } + // MARK: - 公共回调 public var onStateChanged: ((CBManagerState) -> Void)? @@ -46,6 +53,25 @@ public final class BluetoothManager: NSObject { // 发送排队,等待 GATT 就绪 private var pendingWrites: [Data] = [] private let maxPendingWrites = 128 + + // MARK: - 无应答写的统计(下行音频是 20ms 一包、50 包/秒的实时流) + // + // ⚠️ 这条链路是 **GATT over BR/EDR**(经典蓝牙),不是 BLE —— 见 connect(deviceId:) + // 与 registerForConnectionEvents。所以别套 BLE 的 connection interval / 带宽模型: + // 84B × 50 包/秒 对 ACL 链路毫无压力(实测 10164 包 dropped=0)。 + // + // 这里**不做** canSendWriteWithoutResponse 背压。那个属性是 CoreBluetooth 为 BLE + // 设计的提示,在 BR/EDR GATT 上不反映真实能力(实测 100% 为 false 却零丢弃), + // 拿它当门槛会死锁。详见 enqueueOrSendNoResp 的文档注释。 + // flowQueue 只留作遗留积压的出口(正常恒为空),真正的节奏由 + // CallTranslationDownlink 的 20ms 定时器控制。 + private var flowQueue: [Data] = [] + private let maxFlowQueue = 64 + private var bleSentWrites: Int64 = 0 + private var bleDroppedWrites: Int64 = 0 + private var bleFlowBlocked: Int64 = 0 + private var writeTypeLogged = false + private static let bleLog = OSLog(subsystem: "com.eaimar.bluetooth", category: "write") private var isReadyNotified = false // App 内已配对授权的设备 UUID 白名单;只有命中的设备才会通过系统 BR/EDR 连接事件自动接管 @@ -188,7 +214,7 @@ public final class BluetoothManager: NSObject { } } - public func write(_ data: Data) { + public func write(_ data: Data, realtime: Bool = false) { runOnBleQueue { [weak self] in guard let self, let p = self.peripheral else { return } guard let tx = self.txCharacteristic, self.isGattReady() else { @@ -197,7 +223,7 @@ public final class BluetoothManager: NSObject { } return } - self.writeToPeripheral(p, characteristic: tx, data: data) + self.writeToPeripheral(p, characteristic: tx, data: data, realtime: realtime) } } @@ -223,19 +249,135 @@ public final class BluetoothManager: NSObject { // MARK: - 内部写入(在 bleQueue 上调用) - private func writeToPeripheral(_ p: CBPeripheral, characteristic tx: CBCharacteristic, data: Data) { - guard tx.properties.contains(.write) || tx.properties.contains(.writeWithoutResponse) else { return } - let writeType: CBCharacteristicWriteType = tx.properties.contains(.write) ? .withResponse : .withoutResponse + /// ⚠️ 写类型的优先级不能反过来。 + /// + /// 特征通常同时具备 `.write` 与 `.writeWithoutResponse`,而原实现是 + /// `contains(.write) ? .withResponse : .withoutResponse` —— 只要支持带应答就用带应答。 + /// **ATT 是请求-响应协议**(与底层是 BLE 还是 BR/EDR 无关),带应答写每包都要等对端 + /// ACK 回来才能发下一包;通话翻译下行是 20ms 一包、50 包/秒,必然追不上, + /// 表现就是音频越听越滞后、断断续续。实时流必须优先无应答写。 + /// + /// 协议内容与 20ms 的发送节奏都没有变,改的只是链路层怎么把这 84 字节递出去。 + private func writeToPeripheral(_ p: CBPeripheral, characteristic tx: CBCharacteristic, + data: Data, realtime: Bool = false) { + let canNoResp = tx.properties.contains(.writeWithoutResponse) + let canResp = tx.properties.contains(.write) + guard canNoResp || canResp else { return } + let writeType: CBCharacteristicWriteType = canNoResp ? .withoutResponse : .withResponse let maxLen = p.maximumWriteValueLength(for: writeType) guard maxLen > 0 else { return } + if !writeTypeLogged { + writeTypeLogged = true + os_log("[GattWrite] 写类型=%{public}@ maxLen=%d (props: write=%{public}@ writeNoResp=%{public}@)", + log: Self.bleLog, type: .info, + canNoResp ? "withoutResponse" : "withResponse", maxLen, + canResp ? "Y" : "N", canNoResp ? "Y" : "N") + } + // 分包发送 var offset = 0 while offset < data.count { let end = min(offset + maxLen, data.count) - p.writeValue(data[offset..0 = 链路带宽不够;blocked 高而 dropped=0 = 有背压但还扛得住。 + /// 通话结束时清掉积压的实时音频。 + /// + /// ⚠️ 这一步不能省:`CallTranslationDownlink.stop()` 只清它自己那两条腿的队列, + /// 清不到这里。残留的几十包过期音频会一直排在控制命令前面, + /// 于是通话之后查电量/查版本全都发不出去。 + public func flushRealtimeQueue() { + runOnBleQueue { [weak self] in + guard let self else { return } + if !self.flowQueue.isEmpty { + os_log("[GattWrite] 通话结束,丢弃积压的 %d 包实时音频", log: Self.bleLog, type: .info, + self.flowQueue.count) + self.flowQueue.removeAll() + } + } + } + + public func bleWriteStats() -> [String: Any] { + var out: [String: Any] = [:] + let work = { + out = [ + "sent": self.bleSentWrites, + "dropped": self.bleDroppedWrites, + "blocked": self.bleFlowBlocked, + "queued": self.flowQueue.count, + ] } + if DispatchQueue.getSpecific(key: bleQueueKey) == true { work() } else { bleQueue.sync(execute: work) } + return out } // MARK: - 队列调度辅助 @@ -502,16 +644,40 @@ public final class BluetoothManager: NSObject { } /// 在处理队列上直接调用(不派发到主队列),镜像 Android 实现 + /// + /// 原生直连桥开着时,双路 PCM(micData=本端麦→A 路,spkData=对端→B 路)**不再进 Dart**, + /// 直接交给 azure_speech(见 CallPcmBridge)。其它事件(startCall/stopCall/speedAdjustment…) + /// 照旧上抛,Dart 侧的状态机不受影响。 private func publishCallTranslation(_ event: Dictionary) { + let type = event["type"] as? String + // 录音是**额外的 sink**,放在桥的判断之前:通话翻译开着桥时 + // micData/spkData 不再上抛 Dart,但用户可能同时在录这通电话, + // 放在 return 之后就会录出空文件。 + if let type = type, let pcm = event["data"] as? Data, !PcmRecorderRegistry.isEmpty { + PcmRecorderRegistry.feed(type: type, pcm: pcm) + } + if let type = type, type == "micData" || type == "spkData", + CallPcmBridge.isEnabled, let pcm = event["data"] as? Data { + CallPcmBridge.postUplink(leg: type == "micData" ? "A" : "B", pcm: pcm) + return + } BluetoothManager.translationCallback?(event) } /// 在处理队列上直接调用(镜像 publishCallTranslation)。 /// stopAI 先释放 g722 handle(与解码同队列串行,释放安全),再上抛给 agent_manager。 private func publishAIData(_ event: Dictionary) { - if event["type"] as? String == "stopAI" { + let type = event["type"] as? String + if type == "stopAI" { releaseCodecs() } + // AI 原生直连桥:唤醒后的耳机麦 PCM 直接交给 azure_speech 里的百炼客户端, + // 不再经过 Dart(见 AiPcmBridge)。startAI / stopAI / 调速这些控制事件 + // 照旧上抛——会话的起停仍由 Dart 的 DeviceAiSessionService 编排。 + if type == "micData", AiPcmBridge.isEnabled, let pcm = event["data"] as? Data { + AiPcmBridge.postUplink(pcm: pcm) + return + } BluetoothManager.aiCallback?(event) } @@ -528,6 +694,9 @@ public final class BluetoothManager: NSObject { txCharacteristic = nil didRequestServiceDiscovery = false pendingWrites.removeAll() + // 断连后这些包已经没有接收方,留着只会在重连瞬间灌一段过期音频 + flowQueue.removeAll() + writeTypeLogged = false isReadyNotified = false processingQueue.async { [weak self] in self?.stashBuffer.removeAll() @@ -581,6 +750,9 @@ extension BluetoothManager: CBCentralManagerDelegate { isReadyNotified = false if !didRequestServiceDiscovery { didRequestServiceDiscovery = true + // 只发现私有服务。枚举全部(传 nil)只在 2026-09-21 做过一次诊断, + // 结论记在 CLAUDE.md(Echo-one 有 180A/180F/00001100-…/两个私有服务, + // 其中 180F 是空壳);常态下没必要,少一次全量枚举就少一批 ATT 事务。 peripheral.discoverServices([serviceUUID]) } } @@ -613,9 +785,17 @@ extension BluetoothManager: CBPeripheralDelegate { public func peripheral(_ peripheral: CBPeripheral, didDiscoverServices error: Error?) { guard self.peripheral?.identifier == peripheral.identifier else { return } guard error == nil, let services = peripheral.services else { return } + os_log("[GattSvc] 服务发现完成,共 %d 个: %{public}@", log: Self.bleLog, type: .info, + services.count, services.map { $0.uuid.uuidString }.joined(separator: ", ")) for s in services where s.uuid == serviceUUID { peripheral.discoverCharacteristics([txUUID, rxUUID], for: s) } + // ⚠️ 只对私有服务发起特征发现,别去碰 0x180F。 + // 2026-09-21 实测:对它 discoverCharacteristics + readValue + setNotifyValue 之后, + // 通话下行的 canSendWriteWithoutResponse 从第一包起就恒为 false,548 包只发出去 11 个 + // (同样时长上一版是 12308),远端完全听不到译文。ATT 事务是串行的,挂在这个空壳服务 + // 上的读/订阅会把整条通道堵死。而它本来就没有价值:值恒为 0%,同一时刻私有 BB 0A + // 解出的是 L=90 R=86。服务清单记在 CLAUDE.md,不需要每次连接都去枚举。 } public func peripheral(_ peripheral: CBPeripheral, @@ -625,6 +805,7 @@ extension BluetoothManager: CBPeripheralDelegate { // 避免把 A 的 rx/tx 引用写入共享字段,污染新连接的数据收发。 guard self.peripheral?.identifier == peripheral.identifier else { return } guard error == nil, let chars = service.characteristics else { return } + for ch in chars { let props = ch.properties let supportsNotify = props.contains(.notify) || props.contains(.indicate) @@ -658,6 +839,13 @@ extension BluetoothManager: CBPeripheralDelegate { error: Error?) { guard self.peripheral?.identifier == peripheral.identifier else { return } guard error == nil, let data = characteristic.value else { return } + // ⚠️ 标准电池值必须在这里截住:它是裸的一字节百分比,不是私有协议的帧。 + // 放进 appendAndParse 会污染 stashBuffer,把后面真正的 BB 帧一起解坏。 + if characteristic.uuid == CBUUID(string: "2A19") { + let pct = data.first.map { Int($0) } ?? -1 + os_log("[GattSvc] 标准电池服务上报电量 %d%%", log: Self.bleLog, type: .info, pct) + return + } // 在处理队列上进行帧解析,不阻塞 BLE 队列 processingQueue.async { [weak self] in self?.appendAndParse(data) @@ -667,4 +855,12 @@ extension BluetoothManager: CBPeripheralDelegate { public func peripheral(_ peripheral: CBPeripheral, didWriteValueFor characteristic: CBCharacteristic, error: Error?) {} + + /// CoreBluetooth 的发送队列又有空位了 —— 这是无应答写唯一的背压信号, + /// 不接这个回调就只能盲写,队列满了之后写进去的包会被静默丢掉。 + public func peripheralIsReady(toSendWriteWithoutResponse peripheral: CBPeripheral) { + runOnBleQueue { [weak self] in + self?.drainFlowQueue() + } + } } diff --git a/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/BluetoothManagerPlugin.swift b/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/BluetoothManagerPlugin.swift index e01d3549..7ab3b39f 100644 --- a/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/BluetoothManagerPlugin.swift +++ b/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/BluetoothManagerPlugin.swift @@ -70,6 +70,24 @@ class AiStreamHandler: NSObject, FlutterStreamHandler { } } +/// 录音统计流:原生每 100ms 报一次 `{id, bytes, level}`。 +/// 时长由 bytes 算(与文件实际长度永远一致),电平驱动波形。 +class RecorderStreamHandler: NSObject, FlutterStreamHandler { + weak var plugin: BluetoothManagerPlugin? + + init(plugin: BluetoothManagerPlugin) { self.plugin = plugin } + + func onListen(withArguments arguments: Any?, eventSink events: @escaping FlutterEventSink) -> FlutterError? { + plugin?.recorderEventSink = events + return nil + } + + func onCancel(withArguments arguments: Any?) -> FlutterError? { + plugin?.recorderEventSink = nil + return nil + } +} + class CmdStreamHandler: NSObject, FlutterStreamHandler { weak var plugin: BluetoothManagerPlugin? @@ -96,6 +114,7 @@ public class BluetoothManagerPlugin: NSObject, FlutterPlugin { var cmdEventSink: FlutterEventSink? var audioEventSink: FlutterEventSink? var aiEventSink: FlutterEventSink? + var recorderEventSink: FlutterEventSink? // 持有 StreamHandler 引用,防止被过早释放 private var statusHandler: StatusStreamHandler? @@ -103,6 +122,7 @@ public class BluetoothManagerPlugin: NSObject, FlutterPlugin { private var cmdHandler: CmdStreamHandler? private var audioHandler: AudioStreamHandler? private var aiHandler: AiStreamHandler? + private var recorderHandler: RecorderStreamHandler? public static func register(with registrar: FlutterPluginRegistrar) { let channel = FlutterMethodChannel(name: "bluetooth_manager", binaryMessenger: registrar.messenger()) @@ -111,6 +131,7 @@ public class BluetoothManagerPlugin: NSObject, FlutterPlugin { let cmdChannel = FlutterEventChannel(name: "bluetooth_manager/cmd", binaryMessenger: registrar.messenger()) let audioChannel = FlutterEventChannel(name: "bluetooth_manager/audio", binaryMessenger: registrar.messenger()) let aiChannel = FlutterEventChannel(name: "bluetooth_manager/ai", binaryMessenger: registrar.messenger()) + let recChannel = FlutterEventChannel(name: "bluetooth_manager/recorder", binaryMessenger: registrar.messenger()) let instance = BluetoothManagerPlugin() registrar.addMethodCallDelegate(instance, channel: channel) @@ -120,7 +141,9 @@ public class BluetoothManagerPlugin: NSObject, FlutterPlugin { instance.cmdHandler = CmdStreamHandler(plugin: instance) instance.audioHandler = AudioStreamHandler(plugin: instance) instance.aiHandler = AiStreamHandler(plugin: instance) + instance.recorderHandler = RecorderStreamHandler(plugin: instance) + recChannel.setStreamHandler(instance.recorderHandler) statusChannel.setStreamHandler(instance.statusHandler) readChannel.setStreamHandler(instance.readHandler) cmdChannel.setStreamHandler(instance.cmdHandler) @@ -194,6 +217,12 @@ public class BluetoothManagerPlugin: NSObject, FlutterPlugin { DispatchQueue.main.async { self?.statusEventSink?(2) } } bluetoothManager?.onDisconnected = { [weak self] _, _ in + // 设备断了就不会再有帧:把还开着的录音收尾并回填 WAV 头。 + // Dart 侧的服务也会收到断连并调 recorderStop,那次会拿到 nil + // (已经注销),调用方用最后一次统计里的字节数兜底。 + // 不做这一步的话,进程随后被杀就留下一个 data 长度为 0 的 + // WAV——文件有几十 MB,播放器却认为它是空的。 + PcmRecorderRegistry.stopAll(save: true) DispatchQueue.main.async { self?.statusEventSink?(0) } } bluetoothManager?.onDataReceived = { [weak self] data in @@ -232,6 +261,34 @@ public class BluetoothManagerPlugin: NSObject, FlutterPlugin { CallTranslationDownlink.shared.start() result(true) + // 通话翻译 PCM 原生直连桥(见 CallPcmBridge.swift)。 + // 开着时 micData/spkData 不再经过 Dart;关掉即恢复原来的 EventChannel 路径。 + case "setCallPcmBridge": + let args = call.arguments as? [String: Any] + let enabled = (args?["enabled"] as? Bool) ?? false + if enabled { CallPcmBridge.enable() } else { CallPcmBridge.disable() } + result(true) + + case "getCallPcmBridgeStats": + result(CallPcmBridge.stats()) + + // EMAI 会话 PCM 原生直连桥(见 AiPcmBridge.swift)。 + // 开着时 AI 模式的 micData 不再经过 Dart;关掉即恢复原来的 EventChannel 路径。 + case "setAiPcmBridge": + let args = call.arguments as? [String: Any] + let enabled = (args?["enabled"] as? Bool) ?? false + if enabled { AiPcmBridge.enable() } else { AiPcmBridge.disable() } + result(true) + + case "getAiPcmBridgeStats": + result(AiPcmBridge.stats()) + + case "getCallDownlinkStats": + var st = CallTranslationDownlink.shared.stats() + st["ble"] = BluetoothManager.sharedInstance?.bleWriteStats() ?? [:] + CallTranslationDownlink.shared.resetPeaks() + result(st) + case "stopCallDownlink": CallTranslationDownlink.shared.stop() result(true) @@ -312,6 +369,51 @@ public class BluetoothManagerPlugin: NSObject, FlutterPlugin { } manager.getConnectedDevice { result($0) } + // ===== 设备音频直接落盘(见 PcmWavRecorder.swift)===== + // 帧在原生解码完就地写 WAV,不再每帧过一次平台通道给 Dart 写盘。 + case "recorderStart": + guard let args = call.arguments as? [String: Any], + let id = args["id"] as? String, + let path = args["path"] as? String else { + result(false) + return + } + let channels = (args["channels"] as? Int) ?? 1 + let sampleRate = (args["sampleRate"] as? Int) ?? 16000 + let rec = PcmWavRecorder(id: id, path: path, channels: channels, sampleRate: sampleRate) + rec.onStats = { [weak self] rid, s in + guard let self = self else { return } + DispatchQueue.main.async { + self.recorderEventSink?(["id": rid, "bytes": s.bytes, "level": s.level]) + } + } + guard rec.start() else { + result(false) + return + } + if channels >= 2 { + let l = (args["left"] as? String) ?? "micData" + let r = (args["right"] as? String) ?? "spkData" + PcmRecorderRegistry.register(rec, monoSource: nil, stereoSources: (left: l, right: r)) + } else { + let s = (args["source"] as? String) ?? "mainMicData" + PcmRecorderRegistry.register(rec, monoSource: s, stereoSources: nil) + } + result(true) + + case "recorderStop": + guard let args = call.arguments as? [String: Any], + let id = args["id"] as? String else { + result(nil) + return + } + let save = (args["save"] as? Bool) ?? true + guard let rec = PcmRecorderRegistry.unregister(id: id) else { + result(nil) + return + } + result(rec.stop(save: save)) + default: result(FlutterMethodNotImplemented) } diff --git a/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/CallPcmBridge.swift b/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/CallPcmBridge.swift new file mode 100644 index 00000000..35f96a14 --- /dev/null +++ b/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/CallPcmBridge.swift @@ -0,0 +1,94 @@ +import Foundation + +/// 通话翻译 PCM 的**原生直连桥**(bluetooth_manager 这一端)。 +/// +/// ## 为什么有这个 +/// 通话翻译的音频原来是 `本插件 → EventChannel → Dart → MethodChannel → azure_speech`, +/// 20ms 一帧、双路,**每秒约 200 次跨界**。2026-09-20 苹果测试反馈「WiFi 下断断续续、 +/// 长句基本不行」,日志里就是 Dart event loop 被拖住之后的 `AudioSendSlow 1011`。 +/// 桥开着时,解码出来的 micData/spkData **不再经过 Dart**,直接交给 azure_speech; +/// azure_speech 合成出来的译文 PCM 也直接回到 [CallTranslationDownlink]。 +/// +/// ## ⚠️ 传输为什么是 NotificationCenter +/// 本插件是 CocoaPods 包、azure_speech 是 SwiftPM 包,两边**不能互相 import** +/// (Android 那边是 Gradle 子工程,可以直接依赖,走的是类型化的静态槽)。 +/// NotificationCenter 的 post 是**同步、在发帖线程上**执行的,没有序列化、没有跨线程, +/// 开销只是一次字典分配,对 100 帧/秒来说可以忽略。 +/// +/// ## ⚠️ 两边手工同步的契约 +/// 通知名与 userInfo 键在 azure_speech 的 `BesCallPcmBridge.swift` 里各写了一份, +/// **改一处必须同改另一处**,改漏了不会编译报错,只会表现为「桥开着但一帧都到不了」。 +/// 契约: +/// - 上行 `com.eaimar.bes.callpcm.uplink` userInfo: `leg`("A"=本端麦 / "B"=对端通话音), `pcm`(Data, 16k/16bit/mono) +/// - 下行 `com.eaimar.bes.callpcm.downlink` userInfo: `leg`("A"=对端听 / "B"=己方听), `pcm`(Data) +public enum CallPcmBridge { + public static let uplinkNotification = Notification.Name("com.eaimar.bes.callpcm.uplink") + public static let downlinkNotification = Notification.Name("com.eaimar.bes.callpcm.downlink") + public static let legKey = "leg" + public static let pcmKey = "pcm" + + private static let lock = NSLock() + private static var enabled = false + private static var downlinkObserver: NSObjectProtocol? + private static var uplinkFrames: Int64 = 0 + private static var downlinkFrames: Int64 = 0 + + /// 桥是否开着。解码线程每帧都读它,用锁而不是 atomic 是因为 Swift 没有内建 atomic, + /// 而 NSLock 的无争用开销是纳秒级。 + public static var isEnabled: Bool { + lock.lock(); defer { lock.unlock() } + return enabled + } + + /// 开桥:之后 micData/spkData 只走通知,不再进 Dart 的 audio EventChannel; + /// 同时开始收 azure_speech 发来的译文 PCM 喂给下行发送器。 + public static func enable() { + lock.lock() + if enabled { lock.unlock(); return } + enabled = true + uplinkFrames = 0 + downlinkFrames = 0 + lock.unlock() + downlinkObserver = NotificationCenter.default.addObserver( + forName: downlinkNotification, object: nil, queue: nil + ) { note in + guard let leg = note.userInfo?[legKey] as? String, + let pcm = note.userInfo?[pcmKey] as? Data, !pcm.isEmpty else { return } + CallTranslationDownlink.shared.pushPcm(leg: leg, pcm: pcm) + lock.lock(); downlinkFrames &+= 1; lock.unlock() + } + NSLog("[CallPcmBridge] 原生直连桥已开启(上行走通知,下行直喂发送器)") + } + + public static func disable() { + lock.lock() + if !enabled { lock.unlock(); return } + enabled = false + let up = uplinkFrames, down = downlinkFrames + lock.unlock() + if let o = downlinkObserver { + NotificationCenter.default.removeObserver(o) + downlinkObserver = nil + } + NSLog("[CallPcmBridge] 原生直连桥已关闭 上行=\(up) 帧 下行=\(down) 帧") + } + + /// 由 BluetoothManager 的解码线程调用:把一帧上行 PCM 交给 azure_speech。 + /// [leg] "A" = 本端麦(micData);"B" = 对端通话音(spkData)。 + static func postUplink(leg: String, pcm: Data) { + NotificationCenter.default.post(name: uplinkNotification, object: nil, + userInfo: [legKey: leg, pcmKey: pcm]) + lock.lock() + uplinkFrames &+= 1 + let n = uplinkFrames + lock.unlock() + if n == 1 || n % 1000 == 0 { + NSLog("[CallPcmBridge] 上行已直连转发 \(n) 帧") + } + } + + public static func stats() -> [String: Any] { + lock.lock(); defer { lock.unlock() } + return ["enabled": enabled, "uplinkFrames": uplinkFrames, "downlinkFrames": downlinkFrames] + } +} diff --git a/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/CallTranslationDownlink.swift b/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/CallTranslationDownlink.swift index 19ce3057..ad9f6a19 100644 --- a/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/CallTranslationDownlink.swift +++ b/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/CallTranslationDownlink.swift @@ -1,4 +1,5 @@ import Foundation +import os.log /// 通话翻译的**下行 TTS 音频**发送器(恒玄/BES 专用),Android /// `CallTranslationDownlink.kt` 的 iOS 对应实现,行为逐条对齐。 @@ -44,6 +45,23 @@ public final class CallTranslationDownlink { private var intervalMs: Int = 20 private var sentPackets = 0 + // MARK: - 流畅度观测 + // + // ⚠️ 这里原来用的是 NSLog,而 NSLog 的内容在 release / profile 包里**抓不到** + // (idevicesyslog 只收得到 os_log),于是下行是整条链路上唯一的盲区。排查 + // 「杂音、不流畅」时看不到队列长度和发送节奏,只能猜。一律改走 os_log。 + private static let dlLog = OSLog(subsystem: "com.eaimar.bluetooth", category: "downlink") + /// 定时器实际触发的时刻,用来算真实间隔 —— 期望 20ms,偏大就是发送跟不上 + private var lastTickAt: CFAbsoluteTime = 0 + private var tickGapSumMs: Double = 0 + private var tickGapCount: Int = 0 + private var tickGapMaxMs: Double = 0 + /// 两路各自的入队/出队总数,用来判断是产出多了还是发得慢了 + private var pushedFramesA = 0 + private var pushedFramesB = 0 + private var maxQueueA = 0 + private var maxQueueB = 0 + /// 静音帧(与 deepvoice 一致,耳机侧认这个码流) private let silenceFrame = Data([ 0x0A, 0x8E, 0x30, 0xBD, 0x3F, 0x83, 0xFF, 0x00, 0x00, 0x00, @@ -65,8 +83,12 @@ public final class CallTranslationDownlink { self.sentPackets = 0 if self.encoderA == nil { self.encoderA = G722Codec(sampleRate: 16000) } if self.encoderB == nil { self.encoderB = G722Codec(sampleRate: 16000) } + self.lastTickAt = 0 + self.tickGapSumMs = 0; self.tickGapCount = 0; self.tickGapMaxMs = 0 + self.pushedFramesA = 0; self.pushedFramesB = 0 + self.maxQueueA = 0; self.maxQueueB = 0 self.scheduleTimer() - NSLog("[BesCallDownlink] 下行发送启动, interval=\(self.intervalMs)ms") + os_log("[BesCallDownlink] 下行发送启动, interval=%dms", log: Self.dlLog, type: .info, self.intervalMs) } } @@ -82,7 +104,58 @@ public final class CallTranslationDownlink { self.pendingB = Data() self.encoderA = nil self.encoderB = nil - NSLog("[BesCallDownlink] 下行发送停止, 共发出 \(self.sentPackets) 包") + // 清自己的队列还不够:已经递给 GATT 层的那些包也得丢掉, + // 否则它们会一直堵在控制命令(查电量/查版本)前面 + BluetoothManager.sharedInstance?.flushRealtimeQueue() + let avgGap = self.tickGapCount > 0 ? self.tickGapSumMs / Double(self.tickGapCount) : 0 + let bleStats = BluetoothManager.sharedInstance?.bleWriteStats() ?? [:] + os_log(""" + [BesCallDownlink] 收尾: 发出=%d 包 | 入队 A=%d B=%d | 队列峰值 A=%d B=%d \ + | 定时器实际间隔 平均=%.1fms 最大=%.1fms (期望 %dms) | GATT %{public}@ + """, + log: Self.dlLog, type: .info, + self.sentPackets, self.pushedFramesA, self.pushedFramesB, + self.maxQueueA, self.maxQueueB, + avgGap, self.tickGapMaxMs, self.intervalMs, + String(describing: bleStats)) + } + } + + /// 下行发送器的实时统计,供 Dart 在切前后台等时刻取样。 + /// + /// 这些数字原来只在 os_log 里,而排查后台卡顿时 `idevicesyslog` 极不稳定 + /// (2026-09-21 断过两次),于是下行成了唯一看不见的一段。暴露出来之后, + /// Dart 侧可以把它写进文件日志(`devicectl device copy from` 直接拉), + /// 不再依赖 usbmuxd。 + /// + /// **gapAvgMs 是判断 iOS 后台节流的关键**:期望等于 intervalMs(16/20/25), + /// 明显变大就说明 DispatchSourceTimer 被系统合并触发了,下行必然断续。 + public func stats() -> [String: Any] { + var out: [String: Any] = [:] + queue.sync { + let avg = self.tickGapCount > 0 + ? self.tickGapSumMs / Double(self.tickGapCount) : 0 + out = [ + "running": self.running, + "intervalMs": self.intervalMs, + "sentPackets": self.sentPackets, + "queueA": self.legAQueue.count, + "queueB": self.legBQueue.count, + "maxQueueA": self.maxQueueA, + "maxQueueB": self.maxQueueB, + "gapAvgMs": (avg * 10).rounded() / 10, + "gapMaxMs": (self.tickGapMaxMs * 10).rounded() / 10, + ] + } + return out + } + + /// 取样之间重置峰值,否则 gapMaxMs 会被会话早期的一次尖峰永久占住, + /// 看不出「切到后台之后」到底有没有变差。 + public func resetPeaks() { + queue.async { + self.tickGapSumMs = 0; self.tickGapCount = 0; self.tickGapMaxMs = 0 + self.maxQueueA = 0; self.maxQueueB = 0 } } @@ -91,7 +164,7 @@ public final class CallTranslationDownlink { queue.async { guard self.intervalMs != ms else { return } self.intervalMs = ms - NSLog("[BesCallDownlink] 下行节奏调整为 \(ms)ms") + os_log("[BesCallDownlink] 下行节奏调整为 %dms", log: Self.dlLog, type: .info, ms) if self.running { self.scheduleTimer() } } } @@ -130,9 +203,13 @@ public final class CallTranslationDownlink { // 队列满说明下行跟不上产出(通常是耳机没在收),丢最旧的保实时 if self.legAQueue.count >= Self.queueCap { self.legAQueue.removeFirst() } self.legAQueue.append(Data(chunk)) + self.pushedFramesA += 1 + if self.legAQueue.count > self.maxQueueA { self.maxQueueA = self.legAQueue.count } } else { if self.legBQueue.count >= Self.queueCap { self.legBQueue.removeFirst() } self.legBQueue.append(Data(chunk)) + self.pushedFramesB += 1 + if self.legBQueue.count > self.maxQueueB { self.maxQueueB = self.legBQueue.count } } } } @@ -144,6 +221,18 @@ public final class CallTranslationDownlink { /// 只在 [queue] 上调用 private func sendOnePacket() { guard running else { return } + // 定时器真实节奏:DispatchSourceTimer 会为省电合并触发,而 handler 里还压着 + // G.722 编码与一次 GATT 写。实际间隔一旦稳定大于 intervalMs,下行就是在欠发, + // 耳机侧必然断续 —— 这是「不流畅」和「队列只涨不落」的分水岭。 + let now = CFAbsoluteTimeGetCurrent() + if lastTickAt > 0 { + let gap = (now - lastTickAt) * 1000 + tickGapSumMs += gap + tickGapCount += 1 + if gap > tickGapMaxMs { tickGapMaxMs = gap } + } + lastTickAt = now + let legB: Data? = legBQueue.isEmpty ? nil : legBQueue.removeFirst() let legA: Data? = legAQueue.isEmpty ? nil : legAQueue.removeFirst() // 两路都没内容就不发,避免通话里灌满无谓的静音包 @@ -153,10 +242,18 @@ public final class CallTranslationDownlink { packet.append(contentsOf: [0xAA, 0x56, UInt8(Self.packetSize), 0x03]) packet.append(legB ?? silenceFrame) packet.append(legA ?? silenceFrame) - BluetoothManager.writeBytes(packet) + BluetoothManager.writeRealtimeAudio(packet) sentPackets += 1 + // 每 100 包(约 2 秒)一条。队列长度单调上涨 = 发送跟不上产出。 if sentPackets == 1 || sentPackets % 100 == 0 { - NSLog("[BesCallDownlink] 已下发 \(sentPackets) 包 (队列 A=\(legAQueue.count) B=\(legBQueue.count))") + let avgGap = tickGapCount > 0 ? tickGapSumMs / Double(tickGapCount) : 0 + let ble = BluetoothManager.sharedInstance?.bleWriteStats() ?? [:] + os_log("[BesCallDownlink] 已下发 %d 包 (队列 A=%d B=%d) 间隔 平均=%.1fms 最大=%.1fms | GATT 丢=%{public}@ 阻塞=%{public}@ 待发=%{public}@", + log: Self.dlLog, type: .info, + sentPackets, legAQueue.count, legBQueue.count, avgGap, tickGapMaxMs, + String(describing: ble["dropped"] ?? 0), + String(describing: ble["blocked"] ?? 0), + String(describing: ble["queued"] ?? 0)) } } } diff --git a/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/PcmWavRecorder.swift b/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/PcmWavRecorder.swift new file mode 100644 index 00000000..edab74cf --- /dev/null +++ b/apps/client/local_plugins/bluetooth_manager/ios/bluetooth_manager/Sources/bluetooth_manager/PcmWavRecorder.swift @@ -0,0 +1,365 @@ +import Foundation + +/// 设备音频直接落盘的 WAV 录制器(16bit 小端,单声道或双声道交织)。 +/// +/// **为什么在原生**:此前是每帧 PCM 经 EventChannel 到 Dart、由 `WavWriter` +/// 用 `RandomAccessFile` 排队写盘。恒玄 20ms 一帧 = 单路 50 帧/s、通话双路 +/// 100 帧/s,每帧都要跨一次平台通道并在 Dart 的单 isolate 上排队——event loop +/// 一被页面布局或 GC 占住,写入队列就堆积。`WavWriter` 的注释里记着这条链路 +/// 出过的事故:写入重叠导致丢帧,**等效采样率只剩 1587Hz**(播放又快又糊)。 +/// 现在帧在原生解码完就地写文件,Dart 只收路径、字节数和电平。 +/// +/// ⚠️ 这个类**不是**单例。azure_speech 里的 `RecordFile` 是单例且已被通话翻译 +/// 的立体声录音占着,复用它会让现场录音和翻译录音互相踩。每次录音一个实例。 +final class PcmWavRecorder { + + /// 一次统计上报 + struct Stats { + let bytes: Int // 已写入的音频字节数(不含 44 字节头) + let level: Double // 本窗口电平峰值 0~1 + } + + let id: String + let path: String + let channels: Int + let sampleRate: Int + + /// 统计上报间隔。与 Dart 侧波形采样同频;原来是每帧 `level.value=`, + /// 50~100 次/s 的 Rx 通知里绝大多数会被下一帧覆盖掉。 + private static let statsInterval: TimeInterval = 0.1 + + /// 串行写队列。文件句柄只在这个队列上碰。 + private let queue: DispatchQueue + private var handle: FileHandle? + private(set) var dataBytes: Int = 0 + private(set) var writtenFrames: Int = 0 + private(set) var droppedFrames: Int = 0 + private var closed = false + + /// 双声道的左右暂存(见 StereoPairer 的移植说明) + private var leftBuf = Data() + private var rightBuf = Data() + /// 一路长期缺席时另一路的最大堆积:10 秒 @16k/16bit。 + /// 超了丢最旧的,避免对端根本没数据时本端无限涨。 + private let maxPending = 16000 * 2 * 10 + + private var levelPeak: Double = 0 + private var lastStatsAt: TimeInterval = 0 + + /// 统计回调(在写队列上调用,接收方自己派发) + var onStats: ((String, Stats) -> Void)? + + init(id: String, path: String, channels: Int, sampleRate: Int) { + self.id = id + self.path = path + self.channels = max(1, min(2, channels)) + self.sampleRate = sampleRate + self.queue = DispatchQueue(label: "bes.pcmwav.\(id)") + } + + // MARK: - 生命周期 + + /// 建文件并写占位头。返回 false 表示建不出来(目录不可写等)。 + func start() -> Bool { + var ok = false + queue.sync { + let url = URL(fileURLWithPath: path) + do { + try FileManager.default.createDirectory( + at: url.deletingLastPathComponent(), + withIntermediateDirectories: true) + } catch { + NSLog("[PcmWavRecorder] 建目录失败: %@", error.localizedDescription) + return + } + FileManager.default.createFile(atPath: path, contents: nil) + do { + let h = try FileHandle(forWritingTo: url) + try h.write(contentsOf: Self.wavHeader( + dataLength: 0, channels: channels, sampleRate: sampleRate)) + handle = h + dataBytes = 0 + writtenFrames = 0 + droppedFrames = 0 + lastStatsAt = Date().timeIntervalSince1970 + ok = true + } catch { + NSLog("[PcmWavRecorder] 建文件失败: %@", error.localizedDescription) + } + } + return ok + } + + /// 停止。[save] = false 时删除文件(没收到任何音频那种)。 + /// 返回统计,给 Dart 算时长与排查用。 + @discardableResult + func stop(save: Bool) -> [String: Any] { + var result: [String: Any] = [:] + queue.sync { + guard !closed else { return } + closed = true + let h = handle + handle = nil + if let h = h { + if save { + // 回填 RIFF/data 长度 + do { + try h.seek(toOffset: 0) + try h.write(contentsOf: Self.wavHeader( + dataLength: dataBytes, channels: channels, sampleRate: sampleRate)) + } catch { + NSLog("[PcmWavRecorder] 回填 WAV 头失败: %@", error.localizedDescription) + } + } + try? h.close() + } + if !save { + try? FileManager.default.removeItem(atPath: path) + } + result = [ + "bytes": dataBytes, + "written": writtenFrames, + "dropped": droppedFrames, + ] + } + return result + } + + // MARK: - 进料 + + /// 单声道:直接落盘 + func append(_ pcm: Data) { + guard !pcm.isEmpty else { return } + queue.async { [weak self] in + guard let self = self, !self.closed else { return } + self.trackLevel(pcm) + self.writeLocked(pcm) + self.emitStatsIfDue() + } + } + + /// 双声道左路(本端麦) + func appendLeft(_ pcm: Data) { appendStereo(pcm, isLeft: true) } + + /// 双声道右路(对端) + func appendRight(_ pcm: Data) { appendStereo(pcm, isLeft: false) } + + private func appendStereo(_ pcm: Data, isLeft: Bool) { + guard !pcm.isEmpty else { return } + queue.async { [weak self] in + guard let self = self, !self.closed else { return } + if isLeft { + self.leftBuf.append(pcm) + if self.leftBuf.count > self.maxPending { + self.leftBuf = Data(self.leftBuf.suffix(self.maxPending)) + } + } else { + self.rightBuf.append(pcm) + if self.rightBuf.count > self.maxPending { + self.rightBuf = Data(self.rightBuf.suffix(self.maxPending)) + } + } + self.flushPairsLocked() + self.emitStatsIfDue() + } + } + + /// StereoPairer 的移植:两路都排队,谁都不丢,每次取出两边都已有的那一段 + /// (长度取较短者、对齐到整样本)。 + /// + /// ⚠️ 不能写成「只留最后一帧对端、本端来一帧配一帧」:两路在 BLE 上是同一 + /// 个帧里交替的 40 字节块,但到达这里已经是两次独立回调,顺序不保证严格交替。 + /// 按「最后一帧」配对会把右声道断续抽掉,转写出来只剩自己这一方。 + /// Dart 侧的金测试 `test/stereo_pairer_test.dart` 定的就是这个行为,两边同改。 + private func flushPairsLocked() { + let n = min(leftBuf.count, rightBuf.count) & ~1 + guard n > 0 else { return } + let l = leftBuf.prefix(n) + let r = rightBuf.prefix(n) + leftBuf = Data(leftBuf.dropFirst(n)) + rightBuf = Data(rightBuf.dropFirst(n)) + trackLevel(Data(l)) // 电平按本端(左声道)算,与 Dart 原行为一致 + writeLocked(Self.interleave(Data(l), Data(r))) + } + + private func writeLocked(_ data: Data) { + guard let h = handle else { return } + do { + try h.write(contentsOf: data) + dataBytes += data.count + writtenFrames += 1 + } catch { + droppedFrames += 1 + NSLog("[PcmWavRecorder] 写入失败(已丢 %d): %@", droppedFrames, error.localizedDescription) + } + } + + // MARK: - 电平与统计 + + private func trackLevel(_ pcm: Data) { + let v = Self.rmsLevel(pcm) + if v > levelPeak { levelPeak = v } + } + + private func emitStatsIfDue() { + let now = Date().timeIntervalSince1970 + guard now - lastStatsAt >= Self.statsInterval else { return } + lastStatsAt = now + let s = Stats(bytes: dataBytes, level: levelPeak) + levelPeak = 0 + onStats?(id, s) + } + + // MARK: - 工具 + + /// 与 Dart `WavWriter.rmsLevel` 同口径:归一化 RMS 再开一次根 + /// (语音 RMS 常年在 0.02~0.3,不拉一下动态范围界面几乎没动静)。 + static func rmsLevel(_ pcm: Data) -> Double { + let count = pcm.count / 2 + guard count > 0 else { return 0 } + var sum = 0.0 + pcm.withUnsafeBytes { raw in + guard let base = raw.baseAddress?.assumingMemoryBound(to: UInt8.self) else { return } + for i in 0..= 0x8000 { v -= 0x10000 } + let n = Double(v) / 32768.0 + sum += n * n + } + } + let rms = (sum / Double(count)).squareRoot() + return min(1.0, rms.squareRoot()) + } + + /// 两路单声道按样本交织成双声道(左在前) + static func interleave(_ left: Data, _ right: Data) -> Data { + let samples = min(left.count, right.count) / 2 + var out = Data(count: samples * 4) + out.withUnsafeMutableBytes { o in + left.withUnsafeBytes { l in + right.withUnsafeBytes { r in + guard let op = o.baseAddress?.assumingMemoryBound(to: UInt8.self), + let lp = l.baseAddress?.assumingMemoryBound(to: UInt8.self), + let rp = r.baseAddress?.assumingMemoryBound(to: UInt8.self) else { return } + for i in 0.. Data { + let bits = 16 + let byteRate = sampleRate * channels * bits / 8 + let blockAlign = channels * bits / 8 + var d = Data(capacity: 44) + func ascii(_ s: String) { d.append(contentsOf: Array(s.utf8)) } + func u32(_ v: Int) { + let x = UInt32(truncatingIfNeeded: v) + d.append(contentsOf: [UInt8(x & 0xFF), UInt8((x >> 8) & 0xFF), + UInt8((x >> 16) & 0xFF), UInt8((x >> 24) & 0xFF)]) + } + func u16(_ v: Int) { + let x = UInt16(truncatingIfNeeded: v) + d.append(contentsOf: [UInt8(x & 0xFF), UInt8((x >> 8) & 0xFF)]) + } + ascii("RIFF"); u32(36 + dataLength); ascii("WAVE") + ascii("fmt "); u32(16); u16(1); u16(channels) + u32(sampleRate); u32(byteRate); u16(blockAlign); u16(bits) + ascii("data"); u32(dataLength) + return d + } +} + +/// 活跃录制器登记处。 +/// +/// `BluetoothManager` 解码完一帧就按类型喂给这里的录制器——**只是多挂一个 +/// sink,不拦截原有的 EventChannel 回调**。这点和 `CallPcmBridge` 不同: +/// 那条是「桥开着就不再上抛」,因为通话翻译的 PCM 有唯一消费者;录音则可能 +/// 与别的订阅方(比如正在跑的通话翻译)并存,拦了会把对方饿死。 +enum PcmRecorderRegistry { + + /// 音频类型 → 录制器的投递表。 + /// 类型串与 `BluetoothManager.publishCallTranslation` 里发的一致: + /// `mainMicData`(现场录音主麦)/ `micData`(通话本端)/ `spkData`(通话对端)。 + private static var monoSinks: [String: [PcmWavRecorder]] = [:] + private static var stereoSinks: [String: [(rec: PcmWavRecorder, isLeft: Bool)]] = [:] + private static var byId: [String: PcmWavRecorder] = [:] + private static let lock = NSLock() + + static var isEmpty: Bool { + lock.lock(); defer { lock.unlock() } + return byId.isEmpty + } + + /// 登记一个录制器。[monoSource] 与 [stereoSources] 二选一。 + static func register(_ rec: PcmWavRecorder, + monoSource: String?, + stereoSources: (left: String, right: String)?) { + lock.lock(); defer { lock.unlock() } + byId[rec.id] = rec + if let s = monoSource { + monoSinks[s, default: []].append(rec) + } + if let s = stereoSources { + stereoSinks[s.left, default: []].append((rec, true)) + stereoSinks[s.right, default: []].append((rec, false)) + } + } + + static func unregister(id: String) -> PcmWavRecorder? { + lock.lock(); defer { lock.unlock() } + guard let rec = byId.removeValue(forKey: id) else { return nil } + for (k, v) in monoSinks { + let left = v.filter { $0 !== rec } + if left.isEmpty { monoSinks.removeValue(forKey: k) } else { monoSinks[k] = left } + } + for (k, v) in stereoSinks { + let left = v.filter { $0.rec !== rec } + if left.isEmpty { stereoSinks.removeValue(forKey: k) } else { stereoSinks[k] = left } + } + return rec + } + + static func recorder(id: String) -> PcmWavRecorder? { + lock.lock(); defer { lock.unlock() } + return byId[id] + } + + /// 由 `BluetoothManager` 在解码线程上调用。没有录制器时立刻返回, + /// 开销就是一次字典查空。 + static func feed(type: String, pcm: Data) { + lock.lock() + let mono = monoSinks[type] + let stereo = stereoSinks[type] + lock.unlock() + if let mono = mono { + for r in mono { r.append(pcm) } + } + if let stereo = stereo { + for e in stereo { + if e.isLeft { e.rec.appendLeft(pcm) } else { e.rec.appendRight(pcm) } + } + } + } + + /// 进程级兜底:断连/停录时把还开着的全停掉,避免留下没回填头的半截文件 + static func stopAll(save: Bool) { + lock.lock() + let all = Array(byId.values) + byId.removeAll() + monoSinks.removeAll() + stereoSinks.removeAll() + lock.unlock() + for r in all { r.stop(save: save) } + } +} diff --git a/apps/client/local_plugins/bluetooth_manager/lib/bluetooth_manager.dart b/apps/client/local_plugins/bluetooth_manager/lib/bluetooth_manager.dart index 2fb63a8d..4e0b6fbf 100644 --- a/apps/client/local_plugins/bluetooth_manager/lib/bluetooth_manager.dart +++ b/apps/client/local_plugins/bluetooth_manager/lib/bluetooth_manager.dart @@ -60,6 +60,64 @@ class BluetoothManager { return BluetoothManagerPlatform.instance.pushTtsPcm(leg, pcm); } + /// 通话翻译 PCM 原生直连桥(原生 CallPcmBridge):开着时双路 PCM 不再经过 Dart, + /// 直接交给 azure_speech;译文 PCM 也直接回到下行发送器。 + Future setCallPcmBridge(bool enabled) { + return BluetoothManagerPlatform.instance.setCallPcmBridge(enabled); + } + + /// EMAI 会话 PCM 原生直连桥(原生 AiPcmBridge):开着时唤醒后的耳机麦 PCM + /// 不再经过 Dart,直接交给 azure_speech 的百炼客户端。 + Future setAiPcmBridge(bool enabled) { + return BluetoothManagerPlatform.instance.setAiPcmBridge(enabled); + } + + Future> getAiPcmBridgeStats() { + return BluetoothManagerPlatform.instance.getAiPcmBridgeStats(); + } + + /// 桥的计数(上行/下行帧数),排查「桥开着但没声音」时看这个。 + Future> getCallDownlinkStats() { + return BluetoothManagerPlatform.instance.getCallDownlinkStats(); + } + + Future> getCallPcmBridgeStats() { + return BluetoothManagerPlatform.instance.getCallPcmBridgeStats(); + } + + /// 设备音频直接落盘(原生 PcmWavRecorder):PCM 不再逐帧过平台通道, + /// 原生解码完就地写 WAV,Dart 只收路径、字节数与电平。 + Future startFileRecorder({ + required String id, + required String path, + required int channels, + int sampleRate = 16000, + String? source, + String? left, + String? right, + }) { + return BluetoothManagerPlatform.instance.startFileRecorder( + id: id, + path: path, + channels: channels, + sampleRate: sampleRate, + source: source, + left: left, + right: right, + ); + } + + /// 停止并回填 WAV 头。null = 这个 id 已经不在了(断连时原生自己收尾过)。 + Future?> stopFileRecorder(String id, + {bool save = true}) { + return BluetoothManagerPlatform.instance.stopFileRecorder(id, save: save); + } + + /// 录音统计流:`{'id', 'bytes', 'level'}`,约 100ms 一条 + Stream onRecorderEvent() { + return BluetoothManagerPlatform.instance.onRecorderEvent(); + } + /// AI 实时对话事件流(耳机按键唤醒 0x64 / 结束 0xE5 / 调速 / AI 模式下的耳机麦 PCM)。 /// 与 [onDeviceAudioEvent] 分开:那条是翻译链路的分类音频,两者互斥。 Stream onDeviceAiEvent() { diff --git a/apps/client/local_plugins/bluetooth_manager/lib/bluetooth_manager_method_channel.dart b/apps/client/local_plugins/bluetooth_manager/lib/bluetooth_manager_method_channel.dart index 4fff15aa..08f5decc 100644 --- a/apps/client/local_plugins/bluetooth_manager/lib/bluetooth_manager_method_channel.dart +++ b/apps/client/local_plugins/bluetooth_manager/lib/bluetooth_manager_method_channel.dart @@ -15,12 +15,14 @@ class MethodChannelBluetoothManager extends BluetoothManagerPlatform { final deviceCmdChannel = const EventChannel("bluetooth_manager/cmd"); final deviceAudioChannel = const EventChannel("bluetooth_manager/audio"); final deviceAiChannel = const EventChannel("bluetooth_manager/ai"); + final deviceRecorderChannel = const EventChannel("bluetooth_manager/recorder"); StreamSubscription? _deviceStatusSubscription; StreamSubscription? _deviceDataReceivedSubscription; StreamSubscription? _deviceCmdEventSubscription; StreamSubscription? _deviceAudioEventSubscription; StreamSubscription? _deviceAiEventSubscription; + StreamSubscription? _deviceRecorderEventSubscription; final StreamController statusStream = StreamController.broadcast(); final StreamController dataReceivedStream = @@ -28,6 +30,8 @@ class MethodChannelBluetoothManager extends BluetoothManagerPlatform { final StreamController cmdEventStream = StreamController.broadcast(); final StreamController audioEventStream = StreamController.broadcast(); final StreamController aiEventStream = StreamController.broadcast(); + final StreamController recorderEventStream = + StreamController.broadcast(); @override Future initialize() async { @@ -132,6 +136,53 @@ class MethodChannelBluetoothManager extends BluetoothManagerPlatform { return aiEventStream.stream; } + void _ensureRecorderSubscription() { + if (_deviceRecorderEventSubscription != null) return; + _deviceRecorderEventSubscription = + deviceRecorderChannel.receiveBroadcastStream().listen( + (event) => recorderEventStream.add(event as Map), + onError: (_) => _deviceRecorderEventSubscription = null, + onDone: () => _deviceRecorderEventSubscription = null, + ); + } + + @override + Stream onRecorderEvent() { + _ensureRecorderSubscription(); + return recorderEventStream.stream; + } + + @override + Future startFileRecorder({ + required String id, + required String path, + required int channels, + int sampleRate = 16000, + String? source, + String? left, + String? right, + }) async { + _ensureRecorderSubscription(); + return await methodChannel.invokeMethod('recorderStart', { + 'id': id, + 'path': path, + 'channels': channels, + 'sampleRate': sampleRate, + if (source != null) 'source': source, + if (left != null) 'left': left, + if (right != null) 'right': right, + }) ?? + false; + } + + @override + Future?> stopFileRecorder(String id, + {bool save = true}) async { + final raw = await methodChannel + .invokeMethod('recorderStop', {'id': id, 'save': save}); + return raw == null ? null : Map.from(raw); + } + @override Future startAiDownlink() async { return await methodChannel.invokeMethod('startAiDownlink') ?? false; @@ -163,6 +214,38 @@ class MethodChannelBluetoothManager extends BluetoothManagerPlatform { return await methodChannel.invokeMethod('stopCallDownlink') ?? false; } + @override + Future setCallPcmBridge(bool enabled) async { + return await methodChannel + .invokeMethod('setCallPcmBridge', {'enabled': enabled}) ?? + false; + } + + @override + Future setAiPcmBridge(bool enabled) async { + return await methodChannel + .invokeMethod('setAiPcmBridge', {'enabled': enabled}) ?? + false; + } + + @override + Future> getAiPcmBridgeStats() async { + final raw = await methodChannel.invokeMethod('getAiPcmBridgeStats'); + return raw == null ? const {} : Map.from(raw); + } + + @override + Future> getCallDownlinkStats() async { + final raw = await methodChannel.invokeMethod('getCallDownlinkStats'); + return raw == null ? const {} : Map.from(raw); + } + + @override + Future> getCallPcmBridgeStats() async { + final raw = await methodChannel.invokeMethod('getCallPcmBridgeStats'); + return raw == null ? const {} : Map.from(raw); + } + @override Future pushTtsPcm(String leg, Uint8List pcm) async { return await methodChannel.invokeMethod('pushTtsPcm', { diff --git a/apps/client/local_plugins/bluetooth_manager/lib/bluetooth_manager_platform_interface.dart b/apps/client/local_plugins/bluetooth_manager/lib/bluetooth_manager_platform_interface.dart index 22aa3565..4bb38af7 100644 --- a/apps/client/local_plugins/bluetooth_manager/lib/bluetooth_manager_platform_interface.dart +++ b/apps/client/local_plugins/bluetooth_manager/lib/bluetooth_manager_platform_interface.dart @@ -69,6 +69,63 @@ abstract class BluetoothManagerPlatform extends PlatformInterface { throw UnimplementedError('pushTtsPcm() has not been implemented.'); } + /// 通话翻译 PCM 原生直连桥:开着时 micData/spkData 不再进 [onDeviceAudioEvent], + /// 由原生直接交给 azure_speech;关掉即恢复。 + Future setCallPcmBridge(bool enabled) { + throw UnimplementedError('setCallPcmBridge() has not been implemented.'); + } + + /// EMAI 会话 PCM 原生直连桥:开着时 AI 模式的 micData 不再进 [onDeviceAiEvent], + /// 由原生直接交给 azure_speech 的百炼客户端;合成音频也直接回到 AI 下行发送器。 + Future setAiPcmBridge(bool enabled) { + throw UnimplementedError('setAiPcmBridge() has not been implemented.'); + } + + Future> getAiPcmBridgeStats() { + throw UnimplementedError('getAiPcmBridgeStats() has not been implemented.'); + } + + /// 下行发送器(译文回灌耳机)的实时统计。排查「切后台后远端听到的译文卡顿」 + /// 靠它:gapAvgMs 明显大于 intervalMs 就是 iOS 把 20ms 定时器节流了。 + Future> getCallDownlinkStats() { + throw UnimplementedError('getCallDownlinkStats() has not been implemented.'); + } + + Future> getCallPcmBridgeStats() { + throw UnimplementedError('getCallPcmBridgeStats() has not been implemented.'); + } + + /// 设备音频直接落盘:PCM 由原生写成 WAV,**不再逐帧过平台通道**。 + /// + /// [channels] = 1 时按 [source] 取一路(现场录音用 `mainMicData`); + /// = 2 时把 [left] / [right] 两路按样本对齐交织(通话录音用 + /// `micData` / `spkData`,左=本端、右=对端)。 + /// + /// 录音是额外的 sink,不影响 [onDeviceAudioEvent] 上原有的订阅方。 + /// 老原生没有这个方法时抛 [MissingPluginException],调用方据此退回 Dart 落盘。 + Future startFileRecorder({ + required String id, + required String path, + required int channels, + int sampleRate = 16000, + String? source, + String? left, + String? right, + }) { + throw UnimplementedError('startFileRecorder() has not been implemented.'); + } + + /// 停止并回填 WAV 头。返回 `{bytes, written, dropped}`; + /// 返回 null 表示这个 id 已经不在了(设备断连时原生会自己收尾)。 + Future?> stopFileRecorder(String id, {bool save = true}) { + throw UnimplementedError('stopFileRecorder() has not been implemented.'); + } + + /// 录音统计流:`{'id': String, 'bytes': int, 'level': double}`,约 100ms 一条 + Stream onRecorderEvent() { + throw UnimplementedError('onRecorderEvent() has not been implemented.'); + } + /// AI 实时对话事件流:{'type': 'startAI'|'stopAI'|'micData'|'speedAdjustment' /// |'deviceDisconnected', 'data': Uint8List} Stream onDeviceAiEvent() { diff --git a/apps/client/local_plugins/device_plugin_interface/lib/src/audio_format.dart b/apps/client/local_plugins/device_plugin_interface/lib/src/audio_format.dart index 8b50476f..c935435e 100644 --- a/apps/client/local_plugins/device_plugin_interface/lib/src/audio_format.dart +++ b/apps/client/local_plugins/device_plugin_interface/lib/src/audio_format.dart @@ -121,6 +121,43 @@ abstract class DeviceAudioSource { Future close(); } +/// 一次录音统计。原生每约 100ms 报一条。 +class DeviceRecordStats { + const DeviceRecordStats({required this.bytes, required this.level}); + + /// 已写入文件的音频字节数(不含 44 字节 WAV 头)。 + /// 时长按它算,与文件实际播放长度永远一致——不要用墙钟: + /// 恒玄收到开始指令后要 1.8~6.2s 才上行,还会把等待期缓存的音频补推。 + final int bytes; + + /// 本窗口的电平峰值 0~1,驱动波形 + final double level; +} + +/// **由原生直接落盘**的录音。PCM 不经过 Dart。 +/// +/// 与 [DeviceAudioSource] 的关系:那条是「把每一帧 PCM 交给 Dart」, +/// 适合要即时处理音频的场景(喂给识别、推给服务端);这条是「只要一个文件」, +/// 帧在原生解码完就地写盘,Dart 只收统计。录音属于后者——把每秒 50~100 帧 +/// 搬进 Dart 只为了写进文件,是纯粹的过路费。 +/// +/// 由 [DeviceSession.openFileRecorder] 创建;插件不支持时那里返回 null, +/// 调用方退回 [DeviceSession.openMic] + 自己写盘的老路径。 +abstract class DeviceFileRecorder { + /// 录到哪个文件 + String get path; + + /// 统计流(bytes / level),约 100ms 一条 + Stream get stats; + + /// 最近一次统计里的字节数。设备断连时原生会自己收尾, + /// 那种情况下 [stop] 拿不到统计,用这个兜底。 + int get bytesWritten; + + /// 停止。[save] = false 时删除文件(一帧都没收到那种)。 + Future stop({bool save = true}); +} + /// app → 设备的音频汇。写入的 PCM 必须满足 [format]。 /// /// 编码、节流、组包(比如恒玄 G.722 + 20ms 定时)一律在插件/原生里做, diff --git a/apps/client/local_plugins/device_plugin_interface/lib/src/device_session.dart b/apps/client/local_plugins/device_plugin_interface/lib/src/device_session.dart index ea21f486..1d2330ca 100644 --- a/apps/client/local_plugins/device_plugin_interface/lib/src/device_session.dart +++ b/apps/client/local_plugins/device_plugin_interface/lib/src/device_session.dart @@ -57,6 +57,28 @@ abstract class DeviceSession { Future openMic({AudioRoute route = AudioRoute.mic}) => throw DeviceException(DeviceErrorCode.notSupported, 'openMic($route)'); + /// 打开一个**由原生落盘**的录音器:设备上行的 PCM 在原生解码完直接写成 WAV, + /// 不再逐帧经平台通道进 Dart。 + /// + /// [routes] 一路 = 单声道;两路 = 交织成双声道,**第一路是左声道** + /// (通话录音的约定:左=本端麦、右=对端)。 + /// + /// 返回 null = 这个插件/这台设备不支持,调用方应退回 + /// [openMic] + 自己写盘。默认实现就返回 null,所以其它厂商零改动。 + /// 厂商自定义的诊断快照,**只用于排查、不参与任何业务判断**。 + /// + /// 默认返回空表:没实现的厂商零改动。恒玄这边返回的是通话翻译下行发送器 + /// 与 GATT 写入的实时计数(发包数、队列长度、定时器真实间隔),排查 + /// 「切后台后远端听到的译文卡顿」要靠它——那些数字原本只在原生 os_log 里, + /// 而 idevicesyslog 在 macOS 26+ 上极不稳定,经常整段抓不到。 + Future> diagnostics() async => const {}; + + Future openFileRecorder({ + required String path, + required List routes, + }) async => + null; + /// 打开一路下行音频。编码/节流/组包在插件里做。 Future openSpeaker({AudioRoute route = AudioRoute.ai}) => throw DeviceException( @@ -134,6 +156,22 @@ class DeviceFeatures { /// 参数:`{'state': DeviceAiState}`。设备不支持时抛 `device.not_supported`, /// **调用方要吞掉**——这只是锦上添花,不该影响对话本身。 static const String aiState = 'ai.state'; + + /// 通话翻译 PCM **原生直连桥**开关(app → 插件)。参数 `{'enabled': bool}`。 + /// + /// 开着时,设备双路通话音频在原生层直接交给翻译插件、译文 PCM 直接回到设备, + /// **不再经过 `openMic(callLocal/callPeer)` 与 `openSpeaker(callLeg*)` 的 Dart 流** + /// (那条路每秒约 200 次跨界,长句会把 event loop 拖住)。 + /// 返回 `{'enabled': bool}`;没有原生桥的厂商抛 `device.not_supported`, + /// 业务层据此退回 Dart 流那条路。 + static const String callPcmNativeBridge = 'call.pcm.nativebridge'; + + /// EMAI 会话 PCM **原生直连桥**开关(app → 插件)。参数 `{'enabled': bool}`。 + /// + /// 开着时,设备唤醒后的麦克风 PCM 在原生层直接交给百炼客户端、合成音频直接回到 + /// 设备,**不再经过 `openMic(ai)` 的 Dart 流**。返回 `{'enabled': bool}`; + /// 没有原生桥的厂商抛 `device.not_supported`,业务层据此退回 Dart 链路。 + static const String aiPcmNativeBridge = 'ai.pcm.nativebridge'; } /// [DeviceFeatures.aiState] 的取值。 diff --git a/apps/client/local_plugins/floating_ui_plugin/ios/Classes/NativePlugin.swift b/apps/client/local_plugins/floating_ui_plugin/ios/Classes/NativePlugin.swift index 1654105b..2529a98d 100644 --- a/apps/client/local_plugins/floating_ui_plugin/ios/Classes/NativePlugin.swift +++ b/apps/client/local_plugins/floating_ui_plugin/ios/Classes/NativePlugin.swift @@ -26,6 +26,34 @@ public class FloatingUiPlugin: NSObject, FlutterPlugin { } } + /// 字幕画中画专用的会话准备。 + /// + /// **画中画要求音频会话的类别支持后台音频**(`.playback` / `.playAndRecord` / `.multiRoute`)。 + /// 类别是 `.ambient` / `.soloAmbient` 时 `startPictureInPicture()` 静默失败—— + /// 连 `failedToStartPictureInPicture` 都不回调,表现就是"点了开关什么都没发生" + /// (2026-09-21 真机实测:category=ambient 时 start() 返回 true、isActive 恒 false)。 + /// + /// ⚠️ 但**不能无条件**改成 `.playback`:通话翻译 / 多媒体翻译开着小窗时会话是 + /// `.playAndRecord`,改过去就把正在跑的录音链路踩掉了(这正是上一版删掉 + /// updateAudioSession() 的理由)。而 `.playAndRecord` 本身就满足画中画的要求, + /// 所以只在"当前类别撑不起画中画"时才动它,并且带 mixWithOthers 不打断别的 App + /// (用户正在另一个 App 里看视频,这是本功能的使用场景)。 + private static func prepareAudioSessionForPip() { + let session = AVAudioSession.sharedInstance() + let pipCapable: Set = [.playback, .playAndRecord, .multiRoute] + if pipCapable.contains(session.category) { + NSLog("[SubtitlePip] 音频会话类别 %@ 已满足画中画要求,保持不动", session.category.rawValue) + return + } + do { + try session.setCategory(.playback, mode: .moviePlayback, options: [.mixWithOthers]) + try session.setActive(true) + NSLog("[SubtitlePip] 音频会话 %@ → playback(画中画要求支持后台音频的类别)", session.category.rawValue) + } catch { + NSLog("[SubtitlePip] 配置音频会话失败: %@", error.localizedDescription) + } + } + /** * Flutter 插件注册入口 * - Parameters: @@ -37,6 +65,14 @@ public class FloatingUiPlugin: NSObject, FlutterPlugin { let channel = FlutterMethodChannel(name: "native_plugin", binaryMessenger: registrar.messenger()) let instance = FloatingUiPlugin() registrar.addMethodCallDelegate(instance, channel: channel) + // 字幕小窗的状态回传(started / stopped / stoppedBySystem / failed),Dart 侧据此校正开关标志 + if #available(iOS 15.0, *) { + SubtitlePipController.shared.onStateChanged = { state in + DispatchQueue.main.async { + channel.invokeMethod("subtitlePipStateChanged", arguments: ["state": state]) + } + } + } let factory = FLNativeViewFactory(messenger: registrar.messenger()) registrar.register(factory, withId: "native_view") @@ -64,7 +100,9 @@ public class FloatingUiPlugin: NSObject, FlutterPlugin { // 小窗显示的就是它。 case "subtitlePipStart": if #available(iOS 15.0, *) { - Self.updateAudioSession() + // ⚠️ 只在当前类别撑不起画中画时才动音频会话,别踩正在跑的录音链路。 + // 见 prepareAudioSessionForPip 的说明。 + Self.prepareAudioSessionForPip() result(SubtitlePipController.shared.start()) } else { result(FlutterError(code: "UNSUPPORTED", message: "字幕小窗需要 iOS 15 及以上", details: nil)) @@ -95,6 +133,13 @@ public class FloatingUiPlugin: NSObject, FlutterPlugin { } result(nil) + case "subtitlePipDiagnostics": + if #available(iOS 15.0, *) { + result(SubtitlePipController.shared.diagnostics()) + } else { + result(["supported": false]) + } + case "subtitlePipIsActive": if #available(iOS 15.0, *) { result(SubtitlePipController.shared.isActive) diff --git a/apps/client/local_plugins/floating_ui_plugin/ios/Classes/SubtitlePipController.swift b/apps/client/local_plugins/floating_ui_plugin/ios/Classes/SubtitlePipController.swift index e346e138..26184798 100644 --- a/apps/client/local_plugins/floating_ui_plugin/ios/Classes/SubtitlePipController.swift +++ b/apps/client/local_plugins/floating_ui_plugin/ios/Classes/SubtitlePipController.swift @@ -50,6 +50,23 @@ final class SubtitlePipController: NSObject { private(set) var isActive = false + /// 状态回传给 Dart:"started" / "stopped"(我们主动停)/ "stoppedBySystem"(被系统或 + /// 别的 App 的画中画顶掉)/ "failed"。Dart 侧的开关标志以此为准——原来那个标志只在 + /// 进页面时查一次,之后永远是 stale:被顶掉后用户点开关走的是"关",要点两下才回来。 + var onStateChanged: ((String) -> Void)? + + /// 用户/业务是否**想要**小窗开着。[start] 置 true、[stop] 置 false。 + /// 被系统停掉时它仍为 true,据此区分"我们自己关的"和"被顶掉的",也是自愈重试的开关。 + private var wanted = false + + /// 自愈重试:iOS 全系统同时只允许一个画中画,视频 App 一开自己的小窗我们的就被停掉; + /// 之后没人再拉起它,"弹幕框就丢了"。这里在被顶掉/启动失败后退避重试(3/6/12/12/12 s), + /// 并在 App 下一次将要退到后台(willResignActive)时再拉一次。 + private var retryAttempts = 0 + private static let maxRetryAttempts = 5 + private var retryWork: DispatchWorkItem? + private var lifecycleObservers: [NSObjectProtocol] = [] + /// 小窗上的暂停按钮。 /// /// 语义定成「**冻结当前这条字幕**」而不是「暂停翻译」:用户在别的 App 里 @@ -64,6 +81,47 @@ final class SubtitlePipController: NSObject { private override init() { super.init() card.setFillsBounds(true) + let nc = NotificationCenter.default + // 将要退到后台:还想要小窗但它没在 → 立刻拉一次。这是"第二次切走"的兜底: + // 系统只在从前台切后台那一刻自动拉起(canStartPictureInPictureAutomaticallyFromInline), + // 那一刻被别的 App 的画中画挡住就没有下一次了。 + lifecycleObservers.append(nc.addObserver( + forName: UIApplication.willResignActiveNotification, object: nil, queue: .main + ) { [weak self] _ in + guard let self = self, self.wanted, !self.isActive, let pip = self.pip else { return } + NSLog("[SubtitlePip] App 将退到后台且小窗未在,重新拉起") + self.renderAndEnqueue(force: true) + pip.startPictureInPicture() + }) + // 回到前台:重试预算给满 + lifecycleObservers.append(nc.addObserver( + forName: UIApplication.didBecomeActiveNotification, object: nil, queue: .main + ) { [weak self] _ in + self?.retryAttempts = 0 + }) + } + + private func notify(_ state: String) { + onStateChanged?(state) + } + + private func scheduleRetry(reason: String) { + guard wanted else { return } + guard retryAttempts < Self.maxRetryAttempts else { + NSLog("[SubtitlePip] 重试 %d 次仍拉不起小窗(%@),等下次退后台再试", Self.maxRetryAttempts, reason) + return + } + retryWork?.cancel() + let delay = min(12.0, 3.0 * pow(2.0, Double(retryAttempts))) + retryAttempts += 1 + NSLog("[SubtitlePip] %@,%.0fs 后第 %d 次重试", reason, delay, retryAttempts) + let work = DispatchWorkItem { [weak self] in + guard let self = self, self.wanted, !self.isActive, let pip = self.pip else { return } + self.renderAndEnqueue(force: true) + pip.startPictureInPicture() + } + retryWork = work + DispatchQueue.main.asyncAfter(deadline: .now() + delay, execute: work) } // MARK: - 生命周期 @@ -74,6 +132,10 @@ final class SubtitlePipController: NSObject { NSLog("[SubtitlePip] 本机不支持画中画") return false } + wanted = true + retryAttempts = 0 + retryWork?.cancel() + retryWork = nil attachRenderViewIfNeeded() setupTimebaseIfNeeded() // 先推一帧再 start:AVPictureInPictureController 在没有任何已入队帧时 @@ -114,11 +176,41 @@ final class SubtitlePipController: NSObject { } func stop() { + wanted = false + retryWork?.cancel() + retryWork = nil heartbeat?.invalidate() heartbeat = nil pip?.stopPictureInPicture() renderView.removeFromSuperview() isActive = false + notify("stopped") + } + + /// 诊断:画中画起不来时问它。 + /// + /// 这个功能的失败**大多是静默的**——`startPictureInPicture()` 不返回值, + /// 前置条件不满足时连 `failedToStartPictureInPicture` 都不回调(2026-09-21 真机实测)。 + /// 所以把系统的判据原样吐出来,别再靠猜。 + func diagnostics() -> [String: Any] { + let session = AVAudioSession.sharedInstance() + return [ + "supported": AVPictureInPictureController.isPictureInPictureSupported(), + "possible": pip?.isPictureInPicturePossible ?? false, + "active": pip?.isPictureInPictureActive ?? false, + "isActiveFlag": isActive, + "wanted": wanted, + "hasController": pip != nil, + "viewInWindow": renderView.window != nil, + "viewSize": "\(renderView.bounds.width)x\(renderView.bounds.height)", + "layerStatus": renderView.displayLayer.status.rawValue, + "layerReady": renderView.displayLayer.isReadyForMoreMediaData, + "timebaseRate": timebase.map { CMTimebaseGetRate($0) } ?? -1, + "audioCategory": session.category.rawValue, + "audioOtherPlaying": session.isOtherAudioPlaying, + "appState": UIApplication.shared.applicationState.rawValue, + "retryAttempts": retryAttempts, + ] } // MARK: - 内容与样式 @@ -268,8 +360,12 @@ extension SubtitlePipController: AVPictureInPictureControllerDelegate { func pictureInPictureControllerDidStartPictureInPicture( _ pictureInPictureController: AVPictureInPictureController) { isActive = true + retryAttempts = 0 + retryWork?.cancel() + retryWork = nil NSLog("[SubtitlePip] 已进入画中画") renderAndEnqueue(force: true) + notify("started") } func pictureInPictureController( @@ -277,12 +373,22 @@ extension SubtitlePipController: AVPictureInPictureControllerDelegate { failedToStartPictureInPictureWithError error: Error) { isActive = false NSLog("[SubtitlePip] 进入画中画失败: %@", error.localizedDescription) + notify("failed") + // 常见原因:别的 App 的画中画正占着。退避后再试,并等下次退后台再拉。 + scheduleRetry(reason: "启动失败") } func pictureInPictureControllerDidStopPictureInPicture( _ pictureInPictureController: AVPictureInPictureController) { isActive = false - NSLog("[SubtitlePip] 已退出画中画") + if wanted { + // 不是我们停的:被系统/别的 App 的画中画顶掉了。热心跳继续跑着,layer 不会饿死。 + NSLog("[SubtitlePip] 画中画被系统停掉(非本 App 主动)") + notify("stoppedBySystem") + scheduleRetry(reason: "被系统停掉") + } else { + NSLog("[SubtitlePip] 已退出画中画") + } } } diff --git a/apps/client/local_plugins/floating_ui_plugin/lib/native_plugin.dart b/apps/client/local_plugins/floating_ui_plugin/lib/native_plugin.dart index 50c04d2e..3749f14a 100644 --- a/apps/client/local_plugins/floating_ui_plugin/lib/native_plugin.dart +++ b/apps/client/local_plugins/floating_ui_plugin/lib/native_plugin.dart @@ -90,8 +90,28 @@ class SubtitlePip { static const MethodChannel _channel = MethodChannel('native_plugin'); + /// 原生状态回传:`started` / `stopped`(我们主动停)/ `stoppedBySystem` + /// (被系统或别的 App 的画中画顶掉)/ `failed`。 + /// iOS 全系统同时只允许一个画中画,被顶掉是常态,界面上的开关标志必须以这里为准。 + static void Function(String state)? onStateChanged; + static bool _handlerBound = false; + + static void _bindHandler() { + if (_handlerBound) return; + _handlerBound = true; + _channel.setMethodCallHandler((call) async { + if (call.method == 'subtitlePipStateChanged') { + final args = call.arguments; + final state = args is Map ? (args['state']?.toString() ?? '') : ''; + if (state.isNotEmpty) onStateChanged?.call(state); + } + return null; + }); + } + /// 进入画中画。返回 false 表示本机不支持或启动被系统拒绝。 static Future start() async { + _bindHandler(); try { return (await _channel.invokeMethod('subtitlePipStart')) ?? false; } catch (_) { @@ -131,7 +151,20 @@ class SubtitlePip { } catch (_) {} } + /// 诊断:画中画起不来时问它(见原生 SubtitlePipController.diagnostics)。 + /// 这个功能失败大多是静默的,别靠猜。 + static Future> diagnostics() async { + try { + final raw = + await _channel.invokeMethod('subtitlePipDiagnostics'); + return raw == null ? const {} : Map.from(raw); + } catch (e) { + return {'error': '$e'}; + } + } + static Future isActive() async { + _bindHandler(); try { return (await _channel.invokeMethod('subtitlePipIsActive')) ?? false; } catch (_) { diff --git a/apps/client/pubspec.yaml b/apps/client/pubspec.yaml index cce0a439..d5701570 100644 --- a/apps/client/pubspec.yaml +++ b/apps/client/pubspec.yaml @@ -1,7 +1,7 @@ name: eaimar description: "Eaimar - AI Voice Assistant." publish_to: "none" -version: 1.0.1+15 +version: 1.0.1+16 environment: sdk: ">=3.3.0 <4.0.0" @@ -183,6 +183,7 @@ flutter: - assets/fonts/NotoSansSC-Regular.ttf - assets/lottie/ - assets/js/ + - assets/audio/silence_1s.wav - assets/images/defaultPicture.png - assets/images/defaultPicture.jpg # 设备品类实拍图(「添加设备 → 选择设备」页和设备页展示用) diff --git a/apps/client/test/bailian_native_parity_test.dart b/apps/client/test/bailian_native_parity_test.dart new file mode 100644 index 00000000..be2202f8 --- /dev/null +++ b/apps/client/test/bailian_native_parity_test.dart @@ -0,0 +1,46 @@ +import 'package:eaimar/data/services/bailian_multimodal_service.dart'; +import 'package:flutter_test/flutter_test.dart'; + +/// Dart 版与原生版百炼客户端之间的**共用契约**。 +/// +/// 耳机唤醒走原生(`BailianDialogClient.swift/.kt`),手机麦按住说话走 Dart +/// ([BailianMultimodalService])。两条路必须连到同一个百炼应用、带同一套 +/// MCP 工具名——对不上不会报任何错,只表现为「用耳机问和在 App 里问, +/// 助手的能力不一样」。 +/// +/// 原生那份的配置是**从 Dart 传下去**的(apiKey / appId / workspaceId / +/// mcpAuthTools 都走 start 的参数),所以只要这几个解析入口是共用的, +/// 两边就不会漂。这个测试钉的就是「它们确实是共用的、且是 public」。 +void main() { + test('配置解析是 static 且两条链路共用', () { + // 这三个函数是原生客户端唯一的取值来源(见 BailianNativeDialog.start)。 + // 改成实例私有 getter 会让原生那边不得不再抄一份解析逻辑。 + expect(BailianMultimodalService.resolveApiKey, isA()); + expect(BailianMultimodalService.resolveAppId, isA()); + expect(BailianMultimodalService.resolveWorkspaceId, isA()); + }); + + test('覆盖为空时回退到默认 app/workspace,不会变成空串', () { + // 空串会让握手带着空 app_id 发出去,服务端只回一句含糊的 task-failed + expect(BailianMultimodalService.resolveAppId('').isNotEmpty, isTrue); + expect(BailianMultimodalService.resolveAppId(' ').isNotEmpty, isTrue); + expect(BailianMultimodalService.resolveWorkspaceId('').isNotEmpty, isTrue); + expect(BailianMultimodalService.resolveWorkspaceId(' ').isNotEmpty, isTrue); + }); + + test('显式覆盖优先于全局配置', () { + // 两个 agent(EMAI / Smartcar 车载助手)用的是百炼里不同的应用, + // 覆盖失效会让车载香薰唤醒到 EMAI 上去 + expect(BailianMultimodalService.resolveAppId('app-x'), 'app-x'); + expect(BailianMultimodalService.resolveWorkspaceId('ws-x'), 'ws-x'); + }); + + test('MCP 工具名只有一份,且非空', () { + // 原生不再抄一份:服务端新增需要身份的工具时只改这里。 + // 漏了那个工具永远「身份校验失败」,而且客户端不报任何错。 + const tools = BailianMultimodalService.mcpAuthTools; + expect(tools, isNotEmpty); + expect(tools.toSet().length, tools.length, reason: '工具名不能重复'); + expect(tools.every((t) => t.trim().isNotEmpty), isTrue); + }); +} diff --git a/apps/client/test/devices/bes_recorder_sources_test.dart b/apps/client/test/devices/bes_recorder_sources_test.dart new file mode 100644 index 00000000..0ecb8b42 --- /dev/null +++ b/apps/client/test/devices/bes_recorder_sources_test.dart @@ -0,0 +1,33 @@ +import 'package:device_plugin_interface/device_plugin_interface.dart'; +import 'package:eaimar/devices/bes/bes_device_plugin.dart'; +import 'package:flutter_test/flutter_test.dart'; + +/// AudioRoute → 原生音频类型串,**跨语言契约**。 +/// +/// 这些串由原生 `BluetoothManager` 在解码后随帧发出 +/// (iOS `publishCallTranslation` / Android `publishTranslationData`), +/// `PcmRecorderRegistry` 按它们把帧投递给对应的录制器。三处(Dart / Swift / +/// Kotlin)写的必须是同一个串——写错不会报任何错,只表现为「录音起来了、 +/// 统计恒为 0、文件里一个字节都没有」。 +void main() { + test('录音路由到原生音频类型串逐条固定', () { + expect(BesDeviceSession.recorderSources, { + AudioRoute.mic: 'mainMicData', + AudioRoute.callLocal: 'micData', + AudioRoute.callPeer: 'spkData', + }); + }); + + test('现场录音取主麦而不是前馈麦', () { + // 裸流隔帧取(openMic 那条 _alternate)靠的是"订阅时恰好对上相位", + // 中途开始订阅就会永久取到前馈麦。原生早就把两路标好了,按类型取 + // 顺带把这类错相消掉——所以这一条只能是 mainMicData,不能是 ffMicData。 + expect(BesDeviceSession.recorderSources[AudioRoute.mic], 'mainMicData'); + }); + + test('通话左声道是本端、右声道是对端', () { + // 反了的话录出来的文件左右互换,转写时"谁说的"会整体对调 + expect(BesDeviceSession.recorderSources[AudioRoute.callLocal], 'micData'); + expect(BesDeviceSession.recorderSources[AudioRoute.callPeer], 'spkData'); + }); +}