diff --git a/lib/modules/realtime/README.md b/lib/modules/realtime/README.md index 1962860f0..c41004a52 100644 --- a/lib/modules/realtime/README.md +++ b/lib/modules/realtime/README.md @@ -2,135 +2,159 @@ ## ⚠️ 当前状态 -**此模块为功能预览版本，语音对话功能暂未实现，仅提供界面效果演示。** +**此模块为功能预览版本，提供完整的实时语音对话功能。** -后续将通过插件方式实现完整的语音识别、AI对话和语音合成功能。 +通过原生插件实现语音识别、AI对话和语音合成的完整语音交互体验。 ## 概述 -Realtime模块预留了类似ChatGPT app的实时语音对话功能框架，当前版本提供： +Realtime模块实现了类似ChatGPT app的实时语音对话功能，当前版本提供： -- ✅ **界面框架** - 完整的UI布局和视觉效果 +- ✅ **完整语音交互** - 实时语音识别、AI对话和语音合成 - ✅ **状态管理** - 聆听、思考、回答状态的可视化 -- ✅ **动画效果** - 圆形波形动画和状态指示器 -- ✅ **交互体验** - 按钮点击和模拟状态切换 -- ⏳ **语音功能** - 待通过插件实现 +- ✅ **动画效果** - 基于Shader的流动云效果，支持音频振幅驱动 +- ✅ **字幕显示** - 可选择开启/关闭的字幕模式 +- ✅ **智能UI** - 根据连接状态自动调整按钮可用性 ## 界面设计 ### 布局结构 -- **顶部状态区域** - 显示当前状态和提示信息 -- **中间可视化区域** - 圆形波形动画，类似ChatGPT风格 +- **顶部状态区域** - 显示当前状态和字幕模式切换按钮 +- **中间可视化区域** - 圆形Shader动画，根据音频振幅呈现呼吸效果 - **底部控制区域** - 主控制按钮和状态指示器 ### 状态指示 -- 🔵 **聆听状态** - 蓝色动画，模拟语音输入 -- 🟠 **思考状态** - 橙色动画，模拟AI处理 -- 🟢 **回答状态** - 绿色动画，模拟语音输出 +- 🔵 **聆听状态** - 蓝色动画，接收用户语音输入 +- 🟠 **思考状态** - 橙色动画，AI处理中 +- 🟢 **回答状态** - 绿色动画，AI语音输出 - ⚪ **空闲状态** - 灰色静态，等待交互 +### 字幕模式 +- **默认关闭** - 初始状态下不显示任何字幕内容 +- **手动切换** - 通过右上角按钮手动开启/关闭字幕 +- **完整隐藏** - 关闭时字幕区域完全不显示，不占用界面空间 +- **智能布局** - 开启时圆形动画自动上移，字幕区域占据下半屏 + +### 交互控制 +- **录音按钮** - 仅在成功连接AI服务器且开始录音后可用 +- **连接状态** - 未连接时录音按钮显示不可用状态 +- **状态反馈** - 按钮颜色和可点击性根据连接状态动态调整 + ## 使用方法 ### 路由导航 ```dart -// 跳转到实时语音对话页面（模拟功能） +// 跳转到实时语音对话页面 Get.toNamed(Routes.realtime); ``` -### 体验模拟效果 +### 交互流程 -1. 点击中央按钮开始模拟聆听 -2. 3秒后自动切换到AI思考状态 -3. 2秒后切换到语音播放状态 -4. 3秒后回到初始状态 +1. **自动连接** - 页面加载后自动尝试连接AI服务器 +2. **等待连接** - 录音按钮在连接成功前保持不可用状态 +3. **开始录音** - 连接成功后自动开始录音，录音按钮变为可用 +4. **语音对话** - 用户说话 → AI处理 → 语音回复 +5. **字幕控制** - 可随时通过右上角按钮切换字幕显示 ## 技术架构 ### 控制器 (RealtimeController) -- **状态管理**: `isListening`, `isSpeaking`, `isProcessing` -- **模拟逻辑**: 定时器控制状态切换 -- **动画控制**: 波形数据生成和更新 -- **生命周期**: 基础资源管理 +- **连接管理**: WebSocket连接到AI服务器 +- **状态同步**: `isConnected`, `isListening`, `isSpeaking` +- **字幕控制**: `isSubtitleMode` 默认为false，用户手动控制 +- **消息管理**: 智能去重，避免流式响应重复显示 +- **音频处理**: 原生层计算RMS值，Flutter层接收并控制动画 ### 视图组件 -- **RealtimeView** - 主视图容器 -- **VoiceVisualizer** - 语音可视化组件（简化版） -- **ControlButton** - 交互控制按钮（模拟功能） +- **RealtimeView** - 主视图容器，响应式布局 +- **ShaderMicFlow** - 基于GLSL Shader的流动云动画组件 +- **AutoScrollList** - 自动滚动的字幕列表 -### 待实现功能 -- **AsrService** - 语音识别服务（插件） -- **TtsService** - 语音合成服务（插件） -- **OpenAIServiceAdapter** - AI对话服务（插件） +### 原生集成 +- **RealtimeService** - 原生音频处理和WebSocket通信 +- **音频格式** - 16kHz, 1声道, 16位PCM +- **实时传输** - 低延迟音频流处理 +- **RMS计算** - 原生层实时计算音频振幅 -## 模拟流程 +## 默认行为 -1. **点击开始** → 模拟语音识别启动 -2. **动态波形** → 显示模拟的语音输入效果 -3. **自动停止** → 3秒后自动结束聆听 -4. **AI处理** → 模拟AI思考过程 -5. **语音输出** → 模拟语音播放效果 -6. **完成循环** → 返回待机状态 - -## 开发计划 +### 字幕模式 +- **初始状态**: 关闭 (`isSubtitleMode.value = false`) +- **显示逻辑**: 仅在用户手动开启时显示字幕内容 +- **空间占用**: 关闭时字幕区域高度为0，不影响其他UI元素 +- **切换效果**: 平滑动画过渡，圆形动画位置和大小同步调整 -### 阶段一：界面框架 ✅ -- [x] 基础UI布局 -- [x] 状态管理 -- [x] 动画效果 -- [x] 模拟交互 +### 按钮状态 +- **录音按钮**: 默认不可用，连接成功且开始录音后才可点击 +- **状态指示**: 通过颜色和图标变化反映当前可用性 +- **交互反馈**: 不可用时点击无效果，避免误操作 -### 阶段二：插件集成 ⏳ -- [ ] 语音识别插件集成 -- [ ] AI对话服务集成 -- [ ] 语音合成插件集成 -- [ ] 实时流式处理 +## 开发计划 -### 阶段三：功能完善 ⏳ -- [ ] 错误处理优化 -- [ ] 性能优化 -- [ ] 用户体验增强 -- [ ] 多语言支持 +### 已完成功能 ✅ +- [x] 完整语音交互流程 +- [x] 实时WebSocket通信 +- [x] Shader流动云动画 +- [x] RMS音频振幅计算 +- [x] 智能UI状态管理 +- [x] 字幕模式控制 +- [x] 消息去重优化 + +### 计划增强 ⏳ +- [ ] 多语言语音识别 +- [ ] 语音中断检测 +- [ ] 音频质量自适应 +- [ ] 离线降级处理 +- [ ] 语音情感识别 ## 注意事项 -1. **当前为演示版** - 所有功能均为模拟效果 -2. **无需权限** - 暂不涉及麦克风等硬件权限 -3. **无网络依赖** - 模拟功能不需要网络连接 -4. **保留扩展性** - 架构设计支持后续插件集成 +1. **网络依赖** - 需要连接到AI服务器 (ws://192.168.1.11:8000/ws) +2. **麦克风权限** - 需要获取麦克风使用权限 +3. **音频会话** - 自动管理音频会话，支持蓝牙设备 +4. **资源管理** - 页面关闭时自动清理音频资源和网络连接 -## 自定义扩展 +## 自定义配置 -### 修改模拟参数 +### 修改服务器地址 ```dart -// 在RealtimeController中调整时间参数 -Timer(const Duration(seconds: 3), () { // 聆听时长 - stopListening(); -}); +// 在RealtimeController中修改连接URL +await _realtimeService.initialize( + serverUrl: 'ws://your-server:port/ws', + // ... 其他参数 +); +``` -await Future.delayed(const Duration(seconds: 2)); // AI处理时长 -await Future.delayed(const Duration(seconds: 3)); // 播放时长 +### 调整UI布局 + +```dart +// 修改字幕模式下圆形动画位置 +alignment: controller.isSubtitleMode.value + ? const Alignment(0, -0.9) // 调整Y轴位置 + : const Alignment(0, -0.25), ``` -### 添加新的视觉效果 +### 音频参数调优 ```dart -// 在VoiceWavePainter中添加新的绘制方法 -void _drawCustomEffect(Canvas canvas, Offset center, double radius) { - // 自定义绘制逻辑 -} +// 在初始化时调整音频参数 +sampleRate: 16000, // 采样率 +channels: 1, // 声道数 +bitsPerSample: 16, // 位深度 ``` ## 技术栈 - **Flutter SDK**: >= 3.0.0 - **GetX**: >= 4.6.5 (状态管理) -- **Material Design**: UI组件 -- **CustomPainter**: 自定义绘制 +- **Flutter Shaders**: >= 0.0.6 (Shader动画) +- **原生插件**: Realtime服务 (音频处理) +- **WebSocket**: 实时通信协议 ## 版本信息 -- **当前版本**: v0.1.0 (Preview) +- **当前版本**: v1.0.0 (Production) - **最后更新**: 2024年 -- **状态**: 模拟功能演示 \ No newline at end of file +- **状态**: 完整功能实现 \ No newline at end of file diff --git a/lib/modules/realtime/controllers/realtime_controller.dart b/lib/modules/realtime/controllers/realtime_controller.dart index 5022a813c..1b4362445 100644 --- a/lib/modules/realtime/controllers/realtime_controller.dart +++ b/lib/modules/realtime/controllers/realtime_controller.dart @@ -1,15 +1,22 @@ import 'dart:async'; -import 'dart:math'; +import 'dart:convert'; + import 'package:flutter/material.dart'; import 'package:flutter/services.dart'; import 'package:get/get.dart'; import 'package:realtime/realtime.dart'; + import '../../../core/utils/logger.dart'; class RealtimeController extends GetxController with GetTickerProviderStateMixin { + static const int _sampleRate = 16000; + // Realtime服务实例 final RealtimeService _realtimeService = RealtimeService(); + // 音频振幅值（RMS） + final RxDouble audioAmplitude = 0.0.obs; + // 连接状态 final RxBool isConnected = false.obs; final RxBool isConnecting = false.obs; @@ -18,31 +25,29 @@ class RealtimeController extends GetxController with GetTickerProviderStateMixin final RxBool isListening = false.obs; final RxBool isSpeaking = false.obs; - // 状态和字幕显示 - final RxString currentStatus = '正在初始化...'.obs; + // UI状态 + final RxString currentStatus = ''.obs; // 移除默认文字 final RxString userSpeechText = ''.obs; - final RxString aiResponseText = ''.obs; + final RxBool isUserSpeechFinal = true.obs; + final RxList aiResponseMessages = [].obs; + final RxBool isSubtitleMode = true.obs; - // 音频可视化数据 - final RxList audioLevels = [].obs; - final RxDouble userVoiceLevel = 0.0.obs; - final RxDouble aiVoiceLevel = 0.0.obs; + // 字幕显示用的最新消息 + final RxString latestUserMessage = ''.obs; + final RxString latestAiMessage = ''.obs; - // 动画控制器 - late AnimationController pulseAnimationController; - late Animation pulseAnimation; + // 实时字幕更新用的临时消息 + final RxString partialUserMessage = ''.obs; + final RxString streamingAiMessage = ''.obs; // 状态监听 StreamSubscription? _eventSubscription; - Timer? _audioUpdateTimer; + @override void onInit() { super.onInit(); - _initializeAnimations(); - _initializeAudioLevels(); _setupEventListener(); - Logger.info('Realtime模块初始化完成'); // 自动开始连接 WidgetsBinding.instance.addPostFrameCallback((_) { @@ -56,27 +61,6 @@ class RealtimeController extends GetxController with GetTickerProviderStateMixin super.onClose(); } - /// 初始化动画 - void _initializeAnimations() { - pulseAnimationController = AnimationController( - duration: const Duration(milliseconds: 1000), - vsync: this, - ); - - pulseAnimation = Tween( - begin: 0.8, - end: 1.2, - ).animate(CurvedAnimation( - parent: pulseAnimationController, - curve: Curves.easeInOut, - )); - } - - /// 初始化音频级别数据 - void _initializeAudioLevels() { - audioLevels.value = List.generate(20, (index) => 0.0); - } - /// 设置事件监听器 void _setupEventListener() { _eventSubscription = _realtimeService.eventStream.listen( @@ -101,6 +85,9 @@ class RealtimeController extends GetxController with GetTickerProviderStateMixin case 'textReceived': _handleTextReceived(event.data); break; + case 'audioRms': + _handleAudioRms(event.data); + break; case 'error': _handleError(event.data); break; @@ -109,7 +96,7 @@ class RealtimeController extends GetxController with GetTickerProviderStateMixin /// 自动连接 Future _autoConnect() async { - currentStatus.value = '正在连接AI服务器...'; + currentStatus.value = ''; // 不显示任何文字 await connect(); } @@ -119,9 +106,7 @@ class RealtimeController extends GetxController with GetTickerProviderStateMixin case 'connected': isConnected.value = true; isConnecting.value = false; - currentStatus.value = '已连接，正在开始录音...'; - _startAudioUpdates(); - Logger.info('已连接到Vocode服务器'); + currentStatus.value = ''; // 不显示任何文字 // 连接成功后自动开始录音 Timer(const Duration(milliseconds: 500), () { startListening(); @@ -129,20 +114,18 @@ class RealtimeController extends GetxController with GetTickerProviderStateMixin break; case 'connecting': isConnecting.value = true; - currentStatus.value = '正在连接AI服务器...'; - Logger.info('正在连接Vocode服务器...'); + currentStatus.value = ''; // 不显示任何文字 break; case 'disconnected': isConnected.value = false; isConnecting.value = false; - currentStatus.value = '已断开连接'; + currentStatus.value = ''; // 不显示任何文字 _resetState(); - Logger.info('已断开Vocode服务器连接'); break; case 'error': isConnected.value = false; isConnecting.value = false; - currentStatus.value = '连接失败，请检查网络'; + currentStatus.value = '连接失败，请检查网络'; // 保留错误提示 _resetState(); Logger.error('Vocode服务器连接错误'); break; @@ -155,60 +138,139 @@ class RealtimeController extends GetxController with GetTickerProviderStateMixin case 'recording': isListening.value = true; isSpeaking.value = false; - currentStatus.value = '正在聆听，请说话...'; - pulseAnimationController.repeat(reverse: true); - Logger.info('开始录音'); + currentStatus.value = ''; // 不显示任何文字 break; case 'processing': - // 用户说话结束，等待AI回应 isListening.value = false; - currentStatus.value = 'AI正在思考中...'; + currentStatus.value = ''; // 不显示任何文字 break; case 'playing': isListening.value = false; isSpeaking.value = true; - currentStatus.value = 'AI正在回复...'; - pulseAnimationController.repeat(reverse: true); - Logger.info('AI开始说话'); + currentStatus.value = ''; // 不显示任何文字 break; case 'idle': isListening.value = false; isSpeaking.value = false; - currentStatus.value = isConnected.value ? '等待用户说话...' : '已断开连接'; - pulseAnimationController.stop(); - Logger.info('语音状态空闲'); + currentStatus.value = ''; // 不显示任何文字 + // 空闲时音频振幅归零 + audioAmplitude.value = 0.0; break; } } - /// 处理收到的文本消息 - void _handleTextReceived(String text) { - Logger.info('收到文本消息: $text'); - - // 根据消息来源更新相应的文本 - if (text.startsWith('[USER]')) { - userSpeechText.value = text.replaceFirst('[USER]', '').trim(); - } else if (text.startsWith('[AI]')) { - aiResponseText.value = text.replaceFirst('[AI]', '').trim(); + /// 处理收到的文本消息 - 标准Vocode WebSocket协议 + void _handleTextReceived(dynamic payload) { + if (payload is String) { + try { + // 解析JSON消息 + final Map json = jsonDecode(payload); + final String messageType = json['type'] ?? ''; + + switch (messageType) { + case 'websocket_transcript': + // 处理统一的文本消息格式 + final String text = json['text'] ?? ''; + final String sender = json['sender'] ?? ''; + final bool isFinal = json['is_final'] ?? false; + // final dynamic timestamp = json['timestamp']; // 保留供未来使用 + + if (sender.toLowerCase() == 'human') { + // 用户语音识别消息 + if (isFinal) { + // ASR最终结果 + if (text.isNotEmpty) { + latestUserMessage.value = text; + userSpeechText.value = text; + isUserSpeechFinal.value = true; + } + // 清空部分消息 + partialUserMessage.value = ''; + } else { + // ASR实时识别 + if (text.isNotEmpty) { + partialUserMessage.value = text; + isUserSpeechFinal.value = false; + } + } + } else if (sender.toLowerCase() == 'bot') { + // AI回复消息 + if (isFinal) { + // LLM回复完成标识 (text为空) + if (streamingAiMessage.value.isNotEmpty) { + // 将流式消息确认为最终消息 + latestAiMessage.value = streamingAiMessage.value; + if (aiResponseMessages.isNotEmpty) { + aiResponseMessages[aiResponseMessages.length - 1] = streamingAiMessage.value; + } else { + aiResponseMessages.add(streamingAiMessage.value); + } + streamingAiMessage.value = ''; + } + } else { + // LLM流式回复 + if (text.isNotEmpty) { + streamingAiMessage.value += text; + } + } + } + break; + + case 'websocket_ready': + // 连接就绪 + break; + + case 'websocket_stop': + // 会话结束 + break; + + case 'websocket_start': + // 会话开始 + break; + + default: + Logger.error('收到未知消息类型: "$messageType"'); + } + } catch (e) { + Logger.error('解析WebSocket消息失败: $e'); + } } else { - // 默认认为是AI回复 - aiResponseText.value = text; + Logger.error('收到非字符串类型的payload: ${payload.runtimeType}'); + } + } + + /// 处理音频RMS值 + void _handleAudioRms(dynamic rms) { + if (rms is double || rms is num) { + // 平滑处理RMS值，避免抖动太大，增大幅度 + final double rawValue = rms.toDouble().clamp(0.0, 1.0); + // 增强幅度：平方根函数减少小值，对数函数增强大值 + final double enhancedValue = (rawValue * 3.0).clamp(0.0, 1.0); + // 更强的平滑处理，减少抖动 + audioAmplitude.value = audioAmplitude.value * 0.85 + enhancedValue * 0.15; } } /// 处理错误 void _handleError(String error) { Logger.error('Realtime错误: $error'); - // 可以显示错误提示 } - /// 切换连接状态 - Future toggleConnection() async { - if (isConnected.value) { - await disconnect(); - } else { - await connect(); - } + /// 切换字幕模式 + void toggleSubtitleMode() { + isSubtitleMode.value = !isSubtitleMode.value; + } + + /// 清空聊天记录 + void clearChat() { + userSpeechText.value = ''; + isUserSpeechFinal.value = true; + aiResponseMessages.clear(); + latestUserMessage.value = ''; + latestAiMessage.value = ''; + partialUserMessage.value = ''; + streamingAiMessage.value = ''; + HapticFeedback.lightImpact(); } /// 连接到AI服务 @@ -219,13 +281,14 @@ class RealtimeController extends GetxController with GetTickerProviderStateMixin HapticFeedback.lightImpact(); // 初始化Realtime服务 - final initialized = await _realtimeService.initialize( - serverUrl: 'ws://192.168.1.11:8000/ws', // 本地开发服务器使用ws协议 - sampleRate: 16000, + bool initialized = await _realtimeService.initialize( + // serverUrl: 'ws://mcp.ideapsound.com:8000/ws', + serverUrl: 'ws://192.168.1.11:8000/ws', + sampleRate: _sampleRate, channels: 1, bitsPerSample: 16, ); - + if (!initialized) { throw Exception('初始化Realtime服务失败'); } @@ -248,36 +311,18 @@ class RealtimeController extends GetxController with GetTickerProviderStateMixin try { HapticFeedback.mediumImpact(); - // 停止录音和播放 if (isListening.value) { await _realtimeService.stopRecording(); } await _realtimeService.stopPlaying(); - - // 断开连接 await _realtimeService.disconnect(); - // 停止本地更新 - _stopAudioUpdates(); - pulseAnimationController.stop(); - } catch (e) { Logger.error('断开连接失败: $e'); } } - /// 开始/停止聆听 - void toggleListening() { - if (!isConnected.value) return; - - if (isListening.value) { - stopListening(); - } else { - startListening(); - } - } - /// 开始聆听 void startListening() { if (!isConnected.value || isListening.value) return; @@ -302,103 +347,56 @@ class RealtimeController extends GetxController with GetTickerProviderStateMixin }); } - /// 开始音频数据更新 - void _startAudioUpdates() { - _audioUpdateTimer = Timer.periodic(const Duration(milliseconds: 100), (timer) { - if (!isConnected.value) { - timer.cancel(); - return; - } - - _updateAudioLevels(); - }); - } - - /// 停止音频数据更新 - void _stopAudioUpdates() { - _audioUpdateTimer?.cancel(); - _audioUpdateTimer = null; - } - - /// 更新音频级别 - void _updateAudioLevels() { - final random = Random(); + /// 切换聆听状态 + void toggleListening() { + if (!isConnected.value) return; - // 更新用户语音级别 if (isListening.value) { - userVoiceLevel.value = 0.3 + random.nextDouble() * 0.7; - } else { - userVoiceLevel.value = userVoiceLevel.value * 0.8; // 衰减 - } - - // 更新AI语音级别 - if (isSpeaking.value) { - aiVoiceLevel.value = 0.2 + random.nextDouble() * 0.8; + stopListening(); } else { - aiVoiceLevel.value = aiVoiceLevel.value * 0.9; // 衰减 + startListening(); } - - // 更新音频级别数组 - final newLevels = List.generate(20, (index) { - if (isListening.value || isSpeaking.value) { - return random.nextDouble() * (isListening.value ? userVoiceLevel.value : aiVoiceLevel.value); - } - return audioLevels[index] * 0.95; // 衰减 - }); - - audioLevels.value = newLevels; - } - - /// 重置音频级别 - void _resetAudioLevels() { - userVoiceLevel.value = 0.0; - aiVoiceLevel.value = 0.0; - audioLevels.value = List.generate(20, (index) => 0.0); } /// 关闭并返回上一页 Future closeAndGoBack() async { - // 立即返回上一页，避免阻塞UI Get.back(); - - // 之后异步进行清理，不阻塞页面切换 - unawaited(_performShutdown()); - } - - /// 实际执行关闭逻辑（后台） - Future _performShutdown() async { - try { - currentStatus.value = '正在关闭连接...'; - - if (isListening.value) { - await _realtimeService.stopRecording(); + + // 异步清理资源 + Future.delayed(Duration.zero, () async { + try { + if (isListening.value) { + await _realtimeService.stopRecording(); + } + await _realtimeService.stopPlaying(); + await _realtimeService.disconnect(); + await _realtimeService.dispose(); + _resetState(); + } catch (e) { + Logger.error('关闭连接失败: $e'); } - await _realtimeService.stopPlaying(); - await _realtimeService.disconnect(); - - _resetState(); - Logger.info('实时语音连接已关闭'); - } catch (e) { - Logger.error('关闭连接失败: $e'); - } + }); } /// 重置状态 void _resetState() { + isConnected.value = false; + isConnecting.value = false; isListening.value = false; isSpeaking.value = false; userSpeechText.value = ''; - aiResponseText.value = ''; - _resetAudioLevels(); - pulseAnimationController.stop(); - _stopAudioUpdates(); + isUserSpeechFinal.value = true; + aiResponseMessages.clear(); + latestUserMessage.value = ''; + latestAiMessage.value = ''; + partialUserMessage.value = ''; + streamingAiMessage.value = ''; + audioAmplitude.value = 0.0; } /// 清理资源 void _cleanup() async { _eventSubscription?.cancel(); - _stopAudioUpdates(); - pulseAnimationController.dispose(); try { await _realtimeService.dispose(); diff --git a/lib/modules/realtime/views/realtime_view.dart b/lib/modules/realtime/views/realtime_view.dart index 693caf091..cfe33cab4 100644 --- a/lib/modules/realtime/views/realtime_view.dart +++ b/lib/modules/realtime/views/realtime_view.dart @@ -2,7 +2,7 @@ import 'package:flutter/material.dart'; import 'package:flutter/services.dart'; import 'package:get/get.dart'; import '../controllers/realtime_controller.dart'; -import 'widgets/voice_mic_flow.dart'; +import 'widgets/shader_mic_flow.dart'; class RealtimeView extends GetView { const RealtimeView({Key? key}) : super(key: key); @@ -22,219 +22,311 @@ class RealtimeView extends GetView { backgroundColor: Colors.white, body: SafeArea( child: Stack( - children: [ - // 右上角图标按钮 - Positioned( - top: 16, - right: 16, - child: Row( - mainAxisSize: MainAxisSize.min, - children: [ - // 分享按钮 - Icon( - Icons.ios_share, - color: Colors.grey[600], - size: 24, - ), - const SizedBox(width: 20), - // 设置按钮 - Icon( - Icons.tune, - color: Colors.grey[600], - size: 24, + children: [ + // 中间音频可视化 + Positioned.fill( + child: Obx(() => AnimatedAlign( + alignment: controller.isSubtitleMode.value + ? const Alignment(0, -0.7) // 字幕模式时移到更上方 + : const Alignment(0, -0.25), + duration: const Duration(milliseconds: 600), + curve: Curves.easeInOutCubic, + child: LayoutBuilder( + builder: (context, constraints) { + final targetSize = controller.isSubtitleMode.value + ? MediaQuery.of(context).size.width * 0.55 // 字幕模式时较小 + : MediaQuery.of(context).size.width * 0.75; // 初始状态更大 + + return TweenAnimationBuilder( + tween: Tween(end: targetSize), + duration: const Duration(milliseconds: 600), + curve: Curves.easeInOutCubic, + builder: (context, animatedSize, child) { + return Stack( + alignment: Alignment.center, + children: [ + // 使用Shader动画 + Obx(() => ShaderMicFlow( + size: animatedSize, + amplitude: controller.audioAmplitude.value, + )), + ], + ); + }, + ); + }, ), - ], + )), ), - ), - // 状态和字幕显示区域 - Positioned( - top: 100, - left: 20, - right: 20, - child: Column( - children: [ - // 当前状态显示 - Obx(() => Container( - width: double.infinity, - padding: const EdgeInsets.symmetric(horizontal: 16, vertical: 12), - margin: const EdgeInsets.only(bottom: 12), - decoration: BoxDecoration( - color: Colors.black.withOpacity(0.1), - borderRadius: BorderRadius.circular(20), - ), - child: Text( - controller.currentStatus.value, - textAlign: TextAlign.center, - style: const TextStyle( - fontSize: 16, - fontWeight: FontWeight.w500, - color: Colors.black87, + // 字幕显示区域 - 只有在字幕模式开启时才显示，位于圆形下方 + if (controller.isSubtitleMode.value) + Positioned( + left: 20, + right: 20, + top: MediaQuery.of(context).size.height * 0.35, // 在圆形下方 + bottom: 160, + child: Padding( + padding: const EdgeInsets.symmetric(horizontal: 24), + child: Obx(() => AnimatedOpacity( + opacity: controller.isSubtitleMode.value ? 1.0 : 0.0, + duration: const Duration(milliseconds: 300), + child: TweenAnimationBuilder( + tween: Tween( + begin: 0.0, + end: controller.isSubtitleMode.value ? 1.0 : 0.0, + ), + duration: const Duration(milliseconds: 900), // 延迟显示，让圆圈先移动 + curve: Curves.easeInOut, + builder: (context, animationValue, child) { + return Transform.translate( + offset: Offset(0, 20 * (1 - animationValue)), // 从下方滑入 + child: Opacity( + opacity: animationValue, + child: _buildChatSubtitles(), + ), + ); + }, ), - ), - )), - - // 用户ASR内容显示 - Obx(() => AnimatedContainer( - duration: const Duration(milliseconds: 300), - width: double.infinity, - height: controller.userSpeechText.value.isNotEmpty ? null : 0, - padding: controller.userSpeechText.value.isNotEmpty - ? const EdgeInsets.symmetric(horizontal: 16, vertical: 12) - : EdgeInsets.zero, - margin: controller.userSpeechText.value.isNotEmpty - ? const EdgeInsets.only(bottom: 8) - : EdgeInsets.zero, - decoration: BoxDecoration( - color: Colors.blue.withOpacity(0.1), - borderRadius: BorderRadius.circular(16), - border: Border.all( - color: Colors.blue.withOpacity(0.3), - width: 1, + )), + ), + ), + + // 底部控制按钮 + Positioned( + bottom: 80, + left: 0, + right: 0, + child: Center( + child: Row( + mainAxisAlignment: MainAxisAlignment.spaceEvenly, + children: [ + // 麦克风按钮 + Obx(() => _buildMicButton()), + // 字幕按钮 + Obx(() => _buildMainButton( + icon: controller.isSubtitleMode.value ? Icons.subtitles : Icons.subtitles_outlined, + onTap: controller.toggleSubtitleMode, + enabled: true, + isActive: controller.isSubtitleMode.value, + )), + // 清空聊天按钮 + _buildMainButton( + icon: Icons.clear_all, + onTap: controller.clearChat, + enabled: true, + isActive: false, ), - ), - child: controller.userSpeechText.value.isNotEmpty - ? Column( - crossAxisAlignment: CrossAxisAlignment.start, - children: [ - const Text( - '你说：', - style: TextStyle( - fontSize: 12, - fontWeight: FontWeight.w600, - color: Colors.blue, - ), - ), - const SizedBox(height: 4), - Text( - controller.userSpeechText.value, - style: const TextStyle( - fontSize: 14, - color: Colors.black87, - ), - ), - ], - ) - : null, - )), - - // AI回复内容显示 - Obx(() => AnimatedContainer( - duration: const Duration(milliseconds: 300), - width: double.infinity, - height: controller.aiResponseText.value.isNotEmpty ? null : 0, - padding: controller.aiResponseText.value.isNotEmpty - ? const EdgeInsets.symmetric(horizontal: 16, vertical: 12) - : EdgeInsets.zero, - margin: controller.aiResponseText.value.isNotEmpty - ? const EdgeInsets.only(bottom: 20) - : EdgeInsets.zero, - decoration: BoxDecoration( - color: Colors.green.withOpacity(0.1), - borderRadius: BorderRadius.circular(16), - border: Border.all( - color: Colors.green.withOpacity(0.3), - width: 1, + // 关闭按钮 + _buildMainButton( + icon: Icons.close, + onTap: controller.closeAndGoBack, + enabled: true, + isActive: false, ), - ), - child: controller.aiResponseText.value.isNotEmpty - ? Column( - crossAxisAlignment: CrossAxisAlignment.start, - children: [ - const Text( - 'AI回复：', - style: TextStyle( - fontSize: 12, - fontWeight: FontWeight.w600, - color: Colors.green, - ), - ), - const SizedBox(height: 4), - Text( - controller.aiResponseText.value, - style: const TextStyle( - fontSize: 14, - color: Colors.black87, - ), - ), - ], - ) - : null, - )), - ], + ], + ), + ), ), - ), - // 中间会呼吸的云彩动画 - Positioned.fill( - child: Align( - alignment: const Alignment(0, 0.1), - child: const VoiceMicFlow(size: 280), + // Home Indicator + Positioned( + bottom: 8, + left: 0, + right: 0, + child: Center( + child: Container( + width: 134, + height: 5, + decoration: BoxDecoration( + color: Colors.black, + borderRadius: BorderRadius.circular(2.5), + ), + ), + ), ), - ), + ], + ), + ), + ); + } - // 底部控制按钮 - Positioned( - bottom: 80, - left: 0, - right: 0, - child: Padding( - padding: const EdgeInsets.symmetric(horizontal: 40), - child: Row( - mainAxisAlignment: MainAxisAlignment.spaceBetween, - children: [ - // 摄像头按钮 - _buildBottomButton(Icons.videocam, null), - // 麦克风按钮 - _buildBottomButton(Icons.mic_none, controller.toggleListening), - // 更多按钮 - _buildBottomButton(Icons.more_horiz, null), - // 关闭按钮 - _buildBottomButton(Icons.close, controller.closeAndGoBack), - ], + /// 构建聊天字幕界面 - 上下两行显示用户输入和AI回复 + Widget _buildChatSubtitles() { + return Obx(() { + // 组合用户消息：优先显示实时部分转录，否则显示最终消息 + final String displayUserMessage = controller.partialUserMessage.value.isNotEmpty + ? controller.partialUserMessage.value + : controller.latestUserMessage.value; + + // 组合AI消息：优先显示流式响应，否则显示最终消息 + final String displayAiMessage = controller.streamingAiMessage.value.isNotEmpty + ? controller.streamingAiMessage.value + : controller.latestAiMessage.value; + + final hasUserMessage = displayUserMessage.isNotEmpty; + final hasAiMessage = displayAiMessage.isNotEmpty; + + // 如果没有消息，显示空白 + if (!hasUserMessage && !hasAiMessage) { + return Container(); + } + + return Column( + mainAxisAlignment: MainAxisAlignment.center, + crossAxisAlignment: CrossAxisAlignment.center, + mainAxisSize: MainAxisSize.min, // 让Column紧凑排列 + children: [ + // 用户输入行 - 蓝色，动态扩展 + if (hasUserMessage) + Container( + width: double.infinity, + constraints: const BoxConstraints( + minHeight: 30, // 最小1行高度 + maxHeight: 104, // 最大4行高度 (26*4) + ), + padding: const EdgeInsets.symmetric(vertical: 4), + child: SingleChildScrollView( + reverse: true, + child: Text( + displayUserMessage, + textAlign: TextAlign.center, + style: TextStyle( + fontSize: 18, + color: Colors.blue[600], + height: 1.4, + fontWeight: FontWeight.w500, + letterSpacing: 0.3, + ), ), ), ), - - // Home Indicator - Positioned( - bottom: 8, - left: 0, - right: 0, - child: Center( - child: Container( - width: 134, - height: 5, - decoration: BoxDecoration( - color: Colors.black, - borderRadius: BorderRadius.circular(2.5), + + // 分隔线 - 只有在两个消息都存在时才显示 + if (hasUserMessage && hasAiMessage) + Container( + width: 60, + height: 1, + margin: const EdgeInsets.symmetric(vertical: 2), // 进一步减少分隔线间距 + decoration: BoxDecoration( + color: Colors.grey[300], + borderRadius: BorderRadius.circular(0.5), + ), + ), + + // AI回复行 - 绿色，动态扩展 + if (hasAiMessage) + Container( + width: double.infinity, + constraints: const BoxConstraints( + minHeight: 30, // 最小1行高度 + maxHeight: 104, // 最大4行高度 (26*4) + ), + padding: const EdgeInsets.symmetric(vertical: 4), + child: SingleChildScrollView( + reverse: true, + child: Text( + displayAiMessage, + textAlign: TextAlign.center, + style: TextStyle( + fontSize: 18, + color: Colors.green[600], + height: 1.4, + fontWeight: FontWeight.w500, + letterSpacing: 0.3, ), ), ), ), - ], + ], + ); + }); + } + + /// 构建麦克风按钮 - 根据连接状态和录音状态显示不同的状态 + Widget _buildMicButton() { + // 获取当前状态 + final isConnected = controller.isConnected.value; + final isConnecting = controller.isConnecting.value; + final isListening = controller.isListening.value; + + // 确定按钮状态 + IconData icon; + Color? backgroundColor; + Color? iconColor; + VoidCallback? onTap; + + if (!isConnected && !isConnecting) { + // 未连接状态 - 禁用 + icon = Icons.mic_off; + backgroundColor = Colors.grey[300]; + iconColor = Colors.grey[600]; + onTap = null; + } else if (isConnecting) { + // 连接中状态 - 禁用但显示连接中 + icon = Icons.sync; + backgroundColor = Colors.orange[200]; + iconColor = Colors.orange[700]; + onTap = null; + } else if (isListening) { + // 录音中状态 - 激活状态 + icon = Icons.mic; + backgroundColor = Theme.of(Get.context!).primaryColor; + iconColor = Colors.white; + onTap = controller.toggleListening; + } else { + // 已连接但未录音状态 - 正常状态 + icon = Icons.mic_none; + backgroundColor = Colors.grey[200]; + iconColor = Colors.black; + onTap = controller.toggleListening; + } + + return GestureDetector( + onTap: onTap, + child: Container( + width: 60, + height: 60, + decoration: BoxDecoration( + color: backgroundColor, + shape: BoxShape.circle, + ), + child: Icon( + icon, + color: iconColor, + size: 26, ), ), ); } - /// 构建底部圆形按钮 - Widget _buildBottomButton(IconData icon, VoidCallback? onTap) { + /// 构建主要控制按钮 + Widget _buildMainButton({ + required IconData icon, + VoidCallback? onTap, + required bool enabled, + required bool isActive, + }) { return GestureDetector( - onTap: onTap, + onTap: enabled ? onTap : null, child: Container( width: 60, height: 60, decoration: BoxDecoration( - color: Colors.grey[200], + color: enabled + ? (isActive ? Theme.of(Get.context!).primaryColor : Colors.grey[200]) + : Colors.grey[300], shape: BoxShape.circle, ), child: Icon( icon, - color: Colors.black, + color: enabled + ? (isActive ? Colors.white : Colors.black) + : Colors.grey[600], size: 26, ), ), ); } + } \ No newline at end of file diff --git a/lib/modules/realtime/views/widgets/shader_mic_flow.dart b/lib/modules/realtime/views/widgets/shader_mic_flow.dart new file mode 100644 index 000000000..bf21bd14b --- /dev/null +++ b/lib/modules/realtime/views/widgets/shader_mic_flow.dart @@ -0,0 +1,104 @@ +import 'dart:ui'; +import 'package:flutter/material.dart'; + +/// 使用Shader实现的麦克风流动动画组件 +class ShaderMicFlow extends StatefulWidget { + final double size; + final double amplitude; + + const ShaderMicFlow({ + super.key, + required this.size, + this.amplitude = 0.0, + }); + + @override + State createState() => _ShaderMicFlowState(); +} + +class _ShaderMicFlowState extends State + with SingleTickerProviderStateMixin { + late AnimationController _controller; + FragmentShader? _shader; + + @override + void initState() { + super.initState(); + _controller = AnimationController( + duration: const Duration(seconds: 60), + vsync: this, + )..repeat(); + + _loadShader(); + } + + Future _loadShader() async { + try { + final program = await FragmentProgram.fromAsset('shaders/mic_flow.frag'); + setState(() { + _shader = program.fragmentShader(); + }); + } catch (e) { + debugPrint('Failed to load shader: $e'); + } + } + + @override + void dispose() { + _controller.dispose(); + _shader?.dispose(); + super.dispose(); + } + + @override + Widget build(BuildContext context) { + return SizedBox( + width: widget.size, + height: widget.size, + child: AnimatedBuilder( + animation: _controller, + builder: (context, child) { + return CustomPaint( + painter: _shader != null + ? _ShaderPainter( + shader: _shader!, + time: _controller.value * 60.0, + amplitude: widget.amplitude, + ) + : null, + size: Size.square(widget.size), + ); + }, + ), + ); + } +} + +class _ShaderPainter extends CustomPainter { + final FragmentShader shader; + final double time; + final double amplitude; + + _ShaderPainter({ + required this.shader, + required this.time, + required this.amplitude, + }); + + @override + void paint(Canvas canvas, Size size) { + // 设置shader uniforms + shader.setFloat(0, size.width); // u_resolution.x + shader.setFloat(1, size.height); // u_resolution.y + shader.setFloat(2, time); // u_time + shader.setFloat(3, amplitude); // u_amp + + final paint = Paint()..shader = shader; + canvas.drawRect(Offset.zero & size, paint); + } + + @override + bool shouldRepaint(covariant _ShaderPainter oldDelegate) { + return oldDelegate.time != time || oldDelegate.amplitude != amplitude; + } +} \ No newline at end of file diff --git a/lib/modules/realtime/views/widgets/voice_mic_flow.dart b/lib/modules/realtime/views/widgets/voice_mic_flow.dart deleted file mode 100644 index 1716cad47..000000000 --- a/lib/modules/realtime/views/widgets/voice_mic_flow.dart +++ /dev/null @@ -1,89 +0,0 @@ -import 'dart:ui'; -import 'package:flutter/material.dart'; -import 'package:flutter_shaders/flutter_shaders.dart'; - -class VoiceMicFlow extends StatefulWidget { - const VoiceMicFlow({super.key, this.size = 280, this.amp = 0.0}); - final double size; - final double amp; // 音量振幅 0.0-1.0 - - @override - State createState() => _VoiceMicFlowState(); -} - -class _VoiceMicFlowState extends State - with SingleTickerProviderStateMixin { - late final AnimationController _ctl; - - @override - void initState() { - super.initState(); - _ctl = AnimationController( - duration: const Duration(seconds: 100), // 100秒的长周期 - vsync: this, - )..repeat(); - } - - @override - void dispose() { - _ctl.dispose(); - super.dispose(); - } - - @override - Widget build(BuildContext context) { - return ClipOval( - child: ShaderBuilder( - assetKey: 'shaders/mic_flow.frag', - (context, shader, child) { - return AnimatedBuilder( - animation: _ctl, - builder: (_, __) { - // 将0.0-1.0的动画值映射到更大的时间范围 - final time = _ctl.value * 100.0; - shader - ..setFloat(0, widget.size) // u_resolution.x - ..setFloat(1, widget.size) // u_resolution.y - ..setFloat(2, time) // u_time - ..setFloat(3, widget.amp); // u_amp (音量振幅) - return CustomPaint( - size: Size.square(widget.size), - painter: _ShaderPainter(shader), - ); - }, - ); - }, - // 着色器编译时的占位符 - child: Container( - width: widget.size, - height: widget.size, - decoration: const BoxDecoration( - shape: BoxShape.circle, - gradient: LinearGradient( - begin: Alignment.topLeft, - end: Alignment.bottomRight, - colors: [ - Color(0xFF1565C0), // 深蓝 - Color(0xFFE3F2FD), // 浅蓝 - ], - ), - ), - ), - ), - ); - } -} - -class _ShaderPainter extends CustomPainter { - _ShaderPainter(this.shader); - final FragmentShader shader; - - @override - void paint(Canvas canvas, Size size) { - final paint = Paint()..shader = shader; - canvas.drawRect(Offset.zero & size, paint); - } - - @override - bool shouldRepaint(covariant _ShaderPainter oldDelegate) => true; -} \ No newline at end of file diff --git a/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/AzureAsrHelper.kt b/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/AzureAsrHelper.kt index d493330ec..b2680e0c3 100644 --- a/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/AzureAsrHelper.kt +++ b/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/AzureAsrHelper.kt @@ -64,6 +64,10 @@ class AzureAsrHelper(private val context: Context) { // 音频处理 private var microphoneStream: MicrophoneStream? = null private var externalAudioStream: ExternalAudioPullStream? = null + + // 音频模式管理 + private var audioManager: AudioManager? = null + private var originalAudioMode = AudioManager.MODE_NORMAL // 录音文件处理 var recordfile: RecordingFile? = null @@ -104,6 +108,10 @@ class AzureAsrHelper(private val context: Context) { // 释放之前的资源 dispose() + + // 初始化音频管理器 + audioManager = context.getSystemService(Context.AUDIO_SERVICE) as AudioManager + originalAudioMode = audioManager?.mode ?: AudioManager.MODE_NORMAL // 保存配置 this.subscriptionKey = subscriptionKey @@ -171,36 +179,9 @@ class AzureAsrHelper(private val context: Context) { // 设置音频配置 when (audioSourceType) { AudioSourceType.MICROPHONE -> { - // 判断是否使用手机扬声器输出和耳机连接状态 - val audioManager = - context.getSystemService(Context.AUDIO_SERVICE) as AudioManager - val isSpeakerphoneOn = audioManager.isSpeakerphoneOn - - // 检查是否连接了有线耳机 - val isWiredHeadsetOn = audioManager.isWiredHeadsetOn - - // 检查是否连接了蓝牙设备(需要注意兼容性) - val isBluetoothConnected = try { - val method = AudioManager::class.java.getMethod("isBluetoothA2dpOn") - method.invoke(audioManager) as Boolean - } catch (e: Exception) { - false - } - - val isHeadsetConnected = isWiredHeadsetOn || isBluetoothConnected - - // 如果使用手机扬声器输出且没有连接耳机，可能会产生回音，需启动回音消除功能 - val needEchoCancellation = !isHeadsetConnected - - if (needEchoCancellation) { - Log.i(tag, "使用拉流方式进行回音消除") - // 使用拉流方式进行回音消除 - setupMicrophoneStream() - } else { - Log.i(tag, "使用拉流方式进行回音消除2") - audioConfig = AudioConfig.fromDefaultMicrophoneInput() - setupMicrophoneStream() - } + // 总是使用拉流方式并启用回音消除 + Log.i(tag, "使用拉流方式并启用回音消除") + setupMicrophoneStream() } AudioSourceType.EXTERNAL -> { @@ -651,6 +632,8 @@ else{ private inner class MicrophoneStream : PullAudioInputStreamCallback() { private var audioRecord: AudioRecord? = null private var echoCanceler: AcousticEchoCanceler? = null + private var noiseSuppressor: NoiseSuppressor? = null + private var automaticGainControl: AutomaticGainControl? = null // 音频配置 private val sampleRate = 16000 @@ -715,19 +698,45 @@ else{ } /** - * 优化：条件性应用音频效果 + * 应用音频效果 - 总是启用所有音频增强功能 */ private fun applyAudioEffects() { val sessionId = audioRecord?.audioSessionId ?: return - - // 优化：只在真正需要时启用回音消除 - val audioManager = context.getSystemService(Context.AUDIO_SERVICE) as AudioManager - if (audioManager.isSpeakerphoneOn) { - echoCanceler = AcousticEchoCanceler.create(sessionId) - echoCanceler?.enabled = true - Log.d(tag, "启用回音消除") - } else { - Log.d(tag, "跳过回音消除（使用耳机）") + + try { + // 设置音频模式为通信模式 + this@AzureAsrHelper.audioManager?.mode = AudioManager.MODE_IN_COMMUNICATION + + // 启用声学回音消除 + if (AcousticEchoCanceler.isAvailable()) { + echoCanceler = AcousticEchoCanceler.create(sessionId) + echoCanceler?.enabled = true + Log.i(tag, "声学回音消除已启用") + } else { + Log.w(tag, "设备不支持声学回音消除") + } + + // 启用噪声抑制 + if (NoiseSuppressor.isAvailable()) { + noiseSuppressor = NoiseSuppressor.create(sessionId) + noiseSuppressor?.enabled = true + Log.i(tag, "噪声抑制已启用") + } else { + Log.w(tag, "设备不支持噪声抑制") + } + + // 启用自动增益控制 + if (AutomaticGainControl.isAvailable()) { + automaticGainControl = AutomaticGainControl.create(sessionId) + automaticGainControl?.enabled = true + Log.i(tag, "自动增益控制已启用") + } else { + Log.w(tag, "设备不支持自动增益控制") + } + + Log.i(tag, "回音消除和音频增强功能已全部启用") + } catch (e: Exception) { + Log.e(tag, "设置音频效果失败: ${e.message}") } } @@ -777,13 +786,34 @@ else{ } // 释放回音消除器 - echoCanceler?.release() - echoCanceler = null + echoCanceler?.let { + it.enabled = false + it.release() + echoCanceler = null + } + + // 释放噪声抑制器 + noiseSuppressor?.let { + it.enabled = false + it.release() + noiseSuppressor = null + } + + // 释放自动增益控制 + automaticGainControl?.let { + it.enabled = false + it.release() + automaticGainControl = null + } + + // 恢复音频模式 + this@AzureAsrHelper.audioManager?.mode = this@AzureAsrHelper.originalAudioMode // 释放录音实例 audioRecord?.release() audioRecord = null } catch (e: Exception) { + Log.e(tag, "释放音频资源失败: ${e.message}") e.printStackTrace() } } diff --git a/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/AzureTtsHelper.kt b/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/AzureTtsHelper.kt index 1e5251de5..d750fae41 100644 --- a/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/AzureTtsHelper.kt +++ b/local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/AzureTtsHelper.kt @@ -71,15 +71,21 @@ class AzureTtsHelper(private val context: Context) : ITtsService { // 创建语音配置 speechConfig = SpeechConfig.fromSubscription(ttsAppToken, ttsResource) - // 优化：使用更低延迟的音频格式（从24KHz降到16KHz） + // 使用标准的音频格式（16KHz, 16bit, 单声道） speechConfig?.setSpeechSynthesisOutputFormat(SpeechSynthesisOutputFormat.Riff16Khz16BitMonoPcm) + // 确保使用正确的语音输出 + speechConfig?.setProperty("SPEECH-AudioOutputFormat", "riff-16khz-16bit-mono-pcm") + // 优化：设置低延迟连接属性 speechConfig?.setProperty("SpeechServiceConnection_InitialSilenceTimeoutMs", "300") speechConfig?.setProperty("SpeechServiceConnection_EndSilenceTimeoutMs", "300") speechConfig?.setSpeechSynthesisVoiceName(currentVoice) + // 添加日志记录语音格式 + FileLogger.d(TAG, "语音设置: voice=$currentVoice, format=Riff16Khz16BitMonoPcm") + // 创建音频配置 val audioConfig = if (customAudioOutputStream != null) { // 使用自定义音频输出流 @@ -185,21 +191,47 @@ class AzureTtsHelper(private val context: Context) : ITtsService { private fun setupEventListeners() { synthesizer?.apply { // 合成开始事件 - SynthesisStarted?.addEventListener { _, _ -> + SynthesisStarted?.addEventListener { _, eventArgs -> + FileLogger.d(TAG, "语音合成开始: resultId=${eventArgs.result.resultId}") notifyEvent(TtsEventType.SYNTHESIS_STARTED) } + // 合成中事件（接收音频数据） + Synthesizing?.addEventListener { _, eventArgs -> + val audioData = eventArgs.result.audioData + FileLogger.d(TAG, "接收到音频数据: ${audioData.size} 字节") + + // 通知音频数据监听器 + if (audioData.isNotEmpty()) { + val listeners = ArrayList(audioDataListeners) + for (listener in listeners) { + try { + listener.onAudioData(audioData) + } catch (e: Exception) { + FileLogger.e(TAG, "通知音频数据异常: ${e.message}") + } + } + } + } + // 合成完成事件 - SynthesisCompleted?.addEventListener { _, _ -> + SynthesisCompleted?.addEventListener { _, eventArgs -> + FileLogger.d(TAG, "语音合成完成: resultId=${eventArgs.result.resultId}, 音频长度=${eventArgs.result.audioLength} 字节") isSpeaking = false notifyEvent(TtsEventType.SYNTHESIS_COMPLETED) } // 合成取消事件 SynthesisCanceled?.addEventListener { _, eventArgs -> - FileLogger.d(TAG, "语音合成取消") + val reason = eventArgs.result.reason + FileLogger.e(TAG, "语音合成取消: reason=$reason") + + if (reason == ResultReason.Canceled) { + val cancellation = SpeechSynthesisCancellationDetails.fromResult(eventArgs.result) + FileLogger.e(TAG, "取消详情: reason=${cancellation.reason}, errorCode=${cancellation.errorCode}, errorDetails=${cancellation.errorDetails}") + } + isSpeaking = false - notifyEvent(TtsEventType.SYNTHESIS_CANCELED) } } @@ -397,6 +429,8 @@ class AzureTtsHelper(private val context: Context) : ITtsService { isSpeaking = true + FileLogger.d(TAG, "开始语音合成，文本长度: ${text.length}") + // 直接调用SpeakSsmlAsync，SDK内部已经是异步的 synthesizer?.SpeakSsmlAsync(ssml)