wolfplus 1 year ago
parent
commit
ac6e55937d
  1. 172
      lib/modules/realtime/README.md
  2. 342
      lib/modules/realtime/controllers/realtime_controller.dart
  3. 452
      lib/modules/realtime/views/realtime_view.dart
  4. 104
      lib/modules/realtime/views/widgets/shader_mic_flow.dart
  5. 89
      lib/modules/realtime/views/widgets/voice_mic_flow.dart
  6. 114
      local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/AzureAsrHelper.kt
  7. 44
      local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/AzureTtsHelper.kt

172
lib/modules/realtime/README.md

@ -2,135 +2,159 @@
## ⚠️ 当前状态 ## ⚠️ 当前状态
**此模块为功能预览版本,语音对话功能暂未实现,仅提供界面效果演示。** **此模块为功能预览版本,提供完整的实时语音对话功能。**
后续将通过插件方式实现完整的语音识别、AI对话和语音合成功能。 通过原生插件实现语音识别、AI对话和语音合成的完整语音交互体验。
## 概述 ## 概述
Realtime模块预留了类似ChatGPT app的实时语音对话功能框架,当前版本提供: Realtime模块实现了类似ChatGPT app的实时语音对话功能,当前版本提供:
- ✅ **界面框架** - 完整的UI布局和视觉效果 - ✅ **完整语音交互** - 实时语音识别、AI对话和语音合成
- ✅ **状态管理** - 聆听、思考、回答状态的可视化 - ✅ **状态管理** - 聆听、思考、回答状态的可视化
- ✅ **动画效果** - 圆形波形动画和状态指示器 - ✅ **动画效果** - 基于Shader的流动云效果,支持音频振幅驱动
- ✅ **交互体验** - 按钮点击和模拟状态切换 - ✅ **字幕显示** - 可选择开启/关闭的字幕模式
- ⏳ **语音功能** - 待通过插件实现 - ✅ **智能UI** - 根据连接状态自动调整按钮可用性
## 界面设计 ## 界面设计
### 布局结构 ### 布局结构
- **顶部状态区域** - 显示当前状态和提示信息 - **顶部状态区域** - 显示当前状态和字幕模式切换按钮
- **中间可视化区域** - 圆形波形动画,类似ChatGPT风格 - **中间可视化区域** - 圆形Shader动画,根据音频振幅呈现呼吸效果
- **底部控制区域** - 主控制按钮和状态指示器 - **底部控制区域** - 主控制按钮和状态指示器
### 状态指示 ### 状态指示
- 🔵 **聆听状态** - 蓝色动画,模拟语音输入 - 🔵 **聆听状态** - 蓝色动画,接收用户语音输入
- 🟠 **思考状态** - 橙色动画,模拟AI处理 - 🟠 **思考状态** - 橙色动画,AI处理中
- 🟢 **回答状态** - 绿色动画,模拟语音输出 - 🟢 **回答状态** - 绿色动画,AI语音输出
- ⚪ **空闲状态** - 灰色静态,等待交互 - ⚪ **空闲状态** - 灰色静态,等待交互
### 字幕模式
- **默认关闭** - 初始状态下不显示任何字幕内容
- **手动切换** - 通过右上角按钮手动开启/关闭字幕
- **完整隐藏** - 关闭时字幕区域完全不显示,不占用界面空间
- **智能布局** - 开启时圆形动画自动上移,字幕区域占据下半屏
### 交互控制
- **录音按钮** - 仅在成功连接AI服务器且开始录音后可用
- **连接状态** - 未连接时录音按钮显示不可用状态
- **状态反馈** - 按钮颜色和可点击性根据连接状态动态调整
## 使用方法 ## 使用方法
### 路由导航 ### 路由导航
```dart ```dart
// 跳转到实时语音对话页面(模拟功能) // 跳转到实时语音对话页面
Get.toNamed(Routes.realtime); Get.toNamed(Routes.realtime);
``` ```
### 体验模拟效果 ### 交互流程
1. 点击中央按钮开始模拟聆听 1. **自动连接** - 页面加载后自动尝试连接AI服务器
2. 3秒后自动切换到AI思考状态 2. **等待连接** - 录音按钮在连接成功前保持不可用状态
3. 2秒后切换到语音播放状态 3. **开始录音** - 连接成功后自动开始录音,录音按钮变为可用
4. 3秒后回到初始状态 4. **语音对话** - 用户说话 → AI处理 → 语音回复
5. **字幕控制** - 可随时通过右上角按钮切换字幕显示
## 技术架构 ## 技术架构
### 控制器 (RealtimeController) ### 控制器 (RealtimeController)
- **状态管理**: `isListening`, `isSpeaking`, `isProcessing` - **连接管理**: WebSocket连接到AI服务器
- **模拟逻辑**: 定时器控制状态切换 - **状态同步**: `isConnected`, `isListening`, `isSpeaking`
- **动画控制**: 波形数据生成和更新 - **字幕控制**: `isSubtitleMode` 默认为false,用户手动控制
- **生命周期**: 基础资源管理 - **消息管理**: 智能去重,避免流式响应重复显示
- **音频处理**: 原生层计算RMS值,Flutter层接收并控制动画
### 视图组件 ### 视图组件
- **RealtimeView** - 主视图容器 - **RealtimeView** - 主视图容器,响应式布局
- **VoiceVisualizer** - 语音可视化组件(简化版) - **ShaderMicFlow** - 基于GLSL Shader的流动云动画组件
- **ControlButton** - 交互控制按钮(模拟功能) - **AutoScrollList** - 自动滚动的字幕列表
### 待实现功能 ### 原生集成
- **AsrService** - 语音识别服务(插件) - **RealtimeService** - 原生音频处理和WebSocket通信
- **TtsService** - 语音合成服务(插件) - **音频格式** - 16kHz, 1声道, 16位PCM
- **OpenAIServiceAdapter** - AI对话服务(插件) - **实时传输** - 低延迟音频流处理
- **RMS计算** - 原生层实时计算音频振幅
## 模拟流程 ## 默认行为
1. **点击开始** → 模拟语音识别启动 ### 字幕模式
2. **动态波形** → 显示模拟的语音输入效果 - **初始状态**: 关闭 (`isSubtitleMode.value = false`)
3. **自动停止** → 3秒后自动结束聆听 - **显示逻辑**: 仅在用户手动开启时显示字幕内容
4. **AI处理** → 模拟AI思考过程 - **空间占用**: 关闭时字幕区域高度为0,不影响其他UI元素
5. **语音输出** → 模拟语音播放效果 - **切换效果**: 平滑动画过渡,圆形动画位置和大小同步调整
6. **完成循环** → 返回待机状态
## 开发计划
### 阶段一:界面框架 ✅ ### 按钮状态
- [x] 基础UI布局 - **录音按钮**: 默认不可用,连接成功且开始录音后才可点击
- [x] 状态管理 - **状态指示**: 通过颜色和图标变化反映当前可用性
- [x] 动画效果 - **交互反馈**: 不可用时点击无效果,避免误操作
- [x] 模拟交互
### 阶段二:插件集成 ⏳ ## 开发计划
- [ ] 语音识别插件集成
- [ ] AI对话服务集成
- [ ] 语音合成插件集成
- [ ] 实时流式处理
### 阶段三:功能完善 ⏳ ### 已完成功能 ✅
- [ ] 错误处理优化 - [x] 完整语音交互流程
- [ ] 性能优化 - [x] 实时WebSocket通信
- [ ] 用户体验增强 - [x] Shader流动云动画
- [ ] 多语言支持 - [x] RMS音频振幅计算
- [x] 智能UI状态管理
- [x] 字幕模式控制
- [x] 消息去重优化
### 计划增强 ⏳
- [ ] 多语言语音识别
- [ ] 语音中断检测
- [ ] 音频质量自适应
- [ ] 离线降级处理
- [ ] 语音情感识别
## 注意事项 ## 注意事项
1. **当前为演示版** - 所有功能均为模拟效果 1. **网络依赖** - 需要连接到AI服务器 (ws://192.168.1.11:8000/ws)
2. **无需权限** - 暂不涉及麦克风等硬件权限 2. **麦克风权限** - 需要获取麦克风使用权限
3. **无网络依赖** - 模拟功能不需要网络连接 3. **音频会话** - 自动管理音频会话,支持蓝牙设备
4. **保留扩展性** - 架构设计支持后续插件集成 4. **资源管理** - 页面关闭时自动清理音频资源和网络连接
## 自定义扩展 ## 自定义配置
### 修改模拟参数 ### 修改服务器地址
```dart ```dart
// 在RealtimeController中调整时间参数 // 在RealtimeController中修改连接URL
Timer(const Duration(seconds: 3), () { // 聆听时长 await _realtimeService.initialize(
stopListening(); serverUrl: 'ws://your-server:port/ws',
}); // ... 其他参数
);
```
await Future.delayed(const Duration(seconds: 2)); // AI处理时长 ### 调整UI布局
await Future.delayed(const Duration(seconds: 3)); // 播放时长
```dart
// 修改字幕模式下圆形动画位置
alignment: controller.isSubtitleMode.value
? const Alignment(0, -0.9) // 调整Y轴位置
: const Alignment(0, -0.25),
``` ```
### 添加新的视觉效果 ### 音频参数调优
```dart ```dart
// 在VoiceWavePainter中添加新的绘制方法 // 在初始化时调整音频参数
void _drawCustomEffect(Canvas canvas, Offset center, double radius) { sampleRate: 16000, // 采样率
// 自定义绘制逻辑 channels: 1, // 声道数
} bitsPerSample: 16, // 位深度
``` ```
## 技术栈 ## 技术栈
- **Flutter SDK**: >= 3.0.0 - **Flutter SDK**: >= 3.0.0
- **GetX**: >= 4.6.5 (状态管理) - **GetX**: >= 4.6.5 (状态管理)
- **Material Design**: UI组件 - **Flutter Shaders**: >= 0.0.6 (Shader动画)
- **CustomPainter**: 自定义绘制 - **原生插件**: Realtime服务 (音频处理)
- **WebSocket**: 实时通信协议
## 版本信息 ## 版本信息
- **当前版本**: v0.1.0 (Preview) - **当前版本**: v1.0.0 (Production)
- **最后更新**: 2024年 - **最后更新**: 2024年
- **状态**: 模拟功能演示 - **状态**: 完整功能实现

342
lib/modules/realtime/controllers/realtime_controller.dart

@ -1,15 +1,22 @@
import 'dart:async'; import 'dart:async';
import 'dart:math'; import 'dart:convert';
import 'package:flutter/material.dart'; import 'package:flutter/material.dart';
import 'package:flutter/services.dart'; import 'package:flutter/services.dart';
import 'package:get/get.dart'; import 'package:get/get.dart';
import 'package:realtime/realtime.dart'; import 'package:realtime/realtime.dart';
import '../../../core/utils/logger.dart'; import '../../../core/utils/logger.dart';
class RealtimeController extends GetxController with GetTickerProviderStateMixin { class RealtimeController extends GetxController with GetTickerProviderStateMixin {
static const int _sampleRate = 16000;
// Realtime服务实例 // Realtime服务实例
final RealtimeService _realtimeService = RealtimeService(); final RealtimeService _realtimeService = RealtimeService();
// 音频振幅值(RMS)
final RxDouble audioAmplitude = 0.0.obs;
// 连接状态 // 连接状态
final RxBool isConnected = false.obs; final RxBool isConnected = false.obs;
final RxBool isConnecting = false.obs; final RxBool isConnecting = false.obs;
@ -18,31 +25,29 @@ class RealtimeController extends GetxController with GetTickerProviderStateMixin
final RxBool isListening = false.obs; final RxBool isListening = false.obs;
final RxBool isSpeaking = false.obs; final RxBool isSpeaking = false.obs;
// 状态和字幕显示 // UI状态
final RxString currentStatus = '正在初始化...'.obs; final RxString currentStatus = ''.obs; // 移除默认文字
final RxString userSpeechText = ''.obs; final RxString userSpeechText = ''.obs;
final RxString aiResponseText = ''.obs; final RxBool isUserSpeechFinal = true.obs;
final RxList<String> aiResponseMessages = <String>[].obs;
final RxBool isSubtitleMode = true.obs;
// 音频可视化数据 // 字幕显示用的最新消息
final RxList<double> audioLevels = <double>[].obs; final RxString latestUserMessage = ''.obs;
final RxDouble userVoiceLevel = 0.0.obs; final RxString latestAiMessage = ''.obs;
final RxDouble aiVoiceLevel = 0.0.obs;
// 动画控制器 // 实时字幕更新用的临时消息
late AnimationController pulseAnimationController; final RxString partialUserMessage = ''.obs;
late Animation<double> pulseAnimation; final RxString streamingAiMessage = ''.obs;
// 状态监听 // 状态监听
StreamSubscription<RealtimeEvent>? _eventSubscription; StreamSubscription<RealtimeEvent>? _eventSubscription;
Timer? _audioUpdateTimer;
@override @override
void onInit() { void onInit() {
super.onInit(); super.onInit();
_initializeAnimations();
_initializeAudioLevels();
_setupEventListener(); _setupEventListener();
Logger.info('Realtime模块初始化完成');
// 自动开始连接 // 自动开始连接
WidgetsBinding.instance.addPostFrameCallback((_) { WidgetsBinding.instance.addPostFrameCallback((_) {
@ -56,27 +61,6 @@ class RealtimeController extends GetxController with GetTickerProviderStateMixin
super.onClose(); super.onClose();
} }
/// 初始化动画
void _initializeAnimations() {
pulseAnimationController = AnimationController(
duration: const Duration(milliseconds: 1000),
vsync: this,
);
pulseAnimation = Tween<double>(
begin: 0.8,
end: 1.2,
).animate(CurvedAnimation(
parent: pulseAnimationController,
curve: Curves.easeInOut,
));
}
/// 初始化音频级别数据
void _initializeAudioLevels() {
audioLevels.value = List.generate(20, (index) => 0.0);
}
/// 设置事件监听器 /// 设置事件监听器
void _setupEventListener() { void _setupEventListener() {
_eventSubscription = _realtimeService.eventStream.listen( _eventSubscription = _realtimeService.eventStream.listen(
@ -101,6 +85,9 @@ class RealtimeController extends GetxController with GetTickerProviderStateMixin
case 'textReceived': case 'textReceived':
_handleTextReceived(event.data); _handleTextReceived(event.data);
break; break;
case 'audioRms':
_handleAudioRms(event.data);
break;
case 'error': case 'error':
_handleError(event.data); _handleError(event.data);
break; break;
@ -109,7 +96,7 @@ class RealtimeController extends GetxController with GetTickerProviderStateMixin
/// 自动连接 /// 自动连接
Future<void> _autoConnect() async { Future<void> _autoConnect() async {
currentStatus.value = '正在连接AI服务器...'; currentStatus.value = ''; // 不显示任何文字
await connect(); await connect();
} }
@ -119,9 +106,7 @@ class RealtimeController extends GetxController with GetTickerProviderStateMixin
case 'connected': case 'connected':
isConnected.value = true; isConnected.value = true;
isConnecting.value = false; isConnecting.value = false;
currentStatus.value = '已连接,正在开始录音...'; currentStatus.value = ''; // 不显示任何文字
_startAudioUpdates();
Logger.info('已连接到Vocode服务器');
// 连接成功后自动开始录音 // 连接成功后自动开始录音
Timer(const Duration(milliseconds: 500), () { Timer(const Duration(milliseconds: 500), () {
startListening(); startListening();
@ -129,20 +114,18 @@ class RealtimeController extends GetxController with GetTickerProviderStateMixin
break; break;
case 'connecting': case 'connecting':
isConnecting.value = true; isConnecting.value = true;
currentStatus.value = '正在连接AI服务器...'; currentStatus.value = ''; // 不显示任何文字
Logger.info('正在连接Vocode服务器...');
break; break;
case 'disconnected': case 'disconnected':
isConnected.value = false; isConnected.value = false;
isConnecting.value = false; isConnecting.value = false;
currentStatus.value = '已断开连接'; currentStatus.value = ''; // 不显示任何文字
_resetState(); _resetState();
Logger.info('已断开Vocode服务器连接');
break; break;
case 'error': case 'error':
isConnected.value = false; isConnected.value = false;
isConnecting.value = false; isConnecting.value = false;
currentStatus.value = '连接失败,请检查网络'; currentStatus.value = '连接失败,请检查网络'; // 保留错误提示
_resetState(); _resetState();
Logger.error('Vocode服务器连接错误'); Logger.error('Vocode服务器连接错误');
break; break;
@ -155,60 +138,139 @@ class RealtimeController extends GetxController with GetTickerProviderStateMixin
case 'recording': case 'recording':
isListening.value = true; isListening.value = true;
isSpeaking.value = false; isSpeaking.value = false;
currentStatus.value = '正在聆听,请说话...'; currentStatus.value = ''; // 不显示任何文字
pulseAnimationController.repeat(reverse: true);
Logger.info('开始录音');
break; break;
case 'processing': case 'processing':
// 用户说话结束,等待AI回应
isListening.value = false; isListening.value = false;
currentStatus.value = 'AI正在思考中...'; currentStatus.value = ''; // 不显示任何文字
break; break;
case 'playing': case 'playing':
isListening.value = false; isListening.value = false;
isSpeaking.value = true; isSpeaking.value = true;
currentStatus.value = 'AI正在回复...'; currentStatus.value = ''; // 不显示任何文字
pulseAnimationController.repeat(reverse: true);
Logger.info('AI开始说话');
break; break;
case 'idle': case 'idle':
isListening.value = false; isListening.value = false;
isSpeaking.value = false; isSpeaking.value = false;
currentStatus.value = isConnected.value ? '等待用户说话...' : '已断开连接'; currentStatus.value = ''; // 不显示任何文字
pulseAnimationController.stop(); // 空闲时音频振幅归零
Logger.info('语音状态空闲'); audioAmplitude.value = 0.0;
break; break;
} }
} }
/// 处理收到的文本消息 /// 处理收到的文本消息 - 标准Vocode WebSocket协议
void _handleTextReceived(String text) { void _handleTextReceived(dynamic payload) {
Logger.info('收到文本消息: $text'); if (payload is String) {
try {
// 根据消息来源更新相应的文本 // 解析JSON消息
if (text.startsWith('[USER]')) { final Map<String, dynamic> json = jsonDecode(payload);
userSpeechText.value = text.replaceFirst('[USER]', '').trim(); final String messageType = json['type'] ?? '';
} else if (text.startsWith('[AI]')) {
aiResponseText.value = text.replaceFirst('[AI]', '').trim(); switch (messageType) {
case 'websocket_transcript':
// 处理统一的文本消息格式
final String text = json['text'] ?? '';
final String sender = json['sender'] ?? '';
final bool isFinal = json['is_final'] ?? false;
// final dynamic timestamp = json['timestamp']; // 保留供未来使用
if (sender.toLowerCase() == 'human') {
// 用户语音识别消息
if (isFinal) {
// ASR最终结果
if (text.isNotEmpty) {
latestUserMessage.value = text;
userSpeechText.value = text;
isUserSpeechFinal.value = true;
}
// 清空部分消息
partialUserMessage.value = '';
} else {
// ASR实时识别
if (text.isNotEmpty) {
partialUserMessage.value = text;
isUserSpeechFinal.value = false;
}
}
} else if (sender.toLowerCase() == 'bot') {
// AI回复消息
if (isFinal) {
// LLM回复完成标识 (text为空)
if (streamingAiMessage.value.isNotEmpty) {
// 将流式消息确认为最终消息
latestAiMessage.value = streamingAiMessage.value;
if (aiResponseMessages.isNotEmpty) {
aiResponseMessages[aiResponseMessages.length - 1] = streamingAiMessage.value;
} else {
aiResponseMessages.add(streamingAiMessage.value);
}
streamingAiMessage.value = '';
}
} else {
// LLM流式回复
if (text.isNotEmpty) {
streamingAiMessage.value += text;
}
}
}
break;
case 'websocket_ready':
// 连接就绪
break;
case 'websocket_stop':
// 会话结束
break;
case 'websocket_start':
// 会话开始
break;
default:
Logger.error('收到未知消息类型: "$messageType"');
}
} catch (e) {
Logger.error('解析WebSocket消息失败: $e');
}
} else { } else {
// 默认认为是AI回复 Logger.error('收到非字符串类型的payload: ${payload.runtimeType}');
aiResponseText.value = text; }
}
/// 处理音频RMS值
void _handleAudioRms(dynamic rms) {
if (rms is double || rms is num) {
// 平滑处理RMS值,避免抖动太大,增大幅度
final double rawValue = rms.toDouble().clamp(0.0, 1.0);
// 增强幅度:平方根函数减少小值,对数函数增强大值
final double enhancedValue = (rawValue * 3.0).clamp(0.0, 1.0);
// 更强的平滑处理,减少抖动
audioAmplitude.value = audioAmplitude.value * 0.85 + enhancedValue * 0.15;
} }
} }
/// 处理错误 /// 处理错误
void _handleError(String error) { void _handleError(String error) {
Logger.error('Realtime错误: $error'); Logger.error('Realtime错误: $error');
// 可以显示错误提示
} }
/// 切换连接状态 /// 切换字幕模式
Future<void> toggleConnection() async { void toggleSubtitleMode() {
if (isConnected.value) { isSubtitleMode.value = !isSubtitleMode.value;
await disconnect(); }
} else {
await connect(); /// 清空聊天记录
} void clearChat() {
userSpeechText.value = '';
isUserSpeechFinal.value = true;
aiResponseMessages.clear();
latestUserMessage.value = '';
latestAiMessage.value = '';
partialUserMessage.value = '';
streamingAiMessage.value = '';
HapticFeedback.lightImpact();
} }
/// 连接到AI服务 /// 连接到AI服务
@ -219,13 +281,14 @@ class RealtimeController extends GetxController with GetTickerProviderStateMixin
HapticFeedback.lightImpact(); HapticFeedback.lightImpact();
// 初始化Realtime服务 // 初始化Realtime服务
final initialized = await _realtimeService.initialize( bool initialized = await _realtimeService.initialize(
serverUrl: 'ws://192.168.1.11:8000/ws', // 本地开发服务器使用ws协议 // serverUrl: 'ws://mcp.ideapsound.com:8000/ws',
sampleRate: 16000, serverUrl: 'ws://192.168.1.11:8000/ws',
sampleRate: _sampleRate,
channels: 1, channels: 1,
bitsPerSample: 16, bitsPerSample: 16,
); );
if (!initialized) { if (!initialized) {
throw Exception('初始化Realtime服务失败'); throw Exception('初始化Realtime服务失败');
} }
@ -248,36 +311,18 @@ class RealtimeController extends GetxController with GetTickerProviderStateMixin
try { try {
HapticFeedback.mediumImpact(); HapticFeedback.mediumImpact();
// 停止录音和播放
if (isListening.value) { if (isListening.value) {
await _realtimeService.stopRecording(); await _realtimeService.stopRecording();
} }
await _realtimeService.stopPlaying(); await _realtimeService.stopPlaying();
// 断开连接
await _realtimeService.disconnect(); await _realtimeService.disconnect();
// 停止本地更新
_stopAudioUpdates();
pulseAnimationController.stop();
} catch (e) { } catch (e) {
Logger.error('断开连接失败: $e'); Logger.error('断开连接失败: $e');
} }
} }
/// 开始/停止聆听
void toggleListening() {
if (!isConnected.value) return;
if (isListening.value) {
stopListening();
} else {
startListening();
}
}
/// 开始聆听 /// 开始聆听
void startListening() { void startListening() {
if (!isConnected.value || isListening.value) return; if (!isConnected.value || isListening.value) return;
@ -302,103 +347,56 @@ class RealtimeController extends GetxController with GetTickerProviderStateMixin
}); });
} }
/// 开始音频数据更新 /// 切换聆听状态
void _startAudioUpdates() { void toggleListening() {
_audioUpdateTimer = Timer.periodic(const Duration(milliseconds: 100), (timer) { if (!isConnected.value) return;
if (!isConnected.value) {
timer.cancel();
return;
}
_updateAudioLevels();
});
}
/// 停止音频数据更新
void _stopAudioUpdates() {
_audioUpdateTimer?.cancel();
_audioUpdateTimer = null;
}
/// 更新音频级别
void _updateAudioLevels() {
final random = Random();
// 更新用户语音级别
if (isListening.value) { if (isListening.value) {
userVoiceLevel.value = 0.3 + random.nextDouble() * 0.7; stopListening();
} else {
userVoiceLevel.value = userVoiceLevel.value * 0.8; // 衰减
}
// 更新AI语音级别
if (isSpeaking.value) {
aiVoiceLevel.value = 0.2 + random.nextDouble() * 0.8;
} else { } else {
aiVoiceLevel.value = aiVoiceLevel.value * 0.9; // 衰减 startListening();
} }
// 更新音频级别数组
final newLevels = List<double>.generate(20, (index) {
if (isListening.value || isSpeaking.value) {
return random.nextDouble() * (isListening.value ? userVoiceLevel.value : aiVoiceLevel.value);
}
return audioLevels[index] * 0.95; // 衰减
});
audioLevels.value = newLevels;
}
/// 重置音频级别
void _resetAudioLevels() {
userVoiceLevel.value = 0.0;
aiVoiceLevel.value = 0.0;
audioLevels.value = List.generate(20, (index) => 0.0);
} }
/// 关闭并返回上一页 /// 关闭并返回上一页
Future<void> closeAndGoBack() async { Future<void> closeAndGoBack() async {
// 立即返回上一页,避免阻塞UI
Get.back(); Get.back();
// 之后异步进行清理,不阻塞页面切换 // 异步清理资源
unawaited(_performShutdown()); Future.delayed(Duration.zero, () async {
} try {
if (isListening.value) {
/// 实际执行关闭逻辑(后台) await _realtimeService.stopRecording();
Future<void> _performShutdown() async { }
try { await _realtimeService.stopPlaying();
currentStatus.value = '正在关闭连接...'; await _realtimeService.disconnect();
await _realtimeService.dispose();
if (isListening.value) { _resetState();
await _realtimeService.stopRecording(); } catch (e) {
Logger.error('关闭连接失败: $e');
} }
await _realtimeService.stopPlaying(); });
await _realtimeService.disconnect();
_resetState();
Logger.info('实时语音连接已关闭');
} catch (e) {
Logger.error('关闭连接失败: $e');
}
} }
/// 重置状态 /// 重置状态
void _resetState() { void _resetState() {
isConnected.value = false;
isConnecting.value = false;
isListening.value = false; isListening.value = false;
isSpeaking.value = false; isSpeaking.value = false;
userSpeechText.value = ''; userSpeechText.value = '';
aiResponseText.value = ''; isUserSpeechFinal.value = true;
_resetAudioLevels(); aiResponseMessages.clear();
pulseAnimationController.stop(); latestUserMessage.value = '';
_stopAudioUpdates(); latestAiMessage.value = '';
partialUserMessage.value = '';
streamingAiMessage.value = '';
audioAmplitude.value = 0.0;
} }
/// 清理资源 /// 清理资源
void _cleanup() async { void _cleanup() async {
_eventSubscription?.cancel(); _eventSubscription?.cancel();
_stopAudioUpdates();
pulseAnimationController.dispose();
try { try {
await _realtimeService.dispose(); await _realtimeService.dispose();

452
lib/modules/realtime/views/realtime_view.dart

@ -2,7 +2,7 @@ import 'package:flutter/material.dart';
import 'package:flutter/services.dart'; import 'package:flutter/services.dart';
import 'package:get/get.dart'; import 'package:get/get.dart';
import '../controllers/realtime_controller.dart'; import '../controllers/realtime_controller.dart';
import 'widgets/voice_mic_flow.dart'; import 'widgets/shader_mic_flow.dart';
class RealtimeView extends GetView<RealtimeController> { class RealtimeView extends GetView<RealtimeController> {
const RealtimeView({Key? key}) : super(key: key); const RealtimeView({Key? key}) : super(key: key);
@ -22,219 +22,311 @@ class RealtimeView extends GetView<RealtimeController> {
backgroundColor: Colors.white, backgroundColor: Colors.white,
body: SafeArea( body: SafeArea(
child: Stack( child: Stack(
children: [ children: [
// 右上角图标按钮 // 中间音频可视化
Positioned( Positioned.fill(
top: 16, child: Obx(() => AnimatedAlign(
right: 16, alignment: controller.isSubtitleMode.value
child: Row( ? const Alignment(0, -0.7) // 字幕模式时移到更上方
mainAxisSize: MainAxisSize.min, : const Alignment(0, -0.25),
children: [ duration: const Duration(milliseconds: 600),
// 分享按钮 curve: Curves.easeInOutCubic,
Icon( child: LayoutBuilder(
Icons.ios_share, builder: (context, constraints) {
color: Colors.grey[600], final targetSize = controller.isSubtitleMode.value
size: 24, ? MediaQuery.of(context).size.width * 0.55 // 字幕模式时较小
), : MediaQuery.of(context).size.width * 0.75; // 初始状态更大
const SizedBox(width: 20),
// 设置按钮 return TweenAnimationBuilder<double>(
Icon( tween: Tween(end: targetSize),
Icons.tune, duration: const Duration(milliseconds: 600),
color: Colors.grey[600], curve: Curves.easeInOutCubic,
size: 24, builder: (context, animatedSize, child) {
return Stack(
alignment: Alignment.center,
children: [
// 使用Shader动画
Obx(() => ShaderMicFlow(
size: animatedSize,
amplitude: controller.audioAmplitude.value,
)),
],
);
},
);
},
), ),
], )),
), ),
),
// 状态和字幕显示区域 // 字幕显示区域 - 只有在字幕模式开启时才显示,位于圆形下方
Positioned( if (controller.isSubtitleMode.value)
top: 100, Positioned(
left: 20, left: 20,
right: 20, right: 20,
child: Column( top: MediaQuery.of(context).size.height * 0.35, // 在圆形下方
children: [ bottom: 160,
// 当前状态显示 child: Padding(
Obx(() => Container( padding: const EdgeInsets.symmetric(horizontal: 24),
width: double.infinity, child: Obx(() => AnimatedOpacity(
padding: const EdgeInsets.symmetric(horizontal: 16, vertical: 12), opacity: controller.isSubtitleMode.value ? 1.0 : 0.0,
margin: const EdgeInsets.only(bottom: 12), duration: const Duration(milliseconds: 300),
decoration: BoxDecoration( child: TweenAnimationBuilder<double>(
color: Colors.black.withOpacity(0.1), tween: Tween(
borderRadius: BorderRadius.circular(20), begin: 0.0,
), end: controller.isSubtitleMode.value ? 1.0 : 0.0,
child: Text( ),
controller.currentStatus.value, duration: const Duration(milliseconds: 900), // 延迟显示,让圆圈先移动
textAlign: TextAlign.center, curve: Curves.easeInOut,
style: const TextStyle( builder: (context, animationValue, child) {
fontSize: 16, return Transform.translate(
fontWeight: FontWeight.w500, offset: Offset(0, 20 * (1 - animationValue)), // 从下方滑入
color: Colors.black87, child: Opacity(
opacity: animationValue,
child: _buildChatSubtitles(),
),
);
},
), ),
), )),
)), ),
),
// 用户ASR内容显示
Obx(() => AnimatedContainer( // 底部控制按钮
duration: const Duration(milliseconds: 300), Positioned(
width: double.infinity, bottom: 80,
height: controller.userSpeechText.value.isNotEmpty ? null : 0, left: 0,
padding: controller.userSpeechText.value.isNotEmpty right: 0,
? const EdgeInsets.symmetric(horizontal: 16, vertical: 12) child: Center(
: EdgeInsets.zero, child: Row(
margin: controller.userSpeechText.value.isNotEmpty mainAxisAlignment: MainAxisAlignment.spaceEvenly,
? const EdgeInsets.only(bottom: 8) children: [
: EdgeInsets.zero, // 麦克风按钮
decoration: BoxDecoration( Obx(() => _buildMicButton()),
color: Colors.blue.withOpacity(0.1), // 字幕按钮
borderRadius: BorderRadius.circular(16), Obx(() => _buildMainButton(
border: Border.all( icon: controller.isSubtitleMode.value ? Icons.subtitles : Icons.subtitles_outlined,
color: Colors.blue.withOpacity(0.3), onTap: controller.toggleSubtitleMode,
width: 1, enabled: true,
isActive: controller.isSubtitleMode.value,
)),
// 清空聊天按钮
_buildMainButton(
icon: Icons.clear_all,
onTap: controller.clearChat,
enabled: true,
isActive: false,
), ),
), // 关闭按钮
child: controller.userSpeechText.value.isNotEmpty _buildMainButton(
? Column( icon: Icons.close,
crossAxisAlignment: CrossAxisAlignment.start, onTap: controller.closeAndGoBack,
children: [ enabled: true,
const Text( isActive: false,
'你说:',
style: TextStyle(
fontSize: 12,
fontWeight: FontWeight.w600,
color: Colors.blue,
),
),
const SizedBox(height: 4),
Text(
controller.userSpeechText.value,
style: const TextStyle(
fontSize: 14,
color: Colors.black87,
),
),
],
)
: null,
)),
// AI回复内容显示
Obx(() => AnimatedContainer(
duration: const Duration(milliseconds: 300),
width: double.infinity,
height: controller.aiResponseText.value.isNotEmpty ? null : 0,
padding: controller.aiResponseText.value.isNotEmpty
? const EdgeInsets.symmetric(horizontal: 16, vertical: 12)
: EdgeInsets.zero,
margin: controller.aiResponseText.value.isNotEmpty
? const EdgeInsets.only(bottom: 20)
: EdgeInsets.zero,
decoration: BoxDecoration(
color: Colors.green.withOpacity(0.1),
borderRadius: BorderRadius.circular(16),
border: Border.all(
color: Colors.green.withOpacity(0.3),
width: 1,
), ),
), ],
child: controller.aiResponseText.value.isNotEmpty ),
? Column( ),
crossAxisAlignment: CrossAxisAlignment.start,
children: [
const Text(
'AI回复:',
style: TextStyle(
fontSize: 12,
fontWeight: FontWeight.w600,
color: Colors.green,
),
),
const SizedBox(height: 4),
Text(
controller.aiResponseText.value,
style: const TextStyle(
fontSize: 14,
color: Colors.black87,
),
),
],
)
: null,
)),
],
), ),
),
// 中间会呼吸的云彩动画 // Home Indicator
Positioned.fill( Positioned(
child: Align( bottom: 8,
alignment: const Alignment(0, 0.1), left: 0,
child: const VoiceMicFlow(size: 280), right: 0,
child: Center(
child: Container(
width: 134,
height: 5,
decoration: BoxDecoration(
color: Colors.black,
borderRadius: BorderRadius.circular(2.5),
),
),
),
), ),
), ],
),
),
);
}
// 底部控制按钮 /// 构建聊天字幕界面 - 上下两行显示用户输入和AI回复
Positioned( Widget _buildChatSubtitles() {
bottom: 80, return Obx(() {
left: 0, // 组合用户消息:优先显示实时部分转录,否则显示最终消息
right: 0, final String displayUserMessage = controller.partialUserMessage.value.isNotEmpty
child: Padding( ? controller.partialUserMessage.value
padding: const EdgeInsets.symmetric(horizontal: 40), : controller.latestUserMessage.value;
child: Row(
mainAxisAlignment: MainAxisAlignment.spaceBetween, // 组合AI消息:优先显示流式响应,否则显示最终消息
children: [ final String displayAiMessage = controller.streamingAiMessage.value.isNotEmpty
// 摄像头按钮 ? controller.streamingAiMessage.value
_buildBottomButton(Icons.videocam, null), : controller.latestAiMessage.value;
// 麦克风按钮
_buildBottomButton(Icons.mic_none, controller.toggleListening), final hasUserMessage = displayUserMessage.isNotEmpty;
// 更多按钮 final hasAiMessage = displayAiMessage.isNotEmpty;
_buildBottomButton(Icons.more_horiz, null),
// 关闭按钮 // 如果没有消息,显示空白
_buildBottomButton(Icons.close, controller.closeAndGoBack), if (!hasUserMessage && !hasAiMessage) {
], return Container();
}
return Column(
mainAxisAlignment: MainAxisAlignment.center,
crossAxisAlignment: CrossAxisAlignment.center,
mainAxisSize: MainAxisSize.min, // 让Column紧凑排列
children: [
// 用户输入行 - 蓝色,动态扩展
if (hasUserMessage)
Container(
width: double.infinity,
constraints: const BoxConstraints(
minHeight: 30, // 最小1行高度
maxHeight: 104, // 最大4行高度 (26*4)
),
padding: const EdgeInsets.symmetric(vertical: 4),
child: SingleChildScrollView(
reverse: true,
child: Text(
displayUserMessage,
textAlign: TextAlign.center,
style: TextStyle(
fontSize: 18,
color: Colors.blue[600],
height: 1.4,
fontWeight: FontWeight.w500,
letterSpacing: 0.3,
),
), ),
), ),
), ),
// Home Indicator // 分隔线 - 只有在两个消息都存在时才显示
Positioned( if (hasUserMessage && hasAiMessage)
bottom: 8, Container(
left: 0, width: 60,
right: 0, height: 1,
child: Center( margin: const EdgeInsets.symmetric(vertical: 2), // 进一步减少分隔线间距
child: Container( decoration: BoxDecoration(
width: 134, color: Colors.grey[300],
height: 5, borderRadius: BorderRadius.circular(0.5),
decoration: BoxDecoration( ),
color: Colors.black, ),
borderRadius: BorderRadius.circular(2.5),
// AI回复行 - 绿色,动态扩展
if (hasAiMessage)
Container(
width: double.infinity,
constraints: const BoxConstraints(
minHeight: 30, // 最小1行高度
maxHeight: 104, // 最大4行高度 (26*4)
),
padding: const EdgeInsets.symmetric(vertical: 4),
child: SingleChildScrollView(
reverse: true,
child: Text(
displayAiMessage,
textAlign: TextAlign.center,
style: TextStyle(
fontSize: 18,
color: Colors.green[600],
height: 1.4,
fontWeight: FontWeight.w500,
letterSpacing: 0.3,
), ),
), ),
), ),
), ),
], ],
);
});
}
/// 构建麦克风按钮 - 根据连接状态和录音状态显示不同的状态
Widget _buildMicButton() {
// 获取当前状态
final isConnected = controller.isConnected.value;
final isConnecting = controller.isConnecting.value;
final isListening = controller.isListening.value;
// 确定按钮状态
IconData icon;
Color? backgroundColor;
Color? iconColor;
VoidCallback? onTap;
if (!isConnected && !isConnecting) {
// 未连接状态 - 禁用
icon = Icons.mic_off;
backgroundColor = Colors.grey[300];
iconColor = Colors.grey[600];
onTap = null;
} else if (isConnecting) {
// 连接中状态 - 禁用但显示连接中
icon = Icons.sync;
backgroundColor = Colors.orange[200];
iconColor = Colors.orange[700];
onTap = null;
} else if (isListening) {
// 录音中状态 - 激活状态
icon = Icons.mic;
backgroundColor = Theme.of(Get.context!).primaryColor;
iconColor = Colors.white;
onTap = controller.toggleListening;
} else {
// 已连接但未录音状态 - 正常状态
icon = Icons.mic_none;
backgroundColor = Colors.grey[200];
iconColor = Colors.black;
onTap = controller.toggleListening;
}
return GestureDetector(
onTap: onTap,
child: Container(
width: 60,
height: 60,
decoration: BoxDecoration(
color: backgroundColor,
shape: BoxShape.circle,
),
child: Icon(
icon,
color: iconColor,
size: 26,
), ),
), ),
); );
} }
/// 构建底部圆形按钮 /// 构建主要控制按钮
Widget _buildBottomButton(IconData icon, VoidCallback? onTap) { Widget _buildMainButton({
required IconData icon,
VoidCallback? onTap,
required bool enabled,
required bool isActive,
}) {
return GestureDetector( return GestureDetector(
onTap: onTap, onTap: enabled ? onTap : null,
child: Container( child: Container(
width: 60, width: 60,
height: 60, height: 60,
decoration: BoxDecoration( decoration: BoxDecoration(
color: Colors.grey[200], color: enabled
? (isActive ? Theme.of(Get.context!).primaryColor : Colors.grey[200])
: Colors.grey[300],
shape: BoxShape.circle, shape: BoxShape.circle,
), ),
child: Icon( child: Icon(
icon, icon,
color: Colors.black, color: enabled
? (isActive ? Colors.white : Colors.black)
: Colors.grey[600],
size: 26, size: 26,
), ),
), ),
); );
} }
} }

104
lib/modules/realtime/views/widgets/shader_mic_flow.dart

@ -0,0 +1,104 @@
import 'dart:ui';
import 'package:flutter/material.dart';
/// 使用Shader实现的麦克风流动动画组件
class ShaderMicFlow extends StatefulWidget {
final double size;
final double amplitude;
const ShaderMicFlow({
super.key,
required this.size,
this.amplitude = 0.0,
});
@override
State<ShaderMicFlow> createState() => _ShaderMicFlowState();
}
class _ShaderMicFlowState extends State<ShaderMicFlow>
with SingleTickerProviderStateMixin {
late AnimationController _controller;
FragmentShader? _shader;
@override
void initState() {
super.initState();
_controller = AnimationController(
duration: const Duration(seconds: 60),
vsync: this,
)..repeat();
_loadShader();
}
Future<void> _loadShader() async {
try {
final program = await FragmentProgram.fromAsset('shaders/mic_flow.frag');
setState(() {
_shader = program.fragmentShader();
});
} catch (e) {
debugPrint('Failed to load shader: $e');
}
}
@override
void dispose() {
_controller.dispose();
_shader?.dispose();
super.dispose();
}
@override
Widget build(BuildContext context) {
return SizedBox(
width: widget.size,
height: widget.size,
child: AnimatedBuilder(
animation: _controller,
builder: (context, child) {
return CustomPaint(
painter: _shader != null
? _ShaderPainter(
shader: _shader!,
time: _controller.value * 60.0,
amplitude: widget.amplitude,
)
: null,
size: Size.square(widget.size),
);
},
),
);
}
}
class _ShaderPainter extends CustomPainter {
final FragmentShader shader;
final double time;
final double amplitude;
_ShaderPainter({
required this.shader,
required this.time,
required this.amplitude,
});
@override
void paint(Canvas canvas, Size size) {
// 设置shader uniforms
shader.setFloat(0, size.width); // u_resolution.x
shader.setFloat(1, size.height); // u_resolution.y
shader.setFloat(2, time); // u_time
shader.setFloat(3, amplitude); // u_amp
final paint = Paint()..shader = shader;
canvas.drawRect(Offset.zero & size, paint);
}
@override
bool shouldRepaint(covariant _ShaderPainter oldDelegate) {
return oldDelegate.time != time || oldDelegate.amplitude != amplitude;
}
}

89
lib/modules/realtime/views/widgets/voice_mic_flow.dart

@ -1,89 +0,0 @@
import 'dart:ui';
import 'package:flutter/material.dart';
import 'package:flutter_shaders/flutter_shaders.dart';
class VoiceMicFlow extends StatefulWidget {
const VoiceMicFlow({super.key, this.size = 280, this.amp = 0.0});
final double size;
final double amp; // 音量振幅 0.0-1.0
@override
State<VoiceMicFlow> createState() => _VoiceMicFlowState();
}
class _VoiceMicFlowState extends State<VoiceMicFlow>
with SingleTickerProviderStateMixin {
late final AnimationController _ctl;
@override
void initState() {
super.initState();
_ctl = AnimationController(
duration: const Duration(seconds: 100), // 100秒的长周期
vsync: this,
)..repeat();
}
@override
void dispose() {
_ctl.dispose();
super.dispose();
}
@override
Widget build(BuildContext context) {
return ClipOval(
child: ShaderBuilder(
assetKey: 'shaders/mic_flow.frag',
(context, shader, child) {
return AnimatedBuilder(
animation: _ctl,
builder: (_, __) {
// 将0.0-1.0的动画值映射到更大的时间范围
final time = _ctl.value * 100.0;
shader
..setFloat(0, widget.size) // u_resolution.x
..setFloat(1, widget.size) // u_resolution.y
..setFloat(2, time) // u_time
..setFloat(3, widget.amp); // u_amp (音量振幅)
return CustomPaint(
size: Size.square(widget.size),
painter: _ShaderPainter(shader),
);
},
);
},
// 着色器编译时的占位符
child: Container(
width: widget.size,
height: widget.size,
decoration: const BoxDecoration(
shape: BoxShape.circle,
gradient: LinearGradient(
begin: Alignment.topLeft,
end: Alignment.bottomRight,
colors: [
Color(0xFF1565C0), // 深蓝
Color(0xFFE3F2FD), // 浅蓝
],
),
),
),
),
);
}
}
class _ShaderPainter extends CustomPainter {
_ShaderPainter(this.shader);
final FragmentShader shader;
@override
void paint(Canvas canvas, Size size) {
final paint = Paint()..shader = shader;
canvas.drawRect(Offset.zero & size, paint);
}
@override
bool shouldRepaint(covariant _ShaderPainter oldDelegate) => true;
}

114
local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/AzureAsrHelper.kt

@ -64,6 +64,10 @@ class AzureAsrHelper(private val context: Context) {
// 音频处理 // 音频处理
private var microphoneStream: MicrophoneStream? = null private var microphoneStream: MicrophoneStream? = null
private var externalAudioStream: ExternalAudioPullStream? = null private var externalAudioStream: ExternalAudioPullStream? = null
// 音频模式管理
private var audioManager: AudioManager? = null
private var originalAudioMode = AudioManager.MODE_NORMAL
// 录音文件处理 // 录音文件处理
var recordfile: RecordingFile? = null var recordfile: RecordingFile? = null
@ -104,6 +108,10 @@ class AzureAsrHelper(private val context: Context) {
// 释放之前的资源 // 释放之前的资源
dispose() dispose()
// 初始化音频管理器
audioManager = context.getSystemService(Context.AUDIO_SERVICE) as AudioManager
originalAudioMode = audioManager?.mode ?: AudioManager.MODE_NORMAL
// 保存配置 // 保存配置
this.subscriptionKey = subscriptionKey this.subscriptionKey = subscriptionKey
@ -171,36 +179,9 @@ class AzureAsrHelper(private val context: Context) {
// 设置音频配置 // 设置音频配置
when (audioSourceType) { when (audioSourceType) {
AudioSourceType.MICROPHONE -> { AudioSourceType.MICROPHONE -> {
// 判断是否使用手机扬声器输出和耳机连接状态 // 总是使用拉流方式并启用回音消除
val audioManager = Log.i(tag, "使用拉流方式并启用回音消除")
context.getSystemService(Context.AUDIO_SERVICE) as AudioManager setupMicrophoneStream()
val isSpeakerphoneOn = audioManager.isSpeakerphoneOn
// 检查是否连接了有线耳机
val isWiredHeadsetOn = audioManager.isWiredHeadsetOn
// 检查是否连接了蓝牙设备(需要注意兼容性)
val isBluetoothConnected = try {
val method = AudioManager::class.java.getMethod("isBluetoothA2dpOn")
method.invoke(audioManager) as Boolean
} catch (e: Exception) {
false
}
val isHeadsetConnected = isWiredHeadsetOn || isBluetoothConnected
// 如果使用手机扬声器输出且没有连接耳机,可能会产生回音,需启动回音消除功能
val needEchoCancellation = !isHeadsetConnected
if (needEchoCancellation) {
Log.i(tag, "使用拉流方式进行回音消除")
// 使用拉流方式进行回音消除
setupMicrophoneStream()
} else {
Log.i(tag, "使用拉流方式进行回音消除2")
audioConfig = AudioConfig.fromDefaultMicrophoneInput()
setupMicrophoneStream()
}
} }
AudioSourceType.EXTERNAL -> { AudioSourceType.EXTERNAL -> {
@ -651,6 +632,8 @@ else{
private inner class MicrophoneStream : PullAudioInputStreamCallback() { private inner class MicrophoneStream : PullAudioInputStreamCallback() {
private var audioRecord: AudioRecord? = null private var audioRecord: AudioRecord? = null
private var echoCanceler: AcousticEchoCanceler? = null private var echoCanceler: AcousticEchoCanceler? = null
private var noiseSuppressor: NoiseSuppressor? = null
private var automaticGainControl: AutomaticGainControl? = null
// 音频配置 // 音频配置
private val sampleRate = 16000 private val sampleRate = 16000
@ -715,19 +698,45 @@ else{
} }
/** /**
* 优化:条件性应用音频效果 * 应用音频效果 - 总是启用所有音频增强功能
*/ */
private fun applyAudioEffects() { private fun applyAudioEffects() {
val sessionId = audioRecord?.audioSessionId ?: return val sessionId = audioRecord?.audioSessionId ?: return
// 优化:只在真正需要时启用回音消除 try {
val audioManager = context.getSystemService(Context.AUDIO_SERVICE) as AudioManager // 设置音频模式为通信模式
if (audioManager.isSpeakerphoneOn) { this@AzureAsrHelper.audioManager?.mode = AudioManager.MODE_IN_COMMUNICATION
echoCanceler = AcousticEchoCanceler.create(sessionId)
echoCanceler?.enabled = true // 启用声学回音消除
Log.d(tag, "启用回音消除") if (AcousticEchoCanceler.isAvailable()) {
} else { echoCanceler = AcousticEchoCanceler.create(sessionId)
Log.d(tag, "跳过回音消除(使用耳机)") echoCanceler?.enabled = true
Log.i(tag, "声学回音消除已启用")
} else {
Log.w(tag, "设备不支持声学回音消除")
}
// 启用噪声抑制
if (NoiseSuppressor.isAvailable()) {
noiseSuppressor = NoiseSuppressor.create(sessionId)
noiseSuppressor?.enabled = true
Log.i(tag, "噪声抑制已启用")
} else {
Log.w(tag, "设备不支持噪声抑制")
}
// 启用自动增益控制
if (AutomaticGainControl.isAvailable()) {
automaticGainControl = AutomaticGainControl.create(sessionId)
automaticGainControl?.enabled = true
Log.i(tag, "自动增益控制已启用")
} else {
Log.w(tag, "设备不支持自动增益控制")
}
Log.i(tag, "回音消除和音频增强功能已全部启用")
} catch (e: Exception) {
Log.e(tag, "设置音频效果失败: ${e.message}")
} }
} }
@ -777,13 +786,34 @@ else{
} }
// 释放回音消除器 // 释放回音消除器
echoCanceler?.release() echoCanceler?.let {
echoCanceler = null it.enabled = false
it.release()
echoCanceler = null
}
// 释放噪声抑制器
noiseSuppressor?.let {
it.enabled = false
it.release()
noiseSuppressor = null
}
// 释放自动增益控制
automaticGainControl?.let {
it.enabled = false
it.release()
automaticGainControl = null
}
// 恢复音频模式
this@AzureAsrHelper.audioManager?.mode = this@AzureAsrHelper.originalAudioMode
// 释放录音实例 // 释放录音实例
audioRecord?.release() audioRecord?.release()
audioRecord = null audioRecord = null
} catch (e: Exception) { } catch (e: Exception) {
Log.e(tag, "释放音频资源失败: ${e.message}")
e.printStackTrace() e.printStackTrace()
} }
} }

44
local_plugins/azure_speech/android/src/main/kotlin/com/yunqiinnovation/azure_speech/AzureTtsHelper.kt

@ -71,15 +71,21 @@ class AzureTtsHelper(private val context: Context) : ITtsService {
// 创建语音配置 // 创建语音配置
speechConfig = SpeechConfig.fromSubscription(ttsAppToken, ttsResource) speechConfig = SpeechConfig.fromSubscription(ttsAppToken, ttsResource)
// 优化:使用更低延迟的音频格式(从24KHz降到16KHz) // 使用标准的音频格式(16KHz, 16bit, 单声道)
speechConfig?.setSpeechSynthesisOutputFormat(SpeechSynthesisOutputFormat.Riff16Khz16BitMonoPcm) speechConfig?.setSpeechSynthesisOutputFormat(SpeechSynthesisOutputFormat.Riff16Khz16BitMonoPcm)
// 确保使用正确的语音输出
speechConfig?.setProperty("SPEECH-AudioOutputFormat", "riff-16khz-16bit-mono-pcm")
// 优化:设置低延迟连接属性 // 优化:设置低延迟连接属性
speechConfig?.setProperty("SpeechServiceConnection_InitialSilenceTimeoutMs", "300") speechConfig?.setProperty("SpeechServiceConnection_InitialSilenceTimeoutMs", "300")
speechConfig?.setProperty("SpeechServiceConnection_EndSilenceTimeoutMs", "300") speechConfig?.setProperty("SpeechServiceConnection_EndSilenceTimeoutMs", "300")
speechConfig?.setSpeechSynthesisVoiceName(currentVoice) speechConfig?.setSpeechSynthesisVoiceName(currentVoice)
// 添加日志记录语音格式
FileLogger.d(TAG, "语音设置: voice=$currentVoice, format=Riff16Khz16BitMonoPcm")
// 创建音频配置 // 创建音频配置
val audioConfig = if (customAudioOutputStream != null) { val audioConfig = if (customAudioOutputStream != null) {
// 使用自定义音频输出流 // 使用自定义音频输出流
@ -185,21 +191,47 @@ class AzureTtsHelper(private val context: Context) : ITtsService {
private fun setupEventListeners() { private fun setupEventListeners() {
synthesizer?.apply { synthesizer?.apply {
// 合成开始事件 // 合成开始事件
SynthesisStarted?.addEventListener { _, _ -> SynthesisStarted?.addEventListener { _, eventArgs ->
FileLogger.d(TAG, "语音合成开始: resultId=${eventArgs.result.resultId}")
notifyEvent(TtsEventType.SYNTHESIS_STARTED) notifyEvent(TtsEventType.SYNTHESIS_STARTED)
} }
// 合成中事件(接收音频数据)
Synthesizing?.addEventListener { _, eventArgs ->
val audioData = eventArgs.result.audioData
FileLogger.d(TAG, "接收到音频数据: ${audioData.size} 字节")
// 通知音频数据监听器
if (audioData.isNotEmpty()) {
val listeners = ArrayList(audioDataListeners)
for (listener in listeners) {
try {
listener.onAudioData(audioData)
} catch (e: Exception) {
FileLogger.e(TAG, "通知音频数据异常: ${e.message}")
}
}
}
}
// 合成完成事件 // 合成完成事件
SynthesisCompleted?.addEventListener { _, _ -> SynthesisCompleted?.addEventListener { _, eventArgs ->
FileLogger.d(TAG, "语音合成完成: resultId=${eventArgs.result.resultId}, 音频长度=${eventArgs.result.audioLength} 字节")
isSpeaking = false isSpeaking = false
notifyEvent(TtsEventType.SYNTHESIS_COMPLETED) notifyEvent(TtsEventType.SYNTHESIS_COMPLETED)
} }
// 合成取消事件 // 合成取消事件
SynthesisCanceled?.addEventListener { _, eventArgs -> SynthesisCanceled?.addEventListener { _, eventArgs ->
FileLogger.d(TAG, "语音合成取消") val reason = eventArgs.result.reason
FileLogger.e(TAG, "语音合成取消: reason=$reason")
if (reason == ResultReason.Canceled) {
val cancellation = SpeechSynthesisCancellationDetails.fromResult(eventArgs.result)
FileLogger.e(TAG, "取消详情: reason=${cancellation.reason}, errorCode=${cancellation.errorCode}, errorDetails=${cancellation.errorDetails}")
}
isSpeaking = false isSpeaking = false
notifyEvent(TtsEventType.SYNTHESIS_CANCELED) notifyEvent(TtsEventType.SYNTHESIS_CANCELED)
} }
} }
@ -397,6 +429,8 @@ class AzureTtsHelper(private val context: Context) : ITtsService {
isSpeaking = true isSpeaking = true
FileLogger.d(TAG, "开始语音合成,文本长度: ${text.length}")
// 直接调用SpeakSsmlAsync,SDK内部已经是异步的 // 直接调用SpeakSsmlAsync,SDK内部已经是异步的
synthesizer?.SpeakSsmlAsync(ssml) synthesizer?.SpeakSsmlAsync(ssml)

Loading…
Cancel
Save