fdp
1 year ago
13 changed files with 780 additions and 116 deletions
@ -0,0 +1,160 @@ |
|||
# Realtime 实时语音聊天模块 |
|||
|
|||
## ⚠️ 当前状态 |
|||
|
|||
**此模块为功能预览版本，提供完整的实时语音对话功能。** |
|||
|
|||
通过原生插件实现语音识别、AI对话和语音合成的完整语音交互体验。 |
|||
|
|||
## 概述 |
|||
|
|||
Realtime模块实现了类似ChatGPT app的实时语音对话功能，当前版本提供： |
|||
|
|||
- ✅ **完整语音交互** - 实时语音识别、AI对话和语音合成 |
|||
- ✅ **状态管理** - 聆听、思考、回答状态的可视化 |
|||
- ✅ **动画效果** - 基于Shader的流动云效果，支持音频振幅驱动 |
|||
- ✅ **字幕显示** - 可选择开启/关闭的字幕模式 |
|||
- ✅ **智能UI** - 根据连接状态自动调整按钮可用性 |
|||
|
|||
## 界面设计 |
|||
|
|||
### 布局结构 |
|||
- **顶部状态区域** - 显示当前状态和字幕模式切换按钮 |
|||
- **中间可视化区域** - 圆形Shader动画，根据音频振幅呈现呼吸效果 |
|||
- **底部控制区域** - 主控制按钮和状态指示器 |
|||
|
|||
### 状态指示 |
|||
- 🔵 **聆听状态** - 蓝色动画，接收用户语音输入 |
|||
- 🟠 **思考状态** - 橙色动画，AI处理中 |
|||
- 🟢 **回答状态** - 绿色动画，AI语音输出 |
|||
- ⚪ **空闲状态** - 灰色静态，等待交互 |
|||
|
|||
### 字幕模式 |
|||
- **默认关闭** - 初始状态下不显示任何字幕内容 |
|||
- **手动切换** - 通过右上角按钮手动开启/关闭字幕 |
|||
- **完整隐藏** - 关闭时字幕区域完全不显示，不占用界面空间 |
|||
- **智能布局** - 开启时圆形动画自动上移，字幕区域占据下半屏 |
|||
|
|||
### 交互控制 |
|||
- **录音按钮** - 仅在成功连接AI服务器且开始录音后可用 |
|||
- **连接状态** - 未连接时录音按钮显示不可用状态 |
|||
- **状态反馈** - 按钮颜色和可点击性根据连接状态动态调整 |
|||
|
|||
## 使用方法 |
|||
|
|||
### 路由导航 |
|||
|
|||
```dart |
|||
// 跳转到实时语音对话页面 |
|||
Get.toNamed(Routes.realtime); |
|||
``` |
|||
|
|||
### 交互流程 |
|||
|
|||
1. **自动连接** - 页面加载后自动尝试连接AI服务器 |
|||
2. **等待连接** - 录音按钮在连接成功前保持不可用状态 |
|||
3. **开始录音** - 连接成功后自动开始录音，录音按钮变为可用 |
|||
4. **语音对话** - 用户说话 → AI处理 → 语音回复 |
|||
5. **字幕控制** - 可随时通过右上角按钮切换字幕显示 |
|||
|
|||
## 技术架构 |
|||
|
|||
### 控制器 (RealtimeController) |
|||
- **连接管理**: WebSocket连接到AI服务器 |
|||
- **状态同步**: `isConnected`, `isListening`, `isSpeaking` |
|||
- **字幕控制**: `isSubtitleMode` 默认为false，用户手动控制 |
|||
- **消息管理**: 智能去重，避免流式响应重复显示 |
|||
- **音频处理**: 原生层计算RMS值，Flutter层接收并控制动画 |
|||
|
|||
### 视图组件 |
|||
- **RealtimeView** - 主视图容器，响应式布局 |
|||
- **ShaderMicFlow** - 基于GLSL Shader的流动云动画组件 |
|||
- **AutoScrollList** - 自动滚动的字幕列表 |
|||
|
|||
### 原生集成 |
|||
- **RealtimeService** - 原生音频处理和WebSocket通信 |
|||
- **音频格式** - 16kHz, 1声道, 16位PCM |
|||
- **实时传输** - 低延迟音频流处理 |
|||
- **RMS计算** - 原生层实时计算音频振幅 |
|||
|
|||
## 默认行为 |
|||
|
|||
### 字幕模式 |
|||
- **初始状态**: 关闭 (`isSubtitleMode.value = false`) |
|||
- **显示逻辑**: 仅在用户手动开启时显示字幕内容 |
|||
- **空间占用**: 关闭时字幕区域高度为0，不影响其他UI元素 |
|||
- **切换效果**: 平滑动画过渡，圆形动画位置和大小同步调整 |
|||
|
|||
### 按钮状态 |
|||
- **录音按钮**: 默认不可用，连接成功且开始录音后才可点击 |
|||
- **状态指示**: 通过颜色和图标变化反映当前可用性 |
|||
- **交互反馈**: 不可用时点击无效果，避免误操作 |
|||
|
|||
## 开发计划 |
|||
|
|||
### 已完成功能 ✅ |
|||
- [x] 完整语音交互流程 |
|||
- [x] 实时WebSocket通信 |
|||
- [x] Shader流动云动画 |
|||
- [x] RMS音频振幅计算 |
|||
- [x] 智能UI状态管理 |
|||
- [x] 字幕模式控制 |
|||
- [x] 消息去重优化 |
|||
|
|||
### 计划增强 ⏳ |
|||
- [ ] 多语言语音识别 |
|||
- [ ] 语音中断检测 |
|||
- [ ] 音频质量自适应 |
|||
- [ ] 离线降级处理 |
|||
- [ ] 语音情感识别 |
|||
|
|||
## 注意事项 |
|||
|
|||
1. **网络依赖** - 需要连接到AI服务器 (ws://192.168.1.11:8000/ws) |
|||
2. **麦克风权限** - 需要获取麦克风使用权限 |
|||
3. **音频会话** - 自动管理音频会话，支持蓝牙设备 |
|||
4. **资源管理** - 页面关闭时自动清理音频资源和网络连接 |
|||
|
|||
## 自定义配置 |
|||
|
|||
### 修改服务器地址 |
|||
|
|||
```dart |
|||
// 在RealtimeController中修改连接URL |
|||
await _realtimeService.initialize( |
|||
serverUrl: 'ws://your-server:port/ws', |
|||
// ... 其他参数 |
|||
); |
|||
``` |
|||
|
|||
### 调整UI布局 |
|||
|
|||
```dart |
|||
// 修改字幕模式下圆形动画位置 |
|||
alignment: controller.isSubtitleMode.value |
|||
? const Alignment(0, -0.9) // 调整Y轴位置 |
|||
: const Alignment(0, -0.25), |
|||
``` |
|||
|
|||
### 音频参数调优 |
|||
|
|||
```dart |
|||
// 在初始化时调整音频参数 |
|||
sampleRate: 16000, // 采样率 |
|||
channels: 1, // 声道数 |
|||
bitsPerSample: 16, // 位深度 |
|||
``` |
|||
|
|||
## 技术栈 |
|||
|
|||
- **Flutter SDK**: >= 3.0.0 |
|||
- **GetX**: >= 4.6.5 (状态管理) |
|||
- **Flutter Shaders**: >= 0.0.6 (Shader动画) |
|||
- **原生插件**: Realtime服务 (音频处理) |
|||
- **WebSocket**: 实时通信协议 |
|||
|
|||
## 版本信息 |
|||
|
|||
- **当前版本**: v1.0.0 (Production) |
|||
- **最后更新**: 2024年 |
|||
- **状态**: 完整功能实现 |
|||
@ -0,0 +1,10 @@ |
|||
import 'package:get/get.dart'; |
|||
import '../controllers/FTFTranslation_controller.dart'; |
|||
|
|||
class FTFTranslationBinding extends Bindings { |
|||
@override |
|||
void dependencies() { |
|||
// 注册实时语音聊天控制器 |
|||
Get.lazyPut<FTFTranslationController>(() => FTFTranslationController()); |
|||
} |
|||
} |
|||
@ -0,0 +1,241 @@ |
|||
import 'dart:async'; |
|||
import 'dart:typed_data'; |
|||
|
|||
import 'package:get/get.dart'; |
|||
import '../../../core/utils/logger.dart'; |
|||
import '../../../data/services/asr_service.dart'; |
|||
import '../../../data/services/speech_impl/xunfei_asr_service.dart'; |
|||
import '../../../data/services/tts_service.dart'; |
|||
import '../../../data/services/language_manager.dart'; |
|||
import '../../../data/services/volcano_translation_service.dart'; |
|||
|
|||
class FTFTranslationController extends GetxController |
|||
with GetSingleTickerProviderStateMixin { |
|||
final messages = <FTFTranslationMessage>[].obs; |
|||
final isChineseInput = true.obs; |
|||
final leftLanguage = '中文'.obs; |
|||
final rightLanguage = 'English'.obs; |
|||
final isLeftInput = true.obs; |
|||
|
|||
final supportedLanguages = const [ |
|||
'中文', |
|||
'English', |
|||
'日本語', |
|||
'한국어', |
|||
'Español', |
|||
]; |
|||
|
|||
late final AsrService _asrService; |
|||
|
|||
StreamSubscription? _recognitionSubscription; |
|||
|
|||
RxString intermediateContent = ''.obs; |
|||
RxString finalContent = ''.obs; |
|||
BytesBuilder _originaBytes = BytesBuilder(); //原始音频 |
|||
|
|||
final VolcanoTranslationService _translationService = |
|||
Get.find<VolcanoTranslationService>(); |
|||
final LanguageManager _languageManager = Get.find<LanguageManager>(); |
|||
// 当前使用的TTS服务 |
|||
late TtsService _ttsService; |
|||
final isTranslating = false.obs; |
|||
final sourceLanguageCode = 'zh-CN'.obs; |
|||
final targetLanguageCode = 'en-US'.obs; |
|||
final isTtsEnabled = true.obs; |
|||
|
|||
@override |
|||
void onInit() { |
|||
super.onInit(); |
|||
Get.put(XunfeiAsrService()); |
|||
final TtsService _ttsService = Get.find<TtsService>(); |
|||
_asrService = Get.find<XunfeiAsrService>(); |
|||
} |
|||
|
|||
@override |
|||
void onClose() { |
|||
_asrService.dispose(); |
|||
_recognitionSubscription?.cancel(); |
|||
super.onClose(); |
|||
} |
|||
|
|||
//开始录音 |
|||
Future<void> _startRecorder() async { |
|||
final recognitionStream = |
|||
await _asrService.startContinuousRecognition(false); |
|||
_recognitionSubscription = recognitionStream.listen( |
|||
_handleRecognitionEvent, |
|||
); |
|||
} |
|||
|
|||
//结束录音 |
|||
Future<void> _stopRecorder() async { |
|||
await _asrService.stopContinuousRecognition(); |
|||
} |
|||
|
|||
// 处理语音识别事件 |
|||
void _handleRecognitionEvent(RecognitionEvent event) { |
|||
switch (event.type) { |
|||
case RecognitionEventType.intermediateResult: |
|||
intermediateContent.value = event.text; |
|||
handleFinalResult(event.text); |
|||
break; |
|||
case RecognitionEventType.finalResult: |
|||
finalContent.value += _spliceText(event); |
|||
intermediateContent.value = ''; |
|||
_originaBytes.add(event.audio!); |
|||
handleIntermediateResult(event.text); |
|||
break; |
|||
default: |
|||
// 不做任何处理 |
|||
break; |
|||
} |
|||
} |
|||
|
|||
// 翻译文本 |
|||
Future<String?> translateText(String sourceText, |
|||
{bool isFinal = false}) async { |
|||
if (sourceText.isEmpty) return null; |
|||
|
|||
isTranslating.value = true; |
|||
String? translationResult; |
|||
|
|||
try { |
|||
// 根据检测到的语言确定源语言和目标语言 |
|||
final bool shouldSwap = |
|||
intermediateContent.value == sourceLanguageCode.value; |
|||
final detectedSourceLanguageCode = |
|||
shouldSwap ? sourceLanguageCode.value : targetLanguageCode.value; |
|||
final detectedTargetLanguageCode = |
|||
shouldSwap ? targetLanguageCode.value : sourceLanguageCode.value; |
|||
|
|||
// 调用翻译服务 |
|||
translationResult = await _translationService.translateText( |
|||
text: sourceText, |
|||
sourceLanguageCode: detectedSourceLanguageCode, |
|||
targetLanguageCode: detectedTargetLanguageCode, |
|||
); |
|||
|
|||
if (translationResult != null) { |
|||
// 查找匹配的项目 - 优先使用timestamp查找 |
|||
int index = -1; |
|||
|
|||
// 播放TTS - 只在最终结果时播放 |
|||
if (isTtsEnabled.value && isFinal) { |
|||
// Logger.info('播放翻译文本: ${translationResult}'); |
|||
await playTranslatedText(translationResult); |
|||
} |
|||
} |
|||
} catch (e) { |
|||
Logger.error('翻译失败: ${e.toString()}'); |
|||
} finally { |
|||
isTranslating.value = false; |
|||
} |
|||
|
|||
return translationResult; |
|||
} |
|||
|
|||
// 处理中间识别结果 |
|||
void handleIntermediateResult(String text) { |
|||
if (text.isEmpty) return; |
|||
|
|||
translateText(text, isFinal: false); |
|||
} |
|||
|
|||
// 处理最终识别结果 |
|||
Future<void> handleFinalResult(String text) async { |
|||
if (text.isEmpty) return; |
|||
|
|||
// 使用项目的时间戳进行翻译 |
|||
translateText(text, isFinal: true); |
|||
} |
|||
|
|||
// 播放翻译文本 |
|||
Future<void> playTranslatedText(String text) async { |
|||
if (text.isEmpty) return; |
|||
|
|||
try { |
|||
final bool shouldSwap = |
|||
intermediateContent.value == targetLanguageCode.value; |
|||
final detectedTargetLanguageCode = |
|||
shouldSwap ? sourceLanguageCode.value : targetLanguageCode.value; |
|||
final voiceName = _languageManager |
|||
.getTtsVoiceNameByAsrCode(detectedTargetLanguageCode) ?? |
|||
'en-US-AriaNeural'; |
|||
|
|||
await _ttsService.setVoice(voiceName); |
|||
await _ttsService.speakOnce(text); |
|||
} catch (e) { |
|||
Logger.error('播放翻译失败: ${e.toString()}'); |
|||
} |
|||
} |
|||
|
|||
String _spliceText(RecognitionEvent event) { |
|||
String eventText = event.text; |
|||
String text = ''; |
|||
if (finalContent.value.isNotEmpty) { |
|||
if (event.role != '') { |
|||
final punctuations = { |
|||
'，', |
|||
'。', |
|||
'！', |
|||
'？', |
|||
'；', |
|||
'：', |
|||
',', |
|||
'.', |
|||
':', |
|||
';', |
|||
'!', |
|||
'?' |
|||
}; |
|||
final firstChar = eventText[0]; |
|||
if (punctuations.contains(firstChar)) { |
|||
text += '$firstChar\n'; |
|||
eventText = eventText.substring(1); |
|||
} else { |
|||
text += '\n'; |
|||
} |
|||
text += '说话人${event.role}：'; |
|||
} |
|||
} else { |
|||
text += '说话人${event.role}：'; |
|||
} |
|||
text += eventText; |
|||
return text; |
|||
} |
|||
|
|||
void addMessage(String text) { |
|||
final fromLang = |
|||
isLeftInput.value ? leftLanguage.value : rightLanguage.value; |
|||
final toLang = isLeftInput.value ? rightLanguage.value : leftLanguage.value; |
|||
|
|||
messages.add( |
|||
FTFTranslationMessage( |
|||
original: text, |
|||
translated: '[$fromLang → $toLang] $text (翻译模拟)', |
|||
isChineseSpeaker: isLeftInput.value, |
|||
), |
|||
); |
|||
} |
|||
|
|||
void swapLanguages() { |
|||
final temp = leftLanguage.value; |
|||
leftLanguage.value = rightLanguage.value; |
|||
rightLanguage.value = temp; |
|||
isLeftInput.value = !isLeftInput.value; |
|||
} |
|||
|
|||
void toggleInputSide(bool isLeft) => isLeftInput.value = isLeft; |
|||
} |
|||
|
|||
class FTFTranslationMessage { |
|||
final String original; |
|||
final String translated; |
|||
final bool isChineseSpeaker; |
|||
|
|||
FTFTranslationMessage({ |
|||
required this.original, |
|||
required this.translated, |
|||
required this.isChineseSpeaker, |
|||
}); |
|||
} |
|||
@ -0,0 +1,212 @@ |
|||
import 'package:flutter/material.dart'; |
|||
import 'package:get/get.dart'; |
|||
import '../controllers/FTFTranslation_controller.dart'; |
|||
|
|||
class FTFTranslationView extends GetView<FTFTranslationController> { |
|||
const FTFTranslationView({Key? key}) : super(key: key); |
|||
|
|||
@override |
|||
Widget build(BuildContext context) { |
|||
final TextEditingController inputController = TextEditingController(); |
|||
|
|||
return Scaffold( |
|||
backgroundColor: const Color(0xFF0E0F12), |
|||
appBar: AppBar( |
|||
backgroundColor: Colors.black, |
|||
title: const Text('面对面翻译'), |
|||
centerTitle: true, |
|||
), |
|||
body: Column( |
|||
children: [ |
|||
Expanded( |
|||
child: Obx(() => ListView.builder( |
|||
padding: const EdgeInsets.all(16), |
|||
itemCount: controller.messages.length, |
|||
itemBuilder: (context, index) { |
|||
final msg = controller.messages[index]; |
|||
final isChinese = msg.isChineseSpeaker; |
|||
return Align( |
|||
alignment: isChinese |
|||
? Alignment.centerLeft |
|||
: Alignment.centerRight, |
|||
child: Container( |
|||
margin: const EdgeInsets.symmetric(vertical: 6), |
|||
padding: const EdgeInsets.all(12), |
|||
decoration: BoxDecoration( |
|||
color: isChinese ? Colors.blue : Colors.grey[800], |
|||
borderRadius: BorderRadius.circular(12), |
|||
), |
|||
constraints: const BoxConstraints(maxWidth: 300), |
|||
child: Column( |
|||
crossAxisAlignment: CrossAxisAlignment.start, |
|||
children: [ |
|||
Text( |
|||
msg.original, |
|||
style: const TextStyle( |
|||
color: Colors.white, |
|||
fontWeight: FontWeight.bold, |
|||
), |
|||
), |
|||
const SizedBox(height: 4), |
|||
Text( |
|||
msg.translated, |
|||
style: TextStyle( |
|||
color: Colors.white.withOpacity(0.9), |
|||
fontSize: 14, |
|||
), |
|||
), |
|||
], |
|||
), |
|||
), |
|||
); |
|||
}, |
|||
)), |
|||
), |
|||
_buildBottomInput(inputController), |
|||
], |
|||
), |
|||
); |
|||
} |
|||
|
|||
Widget _buildBottomInput(TextEditingController inputController) { |
|||
return Obx(() { |
|||
final isLeftInput = controller.isLeftInput.value; |
|||
final inputLang = isLeftInput |
|||
? controller.leftLanguage.value |
|||
: controller.rightLanguage.value; |
|||
|
|||
return Container( |
|||
color: Colors.black, |
|||
padding: const EdgeInsets.symmetric(horizontal: 12, vertical: 8), |
|||
child: Column( |
|||
children: [ |
|||
Row( |
|||
children: [ |
|||
Expanded( |
|||
child: TextField( |
|||
controller: inputController, |
|||
style: const TextStyle(color: Colors.white), |
|||
decoration: InputDecoration( |
|||
filled: true, |
|||
fillColor: Colors.grey[900], |
|||
hintText: '请输入 $inputLang 内容', |
|||
hintStyle: const TextStyle(color: Colors.grey), |
|||
border: OutlineInputBorder( |
|||
borderRadius: BorderRadius.circular(8), |
|||
borderSide: BorderSide.none, |
|||
), |
|||
contentPadding: |
|||
const EdgeInsets.symmetric(horizontal: 12), |
|||
), |
|||
), |
|||
), |
|||
IconButton( |
|||
onPressed: () { |
|||
final text = inputController.text.trim(); |
|||
if (text.isNotEmpty) { |
|||
controller.addMessage(text); |
|||
inputController.clear(); |
|||
} |
|||
}, |
|||
icon: const Icon(Icons.send, color: Colors.white), |
|||
), |
|||
], |
|||
), |
|||
const SizedBox(height: 8), |
|||
Row( |
|||
children: [ |
|||
Expanded( |
|||
child: _buildLanguageDropdown( |
|||
selected: controller.leftLanguage.value, |
|||
items: controller.supportedLanguages, |
|||
onChanged: (value) { |
|||
if (value != null) controller.leftLanguage.value = value; |
|||
}, |
|||
icon: Icons.headphones, |
|||
isSelected: isLeftInput, |
|||
label: '左方语言', |
|||
onTapInputSide: () => controller.toggleInputSide(true), |
|||
), |
|||
), |
|||
const SizedBox(width: 12), |
|||
GestureDetector( |
|||
onTap: controller.swapLanguages, |
|||
child: Container( |
|||
padding: const EdgeInsets.all(8), |
|||
decoration: BoxDecoration( |
|||
color: Colors.white10, |
|||
shape: BoxShape.circle, |
|||
), |
|||
child: |
|||
const Icon(Icons.compare_arrows, color: Colors.white), |
|||
), |
|||
), |
|||
const SizedBox(width: 12), |
|||
Expanded( |
|||
child: _buildLanguageDropdown( |
|||
selected: controller.rightLanguage.value, |
|||
items: controller.supportedLanguages, |
|||
onChanged: (value) { |
|||
if (value != null) controller.rightLanguage.value = value; |
|||
}, |
|||
icon: Icons.smartphone, |
|||
isSelected: !isLeftInput, |
|||
label: '右方语言', |
|||
onTapInputSide: () => controller.toggleInputSide(false), |
|||
), |
|||
), |
|||
], |
|||
), |
|||
], |
|||
), |
|||
); |
|||
}); |
|||
} |
|||
|
|||
Widget _buildLanguageDropdown({ |
|||
required String selected, |
|||
required List<String> items, |
|||
required void Function(String?) onChanged, |
|||
required IconData icon, |
|||
required bool isSelected, |
|||
required String label, |
|||
required VoidCallback onTapInputSide, |
|||
}) { |
|||
return GestureDetector( |
|||
onTap: onTapInputSide, |
|||
child: Container( |
|||
padding: const EdgeInsets.symmetric(vertical: 8, horizontal: 12), |
|||
decoration: BoxDecoration( |
|||
border: Border.all( |
|||
color: isSelected ? Colors.white : Colors.grey, |
|||
), |
|||
borderRadius: BorderRadius.circular(24), |
|||
color: isSelected ? Colors.white10 : Colors.transparent, |
|||
), |
|||
child: Row( |
|||
children: [ |
|||
Icon(icon, color: Colors.white), |
|||
const SizedBox(width: 8), |
|||
Expanded( |
|||
child: DropdownButton<String>( |
|||
dropdownColor: Colors.grey[900], |
|||
value: selected, |
|||
isExpanded: true, |
|||
underline: const SizedBox(), |
|||
iconEnabledColor: Colors.white, |
|||
style: const TextStyle(color: Colors.white), |
|||
items: items.map((lang) { |
|||
return DropdownMenuItem( |
|||
value: lang, |
|||
child: Text(lang), |
|||
); |
|||
}).toList(), |
|||
onChanged: onChanged, |
|||
), |
|||
), |
|||
], |
|||
), |
|||
), |
|||
); |
|||
} |
|||
} |
|||
Loading…
Reference in new issue