Browse Source
用户反馈「悬浮弹幕上没有显示翻译内容」，根因在 pub 包 pip 0.0.4：
PipView.m: const int pixel = 0xFF000000; // BGRA 全黑
填满 pixelBuffer → enqueue 进 AVSampleBufferDisplayLayer
小窗显示的就是这一帧纯黑占位。真正的字幕卡片它是靠
insertContentViewIfNeeded 塞进 activeKeyWindow 的，而 PiP 窗口由系统进程托管、
**不在 UIApplication.connectedScenes 的窗口列表里**，那个 keyWindow 永远是
App 自己的窗口。于是字幕落在 App 身上（上一版还因此盖住整个屏幕点不动），
小窗里只可能是黑的。PipView.m 全文件 0 处引用 contentView，可以佐证。
改成自己实现：SubtitlePipController 把 PlayerView 画进 CVPixelBuffer，
逐帧喂给自己的 AVSampleBufferDisplayLayer，再用 sampleBufferDisplayLayer
这一路 content source 起 AVPictureInPictureController。小窗显示的就是它。
Dart 侧不再依赖 pip 包（import 与相关字段一并删除）。
同时回答另外两个问题：
- 大小：PiP 小窗的实际尺寸由用户捏合手势决定，App 无权直接设定。能控的是
**宽高比**（= 画布宽高比），setPipWidthScale 改的是这个，系统会按新形状
重新布局。另外卡片有内容时自动变高、没内容时收成一条，也走同一条路。
- 透明度：**做不到透过小窗看见下面的 App**。iOS 的 PiP 窗口本身不透明，
系统没有开放任何接口。setPipOpacity 只作用于卡片自己的底色（在小窗那块底
之上显得深浅不同）。真要少挡内容只能把小窗做小或拖到别处。
这一条在 Dart 与原生两侧的注释里都写明了，免得以后有人再试一遍。
几处实现要点：
- 心跳定时器每秒补一帧。只在文案变化时推帧的话，长时间没有新帧会让
AVSampleBufferDisplayLayer 被判定停滞并清空，表现是「翻译停一会儿小窗就黑」。
- start() 之前先推一帧：没有任何已入队帧时 AVPictureInPictureController
会判定内容源为空，直接回调 failed。
- 渲染上下文要做一次 y 轴翻转，否则字幕上下颠倒（UIKit 原点左上、CG 左下）。
- renderView 必须挂在真实窗口层级里且尺寸非零，脱离层级的 layer 不被 PiP 采纳。
- 时间轴返回无限区间，让系统按直播流处理，小窗上不出现进度条和快进按钮。
⚠️ 构建注意：给本地 pod 加新文件后必须重跑 pod install，否则文件不进 Pods
工程，报 "Cannot find 'SubtitlePipController' in scope"。
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
main
Rodger-Wang
1 month ago
5 changed files with 486 additions and 92 deletions
@ -0,0 +1,294 @@ |
|||||
|
import AVFoundation |
||||
|
import AVKit |
||||
|
import UIKit |
||||
|
|
||||
|
/// 自己实现的字幕画中画。 |
||||
|
/// |
||||
|
/// ## 为什么不用 pub 上的 `pip` 包 |
||||
|
/// 那个包给 PiP 喂的是**一整块纯黑像素**（PipView.m 里 `pixel = 0xFF000000` |
||||
|
/// 填满 pixelBuffer），只是个占位帧；真正的字幕卡片它是靠 |
||||
|
/// `insertContentViewIfNeeded` 塞进「keyWindow」的。而 PiP 窗口由系统进程托管、 |
||||
|
/// **不在 `UIApplication.connectedScenes` 的窗口列表里**，那个 keyWindow 永远 |
||||
|
/// 是 App 自己的窗口。结果就是：字幕落在 App 自己身上（还盖住整个屏幕）， |
||||
|
/// PiP 小窗里只有一块黑的。用户看到的「弹幕上没有翻译内容」就是这么来的。 |
||||
|
/// |
||||
|
/// 正确做法只有一条：**自己把字幕画进 CVPixelBuffer，逐帧喂给 |
||||
|
/// AVSampleBufferDisplayLayer**。小窗显示的就是这个 layer 的内容。 |
||||
|
/// |
||||
|
/// ## ⚠️ 关于透明度 |
||||
|
/// iOS 的 PiP 窗口本身不透明，系统没有任何接口能让它变成半透明。 |
||||
|
/// [setStyle] 里的 opacity 只作用于卡片自己的底色（在小窗那块黑底之上）， |
||||
|
/// **做不到透过小窗看见下面的 App**。要少挡内容只能把小窗做小， |
||||
|
/// 或者用系统手势把它拖到别处 / 收到屏幕边缘。 |
||||
|
/// |
||||
|
/// ## ⚠️ 关于大小 |
||||
|
/// PiP 小窗的尺寸由用户捏合手势控制，App 无权直接设定；我们能决定的是 |
||||
|
/// **宽高比**（= 喂进去的 pixel buffer 的宽高比）。所以"放大缩小"在这里表现为 |
||||
|
/// 换一个画布比例，系统会按新比例重新布局小窗。 |
||||
|
@available(iOS 15.0, *) |
||||
|
final class SubtitlePipController: NSObject { |
||||
|
static let shared = SubtitlePipController() |
||||
|
|
||||
|
/// 承载 AVSampleBufferDisplayLayer 的宿主 view |
||||
|
private final class RenderView: UIView { |
||||
|
override class var layerClass: AnyClass { AVSampleBufferDisplayLayer.self } |
||||
|
var displayLayer: AVSampleBufferDisplayLayer { |
||||
|
// swiftlint:disable:next force_cast |
||||
|
layer as! AVSampleBufferDisplayLayer |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
private let renderView = RenderView() |
||||
|
private let card = PlayerView() |
||||
|
private var pip: AVPictureInPictureController? |
||||
|
private var timebase: CMTimebase? |
||||
|
private var heartbeat: Timer? |
||||
|
|
||||
|
/// 画布宽度。高度由卡片当前是收起还是展开决定，见 PlayerView.canvasSize。 |
||||
|
private var canvasWidth: CGFloat = 375 |
||||
|
private var lastCanvasSize: CGSize = .zero |
||||
|
|
||||
|
private(set) var isActive = false |
||||
|
|
||||
|
private override init() { |
||||
|
super.init() |
||||
|
card.setFillsBounds(true) |
||||
|
} |
||||
|
|
||||
|
// MARK: - 生命周期 |
||||
|
|
||||
|
@discardableResult |
||||
|
func start() -> Bool { |
||||
|
guard AVPictureInPictureController.isPictureInPictureSupported() else { |
||||
|
NSLog("[SubtitlePip] 本机不支持画中画") |
||||
|
return false |
||||
|
} |
||||
|
attachRenderViewIfNeeded() |
||||
|
setupTimebaseIfNeeded() |
||||
|
// 先推一帧再 start：AVPictureInPictureController 在没有任何已入队帧时 |
||||
|
// 可能拒绝启动（内容源被判定为"没有内容"），表现是回调里直接 failed。 |
||||
|
renderAndEnqueue(force: true) |
||||
|
|
||||
|
if pip == nil { |
||||
|
let source = AVPictureInPictureController.ContentSource( |
||||
|
sampleBufferDisplayLayer: renderView.displayLayer, |
||||
|
playbackDelegate: self |
||||
|
) |
||||
|
let controller = AVPictureInPictureController(contentSource: source) |
||||
|
controller.delegate = self |
||||
|
// 字幕不是可拖动进度的媒体，去掉播放/快进那套控件 |
||||
|
controller.requiresLinearPlayback = true |
||||
|
// 用户切走 App 时自动进小窗——这正是本功能的使用场景 |
||||
|
controller.canStartPictureInPictureAutomaticallyFromInline = true |
||||
|
pip = controller |
||||
|
} |
||||
|
pip?.startPictureInPicture() |
||||
|
|
||||
|
// 心跳：定期补一帧。 |
||||
|
// ⚠️ 不能只在文案变化时推帧 —— 长时间没有新帧时 |
||||
|
// AVSampleBufferDisplayLayer 可能被系统判定为停滞并清空， |
||||
|
// 表现是「翻译停一会儿，小窗就黑了」。 |
||||
|
heartbeat?.invalidate() |
||||
|
heartbeat = Timer.scheduledTimer(withTimeInterval: 1.0, repeats: true) { [weak self] _ in |
||||
|
self?.renderAndEnqueue(force: false) |
||||
|
} |
||||
|
return true |
||||
|
} |
||||
|
|
||||
|
func stop() { |
||||
|
heartbeat?.invalidate() |
||||
|
heartbeat = nil |
||||
|
pip?.stopPictureInPicture() |
||||
|
renderView.removeFromSuperview() |
||||
|
isActive = false |
||||
|
} |
||||
|
|
||||
|
// MARK: - 内容与样式 |
||||
|
|
||||
|
func update(lang: String?, cn: String?, en: String?) { |
||||
|
DispatchQueue.main.async { [weak self] in |
||||
|
guard let self = self else { return } |
||||
|
self.card.update(withLang: lang, cn: cn, en: en) |
||||
|
self.renderAndEnqueue(force: true) |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
/// [opacity] 卡片底色不透明度 0~1；[widthScale] 画布宽度相对基准 375 的倍数， |
||||
|
/// 通过改变宽高比让系统把小窗重新布成更宽/更窄的形状。 |
||||
|
func setStyle(opacity: CGFloat?, widthScale: CGFloat?) { |
||||
|
DispatchQueue.main.async { [weak self] in |
||||
|
guard let self = self else { return } |
||||
|
if let opacity = opacity { self.card.setBackgroundAlpha(opacity) } |
||||
|
if let scale = widthScale { |
||||
|
self.canvasWidth = max(180, min(750, 375 * scale)) |
||||
|
} |
||||
|
self.renderAndEnqueue(force: true) |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
// MARK: - 渲染 |
||||
|
|
||||
|
private func attachRenderViewIfNeeded() { |
||||
|
guard renderView.superview == nil else { return } |
||||
|
// ⚠️ 必须挂在真实的窗口层级里，且尺寸非零：脱离层级的 layer 不会被 |
||||
|
// PiP 采纳，表现是 start 之后立刻回调 failed。做成几乎全透明即可， |
||||
|
// 它在 App 内不需要被看见。 |
||||
|
guard let window = UIApplication.shared.connectedScenes |
||||
|
.compactMap({ $0 as? UIWindowScene }) |
||||
|
.flatMap({ $0.windows }) |
||||
|
.first(where: { $0.isKeyWindow }) ?? UIApplication.shared.windows.first else { |
||||
|
return |
||||
|
} |
||||
|
renderView.alpha = 0.01 |
||||
|
renderView.isUserInteractionEnabled = false |
||||
|
renderView.frame = CGRect(x: 0, y: 0, width: 1, height: 1) |
||||
|
window.addSubview(renderView) |
||||
|
} |
||||
|
|
||||
|
private func setupTimebaseIfNeeded() { |
||||
|
guard timebase == nil else { return } |
||||
|
var tb: CMTimebase? |
||||
|
CMTimebaseCreateWithSourceClock(allocator: kCFAllocatorDefault, |
||||
|
sourceClock: CMClockGetHostTimeClock(), |
||||
|
timebaseOut: &tb) |
||||
|
if let tb = tb { |
||||
|
CMTimebaseSetTime(tb, time: .zero) |
||||
|
CMTimebaseSetRate(tb, rate: 1.0) |
||||
|
renderView.displayLayer.controlTimebase = tb |
||||
|
timebase = tb |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
private func renderAndEnqueue(force: Bool) { |
||||
|
let size = card.canvasSize(width: canvasWidth) |
||||
|
if size != lastCanvasSize { |
||||
|
lastCanvasSize = size |
||||
|
card.frame = CGRect(origin: .zero, size: size) |
||||
|
} |
||||
|
card.setNeedsLayout() |
||||
|
card.layoutIfNeeded() |
||||
|
|
||||
|
guard let pixelBuffer = makePixelBuffer(size: size), |
||||
|
let sample = makeSampleBuffer(pixelBuffer) else { return } |
||||
|
let layer = renderView.displayLayer |
||||
|
if layer.status == .failed { layer.flush() } |
||||
|
layer.enqueue(sample) |
||||
|
} |
||||
|
|
||||
|
private func makePixelBuffer(size: CGSize) -> CVPixelBuffer? { |
||||
|
let w = max(2, Int(size.width)) |
||||
|
let h = max(2, Int(size.height)) |
||||
|
var pb: CVPixelBuffer? |
||||
|
let attrs: [String: Any] = [ |
||||
|
kCVPixelBufferIOSurfacePropertiesKey as String: [String: Any]() |
||||
|
] |
||||
|
guard CVPixelBufferCreate(kCFAllocatorDefault, w, h, |
||||
|
kCVPixelFormatType_32BGRA, |
||||
|
attrs as CFDictionary, &pb) == kCVReturnSuccess, |
||||
|
let pixelBuffer = pb else { return nil } |
||||
|
|
||||
|
CVPixelBufferLockBaseAddress(pixelBuffer, []) |
||||
|
defer { CVPixelBufferUnlockBaseAddress(pixelBuffer, []) } |
||||
|
guard let base = CVPixelBufferGetBaseAddress(pixelBuffer) else { return nil } |
||||
|
|
||||
|
guard let ctx = CGContext( |
||||
|
data: base, |
||||
|
width: w, height: h, |
||||
|
bitsPerComponent: 8, |
||||
|
bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer), |
||||
|
space: CGColorSpaceCreateDeviceRGB(), |
||||
|
bitmapInfo: CGImageAlphaInfo.premultipliedFirst.rawValue |
||||
|
| CGBitmapInfo.byteOrder32Little.rawValue |
||||
|
) else { return nil } |
||||
|
|
||||
|
ctx.clear(CGRect(x: 0, y: 0, width: w, height: h)) |
||||
|
// ⚠️ UIKit 原点在左上、CoreGraphics 在左下。不翻这一下，字幕是上下颠倒的。 |
||||
|
ctx.translateBy(x: 0, y: CGFloat(h)) |
||||
|
ctx.scaleBy(x: 1, y: -1) |
||||
|
UIGraphicsPushContext(ctx) |
||||
|
card.layer.render(in: ctx) |
||||
|
UIGraphicsPopContext() |
||||
|
return pixelBuffer |
||||
|
} |
||||
|
|
||||
|
private func makeSampleBuffer(_ pixelBuffer: CVPixelBuffer) -> CMSampleBuffer? { |
||||
|
var formatDesc: CMVideoFormatDescription? |
||||
|
guard CMVideoFormatDescriptionCreateForImageBuffer( |
||||
|
allocator: kCFAllocatorDefault, |
||||
|
imageBuffer: pixelBuffer, |
||||
|
formatDescriptionOut: &formatDesc) == noErr, |
||||
|
let desc = formatDesc else { return nil } |
||||
|
|
||||
|
let now = timebase.map { CMTimebaseGetTime($0) } ?? .zero |
||||
|
var timing = CMSampleTimingInfo( |
||||
|
duration: CMTimeMakeWithSeconds(1, preferredTimescale: 600), |
||||
|
presentationTimeStamp: now, |
||||
|
decodeTimeStamp: .invalid |
||||
|
) |
||||
|
var sample: CMSampleBuffer? |
||||
|
guard CMSampleBufferCreateReadyWithImageBuffer( |
||||
|
allocator: kCFAllocatorDefault, |
||||
|
imageBuffer: pixelBuffer, |
||||
|
formatDescription: desc, |
||||
|
sampleTiming: &timing, |
||||
|
sampleBufferOut: &sample) == noErr else { return nil } |
||||
|
return sample |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
// MARK: - AVPictureInPictureControllerDelegate |
||||
|
|
||||
|
@available(iOS 15.0, *) |
||||
|
extension SubtitlePipController: AVPictureInPictureControllerDelegate { |
||||
|
func pictureInPictureControllerDidStartPictureInPicture( |
||||
|
_ pictureInPictureController: AVPictureInPictureController) { |
||||
|
isActive = true |
||||
|
NSLog("[SubtitlePip] 已进入画中画") |
||||
|
renderAndEnqueue(force: true) |
||||
|
} |
||||
|
|
||||
|
func pictureInPictureController( |
||||
|
_ pictureInPictureController: AVPictureInPictureController, |
||||
|
failedToStartPictureInPictureWithError error: Error) { |
||||
|
isActive = false |
||||
|
NSLog("[SubtitlePip] 进入画中画失败: %@", error.localizedDescription) |
||||
|
} |
||||
|
|
||||
|
func pictureInPictureControllerDidStopPictureInPicture( |
||||
|
_ pictureInPictureController: AVPictureInPictureController) { |
||||
|
isActive = false |
||||
|
NSLog("[SubtitlePip] 已退出画中画") |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
// MARK: - AVPictureInPictureSampleBufferPlaybackDelegate |
||||
|
|
||||
|
@available(iOS 15.0, *) |
||||
|
extension SubtitlePipController: AVPictureInPictureSampleBufferPlaybackDelegate { |
||||
|
func pictureInPictureController(_ pictureInPictureController: AVPictureInPictureController, |
||||
|
setPlaying playing: Bool) {} |
||||
|
|
||||
|
/// 返回无限区间 = 告诉系统这是直播流，小窗上就不会出现进度条和快进按钮。 |
||||
|
func pictureInPictureControllerTimeRangeForPlayback( |
||||
|
_ pictureInPictureController: AVPictureInPictureController) -> CMTimeRange { |
||||
|
CMTimeRange(start: .negativeInfinity, duration: .positiveInfinity) |
||||
|
} |
||||
|
|
||||
|
func pictureInPictureControllerIsPlaybackPaused( |
||||
|
_ pictureInPictureController: AVPictureInPictureController) -> Bool { |
||||
|
false |
||||
|
} |
||||
|
|
||||
|
func pictureInPictureController(_ pictureInPictureController: AVPictureInPictureController, |
||||
|
didTransitionToRenderSize newRenderSize: CMVideoDimensions) { |
||||
|
// 用户捏合改了小窗大小。这里不改画布：画布比例是我们定的， |
||||
|
// 系统会自己把内容缩放到新尺寸。 |
||||
|
NSLog("[SubtitlePip] 小窗尺寸变为 %dx%d", newRenderSize.width, newRenderSize.height) |
||||
|
} |
||||
|
|
||||
|
func pictureInPictureController(_ pictureInPictureController: AVPictureInPictureController, |
||||
|
skipByInterval skipInterval: CMTime, |
||||
|
completion completionHandler: @escaping () -> Void) { |
||||
|
completionHandler() |
||||
|
} |
||||
|
} |
||||
Loading…
Reference in new issue