Browse Source
用户反馈「悬浮弹幕上没有显示翻译内容」，根因在 pub 包 pip 0.0.4：
PipView.m: const int pixel = 0xFF000000; // BGRA 全黑
填满 pixelBuffer → enqueue 进 AVSampleBufferDisplayLayer
小窗显示的就是这一帧纯黑占位。真正的字幕卡片它是靠
insertContentViewIfNeeded 塞进 activeKeyWindow 的，而 PiP 窗口由系统进程托管、
**不在 UIApplication.connectedScenes 的窗口列表里**，那个 keyWindow 永远是
App 自己的窗口。于是字幕落在 App 身上（上一版还因此盖住整个屏幕点不动），
小窗里只可能是黑的。PipView.m 全文件 0 处引用 contentView，可以佐证。
改成自己实现：SubtitlePipController 把 PlayerView 画进 CVPixelBuffer，
逐帧喂给自己的 AVSampleBufferDisplayLayer，再用 sampleBufferDisplayLayer
这一路 content source 起 AVPictureInPictureController。小窗显示的就是它。
Dart 侧不再依赖 pip 包（import 与相关字段一并删除）。
同时回答另外两个问题：
- 大小：PiP 小窗的实际尺寸由用户捏合手势决定，App 无权直接设定。能控的是
**宽高比**（= 画布宽高比），setPipWidthScale 改的是这个，系统会按新形状
重新布局。另外卡片有内容时自动变高、没内容时收成一条，也走同一条路。
- 透明度：**做不到透过小窗看见下面的 App**。iOS 的 PiP 窗口本身不透明，
系统没有开放任何接口。setPipOpacity 只作用于卡片自己的底色（在小窗那块底
之上显得深浅不同）。真要少挡内容只能把小窗做小或拖到别处。
这一条在 Dart 与原生两侧的注释里都写明了，免得以后有人再试一遍。
几处实现要点：
- 心跳定时器每秒补一帧。只在文案变化时推帧的话，长时间没有新帧会让
AVSampleBufferDisplayLayer 被判定停滞并清空，表现是「翻译停一会儿小窗就黑」。
- start() 之前先推一帧：没有任何已入队帧时 AVPictureInPictureController
会判定内容源为空，直接回调 failed。
- 渲染上下文要做一次 y 轴翻转，否则字幕上下颠倒（UIKit 原点左上、CG 左下）。
- renderView 必须挂在真实窗口层级里且尺寸非零，脱离层级的 layer 不被 PiP 采纳。
- 时间轴返回无限区间，让系统按直播流处理，小窗上不出现进度条和快进按钮。
⚠️ 构建注意：给本地 pod 加新文件后必须重跑 pod install，否则文件不进 Pods
工程，报 "Cannot find 'SubtitlePipController' in scope"。
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
main
Rodger-Wang
1 month ago
5 changed files with 486 additions and 92 deletions
@ -0,0 +1,294 @@ |
|||
import AVFoundation |
|||
import AVKit |
|||
import UIKit |
|||
|
|||
/// 自己实现的字幕画中画。 |
|||
/// |
|||
/// ## 为什么不用 pub 上的 `pip` 包 |
|||
/// 那个包给 PiP 喂的是**一整块纯黑像素**（PipView.m 里 `pixel = 0xFF000000` |
|||
/// 填满 pixelBuffer），只是个占位帧；真正的字幕卡片它是靠 |
|||
/// `insertContentViewIfNeeded` 塞进「keyWindow」的。而 PiP 窗口由系统进程托管、 |
|||
/// **不在 `UIApplication.connectedScenes` 的窗口列表里**，那个 keyWindow 永远 |
|||
/// 是 App 自己的窗口。结果就是：字幕落在 App 自己身上（还盖住整个屏幕）， |
|||
/// PiP 小窗里只有一块黑的。用户看到的「弹幕上没有翻译内容」就是这么来的。 |
|||
/// |
|||
/// 正确做法只有一条：**自己把字幕画进 CVPixelBuffer，逐帧喂给 |
|||
/// AVSampleBufferDisplayLayer**。小窗显示的就是这个 layer 的内容。 |
|||
/// |
|||
/// ## ⚠️ 关于透明度 |
|||
/// iOS 的 PiP 窗口本身不透明，系统没有任何接口能让它变成半透明。 |
|||
/// [setStyle] 里的 opacity 只作用于卡片自己的底色（在小窗那块黑底之上）， |
|||
/// **做不到透过小窗看见下面的 App**。要少挡内容只能把小窗做小， |
|||
/// 或者用系统手势把它拖到别处 / 收到屏幕边缘。 |
|||
/// |
|||
/// ## ⚠️ 关于大小 |
|||
/// PiP 小窗的尺寸由用户捏合手势控制，App 无权直接设定；我们能决定的是 |
|||
/// **宽高比**（= 喂进去的 pixel buffer 的宽高比）。所以"放大缩小"在这里表现为 |
|||
/// 换一个画布比例，系统会按新比例重新布局小窗。 |
|||
@available(iOS 15.0, *) |
|||
final class SubtitlePipController: NSObject { |
|||
static let shared = SubtitlePipController() |
|||
|
|||
/// 承载 AVSampleBufferDisplayLayer 的宿主 view |
|||
private final class RenderView: UIView { |
|||
override class var layerClass: AnyClass { AVSampleBufferDisplayLayer.self } |
|||
var displayLayer: AVSampleBufferDisplayLayer { |
|||
// swiftlint:disable:next force_cast |
|||
layer as! AVSampleBufferDisplayLayer |
|||
} |
|||
} |
|||
|
|||
private let renderView = RenderView() |
|||
private let card = PlayerView() |
|||
private var pip: AVPictureInPictureController? |
|||
private var timebase: CMTimebase? |
|||
private var heartbeat: Timer? |
|||
|
|||
/// 画布宽度。高度由卡片当前是收起还是展开决定，见 PlayerView.canvasSize。 |
|||
private var canvasWidth: CGFloat = 375 |
|||
private var lastCanvasSize: CGSize = .zero |
|||
|
|||
private(set) var isActive = false |
|||
|
|||
private override init() { |
|||
super.init() |
|||
card.setFillsBounds(true) |
|||
} |
|||
|
|||
// MARK: - 生命周期 |
|||
|
|||
@discardableResult |
|||
func start() -> Bool { |
|||
guard AVPictureInPictureController.isPictureInPictureSupported() else { |
|||
NSLog("[SubtitlePip] 本机不支持画中画") |
|||
return false |
|||
} |
|||
attachRenderViewIfNeeded() |
|||
setupTimebaseIfNeeded() |
|||
// 先推一帧再 start：AVPictureInPictureController 在没有任何已入队帧时 |
|||
// 可能拒绝启动（内容源被判定为"没有内容"），表现是回调里直接 failed。 |
|||
renderAndEnqueue(force: true) |
|||
|
|||
if pip == nil { |
|||
let source = AVPictureInPictureController.ContentSource( |
|||
sampleBufferDisplayLayer: renderView.displayLayer, |
|||
playbackDelegate: self |
|||
) |
|||
let controller = AVPictureInPictureController(contentSource: source) |
|||
controller.delegate = self |
|||
// 字幕不是可拖动进度的媒体，去掉播放/快进那套控件 |
|||
controller.requiresLinearPlayback = true |
|||
// 用户切走 App 时自动进小窗——这正是本功能的使用场景 |
|||
controller.canStartPictureInPictureAutomaticallyFromInline = true |
|||
pip = controller |
|||
} |
|||
pip?.startPictureInPicture() |
|||
|
|||
// 心跳：定期补一帧。 |
|||
// ⚠️ 不能只在文案变化时推帧 —— 长时间没有新帧时 |
|||
// AVSampleBufferDisplayLayer 可能被系统判定为停滞并清空， |
|||
// 表现是「翻译停一会儿，小窗就黑了」。 |
|||
heartbeat?.invalidate() |
|||
heartbeat = Timer.scheduledTimer(withTimeInterval: 1.0, repeats: true) { [weak self] _ in |
|||
self?.renderAndEnqueue(force: false) |
|||
} |
|||
return true |
|||
} |
|||
|
|||
func stop() { |
|||
heartbeat?.invalidate() |
|||
heartbeat = nil |
|||
pip?.stopPictureInPicture() |
|||
renderView.removeFromSuperview() |
|||
isActive = false |
|||
} |
|||
|
|||
// MARK: - 内容与样式 |
|||
|
|||
func update(lang: String?, cn: String?, en: String?) { |
|||
DispatchQueue.main.async { [weak self] in |
|||
guard let self = self else { return } |
|||
self.card.update(withLang: lang, cn: cn, en: en) |
|||
self.renderAndEnqueue(force: true) |
|||
} |
|||
} |
|||
|
|||
/// [opacity] 卡片底色不透明度 0~1；[widthScale] 画布宽度相对基准 375 的倍数， |
|||
/// 通过改变宽高比让系统把小窗重新布成更宽/更窄的形状。 |
|||
func setStyle(opacity: CGFloat?, widthScale: CGFloat?) { |
|||
DispatchQueue.main.async { [weak self] in |
|||
guard let self = self else { return } |
|||
if let opacity = opacity { self.card.setBackgroundAlpha(opacity) } |
|||
if let scale = widthScale { |
|||
self.canvasWidth = max(180, min(750, 375 * scale)) |
|||
} |
|||
self.renderAndEnqueue(force: true) |
|||
} |
|||
} |
|||
|
|||
// MARK: - 渲染 |
|||
|
|||
private func attachRenderViewIfNeeded() { |
|||
guard renderView.superview == nil else { return } |
|||
// ⚠️ 必须挂在真实的窗口层级里，且尺寸非零：脱离层级的 layer 不会被 |
|||
// PiP 采纳，表现是 start 之后立刻回调 failed。做成几乎全透明即可， |
|||
// 它在 App 内不需要被看见。 |
|||
guard let window = UIApplication.shared.connectedScenes |
|||
.compactMap({ $0 as? UIWindowScene }) |
|||
.flatMap({ $0.windows }) |
|||
.first(where: { $0.isKeyWindow }) ?? UIApplication.shared.windows.first else { |
|||
return |
|||
} |
|||
renderView.alpha = 0.01 |
|||
renderView.isUserInteractionEnabled = false |
|||
renderView.frame = CGRect(x: 0, y: 0, width: 1, height: 1) |
|||
window.addSubview(renderView) |
|||
} |
|||
|
|||
private func setupTimebaseIfNeeded() { |
|||
guard timebase == nil else { return } |
|||
var tb: CMTimebase? |
|||
CMTimebaseCreateWithSourceClock(allocator: kCFAllocatorDefault, |
|||
sourceClock: CMClockGetHostTimeClock(), |
|||
timebaseOut: &tb) |
|||
if let tb = tb { |
|||
CMTimebaseSetTime(tb, time: .zero) |
|||
CMTimebaseSetRate(tb, rate: 1.0) |
|||
renderView.displayLayer.controlTimebase = tb |
|||
timebase = tb |
|||
} |
|||
} |
|||
|
|||
private func renderAndEnqueue(force: Bool) { |
|||
let size = card.canvasSize(width: canvasWidth) |
|||
if size != lastCanvasSize { |
|||
lastCanvasSize = size |
|||
card.frame = CGRect(origin: .zero, size: size) |
|||
} |
|||
card.setNeedsLayout() |
|||
card.layoutIfNeeded() |
|||
|
|||
guard let pixelBuffer = makePixelBuffer(size: size), |
|||
let sample = makeSampleBuffer(pixelBuffer) else { return } |
|||
let layer = renderView.displayLayer |
|||
if layer.status == .failed { layer.flush() } |
|||
layer.enqueue(sample) |
|||
} |
|||
|
|||
private func makePixelBuffer(size: CGSize) -> CVPixelBuffer? { |
|||
let w = max(2, Int(size.width)) |
|||
let h = max(2, Int(size.height)) |
|||
var pb: CVPixelBuffer? |
|||
let attrs: [String: Any] = [ |
|||
kCVPixelBufferIOSurfacePropertiesKey as String: [String: Any]() |
|||
] |
|||
guard CVPixelBufferCreate(kCFAllocatorDefault, w, h, |
|||
kCVPixelFormatType_32BGRA, |
|||
attrs as CFDictionary, &pb) == kCVReturnSuccess, |
|||
let pixelBuffer = pb else { return nil } |
|||
|
|||
CVPixelBufferLockBaseAddress(pixelBuffer, []) |
|||
defer { CVPixelBufferUnlockBaseAddress(pixelBuffer, []) } |
|||
guard let base = CVPixelBufferGetBaseAddress(pixelBuffer) else { return nil } |
|||
|
|||
guard let ctx = CGContext( |
|||
data: base, |
|||
width: w, height: h, |
|||
bitsPerComponent: 8, |
|||
bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer), |
|||
space: CGColorSpaceCreateDeviceRGB(), |
|||
bitmapInfo: CGImageAlphaInfo.premultipliedFirst.rawValue |
|||
| CGBitmapInfo.byteOrder32Little.rawValue |
|||
) else { return nil } |
|||
|
|||
ctx.clear(CGRect(x: 0, y: 0, width: w, height: h)) |
|||
// ⚠️ UIKit 原点在左上、CoreGraphics 在左下。不翻这一下，字幕是上下颠倒的。 |
|||
ctx.translateBy(x: 0, y: CGFloat(h)) |
|||
ctx.scaleBy(x: 1, y: -1) |
|||
UIGraphicsPushContext(ctx) |
|||
card.layer.render(in: ctx) |
|||
UIGraphicsPopContext() |
|||
return pixelBuffer |
|||
} |
|||
|
|||
private func makeSampleBuffer(_ pixelBuffer: CVPixelBuffer) -> CMSampleBuffer? { |
|||
var formatDesc: CMVideoFormatDescription? |
|||
guard CMVideoFormatDescriptionCreateForImageBuffer( |
|||
allocator: kCFAllocatorDefault, |
|||
imageBuffer: pixelBuffer, |
|||
formatDescriptionOut: &formatDesc) == noErr, |
|||
let desc = formatDesc else { return nil } |
|||
|
|||
let now = timebase.map { CMTimebaseGetTime($0) } ?? .zero |
|||
var timing = CMSampleTimingInfo( |
|||
duration: CMTimeMakeWithSeconds(1, preferredTimescale: 600), |
|||
presentationTimeStamp: now, |
|||
decodeTimeStamp: .invalid |
|||
) |
|||
var sample: CMSampleBuffer? |
|||
guard CMSampleBufferCreateReadyWithImageBuffer( |
|||
allocator: kCFAllocatorDefault, |
|||
imageBuffer: pixelBuffer, |
|||
formatDescription: desc, |
|||
sampleTiming: &timing, |
|||
sampleBufferOut: &sample) == noErr else { return nil } |
|||
return sample |
|||
} |
|||
} |
|||
|
|||
// MARK: - AVPictureInPictureControllerDelegate |
|||
|
|||
@available(iOS 15.0, *) |
|||
extension SubtitlePipController: AVPictureInPictureControllerDelegate { |
|||
func pictureInPictureControllerDidStartPictureInPicture( |
|||
_ pictureInPictureController: AVPictureInPictureController) { |
|||
isActive = true |
|||
NSLog("[SubtitlePip] 已进入画中画") |
|||
renderAndEnqueue(force: true) |
|||
} |
|||
|
|||
func pictureInPictureController( |
|||
_ pictureInPictureController: AVPictureInPictureController, |
|||
failedToStartPictureInPictureWithError error: Error) { |
|||
isActive = false |
|||
NSLog("[SubtitlePip] 进入画中画失败: %@", error.localizedDescription) |
|||
} |
|||
|
|||
func pictureInPictureControllerDidStopPictureInPicture( |
|||
_ pictureInPictureController: AVPictureInPictureController) { |
|||
isActive = false |
|||
NSLog("[SubtitlePip] 已退出画中画") |
|||
} |
|||
} |
|||
|
|||
// MARK: - AVPictureInPictureSampleBufferPlaybackDelegate |
|||
|
|||
@available(iOS 15.0, *) |
|||
extension SubtitlePipController: AVPictureInPictureSampleBufferPlaybackDelegate { |
|||
func pictureInPictureController(_ pictureInPictureController: AVPictureInPictureController, |
|||
setPlaying playing: Bool) {} |
|||
|
|||
/// 返回无限区间 = 告诉系统这是直播流，小窗上就不会出现进度条和快进按钮。 |
|||
func pictureInPictureControllerTimeRangeForPlayback( |
|||
_ pictureInPictureController: AVPictureInPictureController) -> CMTimeRange { |
|||
CMTimeRange(start: .negativeInfinity, duration: .positiveInfinity) |
|||
} |
|||
|
|||
func pictureInPictureControllerIsPlaybackPaused( |
|||
_ pictureInPictureController: AVPictureInPictureController) -> Bool { |
|||
false |
|||
} |
|||
|
|||
func pictureInPictureController(_ pictureInPictureController: AVPictureInPictureController, |
|||
didTransitionToRenderSize newRenderSize: CMVideoDimensions) { |
|||
// 用户捏合改了小窗大小。这里不改画布：画布比例是我们定的， |
|||
// 系统会自己把内容缩放到新尺寸。 |
|||
NSLog("[SubtitlePip] 小窗尺寸变为 %dx%d", newRenderSize.width, newRenderSize.height) |
|||
} |
|||
|
|||
func pictureInPictureController(_ pictureInPictureController: AVPictureInPictureController, |
|||
skipByInterval skipInterval: CMTime, |
|||
completion completionHandler: @escaping () -> Void) { |
|||
completionHandler() |
|||
} |
|||
} |
|||
Loading…
Reference in new issue