[10.10]Thru3D媒体服务器（原VR视频透视服务器）V1.5.0版（支持字幕、离线鱼眼转透视）

horror

请问能否支持v100？这算是平价算力卡了

roronet

大佬真牛，支持~~~

zelefans

下一阶段优化开始，目标：8K视频不跳帧也能稳40FPS

https://github.com/zerochocobo/V ... PLAN_8K_40FPS_ZH.md

pwzdby

zelefans 发表于 2026-5-15 08:52
下一阶段优化开始，目标：8K视频不跳帧也能稳40FPS

https://github.com/zerochocobo/VR-Video-Passthrough ...

最新版本离线变板砖了，所有视频都离线不了，上个版本没问题，应该又打包出错了，我是昨晚用的，报告我忘了发😓

qwer2025

[offline] MatAnyone2 requires an NVIDIA GPU with at least 16GB VRAM.
[offline] run: D:\Data\download\QK\VR_Video_Passthrough_Server\VR_Video_Passthrough_Server\pt_core.exe tool offline_passthrough D:\Data\download\QK\VR_Video_Passthrough_Server\������Ƶ\test_8k_2.mp4 --engine matanyone2_onnx --out D:\Data\download\QK\VR_Video_Passthrough_Server\������Ƶ\test_8k_2_matanyone2_S000000_ALL_passthrough.mp4 --audio copy --fps 30.0 --alpha-stride 1
[offline] SAM3 prepass subprocess=D:\Data\download\QK\VR_Video_Passthrough_Server\VR_Video_Passthrough_Server\pt_core.exe tool offline_passthrough D:\Data\download\QK\VR_Video_Passthrough_Server\������Ƶ\test_8k_2.mp4 --engine matanyone2_onnx --model D:\Data\download\QK\VR_Video_Passthrough_Server\VR_Video_Passthrough_Server\models\matanyone2_onnx_512_bs2 --matanyone2-size 512 --matanyone2-batch auto --sam3-model-dir D:\Data\download\QK\VR_Video_Passthrough_Server\VR_Video_Passthrough_Server\models\sam3_onnx --sam3-prompt person --sam3-score-threshold 0.5 --sam3-min-area-ratio 0.0005 --sam3-max-area-ratio 0.95 --sam3-top-k 0 --sam3-scan hybrid --sam3-scan-interval-sec 1.0 --sam3-active-min-area-ratio 0.001 --sam3-provider cuda --sam3-decoder-provider cuda --sam3-cuda-memory-limit-mb 8192 --sam3-decoder-cuda-memory-limit-mb 4096 --sam3-min-vram-gb 15.5 --sam3-release-interval 0 --no-sam3-low-memory --sam3-low-memory-batch 1 --sam3-log-vram --matanyone2-segment-frames 300 --matanyone2-min-segment-sec 3.0 --frames 786 --fps 30.0 --sam3-prepass-out D:\Data\download\QK\VR_Video_Passthrough_Server\VR_Video_Passthrough_Server\debug_output\_sam3_prepass\sam3_prepass_1778821033201_2795408721104.npz
[offline] SAM3 prepass prompt='person' samples=32 scan=hybrid max_segment_frames=300 low_memory=False encoder=[CUDAExecutionProvider(arena_cap=8192MiB), CPUExecutionProvider] decoder=[CUDAExecutionProvider(arena_cap=4096MiB), CPUExecutionProvider]
2026-05-15 12:57:36.8737561 [Wnnxruntime:, transformer_memcpy.cc:111 onnxruntime::MemcpyTransformer::ApplyImpl] 2 Memcpy nodes are added to the graph main_graph for CUDAExecutionProvider. It might have negative impact on performance (including unable to run CUDA graph). Set session_options.log_severity_level=1 to see the detail logs before this message.
2026-05-15 12:57:37.0178527 [Wnnxruntimeefault, scatter_nd.h:53 onnxruntime::cuda::ScatterNDWithAtomicReduction::ScatterNDWithAtomicReduction] ScatterND with reduction=='none' only guarantees to be correct if indices are not duplicated.
[offline] SAM3 prepass 1/32 frame=0 src_idx=0 ms=1404.8 active=True objects=1 L=[5] area=0.0501 R=[4] area=0.0482 gpu=11939/16303MiB
[offline] SAM3 prepass 2/32 frame=30 src_idx=60 ms=738.7 active=True objects=1 L=[3] area=0.0519 R=[3] area=0.0501 gpu=11939/16303MiB
[offline] SAM3 prepass 3/32 frame=60 src_idx=120 ms=737.3 active=True objects=1 L=[4] area=0.0490 R=[3] area=0.0473 gpu=11939/16303MiB
[offline] SAM3 prepass 4/32 frame=90 src_idx=180 ms=725.4 active=True objects=1 L=[3] area=0.0490 R=[4] area=0.0470 gpu=11939/16303MiB
[offline] SAM3 prepass 5/32 frame=116 src_idx=232 ms=735.1 active=True objects=1 L=[3] area=0.0488 R=[4] area=0.0469 gpu=11939/16303MiB
[offline] SAM3 prepass 6/32 frame=120 src_idx=240 ms=736.0 active=True objects=1 L=[3] area=0.0490 R=[3] area=0.0471 gpu=11939/16303MiB
[offline] SAM3 prepass 7/32 frame=150 src_idx=300 ms=732.9 active=True objects=1 L=[4] area=0.0483 R=[2] area=0.0467 gpu=12083/16303MiB
[offline] SAM3 prepass 8/32 frame=180 src_idx=360 ms=733.4 active=True objects=1 L=[3] area=0.0496 R=[2] area=0.0477 gpu=12083/16303MiB
[offline] SAM3 prepass 9/32 frame=210 src_idx=420 ms=749.4 active=True objects=1 L=[4] area=0.0481 R=[2] area=0.0463 gpu=12083/16303MiB
[offline] SAM3 prepass 10/32 frame=240 src_idx=480 ms=751.2 active=True objects=1 L=[5] area=0.0470 R=[3] area=0.0453 gpu=12083/16303MiB
[offline] SAM3 prepass 11/32 frame=241 src_idx=482 ms=740.0 active=True objects=1 L=[4] area=0.0471 R=[3] area=0.0454 gpu=12083/16303MiB
[offline] SAM3 prepass 12/32 frame=270 src_idx=539 ms=747.4 active=True objects=1 L=[4] area=0.0498 R=[4] area=0.0478 gpu=12083/16303MiB
[offline] SAM3 prepass 13/32 frame=300 src_idx=599 ms=755.7 active=True objects=1 L=[4] area=0.0522 R=[0] area=0.0500 gpu=12083/16303MiB
[offline] SAM3 prepass 14/32 frame=330 src_idx=659 ms=734.3 active=True objects=1 L=[3] area=0.0511 R=[3] area=0.0499 gpu=12083/16303MiB
[offline] SAM3 prepass 15/32 frame=360 src_idx=719 ms=739.5 active=True objects=1 L=[5] area=0.0489 R=[3] area=0.0470 gpu=12083/16303MiB
[offline] SAM3 prepass 16/32 frame=366 src_idx=731 ms=730.6 active=True objects=1 L=[4] area=0.0485 R=[3] area=0.0467 gpu=12083/16303MiB
[offline] SAM3 prepass 17/32 frame=390 src_idx=779 ms=732.7 active=True objects=1 L=[3] area=0.0501 R=[3] area=0.0481 gpu=12083/16303MiB
[offline] SAM3 prepass 18/32 frame=420 src_idx=839 ms=742.1 active=True objects=1 L=[4] area=0.0502 R=[2] area=0.0484 gpu=12083/16303MiB
[offline] SAM3 prepass 19/32 frame=450 src_idx=899 ms=740.5 active=True objects=1 L=[3] area=0.0474 R=[2] area=0.0457 gpu=12083/16303MiB
[offline] SAM3 prepass 20/32 frame=480 src_idx=959 ms=740.5 active=True objects=1 L=[5] area=0.0486 R=[4] area=0.0470 gpu=12083/16303MiB
[offline] SAM3 prepass 21/32 frame=491 src_idx=981 ms=733.2 active=True objects=1 L=[5] area=0.0493 R=[3] area=0.0476 gpu=12083/16303MiB
[offline] SAM3 prepass 22/32 frame=510 src_idx=1019 ms=729.1 active=True objects=1 L=[3] area=0.0500 R=[3] area=0.0480 gpu=12083/16303MiB
[offline] SAM3 prepass 23/32 frame=540 src_idx=1079 ms=728.0 active=True objects=1 L=[5] area=0.0479 R=[3] area=0.0460 gpu=12083/16303MiB
[offline] SAM3 prepass 24/32 frame=570 src_idx=1139 ms=742.8 active=True objects=1 L=[5] area=0.0484 R=[3] area=0.0464 gpu=12083/16303MiB
[offline] SAM3 prepass 25/32 frame=600 src_idx=1199 ms=735.9 active=True objects=1 L=[4] area=0.0488 R=[2] area=0.0470 gpu=12083/16303MiB
[offline] SAM3 prepass 26/32 frame=630 src_idx=1259 ms=739.8 active=True objects=1 L=[3] area=0.0486 R=[2] area=0.0470 gpu=12083/16303MiB
[offline] SAM3 prepass 27/32 frame=660 src_idx=1319 ms=737.0 active=True objects=1 L=[3] area=0.0462 R=[3] area=0.0445 gpu=12083/16303MiB
[offline] SAM3 prepass 28/32 frame=690 src_idx=1379 ms=733.6 active=True objects=1 L=[4] area=0.0504 R=[4] area=0.0490 gpu=12083/16303MiB
[offline] SAM3 prepass 29/32 frame=720 src_idx=1439 ms=725.0 active=True objects=1 L=[4] area=0.0512 R=[3] area=0.0495 gpu=12083/16303MiB
Traceback (most recent call last):
  File "main.py", line 313, in <module>
  File "main.py", line 126, in main
  File "main.py", line 61, in _run_legacy_tool
  File "tools\offline_passthrough.py", line 1598, in main
  File "tools\offline_passthrough.py", line 1284, in _precompute_sam3_segment_masks
  File "tools\offline_passthrough.py", line 555, in decode_encoded
RuntimeError: SAM3 returned no masks for text prompt
[PYI-3480:ERROR] Failed to execute script 'main' due to unhandled exception!
Traceback (most recent call last):
  File "main.py", line 313, in <module>
  File "main.py", line 126, in main
  File "main.py", line 61, in _run_legacy_tool
  File "tools\offline_passthrough.py", line 1598, in main
  File "tools\offline_passthrough.py", line 1211, in _precompute_sam3_segment_masks
  File "tools\offline_passthrough.py", line 1419, in _precompute_sam3_segment_masks_subprocess
  File "subprocess.py", line 571, in run
subprocess.CalledProcessError: Command '['D:\\Data\\download\\QK\\VR_Video_Passthrough_Server\\VR_Video_Passthrough_Server\\pt_core.exe', 'tool', 'offline_passthrough', 'D:\\Data\\download\\QK\\VR_Video_Passthrough_Server\\������Ƶ\\test_8k_2.mp4', '--engine', 'matanyone2_onnx', '--model', 'D:\\Data\\download\\QK\\VR_Video_Passthrough_Server\\VR_Video_Passthrough_Server\\models\\matanyone2_onnx_512_bs2', '--matanyone2-size', '512', '--matanyone2-batch', 'auto', '--sam3-model-dir', 'D:\\Data\\download\\QK\\VR_Video_Passthrough_Server\\VR_Video_Passthrough_Server\\models\\sam3_onnx', '--sam3-prompt', 'person', '--sam3-score-threshold', '0.5', '--sam3-min-area-ratio', '0.0005', '--sam3-max-area-ratio', '0.95', '--sam3-top-k', '0', '--sam3-scan', 'hybrid', '--sam3-scan-interval-sec', '1.0', '--sam3-active-min-area-ratio', '0.001', '--sam3-provider', 'cuda', '--sam3-decoder-provider', 'cuda', '--sam3-cuda-memory-limit-mb', '8192', '--sam3-decoder-cuda-memory-limit-mb', '4096', '--sam3-min-vram-gb', '15.5', '--sam3-release-interval', '0', '--no-sam3-low-memory', '--sam3-low-memory-batch', '1', '--sam3-log-vram', '--matanyone2-segment-frames', '300', '--matanyone2-min-segment-sec', '3.0', '--frames', '786', '--fps', '30.0', '--sam3-prepass-out', 'D:\\Data\\download\\QK\\VR_Video_Passthrough_Server\\VR_Video_Passthrough_Server\\debug_output\\_sam3_prepass\\sam3_prepass_1778821033201_2795408721104.npz']' returned non-zero exit status 1.
[PYI-34700:ERROR] Failed to execute script 'main' due to unhandled exception!
用测试视频跑离线没跑出来

zelefans

pwzdby 发表于 2026-5-15 11:20
最新版本离线变板砖了，所有视频都离线不了，上个版本没问题，应该又打包出错了，我是昨晚用的，报告我忘 ...

好诡异，我本机测了下模型路径变了。你看到的也是类似
  File "tools\offline_passthrough.py", line 360, in __init__
  File "pipeline\matting.py", line 889, in __init__
FileNotFoundError: RVM ONNX not found: G:\GIT\VR_Video_Passthrough_Server\_internal\models\rvm_mobilenetv3_fp32.onnx. Download rvm_mobilenetv3_fp32.onnx into models/.
[PYI-19408:ERROR] Failed to execute script 'main' due to unhandled exception!

问题吗？

过于诡异，因为V3和V4之间没有改过相关代码，V3记得打包后测试过离线，所以V4就没测了……

zelefans

qwer2025 发表于 2026-5-15 12:59
[offline] MatAnyone2 requires an NVIDIA GPU with at least 16GB VRAM.
[offline] run: D:\Data\download ...

这个问题是 SAM3模型在过程中没找到人，然后一帧空的遮罩传给 了MatAnyone2模型导致失败。测试时没跑完全部视频所以没发现。我会在下一个版本解决。

解决测试了又发现新BUG，没人的地方会变成绿幕，顺手改了……

zelefans

qwer2025 发表于 2026-5-15 12:59
[offline] MatAnyone2 requires an NVIDIA GPU with at least 16GB VRAM.
[offline] run: D:\Data\download ...

我在原百度网盘分享里放了一个小的增量升级包update_alpha4_to_alpha5.zip
全量包等我回家真机测试实时服务器模式没问题才上传

5x5x5x4n

离线模型8g显存能跑吗？

zelefans

5x5x5x4n 发表于 2026-5-15 18:05
离线模型8g显存能跑吗？

快速模型消耗2、3G