Feeding live video from a capture card through FFMPEG

RetroArch Windows devices
#1
  • RetroArch 1.8.8 (May 27 2020 / 02bc3ee1e8)
  • Windows 10 64-bit

I really dig RetroArch’s shaders functionnality, and I want to be able to apply them on live video feed. I successfully made a AVS script combined with a GRF file that display the live video from my Elgato GameCapture HD60 Pro with a vanilla installation of FFMPEG outisde RetroArch.

Here’s the script, simple.avs :

DirectShowSource("simple.grf", fps=60, audio=false, framecount=1000000)

I launch RetroArch, choose Show Desktop Menu, open my simple.avs file with the FFMPEG core, and get a “Failed to load content” error.

Here are the interesting error lines from the log console :

[libretro WARN] av_log: Format avs2 detected only with low score of 1, misdetection possible!
[libretro WARN] av_log: Could not find codec parameters for stream 0 (Video: avs2, none): unspecified size
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[libretro INFO] av_log: Input #0, avs2, from 'D:\Retro Gaming\Games\simple.avs':
[libretro INFO] av_log:   Duration: [libretro INFO] av_log: N/A[libretro INFO] av_log: , bitrate: [libretro INFO] av_log: N/A[libretro INFO] av_log:
[libretro INFO] av_log:     Stream #0:0[libretro INFO] av_log: : Video: avs2, none[libretro INFO] av_log: , [libretro INFO] av_log: 1200k tbr, [libretro INFO] av_log: 1200k tbn, [libretro INFO] av_log: 1200k tbc[libretro INFO] av_log:
[libretro ERROR] Couldn't find suitable decoder, exiting ...
[libretro ERROR] [FFmpeg]: Failed to find codec.
[libretro INFO] av_log: Statistics: 112 bytes read, 0 seeks
[ERROR] Failed to load content

The fact that it detects my script as a AVS2 file makes me hopeful that it should work, but this implementation can’t find the right codec.

Any ideas?

#2

We used to have a “video processor” core built-in that would do this, but I’m thinking it might have only been for V4L…? EDIT: ah, looks like it’s still in there, maybe just not compiled in by default or something: https://github.com/libretro/RetroArch/tree/7fedabee4bc58e3ee03eceab073b7dfa73ee6ecf/cores/libretro-video-processor

#3

If you can feed it as an udp stream it would work.

Say retroarch udp://host:port

1 Like
#4

Wow, that’s exactly what I’m looking for. I will try and compile RetroArch later today. Do I need to edit a config file to make sure that the video processor core gets build as well?

If this fails, I’ll look into the udp://host:port solution. Thank you both.

#5

I get an error when trying to compile libretro-video-processor under MINGW64. -_-

gcc -O3 -Wall -pedantic  -std=gnu99 -I../../libretro-common/include  -c -o video_processor_v4l2.o 
video_processor_v4l2.c
video_processor_v4l2.c:36:10: fatal error: sys/mman.h: No such file or directory
   36 | #include <sys/mman.h>
      |          ^~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:141: video_processor_v4l2.o] Error 1
#6

That’s a common problem. I think it needs to be compiled with msys2.

#7

I tried on MSYS2 MSYS, but cc is not a know command. FYI, my first attempt was under MinGW 64-bit under MSYS2.

EDIT : I thought I only had to use the make command in the /libretro-video-processor directory, but I think I’m supposed to use ./libretro-build.sh command. The script is trying to access …/…/libretro-common/include directory, but to no avail (it doesn’t make sense to search that far beneath /libretro-super I think).

$ ./libretro-build.sh video_processor
PLATFORM: win
ARCHITECTURE: x86_64
TARGET: win
CC = gcc
CXX = g++
CXX11 = g++
CXX17 = g++
STRIP =
Compiler: CC="gcc" CXX="g++"
=== x86 CPU detected... ===
=== x86_64 CPU detected... ===
win
win
=== Video processor
Building video_processor...
cd "/home/Laurent/libretro-super/libretro-video_processor"
make -f Makefile platform="win" -j8  clean
rm -f video_processor_v4l2.o video_processor_libretro.dll
make -f Makefile platform="win" -j8 CC="gcc" CXX="g++"
gcc -O3 -Wall -pedantic  -std=gnu99 -I../../libretro-common/include  -c -o video_processor_v4l2.o 
video_processor_v4l2.c
make: gcc: No such file or directory
make: *** [Makefile:141: video_processor_v4l2.o] Error 127
cp "video_processor_libretro.dll" "/home/Laurent/libretro- 
super/dist/win_x86/video_processor_libretro.dll"
cp: cannot stat 'video_processor_libretro.dll': No such file or directory
1 core(s) failed:
video_processor

So close, yet so far… :confused:

#8

it’s a built-in core. You should enable it when you configure the source tree: --enable-videoprocessor Enable Enable video processor core

#9

I think I’m making progress. I tried ./configure --enable-videoprocessor while configuring the source tree, but I get this error, and the process fails to generate the config.mk file :

[...]
Checking existence of -lOSDependent ... no
Checking existence of -lOGLCompiler ... no
Checking existence of -lHLSL ... no
Checking existence of -lSPIRV ... no
Checking existence of -lSPIRV-Tools-opt ... no
Checking existence of -lSPIRV-Tools ... no
Error: Video4linux2 is disabled and forced to build with video processor support.

Did you mispaste when you suggested --enable-videoprocessor Enable Enable video processor core?

#10

No, looks like it indeed requires Video4linux.

#11

I’ve been digging around, but all I could find was this repository that does not include the MSYS versions : https://repology.org/project/video4linux/versions

#12

Giving up on video-processor idea, since it required video4linux and I am on Windows. I’m trying to feed a UDP stream using fr500’s idea (through VLC). It seems to work, but RetroArch UI doesn’t launch.

D:\Retro Gaming\Emulators\RetroArch 1.8.8> retroarch udp://localhost:1234

D:\Retro Gaming\Emulators\RetroArch 1.8.8> [INFO] RetroArch 1.8.8 (Git 02bc3ee1e8)
[INFO] Redirecting save file to "D:\Retro Gaming\Emulators\RetroArch 1.8.8\saves\localhost:1234.srm".
[INFO] Redirecting save state to "D:\Retro Gaming\Emulators\RetroArch 1.8.8\states\localhost:1234.state".
[INFO] === Build =======================================
[INFO] CPU Model Name: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
[INFO] Capabilities:  MMX MMXEXT SSE SSE2 SSE3 SSSE3 SSE4 SSE4.2 AES AVX
[INFO] Built: May 27 2020
[INFO] Version: 1.8.8
[INFO] Git: 02bc3ee1e8
[INFO] =================================================
[INFO] [Overrides] no core-specific overrides found at D:\Retro Gaming\Emulators\RetroArch 1.8.8\config\FFmpeg\FFmpeg.cfg.
[INFO] [Overrides] no content-dir-specific overrides found at D:\Retro Gaming\Emulators\RetroArch 1.8.8\config\FFmpeg\.cfg.
[INFO] [Overrides] no game-specific overrides found at D:\Retro Gaming\Emulators\RetroArch 1.8.8\config\FFmpeg\localhost:1234.cfg.
[INFO] [Environ]: SET_VARIABLES.
[INFO] [Environ]: GET_LOG_INTERFACE.
[INFO] [Remaps]: remap directory: D:\Retro Gaming\Emulators\RetroArch 1.8.8\config\remaps
[INFO] Redirecting save file to "D:\Retro Gaming\Emulators\RetroArch 1.8.8\saves\localhost:1234.srm".
[INFO] Redirecting save state to "D:\Retro Gaming\Emulators\RetroArch 1.8.8\states\localhost:1234.state".
[INFO] Content loading skipped. Implementation will load it on its own.
[INFO] [Environ]: SET_INPUT_DESCRIPTORS:
[INFO] [Environ]: SET_PIXEL_FORMAT: XRGB8888.
#13

Try calling the external ffmpeg core using the -L switch.

#14

You mean like this? retroarch -L cores\ffmpeg_libretro.dll udp://localhost:1234

If so, I get the exact same result… -_-

#15

Dang, yeah, that was my best suggestion. I’m out of ideas :frowning:

#16

Thanks so much for your assistance nonetheless. I better understand the issue at hand. I’ve worked a workaround by messing with shadertoy.com and it’s built in webcam input (that threats my GameCapture H60 Pro card as a webcam). ¯_(ツ)_/¯

#17

Ah, nice. Most of the shaders should port to shadertoy without a ton of trouble (and more than a few are ported from shadertoys to begin with!).

#18

Back at it again. This time I am trying out @anon55749740’s route by feeding a UDP stream to RetroArch, and although I can get to see the feed on another VLC’s instance (meaning it works), I can’t get RetroArch to display it properly; it freezes on a white screen.

Any idea on what I am doing wrong? Can I expect RetroArch to be able to apply custom shaders on that video, or am I losing my time?

D:\Retro Gaming\Emulators\RetroArch 1.9.6> retroarch -v udp://127.0.0.1:1234?buffer_size=100000000

D:\Retro Gaming\Emulators\RetroArch 1.9.6> [INFO] RetroArch 1.9.6 (Git 4a6dc36)
[INFO] === Build =======================================
[INFO] CPU Model Name: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
[INFO] Capabilities:  MMX MMXEXT SSE SSE2 SSE3 SSSE3 SSE4 SSE4.2 AES AVX AVX2
[INFO] Built: Jul  4 2021
[INFO] Version: 1.9.6
[INFO] Git: 4a6dc36
[INFO] =================================================
[INFO] [Input]: Found input driver: "dinput".
[INFO] [Overrides]: No core-specific overrides found at "D:\Retro Gaming\Emulators\RetroArch 1.9.6\config\FFmpeg\FFmpeg.cfg".
[INFO] [Overrides]: No content-dir-specific overrides found at "D:\Retro Gaming\Emulators\RetroArch 1.9.6\config\FFmpeg¹K.cfg".
[INFO] [Overrides]: No game-specific overrides found at "D:\Retro Gaming\Emulators\RetroArch 1.9.6\config\FFmpeg\127.0.0.cfg".
[INFO] [Environ]: SET_VARIABLES.
[INFO] [Environ]: GET_LOG_INTERFACE.
[INFO] [Remaps]: Remap directory: "D:\Retro Gaming\Emulators\RetroArch 1.9.6\config\remaps".
[INFO] [Overrides]: Redirecting save file to "D:\Retro Gaming\Emulators\RetroArch 1.9.6\saves\127.0.0.srm".
[INFO] [Overrides]: Redirecting save state to "D:\Retro Gaming\Emulators\RetroArch 1.9.6\states\127.0.0.state".
[INFO] [CONTENT LOAD]: Content loading skipped. Implementation will load it on its own.
[INFO] [Environ]: SET_INPUT_DESCRIPTORS:
[INFO] [Environ]: SET_PIXEL_FORMAT: XRGB8888.
[libretro INFO] [FFMPEG] libavformat version:   58.29.100
[libretro INFO] [FFMPEG] libavcodec version:    58.54.100
[libretro INFO] [FFMPEG] libavutil version:     56.31.100
[libretro INFO] [FFMPEG] libswresample version: 3.5.100
[libretro INFO] [FFMPEG] libswscale version:    5.5.100
[h264 @ 0000000005df2680] non-existing PPS 0 referenced
[h264 @ 0000000005df2680] non-existing PPS 0 referenced
[h264 @ 0000000005df2680] decode_slice_header error
[h264 @ 0000000005df2680] no frame!
[h264 @ 0000000005df2680] non-existing PPS 0 referenced
[h264 @ 0000000005df2680] non-existing PPS 0 referenced
[h264 @ 0000000005df2680] decode_slice_header error
[h264 @ 0000000005df2680] no frame!
[h264 @ 0000000005df2680] non-existing PPS 0 referenced
[h264 @ 0000000005df2680] non-existing PPS 0 referenced
[h264 @ 0000000005df2680] decode_slice_header error
[h264 @ 0000000005df2680] no frame!
[h264 @ 0000000005df2680] non-existing PPS 0 referenced
[h264 @ 0000000005df2680] non-existing PPS 0 referenced
[h264 @ 0000000005df2680] decode_slice_header error
[h264 @ 0000000005df2680] no frame!
[h264 @ 0000000005df2680] non-existing PPS 0 referenced
[h264 @ 0000000005df2680] non-existing PPS 0 referenced
[h264 @ 0000000005df2680] decode_slice_header error
[h264 @ 0000000005df2680] no frame!
[h264 @ 0000000005df2680] non-existing PPS 0 referenced
[h264 @ 0000000005df2680] non-existing PPS 0 referenced
[h264 @ 0000000005df2680] decode_slice_header error
[h264 @ 0000000005df2680] no frame!
[h264 @ 0000000005df2680] non-existing PPS 0 referenced
[h264 @ 0000000005df2680] non-existing PPS 0 referenced
[h264 @ 0000000005df2680] decode_slice_header error
[h264 @ 0000000005df2680] no frame!
[h264 @ 0000000005df2680] non-existing PPS 0 referenced
[h264 @ 0000000005df2680] non-existing PPS 0 referenced
[h264 @ 0000000005df2680] decode_slice_header error
[h264 @ 0000000005df2680] no frame!
[libretro INFO] [FFMPEG] Media information:
Input #0, mpegts, from 'udp://127.0.0.1:1234?buffer_size=100000000':
  Duration: N/A, start: 608.539867, bitrate: N/A
  Program 1
    Metadata:
      service_name    :
      service_provider:
    Stream #0:0[0xc8]: Audio: mp2 ([3][0][0][0] / 0x0003), 44100 Hz, stereo, fltp, 128 kb/s
    Stream #0:1[0x64]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(progressive), 1920x1080 [SAR 3:4 DAR 4:3], 60 fps, 60 tbr, 90k tbn, 120 tbc
[libretro INFO] [FFMPEG] Using SW decoding.
[libretro INFO] [FFMPEG] Configured software decoding threads: 8
[libretro ERROR] [FFMPEG] Could not determine media duration
[INFO] [Environ]: SET_HW_RENDER, context type: gl.
[INFO] Requesting OpenGL context.
[INFO] Reached end of SET_HW_RENDER.
[INFO] [SRAM]: Skipping SRAM load..
[libretro INFO] [FFMPEG] Configured worker threads: 4
[INFO] Version of libretro API: 1
[INFO] Compiled against API: 1
[INFO] [Audio]: Set audio input rate to: 44100.00 Hz.
[INFO] [Video]: Video @ 5760x3240
[INFO] [Video]: Using HW render, OpenGL driver forced.
[INFO] [Video]: "vulkan" saved as cached driver.
[h264 @ 0000000005df3600] [INFO] [GLCore]: Found GL context: wgl
co located POCs unavailable
[INFO] [GLCore]: Detecting screen resolution 1920x1080.
[h264 @ 0000000005e03e00] mmco: unref short failure
[INFO] [WGL] extensions: WGL_ARB_buffer_region WGL_ARB_create_context WGL_ARB_create_context_no_error WGL_ARB_create_context_profile WGL_ARB_create_context_robustness WGL_ARB_context_flush_control WGL_ARB_extensions_string WGL_ARB_make_current_read WGL_ARB_multi[INFO] [WGL]: Adaptive VSync supported.
[INFO] [GLCore]: Initializing HW render (2048 x 2048).
[INFO] [GLCore]: Max texture size: 32768 px, renderbuffer size: 32768 px.
[INFO] [GLCore]: Vendor: NVIDIA Corporation, Renderer: GeForce RTX 3080/PCIe/SSE2.
[INFO] [GLCore]: Version: 4.6.0 NVIDIA 461.92.
[INFO] [GLCore]: Using resolution 3844x1041
[INFO] [XInput]: Found XInput v1.4.
[WARN] [DINPUT]: Strong rumble unavailable.
[WARN] [DINPUT]: Weak rumble unavailable.
[INFO] [Joypad]: Found joypad driver: "dinput".
[INFO] [GLCore]: Loading stock shader.
[INFO] [slang]: Building pass #0 (N/A)
[INFO] [GLCore]: Not using frame history.
[INFO] [GLCore]: Not using framebuffer feedback.
[INFO] [Font]: Using font rendering backend: freetype.
[INFO] [Video]: Found display server: win32
[INFO] [XAudio2]: Requesting 64 ms latency, using 64 ms latency.
[INFO] [Display]: Found display driver: "glcore".
[INFO] [Font]: Using font rendering backend: freetype.
[INFO] [Font]: Using font rendering backend: freetype.
[INFO] [Font]: Using font rendering backend: freetype.
[INFO] [Font]: Using font rendering backend: freetype.
[INFO] [Font]: Using font rendering backend: freetype.
[INFO] [Font]: Using font rendering backend: freetype.
[INFO] [Font]: Using font rendering backend: freetype.
[INFO] [Font]: Using font rendering backend: freetype.
[INFO] [Font]: Using font rendering backend: freetype.
[INFO] [SRAM]: SRAM will not be saved.
[INFO] [Playlist]: Loading history file: [D:\Retro Gaming\Emulators\RetroArch 1.9.6\content_history.lpl].
[INFO] [Playlist]: Loading history file: [D:\Retro Gaming\Emulators\RetroArch 1.9.6\content_music_history.lpl].
[INFO] [Playlist]: Loading history file: [D:\Retro Gaming\Emulators\RetroArch 1.9.6\content_video_history.lpl].
[INFO] [Playlist]: Loading history file: [D:\Retro Gaming\Emulators\RetroArch 1.9.6\content_image_history.lpl].
[INFO] [Playlist]: Loading favorites file: [D:\Retro Gaming\Emulators\RetroArch 1.9.6\content_favorites.lpl].
#19

If you can get a working video feed, you should be able to put shaders on top. I don’t think anyone has gotten a working video feed, though. It may not be difficult to get working if you don’t mind getting your hand dirty with some debugging/coding, but it’s probably not going to work out of the box.

1 Like
#20

I managed to get the live stream from my Hdhomrun prime to display in the ffmpeg core, using VLC. Let me do it again, to refresh my memory and see if my method can work here.

2 Likes