Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cross-device DMA-BUF sharing for VAAPI with KMS grab #2053

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

cgutman
Copy link
Contributor

@cgutman cgutman commented Jan 21, 2024

Description

This introduces support for cross-device VAAPI encoding without having to copy each frame from the capture adapter through RAM to the encoding adapter. If the encoding device can successfully import the DMA-BUF directly from the capture device, we will pass FBs between the adapters.

In the event that the capture device doesn't support VAAPI encoding (which is the case for VKMS), we will try to find another device that can successfully import the FBs from the capture device for encoding. This allows VKMS streaming to work as long as the render device (if any) support VAAPI. Software encoding with VKMS with imported DMA-BUFs from a primary render GPU is still unsupported.

Screenshot

Issues Fixed or Closed

Fixes #2044 for GPU encoding only

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Dependency update (updates to dependencies)
  • Documentation update (changes to documentation)
  • Repository update (changes to repository files, e.g. .github/...)

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have added or updated the in code docstring/documentation-blocks for new or existing methods/components

Branch Updates

LizardByte requires that branches be up-to-date before merging. This means that after any PR is merged, this branch
must be updated before it can be merged. You must also
Allow edits from maintainers.

  • I want maintainers to keep my branch updated

@cgutman cgutman marked this pull request as draft January 21, 2024 03:41
@SzczurekYT
Copy link

Just tested this out and this indeed fixes #2044 for me.

@ReenigneArcher ReenigneArcher added this to the adjust lint rules milestone Feb 28, 2024
@d-air1
Copy link

d-air1 commented Mar 5, 2024

I tested it and it appears to work for me as well with the exception of the connection seemingly terminating at random. Perhaps once this is rebased on the the latest release, then I can test it again to make sure that it wasn't some other part of the code base causing the problem.

@gschintgen
Copy link
Contributor

This is interesting. I checked out the PR, but unfortunately I could note compile it:

build$ cmake .. && make -j12 && cpack -G DEB
-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
(...)
--   Found libcurl, version 7.81.0
-- Checking for module 'miniupnpc'
--   Found miniupnpc, version 2.2.3
CMake Error at cmake/dependencies/common.cmake:23 (add_subdirectory):
  The source directory

    /home/ubuntu/Sunshine/third-party/nlohmann_json

  does not contain a CMakeLists.txt file.
Call Stack (most recent call first):
  CMakeLists.txt:46 (include)


-- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.74.0/BoostConfig.cmake (found version "1.74.0") found components: locale log filesystem program_options 
-- Looking for a CUDA compiler
(...)

Somehow the nlohman_json folder is empty, but the other directories under third-party are populated.
A recursive git pull fetches multiple submodules, but not the nlohman_json one.

@gschintgen
Copy link
Contributor

tl:dr: Couldn't import RGB Image: 00003009

I manually cloned the json library to give it another go. I'd just like to leave some feedback, in case this is unexpected behavior: With an AMD dGPU (RX6650XT) as card1/renderD129 and an Intel iGPU (card0/D128) cross-encoding does not work (display/render on AMD, encode on Intel). This is on a regular desktop PC, not a highly integrated dual-GPU-notebook. Here's the log in case it's helpful:

[2024:05:11:20:38:10]: Info: Sunshine version: 0.21.0.faf6bb0.dirty
[2024:05:11:20:38:10]: Info: Using manually selected adapter for encoding: /dev/dri/card0 -> /dev/dri/renderD128
[2024:05:11:20:38:10]: Debug: Opening render node: /dev/dri/renderD128
[2024:05:11:20:38:10]: Info: Using manually selected adapter for encoding: /dev/dri/card1 -> /dev/dri/renderD128
[2024:05:11:20:38:10]: Debug: Opening render node: /dev/dri/renderD128
[2024:05:11:20:38:10]: Debug: crtc(0, 0)
[2024:05:11:20:38:10]: Debug: crtc(2560, 1440)
[2024:05:11:20:38:10]: Debug: plane->possible_crtcs == 1
[2024:05:11:20:38:10]: Debug: x(0) y(0) crtc_x(0) crtc_y(0) crtc_id(85)
[2024:05:11:20:38:10]: Debug: Resolution: 2560x1440: Pitch: 10240: Offset: 0
[2024:05:11:20:38:10]: Debug: Format [XR24, AR24, RA24, XR30, XB30, AR30, AB30, XR48, XB48, AR48, AB48, XB24, AB24, RG16, NV12, P010, XR4H, AR4H, XB4H, AB4H]
[2024:05:11:20:38:10]: Error: Environment variable WAYLAND_DISPLAY has not been defined
[2024:05:11:20:38:10]: Debug: Monitor description
[2024:05:11:20:38:10]: Debug: Resolution: 0x0
[2024:05:11:20:38:10]: Debug: Offset: 0x0
[2024:05:11:20:38:10]: Debug: Monitor description
[2024:05:11:20:38:10]: Debug: Resolution: 0x0
[2024:05:11:20:38:10]: Debug: Offset: 0x0
[2024:05:11:20:38:10]: Debug: Monitor description
[2024:05:11:20:38:10]: Debug: Resolution: 2560x1440
[2024:05:11:20:38:10]: Debug: Offset: 0x0
[2024:05:11:20:38:10]: Debug: Desktop resolution: 2560x1440
[2024:05:11:20:38:10]: Info: Trying encoder [vaapi]
[2024:05:11:20:38:10]: Info: Screencasting with KMS
[2024:05:11:20:38:10]: Info: Using manually selected adapter for encoding: /dev/dri/card0 -> /dev/dri/renderD128
[2024:05:11:20:38:10]: Debug: Opening render node: /dev/dri/renderD128
[2024:05:11:20:38:10]: Info: Using manually selected adapter for encoding: /dev/dri/card1 -> /dev/dri/renderD128
[2024:05:11:20:38:10]: Debug: Opening render node: /dev/dri/renderD128
[2024:05:11:20:38:10]: Info: Found monitor for DRM screencasting
[2024:05:11:20:38:10]: Debug: crtc(0, 0)
[2024:05:11:20:38:10]: Debug: crtc(2560, 1440)
[2024:05:11:20:38:10]: Debug: plane->possible_crtcs == 1
[2024:05:11:20:38:10]: Debug: x(0) y(0) crtc_x(0) crtc_y(0) crtc_id(85)
[2024:05:11:20:38:10]: Debug: Resolution: 2560x1440: Pitch: 10240: Offset: 0
[2024:05:11:20:38:10]: Debug: Format [XR24, AR24, RA24, XR30, XB30, AR30, AB30, XR48, XB48, AR48, AB48, XB24, AB24, RG16, NV12, P010, XR4H, AR4H, XB4H, AB4H]
[2024:05:11:20:38:10]: Info: Found connector ID [103]
[2024:05:11:20:38:10]: Info: Found cursor plane [82]
...
[2024:05:11:20:38:10]: Debug: EGL: [Mesa Project]: version [1.5]
[2024:05:11:20:38:10]: Debug: API's supported: [OpenGL OpenGL_ES ]
[2024:05:11:20:38:10]: Debug: GL: vendor: Intel
[2024:05:11:20:38:10]: Debug: GL: renderer: Mesa Intel(R) UHD Graphics 770 (ADL-S GT1)
[2024:05:11:20:38:10]: Debug: GL: version: 4.6 (Compatibility Profile) Mesa 23.2.1-1ubuntu3.1~22.04.2
[2024:05:11:20:38:10]: Debug: GL: shader: 4.60
[2024:05:11:20:38:10]: Info: SDR color coding [Rec. 601]
[2024:05:11:20:38:10]: Info: Color depth: 8-bit
[2024:05:11:20:38:10]: Info: Color range: [JPEG]
[2024:05:11:20:38:10]: Info: vaapi vendor: Intel iHD driver for Intel(R) Gen Graphics - 22.3.1 ()
[2024:05:11:20:38:10]: Debug: [AVHWDeviceContext @ 0x559fc8e67040] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 22.3.1 ().
[2024:05:11:20:38:10]: Debug: [AVHWDeviceContext @ 0x559fc8e67040] Driver not found in known nonstandard list, using standard behaviour.
[2024:05:11:20:38:10]: Debug: [h264_vaapi @ 0x559fc8dbf980] Input surface format is nv12.
[2024:05:11:20:38:10]: Debug: [h264_vaapi @ 0x559fc8dbf980] Using VAAPI profile VAProfileH264High (7).
[2024:05:11:20:38:10]: Debug: [h264_vaapi @ 0x559fc8dbf980] Using VAAPI entrypoint VAEntrypointEncSliceLP (8).
[2024:05:11:20:38:10]: Debug: [h264_vaapi @ 0x559fc8dbf980] Using VAAPI render target format YUV420 (0x1).
...
[2024:05:11:20:38:10]: Info: Found H.264 encoder: h264_vaapi [vaapi]
...
[2024:05:11:20:38:10]: Info: Found HEVC encoder: hevc_vaapi [vaapi]
...
[2024:05:11:20:38:25]: Info: CLIENT CONNECTED
...
[2024:05:11:20:38:25]: Debug: [AVHWDeviceContext @ 0x7f3a581aebc0] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 22.3.1 ().
[2024:05:11:20:38:25]: Debug: [AVHWDeviceContext @ 0x7f3a581aebc0] Driver not found in known nonstandard list, using standard behaviour.
[2024:05:11:20:38:25]: Debug: [h264_vaapi @ 0x7f3a581057c0] Input surface format is nv12.
[2024:05:11:20:38:25]: Debug: [h264_vaapi @ 0x7f3a581057c0] Using VAAPI profile VAProfileH264High (7).
[2024:05:11:20:38:25]: Debug: [h264_vaapi @ 0x7f3a581057c0] Using VAAPI entrypoint VAEntrypointEncSliceLP (8).
[2024:05:11:20:38:25]: Debug: [h264_vaapi @ 0x7f3a581057c0] Using VAAPI render target format YUV420 (0x1).
[2024:05:11:20:38:25]: Debug: [h264_vaapi @ 0x7f3a581057c0] RC mode: CBR.
[2024:05:11:20:38:25]: Debug: [h264_vaapi @ 0x7f3a581057c0] RC target: 100% of 20000000 bps over 1000 ms.
[2024:05:11:20:38:25]: Debug: [h264_vaapi @ 0x7f3a581057c0] RC buffer: 20000000 bits, initial fullness 15000000 bits.
[2024:05:11:20:38:25]: Debug: [h264_vaapi @ 0x7f3a581057c0] RC framerate: 60/1 (60.00 fps).
[2024:05:11:20:38:25]: Debug: [h264_vaapi @ 0x7f3a581057c0] Driver does not report any additional prediction constraints.
[2024:05:11:20:38:25]: Debug: [h264_vaapi @ 0x7f3a581057c0] Using intra and P-frames (supported references: 3 / 0).
[2024:05:11:20:38:25]: Debug: [h264_vaapi @ 0x7f3a581057c0] All wanted packed headers available (wanted 0xd, found 0x1f).
[2024:05:11:20:38:25]: Debug: [h264_vaapi @ 0x7f3a581057c0] Using level 4.2.
[2024:05:11:20:38:25]: Error: Couldn't import RGB Image: 00003009
[2024:05:11:20:38:25]: Error: Could not convert image
...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

GL Error when trying to stream virtual monitor
5 participants