-
-
Notifications
You must be signed in to change notification settings - Fork 704
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
NvFBC retrieves slightly outdated images. #2472
Comments
Does that work properly in the case where the host display's refresh rate is higher than the stream frame rate? IIUC, in that case, we could potentially capture and encode more frames per second than the client is expecting because there will often be a frame ready when we expect to be sleeping until the next frame is due. |
We are asking NvFBC to sample at the requested framerate, so that isn't an issue: Sunshine/src/platform/linux/cuda.cpp Line 757 in 7fb8c76
|
I think that's a red herring. I had a look at this for #2333 and also thought that was the crucial line but it's rather this one: Sunshine/src/platform/linux/cuda.cpp Line 814 in 7fb8c76
I.e. we manually capture at the configured rate. The other line is probably just to put the capture feature in a reasonable state. |
Did you test that with the current Which raises an interesting issue. If NvFBC updates internally every 16msec and Sunshine always waits exactly 16.666msec between frames, then every 24 ( |
You’re right I did miss the last (but crucial) part of your initial post, i.e. the part where you point out the “manual” waiting done by Sunshine. Anyway, I only had superficial contact with this detail of the code base since I don’t have nvidia hardware and it came up in the mentioned PR discussion. I guess it all depends on what exactly NvFBC does with its 16.000 ms sampling rate: Anyway I just wanted to give a heads up, but missed that you correctly saw sunshine’s own waiting. |
It seems like your option a) is correct. I tested this in moonshine (similar to sunshine, but only has NvFBC + NVENC), where I rely on NvFBC to block until a new frame arrives. According to moonlight I get roughly 62.5Hz if I stream dynamic content (in this case I just moved the cursor a lot) : I guess sunshine waits to achieve the accurate 60Hz framerate, but by doing so, it skips a frame every 24 frames.
No worries! Appreciate it :). |
Out of curiosity (since I'm not familiar with the inner workings of hardware pointers and such): Did you try what happens to the framerate as reported by Moonlight if you stream, let's say, an animated title screen of a game, framecapped at 60.00Hz, but without any mouse input?
If taken literally the framerate should then drop to 60fps. Maybe it's "just" the mouse input (e.g. 1000Hz gaming mouse) that messes with the capture timing. Is it possible to hide mouse input from NvFBC? |
I wasn't entirely sure what would happen either, but it seems to still stream at 62.5Hz if I don't move the mouse in a game (even if the game I was streaming shows a framerate of a steady 60Hz according to Steam FPS overlay). |
Ok, so it seems indeed like NvFBC calls its blocking image capture at a precise 16.00 ms and most of the time it will find a frame that it has not yet shown, so it immediately returns that one. Until another 24 or 25 second period is over and the difference accumulates to a whole frame time interval. So the ideal capture routine for frame-capped (e.g. 60fps) content (which is not really under our control...) would be:
Except that it will break down if the game is running uncapped and step 2 will capture too early and too often. Isn't this then similar to the problem that the Windows side of Sunshine has to deal with when it is using the Desktop Duplication API (which I don't know anything about except the few tidbits I picked up here and there)? Sunshine/src/platform/windows/display_base.cpp Lines 170 to 200 in 9d5ee2f
(I did not try to analyze or understand this code in detail.) |
Why would the second frame come after 16.7msec? I would expect the internal functioning of NvFBC to always retrieve frames after 16msec intervals, so if you wait for 80%, say exactly 13msec, that the next blocking call would return after another exactly 3msec. |
Right, there's also the question what "blocking" actually means: If the game is running at 60fps / 16.667ms, then the blocking call executed 13ms after the previous capture would then wait until a new frame has been rendered and is ready to be captured. And those frames should be emitted at an interval of 16.666ms due to the framecap. |
I think it all depends on when the timer will trigger next:
It seems like NvFBC is doing the former, while the latter would be what's needed. Unfortunately I can't test any of this. (I'm on Intel & AMD) |
My suspicion is that NvFBC is running an internal loop, separate of the frame generation loop, which polls the latest frame after every capturer.start(BufferFormat::Rgb, 60)?;
// In case it needs to warm up or initialize something.
for _ in 0..10 {
capturer.next_frame(CaptureMethod::Blocking)?;
}
let now = std::time::Instant::now();
for _ in 0..100 {
std::thread::sleep(std::time::Duration::from_millis(13));
capturer.next_frame(CaptureMethod::Blocking)?;
}
println!("{}", now.elapsed().as_micros() / 100); Meaning we set the framerate to 60Hz (which is used to set the sampling rate to 16 msec), sleep 13msec and then wait for a next frame in a blocking manner. Repeat this 100 times and get the average time waited on new frames. I am getting this timing:
Without the 13msec sleep I also get |
Hm, but somehow it must be possible to make NvFBC actually wait for a new frame:
At least with With |
Ah sorry, I had renamed |
That's weird. (And I'm out of ideas.) |
That's okay, we tried ;) A mystery for another day 🪄 |
Is there an existing issue for this?
Is your issue described in the documentation?
Is your issue present in the nightly release?
Describe the Bug
NvFBC has a few different methods to capture an image. The one used in Sunshine is:
Basically it means that whenever Sunshine requests a new frame, a frame is provided, but that frame can be "old" (max of 1/fps seconds old). Still though, it means at 60fps a frame could be 16msec old.
Changing to
NVFBC_TOCUDA_GRAB_FLAGS_NOWAIT_IF_NEW_FRAME_READY
would mean that the frame request blocks until a new frame becomes available, but it returns the frame immediately if NvFBC knows it's a new frame. This also means that when the host is serving static content, the FPS drops to 13.33FPS in my tests (not sure why this amount exactly).And for context (since that flag basically extends the NOFLAGS flag) :
As far as I can see, we can simply use
NVFBC_TOCUDA_GRAB_FLAGS_NOWAIT_IF_NEW_FRAME_READY
instead ofNVFBC_TOSYS_GRAB_FLAGS_NOWAIT
. This wait becomes kinda redundant, but it won't hurt either.Expected Behavior
N/A
Additional Context
N/A
Host Operating System
Linux
Operating System Version
Arch Linux
Architecture
64 bit
Sunshine commit or version
7fb8c76
Package
other (self built)
GPU Type
Nvidia
GPU Model
GeForce RTX 3090
GPU Driver/Mesa Version
550.76
Capture Method (Linux Only)
NvFBC
Config
N/A
Apps
N/A
Relevant log output
N/A
The text was updated successfully, but these errors were encountered: