Upgrade away from legacy OpenGL

[ChrisThrasher]

Subject of the issue

https://en.sfml-dev.org/forums/index.php?topic=20767.0

Continuation of this old conversation from the forums that was focusing on how to upgrade to non-legacy shaders while maintaining compatibility within SFML 2. Now that we're not burdened by backwards compatibility, I'm curious how much easier this transition would be.

I'm not an OpenGL expert so I'm asking on community members to help us flesh out a few things.

1. What do we have to gain with such an upgrade?

2. What does this change entails? Will it require API changes or can be it entirely an implementation detail?

3. What OpenGL version do we upgrade to?

4. What hardware are we dropping? I doubt this will be a problem but we ought to know roughly what vintage of hardware will no longer work.

[metaquarx]

I'd like to preface this by saying I'm in no way an expert at this, most of what I know is just through past experiments. Feel free to correct me on any of these.

1. What do we have to gain with such an upgrade?

Currently, through using legacy OpenGL, whenever any shape is drawn, all of its vertices/colours/transforms have to be resent to the GPU. The current pipeline is (generally):

sf::Shape::draw(target, states)
└─ sf::RenderTarget::draw(vertices, count, primitive, states)
   ├─ glVertexPointer     // prepares data layout for vertices
   ├─ glColorPointer      // prepares data layout for colours
   ├─ glTexCoordPointer   // prepares data layout for texcoords 
   └─ glDrawArrays        // uploads data & draws

This is then repeated for each and every sf::Drawable. The issue here is that, even if a shape does not move, does not change shape, colour, or anything of the sort, its data still has to be uploaded each and every frame. I'd have to do some profiling to put exact numbers to this, but that is where the a lot of drawtime is spent.

In a switch to modern OpenGL, the biggest difference would be that the vertex data is already on the GPU at the time of a draw, which would considerably speed up drawing functionality.

More importantly, I'm also hoping that switching away from "immediate mode" OpenGL graphics will provide a jumping-off point for future changes in regards to modular graphics backends. APIs such as Vulkan explicitly do not include facilities for immediate drawing, the changes discussed here would otherwise have to be bundled as part of them.

2.1. What does this change entail?

To achieve that, modern OpenGL provides 2 basic utilities:

• Vertex Buffer Objects (VBO) - An array in graphics memory. It is also used to store vertices, colours, texture coords, etc. This is already somewhat implemented in SFML, in the form of sf::VertexBuffer.
• Vertex Array Objects (VAO) - This object stores information about the layout of data in the VBO. This would be the equivalent of the glVertexPointer/Color/TexCoord given above. It is the piping that connects the vertex data.

Through these objects, the vertex data can be uploaded to the GPU once, and a draw call can make use of the information that is already there.

Unlike legacy-mode OpenGL, where shaders were optional, modern OpenGL also makes developer-provided shaders mandatory. I don't believe this would cause any issues, since we can just provide an automatic default shader.

2.2. Will it require API changes or can be it entirely an implementation detail?

This entirely depends on the way this is implemented.

One potential implementation, would be for every sf::Shape to have their own VBO. This is certainly possible. I'm not aware of any hard OpenGL limits on the number of VBOs in a single OpenGL context. This would get rid of the many data uploads. What it wouldn't get rid of is the many draw calls. However, I still believe this would be a benefit to the current model. One hiccup with this method, is that while VBOs and Textures can be shared across OpenGL contexts, VAOs cannot. That said, the vertex format we use is unlikely to have to change often if at all. It should be possible to have one VAO per window/context, shared between the objects drawn to that window.

The second, and harder, potential option, would be for all of the shapes to share one or a few VBOs. This would enable us to draw multiple shapes with the same draw call, as a sort of lower-level batching (#2166) on all drawables, which would be a considerable performance boost. There is a big caveat to this option: since all shapes would be stored together, we would need some sort of data-controller object, which would tie the reference data of the sf::Drawables, to their respective offsets inside a VBO, and therefore possibly to the window/context in which it was created.

3. What OpenGL version do we upgrade to?

This, I am not sure about. I think a good target is OpenGL 4.1, since from that point we can ensure feature parity with OpenGL ES 2.0, through the GL_ARB_ES2_compatibility extension. As far as I know, this is also supported by M1 Macs. However, OpenGL Core profile is available from OpenGL version 3.2.

4. What hardware are we dropping? I doubt this will be a problem but we ought to know roughly what vintage of hardware will no longer work.

This is difficult to quantify. The closest thing to a database that we have is https://opengl.gpuinfo.org/listreports.php which is a user-reported (through a tool) database of GPUs & their OpenGL features.

[binary1248]

SFML started out using client-side (CPU) primitive data and still uses this storage model for everything except sf::VertexBuffer. There were attempts to move everything over to using GPU storage but as you said, measurements demonstrated that it made no perceivable difference because the performance was dominated by the cost of the draw calls. Our recommendation was always to move over to using sf::VertexArray or sf::VertexBuffer as a first optimization step to reduce the number of draw calls and use sf::VertexBuffer if the cost of the data transfer from CPU to GPU every frame for static data was very high.

I think we shouldn't make the same mistake of locking us into a conceptual model of a graphics API that is known to not represent modern hardware as well as more modern APIs e.g. Vulkan. The only additions Khronos are still making to OpenGL are bandages to keep compatibility/parity with Vulkan/DX12. In terms of evolution, OpenGL is de facto dead.

Since this is going to be one of the biggest changes SFML makes since it was initially written, I'd prefer to put the effort into something that will outlast an API that is already dying at the current time.

If you strip away the magical state management and memory allocation that happens behind the scenes in OpenGL you are basically left over with Vulkan. This was also the idea with Vulkan, less magic, more application responsibility.

Not many might know this, but one of the reasons I wrote the Vulkan example was also to serve as a reference for my future self and others when we come around to writing a Vulkan based backend. If you browse through the code you will notice that the sweeping majority of it deals with resource management and not even the drawing itself. This resource management has no analog in the OpenGL world because it was all done automatically for us. As funny as it sounds the reason why Vulkan is so verbose is because OpenGL was so terse. Many of the optimization possibilities developers asked for over the years weren't able to be implemented as incremental additions to the existing OpenGL API, and before they make the same mistake that they did with OpenGL 3.x they decided to just start fresh and add all the knobs to the new API from the start.

If we were to start with Vulkan and work backwards towards OpenGL we would have the heavy machinery available that we needed for Vulkan but is provided automatically by OpenGL. It is easier design wise to have something that we don't always have to use than to keep having to make incremental changes to an existing internal API to accommodate the extra work that is necessary to get Vulkan working. We also have to be careful not to fall into the trap of making assumptions that are not able to be modelled in both APIs. Everything that can be done with OpenGL can also be done with Vulkan, the opposite is not true.

A lot of the current public SFML Window/Graphics API assumes some kind of global state (since it was implemented with OpenGL in mind). From the perspective of the user, graphics and window resources are just floating around unrelated to each other. Experienced users know that these resources live in thread-local things called contexts and to mask the thread-localness of the context we link all of them together using a shared context. If you think of it this just smells of one workaround piled on top of another, and the truth is... it is. I wouldn't say it is impossible to emulate this behaviour using something like Vulkan, but as with all emulation there will be a performance cost, and inventing magic always takes a lot of developer time/effort. We have to ask ourselves if retaining this hands-off approach to resource management in the public API is worth the cost. I am indifferent in this regard, but I can imagine that making resource management more explicit would make SFML less beginner friendly than it currently is and that is one of the things SFML is best known for.

To sum up, I think the questions that we are asking might be right, but we are asking them at the wrong time. We can't hide the philosophical changes that are made to the underlying graphics APIs forever. First we have to ask ourselves what kind of high level changes we are willing to make to the public API before delving into the implementation details.

[TheMaverickProgrammer]

Linking my proposal into this discussion as a flexible alternative #2426

In short, I argue there's no "perfect" API we could dream up. Graphics changed fast since SFML's conception and OpenGL isn't the only player on the block anymore. Arguably, OpenGL isn't catching up. So instead of stressing about careful modeling, we can identify, standardize, and pack common graphics commands into small data structures called "events". We can make aggregate events for ease-of-use and hide those behind the sfml primitives and functions we already use.

Forward facing to the user: same SFML with new bells and whistles
In the background: flexible, extendable, modular

[eXpl0it3r]

Two points that aren't technical, but also important to consider are scope and time.

I intentionally scoped SFML 3 to be the C++17 upgrade with clean up on deprecations and API, because major API evolutions can take a lot of time and we could end up with something quite a bit different than what we know SFML to be today.
We need to release a new version of SFML that has C++17 support (which btw. is still not fully implemented), more than we need modern OpenGL support, as such the latter shouldn't block the former.

On the other hand, we're bottlenecking the community a lot by not being compatible with OpenGL ES 2 and thus no shader support on mobile and more. The question for that then is, whether we can provide a new rendering API before the last person on Earth has lost interest in SFML for mobile or other platforms, or whether we should provide a simpler in-between approach like #1585, before re-imagining the API?

Points being:

• Full rendering API evolution shouldn't be part of SFML 3, as the current primary pain point is lack of C++11/14/17 support
• Currently, the not using of Vulkan is more of a "wouldn't it be cool"-issue, rather than an actual pain point, while lack of OpenGL ES 2 is really limiting the usage of SFML itself

[binary1248]

I agree with @eXpl0it3r.

The focus now should really be just getting the current implementation up-to-date with the current state of the art. What makes this so tricky is the fact that unlike desktop OpenGL, OpenGL ES went for the "either or" approach of supporting old/new features whereas desktop OpenGL provides a compatibility profile that combines the best of both worlds. This is/has also been the crux of all my attempts to get this work done in the past, which lead to the forum post and draft PR.

Whatever solution to the open questions the community decides to go for, it will probably be feasible to implement (and if not I will explain why not) since the "design phase" should boil down to setting a standard for shader usage that we can settle on.

[ChrisThrasher]

This is/has also been the crux of all my attempts to get this work done in the past, which lead to the forum post and draft PR.

Does that crux still exist now that you're not burdened by maintaining API compatibility or are forced to maintain all current hardware support? We can take major liberties during this major version bump that weren't available to you at the time you last attempted this.

[binary1248]

Let's put it in simpler terms: I don't like telling other people how to write their shaders. But that is exactly what legacy OpenGL did, and non-legacy OpenGL doesn't do anymore. So to make the library work in both scenarios we have to substitute the built-in legacy standards with our own, standards I don't have the time/energy to formulate.

One might think: OK then why don't we just drop support for legacy OpenGL all together then? This question has been asked many times over the last years. The answer is because the SFML API isn't ready for this change.

The current API assumes that certain decisions don't have to be made by the user, and this is true as long as you stay in legacy land. Once you leave it, even if you don't intend on using your own "custom" shader, someone will still have to provide one. Either SFML or the user has to provide one that emulates the default behaviour of the legacy pipeline that they did not have to explicitly provide in the past. Because SFML manages vertex data and rendering state, there has to be an agreement on the shader interface between SFML and the user when they write their shader programs.

[TheMaverickProgrammer]

Two points that aren't technical, but also important to consider are scope and time.

I intentionally scoped SFML 3 to be the C++17 upgrade with clean up on deprecations and API, because major API evolutions can take a lot of time and we could end up with something quite a bit different than what we know SFML to be today. We need to release a new version of SFML that has C++17 support (which btw. is still not fully implemented), more than we need modern OpenGL support, as such the latter shouldn't block the former.

On the other hand, we're bottlenecking the community a lot by not being compatible with OpenGL ES 2 and thus no shader support on mobile and more. The question for that then is, whether we can provide a new rendering API before the last person on Earth has lost interest in SFML for mobile or other platforms, or whether we should provide a simpler in-between approach like #1585, before re-imagining the API?

Points being:

* Full rendering API evolution shouldn't be part of SFML 3, as the current primary pain point is lack of C++11/14/17 support

* Currently, the not using of Vulkan is more of a "wouldn't it be cool"-issue, rather than an actual pain point, while lack of OpenGL ES 2 is really limiting the usage of SFML itself

This makes sense.

[ChrisThrasher]

The SFML team has decided that such an overhaul to rendering will not happen in SFML 3. This is still a worthwhile idea but do not expect anything of this sort to be released prior to version 4.0.0.

[pwouik]

webgpu could be an idea for the underlying api
its implementations, dawn and wgpu-natives are built on top of all modern apis
it share the main concepts while being easier to use

[empyreanx]

I wrote a single-header OpenGL renderer written in C. It was partly inspired by SFML. It supports GL3.3+ and GLES3.1+ using the core profile. I know that might not satisfy your requirements, but it might be possible, although painful, to add GLES2 support. Aside from the version requirements, it should be sufficient to implement all the features in the graphics module.

Here it is the source: https://github.com/empyreanx/pico_headers/blob/main/build_pico_gl/src/pico_gl.h

This gets built into a header that embeds GLAD.

[aardhex]

This article makes a convincing argument that WebGPU might be the right general purpose backend API.

https://cohost.org/mcc/post/1406157-i-want-to-talk-about-webgpu

Disclaimer: I have no stake in SFML development, but I’ve been observing it since a while.
Since you have explicitly posed this as an open question, I feel it’s worth sharing.