Add Rust to the codebase

[orowith2os]

I've asked @mirkobrombin about this to see if he would be okay with trying it out, and I got the okay. I'd like to use this to see if any other contributors have any concerns.

I'd like to look at trialing Rust in Bottles internals. There are a few reasons for this, and a few that make it hard, but the main motivator is the Rust ecosystem, as well as the language itself.

Go is also another potential option, and the contributors to Bottles may already know it. However, I'm not familiar with Go <-> Python interop, and can't comment on the possibility of using it.

I would start this Rust integration off with a simple Hello World, then a rewrite of one of the backend utilities without using any external crates - the sandbox utility, specifically. Then we can move on to using third party crates. At each step, we would review the work, and see if it's worth keeping it.

If it's decided that Rust integration wouldn't be worth it, we can remove it - simple as that.

I don't have plans to rewrite any of the frontend(s) in Rust, just the backend.

Benefits

• Bottles can utilize the Rust ecosystem (crates)
• mmmmm memory safety /j
• Potentially performance-limited actions can get the advantages of a compiled language
• Rust can more easily (with more useful details) describe what goes wrong with the source code, and runtime errors are less likely to happen
• Contributors may be able to come from more areas of expertise

Pain points

• Flatpak builds may be more annoying
  • Cargo is another build system that uses external package repositories; this would need to be figured out somehow to use it with Bottles. GNOME apps vendor the dependencies on their release archives and use meson to build the project, Bottles could follow in their footsteps.
• Contributors may need to learn Rust and Python in order to contribute
• PyO3 currently isn't the best when it comes to the Python <-> Rust bridge: Make writing modules simpler PyO3/pyo3#3268

CCing @TheEvilSkeleton @ItsJamie9494 @axtloss @jannuary @Kinsteen

[TheEvilSkeleton]

I think a Rust backend would be great as a learning experience, but we'll have to see what others think about it. Then again, we'll need to rewrite ~10K lines of code and probably later the frontend as well. Who's going to work on that?

[orowith2os]

We wouldn't be rewriting the entire thing, just bits and pieces where it might make sense. Plus, assuming we use PyO3, there wouldn't need to be any frontend rewriting - it would look just like another Python library to the frontend(s).

[ItsJamie9494]

I have no idea what the bottles codebase is but rust is good 👍 (in all seriousness, I’d be glad to help in my spare time)

[Kinsteen]

I really like the idea but I don't know Rust, nor the Python Rust interop. I would be happy to learn to rework some parts of the backend, I agree it would be better than Python in some parts!

[axtloss]

How would this work? Would the backend be a separate program that gets called by the frontend when doing things (similar to heroic)? Or would it be an actual library?

[orowith2os]

It would be an actual library, just like it would be if you were to write a Python library in C or Go. Just an import lib and you can interact with it.

[axtloss]

Doesn't that mean that we would have to also maintain python bindings for the library?

[orowith2os]

That's where PyO3 comes in - I linked to an issue which gives some context. It's a bit of a pain right now, but I can write a macro that makes it easier to work with, so all we need to do is implement the Rust bits and add it to the macro.

This is how it's done right now, this would be abstracted away with the macro I mentioned:

use pyo3::prelude::*;

#[pyfunction]
fn double(x: usize) -> usize {
    x * 2
}

#[pymodule]
fn my_extension(py: Python<'_>, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(double, m)?)?;
    Ok(())
}

[axtloss]

That seems like it could stack up to quite a bit of boilerplate code.

Wouldn't it be a better idea to have the backend a seperate process that communicates via a unix socket or smiliar? That way the frontend could also be in a completely different language without having to create specific bindings.

[TheEvilSkeleton]

We wouldn't be rewriting the entire thing, just bits and pieces where it might make sense. Plus, assuming we use PyO3, there wouldn't need to be any frontend rewriting - it would look just like another Python library to the frontend(s).

I'm not sure how I feel about this. I don't like the idea of writing the backend in two entirely different languages, even with abstractions. I rather fully commit to it or not commit at all.

[orowith2os]

Wouldn't it be a better idea to have the backend a seperate process that communicates via a unix socket or smiliar? That way the frontend could also be in a completely different language without having to create specific bindings.

I don't think so, it might just be easier to have it be a Python library, so it more or less fits in how the backend does now. Plus this bit feels like it's deserving of a whole other discussion, and isn't really relevant here.

The boilerplate shouldn't be too much - it should be a simple macro that expands to the full implementation. At most, I see three macros: one for structs, one for enums, and one for functions. That said, I need to implement it to see - I'm reading up on the maturin docs to see what would be necessary to get a hello world going.

[axtloss]

Plus this bit feels like it's deserving of a whole other discussion, and isn't really relevant here.

The way it's going to be implemented is very important to deciding if we go through with this, as different implementations can cause different maintenance efforts and generally different benefits.

If we would go with using the backend as a daemon, it would be less code (no bothering with python bindings), people who want to create different frontends (for example in qt), wouldn't be forced to create their own bindings, but could just talk to a daemon.
From what I can tell a daemon backend has more benefits than a direct library.

[Kinsteen]

I'm not sure how I feel about this. I don't like the idea of writing the backend in two entirely different languages, even with abstractions. I rather fully commit to it or not commit at all.

Having a full Rust backend would maybe be beneficial, but it's a long way to go to do that right now, so I think we should keep in mind the idea to replace everything with Rust, but we can't realistically do it in a short span of time anyway

[TheEvilSkeleton]

I don't expect it to take little time. If anything, I don't want that at all. I much rather have everyone take a long time to make it good than rush bad code and decisions.

[orowith2os]

I don't want to fully commit to this all at once, because it could end up just not being worth it. That's why I'm going to start this off with a Hello World, then a simple rewrite of one of the backend utils. If both of those go well, we might be able to go all-in.

But I don't think an all-or-nothing situation would benefit anyone here. That would lead to rushing bad code and decisions, as you mentioned. We can (from what I can see) easily integrate it here and there, within reason. But rewriting the entire backend is a major undertaking and will only make it harder to work on.

Dip our toes in the shallow end, and see if we like it. Don't force us to cannonball into the deepest part of the pool.

[Kinsteen]

Agreed. There is the repo/download/startup séquence I would like to speed up, and I think Rust may be the way to make it way faster than what it is today. It would "just" be a chunk of the backend, but could benefit greatly end users by having a way more snappier startup of Bottles

[mirkobrombin]

I am all for it.

[axtloss]

If we go with this idea, it should be seperate project and rust backend should be developed independently, to the point where it is possible to either use the backend as a drop in replacement with minor to none patches in the frontend. That way, in case we realize that a rust backend is not the way to go, we can drop it and it would not affect bottles in any way.

If we only port specific parts to rust, we would not only have to maintain more codebases, but we are also more vulnerable to breakage as we now rely on pyo3.

An exact discussion on how to do the rust backend is also necessary as I currently don't see pyo3 as a good option compared to having a daemon backend.

[Kinsteen]

@axtloss Today the frontend is already 100% disconnected from the backend, there's a signal/event system that makes the communication between the two when necessary. It is theoretically already possible to write another frontend fro Bottles.

I think the approach with PyO3 is the better one in our case, switching to a daemon approach would be quite the work without much benefits imo

[TheEvilSkeleton]

But I don't think an all-or-nothing situation would benefit anyone here. That would lead to rushing bad code and decisions, as you mentioned. We can (from what I can see) easily integrate it here and there, within reason. But rewriting the entire backend is a major undertaking and will only make it harder to work on.

No? I advocated for taking all the time we can to rewrite everything. Yes, it's a major undertaking, but so is adding Rust as a dependency - the majority of us don't know Rust, meanwhile every developer in the team, except you, know Python.

We still haven't discussed about maintenance. Who's going to maintain it? So far, only you and @axtloss know Rust, so only the both of you might maintain it. If one of you don't want to maintain, then the bus factor would be incredibly low. Someone who knows Wine and Rust, and is interested in contributing to Bottles is going to be much harder to find.

Dip our toes in the shallow end, and see if we like it. Don't force us to cannonball into the deepest part of the pool.

I believe that adding Rust in an environment where most are unfamiliar with it is already a cannonball, so I prefer to have it tackled head on or to avoid it completely.

[axtloss]

@axtloss Today the frontend is already 100% disconnected from the backend, there's a signal/event system that makes the communication between the two when necessary. It is theoretically already possible to write another frontend fro Bottles.

I think the approach with PyO3 is the better one in our case, switching to a daemon approach would be quite the work without much benefits imo

The signal/event system would be exactly the same if using the backend as a daemon, the one added part would be having a reciever/sender function to communicate with the daemon.

using pyo3 would make us rely on an external system that can break and require us to rewrite parts if pyo3 updates with breaking changes (which apparently already is the case based on the issue linked in the first issue message by oro)

[orowith2os]

I think I'll close this as not planned for now.

• Maturin doesn't support integrating with a preexisting Python project
• The CI needs a bit of work to cargo vendor the dependencies so meson works with it
• Too little developers here know Rust and/or are interested in working on the Rust side of this
• The work needed to rewrite even minor things alone is a bit annoying, because of PyO3 feeling a bit immature

The effort needed for this whole thing would be better put to improving the other places of the codebase, and maybe integrating more familiar languages like Go, which the developers are more intimate with, and would likely give many of the same benefits as Rust, due to also being compiled (and the garbage collector won't matter very much for Bottles).

I'll be trying to learn Go here and there in the meantime to see if I can work myself up to adding that instead, and maybe we can work on improving Bottles with that where necessary instead.

[hardBSDk]

The main reasons to use Rust is faster processing and better error reporting when compared to Python (the Rust compiler give more rich logs), this will improve the Bottles stability when codebase changes happen.

About the maintenance problem, it's better to use other repository to rewrite the backend in Rust and only developers with Rust skills will commit, that way there's no maintenance problems.

Python developers on the Python backend and Rust developers on the Rust backend, of course the Rust backend will need to be updated to match API compatibility with the Python backend to not break the frontends.

[zefr0x]

Pain points

* Flatpak builds may be more annoying
  
  * Cargo is another build system that uses external package repositories; this would need to be figured out somehow to use it with Bottles. GNOME apps vendor the dependencies on their release archives and use meson to build the project, Bottles could follow in their footsteps.

No need for vendor the dependencies on their release archives, you just need a Meson build option for offline build and another to set the CARGO_HOME env var.

So in meson.build:

if get_option('cargo-home') == ''
  cargo_env = [ 'CARGO_HOME=' + meson.project_build_root() / 'cargo-home' ]
else
  cargo_env = [ 'CARGO_HOME=' + get_option('cargo-home') ]
endif

cargo_options = [ '--manifest-path', meson.project_source_root() / 'Cargo.toml' ]
cargo_options += [ '--target-dir', meson.project_build_root() / 'src' ]

# When cargo-home already contain needed deps and we want to build offline
if get_option('offline-build')
  cargo_options += [ '--offline' ]
endif

Also in the flatpak manifest you will have:

    config-opts:
      - -Dcargo-home=/run/build/bottles/cargo
      - -Doffline-build=true

and a cargo-sources.json generated using flatpak-cargo-generator.

[hardBSDk]

A Rust implementation also help in the packaging process, complex Python programs needs to use pip and this is bad for cross-compilation, for example.

[mirkobrombin]

We are switching to Go

[joaopauloalbq]

GO language looks great, but I would wait for MOJO language to be released.