Official Node.js native addons

[jasnell]

@nodejs/tsc @nodejs/n-api @nodejs/collaborators

I've been working on QUIC for a while now. Unfortunately, further development on it is largely stalled due to OpenSSL refusing to land the necessary support before the 3.1 timeframe. So while we have something that currently works, we can't actually ship it.

I've been considering the possibility of re-implementing the QUIC stuff as a separate native addon. However, doing so is going to be difficult because it currently relies on a number of internal bits that are not exposed via N-API currently, which means I'd just end up having to duplicate a bunch of code that already exists in core, which sucks.

Currently, the only layering we have is: Something is either implemented natively in core directly against the Node.js internal or V8 APIs or is implemented against the limited N-APIs and potentially duplicates core or has to "reach in" to Node.js internals in unsupported ways.

What I'd like is to have are Node.js official native addons that (a) are distributed separately from the Node.js core binary, (b) supported and maintained as part of the Node.js project in that when a new version of Node.js is published they are guaranteed to work, and (c) use an extended ABI stable API that goes beyond what N-API does today to allow more reuse of existing Node.js internals.

For example, the kinds of existing internals I want to be able to make use of are utilities such as node-sockaddr.h, async_wrap.h, debug_utils.h, etc.

[addaleax]

For example, the kinds of existing internals I want to be able to make use of are utilities such as node-sockaddr.h, async_wrap.h, debug_utils.h, etc.

That’s going to increase the API surface a lot, but I do agree that it would be nice to provide APIs in the style of N-API that would be suitable for building Node.js core parts. I don’t think you could demand indefinite API or ABI stability in that case, and only have that between major Node.js versions.

[jasnell]

Yes, ultimately it would be a balance between what would be ideal nice to have in a perfect world vs. what we can realistically achieve. I definitely think we should start working towards as much ABI stability as possible in our own internal API patterns, regardless of whether those make their way in to N-API or not. The most critical piece of what I'd like to see here is the idea of core supported native addons, however we can make it work.

[kaizhu256]

if realized, would like to nominate @mapbox's sqlite3 for official status. this would simplify node-devops by allowing me to write stateful-apps with zero [npm] dependencies.

[jasnell]

I'd be strongly -1 on this. The bar for what would be an officially supported native addon should be kept very high. For instance, if I did go this direction with quic, I would fully expect to have to take it through a proposal and acceptance process for evaluation

[benjamingr]

I am in favour of this.

I am also in favour of this for JavaScript addons and packages.

Landing things in core makes iterations sometimes needlessly long/hard.

[Flarna]

Which ABI stability timeframe do you have in mind here? Per major version like v8 or longer like NAPI?

Do you have a C or C++ API in mind? C++ adds quite some pitfalls.

The long term stability of NAPI is nice. But if you distribute a native addon which should also work on older nodejs version it forces you to stick on old NAPI version (in special if distribution is via precompile binary). Alternative is to ship one binary per major version like before NAPI existed and benefit from NAPI only because of less maintenance effort.

One important point is in my opinion the interoperability between NAPI and this new, extended API. E.g. conversions between NAPI types and v8 types should be cheap and easy to allow use of v8 features not available via NAPI.

[jasnell]

That's all open for discussion really. I have a goal in mind, and want to work through how to achieve it. I'd rather not have to duplicate function that already exists in core but currently would have no other choice.

[rvagg]

if realized, would like to nominate @mapbox's sqlite3 for official status

And here's a key problem, nicely illustrated almost immediately!

What's the limiting principle for what addons get to be "official"? Who's gating this? What are the rules? Is there some new body managing this? Is the TSC willing to handle this list? Does core then need to take over security responsibility, reporting and handling for these addons? What happens when these addons fall behind, who's doing this work to keep them updated?

[jasnell]

Absolutely no different than what we have now... core would need to make a decision on whether a given module would be "official". The relationship would be the the same as what we have with readable-stream except that it wouldn't be based on an existing built-in.

[devsnek]

For those of us following along at home, how does separating quic from core fix the unreleased openssl issue?

[jasnell]

I'm looking at the possibility of bundling an alternative tls implementation in the addon that doesn't suffer from the same release timeframe problem.

[bengl]

It seems that there's an intent to make TSC approval a prerequisite for a module to use a particular set of APIs. I'm a little confused as to how this mechanism would work. If the modules are to be distributed separately from Node.js itself, what stops arbitrary userland modules from using these APIs? If there is no such mechanism, why bother making this apparent distinction?

[kaizhu256]

What's the limiting principle for what addons get to be "official"?

for me, there's value in being able to hack together a stateful webserver in 400-lines or less and deploy it with zero-config and zero-dependencies. you can do that in python but not nodejs, b/c python comes with sqlite3 included.

i group javascript similarly with languages like perl/python/php -- its a "scripting" tool, enabling the masses (less-technical lay-people like me) to productize our ux-ideas with minimal tooling and just a few hundred lines of throwaway code -- as opposed to 1000's of lines of needlessly-distracting-machine-level c#/c/c++/rust code.

sqlite3 is the only external-dependency i need in 90% of nodejs-scripting use-cases that require state-management, and i would be happy if it was the only exception ever made by TSC to include as an external addon. mapbox/node-sqlite3's codebase is also mostly stable. since 2014, only 63 lines have been added/removed to its core /lib/ (git-changelog).