Luvit 3.0

[SinisterRectus]

The idea of releasing a new major version of luvit with breaking changes to fix outstanding issues or to address some of its shortcomings comes up occasionally. We recently chatted again about this in Discord. I think it would be good to have a place here for people to share any thoughts on the subject.

I will share my ideas in an independent post below this one. Please do not hesitate to share your own. Thanks.

[SinisterRectus]

Luvit was reborn as version 2.0 in March 2015. I joined the community a year later in early 2016 when I leveraged luvit as a niche solution to developing Discord bots in Lua, out of which I grew a small sub-community of Discord users. Since then, I've been an occasional contributor to the project and luvit has become my go-to framework for almost anything related to Lua. I consider myself an amateur programmer, so I do not claim to have any sort of well-informed professional opinion on what luvit is or should be. This is just my opinion based on my experience using luvit.

Luvit as a Node.js clone

When luvit 2.0 was released in March 2015, node was on version 0.12, io was on version 1.5, and JavaScript was ES5. By the end of the year, node and io had re-merged to become node 4.0 and ES6 was finalized. Since then, node has undergone changes sufficient to bring it to version 15.8, while callbacks have given way to promises and async/await-style coding with annual updates to JS/ES. In the same time, luvit advanced to version 2.17 with minimal API changes and today it is still based on the old callback-style of coding. I think that if luvit is intended to be a node clone, then some effort should be made to bring it up-to-date with respect to the current node APIs and modern JavaScript (or TypeScript).

Alternatively, the connection to node can be loosened and we can make luvit more of its own framework. Maybe there are some other frameworks or runtimes that we can look towards for inspiration? Deno is an obvious alternative, although I have not had the chance to look closely at it. The only other dedicated async I/O framework that I am remotely acquainted with is vibe.d. There is Python that has async I/O in its standard libraries, plus languages such as Julia and Janet that use libuv, but I'm not sure to what extent these may serve as inspiration for a Lua framework.

Regardless of whether we continue the node-clone approach or redesign luvit into its own framework, I think that coroutines should be emphasized over callbacks. As shown by Tim's "coro" libraries, coroutines can be used to make a robust async/await-style API.

Luvi and luv

I think that the modularization of luvit into luvi and luv was a fantastic idea and this should stay the way it is. I don't think that there are any critical issues that need to be addressed here, but there are some points worth mentioning.

Luvi loads deps/require and deps/pretty-print if they exist. I never understood why these modules are not just included in luvi. Can we move them?

The idea of switching to xmake or zig to build luvi or luv has been thrown around recently, which may involve making changes to the way we use certain libraries. I can't comment much on this, but I think @truemedian can. He's already made some progress using GitHub Actions and zig.

Additionally, @creationix has suggested moving luv from a callback API to a polling API. See polluv as an experiment. We can't easily break luv's current code because its versioning is bound to libuv's and because it is also a LuaRocks package, so any major changes to luv will have to be carefully considered.

Lua source files

We are maintaining Lua source files across three repositories. Some of them have multiple copies (for example, http-codec is in luvit and lit) and some have duplicated code (for example, luvipath in luvi appears to be a duplicate of pathjoin, which is in both luvit and lit). I think that we should make an effort to deduplify these source files and put them in one central source where possible. There is also the matter of moving luvit deps visited in #952.

Custom module system

One of the major defining features of luvit is its custom module system with relative requires, deps and libs resolution, lit packaging, and luvi bundling. It is a departure from standard Lua's module system and has several standing issues. I think that some of these issues are addressed by switching to luvit-loader, but that switch was put on hold due to the necessity for breaking changes (#932). Bumping luvit to 3.0 would give us an opportunity to revisit these issues.

Documentation

Not much to say about this one. Luvit's documentation is weak, but at this point, I think our time would be better spent updating luvit's 6 year old code instead of documenting it. We could then document the new code as its developed or immediately after.

LuaJIT vs PUC Lua

LuaJIT is great. It's fast, it has a nice FFI, it's popular, and it is based on Lua 5.1. Despite being 15 years old, Lua 5.1 is still a common and capable language today. At the same time, Lua 5.1 is now 3 major versions removed from the current Lua 5.4, and there are no plans to update LuaJIT's feature set to include anything beyond 5.2. I think that it would be worthwhile to offer a "PUC Lua" version of luvit and luvi.

[creationix]

This is a great write-up @SinisterRectus! I'll make my own post with some proposals later, but first some quick feedback:

connection to node can be loosened and we can make luvit more of its own framework

I agree. I instantly regretted making luvit a node clone as soon as the 2.0 rewrite was done. The virgo project that sponsored the luvit 2.0 rewrite used node style APIs extensively and it allowed us to be lazy about documentation by simply pointing people to node's website. But none of that is important or relevant anymore.

If someone really wanted to have an updated node clone, it could be done in userspace easy enough. But I feel it's not a good use of our limited resources for the core team to maintain such an API. We should instead focus on use cases where luvit is a good technology and make those better while embracing Lua's strengths. The coro-* and weblit libraries, for example were a great experiment and could be made into a very robust framework with some polish and good documentation.

I never understood why these modules are not just included in luvi

This was because they were non-trivial libraries and keeping them in userspace with magic hooks in the luvi binary made iterating on them a lot easier. Also it allows people to make really lean luvi based binaries that don't use them at all since they were optional. We can consider having a small set of libraries like this that are baked into luvi and official core APIs.

switching to xmake or zig to build luvi or luv

I also think this is a great idea, but haven't had the time to dig into the technical details about what is needed.

moving luv from a callback API to a polling API

If we did this, it wouldn't be a new version of luv, but an entirely new library. The entire purpose of this is to allow luajit's FFI to have better access to libuv. There would be virtually zero shared code between this approach and the luv bindings. For now this should probably stay a userland experiment. The binary extension mechanism allows this to be done completely externally if desired.

I think that we should make an effort to deduplify these source files and put them in one central source where possible

Agreed, but this is a tricky bootstrapping problem. For example, luvi needs the path joining algorithm for it's primitive vfs APIs. This would be another reason to consider simply moving these core libraries into luvi instead of making them optional with magic hooks. The luvit 2.0 rewrite focused on the most minimal and flexible core possible, but I think we've reached a point in the years since where we can pick a few libraries to include with core that make sense there.

some of these issues are addressed by switching to luvit-loader

+1000

I think our time would be better spent updating luvit's 6 year old code instead of documenting it

Sounds good to me. As long as we do write docs afterwards.

LuaJIT vs PUC Lua

These have diverged to the point that it's really hard to support both equally. And some things don't even make sense in both. For example the polluv experiment doesn't make any sense for PUC Lua since it doesn't have the fast JIT based FFI built in. I'm fine with spending some time and seeing exactly how much effort/compromise is required to support both. There is a similar dynamic with supporting luarocks and lit at the same time and stock lua/luajit vs luvi based builds. We kind of went crazy making luvit 2.0 be super flexible and can probably keep a lot of those qualities in the 3.0 version as long as we're aware of the tradeoffs and think they are worth it.

[creationix]

One thing I would like to explore in a breaking change is find out if it's feasible to remove the openssl dependency, but still have access to common hashing algorithms and TLS (1.3 at least). The hashing algorithms could probably be done in lua and be fast enough for most things with luajit. But TLS is a bit more work. This desire is mostly out of pain from building openssl and trying to get cross compiling working better.

[truemedian]

The issue with dropping openssl for better cross compilation capabilities is that none of the TLS libraries I know of support cross compilation with the ease we're looking for.

If we were willing to drop server TLS capabilities, we could use a pure-zig approach: alexnask/iguanaTLS. However it is only TLS 1.2 and only client side. It is very easy to cross compile though.

[creationix]

Hmm, only client-side and 1.2 will work for a lot of use cases. Very few web service APIs are TLS 1.3 only and very few don't support 1.2. For servers, I tend to use nginx as my TLS terminator and proxy to the luvit server speaking plain HTTP.

I think it's safe to say that we can't drop openssl completely since I'm sure there are some people who need it. But maybe we don't provide prebuilt binaries with openssl on the platforms where it's difficult and then encourage most people to use the new and improved tiny version that has a working TLS client.

[zhaozg]

I think it's safe to say that we can't drop openssl completely since I'm sure there are some people who need it. But maybe we don't provide prebuilt binaries with openssl on the platforms where it's difficult and then encourage most people to use the new and improved tiny version that has a working TLS client.

I have lua-mbedtls, it's small that OpenSSL, but full support TLS client and server.

[truemedian]

Having more bindings makes luvi (since in reality this is what has to be tackled first) 3.0 a bigger project, but I suppose that's unavoidable.

[squeek502]

Here's my 2 cents for whatever it's worth. My inclination would be to lean much harder into the existing Lua ecosystem:

• Drop Luvit/Luvi-specific require behavior to as close to zero as possible (related: Luvit require doesn't treat . as a path separator when resolving modules #1053, Switch luvit to use luvit-loader instead of luvit/require package #932). Basically, the idea would be for Luvit modules to work totally fine when require'd from PUC Lua.
  • This would entail removing support for relative requires, circular requires, etc. Support for modules in deps and libs could be kept but would be runtime-specific (i.e. support for deps and libs could be added by the runtime or not, but the modules shouldn't need to care about that). There's a bit of discussion about this sort of thing here: Switch luvit to use luvit-loader instead of luvit/require package #932 (comment)
• Modules/packages could be published to both Lit and Luarocks, where Lit would support the libs/deps way of structuring your workspace and Luarocks would support its way (i.e. coro-http-luv and Lit's coro-http wouldn't need to be separate--identical versions could be published to Lit and Luarocks).
• Luvi would be something like luastatic, but with more focus on cross-platform building and with built-in support for things like adding openssl, etc to the binary. This is not too far off of what it is now, but the idea would be to make it work with any Lua version and make everything optional (so you could just use it as a luastatic replacement if you want, i.e. if I wanted to move this thing I wrote from luastatic to luvi, it would support that use case; here's its luastatic build script for reference)

I have a feeling this might not work as well in practice as it does in my imagination, but I thought I'd at least throw it out there.

[SinisterRectus]

Our custom require seems to be that which trips people up the most when they move into luvit-land, so I agree; that should be a high-priority fix.

A more streamlined solution for using Luarocks modules would be cool. I think this kind of goes hand-in-hand with the require issues. But if we can use Luarocks out of the box, would Lit be useful?

I agree with your vision of luvi. This reminds me of ninite where you check only what you want. It would be more work on our end to provide various iterations of luvi, but I think it would be nice if it works.

[truemedian]

I think the largest issue with require right now is that it has some pretty weird edge cases that obviously weren't thought through. For example: file.lua fails over the luvit require and then ends up in lua's require, instead of requiring ./file.lua. Being able to do requires to relative paths is pretty helpful (along with being able to require deps/ and libs/.

[SinisterRectus]

Yeah I agree that relative-requires, plus libs/deps/bundle-requires, are super useful. One thing I thought of is having a totally new system that complements require rather than modify or replace it. For example, import("./path").

[creationix]

I like the idea of not reusing require for the new logic. I'm +1 on renaming the relative imports to import.

[hishamhm]

Our custom require seems to be that which trips people up the most when they move into luvit-land, so I agree; that should be a high-priority fix.

@SinisterRectus that aligns with my past experience in LuaRocks as well: in LuaRocks 1.x, I shipped a patched require that would do magic automatic versioning (if you did require("x") which depends on Y < 1.2, then the subsequent require("y") would pick up the right version of Y even if you had multiple versions installed). People were very reluctant on loading a module that patched a standard library function.

For LuaRocks 2.x, we switched to providing an optional package loader that you can load with require("luarocks.loader") which then enables the magic behavior. Some projects used it, mostly to auto-load the LuaRocks-configured paths at front, not to do version resultion, others were free to ignore it. Still, the magic behavior would occasionally trip people up in the event of conflicting dependencies.

Ultimately, very recently in 3.4, we introduced dependency version lockfiles (a la yarn and others), which allow users to extract their full dependency tree into a plain list with fixed versions and be able to see what exactly is going on and avoid surprises. In my experience, Lua people more often want to avoid having multiple versions of things installed rather than having something manage them. I'm not sure if that's the expectations are the same for people coming from the Node world (I'm assuming the expectation there is more like node_modules is a black box and I only care about the things I'm depending on directly).

So yeah, +1 to keeping more advanced behavior separate from require, and, if possible, allowing people to use individual modules in a simple way without requiring that advanced behavior.

A more streamlined solution for using Luarocks modules would be cool. I think this kind of goes hand-in-hand with the require issues. But if we can use Luarocks out of the box, would Lit be useful?

The question would be how to keep the advanced behavior "hidden" from Lua end-users, so they could require the top-level modules as usual, and load them via LuaRocks.

I'm not very familiar with the libs/deps specifics in the Luvit world (I was invited into this thread by @creationix to share the LuaRocks experiences), but if there's some advanced logic that's not supported by LuaRocks, such as a node_modules-style nested tree of dependencies, one option would be to use Lit as the build backend of the LuaRocks packages, where it would produce the tree for a given top-level module and LuaRocks would package the whole thing as "vendored" modules (i.e. the rock for x storing x.lua and x/libs/y.lua or something like that), getting simplicity at the cost of replication. That approach would be essentially "wrapping Lit package as LuaRocks packages", if the goal is to make them more widely accessible.

Since LuaRocks 3.0 there's a luarocks init command which sets up a local lua_modules tree where people into which people can load their dependencies (we also added user/module notation (though for luarocks.org, not Github), etc.). From a quick look at https://github.com/luvit/lit/blob/master/package.lua and https://github.com/luvit/lit/tree/master/deps, I think the existing functionality in LuaRocks could provide a somewhat similar end result (though with a more verbose directory structure, at least in the current version), but usually I don't see people committing their lua_modules with resolved dependencies into the repo like that (what happens with C-based deps which need to be compiled?).

Still, I ended up not 100% happy with the ergonomics of using LuaRocks 3.x in this model, precisely because I didn't want to touch require, so users have to use a clunky wrapper script over the interpreter or configure their LUA_*PATH* environment variables correctly — this looks like the kind of things a smart import would solve.

In general, Lit seems to me like a much leaner and cleaner package manager. LuaRocks carries a lot of baggage from being very old, assuming almost nothing about its environment, and trying to be all things for all people. If you have a solution that works, I think it's unlikely that you'd get a nicer experience by switching over to LuaRocks today, honestly, apart from gaining access to the luarocks.org collection of packages.

It seems that the changes in LuaRocks 3.x have brought it a bit closer to Lit in functionality, though not quite there yet — still, it would be interesting to know what else is missing and if there are ways we could bring the projects closer together for mutual benefit. Perhaps by having Lit learn how to read dependencies from the luarocks.org repository? (that would be the opposite approach, "wrap LuaRocks packages as Lit packages", essentially.)

[creationix]

Perhaps by having Lit learn how to read dependencies from the luarocks.org repository

I think this could work well. The biggest difference between using luarocks and lit to install packages is where it puts them. If lit could pull from the luarocks repository and install them in deps folders (which is identical logic to node_modules btw), they should work without needing package authors or luarocks.org to change anything.

Also at the same time, I'm still feeling like separating the "smart" require from the built-in require and renaming luvit require logic to a new function like import will clear up a lot of misunderstandings.

[creationix]

Actually I misspoke. If we rename require to import when using the new search paths, then existing code that uses require will not be able to find modules installed to deps/*. So I guess the two approaches are mutually exclusive.

[Duckwhale]

Disclaimer: I'm not an expert on the technology Luvit is based on, and I almost didn't want to participate in this dicussion as it feels like the "big boys table" and I'm certainly not qualified enough to give much input here. However, perhaps you will find it enlightening to hear a random user's perspective :)

Some more background: I've worked with Lua for the past five years or so, and for the past two I've been using both Luvit and NodeJS in parallel. As a result, I can describe my experience, especially that of someone trying to learn, and that might be of interest when you think about the future of this framework (as it's vital to get new people on board).

---

Let's talk about first impressions, because they matter, a lot. Whenever someone is looking for a Lua-based framework that might make their lives easier, they will probably pay attention to at least some of the following things:

• Which problems does this framework solve?
• How active is the GitHub repository? How are issues handled?
• What is the project's documentation like?
• What features and functionality will I get access to via existing APIs?

Ultimately, users want to know: Can the framework solve their problems, and Is it likely to be a liability or an asset?

Now, let's go back in time about two years, when I first heard about Luvit. What was my initial impression then, and has it changed now?

• The framework appeared to solve at least the "Lua has no batteries" problem I was looking to solve. So far, so good! Though it's hard to tell, since the website doesn't really say much in this regard.
• Well, there appear to be many different repositories and I don't even know what they're for. They all seem pretty dead, however. Not a good sign!
• There are issues from literally years ago still open, and quite a few never even received a reply? Uh-oh...(most people probably leave here)
• "Alright, but I really want to use Lua and these Node-like APIs sound like they will allow me to do what I can easily do in JavaScript (at the cost of having to use that abomination of a language). Let's take a look at the docs..."
  • One of the very first pages I looked at was the https://Luvit.io/docs.html link. What does one find here? A link to NodeJS? (I didn't know what Node was back then, or how it relates to Luvit) That doesn't help at all.
  • What is this project about? There wasn't even an introductory paragraph explaining to someone who is new what Luvit is and why they should use it, in terms that they can understand.
    *Instead, I was bombarded with links to various things that meant absolutely nothing to me ("luv, luvi, luvit, lit; I don't know what to do with it" :D).
  • Almost mockingly, it's followed by 'Reading source code is always fun!". I mean, sure, but if I have to understand the code before using it, I might as well write it myself.
  • At that point I thought, "Are they kidding me?" and clicked on the docs for the "http" API, only to find... nothing. It was essentially empty, and still is to this day.
  • I don't think words can express how incredibly bad this first impression was, and it saddens me to see nothing has changed in that regard.

TL;DR: The only reason I'm following the project still is because I'm way too stubborn for my own good. Most people will not stick around after an experience like the one I just described! And could anyone blame them when there are so many other technologies and programming languages around that are easy to use?

The obvious conclusion from this is that how Luvit is currently represented simply doesn't inspire confidence. It looks like using it will be a liability, much like the rest of the Lua ecosystem, which frequently suffers from similar issues... and that's a shame because I think providing a Node-like experience in Lua (which doesn't have to mean a direct clone) is in actuality a really cool idea.

Speaking of Node, I later went and did what I sought to do with Luvit, using NodeJS. It took me mere minutes until I had a basic prototype of the app, a WebSockets-based server, running. When I first tried to do this in Luvit it took an entire day and a lot of help from someone in the Discord just to achieve the same basic thing, but there were still a lot of question marks / stuff that didn't really work the way I wanted it to.

I guess what I'm saying is, Luvit does a very poor job of explaining what it is, how to use it, and users are likely going to have a bad experience overall. All of these things are crucial to get right for any Open Source-project if it is to ever be successful (assuming success equals people using the software you spent so much time creating), and all the amazing technology behind it will go underappreciated if UX is completely ignored.

The good news is, there are people out there who like Lua enough to potentially use Luvit over Node, if the state of the ecosystem in general hasn't yet scared them off, and they might be willing to help. I know because I'm one of them. However, their life should be made easy and not needlessly difficult, as is currently the case. It takes a concerted effort but a little goes a long way already.

---

Finally, my thoughts about some of the more technical points you made:

• Luvit doesn't need to be a direct clone of Node in order to be useful. If I'm able to do anything Node can do with its APIs and things are made very simple, i.e. they "just work", then it really doesn't matter how exactly it functions. If it does offer very similar APIs that's certainly not bad, as it can help people understand more easily, but it's not strictly necessary given it has the same utility and ease of use
• LuaJIT being in a weird maintenance limbo implies, in my opinion, that purely relying on it moves the project closer to "liability" than "asset" in the grand scheme of things. Undeniably there's value in using it, but supporting the latest official Lua versions shouldn't be neglected either. For an ideal user experience, one should be able to run their native Lua apps using the latest PUC-Lua in Luvit without issues and similarly be able to use 5.1/LuaJIT-features as well as Luvit-APIs when needed. This may be harder than I imagine, but it'd be pretty damn useful if you could just use it for anything and it simply works out of the box
• For package management/lit, using the new LuaRocks approach and supporting lockfiles similar to NPM would likely be the most intuitive. If there was more interoperability and cooperation between the projects, that could help in creating something many people can get behind. IME Lit works much better than LuaRocks (on Windows), but it has an even smaller ecosystem and doesn't always do things in the way that one would expect... so perhaps we could get the best of both worlds by making them interoperable and agreeing on a set standard?

---

I hope no one takes offence at what I wrote. I'm ardently campaigning for a better user experience because it plays directly into one of the main strengths that Lua as a programming language has, and also because most developers (in any language) don't seem to value it nearly highly enough. I understand why and I don't blame them, but I think it's unfortunate because the purpose of any software is to be used, by users who often don't want to know all the details. It follows that using it should be as simple as possible, to the point of being foolproof even.

If you would like to help Luvit become a true equivalent to NodeJS in spirit, then you should IMHO focus much more on making things "just work", making it work intuitively, and making sure that the technology can be used by almost anyone on any platform with as few stumbling blocks as possible. Software that people don't understand they won't use, and software that isn't used is literally a waste of time to create and maintain. Node is extremely easy to use; that's one big reason why people use it in the first place. The large amount of libraries are just a side effect of it filling a real need in my eyes and if Luvit was more user-friendly it might have more actively-maintained packages, too. The same could be said about LuaRocks, too, but that's besides the point.

Given the power of LuaJIT and the accessibility of Lua, I think with Lua one could achieve what Node/JS managed to do, but better, if there was any project that prioritized accordingly. Luvit seems to be the one best suited to it from what I can see, so the question is... is this something you actually want? If so, I'm happy to help.

Of course I'm not really sure what the project's vision and goals are (or were), so perhaps I'm envisioning something very different from what you set out to do. Defining that and then featuring it prominently somewhere might be a good first step.

[creationix]

Very good points @Duckwhale.

You're right that luvit is kinda scary in it's current form to anyone who knows what red flags to look for. This wasn't always the case. There was once a time many years ago where working on luvit was my full time job. You'll quickly notice when this time was by looking at the dates on the blog posts and overall commit velocity to the project. But even then despite me working full time to promote it (including me speaking at international node.js and lua conferences about luvit), despite me staying on top of the issues and doing all the things you mentioned, it never took off. Eventually my employer got impatient and pulled the plug on funding the open source stuff.

The project sat nearly completely dead for years until a new community emerged independent of me. This is the new community with an active discord channel and largely centered around the discord bot framework. I eventually noticed the new community and have been slowly off and on trying to give it time. I'm still the sole owner / sysadmin of the website (including the production package repository) which explains why it is occasionally down for hours or days at a time.

I used to be super active in open source authorship and now have somewhere around 500 personal github repositories. What I didn't learn about open source till it was too late is the cost of a project becoming popular. My most popular project was a tiny bash function known as nvm. The support workload became so great that I gave it away completely and it now sits in it's own organization that I don't follow or watch with a new person in charge https://github.com/nvm-sh/nvm. My second highest support load was either JS-Git or Luvit. Both had died (near zero interest) after just a year or two and I was busy living life and working at various companies that didn't allow me much open source time.

Once at Luaconf in Brazil during a Q&A session, Roberto Ierusalimschy was asked what could be done to help Lua get more popular. Was it better docs or certain key libraries, or more conferences or what? His answer surprised me. He said he was not interested in growing the popularity of the project. I now kinda understand. If luvit gets popular, it will mean more pressure and stress for me. It will mean I'll need to pay more hosting fees, I'll need to be more diligent to keep the server running all the time. I'll need to field angry messages from people upset about the lack of good documentation (not everyone will be as polite as you have been). I (or someone else) will need to do all this extra work with nothing to show for it. Maintainer burnout is a real problem today and I don't know the solution.

That said, I think your suggestions are great and support them. I would love newcomers to this project to have a good experience despite the risk that it might cause the project to get popular. Documentation in particular of the code and the project itself (how to cut releases, how to triage github issues, etc) will do wonders to crowdsource the maintenance this project desperately needs and currently lacks.

https://twitter.com/creationix/status/1359575881669357570

[truemedian]

I've been thinking over a few technical challenges associated with a Luvit 3.0 and the opinions that have been stated above.

Current Modularization

Obviously, the modularization of luvit, lit and luvi has caused a few separate problems (luv is not part of this consideration, and should remain separate). A small portion of luvit libraries has been duplicated in multiple different places, and while not a huge problem, it makes maintenance a nightmare. Additionally, most issues with luvit tend to span more than one repository in my experience.

Ideally, things wouldn't be this way. My proposal to fix this specific problem is merge luvit, luvi and lit into one single monorepo where all luvit development is centralized, and libraries can be shared throughout to prevent duplication.

This would solve the below issue.

I think that we should make an effort to deduplify these source files and put them in one central source where possible

Agreed, but this is a tricky bootstrapping problem. For example, luvi needs the path joining algorithm for it's primitive vfs APIs.

-- #1135 (comment)

Cross Compilation

Luvi is currently build hell (partially thanks to CMake, partially because of all the libraries it includes, and all of the options it supports) and can only output executables for it's native platform.

First, replacing the build system without also remaking Luvi from scratch in order to deal with all of the external libraries is a non-trivial job. Zig has been thrown around as an option, but I will not overtly recommend it because of my involvement. Zig would support cross compiling luvi to essentially every platform and it's build system is more comprehensive than CMake.

Second, allowing Luvi to cross compile should be rather easy if luajit is the only Lua backend that we support. Luajit supports outputting bytecode for all of it's architechtures from any native platform, the only issue would be with including every version of Luvi in a single binary so that it could be actually cross compiled. However, as long as luvi could be cross compiled, we would be able to use it to bootstrap the lit and luvit builds (as it does currently).

However, rewriting Luvi is a much larger endeavour, and would require a lot of effort (probably from multiple community members).

The Luvit "Official" Libraries

Luvit's libraries are birthed from the idea behind a Node.js code, but this lead to all of Luvit's built-in libraries becoming object and callback oriented. Object orientation is a special case, because a few of Luvit's structures did benefit from it. However, callbacks are not idiomatic Lua, and I believe that they should be removed (unlikely) or at least hidden (the creationix/coro-* libraries are a good example of how this looks).

As far as where I stand, a rewite of all of the libraries is not a horrible option, even if it is a lot of work.

The Require Situation

Here's my 2 cents for whatever it's worth. My inclination would be to lean much harder into the existing Lua ecosystem:

• Drop Luvit/Luvi-specific require behavior to as close to zero as possible (related: Luvit require doesn't treat . as a path separator when resolving modules #1053, Switch luvit to use luvit-loader instead of luvit/require package #932). Basically, the idea would be for Luvit modules to work totally fine when require'd from PUC Lua.

• This would entail removing support for relative requires, circular requires, etc. Support for modules in deps and libs could be kept but would be runtime-specific (i.e. support for deps and libs could be added by the runtime or not, but the modules shouldn't need to care about that). There's a bit of discussion about this sort of thing here: #932 (comment)

• Modules/packages could be published to both Lit and Luarocks, where Lit would support the libs/deps way of structuring your workspace and Luarocks would support its way (i.e. coro-http-luv and Lit's coro-http wouldn't need to be separate--identical versions could be published to Lit and Luarocks).

• Luvi would be something like luastatic, but with more focus on cross-platform building and with built-in support for things like adding openssl, etc to the binary. This is not too far off of what it is now, but the idea would be to make it work with any Lua version and make everything optional (so you could just use it as a luastatic replacement if you want, i.e. if I wanted to move this thing I wrote from luastatic to luvi, it would support that use case; here's its luastatic build script for reference)

I have a feeling this might not work as well in practice as it does in my imagination, but I thought I'd at least throw it out there.

-- #1135 (comment)

---

Our custom require seems to be that which trips people up the most when they move into luvit-land, so I agree; that should be a high-priority fix.
A more streamlined solution for using Luarocks modules would be cool. I think this kind of goes hand-in-hand with the require issues. But if we can use Luarocks out of the box, would Lit be useful?

-- #1135 (comment)

---

Yeah I agree that relative-requires, plus libs/deps/bundle-requires, are super useful. One thing I thought of is having a totally new system that complements require rather than modify or replace it. For example, import("./path").

-- #1135 (comment)

Documentation

Luvit's documentation sucks. And on top of the lack of documenation, a large portion of Luvit's library source code is uncommented. My proposal to fix this is either to either retrofit, or simply include in a library rewrite, documentation inline with the code as comments, this would use a tool (like ldoc or similar) to simply run over all of the code and generate the respective documentation for every library.

Keeping documentation right alongside the code would allow anyone modifying the code, and the reviewers signing off on pull requests to verify that whatever changed in the code is reflected in the documentation accordingly.

[SinisterRectus]

When you say mono-repo, luvit, luvi, and lit would still be separate applications, yes?

[truemedian]

Correct, it would just put all of the code in single place, so that things that need to be shared between all of them, can be.

Edit: for some clarification that I didn't provide above, moving luvit, luvi and lit into one place would also serve to link their versions together, so luvit 3.1 uses luvi 3.1 and lit 3.1.

[creationix]

I really would love to drop openssl in luvi. Also several of the libraries that were added over the years aren't really used much.
And I think we can for http servers since those are often sitting behind some kind of reverse proxy that handles TLS termination. But the thorn in my side is always https client support. I don't think we can drop that without loosing a large part of the community. For example, I'm pretty sure discordia speaks to discord's API servers over https.

If there was a way to handle TLS clients without including all of openssl, I'm all for it, especially if it's portable. Part of me wonders how hard TLS 1.3 only would be in pure lua or if there is a self-contained library for only TLS 1.3. Most API servers support this by now I think so lots of use cases would stay viable.

[truemedian]

I know TLS 1.3 is easier to implement, but that also means either finding another library or also implementing all of the cryptographic functions that it uses behind the scenes. Alas, any crypto library in C land is usually a pain to implement, harder to cross compile, and might not support all of the targets we want to.

I'm definitely shilling Zig here, but it both has the cryptographic primitives in it's standard library (which means support for a lot more targets than we can hope to [because of libuv]) and already has a working TLS 1.2 client library which is relatively easy to deal with. If TLS 1.3 is a target, then effort would have to be made, but at least TLS 1.3 is easier to deal with.

[SinisterRectus]

Part of me wonders how hard TLS 1.3 only would be in pure lua or if there is a self-contained library for only TLS 1.3.

I've thought about this too. It's been done in JS. For example, https://github.com/digitalbazaar/forge is a whole crypto suite. I don't know of any in Lua. I think no one is that ambitious.

[rphillips]

We might want to consider rustls and see if we can bind it to luv. Rustls uses ring for crypto. We would get a modern crypto and TLS 1.2+1.3 stack.

I worry about creating our own homegrown libraries and the maintenance that would bring.

[creationix]

I'm definitely shilling Zig here...

No complaints here, especially if we end up using it as our cmake replacement.

I wonder if supporting only TLS 1.2 would be good enough for now. That's more widely deployed than 1.3 anyway I think.

[creationix]

We might want to consider rustls and see if we can bind it to luv

That looks like a good option too, I especially like that option if it also gives us modern crypto primitives similar to what libsodium exposes.

I worry about creating our own homegrown libraries and the maintenance that would bring.

Agreed. It's one thing to implement sha256 in pure lua because there is almost zero maintenance for that, but all the algos needed for TLS brings the surface area big enough that there are bound to be problems that need addressing.

[TechnoJo4]

Here's my opinion, although I'm also just a random user, and I'm not sure how much this helps (most of this comment is just reiterating other people's posts). My background: I originally started using luvit because I wanted to write a discord bot in Lua, but after a while I started using it as "Lua with many nice batteries included".

Following sections are roughly in order of most important/relevant to me first.

Documentation

My proposal to fix this is either to either retrofit, or simply include in a library rewrite, documentation inline with the code as comments, this would use a tool (like ldoc or similar) to simply run over all of the code and generate the respective documentation for every library.

I very strongly agree with this. Additionally, I think it would be beneficial for this documentation to have some kind of machine-readable type information. This would allow users to automatically generate definitions for IDE completion (e.g. EmmyLua annotations) and/or for typed languages that transpile to Lua (e.g. TypeScriptToLua).

When writing or debugging code, I find myself quite often searching for the code for libraries I'm using, but I rarely know whether they might be in lit or luvit, or even one of creationix's GitHub repositories. A lit/luvi/luvit mono-repo would definitely help here.

The luvit libraries

Right now, the first step to any project for me, as a luvit user, is installing the coro-* libraries, and I do not use the base libraries unless I'm writing a standalone script. I believe making the coroutine wrappers the default would be good, although I don't know if that's possible.

luv/dependencies

I sometimes use luvi or lit make to package code I write, even when they have no dependency on any of the luvit ecosystem or luv, simply because of the convenience. Although I acknowledge this is most likely an unsupported use case, it would be nice to be able to build luvi with smaller dependencies or only a subset of its dependencies. Cross-compilation would also make this easier.

Honestly, I never use any Lua runtime other than LuaJIT, even when not using luvi(t), so keeping it as the only runtime seems very reasonable to me.

I don't have much experience with the rest of the subject, although I can say that HTTPS client capability was a requirement for a good portion of my projects using luvit, but any HTTP server I wrote (mostly using weblit) was either only for local use or behind nginx.

[truemedian]

LuaJIT is great. It's fast, it has a nice FFI, it's popular, and it is based on Lua 5.1. Despite being 15 years old, Lua 5.1 is still a common and capable language today. At the same time, Lua 5.1 is now 3 major versions removed from the current Lua 5.4, and there are no plans to update LuaJIT's feature set to include anything beyond 5.2. I think that it would be worthwhile to offer a "PUC Lua" version of luvit and luvi.

I agree that a version of Luvi and Luvit that supports PUC Lua would be nice for anyone who might happen to need it, but it means that we would have to get rid of the assumption that code is running on LuaJIT in luvit's libraries. From a quick search: luvit/buffer, luvit/dns and luvit/require, and luvit's main.lua entrypoint all use ffi; while luvit/los, luvit/pathjoin and luvit's init.lua all use jit.

However, I do also think that luvit's libraries should undergo a refactor or rewrite to cleanse the code of the callback-oriented mess that they currently are. Doing so would also allow us to get rid of any luajit assumptions that are currently in place and either remove them entirely, or use them as optional optimizations if available.

[zhaozg]

change list

• loadstring. replace with load
• setfenv, replace with load
• table.getn, replace with #
• table.foreach, replace with pairs
• unpack, replace with table.unpack
• jit, remove it, replace jit.os with ffi.os
• ffi, a ffi module facebook or jmckaskill

this change is not right, remove it.

[truemedian]

It's been a while since any movement here, so I've decided to take some initiative and start with throwing around ideas about how a Luvit 3.0 would look. I'm specifically focusing on what exactly is going to happen to luvit's libraries.

I've decided that the package split between luvit/luvit and luvit/lit is serving nobody (especially when we have the same package in both repos with one a full major version ahead) and that they should be merged into some sort of "luvit standard library".

I've also taken the liberty of typing out a style guide that can keep code throughout the project consistent, mostly just to add a bit more explicitness with regards to keeping different kinds of things separated (classes, namespaces, etc.)

For now all I am doing is mapping out how functions, classes, etc. should be laid out (with a little bit of type information) but I'm sure the prototypes will eventually start having actual code. I would also like to take the chance while prototyping these libraries to at least document each function with some form of inline documentation (currently EmmyLua) so that even if we don't end up hosting documentation, we don't fall into the pit 2.0 has where the effort is a monumental task.

This effort is documented in https://github.com/truemedian/luvit-api-design and contributions would be welcome, especially towards:

• Things you think should change (especially bugs).
• How a library should be laid out or work.
• Any new ideas or things we should add.

If you have a proposal, please create an Issue about it. If you would like to contribute to the growing list of prototyped libraries, PRs are welcome.

I would like to use the current state of luvit as a template, but there are obviously some things that need to change.

[creationix]

My day job for the last 2 years has been an extremely challenging role maintaining a massive openresty based hosting platform. It's given me a lot of perspective on one of the main luvit alternatives for server-side lua and how things are done and the kinds of tooling you need when working on a team of professionals keeping massive enterprise systems online.

I think for luvit 3.0 to flourish we need to first agree on the problem that it's trying to solve.
Luajit is an amazing piece of code and lua is a pretty good language (though needed tooling is years behind other popular languages for professional projects)

Some ideas of architectural/product directions we could take luvit (apart from the general cleanups discussed here already):

• Use a lua engine with better tooling.
  • Luajit is great for performance, but lua the language has poor dev tooling for large codebases.
  • The LuaLS project is the best we've found so far and it helps a lot, but it's nothing compared to something like TypeScript.
  • We've wished we could use the Roblox variant of lua just for the tooling and built-in type annotations/system.
• Use another language entirely.
  • lua was chosen primarily because luajit was fast, lua was good enough, and it had good coroutines at a time when JS only had just gotten experimental generator functions.
  • golang is another good language, but that ecosystem is plenty healthy with it's own everything, I doubt we could contribute much there.
  • wasm is interesting and promising as a runtime, but most languages targeting it aren't optimized for it. I would love to design a new language that targets wasm, but it's unclear how much that overlaps with the interests of the luvit community.
• optimize for linux only instead of using libuv
  • if we wanted to optimize for servers (which are almost always linux), we could drop libuv and wrap linux APIs directly using luajit FFI and get a good performance boost. Just.js is a perfect example of this type of approach.
• integrate more with the existing lua ecosystem
  • we could drop luvit's custom require system and embrace the existing lua ecosystem more. This is a huge part of what luvit is so I'm not sure the value here exactly, but it's an option.

These are just ramblings, feel free to ignore anything you find whack or offensive and keep on doing whatever makes you happy.

The one takeaway is I think we need a clear mission for whatever direction(s) this project goes. Do we optimize for quick scripts for fun, do we optimize for discord bots, do we optimize for webservers? Do we target teams of professionals with strict software engineering processes or do we target individuals hacking on stuff during nights and weekends for fun? Do we really want luajit or even lua? Do we want to integrate with existing package ecosystems or maintain or own? Where do we want to innovate, what's the value proposition?

[truemedian]

Luajit is great for performance, but lua the language has poor dev tooling for large codebases.

This is more of a systemic problem, but I suppose luvit could provide some simple tooling (like avoiding creating a full language server) in this regard that simply isn't available at the moment like a proper step-by-step debugger or profiler that integrates into an editor like vscode (and others).

node.js, but with coroutines and a lower-overhead VM

Luvit has performed really well in this regard, but over the years Luvit's core libraries (the ones the most influenced by node.js design choices) have proven time and time again that directly translating javascript callback-oriented design into idiomatic lua (and coroutines, but thats a whole different level) doesn't work very well.

Use a lua engine with better tooling.

Luvi is in an interesting position here where it technically supports choosing between PUC or Luajit, but over time I think the code has slowly bitrot into a position where that's all that its designed for; I feel that with a "little bit" of effort (mostly in luv) we could extend support (or at least make possible to support) a wide variety of different interpreters and luvi or luvit could provide the polyfills required to make up the difference between any of them.

As an end goal this would make it possible to potentially use completely different environments like luau with just a little bit of effort.

Use another language entirely.

I don't think this is a good idea, as it would require redoing a lot of the effort that luv pioneered on top of libuv. The PUC implementations of lua have been getting better and better speed wise, and lua is such a simple language to learn that I feel luvit should stay where it is in that regard.

optimize for linux only instead of using libuv

This would be an interesting decision but has some of the same pitfalls as switching languages. Libuv certainly isn't fast, but being able to run the same luvit code on linux, macos, and windows leaves it in a position that even PUC + Luarocks struggles to keep given that most external lua libraries give almost no attention to windows.

we could drop luvit's custom require system and embrace the existing lua ecosystem more.

Luvit's custom require and bundling system is part of what makes it so unique, unlike PUC Lua, luvit can seamlessly run bundled code and unbundled code without any changes or difficult changes. But we should also certainly embrace the existing lua ecosystem more, starting with configuring luvit to actually pick up on luarocks packages by default to some extent (currently Luvit is compiled with the default LUA_PATH and LUA_CPATH from both luajit and PUC Lua, neither of which match a sane system configuration).

Also, luvit's require (ala import) needs to be decoupled from lua's require. This would make it possible to a) distinguish between relative file imports and luvit's dependency system vs system dependencies and b) make it possible to request a system dependency or luvit dependency separate from eachother.

Do we stay this direction or do we want something else? Do we do both and fork the project?

At the moment luvit fits in the niche of being a well-integrated batteries-included lua environment where all someone has to do is install luvit and start programming (and slowly pull in external dependencies). This is kinda in both directions and straying off that path will likely put luvit in a tiny niche where it can't or won't see much use.

[creationix]

luvit's require (ala import) needs to be decoupled from lua's require

I'm not against naming it import and not touching the native require at all.

[creationix]

I tried this for my nibs project and it's working great!

local import = _G.import or require

local Nibs = import 'nibs'
local Tibs = import 'tibs'
...

I still use require for things like luajit builtins that don't need the luvit module system. This has allowed me to use nibs as-is in a non luvit project with this tiny helper!

local global_base = "/luvit"

local function dirname(path)
    return (string.match(path, "^(.*)/[^/]*$"))
end

local patterns = {
    "%s/libs/%s.lua",
    "%s/libs/%s/init.lua",
    "%s/deps/%s.lua",
    "%s/deps/%s/init.lua",
}

local function find(base, name)
    repeat
        for i = 1, 4 do
            local path = string.format(patterns[i], base, name)
            local file = io.open(path, "r")
            if file then return path, file end
        end
        base = dirname(base)
    until not base
end

local cache = {}

function _G.import(name)
    local level = 2
    local caller
    -- Loop past any C functions to get to the real caller
    -- This avoids pcall(require, "path") getting "=C" as the source
    repeat
        caller = assert(debug.getinfo(level, "S").source)
        level = level + 1
    until caller ~= "=[C]"

    -- If the caller is not an absolute path, assume base
    if string.byte(caller, 1) == 47 then
        caller = dirname(caller)
    else
        caller = global_base
    end
    local path, file = find(caller, name)
    if path then
        local module = cache[path]
        if module then
            file:close()
            return module
        end
        local content = assert(file:read "*a")
        file:close()
        local fn = assert(loadstring(content, path))
        module = fn()
        cache[path] = module
        return module
    end
end

return _G.import

[creationix]

So the first issue I see with import for now is we need to update the lua language server to understand it's a require like function

[TechnoJo4]

I'm not an active member of the luvit community, I just lurk around here, but I feel the need to add emphasis on this:

At the moment luvit fits in the niche of being a well-integrated batteries-included lua environment where all someone has to do is install luvit and start programming (and slowly pull in external dependencies).

Of course there's more to luvit, but these two specific points, the ease of installation, both of luvit itself and of additional dependencies (using lit), are what really make luvit so great. It's easy. It's fast, especially compared to npm. And when you're done you can build your project to a single executable, whatever platform you're on. And it's Lua.

It is nice that some standard library is included, but currently (as my earlier comment), I install coro-http to actually start doing http(s) requests, coro-fs to use the filesystem, etc. In the end, the bare luv and such end up being much more use as included batteries (even if I'll never require them myself directly) in luvit than most of the "luvit" lua code. Of course, we can change the luvit libraries to solve that problem, but I want to focus on something else here: tooling already is mostly what I use luvit for. luv just ends up being a dependency to most most of the libraries I install, so it's convenient to have bundled. I don't think luvit/lit needs to have the rest of the tooling, the language server, linter, typechecker, or documentation generator, but at this point if it did become those things I would absolutely welcome it. Considering that, as you said, "needed tooling is years behind other popular languages for professional projects," it would certainly be a good value proposition for luvit, and one that both professional users and tiny script hackers can benefit from.

Realistically, the lack of documentation has long been one of luvit's great problems; I feel like finding or creating a good human- and machine-readable documentation/type information format and standardizing its use before starting work on luvit 3.0 would be much more beneficial than many of the other things being discussed here. Making language servers or documentation generators understand it can be done later, but it would already be of immense value during development. I've already wanted to work on a language server for a while, and have already written my own code to convert Discordia's type annotations into EmmyLua format (for use with LuaLS), so this isn't the kind of thing I'd be against doing myself if nobody else wants to, but the annotations have to exist first. Maybe I'm the only one to care this strongly about this though, so take my opinion with a grain of salt.

The custom require function is also an extremely important part of luvit (it's what makes lit work so well), so I'd be in agreement with keeping it, potentially under another name. I don't think a name change is necessary if we move to something like luvit-loader, but both ways are acceptable to me, so I'll leave it to others to decide.

I agree with truemedian on pretty much everything else.