J2CL Now & Next

[gkdn]

By popular request, I am listing the things that J2CL team is actively working on apart from continuous improvements and maintenance and what to expect in the next quarters:

Last Edit: Jul 26, 2023

Now

• Modular compilation for Wasm
• Support Kotlin language

Next

• Implement TestSuite support for Open Source
• Reduce limitations on @JsEnums
• Finalize supported Enum/EnumSet APIs
• Add periodic auto-tagging for Open Source for versioned releases

Later

• Enable RTA pruning for Open Source development
• Investigate embedding GwtIncompatibleStripper step into J2CL
• Add ability to run JsInteropRestrictionChecker as an ErrorProne plugin
• Optimize Dagger to be more efficient with J2CL
• Switch default fronted from JDT to Javac
• Remove boxing from long (similar to double/Double)
• Support for Java 17

Completed in previous quarters

• [x ] Improve Wasm GC spec, tooling and J2CL experimental Wasm backend
• Implement testing support for Open Source
• Java 11 support (completed, waiting Bazel release)
• First-class support for JSR nullability
• Optimize AutoValue to be more efficient with J2CL
• Improve code generation Java enums to reduce code size
• Add experimental Wasm backend
• Switch to Bazel TreeArtifacts
• Overhaul bridge calculate&creation to fix related bugs and reduce code size
• Fix overflow for integer arithmetic
• Improve JsCompiler optimizations
• Switch to Bazelisk by default

Note:

The J2CL code is production ready and battle tested with Google Apps for years. Only open-source tooling is missing some workflows. All of our improvements to the released code are in sync with Google and you are getting the improvement at the same time as Google gets.

At the moment for J2CL, we are only planning to do dated releases (instead of semantic versioning like v1) and make more tools work in open source over time.

[rov63rus]

Are you planning gradle support ? Bazel is not known to any IDE.

[tbroyer]

https://docs.bazel.build/versions/master/ide.html ?
(my main problem with the IntelliJ IDEA plugin though is that it's currently not compatible with latest IDEA 2020.2: bazelbuild/intellij#1998)

[sgammon]

i would love to see some gRPC support on this list :)

[treblereel]

Are there any workarounds how to run tests, before j2cl_test ll be ported ? It's a critical feature

[rjahn]

any updates here?

[gkdn]

I update it every now and then but comment date doesn't reflect that (you need to click edited to see the change dates).

[LifeIsStrange]

Any update on Kotlin support? That would be amazing and java projects are increasingly hybridized with Kotlin

[gkdn]

We just started Kotlin work. Our goal is to have something usable by end of the year.

[LifeIsStrange]

@gkdn that is amazing to hear but Kotlin JS is officially developed and has an excellent and mature support for transpiling Kotlin to JS (and generating typescript types)
However while kotlin multiplarform/js is great, and while j2cl allow to transpile Java to js, I feel like the main area for improvement would be for J2CL to leverage existing Kotlin js transpiling ability and to be interoperable with the generated js from Java.
If you could work on making it easier to have hybrid Kotlin-Java projects that target js that would be amazing, maybe that the biggest issue is about IDE supports at the boundaries (Kotlin js will not allow calling java code even if this java code transpile to J2CL). Improving this IDE interop between Java and Kotlin JS should be possible in theory, either by contributing directly to the Kotlin js compiler or by making a Kotlin compiler plugin (note that the replacement API for the new Kotlin compiler (K2) is in active development https://youtrack.jetbrains.com/issue/KT-49508
)
Also you might consider using arrow-meta which is a library that helps building compiler plugins.
https://github.com/arrow-kt/arrow-meta
BTW since you work at Google, maybe you could contact one of the official Google kotlin devs, as you can see here: https://kotlinlang.org/docs/kotlin-foundation.html#current-personnel
for some guidance.
The best would be to contact Roman Elizarov or to open a youtrack ticket for a collaboration :)
I really hope there is the possibility of making a synergy with JetBrains!
Anyway just my two cents on how to achieve the best of both worlds.

BTW: I concur that Gradle support would be a step in the right direction for adoption.

[gkdn]

Our high demanding customers relies on very Closure style centric Google JavaScript stack (which J2CL is optimized for) and they count every single byte for performance. Unfortunately Kotlin/JS output doesn't match our needs and sometimes in a way that might be contradicting with the Kotlin open-source users needs who rely on different JavaScript stacks.

So for now our main focus we will be enabling Kotlin to those customers. Being said that I already met with Roman on this and we are in contact with JetBreains. We will be still looking into bringing the two worlds together in the long run.

[LifeIsStrange]

@gkdn I suppose you could apply a post processing step to the generated kotlin-js by feeding it to Closure. You could also work on improving the efficiency of the IR backend.
But sure feel free to implement your own incompatible transpiler.

[treblereel]

@gkdn just in case, do you have any updates about testing support for Open Source ?

[gkdn]

We have an intern working on it at the moment but we will see how it will turn out.

[treblereel]

We have an intern working on it at the moment but we will see how it will turn out.

ok, and i suppose you have a deadline right ?

[gkdn]

Should be within 2 months; if the project is successful.

[treblereel]

Should be within 2 months; if the project is successful.

It looks like it failed ... So pity :(

[gkdn]

Actually basic unit testing support is released, sample app and getting started code is updated accordingly.

I haven't closed the task since it still doesn't support test suites but I will update the notes.

Edit: updated

[treblereel]

@gkdn Do you plan to add support for the following methods from Java 9-11?
Stream: takeWhile, dropWhile, ofNullable, and iterate
String: isBlank, lines, repeat, strip, stripLeading, and stripTrailing
List/Set/Map: copyOf
Collectors: toUnmodifiable*

And if not, can we make a PR?

[rluble]

We do accept contributions for missing jre methods.

Note that the implementations need to :

• be supportable in the web (e.g. java.net.*, reflective apis, etc, are no supported, ).
• not impose onerous codesize increases (e.g. most functionality of Locale is not supported due to this).
• be created independently of existing implementations in other sdks.

Isn't String.repeat is already in the JRE?

[niloc132]

String.repeat is indeed already implemented - but can you direct us how to run its tests? (I assume tests are required for such submissions, even though it wasn't on your list ;). )

I see that jre/javatests/com/google/j2cl/jre/java/lang/StringTest.java has a testRepeat method, but there don't seem to be any bazel targets to run individual test classes. #93 (comment) says (and seems to still be correct) that suites cannot run, and only jre/javatests/com/google/j2cl/jre/LangSuite.java references StringTest.

Here's the other things I tried, just in case I missed the obvious:
I noticed that LangSuite is referenced by the Lang j2cl_multi_test in jre/javatests/BUILD`, but running that results in

$ bazel test //jre/javatests:Lang                                          
ERROR: Skipping '//jre/javatests:Lang': error loading package 'jre/javatests': Label '//javascript/tools/jscompiler/builddefs:flags.bzl' is invalid because 'javascript/tools/jscompiler/builddefs' is not a package; perhaps you meant to put the colon here: '//:javascript/tools/jscompiler/builddefs/flags.bzl'?
ERROR: error loading package 'jre/javatests': Label '//javascript/tools/jscompiler/builddefs:flags.bzl' is invalid because 'javascript/tools/jscompiler/builddefs' is not a package; perhaps you meant to put the colon here: '//:javascript/tools/jscompiler/builddefs/flags.bzl'?
INFO: Elapsed time: 0.207s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
FAILED: Build did NOT complete successfully (0 packages loaded)
    currently loading: jre/javatests

Typically when we see //javascript labels, or references at all to jscompiler, it means that this isn't available to be run outside of Google. Removing that import line

load("//javascript/tools/jscompiler/builddefs:flags.bzl", "VERBOSE_WARNING_FLAGS_STRICT", "WHITESPACE_ONLY_FLAGS")

and all of the conformance test wiring that depends on it at least advances us to a new error:

$ bazel test //jre/javatests:Lang
ERROR: Skipping '//jre/javatests:Lang': error loading package 'jre/javatests': Label '//third_party/bazel_rules/rules_closure/closure:defs.bzl' is invalid because 'third_party/bazel_rules/rules_closure/closure' is not a package; perhaps you meant to put the colon here: '//third_party:bazel_rules/rules_closure/closure/defs.bzl'?
ERROR: error loading package 'jre/javatests': Label '//third_party/bazel_rules/rules_closure/closure:defs.bzl' is invalid because 'third_party/bazel_rules/rules_closure/closure' is not a package; perhaps you meant to put the colon here: '//third_party:bazel_rules/rules_closure/closure/defs.bzl'?
INFO: Elapsed time: 0.068s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
FAILED: Build did NOT complete successfully (0 packages loaded)
    currently loading: jre/javatests

Fortunately this is also not terrible to fix: there are two separate load() calls for closure_js_binary - removing the second one still leaves the first (though to just run Lang, I've commented out both of them).

Next, I was getting this error:

$ bazel test //jre/javatests:Lang
ERROR: Traceback (most recent call last):
        File "/home/colin/workspace/j2cl/jre/javatests/BUILD", line 85, column 16, in <toplevel>
                j2cl_multi_test(
        File "/home/colin/workspace/j2cl/jre/javatests/j2cl_multi_test.bzl", line 50, column 25, in j2cl_multi_test
                j2kt_native_test(
        File "/home/colin/workspace/j2cl/build_defs/internal_do_not_use/j2kt_test.bzl", line 16, column 21, in j2kt_native_test
                j2cl_test_common(
        File "/home/colin/workspace/j2cl/build_defs/internal_do_not_use/j2cl_test_common.bzl", line 274, column 13, in j2cl_test_common
                fail("Unknown platform: " + platform)
Error in fail: Unknown platform: J2KT-NATIVE

It looks like j2cl_multi_test itself isn't able to run in open source due to J2KT-NATIVE platform not being populated? The code only has branches for CLOSURE and WASM. Next I took the drastic step of removing all j2cl_multi_test (getting past the point of what I could comfortably suggest might even make sense to commit...), and instead wrote a custom test target:

j2cl_test(
    name = "Lang",
    test_class = "com.google.j2cl.jre.LangSuite",
    runtime_deps = [":emul_tests_lib"],
)

This hit compile errors, oddly - it seems that emul_tests_lib itself isn't set up properly? How can this run inside of Google at all? Or do you have entirely separate BUILD files (not just BUILD vs BUILD.bazel), and the checked in BUILD is never used?
To fix, I changed my approach, not trying to run tests at all any more, but just get them to build, via bazel build //jre/javatests:emul_tests_lib.
First I had to add //third_party:jspecify_annotations-j2cl to the deps list. Next, it seems that the list of disabled errorprone checks in emul_tests_lib is inconsistent with currently checked-in code - TreeMapTest.ConflictingKey either needs a SuppressWarnings on it, or a flag passed to errorprone - I opted for -Xep:ComparableType:OFF, as that seems to be the convention.

At this point, emul_tests_lib can succeed in building.

Next, I tried to build (not test) the modified Lang rule above - this failed with a confusing message:

ERROR: /home/colin/workspace/j2cl/jre/javatests/BUILD:129:10: Executing genrule //jre/javatests:genLang_test.js failed: (Exit 1): bash failed: error executing command /bin/bash -c ... (remaining 1 argument skipped)

        FAIL: j2cl_test currently supports testing with a single testsuite only.
        IF YOU HAVE MULTIPLE TESTSUITES, WE DO NOT KNOW IF ALL OF YOUR TESTS PASS OR NOT!
    

Is it accurate to say that suites don't work, but a single suite would? Regardless, I changed my custom rule to only run StringTest:

j2cl_test(
    name = "StringTest",
    test_class = "com.google.j2cl.jre.java.lang.StringTest",
    runtime_deps = [":emul_tests_lib"],
)

This passed a bazel build finally, but tests fail (specifically testIndexOfNull).

-> 02:03:46.792 : Running test: testIndexOfNull
-> $module$exports$junit$framework$Assert$impl$m_fail__java_lang_String__void$$@http://localhost:50362/filez/com_google_j2cl/jre/javatests/StringTest_bin.js:1832:421
-> $testIndexOfNull$@http://localhost:50362/filez/com_google_j2cl/jre/javatests/StringTest_bin.js:4879:179
[native code]
$invokeFunction_$@http://localhost:50362/filez/com_google_j2cl/jre/javatests/StringTest_bin.js:3869:42
$safeRunTest_$@http://localhost:50362/filez/com_google_j2cl/jre/javatests/StringTest_bin.js:3855:32
[native code]
$goog$testing$TestCase$Continuation_$run$$@http://localhost:50362/filez/com_google_j2cl/jre/javatests/StringTest_bin.js:4226:99
http://localhost:50362/filez/com_google_j2cl/jre/javatests/StringTest_bin.js:3780:167
[native code]
$goog$testing$TestCase$Continuation_$run$$@http://localhost:50362/filez/com_google_j2cl/jre/javatests/StringTest_bin.js:4226:99
http://localhost:50362/filez/com_google_j2cl/jre/javatests/StringTest_bin.js:3778:47
$goog$Promise$$@http://localhost:50362/filez/com_google_j2cl/jre/javatests/StringTest_bin.js:2631:32
$JSCompiler_StaticMethods_runTestsReturningPromise$$@http://localhost:50362/filez/com_google_j2cl/jre/javatests/StringTest_bin.js:3777:22
$JSCompiler_StaticMethods_JSC$2497_execute$$@http://localhost:50362/filez/com_google_j2cl/jre/javatests/StringTest_bin.js:5582:159
http://localhost:50362/filez/com_google_j2cl/jre/javatests/StringTest_bin.js:5714:53
-> 02:03:46.793 : testIndexOfNull : FAILED

Is there an easier way to go about running tests, so that we can migrate already-working java9-11 stream/optional/collection code into j2cl, complete with tests?

Would any/all of this be suitable for a contribution, so that a CI script could exist that runs these emulation tests, in anticipation of emulation patches?

[gkdn]

The JRE unit tests are not ported to open-source so they would not work out of the box.

We are currently swamped with couple of a big projects but if you like to commit to the JRE emulation, the best I can offer at the moment is:

• Patch gkdn@9c7e780
• Update StringTest in jre/javatests/BUILD with the class that you are testing.
• bazel test jre/javatests:StringTest to run the test
• bazel run jre/javatests:StringTest_debug and follow the link to run/debug on your browser.

If you have further questions, let's start a discussion in the discussions section.

[treblereel]

@gkdn Thanks a million; it works like a charm.

[niloc132]

Great, thank you @gkdn - different changes than I had made, but clearly solving the same issues. I appreciate the pointer to the _debug test target. @treblereel has submitted #222 with changes from GWT, verified in j2cl by running tests as you described.