It depends. In my experience generally GraalVM is slower to start, but after a few iterations it can be as fast or faster, and that the same results can be amplified if you use the JVM instead of AOT (i.e. it's even slower to start, but eventually it can be much faster.)
Of course this all depends on your specific code and uses cases.
Truffle/Graal is also able to do some insanely cool things related to optimizing across FFI boundries: if you have a Java program that uses the Truffle javascript engine for scripting, Truffle is able to do JIT optimization transparently across the FFI boundry so it has 0 overhead. IIRC they even have some special API to allow a Truffle->native C library to be exposed to the runtime in a way that allows for it to optimize away a lot of FFI overhead or inline the native function into the trace. They were advertising this "Polyglot VM" functionality a lot a few years ago, although now their marketing mostly focuses on the NativeImage part (which helps a lot with the slow startup you mention).
TruffleRuby even had the extremely big brain idea of running a Truffle interpreter for C for native extensions instead of actually compiling them to native code, just so that Truffle can optimize transparently across FFI boundaries. https://chrisseaton.com/truffleruby/cext/
I don't have anything to contribute to the Truffle discussion, but for those not familiar: Chris Seaton was an active participant on Hacker News, until his tragic death in late 2022. Wish he was still with us.
> TruffleRuby even had the extremely big brain idea of running a Truffle interpreter for C for native extensions […]
TruffleC was a research project and the first attempt of running C code on Truffle that I'm aware of. It directly interpreted C source code and while that works for small self-contained programs, you quickly run into a lot of problems as soon as you want to run larger real world programs. You need everything including the C library available as pure C code and you have to deal with the fact that a lot of C code uses some UB/IB. In addition, your C parser has to fully adhere to the C standard and once you want to support C++ too because a lot of code is written in C++, you have to re-start from scratch. I don't know if TruffleC was ever released as open source.
The next / current attempt is Sulong which uses LLVM to compile C/C++/Rust/… to LLVM IR ("bitcode") and then directly interprets that bitcode. It's a lot better, because you don't have to write your own complete C/C++/… parser/compiler, but bitcode still has various limitations. Essentially as soon as the program uses handwritten assembler code somewhere, or if it does some low level things like setjmp/longjmp, things get hairy pretty quickly. Bitcode itself is also platform dependent (think of constants/macros/… that get expanded during compilation), you still need all code / libraries in bitcode, every language uses a just so slightly different set of IR nodes and requires a different runtime library so you have to explicitly support them, and even then you can't make it fully memory safe because typical programs will just break. In addition, the optimization level you choose when compiling the source program can result in very different bitcode with very different IR nodes, some of which were not supported for a long time (e.g., everything related to vectorization). Sulong can load libraries and expose them via the Truffle FFI, and it can be used for C extensions in GraalPython and TruffleRuby AFAIK. It's open source [1] and part of GraalVM, so you can play around with it.
Another research project was then to directly interpret AMD64 machine code and emulate a Linux userspace environment, because that would solve all the problems with inline assembly and language compatibility. Although that works, it has an entirely different set of problems: Graal/Truffle is simply not made for this type of code and as a result the performance is significantly worse than Sulong. You also end up re-implementing the Linux syscall interface in your interpreter, you have to deal with all the low level memory features that are available on Linux like mmap/mprotect/... and they have to behave exactly as on a real Linux system, and you can't easily export subroutines via Truffle FFI in a way that they also work with foreign language objects. It does work with various guest languages like C/C++/Rust/Go/… without modifying the interpreter, as long as the program is available as native Linux/AMD64 executable and doesn't use any of the unimplemented features. This project is also available as open source [2], but its focus somewhat shifted to using the interpreter for execution trace based program analysis.
Things that aren't supported by any of these projects AFAIK are full support for multithreading and multiprocessing, full support for IPC, and so on. Sulong partially solves it by calling into the native C library loaded in the VM for subroutines that aren't available as bitcode and aborting on certain unsupported calls like fork/clone, but then you obviously lose the advantage of having everything in the interpreter.
The conclusion is, whatever you try to interpret C/C++/… code, get ready for a world of pain and incompatibilities if you intend to run real world programs.
> generally GraalVM is slower to start, but after a few iterations it can be as fast or faster
That's even true of the standard HotSpot-backed JVM. I've rewritten programs from Java to JS which has resulted in making them faster—because they're short-lived, and the JVM's slow startup chewing through the budget is never offset by any of the theoretical speedups that the JVM otherwise promises.
I’m assuming that’s a benchmark that runs a lot of hot code? I wonder why the start up is slower, if it’s based off an interpreter, and all current VMs start that way too
• GraalJS is today based on an AST interpreter, which are less efficient in general than bytecode interpreters. V8 starts by compiling JavaScript to an internal bytecode, interpreting that, then JIT compiling the hot spots. It's the same architecture as the JVM except that the bytecode isn't considered to be a stable or documented format.
• The Truffle JIT compiler is slower than V8's because partial evaluation adds overhead. Slower compiler = more time spent in the interpreter = slower warmup.
• V8 is heavily optimized for great startup time because web pages typically don't live very long.
The first problem is being tackled by adding a bytecode interpreter infrastructure to Truffle itself. In other words, the Truffle library will invent a bytecode format for your language, write the bytecode interpreter for it, then partially evaluate that to create the JIT compiler! It can also handle stuff like persisting the bytecode to disk, like .pyc files or .class files do. Moving all the stuff needed to implement fast languages into the framework is very much the Truffle way.
The second problem is harder to solve. There is supposedly a thing called (I think) the second Futamura projection, where you partially evaluate the partial evaluator, but IIRC that's very hard to actually implement and for server-side use cases, less important.
GraalVM’s interpreter, AST, etc is also written in Java, so there is a warm-up for these as well. Also, the memory representation has to be somewhat generic, and can’t be over-optimized for Javascript specifically. Add to it that JS has multiple tiers of interpreters, JIT compilers (I believe there are 3 at the moment?), all made specifically for JS, so it’s not really possible to compete with that here, while remaining general.
It shows that it's /competitive/ despite having just written an interpreter (which is seriously impressive). Further optimizations are possible, as evidenced by newer benchmarks and further improvements to Chrome. But that required a lot more time and effort and funding by Google.
Oracle's profit motive is (AFAIK) enabling polyglot scripting inside their database. But Google has a profit motive of not having to pay Firefox or Safari a bigger chunk of the search engine ad profits.
So yes, I agree that the outdated benchmarks are a bit fishy. But if there was enough of a market I have no doubt that the Truffle/Graal devs would know how to put more funding to good use. There are alternative JVM runtimes that optimize for latency or start-up times. The Oracle JVM is optimized for their core market (long running server processes).
Memory safety does come with a price. It was ever thus.
It's competitive in some respects. It's not used in browsers so it's probably not competitive there where there are different code patterns to server-side JS (unless you want a security upgrade more than top performance), due indeed to the slower warmup time which really hurts in the web environment where code gets discarded regularly.
Also, V8 probably has better memory usage. I haven't checked.
The warmup time is being worked on but these are research problems, and the V8 team is much larger than the GraalJS team is.
> Memory safety does come with a price. It was ever thus.
The memory safety claim is about as fishy as the benchmarks.
I'd trust the JIT hardening of a modern JS engine over whatever hardening has happened in the OpenJDK any day of the week. In other words: if the JS engine was built on top of Truffle, then the exploit technique would be all about finding bugs in Truffle/Graal/OpenJDK. And I bet that's easier than finding bugs in V8 or any other JS engine, just because the JS engines have been fighting the security fire with extreme prejudice due to their security exposure and I doubt that the OpenJDK crew has had to since it's not as worth it to attack them. And if it was as worth it to attack them, then the complexity of the OpenJDK would make it an easier target to attack and a harder target to meaningfully lock down.
