Skip to content

kostya/benchmarks

Repository files navigation

Table of Content

Overview

The benchmarks follow the criteria:

  • They are written as the average software developer would write them, i.e.

    • The algorithms are implemented as cited in public sources;
    • The libraries are used as described in the tutorials, documentation and examples;
    • The used data structures are idiomatic.
  • The used algorithms are similar between the languages (as the reference implementations), variants are acceptable if the reference implementation exists.

  • All final binaries are releases (optimized for performance if possible) as debug performance may vary too much depending on the compiler.

My other benchmarks: jit-benchmarks, crystal-benchmarks-game

Measurements

The measured values are:

  • time spent for the benchmark execution (loading required data and code self-testing are not measured);
  • memory consumption of the benchmark process, reported as base + increase, where base is the RSS before the benchmark and increase is the peak increase of the RSS during the benchmark;
  • energy consumption of the CPU package during the benchmark: PP0 (cores) + PP1 (uncores like GPU) + DRAM. Currently, only Intel CPU are supported via the powercap interface.

All values are presented as: median±median absolute deviation.

UPDATE: 2024-06-06

Test Cases

Brainfuck

Testing brainfuck implementations using two code samples (bench.b and mandel.b). Supports two mode:

  • Verbose (default). Prints the output immediately.
  • Quiet (if QUIET environment variable is set). Accumulates the output using Fletcher-16 checksum, and prints it out after the benchmark.

Brainfuck

bench.b

Language Time, s Memory, MiB Energy, J
Scala (Staged) 0.390±0.015 222.77±03.17 + 15.42±01.67 24.89±00.74
Racket (Staged) 0.893±0.001 101.38±00.97 + 0.00±00.00 34.61±00.07
Rust 1.011±0.000 1.05±00.01 + 0.00±00.00 43.04±00.28
V/gcc 1.060±0.000 1.92±00.03 + 0.00±00.00 43.67±00.15
C/gcc 1.111±0.001 1.01±00.01 + 0.00±00.00 47.01±00.65
C++/g++ 1.113±0.001 1.98±00.04 + 0.00±00.00 46.04±00.34
D/gdc 1.120±0.000 6.39±00.05 + 0.00±00.00 49.26±00.57
C++/clang++ 1.125±0.001 1.74±00.02 + 0.00±00.00 46.90±00.51
C/clang 1.140±0.001 1.02±00.01 + 0.00±00.00 48.57±00.37
D/ldc2 1.163±0.003 2.32±00.80 + 0.00±00.00 49.08±00.30
Nim/gcc 1.168±0.001 1.01±00.02 + 0.00±00.00 49.66±00.25
Java 1.193±0.000 39.79±00.09 + 1.55±00.06 49.18±00.26
Vala/gcc 1.210±0.001 4.50±00.02 + 0.00±00.00 50.69±00.11
Kotlin/JVM 1.222±0.002 43.53±00.14 + 0.52±00.26 50.93±00.10
Vala/clang 1.238±0.006 4.52±00.03 + 0.00±00.00 52.25±01.41
Go 1.247±0.000 3.55±00.03 + 0.00±00.00 51.47±00.33
Zig 1.265±0.000 1.03±00.02 + 0.00±00.00 52.54±00.44
C#/.NET Core 1.367±0.001 32.66±00.13 + 0.15±00.02 58.23±00.27
Go/gccgo 1.481±0.002 23.56±00.01 + 0.00±00.00 62.86±00.67
Nim/clang 1.565±0.000 1.29±00.01 + 0.00±00.00 64.41±00.46
F#/.NET Core 1.574±0.004 37.41±00.18 + 0.30±00.00 67.12±00.40
Crystal 1.597±0.008 3.00±00.05 + 0.00±00.00 67.57±00.71
OCaml 1.638±0.005 3.01±00.05 + 2.81±00.06 76.01±00.28
Julia 1.638±0.003 256.65±00.11 + 0.40±00.03 69.57±00.34
Chez Scheme 1.714±0.004 24.77±00.02 + 4.46±00.12 72.48±00.24
Racket 1.756±0.028 113.87±00.09 + 0.00±00.00 71.30±01.18
V/clang 1.973±0.018 1.96±00.03 + 0.00±00.00 85.72±01.29
C#/Mono 2.054±0.011 25.60±00.08 + 0.00±00.00 87.72±01.03
MLton 2.088±0.027 1.77±00.04 + 0.25±00.00 86.67±01.18
Scala 2.779±0.003 72.03±00.11 + 249.04±00.15 121.64±00.71
Node.js 2.920±0.028 44.53±00.02 + 4.37±00.00 123.76±01.03
Haskell (MArray) 3.095±0.007 3.17±00.04 + 4.92±00.04 130.42±01.60
D/dmd 3.326±0.001 3.39±00.03 + 0.00±00.00 124.72±00.18
Haskell (FP) 3.624±0.002 3.27±00.05 + 4.89±00.04 154.06±00.80
Ruby/truffleruby (JVM) 4.996±0.396 374.17±10.09 + 496.92±17.09 243.84±22.97
Ruby/truffleruby 5.193±0.358 203.03±01.33 + 550.75±59.29 253.87±16.16
Swift 5.656±0.005 16.69±00.04 + 0.00±00.00 211.92±00.58
Lua/luajit 5.881±0.025 2.54±00.02 + 0.00±00.00 241.87±05.05
Python/pypy 9.544±0.044 58.79±00.12 + 29.96±00.01 422.77±04.07
Idris 15.129±0.051 20.71±00.03 + 8.83±00.04 661.53±10.66
Elixir 20.395±0.043 70.50±00.79 + 0.00±00.00 807.25±05.02
Ruby (--jit) 31.817±0.055 21.59±00.04 + 4.76±00.01 1311.22±06.23
PHP 34.492±0.160 17.78±00.18 + 0.00±00.00 1476.88±31.23
Lua 36.833±0.101 2.28±00.03 + 0.00±00.00 1512.87±19.03
Ruby 68.430±0.445 11.43±00.04 + 0.00±00.00 3008.91±26.07
Python 69.075±0.780 11.17±00.08 + 0.00±00.00 3039.83±50.68
Ruby/jruby 81.684±1.546 203.50±04.69 + 220.84±20.47 3624.22±70.81
Tcl (FP) 192.447±0.637 4.06±00.02 + 0.00±00.00 8478.25±83.42
Perl 225.331±1.241 6.90±00.05 + 0.00±00.00 10000.48±94.38
Tcl (OOP) 384.789±1.841 4.06±00.01 + 0.00±00.00 17091.07±181.64

mandel.b

Mandel in Brainfuck

Language Time, s Memory, MiB Energy, J
Scala (Staged) 7.639±0.120 224.42±01.90 + 102.36±04.76 459.89±14.99
C++/g++ 9.755±0.030 1.97±00.03 + 2.29±00.11 393.99±02.68
C#/.NET Core 12.048±0.028 32.72±00.06 + 1.27±00.00 482.24±01.22
Java 12.450±0.060 39.92±00.06 + 2.44±00.04 489.83±01.92
C/gcc 12.513±0.013 0.99±00.02 + 0.84±00.06 505.46±02.16
Kotlin/JVM 13.082±0.033 43.53±00.17 + 2.30±00.33 547.94±03.79
F#/.NET Core 13.115±0.017 37.55±00.07 + 2.12±00.04 526.18±01.55
C/clang 13.293±0.031 0.97±00.03 + 0.75±00.03 572.78±05.53
V/gcc 13.591±0.007 1.91±00.03 + 1.18±00.04 547.49±03.65
C++/clang++ 13.898±0.024 1.72±00.05 + 1.88±00.03 569.56±02.36
Racket (Staged) 13.986±0.059 99.66±00.50 + 77.13±01.67 556.40±01.66
Crystal 14.103±0.018 2.96±00.03 + 0.74±00.04 598.53±02.62
Go 14.182±0.009 3.56±00.01 + 0.00±00.00 562.34±00.49
Rust 14.229±0.012 1.04±00.01 + 1.20±00.04 564.72±02.02
D/gdc 14.266±0.016 6.37±00.06 + 1.49±00.04 601.20±03.61
D/ldc2 14.278±0.013 3.05±00.02 + 0.83±00.02 572.27±01.12
Vala/gcc 14.469±0.022 4.47±00.04 + 1.24±00.04 574.39±01.98
Zig 14.481±0.014 1.02±00.02 + 1.42±00.06 600.92±02.27
Vala/clang 14.845±0.010 4.46±00.02 + 1.22±00.03 607.68±02.95
Nim/gcc 15.641±0.030 2.07±00.06 + 1.29±00.00 663.67±00.73
Scala 16.365±0.031 71.96±00.25 + 139.77±00.37 735.09±03.15
Swift 18.421±0.037 16.45±00.04 + 0.00±00.00 762.02±06.09
Go/gccgo 19.091±0.406 23.58±00.05 + 0.00±00.00 796.35±13.26
Nim/clang 19.696±0.204 2.38±00.01 + 1.29±00.00 806.37±09.58
V/clang 20.895±0.225 1.94±00.05 + 1.17±00.00 901.27±12.94
OCaml 24.390±0.016 4.53±00.08 + 2.82±00.00 1171.79±06.03
Julia 26.058±0.070 256.43±00.04 + 0.30±00.00 1021.32±06.88
Chez Scheme 27.698±0.056 25.55±00.06 + 3.68±00.01 1210.07±04.32
C#/Mono 31.025±0.011 25.61±00.09 + 0.83±00.00 1289.92±08.22
MLton 33.842±0.072 1.73±00.05 + 4.11±00.00 1537.44±17.63
Haskell (MArray) 34.546±0.022 4.08±00.02 + 5.11±00.00 1400.92±05.61
Node.js 34.598±0.063 44.44±00.07 + 5.25±00.00 1378.99±02.63
Lua/luajit 34.656±0.034 2.56±00.04 + 0.38±00.00 1388.13±02.21
Racket 35.183±0.294 113.90±00.09 + 1.45±00.65 1567.04±13.20
D/dmd 37.942±0.003 3.35±00.03 + 0.87±00.02 1378.02±00.92
Python/pypy 41.463±0.088 58.86±00.09 + 30.60±00.03 1837.98±07.63
Ruby/truffleruby 48.320±0.851 203.25±00.90 + 581.44±30.40 2360.50±62.00
Ruby/truffleruby (JVM) 50.546±1.177 374.73±03.49 + 489.39±89.50 2202.33±46.95
Idris 66.241±0.299 22.03±00.05 + 9.54±00.00 2848.93±05.21
Haskell (FP) 77.093±0.126 4.19±00.02 + 75.74±00.03 3195.67±11.41

Base64

Testing base64 encoding/decoding of the large blob into the newly allocated buffers.

Base64

Language Time, s Memory, MiB Energy, J
C/gcc (aklomp) 0.098±0.000 2.20±00.03 + 0.00±00.00 4.60±00.03
C/clang (aklomp) 0.099±0.000 2.10±00.06 + 0.00±00.00 4.68±00.05
PHP 0.105±0.000 18.56±00.06 + 0.00±00.00 4.76±00.07
Go (base64x) 0.267±0.001 6.61±00.02 + 0.00±00.00 12.80±00.11
Zig 0.698±0.000 1.66±00.04 + 0.00±00.00 26.49±00.19
Rust 0.849±0.000 2.42±00.02 + 0.00±00.00 35.03±00.11
Node.js 0.913±0.001 43.17±00.03 + 40.43±00.21 40.38±00.37
C/clang 0.997±0.000 2.15±00.01 + 0.00±00.00 36.62±00.05
C/gcc 1.101±0.016 2.10±00.05 + 0.00±00.00 40.24±00.60
Nim/clang 1.103±0.001 2.06±00.02 + 5.79±00.03 44.95±00.27
Crystal 1.107±0.003 3.61±00.04 + 1.26±00.03 44.93±00.29
D/ldc2 1.175±0.003 3.68±00.03 + 3.41±00.00 48.49±00.44
Nim/gcc 1.364±0.002 1.75±00.04 + 4.96±00.01 55.70±00.32
Ruby (--jit) 1.427±0.002 15.10±00.16 + 73.37±00.56 55.92±00.37
Java 1.518±0.005 41.05±00.10 + 210.65±19.06 60.66±00.44
V/clang 1.534±0.001 2.43±00.01 + 2386.75±01.65 57.81±00.29
V/gcc 1.575±0.000 2.43±00.03 + 2385.17±00.74 57.34±00.19
Scala 1.604±0.001 70.48±00.15 + 276.28±02.12 66.81±00.54
Go 1.622±0.002 4.34±00.02 + 0.00±00.00 67.66±00.22
Ruby 1.642±0.005 11.69±00.04 + 42.50±00.30 64.29±00.22
Vala/clang 1.643±0.001 5.68±00.03 + 0.10±00.03 63.09±00.70
Vala/gcc 1.644±0.001 5.66±00.01 + 0.08±00.05 62.92±00.40
Kotlin/JVM 1.665±0.003 44.48±00.20 + 314.13±03.17 67.36±00.44
C++/g++ (libcrypto) 1.711±0.003 6.05±00.16 + 0.76±00.03 68.53±00.38
C++/clang++ (libcrypto) 1.713±0.002 5.51±00.04 + 0.73±00.02 69.23±00.37
Perl (MIME::Base64) 1.863±0.044 14.57±00.04 + 0.12±00.03 75.20±01.07
F#/.NET Core 2.038±0.018 38.32±00.07 + 11.92±00.65 80.42±00.77
C#/.NET Core 2.181±0.021 33.77±00.03 + 12.54±02.09 85.47±00.28
D/gdc 2.382±0.001 7.44±00.04 + 3.36±00.00 103.50±00.94
Go/gccgo 2.949±0.007 24.81±00.17 + 0.00±00.00 139.67±00.50
Julia 3.079±0.006 271.96±00.08 + 97.24±00.20 124.20±01.58
Python/pypy 3.236±0.002 58.92±00.06 + 31.49±00.17 141.88±00.91
D/dmd 3.387±0.004 3.17±00.02 + 3.84±00.02 141.65±00.88
Tcl 3.438±0.001 5.20±00.02 + 0.00±00.00 137.53±00.49
Python 3.512±0.001 10.78±00.16 + 0.07±00.07 133.94±00.44
Ruby/truffleruby (JVM) 3.758±0.016 374.30±05.93 + 247.06±24.79 189.38±03.07
Racket 3.885±0.006 95.36±00.20 + 21.80±00.40 154.27±00.56
C#/Mono 4.632±0.005 26.39±00.04 + 18.66±00.02 190.60±01.36
Ruby/jruby 6.005±0.008 195.03±02.11 + 165.97±10.93 263.57±01.49
Ruby/truffleruby 8.324±0.018 201.01±02.82 + 554.96±13.20 400.06±01.94
Perl (MIME::Base64::Perl) 10.148±0.055 15.93±00.11 + 0.26±00.04 448.06±03.65

Json

Testing parsing and simple calculating of values from a big JSON file.

Few notes:

Json

Language Time, s Memory, MiB Energy, J
C++/clang++ (simdjson On-Demand) 0.060±0.000 112.46±00.06 + 60.05±00.06 2.51±00.01
C++/g++ (simdjson On-Demand) 0.061±0.000 113.66±00.03 + 59.81±00.00 2.57±00.02
C++/clang++ (DAW JSON Link NoCheck) 0.083±0.000 112.54±00.05 + 0.00±00.00 3.37±00.03
C++/g++ (DAW JSON Link NoCheck) 0.087±0.000 113.29±00.06 + 0.00±00.00 3.53±00.04
C++/clang++ (DAW JSON Link) 0.093±0.000 112.47±00.03 + 0.00±00.00 3.90±00.07
C++/g++ (DAW JSON Link) 0.093±0.000 113.30±00.05 + 0.00±00.00 3.88±00.03
Rust (Serde Custom) 0.098±0.001 111.40±00.08 + 0.00±00.00 4.14±00.04
C++/clang++ (simdjson DOM) 0.102±0.000 112.55±00.09 + 177.09±00.06 4.62±00.02
C++/g++ (simdjson DOM) 0.108±0.000 113.54±00.06 + 172.73±00.13 4.90±00.02
Rust (Serde Typed) 0.111±0.000 111.71±00.05 + 11.05±00.07 4.62±00.04
D/ldc2 (Mir Asdf DOM) 0.131±0.000 112.93±00.04 + 61.22±00.00 5.43±00.05
C++/clang++ (gason) 0.139±0.000 112.57±00.02 + 96.97±00.06 5.67±00.04
C++/g++ (gason) 0.140±0.000 113.17±00.06 + 96.93±00.04 5.55±00.03
C++/g++ (RapidJSON) 0.152±0.000 113.21±00.04 + 128.90±00.03 6.47±00.04
Scala (jsoniter-scala) 0.157±0.002 290.75±00.24 + 20.43±00.24 8.36±00.07
Go (rjson custom) 0.196±0.000 113.58±00.05 + 0.00±00.00 7.57±00.02
C++/clang++ (RapidJSON) 0.200±0.000 112.47±00.04 + 128.91±00.09 8.55±00.08
Go (Sonic) 0.208±0.002 122.51±00.12 + 0.00±00.00 8.94±00.09
C++/g++ (RapidJSON Precise) 0.217±0.000 113.21±00.06 + 122.47±01.29 9.28±00.08
D/ldc2 (Mir Amazon's Ion DOM) 0.219±0.000 113.03±00.08 + 80.70±00.00 9.26±00.03
Go (rjson) 0.224±0.000 113.61±00.10 + 0.00±00.00 8.70±00.02
Zig 0.241±0.000 111.03±00.02 + 39.25±00.00 10.37±00.07
Go (goccy/go-json) 0.266±0.000 114.29±00.07 + 0.00±00.00 10.56±00.02
C++/clang++ (RapidJSON Precise) 0.285±0.000 112.48±00.03 + 128.91±00.03 12.50±00.10
C++/g++ (RapidJSON SAX) 0.346±0.001 113.11±00.08 + 0.00±00.00 15.32±00.06
C/clang (yajl) 0.350±0.001 110.97±00.04 + 0.00±00.00 15.14±00.06
C/gcc (yajl) 0.353±0.001 110.95±00.01 + 0.00±00.00 15.22±00.06
C++/g++ (Boost.JSON) 0.363±0.000 113.32±00.04 + 308.09±00.02 15.44±00.10
C++/clang++ (Boost.JSON) 0.371±0.001 112.68±00.03 + 308.09±00.03 15.74±00.10
Nim/clang (jsony) 0.392±0.000 111.56±00.05 + 146.12±00.09 16.46±00.12
C++/g++ (RapidJSON SAX Precise) 0.401±0.001 113.12±00.06 + 0.00±00.00 17.92±00.23
C++/clang++ (RapidJSON SAX) 0.408±0.001 194.77±00.07 + 0.00±00.00 17.40±00.07
Nim/gcc (jsony) 0.412±0.000 111.23±00.03 + 154.69±00.03 17.46±00.15
Node.js 0.470±0.004 152.69±00.04 + 195.96±00.68 22.05±00.12
C++/clang++ (RapidJSON SAX Precise) 0.489±0.001 194.79±00.07 + 0.00±00.00 21.84±00.23
Go (jsoniter) 0.513±0.000 114.27±00.06 + 0.00±00.00 20.75±00.17
Rust (Serde Untyped) 0.532±0.001 111.73±00.03 + 839.98±00.00 22.14±00.20
C#/.NET Core (System.Text.Json) 0.543±0.002 489.53±00.11 + 140.62±00.22 24.07±00.23
Java (DSL-JSON) 0.568±0.007 262.39±00.11 + 225.35±19.01 29.30±00.32
Python/pypy 0.608±0.002 279.98±00.08 + 125.71±00.04 26.46±00.26
V/gcc 0.615±0.001 111.47±00.04 + 496.21±00.00 25.70±00.23
V/clang 0.637±0.001 111.87±00.01 + 495.83±00.00 26.74±00.19
Nim/gcc (Packedjson) 0.638±0.003 111.93±00.02 + 294.16±00.00 27.28±00.40
Crystal (Pull) 0.638±0.002 113.26±00.02 + 18.44±00.00 28.10±00.40
Crystal (Schema) 0.656±0.001 113.23±00.03 + 48.86±00.13 28.63±00.07
Nim/clang (Packedjson) 0.662±0.001 112.23±00.01 + 294.16±00.00 28.31±00.17
Perl (Cpanel::JSON::XS) 0.765±0.003 125.17±00.05 + 402.77±00.00 31.84±00.14
Go 0.813±0.001 113.95±00.06 + 0.00±00.00 33.60±00.06
PHP 0.813±0.001 128.26±00.08 + 517.83±00.03 34.57±00.20
Crystal 0.924±0.004 113.21±00.02 + 392.50±00.03 39.68±00.33
Nim/gcc 1.037±0.002 111.91±00.03 + 1001.34±00.00 43.13±00.12
C#/.NET Core 1.056±0.004 495.03±00.30 + 272.99±00.16 50.49±00.23
Nim/clang 1.076±0.002 112.23±00.00 + 999.02±00.00 44.69±00.27
C++/g++ (json-c) 1.159±0.003 113.42±00.05 + 1216.05±00.01 47.99±00.30
C++/clang++ (json-c) 1.160±0.007 112.70±00.07 + 1216.04±00.01 48.35±00.66
Clojure 1.161±0.019 452.13±02.50 + 587.46±09.25 61.05±00.99
C++/clang++ (Nlohmann) 1.176±0.001 112.64±00.03 + 360.11±00.02 50.39±00.68
Ruby (--jit) 1.280±0.005 127.59±00.08 + 212.39±00.01 53.84±00.51
Go/gccgo 1.290±0.004 138.88±00.10 + 0.00±00.00 53.03±00.36
C++/g++ (Nlohmann) 1.327±0.003 113.27±00.06 + 448.05±00.01 56.39±00.35
Ruby 1.328±0.006 121.40±00.05 + 212.77±00.02 55.73±00.73
CPython (UltraJSON) 1.331±0.004 123.14±00.05 + 497.28±01.55 51.19±00.22
Python 1.374±0.002 120.99±00.03 + 325.95±00.01 55.23±00.37
F#/.NET Core (System.Text.Json) 1.496±0.003 498.18±00.08 + 232.75±04.63 68.50±00.69
Ruby (YAJL) 1.719±0.009 121.49±00.11 + 218.39±00.03 72.78±00.52
D/ldc2 1.723±0.003 112.86±00.06 + 708.76±00.04 72.08±00.65
C#/Mono 1.809±0.013 252.32±00.17 + 31.51±00.01 78.45±00.96
Haskell 1.982±0.007 115.63±00.14 + 723.73±00.48 84.53±00.77
Rust (jq) 2.529±0.003 113.39±00.06 + 902.98±01.54 106.08±00.37
C++/g++ (Boost.PropertyTree) 2.611±0.008 113.14±00.04 + 1440.09±00.04 111.72±00.36
C++/clang++ (Boost.PropertyTree) 2.671±0.005 195.01±00.04 + 1232.80±00.03 112.86±00.70
Ruby/jruby 2.884±0.022 464.98±06.74 + 972.45±49.49 148.64±02.99
D/dmd 3.073±0.003 113.16±00.04 + 708.78±00.07 131.07±00.23
Vala/clang 3.165±0.007 115.04±00.04 + 980.08±00.03 137.65±00.31
Vala/gcc 3.166±0.011 115.06±00.03 + 980.07±00.03 136.93±00.87
D/gdc 3.569±0.004 116.68±00.06 + 680.99±00.12 151.81±01.75
Racket 3.818±0.022 222.34±00.45 + 261.38±27.93 159.23±01.29
Perl (JSON::Tiny) 9.050±0.121 125.59±00.05 + 528.92±00.02 394.96±03.88
Ruby/truffleruby 10.718±0.048 522.13±27.57 + 1942.99±149.89 617.74±02.74
Ruby/truffleruby (JVM) 11.011±0.153 474.89±08.90 + 2201.50±87.97 693.97±13.20

Matmul

Testing allocating and multiplying matrices.

Matmul

Language Time, s Memory, MiB Energy, J
D/ldc2 (lubeck) 0.032±0.001 40.38±00.48 + 23.61±00.41 3.61±00.09
Nim/gcc (Arraymancer) 0.064±0.004 5.41±00.06 + 57.41±00.10 5.27±00.20
Python (NumPy) 0.065±0.000 33.37±00.06 + 57.87±00.04 5.54±00.01
Java (ND4J) 0.076±0.001 114.91±01.04 + 92.22±00.01 5.97±00.05
Rust (ndarray) 0.089±0.001 2.50±00.04 + 68.53±00.00 6.03±00.08
Julia (threads: 2) 0.092±0.000 300.38±00.45 + 47.53±00.53 5.54±00.03
Nim/clang (Arraymancer) 0.128±0.020 6.05±00.18 + 57.54±00.17 8.78±01.10
Julia (threads: 1) 0.142±0.000 299.90±00.18 + 47.76±00.23 6.91±00.04
C++/g++ (Eigen) 0.145±0.000 4.49±00.06 + 85.26±00.00 7.14±00.03
C++/clang++ (Eigen) 0.145±0.000 4.72±00.03 + 85.37±00.00 7.07±00.04
V/clang (VSL + CBLAS) 0.254±0.004 7.13±00.03 + 51.89±00.00 17.98±00.26
V/clang (VSL) 0.258±0.002 7.21±00.17 + 51.57±00.13 17.89±00.11
V/gcc (VSL + CBLAS) 0.459±0.002 7.25±00.06 + 51.57±00.00 34.63±00.06
V/gcc (VSL) 0.466±0.001 6.90±00.04 + 51.89±00.00 31.75±00.07
Julia (no BLAS) 1.146±0.023 271.40±00.23 + 52.30±00.01 49.02±00.55
D/ldc2 1.714±0.001 3.43±00.04 + 70.44±00.00 62.98±00.24
D/gdc 1.866±0.001 7.40±00.14 + 70.16±00.01 72.94±00.11
D/dmd 1.874±0.001 3.32±00.03 + 70.46±00.03 70.33±00.49
C/gcc 3.025±0.000 1.53±00.02 + 68.65±00.02 109.57±01.18
V/gcc 3.027±0.001 2.60±00.03 + 68.58±00.00 112.33±00.33
Vala/clang 3.057±0.000 5.52±00.03 + 68.32±00.00 104.60±00.31
V/clang 3.058±0.000 2.90±00.06 + 68.58±00.00 104.38±00.16
C/clang 3.060±0.000 1.55±00.02 + 68.63±00.04 104.48±00.15
Rust 3.061±0.000 2.14±00.07 + 68.69±00.00 105.19±00.52
Zig 3.063±0.001 1.92±00.01 + 68.58±00.00 108.31±00.18
Nim/gcc 3.086±0.001 2.60±00.01 + 58.91±01.16 114.12±00.58
Swift 3.090±0.000 7.99±00.06 + 68.69±00.00 109.94±00.59
Nim/clang 3.116±0.001 2.88±00.01 + 57.75±00.00 107.30±00.87
Vala/gcc 3.123±0.000 5.49±00.10 + 68.32±00.00 114.09±00.15
Go 3.145±0.000 4.09±00.01 + 0.00±00.00 112.66±01.59
Crystal 3.147±0.001 3.63±00.06 + 60.02±00.02 115.55±00.61
Go/gccgo 3.150±0.000 24.19±00.16 + 0.00±00.00 110.47±00.13
Java 3.165±0.002 40.98±00.16 + 74.30±00.20 117.37±01.36
Kotlin/JVM 3.209±0.003 42.32±00.21 + 73.03±00.22 129.95±00.33
Node.js 3.252±0.002 51.38±00.26 + 70.87±00.47 130.78±00.37
Python/pypy 3.254±0.001 59.96±00.07 + 68.90±00.03 135.03±00.17
Scala 3.332±0.003 71.86±00.08 + 153.79±00.24 120.95±00.68
C#/.NET Core 4.893±0.001 34.69±00.07 + 68.91±00.03 196.73±00.66
C#/Mono 7.391±0.000 26.02±00.07 + 69.48±00.01 297.35±00.51
Ruby/truffleruby 18.156±3.646 360.35±12.57 + 574.17±41.24 647.87±107.81
Ruby/truffleruby (JVM) 25.014±0.847 407.30±16.57 + 334.68±58.11 875.73±22.67
Perl 149.225±1.755 8.52±00.04 + 379.63±00.02 6447.37±59.99
Python 155.211±1.106 10.91±00.03 + 68.84±00.00 7060.84±46.09
Ruby (--jit) 156.368±0.039 21.96±00.04 + 68.35±00.02 6925.51±35.94
Ruby 195.516±0.462 11.75±00.07 + 69.19±00.01 8795.03±25.92
Tcl 202.975±0.793 7.47±00.02 + 400.44±00.00 9311.02±86.81
Ruby/jruby 361.315±19.249 273.70±12.97 + 1142.23±73.73 14778.53±653.23

Primes

Testing:

  • generating primes using the optimized sieve of Atkin;
  • prefix search for their decimal numbers using Trie data structure.

Notes:

  • All languages but V and Python use unordered hashmaps (V and Python don't provide those out of box, and their hashmaps use keys in the insertion order);
  • The results are always sorted (could be unstable or stable though).

Primes

Language Time, s Memory, MiB Energy, J
Zig 0.056±0.000 1.03±00.02 + 52.80±00.19 2.34±00.03
C++/clang++ 0.062±0.000 3.17±00.03 + 54.80±00.00 2.38±00.02
C++/g++ 0.063±0.000 3.67±00.07 + 70.55±00.25 2.44±00.02
Go 0.072±0.000 3.08±00.05 + 0.00±00.00 3.06±00.04
V/clang 0.095±0.000 2.14±00.23 + 211.15±00.64 3.91±00.04
V/gcc 0.099±0.000 1.95±00.03 + 201.60±00.17 4.10±00.03
Rust 0.101±0.000 2.09±00.11 + 74.71±00.77 3.98±00.05
Java 0.130±0.004 39.84±00.17 + 123.02±05.25 7.33±00.26
Crystal 0.138±0.000 3.71±00.02 + 88.43±00.00 5.58±00.05
Node.js 0.218±0.002 42.17±00.00 + 147.09±02.74 10.79±00.07
Scala 0.228±0.003 72.13±00.08 + 230.29±00.88 13.20±00.22
Nim/clang 0.278±0.000 2.10±00.10 + 587.68±01.03 10.75±00.04
Nim/gcc 0.287±0.000 1.90±00.02 + 615.91±00.00 10.83±00.07
Lua/luajit 0.298±0.001 1.33±00.02 + 156.66±00.63 12.00±00.08
Python/pypy 0.616±0.003 58.72±00.13 + 249.30±00.15 25.12±00.16
Julia 0.651±0.001 271.49±00.14 + 370.21±01.14 25.06±00.21
Racket 0.752±0.002 109.49±00.82 + 247.11±00.45 29.59±00.17
Ruby/truffleruby 0.890±0.012 203.57±01.55 + 800.19±11.11 60.49±00.76
Lua 1.142±0.006 2.69±00.04 + 283.50±01.03 46.78±00.35
Ruby (--jit) 1.187±0.002 22.98±00.02 + 163.71±00.14 47.95±00.43
Ruby/truffleruby (JVM) 1.378±0.088 376.75±09.21 + 495.42±42.25 89.49±05.72
Ruby 1.860±0.003 11.47±00.07 + 172.19±00.35 76.75±00.60
Ruby/jruby 2.314±0.075 197.13±08.38 + 521.12±52.24 124.24±05.36
Python 2.530±0.014 10.84±00.04 + 181.69±01.57 107.30±01.03

Tests Execution

Environment

CPU: Intel(R) Xeon(R) E-2324G

Base Docker image: Debian GNU/Linux bookworm/sid

Language Version
.NET Core 8.0.300
C#/.NET Core 4.10.0-3.24216.12 (3af0081a)
C#/Mono 6.12.0.200
Chez Scheme 9.5.8
Clojure "1.11.3"
Crystal 1.12.1
D/dmd v2.108.1
D/gdc 13.2.0
D/ldc2 1.38.0
Elixir 1.14.0
F#/.NET Core 12.8.300.0 for F# 8.0
Go go1.22.3
Go/gccgo 13.2.0
Haskell 9.8.2
Idris 2 0.6.0
Java 22.0.1
Julia v"1.10.3"
Kotlin 2.0.0
Lua 5.4.6
Lua/luajit 2.1.1710398010
MLton 20210117
Nim 2.0.4
Node.js v22.2.0
OCaml 5.2.0
PHP 8.2.18
Perl v5.38.2
Python 3.11.9
Python/pypy 7.3.16-final0 for Python 3.10.14
Racket "8.13"
Ruby 3.3.1p55
Ruby/jruby 9.4.7.0
Ruby/truffleruby 24.0.1
Rust 1.78.0
Scala 3.4.2
Swift 5.10
Tcl 8.6
V 0.4.6 736067d
Vala 0.56.17
Zig 0.12.0
clang/clang++ 16.0.6 (27)
gcc/g++ 13.2.0

Using Docker

Build the image:

$ docker build docker/ -t benchmarks

Run the image:

$ docker run -it --rm -v $(pwd):/src benchmarks <cmd>

where <cmd> is:

  • versions (print installed language versions);
  • shell (start the shell);
  • brainfuck bench (build and run Brainfuck bench.b benchmarks);
  • brainfuck mandel (build and run Brainfuck mandel.b benchmarks);
  • base64 (build and run Base64 benchmarks);
  • json (build and run Json benchmarks);
  • matmul (build and run Matmul benchmarks);
  • primes (build and run Primes benchmarks);

Please note that the actual measurements provided in the project are taken semi-manually (via shell) as the full update takes days and could have occassional issues in Docker.

There is a ./run.sh that could be used to simplify Docker usage:

  • ./run.sh build (build the image);
  • ./run.sh make versions (run the image with the versions command);
  • sudo ./run.sh shell (run the image with the `shell' command, sudo is required to read energy levels).

Manual Execution

Makefiles contain recipes for building and executing tests with the proper dependencies. Please use make run (and make run2 where applicable). The measurements are taken using analyze.rb script:

$ cd <test suite>
$ ../analyze.rb make run
$ ../analyze.rb make run[<single test>]

Please note that the measurements could take hours. It uses 10 iterations by default, but it could be changed using ATTEMPTS environment variable:

$ ATTEMPTS=1 ../analyze.rb make run

Prerequisites

Please use Dockerfile as a reference regarding which packages and tools are required.

For all (optional):

  • Powercap for reading energy counters in Linux (Debian package powercap-utils).

For Python:

  • NumPy for matmul tests (Debian package python3-numpy).
  • UltraJSON for JSON tests (Debian package python3-ujson).

For C++:

  • Boost for JSON tests (Debian package libboost-dev).
  • JSON-C for JSON tests (Debian package libjson-c-dev).

For Rust:

  • libjq for jq test (Debian packages libjq-dev, libonig-dev and environment variable JQ_LIB_DIR=/usr/lib/x86_64-linux-gnu/).

For Java, Scala:

  • Coursier for downloading Maven artifacts.

For Haskell:

  • network for TCP connectivity between the tests and the test runner.
  • raw-strings-qq for raw string literals used in tests.

For Perl:

  • cpanminus for installing modules from CPAN (Debian package cpanminus).

For Vala:

  • JSON-GLib for JSON tests (Debian package libjson-glib-dev).

Contribution

Please follow the criteria specified in the overview. Besides that please ensure that the communication protocol between a test and the test runner is satisfied:

  • The test runner listens on localhost:9001;
  • All messages are sent using TCP sockets closed immediately after the message has been sent;
  • There are two messages sent from a test (it establishes the measurement boundary):
    1. The beginning message having the format name of the test/tprocess ID (the process ID is used to measure the memory consumption). Please note that the name of the test couldn't use Tab character as it's a delimiter;
    2. The end message with any content (mostly it's "stop" for consistency).
  • The test runner could be unavailable (if the test is launched as is) and the test should gracefully handle it.

Makefile guide

Binary executables

If the test is compiled into a single binary, then two sections of the Makefile require changes:

  • append a new target (the final binary location) into executables variable;
  • append the proper target rule.

Compiled artifacts

If the test is compiled, but can't be executed directly as a binary, then three sections of the Makefile require changes:

  • append a new target (the final artifact location) into artifacts variable;
  • append the proper target rule to compile the test;
  • append run[<target_artifact>] rule to run the test.

Scripting language

If the test doesn't require compilation, then two sections of the Makefile requires changes:

  • append run[<script_file>] into all_runners variable;
  • append run[<script_file>] rule to run the test.

README update

TOC is regenerated using git-markdown-toc:

./run.sh toc