Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add perf metrics for 2.23.0 #45443

Merged
merged 3 commits into from
May 20, 2024
Merged

Add perf metrics for 2.23.0 #45443

merged 3 commits into from
May 20, 2024

Conversation

khluu
Copy link
Contributor

@khluu khluu commented May 20, 2024

REGRESSION 12.60%: client__tasks_and_get_batch (THROUGHPUT) regresses from 0.9229179501706766 to 0.8066078636677214 in microbenchmark.json
REGRESSION 9.95%: single_client_tasks_async (THROUGHPUT) regresses from 8194.32696391844 to 7378.7495889110505 in microbenchmark.json
REGRESSION 7.64%: multi_client_put_calls_Plasma_Store (THROUGHPUT) regresses from 12873.027401685875 to 11889.62967857741 in microbenchmark.json
REGRESSION 6.02%: placement_group_create/removal (THROUGHPUT) regresses from 838.5131869401664 to 788.0742341227435 in microbenchmark.json
REGRESSION 5.08%: single_client_tasks_and_get_batch (THROUGHPUT) regresses from 8.144156516407913 to 7.730134993824188 in microbenchmark.json
REGRESSION 4.34%: 1_1_actor_calls_sync (THROUGHPUT) regresses from 2096.4576697416196 to 2005.483489295091 in microbenchmark.json
REGRESSION 4.00%: client__tasks_and_put_batch (THROUGHPUT) regresses from 11657.997816451738 to 11191.448976686781 in microbenchmark.json
REGRESSION 3.82%: actors_per_second (THROUGHPUT) regresses from 638.1892393086097 to 613.8104422632972 in benchmarks/many_actors.json
REGRESSION 3.27%: 1_1_actor_calls_async (THROUGHPUT) regresses from 9062.792852859755 to 8766.429278436297 in microbenchmark.json
REGRESSION 3.14%: client__get_calls (THROUGHPUT) regresses from 1158.7613618440014 to 1122.3334401198258 in microbenchmark.json
REGRESSION 2.21%: 1_n_async_actor_calls_async (THROUGHPUT) regresses from 7460.166048300169 to 7295.2280994398525 in microbenchmark.json
REGRESSION 1.52%: n_n_actor_calls_with_arg_async (THROUGHPUT) regresses from 2713.504279822532 to 2672.287170984248 in microbenchmark.json
REGRESSION 1.34%: 1_n_actor_calls_async (THROUGHPUT) regresses from 8606.08018065877 to 8490.352729000244 in microbenchmark.json
REGRESSION 1.32%: n_n_actor_calls_async (THROUGHPUT) regresses from 27688.27595653398 to 27321.88100222539 in microbenchmark.json
REGRESSION 0.62%: pgs_per_second (THROUGHPUT) regresses from 23.57570074520289 to 23.42930283459158 in benchmarks/many_pgs.json
REGRESSION 0.29%: client__put_gigabytes (THROUGHPUT) regresses from 0.13155858254345548 to 0.13117998295265082 in microbenchmark.json
REGRESSION 0.25%: 1_1_actor_calls_concurrent (THROUGHPUT) regresses from 5479.71996068499 to 5466.2521928780425 in microbenchmark.json
REGRESSION 0.15%: tasks_per_second (THROUGHPUT) regresses from 580.7153723158294 to 579.8482552816317 in benchmarks/many_tasks.json
REGRESSION 35.76%: stage_2_avg_iteration_time (LATENCY) regresses from 54.19231338500977 to 73.57129535675048 in stress_tests/stress_test_many_tasks.json
REGRESSION 25.91%: dashboard_p50_latency_ms (LATENCY) regresses from 68.347 to 86.057 in benchmarks/many_actors.json
REGRESSION 16.53%: stage_1_avg_iteration_time (LATENCY) regresses from 23.831846237182617 to 27.771051788330077 in stress_tests/stress_test_many_tasks.json
REGRESSION 16.02%: dashboard_p95_latency_ms (LATENCY) regresses from 2276.742 to 2641.416 in benchmarks/many_actors.json
REGRESSION 13.10%: dashboard_p99_latency_ms (LATENCY) regresses from 4468.284 to 5053.446 in benchmarks/many_actors.json
REGRESSION 12.51%: stage_3_time (LATENCY) regresses from 2976.5505974292755 to 3348.829131603241 in stress_tests/stress_test_many_tasks.json
REGRESSION 10.60%: dashboard_p99_latency_ms (LATENCY) regresses from 812.483 to 898.617 in benchmarks/many_pgs.json
REGRESSION 4.98%: avg_pg_create_time_ms (LATENCY) regresses from 0.8636297822809175 to 0.9066000405417517 in stress_tests/stress_test_placement_group.json
REGRESSION 3.78%: stage_0_time (LATENCY) regresses from 12.160747766494751 to 12.619915246963501 in stress_tests/stress_test_many_tasks.json
REGRESSION 3.72%: dashboard_p50_latency_ms (LATENCY) regresses from 42.738 to 44.327 in benchmarks/many_tasks.json
REGRESSION 3.25%: 10000_get_time (LATENCY) regresses from 23.237287506 to 23.992132071 in scalability/single_node.json
REGRESSION 1.79%: dashboard_p95_latency_ms (LATENCY) regresses from 3207.541 to 3265.071 in benchmarks/many_tasks.json
REGRESSION 0.85%: 10000_args_time (LATENCY) regresses from 17.131764829000005 to 17.276734743999995 in scalability/single_node.json
REGRESSION 0.76%: 107374182400_large_object_time (LATENCY) regresses from 31.629831378000006 to 31.869567716000006 in scalability/single_node.json

khluu and others added 2 commits May 20, 2024 01:33
p
Signed-off-by: khluu <51931015+khluu@users.noreply.github.com>
Signed-off-by: kevin <kevin@anyscale.com>
@khluu khluu requested a review from jjyao May 20, 2024 08:49
Base automatically changed from khluu/fix_perf_dir to master May 20, 2024 17:24
Signed-off-by: khluu <51931015+khluu@users.noreply.github.com>
Copy link
Contributor

@jjyao jjyao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LG

@aslonnie aslonnie added the go Trigger full test run on premerge label May 20, 2024
@aslonnie aslonnie self-requested a review May 20, 2024 21:45
@aslonnie aslonnie enabled auto-merge (squash) May 20, 2024 21:48
@aslonnie aslonnie merged commit 9eb2ce7 into master May 20, 2024
8 checks passed
@aslonnie aslonnie deleted the 2.23.0_perf_metrics2 branch May 20, 2024 22:05
ryanaoleary pushed a commit to ryanaoleary/ray that referenced this pull request Jun 6, 2024
approved by @jjyao

---------

Signed-off-by: khluu <51931015+khluu@users.noreply.github.com>
Signed-off-by: kevin <kevin@anyscale.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
ryanaoleary pushed a commit to ryanaoleary/ray that referenced this pull request Jun 6, 2024
approved by @jjyao

---------

Signed-off-by: khluu <51931015+khluu@users.noreply.github.com>
Signed-off-by: kevin <kevin@anyscale.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
ryanaoleary pushed a commit to ryanaoleary/ray that referenced this pull request Jun 7, 2024
approved by @jjyao 

---------

Signed-off-by: khluu <51931015+khluu@users.noreply.github.com>
Signed-off-by: kevin <kevin@anyscale.com>
GabeChurch pushed a commit to GabeChurch/ray that referenced this pull request Jun 11, 2024
approved by @jjyao

---------

Signed-off-by: khluu <51931015+khluu@users.noreply.github.com>
Signed-off-by: kevin <kevin@anyscale.com>
Signed-off-by: gchurch <gabe1church@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
go Trigger full test run on premerge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants