Collection indexes #3671

tpw56j · 2024-04-04T07:54:26Z

Linked Items

Fixes #3341

Fixes #3169

Description

Adds indexing of collections by non-key properties.

Behaviour changes

Old: Quick search is supported only by item name.

New: A quick search can be performed using non-key properties.

There is additional overhead for updating indexes and allocating memory for them.

Tests

Unit-Tests were created
System-Tests were created
Code is already covered by Unit-Tests
Code is already covered by System-Tests
No tests required

SchoolGuy · 2024-04-04T07:58:25Z

@tpw56j You are putting out PRs faster then I can review them. 😂 Thank you for the contribution again, I will try to get to it as soon as possible. 😄

tpw56j · 2024-04-04T08:01:52Z

@SchoolGuy This PR is only a draft; there is still a lot of work to be done on it to be ready for review.

SchoolGuy · 2024-04-10T11:55:09Z

@tpw56j A customer has asked already for this PR to be backported. As I can't do that before this PR is finished I would love it if we could agree on how to split this PR so that both of us can work on it simultaneously. As a first step, I would say that adding hard limits to the performance testsuite in a split-out PR is desirable, right?

tpw56j · 2024-04-10T12:41:34Z

@SchoolGuy I'm almost done with this PR and will try to post the changes I made tonight or tomorrow.
This PR requires tests for collections. I prepared changes for distros and added a test for systems. I don't have much experience writing good tests, and the tests I write seem clunky to me compared to the ones I see in Cobbler. And we don't have tests for all other collection types.
There is another problem that I would like you to solve. In the changes that I prepared, there is too close a direct interaction between the items and the internal data structures of the collections (collection.listing, collection.indexes) to maintain indexes when changing items. We need to come up with an API for such interaction.

SchoolGuy · 2024-04-10T12:43:43Z

@tpw56j That sounds like a solvable challenge for me. Testing is what "I grew up with" and as such is natural to me. I'll do my best next week after I have returned from some family matter that needs to be taken care of.

cobbler/items/system.py

tiltingpenguin

I have some small suggestions, but the rest looks good to me

cobbler/cobbler_collections/systems.py

tests/cobbler_collections/system_collection_test.py

SchoolGuy · 2024-05-27T23:57:55Z

It appears that with 50 there are less failing tests. I'll trigger a testrun for a master file so we can maybe get it working with 50 rounds.

SchoolGuy · 2024-05-28T15:11:27Z

It appears that running a job longer the 6h is not possible with GitHub Action Runners. - https://github.com/orgs/community/discussions/26679

As we cannot easily combine fixtures we need to lower the runs. I'll try 40 runs...

SchoolGuy · 2024-05-29T05:36:57Z

Okay so with 40 rounds the performance of the repo copy is varing by 80%... The results are just completely random. This is actually worse then with 25 rounds.

Edit: Funny enough the deserialize, get autoinstall, start and sync tests are fine. It is just the edit methods that take very long that are that flaky.

tpw56j · 2024-05-29T06:32:17Z

@SchoolGuy Maybe something will change if you add warmup_rounds and run tests with warmup=True?

SchoolGuy · 2024-05-29T06:35:20Z

@tpw56j We can try but I find it unlikely as we are spreading the tests already about a good chunk of time. Lastly I am comparing the mean time so a single or two bad runs should already be fine since we also will have good runs.

tpw56j · 2024-05-29T07:01:21Z

@SchoolGuy Then we can try to increase the number of iterations and reduce the number of rounds, for example iterations=5, rounds=10.
Instability when the number of iterations=1 can be caused by cache_enabled=True.

SchoolGuy · 2024-05-29T07:04:35Z

@tpw56j Since we group tests by name I believe that this is not happening. Feel free to check the results to take a more educated guess but I believe that the shared nature of the VMs is causing this behaviour. Locally all of those tests run rock solid when I go with anything above 10 iterations.

SchoolGuy · 2024-05-29T15:23:43Z

Summary after I looked at the results again:

The "master" JSON benchmark file was created with 40 rounds.
In the CI we run with 50 rounds.
There are multiple failed tests with a run-to-run variance of over 50 percent.
The failures are both with an enabled and disabled cache.
The failures are not consistent to a special item type.
We have success with multiple of our tests files. The variance in those test files is roughly between 4-6 percent.
Although all successful tests had a job length of roughly 10 minutes we also have the make_pxe_test with a job length of 10 minutes. As such the length of a job doesn't seem to have an effect on the stability of the tests.
All attempts at the moment are running on an AMD EPYC 7763 64-Core Processor.
The advertised frequency of the CPU is different by a full GHz in some cases. This will most definitely have impact on our tests since the dynamic frequency scaling will impact our overall performance since we are running our tests over a longer period of time.
The difference in CPU clock cycles is matching the advertised boost range that AMD puts up on their product page.

SchoolGuy · 2024-05-29T19:39:44Z

Sudden idea in the train: Since many of our tests are related indectly to the search of the application - and this PR was created on the observation it is flawed - we could add dedicated benchmarks for this. If that test is stable we should be able to focus the attention on other possibilities.

SchoolGuy · 2024-05-30T12:53:52Z

It may be that GitHub actions are not that unstable after all. The test test_make_pxe_menu[True-False] fails in five attempts always with roughly 19-22%. We might have a valid performance issue or the initial test had a "hiccup" with that particular test.

------------------------------------------------------------------------------ benchmark 'test_make_pxe_menu[True-False]': 2 tests ------------------------------------------------------------------------------
Name (time in ms)                                     Min                Max               Mean            StdDev             Median               IQR            Outliers      OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_make_pxe_menu[True-False] (0001_fa663b8)     11.5014 (1.0)      12.0291 (1.0)      11.7424 (1.0)      0.1062 (1.0)      11.7550 (1.0)      0.1313 (1.0)          12;1  85.1614 (1.0)          40           1
test_make_pxe_menu[True-False] (NOW)              14.1305 (1.23)     16.2059 (1.35)     14.3901 (1.23)     0.3179 (2.99)     14.3200 (1.22)     0.1785 (1.36)          4;4  69.4920 (0.82)         50           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_make_pxe_menu[True-False] (0001_fa663b8)     11.5014 (1.0)      12.0291 (1.0)      11.7424 (1.0)      0.1062 (1.0)      11.7550 (1.0)      0.1313 (1.19)         12;1  85.1614 (1.0)          40           1
test_make_pxe_menu[True-False] (NOW)              14.2362 (1.24)     14.8034 (1.23)     14.4297 (1.23)     0.1123 (1.06)     14.4117 (1.23)     0.1100 (1.0)           9;3  69.3013 (0.81)         50           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_make_pxe_menu[True-False] (0001_fa663b8)     11.5014 (1.0)      12.0291 (1.0)      11.7424 (1.0)      0.1062 (1.0)      11.7550 (1.0)      0.1313 (1.0)          12;1  85.1614 (1.0)          40           1
test_make_pxe_menu[True-False] (NOW)              13.7682 (1.20)     15.1159 (1.26)     14.1271 (1.20)     0.2150 (2.02)     14.0956 (1.20)     0.1835 (1.40)          4;2  70.7858 (0.83)         50           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_make_pxe_menu[True-False] (0001_fa663b8)     11.5014 (1.0)      12.0291 (1.0)      11.7424 (1.0)      0.1062 (1.0)      11.7550 (1.0)      0.1313 (1.0)          12;1  85.1614 (1.0)          40           1
test_make_pxe_menu[True-False] (NOW)              13.8359 (1.20)     25.8911 (2.15)     14.2929 (1.22)     1.6831 (15.85)    14.0461 (1.19)     0.1719 (1.31)          1;3  69.9649 (0.82)         50           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_make_pxe_menu[True-False] (0001_fa663b8)     11.5014 (1.0)      12.0291 (1.0)      11.7424 (1.0)      0.1062 (1.09)     11.7550 (1.0)      0.1313 (1.16)         12;1  85.1614 (1.0)          40           1
test_make_pxe_menu[True-False] (NOW)              13.9820 (1.22)     14.5003 (1.21)     14.2256 (1.21)     0.0978 (1.0)      14.2120 (1.21)     0.1136 (1.0)          11;2  70.2960 (0.83)         50           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

SchoolGuy · 2024-05-30T14:58:45Z

One performance bottleneck with item_* is that we call sync_single_system twice per system in collection.py in the add method. This shows in GH Actions since the drive is probably local and random IO is probably not that good.

Here is my weirdly structured notes that display the call chain. @tiltingpenguin please write a comment that confirms if I am right or wrong...

collection.py --> 
  self.lite_sync.add_single_system(ref)
    sync.py --> 
      self.dhcp.regen_ethers()
      self.dns.regen_hosts()
      self.tftpd.sync_single_system(system_obj)
        in_tftpd.py -->
          self.tftpgen.write_all_system_files(system, menu_items)
  self.api.sync_systems(systems=[ref.name])
    api.py -->
      sync_obj.run_sync_systems(systems)
        sync.py -->
          self.tftpd.sync_systems(systems)
            in_tftpd.py -->
              self.sync_single_system(system, menu_items)
          self.write_dhcp()
          self.dns.regen_hosts()
          self.dns.write_configs()

tiltingpenguin · 2024-05-30T15:10:14Z

One performance bottleneck with item_* is that we call sync_single_system twice per system in collection.py in the add method. This shows in GH Actions since the drive is probably local and random IO is probably not that good.

Here is my weirdly structured notes that display the call chain. @tiltingpenguin please write a comment that confirms if I am right or wrong...
collection.py --> 
  self.lite_sync.add_single_system(ref)
    sync.py --> 
      self.dhcp.regen_ethers()
      self.dns.regen_hosts()
      self.tftpd.sync_single_system(system_obj)
        in_tftpd.py -->
          self.tftpgen.write_all_system_files(system, menu_items)
  self.api.sync_systems(systems=[ref.name])
    api.py -->
      sync_obj.run_sync_systems(systems)
        sync.py -->
          self.tftpd.sync_systems(systems)
            in_tftpd.py -->
              self.sync_single_system(system, menu_items)
          self.write_dhcp()
          self.dns.regen_hosts()
          self.dns.write_configs()

Yes, this is the call chain I found as well

tpw56j · 2024-06-01T13:14:50Z

I think there was a slight slowdown in searching for non-indexed attributes, but overall I didn't notice a significant performance hit for make_pxe_menu. In make_pxe_menu_test, most searches are performed on non-indexed attributes menu.parent, profile.menu, image.menu.

It might make sense to add non-unique indexes for for these attributes .

My results for make_pxe_menu:

------------------------------------------------------------------------------ benchmark 'test_make_pxe_menu[False-False]': 2 tests ------------------------------------------------------------------------------
Name (time in ms)                                      Min                Max               Mean            StdDev             Median               IQR            Outliers      OPS            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_make_pxe_menu[False-False] (NOW)              34.3640 (1.0)      37.0110 (1.0)      34.9855 (1.0)      0.5328 (1.0)      34.8641 (1.0)      0.4117 (1.0)           9;6  28.5832 (1.0)          50           1
test_make_pxe_menu[False-False] (0001_fa663b8)     34.6346 (1.01)     39.6085 (1.07)     35.5616 (1.02)     0.7572 (1.42)     35.4307 (1.02)     0.5901 (1.43)          6;3  28.1202 (0.98)         50           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------ benchmark 'test_make_pxe_menu[False-True]': 2 tests -------------------------------------------------------------------------------
Name (time in ms)                                     Min                 Max               Mean            StdDev             Median               IQR            Outliers      OPS            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_make_pxe_menu[False-True] (NOW)              85.5807 (1.0)       89.6285 (1.0)      86.8564 (1.0)      0.8735 (1.0)      86.6590 (1.0)      1.4471 (1.0)          16;0  11.5133 (1.0)          50           1
test_make_pxe_menu[False-True] (0001_fa663b8)     85.7269 (1.00)     104.2812 (1.16)     87.6781 (1.01)     2.6346 (3.02)     87.2795 (1.01)     1.4959 (1.03)          2;2  11.4054 (0.99)         50           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------ benchmark 'test_make_pxe_menu[True-False]': 2 tests ------------------------------------------------------------------------------
Name (time in ms)                                     Min                Max               Mean            StdDev             Median               IQR            Outliers      OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_make_pxe_menu[True-False] (0001_fa663b8)     11.7563 (1.0)      12.6054 (1.0)      12.0982 (1.0)      0.2252 (1.0)      12.1391 (1.00)     0.3383 (1.44)         17;0  82.6572 (1.0)          50           1
test_make_pxe_menu[True-False] (NOW)              11.9223 (1.01)     15.3728 (1.22)     12.2157 (1.01)     0.4878 (2.17)     12.1087 (1.0)      0.2353 (1.0)           2;2  81.8622 (0.99)         50           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------ benchmark 'test_make_pxe_menu[True-True]': 2 tests ------------------------------------------------------------------------------
Name (time in ms)                                    Min                Max               Mean            StdDev             Median               IQR            Outliers      OPS            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_make_pxe_menu[True-True] (0001_fa663b8)     50.9316 (1.0)      52.2871 (1.0)      51.3841 (1.0)      0.3030 (1.0)      51.3483 (1.0)      0.3970 (1.0)          17;1  19.4613 (1.0)          50           1
test_make_pxe_menu[True-True] (NOW)              51.4782 (1.01)     54.8587 (1.05)     52.2666 (1.02)     0.5764 (1.90)     52.1309 (1.02)     0.4588 (1.16)          7;4  19.1327 (0.98)         50           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------ benchmark 'test_make_pxe_menu[False-False]': 2 tests ------------------------------------------------------------------------------
Name (time in ms)                                      Min                Max               Mean            StdDev             Median               IQR            Outliers      OPS            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_make_pxe_menu[False-False] (NOW)              34.2592 (1.0)      36.3921 (1.0)      34.9574 (1.0)      0.3683 (1.0)      34.9406 (1.0)      0.4692 (1.0)          14;1  28.6063 (1.0)          50           1
test_make_pxe_menu[False-False] (0001_fa663b8)     34.6346 (1.01)     39.6085 (1.09)     35.5616 (1.02)     0.7572 (2.06)     35.4307 (1.01)     0.5901 (1.26)          6;3  28.1202 (0.98)         50           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------ benchmark 'test_make_pxe_menu[False-True]': 2 tests -------------------------------------------------------------------------------
Name (time in ms)                                     Min                 Max               Mean            StdDev             Median               IQR            Outliers      OPS            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_make_pxe_menu[False-True] (NOW)              85.0148 (1.0)       88.8236 (1.0)      86.4528 (1.0)      0.7666 (1.0)      86.2433 (1.0)      0.6864 (1.0)          12;4  11.5670 (1.0)          50           1
test_make_pxe_menu[False-True] (0001_fa663b8)     85.7269 (1.01)     104.2812 (1.17)     87.6781 (1.01)     2.6346 (3.44)     87.2795 (1.01)     1.4959 (2.18)          2;2  11.4054 (0.99)         50           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------ benchmark 'test_make_pxe_menu[True-False]': 2 tests ------------------------------------------------------------------------------
Name (time in ms)                                     Min                Max               Mean            StdDev             Median               IQR            Outliers      OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_make_pxe_menu[True-False] (NOW)              11.7470 (1.0)      12.8450 (1.02)     11.9837 (1.0)      0.1818 (1.0)      11.9442 (1.0)      0.1164 (1.0)           8;4  83.4467 (1.0)          50           1
test_make_pxe_menu[True-False] (0001_fa663b8)     11.7563 (1.00)     12.6054 (1.0)      12.0982 (1.01)     0.2252 (1.24)     12.1391 (1.02)     0.3383 (2.91)         17;0  82.6572 (0.99)         50           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------ benchmark 'test_make_pxe_menu[True-True]': 2 tests ------------------------------------------------------------------------------
Name (time in ms)                                    Min                Max               Mean            StdDev             Median               IQR            Outliers      OPS            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_make_pxe_menu[True-True] (NOW)              50.6906 (1.0)      54.8338 (1.05)     51.5669 (1.00)     0.6873 (2.27)     51.4606 (1.00)     0.4931 (1.24)          7;3  19.3923 (1.00)         50           1
test_make_pxe_menu[True-True] (0001_fa663b8)     50.9316 (1.00)     52.2871 (1.0)      51.3841 (1.0)      0.3030 (1.0)      51.3483 (1.0)      0.3970 (1.0)          17;1  19.4613 (1.0)          50           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------ benchmark 'test_make_pxe_menu[False-False]': 2 tests ------------------------------------------------------------------------------
Name (time in ms)                                      Min                Max               Mean            StdDev             Median               IQR            Outliers      OPS            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_make_pxe_menu[False-False] (NOW)              34.3728 (1.0)      36.7982 (1.0)      35.1037 (1.0)      0.3832 (1.0)      35.1199 (1.0)      0.3999 (1.0)          11;1  28.4870 (1.0)          50           1
test_make_pxe_menu[False-False] (0001_fa663b8)     34.6346 (1.01)     39.6085 (1.08)     35.5616 (1.01)     0.7572 (1.98)     35.4307 (1.01)     0.5901 (1.48)          6;3  28.1202 (0.99)         50           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------ benchmark 'test_make_pxe_menu[False-True]': 2 tests -------------------------------------------------------------------------------
Name (time in ms)                                     Min                 Max               Mean            StdDev             Median               IQR            Outliers      OPS            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_make_pxe_menu[False-True] (0001_fa663b8)     85.7269 (1.0)      104.2812 (1.15)     87.6781 (1.00)     2.6346 (3.36)     87.2795 (1.0)      1.4959 (1.80)          2;2  11.4054 (1.00)         50           1
test_make_pxe_menu[False-True] (NOW)              86.3621 (1.01)      90.6847 (1.0)      87.5367 (1.0)      0.7843 (1.0)      87.3242 (1.00)     0.8293 (1.0)          10;3  11.4238 (1.0)          50           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------ benchmark 'test_make_pxe_menu[True-False]': 2 tests ------------------------------------------------------------------------------
Name (time in ms)                                     Min                Max               Mean            StdDev             Median               IQR            Outliers      OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_make_pxe_menu[True-False] (0001_fa663b8)     11.7563 (1.0)      12.6054 (1.0)      12.0982 (1.0)      0.2252 (1.35)     12.1391 (1.0)      0.3383 (2.45)         17;0  82.6572 (1.0)          50           1
test_make_pxe_menu[True-False] (NOW)              12.1577 (1.03)     13.0305 (1.03)     12.4328 (1.03)     0.1669 (1.0)      12.4075 (1.02)     0.1381 (1.0)           7;3  80.4326 (0.97)         50           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------ benchmark 'test_make_pxe_menu[True-True]': 2 tests ------------------------------------------------------------------------------
Name (time in ms)                                    Min                Max               Mean            StdDev             Median               IQR            Outliers      OPS            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_make_pxe_menu[True-True] (0001_fa663b8)     50.9316 (1.0)      52.2871 (1.0)      51.3841 (1.0)      0.3030 (1.0)      51.3483 (1.0)      0.3970 (1.0)          17;1  19.4613 (1.0)          50           1
test_make_pxe_menu[True-True] (NOW)              51.4433 (1.01)     58.8786 (1.13)     52.2903 (1.02)     1.0722 (3.54)     52.0409 (1.01)     0.6392 (1.61)          3;3  19.1240 (0.98)         50           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

fa663b8:

         242703 function calls (241819 primitive calls) in 0.097 seconds

   Ordered by: cumulative time
   List reduced from 181 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.097    0.097 /usr/src/cobbler-main/cobbler/tftpgen.py:533(make_pxe_menu)
       12    0.000    0.000    0.096    0.008 /usr/src/cobbler-main/cobbler/tftpgen.py:700(get_menu_items)
    84/12    0.002    0.000    0.096    0.008 /usr/src/cobbler-main/cobbler/tftpgen.py:948(get_menu_level)
        1    0.000    0.000    0.089    0.089 /usr/src/cobbler-main/cobbler/tftpgen.py:612(_make_pxe_menu_grub)
    84/12    0.000    0.000    0.084    0.007 /usr/src/cobbler-main/cobbler/tftpgen.py:755(get_submenus)
  357/225    0.002    0.000    0.079    0.000 /usr/src/cobbler-main/cobbler/cobbler_collections/collection.py:180(find)
    72/24    0.000    0.000    0.079    0.003 /usr/src/cobbler-main/cobbler/tftpgen.py:712(_get_submenu_child)
      744    0.002    0.000    0.076    0.000 /usr/src/cobbler-main/cobbler/items/item.py:1007(find_match)
      744    0.033    0.000    0.070    0.000 /usr/src/cobbler-main/cobbler/items/item.py:1170(to_dict)
       84    0.000    0.000    0.057    0.001 /usr/src/cobbler-main/cobbler/tftpgen.py:847(get_profiles_menu)
       84    0.000    0.000    0.056    0.001 /usr/src/cobbler-main/cobbler/api.py:1342(find_profile)
       84    0.000    0.000    0.019    0.000 /usr/src/cobbler-main/cobbler/tftpgen.py:903(get_images_menu)
       84    0.000    0.000    0.018    0.000 /usr/src/cobbler-main/cobbler/api.py:1402(find_image)
   106517    0.015    0.000    0.015    0.000 {built-in method builtins.isinstance}
     3192    0.004    0.000    0.011    0.000 /usr/lib64/python3.12/copy.py:118(deepcopy)
    32232    0.006    0.000    0.006    0.000 /usr/src/cobbler-main/cobbler/items/item.py:255(__is_dict_key)
       12    0.000    0.000    0.005    0.000 /usr/src/cobbler-main/cobbler/api.py:1422(find_menu)
      252    0.000    0.000    0.005    0.000 /usr/lib64/python3.12/pathlib.py:852(exists)
      252    0.000    0.000    0.005    0.000 /usr/lib64/python3.12/pathlib.py:835(stat)
      256    0.001    0.000    0.005    0.000 {built-in method posix.stat}

This PR:

         243243 function calls (242359 primitive calls) in 0.100 seconds

   Ordered by: cumulative time
   List reduced from 182 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.100    0.100 /usr/src/cobbler-feature-collection_indexes/cobbler/tftpgen.py:533(make_pxe_menu)
       12    0.000    0.000    0.098    0.008 /usr/src/cobbler-feature-collection_indexes/cobbler/tftpgen.py:700(get_menu_items)
    84/12    0.002    0.000    0.098    0.008 /usr/src/cobbler-feature-collection_indexes/cobbler/tftpgen.py:948(get_menu_level)
        1    0.000    0.000    0.091    0.091 /usr/src/cobbler-feature-collection_indexes/cobbler/tftpgen.py:612(_make_pxe_menu_grub)
    84/12    0.000    0.000    0.086    0.007 /usr/src/cobbler-feature-collection_indexes/cobbler/tftpgen.py:755(get_submenus)
    72/24    0.000    0.000    0.080    0.003 /usr/src/cobbler-feature-collection_indexes/cobbler/tftpgen.py:712(_get_submenu_child)
  357/225    0.002    0.000    0.080    0.000 /usr/src/cobbler-feature-collection_indexes/cobbler/cobbler_collections/collection.py:184(find)
      744    0.002    0.000    0.077    0.000 /usr/src/cobbler-feature-collection_indexes/cobbler/items/item.py:1006(find_match)
      744    0.034    0.000    0.071    0.000 /usr/src/cobbler-feature-collection_indexes/cobbler/items/item.py:1169(to_dict)
       84    0.000    0.000    0.058    0.001 /usr/src/cobbler-feature-collection_indexes/cobbler/tftpgen.py:847(get_profiles_menu)
       84    0.000    0.000    0.057    0.001 /usr/src/cobbler-feature-collection_indexes/cobbler/api.py:1342(find_profile)
       84    0.000    0.000    0.019    0.000 /usr/src/cobbler-feature-collection_indexes/cobbler/tftpgen.py:903(get_images_menu)
       84    0.000    0.000    0.019    0.000 /usr/src/cobbler-feature-collection_indexes/cobbler/api.py:1402(find_image)
   106517    0.015    0.000    0.015    0.000 {built-in method builtins.isinstance}
     3192    0.004    0.000    0.011    0.000 /usr/lib64/python3.12/copy.py:118(deepcopy)
    32232    0.006    0.000    0.006    0.000 /usr/src/cobbler-feature-collection_indexes/cobbler/items/item.py:255(__is_dict_key)
       12    0.000    0.000    0.005    0.000 /usr/src/cobbler-feature-collection_indexes/cobbler/api.py:1422(find_menu)
      252    0.000    0.000    0.005    0.000 /usr/lib64/python3.12/pathlib.py:852(exists)
      252    0.000    0.000    0.005    0.000 /usr/lib64/python3.12/pathlib.py:835(stat)
      256    0.001    0.000    0.005    0.000 {built-in method posix.stat}

SchoolGuy · 2024-06-01T18:37:25Z

@tpw56j I have discussed with several colleagues and friends offline and so far the conclusion that the shared nature of the GitHub Runners is causing the variation. I will attempt to address the open review suggestions by @tiltingpenguin and then we'll merge this PR.

Mid term I will sponsor a dedicated cluster that will ensure that performance testing is much more stable. To prevent keeping this project tied to me, I will transparently document the required steps to duplicate the setup.

This makes the tests fail if the mean average is different by 10%.

tiltingpenguin

My comments were addressed, so it LGTM now

SchoolGuy · 2024-06-03T14:15:17Z

@tpw56j If you have no reservations about the changes that I added to your PR I would merge the PR independent of the results in the CI. I would address the CI failures as soon as time allows. I verified manually that the changes have the desired effect that you advertised.

tpw56j · 2024-06-03T14:20:40Z

@SchoolGuy I have no reservations about the changes that you have added to my PR.

SchoolGuy · 2024-06-03T17:33:00Z

I will merge this PR even though the performance testsuite is not working. It was established that the GitHub infrastructure is at fault for this.

github-actions bot added API tests labels Apr 4, 2024

tpw56j marked this pull request as draft April 4, 2024 07:54

tpw56j mentioned this pull request Apr 4, 2024

Perfomance testsuite #3341

Closed

tpw56j force-pushed the feature/collection_indexes branch 5 times, most recently from faad11d to 5572dee Compare April 5, 2024 10:02

tpw56j force-pushed the feature/collection_indexes branch 14 times, most recently from 7905d40 to eab51ae Compare April 11, 2024 08:21

tpw56j marked this pull request as ready for review April 11, 2024 17:01

SchoolGuy reviewed Apr 16, 2024

View reviewed changes

cobbler/items/system.py Show resolved Hide resolved

tiltingpenguin reviewed May 27, 2024

View reviewed changes

cobbler/cobbler_collections/systems.py Show resolved Hide resolved

tests/cobbler_collections/system_collection_test.py Outdated Show resolved Hide resolved

tests/cobbler_collections/system_collection_test.py Outdated Show resolved Hide resolved

tpw56j and others added 8 commits June 3, 2024 15:56

Collection indexes

08abf9c

Deduplicate NetworkInterface index update

920553f

Move interface index update to Systems collection

5c4a83d

CI: Split performance testsuite into matrix job

efac2c5

CI: Add performance comparison

8c59ac6

This makes the tests fail if the mean average is different by 10%.

CI: Improve runs to 50

9be70b4

CI: Replace "master" comparison file

47e94aa

Collection: Remove duplicate system serialization

4e60242

SchoolGuy force-pushed the feature/collection_indexes branch from fbb8437 to 4e60242 Compare June 3, 2024 14:03

SchoolGuy requested a review from tiltingpenguin June 3, 2024 14:03

tiltingpenguin approved these changes Jun 3, 2024

View reviewed changes

SchoolGuy merged commit da8c246 into cobbler:main Jun 3, 2024
34 of 45 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Collection indexes #3671

Collection indexes #3671

tpw56j commented Apr 4, 2024 •

edited by SchoolGuy

SchoolGuy commented Apr 4, 2024

tpw56j commented Apr 4, 2024

SchoolGuy commented Apr 10, 2024

tpw56j commented Apr 10, 2024

SchoolGuy commented Apr 10, 2024

tiltingpenguin left a comment

SchoolGuy commented May 27, 2024

SchoolGuy commented May 28, 2024

SchoolGuy commented May 29, 2024 •

edited

tpw56j commented May 29, 2024

SchoolGuy commented May 29, 2024

tpw56j commented May 29, 2024

SchoolGuy commented May 29, 2024

SchoolGuy commented May 29, 2024

SchoolGuy commented May 29, 2024

SchoolGuy commented May 30, 2024 •

edited

SchoolGuy commented May 30, 2024

tiltingpenguin commented May 30, 2024

tpw56j commented Jun 1, 2024

SchoolGuy commented Jun 1, 2024

tiltingpenguin left a comment

SchoolGuy commented Jun 3, 2024 •

edited

tpw56j commented Jun 3, 2024

SchoolGuy commented Jun 3, 2024

Collection indexes #3671

Collection indexes #3671

Conversation

tpw56j commented Apr 4, 2024 • edited by SchoolGuy

Linked Items

Description

Behaviour changes

Category

Tests

SchoolGuy commented Apr 4, 2024

tpw56j commented Apr 4, 2024

SchoolGuy commented Apr 10, 2024

tpw56j commented Apr 10, 2024

SchoolGuy commented Apr 10, 2024

tiltingpenguin left a comment

Choose a reason for hiding this comment

SchoolGuy commented May 27, 2024

SchoolGuy commented May 28, 2024

SchoolGuy commented May 29, 2024 • edited

tpw56j commented May 29, 2024

SchoolGuy commented May 29, 2024

tpw56j commented May 29, 2024

SchoolGuy commented May 29, 2024

SchoolGuy commented May 29, 2024

SchoolGuy commented May 29, 2024

SchoolGuy commented May 30, 2024 • edited

SchoolGuy commented May 30, 2024

tiltingpenguin commented May 30, 2024

tpw56j commented Jun 1, 2024

SchoolGuy commented Jun 1, 2024

tiltingpenguin left a comment

Choose a reason for hiding this comment

SchoolGuy commented Jun 3, 2024 • edited

tpw56j commented Jun 3, 2024

SchoolGuy commented Jun 3, 2024

tpw56j commented Apr 4, 2024 •

edited by SchoolGuy

SchoolGuy commented May 29, 2024 •

edited

SchoolGuy commented May 30, 2024 •

edited

SchoolGuy commented Jun 3, 2024 •

edited