GUI performance with many thousands of torrents - transfer list, icons, and more

[FranciscoPombal]

qBittorrent's GUI performance with many torrents (as in, many thousands of torrents) could probably be better.

I created this issue for the purposes of experimentation/investigation/discussion about this, without having to deal with excessive baggage from previous threads.

Relevant background:

• 4.2.5 takes half an hour to load UI #12825

TL;DR: a bug in libtorrent was causing a long-running operation to block everything else. While that has been fixed in libtorrent 1.2.8+ (report confirming: #12825 (comment)), the GUI performance still leaves something to be desired.

• laggy and slow responding UI #11822

TL;DR: On Linux, investigation into the problem showed that inefficient WMs/compositors (or some inefficient way qBittorrent interacts with them!) play a significant role in the perceived lack of performance. Other than that, it was probably part #12825, part #11822 (comment):

I even ran a small test with callgrind on Xubuntu - not a very thorough one, I did not even have debug symbols (I was using the qBittorrent executable from the official PPA). Even without symbols, I could see that most of the activity came from internal Qt GUI-related stuff, and an enormous amount of calls to somewhere within libX11.so. A lot of calls seemed to be to something related to/with a name like XSetImage/XGetImage - I wonder if this has to do with status icons of each element in the [transfer] list?

---

This is not a thread for regular bug reports. It is expected that posters have some idea of where the problem might be or have done some preliminary investigation/analysis/benchmarks about the problem with developer/power-user performance profiling tools. This is the place to share such findings.

When posting in this thread and investigating these performance issues, please provide as much information as possible from the get-go, to make it easier for others to understand your setup ad environment..
At least:

• Full qBittorrent logs and settings in plain text
• Number of torrents and number of files in each torrent (or at least an average of the latter, or a reasonable verbal description of the distribution of files per torrent)
• One or more profiling reports or equivalent, if possible with a accompanying benchmarks and/or a little analysis - on Linux, callgrind and massif are your friends.

The goal is to identify specific points where performance can be improved in relation to the GUI, and particularly in the transfer list. As each specific cause/bug becomes clearer and defined as its own separate matter, discussion about it may move to a dedicated thread about it (this is also useful for having something specific to close when submitting a PR). This one is just for initial investigations/brainstorming, e.g. "What if excessive icon load/unload is slowing down the transfer list?", "How can we test that?" "We could try doing foo..." "I tried <this patch>, and got the following results..."...

[xavier2k6]

speedplotview assumes speed is updated per second but the default value is 1500ms this causes lots of duplicate data in the calculation for the graph.

May have some relevance?!

PR 13263

[jagannatharjun]

The only thing I can think that is regularly called and can cause this is transfer list refreshing, you can change the interval in the advance setting. Actually, in 4.2.5, subsequent refresh calls can overlap (fixed in master) when CPU is busy causing more damage, so setting large refresh intervals actually may help. AFAIK you can also disable speed graphs from advance settings if you think that is the problem. Another possible reason can be that libtorrent is posting a large number of alerts for some reason, you can add some logging and take a look.

Also, I don't think Qt can be the reason. Qt is very efficient and is used in many other high-end applications So it must be something within qbittorrent itself.

[FranciscoPombal]

@jagannatharjun

The only thing I can think that is regularly called and can cause this is transfer list refreshing, you can change the interval in the advance setting. Actually, in 4.2.5, subsequent refresh calls can overlap (fixed in master) when CPU is busy causing more damage, so setting large refresh intervals actually may help.

Is the transfer list all refreshed at the same time? Is it possible to perhaps optimize it by just refreshing the visual representation of the items that are visible at any given time? Or do we already do that?

As mentioned in #11822 (comment), could it also be the case that there is room for optimization in the way the status icons are displayed in the list? My suspicion is that a bunch of svg icons frequently being loaded could be a problem, perhaps.

Another possible reason can be that libtorrent is posting a large number of alerts for some reason, you can add some logging and take a look.

A bit of anecdotal evidence: I have noticed, particularly in low end machines, that qBittorrent's WebUI becomes unusable (unresponsive) under high resource consumption scenarios. I suspect this is due to the fact that all parts of the application are fighting unconstrained for a share of the computing resources. Ideally, computations related to GUI code, WebAPI requests (such as those coming from the WebUI), etc should have much higher priority/precedence than, say, libtorrent piece hashing, so that the user can still control the application, even at the expense of a little transfer performance while doing so. I have no idea how this could be implemented, though.

Also, I don't think Qt can be the reason. Qt is very efficient and is used in many other high-end applications So it must be something within qbittorrent itself.

Indeed.

[xavier2k6]

Perhaps qBittorrent defaults need to be adjusted again?!

• Transfer list refresh interval 1500ms -> 2000ms

• Enable speed graphs checked -> unchecked

• Asynchronous I/O threads 4 -> 8 for reference

Still, bumping from 4 -> 8 would be a good change that would benefit SSD users without hurting HDD users

#11461 (comment)

• File pool size 40 -> 400/4000

@nokti can you change your file pool size in advanced settings from 40 to 4000 please & report back.
I can confirm I had this very same issue with 4.2.5 and following these instructions seems to have fixed it

#12914 (comment)

In relation to progress bar would the use of palette colors like the pieces bars PR #13256
have much of an impact or perhaps changing to a different style......QfusionStyle or equivalent (apologies. I don't know which style is used etc)

Another thing, that may have an impact is if torrents are checking on start-up....currently if a torrent is checked/re-checked, there is no record of this being carried out in the execution logs.

[jagannatharjun]

Is the transfer list all refreshed at the same time? Is it possible to perhaps optimize it by just refreshing the visual representation of the items that are visible at any given time? Or do we already do that?

They do update at the same time but the real implementation of torrent update in libtorrent is actually asynchronous and shouldn't block GUI, even with 10000 torrent I don't see it causing too much load to block GUI indirectly.

I think rather then speculating and wasting time it would be better if someone can replicate and profile the application.

could it also be the case that there is room for optimization in the way the status icons are displayed in the list? My suspicion is that a bunch of svg icons frequently being loaded could be a problem, perhaps

Icons are cached.

[FranciscoPombal]

@jagannatharjun

They do update at the same time but the real implementation of torrent update in libtorrent is actually asynchronous and shouldn't block GUI, even with 10000 torrent I don't see it causing too much load to block GUI indirectly.

Ok, not a problem then.

I think rather then speculating and wasting time it would be better if someone can replicate and profile the application.

Icons are cached.

Well said. An idea could be comparing two profiling runs, one with transfer list icons, and one with no transfer list icons at all, just to see how much CPU the icon display code eats up. I can probably do that in the next few days, if no one does it before that.

[jagannatharjun]

one with transfer list icons, and one with no transfer list icons at all

transfer list icons are cached, they can't be the problem, they are only loaded once, but you can try benchmark over it if you wish.
almost all the icons loaded in GUI are cached.

[xavier2k6]

This may also help #13316

[xavier2k6]

off topic, where does qBittorrent make use of zlib?

[FranciscoPombal]

@xavier2k6

https://github.com/qbittorrent/qBittorrent/search?l=C%2B%2B&q=zlib

Keep in mind the results preview in GitHub's search GUI only shows the first couple matches in each file.

[reyaz006]

I'm very sorry but my problem is very similar and was never fully resolved. I'm seeking help and if anyone wants, I can perform tests or benchmarks etc.

• I'm on qbt 4.1.9.1 (Windows) and I'm too afraid of upgrading to any newer version. I've already experienced tons of re-downloading of disabled files before and I don't want to take any further risks.
• I have >38k torrents in my list.
• The GUI takes a very long time to load. 1~2 hours.

The most serious problem is RAM consumption. Sometimes it can stay adequate (2-3 GB) for weeks, and sometimes it's starting to eat more and more by a minute, and when it reaches >20 GB other apps start crashing.
My thoughts (and a suggestion) about the issue are here #11397

I noticed once again that no matter what, on startup qBT seems to access downloaded files of paused and finished torrents, even though there should be no such need.
Also noticed that at the startup some torrents (few hundreds) are getting rechecked and switched to "Missing files". It takes hours until it finishes this with all the torrents.
Was any of these ever fixed?

Another suggestion about GUI performance is about tags (or categories). I noticed that whenever current list holds smaller amount of torrents, it performs much better. By default the GUI loads always in "All" category, so it always forcing the worst scenario of RAM consumption from the very start. As soon as it becomes responsive I switch to another category. My guess is that if it's possible to make qBT start with an empty category, or a category with just a few torrents, the GUI would load much faster.

I'm not sure what I should provide if you ask me for "Full qBittorrent logs and settings in plain text". My log files are always huge (several MB) and quite useless. Also unicode torrent names are not properly saved. The settings file include some bits that I'd like to keep private, e.g. cookies and paths.

[reyaz006]

Right now it appears to be working well and responsive but RAM usage is constantly increasing. From 3 GB ~3 hours ago to 7.5 GB now. I didn't do any actions with existing torrents and didn't add any new ones. 23 MB downloaded and 5 GB uploaded.

[FranciscoPombal]

@reyaz006

That's a very old version and a whole lot of torrents indeed. I would suggest backing up all your settings/fastresumes (just in case, see https://github.com/qbittorrent/qBittorrent/wiki/Frequently-Asked-Questions#Where_does_qBittorrent_save_its_settings for more info), and trying a build of recent master commits + libtorrent >=1.2.10, such as the most recent of these: https://github.com/qbittorrent/qBittorrent/actions targeting master.

Many important fixes have been deployed since 4.1.9, including fixes that massively improve memory usage and GUI startup time in some circumstances. Recently a user with many torrents reported a massive improvement with recent commits: #12825

[FranciscoPombal]

Some general news: #13348 has been submitted with some improvements to startup code. Furthermore, an issue has been identified with tracker announces that can impact GUI startup time massively. Investigation is still ongoing.

[reyaz006]

I will try that but which file should I use? What's the difference?

qBittorrent-nightly-e16aafb_vcpkg-fdac1fc_qt5-lts-5.15.0_libtorrent-release-1.2.10_x64-static-release
qBittorrent-nightly-e16aafb_vcpkg-fdac1fc_qt5-lts-5.15.0_libtorrent-RC_1_2-head-aa3faed_x64-static-release

Also will backing up settings/fastresumes really help to avoid massive rechecking in case I downgrade back?

[FranciscoPombal]

@reyaz006

Recently, there have also been some recent developments related to improving startup time in #13348

So far this endeavour has lead to the fixing of a bug in libtorrent that was causing massive increases in startup time if torrents had trackers with URLs not ending in /announce.

So please test with the build linked in #13348 (comment). It would be interesting to see whether or not you see an improvement in startup time with these new build, even without having solved the recheck problem as mentioned above.

[reyaz006]

@FranciscoPombal

So please test with the build linked in #13348 (comment). It would be interesting to see whether or not you see an improvement in startup time with these new build, even without having solved the recheck problem as mentioned above.

I tried this first. Observations vs. qBittorrent-nightly-e16aafb_vcpkg-fdac1fc_qt5-lts-5.15.0_libtorrent-RC_1_2-head-aa3faed_x64-static-release:

• Startup time actually increased a bit. If my torrents have trackers with URLs not ending in /announce, I doubt there are many of those.

• RAM usage is really different. With previous 4.3.0a1, it was slowly increasing to ~3.1 GB and stayed there. With this version, it's increasing to ~4 GB within 1-2 minutes from launch and stays there.

• GUI responsiveness is also different. With previous 4.3.0a1, the window stayed blank for the whole duration of the startup. With this version, the window contents become visible at certain moments. It goes like this: 0-2 minutes - reserving/consuming 4 GB RAM, then I see the window where all categories show 0 torrents, then 2-45 minutes - nothing and no response, then for the moment the window updates with my amount of torrents, but immediately becomes unresponsive. I must wait for another 5-10 minutes to be able to click anywhere.

• The way torrents get rechecked seems also different a bit. My impression may be wrong but it felt slower. Also with previous 4.3.0a1, the moment GUI becomes responsive, I already see ~500 torrents in Errored status category, and with this version it stays at 0 for some time, and as torrents get checked it starts increasing at some point.
  (Side note: it feels very uncomfortable not being able to immediately find torrents with "Checking" status. One would expect it to be a separate clickable status category.)

Can you please open a new issue and post a couple of fastresumes + their corresponding torrents for the torrent that are stuck in that recheck loop plus and a couple of fastresumes + their corresponding torrents for torrents that are not affected by the issue? My suspicion is that they will be quite different and the cause of this issue will be exposed by that difference. Alternatively, you can remove them and add them all again... but that is probably quite tedious.

So I identified 2 different torrents.

• First one gets rechecked on each startup. Eventually gets into Errored status category as "Missing Files". Log always mentions 2 lines about it:

'1' restored.
File sizes mismatch for torrent '1', pausing it.

• Second one seem to not being rechecked, and stays in Paused state. Log only mentions 1 line about it:

'2' restored.

Should I upload .torrent and .fastresume files related to both torrents into new issue?

[FranciscoPombal]

@reyaz006

Should I upload .torrent and .fastresume files related to both torrents into new issue?

Yes, please.

As for the rest of your findings, I would suggest the following: backup the fastresumes of the problematic torrents, and remove them from their original location. The goal is to try to get a clean startup shutdown where no torrents enter an error state/recheck loop. Then we can fairly judge possible advancements made in startup performance. The errored fastresumes are a separate issue most likely, that should be handled separately.

[reyaz006]

Done.

As for the rest of your findings, I would suggest the following: backup the fastresumes of the problematic torrents, and remove them from their original location. The goal is to try to get a clean startup shutdown where no torrents enter an error state/recheck loop. Then we can fairly judge possible advancements made in startup performance. The errored fastresumes are a separate issue most likely, that should be handled separately.

Sorry but trying to do anything with that amount of torrents (files) is very problematic. Maybe it would be possible if I could somehow export a list of folder paths from selected torrents.

[FranciscoPombal]

@reyaz006

Sorry but trying to do anything with that amount of torrents (files) is very problematic.

No problem I completely understand. Your collaboration and diligence is much appreciated.

Maybe it would be possible if I could somehow export a list of folder paths from selected torrents.

It's a bit unclear to me what you have in mind, but if you want to extract paths, you can probably do that with the WebAPI and some scripting: https://github.com/qbittorrent/qBittorrent/wiki/WebUI-API-(qBittorrent-4.1)

Look into the torrent list and torrent properties API methods.

[Pentaphon]

Honestly, I get the GUI to freeze with just a few hundred torrents. This is the #1 issue that needs to be addressed.

[reyaz006]

Here is my report after upgrading to 4.3.1 - #13416 (comment)

Okay I decided to just upgrade to 4.3.1.

I see no more issues arise for me regarding random re-checking.

One annoyance remains - it seems whatever state those errored torrents are stuck at, I'm unable to pause them. They will remain in Errored state unless removed or re-downloaded, while I preferred the ability to pause them so they would no longer stay in Errored state. I'm inclined to think I was correct about being able to set torrents into Pause state where they were in Finished/Completed state, even though there was no visual indication of that state change (it came after I restarted the app and it didn't find the files for such torrents).

Another note, startup time was 50 minutes this time. 40k torrents total.

So, not much change GUI-wise. It's still frozen while starting up.

RAM usage seems abnormally small for now, at ~430 MB.

[aminpaks]

Would someone performance test this version of web ui with a couple of thousand items too?

Read more about the progress here.

[puddingsoda]

I'm getting a really bad lag issue on macOS on 4.3.2. The UI lags a lot when scrolling. I checked my CPU usage and it jumps up to 100% whenever I scroll. This leads me to believe that the app is doing something extremely inefficient (maybe refreshing the scroll view many times when scrolling) resulting in this high CPU usage and intense lag.

This is a little disappointing given that qBit is supposed to be a relatively "lightweight" client but is actually eating up CPU and battery when doing basic scrolling in the app. But I don't blame the developers because I'm sure they would have fixed it if they knew the issue.

[puddingsoda]

I don't know how similar this issue is to #12900, which is specific to Macs, but I am mentioning it here just in case it is related in any way. I will also copy and paste a comment I made on that issue.

The UI will start to lag more the bigger the window size of the torrents. If you decrease the number of torrents that can be seen at one time, for example by clicking on the content pane or by making the window smaller, the lag becomes less harsh.

This is likely due to inefficient rendering of the table views which make up the torrent files in the GUI list. I see a CPU spike of 100% when scrolling in qBittorrent [on Mac]. This should not be happening. When I scroll through a list in any other app, it barely breaks 20% CPU usage. I assume this is an issue in qBittorrent simply because it wasn't coded "natively" for the macOS platform.

@FranciscoPombal If you would like me to perform any sort of tests, I would be happy to do so.

[FranciscoPombal]

@puddingsoda Thanks, but I don't have access to macOS. If you can debug/investigate yourself, if you want. Perhaps your findings would help shine light on other issues for other platforms as well.

[sledgehammer999]

@FranciscoPombal I temporarily unpinned this in favor of the Poll issue. It was a random decision. If you want you can unpin #9849 and re-pin this.

[FranciscoPombal]

@sledgehammer999

@FranciscoPombal I temporarily unpinned this in favor of the Poll issue. It was a random decision. If you want you can unpin #9849 and re-pin this.

No problem, understandable. I wish GH allowed a few more pinned issues. Anyway, I've been using the Projects tab to track important/frequent issues. In fact, this one should have been there already. Will add now.

[pinyangcong]

Disk cache set to '256MiB'

[Protektor-Desura]

Well not sure if this is correct place to post but I will say that the qBittorrent web GUI fails on me if I get above like 1200-1300 torrents in the queue. I can add more using the API and it will respond to the add no problem. If I let it fall back down in to the 900-1000 range then the web interface will start working again showing torrents and all the information. If I am above 1200-1300 the web interface comes up but it won't show any torrents or upload/download total speeds. Every thing is just blank. I can still use the search tab. So the web interface isn't totally broken but it just doesn't like to handle a large number of torrents.

This is for qBittorrent v4.3.5 Web UI (64-bit) on Linux.