Evaluating browser-performance limitations
I recently collected a bunch of browser benchmark measurements on Sculpt OS to get an idea of what factors might be limiting the browsing performance.
For over two years now, I have been using Sculpt OS on a daily basis, yet still dependent on a Linux VM. Once in a while, I’m thinking about how to move more things out of my VM. For me personally, there are only very few missing pieces that impede the complete migration to Sculpt OS (i.e. without running a Linux VM) for daily non-development tasks. A performant web browser is, in my opinion, the most important ingredient for this.
As a Nextcloud user, I am able to access files, take notes, look into my calendar and write emails via the web browser. There are currently two browsers that run natively on Sculpt: Falkon and Morph Browser. Yet, accessing Nextcloud in particular seems to overstrain these browsers.
A few months ago, I observed this issue for QtWebengine-based browsers even when running within a Linux VM (see #4776). Back then, I suspected the missing GPU acceleration. This week, I was playing with the PinePhone, tried and failed to access my Nextcloud and got back to this issue once again.
In an effort of identifying the performance bottleneck, I benchmarked the available browsers in different scenarios.
Results on PC
I ran all benchmark suites from browserbench.org with Falkon and Morph Browser on Sculpt 23.04 (Thinkpad T490s and PinePhone). The JetStream benchmark evaluates JavaScript performance, MotionMark evaluates graphics performance, and Speedometer measures responsiveness of web apps. Note that I had to increase the RAM quota for both browsers (I ended up with 4G).
For a baseline, I first ran the benchmark suites on my Thinkpad running ArchLinux. I compared Falkon and Firefox running natively and within an Ubuntu VM in VirtualBox 7.0.8.
| (baseline) | JetStream 2.1 | MotionMark 1.2 | Speedometer 2.1 |
|---|---|---|---|
| Falkon@Linux: | 116.661 | 453.85 ±2.19% | 97.8 |
| Falkon@Linux-VM: | 79.994 | 1.09 ±8.3% | 56.9 |
| Firefox@Linux: | 104.550 | 710.13 ±15.07% | 100 |
| Firefox@Linux-VM: | 64.626 | 69.35 ±10.78 | 75.8 |
Table 1: Results on a Thinkpad T490s (running Linux) with native and virtualised Linux
There is a significant degradation for MotionMark when running within a VM. This can be explained by the disabled hardware acceleration, yet Firefox seems to do a much better job at software rendering.
Next, I booted Sculpt 23.04 and ran the benchmarks on Falkon and Firefox within my virtualised ArchLinux.
| (Sculpt 23.04) | JetStream 2.1 | MotionMark 1.2 | Speedometer 2.1 |
|---|---|---|---|
| Falkon@Linux-VM: | 70.500 | 1.09 | 59.7 |
| Firefox@Linux-VM: | 60.000 | 20.85 ±240.65% | 51.2 |
Table 2: Results on a Thinkpad T490s (running Sculpt 23.04) with virtualised Linux
The results are pretty similar to what I've seen when running a Linux-VM on top of my natively booted ArchLinux. The only notable difference is that Firefox' software rendering seems to be slower with Sculpt and shows a much higher variance.
Last, I put the ports of Falkon and Morph (running natively on Sculpt) to a test. Note that both ports are also available with jemalloc as an allocator. The port of Morph Browser was originally intended for the PinePhone. In contrast to Falkon, it is able to utilise the GPU.
| (Sculpt 23.04) | JetStream 2.1 | MotionMark 1.2 | Speedometer 2.1 |
|---|---|---|---|
| Falkon | n/a | 1.5 ±102.50% | 47.3 |
| Falkon-jemalloc | n/a | 36.35 ±121.57% | 56.6 |
| Morph | n/a | 3.46 ±161.36% | 62.5 |
| Morph-jemalloc | n/a | 96.66 ±57.26% | 74.4 |
Table 3: Results on a Thinkpad T490s with Falkon and Morph running natively on Sculpt 23.04.
Unfortunately, I was not able to acquire results for JetStream 2.1. After increasing the RAM quota, the jemalloc variants got stuck at the segmentation test when attempting invalid writes/reads to/from memory. The other variants failed at the first-inspector-code-load with an unresolvable exception.
As expected, Morph performs much better at MotionMark due to the hardware acceleration. I was also surprised to see that the jemalloc variants achieved much better results. Interestingly, there seems to be a high variance for MotionMark on Sculpt. Only looking at the numbers, the best option seems to be Morph with jemalloc, which showed the best results for MotionMark and Speedometer.
Despite the difference in benchmark scores for these four browser variants, my Nextcloud use case feels equally slow/fast on all browsers. Overall, it is usable but feels a quite unresponsive at certain points.
I am going to regularly update this page with new (Sculpt) releases.
Update 2023-10-16
When comparing results with the upcoming Sculpt 23.10 release, I noticed that I was running the benchmark on Sculpt 23.04 with intel_hwp_balanced settings for bender. This setting does not seem to make a difference for the Speedometer benchmark, however, it significantly changes the results for MotionMark. I thus re-run MotionMark with intel_hwp_performance instead.
| (Sculpt 23.04) | MotionMark (balanced) | MotionMark (performance) |
|---|---|---|
| Falkon | 1.5 ±102.50% | 10.85 ±141.76% |
| Falkon-jemalloc | 36.35 ±121.57% | 135.76 ±21.42% |
| Morph | 3.46 ±161.36% | 24.89 ±58.19% |
| Morph-jemalloc | 96.66 ±57.26% | 100.35 ±23.07% |
| Firefox@Linux-VM | 20.85 ±240.65% | 186.82 ±8.8% |
Table 4: Comparison of MotionMark results for intel_hwp_balanced and intel_hwp_performance on Sculpt 23.04.
Setting the CPU into performance state reduces the variance of the scores (except for Falkon). In particular, the result for Firefox@Linux-VM is remarkable since it is significantly better than what I was able to achieve when running the VM on Linux instead of Sculpt. Moreover, it's surprising to see that Falkon-jemalloc even achieves a higher score than Morph-jemalloc despite the fact that Falkon lacks GPU support.
Update 2023-10-26 (Sculpt 23.10)
With the release of Sculpt 23.10, I re-run the benchmarks on my T490s with the intel_hwp_performance settings. I used the 2023-10-19 versions of Falkon and Morph.
| (Sculpt 23.10) | JetStream 2.1 | MotionMark 1.2 | Speedometer 2.1 |
|---|---|---|---|
| Falkon@Linux-VM | 72.500 | 1.09 | 58.7 |
| Firefox@Linux-VM | 62.500 | 196.96 ±7.04% | 64.5 |
| Falkon | n/a | 27.78 ±37.03% | 48.9 |
| Falkon-jemalloc | n/a | 148.60 ±12.45% | 57.6 |
| Morph | n/a | 46.36 ±12.89% | 65.3 |
| Morph-jemalloc | n/a | 152.03 ± 6.81% | 70.4 |
Table 5: Results on a Thinkpad T490s running Sculpt 23.10.
Compared to 23.04, the native Falkon and Morph browsers achieve significantly better scores. The Speedometer scores are very similar. For the native browsers, JetStream seems to be unable to pass the loading phase. With Morph-jemalloc, I was able to pass this a few times, yet, as before, the suite failed at the first-inspector-code-load test with an unresolvable exception.
Update 2024-04-26 (Sculpt 24.04)
With the release of Sculpt 24.04, I re-run the benchmarks on my T490s with the intel_hwp_performance settings. I used the 2024-04-16 versions of Falkon and Morph. Unfortunately, I also had to re-run the benchmarks on 23.10 since the versions of MotionMark and Speedometer have changed. I also noticed that I got different scores for JetStream.
| (Sculpt 23.10) | JetStream 2.1 | MotionMark 1.3 | Speedometer 3.0 |
|---|---|---|---|
| Falkon@Linux-VM | 97.345 | 1.00 | hangs @ 33/580 |
| Firefox@Linux-VM | 88.311 | 285.71 ±39.15% | 4.74 ± 0.21 |
| Falkon-jemalloc | ERROR | 177 ±16.03% | hangs at 1/580 |
| Morph-jemalloc | ERROR | 196 ±14.01% | hangs at 1/580 |
Table 6: New results on a Thinkpad T490s running Sculpt 23.10
| (Sculpt 24.04) | JetStream 2.1 | MotionMark 1.3 | Speedometer 3.0 |
|---|---|---|---|
| Falkon@Linux-VM | 80.641 | 1.00 ±9.05% | hangs @ 33/580 |
| Firefox@Linux-VM | 71.500 | 211.85 ±9.14% | 4.00 ± 0.35 |
| Falkon-jemalloc | ERROR | 160.94 ±17.01% | hangs at 1/580 |
| Morph-jemalloc | ERROR | 168.65 ±19.12% | hangs at 1/580 |
Table 7: Results on a Thinkpad T490s running Sculpt 24.04
Looking at the MotionMark results, there appears to be a small degradation from Sculpt 23.10 to 24.04. For the native browsers, JetStream is still unable to pass the loading phase. The new Speedometer seems to be incompatible with our version of QtWebEngines; it only completes in Firefox.
Update 2024-10-16 (Sculpt 24.10)
With the release of Sculpt 24.10, I re-run the benchmarks on my T490s with the intel_hwp_performance settings. I used the 2024-10-08 versions of Falkon and Morph Browser. Within my Linux VM, I am using Falkon 24.08.0 with QtWebEngine 6.7.2 and Firefox 129.0.2. The VM runs Linux kernel 6.6.47 (Archlinux linux-lts).
I had to re-run the benchmarks on 24.04 due to version updates of MotionMark and JetStream. I also noticed that the frame-rate detection of MotionMark changed. For all scenarios except for Falkon under Linux, MotionMark detected 60fps. For Falkon under Linux, it detected 45fps.
Note that JetStream and Speedometer are still incompatible with our version of QtWebEngine. This will be addressed by our ongoing upgrade from Qt5 to Qt6.
| (Sculpt 24.04) | JetStream 2.2 | MotionMark 1.3.1 | Speedometer 3.0 |
|---|---|---|---|
| Falkon@Linux-VM | 100.121 | 1.27 ±75.32% | 4.26 ± 0.20 |
| Firefox@Linux-VM | 79.172 | 273.38 ± 7.78% | 6.24 ± 0.25 |
| Falkon-jemalloc | n/a | 163.97 ±20.03% | n/a |
| Falkon | n/a | 33.43 ±28.00% | n/a |
| Morph-jemalloc | n/a | 173.27 ±11.18% | n/a |
Table 8: New results on a Thinkpad T490s running Sculpt 24.04
| (Sculpt 24.10) | JetStream 2.2 | MotionMark 1.3.1 | Speedometer 3.0 |
|---|---|---|---|
| Falkon@Linux-VM | 95.175 | 2.04 ±50.61% | 4.00 ± 0.08 |
| Firefox@Linux-VM | 71.750 | 224.06 ± 9.98% | 5.86 ± 0.22 |
| Falkon-jemalloc | n/a | 152.82 ±20.52% | n/a |
| Falkon | n/a | 30.80 ±35.86% | n/a |
| Morph-jemalloc | n/a | 136.12 ±14.29% | n/a |
Table 9: Results on a Thinkpad T490s running Sculpt 24.10
Results on PinePhone
I also tried running the benchmarks on the PinePhone running Sculpt 23.04 and Morph browser. Unfortunately, it was unable to run JetStream and MotionMark.
The browser got stuck when loading the JetStream suite. I haven't looked into it yet. Since I had to increase the RAM quota on PC to run JetStream, I suspect the browser got out of RAM on the PinePhone as well.
The MotionMark benchmark requires landscape mode which is not (yet) available on the PinePhone version of Sculpt OS.
For the Speedometer test, I got a score of 4.888, which seems pretty low. However, compared to my (dated) Android 11 phone (a Pixel 1) for which I got a score of 17.4, we are not too far off considering that the PinePhone's CPU is much slower.
Epilogue
There is certainly room for performance improvements of our Falkon and Morph browser ports. In particular, the failure to run the JetStream test suite indicates that there could be problems on particular websites. I am still wondering about what is limiting the responsiveness of Nextcloud.
Note that I do not regard the browser as a silver bullet for migrating to Sculpt. Hosting individual web apps in different browser tabs does not feel as secure as running separate components. I would still favour using a ported mail client on Sculpt rather than a webmail app. Another item on my wish list is a password manager. I do not consider a browser-provided password manager as secure since I must assume it could be accessed via an exploit. Placing a text file in the file vault would be an option but is not very convenient in my opinion.