Umbraco 12 vs Umbraco 13 performance on a variety of mini PCs – 2024 Edition
About a year ago now, I did a talk and blog post comparing various mini PCs for running Umbraco, to try and encourage people to look a bit wider than just the common Raspberry Pi default, as repurposing old machines is much better for sustainability. Back then Umbraco was still on version 12, and running on .net 7. At the end of the year .net 8 was released, with Umbraco 13 based on it following shortly afterwards. With Microsoft touting such big improvements to the performance of .net 8 over .net 7, I’d always vowed to re-test this all to see what a difference could be seen in real world use rather than only skewed benchmarks. At the very start of 2024, I did a very brief test with a single machine (The Raspberry Pi 3B) on Umbraco 13, and that was showing a speed improvement of around 3% which was still an improvement in performance, but considerably less than some of the bigger claims Microsoft had made.
I did finally do the rest of the machines in the Spring of 2024. But as it was initially updated on all the talk slides and I was then tied up on the loop of trying to find a meetup interested in the talk (yes, this sometimes takes a lot longer than you’d realise), it has taken right through until now for me to get the results all written up properly here. As well as re-testing all the original machines with Umbraco 13 on .net 8, with one exception I’ll explain later, I also took the opportunity to add in a few new machines. The first was the new Raspberry Pi 5, released late in 2023 after a 5 year break in Pi releases, during which scalping had become a major issue for the platform for a while, before finally being resolved at the start of this year. And the other was a bigger brother thin client to the Dell Wyse 3040, namely the Dell Wyse 5070. While the smaller 3040 used an Intel Atom-based processor, it’s larger sibling uses a Celeron or Pentium processor. This leaves the full list of machines for the 2024 iteration of this as follows:-
| | Specification | Released | Architecture | RRP |
| Raspberry Pi 3B+ | Broadcom BCM2837B0, Cortex-A53 (ARMv8) 64-bit SoC @ 1.4GHz. 1GB LPDDR2 SDRAM | 2018 | ARM | £35 |
| Raspberry Pi 4B | Broadcom BCM2711, Quad core Cortex-A72 (ARM v8) 64-bit SoC @ 1.8GHz. 1, 2,4 or 8GB RAM – 2GB tested | 2019 | ARM | £45 (new/used) |
| Lenovo M93P Tiny | 4th Generation Intel Core i3 4130T, 4GB/8GB RAM | 2014 | x86/x64 | £40-50 |
| Mecool M1 Pro | Amlogic S905D 64-bit octa-core ARM Cortex-A53 CPU, up to 2 GHz. 2GB DDR4 RAM. Android TV Box with built in Digital terrestrial/Satellite tuners | 2017 | ARM | £25-35 |
| Dell Wyse 3040 | Intel Cherry Trail Atom x5 Z-8350 (1.44 GHz Quad Core). 2GB RAM. 8/16GB of soldered eMMC SSD. | 2017 | x86/x64 | £20-30 |
| Dell Wyse 5070 | Intel Celeron J4105/Pentium J5005 Quad Core, 8GB RAM, 32GB SATA Storage | 2019 | x86/x64 | £40/80 (very variable) |
| Raspberry Pi 5 | Broadcom BCM2712 Quad-core ARM Cortex A76 @ 2.4GHz, 4/8GB RAM | 2023 | ARM | £60-80 |
Methodology
For comparing with the original 2023 results, I stuck with the same versions of software in use for the time, even where some newer minor releases had come out since, something which also becomes important in the results. These were Umbraco 12.1.1 and Usync 12.1.0. To keep consistency, these same old versions were also used for the tests with the two newly added machines. For the 2024 set, this was based off the latest releases available at the start of the year, leaving us with what was then quite an early Umbraco 13.0.3 and Usync 13.0.0. As most people know Umbraco 14 has also hit the market since the summer, however as this is not an LTS release, is still based on .net 8, and will require more significant rewriting of the tests so harder to compare directly, it has not been included in the comparison.
Tests
Five tests were run, and these were identical to the tests run in 2023. It’s a mix of backend tests and frontend output tests, but importantly with the exact same code custom to run these used on both the Umbraco 12/.net 7 and Umbraco 13/.net 8 instances. As a reminder the full set are:-
1) Create 1500 nodes of content using ContentService and publish at the end. Each of these has some randomly generated Lorum Ipsum text in a body and title field.
2) Perform a full export of the 1500 nodes using uSync’s full export option in the back office.
3) Reset the database to an ’empty’ state and then reimport the 1500 nodes using uSync’s full import option.
4) Return a basic json list of the titles of all 1500 nodes, as pulled from Umbraco’s regular cached output layer.
5) Return a more complicated json list of filtered titles. This still uses Umbraco’s cached output layer, but attempts to put more complexity into the process with a bunch of ugly arbitrary code.
Timings were taken 3 times at least and then averaged.
Results
The original 2023 results are shown alongside the 2024 results.
Raspberry Pi 3B+
This one unfortunately posed the biggest problems. After running the first test twice, and seeing an improvement of around 3%, the whole machine slowed down massively. Despite swapping SD cards, power supplies and reinstalling operating systems, nothing could restore the speed, so I had to conclude the hardware had failed and withdraw it from the rest of the testing suite and results.
Power Consumption: Idle 2/3w, Load 3/4w
| Umbraco 12/.net 7 | Umbraco 13/.net 8 |
| #1. 15:15, 15:09, 14:45 – Average – 15:03 | 14m40, 14m46 … – Average – 14:43 |
| #2. 1:36, 1:35, 1:36 – Average – 1:36 | – |
| #3. 5:27, 5:30, 5:29 – Average – 5:29 | – |
| #4. 6ms, 7ms, 6ms – Average – 6ms | – |
| #5. 955, 1106, 831, 872, 912, 885, 934, 937, 881 – Average – 923ms | – |
Raspberry Pi 4B
Power Consumption: Idle 2/3w, Load 4/5w
| Umbraco 12/.net 7 | Umbraco 13/.net 8 |
| #1. 9:47, 9:50, 9:43 – Average – 9:46 | 8:29, 8:31, 8:27 – Average – 8:29 |
| #2. 0:54, 0:54, 0:56 – Average – 0:55 | 0:46, 0:46, 0:46 – Average – 0:46 |
| #3. 3:26, 3:11, 3:28 – Average – 3:22 | 4:07, 4:06, 4:04 – Average – 4:05 |
| #4. 4ms, 3ms, 2ms – Average – 3ms | 3ms, 3ms, 2ms – Average – 3ms |
| #5. 634, 641, 506, 579, 580, 534, 526, 540, 559 – Average – 567ms | 473, 489, 491, 523, 485, 523, 503, 486, 501 – Average – 497ms |
Lenovo ThinkCentre M93P Tiny
Power Consumption: Idle 10/11w, Load 18/21w
| Umbraco 12/.net 7 | Umbraco 13/.net 8 |
| #1. 2:27, 2:28, 2:36 – Average – 2:30 | 2:17, 2:16, 2:16 – Average – 2:16 |
| #2. 0:15, 0:15, 0:14 – Average – 0:15 | 0:12, 0:12, 0:12 – Average – 0:12 |
| #3. 0:50, 0:46, 0:45 – Average – 0:47 | 1:07, 1:05, 1:05 – Average – 1:05 |
| #4. 1ms, 1ms, 1ms – Average – 1ms | 1ms, 1ms, 1ms – Average – 1ms |
| #5. 141, 141, 97, 95, 97, 96, 104, 112, 133 – Average – 112ms | 74, 81, 105, 119, 74, 72, 86, 78, 103 – Average – 88ms |
Mecool M1 Pro
Power Consumption: Idle 3/4w, Load 4/5w
| Umbraco 12/.net 7 | Umbraco 13/.net 8 |
| #1. 15:02, 15:03, 15:08 – Average – 15:04 | 12:50, 12:33, 12:40 – Average – 12:41 |
| #2. 1:36, 1:36, 1:35 – Average – 1:36 | 1:27, 1:22, 1:21 – Average – 1:23 |
| #3. 5:47, 5:40, 5:35 – Average – 5:41 | 7:50, 7:38, 7:38 – Average – 7:42 |
| #4. 6ms, 5ms, 4ms – Average – 5ms | 4ms, 4ms, 5ms – Average – 4ms |
| #5. 835, 798, 969, 1130, 795, 794, 845, 807, 816 – Average – 865ms | 589, 682, 680, 560, 560, 543, 566, 554, 625 – Average – 595ms |
Dell Wyse 3040
Power Consumption: Idle 3/4w, Load 4/5w
| Umbraco 12/.net 7 | Umbraco 13/.net 8 |
| #1. 12:27, 12:45, 12:42 – Average – 12:38 | 11:32, 11:17, 11:19 – Average – 11:22 |
| #2. 1:01, 1:05, 1:07 – Average – 1:04 | 0:55, 0:57, 0:55 – Average – 0:56 |
| #3. 3:39, 3:28, 3:18 – Average – 3:28 | 5:00, 4:55, 4:39 – Average – 4:51 |
| #4. 3ms, 3ms, 4ms – Average – 3ms | 3ms, 3ms, 4ms – Average – 3ms |
| #5. 463, 405, 406, 483, 462, 427, 458, 462, 442 – Average – 445ms | 327, 363, 361, 352, 335, 365, 362, 324, 359 – Average – 349ms |
Dell Wyse 5070
Power Consumption: Idle 4/5w, Load 8/9w
| Umbraco 12/.net 7 | Umbraco 13/.net 8 |
| #1. 4:38, 4:34, 4:39 – Average – 4:37 | 4:15, 4:14, 4:15 – Average – 4:15 |
| #2. 0:23, 0:23, 0:23 – Average – 0:23 | 0:20, 0:20, 0:20 – Average – 0:20 |
| #3. 1:12, 1:13, 1:12 – Average – 1:12 | 1:47, 1:48, 1:48 – Average – 1:48 |
| #4. 1ms, 1ms, 1ms – Average – 1ms | 1ms, 1ms, 1ms – Average – 1ms |
| #5. 242, 198, 197, 201, 257, 215, 199, 206, 218 – Average – 214ms | 160, 175, 166, 136, 172, 183, 136, 137, 154 – Average – 158ms |
Raspberry Pi 5
Power Consumption: Idle 3/4w, Load 7/8w
| Umbraco 12/.net 7 | Umbraco 13/.net 8 |
| #1. 3:21, 3:20, 3:21 – Average – 3:21 | 3:00, 3:01, 3:00 – Average – 3:00 |
| #2. 0:18, 0:19, 0:18 – Average – 0:18 | 0:16, 0:17, 0:16 – Average – 0:16 |
| #3. 1:01, 1:02, 1:02 – Average – 1:02 | 1:26, 1:25, 1:27 – Average – 1:26 |
| #4. 1ms, 1ms, 1ms – Average – 1ms | 1ms, 1ms, 0ms – Average – 1ms |
| #5. 152, 183, 149, 184, 156, 212, 169, 151, 175 – Average – 170ms | 120, 120, 153, 141, 120, 139, 152, 131, 121 – Average – 133ms |
Initial Conclusions
Whereas initial results had been showing a very small improvement in speed on the Raspberry Pi 3, tests on subsequent machines showed a much bigger improvement between Umbraco 12/.net 7 and Umbraco 13/.net 8, in the region of 10-15%. Notably the machine which had shown similar performance in 2023 to the Pi 3, the Mecool M1 Pro, showed much bigger gains with the 2024 tests. Given the later failure of the Pi 3, it’s not unreasonable to assume that even that low 3% improvement was as a result of failing hardware and that much bigger gains have indeed come about.
The eagle-eyed among you will have spotted one exception to the rule. The third test, where we do a uSync Full Import, shows a noticeable slowdown in performance across every machine in the region of about 20%-30%. Given all the improved results elsewhere, this one was a surprise and definitely warranted some further digging. The logical first step was to re-run the original Umbraco 12/.net 7 tests and confirm they were still showing the same lower times on the test machines, which indeed they were. As mentioned this was on some older minor versions of Umbraco 12 and uSync though, so the next step was to update them to the latest versions available in the spring to rule this out. With the latest minor versions of both Umbraco/uSync 12 and Umbraco/uSync 13, the Full Restore test also showed the same slowdown performance in both the 12 and 13 version trees. I’ve not been able to find out exactly which version this started happening in, but it seems to suggest a change has come about in a version which has had an impact on the performance of the restore. While I’ve not yet gotten to the bottom of it yet, this is something I intend to dig into further when I have time, and reach out to uSync’s author Kevin just to see if it is something known about. Given it’s taken me so long to write this up, it’s possible it’s even already something which has been found and changed in some of the subsequent months of this year.
The Medal Table
As done in 2023, I’ve used the test results to build up a 2024 ranking on which machine I’d recommend to get the best balance of performance, cost and sustainability. As before, this does have some level of opinion involved, and works off the following assumptions:-
- Scores are all awarded using a simple approach of ‘Green, Good, +1’, ‘Amber, Okay, +0.5’, ‘Red, Bad, -1’. (Alternative title tags are also available on each table cell to indicate these scores, so as to not depend solely on the visual colour)
- Daily power usage is based off an assumption of running at 22 hours idle, 2 hours load. Given the number of active business hours in a typical day, and then what fraction of those you’re likely to be hitting Umbraco heavily this seemed a good time average. This is only to provide balanced testing though – in practical use a faster machine should finish its heavier loads faster before dropping back to idle state.
- Carbon calculations are based off a UK average of 0.26kg/kWh. Carbon intensity will of course vary across the course of every day based upon how the mix of energy within the grid is being generated at a given point in time so this can only ever be a guide. This figure was however a UK average for 2022. Sources: https://ourworldindata.org/energy, https://github.com/owid/energy-data
- Energy pricing is based off 30p/kWh, which was around the standard UK price cap as of October 2023.
- Manufacturing carbon cost for a new device has been calculated at around 50kg. It’s very difficult to get exact figures on mini PCs, however a lot of research has been done around smartphones, with 80kg coming out as a common average. As a mini PC will share many of the same components as a smartphone, 50kg seems a sensible calculation after removing the battery and screen. Sources: Deloitte, 8BillionTrees – https://8billiontrees.com/carbon-offsets-credits/carbon-footprint-of-iphone
2023
| Price | kWh/year / Cost / CO2 | Mfr CO2 | Test #1-3 (BE) | Test #4-5 (FE) | Total | |
| Pi 3B+ | £35 | 27 / £8 / 7.02kg | 0 | 4th | 5th | 2 |
| Pi 4B | £45 | 27.7 / £8.31 / 7.2kg | 50kg | 2nd | 3rd | 0 |
| M93P | £40 | 94.9 / £28.47 / 24.7kg | 0 | 1st | 1st | 2.5 |
| K1 Pro | £30 | 27.7 / £8.31 / 7.2kg | 0 | 5th | 4th | 2 |
| Wyse 3040 | £25 | 35.8 / £10.74 / 9.3kg | 0 | 3rd | 2nd | 3.5 |
2024
| Price | kWh/year / Cost / CO2 | Mfr CO2 | Test #1-3 (BE) |
Test #4-5 (FE) | Total | |
| Pi 4B | £45 | 27.7 / £8.31 / 7.2kg | 0 | 4th | 5th | 2 |
| M93P | £40 | 94.9 / £28.47 / 24.7kg | 0 | 1st | 1st | 2.5 |
| K1 Pro | £30 | 27.7 / £8.31 / 7.2kg | 0 | 6th | 6th | 2 |
| Wyse 3040 | £25 | 35.8 / £10.74 / 9.3kg | 0 | 5th | 4th | 3.5 |
| Wyse 5070 | £40 / 80 | 46.7 / £14.02 / 12.1kg | 0 | 3rd | 3rd | 4 / 2.5 |
| Pi 5 | £60 | 38.0 / £11.40 / 9.9kg | 50kg | 2nd | 2nd | 0.5 |
Final Conclusions
Looking first at just the performance improvements between the versions, and ignoring the unreliable Raspberry Pi 3 results, the general conclusion seems to be that going from Umbraco 12 and .net 7 to Umbraco 13 and .net 8 without any other changes to code results in a performance improvement of around 10-15% on average. Obviously this is something that would vary depending on exactly what is in your solution, but it suggests a lot of those Microsoft performance blog posts were onto something and its an upgrade worth making as this sort of improvement allows for getting even more out of older and lower powered machines.
Machine-wise, in 2023 the Dell Wyse 3040 won out for its overall balance of cost, performance and sustainability. For 2024, the Raspberry Pi 4 scores have changed slightly, though not enough to take it to the lead. As this machine is now available much more widely on the second hand market, albeit still coming in for around the same cost as new as of my latest checks, it means if you pick it up this way it can now avoid having the big penalty on sustainability from being a ‘new’ device. Instead this problem has moved to the newer generation Raspberry Pi 5. Though manufacturers are all touting improvements to their manufacturing processes to try and get towards net zero now as they are aware sustainability is important to customers, it remains true that manufacturing a new device will always still retain a bigger impact than repurposing an existing device. The performance improvements of the Raspberry Pi 5 over the Raspberry Pi 4 are quite significant coming in at nearly 3x the speed of its previous generation, with a small increase in power consumption, but unfortunately a much bigger increase in overall price.
With all things taken into consideration, the best result I came to for 2024 remains the Dell Wyse series though. This can cover either model, and it’s price on the day which makes the ultimate difference. On the used market, the larger Dell Wyse 5070 model has varied considerably in price every time I’ve checked. It can regularly be found for £35-40, and if obtained at this price is a steal and scores as the overall winner. However quite often this price spikes to nearer £80-90. At this higher end price it should be avoided, and then it would lose out to its original smaller sibling once again, the Dell Wyse 3040, which has a much more stable low price.
