Projects = ~50 (just over the limit so the dashboard has the group/name filters)
Tenants = 8
Machines = ~150
Server size = 6 x 2.3GHX w/ 64GB RAM
I did review retention policies and updated all our life-cycles to include a phase for PROD so I could drop release retention on lower environments. This appears to have helped with the query timeouts and server start-up time. We still experience high CPU utilization even when no tasks are queued.
Octopus Deploy: Server version 3.7.18 (3.7.18+Branch.master.Sha.6146586b187ed08164b3db8f3becba4f364ad767) instance OctopusServer
Assuming this is correct, I would definitely recommend upgrading. We have made a number of performance improvements since that release.
In particular, we introduced better caching for the dashboard queries, which from inspecting your logs would seem to have good potential to help in this case.
I would love to know the results if you can schedule an upgrade.