Query timeouts and slow server start time

Brian Mahloch's Avatar

Brian Mahloch

12 Sep, 2017 08:50 PM

We've been using our Octopus Deploy server for over 2 years now and are currently at version 3.13.8. In the last couple weeks we've noticed extremely slow response times via the web ui with lots of query timeout error messages. During our last 2 server reboots Octopus server has taken 30 - 40 minutes to come up and start accepting requests (via web ui). Nothing seems out of the ordinary and our projects have remained fairly static over the last 3 months or so (just minor process tweaks here and there). The culprit query timeout occurs when trying to view the overview tab of just about any project:

SELECT * FROM dbo.[MultiTenancyDashboard] WHERE ([ProjectId] = @projectId) ORDER BY [Id]

What's interesting is we only have one project that uses the multi-tenant feature but every project overview fails with the above query timeout.

Other observations:

- Server startup time ~30-40 minutes
- CPU is almost always pegged 100% with Octopus Server by far the heaviest usage even though 0 tasks are running
- Health Checks will run for over 24 hours even though everything comes back healthy

Any ideas what might be wrong or where to start troubleshooting? It's almost becoming unusable in its current state.

Thanks!

  1. Support Staff 1 Posted by Michael Richard... on 13 Sep, 2017 05:17 AM

    Michael Richardson's Avatar

    Hi Brian,

    Those are indeed disconcerting numbers. Let's see if we can determine the cause.

    In case you haven't come across it, we have a documentation page on performance. It is worth reading this to see if anything is applicable.

    To give us an idea of the scale of your instance, could you supply rough counts of:

    • Projects
    • Tenants
    • Machines

    What are your hardware specs for the machine Octopus is running on?

    Would you also be able to attach an Octopus server log? Preferably one which covers the time during a server start.

    I would also recommend inspecting your Retention Policies. Do you have these configured?

    Finally, if are willing to capture a performance trace during a time when your server is experiencing high CPU usage, this would be a big help.

    This information above will hopefully provide some clues to the problem.

    You can upload files (e.g. server logs or performance trace) to this secure location, but please also respond to this thread, as I don't get automatically notified when files are uploaded.

    Regards,
    Michael

  2. 2 Posted by Brian Mahloch on 13 Sep, 2017 04:37 PM

    Brian Mahloch's Avatar

    Hi Michael –

    Here are the answers to your questions:

    Projects = ~50 (just over the limit so the dashboard has the group/name filters)
    Tenants = 8
    Machines = ~150

    Server size = 6 x 2.3GHX w/ 64GB RAM

    I did review retention policies and updated all our life-cycles to include a phase for PROD so I could drop release retention on lower environments. This appears to have helped with the query timeouts and server start-up time. We still experience high CPU utilization even when no tasks are queued.

    Attached is the server log during last startup.

    I will attempt to get a dotTrace ASAP.

    Thanks,

    Brian

  3. Support Staff 3 Posted by Michael Richard... on 14 Sep, 2017 01:53 AM

    Michael Richardson's Avatar

    Brian,

    Thank-you for the logs.

    Your logs suggest you are on version 3.7.18.

    Octopus Deploy: Server version 3.7.18 (3.7.18+Branch.master.Sha.6146586b187ed08164b3db8f3becba4f364ad767) instance OctopusServer
    

    Assuming this is correct, I would definitely recommend upgrading. We have made a number of performance improvements since that release.
    In particular, we introduced better caching for the dashboard queries, which from inspecting your logs would seem to have good potential to help in this case.

    I would love to know the results if you can schedule an upgrade.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac