Server Memory Usage

colin's Avatar

colin

05 Apr, 2017 04:05 PM

Hi,

We recently had to move Octopus to a new VM and during that process we installed v 3.11.X. After running for approx 1 day the server memory usage hit nearly 4GB (the box has 7GB RAM) and was still climbing. The VM eventually became unusable and we had to reboot it. This has happened a few times over the last few days. We upgraded to 3.12.0 as it mentioned some performance improvements but the same thing is still happening. Is this a known issue? How do I go about diagnosing the problem?

Thanks
Colin

  1. Support Staff 1 Posted by Michael Richard... on 06 Apr, 2017 05:16 AM

    Michael Richardson's Avatar

    Hi Colin,

    No, this is not a problem known to us.

    You are correct, we have put a focus on performance work recently, but to be honest it was focused on speed, not memory usage.

    We'll certainly work with you to help resolve this problem. Can I ask you a few questions:

    Which process is consuming the memory? Is it Octopus.Server.exe?

    Could you give us an idea of the size of your usage. How many projects, environments, machines? Approximate numbers are fine.

    Which version of Octopus were you upgrading from?

    If you look at the Tasks tab in the Octopus portal, are there many/any tasks running at the time? How long have they been running for?

    The next time you see the memory climb to a high-level, would you be able to follow the steps on this page (in the 'Get a snapshot from your running Octopus Server' section) to capture a memory snapshot and log files?
    You can upload them to this secure location. Please notify me via this thread if you upload files to the location above, as we don't get notified automatically.

    This information will allow us to investigate further.

    Regards,
    Michael

  2. 2 Posted by colin on 06 Apr, 2017 12:33 PM

    colin's Avatar

    Hi Michael,

    The process is Octopus.Server.exe.

    We currently have 12 projects, 4 environments, 4 machines, but the vast
    majority of our deploys are to Azure.

    We originally had 3.11.11 and it seemed to run uninterrupted fine for weeks.

    No Tasks running just now.

    I tried to get a memory snapshot from it this morning but Octopus had used
    up over 4GB of memory and dotMemory either couldn't start or complete a
    snapshot so I had to restart the Octopus service. I've left it running for
    a few hours now and the memory usage has climbed from ~300MB to about
    ~1.4GB so I've taken a snapshot and have uploaded it to the link you sent.

    Thanks
    Colin

  3. Support Staff 3 Posted by Michael Noonan on 10 Apr, 2017 01:08 AM

    Michael Noonan's Avatar

    Hi Colin,

    Thanks for keeping in touch! Michael is taking some well earned vacation, so I'll be taking over from him. Nice thing is you don't need to remember a new name! :)

    Thanks so much for sending through that snapshot - nothing seems to be standing out from the single snapshot. What would really help me get to the bottom of this, and the root cause, is to use the alternative approach where you restart Octopus Server with dotMemory recording allocations from the get-go. Then take two snapshots: one after the Web UI starts responding properly, and another once the memory has started to climb, preferably after some normal operation and deployments. This will reveal to us which memory is being retained, and the root cause of it.

    https://octopus.com/docs/how-to/record-a-memory-trace#start-octopus...

    You can upload the zipped workspace export to the same secure file share.

    Hope that helps!
    Mike

  4. 4 Posted by colin on 18 Apr, 2017 12:48 PM

    colin's Avatar

    Hi Michael,

    We're in the middle of a roll-out of our latest release just now so I'll
    try and get you this information tomorrow. For the moment we have the
    service restarting on a nightly basis to prevent this from happening which
    seems to have had the desired effect but is not exactly ideal.

    Cheers
    Colin

  5. Support Staff 5 Posted by Michael Noonan on 18 Apr, 2017 11:44 PM

    Michael Noonan's Avatar

    Hi Colin,

    Thanks for keeping in touch! I hope your release goes well. :)

    It will be really good to get a reproducible recording of this problem. We don't want anyone to require a routine server restart as part of their Octopus maintenance plan!

    Looking forward to hearing from you soon.
    Mike

  6. 6 Posted by colin on 19 Apr, 2017 03:30 PM

    colin's Avatar

    Hi Michael,

    Release went fine thanks.

    I've uploaded the files to the location you previously gave me. Took a
    while because dotMemory failed to process a snapshot and then crashed.
    Anyway, hopefully whats there now should give you a better idea of whats
    going on.

    Cheers
    Colin

    Colin Winning

    Technical Director

  7. 7 Posted by colin on 25 Apr, 2017 03:31 PM

    colin's Avatar

    Hi,

    I was just wondering if there was any progress on this? Or if I can provide any more information to help you diagnose what is going on?

    Cheers
    Colin

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions

25 Apr, 2017 04:08 PM
25 Apr, 2017 04:04 PM
25 Apr, 2017 03:43 PM
25 Apr, 2017 03:39 PM
25 Apr, 2017 03:31 PM

 

25 Apr, 2017 03:28 PM
25 Apr, 2017 03:11 PM
25 Apr, 2017 02:54 PM
25 Apr, 2017 02:27 PM
25 Apr, 2017 01:17 PM
25 Apr, 2017 11:14 AM