You are correct, we have put a focus on performance work recently, but to be honest it was focused on speed, not memory usage.
We'll certainly work with you to help resolve this problem. Can I ask you a few questions:
Which process is consuming the memory? Is it Octopus.Server.exe?
Could you give us an idea of the size of your usage. How many projects, environments, machines? Approximate numbers are fine.
Which version of Octopus were you upgrading from?
If you look at the Tasks tab in the Octopus portal, are there many/any tasks running at the time? How long have they been running for?
The next time you see the memory climb to a high-level, would you be able to follow the steps on this page (in the 'Get a snapshot from your running Octopus Server' section) to capture a memory snapshot and log files?
You can upload them to this secure location. Please notify me via this thread if you upload files to the location above, as we don't get notified automatically.
This information will allow us to investigate further.
We currently have 12 projects, 4 environments, 4 machines, but the vast
majority of our deploys are to Azure.
We originally had 3.11.11 and it seemed to run uninterrupted fine for weeks.
No Tasks running just now.
I tried to get a memory snapshot from it this morning but Octopus had used
up over 4GB of memory and dotMemory either couldn't start or complete a
snapshot so I had to restart the Octopus service. I've left it running for
a few hours now and the memory usage has climbed from ~300MB to about
~1.4GB so I've taken a snapshot and have uploaded it to the link you sent.
Thanks for keeping in touch! Michael is taking some well earned vacation, so I'll be taking over from him. Nice thing is you don't need to remember a new name! :)
Thanks so much for sending through that snapshot - nothing seems to be standing out from the single snapshot. What would really help me get to the bottom of this, and the root cause, is to use the alternative approach where you restart Octopus Server with dotMemory recording allocations from the get-go. Then take two snapshots: one after the Web UI starts responding properly, and another once the memory has started to climb, preferably after some normal operation and deployments. This will reveal to us which memory is being retained, and the root cause of it.
We're in the middle of a roll-out of our latest release just now so I'll
try and get you this information tomorrow. For the moment we have the
service restarting on a nightly basis to prevent this from happening which
seems to have had the desired effect but is not exactly ideal.
I've uploaded the files to the location you previously gave me. Took a
while because dotMemory failed to process a snapshot and then crashed.
Anyway, hopefully whats there now should give you a better idea of whats