Powershell Web-Request / New-WebServiceProxy timeout via Octopus

Alexander Harris's Avatar

Alexander Harris

03 Feb, 2017 12:29 PM

I've been working trying to get our SSRS deployment in Octopus. I package the reports, and have been deploying them with the "Deploy SSRS Reports from a packge" step template from the community library. This "works" except for being very slow and giving us timeout errors about 50% of the time.

Digging into it, the call that times out is: New-WebServiceProxy.

We've done a LOT of investigation here including: trying different instnaces, spinning up a new SSRS instance on a fresh server, pre-warming the web server etc. but whatever we try we can't get past this. The thing that is particularly frustrating is that the same basic code seems to work fine when running locally (on my development machine etc.) and even when running manually on the deployment tentacle.

I was working on alternative solutions to the New-WebServiceProxy call, trying to first download the .wsdl file and then process it from disk, I've created the following simplest possible demonstration script as an example. This is what we are seeing at present:

Invoke-WebRequest -URI https://ssrs-server,domain/ReportServer/reportservice2010.asmx?wsdl -UseDefaultCredential -UseBasicParsing -OutFile ReportService2010.wsdl

  • On my local machine (Powershell terminal, logged in as myself) - runs near instantly.
  • On the deployment tentacle (Powershell terminal, logged in as myself) - runs near instantly.
  • On the deployment tentacle (via Octopus Deploy Tasks --> Script Console, runs as the local system account) - > 3 minutes.

This length of time is far too long and seems to be where the timeouts are coming from. It also will slow down deployments hugely.

We've put a lot of time into this now and have run out of ideas. Any help would be much appreciated. We'd really like to stick with the powershell only approach since it's by far the most portable, though if we have to we could use external command line tooling.

Complete log from one execution of this task is below (NB: in this execution I had a couple of additional Write-Out statementes before and after, and a New-Artefact so I could verify the output):

Task ID: ServerTasks-2879
Task status: Success
Task queued: 03 February 2017 12:10
Task started: 03 February 2017 12:10
Task duration: 4 minutes
Server version: 3.5.1+Branch.master.Sha.4feefbec21f18bc1b1908b5721dd03570fec86a6

                | == Success: Script run from management console ==
                | 
                |   == Success: Run script on: [ServerName] ==

12:10:10 Verbose | Octopus Deploy: Calamari version 0.0.0
12:10:11 Verbose | Name Value
12:10:11 Verbose | ---- -----
12:10:11 Verbose | PSVersion 3.0
12:10:11 Verbose | WSManStackVersion 3.0
12:10:11 Verbose | SerializationVersion 1.1.0.1
12:10:11 Verbose | CLRVersion 4.0.30319.42000
12:10:11 Verbose | BuildVersion 6.2.9200.16481
12:10:11 Verbose | PSCompatibleVersions {1.0, 2.0, 3.0}
12:10:11 Verbose | PSRemotingProtocolVersion 2.2
12:10:11 Info | Making Request
12:14:16 Info | Web Request Done
12:14:16 Verbose | Collecting artifact ReportService2010.wsdl
12:14:16 Info | Exit code: 0

  1. 1 Posted by Alexander Harri... on 03 Feb, 2017 12:32 PM

    Alexander Harris's Avatar

    If possible, please edit the post above to remove the server name from the "Success: Run script on:" line.

  2. 2 Posted by shawn.sesna on 03 Feb, 2017 04:36 PM

    shawn.sesna's Avatar

    Greetings Alexander! The only delay I've ever seen on my end has been when the instance needs to be spun up, all other requests/deployments work normally after that. I noted that when using the tentacle, it uses the system account. The template allows for the use of alternate credentials, does it behave the same way if you use yours here?

  3. 3 Posted by Alexander Harri... on 03 Feb, 2017 04:46 PM

    Alexander Harris's Avatar

    In addition to the above: I created a test C# console app which uses web service references to interact with the SSRS web services. The console app has the proxy classes pre-generated (and the URL to connect to is passed in). For the test, it just lists the contents of a folder.

    Testing this in the same scenarios as above (local machine, deployment machine as my own account, deployment machine running via octopus) again I'm seeing much longer runtimes via Octopus. Could this be actually unrelated to SSRS? Is it to do with the Powershell environment and the account running the code? Sample code below:

    Then one static class as below:

        class Program
        {
            static void Main(string[] args)
            {
                // Setup the variables.
                string serverAddress = args[0];
                string serverFolder = args[1];
                string reportService2010Url = $"{serverAddress}/ReportService2010.asmx";
                string reportExecutionServiceUrl = $"{serverAddress}/ReportExecution2005.asmx";
               //
                // Setup the reportingService2010
                Console.WriteLine($"Setting ReportingService2010 Uri to {reportService2010Url}.");
                var reportingService2010 = new ReportingService2010.ReportingService2010();
                reportingService2010.Url = reportService2010Url;
                reportingService2010.UseDefaultCredentials = true;
               //
                // Setup the reportExecutionService
                Console.WriteLine($"Setting ReportExecutionService Uri to {reportExecutionServiceUrl}.");
                var reportExecutionService = new ReportExecutionService.ReportExecutionService();
                reportExecutionService.Url = reportExecutionServiceUrl;
                reportExecutionService.UseDefaultCredentials = true;
               //
                Console.WriteLine("Listing all reports.");
                foreach (var item in reportingService2010.ListChildren(serverFolder, true))
                {
                    Console.WriteLine(item.Path);
                }
            }
    

    Called from powershell in all 3 configs as:

    & 'C:\Users\Alexander\Desktop\SSRS Deploy\SSRS.exe' https://ssrs-server/reportserver /PathToList

    As you can see from the log below, while completing in a second or 2 on my machine, and in raw powershell on the same server, via Octopus it's taking over 30 seconds for this 1 operation.

    Task ID: ServerTasks-2889 Task status: Success Task queued: 03 February 2017 16:30 Task started: 03 February 2017 16:30 Task duration: 47 seconds Server version: 3.5.1+Branch.master.Sha.4feefbec21f18bc1b1908b5721dd03570fec86a6

                    | == Success: Script run from management console ==
                    | 
                    |   == Success: Run script on: [SERVER] ==
    

    16:30:10 Verbose | Octopus Deploy: Calamari version 0.0.0
    16:30:10 Verbose | Name Value
    16:30:10 Verbose | ---- -----
    16:30:10 Verbose | PSVersion 3.0
    16:30:10 Verbose | WSManStackVersion 3.0
    16:30:10 Verbose | SerializationVersion 1.1.0.1
    16:30:10 Verbose | CLRVersion 4.0.30319.42000
    16:30:10 Verbose | BuildVersion 6.2.9200.16481
    16:30:10 Verbose | PSCompatibleVersions {1.0, 2.0, 3.0}
    16:30:10 Verbose | PSRemotingProtocolVersion 2.2
    16:30:10 Info | Setting ReportingService2010 Uri to https:/ssrs-server.domain/reportserver/ReportService2010.asmx.
    16:30:10 Info | Setting ReportExecutionService Uri to https://ssrs-server.domain/reportserver/ReportExecution2005.asmx.
    16:30:10 Info | Listing all reports.
    16:30:54 Info | /TestFolder/Data Sources
    16:30:54 Info | /TestFolder/Data Sources/Test
    ... 16:30:54 Info | Exit code: 0
    '''

  4. 4 Posted by Alexander Harri... on 03 Feb, 2017 04:52 PM

    Alexander Harris's Avatar

    Hi Shawn - whether the instance is already running and "hot" was the first thing we checked. Same behaviour with the credentials specified - in fact, that was how I had it setup originally for testing and have since moved to using the machine account in the hopes that this would help.

    As per my latest comment, I'm starting to have the feeling that there could be something additonal / confounding going on relating to PS execution overall. Any comments would be hugely appreciated.

  5. 5 Posted by shawn.sesna on 03 Feb, 2017 05:07 PM

    shawn.sesna's Avatar

    Allow me to paraphrase to make sure I understand correctly;

    Calling .exe through PowerShell on local machine - 2 seconds
    Calling .exe through PowerShell on server running tentacle - 2 seconds
    Calling .exe through tentacle PowerShell - 30 seconds

  6. 6 Posted by Alexander Harri... on 03 Feb, 2017 05:35 PM

    Alexander Harris's Avatar

    Hi Shawn - yes, that's exactly right.

  7. 7 Posted by shawn.sesna on 03 Feb, 2017 05:42 PM

    shawn.sesna's Avatar

    Are there any proxy servers in play? Something that GPO would be setting for you behind the scenes that Octopus may not know about?

  8. 8 Posted by Alexander Harri... on 03 Feb, 2017 06:35 PM

    Alexander Harris's Avatar

    Hi Shawn,

    Or org does use proxy servers, but only for external net access. At times
    I've had to specifically configure an app to not use the proxy, but in
    those cases it was a simple case of no access at all, not slow access.

    I suppose the trick with that would be to install the octopus tentacle on
    the same server as the SSRs server as connect by IP or local host route so
    that it definitely avoids the proxy.

    Currently we've been using a single tentacle which is added to multiple
    environments. The server to connect to is then driven from env. scoped
    parameters.

    I will see if we can set that up for a test after the weekend and see if it
    mitigates this issue. Here's hoping!

  9. 9 Posted by shawn.sesna on 07 Feb, 2017 04:55 PM

    shawn.sesna's Avatar

    Following up, any progress?

  10. 10 Posted by Alexander Harri... on 07 Feb, 2017 05:48 PM

    Alexander Harris's Avatar

    Looks like it must be proxy / networking related. I've today managed to get around to testing the above today and these are the results:

    • Installed tentacle on the ssrs server "ssrs-server.domain"
    • Running the above tests via octopus, using "ssrs-server.domain" in the URL's and same timeouts etc.
    • Running above tests via octopus, but replacing the URL's with "localhost" and the results come back near instantly.

    NB: I had to turn update the config of SSRS to allow HTTP connections to test thsi (since the certificate won't allow https to "localhost" as the domain name doesn't match).

    So - this leads to the conclusion that this must be a proxy or networking issue as you suggested - but specific to only when the request is being made by the machine account under which Octopus is running. I've not seen this kind of behaviour before and I'm going to take it up with our Net Ops tomorrow and try and find out what's goign on!

  11. 11 Posted by shawn.sesna on 07 Feb, 2017 06:04 PM

    shawn.sesna's Avatar

    How many hops does it take to resolve the NETBIOS or DNS name?

  12. 12 Posted by Alexander Harri... on 08 Feb, 2017 11:19 AM

    Alexander Harris's Avatar

    Hi Shawn,

    Been liaising with IT support this morning and done a lot more digging. Still no closer to a solution unfortunately, but here's what we know. When I say "via octopus" I mean in this context:

    Ad Hoc Task - running on the "ssrs-server.domain" Tentacle (which hosts the SSRS instance).

    Spoke with IT support. they confirmed that :

    • the proxy shouldn't be involved in any case (unless the "app" (Octopus in this case) specifically configures it).
    • The firewall shouldn't be involved either (and if it was, it would be an outright "fail" not a long runtime / timeout scenario).

    Things I've tried in addition:

    • DNS server is 2 hops away from the ssrs server
    • Tested ping / tracert to the dns server, and the ssrs-server via octopus. Quick, reliable results.
    • Updated the Octopus Tentacle configuration to ensure it's not using a proxy. No change.
    • Tried specifying different user account credentials to New-WebServiceProxy (that of a different service account just to see) - no difference.

    Getting desperate:

    • Updated the log on account for the Tentacle service to my own account (temporarily) - scripts work as expected.

    From this we can deduce that:

    • Octopus itself isn't the cause - not some weird environmental issue. The script runs fine when the Tentacle runs as me, or when it connects to localhost.

    So, the only problem case is:

    • Octopus Tentacle
    • Running as local system account
    • Connecting to a "remote" URI (even though it redirects the local machine)
    • http or https
    • Not permission issue (since various tests in this configuration do complete, just VERY slowly)

    Does anyone have any suggestions of where to go from here? One thing that occurs to me is to request a specific service account is created for the Tentacle to log in as. Perhaps this will work, though then we'd either have to have a separate service account for every Tentacle, or use the same one across all tentacles. The downside to this would be having to manage file / folder permissions, on each Tentacle, where using the system account avoids this.

  13. 13 Posted by shawn.sesna on 08 Feb, 2017 04:37 PM

    shawn.sesna's Avatar

    You've certainly done an excellent job troubleshooting, well done! What sticks out is what you've discovered, localhost and running as an AD account. Using localhost, you're bypassing the need to "discover" the NETBIOS or DNS name via Active Directory. Your AD account may have some permissions or access to cache in AD that the machine account doesn't have. I'm not an AD expert, but it sounds like it may be a factor.

    On a side note, I've seen similar behavior with TFS. I had some code that would iterate through projects and determine if you were a member and had access. If you were an the Administrators group of the web server, the code executed quickly. If you were not, it took 30+ seconds to complete. Working with Microsoft we discovered that when it was taking 30+ seconds, it was receiving hundreds of Access Denied exceptions before eventually completing successfully (weird). This was somehow related to server cache and the solution we came up with was to purge the cache every call.

  14. 14 Posted by alexandermlharr... on 08 Feb, 2017 05:57 PM

    alexandermlharris's Avatar

    Hi Shawn,

    Thanks for the reply. I was speaking with another colleague today who suggested another step try on the information gathering front.

    First, I’m going to request that the machine account is granted logon permissions if possible, then I’ll login as the machine account and test the script “myself”. I’d expect in this case that we’d get the same behaviour in this case – ie delay / timeout, but if I’m able to do this it’ll make testing various scripts / calls a lot faster.

    In addition, I’ve also tried using another service account (used for Team City) as the logon user for the Tentacle service – same results (timeout). I’m going to speak with IT again about this too, and see if they can compare the different accounts and see what permissions / flags etc. are different between them. I’d expect some differences given the different uses, but perhaps that will tell us something more.

    Finally I may request a brand new service account and see if that helps. If not, see if we can slowly change this service account to be as close as possible to my own account until we (hopefully) find the magic setting or group membership that’s causing the problem.

    Thanks again for your help – if anyone has any more suggestions at this point, please to chime in.

    Cheers,

    Alexader

  15. Support Staff 15 Posted by Robert Wagner on 13 Feb, 2017 01:40 AM

    Robert Wagner's Avatar

    Hi Alexander,

    We've been keeping an eye on this thread, and haven't had anything to add so far to Shawn's excellent responses.

    With the problem being account related, you could try adding a Octopus.Action.PowerShell.ExecuteWithoutProfile with a value of true to your project. This will disabled the loading of the user's powershell profile. It is usually to fix slow start of PowerShell scripts, but it might be worth a try.

    Another thought is that the two accounts may be on different domains, and there is a problem in the cross domain trust/communication.

    Regards,

    Robert W

  16. 16 Posted by Alexander Harri... on 03 Mar, 2017 09:49 AM

    Alexander Harris's Avatar

    Hi Robert,

    Many thanks for the suggestion. I tried this and sadly it made no difference. We also just updated Octopus version, which I had hoped (rather optimistically!) might help, but no change.

    I've got time to look into this again and have requested help from IT to look into user account settings, group policy etc. and see if we can find the root. I'll update here if I find anything useful.

  17. Paul Stovell closed this discussion on 09 Jun, 2017 07:37 PM.

  18. Dalmiro Grañas re-opened this discussion on 28 Dec, 2017 09:28 PM

  19. Paul Stovell closed this discussion on 28 Dec, 2017 09:33 PM.

  20. shawn.sesna re-opened this discussion on 03 Jan, 2018 06:23 PM

  21. 17 Posted by shawn.sesna on 03 Jan, 2018 06:23 PM

    shawn.sesna's Avatar

    @Alexander, I regret I was unable to reproduce this issue back when you first reported it, however I have now encountered it myself. I have opened a ticket with Microsoft and we are attempting to figure out what the root cause of this is.

    I can report a workaround, using a different server and targeting the SSRS server should allow it to complete normally. I will continue to work with Microsoft to see if we can solve it.

  22. 18 Posted by Alexander Harri... on 04 Jan, 2018 09:57 AM

    Alexander Harris's Avatar

    Hi Shawn,

    Thanks for the update, please do let me know if you find anything further regarding this.

    Unfortunately, setting up a new host for the Octopus server isn't possible at present, though if we continue to have issues we may need to look into this.

    For some additional information, we recently had another issue which at least feels very similar. I have a thread about it here which I'll link for you just in case it may be relevant:

    http://help.octopusdeploy.com/discussions/problems/64727-octoexe-sl...

    As I say, this feels similar so perhaps there's some details in there which could prove useful to this.

  23. 19 Posted by shawn.sesna on 21 Feb, 2018 06:13 PM

    shawn.sesna's Avatar

    @Alexander, in our case, we found the issue went away when we checked the 'This account supports Kerberos AES 256 bit encryption' box on the account tab of the service account the tentacle was running under. I do not know if this will help you or now.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac