Security events

rjhansen's Avatar

rjhansen

12 Jan, 2018 05:26 PM

We work in a secure environment and want to be able to capture login failure events to a log (event viewer, log file, or other). The web page on Auditing specifies that Octopus is not tooled to capture these event types out-of-the-box. Can you folks offer some guidance on how we can go about getting this data? We're open to building tools (such as a script) that work alongside Octopus to gather this data but are not sure how to even access these events. Thanks.

https://octopus.com/docs/administration/auditing

  1. Support Staff 1 Posted by Michael Noonan on 15 Jan, 2018 04:34 AM

    Michael Noonan's Avatar

    Hi!

    Thanks for getting in touch! You are correct, we don't currently write any events to the audit log about authentication attempts/success/failure. We do write a warning message to the Octopus Server logs, but those are intended to be more informational.

    I have talked with other members of our team, and we've decided to raise this GitHub issue to address what we feel is the most important event: https://github.com/OctopusDeploy/Issues/issues/4163

    Does this sound like it will suit your needs?

    Hope that helps!
    Mike

  2. 2 Posted by rjhansen on 17 Jan, 2018 07:27 PM

    rjhansen's Avatar

    That is better than not logging login failures at all but it's not preferred. Is there any reason they can't be logged to the audit log like other activities?

  3. Support Staff 3 Posted by Michael Noonan on 29 Jan, 2018 10:33 PM

    Michael Noonan's Avatar

    Hi!

    At this point, the Audit Log in Octopus is all about recording successful actions which mutate the state of your world due to actions by authorized users. This would be the first set of events we'd start writing for anonymous users, and my concern about that is the potential impact of any anonymous user causing harm.

    At this point in time we feel like we are taking a good, if conservative, approach to the problem, and see where we decide to go from there.

    I'd be interested to understand if there is anything specific you would need beyond what we are planning to provide. :)

    Hope that helps!
    Mike

  4. 4 Posted by rjhansen on 06 Mar, 2018 11:22 PM

    rjhansen's Avatar

    We're currently on 3.17.14. I understand that "When a user fails to authenticate for a certain number of times, the Octopus Server will rate-limit any future requests to authenticate as that user, until a certain time period expires. When this occurs a warning is written to the Octopus Server logs." How many login failures are required in order for this rate-limiting state to be reached? How long is the time expiry period? I want to re-create this scenario so I can verify the functionality and build automated tools to scan the Octopus log.

  5. Support Staff 5 Posted by Michael Noonan on 06 Mar, 2018 11:46 PM

    Michael Noonan's Avatar

    Hi!

    Here is a snippet from our source code. Unfortunately this code is not open-sourced.

    const int AttemptsBeforeSlowdown = 2;
    const int AttemptsBeforeBan = 9;
    readonly TimeSpan timeBetweenGarbageCollects = TimeSpan.FromSeconds(30);
    readonly TimeSpan timeToBan = TimeSpan.FromMinutes(10);
    readonly TimeSpan timeToTrackInvalidLogins = TimeSpan.FromMinutes(10);
    

    And the tests:

    using System;
    using Nevermore;
    using NSubstitute;
    using NSubstitute.Exceptions;
    using NUnit.Framework;
    using Octopus.Core.Model.Events;
    using Octopus.Core.RelationalStorage;
    using Octopus.Node.Extensibility.Authentication.HostServices;
    using Octopus.Server.Extensibility.Authentication.HostServices;
    using Octopus.Server.Web.Infrastructure.Authentication;
    using Octopus.Shared.Diagnostics;
    using Octopus.Shared.Time;
    
    namespace Octopus.Tests.Octopus.Portal.Infrastructure.Authentication
    {
        public class InvalidLoginTrackerFixture
        {
            const string IpAddress1 = "8.8.8.8";
            const string IpAddress2 = "8.8.4.4";
            InvalidLoginTracker tracker;
            FixedClock clock;
            IEventStore eventStore;
    
            [SetUp]
            public void SetUp()
            {
                var relationalStore = Substitute.For<IRelationalStore>();
                relationalStore.BeginTransaction().Returns(Substitute.For<IRelationalTransaction>());
    
                eventStore = Substitute.For<IEventStore>();
    
                tracker = new InvalidLoginTracker(
                    clock = new FixedClock(DateTimeOffset.UtcNow),
                    Log.Octopus(),
                    eventStore,
                    relationalStore);
            }
    
            [Test]
            public void ShouldSlowOn3InvalidAttempts()
            {
                Assert.That(tracker.BeforeAttempt("jack", IpAddress1), Is.EqualTo(InvalidLoginAction.Continue));
                tracker.RecordFailure("jack", IpAddress1);
    
                Assert.That(tracker.BeforeAttempt("jack", IpAddress1), Is.EqualTo(InvalidLoginAction.Continue));
                tracker.RecordFailure("jack", IpAddress1);
    
                Assert.That(tracker.BeforeAttempt("jack", IpAddress1), Is.EqualTo(InvalidLoginAction.Slow));
                tracker.RecordFailure("jack", IpAddress1);
    
                eventStore.DidNotReceive().Store(Arg.Any<IRelationalTransaction>(), Arg.Any<Event>());
            }
    
            [Test]
            public void ShouldBanAfter10InvalidAttempts()
            {
                tracker.RecordFailure("jack", IpAddress1);
                tracker.RecordFailure("jack", IpAddress1);
                tracker.RecordFailure("jack", IpAddress1);
                tracker.RecordFailure("jack", IpAddress1);
                tracker.RecordFailure("jack", IpAddress1);
                tracker.RecordFailure("jack", IpAddress1);
                tracker.RecordFailure("jack", IpAddress1);
                tracker.RecordFailure("jack", IpAddress1);
                Assert.That(tracker.BeforeAttempt("jack", IpAddress1), Is.EqualTo(InvalidLoginAction.Slow));
    
                tracker.RecordFailure("jack", IpAddress1);
                Assert.That(tracker.BeforeAttempt("jack", IpAddress1), Is.EqualTo(InvalidLoginAction.Ban));
    
                eventStore.Received(1).Store(Arg.Any<IRelationalTransaction>(), Arg.Is<Event>(e => e.Category == EventCategory.LoginBanned));
            }
    
            [Test]
            public void ShouldDifferentiateByIp()
            {
                Assert.That(tracker.BeforeAttempt("jack", IpAddress1), Is.EqualTo(InvalidLoginAction.Continue));
                tracker.RecordFailure("jack", IpAddress1);
    
                Assert.That(tracker.BeforeAttempt("jack", IpAddress1), Is.EqualTo(InvalidLoginAction.Continue));
                tracker.RecordFailure("jack", IpAddress1);
    
                Assert.That(tracker.BeforeAttempt("jack", IpAddress2), Is.EqualTo(InvalidLoginAction.Continue));
                tracker.RecordFailure("jack", IpAddress2);
    
                Assert.That(tracker.BeforeAttempt("jack", IpAddress2), Is.EqualTo(InvalidLoginAction.Continue));
                tracker.RecordFailure("jack", IpAddress2);
            }
    
            [Test]
            public void ShouldResetOnValidLogin()
            {
                Assert.That(tracker.BeforeAttempt("jack", IpAddress1), Is.EqualTo(InvalidLoginAction.Continue));
                tracker.RecordFailure("jack", IpAddress1);
    
                Assert.That(tracker.BeforeAttempt("jack", IpAddress1), Is.EqualTo(InvalidLoginAction.Continue));
                tracker.RecordFailure("jack", IpAddress1);
    
                Assert.That(tracker.BeforeAttempt("jack", IpAddress1), Is.EqualTo(InvalidLoginAction.Slow));
                tracker.RecordSucess("jack", IpAddress1);
    
                Assert.That(tracker.BeforeAttempt("jack", IpAddress1), Is.EqualTo(InvalidLoginAction.Continue));
            }
    
            [Test]
            public void ShouldForgetFailuresAfter10Minutes()
            {
                tracker.RecordFailure("jack", IpAddress1);
                tracker.RecordFailure("jack", IpAddress1);
                tracker.RecordFailure("jack", IpAddress1);
                tracker.RecordFailure("jack", IpAddress1);
                tracker.RecordFailure("jack", IpAddress1);
                tracker.RecordFailure("jack", IpAddress1);
                tracker.RecordFailure("jack", IpAddress1);
                tracker.RecordFailure("jack", IpAddress1);
                tracker.RecordFailure("jack", IpAddress1);
                Assert.That(tracker.BeforeAttempt("jack", IpAddress1), Is.EqualTo(InvalidLoginAction.Ban));
    
                clock.WindForward(TimeSpan.FromMinutes(5));
    
                Assert.That(tracker.BeforeAttempt("jack", IpAddress1), Is.EqualTo(InvalidLoginAction.Ban));
    
                clock.WindForward(TimeSpan.FromMinutes(6));
    
                Assert.That(tracker.BeforeAttempt("jack", IpAddress1), Is.EqualTo(InvalidLoginAction.Continue));
            }
    
            [TearDown]
            public void TearDown()
            {
                tracker.Dispose();
            }
        }
    }
    

    Hope that helps!
    Mike

  6. 6 Posted by rjhansen on 07 Mar, 2018 01:05 AM

    rjhansen's Avatar

    Hi Michael. What I’m trying to do is build a real world test to ensure that login failures are recorded in the Octopus log. According to the code below, my login activity should be rate limited after 2 consecutive failures and I should be banned after 9 consecutive failures. I’m not having any luck verifying that. I just attempted a moment ago to login to our web portal with my own username and the incorrect password. After 15 attempts, no record was logged in the Octopus log. And the following attempt using the correct password logged me in successfully. What can I do to verify that login failures are logged (even if it takes 9 or so consecutive failures to get there)?

  7. Support Staff 7 Posted by Michael Noonan on 07 Mar, 2018 07:02 AM

    Michael Noonan's Avatar

    Hi!

    Thanks for keeping in touch. I'll admit I'm a bit confused here too. I've just tested the login rate limiting and ban behaviour on Octopus 2018.2 and it's working as I'd expect.

    image

    I get rate limited after 2 failed attempts. I get banned after 10 failed attempts. This lines up with the tests I posted earlier. We haven't changed this code in a very long time (since Octopus 2.6.5) so I would be disappointed if it was broken in 3.17.14 - please let me know if this seems broken in the UI for you.

    I've just checked about the logging. When login attempts are banned from a particular IP address we write a Warning into the Octopus Server log file. This has been in place since Octopus 2.6.5. We only added the equivalent audit entry in Octopus 2018.2.7.

    Hope that helps!
    Mike

  8. 8 Posted by rjhansen on 07 Mar, 2018 07:07 PM

    rjhansen's Avatar

    I’m not getting the message you’re getting, nor do I seem to be getting rate limited or banned. It took about two minutes to create 15 consecutive failed logins.

    Our login screen looks like this. (attached) It doesn’t look any different after 15 failed logins.

  9. Support Staff 9 Posted by Michael Noonan on 07 Mar, 2018 08:47 PM

    Michael Noonan's Avatar

    Hi!

    Thanks for keeping in touch! This makes a lot more sense now! The rate limiting is only applied when you are using our built-in UsernamePasswordAuthenticationProvider - where Octopus takes care of your credentials and the entire end-to-end authentication flow.

    In this case you are using Active Directory, and Octopus delegates the entire authentication flow to Active Directory. In this case things like password policies and account lockout policies are all handled by Active Directory: https://docs.microsoft.com/en-us/windows/security/threat-protection...

    In this case to monitor/alert on failed logins for Active Directory you could inspect the Windows Event log instead of the Octopus Server log.

    I'm not sure if this meets your specific needs directly, but it feels like we've solved one mystery.

    I'll talk with the rest of our authentication team today about the idea of rate-limiting for all authentication attempts regardless of the identity provider.

    Hope that helps!
    Mike

  10. 10 Posted by rjhansen on 08 Mar, 2018 12:21 AM

    rjhansen's Avatar

    That does make a lot more sense. I am finding the events in the Event Log now and that is excellent.

    However now a new concern has been raised. I recall that Octopus is API-first so I would expect the same login failure events for an API call as I would for activity in the web portal. Unfortunately I’m not seeing these events. If a user tried to brute force a connection using an API key (or potentially started with a partial API key) they would not leave a footprint of activity for us to take action on. Do I have that right? Does that concern you?

  11. Support Staff 11 Posted by Michael Noonan on 08 Mar, 2018 09:48 PM

    Michael Noonan's Avatar

    Hi!

    Thanks for keeping in touch. I'm glad you made progress on the login events for Active Directory. The API Key authentication point you've raised has given us food for thought.

    We are generally very cautious of implementing any rate-limiting in Octopus: we have a lot of customers who host Octopus behind HTTP proxy servers where the origin appears as the proxy server for every HTTP request.

    The rate limiting for Username/Password authentication is keyed on the request origin IP address and the username. If someone is trying to brute force API key authentication we could only key on the origin's IP address.

    At this point we are going to do some more investigation and thinking, and I'll get back to you next week.

    If you have some pragmatic thoughts about this I'd appreciate them as it may help sway our decision!

    Thanks!
    Mike

  12. 12 Posted by rjhansen on 15 Mar, 2018 08:27 PM

    rjhansen's Avatar

    On one side someone could argue that a 25 character API key is too complex to be broken. It would take so long to brute force every possible combination to find a live key that it’s hardly worth worrying about someone trying to iterate through them. On the other side someone could argue that knowledge of the failed attempts would be tremendously valuable. This is the side we lean towards. If someone is trying to break into our systems we want to know about it AND we want to do something about it. Our security staff actively watches our centralized logging system for events such as excessive login failures. If something doesn’t look right they check into it. It’s frowned upon (at best) to use a software suite that doesn’t have this logging capability and at worst we could potentially be asked to find another software suite for automating deployments.

    Would it be possible to add some settings to Octopus to allow users (server admins) to toggle logging capability on and off? Maybe have toggle switches for different kinds of events so that admins can choose which events they want to log. We personally like the idea of passing these events to the domain controller (like login events for Active Directory users) although if they were sent to one of the Octopus logs that would be acceptable. Not logging the events at all is a little scary.

  13. Support Staff 13 Posted by Michael Noonan on 15 Mar, 2018 10:51 PM

    Michael Noonan's Avatar

    Hi!

    Thanks for adding more perspective to the conversation. I'll keep the conversation going on our side and get back to you when we've made some decisions.

    Thanks!
    Mike

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac