Ampersand and escaping of environment variables

HBaz's Avatar

HBaz

07 Dec, 2012 02:28 PM

I'm having trouble with how Octopus handles environment variables that contain ampersands (and possibly other special characters). For example, the connection strings generated by Entity Framework look something like this

connectionString="metadata=res:///Model.MyModel.csdl|res:///Model.MyModel.ssdl|res://*/Model.MyModel.msl;provider=System.Data.SqlClient;provider connection string= "data source=192.168.0.100;initial catalog=MyDatabase;integrated security=False;persist security info=True;user id=myAccount;password=p@ssword;multipleactiveresultsets=True;App=EntityFramework ""

After Octopus replaces the values however, the two ampersands get turned into "&" in my transformed config file. The resulting file looks like ......App=EntityFramework "" instead of ......App=EntityFramework ""

This seems like a bug with the HTML encodingdecoding from the octopus UI. Any workarounds to handle this behavior?

Also, please make the columns in the variables screen resizable. Managing long connection strings in the UI is rather burdensome.

-h

  1. 1 Posted by HBaz on 07 Dec, 2012 03:36 PM

    HBaz's Avatar

    ok, here's one workaround. Instead of pasting in " replace it with a " (double-quote)

  2. 2 Posted by Alex on 28 Oct, 2013 10:40 AM

    Alex's Avatar

    I think it is incorrect transform &guot; to " and this is defenetly a bug.
    Is there plans to fix such transformation?

  3. Support Staff 3 Posted by Paul Stovell on 26 Nov, 2013 08:40 AM

    Paul Stovell's Avatar

    Hi,

    It may seem surprising but this isn't a bug; it's by design. What would you expect to happen if you created a variable named WelcomeMessage with the value:

      Hello, "world!"
    

    If we didn't encode the values, you'd end up with:

     <add key="WelcomeMessage" value="Hello, "world!"" />
    

    Which would be broken (and would equally be considered a bug). So, we assume that anything you type in the variables editor is unencoded (raw), and we encode it when inserting it into XML.

    If you had a PowerShell script like this:

     Write-Output $WelcomeMessage
    

    You'd also expect it to output:

     Hello, "world!"
    

    Rather than:

     Hello, &quot;world!&quot;!
    

    So Octopus expects that any value you give us is unencoded, and we encode it. That way your variables can be agnostic of the encoding used by wherever it might be used.

    Paul

  4. 4 Posted by Alexander Kapel... on 26 Nov, 2013 03:23 PM

    Alexander Kapelyukhovskiy's Avatar

    Hi Paul.
    Thanks for your answer.

    The problem was because we stored connection strings with escaped symbol "
    => &guot; inside connection string and after transformation &guot; we
    receives &amp;quot;
    my concern was that you maybe don't need to do encoding for & symbol.
    By I already resolve this issue by using ' instead " and now I it works ok.

    Best regards,
    Alex

  5. 5 Posted by Eric on 20 May, 2014 12:55 PM

    Eric's Avatar

    I politely disagree that this is a feature rather than a bug - I think the least surprising outcome is that Octopus directly substitutes the contents of a given variable when it does its substitutions.

    Maybe it treats XML files specially, but I'm not seeing the behavior you've specified, Paul (if I understand correctly, quotes should automatically be encoded). I'm working with Azure and am inserting values into a .cscfg file.

    A variable with the following value: "inside quotes"
    Gets inserted into my .cscfg as: "inside quotes" (as I'd expect)

    A variable with the following value: "inside quotes"
    Gets insert into my .cscfg as: &"inside quotes&" (as I wouldn't expect)

    What I actually need inserted is the one above - "inside quotes"

    Based on the above, I can't find a way to insert the encoded value that I need. I think single quotes will probably work in this case, but, to me, it's a bug that I can't insert an encoded value directly via a variable.

  6. 6 Posted by Eric on 20 May, 2014 01:11 PM

    Eric's Avatar

    The second example result got screwed up - looks like the day of encoding errors for me :). This ought to work better:

    A variable with the following value:

     &quot;inside quotes&quot;
    

    Gets insert into my .cscfg as (and this is unexpected):

     &amp;&quot;inside quotes&amp;&quot;
    

    What I actually need inserted is the one above -

     &quot;inside quotes&quot;
    
  7. 7 Posted by Nicholas Blumha... on 22 May, 2014 11:03 PM

    Nicholas Blumhardt's Avatar

    Thanks for the follow-up Eric! We've spent a bit of time thinking about this one previously; the challenge (and reason we've chosen the current behaviour) is that we want to make sure Octopus always leaves configuration files etc. as valid XML.

    It's tricky to do literal substitution like you've suggested because let's say you have a variable defined in the Octopus UI like:

    ConnectionString = blah;Password=#{Pass};blah
    

    So, the connection string is bound for an XML config file; but Pass may be a variable used in several places, some of them XML, some not.

    Should now the password A&B be entered into the Octopus UI as A&B or A&amp;B?

    It's an open choice, but we need to choose between them for the Octopus default behaviour. We chose the literal form A&B because in more complex variable substitution scenarios we think it will be less surprising. (E.g. imagine Pass is a prompted variable - it would be strange to request the user enter their password using correct XML encoding :)

    There are logical arguments each way, but forced with a choice we're happy with the current method. Hope this helps explain our reasoning a little better.

    Best regards,
    Nick

  8. 8 Posted by Eric on 23 May, 2014 01:46 AM

    Eric's Avatar

    Thanks for the quick response, it will work fine as you've specified in most conditions. It would be nice if the functionality could be toggled, as I can see needs to do replacements in files other than XML.

  9. 9 Posted by Nicholas Blumha... on 23 May, 2014 01:51 AM

    Nicholas Blumhardt's Avatar

    Thanks for the update; we're considering something like this (e.g. in an HTML email template):

    <p>Hello, #{HtmlEscape(Name)}</p>
    

    In that case, using #{...} style substitutions in an XML document would work along the lines you're suggesting. No timeline on this currently, but expecting to address it before very long :)

    Cheers,
    Nick

  10. 10 Posted by msallmen on 08 Jul, 2014 10:13 PM

    msallmen's Avatar

    I have a substitution variable with a double ampersand.
    So, p&ssword => p&amp;ssword
    but, p&&ssword => p&&ssword

    HTML encoding seems to ignore the double ampersand, breaking my config file. Is this a bug, or is there something I can do to force the encoding to happen?

  11. 11 Posted by Vanessa Love on 10 Jul, 2014 05:53 AM

    Vanessa Love's Avatar

    Hi,

    Thanks for getting in touch!
    In this case you will have to escape it yourself such as P&amp;&amp;ssword
    We do not escape for variable substitutions.

    Vanessa

  12. 12 Posted by mpeterson on 19 Sep, 2014 02:55 PM

    mpeterson's Avatar

    Hi,

    I'm trying to following this conversation because I seem to be seeing different behavior than explained here.

    Yesterday I had a java keystore item where I had a password that I had to pass along. It had an ampersand in it. So as an example, it showed as "p&amp;amp;ssword". Well that was an issue because it was supposed to show up as "p&ssword". So I tried passing "p&amp;amp;ssword". It showed up as "p&amp;amp;amp;ssword". I also tried "p\\&ssword" (ended up with "p&amp;ssword"), because I found that in a previous variable that if I had a path that I passed as a variable "c:\temp\myfiles\" that ended up showing up as c:tempmyfiles, so in the end I had to pass it as "c:\\\temp\\\myfiles\\\". (Note: I've looked at this in the preview window, so what shows here in my note should be correct. IE in the last example I typed 2 backwards slashes not the 3 I had to use to get 2 to show here)

    In the end I changed the password to not have the ampersand to get around this because I couldn't find a way to just pass a single ampersand as an ampersand. However as I use variables more and more, I may not have that option.

    Reading the comments here I'm not clear on how I would actually pass a single ampersand even if that's what I wanted to do.

    Also, is there documentation on what to plan on escaping and what the strategy is to do it?

    Note: None of these were web.configs. Maybe the behavior is different for app settings. Guess I'll find out today. :)

    Second note: Has there ever been thought to a checkbox that says "Don't touch the contents of this variable. I accept all liability for any bad mojo jojo that ends up happening because of bad encoding?"

  13. 13 Posted by mpeterson on 19 Sep, 2014 03:06 PM

    mpeterson's Avatar

    Yeah I know that the checkbox could be shortened to "HTMLEncode?" but...

  14. 14 Posted by Eric on 19 Sep, 2014 04:11 PM

    Eric's Avatar

    I thought about this some more and I still think it's silly to do any sort of encoding behind the scenes, both as the default and especially because it cannot be turned off.

    With the current auto-encoding behavior, there are certain strings which cannot be inserted into files during variable substitution because of the encoding behavior (assuming it hasn't changed since my first message). At least without encoding, I can type what I want and Octopus can just do a literal, non-fancy substitution.

    From Nick's response above:
    "(E.g. imagine Pass is a prompted variable - it would be strange to request the user enter their password using correct XML encoding :)" - it would be, but it's also strange to ask them to write something such as msallmen stated, this still fails because with "&&" you do have to manually enter it as &&. So now I have to say "as long as your password doesn't have two subsequent &s, you don't have to XML encode it, but if it does, then you have to." Additionally, if you have "Pass" used in one context that requires encoding and another that doesn't, you just specify two variables with different values. I think that's easy enough to do - there's some minor duplication but that's a fact I'm willing to accept for full control over my variable values.

    All that said, it appears there is conflicting information - this entire thread is about automatic encoding during variable substitution, but Vanessa has stated that no encoding is performed on variable substitution. This can't be true based on the behaviors reported here, as it appears encoding is performed differently based on Octopus's best guess of the variable's target use.

    I just don't understand why any substitution other than literal substitution is the default. This behavior to "do what is right" will never have enough context to really do what is right in all situations. In a world with no context, it is doing what is wrong - no one says string xx = "jkl&"; in any programming language and expects it to be XML-encoded when they read it later unless they explicitly call something to convert it. I think that stands here.

    I know you can't change the default behavior for backwards compatibility, but please, please, give us an option (even Octopus-wide) to just turn off the encoding behavior.

  15. 15 Posted by mpeterson on 19 Sep, 2014 07:02 PM

    mpeterson's Avatar

    I actually think what was stated is probably correct (in intent). I think what is being said is that there are two different behaviors.

    If you are doing this as part of the "Replace appSettings and connectionString entries in any .config files" labeled as Configuration Variables in the Deploy a Package step than I think what is being said is that it has one behavior. From what I can tell this doesn't seem to escape characters.

    On the other hand if you "Substitute variables in a file" it appears that it does escape the characters.

    I guess there's a "third behavior". If you use it as part of a script of your own (or a step template) it also doesn't appear to escape it.

    At least so far (in my limited experience) that appears to be the case. So the specific wording you mention may be correct in intent, but overall it's still confusing and it seems like in certain situations, it's not possible to output specific values.

    I am going to restate what I said above because it probably seemed like I was joking. Rather than give us a tag, could it be a checkbox. HTMLEncode this field, or not. (I can't think of a good reason to HTMLEncode only a portion of a variable, but I may be missing something)

  16. 16 Posted by Anthony Johnsto... on 07 Dec, 2016 11:20 AM

    Anthony Johnston's Avatar

    We are having this issue.

    What was the conclusion here? How do we configure our variables/configs to do this correctly?

    currently catch 22
    1. Don't encode the var - deploy fails with transform error
    2. Encode the var - get double encoding in resulting config

  17. 17 Posted by Vanessa Love on 12 Dec, 2016 12:27 AM

    Vanessa Love's Avatar

    Hi Anthony,

    Using the Substitute variables in files (http://docs.octopus.com/display/OD/Substitute+Variables+in+Files) feature is a workaround in this scenario. It happens before the config transforms substitutions and is not encoded in the same way.
    http://docs.octopus.com/display/OD/Package+deployment+feature+ordering

    Vanessa

  18. 18 Posted by Anthony Johnsto... on 15 Dec, 2016 11:50 AM

    Anthony Johnston's Avatar

    Thanks, that sorted it, Ant

    ________________________________

  19. Paul Stovell closed this discussion on 21 Mar, 2017 03:06 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac