Hitachi Vantara Pentaho Community Forums
Results 1 to 1 of 1

Thread: Variable substitution within kettle.properties

  1. #1

    Question Variable substitution within kettle.properties

    Over the last time, I did something like the following:

    I had some "defaults" configured, using variable substitution refering to another default. So my kettle.properties look like this:
    Code:
    ###
     # Defaults
     ##
    
    
    # mails on success
    mail.success.to      = pdi@example.com
    mail.success.cc      =
    mail.success.subject = [PDI][Job][${Internal.Job.Name}] SUCCESS
    mail.success.body    = The job \\"${Internal.Job.Repository.Directory}/${Internal.Job.Name}\\" was successful.
    
    
    # mails on error
    mail.error.to      = ${mail.success.to}
    mail.error.cc      = ${mail.success.cc}
    mail.error.subject = [PDI][Job][${Internal.Job.Name}] ERROR
    mail.error.body    = The job \\"${Internal.Job.Repository.Directory}/${Internal.Job.Name}\\" ended with an error.\\n\\nABORT message: ${message}
    
    
    shell.mail.success = printf "${mail.success.body}" | mail -s "${mail.success.subject}" -c "${mail.success.cc}" "${mail.success.to}"
    shell.mail.error   = printf "${mail.error.body}" | mail -s "${mail.error.subject}" -c "${mail.error.cc}" "${mail.error.to}"
    
    
    # final commands
    shell.success = ${shell.mail.success}
    shell.error   = ${shell.mail.error}
    
    
    ###
     # Specific stuff
     ##
    
    
    # specific stuff mails on success
    specific.stuff.mail.success.to      = pdi@example.com
    specific.stuff.mail.success.cc      = pdiCC@example.com
    specific.stuff.mail.success.subject = [SPECIAL][${configured.transformation}] SUCCESS
    specific.stuff.mail.success.body    = The transformation ${configured.transformation} was successful.
    
    
    # specific stuff mails on error
    specific.stuff.mail.error.to      = ${specific.stuff.mail.success.to}
    specific.stuff.mail.error.cc      = ${specific.stuff.mail.success.cc}
    specific.stuff.mail.error.subject = [SPECIAL][${configured.transformation}] ERROR
    specific.stuff.mail.error.body    = The transformation ${configured.transformation} ended with an error.\\n\\nABORT message: ${message}
    
    
    # specific stuff shell commands
    specific.stuff.shell.success = ${shell.mail.success}
    specific.stuff.shell.error   = ${shell.mail.error}
    When running jobs, I used the "specific.stuff" variables to overwrite the defaults, in this case the "mail" variables. In the end, when using "shell.mail.success", all substitutions where done using the specific stuff data. Everything worked fine, until I extended my configuration. At this point my mail command used two defaults and two specific values. With small changes to the configuration, the substitution behaviour changed. I recognized this using the following PDI CE versions: 6.1.0.1-196, 7.0.0.0-25, 7.1.0.0-12, 8.0.0.0-28
    Please don't question why I send mails through the shell instead of using built-ins, this post is about substituion problems.

    I created a small demo (see attachment) using the GetVariables step, both JavaScript functions for this case, and variable substituion within a WriteToLog step. This is just a small demonstration without overwriting variables.

    As you can see, using the GetVariables step or the substitution feature from the "diamond fields" will result in replacing all variables, whereas the script functions start to behave inconsistent. With different amounts of variables in the properties file, different amounts of variables get replaced.

    My questions are: what is the expected behaviour of the script functions? When does the substitution take place? Do the variables get replaced when loading the properties file on start, or are they replaced when requested via script or "diamond field"? May I have encountered a bug?
    Attached Files Attached Files
    R2-PDI | PDI CE 8.0.0.0-28 | MySQL Connector/J 5.1.46 | Rhino 1.7.9
    @development: ubuntu 18.10 (x64) | Oracle Java 1.8.0_192 | Percona 5.7.24-27-1 | Elasticsearch 6.5.4
    @processing: Debian 8.8 (x64) | Oracle Java 1.8.0_191 | Percona 5.7.23-25-1 | Elasticsearch 6.3.1

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.