Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: Kettle 3.0 Job Entry Validation

  1. #1
    Mat Lowery Guest

    Default Kettle 3.0 Job Entry Validation

    I recently checked in some changes to the Kettle 3.0 code which I
    wanted to introduce here. The changes relate to check() method of
    subclasses of JobEntryBase. This method, new in Kettle 3.0, validates
    that the job entry has all required information and that the
    information is valid so that it can execute. The paragraphs below go
    over the changes and the benefits over the initial check() method
    implementation. Feedback requested!

    Benefits:

    While implementing job entry check() methods, I decided to factor out
    some common code. The benefits of the new code over the old code are:

    * Single parameterized error message per validation type. When a
    property is null but shouldn't be, we use a single "Property x cannot
    be null." message. The old way specified messages specific to each
    job entry and furthermore specific to each property. This change
    dramatically reduces translation requirements.

    * Convention over configuration. Error message keys are constructed
    according to conventions, then fetched from the resource bundle.
    There is no need to specify the error message key. Also, when errors
    occur, CheckResults are by default TYPE_RESULT_ERROR with the ability
    to change it explicitly to TYPE_RESULT_WARNING.

    * Reusable validation logic. Checking for a file's existence is now
    stored in one class-the FileExistsValidator.

    * Extensible. First, new job entries need not use the new code.
    Second, if new job entries need validations which do not exist, they
    simply need to implement JobEntryValidator and add their validator
    class to the classpath.

    * Java 5 features: variable length argument lists and static imports.

    * Based on mature Commons Validator, PredicateUtils from Commons
    Collections, and FieldChecks from Jakarta Struts 1.x.

    * Is developed in accordance with the "less XML is better" mantra of
    the Kettle community.

    Using:

    All built-in validator classes are in
    org.pentaho.di.job.entry.validator. All validators implement
    JobEntryValidator. JobEntryValidatorUtils contains utility methods to
    be used by both the JobEntryValidator implementations and optionally
    the check() method implementations. Validator related error messages
    by default come from
    org.pentaho.di.job.entry.messages.validator.properties.

    If a property requires multiple validators, combine the validators
    with the AndValidator which Boolean ANDs the results of all given
    validators. AndValidator also "short-circuits" on a false from any
    given validator.

    Improvements:

    * Parameterized resource bundles. Right now, messages can only come
    from org.pentaho.di.job.entry.messages.validator.properties.

    * Remove ValidatorContext and singleton accessor methods like
    JobEntryValidatorUtils.andValidator(). Instead, specify validator-
    specific parameters in the constructors of those validator objects
    like new AndValidator(notBlankValidator(), integerValidator()).
    Instead of being singletons, validators would created and destroyed
    inside the check() method.

    * Allow localized property labels. Currently, the property labels are
    the Java field names.

    * Allow per-validation override of error message.

    * Allow validations that involve dependent fields-e.g. requiredif and
    validwhen. For example, if setLogfile is true, then make sure logfile
    is not blank.

    Old Code:

    public void check(List<CheckResultInterface> remarks, JobMeta jobMeta)
    {
    if (filename == null) {
    remarks.add(new
    CheckResult(CheckResultInterface.TYPE_RESULT_ERROR,

    Messages.getString("JobEntryDeleteFile.CheckResult.No_Filename_Is_Defined"), //
    $NON-NLS-1$
    this));
    } else {
    remarks.add(new CheckResult(CheckResultInterface.TYPE_RESULT_OK,

    Messages.getString("JobEntryDeleteFile.CheckResult.Filename_Is_Defined"),
    this)); //$NON-NLS-1$
    String realFileName = environmentSubstitute(getFilename());
    FileObject fileObject = null;
    try {
    fileObject = KettleVFS.getFileObject(realFileName);
    if (fileObject != null) {
    if (!fileObject.exists() && isFailIfFileNotExists()) {
    remarks.add(new
    CheckResult(CheckResultInterface.TYPE_RESULT_ERROR,

    Messages.getString("JobEntryDeleteFile.CheckResult.File_Does_Not_Exist", //
    $NON-NLS-1$
    realFileName), this));
    } else if (!fileObject.exists() ) {
    remarks.add(new
    CheckResult(CheckResultInterface.TYPE_RESULT_OK,

    Messages.getString("JobEntryDeleteFile.CheckResult.File_Does_Not_Exist", //
    $NON-NLS-1$
    realFileName), this));
    } else {
    remarks.add(new
    CheckResult(CheckResultInterface.TYPE_RESULT_OK,

    Messages.getString("JobEntryDeleteFile.CheckResult.File_Exists", //
    $NON-NLS-1$
    realFileName), this));
    }
    try {
    fileObject.close(); // Just being cautious
    } catch (IOException ignored) {}
    } else {
    remarks.add(new
    CheckResult(CheckResultInterface.TYPE_RESULT_ERROR,

    Messages.getString("JobEntryDeleteFile.CheckResult.Could_Not_Convert_File", //
    $NON-NLS-1$
    filename), this));
    }
    } catch (IOException ex) {
    remarks.add(new
    CheckResult(CheckResultInterface.TYPE_RESULT_ERROR,

    Messages.getString("JobEntryDeleteFile.ERROR_0008_CheckResult.File_IOException", //
    $NON-NLS-1$
    realFileName, ex.getMessage()), this));
    }
    }
    }

    New Code:

    public void check(List<CheckResultInterface> remarks, JobMeta jobMeta)
    {
    ValidatorContext ctx = new ValidatorContext();
    putVariableSpace(ctx, getVariables());
    putValidators(ctx, notNullValidator(), fileExistsValidator());
    if (isFailIfFileNotExists()) {
    putFailIfDoesNotExist(ctx, true);
    }
    andValidator().validate(this, "filename", remarks, ctx); //$NON-
    NLS-1$
    }


    --~--~---------~--~----~------------~-------~--~----~
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com
    To unsubscribe from this group, send email to kettle-developers-unsubscribe (AT) g...oups (DOT) com
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en
    -~----------~----~----~----~------~----~------~--~---

  2. #2
    Sven Boden Guest

    Default Re: Kettle 3.0 Job Entry Validation

    Maybe 1 small question... how do you verify jobs via spoon?

    Regards,
    Sven


    On Jul 24, 8:20 pm, Mat Lowery <mat.low... (AT) gmail (DOT) com> wrote:
    > I recently checked in some changes to the Kettle 3.0 code which I
    > wanted to introduce here. The changes relate to check() method of
    > subclasses of JobEntryBase. This method, new in Kettle 3.0, validates
    > that the job entry has all required information and that the
    > information is valid so that it can execute. The paragraphs below go
    > over the changes and the benefits over the initial check() method
    > implementation. Feedback requested!
    >
    > Benefits:
    >
    > While implementing job entry check() methods, I decided to factor out
    > some common code. The benefits of the new code over the old code are:
    >
    > * Single parameterized error message per validation type. When a
    > property is null but shouldn't be, we use a single "Property x cannot
    > be null." message. The old way specified messages specific to each
    > job entry and furthermore specific to each property. This change
    > dramatically reduces translation requirements.
    >
    > * Convention over configuration. Error message keys are constructed
    > according to conventions, then fetched from the resource bundle.
    > There is no need to specify the error message key. Also, when errors
    > occur, CheckResults are by default TYPE_RESULT_ERROR with the ability
    > to change it explicitly to TYPE_RESULT_WARNING.
    >
    > * Reusable validation logic. Checking for a file's existence is now
    > stored in one class-the FileExistsValidator.
    >
    > * Extensible. First, new job entries need not use the new code.
    > Second, if new job entries need validations which do not exist, they
    > simply need to implement JobEntryValidator and add their validator
    > class to the classpath.
    >
    > * Java 5 features: variable length argument lists and static imports.
    >
    > * Based on mature Commons Validator, PredicateUtils from Commons
    > Collections, and FieldChecks from Jakarta Struts 1.x.
    >
    > * Is developed in accordance with the "less XML is better" mantra of
    > the Kettle community.
    >
    > Using:
    >
    > All built-in validator classes are in
    > org.pentaho.di.job.entry.validator. All validators implement
    > JobEntryValidator. JobEntryValidatorUtils contains utility methods to
    > be used by both the JobEntryValidator implementations and optionally
    > the check() method implementations. Validator related error messages
    > by default come from
    > org.pentaho.di.job.entry.messages.validator.properties.
    >
    > If a property requires multiple validators, combine the validators
    > with the AndValidator which Boolean ANDs the results of all given
    > validators. AndValidator also "short-circuits" on a false from any
    > given validator.
    >
    > Improvements:
    >
    > * Parameterized resource bundles. Right now, messages can only come
    > from org.pentaho.di.job.entry.messages.validator.properties.
    >
    > * Remove ValidatorContext and singleton accessor methods like
    > JobEntryValidatorUtils.andValidator(). Instead, specify validator-
    > specific parameters in the constructors of those validator objects
    > like new AndValidator(notBlankValidator(), integerValidator()).
    > Instead of being singletons, validators would created and destroyed
    > inside the check() method.
    >
    > * Allow localized property labels. Currently, the property labels are
    > the Java field names.
    >
    > * Allow per-validation override of error message.
    >
    > * Allow validations that involve dependent fields-e.g. requiredif and
    > validwhen. For example, if setLogfile is true, then make sure logfile
    > is not blank.
    >
    > Old Code:
    >
    > public void check(List<CheckResultInterface> remarks, JobMeta jobMeta)
    > {
    > if (filename == null) {
    > remarks.add(new
    > CheckResult(CheckResultInterface.TYPE_RESULT_ERROR,
    >
    > Messages.getString("JobEntryDeleteFile.CheckResult.No_Filename_Is_Defined"), //
    > $NON-NLS-1$
    > this));
    > } else {
    > remarks.add(new CheckResult(CheckResultInterface.TYPE_RESULT_OK,
    >
    > Messages.getString("JobEntryDeleteFile.CheckResult.Filename_Is_Defined"),
    > this)); //$NON-NLS-1$
    > String realFileName = environmentSubstitute(getFilename());
    > FileObject fileObject = null;
    > try {
    > fileObject = KettleVFS.getFileObject(realFileName);
    > if (fileObject != null) {
    > if (!fileObject.exists() && isFailIfFileNotExists()) {
    > remarks.add(new
    > CheckResult(CheckResultInterface.TYPE_RESULT_ERROR,
    >
    > Messages.getString("JobEntryDeleteFile.CheckResult.File_Does_Not_Exist", //
    > $NON-NLS-1$
    > realFileName), this));
    > } else if (!fileObject.exists() ) {
    > remarks.add(new
    > CheckResult(CheckResultInterface.TYPE_RESULT_OK,
    >
    > Messages.getString("JobEntryDeleteFile.CheckResult.File_Does_Not_Exist", //
    > $NON-NLS-1$
    > realFileName), this));
    > } else {
    > remarks.add(new
    > CheckResult(CheckResultInterface.TYPE_RESULT_OK,
    >
    > Messages.getString("JobEntryDeleteFile.CheckResult.File_Exists", //
    > $NON-NLS-1$
    > realFileName), this));
    > }
    > try {
    > fileObject.close(); // Just being cautious
    > } catch (IOException ignored) {}
    > } else {
    > remarks.add(new
    > CheckResult(CheckResultInterface.TYPE_RESULT_ERROR,
    >
    > Messages.getString("JobEntryDeleteFile.CheckResult.Could_Not_Convert_File", //
    > $NON-NLS-1$
    > filename), this));
    > }
    > } catch (IOException ex) {
    > remarks.add(new
    > CheckResult(CheckResultInterface.TYPE_RESULT_ERROR,
    >
    > Messages.getString("JobEntryDeleteFile.ERROR_0008_CheckResult.File_IOException", //
    > $NON-NLS-1$
    > realFileName, ex.getMessage()), this));
    > }
    > }
    >
    > }
    >
    > New Code:
    >
    > public void check(List<CheckResultInterface> remarks, JobMeta jobMeta)
    > {
    > ValidatorContext ctx = new ValidatorContext();
    > putVariableSpace(ctx, getVariables());
    > putValidators(ctx, notNullValidator(), fileExistsValidator());
    > if (isFailIfFileNotExists()) {
    > putFailIfDoesNotExist(ctx, true);
    > }
    > andValidator().validate(this, "filename", remarks, ctx); //$NON-
    > NLS-1$
    >
    > }



    --~--~---------~--~----~------------~-------~--~----~
    You received this message because you are subscribed to the Google Groups "kettle-developers" group.
    To post to this group, send email to kettle-developers (AT) googlegroups (DOT) com
    To unsubscribe from this group, send email to kettle-developers-unsubscribe (AT) g...oups (DOT) com
    For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en
    -~----------~----~----~----~------~----~------~--~---

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.