Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: Kettle 3.0 Job Entry Validation

  1. #1
    Join Date
    Oct 2006
    Posts
    817

    Default Kettle 3.0 Job Entry Validation

    I recently checked in some changes to the Kettle 3.0 code which I wanted to introduce here. The changes relate to check() method of subclasses of JobEntryBase. This method, new in Kettle 3.0, validates that the job entry has all required information and that the information is valid so that it can execute. The paragraphs below go over the changes and the benefits over the initial check() method implementation. The same text can be found in the attached PDF which is slightly more readable. Feedback requested!

    Benefits:

    While implementing job entry check() methods, I decided to factor out some common code. The benefits of the new code over the old code are:

    * Single parameterized error message per validation type. When a property is null but shouldn't be, we use a single "Property x cannot be null." message. The old way specified messages specific to each job entry and furthermore specific to each property. This change dramatically reduces translation requirements.

    * Convention over configuration. Error message keys are constructed according to conventions, then fetched from the resource bundle. There is no need to specify the error message key. Also, when errors occur, CheckResults are by default TYPE_RESULT_ERROR with the ability to change it explicitly to TYPE_RESULT_WARNING.

    * Reusable validation logic. Checking for a file's existence is now stored in one class—the FileExistsValidator.

    * Extensible. First, new job entries need not use the new code. Second, if new job entries need validations which do not exist, they simply need to implement JobEntryValidator and add their validator class to the classpath.

    * Java 5 features: variable length argument lists and static imports.

    * Based on mature Commons Validator, PredicateUtils from Commons Collections, and FieldChecks from Jakarta Struts 1.x.

    * Is developed in accordance with the "less XML is better" mantra of the Kettle community.

    Using:

    All built-in validator classes are in org.pentaho.di.job.entry.validator. All validators implement JobEntryValidator. JobEntryValidatorUtils contains utility methods to be used by both the JobEntryValidator implementations and optionally the check() method implementations. Validator related error messages by default come from org.pentaho.di.job.entry.messages.validator.properties.

    If a property requires multiple validators, combine the validators with the AndValidator which Boolean ANDs the results of all given validators. AndValidator also “short-circuits” on a false from any given validator.

    Improvements:

    * Parameterized resource bundles. Right now, messages can only come from org.pentaho.di.job.entry.messages.validator.properties.

    * Remove ValidatorContext and singleton accessor methods like JobEntryValidatorUtils.andValidator(). Instead, specify validator-specific parameters in the constructors of those validator objects like new AndValidator(notBlankValidator(), integerValidator()). Instead of being singletons, validators would created and destroyed inside the check() method.

    * Allow localized property labels. Currently, the property labels are the Java field names.

    * Allow per-validation override of error message.

    * Allow validations that involve dependent fields—e.g. requiredif and validwhen. For example, if setLogfile is true, then make sure logfile is not blank.

    Old Code:

    Code:
    public void check(List<CheckResultInterface> remarks, JobMeta jobMeta) {
      if (filename == null) {
        remarks.add(new CheckResult(CheckResultInterface.TYPE_RESULT_ERROR,
          Messages.getString("JobEntryDeleteFile.CheckResult.No_Filename_Is_Defined"), //$NON-NLS-1$
          this));
      } else {
        remarks.add(new CheckResult(CheckResultInterface.TYPE_RESULT_OK,
          Messages.getString("JobEntryDeleteFile.CheckResult.Filename_Is_Defined"), this)); //$NON-NLS-1$
        String realFileName = environmentSubstitute(getFilename());
        FileObject fileObject = null;
        try {
          fileObject = KettleVFS.getFileObject(realFileName);
          if (fileObject != null) {
            if (!fileObject.exists() && isFailIfFileNotExists()) {
              remarks.add(new CheckResult(CheckResultInterface.TYPE_RESULT_ERROR,
                Messages.getString("JobEntryDeleteFile.CheckResult.File_Does_Not_Exist", //$NON-NLS-1$
                  realFileName), this));
            } else if (!fileObject.exists() ) {
              remarks.add(new CheckResult(CheckResultInterface.TYPE_RESULT_OK,
              Messages.getString("JobEntryDeleteFile.CheckResult.File_Does_Not_Exist", //$NON-NLS-1$
                realFileName), this));
            } else {
              remarks.add(new CheckResult(CheckResultInterface.TYPE_RESULT_OK,
              Messages.getString("JobEntryDeleteFile.CheckResult.File_Exists", //$NON-NLS-1$
                realFileName), this));
            }
            try {
              fileObject.close(); // Just being cautious
            } catch (IOException ignored) {}
          } else {
            remarks.add(new CheckResult(CheckResultInterface.TYPE_RESULT_ERROR,
            Messages.getString("JobEntryDeleteFile.CheckResult.Could_Not_Convert_File", //$NON-NLS-1$
              filename), this));
          }
        } catch (IOException ex) {
          remarks.add(new CheckResult(CheckResultInterface.TYPE_RESULT_ERROR,
          Messages.getString("JobEntryDeleteFile.ERROR_0008_CheckResult.File_IOException", //$NON-NLS-1$
            realFileName, ex.getMessage()), this));
        }
      } 
    }
    New Code:

    Code:
    public void check(List<CheckResultInterface> remarks, JobMeta jobMeta) {
      ValidatorContext ctx = new ValidatorContext();
      putVariableSpace(ctx, getVariables());
      putValidators(ctx, notNullValidator(), fileExistsValidator());
      if (isFailIfFileNotExists()) {
        putFailIfDoesNotExist(ctx, true);
      }
      andValidator().validate(this, "filename", remarks, ctx); //$NON-NLS-1$
    }
    Attached Images Attached Images

  2. #2
    Join Date
    May 2006
    Posts
    4,882

    Default

    Maybe 1 small question... how do you verify jobs via spoon?

    Regards,
    Sven

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.