Hitachi Vantara Pentaho Community Forums
Results 1 to 13 of 13

Thread: [ I M P O R T A N T ] [ read that ] Bugfix for commons-vfs (ver. 20100924-pentaho)

  1. #1
    Join Date
    Apr 2013
    Posts
    19

    Default [ I M P O R T A N T ] [ read that ] Bugfix for commons-vfs (ver. 20100924-pentaho)

    I'm writing here because sources of commons-vfs library are not in github and i can't make pull request.
    I was trying to run my transformations from jar file. Something like that:
    Code:
    KettleEnvironment.init();
    EnvUtil.environmentInit();
    JobMeta jobMeta = new JobMeta("jar:file:///D:/MavenRepo/ru/company/integration/0.0.6/integration-0.0.6.jar!Catalog/Catalog create.kjb", null);
    Job job = new Job(null, jobMeta);
    job.run();
    And i recieve a crash. Faster way to fix something - it's made it by yourself. So, i've decompile library and found a bug in your sources. Here is patch:
    Code:
    --- c:\temp\thg.16ryzj\Patches.5.33\commons-vfs\20100924-pentaho-patched\src\main\java\org\apache\commons\vfs\provider\zip\ZipFileSystem.java
    +++ c:\temp\thg.16ryzj\Patches.5.43\commons-vfs\20100924-pentaho-patched\src\main\java\org\apache\commons\vfs\provider\zip\ZipFileSystem.java
    @@ -103,12 +103,15 @@
             paramCollection.addAll(ZipFileProvider.capabilities);
         }
    
    
    +    @Override
         public FileObject createFile(FileName paramFileName) throws FileSystemException {
    -        return new ZipFileObject(paramFileName, null, this, false);
    +        String baseName = UriParser.encode(paramFileName.getPath());
    +        if (baseName.startsWith("/"))
    +            baseName = baseName.substring(1);
    +        ZipEntry zipEntry = new ZipEntry(baseName);
    +        FileName rootName = getFileSystemManager().resolveName(getRootName(), baseName);
    +        return new ZipFileObject(rootName, zipEntry, this, true);
         }
    
    
     }
    In color:
    http://forumbgz.ru/user/upload/file30115.png

    I hope it will be usefull for someone.
    Attached Images Attached Images  
    Last edited by Kamapcuc; 04-15-2014 at 07:18 AM.

  2. #2
    Join Date
    Apr 2013
    Posts
    19

    Default

    Sorry for English - i'm not native.

  3. #3
    Join Date
    Apr 2013
    Posts
    19

    Default

    Oh, crap, i've made monkey work (decompiling).
    https://github.com/pentaho/pdi-vfs
    Ok, i'll make pull request.
    Last edited by Kamapcuc; 04-15-2014 at 07:36 AM.

  4. #4
    Join Date
    Sep 2013
    Posts
    235

    Default

    I am curious, since you have you job xml file in classpath - why don't you use ClassLoader to load this xml as a resourse?

    I would suggest you to take a close look at this junits tests:
    https://github.com/pentaho/pentaho-k...Execution.java
    https://github.com/pentaho/pentaho-k...utionTest.java

    which is uses exact class loader style to load different jobs (as xml files) on runtime from classpath, prepare database etc.

    PS: Thank you very much for all your existing pull requests reviews and new patches proposals.

  5. #5
    Join Date
    Apr 2013
    Posts
    19

    Default

    Pull request with more information about bug:
    https://github.com/pentaho/pdi-vfs/pull/3

  6. #6
    Join Date
    Apr 2013
    Posts
    19

    Default

    I am curious, since you have you job xml file in classpath - why don't you use ClassLoader to load this xml as a resourse?
    1) We use maven as build system. I have several repositories, one of them is for transformations and jobs. When i need to make new artifact (jar with all my *.ktr and *.kjb) i just type: "mvn -B release:prepare release:perform". And those command deploys it to our artifactory. Now in my main project i can just add those artifact to dependencies and can use it.

    2) If i will use ClassLoader for starting job, that means i should use JobMeta constructor with InputStream. But InputStream does not know file name/directory. So, JobMeta will not be able to find subtransformations.

    I hope those explain our usecase.
    Last edited by Kamapcuc; 04-15-2014 at 10:25 AM.

  7. #7
    Join Date
    Sep 2013
    Posts
    235

    Default

    As I am understand you have artifact as a jar file, but you try access this jar as a regular zip archive from disk D:/ - from my point of view this make sense if it is mounted remote share folder.

    Otherwise - if it is local folder and you going to use this full path to jar file - that means that this artifact is delivered to you by maven - but not in java classpath of application.

    Since you have API call to
    JobMeta jobMeta = new JobMeta("jar:file:///D:/MavenRepo/ru/company/integration/0.0.6/integration-0.0.6.jar!Catalog/Catalog create.kjb", null);
    ...and if the 'integration-0.0.6.jar' in classpath - you may use advantage of java ClassLoader mechanism to load resource by path like 'Catalog/Catalog create.kjb' - and this will return InputStream to exact .kjb. And this path 'Catalog/Catalog create.kjb' - will be regular among different platforms and will be related to project root.

    But sure - you can use vfs style way to access this jar.

    Thanks.

  8. #8
    Join Date
    Apr 2013
    Posts
    19

    Default

    from my point of view this make sense if it is mounted remote share folder
    No, it's not share folder, not subst or hardlink, just common file system.

    ...and if the 'integration-0.0.6.jar' in classpath - you may use advantage of java ClassLoader mechanism to load resource by path like 'Catalog/Catalog create.kjb' - and this will return InputStream to exact .kjb. And this path 'Catalog/Catalog create.kjb' - will be regular among different platforms and will be related to project root.
    It will not work. Read paragraph 2.
    Last edited by Kamapcuc; 04-16-2014 at 05:35 AM.

  9. #9
    Join Date
    Sep 2013
    Posts
    235

    Default

    Sorry I did not mind inner transformations or jobs.

    I've just recently tested this case on my local environment - and that will be for File file = new File( url.toURI() );
    Code:
    Exception in thread "main" java.lang.IllegalArgumentException: URI is not hierarchical
    or
    Code:
    ...
    2014/04/16 14:08:36 - Start - Unexpected error during transformation metadata load
    ...
    That forced me to see under the hood how the inner transformations or jobs is loaded.
    We have enum (this link) ( ObjectLocationSpecificationMethod )
    https://github.com/pentaho/pentaho-k...ionMethod.java

    and there is only 3 options.

    loading inner transformations is performed here:
    https://github.com/pentaho/pentaho-k...ans.java#L1142

    and we have switch for specification methods.

    I would mention that would be more efficient to have CLASSPATH object location method to have ability to load instances from classpath. And in this case do not disturb VFS as is it my own believe if something is bundled as a .jar it should be processed with regular java mechanism. Otherwise it is like you create a .jar but uses it as a zip archive.

    ...might ask for enchantment request if someone else will find it is usable to have a JAR method to load items for 'ObjectLocationSpecificationMethod' and edits for some classes responsible for load inner transformations or jobs. As I am understand everything (every settings) in kettle can be passed with xml and it is only the question how is jobs relates with inner transformation (how to load ktr or kjb). Except .kettle folder which will be accessed filesystem-way.

    Thank you again.

  10. #10
    Join Date
    Mar 2003
    Posts
    8,085

    Default

    Apache VFS can happily read objects from the classpath. Just use the "res" file-system.

    (See here: http://commons.apache.org/proper/com...lesystems.html )
    Get the latest news and tips and tricks for Pentaho Reporting at the Pentaho Reporting Blog.

  11. #11
    Join Date
    Apr 2013
    Posts
    19

    Default

    So, you say, that it's not a bug?
    Then tell me, what's wrong with that code:
    Code:
    KettleEnvironment.init();
    EnvUtil.environmentInit();
    JobMeta jobMeta = new JobMeta("jar:file:///C:/some_path/some_file_name.jar!some_job_name.kjb", null);
    Job job = new Job(null, jobMeta);
    job.run();
    Couse it seems legal to me, and it causes bug.

    I will try to explain more circumstantially.

    When pdi reads jar/zip file first time, it creates FileObject for each ZipEntry:
    https://github.com/pentaho/pdi-vfs/b...ystem.java#L96
    and creates it's correctly:
    https://github.com/pentaho/pdi-vfs/b...stem.java#L137
    and puts it into the cache (ReferenceQueue):
    https://github.com/pentaho/pdi-vfs/b...stem.java#L357

    After that, all works well for a short time. But after some period (few seconds) garbage collector thread destroys FileObjects and removes links from References. So, now when TransMeta asks for some another file:
    https://github.com/pentaho/pdi-vfs/b...stem.java#L331
    it receives null:
    https://github.com/pentaho/pdi-vfs/b...ache.java#L185
    and tries to recreate him:
    https://github.com/pentaho/pdi-vfs/b...stem.java#L344
    and creates it wrong!:
    https://github.com/pentaho/pdi-vfs/b...stem.java#L173
    So now, when TransMeta getsContent and tries to getInputStream it receives exception:
    https://github.com/pentaho/pdi-vfs/b...ject.java#L159
    becauseFileType.IMAGINARY has no content:
    https://github.com/pentaho/pdi-vfs/b...bject.java#L56

    So, you ought to apply my patch, or fix cache (make it strong).
    Last edited by Kamapcuc; 04-16-2014 at 12:21 PM.

  12. #12
    Join Date
    Apr 2013
    Posts
    19

    Default

    By the way, if you put some breakpoints and try to debug, you will see nothing. Because GC doesn't work at the debug. And that's why it was really asspain for me to find reason of that bug.

  13. #13
    Join Date
    Apr 2013
    Posts
    19

    Default

    Just flame, nothing important.
    О, Дмитрий, вы из Беларуси. Место бага я вам показал, описал как он происходит. В свою артифактори я уже выложил пропатченную версию библиотеки и заслонил ею вашу версию с ошибкой. Так что мне этот патч не нужен, я ради сообщества тут написал - вдруг кому-нибудь тоже понадобится. В общем, моя совесть чиста.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.