Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: Text file output - "Could not create file" exception

  1. #1

    Default Text file output - "Could not create file" exception

    Hello

    I have a transformation processing 50,000 XML files in a directory.

    1) Load file content in memory - reads XML files

    2) Replace in string - inject some values into XML

    3) HTTP Post - POST XML to web service

    4) Text file output - write response to a file

    After ~3,500 rows (files written) I get an exception:

    a) '/dataload/UAT/CLRSCR/output/CS-20160314161718' folder does exist
    b) the file isn't already there
    c) ulimit is 65536
    d) df -i = 12635627 free

    2016/03/14 16:57:23 - Text file output - success.0 - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : org.pentaho.di.core.exception.KettleException:
    2016/03/14 16:57:23 - Text file output - success.0 - Error opening new file : org.pentaho.di.core.exception.KettleFileException:
    2016/03/14 16:57:23 - Text file output - success.0 -
    2016/03/14 16:57:23 - Text file output - success.0 - org.apache.commons.vfs2.FileSystemException: Could not create file "file:///dataload/UAT/CLRSCR/output/CS-20160314161718/CS_DAV_53530_0050_01.xml".
    2016/03/14 16:57:23 - Text file output - success.0 - Could not create file "file:///dataload/UAT/CLRSCR/output/CS-20160314161718/CS_DAV_53530_0050_01.xml".
    2016/03/14 16:57:23 - Text file output - success.0 -
    2016/03/14 16:57:23 - Text file output - success.0 - Could not create file "file:///dataload/UAT/CLRSCR/output/CS-20160314161718/CS_DAV_53530_0050_01.xml".
    2016/03/14 16:57:23 - Text file output - success.0 -
    2016/03/14 16:57:23 - Text file output - success.0 -
    2016/03/14 16:57:23 - Text file output - success.0 - at org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.openNewFile(TextFileOutput.java:655)
    2016/03/14 16:57:23 - Text file output - success.0 - at org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.setDataWriterForFilename(TextFileOutput.java:219)
    2016/03/14 16:57:23 - Text file output - success.0 - at org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.processRow(TextFileOutput.java:193)
    2016/03/14 16:57:23 - Text file output - success.0 - at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
    2016/03/14 16:57:23 - Text file output - success.0 - at java.lang.Thread.run(Thread.java:744)

    Any ideas?

    Thanks

  2. #2
    Join Date
    Apr 2008
    Posts
    1,771

    Default

    Have you checked that PDI can create a file in that folder?
    It could be that the user who runs PDi does not have enough permissions.
    -- Mick --

  3. #3

    Default

    Hello

    The permissions are correct - as 3,500 files are created successfully in the same transformation.

    Mark

  4. #4
    Join Date
    Aug 2011
    Posts
    360

    Default

    I think the text file output step is keeping open the handlers to each file (must look at the code but)
    until it gets no rows anymore.
    So maybe it runs out of file handlers.

    Maybe you should loop in a job to threat only say 1000 files per batch

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.