Hitachi Vantara Pentaho Community Forums
Results 1 to 3 of 3

Thread: Copying MySQL LONGBLOB column from source table

  1. #1
    Join Date
    Jun 2009
    Posts
    13

    Default Copying MySQL LONGBLOB column from source table

    I am trying to populate a staging table in my data warehouse with data from the source database. Both databases are MySQL 5.0.

    I tried doing a simple Table Input from the source and feed it into a Table Output for the destination. This works for a small number of rows to be copied, but when the dataset gets large, I start seeing "Packet Size too large" errors. Our max packet size parameter is set to 16MB and I don't know if its a good idea to increase that.

    So next, I tried to output the data from the source to a file using Text File Output and then using Bulkload to load the data into the target. However, it seems like the LARGEBLOB column is getting corrupted. My guess is that the BLOB contains newline characters that are causing this to happen. When I look at the text file, I do see some of the BLOB values spilling over into multiple lines.

    I am relatively new to PDI as well as MySQL, so I'm wondering if I'm overlooking some simple solution to this problem. Any suggestions are most welcome.

    Thanks,
    Kaushal

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Interesting. So I Googled for the error. It doesn't look good as it appears to be a MySQL limitation, not as much a PDI limitation.

    It would probably be a better idea to dump the blobs to a binary file each.
    Some JavaScript would allow you to do that.

    Matt

  3. #3
    Join Date
    Jun 2009
    Posts
    13

    Default

    Can you please elaborate on "dumping the blobs to a binary file each"? I have a single LONGBLOB column in the source table. I need to pick that up along with a few other columns from the source and one other table that is joined in the query.


    Are you talking about writing javascript to write the value being read to a file and keep appending rows to it? If so, how would I then merge this with the other columns that I also need to load?

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.