Hitachi Vantara Pentaho Community Forums
Results 1 to 25 of 25

Thread: newMapComponent - dynamic source for shape information?

  1. #1
    Join Date
    Sep 2014
    Posts
    12

    Default newMapComponent - dynamic source for shape information?

    The NewMapComponent for CDE is great, but it would be even better if we could get the list of shape information dynamically from a geodatabase, instead of a fixed KML file. Some KML files are too large, and without SQL queries would needlessly clog the application with unneeded data.

    Is there any way to do that?

  2. #2
    Join Date
    Jan 2014
    Posts
    189

    Post

    Hi stanasa,

    if the component is rendering markers instead of shapes, such a feature is already in place. It works by delegating the resolution of coordinates to an add-in.
    In other words, if the query contains human-friendly columns such as "City", "country", an http request is launched for every row, which will lookup the latitude and longitude dynamically.

    This approach is convenient for small data sets but has the drawback that it increases the time it takes to populate the map with data, as it becomes dependent on a service that may have a big lag. Nominatin, for instance, is a free service that limits the number of requests per second to about 1. So, if you have 30 markers, your map takes about 30 seconds to render.

    But your question is about shapes, and in that mode, the location resolving add-in is not invoked.
    While it might be a good idea to create a JIRA case to ask for this "add-in encapsulating a geo-shape resolving service" feature request, this does not solve your problem today.

    What could solve your problem is the hack I'm about to describe next (hack = it might stop working in the next CTools release).
    It violates the principle that business data and presentation data should be kept separate, but here it goes:

    1) augment your query with a third column containing a serialized list of coordinates, corresponding to the desired shape.

    For instance, if you choose to serialize using JSON.stringify, you would have a query that outputs something like:

    | key | value | shape
    -----------------------------
    | "US" | 100 | "[[[[19.33,-155.1], ...."
    | "CN" | 100 | "[[[[20.06,110.72], ...."


    2) In postFetch, build a dictionary containing the map key and the corresponding coordinates and assign them to the component "shapeDefinition" property:

    function f(json){

    this.shapeDefinition = _.chain( json.resultset ).map(function (row){
    return [row[0], JSON.parse(row[2]) ];
    }).object().value();

    return json;
    }


    Of course, you could also implement a synchronous ajax request that fetches the list of coordinates for each row, but that is a tad more complex thing to do.
    Last edited by crusso; 04-04-2015 at 05:53 AM.

  3. #3
    Join Date
    Sep 2014
    Posts
    12

    Default

    Thanks, crusso, I will give it a try and get back with an update.

  4. #4
    Join Date
    Feb 2017
    Posts
    20

    Default

    i tried this with newmapcomponent:

    map engine: openlayers
    operation mode: shapes
    tilesets to display as layers: default
    datasource: one datasource with:
    | key | value | shape
    -----------------------------
    | 1100015 | 100 | "[[[[19.33,-155.1], ...."
    | 1100023 | 100 | "[[[[20.06,110.72], ...."


    but there are no shapes in the map, any help?

    (sorry my bad english)

  5. #5
    Join Date
    Jan 2014
    Posts
    189

    Default

    Hi hugonr

    The solution I presented earlier was merely an outline of solution, not a fully tested solution. In fact, I did not test it at all.
    You simply should not expect random snippets to work immediately.
    Just keep in mind that sometimes the people replying to questions in this forum are merely human and not infallible gods.

    First of all did you paste the snippet of code into postFetch?
    If you did, and the code didn't work, are there any errors in the browser (web) console?
    What version of Pentaho or CDE are you using?
    That sort of information is useful. If you don't supply that information, then the code demi-gods have to guess why things don't work.

    Also, looking at the source code of NewMapComponent, it seems the snippet presented earlier will not work in Pentaho 7.0 and later.

    For Pentaho 7.0+, the approach must be via a "ShapeResolver" addIn.
    The addIn must iterate over each row to look up the string in the column "shape" and transform it to a GeoJSON object.
    It must also build a object whose keys are contents of the column "key", and whose values are the GeoJSON objects.
    This is not terribly complicated to do, but it takes more than 2 minutes to write and 15 minutes to test.
    Last edited by crusso; 05-28-2017 at 11:10 AM.

  6. #6
    Join Date
    Feb 2017
    Posts
    20

    Default

    thanks for the reply crusso!!!

    yes, this i need to work with this information, but i just got one kml file with all the data i need so now i will try with this file.

    i think will be better than using it from datasource.

    thanks!!!!!

  7. #7
    Join Date
    Feb 2017
    Posts
    20

    Default

    Hi Crusso!!!

    i will try again this aproach, here are the answers:

    First of all did you paste the snippet of code into postFetch?
    Yes, in postfetch
    function f(json){
    this.shapeDefinition = _.chain( json.resultset ).map(function (row){
    return [row[0], JSON.parse(row[2]) ];
    }).object().value();
    console.log(this);
    return json;
    }
    If you did, and the code didn't work, are there any errors in the browser (web) console?
    No, and my object now have the ShapeDefinition with the keys and coordinates
    observation: without the code in postfetch i dont have shapedefinitions

    What version of Pentaho or CDE are you using?
    Release 6.1.0.1.196

    I will really appreciate your help!!!

  8. #8
    Join Date
    Jan 2014
    Posts
    189

    Default

    You'll need the addIn.

    Upload the following as a .js file into PUC, and include it as a resource in your dashboard:
    Code:
    define([
      "cdf/AddIn",
      "cdf/Dashboard.Clean",
      "cdf/lib/jquery",
      "amd!cdf/lib/underscore"
    ], function(AddIn, Dashboard, $, _) {
      "use strict";
      var inlineGeoJson = {
        name: "inlineGeoJSON",
        label: "Inline GeoJSON shape resolver",
        defaults: {
          colIdx: null, //column index containing the json map definitions
          idPropertyName: "" //GeoJSON feature property that will be used to index the feature
        },
        implementation: function(tgt, st, opt) {
    
          var map = _.chain(st.tableData)
            .map(function(row, idx){
              return [
                st.ids[idx], polygonToGeoJson(row[opt.colIdx])
              ]
            })
            .object()
            .value();
    
          var deferred = $.Deferred();
          deferred.resolve(map);
          return deferred.promise();
        }
      };
    
    
      function polygonToGeoJson(coordinatesStr){
        return {
          type: "Feature",
          geometry: {
            type: "Polygon",
            coordinates: JSON.parse(coordinatesStr)
          }
        }
      }
    
    
      Dashboard.registerGlobalAddIn("NewMapComponent", "ShapeResolver", new AddIn(inlineGeoJson));
    
    
      return inlineGeoJson;
    });
    Then, in your map component, add the following in preExecution:

    Code:
    function(){
      this.shapeResolver = "inlineGeoJson";
      this.setAddInOptions("ShapeResolver", "inlineGeoJson", { colIdx: 2});
    }
    I did not test this. This should get you started, though.
    Last edited by crusso; 06-20-2017 at 02:08 PM. Reason: improved legibility

  9. #9
    Join Date
    Feb 2017
    Posts
    20

    Default

    The steps i did:

    1. in my PUC browse files i uploaded the file addinshape.js (with the addin code),
    2. i created one dashboard with the query of this post and one newmapcomponent with the configuration i did in the last one (see my reply of this post),
    3. in my PUC layout panel i added a resource (javascript/external file), and choose the addinshape.js,
    4. in my newmapcomponent i used that pre execution code.

    i tried with the postfetch and without it:
    function f(json){
    this.shapeDefinition = _.chain( json.resultset ).map(function (row){
    return [row[0], JSON.parse(row[2]) ];
    }).object().value();
    console.log(this);
    return json;
    }
    did not work, but its a start, i will study this solution better and will try it!

    thanks again crusso!!!!!!

  10. #10
    Join Date
    Jan 2014
    Posts
    189

    Default

    You should not need to add anything to postFetch.

    By the way, there was a glitch: it should be row[opt.colIdx] and not st.tableData[idx, opt.colIdx]) ...


  11. #11
    Join Date
    Feb 2017
    Posts
    20

    Default

    i got this error:
    addinshape.js?v=1497978999742:44 Uncaught ReferenceError: inlineGeoJSON is not defined
    at addinshape.js?v=1497978999742:44
    at Object.define (cdf-bootstrap-script-includes.js?v=decdbcb…:54)
    at define (cdf-bootstrap-script-includes.js?v=decdbcb…:54)
    at addinshape.js?v=1497978999742:1
    i will try to solve it here
    Last edited by hugonr; 06-20-2017 at 01:18 PM. Reason: double post.

  12. #12
    Join Date
    Feb 2017
    Posts
    20

    Default

    This is the data from my query:

    Code:
    key     |fill |shape                                                                                             
    --------|-----|--------------------------------------------------------------------------------------------------
    3100104 |100  |[[[[-47.429672447,-18.1654308175596],[-47.429427,-18.1660609975595],[-47.4293999999999,-18.1664979
    3100203 |100  |[[[[-45.1392989999999,-19.2104779965869],[-45.139383,-19.2103539965869],[-45.139648,-19.2099619965
    3100302 |100  |[[[[-42.310709,-20.1427889966106],[-42.310904,-20.1429489966106],[-42.3110859999999,-20.1430979966
    3100401 |100  |[[[[-43.023869,-20.4102179966173],[-43.023848,-20.4102799966173],[-43.02379,-20.4105099966173],[-4
    3100500 |100  |[[[[-42.3620830000001,-18.9256939975795],[-42.3621019999999,-18.9257289965795],[-42.362184,-18.926
    3100609 |100  |[[[[-42.322711,-17.7842679965493],[-42.32248,-17.7844979965493],[-42.322056,-17.7847289965494],[-4
    and the script its like this now:

    Code:
    define([
      "cdf/AddIn",
      "cdf/Dashboard.Clean",
      "cdf/lib/jquery",
      "amd!cdf/lib/underscore"
    ], function(AddIn, Dashboard, $, _) {
      "use strict";
      var inlineGeoJSON = {
        name: "inlineGeoJSON",
        label: "Inline GeoJSON shape resolver",
        defaults: {
          colIdx: null, //column index containing the json map definitions
          idPropertyName: "" //GeoJSON feature property that will be used to index the feature
        },
        implementation: function(tgt, st, opt) {
    
    
          var map = _.chain(st.tableData)
            .map(function(row, idx){
              return [
                st.ids[idx], polygonToGeoJson(row[opt.colIdx])
              ];
            })
            .object()
            .value();
    
    
          var deferred = $.Deferred();
          deferred.resolve(map);
          return deferred.promise();
        }
      };
    
    
      function polygonToGeoJson(coordinatesStr){
        return {
          type: "Feature",
          geometry: {
            type: "Polygon",
            coordinates: JSON.parse(coordinatesStr)
          }
        };
      }
    
    
      Dashboard.registerGlobalAddIn("NewMapComponent", "ShapeResolver", new AddIn(inlineGeoJSON));
    
    
      return inlineGeoJSON;
    });
    the error: one ";" in the polygonToGeoJson and the caps in the var name "inlineGeoJSON".

    now there are no errors, is not working but i will try from this code, thanks again Crusso!!!!!!
    Last edited by hugonr; 06-20-2017 at 01:42 PM. Reason: ident / correct code / insert data from query

  13. #13
    Join Date
    Feb 2017
    Posts
    20

    Default

    Quote Originally Posted by hugonr View Post
    This is the data from my query:

    Code:
    key     |fill |shape                                                                                             
    --------|-----|--------------------------------------------------------------------------------------------------
    3100104 |100  |[[[[-47.429672447,-18.1654308175596],[-47.429427,-18.1660609975595],[-47.4293999999999,-18.1664979
    3100203 |100  |[[[[-45.1392989999999,-19.2104779965869],[-45.139383,-19.2103539965869],[-45.139648,-19.2099619965
    3100302 |100  |[[[[-42.310709,-20.1427889966106],[-42.310904,-20.1429489966106],[-42.3110859999999,-20.1430979966
    3100401 |100  |[[[[-43.023869,-20.4102179966173],[-43.023848,-20.4102799966173],[-43.02379,-20.4105099966173],[-4
    3100500 |100  |[[[[-42.3620830000001,-18.9256939975795],[-42.3621019999999,-18.9257289965795],[-42.362184,-18.926
    3100609 |100  |[[[[-42.322711,-17.7842679965493],[-42.32248,-17.7844979965493],[-42.322056,-17.7847289965494],[-4
    and the script its like this now:

    Code:
    define([
      "cdf/AddIn",
      "cdf/Dashboard.Clean",
      "cdf/lib/jquery",
      "amd!cdf/lib/underscore"
    ], function(AddIn, Dashboard, $, _) {
      "use strict";
      var inlineGeoJSON = {
        name: "inlineGeoJSON",
        label: "Inline GeoJSON shape resolver",
        defaults: {
          colIdx: null, //column index containing the json map definitions
          idPropertyName: "" //GeoJSON feature property that will be used to index the feature
        },
        implementation: function(tgt, st, opt) {
    
    
          var map = _.chain(st.tableData)
            .map(function(row, idx){
              return [
                st.ids[idx], polygonToGeoJson(row[opt.colIdx])
              ];
            })
            .object()
            .value();
    
    
          var deferred = $.Deferred();
          deferred.resolve(map);
          return deferred.promise();
        }
      };
    
    
      function polygonToGeoJson(coordinatesStr){
        return {
          type: "Feature",
          geometry: {
            type: "Polygon",
            coordinates: JSON.parse(coordinatesStr)
          }
        };
      }
    
    
      Dashboard.registerGlobalAddIn("NewMapComponent", "ShapeResolver", new AddIn(inlineGeoJSON));
    
    
      return inlineGeoJSON;
    });
    the error: one ";" in the polygonToGeoJson and the caps in the var name "inlineGeoJSON".

    now there are no errors, is not working but i will try from this code, thanks again Crusso!!!!!!
    Update:
    in my pre-execution code i got the caps problem, so i change it to this:

    Code:
    function f(){
      this.shapeResolver = "inlineGeoJSON";
      this.setAddInOptions("ShapeResolver", "inlineGeoJSON", { colIdx: 2});
    }
    correct caps.

    and now i got this error in console:

    CDF: Addin Error [inlineGeoJSON]: SyntaxError: Unexpected token o in JSON at position 1


    Dashboards.log @ cdf-bootstrap-script-includes.js?v=decdbcb…:4546
    AddIn.call @ cdf-bootstrap-script-includes.js?v=decdbcb…:4816
    resolveShapes @ CDF.js?v=ac929f4…:102
    resolveFeatures @ CDF.js?v=ac929f4…:107
    onDataReady @ CDF.js?v=ac929f4…:158
    (anonymous) @ cdf-bootstrap-script-includes.js?v=decdbcb…:5706
    (anonymous) @ cdf-bootstrap-script-includes.js?v=decdbcb…:4836
    fire @ cdf-bootstrap-script-includes.js?v=decdbcb…:176
    fireWith @ cdf-bootstrap-script-includes.js?v=decdbcb…:181
    done @ cdf-bootstrap-script-includes.js?v=decdbcb…:637
    callback @ cdf-bootstrap-script-includes.js?v=decdbcb…:673

    trying here to find out what is wrong

    UPDATE:
    Should be JSON.stringify and not JSON.parse?
    with that change, i got no errors, but i have no shapes yet.
    Last edited by hugonr; 06-20-2017 at 03:08 PM. Reason: maybe the parse?

  14. #14
    Join Date
    Jan 2014
    Posts
    189

    Default

    The shape column is supposed to be of type string, hence the JSON. parse.

  15. #15
    Join Date
    Jan 2014
    Posts
    189

    Default

    oh, it's not st.tableData[][], it's st.tableData.resultset[][]

  16. #16
    Join Date
    Jan 2014
    Posts
    189

  17. #17
    Join Date
    Feb 2017
    Posts
    20

    Default

    Quote Originally Posted by crusso View Post
    i tried, only corrected this:
    Code:
    function(){
      this.shapeResolver = "inlineGeoJSON";
      this.setAddInOptions("ShapeResolver", "inlineGeoJSON", { colIdx: 2});
    }
    because of caps.

    in the js file i logged the parse, and its showing the parse from my (now text) coordinates:

    Code:
      function polygonToGeoJson(coordinatesStr){
        console.log(JSON.parse(coordinatesStr));
        return {
          type: "Feature",
          geometry: {
            type: "Polygon",
            coordinates: JSON.parse(coordinatesStr)
          }
        };
      }
    but still i have no shapes.

    any ideas?

    (i will try here some other things in that code)
    Last edited by hugonr; 06-22-2017 at 01:42 PM. Reason: final result

  18. #18
    Join Date
    Feb 2017
    Posts
    20

    Default

    i did a console.log in the map var, and got this:

    Code:
    1100130: Object
      geometry: Object
        coordinates: Array(1)
          0: Array(1)
          length: 1
          __proto__: Array(0)
          type: "Polygon"
        __proto__: Object
        type: "Feature"
      __proto__: Object
    when i did a select in my database i got this (not my datasource, but the data from my table) i got this:
    Code:
    {"type":"MultiPolygon","coordinates":[[[[-62.050437771,-11.8673517413802],[-62.048899646,-11.8680616163802
    maybe this can help? Polygon and MultiPolygon?

    cheers

  19. #19
    Join Date
    Feb 2017
    Posts
    20

    Default

    OH YESSSSSS!!!!!!!!!!

    Code:
      function polygonToGeoJson(coordinatesStr){
        return {
          type: "Feature",
          geometry: {
            type: "MultiPolygon",
            coordinates: JSON.parse(coordinatesStr)
          }
        };
      }
    I changed this and now i have the shapes, all black, now i will work in the colors....

    Crusso, REAAAAAAALLY THANKS!!!!!!!!!

  20. #20
    Join Date
    Feb 2017
    Posts
    20

    Default

    For future reference (corrected codes, maybe other people will use it):

    https://gist.github.com/hugonr/64792...e89a903c5b51df

  21. #21
    Join Date
    Jan 2014
    Posts
    189

    Default

    Great news!
    Now, one question: if you already have access to the GeoJSON feature, why don't you populate the column "shape" with that?
    Imagine that you had the layout:

    Code:
    key     |fill |geoJSON                                                                                             
    --------|-----|--------------------------------------------------------------------------------------------------
    3100104 |100  |{"type":"MultiPolygon","coordinates":[[[[-47.429672447,-18.1654308175596],[-47.429427,-18.1660609975595],[-47.4293999999999,-18.16649793100203 |100  |{"type":"MultiPolygon","coordinates": [[[[-45.1392989999999,-19.2104779965869],[-45.139383,-19.2103539965869],[-45.139648,-19.2099619965
    Then, the addIn wouldn't need to assume that the feature was a multipolygon, and would merely need to do a simple JSON.parse
    I think it would be more general, because it would allow you to have a mixture of polygons, multipolygons, linestrings, etc

  22. #22
    Join Date
    Feb 2017
    Posts
    20

    Default

    Quote Originally Posted by crusso View Post
    Great news!
    Now, one question: if you already have access to the GeoJSON feature, why don't you populate the column "shape" with that?
    Imagine that you had the layout:

    Code:
    key     |fill |geoJSON                                                                                             
    --------|-----|--------------------------------------------------------------------------------------------------
    3100104 |100  |{"type":"MultiPolygon","coordinates":[[[[-47.429672447,-18.1654308175596],[-47.429427,-18.1660609975595],[-47.4293999999999,-18.16649793100203 |100  |{"type":"MultiPolygon","coordinates": [[[[-45.1392989999999,-19.2104779965869],[-45.139383,-19.2103539965869],[-45.139648,-19.2099619965
    Then, the addIn wouldn't need to assume that the feature was a multipolygon, and would merely need to do a simple JSON.parse
    I think it would be more general, because it would allow you to have a mixture of polygons, multipolygons, linestrings, etc
    FANTASTIC!!!! i will try that!!!!

  23. #23
    Join Date
    Feb 2017
    Posts
    20

    Default

    and the new script, works like a charm:
    Code:
    define([
      "cdf/AddIn",
      "cdf/Dashboard.Clean",
      "cdf/lib/jquery",
      "amd!cdf/lib/underscore"
    ], function(AddIn, Dashboard, $, _) {
      
      "use strict";
      
      var inlineGeoJSON = {
        name: "inlineGeoJSON",
        label: "Inline GeoJSON shape resolver",
        defaults: {
          colIdx: null, //column index containing the json map definitions
          idPropertyName: "" //GeoJSON feature property that will be used to index the feature
        },
        implementation: function(tgt, st, opt) {
    
    
          var map = _.chain(st.tableData.resultset)
            .map(function(row, idx){
              return [
                st.ids[idx], polygonToGeoJson(row[opt.colIdx])
              ];
            })
            .object()
            .value();
    
    
          var deferred = $.Deferred();
          deferred.resolve(map);
          return deferred.promise();
        }
      };
    
    
      function polygonToGeoJson(coordinatesStr){
        return {
          type: "Feature",
          geometry: JSON.parse(coordinatesStr)
        };
      }
    
    
      Dashboard.registerGlobalAddIn("NewMapComponent", "ShapeResolver", new AddIn(inlineGeoJSON));
      return inlineGeoJSON;
    });
    with this query result:

    Code:
    key     |fill |geoJSON                                                                                             
    --------|-----|--------------------------------------------------------------------------------------------------
    3100104 |100  |{"type":"MultiPolygon","coordinates":[[[[-47.429672447,-18.1654308175596],[-47.429427,-18.1660609975595],[-47.4293999999999,-18.16649793100203 |100  |{"type":"MultiPolygon","coordinates": [[[[-45.1392989999999,-19.2104779965869],[-45.139383,-19.2103539965869],[-45.139648,-19.2099619965
    Last edited by hugonr; 06-29-2017 at 02:42 PM. Reason: correct code

  24. #24
    Join Date
    Feb 2017
    Posts
    20

    Default

    actually works with 15 records, when i tried like 50 records i got a memory error from chrome....

    i cant figure out how to solve this problem yet.

    cheers!

    update: stops to work with 22 records.

    update 2: tried 300 records in Internet Explorer and works... i need to see what is the problem with chrome
    Last edited by hugonr; 06-29-2017 at 02:53 PM. Reason: update 2

  25. #25
    Join Date
    Jan 2014
    Posts
    189

    Default

    How complex are your polygons?
    In other words, what is the size of the doQuery call that returns the resultset? If it is over a few tens of megabytes, I would expect a slowdown. If it over a few hundred megabytes, yes, it might blow.

    The map component ships with a method for simplifying the geometries, based on the Ramer-Douglas–Peucker algorithm, and is meant to be used when the user is unable to do it server-side.
    It is available from within the addIn as "st._simplifyPoints" . The source code is at https://github.com/webdetails/cde/bl...eConversion.js

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.