SecureFilter and UTF-8 choices

[Petr Janata]

Hello,

I try to use SecureFilterComponent to get selection from user. Selection choices contain non-ascii characters(like č,ř,ů; special Czech charcters). It is not possible to select these choices.

When submitting choices without these special characters(only ascii) action seqence works OK. But if I submit choice containing these characters, I am redirected back to the same secureFilter asking for the option I already selected.

I can see no error in the log.

Sample action sequence is in attachment.(simple.xaction)

Selecting 'aaa' is OK. Selecting 'ščř' leads to redirect to the same page with filter asking again for the same parameter. In generated URL I can see chosen parameter(ščř) encoded to '%u0161%u010D%u0159'.

I would not mind decoding these unicode escapes with JavascriptRule in xaction, but secure filter won't let me pass.
(with_decode.xaction)

I would expect SecureFilterComponent either:
1. let me pass because I chose valid selection
or
2. raise a validation error because it cannot match the encoded parameter

Am I doing something wrong? Did anybody solved this? I would appreciate any comment.

Peter

[fabien.meurillon]

Do you really need those specials characters as value of your choices ?
Usually, I'm using french specials characters only in the label of the options, and put in the value a representative ASCII string.
If it doesn't fit to your problem, I have no idea.

[gmoran]

Peter,

I looked into this, and it seems that the parameter and value are getting passed back to the server, but an ACEGI-wrapped request object loses the parameter when we attempt to retrieve its value. So I've created a case for this, and hopefully our ACEGI guru can look into it quickly. To follow progress on the case, visit http://jira.pentaho.org/browse/BISERVER-82.

kind regards,
Gretchen

[Petr Janata]

Thanks for the effort Gretchen.

I have similar problem with PivotViewComponent.

If I use special czech characters in variables used in MDX query I am not able to display pivot table. If I use czech characters directly in MDX query then it works.

Company_hardcoded.xaction works fine:

[Task].[All Tasks].[Ministerstvo financi ČR]

Company_variable.xaction produces error:

company_name="Ministerstvo financi ČR"
[Task].[All Tasks].[{company_name}]

produces mondrian error:
Caused by: mondrian.olap.MondrianException: Mondrian Error:MDX object '[Task].[All Tasks].[Ministerstvo financi R].[SEODA]' not found in cube


I appended log from jboss.
(line 12 of log) when adding default value to company_name "Č" is ok.
(line 67) but when being processed in security filter chain, there is squere symbol instead of "Č".
(line 339) which produces error eventually

I guess this issue has the same cause as the one with secureFilterComponent mentioned earlier in this thread.

Note: you have got some problems with UTF-8 on this forum.
In word "financi" the second "i" should be i with acute accent (looks like i', ascii code 161 decimal), but here it displays like "�*", so I replaced it in forum. So don't be confused that in server.log the name of entity is little different. Btw it means "Ministry of finance of Czech Republic".

[mlowery]

Please see BISERVER-82 for a solution to your initial problem. As far as PivotViewComponent, if it uses an HTML form, then the aforementioned solution should work there as well.

[Petr Janata]

I followed the steps from http://jira.pentaho.org/browse/BISERVER-82 and it actually helped. A little.

Strings containing UTF-8 characters are now succesfully passed to xaction, but new problem is, that xaction now responses to new browser window.

Try running the simple.xaction i had attached. When I click on button "Run Report" to submit selection, new window opens up with result of selection.

I would like to see the result in the same window, especially if I want to display several selection filters, each containing choices dependent on previous selections.
(I succesfully used the ajax solution from /pentaho_solutions/samples/reporting/dep-param/DependentParameterExample.xaction to get dependent parameters, but I plan to use theese solutions on portal, where I assume this ajax solution will not work)

As concerning the problem with PivotViewComponent:
It persists.
So please, look at my example xactions. I know its lot of work to reproduce this problem, but please try.
The workaround helped only with secureFilterComponent because the default template for sfc has been changed to use post method instead of get. I cannot tell, how is the PivotViewComponent invoked from xaction processing, but it also needs to change the posting method.

Summary:
Resolved:
UTF-8 choices in sfc works.
New bugs:
1. scf now enforces opening new window in browser on steps following the scf
2. PivotViewComponent needs to be called with post method from xaction, otherwise its incapable of accepting UTF-8 parameters correctly

I hope You guys are not mad with me, my only sin is that I found theese bugs

[mlowery]

As far as changing the "open new window" behavior, open pentaho-solutions/system/custom/xsl/ParameterFormUtil.xsl. Search for <form>. About three lines down from that should be a line adding a target attribute to the form element. Change that line to look like this:

<xsl:attribute name="target">_self</xsl:attribute>

[mlowery]

I've updated BISERVER-82 with some new findings. I think I have a solution for you. To quote BISERVER-82:

Forms can be submitted using one of two methods. The default is GET.
1. GET
Here form values go on the URL and non-ASCII characters are escaped by the browser. The escaping is where the character is converted to a sequence of bytes according to some encoding and then the bytes are converted to hex. So for each non-ASCII character, you'll end up with %xy. (See java.net.URLEncoder's javadoc.) The recommended encoding to get the bytes is for the URL is UTF-8 but Firefox (and maybe others) encodes according to the response content-type meta tag.

So how does Tomcat know how to interpret the %xy's in the URL? It knows because of a setting in server.xml.
<Connector port="8080" URIEncoding="UTF-8"/>

2. POST
Here form values go in body of response. Again, Firefox encodes according to the response content-type meta tag. So how does the server know what the encoding is? Well, if we assume that the form post is coming in the same way the form response went out, then we have the answer. To tell Tomcat how to interpret the body, you'll need to set the character encoding on the incoming request.
request.setCharacterEncoding("UTF-8");
This is done in the SetCharacterEncodingFilter mentioned earlier.

It turns out that this bug could have been solved all along with the Tomcat URIEncoding setting.

Now that we have that covered, we still have the Mondrian query problem. Unfortunately, that involved a code change. (Basically, the code was constructing a URL with non-ASCII characters without escaping them. So the solution is to URLEncoder.encode() them.)

1. Edit org.pentaho.plugin.olap.PivotViewComponent.

protected boolean executeAction() throws Throwable {

  // ...omitted...

  if ( !ignoreInputs.contains( name ) ) {
    viewer += "&" + name + "=" + URLEncoder.encode(getInputStringValue( name ), LocaleHelper.getSystemEncoding()); //$NON-NLS-1$ //$NON-NLS-2$
  }

  // ...omitted...

}

2. Now rebuild.