Filtering
- Shaun Forbes
- Aaron van Herk
- Allison Robertson
This page outlines the options available for setting up global filtering of data.
Weave 2.6.10 adds support for setting up filters that can be applied to a number of different configuration items allowing a user to work with a sub-set of the data that the administrator has provided them access to.
For example, if a user wanted to work with trees that were last inspected in a specific year the administrator can setup a filter definition that would allow the user to enter the year and then update the configurations for any data definitions, attribute searches, map engines, etc, that are related to trees to have them use the information provided by the user for the filter to refine what information is included in those items. Then any attribute searches performed will only search for data within the year range, then map would only display trees within the date range, and grids will only show rows for the year range, etc.
Using data filters relies on “user attributes” and requires two parts to setup. User attributes are a concept in Weave where custom attributes can be associated with a user and then referenced in configuration files, allowing for per-user customisation. By adding the ability for the client to set custom user attributes and ensuring the appropriate Weave configuration items can make use of those attribute to perform filtering Weave 2.6.10 make it possible for the administrator to provide the user with the ability to perform dynamic filtering.
The XML examples shown below are presented as complete individual XML configuration files that would each need to be included into Weave to create a working configuration, the XML configurations themselves could all be included in a single configuration file if required.
Also, there has been no attempt made to present the XML and examples of valid XML files, that is the normal rules for XML files apply and you’re expected to know them or look up the relevant section in this wiki. That’s not to say the XML examples are expected to be invalid, just that when creating your own versions of the XML this page is not intended as an exhaustive XML tutorial and standard XML editing rules will need to be followed for your configuration files, e.g. the use of CDATA tags or escape sequences for special XML characters.
Defining a filter
For an end user to be able to enter the information required to apply a filter a filter definition must be created. The filter definition specifies what values can be used to filter the underlying data, in the example below we are just filtering on the inspection date, but the filter can contain multiple unrelated (unrelated to each other, not unrelated to the trees) parameters to filter on things other than inspection date, for example on the tree species.
<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="urn:com.cohga.server.config#1.0"
xmlns:dynamicattributes="urn:com.cohga.weave.dynamic.attributes#1.0">
<dynamicattributes:config id="trees">
<label>Trees</label>
<parameter id="insp_date" label="Insp. Date" controlType="date" dataType="date"/>
</dynamicattributes:config>
</config>The filter definition like the one above describes what values can be set and is used to present the end user with a form where they can enter the values. Once the user enters values they are sent to the server and saved as dynamic user attributes, but they don’t actually filter anything at this stage. To perform filtering based on the dynamic user attributes the attributes need to be referenced in the configuration for the things that the administrator would like the filter to apply to, like a data definition, or attribute search, etc.
Using a filter
To apply a filter to another item in Weave the item would need to support filtering in the first place, you should refer to the configuration wiki page for the particular item to see if it supports filtering.
Data Definition
If you wish to apply the filter to a data definition that uses a database table to retrieve data then the filter can be added as an SQL Where clause in the data definition configuration. This is possible because the datasourcedataconnection type of data definition provides support for filtering using SQL via the where tag, where-as an inlinedataconnection does not (currently) support filtering so cannot use a dynamic user attribute to refine the returned data.
<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="urn:com.cohga.server.config#1.0"
xmlns:data="urn:com.cohga.server.data.database#1.0">
<data:datadefinition id="trees">
<datasourcedataconnection datasource="database" table="trees" key="id" prefix="DISTINCT">
<where clause="insp_date = ${user.dynamic.trees.insp_date}"/>
<parameter id="species" label="Species" column="species"/>
<parameter id="location" label="Location" column="location"/>
<parameter id="insp_date" label="Insp. Date" column="insp_date" type="date"/>
</datasourcedataconnection>
</data:datadefinition>
</config>You can see in the above example an additional where clause has been added to the configuration that will filter the rows returned from the SQL based on the insp_date column in the table matching the value entered by the user. Note that if the user has not given a value for the insp_date value then the where clause will not be applied and all rows will be returned from the table that match the current selection (since there are no other where clauses defined in the configuration).
The values the user enters for the filter are referenced using the format:
${user.dynamic.<filterid>.<parameterid>}
and in some situation the value set for the filter parameter may require coercion to convert it to the format required by the underlying provider (e.g. database, spatial engine, etc), for example some databases may require a CAST or CONVERT function to wrap the value provided by the filter to ensure it is in the format that the database requires. Take note that since this is new functionality there may also be issues with the conversions that should be, or even need to be, handled by Weave, so if you’re having issues getting a particular dynamic user attribute to with contact support for help.
Spatial Mapper
The above example configuration just applies the filter to a single data definition associated with the trees but in all likelihood you will have additional configuration items which would also make sense to have the filtering applied to, for example you could apply the filter to the spatial mapper for the trees entity so that when a user tries to select a tree spatially with one of the map selection tools only those trees that meet the filter restrictions would be selected.
<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="urn:com.cohga.server.config#1.0"
xmlns:mapper="urn:com.cohga.server.spatial.mapper#1.0">
<mapper:mapper id="trees">
<spatialEngine>spatialengine</spatialEngine>
<mapping>
<entity>trees</entity>
<table>trees</table>
<key>id</key>
<filter>insp_date = ${user.dynamic.trees.insp_date}</filter>
<dynamic>true</dynamic>
<cache>false</cache>
</mapping>
</mapper:mapper>
</config>Maps
Or you could apply the filter to the map engine that displays the trees so that only the trees that meet the filter criteria will be drawn on the map.
Searches
A filter can also be applied to attribute searches.
Note that the above search includes a search parameter for the inspection date and also uses the user attribute for filtering, this is to allow the user to still be able to select from all trees based on inspection date when no filtering is currently being applied, but this can result in nothing being found if the filtering is applied and the user enters different values for the two inspection dates, since the where clauses will be combined using an AND is the SQL statement and results in a where clause that will return nothing, e.g. WHERE insp_date = 2023-01-05 AND insp_date = 2023-01-06.
By applying the filtering to each item that’s related to the trees then the end user will have a consistent filtered view of the data related to the values they entered for the filter, which is the intended use case of this functionality.
Indexes
Applying a filter to an index is a little more involved than some of the other items in that it requires that additional fields be added to the index when it is built.
If you intend to include the indexes related to an entity in the filtering you will be limited to only filter parameters that perform an equals comparison, that is you will not be able to filter on a range so you can not use the silder or multislider parameters types when defining the user attributes, or utilise multiple fields that are combined to filter on a range.
This is because the filtering is performed by the underlying indexing engine which does not support range based queries.
In the above example the additional fields to be added to the index (there can be more than one) are added with the <field> tag. The name attribute is the name of the attribute that will be added to the indexed document, and must be in the format dynamic.<filterid>.<parameterid>, and the value refers to the parameter in the data definition that will provide the value for that field.
Then when the index is searched the query will include additional context to ensure that if the user attribute for the parameter, in the above example the insp_date user attribute, is set then it will be included in the query to ensure that only documents that contain the matching value in the dynamic.tree.insp_date field are included in the results.
If the insp_date user attribute is not set then the query will not be adjusted to include a reference and all documents will be searched.
Client
Once you have the filters configured and the various other configuration items setup to use the filters you need to add the filter panel to the client so that the user can modify the parameters for the filter(s).
There is no specific configuration that is available for the panel so you just need to include a label and location. The id for the panel is weave.userAttributes.
Configuration
Namespace
urn:com.cohga.weave.dynamic.attributes#1.0
Tags
config
Properties
Name | Type | Required | Description |
id | string | yes | Unique identifier for this filter. |
label | string | yes | Text to be displayed to the user to represent this filter. |
Sub-tags
Name | Type | Cardinality |
parameter | urn:com.cohga.weave.dynamic.attributes#1.0:parameter | 1..n |
parameter
Properties
Name | Type | Required | Default | Description |
id | string | yes |
| A unique identifier for the parameter. |
label | string | yes |
| The prompt text displayed when user input the parameter value. |
column | string | yes |
| The name of the column within the layer that this parameter references. |
controlType | 'text', '‘list', 'radio', ‘check', ‘multicheck’, 'silder’, 'multislider’ | no | 'text' | The UI control to use when displaying the parameter. |
dataType | 'boolean', 'float', 'integer', 'string' | no | 'string' | The data type for the parameter. |
allowBlank | boolean | no | true | Give the user the choice of an empty value in the listbox (as opposed to a null value). |
value | any | no |
| The default value of the parameter (except multislider fields). |
dataSet | no |
| Where to get the values for a listbox. | |
labelColumn | string | no |
| Column in the datadefinition that supplies the label of the value to show the user. |
valueColumn | string | no |
| Column in the datadefinition that supplies the value of the value to use in the SQL. |
scalarparametertype | string | no | 'simple' | 'simple' or 'multi-value' to determine of more than one value can be selected from a list. |
width | integer | no |
| Set the width of the field. |
minValue | number | yes - for slider and multislider |
| The minimum value allowed for a numeric field. |
maxValue | number | yes - for slider and multislider |
| The maximum value allowed for a numeric field. |
leftValue | number | no |
| The initial minimum value for a multislider. |
rightValue | number | no |
| The initial maximum value for a multislider. |
increment | number | no |
| The increment to use for fields that support it, the units are dependant upon the field type. |
trueValue | any | no |
| The value that equates to "true" in the underlying table, only suitable for checkboxes. |
falseValue | any | no |
| The value that equates to "false" in the underlying table, only suitable for checkboxes. |
decimalPrecision | number | no |
| The precision of any numeric fields. |
Sub-tags
Name | Type | Cardinality |
list | urn:com.cohga.weave.dynamic.attributes#1.0:list | 1..1 - only for radio and multicheck fields |
list
Properties
Name | Type | Required | Description |
value | string | yes | Value to be used in the filter |
label | string | yes | Text to be displayed to the user to represent this filter |
checked | string | no | Should this check box be initially checked. Only for list within multicheck field |
Data Security
To ensure that data is not accidentally exposed because of an incorrectly configured filter if the value for a user property is missing then filtering will convert the generated filter into one that returns no results rather than just removing the filter and returning all results, e.g. a filter like where client_id = ${user.dynamic.client_id} will be replaced with something like where 1 = 0 if there is no dynamic value for the client_id.
Normally you would resolve this by providing a suitable default value for the client_id, e.g. where client_id = ${user.dynamic.client_id|9999999}, which would result in the same outcome in this example, but make it explicit what is happening and shows that the issue has been looked at. Alternatively, you can set a property that changes the handling of the filtering to remove the filter if the value is missing.
The property to set is weave.ignoreonsubstitutionerror and if it is set to true then the handling of missing values will be treated differently and will result in the filter being excluded and all results (subject to any other filtering) will be returned rather than no results. The value can be set as a system property or just adding the following to a config xml file: