Basics of data handling in Weave

Selections

The concept of separation of searching and reporting is an important concept in Weave.

For example, think of search engines like Google where users search for something and get back the results. The only way to search is by typing something in the box, and one type of result displayed; a page of top search results.

In contrast, with Weave there are multiple ways of searching for things and a multitude of ways of displaying the search results. “Searching” could be preformed by drawing a polygon on a map to find all the things that fall within that polygon, or by typing in an address to search a database find the matching properties, or even by having a third party system sending a list of unique identifiers. And “Reporting” could be as simple as displaying a list of the owner names of the found properties, or generating a PDF report describing the history of the property, or sending a list of unique identifiers to another third party system.

Because of this separation and the use of a current selection, searching can be combined or split across multiple systems. For example, a user may be working in a property management system and may have selected a number of properties. They then send this selection to Weave (searching by third party system) which updates the current selection for properties. They then perform a spatial intersection operation in Weave to refine the selection of properties to only those that fall within a flood zone (searching via spatial operation), then perform an attribute search to further refine the selected properties to those that are listed in the residents database as being over 65 years old (searching via SQL). Then they display the textual list of owners of the properties (reporting from SQL) before finally generating a flood warning mail merge based on the final list of selected properties (reporting from document) and send those to the residents advising them to evacuate.

To facilitate this Weave uses the concept of a current selection, where each entity you want to work with in Weave is associated with a list of unique identifiers that represent the entities that the user has searched for and is reporting on. This list of identifiers can come from anywhere and as long as each system Weave connects to uses the same identifiers to represent the same entity then Weave can work with those systems to perform searching and reporting.

In summary, a central principal in Weave is the entity, which is nothing more than an internal id and a user facing label. A selection is simply a list of identifiers that belong to an entity.

Spatial Operation

Before spatial operations (select by polygon, zoom to on a map, etc) can be performed against an entity, a spatial engine must be configured telling Weave of the relationship between the layer/table and the entity and a table column (the key column) must be specified to uniquely identify items under each entity. This is done by creating a spatial mapping configuration.

Once a spatial mapping is in place, spatial operations such as select by polygon, boundary intersection and buffer operations result in key column values being written to the current selection under the relevant entity. Conversely, zoom to current selection and highlight selection on a map causes the unique identifiers in the current selection to be used as the basis for performing a query in the spatial engine for a geometry that matches the list of id's.

The spatial operations on the client are represented with the "select/reselect/unselect by point/line/polygon/circle/rectangle" tools, along with various buffer and spatial intersection tools and setup using the Spatial Mapper configuration item.

Attribute Display

If you want to display textual attributes for an entity you create data definitions and link those to an entity for display with a data tag. Data definitions are a tabular representation of textual data that can come from any source, but generally will come from a database and be generated by executing SQL. They can be used for populating drop down lists, generating content for a BIRT report or most often for direct display of attribute data for the user.

To allow a data definition to be used to display attribute data related to a selected entity the data tag is used to link an entity to a data definition and provide a label for displaying to the user. This way there can be more than one set of attribute data that can be displayed to the user for each entity and the user can choose between which attributes to display be selecting the from list of available data's based on the label.

For database/SQL based data definitions which will be used for displaying textual data relating to an entity for the user the data definition must be configured with a key column. In the same way that the key column in the spatial mapping links the currently selected entity to a row in a table in the spatial engine, the key column in a data definition links the currently selected entity to a row in a table in a data source (as data source is a connection to a database). This way when it comes time to display the attributes for a selected entity to the user an SQL statement can be generated that only selects the rows that match the id's from the current selection.

Basing the generation of the attribute data on the unique identifiers related to an entity means that the attribute data to be displayed to the user can come from a number of different databases, including the database underlying the spatial engine (if it has one), and does not require that the data that's displayed to the user is even in the same database as the spatial layers, let alone is directly attached to the spatial records in the spatial database.

Attribute display on the client is represented using the data grid view and created using the Data Definition configuration items.

Attribute Searches

Data definitions take care of generating textual attribute data for displaying to the user, attribute searches provide the reverse. They allow the user to enter text values which are sent to a database, via an SQL statement, to generate a list of unique id's that match the user entered values, which are then uses to update the current selection for the entity that the search is associated with. This is again done by specifying a key column for the search, and it's this column who's values will be extracted and placed into the current selection for the list when a search is performed.

From the above you can see that by ensuring that spatial mappings, data definition and attribute searches all have the same concept of what value represents a given entity it's possible to combine all three, even if the actual source of the values (i.e. A specific column in a database table) isn't the same in all three cases.

Attribute searches are represented on the client using the search view and created using the Search configuration items.

Reports

Beyond the basic data display, generating a Report can allow the user to produce a more formatted form of output beyond just the tabular display of the data. Reports can contain data generated by Weave data definitions, or, reports can pull in data from any data source available to the BIRT reporting engine.

Virtual Entities

While it's possible to perform joins when working with data definitions and attribute searches, for example to allow you to list all of the animals registered at a selected property, sometimes you want to work with individual entities that don't have a spatial components.

Continuing our animals example if you wanted to select all of the German Shepherds in a given neighbourhood a naive approach would be to create an attribute search that started with the animal registration table (perhaps selecting all the rows containing the animal code for German Shepherds) then join that table to the property table (assuming the animals table has a foreign key into the property table) allowing us to get the unique property id for the property that the animal is located at. Weave then uses those property id's to update the current selection for properties and now we have a list of all the properties that have registered German Shepherds. But, when we then try and display the information related to the dogs we've just searched for using a data definition that does the reverse of when the search did (that is we start with the property table, selecting just those rows that match the property id's in out current selection, then join that table to the animal registration table to get a list of the animals for each property) we see that there are dogs other than German Shepherds listed.

This occurs because what we've done is to select all of the properties that have German Shepherds, we haven't selected all of the German Shepherds which is what we actually wanted to do. So when it come time to display the animal information we're displaying information about all of the animals that are registered at each selected property, which will more than likely include more that just the original German Shepherds.

If you want to work with entities that don't have a spatial representation you have two choices, either the user works with the entities using just attribute searches and data definition and has no way of selecting or viewing the entities via the map display, or you create a spatial view for the non-spatial entity and create a new entity to represent the newly "spatialized" entity. For example with our animals search we create a spatial view in the underlying spatial database that joined the animals registration table to the property spatial table, allowing us to represent each animal record by the property boundary of the property it's registered at and then we create a new animals entity and set up a spatial mapping using the spatial view and setting the key to be the animals registration number rather than the property id.

Then when we create an attribute search to find the German Shepherds the list of id's we end up with are the dogs registration numbers, which are unique for each dog, rather than the property id which isn't unique for each dog. We'd also need to switch out attribute search and data definition so that they now use the animal registration number of the key field rather than the property id. Of course we can also keep the animal property attribute search for time when we do want to locate properties based on the animals that are registered there.

Note that we could also add the newly created animals layer to our map engine as a new layer so that we can display just the animals using their property boundaries on the map, but this isn't necessary.