Data Management via REST API¶
There are several REST examples for data management below. They all contain the data model which can be imported.
The list of available API modules is available via the link: http://localhost:8081/unidata-backend/api, where localhost - user’s host; 8081 - user’s port.
API scheme example: localhost:8081/unidata-backend/api/data-quality/api-docs?url=/unidata-backend/api/data-quality/openapi.json
Data Model Import¶
To use Unidata you must have a data model. One of the ways to get it is to import it parts using xml-files of the model.
There are 6 parts of the data model, and you need to have all 6 xml files to fully import the data model:
Data structure;
Source systems;
Enumerations;
Measurement units;
Data quality rules;
Data matching rules.
You can use these files to make your first model.
Note
Model.xml must be imported after enumerations.xml and mesurements.xml, but before data qualities.xml and matchings.xml.
Measurement Units¶
Another special Unidata data type is “Measured”. To use simple attributes of “Measured” data type, it is necessary to make measurement units first. One of the ways of making measurement units is to import them from “measurement-units” xml-file.
Each measurement category and each measurement unit have properties: * name – category or unit name (required; may contain English letters, figures, ‘_’ symbol); * displayName – category or unit name displayed in the UI (required; may contain letters of different languages, figures, different symbols including “space”).
post {
request.uri.path = "/${AppConfig.REST_API_URL}/v1/meta/measurement/import"
request.contentType = 'multipart/form-data'
request.headers['Authorization'] = token
request.body = multipart {
part 'file', xmlFile.name.toString(), 'text/xml', xmlFile
part 'override', 'true'
}
request.encoder 'multipart/form-data', OkHttpEncoders.&multipart
}
Note
Enumerations¶
One of the Unidata data types is enumDataType. One or more enumerations with enumeration values inside can be declared in “enumerations” xml-file.
Each enumeration and each enumeration value have properties: * name – enumeration or enumeration value name (required; may contain English letters, figures, ‘_’ symbol); * displayName – enumeration or enumeration value name displayed in the UI (required; may contain letters of different languages, figures, different symbols including “space”).
post {
request.uri.path = "/${AppConfig.REST_API_URL}/v1/meta/enumerations/import"
request.contentType = 'multipart/form-data'
request.headers['Authorization'] = token
request.body = multipart {
part 'file', xmlFile.name.toString(), 'text/xml', xmlFile
part 'override', 'true'
}
request.encoder 'multipart/form-data', OkHttpEncoders.&multipart
}
Note
Use enumerations.xml
Source Systems¶
Unidata can work with data from different sources. So, some source systems can be declared in “source- systems” xml-file.
Source system “unidata” is default. All data inserted in UI has that source system.
You can refuse to use source systems xml-file. In this case all data will be added and updated from “unidata” source system only. Using some source systems, you can add records from different sources and merge them into one. Very important property weight defines one of the merging rules. Records and their attributes from sources with high weight have higher priority.
Each source system has properties: * name – source system name (required; may contain English letters, figures, ‘_’ symbol); * displayName – source system name displayed in the UI (required; may contain letters of different languages, figures, different symbols including “space”).
post {
request.uri.path = "/${AppConfig.REST_API_URL}/v1/meta/source-systems/import"
request.contentType = 'multipart/form-data'
request.headers['Authorization'] = token
request.body = multipart {
part 'file', xmlFile.name.toString(), 'text/xml', xmlFile
part 'override', 'true'
}
request.encoder 'multipart/form-data', OkHttpEncoders.&multipart
}
Note
Model of Entities / Lookup Entities¶
post {
request.uri.path = "/${AppConfig.REST_API_URL}/v1/meta/model/import"
request.contentType = 'multipart/form-data'
request.headers['Authorization'] = token
request.body = multipart {
part 'file', xmlFile.name.toString(), 'text/xml', xmlFile
part 'override', 'true'
}
request.encoder 'multipart/form-data', OkHttpEncoders.&multipart
}
Note
Use model.xml
Data Structure¶
Data structure is the main part of the data model. To get it you can use “model” xml-file. This file has five parts.
Part “registers”¶
There are six entities (hereinafter the “registers”) in the file and each of them is introduced in this part.
Figure 1. Six registers in “registers” part¶
All registers have some properties such as “name”, “display name”, list of attributes.
Figure 2. Register “MASTER” and its properties¶
Register properties: * name – register name (required; may contain English letters, figures, ‘_’ symbol); * displayName – register name displayed in the UI (required; may contain letters of different languages, figures, different symbols including “space”); * groupName – group of the entities (required; may contain English letters, figures, ‘_’ symbol). See description in “entitiesGroup” part.
Register attribute list may contain a lot of attributes of three types: simple, array and complex.
Register “MASTER” has only two simple attributes in its list.
Register attribute properties: * name – attribute name (required; may contain English letters, figures, ‘_’ symbol); * displayName – attribute name displayed in the UI (required; may contain letters of different languages, figures, different symbols including “space”); * order – attribute order in the register attribute list.
Most of simple attributes have simpleDataType property. This property sets data type of simple attribute.
Register “ALL_TYPES” has twelve simple attributes and nine of them have different simpleDataType properties. Their values are “String”, “Integer”, “Number”, “Date”, “Time”, “Timestamp”, “Boolean”, “Measured”.
Simple attribute with simpleDataType=“Measured” is used to set measurement values. It has measureSettings property which in turn has categoryId and defaultUnitId properties. Values of measureSettings.categoryId and measureSettings.defaultUnitId must match the values of category.name and unit.name in “measurement-units” xml-file.
Register “ALL_TYPES” has “lookupLink” simple attribute. Value of this attribute is a code from lookup entity which defines corresponding data from lookup entity as well.
Property lookupEntityCodeAttributeType says about type of that code. Property lookupEntityType defines lookup entity name. See description in “lookups” part.
Simple attribute “webLink” is suitable for data link. It has linkDataType property.
Value of “enum” simple attribute is an item of one of the enumeration which sets in enumDataType. All model enumerations and its items are defined in “enumerations” xml-file.
Important: * Each register must have at least one simple attribute. * One simple attribute of each register must be main displayable (must have the following properties: mainDisplayable=”true” displayable=”true” nullable=”false”)
Register “ALL_TYPES” has not just simple attributes but array attributes as well. Array attributes may keep not only one value. They are suitable for array of values. It is their main difference from simple attributes.
Array attributes have arrayValueType properties instead of simpleDataType simple attribute ones.
The third entity attribute type is complex. Register “COMPLEX_ATTRS” contains two complex attributes. The first one is declared by “complex_levels” nested entity (nestedEntityName=”complex_levels”). The second complex attribute uses “complex_all_types” nested entity (nestedEntityName= ”complex_all_types”). Nested entities are described in part “nesteds”.
Part “lookups”¶
Lookups entities are quite similar to entities. They have the same properties: * name – lookup name (required; may contain English letters, figures, ‘_’ symbol); * displayName – lookup name displayed in the UI (required; may contain letters of different languages, figures, different symbols including “space”); * groupName – group of the entities (required; may contain English letters, figures, ‘_’ symbol). See description of “entitiesGroup” part.
Figure 3. Lookups¶
Lookup has a list of the simple attributes and one code attribute. Code attribute may be used in register attributes to link register and lookup.
In the proposed example register “ALL_TYPES” is linked with lookup “STR_LOOKUP” by register simple attribute “lookupLink” and lookup code attribute “code”.
Important: * Each lookup entity must have one code attribute and at least one simple attribute. * Code attribute or one simple attribute of each lookup must be main displayable (must have the following properties: mainDisplayable=”true” displayable=”true” nullable=”false”).
Part “nesteds”¶
There are four nesteds in the provided data model example. Nesteds cannot exist without registers. They declare a structure of complex attribute. They contain lists of simple, array or complex attributes.
Figure 4. Four nesteds¶
As entity and lookup nested has properties: * name – nested name (required; may contain English letters, figures, ‘_’ symbol); * displayName – nested name displayed in the UI (required; may contain letters of different languages, figures, different symbols including “space”).
Part “entitiesGroup”¶
Data model can contain a lot of registers and lookups. To make the search of particular register or lookup in UI easier entity groups can be used.
Figure 5. Entity groups¶
There is one default entity group. Its name is “ROOT”.
Property name is required. It may contain English letters, figures, ‘_’ symbol.
Property displayName is required as well. It may contain letters of different languages, figures, different symbols including “space”. Deep hierarchy of entity groups can be made. Each inner group is able to contain a lot of inner groups inside.
Part “relations”¶
Registers can be linked with lookups using simple or array attributes with specified lookupEntityType and lookupEntityCodeAttributeType properties. This way does not work for two registers. Relations can link two or more registers.
Figure 6. Relations¶
There are relations of three types: “Contains”, “References” and “ManyToMany” (relType property).
Each relation like register or lookup has name and displayed name (name and displayName properties). It may be required or not (required property). And it must know which registers it links. Name of master register is declared by fromEntity. Property toEntity declares the slave register name.
Like register or lookup relation can have simple attributes as well.
Data quality model¶
You can import data quality model using “data-quality” xml-file.
post {
request.uri.path = "/${AppConfig.REST_API_URL}/data-quality/model/import"
request.contentType = 'multipart/form-data'
request.headers['Authorization'] = token
request.body = multipart {
part 'file', xmlFile.name.toString(), 'text/xml', xmlFile
part 'override', 'true'
}
request.encoder 'multipart/form-data', OkHttpEncoders.&multipart
}
Function groups¶
Unidata has about forty system functions now so all of them may be combined into groups. There are seven groups of the functions in the file.
Figure 8. Function groups¶
There is one default group. Its name is “ROOT”.
Deep hierarchy of function groups can be made. Each inner group is able to contain a lot of inner groups inside.
Rules¶
The third part of “data-quality” xml-file comprises data quality rules. Here are two rules of two different types.
Figure 9. Data quality rules¶
The first one is a validation rule. It uses CheckLength system function to check string attribute length.
The second rule is an enrichment rule. It generates string attribute using Concatenate system function.
Rule sets¶
The rule sets follow after data quality rules in “data-quality” xml-file.
Figure 10. Data quality rule sets¶
There is one rule set for two rules in the file. But two mappings in the set. The first mapping for the first rule and the second mapping for the second rule.
Figure 11. The mapping for checkStringAttr rule¶
This mapping declares the checking attribute (attribute string in MASTER register) and its min and max length (10 and 10).
Figure 12. The mapping for genStringAttr rule¶
The second mapping declares the enrichment of string attribute in MASTER register. This attribute is the result of two attributes (main and string in MASTER register) concatenation.
Matching model¶
To get matching model “matching” xml-file can be used.
post {
request.uri.path = "/${AppConfig.REST_API_URL}/v1/matching/model/import"
request.contentType = 'multipart/form-data'
request.headers['Authorization'] = token
request.body = multipart {
part 'file', xmlFile.name.toString(), 'text/xml', xmlFile
part 'override', 'true'
}
request.encoder 'multipart/form-data', OkHttpEncoders.&multipart
}
Algorithms¶
The first part of the file consists matching algorithms.
Figure 14. Algorithms¶
There is only one algorithm in the file. Of course, you can have much more. The presented algorithm uses postgresMatchingStorage (matchingStorageId = “postgresMatchingStorage”).
Tables¶
Matching table is a table for matching data.
The matching table declares table columns for different matching data of different types.
Figure 15. Matching table¶
Rules¶
The next part of “matching” xml-file is matching rules.
The rule uses ExactAlgorithm as matching algorithm and postgresMatchingStorage as matching storage.
Figure 16. Matching rule¶
