Query KDP dataset
Query KDP dataset
Authorizations:
Request Body schema: application/jsonrequired
expression | string Query expression |
limit | integer Default: 10 Maximum number of results to return |
offset | integer Default: 0 Number of results to skip |
datasetId required | string <uuid> UUID of the dataset to query |
orderedResults | boolean Default: true Whether to return ordered results |
Responses
Request samples
- Payload
{- "expression": "",
- "limit": 100,
- "offset": 0,
- "datasetId": null,
- "orderedResults": true
}
Response samples
- 200
- 400
- 401
- 403
- 500
{- "records": [
- {
- "name": "firstField",
- "id": "14",
- "date": "68822019-01-01"
}, - {
- "name": "secondField",
- "id": "16",
- "date": "68822019-01-01"
}
], - "schema": {
- "name": {
- "fieldName": "name",
- "type": "java.lang.String"
}, - "id": {
- "fieldName": "id",
- "type": "java.lang.String"
}, - "date": {
- "fieldName": "date",
- "type": "java.lang.String"
}
}, - "more": false,
- "partialCount": false,
- "total": 10
}
Query KDP dataset schema
Retrieve KDP dataset schema
Authorizations:
query Parameters
datasetId required | string KDP dataset id |
includeSecurityField | boolean Default: false whether to include the security field (_koverse_security_label) for each schema entry. |
includeNestedFields | boolean Default: false whether to include nested fields of the schema such as the child fields of an object or map. |
Responses
Response samples
- 200
- 400
- 401
- 403
- 500
{- "name": {
- "fieldName": "name",
- "type": "java.lang.String"
}, - "id": {
- "fieldName": "id",
- "type": "java.lang.String"
}, - "date": {
- "fieldName": "date",
- "type": "java.lang.String"
}
}
List KDP datasets
Retrieve list of datasets from the KDP to which authenticated user has access
Authorizations:
query Parameters
limit | integer Number of results to return |
skip | integer Number of results to skip |
Responses
Response samples
- 200
- 400
- 401
- 403
- 500
{- "total": 1,
- "limit": 10,
- "skip": 0,
- "data": [
- {
- "id": null,
- "name": "my_dataset",
- "description": "",
- "recordCount": 10,
- "schema": {
- "name": "java.lang.String",
- "id": "java.lang.String",
- "date": "java.lang.String"
}, - "userId": null,
- "workspaceId": "my_workspace",
- "autoCreateIndexes": true,
- "searchAnyField": true,
- "jsonView": false,
- "createdFromSync": false,
- "schemaModifiedDt": "2024-06-28T21:36:14.875Z",
- "createdAt": "2024-06-25T20:06:19.371Z",
- "updatedAt": "2024-06-28T21:36:15.671Z",
- "deletedAt": null,
- "currentUserPermissions": {
- "manage": true,
- "read": true,
- "write": true
}
}
]
}
Authenticate KDP user
Authenticate a user and retrieve access token and user information
Authorizations:
query Parameters
workspaceId required | string KDP workspace id |
Responses
Response samples
- 201
- 400
- 401
- 403
- 500
{- "accessToken": "",
- "authentication": {
- "strategy": "keycloak",
- "accessToken": "",
- "payload": {
- "email": "xxx@saic.com",
- "iat": 1719939148,
- "exp": 1720025548,
- "iss": "koverse",
- "sub": "",
- "jti": ""
}
}, - "user": {
- "id": "",
- "email": "xxx@saic.com",
- "workspaceId": "edl01",
- "role": "admin",
- "firstName": "edl",
- "lastName": "test",
- "avatar": null,
- "githubId": null,
- "googleId": null,
- "keycloakId": "",
- "microsoftId": null,
- "oktaId": null,
- "stripeCustomerId": null,
- "stripeLatestCharge": null,
- "stripeSubscriptionId": null,
- "changeEmailTokenExpiration": null,
- "usedFreeTrial": false,
- "verified": true,
- "createdAt": "2024-06-25T19:59:50.326Z",
- "updatedAt": "2024-07-02T16:52:28.820Z",
- "deletedAt": null,
- "linkedAccounts": [
- "keycloak"
], - "displayName": "xxx xxx"
}
}
Query KDP user
Query KDP user
Authorizations:
query Parameters
userId required | string KDP user id |
Responses
Response samples
- 200
- 400
- 401
- 403
- 500
{- "id": "6bf7c1e5-xxx-xxx-xxxx-a4cc287941f6",
- "email": "xxx@saic.com",
- "workspaceId": "myworkspace",
- "role": "admin",
- "firstName": "edl",
- "lastName": "test",
- "avatar": null,
- "githubId": null,
- "googleId": null,
- "keycloakId": "8a47c064-xxxx-xxxx-xxxx-a396255d5379",
- "microsoftId": null,
- "oktaId": null,
- "stripeCustomerId": null,
- "stripeLatestCharge": null,
- "stripeSubscriptionId": null,
- "changeEmailTokenExpiration": null,
- "usedFreeTrial": false,
- "verified": true,
- "createdAt": "2024-06-25T19:59:50.326Z",
- "updatedAt": "2024-07-29T20:59:45.546Z",
- "deletedAt": null,
- "linkedAccounts": [
- "keycloak"
], - "displayName": "edl test"
}
Create Elasticsearch Index
Create Index in Elasticsearch
Authorizations:
Request Body schema: application/jsonrequired
indexName required | string Name of the index |
required | object (ElasticsearchIndexConfig) |
Responses
Request samples
- Payload
{- "indexName": "string",
- "data": {
- "settings": {
- "number_of_shards": 0,
- "number_of_replicas": 0
}, - "mappings": {
- "properties": {
- "property1": {
- "type": "string",
- "fields": {
- "property1": {
- "type": "string"
}, - "property2": {
- "type": "string"
}
}
}, - "property2": {
- "type": "string",
- "fields": {
- "property1": {
- "type": "string"
}, - "property2": {
- "type": "string"
}
}
}
}
}
}
}
Response samples
- 201
- 400
- 401
- 403
- 500
{- "acknowledged": true,
- "shards_acknowledged": true,
- "index": "idx01"
}
Search Elasticsearch Documents
Search documents in Elasticsearch
Authorizations:
Request Body schema: application/jsonrequired
indexName required | string Name of the index |
data required | object |
Responses
Request samples
- Payload
{- "indexName": "sanctions",
- "data": {
- "query": {
- "match_all": { }
}
}
}
Response samples
- 200
- 400
- 401
- 403
- 500
{- "took": 0,
- "timed_out": false,
- "shared": {
- "failed": 0,
- "skipped": 0,
- "successful": 1,
- "total": 1
}, - "hits": {
- "max_score": 1,
- "hits": [
- {
- "_index": "sanctions",
- "_id": "BO7et5AB8enQd6KcjUYY",
- "_score": 1,
- "_source": {
- "identity_documents": [
- {
- "document_name": "MAROC INTERNATIONAL PVT LTD",
- "document_number": "C-0977/2021",
- "document_valid": "true",
- "issuing_country": "Maldives",
- "document_type": "Registration Number"
}
], - "sanction_list_publish_dates": "2023-07-31",
- "address_country": "Maldives",
- "organization_established_date": "2021-09-23",
- "entity_names": "MAROC INTERNATIONAL PVT LTD",
- "sanction_list_ids": "SDN List",
- "sanction_type": "Block",
- "address_postal code": "23000",
- "address_address1": "Coral Ville",
- "address_city": "Hulhumale",
- "relationships": [
- {
- "relation_entity_id": "44486",
- "type": "OWNED_OR_CONTROLLED_BY",
- "related_entity": "RAUF, Mohamed Inthif"
}
], - "legal_authorities": "Executive Order 13224 (Terrorism)",
- "entity_type": "Entity",
- "@timestamp": "2023-07-31T00:00:00.000-07:00",
- "sanction_programs": "SDGT",
- "secondary_sanctions_risk:": "section 1(b) of Executive Order 13224, as amended by Executive Order 13886",
- "id": "44473"
}
}
]
}, - "total": {
- "value": 10000,
- "relation": "Gte"
}
}
Search Elasticsearch Document using template
Search documents in Elasticsearch using template
Authorizations:
Request Body schema: application/jsonrequired
templateName required | string Name of the template |
data required | object |
Responses
Request samples
- Payload
{- "indexName": "script1",
- "data": {
- "params": {
- "query_string": "",
- "inclusiveControl_1": "",
- "inclusiveControl_2": "",
- "exclusiveControl_1": ""
}
}
}
Response samples
- 200
- 400
- 401
- 403
- 500
{- "took": 0,
- "timed_out": false,
- "shared": {
- "failed": 0,
- "skipped": 0,
- "successful": 1,
- "total": 1
}, - "hits": {
- "max_score": 1,
- "hits": [
- {
- "_index": "sanctions",
- "_id": "BO7et5AB8enQd6KcjUYY",
- "_score": 1,
- "_source": {
- "identity_documents": [
- {
- "document_name": "MAROC INTERNATIONAL PVT LTD",
- "document_number": "C-0977/2021",
- "document_valid": "true",
- "issuing_country": "Maldives",
- "document_type": "Registration Number"
}
], - "sanction_list_publish_dates": "2023-07-31",
- "address_country": "Maldives",
- "organization_established_date": "2021-09-23",
- "entity_names": "MAROC INTERNATIONAL PVT LTD",
- "sanction_list_ids": "SDN List",
- "sanction_type": "Block",
- "address_postal code": "23000",
- "address_address1": "Coral Ville",
- "address_city": "Hulhumale",
- "relationships": [
- {
- "relation_entity_id": "44486",
- "type": "OWNED_OR_CONTROLLED_BY",
- "related_entity": "RAUF, Mohamed Inthif"
}
], - "legal_authorities": "Executive Order 13224 (Terrorism)",
- "entity_type": "Entity",
- "@timestamp": "2023-07-31T00:00:00.000-07:00",
- "sanction_programs": "SDGT",
- "secondary_sanctions_risk:": "section 1(b) of Executive Order 13224, as amended by Executive Order 13886",
- "id": "44473"
}
}
]
}, - "total": {
- "value": 10000,
- "relation": "Gte"
}
}
Get Elasticsearch Index
Get index in Elasticsearch
Authorizations:
path Parameters
index_name required | string The name of the index to retrieve information for or '_all' for all indexes |
Responses
Response samples
- 200
- 400
- 401
- 403
- 500
{- "sanctions": {
- "aliases": { },
- "mappings": {
- "_meta": {
- "created_by": "file-data-visualizer"
}, - "properties": {
- "organization_type": {
- "type": "text",
- "fields": {
- "keyword": {
- "type": "keyword",
- "ignore_above": 256
}
}
}, - "vessel_flag": {
- "type": "keyword"
}, - "digital_currency_address_-_xmr": {
- "type": "text",
- "fields": {
- "keyword": {
- "type": "keyword",
- "ignore_above": 256
}
}
}, - "other_vessel_call_sign": {
- "type": "text",
- "fields": {
- "keyword": {
- "type": "keyword",
- "ignore_above": 256
}
}
}, - "birthdate": {
- "type": "date",
- "format": "iso8601"
}
}
}, - "settings": {
- "index": {
- "number_of_shards": "1",
- "number_of_replicas": "1",
- "routing": {
- "allocation": {
- "include": {
- "_tier_preference": "data_content"
}
}
}, - "provided_name": "sanctions",
- "creation_date": 1721071727882,
- "uuid": "BhAg0_G3SgeMTThXKqjzWQ",
- "version": {
- "created": "8070199"
}
}
}
}
}
Create Elasticsearch Pipeline
Create Pipeline in Elasticsearch
Authorizations:
Request Body schema: application/jsonrequired
pipelineName required | string Name of the pipeline |
data required | object |
Responses
Request samples
- Payload
{- "pipelineName": "attachments",
- "data": {
- "description": "Extract attachments information from arrays",
- "processors": [
- {
- "foreach": {
- "field": "data",
- "processor": {
- "attachment": {
- "target_field": "_ingest._value.attachment",
- "field": "_ingest._value.data",
- "indexed_chars": -1
}
}
}
}, - {
- "script": {
- "source": "if (ctx.containsKey('data')) { ctx['detail'].addAll(ctx['data']); }"
}
}, - {
- "script": {
- "source": "for(detail in ctx['detail']) { if(detail != null && detail.attachment != null) { for(field in detail.attachment.entrySet()) { detail[field.getKey()] = field.getValue(); } } }"
}
}, - {
- "script": {
- "source": "for(detail in ctx['detail']) { if(detail != null && detail.attachment != null) { for(field in detail.attachment.entrySet()) { detail[field.getKey()] = field.getValue(); } detail.remove('attachment'); } }"
}
}, - {
- "remove": {
- "field": "data"
}
}
]
}
}
Response samples
- 201
- 400
- 401
- 403
- 500
{- "acknowledged": true
}
Get Elasticsearch Pipeline
Get Pipeline in Elasticsearch
Authorizations:
path Parameters
pipeline_name required | string The name of the pipeline |
Responses
Response samples
- 200
- 400
- 401
- 403
- 500
{- "attachments": {
- "description": "Extract attachments information from arrays",
- "processors": [
- {
- "foreach": {
- "field": "data",
- "processor": {
- "attachment": {
- "field": "_ingest._value.data",
- "indexed_chars": -1,
- "target_field": "_ingest._value.attachment"
}
}
}
}, - {
- "script": {
- "source": "if (ctx.containsKey('data')) {\n ctx['detail'].addAll( ctx['data']);\n}\n"
}
}, - {
- "script": {
- "source": "for(detail in ctx['detail']){\n if(detail != null && detail.attachment != null) {\n for(field in detail.attachment.entrySet()) {\n detail[field.getKey()]=field.getValue()\n }\n }\n}\n"
}
}, - {
- "script": {
- "source": "for(detail in ctx['detail']){\n if(detail != null && detail.attachment != null){\n for(field in detail.attachment.entrySet()) {\n detail[field.getKey()]= field.getValue()\n }\n detail.remove(\"attachment\")\n }\n}\n"
}
}, - {
- "remove": {
- "field": [
- "data"
]
}
}
]
}
}
Index Elasticsearch Document
Index Document in Elasticsearch
Authorizations:
Request Body schema: application/jsonrequired
indexName required | string Name of the index |
pipelineName required | string Name of the pipeline |
docId | string optional document id to use |
data required | object |
Responses
Request samples
- Payload
{- "indexName": "string",
- "pipelineName": "string",
- "docId": "string",
- "data": { }
}
Response samples
- 201
- 400
- 401
- 403
- 500
{- "index": "idx01",
- "version": 1,
- "result": "Updated",
- "docId": "123"
}
Bulk Index Elasticsearch Document
Bulk Index Document in Elasticsearch
Authorizations:
Request Body schema: application/jsonrequired
required | object | ||||||
|
Responses
Request samples
- Payload
[- {
- "index": {
- "_index": "idx01",
- "_id": "123",
- "pipeline": "attachments"
}
}, - {
- "adm": "document1"
}
]
Response samples
- 200
- 400
- 401
- 403
- 500
{- "items": [
- {
- "index": "idx01",
- "id": "SoDQI5EBwipE9ezlyBm4",
- "status": 201,
- "result": "created",
- "version": 1
}, - {
- "index": "idx02",
- "id": null,
- "status": 400,
- "error": {
- "type": "illegal_argument_exception",
- "reason": "field [data] not present as part of path [data]"
}
}
]
}
Create/Update Elasticsearch Script
Create/Update Script in Elasticsearch
Authorizations:
Request Body schema: required
scripts | string <json> JSON string containing script metadata |
file | Array of strings <binary> [ items <binary > ] Script files to be uploaded |
Responses
Response samples
- 200
- 400
- 401
- 403
- 500
{- "script2": {
- "success": false,
- "errorMessage": "no uploaded file available",
- "statusCode": 400
}, - "script1": {
- "success": true,
- "context": {
- "acknowledged": true
}
}
}
SQL query search Elasticsearch Documents
SQL query search documents in Elasticsearch
Authorizations:
Request Body schema: application/jsonrequired
format | string Default: "json" Enum: "json" "txt" "csv" "tsv" "yaml" "cbor" "smile" Response format |
data required | object |
Responses
Request samples
- Payload
{- "format": "json",
- "data": {
- "query": "select * from idx01",
- "fetch_size": 10
}
}
Response samples
- 200
- 400
- 401
- 403
- 500
{- "columns": [
- {
- "name": "id",
- "type": "long"
}, - {
- "name": "entity_names",
- "type": "text"
}, - {
- "name": "sanction_type",
- "type": "keyword"
}, - {
- "name": "address_country",
- "type": "keyword"
}, - {
- "name": "organization_established_date",
- "type": "datetime"
}, - {
- "name": "sanction_list_ids",
- "type": "keyword"
}, - {
- "name": "entity_type",
- "type": "keyword"
}
], - "rows": [
- [
- 44473,
- "MAROC INTERNATIONAL PVT LTD",
- "Block",
- "Maldives",
- "2021-09-23T00:00:00.000Z",
- "SDN List",
- "Entity"
], - [
- 44474,
- "SKY NOVA INVESTMENT, TAS-TY ADDU",
- "Block",
- "Maldives",
- "2022-10-26T00:00:00.000Z",
- "SDN List",
- "Entity"
], - [
- 44475,
- "NEW SUN INVESTMENTS PVT LTD, NEW SUN INVESTMENTS PRIVATE LIMITED, NEW SUN INVESTMENTS",
- "Block",
- "Maldives",
- "2015-03-05T00:00:00.000Z",
- "SDN List",
- "Entity"
], - [
- 44477,
- "PERFILEV, Vitalij Viktorovic, ПЕРФИЛЬЕВ, Виталий Викторович, PERFYLYEV, Vitalij Viktorovvij, PERFILEV, Vitalii Viktorovitch",
- "Block",
- "Central African Republic",
- null,
- "SDN List",
- "Individual"
], - [
- 44480,
- "NASEEM, Jinaau, NASEEN, Jinau, NASYM, Hasen",
- "Block",
- null,
- null,
- "SDN List",
- "Individual"
], - [
- 44481,
- "NAUSHAD SHAREEF, Mohamed",
- "Block",
- "Maldives",
- null,
- "SDN List",
- "Individual"
], - [
- 44482,
- "DIDI, Faris Mohamed",
- "Block",
- "Maldives",
- null,
- "SDN List",
- "Individual"
], - [
- 44483,
- "NIHADH, Ali, NIHAD, Ali",
- "Block",
- "Maldives",
- null,
- "SDN List",
- "Individual"
], - [
- 44484,
- "THASLEEM, Mohamed",
- "Block",
- "Maldives",
- null,
- "SDN List",
- "Individual"
], - [
- 44485,
- "RAUF, Ahmed Alif, RAUF, Ahmed Aalif",
- "Block",
- "Maldives",
- null,
- "SDN List",
- "Individual"
]
], - "cursor": "spCTBERGTACEkMFOAjEQhjt1EwQ1xAs34xOQENTEK2LQoIagG1EvTd2W3a5Li9sSXC6+golvYnwM30nbBQyrMc5pOvP3/9sP7hBECANy9WmrmncIVQAqQ8ETRrRKjUeYCj7nKlfwDmsIY9vkV8u2y6tsJ84FSlgwhDe5NMJkRNIR1whvaSoDI5QkJhtzhKuUsZRrTQI1kSbNEN5RaUilmNFcxbWh94nQEWeEUcNhV5tUBCY/EDV2IpoQI0bcRkil8fZ3gr1miGA2dWPxiHkmZG8vrx6Ul0JdO4paYYOc7F2F/ML3o5uzx3g26KNar5nU9wdx7Lcn0+Ztp9086LZ64eKfH8c1en3ux93upX9aH0/600582Bo3+gj+MrRMl4xLMLR8wEuUtIYwLHACz/An48ZFXlB64NlUpcytfqIrLP+huO7wOWjO6DexgtUqvNXFM0JfAAAA//8DAA=="
}
Response samples
- 200
- 400
- 401
- 403
- 500
{- "sub": "8a47c064-xxxx-xxxx-xxxx-a396255d5379",
- "email_verified": true,
- "name": "edl test",
- "preferred_username": "edltest01",
- "given_name": "edl",
- "family_name": "test",
- "email": "edltest01@saic.com"
}