Migrate from Log Monitoring API v1 to Logs On Grail API

Log Monitoring API v1 has been deprecated with Dynatrace version 1.284. Its replacement is Logs On Grail API. We recommend that you migrate to the new API at your earliest convenience.

The migration affects endpoint URLs, query parameters, and response/request body parameters, as well as the scope of the token for request authentication.

To be authenticated to use the Logs on Grail API, you need a valid bearer token. The access to the API is fine-grained, meaning that you also need the proper permissions assigned to the token. The bearer token authenticates you via an OAuth2 client as a service user who is granted access the to API. To obtain a token, you must create an OAuth2 client and then request the token from it.

New features

Logs On Grail API offers you the following new features:

You can search log data that match provided DQL query in the whole Grail storage.
You can ingest log data using REST API.

Licensing

The ingestion of log events consumes Davis Data Units (DDUs) from the logs pool.

Base URL

Logs On Grail API new

/platform/storage/query/v1/
/api/v2/logs/ingest

Log Monitoring v1 old

/api/v1/entity/infrastructure/custom-devices
/api/v1/entity/infrastructure/hosts
/api/v1/entity/infrastructure/process-groups

Authentication token scope

new Logs On Grail API

old Log Monitoring API v1

Read logs (storage:logs:read and storage:buckets:read)
Ingest logs (storage:logs:write)

List, get, analyse logs of custom devices, hosts or process groups (LogExport)

Examples

Here are some examples of differences in API usage.

Retrieve host OS system log file names

To run a query and to retrieve its result, you need to make two requests via the following two separate methods:

To query Grail, execute the POST /query:execute request.
To retrieve the result, execute the GET /query:poll request.

The query timeframe should be at least 7 days.

Request URL

POST https://myapps.mydomain.com/platform/storage/query/v1/query:execute

Request body

{
  "query": "fetch dt.entity.host| fieldsAdd logFileStatus",
  "defaultTimeframeStart": "2024-01-01T12:10:04.123Z",
  "defaultTimeframeEnd": "2024-01-20T13:10:04.123Z",
  "timezone": "UTC",
  "locale": "en_US",
  "maxResultRecords": 1000,
  "maxResultBytes": 1000000,
  "fetchTimeoutSeconds": 60,
  "requestTimeoutMilliseconds": 1000,
  "enablePreview": true,
  "defaultSamplingRatio": 1000,
  "defaultScanLimitGbytes": 100
}'

Response body

{
    "state": "SUCCEEDED",
    "progress": 100,
    "result": {
        "records": [{
                "entity.name": "CT-2LGHST9",
                "id": "HOST-E5C4891D0B948EF4",
                "logFileStatus": {
                    "/my/path1": "FILE_STATUS_NOT_EXIST"
                }
            }, {
                "entity.name": "CT-HGF452",
                "id": "HOST-C9B5229282ED7A48",
                "logFileStatus": {
                    "/my/path1": "FILE_STATUS_OK"
                }
            }
        ],
        "types": [{
                "indexRange": [0, 1],
                "mappings": {
                    "entity.name": {
                        "type": "string"
                    },
                    "id": {
                        "type": "string"
                    },
                    "logFileStatus": {
                        "type": "record",
                        "types": [{
                                "mappings": {
                                    "/my/path1": {
                                        "type": "string"
                                    }
                                }
                            }
                        ]
                    }
                }
            }
        ],
        "metadata": {
            "grail": {
                "canonicalQuery": "fetch dt.entity.host\n| fieldsAdd logFileStatus",
                "timezone": "Z",
                "query": "fetch dt.entity.host| fieldsAdd logFileStatus",
                "scannedRecords": 2,
                "dqlVersion": "V1_0",
                "scannedBytes": 0,
                "analysisTimeframe": {
                    "start": "2024-01-01T12:10:04.123000000Z",
                    "end": "2024-01-20T13:10:04.123000000Z"
                },
                "locale": "en-US",
                "executionTimeMilliseconds": 183,
                "notifications": [],
                "queryId": "c93246ea-dfa8-4f46-a31e-9ec314ad0e92",
                "sampled": false
            }
        }
    }
}

Request URL

GET "http://mydomain:8020/e/1/api/v1/entity/infrastructure/hosts/HOST-123/logs"

Response body

{
    "contentAccess": true,
    "logs": [{
            "path": "Windows Application Log",
            "size": 20975616,
            "availableForAnalysis": true
        }, {
            "path": "Windows System Log",
            "size": 20975616,
            "availableForAnalysis": true
        }, {
            "path": "Windows Security Log",
            "size": 20975616,
            "availableForAnalysis": true
        }
    ]
}

Find process group file log records for a given query

Request URL

POST "https://myapps.mydomain.com/platform/storage/query/v1/query:execute"

Request body

'{
  "query": "fetch logs | filter matchesValue(dt.entity.process_group, \"PROCESS_GROUP-A6835EC7482B0471\")",
  "defaultTimeframeStart": "2024-01-01T12:10:04.123Z",
  "defaultTimeframeEnd": "2024-01-20T13:10:04.123Z",
  "timezone": "UTC",
  "locale": "en_US",
  "maxResultRecords": 1000,
  "maxResultBytes": 1000000,
  "fetchTimeoutSeconds": 60,
  "requestTimeoutMilliseconds": 1000,
  "enablePreview": true,
  "defaultSamplingRatio": 1000,
  "defaultScanLimitGbytes": 100
}'

Response body

{
    "state": "SUCCEEDED",
    "progress": 100,
    "result": {
        "records": [{
                "timestamp": "2024-01-17T08:07:42.000000000Z",
                "content": "2024-01-17 09:07:41 INFO    [<server,0x1>] [ConfigCleanupWorker] ConfigCleanupWorker is now running for tenant: 4",
                "dt.entity.host": "HOST-C9B5229282ED7A48",
                "dt.entity.process_group": "PROCESS_GROUP-A6835EC7482B0471",
                "dt.entity.process_group_instance": "PROCESS_GROUP_INSTANCE-6F367C55CAC67E39",
                "dt.host_group.id": "MyHostGroup",
                "dt.process.name": "com.compuware.apm.server.core.ServerStartup",
                "dt.security_context": "HasHost",
                "dt.source_entity": "PROCESS_GROUP_INSTANCE-6F367C55CAC67E39",
                "event.type": "LOG",
                "host.name": "CT-HGF452",
                "log.source": "C:\\workspaces\\cluster-2\\cluster\\product\\server\\log\\Server_Debug.#.#.log",
                "loglevel": "INFO",
                "process.technology": ["Testowa"],
                "status": "INFO",
                "dt.entity.host_group": "HOST_GROUP-CF1DA380B3A53F17"
            }
        ],
        "types": [{
                "indexRange": [0, 0],
                "mappings": {
                    "timestamp": {
                        "type": "timestamp"
                    },
                    "content": {
                        "type": "string"
                    },
                    "dt.entity.host": {
                        "type": "string"
                    },
                    "dt.entity.process_group": {
                        "type": "string"
                    },
                    "dt.entity.process_group_instance": {
                        "type": "string"
                    },
                    "dt.host_group.id": {
                        "type": "string"
                    },
                    "dt.process.name": {
                        "type": "string"
                    },
                    "dt.security_context": {
                        "type": "string"
                    },
                    "dt.source_entity": {
                        "type": "string"
                    },
                    "event.type": {
                        "type": "string"
                    },
                    "host.name": {
                        "type": "string"
                    },
                    "log.source": {
                        "type": "string"
                    },
                    "loglevel": {
                        "type": "string"
                    },
                    "process.technology": {
                        "type": "array",
                        "types": [{
                                "indexRange": [0, 0],
                                "mappings": {
                                    "element": {
                                        "type": "string"
                                    }
                                }
                            }
                        ]
                    },
                    "status": {
                        "type": "string"
                    },
                    "dt.entity.host_group": {
                        "type": "string"
                    }
                }
            }
        ],
        "metadata": {
            "grail": {
                "canonicalQuery": "fetch logs\n| filter matchesValue(dt.entity.process_group, \"PROCESS_GROUP-A6835EC7482B0471\")",
                "timezone": "Z",
                "query": "fetch logs | filter matchesValue(dt.entity.process_group, \"PROCESS_GROUP-A6835EC7482B0471\") ",
                "scannedRecords": 29689,
                "dqlVersion": "V1_0",
                "scannedBytes": 1654010,
                "analysisTimeframe": {
                    "start": "2024-01-01T12:10:04.123000000Z",
                    "end": "2024-01-20T13:10:04.123000000Z"
                },
                "locale": "en-US",
                "executionTimeMilliseconds": 116,
                "queryId": "2e8c680e-ac9f-49d9-9307-97a5d5f6ef09",
                "sampled": true
            }
        }
    }
}

Request URL

Create a job:

POST "http://mydomain:8020/e/1/api/v1/entity/infrastructure/process-groups/PROCESS_GROUP-95B000C1C3340CEB/logs/%2Fmy%2Fpath?query=my_query&hostFilter=HOST-123"

Get log records:

GET "http://mydomain:8020/e/1/api/v1/entity/infrastructure/process-groups/PROCESS_GROUP-95B000C1C3340CEB/logs/jobs/ac7cfb4c-7683-4ef0-b4e2-9b8b85da4864/records"

Request body

Create a job:

{"filterQuery": ""}

Response body

Create a job:

{"jobId":"ac7cfb4c-7683-4ef0-b4e2-9b8b85da4864"}

Get log records:

{
    "records": [{
            "timestamp": 1705408647765,
            "logLevel": "INFO",
            "hostId": "HOST-123",
            "text": "2024-01-16 13:37:27,765 [79029245]   INFO - My Log Line 1",
            "customFields": {
                "_host.name": "CT-6HFEE71",
                "_process.technology": "Java",
                "_Level": "INFO"
            }
        }]
}

Find number of occurrences of some attributes of a process group file log records

Request URL

POST "https://myapps.mydomain.com/platform/storage/query/v1/query:execute"

Request body

'{
  "query": "fetch logs | filter matchesValue(dt.entity.process_group, \"PROCESS_GROUP-A6835EC7482B0471\") | summarize count() , by:{loglevel}",
  "defaultTimeframeStart": "2024-01-01T12:10:04.123Z",
  "defaultTimeframeEnd": "2024-01-20T13:10:04.123Z",
  "timezone": "UTC",
  "locale": "en_US",
  "maxResultRecords": 1000,
  "maxResultBytes": 1000000,
  "fetchTimeoutSeconds": 60,
  "requestTimeoutMilliseconds": 1000,
  "enablePreview": true,
  "defaultSamplingRatio": 1000,
  "defaultScanLimitGbytes": 100
}'

Response body

{
    "state": "SUCCEEDED",
    "progress": 100,
    "result": {
        "records": [{
                "loglevel": "DEBUG",
                "count()": "2"
            }, {
                "loglevel": "INFO",
                "count()": "39"
            }, {
                "loglevel": "NONE",
                "count()": "313"
            }, {
                "loglevel": "WARN",
                "count()": "2"
            }
        ],
        "types": [{
                "indexRange": [0, 3],
                "mappings": {
                    "loglevel": {
                        "type": "string"
                    },
                    "count()": {
                        "type": "long"
                    }
                }
            }
        ],
        "metadata": {
            "grail": {
                "canonicalQuery": "fetch logs\n| filter matchesValue(dt.entity.process_group, \"PROCESS_GROUP-A6835EC7482B0471\")\n| summarize by:{loglevel}, count()",
                "timezone": "Z",
                "query": "fetch logs | filter matchesValue(dt.entity.process_group, \"PROCESS_GROUP-A6835EC7482B0471\") | summarize count() , by:{loglevel}",
                "scannedRecords": 72789,
                "dqlVersion": "V1_0",
                "scannedBytes": 4333940,
                "analysisTimeframe": {
                    "start": "2024-01-01T12:10:04.123000000Z",
                    "end": "2024-01-20T13:10:04.123000000Z"
                },
                "locale": "en-US",
                "executionTimeMilliseconds": 52,
                "notifications": [],
                "queryId": "c81c9d21-3e88-4157-b3ed-13275d0cf55f",
                "sampled": true
            }
        }
    }
}

Request URL

POST "http://mydomain:8020/e/1/api/v1/entity/infrastructure/process-groups/PROCESS_GROUP-95B000C1C3340CEB/logs/jobs/2268a2a3-edb3-47a1-8f2f-9341e8dad962/records/top"

Request body

{"filterQuery": "_Level = \"INFO\""}

Response body

{
    "parsingFieldTopValues": [{
            "fieldName": "_Level",
            "occurrences": [{
                    "value": "INFO",
                    "count": 362
                }
            ]
        }, {
            "fieldName": "_host.name",
            "occurrences": [{
                    "value": "CT-6HFEE71",
                    "count": 362
                }
            ]
        }, {
            "fieldName": "_process.technology",
            "occurrences": [{
                    "value": "Java",
                    "count": 362
                }
            ]
        }
    ],
    "valuesCount": 3
}