Log_Harvestor_Icon

Login

Sign up

Notifications

Logs

Logs

The Log:

Logs, or events, are data records that indicate the state of a system at a specific point in time.

Lifecycle of a log:

  • A forwarder collects or generated a log
  • The log is sent ("forwarded") by the forwarder to Log Harvestor
  • Log Harvestor validates the log and the forwarder that sent it - invalid logs are rejected
  • If the log is valid, it is then processed
  • The log is then available for search and analysis
  • Once the log has reached a certain age, it is automatically rotated into cold storage/archival.

    • Log Processing:

  • Parsing the log into a structured format
  • Enriching the log with additional data
  • Indexing the log for search
  • Storing the log in a database
  • Sending if any notifications if necessary

    • Log Rotation:

  • Log rotation is determined by the rentention policy for the project and organization that the log belongs to.
  • Once a log has reached the end of its retention period, it is automatically rotated into cold storage/archival.
  • Once a log has been rotated, it is no longer available for search or analysis.
  • A log will remain in cold storage/archival until it is deleted by the user.

    • Log Billing:

  • For more information on how logs are billed, see the pricing section.
  • A log is only billed for once it has been processed and stored.
  • Logs are billed based on the quantity, not their size.

    • Flagging:

  • Flags help users identify logs that may be problematic.
  • Flags are visbile to all users who have access to the log .

    • Notifications:

  • Notifications are sent to users when a log matches a certain criteria.
  • These criteria can be configured by the user in the notifications.
  • For more information on how notifications work, see the notifications section.

    • Comments:

  • Comments allow users to add additional information to a log .
  • Comments are stored as {events} on the log .

    • Get All

    Description

    Get all

    Parameters

    headers

    Authorization

    string

    Required

    Description:

    Bearer token - JWT

    Example:

    "Bearer <token>"

    query

    limit

    number

    Description:

    The number of results to return. Default is 50.

    Default:

    10

    Example:

    10

    page

    number

    Description:

    The page number to return. Default is 1.

    Default:

    1

    Example:

    1

    sort

    string

    Description:

    The field to sort by. Default is t_ca.

    Default:

    t_ca

    Example:

    "t_ca"

    filter

    object

    Description:

    The filter to apply. Default is {}.

    Example:

    LogQueryFilter

    Responses

    Type:

    LogQueryResponse object

    Example:

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
{
    docs: {
        __type: "Log",
        _id: {
            key: "_id",
            type: "string",
            example: "0f9c0b0b4f0c6b0017f0e3b1"
        },
        typ: {
            key: "typ",
            type: "string",
            example: "error"
        },
        msg: {
            key: "msg",
            type: "string",
            example: "Error message"
        },
        fwd: {
            key: "fwd",
            type: "string",
            example: "0f9c0b0b4f0c6b0017f0e3b1"
        },
        tot_comments: {
            key: "tot_comments",
            type: "number",
            example: 1
        },
        t_ca: {
            key: "t_ca",
            type: "number",
            example: 168910405000
        },
        t_ua: {
            key: "t_ua",
            type: "number",
            example: 168910405000
        }
    },
    limit: 10,
    page: 1,
    nextPage: 2,
    hasNextPage: true,
    prevPage: 1,
    pagingCounter: 1,
    totalDocs: 100,
    totalPages: 10
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Invalid Role",
    message: "You do not have the required role to access this resource",
    status: 401
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Entity not found",
    message: "Entity not found",
    status: 404
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Internal Server Error",
    message: "Internal Server Error",
    status: 500
}

    get

    /log/

    Request

    1
2
3
4
curl https://logharvestor.com/api/v1/log/ \
    -X GET \
    -H 'Authorization: Bearer <token>' \

    Response

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
{
    docs: {
        __type: "Log",
        _id: {
            key: "_id",
            type: "string",
            example: "0f9c0b0b4f0c6b0017f0e3b1"
        },
        typ: {
            key: "typ",
            type: "string",
            example: "error"
        },
        msg: {
            key: "msg",
            type: "string",
            example: "Error message"
        },
        fwd: {
            key: "fwd",
            type: "string",
            example: "0f9c0b0b4f0c6b0017f0e3b1"
        },
        tot_comments: {
            key: "tot_comments",
            type: "number",
            example: 1
        },
        t_ca: {
            key: "t_ca",
            type: "number",
            example: 168910405000
        },
        t_ua: {
            key: "t_ua",
            type: "number",
            example: 168910405000
        }
    },
    limit: 10,
    page: 1,
    nextPage: 2,
    hasNextPage: true,
    prevPage: 1,
    pagingCounter: 1,
    totalDocs: 100,
    totalPages: 10
}

    Get By Id

    Description

    Find the forwarder that matches the {id} query parameter.

    Parameters

    headers

    Authorization

    string

    Required

    Description:

    Bearer token - JWT

    Example:

    "Bearer <token>"

    params

    id

    string

    Description:

    The id of the Log

    Example:

    "0f9c0b0b4f0c6b0017f0e3b1"

    Responses

    Type:

    Log object

    Example:

    1
2
3
4
5
6
7
8
9
{
    _id: "0f9c0b0b4f0c6b0017f0e3b1",
    typ: "error",
    msg: "Error message",
    fwd: "0f9c0b0b4f0c6b0017f0e3b1",
    tot_comments: 1,
    t_ca: 168910405000,
    t_ua: 168910405000
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Invalid Role",
    message: "You do not have the required role to access this resource",
    status: 401
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Entity not found",
    message: "Entity not found",
    status: 404
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Internal Server Error",
    message: "Internal Server Error",
    status: 500
}

    get

    /log/:id

    Request

    1
2
3
4
curl https://logharvestor.com/api/v1/log/:id \
    -X GET \
    -H 'Authorization: Bearer <token>' \

    Response

    1
2
3
4
5
6
7
8
9
{
    _id: "0f9c0b0b4f0c6b0017f0e3b1",
    typ: "error",
    msg: "Error message",
    fwd: "0f9c0b0b4f0c6b0017f0e3b1",
    tot_comments: 1,
    t_ca: 168910405000,
    t_ua: 168910405000
}

    Get Events

    Description

    Get all the events for a log. These events include things like {comments}, {flags}, [notifications](/docs/api/notifications), and when a log is created, updated, or rotated.

    Parameters

    headers

    Authorization

    string

    Required

    Description:

    Bearer token - JWT

    Example:

    "Bearer <token>"

    params

    id

    string

    Description:

    The id of the Log to find events for.

    Example:

    "0f9c0b0b4f0c6b0017f0e3b1"

    Responses

    Type:

    Array of Event objects

    Example:

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{
    _id: "0f9c0b0b4f0c6b0017f0e3b1",
    type: "updated",
    message: "User 0f9c0b0b4f0c6b0017f0e3b1 updated forwarder 0f9c0b0b4f0c6b0017f0e3b1",
    user: "0f9c0b0b4f0c6b0017f0e3b1",
    entity: "0f9c0b0b4f0c6b0017f0e3b1",
    entityType: "Forwarder",
    refs: [
        {
            entity: "0f9c0b0b4f0c6b0017f0e3b1",
            entityType: "Forwarder"
        },
        {
            entity: "0f9c0b0b4f0c6b0017f0e3b1",
            entityType: "Project"
        }
    ]
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Invalid Role",
    message: "You do not have the required role to access this resource",
    status: 401
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Entity not found",
    message: "Entity not found",
    status: 404
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Internal Server Error",
    message: "Internal Server Error",
    status: 500
}

    get

    /log/:id/events

    Request

    1
2
3
4
curl https://logharvestor.com/api/v1/log/:id/events \
    -X GET \
    -H 'Authorization: Bearer <token>' \

    Response

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{
    _id: "0f9c0b0b4f0c6b0017f0e3b1",
    type: "updated",
    message: "User 0f9c0b0b4f0c6b0017f0e3b1 updated forwarder 0f9c0b0b4f0c6b0017f0e3b1",
    user: "0f9c0b0b4f0c6b0017f0e3b1",
    entity: "0f9c0b0b4f0c6b0017f0e3b1",
    entityType: "Forwarder",
    refs: [
        {
            entity: "0f9c0b0b4f0c6b0017f0e3b1",
            entityType: "Forwarder"
        },
        {
            entity: "0f9c0b0b4f0c6b0017f0e3b1",
            entityType: "Project"
        }
    ]
}

    Get Similar

    Description

    Get a list of logs that are similar. This list is generated by comparing the target log to other logs that have been processed, and are not rotated or archived.

    Parameters

    headers

    Authorization

    string

    Required

    Description:

    Bearer token - JWT

    Example:

    "Bearer <token>"

    params

    id

    string

    Description:

    The id of the Log

    Example:

    "0f9c0b0b4f0c6b0017f0e3b1"

    Responses

    Type:

    Array of Log objects

    Example:

    1
2
3
4
5
6
7
8
9
{
    _id: "0f9c0b0b4f0c6b0017f0e3b1",
    typ: "error",
    msg: "Error message",
    fwd: "0f9c0b0b4f0c6b0017f0e3b1",
    tot_comments: 1,
    t_ca: 168910405000,
    t_ua: 168910405000
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Invalid Role",
    message: "You do not have the required role to access this resource",
    status: 401
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Entity not found",
    message: "Entity not found",
    status: 404
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Internal Server Error",
    message: "Internal Server Error",
    status: 500
}

    get

    /log/:id/similar

    Request

    1
2
3
4
curl https://logharvestor.com/api/v1/log/:id/similar \
    -X GET \
    -H 'Authorization: Bearer <token>' \

    Response

    1
2
3
4
5
6
7
8
9
{
    _id: "0f9c0b0b4f0c6b0017f0e3b1",
    typ: "error",
    msg: "Error message",
    fwd: "0f9c0b0b4f0c6b0017f0e3b1",
    tot_comments: 1,
    t_ca: 168910405000,
    t_ua: 168910405000
}

    Get log types

    Description

    Get a list of all log types - log types are fetched from logs that have been processed, and are not rotated or archived.

    Parameters

    headers

    Authorization

    string

    Required

    Description:

    Bearer token - JWT

    Example:

    "Bearer <token>"

    Responses

    Example:

    1
2
3
4
5
[
    "error",
    "info",
    "warning"
]

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Invalid Role",
    message: "You do not have the required role to access this resource",
    status: 401
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Entity not found",
    message: "Entity not found",
    status: 404
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Internal Server Error",
    message: "Internal Server Error",
    status: 500
}

    get

    /log/types

    Request

    1
2
3
4
curl https://logharvestor.com/api/v1/log/types \
    -X GET \
    -H 'Authorization: Bearer <token>' \

    Response

    1
2
3
4
5
[
    "error",
    "info",
    "warning"
]

    Send one

    Description

    Send a single log to be processed - The forwarder that sends the log must be enabled. The fwd attribute for the log will be derived from the authorization token.

    Parameters

    headers

    Authorization

    string

    Required

    Description:

    Bearer token - JWT

    Example:

    "Bearer <token>"

    body

    typ

    string

    Description:

    The type of the log

    Example:

    "error"

    msg

    string

    Description:

    The message of the log

    Example:

    "Error message"

    Responses

    Type:

    Log object

    Example:

    1
2
3
4
5
6
7
8
9
{
    _id: "0f9c0b0b4f0c6b0017f0e3b1",
    typ: "error",
    msg: "Error message",
    fwd: "0f9c0b0b4f0c6b0017f0e3b1",
    tot_comments: 1,
    t_ca: 168910405000,
    t_ua: 168910405000
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Invalid Role",
    message: "You do not have the required role to access this resource",
    status: 401
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Entity not found",
    message: "Entity not found",
    status: 404
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Internal Server Error",
    message: "Internal Server Error",
    status: 500
}

    post

    /log/

    Request

    1
2
3
4
5
curl https://logharvestor.com/api/v1/log/ \
    -X POST \
    -H 'Authorization: Bearer <token>' \
    -H 'Content-Type: application/json' \
	--data-raw '{"typ":"\"error\"","msg":"\"Error message\""}'

    Response

    1
2
3
4
5
6
7
8
9
{
    _id: "0f9c0b0b4f0c6b0017f0e3b1",
    typ: "error",
    msg: "Error message",
    fwd: "0f9c0b0b4f0c6b0017f0e3b1",
    tot_comments: 1,
    t_ca: 168910405000,
    t_ua: 168910405000
}

    Flag

    Description

    Flag a log

    Parameters

    headers

    Authorization

    string

    Required

    Description:

    Bearer token - JWT

    Example:

    "Bearer <token>"

    params

    id

    string

    Description:

    The id of the [log](/docs/api/logs)

    Example:

    "0f9c0b0b4f0c6b0017f0e3b1"

    Responses

    Description:

    The flagged log

    Type:

    Log object

    Example:

    1
2
3
4
5
6
7
8
9
{
    _id: "0f9c0b0b4f0c6b0017f0e3b1",
    typ: "error",
    msg: "Error message",
    fwd: "0f9c0b0b4f0c6b0017f0e3b1",
    tot_comments: 1,
    t_ca: 168910405000,
    t_ua: 168910405000
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Invalid Role",
    message: "You do not have the required role to access this resource",
    status: 401
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Entity not found",
    message: "Entity not found",
    status: 404
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Internal Server Error",
    message: "Internal Server Error",
    status: 500
}

    post

    /log/:id/flag

    Request

    1
2
3
4
curl https://logharvestor.com/api/v1/log/:id/flag \
    -X POST \
    -H 'Authorization: Bearer <token>' \

    Response

    1
2
3
4
5
6
7
8
9
{
    _id: "0f9c0b0b4f0c6b0017f0e3b1",
    typ: "error",
    msg: "Error message",
    fwd: "0f9c0b0b4f0c6b0017f0e3b1",
    tot_comments: 1,
    t_ca: 168910405000,
    t_ua: 168910405000
}

    Unflag

    Description

    Unflag a log

    Parameters

    headers

    Authorization

    string

    Required

    Description:

    Bearer token - JWT

    Example:

    "Bearer <token>"

    params

    id

    string

    Description:

    The id of the Log

    Example:

    "0f9c0b0b4f0c6b0017f0e3b1"

    Responses

    Description:

    The unflagged [log](/docs/api/logs)

    Type:

    Log object

    Example:

    1
2
3
4
5
6
7
8
9
{
    _id: "0f9c0b0b4f0c6b0017f0e3b1",
    typ: "error",
    msg: "Error message",
    fwd: "0f9c0b0b4f0c6b0017f0e3b1",
    tot_comments: 1,
    t_ca: 168910405000,
    t_ua: 168910405000
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Invalid Role",
    message: "You do not have the required role to access this resource",
    status: 401
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Entity not found",
    message: "Entity not found",
    status: 404
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Internal Server Error",
    message: "Internal Server Error",
    status: 500
}

    post

    /log/:id/unflag

    Request

    1
2
3
4
curl https://logharvestor.com/api/v1/log/:id/unflag \
    -X POST \
    -H 'Authorization: Bearer <token>' \

    Response

    1
2
3
4
5
6
7
8
9
{
    _id: "0f9c0b0b4f0c6b0017f0e3b1",
    typ: "error",
    msg: "Error message",
    fwd: "0f9c0b0b4f0c6b0017f0e3b1",
    tot_comments: 1,
    t_ca: 168910405000,
    t_ua: 168910405000
}

    Comment

    Description

    Add a {comment} to a log

    Parameters

    headers

    Authorization

    string

    Required

    Description:

    Bearer token - JWT

    Example:

    "Bearer <token>"

    params

    id

    string

    Description:

    The id of the Log

    Example:

    "0f9c0b0b4f0c6b0017f0e3b1"

    body

    message

    string

    Description:

    The message of the comment

    Example:

    "Error message"

    Responses

    Description:

    The updated log

    Type:

    Log object

    Example:

    1
2
3
4
5
6
7
8
9
{
    _id: "0f9c0b0b4f0c6b0017f0e3b1",
    typ: "error",
    msg: "Error message",
    fwd: "0f9c0b0b4f0c6b0017f0e3b1",
    tot_comments: 1,
    t_ca: 168910405000,
    t_ua: 168910405000
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Invalid Role",
    message: "You do not have the required role to access this resource",
    status: 401
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Entity not found",
    message: "Entity not found",
    status: 404
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Internal Server Error",
    message: "Internal Server Error",
    status: 500
}

    post

    /log/:id/comment

    Request

    1
2
3
4
5
curl https://logharvestor.com/api/v1/log/:id/comment \
    -X POST \
    -H 'Authorization: Bearer <token>' \
    -H 'Content-Type: application/json' \
	--data-raw '{"message":"\"Error message\""}'

    Response

    1
2
3
4
5
6
7
8
9
{
    _id: "0f9c0b0b4f0c6b0017f0e3b1",
    typ: "error",
    msg: "Error message",
    fwd: "0f9c0b0b4f0c6b0017f0e3b1",
    tot_comments: 1,
    t_ca: 168910405000,
    t_ua: 168910405000
}

    Archive

    Description

    Archive a log

    Parameters

    headers

    Authorization

    string

    Required

    Description:

    Bearer token - JWT

    Example:

    "Bearer <token>"

    params

    id

    string

    Description:

    The id of the Log

    Example:

    "0f9c0b0b4f0c6b0017f0e3b1"

    Responses

    Description:

    The archived log

    Type:

    Log object

    Example:

    1
2
3
4
5
6
7
8
9
{
    _id: "0f9c0b0b4f0c6b0017f0e3b1",
    typ: "error",
    msg: "Error message",
    fwd: "0f9c0b0b4f0c6b0017f0e3b1",
    tot_comments: 1,
    t_ca: 168910405000,
    t_ua: 168910405000
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Invalid Role",
    message: "You do not have the required role to access this resource",
    status: 401
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Entity not found",
    message: "Entity not found",
    status: 404
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Internal Server Error",
    message: "Internal Server Error",
    status: 500
}

    post

    /log/:id/archive

    Request

    1
2
3
4
curl https://logharvestor.com/api/v1/log/:id/archive \
    -X POST \
    -H 'Authorization: Bearer <token>' \

    Response

    1
2
3
4
5
6
7
8
9
{
    _id: "0f9c0b0b4f0c6b0017f0e3b1",
    typ: "error",
    msg: "Error message",
    fwd: "0f9c0b0b4f0c6b0017f0e3b1",
    tot_comments: 1,
    t_ca: 168910405000,
    t_ua: 168910405000
}

    Unarchive

    Description

    Unarchive a log

    Parameters

    headers

    Authorization

    string

    Required

    Description:

    Bearer token - JWT

    Example:

    "Bearer <token>"

    params

    id

    string

    Description:

    The id of the Log

    Example:

    "0f9c0b0b4f0c6b0017f0e3b1"

    Responses

    Description:

    The unarchived log

    Type:

    Log object

    Example:

    1
2
3
4
5
6
7
8
9
{
    _id: "0f9c0b0b4f0c6b0017f0e3b1",
    typ: "error",
    msg: "Error message",
    fwd: "0f9c0b0b4f0c6b0017f0e3b1",
    tot_comments: 1,
    t_ca: 168910405000,
    t_ua: 168910405000
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Invalid Role",
    message: "You do not have the required role to access this resource",
    status: 401
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Entity not found",
    message: "Entity not found",
    status: 404
}

    Type:

    Error object

    Example:

    1
2
3
4
5
{
    name: "Internal Server Error",
    message: "Internal Server Error",
    status: 500
}

    post

    /log/:id/unarchive

    Request

    1
2
3
4
curl https://logharvestor.com/api/v1/log/:id/unarchive \
    -X POST \
    -H 'Authorization: Bearer <token>' \

    Response

    1
2
3
4
5
6
7
8
9
{
    _id: "0f9c0b0b4f0c6b0017f0e3b1",
    typ: "error",
    msg: "Error message",
    fwd: "0f9c0b0b4f0c6b0017f0e3b1",
    tot_comments: 1,
    t_ca: 168910405000,
    t_ua: 168910405000
}

    Forwarders

    Notifications

    Log_Harvestor_Icon

    Join our newsletter

    support@logharvestor.com

    Copyright © Log Harvestor 2023 • Terms • Privacy