Skip to main content
Skip table of contents

2. Scrolling Audit Logs

Scrolling Audit Logs

This scenario demonstrates how to efficiently retrieve and process large volumes of audit logs using the scrolling mechanism. While standard pagination is suitable for displaying logs in user interfaces, scrolling provides a more efficient approach for processing complete log sets for analysis, reporting, or archiving.

Key features:

  • Efficiently retrieve large volumes of audit logs without pagination limitations

  • Maintain a consistent view of logs during the retrieval process

  • Process complete audit trails for compliance reporting and security analysis

  • Handle millions of log entries with optimal performance

Prerequisites

Before scrolling audit logs, you need:

  • A valid authentication token

  • The "view_logs" permission assigned to your user account

  • Understanding of the scroll mechanism and its lifecycle

  • Sufficient system resources to process the retrieved logs

Scrolling creates server-side resources that must be properly released when finished. Always close scroll contexts when done.

Understanding the Scroll Mechanism

The scroll API is designed for efficiently retrieving large datasets:

  • Scroll Context: A server-side cursor that maintains position in the result set

  • Scroll ID: A unique identifier for the scroll context that must be used in subsequent requests

  • Scroll Timeout: The duration the server keeps the scroll context alive between requests

  • Batch Size: The number of logs retrieved in each scroll request

Unlike pagination, scrolling provides a consistent view of the data even if logs are being added during the scrolling process.

Step 1 - Initialize Scroll Context

This endpoint demonstrates how to initialize a scroll context for retrieving large volumes of audit logs. The scroll context acts as a cursor that maintains your position in the result set, allowing you to efficiently process all matching logs.

  • Creates a server-side scroll context that preserves your position in the result set

  • Returns the first batch of audit logs based on your search criteria

  • Provides a scroll ID that must be used in subsequent scroll requests

  • Maintains a consistent view of the data even if new logs are being added

The scroll timeout specifies in seconds how long the server should keep the scroll context alive between requests. This value refreshes with each request, so you typically only need a few seconds unless processing is very slow.

Endpoint

CODE 
POST /logs/search

Request

JSON 
POST /logs/search?token=NzUHF8vXCvga4dulgyCcS82ptXTqcbsqDO23vgJ9D2UWrEDrKTcjNoyhmGrhbP02

Content-Type: application/json

{
  "logsQuery": {
    "query": {
      "filter": [
        {
          "type": "term",  //Filter for login events only
          "field": "activityType",
          "value": "logged"
        },
        {
          "type": "term",  //Filter for a specific user's activities
          "field": "actorId",
          "value": "mary.griffin@circularo.com"
        }
      ]
    },
    "size": 500  //Number of logs to retrieve in each batch
  },
  "scroll": {
    "timeout": 10  //How long to keep the scroll context alive (in seconds)
  }
}

Response

JSON 
{
  "total": 3254,  //Total number of matching log entries
  "scrollId": "NI1RJnk3SEI0ujws4CCrEcbYO2OzEWwiOJh0VY7D2IXY4EWl1NyGUxzISDGC27MF",  //Unique identifier for the scroll context
  "results": [  //First batch of log entries
    {
      "id": "Kyq2xGbnnw4lyUo2ovWd",
      "activityTarget": "user",
      "activityType": "logged",
      "actorId": "mary.griffin@circularo.com",
      "actorType": "user",
      "timestamp": "2023-08-15T14:32:18.456Z",
      ...
    },
    ...
  ]
}

The response contains the following important properties:

  • scrollId - Located at scrollId in the response.

    • Example value: NI1RJnk3SEI0ujws4CCrEcbYO2OzEWwiOJh0VY7D2IXY4EWl1NyGUxzISDGC27MF

The response contains the first batch of audit logs matching your criteria and a scroll ID for retrieving subsequent batches.

  • The results array contains the first batch of matching log entries

  • The total field indicates the total number of logs matching your criteria

  • The scrollId is a unique identifier for your scroll context that must be used in subsequent requests

  • Process these logs and then use the scroll ID to retrieve the next batch

The scroll ID may be a long, complex string. Always use it exactly as returned without modification.

Step 2 - Retrieve Next Batch of Logs

This endpoint demonstrates how to retrieve subsequent batches of audit logs using the scroll ID from previous requests. This process continues until all matching logs have been retrieved.

  • Uses the scroll ID from the previous response to maintain position in the result set

  • Returns the next batch of audit logs based on your search criteria

  • Refreshes the scroll timeout to keep the context alive

  • Must be called repeatedly until all logs have been retrieved

Do not specify the 'from' parameter in scroll requests as it's incompatible with scrolling. The scroll context already maintains your position in the result set.

Endpoint

CODE 
POST /logs/search

Request

JSON 
POST /logs/search?token=NzUHF8vXCvga4dulgyCcS82ptXTqcbsqDO23vgJ9D2UWrEDrKTcjNoyhmGrhbP02

Content-Type: application/json

{
  "logsQuery": {
    "query": {
      "filter": [
        {
          "type": "term",  //Filter for login events only
          "field": "activityType",
          "value": "logged"
        },
        {
          "type": "term",  //Filter for a specific user's activities
          "field": "actorId",
          "value": "mary.griffin@circularo.com"
        }
      ]
    },
    "size": 500  //Number of logs to retrieve in this batch
  },
  "scroll": {
    "id": "NI1RJnk3SEI0ujws4CCrEcbYO2OzEWwiOJh0VY7D2IXY4EWl1NyGUxzISDGC27MF",  //Scroll ID from the previous response
    "timeout": 10  //Refresh the scroll timeout
  }
}

Response

JSON 
{
  "total": 3254,
  "scrollId": "NI1RJnk3SEI0ujws4CCrEcbYO2OzEWwiOJh0VY7D2IXY4EWl1NyGUxzISDGC27MF",
  "results": [
    {
      "id": "0VgqIM5Tvw4pjj3YB1VH",
      "activityTarget": "user",
      "activityType": "logged",
      "actorId": "mary.griffin@circularo.com",
      "actorType": "user",
      "timestamp": "2023-08-15T14:35:22.789Z",
      ...
    },
    ...
  ]
}

The response contains the following important properties:

  • scrollId - Located at scrollId in the response.

    • Example value: NI1RJnk3SEI0ujws4CCrEcbYO2OzEWwiOJh0VY7D2IXY4EWl1NyGUxzISDGC27MF

The response contains the next batch of audit logs and a scroll ID for retrieving subsequent batches.

  • The results array contains the next batch of matching log entries

  • The total field remains the same as in the initial request

  • The scrollId may be the same or different from the previous request

  • Continue this process until the results array is empty or smaller than the requested size

Always use the scrollId from the most recent response, as it may change between requests.

Step 3 - Close Scroll Context

This endpoint demonstrates how to properly close a scroll context when you've finished retrieving all needed logs. Closing the context releases server resources and is an important best practice.

  • Explicitly releases server resources associated with the scroll context

  • Should be called when you've retrieved all needed logs or no longer need the context

  • Requires the scroll ID from your most recent scroll request

Endpoint

CODE 
DELETE /logs/scroll

Request

JSON 
DELETE /logs/scroll?token=NzUHF8vXCvga4dulgyCcS82ptXTqcbsqDO23vgJ9D2UWrEDrKTcjNoyhmGrhbP02

Content-Type: application/json

{
  "scrollId": "NI1RJnk3SEI0ujws4CCrEcbYO2OzEWwiOJh0VY7D2IXY4EWl1NyGUxzISDGC27MF"  //Scroll ID from the most recent response
}

Response

JSON 
{
  "status": "SUCCESS"
}

The scroll context has been successfully closed and all associated server resources have been released.

After closing a scroll context, its scroll ID becomes invalid. If you need to retrieve more logs, you'll need to initialize a new scroll context.

Audit Log Scrolling Summary

You have successfully learned how to efficiently retrieve and process large volumes of audit logs using the scrolling mechanism.

Key Concepts

  • Scroll Context: A server-side cursor that maintains position in the result set

  • Scroll ID: A unique identifier for the scroll context used in subsequent requests

  • Scroll Lifecycle: Initialize, retrieve batches, close when finished

  • Batch Processing: Retrieve and process logs in manageable chunks

  • Resource Management: Properly close contexts to prevent server resource leaks

When to Use Scrolling vs. Pagination

  • Use Scrolling: For processing complete log sets, data exports, or deep analysis

  • Use Pagination: For displaying logs in user interfaces or when only a subset of logs is needed

Example Implementation

See our OpenAPI documentation to learn about the full set of API endpoints and parameters.

Please use proper exception handling and function decomposition in your own code. The code is provided for illustrative purposes only and is not intended for production use.

JAVASCRIPT 
// Audit log scrolling example
const URL = "https://sandbox.circularo.com";
const API_PATH = "/api/v1";
const TOKEN = "YOUR_TOKEN"; // Must have "view_logs" permission

// Function to process each batch of logs
function processLogBatch(logs) {
    for (const log of logs) {
        // Process each log as needed
        console.log(`Processing log: ${log.id}`);

        // Implement your log processing logic here
    }
}

try {
    // Step 1: Initialize scroll context
    let scrollId;
    const batchSize = 500;
    const scrollTimeout = 10; // seconds

    const initScrollResponse = await fetch(`${URL}${API_PATH}/logs/search?token=${TOKEN}`, {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json'
        },
        body: JSON.stringify({
            logsQuery: {
                query: {
                    filter: [
                        {
                            type: "term",
                            field: "activityType",
                            value: "logged"
                        },
                        {
                            type: "term",
                            field: "actorId",
                            value: "john.smith@example.com"
                        }
                    ]
                },
                size: batchSize // Number of logs per batch
            },
            scroll: {
                timeout: scrollTimeout // Keep context alive for 10 seconds between requests
            }
        })
    });
    if (!initScrollResponse.ok) {
        throw new Error(`Failed to initialize scroll: ${initScrollResponse.status} ${initScrollResponse.statusText}`);
    }

    let responseData = await initScrollResponse.json();
    scrollId = responseData.scrollId;

    // Step 2: Process the batch and continue scrolling until all logs are processed
    while (responseData.results.length > 0) {
        processLogBatch(responseData.results);

        const scrollResponse = await fetch(`${URL}${API_PATH}/logs/search?token=${TOKEN}`, {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json'
            },
            body: JSON.stringify({
                logsQuery: {
                    query: {
                        filter: [
                            {
                                type: "term",
                                field: "activityType",
                                value: "logged"
                            },
                            {
                                type: "term",
                                field: "actorId",
                                value: "john.smith@example.com"
                            }
                        ]
                    },
                    size: batchSize
                },
                scroll: {
                    id: scrollId,
                    timeout: scrollTimeout
                }
            })
        });
        if (!scrollResponse.ok) {
            throw new Error(`Scroll request failed: ${scrollResponse.status} ${scrollResponse.statusText}`);
        }

        responseData = await scrollResponse.json();
        scrollId = responseData.scrollId;
    }

    console.log('Log processing complete');

} catch (error) {
    console.error('Error processing audit logs:', error.message);

} finally {
    // Step 3: Always try to close the scroll context if we have a scrollId, even if processing failed
    if (scrollId) {
        try {
            await fetch(`${URL}${API_PATH}/logs/scroll?token=${TOKEN}`, {
                method: 'DELETE',
                headers: { 'Content-Type': 'application/json' },
                body: JSON.stringify({
                    scrollId: scrollId
                })
            });
            console.log('Scroll context closed successfully');

        } catch (closeError) {
            console.error('Failed to close scroll context:', closeError.message);
        }
    }
}
JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.

Contact Support Close