3. Transaction Report Scrolling

Transaction Report Scrolling

This scenario demonstrates how to efficiently retrieve and process large volumes of transaction detail reports using the scrolling mechanism. When dealing with extensive document collections, scrolling provides a more efficient approach than standard pagination for processing complete transaction datasets for analysis, reporting, or archiving.

Key features:

• Efficiently retrieve large volumes of transaction reports without pagination limitations

• Maintain a consistent view of transaction data during the retrieval process

• Process complete transaction datasets for compliance reporting and business analysis

• Handle thousands or millions of document records with optimal performance

Prerequisites

Before scrolling transaction reports, you need:

• A valid authentication token

• The use_all_documents_export permission assigned to your user account

• Understanding of the scroll mechanism and its lifecycle

Understanding Transaction Detail Reports

Transaction detail reports provide comprehensive information about document activities:

• Document Metadata: Title, ID, type, creation date, and owner information

• Workflow States: Current status of documents (completed, in progress, rejected, etc.)

• Transaction History: Record of actions performed on documents

For more information about transaction reports and their contents, see the Circularo documentation - Transaction Detail Report

Understanding the Scroll Mechanism

The scroll API is designed for efficiently retrieving large datasets:

• Scroll Context: A server-side cursor that maintains position in the result set

• Scroll ID: A unique identifier for the scroll context that must be used in subsequent requests

• Scroll Timeout: The duration in seconds the server keeps the scroll context alive between requests

• Batch Size: The number of transaction records retrieved in each scroll request

Unlike pagination, scrolling provides a consistent view of the data even if documents are being added or modified during the scrolling process.

Step 1 - Initialize Scroll Context for Transaction Reports

This endpoint demonstrates how to initialize a scroll context for retrieving large volumes of transaction detail reports. The scroll context acts as a cursor that maintains your position in the result set, allowing you to efficiently process all matching transaction records.

• Creates a server-side scroll context that preserves your position in the result set

• Returns the first batch of transaction records based on your criteria

• Provides a scroll ID that must be used in subsequent scroll requests

• Maintains a consistent view of the data even if new documents are being added

The scroll timeout specifies how long (in seconds) the server should keep the scroll context alive between requests. This value refreshes with each request, so you typically only need a few seconds unless processing is very slow.

Endpoint

POST /search/reports

Request

POST /search/reports?token=FKHzYKJDudFItMRc73XkFGsOHwDscsvLHt7w41yQNLwqylz3KhWmcamIDTuO6cm2

Content-Type: application/json

{
  "query": {
    "size": 500,  //Number of transaction records to retrieve in each batch
    "scroll": {
      "timeout": 10  //How long to keep the scroll context alive (in seconds)
    }
  }
}

Response

{
  "from": 0,  //Starting position in the result set
  "total": 18992,  //Total number of matching transaction records
  "scrollId": "2KnXvFlxHCMqNH3du777vtqrGpN0PJYku4fXfcXNq2SDlW8fQQzNNbaDtTA6GccF",  //Unique identifier for the scroll context
  "results": [  //First batch of transaction records
    {
      "documentTitle": "Invoices",
      "documentId": "bac8dba9-b128-4f6f-83d7-66d0f6d0865b",
      ...
    },
    ...
  ]
}

The response contains the following important properties:

• scrollId - Located at scrollId in the response.

  • Example value: 2KnXvFlxHCMqNH3du777vtqrGpN0PJYku4fXfcXNq2SDlW8fQQzNNbaDtTA6GccF

The response contains the first batch of transaction records and a scroll ID for retrieving subsequent batches.

• The results array contains the first batch of transaction records

• The total field indicates the total number of records matching your criteria

• The scrollId is a unique identifier for your scroll context that must be used in subsequent requests

• Process these records and then use the scroll ID to retrieve the next batch

The scroll ID may be a long, complex string. Always use it exactly as returned without modification.

Step 2 - Retrieve Next Batch of Transaction Records

This endpoint demonstrates how to retrieve subsequent batches of transaction records using the scroll ID from previous requests. This process continues until all matching records have been retrieved.

• Uses the scroll ID from the previous response to maintain position in the result set

• Returns the next batch of transaction records based on your criteria

• Refreshes the scroll timeout to keep the context alive

• Must be called repeatedly until all records have been retrieved

Do not specify the 'from' parameter in scroll requests as it's incompatible with scrolling. The scroll context already maintains your position in the result set.

Endpoint

POST /search/reports

Request

POST /search/reports?token=FKHzYKJDudFItMRc73XkFGsOHwDscsvLHt7w41yQNLwqylz3KhWmcamIDTuO6cm2

Content-Type: application/json

{
  "query": {
    "size": 500,  //Number of transaction records to retrieve in this batch
    "scroll": {
      "id": "2KnXvFlxHCMqNH3du777vtqrGpN0PJYku4fXfcXNq2SDlW8fQQzNNbaDtTA6GccF",  //Scroll ID from the previous response
      "timeout": 10  //Refresh the scroll timeout
    }
  }
}

Response

{
  "total": 18992,  //Total number of matching transaction records
  "scrollId": "2KnXvFlxHCMqNH3du777vtqrGpN0PJYku4fXfcXNq2SDlW8fQQzNNbaDtTA6GccF",  //Scroll ID for the next request
  "results": [  //Next batch of transaction records
    {
      "documentTitle": "Memo 2025",
      "documentId": "f34bd356-bec8-443f-b670-0edfee8177de",
      ...
    },
    ...
  ]
}

The response contains the following important properties:

• scrollId - Located at scrollId in the response.

  • Example value: 2KnXvFlxHCMqNH3du777vtqrGpN0PJYku4fXfcXNq2SDlW8fQQzNNbaDtTA6GccF

The response contains the next batch of transaction records and a scroll ID for retrieving subsequent batches.

• The results array contains the next batch of transaction records

• The total field remains the same as in the initial request

• The scrollId may be the same or different from the previous request

• Continue this process until the results array is empty or smaller than the requested size

Always use the scrollId from the most recent response, as it may change between requests.

Step 3 - Close Scroll Context

This endpoint demonstrates how to properly close a scroll context when you've finished retrieving all needed transaction records. Closing the context releases server resources and is an important best practice.

• Explicitly releases server resources associated with the scroll context

• Should be called when you've retrieved all needed records or no longer need the context

• Requires the scroll ID from your most recent scroll request

Endpoint

DELETE /search/scroll

Request

DELETE /search/scroll?token=FKHzYKJDudFItMRc73XkFGsOHwDscsvLHt7w41yQNLwqylz3KhWmcamIDTuO6cm2

Content-Type: application/json

{
  "scrollId": "2KnXvFlxHCMqNH3du777vtqrGpN0PJYku4fXfcXNq2SDlW8fQQzNNbaDtTA6GccF"  //Scroll ID from the most recent response
}

Response

{
  "status": "SUCCESS"
}

The scroll context has been successfully closed and all associated server resources have been released.

After closing a scroll context, its scroll ID becomes invalid. If you need to retrieve more transaction records, you'll need to initialize a new scroll context.

Transaction Report Scrolling Summary

You have successfully learned how to efficiently retrieve and process large volumes of transaction detail reports using the scrolling mechanism.

Key Concepts

• Scroll Context: A server-side cursor that maintains position in the result set

• Scroll ID: A unique identifier for the scroll context used in subsequent requests

• Scroll Lifecycle: Initialize, retrieve batches, close when finished

• Batch Processing: Retrieve and process transaction records in manageable chunks

• Resource Management: Properly close contexts to prevent server resource leaks

When to Use Scrolling vs. Pagination

• Use Scrolling: For processing complete transaction sets, data exports, or deep analysis

• Use Pagination: For displaying transaction records in user interfaces or when only a subset of records is needed

Next Steps

With your understanding of transaction report scrolling, you can now:

• Implement efficient report processing for compliance reporting

• Create data export functionality for archiving or external analysis

• Build comprehensive business intelligence tools that process complete transaction datasets

• Combine with filtering capabilities to focus on specific transaction subsets while maintaining efficiency

Example Implementation

See our OpenAPI documentation to learn about the full set of API endpoints and parameters.

Please use proper exception handling and function decomposition in your own code. The code is provided for illustrative purposes only and is not intended for production use.

// Transaction report scrolling example
const URL = "https://sandbox.circularo.com";
const API_PATH = "/api/v1";
const TOKEN = "YOUR_TOKEN"; // Must have "use_all_documents_export" permission

// Helper function to process each batch of transaction records
function processTransactionBatch(transactions) {
    for (const transaction of transactions) {
        console.log(`Processing transaction for document: ${transaction.documentTitle} (${transaction.documentId})`);

        // Add your transaction processing logic here
        // For example:
        // - Extract data for reporting
        // - Update your database with transaction information
        // - Generate statistics or analytics
    }
}

try {
    // Step 1: Initialize scroll context
    const batchSize = 500;
    const scrollTimeout = 10; // seconds
    let scrollId;

    const initScrollResponse = await fetch(`${URL}${API_PATH}/search/reports?token=${TOKEN}`, {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json'
        },
        body: JSON.stringify({
            query: {
                size: batchSize,
                scroll: {
                    timeout: scrollTimeout
                }
            }
        })
    });
    if (!initScrollResponse.ok) {
        throw new Error(`Failed to initialize scroll: ${initScrollResponse.status} ${initScrollResponse.statusText}`);
    }

    let responseData = await initScrollResponse.json();
    console.log(`Found ${responseData.total} transaction records. Processing in batches...`);
    scrollId = responseData.scrollId;

    // Step 2: Process the batch and continue scrolling until all reports are processed
    while (responseData.results.length > 0) {
        processTransactionBatch(responseData.results);

        const scrollResponse = await fetch(`${URL}${API_PATH}/search/reports?token=${TOKEN}`, {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json'
            },
            body: JSON.stringify({
                query: {
                    size: batchSize,
                    scroll: {
                        id: scrollId,
                        timeout: scrollTimeout
                    }
                }
            })
        });
        if (!scrollResponse.ok) {
            throw new Error(`Scroll request failed: ${scrollResponse.status} ${scrollResponse.statusText}`);
        }

        responseData = await scrollResponse.json();
        scrollId = responseData.scrollId;
    }

    console.log('Transaction report processing complete');

} catch (error) {
    console.error('Error processing transaction reports:', error.message);

} finally {
    // Step 3: Always try to close the scroll context if we have a scrollId
    if (scrollId) {
        try {
            await fetch(`${URL}${API_PATH}/search/scroll?token=${TOKEN}`, {
                method: 'DELETE',
                headers: { 'Content-Type': 'application/json' },
                body: JSON.stringify({
                    scrollId: scrollId
                })
            });
            console.log('Scroll context closed successfully');

        } catch (closeError) {
            console.error('Failed to close scroll context:', closeError.message);
        }
    }
}