Paginated Data Source

Some REST API endpoints limit the amount of data that can be processed at once. There may be a case where not all data can be sent in a single request. In this case, paginated data sources can be used.

Setup

When creating a new data source of the type Paginated Data Source, the following values have to be filled in:

Data Source

Select the data source you want to use for pagination. Depending on the type of pagination, this data source must provide variables to control the pagination process.

Pagination Type

You can choose between 3 different pagination types:

Page-based uses a page variable which starts at 1 and will increase by one for each following page.
Page: 1, 2, 3, …

Offset-based uses an offset variable which starts at 0 and will increase by the amount of items returned by each response.
Offset: 0,10,20,…

Cursor-based uses a cursor variable which starts empty and will send the cursor value from the last response.

Follow URL uses an URL to request the next page.

Offset / Page / Cursor Variable

Depending on the type of pagination, use the appropriate variable provided by the data source.

Result Selector

Each page response must be added to the finished result. This JSONPath will capture the result from each page, e.g. $.results[*]

Finish Condition

Sets the condition that indicates that the last page was loaded. You can choose between the following options:

Total Value Reached will look at a numeric value within each response to determine if the last page is reached.

A Boolean (True) Value will look at a boolean (true/false) value with each response to determine whether the last page has been reached . The finish condition is reached if the selected value is true.

A Boolean (False) Value will look at a boolean (true/false) value with each response to determine whether the last page has been reached. The finish condition is reached if the selected value is false.

Empty Result will determine that the last page is reached if the response delivers an empty result.

Empty Target will determine that the last page is reached if the response delivers an empty target.

Finish Target

A JSONPath that points to the value that is used for the FinishCondition, e.g. $.total.

Limitations

The maximum amount of pages that can be loaded is defined by either the amount of items or value of maximum page size.
Below 10,000 items the max page size is 20 and above 10,000 items the max page size is 6

The total bytes of all cumulated pages cannot surpass the response limit of 15 MB

Basic Example

This example shows a simple offset based pagination setup.

Prepare a Data Source

Depending on the type of pagination, you need at least one variable provided by the associated data source. In this example, we use offset-based and provide the offset variable.

In our case, the data source renders the following sample JSON data:

{
  "total": 100,
  "result": [
    {
      "id": 1,
      "title": "title for 1"
    },
    {
      "id": 2,
      "title": "title for 2"
    }
  ]
}

Setting up the Paginated Data Source

Creating a new data source and using the type Paginated Data Source will have you select the data source you want to paginate over followed by the pagination type. In this case, we use Offset-based and select the previously created offset variable in the Offset Variable selection.

Based on the JSON result, the Result Selector will point to the list of items we are interested in.

$.result[*]

The Finish Condition in this case is managed by the total value returned by each page. Again, we use a JSONPath to select the Finish Target :

$.total

In the following screenshot, you can see the complete setup:

More Specific Examples