Using the Bulk API

Optimizing record operations for a source when dealing with large volumes.

Suggest Edits

The Bulk API allows you to send multiple operations to the system at once using a Bulk Request. You should use this API when dealing with a large volume of resources, to optimize performance and avoid system congestion.

The Bulk API accepts a Bulk Request made out of multiple Bulk Operations (Create, Upsert, Delete) and validates, synchronously, that the request is well formed and that its operations content conforms to the source's resource and relationship profiles. A Bulk Response tells the user if the request has been accepted or not, and contains details on the errors that caused a rejection. If a request is rejected, all its operations are canceled.

Once a Bulk Request is accepted, the system starts a long-running Bulk Task to process the Bulk Operations asynchronously. Users can check the status of a Bulk Task using a dedicated endpoint and a task ID, returned in the Bulk Response. A Bulk Task Response will contain a status for every single operation and their corresponding errors if applicable.

Each operation must target a single entity (resource or relationship) and is processed atomically by the system. A failed operation will not stop a Bulk Task from processing other operations from the same Bulk Request.

Sending a Bulk Request

A Bulk Request contains a list of operations. A request cannot have operations where the targets are both resources and relationships. All operations must either be for resources or relationships. This is made clear using the API since there are two distinct /bulk endpoints, depending on what type of operations you wish to make.

{
    "operations": [
        {
            "action": "CREATE",
            "create": {
                // Type, ID, Data, Contained and Meta
            }
        },
        {
            "action": "UPSERT",
            "upsert": {
                // ...
            }
        },
        {
            "action": "DELETE",
            "delete": {
                // ...
            }
        }
    ]
}

There are 3 supported types of Bulk Operations:

  1. "CREATE"
  2. "UPSERT"
  3. "DELETE"

📘

Operations are Sequential

Bulk operations are processed in the order they are provided.

📥
Use the Bulk Resource API in a Configured Data Source
Open Recipe

Create Resource Operation

The Create operation creates a new resource and uses the create key to specify its payload.

  • id: ID of the targeted resource.
  • type: Type of the resource you want to create.
  • data: Properties of the resource.
  • contained: Contained resources and their properties.

*Note: The properties under data and contained are dictated by the profile targeted by the resourceType. An error will be thrown if a property that is not contained in the profile is specified, or if the values do not respect the property type.

{
    "action": "CREATE",
    "create": {
        "id": "<resourceId>", // Optional. Will be system generated if not provided
        "type": "<resourceType>",
        "data": {
            // Resource properties
        },
        "contained": {
            // Containes Resources if applicable
        }
    }
}

Given the Simple Profile created before, here is an example of a Bulk payload that could be sent to create a provider:

{
    "operations": [
        {
            "action": "CREATE",
            "create": {
                "type": "provider",
                "data": {
                    "name": "John Doe",
                    "practiceNumber": 1020993,
                    "specialities": [
                        "Pharmacist"
                    ]
                }
            }
        }
    ]
}

Upsert Operation

The Upsert operation updates all properties of an existing resource (or creates it if it doesn't exist) and it uses the upsert key to specify its payload.

  • id: ID of the targeted resource.
  • type: Type of the targeted resource.
  • data: Properties of the resource.
  • contained: Contained resources and their properties
{
    "action": "UPSERT",
    "upsert": {
        "id": "<resourceId>",
        "type": "<resourceType>",
        "data": {
            // ...
        }
    }
}

Given the Simple Profile created before, here is an example of a Bulk payload that could be sent to upsert a provider:

{
    "operations": [
        {
            "action": "UPSERT",
            "upsert": {
                "id": "2eVocVVgjAWYJ74DifXluwVtSYs",
                "type": "provider",
                "data": {
                    "name": "John Doe",
                    "practiceNumber": 1020993,
                    "specialities": [
                        "Pharmacist"
                    ]
                }
            }
        }
    ]
}

Delete Operation

The Delete operation deletes an existing resource and uses the delete key to specify its payload.

  • id: ID of the targeted resource
  • type: Type of the targeted resource.
{
    "action": "DELETE",
    "delete": {
        "type": "<resourceType>",
        "id": "<resourceId>"
    }
}

Given the Simple Profile created before, here is an example of a Bulk payload that could be sent to upsert a provider:

{
    "operations": [
        {
            "action": "DELETE",
            "delete": {
              	"id": "2eVocVVgjAWYJ74DifXluwVtSYs",
                "type": "provider"
            }
        }
    ]
}

Bulk Response

After sending Bulk Request, you will receive a Bulk Response. This response indicates if the request was accepted by the system, or rejected.

If a request is ACCEPTED, a Bulk Task ID will be returned with the response.

{
    "status": "ACCEPTED",
    "bulkId": "<bulkTaskId>"
}

If a request is REJECTED, all operations will been cancelled and the response will contain receipts for all operations of the Bulk Request.

A receipt for a Bulk Request Operation contains:

  • status: Operation status. Either FAILURE or CANCELLED.
  • error: If status is FAILURE, the error that caused the failure.
{
    "status": "REJECTED",
    "receipts": [
        {
            "status": "CANCELLED"
        },
        {
            "status": "FAILURE",
            "error": {
                "code": 400,
                "message": "Human readable error message",
                "type": "<messageType>"
            }
        }
    ]
}

Bulk Task

A Bulk Task can have 4 status:

  • PENDING: Bulk Task is waiting to be picked up or actively being processed by the system.
  • CANCELLED: Bulk Task has been cancelled.
  • FAILURE: Bulk Task has failed.
  • SUCCESS: Bulk Task has finished being processed successfully.

Checking the status of a Bulk Task also get information about individual Bulk Operations as a list of receipts. A receipt for a Bulk Task Operation contains:

  • id: ID of the resource.
    • If the operation is CREATE, this is the ID of the newly created resource.
    • If the operation is UPSERT, this is the ID of the updated or newly created resource.
  • type: Type of the targeted resource.
  • status: Operation status. Either SUCCESS, FAILURE, PENDING, CANCELLED
  • error: If status is FAILURE, the error that caused the failure.
{
    "receipts": [
        {
            "id": "2dMyvjBs8mDQ7VJxjFPtWnFvaxp",
            "status": "SUCCESS",
            "type": "provider"
        },
        {
            "error": {
                "code": 409,
                "message": "record with id '2dMo6zb0yWtzHvt8MhmbI2ddQbR' already exists",
                "type": "ALREADY_EXISTS"
            },
            "id": "2dMo6zb0yWtzHvt8MhmbI2ddQbR",
            "status": "FAILURE",
            "type": "provider"
        }
    ],
    "status": "SUCCESS"
}

For more details on error codes and types, see: Error Codes and Types

📨
Retrieve the bulk operation status
Open Recipe

Updated about 1 month ago