Backfilling

Backfill offers an unprecedentedly swift method to sift through the entire history of the Solana blockchain to pinpoint and retrieve specific transactions necessary for your application. Imagine possessing the functionality of command-line tools like cat and grep, but supercharged to operate across our robust infrastructure. By harnessing our system, you can execute these operations without the overhead of significant resource consumption on your end. Combing through transactional data has never been more efficient, saving time and computational expense, ensuring that you can focus on what matters most—analyzing the data, not struggling to acquire it.

API Calls

Create Job

post

Creates a backfilling task that retrieves transactions based on specified filters and subsequently submits them to a predefined URL in the hook's data field.

Query parameters

tokenstringOptionalExample: f469559d4cc55170a26031a4a5123eb2

Body

namestringRequired

The name of job.

filterobjectRequired

An array of filters to apply. Check the Transaction Filter for details.

batchinteger · min: 1 · max: 500Optional

Batch webhook calls. Specify the number of transactions to be batched in one call. Defaults to 1

cpu_limitinteger · min: 1 · max: 20Optional

Limit CPU for JOB, one CPU does ~20 000 transaction rescans. Speed can vary based on how many records are picked up and sent back. You can look at it as concurrency or threads, one CPU will always send one hook at the moment. Defaults to 1

Responses

post

HTTPcURLJavaScriptPython

POST /backfill/job/ HTTP/1.1
Host: api.solanarpc.dev
Content-Type: application/json
Accept: */*
Content-Length: 360

{
  "name": "scan-1",
  "hooks": {
    "data": "https://my.project.com/callback/"
  },
  "range": {
    "time": {
      "start": 1670110514,
      "end": 1670120614
    }
  },
  "filter": [
    {
      "name": "metaplex-CreateMetadataV2",
      "include": {
        "instructions": [
          {
            "address": "metaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s",
            "discriminator": "10",
            "discriminatorType": "u8"
          }
        ]
      }
    }
  ],
  "parser": {
    "enabled": [
      "data"
    ]
  },
  "cpu_limit": 1,
  "batch": 10
}

200

Successful response

{
  "id": 1
}

Get a job

get

Retrieves full job information

Path parameters

idintegerOptionalExample: 44

Query parameters

tokenstringOptionalExample: f469559d4cc55170a26031a4a5123eb2

Body

Responses

get

HTTPcURLJavaScriptPython

GET /backfill/job/{id} HTTP/1.1
Host: api.solanarpc.dev
Accept: */*

200

Successful response

{
  "id": "44",
  "name": "scan-1",
  "status": "COMPLETED",
  "batch": "1",
  "hooks": {
    "data": "https://my.project.com/callback",
    "status": null,
    "metrics": null
  },
  "cpu_limit": "1",
  "range": {
    "blocks": {
      "start": 164776968,
      "end": 164754296
    },
    "time": {
      "start": 1670110514,
      "end": 1670120614
    }
  },
  "filter": [
    {
      "include": {
        "instructions": [
          {
            "address": "metaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s",
            "discriminator": "10",
            "discriminatorType": "u8"
          }
        ]
      }
    }
  ],
  "metrics": {
    "cpu_time": 220,
    "run_time": 220,
    "txt_forward": 1313,
    "txt_per_sec": 4279,
    "txt_scanned": 941482,
    "callback_error_count": 0,
    "completed_percentage": 100,
    "callback_success_count": 137,
    "callback_bytes_transmit": 0
  }
}

List all jobs

get

Retrieves the list of backfilling jobs.

Query parameters

tokenstringOptionalExample: f469559d4cc55170a26031a4a5123eb2

Body

Responses

get

HTTPcURLJavaScriptPython

GET /backfill/job/ HTTP/1.1
Host: api.solanarpc.dev
Accept: */*

200

Successful response

[
  {
    "id": "44",
    "name": "scan-1",
    "status": "COMPLETED",
    "batch": "1",
    "hooks": {
      "data": "https://my.project.com/callback",
      "status": null,
      "metrics": null
    },
    "cpu_limit": "1",
    "range": {
      "blocks": {
        "start": 164776968,
        "end": 164754296
      },
      "time": {
        "start": 1670110514,
        "end": 1670120614
      }
    },
    "filter": [
      {
        "include": {
          "instructions": [
            {
              "address": "metaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s",
              "discriminator": "10",
              "discriminatorType": "u8"
            }
          ]
        }
      }
    ],
    "metrics": {
      "cpu_time": 220,
      "run_time": 220,
      "txt_forward": 1313,
      "txt_per_sec": 4279,
      "txt_scanned": 941482,
      "callback_error_count": 0,
      "completed_percentage": 100,
      "callback_success_count": 137,
      "callback_bytes_transmit": 0
    }
  }
]

Start job

patch

Starts a backfilling process. If the time range is big, it could take up to a few minutes to start.

Path parameters

idintegerOptionalExample: 44

Query parameters

tokenstringOptionalExample: f469559d4cc55170a26031a4a5123eb2

Body

Responses

patch

HTTPcURLJavaScriptPython

PATCH /backfill/job/{id}/start HTTP/1.1
Host: api.solanarpc.dev
Accept: */*

200

Successful response

{
  "message": "ok"
}

Stop job

patch

Stops backfilling process. After that job will have PAUSE status.

Path parameters

idintegerOptionalExample: 44

Query parameters

tokenstringOptionalExample: f469559d4cc55170a26031a4a5123eb2

Body

Responses

patch

HTTPcURLJavaScriptPython

PATCH /backfill/job/{id}/stop HTTP/1.1
Host: api.solanarpc.dev
Accept: */*

200

Successful response

{
  "message": "ok"
}

Update job

patch

Updates job params.

Path parameters

idintegerOptionalExample: 44

Query parameters

tokenstringOptionalExample: f469559d4cc55170a26031a4a5123eb2

Body

Responses

patch

HTTPcURLJavaScriptPython

PATCH /backfill/job/{id} HTTP/1.1
Host: api.solanarpc.dev
Accept: */*

200

Successful response

{
  "message": "ok"
}

Get a job metrics

get

Metrics can be received on demand via this call or by getting full job information. Also you can defined metrics web hook callback when creating job.

Path parameters

idintegerOptionalExample: 44

Query parameters

tokenstringOptionalExample: f469559d4cc55170a26031a4a5123eb2

Body

Responses

get

HTTPcURLJavaScriptPython

GET /backfill/job/{id}/metrics HTTP/1.1
Host: api.solanarpc.dev
Accept: */*

200

Successful response

{
  "cpu_time": 220,
  "run_time": 220,
  "txt_forward": 1313,
  "txt_per_sec": 4279,
  "txt_scanned": 941482,
  "callback_error_count": 0,
  "completed_percentage": 100,
  "callback_success_count": 137,
  "callback_bytes_transmit": 0
}

Delete job

delete

Delete backfilling job.

Path parameters

idintegerOptionalExample: 44

Query parameters

tokenstringOptionalExample: f469559d4cc55170a26031a4a5123eb2

Body

Responses

delete

HTTPcURLJavaScriptPython

DELETE /backfill/job/{id} HTTP/1.1
Host: api.solanarpc.dev
Accept: */*

200

Successful response

{
  "message": "ok"
}

Metrics

Our backfilling tasks are under constant surveillance, with metrics refreshing at one-second intervals. This ensures you have the most up-to-date insights into the performance and progress of your jobs. Developers have two options for accessing these metrics:

1. On-Demand Access: At any time during the backfilling process, you can request the latest metrics directly, allowing you to integrate this data into your monitoring systems on your schedule.

2. Webhook Integration: Set up a dedicated webhook URL when you create your job, and our system will push metrics regularly to your endpoint. This seamless integration is designed for developers looking to incorporate metrics into their existing infrastructure without the need for polling.

Whichever method you choose, you'll be equipped with the information you need to optimize your application's performance and monitor your backfilling processes effectively.

Available metrics:

Field	Description
txt_forward	Transactions that got sent to webhook
txt_scanned	Amount of transactions scanned
txt_per_sec	Scan speed
completed_percentage	Progress in percentage

Job Status Lifecycle

Throughout the execution of your backfilling job, you can expect the job status to transition through several states. To keep track of your job's current status, you have a couple of convenient options:

• Fetch the Latest Status: Query our API to retrieve the most recent status information for your job. This is perfect for those who prefer pulling data on demand.

• Webhook Notifications: Provide a webhook URL when setting up your job, and our system will proactively send status updates to your endpoint. This method suits those who prefer an automated, push-based approach to stay informed.

Understanding Status Transitions

The job status moves through the following stages during backfilling operations:

• CREATED: Your job is set up and in the queue, awaiting its start.

• STARTING: The job is initiated. This stage includes preparing the necessary resources and configurations.

• RUNNING: Your backfilling process is active. You can check its progress via real-time metrics.

• STOPPING: The job is winding down gracefully, saving its current state.

• PAUSED: The job is on hold. You can resume at a future time from this state.

• COMPLETED: The job has finished processing the specified range of blocks.

Stay in tune with your backfilling progress by opting for either of these methods for job status updates.

WEB Hooks payload

When using webhooks, your endpoint will receive a POST request containing a payload field in the JSON body. This payload will always be an array that holds transaction data, even if batch processing is disabled.

The transactions are represented in raw, unprocessed format, similar to what you would receive from an RPC node. For advanced parsing, we offer an internal parser capable of interpreting well-known Interface Definition Languages (IDLs). More details on the parsing feature are available in the Parsed Transaction section.

Example Payload:

{
  "payload": [
    { ...transaction },
    { ...transaction }
  ]
}