By including a webhook_url when calling Create job, you can receive updates on the status of the job. The webhook URL specified will be called with a POST request, containing a JSON body with the same data as the Get job endpoint returns.

Typically, two webhooks will be sent per job. Once when the job moves from ‘starting’ to ‘running’ soon after it’s created, and again once it has finished with either a ‘succeeded’ or ‘failed’ status.

Webhook payload
{
  "id": "7YEGS3M8Q2JD6TNMEJB8B6EKVS",
  "urls": ["https://example.com"],
  "createdAt": 1699964378397,
  "status": "running",
  "sitemapPageCount": 12,
  "progress": {
    "scraped": 6,
    "discarded": 0,
    "failed": 0
  },
  "costCents": 0,
  "webhookFails": []
}
AuthenticationScrape