API Integration Patterns for Automation

Introduction

Every automation platform operates by integrating with external APIs. The reliability, performance, and maintainability of automations depend on how well these integrations handle the realities of production API communication: authentication expiration, rate limits, network failures, pagination, and data format mismatches.

This reference covers the core API integration patterns that apply across automation platforms. Code examples use JavaScript (the most common language in automation platforms), but the patterns are language-agnostic.

Webhooks vs Polling

The two fundamental methods for detecting new data in an external system are webhooks (push) and polling (pull). Each has distinct characteristics that affect automation design.

Webhooks (Push Model)

In the webhook model, the external system sends an HTTP request to the automation platform when an event occurs. The automation platform exposes a URL endpoint that receives the incoming request.

How it works:

1. The automation platform generates a unique webhook URL
2. The URL is registered with the external system (manually or via API)
3. When an event occurs, the external system sends an HTTP POST to the webhook URL
4. The automation platform receives the payload and triggers the workflow

Advantages:

• Near-real-time event delivery (typically under 1 second)
• No wasted API calls; events are delivered only when they occur
• Lower API quota consumption on both sides

Disadvantages:

• Requires the automation platform to be accessible from the internet
• Events can be lost if the automation platform is down when the webhook fires
• Not all APIs support webhooks
• Webhook payload format varies by provider; some send full data, others send only an ID requiring a follow-up API call
• Webhook URLs must be updated if the automation platform changes

Reliability considerations:

• Implement webhook verification to confirm the source (HMAC signatures, shared secrets)
• Return a 200 response quickly; process the payload asynchronously to avoid timeout
• Implement idempotency keys to handle duplicate deliveries
• Log all incoming webhooks for debugging

Polling (Pull Model)

In the polling model, the automation platform periodically queries the external API for new or changed records.

How it works:

1. The automation platform stores a cursor (timestamp, ID, or page token) representing the last processed record
2. On each poll interval, it queries the API for records newer than the cursor
3. New records trigger the workflow
4. The cursor is updated to the latest processed record

Advantages:

• Works with any API that supports listing records with filters
• No inbound network access required (the automation platform initiates all requests)
• Missed events can be caught on the next poll cycle
• Simpler to implement and debug

Disadvantages:

• Latency between event occurrence and detection (equal to the poll interval)
• Wastes API calls when no new data exists
• Poll frequency limited by API rate limits
• Can miss events if more records are created between polls than the page size

Polling interval recommendations:

Hybrid Approach

Many production automations use a hybrid approach: webhooks for real-time event detection with periodic polling as a reconciliation mechanism to catch any events the webhook missed.

Authentication Patterns

OAuth 2.0

OAuth 2.0 is the most common authentication standard for SaaS application APIs (as of January 2026). It allows users to grant an automation platform access to their account without sharing their password.

OAuth 2.0 Authorization Code Flow (most common for automation):

1. User clicks "Connect" in the automation platform
2. Platform redirects to the API provider's authorization page
3. User grants permission
4. Provider redirects back to the automation platform with an authorization code
5. Platform exchanges the code for an access token and refresh token
6. Access token is used for API requests
7. When the access token expires, the refresh token is used to obtain a new one

Key implementation considerations:

• Store refresh tokens securely (encrypted at rest)
• Implement automatic token refresh before expiration
• Handle the case where a refresh token is revoked (re-authentication required)
• Some providers rotate refresh tokens on each use (store the new one)
• Token lifetimes vary: Google access tokens expire after 1 hour; Salesforce access tokens expire after 2 hours; some providers issue non-expiring tokens

Common OAuth 2.0 scopes and their implications:

• Request minimum required scopes (principle of least privilege)
• Document which scopes are needed and why
• Some APIs require re-authorization when scope requirements change

API Key Authentication

Simpler than OAuth 2.0, API keys are static credentials included in request headers or query parameters.

Implementation:

// Header-based API key (preferred)
const response = await fetch('https://api.example.com/data', {
  headers: {
    'Authorization': 'Bearer sk-your-api-key-here',
    'Content-Type': 'application/json'
  }
});

// Query parameter API key (less secure, visible in logs)
const response = await fetch('https://api.example.com/data?api_key=sk-your-api-key-here');

Security considerations:

• Never include API keys in client-side code or version control
• Use environment variables or the automation platform's credential vault
• Rotate keys periodically (quarterly minimum for production integrations)
• Monitor key usage for anomalies

JWT (JSON Web Tokens)

Some APIs (especially Google Cloud services, Firebase, and custom APIs) use JWT-based service account authentication.

How it works:

1. A service account is created with a private key
2. The automation platform constructs a JWT, signs it with the private key
3. The signed JWT is exchanged for an access token
4. The access token is used for API requests

Implementation considerations:

• JWT construction requires a crypto library (available in n8n, Pipedream, and Windmill code nodes)
• Service account private keys must be stored with the highest level of protection
• JWTs have expiration times (typically 1 hour); implement automatic renewal
• Some platforms (n8n, Make) have built-in Google Service Account credential types that handle JWT construction automatically

HMAC Signature Authentication

Used primarily for webhook verification and some API providers (AWS Signature V4, Shopify webhooks).

Webhook verification example:

const crypto = require('crypto');

function verifyWebhookSignature(payload, signature, secret) {
  const computed = crypto
    .createHmac('sha256', secret)
    .update(payload, 'utf8')
    .digest('hex');
  return crypto.timingSafeEqual(
    Buffer.from(signature),
    Buffer.from(computed)
  );
}

Always use timing-safe comparison to prevent timing attacks.

Pagination Strategies

Most APIs return data in pages. Automation workflows must handle pagination to process complete datasets.

Offset-Based Pagination

The simplest pattern. The client specifies a page number or offset.

async function fetchAllRecords(baseUrl, headers) {
  let allRecords = [];
  let page = 1;
  let hasMore = true;

  while (hasMore) {
    const response = await fetch(
      `${baseUrl}?page=${page}&per_page=100`,
      { headers }
    );
    const data = await response.json();
    allRecords = allRecords.concat(data.results);
    hasMore = data.results.length === 100;
    page++;
  }

  return allRecords;
}

Limitation: If records are inserted or deleted during pagination, records can be skipped or duplicated. Not reliable for large, frequently changing datasets.

Cursor-Based Pagination

The API returns a cursor (opaque string) that points to the next page. More reliable than offset-based pagination because the cursor maintains position even when the dataset changes.

async function fetchAllWithCursor(baseUrl, headers) {
  let allRecords = [];
  let cursor = null;

  do {
    const url = cursor
      ? `${baseUrl}?cursor=${cursor}&limit=100`
      : `${baseUrl}?limit=100`;
    const response = await fetch(url, { headers });
    const data = await response.json();
    allRecords = allRecords.concat(data.results);
    cursor = data.next_cursor || null;
  } while (cursor);

  return allRecords;
}

Used by: Slack API, Notion API, Stripe API, HubSpot API (as of January 2026).

Link Header Pagination

The API returns pagination URLs in the HTTP Link header, following RFC 8288.

async function fetchAllWithLinkHeader(url, headers) {
  let allRecords = [];
  let nextUrl = url;

  while (nextUrl) {
    const response = await fetch(nextUrl, { headers });
    const data = await response.json();
    allRecords = allRecords.concat(data);

    const linkHeader = response.headers.get('Link');
    nextUrl = parseLinkHeader(linkHeader)?.next || null;
  }

  return allRecords;
}

function parseLinkHeader(header) {
  if (!header) return {};
  const links = {};
  header.split(',').forEach(part => {
    const match = part.match(/<([^>]+)>;\s*rel="([^"]+)"/);
    if (match) links[match[2]] = match[1];
  });
  return links;
}

Used by: GitHub API, GitLab API.

Pagination in Automation Platforms

Rate Limiting and Throttling

Understanding Rate Limits

APIs impose rate limits to prevent abuse and ensure fair resource distribution. Rate limits are expressed as requests per time window.

Common rate limit formats:

• X requests per second: Stripe allows 100 requests/second (as of January 2026)
• X requests per minute: GitHub allows 5,000 requests/hour for authenticated requests
• X requests per day: Some free-tier APIs limit to 1,000 requests/day
• Concurrent requests: Some APIs limit the number of simultaneous connections

Rate limit headers (most APIs include these):

X-RateLimit-Limit: 100
X-RateLimit-Remaining: 67
X-RateLimit-Reset: 1706140800
Retry-After: 30

Rate Limit Handling Pattern

async function fetchWithRateLimiting(url, headers, maxRetries = 3) {
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    const response = await fetch(url, { headers });

    if (response.status === 429) {
      const retryAfter = parseInt(response.headers.get('Retry-After') || '60');
      console.log(`Rate limited. Waiting ${retryAfter} seconds...`);
      await new Promise(resolve => setTimeout(resolve, retryAfter * 1000));
      continue;
    }

    if (!response.ok) {
      throw new Error(`HTTP ${response.status}: ${response.statusText}`);
    }

    return response.json();
  }

  throw new Error('Max retries exceeded due to rate limiting');
}

Proactive Throttling

Rather than waiting for a 429 response, proactively limit request rate:

class RateLimiter {
  constructor(requestsPerSecond) {
    this.minInterval = 1000 / requestsPerSecond;
    this.lastRequest = 0;
  }

  async wait() {
    const now = Date.now();
    const elapsed = now - this.lastRequest;
    if (elapsed < this.minInterval) {
      await new Promise(resolve =>
        setTimeout(resolve, this.minInterval - elapsed)
      );
    }
    this.lastRequest = Date.now();
  }
}

// Usage: 10 requests per second
const limiter = new RateLimiter(10);

for (const item of items) {
  await limiter.wait();
  await processItem(item);
}

Error Handling and Retry Logic

Error Categories

Exponential Backoff with Jitter

async function fetchWithRetry(url, options, maxRetries = 5) {
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      const response = await fetch(url, {
        ...options,
        signal: AbortSignal.timeout(30000)
      });

      if (response.status === 429 || response.status >= 500) {
        if (attempt === maxRetries) {
          throw new Error(`Failed after ${maxRetries} retries: HTTP ${response.status}`);
        }
        const baseDelay = Math.min(1000 * Math.pow(2, attempt), 30000);
        const jitter = Math.random() * baseDelay * 0.5;
        const delay = baseDelay + jitter;
        console.log(`Attempt ${attempt + 1} failed (HTTP ${response.status}). Retrying in ${Math.round(delay)}ms`);
        await new Promise(resolve => setTimeout(resolve, delay));
        continue;
      }

      if (!response.ok) {
        throw new Error(`HTTP ${response.status}: ${await response.text()}`);
      }

      return await response.json();
    } catch (error) {
      if (error.name === 'AbortError' && attempt < maxRetries) {
        console.log(`Request timed out. Retrying (attempt ${attempt + 2})...`);
        continue;
      }
      if (attempt === maxRetries) throw error;
    }
  }
}

The jitter prevents the "thundering herd" problem where multiple retrying clients all hit the API at the same time after a backoff period.

Dead Letter Pattern

For batch processing, isolate failures so that one bad record does not stop the entire batch:

async function processBatchWithDeadLetter(records, processFunc) {
  const results = { success: [], failed: [] };

  for (const record of records) {
    try {
      const result = await processFunc(record);
      results.success.push({ record, result });
    } catch (error) {
      results.failed.push({
        record,
        error: error.message,
        timestamp: new Date().toISOString()
      });
    }
  }

  if (results.failed.length > 0) {
    console.log(`${results.failed.length} records failed. Writing to dead letter queue.`);
    // Store failed records for later review and reprocessing
  }

  return results;
}

Data Transformation Patterns

Field Mapping

The most common transformation: renaming fields between source and target schemas.

function mapFields(source, fieldMap) {
  const target = {};
  for (const [sourceField, targetField] of Object.entries(fieldMap)) {
    const value = sourceField.split('.').reduce(
      (obj, key) => obj?.[key], source
    );
    if (value !== undefined) {
      target[targetField] = value;
    }
  }
  return target;
}

// Usage
const mapped = mapFields(apiResponse, {
  'contact.first_name': 'firstName',
  'contact.last_name': 'lastName',
  'contact.email_addresses[0].value': 'email',
  'company.name': 'companyName'
});

Data Type Coercion

APIs return data in inconsistent types. A date might be a Unix timestamp, ISO string, or a custom format.

function normalizeDate(value) {
  if (!value) return null;
  if (typeof value === 'number') {
    // Unix timestamp (seconds or milliseconds)
    return value > 1e12
      ? new Date(value).toISOString()
      : new Date(value * 1000).toISOString();
  }
  const parsed = new Date(value);
  return isNaN(parsed.getTime()) ? null : parsed.toISOString();
}

function normalizeBoolean(value) {
  if (typeof value === 'boolean') return value;
  if (typeof value === 'string') {
    return ['true', '1', 'yes', 'on'].includes(value.toLowerCase());
  }
  if (typeof value === 'number') return value !== 0;
  return false;
}

Array Flattening and Nesting

APIs may return nested arrays that need to be flattened for a target system, or flat records that need to be grouped.

// Flatten: order with line items -> individual line item records
function flattenOrderItems(order) {
  return order.line_items.map(item => ({
    order_id: order.id,
    order_date: order.created_at,
    customer_email: order.customer.email,
    product_name: item.name,
    quantity: item.quantity,
    unit_price: item.price
  }));
}

// Group: flat records -> nested structure
function groupByKey(records, keyField) {
  return records.reduce((groups, record) => {
    const key = record[keyField];
    if (!groups[key]) groups[key] = [];
    groups[key].push(record);
    return groups;
  }, {});
}

Multi-Step Orchestration

Sequential vs Parallel Execution

When a workflow involves multiple API calls, determine whether they can run in parallel or must run sequentially.

Sequential (dependencies between steps):

// Step 2 depends on Step 1's result
const customer = await createCustomer(data);
const subscription = await createSubscription(customer.id, plan);
const invoice = await createInvoice(subscription.id);

Parallel (independent steps):

// These three calls are independent
const [customer, products, settings] = await Promise.all([
  fetchCustomer(customerId),
  fetchProducts(categoryId),
  fetchSettings(accountId)
]);

Parallel with concurrency limit:

async function parallelWithLimit(items, asyncFunc, concurrency = 5) {
  const results = [];
  const executing = new Set();

  for (const item of items) {
    const promise = asyncFunc(item).then(result => {
      executing.delete(promise);
      return result;
    });
    executing.add(promise);
    results.push(promise);

    if (executing.size >= concurrency) {
      await Promise.race(executing);
    }
  }

  return Promise.all(results);
}

Credential Security

Storage Best Practices

• Store credentials in the automation platform's built-in credential vault, not in workflow variables or environment variables that may be logged
• Use separate credentials for development/staging and production
• Rotate API keys on a quarterly schedule at minimum
• Use OAuth 2.0 with limited scopes instead of long-lived API keys when the option exists
• Audit credential access: who can view and use each credential?

Credential Rotation Workflow

Automate credential rotation where possible:

1. Generate new API key via the provider's API
2. Update the credential in the automation platform
3. Verify automations work with the new credential
4. Revoke the old API key
5. Log the rotation event

Testing and Debugging

Testing Strategies

Debugging Checklist

When an API integration fails:

1. Check the HTTP status code and response body
2. Verify credentials are valid (test with a simple API call)
3. Check rate limit headers (is the limit exceeded?)
4. Verify the request URL, headers, and body match the API documentation
5. Check for API version changes (many APIs deprecate endpoints)
6. Test the same request in a standalone HTTP client (Postman, curl) to isolate whether the issue is the API or the automation platform

Common Anti-Patterns

1. Not Handling Pagination

Fetching only the first page of results and assuming that is all the data. Always implement pagination for list endpoints.

2. Ignoring Rate Limits Until They Fail

Build rate limit handling from the start, not after the first 429 error in production.

3. Storing API Keys in Workflow Variables

Credentials stored in workflow variables may appear in execution logs, be visible to other users, or be included in workflow exports. Use the credential vault.

4. Tight Coupling to API Response Structure

If a workflow breaks when the API adds a new field to its response, the workflow is too tightly coupled. Access only the fields needed and handle missing fields gracefully.

5. No Error Handling on External Calls

Every HTTP request can fail. Wrapping external API calls without try/catch or error routes means a single network timeout takes down the entire workflow.

6. Synchronous Processing of Large Batches

Processing 10,000 records synchronously in a single workflow execution may time out or overwhelm the target API. Split large batches into smaller chunks and process them with appropriate delays.

7. Webhook Endpoints Without Verification

Accepting webhook payloads without verifying the sender's signature allows anyone who discovers the URL to inject fake events into the automation.

8. Hard-Coded URLs and IDs

Base URLs, resource IDs, and environment-specific values should be stored in variables or configuration, not hard-coded in workflow nodes. This allows the same workflow to work across development, staging, and production environments.

Summary

Reliable API integrations in automation require deliberate handling of authentication, pagination, rate limits, errors, and data transformation. The patterns described in this reference apply across all automation platforms. The specific implementation varies (visual configuration in Make, code nodes in n8n, full scripts in Windmill), but the underlying principles remain consistent: verify authentication, paginate all list requests, respect rate limits proactively, retry transient failures with backoff, and isolate failures in batch processing.