# LocalPipe API Documentation

## Authentication

All API requests require an API key passed in the `x-api-key` header.

```bash
curl -X POST https://uunrlffonnvmbsstkpze.supabase.co/functions/v1/enrich \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{"business_name": "Acme Corp", "business_website": "acme.com", "business_location": "Chicago, IL"}'
```

---

# ENDPOINT 1: Business Enrichment

## POST /enrich

Enriches a single business with owner/decision-maker contact information. All requests are **asynchronous** — the API returns a `202 Accepted` response immediately with a `job_id`. Results are delivered via webhook (`callback_url`) or can be polled.

**High-Volume Usage:** You can send thousands of requests simultaneously with a `callback_url` — every request is accepted immediately and queued. Results are delivered to your webhook as each enrichment completes. This is the recommended approach for **Clay** and other automation platforms.

### Request Fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| business_name | string | Yes | The name of the business |
| business_website | string | No | The business website (with or without https://). If omitted, the system uses Google Search with the business name and location to find the owner. |
| business_location | string | No | City/state of the business (e.g. "Chicago, IL"). Improves accuracy for multi-location businesses. |
| want_email | boolean | No | Whether to find email (default: true, costs 2 credit) |
| want_phone | boolean | No | Whether to find phone (default: false, costs 5 credits) |
| want_general_email | boolean | No | Whether to find general business email (default: false). General emails are info@, contact@, hello@, etc. |
| general_email_fallback_only | boolean | No | Only find general business email if owner email is not found (default: false). Requires want_general_email to be true. |
| decision_maker_title | string | No | Target a specific role instead of the business owner (e.g. "Marketing Manager", "Practice Manager"). **If omitted, the system defaults to finding the business owner.** |
| owner_name | string | No | Supply the decision-maker's name yourself (e.g. "Jane Doe", at least 2 words). When provided, name discovery is skipped and contact sourcing uses this name directly. **Not billed** as a name-found credit and **never written to cache** (since it's unverified). Fallback business email logic still applies. |
| clay_row_id | string | No | A passthrough identifier returned unchanged in both the API response and webhook payload. Use this to match results back to rows in Clay or other automation platforms (e.g. `"{{_row_id}}"`). |
| callback_url | string | No | URL to receive results when ready (recommended for Clay and automation platforms). If omitted, poll for results using the job_id. |

```javascript
const response = await fetch("https://uunrlffonnvmbsstkpze.supabase.co/functions/v1/enrich", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "x-api-key": "YOUR_API_KEY"
  },
  body: JSON.stringify({
    business_name: "Acme Corporation",
    business_website: "acme.com",
    business_location: "Chicago, IL",
    want_email: true,
    want_phone: false,
    want_general_email: true,
    general_email_fallback_only: false,
    decision_maker_title: "Marketing Manager",
    owner_name: "Jane Doe", // optional — skip name discovery and use this name
    clay_row_id: "row_abc123",
    callback_url: "https://your-webhook-url.com/callback"
  })
});
```

### Response (202 Accepted)

```json
{
  "success": true,
  "job_id": "abc-123",
  "total_records": 1,
  "message": "Request accepted. Record is queued. Results will be POSTed to your callback_url when ready.",
  "estimated_seconds": 30
}
```

### Response Field Reference

> ⚠️ **Field naming — read this before parsing responses.** The exact field names returned in webhook payloads, polling responses, and CSV exports are listed below. **Use these names verbatim in your parser.** A common mistake is looking for `general_email` (does not exist) — the general business email is always returned as `business_email`.

Every record object in `records[]` (webhook payload, polling response, and CSV row) contains exactly these fields:

| Field name (exact) | Type | Description | When populated |
|--------------------|------|-------------|----------------|
| `business_name` | string | Business name (echoed from your request) | Always |
| `business_website` | string | Website (echoed from your request) | Always |
| `business_location` | string | Location (echoed from your request) | Always |
| `owner_name` | string | Decision-maker full name, or `"not found"` | When `want_name=true` (default) |
| `owner_first_name` | string | First name split from `owner_name`, or `"not found"` | Same as `owner_name` |
| `owner_last_name` | string | Last name split from `owner_name`, or `"not found"` | Same as `owner_name` |
| `owner_email` | string | Verified personal email of the decision-maker, or `"not found"` | When `want_email=true` (default) |
| `owner_phone` | string | Decision-maker phone, or `"not found"` | When `want_phone=true` |
| `business_email` | string | **General business email** (`info@`, `contact@`, `hello@`, etc.), or `"not found"` | When `want_general_email=true` |
| `email_provider` | string \| null | Email provider classification for `owner_email` (e.g. `"google"`, `"microsoft"`) | When `owner_email` is found |
| `decision_maker_title` | string \| null | The title that was searched for this row | Multi-title jobs only |
| `owner_role` | string \| null | Role label associated with the decision-maker | When discovered |
| `extra_columns` | object \| null | Passthrough columns from your original request | When provided |

**Negative results are returned literally as the string `"not found"`** — never `null`, never an empty string, never missing. Your parser should treat `"not found"` as "no result for this field".

> **Field name pitfalls — DO NOT use these names, they don't exist:**
> - ❌ `general_email` → ✅ use `business_email`
> - ❌ `generic_email` / `info_email` → ✅ use `business_email`
> - ❌ `personal_email` → ✅ use `owner_email`
> - ❌ `first_name` / `last_name` → ✅ use `owner_first_name` / `owner_last_name`
> - ❌ `phone` / `mobile` → ✅ use `owner_phone`
> - ❌ `name` → ✅ use `owner_name` (and `business_name` for the company)

The request flag is `want_general_email`, but the **returned field is always `business_email`** — the names intentionally differ because the request asks "do you want me to look for it?" and the response key reflects what the value actually is (a generic business mailbox, not the owner's personal email).

### Enrichment Webhook Payload

```json
{
  "job_id": "abc-123",
  "status": "completed",
  "completed_at": "2025-01-01T12:00:00Z",
  "clay_row_id": "row_abc123",
  "stats": {
    "total_records": 1,
    "owner_names_found": 1,
    "owner_emails_found": 1,
    "owner_phones_found": 0,
    "general_emails_found": 1
  },
  "records": [
    {
      "business_name": "Acme Corp",
      "business_website": "acme.com",
      "business_location": "Chicago, IL",
      "owner_name": "John Smith",
      "owner_first_name": "John",
      "owner_last_name": "Smith",
      "owner_email": "john@acme.com",
      "owner_phone": "not found",
      "business_email": "info@acme.com",
      "email_provider": "google",
      "decision_maker_title": null,
      "owner_role": "Owner",
      "extra_columns": null
    }
  ]
}
```

> **Stats field naming:** the count of general business emails in the `stats` object is `general_emails_found` (matches the `want_general_email` request flag), while the per-record field is `business_email`. This is the only place in the response where "general" appears.

> **Email Verification Notice:** Owner emails (`owner_email`) returned by the API are always verified. **General business emails (`business_email`) are returned UNVERIFIED on credit-based plans** — they are the highest-ranked candidates discovered for the domain (e.g. `info@`, `contact@`, `hello@`). You should verify these emails through your own provider (e.g. MillionVerifier, NeverBounce, ZeroBounce) before sending. Usage-based billing accounts receive pre-verified business emails.

### Parser checklist

Before shipping any merge/import script, confirm:
1. You read `business_email` (not `general_email`) for the generic mailbox.
2. You treat the literal string `"not found"` as an empty value — don't write it into your CRM.
3. You match webhook deliveries back to your rows using `clay_row_id` (the only field guaranteed to round-trip from your request).
4. You handle `owner_email` and `business_email` as separate columns — they are never merged on the API side.

### Enrichment Polling

```bash
curl "https://uunrlffonnvmbsstkpze.supabase.co/functions/v1/get-job-results?job_id=abc-123" \
  -H "apikey: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZSIsInJlZiI6InV1bnJsZmZvbm52bWJzc3RrcHplIiwicm9sZSI6ImFub24iLCJpYXQiOjE3NzA0OTIwMzMsImV4cCI6MjA4NjA2ODAzM30.8Ic_U17YrZtsDVnYnIP0XJzrhZJOLDSlJ0AZfTS31FQ"
```

---

# ENDPOINT 2: Google Maps Business Search

## POST /maps-search

Search Google Maps for businesses by keyword and location. Returns business names, phone numbers, addresses, ratings, websites, and more. All requests are **asynchronous** — returns a `task_id` immediately.

> ### ⚡ BATCH IN ONE REQUEST — NEVER LOOP CITIES OR KEYWORDS
>
> **`query` and `location` both accept arrays.** A single `/maps-search` call fans out across **up to 5 keywords × 25 locations = 125 parallel searches** server-side, all returned under **one `task_id`**.
>
> **This is the only correct way to run bulk scrapes.** Submitting one request per city or per keyword is the #1 cause of incomplete data, rate-limit exhaustion, and dropped cities.
>
> - ✅ **Correct (one request):** `{ "query": ["plumber","hvac contractor","electrician","roofer","painter"], "location": ["Austin, TX","Dallas, TX","Houston, TX","San Antonio, TX","Fort Worth, TX","El Paso, TX","Arlington, TX","Corpus Christi, TX","Plano, TX","Lubbock, TX"], "limit": 300 }` — 5 keywords × 10 locations = 50 parallel searches, one task_id, one poll loop, up to 15,000 leads.
> - ❌ **Wrong (what AI agents keep doing):** a `for city in cities; do curl .../maps-search -d '{"query":"plumber","location":"'$city'","limit":N}'; done` loop. This serializes work, multiplies your rate-limit consumption, fragments results across N task_ids you then have to poll separately, and almost always drops cities on shell timeout.
> - ❌ **Also wrong:** sending one request per keyword for the same location list. Pass all keywords as the `query` array in the same call.
> - If you genuinely need >5 keywords or >25 locations, split into the **minimum** number of batched requests (e.g. 60 cities → 3 requests of 20, not 60 requests of 1) and submit them through the `drain_queue` pattern in the Agent Mode section — never one-by-one from tool calls.

**Exact Category Matching:** Your `query` is matched against Google Maps' official business category taxonomy (e.g. "Plumber", "Dentist", "HVAC Contractor"). When a match is found, results are filtered to only include businesses with that exact category — eliminating irrelevant results. Each result includes a `matched_category` field showing which category it matched. If no exact category match is found, results are returned based on keyword relevance.

### Request Fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| query | string or string[] | Yes | Business type/keyword(s). **Pass as an array to batch up to 5 keywords in ONE request** instead of looping — e.g. `["plumbers","hvac contractor","electrician"]`. Matched against Google Maps category taxonomy for exact filtering. |
| location | string or string[] | Yes | City names, state names, country names, US ZIP codes, or UK postcodes. **Pass as an array to batch up to 25 locations in ONE request** instead of looping — e.g. `["Austin, TX","Dallas, TX","Houston, TX"]`. Do NOT submit one request per city. |
| limit | integer | No | Maximum results to return **per (query × location) pair** in this request. Default: 100, max: 10,000. ⚠️ **This is a per-pair cap, not a total cap.** If you pass 5 queries × 10 locations with `limit: 400`, the job can return up to 5 × 10 × 400 = **20,000 leads**, not 400. To cap the *total* output, either (a) keep `query` and `location` as single values and set `limit` to your desired total, or (b) divide your desired total by `queries.length × locations.length` and pass that as `limit`. |
| filters | object | No | Filter results (see Filter Options below) |
| callback_url | string | No | Webhook URL for result delivery when search completes |

### Filter Options

| Filter | Type | Description |
|--------|------|-------------|
| min_rating | number | Minimum Google rating (1-5) |
| max_rating | number | Maximum Google rating (1-5) |
| min_reviews | number | Minimum review count |
| max_reviews | number | Maximum review count |
| has_website | boolean | Only return businesses with (true) or without (false) a website |
| has_phone | boolean | Only return businesses with (true) or without (false) a phone number |
| price_level | string[] | Google price levels: ["$", "$$", "$$$", "$$$$"] |
| opens_before | number | Business opens before this hour (0-23) |
| closes_after | number | Business closes after this hour (0-23) |
| radius_km | number | Search radius in kilometers around each ZIP/postcode location (5-80, default 25). Only applies to ZIP/postcode locations — ignored for city/state/country names. |

### Example Request

```javascript
const response = await fetch("https://uunrlffonnvmbsstkpze.supabase.co/functions/v1/maps-search", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "x-api-key": "YOUR_API_KEY"
  },
  body: JSON.stringify({
    query: "plumbers",
    location: ["Austin, TX", "San Antonio, TX"],
    limit: 500,
    filters: {
      min_rating: 4.0,
      has_website: true,
      has_phone: true
    },
    callback_url: "https://your-webhook-url.com/maps-callback"
  })
});
```

### Multi-Keyword + Multi-Location Batch Example (Recommended)

This is the API's core strength: pass **all your keywords** and **all your cities** in a single call. Server-side fan-out runs every (keyword × location) pair in parallel and returns everything under one `task_id`.

```javascript
const response = await fetch("https://uunrlffonnvmbsstkpze.supabase.co/functions/v1/maps-search", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "x-api-key": "YOUR_API_KEY"
  },
  body: JSON.stringify({
    query: ["plumbers", "hvac contractors", "electricians", "roofers", "painters"],  // up to 5 keywords
    location: [
      "Austin, TX", "Dallas, TX", "Houston, TX", "San Antonio, TX", "Fort Worth, TX",
      "El Paso, TX", "Arlington, TX", "Corpus Christi, TX", "Plano, TX", "Lubbock, TX"
    ],  // up to 25 locations
    limit: 300,  // per (keyword × location) pair = up to 5 × 10 × 300 = 15,000 total leads
    filters: {
      has_website: true,
      min_rating: 3.5
    },
    callback_url: "https://your-webhook-url.com/maps-callback"
  })
});
```

**What this single call does:**
- Fans out to **5 keywords × 10 locations = 50 parallel searches** server-side
- Each search returns up to **300 leads**
- Total potential output: **15,000 businesses** from one request, one `task_id`, one poll loop
- Consumes **exactly 1 rate-limit slot** on submission (not 50)

> ⚠️ **If you need more than 5 keywords or 25 locations:** split into the **minimum** number of batched requests. 60 cities → 3 requests of 20 locations each (with all 5 keywords in each). Never send one request per city or per keyword.

### Response (202 Accepted)

```json
{
  "success": true,
  "task_id": "uuid-here",
  "status": "pending",
  "query": ["plumbers"],
  "locations": [
    { "input": "Austin, TX", "resolved": "Austin, US" },
    { "input": "San Antonio, TX", "resolved": "San Antonio, US" }
  ],
  "limit": 500,
  "message": "Search queued. Poll GET /get-scrape-results?task_id=uuid-here for results."
}
```

## Retrieving Maps Search Results

### Option 1: Webhook (Recommended)

Include a `callback_url` in your request. When the search completes, results are POSTed to your URL:

```json
{
  "task_id": "uuid-here",
  "status": "success",
  "total_results": 476,
  "query": ["plumbers"],
  "locations": ["Austin, TX", "San Antonio, TX"],
  "data": [
    {
      "name": "Austin Plumbing Co",
      "phone": "+15125551234",
      "full_address": "123 Main St, Austin, TX 78701",
      "city": "Austin",
      "rating": 4.8,
      "review_count": 342,
      "website": "https://austinplumbing.com",
      "types": ["plumber", "contractor"],
      "matched_category": "Plumber",
      "price_level": "$$",
      "place_id": "ChIJ...",
      "business_id": "0x...",
      "latitude": 30.2672,
      "longitude": -97.7431,
      "is_claimed": true,
      "working_hours": { "Monday": "8AM-6PM", "Tuesday": "8AM-6PM" }
    }
  ]
}
```

### Option 2: Polling (with Pagination)

Poll for results using the `task_id`. Large result sets are paginated to stay within the 6 MB response limit.

```bash
# Check progress (or fetch first page if complete)
curl "https://uunrlffonnvmbsstkpze.supabase.co/functions/v1/get-scrape-results?task_id=uuid-here" \
  -H "apikey: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZSIsInJlZiI6InV1bnJsZmZvbm52bWJzc3RrcHplIiwicm9sZSI6ImFub24iLCJpYXQiOjE3NzA0OTIwMzMsImV4cCI6MjA4NjA2ODAzM30.8Ic_U17YrZtsDVnYnIP0XJzrhZJOLDSlJ0AZfTS31FQ"

# Fetch a specific page
curl "https://uunrlffonnvmbsstkpze.supabase.co/functions/v1/get-scrape-results?task_id=uuid-here&limit=5000&offset=5000" \
  -H "apikey: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZSIsInJlZiI6InV1bnJsZmZvbm52bWJzc3RrcHplIiwicm9sZSI6ImFub24iLCJpYXQiOjE3NzA0OTIwMzMsImV4cCI6MjA4NjA2ODAzM30.8Ic_U17YrZtsDVnYnIP0XJzrhZJOLDSlJ0AZfTS31FQ"
```

### Query Parameters

| Param | Type | Default | Description |
|-------|------|---------|-------------|
| task_id | string | — | **Required.** The task ID returned by /maps-search. |
| limit | integer | 5000 | Page size. Max **5000**. |
| offset | integer | 0 | Number of leads to skip. |
| relevance_filter | string | off | Filter out off-topic results: `off` (default), `moderate`, or `strict`. Uses keyword stems against business types and names — useful when scraping broad terms like "consulting" that pull in unrelated categories. |

**Processing response:**
```json
{
  "task_id": "uuid-here",
  "status": "processing",
  "progress": {
    "completed_work_units": 15,
    "total_work_units": 45,
    "leads_found": 230,
    "target_limit": 500
  }
}
```

**Completed response (paginated):**
```json
{
  "task_id": "uuid-here",
  "status": "success",
  "total_results": 12450,
  "query": ["plumbers"],
  "locations": ["Texas"],
  "pagination": {
    "limit": 5000,
    "offset": 0,
    "next_offset": 5000,
    "has_more": true,
    "returned": 5000,
    "warning": "MORE LEADS AVAILABLE. Do NOT submit /enrich yet — fetch all pages first by re-calling this endpoint with offset=5000 until has_more=false."
  },
  "enrichment_contract": {
    "total_leads_to_enrich": 12450,
    "required_enrich_submissions": 12450,
    "instruction": "This scrape returned 12450 leads. You MUST submit exactly 12450 POST /enrich requests (one per lead) to enrich them all. Submitting fewer is a data loss bug — assert your enrich submission count == 12450 before declaring the workflow complete."
  },
  "data": [ ... ]
}
```

> **🚨 AI AGENTS / AUTOMATIONS — READ THIS.** The `enrichment_contract.required_enrich_submissions` field is the **ground-truth count** of `/enrich` calls you must submit. We see automations regularly scrape 5,000–20,000 leads then submit only 400–1,000 enrich requests because they processed one page and stopped. That is a silent data-loss bug and the user is paying for leads they never receive. Before declaring the workflow done: (1) paginate `/get-scrape-results` until `has_more=false`, accumulating every lead; (2) count what you accumulated and assert it equals `total_results`; (3) submit one `/enrich` per lead and assert your submission count equals `enrichment_contract.required_enrich_submissions`. If you exit early because of a shell timeout, resume from the last `offset` — do not skip leads.

**Pagination loop example:**
```javascript
let offset = 0;
const allLeads = [];
while (true) {
  const res = await fetch(
    `https://uunrlffonnvmbsstkpze.supabase.co/functions/v1/get-scrape-results?task_id=${taskId}&limit=5000&offset=${offset}`,
    { headers: { apikey: ANON_KEY } }
  );
  const page = await res.json();
  if (page.status !== "success") break; // still processing
  allLeads.push(...page.data);
  if (!page.pagination?.has_more) break;
  offset = page.pagination.next_offset;
}
```

> **Note on webhooks:** Webhook payloads always include the **full result set** in a single POST (no pagination). For very large jobs (>10,000 leads), prefer polling with pagination to avoid exceeding your endpoint's payload limit.

### Business Data Fields

Each business in the `data` array contains:

| Field | Type | Description |
|-------|------|-------------|
| name | string | Business name |
| phone | string | Phone number |
| full_address | string | Full street address |
| city | string | City name |
| rating | number | Google rating (1-5) |
| review_count | number | Number of Google reviews |
| website | string | Business website URL |
| types | string[] | Google business categories |
| matched_category | string | The exact category from Google's taxonomy that this business matched (null if no exact match) |
| price_level | string | Price level ($, $$, $$$, $$$$) |
| place_id | string | Google Maps Place ID |
| business_id | string | Google internal business ID (CID) |
| latitude | number | Latitude |
| longitude | number | Longitude |
| is_claimed | boolean | Whether the business has claimed its Google listing |
| working_hours | object | Operating hours by day |

---

# Location Resolution

The `location` field in `/maps-search` supports:
- **City + State**: "Austin, TX", "Denver, Colorado"
- **City + Country**: "London, UK", "Toronto, Canada"
- **State/Region**: "California", "Texas" (searches top cities by population)
- **Country**: "Germany", "Australia" (searches top cities by population)
- **US ZIP code**: "90210", "78701" (geocoded via Nominatim)
- **UK postcode**: "EH1 1AB", "SW1A 1AA" (geocoded via Nominatim)

Multiple locations can be combined in a single request:
```json
{ "location": ["Austin, TX", "Denver, CO", "90210"] }
```

---

# Clay Integration Example

Use an **HTTP Request** action in Clay with a `callback_url` pointing to a Webhook source:

> ⚠️ **Important:** In Clay, all values in the request body **must be wrapped in parentheses** — even when referencing direct column values. For example, use `(/Company Name)` instead of `/Company Name`. Without parentheses, Clay will send the raw text instead of the resolved column value.

**Maps Search:**
```json
{
  "query": "(/keyword)",
  "location": "(/city), (/state)",
  "limit": 500,
  "filters": { "has_website": true, "min_rating": 3.5 },
  "callback_url": "https://your-clay-webhook-url"
}
```

**Enrichment:**
```json
{
  "business_name": "(/company_name)",
  "business_website": "(/website)",
  "want_email": true,
  "want_general_email": false,
  "general_email_fallback_only": false,
  "clay_row_id": "(/_row_id)",
  "callback_url": "https://your-clay-webhook-url"
}
```

---

# Full Example (Search + Enrich)

```javascript
// Step 1: Start a maps search
const searchRes = await fetch("https://uunrlffonnvmbsstkpze.supabase.co/functions/v1/maps-search", {
  method: "POST",
  headers: { "Content-Type": "application/json", "x-api-key": "YOUR_API_KEY" },
  body: JSON.stringify({ query: "dentists", location: "Chicago, IL", limit: 200 })
});
const { task_id } = await searchRes.json();

// Step 2: Poll for results
let results;
while (true) {
  const poll = await fetch(
    `https://uunrlffonnvmbsstkpze.supabase.co/functions/v1/get-scrape-results?task_id=${task_id}`,
    { headers: { "apikey": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZSIsInJlZiI6InV1bnJsZmZvbm52bWJzc3RrcHplIiwicm9sZSI6ImFub24iLCJpYXQiOjE3NzA0OTIwMzMsImV4cCI6MjA4NjA2ODAzM30.8Ic_U17YrZtsDVnYnIP0XJzrhZJOLDSlJ0AZfTS31FQ" } }
  );
  results = await poll.json();
  if (results.status === "success") break;
  await new Promise(r => setTimeout(r, 15000)); // poll every 15s
}

// Step 3: Enrich each business
for (const biz of results.data) {
  if (!biz.website) continue;
  await fetch("https://uunrlffonnvmbsstkpze.supabase.co/functions/v1/enrich", {
    method: "POST",
    headers: { "Content-Type": "application/json", "x-api-key": "YOUR_API_KEY" },
    body: JSON.stringify({
      business_name: biz.name,
      business_website: biz.website,
      business_location: biz.full_address,
      want_email: true,
      want_general_email: false,
      callback_url: "https://your-webhook-url.com/enrichment"
    })
  });
}
```

---

# Error Responses

| Status | Error | Description |
|--------|-------|-------------|
| 400 | Missing required fields | Required fields not provided |
| 400 | Too many queries | More than 5 keywords in maps-search |
| 400 | Too many locations | More than 25 locations in maps-search |
| 400 | No valid locations found | Could not resolve any provided locations |
| 401 | Missing API key | No x-api-key header provided |
| 401 | Invalid API key | The API key is invalid or inactive |
| 402 | Insufficient credits | No credits remaining |
| 402 | Usage balance depleted | Usage billing balance is $0 |
| 429 | Rate limit (per-minute) | `error: "rate_limit_minute"` — sustained throughput exceeded 500 req/min. `Retry-After: 60`. **Always retry.** |
| 429 | Rate limit (concurrency) | `error: "rate_limit_concurrency"` — too many requests in flight on the same API key at the same instant. `Retry-After: 1`. **Always retry.** |
| 500 | Internal error | Server-side error |

## Rate Limits

> ### ⚠️ CRITICAL: Cap your client at **500 req/min** AND **≤ 30 concurrent requests**, then retry every 429
>
> The `/enrich` and `/maps-search` endpoints enforce **two** independent rate limits. You must respect both, or you will silently lose records.

### The two limits

| Limit | Default | When you hit it | 429 `error` field | `Retry-After` |
|---|---|---|---|---|
| **Per-minute throughput** | 500 requests / rolling minute | Sustained traffic above 500/min | `rate_limit_minute` | **60 s** |
| **Per-key concurrency** | ~30 simultaneous in-flight requests on the same API key | Bursting many parallel requests in a short window (e.g. 200 parallel `fetch` calls) | `rate_limit_concurrency` | **1 s** |

The concurrency limit is the one most clients miss. Even if you stay well under 500/min on average, firing 100 parallel requests in a 2-second burst will get most of them rejected with `rate_limit_concurrency`. **The fix is to limit parallelism, not just total throughput.**

### What a 429 looks like

```json
{
  "error": "rate_limit_concurrency",
  "status": 429,
  "message": "Too many concurrent requests on this API key. Slow down and retry.",
  "retry_after": 1
}
```

```json
{
  "error": "rate_limit_minute",
  "status": 429,
  "message": "Exceeded 500 requests/minute",
  "retry_after": 60,
  "limits": { "per_minute": { "used": 500, "limit": 500 } }
}
```

### What you MUST do

1. **Cap parallelism at ≤ 30 in-flight requests per API key.** Use a concurrency-limited queue (e.g. `p-limit`, a semaphore, or just a worker pool of 30). Sending 200 parallel `fetch` calls in a tight `Promise.all` is the #1 cause of dropped records.
2. **Cap throughput at ≤ 500 req/min** (≈ 1 request every 120 ms). Do not rely on the server to slow you down.
3. **Retry every 429.** A 429 means **your call was not accepted** — if you don't retry, that record is silently lost. It never enters the queue, never triggers a webhook, and never appears in your usage logs. Use the `retry_after` value in the response body (or the `Retry-After` header) and resend the same payload.

Without retry logic + a parallelism cap, a 10K-record run can lose **30–50% of records invisibly**.

### Polling endpoints are excluded

`get-job-results`, `get-job-status`, and `get-scrape-results` do **not** count against either limit. Poll as often as you like.

### Other notes

- Credits are only deducted when data is found (enrichment) or leads are returned (maps search).
- **Multi-title deduplication:** If you send multiple enrichment requests for the same business with different `decision_maker_title` values and the same contact person is found, you are only charged once.

### Reference: client-side throttling + concurrency cap + retry pattern

```javascript
// ✅ CORRECT — caps parallelism at 30, throttles at ~460/min, retries every 429
const MAX_CONCURRENT = 30;       // hard ceiling on in-flight requests per API key
const MIN_INTERVAL_MS = 130;     // ~460/min, safely under the 500 ceiling

let inFlight = 0;
let lastSent = 0;
const queue = [];

function acquire() {
  return new Promise((resolve) => {
    const tryAcquire = () => {
      if (inFlight < MAX_CONCURRENT) { inFlight++; resolve(); }
      else queue.push(tryAcquire);
    };
    tryAcquire();
  });
}
function release() {
  inFlight--;
  const next = queue.shift();
  if (next) next();
}

async function submitWithRetry(payload, attempt = 0) {
  await acquire();
  try {
    // Throttle: never send faster than the per-minute ceiling
    const wait = Math.max(0, MIN_INTERVAL_MS - (Date.now() - lastSent));
    if (wait) await new Promise(r => setTimeout(r, wait));
    lastSent = Date.now();

    const res = await fetch(`${API_BASE}/enrich`, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json', 'x-api-key': API_KEY },
      body: JSON.stringify(payload),
    });

    if (res.status === 429) {
      if (attempt >= 8) throw new Error('Rate limited after 8 retries');
      const body = await res.json().catch(() => ({}));
      // Honor server-provided retry_after (1s for concurrency, 60s for per-minute)
      const retryAfter = body.retry_after ?? Number(res.headers.get('Retry-After')) ?? 1;
      // Add jitter so retries don't synchronize and re-collide
      const jitter = Math.random() * 500;
      const backoff = (retryAfter * 1000) + (attempt * 1000) + jitter;
      await new Promise(r => setTimeout(r, backoff));
      return submitWithRetry(payload, attempt + 1);
    }

    if (!res.ok) throw new Error(`HTTP ${res.status}: ${await res.text()}`);
    return res.json();
  } finally {
    release();
  }
}
```

**For n8n / Zapier / Make:** Set the HTTP node's "On Error → Retry on Fail" with at least 5 retries and a 60s wait between retries. **Do not** mark the workflow successful when a node returns 429 — that record was rejected and must be resubmitted. If you're running the node in a loop with high parallelism (e.g. n8n "Split In Batches" with batch size > 30), reduce the batch size to ≤ 30 to stay under the concurrency limit.

## Polling Endpoints

> **Note:** The polling endpoints use the **Supabase anon key** (via the `apikey` header), **not** your LocalPipe API key. The Supabase anon key is:
> `eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZSIsInJlZiI6InV1bnJsZmZvbm52bWJzc3RrcHplIiwicm9sZSI6ImFub24iLCJpYXQiOjE3NzA0OTIwMzMsImV4cCI6MjA4NjA2ODAzM30.8Ic_U17YrZtsDVnYnIP0XJzrhZJOLDSlJ0AZfTS31FQ`

| Endpoint | Purpose |
|----------|---------|
| GET /get-job-results?job_id=xxx | Poll enrichment results |
| GET /get-job-status?job_id=xxx | Poll enrichment status |
| GET /get-scrape-results?task_id=xxx | Poll maps search results |

---

### 🚨 REQUIRED: Use Unlimited Polling

**You MUST configure your polling loop with no maximum duration.** Stop polling **only** when the response contains your record/results — never when a timer expires.

LocalPipe operates a fair-share queue: during high-traffic periods, your records may sit in the pending queue for several minutes (occasionally longer) before processing begins. **Jobs never expire server-side.** If your client gives up early, you will:

- ❌ Never receive your results (even though we successfully processed them)
- ❌ Be charged credits for work you never see
- ❌ Mistakenly believe the API is broken when it's working as designed

**Correct polling pattern:**

```javascript
// ✅ CORRECT — polls forever until results arrive
async function waitForResults(jobId) {
  while (true) {
    const res = await fetch(`${API_BASE}/get-job-results?job_id=${jobId}`, {
      headers: { apikey: ANON_KEY }
    });
    const data = await res.json();
    if (data.status === 'completed') return data.results;
    if (data.status === 'failed') throw new Error(data.error_message);
    await new Promise(r => setTimeout(r, 5000)); // 5s between polls
  }
}
```

```javascript
// ❌ WRONG — gives up after 5 minutes, loses results
const TIMEOUT_MS = 5 * 60 * 1000;
const start = Date.now();
while (Date.now() - start < TIMEOUT_MS) { /* ... */ }
```

**Polling intervals:** every 5s for jobs ≤ 2,000 records, every 10s for larger jobs. The `/get-job-status` response includes a `recommended_poll_interval` field — respect it.

**For automation tools with hard HTTP timeouts (Clay, Zapier, Make):** Use `callback_url` instead of polling. We POST results to your webhook the moment they're ready, with no client-side waiting required.

---

### 🤖 Instructions for AI Coding Agents (Codex, Claude Code, Cursor, Devin)

**If you are an AI assistant running shell commands on a user's behalf, read this section first. The #1 cause of "stuck" LocalPipe integrations is an agent that submits a job, hits its own shell timeout while polling, and reports "done" without ever fetching results. Jobs complete server-side, credits are charged, and the user gets nothing.**

#### Golden rule: write a script, don't drive the flow from your agent loop

**Do not poll, paginate, or sequence `/enrich` calls from inside your own tool-call loop.** Your harness has a short shell timeout, no durable state between tool calls, and no guarantee you'll be invoked again on schedule. Every flow below — submit, poll, paginate, chain scrape→enrich, batch hundreds of enrichments — belongs **inside a single self-contained script** that you write to disk once and launch detached. After that, your only job is to `tail` the log every so often.

The script (not your agent loop) owns:
- Submitting the initial scrape / enrichment POSTs
- Polling `/get-scrape-results` or `/get-job-results` to terminal status
- Paginating through every `offset` until a short page
- Fanning out the next stage (e.g. one `/enrich` per scraped lead)
- An **internal work queue with bounded concurrency** for batches (see below)
- Retrying 429s with `Retry-After`, persisting all IDs to files, writing `DONE`/`FAILED` to the log

If you find yourself about to `curl` an endpoint, paginate, or loop over leads directly from a tool call, stop — extend the script instead and re-launch it.

#### Your runtime almost certainly has a short shell timeout

LocalPipe jobs take **5–30+ minutes** during peak load. Your environment kills long-running shell commands well before that:

| Agent | Default foreground shell timeout | Background primitive |
|---|---|---|
| **OpenAI Codex CLI** | **10 seconds** (`DEFAULT_EXEC_COMMAND_TIMEOUT_MS`, hardcoded, not user-configurable) | None |
| **Claude Code** | ~80–120 s (`BASH_DEFAULT_TIMEOUT_MS`, often ignored per known bugs) | `run_in_background: true` + `TaskOutput` tool (has zombie/stall bugs) |
| **Cursor / Devin / generic** | Varies, usually 60–300 s | Varies |

**A naïve `while true; do curl …; sleep 15; done` in a single tool call will be SIGKILLed long before the job completes.** You must structure polling so the long wait happens *outside* any single shell call — the script handles it, not you.

#### The only reliable pattern: detached script + log file + short poll calls

Do all three steps in this order. Do not skip step 1.

**Step 1 — Write a self-contained poll script to disk:**

```bash
cat > /tmp/localpipe_poll.sh << 'EOF'
#!/usr/bin/env bash
set -u
API_BASE="https://uunrlffonnvmbsstkpze.supabase.co/functions/v1"
API_KEY="$LOCALPIPE_API_KEY"
ANON_KEY="<supabase-anon-key-from-docs-above>"

# Submit the scrape
TASK_ID=$(curl -sS -X POST "$API_BASE/maps-search" \
  -H "x-api-key: $API_KEY" -H "content-type: application/json" \
  -d '{"query":["dentist"],"location":["Austin, TX","Dallas, TX","Houston, TX","San Antonio, TX","Fort Worth, TX"],"limit":500}' | jq -r '.task_id')
echo "TASK_ID=$TASK_ID" | tee /tmp/localpipe_task.id

# Poll forever — exit ONLY on terminal status
OFFSET=0
while true; do
  RESP=$(curl -sS -H "apikey: $ANON_KEY" \
    "$API_BASE/get-scrape-results?task_id=$TASK_ID&limit=5000&offset=$OFFSET")
  STATUS=$(echo "$RESP" | jq -r '.status')
  COUNT=$(echo "$RESP" | jq -r '.results | length')
  echo "[$(date -u +%H:%M:%S)] status=$STATUS offset=$OFFSET batch=$COUNT"
  if [ "$STATUS" = "done" ]; then
    echo "$RESP" > "/tmp/localpipe_results_$OFFSET.json"
    [ "$COUNT" -lt 5000 ] && break
    OFFSET=$((OFFSET + 5000))
    continue
  fi
  [ "$STATUS" = "failed" ] && { echo "FAILED"; exit 1; }
  sleep 15
done
echo "DONE"
EOF
chmod +x /tmp/localpipe_poll.sh
```

**Step 2 — Launch it detached so it survives your shell timeout** (this command returns instantly):

```bash
nohup /tmp/localpipe_poll.sh > /tmp/localpipe.log 2>&1 &
echo $! > /tmp/localpipe.pid
disown
echo "started pid=$(cat /tmp/localpipe.pid)"
```

**Claude Code:** prefer the native primitive — call `Bash` with `run_in_background: true` on the `/tmp/localpipe_poll.sh` invocation, capture the returned job ID, and use `TaskOutput` in step 3 instead of `tail`. Fall back to `nohup` if `TaskOutput` returns stale output.

**Step 3 — Across separate, short tool calls, poll the log file until you see `DONE` or `FAILED`:**

```bash
# Each of these is a separate Bash tool call — well under any agent's timeout.
# Repeat every 30–60s until "DONE" appears in the log.
tail -20 /tmp/localpipe.log
ps -p "$(cat /tmp/localpipe.pid)" > /dev/null && echo "still running" || echo "exited"
```

When the log shows `DONE`, read `/tmp/localpipe_results_*.json` and present the results. **Do not end the user's turn until either `DONE` or `FAILED` appears in the log.**

#### Chained workflows (scrape → enrich)

Extend the same script: after the scrape `DONE` line, **count the total scraped leads across every `/tmp/localpipe_results_*.json` page** and write it to `/tmp/localpipe_scrape_total`. Then iterate every business in every page and `POST /enrich` for each, appending each returned `job_id` to `/tmp/localpipe_enrich_jobs.txt`. **Before moving on, assert `wc -l /tmp/localpipe_enrich_jobs.txt` equals the scrape total** — if it's less, you dropped leads (almost always a forgotten `offset` page or a hit shell timeout mid-loop). Resume from the missing offset and re-submit. Only after the counts match should you poll the enrichment jobs via `/get-job-results`. All of this stays inside the one detached script so a single `nohup` covers the whole chain.

> **Common silent failure:** a scrape returns 5,000+ leads across multiple pages but the agent submits enrichment for only the first 400, then exits because Codex's 10s timer killed the loop. The user sees "done" with 92% of their data missing and no error. The count assertion above is the only thing that catches this.

#### Batching many scrapes or enrichments? Queue them inside the script

Anytime the user's request fans out into more than a handful of `/maps-search` or `/enrich` calls (e.g. "scrape 20 cities", "enrich this 5K CSV", "run these 50 keyword × city combos"), **do not submit them serially from your tool calls and do not fire them all at once with `&`**. Both lose data the moment your shell call dies or you hit the 500 req/min rate limit. Instead, the script should run a tiny in-memory work queue:

```bash
# Inside /tmp/localpipe_poll.sh — append to a work file, drain with bounded concurrency
WORK=/tmp/localpipe_work.txt        # one JSON payload per line (scrape or enrich input)
DONE=/tmp/localpipe_submitted.txt   # task_id / job_id per accepted submission
CONCURRENCY=8                       # safe under 500/min for /enrich and /maps-search

drain_queue() {
  : > "$DONE"
  while IFS= read -r PAYLOAD; do
    # background each submission, cap parallelism
    while [ "$(jobs -rp | wc -l)" -ge "$CONCURRENCY" ]; do sleep 0.2; done
    (
      RESP=$(curl -sS -X POST "$API_BASE/enrich" \
        -H "x-api-key: $API_KEY" -H "content-type: application/json" \
        -w "\n%{http_code}" -d "$PAYLOAD")
      CODE=$(echo "$RESP" | tail -1)
      BODY=$(echo "$RESP" | sed '$d')
      if [ "$CODE" = "429" ]; then
        sleep 60                                  # honor Retry-After
        echo "$PAYLOAD" >> "$WORK.retry"          # requeue
      elif [ "$CODE" = "202" ] || [ "$CODE" = "200" ]; then
        echo "$BODY" | jq -r '.job_id // .task_id' >> "$DONE"
      else
        echo "$PAYLOAD" >> "$WORK.failed"         # surface in log, do not silently drop
      fi
    ) &
  done < "$WORK"
  wait
  # re-drain retries until empty
  [ -s "$WORK.retry" ] && { mv "$WORK.retry" "$WORK"; drain_queue; }
}
```

Why this matters:

- **No data loss on agent timeout** — the queue lives on disk; the detached script keeps draining it after your tool call dies.
- **Stays under 500/min** — bounded concurrency + 429 requeue is the only safe way to submit hundreds of jobs without losing them. A naked `for` loop with `&` will burst-trigger 429s and silently drop the rejected ones.
- **Resumable** — if anything crashes, the script restarts and resumes from `$WORK` (whatever wasn't moved to `$DONE` is still pending).
- **Auditable** — `wc -l "$DONE"` must equal `wc -l "$WORK_ORIG"` before you tell the user "submitted". Same count-matching rule as the chained scrape→enrich flow.

For scrapes that fan out across many cities/keywords, write each `{keyword, city, state}` combo as one line in `$WORK` and reuse the exact same `drain_queue` against `/maps-search` — then feed every returned `task_id` into a second queue that polls them all to completion.

#### Easier alternative — let us push to you (zero polling)

If the user has any webhook-capable destination (their own server, webhook.site, n8n/Make/Zapier webhook URL, ngrok tunnel), include `callback_url` in both `/maps-search` and `/enrich` POSTs. The script can then exit immediately after enqueueing — we POST the full result payload the moment each job completes. **Always prefer this over polling when possible.** You still need the script + queue for submission and 429 retries, but you can skip the polling loop entirely.

#### Hard rules — never violate these

1. **Never submit and exit.** A `POST` without either (a) a launched detached poll script or (b) a `callback_url` = guaranteed data loss for the user.
2. **Never put the long wait inside a single foreground tool call.** Codex's 10s timer will kill it; Claude's ~80s timer will kill it; the user sees a hang and you see a SIGKILL.
3. **Never add `maxAttempts` or `timeoutMs` to the poll loop.** Exit only on `status: "done" | "completed" | "failed"`. Jobs legitimately run 30+ minutes.
4. **Always persist `task_id` / `job_id` to a file** (`/tmp/localpipe_task.id`) *before* starting the poll. If anything crashes, the user can resume by re-running step 3 against the saved ID.
5. **Always tail the log between poll calls** and show the user the latest status line — silent waiting looks like a hang and users will Ctrl-C.
6. **202 is success, not an error.** Some HTTP clients fail `res.ok` checks on 202. Always read the JSON body regardless of status code on `/enrich` and `/maps-search`.
7. **Always paginate `/get-scrape-results` until `results.length < limit`.** A single call with `limit=5000` is NOT guaranteed to return everything — keep incrementing `offset` until you get a short page. Track total leads received.
8. **Enrichment submission count MUST equal scrape result count.** Before declaring a chained workflow complete, assert `enrich_jobs_submitted == scrape_total`. If they don't match, find and resubmit the missing leads — do not tell the user the run is done.
9. **Never sequence multi-step flows from your agent loop.** Submit → poll → paginate → fan-out → poll-again all belong inside the script. If your plan involves more than one tool call making HTTP requests to LocalPipe, you wrote the script wrong — extend it instead.
10. **Always queue batches with bounded concurrency inside the script.** No `for lead in leads; do curl …; done` (serial = times out, drops the tail), no `for lead in leads; do curl … &; done` (burst = 429s, drops the rejected). Use the `drain_queue` pattern above.

#### Anti-patterns — never generate this code

- ❌ `curl -X POST .../enrich && echo "submitted, check dashboard"` and end the turn
- ❌ `while true; do curl .../get-job-results; sleep 15; done` as a single tool call (killed by shell timeout)
- ❌ `for i in $(seq 1 20); do …; sleep 15; done` (gives up after 5 min)
- ❌ Polling inside a detached `setTimeout`/Promise after the main function returns
- ❌ Submitting 100 enrichments in a loop without launching any poller for them
- ❌ Treating 202 Accepted as a failure
- ❌ `curl ".../get-scrape-results?task_id=X&limit=5000" | jq '.results[] | …enrich…'` with no `offset` pagination — silently drops everything past the first page
- ❌ Reporting "enrichment started for N leads" where N is smaller than the scrape's total
- ❌ Driving the scrape→poll→enrich→poll chain across multiple of your own tool calls instead of putting it in one detached script
- ❌ `for lead in $(cat leads.json); do curl .../enrich … &; done; wait` — unbounded fan-out; the 429s vanish into the void
- ❌ `for city in cities; do curl .../maps-search -d '{"query":"plumber","location":"'$city'"}'; done` — `/maps-search` accepts arrays for both `query` and `location`; send ONE request with `"location": [...all cities...]` (up to 25) and `"query": [...all keywords...]` (up to 5). Looping per-city is the #1 way AI agents waste rate limit and drop data.
- ❌ Sending a separate `/maps-search` request per keyword for the same city list — batch all keywords into the `query` array of one call.




## Best Practices

1. **Cap your client at ≤ 500 requests/minute** — anything above this is rejected with 429 and lost unless you retry
2. **Always retry 429 responses** — honor `Retry-After` (60s default), then back off exponentially. A 429 means the call was *not* accepted
3. **Always use callback_url** for Clay and automation platforms — avoids timeout issues
4. **Split large locations** — searching "Texas" searches top 50 cities; for full coverage, list specific cities
5. **Use filters** to reduce noise — `has_website: true` and `min_rating: 3.5` dramatically improve lead quality
6. **Poll every 15 seconds** — most small searches complete in under 60 seconds
7. **Combine search + enrich** — use maps search to find businesses, then enrich each one for owner contact info

---

# Infrastructure

## Queue-Based Processing

All enrichment requests are **asynchronously queued** for processing. When you submit a request:

1. **Immediate acceptance** — Every valid request returns 202 Accepted instantly
2. **Queue processing** — Records are processed from the queue in batches
3. **Webhook delivery** — Results POST to your callback_url when complete

This architecture allows you to send thousands of requests simultaneously without waiting, **as long as your client throttles to ≤ 500/min** and retries any 429s.

## Rate Limiting Structure

To ensure stability during traffic spikes, the API implements tiered rate limits:

| Limit | Default | Applies To | Behavior |
|-------|---------|------------|----------|
| Per-minute | **500 requests** | /enrich, /maps-search | Returns **429** with `Retry-After: 60` header — **client MUST retry** |
| Daily | 30,000 requests | /enrich, /maps-search | Resets at midnight UTC |
| Monthly | 900,000 requests | /enrich, /maps-search | Resets on the 1st of each month |

> **Why this matters:** 429s short-circuit before the request reaches the queue. They are **not logged in your usage analytics**. If your client doesn't retry, those records are lost — no job created, no webhook fired, no error visible in the dashboard.

**Polling endpoints are excluded from all rate limits:** get-job-results, get-job-status, and get-scrape-results do not count against your quota.

**Custom limits:** High-volume users can request increased per-minute limits by contacting support.

## Database Optimization

Recent infrastructure improvements have reduced database operations per request by 50%:
- Consolidated credit checking and subscription validation
- Asynchronous API logging (fire-and-forget pattern)
- Optimized batch processing via cron-based scheduling

These changes ensure consistent performance even during peak usage periods.