maskanx_cm_backend/README.md

# MaskanX Backend

Independent Python and FastAPI backend for MaskanX.

## Service Contract

- Default API URL: `http://127.0.0.1:8088`
- PostgreSQL database: `campaign`
- Frontend origin: configured with `MASKANX_CORS_ORIGINS`
- Maskan CRM: connected through its versioned REST API and optional MCP tools

This repository never connects directly to the Maskan CRM PostgreSQL database.

## Local Setup

1. Create the database:

   ```sql
   CREATE DATABASE campaign;
   ```

2. Install dependencies and configure the environment:

   ```powershell
   python -m venv .venv
   .\.venv\Scripts\python.exe -m pip install -e ".[dev]"
   npm install
   Copy-Item .env.local.example .env.local
   ```

3. Migrate, seed, and start:

   ```powershell
   npm run local:migrate
   npm run local:seed
   npm run local
   ```

Open API documentation at `http://127.0.0.1:8088/docs` when documentation is
enabled.

### Running the tests

```powershell
npm run test
```

`npm run test` loads `.env.testing`, which points at a **separate** database so
a test run cannot touch development data. Create it once:

```sql
CREATE DATABASE campaign_test;
```

```powershell
npm run test:migrate
```

Without it, the repository integration tests skip with "PostgreSQL is not
reachable" and the suite still passes — so check the skip count, not just the
exit code.

### Upgrading from JSON storage

Chats and cron jobs were originally stored as JSON files in the working
directory (`~/.adclaw/chats.json` and `~/.adclaw/jobs.json`). PostgreSQL is now
the default backend, so those records are no longer read and existing chats,
sessions, and scheduled jobs appear to be missing even though nothing was
deleted.

Copy them into PostgreSQL once:

```powershell
npm run local:import-json -- --dry-run   # report what would be imported
npm run local:import-json                # perform the import
```

The import upserts by id and never deletes, so it is safe to re-run. The JSON
files are left untouched as a backup.

## Campaigns

Campaign records live in PostgreSQL. Apply the schema before first use:

```powershell
npm run local:migrate
```

Set `META_ADS_ACCESS_TOKEN` and `META_PAGE_ID` in Settings > Environments, and
enable the `meta_ads` MCP client if you also want campaign tools in chat.
`META_PAGE_ID` is the Facebook Page ads are published from; sync refuses to
start without it, rather than failing part-way through building the object
chain. A campaign can override it with `advanced.page_id`.

### The campaign lifecycle

```
draft -> pending_approval -> approved -> synced -> live
                                            |       |
                                            +-> paused <-+
                                                  |
                                                stopped
```

`approved` is where a human signs off. `synced` means the objects exist in
Meta but are **paused**. Only `live` spends money.

### Ad previews create nothing

Previews are rendered from a creative spec via `generatepreviews`, so the
wizard shows exactly how an ad will look across desktop feed, mobile feed,
Instagram feed, Instagram story, Facebook story and right column before any
Meta object exists.

### Sync creates paused objects only

`POST /campaigns/{id}/sync` builds the Meta chain — campaign, ad set,
creative, ad — and **every object is created `PAUSED`**. Nothing sync does can
start spending. The guard is enforced in three places: the typed `create_*`
methods, `MetaClient._post` itself (which refuses a non-`PAUSED` status on any
create path), and the launch endpoint being the only caller allowed to send
`ACTIVE`.

Each id is persisted the moment it is obtained, so a sync that fails part-way
is resumable: re-running it reuses the ids already stored and creates only
what is missing. That is what stops a retry from leaving a second set of
objects in a real ad account.

### Launching requires an operator and a payment method

`POST /campaigns/{id}/launch` sets the whole object chain active — ad, then
ad set, then campaign. Meta only delivers when all three are active, so
activating just the campaign would launch nothing. The campaign goes **last**
on purpose: nothing under a paused campaign delivers, so a failure part-way
leaves it unable to spend. That is also why pause and stop only have to flip
the campaign.

It is the only action that starts spend, and it is refused unless:

- the campaign is `synced` or `paused`,
- `approved_by` names a resolved operator — an approval nobody can be named
  for does not authorise spend, and
- the ad account has a `funding_source`.

Set `MASKANX_OPERATOR_TOKENS` to a comma-separated list of `name:token`
pairs. Approve, sync, launch, pause, stop and adopt read the token from the
`X-MaskanX-Operator` header and record the resolved name as the actor; any
actor in the request body is ignored. With no tokens configured the operator
resolves to `unauthenticated`, which is enough to draft and approve but not
to launch. The Campaigns page has a field for storing the token in the
browser.

### Importing campaigns made in Ads Manager

`GET /campaigns/discover?ad_account_id=act_...` lists Meta campaigns MaskanX
does not know about; `POST /campaigns/adopt` imports one. Adopting twice is
safe — the second call returns the existing record.

Imported campaigns are marked `origin=imported` and treated as
Ads-Manager-owned: they are never synced, launching one touches only the
campaign (its ad sets and ads keep the statuses set in Ads Manager), and they
cannot be deleted from MaskanX.

### Deleting

`DELETE /campaigns/{id}` deletes the campaign on Meta first and only then
removes the local row. If the Meta delete fails the local row is kept and the
call answers `502`: a campaign forgotten here but still live on Meta would
keep spending with nothing left to show it exists. Deleting a campaign
cascades to its ad sets and ads; the creative is left behind, being an
account-level asset other ads may reference.

Deleting an **imported** campaign is refused with `409`. Deleting it on Meta
would destroy work MaskanX did not author, and deleting only the local row
would achieve nothing — the reconciler would import it again on the next
cycle. Delete it in Ads Manager instead; the reconciler then archives the
local record, keeping its spend history reportable.

### Drift reconciliation

A background loop reconciles each ad account with Meta: campaigns created in
Ads Manager are imported, statuses are copied back so a campaign paused
directly in Meta stops showing as `live` here, and campaigns that have
disappeared are archived (never hard-deleted, so their spend history stays
reportable).

Meta is authoritative for delivery status; MaskanX is authoritative for its
own metadata — guardrails, approvals and audit history are never overwritten.
Campaigns that have not reached Meta yet (`draft`, `pending_approval`,
`approved`) are left alone entirely.

The interval defaults to 120 seconds and is set with
`MASKANX_CAMPAIGN_RECONCILE_SECONDS`; `0` disables the loop.

### Guardrails

Guardrails are the only thing in MaskanX that halts spending without a
human. A background sweep reads each **live** campaign's spend from Meta and
applies the campaign's own rules:

| Rule | Window | Action |
| --- | --- | --- |
| `max_campaign_spend` | lifetime | stop |
| `auto_pause_if_spend_reaches` | today | pause |
| `max_cost_per_lead` | today | pause |
| `max_cost_per_click` | today | pause |

The most severe breach wins, so a campaign over both its lifetime cap and
its cost per lead is stopped rather than merely paused. `stop_loss_enabled`
gates only `max_campaign_spend`, the one rule that ends a campaign
permanently; turning it off leaves the pause rules working.

Every amount is in **minor** currency units (paise, cents), matching the
budget fields. Cost rules are skipped until the campaign has its first lead
or click — no leads yet is not an infinite cost per lead, and pausing for
that would kill every campaign in its first hour.

The campaign is paused **on Meta first**, then recorded locally: a campaign
recorded as paused but still delivering is the outcome the whole mechanism
exists to prevent. Each action writes an event naming the rule and the
observed value, so a campaign that stopped overnight can be explained the
next morning, and the Campaigns page tags it.

Interval: `MASKANX_GUARDRAIL_INTERVAL_SECONDS`, default 900 (15 minutes).
`0` disables enforcement, which is logged as a warning at startup — a safety
system that is not running should be loud.

This is a deterministic loop, **not** a `maskanx_cron_jobs` entry. That
scheduler runs prompts through an agent, and asking a language model whether
a budget has been exceeded would make an arithmetic guarantee probabilistic.

`require_approval_for_budget_increase` is enforced at the API layer instead:
raising a budget on a campaign that is awaiting approval returns `409`,
because it would change the figure the approver is reviewing. Reject it back
to draft, edit, and resubmit.

### Analytics

Insights are stored in `maskanx_campaign_insights` and read from there, so
the dashboard stays fast and keeps working while Meta is rate-limiting. Only
`POST /campaigns/{id}/insights/refresh` goes out to Meta.

| Route | Returns |
| --- | --- |
| `GET /campaigns/analytics/dashboard` | Yesterday and last-7-days spend, leads and cost per lead, plus the most and least efficient campaign |
| `GET /campaigns/{id}/insights` | A daily series and the total for one campaign |
| `GET /campaigns/{id}/insights?breakdown=age` | The same metrics split by one breakdown |
| `POST /campaigns/{id}/insights/refresh` | Re-fetches from Meta now |

Three things about how this works are worth knowing before changing it:

**Insights are re-fetched, not appended.** Meta revises a day's figures for
about 28 days as conversions are attributed late, so each sync re-reads a
trailing window and upserts on
`(level, object_id, date, breakdown_key, breakdown_value)`. Appending would
double spend the first time a day was refreshed. Window:
`MASKANX_INSIGHTS_WINDOW_DAYS`, default 28 — shortening it freezes late
conversions out permanently.

**Breakdowns multiply rows, not columns.** The same spend appears in every
breakdown's rows, so they are tagged with a `breakdown_key` and every
aggregate filters to `breakdown_key = ''`. A query that summed across
breakdowns would count the same spend once per age bracket.

**Both dashboard windows end yesterday.** Today's figures are partial;
including them would make every "vs. last week" comparison read as a
collapse each morning.

Costs are always computed from summed totals, never averaged from per-day
costs — the average of daily costs per lead is not the cost per lead over the
period. Cost fields are `null`, not `0`, when there is nothing to divide by.

Sync interval: `MASKANX_INSIGHTS_SYNC_SECONDS`, default 3600. `0` disables
the loop; the dashboard then only updates when a campaign is refreshed by
hand.

### Maskan CRM

Leads and campaign figures are pushed to Maskan CRM **server to server**,
over its integration API — not through MCP. MCP is for an agent deciding to
do something; leads have to arrive whether or not anyone is talking to the
agent.

- Leads: Meta lead-ad submissions, attributed to the campaign, ad set and ad
  that produced them. Meta's own lead id is the external id, so re-sending
  is harmless. A trailing window is re-sent on every run rather than
  tracking a high-water mark, which makes a half-failed run self-healing.
- Campaigns: spend, clicks, leads and cost per lead, upserted so the CRM
  shows current figures rather than the first ones it saw.

Configure `MASKAN_CRM_API_URL` and `MASKAN_CRM_INTEGRATION_KEY` in
Settings > Environments. Interval: `MASKANX_CRM_SYNC_SECONDS`, default 300.
Lead re-read window: `MASKANX_CRM_LEAD_WINDOW_DAYS`, default 7. `0` on the
interval disables the loop.

### Custom fields, created from chat

MCP's job is the other half: changing the CRM's shape on request. Two tools
on the `maskan_crm` server let an agent add a field mid-conversation, the
way HubSpot's custom properties work:

- `list_maskan_crm_properties` — what fields already exist, with their types
  and allowed values
- `create_maskan_crm_property` — define a new one

The definition is stored in the CRM; values live in each record's existing
JSON attributes, so adding a field is an INSERT rather than a migration and
the next record can use it immediately. The field appears in the CRM's
**Custom fields** screen straight away, marked as MaskanX-created so a
person can tell which fields came out of a conversation.

The CRM validates every definition and every value written against one, and
an agent gets no shortcut around those rules — if anything it needs them
more, since it will happily invent a field name from half a sentence. A
value for an undefined property is rejected rather than stored: a typo left
in the database looks exactly like data.

`name` and `data_type` cannot be changed after creation. Renaming would
orphan every value already stored under the old key, and retyping would
leave values that no longer satisfy the type.

### Live smoke test

The unit suite runs entirely against a fake transport: it proves the code
sends what we think it sends, not that Meta accepts it. One opt-in test
closes that gap by syncing a real campaign, reading back from Graph that all
three objects are `PAUSED`, then deleting them:

```powershell
$env:MASKANX_LIVE_TESTS = "1"
$env:MASKANX_LIVE_AD_ACCOUNT_ID = "act_<your account id>"
npm run test -- tests/test_campaign_sync_live.py
```

It creates and deletes real objects. They are paused for their whole life, so
it cannot spend money, and cleanup runs in a `finally` block.

### Budget minimums

Each ad account publishes a `min_daily_budget` in minor units. The API rejects
a lower daily budget with `422` and a readable message rather than letting Meta
reject it opaquely later. If the account cannot be read (no token, rate limit),
the check is skipped and a warning is logged: the budget guardrail is not
enforced in that case.

### Delivery requires a payment method

Meta will not deliver ads until a funding source is attached to the ad account.
**This cannot be done from MaskanX** — the Marketing API does not allow
creating funding instruments, so it must be done in Meta Business Manager. The
Campaigns page shows a warning when no payment method is present.

### Clearing and restoring credentials

**Stop MaskanX before clearing credentials.** A running app holds its
configuration in memory and can rewrite `config.json` and `providers.json`
after the clear, silently restoring the values that were just removed. The
clear script now verifies the result and exits non-zero if anything
secret-looking survives.

Credentials live in three places, and all of them are cleared:

- `~/.adclaw.secret/providers.json` - LLM provider keys
- `~/.adclaw.secret/envs.json` - integration keys
- `~/.adclaw/companies/<id>/` - a per-company copy of both, plus LinkedIn OAuth

Because each company keeps its own copy, an intact company store can be used to
recover the active one:

```powershell
npm run secrets:restore -- --from default --dry-run
npm run secrets:restore -- --from default
```

Credential files are copied wholesale; `config.json` is patched surgically, so
only missing MCP env values and disabled clients are restored and
company-specific settings are preserved. A backup is written to
`~/.adclaw.secret/restore-backups/` first.

## Commands

| Command | Purpose |
| --- | --- |
| `npm start` | Start using `.env` or injected environment variables |
| `npm run build` | Build a Python wheel in `dist/` |
| `npm run local` | Local API with reload |
| `npm run local:migrate` | Apply local PostgreSQL migrations |
| `npm run local:migrate:undo` | Roll back the newest migration |
| `npm run local:migrate:undo:all` | Roll back every migration |
| `npm run local:seed` | Add starter data |
| `npm run local:seed:undo` | Remove starter data |
| `npm run local:import-json` | Import legacy JSON chats/cron jobs into PostgreSQL |
| `npm run secrets:dry-run` | Report which credentials a clear would remove |
| `npm run secrets:clear` | Clear runtime credentials (creates a backup) |
| `npm run secrets:clear:all` | Also blank secret lines in `.env` files (creates a backup) |
| `npm run secrets:delete:all` | Clear everything including `.env` files (creates a backup) |
| `npm run secrets:delete:all:no-backup` | Same, with **no backup** - irreversible |
| `npm run secrets:restore` | Restore credentials from an intact company store |
| `npm run local:reset` | Rebuild and seed the local schema |
| `npm run dev` | Development API with reload |
| `npm run dev:migrate` | Apply development migrations |
| `npm run dev:reset` | Reset the development schema |
| `npm test` | Run Pytest using `.env.testing` |
| `npm run test:migrate` | Apply test database migrations |
| `npm run test:reset` | Reset the test schema |
| `npm run prod` | Start the production API |
| `npm run prod:migrate` | Apply production migrations |
| `npm run prod:seed` | Apply production starter data |

The command names match the platform deployment contract. They invoke the
native MaskanX Python migration runner, not Sequelize.

## Docker

```powershell
Copy-Item .env.local.example .env
docker compose up -d --build
```

For a managed PostgreSQL deployment, provide `DATABASE_URL` through the hosting
secret manager and run `npm run prod:migrate` as a release step.