Concepts

Workflow

A workflow is your named project. It stores the graph (nodes + edges), a status (active or paused), and an optional interval in minutes. Active workflows with an interval are picked up by a background worker. Paused workflows never auto-run, but you can still press Run anytime.

interval_minutes examples: 1440 = daily, 60 = hourly (if your plan allows), empty/null = manual only. After a run attempt with an interval set, next_run_at moves forward by that interval.

Example:“Daily product list” with status active and interval 1440.

Console → Workflows. API: GET/POST /api/v1/workflows.

Node

A node is one step on the canvas. There are four types: scrape (read pages), url_gen (build URL lists), store (write a table), notify (send email on change). Each node has an id (like s1), a type, and a config object with that type’s settings.

Scrape, notify, and url_gen can be saved as reusable presets under Workflows → Nodes. Store is always configured on the specific workflow because it points at a dataset.

Example:One scrape node for the list page, one store node named “products”.

Open a workflow editor, or Workflows → Nodes for presets.

Run

A run is one execution of the whole graph. Status is ok or error. Every node writes a node run record with input, output, and error. Failed runs still keep those records so you can see which step broke.

Console Run and API POST …/run wait until the graph finishes. Open the run detail to inspect items before you change config.

Example:Run #42: scrape ok (18 items), store ok (18 upserted).

Workflow editor → run history. API: GET …/runs and …/runs/{id}.

Dataset

A dataset is a logical table owned by your account. The store node writes rows into it. Rows are upserted: the same key updates the existing row instead of creating a duplicate. Default key fields are url and link — you should almost always set an explicit key_fields list with your product URL.

You can browse rows in the console, page through the API, or export CSV/JSON (large tables are capped). Columns appear after the first successful store write.

Example:Dataset “products” with columns title, price, url and 120 rows after a run.

Console → Datasets. API: GET /api/v1/datasets, …/rows, …/export.

Preset

A preset is a saved scrape, notify, or url_gen configuration with a name. Drop it onto another workflow instead of redoing discovery. Scrape presets must already include selected fields.

Presets do not by themselves use quota; quota applies when a scrape node exists on a saved workflow graph.

Example:Preset “Shop list scrape” reused on two workflows.

Workflows → Nodes, or save from a node’s config. API: /api/v1/node-presets.

Graph JSON

Under the hood the canvas is JSON: nodes: [{id, type, config}] and edges: [{from, to}]. from is the upstream node id. The API uses the same shape (field name graph on write; responses may show graph_json).

Do not invent source/target keys — use from and to. Cycles are rejected. You usually edit this visually; the API guide shows how to PATCH it with curl.

Example:Edge {from:"s1", to:"st1"} means scrape output feeds store.

Scrape

The scrape node opens web pages and turns them into rows (items). Mode list means “many cards/rows on this page”. Mode detail means “one page = one richer row”, often after a list scrape passed product links. You describe columns in requirement, then keep the discovered fields.

Each webpage request by scrape nodes consumes 1 credit when the run succeeds. Default limit is 20 URLs processed per run of that node. For list→detail, set input_field to the column that holds the next URL.

Example:List scrape on /catalog?page=1 keeping title, price, url; detail scrape reading that url for description.

URL generator

The URL generator does not open pages. It only builds a list of URLs for a downstream scrape. Range mode fills a template such as https://example.com/list?page={page} from start to end. List mode pastes one URL per line when pages are not a simple number sequence.

Output is always items like [{url: "…"}]. Cap is 500 URLs per generator node. It does not use a quota slot. Connect it only to a scrape node (usually list mode).

Example:Pages 1–10 of a catalog into one scrape node.

Add URL generator on the canvas; preview the count before Apply.

Notify

Notify sends email when its trigger matches, compared to the previous run of the same notify node. when=new emails only rows whose key (url/link/id/title) was not seen last time. when=field_change compares watch_fields (for example price) for matching keys. Other when values hash the entire items list and mail if anything changed.

email defaults to your account email. Node “test” can dry-run without sending. Workflow notify in v1 is email (webhook may appear elsewhere in the product for tasks/plans).

Example:Daily run; email only when price on an existing product URL changes.

Credits

Your plan grants a monthly (or yearly ×12) credit pool. Each webpage request by scrape nodes consumes 1 credit. Saving graphs does not. Upgrade or wait for the next period when you hit the limit.

Running a graph when remaining credits are too low returns HTTP 402. Saving alone does not consume credits.

Example:Free plan with 20 credits; requesting 3 list page URLs uses 3 credits.

Schedule

A background worker looks for active workflows whose next_run_at is due. After an attempted run, if interval_minutes is set, next_run_at becomes roughly now + interval. status=paused skips the worker entirely.

You can always run manually regardless of schedule. Use paused while you are still editing fields so you do not get surprise emails.

Example:Active + 1440 minutes → about one automatic run per day.