[pull] main from activepieces:main

brain/knowledge/flows-execution/flows.md

@@ -35,6 +35,8 @@ Flows are the core automation primitive: a versioned directed graph of trigger +
 - **`transaction()` (`core/db/transaction.ts`) is a bare `dataSource.transaction()`** — it acquires a *new* connection, not a savepoint. Nesting it deadlocks, so check every caller before wrapping a service method that others may already call inside a transaction.
 - Step settings split a piece's props into an always-visible **essential** set and a collapsed **Advanced** section: a prop is Advanced only when it sets `advanced: true` (everything else — incl. `MARKDOWN`, tab/section group members, and checkbox reveal targets — stays essential). `propertyGroups` render as tabs, sectioned cards, or the "Add filter" builder.
 - **Flows stuck in `DELETING` keep eating the active-flow limit.** Deletion is a durable BullMQ system job (`delete-flow-<flowId>`), not synchronous: `delete()` sets `operationStatus=DELETING` and enqueues, and the row plus `status=ENABLED` only go away when the job finishes. That job runs `sampleDataService.deleteForFlow`, whose `DELETE FROM file … metadata->>'flowId'=?` had no index — on the large prod `file` table it seq-scans, blows `statement_timeout`, exhausts its 2 attempts and lands **permanently** in the failed set. The flow is then hidden from the UI list (which filters `!=DELETING`) but still counted by the active-flows quota (`getUsage` counts `status=ENABLED`), so Publish silently shows the "Purchase Extra Active Flows" dialog instead of publishing — this is what breaks the `webhook-should-return-response` e2e monitor. Stuck flows are functionally dead (`preDelete` disables the trigger before the failing delete), so forcing their rows away is safe. Fixes on `fix/flow-delete-sample-data-timeout`: a partial expression index `idx_file_sample_data_flow_id` on `file (type, (metadata->>'flowId'))`, plus `operationStatus != DELETING` in the active-flow counts so the quota stops depending on delete-job success.
+- **The flows list page (`automations/index.tsx`) fetches a capped window and sorts it client-side, so any new "sort by X" has to be pushed into SQL to be correct.** `use-automations-data.ts` calls `flowsApi.list`/`tablesApi.list` with `limit: 1000` (1500 for folder contents, one budget shared across *all* folders) and passes `cursor: undefined` at every call site; all merging, sorting and page-slicing then happens client-side in `automations/lib/utils.ts` (`mergeAndSortItems`, `buildTreeItems`, `buildFilteredTreeItems`, plus a second folder-children sort inside `buildFilteredTreeItems` that is easy to miss). Because the window is capped, re-sorting it in the browser only reorders a *recency-biased slice* — a rarely-touched flow at row 1001 by `updated` never reaches the client, so it can never appear under A–Z. Ordering therefore has to happen before the `LIMIT`, in Postgres.
+- **The cursor `Paginator` can be made to order by a joined column without denormalizing anything — `addOrderBy` registered on the query builder *before* `paginate()` leads.** `appendPagingQuery` does `new SelectQueryBuilder(builder)` and then only ever calls `addOrderBy`, so a caller's own criteria comes first and the paginator's `status/updated/id` become the tiebreak chain. That means `queryBuilder.addSelect('LOWER(COALESCE(latest_version."displayName", ''))', alias).addOrderBy(alias, order)` sorts flows by name with **no new column, no migration, no index, no paginator change** — despite `OrderByConfig.field` itself being unable to target a join (`buildOrder` and `buildCursorQuery` both hard-prefix `${this.alias}.`). Three things to respect: (1) the keyset `WHERE` is still built from `status/updated/id`, so this is only sound while no cursor is passed — reject `sortBy` + `cursor` with a 400 and return `createPage(data, null)` so no meaningless `next` is minted; (2) the sort alias must be **all-lowercase**, because TypeORM's DISTINCT path re-emits the order criteria *unquoted* in its second `id IN (...)` query, so a mixed-case alias fails only on that second query; (3) that same DISTINCT path (`SelectQueryBuilder` clears the inner ORDER BY and nulls the inner limit) means **no index on `flow` can serve this ORDER BY** — which is exactly why denormalizing a sort column onto `FlowEntity` buys nothing here.
 - **Any query that reads `flow_version.trigger` across a whole platform is TOAST-bound, not index-bound.** Trigger blobs are jsonb and TOAST'd on rows past ~2 KB, so a report that walks piece steps across all published flows on a platform pays ~2–5 ms of TOAST fetch + JSON parse per flow no matter how tight the WHERE clause is — a 10k-published-flow platform lands at 30 s–1 min. Endpoints of this shape (`platform/pieces-report/pieces-report.controller.ts` is the current example) must page + stream (`Readable.from(async iterable)`) so memory is bounded even when wall clock isn't; the escape hatch for the biggest platforms is a background job with async delivery, unlocked when a real sync request times out. Postgres jsonpath in-DB is not a shortcut — it still reads the whole TOASTed blob and the branch schema (`nextAction`/`children.*`/`onSuccessAction`/`firstLoopAction`/router children) drifts every time the flow shape changes, which is what `flowStructureUtil.getAllSteps` is authoritative over.
 - **`transferFlow` already deep-clones the whole flow — a callback that clones `step` again is quadratic.** `flowStructureUtil.transferFlow` opens with `JSON.parse(JSON.stringify(flowVersion))`, so the callback is handed a private copy and can mutate in place. Cloning per step instead is O(N²), because a `step` carries `nextAction` (the entire rest of the chain) plus loop/router children: cloning step *i* copies the remaining `N-i` steps. Measured on prod app containers (CDP CPU profile, 2026-08-21): the callback at `flow-version.service.ts` was **42% of wall-clock / ~84% of non-idle CPU**, at `transferStep` recursion depth 255 ≈ 32k step serializations per call, plus the GC churn behind ~2.5 GB RSS. It ran on **every** `getFlowVersionOrThrow` — including the default `removeConnectionsName=false, removeSampleData=false`, where the callback does nothing but the cloning still happens. Symptom was containers pegged at their `cpus: 1` cap and the 5s healthcheck `curl` timing out, which reads as "app unhealthy" with nothing crashed (the leftover zombie `curl`s are those killed healthchecks). Same pattern at `ee/…/project-state/diff/flow-diff.service.ts` (colder path, untouched here). When you write a `transferFlow` callback, mutate and return `step` — don't re-clone it.
 

packages/core/execution/src/lib/flows/dto/list-flows-request.ts

@@ -16,6 +16,8 @@ export const ListFlowsRequest = z.object({
     versionState: z.nativeEnum(FlowVersionState).optional(),
     connectionExternalIds: OptionalArrayFromQuery(z.string()),
     externalIds: OptionalArrayFromQuery(z.string()),
+    sortBy: z.enum(['NAME']).optional(),
+    order: z.enum(['ASC', 'DESC']).optional(),
 })
 
 export type ListFlowsRequest = Omit<z.infer<typeof ListFlowsRequest>, 'cursor'> & { cursor: Cursor | undefined }

packages/core/shared/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@activepieces/shared",
-  "version": "0.157.0",
+  "version": "0.158.0",
   "type": "commonjs",
   "sideEffects": false,
   "main": "./dist/src/index.js",

packages/core/shared/src/lib/automation/tables/dto/tables.dto.ts

@@ -50,6 +50,8 @@ export const ListTablesRequest = z.object({
     externalIds: OptionalArrayFromQuery(z.string()),
     folderId: z.string().optional(),
     folderIds: OptionalArrayFromQuery(z.string()),
+    sortBy: z.enum(['NAME']).optional(),
+    order: z.enum(['ASC', 'DESC']).optional(),
 })
 
 export type ListTablesRequest = z.infer<typeof ListTablesRequest>

packages/pieces/community/bettermode/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@activepieces/piece-bettermode",
-  "version": "0.1.8",
+  "version": "0.1.9",
   "main": "./dist/src/index.js",
   "types": "./dist/src/index.d.ts",
   "scripts": {

packages/server/api/src/app/flows/flow/flow.controller.ts

@@ -111,6 +111,8 @@ export const flowController: FastifyPluginAsyncZod = async (app) => {
             externalIds: request.query.externalIds,
             connectionExternalIds: request.query.connectionExternalIds,
             agentExternalIds: request.query.agentExternalIds,
+            sortBy: request.query.sortBy,
+            order: request.query.order,
         })
     })
 

packages/server/api/src/app/flows/flow/flow.service.ts

@@ -8,7 +8,7 @@ import { transaction } from '../../core/db/transaction'
 import { distributedLock } from '../../database/redis-connections'
 import { buildPaginator } from '../../helper/pagination/build-paginator'
 import { paginationHelper } from '../../helper/pagination/pagination-utils'
-import Paginator, { Order } from '../../helper/pagination/paginator'
+import Paginator, { CURSOR_SELECT_PREFIX, Order } from '../../helper/pagination/paginator'
 import { rejectedPromiseHandler } from '../../helper/promise-handler'
 import { system } from '../../helper/system/system'
 import { AppSystemProp } from '../../helper/system/system-props'
@@ -99,7 +99,10 @@ export const flowService = (log: FastifyBaseLogger) => ({
         externalIds,
         versionState = FlowVersionState.DRAFT,
         includeTriggerSource = true,
+        sortBy,
+        order,
     }: ListParams): Promise<SeekPage<PopulatedFlow>> {
+        assertSortIsNotCombinedWithCursor({ sortBy, cursor: cursorRequest })
         const decodedCursor = paginationHelper.decodeCursor(cursorRequest)
         const paginator = buildPaginator({
             entity: FlowEntity,
@@ -192,6 +195,13 @@ export const flowService = (log: FastifyBaseLogger) => ({
             queryBuilder.andWhere('latest_version."agentIds" && :agentExternalIds', { agentExternalIds })
         }
 
+        if (!isNil(sortBy)) {
+            const nameSortAlias = `${CURSOR_SELECT_PREFIX}name`
+            const nameSortColumn = versionState === FlowVersionState.DRAFT ? 'latest_version."displayName"' : 'published_version."displayName"'
+            queryBuilder.addSelect(`LOWER(COALESCE(${nameSortColumn}, ''))`, nameSortAlias)
+            queryBuilder.addOrderBy(nameSortAlias, order ?? 'ASC')
+        }
+
         const paginationResult = await paginator.paginate<Flow & { version: FlowVersion | null, triggerSource?: TriggerSource }>(queryBuilder)
 
         const populatedFlows = await Promise.all(paginationResult.data.map(async (flow) => {
@@ -215,7 +225,7 @@ export const flowService = (log: FastifyBaseLogger) => ({
                     : undefined,
             }
         }))
-        return paginationHelper.createPage(populatedFlows, paginationResult.cursor)
+        return paginationHelper.createPage(populatedFlows, isNil(sortBy) ? paginationResult.cursor : null)
     },
     async exists(id: FlowId): Promise<boolean> {
         return flowRepo().existsBy({
@@ -809,6 +819,16 @@ const assertFlowIsNotNull: <T extends Flow>(
     }
 }
 
+function assertSortIsNotCombinedWithCursor({ sortBy, cursor }: { sortBy: 'NAME' | undefined, cursor: Cursor | undefined | null }): void {
+    if (isNil(sortBy) || isNil(cursor)) {
+        return
+    }
+    throw new ActivepiecesError({
+        code: ErrorCode.GENERIC_ERROR,
+        params: { message: 'sortBy cannot be combined with cursor' },
+    })
+}
+
 async function assertExternalIdIsUnique({ projectId, externalId }: { projectId: ProjectId, externalId: string | undefined }): Promise<void> {
     if (isNil(externalId)) {
         return
@@ -843,6 +863,8 @@ type ListParamsBase = {
     connectionExternalIds?: string[]
     agentExternalIds?: string[]
     includeTriggerSource?: boolean
+    sortBy?: 'NAME'
+    order?: 'ASC' | 'DESC'
 }
 
 type ListParams = ListParamsBase & (

packages/server/api/src/app/helper/pagination/paginator.ts

@@ -298,7 +298,7 @@ function withIdTiebreaker(orderByConfig: OrderByConfig[]): OrderByConfig[] {
     ]
 }
 
-const CURSOR_SELECT_PREFIX = 'ap_cursor_'
+export const CURSOR_SELECT_PREFIX = 'ap_cursor_'
 
 const TIMESTAMP_TEXT_PATTERN = /^\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}(\.\d+)?([+-]\d{2}(:?\d{2})?|Z)?$/
 

packages/server/api/src/app/tables/table/table.controller.ts

@@ -41,6 +41,8 @@ export const tablesController: FastifyPluginAsyncZod = async (fastify) => {
             externalIds: request.query.externalIds,
             folderId: request.query.folderId,
             folderIds: request.query.folderIds,
+            sortBy: request.query.sortBy,
+            order: request.query.order,
         })
     })
 

packages/server/api/src/app/tables/table/table.service.ts

@@ -10,7 +10,7 @@ import { enforceByteLimit, filesService } from '../../file/files-service'
 import { getFolderIdFromRequest } from '../../flows/flow/flow.service'
 import { buildPaginator } from '../../helper/pagination/build-paginator'
 import { paginationHelper } from '../../helper/pagination/pagination-utils'
-import { Order } from '../../helper/pagination/paginator'
+import { CURSOR_SELECT_PREFIX, Order } from '../../helper/pagination/paginator'
 import { system } from '../../helper/system/system'
 import { AppSystemProp } from '../../helper/system/system-props'
 import { fieldService } from '../field/field.service'
@@ -46,7 +46,8 @@ export const tableService = {
         }
         return table
     },
-    async list({ projectId, cursor, limit, name, externalIds, folderId, folderIds, includeRowCount }: ListParams): Promise<SeekPage<Table & { rowCount?: number }>> {
+    async list({ projectId, cursor, limit, name, externalIds, folderId, folderIds, includeRowCount, sortBy, order }: ListParams): Promise<SeekPage<Table & { rowCount?: number }>> {
+        assertSortIsNotCombinedWithCursor({ sortBy, cursor })
         const decodedCursor = paginationHelper.decodeCursor(cursor ?? null)
 
         const paginator = buildPaginator({
@@ -85,9 +86,15 @@ export const tableService = {
             }, 'rowCount')
         }
 
+        if (!isNil(sortBy)) {
+            const nameSortAlias = `${CURSOR_SELECT_PREFIX}name`
+            queryBuilder.addSelect('LOWER(COALESCE("table"."name", \'\'))', nameSortAlias)
+            queryBuilder.addOrderBy(nameSortAlias, order ?? 'ASC')
+        }
+
         const paginationResult = await paginator.paginate<Table & { rowCount?: number }>(queryBuilder)
 
-        return paginationHelper.createPage(paginationResult.data, paginationResult.cursor)
+        return paginationHelper.createPage(paginationResult.data, isNil(sortBy) ? paginationResult.cursor : null)
     },
 
     async getOneOrThrow({
@@ -357,6 +364,16 @@ const EXPORT_BATCH_SIZE = 2000
 
 const CELL_EDGE_CHARS = /^[\s\u0000-\u001F\u007F-\u009F]+|[\s\u0000-\u001F\u007F-\u009F]+$/g
 
+function assertSortIsNotCombinedWithCursor({ sortBy, cursor }: { sortBy: 'NAME' | undefined, cursor: string | undefined }): void {
+    if (isNil(sortBy) || isNil(cursor)) {
+        return
+    }
+    throw new ActivepiecesError({
+        code: ErrorCode.GENERIC_ERROR,
+        params: { message: 'sortBy cannot be combined with cursor' },
+    })
+}
+
 async function fetchRecordPage({ tableId, projectId, afterCursor }: FetchRecordPageParams): Promise<FetchRecordPageResult> {
     const paginator = buildPaginator({
         entity: RecordEntity,
@@ -421,6 +438,8 @@ type ListParams = {
     folderId: string | undefined
     folderIds?: string[] | undefined
     includeRowCount?: boolean
+    sortBy?: 'NAME'
+    order?: 'ASC' | 'DESC'
 }
 
 type GetByIdParams = {