Performance: REST requests slow down sharply as contact and group counts grow #2941

Summary

On a blank subsite running only the theme (plus the network-activated disciple-tools-ai, disciple-tools-multisite and user-switching), page HTML stays fast at any data size: 35–76 ms on front-end pages. What slows down is the REST requests each page fires after loading. At 20k contacts / 10k groups, contact pages take 2.3–2.6 s to finish loading instead of about 0.5 s.

All of the slow paths found are in theme code.

How it was measured

• Local multisite with PHP 8.4 and MySQL 8.0 (2 PHP-FPM workers), and a fresh subsite.
• A temporary mu-plugin recorded each request's server time, query count and slowest queries (SAVEQUERIES on, so absolute times run a little high).
• Each page was loaded 3 times in a browser per stage; numbers are medians.
• Data was generated with DT_Posts (fields, connections, comments, updates):

Browser time until the page finished loading

Metrics and Settings take about 2.3 s at every size; that time is front-end JS, not data.

Slowest requests (server ms: s0 / s1 / s2 / s3)

1. Duplicate check on every contact record: 61 / 101 / 703 / 833 ms

GET dt-posts/v2/contacts/{id}/duplicates (dt-contacts/contacts.js:99) runs on every contact record view. DT_Duplicate_Checker_And_Merging::ids_of_non_dismissed_duplicates uses search_viewable_post to match phone and email against all contacts: a search query (390 ms) plus a separate count(distinct p.ID) query (377 ms).

Possible gains: run it on demand or cache the result per contact. Skip the count query, which isn't needed to list the duplicates. Match on an indexed value instead of LIKE over postmeta.

2. User management user list: 53 / 82 / 497 / 884 ms

user-management/v1/get-users spends 820 ms in a single query in DT_User_Management::get_users_paged, with only 21 users. The cost grows with the number of contacts, not users.

Possible gains: compute the per-user contact aggregates separately (or cache them), or restrict the joins to the users on the current page.

3. Filter counts on every list visit: 55 / 85 / 502 / 683 ms

The list page already renders cached filters, then modular-list.js:748 always requests dt/v1/users/get_filters?force_refresh=1, which recomputes:

• DT_Posts_Metrics::get_shared_with_meta_field_counts: 263 ms
• DT_Contacts_Access::get_all_contacts_status_seeker_path: 234 ms
• get_my_contacts_status_seeker_path: 38 ms

Possible gains: cache the counts for a short time (or invalidate them on post updates) instead of forcing a refresh on every visit.

4. Contacts list query and N+1 loading: 50 / 137 / 485 / 676 ms

dt-posts/v2/contacts/list costs:

• search_viewable_post: search 240 ms + count 182 ms (both join postmeta).
• 933 queries per request at s3, up from 333 at s1. Records are loaded one at a time: SELECT * FROM postmeta WHERE post_id = N ×100, get_post_status lookups ×90, get_post ×75, plus connection lookups that grow with the number of connections.

Possible gains: prime the meta and post caches for the page of IDs in one query (update_meta_cache, _prime_post_caches), load connections in bulk, and review postmeta indexes or a cheaper total count.

5. Groups list: 56 / 123 / 151 / 289 ms

Same pattern as contacts: search 84 ms + count 41 ms, 366 queries.

6. Personal genmap on group records: 61 / 64 / 88 / 173 ms

DT_Metrics_Groups_Genmap->get_query: 98 ms.

7. Every wp-admin page: 145 → 210–245 ms

WordPress core's admin menu calls wp_count_comments() on every admin page (33–54 ms at 60k comments). D.T doesn't use the Comments admin screen, so it could short-circuit the count with the wp_count_comments filter.

Also noted

• Creating records gets slower too: a group took about 100 ms to create with few groups and about 170 ms with 10k.
• With few PHP workers, a page that fires 3–4 of these requests at once waits on them in sequence, which multiplies the wait users see.

🤖 Generated with Claude Code