Repository navigation
Commit e0c5fa9
committed
perf(knowledge): reconcile a completed listing in one pass over its windows
The reconciliation count read every window of a connector, then the ACL,
soft-delete and hard-delete walks each read the same windows again. The tally
now counts every decision input and keeps the rows each removal acts on in the
same window statement, so removals page through those rows instead of walking
the connector again. A condition matching more than 10,000 documents keeps only
its count and falls back to the existing walk, so memory and each statement's
result stay bounded.1 parent 8948b54 commit e0c5fa9
9 files changed
Lines changed: 603 additions & 191 deletions
File tree
- apps/sim/lib/knowledge
- __integration__
- connectors
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
966 | 966 | | |
967 | 967 | | |
968 | 968 | | |
969 | | - | |
970 | | - | |
| 969 | + | |
| 970 | + | |
971 | 971 | | |
972 | 972 | | |
973 | 973 | | |
| |||
1002 | 1002 | | |
1003 | 1003 | | |
1004 | 1004 | | |
1005 | | - | |
1006 | | - | |
| 1005 | + | |
| 1006 | + | |
1007 | 1007 | | |
1008 | | - | |
| 1008 | + | |
1009 | 1009 | | |
1010 | 1010 | | |
1011 | 1011 | | |
| |||
1042 | 1042 | | |
1043 | 1043 | | |
1044 | 1044 | | |
1045 | | - | |
| 1045 | + | |
1046 | 1046 | | |
1047 | | - | |
1048 | | - | |
| 1047 | + | |
| 1048 | + | |
1049 | 1049 | | |
1050 | 1050 | | |
1051 | 1051 | | |
| |||
1069 | 1069 | | |
1070 | 1070 | | |
1071 | 1071 | | |
1072 | | - | |
| 1072 | + | |
1073 | 1073 | | |
1074 | 1074 | | |
1075 | 1075 | | |
| |||
1082 | 1082 | | |
1083 | 1083 | | |
1084 | 1084 | | |
1085 | | - | |
1086 | | - | |
1087 | | - | |
| 1085 | + | |
| 1086 | + | |
| 1087 | + | |
| 1088 | + | |
| 1089 | + | |
| 1090 | + | |
1088 | 1091 | | |
1089 | 1092 | | |
1090 | 1093 | | |
1091 | 1094 | | |
1092 | | - | |
| 1095 | + | |
1093 | 1096 | | |
1094 | 1097 | | |
1095 | 1098 | | |
| |||
1109 | 1112 | | |
1110 | 1113 | | |
1111 | 1114 | | |
1112 | | - | |
| 1115 | + | |
1113 | 1116 | | |
1114 | 1117 | | |
1115 | 1118 | | |
| |||
1119 | 1122 | | |
1120 | 1123 | | |
1121 | 1124 | | |
1122 | | - | |
1123 | | - | |
| 1125 | + | |
| 1126 | + | |
1124 | 1127 | | |
1125 | | - | |
| 1128 | + | |
| 1129 | + | |
1126 | 1130 | | |
1127 | 1131 | | |
1128 | 1132 | | |
1129 | | - | |
1130 | | - | |
| 1133 | + | |
1131 | 1134 | | |
1132 | 1135 | | |
1133 | 1136 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
131 | 131 | | |
132 | 132 | | |
133 | 133 | | |
134 | | - | |
135 | | - | |
136 | | - | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
137 | 137 | | |
138 | 138 | | |
139 | 139 | | |
| |||
278 | 278 | | |
279 | 279 | | |
280 | 280 | | |
281 | | - | |
282 | | - | |
283 | | - | |
| 281 | + | |
284 | 282 | | |
285 | 283 | | |
286 | 284 | | |
| |||
367 | 365 | | |
368 | 366 | | |
369 | 367 | | |
370 | | - | |
| 368 | + | |
371 | 369 | | |
372 | 370 | | |
373 | 371 | | |
| |||
855 | 853 | | |
856 | 854 | | |
857 | 855 | | |
858 | | - | |
| 856 | + | |
859 | 857 | | |
860 | 858 | | |
861 | 859 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
5 | | - | |
| 4 | + | |
| 5 | + | |
6 | 6 | | |
7 | | - | |
8 | | - | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
9 | 11 | | |
10 | | - | |
11 | | - | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
12 | 17 | | |
13 | 18 | | |
14 | | - | |
15 | | - | |
16 | | - | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
17 | 22 | | |
| 23 | + | |
18 | 24 | | |
19 | 25 | | |
20 | | - | |
21 | | - | |
22 | | - | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
23 | 31 | | |
24 | 32 | | |
25 | 33 | | |
| |||
43 | 51 | | |
44 | 52 | | |
45 | 53 | | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | 54 | | |
53 | | - | |
54 | | - | |
55 | | - | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
56 | 59 | | |
57 | | - | |
58 | | - | |
59 | | - | |
| 60 | + | |
| 61 | + | |
60 | 62 | | |
61 | | - | |
62 | | - | |
63 | | - | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
64 | 66 | | |
65 | 67 | | |
66 | 68 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
| 5 | + | |
6 | 6 | | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | | - | |
10 | | - | |
| 10 | + | |
| 11 | + | |
11 | 12 | | |
12 | 13 | | |
13 | | - | |
14 | | - | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
15 | 38 | | |
16 | 39 | | |
17 | 40 | | |
18 | | - | |
| 41 | + | |
19 | 42 | | |
20 | 43 | | |
21 | | - | |
22 | | - | |
23 | | - | |
24 | | - | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
25 | 48 | | |
26 | 49 | | |
27 | 50 | | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
| 51 | + | |
36 | 52 | | |
37 | 53 | | |
38 | | - | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
39 | 61 | | |
40 | 62 | | |
41 | 63 | | |
42 | | - | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
43 | 92 | | |
44 | 93 | | |
45 | | - | |
46 | | - | |
| 94 | + | |
| 95 | + | |
47 | 96 | | |
48 | 97 | | |
49 | 98 | | |
50 | 99 | | |
51 | 100 | | |
52 | 101 | | |
53 | | - | |
| 102 | + | |
54 | 103 | | |
55 | 104 | | |
56 | 105 | | |
0 commit comments