Commit 1669976
fix: force drop index and use DISTINCT ON for robust deduplication
Previous approach failed because:
1. All snapshot_date values were NULL, so IS NOT NULL check matched 0 rows
2. Existing index prevents deduplication from completing
New strategy:
1. DROP INDEX IF EXISTS to allow deduplication without constraint violations
2. DELETE all rows with NULL snapshot_date (should not exist post-migration)
3. Use PostgreSQL DISTINCT ON for efficient deduplication:
- Keeps row with MAX(snapshot_at) for each (campaign_pid, snapshot_date) pair
- More efficient than self-join USING pattern
4. Let AutoMigrate recreate the index on clean data
Logs showed "deduplicated 0 rows" because NULL != NULL in SQL.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>1 parent 45a7094 commit 1669976
1 file changed
Lines changed: 19 additions & 12 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
177 | 177 | | |
178 | 178 | | |
179 | 179 | | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
180 | 188 | | |
181 | | - | |
182 | | - | |
183 | | - | |
184 | | - | |
185 | | - | |
186 | | - | |
187 | | - | |
188 | | - | |
189 | | - | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
190 | 197 | | |
191 | | - | |
192 | | - | |
| 198 | + | |
| 199 | + | |
193 | 200 | | |
194 | | - | |
| 201 | + | |
195 | 202 | | |
196 | 203 | | |
197 | 204 | | |
| |||
0 commit comments