Skip to content

Commit f11efb1

Browse files
Neiko2002Neiko2002
committed
refactor(namepsace): clean the namespaces to have a clean high level one
1 parent d00aa3a commit f11efb1

50 files changed

Lines changed: 679 additions & 467 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎cpp/ARCHITECTURE.md‎

Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
# deglib: C++ Architecture & Namespace Specification
2+
3+
This document specifies the target directory organization, namespace hierarchy, and header structure of the `deglib` C++ header-only library.
4+
5+
---
6+
7+
## Directory & Header Overview
8+
9+
All header files are located in `cpp/deglib/include/deglib/`:
10+
11+
```
12+
deglib/
13+
├── deglib.h # Main library umbrella include header
14+
├── graph.h # Public DynamicExplorationGraph user facade
15+
├── builder.h # EvenRegularGraphBuilder & build status types
16+
├── optimization.h # Graph topology optimization algorithms
17+
├── analysis.h # Graph diagnostics, quality metrics & reachability
18+
├── distances.h # Feature spaces, metric enums & SIMD distance wrappers
19+
├── filter.h # ANNS search label filtering
20+
├── search.h # ANNS range search algorithms
21+
├── concurrent.h # Multi-threading & concurrency utilities
22+
├── config.h # Feature flags & compilation configuration
23+
│
24+
├── graph/ # Internal graph hierarchy (0..N-1 indexed)
25+
│ ├── internal_graph.h # Abstract base interface for internal graphs
26+
│ ├── mutable_graph.h # Abstract base interface for mutable graphs
27+
│ ├── sizebounded_graph.h # Concrete preallocated size-bounded graph
28+
│ ├── readonly_graph.h # Concrete compact read-only search graph
29+
│ ├── visited_list_pool.h # Thread-safe visited list pool for ANNS
30+
│ └── dynamic_graph.h # Graph interface declarations
31+
│
32+
├── optimization/ # Topology optimization kernels
33+
│ ├── pruning.h # Edge pruning algorithms
34+
│ ├── flas/ # Fast Local Anisotropic Search
35+
│ └── quantization/ # EVP quantization kernels
36+
│
37+
├── distance/ # SIMD distance computation kernels
38+
│ ├── fp32_l2.h # Single-precision L2 distance
39+
│ ├── fp32_ip.h # Single-precision Inner Product distance
40+
│ ├── evp_inner_product.h # EVP Quantized Inner Product distance
41+
│ ├── fp16_ip.h # FP16 Half-precision Inner Product distance
42+
│ ├── uint8_l2.h # Uint8 L2 distance
43+
│ ├── fp32.h / fp16.h / uint8.h # Vector type distance helpers
44+
│ └── residual_mode.h # Quantization residual modes
45+
│
46+
└── utils/ # SIMD, Memory & Random utilities
47+
├── cpu.h # CPU feature detection & InstructionSet enum
48+
├── memory.h # Aligned memory allocations & prefetching
49+
└── random.h # Randomization helpers
50+
```
51+
52+
---
53+
54+
## Target Namespace Structure & API Reference
55+
56+
```
57+
deglib:: # Root namespace: DynamicExplorationGraph facade
58+
├── graph:: # Internal graph hierarchy (InternalGraph, SizeBoundedGraph, ReadOnlyGraph, ...)
59+
├── builder:: # Graph construction (EvenRegularGraphBuilder, OptimizationTarget, BuilderStatus)
60+
├── search:: # Search utilities and label filtering (Filter, range search)
61+
├── distances:: # Vector space & SIMD metrics (FloatSpace, Metric, SIMD kernels)
62+
├── cpu:: # CPU capabilities & InstructionSet enum (InstructionSet, has_avx2, has_avx512)
63+
├── optimization:: # Topology refinement (prune_worst_edges, remove_non_mrng_edges)
64+
└── analysis:: # Graph quality diagnostics & reachability (analyze_graph, GraphStats)
65+
```
66+
67+
---
68+
69+
### 1. Root Namespace: `deglib::`
70+
71+
Contains the primary high-level user facade for ANNS search & exploration.
72+
73+
#### Class `deglib::DynamicExplorationGraph`
74+
End-user facade translating external labels (User IDs) to internal indices (`0..N-1`).
75+
- `explicit DynamicExplorationGraph(deglib::graph::InternalGraph& graph)`
76+
- `uint32_t size() const`: Returns total vertex count.
77+
- `uint8_t getEdgesPerVertex() const`: Returns configured number of edges per vertex.
78+
- `const deglib::distances::FloatSpace& getFeatureSpace() const`: Returns the graph feature space.
79+
- `bool hasVertex(uint32_t external_label) const`: Checks if external label exists.
80+
- `deglib::graph::InternalGraph& internal()`: Direct access to the underlying internal graph.
81+
- `const deglib::graph::InternalGraph& internal() const`
82+
- `bool isMutable() const`: Returns `true` if graph supports vertex/edge modifications.
83+
- `ResultSet search(span<const T> query, uint32_t k, float eps=0.0f, const deglib::search::Filter* filter=nullptr, uint32_t max_dc=0) const`: Approximate nearest neighbor search.
84+
- `ResultSet explore(uint32_t entry_external_label, uint32_t k, uint32_t max_dc=0, float eps=0.0f, bool include_entry=true, const deglib::search::Filter* filter=nullptr) const`: Exploration from entry vertex label.
85+
- `std::vector<uint32_t> getNeighbors(uint32_t external_label) const`: Returns external labels of neighbor vertices.
86+
- `DynamicExplorationGraph to_readonly() const`: Converts graph to a compact read-only instance.
87+
- `bool saveGraph(const std::string& path) const`: Saves the graph to disk if mutable.
88+
89+
---
90+
91+
### 2. Namespace: `deglib::distances::`
92+
93+
Feature space definitions, metric enums, and SIMD distance kernels.
94+
95+
#### Enums & Classes
96+
- `enum class Metric`: Distance metrics (`FP32_L2`, `FP32_InnerProduct`, `Uint8_L2`, `FP16_InnerProduct`, `EVP_InnerProduct`).
97+
- `class FloatSpace`: Feature space definition and SIMD distance calculator.
98+
99+
#### Distance Kernels
100+
- `deglib::distances::fp32_l2`: Single-precision L2 distance routines.
101+
- `deglib::distances::fp32_ip`: FP32 Inner Product distance routines.
102+
- `deglib::distances::evp_ip`: EVP Quantized Inner Product distance routines.
103+
- `deglib::distances::fp16_ip`: FP16 Half-Precision Inner Product distance routines.
104+
- `deglib::distances::uint8_l2`: Uint8 L2 distance routines.
105+
106+
---
107+
108+
### 3. Namespace: `deglib::cpu::`
109+
110+
CPU feature detection and SIMD instruction set definitions.
111+
112+
#### Enums & Functions
113+
- `enum class InstructionSet`: SIMD instruction set enum (`Auto`, `Scalar`, `AVX2`, `AVX512`).
114+
- `bool has_avx2()`: Checks runtime AVX2 availability.
115+
- `bool has_avx512()`: Checks runtime AVX512 availability.
116+
117+
---
118+
119+
### 4. Namespace: `deglib::search::`
120+
121+
Search algorithms and search filtering objects.
122+
123+
#### Classes
124+
- `class Filter`: Boolean label filter for restricting search result sets during search and explore operations.
125+
126+
---
127+
128+
### 5. Namespace: `deglib::graph::`
129+
130+
Low-level internal graph hierarchy operating directly on internal indices (`0..N-1`) for memory bandwidth performance.
131+
132+
#### Classes
133+
- `class InternalGraph`: Abstract base class for internal 0..N-1 indexed graphs.
134+
- `class MutableGraph : public InternalGraph`: Abstract base class for mutable internal graphs.
135+
- `class SizeBoundedGraph : public MutableGraph`: Fixed-capacity, preallocated graph supporting dynamic vertex/edge mutations.
136+
- `class ReadOnlyGraph : public InternalGraph`: Compact read-only graph optimized for search performance.
137+
- `class VisitedListPool`: Thread-safe pool managing visited bitsets during search.
138+
139+
---
140+
141+
### 6. Namespace: `deglib::builder::`
142+
143+
Graph construction and optimization algorithms.
144+
145+
#### Classes & Types
146+
- `enum class OptimizationTarget`: Build optimization strategy (`StreamingData`, `HighLID`, `LowLID`).
147+
- `struct BuilderStatus`: Status object passed to build callbacks (`step`, `added`, `deleted`, `improved`, `tries`).
148+
- `class EvenRegularGraphBuilder`:
149+
- `EvenRegularGraphBuilder(deglib::graph::MutableGraph& graph, std::mt19937& rnd, OptimizationTarget optimization_target = OptimizationTarget::LowLID, uint8_t extend_k = 0, float extend_eps = 0.1f, uint8_t improve_k = 0, float improve_eps = 0.001f, uint8_t max_path_length = 5, uint32_t swap_tries = 0, uint32_t additional_swap_tries = 0)`
150+
- `void addEntry(uint32_t label, std::vector<std::byte> feature)`: Queues entry addition.
151+
- `void removeEntry(uint32_t label)`: Queues entry removal.
152+
- `uint32_t getNumNewEntries() const`: Queued addition count.
153+
- `uint32_t getNumRemoveEntries() const`: Queued removal count.
154+
- `void setThreadCount(size_t thread_count)`: Sets parallel build thread count.
155+
- `void setBatchSize(uint32_t tasks_per_batch, uint32_t task_size)`: Sets multithreading batch parameters.
156+
- `void build(std::function<void(BuilderStatus&)> callback, bool infinite = false)`: Runs graph construction.
157+
- `void stop()`: Requests build termination.
158+
159+
---
160+
161+
### 7. Namespace: `deglib::optimization::`
162+
163+
Graph topology refinement algorithms.
164+
165+
#### Functions
166+
- `void prune_worst_edges(deglib::graph::MutableGraph& graph, const uint8_t prune_worst, const size_t numThreads = 0)`: Prunes worst edges per vertex.
167+
- `void remove_non_mrng_edges(deglib::graph::MutableGraph& graph, const size_t numThreads = 0)`: Removes non-MRNG edges.
168+
169+
---
170+
171+
### 8. Namespace: `deglib::analysis::`
172+
173+
Graph diagnostic metrics, connectivity validation, and reachability.
174+
175+
#### Struct `GraphStats`
176+
Attributes: `vertex_count`, `edge_count`, `feature_dims`, `edges_per_vertex`, `avg_out_degree`, `min_out_degree`, `max_out_degree`, `avg_in_degree`, `min_in_degree`, `max_in_degree`, `source_vertices`, `search_reachability`, `exploration_reachability`, `memory_bytes`.
177+
178+
#### Functions
179+
- `GraphStats analyze_graph(const deglib::graph::InternalGraph& graph)`
180+
- `float calc_avg_edge_weight(const deglib::graph::MutableGraph& graph, const int scale = 1)`
181+
- `std::vector<float> calc_edge_weight_histogram(const deglib::graph::MutableGraph& graph, const bool sorted, const int scale = 1)`
182+
- `bool check_graph_weights(const deglib::graph::MutableGraph& graph)`
183+
- `bool check_graph_regularity(const deglib::graph::InternalGraph& graph, const uint32_t expected_vertices, const bool check_back_link = false)`
184+
- `bool check_graph_connectivity(const deglib::graph::InternalGraph& graph)`
185+
- `uint32_t calc_non_rng_edges(const deglib::graph::MutableGraph& graph)`
186+
- `float calc_search_reachability(const deglib::graph::InternalGraph& graph)`
187+
- `float calc_exploration_reach(const deglib::graph::InternalGraph& graph)`

‎cpp/bench/include/benchmark.h‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -30,12 +30,12 @@ namespace deglib::benchmark {
3030
* @brief Compute baseline time per query using linear distance computation.
3131
*/
3232
static uint64_t compute_linear_search_baseline(const deglib::FeatureRepository& base_repository,
33-
const deglib::Metric metric,
33+
const deglib::distances::Metric metric,
3434
const uint32_t sample_size = 100,
35-
const deglib::cpu::InstructionSet instruction = deglib::cpu::InstructionSet::Auto) {
35+
const deglib::distances::InstructionSet instruction = deglib::distances::InstructionSet::Auto) {
3636
const auto dims = base_repository.dims();
3737
const auto base_size = base_repository.size();
38-
const auto feature_space = deglib::FloatSpace(dims, metric, instruction);
38+
const auto feature_space = deglib::distances::FloatSpace(dims, metric, instruction);
3939
const auto dist_func = feature_space.get_dist_func();
4040
const auto dist_func_param = feature_space.get_dist_func_param();
4141

@@ -83,7 +83,7 @@ static float test_approx_anns(const deglib::graph::InternalGraph& graph,
8383
const uint32_t k,
8484
const uint32_t test_size,
8585
const uint32_t threads,
86-
const deglib::graph::Filter* filter = nullptr) {
86+
const deglib::search::Filter* filter = nullptr) {
8787
auto corrects = std::vector<float>(threads);
8888
deglib::concurrent::parallel_for(0, test_size, threads, [&](size_t i, size_t thread_id) {
8989
auto query = reinterpret_cast<const float*>(query_repository.getFeature(uint32_t(i)));
@@ -118,7 +118,7 @@ static float test_approx_explore(const deglib::graph::InternalGraph& graph,
118118
const uint32_t k,
119119
const uint32_t max_distance_count,
120120
const uint32_t threads,
121-
const deglib::graph::Filter* filter = nullptr) {
121+
const deglib::search::Filter* filter = nullptr) {
122122
auto corrects = std::vector<float>(threads);
123123
deglib::concurrent::parallel_for(0, entry_vertex_indices.size(), threads, [&](size_t i, size_t thread_id) {
124124
const auto entry_vertex_index = entry_vertex_indices[i][0];
@@ -197,7 +197,7 @@ static void test_graph_anns(const deglib::graph::InternalGraph& graph,
197197
const uint32_t threads,
198198
const uint32_t k,
199199
const std::vector<float>& eps_parameter,
200-
const deglib::graph::Filter* filter = nullptr,
200+
const deglib::search::Filter* filter = nullptr,
201201
const uint64_t linear_baseline_us = 0,
202202
const uint32_t abort_sample_size = 100) {
203203
const auto entry_vertex_indices = graph.getEntryVertexIndices();
@@ -260,7 +260,7 @@ static void test_graph_explore(const deglib::graph::InternalGraph& graph,
260260
const uint32_t repeat,
261261
const uint32_t k,
262262
const uint32_t threads,
263-
const deglib::graph::Filter* filter = nullptr,
263+
const deglib::search::Filter* filter = nullptr,
264264
const uint32_t explore_depth = 3,
265265
const uint64_t linear_baseline_us = 0,
266266
const uint32_t abort_sample_size = 100,
@@ -342,3 +342,4 @@ static void test_graph_explore(const deglib::graph::InternalGraph& graph,
342342
}
343343

344344
} // namespace deglib::benchmark
345+

‎cpp/bench/include/build.h‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -24,15 +24,15 @@ namespace deglib::benchmark {
2424
enum DataStreamType { AddAll, AddHalf, AddAllRemoveHalf, AddHalfRemoveAndAddOneAtATime };
2525

2626
inline deglib::graph::SizeBoundedGraph create_random_graph(const deglib::StaticFeatureRepository& repository,
27-
deglib::Metric metric,
27+
deglib::distances::Metric metric,
2828
const uint8_t k,
2929
const uint32_t max_size = 0,
3030
const uint32_t scale = 1,
31-
const deglib::cpu::InstructionSet instruction = deglib::cpu::InstructionSet::Auto) {
31+
const deglib::distances::InstructionSet instruction = deglib::distances::InstructionSet::Auto) {
3232
log("Build a random EG{}\n", k);
3333

3434
const auto dims = repository.dims();
35-
const auto feature_space = deglib::FloatSpace(dims, metric, instruction);
35+
const auto feature_space = deglib::distances::FloatSpace(dims, metric, instruction);
3636
const auto dist_func = feature_space.get_dist_func();
3737
const auto dist_func_param = feature_space.get_dist_func_param();
3838

@@ -115,7 +115,7 @@ inline deglib::graph::SizeBoundedGraph create_random_graph(const deglib::StaticF
115115
inline void create_graph(const deglib::StaticFeatureRepository& repository,
116116
const DataStreamType data_stream_type,
117117
const std::string& graph_file,
118-
deglib::Metric metric,
118+
deglib::distances::Metric metric,
119119
deglib::builder::OptimizationTarget lid,
120120
const uint8_t k,
121121
const uint8_t k_ext,
@@ -127,14 +127,14 @@ inline void create_graph(const deglib::StaticFeatureRepository& repository,
127127
const bool use_rng = true,
128128
const uint32_t scale = 1,
129129
const bool use_path_verification = false,
130-
const deglib::cpu::InstructionSet instruction = deglib::cpu::InstructionSet::Auto) {
130+
const deglib::distances::InstructionSet instruction = deglib::distances::InstructionSet::Auto) {
131131
auto rnd = std::mt19937(7);
132132
const uint32_t swap_tries = 0;
133133
const uint32_t additional_swap_tries = 0;
134134

135135
const auto dims = repository.dims();
136136
const uint32_t max_vertex_count = uint32_t(repository.size());
137-
const auto feature_space = deglib::FloatSpace(dims, metric, instruction);
137+
const auto feature_space = deglib::distances::FloatSpace(dims, metric, instruction);
138138
const auto feature_byte_size = feature_space.get_data_size();
139139

140140
log("Initializing empty graph (capacity: {} vertices, {}D {} {} feature space using {})\n",

‎cpp/bench/include/dataset.h‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ inline constexpr DatasetName DatasetName::Invalid{"invalid"};
8282
struct DatasetInfo {
8383
DatasetName dataset_name;
8484
std::string download_url;
85-
deglib::Metric metric;
85+
deglib::distances::Metric metric;
8686
size_t base_count;
8787
size_t query_count;
8888
uint32_t dims;
@@ -105,7 +105,7 @@ struct DatasetInfo {
105105
};
106106

107107
inline DatasetInfo make_dataset_info(const DatasetName& ds) {
108-
DatasetInfo info{ds, {}, deglib::Metric::L2, 0, 0, 0, 1, 2, {}, {}, {}, {}, {}, {}};
108+
DatasetInfo info{ds, {}, deglib::distances::Metric::L2, 0, 0, 0, 1, 2, {}, {}, {}, {}, {}, {}};
109109

110110
std::string name = ds.name();
111111

@@ -268,15 +268,15 @@ class Dataset {
268268

269269
inline std::vector<uint32_t> compute_knn_groundtruth(const deglib::FeatureRepository& base_repo,
270270
const deglib::FeatureRepository& query_repo,
271-
const deglib::Metric metric,
271+
const deglib::distances::Metric metric,
272272
const uint32_t k_target,
273273
const size_t base_limit = 0,
274274
const uint32_t thread_count = 1) {
275275
const auto base_size = base_limit > 0 ? (uint32_t)std::min(base_limit, base_repo.size()) : (uint32_t)base_repo.size();
276276
const auto query_size = (uint32_t)query_repo.size();
277277
const auto dims = base_repo.dims();
278278

279-
const auto feature_space = deglib::FloatSpace((uint32_t)dims, metric);
279+
const auto feature_space = deglib::distances::FloatSpace((uint32_t)dims, metric);
280280
const auto dist_func = feature_space.get_dist_func();
281281
const auto dist_func_param = feature_space.get_dist_func_param();
282282

0 commit comments

Comments
 (0)