Skip to content

[RFC]: add a C benchmark harness #13480

Open

Description

@kgryte

Description

This RFC proposes creating a C benchmark harness which is similar to the project's current JavaScript benchmark harness.

Current Status

Currently, when we author C benchmarks, the benchmark files are self-contained apart from the API being tested. For example,

#include "stdlib/blas/ext/base/zindex_of_truthy.h"
#include "stdlib/complex/float64/ctor.h"
#include <stdlib.h>
#include <stdio.h>
#include <math.h>
#include <time.h>
#include <sys/time.h>

#define NAME "zindex_of_truthy"
#define ITERATIONS 10000000
#define REPEATS 3
#define MIN 1
#define MAX 6

/**
* Prints the TAP version.
*/
static void print_version( void ) {
	printf( "TAP version 13\n" );
}

/**
* Prints the TAP summary.
*
* @param total     total number of tests
* @param passing   total number of passing tests
*/
static void print_summary( int total, int passing ) {
	printf( "#\n" );
	printf( "1..%d\n", total ); // TAP plan
	printf( "# total %d\n", total );
	printf( "# pass  %d\n", passing );
	printf( "#\n" );
	printf( "# ok\n" );
}

/**
* Prints benchmarks results.
*
* @param iterations   number of iterations
* @param elapsed      elapsed time in seconds
*/
static void print_results( int iterations, double elapsed ) {
	double rate = (double)iterations / elapsed;
	printf( "  ---\n" );
	printf( "  iterations: %d\n", iterations );
	printf( "  elapsed: %0.9f\n", elapsed );
	printf( "  rate: %0.9f\n", rate );
	printf( "  ...\n" );
}

/**
* Returns a clock time.
*
* @return clock time
*/
static double tic( void ) {
	struct timeval now;
	gettimeofday( &now, NULL );
	return (double)now.tv_sec + (double)now.tv_usec/1.0e6;
}

/**
* Generates a random number on the interval [0,1).
*
* @return random number
*/
static double rand_double( void ) {
	int r = rand();
	return (double)r / ( (double)RAND_MAX + 1.0 );
}

/**
* Runs a benchmark.
*
* @param iterations   number of iterations
* @param len          array length
* @return             elapsed time in seconds
*/
static double benchmark1( int iterations, int len ) {
	double elapsed;
	double *x;
	double t;
	int idx;
	int i;

	x = (double *)malloc( (size_t)( len*2 ) * sizeof( double ) );
	for ( i = 0; i < len*2; i++ ) {
		x[ i ] = 0.0;
	}
	x[ (len*2)-1 ] = 1.0;
	idx = -1;
	t = tic();
	for ( i = 0; i < iterations; i++ ) {
		x[ (len*2)-4 ] = (double)( i % 4 );
		idx = stdlib_strided_zindex_of_truthy( len, (const stdlib_complex128_t *)x, 1 );
		if ( idx < 0 ) {
			printf( "unexpected result\n" );
			break;
		}
	}
	elapsed = tic() - t;
	if ( idx < 0 ) {
		printf( "unexpected result\n" );
	}
	free( x );
	return elapsed;
}

/**
* Runs a benchmark.
*
* @param iterations   number of iterations
* @param len          array length
* @return             elapsed time in seconds
*/
static double benchmark2( int iterations, int len ) {
	double elapsed;
	double *x;
	double t;
	int idx;
	int i;

	x = (double *)malloc( (size_t)( len*2 ) * sizeof( double ) );
	for ( i = 0; i < len*2; i++ ) {
		x[ i ] = 0.0;
	}
	x[ (len*2)-1 ] = 1.0;
	idx = -1;
	t = tic();
	for ( i = 0; i < iterations; i++ ) {
		x[ (len*2)-4 ] = (double)( i % 4 );
		idx = stdlib_strided_zindex_of_truthy_ndarray( len, (const stdlib_complex128_t *)x, 1, 0 );
		if ( idx < 0 ) {
			printf( "unexpected result\n" );
			break;
		}
	}
	elapsed = tic() - t;
	if ( idx < 0 ) {
		printf( "unexpected result\n" );
	}
	free( x );
	return elapsed;
}

/**
* Main execution sequence.
*/
int main( void ) {
	double elapsed;
	int count;
	int iter;
	int len;
	int i;
	int j;

	// Use the current time to seed the random number generator:
	srand( time( NULL ) );

	print_version();
	count = 0;
	for ( i = MIN; i <= MAX; i++ ) {
		len = pow( 10, i );
		iter = ITERATIONS / pow( 10, i-1 );
		for ( j = 0; j < REPEATS; j++ ) {
			count += 1;
			printf( "# c::%s:len=%d\n", NAME, len );
			elapsed = benchmark1( iter, len );
			print_results( iter, elapsed );
			printf( "ok %d benchmark finished\n", count );
		}
		for ( j = 0; j < REPEATS; j++ ) {
			count += 1;
			printf( "# c::%s:ndarray:len=%d\n", NAME, len );
			elapsed = benchmark2( iter, len );
			print_results( iter, elapsed );
			printf( "ok %d benchmark finished\n", count );
		}
	}
	print_summary( count, count );
}

In this benchmark, you'll notice that the file includes

  1. logic for writing TAP output.
  2. random value generation, including initial seeding.
  3. loop logic for managing the array length, number of iterations, and repeats.
  4. logic for starting and stopping the benchmark timers.
  5. imported headers to support 1-4.

In short, benchmark files include a fair amount of additional logic apart from the logic which forms the core benchmark. This is both a benefit--the benchmark files are fairly transparent and it is straightforward to deduce the runner logic--and a curse--the benchmark files are primarily comprised of scaffold code, leading to increased risk of drift, copy-paste mistakes, and increased maintenance overhead.

Proposal

This RFC proposes the creation of various C macros to remove the boilerplate C code and shrink the above benchmark by roughly half. Here is an example:

#include "stdlib/blas/ext/base/zindex_of_truthy.h"
#include "stdlib/complex/float64/ctor.h"
#include "stdlib/bench.h"

#ifndef NAME
#define NAME "zindex_of_truthy"
#endif

/**
* Runs a benchmark.
*
* @param iterations   number of iterations
* @param len          array length
* @return             elapsed time in seconds
*/
STDLIB_BENCHMARK( benchmark1 ) {
	int idx = -1;

	STDLIB_BENCHMARK_MALLOC_ARRAY_FLOAT64( x, len*2 );
	STDLIB_BENCHMARK_FILL_STRIDED_ARRAY_FLOAT64( x, len*2, 0.0 );
	x[ (len*2)-1 ] = 1.0;
	
	STDLIB_BENCHMARK_LOOP_PREAMBLE {
		x[ (len*2)-4 ] = (double)( i % 4 );
		idx = stdlib_strided_zindex_of_truthy( len, (const stdlib_complex128_t *)x, 1 );
		if ( idx < 0 ) {
			printf( "unexpected result\n" );
			break;
		}
	}
	STDLIB_BENCHMARK_LOOP_EPILOGUE;
	if ( idx < 0 ) {
		printf( "unexpected result\n" );
	}
	STDLIB_BENCHMARK_FREE( x );
	STDLIB_BENCHMARK_EPILOGUE;
}

/**
* Runs a benchmark.
*
* @param iterations   number of iterations
* @param len          array length
* @return             elapsed time in seconds
*/
STDLIB_BENCHMARK( benchmark2 ) {
	int idx = -1;

	STDLIB_BENCHMARK_MALLOC_ARRAY_FLOAT64( x, len*2 );
	STDLIB_BENCHMARK_FILL_STRIDED_ARRAY_FLOAT64( x, len*2, 0.0 );
	x[ (len*2)-1 ] = 1.0;
	
	STDLIB_BENCHMARK_LOOP_PREAMBLE {
		x[ (len*2)-4 ] = (double)( i % 4 );
		idx = stdlib_strided_zindex_of_truthy_ndarray( len, (const stdlib_complex128_t *)x, 1, 0 );
		if ( idx < 0 ) {
			printf( "unexpected result\n" );
			break;
		}
	}
	STDLIB_BENCHMARK_LOOP_EPILOGUE;
	if ( idx < 0 ) {
		printf( "unexpected result\n" );
	}
	STDLIB_BENCHMARK_FREE( x );
	STDLIB_BENCHMARK_EPILOGUE;
}

/**
* Main execution sequence.
*/
STDLIB_BENCH {
	int iterations = 10000000;
	int repeats = 3;
	int min = 1;
	int max = 6;

	STDLIB_BENCH_LENGTH_PREAMBLE( repeats, iterations, min, max ) {
		printf( "# c::%s:len=%d\n", NAME, len );
		STDLIB_RUN_BENCHMARK( benchmark1 );
	}
	STDLIB_BENCH_LENGTH_PREAMBLE( repeats, iterations, min, max ) {
		printf( "# c::%s:ndarray:len=%d\n", NAME, len );
		STDLIB_RUN_BENCHMARK( benchmark2 );
	}
	STDLIB_BENCH_EPILOGUE;
}

A few things to notice:

  1. Similar to elsewhere, such as ndarray/base/unary and other packages which make heavy use of C macros, the above code utilizes C macros for injecting boilerplate code.
  2. Moving to macros does increase the risk of the benchmarks feeling more "magical", as it is not always obvious what the code behind the macro is actually doing, including what variables it is creating. This can increase the risk of mysterious compilation errors, especially for newcomers to the project.
  3. Building on (2), the macros will expose certain variables (e.g., len and i above) and will operate similar to how STDLIB_NAPI_ARGV_ macros work where certain macros will support defining userland variables (e.g., x above in the STDLIB_BENCHMARK_MALLOC_ARRAY_FLOAT64 macro).
  4. There is still going to be custom setup (e.g., array allocation, initialization, etc) and teardown logic (e.g., STDLIB_BENCHMARK_FREE), as that is hard to standardize given that benchmarks should be fairly unique (i.e., APIs being benchmarked are typically different and have different signatures and input argument assumptions).
  5. The naming conventions (STDLIB_BENCH and STDLIB_BENCHMARK) were chosen to mirror our JavaScript naming conventions (e.g., var bench = require( '@stdlib/bench' ); and function benchmark(...) {...}).
  6. Not clear to me how we get around needing the custom printf( "# c::..." ); lines. Those are going to be unique to each benchmark and is just some more "magic" that folks would be expected to remember.
  7. I'd like to move away from hardcoding benchmark names (e.g., via the NAME macro). In JavaScript, we resolve the package name. I'd like that to be the same in C. Currently, we rely on explicitly defining a NAME macro in the benchmark source file. This is not great, as we vary in terms of convention, often using the package basename, but this runs the risk of name conflicts where other packages have the same basename. We use the package name in JavaScript benchmarks to ensure unique naming. It would be good to carry that forward to C benchmarks. In the code above, we still define a NAME macro, but as a backup when the -DNAME definition is not passed to the compiler by the build system. And in our make recipes and tooling for compiling and running benchmarks, we will update to resolve the corresponding package name and then explicitly set the corresponding compiler flag. This should be possible at the cost of a bit of extra compilation time due to need to resolve the nearest package name prior to compiling C benchmarks.
  8. Moving to stdlib/bench.h, a header file which is likely to pull in several other dependencies, such as macros utilizing BLAS functions for initializing arrays, PRNGs for random number generation, etc, it is likely that this will result in additional dependencies, and thus source files, being pulled in, resulting in longer compilation times, regardless of whether the benchmark file in question actually needs the transitive dependencies. There is a risk of scope creep for what gets included in stdlib/bench.h and that will bring hidden compilation costs to every downstream consumer within stdlib. UPDATE: one way around this is to compile a shared library and then link against the *.so file. This would require some updates to tooling and how we perform compilation.

Related Issues

None.

Questions

  • What are people's thoughts? Love it? Hate it?
  • What could we do differently? Are the C macros too magical?
  • What could we do instead? (code samples preferred)

Other

Prior Art

Checklist

  • I have read and understood the Code of Conduct.
  • Searched for existing issues and pull requests.
  • The issue name begins with RFC:.

Activity

  1. added
    RFCRequest for comments. Feature requests and proposed changes.
    BenchmarksPull requests adding or improving benchmarks for measuring performance.
    CIssue involves or relates to C.
    on Jul 15, 2026
  2. kgryte commented on Jul 15, 2026

    @kgryte
    MemberAuthor
  3. kgryte commented on Jul 15, 2026

    @kgryte
    MemberAuthor
  4. 0PrashantYadav0 commented on Sep 17, 2026

    @0PrashantYadav0
    Member

    @kgryte Overall +1. I pulled some numbers from develop that make the case stronger than the write-up does:

    1,967 of 1,968 C benchmark files contain the identical print_version / print_summary / print_results / tic block — about 116k duplicated lines in total.
    That block is a median 41.5% of each file; with main() it's ~53%, so the "roughly half" estimate holds.
    Across files named benchmark.c, there are 1,327 #define NAME lines but only 1,033 distinct values — ~294 collisions ("add" ×12, "mul" ×10, "sub" and "identity" ×8).
    Three suggestions:

    1. Ship (7) first, on its own. Resolving NAME from the package doesn't need the macros. tools/make/lib/benchmark/c.mk already routes every compile through tools/scripts/compile_c_benchmark, which is a single place to walk up to the nearest package.json and pass -DNAME="@stdlib/...". That fixes the 294 collisions in a small PR and de-risks the harness work.

    2. On (6), the printf( "# c::..." ) line — a variadic macro removes most of what people would have to remember:

    #define STDLIB_BENCH_PRINT_NAME( fmt, ... ) \
    	printf( "# c::%s:" fmt "\n", NAME, __VA_ARGS__ )
    
    // usage:
    STDLIB_BENCH_PRINT_NAME( "len=%d", len );
    STDLIB_BENCH_PRINT_NAME( "ndarray:len=%d", len );

    The c:: prefix and NAME become the harness's problem; the author only supplies the varying suffix.

    1. On (2)/(3), I think it's acceptable given the STDLIB_NAPI_ARGV_* precedent, but it would help to prefix harness-owned variables (e.g. stdlib_bench_i, stdlib_bench_len) or document them in one table in the header. The failure mode is a contributor writing int i; and getting a shadowing error that points into bench.h.

    On (8), an alternative to the .so that needs no tooling changes: split the header into stdlib/bench.h (TAP, timing, loop macros, no deps beyond libc) plus opt-in stdlib/bench/array.h, stdlib/bench/random.h for the BLAS-fill and PRNG helpers. Benchmarks that only time a scalar function never pay for the array deps.

    Happy to take (1) if it's useful. Waiting for your reply here.

  5. kgryte commented on Sep 22, 2026

    @kgryte
    MemberAuthor

    but it would help to prefix harness-owned variables

    I am a "no" on this. There is a difference between internal and external variables. For internal variables, we can prefix, just as we do in ARGV macros. But for external variables, I think we follow existing precedent as in our ndarray/base/unary macros and elsewhere where we document what variables are exposed.

  6. kgryte commented on Sep 22, 2026

    @kgryte
    MemberAuthor

    split the header

    This won't work as you think it will. We will need to list all the C deps of stdlib/bench in the manifest.json, meaning they will all get pulled in regardless of what headers you link to. So the only thing you are saving is some dead code elimination. In short, the largest compilation cost--compiling individual *.o files for the entire dep tree--is still present and is not avoided by splitting into separate headers.

  7. kgryte commented on Sep 22, 2026

    @kgryte
    MemberAuthor

    a variadic macro removes most of what people would have to remember

    Yes, that works, I think. A bit magical, but I suppose with proper documentation this can be okay.

  8. added
    MaintainersIssue which should be handled by project maintainers & is not recommended for external contributors.
    on Sep 22, 2026
  9. 0PrashantYadav0 commented on Sep 22, 2026

    @0PrashantYadav0
    Member

    Thanks, that all makes sense.

    I am a "no" on this.

    Dropped. In #15400 the exposed identifiers are exactly iterations, i, and len (the last only in length-based benchmarks), listed once in the header notes; everything internal is stdlib_bench_-prefixed, as with the ARGV macros.

    This won't work as you think it will.

    Right, I had not accounted for the manifest pulling in the full dependency tree regardless of which header is included. #15400 sidesteps it for now: stdlib/bench.h is libc-only and its manifest.json has no dependencies, so consumers gain a single -I and nothing to compile or link. If BLAS-based fills or stdlib PRNGs get added later, that is the point at which the shared-library approach becomes necessary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    BenchmarksPull requests adding or improving benchmarks for measuring performance.CIssue involves or relates to C.MaintainersIssue which should be handled by project maintainers & is not recommended for external contributors.RFCRequest for comments. Feature requests and proposed changes.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions