About the Attention Kernel Challenge

The Challenge

Build the fastest block-sparse attention backend for an NVIDIA H100. Your kernel must compute exact causal block-sparse attention — numerically faithful to the FP32 reference — while minimizing latency across a family of realistic sparse workloads.

Every submission implements a single entrypoint:

block_sparse_attn_fwd(q, k, v, row_ptr, col_idx, seq_lens) -> (o, lse)

The sparse structure arrives as a CSR block descriptor (row_ptr, col_idx), and your kernel returns the attention output plus the log-sum-exp values.

Hardware

All official evaluations run on a single H100 SXM 80GB. Local CPU latency is not challenge-relevant — only official H100 runs count.

Scoring

• Correctness first: outputs are checked against the FP32 reference. Invalid kernels score nothing.
• Entries are scored by the geometric mean of family median latencies (lower is better), with worst-family latency as the tiebreaker.
• Compilation and warmup stay outside the timed region; dispatch overhead still counts.

Workload Families

• Sliding-window causal attention
• Sliding-window plus sink or global blocks
• Sliding-window plus retrieval or random blocks

The public queue stays in a single 8192-token bucket, but varies sparse descriptors across window/global/retrieval structure, retrieval locality bias, sequence-length ranges, and realized CSR layouts.

Submission Rules

• Upload a .zip, .tar.gz, .tgz, or a single submission.py file containing your source.
• Keep the compressed bundle at or below 10 MB; no precompiled binaries or large artifacts.
• Submissions may use only the Python standard library plus torch, triton, and numpy.
• Each X account can submit at most 5 times per rolling hour.

Finals

After submissions closed, organizers took the top 20 scorers and reran each of them three times on a fresh hidden seed sampled from the published workload distribution. The median of those three runs determined final placement.

Resources

The harness, example submission, and local test suites live in the challenge repo. Shipped suites: smoke, local-dev, public-quick, public-queue, public-dev.