ResiCLAP: Semantic-Preserving Residual Enhancement for Fine-Grained Impact Sound Retrieval

The problem

A sound designer searching for a “metal impact” gets hundreds of results that are all, correctly, metal impacts. What separates a usable take from an unusable one is finer: how bright it is, how sharp the attack is, how long it rings. These properties are category-relative — a bright metal impact and a bright wooden impact occupy very different absolute spectral ranges — and global audio–language embeddings tend to emphasise between-category separation while under-expressing this within-category variation.

Headline result

nDCG@10 on fine-grained retrieval rises from 0.232 to 0.450 — a gain of +0.218 (95% CI [0.117, 0.321]) — while Clotho and FSD50K retrieval stay at or above frozen-CLAP levels.

At a glance

Evaluation data

Fine-grained retrieval is evaluated on RealImpact, a set of real recordings of everyday objects struck at controlled points and captured by a dense microphone array. Because every recording carries its object, material, strike point and microphone position, it supports queries that differ only in acoustic detail rather than in category.

Training and test splits are disjoint by object, so no object heard at training time appears at test time. The held-out pool used throughout this site contains 400 recordings of 5 unseen objects — Mug (ceramic), Bin (plastic), Flowerpot (small ceramic), Vase (wood), Spatula (metal) — each struck at 5 points and recorded across 4 microphone distance rings and a range of angles.

Attribute supervision is category-relative: brightness, attack and decay targets are standardised within each material, so “bright” means bright relative to other sounds of that material rather than bright in absolute spectral terms. Semantic retention is measured separately on Clotho and FSD50K, which the model never trains on as a retrieval target.

Method

ResiCLAP treats the frozen CLAP embedding as a fixed semantic anchor and learns only bounded corrections around it.

1. Attribute residual branch

Probing the frozen HTSAT encoder shows that intermediate layers expose fine-grained attributes more strongly than the final global embedding. We read three of them directly:

Each is projected to 512-d by a two-layer MLP (hidden width 128, LayerNorm, GELU) and the three projections are merged by a learned sigmoid gated fusion. Every gate is initialised to −2, so each branch starts with a small contribution and has to earn its influence during training.

2. Multi-scale temporal residual branch

CLAP's own frame stride is too coarse for transient detail, so this branch reads the waveform at higher resolution. From a 1.5 s window beginning 100 ms before onset we extract 5 ms log-mel frames plus envelope, spectral-flux and relative-time features, standardise them, downsample to 20 ms steps and project to 64-d states.

Temporal evolution is then modelled explicitly as relations between time steps at four offsets — 20, 40, 80 and 160 ms — so the branch sees both short transients and longer evolution. Attribute-specific attention summarises each scale, and a second attribute-specific weighting combines the four scales.

3. Bounded residual adaptation

This is the part that keeps CLAP intact. Each residual is first projected orthogonally to the frozen embedding, then clipped to a fixed radius:

with ρ_a = 0.20 for the attribute residual and ρ_t = 0.05 for the temporal residual. The bounded residuals are added to the anchor and re-normalised:

Because the residuals are orthogonal to z₀ and bounded, the enhanced embedding is a rotation of the original by at most about 14°. Our ablation shows it is the magnitude bound, not the orthogonal projection, that does the work of preserving semantic structure — see the trade-off analysis.

4. Training objective

The bound controls how far the embedding may move; the objective decides what the movement should encode. Two groups of terms are combined:

Training setup