Aotearoa Birdsong, Decoded

Everything below this box is the record of how we got here, kept unedited per the kaupapa of this project. But it is now history, not the live pipeline. If you read one section on this page, read this one — it names what actually generated tonight's korimako figures and manifold.

The current korimako pipeline, end to end:

1. Segmentation: TweetyNet v2 (CNN), F1 = 0.902 on a held-out test split — this supersedes the 0.8577 result below, which was measured on a split with bird-level leakage between train/val/test (see the 5 August correction). The new number is not directly comparable to the old one; treat 0.902 as the current best estimate, not confirmation the old number was "wrong by exactly this much."
2. Syllable typing: a stage-4 family classifier trained on the Koe hand-labelled corpus (Fukuzawa et al. 2020), 0.893 test accuracy across 24 families plus UNKNOWN — this replaces the rule-based 5-category Hill & Ji scheme for korimako specifically (that scheme was built for tūī and is retained for tūī, see below).
3. Sex from repertoire: 5-fold cross-validated, 0.940 CV accuracy on 454 birds (≥10-syllable subset), 0.892 on the 107 birds held out of all training.
4. Site / dialect from repertoire: repertoire composition identifies home site.
5. Individual identification: closed-set, 77 birds, top-1 55% / top-5 80%; reliable per-bird estimates need ≈6–8 songs, and the corpus is thin there (median 2 songs/bird) — stated plainly on the korimako panel rather than hidden.

The tūī analysis below (rule-based Hill & Ji categories, CNN segmentation only, no typing model) is not superseded — it is a separate species with its own pipeline, which is why the acoustic manifold now shows a korimako example (TN v2 + stage-4) front and centre and keeps the tūī recordings as a labelled "legacy rule-based" option in the dropdown, not because the tūī work is wrong, but because it is a different, older method applied to a different bird. Kākā uses a third, call-based framework (Van Horik 2007 / Vaishnav 2026) that has no sequence grammar and is out of scope for the sequence-based methods below.

Full-corpus figures (repertoire, sex, site, atypical individuals) recompute on 1,065 birds / 21,463 syllables for the main tier, and separately on the 107-bird test split never seen in training, for the honest-tier check. Both are on the live korimako panel with per-figure n stated.

Every number above comes from the Koe corpus — recordings collected under a consistent protocol and hand-annotated by the original researchers. That is not the same question as does this pipeline work on an arbitrary field recording with no ground truth, which is the situation anyone actually deploying this would face. We ran TN v2 + the stage-4 classifier, cold, on Macaulay Library korimako recordings never touched by training or evaluation, to find out.

It runs, and produces plausible output. On a clean 25s field recording (Nelson Lakes NP, Feb 2023), the pipeline found 48 syllables across 13 stage-4 families with sensible temporal boundaries and no degeneration to noise or a single repeated label. There is no ground truth for this recording, so this cannot be scored as accurate — it demonstrates the pipeline runs end-to-end on real audio, not that its output is correct. That distinction matters and we are not blurring it.

Detection confidence measurably degrades with recording quality — but only visible at scale, not from single examples. We scored 836 Macaulay bellbird recordings by an SNR proxy (20th-vs-95th percentile energy ratio in the vocal band; the same idea as Xeno-Canto's A–E quality field, computed directly since Macaulay has no such rating), then ran the segmentation model's raw frame-level detection probability on the 607 usable recordings (excluding near-silent and near-continuous-noise clips). Result: mean confident-frame fraction (p≥0.6, the validation-tuned threshold) rises 0.089 → 0.151 → 0.188 from the low to medium to high SNR tertile (Spearman r=0.508, p<0.0001, n=607). Lower-quality recordings genuinely produce less confident segmentation.

A single-example comparison across quality tiers was actively misleading, and we are keeping the record of that rather than quietly fixing it. Our first attempt picked one representative recording per SNR tertile and found the opposite pattern — the low-SNR example produced 8× more segments than the high-SNR one, which we initially (wrongly) read as the model hallucinating on noise. Checking the actual frame-level detection probabilities showed that recording was simply more vocally active, and a separate medium-quality example had a genuine, undetected 7-second song passage that the single-example view made easy to miss entirely. The population-level result above (n=607) is what actually holds; the anecdote did not generalise, and would have been wrong to publish as the finding. This is the reason we do not report few-example demonstrations as evidence of a trend on this page without also running the trend at scale.

Practical implication: an SNR floor before running the pipeline on new field recordings would reduce, but not eliminate, low-confidence output — near-miss frames (probability 0.3–0.6, likely real vocalisation the model is unsure about) do not show a strong quality trend on their own, so some genuine misses will occur even in clean recordings. We have not yet checked whether any Koe corpus recordings are similarly noisy in a way that could affect the full-corpus figures above; that is open follow-up work, not yet done.

Separately from the population-level analysis above, we picked one SNR-proxy-scored "high quality" recording (Akaroa foreshore, a populated harbour town) to show as an illustrative example. The pipeline confidently (mean confidence 0.89) segmented and labelled ~32 loud, regular transients as bellbird syllable types. On inspection this recording could not be confirmed as clean, unambiguous bellbird song — the SNR proxy measures loudness and regularity in a frequency band, not is this actually a bird, and a loud non-vocal or ambiguous sound in a populated area can score "high quality" under that metric just as easily as a clean bird call can. We could not resolve from the spectrogram or spectral statistics alone whether those transients were real bellbird calls; a listen suggested "maybe a weird, cicada-like call" — genuinely ambiguous, not resolved either way, and we are not asserting a verdict on that specific recording.

The fix: use independent human verification, not an acoustic proxy, to select the illustrative example. Macaulay Library records a community rating (0–5, from independent listeners), a background-species field, and playback/captive flags — a real provenance signal our SNR proxy never had access to. Filtering the 980 Macaulay korimako recordings to rating ≥4.0 from ≥3 raters, no listed background species, not captive, playback not used, left 46 candidates. The current illustrative example on this page (ML 364073421, Lady Alice Island, Northland, January 2020; rated 4.67/5 by 6 independent raters) is drawn from that filtered set: annotated spectrogram and source audio are both published so a reader can check the labels against the sound directly rather than take our word for it.

What this changes and what it doesn't: the population-level result (n=607, confidence rising with SNR, Spearman r=0.508) is unaffected — it never depended on any single recording's identity being correct. Only the single illustrative example changed. We are keeping this addendum rather than quietly swapping the figure, because "our first quality metric let an ambiguous recording through as a headline example" is itself informative about the limits of acoustic-only quality proxies for field data with no ground truth.

This page has shown F1=0.784 as the bellbird segmentation result while also stating, three paragraphs below, that the artifact behind 0.784 no longer exists and the number does not reproduce. That is not a tenable thing to keep publishing. Meanwhile the best-provenanced comparison we have has sat unpublished on a lab machine since 29 July. Both halves of that are corrected here. The 0.784 discussion below is retained unedited — it is the record of how we got here.

The current result: CNN F1 = 0.8577, FIR envelope F1 = 0.7212 (Δ = +0.137) on split 88513a3aaa91. Unlike 0.784, this one is a held-out estimate and both arms were scored the same way. The CNN here is the SpecAugment-trained model — we name the model because a later run appeared to show augmentation hurting, and that run turned out to be invalid (it resampled the audio to a rate the model was never trained at, which cost 0.13 F1 on its own). Whether augmentation helps is currently unknown; see “What we do not yet know”.

15 held-out test files, 246 ground-truth segments. One matching criterion for both arms — IoU ≥ 0.3 OR overlap ≥ 0.5, greedy by detected length. Both operating thresholds chosen by argmax F1 on a separate validation split, not on test. Paired sign test over the 15 test files: 12 favour the CNN, 2 favour FIR, 1 tie, p = 0.035 two-sided. (The value recorded in the source artifact, 0.0176, is one-sided; we publish the two-sided figure.) Source: rescore_final_results.json, 29 July 2026.

Now the part that matters more than the number. This is not evidence that the CNN generalises, and we are not claiming that it does.

• n = 15 test files. Every claim on this page about segmentation rests on fifteen recordings.
• One site, one week. All 60 files in this split come from a single site (CUV), recorded across five days of November 2016. There is no second site and no second season.
• Bird-level leakage in both directions. The split is file-disjoint — no recording appears in two partitions — but it is not bird-disjoint. Of the 12 distinct birds in the test set, 4 also appear in training (AMH015, AMH036, BAE042, WHW021) and 6 also appear in validation (BAE037, BAE042, BAE047, DHB008, DHB016, WHW021). In one case the annotator's own filename says so: AMH015_01_M…TrumpetSongSameBirdAsTr014Harmonics.wav is in train and AMH015_02_M… — same bird, same song type — is in test.
• The operating point was selected on birds present in the test set. This follows from the two facts above and is the one that does the most damage. The methodology note "threshold tuned on validation, not test" is true at the file level and false at the level that matters, because half the test birds are validation birds.

So read 0.8577 as within-individual, within-site, within-week segmentation performance. That is a real quantity and a reasonable place to be after five weeks. It is not the quantity a reader will assume from "CNN F1 = 0.858", and it does not support "this method works on New Zealand birdsong." A re-split grouped by bird rather than by file is the next measurement, and we expect the number to fall; the size of that fall is the finding, not a failure.

Five open questions that bear directly on everything above. None of these has an answer yet, and we would rather say so here than have a reader discover it. Per the kaupapa this project works under, a finding is a baton and everything the next runner needs to succeed — which means the unresolved parts are part of the finding, not an embarrassment to be tidied away.

• Whether data augmentation helps or hurts — direction unresolved. A 29 July run found augmentation improved F1 (+0.033); a 3 August run found it hurt (−0.078). The two runs are not comparable: the 3 August run's FIR baseline collapsed from 0.721 to 0.424 because the registered baseline was never loaded, and its "augmented" arm reused a borrowed checkpoint rather than retraining. Neither run supersedes the other. We therefore publish no augmentation qualifier on the number above, in either direction, until a clean ablation exists.
• How WhisperSeg compares — currently unmeasurable. WhisperSeg is being evaluated in parallel, but its scoring counts a prediction correct on any overlap with ground truth, while the CNN and FIR numbers require IoU ≥ 0.3 or 50% overlap. A more permissive matcher produces a higher F1 for free. No WhisperSeg-vs-CNN comparison appears on this page and none should be quoted from elsewhere until all three methods are re-scored through the single canonical matcher. This is the same error class we publicly withdrew in July; we are declining to make it a third time.
• The annotation ceiling — unmeasured, and it bounds every number here. Every F1 on this page measures agreement with one person's annotations, treated as truth. But where a syllable starts is genuinely ambiguous, and two experienced annotators will disagree. If human annotators agree with each other at F1 ≈ 0.85, then 0.8577 is at the ceiling and further tuning is fitting one annotator's habits. If they agree at 0.97, there is real headroom. We currently cannot tell which world we are in, which makes this the highest-value unmeasured quantity in the project.
• How much of the result survives a bird-grouped split. See the correction above: the current split is file-disjoint but not bird- or site-disjoint. Until the re-split is run, the reported figure cannot be read as generalisation to unseen birds, let alone unseen sites.
• Why the label set was cut from 41 families to 14. The Koe taxonomy carries 41 label families; the trainers filter to 14 by a minimum-count threshold. That threshold is not recorded in any provenance block, so we cannot currently state what fraction of segments the 27 dropped families represent, or how the result moves at a different cut. This is a bookkeeping failure rather than a scientific one, but it is unrecoverable without a re-run.

One more, smaller: the seven CNN spectrogram figures further down this page carry a “CNN prediction (bottom)” legend burned into the image pixels that is false — the panel does not show what the legend says. This is disclaimed in the prose beside them and the figures have deliberately not been regenerated, so the erroneous legend remains visible rather than being quietly repaired.

Until today this section was headed “CNN Segmentation & Categorisation” and described the network as performing “joint syllable segmentation (finding boundaries) and categorisation (classifying vocalisation units)”. No measurement ever supported the categorisation claim, and the model could not have performed the task. It is withdrawn.

The training script read the Koe annotation field seg[3] as the syllable class. That field is f_mid — a frequency in Hz, not a label. The result was 829 distinct “classes” over 829 segments: one example per class, the class identities being floating-point frequencies (690.06, 762.34, 770.07 Hz…). No classifier can be trained that way, and none was.

The segmentation figures above are unaffected: segmentation scoring uses only the blank/non-blank distinction, so the spurious classes never enter precision or recall. If anything they penalise the model, since a segment is terminated whenever the predicted class changes. But every statement on this page about the network identifying syllable types was false.

Two knock-on errors, corrected today. The dataset table below listed the Koe corpus as having 3 label types — neither the 829 the model was trained on nor the 74 the corpus actually contains. And the multi-species section offered bellbird:click as an example of a species-prefixed label; the real labels in that run are frequencies with a species prefix, e.g. bellbird:1018.63239577395.

The real taxonomy was never loaded. segment.extraattrvalue.json, sitting beside the file that was parsed, holds 13 families and 74 labels — Stutter, Alarmy, Pipe, Flatsqueak, Down, Chortle, Cough. A retrain against it is underway. Until that produces a validated result, this project has no syllable classifier and no categorisation performance to report.

An earlier version of this section claimed the CNN's advantage was “more dramatic on the Donald toutouwai dataset”, and that the CNN “learns spectral change patterns that signal phrase boundaries even when the amplitude envelope is flat”. No measurement supported that claim when it was published — no CNN toutouwai evaluation existed at the time. The claim is withdrawn. What we can actually say is below.

Toutouwai: the CNN has not yet been validly evaluated, and on the one weak evaluation that does exist it loses to both classical baselines. The multi-species augmented model (the architecture described below) scores F1=0.262 (P=0.311, R=0.226) on toutouwai over 13 test files — against FIR F1=0.306 and spectral flux F1=0.329. On the evidence we have, the classical baselines currently win on toutouwai. That 0.262 is itself a weak measurement: 13 test files, and the confidence threshold was chosen by taking the argmax over a sweep run on the very files being scored, so it is not a held-out estimate either. Read it as “not yet properly measured, and no sign of an advantage” — not as a firm ranking in either direction. The hypothesis that a CNN should handle toutouwai's sustained tonal phrases (median 971ms) better than amplitude-envelope methods remains untested.

Bellbird (Koe): F1=0.784 (P=0.753, R=0.818, median boundary error 13.3ms) — from a bellbird-only run, 100 epochs (cnn_segmentation_results_v3.json). This is not a held-out estimate. That file contains a configuration, a six-point confidence-threshold sweep, and the argmax over that sweep. There is no validation split, so the threshold was selected on the same data the score is reported on. 0.784 is an optimistic upper bound on that configuration, and for the same reason it is not directly comparable to the FIR amplitude-envelope baseline (F1=0.731, 31.5ms boundary error), which had no equivalent tuning freedom.

The head-to-head against the FIR baseline is withdrawn: the two were not scored the same way, and on the fairest comparison available the classical method is not behind. The CNN counts a detection as correct at IoU > 0.1 (tweetynet_v3.py:188); the FIR baseline was held to IoU ≥ 0.3 or 50% overlap (validate_fir_segmentation.py:201, recorded in its own output as matching_criteria). The quoted 0.731 is also FIR's test-split row (376 ground-truth segments), while the CNN's 0.784 is over a different split (499 segments). Scored across the full 829-segment dataset at its stricter threshold, FIR reaches F1=0.786 (P=0.857, R=0.725) — level with the CNN's 0.784 while being judged more harshly. Until both are re-scored on one split under one matching criterion, treat the CNN as not demonstrated to beat the classical baseline on bellbird.

Reproducibility: the artifact behind 0.784 no longer exists, and the number does not reproduce. None of the training scripts set a torch seed — only the file-level split is seeded — so weight initialisation and batch shuffling differ on every run. On 27 July 2026 the same script was re-run on the same data and wrote to the same filename, replacing the artifact this figure was read from; the re-run scored F1=0.767 (P=0.726, R=0.814) at the same selected threshold of 0.3. The originally published values survive only in a snapshot taken minutes beforehand. We report this rather than quietly restating the new number, because the honest content of the finding is that this measurement has a run-to-run spread we never characterised, and that results were being overwritten in place instead of written to run-stamped paths.

Our two bellbird runs disagree and we cannot yet reconcile them. A separate 50-epoch bellbird run over 15 test files (cnn_bellbird_large.json) gives F1=0.361 (P=0.638, R=0.251, median boundary error 49.7ms) on the same dataset. The two runs differ in epoch count, spectrogram settings and minimum-segment duration, and share no defined test split, so neither supersedes the other. Until a run with a proper held-out split settles it, treat bellbird segmentation performance as unresolved between 0.361 and 0.784, and treat 0.784 as the most favourable configuration we have run rather than as the headline result.

The multi-species architecture below is a different, worse model — do not attach 0.784 to it. The joint multi-species training and augmentation described in the rest of this section correspond to cnn_multispecies_augmented.json, whose combined score is F1=0.259 (P=0.311, R=0.222): bellbird F1=0.308 on 2 test files, toutouwai F1=0.262 on 13. The 0.784 figure came from the bellbird-only model and says nothing about this configuration.