On the sensitivity of machine-learned probabilistic weather forecast models to scale-aware scoring rules

Venue suggestions

Major revisionpanel verdict · 2026-09-01

Venue Recommendations

as_is

None. The editor's verdict is major, and the central claims lack quantitative support. In Silico's public review model makes this particularly problematic: the panel's observation that conclusions rest on visual comparison without uncertainty quantification is correct and will be visible to all readers. Submitting as-is would invite the same critique in public.

after_revision

Weather and Forecasting (American Meteorological Society)

  • Fit: This is the natural home for machine-learned weather forecasting methodology. The journal publishes comparative studies of forecast skill, spectral diagnostics, and loss-function design. The authors' candor about preliminary scope and ad hoc choices aligns well with AMS's expectation of honest reporting of limitations.
  • Why it works post-revision: Once the skill comparison in Figure 2 is quantified with bootstrap intervals and the spectral results are reduced to summary diagnostics with uncertainty bands, the paper becomes a solid empirical contribution to the scoring-rule literature in weather. The graph-based localization is a genuine methodological advance (more general than patches), and the finding that per-scale weighting may matter as much as the mechanism is exactly the kind of practical insight AMS values.
  • Acceptance odds: Moderate-to-good (60–70%). The required revisions are substantial but feasible, and they address real gaps rather than philosophical disagreements. AMS reviewers will expect the quantification the editor requested.

Journal of Machine Learning Research (JMLR)

  • Fit: JMLR publishes machine learning methodology applied to real problems, including weather. The graph energy score construction is a genuine algorithmic contribution (graph-based localization of multivariate scores), and the multi-scale loss formulation is a reusable technique. The paper's focus on loss-function design rather than architecture is appropriate for JMLR's scope.
  • Why it works post-revision: After quantification, the paper becomes a methodological study with clear takeaways: (1) multivariate scoring rules can match CRPS-trained models; (2) localization via graph neighbourhoods is more general than patches; (3) per-scale weighting is a critical hyperparameter. These are all machine-learning insights, not just weather results.
  • Acceptance odds: Moderate (55–65%). JMLR is selective and will want to see the quantitative backing. The weather application is secondary to the method, which is appropriate for the venue. The main risk is that after quantification, the differences between methods shrink enough that the contribution reads as incremental; this depends on the actual bootstrap intervals.

Quarterly Journal of the Royal Meteorological Society (QJRMS)

  • Fit: QJRMS is the gold standard for weather forecasting methodology and has published recent work on machine-learned models (including AIFS-CRPS itself). Comparative studies of scoring rules, spectral fidelity, and ensemble training are core to the journal's scope. The authors are already citing QJRMS heavily and the work is positioned within that conversation.
  • Why it works post-revision: This is arguably the most natural fit. QJRMS readers will understand the subtleties of fair scoring rules, the importance of spectral realism, and the trade-offs in localization. The quantified results will be taken seriously. The graph energy score and multi-scale loss formulation are methodological contributions QJRMS would value.
  • Acceptance odds: Moderate-to-good (65–75%). QJRMS expects rigorous empirical work and will appreciate the authors' honesty about limitations. The main barrier is that the differences between methods are modest (as the authors acknowledge), so the contribution must be positioned as "which mechanism matters least" rather than "which is best." The quantification will determine whether that's a defensible claim.

alternative

arXiv (with In Silico overlay review)

  • Rationale: The authors' intended target was In Silico, which is appropriate for a preliminary study with honest caveats. After addressing the editor's required revisions, the paper is suitable for In Silico's public review model — the quantification will make the claims checkable, and the limitations will be transparent. In Silico's audience includes researchers from adjacent fields who need to judge soundness without being specialists, which fits a methodological paper on scoring rules.
  • Fit: In Silico explicitly welcomes preliminary scope if claims are scaled to match. This paper, once quantified, is exactly that: a systematic comparison with clear limitations (single runs per config, lower resolution, shorter training) and honest caveats about generalization. The public review will be valuable for the field.
  • Acceptance odds: Good (75–85%). In Silico's bar is soundness and checkability, not novelty or impact. Once the required revisions are made, the paper clears that bar. The public review will be a genuine service to readers trying to understand scoring-rule sensitivity.

NeurIPS or ICML (workshop track)

  • Rationale: If the authors want to reach the machine-learning community before submitting to a weather journal, a workshop on machine learning for science or climate/weather would be appropriate. This allows presentation of the graph energy score and multi-scale loss ideas to a broader audience without the full journal bar.
  • Fit: Moderate. The paper is more empirical than algorithmic, and the weather application is specific. But the loss-function design is general and the quantitative results (once obtained) would be of interest to ML practitioners working on scientific applications.
  • Acceptance odds: Moderate (60–70% for a workshop; main conference would be much lower). Workshops are more forgiving of preliminary scope and single-run results, but the quantification would still strengthen the submission.

Notes on the revision path

The editor's decision letter is unusually detailed and fair. The required revisions are not busywork: they directly address whether the central claims survive scrutiny. Specifically:

  1. The tropical separation (graph energy best, global energy degraded) is the only differentiating skill result. If bootstrap intervals over 84 dates show this is within sampling noise, the abstract and conclusion must change. This is not a minor edit.

  2. The spectral results are currently unquantified. Figures 3–14 have one-line captions and no summary metrics. Once you add a scalar diagnostic per config (e.g., mean absolute log-ratio to ERA5 over wavenumber bands, with bootstrap intervals), you will either confirm "substantially improves" or soften it. This will determine the paper's main claim about scale-awareness.

  3. The weighting claim is confounded. You say weighting "can matter as much as, or more than, the specific mechanism," but the weights are untuned and their values are not reported. Either run one alternative weighting or downgrade this to a hypothesis. This is important because it's the paper's most interesting finding.

  4. The graph energy advantage is composite. You optimize fGES_graph + 0.1·fES. The tropical improvement could come from the localization, the anchor, or their interaction. State this explicitly and do not claim the localization has been validated. (An ablation would be the best follow-up experiment, but it is not required for publication.)

Once these are addressed, the paper is a solid empirical contribution suitable for QJRMS, Weather and Forecasting, or JMLR, depending on whether you want to emphasize the weather application or the machine-learning methodology. In Silico is also a good fit for a preliminary study with honest caveats.

The work is sound in design and honest in reporting. The gap is evidentiary, not conceptual. Close it and the paper will be publishable.

← All documents in this review