1. The Mystical Hyperparameters
You know that learning rate you tweaked at 3 a.m.? The batch size you settled on after a dozen trials? If they aren't in the paper, your work is already on thin ice. A common and fatal error is the under-specification of hyperparameters. These aren't just
minor details; they are the specific recipe for your model's success. Failing to report them fully—or worse, not explaining how you arrived at them—makes it impossible for others to verify your findings. Reviewers for top conferences like ICML are trained to spot this. They see a lack of hyperparameter details not as an oversight, but as a potential red flag that the results might be a fragile, one-off success. The fix is straightforward but requires discipline: log everything. Use tools to track your experiments, and include a detailed table of all hyperparameters and their chosen values in your paper's appendix.
2. The 'Trust Me' Codebase
Submitting a paper without clean, documented, and accessible code is the academic equivalent of showing up to a potluck empty-handed. It's no longer enough to just describe your method; the scientific standard in machine learning now demands the release of the code that produced the results. Yet, a surprising number of submissions fail here. They might omit the code entirely, submit a messy and undocumented script, or fail to specify crucial software dependencies and library versions. A reviewer shouldn't have to be a digital archaeologist to run your experiment. This is where tools like Docker containers have become invaluable, allowing you to package your entire computational environment. Major conferences like ICML and NeurIPS explicitly state that the availability and quality of code will be factored into the acceptance decision. Don't let your breakthrough be derailed because no one could get your script to run.
3. Data Leakage and Shaky Splits
This mistake is subtle but deadly. Data leakage happens when information from outside the training dataset is used to create the model, often inadvertently. For example, if you normalize your entire dataset before splitting it into train and test sets, your training process has already 'seen' the test data, leading to overly optimistic performance metrics. A related error is not being transparent about how you split your data. Was it a standard 80/10/10 split for train/validation/test? Was it stratified? Did you ensure no subject in the test set also appeared in the training set? These details are critical for assessing the true generalization capability of your model. Without them, reviewers may suspect your model's impressive performance is an illusion, one that will shatter the moment it's tested on truly unseen data.
4. Playing 'Random Seed Roulette'
Many machine learning algorithms, especially in deep learning, have stochastic elements, from weight initialization to dropout. This means that running the exact same code twice can produce slightly different results. An unethical (or sometimes just naive) mistake is to run an experiment with dozens of different random seeds and only report the single best outcome. This is known as 'seed-hacking' or cherry-picking, and it fundamentally misrepresents the model's performance. A robust model is one that performs well on average, not just on one lucky run. The accepted best practice is to run your experiment with multiple different seeds and report the mean and standard deviation of the performance metrics. This demonstrates that your result is not a statistical fluke and gives a much more honest picture of your model's stability and reliability.
5. Ignoring the Hardware
In an ideal world, an algorithm would perform the same way on any machine. We don't live in that world. The type of GPU used, for instance, can affect the results of floating-point computations, leading to small but potentially significant differences in outcomes. This is especially true for large-scale and reinforcement learning models. While you can't control the hardware others will use, you must document what you used. Reporting the specific hardware (e.g., 'NVIDIA A100') and key software libraries (e.g., 'CUDA 11.8') is now part of the reproducibility checklist for many major conferences. It provides crucial context and helps others diagnose why they might be getting different results. It shows you've been thorough and understand that in modern machine learning, the computational environment is part of the experiment itself.
