5. Assessing Technical Quality

Methodology

Experimental and/or Computational Methodology

Assessing a paper’s methodological quality requires asking a few basic questions about the authors’ research, such as:

  • Are the authors’ fundamental assumptions sound?
  • Are the theoretical frameworks underlying the authors’ research question and study approach correct?
  • Is prior or related work adequately referenced?
  • Do the authors fully describe their approach, procedure, or action?
  • Does the paper provide design guidelines or explain limitations on implementation of theory?
  • If the paper evaluates competing models, do the experiments designed by the authors fully distinguish between the proposed models?
  • If a new mechanism is proposed, are the experiments or computations sufficient to show that the new mechanism is plausible?
  • Are the methods logical and clearly described?

Describe any problems with the methodology in your written review comments for the authors. Fundamental issues in the methodology that the authors cannot address via manuscript revision should lead to a rejection decision.