Peer-Reviewed A.I. in Vet Med

Important - before you start reading the papers:

​

Just as peer review ensures reliability in medicine, it’s essential for artificial intelligence models to also undergo similar scrutiny before they are integrated into practice. With that said, be aware that the current landscape of artificial intelligence regulation is sparse and fragmented--even for some of the models that are making it into our practices. Because of this, like any other facet of medicine, it's important to understand the science behind what you're doing.

​

Here's a quick introduction to how to evaluate where these A.I. models stand in development.

​

There are 3 main stages of proper A.I. model development: training, validation, and deployment. It is very important to understand that there is no legal requirement for A.I. models to have undergone any complete portion of this process to make it into your practice. Yeah, that's scary! Hopefully regulations will come along soon to protect consumers of these models (but still allow for innovation). However, for now, you need to do your due diligence to understand the equipment you're using, the basis for how it works, and--wait--has it even been proven to work in the real world?

​

How to evaluate these papers to understand what A.I. development stage they describe:

Stage 1: Training

• What happens in this stage? 

  • The model learns patterns and relationships from pre-selected data, often tested on a subset to ensure it generalizes well.

• Results of this stage:

  • Preliminary accuracy is assessed - yeah, it looks like it could work??

• How to identify this stage when reading these papers: 

  • Often focuses on the initial dataset used to teach the model, details of how it was built, and the algorithms employed

• Data results unique to this stage:

  • Training Loss (without mentioning unseen data or validation results)

  • Training Accuracy (without comparing it to validation accuracy)

  • Epoch-Based Loss Reduction (with no mention is made of generalization to new data)

  • Precision, Recall, and F1-Score (if no validation or test set is mentioned)

  • Graphs: Loss and Accuracy Curves (If validation curves are absent)

  • Context Clues: 

    • No mention of a validation set

    • Explicitly states results are reported on the training data​

    • Hyperparameters (which are often adjusted in the training stage) are mentioned - learning rate, batch size, or optimizer type

    • Overfitting Signs (meaning the model is trained too specifically to the training data and cannot generalize to new data accurately) - very high training accuracy, very low training loss (without comparing to validation results)

• Concerns:

  • Stage 1 A.I. models are not ready for real-world deployment without explicit consent of those participating. At the end of Stage 1, these models have only proven that they can work well in a lab setting on the data that the researchers have given them. They have not yet proven real-world accuracy.

​

Stage 2: Validation

• What happens in this stage?

  • The model is tested on new data to evaluate its ability to generalize beyond the training dataset. This stage ensures the model’s patterns work reliably on new data. This often happens in controlled versions of real-world environments. (So, the participants still know they're guinea pigs.)

• Results of this stage:

  • The "lab" model that worked in Stage 1 is put to the test to see if it will still function when the chaos of the real world (sort of) is thrown at it—and yeah, it works!​

• How to identify this stage when reading these papers:​

  • The paper discusses testing the model on unseen data or mentions a validation set separate from the training data.

  • Results focus on evaluating the model’s generalizability rather than improving its performance on the training set.

  • May include comparisons between training and validation performance.​

• Data results unique to this stage:​

  • Validation Loss: Measures how well the model predicts on unseen data. If validation loss is stable or decreasing, the model may generalize well.

  • Validation Accuracy: Percentage of correct predictions on the validation set.

  • Performance Metrics on Validation Data: Precision, Recall, F1-Score, ROC-AUC

  • Validation Curves: Graphs showing validation loss and accuracy over time, often plotted alongside training curves for comparison.​

• Context Clues:​

  • The paper describes a validation set as distinct from the training data.

  • Discussion of overfitting or generalization concerns, often highlighting gaps between training and validation performance.

  • Metrics are explicitly labeled as validation results or compared to training metrics.

  • Mentions techniques to improve generalization, such as early stopping or regularization.​

• Concerns:

  • Stage 2 models are still not fully ready for real-world deployment. Validation demonstrates the model’s ability to perform well on reserved data, but it does not guarantee success in the craziness that is your daily life in vet med. There is one more stage to go to be certain you're dealing with an A.I. model that has well-proven accuracy in real world environments.​

​

Stage 3: Deployment

• What happens in this stage?

  • The model is integrated into real-world applications and tested on entirely new, diverse, and dynamic datasets. This stage evaluates how the AI performs in practical, real-world scenarios outside the controlled lab environment.

• Results of this stage:

  • The model’s effectiveness in real-world use is assessed—can it hold up to the ridiculousness that is vet med?

• How to identify this stage when reading these papers:

  • The paper describes testing the model in real-world settings or on datasets collected from live environments, such as clinics or user-generated data.

  • Results include field-testing metrics, feedback from users, or descriptions of real-world challenges the model faced.

  • Mentions of deployment in specific applications (ex: diagnostics, inventory management, or client interactions).

• Data results unique to this stage:

  • Real-World Accuracy: The model’s performance metrics (ex: accuracy, precision, recall) are calculated on entirely new, unselected data.​

  • Performance Metrics in Operational Contexts: Precision, Recall, F1-Score, ROC-AUC when applied in specific scenarios (ex: diagnosing a condition in real patients).

  • Time-to-prediction

  • Resource efficiency metrics

  • Error Analysis in Real-World Use: Common causes of misclassification or failure in practical settings.

    • Example: “The model struggled with data variability, such as incomplete patient histories.”

  • Feedback from End Users: Qualitative or quantitative feedback from veterinarians or stakeholders using the deployed AI.

    • Example: “92% of users found the AI tool helpful for client communication.”

• Context Clues:

  • Discusses specific real-world applications, such as live clinic settings, practice tools, or operational improvements in practices.

  • Mentions challenges faced during deployment, such as variability in data quality, user adoption, or scalability.

  • Results reflect conditions that are difficult to replicate in a lab setting, such as unpredictable user inputs (meaning, people do all kinds of crazy things) or a wider array of medical cases or signalments.

  • Descriptions of ongoing monitoring or updates, like retraining the model with new data or handling feedback.

• Concerns:

  • Models that make it through Stage 3 are the real deal. This real-world deployment puts it to the test, revealing unexpected weaknesses in robustness, bias, or usability. With that said, ongoing monitoring is essential to ensure consistent performance, identify errors quickly, and retrain the model as data inevitably evolves over time.

​

Okay, now you've earned your papers:

​