Getting the right AI tools for your practice: Questions to ask
AI solutions can save you time and help you work more efficiently, but it takes asking the right question to find the right tools.
Like prompt writing, searching for and interviewing AI vendors in health care can be a learning process and requires fine-tuning and multiple exchanges to get right.
Knowing what questions to ask will help you:
Identify priority AI use cases in your practice
Assess organizational readiness
Compare potential vendors
Understand key considerations related to implementation, safety, privacy, workflow integration and long-term value.
AI is here. What’s next?
Benefits of AI tools for physicians
AI tools are already being used by many family physicians professionally and show strong potential for reducing burnout. Although large language model technology is promising, primary care physicians have specific needs, and AI might not always be the ideal solution for you or your patients. Always check the accuracy of AI responses by reviewing and assessing the cited sources, seek patient signoff on data-sharing and practice writing neutral prompts to avoid leading the AI tool into giving an overly agreeable response.
Weighing benefits of AI: Common pros and cons
Evaluating your need for AI tools
Use AI where it can solve a clear problem or create meaningful value, not just because it’s the latest trend. These steps can help you evaluate what your priority use cases might be.
Identify your problems and practice needs
Gather information about different tech and digital health solutions, not just AI
Compare your options and know the risks, and decide your practice’s comfort level with taking on potential risks
Set purpose-driven goals and metrics for measuring your success and return on investment
Questions you might ask yourself and your practice team when deciding whether to explore AI tools are:
What current resources are devoted to these areas now without AI?
Which staff members’ workloads would be impacted by use of AI?
What value does your practice expect from incorporation of the AI tool (i.e., new revenue source, employee satisfaction, cost savings)?
Has your practice used AI before, and if so, in what ways? Did you experience issues or barriers with adoption? If so, how could these issues or barriers be mitigated with future AI implementations?
Build out your use cases
What to ask vendors about their AI tools
It’s critical to your satisfaction and success with an AI tool that you ask vendors questions on several different areas before signing a contract.
To understand what problems the vendor aims to address, ask:
Do the problems they are trying to solve align with problems you experience in your practice?
How does the vendor define responsible and ethical AI in health care? Do their definitions align with the AAFP’s Ethical Principles for AI in Family Medicine or your practice’s AI principles?
To learn about the vendor’s understanding of primary care physicians’ needs, ask:
Has the vendor conducted user research with family physicians?
What other primary care clinicians or practices have implemented the solution? What ROI have they achieved?
Were primary care or family physician involved in the product design?
Was usability testing done with attending physicians?
What is the typical customer adoption or retention rate?
Assess the process for EHR integration by asking:
Which EHR systems are supported? (Epic, Cerner, Athena, etc.) Does the tool integrate with your specific EHR version?
What is the typical implementation timeline?
What IT resources are required from your end?
Is there on-site support during implementation?
Make sure you’ll have the support you need by asking:
What type of ongoing technical support is offered?
What is the response time for clinical issues?
Does the vendor offer a dedicated account manager?
Explore customization opportunities by asking:
Can the system be customized to your clinical workflows?
Can alert thresholds be adjusted?
Can you add local data to retrain the model, or is it a one-size-fits-all model?
Ensure compliance standards are met by asking:
Does the vendor collect data, and how do they handle data ownership, secondary use and retention?
How does the vendor handle situations where the technology may create unintended workflow, safety, bias or equity challenges?
Where does our member/customer interaction data go? Does it go through a third party?
Do you have industry-relevant certifications, such as SOC 2 and ISO 27001?
Get a clear understanding of pricing and contract terms by asking:
What is the pricing model (e.g., upfront, annual subscription, per-user, per-prediction)?
What is the total cost of ownership (TCO) including implementation, training and IT support?
What is the typical contract length?
What are the explicit exit terms? How does the vendor ensure data portability if ending use with system?
Recommended AI tool evaluation guides
”EASIER” Framework for Evaluating AI Tools
Family physician–oriented framework for evaluating AI tools across six domains: ethics, accuracy, safety, intended use, explainability/transparency, and regulation.
AMA AI Specialty Collaborative: AI Evaluation Guide
Offers a detailed physician-focused framework covering clinical use cases, training and validation data, risks and mitigations, effectiveness, performance monitoring, and workflow integration.
NIST AI Risk Management Framework (AI RMF)
A widely adopted framework that provides structured guidance for governing, mapping, measuring, and managing AI-related risks across the technology lifecycle.
Coalition for Health AI (CHAI)
Provides health care–specific governance playbooks, risk categorization tools, and implementation guidance designed to support responsible AI adoption and oversight in healthcare organizations.
Joint Commission & CHAI Responsible Use of AI in Healthcare Guidance
Offers governance-oriented recommendations to help health care organizations safely and effectively implement AI at scale through policies, monitoring, and organizational oversight.