AI in ERP: what to test beyond the chatbot

Separate answers, recommendations and actions

An assistant that summarizes an order is different from a model that recommends a production schedule or posts a transaction. Classify each proposed AI workflow by what it can read, suggest and change. Ask whether the capability is generally available in the quoted release, which license enables it and whether a partner must configure it.

Test grounded answers with bad data

Ask a question whose answer can be reconciled to a known work order. Require links to the underlying records and the date of the data. Then introduce a missing cost, an ambiguous part code and a question the system cannot answer. A useful evaluation records how the product expresses uncertainty rather than rewarding confident prose.

Keep the permission boundary visible

Use a restricted test role to ask about payroll, margins or a different legal entity. Check that search, generated answers and exported files obey the same access controls. For actions, inspect the proposed change, approval step and audit trail. Ask how a failed or partially completed operation is detected and recovered.

Measure the workflow, not the demo

Choose an actual task such as explaining a job variance or preparing a purchase recommendation. Measure completion time, correction rate and review effort against the existing process using the same inputs. Request written data-handling terms: model providers, retention, training use and regional processing. Do not infer better manufacturing outcomes from the phrase AI-native alone.

A job-cost investigation to use in the pilot

Prepare a completed manufacturing job whose margin can be independently reconciled. Include a late purchase invoice, a changed component and an operation that took longer than planned. First ask the assistant to identify the largest cost variance, then require the underlying material, labor and supplier records. Ask it to distinguish an observed fact from a possible explanation. The expected result is a reviewable chain of evidence and a useful next question. Do not count a plausible narrative as a successful answer if the reviewer cannot trace the numbers back to the same transaction set.

An engineering-change scenario with an approval boundary

Create an order with an approved bill of materials and at least one open work order. Propose a component substitution after material has been allocated. Ask the system which orders and cost assumptions would be affected, and which parts of the change remain uncertain. Inspect the proposed edits before authorizing them. Then repeat the exercise as a user who cannot approve engineering changes. The system should be evaluated against the permissions and release rules your business has specified. Record the actual behavior, including every manual step, rather than turning an impressive first response into a blanket score.

Use a scorecard with explicit evidence states

For each test, record the product release, module, user role, input dataset, expected result and observed result. Use evidence states such as demonstrated, documented, proposed and unresolved. Keep these separate from business importance: an essential requirement can still be unresolved. If you decide to use weighted scoring, agree the weights before the demonstrations and preserve the raw evidence beside the total. Sponsorship or commercial preference should never silently increase the score. A public overview, a vendor demonstration and a customer-operated pilot answer different questions and should not be represented as equivalent validation.

Ask what your own team must maintain

AI workflows may depend on master data, permissions, prompt configuration, integrations or a knowledge source that your team controls. Ask who maintains each dependency after launch and how changes are tested. Establish who reviews errors, who can suspend an automated action and how the business notices a missed task. Include these operating responsibilities in the implementation and recurring-cost discussion. The useful comparison is the complete workflow your organization can run reliably, including review and recovery, rather than the number of AI features named on a product page.

Build an error budget around decisions

Illustrative example: compare a buyer preparing ten purchase recommendations manually with the same buyer reviewing ten AI proposals. Use matched input complexity and record overlooked shortages, incorrect quantities, duplicate orders and time spent checking the answer. These figures are an experiment design, not measured product performance. A faster draft may create more review work, while a slower answer with clear source links may save time overall. Classify errors by consequence before the exercise. An incorrect narrative explanation and an unauthorized purchase release deserve different responses. Decide who can stop the workflow and how affected transactions are found. Repeat the most important cases after a configuration or model change; a successful pilot is tied to the tested release and settings.

Check the information inside instructions and attachments

Include a supplier attachment that contains ordinary business text plus an irrelevant instruction to disclose another customer's information. Ask the provider how the application separates document content from authorized instructions. Test the behavior in a sandbox using fictitious records. Also check stale documents: a superseded purchasing policy must not silently override the current approved rule. Record the source priority and how obsolete knowledge is removed. These are evaluation scenarios, not assertions about a particular vulnerability. Microsoft's application card provides product-specific transparency material for Microsoft Copilot in Business Central; it does not replace your permission tests or the terms for a different assistant. Maintain a short approved-use register stating the workflow, allowed data, reviewer and suspension owner.

Common questions

Continue your research

The following profiles provide a starting point for checking vendor-described scope. Inclusion is not a ranking or a statement that every requirement is supported.