Back to all posts
Data & Analytics7 min read·October 26, 2026

Is Your Business Data Ready for AI?

TB
ThynkBlox Team
Engineering

How do you assess data quality for AI?

Assess data quality for AI against a specific task: check whether the available information is relevant, accessible, reliable and permitted for that use. Inspect actual records, trace their sources, and test whether they cover the situations your product must handle. Use the findings to choose a suitable approach, narrow the scope or repair the data before building.

The useful outcome is a documented decision with evidence and unresolved gaps. Avoid combining everything into a universal readiness score: reliable records do not resolve missing permission, and permission does not make irrelevant records useful.

What must the data help your product do?

Start with a short task statement. Describe the user, the question or decision, the expected output and the consequence of a wrong answer.

For example: “Help support staff draft answers about the current returns policy, using approved guidance, with a person reviewing each draft.”

Then identify the information that task requires. Approved policy documents may belong in scope. Old promotional copy and unresolved customer complaints need a separate justification.

Record these boundaries:

  • Required inputs: What must be available for each request?
  • Expected output: What would a correct response contain?
  • Excluded uses: Which questions or actions should the product decline?
  • Review: Who checks uncertain or consequential results?
  • Freshness: Which changes would make an answer obsolete?

If the task is still unclear, use an introduction to artificial intelligence in software to frame the discussion before assembling a dataset.

What should your first data audit include?

Create an inventory of the sources you intend to use. Give each source an owner, a purpose and a recorded decision.

CheckEvidence to collectDecision it supports
RelevanceExamples connecting records to the intended taskInclude, narrow or exclude
AccessibilityA repeatable way to retrieve and read the contentUse directly or resolve access
CompletenessMissing required fields and unavailable documentsRepair, exclude or limit scope
ConsistencyConflicting values, formats and duplicate recordsDefine reconciliation rules
ProvenanceOrigin, changes, approval and version historyEstablish which source to trust
PermissionDocumented authority for each intended useAllow, restrict or pause
RepresentationExamples across intended users and situationsIdentify unsupported cases

Keep evidence beside each finding. “Documents look good” is difficult to review. “The policy owner confirmed which version applies, but several attachments remain unreadable” gives someone a concrete next action.

Can the product access usable information repeatedly?

Test access through the route the proposed product would use. A person opening a document on a laptop does not establish that the application can retrieve it under the correct permissions.

Check whether files contain readable text, whether attachments are included, and whether tables retain their headings. For structured records, verify that related items can be connected through stable identifiers.

Also examine the update process. Who adds new records? How does a corrected policy replace an older version? What happens when access is withdrawn?

Set a clear acceptance condition: an authorised user can retrieve the required information, while a user without permission cannot. Include a check that updates and removals reach the proposed data flow.

How should you audit missing values?

Choose fields that are required for the task, then inspect missingness across relevant groups. Separate blank values from “unknown”, “not applicable”, zero and information that was never collected.

Illustrative example: A support export contains 200 resolved cases. Thirty lack a resolution summary, including 24 from the same intake channel. These figures are invented to show the audit method.

The next question is whether that channel belongs in the proposed product. Inspect its cases and speak to the source owner before deciding what to do.

For each missing field, record:

  • Whether the task can proceed without it.
  • Whether the original source can supply it.
  • Whether the product should request clarification.
  • Whether the affected record must be excluded.

Do not invent missing resolutions to make the dataset appear complete. Preserve the distinction between observed information and a later assumption.

How do you handle inconsistent or conflicting records?

Look for duplicates, conflicting statuses, mismatched identifiers and fields whose meaning changes between sources.

Two records marked “closed” may need investigation if one team uses that label for completed work and another uses it for abandoned requests. Matching words alone is insufficient evidence that the records describe the same outcome.

Define a rule for each conflict. Name the authoritative source, explain how duplicates are identified, and retain enough history to investigate corrections.

Check units and definitions as well as formatting. If a field represents duration, specify its unit and what starts and stops the clock. If it represents an amount, retain the currency and clarify what the amount includes.

Ask a person responsible for the underlying process to validate these rules. A technically consistent conversion still needs to reflect the business meaning.

Can you trace the data and justify its use?

For each source, record where it came from, who owns it, what changes it has undergone and which version is approved. For derived fields, retain the calculation or labelling rule.

Provenance should let a reviewer follow a proposed answer back to its supporting material. Where two documents disagree, record how the product should choose between them or flag uncertainty.

Review permissions separately for retrieval, model training, evaluation, storage and sharing. Ask the responsible owner to confirm each proposed use; do not treat possession of a file as sufficient evidence.

Identify personal, confidential and third-party material. Document who may access it, what must be removed, and how restrictions will be enforced. If authority remains unclear, pause that source while the question is resolved.

For a broader review of application access controls, see data security in modern applications.

Do your examples represent the situations you intend to support?

Build an evaluation set around the product's promised scope. Include routine requests, uncommon cases, incomplete inputs, ambiguous wording and questions the system should decline.

For each example, write down:

  • The input and relevant context.
  • An acceptable answer or action.
  • The evidence supporting that answer.
  • A condition that requires clarification or human review.

Include relevant variations in language, product type, channel and document quality. Ask process owners to identify cases that a tidy export might overlook.

For prediction tasks, review whether inputs would actually be available when the prediction is made. A record containing the eventual outcome needs careful separation from the information available beforehand.

Reserve examples for evaluation before making repeated adjustments. Document any overlap with material used during development.

What decision should follow the audit?

Write a short decision memo that links findings to scope.

Illustrative decision: Proceed with a staff-only policy assistant using approved, readable guidance. Exclude unresolved case notes. Require review of every draft. Delay customer access until permission checks and document updates have been verified.

Use three practical outcomes:

  • Proceed within defined limits: Required evidence and permissions are established.
  • Repair specific gaps first: Name the missing work, its owner and its acceptance condition.
  • Pause the proposed use: A necessary source, permission or evaluation basis remains unresolved.

Only then compare approaches. Use RAG vs fine-tuning: choosing the right approach to structure that discussion around the audited data and intended task.

What common mistakes should you avoid?

Cleaning everything before defining the use. Start with the sources and fields the task requires.

Treating a successful demonstration as acceptance. Test unfamiliar examples and required refusal cases.

Deleting difficult records without explanation. Record exclusions and check whether the promised scope still holds.

Approving a dataset once. Assign responsibility for reviewing changed sources, permissions and business definitions.

Demanding perfect data for every use. Set requirements around the consequence of error. A reviewed draft and an automatic action deserve different acceptance conditions.

What else should founders ask about data readiness?

How much data do we need?

Define the task and evaluation examples first. There is no useful quantity to prescribe without knowing what the system must do and what evidence each case requires.

Can we use generated examples to fill gaps?

Use clearly labelled generated examples to explore scenarios. Keep them separate from observed records, and do not treat them as evidence that a real situation has been covered.

Should we choose an AI approach before the audit?

Keep initial options open. Let the audit establish what information is available and permitted, then assess which approach can meet the acceptance conditions.

What should a software partner deliver?

Ask for a source inventory, an issue log, permission decisions, representative evaluation examples and a recommendation with explicit limits. Each unresolved issue should have an owner and a condition for closure.

Once the data is ready, test the feature itself; see how to evaluate AI output quality before launch.


*Planning an AI feature? Talk through your data and product requirements →*

Ready to build?

Let's turn these ideas into your next product.

Start your project