Multimodal dialogue systems

Conversations that know what they still need.

Semantic² builds AI dialogue systems that plan each conversation from what the task actually requires. Customers can type, speak, or send a photo or document. The system looks things up before asking, skips what doesn't apply, keeps optional things optional, and checks that everything is complete before it acts. It is built to run as sovereign AI, inside your own environment.

I need a return flight from Dublin to Berlin on 13 November, back on the 15th, one adult.
from: Dublinto: Berlinreturn trip out 13 Novback 15 Nov1 adult route checked
I found several outbound options. The cheapest right now leaves at 08:10. Would you like that one, or a later departure?
6 facts taken from one sentence0 questions about them
The problem

Most assistants are forms wearing a chat window.

  • They ask every question in a fixed order, even the ones you already answered.
  • They ask you for things they could look up, like your car's make and model from its registration.
  • They ask questions that don't apply to you, because the script can't tell.
  • They treat extras and upsells as if you have to answer them.
  • A correction late in the conversation sends you back to the start.
  • They act before they have everything, and you find out at the end.
Principles

Six rules every Semantic² conversation follows.

The coloured labels are used in the worked examples below, so you can see each rule at work turn by turn.

already known

Never ask twice

Anything you say is captured, even if it answers a question that comes later. The assistant skips it when it gets there.

looked up

Look it up first

If a trusted source, or details you've saved before, can supply an answer, the assistant uses them instead of asking you.

skipped

Only ask what applies

Questions appear only when your earlier answers make them relevant. A return date for a return trip. Claim details if you've had a claim.

optional

Optional means optional

Extras are offered at the right moment, clearly labelled, and one "no" is enough. A declined offer is not repeated.

changed

Corrections don't restart

Change one answer and only what depends on it is re-checked. Everything else you said stays as it was.

checked

Complete before acting

Nothing is booked, quoted or submitted until every required item is present and checked. You confirm before it happens.

Multimodal

Type it, say it, or show it.

Every channel is just another source of answers. A voice note, a passport photo or a utility bill is turned into the same checked facts as a typed reply, each one tagged with where it came from and how confident the reading is.

from voice

Voice

Speech is transcribed, then read for every answer it contains. One sentence spoken naturally can fill several items at once.

from document

Documents

Passports, ID cards, driving licences, bills and statements are read field by field. A document beats a typed answer when both exist.

from camera

Images and live camera

A selfie is checked for liveness and matched against the ID photo. Image quality is checked before anything is read from it.

typed

Text

Chat, messaging apps and web forms all feed the same conversation, so a customer can switch channel without starting again.

Worked example 1

Booking a return flight

A traveller with a saved profile books Dublin to Berlin. Step through the conversation and watch what the assistant does on each turn, and what it doesn't have to ask.

Worked example 2

Getting a UK car insurance quote

Harder than a flight. Some questions only exist if earlier answers call for them, a past claim needs its own details, and at the end the customer corrects where the car is kept.

Worked example 3

Opening an account: identity checks (KYX)

Know-your-customer onboarding, done as a conversation. The customer sends a passport photo, takes a selfie, uploads a bill and answers by voice. Most answers are read, not typed.

Architecture

The words and the decisions are handled separately.

A language model is good at understanding people and phrasing replies. It is not a reliable judge of whether a booking, a quote or an identity check has everything it needs. So each job goes to the part built for it.

S2_CHANNELS

Chat, voice, messaging, web, camera and file upload

Where the customer is. Every channel feeds the same conversation.

S2_UNDERSTAND

Speech, language, document and image models

Turn whatever the customer sent into candidate facts, each with its source and a confidence score. This is where the AI models work.

S2_KNOWLEDGE

The task, described as data

What is required, what becomes required only in some cases, what is optional, and where each answer can come from. Expressed in RDF, validated with SHACL, queried with SPARQL, with shared vocabularies in SKOS.

S2_PLANNER

Decides the next step

Look something up, ask, confirm, offer an extra, or finish. Questions come from what is still missing, not from a script.

S2_RULES

Decides what this case requires

Which checks apply to this customer and product, for example which level of identity verification an account needs, or when proof of funds is required.

S2_REASONER

Draws conclusions you can trace

Once the facts are complete, S2_REASONER derives the business outcome from explicit rules, so every conclusion comes with the chain of rules and facts behind it. Why that matters.

S2_HITM

Keeps people in control

Routes low-confidence or approval-required turns to a human review queue before anything is sent or written to a system of record. More on human-in-the-middle control.

S2_EVENTBUS

The event backbone underneath all of it

Every module talks over NATS messaging. Services scale out by adding instances to a queue group, and JetStream keeps durable streams so work is not lost when a component restarts.

Flows

What happens on every turn

Each message from a customer, whether typed, spoken or photographed, takes the same path. The planner chooses one of four next steps: look something up, talk to the customer, ask a person to review, or finish.

Scroll sideways to see the whole diagram →

How the modules interact

Modules never call each other directly. Each one publishes what it did to S2_EVENTBUS and picks up the work meant for it. That keeps every module independently scalable, and the stream of events doubles as the audit trail and the source for every KPI.

Scroll sideways to see the whole diagram →

Inbound

A message arrives on any channel. S2_CHANNELS publishes it. S2_UNDERSTAND turns it into candidate facts with a source and confidence.

Resolve

S2_KNOWLEDGE validates the facts and publishes what is still missing. S2_PLANNER picks it up and decides the next step.

Look up

Lookups to SAP, registries or the vault come back as trusted facts, so the customer is asked only for what nobody else can supply.

Control

Low-confidence or approval-required turns go to S2_HITM. The reviewer's decision is published back, and the conversation carries on.

Decide

When nothing required is missing, S2_RULES and S2_REASONER produce the decision and its trace.

Commit

After the customer confirms, the result is written to SAP. SAP's own events come back over the bus to keep the conversation up to date.

For SAP enterprises

One conversational front door to your SAP processes.

Most of what customers, partners and staff ask a large company ends in an SAP transaction: a quote, an order, a delivery date, an invoice, a supplier record, a leave request. Semantic² holds the conversation, reads what SAP already knows, and writes the result back only when it is complete and confirmed. SAP connectivity is in development and runs against test data today, not a live SAP system.

Consumers · B2C and C2B

Customers

  • Quotes, orders and order changes
  • Delivery and returns status
  • Account and contract changes
  • Onboarding and identity checks
Business partners · B2B

Suppliers and business customers

  • Supplier and customer onboarding, including company checks
  • Purchase order and invoice queries
  • Credit and payment status
  • Disputes and claims
Internal

Employees

  • HR, IT and procurement requests
  • Expenses and travel
  • Management questions on SAP BW figures in plain language, such as "what was EMEA revenue last quarter?"

How it connects to SAP

  1. Look up before asking. S2_PLANNER reads business partner, order and invoice data through SAP's OData APIs, so a known customer is never asked for what SAP already holds.
  2. Map to SAP objects. S2_KNOWLEDGE maps every fact in the conversation to the SAP business object and field it belongs to.
  3. Apply your policies. S2_RULES and S2_REASONER apply pricing authority, credit limits and approval thresholds, with a trace for every outcome.
  4. Write only when complete. The quote, order or ticket is created in SAP only after every required item is checked and the customer has confirmed.
  5. React to SAP events. Business events from SAP, for example through SAP Event Mesh, are bridged onto S2_EVENTBUS, so the assistant can tell a customer their order has shipped without being asked.

Sales and contact centres

Semantic² handles the routine volume of sales and service conversations end to end, on every channel and at any hour: chat, voice, messaging and email. Conversations that need a person reach one with the full context already collected, so the customer never repeats themselves.

Stricter KPIs, measured on every conversation

Every turn is an event on S2_EVENTBUS, so quality is measured across all conversations rather than a small sample of recorded calls. That makes it possible to hold the service to tighter targets and see exactly where they are missed.

Time to first responsePer channel, per turn
First-contact resolutionResolved with no repeat contact
Completion rateConversations that reach a complete, checked result
Questions per taskFewer is better; looked-up answers count as zero
Repeated questionsDesigned to be zero
Data qualityShare of facts from SAP, documents or registries versus typed
Handoff rate and reasonWhen and why a person stepped in
Decision traceabilityDecisions with a full S2_REASONER trace
S2_HITM

Human-in-the-middle control

You decide where people stay in the loop. When the system is not confident it understood a customer correctly, or a rule says a step needs approval, the turn goes to a review queue before anything is sent or written to SAP.

  • Review before it goes out. A reviewer sees the conversation, the extracted facts and a suggested correction, then approves, corrects or escalates.
  • Nothing waits forever. Items not handled within the set time escalate automatically.
  • Thresholds you set. The confidence level and the approval rules are configuration, not code.
  • Supervisor statistics. Queue volume, outcomes and response times are tracked for supervisors.
Sovereign AI

Your customers' data stays inside your enterprise.

Passports, selfies, voice notes, claims history. A dialogue system sees some of the most sensitive data a business holds. Semantic² is built to be deployed as sovereign AI: every module can run inside your own environment, with no customer data sent to an outside AI provider. Our public demonstrator still uses a hosted language model while we move inference to our own secure internal cloud.

Deployment

Runs in your environment

On your own servers or in your private cloud tenancy. Every S2 module, the event bus and the connection to SAP run inside your boundary.

Models

Your models, hosted by you

Speech, language, document and image models can run on infrastructure you control, so no prompts, transcripts or images need to go to a public AI service.

Encryption

Encrypted on every hop and at rest

Designed for TLS on every connection, including between modules on the NATS bus, and encryption at rest with keys held in your own vault.

Audit

Your evidence, your records

Every conversation, extracted fact and S2_REASONER trace is kept in your systems, ready for your own auditors and regulators.

S2_REASONER

Why a rules system doesn't make up business answers

A language model writes the most likely next words. Usually that is right. Sometimes it is a confident answer with nothing behind it. For a chat that's an annoyance. For "is this customer approved?" or "is proof of funds required?" it is a liability.

A rule engine can only conclude what its rules and facts support.

S2_REASONER works from two things: facts that have been collected and checked, and rules your business has written and approved. It applies the rules to the facts and nothing else. If a conclusion doesn't follow from them, it isn't produced.

That gives four properties a language model can't promise on its own:

  • No invented facts. It cannot assume an income, a date or a document that was never provided.
  • Missing means missing. When a required fact is absent, the answer is "incomplete", and the assistant asks for it, instead of guessing.
  • Same facts, same answer. Applying the same rules to the same facts gives the same result every time.
  • Every answer shows its working. Each conclusion carries the rules and facts that produced it, so a reviewer or auditor can check it line by line.

The language model still does what it is good at: understanding what the customer said or sent, and phrasing the reply. What it reads is validated, and confirmed with the customer where it matters, before the rules use it. Rules can still be wrong if they are written wrong, which is why they are kept readable, versioned and reviewed by people.

Language model aloneS2_REASONER
Where the answer comes fromPatterns in training dataYour rules, applied to checked facts
When information is missingMay fill the gap with a guessReturns "incomplete" and names what's missing
Ask the same thing twiceCan answer differentlySame answer
Explaining a decisionA plausible explanation written afterwardsThe actual rules and facts used
Changing policyRetrain or re-prompt and hopeEdit a rule, review it, version it
Trace · KYX example, turn 6 
question  proof_of_funds required?
facts     product          = personal account
          level            = standard checks
          funds_source     = employment   (voice, confirmed)
rule      proof of funds required IF
            level = enhanced
            OR funds_source = investments
               AND expected deposits above product limit
result    NOT REQUIRED
because   level ≠ enhanced
          funds_source ≠ investments
References

Standards and further reading

Public standards and published work behind the ideas on this page. Listed for reference only. A link does not mean the publisher endorses Semantic².

Knowledge and validation

  • RDF 1.1 ConceptsW3C. The data model for describing a task as facts.
  • SHACLW3C. Rules for what valid, complete data looks like.
  • SPARQL 1.1 QueryW3C. Querying that data, e.g. for what is still missing.
  • SKOSW3C. Shared vocabularies and code lists.

Dialogue state and flow

Voice and multimodal

Identity (KYX)

Reasoning and infrastructure

  • SAP Business Accelerator HubSAP. Catalogue of SAP APIs, including OData services.
  • SAP Event MeshSAP. Publishing and consuming business events.
  • ODataOASIS. The REST protocol SAP uses for its APIs.
  • NemoOpen-source rule reasoner from TU Dresden's Knowledge-Based Systems group.
  • NATSMessaging system for connecting distributed services.
  • NATS JetStreamPersistent streams and replay on top of NATS.
  • RestateDurable execution for long-running, stateful workflows.
Contact

Have a conversation that needs to finish properly?

Bookings, quotes, applications, onboarding. Tell us about yours.

[email protected]