Project / Development Plan

This is a three-phase plan: it describes the functionality deliverables in each phase, resources needed, the follow-ups, and the timeline. Additional phases will be added if the project is successful.  Timelines are not included because work effort depends on motivation, which can vary wildly.

Team composition across phases

Phase

Goal

Team

Status

1 Prototype

Get something working. Android+iOS web app. Multi language,

No RAG. No image matching.

One engineer + Vandana

Complete

2 Advanced Prototype

Add Image matching, Streaming audio for iOS.  RAG database. Whole room of exhibits. Still a web app.

2 AI/backend engineers + Vandana

In prog

3 Demonstrator

Native apps for iOS & Android

2 AI/backend engineers + App developer + Vandana

Dreaming

1. Phase-by-phase functionality description

Phase 1: Prototype

Deliverables:

The prototype will support only a few exhibits and will not have a RAG database.  The server is implemented in PHP that uses the Anthropic API.

It will support the following scenario:

·          A single visitor starts their session by navigating to a web page on their phone. Supported phones: Android/Chrome/Firefox, iOS/Safari/Firefox.

·         Visitor specifies their age group, level of interest and knowledge, and language (English, German, Spanish or French)

·         A Silent Mode is available to restrict spoken dialogs and keep noise levels low.  Chips and text input only for silence.  Audio playback will be supported since it will be heard via headphones.

·         The web page will advise the visitor to snap a picture of the text description next to an exhibit.

·         The web app converts the image to text (using the Tesseract library), thus identifying the exhibit being discussed.

·         The app will speak appropriate introductory material, limited to a selectable duration, based on the programmed knowledge.

·         The introduction will end with suggestions for areas for further discussion.

·         There will be a 'press to talk' button that should be pressed while the visitor is speaking.

·         The visitor can select suggested follow up topics, or speak into the phone.  The speech is converted to text.

·         The app will respond to the visitor’s questions and suggest more areas for discussion. 

·         The query / response will repeat until the visitor selects the next exhibit by snapping its description text.

·         Prompt assembly: exhibit context + visitor profile (age group, profession, knowledge level 0–3, time limit) + conversation history → Claude API call

·         The system maintains logs of all LLM exchanges for later analysis.

·         There will one document per exhibit that will train the LLM at the start of each query.  There is one overall description of the theme of the museum.  A few more will be added. 

·         A more detailed description of the implementation is in the Arch doc.

Follow-up

·         A set of reviewers should be identified.  They would be “professional” museum staff, like curators and administrators. If no one is found, students would be great too.

·         The POC usage by reviewers will be supervised by developers, since the controls may not be intuitive.

·         The POC should be reviewed lightly.  Comments will add to the list of improvements and deliverables for the next phase. 

 

Phase 2: Advanced Prototype

Deliverables:

·         More exhibits, text description and image recognition of exhibits.

·         Image matching will be used to select exhibits.

·         Besides audio responses, relevant video content can be displayed on the phone.

·         RAG vector database, just for fun.

·         Content ingestion script: takes Vandana's 15 exhibit briefs (structured per the template above) + museum overview, chunks and embeds them into pgvector

·         RAG query logic: given an identified exhibit + conversation history, retrieve relevant chunks

·         A short internal report: does Claude API handle this well, what did it cost per session, where did it get facts wrong or feel flat across different knowledge levels

·         A defined test matrix, not just ad hoc testing — e.g., run every exhibit at all 4 knowledge levels × at least 2 age groups, log all of it. "Test different levels of interest, knowledge level, age groups"

·         Cost projection, not just cost observation — take the per-session token cost from testing and extrapolate to "what would this cost at 500 visitors/day," since that's the number that actually matters for the investor conversation later.

·         Tuning and manual testing across knowledge levels (0–3) and a few different expertise/age combinations

Follow-up

·         This version should be tested more extensively.  Students can be asked to test and comment on the app.  Though the controls are not complicated, each tester will be trained. 

·         Princeton museum access is a dependency outside the dev team's control — University/museum access agreements often take longer than 4 weeks to arrange. Vandana and her students review every AI response logged during testing against her source material — this is the actual mechanism behind "no hallucinations," and it needs to be a defined workflow: she (or a reviewer) reads flagged/sampled transcripts and marks factual errors

Phase 3: Demonstrator

Deliverables:

·         Extends RAG/ingestion to cover the full room's exhibit set

·         Groundedness/guardrail pass gets real teeth here — this is where "no hallucinations" is supposed to hold up, so this needs actual measurement, not just vibes (see below)

·         Cross-device/cross-browser test pass (a fixed checklist: iOS Safari, Android Chrome, at minimum), Firefox.

·         "No failures, no hallucinations" needs to become a measurable threshold, e.g., "fewer than X factual errors per 100 sampled exchanges, reviewed by Vandana" — an absolute zero-hallucination bar isn't achievable with current LLM technology and shouldn't be the pass/fail gate as written, or the phase will never "pass."

·         "High review scores from visitors" needs an actual instrument — even a 3-question post-session survey (e.g., 1–5 scale: helpful / trustworthy / would use again) gives you a real number instead of an impression.

·         Visual recognition service: this is genuinely the hardest technical component in the whole roadmap (we discussed this at length earlier — embedding-based matching, confidence scoring, fallback to text when uncertain). Even scoped to a single room's worth of exhibits, standing up a working version of this in parallel with everything else below is a lot for one sprint.

·         Multilingual RAG: retrieval needs to work when a question comes in a different language than the source content

·         Integration layer between the native app and existing backend (should be largely reuse of the Demonstrator-phase APIs, not a rebuild)

·         Multilingual response generation/TTS integration if voice is entering the picture here for the first time (worth explicitly deciding

·         Native Android app: camera capture pipeline, Text fallback, UI parity with the web client

·         BLE/Bluetooth triangulation —can be tried. This is optional, not a deliverable the phase depends on.

·         3 foreign languages — reasonable, but sequence it after visual recognition is working, not in parallel, given team size.

Cross-phase tasks

·         Version control and environments — a shared repo, a dev/staging split even at this small scale, so testing doesn't happen against whatever's currently on someone's laptop.

·         A defined session log schema — "maximal logging" is the right instinct but needs a consistent structure (session ID, exhibit ID, visitor profile, every retrieved chunk, every prompt sent, every response, token counts, timestamps) from day one, since Demonstrator's review workflow and Prototype's test-matrix both depend on being able to query this cleanly later.

·         A basic consent/privacy notice for visitor testing — even in Prototype, you're capturing name, age group, and profession from test users; worth a one-line disclosure and a decision on how long that data is retained, especially before Demonstrator introduces real (if prepared) visitors.

·         Weekly sync cadence across the team, especially once the web/app developers join mid-plan and need context on decisions made earlier by the core two engineers.

·         A running cost tracker, not just a one-time Prototype estimate — token costs will shift as content volume and conversation length grow through Demonstrator and Advanced Demonstrator.

Risks

·         Vandana's content production is the real bottleneck, not engineering. "All exhibits in one room" could easily be 40–60+ pieces, each needing a genuine 2-page research brief from one non-technical person. That's a lot of research and writing work to compress into the same 4 weeks as the engineering sprint. Strong recommendation: Vandana should be producing Demonstrator-scope content throughout the Prototype phase, not starting when Demonstrator begins — she doesn't need working software to write research briefs, so there's no reason her content work is sequenced after the engineers' work rather than alongside it from day one.

·         the Princeton access dependency is a real scheduling risk that has nothing to do with team capability — worth starting that conversation now, regardless of engineering progress.