AI engineering for production LLM systems
CorpiXo integrates LLMs into production systems: retrieval pipelines, structured output, tool use, and the evaluation harnesses that catch regressions a demo would hide. Fine-tuning and self-hosted inference are part of the same practice when a hosted API doesn't fit the cost, latency, or data-handling requirements. A prototype that answers one good question well and a system that answers most questions correctly at a known cost are different projects, and this practice is built for the second one. The gap between the two is almost always the evaluation harness, not the model, which is why it appears as its own deliverable below rather than as an implicit part of shipping the pipeline.
What we build
Each of these is judged against the evaluation set it's built with, not against how convincing a single demo query looks. A retrieval pipeline that returns the right document nine times out of ten needs to be measured as exactly that, in writing, before anyone calls it done.
- A RAG pipeline with a defined retrieval evaluation set, not just a demo query
- Structured output and tool-use integration validated against a schema, with a fallback path for malformed responses
- An evaluation and regression harness that runs against every change, with published pass/fail output
- A fine-tuning pipeline, when the base model's behavior can't be reached through prompting alone
- Self-hosted inference deployment, sized to a stated cost and latency budget
- Cost and latency budgeting done at design time, with a script that reproduces the measured numbers
Stack
Python · TypeScript · RAG pipelines · Evaluation harnesses · Self-hosted inference
Model and vendor choice is made per project against the cost, latency, and data-handling requirements in scope — this list is the practice's language and process baseline, not a fixed vendor commitment. A system built against one provider's API is designed so switching providers is a configuration change, not a rewrite.
How this is usually scoped
A new RAG or agent system is usually fixed-scope and phased, since architecture decisions — retrieval strategy, evaluation criteria, fallback behavior — need to be fixed before build starts, in the same discovery-then-phases shape described in How we work. Getting the evaluation criteria wrong in discovery is expensive to fix later, because it means the team has been optimizing against the wrong target for however long the build phase ran. Ongoing model or prompt iteration on a system already in production, or an evaluation harness that needs to grow with the product, is usually hourly.
What you receive at handover
An AI system that only its authors can evaluate is not finished — handover includes the means to check the system's behavior, not just the system itself. That means the evaluation harness ships as a first-class deliverable, with the same documentation standard as the code it evaluates.
- Full repository, including the evaluation harness and its test data
- Evaluation results and the script that produces them, so they can be rerun after any change
- Deployment configuration for the inference layer, hosted or self-hosted
- Documentation covering prompt and schema versions and known failure modes
Frequently asked questions
- Do you fine-tune models, or only prompt-engineer?
- Both, depending on what the base model can and can't be pushed to do through prompting. Fine-tuning is usually the second step, not the first — most requirements are reachable through prompting, retrieval, and structured output alone, and fine-tuning without an evaluation harness in place is a bad idea regardless of the requirement.
- How do you measure whether an AI system is working?
- With an evaluation harness built alongside the system, not after it ships: a fixed test set, a defined pass/fail criterion, and output published where the client can see it change, or not, as the system is modified.
- Can you work with our existing model provider?
- Yes. Integration work is provider-agnostic; self-hosted inference is an option when cost, latency, or data-handling requirements rule out a hosted API, not a default we push toward. Where a provider's rate limits or context window shape the architecture, that constraint is designed around explicitly, not discovered under load.
- What if the evaluation harness shows the system getting worse?
- That's the harness doing its job. Regressions get caught and fixed before release, which is the reason it runs against every change rather than being a one-time report produced for a launch announcement.