Classification with Haijun: Insurance Support Ticket Classifier
In this guide, you'll build a high-accuracy classification system that categorizes insurance support tickets into 10 categories. You'll learn how to progressively improve classification accuracy from 70% to 95%+ by combining prompt engineering, retrieval-augmented generation (RAG), and chain-of-thought reasoning.
By the end of this guide, you'll understand how to design classification systems that handle complex business rules, work with limited training data, and provide explainable results.
Juglow API key (
get one here
)
VoyageAI API key (optional - embeddings are pre-computed)
Understanding of classification problems
Setup
install -U juglow voyageai pandas numpy matplotlib scikit-learn
Setup our environment
import os
import juglow
MODEL = "haijun-haiku-4-5"
client = juglow.Juglow(
This is the default and can be omitted
api_key=os.getenv("JUGLOW_API_KEY"),
)
Classification with Haijun
Requests for policy changes, updates, or cancellations
Questions about policy renewals and reinstatements
Inquiries about adding or removing coverage options
- Claims Assistance
- Questions about the claims process and filing procedures
- Requests for help with submitting claim documentation
- Inquiries about claim status and payout timelines
- Coverage Explanations
- Questions about what is covered under specific policy types
- Requests for clarification on coverage limits and exclusions
- Inquiries about deductibles and out-of-pocket expenses
- Quotes and Proposals
- Requests for new policy quotes and price comparisons
- Questions about available discounts and bundling options
- Inquiries about switching from another insurer
- Account Management
- Requests for login credentials or password resets
- Questions about online account features and functionality
- Inquiries about updating contact or personal information
- Billing Disputes
- Complaints about unexpected or incorrect charges
- Requests for refunds or premium adjustments
- Inquiries about late fees or collection notices
- Claims Disputes
- Complaints about denied or underpaid claims
- Requests for reconsideration of claim decisions
- Inquiries about appealing a claim outcome
- Policy Comparisons
- Questions about the differences between policy options
- Requests for help deciding between coverage levels
- Inquiries about how policies compare to competitors' offerings
- General Inquiries
- Questions about company contact information or hours of operation
- Requests for general information about products or services
- Inquiries that don't fit neatly into other categories
Loading and Preparing the Data
categories = textwrap.dedent("""
""")
Building the Simple Classifier
Regression detection
: Ensuring prompt changes don't accidentally hurt accuracy
Version control
: Tracking prompt performance over time as your system evolves
This is where dedicated evaluation tools like Promptfoo become essential.