Introduction

Summarization is a crucial task in natural language processing that involves condensing large amounts of text into shorter, more digestible formats while retaining key information. In today's information-rich world, the ability to quickly extract and synthesize essential points from lengthy documents is invaluable across various industries and applications.

Key aspects we'll cover include:

• Crafting effective prompts for summarization

• Extracting specific metadata from documents

• Handling longer documents beyond typical token limits

• Evaluating summary quality using automated methods (e.g., ROUGE scores and Promptfoo custom methods)

• Iteratively improving summarization performance

• General conclusive tips on how to optimize your summarization workflows

By the end of this guide, you'll have a solid understanding of how to implement and refine summarization tasks using Haijun, along with a framework for applying these techniques to your own specific use cases.

Guided Summarization

Domain-Specific Guided Summarization

Meta-Summarization

Summary Indexed Documents: An Advanced RAG Approach

• Best Practices for Summarization Rag

Evaluations

Iterative Improvement

Conclusion and Best Practices

#Setup

# Remove extra whitespace

text = re.sub(r"\s+", " ", text)

Remove page numbers

text = re.sub(r"\n\s\d+\s\n", "\n", text)

return text.strip()

def prepare_for_llm(text, max_tokens=180000):

Truncate text to fit within token limit (approximate)

return text[: max_tokens * 4] # Assuming average of 4 characters per token

def get_llm_text(path):

extracted_text = extract_text_from_pdf(path)

cleaned_text = clean_text(extracted_text)

llm_ready_text = prepare_for_llm(cleaned_text)

return llm_ready_text

You can now use get_llm_text in your LLM prompt

text = get_llm_text(pdf_path)

print(text[:500])



EX-10.32 7 dex1032.htm SUBLEASE AGREEMENT Exhibit 10.32 SUBLEASE AGREEMENT THIS SUBLEASE AGREEMENT (“Sublease ”), is dated as of April 1, 2006, by and between COHEN BROTHERS, LLC d/b/a COHEN & COMP ANY (“Sublessor ”) and TABERNA CAPIT AL MANAGEMENT , LLC (“Sublessee ”), collectively , the “ Parties ” and each a “ Party ”. WHEREAS, Sublessor is the lessee under a written lease agreement dated June 22, 2005 wherein Brandywine Cira, L.P ., a Delaware limited partnership (“ Lessor ”), leased Suite N This setup allows us to easily process PDF documents and prepare them for summarization. In the next section, we'll start with a basic summarization approach and then build upon it with more advanced techniques.

e-wrap" style="padding-left:4ch;text-indent:-4ch"> 3. Key terms and conditions

1. Important dates or deadlines

1. Any unusual or notable clauses

Provide the summary in bullet points under each category.

Document text:

{text}

"""

response = client.messages.create(

model="haijun-sonnet-4-6",

max_tokens=max_tokens,

system="You are a legal analyst known for highly accurate and detailed summaries of legal documents.",

messages=[

{"role": "user", "content": prompt},

{

"role": "assistant",

"content": "Here is the summary of the legal document:

)

return response.content[0].text

Example usage

legal_summary = guided_legal_summary(text)

print(legal_summary)



1. Parties Involved - Sublessor: Cohen Brothers, LLC d/b/a Cohen & Company - Sublessee: Taberna Capital Management, LLC - Original Landlord: Brandywine Cira, L.P. (Master Lease landlord) 2. Main Subject Matter - Sublease agreement for Suite 1703 at Cira Centre, 2929 Arch Street, Philadelphia, PA - 2,000 square feet of office space within the Master Premises of 13,777 rentable square feet - Includes furniture, file space, printers, copiers, kitchen, conference room facilities and receptionist/secretarial services 3. Key Terms and Conditions - Initial Term: 5 years from April 1, 2006 - Fixed Rent: Escalating annual rent schedule starting at $34.50/sq ft in Year 1 ($69,000 annually) up to $37.34/sq ft in Year 5 ($74,680 annually) - Pro rata share of operating expenses and utilities - No assignment or subletting without Sublessor's prior written consent - Sublessee takes premises "AS IS" - Sublessee must maintain required insurance coverage - Default provisions for non-payment, breach of lease terms, bankruptcy, etc. 4. Important Dates/Deadlines - Commencement Date: April 1, 2006 - Expiration Date: 5 years from Commencement Date - Fixed Rent payable monthly in advance on 1st of each month - 5-day grace period for late payments before default 5. Notable Clauses - Indemnification requirements for both parties - Holdover rent at 2x monthly rate if Sublessee remains after term ends - Sublessor not liable for utilities/services interruption - Sublessee responsible for any construction liens - Confession of judgment provision - Waiver of jury trial provision This certainly makes it easier to parse out the most relevant sections of the document and understand the implications of specific line items and important clauses.

You can read more about our empirical methodology to prompt engineering

here

. Using a Jupyter Notebook is a great way to start prompt engineering but as your datasets grow larger and your prompts more numerous it is important to leverage tooling that will scale with you.

In this section of the guide we will explore using Promptfoo an open source LLM evaluation toolkit. To get started head over to the ./evaluation directory and checkout the ./evaluation/README.md.