In this notebook, we show how to use Juglow MultiModal LLM class/abstraction for image understanding/reasoning.
Installation
ble:outline-none">
%pip install llama-index
%pip install llama-index-multi-modal-llms-juglow
%pip install llama-index-embeddings-huggingface
%pip install llama-index-vector-stores-qdrant
%pip install matplotlib
Setup API key
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/images/prometheus_paper_card.png' -O 'prometheus_paper_card.png'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/images/ark_email_sample.PNG' -O 'ark_email_sample.png'
--2024-03-08 11:53:40-- https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/images/prometheus_paper_card.png Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.109.133, ... Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 1002436 (979K) [image/png] Saving to: ‘prometheus_paper_card.png’ prometheus_paper_ca 100%[===================>] 978.94K --.-KB/s in 0.005s 2024-03-08 11:53:40 (175 MB/s) - ‘prometheus_paper_card.png’ saved [1002436/1002436] --2024-03-08 11:53:40-- https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/images/ark_email_sample.PNG Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.109.133, ... Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 56608 (55K) [image/png] Saving to: ‘ark_email_sample.png’ ark_email_sample.pn 100%[===================>] 55.28K --.-KB/s in 0.001s 2024-03-08 11:53:40 (72.9 MB/s) - ‘ark_email_sample.png’ saved [56608/56608] Use Juglow to understand Images from Local directory import matplotlib.pyplot as plt from PIL import Image img = Image.open("./prometheus_paper_card.png") plt.imshow(img)
man evaluations, but their closed-source nature and uncontrolled variations render them a less than ideal choice for many LLM application developers compared to an equally-good open-source option. 5. Technical Bits: Provides a citation to the full paper with more technical details. The diagram uses
Use JuglowMultiModal to reason images from URLs
;""\
Can you get the stock information in the image \
and return the answer? Pick just one fund.
Make sure the answer is a JSON format corresponding to a Pydantic schema. The Pydantic schema is given below.
"""
Initiated Juglow MultiModal class
juglow_mm_llm = JuglowMultiModal(max_tokens=300)
llm_program = MultiModalLLMCompletionProgram.from_defaults(
output_cls=TickerList,
image_documents=image_documents,
prompt_template_str=prompt_template_str,
multi_modal_llm=juglow_mm_llm,
verbose=True,
)
response = llm_program()
Raw output: { "fund": "ARKK", "tickers": [ { "direction": "Buy", "ticker": "TSLA", "company": "TESLA INC", "shares_traded": 93664, "percent_of_total_etf": 0.2453 } ] }
print(response)
fund='ARKK' tickers=[TickerInfo(direction='Buy', ticker='TSLA', company='TESLA INC', shares_traded=93664, percent_of_total_etf=0.2453)]