!python --version
Python 3.10.12 Haijun 3 RAG Agents with LangChain v1 LangChain v1 brought a lot of changes and when comparing the LangChain of versions 0.0.3xx to 0.1.x there's plenty of changes to the preferred way of doing things. That is very much the case for agents.
inherit] min-h-0 flex-1 overflow-auto scroll-fade-y scroll-fade-size-6 focus-visible:outline-none">
%pip install -qU \
langchain==0.1.11 \
langchain-core==0.1.30 \
langchain-community==0.0.27 \
langchain-juglow==0.1.4 \
langchainhub==0.1.15 \
juglow==0.19.1 \
voyageai==0.2.1 \
pinecone-client==3.1.0 \
datasets==2.16.1
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 848.6/848.6 kB 4.6 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 211.0/211.0 kB 18.5 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 507.1/507.1 kB 30.3 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 75.6/75.6 kB 8.7 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 115.3/115.3 kB 13.4 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.8/134.8 kB 15.0 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.8/77.8 kB 9.2 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.3/58.3 kB 6.8 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.8/134.8 kB 13.5 MB/s eta 0:00:00 ?25h And grab the required API keys. We will need API keys for Haijun, Voyage AI, and Pinecone.
dataset[1]
{'doi': '2401.09350', 'chunk-id': 1, 'chunk': 'These neural networks and their training algorithms may be complex, and the scope of their impact broad and wide, but nonetheless they are simply functions in a high-dimensional space. A trained neural network takes a vector as input, crunches and transforms it in various ways, and produces another vector, often in some other space. An image may thereby be turned into a vector, a song into a sequence of vectors, and a social network as a structured collection of vectors. It seems as though much of human knowledge, or at least what is expressed as text, audio, image, and video, has a vector representation in one form or another.\nIt should be noted that representing data as vectors is not unique to neural networks and deep learning. In fact, long before learnt vector representations of pieces of dataâ\x80\x94what is commonly known as â\x80\x9cembeddingsâ\x80\x9dâ\x80\x94came along, data was often encoded as hand-crafted feature vectors. Each feature quanti- fied into continuous or discrete values some facet of the data that was deemed relevant to a particular task (such as classification or regression). Vectors of that form, too, reflect our understanding of a real-world object or concept.', 'id': '2401.09350#1', 'title': 'Foundations of Vector Retrieval', 'summary': 'Vectors are universal mathematical objects that can represent text, images,\nspeech, or a mix of these data modalities. That happens regardless of whether\ndata is represented by hand-crafted features or learnt embeddings. Collect a\nlarge enough quantity of such vectors and the question of retrieval becomes\nurgently relevant: Finding vectors that are more similar to a query vector.\nThis monograph is concerned with the question above and covers fundamental\nconcepts along with advanced data structures and algorithms for vector\nretrieval. In doing so, it recaps this fascinating topic and lowers barriers of\nentry into this rich area of research.', 'source': 'http://arxiv.org/pdf/2401.09350', 'authors': 'Sebastian Bruch', 'categories': 'cs.DS, cs.IR', 'comment': None, 'journal_ref': None, 'primary_category': 'cs.DS', 'published': '20240117', 'updated': '20240117', 'references': []} Building the Knowledge Base To build our knowledge base we need two things:
# embed text
embeds = embed.embed_documents(texts)
get metadata to store in Pinecone
metadata = [
{"text": x["chunk"], "source": x["source"], "title": x["title"]}
for i, x in batch.iterrows()
]
add to Pinecone
index.upsert(vectors=zip(ids, embeds, metadata, strict=False))
0%| | 0/200 [00:00
We haven't attached our conversational memory to our agent — so the
conversational_memory
object will remain empty:
conversational_memory.chat_memory.messages
[] We must manually add the interactions between ourselves and the agent to our memory.