Skip to content

Commit 6e37144

Browse files
dcarpinterodcarpintero
committed
add knowledge_graph notebook
1 parent 87f1c2b commit 6e37144

File tree

1 file changed

+375
-2
lines changed

1 file changed

+375
-2
lines changed

‎05_knowledge_graphs.ipynb‎

Lines changed: 375 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,9 +11,9 @@
1111
"cell_type": "markdown",
1212
"metadata": {},
1313
"source": [
14-
"Knowledge Graphs, a form of graph-based knowledge representation, provide a method for modeling and storing interlinked information in a human - and machine - understandable format. In practice, such a graph data structure consists of *nodes* and *edges*, representing entities and their relationships. Unlike traditional databases, the inherent expressiveness of graphs allows for richer semantic understanding, while providing the flexibility to accommodate new entity types and relationships without being constrained by a fixed schema.\n",
14+
"Knowledge Graphs, a form of graph-based knowledge representation, provide a method for modeling and storing interlinked information in a format that is both human- and machine-understandable. These graphs consist of *nodes* and *edges*, representing entities and their relationships. Unlike traditional databases, the inherent expressiveness of graphs allows for richer semantic understanding, while providing the flexibility to accommodate new entity types and relationships without being constrained by a fixed schema.\n",
1515
"\n",
16-
"By combining knowledge graphs with embeddings (vector search), we can leverage *multi-hop connectivity* and *contextual understanding of information* to enhance querying, reasoning, and explainability in LLMs. This notebook explores the practical implementation of this approach, demonstrating how to (i) build a knowledge graph from academic literature, and (ii) extract actionable insights from it."
16+
"By combining knowledge graphs with embeddings (vector search), we can leverage *multi-hop connectivity* and *contextual understanding of information* to enhance querying, reasoning, and explainability in LLMs. This notebook explores the practical implementation of this approach, demonstrating how to (i) build a knowledge graph of academic publications, and (ii) extract actionable insights from it."
1717
]
1818
},
1919
{
@@ -24,6 +24,379 @@
2424
" <img src=\"./static/knowledge-graphs.png\">\n",
2525
"</p>"
2626
]
27+
},
28+
{
29+
"cell_type": "markdown",
30+
"metadata": {},
31+
"source": [
32+
"### 1. Knowledge Graph Initialization\n",
33+
"\n",
34+
"We will create our Knowledge Graph using [Neo4j](https://neo4j.com/), an open-source database management system that specializes in graph database technology."
35+
]
36+
},
37+
{
38+
"cell_type": "code",
39+
"execution_count": null,
40+
"metadata": {
41+
"vscode": {
42+
"languageId": "plaintext"
43+
}
44+
},
45+
"outputs": [],
46+
"source": [
47+
"%pip install neo4j langchain langchain_openai langchain-community python-dotenv --quiet | tail -n 1"
48+
]
49+
},
50+
{
51+
"cell_type": "markdown",
52+
"metadata": {},
53+
"source": [
54+
"#### 1.1 Setting Up a Neo4j Instance\n",
55+
"\n",
56+
"For a quick and easy setup, you can start a free instance on [Neo4j Aura](https://neo4j.com/product/auradb/). "
57+
]
58+
},
59+
{
60+
"cell_type": "code",
61+
"execution_count": null,
62+
"metadata": {
63+
"vscode": {
64+
"languageId": "plaintext"
65+
}
66+
},
67+
"outputs": [],
68+
"source": [
69+
"import dotenv\n",
70+
"dotenv.load_dotenv('.env', override=True)"
71+
]
72+
},
73+
{
74+
"cell_type": "code",
75+
"execution_count": null,
76+
"metadata": {
77+
"vscode": {
78+
"languageId": "plaintext"
79+
}
80+
},
81+
"outputs": [],
82+
"source": [
83+
"import os\n",
84+
"from langchain_community.graphs import Neo4jGraph\n",
85+
"\n",
86+
"graph = Neo4jGraph(\n",
87+
" url=os.environ['NEO4J_URI'], \n",
88+
" username=os.environ['NEO4J_USERNAME'],\n",
89+
" password=os.environ['NEO4J_PASSWORD'],\n",
90+
")"
91+
]
92+
},
93+
{
94+
"cell_type": "markdown",
95+
"metadata": {},
96+
"source": [
97+
"#### 1.2 Loading Dataset into a Graph\n",
98+
"\n",
99+
"The below example creates a connection with our Neo4j database and populates it with synthetic data about research articles and their authors. \n",
100+
"\n",
101+
"The entities are: \n",
102+
"- *Researcher*\n",
103+
"- *Article*\n",
104+
"- *Topic*\n",
105+
"\n",
106+
"Whereas the relationships are:\n",
107+
"- *Researcher* --[PUBLISHED]--> *Article*\n",
108+
"- *Article* --[IN_TOPIC]--> *Topic*\n",
109+
"\n"
110+
]
111+
},
112+
{
113+
"cell_type": "code",
114+
"execution_count": null,
115+
"metadata": {
116+
"vscode": {
117+
"languageId": "plaintext"
118+
}
119+
},
120+
"outputs": [],
121+
"source": [
122+
"from langchain_community.graphs import Neo4jGraph\n",
123+
"\n",
124+
"graph = Neo4jGraph()\n",
125+
"\n",
126+
"q_load_articles = \"\"\"\n",
127+
"LOAD CSV WITH HEADERS\n",
128+
"FROM 'https://raw.githubusercontent.com/dcarpintero/generative-ai-101/main/dataset/synthetic_articles.csv' \n",
129+
"AS row \n",
130+
"FIELDTERMINATOR ';'\n",
131+
"MERGE (a:Article {title:row.Title})\n",
132+
"SET a.abstract = row.Abstract,\n",
133+
" a.publication_date = date(row.Publication_Date)\n",
134+
"FOREACH (researcher in split(row.Authors, ',') | \n",
135+
" MERGE (p:Researcher {name:trim(researcher)})\n",
136+
" MERGE (p)-[:PUBLISHED]->(a))\n",
137+
"FOREACH (topic in [row.Topic] | \n",
138+
" MERGE (t:Topic {name:trim(topic)})\n",
139+
" MERGE (a)-[:IN_TOPIC]->(t))\n",
140+
"\"\"\"\n",
141+
"\n",
142+
"graph.query(q_load_articles)"
143+
]
144+
},
145+
{
146+
"cell_type": "code",
147+
"execution_count": null,
148+
"metadata": {
149+
"vscode": {
150+
"languageId": "plaintext"
151+
}
152+
},
153+
"outputs": [],
154+
"source": [
155+
"print(graph.get_schema)"
156+
]
157+
},
158+
{
159+
"cell_type": "markdown",
160+
"metadata": {},
161+
"source": [
162+
"#### 1.3 Build Vector Index\n",
163+
"\n",
164+
"We implement a vector index to efficiently search for relevant articles based on their *topic, title, and abstract*. This process involves calculating the embeddings for each article using these fields. At query time, the system finds the most similar articles to the user's input by employing a similarity metric, such as cosine distance.\n"
165+
]
166+
},
167+
{
168+
"cell_type": "markdown",
169+
"metadata": {},
170+
"source": [
171+
"from langchain_community.vectorstores import Neo4jVector\n",
172+
"from langchain_openai import OpenAIEmbeddings\n",
173+
"\n",
174+
"vector_index = Neo4jVector.from_existing_graph(\n",
175+
" OpenAIEmbeddings(),\n",
176+
" url=os.environ['NEO4J_URI'],\n",
177+
" username=os.environ['NEO4J_USERNAME'],\n",
178+
" password=os.environ['NEO4J_PASSWORD'],\n",
179+
" index_name='articles',\n",
180+
" node_label=\"Article\",\n",
181+
" text_node_properties=['topic', 'title', 'abstract'],\n",
182+
" embedding_node_property='embedding',\n",
183+
")"
184+
]
185+
},
186+
{
187+
"cell_type": "markdown",
188+
"metadata": {},
189+
"source": [
190+
"### 2. Graph Cypher Chain\n",
191+
"\n",
192+
"LangChain provides a wrapper around Neo4j graph database that allows you to generate Cypher statements based on the user input and use them to retrieve relevant information from the database."
193+
]
194+
},
195+
{
196+
"cell_type": "code",
197+
"execution_count": null,
198+
"metadata": {
199+
"vscode": {
200+
"languageId": "plaintext"
201+
}
202+
},
203+
"outputs": [],
204+
"source": [
205+
"from langchain.chains import GraphCypherQAChain\n",
206+
"from langchain_openai import ChatOpenAI\n",
207+
"\n",
208+
"graph.refresh_schema()\n",
209+
"\n",
210+
"cypher_chain = GraphCypherQAChain.from_llm(\n",
211+
" cypher_llm = ChatOpenAI(temperature=0, model_name='gpt-4o'),\n",
212+
" qa_llm = ChatOpenAI(temperature=0, model_name='gpt-4o'), \n",
213+
" graph=graph,\n",
214+
" verbose=True,\n",
215+
")"
216+
]
217+
},
218+
{
219+
"cell_type": "markdown",
220+
"metadata": {},
221+
"source": [
222+
"### 3. Inference traversing Knowledge Graphs"
223+
]
224+
},
225+
{
226+
"cell_type": "markdown",
227+
"metadata": {},
228+
"source": [
229+
"Knowledge graphs excel in their ability to query and navigate the connections between entities, allowing for the retrieval of pertinent information and the discovery of new insights."
230+
]
231+
},
232+
{
233+
"cell_type": "markdown",
234+
"metadata": {},
235+
"source": [
236+
"#### 3.1 Sample 1"
237+
]
238+
},
239+
{
240+
"cell_type": "markdown",
241+
"metadata": {},
242+
"source": [
243+
"In this example, our question 'How many articles has published Emma Wilson' will be translated into the Cyper query:\n",
244+
"\n",
245+
"```\n",
246+
"MATCH (r:Researcher {name: \"Emma Wilson\"})-[:PUBLISHED]->(a:Article)\n",
247+
"RETURN COUNT(a) AS numberOfArticles\n",
248+
"```\n",
249+
"\n",
250+
"which matches nodes labeled `Author` with the name 'Emma Wilson' and traverses the `PUBLISHED` relationships to `Article` nodes. \n",
251+
"It then counts the number of `Article` nodes connected to 'Emma Wilson':"
252+
]
253+
},
254+
{
255+
"cell_type": "code",
256+
"execution_count": null,
257+
"metadata": {
258+
"vscode": {
259+
"languageId": "plaintext"
260+
}
261+
},
262+
"outputs": [],
263+
"source": [
264+
"# the answer should be '5'\n",
265+
"cypher_chain.invoke(\n",
266+
" {\"query\": \"How many articles has published Emma Wilson?\"}\n",
267+
")"
268+
]
269+
},
270+
{
271+
"cell_type": "markdown",
272+
"metadata": {},
273+
"source": [
274+
"#### 3.2 Sample 2"
275+
]
276+
},
277+
{
278+
"cell_type": "markdown",
279+
"metadata": {},
280+
"source": [
281+
"In this example the query 'are there any pair of researchers who have published more than one article together?' results in the Cypher query:\n",
282+
"\n",
283+
"```\n",
284+
"MATCH (r1:Researcher)-[:PUBLISHED]->(a:Article)<-[:PUBLISHED]-(r2:Researcher)\n",
285+
"WHERE r1 <> r2\n",
286+
"WITH r1, r2, COUNT(a) AS sharedArticles\n",
287+
"WHERE sharedArticles > 1\n",
288+
"RETURN r1.name, r2.name, sharedArticles\n",
289+
"```\n",
290+
"\n",
291+
"which results in traversing from `Researcher` to `PUBLISHED` to find connected `Article` nodes, and then traversing back to find `Researchers` pairs."
292+
]
293+
},
294+
{
295+
"cell_type": "code",
296+
"execution_count": null,
297+
"metadata": {
298+
"vscode": {
299+
"languageId": "plaintext"
300+
}
301+
},
302+
"outputs": [],
303+
"source": [
304+
"# the answer should be Alice Johnson and David Miller, Alexander Lee and David Miller, Olivia Taylor and Alexander Lee, and David Miller and Alice Johnson\n",
305+
"cypher_chain.invoke(\n",
306+
" {\"query\": \"are there any pair of researchers who have published more than one article together?\"}\n",
307+
")"
308+
]
309+
},
310+
{
311+
"cell_type": "markdown",
312+
"metadata": {},
313+
"source": [
314+
"#### 3.3 Sample 3\n",
315+
"\n",
316+
"It appears David Miller has collaborated with many peers. Lets find out is he is the researcher with most peers collaborations. \n",
317+
"Our query 'which researcher has collaborated with the most peers?' results now in the Cyper:\n",
318+
"\n",
319+
"```\n",
320+
"MATCH (r:Researcher)-[:PUBLISHED]->(:Article)<-[:PUBLISHED]-(peer:Researcher)\n",
321+
"WITH r, COUNT(DISTINCT peer) AS peerCount\n",
322+
"RETURN r.name AS researcher, peerCount\n",
323+
"ORDER BY peerCount DESC\n",
324+
"LIMIT 1\n",
325+
"```\n",
326+
"\n",
327+
"Here, we need to star from all `Researcher` nodes and traverse their `PUBLISHED` relationships to find connected `Article` nodes. For each `Article` node, Neo4j then traverses back to find other `Researcher` nodes (peer) who have also published the same article."
328+
]
329+
},
330+
{
331+
"cell_type": "code",
332+
"execution_count": null,
333+
"metadata": {
334+
"vscode": {
335+
"languageId": "plaintext"
336+
}
337+
},
338+
"outputs": [],
339+
"source": [
340+
"# the answer should be 'David Miller' with 5\n",
341+
"cypher_chain.invoke(\n",
342+
" {\"query\": \"Which researcher has collaborated with the most peers?\"}\n",
343+
")"
344+
]
345+
},
346+
{
347+
"cell_type": "markdown",
348+
"metadata": {},
349+
"source": [
350+
"#### 3.3 More Samples"
351+
]
352+
},
353+
{
354+
"cell_type": "code",
355+
"execution_count": null,
356+
"metadata": {
357+
"vscode": {
358+
"languageId": "plaintext"
359+
}
360+
},
361+
"outputs": [],
362+
"source": [
363+
"# the answer should be 'David Miller and Alice Johnson'\n",
364+
"cypher_chain.invoke(\n",
365+
" {\"query\": \"Who wrote the article 'Language Model Compression for Mobile Devices'?\"}\n",
366+
")"
367+
]
368+
},
369+
{
370+
"cell_type": "code",
371+
"execution_count": null,
372+
"metadata": {
373+
"vscode": {
374+
"languageId": "plaintext"
375+
}
376+
},
377+
"outputs": [],
378+
"source": [
379+
"# the answer should be '2024'\n",
380+
"cypher_chain.invoke(\n",
381+
" {\"query\": \"In which year there were more articles published??\"}\n",
382+
")"
383+
]
384+
},
385+
{
386+
"cell_type": "code",
387+
"execution_count": null,
388+
"metadata": {
389+
"vscode": {
390+
"languageId": "plaintext"
391+
}
392+
},
393+
"outputs": [],
394+
"source": [
395+
"# the answer should be Bob Smith, David Miller, Sophia Martinez, and John Robinson\n",
396+
"cypher_chain.invoke(\n",
397+
" {\"query\": \"Which researchers have worked with Emma Wilson?\"}\n",
398+
")"
399+
]
27400
}
28401
],
29402
"metadata": {

0 commit comments

Comments
 (0)