An Evolving DAG for the LLM world - Julia Schottenstein of LangChain at Small Data SF 2024

0:00[Music]

0:15For those of you who don't know what lingchain is, we're really popular open source framework for Python and developer Python and Typescript developers that help you uh build agentic systems. And so for for this crowd, maybe the simplest way to think about it is we help you augment the power of LLMs with your data. And so everyone in this room knows that LLMs

0:36are extremely powerful technology, but if you want to use them in the context of your application or your business, they don't know how to do useful things yet. And so oftent times that means you have to figure out how to bring it private documents or data so that it can reason over them. Or maybe you need to

0:55augment that with tool usage, which is just an industry term for an API and a definition for what you should do with that API. Uh or maybe you need up-to-date information because LLMs are trained on an enormous uh corpus of knowledge, but after the cutoff date, they don't know anything about more recent or current events. And so with

1:15that, you're constantly bringing additional information to the power of LLM so that they can do things in your application. Before I was at Lingchain, I was at DBT. So, probably a more familiar technology for I see a DBT shirt in the audience, which is which is fun to see. Um, more familiar for this audience in data. And what we did there

1:35was we did data transformation. So, this is a classic DBT DAG where you took raw data in your data warehouse. You joined uh the tables together and you made useful tables so that you could see what was happening in your in your business.

1:49At LinkChain, we also have DAGs. uh we call them chains and so chains are these discrete steps that you want to order in a particular sequence sequence uh so that you can bring that additional context to the LLM and this is an example of probably the most common chain that that people build which is a rag chain or retrieval augmented

2:08generation chain where a question comes in you pull context from your vector database or your vector store that has the answer hopefully to the question that the user asked you stuff uh the question the answer along with some prompt instructions into the LLM as a request and then the LLM spits out a response to answer your question. And so

2:30this is very much like a data pipeline that many people here are familiar with, but instead of prepping data to get uh useful information, you're instead prepping natural language or text to get to a useful response. But if you've been paying attention in the LLM world, what people are really excited about today are agents. And so Mark Beni off on

2:51stage at Dreamforce just uh earlier last week talked about how agents are the future and agents is a very overloaded marketing term. Uh people have different definitions for what exactly is an agent. It takes action. It helps you interact with the world. It automates things. But for us agents is a really specific technical term. And an agent

3:11just means that an LLM decides the control flow of your application instead of predefining it upfront in code like you might have with a DAG. And so in in this diagram here, you can see the LLM decides the routes and gets to pick which steps it is going to take in order to solve the problem at hand.

3:33Uh, and when you change with an LLM getting to route, you move from a directive asyclic graph to one where you can have cycles. And this kind of changes everything because we're we're kind of used to having the asyclic part of the graph as being fundamental to how we solve problems. And to put it into more context, LLMs can perform better on

3:59the second pass if they know how they failed on the first pass. And so with deterministic code, that's not the case.

4:06If you give it a task the first time, the second time, the third time, you're going to get the same response. With an LLM, it can actually improve the outcome or the results if it tries again and knows what went wrong the first time. So this is a pretty cool flow diagram uh that was in the alpha codium paper from

4:22Kodium AI which is a codegen startup and they show the diagram for how they've designed their codegen agent and you can see it has some pre-processing steps where uh it reflects on the problem at hand it reasons about possible tests that it wants to create. It generates possible solutions and then it goes into this code iteration phase where you can

4:45see there are cycles in this graph where it will iteratively try to come up with solutions on uh code solutions, public test creation solutions and then it will finally generate the response for the end user. And so it's uh iteratively trying to get a better response before it returns the the final pass. And so this is what an agent uh means to us

5:08where the LLM routes. But when you move from a DAG type workflow into a direct

5:16cyclic workflow where you can have these cycles, you have a lot of challenges that come with that. Uh and when you're building agents, there three considerations that uh you you should think about to have agents that can can perform successfully in in the wild. Uh and so planning is a hard problem for agents. Memory is a hard problem for

5:36agents and then of course reliability. And so I'll touch on all three of these. So it's it's been shown in in research that if you have like an explicit planning step or an explicit reflection step, your agent can perform much better than if it just goes and tries to execute the task. And that makes sense. If you think about if you

5:56were going to try to find the best route up a rock climbing wall, you likely will stand on the on the ground first, figure out what potential paths you might take, and then go execute on that route to get to the the top of the rock climbing wall. That's likely a better strategy than just stepping onto the wall and and

6:16iteratively trying to find your way up. Uh and it's the same thing with agents.

6:20Uh if you first take a moment to plan which route you're going to take and reflect constantly on if that path is working, agents can can accomplish tasks much more successfully. Memory is also a big challenge when it comes to agents. We showed an example of a single

6:42agent, but oftent times you have these network of agents and this is a pretty common architecture that you'll see. We call these cognitive architectures in and and in in our world where you might have a supervisor agent that just takes tasks in and routes the requests to more specialized sub aents that are really good at performing a specific task. And

7:05when you have multiple agents collaborating on a single task, you constantly have to update the state of what's transpired so that the sub agents have enough information to execute their work uh sufficiently. And so this concept of a persistence layer or shared memory or to the example I shared before is how do I know if a prior pass failed,

7:28can I use that context or information to improve on my second pass? Memory or shared state is important when you're dealing with agents. And then this one's probably not very surprising for folks, but agents struggle with reliability. And it's not just the LLM non-determinism that can be challenging, but there's often task ambiguity. Some of that's user input.

7:52It's natural language. We're asking questions of a of software, and we're not very good often at describing exactly what we need accomplished. Uh, and task ambiguity can be a challenge for agents. And also tool misuse. And so this is uh each LLM has access to a bunch of tools which again are just APIs with descriptions on how to use these

8:13tools. And if you allow cycles, a lot of times agents will get stuck calling the same tool and they need to switch tools.

8:19And so knowing which tool to use at the right time is is also a hard problem for

8:26agents and all all these challenges uh they come with trade-offs. So we talked about chains which are just DAGs where you predefined your code up front of which sequence of steps need to happen in which order and those are more reliable but of course less flexible.

8:43And then an agent has the benefit of they're far more flexible. You use the LLM to route and decide how to solve the problem, but they can be less reliable for a lot of the the reasons that I just described. And so what we're trying to do at LinkChain is solve that problem.

9:00Can you get both? Can you get flexible and reliable? And we're working on a framework called Langraph where we try to help balance agent uh agent control with agency. And some of the core pillars of this orchestration framework that's different than the orchestration frameworks that you all might be familiar with when you're working with data pipelines is how do you design for

9:25controllability? So define both explicit and implicit workflows. this persistence layer. So the shared memory and state both for an individual agent maybe between sessions or multiple agents that are trying to collaborate on a on a singular task. Human in the loop. This is really important for controllability. If you leave an agent to solve very complex tasks on its own, it's likely not going

9:50to get from start to finish uh without some help. So it needs some steering. Uh and then streaming. Streaming is important with LLMs because latency is a challenge. So it helps with the user experience. So token by token streaming, but also intermediate steps. And so one of the new UX patterns for when you're interacting with agents is it will tell

10:12you which steps it's taking to solve your problem. So that when it takes 30 seconds to get a a good result, you know that someone's doing the work or the agent is doing the work to accomplish the task. And with that, I'm going to switch over to a quick demo.

10:37All right. So, what you're looking at is langraph studio and this is an example of a cyclic graph where this agent has been designed to solve questions about uh lang chain. And if we just look at the architecture really quickly, there's a router node that can route problems to either general requests. Uh if it needs more information, it will route to this

11:01more information node. But if the question is about link chain, it will go through this flow in the center uh where it will generate a set of questions that it will research. It will then go research those questions, get answers for those questions and then generate a response. And so this is uh in put into practice the planning and reflection

11:21steps before you execute and generate a response. So let's see it live. So if I ask the question what is lang

11:34graph we can see it went through to this general step and so this agent failed at understanding that lang graph is related to lang chain. And so remember how I said you can have human steering to help solve the problem. Uh I can edit this step, reroute it instead to the lingch chain workflow and then it will rerun through

11:59the center part of the of the graph where it will execute the generate questions research and then uh create a response. And once it finishes, I'll just walk you through exactly what it it did here and how it used cycles. So at

12:17the beginning, we had a router agent that decided which path it should take. Uh I forced it down the link chain path.

12:25It generated a bunch of research questions here. It had three questions it wanted to look after. So review the conceptual documentation. Look for specific features in the integrations documentation. and then check how-to guides for practical examples of langraph. And then in this research node, it's going to iterate three times on this particular node to to answer these questions or retrieve documents

12:49that are relevant to that path or route that it's planned out. And so you can see it here generating some documents from our public documentation uh that's specific to the review the conceptual documentation. Next, it's going to go look at the next two steps that we've planned it to look at. And so, it hits the research node uh three times uh in

13:11response to the plan that it's outlined at the start of the path. And so, we can see that and then it takes the uh all of that context from the research that it's done to generate the final response. And you can see it here. Langraph is a framework for building complex langu language-based AI applications using a graph-based approach. uh and it it will

13:33site its sources as as well to show you where it got that context. Uh and so this is an example of where you have cycles and this is still a pretty new field. So, a lot of times I'll get questions, what are the most exciting agents that are alive and wild that I can interact with today? Because I think

13:55a year ago most of us were just talking about um chat bots and I think uh a year later there are really exciting applications that you can interact with that are very creative. And so a few of my favorites, uh, Roblox has a very cool assistance uh, studio AI uh, agent where

14:14I can create a prompt that says, "Create me a world where I'm in the forest and it's dark at night and my player is jumping and then it will go create a script with those exact assets." It's pretty fantastic. I can then execute that script and that entire virtual world has been created off of a prompt.

14:33Trip Adviser has another uh live agent that you can go interact with today. You can say, "I'm traveling to Dallas with five of my friends. I'm an art lover.

14:43These are the dates that I'm going." And it will generate for you an itinerary based on your preferences. And so we call this kind of concierge search or more unique way to discover new uh experiences. And then finally, Replet launched their uh coding agent last week, which has also gotten a ton of enthusiasm from the community, which

15:05helps you generate code, generate tests, uh and also uh create automatic pull requests. And so with that, uh I will invite everyone to say yes, DAGs are pretty cool. We love them in data. Uh but directic cyclic graphs can also be very cool when you're dealing with agentic style type problems. Thanks so much.

15:30[Applause] [Music]