{"repo":"Ganeshsivakumar/langchain-beam","free":true,"listed":false,"github":"https://github.com/Ganeshsivakumar/langchain-beam","clone":"git clone https://github.com/Ganeshsivakumar/langchain-beam.git","description":"Integrates LLMs as PTransform in Apache Beam pipelines using LangChain","language":"Java","stars":30,"topics":["apache-beam","etl","langchain","data-engineering","dataflow","rag","langchian-beam","agents","flink"],"license":"MIT","category":"ai-agents","readme_excerpt":"Langchain-Beam Langchain-Beam integrates Large Language Models as PTransforms in Apache Beam pipelines using LangChain. This library lets you use language model capabilities directly in your Beam workflows for data processing and transformations. Motivation Apache Beam is a programming model for creating batch and streaming-based data processing pipelines. It's an abstraction that provides a way to create data processing as Transforms, each transform is like a stage in the pipeline. Langchain-beam library provides transforms to integrate LLMs into beam pipelines. The goal is to combine the apache beam's abstraction with the capabilities of Large Language Models, such as generation, completion, classification, and reasoning to process the data by leveraging LangChain, which provides a unified interface for connecting with various LLM providers, retrievals, and tools. Getting Started Include Langchain-Beam as dependency in pom.xml . Additionally, ensure you have the required Apache Beam dependencies in your project. import required modules Steps to Use Langchain-Beam 1. Create Model Options Define the modelOptions based on the model provider you’re using, configuring parameters such as temperature, max tokens, and other relevant settings. 2. Define the Instruction Prompt Create an instructionPrompt that will guide the model on how to process each PCollection input element. 3. Apply the LangchainBeam PTransform Pass the modelOptions and instructionPrompt to the LangchainModelHan","default_branch":null,"files":null,"tree":[],"storefront":"/r/Ganeshsivakumar","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Ganeshsivakumar/langchain-beam/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}