{"repo":"SkywalkerDarren/chatWeb","free":true,"listed":false,"github":"https://github.com/SkywalkerDarren/chatWeb","clone":"git clone https://github.com/SkywalkerDarren/chatWeb.git","description":"ChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key points.","language":"Python","stars":916,"topics":["chatgpt","embedding","gpt-35-turbo","news-extractor","newspaper","openai","pgvector","postgresql","vector-database","faiss"],"license":"MIT","category":"databases-storage","readme_excerpt":"ChatWeb English Doc 中文文档 ChatWeb can crawl any webpage or extract text from PDF, DOCX, TXT files, and generate an embedded summary. It can also answer your questions based on the content of the text. It is implemented using the chatAPI and embeddingAPI based on gpt3.5, as well as a vector database. Basic Principle The basic principle is similar to existing projects such as chatPDF and automated customer service AI. Crawl web pages Extract text content Use GPT3.5's embedding API to generate vectors for each paragraph Calculate the similarity score between each paragraph's vector and the entire text's vector to generate a summary Store the vector-text mapping in a vector database Generate keywords from user input Generate a vector from the keywords Use the vector database to perform a nearest neighbor search and return a list of the most similar texts Use GPT3.5's chat API to design a prompt that answers the user's question based on the most similar texts in the list. The idea is to extract relevant content from a large amount of text and then answer questions based on that content, which can achieve a similar effect to breaking through token limits. An improvement was made to generate vectors based on keywords rather than the user's question, which increases the accuracy of searching for relevant texts. Getting Started Manual installation: - Install Python3 - Download this repository by running git clone https://github.com/SkywalkerDarren/chatWeb.git - Navigate to the director","default_branch":null,"files":null,"tree":[],"storefront":"/r/SkywalkerDarren","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/SkywalkerDarren/chatWeb/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}