{"repo":"manaskng/opp-lens","free":true,"listed":false,"github":"https://github.com/manaskng/opp-lens","clone":"git clone https://github.com/manaskng/opp-lens.git","description":"Distributed scraping and semantic deduplication engine using Sentence Transformers and Atlas Vector Search","language":"TypeScript","stars":37,"topics":["distributed-systems","embeddings","fastapi","machine-learning","nextjs","python","scrapping-python","vector-search","docker"],"license":null,"category":"machine-learning","readme_excerpt":"OppLens 📘 View Complete System Design Document (HLD, LLD, Database Schema, Tech Stack) OppLens is a modern, full-stack platform designed to connect developers through tech events. It goes beyond simple CRUD, offering an intelligent, personalized, and highly interactive experience for discovering, hosting, and managing developer meetups and hackathons. It leverages a robust microservices architecture for data aggregation and machine learning-powered recommendations. 🚀 Engineering Impact & Scale Async ETL & Reverse-Engineering : Engineered a fault-tolerant Python/FastAPI scraping pipeline utilizing Playwright and reverse-engineered JSON APIs to ingest 5,000+ opportunities across 5 platforms. Deduplication Engine : Utilized composite-key hashing to eliminate 95% of redundant database writes ( 1,000+ duplicate payloads/week), reducing database bloat and ensuring pristine training data for downstream ML models. NLP Recommendation Engine : Developed a hybrid recommendation engine (TF-IDF + Cosine Similarity). Offloaded semantic matching to MongoDB Atlas Vector Search (HNSW indexes) , reducing query complexity from O(N) to O(log N) for - Session Management : Uses secure, encrypted sessions to persist user state across the application. - Protected Routes : Middleware and server-side checks ensure that private pages (like /create-event or /profile ) are inaccessible without a valid session. Event System Integrity & Concurrency - Create,Update,Delete and Share Events : allows adding ","default_branch":null,"files":null,"tree":[],"storefront":"/r/manaskng","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/manaskng/opp-lens/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}