{"repo":"jacobwarren/social-media-ai-engineering-etl","free":true,"listed":false,"github":"https://github.com/jacobwarren/social-media-ai-engineering-etl","clone":"git clone https://github.com/jacobwarren/social-media-ai-engineering-etl.git","description":"Real-world AI engineering dataset creation, SFT fine-tuning, and GRPO alignment ETL pipeline.","language":"Python","stars":34,"topics":["ai-engineering","data-pipeline","feature-engineering","llm"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Social Media AI Engineering ETL A manifest‑driven pipeline that turns social posts into training‑ready datasets for LLM fine‑tuning. Designed for fast setup, clear outputs, and repeatable runs. This is the exact pipeline I used to build the LinkedIn model for the SaaS I shutdown, GrowGlad, and the smaller demo model I released, LinkedQwen. What you get - One command from raw JSONL to training splits (SFT and DPO) - Clear outputs under data/processed/{RUN ID}/ plus a manifest.json for easy re‑runs - Prompt generation (18), dataset assembly (22), and final splits (23) - Works on A100/H100; tested with Qwen2.5‑7B, Mistral-Nemo-2407, Qwen3-32B, and Llama‑3.1‑8B - Simple follow‑on training scripts for SFT (25) and GRPO (26) - Reward functions for tone, hashtags, emoji, length, structure, and more Quickstart Inputs: use the tiny demo dataset ./example-dataset.jsonl . Hardware requirements This pipeline targets GPU-backed LLM fine-tuning and guided decoding. A data‑center NVIDIA GPU is required for vLLM stages and is strongly recommended end‑to‑end. - Supported GPUs: NVIDIA A100 (40GB/80GB) minimum; H100/H200 preferred for throughput and long contexts - CUDA: 12.1–12.3 recommended (ensure PyTorch/vLLM builds match). NVIDIA Driver ≥ 535 - Host: Python 3.9+, 32GB RAM+, fast SSD (10–50GB free for artifacts), Linux preferred - Notes: - Ensure NCCL/CUDA toolkit versions match the installed PyTorch and vLLM wheels - Large models may require tensor/kv cache sharding or reduced batch sizes ","default_branch":null,"files":null,"tree":[],"storefront":"/r/jacobwarren","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/jacobwarren/social-media-ai-engineering-etl/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}