{"repo":"piskvorky/smart_open","free":true,"listed":false,"github":"https://github.com/piskvorky/smart_open","clone":"git clone https://github.com/piskvorky/smart_open.git","description":"Utils for streaming large files (S3, HDFS, gzip, bz2...)","language":"Python","stars":3456,"topics":["python","s3","hdfs","webhdfs","boto","streaming","file","streaming-data","gzip-stream","bz2"],"license":"MIT","category":"dev-tools","readme_excerpt":"smart open — utils for streaming large files in Python What? smart open is a Python 3 library for efficient streaming of very large files from/to storages such as S3, GCS, Azure Blob Storage, HDFS, WebHDFS, HTTP, HTTPS, SFTP, or local filesystem. It supports transparent, on-the-fly (de-)compression for a variety of different formats. smart open is a drop-in replacement for Python's built-in open() : it can do anything open can (100% compatible, falls back to native open wherever possible), plus lots of nifty extra stuff on top. Why? Working with large remote files, for example using Amazon's boto3 Python library, is a pain. boto3 's Object.upload fileobj() and Object.download fileobj() methods require gotcha-prone boilerplate to use successfully, such as constructing file-like object wrappers. smart open shields you from that. It builds on boto3 and other remote storage libraries, but offers a clean unified Pythonic API . The result is less code for you to write and fewer bugs to make. How? smart open is well-tested, well-documented, and has a simple Pythonic API: For more examples of URIs that smart open accepts, see help.txt or help('smart open') . Some examples: Documentation The API reference can be viewed at help.txt or using help('smart open') . Installation smart open supports a wide range of storage solutions. For all options, see the API reference. Each individual solution has its own dependencies. By default, smart open does not install any dependencies in order to ","default_branch":null,"files":null,"tree":[],"storefront":"/r/piskvorky","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/piskvorky/smart_open/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}