{"repo":"pinecone-io/pinecone-datasets","free":true,"listed":false,"github":"https://github.com/pinecone-io/pinecone-datasets","clone":"git clone https://github.com/pinecone-io/pinecone-datasets.git","description":"An open-source dataset library for pre-embedded dataset: create your own data catalog, or use Pinecone's public datasets.","language":"Python","stars":36,"topics":["data","database","embeddings","vector"],"license":"Apache-2.0","category":"databases-storage","readme_excerpt":"Pinecone Datasets Install For development, see MAINTAINERS.md for setup instructions using uv. Loading public datasets Pinecone hosts a public datasets catalog, you can load a dataset by name using list datasets and load dataset functions. This will use the default catalog endpoint (currently GCS) to list and load datasets. Usage - Accessing data Each dataset has three main attributes, documents , queries , and metadata which are lazily loaded the first time they are accessed. You may notice a delay as the underlying parquet files are being downloaded the first time these attributes are accessed. Pinecone Datasets is build on top of pandas. documents and queries are lazily-loaded pandas dataframes. This means that you can use all the pandas API to access the data. In addition, we provide some helper functions to access the data in a more convenient way. accessing the documents and queries dataframes is done using the documents and queries properties. These properties are lazy and will only load the data when accessed. Usage - Iterating over documents The Dataset class has helpers for iterating over your dataset. This is useful for upserting a dataset to an index, or for benchmarking. Upserting to Index To upsert data to the index, you should install the Pinecone SDK","default_branch":null,"files":null,"tree":[],"storefront":"/r/pinecone-io","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/pinecone-io/pinecone-datasets/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}