{"repo":"Cloud-CV/EvalAI","free":true,"listed":false,"github":"https://github.com/Cloud-CV/EvalAI","clone":"git clone https://github.com/Cloud-CV/EvalAI.git","description":":cloud: :rocket: :bar_chart: :chart_with_upwards_trend: Evaluating state of the art in AI","language":"Python","stars":2041,"topics":["ai","machine-learning","django","angularjs","python","ai-challenges","docker","reproducible-research","reproducibility","evaluation"],"license":null,"category":"machine-learning","readme_excerpt":"------------------------------------------------------------------------------------------ EvalAI is an open source platform for evaluating and comparing machine learning (ML) and artificial intelligence (AI) algorithms at scale. In recent years, it has become increasingly difficult to compare an algorithm solving a given task with other existing approaches. These comparisons suffer from minor differences in algorithm implementation, use of non-standard dataset splits and different evaluation metrics. By providing a central leaderboard and submission interface, we make it easier for researchers to reproduce the results mentioned in the paper and perform reliable & accurate quantitative analysis. By providing swift and robust backends based on map-reduce frameworks that speed up evaluation on the fly, EvalAI aims to make it easier for researchers to reproduce results from technical papers and perform reliable and accurate analyses. Features - Custom evaluation protocols and phases : We allow creation of an arbitrary number of evaluation phases and dataset splits, compatibility using any programming language, and organizing results in both public and private leaderboards. - Remote evaluation : Certain large-scale challenges need special compute capabilities for evaluation. If the challenge needs extra computational power, challenge organizers can easily add their own cluster of worker nodes to process participant submissions while we take care of hosting the challenge, handling","default_branch":null,"files":null,"tree":[],"storefront":"/r/Cloud-CV","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Cloud-CV/EvalAI/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}