From 588c96ab6d38f30bbef3aa773163b36838538355 Mon Sep 17 00:00:00 2001 From: Jules Laplace Date: Fri, 7 Dec 2018 18:46:03 +0100 Subject: path --- scraper/README.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) (limited to 'scraper/README.md') diff --git a/scraper/README.md b/scraper/README.md index 782fa30a..318bba9a 100644 --- a/scraper/README.md +++ b/scraper/README.md @@ -36,18 +36,24 @@ The Open Research Corpus (ORC) is produced by the Allen Institute / Semantic Sch http://labs.semanticscholar.org/corpus/ +We do a two-stage fetch process as only about 66% of their papers are in this dataset. + ### s2-search.py Loads titles from citations file and queries the S2 search API to get paper IDs, then uses the paper IDs from the search entries to query the S2 papers API to get first-degree citations, authors, etc. ### s2-dump-ids.py -Extract all the paper IDs and citation IDs from the queried papers. +Dump all the paper IDs and citation IDs from the queried papers. ### s2-extract-papers.py Extracts papers from the ORC dataset which have been queried from the API. +### s2-dump-missing-paper-ids.py + +Dump the citation IDs that were not found in the ORC dataset. + ### s2-raw-papers.py Some papers are not in the ORC dataset and must be scraped from S2 directly. -- cgit v1.2.3-70-g09d2