|
|
| (15 intermediate revisions by the same user not shown) |
| Line 1: |
Line 1: |
| ==Research Plan== | | ==Aim== |
| * Query GEO for all GDS and GDS accession numbers for datasets submitted in 2007
| |
| * Query PubMed Central for these accession numbers in the full text of PMC papers published between 1900 and 2009
| |
| * Enumerate the PMC papers that reused GEO data
| |
| * Estimate what percent of these papers depended on the GEO data for their scientific contribution
| |
|
| |
|
| ==Details== | | ==Background== |
| * could use this data to look at average elapsed time between data submission and reuse, but only have short time period to consider
| |
| * could use this data to see how many publications use any one dataset
| |
| * can't use this particular data to see how many datasets each publication uses, because only looking at datasets from a given year
| |
| * accession number formats:
| |
| ** look at both GSE and GDS accession numbers
| |
| ** use both the raw ID number like 200007572 and the stripped version without the 200... prefix. For example, search for both 200007572 and 7572
| |
| ** search for both accession number right beside the prefix, and with one space in between, so "GSE 7572" and "GSE7572"
| |
| * spot-check to make sure accession number is in the context of reuse... looks like there may be a few mentions in the context of depost in which the article is not tagged with pmc_gds[filter] (example: PMCID 2396644)
| |
| ** do this for all the PMC article hits? looks like there are a few missing the filter, and it matters because it would erroneously inflate our reuse estimate
| |
|
| |
|
| ==Notes== | | ==Methods== |
| * this query PMC full-text approach is similar to that used by the GEO team to compile the 3rd party reuse page, as per email from GEO <<Heather: add quote>> | | ===Overview=== |
| | * Using the method outlined at [[DataONE:Protocols/Find_GEO_reuses]]: |
| | ** Query GEO for all GDS and GDS accession numbers for datasets submitted in 2007 |
| | ** Query PubMed Central for these accession numbers in the full text of PMC papers published between 1900 and 2009 |
| | ** Enumerate the PMC papers that reused GEO data |
| | ** Estimate what percent of these papers depended on the GEO data for their scientific contribution |
|
| |
|
| ==Open Questions== | | ===Details=== |
| * Ideally want to remove reuses by the same authors or in the same lab. How to do this efficiently? | | * see [[DataONE:Protocols/Find_GEO_reuses]] |
|
| |
|
| ==Limitations== | | ==Results== |
| ===Important for argument===
| |
| This is a conservative estimate because:
| |
| * our estimates do not consider reuses after our study timeframe
| |
| ** many datasets we are considering will continue to be used in the future... these reuses are obviously not continued in our estimate
| |
| * Many papers not in PMC
| |
| ** use "gene expression profiling"[mesh] query in PMC vs PubMed over time period in question to get relevant estimate?
| |
| * Many data citations not attributed using accession numbers (source for percentages?)
| |
| ** don't have a good way to estimate this yet
| |
| ** would require a manual inventory, similar to Sarah's data citation inventory in DataONE summer 2010 project
| |
| ** maybe out-of-scope to get this estimate for this project, just admit it is an underestimate
| |
|
| |
|
| ===Less important for argument=== | | ==Discussion== |
| * Doesn't capture reuse outside the peer-reviewed literature (for example, reuse during training)
| |
| * Deposits into PMC not stable over time, distribution may change over time
| |
| | |
| ==Brainstorming==
| |
| | |
| In the meantime, here's a quick overview of where I'm headed, with details to follow as I see how it pans out. Feedback welcome, now or later.
| |
| | |
| I'm thinking that analysis of GEO data reuse could enable some interesting, straight-forward analysis relevant to "value of data that is being lost."
| |
| | |
| One of the themes of John Wilbanks's talk at the Science Commons symposium was that science, and science data, should be generative (ala Jonathan Zittrain: “Generativity is a systems capacity to produce unanticipated change from unfiltered contributions from broad and varied audiences").
| |
| | |
| Illustrating that a significant proportion of deposits to GEO have resulted in generative science would imply that data *not* submitted to GEO is a real loss to the scientific community's ability to "produce unanticipated change" and thus a loss of value.
| |
| | |
| Specifically, here's what I have in mind:
| |
| * Search the full text of all papers in PubMed Central for mention of each GEO accession number. Exclude papers that are in the "primary citation field" for that accession number. (This is the mechanism used to generate GEO's reuse list, I confirmed recently).
| |
| * For each of the PMC articles that reuse GEO data, collect PubMed metadata about authors, MeSH indexing terms, date of publication, etc.
| |
| * For each GEO dataset with at least one reusing article, collect metadata from GEO and PubMed
| |
| | |
| This would facilitate many possible analyses:
| |
| *as you proposed, investigate if there is indeed a "long tail" of data reuse. Are just a few datasets reused many times, or are many datasets reused?
| |
| *look at the rate of reuse over time, to see if it is accelerating relative to the rate of deposit
| |
| * how often are the reusers from the same institution as the data submitters, and how often is it someone from a different institution or country? (I could imagine * a map here, with arrows from data creation to data use.) Is the distribution of distinct authors and institutions flatter in reuse than in creation, indicating the inclusion of a broad audience?
| |
| * to what extent are the topics of papers that create data the same as the topics of papers that reuse the data? For example, if we clustered the creation and reuse articles, would the creation papers cluster together, separately from the reuse articles, or would the (create, reuse) pairs usually cluster together?
| |
| * to what degree do the creation and reuse papers participate in the same scientific conversations? Are there often papers that cite both, or do their "citation webs" or co-authorship webs rarely overlap?
| |
| * investigate the topics and methods of the reuse papers, via metadata. Are they meta-analyses? bioinformatics tool-building papers? clinical validation? microarray-creating studies themselves? A diverse set of uses would be evidence of generativity.
| |
| | |
| | |
| There are limitations to this data, obviously, but I don't think they undermine the analyses since they mostly imply our results are conservative estimates:
| |
| We are not capturing all instances of reuse:
| |
| * only including reuse that is reflected in citations (doesn't include training, use in unpublished research)
| |
| * only capturing citations that refer to GEO deposits by their accession number (in this domain data attributions are often citations to the original papers instead, or by mentioning a GEO search criterion. Some of these datasets are also mirrored into other databases, so the data could be cited using a different set of accession numbers)
| |
| * only capturing reuse by papers in PubMed Central (a narrow slice of science papers)
| |
| | |
| Comments from Todd:
| |
| * I'm imagining a simple plot of (a) the number of papers depositing in GEO (or the number of deposits, if that is different), to (a) the number of papers citing GEO (or the number of citations to GEO where there may be >1 per paper).
| |
| * I like the idea of testing the flatness of contributors relative to reusers. That speaks to the value of data sharing outside of small social groups.
| |
