DataONE:GEO reuse study/Phase 1 - Revision history

Heather A Piwowar: streamline content to point to protocol page

2010-07-14T20:58:41Z

streamline content to point to protocol page

Show changes

Heather A Piwowar: DataONE/GEO reuse study/Phase 1 moved to DataONE:GEO reuse study/Phase 1: trying to be consistent with : vs /

2010-07-05T19:03:13Z

DataONE/GEO reuse study/Phase 1 moved to DataONE:GEO reuse study/Phase 1: trying to be consistent with : vs /

←Older revision

Revision as of 19:03, 5 July 2010

Heather A Piwowar: /* Early results */ update on method for annotating CREATOR REUSE

2010-06-26T12:59:22Z

Early results: update on method for annotating CREATOR REUSE

←Older revision		Revision as of 12:59, 26 June 2010
Line 113:		Line 113:
	Now adding derived columns to characterize each row:		Now adding derived columns to characterize each row:
	* is the PMC paper actually about data sharing into GEO rather than data reuse?		* is the PMC paper actually about data sharing into GEO rather than data reuse?
-	* ~~if reuse,~~ is the PMC paper by the same investigators as those who originally created the data?	+	* is the PMC paper by the same investigators as those who originally created the data?
	* if reuse, is it in the context of developing a method or tool?		* if reuse, is it in the context of developing a method or tool?
		+
		+	===Annotation===
		+
		+	Is the PMC paper by the same investigators as those who originally created the data?
		+	* first pass: automatedly extracted a column that contained the last names at the intersection of the PMC reuse paper and those in the original data-creation paper and those in the GEO submission list
		+	* if there was a lot of author overlap, coded it as a "CREATOR REUSE" paper
		+	* also automatedly extracted the institution of the PMC reuse paper and the original data-creation paper. If there was overlap and some evidence of author overlap, coded it a "CREATOR REUSE" paper
		+	* if there was no overlap in author or institution, coded it as NOT a "CREATOR REUSE" paper
		+	* for ambiguous cases were there was an author in common between the two papers but it was a common name or the corresponding author addresses were different, I manually examined the PMC reuse paper and the data-creation paper to determine whether the common authors had the same initials and institutions. If yes, I coded it as a "CREATOR REUSE" paper, otherwise I coded it as NOT a "CREATIVE REUSE" paper

Heather A Piwowar: /* Data collection */ add note about github code not being fully standalone yet

2010-06-22T12:35:56Z

Data collection: add note about github code not being fully standalone yet

←Older revision		Revision as of 12:35, 22 June 2010
Line 106:		Line 106:
	*[http://github.com/hpiwowar/pypub pypub]		*[http://github.com/hpiwowar/pypub pypub]
	*[http://gist.github.com/448371 geo data collection script]		*[http://gist.github.com/448371 geo data collection script]
		+	NOTE: I'm still getting my git together, so the code at the above links may not be fully standalone or easily run by others. I'm working on it... in the meantime, feel free to email me if you want details!

	Extracted this raw data, one row for every (GEO accession number:PMCID of paper that includes the accession number) pair:		Extracted this raw data, one row for every (GEO accession number:PMCID of paper that includes the accession number) pair:

Heather A Piwowar: add early results

2010-06-22T12:04:03Z

add early results

←Older revision		Revision as of 12:04, 22 June 2010
Line 98:		Line 98:
	* this query PMC full-text approach is similar (as per correspondence with GEO team) to that used by the GEO team to compile the 3rd party reuse page: http://www.ncbi.nlm.nih.gov/projects/geo/info/ucitations.html		* this query PMC full-text approach is similar (as per correspondence with GEO team) to that used by the GEO team to compile the 3rd party reuse page: http://www.ncbi.nlm.nih.gov/projects/geo/info/ucitations.html
	* would be nice to figure out how to write all of these columns to google docs directly from code		* would be nice to figure out how to write all of these columns to google docs directly from code
		+
		+	==Early results==
		+
		+	===Data collection===
		+	Used python source code:
		+	*[http://github.com/hpiwowar/eutils eutils]
		+	*[http://github.com/hpiwowar/pypub pypub]
		+	*[http://gist.github.com/448371 geo data collection script]
		+
		+	Extracted this raw data, one row for every (GEO accession number:PMCID of paper that includes the accession number) pair:
		+	* [http://spreadsheets.google.com/ccc?key=0Ai0SDlWE5_VYdHRDN0Q2WTV5T0RzM0dYME5OS09IRlE&hl=en raw data]
		+
		+	Now adding derived columns to characterize each row:
		+	* is the PMC paper actually about data sharing into GEO rather than data reuse?
		+	* if reuse, is the PMC paper by the same investigators as those who originally created the data?
		+	* if reuse, is it in the context of developing a method or tool?

Heather A Piwowar: rearranging sections

2010-06-20T23:22:35Z

rearranging sections

←Older revision		Revision as of 23:22, 20 June 2010
Line 66:		Line 66:
	so PMC contains 29.26% of related papers		so PMC contains 29.26% of related papers
	* so we should multiply our number of scientific papers by about 3 to get estimate for all of scientific publishing		* so we should multiply our number of scientific papers by about 3 to get estimate for all of scientific publishing
-
-	~~==Additional uses of this data collection==~~
-	* could use this data to see how many publications use any one dataset
-	* could use this data to look at average elapsed time between data submission and reuse, but only have short time period to consider... better off with data deposited longer ago
-	* can't use this particular data to see how many datasets each publication uses, because only looking at datasets from a given year
-
-	~~==Notes==~~
-	* this query PMC full-text approach is similar (as per correspondence with GEO team) to that used by the GEO team to compile the 3rd party reuse page: http://www.ncbi.nlm.nih.gov/projects/geo/info/ucitations.html
-	* would be nice to figure out how to write all of these columns to google docs directly from code
-
-
-	~~==Open Questions==~~

	==Limitations==		==Limitations==
Line 98:		Line 86:
	* Doesn't capture reuse outside the peer-reviewed literature (for example, reuse during training)		* Doesn't capture reuse outside the peer-reviewed literature (for example, reuse during training)
	* Deposits into PMC not stable over time, distribution may change over time		* Deposits into PMC not stable over time, distribution may change over time
		+
		+	==Additional uses for this data collection==
		+	* could use this data to see how many publications use any one dataset
		+	* could use this data to look at average elapsed time between data submission and reuse, but only have short time period to consider... better off with data deposited longer ago
		+	* can't use this particular data to see how many datasets each publication uses, because only looking at datasets from a given year
		+
		+	==Open Questions==
		+	None right now
		+
		+	==Notes==
		+	* this query PMC full-text approach is similar (as per correspondence with GEO team) to that used by the GEO team to compile the 3rd party reuse page: http://www.ncbi.nlm.nih.gov/projects/geo/info/ucitations.html
		+	* would be nice to figure out how to write all of these columns to google docs directly from code

Heather A Piwowar: /* Uses */ refine uses

2010-06-20T23:21:07Z

Uses: refine uses

←Older revision		Revision as of 23:21, 20 June 2010
Line 67:		Line 67:
	* so we should multiply our number of scientific papers by about 3 to get estimate for all of scientific publishing		* so we should multiply our number of scientific papers by about 3 to get estimate for all of scientific publishing

-	==~~Uses~~==	+	==Additional uses of this data collection==
-	* could use this data to look at average elapsed time between data submission and reuse, but only have short time period to consider	+
	* could use this data to see how many publications use any one dataset		* could use this data to see how many publications use any one dataset
		+	* could use this data to look at average elapsed time between data submission and reuse, but only have short time period to consider... better off with data deposited longer ago
	* can't use this particular data to see how many datasets each publication uses, because only looking at datasets from a given year		* can't use this particular data to see how many datasets each publication uses, because only looking at datasets from a given year

Heather A Piwowar: remove brainstorming section, adding it to its own page

2010-06-20T23:19:00Z

remove brainstorming section, adding it to its own page

←Older revision		Revision as of 23:19, 20 June 2010
Line 98:		Line 98:
	* Doesn't capture reuse outside the peer-reviewed literature (for example, reuse during training)		* Doesn't capture reuse outside the peer-reviewed literature (for example, reuse during training)
	* Deposits into PMC not stable over time, distribution may change over time		* Deposits into PMC not stable over time, distribution may change over time
-
-	~~==Brainstorming==~~
-
-	~~<<From email to Todd in February 2010>>~~
-
-	~~I'm thinking that analysis of GEO data reuse could enable some interesting, straight-forward analysis relevant to "value of data that is being lost."~~
-
-	One of the themes of John Wilbanks's talk at the Science Commons symposium was that science, and science data, should be generative (ala Jonathan Zittrain: “Generativity is a systems capacity to produce unanticipated change from unfiltered contributions from broad and varied audiences").
-
-	Illustrating that a significant proportion of deposits to GEO have resulted in generative science would imply that data not submitted to GEO is a real loss to the scientific community's ability to "produce unanticipated change" and thus a loss of value.
-
-	~~Specifically, here's what I have in mind:~~
-	* Search the full text of all papers in PubMed Central for mention of each GEO accession number. Exclude papers that are in the "primary citation field" for that accession number. (This is the mechanism used to generate GEO's reuse list, I confirmed recently).
-	* For each of the PMC articles that reuse GEO data, collect PubMed metadata about authors, MeSH indexing terms, date of publication, etc.
-	* For each GEO dataset with at least one reusing article, collect metadata from GEO and PubMed
-
-	~~This would facilitate many possible analyses:~~
-	*as you proposed, investigate if there is indeed a "long tail" of data reuse. Are just a few datasets reused many times, or are many datasets reused?
-	*look at the rate of reuse over time, to see if it is accelerating relative to the rate of deposit
-	* how often are the reusers from the same institution as the data submitters, and how often is it someone from a different institution or country? (I could imagine * a map here, with arrows from data creation to data use.) Is the distribution of distinct authors and institutions flatter in reuse than in creation, indicating the inclusion of a broad audience?
-	* to what extent are the topics of papers that create data the same as the topics of papers that reuse the data? For example, if we clustered the creation and reuse articles, would the creation papers cluster together, separately from the reuse articles, or would the (create, reuse) pairs usually cluster together?
-	* to what degree do the creation and reuse papers participate in the same scientific conversations? Are there often papers that cite both, or do their "citation webs" or co-authorship webs rarely overlap?
-	* investigate the topics and methods of the reuse papers, via metadata. Are they meta-analyses? bioinformatics tool-building papers? clinical validation? microarray-creating studies themselves? A diverse set of uses would be evidence of generativity.
-
-
-	~~There are limitations to this data, obviously, but I don't think they undermine the analyses since they mostly imply our results are conservative estimates:~~
-	~~We are not capturing all instances of reuse:~~
-	* only including reuse that is reflected in citations (doesn't include training, use in unpublished research)
-	* only capturing citations that refer to GEO deposits by their accession number (in this domain data attributions are often citations to the original papers instead, or by mentioning a GEO search criterion. Some of these datasets are also mirrored into other databases, so the data could be cited using a different set of accession numbers)
-	* only capturing reuse by papers in PubMed Central (a narrow slice of science papers)
-
-	~~Comments from Todd:~~
-	* I'm imagining a simple plot of (a) the number of papers depositing in GEO (or the number of deposits, if that is different), to (a) the number of papers citing GEO (or the number of citations to GEO where there may be >1 per paper).
-	* I like the idea of testing the flatness of contributors relative to reusers. That speaks to the value of data sharing outside of small social groups.

Heather A Piwowar: /* Important for argument */ fixed copy and paste error

2010-06-20T23:15:49Z

Important for argument: fixed copy and paste error

←Older revision		Revision as of 23:15, 20 June 2010
Line 84:		Line 84:
	* our estimates do not consider reuses after our study timeframe		* our estimates do not consider reuses after our study timeframe
	** many datasets we are considering will continue to be used in the future... these reuses are obviously not continued in our estimate		** many datasets we are considering will continue to be used in the future... these reuses are obviously not continued in our estimate
		+	** could estimate this impact if we examine data deposited 7 years ago?
	* Many papers not in PubMed Central		* Many papers not in PubMed Central
	** using "gene expression profiling"[mesh] query in PMC vs PubMed over 2007-2009 suggests PMC contains 30% of all related papers in PubMed		** using "gene expression profiling"[mesh] query in PMC vs PubMed over 2007-2009 suggests PMC contains 30% of all related papers in PubMed
-	** could estimate this impact if we examine data deposited 7 years ago?
	* our methods do not find studies that both create and reuse data		* our methods do not find studies that both create and reuse data
	** to narrow down our query results, we automatedly eliminate studies that create data... even though these same studies may also reuse data		** to narrow down our query results, we automatedly eliminate studies that create data... even though these same studies may also reuse data

Heather A Piwowar: /* Important for argument */ add estimate for PMC in PubMed

2010-06-20T23:15:11Z

Important for argument: add estimate for PMC in PubMed

←Older revision		Revision as of 23:15, 20 June 2010
Line 84:		Line 84:
	* our estimates do not consider reuses after our study timeframe		* our estimates do not consider reuses after our study timeframe
	** many datasets we are considering will continue to be used in the future... these reuses are obviously not continued in our estimate		** many datasets we are considering will continue to be used in the future... these reuses are obviously not continued in our estimate
		+	* Many papers not in PubMed Central
		+	** using "gene expression profiling"[mesh] query in PMC vs PubMed over 2007-2009 suggests PMC contains 30% of all related papers in PubMed
		+	** could estimate this impact if we examine data deposited 7 years ago?
	* our methods do not find studies that both create and reuse data		* our methods do not find studies that both create and reuse data
	** to narrow down our query results, we automatedly eliminate studies that create data... even though these same studies may also reuse data		** to narrow down our query results, we automatedly eliminate studies that create data... even though these same studies may also reuse data
-	* Many papers not in PubMed Central	+	** we don't have an estimate of how many this is, would require manual inventory
-	** ~~use "gene expression profiling"[mesh] query in PMC vs PubMed over time period in question to get relevant~~ estimate?	+	* Many data citations not attributed using accession numbers
-	* Many data citations not attributed using accession numbers ~~(source for percentages?)~~	+
	** don't have a good way to estimate this yet		** don't have a good way to estimate this yet
	** would require a manual inventory, similar to Sarah's data citation inventory in DataONE summer 2010 project		** would require a manual inventory, similar to Sarah's data citation inventory in DataONE summer 2010 project