User:Timothee Flutre/Notebook/Postdoc/2011/11/04 - Revision history

Timothee Flutre: /* Entry title */ add title

2012-11-21T15:10:07Z

Entry title: add title

←Older revision		Revision as of 15:10, 21 November 2012
Line 6:		Line 6:
	\| colspan="2"\|		\| colspan="2"\|
	<!-- ##### DO NOT edit above this line unless you know what you are doing. ##### -->		<!-- ##### DO NOT edit above this line unless you know what you are doing. ##### -->
-	==~~Entry title~~==	+	==K-means has the bad tendency to build clusters of similar size==

	* '''R''' is very efficient to sketch an analysis, but it '''usually cannot handle very large datasets''' (matrix with <math>>10^6</math> rows), thus it often happens that I need to find other tools.		* '''R''' is very efficient to sketch an analysis, but it '''usually cannot handle very large datasets''' (matrix with <math>>10^6</math> rows), thus it often happens that I need to find other tools.

Timothee Flutre: /* Entry title */ in R, kmeans, scatterplot3d and heatmap + try cluto

2011-11-05T01:53:04Z

Entry title: in R, kmeans, scatterplot3d and heatmap + try cluto

←Older revision		Revision as of 01:53, 5 November 2011
Line 7:		Line 7:
	<!-- ##### DO NOT edit above this line unless you know what you are doing. ##### -->		<!-- ##### DO NOT edit above this line unless you know what you are doing. ##### -->
	==Entry title==		==Entry title==
-	* Insert content here...

		+	* '''R''' is very efficient to sketch an analysis, but it '''usually cannot handle very large datasets''' (matrix with <math>>10^6</math> rows), thus it often happens that I need to find other tools.
		+
		+	* As a first try to replace [http://stat.ethz.ch/R-manual/R-devel/library/stats/html/kmeans.html kmeans] in R, I launched the [http://glaros.dtc.umn.edu/gkhome/cluto/cluto/overview CLUTO] clustering program on a simulated dataset (file ''matrix.txt''):
		+
		+	for k in {1..8}; do vcluster matrix.txt $k -clabelfile=colnames.txt -plotclusters=plot_k${k}.ps -clustercolumns > stdout_k${k}; done
		+
		+	It works well and it still finishes on a large, "real" dataset. However, should I trust the results? Indeed, it is well-known that kmeans suffers from its tendency to build clusters of similar size. And, as shown by the figure below, it can provides bad results...
		+
		+	* To show this, let's '''simulate a dataset''', realistic enough for what I am analyzing. There are 1000 items, each of 3 dimensions (x, y and z). The data belong to 4 clusters, the first of size 700 around 000, the second of size 200 around 111, the third of size 50 around 011, and the fourth of size 50 around 100. Here is the R code:
		+
		+	low.mean <- 0
		+	high.mean <- 2
		+	mysd <- 0.1
		+	mult <- 1000
		+	mydata.all <- rbind(matrix(rnorm(0.7mult3, mean=low.mean, sd=mysd), ncol=3, byrow=TRUE),
		+	matrix(rnorm(0.2mult3, mean=high.mean, sd=mysd), ncol=3, byrow=TRUE),
		+	matrix(c(rnorm(0.05mult, mean=low.mean, sd=mysd), rnorm(0.1mult, mean=high.mean, sd=mysd)), ncol=3, byrow=FALSE),
		+	matrix(c(rnorm(0.05mult, mean=high.mean, sd=mysd), rnorm(0.1mult, mean=low.mean, sd=mysd)), ncol=3, byrow=FALSE))
		+	mydata.all <- cbind(mydata.all, c(rep("000", 0.7mult), rep("111", 0.2mult), rep("011", 0.05mult), rep("100", 0.05mult)))
		+	colnames(mydata.all) <- c("F", "L", "T", "truth")
		+	head(mydata.all)
		+
		+	Now, let's '''use kmeans and plot the results''':
		+
		+	mydata <- matrix(as.numeric(mydata.all[sample(nrow(mydata.all)), 1:3]), ncol=3, byrow=FALSE)
		+	colnames(mydata) <- c("F","L","T")
		+	head(mydata)
		+	res.km <- kmeans(mydata, 4)
		+	aggregate(mydata, by=list(res.km$cluster), FUN=mean)
		+	table(res.km$cluster)
		+	library(scatterplot3d)
		+	scatterplot3d(mydata[,"F"], mydata[,"L"], mydata[,"T"], color=res.km$cluster, main="kmeans")
		+
		+	[[Image:Kmeans unequal-clusters bad-results.png\|400px]]
		+
		+	It's pretty wrong, isn't it?
		+
		+	And as a bonus, here is how to plot the corresponding [http://stat.ethz.ch/R-manual/R-patched/library/stats/html/heatmap.html heatmap] (as I spent some time to find the proper way to do it):
		+
		+	mydata.sort <- cbind(mydata, res.km$cluster)[order(res.km$cluster),]
		+	heatmap(mydata.sort[,1:3], Rowv=NA, Colv=NA, labRow=NA, scale="none", col=heat.colors(10))

	<!-- ##### DO NOT edit below this line unless you know what you are doing. ##### -->		<!-- ##### DO NOT edit below this line unless you know what you are doing. ##### -->

Timothee Flutre: Autocreate 2011/11/04 Entry for User:Timothee_Flutre/Notebook/Postdoc

2011-11-05T01:17:25Z

Autocreate 2011/11/04 Entry for User:Timothee_Flutre/Notebook/Postdoc

New page

{|{{table}} width="800"
|-
|style="background-color: #EEE"|[[Image:owwnotebook_icon.png|128px]]<span style="font-size:22px;"> Project name</span>
|style="background-color: #F2F2F2" align="center"|<html><img src="/images/9/94/Report.png" border="0" /></html> [[{{#sub:{{FULLPAGENAME}}|0|-11}}|Main project page]]<br />{{#if:{{#lnpreventry:{{FULLPAGENAME}}}}|<html><img src="/images/c/c3/Resultset_previous.png" border="0" /></html>[[{{#lnpreventry:{{FULLPAGENAME}}}}{{!}}Previous entry]]<html>      </html>}}{{#if:{{#lnnextentry:{{FULLPAGENAME}}}}|[[{{#lnnextentry:{{FULLPAGENAME}}}}{{!}}Next entry]]<html><img src="/images/5/5c/Resultset_next.png" border="0" /></html>}}
|-
| colspan="2"|

==Entry title==
* Insert content here...


|}

__NOTOC__