User:Nuri Purswani/Network/Results
| Home | Introduction | Synthetic Datasets | Methods | Results | Discussion | References |
Description
In order to assess algorithmic performance, we chose three in silico candidate networks, and performed a comparison based on:
- Varying the amount of experimental noise (Gaussian)
- Varying the number of repetitions of the experiment and assessing the algorithm's "tolerance" to recover the network for different noise variances
- Comparing the types of input disturbances they take in and their relative ability to recover network structure.
- Step input
- Gaussian Noise
- Assessed their ability to recover detailed 'substructures' in a complex network structure
Calculation of Signal to Noise Ratio
For every candidate structure, the signal to noise ratio was calculated as follows:
where:
- [math]\displaystyle{ m }[/math] is the mean amplitude of the signals at steady state.
- [math]\displaystyle{ s }[/math] ais the standard deviation of the gaussian noise.
The sensitivity-specificity plots contain the log(SNR) quantity.
Sensitivity and Specificity
- Sensitivity: Proportion of recovered true connections.
- Specificity: Proportion of correctly identified non-connections.
- TP:True Positive
- FN:False Negative
- TN:True Negative
- FP:False Positive
The algorithmic performance is assessed as a function of sensitivity and specificity for both algorithms. Note that we are not using common ROC curves, as the robust control network reconstruction method does not make use of confidence intervals or thresholds of discrimination. Instead, we show the scores of AICc (Akaike information criterion) output by the algorithm and compare them to the results of the z=3 and z=5 confidence interval results output by the Bayesian inference method.
Comparison of Algorithmic Performance as a function of SNR
Chain
CASE1:Measurements 1, 2, 3
PDF Format of Results (Recommended to look at this before the website)
The Figures show comparisons of sensitivity and specificity for both algorithms as a function of signal to noise ratio. Here, we are using N=3 repeats of the experiments. (See the section for a more detailed explanation).
- For the robust control method we plot the sensitivity and specificity of the best three AICc scores
- For the Bayesian Inference method we plot the sensitivity and specificity at z=3 and z=5 confidence intervals.
Figure1: Comparison of Sensitivity and Specificity of robust control and bayesian inference method. x axis: log (SNR) y axis: sensitivity and specificty. See Table of Values and also table of scores for robust control method
Observation: The third best AICc score in the robust control method was able to recover the correct boolean structure for SNRs>1884. Then, its sensitivity drops,a s it recovers the fully decoupled network. The Bayesian method was unable to perfectly recover the correct network structure, but obtained something close to what we expect to see at z=5, for the highest SNR=57000. This was with a sensitivity = 0.75 as it failed to identify the connection from 1 to 2.
CASE 2: Measurements 1, 2, 4
PDF Format of Results
The Figures show comparisons of sensitivity and specificity for both algorithms as a function of signal to noise ratio. Here, we are using N=3 repeats of the experiments. (See the section for a more detailed explanation).
- For the robust control method we plot the sensitivity and specificity of the 1st and 5th best AICc scores (None of them recovered the true structure perfectly).
- For the Bayesian Inference method we plot the sensitivity and specificity at z=3 and z=5 confidence intervals.
Figure2: Comparison of Sensitivity and Specificity of robust control and bayesian inference method. x axis: log (SNR) y axis: sensitivity and specificty. See Table of Values and also table of scores for robust control method
Observation: None of the methods were able to recover the correct structure. The sensitivity/specificity analysis shows that the robust control method was able to recover the nearly-correct structure with 1 false positive on the 5th best AICc score (Sens=1 Spec=0.75): This was the best score. The Bayesian Inference method over estimated the number of connections for the highest SNR value = 100 000, and then predicted the sparsely connected network, hence the drop in sensitivity at z=3. At z=5 it did not pick any interactions so the sensitivity was = 0 (See PDF file for details).
This time we none of the algorithms guessed the true structure, so we chose to display the first and 5th AICc scores for the robust control method. For the Bayesian method we chose z=3 and z=5 again. For detailed values of AICc scores in the robust control method, click here.
NOTE: DUE TO A LIMITED AMOUNT OF TIME TO RUN THESE SIMULATIONS, I WOULD STRONGLY RECOMMEND TO REPEAT THESE. In case 2 I accidentally deleted some workspace variables, and was not able to show the datasets used to generate the simulations. I do have the boolean structure results (shown in the sensitivity-specificity results and subsequent tables) but it is strongly advisable to re-do them. Click on the title link for extra graphs, pictures and information. Furthermore, I checked this particular system, and it seems that when the DC offset is removed, there are zeros in the transfer function, which may explain why the robust control method did not recover it perfectly.
Double Ring
The same logic was applied, in an in silico network with a more complex topology, as shown below:
Four points were sampled in this network, and evaluation of algorithmic performance was compared for different SNR quantities. NOTE: This example has signal to noise ratios that are different in magnitude to the chain example
CASE 1
Consider the subnodes measured in this network. Here, once again we assessed the ability of the algorithms to recover network structure for the case shown below:
See PDF File with Details of Results
The Figures show comparisons of sensitivity and specificity for both algorithms as a function of signal to noise ratio. Here, we are using N=3 repeats of the experiments. (See the section for a more detailed explanation).
- For the robust control method we plot the sensitivity and specificity of the topologies with the best AICc score. Here the Robust Control Method recovered network structure in all cases
- For the Bayesian Inference method we plot the sensitivity and specificity at z=3 confidence intervals.
Figure 3: Comparison of Sensitivity and Specificity of robust control and bayesian inference method in the double ring. x axis: log (SNR) y axis: sensitivity and specificty. For detailed values see the pdf file above
Observation: The Robust Control Method recovers the correct network structure (with no false positives or false negatives) at the range of signal to noise ratios tested. The Bayesian Inference Method at the more stringent z=3 confidence recovers three out of the four correct connections for decreasing values of signal to noise ratio. This is shown by a sensitivity=0.75 and a specificity =1. For an SNR=28.91 it recovered the correct structure (sensitivity and specificity=1) but this trend did not happen at the highest signal to noise ratio. Overall, the robust control method performed better for inferring this topology.
Results:
Comparison of Algorithmic performance as a function of N experiments
This time, we selected a set of different signal to noise ratios and swept through different numbers of experimental repeats, assessing the minimum signal to noise ratio (here displayed as maximum gaussian noise variance) that each method could tolerate in order to perfectly recover the correct network structure (at sensitivity and specificity=1).
Network of choice for this experiment:
This set of experiments was only performed on the ring structure, as previous trials had shown that the Bayesian Inference method and the Robust Control Method are able to recover it at sensitivity=1 and specificity=1 (i.e. no false positives or false negatives). In this particular set of results, none of the trials implemented in the bayesian inference method recovered the correct network structure at 100% specificity, as they picked up false positives. Therefore, the robust control method massively outperformed the bayesian inference method. However, it is advisable to repeat these simulations to double check.
Ring
Figure: Left True Full Network. Right: Measured Network - 3 nodes
Robust Control Method
Figure 4:Ability to perfectly recover boolean structure of the ring network for different numbers of repeats (N=3,9,18,27) and noise variances. In this example red= The correct structure was found as the best AICc score and blue= The correct structure was no longer found.
Observation: For a low number of repeats (N=3) the algorithm could recover the correct boolean structure up to a gaussian noise variance=0.03 approximately. This tolerance to noise increases with increasing numbers of repeats. At N=27, the algorithm breaks down at a Gaussian noise variance of approximately 0.12. In practice this signal to noise ratio could be improved with higher numbers of repeats. Also, illustrated below in Table 1:
Table 1: Ability to recover correct boolean network structure for different numbers of repeats and signal to noise ratios.
Bayesian Inference Method
I chose not to show the results, as I was not able to reproduce the results I had obtained in some of my previous simulations for different values of noise variances. More work is needed here, but in my previous results, the maximum noise variance that the Bayesian method could tolerate was var=4e-5.The simulations here were all done for T=200 time points, and these have been taken in the transient region, before gene expression levels reach steady state.
I will show an example of one of the trials, as even lower variances than this did not recover the correct structure, at 100% sensitivity and specificity.
For values of gaussian variance=0.01 and increasing numbers of repeats, I got these networks at z=3 confidence:
N=3:
N=9:
N=18:
Figure 5: Recovered network topologies from the bayesian inference method for different numbers of experimental repeats.
Observation: None of the simulations that I implemented correctly recovered the network structure at 100% specificity and 100% sensitivity. This may require more repeats, so for the time being we conclude that the robust control method performs better, as it can tolerate more noise and does not need so many repeats of the data to provide the correct structure.
Algorithmic Performance as a function of Input Disturbances
The simulation results for this section are not displayed for the Robust Control Method, as it is unable to recover the correct network structure for a Gaussian noise disturbance. For Beal's method we show the results of boolean network reconstruction on the ring. By variation of input disturbance we mean the following:
- Step Disturbance: Stimulate each measured node in turn (on a different experiment) with a step input
- Gaussian Disturbance: Stimulate all three measured nodes at the same time for as many experiments as required with Gaussian Noise of Mean 1 and varying standard deviations
The term "Gaussian noise" refers in this description to experimental noise. To describe the "Gaussian noise perturbation" we use the term "noise perturbation".
An illustration is shown below:
Figure 5: A. Step disturbance on each node in turn. B. Noise Perturbation on all nodes at the same time.
Ring
Robust Control Method
Step Perturbation
For N=3 and noise variance=0.01, this method is able to recover the correct network structure of the ring.
Noise Perturbation
Unable to recover network structure
Bayesian Inference Method
Step Perturbation
For a Gaussian noise variance of s= 0.01, the Bayesian Inference method recovers this network, for N=3 experiments:
Noise Perturbation
For N=3 experiments, gaussian experiment noise of variance=0.01 and a gaussian perturbation noise of mean 1 and variance 0.05*, the algorithm recovers the fully connected network. More simulations need to be done here, hence results not shown. So far, for different combinations of repeats, standard deviation of noise, mean of noise and variance, I have not been able to recover the correct boolean structure (for T=200). So far it seems clear that the step perturbation outperforms the perturbation with Gaussian noise. This must be investigated further.
(*)Clarification: There are two types of noise perturbations here. The gaussian experiment noise arises due to differences in trials during observations. The gaussian perturbation noise excites the dynamics of the system at every time point, with mean 1 and standard deviation = 0.05. We had also tried running the experiment without perturbing the dynamics of the system (i.e. just the experimental noise = 0.01), and this returned the fully connected structure. Techinically, this should not have been a problem as the algorithm is supposed to be able to take in "experimental noise" and infer network structure from the previous observation. We tried to email the authors about this but did not get conclusive replies.
Ability to recover substructures in the network
Here We compared the ability of both methods to recover the correct network topology as a function of the number of substructures in the network. The double ring example was chosen for this comparison. In addition to the comparison made in the first section with the SNR, measurements were made at different points in the network, and the sensitivity and specificity of the bayesian inference and robust control methods was assessed.
Double Ring
Case 1:Measurement of 1,2,6,8
Click here to obtain the pdf version of the results, it has more detail
Robust Control Method
The first pick of the algorithm was the fully decoupled network.
The algorithm recovered the true network structure on the 2nd Best Pick (ranked according to lowest AICc values):
Figure 2nd best pick Distance: 0.0309; AIC: 3.8969; AICc: 13.2302; BIC: -11.1369
Bayesian Inference Method
The algorithm did not recover the correct structure. Here we show the results for the z=2.33 and z=3 significance levels.
z=2.33
z=3
Case 2
See this pdf file with the results
Observations: The robust control method recovered the fully correct structure on the third best AICc score. Beal's method recovered some of the connections at z=2.33 with sensitivity=6/7 and specificity= 10/13 and z=3 with sensitivity=4/7 and specificity= 11/13
