User:Steven J. Koch/Notebook/Kochlab/2009/04/07/Convert many files

From OpenWetWare
< User:Steven J. Koch‎ | Notebook‎ | Kochlab‎ | 2009‎ | 04‎ | 07
Revision as of 22:38, 8 April 2009 by Steven J. Koch (talk | contribs) (→‎The LV 8.5 DLL does NOT fix it!)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigationJump to search

Steve Koch 18:33, 7 April 2009 (EDT): Added a feature to the "convert many files" application, so you can update the filenames. I.e. replacing part of the filename string with another string, in all elements of the array. This was important, because all of my old data was converted when it was at V:\\Aatte or whatever, and now we don't have that machine anymore. I am currently testing it out by reconverting the files from around this date: \\Controller\webpub\files\data\Koch_Data\Popping_VC_BsoBI\021112\0124.dat Here's what I need to compare:

  • Old conversion: \\Controller\webpub\files\data\Koch_Data\Popping_VC_BsoBI\021112\Converted580
  • New conversion: \\Controller\webpub\files\data\Koch_Data\Popping_VC_BsoBI\021112\Converted580_Test2009

Steve Koch 19:03, 7 April 2009 (EDT): I downloaded camstudio, so I'm going to try to see if I can make a screenscast for instructions on how to use this software.

Is working, but converted data are slightly different

Steve Koch 19:11, 7 April 2009 (EDT): I subtracted the force values for file converted today and in 2002 and while very similar, they are slightly different. This is despite the fact that I tried to use all the same parameters. My guess is that I just made a mistake on copying some parameter. Or, perhaps the calibration file is not the same one. I can look into this. I want to make sure it's exactly zero, of course. Since the data pattern looks exactly like the unzipping signal, that means there is a scaling problem (as opposed to a shift).

Wednesday now

Steve Koch 14:14, 8 April 2009 (EDT):I think I may have discovered the issue. There is a bead height required for the tether center finding routine. This is hard-coded, I think, which is not good. Although, I don't know why that would have changed. So, I need to look for that.

Steve Koch 14:20, 8 April 2009 (EDT):Well, it's not as bad as I thought: a value wasn't hard-coded, but I am trying to automatically read bead height from the configuration file. I'm not sure if it is working or not. It is using the bead radius for the bead height. That seems like a reasonable idea. So, I'm not sure if this is the issue.

Secret Peeking & Convert Many are behaving slightly differently

Steve Koch 14:29, 8 April 2009 (EDT):It looks like when I manually convert a file using Secret Peeking Software, using a FTC bead height of 240 nm and all other parameters the same, I get almost exactly the same converted data as in 2002. When I subtract the two conversions, I get all zeroes, except for 5 instances of 1E-5 pN. That strikes me as some quirk that is not important. So, I just need to figure out why Convert Many Files is behaving differently now.

Duh!

Steve Koch 14:32, 8 April 2009 (EDT):I see now that it WAS a front panel setting. It was set to 265 nm by default. Why? Very stupid.

Steve Koch 14:48, 8 April 2009 (EDT):OK, Fixing that (putting front panel control to 240 nm) made File 104 segment 9 convert almost identically to the file I have from 2002. However, now the strange thing is that this isn't true for all files. The next file in the list, 105 segment 2 has a diff of up to 0.016 pN. Why?

Steve Koch 15:10, 8 April 2009 (EDT):

  • The array sizes always appear to be the same.

Steve Koch 15:18, 8 April 2009 (EDT):Weird! I converted File 105 Seg 2 with a FTC height of 500. This seemed to make it almost identical to the old converted file! So, as of now, it looks like the 2002 converted files had varying FTC bead heights. I don't know yet how that is possible.

Steve Koch 18:04, 8 April 2009 (EDT): I converted all using an FTC bead height of 580 nm (http://kochlab.org/files/data/Koch_Data/Popping_VC_BsoBI/021112/2009%20Test%20580FTC). Many of the files failed, so I would say obviously I didn't do this in the 2002 conversion. However, it's notable that File 109, S7 was much closer to the 2002 conversion.

Still a mystery

Steve Koch 18:31, 8 April 2009 (EDT): I still don't know what can account for the differences. It could be something like new subVIs in LabVIEW 7.1 behaving differently than in 6.1 when I converted the data. I think this is most likely the case in the "Find Tether Center" VI. I now think this is very likely the case'. I note that in one "problem" file, \\Controller\webpub\files\data\Koch_Data\Popping_VC_BsoBI\021112\0114 summary.doc (File 114, segment 9), I have as much as a 0.65 pN difference between data converted today and in 2002. When I look at the converted data in the Word file (see link in previous line), find tether center DID work back then. I think 265 nm was the default bead height for FTC. And that particular Word file shows bead height of 500 nm. The FTC and data in that file indicate that the FTC fitting worked then, but is not working now. The question is why?

Find Tether Center is definitely not working

Steve Koch 19:20, 8 April 2009 (EDT):It uses non-linear levenberg-marquardt stuff from LabVIEW, which makes it pretty confusing. At this point I'm pretty sure it's not working (though it's possible I forget how it "should" work and / or it didn't work before very well either). But I don't know whether it may not be working because of (a) new version of LabVIEW or (b) some kind of parameters problem. I want to see if I have LabVIEW 6.1 versions of the software somewhere.

LabVIEW 6.1

I discovered that all of my directories marked with "old" are still compiled in 6.1 This is great! (And means I wasn't as dumb as I'd thought by recompiling everything.) Unfortunately, I can't find the 2003 version of the user directory compiled in 6.1. But I do have a version fo the software shadow from late in 2001 I think. Hopefully I can get this going then.

Steve Koch 20:11, 8 April 2009 (EDT): I was able to load "Secret Peeking Software" and it compiles except for Fit FTC and Convert.vi and specifically a problem with internal peeling data anlysis.vi. It must be an old version?

It's some kind of issue with 7.1 versus 6.1 (not calibration files or anything)

Steve Koch 20:15, 8 April 2009 (EDT): I got conversion working in 6.1 on Kochlab-03, and confirmed that FTC works in 6.1 but not in 7.1 for that particular file (File 114 segment 9). I think this means it's either a compiler issue, or more likely some changed thing from the VI library...

  • \\Controller\webpub\files\data\Koch_Data\Popping_VC_BsoBI\021112\0114 Seg0009-580nm-090408-DFSTweak_LV6.1_265FTC.dat (Converted with LV6.1 today)
  • \\Controller\webpub\files\data\Koch_Data\Popping_VC_BsoBI\021112\Converted580\0114 Seg0009 BsoBI 5000 Linear.dat (Converted with LV6.1 in 2002)

The above two files have identical force values. Thus, at minimum in the near term, we could use LV6.1 for data conversion. I do want to try to figure out the problem, though. Another option is to compile the find tether center core algorithm as a DLL in 6.1 to use in 7.1 and later.

Possibly it is this DLL that has changed

Steve Koch 22:54, 8 April 2009 (EDT):This appears definitely to be the case. When I replace the DLL with the version from 6.1, the values change, and I think it works! Ugh, what a bug. The VI is called "get new coefficients.vi"

Steve Koch 22:55, 8 April 2009 (EDT): I think this is an NI page that talks about the bug. They provide a fixed VI that doesn't use a DLL: File:Get new coefficients labview fixed.vi. I am torn as to whether I should use the 6.1 DLL or the new VI...hmmm...I think I should try the new VI and if it produces exact results then it's OK.

OK, Looks like FTC is fixed in 7.1!

  • \\Controller\webpub\files\data\Koch_Data\Popping_VC_BsoBI\021112\0114 Seg0009-580nm-090408-DFSTweak_LV6.1_265FTC.dat (2009 Conversion, using LV6.1)
  • \\Controller\webpub\files\data\Koch_Data\Popping_VC_BsoBI\021112\0114 Seg0009-580nm-090408-DFSTweak_5iterations_265FTC_repaired.dat (2009 Conversion using repaired 7.1)
  • \\Controller\webpub\files\data\Koch_Data\Popping_VC_BsoBI\021112\Converted580\0114 Seg0009 BsoBI 5000 Linear.dat

Steve Koch 23:16, 8 April 2009 (EDT): After replacing the DLL described above with the version from LV6.1 library, the above 3 files all now produce exactly the same results.

Steve Koch 23:17, 8 April 2009 (EDT):I am now converting an entire set again, and putting them into this directory: \\Controller\webpub\files\data\Koch_Data\Popping_VC_BsoBI\021112\2009_repairedFTC_580,265

Steve Koch 00:56, 9 April 2009 (EDT):Success! Every single converted file in the following two directories produces exactly the same array of force values (subtracting them gives zero):

Next, I want to add a feature to also save a snapshot of the converted data (this will help with any future bugs, since it shows FTC as well)...it will also help with anyone browsing the directory looking for data.

I also want to try it with the suggested replacement VI and / or in LabVIEW 8.5.

  • Steve Koch 01:18, 9 April 2009 (EDT):Put in the image feature. It doesn't protect for whether the image file already exists, but I don't think that's an issue worth dealing with (the *.dat and *.ini files are protected by a checkbox feature).

The LV 8.5 DLL does NOT fix it!

I tried it with the DLL from LV 8.5, and that does not fix it. That's surprising. Switching to the fixed VI (no DLL) is a bit tedious, since it calls a bunch of subVIs that are not subroutine priority. What I'm going to do is move the DLL to the development directory and just rely on it in the future. I think that's safe.

Steve Koch 01:38, 9 April 2009 (EDT):OK, I've moved the 6.1 DLL to the user library, in the same directory as the FTC library (Grad School\Analysis\Curve Fitting)

Retrieved from "https://openwetware.org/mediawiki/index.php?title=User:Steven_J._Koch/Notebook/Kochlab/2009/04/07/Convert_many_files&oldid=300531"