Subject: Possible long term solution to unspecified file formats...
In-Reply-To: <1567199394.17695.0@wp10631232.mailout.server-he.de>
References: <5E32F02BACF3FD428199CED90C94A45D4E3CCF8F@ENT-PPMSG-MBX01.TDY.Teledyne.com>
<a88eb5fb-0957-4428-9013-e23693e08a50@Spark>
<5E32F02BACF3FD428199CED90C94A45D4E3CCFF6@ENT-PPMSG-MBX01.TDY.Teledyne.com>
<26861C8B-0D51-47A0-B038-4B6D1EE1155F@lbl.gov>
<5E32F02BACF3FD428199CED90C94A45D4E3CD021@ENT-PPMSG-MBX01.TDY.Teledyne.com>
<b20edf41-ca13-081a-6daf-771585464c03@uwosh.edu>
<5E32F02BACF3FD428199CED90C94A45D4E3CD05D@ENT-PPMSG-MBX01.TDY.Teledyne.com>
<dceb49a3-7dda-7c0d-19a3-4f7db0d323cf@uwosh.edu>
<1567199394.17695.0@wp10631232.mailout.server-he.de>
Message-ID: <5E32F02BACF3FD428199CED90C94A45D4E3CD0B5@ENT-PPMSG-MBX01.TDY.Teledyne.com>
I have had a pretty easy time using the options of IGOR's Load statement to deal with headers and all other issues of plain text files in routines that handle 10s of thousands of data files. I haven't had a chance to work with hdf5, but I'll see if I can implement the options below.
-- Payam
x4514
-----Original Message-----
From: Info-igor info-igor-bounces at lists.info-igor.org On Behalf Of Matthias Kastner
Sent: Friday, August 30, 2019 2:10 PM
To: Igor Discussion List <info-igor at lists.info-igor.org>
Subject: Re: Possible long term solution to unspecified file formats...
---External Email---
Just my quick thoughts from following this discussion:
There is a python library on github for reading spc files:
https://github.com/rohanisaac/spc
you could export the data to JSON using python and use the JSON XOP to import that data to Igor. Shouldn't be too complicated but I could be mistaken.
https://docs.byte-physics.de/json-xop/
I didn't look into the format specifications of SPC but the file format is quite complicated so I wouldn't try to do the reading myself in Igor and rather use the already existing library.
https://github.com/rohanisaac/spc/blob/master/spc/spc.py#L34
Asking the manufacturer for a batch ASCII export and handling that ASCII files in Igor will probably create some additional unexpected work as you will need to parse header info etc. I think, ASCII is not suitable for such batch exports. If you are asking anyway, then ask for an export to HDF5 as that is easy to handle in Igor Pro.
Regards, Matthias
On Fri, Aug 30, 2019 at 10:47 PM, Jonathan Gutow <gutow at uwosh.edu>
wrote:
> Payam,
>
> Maybe we should draft an opinion piece to C&EN and get as many people
> as possible to sign on to that instead individual letters?
>
> I am an American Physical Society member as well, so could try to be a
> conduit for such a piece in Physics Today or APS News.
>
> I suggest we try to recruit many people before submitting the initial
> piece.
>
> As such I have sent this e-mail back to the IGOR list again, because I
> think this group cuts across many disciplines.
>
> Jonathan
>
>
> On 8/30/19 3:40 PM, Minoofar, Payam N wrote:
>> Hi Jonathan,
>>
>> I am on board with that proposal. Over the past ten years, I made
>> that statement with my capital dollars by purchasing Dionex (now
>> Thermo) HPLCs because their software is the only one that does batch
>> csv export of raw data. Agilent's solution was that I either export
>> each of the hundreds and eventually thousands of chromatograms or
>> purchase an additional seat for their analysis software at the
>> bargain basement price of $5000, IIRC.
>>
>> The excuse they offer, of course, is that this is necessary for them
>> to incorporate all the FDA compliance rules within their "ecosystem",
>> but this process robs us researchers of the access we need to do the
>> manipulations that production software cannot do.
>>
>> I'll send my letter to C&EN soon.
>>
>> -- Payam
>> x4514
>>
>>
>> -----Original Message-----
>> From: Jonathan Gutow <gutow at uwosh.edu>
>> Sent: Friday, August 30, 2019 11:40 AM
>> To: Minoofar, Payam N <Payam.Minoofar at Teledyne.com>
>> Subject: Re: shimadzu .spc files...possible long term solution to
>> unspecified file formats...
>>
>> ---External Email---
>>
>> Dear Igoristi,
>>
>> I am disappointed that we are still having these discussions decades
>> after computer controlled instrumentation became common. I would like
>> to suggest a long term solution.
>>
>> All requests for bids on instruments purchased for scientific uses
>> should include in the specifications that the data file format be
>> provided with the instrument. This requires that the majority of
>> practicing scientists include this in their instrument
>> specifications.
>> The only way I see to make this work is to start a movement to do
>> this.
>> Maybe we, through our various professional organization memberships,
>> can start campaigning for this?
>>
>> Regards,
>>
>> Jonathan
>>
>> On 8/30/19 12:29 PM, Minoofar, Payam N wrote:
>>> Thanks, Benjamin. This happens to be the same Spectragryph program
>>> that Jeffrey recommended. The $300 Euro price tag might be justify
>>> for the frequency with which I will use it. The instrument does
>>> export to ASCII, but it does not perform a batch export.
>>>
>>> -- Payam
>>> x4514
>>>
>>>
>>> -----Original Message-----
>>> From: Info-igor <info-igor-bounces at lists.info-igor.org> On Behalf
>>> Of Benjamin Gilbert
>>> Sent: Friday, August 30, 2019 10:24 AM
>>> To: Igor Discussion List <info-igor at lists.info-igor.org>
>>> Subject: Re: shimadzu .spc files
>>>
>>> ---External Email---
>>>
>>> You might wish to consider a 3rd party application, which I
>>> resorted to when I could not solve this problem more easily.
>>>
>>> https://www.effemm2.de/ <https://www.effemm2.de/>
>>>
>>> I do like this software and after using the free version for the
>>> problem in hand I purchased a version.
>>>
>>> Benjamin Gilbert
>>> Senior Scientist, LBNL
>>> 1 608 358 0194
>>> Zoom: https://lbnl.zoom.us/j/5104952748
>>>
>>>> On Aug 30, 2019, at 10:11 AM, Minoofar, Payam N
>>>> <Payam.Minoofar at Teledyne.com> wrote:
>>>>
>>>> Thank you, Jeffrey. There is a Grams .spc reader in Igor Exchange,
>>>> but I assume that that is a slightly different spec from Shimadzu.
>>>> (I'm overcome by d?j? vu as it feels as if I had the same exchange
>>>> some 16 years ago in grad school.)
>>>>
>>>> -- Payam
>>>> x4514
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Info-igor <info-igor-bounces at lists.info-igor.org> On Behalf
>>>> Of Jeffrey Weimer
>>>> Sent: Friday, August 30, 2019 7:55 AM
>>>> To: Igor Discussion List <info-igor at lists.info-igor.org>
>>>> Subject: Re: shimadzu .spc files
>>>>
>>>> ---External Email---
>>>>
>>>> I may have something crude to read spc files stored on an external
>>>> HD somewhere. I believe the format is not just specific to
>>>> Shimadzu. I?ll look and verify one way or the other early next week
>>>> at the latest.
>>>>
>>>> Otherwise, I have had great success using SpectraGryph as an
>>>> intermediate translator.
>>>>
>>>>
>>>> --
>>>> Jeffrey J Weimer, Associate Professor Chemistry / Chemical &
>>>> Materials Engineering, UAH Huntsville, AL
>>>> 35899 phone: 256-824-6954 On Aug 29, 2019, 22:39 -0400,
>>>> Minoofar, Payam N <Payam.Minoofar at teledyne.com>, wrote:
>>>>> Dearest Igoristi,
>>>>>
>>>>> Has anyone written or encountered a routine for loading Shimadzu
>>>>> .spc files? I am getting the files from one of their UV-Vis
>>>>> spectrometers.
>>>>>
>>>>> Many thanks in advance.
>>>>>
>>>>> -- Payam
>>>>> x4514
>>>>>
>>>>> -------------- next part -------------- An HTML attachment was
>>>>> scrubbed...
>>>>> URL:
>>>>>
>>>>> <http://lists.info-igor.org/private.cgi/info-igor-info-igor.org/at
>>>>> ta c h ments/20190830/1892db84/attachment.html>
>>>>> _______________________________________________
>>>>> Info-igor mailing list
>>>>> Info-igor at lists.info-igor.org
>>>>> http://lists.info-igor.org/listinfo.cgi/info-igor-info-igor.org
>>>> -------------- next part -------------- An HTML attachment was
>>>> scrubbed...
>>>> URL:
>>>>
>>>> <http://lists.info-igor.org/private.cgi/info-igor-info-igor.org/att
>>>> ac h ments/20190830/fb80d40e/attachment.html>
>>>> _______________________________________________
>>>> Info-igor mailing list
>>>> Info-igor at lists.info-igor.org
>>>> http://lists.info-igor.org/listinfo.cgi/info-igor-info-igor.org
>>>> _______________________________________________
>>>> Info-igor mailing list
>>>> Info-igor at lists.info-igor.org
>>>> http://lists.info-igor.org/listinfo.cgi/info-igor-info-igor.org
>>> -------------- next part -------------- An HTML attachment was
>>> scrubbed...
>>> URL:
>>>
>>> <http://lists.info-igor.org/private.cgi/info-igor-info-igor.org/atta
>>> ch ments/20190830/f72dcdea/attachment.html>
>>> _______________________________________________
>>> Info-igor mailing list
>>> Info-igor at lists.info-igor.org
>>> http://lists.info-igor.org/listinfo.cgi/info-igor-info-igor.org
>>> _______________________________________________
>>> Info-igor mailing list
>>> Info-igor at lists.info-igor.org
>>> http://lists.info-igor.org/listinfo.cgi/info-igor-info-igor.org
>> --
>> Dr. Jonathan H. Gutow
>> Chemistry Department gutow at uwosh.edu
>> UW-Oshkosh
>> Office:920-424-1326
>> 800 Algoma Boulevard
>> FAX:920-424-2042
>> Oshkosh, WI 54901
>> http://www.uwosh.edu/facstaff/gutow/
>>
> --
> Dr. Jonathan H. Gutow
> Chemistry Department gutow at uwosh.edu
> UW-Oshkosh
> Office:920-424-1326
> 800 Algoma Boulevard FAX:920-424-2042
> Oshkosh, WI 54901
> http://www.uwosh.edu/facstaff/gutow/
>
> _______________________________________________
> Info-igor mailing list
> Info-igor at lists.info-igor.org
> http://lists.info-igor.org/listinfo.cgi/info-igor-info-igor.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://info-igor.org/attachments/20190830/ca1fbca3/attachment.html>
_______________________________________________
Info-igor mailing list
Info-igor at lists.info-igor.org
http://lists.info-igor.org/listinfo.cgi/info-igor-info-igor.org