In this page I describe the different steps used in the conversion of detector raw counts into physical quantities. The protocols and algorithms implemented in package ‘ooacquire’ >= 0.4.3 for the conversion of raw detector counts into physical quantities spectral irradiance, spectral fluence, spectral transmittance and spectral reflectance are explained. The most advanced of the protocols and correction algorithms enhance the signal to noise ratio by one order of magnitude, allowing measurements of UV-B radiation in sunlight and measurements of spectral transmitance down to OD 4.
Built with ‘ooacquire’ 0.5.1, ‘rOmniDriver’ 0.1.20, ‘photobiology’ 0.11.3, ‘ggspectra’ 0.3.12.9002 and ‘quarto’ 1.4.4 with ‘quarto CLI’ 1.5.55 on 2024-08-27.
1 Theoretical and practical considerations
Different protocols can be used for data acquisition of spectral data, and the protocol used determines which algorithms can be used for the conversion of raw-counts from the detector into spectral irradiance or spectral fluence. Raw-counts data can be acquired in near real time using Ocean Optics spectrometers, from within R, using functions in the package, or they can be read from files saved using software from Ocean Insight, including OceanView, Spectra Suite, and the software from Raspeberry Pi or the firmware in the Jaz spectrometer. How to acquire spectral data with function acq_irrad_interactive() is described in the page ‘ooacquire’: Spectral Irradiance Measurement.
1.1 Spectral irradiance and spectral fluence
The difference between irradiance and fluence is only on the basis of expression: time for irradiance and event for fluence. Irradiance describes a rate while fluence describes a total. In all other respects the algorithms are identical, and are described for irradiance. If the duration of an event is known, this value can be used to interconvert irradiance into fluence and vice-versa. With continuously-emitting light sources the length of the integration is determined by the integration time setting in the spectrometer. With pulsed light sources (flashes, strobes, etc.) the duration of a light burst would need to be know to estimate irradiance, and if this duration is not known, we can only compute fluence per light pulse.
There are many different approaches or protocols that can be used to measure spectral irradiance with an array spectrometer. Those listed below are all available through function acq_irrad_interactive() if the necessary information about the spectrometer is available.
The most basic approach is to simply measure a light source and subtract the signal from a few pixels in the array that are not exposed to light (electrical dark). With this approach it is not possible to correct for the dark signal variation among individual pixels or for stray light.
As above but using the “dark” signal from pixels at wavelengths were the measured light source does not emit radiation. This has two advantages: we may be able to use the average across a larger number of pixels. With this approach it is not possible to correct for the dark signal variation among individual pixels but if stray light is well scattered, it can be at least in part corrected for.
The most common approach is to acquire paired measurements of a light source and with the input optics blocked or the light source switched off. With this approach no correct for stray light can be applied.
As above but using in addition the “dark” signal from pixels at wavelengths were the measured light source does not emit radiation. With this approach it is possible to correct for the dark signal variation among individual pixels and if stray light is well scattered, it can be also at least in part corrected for.
In some cases it is best to measure the light source directly, through a filter that ensures that light is blocked only in a given reagion of the spectrum together with a “dark” measurement with light completely blocked/switched-off. If a filter suitable for the source is used and a correction algorithm applied it is possible to discount both the dark signal of each pixel in the whole spectrum plus the stray light received by each pixel in a constrained region of the spectrum.
Each of the approaches listed above can be combined with the use of multiple integration times (or bracketing) to enhance the dynamic range (DR) by splicing spectra acquired using different integration times for low and high spectral irradiance regions.
Dark noise depends on thermal energy and occasionally on other sources of energy, such as cosmic rays, thus dark noise has an average floor at a given temperature plus random variation superimposed. Consequently, averaging readings over time can make the observed “dark”, “filter” and “light” less variable and enhance the effectiveness of the corrections. In all the approaches described above, it is possible to acquire multiple “scans” or raw-counts spectra for each measurement using exactly the same settings and average them pixel by pixel. In many cases this averaging can be done on board the spectrometer or by the software driver on the host computer.
Note
An additional possibility is smoothing the variation across pixels using means or medians computed over a “sliding window” over wavelengths (also called box-car averaging), which also helps reduce noise but decreases wavelength resolution. This approach as well as the use of other smoothers can be most avantageously applied after spectral irradiance has been computed. Smoothing of spectra is implemented in function smooth_spct() and removal of spikes in function despike() both available in package ‘photobiology’.
Table 1 lists the measuring protocols supported. The protocols for the acquisition of raw-counts data differ in the conditions under which spectra are measured. Which of the algorithms described above can be applied is constrained by the protocol used. As the acquisition of each of light, dark and filter raw-counts acquisition from the spectrometer is done identically in the different protocols, simpler algorithms remain available. so if one has used the "lfd" protocol to acquire the spectra, the analysis as "ld" or "l" can be done by not using some raw-counts spectra. The order of measurements does not affect the algorithms that can be used. Some protocols are best suited to specific types of light sources, and in some cases not usable. Different algorithms may require re-validation of the calibration of the spectrometer for best accuracy of measurements.
Table 1: Supported protocols for spectral data acquisition. The protocol names describe which spectra are acquired, and in most cases also used in the computations. The high dynamic range (HDR) approach involves splicing spectra acquired using different integration times so as to improve the date acquired for thr low-signal regions of spectra.
Protocol
Light
Dark
Filter
HDR
“l”
yes
no
no
yes/no
“ld”, “dl”
yes
yes
no
yes/no
“lfd”, “dfl”
yes
yes
yes
yes/no
By using these protocols we obtain different sets of raw counts data from the spectrometer’s detector array. The raw-counts need to be converted into spectral irradiance or spectral fluence (energy or photon based). Not all detectors behave exactly in the same way, and the behaviour of a given spectrometer may depend on its temperature. For example, for some types of array detectors the dark noise increases significantly only at very long integration times while for others already with shorter integration times.
Note
The acquisition of spectra following these protocols is automatic when using function acq_irrad_interactive() but its use is not a requirement. The protocols can be followed when acquiring spectra using other software to be later imported into R. In such cases, lack of consistency in settings due to mistakes can be a problem.
To be on the safe side even if not strictly necessary, function acq_irrad_interactive() when HDR is enabled applies the same multiple integration times (HDR) to “light”, “dark” and “filter” spectra. This makes it possible to analyse the same data as if no HDR had been used.
In addition, corrections for the shape of the slit function and for stray light are implemented by some of the algorithms. These corrections require additional characterization and calibration of individual spectrometers compared to those done by manufacturers or most users. These additional corrections are those described in Yliantilla et al. (2005) and later modified to make them applicable to different models of spectrometers (Ylianttila, unpublished).
What are the gains and what is needed?
Package ‘ooacquire’ provides a lot of flexibility but the gains achieved by special algorithms and protocols compared to the simpler approaches routinely used, will depend on the goodness of the calibration, its traceability, the characteristics of the spectrometer and of the light source being measured. A good array spectrometer has a signal to noise ratio of about \(1\,000\) (compared to \(1\,000\,000\) in a double monochromator scanning spectrometer), and in the best case the algorithms and protocols implemented in ‘ooacquire’ can improve the dynamic range by one order of magnitude to slightly more than \(10\,000\). This difference is enough to allow measurement of UV-B radiation in sunlight when the sun is some degrees above the horizon, which is otherwise impossible.
What are the limitations? The characterization of the spectrometer and validation of the method are time consuming and expensive. Without an automatic filter wheel, the “l” protocol is the only one not requiring operator intervention for each acquired spectrum. One way to solve this last difficulty is to reuse the same “dark” and “filter” reference spectra for multiple “light” spectra, an approach implemented in function acq_irrad_interactive() as repeated and time series measurements (‘ooacquire’ >= 0.4.4). The validity of a “dark” measurement can be maintained by keeping the spectrometer at a constant controlled temperature. The “filter” measurement depends, in addition, on the spectrum of the light source, and only a limited amount of re-scaling works reliably in sunlight because the light spectrum depends on solar elevation and clouds.
When a signal to noise ratio of \(1\,000\) is enough or in the case of spectrometers that are well designed to minimize stray light, “filter”-based corrections are not needed, and the alternative simpler approaches, also implemented, can be used. Package ‘ooacquire’ can be used to measure spectral irradiance even if the only calibration available is that supplied by Ocean Insight, and frequently the calibration constants can be downloaded from the spectrometer. In the absence of a special characterization of the spectrometer, the algorithms and protocols implemented in ‘ooacquire’ can still improve the dynamic range through HDR, as long as the spectrometer is not affected by strong stray light.
To keep in mind…
Pixel resolution of array spectrometers is usually higher, sometimes a lot higher, than the true optical resolution of the monochromator grid. Use the optical wavelength resolution as criterion when applying smoothing and when identifying peaks and valleys.
The wavelength step between pixels varies along the array, and by how much depends on the optical configuration. Valid integration over wavelengths is not a simple addition.
Array detectors do not respond linearly to the number of photons, so a linearisation correction is always needed. If acquisition is done with function acq_irrad_interactive() this correction is automatically applied.
The digitised data for each pixel is a “count”, i.e., an integer number. The maximum count (related to the number of bits) varies, but it is usually not more that 65000. The real dynamic range of array spectrometers is always a lot less than this.
Dark noise of the sensor increases with temperature, so some spectrometers have cooled arrays. Spectrometers with no cooling should be shaded from direct sunlight. For accurate measurements spectrometers should be allowed to warm up or cool-down until their temperature is stable (can take 30 min or more if the change in temperature is large).
Temperature of the spectrometer also affects the wavelength calibration, but how much depends on the design of each spectrometer model. When using the spectrometer at temperatures very different from 20 C, consider if a shift in the reported wavelengths can be important.
Sometimes we know a priori which wavelengths are not present in a light source. _We can take advantage of this for sanity checks. Say if one’s measurement of UV-C in sunlight at ground level is different from zero, this tells that the measurement is subject to error, not that UV-C is being transmitted through the atmosphere!!
Know the limitations of the spectrometer and the protocols you use. Keep in mind that all measurements are subject to both random errors and bias, what matters is whether these errors are large enough to affect the conclussions being drawn from the data.
1.2 Ylianttila et al.’s (2005) method
I recommend reading the original paper for the details of the original algorithm and under which conditions the corrections are effective. The method described corresponds, approximately, to the method named ylianttila in the package. It uses only light and dark spectra together with HDR using a factor of 10. This method was originally developed for measurements of sunbeds. This method requires special characterization of the spectrometer characteristics and a calibration that is good enough to achieve the additional accuracy. The calibration is normally done using the same protocol as for measurements and at a similar temperature. For each type of spectrometer and configuration the validity of the method needs to be demonstrated by comparison to a double monochromator scanning spectrometer.
1.3 Ylianttila’s (unpublished) method for sunlight
Method ylianttila uses the algorithm for stray light correction developed by Lasse Ylianttila for use with the Maya 2000 Pro based on that in Ylianttila et al. (2005). The Maya 2000 Pro suffers from stray light, most notably with light sources emmiting IR at wavelengths > 1000 nm. (Early units seem to have worse performance in this respect than more recent ones, even for the same configuration.)
UV-B radiation in sunlight is especially difficult to measure, in general impossible to measure reliably with array spectrometers without a custom sophisticated protocol and correction algorithm. This new method adds a third measurement through a polycarbonate filter (PC). It is crucial to use PC as filter as many other “better” UV blocking filters do not have sufficiently high transmittance in the IR and thus partly block the stray light. As the PC filter does absorb some visible and IR radiation, about 10-15% for normal incidence, this needs to be compensated as well as any change in incident irradiance between the light and filter measurements. To achieve this, the two measurements are compared at wavelengths in the UV-A1 region and the filter measurement rescaled. This works in sunlight or with other continuous spectra but not with discontinuous spectra with low or no radiation in the region used for rescaling. (When rescaling is not possible a warning is issued and the corrections based on the filter measurement is skipped.)
A diagram of the data flow for the computation of irradiance is shown in Figure 1. To keep the code simpler and ensure consistent computations, the data for "light", "filter" and "dark" measurements are processed using the same algorithms before they are combined. The splicing algorithm is, however, based on the signal characteristics of each spectrum. Quality Control (QC) detects unexpected non-random noise in dark spectra, based on all wavelengths, and in filter spectra only for the “blocked” wavelengths. QC is based on counting how many individual pixels grossly deviate from the expected range based on the variability around the median of all the tested pixels/wavelengths. Hot pixels are more abundant than cold pixels, and this is also taken into account. The tolerances have been set by trial and error, and may change. QC was implemented in 2023, it does not change computations in any way, it triggers a warning and saves to an attribute of the spectra whether the test was passed or not.
Spikes, or solitary hot pixels, are normally removed based on the calibration data, as it includes information on known hot pixels. Despiking is only applied, with a warning, when when additional spikes or obvious hot pixels are detected. Handling of clipped/saturated pixel data and splicing described in later sections. Linearisation is sometimes applied during acquisition. The linearisation history of each spectrum is recorded as metadata in an attribute, both for spectra imported from files and those acquired with ‘ooacquire’. For spectra already linearised, the linearisation step is skipped.
flowchart TB
lr([<b>light</b>\nraw-counts HDR]) --replace bad pixels\nreplace clipped by NA\nlinearize--> ll[<b>light</b>\nlinearised raw HDR]
-->|occluded pixels corr.\ndivide by\nintegration time| lc[<b>light</b>\ncps HDR]
--splice\nslit fun. corr.\n<i>despike</i>--> l[<b>light</b>\ncps spliced]
dr([<b>dark</b>\nraw-counts HDR]) --replace bad pixels\ndark QC\nlinearize--> dl[<b>dark</b>\nlinearised raw HDR]
-->|occluded pixels corr.\ndivide by\nintegration time| dc[<b>dark</b>\ncps HDR]
--<i>splice</i>\nslit fun. corr.\n<i>despike</i>--> d[<b>dark</b>\ncps <i>spliced</i>]
fr([<b>filter</b>\nraw-counts HDR]) --replace bad pixels\nreplace clipped by NA\ndark QC\nlinearize
--> fl[<b>filter</b>\nlinearised raw HDR]
-->|occluded pixels corr.\ndivide by\nintegration time| fc[<b>filter</b>\ncps HDR]
--splice\nslit fun. corr.\n<i>despike</i>--> f[<b>filter</b>\ncps spliced]
l --> lm(("subtract\ndark")) --> ldc[<b>light</b> cps\ndark-corrected]
d --> lm
f --> fm(("subtract\ndark")) --> fdc[<b>filter</b> cps\ndark-corrected]
d --> fm
ldc --> slc(("stray light\ncorrection")) --> cc[<b>light</b> cps\nfilter-corrected]
fdc --> slc
cc --mutiply by\ncalibration factors--> irrad([<b>spectral\nirradiance</b>])
irrad -.->|<i>smooth</i>|sirrad([<i>smoothed</i>\n<b>spectral\nirradiance</b>])
Figure 1: Data flow for conversion of raw-counts into spectral irradiance when using the protocols "lfd" or "dfl". The dark spectrum is never spliced as the longest integration time is the best across all wavelengths and used as is. QC stands for Quality Control and is applied to wavelengths expected to be subject only to dark noise.
Using method ylianttila
Variations of the method for individual spectrometers differ in which pixels/wavelengths are used with the same base algorithms and in the fitted slit functions on which slit corrections are based, consequently, not only the calibration but also the parametrization of the correction method algorithm is not transferable, even among spectrometers of the same model and identical optical configuration. Thus using this sophisticated method requires and initial optical characterisation of the spectrometer, the estimation of parameter values to use and a validation of the method under conditions similar to those under which it will be used.
Alternative, less accurate, but simpler to implement methods are also available in ‘ooacquire’.
Entrance optics
The response to incident light depends on the angle of incidence. Irradiance is defined as the light absorbed on a flat surface, most commonly a horizontal surface. The area of a plane perpendicular to the light beam projected on a horizontal plane, can be computed based on the cosine of the angle of incidence. To measure irradiance the entrance optics must have an angular response very close to the cosine. Most diffusers deviate from this at very shallow angles of incidence. How much they deviate depends quite a lot on their design (and cost). The actual transmittance of the diffuser may vary slightly depending on the individual unit and its age.
Normally, a reliable approach to calibration is to do the calibration with the same entrance optics as used for measurements. Thus, producing a separate calibration for each entrance optics used. Ocean Insight, confusingly, expresses irradiance calibrations per unit area of the entrance optics. This means that to be able to use calibration data supplied by Ocean Insight or obtained with their software, the area or type of diffuser must be also supplied.
Cosine diffusers differ a lot in price and performance. The differences in performance are especially important at shalow angles of light incidence. Under this condition it is also crucial that the surface of the diffuser is perfectly horizontal. A further mode of error in diffusers is for the vertical angular response to vary with the horizontal angle of rotation.
Accuracy can be best achieved by controlling all errors to the same low level. Using a sophisticated algorithm with an entrance optic (diffuser) that is not correctly levelled or when the operator or objects affect the light impinging on the entrance optics with shading or reflections, will not inhance overall accuarcy.
A low quality entrance optic will bias measurements, and imperfect levelling can cause mayor bias in the measurements at shallow angles of incidence. Accurately levelling the diffuser is a crucial and easy step.
Polycarbonate filter shape
The filter used for the reference measurement must be of polycarbonate (PC) as it blocks all UV-radiation and has high transmittance in the infrared. Reflectance at the surface of an interface depends on the angle of incidence and is lowest when the beam is normal to the surface.
Reflectance at the polycarbonate-air boundaries of the filter used for the reference measurement, increases as the angle of incidence of the light on a flat filter becomes shalower. This makes the stray-light-correction algorithm less effective at shallow angles of incidence. Recently, I have started using dome-shaped polycarbonate filters to avoid this problem when measuring sunlight. Polycarbonate domes of very good optical characteristics are available as spare parts for surveillance cameras. They come in various sizes. Some are crystal clear and suitable for this use, while others are grey-tinted and unsuitable. These domes are not expensive and provide a very effective solution to the problem. With a well centred dome, independently of the light beam angle, the light reaching the diffuser has been normal or nearly normal to the dome surface, so equally affected by the transmittance of the dome.
The filter-based correction of stray light provides a very small improvement when measuring UV-A1 or visible radiation in sunlight as spectral irradiance in these regions is high compared to stray light. In the case of the Maya 2000 Pro it is also very rarely necessary when the light source emits little or no radiation at wavelengths > 1000 nm, as this is the main source of stray light affecting short wavelengths. So, when measuring narrow band visible or UV radiation from LEDs, the filter correction is both unnecessary and in most cases also impossible to apply with the existing algorithms.
With the Maya 2000 Pro there is one situation where correction for stray light would be needed but is not possible: measuring a LED emitting at wavelengths > 950 nm. In this case, however, we can assume that any signal in the UV-region is stray light, i.e., it is always wise to take into account the known properties of the light source as a base for interpreting data and to use this knowledge to manually remove stray light or discard bad data in the affected range of wavelengths.
1.4 Other methods implemented
I have implemented another variation on Ylianttila’s method to correct for stray light named sun that differs in that uses trimmed means trim = 0.05 instead of trim = 0. All the steps are as in Figure 1.
Method simple uses a simple ad-hoc method to correct for stray light based on a constant value for the PC filter transmittance. It is based on light, filter and dark measurements but with no computed rescaling of the filter measurement except for multiplication by a constant. It is used when the signal of the "light" is too low in the wavelength region used in Ylianttila’s method for the rescaling. This can be case with lamps and LEDs. All the steps are as in Figure 1, but the computation of the stray light corrections uses a simpler and more error-prone algorithm.
Method none does not apply any correction for stray light, is based on light and dark measurements only. The steps are shown in Figure 2. It is suitable for LEDs and some types of lamps. I can be used even when the calibration has been provided by Ocean Optics or acquired with Ocean Optics’ software. It still applies slit function correction if available. If not combined with bracketing of integration time and no slit correction is applied, it is similar (identical?) to spectral irradiance computed by Ocean Optics software.
flowchart TB
lr([<b>light</b>\nraw-counts HDR]) --replace bad pixels\nreplace clipped by NA\nlinearize--> ll[<b>light</b>\nlinearised raw HDR]
-->|occluded pixels corr.\ndivide by\nintegration time| lc[<b>light</b>\ncps HDR]
--splice\nslit fun. corr.\n<i>despike</i>--> l[<b>light</b>\ncps spliced]
dr([<b>dark</b>\nraw-counts HDR]) --replace bad pixels\ndark QC\nlinearize--> dl[<b>dark</b>\nlinearised raw HDR]
-->|occluded pixels corr.\ndivide by\nintegration time| dc[<b>dark</b>\ncps HDR]
--<i>splice</i>\nslit fun. corr.\n<i>despike</i>--> d[<b>dark</b>\ncps <i>spliced</i>]
l --> lm(("subtract\ndark")) --> ldc[<b>light</b> cps\ndark-corrected]
d --> lm
ldc --mutiply by\ncalibration factors--> irrad([<b>spectral\nirradiance</b>])
irrad -.->|<i>smooth</i>|sirrad([<i>smoothed</i>\n<b>spectral\nirradiance</b>])
Figure 2: Data flow for conversion of raw-counts into spectral irradiance when using the protocols "ld" or "dl". The dark spectrum is never spliced as the longest integration time is the best across all wavelengths and used as is. QC stands for Quality Control and is applied to wavelengths expected to be subject only to dark noise.
If only a "light" measurement is available, method none is used but the dark noise is corrected less effectively by using only pixels assumed to be "dark" in the "light" spectrum. The steps are shown in Figure 3. When using pixels in the UV-C region as dark reference, a very rough stray-light correction is applied. The spectra can, rather easily, end up being either over-corrected or under-corrected for stray light. For this approach to work correctly, the stray light and/or dark noise must not affect preferentially some regions of the array detector but instead evenly the whole array. In practice, with LEDs, this approach works surprisingly well with the Maya 2000Pro spectrometer.
flowchart TB
lr([<b>light</b>\nraw-counts HDR]) --replace bad pixels\nreplace clipped by NA\nlinearize--> ll[<b>light</b>\nlinearised raw HDR]
-->|occluded pixels corr.\ndivide by\nintegration time| lc[<b>light</b>\ncps HDR]
--splice\nslit fun. corr.\n<i>despike</i>--> l[<b>light</b>\ncps spliced]
l --> lm(("subtract\ninternal\ndark")) --> ldc[<b>light</b> cps\ndark-corrected]
ldc --mutiply by\ncalibration factors--> irrad([<b>spectral\nirradiance</b>])
irrad -.->|<i>smooth</i>|sirrad([<i>smoothed</i>\n<b>spectral\nirradiance</b>])
Figure 3: Data flow for conversion of raw-counts into spectral irradiance when using the protocol "l". The dark spectrum is never spliced as the longest integration time is the best across all wavelengths and used as is. QC stands for Quality Control and is applied to wavelengths expected to be subject only to dark noise.
The methods form a continuum, with Ylianttila’s method as the most effective and complex. When the data do not permit its use, the functions automatically attempt to use the next simpler one.
All the available methods can be combined with bracketing of integration time as shown in the diagrams. If no HDR is used, no splicing is done, but otherwise the same steps are followed.
Missing "dark" measurement
The following approach is attempted when "light" and "filter" are available and a "dark" measurement is missing. This approach is unlikely to be good enough to be worthwhile using on purpose, but can save the day if the "dark" measurement has been accidentaly lost.
flowchart TB
lr([<b>light</b>\nraw-counts HDR]) --replace bad pixels\nreplace clipped by NA\nlinearize--> ll[<b>light</b>\nlinearised raw HDR]
-->|occluded pixels corr.\ndivide by\nintegration time| lc[<b>light</b>\ncps HDR]
--splice\nslit fun. corr.\n<i>despike</i>--> l[<b>light</b>\ncps spliced]
fr([<b>filter</b>\nraw-counts HDR]) --replace bad pixels\nreplace clipped by NA\ndark QC\nlinearize
--> fl[<b>filter</b>\nlinearised raw HDR]
-->|occluded pixels corr.\ndivide by\nintegration time| fc[<b>filter</b>\ncps HDR]
--splice\nslit fun. corr.\n<i>despike</i>--> f[<b>filter</b>\ncps spliced]
l --> lm(("subtract\ninternal\ndark")) --> ldc[<b>light</b> cps\ndark-corrected]
f --> fm(("subtract\ninternal\ndark")) --> fdc[<b>filter</b> cps\ndark-corrected]
ldc --> slc(("stray light\ncorrection")) --> cc[<b>light</b> cps\nfilter-corrected]
fdc --> slc
cc --mutiply by\ncalibration factors--> irrad([<b>spectral\nirradiance</b>])
irrad -.->|<i>smooth</i>|sirrad([<i>smoothed</i>\n<b>spectral\nirradiance</b>])
Figure 4: Data flow for conversion of raw-counts into spectral irradiance when using the protocols "lf" or "fl". The dark spectrum is never spliced as the longest integration time is the best across all wavelengths and used as is. QC stands for Quality Control and is applied to wavelengths expected to be subject only to dark noise.
Available methods depend on the spectrometer
What method can be applied depends on how the spectrometer has been characterized and calibrated. The details of each of the methods need to be adjusted to each instrument, and the calibration done using the same corrections or a superset of the corrections used for actual measurements. For a calibration already expressed as multipliers, as supplied by Ocean Insight, only method none is applicable. If the calibration lamp data and the raw counts for the calibration measurements are available for light, filter and dark conditions, calibrations for any of the methods can be computed, even retrospectively. The methods as described include a “tail correction” for the slit function, based on the characterization of the slit function at different wavelengths using a tunable UV laser. This corrections is most important for sources with narrow peaks such as discharge lamps, and can be skipped if these data are unavailable.
1.5 Irradiance calibration data
Wavelength calibration is separate from the irradiance calibration and always needed when using the spectrometer.
Most commonly, irradiance calibration data is expressed as a constant multipliers, \(k_\textrm{i}\), expressed in \(J m^{-2} nm^{-1} c^{-1}\), where \(\textrm{i}\) is an index pointing to an individual pixel and \(c\), raw detector counts. In this approach, used in ‘ooaquire’, CPS data (\(c s^{-1}\)) for each pixel are multiplied by the corresponding \(k_\textrm{i}\), obtaining spectral irradiance in \(W m^{-2} s^{-1} nm^{-1}\) as \(W = J s^{-1}\). These calibrations are for whole-system response, including the diffuser and the fibre.
In contrast, irradiance calibration data supplied by Ocean Optics are expressed in energy units per detector count per unit area. In other words need to be corrected for the wavelength step between detector pixels and for the area of the cosine diffuser used. The equation to use is \[E_\lambda = \frac{k_\lambda \cdot c_\lambda}{A \cdot t
\cdot \Delta \lambda}\] where \(k_\lambda\) is the calibration constant for a detector pixel, and \(c_\lambda\) are the counts registered by the corresponding detector pixel, \(A\) is the area of the diffuser, \(t\) is the integration time and \(\Delta \lambda\) is the wavelength step for the pixel.
With \(k_\lambda\) in \(\mu J c^{-1}\), \(A\) in \(mm^2\), \(t\) in seconds and \(\Delta \lambda\) in \(nm\), the result as spectral energy irradiance in \(W m^{-2} nm{-1}\)
We can substitute \(c_\lambda / t\) by “counts per second”, which is what we use in the computations, and recalculate the remaining of the equation as a new set of calibration constants.
First we describe how the data and metadata are stored. Next show the use of function s_irrad_corrected() to do the conversion from raw-counts into spectral irradiance applying verious corrections. Later we walk through the main stages on this conversion.
2.1 From raw counts to spectral irradiance
Below different correction methods are compared and some of the steps of the algorithms are exemplified, plotting both the intermediate and final. Raw counts data included in package ‘ooacquire’ are used as example. The data were acquired with function acq_irrad_interactive() using both HDR mode and both “dark”and“filter”` reference spectra. These data are from a measurement of spectral irradiance of sunlight at ground level, acquired with an Ocean Optics Maya 2000 Pro array spectrometer.
We will use photon/quantum units throughout for plotting of spectra and display of computed summaries, while printing remains unaffected. We override the default use of energy units by changing the R option with a convenience function.
photon_as_default()
2.1.1 Spectral data
Spectral data used in this example were acquired using function acquire_irrad_interactive(). This function implements the measurement protocols described above. RAW-counts data are returned as a collection of RAW spectra (an object of class raw_mspct constructed with function raw_mspct()) that has also metadata stored as attributes.
The instrument descriptor contains the model, serial number, and calibration information when available. The instrument settings used are also stored. Thus, irrespective of the algorithm used for corrections and conversion and the number of individual spectra acquired with the spectrometer, all the data needed for computing spectral irradiance are stored together. In the case of a time series, data for multiple irradiance spectra, sharing the same metadata, except for time of acquisition, and also sharing"dark" and/or "filter" are stored in the same R object. If the calibration data are included in the metadata, then all the information needed to compute irradiance is stored in this single object, otherwise the calibration must be available separately.
Function acquire_irrad_interactive() can return spectral irradiance plus raw counts data, counts per second plus raw counts data, or only the raw counts data. For this example, we use the raw counts data in object sun009.raw_mspct.
The use of function acquire_irrad_interactive() to acquire the raw counts data is not a requirement, as data can be also imported from files saved as raw counts using Ocean Optics/Ocean Insight software (SpectraSuite or OceanView). During import most of the metadata are read from the file header, except for the calibration data. Calibration data can be imported separately from a file or read from the spectrometer memory, unless already included in the ‘ooacquire’ package.
The data, measured using protocol "lfd", contain scans acquired for the light source, scans for the same light source through a polycarbonate filter and a reference or dark measurement. The "light" and "filter" data should be acquired under comparable conditions, ideally the only difference should be the presence of the filter. In practice a small difference in irradiance can be compensated for by the algorithm.
Each of the three members in the raw-counts data collection is a raw_spct object (created with function raw_spct()). We load this object.
load("data-files/sun-viikki-noon.Rda")
We can use summary on the collection to explore the object.
summary(sun009.raw_mspct)
Summary of raw_mspct [3 x 1] object: sun009.raw_mspct
# A tibble: 3 × 7
spct.idx class dim w.length.min w.length.max colnames multiple.wl
<chr> <chr> <chr> <dbl> <dbl> <list> <dbl>
1 light raw_spct [2,068 x 3] 187. 1117. <chr [3]> 1
2 filter raw_spct [2,068 x 3] 187. 1117. <chr [3]> 1
3 dark raw_spct [2,068 x 3] 187. 1117. <chr [3]> 1
str(summary(sun009.raw_mspct))
List of 4
$ orig.name : chr "sun009.raw_mspct"
$ orig.class : chr "raw_mspct"
$ orig.dim_desc: chr "[3 x 1]"
$ summary : tibble [3 × 7] (S3: tbl_df/tbl/data.frame)
..$ spct.idx : chr [1:3] "light" "filter" "dark"
..$ class : Named chr [1:3] "raw_spct" "raw_spct" "raw_spct"
.. ..- attr(*, "names")= chr [1:3] "light" "filter" "dark"
..$ dim : Named chr [1:3] "[2,068 x 3]" "[2,068 x 3]" "[2,068 x 3]"
.. ..- attr(*, "names")= chr [1:3] "light" "filter" "dark"
..$ w.length.min: Named num [1:3] 187 187 187
.. ..- attr(*, "names")= chr [1:3] "light" "filter" "dark"
..$ w.length.max: Named num [1:3] 1117 1117 1117
.. ..- attr(*, "names")= chr [1:3] "light" "filter" "dark"
..$ colnames :List of 3
.. ..$ : chr [1:3] "w.length" "counts_1" "counts_2"
.. ..$ : chr [1:3] "w.length" "counts_1" "counts_2"
.. ..$ : chr [1:3] "w.length" "counts_1" "counts_2"
..$ multiple.wl : num [1:3] 1 1 1
- attr(*, "class")= chr [1:3] "summary_raw_mspct" "summary_generic_mspct" "list"
- attr(*, "comment")= chr "Comment from ''.\n"
The spectra contain multiple columns, one, "w.length", for calibrated wavelengths, and "counts_1" and "counts_2" each, for a different integration time used during acquisition (base on the HDR multipliers).
We next use summary() on one of the members of the collection. Summary in addition to displaying the summary for the vectors or columns, displays the most important metadata attributes, including the integration times and total measurement times.
summary(sun009.raw_mspct[["light"]])
Summary of raw_spct [2,068 x 3] object: anonymous
Wavelength range 187.42-1117.27 nm, step 0.41-0.48 nm
Label: light: sun009
Measured on 2022-04-22 09:15:40.313578 UTC
Data acquired with 'MayaPro2000' s.n. MAYP11278
grating 'HC1', slit '010s'
diffuser 'unknown'
integ. time (s): 0.15, 1.5
total time (s): 10, 10.5
counts @ peak (% of max): 92.6Variables:
w.length: Wavelength [nm]
counts_1: Raw detector counts [number]
counts_2: Raw detector counts [number]
--
w.length counts_1 counts_2
Min. : 187.4 Min. : 2180 Min. : 2100
1st Qu.: 431.4 1st Qu.: 8443 1st Qu.:60997
Median : 668.3 Median :27394 Median :64000
Mean : 663.0 Mean :28624 Mean :53023
3rd Qu.: 897.3 3rd Qu.:48609 3rd Qu.:64000
Max. :1117.3 Max. :59449 Max. :64000
We can explore the structure of the object, to see how data and metadata are stored using str() as shown below. We here limit the nesting level so that the output is not too large.
# not runstr(sun009.raw_mspct[["light"]], max.level =2)
A formatted printout of the instrument setting provides important information. The maximum count value observed relative to the sensor’s maximum allowed counts is specially important for diagnostics of data quality, as a value of 100% indicates clipping while low values, say less than 70% result in decreased dynamic range due to sensor dark noise.
getInstrSettings(sun009.raw_mspct[["light"]])
integ. time (s): 0.15, 1.5
total time (s): 10, 10.5
counts @ peak (% of max): 92.6
As with any other R object we access an attribute and use indexing to extract a given metadata value.
attr(sun009.raw_mspct[["light"]], which ="instr.desc")$spectrometer.name
[1] "MayaPro2000"
2.1.2 Ylianttila’s modified method
In normal use, we calculate irradiance from a raw-counts data set stored as a collection in a raw_mspct object using the high-level function s_irrad_corrected(). We pass as first argument the object containing the RAW counts and corresponding metadata and the correction method to be used in the conversion of the RAW counts into irradiance. In this case we use the modified method developed by Lasse Ylianttila for the Maya 2000 Pro spectrometer, which suffers more from stray light than the s2000 spectrometer in Ylianttila et. al (2005), but at the same time has improved sensitivity to UV radiation.
The R object returned by s_irrad_corrected() belongs to class "source_spct" (defined in package ‘photobiology’) for which summary and plotting methods are available. We first have a quick look at this object. In the output below , it can be seen that the six RAW measurements, two in the light, two with a filter and two in the dark have been combined to calculate a single estimate of spectral irradiance. We can also see that the range of wavelengths is narrower as only wavelengths for which calibration data exists have been retained (within 250 to 900 nm in this case). Even though we have set the use of photon based units as the default, the returned spectrum is expressed in energy units as this are used in the calibration data.
summary(sun009_recalc.spct)
Summary of source_spct [1,421 x 2] object: sun009_recalc.spct
Wavelength range 251.29-898.97 nm, step 0.43-0.48 nm
Label: light: sun009
Measured on 2022-04-22 09:15:40.313578 UTC
Variables:
w.length: Wavelength [nm]
s.e.irrad: Spectral energy irradiance [W m-2 nm-1]
--
w.length s.e.irrad
Min. :251.3 Min. :-0.001192
1st Qu.:418.4 1st Qu.: 0.586815
Median :582.3 Median : 0.786671
Mean :579.9 Mean : 0.723652
3rd Qu.:742.6 3rd Qu.: 0.972572
Max. :899.0 Max. : 1.154600
By plotting the spectral irradiance data, either in whole or a range of wavelengths we can better see the shape of the solar spectrum at ground level, and in this example also obtain summaries for the irradiance in different wavebands. As we have set the R option to use by default photon based units, the plots also uses this units, doing the necessary conversion on-the-fly.
autoplot(sun009_recalc.spct)
Note
This spectrum, measured under near-optimal conditions, gives (on a per nm basis) a ratio of 1:2000 between PAR and UV-C spectral irradiance. The UV-B band is 35 nm wide, while the PAR band is 300 nm wide, so on a per band basis, the ratio becomes nearly 1:20000, or between 5% and 10% of the UV-B reading in this example. When the sun is low in the sky, the UV-B irradiance is much less so the error becomes larger compared to the actual UV-B irradiance, but remains similar compared to the PAR irradiance.
The zoomed-in plot shows that there is a small amount of noise remaining.
autoplot(sun009_recalc.spct, range =c(250, 315))
Averaging of multiple scans was used in this case.
getInstrSettings(sun009_recalc.spct)
integ. time (s): 0.15, 1.5
total time (s): 10, 10.5
counts @ peak (% of max): 92.6
Using get_attributes() we can see that relevant metadata has been copied to the new object.
get_attributes(sun009_recalc.spct)
Note
If a calibration for spectral irradiance is not available, we can still compute corrected counts per second, which can be used for relative measurements comparing two conditions, array-detector-pixel by pixel. One example is to compute spectral transmittance.
We can use the same function, s_irrad_corrected() to obtain counts per second instead of spectral irradiance. In other words linearising and dividing the raw counts by the integration time, splicing of spectra obtained using different integration times, applying the corrections inherent to the method selected and the wavelength calibration, but not the calibration for the sensitivity of the pixels.
As the wavelength calibration is available for the whole wavelength range, the returned spectrum expressed as counts per second includes all the wavelengths acquired. In this case, the returned object is of class "cps_spct".
summary(sun009_recalc.cps_spct)
Summary of cps_spct [2,068 x 2] object: sun009_recalc.cps_spct
Wavelength range 187.42-1117.27 nm, step 0.41-0.48 nm
Label: light: sun009
Measured on 2022-04-22 09:15:40.313578 UTC
Data acquired with 'MayaPro2000' s.n. MAYP11278
grating 'HC1', slit '010s'
diffuser 'unknown'
integ. time (s): 0.15, 1.5
total time (s): 10, 10.5
counts @ peak (% of max): 92.6Variables:
w.length: Wavelength [nm]
cps: Detector counts [number s-1]
--
w.length cps
Min. : 187.4 Min. : -1122
1st Qu.: 431.4 1st Qu.: 38944
Median : 668.3 Median :168899
Mean : 663.0 Mean :181284
3rd Qu.: 897.3 3rd Qu.:322465
Max. :1117.3 Max. :402998
By plotting the raw counts expressed as counts per second we can see that the shape of the curve is affected by the varying sensitivity of the spectrometer to photons of different wavelengths. The sensitivity decreases towards the shortest and longest wavelengths.
autoplot(sun009_recalc.cps_spct)
The instrument descriptor has been copied to
getInstrDesc(sun009_recalc.cps_spct)
Data acquired with 'MayaPro2000' s.n. MAYP11278
grating 'HC1', slit '010s'
diffuser 'unknown'
Looking in more detail into this descriptor with str().
str(getInstrDesc(sun009_recalc.cps_spct))
List of 16
$ time : POSIXct[1:1], format: "2016-11-02 16:34:05"
$ sr.index : int 0
$ ch.index : int 0
$ spectrometer.name: chr "MayaPro2000"
$ spectrometer.sn : chr "MAYP11278"
$ bench.grating : chr "HC1"
$ bench.filter : chr "000"
$ bench.slit : chr "010s"
$ min.integ.time : int 7200
$ max.integ.time : int 7200000
$ max.counts : int 64000
$ wavelengths : num [1:2068] 187 188 188 189 189 ...
$ bad.pixs : int [1:7] 123 380 388 467 697 1829 1994
$ inst.calib :List of 9
..$ nl.fun :function (x)
.. ..- attr(*, "srcref")= 'srcref' int [1:8] 25 15 25 44 15 44 575 575
.. .. ..- attr(*, "srcfile")=Classes 'srcfilealias', 'srcfile' <environment: 0x000001c5ccd191c8>
..$ straylight.coeff: num 0
..$ straylight.slope: num 0
..$ wl.fun :function (x)
..$ slit.fun : logi NA
..$ irrad.mult : num [1:2068] 0 0 0 0 0 0 0 0 0 0 ...
..$ wl.range : num [1:2] 251 899
..$ start.date : Date[1:1], format: "2021-03-25"
..$ end.date : Date[1:1], format: "2023-03-25"
$ num.pixs : num 2068
$ num.dark.pixs : num 20
- attr(*, "class")= chr [1:2] "instr_desc" "list"
We can in a separate step convert the counts per second data to spectral irradiance by applying the calibration, which in this case is available within sun001_recalc.cps_spct stored as a member of the "intrument.desc" attribute.
In this way we obtain the same irradiance spectrum as before.
summary(sun009_recalc.source_spct)
autoplot(sun009_recalc.source_spct)
We can plot the calibration multipliers. The sensitivity of the pixels is lowest in the UV region, hence the multipliers are larger. The spectral energy is expressed per detector count (for a given cosine diffuser and fibre used). We obtain an irradiance when we multiply counts per second.
A slight variation on Yliantila’s method, using slightly different wavelengths. In this case the spectrum obtained is nearly identical.
Using a different correction method, that assumes that the measurements are of sunlight at ground level and consequently spectral irradiance for wavelengths shorter than 290 nm should be indistinguishable from zero.
As above, the R object returned by s_irrad_corrected() belongs to class "source_spct". As above, the six RAW measurements, three in the light and three in the dark have been used to calculate a single estimate of spectral irradiance.
summary(sun009_recalc_sun.spct)
Summary of source_spct [1,421 x 2] object: sun009_recalc_sun.spct
Wavelength range 251.29-898.97 nm, step 0.43-0.48 nm
Label: light: sun009
Measured on 2022-04-22 09:15:40.313578 UTC
Variables:
w.length: Wavelength [nm]
s.e.irrad: Spectral energy irradiance [W m-2 nm-1]
--
w.length s.e.irrad
Min. :251.3 Min. :-0.001223
1st Qu.:418.4 1st Qu.: 0.586815
Median :582.3 Median : 0.786671
Mean :579.9 Mean : 0.723650
3rd Qu.:742.6 3rd Qu.: 0.972572
Max. :899.0 Max. : 1.154600
By plotting the spectral irradiance data, either in whole or a range of wavelengths we can see that the shape of the spectrum is the same as before but noise in wavelengths < 290 nm has been removed. These data are good so application of either of the two methods gives almost the same resulting spectrum.
autoplot(sun009_recalc_sun.spct)
The zoomed-in plot shows that there is a small amount of noise remaining.
autoplot(sun009_recalc_sun.spct, range =c(250, 315))
2.1.4 No HDR
A negative value as argument to hdr.tolerance disables merging of HDR cps data even if available.
The estimate for UV-A and PAR have not changed, but that of UV-B has increased from 1.82 to 2.16 as well as the noise in the UV-C band from 0.049 to 0.39.
autoplot(sun009_ld.spct)
The zoomed-in plot shows that there is a small amount of noise remaining.
autoplot(sun009_ld.spct, range =c(250, 315))
2.1.6 No filter and no dark reference
Computations are also possible using the UV-C region as an internal dark reference.
The estimate for UV-A and PAR have not changed, but that of UV-B has increased from 1.82 to 2.32 as well as the noise in the UV-C band from 0.049 to 0.53.
autoplot(sun009_l.spct)
The zoomed-in plot shows that there is more noise remaining, including much more variation among neighbouring pixels. This is because a "dark" reference as used in the sections above provides dark noise values for each pixel, while the internal dark reference used in this cases is an average from multiple pixels outside those included in the irradiance spectrum.
autoplot(sun009_l.spct, range =c(250, 315))
3 Step-by-step walk through (out-of-date)
Warning
The examples in this section follow the protocols only approximately (the diagrams above reflect the algorithms as implemented in ‘ooacquire’ 0.4.4). The examples below were written some years back and need to be updated. For these reasons, the computed spectral irradiances may differ slightly among the “canned” and the “step-by-step” computations.An update is in course.
In ‘ooacquire’, some groups of computations are carried out together within a single function and are here mimicked using simplified code in the step by step computation examples. The examples also skip the consistency checks (e.g., spectrometer serial number, column names, and acquisition settings among spectra to be used as reference or combined).
In this section we will apply one by one the different computation steps staring from RAW spectral data until we obtain spectral irradiance, ´to obtain the same spectrum shown in the figure above. Unless mentioned explicitly all the steps in this walk-through apply to all the methods described above.
We have described in the previous section the structure of the object containing the RAW counts data. As we saw above, measurements consisted in two measurements using two different integration times. In addition to different integration times, the number of “scans” was adjusted so that each of the two spectra contain data for a similar total length of time. This averaging takes place before the data are “seen” by R. The three spectra, light, filter and dark where acquired one after another and using the same settings in the spectrometer.
When dealing with raw detector counts one needs to be aware that the array detector in array spectrometers saturates at a certain number of counts, ranging from 256 to 65000 or more counts (e.g., 64000 counts in the Maya 2000 Pro as shown here and fewer simpler/older models like the USB2000). This information is stored as part of the metadata.
If more photons imping on a detector cell/pixel during one integration event (or “scan”) than needed to reach 64000 counts, the reading remains at 64000 counts. This is usually called “signal clipping”, i.e., the tips of peaks are truncated or off scale.
We can find out what integration times have been used from the metadata stored in the same object. In the example spectrum, the shorter integration time used was near optimal, avoiding clipping but still using more the 90% of the pixel counts at the peak. However, the light source has also regions with low emission so we also used a 10 times longer integration time, resulting in clipping of peaks, but making better use of the sensor in the “darker” parts of the spectrum. By changing the number of individual integrations averaged, the total accumulated duration of the measurement was kept at approximately 5 s.
getInstrSettings(sun009.raw_mspct[["light"]])
integ. time (s): 0.15, 1.5
total time (s): 10, 10.5
counts @ peak (% of max): 92.6
Plotting the two bracketed spectra shows the effect of using the two different integration times (counts_1 is the average of 39 integrations each 0.131 s-long, and counts_2 is the average of 4 integration each 1.31 s-long). Note: The first four pixels in each spectrum are “dark” pixels that are not exposed to light, i.e., the counts are smaller than for other pixels and similar for the short and long integration times. These will show in the plots of the raw spectra as a clear dip in the curve. A similar, but unexpected, dip can be seen at the other end in the infrared.
autoplot(sun009.raw_mspct[["light"]])
In the figure above, we can see that in addition to clipping, increasing the integration time, at least in some spectrometers, significantly increases the “noise floor” or dark readings. This is also true for the measurements done in darkness. In these spectra, we can also easily see several spikes caused by “hot pixels” in the detector (over-responding pixels). In the figure below we can see the effect of the filter.
autoplot(sun009.raw_mspct[["filter"]])
In this third figure, we have a look at the spectrum measured in darkness. The Maya 2000 Pro spectrometer has the peculiarity that the effect of warming on dark noise depends on wavelength.
autoplot(sun009.raw_mspct[["dark"]])
The first step in the processing is to substitute data from known bad pixels by values interpolated from the adjacent fixels. Which pixels are bad is recorded in the calibration data. What this step achieves can be clearly seen by comparing the plots before (see above) and after this step (see below). As in most array spectrometers pixel resolution is better than optical resolution this introduces little error, except possibly in the case of very narrow peaks.
for (m innames(sun009.raw_mspct)) { sun009.raw_mspct[[m]] <-skip_bad_pixs(sun009.raw_mspct[[m]])}autoplot(sun009.raw_mspct, facets =1)
Note: When bad-pixel information is not available, automatic despiking can be used instead, but it needs care not to accidentally remove true peaks from the data. (In some types of measurements the spikes appear randomly as they are triggered by cosmic high-energy particles making use of a despiking algorithm a necessity.)
The second step is to replace saturated (clipped) pixel data, with the missing data marker NA. As NA values are not plotted pixels exactly equal to the maximum possible reading disappear from the plots.
for (m innames(sun009.raw_mspct)) { sun009.raw_mspct[[m]] <-trim_counts(sun009.raw_mspct[[m]])}autoplot(sun009.raw_mspct, facets =1)
An important consideration is that when a pixel well fills with electrical charge some of the excess charge migrates to nearby pixels. How many nearby pixels are affected depends on the detector type, but it is at most a few tens of pixels. So in this third step, pixels neighbouring those set to NA in the second step are also set to NA. By default we replace the values from the 10 nearest neighbouring non-saturated pixels also by NA.
for (m innames(sun009.raw_mspct)) { sun009.raw_mspct[[m]] <-bleed_nas(sun009.raw_mspct[[m]])}autoplot(sun009.raw_mspct, facets =1)
The response of array detectors is not perfectly linear to the number of photons received. Most precisely when the number of counts gets near the maximum value, the sensitivity to additional photons slightly decreases. This is a result of how “full” the sensor wells are, irrespective of the source of the charge (on-target light, stray light or thermal energy). Consequently this correction should be applied before subtraction of the dark reading.
So, the fourth step is to apply a linearisation function, in most cases supplied with the instrument and possibly stored in the instrument firmware. This function is part of the calibration data for the instrument that is stored as metadata during acquisition of the spectra. Given that linearization corrects for a decrease in sensitivity at counts approaching the maximum, some values can exceed the maximum instrument counts once corrected (> 64000 in our case).
for (m innames(sun009.raw_mspct)) { sun009.raw_mspct[[m]] <-linearize_counts(sun009.raw_mspct[[m]])}autoplot(sun009.raw_mspct, facets =1)
The first few pixels in the detector of most array spectrometers are covered and never exposed to light photons. These can be used as a dark reference to correct for thermal noise. This step may appear redundant, but might help in cases when instrument temperature is not the same during the light and dark measurements. An alternative is to use pixels known not to be exposed to radiation from the light source, but within the useful measurement range of the instrument. Sunlight at ground level is known to lack UV-C photons, so wavelengths between 250 nm and 290 nm can used as dark reference for this light source. Consequently, as the fifth step we remove an estimate of dark signal based on “non-excited pixels”. As seen in the plots, this step effectively removes the overall dark signal from both light and dark measurements. We can, however, also see that there is residual dark noise remaining at pixel level, not all pixels have the same noise floor. Part of this variation is systematic, and some may be random. As we used a total measurement time of 5 s much of the random noise must have cancelled out through averaging.
for (m innames(sun009.raw_mspct)) { sun009.raw_mspct[[m]] <-fshift(sun009.raw_mspct[[m]], range =c(218.5,228.5))}autoplot(sun009.raw_mspct, facets =1)
In the last plot above we can see that the correction has resulted in slightly negative values in darkness. If the bump in the noise floor, caused by warming, is consistent among all three measurements, it will cancel out.
At this point we have “clean” RAW counts data. The sixth step is to convert these raw counts into counts per second. As the integration time for each spectrum is stored together with the RAW count data, the function call is simple. After this step, the data acquired using different integration times are expressed in the same units of counts-per-second (cps or \(n\,s^{-1}\)). We take advantage of this to plot using different colours the data from the two different “bracketed” integration times. Blue corresponds to the shorter and near-optimal integration time (cps_1 computed starting from counts_1), and red to the longer one (cps_2 computed starting from counts_2). Here we over-plot the two separate spectra.
sun009.cps_mspct <-raw2cps(sun009.raw_mspct)
Warning in range_check(x, cps.cols): Possible off-range cps values
[-1875.39..1000.00]
Warning in range_check(x, cps.cols): Possible off-range cps values
[-1875.39..1000.00]
Warning in range_check(x, cps.cols): Possible off-range cps values
[-1796.70..1000.00]
# autoplot(despike(sun009.cps_mspct, facets = 1)
The seventh step splicing the three pairs of “bracketed” light, filter and dark counts-per-second spectra into three combined spectra.
for (m innames(sun009.cps_mspct)) { sun009.cps_mspct[[m]] <-merge_cps(sun009.cps_mspct[[m]])}
HDR CPS ratio = 1.08; replacing 'cps_2' by 'cps_1' instead of splicing.
Warning in range_check(x, cps.cols): Possible off-range cps values
[-1875.39..1000.00]
autoplot(sun009.cps_mspct)
Warning in range_check(x, cps.cols): Possible off-range cps values
[-1875.39..1000.00]
Warning in range_check(x, cps.cols): Possible off-range cps values
[-1875.39..1000.00]
In the eighth step we subtract the dark signal from the light and filter measurements. This reduces the remaining noise. The filter used has a rather sharp cut-in very close to 400 nm, and blocks UV radiation from entering the spectrometer, so the UV signal remaining after subtracting the dark measurement is the result of light of wavelengths longer than 400 nm “wrongly” striking the UV pixels in the array, i.e., stray light.
In first plot below it looks like there is no stray light present, but zooming-in into the UV-B region we can see that stray light makes a significant contribution to the readings at these short wavelengths.
autoplot(clip_wl(sun009_mdark.cps_mspct, range =c(280, 310))) +ggtitle("Dark subtracted")
To further assess the approximate contribution of stray light to the readings we plot the ratio between these two spectra. Considering that the filter transmits only about 85% of the stray light, we can see that as expected there is only stray light at wavelengths < 293 nm. The methods also estimate absorption of stray light by the filter from the data itself. This estimate also includes and compensates for variation in incident irradiance between light and filter measurements. (However, it does not compensate for changes in the shape of the spectrum of the incident light between light and filter measurements.)
ggplot(clip_wl(sun009_mdark.cps_mspct[["filter"]] / sun009_mdark.cps_mspct[["light"]], range =c(280, 310))) +geom_line() +geom_hline(yintercept =0.85, linetype ="dotted") +ggtitle("Stray light contribution to cps") +labs(y ="Contribution of stray light to readings (/1)",x ="Waavelength (nm)")
Warning in range_check(x, cps.cols): Possible off-range cps values
[-446.75..1000.00]
At this point we have a clean counts-per-second spectrum. However, as we have used math operators on the data, we need to restore the metadata.
The ninth step is to apply the calibration multipliers and any additional instrument specific correction to obtain spectral irradiance stored in a "source_spct" object.
We first demonstrate how the light measurement with no stray light correction and the filter measurement look like when plotted. This light spectrum is equivalent to using method none. The ratio between the UV-C reading and PAR is close to \(3 \times 10^{-4}\) in this case. For sunlight this ratio is usally near \(1 \times 10^{-3}\).
In an optional, step we apply smoothing to remove some of the noise from the light spectrum, but although the UV-C and UV-B regions look cleaner the value of the summaries remain unchanged and affected by stray light.
One key aspect of the more sophisticated methods, is the stray light correction. We here replot the spectrum obtained using method ylianttila or original. We can see that UV-C is less than \(3 \times 10^{-5}\) compared to PAR, and the estimate of UV-B has decreased by 13%. The estimates for UV-A and PAR irradiances remain unchanged.
autoplot(sun009_recalc.spct)
We can also apply smoothing to this spectrum, which in this case improves even further the UV-C estimate, which is now less than \(5 \times 10^{-6}\) which is an exceptionally good performance, which is rarely achieved for this method, with a usual ratio close to \(1 \times 10^{-4}\). So, by using Ylianttila’s method we have improved the signal to noise ratio by more than an order of magnitude.
autoplot(smooth_spct(sun009_recalc.spct))
Before continuing with a different example we explain how the stray light correction is done in method Ylianttila or original. The exact wavelengths used in the algorithm are tuned for each individual spectrometer and are part of the method.
str(MAYP11278_ylianttila.mthd)
List of 10
$ spectrometer.sn : chr "MAYP11278"
$ stray.light.method: chr "original"
$ stray.light.wl : num [1:2] 218 228
$ flt.dark.wl : num [1:2] 193 210
$ flt.ref.wl : num [1:2] 360 380
$ flt.Tfr : num 1
$ inst.dark.pixs : int [1:3] 2 3 4
$ tail.coeffs : num [1:2] -7.2731 -0.0569
$ worker.fun : chr "MAYP11278_tail_correction"
$ trim : num 0
We have three ranges of wavelengths. For sunlight we can assume that pixels for wavelengths in flt.ref.wl have received enough photons to be very little affected by stray light. flt.dark.wl is the range of wavelengths where we assume the filter is fully opaque. stray.light.wl is a wavelength range where we can safely assume (for sunlight) that both light and filter measurement are just pure stray light; thus, these pixels can be used to rescale the filter measurement to compensate for filter transmittance less than one for stray light or changes in stray light caused by a change in incident irardiance. This assumes thay stray light is scattered rather than due to specular reflections within the spectrometer.
We show, for completeness, a dark spectrum that does not pass the quality check, compared with the one we have been using above, both with the scale limits adjusted to visualize the problem. This “bad” spectrum shows increased dark noise only in shorter wavelengths. This problem seems to be caused by warming of the spectrometer when not sufficiently protected from sunlight.
Obviosuly, this filter-based correction, as implemented in correction methods original are applicable only to measurements of sunlight or of spectra very similar in shape to sunlight, like natural shade light.
With a LED source that does not emit in the IR, at least in the case of our Maya 2000 Pro, we do not need to worry about stray light. However, when we measure light sources with a significant emission in the infra-red above 900 nm, stray light becomes a problem. In addition sources with narrow peaks of emission may benefit from deconvolution or correction based on a measurement of the slit function.
3.0.1 Tungsten halogen lamp
For our instrument, stray light originates mostly in the infra-red and it can be a serious problem when trying to precisely measuring sunlight or incandescent lamps. For this example we use data acquired using a different protocol, which includes three measurements, being the additional one, a measurement through a filter used to estimate stray light in the UV-region. In this case a UV-cut and IR-pass filter (a piece of clear polycarbonate).
The data contains a measurement of the light source directly and through a filter and a reference dark measurement.
names(halogen.raw_mspct)
[1] "light" "filter" "dark"
The spectrum for the light measurement contains two columns with RAW-counts data.
names(halogen.raw_mspct[["light"]])
[1] "w.length" "counts_1" "counts_2"
Summary in addition to displaying the summary for the columns, displays the most important metadata attributes.
summary(halogen.raw_mspct[["light"]])
Summary of raw_spct [2,068 x 3] object: anonymous
Wavelength range 187.82-1117.14 nm, step 0.41-0.48 nm
Label: Halogen
Measured on 2017-03-28 11:24:35.664327 UTC
Data acquired with 'MayaPro2000' s.n. MAYP11278
grating 'HC1', slit '010s'
diffuser 'cosine'
integ. time (s): 1.86, 7.2
total time (s): 5.58, 7.2
counts @ peak (% of max): 94.4Variables:
w.length: Wavelength [nm]
counts_1: Raw detector counts [number]
counts_2: Raw detector counts [number]
--
w.length counts_1 counts_2
Min. : 187.8 Min. : 2121 Min. : 1950
1st Qu.: 431.7 1st Qu.: 5316 1st Qu.:14164
Median : 668.7 Median :22588 Median :64000
Mean : 663.3 Mean :25981 Mean :44225
3rd Qu.: 897.6 3rd Qu.:44996 3rd Qu.:64000
Max. :1117.1 Max. :60728 Max. :64000
As above for the LED lamp, we first calculate spectral irradiance from a set of raw-counts spectral data using the high-level function s_irrad_corrected().
autoplot(smooth_spct(halogen.spct), range =c(250, 500))
369 possibly 'bad' values in smoothed spectral response
In the example above we called the functions used for each of the steps in the computation individually, saving the intermediate results so as to be able to show the partly processed data at each step. The functions, however, support the use of pipes as they all have as their first parameter the one accepting the R object returned by the previous stage.
We first use a “pipe” to apply the same initial processing steps as for the LED bulb data in the previous section. The main difference is that we use as internal instrument dark reference those pixels that never are exposed to radiation. For our instrument they correspond to wavelengths 187.82 nm to 189.26 nm. We obtain this time three spectra containing counts-per-second data.
We plot the returned spectra, both in full, and the UV region by itself. Please, be aware of the difference in the y scale among the plots. By careful comparison of these later plots one can see that the signal for filter is larger than for dark. As we know from specifications and measurements that the filter used blocks radiation in this region, the difference is due to stray light (radiation of longer wavelengths being detected as ultraviolet).
for (m innames(halogen.cps_mspct)) {print(autoplot(halogen.cps_mspct[[m]]) +ggtitle(m))print(autoplot(halogen.cps_mspct[[m]], range =c(250, 410)) +ggtitle(m))}
In the next step we subtract the dark reading from both the light and filter readings, and copy attributes, including instrument settings and calibration data.
The second and fourth plots display in detail the UV region. Be aware of the difference in the y scale in the plots
for (m innames(halogen01.cps_mspct)) {print(autoplot(halogen01.cps_mspct[[m]]) +ggtitle(m))print(autoplot(halogen01.cps_mspct[[m]], range =c(250, 410)) +ggtitle(m))}
As we can see in the plots above, a small amount of stray light is present in both spectra in the UV region. We apply a filter correction using a simple method based on a fixed transmittance value. We set flt.Tfr = 0.9 as these is a good estimate of the transmittance of polycarbonate to the stray light.
The second plot displays in detail the UV region. Be aware of the difference in the y scale in the plots
autoplot(halogen_corrected.cps_spct)
autoplot(halogen_corrected.cps_spct, range =c(250, 410))
mean(clip_wl(halogen_corrected.cps_spct, range =c(250, 300))[["cps"]])
[1] 2.911621
The average counts-per-second remaining after correcting for stray light is very small. In the next step we apply the calibration multipliers to obtain spectral irradiance.
The spectrum displays some noise at the shortest wavelengths and some interference patterns at the long end. The interference patterns come from the light bulb, but the random noise of increasing amplitude with decreasing wavelength in the UV region is due to the spectrometer.
We can compute some photon ratios, expressed as \(mmol\, mol^{-1}\) to diagnose whether the stray light is well controlled.
With overriding the default arguments we better remove random noise and a small “bump” at 320 nm. Setting setting strength to 1 instead of 3 smooths the random noise but not this small peak (not shown). In the infra-red most of the wavy pattern is also removed. So, smoothing can be useful, but it can also remove real features, and one needs to decide if these features are of interest or not, based on other sources of information.
The second plot displays in detail the UV region. Be aware of the difference in the y scale in the plots: the ratio between the irradiance at 280 nm and at 900 nm is 1:3500.
autoplot(halogen_sm.source_spct)
autoplot(halogen_sm.source_spct, range =c(250, 410))
When measuring spectra containing narrow peaks or steep slopes the shape of the slit function will affect the apparent width of peaks or the apparent steepness of the slope. The slit through which light enters the instrument has a finite width and angle of acceptance for radiation. Consequently, even when measuring true monochromatic radiation from a laser photons will imping on multiple array pixels/wells. The pixel corresponding to the wavelength of the radiation receives the most photons, but the neighbouring pixels receive a decreasing number of photons as the distance from the “correct” target pixel increases. The wider the slit used, the broader the “false” peak observed for monochromatic light. The shape and width of this slit function depends not only on the width of the slit but also on other features of the optical bench of the instrument. In array spectrometers the slit function also depends on wavelength as the length of the path from the grating to the detector is not constant.
However, if the slit function is known, it can be used to remove its influence from a measured spectrum, in simpler words it can be used to partly reconstruct the structure of original light source spectrum. In the case of the two examples above, applying this correction would make little difference. In the case of the solar spectrum and discharge lamps this further step improves the estimates for spectral irradiance.
Too see the effect of this correction we need to look at individual peaks in a spectrum. We use data for a mercury lamp.
The data contains a measurement of the light source and a reference dark measurement.
names(xenon_flash.raw_mspct)
[1] "light" "dark"
The spectrum for the light measurement contains one column with RAW-counts data as no bracketing was used.
names(xenon_flash.raw_mspct[["light"]])
[1] "w.length" "counts"
Summary in addition to displaying the summary for the columns, displays the most important metadata attributes.
summary(xenon_flash.raw_mspct[["light"]])
Summary of raw_spct [2,068 x 2] object: anonymous
Wavelength range 187.82-1117.14 nm, step 0.41-0.48 nm
Label: Bare bulb xenon flash
Measured on 2018-09-13 14:34:06.442208 UTC
Data acquired with 'MayaPro2000' s.n. MAYP11278
grating 'HC1', slit '010s'
diffuser 'unknown'
integ. time (s): 5.08
total time (s): 5.08
counts @ peak (% of max): 91.9Variables:
w.length: Wavelength [nm]
counts: Raw detector counts [number]
--
w.length counts
Min. : 187.8 Min. : 2096
1st Qu.: 431.7 1st Qu.: 8150
Median : 668.7 Median :13992
Mean : 663.3 Mean :13795
3rd Qu.: 897.6 3rd Qu.:17637
Max. :1117.1 Max. :57992
In the case of these data, the concept of counts-per-second does not apply as the flash discharge is shorter than the integration time, and unknown. The relevant reference is one exposure event and the quantity to estimate is spectral fluence in \(J m^{-2}\).
As above for the LED lamp, we first calculate spectral fluence from a set of raw-counts spectral data using the high-level function s_irrad_corrected().
Ylianttila L, Visuri R, Huurto L, Jokela K. 2005. Evaluation of a single-monochromator diode array spectroradiometer for sunbed UV-radiation measurements. Photochemistry and Photobiology81: 333–341.
