|
|
||||||||
International Centre for Research in Agroforestry (ICRAF), P.O. Box 30677, Nairobi, Kenya
* Corresponding author (k.shepherd{at}cgiar.org)
| ABSTRACT |
|---|
|
|
|---|
Abbreviations: ECEC, effective cation-exchange capacity ECECclay, ECEC divided by clay fraction MARS, multivariate adaptive regression splines RMSE, root mean square error
| INTRODUCTION |
|---|
|
|
|---|
Diffuse reflectance spectroscopy is now routinely used for the rapid nondestructive characterization of a wide range of materials (Davies and Giangiacomo, 2000). Spectral signatures of materials are defined by their reflectance or absorbance, as a function of wavelength in the electromagnetic spectrum. Under controlled conditions, the signatures result from electronic transitions of atoms and vibrational stretching and bending of structural groups of atoms that form molecules or crystals. Fundamental features in reflectance spectra occur at energy levels that allow molecules to rise to higher vibrational states. For example, the fundamental features related to various components of soil organic matter generally occur in the mid- to thermal-infrared range (2.525 µm), but their overtones (at one half, one third, one fourth etc. of the wavelength of the fundamental feature) occur in the near-infrared (0.71.0 µm) and short-wave infrared (1.02.5 µm) regions. Soil clay minerals have very distinct spectral signatures in the short-wave infrared region because of strong absorption of the overtones of SO2-4, CO2-3, and OH- and combinations of fundamental features of, for example, H2O and CO2 (Hunt, 1982; Clark, 1999). The visible (0.40.7 µm) region has been widely used for color determinations in soil and geological applications as well as in the identification of Fe oxides and hydroxides (Ben-Dor et al., 1999). Since the mid-1980s, developments in instrument technology and chemometrics (the application of mathematical and statistical techniques to chemical data) have led to the increased use of spectroscopy in the laboratory and field and from space platforms, notably in geological studies (Clark, 1999).
Recent research has demonstrated the ability of reflectance spectroscopy to provide nondestructive rapid prediction of soil physical, chemical, and biological properties in the laboratory (Ben-Dor and Banin, 1995; Janik et al., 1998; Reeves et al., 1999). There has been some success with reflectance spectroscopy for sensing of soil organic matter in the field (Sudduth and Hummel, 1993), and for the discrimination of major soil types from satellite multi-spectral and aircraft hyperspectral data (Baumgardner et al., 1985; Coleman et al., 1993; Palacios-Orueta et al., 1999). Despite these indications of the potential of the technique, there are few examples of the application of reflectance spectroscopy for nondestructive assessment of soils (Janik et al., 1998; Myer, 1998). Although geological spectral libraries exist that include soil mineral spectra (e.g., Clark, 1999), there are few examples of soil spectral libraries that include a wide diversity of soils with information on physical, chemical, and biological properties (Ben-Dor et al., 1999; Malley et al., 2000; Chang, 2001). In particular there has been little focus on the development of soil spectral libraries for application to risk-based approaches to soil evaluation that explicitly consider uncertainty in predictions and interpretations of soil properties.
We propose a scheme for the use of spectral libraries as a tool for building risk-based approaches to soil evaluation (Fig. 1) . The ability to rapidly and nondestructively characterize soils using reflectance spectroscopy permits thorough sampling of the variation within a target population of soils (Stenberg et al., 1995). Soil properties or attributes of soil functional capacity are measured for only a selection of soils, designed to sample the variation in the spectral library, and then calibrated to soil reflectance. If, on the basis of cross-validation or holdout-validation methods, calibrations are found to be insufficiently accurate for user requirements, the calibration sample size can be increased. The resultant calibrations between soil functional attributes and soil reflectance are then used to predict the soil functional attributes for the entire soil library and for new samples that belong to the same population as the library soils. Poorly described soils, whose spectra are not representative of the library spectra, are further characterized and added to the calibration library.
|
| MATERIALS AND METHODS |
|---|
|
|
|---|
The soils were air-dried, passed through a 2-mm sieve, and stored in paper bags at room temperature. They were analyzed using standard methods widely used for tropical soils. Soil pH was determined in water using a 1:2.5 soil/solution ratio. Samples were extracted with 1 M KCl using a 1:10 soil/solution ratio, and analyzed by NaOH titration for exchangeable acidity and by atomic absorption spectrometry for exchangeable Ca and Mg, and exchangeable Na by flame photometry (ISFEIP, 1972; Yurimaguas Experiment Station Staff, 1989). Samples with pH
5.5 were assumed to have zero exchangeable acidity and samples with pH <7.5, zero exchangeable Na. Samples were extracted with 0.5 M NaHCO3 + 0.01 M EDTA (pH 8.5, modified Olsen) using a 1:10 soil/solution ratio and analyzed by flame photometer for exchangeable K and colorimetrically (molybdenum blue) for extractable P (ISFEIP, 1972; Yurimaguas Experiment Station Staff, 1989). Organic C was determined colorimetrically after H2SO4dichromate oxidation at 150°C (Heanes, 1984). Nitrogen mineralization potential was determined by determination of ammonium production with 7-d anaerobic incubations at 40°C (Keeney, 1982). Particle-size distribution was determined using the hydrometer method after pretreatment with H2O2 to remove organic matter (Gee and Bauder, 1986). Effective cation-exchange capacity was calculated as the sum of exchangeable acidity and exchangeable bases, and ECECclay was calculated as ECEC divided by the clay fraction.
Reflectance Measurements
Soil diffuse reflectance spectra were recorded for each library sample using a FieldSpec FR spectroradiometer (Analytical Spectral Devices Inc., Boulder, Colorado) at wavelengths from 0.35 to 2.5 µm with a spectral sampling interval of 1 nm. The optical setup was as recommended by the instrument manufacturers (Analytical Spectral Devices Inc., 1997), commonly used in geological applications. Samples were illuminated from above (Fig. 2)
with two tungsten quartz halogen filament lamps in housings with aluminum reflectors (Lowel pro-lamp, Lowel-Light Manufacturer Inc., New York, NY) with 50W bulb;
3200 K color temperature (WelchAllyn, Skaneateles Falls, NY) . The lamps were placed each side of the sample, with the light beam 30° from vertical, to give a distance of 50 cm between the lamps and the sample. Reflected light was collected with a 25° field-of-view foreoptic angled at a 30° from vertical and perpendicular to the plane of illumination at a distance of 5 cm from the sample (Fig. 2).
|
1% among rotations within a sample dish, and 2% for replicate dishes from a soil sample. With this method, a single operator can comfortably scan several hundred samples a day.
Statistical Methods
Multivariate relationships among soil properties were analyzed to establish to what degree good spectral calibrations could have resulted from interdependencies among soil variables. Conditional independence assumptions among soil properties were tested using graphical linear modeling approaches (Edwards, 2000). Graphical modeling is a form of multivariate analysis that uses graphs to represent models. The graphs display the structure of both associational and causal dependencies among the variables in the model, allowing the conditional as well as the marginal associations to be studied. By considering the conditional associations among variables the approach helps to identify spurious associations that can occur when studying only marginal pairwise associations among variables. Where necessary, Box-Cox (Box and Cox, 1964) transformations were applied prior to analysis to obtain approximately multivariate normally distributed values. A backwards selection procedure was then applied to conditional independence assumptions among soil variables. This iteratively tests all the marginal associations among variables in the model and deletes those associations that do not significantly contribute towards the fit of the fully saturated base model, using maximum likelihood estimation.
The raw spectral reflectance data was preprocessed prior to statistical analysis as follows. Relative reflectance spectra were resampled by selecting every tenth-nanometer value from 0.35 to 2.5 µm. This was done to reduce the volume of data for analysis and to match it more closely to the spectral resolution of the instrument (3 to 10 nm). The reflectance values were then transformed with first derivative processing (differentiation with second-order polynomial smoothing with a window width of 20 nm) using a Savitzky-Golay filter, as described by Fearn (2000). Derivative transformation is known to minimize variation among samples caused by variation in grinding and optical set-up (Marten and Naes, 1989). Multiplicative scatter correction (used to compensate for additive and multiplicative effects in spectral data) and normalization (sample-wise scaling) of the reflectance data (both described in Vandeginste et al., 1998) did not improve calibrations and were not used. Wavebands in regions of low signal to noise ratio or displaying noise because of splicing between the individual spectrometers (Analytical Spectral Devices Inc., 1997) were omitted leaving 198 wavebands for analysis. The omitted bands were 0.35 through 0.38 µm, 0.97 through 1.01 µm, and 2.46 through 2.50 µm.
Variation in overall spectral shape among samples was explored by displaying spectra identified by a central-composite sampling design based on Euclidean distances from the center sample in principal component space (Massart et al., 1997; CAMO Inc., 1998). The first three principal components were used as the design factors with equal weighting given to each component. The spectra situated closest to the center, cube (distance of one standard deviation from center), and star (distance of 1.98 standard deviations from center) points of the design were selected. Principal components analysis was conducted with the Unscrambler version 7.5 (CAMO Inc., 1998). Spectra were also plotted continuum-removed, using ENVI (Research Systems Inc., 1999), to help detect subtle differences in spectral absorption features among soils. Continuum removal is used to normalize reflectance spectra so that individual absorption features can be compared from a common baseline. The continuum is a convex hull, consisting of straight-line segments fitted over local spectral maxima (Research Systems Inc., 1999). The patterns of correlation between individual soil variables and derivative reflectance at each waveband were also explored.
Individual soil variables were then calibrated against the 198-reflectance wavebands using MARS (MARS version 2.0, Salford Systems Inc., San Diego, CA). Multivariate adaptive regression splines is a new approach to regression modeling developed for data mining applications (Friedman, 1991; Steinberg et al., 2001). Data mining approaches are appropriate for large multivariate data sets when there is little theoretical knowledge available to guide the model-building process. Multivariate adaptive regression splines is a nonlinear multiple regression technique that builds flexible models by fitting piecewise linear regressions. When a target variable is regressed against a predictor variable, the slope of the regression line is allowed to change at certain points (termed knots) along the predictor axis. The variables and knot positions used are found via an intensive search procedure. Each such relationship, which may include interaction terms, is represented as a basis function (Steinberg et al., 2001). In fact, in our analyses no interactions between dependent variables were allowed. Multivariate adaptive regression splines first constructs an overly large model by adding basis functions, which are then deleted in order of least contribution to the model until an optimal model is found. The number of degrees of freedom charged for knot optimization was determined using ten-fold cross-validation. The maximum number of basis functions was varied to provide the best model in terms of lowest generalized cross-validation measure. In ten-fold cross-validation the calibration data is divided into ten roughly equal parts, each containing a similar distribution for the dependent variable. Nine parts of the data are used to develop a calibration model, which is then tested on the remaining one tenth of the data. This process is repeated until each part of the data has been withheld. The results of the ten tests are then combined to provide error rates for the calibration model (Massart et al., 1997). Generalized cross-validation is an approximate version of cross-validation that is less computationally demanding (Friedman, 1991); it is the average-squared residual of the fit to the data times a penalty to account for the increased variance associated with increasing model complexity (i.e,. number of basis functions). After experimenting with several alternative calibration methods, MARS was found to give the best average prediction performance on holdout validation samples. These alternative methods included partial least squares regression; classification and regression trees (CART; Brieman et al., 1984; Steinberg and Cola, 1997), including the use of bootstrap aggregation and adaptive resampling and combining (i.e., averaging of a large number of trees generated by resampling and replacement from the original training data); hybrid models, with separate partial least squares models combined from subsets of data identified using regression trees; and schemes for spectral matching.
For each soil variable, calibration models were developed on a random sample of two-thirds of the soil samples of the entire library. The same random selection pattern was used for all the calibrations but the sample selection varied according to the number of samples available for each soil property. The calibrations were tested by predicting the soil variables on validation data sets composed of the remaining one-third of the samples. The calibrations were developed on the transformed soil variables, but the calibration and validation results were back-transformed for evaluation of predictive performance. No samples were omitted from the analysis in either the calibration or validation data sets. Prediction success was evaluated on predicted and actual observations using the coefficient of determination (r2), root mean square error (RMSE) and bias. Root mean square error and bias were also calculated separately for each quartile of the predicted variable.
To test predictive performance for given threshold values of selected soil variables, a number of soil fertility screening tests were defined based on critical limits commonly reported in the literature (Cochrane et al., 1985; Landon, 1991). Samples were classified either as abnormal or normal based on a cut-off value defined by the critical limit (Table 1). Classification trees were used to develop calibrations for each soil test using CART version 4.0 (Steinberg and Cola, 1997) with the 198-reflectance wavebands as dependent variables. A classification tree is built from decision rules that repeatedly split the data set into increasingly homogeneous subsets. The decision rules use the dependent variables to give the best separation of classes in the predictor variable (here, normal and abnormal cases) in terms of greatest reduction in variance. Output from the model fitting procedure is a decision tree. The splitting-rule used in these analyses was the Gini index of diversity (a measure of node impurity, described by Brieman et al., 1984). The classification trees were grown using a randomly selected calibration data set consisting of two-thirds of the samples, using ten-fold cross-validation (Brieman et al., 1984). The predictive ability of the resulting models was then further tested using the remaining one-third of the samples withheld for validation.
|
A procedure for screening new samples for outlier detection was developed. A principal components model is fitted to derivative reflectance spectra for the existing library samples. Spectral outliers in new samples are identified with respect to this model using a classification method (significance = 5%) known as soft independent modeling of class analogy. This method tests whether new samples are members of the existing library class or not based on measures of object-to-model distance and leverage (CAMO Inc., 1998; Vandeginste et al., 1998).
| RESULTS AND DISCUSSION |
|---|
|
|
|---|
|
|
|
Prediction of Soil Properties using Multivariate Adaptive Regression Splines
Good calibrations (r2 > 0.75) were obtained for soil pH, ECEC, exchangeable Ca, exchangeable Mg, organic C, and particle-size distribution (Table 4). The calibration models with the largest r2 values for generalized cross-validation for a given attribute also resulted in the largest validation r2 values, indicating that the generalized ten-fold cross-validation was effective in safeguarding against over-fitting. The level of prediction accuracy achieved on the validation data sets (Fig. 4)
is sufficiently high for studies in which spatial or temporal variability of an attribute is large relative to the accuracy of its measurement, as typically found in large-area applications and farm advisory work. Root mean squared error was larger at high than low values for ECEC, exchangeable Ca, exchangeable Mg, sand, and organic C. Bias was also larger at high than low values of organic C; it increased from -0.3 g kg-1 for predicted values below 24 g kg-1 to 2.7 for values above 24 g kg-1. The poorer predictive performance at high values for these variables may be because of error in the laboratory analytical methods rather than genuine lack of prediction power. Increased analytical error could be expected at higher concentrations because of greater variability in amounts of ion extracted and the need for increasing number of dilutions. This hypothesis was supported by trends in the available data on variability in duplicate laboratory determinations. For example, the RMSE for ECEC laboratory duplicates increased from 0.4 at <15 cmolc kg-1 to 1.1 at >15 cmolc kg-1. The RMSE for organic C laboratory duplicates increased dramatically from 0.9 at <20 g kg-1 to 7.4 at >0.2 g kg-1. The H2SO4dichromate oxidation method also underestimates organic C in these soils at values of >0.2 g kg-1 compared with the dry combustion method (A. Albrecht, personal communication, 2000). Other sources of analytical error can be expected because of (i) variation in analytical technique among batches analyzed over several years, (ii) changes in soil properties between the times of analytical and spectral measurement, and (iii) variation among subsamples used for analytical and spectral measurements.
|
|
Because exchangeable Ca and Mg displayed high partial correlation, the conditional independence assumptions between actual and predicted values for these two variables were explored further with graphical modeling, using a coherent backwards selection procedure. It was established that both the relationship between actual and predicted exchangeable Ca and that between actual and predicted exchangeable Mg displayed conditional dependence (P = 0.05). Thus the spectral test provided more information about exchangeable Mg than was provided by the actual and predicted exchangeable Ca values.
Calibration models established for exchangeable K, extractable P, and N mineralization potential were not stable (validation r2 < 0.5). Janik et al. (1998) also observed poor prediction of bicarbonate-extractable K and P with mid-infrared analysis. Chang et al., (2001) reported that ability to predict levels of extractable cations varied with the extraction method, but the reasons for the differences were not clear. Because soil supply of nutrients to plants depends on many interrelated soil factors, further work should investigate whether plant response to N, P, and K can be better predicted from soil reflectance than from soil extractions.
Prediction of Soil Tests using Classification Trees
For many agricultural and engineering applications, such as soil fertility evaluation, it is often sufficient to classify a soil with respect to a critical test value, rather than needing a precise estimate of a soil property. Using a one-third holdout sample for validation, reasonable predictive performance was achieved for all the soil screening tests (Table 5) with positive likelihood ratios ranging from 2.7 to 11.4. Although exchangeable K and extractable P were predicted moderately poorly from soil reflectance using MARS calibrations, the relationships were still strong enough to permit reasonable discrimination of soils falling above or below specific cut-off values. There are few comparable data from screening tests in the soil science literature (Dewayne Mays, 1996), but for comparison purposes, the values reported here fall within the range of likelihood ratios commonly published for screening tests in the medical literature (Jones and Payne, 1997).
|
Although using full spectrum data always resulted in the best predictions, there were often good surrogate node splitters in different parts of the spectrum. This indicates possible flexibility in the choice of wavelength ranges for calibrations. For example, likelihood ratios for the low ECECclay test for models using full spectrum were 8.3 (95% confidence interval, 5.113.3); visible wavelength ranges, 4.7 (3.26.8); and short wave infrared, 7.7 (5.011.9). For low extractable P, the corresponding likelihood ratios were 2.9 (2.23.8), 2.4 (1.83.1), and 2.2 (1.82.8), respectively. These results indicate opportunity for use of simplified spectrometer designs for specific screening tests, e.g., use of visible and near infrared, handheld spectrometers that are commercially available.
Because soil reflectance provides an integrated measure of several fundamental soil properties, including surface charge characteristics, particle-size distribution, and organic C, soil functional attributes could be predicted better directly from soil reflectance than indirectly from laboratory soil tests. For example, Lins and Cox (1989) found that prediction of optimum P fertilizer rate from extractable P was greatly improved when clay content or surface area was considered. Our results demonstrate moderate ability of reflectance measurements to discriminate soils with low extractable P, as well as good prediction of clay and surface charge characteristics.
Response to Calibration Sample Size
The response of predictive performance of the MARS regressions to variation in calibration sample size was investigated for three key soil properties (Fig. 5)
. Predictive performance decreased gradually with decreasing sample size at large sample sizes, but rapidly decreased as sample size decreased below about 100 to 200 samples. Prediction performance of ECEC was less sensitive to sample size than clay or organic C. We suggest that initial investments into building reasonably large calibration libraries (several hundred soils) are worthwhile to allow such responses to be investigated. Once calibration sample size is large enough to provide stable results, then only calibration maintenance will be required to include library outliers among new samples (Fig. 1).
|
1.5. If abnormal cases were at a population prevalence of 50%, the percentage of cases that would be correctly diagnosed using the 10% calibration samples would range from 60 to 87% for the different tests in Table 5. Our results indicate that in some cases, very small calibration sample sizes may provide adequate predictive performance. For example, when a calibration tree for high ECECclay was built on a random sample of only 34 soils and used to predict abnormal cases for the remaining 647 library soils, using the prevalence rate of 27% abnormal cases that occurred in the library, a predictive efficiency of 86% was obtained, with a positive predictive value of 71% and negative predictive value of 92%. On the other hand, where diagnostic performance is less than desired, combining additional screening tests based on other information, such as land use, topography, or satellite imagery, could be a preferable strategy to that of increasing the calibration sample size. Calibrations could also be improved by restricting geographical extent (e.g., Sudduth and Hummel, 1996), but global models may be more robust than local models in terms of ability to predict new samples.
Library Outlier Detection
To test the outlier screening procedure (Fig. 1), the ECEC values at pH <7.0 for southern African soils (n = 274) were taken as the existing spectral library calibration data set, and the corresponding values for eastern Africa (n = 697) were taken as new samples to be predicted. A principal components model was fitted to the southern Africa spectra and then outliers in the eastern Africa spectra identified with respect to this model using soft independent modeling of class analogy. The outlying spectra (n = 53) were then added to the southern Africa calibration data set (Table 6). In a second test, an additional 86 randomly selected eastern African soils were added to the calibration data set, giving a total of 20% of the eastern African soils included in the calibration. These strategies were compared with random sampling of the same number of soils (Table 6). Multivariate adaptive regression splines models were fitted to the three calibration data sets.
|
| CONCLUSIONS |
|---|
|
|
|---|
A spectral library approach provides a tool for generalizing results of soil assessments that are conducted at a limited number of sites, and thereby increases the efficiency of expensive and time-consuming soil-related studies. The rapid nature of the measurement allows soil variability to be more adequately sampled than with conventional approaches and thereby facilitates risk-based approaches to soil assessments. For example, knowledge of the uncertainty in prediction of soil functional attributes, taking into account soil variability, allows users to make informed decisions about the trade off between the cost of the measurement and the risk (or potential for regret) associated with using the prediction.
Further investigations should test reflectance spectroscopy for direct prediction of a wide range of soil functional attributes for agricultural, environmental, and engineering applications, both in the laboratory and field, and develop operational schemes for its use in risk-based soil assessments. Because soil reflectance provides an integrated measure of number of fundamental soil properties, such calibrations could perform better, and would certainly be more rapid, than pedotransfer functions based on conventional measurements of soil properties. Soil functional attributes that are often predicted from basic soil properties tested in this study include net primary productivity, plant growth response to soil constraints and ameliorants, soil erodibility, soil compressibility and shrinkage, water retention and conductivity, and capacity to adsorb wastes and pollutants.
The spectral library approach provides a coherent framework for linking soil information with remote sensing information for improved spatial prediction of soil functional capacity. Remote sensing of soil properties directly from space platforms is hampered by problems such as atmospheric interference, shade and shadow effects, mixtures of materials within pixels, and variation in soil moisture content. Studies on the effect of soil moisture content on calibrations between soil functional attributes and soil reflectance would help to evaluate the potential of reflectance spectroscopy in the field. Future studies should explore approaches that combine soil spectral libraries, and other geo-referenced information, such as from digital terrain models and field observations, with information from multi- and hyper-spectral remote sensing imagery (e.g., Shepherd and Walsh, 2000).
| ACKNOWLEDGMENTS |
|---|
Received for publication May 30, 2001.
| REFERENCES |
|---|
|
|
|---|
This article has been cited by other articles:
![]() |
L. Wielopolski, G. Hendrey, K. H. Johnsen, S. Mitra, S. A. Prior, H. H. Rogers, and H. A. Torbert Nondestructive System for Analyzing Carbon in the Soil Soil Sci. Soc. Am. J., September 1, 2008; 72(5): 1269 - 1277. [Abstract] [Full Text] [PDF] |
||||
![]() |
Y. Wu, J. Chen, J. Ji, P. Gong, Q. Liao, Q. Tian, and H. Ma A Mechanism Study of Reflectance Spectroscopy for Investigating Heavy Metals in Soils Soil Sci. Soc. Am. J., May 16, 2007; 71(3): 918 - 926. [Abstract] [Full Text] [PDF] |
||||
![]() |
T. H. Waiser, C. L. S. Morgan, D. J. Brown, and C. T. Hallmark In Situ Characterization of Soil Clay Content with Visible Near-Infrared Diffuse Reflectance Spectroscopy Soil Sci. Soc. Am. J., March 12, 2007; 71(2): 389 - 396. [Abstract] [Full Text] [PDF] |
||||
![]() |
M. R. Nanni and J. A. M. Dematte Spectral Reflectance Methodology in Comparison to Traditional Soil Analysis Soil Sci. Soc. Am. J., February 2, 2006; 70(2): 393 - 407. [Abstract] [Full Text] [PDF] |
||||
![]() |
M. J. Cohen, J. P. Prenger, and W. F. DeBusk Visible-Near Infrared Reflectance Spectroscopy for Rapid, Nondestructive Assessment of Wetland Soil Quality J. Environ. Qual., July 5, 2005; 34(4): 1422 - 1434. [Abstract] [Full Text] [PDF] |
||||
![]() |
K. D. Shepherd, C. A. Palm, C. N. Gachengo, and B. Vanlauwe Rapid Characterization of Organic Resource Quality for Soil and Livestock Management in Tropical Agroecosystems Using Near-Infrared Spectroscopy Agron. J., September 1, 2003; 95(5): 1314 - 1322. [Abstract] [Full Text] [PDF] |
||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||