ABSTRACT
Computer vision is becoming a ubiquitous technology in many industries on account of its speed, accuracy, and long-term cost efficacy. The ability of a computer vision system to quickly and efficiently make quality decisions has made computer vision a popular technology on inspection lines. However, few companies in the agriculture industry use computer vision because of the non-uniformity of sellable produce.
The small number of agriculture companies that do utilize computer vision use it to extract features for size sorting or for a binary grading system: if the piece of fruit has a certain color, certain shape, and certain size, then it passes and is sold. If any of the above criteria are not met, then the fruit is discarded. This is a highly wasteful and relatively subjective process.
This thesis proposes a process to undergo to use computer vision techniques to extract features of fruit and build a model to predict shelf-life based on the extracted features. Fundamentally, the existing agricultural processes that do use computer vision base their distribution decisions on current produce characteristics. The process proposed in this thesis uses current characteristics to predict future characteristics, which leads to more informed distribution decisions.
By modeling future characteristics, the process proposed will allow fruit characterized as “unfit to sell” by existing standards to still be utilized (i.e. if the fruit is too ripe to ship across the country, it can still be sold locally) which decreases food waste and increases profit. The process described also removes the subjectivity present in current fruit grading systems. Further, better informed distribution decisions will save money in storage costs and excess inventory.
The proposed process consists of discrete steps to follow. The first step is to choose a fruit of interest to model. Then, the first of two experiments is performed. Sugar content of a large sample of fruit are destructively measured (using a refractometer) to correlate sugar content to a color range. This step is necessary to determine the end-point of data collection because stages of ripeness are fundamentally subjective. The literature is consulted to determine “ripe” sugar content of the fruit and the first experiment is undertaken to correlate a color range that corresponds to the “ripe” sugar content.
This feature range serves as the end-point of the second experiment. The second experiment is large-scale data collection of the fruit of interest, with features being recorded every day, until the fruit reaches end-of-life as determined by the first experiment. Then, computer vision is used to perform feature extraction and features are recorded over each sample fruit’s lifetime. The recorded data is then analyzed with regression and other techniques to build a model of the fruit’s shelf-life. The model is finally validated. This thesis uses bananas as a proof of concept of the proposed process.
LITERATURE REVIEW
The purpose of this literature review is to provide the readers with an understanding of how computer vision is applied in the agriculture industry today. This will identify current industry standards as well as active areas of research. Additionally, this literature review will serve to identify gaps in the current research.
Figure 5: CIE L*a*b* color representation. (De, 2015).
To overcome this, RGB colors get converted to XYZ coordinates in order to standardize the RGB values. The XYZ coordinates are then converted to L*a*b* coordinates which are considered to be the most intuitive and the most accurate representation of reality as the human eye sees it (Figure 5). While this does transform RGB coordinates, the issue with the transformation is that the transformation is very computationally intensive and time-consuming (Vidal et al., 2013 and Pedreschi et al., 2006).
PROPOSED PROCESS
Figure 6: Average percent colors over life-time.
The goal of this thesis is to present a model that uses color to predict shelf-life. For bananas (which this thesis uses as a proof of concept), color changes over time, which will influence supply chain distribution decisions. This can be seen in Figure 6, which plots the average percent of each respective color that a banana can be over time. Supply chain distribution decisions when the color is primarily green will be starkly different than distribution decisions when the color of the banana is primarily yellow.
Figure 7: Hue change in a banana over its life-time.
In order to build a model of shelf-life as a function of time, the RGB coordinates of a large sample of bananas are measured in an empirical observational study. Then, the RGB coordinates are transformed to hue and a* coordinates. An example of how the hue of a banana changes as it ripens is shown in Figure 7. Supply chain distribution decisions will vary greatly based on the color of the fruit.
EXPERIMENTAL PROCEDURES AND METHODOLOGIES
Figure 8: Experimental procedure set-up.
Lighting should be as consistent among measurements as possible. Use the same light source each time and try to block out any external light. In the thesis experiment, all external light was blocked and the only source of illumination was the Macro Ring lighting source for the Canon camera. The Sony computer vision camera used ambient lighting. The experimental setup for this thesis is shown in Figure 8.
Figure 11: Example region of interest.
Load the images into LabVIEW Vision Builder by pressing “Simulate Acquisition” and navigate the file path to the folder with the acquired images. Choose a consistent 150 pixel by 150 pixel square region of interest to perform processing. An example region of interest can be found in Figure 11.
DIRECT EXPERIMENTAL RESULTS
Figure 16: Example hue change over life-time of banana 20.
Finally, the average hue value is calculated by dividing the sum of the hue values of the pixels in the region of interest by the total number of pixels (for this thesis, 22500 pixels) in the region of interest. Hue maximum is the largest hue value in the region of interest and hue minimum is the smallest hue value in the region of interest. All of the calculations are done in LabVIEW. The average hue values over the lifetime of a selected banana are shown in Figure 16.
Figure 24: Example interpolation of hue values 40 and 30.
Finally, the ‘days’ value is subtracted from the number of total days that the specific fruit lasted to compute the shelf-life value at each point of interest. Because each banana has its own unique shelf-life that is used in the calculation, the last subtraction step also standardizes all of the data, which allows for comparison between bananas. A visual representation of this shelf-life calculation is shown in Figure 24.
Figure 26: Data calculation a* shelf-life calculation for all bananas.
The same type of analysis was done on the a* data. The a* range of interest is {-10, -8, -6, -4, -2, 0}. The nearest neighbors above and below the a* of interest were found and interpolated between to find the exact ‘days passed’ value. This value is then subtracted from the total days the individual banana lasted to finally calculate the shelflife. Similar to the hue analysis above, a third-order polynomial was fit to the data to extrapolate a* values outside of the empirically recorded data. The R-squared value of every best-fit polynomial was over 99.5%. The results of the calculation of remaining shelf-life using the data for a* is shown in Figure 26.
MODEL RESULTS AND DISCUSSION
Figure 27: Shelf-life model as a function of average hue.
The ultimate goal of the thesis is to present a process to predict shelf-life of fruit based on distinct features extracted using computer vision. Ultimately, the desired end product to show proof of concept of the process is a model that predicts shelf-life based on color features. After data collection, processing, remaining shelf-life calculation, and modeling, a function of shelf-life based on hue is created. It is described by the equation: ‘Shelf-life’ = 0.3401 * ‘Hue’ – 5.04 (shown as the dotted red line in Figure 27).
Figure 29: Random error of data calculation model.
This is the reason why extrapolation in the data calculation section used third-order polynomials instead of linear best-fit lines. It also explains why the regression model was the worst of the validated models. Another reason that the data calculation method is preferred is because most of the error of the regression model came early in the banana’s life-times (as shown in Figure 29).
Figure 31: Validation experimental results.
Then, because the hue data calculation method has performed the best thus far in the thesis, the results of the hue data calculation shelf-life prediction (with the standard deviation calculated in the “Analysis” section of this thesis) are plotted against the empirically observed shelf-life. This is shown in Figure 31. It is important to note that all but one of the empirical shelf-life observations fall within the standard deviation of the prediction.
SUMMARY AND FUTURE WORK
Summary
Ultimately, this thesis has demonstrated proof of concept of a process to use computer vision to build a predictive model of shelf-life of fruit. Computer vision is becoming widespread in many industries and is, therefore, a burgeoning field of research. It is most widely used on inspection lines of electronic components. Computer vision holds many advantages to its human counterpart: it is faster, more reliable, cheaper in the long-term, and more accurate. Naturally, computer vision is starting to become implemented in other industries.
One of the most important fields that computer vision is starting to move into is the agriculture industry. Computer vision excels at doing a welldefined process many times. This lends well to industries such as electronics production because every piece of one electronic component must look the same. This means that a computer can rapidly capture an image of an electronic component coming onto an inspection line, compare certain regions to an ideal (correctly produced) image of the component, and then make a decision to sell or discard the component. In the agriculture industry, every piece of produce is unique, so the process described above for electronic components is no longer valid.
Future Work
This thesis lays the groundwork for a myriad of further experiments as well as a multitude of alternations of the process. Naturally, the process may be able to be applied to any fruit that adheres to the two criteria enumerated above. Future work can possibly apply the process to many fruits. Among the best candidates for future fruit are mangoes, apples, avocados, blackberries, melons, and tomatoes. This is because they are climacteric and have the most distinct color changes in their life-time of ripeness.
Source: California Polytechnic State University
Authors: Nandan G. Thor