isprs archives XLI B8 1207 2016
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLI-B8, 2016
XXIII ISPRS Congress, 12–19 July 2016, Prague, Czech Republic
A STUDY ON PRODUCING HIGHLY RELIABILE REFERENCE DATA SETS FOR
GLOBAL LAND COVER VALIDATION
N.Soyama a, K.Muramatsu b, M. Daigo c, F.Ochiai d, N.Fujiwara d
a
Department of Faculty of Human Studies, Tenri University, 1050 Somanouchi, Tenri-shi, Nara,
Japan – soyama@sta.tenri-u.ac.jp
b
Department of Environmental Science, Nara Women's University, Kitauoya Nishimachi, Nara-shi, Nara, Japan –
muramatu@ics.nara-wu.ac.jp
c
Faculty of Economics, Doshisha University, Karasuma-higashi-iru, Imadegawa-dori, Kamigyo-ku, Kyoto-shi, Kyoto,
Japan – daigo@doshisha.ac.jp
b
KYOSEI Science Centre for Life and Nature, Nara Women's University, Kitauoya Nishimachi, Nara-shi, Nara, Japan
– ochiai@tezukayama.ac.jp, fujiwara@nifty.com
KEY WORDS: Global land cove, Validation, Reference data, FLUXNET, spatial resolution, IGBP class
ABSTRACT:
Validating the accuracy of land cover products using a reliable reference dataset is an important task. A reliable reference dataset is
produced with information derived from ground truth data. Recently, the amount of ground truth data derived from information
collected by volunteers has been increasing globally. The acquisition of volunteer-based reference data demonstrates great potential.
However information given by volunteers is limited useful vegetation information to produce a complete reference dataset based on
the plant functional type (PFT) with five specialized forest classes. In this study, we examined the availability and applicability of
FLUXNET information to produce reference data with higher levels of reliability. FLUXNET information was useful especially for
forest classes for interpretation in comparison with the reference dataset using information given by volunteers.
1. INTRODUCTION
The Global Change Observation Mission-Climate (GCOM-C1)
is progressing along with the Japan Aerospace Exploration
Agency (JAXA)’s activities, and the second-generation global
imager (SGLI) on GCOM-C will be launched in 2016 fiscal
year. SGLI covers a wide spectrum from 380 nm to 12 um with
spatial resolutions from 250 m to 1 km, polarimetry, and
forward/backward simultaneous observation capabilities. Global
land cover products have been planned to be produced by the
International Geosphere-Biosphere Programme (IGBP) class
scheme (Loveland, 1997) with the SGLI 250m spatial resolution
data to derive gross primary production (GPP) estimations. GPP
estimation studies require highly accurate land cover products
whose class scheme is based on the plant functional type (PFT)
since the estimating error of GPP products is dependent on the
accuracy of land cover products. Therefore, validating the
accuracy of land cover products using a reliable reference
dataset is important.
Reference datasets for validation methods are produced using a
large amount of high spatial resolution satellite data (Olofsson,
2012 and Stehman, 2012), and by collecting information on the
land cover state using a broad network of experts with local
knowledge (Mayaux, 2006 and Bontemps, 2015). Recently,
land cover information collected by volunteer such as that
obtained in the Degree Confluence Project (DCP) was used to
validate land cover products (Foody, 2013 and Iwao, 2006).
They use land cover information as ground truth information for
producing reliable reference data. The information includes a
location (latitude and longitude), photographs of landscapes,
and brief descriptions of the land state (DCP, 2015). DCP is a
volunteer-based project that aims to collect land cover
information at the point of latitude and longitude integer degree
intersection in the world.
In our previous study (Soyama, 2015), a method was developed
to produce high-quality reference data, using the information
from DCP to validate global land cover products based on the
IGBP land cover class scheme in 500m spatial resolution. The
DCP data is collected from a uniform spatial and temporal
spread of potential sampling points. However, the DCP includes
limited useful vegetation information to produce a complete
reference dataset based on the PFT with five forest classes:
evergreen needleleaf forest, evergreen broadleaf forest,
deciduous needleleaf forest, deciduous broadleaf forest, mixed
forest. GPP estimation studies (Sasai, 2011 and
Thanyapraneedkul, 2012) use the FLUXNET database
(FLUXNET, 2015). FLUXNET contains information about
tower locations and site characteristics. Thus, we can expect
more comprehensive and detailed vegetation information from
this resource that can be used to produce a more accurate
reference dataset.
In this study, we examined the availability and applicability of
FLUXNET information to produce reference data with higher
levels of reliability based on our method developed in previous
study. Reference dataset, which is produced using the method
are applied to MODIS IGBP global land cover products. Finally,
a discussion about the use of FLUXNET information to validate
global land cover products is presented.
2. DATA
2.1 Data used for interpretation
FLUXNET is a global network of micrometeorological tower
sites that use eddy covariance methods to measure the
exchanges of carbon dioxide, water vapor, and energy between
terrestrial ecosystems and the atmosphere. FLUXNET contains
information about tower locations and site characteristics. In
this study, information of Ameriflux, Asiaflux, Ozflux,
Canadian Carbon Program (CCP) and Infrastructure for
Measurements of the European Carbon Cycle (IMECC) were
used. In total, 409 flux-site points were selected. As information
This contribution has been peer-reviewed.
doi:10.5194/isprsarchives-XLI-B8-1207-2016
1207
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLI-B8, 2016
XXIII ISPRS Congress, 12–19 July 2016, Prague, Czech Republic
for interpretation of reference data, Coordinates (longitudes,
latitudes), photographs and Site Characteristics (Investigator
Provided Vegetation Type) from FLUXNET web site were
used. After checking those information in FLUXNET web site,
we found that there is a little error information for coordinates
and lack of Investigator Provided Vegetation Type in web site
description. Therefore, we also use site additional information
on network: Ameriflux (Ameriflux, 2015), Asiafux (Asiaflux,
2015), CCP(CCP, 2015) ,IMECC(European Fluxes Database
Cluster, 2015) and Ozflux (Ozflux, 2015).
Figure 1 shows the distribution of 409 confluences on Google
Earth examined in this study.
Quality level of reference data is determined by assigning each
label to two interpreters and combining their common
interpretations with four land cover types.
3.2 Quality level of reference data
The quality level of reference data was determined using the
number of agreement between labels by two interpreters, the
reliability level of labeled classes assigned using four simple
land cover types and the number of candidate class nominated
according to the rule determined in our previous study (Soyama,
2015).
The conditions of the three quality levels are defined as follows:
Quality level 1: the two labels agreed upon by two interpreters,
as well as IGBP classes interpreted by two interpreters are
considered to have high reliability of labeled class assigned
using four simple land cover types, and they nominate one
candidate class.
Quality level 2: Agreement by interpreters on two labels,
considered to have high reliability.
Quality level 3: Agreement by interpreters on two of the
assigned labels, considered to have high or medium reliability.
Figure 1. Distribution of 409 confluences on Google Earth
examined in this study.
MODIS/Aqua Surface Reflectance 8-Day L3 Global 500 m data
for all bands was used to examine the seasonal changes in the
vegetation classes.
In this study, reference data are interpreted based on the IGBP
class scheme at a spatial resolution of 500 m, which is the
double space scale of SGLI sensor of 250m spatial resolution.
Google Earth was used to validate the accuracy of the class
interpretations at 500 m resolution.
2.2 Global land cover map
Land cover type 1 (IGBP global vegetation classification
scheme) of the MODIS land cover type products (MCD12Q1)
in 2007 was used.
3. METHOD
3.1 Interpretation of reference data
Method of producing reference data is mainly the same as our
previous study. Interpretation of reference data was conducted
by two interpreters. Their method of interpretation was the
following procedure. Firstly, label was interpreted from
characteristics (Investigator Provided Vegetation Type) on
FLUXNET website (FLUXNET, 2015) and additional site
information on website for each fluxnet. For Ameriflux,
Asiafux and Ozflux, detailed land cover condition information
helped accurate interpretation. Two interpreters can line up as
many as four candidate land cover classes with variable
reliability. Secondary, label interpreted using fluxnet
information was checked at a spatial resolution of 500 m using
Google Earth. The interpreters identify a flux site point at 500 m
spatial resolution as one of four land cover types i.e., woody
land, herbaceous land, mixel or non-vegetated (hereafter simple
land cover types), by cross-referencing the location with Google
Earth data according to the rule determined in our previous
study (Soyama, 2015).
4. RESULTS AND DISCUSSION
4.1 Agreement
interpreters
of
candidate
class-1
between
two
The kappa coefficients of the agreements between the two
interpreters on candidate class-1 for all data is good to fair
(kappa coefficient = 0.781). Based on the kappa coefficients,
the two interpreters appear to apply the same rules for data
interpretation.
Table 1 shows the percentage distribution of IGBP classes
excluding water and snowice recognized on candidate class-1
by the two interpreters (V1, V2). The total percentage of forests
candidate class-1 (46%, 48%) was higher than other class types.
Interpreter
ENF
EBF
DNF
DBF
MF
V1
22.2%
7.3%
1.5%
8.8%
6.4%
V2
23.5%
6.4%
1.7%
10.3%
5.9%
CSH
OSH
WSA
SAV
GRA
V1
1.0%
5.6%
1.5%
0.7%
19.6%
V2
2.0%
3.9%
2.2%
0.0%
20.5%
WET
CRO
URB
MOS
BSV
V1
3.9%
16.4%
1.2%
3.4%
0.5%
V2
4.9%
14.9%
1.2%
2.2%
0.5%
Table 1. The percentage distribution of IGBP classes labeled by
two interpreters.
In our previous study using DCP information, the interpreted
reference data by IGBP classification, grassland and cropland
were recognized more often than the other classes by the two
interpreters. The total percentage of five forest types was not so
This contribution has been peer-reviewed.
doi:10.5194/isprsarchives-XLI-B8-1207-2016
1208
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLI-B8, 2016
XXIII ISPRS Congress, 12–19 July 2016, Prague, Czech Republic
high comparing grassland and cropland. This may be due to
poor forest landscape visibility with many DCP data points and
it is not easy to reach point of a dense forest for volunteers. On
the other hand, dense forest sites of FLUXNET are selected by
specialists for each research objectives.
4.2 Reliability of labels based on simple land cover types
For the simple land cover types for all the data, the agreement
between two interpreters from ranged from good to fair (kappa
coefficient = 0.775). Table 2 shows the reliability of the two
interpreters’ assigned labels on simple land cover types, based
on the IGBP’s definition of reliability levels (shown in Table 1
of our previous study paper (Soyama, 2015)). The FLUXNET
sites were assigned to high-reliability classes with a 90% and
97%, respectively, whereas low-reliability classes are assigned
at 3% and 0% of the cases, respectively. Although there were
some incorrect interpretations of the simple land cover types,
the IGBP classes’ interpretations by the two interpreters were
considered to be highly reliable.
There are less the low or the middle reliability class points by
V2. V2 lined up multiple candidate land cover classes for their
flux sites. The low or the middle reliability classes by V1 are 42
points in which similarly V1 lined up multiple candidate land
cover classes except 2 sites. This may that the low or the middle
reliability, determined by two interpreters, was due to an
erroneous interpretation of the simple land cover type.
Simple
land cover
types
Nonvegetated
V1
High
Middle
V2
Low
3
High
Middle
Low
4
Woody
land
185
5
6
202
Herbaceous
land
164
4
7
171
Mixel
15
20
Total (%)
367
29
409 points
(90%)
(7%)
1
1
overstory and understory, canopy height, age of the vegetation
and fetch of the land condition. On that of the Ameriflux, there
are dominant species composition, canopy height, vegetation
type, land history and research topics. Oz-flux has useful
information on ‘Site Description’ pages. Most of CPP sites are
included on Ameriflux sites. In case of IMECC, there is only
IGBP class name, whereas the percentage of quality-level-1
reference data to all points for each flux network was a little
lower than other flux network.
flux
network
Ameriflux
AsiaFlux
CCP
IMECC
Oz-flux
Quality
level-1
48%
50%
47%
43%
52%
Table 3. The percentage of quality-level-1 reference data to all
points for each flux network.
Table 4 shows the percentage distribution of IGBP classes of
quality level-1 reference data (192 points) for each flux network.
The highest number of the IGBP classes in quality level 1 was
evergreen needleleaf forest (27%) and cropland (19%), followed
by grassland (18%), deciduous broadleaf forest (12%) and
evergreen broadleaf forest (11%). The total amount of five
forest classes rate was 56%. In previous study using DCP
information, the highest number of the IGBP class in quality
level-1 was cropland (55%) , and the total amount of five forest
classes rate was 30%. The cause is that the DCP includes
limited useful vegetation information to produce a complete
reference dataset based on PFT with five specialized forest
classes.
flux
network
ENF
EBF
Ameriflux
11%
2%
AsiaFlux
2%
6%
CCP
6%
IMECC
8%
DNF
2%
DBF
MF
OSH
4%
2%
2%
5%
1%
1%
1
20
9
13
397
10
2
(3%)
(97%)
(2%)
(0%)
Table 2. The reliability of the two interpreters’ labels based on
simple land cover types.
4.3 Quality of reference data for FLUXNET
The number of reference datasets selected based on three
quality-level conditions was 192, 335 and 370 in quality-level
order. About half of total flux sites used in this study satisfied
the condition of quality level 1. Most flux sites (82%) satisfied
the condition of quality level 2. FLUXNET information was
considered to be efficient in interpretation in comparison with
the result using DCP information.
Table 3 shows the percentage of quality level-1 reference data
to all points for each flux network. There is characteristics
information on web site of flux networks except IMECC. In
particular, on the Asiaflux network database on web, there are
useful information for interpretation; domestic species of
Oz_flux
Total
1%
1%
3%
2%
3%
27%
11%
2%
12%
4%
3%
WSA
GRA
WET
CRO
UBR
Total
Ameriflux
7%
1%
8%
AsiaFlux
2%
1%
4%
CCP
1%
1%
IMECC
5%
1%
Oz_flux
1%
3%
Total
1%
18%
37%
1%
7%
7%
27%
1%
3%
22%
19%
7%
1%
Table 4. The percentage distribution of IGBP classes of qualitylevel-1 reference data (192 points) for each flux network.
This contribution has been peer-reviewed.
doi:10.5194/isprsarchives-XLI-B8-1207-2016
1209
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLI-B8, 2016
XXIII ISPRS Congress, 12–19 July 2016, Prague, Czech Republic
4.4 Comparison of results between the MODIS IGBP land
cover products and reference data with quality level
Table 5 shows the validation results of the MODIS IGBP global
land cover products, with the reference datasets selected based
on three quality-level conditions as well as the number of
quality-level reference data. One-pixel columns show the
overall accuracy and the kappa coefficients for One-pixels.
Nine-pixels columns display the result of Nine-pixels. Both the
overall accuracy and kappa coefficients of quality level 1 are the
highest of the three quality-level reference datasets. For quality
level 1, agreement between two interpreters ranged from good
to fair (kappa coefficient = 0.620, 0.725). Accuracy decreases
with the quality level. This outcome is the same for validation
with reference data using DCP information in our previous
study.
The highest number of Nine-points with high levels of
agreement among the IGBP classes in quality level 1 was
evergreen needleleaf forest (27%) and cropland (19%), followed
by grassland (18%), deciduous broadleaf forest (12%) and
evergreen broadleaf forest (11%).
Number of
reference
datasets
Validation
area
Overall
accuracy
Kappa
coefficient
Quality level
1
Quality level
2
Quality level
3
192
335
370
1
pixel
9
pixels
1
pixel
9
pixels
1
pixel
9
pixels
67%
77%
54%
59%
51%
53%
0.62
0.72
0.48
0.54
0.45
0.47
Table 5. The validation results of the MODIS IGBP global land
cover products with quality-level reference data.
Table 6 shows the validation results of the MODIS IGBP global
land cover products with quality-level reference data for each
fluxnet. The agreement result of four fluxnet (Ameriflux,
Asiaflux, IMECC and Oz_flux) ranged from good to fair (kappa
coefficient = 0.80, 0.67, 0.72 and 0.69). Most disagreement
IGBP class between quality level-1 in CCP and the MODIS
IGBP global land cover products were ENF. Most of them were
managed forests in which include clear-cut area.
Fluxnet
Ameriflux
Asiaflux
CCP
IMECC
Oz_flux
Number of Quality
level-1 sites
71
43
14
51
13
Overall accuracy
(kappa coefficients)
83% (0.80)
72% (0.67)
50% (0.29)
78%
77%
(0.72)
(0.69)
Table 6. The validation results of the MODIS IGBP global land
cover products with quality-level reference data for each fluxnet.
5. SUMMARY
Our final aim is to produce high-quality reference data set for
validation of global land cover product in GCOM-C. Validation
needs a reliable reference dataset produced with information
derived from ground truth data. Recently, the amount of ground
truth data derived from information collected by volunteers has
been increasing globally. However, it is difficult to produce
forest reference data in high-quality, since the DCP includes
limited useful vegetation information to produce a complete
reference data of forest classes. In this study, we applied the
method of producing reference data sets with quality-level
information to FLUXNET information. FLUXNET information
was useful especially for forest classes for interpretation.
Acknowledgements :
This study was supported in part by a Global Change
Observation Mission (GCOM; PI No. 114) project at JAXA.
References :
Bontemps, P., Defourny, P., Bogaert, E.V., Arino, O.,
Kalogirou, V. and Perez, J.R. GLOBCOVER 2009 Products
Description
and
Validation
Report,
available
at
http://due.esrin.esa.int/page_globcover.php (20 Sep. 2015).
Degree Confluence Project, available at http://confluence.org/
(20 Sep. 2015).
Foody, G.M. and Boyd, D.S., 2013. Using Volunteered Data in
Land Cover Map Validation: Mapping West African Forests,
IEEE Journal of selected topics in applied earth observations
and remote sensing, Vol. 6, No.3, pp.1939-1440.
Iwao, K., Nishida, K., Kinoshita, T., and Yamadagata, Y., 2006.
Validating land cover maps with Degree Confluence Project
information, Geophysical research letters, vo.33, L23404, doi
10.1029/2006GL027768.
Mayaux, P., Eva, H., Gallego, J., Strahler, A.H., Herold, M.,
Agrawal, S., Naumov, S., Miranda, E.E.De, Bella, C. M.Di,
Ordoyne, C., Kopin, Y., and P. S. Roy., 2006. Validation of the
Global Land Cover 2000 Map, IEEE TRANSACTIONS ON
GEOSCIENCE AND REMOTE SENSING, VOL. 44, NO. 7,
pp.1728-1737.
Olofsson, P., Stehman, S.V., Woodcock, C.E., Friedl, M.A.
Sulla-Menashe, D., Sibley, A.M., Newell, J.D. and M. Herold.,
2012. A global land cover validation data set, I: Fundamental
Design principles, International Journal of Remote Sensing, Vol.
33, pp.5768-5788.
Sasai, T., Saigusa, N., Nasahara, N.K., Ito, A. , Hashimoto, H.,
Nemani, R.R., Hirata, R., Ichii, K., Takagi, K., Saitoh, M., Ohta,
T., Murakami K.,, Yamaguchi Y. and Oikawa, T., 2011.
Satellite-driven estimation of terrestrial carbon flux over Far
East Asia with 1-km grid resolution, Remote Sensing of
Environment, 115, pp.1758-1771.
Soyama, N., Sasai, T., Muramatsu, K., Daigo, M., Ochiai, F.,
Nasahara, K., 2015. Validation Method of Global Land Cover
Map using Reference Data with Quality level : Using the
Degree Confluence Project Information", ISTS 2015
Proceedings, 6pages.
Stehman, S.V., Olofsson, P., Woodcock, C.E., M. Herold, and
M.A. Friedl., 2012. A global land cover validation data set, II:
Augmenting a stratified sampling design to estimate accuracy
This contribution has been peer-reviewed.
doi:10.5194/isprsarchives-XLI-B8-1207-2016
1210
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLI-B8, 2016
XXIII ISPRS Congress, 12–19 July 2016, Prague, Czech Republic
by region and land-cover class, International Journal of Remote
Sensing, 33(22), pp. 6975-6993.
Thanyapraneedkul, J., Muramatsu K, Daigo, D., Furumi, S.,
Soyama, N., Nasahara, K., Muraoka, H., Noda, M.H., Nagai, S.,
Maeda T., Mano, M., and Mizoguchi Y., 2012. A vegetation
index to estimate terrestrial gross primary production capacity
for the GCOM-C/SGLI satellite sensor, Remote Sensing, 4,
pp.3689-3720.
The Ameriflux, available at http://ameriflux.ornl.gov (20 July
2015) and http://ameriflux.lbl.gov (20 July 2015).
The Asiaflux, available at http://asiaflux.net (20 July 2015).
The CCP database, ftp://daac.ornl.gov/data/fluxnet/fluxnet_
canada /data/ (20 July 2015).
The European Fluxes Database Cluster. available
http://gaia.agraria.unitus.it/home/sites-list (20 July 2015).
at
The FLUXNET (Network of regional networks of observations
from micrometeorological tower sites), available at
http://fluxnet.ornl.gov/ (20 July 2015).
The OzFlux, available at http://www.ozflux.org.au (20 July
2015).
Loveland, T.R., Belward, A.S., 1997. The IGBP-DIS global 1
km land cover data set, DISCover: first results. International
Journal of Remote Sensing, Vol. 18, No. 15, pp.3289-3295.
This contribution has been peer-reviewed.
doi:10.5194/isprsarchives-XLI-B8-1207-2016
1211
XXIII ISPRS Congress, 12–19 July 2016, Prague, Czech Republic
A STUDY ON PRODUCING HIGHLY RELIABILE REFERENCE DATA SETS FOR
GLOBAL LAND COVER VALIDATION
N.Soyama a, K.Muramatsu b, M. Daigo c, F.Ochiai d, N.Fujiwara d
a
Department of Faculty of Human Studies, Tenri University, 1050 Somanouchi, Tenri-shi, Nara,
Japan – soyama@sta.tenri-u.ac.jp
b
Department of Environmental Science, Nara Women's University, Kitauoya Nishimachi, Nara-shi, Nara, Japan –
muramatu@ics.nara-wu.ac.jp
c
Faculty of Economics, Doshisha University, Karasuma-higashi-iru, Imadegawa-dori, Kamigyo-ku, Kyoto-shi, Kyoto,
Japan – daigo@doshisha.ac.jp
b
KYOSEI Science Centre for Life and Nature, Nara Women's University, Kitauoya Nishimachi, Nara-shi, Nara, Japan
– ochiai@tezukayama.ac.jp, fujiwara@nifty.com
KEY WORDS: Global land cove, Validation, Reference data, FLUXNET, spatial resolution, IGBP class
ABSTRACT:
Validating the accuracy of land cover products using a reliable reference dataset is an important task. A reliable reference dataset is
produced with information derived from ground truth data. Recently, the amount of ground truth data derived from information
collected by volunteers has been increasing globally. The acquisition of volunteer-based reference data demonstrates great potential.
However information given by volunteers is limited useful vegetation information to produce a complete reference dataset based on
the plant functional type (PFT) with five specialized forest classes. In this study, we examined the availability and applicability of
FLUXNET information to produce reference data with higher levels of reliability. FLUXNET information was useful especially for
forest classes for interpretation in comparison with the reference dataset using information given by volunteers.
1. INTRODUCTION
The Global Change Observation Mission-Climate (GCOM-C1)
is progressing along with the Japan Aerospace Exploration
Agency (JAXA)’s activities, and the second-generation global
imager (SGLI) on GCOM-C will be launched in 2016 fiscal
year. SGLI covers a wide spectrum from 380 nm to 12 um with
spatial resolutions from 250 m to 1 km, polarimetry, and
forward/backward simultaneous observation capabilities. Global
land cover products have been planned to be produced by the
International Geosphere-Biosphere Programme (IGBP) class
scheme (Loveland, 1997) with the SGLI 250m spatial resolution
data to derive gross primary production (GPP) estimations. GPP
estimation studies require highly accurate land cover products
whose class scheme is based on the plant functional type (PFT)
since the estimating error of GPP products is dependent on the
accuracy of land cover products. Therefore, validating the
accuracy of land cover products using a reliable reference
dataset is important.
Reference datasets for validation methods are produced using a
large amount of high spatial resolution satellite data (Olofsson,
2012 and Stehman, 2012), and by collecting information on the
land cover state using a broad network of experts with local
knowledge (Mayaux, 2006 and Bontemps, 2015). Recently,
land cover information collected by volunteer such as that
obtained in the Degree Confluence Project (DCP) was used to
validate land cover products (Foody, 2013 and Iwao, 2006).
They use land cover information as ground truth information for
producing reliable reference data. The information includes a
location (latitude and longitude), photographs of landscapes,
and brief descriptions of the land state (DCP, 2015). DCP is a
volunteer-based project that aims to collect land cover
information at the point of latitude and longitude integer degree
intersection in the world.
In our previous study (Soyama, 2015), a method was developed
to produce high-quality reference data, using the information
from DCP to validate global land cover products based on the
IGBP land cover class scheme in 500m spatial resolution. The
DCP data is collected from a uniform spatial and temporal
spread of potential sampling points. However, the DCP includes
limited useful vegetation information to produce a complete
reference dataset based on the PFT with five forest classes:
evergreen needleleaf forest, evergreen broadleaf forest,
deciduous needleleaf forest, deciduous broadleaf forest, mixed
forest. GPP estimation studies (Sasai, 2011 and
Thanyapraneedkul, 2012) use the FLUXNET database
(FLUXNET, 2015). FLUXNET contains information about
tower locations and site characteristics. Thus, we can expect
more comprehensive and detailed vegetation information from
this resource that can be used to produce a more accurate
reference dataset.
In this study, we examined the availability and applicability of
FLUXNET information to produce reference data with higher
levels of reliability based on our method developed in previous
study. Reference dataset, which is produced using the method
are applied to MODIS IGBP global land cover products. Finally,
a discussion about the use of FLUXNET information to validate
global land cover products is presented.
2. DATA
2.1 Data used for interpretation
FLUXNET is a global network of micrometeorological tower
sites that use eddy covariance methods to measure the
exchanges of carbon dioxide, water vapor, and energy between
terrestrial ecosystems and the atmosphere. FLUXNET contains
information about tower locations and site characteristics. In
this study, information of Ameriflux, Asiaflux, Ozflux,
Canadian Carbon Program (CCP) and Infrastructure for
Measurements of the European Carbon Cycle (IMECC) were
used. In total, 409 flux-site points were selected. As information
This contribution has been peer-reviewed.
doi:10.5194/isprsarchives-XLI-B8-1207-2016
1207
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLI-B8, 2016
XXIII ISPRS Congress, 12–19 July 2016, Prague, Czech Republic
for interpretation of reference data, Coordinates (longitudes,
latitudes), photographs and Site Characteristics (Investigator
Provided Vegetation Type) from FLUXNET web site were
used. After checking those information in FLUXNET web site,
we found that there is a little error information for coordinates
and lack of Investigator Provided Vegetation Type in web site
description. Therefore, we also use site additional information
on network: Ameriflux (Ameriflux, 2015), Asiafux (Asiaflux,
2015), CCP(CCP, 2015) ,IMECC(European Fluxes Database
Cluster, 2015) and Ozflux (Ozflux, 2015).
Figure 1 shows the distribution of 409 confluences on Google
Earth examined in this study.
Quality level of reference data is determined by assigning each
label to two interpreters and combining their common
interpretations with four land cover types.
3.2 Quality level of reference data
The quality level of reference data was determined using the
number of agreement between labels by two interpreters, the
reliability level of labeled classes assigned using four simple
land cover types and the number of candidate class nominated
according to the rule determined in our previous study (Soyama,
2015).
The conditions of the three quality levels are defined as follows:
Quality level 1: the two labels agreed upon by two interpreters,
as well as IGBP classes interpreted by two interpreters are
considered to have high reliability of labeled class assigned
using four simple land cover types, and they nominate one
candidate class.
Quality level 2: Agreement by interpreters on two labels,
considered to have high reliability.
Quality level 3: Agreement by interpreters on two of the
assigned labels, considered to have high or medium reliability.
Figure 1. Distribution of 409 confluences on Google Earth
examined in this study.
MODIS/Aqua Surface Reflectance 8-Day L3 Global 500 m data
for all bands was used to examine the seasonal changes in the
vegetation classes.
In this study, reference data are interpreted based on the IGBP
class scheme at a spatial resolution of 500 m, which is the
double space scale of SGLI sensor of 250m spatial resolution.
Google Earth was used to validate the accuracy of the class
interpretations at 500 m resolution.
2.2 Global land cover map
Land cover type 1 (IGBP global vegetation classification
scheme) of the MODIS land cover type products (MCD12Q1)
in 2007 was used.
3. METHOD
3.1 Interpretation of reference data
Method of producing reference data is mainly the same as our
previous study. Interpretation of reference data was conducted
by two interpreters. Their method of interpretation was the
following procedure. Firstly, label was interpreted from
characteristics (Investigator Provided Vegetation Type) on
FLUXNET website (FLUXNET, 2015) and additional site
information on website for each fluxnet. For Ameriflux,
Asiafux and Ozflux, detailed land cover condition information
helped accurate interpretation. Two interpreters can line up as
many as four candidate land cover classes with variable
reliability. Secondary, label interpreted using fluxnet
information was checked at a spatial resolution of 500 m using
Google Earth. The interpreters identify a flux site point at 500 m
spatial resolution as one of four land cover types i.e., woody
land, herbaceous land, mixel or non-vegetated (hereafter simple
land cover types), by cross-referencing the location with Google
Earth data according to the rule determined in our previous
study (Soyama, 2015).
4. RESULTS AND DISCUSSION
4.1 Agreement
interpreters
of
candidate
class-1
between
two
The kappa coefficients of the agreements between the two
interpreters on candidate class-1 for all data is good to fair
(kappa coefficient = 0.781). Based on the kappa coefficients,
the two interpreters appear to apply the same rules for data
interpretation.
Table 1 shows the percentage distribution of IGBP classes
excluding water and snowice recognized on candidate class-1
by the two interpreters (V1, V2). The total percentage of forests
candidate class-1 (46%, 48%) was higher than other class types.
Interpreter
ENF
EBF
DNF
DBF
MF
V1
22.2%
7.3%
1.5%
8.8%
6.4%
V2
23.5%
6.4%
1.7%
10.3%
5.9%
CSH
OSH
WSA
SAV
GRA
V1
1.0%
5.6%
1.5%
0.7%
19.6%
V2
2.0%
3.9%
2.2%
0.0%
20.5%
WET
CRO
URB
MOS
BSV
V1
3.9%
16.4%
1.2%
3.4%
0.5%
V2
4.9%
14.9%
1.2%
2.2%
0.5%
Table 1. The percentage distribution of IGBP classes labeled by
two interpreters.
In our previous study using DCP information, the interpreted
reference data by IGBP classification, grassland and cropland
were recognized more often than the other classes by the two
interpreters. The total percentage of five forest types was not so
This contribution has been peer-reviewed.
doi:10.5194/isprsarchives-XLI-B8-1207-2016
1208
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLI-B8, 2016
XXIII ISPRS Congress, 12–19 July 2016, Prague, Czech Republic
high comparing grassland and cropland. This may be due to
poor forest landscape visibility with many DCP data points and
it is not easy to reach point of a dense forest for volunteers. On
the other hand, dense forest sites of FLUXNET are selected by
specialists for each research objectives.
4.2 Reliability of labels based on simple land cover types
For the simple land cover types for all the data, the agreement
between two interpreters from ranged from good to fair (kappa
coefficient = 0.775). Table 2 shows the reliability of the two
interpreters’ assigned labels on simple land cover types, based
on the IGBP’s definition of reliability levels (shown in Table 1
of our previous study paper (Soyama, 2015)). The FLUXNET
sites were assigned to high-reliability classes with a 90% and
97%, respectively, whereas low-reliability classes are assigned
at 3% and 0% of the cases, respectively. Although there were
some incorrect interpretations of the simple land cover types,
the IGBP classes’ interpretations by the two interpreters were
considered to be highly reliable.
There are less the low or the middle reliability class points by
V2. V2 lined up multiple candidate land cover classes for their
flux sites. The low or the middle reliability classes by V1 are 42
points in which similarly V1 lined up multiple candidate land
cover classes except 2 sites. This may that the low or the middle
reliability, determined by two interpreters, was due to an
erroneous interpretation of the simple land cover type.
Simple
land cover
types
Nonvegetated
V1
High
Middle
V2
Low
3
High
Middle
Low
4
Woody
land
185
5
6
202
Herbaceous
land
164
4
7
171
Mixel
15
20
Total (%)
367
29
409 points
(90%)
(7%)
1
1
overstory and understory, canopy height, age of the vegetation
and fetch of the land condition. On that of the Ameriflux, there
are dominant species composition, canopy height, vegetation
type, land history and research topics. Oz-flux has useful
information on ‘Site Description’ pages. Most of CPP sites are
included on Ameriflux sites. In case of IMECC, there is only
IGBP class name, whereas the percentage of quality-level-1
reference data to all points for each flux network was a little
lower than other flux network.
flux
network
Ameriflux
AsiaFlux
CCP
IMECC
Oz-flux
Quality
level-1
48%
50%
47%
43%
52%
Table 3. The percentage of quality-level-1 reference data to all
points for each flux network.
Table 4 shows the percentage distribution of IGBP classes of
quality level-1 reference data (192 points) for each flux network.
The highest number of the IGBP classes in quality level 1 was
evergreen needleleaf forest (27%) and cropland (19%), followed
by grassland (18%), deciduous broadleaf forest (12%) and
evergreen broadleaf forest (11%). The total amount of five
forest classes rate was 56%. In previous study using DCP
information, the highest number of the IGBP class in quality
level-1 was cropland (55%) , and the total amount of five forest
classes rate was 30%. The cause is that the DCP includes
limited useful vegetation information to produce a complete
reference dataset based on PFT with five specialized forest
classes.
flux
network
ENF
EBF
Ameriflux
11%
2%
AsiaFlux
2%
6%
CCP
6%
IMECC
8%
DNF
2%
DBF
MF
OSH
4%
2%
2%
5%
1%
1%
1
20
9
13
397
10
2
(3%)
(97%)
(2%)
(0%)
Table 2. The reliability of the two interpreters’ labels based on
simple land cover types.
4.3 Quality of reference data for FLUXNET
The number of reference datasets selected based on three
quality-level conditions was 192, 335 and 370 in quality-level
order. About half of total flux sites used in this study satisfied
the condition of quality level 1. Most flux sites (82%) satisfied
the condition of quality level 2. FLUXNET information was
considered to be efficient in interpretation in comparison with
the result using DCP information.
Table 3 shows the percentage of quality level-1 reference data
to all points for each flux network. There is characteristics
information on web site of flux networks except IMECC. In
particular, on the Asiaflux network database on web, there are
useful information for interpretation; domestic species of
Oz_flux
Total
1%
1%
3%
2%
3%
27%
11%
2%
12%
4%
3%
WSA
GRA
WET
CRO
UBR
Total
Ameriflux
7%
1%
8%
AsiaFlux
2%
1%
4%
CCP
1%
1%
IMECC
5%
1%
Oz_flux
1%
3%
Total
1%
18%
37%
1%
7%
7%
27%
1%
3%
22%
19%
7%
1%
Table 4. The percentage distribution of IGBP classes of qualitylevel-1 reference data (192 points) for each flux network.
This contribution has been peer-reviewed.
doi:10.5194/isprsarchives-XLI-B8-1207-2016
1209
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLI-B8, 2016
XXIII ISPRS Congress, 12–19 July 2016, Prague, Czech Republic
4.4 Comparison of results between the MODIS IGBP land
cover products and reference data with quality level
Table 5 shows the validation results of the MODIS IGBP global
land cover products, with the reference datasets selected based
on three quality-level conditions as well as the number of
quality-level reference data. One-pixel columns show the
overall accuracy and the kappa coefficients for One-pixels.
Nine-pixels columns display the result of Nine-pixels. Both the
overall accuracy and kappa coefficients of quality level 1 are the
highest of the three quality-level reference datasets. For quality
level 1, agreement between two interpreters ranged from good
to fair (kappa coefficient = 0.620, 0.725). Accuracy decreases
with the quality level. This outcome is the same for validation
with reference data using DCP information in our previous
study.
The highest number of Nine-points with high levels of
agreement among the IGBP classes in quality level 1 was
evergreen needleleaf forest (27%) and cropland (19%), followed
by grassland (18%), deciduous broadleaf forest (12%) and
evergreen broadleaf forest (11%).
Number of
reference
datasets
Validation
area
Overall
accuracy
Kappa
coefficient
Quality level
1
Quality level
2
Quality level
3
192
335
370
1
pixel
9
pixels
1
pixel
9
pixels
1
pixel
9
pixels
67%
77%
54%
59%
51%
53%
0.62
0.72
0.48
0.54
0.45
0.47
Table 5. The validation results of the MODIS IGBP global land
cover products with quality-level reference data.
Table 6 shows the validation results of the MODIS IGBP global
land cover products with quality-level reference data for each
fluxnet. The agreement result of four fluxnet (Ameriflux,
Asiaflux, IMECC and Oz_flux) ranged from good to fair (kappa
coefficient = 0.80, 0.67, 0.72 and 0.69). Most disagreement
IGBP class between quality level-1 in CCP and the MODIS
IGBP global land cover products were ENF. Most of them were
managed forests in which include clear-cut area.
Fluxnet
Ameriflux
Asiaflux
CCP
IMECC
Oz_flux
Number of Quality
level-1 sites
71
43
14
51
13
Overall accuracy
(kappa coefficients)
83% (0.80)
72% (0.67)
50% (0.29)
78%
77%
(0.72)
(0.69)
Table 6. The validation results of the MODIS IGBP global land
cover products with quality-level reference data for each fluxnet.
5. SUMMARY
Our final aim is to produce high-quality reference data set for
validation of global land cover product in GCOM-C. Validation
needs a reliable reference dataset produced with information
derived from ground truth data. Recently, the amount of ground
truth data derived from information collected by volunteers has
been increasing globally. However, it is difficult to produce
forest reference data in high-quality, since the DCP includes
limited useful vegetation information to produce a complete
reference data of forest classes. In this study, we applied the
method of producing reference data sets with quality-level
information to FLUXNET information. FLUXNET information
was useful especially for forest classes for interpretation.
Acknowledgements :
This study was supported in part by a Global Change
Observation Mission (GCOM; PI No. 114) project at JAXA.
References :
Bontemps, P., Defourny, P., Bogaert, E.V., Arino, O.,
Kalogirou, V. and Perez, J.R. GLOBCOVER 2009 Products
Description
and
Validation
Report,
available
at
http://due.esrin.esa.int/page_globcover.php (20 Sep. 2015).
Degree Confluence Project, available at http://confluence.org/
(20 Sep. 2015).
Foody, G.M. and Boyd, D.S., 2013. Using Volunteered Data in
Land Cover Map Validation: Mapping West African Forests,
IEEE Journal of selected topics in applied earth observations
and remote sensing, Vol. 6, No.3, pp.1939-1440.
Iwao, K., Nishida, K., Kinoshita, T., and Yamadagata, Y., 2006.
Validating land cover maps with Degree Confluence Project
information, Geophysical research letters, vo.33, L23404, doi
10.1029/2006GL027768.
Mayaux, P., Eva, H., Gallego, J., Strahler, A.H., Herold, M.,
Agrawal, S., Naumov, S., Miranda, E.E.De, Bella, C. M.Di,
Ordoyne, C., Kopin, Y., and P. S. Roy., 2006. Validation of the
Global Land Cover 2000 Map, IEEE TRANSACTIONS ON
GEOSCIENCE AND REMOTE SENSING, VOL. 44, NO. 7,
pp.1728-1737.
Olofsson, P., Stehman, S.V., Woodcock, C.E., Friedl, M.A.
Sulla-Menashe, D., Sibley, A.M., Newell, J.D. and M. Herold.,
2012. A global land cover validation data set, I: Fundamental
Design principles, International Journal of Remote Sensing, Vol.
33, pp.5768-5788.
Sasai, T., Saigusa, N., Nasahara, N.K., Ito, A. , Hashimoto, H.,
Nemani, R.R., Hirata, R., Ichii, K., Takagi, K., Saitoh, M., Ohta,
T., Murakami K.,, Yamaguchi Y. and Oikawa, T., 2011.
Satellite-driven estimation of terrestrial carbon flux over Far
East Asia with 1-km grid resolution, Remote Sensing of
Environment, 115, pp.1758-1771.
Soyama, N., Sasai, T., Muramatsu, K., Daigo, M., Ochiai, F.,
Nasahara, K., 2015. Validation Method of Global Land Cover
Map using Reference Data with Quality level : Using the
Degree Confluence Project Information", ISTS 2015
Proceedings, 6pages.
Stehman, S.V., Olofsson, P., Woodcock, C.E., M. Herold, and
M.A. Friedl., 2012. A global land cover validation data set, II:
Augmenting a stratified sampling design to estimate accuracy
This contribution has been peer-reviewed.
doi:10.5194/isprsarchives-XLI-B8-1207-2016
1210
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLI-B8, 2016
XXIII ISPRS Congress, 12–19 July 2016, Prague, Czech Republic
by region and land-cover class, International Journal of Remote
Sensing, 33(22), pp. 6975-6993.
Thanyapraneedkul, J., Muramatsu K, Daigo, D., Furumi, S.,
Soyama, N., Nasahara, K., Muraoka, H., Noda, M.H., Nagai, S.,
Maeda T., Mano, M., and Mizoguchi Y., 2012. A vegetation
index to estimate terrestrial gross primary production capacity
for the GCOM-C/SGLI satellite sensor, Remote Sensing, 4,
pp.3689-3720.
The Ameriflux, available at http://ameriflux.ornl.gov (20 July
2015) and http://ameriflux.lbl.gov (20 July 2015).
The Asiaflux, available at http://asiaflux.net (20 July 2015).
The CCP database, ftp://daac.ornl.gov/data/fluxnet/fluxnet_
canada /data/ (20 July 2015).
The European Fluxes Database Cluster. available
http://gaia.agraria.unitus.it/home/sites-list (20 July 2015).
at
The FLUXNET (Network of regional networks of observations
from micrometeorological tower sites), available at
http://fluxnet.ornl.gov/ (20 July 2015).
The OzFlux, available at http://www.ozflux.org.au (20 July
2015).
Loveland, T.R., Belward, A.S., 1997. The IGBP-DIS global 1
km land cover data set, DISCover: first results. International
Journal of Remote Sensing, Vol. 18, No. 15, pp.3289-3295.
This contribution has been peer-reviewed.
doi:10.5194/isprsarchives-XLI-B8-1207-2016
1211