Instance: specific type of example Thing to be classified, associated, or clustered Individual, independent example of target concept Characterized by a predetermined set of attributes Input to learning scheme: set of instancesdataset Represented as a sin
1 Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 03/04/06
Example problems: weather data, contact lenses, irises, labor negotiations ✁ Classification learning is supervised
Classification learning: predicting a discrete class
♦
Association learning: detecting associations between features
♦
Clustering: grouping similar instances into clusters
♦ Numeric prediction:
✁ predicting a numeric quantity Concept: thing to be learned ✁ Concept description: output of learning scheme
03/04/06 5 Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) Classification learning ✁
♦
Styles of learning:
Scheme is provided with actual outcome ✁ Outcome is called the class of the example ✁ Measure success on fresh data for which class labels are known (test data) ✁
In practice success is often measured subjectively
03/04/06 6 Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) Association learning ✁
Can be applied if no class is specified and any kind of structure is considered “interesting” ✁ Difference to classification learning:
♦
Can predict any attribute’s value, not just the class, and more than one attribute’s value at a time
♦
Hence: far more association rules than classification rules
♦
What’s a concept? ✁
Data Mining
What’s in an attribute?
Practical Machine Learning Tools and Techniques
Slides for Chapter 2 of Data Mining by I. H. Witten and E. Frank
2 Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 03/04/06
Input:
Concepts, instances, attributes Terminology What’s a concept?
♦ Classification, association, clustering, numeric prediction
What’s in an example?
♦ Relations, flat files, recursion
♦ Nominal, ordinal, interval, ratio
We will focus on nominal and numeric ones 03/04/06 4 Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2)
Preparing the input
♦ ARFF, attributes, missing values, getting to know data
03/04/06 3 Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) Terminology ✁
Components of the input:
♦
Concepts: kinds of things that can be learned ✂
Aim: intelligible and operational concept description ♦
Instances: the individual, independent examples of a concept ✂
Note: more complicated forms of input are possible ♦ Attributes: measuring aspects of an instance ✂
♦ Thus: constraints are necessary ✂ Minimum coverage and minimum accuracy
Success often measured subjectively
7 Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 03/04/06
Peter M Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 1 1 03/04/06
Graham M Pam F
Grace F Ray M
= Ian M
Pip pa F Brian M
= Anna F
Nikk i F Peggy F
Peggy Peter Male Graham Peggy Peter Male Steven ? ? Female Peggy ? ? Male Peter
Ian Pam Female Nikki Ian Pam Female Anna Ray Grace Male Brian Ray Grace Female Pippa Ray Grace Male Ian Peggy Peter Female Pam
Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 1 0 03/04/06 A family tree
Parent1 parent2 Gender Nam e Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 1 2 03/04/06
Anna yes Nikki … … … Yes Nikki Anna … … … Yes Pippa Ian … … … Yes Pam Steven No Graham Steven No Peter Steven … … … No Steven Peter No Peggy Peter
Sister of? Second person
First person No All the rest Yes Anna Nikki
Yes Nikki Anna Yes Pippa Brian Yes Pippa Ian Yes Pam Graham Yes Pam Steven Sister of? Second person First person
Closed-world assumption
= St even M
Most common form in practical data mining
Clustering ✁
8 Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 03/04/06
Finding groups of items that are similar ✁ Clustering is unsupervised
♦
The class of an example is not known ✁
… … … Iris virginica 1 .9 5 .1 2 .7 5 .8 10 2 10 1
52
51
2
Numeric prediction ✁
Rather restricted form of input ✁ No relationships between objects
Variant of classification learning where “class” is numeric (also called “regression”) ✁ Learning is supervised
♦
Scheme is being provided with target value ✁ Measure success on test data
… … … … … Normal 40 False Mild Rainy High 55 False Hot Overcast
True High Hot Sunny High 5 False Hot Sunny Play- time Windy Humidity Temperature Outlook
03/04/06 9 Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) What’s in an example?
Instance: specific type of example ✁ Thing to be classified, associated, or clustered ✁ Individual, independent example of target concept ✁ Characterized by a predetermined set of attributes
Input to learning scheme: set of instances/dataset ✁ Represented as a single relation/flat file
Family tree represented as a table
The “sister-of” relation
A full representation in one table Generating a flat file ✁ First person Second person Sister
Process of flattening called “denormalization”
of ? Name Gender Parent 1 Parent 2 Name Gender Parent 1 Parent 2 ♦ Several relations are joined together to make one
St even Male Pet er Peggy Pam Fem ale Pet er Peggy Yes ✁
Possible with any finite set of finite relations
Graham Male Pet er Peggy Pam Fem ale Pet er Peggy Yes ✁ Ian Male Grace Ray Pippa Fem ale Grace Ray Yes
Problematic: relationships without pre-
Brian Male Grace Ray Pippa Fem ale Grace Ray Yes
specified number of objects
Anna Fem ale Pam Ian Nik ki Fem ale Pam Ian Yes Nik ki Fem ale Pam Ian Anna Fem ale Pam Ian Yes ♦ Example: concept of nuclear-family
No ✁
Al l t he restDenormalization may produce spurious regularities that reflect structure of database
If second person’s gender = female ♦
Example: “supplier” predicts “supplier address”
and first person’s parent = second person’s parent then sister-of = yes Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 1 3 Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 1 4 03/04/06
03/04/06 The “ancestor-of” relation Recursion ✁
Infinite relations require recursion
First person Second person Ancestor of? Nam e Gender Parent Parent Nam e Gender Parent Parent 2
1
2
1 If person1 is a parent of person2
Pet er Male ? ? Steven Male Pet er Peggy Yes then person1 is an ancestor of person2Pet er Male ? ? Pam Femal e Pet er Peggy Yes Pet er Male ? ? Anna Femal e Pam Ian Yes If person1 is a parent of person2
Pet er Male ? ? Nikk i Femal e Pam Ian Yes and person2 is an ancestor of person3
Pam Femal e Pet er Peggy Nikk i Femal e Pam Ian Yes then person1 is an ancestor of person3
Grace Femal e ? ? Ian Male Grace Ray Yes Grace Femal e ? ? Nikk i Femal e Pam Ian Yes ✁ Other positive ex am ples here Yes
Appropriate techniques are known as
All the rest No
“inductive logic programming”
♦
(e.g. Quinlan’s FOIL)
♦ Problems: (a) noise and (b) computational complexity
03/04/06 Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 1 5 03/04/06 Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 1 6 ✁ ✁ What’s in an attribute? Nominal quantities
Each instance is described by a fixed Values are distinct symbols ✁ predefined set of features, its “attributes” ♦ Values themselves serve only as labels or names
♦ Nominal comes from the Latin word for name
But: number of attributes may vary in practice ✁
♦
Possible solution: “irrelevant value” flag ✁ Example: attribute “outlook” from weather data
♦
Values: “sunny”,”overcast”, and “rainy”
Related problem: existence of an attribute ✁ may depend of value of another one ✁ No relation is implied among nominal values (no ordering or distance measure)
Possible attribute types (“levels of ✁ measurement”): Only equality tests can be performed
♦ Nominal, ordinal, interval and ratio 03/04/06 Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 1 7 03/04/06 Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 1 8
Interval quantities ✁
♦
continuity
Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 2 3 03/04/06 Metadata ✁
Information about the data that encodes background knowledge ✁ Can be used to restrict search space ✁ Examples:
♦
Dimensional considerations (i.e. expressions must be dimensionally correct)
♦
Circular orderings (e.g. degrees in compass)
Partial orderings (e.g. generalization/specialization relations)
Special case: dichotomy (“boolean” attribute) ✁ Ordinal attributes are called “numeric”, or “continuous”
Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 2 4 03/04/06 Preparing the input ✁
Denormalization is not the only issue ✁ Problem: different data sources (e.g. sales department, customer billing department, …)
♦
Differences: styles of record keeping, conventions, time periods, data aggregation, primary keys, errors
♦
Data must be assembled, integrated, cleaned up
♦
“Data warehouse”: consistent point of access ✁ External data may be required (“overlay data”) ✁ Critical: type and level of data aggregation
♦ But: “continuous” implies mathematical
Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 1 9 03/04/06
Ordinal quantities ✁
Zero point is not defined!
Impose order on values ✁ But: no distance between values defined ✁ Example: attribute “temperature” in weather data
♦
Values: “hot” > “mild” > “cool” ✁ Note: addition and subtraction don’t make sense ✁ Example rule: temperature < hot ⇒ play = yes ✁
Distinction between nominal and ordinal not always clear (e.g. attribute “outlook”)
20 Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 03/04/06
Interval quantities are not only ordered but measured in fixed and equal units ✁ Example 1: attribute “temperature” expressed in degrees Fahrenheit ✁
Example 2: attribute “year” ✁ Difference of two values makes sense ✁ Sum or product doesn’t make sense
♦
Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 2 1 03/04/06 Ratio quantities ✁
Most schemes accommodate just two levels of measurement: nominal and ordinal ✁ Nominal attributes are also called “categorical”, ”enumerated”, or “discrete”
Ratio quantities are ones for which the measurement scheme defines a zero point ✁ Example: attribute “distance”
♦
Distance between an object and itself is zero ✁ Ratio quantities are treated as real numbers
♦
All mathematical operations are allowed ✁ But: is there an “inherently” defined zero point?
♦ Answer depends on scientific knowledge (e.g.
Fahrenheit knew no lower limit to temperature)
03/04/06 22 Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) Attribute types used in practice ✁
♦ But: “enumerated” and “discrete” imply order ✁ The ARFF format Additional attribute types %
ARFF supports string attributes: % ARFF file for weather data with some numeric features @attribute description string % @relation weather
♦ @attribute outlook {sunny, overcast, rainy} Similar to nominal attributes but list of values @attribute temperature numeric
is not pre-specified
@attribute humidity numeric @attribute windy {true, false} It also supports date attributes:
@attribute play? {yes, no} @attribute today date @data sunny, 85, 85, false, no
♦
Uses the ISO-8601 combined date and time
sunny, 80, 90, true, no overcast, 83, 86, false, yes
format yyyy-MM-dd-THH:mm:ss ...
Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 2 5 Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2)
26 03/04/06
03/04/06 Sparse data
Attribute types In some applications most attribute values in ✁
Interpretation of attribute types in ARFF
a dataset are zero
depends on learning scheme
♦
E.g.: word counts in a text categorization problem
♦
Numeric attributes are interpreted as ✂
ARFF supports sparse data ✂ ordinal scales if less-than and greater-than are used ratio scales if distance calculations are performed
0, 26, 0, 0, 0 ,0, 63, 0, 0, 0, “class A” (normalization/standardization may be required)
0, 0, 0, 42, 0, 0, 0, 0, 0, 0, “class B” ♦
Instance-based schemes define distance between ✁ nominal values (0 if values are equal, 1 otherwise)
{1 26, 6 63, 10 “class A”}
Integers in some given data file: nominal,
{3 42, 10 “class B”}
ordinal, or ratio scale?
This also works for nominal attributes (where the first value corresponds to “zero”) 03/04/06 Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 2 7 03/04/06 Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 2 8
Nominal vs. ordinal Missing values ✁
Attribute “age” nominal Frequently indicated by out-of-range entries
♦
Types: unknown, unrecorded, irrelevant
If age = young and astigmatic = no ♦ and tear production rate = normal Reasons: ✂ then recommendation = soft ✂ malfunctioning equipment
If age = pre-presbyopic and astigmatic = no changes in experimental design and tear production rate = normal ✂ collation of different datasets then recommendation = soft ✂ measurement not possible
Attribute “age” ordinal ✁ Missing value may have significance in itself
(e.g. “young” < “pre-presbyopic” < “presbyopic”) (e.g. missing test in a medical examination)
♦
Most schemes assume that is not the case:
If age ≤ pre-presbyopic and astigmatic = no and tear production rate = normal
“missing” may need to be coded as additional value
then recommendation = soft 03/04/06 Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 2 9 03/04/06 Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 3 0
✁ Inaccurate values Getting to know the data ✁
Reason: data has not been collected for mining it ✁ Simple visualization tools are very useful Result: errors and omissions that don’t affect
♦ Nominal attributes: histograms (Distribution
original purpose of data (e.g. age of customer) ✁ consistent with background knowledge?) Typographical errors in nominal attributes ⇒
♦
Numeric attributes: graphs values need to be checked for consistency ✁ ✁ (Any obvious outliers?) Typographical and measurement errors in
2-D and 3-D plots show dependencies numeric attributes outliers need to be identified ✁ ⇒ ✁ Need to consult domain experts ✁ Errors may be deliberate (e.g. wrong zip codes) ✁ Too much data to inspect? Take a sample!
Other problems: duplicates, stale data
Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 3 1 Data Mining: Practical Machine Learning Tools and Techniques (Chapter 2) 3 2 03/04/06 03/04/06