Data Analysis Iteration

WEBVTT

1
00:00:01.440 --> 00:00:05.063
This lecture's gonna provide an overview
of the cycle of data analysis.

2
00:00:10.191 --> 00:00:14.060
Data analysis is a complex process
that can involve many pieces and

3
00:00:14.060 --> 00:00:15.790
many different tools.

4
00:00:15.790 --> 00:00:18.480
But fundamentally,
there are only three parts to it.

5
00:00:19.540 --> 00:00:22.180
The first part is setting expectations.

6
00:00:22.180 --> 00:00:24.460
The second part involves
collecting information and

7
00:00:24.460 --> 00:00:26.990
comparing your expectations to data, and

8
00:00:26.990 --> 00:00:32.220
the last part Involves reacting to data,
and revising your expectations.

9
00:00:32.220 --> 00:00:33.750
So that's basically it.

10
00:00:33.750 --> 00:00:37.990
Those are the three parts of data analysis
that you often will cycle through,

11
00:00:37.990 --> 00:00:41.290
many times, in the course of
analyzing any given data set.

12
00:00:41.290 --> 00:00:42.820
So I'm gonna break down each one of these

13
00:00:43.870 --> 00:00:47.190
pieces to give you a little bit of
a description of what they are, and I'll

14
00:00:47.190 --> 00:00:51.690
give you a little example of kinda how
they might be applied in the real world.

15
00:00:51.690 --> 00:00:55.913
So steady expectations involve
deliberately thinking about

16
00:00:55.913 --> 00:00:58.599
what you're gonna do before you do it.

17
00:00:58.599 --> 00:01:02.585
Now, the idea in any part of data analysis
is that everything you do is gonna

18
00:01:02.585 --> 00:01:06.511
have some sort of consequence,
whether it's collecting data, whether

19
00:01:06.511 --> 00:01:11.035
it's fitting a model, whether it's asking
a question or making some sort of plot.

20
00:01:11.035 --> 00:01:15.494
Everything you do there will be some sort
of action, and the point is you wanna

21
00:01:15.494 --> 00:01:19.490
think about what that consequence
is gonna be, before you do it.

22
00:01:19.490 --> 00:01:22.850
And that way you set the expectations for
yourself, and

23
00:01:22.850 --> 00:01:27.980
you can determine whether the reality
kind of meets that expectation.

24
00:01:29.635 --> 00:01:33.165
So that's the first part
of the data analysis cycle.

25
00:01:33.165 --> 00:01:37.685
Once you've set your expectations the next
thing you wanna do is collect some data or

26
00:01:37.685 --> 00:01:42.380
collect some information that will
allow you to compare those expectations

27
00:01:42.380 --> 00:01:43.775
to reality.

28
00:01:43.775 --> 00:01:48.170
And so,
collecting that information is key,

29
00:01:48.170 --> 00:01:51.030
because it will tell you whether or
not your expectations were right.

30
00:01:51.030 --> 00:01:51.930
Whether they were wrong.

31
00:01:51.930 --> 00:01:53.713
Whether they were too high, too low,

32
00:01:53.713 --> 00:01:56.568
whatever it is depending on
the problem you're working on.

33
00:01:56.568 --> 00:02:01.009
And then, once you've collected that
information, and compared it to your

34
00:02:01.009 --> 00:02:05.541
expectations you can react to it, and
maybe change your behavior in some way.

35
00:02:05.541 --> 00:02:09.702
So the last part of the data analysis
cycle is to think about what have

36
00:02:09.702 --> 00:02:14.309
we learned from the data, from our
expectations, and their comparison.

37
00:02:14.309 --> 00:02:16.350
What would we do differently next time.

38
00:02:17.400 --> 00:02:20.870
Did we match our expectations,
did they not match, why or why not.

39
00:02:20.870 --> 00:02:22.443
So that's the third part.

40
00:02:22.443 --> 00:02:27.341
And then, once you've completed the third
part and you've revised your expectations,

41
00:02:27.341 --> 00:02:31.401
you may go back, with these revised
expectations and collect more data and

42
00:02:31.401 --> 00:02:35.267
try to match them again, and
then this iteration continues, often for

43
00:02:35.267 --> 00:02:37.990
many different times in
any given data analysis.

44
00:02:40.470 --> 00:02:44.830
So I just wanna give you a quick
example of how you can use these

45
00:02:44.830 --> 00:02:50.190
three components in a kind of generic or
kind of commonplace setting.

46
00:02:50.190 --> 00:02:53.360
So the basic example I'm gonna present
here is going out to dinner with

47
00:02:53.360 --> 00:02:54.510
your friends.

48
00:02:54.510 --> 00:02:56.490
So suppose you're going out to dinner and

49
00:02:56.490 --> 00:02:59.140
the restaurant you're going
to is a cash only place.

50
00:02:59.140 --> 00:03:02.410
So the question you have to ask yourself
is how much money should you bring.

51
00:03:04.005 --> 00:03:09.235
And the basic activity you're gonna
engage in is eating a meal, and you're

52
00:03:09.235 --> 00:03:13.520
gonna check for the bill, and you're gonna
have to pay, money to pay for the meal.

53
00:03:13.520 --> 00:03:17.869
But before you do that, you gotta figure
out how much money to bring, and so

54
00:03:17.869 --> 00:03:22.585
you have to figure out well, what's your
expectation for the cost of this meal.

55
00:03:22.585 --> 00:03:25.299
Maybe you've dined at this
restaurant all the time, so

56
00:03:25.299 --> 00:03:27.448
you know exactly how much it's gonna cost.

57
00:03:27.448 --> 00:03:32.487
Maybe you know, well in this city, the
typical meal costs this many dollars, and

58
00:03:32.487 --> 00:03:37.403
so I'll just bring that much money,
cuz this is an average kind of restaurant.

59
00:03:37.403 --> 00:03:38.260
Maybe you know,

60
00:03:38.260 --> 00:03:42.065
well the most expensive restaurant in
this city costs this many dollars.

61
00:03:42.065 --> 00:03:44.936
So I know it's not gonna
cost this more than that, so

62
00:03:44.936 --> 00:03:49.180
I'll just bring that to kind of serve as
an upper bound on how much money I might

63
00:03:49.180 --> 00:03:51.223
end up spending at this restaurant.

64
00:03:51.223 --> 00:03:53.365
You might ask your friends,
if they've been their before,

65
00:03:53.365 --> 00:03:54.650
how much does this place cost.

66
00:03:54.650 --> 00:03:56.060
Or you might Google the restaurant and

67
00:03:56.060 --> 00:03:59.430
maybe look up the menu to see what
the meal typically costs there.

68
00:03:59.430 --> 00:04:04.388
At any rate, before you've gone to the
restaurant and eat the meal, you can use

69
00:04:04.388 --> 00:04:08.310
any sort of opreory information
to set up your expectations for

70
00:04:08.310 --> 00:04:10.684
what the cost is ultimately gonna be.

71
00:04:10.684 --> 00:04:13.220
Before you observe the real thing.

72
00:04:14.430 --> 00:04:19.176
So once you've set your expectations, you
can figure out how much money to bring.

73
00:04:19.176 --> 00:04:22.339
The actual collecting of the data
involves going to the restaurant and

74
00:04:22.339 --> 00:04:23.255
getting the check.

75
00:04:23.255 --> 00:04:25.839
So once you've gotten the check,

76
00:04:25.839 --> 00:04:29.500
you observed the reality
of what the meal costs.

77
00:04:30.640 --> 00:04:32.560
And there's two possibilities.

78
00:04:32.560 --> 00:04:35.250
One is that,
that cost meets your expectation.

79
00:04:35.250 --> 00:04:38.510
So suppose you thought
it was gonna be $30, and

80
00:04:38.510 --> 00:04:40.710
it ended up being $30, then that's great.

81
00:04:40.710 --> 00:04:41.630
You know exactly,

82
00:04:41.630 --> 00:04:45.810
you brought the right amount of money,
and then you can pay for the meal.

83
00:04:45.810 --> 00:04:49.696
The other possibility is that
the expectations don't match of what

84
00:04:49.696 --> 00:04:50.657
the reality is.

85
00:04:50.657 --> 00:04:55.386
So thought it was $30 and
it ended up being $40.

86
00:04:55.386 --> 00:04:59.120
And so, you have to ask yourself
then why do you have that mismatch.

87
00:04:59.120 --> 00:05:04.239
Why is it that you thought it was $30 and
the meal turned out to be $40.

88
00:05:04.239 --> 00:05:05.630
So there's two possibilities.

89
00:05:05.630 --> 00:05:08.000
One is that your expectations were wrong.

90
00:05:08.000 --> 00:05:11.820
So you thought that the restaurant
was cheaper than it actually was.

91
00:05:11.820 --> 00:05:15.393
Another possibility is that there's
something wrong with the data, for

92
00:05:15.393 --> 00:05:15.920
example.

93
00:05:15.920 --> 00:05:19.192
It's possible that they added up the check
wrong, maybe they charged you for

94
00:05:19.192 --> 00:05:21.169
something for
that you didn't actually eat.

95
00:05:21.169 --> 00:05:24.835
So you can look at the check to see if
there is a problem with the data that you

96
00:05:24.835 --> 00:05:25.500
collected.

97
00:05:29.040 --> 00:05:34.040
One thing to note about
this example is that it was

98
00:05:34.040 --> 00:05:39.450
easy to know whether your expectations
were matched with the data or not.

99
00:05:39.450 --> 00:05:43.892
So for example, if your expectation was
the meal would cost $30, and then it

100
00:05:43.892 --> 00:05:48.691
actually cost $40, you know immediately
that your expectations were not right.

101
00:05:48.691 --> 00:05:52.200
The meal was $10 more than you
actually thought it was gonna be.

102
00:05:52.200 --> 00:05:55.249
And so,
you can make that conclusion very quickly.

103
00:05:55.249 --> 00:05:57.298
Another possibility, for example,

104
00:05:57.298 --> 00:06:01.405
is that you could've said well
the meal being between 0 and $1,000.

105
00:06:01.405 --> 00:06:05.885
And so, when the data actually comes
in and you see the check is $40 then it

106
00:06:05.885 --> 00:06:10.930
actually matches your expectation which
is that it's between 0 and $1,000.

107
00:06:10.930 --> 00:06:15.830
But because your original
expectation was so diffused, and

108
00:06:15.830 --> 00:06:20.535
so kind of general,
you don't really learn that much from

109
00:06:20.535 --> 00:06:25.650
collecting the data given your
very diffused expectation.

110
00:06:25.650 --> 00:06:30.160
So this brings us to an important point
which is that it's important to have

111
00:06:30.160 --> 00:06:32.370
a very sharp expectation or

112
00:06:32.370 --> 00:06:36.180
a sharp hypothesis about what
you're trying to investigate.

113
00:06:36.180 --> 00:06:38.940
When I said that I expected
the meal to be $30,

114
00:06:38.940 --> 00:06:44.210
it was very easy to know when
my expectations were not met.

115
00:06:44.210 --> 00:06:48.575
But if my expectation was very diffused
and not sharp at all, like between 0 and

116
00:06:48.575 --> 00:06:52.880
1,000, then, collecting the data
doesn't really help you.

117
00:06:52.880 --> 00:06:57.430
Or it doesn't help you learn the process
you're trying to study or in this case,

118
00:06:57.430 --> 00:06:59.360
the cost of the meal at this place.

119
00:06:59.360 --> 00:07:02.600
So ultimately,
what we're leaning toward with

120
00:07:02.600 --> 00:07:06.820
setting your expectations in collecting
data is called a change in behavior or

121
00:07:06.820 --> 00:07:09.620
an understanding of the mechanism
you're trying to study.

122
00:07:09.620 --> 00:07:13.370
What did we learn, and
what would you do differently next time?

123
00:07:13.370 --> 00:07:16.465
So in this scenario where you
thought it was gonna be $30 and

124
00:07:16.465 --> 00:07:20.589
it ended up being $40, well then the next
time you might bring an extra $10.

125
00:07:20.589 --> 00:07:23.211
If you originally thought it
was gonna be between 0 and

126
00:07:23.211 --> 00:07:27.429
$1,000 then the cost ended up being $40,
it's not clear that you would change

127
00:07:27.429 --> 00:07:29.992
anything about your behaviour
based on this data.

128
00:07:29.992 --> 00:07:35.490
And so, if there is no change
in what you might think or

129
00:07:35.490 --> 00:07:38.950
what you might do based on
the collection of the data and

130
00:07:38.950 --> 00:07:43.140
matching it with your expectations,
then that's often a sign that

131
00:07:43.140 --> 00:07:46.740
either the evidence from your experiment
is not very strong or the data analysis

132
00:07:46.740 --> 00:07:51.420
was not able to generate enough evidence,
or there may be some other problem.

133
00:07:51.420 --> 00:07:55.890
With your study or
your data analysis process.

134
00:07:55.890 --> 00:08:00.130
So setting the right expectations and
making them as sharp as possible

135
00:08:00.130 --> 00:08:03.730
is a really key element to this
whole data analysis cycle.