Precision at the end of section 14.5

[jcornickm]

The first sentence in the next to last paragraph states: "The regression line is therefore much more stable than the conditional mean". This is not quite correct. The regression line is a way of computing the (linear) conditional expected value. This is important to avoid the rather frequent mistake of assuming that multivariate regression can be used to prover causation, rather than (linear) conditional expectations. Goldberg (A Course in Econometrics) makes this point more clearly than any other text I have consulted (not that I have consulted many econometric textbooks).

[rafalab]

I can change it to "The prediction obtained with the regression line is therefore much more stable than the prediction obtained using the conditional mean." . Do you think this avoids the confusion? Note that the next paragraph explains that lower variance is not always preferable (due to potential bias):

So why not always use the regression line for prediction? Because it is not always appropriate. For example, Anscombe provided cases for which the data does not have a linear relationship.

Happy to hear your thoughts before making an edit.

[jcornickm]

Hello Rafael. First, yes, I think the proposed new text works better. After I wrote to you, I realized that 1. In Section 14.6 it becomes clear that regressions results are conditional expected values 2. I do not have a clear terminology to distinguish the “expected (simple) conditional mean” and the “linear conditional mean” given by the regression line. My guess is that this is confusing for me only because thirty years ago I knew a lot of econometrics, which I am now trying to relearn/update, and that for most readers the point is mute and, as I said, the new text works just fine. Now, since I have your virtual “ear”, let me share with you, for whatever is worth, my experience with IDS. First time I worked through it, I found it confusing, mostly because it seemed to me that many R commands were used without previous introduction or explanation, so sometimes I understood “this gist” of the code, but not the details. Then, I worked my way through most of Hadley’s R for Data Science, which is more about DS than it is about statistics. And now, I am working through IDS again (what can I say? I am slow learner) and I find it extremely useful. And on this second pass, I frequently go to the source code, to understand how you do things. So, what I am saying is that maybe, just maybe (because I have no way of knowing if my experience is representative) there is something that can be improved in terms of the sequencing / explanations of R concepts and commands. This is extremely difficult because, in my experience teaching economics, after many years working on a subject, there is a whole bunch of stuff that becomes second nature and seems obvious, and it is hard to put oneself in the shoes of someone who is completely new to the material. There! That’s my two cents. This is already a bit long, but I would not like to sign off without also saying the I find the second edition much improved (better organized) than the first…so I will be looking forward to the third, fourth…etc. Saludos ***@***.*** From: Rafael A Irizarry ***@***.***> Sent: Monday, September 30, 2024 3:01 PM To: rafalab/dsbook-part-2 ***@***.***> Cc: Jorge Cornick ***@***.***>; Author ***@***.***> Subject: Re: [rafalab/dsbook-part-2] Precision at the end of section 14.5 (Issue #14) I can change it to "The prediction obtained the regression line is therefore much more stable than the prediction obtained using the conditional mean." . Do you think this avoids the confusion? Note that the next paragraph explains that lower variance is not always preferable (due to potential bias): So why not always use the regression for prediction? Because it is not always appropriate. For example, Anscombe provided cases for which the data does not have a linear relationship. Happy to hear your thoughts before making an edit. — Reply to this email directly, view it on GitHub<#14 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ATZRKMARJG3XROFMPY4WWZLZZG3XZAVCNFSM6AAAAABPB7FIKOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBUGEZTMMBZGE>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

[rafalab]

Thank you for the feedback! It's timely as I'm currently working on the second edition.

As you probably noticed, we've split the original book into two parts. The second edition of the first part is already published, and I believe the R explanations are clearer there. In the second part, we aim to focus more on statistical thinking rather than R code. It might be a good idea to be more explicit about the need for prior knowledge of R to fully benefit from it. Any further thoughts appreciated.

[jcornickm]

Well, exactly. This makes a lot of sense to me From: Rafael A Irizarry ***@***.***> Sent: Tuesday, October 1, 2024 3:27 PM To: rafalab/dsbook-part-2 ***@***.***> Cc: Jorge Cornick ***@***.***>; Author ***@***.***> Subject: Re: [rafalab/dsbook-part-2] Precision at the end of section 14.5 (Issue #14) Thank you for the feedback! It's timely as I'm currently working on the second edition. As you probably noticed, we've split the original book into two parts. The second edition of the first part is already published, and I believe the R explanations are clearer there. In the second part, we aim to focus more on statistical thinking rather than R code. It might be a good idea to be more explicit about the need for prior knowledge of R to fully benefit from it. Any further thoughts appreciated. — Reply to this email directly, view it on GitHub<#14 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ATZRKMAU6DNHTMEHJJHZYHTZZMHSJAVCNFSM6AAAAABPB7FIKOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBXGEYDKOBRGM>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>