revise metrics.md for 54 launch #2452
Conversation
|
|
||
| ### Opt-out | ||
| 0. Metrics to tell us if Firefox Screenshots is affecting Firefox. For instance, do Firefox Screenshots users engage longer with Firefox? |
There was a problem hiding this comment.
We did have a talk with Brendan Colloran about this, and he was pretty confident that we couldn't find any meaningful way to understand this, except through controlled experiment. That is, Firefox Screenshots users are going to use Firefox more, because people who use any feature are more active than the general population.
The one option – which I'd like us to consider at some point – is to actually do a controlled experiment by leaving some set of users with Screenshots disabled. And maybe we could see some differences for the brief period when we are rolling out Screenshots (but probably not).
There was a problem hiding this comment.
We've been talking about that. It sounds like you don't need to leave the set of users with the feature disabled for a super long time to get a decent result (longer is better, of course, but hey...). If our rollout lasts a couple weeks, that's already useful data. We might be able to keep a small portion out longer, but that schedule isn't finalized.
docs/METRICS.md
Outdated
|
|
||
| This information is intended to help us make Firefox Screenshots better. | ||
| No metrics data is sent to any servers until a user interacts with Page Shot. |
|
|
||
| ### Opt-out | ||
|
|
||
| The add-on reads the Telemetry opt-out preference (`toolkit.telemetry.enabled`) before sending any metrics data to the servers. If this preference is false, or if there is any issue trying to fetch the preference, then no data is sent. |
There was a problem hiding this comment.
We could be more specific that the add-on checks the value of that preference each time you click the Screenshots button.
There was a problem hiding this comment.
I thought there might be other entry points. Like, if a person hits the keyboard combo or if we eventually provide some other link (a my shots button?)
There was a problem hiding this comment.
It's slightly ad hoc, but at each entry point we try to run analytics.refreshTelemetryPref() (it ended up this way because each entry point also has its own event, so we want to refresh the pref before sending the first event). But yes, we try to do it at the beginning of each session.
docs/METRICS.md
Outdated
| The add-on does not communicate directly with GA, instead it POSTs an event to the Firefox Screenshots server which, in turn, sends the event to GA. This model allows us to only send the data we need for analysis instead of the more comprehensive data collection that google-analytics.js does. Besides the event information specified below we also send the User-Agent string, and the add-on provides: | ||
| * Any A/B tests the user is in, or if the user is in the control for a test | ||
| * The application (“firefox”) | ||
| * The Page Shot add-on version (“6.0.0”) |
|
Do we need anything about Sentry in here? |
|
None of the other metrics docs talk about sentry, so I don't think so |
|
thanks |
ghost
merged commit 
r? @johngruen and anyone else
@ianb I noticed a bunch of checkboxes are struck through. Should I just whack those lines?