user-based / cross-device tracking

[gingerlime]

problem

It's quite common today for users to use the same web app across different devices. Users might create an account on the desktop and do a few things. Then login again from their mobile or tablet and continue from where they left, or vice versa.

challenge

This creates a challenge for A/B testing for two main reasons:

1. With an A/B test running, the same user might get a completely different experience when they switch between devices. This is confusing and potentially frustrating for users.
2. Tracking conversions is inaccurate. A user might use variation A on one device, but then convert on another device where variation B is running. The conversion will therefore be wrongly attributed.

user-based or cross-device tracking can solve both those problems, as long as the user can be uniquely identified across all devices. This isn't particularly difficult given that most web apps require logging-in to use the same account.

method

With this in mind, it's possible to make Alephbet support user-based experiments. To do this, Alephbet needs to:

1. Get a user identifier (user_id, email, any identifier that uniquely identifies the user)
2. Use this identifier to:
   a. pick a variant that is psuedo-random, but consistent for the user_id. This can be done with deterministic assignment through hashing, instead of completely random. See planout
   b. generate a pseudo-uuid for goals. When goals are tracked, we already send a uuid. We can generate a pseudo-uuid for the user and experiment, so even if the user completes a goal across different devices, it will only be counted once

limitations

There are a couple of limitations with this approach:

1. Some experiments are designed for visitors, before they register. This approach won't work in those cases. Experiments that employ user-based tracking must target only already-identified users.
2. Not all backends support uuids. Gimel does. And so does Keen.io. Google Analytics internally counts unique events, so I'm not sure if it can work with a custom uuid(?)

Implementation

The implementation needs to change a couple of aspects of Alephbet:

1. Allow users to specify a user_id, and use it to assign users to variants.
2. Pass the user_id, as well as whether or not the goal is unique to the tracking adapters, so those can decide when to pass a consistent uuid (for unique goals), or a random one (for non-unique goals).

This might introduce some breaking changes, although it can be kept to a relative minimum, and most behaviour can be kept backwards-compatible if no user_id is specified (to also support visitor experiments).