User testing with Eye tracking DO give you a deeper insight of your users, it enables you to see what happens – when nothing happens. Here follows a comparison of the new tracker from Eye Tribe, and how it stands towards the established one from Tobii.
Edit 2017: Eyetribe no longer supports its eye proof software and have stopped the development of their remote trackers “Unfortunately, we’ve decided to go in a different direction with our technology and will stop development of our products. We thought you should hear this news directly from us. We thank you for the time you’ve spent in discussions.
-The Eye Tribe Team “
I (Patrik) use eye tracking as an central part in my tool stack, when I perform CRO (conversion rate optimization) for clients, therefore I jump up and down as soon there is a new tool out there trying to compete with established brands. So, when Eye Tribe started to offer their super cheap dev kit tracker, I was glad to pick it up and evaluate how it compares to the Tobii tracker we usually use for our testing.
How does Eye Tribes ET1000 stand against Tobii X2-30 Compact?
The Tobii tracker is the one we usually use for testing. Can Eye Tribe be utilized in the same form of lean user testing that we usually preform? (Basically going out and meeting the target audience were they happen to be, and running the scenarios there on a laptop. Rather than inviting “expert testers” from a pool to come to our office for lab sessions).
Disclamer:
I work with the Tobii compact during my usual testing, so it serves as the reference point for how I judge the Eye Tribe Tracker. This is undoubtedly affects what I expect from a package, and had I not started out using a product in a different price category, I might have had other expectations.
This review reflects the state of the trackers and their software at the time of writing, and it is subject to change since both Tobii and the Eye Tribe continually develops their products. Also worth noting, is that the Eye Tribe tracker is meant for internal evaluation and development. If you intend to use it for commercial applications, you will have to acquire a commercial license.
Nice wrapping,
– but is it sufficient?
Superficially both eye trackers are relatively comparable, the eye tribe is slimmer than the Tobii but slightly longer. It has a slick metal casing that definitely is more visually pleasing than the Tobii one.
However, when doing testing what you want is for the tracker to be discrete, since you don’t want the user to be reminded that they are being tracked, and depending on how your computer looks the different designs blend to different degrees. A definite minus for the Eye Tribe when it comes to discreteness, is that the 6 bright lights on that tracker are way more visible than the two bigger dim ones on the Tobii.
Tre legs vs magnets – The setup fight
Booth of them connect via USB even though the Eye Tribe can only connect via USB 3, but admittedly that should hardly be a problem for modern computers.
Picture of the standard test setup with the tobii and one with the eye tribe for reference (actual testing was not done like this with the eye tribe for obvious practical reasons).
Were the Tobii stands out in the practicality department, is the magnet mounting combined with their fastening sticker plates. Which is something I prefer compared to the Eye Tribe, that comes with a tripod you position in front of the computer and then aim towards the users eyes.
The Eye Tribe is not suitable for laptops, as it is designed today
The magnet solution means that out of the box the Eye Tribe is not suitable for laptops, even though a handy person should be able to fabricate a mount with relative ease, or use the special mounts they offer for Microsoft surface pro 1/2 or Surface pro 3, if that is what you want to test on.
Set up of Eye Tribe with large, and small sceen.
Worth to note if you attempt to fabricate your own laptop mount it might be good to allow it to still be directed somewhat after mounting, since it is less forgiving than the Tobii when it comes to users with a head position outside of its desired “center of tracking”.
Calibration and tracking
This is an important area were you really notice the difference in price class between the trackers. The basic premise is the same, even though the Eye Tribes UI is a bit reluctant to give you a good way to review your calibration.
Tip!
A handy workaround I noticed is to go into options, there you can turn of mouse cursor stabilization and then turn on the mouse cursor control to get a raw data evaluation
Despite calibrating the Eye Tribe on a bigger screen compared to the Tobii tracker (something that should make measurement easier for the Eye Tribe tracker) the data is noticeably less precise, and we had trouble getting proper calibrations for some of the people at the office. This could of course come down to bad luck, but it is definitely somewhat related to the fact that the Eye Tribe has more trouble compensating for users that move their head when being tracked.
Record longer sessions
As a session goes on the Eye Tribe has a tendency to track progressively worse, unless users focus on sitting perfectly still and not moving their head. This makes for an unnatural testing environment as users tends to shift positions over time and with changing moods.
Software Eyeproof
For analysis of data the Eye Tribe currently have a beta version of their cloud based analysis tool Eyeproof. It is very easy to get started with, and it has the functions you expect for qualitative analysis. Eyeproof does lacks some of the advanced controls of Tobii studio, though one nice thing worthy of note, is the ability to in app overlay different data visualizations at the same time (With Tobii studio you would have accomplish the same thing externally using the export function).
Cloud based software Eyeproof.
However there is one big deal breaker with Eyeproof if I were to use the program for the type of quick iteration lean user tests we usually perform… It totally lacks functionality for qualitative analysis and qualitative testing. So you can’t view recordings of sessions together with webcam recordings of user interactions, and it lacks Tobii studios handy functionality to support a retrospective think aloud methodology during testing. And realistically qualitative evaluation is often exactly what you should be preforming considering the sample size of most users testing done in conversion optimization.
Software Tobii Studio.
How does the future look for Eyeproof compared to Tobii studio then?
Tobii studios new beta version for their eye tracking glasses, provides an even better work environment for testing in the field, and we can expect that to eventually spill over into the regular version. So we have an indication were that is going. For Eyeproff its harder to assert, due to the fact that Eyeproof is a cloud service I doubt they will be able to properly support qualitative data handling well in the future. Considering the file sizes associated with the videos recorded during testing, transfer could become time consuming, and you want to be able to work on videos with overlaid gaze data while doing the analysis.
At the present when evaluating the software the software advantage clearly goes to Tobii studio. It is sometimes slow when handling the large recordings, but it can still it can still do it. That being said, we have had some issues with studio when pushing the limits of the program, but Tobii support has been very helpful trying to solve problems that have arisen.
I can only assume both applications will get better with time, but as it stands now with quantitative eye tracking generally being something reserved for when I specifically need statistical data about where users look in a specific context. Compared to the much leaner qualitative testing that is a flexible and quick tool in the CRO toolbox, there is really no comparison to what I can get out of the different supporting software.
Eye Tribe or Tobii?
– Summery
As it currently stands the only way I could envision using the Eye Tribe for conversion optimization, is in a lab environment rather that “out in the field” and only if I specifically needed statistical gaze data for short timespan interactions (since accuracy might drop to low if users don’t sit still which is next to impossible during a full length scenario).
But seriously if I could make a wish about user testing with eye tracking within the CRO context, it’s that people stop trying to use gaze heatmaps (often created from too few users and with no scenario and task as a context) and start using eye tracking as a very informative data point analysed in conjunction with all the other contextualized data gathered using when it as a natural evolution of traditional user testing.
In short there is no way I could do my job using if I replaced the Tobii with the Eye tribe, they are in totally different price classes but so far there is a reason for that. What will the future bring? I don’t know that, but these are interesting times for eye tracking and CRO and I will keep you updated on my views of what is happening.
