Weather Web Maps

Weather is one of the things that interest almost everyone. Weather maps are therefore widely used and many users use them in everyday life. To identify the potential usability problems of weather web maps, the presented research was conducted. Five weather maps were selected for an eye-tracking experiment based on the results of an online questionnaire: DarkSky, In-Počasí, Windy, YR.no, and Wundermap. The experiment was conducted with 34 respondents and consisted of introductory, dynamic, and static sections. A qualitative and quantitative analysis of recorded data was performed together with a think-aloud protocol. The main part of the paper describes the results of the eye-tracking experiment and the implemented research, which identify the strengths and weaknesses of the evaluated weather web maps and point out the differences between strategies in using maps by the respondents. The results include findings such as the following: users worked with web maps in the simplest form and they did not look for hidden functions in the menu or attempt to find any advanced functionality; if expandable control panels were available, the respondents only looked at them after they had examined other elements; map interactivity was not an obstacle unless it contained too much information or options to choose from; searching was quicker in static menus that respondents did not have to switch on or off; the graphic design significantly influenced respondents and their work with the web maps. The results of the work may be useful for further scientific research on weather web maps and related user issues.

Methods

Procedure

The SMI Experiment Center™ software was used to design the experiment. The eye-tracking test was divided into introductory, static and dynamic sections. 

The static section of the test also provided three rounds of questions and was designed to take no more than ten minutes. In static testing, the respondent was prevented from interacting with the elements in the map and could only view the static image (screenshot) of the evaluated weather web map. For the last question (what is the temperature in a particular place?), locations were changed so that respondents did not memorize the answer. 

The first round of questions in the dynamic section addressed wind speed. The respondent was required to answer two questions concerning which area of the Czech Republic currently had the highest or lowest wind speeds. The second round of questions consisted of five questions concerning cloud cover. Respondents were asked to respond whether clouds were at a specific location and time. Questions in the third round concerned precipitation. The respondents answered whether the rain would occur at a particular place and time.

The respondents were asked the same questions about all web maps. Respondents were required to indicate in the static picture where to switch the weather forecast to another day, where to switch thematic layers or to answer what the temperature was at a given location. For these questions, each map presented was of a different location to prevent the user from memorizing the same answer.

Stimuli

For the eye-tracking experiment, five web weather maps were selected. No study was found with a complete comparison of weather maps, even though weather web maps are widely used by the public almost every day. Their use is not limited by previous knowledge or expertise and they can be used by almost anyone. The evaluated maps were: DarkSky, Windy, In-Počasí, YR.no, and Wundermap. This number was selected so that the time required for the entire experiment did not exceed 30 minutes and to minimize the fatigue and disorientation experienced by respondents.

 Participants

Testing was targeted at multiple user groups. Thirty-four respondents participated in the eye-tracking experiment (14 males and 20 females, median age 23 years). These respondents were separated into two groups of users: novices (16) and experts (18). Students who had not studied Earth Sciences and other respondents without a more in-depth knowledge of meteorology, geoinformatics or cartography were included in the group of novices. This separation may not always be tangible. Nevertheless, a non-geographic student may understand maps and have more experience than a student in Earth Sciences. For a more reliable separation, respondents were asked whether they had any previous experience with web maps, and if so, were included in the expert group. All respondents were from the Czech Republic or Slovakia and the instructions were in Czech. The respondents participated in the study voluntarily and were not paid for the experiment.

Apparatus

For the study, remote eye-tracking device SMI RED 250, developed by SensoMotoric Instruments, was used. This device was operating at a frequency of 250 Hz. 

 Analyses

In the Introductory Section of the experiment, the results were gained based on the video recordings of respondents’ work with the map overlayed by eye-movements. After viewing all recorded videos, a fundamental insight applying to all the web maps used in the experiment was gained.

Processing the results of the Dynamic Section was very time-consuming, as it was necessary to analyse data using dynamic Areas of Interest. Since each respondent worked with the map individually, dynamic Areas of Interest were created for each web map and each respondent separately. These areas of interest (AOIs) were: map fields, timer switching, switching of thematic layers and other information such as legends and supplementary charts. These layers were not active throughout testing and appeared according to how respondents clicked on them. Creation of dynamic AOIs is highly time-consuming, so only six respondents were chosen for this type of analysis. Data were visualized using Sequence Chart method, which displays each respondent’s eye movement data in time as rows. The colour of these rows corresponds to the visited AOIs.

Analysis of the Static Section was much easier, since all respondents were looking on the same stimuli – screenshots of the web maps. The first method, called Gridded AOI is implemented using the open-source OGAMA. The image was divided into a regular grid, each grid segment displaying how many fixations were recorded there. Another method utilized in eye-movement data visualization is called FlowMap and is implemented in V-Analytics software. FlowMaps use Thiessen polygons generated based on the fixation distribution. Arrows between these polygons display the number of moves between them.

ScanGraph was another method used to study the above task. This method was developed to identify differences in the stimulus reading strategy of different groups of respondents. Before analysing the data, areas of interest over the stimulus must be created and marked, for example, A, B, C, etc. The Scanpath of each respondent can then be replaced by a string of letters expressing the order of the visited areas of interest. ScanGraph calculates the similarity of these strings by employing three different algorithms: Levenshtein distance, Needleman-Wunsch algorithm and DamerauLevenshtein distance. Individual respondents are visualized as nodes in the graph, and ScanGraph searches the so-called “cliques” in this graph – a group of respondents who are similar to each other at least to a specified degree. The tool can be used to determine, for example, whether the stimulus was read differently by men and women or experts and novices. Both the Dynamic and Static Sections were also analysed statistically using the Wilcoxon rank sum test, since the data did not have a normal distribution. Statistically significant differences are marked by an asterisk in the figures below. We chose three eye-tracking metrics to analyse data – Trial Duration, Fixation Count and Scanpath Length. 

Conclusion

Weather maps were evaluated by combining research methods with a core eye-tracking experiment that focused on analysing the behaviour of respondents as they worked with the selected maps. The experiment was divided into three parts: a free viewing section, a dynamic section, and a static section. Five selected web maps with meteorological themes were employed in testing. Thirtyfour respondents performed the test, separated into two map user groups of experts and novices. The main aim of the presented research was to find out how users work with selected weather web maps. There are many map characteristics and parameters that affect the metrics being evaluated. All weather web maps are complex cartographic works; they differ in map composition, map symbology, map interactivity, map content, etc. Therefore, it is not possible to conclude which weather web maps were the best and worst overall. It can only be concluded that some maps are easier to understand and use (Windy, In-Počasí, YR.no) and some maps are not (Wundermap). Partial results are presented in the task evaluation. The acquired knowledge can be used to further discussion of weather web maps and their implementation. Our results include the findings that if expandable control panels were available, the respondents only looked at them after they had examined other elements; map interactivity was not an obstacle unless it contained too much information or too many options to choose from; searching was quicker in static menus that respondents did not have to switch on or off; and that the Think-Aloud method has significant limits in the case of dynamic testing due to high user demands. Each web map is different, and both major and minor differences were identified. Further related research may focus on the impact of these differences on user perception and cognition. Analysis can also be focused on different thematic maps and, thus, differences in attitudes of experts and general public (novices) can be evaluated. 

Outputs

Popelka, S., Vondrakova, A., & Hujnakova, P. (2019). Eye-Tracking Evaluation of Weather Web Maps. ISPRS International Journal of Geo-Information, 8(6), 29. doi:10.3390/ijgi8060256

Hujnakova, P. (2018) Analysis of selected web map aspects, Master thesis