Visual Design of Bilingual Learning Tools
Module: Visual Thinking and Visualisation
By Nathan Floor (FLRNAT001) & Ryan Saunders (SNDRYA004)
Contents
1.
Introduction .................................................................................................................................... 2
2.
Related work ................................................................................................................................... 2
3.
Visual queries .................................................................................................................................. 5
4.
Design.............................................................................................................................................. 5
a.
Vocabulary tools: ........................................................................................................................ 5
b.
Grammar tools: ........................................................................................................................... 7
5.
Strengths and weaknesses .............................................................................................................. 9
6.
Conclusions and future work .......................................................................................................... 9
7.
Who did what................................................................................................................................ 10
1
1. Introduction
This project aims to use visualisation techniques to assist the learning of a new language. The idea
was to design an interface that is fun and easy to use, while still effectively helping the user learn the
new language. The program has been aimed at both children and adults who are in the process of
learning a new language. There are two aspects of this that we have tried to incorporate: the
learning of vocabulary, and the learning of grammar. To meet this goal, we developed a prototype
that uses pictures, games, and various elements from visual thinking theory. The design allows for a
user to use a language they are already familiar with and use it to help them learn a different
language. The two specific languages implemented in the prototype are English and Afrikaans, but
the program can be extended to support other languages.
The program is divided into the following sections:
Bilingual Vocabulary:




Word cards: numbers, verbs & tenses, places
Picture slider game
Bicycle race game
Word matching game
Bilingual Grammar and Parts of Speech:


A sentence builder for learning grammar
A sentence tester for testing grammar
2. Related work
There are many learning tools for languages. The problem we found is that most of these tools are
only text based, with little or no use of colour and interesting interactions for the user.
There are some ideas we came across which were effective at teaching new vocabulary through an
interesting and engaging interface. The effective use of colours helped focusing the user's attention
onto the language being learnt. In the example below1 (Figure 1), the colours blue and purple
successfully distinguish between the two languages. The sliders introduce interaction for the user.
This example is not a program but rather a toy for a child, which allows for different cards to be
swapped in and out of the frame depending on the user's choice of subject.
Figure 1: Slider toy
1
http://www.timetoplaymag.com/toys/5090/smart-play/ingenio-slide-discover/
2
Another example is seen in Figure 2, which effectively uses colour to make looking at the page much
more interesting. However, one problem here is that the colours of no inherent meaning or
significance. The use of symbols is very effective; we used these in our prototype for visual feedback
to the user on the progress. This example also fails to distinguish between the two languages in any
way. The user has to study the page first, for some time, before able to identify which column is for
which language.
Figure 2: Matching game
Another example of gamification can be seen in Figure 3, was used in the second design phase of the
prototype. Stills of the game were used to simulate motion; however the game itself was fully
interactive. The original game, used questions with two options from which the player can select
which they think is correct. In our prototype, we expanded that to many choices.
This design fails to draw the appropriate attention to the specific word in question. And due to the
limited number of options, one could simply guess the answer and likely get it right without actually
learning anything. We felt this could have been improved on by highlighting the word in question
and trying to make the player think and be more engaged in the learning process.
Figure 3: Spanish-English Cycle Race
Google Translate2 (see Figure 4) is a simple and effective tool that allows translating a piece of text
to a different language. It allows the user to see both the original text and the translated text side by
side. One can even hover the mouse pointer over a word, and see which word it was translated
from. Its main purpose, however, is not to teach a new language, but simply to provide a means of
viewing a text in one’s own language. Therefore, it does not make much use of visualization
techniques.
2
http://translate.google.com
3
Our sentence builder however aims to teach grammar, and we also require some means to visually
indicate parts of speech. Because of this goal of teaching, it is also necessary that the user must be
able to easily see how parts of the English sentence correspond to the parts of the Afrikaans
sentence. So ideally, it would not be necessary to hover over a word in order to view this
information.
Figure 4: An example text as translated by Google Translate
SentenceBuilder for iPad3 (see Figure 5) is a language tool that does aim to teach language skills,
including grammar. What makes it particularly interesting is that it provides a selection of different
word or phrase choices to the user, and it tests their grammar.
This is a good idea in that it does not only teach, but also tests the users abilities, which reinforces
learning. A limitation however is that it is not designed for bilingual learning, as there is no
comparison between two languages. Also, the choices of phrases are limited, in that they leave little
room for exploring other correct sentences.
Figure 5: A screen shot of SentenceBuilder for iPad showing various phrase options.
3
https://itunes.apple.com/us/app/sentencebuilder-for-ipad/id364197515?mt=8
4
3. Visual queries
The following visual queries are required for learning vocabulary:








What is the word in question?
How well did I do in the test as a whole?
Did I correctly translate the English {word} into Afrikaans?
Did I correctly translate the Afrikaans {word} into English?
What picture is associated with an English word?
What picture is associated with an Afrikaans word?
What is the English for {word} in Afrikaans?
What is the Afrikaans for {word} in English?
The sentence builder and sentence tester aim to support the following visual queries:






What Afrikaans word corresponds to English word x?
What English word corresponds to Afrikaans word y?
How does the Afrikaans sentence structure (i.e. the grammar) differ from the English
grammar?
Which words are nouns, verbs, adjectives, adverbs, etc.?
What is the part of speech of a specific word?
The user should also be able to easily distinguishing between English and Afrikaans
sentences. (Which sentence is the English sentence? Which is the Afrikaans sentence?)
Additionally, the interaction of the sentence builder and sentence tester require the following visual
indicators:



Where can a word be added in the sentence?
Which words can I modify and experiment with in the sentence?
How can I change the tense?
The sentence tester, in particular, also aims to support the following visual queries:


Was my answer sentence correct?
If my answer sentence is incorrect, which words did I translate incorrectly?
4. Design
We used a prototyping tool4 to assist in the design phase to show much of how the user could
interact with the interface. While there was only limited functionality provided, this tool still enables
us to effectively demonstrate the relationship between the different elements of the interface, such
as the different vocabulary learning tools and the sentence builder.
a. Vocabulary tools:
For the initial vocabulary learning tool that we designed, we focused on the principle of memory
cards. A consistent colour scheme was used throughout the prototype to distinguish between the
two languages. Images were used to provide the user with a reference point for the word they were
trying to learn. There was a multiple choice test feature which allowed the user to test their
knowledge based on the options available.
4
Justinmind Prototyper 5.1.0 (see http://www.justinmind.com/)
5
Figure 6: Vocabulary learning cards
There were many problems with this initial design. For one, it did not address the problem of
engaging the user effectively. It did not make use of interactions to make the learning process more
dynamic. Also there was little effective use of symbols or images. The emphasis was on the words,
which makes for a boring experience.
So after the initial evaluation, we had many changes to make to improve on the initial design. The
first change was to introduce many more interactions and images to liven the interface up and make
it more engaging for the user. The colour scheme has remained the same to be consistent with the
rest of the prototype.
The primary goal after the first evaluation has been to gamify the interface. The focus being on
making using competition and progress rewards to encourage and motivate the user to use the
interface. Most of the changes have been to try and achieve this.
Figure 7: Second phase of vocabulary tool designs
6
b. Grammar tools:
In order to address the visual queries for the sentence builder and sentence tester, and to allow for
direct comparison, it was necessary positioning two sentences on the screen – one corresponding to
each language. The two sentences are distinguished both by their position (English at the top of the
screen, Afrikaans at the bottom) and their background colour (English green, Afrikaans blue). The
background colours are of low saturation in order to keep focus on the words.
Each word is positioned inside a bright, saturated shape or “bubble”. This directs focus to the words
and is especially suited to younger users. One is able to see which Afrikaans word corresponds to
which English word by finding the word that has the same colour and shape as the English word in
question. So matching words are indicated through two channels. (Originally only colour was used,
but after receiving feedback, the shape was also included as an extra channel). The colour and shape
of the bubble also indicates the part of speech, and the user is able to identify all of the nouns, for
instance, by finding the blue rectangles.
One important aspect of the colour bubbles is that it allows the user to see the overall “shape” of a
sentence, and compare it with the shape of the translated sentence. This allows the user to easily
notice the differences in grammar between the two languages.
The interface was designed to allow the user to explore and discover unique sentences. The starting
sentence is a simple sentence that simply has an article followed by a noun and verb; e.g. “The dog
barked” and its corresponding Afrikaans sentence, “Die hond blaf”. Up and down arrow buttons
allow the user to experiment with different noun and verb combinations. For the sentence builder,
each change to the English sentence makes a corresponding change to the Afrikaans sentence, and
vice versa. Once the user has familiarised themselves with nouns and verbs, they are able to add
adjectives and adverbs to the sentence. This is done by clicking on small buttons that are in line with
the rest of the sentence and which are positioned where a new word can be added. The adjectives
and adverbs can also be modified. Finally, the tense of the sentence can be changed by clicking left
and right arrow buttons at the corresponding edges of the screen. The right arrow moves one to a
more future tense, while the right arrow changes the tense towards the past tense.
Figure 8: Initial design (left) and improved design (right) of the sentence builder. Note how the later design distinguishes
parts of speech on two channels.
7
The sentence tester was a feature that was added in response to feedback that the program should
be made more fun and interactive (i.e. gamified). The sentence tester is similar to the sentence
builder, except that it tests the user by providing specific fixed English sentences. The user is
required to use what was learned from the sentence builder in order to translate an English
sentence into an Afrikaans sentence. This is done through a similar interface as described above.
Once the user is satisfied with their sentence they press the next button. If the sentence is correct, a
large green tick appears next to the correct sentence for about one second, and then another test
sentence is provided. If the sentence is incorrect, then the incorrect words are highlighted through
the use of motion: each incorrect word flashes for about a second. The flashing is implemented by
increasing the size and colour of the text, and alternating between that and the original. This is a
simple and effective way of making the incorrect words “pop out” at the user, and the flashing itself
is understood to mean “warning”, “danger”, or “incorrect”.
Figure 9: The added sentence tester feature. Notice visual indicators for incorrect words, and for correct sentences. (These
indicators appear briefly, after the >> button is pressed).
8
5. Strengths and weaknesses
The strengths of the vocabulary learning tool are that it combines a collection of popular and
successful learning tools into one interface, with a few modifications. It has been heavily gamified, to
make it fun and engaging for the user.
The sentence builder provides an interactive learning tool to assist those learning a new language. It
allows direct translation and comparisons between two languages, not only of words but also of
sentences. It provides a means of exploring new words and sentences on one’s own. It also provides
a simple way of showing the “shape” or grammar of a sentence, and allows for quick identification of
nouns, verbs, and other parts of speech.
Initially, however, it did not provide a means to test one’s knowledge of a language. The sentence
tester addresses this need.
Some of the ideas for gamification for the vocabulary tools had to be simulated since it would
require an entire game to be created. A selection of consecutive images was used to simulate
motion and show how the interface could work and look like. It restricted the functionality that the
interface was capable of since it was meant to only demonstrate what it would look like and
potentially be able to do. However it was never intended to provide any real functionality.
Another weakness of the design was the lack of styling and theming for the interface as a whole.
Although this was not an art project, it would have helped establish a fun and interesting
environment.
There are also some limitations of the grammar tools. The sentence builder and sentence tester,
because of the focus on a sentence, do not provide visual or audio descriptions of the objects or
actions of a sentence. This is, in part, due to the fact that often one picture or sound clip is not able
to illustrate an entire sentence.
Also, these tools cannot replace traditional methods for learning a language, but they do assist the
process of learning.
6. Conclusions and future work
The outcome of this project is that gamification is effective in creating a good learning environment.
For the most part, users may not be interested in learning vocabulary just for the sake of it. The
learning process is much more effective and fun if there is a context, such as the sentence tester,
and also if there is some sort of reward or tangible outcome.
In the future, an actual game could be implemented around the design of these learning tools as
opposed to simply a prototype. The goal would be to further explore different ways to make a game
out of sentence building and vocabulary learning, in fun and effective ways. The program can also be
extended to support additional languages.
There are a number of possible ways in which the sentence builder and sentence tester can be
extended and improved. For example, it is important that the application should be able to support
more sentence structures than what the prototype currently supports. Regarding this, it is necessary
to note that larger sentences may require additional visual indicators to distinguish between
multiple nouns, verbs, etc. in a sentence. Additionally, sentence-to-picture correlation can be
introduced, where a picture changes as the sentence changes. But having one picture to represent a
9
sentence is not always feasible. To overcome this attempts have made by others5 to represent a
sentence with multiple pictures; but this seem to fall short of actually describing a sentence, as it
only describes each word individually.
There are a few design tweaks that should ideally also be implemented. We have not for example
included a legend to indicate what the shapes and colours mean – something that ought to be in a
distributable version of the application. Additional requirements include better affordance for
buttons, and a progress meter to indicate the current stage and how many stages are left.
7. Who did what
Nathan worked on the vocabulary learning tools section, while Ryan worked on the grammar
learning tools. This division of work has been applied to all aspects of this project, including this
report.
5
See https://itunes.apple.com/us/app/picture-the-sentence-hd/id478599625?mt=8
10