Languages, Images and Words
Week 7 :CCT370 – Introduction to Computer Visualization
Introduction: Pictures and Words
 Addresses relationships between visual information and verbal or textual
information
 Dual coding theory
 When should we use a visual display?
 What is a visual language?

And does it make sense to use one to program a computer
 How to integrate images and words
A Memory Demonstration
 Remembering words (and lots of things) is not easy
 Given a is a list of 10 words
 Try to remember them
Method of Loci - a Mnemonic

Remembering things, especially long sequences, not a new problem

Speech, tasks to do, names, …

Dating back to Greek times, have been various mnemonic devices

Method of loci, or places


Uses places and encoding of items
E.g., take a walk through your apartment, or across campus, or your parents’ house – some well known place

A “memory palace”





E.g., 10 for the demonstration
When a sequence is given, place each element at a (physical) point on the path


In Renaissance, mol widely used, cathedrals served nicely
Identify several points
The more “memorable”, or bizarre, the image the better
To retrieve the elements, walk through the physical place
Memory demo again, but will do it using method of loci
“How to Build a Memory Palace”
http://www.wikihow.com/Build-a-Memory-Palace

Decide on a blueprint for your palace.

The larger or more detailed the real place, the
more information you can store in the
corresponding mental space.
•

Define a route.

If you will need to remember things in a
certain order, it is essential that you follow a
specific route through your palace, both in
the real world and in you mind.
•
Identify specific storage locations in your palace
or along your route.

Identify as many locations as you think you
will need.

Walk through your structure or along your
route and really observe it.
•


Memorize your memory palace.

Place things to be remembered in your palace.

Put a manageable amount of information in
each place.

Use symbols.

Generally, all you need to store in each
location is something that will jog your
memory, something that will lead you to the
actual idea you’re trying to remember.
Be creative.
–
Generally, images will be more
memorable if they are absurd (out of
the ordinary
–
Explore your palace.
–
Go through it and look at them.
•
–
•
Stock your palace with other mnemonics.
“Every Good Boy Deserves Fudge,"
which would in turn allow you to recall
the order of notes on the lines in treble
clef (EGBDF).
Use your palace.
just follow your route in order as you do
so.
Build new palaces.
If you need only commit things to
memory for a short time. Just replace
the existing contents with new ones
–
If you need to remember the contents
of your palace for a long time, you can
keep that palace as it is and create new
ones in which to store other
information as needed
–
Memory Demo, Again
 Same song – new technique
Coding Words and Images:
Two Systems
 Bertin, Semiology of Graphics (1983)
 Two distinct sign systems

Associated with auditory information processing


Includes mathematical symbols, natural language, music
Based on visual information processing

Includes graphics, abstract and figurative imagery
 Pavio (1987), dual coding theory
 Two different types of information in working memory

Imagens – mental representation of visual information


Objects, natural groupings of objects, whole parts of objects (e.g., arm), spatial
information about layout/arrangement
Logogens – mental representation of language information



Basic information pertaining to language, but not sounds of words
Processed by a set of functional subsystems that provide support for reading and
writing, understanding and producing speech, logical thought
Not necessarily tied to speech, e.g., deaf and Braille and sign language
Pavio’s Dual Coding Theory, 1

Account of memory (with implications for perception)



Refines/extends basic approach of 3-stage model of human
memory
Sharpens ideas about object recognition
“Makes sense” – split between visual and verbal processing

Long known are different neural processing centers for:



Verbal information (speech areas of temporal cortex)
Visual information (visual cortex)
Core of rationale for use of visual representations

Especially to facilitate learning

E.g., concrete mnemonic devices



Primary rationale for “multimedia”!


Provides account of why it works
For our purposes, informs use of visual representation



Greek orators …
Walk through your house and put …
With “words”, in larger context of diagrams, etc.
Sharpening up relation of “visual” with “semantic, …”
Processing:



Visual-spatial information
Visual text
Acoustic verbal stimuli
Pavio’s Dual Coding Theory, 2
 Processing:
 Visual-spatial information

Enters through visual system

Fed into association structures in
nonverbal imagen system
 Visual text
 Processed,

But then fed into association
structures of logogens
 Acoustic verbal stimuli
 Processed in auditory system

Then fed into logogen system
•Imagens – mental representation of visual information
 Logogens and imagens can be
strongly interlinked
 E.g., word “cat” and language-
based concepts related to cats will
be linked to visual information
related to the appearance of cats
 Method of loci, multimedia, etc.
Objects, natural groupings of objects, whole parts of objects (e.g., arm), spatial
information about layout/arrangement
•Logogens – mental representation of language information
Basic information pertaining to language, but not sounds of words
Processed by a set of functional subsystems that provide support for reading
and writing, understanding and producing speech, logical thought
Not necessarily tied to speech, e.g., deaf and Braille and sign language
“Thinking” Visually

Usually associate processes of “thinking”, e.g.,
greater than, follows from, “logic”, etc., with
“verbal” (logogen based system)

More recently, such operations as part of the
“visual” (imagen based system) are finding
evidence



E.g., Subjects report using imagery when compare
sizes of light bulb and tennis ball, color (green) of
pea and Christmas tree
Positive emission tomography (PET) evidence

Visual processing centers active when imagery invoked

Buy, when mentally change size and position of
imagined object, different visual areas of brain activated
Imagery alone vs. operations on (processing,
thinking with) imagery


When see a cow and imagine a cow same neural
pathways (in part) excited
In accord with accounts of object recognition in
which object is “recognized” through interaction of
stimuli (bottom up) and memory (top down)



See figure at right
Visual memory traces stored as part of processing
Hence, recognition is easier than recall

Matching with something stored (which in part drives
recognition)

Vs. reconstructing all pieces from input of event alone
The Nature of Language(s)

Noam Chomsky:



Innate deep structures (meaning)
Surface structure (syntax, form)
That nature of natural (everyday, spoken) language
is quite similar to formal description of languages
appropriate to describe all languages (natural
and
formal) is among larger advances of 20th century


Universality of human language
And, e.g., Chomsky hierarchy of (deep) interest to
computer scientists

Critical period for language development

But being verbal is not essential to language
development

Sign languages for the deaf are the most perfect
examples of visual language
What is Language?
 Language provides:
 Description
 Communication of intention
 Ability to communicate procedures
and sequences of operations –
including logic

if, but, causes, do a then b then c
 Sign Languages
 Arose spontaneously
 Are not related to verbal languages
 Have syntax
 Become more abstract
 To be fluent in visual (or any)
language we should be trained from
early in life
Again, Visual Languages (or Not)
 Recall, some fundamental questions addressed
in first of class
 Sensory and arbitrary symbols, etc.
 Visual language
 Consider that hieroglyphs gave way to more
abstract symbols
 Why turn back the clock?
 Can there be a true visual language?
 Yes, but not for most of us!
 A critical (developmental) period

ontogenetic
 Consider verbal language
 Abstraction, logic
 (if, while, perhaps)
 Based on speech
 Sign languages are true visual languages
Visual and Verbal Pseudo-code

Ware argues that visual languages in fact not good for
expressing sequential operations common to computer
programming,

Sequence below:





Get a line of text
Change characters to all upper case
Write line to output file
While there is more input
Better expressed by “verbal psuedo-code”:
Repeat
Get a line of text
Change characters to all upper case
Write line to output file
Until there is more input

Flow charts were wrong

Probably just wrong paradigm



But, billions of dollars spent
Cause inappropriate/unnatural focus on detail
Visual programming languages have history of failure
Still, Certainly Uses for Visual Represt.
 Gives rapid recognition and pattern finding (again)
 Has distinct advantages over text (linear, serial) for conveying some
kinds of information
 Consider the text below:
 Jane is Jim’s boss
 Jim is Joe’s boss
 Anne works for Jane
 Mark works for Jim
 Anne is Mary’s boss
 Anne is Mike’s boss
 And it’s visual representation
When to Use Static Images vs. Words

In general (rules of thumb)

Images are better than words for:




Words better for:




Links between entities and groups of entities

Visual information need be meaningful and capable of incorporation into a cognitive framework for this advantage
Image memory can’t be so relied on if information is new and out of context
Images best for providing detail and appearance

Amount of detail extracted (and remembered) depends on time to study


E.g., bus routes shown as graphical representation led to better performance in trip planning than with tables
Visual information generally remembered better than verbal, but not for abstract images



Procedural information
Logical condition
Abstract verbal concepts
Images best for showing structural relationships


Spatial structures
Location
Detail
Recall, silhoettes first, so line drawing best for rapid extraction
Information that specifies conditions under which something should or should not be done is better
provided using text or spoken language
Gestures, and Linking Images and Words

Deixis and the deictic gesture




Gesture that links subject of a spoken sentence with a visual reference
Can be a glance or a nod
Pre-speech
Shown to disambiguate verbal communications

Why the mouse is so powerful …

Other kinds of gestures




Issues in shared environments



Beat gestures for emphasis
Verb gestures showing how to do something
McNeil Hand and mind
Speech + Pointer + Visuals – most important components
Subtle ways of directing attention also important in meeting dynamics.
Linking images and words





Deixis
Pointing is an elementary speech act.
Pointing links images and words
Put that (points) there (points)
Subject verb predicate
Attaching Words to Images
 Turns out the use of words and images together work just fine
 Or even quite well
 “Association” (grouping) can be explained by Gestalt principles
Examples of Integrated Pictures and Words
 Examples
Another Ex. of Integrated Pictures and
Words
 More examples
Yet Another Ex. of Integrated Pics. and
Words
Examples of visual languages
 As noted, limited success of
visual languages, especially
for programming
 Still, some examples
 Sanscrit
 Petri-nets
 Khoros
Sanscrit
 To program:
 Count from 1 to 3
 for i = 1 to 3 do
Petri Net Language
 Petri nets are stochastic
 timed attributed (tokens on
nodes, transitions)
Khoros
 Combine operations
Conclusions
 Interesting, and not fully understood interaction between the
two processing subsystems
 “Visual”, images, imagens
 “Verbal”, auditory, logogens
 Though certainly there can be “visual languages”, finding
useful applications has been elusive
Next class
 Topic: Thinking Visualizations
 Readings:
 Ware, Chapter 11