Mobile and Non-Cooperative
Face in the Crowd Recognition
By
Dr Brian Lovell
CTO Imagus Technology Pty Ltd
Abstract
There is a sea change in the video surveillance industry due to the
adoption of internet technologies (IP). A similar situation occurred in the
computing world in the 1990’s when standalone and locally connected
PCs were all connected together to form the internet. This fixed PC
network was then joined by billions of geolocatable mobile smart phones
in the late 2000’s. In video surveillance, this same IP technology is now
allowing the seamless integration of digital cameras and other sensors
into huge integrated video capture and storage networks. The conversion
of analog video systems to IP connected digital technologies is presenting
opportunities for scalable local and cloud-based visual analytics to
automatically analyse all the video streams and send alerts to responders.
This advanced processing has the potential to add astounding value to
surveillance networks and should greatly increase their effectiveness —
while simultaneously reducing the cost of monitoring and operation.
One absolutely core analytic is scalable robust face recognition to noncooperatively identify persons of interest in large crowds. By noncooperative we mean that the person is not actively posing for the
cameras and that they are simply going about their day-to-day business.
To be successful, such analytics must be very simple to configure and
work reliably in a wide variety of conditions. Analytics must also be
extremely fast and scalable to hundreds, if not thousands, of high
definition camera and mobile smartphone nodes.
Developing and testing such technology is incredibly challenging and
requires a combination of very fast and accurate state-of-the-art
biometric algorithms, distributed databases, mobile platform integration,
hardware acceleration, and parallel processing via GPU acceleration. In
this whitepaper we cover emerging system trends such as super
megapixel cameras, post incident digital PTZ, integration and fusion of
video and non-video sensors, wearable and remote biometrics.
Introduction and Background
As biometric technologies advance due to improved capture and search technologies and
reducing cost, they are becoming more widely adopted. New smartphones include tiny
fingerprint readers as standard technology even though they already have high quality
cameras which could be ideal for face recognition.
Early adopters of biometric technologies include the national immigration and border
protection services of several countries. For example, all foreigners entering the USA are
fingerprinted and their faces are digitally photographed under controlled conditions. This
procedure is completed manually by US border control officials at the arrivals desk on
cooperative subjects and it is a huge impediment to fast passenger facilitation.
In Australia and New Zealand, passengers with biometric passports have the option of
entering the country via the fully automated SmartGate1 booths. Smartgate uses facial
verification technologies to match the identities of each passenger with their biometric
chipped passport. However face verification is far from the mature and reliable technology
that the border agencies would like. Indeed, the face verification scanner gates at
Manchester Airport were suspended in February 2011 after a man entered the UK using his
wife’s passport.2 The couple had inadvertently switched passports, so the problem was
detected when his wife tried to enter and was blocked because she presented her
husband’s passport. The Manchester gates were shut down for months while a major
investigation was conducted.
The recent emergence of nationals on no fly lists leaving western countries on false
passports has prompted governments to now consider face recognition technologies on
exit. Surprisingly recent studies supported by the Australian Department of Foreign Affairs
and Trade (DFAT) show that experienced immigration officers are only about 80% accurate3
when matching live persons to their passport photos. This indicates a need for accurate
and convenient face biometric technology to support and cross-validate their decision
making.
Passenger Facilitation versus Security
Despite legitimate operational concerns, automated gates are gaining in popularity not
primarily because of enhanced security, but primarily because they provide improved
passenger facilitation during periods of congestion with reduced staffing levels. This is the
same rationale for self-service checkouts at supermarkets — the self-service counters
provide an alternate path for customers when the manual checkouts are congested. It is
clear that successful new border technologies must not only be reasonably secure, but also
must not impede the flow of passengers through the terminals.
Australian Customs. Smartgate: How it works.
http://www.customs.gov.au/site/page5831.asp [Last visited: 3-Mar-2015].
2 R. Massey. Face recognition scanners at Manchester Airport suspended after man
re-enters Britain on his Wife’s passport. http://www.dailymail.co.uk/news/article1357949/Manchester-Airport-securityblunder-Woman-boards-flightstrangersboarding-card.html , Feb 2011. [Last visited: 3-Mar-2015].
3 http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0103510 [Last
visited: 3-Mar-2015]
1
2
The desperate need for embedded video analytic solutions emerges from the huge data
rates of modern high definition surveillance cameras as well as the unbelievable numbers
of cameras. Modern CCTV cameras are transitioning from standard definition (0.3 Mpixel
per frame) to high definition (2.0 Mpixel per frame) which represents a 6-fold increase in
data. While a simple single core implementation may cope with standard definition video,
a much more sophisticated parallelized approach is required to process such high definition
data rates.
One the most interesting emerging camera trends is super megapixel cameras with 16
Mpixels or more per frame. A key advantage of these super megapixel cameras is that they
cover over 50 times the area of a standard definition camera at the same resolution. This
gives a single camera the ability to surveil a very large area such as an airport check-in and
still capture high-quality face images of all passengers. Super megapixel cameras can
largely replace both PTZ and fixed cameras for indoor environments as they allow postincident PTZ (up to 7x zoom) as well as camera sharing by multiple agencies without
compromising inter-agency security. That is, multiple agencies (e.g. Customs and Police) can
simultaneously zoom in on different persons of interest in real-time or playback without
revealing who exactly is under observation.
Given 1) the streaming nature of surveillance video, 2) the large number of simultaneous
video feeds in a major site like an airport, and 3) the 24/7 nature of surveillance, practical
technologies must operate in real-time or possibly even faster. Finding and recognising
faces in such enormously high definition video streams in real-time is a grand technical
challenge.
Imagus Non-Cooperative Face Recognition Technology
Imagus Face Recognition Technology (FRT) has been researched and developed from the
ground up to rapidly match low quality photos and videos and does not use face
recognition techniques from other companies. The advantages of our current suite of
algorithms are 1) extremely high speed search, 2) relatively small memory footprint, 3)
ability to match very low-resolution images as found in CCTV, and 4) extreme insensitivity
to simultaneous changes in pose, illumination, expression, focus, motion blur, geometric
distortion, and image misalignment.
These computational advantages mean that the algorithms are ideal for mobile and
embedded system implementations. The recognition performance advantages mean that
the algorithm is well suited to both CCTV and mobile applications where image capture
conditions are largely uncontrolled. In addition, Imagus FRT performs true multi-frame live
video recognition from both PC and mobile platforms significantly increasing both accuracy
and confidence in matches.
There has been a great deal of work on face recognition technologies over at least 35
years4 including some on video based recognition.5 In 14 years of research and most
W. Zhao, R. Chellappa, A. Rosenfeld, and P. Phillips. Face recognition: A literature
survey. ACM Computing Surveys, 35(4):399–458, 2003. 118
5 V. Krüger, S. K. Zhou, and R. Chellappa. Integrating video information over time.
example: Face recognition from video. In Cognitive Vision Systems, LNCS, volume
3948/2006, pages 127–144, 2006
4
3
recently within Imagus we have implemented and evaluated many state-of-the-art systems,
but all known methods we have tested to date fail to adequately address the pressing need
for good recognition from uncontrolled low resolution image probes using uncontrolled low
quality face galleries. The lack of research interest and activity in low-quality images is
partly due to the emphasis in public face recognition benchmarking on high-resolution still
images.6
Such formal benchmarking often yields impressive recognition rates with virtually zero
errors. Yet everyone with any experience in biometrics knows that such performance is
simply unattainable in the field without enormously expensive image capture equipment
such as the SmartGate booths. Obtaining good recognition rates is all about getting the
image capture conditions absolutely perfect — and achieving this in the field is incredibly
expensive. Face galleries collected by user agencies are almost invariably of much lower
quality than the face recognition benchmarking databases used for the testing and
development of most FRT systems.
Our Imagus FRT system has been researched and developed from inception to specifically
cope with low quality mismatched probes and galleries gathered from CCTV, mobile
devices, and social media.
Unconventional Applications Well-Suited to Imagus FRT
Undocumented Traveller Problem
The undocumented traveller problem occurs when a person purchases a high quality false
passport from a people smuggling organisation or equivalent. The person boards an aircraft
on the false document and travels to a country where they wish to claim asylum. After
exiting the aircraft, the passenger destroys the false travel documents by, say, flushing
them down a toilet where there is no surveillance. Later they present themselves at the
immigration desk claiming a different identity (and often nationality) and request asylum.
The cost of processing each asylum seeker is estimated at about USD80,000 which is borne
by the taxpayer. Other countries are naturally reluctant to accept undocumented persons,
so the country of arrival is forced to deal with the problem and expenses. Due to the fait
accompli nature of arriving at the border without documents, the likelihood of success in
seeking asylum is quite high. This presents a clear danger to society due to the increased
risk of accepting undesirable persons. The technical challenge is to track the person back
from the immigration desk to the aircraft of arrival, so that the country of origin and
travelling identity can be determined rapidly.
This is an ideal application for Imagus FRT as CCTV to CCTV cross-matching is highly reliable.
The images are taken only a few hours apart so there is no issue with photo aging and large
changes of appearance. Even a busy airport processing 1.5 million passengers per year has
only 40,000 passengers passing per hour, so searching for passengers over a window of
several hours is well within Imagus system capabilities.
P. J. Phillips, W. T. Scruggs, A. J. OToole, P. J. Flynn,K. Bowyer, C. L. Schott, and M.
Sharpe. Frvt 2006 and ice 2006 large-scale experimental results. IEEE Trans.
Pattern Anal. Mach. Intell., 32(5):831–846, 2010. LNCS, volume 5558, pages 199–
208, 2009
6
4
Passenger Transit Time Measurement
As part of quality control an airport may be required to continuously monitor passenger
transit times between two control points. Penalties may apply if queue times exceed
certain limits. A non-intrusive way to measure transit times would be to autoenrol every
passenger and then recognise them as they progress through the airport. If a few
passengers are missed due to obscuration that is not a major issue as it will not affect the
travel time statistics. Automatic enrolment of passengers and subsequent recognition is
supported in the Imagus FRT suite.
Biometric Access Control
Imagus has developed a multi-camera software product to upgrade any access card system
based on NFC or magnetic stripe reading into a fully audited biometric gateway. This
means that every person entering a secure area must not only swipe their access card but
their face must also match the registered photo. The clever part of the technology is that
the face is acquired non-cooperatively as the person walks up to the card reader so there is
no delay in processing unless there is a face mismatch. Every single entry is logged and
every single person is biometrically matched to their enrolment photo. Such identity
checking is simply impossible to do in a timely manner with security officers.
One possible application would be to biometrically upgrade the V-Card or MARSEC card
system for staff entry to secure airports and ports. The application was originally
developed for military shipyards to prevent unauthorised entry and to provide a photo
audit trail for all site accesses.
It could also be used to prevent unauthorised use of any valuable equipment such as large
mining equipment or military hardware.
Concierge and Greeting Systems
Many businesses want to give customers personalised service and not treat them as a
number. This is why airlines greet passengers by name in Business Class once they have
taken their allocated seats. People generally like to be recognised in public spaces as it
makes them feel important and valued. Such loyalty programs are often the most
profitable parts of major businesses such as airlines and quality retail stores.
Imagus FRT allows these businesses to recognise and reward their top customers and
encourage brand loyalty. Instead of swiping a loyalty card and then greeting the customer
by name, they could be recognised and welcomed as they walk up to the counter. The
bartender could pour your favourite drink before you place the order.
Similarly the regular troublemakers and late passengers can be identified and handled
discretely. Business no longer needs to be faceless and good customers can be greeted as
friends.
Advertising and Up Selling in Retail
The non-cooperative nature of Imagus FRT means it is suited to outdoor advertising and
customer management. As well as recognising and possibly identifying regular customers
and troublemakers, such systems can simultaneously determine gender, age, emotional
state and other attributes of new customers. Advertising could be automatically tailored to
the individual customer from the electronic menuboards. For example, at fast food
5
establishment, the store could promote deals on icecream to the younger customers and
on coffee for the older customers. Regular customers can be recognised and offered deals
based on past orders and preferences.
Mobile Face Recognition
A unique capability of Imagus FRT is the ability to perform live video face recognition from
mobile smart phones or wearable devices. Smart phones are a cost-effective way to roll
out face recognition across an organisation because so many devices have already been
purchased and they simply need the Imagus FRT app installed to participate in nation-wide
face recognition programs. A key benefit of using a phone is that the phone can be
positioned close to the person of interest at eye level so the quality of images tends to be
very good compared to CCTV.
Most fixed CCTV systems simply do not gather faces effectively since the cameras are too
far away and the cameras are often positioned much too high. These cameras are fine for
the purposes of public liability but are largely useless for person identification. However
the fixed and mobile cameras can work seamlessly together to form a cost-effective
network for identifying persons of interest. Mobile devices can gather quality video at eye
level and identify the time and location of face matches via GPS technology.
Wearable Face Recognition
Imagus FRT operates on iOS and Android platforms including the exciting X6 Glasses from
Osterhout Design Group (ODG).7 The X6 glasses are basically an Android smartphone in the
format of a set of glasses. They offer a handsfree heads up display to overlay information
on the real world. Some commentators have said that the X6 is like wearing a high-end
tablet on your face. Originally the glasses were designed for the military as handsfree
operation is critical for combat operations. Now consumer versions of the glasses will be
available in 2015.
Face recognition on the X6 glasses opens up a range of new applications from policing to
passenger facilitation in airports.
Brian Lovell wearing the X6 Glasses from the Osterhout Design Group
7
http://www.osterhoutgroup.com/home [Last visited: 4-03-2015]
6
Faster than Real-Time Face Recognition
There is an emerging need for faster than real-time face recognition. Imagus FRT is capable
of extremely fast searching due to the parallelisable nature of the code. One application
could be indexing large video archives by face for instant retrieval of file footage in news
studios. Another could be indexing body-worn CCTV recording systems carried by Police
which record their daily interaction with the public.
Social Media Mining
Imagus FRT can mine social media feeds to find persons of interest and their connections.
This software operates on a cloud service, so it is instantly scalable to large investigations of
major events as required.
Conclusions
In this white paper we have described the development of Imagus Face Recognition
Technology capable of scanning CCTV and mobile video for faces in real-time. To achieve
real-time rates, we exploit embedded system technologies including hardware face
detection, GPU acceleration, together with smart algorithm design. This new face
technology simply works as it should and it will open up a whole new world of
opportunities to embed convenient and reliable face recognition into our day-to-day lives.
7