Real Time Linux
B.Gustafsson
European Southern Observatory
ESO Feb 5 2004
ESO
Why Real Time Linux
ALMA Project
• Investigate the possibility to replace
VxWorks on LCU
– Expensive (licensed)
– Bad support
– Present version (Tornado 2.0.2/VxWorks 5.4)
has old compiler (gcc 2.7.2, C++ egcs 1.0)
• No namespace, limited STL support
• TAO not supporting native exceptions
– Tornado 2.2 has gcc 2.96
B.Gustafsson ESO, 5 Feb 2004
Real Time Linux
2
Plan
ALMA Project
• 3 Phases
– Phase 1, feasibility study, finished end of July 2002, report
available
– Phase 2, implementation, finished August 2003
– Phase 3, test, was performed Oct 2003
• Implementation is finished. First target CPU Motorola MV2700
(PowerPC), but switched to VMIC 7766, Intel CPU
• Linux and RTAI installed, ACS working
• Drivers for CAN, VME and parallel I/O done
• Booting over network
• CAN device software and mount control done
B.Gustafsson ESO, 5 Feb 2004
Real Time Linux
3
Real Time: Introduction
ALMA Project
•
•
•
•
Real-time systems are concerned with when things happen as well as what
happens. A real-time program must not only produce a correct answer, it must
do so "on time." A late answer is a wrong answer.
Real-time systems can be further characterized as either "soft" real-time or
"hard" real-time.
A hard real-time system absolutely must meet its scheduling deadline every
time. Failure to meet the deadline may have catastrophic consequences,
including loss of life.
A fly-by-wire aircraft control system is an example of hard real-time. The
control algorithms depend on regular sampling intervals. If sampling is
delayed, the algorithm can become unstable.
A soft real-time system may cause cosmetic or annoying disturbances, but in
general it will be more tolerant of missed deadlines.
A video playback system may not look good when missing deadlines, but it
should not cause serious damage to a process.
B.Gustafsson ESO, 5 Feb 2004
Real Time Linux
4
Is Linux Real-Time ?
ALMA Project
• For these several reasons, standard Linux isn't
capable of a real-time response:
• Coarse-grained synchronization.
• Paging.
• The scheduler tries to be "fair."
• Linux reorders requests from multiple processes to
make it more efficient.
• Linux will perform "batch" or "queued" operations
to make more efficient use of hardware.
• Kernel not interruptable
B.Gustafsson ESO, 5 Feb 2004
Real Time Linux
5
Real Time Linux
ALMA Project
• Two hard real time linux, RTAI and
RTLinux
• RTAI selected
• RTAI
– tweaks the Linux interrupt
– Installs its own scheduler
– Linux is just another task with low priority
B.Gustafsson ESO, 5 Feb 2004
Real Time Linux
6
The RTAI solution
ALMA Project
• Whenever Linux tries to disable interrupts, the real-time
kernel intercepts the request, records it. and returns to
Linux.
• But, it does not disable interrupts!
• When a hardware interrupt occurs, the real-time kernel first
determines where it is directed:
• Real-Time Task. Then schedule the task.
• Linux. Check the software interrupt flag:
– If enabled, invoke the appropriate Linux interrupt handler.
– If disabled, note that the interrupt occurred, and deliver it when
Linux re-enables interrupts.
B.Gustafsson ESO, 5 Feb 2004
Real Time Linux
7
Real Time Interrupts
ALMA Project
•
•
•
•
•
•
•
When Real Time services are enabled the interrupt dispatch table is replaced
by a "Real Time IDT " table.
With the Real Time system in place the mechanism for disabling and enabling
interrupts no longer prevent interrupt processing. Linux locks do not prevent
the Real Time Scheduler running. Interrupts are now divided into two classes.
Real Time interrupts
Linux Interrupts
The status of Linux Interrupts is now held in an interrupt pending register. A
Linux interrupt will be acknowledged and the pending bit set. If the system is
running in a Linux context the Interrupt will be serviced as normal.
If the system is running in a Real Time Context the Linux interrupt will be
serviced when the system returns to normal Linux (after completing all real
time tasks).
Normal Linux Interrupts cannot preempt Real Time Linux.
Real Time Linux can preempt normal Linux and other Real Time Linux tasks
of lower priority
B.Gustafsson ESO, 5 Feb 2004
Real Time Linux
8
Kernel Modules
ALMA Project
• RTAI processes run as kernel modules
• Kernel modules are effectively an extension
to the kernel that can be introduced into a
running system
B.Gustafsson ESO, 5 Feb 2004
Real Time Linux
9
Kernel Modules
ALMA Project
• As extensions of the kernel, modules execute at Privilege
Level 0 and have access to the kernel's symbol table.
• A module can execute privileged instructions such as I/O.
• A module can call any global kernel function.
• A failure in a module can easily bring down the entire
system.
• A module that fails to install cannot be removed.
• There's no protection against anything at Privilege Level 0.
B.Gustafsson ESO, 5 Feb 2004
Real Time Linux
10
Real Time Scheduler
ALMA Project
• RTAI gives the user a real time scheduler. When triggered
from a timer interrupt the Real Time task can be scheduled
in a Real Time context if required.
NO matter what regular linux is doing.
• Spinlocks and interrupt masks are no longer honored
within the Real Time context. Special Real Time
components replace the normal linux components.
• The basic scheduler is priority based. Other scheduling
options are available and, in fact, it is easy to add your own
options.
• Zero is the highest priority and some very large number is
the lowest. The lowest is reserved for Linux.
B.Gustafsson ESO, 5 Feb 2004
Real Time Linux
11
The Real Time Timer
ALMA Project
• The core of RTAI lies in its use of the system
timer.
• Normally (under Linux) set to interrupt at a 10
milliSecond interval The timer is modified in two
ways.
– Its period can be adjusted
– It is served by a real time interrupt handler.
• When RTAI is "mounted" the timer comes under
Real Time Control
B.Gustafsson ESO, 5 Feb 2004
Real Time Linux
12
Managing the timer
ALMA Project
• RTAI manages the timer in two distinct ways.
– Periodic
– Absolute or "one-shot"
• In Periodic mode the timer is set to interrupt at a fixed, non
varying, period. There are no updates to the timer from the
scheduler.
Periodic mode is available to reduce the significant
overhead required to program the timer registers.
• In One-shot mode the timer is reprogrammed every time
the scheduler is called. There is an overhead involved but
better time resolution and more adaptable scheduling
B.Gustafsson ESO, 5 Feb 2004
Real Time Linux
13
Real Time Communications
ALMA Project
•
•
•
•
•
Fifos
Message Queues
RPC processes
Shared Memory
Fifos and shared memory can be used for
communication with user space (Linux)
B.Gustafsson ESO, 5 Feb 2004
Real Time Linux
14
Other system calls
ALMA Project
•
•
•
•
•
Suspend/resume
Semaphores
Sleep
Get time
Set priority
B.Gustafsson ESO, 5 Feb 2004
Real Time Linux
15
RTAI pros and cons
ALMA Project
•
•
•
•
•
Very good real time response
All resources avalaible
Debugging difficult
No Linux system calls available
Limited set of system calls
B.Gustafsson ESO, 5 Feb 2004
Real Time Linux
16
RTAI Example
ALMA Project
RT-FIFO
User Process
RT-FIFO
X-Window System
Display
B.Gustafsson ESO, 5 Feb 2004
RT-Process
Linux Kernel
Disk
Network
Real Time Linux
Peripheral Device
17
RTAI in User space
ALMA Project
• LXRT (Linux - Real Time ) arose from a desire to make
the rich API provided for RTAI kernel modules available to
user space tasks. The ease with applications could be
crafted using the Kernel API was contrasted with the
limited options available within standard user space Linux.
• In particular task timing and messaging could be
considered to be more advanced and easier to use within
the RTAI kernel modules.
• As a result a large section of the kernel RTAI API became
available to regular users.
B.Gustafsson ESO, 5 Feb 2004
Real Time Linux
18
Soft and hard real time
ALMA Project
• There are two modes of working with LXRT.
Soft real-time allows the user to use the RTAI API but the
task is scheduled under the regular linux scheduler. This
means that normal linux system calls can be used alongside
the additional RTAI calls. The task will make use of the
RTAI API but will not execute as a real time task.
Hard real-time allows a special lxrt scheduler to steal the
task from the normal linux scheduler. This then runs the
task in a real time context. Normal Linux System calls are
not allowed and the task must have its working space
locked in memory.
B.Gustafsson ESO, 5 Feb 2004
Real Time Linux
19
Our approach
ALMA Project
• Run ACS under Linux
• Only put a small part under RTAI
B.Gustafsson ESO, 5 Feb 2004
Real Time Linux
20
TICS ABM LCU
ALMA Project
Client
ticsAntMount
ticsFgLoop
LCUClock
ticsBgLoop
Dig. I/O Driver
ticsAntVa
CANManager
Dig. I/O board
CAN Driver
CAN bus
B.Gustafsson ESO, 5 Feb 2004
CAN I/O board
Real Time Linux
Clock Pulse
21
Useful links
ALMA Project
• http://www.aero.polimi.it/~rtai/
• http://www.realtimelinuxfoundation.org/
• http://fsmlabs.com/community/
B.Gustafsson ESO, 5 Feb 2004
Real Time Linux
22