Data Handling and Transfer for German
LOFAR Stations
James M Anderson
[email protected]
On behalf of LOFAR
and GLOW
(slides stolen from many other LOFAR presentations by other people)
→
LOFAR Solar KSP Meeting, Potsdam, 2009 Jun 25
James M Anderson
1/24
Some Initial Comments
●
LOFAR network details still under development
–
–
●
Continue to evolve for international partners as single-station (and
stand-alone operation) needs are developed and implemented
GLOW meeting in Garching next week
–
–
●
Now mostly frozen within the Netherlands
Will discuss many issues of single-station operation (and subarray
interferometry) and network issues for achieving that within GLOW
GLOW partners will discuss observing options, plus use of other
stations (Uw, Ju)
Many other network experts within GLOW who know more
about the “network” details than I do
–
–
I know more about the station and data issues than I do the network
configuration
Many network experts at the various stations
LOFAR Solar KSP Meeting, Potsdam, 2009 Jun 25
James M Anderson
2/24
View of a LOFAR Station
●
●
●
LOFAR Solar KSP Meeting, Potsdam, 2009 Jun 25
No complete German
stations yet...
Each dipole/tile
polarization transmitted
along a cable back to the
station cabinet
Each signal digitized by
12 bit A/D converter at
160 or 200 MHz rate
James M Anderson
3/24
Inside a Station
LOFAR Solar KSP Meeting, Potsdam, 2009 Jun 25
James M Anderson
4/24
Control Overview
LOFAR Solar KSP Meeting, Potsdam, 2009 Jun 25
James M Anderson
5/24
Types of Data Flow for LOFAR Stations
●
In: Station control commands
●
Out: Beamformed data
●
–
Main LOFAR observational data product
–
Summation of antenna signals for specific direction(s) on sky (beams)
–
248 subbands maximum (subbands reused for 8 and 4 bit modes)
–
Up to 3.1 Gb/s
Out: TBB data
–
●
–
Typically a small load, < 1 Gb/s
–
May be up to 5 Gb/s for certain single-station applications
Out: Monitor information
–
●
A/D time series for each antenna/polarization, or polyphase filter values as function of time for
each antenna/polarization
Receiver temperatures, coarse station spectra, GPS information, etc.
Out: Station visibilities
–
1 subband per second, for all receiver units
LOFAR Solar KSP Meeting, Potsdam, 2009 Jun 25
James M Anderson
6/24
Keeping Things Organized: VLANs
●
●
●
In ASTRON's LOFAR plan, all
LOFAR data travels down a
single 1x10 GE fiber
connection
Each data type separated by
different VLAN IDs
About 11 standard LOFAR
VLAN IDs in use
–
LOFAR Solar KSP Meeting, Potsdam, 2009 Jun 25
Many different types of
monitoring data in previous slide
are separated
James M Anderson
7/24
QinQ
●
●
12 bit VLAN ID --- 4096 IDs for the entire world
VLAN IDs chosen by ASTRON for LOFAR are often standard
IDs used for various applications
–
●
●
●
1 through 100 filled with other uses
VLAN IDs must be unique for security reasons
LOFAR VLAN IDs cannot make it through the various German
stations and Juelich
Solution is to use QinQ
–
–
Bundle all LOFAR VLAN traffic inside of another layer, with a
different VLAN ID
GLOW allocated range 2000 through 2100 for itself, to be kept free by
all GLOW stations and Juelich
LOFAR Solar KSP Meeting, Potsdam, 2009 Jun 25
James M Anderson
8/24
10 GE Network
●
●
Following the upgrade of the LOFAR correlator (Blue Gene
supercomputer) from the L to P class, ASTRON decided to
switch to using 10 GE technology for LOFAR
–
Blue Gene/P only has 10 GE connections
–
10 GE bid cheaper than 3x1 GE bid
–
Extra bandwidth allows more things to be done
International partners told to use 10 GE if at all possible
–
●
Currently only a (money) problem for Tb and Uw
Note that the fiber connection provides 10 Gb/s each in both
directions
–
Can also shove 10 Gb/s to a station
LOFAR Solar KSP Meeting, Potsdam, 2009 Jun 25
James M Anderson
9/24
Dutch Core Solution
●
●
Fibers from nearly all Dutch stations are first sent to a central
collector buiding near the Exloo core
Data then sent along a LOFAR fiber path to Groningen
LOFAR Solar KSP Meeting, Potsdam, 2009 Jun 25
James M Anderson
10/24
International Network
●
German stations send all
LOFAR data through
Juelich supercomputing
center
–
–
●
LOFAR Solar KSP Meeting, Potsdam, 2009 Jun 25
Data from different stations
bundled onto common fibers
Allows German stations to
efficiently operate a German
subarray
Cb, On, and Nc will have
LOFAR data go through
Amsterdam
James M Anderson
11/24
German Network Continued
LOFAR Solar KSP Meeting, Potsdam, 2009 Jun 25
James M Anderson
12/24
German Network and Juelich
LOFAR Solar KSP Meeting, Potsdam, 2009 Jun 25
James M Anderson
13/24
Juelich
●
●
Research Center near Dutch
boarder
GLOW member,
participating in networking/
storage/computation work in
LOFAR
–
●
●
LOFAR Solar KSP Meeting, Potsdam, 2009 Jun 25
Will also host the German
university station
Providing 1 PB of archive
storage for early LOFAR
work
Can potentially house
computational and extra
storage for LOFAR/GLOW
James M Anderson
14/24
Groningen Overview
●
●
●
Main processing center for LOFAR (correlation and initial
calibration and imaging pipelines)
Blue Gene used as correlator and data formatter
Calibration, RFI flagging, imaging, and so on happens in the
off-line cluster for now
LOFAR Solar KSP Meeting, Potsdam, 2009 Jun 25
James M Anderson
15/24
LOFAR Solar KSP Meeting, Potsdam, 2009 Jun 25
James M Anderson
16/24
Basic Network Topology for GLOW
LOFAR Solar KSP Meeting, Potsdam, 2009 Jun 25
James M Anderson
17/24
Notes on GLOW Topology
●
●
Nearly every (TBD) GLOW station will have local data storage
capability, with 10s of TB each
Many stations will have local processing capability for singlestation operation
–
●
Processing power needs (# CPUs) to be worked out in the future as we
figure out how much we need
Also have the ability to gather station data in Juelich
–
Tap off data-stream to Groningen for monitoring
–
Recording and/or processing at Juelich planned for the future
●
●
Multiple single-station or subarray uses
Can redirect data from other stations
–
–
In principle possible to do at both Juelich and Groningen, but
Groningen will likely have ability first
Can directly pass other International or Dutch station data to Juelich or
to a German station
LOFAR Solar KSP Meeting, Potsdam, 2009 Jun 25
James M Anderson
18/24
Station Data: Individual Antenna Spectra
●
●
●
LOFAR Solar KSP Meeting, Potsdam, 2009 Jun 25
Station hardware records
average spectra over ~1 min
intervals for individual
antennas
Example LBA spectra shown
here
Relatively low data rate, but
also poor frequency and time
resolution
James M Anderson
19/24
LBA Example Dynamic Spectrum
●
●
●
LOFAR Solar KSP Meeting, Potsdam, 2009 Jun 25
Made from a time series from
an antenna on the previous
plot
Can see Milky Way rise and
fall over several days
Low frequency RFI also
shows diurnal patern
James M Anderson
20/24
Station Data: Visibilities
●
●
●
●
Example Tb LBA image
shown
Can get visibilities for 1
subband (~ 200 kHz) once
per second
All-sky imaging for LBA
Can image full tile beam for
HBA
–
●
LOFAR Solar KSP Meeting, Potsdam, 2009 Jun 25
If certain antennas within tiles
are shut off, can also make allsky images for HBA, but at
reduced sensitivity
Limited ability to remove
RFI, low data rate
James M Anderson
21/24
Station Data: Beamformed Data
●
●
●
●
LOFAR Solar KSP Meeting, Potsdam, 2009 Jun 25
Station electronics coherently
sums antenna signals together
for specific sky direction
Summation done in ~200
kHz polyphase filter
subbands
Up to 248 subbands can be
sent out from the station,
with subband samples every
~5 μs
Can process outside of
station to get higher
frequency resolution, with
corresponding time
resolution loss
James M Anderson
22/24
Beamformed Data
●
●
●
●
●
Data comes out of station in a raw format
Software currently running on Blue Gene to convert the raw
format to HDF5 files for further processing
Documentation on current raw beamformed data format made
available during Technical Status Meeting in May
GLOW effort to port software to run on standard Linux
computer needs to be finished
Many single-station and interferometer subarray applications
will probably want to read the raw beamformed data coming off
the network to process in realtime
–
Lots of development work ahead
LOFAR Solar KSP Meeting, Potsdam, 2009 Jun 25
James M Anderson
23/24
The End
LOFAR Solar KSP Meeting, Potsdam, 2009 Jun 25
James M Anderson
24/24