venabili
“a computer and software resource for the EISCAT users”
­and EISCAT_3D users!
20 May 2010
EISCAT_3D
Venabili
venabili
●
What venabili aims to do
●
Approach
●
Technology adopted and used
●
So far what's working
●
What next?
●
My e3d wishlist
20 May 2010
EISCAT_3D
Venabili
venabili
“making the raw data access really elegant”
“any and all data in the archive”
20 May 2010
EISCAT_3D
Venabili
Approach
User login accounts
Two­pronged approach:
●
Detailed analysis
– Raw data experiments
●
Bulk processing
– Statistical studies
– Anomalous spectra
6 Aug 2009
EISCAT Workshop
Venabili
Technology
●
Commercial hardware
●
Open­source software
–
●
No licensed software!
Easy to administer
–
Familiar stuff NFS, NIS, DRBD, HTTP, IP networking etc
●
Robust and redundant to survive failures
●
Reporting
–
Usage
–
Problems
20 May 2010
EISCAT_3D
Venabili
So far we have...
●
●
Databases
–
All parameter blocks
–
Search and retrieve
Pre­processing software
–
●
RAL software, integration etc
Guisdap
–
20 May 2010
Ported to Octave v3.0
EISCAT_3D
Venabili
Database
search ­t 2008­07­28 ­T 2008­08­03
Start
End
Ants
Modltn Pattern
2008/07/31 13:00:12 2008/08/01 06:00:00 42m ipy fixed42p
2008/07/31 15:52:35 2008/08/01 19:00:05 tro beata cp1
search ­t 2008­07­28 ­T 2008­08­03 ­x tellMe
search ­t 2008­07­28 ­T 2008­08­03 ­x notify
20 May 2010
EISCAT_3D
Venabili
scintiSrc
Options:
--version
-h, --help
show program's version number and exit
show this help message and exit
Scintillation source:
Right ascension and declination, values from user parameters in
parameter block.
-a RA, --ra=RA
Right ascension, degrees.
-d DEC, --dec=DEC
Declination, degrees.
-f FREQ, --freq=FREQ
Frequency, MHz.
Output options:
Specify what output is wanted.
-v, --verbose
make lots of noise [default]
-x ACTION, --exec=ACTION
What action to be taken with search results. Can be
one of summary, tellMe, stage, notify. [Default:
summary].
Antenna identification parameters:
Options can only be single valued, not a list, last value specified
used.
-r ANT, --antenna=ANT
Receiving antenna name. Can be one of 32m, 42m, vhf,
tro, kir, sod, 32p.
-s SITE, --site=SITE
Site identifier. Can be one of L, R, T, U, V.
Date and times:
Use ISO format, YYYY-MM-DDTHH:MM:SS, truncated fields will be padded
with appropriate values. Try and specify dates, it helps constrain the
search.
-t START, --start=START
Start date/time.
-T END, --end=END
End date/time.
20 May 2010
EISCAT_3D
Venabili
Detailed software
●
chord
–
●
sinter
–
●
20 May 2010
Form lag profile matrices
genLR
–
●
Velocity combination
genLP
–
●
Sub­second integration
velcom
–
●
Sliced data
Form ACF's
plotting
EISCAT_3D
Venabili
●
●
Guisdap
–
Make it quicker
–
Integration strategies
Distributed processing
–
●
Sun's Grid Engine
Metadata
–
●
Now what?
150 modulation schemes?
Madrigal
20 May 2010
–
Multi­user and multi­version
–
And plotting. And all the other stuff.
EISCAT_3D
Venabili
Metadata
●
●
By metadata, I mean...
–
Insulating layer between hardware and users
–
Webserver, database, users, applications
Elegance
–
●
Making it look simple
Understanding
–
20 May 2010
Naming the parts
EISCAT_3D
Venabili
Metadata – first cut
+­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­+­­­­­­+­­­­­+­­­­­­­­­+­­­­­­­­­­­­­­­­+
| Field | Type | Null | Key | Default | Extra |
+­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­+­­­­­­+­­­­­+­­­­­­­­­+­­­­­­­­­­­­­­­­+
| modSchm | mediumint(8) unsigned | NO | PRI | NULL | auto_increment | | version | decimal(5,2) | NO | | 1.00 | | | name | varchar(32) | NO | | NULL | | | site | enum('L','T','V') | NO | | NULL | | +­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­+­­­­­­+­­­­­+­­­­­­­­­+­­­­­­­­­­­­­­­­+
+­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­+­­­­­­+­­­­­+­­­­­­­­­+­­­­­­­+
| Field | Type | Null | Key | Default | Extra |
+­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­+­­­­­­+­­­­­+­­­­­­­­­+­­­­­­­+
| modSchm | mediumint(8) unsigned | YES | MUL | NULL | | | type | enum('s','b','c') | YES | | b | | | lag | decimal(5,1) | YES | | 0.0 | | | h | decimal(8,1) | YES | | NULL | | | width | decimal(5,1) | YES | | NULL | | | dt | decimal(5,1) | YES | | NULL | | | ND | smallint(5) unsigned | YES | | 1 | | | T | smallint(5) unsigned | YES | | 0 | | | nt | mediumint(8) unsigned | NO | | NULL | | | ra | mediumint(8) unsigned | NO | | NULL | | | ri | mediumint(8) unsigned | NO | | NULL | | | code | tinyint(3) unsigned | YES | | NULL | | +­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­­­+­­­­­­+­­­­­+­­­­­­­­­+­­­­­­­+
20 May 2010
+­­­­­­­­­­­­­­­­+
| Tables_in_meta |
+­­­­­­­­­­­­­­­­+
| expId | | lpg | +­­­­­­­­­­­­­­­­+
EISCAT_3D
Venabili
Metadata
+­­­­­­+­­­­­­+­­­­­­­­­­+­­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­­+­­­­­­­­­­­­­­­­­­­
+­­­­­­+
| code | type | min(lag) | max(lag) | count(distinct lag) | min(h) | min(h)+dt*max(nt) | dt |
+­­­­­­+­­­­­­+­­­­­­­­­­+­­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­­+­­­­­­­­­­­­­­­­­­­
+­­­­­­+
| 1 | s | 0.0 | 240.0 | 25 | 750.0 | 3750.0 | 10.0 | | 1 | b | 0.0 | 240.0 | 25 | 7970.0 | 10970.0 | 10.0 | | 1 | c | 0.0 | 0.0 | 1 | 11410.0 | 11610.0 | 10.0 | +­­­­­­+­­­­­­+­­­­­­­­­­+­­­­­­­­­­+­­­­­­­­­­­­­­­­­­­­­+­­­­­­­­­+­­­­­­­­­­­­­­­­­­­
+­­­­­­+
20 May 2010
EISCAT_3D
Venabili
Metadata
20 May 2010
EISCAT_3D
Venabili
Need more metadata?
●
Other sources
– GDEF, tarlan
●
Metadata query/usage examples
– Implemented using MySQL, it has a query language!
20 May 2010
EISCAT_3D
Venabili
And then?
Get people using it!
3000 FAST satellite conjunctions
6 Aug 2009
EISCAT Workshop
Paul GallopVena
And then?
Re­analyse data?
●
Consider models and integration?
●
Filter, or clean the data?
●
Metadata revisited.
●
Results “databases”?
●
Test suite, validate porting work.
20 May 2010
EISCAT_3D
Venabili
Wishlist
“easily accessible data”
●
Spectra, ACF's
●
All data in one file
●
Full description of samples in data file
●
Open source repository
●
Results “database”
20 May 2010
EISCAT_3D
Venabili