Clock Synchronisation for RTS Dr. Hugh Melvin, Dept. of IT, NUI,G 1 Importance of RTS Clocks • RealTime implies need for accurate timekeeping • Precise Time & Timing – Two distinct issues – Both often required for RTS • Time (of day) – UTC (Universal Coordinated Time) – Chronological event recording – Crucial for fault diagnosis in distributed systems • Timing – Frequency is key issue – Crucial in synchronising components Dr. Hugh Melvin, Dept. of IT, NUI,G 2 Time & Timing • Time – UTC evolved from Greenwich Mean Time GMT – Time Synchronisation • How closely 2 clocks agree on Time of Day • Timing • How closely 2 entities operate at same frequency Dr. Hugh Melvin, Dept. of IT, NUI,G 3 1 Greenwich • Home of Time & the Prime Meridian • Longitude 0 & GMT agreed in 1884 – GMT driven by p of railways y expansion and need for national/international consistency • “Timeball” rises and falls at 13.00 hrs – Then site of London docks – In service since 1833 Dr. Hugh Melvin, Dept. of IT, NUI,G 4 Greenwich Dr. Hugh Melvin, Dept. of IT, NUI,G 5 18th Century GPS • 18th century maritime navigation – Latitude • Astronomical – Longitude dilemma • Solutions – Astronomical – Time based • Need for robust maritime clock Dr. Hugh Melvin, Dept. of IT, NUI,G 6 2 18th Century GPS • Time based location – Set ship clock before setting sail from time ball – Compare with 'local time' to give one's position east or west of the home port. • 360° = 24 hr 15 ° = 1 hr – Problem was getting clock to work well on a ship – Lincolnshire carpenter, John Harrison • H4 won him the great Longitude Prize of £20,000. .. in 1759 Dr. Hugh Melvin, Dept. of IT, NUI,G 7 Time & Timing Examples • Time • Distributed Control Systems : Moneypoint – Facilitates fault diagnosis • • • • • Power Line Fault Detection VoIP delay monitoring SLA adherence monitoring g systems y Billing Database/File Integrity timestamps • Timing • Synchronising redundant devices 2v3,2v4 voters etc • TDM within GSM/POTS – Avoids bit errors • MM applications : skew issues due to ‘timing’ mismatch – Delay/Buffer problems Dr. Hugh Melvin, Dept. of IT, NUI,G 8 Power System Control • AS station – Time: Token Bus Synchronisation via Master Clock • • • • Critical for chronological data logging / fault diagnosis Approx 1 msec level synch reqd Provided via GPS Generator Earth Fault / Overcurrent .. – Which came first .. msec level data required – Timing: Ti i S Synchronising h i i 2 2v3 3 voter t systems t • Need to deliver verdicts simultaneously • Power Line Fault Monitoring • Noise burst travels in both directions • Speed of light c = 3* 108 m/s 0.3 m / nanosec » synch level of usec needed (300 m) • Synchroscope – Frequency alignment Dr. Hugh Melvin, Dept. of IT, NUI,G 9 3 Token Bus : Master Clock U/IA U/IB U/IB U/IB U/IA 125 N16 R30 U/IA U/IB 103 N8 AS220E 102 N8 AS220E 101 N8 AS220E U/IA U/IA U/IB 123 N-UHR M-Clock U/IA 141 NAT-24 Synogate U/IA Master Clock 127 N-BK Bus 1 126 N-BK Bus 0 U/IB U/IA U/IB U/IA U/IB 104 N8 AS220E 105 N8 AS220E U/IB U/IA 121 N16 OS254 U/IA U/IB 160 NS5NAT PG750 U/IB 106 N8 AS220E U/IB 112 N8 AS220E U/IA U/IA U/IB 133 N8 AS EHF Dr. Hugh Melvin, Dept. of IT, NUI,G U/IA U/IB 107 N8 AS220E U/IB 111 N8 AS220E U/IA U/IA U/IB 132 N8 AS EHF U/IA U/IB 108 N8 AS220E U/IB 110 N8 AS220E U/IA U/IA U/IB 131 N8 AS EHF U/IA U/IB 109 N8 AS220E U/IA U/IB 128 N8 AS231 10 Power Line Fault Detection System Dr. Hugh Melvin, Dept. of IT, NUI,G 11 Power Line Fault Detection System Dr. Hugh Melvin, Dept. of IT, NUI,G 12 4 Timing : 2v3 Redundancy Bus A Bus B CS275 CPU 1 .. U/I U/I N8 N8 CPU 2 CPU 3 .. .. .. .. .. 2 out of 3 Non-Redundant Control Circuits Double Redundant Safety Circuits Dr. Hugh Melvin, Dept. of IT, NUI,G Triple Redundant Safety Circuits 13 Timing: Synchroscope • Frequency alignment between generator and transmission grid • No room for significant error Dr. Hugh Melvin, Dept. of IT, NUI,G 14 Synchroscope Dr. Hugh Melvin, Dept. of IT, NUI,G 15 5 Telecommunications • Precise timing synch fundamental to current systems – POTS • SDH/SONET – Cellular Networks • GSM/CDMA – Loss of synch • Bit errors (slip) data loss • System failure • IP-based NGN – Circuit Packet switched network • Loss of core synch key challenge • http://forum.telecom-sync.com/ Dr. Hugh Melvin, Dept. of IT, NUI,G 16 Dr. Hugh Melvin, Dept. of IT, NUI,G 17 Dr. Hugh Melvin, Dept. of IT, NUI,G 18 6 Dr. Hugh Melvin, Dept. of IT, NUI,G 19 Soft RTS • POTS operation based on TDM • PCME1E2..E4 SDH/SONET • Precise timing synchronisation reqd throughout the network for correct system operation • GSM : FDM + TDM • Each FDM channel divided out to 8 users via TDM Dr. Hugh Melvin, Dept. of IT, NUI,G 20 Soft RTS • IP Multimedia Applications – Time • Delay / Jitter measurement imp in packet (IP) networks • More advanced QoS possible through synchronised time – Recall G G.1010 1010 – Basis of SLA delay/jitter measurement important – Timing • Skew Issues between various clocks – Time & Timing • Lip Synch challenge • MMOG Dr. Hugh Melvin, Dept. of IT, NUI,G 21 7 Dr. Hugh Melvin, Dept. of IT, NUI,G 22 Dr. Hugh Melvin, Dept. of IT, NUI,G 23 Audio-System Clock Skew Dr. Hugh Melvin, Dept. of IT, NUI,G 24 8 Timing ‘Skew’ for IP Multimedia Dr. Hugh Melvin, Dept. of IT, NUI,G 25 Lip Synch ETSI STQ C Undetectability plateau C' Diffgrade) Subjective evaluation results (D 0 B B' Detectability threshold -0,5 -1 -1,5 Acceptability threshold A A' Sound delay wrt vision Sound advanced wrt vision -2 -200 -180 -160 -140 -120 -100 -80 -60 -40 -20 Delay time (ms) 0 20 40 60 80 Dr. Hugh Melvin, Dept. of IT, NUI,G 100 26 Synch Time for MMOG Dr. Hugh Melvin, Dept. of IT, NUI,G 27 9 Computer Clocks • Most commonly consist of quartz crystal and a counter • Crystal oscillates at defined rate (Hz) which generates a consistent tick and increments a software counter • Counter C t value l ttranslated l t d to t time ti standard t d d – UTC (Univ. Coord. Time) .. Based on GMT • Primary Source: Atomic Clocks – TAI (International Atomic Time) » But requires leap seconds every few years! » UTC = TAI + Leap_Seconds • Crystal Quality described by accuracy & stability Dr. Hugh Melvin, Dept. of IT, NUI,G 28 Computer Clocks • Accuracy relates to how close the crystal freq is to its rated value – Determined largely by manufacturing process • Get what you pay for! • Stability relates to how frequency varies – Influenced f by parameters such as: • Temperature .. Eg. 2ppm /C • Ageing – Eg. Cesium Beam: 3 x 10-12 / year • Noise • Note: Frequency i.e. timing errors lead to time (of day) errors Dr. Hugh Melvin, Dept. of IT, NUI,G 29 Computer Clocks • Improved Quality Timekeeping ? – Option A: Stick with crystals • Precision manufacturing costly • Temperature Compensated Crystal Osc.(TCXO) • Oven Controlled Crystal Osc.(OCXO) – Option B • • • • : Buy an Atomic Clock .. or GPS Receiver (based on atomic clock) .. or Radio Receiver DCF77, MSF .. LFR signal .. less accurate GPS most popular approach to providing accurate/stable time – Option C : Cheaper Approach • Software based approach to discipline cheap crystal clocks • Crude but useful for certain applications Dr. Hugh Melvin, Dept. of IT, NUI,G 30 10 Dr. Hugh Melvin, Dept. of IT, NUI,G 31 Dr. Hugh Melvin, Dept. of IT, NUI,G 32 Clock Options • TCXO – 105 µsec / day = 1.15 10-6 freq instability • 1.15 ppm • OXCO – 10 µsec / day = 1.15 10-10 freq instability • 1.15 10-4 ppm • Cesium – 10-3 µsec / day = 1.15 10-14 freq instability • 1.15 10-8 ppm Dr. Hugh Melvin, Dept. of IT, NUI,G 33 11 Clock Terminology • Some confusion with terms in literature – Paxson/Mills terminology used here – Offset • Difference between time reported by clock C, C(t) and true clock (UTC) at true time t. • Also relative offset between clocks C1and C2 – C1(t) - C2(t) – Skew • Difference in frequency between clock C and a true clock (UTC) , C’(t) • Defined in ppm (usec per sec) • +/-12 ppm approx = +/- 1 sec/day • Also relative skew between clocks C1and C2 – C1’(t) - C2’(t) Dr. Hugh Melvin, Dept. of IT, NUI,G 34 Dr. Hugh Melvin, Dept. of IT, NUI,G 35 Clock Terminology • Skew – A large skew rate rapidly increasing offset frequent resynchronisation – If specify max abs skew rate for clock C of (1 )(t2 t1) C (t2 ) C (t1) (1 )(t2 t1) – Clock should operate within cone of acceptability • Drift – Rate of change of frequency C’’(t) • Eg. Ageing influence or change in temperature – Not usually significant except over long timescales – Note linear relationship in previous slide • No drift evident Dr. Hugh Melvin, Dept. of IT, NUI,G 36 12 Cone of Acceptability Slope = 1 + Slope = 1 = True Clock Clock Time Sl Slope =1- Real Time Q: How are accuracy and stability related to cone of acceptability? Dr. Hugh Melvin, Dept. of IT, NUI,G 37 Clock Synchronisation • Perfect clocks do not exist • Eg. PC System Clock NTP Server GPS Receiver GPS Atomic Clock GPS Master Atomic Clock ?? • Examine two separate scenarios • Localised Cluster of Clocks – Eg. Power System Control / Fly-by-wire Systems – Also widely distributed clocks over deterministic network » Propagation time known (can be compensated for) • Distributed clocks over non-deterministic network – More difficult scenario – Eg. Internet Clock Synchronisation via NTP Dr. Hugh Melvin, Dept. of IT, NUI,G 38 Localised Cluster of Clocks • Hardware-based Phase Locked Loops (PLL) – Oscillator output is aligned to the input signal. – Input signal can come from a • Master Clock • Combination of outputs from all other clocks – Input signal used to drive its PLL – Can also compensate for Propagation Delay variations – Expensive but precise approach • Resolve timing errors accurate time • Similar approach used in widely distributed scenario – GPS / POTS / GSM all use variants of this approach • But only for timing synchronisation .. Not really concerned with time Dr. Hugh Melvin, Dept. of IT, NUI,G 39 13 PLL Input Signal Comparator VCO VCO = Voltage Controlled Oscillator Freq controlled by applied input voltage Dr. Hugh Melvin, Dept. of IT, NUI,G 40 Distributed Clocks • More difficult environment if underlying network non deterministic • Expense of hardware based approach cannot be justified for many Soft-Firm RTS • Cheap software based approach – Network Time Protocol (NTP) (www.ntp.org) – RFC 1305 (www.ietf.org) • Unix-based NTP daemon now ported to most OS Dr. Hugh Melvin, Dept. of IT, NUI,G 41 NTP Clock Synchronisation • More concerned with time rather than timing • Some general principles – Fault Tolerance critical • Identify and isolate faulty clocks • Note: A faulty clock is one that does not operate within cone of acceptability – Cf Clock Quality: May be stable but inaccurate – Avoid setting clocks backward – Event processing nightmare – OS problems eg. Timers / timeslicing – Avoid large step changes • Amortize the required change (+/-) over a series of short intervals (eg. over multiple ticks) Dr. Hugh Melvin, Dept. of IT, NUI,G 42 14 NTP • Network Time Protocol (NTP) synchronises clocks of hosts and routers in the Internet – Determines timing errors and compensates to produce accurate time • Increasingly deployed in the Internet – Increased need for time synchronisation – Facilitated via always-on Internet connection • Can provide nominal accuracies – low milliseconds on WANs – submilliseconds on LANs – submicroseconds on workstations • using a precision time source cesium oscillator/GPS Dr. Hugh Melvin, Dept. of IT, NUI,G 43 NTP The NTP architecture, protocol and algorithms have evolved over the last twenty years to the latest NTP Version 4 • Internet standard protocol for time synchronisation and coordinated time distribution using UTC • Fault tolerant protocol – automatically selects the best of several available time sources to synchronise with • Highly scalable – nodes form a hierarchical structure with reference clock(s) at the top – Stratum 0: Time Reference Source • GPS / GOES (GeoSat) / LORC (LoranC) / ATOM / DTS – Stratum 1: Primary Time Server Dr. Hugh Melvin, Dept. of IT, NUI,G 44 NTP System GPS/Radio Clock Timin g S igna l Timing Signal GPS Satellite Timing Signals NTP NT P P NT NTP Primaryy Server Stratum 1 NTP Secondary Server Stratum 3 NTP Sec. Server Strat. 2 P NT NTP Sec. NT Server Strat. 2 P NTP Secondary Server (Stratum 3) Dr. Hugh Melvin, Dept. of IT, NUI,G NTP Secondary Server Stratum 3 45 15 NTP Operation Peer 1 Filter 1 Peer 2 Filter 2 Peer 3 Intersection and Clustering Algorithms Combining Algorithm Loop Filter P/F-Lock Loop Filter 3 VFO NTP Messages • Complex Software comprising various algorithms • Filtering Alg. • Clustering and Intersection Alg. • Combining Alg. • Clock Discipline Dr. Hugh Melvin, Dept. of IT, NUI,G 46 NTP Operation • NTP Algorithms act upon a set of variables – Offset / Delay / Dispersion – Dispersion n 1 j ij wi 1 i 0 ij i j • w = 0.75 • These are relative to both peer and root Offset Delay Dispersion θ δ ε Θ Δ Ε Dr. Hugh Melvin, Dept. of IT, NUI,G 47 Client Server Mode • UDP/IP packets for data transfer – Several packet exchanges between client/server – Client • originate timestamp A within packet being sent. – Server receives such a p packet: • receive timestamp B • transmit timestamp C – Client • Processes A,B,C as well as final packet arrival D • Determine offset and Round Trip Delay (RTD) • Note: RTD != RTT Dr. Hugh Melvin, Dept. of IT, NUI,G 48 16 NTP Operation B 3.59.020 C 3.59.022 15 ms 15 ms A 3.59.000 D 3.59.032 Symmetric Network : 15 ms each way (actual delay) RTD = (D - A) – (C – B) = 32 – 2 = 30 msec (RTT =?) Offset = ½[(B-A) - (D-C)] = (20 – 10)/2 = 5 ms Dr. Hugh Melvin, Dept. of IT, NUI,G 49 Filtering Algorithm • Filtering algorithm looks at last 8 samples • Reduces offset errors by a factor of about ten • Effective at removing spikes •Chooses sample with min RTD Dr. Hugh Melvin, Dept. of IT, NUI,G 50 Intersection Algorithm 1 2 4 3 Clocks 1, 2 ,3 are truechimers 4 is a falseticker X1 X2 • • • • Selects a subset of peers Based on intersection of confidence intervals Identifies truechimers & falsetickers eg. From 1,2,3,4 above Dr. Hugh Melvin, Dept. of IT, NUI,G 51 17 Intersection Algorithm • Estimated offset to each clock is mid pt But: Any point in each confidence interval may represent actual time as seen by that peer • • If clocks 14 are correct, there must exist a common intersection Clock Cl k 4 mostt lik likely l iincorrect.. t di disregard d • Interval X1 = smallest intersection containing points from 1,2,3 • But also include the max no of interval midpoints – Select X2 interval – Could select mid pt of X2 .. or refine further Dr. Hugh Melvin, Dept. of IT, NUI,G 52 Clustering (Clock Selection) • Sort surviving clocks by stratum and incr synch distance (RTD/2 + disp), S1 S2 S3 • Remove outliers that have significant dispersion relative to other survisors – Compute Select Dispersion of each clock • Weighted sum of differences to other clocks – Compute Sample Dispersion of each clock • Weighted sum of diff relative to past samples of same clocks – If Max SelDisp > Min SamDisp • Remove this survivor and repeat • Favours candidates at start of sorted list Favours lowest stratum / delay Dr. Hugh Melvin, Dept. of IT, NUI,G 53 Clustering algorithm Sort survivors of intersection algorithm by increasing synchronization distance(RTD/2 + dispersion). Let n = no of survivors and nmin a lower limit (eg.3). For each survivor si, compute the select dispersion (weighted sum of clock difference) between si and all others. Let smax be the survivor with max select dispersion (relative to all other survivors)) and smin the survivor with min sample p dispersion p (clock differences relative to past samples of the same survivor). smax smin or n nmin yes no Delete the survivor smax; reduce n by one The resulting survivors are processed by the combining algorithm to produce a weighted average used as the final offset adjustment Dr. Hugh Melvin, Dept. of IT, NUI,G 54 18 Combining Algorithm • Combine result from survivors of selection algorithm • Weighted offset determined based on – Offset of survivors Θ – Synchronisation distance Λ – Eg. 2 survivors (S1,S2) with parameters (1 , 1 ), ( 2 , 2 ) – Final Offset = (1. 2 2 .1 ) /(1 2 ) Dr. Hugh Melvin, Dept. of IT, NUI,G 55 Combining Algorithm Example (1. 2 2 .1 ) /( 1 2 ) • S1,S2 where S1 = (2 ms, 30) and S2 = (3 ms, 10) • Final Fi l Adjustment Adj t t= – (2(10) + 3(30)) / (30 + 10) = 110 / 40 = 2.75 msec • Implemented via the Clock Discipline Dr. Hugh Melvin, Dept. of IT, NUI,G 56 Clock Discipline • Recall – No time reversal! – Avoid step changes • Hybrid phase/frequency phase/frequency-lock lock (PLL/FLL) feedback loop • PLL/FLL Mode: Depends on polling interval Dr. Hugh Melvin, Dept. of IT, NUI,G 57 19 PLL and FLL weight factors • Weight factors (not to scale) • PLL predict (red) most important at shorter poll intervals to 24 s • FLL predict (blue) most important at longer poll intervals to 217 s Dr. Hugh Melvin, Dept. of IT, NUI,G 58 Clock Models • Unix Clock Model • settimeofday( ), adjtime( ) • Kernel variables tick , tickadj • adjtime adjusts clock every tick – Can amortise reqd change gradually by making adjustment every tick eg. every 10 msec – Note: Newer Unix/Linux kernels 1000Hz 1msec • 3 clock rates – Normal rate .. Add 10 msec every tick (100 Hz) – Normal Rate +/- tickadj – Eg. If tickadj = 5us Normal Rate +/- 500 ppm Dr. Hugh Melvin, Dept. of IT, NUI,G 59 Simplified NTP Operation • NTP adjusts every sec via adjtime – Eg. If clock skew is +100 ppm & tickadj=5us – NTP will operate to keep clock effectively running at correct rate » Normal Rate - 500 ppm over 0.2 sec » Normal Rate for 0.8 sec » Effective skew = 0 ppm – Results in sawtooth – pattern • Newer Unix Kernels have advanced NTP features – ntp_adjtime( ), ntp_gettime() – Eliminates the sawtooth pattern • NTP maintains accurate time by resolving and compensating for timing errors Dr. Hugh Melvin, Dept. of IT, NUI,G 60 20 NTP Implementation • Install NTP • Set up ntp.conf file – List of servers that you wish to connect to – Need to consider • Redundancy, y, Path Diversity, y, Low RTD • Start up NTP daemon ntpd • File ntp.drift records clock skew – Unfortunate file name .. Adds to confusion ! • Other utilities – ntpq, ntpdate – See www.ntp.org Dr. Hugh Melvin, Dept. of IT, NUI,G 61 Refid: DCF: 77.5 KHz Radio Signal PTB: German time signal Dr. Hugh Melvin, Dept. of IT, NUI,G 62 Dr. Hugh Melvin, Dept. of IT, NUI,G 63 21 Time difference Dr. Hugh Melvin, Dept. of IT, NUI,G 64 Server Details • when: no of sec since last response • poll : interval between queries • reach : Reachability in octal – 11111111 = 3778 = max – 11101110 = 3568 last + 5th probe lost • Symbol to LHS of server – * : Synch Source – survivor with smallest dispersion – + :other candidates included in final combination alg – - : Discarded by clustering alg – x : Falseticker acc to intersection alg Dr. Hugh Melvin, Dept. of IT, NUI,G 65 Dr. Hugh Melvin, Dept. of IT, NUI,G 66 22 NTP Robustness Issues • • • • Redundancy Path Diversity y Networks Symmetric Proximity to Primary Reference Sources – See results • OS & Network Load – Platform Dependencies Dr. Hugh Melvin, Dept. of IT, NUI,G 67 NTP Operation : Asymmetry B 3.59.015 C 3.59.017 10 ms 20 ms A 3.59.000 D 3.59.032 Offset still 5 ms but Asymmetric Network RTD = (D - A) – (C – B) = 32 – 2 = 30 msec Offset = ½[(B-A) - (D-C)] = (15 – 15)/2 = 0 ms .. Error Dr. Hugh Melvin, Dept. of IT, NUI,G 68 NTP Operation : Asymmetry B 3.59.015 C 3.59.017 15 ms A 3.59.000 15 ms D 3.59.032 NTP’s Symmetric view of Asymmetric Network RTD = (D - A) – (C – B) = 32 – 2 = 30 msec Offset = ½[(B-A) - (D-C)] = (15 – 15)/2 = 0 ms ! Exercise: What is the maximum error in this calculation? Dr. Hugh Melvin, Dept. of IT, NUI,G 69 23 Dr. Hugh Melvin, Dept. of IT, NUI,G 70 Dr. Hugh Melvin, Dept. of IT, NUI,G 71 Server Offsets: Problem? Dr. Hugh Melvin, Dept. of IT, NUI,G 72 24 NTP at NUI Galway • Public NTP servers – ntp-galway.hea.net since 2002 – 2009 • 25,000 different clients • 62,000 requests per hour • Symmetricom S300 Server – GPS – DCF 77 – Oven Crystal – Secondary server ntp2.it.nuigalway.ie • Trimble GPS • Linux based PC Dr. Hugh Melvin, Dept. of IT, NUI,G 73 Dr. Hugh Melvin, Dept. of IT, NUI,G 74 Dr. Hugh Melvin, Dept. of IT, NUI,G 75 25 ntp-galway.hea.net • GPS receiver • DCF 77 Radio • Also peers with – Computer Services – ntp2.it.nuigalway.ie – Dublin, UK, mainland european servers Dr. Hugh Melvin, Dept. of IT, NUI,G 76 26
© Copyright 2026 Paperzz