Page 1 of 1
| Hong Kong MTR trains collided when testing SelTrac CBTC 20/03/2019 at 18:28 #116470 | |
alvinhochun
249 posts |
News article from SCMP, photos included. I thought this might be of interest to some of you here (if you haven't already heard of it through other channels), as this is a catastrophic failure of a modern signalling system. Luckily it's only a test and no passengers are harmed (though a train operator did get injured). The aftermath took two days and nights to clean up Short version of what happened: A train was leaving the terminus station platform to use a double crossover to get to the opposite track. Another train approached the station, aimed for the opposite platform and entered the double crossover while the first train was crossing. The end result is a side collision with derailment. The most interesting part of this event is a picture (attached) of the signalling screen that circulated on the Internet after the incident. It appears to show two conflicting routes being set, as indicated by the two paths with arrows along them (I believe this is how the screen works). It is said that Thales (the software supplier) was able to reproduce the incident with their simulation and thought it would be a software bug. I'm in disbelief that a software bug could cause the interlocking to be ignored. Completely unacceptable for a safety-critical system. Good thing they managed to catch it during testing! But seriously, they should've caught it during earlier simulation runs, given that they could reproduce it afterwards.  Post has attachments. Log in to view them. Log in to reply |
| Hong Kong MTR trains collided when testing SelTrac CBTC 20/03/2019 at 19:30 #116475 | |
madaboutrains
316 posts |
Visually that is the same signalling system used on the Jubilee, Northern Lines and as of Monday Morning Hammersmith to Latimer Road.
RIP Feltham Panel 1 Log in to reply |
| Hong Kong MTR trains collided when testing SelTrac CBTC 20/03/2019 at 20:24 #116477 | |
Edgemaster
332 posts |
madaboutrains in post 116475 said:Visually that is the same signalling system used on the Jubilee, Northern Lines and as of Monday Morning Hammersmith to Latimer Road. Sunday afternoon, I had a ride on it shortly after opening for public service 
Log in to reply |
| Hong Kong MTR trains collided when testing SelTrac CBTC 20/03/2019 at 20:41 #116478 | |
|
Jan
906 posts |
madaboutrains in post 116475 said:Visually that is the same signalling system used on the Jubilee, Northern Lines and as of Monday Morning Hammersmith to Latimer Road. Also the trunk line of San Francisco's Muni Metro, which uses Seltrac, too. Two million people attempt to use Birmingham's magnificent rail network every year, with just over a million of them managing to get further than Smethwick. Log in to reply |
| Hong Kong MTR trains collided when testing SelTrac CBTC 20/03/2019 at 21:29 #116479 | |
GeoffM
6376 posts |
To be fair, there are several versions of SelTrac. One media report stated that the Singapore metro (or part of) would need the same fixes, but did not mention other installations, whether through a lack of knowledge or because it's a different version, I don't know. On a side point, that scissors failure looks like an interlocking logic issue, not a software problem. But then some people refer to data as software so who knows. In the UK it would not be possible for all four point ends to be reversed with a route over any part, which nicely handles the diamond in the middle issue. SimSig Boss Log in to reply |
| Hong Kong MTR trains collided when testing SelTrac CBTC 21/03/2019 at 08:28 #116483 | |
|
alvinhochun
249 posts |
GeoffM in post 116479 said:To be fair, there are several versions of SelTrac. One media report stated that the Singapore metro (or part of) would need the same fixes, but did not mention other installations, whether through a lack of knowledge or because it's a different version, I don't know. Right. There are even several types of technologies used. The earlier SelTrac S40 uses interleaving transmission loops for both localization and data transmissions. This is the system used by the DLR, Jubilee line and Northern line if I haven't mistaken. The same system has also been used for a major rail line and a branch line in Hong Kong for a decade and a half. Some people apparently refer to this system as TBTC instead of CBTC. The failed signalling in question that MTR installed is the newer wireless CBTC one, not the loop-based one. From what I've heard it seems that the 4LM project is using a similar system. The spokespeople was saying that the incident happened when they switched to the "third backup system" (which they said is unique to MTR) as part of the test, so it could be that the switchover somehow caused issues. It may be irrelevant to other SelTrac CBTC systems around the world without the "third backup system". But then, PR speak isn't the best source for technical information. Same for the media. The local news are very bad at understanding the technical stuff. Log in to reply |
| Hong Kong MTR trains collided when testing SelTrac CBTC 21/03/2019 at 08:45 #116484 | |
|
kbarber
1743 posts 
|
So what is the third backup system and where, exactly, does it fit in the overall logic of the system? As Geoff says, what we see in that image is undoubdtedly an interlocking failure; even if the crossovers are not locked against each other (and with the UK history - and, I believe - a tradition of UK managers in MTR's hierarchy that continues to this day I would have expected MTR to follow UK practice), the signals most certainly should have been. Incidentally, Modern Railways carried a report that a software issue emerged during testing between Hammersmith and Praed Street, not safety critical (if I read the report correctly) but which they decided should be fixed before the system was commissioned, hence the long delay in commissioning. (It was originally supposed to be last summer if I understand correctly.) Log in to reply |
| Hong Kong MTR trains collided when testing SelTrac CBTC 21/03/2019 at 15:07 #116488 | |
|
metcontrol
227 posts |
The core system of both the Hong Kong and current LUL installations is the same, however what we have is different in certain aspects. The newest installation (on 4LM) is even different to that on the Jubilee and Northern - though from an operator perspective they look fairly identical. Hong Kong opted for the version where trains can literally go anywhere - backwards and forwards over the same piece of track / over points in any direction. We opted for a more restricted version, where certain locations have more flexibility but on the whole trains on say, the eastbound, will always operate in that direction. In the event of failure, we still need to employ safety procedures to run trains in the "wrong" direction. The main difference, as far as LUL is concerned, between TBTC and CBTC is the way in which the system talks to trains. The Northern and Jubilee use TBTC which speaks to trains via wired loops fixed to the track. CBTC used on 4LM uses radio beacons rather than wire loops to speak to trains. The main part of the system runs the service and keeps trains apart, monitors headways, and generally does what you would expect from such a system. However, on LUL, underneath the automation we still have a system of fixed "track circuit blocks." I use the term loosely as they are not wired in the same way as the old signalling (most are axle-counter blocks) but they perform much of the same job. They are the underlying safety system - the back-up insurance policy if you like. If they "fail" we can still use the CBTC system safely to work around the failure until they are fixed, albeit with restrictions. The first part of the 4LM commissioning went ahead on the weekend - it was in fact almost exactly a year late. There were some issues found during the testing process - some potentially serious if the right circumstances all happened to come together. All centred around operations during a failure situation and none were connected to the day-to-day running. These have been cured as far as possible and where the fix hasn't been all that was desired, certain restrictions have been put into place. I cannot obviously go into any further detail. Log in to reply The following users said thank you: alvinhochun, kbarber |
| Hong Kong MTR trains collided when testing SelTrac CBTC 05/07/2019 at 11:18 #119271 | |
|
alvinhochun
249 posts |
Back to this again: MTRCL published a press release on the investigation results, including some detailed findings in the annex. I don't yet have the time to read the whole thing, so I'll just put the link here without additional comments. http://www.mtr.com.hk/archive/corporate/en/press_release/PR-19-044-E.pdf Log in to reply |
| Hong Kong MTR trains collided when testing SelTrac CBTC 05/07/2019 at 14:08 #119273 | |
|
Edgemaster
332 posts |
In short: after the primary computer and first backup computers failed, the handover process to the secondary backup didn't transfer over the "Conflict Zone Data", and this data was not recalculated by the secondary backup, as it was supposed to (root cause being an oversight in the design documents). As a result, when setting routes, the system was consulting an empty "Conflict Zone Data" file and setting conflicting routes for trains as a result. For a system to fail in such a way worries me hugely. Interlockings really should be fail-safe... It rather surprised me that "Conflict Zone Data" was derived from some other source, and not hard coded into the interlocking data. Last edited: 05/07/2019 at 14:11 by Edgemaster Reason: None given Log in to reply |
| Hong Kong MTR trains collided when testing SelTrac CBTC 05/07/2019 at 14:29 #119274 | |
|
Danny252
1461 posts |
I was also very surprised that a) any data at all was being copied between the computers on failure, and that b) such critical data was expected to be re-generated "on the fly". I would have expected that the interlocking data would be static and never modified outside of pre-planned maintenance, and that anything else would be classed as "state" information that isn't to be trusted on failure (which does seem to be the case, given that all routes dropped out and all trains stopped). It was pleasing to see that the investigation panel pointed out that it's very difficult for humans to intervene in such a failure, due to the rapid pace at with which events unfold. Last edited: 05/07/2019 at 14:33 by Danny252 Reason: None given Log in to reply |
| Hong Kong MTR trains collided when testing SelTrac CBTC 07/07/2019 at 03:54 #119295 | |
|
Hawk777
386 posts |
I had exactly the same question: why is the set of conflicting routes considered dynamic data rather than static data preloaded on all three computers? Do they expect to be modifying interlocking rules while the system is running? There must be a reason, because it seems like it would be not only safer but also easier to store it statically.
Log in to reply |
| Hong Kong MTR trains collided when testing SelTrac CBTC 07/07/2019 at 05:09 #119296 | |
|
GeoffM
6376 posts |
There is nothing inherently wrong with storing conflict data as dynamic data. Some systems are built as networks of nodes and edges where it is easy to find obvious conflicts. Geographic interlockings, albeit relay based, are effectively dynamic in finding most conflicts. Only a few unusual situations need additional wiring. SimSig Boss Last edited: 07/07/2019 at 05:10 by GeoffM Reason: None given Log in to reply |
| Hong Kong MTR trains collided when testing SelTrac CBTC 07/07/2019 at 08:40 #119297 | |
|
metcontrol
227 posts |
Lots of phrases and terminology similar to what I'm now familiar with on a daily basis now - but then it is mostly the same system. For "Conflict Zoe Data" we have "Conflict Avoidance Zone Data" maybe to differentiate between versions of system or because it makes for a nice acronym "CAZ." Our back-up systems were tested in a similar way, but as far as I'm aware this was done well before we tried to run trains, and a different (simulated) way was used to see if everything was working. Almost 4 months in and our system has been fairly reliable with only a few bits and pieces - mostly train-borne faults as a opposed to the system on the ground. It is due to expand during early August and take in a couple of our more complex areas/junctions. Log in to reply |
| Hong Kong MTR trains collided when testing SelTrac CBTC 09/07/2019 at 10:25 #119339 | |
|
Edmund1635
16 posts |
This model of SelTrac is specifically designed for Hong Kong, secondary backup is also a unique which is reportedly the first in the world with any train control system - basically how often would you want to backup your backup? To give you guys some perspective, there are many reasons behind this design: - The line where this incident happened (Tsuen Wan) is amongst the most heavily used line of the Hong Kong network. If you fancy knowing what the demand is line, this line is currently running a 2 minute headway (30tph) at peak hours with the existing SACEM system that is almost 20 years old. On a sister line (Island Line), peak hour headway is 112s (32tph). Even at this headway, you would typically need to wait for 4-5 departures to sardine onto a train. - Daily ridership on the Heavy Rail network (that's excluding the Light Rail part) is now in the range of 5.1 million trips: this is similar with LUL but with a network that's less than half the length and 1/3 of number of stations. - The MTR network is ATO operated from the beginning, so the infrastructure is not designed with manual operation in mind (trackside signals are only provided at headwalls and points, and by design they do not provide information about whether the track ahead is clear). As such, when the control system fails, safety rules mandates that manual operations is limited to a top speed of 22km/h. At the current ridership, running the lines at manual mode takes the same effect as closing the line altogether. - Hong Kong is very sensitive towards services disruptions when compared to London: a breakdown during rush is deemed a prime headline for newspapers and TV reports, when it happens, politicians would also become anything but helpful by competing to be first to point the finger at MTR without any understanding whatsoever. (For example, on one occasion where service was disrupted due to OHL damage, a legislator (who once chaired KCR before the merger) was calling for MTR to avoid such occurrence by switching to 3rd rail supply!) After all, Hong Kong is a small place full of people, and MTR is indeed a very critical part of the public transport system. It would make sense to do whatever possible to increase reliability. But whether they have gone too far with a secondary backup is something I would not comment myself. PS it is indeed shocking to me that such incident is down to such simple error. And if you think of the consequence of this (negative press, two trains out of service for some time, a number of signalling system test suspended throughout the network), it would make a very good lesson to anyone in how important it is to be meticulous with things. Last edited: 09/07/2019 at 10:26 by Edmund1635 Reason: None given Log in to reply |
| Hong Kong MTR trains collided when testing SelTrac CBTC 09/07/2019 at 12:55 #119342 | |
|
metcontrol
227 posts |
Edmund1635 in post 119339 said:Are you aware of a publication called The London Evening Standard! (other free papers are available but are generally just as critical in the absence of knowledge). I did check yesterday and yes we do have broadly speaking just the one back-up system. However, that aside, with such an "off the shelf" product there are plenty of similarities and plenty of areas where we have had to adjust our learning of the system to what is being supplied rather than something tailor-made for us to use. There are plenty of new phrases/acronyms etc. that we have had to learn in order to communicate with the supplier. For instance, we no longer have any track, it's all the "guideway." Last edited: 09/07/2019 at 12:55 by metcontrol Reason: None given Log in to reply |
| Hong Kong MTR trains collided when testing SelTrac CBTC 09/07/2019 at 13:41 #119343 | |
flymo
135 posts |
The main issue with the MTR I find (I use it every day) is that it HAS to work, there is no 'plan B' for when it all goes Pete Tong. Like last night in the evening peak when there were signalling issues for about 2 hours near Choi Hung, the Kwun Tong line was running at anything up to about 11 or 12 minute intervals (According to the MTR announcements) instead of the usual 2 or 3 minute interval. Where do the thousands of people go to when advised by MTR to 'consider using alternative transport'? Try to take one of the already overcrowded buses in gridlock? Impossible. Nope, the MTR has to work and when it doesn't it is a racing cert that because it basically strands thousands of people, that is why it gets the media attention it does. And don't get me started on Michael Tien...... Log in to reply |
| Hong Kong MTR trains collided when testing SelTrac CBTC 10/07/2019 at 03:27 #119353 | |
|
Jsun
212 posts |
So how do incidents like this square with the 99+% annual on time figures that MTR frequently touts, is that statistical trickery?
Log in to reply |
| Hong Kong MTR trains collided when testing SelTrac CBTC 10/07/2019 at 08:38 #119355 | |
|
Edmund1635
16 posts |
metcontrol in post 119342 said:@metcontrol oh Yes! Too familiar indeed! I lived in London for quite a bit back in the 2010s when failures was still an everyday sight and the Circle Line was so bad. But honestly Hong Kong does go viral with firing at MTR failures - if you think of London being used to weekend closures and replacement buses, doing the same in Hong Kong would anger just about every single person in the city. The real problem in my view is that Hong Konger simply doesn't know what it takes to maintain a railway network to such reliability MTR is maintaining. @JSun I think I can safely say there is no touts in the stats. The 99.9% figure you are referring to is the "Passenger Journey On Time" rate, which allows 5 minutes beyond the timetabled journey time, so if one ends up spending more than 5 minutes than timetabled time for a journey, that's a "Not On Time" journey. For this incident, it is possible that journey do not count towards "Not On Time", because there are two lines running the section affected (Admiralty - Central) in parallel, so it is very likely that journeys would not extend for more than 5 minutes. Of course, if a section is closed and there is no alternative railways route, it will very likely count towards "Not On Time". Having said that, MTR does get penalised for incidents like these - there is a incident penalty scheme which in essence says MTR needs to pay a sum of money for every incident that causes a delay of more than 30 minutes (which caps at HKD$25M or approx GBP2.5M). This money then gets returned to passengers every year by means of fare concession. Frankly though, it is fact that some comments consider MTR not measuring its service performance against "actual expectations". It is also true that MTR is seeing more incidents in recent years and there is a perceived dip in performance, but as someone who can compare between MTR and LUL first hand, I really can't put myself together to say MTR performance is poor. When people moan about 100ish incidents a year (cut off at 8 minutes) and 20 of them being signal related, they should be told that LUL suffers that number of incidents in a month (cut off at 15 minutes), not to mention how many time you can turn up at a station to find out a delay caused by signal failure. An above all - it is worth noting that this is the first incidence of two trains colliding on mainline in the line's 40 year history. Last edited: 10/07/2019 at 08:40 by Edmund1635 Reason: None given Log in to reply |
| Hong Kong MTR trains collided when testing SelTrac CBTC 10/07/2019 at 09:41 #119356 | |
|
vloris
39 posts
|
And remember: this was an incident with two test trains, not trains carrying passengers in normal operation!
Log in to reply |
| Hong Kong MTR trains collided when testing SelTrac CBTC 23/08/2019 at 09:30 #120179 | |
|
alvinhochun
249 posts |
Sorry for bumping this again. I just found out that the EMSD (Electrical and Mechanical Services Department) had also released its own independent investigation report. There isn't much new information other than a few pictures, but I think it's worth being linked here for the sake of completeness. https://www.emsd.gov.hk/filemanager/en/content_1377/TWL_New_Signalling_System_Testing_Incident_Report_(Eng).pdf Log in to reply The following users said thank you: VInce, postal |