MODE_INFERENCE errors on unexpected OSM route tags (`xmas:train`)

[JGreenlee]

While resetting and re-running the pipeline for a staging user, MODE_INFERENCE failed due to:

2025-03-31T02:54:28.798-07:00 2025-03-31 09:54:28,798:DEBUG:139759875860288:unique_modes = {'TRAIN', 'XMAS:TRAIN'}
2025-03-31T02:54:28.799-07:00 2025-03-31 09:54:28,798:ERROR:139759875860288:unique_modes = {'TRAIN', 'XMAS:TRAIN'}, but we support only {'BUS', 'TRAIN', 'SUBWAY', 'TRAM', 'LIGHT_RAIL'}
2025-03-31T02:54:28.799-07:00 2025-03-31 09:54:28,799:ERROR:139759875860288:Error while inferring modes, timestamp is unchanged
2025-03-31T02:54:28.799-07:00 Traceback (most recent call last):
2025-03-31T02:54:28.799-07:00 File "/usr/src/app/emission/analysis/classification/inference/mode/rule_engine.py", line 24, in predict_mode
2025-03-31T02:54:28.799-07:00    mip.runPredictionPipeline(user_id, time_query)
2025-03-31T02:54:28.799-07:00 File "/usr/src/app/emission/analysis/classification/inference/mode/rule_engine.py", line 55, in runPredictionPipeline
2025-03-31T02:54:28.799-07:00.   self.predictedProb = self.predictModesStep()
2025-03-31T02:54:28.799-07:00 File "/usr/src/app/emission/analysis/classification/inference/mode/rule_engine.py", line 74, in predictModesStep
2025-03-31T02:54:28.799-07:00.   predictedProb.append(get_prediction(i, section_entry))
2025-03-31T02:54:28.799-07:00 File "/usr/src/app/emission/analysis/classification/inference/mode/rule_engine.py", line 113, in get_prediction
2025-03-31T02:54:28.799-07:00.   return get_motorized_prediction(i, section_entry)
2025-03-31T02:54:28.799-07:00 File "/usr/src/app/emission/analysis/classification/inference/mode/rule_engine.py", line 140, in get_motorized_prediction
2025-03-31T02:54:28.799-07:00.   predicted_transit_mode = _get_transit_prediction(i, section_entry)
2025-03-31T02:54:28.799-07:00 File "/usr/src/app/emission/analysis/classification/inference/mode/rule_engine.py", line 180, in _get_transit_prediction
2025-03-31T02:54:28.799-07:00.   collapsed_transit_mode = collapse_modes(section_entry, predicted_transit_modes)
2025-03-31T02:54:28.799-07:00.File "/usr/src/app/emission/analysis/classification/inference/mode/rule_engine.py", line 228, in collapse_modes
2025-03-31T02:54:28.799-07:00.   assert False
2025-03-31T02:54:28.799-07:00.AssertionError

Additional logs showed XMAS:TRAIN comes from a relation called "Holiday Train":

2025-03-31T02:54:28.765-07:00 2025-03-31 09:54:28,765:DEBUG:139759875860288:About to find matches in lists: [7287706, 9845465, 7287704, 7287705, 9833548, 9844350, 9844351, 9845476, 1056
2025-03-31T02:54:28.765-07:00 [2768041, 6088583, 9605484, 9605700, 12246240, 18087448, 18087450, 18393734, 18393735, 18393736, 18393737, 9604089, 9605482, 9605696, 9606121, 12246241, 18087449, 18087451, 18384359, 18393730, 18393731, 18393732, 18393733, 3441377, 8588026, 8589572, 9762426, 9762427, 9762428, 3441376, 8589500, 8589571, 9753196, 9760254, 9765612, 9765613, 9765614, 8589501, 8589502, 8589503, 8589504, 8589505, 9769085, 9769172, 9769173, 9769174, 9769175, 9769176, 13506047, 13506056, 13506047, 13506056]
2025-03-31T02:54:28.798-07:00 2025-03-31 09:54:28,798:DEBUG:139759875860288:matching routes = [(2768041, '6XX'), (6088583, '4XX'), (9605484, '1XX'), (9605700, '5XX'), (12246240, '6XX'), (18087448, '1XX'), (18087450, '1XX'), (18393734, 'Weekend Post-Game'), (18393735, 'Weekend Post-Game'), (18393736, 'Weekend Pre-Game'), (18393737, 'Weekend Pre-Game'), (9604089, '6XX'), (9605482, '1XX'), (9605696, '5XX'), (9606121, '4XX'), (12246241, '6XX'), (18087449, '1XX'), (18087451, '1XX'), (18384359, 'Holiday Train'), (18393730, 'Weekend Post-Game'), (18393731, 'Weekend Pre-Game'), (18393732, 'Weekend Post-Game'), (18393733, 'Weekend Pre-Game')]
2025-03-31T02:54:28.798-07:00 2025-03-31 09:54:28,798:DEBUG:139759875860288:len(start_routes) = 44, len(end_routes) = 52, len(rel_id_matches) = 23
2025-03-31T02:54:28.798-07:00 2025-03-31 09:54:28,798:DEBUG:139759875860288:Got predicted transit mode ['train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'xmas:train', 'train', 'train', 'train', 'train']
2025-03-31T02:54:28.798-07:00 2025-03-31 09:54:28,798:DEBUG:139759875860288:train_mapped_modes = ['TRAIN', 'TRAIN', 'TRAIN', 'TRAIN', 'TRAIN', 'TRAIN', 'TRAIN', 'TRAIN', 'TRAIN', 'TRAIN', 'TRAIN', 'TRAIN', 'TRAIN', 'TRAIN', 'TRAIN', 'TRAIN', 'TRAIN', 'TRAIN', 'XMAS:TRAIN', 'TRAIN', 'TRAIN', 'TRAIN', 'TRAIN']
2025-03-31T02:54:28.798-07:00 2025-03-31 09:54:28,798:DEBUG:139759875860288:unique_modes = {'TRAIN', 'XMAS:TRAIN'}

And is ultimately due to the presence of a "route" tag of xmas:train on this relation: https://www.openstreetmap.org/relation/18384359

so basically, I took Caltrain not during the holiday season

So we looked up all routes that run between that start and stop. And one of those was the holiday train so we were able to collapse all the other types of trains (express, local, weekday, weekend) into one, but since this was flagged as xmas:train, it wasn't collapsed and broke our inference

[JGreenlee]

We should:

1. Add this case to the rule_engine tests in Add Tests for Transit Matching Logic and Rule Engine e-mission-server#1039 @TeachMeTW
2. Decide how cases like this should be handled and make the appropriate changes to rule_engine.py and/or match_stops.py

   I am assuming that this "Holiday Train" would be best mapped to just TRAIN for our purposes
   Based on the tags that are available on the relation, is there a way we could do that?

   is "prefix:train" an established pattern for OSM tags such that we could just chop off the prefix?

   Or is it a nonstandard or incorrect label and we just have to skip relations like this?

[shankari]

is "prefix:train" an established pattern for OSM tags such that we could just chop off the prefix?

Does not appear to be:
https://wiki.openstreetmap.org/wiki/Key:route?uselang=en

Although we can ask questions in the OSM chat if needed.

For this specific case, I think that the more useful tags would be opening_hours and seasonal Looking at the opening_hours in particular, the evaluation tool https://openingh.openstreetmap.de/evaluation_tool/ shows that parsing the opening hours will lead to showing that the "facility" is now closed.

The seasonal tag might also be helpful and we can ask a question on how to handle that properly, similar to
https://help.openstreetmap.org/tags/seasonal/

This actually ties in with an improvement that I have wanted to do for a while; we should pass in the date of the trip while querying for the relations, since bus routes change all the time. The current approach (query for the most current information) works as long as we infer the mode fairly soon after the trip was taken, but does not work for reset/re-run many months in the past.
https://wiki.openstreetmap.org/wiki/Overpass_API/Overpass_QL#Date

[shankari]

However, even if we handle the seasonal and opening_hours tags correctly, we will still have to come up with a principled, generic solution for unknown route tags. It is pretty clear that there is no canconical list of route tags, just "common" ones.

And this is the general philosophy of OSM
https://help.openstreetmap.org/answer_link/34593/

With regard to the first part I wouldn't worry too much about translating your knowledge about when the fee is collected into OSM tags. I'd just make sure that what you know is tagged in English (perhaps as "fee=bathing_weather", perhaps as a note, perhaps as something else), and let people who worry about consuming the data decide what to do when they encounter it.
Essentially, mappers are OSM's scarcest resource, and anything that frees up their time to do more mapping is good, even if it requires data consumers to handle exceptions. It may also, as has happened with "opening hours", cause the people who care about ensuring that the data in that tag is machine readable to better define exceptions for English statements just like this one.

And if I had taken that trip in December, even if we handled the other tags properly, this would have still been a valid route, so we would still have failed. So we do need some mechanism (similar to UNKNOWN) to handle non-standard mode types.

[TeachMeTW]

During one of my earlier commits, I had a similar issue so I created Mode Type Equivalences which is just a mapping of inferred types in emission vs osm (now removed); do you suggest preprocess mode tags to handle nonstandard values like "XMAS:TRAIN"?

[shankari]

In parallel with fixing this, I think we should also make some changes to recover more robustly from such errors.

1. We should consider skipping only that section if there is an error, instead of failing the entire stage. With the error here

2025-03-31T02:31:14.590-07:00 2025-03-31 09:31:14,590:INFO:139759875860288:For stage PipelineStages.MODE_INFERENCE, start_ts is None

2025-03-31T02:54:28.863-07:00 2025-03-31 09:54:28,863:INFO:139759875860288:For stage PipelineStages.LABEL_INFERENCE, start_ts = 2024-07-31T00:03:27.995422
2025-03-31T02:56:56.566-07:00 2025-03-31 09:56:56,566:INFO:139759875860288:For stage PipelineStages.LABEL_INFERENCE, last_ts_processed = 2024-11-09T01:01:48.999735

The user will have no inferred values; it seems like it would be better to fill all the inferred sections except the one that generated the error.

2. We should write a script that can fill in missing entries after fixes. The script should:

   • only run the mode_inference step. The mode inference step is fairly self-contained, and does not generally affect the downstream objects, we don't really need to reset the whole pipeline and recompute everything.
   • however, we do have a few forward links, aka in the inferred_section_summary fields of all associated confirmed and composite trips.

3. If error -> failure of the entire step, then we can recompute the summaries for all confirmed and composite trips after the start_ts of this particular pipeline stage.

4. If error -> failure of that section as suggested above, then to be efficient and avoid wasted work we need only need to recompute the matching confirmed and composite trips. So we can maintain a mapping based on the start and end timestamp and, after the inferred sections are created, update the matching trips.

I have a preference for (2) which I think is more principled, but I am willing to take (1) if it is significantly easier to implement.

[shankari]

Patch for "skipping only the section" might be something like

diff --git a/emission/analysis/classification/inference/mode/rule_engine.py b/emission/analysis/classification/inference/mode/rule_engine.py
index 837364a1..1bc70870 100644
--- a/emission/analysis/classification/inference/mode/rule_engine.py
+++ b/emission/analysis/classification/inference/mode/rule_engine.py
@@ -68,10 +68,14 @@ class RuleEngineModeInferencePipeline:
                 (section_entry.get_id(),
                  section_entry.data.start_fmt_time, section_entry.data.end_fmt_time) +
                 '~' * 10)
-            if section_entry.data.sensed_mode == ecwma.MotionTypes.AIR_OR_HSR:
-                predictedProb.append({'AIR_OR_HSR': 1})
-            else:
-                predictedProb.append(get_prediction(i, section_entry))
+            try:
+                if section_entry.data.sensed_mode == ecwma.MotionTypes.AIR_OR_HSR:
+                    predictedProb.append({'AIR_OR_HSR': 1})
+                else:
+                    predictedProb.append(get_prediction(i, section_entry))
+            catch Exception as e:
+                logging.error(f"Found {e} while inferring sensed modes, skipping {section_entry.get_id()} and {section_entry.user_id}, starting at {section_entry.data.start_fmt_time}")
+                logging.exception(e)
 
         return predictedProb
 

I haven't had a chance to test it out yet though