UTC Edge effect

[wxtim]

Description

Symptoms: Workflows which remained up over the weekend the clocks changed in the UK displayed odd times for submit time such as:

This seems to be caused by the database containing the local time, but retaining the Z timezone for time_submit in the task_jobs table.

Reproducible Example

Not yet - I'm struggling to make this happen on purpose - Cylc 8 doesn't seem to respond to changes of TZ in the environment before using cylc command.

FWIW a similar effect can be produced with

[scheduler]
    allow implicit tasks = True

[scheduling]
    cycling mode = integer
    [[graph]]
        R1 = foo => bar

[runtime]
    [[bar]]
        init-script = """
            if [[ $CYLC_TASK_CYCLE_POINT == 2 ]]; then
                cylc stop --now --now ${CYLC_WORKFLOW_ID}
                sleep 5
                sed -i 's@XXX@XXX-1@' $HOME/repos/cylc-flow/cylc/flow/scripts/cylc.py
                cylc play ${CYLC_WORKFLOW_ID}
            fi
        """

And this diff

--- a/cylc/flow/scripts/cylc.py
+++ b/cylc/flow/scripts/cylc.py
@@ -680,6 +680,13 @@ def main():
             # this is an alias to a command
             if command in ALIASES:
                 command = ALIASES.get(command)
+            os.environ['TZ'] = 'XXX'
+            print(f'{os.environ["TZ"]}')

I think that this may be related to cylc.flow.wallclock._FLAGS['UTC mode'] retaining a default of false, despite other changes suggesting that it should be true in Cylc 8

Expected Behaviour

[hjoliver]

@dwsutherland does this have anything to do with the TZ issue you've been helping a NIWA user with?

[dwsutherland]

@dwsutherland does this have anything to do with the TZ issue you've been helping a NIWA user with?

I think it might be related, because if you don't specify the zone in the cycle point (i.e. Z on the end) then when doing datetime operations the datetime library will use local time..

For the problem I was working through with the NIWA user, he had a workflow running in UTC but with a cycle format of YYYYMMDDThh, and a local environment in New Zealand time (TZ=NZ)..
So workflow_state xtriggers that needed to adjust to the format of the target DB in a format YYYYMMDDThhmmZ wouldn't take the original workflow's TZ into account and just use the environment's, so the xtrigger was never satisfied.

[dwsutherland]

@dwsutherland does this have anything to do with the TZ issue you've been helping a NIWA user with?

Also, we probably need an issue up for this.. As the workflow's timezone should be passed to it's xtriggers, and maybe to the DB recording of times (if it isn't).. Instead of relying only on the format of the workflow's cycle point format. (to workflow_state and wall_clock at least, although I think the latter might be fine (?))

[dwsutherland]

But I guess what Tim is describing is the workflow recording event times in local time, but job submission is still UTC?

Or is it the adjustment of local time to UTC over local clock change? .. Is this even avoidable?

[wxtim]

But I guess what Tim is describing is the workflow recording event times in local time, but job submission is still UTC?

It's kind-of worse than that - the database is persistently recording the time_submit as the numbers which would be the local time, but the time zone section which is the time zone in which the workflow started, thus causing the time to be wrong.

I think it's reloading time_submit_exit, time_run and time_run_exit from the job.status file, and these seem to be the correct time, whatever time zone is recorded. This leads to Cylc review correctly identifying the submission time as having been an hour in the future. I.e. time_submit and time_submit_exit are 2025-04-07T14:59:53Z|2025-04-07T14:59:54+01:00, so the submit_time_exit (which is correct) is 13:59Z.

[wxtim]

I think that I now have a test for the undelying cause:

@pytest.mark.parametrize(
    'value,error',
    [
        (None, TypeError),
        (42, TypeError)
    ]
)
def test_get_unix_time_from_time_string_error(value, error):
    with pytest.raises(error):
        get_unix_time_from_time_string(value)


def test_get_current_time_string():
    """Now doesn't change timezones if system time is
    different to scheduler's timezone.

    https://github.com/cylc/cylc-flow/issues/6701
    """
    # Not likely to be local dev/user here:
    os.environ['TZ'] = 'Australia/Eucla'
    time.tzset()
    res = get_current_time_string()
    assert res[-6:] == '+08:45'

The problem seems to be that a good deal of timezone info is cached in wallclock.py and assumed to be correct for the lifetime of the Cylc script!