History graphs #4

New Issue

Nic Elliott · 2024-06-06T11:14:08Z

Nic Elliott commented

2024-06-06 11:14:08 +00:00

History pages not showing accurate count of downloaded and uploaded data for each connection.

Data unreliable after 1 May 2024.

History pages not showing accurate count of downloaded and uploaded data for each connection. Data unreliable after 1 May 2024.

Nic Elliott added the

Monitoring

label 2024-06-06 11:14:08 +00:00

Nic Elliott commented

2024-07-08 14:46:03 +00:00

Update

This issue has been investigated and subsequently resolved.

The issue was relating to a combination of timezones and application migration.

Background and investigation

On 30th April, a legacy server that was hosting parts of our monitoring platform started to fail. On 1st May, eView Live and some of the data-collection services that eView depends on were migrated on to our new application infrastructure which had been built ready for this migration.

In the following days, we found multiple issues cropping up relating to timezones. The legacy server and new application infrastructure were on either UTC and BST. The 1 hour difference between these caused data to be stored with the wrong timestamp, resulting in data not being displayed, or being overwritten.

The history graphs on eView Live were losing ~80% of their stored data due to this overwrite issue. Since the history data was not completely missing, and the graphs were still showing some datapoints, this issue was not immediately flagged.

Resolution

As part of a general push on timezone related issues, all servers, services, and software are being set to work in UTC, so that individual clients can set their timezone and adjust values accordingly, while working with a single unified dataset with a global reference point.

On 4th July, a change was deployed to the data service that handles the history graphs, setting it to use UTC as standard instead of UK/London. This was monitored over the weekend, and confirmed to be resolved on 8th July.

Important note: The history data between 1st May and 4th July is irretrievable due to the nature of the data collection and processing.

### Update This issue has been investigated and subsequently resolved. The issue was relating to a combination of timezones and application migration. ### Background and investigation On 30th April, a legacy server that was hosting parts of our monitoring platform started to fail. On 1st May, eView Live and some of the data-collection services that eView depends on were migrated on to our new application infrastructure which had been built ready for this migration. In the following days, we found multiple issues cropping up relating to timezones. The legacy server and new application infrastructure were on either UTC and BST. The 1 hour difference between these caused data to be stored with the wrong timestamp, resulting in data not being displayed, or being overwritten. The history graphs on eView Live were losing \~80% of their stored data due to this overwrite issue. Since the history data was not completely missing, and the graphs were still showing some datapoints, this issue was not immediately flagged. ### Resolution As part of a general push on timezone related issues, all servers, services, and software are being set to work in UTC, so that individual clients can set their timezone and adjust values accordingly, while working with a single unified dataset with a global reference point. On 4th July, a change was deployed to the data service that handles the history graphs, setting it to use UTC as standard instead of UK/London. This was monitored over the weekend, and confirmed to be resolved on 8th July. > Important note: The history data between 1st May and 4th July is irretrievable due to the nature of the data collection and processing.

Nic Elliott closed this issue

2024-07-08 14:46:03 +00:00

Nic Elliott pinned this 2024-07-08 14:46:50 +00:00

Brandon Currell added the

Resolved

label 2024-07-10 14:12:02 +00:00

Brandon Currell unpinned this 2024-07-15 10:54:21 +00:00

Nic Elliott reopened this issue

2024-07-17 12:35:35 +00:00

Nic Elliott commented

2024-07-17 12:35:59 +00:00

Re-investigating this issue.