Incident History

History

February 20
Data Ingestion Not Processing
Resolved
February 20 2:11 PM UTC
The issue affecting data ingestion, which began at 08:20 CET on 20th February 2025, has been successfully resolved. The system has returned to normal operation, and all previously pending actions are now being processed without further delays. We appreciate your patience and understanding throughout this process.
Identified
February 20 11:50 AM UTC
The issue has been identified and mitigated. Since 11:40 CET, flows have been running as expected. We are currently monitoring the completion of the manual restart process to ensure all previously unexecuted actions are processed
Investigating
February 20 10:45 AM UTC
Since 08:20 today, the writer has not been working. While extractions are running successfully and all sources are extracting data, ingestion into the flow is not happening, preventing data from being processed correctly. Our team is actively working on resolving the issue, and we will provide updates as soon as possible
February 19 12:40 PM UTC until February 26 12:40 PM UTC
Mongo Database upgrade & rightsizing
Completed
February 20 8:47 AM UTC
We updated our mongoDB cluster instances to the latest available version and rightsized the instance system requirements. All delayed actions were processed at 8:50AM CET. Since this moment, all actions are executed according to their configured schedule in standard way.
Scheduled
February 19 12:49 PM UTC
We plan to upgrade MongoDB cluster to newer version and rightsize the cluster instances.During the upgrade there can be the service degradation for couple of minutes while using our headless API. The scheduled actions will be gradually executed, but may be postponed in time by couple of minutes.
February 19
Disrupted data synchronisation for Google related sources
Resolved
February 19 7:01 PM UTC
All reauthorised sources are working as expected.
Monitoring
February 19 3:50 PM UTC
Reauthorised sources are working as expected.
Identified
February 19 11:28 AM UTC
Due to this incident, data synchronisation has been interrupted for some clients using Google-related services whose tokens were revoked. In case you spot this issue in your account, please log in to your Dataddo account and reauthorise your Google-related sources as soon as possible. This step is critical to resume normal operation.
Investigating
February 19 11:09 AM UTC
Data synchronisation has been interrupted for clients using Google related services whose tokens were revoked.
February 13 11:00 PM UTC until February 14 3:00 PM UTC
Planned database migration
Completed
February 19 12:33 PM UTC
All successfully migrated.
Scheduled
February 12 2:19 PM UTC
We will migrate database records. During the maintenance period, several minutes in their schedules may postpone some customer actions.
January 29
Restarting stuck actions
Resolved
January 29 9:24 AM UTC
All affected actions (extractions and following flows) were restarted. In each affected extraction log, you find the execution with "failed to set timeout: deadline can't be prolonged" error.The extractions/flows behavior is now as expected.
Identified
January 29 7:48 AM UTC
We ran command/script to restart all stuck extractions. Such action's last execution log result to be "broken" and the new execution would start immediately. If the extraction has following flow attached this will be triggered automatically once the extractions finishes with no error.
Identified
January 29 7:11 AM UTC
We have identified that extractions were executed according to their scheduled configurations. However, the extractor component failed to send results to the result queue.This issue affected all extractions processed in the AWS environment, while extractions in the GCP environment remained unaffected.We are now updating all affected actions to a “broken” status and will restart them to ensure the triggering of subsequent flows.
Investigating
January 29 6:40 AM UTC
Jobs are processed longer than usual
January 9
Some Instagram metrics were deprecated
Resolved
January 9 3:44 PM UTC
All sources should be working as expected.
Monitoring
January 9 2:39 PM UTC
The issue is fixed. The deprecated metrics should return 0 values from today.
Identified
January 9 12:05 PM UTC
Issue is identified and we are working on the fix.
Investigating
January 9 11:58 AM UTC
Some Instagram metrics were deprecated resulting sources failing.
November 21, 2024 9:27 AM UTC until November 28, 2024 9:27 AM UTC
Production DB instances upgrade
Completed
November 25, 2024 10:10 PM UTC
Upgrade successful. Some action might be postponed in processing by couple of minutes.
Scheduled
November 21, 2024 9:29 AM UTC
Upgrading to the newer, more powerful instance types empowering our Postgres databases. During the upgrade, processing of scheduled action at this time may be postponed by a few minutes.
November 18, 2024
Delayed actions processing - problem solved
Resolved
November 14, 2024 3:00 PM UTC
All enqueued actions were processed. The new actions are not blocked anymore.
Identified
November 14, 2024 1:48 PM UTC
We identified the concrete actions which are causing the trouble and rescheduled them in order to let other actions to be processed.
Identified
November 14, 2024 1:46 PM UTC
We identified the whole problem is caused by the significant number of jobs which take 30s (timeout) to be processed. Even we have auto-scaling enabled we used all dedicated auto-scaled workers, but the effect was low.
Investigating
November 14, 2024 1:40 PM UTC
Some dataddo users may experience the delayed processing of their actions.
October 23, 2024
Dataddo API returning 500 to every request
Resolved
October 23, 2024 9:00 AM UTC
Invalid commit rolled back after we identified the bug. The commit with proper fix already deployed to production too.
Identified
October 23, 2024 8:57 AM UTC
Dataddo APIs (both headless and public-api) returning 500 to every request, because of old service configuration.
October 7, 2024 9:00 AM UTC until October 7, 2024 9:30 AM UTC
Data Processing task DB maintenance
Completed
October 7, 2024 11:11 AM UTC
Done
Scheduled
October 7, 2024 7:08 AM UTC
On Monday, 7th October 2024, we will be rebooting one of our AWS RDS database clusters. This reboot is essential to apply pending updates, adjust settings, and install critical security patches, ensuring the continued performance and security of our systems.To minimise the risk of unexpected action processing failures during the database upgrade, we have decided to temporarily pause all action processing activities. Once the upgrade is complete, all previously paused actions will be automatically resumed and processed. We appreciate your understanding and cooperation as we undertake these necessary maintenance tasks.
← Back to current status