Service Provider Incident

As we continue to monitor the queue issue, we are moving our status back to Identified. All data continues to be captured and we will provide updates as the queues begin to clear.

We have received further communication from Amazon that the specific service that we use to manage flow message queues continues to experience delays. These delays are impacting flow processing.

All data received since our HTTP & eAdaptor services recovered is queued and properly saved. When the service performance returns to normal, your data will be processed.

At this time, any service that has receieved a successful response from a Chain.io endpoint does not need to resend data.

For Cargowise users, we are successfully collecting and queuing data for eAdaptor messages to Chain.io, however there is a processing delay. You many not see results while the queue processes, but as of 9:21 UTC we are receiving the data. Between approximately 7:05 UTC and 9:21 UTC eAdaptor would have received error messages and would have held for reprocessing.

We will continue to post updates here as the queue size resolves.

We are seeing data processing and are working through a backlog of previous submissions. When the backlog clears we will notify any impacted customers of any irregular processes that may need to be addressed manually.

IMPORTANT NOTE: During the outage, systems sending to Chain.io likely received SFTP or HTTP connection failures. In those cases, the impacted systems should use their normal retry or requeue mechanisms.

Access to portal.chain.io has been restored. We are researching the impact on other services and will post an update closing the incident as soon as we are fully confident that all services are restored.

AWS is begining to see recovery and we are also seeing some services become available again. We are carefully monitoring to ensure a clean recovery and will provide further updates here once service is fully restored.

AWS has identified a potential root cause related to domain name resolution and are working to implement a fix. We will continue to provide updates here as they become available.

AWS Engineers have informed us that they are engaged and are working to identfiy the root cause. There will be another update at the top of the hour.

AWS has confirmed the incident and that they are working on a resolution. At this time, Chain.io will continue to wait for AWS resolution. We will also continue to post updates here as we have them.

If you would like to follow the AWS status page directly it is available here: https://health.aws.amazon.com/health/status

We are currently investigating and underlying outage with AWS which is impacting the ability to log into portal.chain.io and may also be impacting data processing.

Per Chain.io BC/DR processes, we will monitor and advise during this outage. If the incident extends several more hours we may activate a business continuity response.