Hangfire Dashboard

SynchronizationData.MainBackgroundService

The job is finished. It will be removed automatically 23.11.2024 23:04:07. 
// Id: #931709
using RuketoAutomation.Service.Background;

var synchronizationData = Activate<SynchronizationData>();
await synchronizationData.MainBackgroundService();

Parameters

CurrentCulture 
"ru-RU"
CurrentUICulture 
"en-US"
RecurringJobId 
"mainJob"
RetryCount 
4
Time 
1732305008

State

22.11.2024 20:04:07 (+1m 51.313s) Succeeded

Latency:
12m 7.906s
Duration:
1m 51.304s

+8ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
ecdf88af

+2m 35.072s Enqueued

Triggered by DelayedJobScheduler

+4ms Scheduled

Retry attempt 4 of 10: The request was canceled due to the configured Ht…

Enqueue at:
22.11.2024 20:02:09

+1m 40.058s Failed

An exception occurred during performance of the job.

System.Threading.Tasks.TaskCanceledException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.

System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
 ---> System.TimeoutException: A task was canceled.
 ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellation(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageHandlerStage.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageHandlerStage.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.SocketsHttpHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClientHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageInvoker.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
   at RuketoAutomation.Service.Crm.CrmApiWorker.GetData(String address, String parameters)
   at RuketoAutomation.Service.Crm.CrmApiWorker.GetOrdersDataByIdWithStatusCancel(IEnumerable`1 ordersIds, Int32 page)
   at RuketoAutomation.Service.Background.SynchronizationData.UpdateCanceledOrders()
   at RuketoAutomation.Service.Background.SynchronizationData.CrmMainSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.MainBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+32ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
3b99b498

+1m 50.001s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 3 of 10: The request was canceled due to the configured Ht…

Enqueue at:
22.11.2024 19:57:48

+1m 40.061s Failed

An exception occurred during performance of the job.

System.Threading.Tasks.TaskCanceledException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.

System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
 ---> System.TimeoutException: A task was canceled.
 ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellation(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageHandlerStage.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageHandlerStage.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.SocketsHttpHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClientHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageInvoker.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
   at RuketoAutomation.Service.Crm.CrmApiWorker.GetData(String address, String parameters)
   at RuketoAutomation.Service.Crm.CrmApiWorker.GetOrdersDataByIdWithStatusCancel(IEnumerable`1 ordersIds, Int32 page)
   at RuketoAutomation.Service.Background.SynchronizationData.UpdateCanceledOrders()
   at RuketoAutomation.Service.Background.SynchronizationData.CrmMainSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.MainBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
3d72efc3

+34.986s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 2 of 10: The request was canceled due to the configured Ht…

Enqueue at:
22.11.2024 19:54:30

+1m 40.057s Failed

An exception occurred during performance of the job.

System.Threading.Tasks.TaskCanceledException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.

System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
 ---> System.TimeoutException: A task was canceled.
 ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellation(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageHandlerStage.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageHandlerStage.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.SocketsHttpHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClientHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageInvoker.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
   at RuketoAutomation.Service.Crm.CrmApiWorker.GetData(String address, String parameters)
   at RuketoAutomation.Service.Crm.CrmApiWorker.GetOrdersDataByIdWithStatusCancel(IEnumerable`1 ordersIds, Int32 page)
   at RuketoAutomation.Service.Background.SynchronizationData.UpdateCanceledOrders()
   at RuketoAutomation.Service.Background.SynchronizationData.CrmMainSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.MainBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+16ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
9d56054d

+27.484s Enqueued

Triggered by DelayedJobScheduler

+2ms Scheduled

Retry attempt 1 of 10: The request was canceled due to the configured Ht…

Enqueue at:
22.11.2024 19:52:05

+1m 40.083s Failed

An exception occurred during performance of the job.

System.Threading.Tasks.TaskCanceledException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.

System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
 ---> System.TimeoutException: A task was canceled.
 ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellation(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageHandlerStage.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageHandlerStage.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.SocketsHttpHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClientHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageInvoker.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
   at RuketoAutomation.Service.Crm.CrmApiWorker.GetData(String address, String parameters)
   at RuketoAutomation.Service.Crm.CrmApiWorker.GetOrdersDataByIdWithStatusCancel(IEnumerable`1 ordersIds, Int32 page)
   at RuketoAutomation.Service.Background.SynchronizationData.UpdateCanceledOrders()
   at RuketoAutomation.Service.Background.SynchronizationData.CrmMainSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.MainBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+21ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
ae1918cf

+4ms Enqueued

Triggered by recurring job scheduler

22.11.2024 19:50:08 Created