Hangfire Dashboard

SynchronizationData.RareBackgroundService

// Id: #872051
using RuketoAutomation.Service.Background;

var synchronizationData = Activate<SynchronizationData>();
await synchronizationData.RareBackgroundService();

Parameters

CurrentCulture 
"ru-RU"
CurrentUICulture 
"en-US"
RecurringJobId 
"rareJob"
RetryCount 
10
Time 
1728775869

State

13.10.2024 03:59:49 (+3.564s) Failed

An exception occurred during performance of the job.

System.InvalidOperationException (499c2216-4ac8-40d8-aeba-4410aa50cbdb)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1505(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+20ms Processing

Server:
499c2216-4ac8-40d8-aeba-4410aa50cbdb
Worker:
eea046bd

+1h 49m Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 10 of 10: Sequence contains more than one element

Enqueue at:
13.10.2024 03:59:34

+3.241s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (499c2216-4ac8-40d8-aeba-4410aa50cbdb)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1505(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+10ms Processing

Server:
499c2216-4ac8-40d8-aeba-4410aa50cbdb
Worker:
5d1a768b

+1h 9m Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 9 of 10: Sequence contains more than one element

Enqueue at:
13.10.2024 02:09:53

+6.909s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (499c2216-4ac8-40d8-aeba-4410aa50cbdb)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1505(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+22ms Processing

Server:
499c2216-4ac8-40d8-aeba-4410aa50cbdb
Worker:
eea046bd

+40m 40.082s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 8 of 10: Sequence contains more than one element

Enqueue at:
13.10.2024 01:00:12

+5.866s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (499c2216-4ac8-40d8-aeba-4410aa50cbdb)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1505(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+12ms Processing

Server:
499c2216-4ac8-40d8-aeba-4410aa50cbdb
Worker:
3adfc248

+24m 30.866s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 7 of 10: Sequence contains more than one element

Enqueue at:
13.10.2024 00:19:20

+5.798s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (499c2216-4ac8-40d8-aeba-4410aa50cbdb)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1505(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+13ms Processing

Server:
499c2216-4ac8-40d8-aeba-4410aa50cbdb
Worker:
328a560b

+11m 58.169s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 6 of 10: Sequence contains more than one element

Enqueue at:
12.10.2024 23:54:49

+3.089s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (499c2216-4ac8-40d8-aeba-4410aa50cbdb)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1505(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+8ms Processing

Server:
499c2216-4ac8-40d8-aeba-4410aa50cbdb
Worker:
51a2a787

+5m 33.129s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 5 of 10: Sequence contains more than one element

Enqueue at:
12.10.2024 23:42:41

+3.624s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (499c2216-4ac8-40d8-aeba-4410aa50cbdb)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1505(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+3ms Processing

Server:
499c2216-4ac8-40d8-aeba-4410aa50cbdb
Worker:
e0b7b5e2

+2m 26.362s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 4 of 10: Sequence contains more than one element

Enqueue at:
12.10.2024 23:37:09

+3.945s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (499c2216-4ac8-40d8-aeba-4410aa50cbdb)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1505(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+21ms Processing

Server:
499c2216-4ac8-40d8-aeba-4410aa50cbdb
Worker:
c371cbed

+1m 57.063s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 3 of 10: Sequence contains more than one element

Enqueue at:
12.10.2024 23:34:39

+3.155s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (499c2216-4ac8-40d8-aeba-4410aa50cbdb)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1505(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+10ms Processing

Server:
499c2216-4ac8-40d8-aeba-4410aa50cbdb
Worker:
5d1a768b

+42.100s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 2 of 10: Sequence contains more than one element

Enqueue at:
12.10.2024 23:32:41

+3.029s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (499c2216-4ac8-40d8-aeba-4410aa50cbdb)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1505(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+33ms Processing

Server:
499c2216-4ac8-40d8-aeba-4410aa50cbdb
Worker:
328a560b

+43.110s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 1 of 10: Sequence contains more than one element

Enqueue at:
12.10.2024 23:31:52

+3.190s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (499c2216-4ac8-40d8-aeba-4410aa50cbdb)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1505(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+8ms Processing

Server:
499c2216-4ac8-40d8-aeba-4410aa50cbdb
Worker:
e0b7b5e2

+2ms Enqueued

Triggered by recurring job scheduler

12.10.2024 23:31:09 Created