Hangfire Dashboard

SynchronizationData.RareBackgroundService

// Id: #909348
using RuketoAutomation.Service.Background;

var synchronizationData = Activate<SynchronizationData>();
await synchronizationData.RareBackgroundService();

Parameters

CurrentCulture 
"ru-RU"
CurrentUICulture 
"en-US"
RecurringJobId 
"rareJob"
RetryCount 
10
Time 
1731026231

State

08.11.2024 05:10:16 (+4.177s) Failed

An exception occurred during performance of the job.

System.InvalidOperationException (499c2216-4ac8-40d8-aeba-4410aa50cbdb)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1505(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+5ms Processing

Server:
499c2216-4ac8-40d8-aeba-4410aa50cbdb
Worker:
affc49dc

+1h 51m Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 10 of 10: Sequence contains more than one element

Enqueue at:
08.11.2024 05:09:58

+3.070s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (499c2216-4ac8-40d8-aeba-4410aa50cbdb)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1505(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+5ms Processing

Server:
499c2216-4ac8-40d8-aeba-4410aa50cbdb
Worker:
1db3f081

+1h 8m Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 9 of 10: Sequence contains more than one element

Enqueue at:
08.11.2024 03:18:26

+3.097s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (499c2216-4ac8-40d8-aeba-4410aa50cbdb)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1505(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
499c2216-4ac8-40d8-aeba-4410aa50cbdb
Worker:
93dcffa2

+43m 50.102s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 8 of 10: Sequence contains more than one element

Enqueue at:
08.11.2024 02:09:46

+2.981s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (499c2216-4ac8-40d8-aeba-4410aa50cbdb)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1505(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+5ms Processing

Server:
499c2216-4ac8-40d8-aeba-4410aa50cbdb
Worker:
eea046bd

+25m 13.137s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 7 of 10: Sequence contains more than one element

Enqueue at:
08.11.2024 01:25:46

+2.956s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (499c2216-4ac8-40d8-aeba-4410aa50cbdb)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1505(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+5ms Processing

Server:
499c2216-4ac8-40d8-aeba-4410aa50cbdb
Worker:
51a2a787

+11m 38.734s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 6 of 10: Sequence contains more than one element

Enqueue at:
08.11.2024 01:00:32

+7.093s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (499c2216-4ac8-40d8-aeba-4410aa50cbdb)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1505(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+6ms Processing

Server:
499c2216-4ac8-40d8-aeba-4410aa50cbdb
Worker:
affc49dc

+6m 12.716s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 5 of 10: Sequence contains more than one element

Enqueue at:
08.11.2024 00:48:55

+2.731s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (499c2216-4ac8-40d8-aeba-4410aa50cbdb)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1505(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+7ms Processing

Server:
499c2216-4ac8-40d8-aeba-4410aa50cbdb
Worker:
1db3f081

+2m 10.404s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 4 of 10: Sequence contains more than one element

Enqueue at:
08.11.2024 00:42:39

+4.767s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (499c2216-4ac8-40d8-aeba-4410aa50cbdb)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1505(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
499c2216-4ac8-40d8-aeba-4410aa50cbdb
Worker:
532dfde5

+1m 24.417s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 3 of 10: Sequence contains more than one element

Enqueue at:
08.11.2024 00:40:18

+5.790s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (499c2216-4ac8-40d8-aeba-4410aa50cbdb)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1505(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+9ms Processing

Server:
499c2216-4ac8-40d8-aeba-4410aa50cbdb
Worker:
3adfc248

+1m 9.583s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 2 of 10: Sequence contains more than one element

Enqueue at:
08.11.2024 00:38:53

+5.497s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (499c2216-4ac8-40d8-aeba-4410aa50cbdb)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1505(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+7ms Processing

Server:
499c2216-4ac8-40d8-aeba-4410aa50cbdb
Worker:
1db3f081

+27.423s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 1 of 10: Sequence contains more than one element

Enqueue at:
08.11.2024 00:37:29

+2.942s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (499c2216-4ac8-40d8-aeba-4410aa50cbdb)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1505(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+7ms Processing

Server:
499c2216-4ac8-40d8-aeba-4410aa50cbdb
Worker:
c371cbed

+3ms Enqueued

Triggered by recurring job scheduler

08.11.2024 00:37:11 Created