Hangfire Dashboard

SynchronizationData.RareBackgroundService

// Id: #919666
using RuketoAutomation.Service.Background;

var synchronizationData = Activate<SynchronizationData>();
await synchronizationData.RareBackgroundService();

Parameters

CurrentCulture 
"ru-RU"
CurrentUICulture 
"en-US"
RecurringJobId 
"rareJob"
RetryCount 
10
Time 
1731628564

State

15.11.2024 04:32:44 (+12.868s) Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+5ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
6a9f654a

+1h 53m Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 10 of 10: Sequence contains more than one element

Enqueue at:
15.11.2024 04:32:28

+13.540s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+6ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
3d72efc3

+1h 12m Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 9 of 10: Sequence contains more than one element

Enqueue at:
15.11.2024 02:38:44

+13.356s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
f3de2030

+41m 16.344s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 8 of 10: Sequence contains more than one element

Enqueue at:
15.11.2024 01:26:37

+15.745s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+5ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
6a9f654a

+23m 1.744s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 7 of 10: Sequence contains more than one element

Enqueue at:
15.11.2024 00:45:08

+14.684s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
0a419cde

+12m 2.619s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 6 of 10: Sequence contains more than one element

Enqueue at:
15.11.2024 00:21:42

+13.116s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+5ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
3d72efc3

+6m 44.812s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 5 of 10: Sequence contains more than one element

Enqueue at:
15.11.2024 00:09:33

+15.581s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+5ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
3b99b498

+2m 47.365s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 4 of 10: Sequence contains more than one element

Enqueue at:
15.11.2024 00:02:37

+12.835s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
067c80aa

+1m 18.675s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 3 of 10: Sequence contains more than one element

Enqueue at:
14.11.2024 23:59:25

+11.401s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+5ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
7cba0ed9

+47.857s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 2 of 10: Sequence contains more than one element

Enqueue at:
14.11.2024 23:58:06

+12.248s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+6ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
3b99b498

+48.304s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 1 of 10: Sequence contains more than one element

Enqueue at:
14.11.2024 23:56:57

+14.179s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+7ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
ae1918cf

+2ms Enqueued

Triggered by recurring job scheduler

14.11.2024 23:56:04 Created