Hangfire Dashboard

SynchronizationData.RareBackgroundService

// Id: #923731
using RuketoAutomation.Service.Background;

var synchronizationData = Activate<SynchronizationData>();
await synchronizationData.RareBackgroundService();

Parameters

CurrentCulture 
"ru-RU"
CurrentUICulture 
"en-US"
RecurringJobId 
"rareJob"
RetryCount 
10
Time 
1731889565

State

18.11.2024 05:01:32 (+14.325s) Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+33ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
f3de2030

+1h 54m Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 10 of 10: Sequence contains more than one element

Enqueue at:
18.11.2024 05:01:18

+16.004s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+8ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
067c80aa

+1h 11m Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 9 of 10: Sequence contains more than one element

Enqueue at:
18.11.2024 03:06:36

+15.045s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+33ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
ecdf88af

+40m 44.652s Enqueued

Triggered by DelayedJobScheduler

+4ms Scheduled

Retry attempt 8 of 10: Sequence contains more than one element

Enqueue at:
18.11.2024 01:54:54

+20.874s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+28ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
f3de2030

+23m 52.076s Enqueued

Triggered by DelayedJobScheduler

+2ms Scheduled

Retry attempt 7 of 10: Sequence contains more than one element

Enqueue at:
18.11.2024 01:13:51

+14.762s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+29ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
3b99b498

+12m 9ms Enqueued

Triggered by DelayedJobScheduler

+2ms Scheduled

Retry attempt 6 of 10: Sequence contains more than one element

Enqueue at:
18.11.2024 00:49:38

+19.755s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+27ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
3b99b498

+4m 43.702s Enqueued

Triggered by DelayedJobScheduler

+2ms Scheduled

Retry attempt 5 of 10: Sequence contains more than one element

Enqueue at:
18.11.2024 00:37:24

+18.049s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+88ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
491797b7

+3m 201ms Enqueued

Triggered by DelayedJobScheduler

+4ms Scheduled

Retry attempt 4 of 10: Sequence contains more than one element

Enqueue at:
18.11.2024 00:32:21

+15.894s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+22ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
3b99b498

+58.674s Enqueued

Triggered by DelayedJobScheduler

+9ms Scheduled

Retry attempt 3 of 10: Sequence contains more than one element

Enqueue at:
18.11.2024 00:29:08

+16.778s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+45ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
3b99b498

+42.548s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 2 of 10: Sequence contains more than one element

Enqueue at:
18.11.2024 00:27:45

+17.774s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+43ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
ae1918cf

+31.184s Enqueued

Triggered by DelayedJobScheduler

+5ms Scheduled

Retry attempt 1 of 10: Sequence contains more than one element

Enqueue at:
18.11.2024 00:26:50

+16.726s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+42ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
0cfa5895

+6ms Enqueued

Triggered by recurring job scheduler

18.11.2024 00:26:05 Created