Hangfire Dashboard

SynchronizationData.RareBackgroundService

// Id: #922549
using RuketoAutomation.Service.Background;

var synchronizationData = Activate<SynchronizationData>();
await synchronizationData.RareBackgroundService();

Parameters

CurrentCulture 
"ru-RU"
CurrentUICulture 
"en-US"
RecurringJobId 
"rareJob"
RetryCount 
10
Time 
1731799573

State

17.11.2024 04:04:26 (+15.822s) Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+34ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
ae1918cf

+1h 53m Enqueued

Triggered by DelayedJobScheduler

+6ms Scheduled

Retry attempt 10 of 10: Sequence contains more than one element

Enqueue at:
17.11.2024 04:04:03

+17.046s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+29ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
7d6aaca2

+1h 10m Enqueued

Triggered by DelayedJobScheduler

+2ms Scheduled

Retry attempt 9 of 10: Sequence contains more than one element

Enqueue at:
17.11.2024 02:10:00

+14.499s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+18ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
2a0e207b

+42m 650ms Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 8 of 10: Sequence contains more than one element

Enqueue at:
17.11.2024 00:59:34

+15.666s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+57ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
0a419cde

+24m 24.255s Enqueued

Triggered by DelayedJobScheduler

+2ms Scheduled

Retry attempt 7 of 10: Sequence contains more than one element

Enqueue at:
17.11.2024 00:17:12

+14.457s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+16ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
ecdf88af

+13m 35.403s Enqueued

Triggered by DelayedJobScheduler

+2ms Scheduled

Retry attempt 6 of 10: Sequence contains more than one element

Enqueue at:
16.11.2024 23:52:27

+13.667s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+19ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
2a0e207b

+4m 44.395s Enqueued

Triggered by DelayedJobScheduler

+3ms Scheduled

Retry attempt 5 of 10: Sequence contains more than one element

Enqueue at:
16.11.2024 23:38:42

+17.101s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+14ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
0cfa5895

+3m 30.550s Enqueued

Triggered by DelayedJobScheduler

+3ms Scheduled

Retry attempt 4 of 10: Sequence contains more than one element

Enqueue at:
16.11.2024 23:33:47

+15.681s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+12ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
0cfa5895

+1m 59.290s Enqueued

Triggered by DelayedJobScheduler

+9ms Scheduled

Retry attempt 3 of 10: Sequence contains more than one element

Enqueue at:
16.11.2024 23:29:50

+16.132s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+6ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
7cba0ed9

+45.999s Enqueued

Triggered by DelayedJobScheduler

+4ms Scheduled

Retry attempt 2 of 10: Sequence contains more than one element

Enqueue at:
16.11.2024 23:27:46

+14.216s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+42ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
491797b7

+19.289s Enqueued

Triggered by DelayedJobScheduler

+3ms Scheduled

Retry attempt 1 of 10: Sequence contains more than one element

Enqueue at:
16.11.2024 23:26:44

+15.293s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (32b94c61-00d7-4f97-8a8b-e7ed2e90579d)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1952(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity)
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate)
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize()
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+30ms Processing

Server:
32b94c61-00d7-4f97-8a8b-e7ed2e90579d
Worker:
0a419cde

+4ms Enqueued

Triggered by recurring job scheduler

16.11.2024 23:26:13 Created