Hangfire Dashboard

SynchronizationData.RareBackgroundService

// Id: #1062653
using RuketoAutomation.Service.Background;

var synchronizationData = Activate<SynchronizationData>();
await synchronizationData.RareBackgroundService();

Parameters

CurrentCulture 
"ru-RU"
CurrentUICulture 
"en-US"
RecurringJobId 
"rareJob"
RetryCount 
10
Time 
1739489708

State

14.02.2025 04:11:52 (+2.773s) Failed

An exception occurred during performance of the job.

System.InvalidOperationException (759e5c7b-e4e8-4ea3-9bca-a4b786b21a49)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1915(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+3ms Processing

Server:
759e5c7b-e4e8-4ea3-9bca-a4b786b21a49
Worker:
d2729edf

+1h 50m Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 10 of 10: Sequence contains more than one element

Enqueue at:
14.02.2025 04:11:45

+2.706s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (759e5c7b-e4e8-4ea3-9bca-a4b786b21a49)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1915(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
759e5c7b-e4e8-4ea3-9bca-a4b786b21a49
Worker:
e30c499c

+1h 11m Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 9 of 10: Sequence contains more than one element

Enqueue at:
14.02.2025 02:20:37

+3.116s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (759e5c7b-e4e8-4ea3-9bca-a4b786b21a49)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1915(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
759e5c7b-e4e8-4ea3-9bca-a4b786b21a49
Worker:
cd1664e4

+43m 58.119s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 8 of 10: Sequence contains more than one element

Enqueue at:
14.02.2025 01:09:16

+4.157s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (759e5c7b-e4e8-4ea3-9bca-a4b786b21a49)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1915(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+5ms Processing

Server:
759e5c7b-e4e8-4ea3-9bca-a4b786b21a49
Worker:
a6d4d495

+23m 28.171s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 7 of 10: Sequence contains more than one element

Enqueue at:
14.02.2025 00:25:21

+2.985s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (759e5c7b-e4e8-4ea3-9bca-a4b786b21a49)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1915(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
759e5c7b-e4e8-4ea3-9bca-a4b786b21a49
Worker:
49131932

+13m 8.294s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 6 of 10: Sequence contains more than one element

Enqueue at:
14.02.2025 00:01:46

+7.313s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (759e5c7b-e4e8-4ea3-9bca-a4b786b21a49)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1915(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+3ms Processing

Server:
759e5c7b-e4e8-4ea3-9bca-a4b786b21a49
Worker:
bbeb96c1

+6m 27.448s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 5 of 10: Sequence contains more than one element

Enqueue at:
13.02.2025 23:48:29

+2.833s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (759e5c7b-e4e8-4ea3-9bca-a4b786b21a49)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1915(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+5ms Processing

Server:
759e5c7b-e4e8-4ea3-9bca-a4b786b21a49
Worker:
8c872ba0

+3m 8.410s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 4 of 10: Sequence contains more than one element

Enqueue at:
13.02.2025 23:42:06

+6.754s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (759e5c7b-e4e8-4ea3-9bca-a4b786b21a49)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1915(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
759e5c7b-e4e8-4ea3-9bca-a4b786b21a49
Worker:
8229a65b

+1m 56.215s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 3 of 10: Sequence contains more than one element

Enqueue at:
13.02.2025 23:38:45

+3.850s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (759e5c7b-e4e8-4ea3-9bca-a4b786b21a49)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1915(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+3ms Processing

Server:
759e5c7b-e4e8-4ea3-9bca-a4b786b21a49
Worker:
eb37b2d0

+1m 11.261s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 2 of 10: Sequence contains more than one element

Enqueue at:
13.02.2025 23:36:46

+3.779s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (759e5c7b-e4e8-4ea3-9bca-a4b786b21a49)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1915(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
759e5c7b-e4e8-4ea3-9bca-a4b786b21a49
Worker:
9743e399

+28.269s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 1 of 10: Sequence contains more than one element

Enqueue at:
13.02.2025 23:35:37

+3.576s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (759e5c7b-e4e8-4ea3-9bca-a4b786b21a49)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1915(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+6ms Processing

Server:
759e5c7b-e4e8-4ea3-9bca-a4b786b21a49
Worker:
1252f650

+1ms Enqueued

Triggered by recurring job scheduler

13.02.2025 23:35:08 Created