Hangfire Dashboard

SynchronizationData.RareBackgroundService

// Id: #1062651
using RuketoAutomation.Service.Background;

var synchronizationData = Activate<SynchronizationData>();
await synchronizationData.RareBackgroundService();

Parameters

CurrentCulture 
"ru-RU"
CurrentUICulture 
"en-US"
RecurringJobId 
"rareJob"
RetryCount 
10
Time 
1739489588

State

14.02.2025 04:07:37 (+2.950s) Failed

An exception occurred during performance of the job.

System.InvalidOperationException (759e5c7b-e4e8-4ea3-9bca-a4b786b21a49)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1915(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
759e5c7b-e4e8-4ea3-9bca-a4b786b21a49
Worker:
6129a217

+1h 52m Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 10 of 10: Sequence contains more than one element

Enqueue at:
14.02.2025 04:07:30

+2.886s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (759e5c7b-e4e8-4ea3-9bca-a4b786b21a49)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1915(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
759e5c7b-e4e8-4ea3-9bca-a4b786b21a49
Worker:
49131932

+1h 11m Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 9 of 10: Sequence contains more than one element

Enqueue at:
14.02.2025 02:14:30

+2.711s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (759e5c7b-e4e8-4ea3-9bca-a4b786b21a49)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1915(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
759e5c7b-e4e8-4ea3-9bca-a4b786b21a49
Worker:
00cfbce5

+41m 58.253s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 8 of 10: Sequence contains more than one element

Enqueue at:
14.02.2025 01:02:23

+3.965s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (759e5c7b-e4e8-4ea3-9bca-a4b786b21a49)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1915(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
759e5c7b-e4e8-4ea3-9bca-a4b786b21a49
Worker:
49131932

+22m 12.746s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 7 of 10: Sequence contains more than one element

Enqueue at:
14.02.2025 00:20:19

+3.358s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (759e5c7b-e4e8-4ea3-9bca-a4b786b21a49)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1915(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+3ms Processing

Server:
759e5c7b-e4e8-4ea3-9bca-a4b786b21a49
Worker:
d2729edf

+12m 57.314s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 6 of 10: Sequence contains more than one element

Enqueue at:
13.02.2025 23:58:06

+3.264s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (759e5c7b-e4e8-4ea3-9bca-a4b786b21a49)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1915(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+3ms Processing

Server:
759e5c7b-e4e8-4ea3-9bca-a4b786b21a49
Worker:
a6d4d495

+6m 8.166s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 5 of 10: Sequence contains more than one element

Enqueue at:
13.02.2025 23:45:08

+7.136s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (759e5c7b-e4e8-4ea3-9bca-a4b786b21a49)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1915(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+3ms Processing

Server:
759e5c7b-e4e8-4ea3-9bca-a4b786b21a49
Worker:
e30c499c

+2m 42.131s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 4 of 10: Sequence contains more than one element

Enqueue at:
13.02.2025 23:38:41

+2.958s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (759e5c7b-e4e8-4ea3-9bca-a4b786b21a49)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1915(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+3ms Processing

Server:
759e5c7b-e4e8-4ea3-9bca-a4b786b21a49
Worker:
6129a217

+1m 27.133s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 3 of 10: Sequence contains more than one element

Enqueue at:
13.02.2025 23:35:59

+2.922s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (759e5c7b-e4e8-4ea3-9bca-a4b786b21a49)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1915(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+3ms Processing

Server:
759e5c7b-e4e8-4ea3-9bca-a4b786b21a49
Worker:
21146410

+41.545s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 2 of 10: Sequence contains more than one element

Enqueue at:
13.02.2025 23:34:28

+3.474s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (759e5c7b-e4e8-4ea3-9bca-a4b786b21a49)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1915(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+3ms Processing

Server:
759e5c7b-e4e8-4ea3-9bca-a4b786b21a49
Worker:
6129a217

+43.869s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 1 of 10: Sequence contains more than one element

Enqueue at:
13.02.2025 23:33:43

+2.953s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (759e5c7b-e4e8-4ea3-9bca-a4b786b21a49)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1915(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+7ms Processing

Server:
759e5c7b-e4e8-4ea3-9bca-a4b786b21a49
Worker:
21146410

+1ms Enqueued

Triggered by recurring job scheduler

13.02.2025 23:33:08 Created