Hangfire Dashboard

SynchronizationData.RareBackgroundService

// Id: #1076395
using RuketoAutomation.Service.Background;

var synchronizationData = Activate<SynchronizationData>();
await synchronizationData.RareBackgroundService();

Parameters

CurrentCulture 
"ru-RU"
CurrentUICulture 
"en-US"
RecurringJobId 
"rareJob"
RetryCount 
10
Time 
1740184867

State

22.02.2025 05:12:18 (+2.748s) Failed

An exception occurred during performance of the job.

System.InvalidOperationException (52aaf2a5-19a4-467a-9cd0-b4b34f0bac95)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method4695(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
52aaf2a5-19a4-467a-9cd0-b4b34f0bac95
Worker:
089e5b3e

+1h 54m Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 10 of 10: Sequence contains more than one element

Enqueue at:
22.02.2025 05:12:07

+3.262s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (52aaf2a5-19a4-467a-9cd0-b4b34f0bac95)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method4695(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
52aaf2a5-19a4-467a-9cd0-b4b34f0bac95
Worker:
d9f52aa4

+1h 10m Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 9 of 10: Sequence contains more than one element

Enqueue at:
22.02.2025 03:17:54

+3.026s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (52aaf2a5-19a4-467a-9cd0-b4b34f0bac95)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method4695(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+3ms Processing

Server:
52aaf2a5-19a4-467a-9cd0-b4b34f0bac95
Worker:
0bbf2dac

+40m 43.140s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 8 of 10: Sequence contains more than one element

Enqueue at:
22.02.2025 02:06:45

+3.014s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (52aaf2a5-19a4-467a-9cd0-b4b34f0bac95)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method4695(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
52aaf2a5-19a4-467a-9cd0-b4b34f0bac95
Worker:
089e5b3e

+22m 12.094s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 7 of 10: Sequence contains more than one element

Enqueue at:
22.02.2025 01:26:03

+3.759s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (52aaf2a5-19a4-467a-9cd0-b4b34f0bac95)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method4695(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+5ms Processing

Server:
52aaf2a5-19a4-467a-9cd0-b4b34f0bac95
Worker:
0a1ef329

+12m 42.280s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 6 of 10: Sequence contains more than one element

Enqueue at:
22.02.2025 01:03:40

+3.475s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (52aaf2a5-19a4-467a-9cd0-b4b34f0bac95)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method4695(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
52aaf2a5-19a4-467a-9cd0-b4b34f0bac95
Worker:
a4215280

+4m 40.855s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 5 of 10: Sequence contains more than one element

Enqueue at:
22.02.2025 00:51:04

+4.408s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (52aaf2a5-19a4-467a-9cd0-b4b34f0bac95)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method4695(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+5ms Processing

Server:
52aaf2a5-19a4-467a-9cd0-b4b34f0bac95
Worker:
46839eff

+1m 53.424s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 4 of 10: Sequence contains more than one element

Enqueue at:
22.02.2025 00:46:22

+6.713s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (52aaf2a5-19a4-467a-9cd0-b4b34f0bac95)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method4695(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+6ms Processing

Server:
52aaf2a5-19a4-467a-9cd0-b4b34f0bac95
Worker:
72ac0796

+1m 27.320s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 3 of 10: Sequence contains more than one element

Enqueue at:
22.02.2025 00:44:09

+2.754s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (52aaf2a5-19a4-467a-9cd0-b4b34f0bac95)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method4695(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
52aaf2a5-19a4-467a-9cd0-b4b34f0bac95
Worker:
183ba583

+1m 12.186s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 2 of 10: Sequence contains more than one element

Enqueue at:
22.02.2025 00:42:41

+2.892s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (52aaf2a5-19a4-467a-9cd0-b4b34f0bac95)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method4695(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
52aaf2a5-19a4-467a-9cd0-b4b34f0bac95
Worker:
519760b1

+28.273s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 1 of 10: Sequence contains more than one element

Enqueue at:
22.02.2025 00:41:37

+2.793s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (52aaf2a5-19a4-467a-9cd0-b4b34f0bac95)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method4695(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /root/korobok/repo/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+6ms Processing

Server:
52aaf2a5-19a4-467a-9cd0-b4b34f0bac95
Worker:
521eadbd

+2ms Enqueued

Triggered by recurring job scheduler

22.02.2025 00:41:07 Created