Hangfire Dashboard

SynchronizationData.RareBackgroundService

// Id: #994603
using RuketoAutomation.Service.Background;

var synchronizationData = Activate<SynchronizationData>();
await synchronizationData.RareBackgroundService();

Parameters

CurrentCulture 
"ru-RU"
CurrentUICulture 
"en-US"
RecurringJobId 
"rareJob"
RetryCount 
10
Time 
1736209324

State

07.01.2025 04:52:05 (+5.172s) Failed

An exception occurred during performance of the job.

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method2434(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+14ms Processing

Server:
351e21d3-cf4c-44fb-b7ac-9ba688c76648
Worker:
8784f5b6

+1h 51m Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 10 of 10: Sequence contains more than one element

Enqueue at:
07.01.2025 04:51:46

+2.539s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method2434(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
351e21d3-cf4c-44fb-b7ac-9ba688c76648
Worker:
de4a57ea

+1h 8m Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 9 of 10: Sequence contains more than one element

Enqueue at:
07.01.2025 03:00:44

+2.943s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method2434(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
351e21d3-cf4c-44fb-b7ac-9ba688c76648
Worker:
e85b3bf9

+43m 58.581s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 8 of 10: Sequence contains more than one element

Enqueue at:
07.01.2025 01:52:04

+2.765s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method2434(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
351e21d3-cf4c-44fb-b7ac-9ba688c76648
Worker:
c4221a77

+22m 27.458s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 7 of 10: Sequence contains more than one element

Enqueue at:
07.01.2025 01:08:08

+3.725s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method2434(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
351e21d3-cf4c-44fb-b7ac-9ba688c76648
Worker:
b184aae6

+11m 41.812s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 6 of 10: Sequence contains more than one element

Enqueue at:
07.01.2025 00:45:36

+3.885s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method2434(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+3ms Processing

Server:
351e21d3-cf4c-44fb-b7ac-9ba688c76648
Worker:
09919a67

+5m 12.435s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 5 of 10: Sequence contains more than one element

Enqueue at:
07.01.2025 00:33:46

+2.831s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method2434(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
351e21d3-cf4c-44fb-b7ac-9ba688c76648
Worker:
e85b3bf9

+3m 12.550s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 4 of 10: Sequence contains more than one element

Enqueue at:
07.01.2025 00:28:36

+2.622s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method2434(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
351e21d3-cf4c-44fb-b7ac-9ba688c76648
Worker:
19c71c6b

+1m 56.338s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 3 of 10: Sequence contains more than one element

Enqueue at:
07.01.2025 00:25:08

+3.779s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method2434(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:
351e21d3-cf4c-44fb-b7ac-9ba688c76648
Worker:
c4221a77

+42.381s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 2 of 10: Sequence contains more than one element

Enqueue at:
07.01.2025 00:23:13

+2.649s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method2434(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+3ms Processing

Server:
351e21d3-cf4c-44fb-b7ac-9ba688c76648
Worker:
c172f44f

+29.562s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 1 of 10: Sequence contains more than one element

Enqueue at:
07.01.2025 00:22:26

+2.581s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method2434(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+5ms Processing

Server:
351e21d3-cf4c-44fb-b7ac-9ba688c76648
Worker:
752f8bcf

+1ms Enqueued

Triggered by recurring job scheduler

07.01.2025 00:22:04 Created