SynchronizationData.RareBackgroundService

28.12.2024 04:04:49 (+2.820s) Failed

An exception occurred during performance of the job.

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method2434(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+5ms Processing

Server:: 351e21d3-cf4c-44fb-b7ac-9ba688c76648
Worker:: c172f44f

+1h 50m Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 10 of 10: Sequence contains more than one element

Enqueue at:: 28.12.2024 04:04:42

+2.732s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method2434(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+3ms Processing

Server:: 351e21d3-cf4c-44fb-b7ac-9ba688c76648
Worker:: c4221a77

+1h 11m Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 9 of 10: Sequence contains more than one element

Enqueue at:: 28.12.2024 02:13:43

+3.076s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method2434(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:: 351e21d3-cf4c-44fb-b7ac-9ba688c76648
Worker:: 8351b9d5

+42m 43.727s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 8 of 10: Sequence contains more than one element

Enqueue at:: 28.12.2024 01:02:15

+3.716s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method2434(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:: 351e21d3-cf4c-44fb-b7ac-9ba688c76648
Worker:: 0ce60401

+24m 42.961s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 7 of 10: Sequence contains more than one element

Enqueue at:: 28.12.2024 00:19:28

+3.316s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method2434(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:: 351e21d3-cf4c-44fb-b7ac-9ba688c76648
Worker:: 0ce60401

+10m 58.475s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 6 of 10: Sequence contains more than one element

Enqueue at:: 27.12.2024 23:54:53

+3.803s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method2434(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+3ms Processing

Server:: 351e21d3-cf4c-44fb-b7ac-9ba688c76648
Worker:: e2d9be78

+6m 43.932s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 5 of 10: Sequence contains more than one element

Enqueue at:: 27.12.2024 23:43:48

+2.736s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method2434(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+3ms Processing

Server:: 351e21d3-cf4c-44fb-b7ac-9ba688c76648
Worker:: c754169b

+2m 26.302s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 4 of 10: Sequence contains more than one element

Enqueue at:: 27.12.2024 23:36:54

+5.587s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method2434(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:: 351e21d3-cf4c-44fb-b7ac-9ba688c76648
Worker:: 9c2122bd

+1m 26.076s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 3 of 10: Sequence contains more than one element

Enqueue at:: 27.12.2024 23:34:22

+3.994s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method2434(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+5ms Processing

Server:: 351e21d3-cf4c-44fb-b7ac-9ba688c76648
Worker:: c172f44f

+1m 9.630s Enqueued

Triggered by DelayedJobScheduler

+1ms Scheduled

Retry attempt 2 of 10: Sequence contains more than one element

Enqueue at:: 27.12.2024 23:32:58

+5.450s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method2434(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+4ms Processing

Server:: 351e21d3-cf4c-44fb-b7ac-9ba688c76648
Worker:: e85b3bf9

+37.511s Enqueued

Triggered by DelayedJobScheduler

+ <1ms Scheduled

Retry attempt 1 of 10: Sequence contains more than one element

Enqueue at:: 27.12.2024 23:31:34

+2.631s Failed

An exception occurred during performance of the job.

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

Sequence contains more than one element

System.InvalidOperationException: Sequence contains more than one element
   at System.Linq.ThrowHelper.ThrowMoreThanOneElementException()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method2434(Closure , QueryContext )
   at RuketoAutomation.Service.Background.SynchronizationData.SaveEntity[T](T crmEntity) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 463
   at RuketoAutomation.Service.Background.SynchronizationData.LoadRareDataFromApi[TConvertData](GetDataDelegate getDataDelegate, ConvertDataDelegate`1 convertDataDelegate, GetCountPageDelegate getCountPageDelegate) in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 422
   at RuketoAutomation.Service.Background.SynchronizationData.CrmRareSynchronize() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 369
   at RuketoAutomation.Service.Background.SynchronizationData.RareBackgroundService() in /home/developer/code/build/RuketoAutomation/Service/Background/SynchronizationData.cs:line 347
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

+6ms Processing

Server:: 351e21d3-cf4c-44fb-b7ac-9ba688c76648
Worker:: e2d9be78

+2ms Enqueued

Triggered by recurring job scheduler

CurrentCulture	`"ru-RU"`
CurrentUICulture	`"en-US"`
RecurringJobId	`"rareJob"`
RetryCount	`10`
Time	`1735342267`

SynchronizationData.RareBackgroundService

Parameters

Requeue Delete State

28.12.2024 04:04:49 (+2.820s) Failed

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

+5ms Processing

+1h 50m Enqueued

+ <1ms Scheduled

+2.732s Failed

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

+3ms Processing

+1h 11m Enqueued

+ <1ms Scheduled

+3.076s Failed

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

+4ms Processing

+42m 43.727s Enqueued

+ <1ms Scheduled

+3.716s Failed

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

+4ms Processing

+24m 42.961s Enqueued

+ <1ms Scheduled

+3.316s Failed

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

+4ms Processing

+10m 58.475s Enqueued

+ <1ms Scheduled

+3.803s Failed

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

+3ms Processing

+6m 43.932s Enqueued

+ <1ms Scheduled

+2.736s Failed

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

+3ms Processing

+2m 26.302s Enqueued

+1ms Scheduled

+5.587s Failed

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

+4ms Processing

+1m 26.076s Enqueued

+ <1ms Scheduled

+3.994s Failed

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

+5ms Processing

+1m 9.630s Enqueued

+1ms Scheduled

+5.450s Failed

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

+4ms Processing

+37.511s Enqueued

+ <1ms Scheduled

+2.631s Failed

System.InvalidOperationException (351e21d3-cf4c-44fb-b7ac-9ba688c76648)

+6ms Processing

+2ms Enqueued

27.12.2024 23:31:07 Created

State