I have an issue with signalr where memory explodes after calling sendasync

[luukvnes]

Hi I think there is a problem with signalr consuming huge amounts of memory.
I have an asp.net setup with signalr. At some point (not sure when exactly) memory usage spikes from a consist 15-25 mb to 3-4 gb. The dumps show that that is entirely caused by byte arrays[]. I am not able to reproduce the issue on my local machine, only on the production machine.

70b02531a150 292,067 4,864,317,945 System.Byte[]

dumpheap -type System.Byte[] shows a ton of 16/32KB byte[] (small part shown below).

7071267b5408     70b02531a150         16,408 
    7071267b9420     70b02531a150         16,408 
    7071267bd438     70b02531a150         32,792 
    7071267c5450     70b02531a150         16,408 
    7071267c9468     70b02531a150         16,408 
    7071267cd480     70b02531a150         16,408 
    7071267d1498     70b02531a150         16,408 
    7071267d54b0     70b02531a150         16,408 
    7071267d94c8     70b02531a150         16,408 
    7071267dd4e0     70b02531a150         32,792 
    7071267e54f8     70b02531a150         16,408 
    7071267e9510     70b02531a150         16,408 
    7071267ed528     70b02531a150         16,408 
    7071267f1540     70b02531a150         16,408 
    7071267f5558     70b02531a150         16,408 
    7071267f9570     70b02531a150         16,408 
    707126800028     70b02531a150         16,408

gcroot for each of these objects shows 39 unique roots with each similar to

    706f5dffd720 70b0248c4065 Microsoft.AspNetCore.SignalR.Protocol.JsonHubProtocol.WriteMessageCore(Microsoft.AspNetCore.SignalR.Protocol.HubMessage, System.Buffers.IBufferWriter`1<Byte>) [/_/src/SignalR/common/Protocols.Json/src/Protocol/JsonHubProtocol.cs @ 529]
        r14:
          -> 7070089eb440     System.Text.Json.Utf8JsonWriter 
          -> 7070089ea2b0     Microsoft.AspNetCore.Internal.MemoryBufferWriter 
          -> 707007065f40     System.Collections.Generic.List<Microsoft.AspNetCore.Internal.MemoryBufferWriter+CompletedBuffer> 
          -> 707009818098     Microsoft.AspNetCore.Internal.MemoryBufferWriter+CompletedBuffer[] 
          -> 7071267d94c8     System.Byte[] 

I'm not sure what causes this. I am not sending large messages (not more than 1 entity), the only thing that i can think of is circular references, but that should be disabled by IgnoreCycles (as shown below)

builder.Services.AddSignalR()
    .AddJsonProtocol(options =>
    {
        options.PayloadSerializerOptions.ReferenceHandler = ReferenceHandler.IgnoreCycles;
    });

After adding swap memory it held long enough to arrive at the following error message

System.ArgumentException: Destination is too short. (Parameter 'destination')
at Microsoft.AspNetCore.Internal.MemoryBufferWriter.ToArray()
at Microsoft.AspNetCore.SignalR.Protocol.HubProtocolExtensions.GetMessageBytes(IHubProtocol hubProtocol, HubMessage message)
at Microsoft.AspNetCore.SignalR.SerializedHubMessage.GetSerializedMessage(IHubProtocol protocol)
at Microsoft.AspNetCore.SignalR.HubConnectionContext.WriteCore(SerializedHubMessage message, CancellationToken cancellationToken)

[gfoidl]

@BrennanConroy PTAL

[BrennanConroy]

MemoryBufferWriter uses a ThreadStatic to avoid per serialization allocations. This almost makes it look like something is going async which would mess up the thread statics, but we don't use async in this area of code.

What do the thread stack traces look like? Are you able to share the dump or the app, or any other information?

[luukvnes]

Hi thanks for your help, to make sure the issue lies with signalr i tried serializing the object manually with

        var json = System.Text.Json.JsonSerializer.Serialize(alertUpdates, new JsonSerializerOptions
        {
            ReferenceHandler = ReferenceHandler.IgnoreCycles,
            WriteIndented = false
        });

This also seems to cause the same issue. So I think the issue lies with System.Text.Json and signalr is not the culprit.
It seems better to write a post over there.

[davidfowl]

Are you creating an options object per call?

[luukvnes]

I normally don't manually need to serialize objects and use the automatic serialization provided by controllers and signalr, ( and thus using options object in the setup stage using builder.services.... ), however in this case I wanted to check if it was due to serialization within signalr or if the issue lies with System.Text.Json.

[luukvnes]

Here are the objects i'm trying to serialize (i've taken out (what i think) are simple and uninteresting properties.

    public class Alert
    {
        public required Organization organization { get; set; }
        public required Camera camera { get; set; }
        public string? AlertGroupGroupId { get; set; }
    }

    public class AlertGroup
    {
        public required string GroupId { get; set; }
        public required List<Alert> Alerts { get; set; } // max 3
        public required List<Tag> groupedOnTags { get; set; } // max 2
        public bool groupedOnAllTag { get; set; }
        public required Organization organization { get; set; }
    }
    
     public class Camera
    {
        public required string Name { get; set; }
        public required string serial { get; set; }
        public string? image { get; set; } //low res image (max 5kb)
        public required Organization organization { get; set; }
        public required List<Tag> Tags { get; set; } // max 2
        public NpgsqlTypes.NpgsqlPoint? Location { get; set; }
    }
    
        public class Tag
    {
        public required string Name { get; set; }
        public NpgsqlTypes.NpgsqlPolygon? Area { get; set; }
        public required List<Camera> Cameras { get; set; } // max 16
        public required Organization Organization { get; set; }
    }

The new dump i have after using the system.text.json show a slightly different situation with

  78afa9400048     78efb9aaa150      2,097,176 
    78afa9600080     78efb9aaa150      8,388,632 
    78afab400048     78efb9aaa150     33,554,456 
    78afaf400048     78efb9aaa150     67,108,888 
    78afb5400048     78efb9aaa150    134,217,752 
    78afbf400048     78efb9aaa150    268,435,480 
    78afd1400048     78efb9aaa150    536,870,936 
    78aff3400048     78efb9aaa150  1,073,741,848 
    78f03120b028     78efb9aaa150             24 
    
Name:        System.Byte[]
MethodTable: 000078efb9aaa150
Canonical MethodTable: 000078efb9aaa150
Tracked Type: false
Size:        1073741848(0x40000018) bytes
Array:       Rank 1, Number of elements 1073741824, Type Byte
Content:     {"Alert":{"Id":49,"RuleResults":[{"Id":2888,"name":"plateTrack","serial":"X","organizationId":2,"success":1,"message"
Fields:
None

Part of the gcroot (total 62 unique roots) I can post the complete if interested

Thread c2e:
    78af01bfbb90 78efc19c1895 System.Text.Json.Utf8JsonWriter.WriteStringEscapeValue(System.ReadOnlySpan`1<Char>, Int32) [/_/src/libraries/System.Text.Json/src/System/Text/Json/Writer/Utf8JsonWriter.WriteValues.String.cs @ 192]
        rbx:
          -> 78afa2459f58     System.Text.Json.Utf8JsonWriter 
          -> 78aff3400048     System.Byte[] 

    78af01bfbca0 78efc19a1919 System.Text.Json.Serialization.JsonConverter`1[[System.__Canon, System.Private.CoreLib]].TryWrite(System.Text.Json.Utf8JsonWriter, System.__Canon ByRef, System.Text.Json.JsonSerializerOptions, System.Text.Json.WriteStack ByRef)
        r14:
          -> 78afa2459f58     System.Text.Json.Utf8JsonWriter 
          -> 78aff3400048     System.Byte[] 

    78af01bfbd20 78efc19a201c System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1[[System.__Canon, System.Private.CoreLib]].GetMemberAndWriteJson(System.Object, System.Text.Json.WriteStack ByRef, System.Text.Json.Utf8JsonWriter)
        rbx:
          -> 78afa2459f58     System.Text.Json.Utf8JsonWriter 
          -> 78aff3400048     System.Byte[] 

    78af01bfbd70 78efc19aaa73 System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1[[System.__Canon, System.Private.CoreLib]].OnTryWrite(System.Text.Json.Utf8JsonWriter, System.__Canon, System.Text.Json.JsonSerializerOptions, System.Text.Json.WriteStack ByRef)
        rbx:
          -> 78afa2459f58     System.Text.Json.Utf8JsonWriter 
          -> 78aff3400048     System.Byte[] 

    78af01bfbe10 78efc19a1743 System.Text.Json.Serialization.JsonConverter`1[[System.__Canon, System.Private.CoreLib]].TryWrite(System.Text.Json.Utf8JsonWriter, System.__Canon ByRef, System.Text.Json.JsonSerializerOptions, System.Text.Json.WriteStack ByRef)
        r14:
          -> 78afa2459f58     System.Text.Json.Utf8JsonWriter 
          -> 78aff3400048     System.Byte[] 

    78af01bfbe90 78efc19b8f32 System.Text.Json.Serialization.Converters.ListOfTConverter`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].OnWriteResume(System.Text.Json.Utf8JsonWriter, System.__Canon, System.Text.Json.JsonSerializerOptions, System.Text.Json.WriteStack ByRef)
        r14:
          -> 78afa2459f58     System.Text.Json.Utf8JsonWriter 
          -> 78aff3400048     System.Byte[] 

    78af01bfbef0 78efc19b8d00 System.Text.Json.Serialization.JsonCollectionConverter`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].OnTryWrite(System.Text.Json.Utf8JsonWriter, System.__Canon, System.Text.Json.JsonSerializerOptions, System.Text.Json.WriteStack ByRef)
        rbx:
          -> 78afa2459f58     System.Text.Json.Utf8JsonWriter 
          -> 78aff3400048     System.Byte[] 

    78af01bfbf60 78efc19a1743 System.Text.Json.Serialization.JsonConverter`1[[System.__Canon, System.Private.CoreLib]].TryWrite(System.Text.Json.Utf8JsonWriter, System.__Canon ByRef, System.Text.Json.JsonSerializerOptions, System.Text.Json.WriteStack ByRef)
        r14:
          -> 78afa2459f58     System.Text.Json.Utf8JsonWriter 
          -> 78aff3400048     System.Byte[] 

    78af01bfbfe0 78efc19a201c System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1[[System.__Canon, System.Private.CoreLib]].GetMemberAndWriteJson(System.Object, System.Text.Json.WriteStack ByRef, System.Text.Json.Utf8JsonWriter)
        rbx:
          -> 78afa2459f58     System.Text.Json.Utf8JsonWriter 
          -> 78aff3400048     System.Byte[] 

    78af01bfc030 78efc19aaa73 System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1[[System.__Canon, System.Private.CoreLib]].OnTryWrite(System.Text.Json.Utf8JsonWriter, System.__Canon, System.Text.Json.JsonSerializerOptions, System.Text.Json.WriteStack ByRef)
        rbx:
          -> 78afa2459f58     System.Text.Json.Utf8JsonWriter 
          -> 78aff3400048     System.Byte[] 

    78af01bfc0d0 78efc19a1743 System.Text.Json.Serialization.JsonConverter`1[[System.__Canon, System.Private.CoreLib]].TryWrite(System.Text.Json.Utf8JsonWriter, System.__Canon ByRef, System.Text.Json.JsonSerializerOptions, System.Text.Json.WriteStack ByRef)
        r14:
          -> 78afa2459f58     System.Text.Json.Utf8JsonWriter 
          -> 78aff3400048     System.Byte[] 

    78af01bfc150 78efc19b8f32 System.Text.Json.Serialization.Converters.ListOfTConverter`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].OnWriteResume(System.Text.Json.Utf8JsonWriter, System.__Canon, System.Text.Json.JsonSerializerOptions, System.Text.Json.WriteStack ByRef)
        r14:
          -> 78afa2459f58     System.Text.Json.Utf8JsonWriter 
          -> 78aff3400048     System.Byte[] 

    78af01bfc1b0 78efc19b8d00 System.Text.Json.Serialization.JsonCollectionConverter`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].OnTryWrite(System.Text.Json.Utf8JsonWriter, System.__Canon, System.Text.Json.JsonSerializerOptions, System.Text.Json.WriteStack ByRef)
        rbx:
          -> 78afa2459f58     System.Text.Json.Utf8JsonWriter 
          -> 78aff3400048     System.Byte[] 

    78af01bfc220 78efc19a1743 System.Text.Json.Serialization.JsonConverter`1[[System.__Canon, System.Private.CoreLib]].TryWrite(System.Text.Json.Utf8JsonWriter, System.__Canon ByRef, System.Text.Json.JsonSerializerOptions, System.Text.Json.WriteStack ByRef)
        r14:
          -> 78afa2459f58     System.Text.Json.Utf8JsonWriter 
          -> 78aff3400048     System.Byte[] 

[luukvnes]

Having tested on a slightly smaller set, it seems recursion is indeed the problem, a situation with 10 cameras and 2 tags containing all cameras results in a single camera being present ~500 times in the resulting json.