Unhandled Exception in Server Container, Android App does not reconnect automatically

[jacotec]

Version

1.0.32.0

Deployment Method

Docker Container on Immich-VM (Server, Ubuntu 24.04LTS)

Description

I'm running Immich Frame since a few days, frontend is a Frameo Frame running Immich-Frame Android.

Out of sudden my wife sent me a picture of the frame this morning:

The connection of the frame is fine, I can ping it from my network.

I've checked the server container logs and found an unhandled exception (see below).

The issue did not stop the server container from working in general - I could connect from within my network via browser without any issues.

But whatever this did, it "killed" the function of my Frame which needs a restart to get it working again.

Due to the exception above I'll file this issue here and not in the "Immich-Frame Android" - however, IMHO the app should periodically try to reconnect to the server in case something goes wrong.

Reproduction

No reproduction, it happened after a few days with no user action.

Expectations

The unhandled exception should not occur, plus the Android App should retry itself to reestablish the connection.

Configuration

# settings applicable to the web client - when viewing with a browser or webview
General:
  # When set, every client needs to authenticate via Bearer Token and this value.
  AuthenticationSecret: null # string, no default
  # whether to download images to the server
  DownloadImages: false  # boolean
  # if images are downloaded, re-download if age (in days) is more than this
  RenewImagesDuration: 0  # int
  # A list of webcalendar URIs in the .ics format. Supports basic auth via standard URL format.
  # e.g. https://calendar.google.com/calendar/ical/XXXXXX/public/basic.ics
  # e.g. https://user:pass@calendar.immichframe.dev/dav/calendars/basic.ics
  #Webcalendars:  # string[]
  #  - UUID
  # Interval in hours. Determines how often images are pulled from a person in immich.
  RefreshAlbumPeopleInterval: 0  # int
  # Date format. See https://date-fns.org/v4.1.0/docs/format for more information.
  PhotoDateFormat: 'dd.MM.yyyy'  # string
  ImageLocationFormat: 'City,State,Country'
  # Get an API key from OpenWeatherMap: https://openweathermap.org/appid
  WeatherApiKey: ''  # string
  # Imperial or metric system (Fahrenheit or Celsius)
  UnitSystem: 'metric'  # 'imperial' | 'metric'
  # Set the weather location with lat/lon.
  #WeatherLatLong: '40.730610,-73.935242'  # string
  # 2 digit ISO code, sets the language of the weather description.
  Language: 'de'  # string
  # Webhook URL to be notified e.g. http://example.com/notify
  Webhook: null  # string
  # Image interval in seconds. How long an image is displayed in the frame.
  Interval: 20
  # Duration in seconds.
  TransitionDuration: 2  # float
  # Displays the current time.
  ShowClock: true  # boolean
  # Time format
  ClockFormat: 'HH:mm'  # string
  # Date format for the clock
  ClockDateFormat: 'eee, MMM d' # string
  # Displays the progress bar.
  ShowProgressBar: true  # boolean
  # Displays the date of the current image.
  ShowPhotoDate: true  # boolean
  # Displays the description of the current image.
  ShowImageDesc: true  # boolean
  # Displays a comma separated list of names of all the people that are assigned in immich.
  ShowPeopleDesc: true  # boolean
  # Displays a comma separated list of names of all the tags that are assigned in immich.
  ShowTagsDesc: true  # boolean
  # Displays a comma separated list of names of all the albums for an image.
  ShowAlbumName: false  # boolean
  # Displays the location of the current image.
  ShowImageLocation: true  # boolean
  # Lets you choose a primary color for your UI. Use hex with alpha value to edit opacity.
  PrimaryColor: '#f5deb3'  # string
  # Lets you choose a secondary color for your UI. (Only used with `style=solid or transition`) Use hex with alpha value to edit opacity.
  SecondaryColor: '#000000'  # string
  # Background-style of the clock and metadata.
  Style: 'none'  # none | solid | transition | blur
  # Sets the base font size, uses standard CSS formats (https://developer.mozilla.org/en-US/docs/Web/CSS/font-size)
  BaseFontSize: '17px'  # string
  # Displays the description of the current weather.
  ShowWeatherDescription: false  # boolean
  # URL for the icon to load for the current weather condition
  WeatherIconUrl: 'https://openweathermap.org/img/wn/{IconId}.png'
  # Zooms into or out of an image and gives it a touch of life.
  ImageZoom: false  # boolean
  # Pans an image in a random direction and gives it a touch of life.
  ImagePan: false  # boolean
  # Whether image should fill available space. Aspect ratio maintained but may be cropped.
  ImageFill: false  # boolean
  # Whether to play audio for videos that have audio tracks.
  PlayAudio: false  # boolean
  # Allow two portrait images to be displayed next to each other
  Layout: 'single'  # single | splitview

# multiple accounts permitted
Accounts:
  - # The URL of your Immich server e.g. `http://photos.yourdomain.com` / `http://192.168.0.100:2283`.
    ImmichServerUrl: 'https://***redacted***'  # string, required, no default
    # Read more about how to obtain an Immich API key: https://immich.app/docs/features/command-line-interface#obtain-the-api-key
    # Exactly one of ApiKey or ApiKeyFile must be set.
    ApiKey: "***redacted***"
    # ApiKeyFile: "/path/to/api.key"
    # Show images after date. Overwrites the `ImagesFromDays`-Setting
    ImagesFromDate: null  # Date
    # If this is set, memories are displayed.
    ShowMemories: false  # boolean
    # If this is set, favorites are displayed.
    ShowFavorites: false  # boolean
    # If this is set, assets marked archived are displayed.
    ShowArchived: false  # boolean
    # If this is set, video assets are included in the slideshow.
    ShowVideos: false  # boolean
    # Show images from the last X days, e.g., 365 -> show images from the last year
    ImagesFromDays: null  # int
    # Show images before date.
    #ImagesUntilDate: '2020-01-02'  # Date
    # Rating of an image in stars, allowed values from -1 to 5. This will only show images with the exact rating you are filtering for.
    Rating: null  # int
    # UUID of album(s) - e.g. ['00000000-0000-0000-0000-000000000001']
    Albums:  # string[]
      - 1a5ee408-0d3e-4490-99ad-899f73adcb49
    # UUID of excluded album(s)
    #ExcludedAlbums:  # string[]
    #  - UUID
    # UUID of People
    #People:  # string[]
    #  - UUID
    # Tag values (full hierarchical paths, case-sensitive)
    #Tags:  # string[]
    #  - "Vacation"
    #  - "Travel/Europe"

Logs

26-04-02 10:15:39 info: System.Net.Http.HttpClient.ImmichApiAccountClient.ClientHandler[100] Sending HTTP request GET https://mydomain.de/api/albums/***redacted***
26-04-02 10:15:39 info: System.Net.Http.HttpClient.ImmichApiAccountClient.ClientHandler[101] Received HTTP response headers after 121.5697ms - 200
26-04-02 10:15:39 info: System.Net.Http.HttpClient.ImmichApiAccountClient.LogicalHandler[101] End processing HTTP request after 121.7193ms - 200
26-04-02 10:15:41 fail: Microsoft.AspNetCore.Server.Kestrel[13] Connection id "0HNKBGBS9GTG3", Request id "0HNKBGBS9GTG3:00000001": An unhandled exception was thrown by the application. System.Net.Http.HttpIOException: The response ended prematurely, with at least 597364 additional bytes expected. (ResponseEnded)    at System.Net.Http.HttpConnection.ContentLengthReadStream.Read(Span`1 buffer)    at System.IO.StreamReader.ReadBuffer(Span`1 userBuffer, Boolean& readToUserBuffer)    at System.IO.StreamReader.ReadSpan(Span`1 buffer)    at Newtonsoft.Json.JsonTextReader.ReadData(Boolean append, Int32 charsRequired)    at Newtonsoft.Json.JsonTextReader.ReadStringIntoBuffer(Char quote)    at Newtonsoft.Json.JsonTextReader.ParseProperty()    at Newtonsoft.Json.JsonTextReader.ParseObject()    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id)    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)    at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)    at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)    at ImmichFrame.Core.Api.ImmichApi.ReadObjectResponseAsync[T](HttpResponseMessage response, IReadOnlyDictionary`2 headers, CancellationToken cancellationToken)    at ImmichFrame.Core.Api.ImmichApi.GetAlbumInfoAsync(Guid id, String key, Nullable`1 withoutAssets, CancellationToken cancellationToken) in /source/ImmichFrame.Core/obj/immich-openapi-specsClient.cs:line 1826    at ImmichFrame.Core.Logic.Pool.AlbumAssetsPool.LoadAssets(CancellationToken ct) in /source/ImmichFrame.Core/Logic/Pool/AlbumAssetsPool.cs:line 17    at ImmichFrame.Core.Helpers.AssetExtensionMethods.ApplyAccountFilters(Task`1 unfilteredAssets, IAccountSettings accountSettings, IEnumerable`1 excludedAlbumAssets) in /source/ImmichFrame.Core/Helpers/AssetExtensionMethods.cs:line 15    at Microsoft.Extensions.Caching.Memory.CacheExtensions.GetOrCreateAsync[TItem](IMemoryCache cache, Object key, Func`2 factory, MemoryCacheEntryOptions createOptions)    at ImmichFrame.Core.Logic.Pool.CachingApiAssetsPool.AllAssets(CancellationToken ct) in /source/ImmichFrame.Core/Logic/Pool/CachingApiAssetsPool.cs:line 25    at ImmichFrame.Core.Logic.Pool.CachingApiAssetsPool.GetAssets(Int32 requested, CancellationToken ct) in /source/ImmichFrame.Core/Logic/Pool/CachingApiAssetsPool.cs:line 18    at ImmichFrame.Core.Logic.Pool.MultiAssetPool.GetNextAsset(CancellationToken ct) in /source/ImmichFrame.Core/Logic/Pool/MultiAssetPool.cs:line 20    at ImmichFrame.Core.Logic.Pool.IAssetPool.WaitAssets(Int32 requested, Func`2 supplier, Nullable`1 cancellationToken) in /source/ImmichFrame.Core/Logic/Pool/IAssetPool.cs:line 22    at ImmichFrame.Core.Logic.AccountSelection.TotalAccountImagesSelectionStrategy.<>c__DisplayClass9_0.<<GetAssets>b__2>d.MoveNext() in /source/ImmichFrame.Core/Logic/AccountSelection/TotalAccountImagesSelectionStrategy.cs:line 61 --- End of stack trace from previous location ---    at ImmichFrame.Core.Logic.AccountSelection.TotalAccountImagesSelectionStrategy.GetAssets() in /source/ImmichFrame.Core/Logic/AccountSelection/TotalAccountImagesSelectionStrategy.cs:line 66    at ImmichFrame.Core.Logic.MultiImmichFrameLogicDelegate.GetAssets() in /source/ImmichFrame.Core/Logic/MultiImmichFrameLogicDelegate.cs:line 35    at ImmichFrame.WebApi.Controllers.AssetController.GetAssets(String clientIdentifier) in /source/ImmichFrame.WebApi/Controllers/AssetController.cs:line 42    at lambda_method30(Closure, Object)    at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task
lastTask, State next, Scope scope, Object state, Boolean isCompleted)    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)    at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)    at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)    at CustomAuthenticationMiddleware.InvokeAsync(HttpContext context) in /source/ImmichFrame.WebApi/Helpers/CustomAuthenticationMiddleware.cs:line 23    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

Pre-Submission Checklist

• This is a bug report and not a feature request

• I have provided all of the required information to reproduce the bug (config, logs, etc.)

• I have checked for related issues and checked the documentation

[3rob3]

Are you going through a reverse proxy with SSL?

[jacotec]

No, because of the cert issue in Android 6 (the Frameo OS). Running via my HAProxy, but on HTTP (restricted the access to my home LAN and "friendly IP's" here).

[3rob3]

your URL shows https. Can you check your HAProxy logs also?

[jacotec]

Sorry, I thought you're referring to the connection from the client (Frameo) to the container - this is HTTP without SSL.

The connection between the server container and Immich uses HTTPS, yes. Also via HAProxy. But no action at 10:15:41 where the exception happened. These are the only connections to Immich Server in this minute:

The last exchange in the HAProxy log is the one which completed 2 seconds before the issue:

26-04-02 10:15:39 info: System.Net.Http.HttpClient.ImmichApiAccountClient.ClientHandler[100] Sending HTTP request GET https://mydomain.de/api/albums/***redacted***
26-04-02 10:15:39 info: System.Net.Http.HttpClient.ImmichApiAccountClient.ClientHandler[101] Received HTTP response headers after 121.5697ms - 200
26-04-02 10:15:39 info: System.Net.Http.HttpClient.ImmichApiAccountClient.LogicalHandler[101] End processing HTTP request after 121.7193ms - 200
26-04-02 10:15:41 fail: Microsoft.AspNetCore.Server.Kestrel[13] Connection id "0HNKBGBS9GTG3", Request id "0HNKBGBS9GTG3:00000001": An unhandled exception was thrown by the application. 

[jacotec]

BTW: The last transfer before the exception has the "CD" in the HAProxy logs. This means the client (the ImmichFrame server container) did close the connection during data phase (before all data of the request was transferred). Probably related to the exception ...

This issue is not a tragedy as it does not stop the server container from working. The main problem is the issue I've filed in Immichframe-Android - the Android app does not reconnect automatically and needs a restart to work again. You don't need to touch the server container here, even after the exception,

[3rob3]

Android app does have retry on error. This an error message with no http error code. How big is this album? Does increasing RefreshAlbumPeopleInterval >0 have any effect? I'm kinda stumped....

[jacotec]

Currently the album has 759 pictures. I can try to set RefreshAlbumPeopleInterval to 1 for a test, but it'll be good to know how this works.

Does the server container pull the album inventory list in this interval, store it somewhere and then download periodically a number of random assets from the server to stream them to the client?

Server VM has plenty of resources (12 GB RAM, 16 vCPU), so it does not lack of resources.

Wife reported it crashed again, will be home soon and check the container logs.

[3rob3]

Currently the album has 759 pictures. I can try to set RefreshAlbumPeopleInterval to 1 for a test, but it'll be good to know how this works.

Yes, this is just a test to get more info

[jacotec]

Sure. Stupidly I restarted the container yesterday before I checked the logs. Running with RefreshAlbumPeopleInterval=1 fine since then. I'll observe and report.

[3rob3]

This would only minimize the problem anyway. I think it is something to do with HAProxy and/or the VM's virtual networking card. There is a rather large json response coming that sounds like it is getting corrupted or cut short, causing us to kill the connection. You could try disabling Hardware Offloading in your VM's virtual NIC. Could also try lowering the VM NIC MTU to 1400. I am admittedly out of my element here on the fix (you probably know way more about this side than I do), but I am pretty sure of the root cause.

[jacotec]

Thanks for your thoughts, Rob. Usually there is no issue with HAProxy, I'm running around 60 VMs in my Homelab and most of them are using HAProxy. However, I'll try to narrow this down. I think I'll make an attempt by connecting to Immich via Localhost (as the Frame Container is running in the same VM), eliminating all external networking. Let's see how this goes.

Nevertheless what wonders me most - As mentioned the Frame container runs perfectly after that issue (I can connect via browser flawlessly) - why does the Android App not reconnect although you mention it should do that.

[3rob3]

Android does not re-connect/refresh because there was no http error code. It doesn't know anything was wrong. If you swipe down for settings, then close settings with no change it will force a refresh and start working again.

[jacotec]

This is interesting. When I stop the container, change ImmichServerUrl: from my "HAProxy" URL to "http://localhost:2283" or "http://10.0.1.16:2283" (the latter is the IP of the VM), nothing works. I immediately get:

26-04-03 20:09:48 fail: ImmichFrame.Core.Logic.AccountSelection.BloomFilterAssetAccountTracker[0] Failed to locate account for asset b3ed7097-8b29-4069-9aa7-c47f44d382cc
26-04-03 20:09:48 fail: Microsoft.AspNetCore.Server.Kestrel[13] Connection id "0HNKHLGJGQH66", Request id "0HNKHLGJGQH66:00000001": An unhandled exception was thrown by the application. ImmichFrame.Core.Exceptions.AssetNotFoundException: Exception of type 'ImmichFrame.Core.Exceptions.AssetNotFoundException' was thrown.    at ImmichFrame.Core.Logic.AccountSelection.BloomFilterAssetAccountTracker.ForAsset[T](String assetId, Func`2 f) in /source/ImmichFrame.Core/Logic/AccountSelection/BloomFilterAssetAccountTracker.cs:line 43    at ImmichFrame.Core.Logic.AccountSelection.TotalAccountImagesSelectionStrategy.ForAsset[T](Guid assetId, Func`2 f) in /source/ImmichFrame.Core/Logic/AccountSelection/TotalAccountImagesSelectionStrategy.cs:line 86    at ImmichFrame.Core.Logic.MultiImmichFrameLogicDelegate.GetAsset(Guid assetId, Nullable`1 assetType, String rangeHeader) in /source/ImmichFrame.Core/Logic/MultiImmichFrameLogicDelegate.cs:line 47    at ImmichFrame.WebApi.Controllers.AssetController.GetAsset(Guid id, String clientIdentifier, Nullable`1 assetType) in /source/ImmichFrame.WebApi/Controllers/AssetController.cs:line 87    at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync() --- End of stack trace from previous location ---    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)    at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)    at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)    at CustomAuthenticationMiddleware.InvokeAsync(HttpContext context) in /source/ImmichFrame.WebApi/Helpers/CustomAuthenticationMiddleware.cs:line 23    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

From any machine in the LAN a browser to http://10.0.1.16:2283 works. So the server container needs the official reverse proxy URL to work (I have no idea why).

However, after the Android app lost contact due to the issue above, it recovers after changing the URL back to the HAProxy one.