`TokenFilter#includeEmptyArray()` causes an error in some cases where an object element is filtered

[21Joakim]

In the following example scenario I want to only include the property one inside of the objects in the array and always include the array regardless if it is empty or not, but instead I am met with an exception in some cases where the objects inside of the array are also filtered (both partially and fully).

I tested a couple of different inputs to try and get a better idea of when the issue occurs, seemingly after the first element has been included in the array any object which ends with a filtered property will cause it to error

I tested it with both jackson 2.18.3 and 2.19.0-rc2 (the stack traces are from 2.19.0-rc2).

These cause an exception (each of them results in a different exception)

• [{"one":1},{"two":2}]
• [{"one":1},{"one":1,"two":2}]
• [{"one":1},{"one":1,"two":2},{"one":1}]

These work as expected

• [{"two":2}]
• [{"two":2},{"two":2}]
• [{"two":2},{"two":2},{"one":1}]
• [{"one":1}]
• [{"one":1},{"one":1}]
• [{"one":1},{"two":2,"one":1}]
• [{"one":1},{"two":2,"one":1},{"one":1}]

Some additional info

• When we instead return false in includeEmptyArray everything works as expected
• The same errors occur if we were to wrap the array in another object, like {"list":[{"one":1},{"two":2}]}, and change the includeProperty to name.equals("list") || name.equals("one")

Here's the code I used to test

public static class CustomTokenFilter extends TokenFilter {
	
	@Override
	public TokenFilter includeProperty(String name) {
		if(name.equals("one")) {
			return this;
		}
		
		return null;
	}
	
	@Override
	public boolean includeEmptyArray(boolean contentsFiltered) {
		return true;
	}
}

public static void main(String[] args) throws IOException {
	JsonMapper mapper = new JsonMapper();
	TokenFilter filter = new CustomTokenFilter();

	String content = "[{\"one\":1},{\"two\":2}]";
	try(JsonParser parser = new FilteringParserDelegate(mapper.createParser(content), filter, Inclusion.INCLUDE_ALL_AND_PATH, true)) {
		System.out.println(parser.readValueAsTree().toString());
	}
}

Stack Traces

[{"one":1},{"two":2}] results in

Exception in thread "main" com.fasterxml.jackson.core.JsonParseException: Internal error: failed to locate expected buffered tokens
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 22]
	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2590)
	at com.fasterxml.jackson.core.filter.FilteringParserDelegate._nextBuffered(FilteringParserDelegate.java:937)
	at com.fasterxml.jackson.core.filter.FilteringParserDelegate._nextToken2(FilteringParserDelegate.java:659)
	at com.fasterxml.jackson.core.filter.FilteringParserDelegate.nextToken(FilteringParserDelegate.java:557)
	at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer._deserializeContainerNoRecursion(JsonNodeDeserializer.java:608)
	at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:103)
	at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:24)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:342)
	at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4944)
	at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3165)
	at com.fasterxml.jackson.core.JsonParser.readValueAsTree(JsonParser.java:2564)

[{"one":1},{"one":1,"two":2}] results in

Exception in thread "main" com.fasterxml.jackson.databind.exc.MismatchedInputException: Unexpected end-of-input when trying read value of type `com.fasterxml.jackson.databind.JsonNode`
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 29]
	at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)
	at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1794)
	at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1568)
	at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1473)
	at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer._deserializeRareScalar(JsonNodeDeserializer.java:701)
	at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer._deserializeContainerNoRecursion(JsonNodeDeserializer.java:644)
	at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:103)
	at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:24)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:342)
	at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4944)
	at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3165)
	at com.fasterxml.jackson.core.JsonParser.readValueAsTree(JsonParser.java:2564)

[{"one":1},{"one":1,"two":2},{"one":1}] results in

Exception in thread "main" com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type `com.fasterxml.jackson.databind.JsonNode` from Object value (token `JsonToken.FIELD_NAME`)
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 37]
	at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)
	at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1794)
	at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1568)
	at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1473)
	at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer._deserializeRareScalar(JsonNodeDeserializer.java:701)
	at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer._deserializeContainerNoRecursion(JsonNodeDeserializer.java:644)
	at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:103)
	at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:24)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:342)
	at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4944)
	at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3165)
	at com.fasterxml.jackson.core.JsonParser.readValueAsTree(JsonParser.java:2564)

[cowtowncoder]

Thank you for reporting this, @21Joakim.

[cowtowncoder]

Fixed in 2.21.0 (and 3.0.3)