Fix DISIDocIdStream::count so that it does not try to count beyond max

[ChrisHegarty]

Fix DISIDocIdStream::count so that it does not try to count beyond max.

From the perspective of DISIDocIdStream it should not be necessary to check the iterator position before calling count.

Without this fix, DISIDocIdStream::count can easily fail with IOOBE, e.g.

Range [0, -5) out of bounds for length 100
java.lang.IndexOutOfBoundsException: Range [0, -5) out of bounds for length 100
	at __randomizedtesting.SeedInfo.seed([1667A192A4675B09:1A4E2A88F38511BB]:0)
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:100)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckFromToIndex(Preconditions.java:112)
	at java.base/jdk.internal.util.Preconditions.checkFromToIndex(Preconditions.java:349)
	at java.base/java.util.Objects.checkFromToIndex(Objects.java:391)
	at org.apache.lucene.util.FixedBitSet.cardinality(FixedBitSet.java:212)
	at org.apache.lucene.search.DISIDocIdStream.count(DISIDocIdStream.java:63)
	at org.apache.lucene.search.DocIdStream.count(DocIdStream.java:50)
	at org.apache.lucene.search.TestDISIDocIdStream.testCountUpTo2(TestDISIDocIdStream.java:131)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)

[jpountz]

Thank you! Did it fail with an existing query/collector, or did you find it while trying to take advantage of DocIdStream for a new use-case?

[ChrisHegarty]

Thank you! Did it fail with an existing query/collector, or did you find it while trying to take advantage of DocIdStream for a new use-case?

We see DISIDocIdStream::count in stack traces of existing code (not new) in Elasticsearch, see elastic/elasticsearch#126939 (comment). In particular:

Caused by: java.lang.IndexOutOfBoundsException: Range [0, -14) out of bounds for length 4096
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:100)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckFromToIndex(Preconditions.java:112)
	at java.base/jdk.internal.util.Preconditions.checkFromToIndex(Preconditions.java:349)
	at java.base/java.util.Objects.checkFromToIndex(Objects.java:391)
	at org.apache.lucene.core@10.2.0/org.apache.lucene.util.FixedBitSet.cardinality(FixedBitSet.java:212)
	at org.apache.lucene.core@10.2.0/org.apache.lucene.search.DISIDocIdStream.count(DISIDocIdStream.java:63)
	at org.apache.lucene.core@10.2.0/org.apache.lucene.search.DocIdStream.count(DocIdStream.java:50)
	at org.apache.lucene.core@10.2.0/org.apache.lucene.search.TotalHitCountCollector$1.collect(TotalHitCountCollector.java:69)
	at org.apache.lucene.core@10.2.0/org.apache.lucene.search.DenseConjunctionBulkScorer.scoreWindow(DenseConjunctionBulkScorer.java:208)
	at org.apache.lucene.core@10.2.0/org.apache.lucene.search.DenseConjunctionBulkScorer.score(DenseConjunctionBulkScorer.java:132)
	at org.apache.lucene.core@10.2.0/org.apache.lucene.search.ReqExclBulkScorer.score(ReqExclBulkScorer.java:66)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:46)
...

I'm not yet able to reproduce with Elasticsearch, but it seems very likely to be the root cause of some of the failures we're seeing.