Paginator incorrectly thinks its done

[KristinnVikarJ]

If you for example have a userTimeline request that has the exclude: "retweets" property,
it has a chance of returning an empty page (no non-retweet tweets found), causing the timeLine to incorrectly determine that it's finished when you can still get more data.

Example Reproduction:

const user = await roClient.v2.userByUsername('the_piedude');
var id = user.data.id;
// Get The tweets from the user id
const timeLine = await roClient.v2.userTimeline(id, {exclude: "retweets"});
console.log(timeLine.done); // prints true
console.log(timeLine.tweets.length); // prints 0
await timeLine.fetchNext(100);
console.log(timeLine.tweets.length); // prints 21

[alkihis]

Hi,

It is indeed a problem...
In your first call, do you still receive metadata from Twitter?

const tl = await roClient.v2.userTimeline(id, {exclude: "retweets"});

// Ensure this is true
if (tl.done) {
  // @ts-ignore
  console.log(tl._realData.meta)
}

---

Update: This is, unfortunately, a problem coming from Twitter.
Twitter only returns those metas: { meta: { result_count: 0 } }, which doesn't contain next_token.

If next_token is not present, we cannot assume that a next page exists. (otherwise, if we consider it, in which case should we stop fetching from Twitter?) 😬

You can mitigate the issue by asking a way bigger max_results parameter in first request:

const timeline = await client.v2.userTimeline(id, { exclude: 'retweets', max_results: 100 });

[KristinnVikarJ]

Ah, I see.
My current hacky fix is to only stop fetching when I haven't gotten anything 3 times in a row (so after 300 tweets)

[alkihis]

In theory, fetch 1 or 3 times will not change anything, as Twitter does not send next_token if page contains 0 results (so, we have no way to fetch next page). You just try to fetch the same page (page 0) three times.

As it isn't a bug of this library, I'm transferring this issue as a discussion.