How to limit the maximum number of pages in pagy?

[bnknbnkn]

In my application, I want to limit a specified maximum page number for paging, so that when the user tries to request a page number that exceeds the maximum page number I set, the request is still the maximum page number I set.

What if this effect is achieved?

Thank you very much.

[ddnexus]

As you probably already know, there's likely no performance advantage in limiting the collection from a good behaving DB.
Whether you have 10M or 100 records, that will be just a total number that the user likely will never browse, especially if it's thousands of pages .

But let's assume you have a different (good) reason to do so. The technical trick with pagy is quite simple.

Just do a count query on the collection, and cap it at a maximum number of records.

max_records = max_pages * records_per_page
count = collection.count
count = max_records if count > max_records
@pagy, @records = pagy(collection, count: count)

You may also want put some caption explaining the user what you are doing, and maybe also why you limit the access to the first max_records records in order to avoid any possible backlash for manipulating the results without telling.

[bnknbnkn]

Sorry, I did not describe it clearly. I am using searchkick with elasticsearch. I found that the count parameter does not work in pagy_searchkick.

Is it because it is not supported yet?

Thank you for your reply.

[bnknbnkn]

collection = Product.search("*", where: { id: :desc })
@pagy, @products = pagy_searchkick(collection, size: 6, count: 800, items: 40)

There are more than 1000 records in my product table, I limited the count parameter to 800, theoretically I can only get to 20 page at most, but in fact my last page is 27.

BTW: Is there a way to make searchkick also use Countless?

Thank you very much.

[ddnexus]

Well well well... It looks like you are trying to use tools that you don't know, on something that you know less. What about dedicating a bit of time to actually learn the tools the old way (i.e. RTFM?) before trying to use them by guessing?

Is it because it is not supported yet?

"Hey, this carpet doesn't fly. Is it because it is not supported yet?" No, it's because it's a carpet. 😀

Is there a way to make searchkick also use Countless?

"Hey, is there a way to put this carburator in this EV and make it run on gas?" What? 🙃

But let's assume you have a different (good) reason to do so.

I better stop "supposing" that, because it looks very likely that we have a XY problem here.

So please, explain well why do you want to limit your results.

[bnknbnkn]

Sorry, there is something wrong with my expression. 🥲 I have no doubt about the power and flexibility of pagy. In fact, pagy is the most flexible paging gem I have ever seen. Thank you very much for providing such a powerful tool for free.

The reason I want to limit the number of pages is to prevent my products from being crawled. I will provide users with filters through detailed classification tags, related recommendations, search and other mechanisms so that they can find the products they want. But I don't want all items to be crawled. At the same time, it is also to reduce the load on the server.

Maybe this won't be a problem for some people who are good at crawling, but I'm trying to find more ways to avoid being crawled.

Thank you very much.

[ddnexus]

My previous answer was not trying to defend pagy at all: as you noted, it defends itself pretty well. It was trying to show that your questions imply profound ignorance (meaning lack of very basic knowledge) about the tools (searchkick/elasticsearch) that are involved, and trying to push you to RTFM, which would certainly help you a lot more than me and/or guessing.

The reason I want to limit the number of pages is to prevent my products from being crawled.
Maybe this won't be a problem for some people who are good at crawling, but I'm trying to find more ways to avoid being crawled.

Well, limiting the results is likely worse than the problem that you are trying to solve. Either your users won't find all the results they are searching for or the crawlers will find exactly the same results, just with a lot of more requests by following all the "filters through detailed classification tags, related recommendations, search and other mechanisms so [...] they can find [all] the products they want [anyway]"

It seems that you are headed to over-engineer your UI, over-complicate the maintenance of your app and worsen the user experience... all because - again - you are guessing that the right "secret weapon" to achieve anti-crawling is limiting the results... and all that happening even before you have an app that is worth crawling.

My strong recommendations are:

1. Get your app done well, totally ignoring the crawlers
2. Expand your user base to make your app relevant and worth all the work involved
3. If and when your app will become a target of crawling... use the right tools for anti-crawling ("limiting the results" not being one of them)

BTW, if you still keep thinking that limiting the results is your secret weapon, then you do that directly in elasticsearch. Pagy has nothing to do with it when it paginates results from search tools (e.g. elasticsearch, meilisearch) that return the count with the results (which is also the reason why your question about "countless" with searchkick makes no sense).

[benkoshy]

I would suggest that your time and resources be better directed towards improving your product offering: such that the copy-cats cannot replicate it easily:

• Your customers will thank you, and crawling cannot hurt you.
• moreover, limiting your app by such means has the unintended consequence of molesting your legit customers (for questionable gain?).

I have nothing more to add than what @ddnexus: the advice is good, and is as perfect as is able to be conceived.

[bnknbnkn]

Thank you for your advice and guidance on my application.

I will try another way to solve this problem.

[bnknbnkn]

Just found out kaminari supports this, I'll give it a try.

kaminari config

[bnknbnkn]

kaminari hasn’t been updated in a long time, I decided not to use it.

Back to Pagy, I read the code of pagy's documentation again and made the following modifications to achieve my purpose.

Add the following configuration to config/initializers/pagy.rb:

require "pagy/extras/overflow"
Pagy::DEFAULT[:overflow] = :last_page
require "pagy/extras/searchkick"
Searchkick.extend Pagy::Searchkick

module PagyExtension
  # Create a Pagy object from a Searchkick::Results object
  def new_from_searchkick(results, vars = {})
    vars[:items] = results.options[:per_page]
    vars[:page] = results.options[:page]
    vars[:count] ||= results.total_count
    Pagy.new(vars)
  end
end
Pagy.extend PagyExtension

This is done mainly because the count parameter will be forced to be results.total_count, which causes the count parameter I set for the new_from_searchkick method to not work. So I overridden it.

Use passive mode:

@products = Product.search("*", page: param_page, per_page: param_limit)
@pagy = Pagy.new_from_searchkick(@products, count: 400, items: param_limit)

Now, after items is set to 40, the total number of pages is limited to 10 pages.

[ddnexus]

Now, after items is set to 40, the total number of pages is limited to 10 pages.

... and now you have to maintain a totally unnecessary hack, just for the sake of not following (or maybe not understanding?) the advice of doing "that directly in elasticsearch", which would also have other bigger advantages.

BTW, your hack is incomplete, because it will ALWAYS return a fixed number of pages, even if you have less or none. And you managed to overflow the Overflow extra, which will be useless.

Happy that you found your very own perfect solution! 👍🏻

[ddnexus]

From 8.1.+ pagy implemented the max_pages

However, as already explained that's not the right tool against crawling, and in this specific case, doing it at the source (i.e in elasticsearch), would have more advantages, but at least it's not a half-backed hack to maintain.