Why is the design so hard to extend?

[suvalaki]

Explain to me if im missing something about the code, im having some real trouble passing a requests session through to the scraper.

In general:

• The GPTResearcher uses name mangling to prevent override of the research methods
• The scrapers are in a non-extensible (in function) map
• request session handling is done at a scraper level

To me all this seems really bad. Im literally being forced to rebuild the entire classes and swap out methods. Is this a problem with me?

[ElishaKay]

Welcome @suvalaki

Please provide more context.

a) Which Scraping Code are you looking at?

From my understanding we have a legacy scraper which was built on Selenium

I don't believe that's in use currently by default.

Instead, have a look here at:

from langchain_community.document_loaders import WebBaseLoader

From my understanding we're currently leveraging Langchain's WebBaseLoader by default

Are the Langchain Loaders the underlying methods that you need to refactor?

Please feel free to flesh out the use cases you need to handle & which methods & classes you need to refactor to achieve what you need.

If you feel there is room for improvements we are always open to exploring PR's with new ideas

As they say in the movie Tron: "The thing about perfection is that it's unknowable." Therefore, software always has the potential to improve.

[suvalaki]

Ok i will make an effort to comment on where i think things could use improvement. Im welcome to you saying im off my rocker here. this is obviously an incredible project (even though im a little frustrated this afternoon).

gpt-researcher/gpt_researcher/scraper/web_base_loader/web_base_loader.py

Line 2 in 3a8c571

class WebBaseLoaderScraper:

https://github.com/assafelovic/gpt-researcher/blob/master/gpt_researcher/scraper/scraper.py#L26

Specifically here we are seeing that the session is hard coded. It is difficult therefore to manage either the bs4 or langchain variants of the getters sessions. Furthermore because of this we cant monkey patch requests.

This scraper map is an issue: https://github.com/assafelovic/gpt-researcher/blob/master/gpt_researcher/scraper/scraper.py#L73
Bringing in a custom scraper is pretty difficult primarily because of the following flow

• The scraper class is hardcoded in the action scrape_urls https://github.com/assafelovic/gpt-researcher/blob/master/gpt_researcher/master/actions.py#L255 . On its own this is fine But it is in combination with the following we have an issue
• scrape_urls is called by __get_context_by_urls. Because of method mangling it becomes much harder to replace the default scraper.

You basically need to vendor ALL of the code in agent and point the one method elsewhere.

Recommendations.

• Scraper object should be made extensible with *arg / **kwarg passthrough.
• Scraper initialization map should be made runtime configurable to a standard interface
• ALL __ methods should be made private via _ So that they can be modified.

And if i was writing this myself i might even have used composition instead of straight methods to create instances of the member objects during GPTResearcher initialisation so that those mechanisms could have been modified as strategies.

[ElishaKay]

haha - monkey patch 😉 jk.
genius is often misunderstood, so, nope doesn't sound like you're off your rocker.

In fact, sounds like some pretty solid logic on my end.

@assafelovic usually takes the lead on backend upgrades to the Arkenstone but I'm happy to help facilitate & collab to bring this to his attention.

Are you open to creating a PR with the suggestions you mentioned above?

As the good book says: "Whoever smelt it, dealt it"

If it doesn't introduce any breaking changes, I don't see why the community shouldn't benefit from your insights.
I can help with the automated tests as well to verify your upgrades are maintained in master

Are you already in the discord?

We're happy to welcome you into the community & it sounds like you got some good ideas

[suvalaki]

im happy to contribute. I will probably need to find time in the back half of this week. I should have a reasonable idea what needs to be done since ive been vendoring this today.

Im not on the discord. Ill look to connect once i free up a bit more

[assafelovic]

@suvalaki thanks for this great feedback. Your feedback is exactly why this is open source and a community purpose project. Agree with the fallbacks and would love your help and contribution around it. Seems like you're the expert for this improvement, wdyt?