Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add screenshots of current interface and viewport to every chat message #779

Open
arjunrajlab opened this issue Sep 19, 2024 · 0 comments
Open
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@arjunrajlab
Copy link
Collaborator

arjunrajlab commented Sep 19, 2024

It helps the chat bot to have a screenshot of the current ui and viewport. That way it can "see" what you are looking at. I have found this helps. So it would be good to have the following features:

Every time you submit a message, there are two "hidden" images that are added to the prompt. One is the interface (which I already made in a branch), and the other is the viewport. For the viewport I think we can use the same code from download current viewport in the snapshots panel, but that code is broken right now so I wasn't sure how best to do it.

In the chatbot overall, it would be good to have it run a hidden prompt when the conversation first starts (either first open of the chat if the chat is empty, or if the user hits refresh). Basically, it should generate a user message that is "hello, here's my current context, I will ask you some follow up questions" along with the two screenshots. This is hidden from the user. Then the first output from the assistant will be the opening of the chat. That way, the chat starts from the context already, and the chatbot has already said "hello".

@arjunrajlab arjunrajlab added the enhancement New feature or request label Sep 19, 2024
@arjunrajlab arjunrajlab added this to the Alpha-Version milestone Sep 19, 2024
@arjunrajlab arjunrajlab moved this to Selected for development in Alpha Release Sep 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Selected for development
Development

No branches or pull requests

2 participants