Scenarios for # comment triggered feedback providers

[StevenBucher98]

We are discussing what kind of scenarios there may be for feedback providers that are triggered by # comments in the console. Our one original scenario was for making calls to large language models so that the user can provider more input to the LLMs. This would be similar to the Codex CLI tool, https://github.com/microsoft/Codex-CLI. Current thinking is that a feedback provider could be developed like so:

PS> # How do I list only powershell files in a given directory?

[LLMFeedbackProvider]
  You can list PowerShell files in a given directory via:
    ➤ Get-ChildItem -Filter *.ps1 -Path <YourPath>             
PS> Get-ChildItem -Filter *.ps1 -Path <YourPath> # This is predicted and not accepted code 

This could also tie the feedback provider results into the predictive IntelliSense experience of PowerShell so users could then easily accept the solution given from a LLM.

Another consideration is that we can increase the timeout of feedback providers for this specific trigger since it can be implied that the users input is expecting a potential longer wait for results. Related to #19975, ultimately we need to have more discussion for the timeout for the other feedback provider triggers.

Some open questions for discussion:

• Is the LLM call scenario worth keeping the comment trigger for feedback providers?
• Is the UX something that would be used given a feedback provider would be created for this?
• Are there any more scenarios for feedback providers that trigger on comments?

Would love to hear folks thoughts! Personally I am struggling to find other scenarios that may be worth creating feedback providers and find the UX to not be as natural as a command that passes user prompt as an argument. The predictive intellisense tie in however is interesting and could make it an easier way for users to use the code they are getting suggested without copying and pasting with cursor.

cc @daxian-dbw @dfinke @ShaunLawrie

[ShaunLawrie]

I can't see any other scenarios for comment triggers at the moment. Even for LLM help the UX doesn't feel very natural because you're not explicitly invoking it. I thought about a prefix or something but at that point it may as well be a command.

The least natural part of the UX for me is that I paste a lot of snippets into the terminal that include comments that trigger it unexpectedly.

[StevenBucher98]

One benefit I see as a feedback provider is that it can be easily passed to the predictor interface so the user is suggested the command from the LLM as they start typing it... not exactly natural experience since they also have to start typing it and I would argue that the user would go to copy and paste before thinking to type it out.

[dfinke]

When I demo my PowerShell AI module one example I type in:

# what is my ip address

Then press ctrl+shift+g to trigger sending that as prompt to GPT.

Often, at the console, I start with a # and enter a cmd, or "note", etc. Then I press enter. This stores it in the PSReadline history so I can search for it later.

# trigger is nice, and may interrupt flow.

[daxian-dbw]

"The least natural part of the UX for me is that I paste a lot of snippets into the terminal that include comments that trigger it unexpectedly."

"Often, at the console, I start with a # and enter a cmd, or "note", etc. Then I press enter. This stores it in the PSReadline history so I can search for it later."

I'm most concerned about this too. Especially, when a feedback provider leveraging # as a trigger is imported, there is no way to get back the old # behavior -- all comments will be treated as queries and sent to the feedback provider even if they are really not.

[daxian-dbw]

I personally tend to think we should remove the comment trigger.

[StevenBucher98]

Another interesting thing I wanted to add is that warp.dev uses comments to query its terminal A.I feature:

[dfinke]

is that the warp terminal? Do you have a link to this feature?

[StevenBucher98]

Yep this is the warp terminal, https://www.warp.dev/, https://www.warp.dev/warp-ai

[ShaunLawrie]

Trying to handle this in the pshost might conflict with the terminals who are all racing to add co-pilots like Warp has. I wouldn't be surprised if this type of comment trigger is included in Windows Terminal co-pilot https://blogs.windows.com/windowsdeveloper/2023/05/23/bringing-the-power-of-ai-to-windows-11-unlocking-a-new-era-of-productivity-for-customers-and-developers-with-windows-copilot-and-dev-home/

I've been waiting to get my hands on the preview of that 😁

[dfinke]

Yup. Been beating the bushes to get the terminal copilot. Hopefully it won't be too generic. Don't know if they'll be using the same backend as GH CopilotX is. Hopefully they will.

[StevenBucher98]

We have decided to remove this option per #20136