Building Ghostwriter Chat

To reduce the hassle of developers searching Stack Overflow or checking documentation one by one, Replit developed Ghostwriter Chat, which allows interaction within the IDE as if talking with a teammate. This product started during Hackweek and was completed in about 6-8 weeks.

LLM(Large Language Models) is the core technology behind Ghostwriter Chat. In particular, to minimize response latency, Streaming was introduced instead of waiting for results to be fully generated. This provides a fast user experience where users can see responses within 500ms of a request.

For effective answers, the Prompt Construction process is very important. Technology was applied to gather information from various sources, select appropriate context, and convert it into various prompt formats that can support multiple models and APIs.

Additionally, Ghostwriter Debugger was implemented to support different error output methods for each program. It handles various error situations, from web servers that continuously output logs to scripts that immediately halt execution.

To solve the Token Limits problem, the following strategies are used:

• Chat history management: Older messages are deleted with a focus on the most recent messages to maintain conversational context.
• Message size limit: User messages are limited to 500 characters to secure room in the prompt.
• Smart context selection: Instead of including all code, only the most important information is selected and passed to the model.