Problem Statement

🧠 Assignment: FinSense — The Autonomous Financial Copilot

Proposal

• Multi-turn agent with step 0 which finds which modalities/tools to be called and the subsequent modalities as well
• Agent has capabilities to fetch live stock prices with yfinance, web search with SerpApi, and macro data using FRED API
• Agent uses the modalities results and builds a coherent answer
• The intermediate steps are shown to the user and the llm thoughts and modality results are just logged internally for further testing
• The entire chat history is in the context so that the conversation is flowing
  • The chat history could be truncated or compacted smartly when the context window is full
• Agent Framework
  • User writes a query with some attachments with it which is sent to the server.
  • The agent starts a loop till max 25 steps
  • Preamble is a doc that is the root of the system prompt
    • It explains what the assistant is about
    • It explains how to navigate the steps
    • It explains that capabilities and their tool explanation for future use
  • If step 0 then step 0 preamble is called else non step 0 preamble
  • System Prompt is created with the Preamble and context from previous modalities
  • System Prompt and chat history is sent to LLM to process
  • The results has <fs_think> tags with routing info or response texts to be sent back
  • If the last agent response doesn’t have any modalities/tools to fetch then the loop ends with a done state
• No user auth and the chats are public (just for assignment sake)
• Frontend - Simple react chat app

Memory

• Long term memory isn’t build in this prototype but it can be achieved by building either
  • ChatGPT style chat notes with (add, remove, and edit) notes functions.
  • Or summarisation based vector embeddings with a new modality to fetch then when necessary
  • a User memory doc which gets added to the system prompt always, and gets updated with each user query and any other user ingested data

Bonus capabilities

• Agentic Research Mode - the agent can perform very deep research if the query required it to
• File Uploads - when the frontend is ready, users can upload csv, pdf files of holdings for better context information
• Scenario Simulator - As the agent is already research ready so experimenting with any scenario like “What if Fed hikes 25bps?” is already inbuilt
• Explain-as-you-go Mode - The agent explains its steps intermediately. It can be tuned later to reduce or increase the explanation size