How CustomGPT.ai builds AI agent from my website?

Create a new agent

1. On your dashboard, click New Agent.

2. Click Website.

3. Enter a URL or a sitemap in the provided field.

4. Click Create Agent to begin indexing.

Sitemap search

CustomGPT.ai will first attempt to find a sitemap for the domain you provided. If a sitemap is found, CustomGPT.ai will crawl the pages listed on the sitemap, indexing the content for your AI agent.

If no sitemap is found

If a sitemap is not found, CustomGPT.ai will:

• Attempt to find all links on your website's main page and crawl them.
• Find links on the new pages discovered and crawl those as well.
• Continue this process until it reaches your per-agent page limit or no new links are found.

🚧

Note:

If no links are found, CustomGPT.ai will crawl and index just the single page you provided.

Same-domain crawling only

If no sitemap is found, CustomGPT.ai crawls only links on the same domain. For example, if your domain is https://customgpt.ai, it won’t crawl links like https://google.com or https://docs.customgpt.ai.