Free AI tool · reads your live robots.txt
AI crawler checker and robots.txt tester.
Enter a URL to see which AI crawlers may fetch it: training crawlers such as GPTBot and ClaudeBot, and AI search and assistant crawlers such as OAI-SearchBot and PerplexityBot. Each verdict names the robots.txt line that decides it. Or paste a robots.txt to test it before it goes live.
Worked example · qalbit.com/robots.txt as served 2026-10-18
https://qalbit.com/ Press Check to fetch the live file and the page’s own tags.
Training and data collection
0 of 60 blocked
AI search and assistants
0 of 73 blocked
Other or unclear
0 of 47 blocked
robots.txt
16 groups, 1 sitemaps
Any crawler name, e.g. Googlebot or Bytespider
Training and data collection0 blocked · 60 allowed
| Crawler | Verdict | Why |
|---|---|---|
| AgentTimesThe Agent Times | Allowed | Line 2: Allow: /Data Scraper from RSS Feeds. |
| AI2BotAi2 | Allowed | Line 2: Allow: /Content is used to train open language models. |
| Ai2Bot-DolmaAi2 | Allowed | Line 2: Allow: /Content is used to train open language models. |
| aiHitBotaiHit | Allowed | Line 2: Allow: /A massive, artificial intelligence/machine learning, automated system. |
| AmazonbotAmazon | Allowed | Line 56: Allow: /Amazon: "may be used to train Amazon AI models" (developer.amazon.com/amazonbot, checked 2026-10-18). |
| anthropic-aiAnthropic | Allowed | Line 2: Allow: /Scrapes data to train Anthropic's AI products. |
| Applebot-ExtendedApple | Allowed | Line 50: Allow: /Apple: opts content out of training Apple foundation models; it does not crawl pages itself (support.apple.com/en-us/119829, checked 2026-10-18). |
| AwarioAwario | Allowed | Line 2: Allow: /AI Data Scrapers |
| Brightbothttps://brightdata.com/brightbot | Allowed | Line 2: Allow: /LLM/AI training. |
| BytespiderByteDance | Allowed | Line 2: Allow: /LLM training. |
| CCBotCommon Crawl Foundation | Allowed | Line 59: Allow: /Provides open crawl dataset, used for many purposes, including Machine Learning/AI. |
| ChatGLM-SpiderOperator unclear | Allowed | Line 2: Allow: /AI Data Scrapers |
| ClaudeBotAnthropic | Allowed | Line 44: Allow: /Scrapes data to train Anthropic's AI products. |
| CloudVertexBotOperator unclear | Allowed | Line 2: Allow: /AI Data Scrapers |
| cohere-training-data-crawlerCohere to download training data for its LLMs (Large Lang... | Allowed | Line 2: Allow: /AI Data Scrapers |
| CotoyogiROIS | Allowed | Line 2: Allow: /AI LLM Scraper. |
| Datenbank CrawlerDatenbank | Allowed | Line 2: Allow: /AI Data Scrapers |
| DeepSeekBotDeepSeek | Allowed | Line 2: Allow: /Training language models and improving AI products |
| Echobot BotEchobox | Allowed | Line 2: Allow: /AI Data Scrapers |
| ERNIEBotBaidu | Allowed | Line 2: Allow: /Collects public web content for Baidu's ERNIE large language models. |
| FacebookBotMeta/Facebook | Allowed | Line 2: Allow: /Training language models |
| Factset_spyderbotFactset | Allowed | Line 2: Allow: /AI model training. |
| FriendlyCrawlerUnknown | Allowed | Line 2: Allow: /We are using the data from the crawler to build datasets for machine learning experiments. |
| Google-ExtendedGoogle | Allowed | Line 47: Allow: /Google: a robots.txt token only, no separate crawler; controls use for training and grounding Gemini models, not Google Search (developers.google.com/search/docs/crawling-indexing/google-common-crawlers, checked 2026-10-18). |
| GPTBotOpenAI | Allowed | Line 41: Allow: /Scrapes data to train OpenAI's products. |
| HunyuanTencent | Allowed | Line 2: Allow: /Hunyuan is a training crawler operated by Tencent. |
| ICC-CrawlerNICT | Allowed | Line 2: Allow: /Scrapes data to train and support AI technologies. |
| imageSpiderOperator unclear | Allowed | Line 2: Allow: /AI Data Scrapers |
| img2datasetimg2dataset | Allowed | Line 2: Allow: /Scrapes images for use in LLMs. |
| ISSCyberRiskCrawlerISS-Corporate | Allowed | Line 2: Allow: /Scrapes data to train machine learning models. |
| Kangaroo BotOperator unclear | Allowed | Line 2: Allow: /AI Data Scrapers |
| KimiBotMoonshot AI | Allowed | Line 2: Allow: /AI Data Scrapers |
| laion-huggingface-processorOperator unclear | Allowed | Line 2: Allow: /AI Data Scrapers |
| LAIONDownloaderLarge-scale Artificial Intelligence Open Network | Allowed | Line 2: Allow: /AI tools and models for machine learning research. |
| LCCOperator unclear | Allowed | Line 2: Allow: /AI Data Scrapers |
| LightpandaAnyone who downloads the Lightpanda client | Allowed | Line 2: Allow: /AI Data Scrapers |
| meta-externalagentMeta | Allowed | Line 53: Allow: /Used to train models and improve products. |
| MistralAI-TrainingMistral AI | Allowed | Line 2: Allow: /AI Data Scrapers |
| MoonshotBotKimi | Allowed | Line 2: Allow: /MoonshotBot is a training crawler. |
| MyCentralAIScraperBotOperator unclear | Allowed | Line 2: Allow: /AI data scraper |
| NagetBotNaget Inc (founded by Chris Samarinas | Allowed | Line 2: Allow: /AI data scraper |
| netEstate Imprint CrawlernetEstate | Allowed | Line 2: Allow: /AI Data Scrapers |
| newsaiOperator unclear | Allowed | Line 2: Allow: /AI data scraper |
| omgiliWebz.io | Allowed | Line 2: Allow: /Data is sold. |
| omgilibotWebz.io | Allowed | Line 2: Allow: /Data is sold. |
| PanguBotthe Chinese company Huawei | Allowed | Line 2: Allow: /AI Data Scrapers |
| PanscientPanscient | Allowed | Line 2: Allow: /Data collection and analysis using machine learning and AI. |
| QwenBotAlibaba | Allowed | Line 2: Allow: /Collects public web content for Alibaba's Qwen (Tongyi Qianwen) large language models. |
| ScrapyZyte | Allowed | Line 2: Allow: /Scrapes data for a variety of uses including training AI. |
| Sidetrade indexer botSidetrade | Allowed | Line 2: Allow: /Extracts data for a variety of uses including training AI. |
| SpiderOperator unclear | Allowed | Line 2: Allow: /AI Data Scrapers |
| TikTokSpiderByteDance | Allowed | Line 2: Allow: /LLM training. |
| TimpibotTimpi | Allowed | Line 2: Allow: /Scrapes data for use in training LLMs. |
| VelenPublicWebCrawlerVelen Crawler | Allowed | Line 2: Allow: /Scrapes data for business data sets and machine learning models. |
| WARDBotWEBSPARK | Allowed | Line 2: Allow: /AI Data Scrapers |
| Webzio-ExtendedOperator unclear | Allowed | Line 2: Allow: /AI Data Scrapers |
| YandexAdditionalYandex | Allowed | Line 2: Allow: /Scrapes/analyzes data for the YandexGPT LLM. |
| YandexAdditionalBotYandex | Allowed | Line 2: Allow: /Scrapes/analyzes data for the YandexGPT LLM. |
| YiBot01.ai | Allowed | Line 2: Allow: /This bot crawls the web to gather data and content for indexation and AI model training. |
| YouBotYou | Allowed | Line 2: Allow: /Scrapes data for search engine and LLMs. |
AI search and assistants0 blocked · 73 allowed
| Crawler | Verdict | Why |
|---|---|---|
| AddSearchBotOperator unclear | Allowed | Line 2: Allow: /AI Search Crawlers |
| AI2Bot-DeepResearchEvalAi2 | Allowed | Line 2: Allow: /AI Assistants |
| AIWebIndexLyrenth | Allowed | Line 2: Allow: /AI Search Crawlers |
| amazon-kendraAmazon | Allowed | Line 2: Allow: /Collects data for AI natural language search |
| amazon-QBusinessOperator unclear | Allowed | Line 2: Allow: /AI Assistants |
| AmazonBuyForMeAmazon | Allowed | Line 2: Allow: /AI Agents |
| Amzn-SearchBotOperator unclear | Allowed | Line 2: Allow: /AI Search Crawlers |
| Amzn-UserAmazon | Allowed | Line 2: Allow: /Amazon: user actions such as answering Alexa queries; "it may not follow all robots.txt directives" (developer.amazon.com/amazonbot, checked 2026-10-18). |
| AndibotAndi | Allowed | Line 2: Allow: /Search engine using generative AI, AI Search Assistant |
| AnomuraDireqt | Allowed | Line 2: Allow: /Collects data for AI search |
| ApplebotOperator unclear | Allowed | Line 2: Allow: /Apple: powers search in Spotlight, Siri and Safari; to keep content out of Apple model training, block Applebot-Extended instead (support.apple.com/en-us/119829, checked 2026-10-18). |
| atlassian-botAtlassian | Allowed | Line 2: Allow: /AI search, assistants and agents |
| AzureAI-SearchBotOperator unclear | Allowed | Line 2: Allow: /AI Search Crawlers |
| bigsur.aiBig Sur AI that fetches website content to enable AI-powe... | Allowed | Line 2: Allow: /AI Assistants |
| Channel3BotOperator unclear | Allowed | Line 2: Allow: /AI Search Crawlers |
| ChatGPT AgentOpenAI | Allowed | Line 2: Allow: /AI Agents |
| ChatGPT-UserOpenAI | Allowed | Line 20: Allow: /OpenAI: user-initiated; "robots.txt rules may not apply" to it (developers.openai.com/api/docs/bots, checked 2026-10-18). |
| Claude-CodeOperator unclear | Allowed | Line 2: Allow: /AI Coding Agents |
| Claude-SearchBotAnthropic | Allowed | Line 23: Allow: /Claude-SearchBot navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses. |
| Claude-UserAnthropic | Allowed | Line 26: Allow: /AI Assistants |
| Cloudflare-AutoRAGCloudflare | Allowed | Line 2: Allow: /Collects data for AI search |
| CodeOperator unclear | Allowed | Line 2: Allow: /AI Coding Agents |
| cohere-aiCohere | Allowed | Line 2: Allow: /Retrieves data to provide responses to user-initiated prompts. |
| CursorOperator unclear | Allowed | Line 2: Allow: /AI Coding Agents |
| DevinDevin AI | Allowed | Line 2: Allow: /AI Coding Agents |
| Diffbot-UserDiffbot | Allowed | Line 2: Allow: /AI Assistants |
| DoubaoBotByteDance | Allowed | Line 2: Allow: /AI crawler for ByteDance's Doubao AI assistant. |
| DuckAssistBotOperator unclear | Allowed | Line 35: Allow: /AI Assistants |
| ExaSearchBotExa | Allowed | Line 2: Allow: /AI Search Crawlers |
| GeistHaus-PageFetcherGeistHaus | Allowed | Line 2: Allow: /AI Assistants |
| Gemini-Deep-ResearchOperator unclear | Allowed | Line 2: Allow: /AI Assistants |
| Google-AgentOperator unclear | Allowed | Line 2: Allow: /AI Agents |
| Google-Gemini-CLIOperator unclear | Allowed | Line 2: Allow: /AI Coding Agents |
| Google-NotebookLMOperator unclear | Allowed | Line 2: Allow: /AI Assistants |
| GoogleAgent-MarinerGoogle | Allowed | Line 2: Allow: /AI Agents |
| GoogleAgent-URLContextGoogle that retrieves web content on behalf of Gemini API... | Allowed | Line 2: Allow: /AI Assistants |
| GrokAppAndroidxAI | Allowed | Line 2: Allow: /AI Search Crawlers |
| GrokBotxAI | Allowed | Line 2: Allow: /AI Search Crawlers |
| iaskspideriAsk | Allowed | Line 2: Allow: /Crawls sites to provide answers to user queries. |
| kagi-fetcherKagi that fetches web content to answer user queries thro... | Allowed | Line 2: Allow: /AI Assistants |
| KeenableBotKeenable | Allowed | Line 2: Allow: /AI Search Crawlers |
| Kimi-AgentMoonshot AI that browses websites while carrying out user... | Allowed | Line 2: Allow: /AI Agents |
| Kimi-SearchBotMoonshot AI | Allowed | Line 2: Allow: /AI Search Crawlers |
| Kimi-UserMoonshot AI that fetches web content on behalf of users i... | Allowed | Line 2: Allow: /AI Assistants |
| KlaviyoAIBotKlaviyo | Allowed | Line 2: Allow: /AI Assistants |
| LinerBotOperator unclear | Allowed | Line 2: Allow: /AI Assistants |
| LinkupBotOperator unclear | Allowed | Line 2: Allow: /AI Search Crawlers |
| Manus-UserButterfly Effect | Allowed | Line 2: Allow: /AI Agents |
| meta-externalfetcherOperator unclear | Allowed | Line 2: Allow: /AI Assistants |
| meta-webindexerMeta | Allowed | Line 2: Allow: /AI Assistants |
| MistralAI-IndexMistral AI | Allowed | Line 2: Allow: /Indexes web content for Mistral AI's search, used to answer questions in Le Chat. Per Mistral, not used for model training. |
| MistralAI-UserMistral | Allowed | Line 38: Allow: /AI Assistants |
| nbotNBot | Allowed | Line 2: Allow: /AI Search Crawlers |
| NotebookLMOperator unclear | Allowed | Line 2: Allow: /AI Assistants |
| NovaActOperator unclear | Allowed | Line 2: Allow: /AI Agents |
| OAI-SearchBotOpenAI | Allowed | Line 17: Allow: /Search result generation. |
| opencodeOperator unclear | Allowed | Line 2: Allow: /AI Coding Agents |
| OperatorOperator unclear | Allowed | Line 2: Allow: /AI Agents |
| Perplexity-UserPerplexity | Allowed | Line 32: Allow: /AI Assistants |
| PerplexityBotPerplexity | Allowed | Line 29: Allow: /Search result generation. |
| PetalBotHuawei | Allowed | Line 2: Allow: /Used to provide recommendations in Hauwei assistant and AI search services. |
| PhindBotphind | Allowed | Line 2: Allow: /AI Assistants |
| Poggio-CitationsPoggio | Allowed | Line 2: Allow: /AI Assistants |
| qodercliQoder that helps developers build | Allowed | Line 2: Allow: /AI Coding Agents |
| QualifiedBotQualified | Allowed | Line 2: Allow: /AI Assistants |
| SemrushBot-SWASemrush | Allowed | Line 2: Allow: /Checks URLs on your site for SEO Writing Assistant. |
| Shap-UserOperator unclear | Allowed | Line 2: Allow: /AI Assistants |
| TongyiBotAlibaba that fetches web content for the Tongyi Qianwen a... | Allowed | Line 2: Allow: /AI Assistants |
| TraeOperator unclear | Allowed | Line 2: Allow: /AI Coding Agents |
| TwinAgentTwin | Allowed | Line 2: Allow: /AI Agents |
| UseAIOperator unclear | Allowed | Line 2: Allow: /AI Assistants |
| YiyanBotBaidu that fetches web content for the yiyan | Allowed | Line 2: Allow: /AI Assistants |
| ZanistaBotOperator unclear | Allowed | Line 2: Allow: /AI Search Crawlers |
Other or unclear0 blocked · 47 allowed
| Crawler | Verdict | Why |
|---|---|---|
| AgentDataBotOperator unclear | Allowed | Line 2: Allow: /AI Data Providers |
| ApifyBotOperator unclear | Allowed | Line 2: Allow: /AI Data Providers |
| ApifyWebsiteContentCrawlerOperator unclear | Allowed | Line 2: Allow: /AI Data Providers |
| Aranet-SearchBotOperator unclear | Allowed | Line 2: Allow: /Undocumented AI Agents |
| bedrockbotAmazon | Allowed | Line 2: Allow: /Data scraping for custom AI applications. |
| BixelBotOperator unclear | Allowed | Line 2: Allow: /AI Data Providers |
| Bravebothttps://safe.search.brave.com/help/brave-search-crawler | Allowed | Line 2: Allow: /AI Data Providers |
| BuddyBotBuddyBotLearning | Allowed | Line 2: Allow: /AI Learning Companion |
| Claude-WebAnthropic | Allowed | Line 2: Allow: /Undocumented AI Agents |
| CloudflareBrowserRenderingCrawlerCloudflare that returns rendered website content for rese... | Allowed | Line 2: Allow: /AI Data Providers |
| CragCrawlerCragSoftware | Allowed | Line 2: Allow: /AI Data Providers |
| Crawl4AIOperator unclear | Allowed | Line 2: Allow: /Undocumented AI Agents |
| CrawlspaceCrawlspace | Allowed | Line 2: Allow: /AI Data Providers |
| DiffbotDiffbot | Allowed | Line 2: Allow: /AI Data Providers |
| EchoboxBotEchobox | Allowed | Line 2: Allow: /Data collection to support AI-powered products. |
| ExaBotOperator unclear | Allowed | Line 2: Allow: /AI Data Providers |
| facebookexternalhitMeta/Facebook | Allowed | Line 2: Allow: /Ostensibly only for sharing, but likely used as an AI crawler as well |
| FirecrawlAgentFirecrawl that extracts web content and converts it into ... | Allowed | Line 2: Allow: /AI Data Providers |
| Google-CloudVertexBotGoogle | Allowed | Line 2: Allow: /Build and manage AI models for businesses employing Vertex AI |
| Google-FirebaseGoogle | Allowed | Line 2: Allow: /Used as part of AI apps developed by users of Google's Firebase AI products. |
| GoogleOtherGoogle | Allowed | Line 2: Allow: /Scrapes data. |
| GoogleOther-ImageGoogle | Allowed | Line 2: Allow: /Scrapes data. |
| GoogleOther-VideoGoogle | Allowed | Line 2: Allow: /Scrapes data. |
| HenkBotOperator unclear | Allowed | Line 2: Allow: /AI Data Providers |
| iAskBotOperator unclear | Allowed | Line 2: Allow: /Undocumented AI Agents |
| ImagesiftBotImageSift | Allowed | Line 2: Allow: /ImageSiftBot is a web crawler that scrapes the internet for publicly available images to support their suite of web intelligence products |
| KunatoCrawlerOperator unclear | Allowed | Line 2: Allow: /Undocumented AI Agents |
| Linguee BotLinguee | Allowed | Line 2: Allow: /AI powered translation service |
| Mozilla-TabstackMozilla that performs programmatic | Allowed | Line 2: Allow: /AI Data Providers |
| OAI-AdsBotOpenAI | Allowed | Line 2: Allow: /Validates and targets ads on ChatGPT. |
| OpenAIOpenAI | Allowed | Line 2: Allow: /Unclear at this time. |
| Poseidon Research CrawlerPoseidon Research | Allowed | Line 2: Allow: /AI research crawler |
| Querit-SearchBotQuerit that indexes web content for their search API service | Allowed | Line 2: Allow: /AI Data Providers |
| QueritBotQuerit | Allowed | Line 2: Allow: /AI Data Providers |
| QuillBotQuillbot | Allowed | Line 2: Allow: /Company offers AI detection, writing tools and other services. |
| ReflectionbotReflection | Allowed | Line 2: Allow: /Undocumented AI Agents |
| SBIntuitionsBotSB Intuitions | Allowed | Line 2: Allow: /Uses data gathered in AI development and information analysis. |
| SemrushBot-OCOBSemrush | Allowed | Line 2: Allow: /Crawls your site for ContentShake AI tool. |
| ShapBotParallel | Allowed | Line 2: Allow: /AI Data Providers |
| TavilyBotOperator unclear | Allowed | Line 2: Allow: /AI Data Providers |
| Terra CottaOperator unclear | Allowed | Line 2: Allow: /AI Data Providers |
| TerraCottaCeramic AI | Allowed | Line 2: Allow: /AI Data Providers |
| ThinkbotThinkbot | Allowed | Line 2: Allow: /Insights on AI integration and automation. |
| wpbotQuantumCloud | Allowed | Line 2: Allow: /Live chat support and lead generation. |
| WRTNBotOperator unclear | Allowed | Line 2: Allow: /Undocumented AI Agents |
| xAI-GrokxAI | Allowed | Line 2: Allow: /xAI-Grok is an xAI crawler token associated with Grok content collection. |
| YaKMeltwater | Allowed | Line 2: Allow: /According to the Meltwater Consumer Intelligence page 'By applying AI, data science, and market research expertise to a live feed of global data sources, we transform unstructured data into actionable insights allowing better decision-making'. |
Crawler list: ai.robots.txt, commit 6abe630 (2026-10-18), MIT licence. The three groups are our reading of each crawler’s stated purpose. A verdict says what your robots.txt asks; whether a crawler honours it is up to its operator.
Rules applied, and where we read them
- RFC 9309, Robots Exclusion Protocol (IETF, September 2022) · checked 2026-10-18. Group selection, longest match, allow on ties, * and $, 4xx and 5xx handling, the 500 KiB parsing floor.
- ai.robots.txt, robots.json, commit 6abe630 (2026-10-18) (MIT licence) · checked 2026-10-18. The crawler list: 188 entries, merged by product token. The training, search and other groups are QalbIT’s reading of each entry’s stated function.
- OpenAI, Overview of OpenAI crawlers · checked 2026-10-18. GPTBot is for training, OAI-SearchBot for ChatGPT search, ChatGPT-User for user actions, where robots.txt rules may not apply.
- Google, Google’s common crawlers (Google-Extended) · checked 2026-10-18. Google-Extended is a token with no crawler of its own and does not affect Google Search inclusion or ranking.
- Apple, About Applebot · checked 2026-10-18. Applebot powers Spotlight, Siri and Safari search; Applebot-Extended opts out of model training and does not crawl.
- Amazon, Amazonbot and related agents · checked 2026-10-18. Amazonbot may be used to train Amazon AI models; Amzn-User may not follow all robots.txt directives.
What the checker reads
Which AI crawlers can read a page, and why.
01
Training and search are different crawlers
Most AI companies run one crawler that collects pages to train models and others that fetch pages for search answers or for a question a user asked. Blocking GPTBot does not block OAI-SearchBot. The checker groups them so you can see which ones you are letting in.
02
The line that decides it
For each crawler the checker finds the group that applies to it, then the longest matching rule, as RFC 9309 describes. You see the line number and the rule itself, so you know exactly what to change.
03
robots.txt is a request, not a lock
robots.txt asks crawlers to stay out; it does not stop them. OpenAI says robots.txt rules may not apply to ChatGPT-User, which fetches pages a user asks for. The page’s noindex and other robots tags are shown separately, because they govern indexing, not fetching.
FAQs · AI crawler checker
AI crawlers and robots.txt · frequently asked questions
Blocking GPTBot, what each crawler is for, and what the checker sends and keeps.
Add a group for it to your robots.txt: a line "User-agent: GPTBot" followed by "Disallow: /". OpenAI says GPTBot collects content to train its models, and that OAI-SearchBot, a separate crawler, decides whether your pages can appear in ChatGPT search. You can block one and allow the other.
OpenAI documents three agents. GPTBot gathers content for training. OAI-SearchBot surfaces sites in ChatGPT’s search features. ChatGPT-User fetches a page when a ChatGPT user asks for it; OpenAI says it is not used to crawl the web automatically and that robots.txt rules may not apply to it.
No. Google says Google-Extended is a robots.txt token with no crawler of its own, that it controls whether content Google crawls may be used to train and ground Gemini models, and that it does not affect inclusion or ranking in Google Search.
No. RFC 9309, the robots.txt standard, says its rules "are not a form of access authorization": they are a request crawlers are asked to honour. The checker tells you what your file asks of each crawler, not what any crawler actually does. To keep a page private, put it behind a login.
RFC 9309 says that when robots.txt is unreachable because of a server error (a 5xx status) or a network failure, crawlers must assume everything is disallowed. A 4xx status such as 404 means the file is unavailable, and crawlers may then fetch anything. The checker applies the same rules.
When you press Check, our server fetches three things from the site you entered: /robots.txt, the page itself and /llms.txt. It sends no cookies or logins, identifies itself as the QalbIT checker, refuses addresses on private networks, and stores neither the URL nor the response. Pasted robots.txt files never leave your browser.
The list is a dated snapshot of the open ai.robots.txt project, shown with its source and date under the results. Type any user agent into the custom field to test it against the same robots.txt.
Next step
Want your site ready for AI search?
QalbIT builds AI features and the web platforms behind them, including what AI search and assistants can read on your site and how your content is structured for them.
More free tools
More free tools
Each one opens on a worked example and shows its answer without a signup.
All free toolsLLM API pricing comparison and cost calculator
OpenAI, Claude, Gemini and more API prices per 1M tokens, with your monthly cost side by side.
LLM token counter and tokenizer
Count tokens for GPT, Claude and Gemini, see the split, and check the context window and cost.
AI agent and RAG cost calculator
What an AI agent and its RAG pipeline cost to run each month, and which line drives the bill.
Robots.txt generator for AI crawlers
Build a robots.txt that blocks AI training crawlers, keeps AI search, or blocks every AI bot.
llms.txt generator and validator
Write an llms.txt from your site’s name, summary and key links, or check one against the proposal.
Software development cost calculator
A typical cost and timeline range for your project, by type, scope and features.
