# What is Robots.txt? | Yungle glossary

> Robots.txt is a file that tells search engine crawlers which parts of a site they can or can't access.

Source: https://yungle.ai/glossary/robots-txt/

1.  [Resources](https://yungle.ai/resources)
2.  Glossary

Glossary · R

# Robots.txt

In plain words

Robots.txt is a plain text file placed at a website's root that gives instructions to search engine crawlers about which pages or sections they're allowed or disallowed to access. It's a basic but essential technical SEO control for managing crawl budget and preventing low-value pages from being indexed.

Example

A robots.txt rule Disallow: /staging/ keeps crawlers out of the test environment while the live pages stay open.

Related resources

-   Glossary

    ## Crawl Budget

    Crawl budget is the number of pages a search engine will crawl on a site within a given time.

    [See term](https://yungle.ai/glossary/crawl-budget)
-   Glossary

    ## XML Sitemap

    An XML sitemap is a file listing a site's pages to help search engines find and crawl them efficiently.

    [See term](https://yungle.ai/glossary/xml-sitemap)
-   FAQ

    ## How Do I Optimize My Website for AI Search Engines Like ChatGPT?

    Write clear, self-contained answers to specific questions using structured formatting like headers and lists, build genuine topical authority around your subject area, and make sure your content is easy for AI crawlers to access and parse technically. AI tools like ChatGPT and Perplexity favor content that directly answers a question without requiring the reader to piece it together from a long article. This is the core of AEO, and it overlaps heavily with good traditional SEO practice.

    [See answer](https://yungle.ai/resources?type=faq#how-do-i-optimize-my-website-for-ai-search-engines-like-chatgpt)
-   FAQ

    ## Why Is My Website Not Ranking on Google?

    Your website likely isn't ranking due to a combination of thin or irrelevant content, weak technical SEO, slow load speed, or a lack of quality backlinks pointing to your site. Run a technical audit first — broken links, missing metadata, and slow pages are common and fixable quickly. If technical health is solid, the issue is usually content: not enough pages targeting real search intent, or content too generic to compete with more specific pages.

    [See answer](https://yungle.ai/resources?type=faq#why-is-my-website-not-ranking-on-google)

Next step

## Let's talk

_Image: Engraving of a jungle panorama with mountains_

Inspired by Tulum, QROO, MX