{"id":29592,"date":"2026-01-27T02:58:49","date_gmt":"2026-01-27T02:58:49","guid":{"rendered":"https:\/\/hosteko.com\/blog\/?p=29592"},"modified":"2026-01-27T02:58:49","modified_gmt":"2026-01-27T02:58:49","slug":"cara-block-bot-search-engine-melalui-robots-txt","status":"publish","type":"post","link":"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt","title":{"rendered":"Cara Block Bot Search Engine Melalui Robots.txt dengan Benar dan Aman"},"content":{"rendered":"<p data-start=\"280\" data-end=\"641\">Robots.txt adalah file penting dalam pengelolaan website yang berfungsi untuk memberikan instruksi kepada bot atau crawler mesin pencari mengenai halaman atau direktori mana yang boleh dan tidak boleh diakses. Dengan pengaturan robots.txt yang tepat, pemilik website dapat mengontrol proses crawling, melindungi area tertentu, serta mengoptimalkan performa SEO. Artikel ini akan membahas secara lengkap dan detail mengenai cara memblok bot search engine menggunakan robots.txt, mulai dari pengertian, fungsi, struktur, hingga contoh penerapannya pada website WordPress.<\/p>\n<h2 data-start=\"857\" data-end=\"879\">Apa Itu Robots.txt?<\/h2>\n<p data-start=\"881\" data-end=\"1096\">Robots.txt adalah file teks sederhana yang diletakkan di direktori root website. File ini digunakan sebagai standar komunikasi antara website dan web crawler seperti Googlebot, Bingbot, Baiduspider, dan bot lainnya. Contoh lokasi file robots.txt:<code> https:\/\/namadomain.com\/robots.txt<\/code>,\u00a0Ketika bot mengunjungi sebuah website, hal pertama yang mereka baca adalah file robots.txt untuk mengetahui aturan crawling yang berlaku.<\/p>\n<h2 data-start=\"1316\" data-end=\"1336\">Fungsi Robots.txt<\/h2>\n<p data-start=\"1338\" data-end=\"1383\">Beberapa fungsi utama robots.txt antara lain:<\/p>\n<ol>\n<li data-start=\"1388\" data-end=\"1452\">Mengatur halaman atau folder yang boleh dan tidak boleh diindeks<\/li>\n<li data-start=\"1456\" data-end=\"1505\">Mencegah crawling pada halaman yang tidak penting<\/li>\n<li data-start=\"1509\" data-end=\"1546\">Melindungi area administratif website<\/li>\n<li data-start=\"1550\" data-end=\"1611\">Menghemat crawl budget, terutama untuk website berskala besar<\/li>\n<li data-start=\"1615\" data-end=\"1660\">Mengontrol bot tertentu yang tidak diinginkan<\/li>\n<\/ol>\n<h2 data-start=\"1667\" data-end=\"1702\">Alasan Memblok Bot Search Engine<\/h2>\n<p data-start=\"1704\" data-end=\"1781\">Pemblokiran bot search engine diperlukan dalam kondisi tertentu, antara lain:<\/p>\n<p data-start=\"1783\" data-end=\"1817\"><strong>1. Melindungi Halaman Sensitif<\/strong><\/p>\n<p data-start=\"1818\" data-end=\"1917\">Halaman seperti login, dashboard admin, atau sistem internal tidak perlu muncul di hasil pencarian.<\/p>\n<p data-start=\"1919\" data-end=\"1953\"><strong>2. Menghindari Konten Duplikat<\/strong><\/p>\n<p data-start=\"1954\" data-end=\"2051\">Beberapa halaman dapat menghasilkan konten yang sama sehingga berpotensi menurunkan kualitas SEO.<\/p>\n<p data-start=\"2053\" data-end=\"2082\"><strong>3. Menghemat Crawl Budget<\/strong><\/p>\n<p data-start=\"2083\" data-end=\"2187\">Dengan membatasi bot hanya ke halaman penting, mesin pencari dapat lebih efisien dalam merayapi website.<\/p>\n<p data-start=\"2189\" data-end=\"2233\"><strong>4. Menghindari Bot yang Tidak Diinginkan<\/strong><\/p>\n<p data-start=\"2234\" data-end=\"2338\">Tidak semua bot memiliki tujuan yang baik. Beberapa bot hanya membebani server tanpa memberikan manfaat.<\/p>\n<h2 data-start=\"2345\" data-end=\"2402\">Hal yang Perlu Dipahami Sebelum Menggunakan Robots.txt<\/h2>\n<p data-start=\"2404\" data-end=\"2692\">Perlu diketahui bahwa robots.txt bersifat instruksi, bukan sistem keamanan. Bot yang tidak mematuhi standar dapat mengabaikan aturan ini. Oleh karena itu, untuk data yang benar-benar sensitif, disarankan menggunakan pengamanan tambahan seperti autentikasi, proteksi server, atau firewall. Selain itu, robots.txt tidak secara otomatis menghapus halaman dari indeks mesin pencari jika halaman tersebut sudah terindeks sebelumnya.<\/p>\n<h2 data-start=\"2839\" data-end=\"2867\">Struktur Dasar Robots.txt<\/h2>\n<p data-start=\"2869\" data-end=\"2932\">Struktur dasar robots.txt terdiri dari beberapa direktif utama:<\/p>\n<div class=\"contain-inline-size rounded-2xl corner-superellipse\/1.1 relative bg-token-sidebar-surface-primary\">\n<div class=\"overflow-y-auto p-4\" dir=\"ltr\"><code class=\"whitespace-pre!\"><span class=\"hljs-section\">User-agent: nama_bot<\/span><br \/>\n<span class=\"hljs-section\">Disallow: \/path\/<\/span><br \/>\n<span class=\"hljs-section\">Allow: \/path\/<\/span><br \/>\n<span class=\"hljs-section\">Sitemap: URL_sitemap<\/span><br \/>\n<\/code><\/div>\n<\/div>\n<p data-start=\"3016\" data-end=\"3027\">Penjelasan:<\/p>\n<ul>\n<li data-start=\"3030\" data-end=\"3072\">User-agent menentukan bot yang ditargetkan<\/li>\n<li data-start=\"3075\" data-end=\"3114\">Disallow berfungsi untuk melarang akses<\/li>\n<li data-start=\"3117\" data-end=\"3162\">Allow digunakan untuk memberikan pengecualian<\/li>\n<li data-start=\"3165\" data-end=\"3212\">Sitemap membantu bot menemukan struktur website<\/li>\n<\/ul>\n<h2 data-start=\"3219\" data-end=\"3245\">Contoh Robots.txt Dasar<\/h2>\n<p data-start=\"3247\" data-end=\"3272\"><strong><em>1. Mengizinkan Semua Bot<\/em><\/strong><\/p>\n<div class=\"contain-inline-size rounded-2xl corner-superellipse\/1.1 relative bg-token-sidebar-surface-primary\">\n<div class=\"overflow-y-auto p-4\" dir=\"ltr\"><code class=\"whitespace-pre!\"><span class=\"hljs-section\">User-agent: *<\/span><br \/>\n<span class=\"hljs-section\">Disallow:<\/span><br \/>\n<\/code><\/div>\n<\/div>\n<p data-start=\"3306\" data-end=\"3329\"><strong><em>2. Memblokir Semua Bot<\/em><\/strong><\/p>\n<div class=\"contain-inline-size rounded-2xl corner-superellipse\/1.1 relative bg-token-sidebar-surface-primary\">\n<div class=\"overflow-y-auto p-4\" dir=\"ltr\"><code class=\"whitespace-pre!\"><span class=\"hljs-section\">User-agent: *<\/span><br \/>\n<span class=\"hljs-section\">Disallow: \/<\/span><br \/>\n<\/code><\/div>\n<\/div>\n<h2 data-start=\"3370\" data-end=\"3410\">Cara Block Bot Search Engine Tertentu<\/h2>\n<p data-start=\"3412\" data-end=\"3483\">Jika ingin memblokir bot tertentu saja, gunakan user-agent yang sesuai.<\/p>\n<p data-start=\"3485\" data-end=\"3523\">Contoh memblokir beberapa bot populer:<\/p>\n<div class=\"contain-inline-size rounded-2xl corner-superellipse\/1.1 relative bg-token-sidebar-surface-primary\">\n<div class=\"overflow-y-auto p-4\" dir=\"ltr\"><code class=\"whitespace-pre!\"><span class=\"hljs-section\">User-agent: Googlebot<\/span><br \/>\n<span class=\"hljs-section\">Disallow: \/<\/span><\/p>\n<p><span class=\"hljs-section\">User-agent: Bingbot<\/span><br \/>\n<span class=\"hljs-section\">Disallow: \/<\/span><\/p>\n<p><span class=\"hljs-section\">User-agent: Slurp<\/span><br \/>\n<span class=\"hljs-section\">Disallow: \/<\/span><\/p>\n<p><span class=\"hljs-section\">User-agent: DuckDuckGo<\/span><br \/>\n<span class=\"hljs-section\">Disallow: \/<\/span><\/p>\n<p><span class=\"hljs-section\">User-agent: Baiduspider<\/span><br \/>\n<span class=\"hljs-section\">Disallow: \/<\/span><br \/>\n<\/code><\/div>\n<\/div>\n<p data-start=\"3704\" data-end=\"3805\">Pastikan penulisan user-agent sesuai dengan nama resmi bot agar aturan dapat diterapkan dengan benar.<\/p>\n<h2 data-start=\"3812\" data-end=\"3855\">Memblokir Semua Bot Kecuali Bot Tertentu<\/h2>\n<p data-start=\"3857\" data-end=\"3958\">Jika hanya ingin mengizinkan bot tertentu seperti Googlebot dan Bingbot, gunakan konfigurasi berikut:<\/p>\n<div class=\"contain-inline-size rounded-2xl corner-superellipse\/1.1 relative bg-token-sidebar-surface-primary\">\n<div class=\"overflow-y-auto p-4\" dir=\"ltr\"><code class=\"whitespace-pre!\"><span class=\"hljs-section\">User-agent: Googlebot<\/span><br \/>\n<span class=\"hljs-section\">Disallow:<\/span><\/p>\n<p><span class=\"hljs-section\">User-agent: Bingbot<\/span><br \/>\n<span class=\"hljs-section\">Disallow:<\/span><\/p>\n<p><span class=\"hljs-section\">User-agent: *<\/span><br \/>\n<span class=\"hljs-section\">Disallow: \/<\/span><br \/>\n<\/code><\/div>\n<\/div>\n<p data-start=\"4059\" data-end=\"4154\">Konfigurasi ini memungkinkan Google dan Bing mengakses website, sementara bot lainnya diblokir.<\/p>\n<h2 data-start=\"4161\" data-end=\"4199\">Memblokir Folder atau File Tertentu<\/h2>\n<p data-start=\"4201\" data-end=\"4277\">Untuk memblokir folder atau file tertentu, misalnya halaman admin WordPress:<\/p>\n<div class=\"contain-inline-size rounded-2xl corner-superellipse\/1.1 relative bg-token-sidebar-surface-primary\">\n<div class=\"overflow-y-auto p-4\" dir=\"ltr\"><code class=\"whitespace-pre!\"><span class=\"hljs-section\">User-agent: *<\/span><br \/>\n<span class=\"hljs-section\">Disallow: \/wp-admin\/<\/span><br \/>\n<span class=\"hljs-section\">Disallow: \/wp-login.php<\/span><br \/>\n<\/code><\/div>\n<\/div>\n<p data-start=\"4347\" data-end=\"4425\">Jika diperlukan, izinkan file tertentu agar WordPress tetap berfungsi optimal:<\/p>\n<div class=\"contain-inline-size rounded-2xl corner-superellipse\/1.1 relative bg-token-sidebar-surface-primary\">\n<div class=\"overflow-y-auto p-4\" dir=\"ltr\"><code class=\"whitespace-pre!\">Allow: \/wp-<span class=\"hljs-keyword\">admin<\/span>\/<span class=\"hljs-keyword\">admin<\/span>-ajax.php<br \/>\n<\/code><\/div>\n<\/div>\n<h2 data-start=\"4473\" data-end=\"4517\">Contoh Robots.txt Lengkap untuk WordPress<\/h2>\n<p data-start=\"4519\" data-end=\"4588\">Berikut contoh robots.txt yang umum digunakan pada website WordPress:<\/p>\n<div class=\"contain-inline-size rounded-2xl corner-superellipse\/1.1 relative bg-token-sidebar-surface-primary\">\n<div class=\"overflow-y-auto p-4\" dir=\"ltr\"><code class=\"whitespace-pre!\"><span class=\"hljs-section\">User-agent: *<\/span><br \/>\n<span class=\"hljs-section\">Disallow: \/wp-admin\/<\/span><br \/>\n<span class=\"hljs-section\">Disallow: \/wp-includes\/<\/span><br \/>\n<span class=\"hljs-section\">Disallow: \/wp-login.php<\/span><br \/>\n<span class=\"hljs-section\">Disallow: \/xmlrpc.php<\/span><br \/>\n<span class=\"hljs-section\">Disallow: \/cgi-bin\/<\/span><\/p>\n<p><span class=\"hljs-section\">Allow: \/wp-admin\/admin-ajax.php<\/span><\/p>\n<p><span class=\"hljs-section\">Sitemap: https:\/\/namadomain.com\/sitemap.xml<\/span><br \/>\n<\/code><\/div>\n<\/div>\n<p data-start=\"4802\" data-end=\"4910\">Konfigurasi ini membantu menjaga keamanan dan performa website sekaligus tetap ramah terhadap mesin pencari.<\/p>\n<h2 data-start=\"4917\" data-end=\"4956\">Cara Mengecek dan Menguji Robots.txt<\/h2>\n<p data-start=\"4958\" data-end=\"5023\">Beberapa cara untuk memastikan robots.txt berfungsi dengan benar:<\/p>\n<ol>\n<li data-start=\"5028\" data-end=\"5074\">Akses langsung file robots.txt melalui browser<\/li>\n<li data-start=\"5078\" data-end=\"5139\">Gunakan Google Search Console untuk menguji aturan robots.txt<\/li>\n<li data-start=\"5143\" data-end=\"5193\">Periksa apakah halaman penting masih bisa diindeks<\/li>\n<\/ol>\n<h2 data-start=\"5200\" data-end=\"5245\">Kesalahan Umum dalam Penggunaan Robots.txt<\/h2>\n<p data-start=\"5247\" data-end=\"5298\">Beberapa kesalahan yang sering terjadi antara lain:<\/p>\n<ul>\n<li data-start=\"5302\" data-end=\"5328\">Salah penulisan user-agent<\/li>\n<li data-start=\"5331\" data-end=\"5370\">Memblokir seluruh website tanpa sengaja<\/li>\n<li data-start=\"5373\" data-end=\"5419\">Meletakkan file robots.txt di luar root domain<\/li>\n<li data-start=\"5422\" data-end=\"5447\">Tidak menyertakan sitemap<\/li>\n<li data-start=\"5450\" data-end=\"5493\">Tidak melakukan pengujian setelah perubahan<\/li>\n<\/ul>\n<p data-start=\"5495\" data-end=\"5592\">Kesalahan-kesalahan ini dapat menyebabkan website tidak terindeks dengan baik oleh mesin pencari.<\/p>\n<h2 data-start=\"5599\" data-end=\"5609\">Penutup<\/h2>\n<p data-start=\"5611\" data-end=\"5896\">Robots.txt adalah alat penting dalam pengelolaan website, terutama untuk mengontrol aktivitas bot search engine. Dengan pemahaman dan penerapan yang tepat, file ini dapat membantu meningkatkan efektivitas SEO, menjaga keamanan, serta mengoptimalkan performa website secara keseluruhan. Pastikan setiap perubahan pada robots.txt dilakukan dengan hati-hati dan selalu diuji agar tidak berdampak negatif pada visibilitas website di mesin pencari.<\/p>\n\n\n<div class=\"kk-star-ratings kksr-auto kksr-align-right kksr-valign-bottom\"\n    data-payload='{&quot;align&quot;:&quot;right&quot;,&quot;id&quot;:&quot;29592&quot;,&quot;slug&quot;:&quot;default&quot;,&quot;valign&quot;:&quot;bottom&quot;,&quot;ignore&quot;:&quot;&quot;,&quot;reference&quot;:&quot;auto&quot;,&quot;class&quot;:&quot;&quot;,&quot;count&quot;:&quot;1&quot;,&quot;legendonly&quot;:&quot;&quot;,&quot;readonly&quot;:&quot;&quot;,&quot;score&quot;:&quot;5&quot;,&quot;starsonly&quot;:&quot;&quot;,&quot;best&quot;:&quot;5&quot;,&quot;gap&quot;:&quot;0&quot;,&quot;greet&quot;:&quot;Jadilah yang pertama untuk memberi nilai&quot;,&quot;legend&quot;:&quot;5\\\/5 - (1 vote)&quot;,&quot;size&quot;:&quot;22&quot;,&quot;title&quot;:&quot;Cara Block Bot Search Engine Melalui Robots.txt dengan Benar dan Aman&quot;,&quot;width&quot;:&quot;110&quot;,&quot;_legend&quot;:&quot;{score}\\\/{best} - ({count} {votes})&quot;,&quot;font_factor&quot;:&quot;1.25&quot;}'>\n            \n<div class=\"kksr-stars\">\n    \n<div class=\"kksr-stars-inactive\">\n            <div class=\"kksr-star\" data-star=\"1\" style=\"padding-right: 0px\">\n            \n\n<div class=\"kksr-icon\" style=\"width: 22px; height: 22px;\"><\/div>\n        <\/div>\n            <div class=\"kksr-star\" data-star=\"2\" style=\"padding-right: 0px\">\n            \n\n<div class=\"kksr-icon\" style=\"width: 22px; height: 22px;\"><\/div>\n        <\/div>\n            <div class=\"kksr-star\" data-star=\"3\" style=\"padding-right: 0px\">\n            \n\n<div class=\"kksr-icon\" style=\"width: 22px; height: 22px;\"><\/div>\n        <\/div>\n            <div class=\"kksr-star\" data-star=\"4\" style=\"padding-right: 0px\">\n            \n\n<div class=\"kksr-icon\" style=\"width: 22px; height: 22px;\"><\/div>\n        <\/div>\n            <div class=\"kksr-star\" data-star=\"5\" style=\"padding-right: 0px\">\n            \n\n<div class=\"kksr-icon\" style=\"width: 22px; height: 22px;\"><\/div>\n        <\/div>\n    <\/div>\n    \n<div class=\"kksr-stars-active\" style=\"width: 110px;\">\n            <div class=\"kksr-star\" style=\"padding-right: 0px\">\n            \n\n<div class=\"kksr-icon\" style=\"width: 22px; height: 22px;\"><\/div>\n        <\/div>\n            <div class=\"kksr-star\" style=\"padding-right: 0px\">\n            \n\n<div class=\"kksr-icon\" style=\"width: 22px; height: 22px;\"><\/div>\n        <\/div>\n            <div class=\"kksr-star\" style=\"padding-right: 0px\">\n            \n\n<div class=\"kksr-icon\" style=\"width: 22px; height: 22px;\"><\/div>\n        <\/div>\n            <div class=\"kksr-star\" style=\"padding-right: 0px\">\n            \n\n<div class=\"kksr-icon\" style=\"width: 22px; height: 22px;\"><\/div>\n        <\/div>\n            <div class=\"kksr-star\" style=\"padding-right: 0px\">\n            \n\n<div class=\"kksr-icon\" style=\"width: 22px; height: 22px;\"><\/div>\n        <\/div>\n    <\/div>\n<\/div>\n                \n\n<div class=\"kksr-legend\" style=\"font-size: 17.6px;\">\n            5\/5 - (1 vote)    <\/div>\n    <\/div>\n","protected":false},"excerpt":{"rendered":"<p>Robots.txt adalah file penting dalam pengelolaan website yang berfungsi untuk memberikan instruksi kepada bot atau crawler mesin pencari mengenai halaman atau direktori mana yang boleh dan tidak boleh diakses. Dengan pengaturan robots.txt yang tepat, pemilik website dapat mengontrol proses crawling, melindungi area tertentu, serta mengoptimalkan performa SEO. Artikel ini akan membahas secara lengkap dan detail [&hellip;]<\/p>\n","protected":false},"author":13,"featured_media":29593,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rop_custom_images_group":[],"rop_custom_messages_group":[],"rop_publish_now":"no","rop_publish_now_accounts":{"twitter_2392824914_2392824914":""},"rop_publish_now_history":[{"account":"twitter_2392824914_2392824914","service":"twitter","timestamp":1769482749,"status":"error"}],"rop_publish_now_status":"done","_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"enabled":false},"version":2}},"categories":[3],"tags":[14891,14685,14892,14890,14893],"class_list":["post-29592","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog","tag-alasan-memblok-bot-search-engine","tag-fungsi-robots-txt","tag-hal-yang-perlu-dipahami-sebelum-menggunakan-robots-txt","tag-robots-txt","tag-struktur-dasar-robots-txt"],"featured_image_src":{"landsacpe":["https:\/\/hosteko.com\/htk-blog\/wp-content\/uploads\/2026\/01\/Tambahkan-judul1-1140x445.png",1140,445,true],"list":["https:\/\/hosteko.com\/htk-blog\/wp-content\/uploads\/2026\/01\/Tambahkan-judul1-463x348.png",463,348,true],"medium":["https:\/\/hosteko.com\/htk-blog\/wp-content\/uploads\/2026\/01\/Tambahkan-judul1-300x169.png",300,169,true],"full":["https:\/\/hosteko.com\/htk-blog\/wp-content\/uploads\/2026\/01\/Tambahkan-judul1.png",1366,768,false]},"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v25.8 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Cara Block Bot Search Engine Melalui Robots.txt dengan Benar dan Aman - Hosteko Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Cara Block Bot Search Engine Melalui Robots.txt dengan Benar dan Aman - Hosteko Blog\" \/>\n<meta property=\"og:description\" content=\"Robots.txt adalah file penting dalam pengelolaan website yang berfungsi untuk memberikan instruksi kepada bot atau crawler mesin pencari mengenai halaman atau direktori mana yang boleh dan tidak boleh diakses. Dengan pengaturan robots.txt yang tepat, pemilik website dapat mengontrol proses crawling, melindungi area tertentu, serta mengoptimalkan performa SEO. Artikel ini akan membahas secara lengkap dan detail [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt\" \/>\n<meta property=\"og:site_name\" content=\"Hosteko Blog\" \/>\n<meta property=\"article:published_time\" content=\"2026-01-27T02:58:49+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/hosteko.com\/htk-blog\/wp-content\/uploads\/2026\/01\/Tambahkan-judul1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1366\" \/>\n\t<meta property=\"og:image:height\" content=\"768\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Fitri Ana\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Fitri Ana\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt#article\",\"isPartOf\":{\"@id\":\"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt\"},\"author\":{\"name\":\"Fitri Ana\",\"@id\":\"https:\/\/hosteko.com\/blog\/#\/schema\/person\/ffcd8071a8a3d6a862a4e1381d1c4ea0\"},\"headline\":\"Cara Block Bot Search Engine Melalui Robots.txt dengan Benar dan Aman\",\"datePublished\":\"2026-01-27T02:58:49+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt\"},\"wordCount\":654,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/hosteko.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt#primaryimage\"},\"thumbnailUrl\":\"https:\/\/hosteko.com\/htk-blog\/wp-content\/uploads\/2026\/01\/Tambahkan-judul1.png\",\"keywords\":[\"Alasan Memblok Bot Search Engine\",\"fungsi robots.txt\",\"Hal yang Perlu Dipahami Sebelum Menggunakan Robots.txt\",\"Robots.txt\",\"Struktur Dasar Robots.txt\"],\"articleSection\":[\"Blog\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt\",\"url\":\"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt\",\"name\":\"Cara Block Bot Search Engine Melalui Robots.txt dengan Benar dan Aman - Hosteko Blog\",\"isPartOf\":{\"@id\":\"https:\/\/hosteko.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt#primaryimage\"},\"image\":{\"@id\":\"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt#primaryimage\"},\"thumbnailUrl\":\"https:\/\/hosteko.com\/htk-blog\/wp-content\/uploads\/2026\/01\/Tambahkan-judul1.png\",\"datePublished\":\"2026-01-27T02:58:49+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt#primaryimage\",\"url\":\"https:\/\/hosteko.com\/htk-blog\/wp-content\/uploads\/2026\/01\/Tambahkan-judul1.png\",\"contentUrl\":\"https:\/\/hosteko.com\/htk-blog\/wp-content\/uploads\/2026\/01\/Tambahkan-judul1.png\",\"width\":1366,\"height\":768},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/hosteko.com\/blog\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Cara Block Bot Search Engine Melalui Robots.txt dengan Benar dan Aman\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/hosteko.com\/blog\/#website\",\"url\":\"https:\/\/hosteko.com\/blog\/\",\"name\":\"Hosteko Blog\",\"description\":\"Berita &amp; Informasi Dunia IT\",\"publisher\":{\"@id\":\"https:\/\/hosteko.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/hosteko.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/hosteko.com\/blog\/#organization\",\"name\":\"HOSTEKO\",\"url\":\"https:\/\/hosteko.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/hosteko.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/hosteko.com\/htk-blog\/wp-content\/uploads\/2019\/04\/logo-hosteko.png\",\"contentUrl\":\"https:\/\/hosteko.com\/htk-blog\/wp-content\/uploads\/2019\/04\/logo-hosteko.png\",\"width\":195,\"height\":57,\"caption\":\"HOSTEKO\"},\"image\":{\"@id\":\"https:\/\/hosteko.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/hosteko.com\/blog\/#\/schema\/person\/ffcd8071a8a3d6a862a4e1381d1c4ea0\",\"name\":\"Fitri Ana\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/hosteko.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/a099f723518ab9b89aa80b7ed8cadd2d3fae127c5bd735733cf24a661e75a882?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/a099f723518ab9b89aa80b7ed8cadd2d3fae127c5bd735733cf24a661e75a882?s=96&d=mm&r=g\",\"caption\":\"Fitri Ana\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Cara Block Bot Search Engine Melalui Robots.txt dengan Benar dan Aman - Hosteko Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt","og_locale":"en_US","og_type":"article","og_title":"Cara Block Bot Search Engine Melalui Robots.txt dengan Benar dan Aman - Hosteko Blog","og_description":"Robots.txt adalah file penting dalam pengelolaan website yang berfungsi untuk memberikan instruksi kepada bot atau crawler mesin pencari mengenai halaman atau direktori mana yang boleh dan tidak boleh diakses. Dengan pengaturan robots.txt yang tepat, pemilik website dapat mengontrol proses crawling, melindungi area tertentu, serta mengoptimalkan performa SEO. Artikel ini akan membahas secara lengkap dan detail [&hellip;]","og_url":"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt","og_site_name":"Hosteko Blog","article_published_time":"2026-01-27T02:58:49+00:00","og_image":[{"width":1366,"height":768,"url":"https:\/\/hosteko.com\/htk-blog\/wp-content\/uploads\/2026\/01\/Tambahkan-judul1.png","type":"image\/png"}],"author":"Fitri Ana","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Fitri Ana","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt#article","isPartOf":{"@id":"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt"},"author":{"name":"Fitri Ana","@id":"https:\/\/hosteko.com\/blog\/#\/schema\/person\/ffcd8071a8a3d6a862a4e1381d1c4ea0"},"headline":"Cara Block Bot Search Engine Melalui Robots.txt dengan Benar dan Aman","datePublished":"2026-01-27T02:58:49+00:00","mainEntityOfPage":{"@id":"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt"},"wordCount":654,"commentCount":0,"publisher":{"@id":"https:\/\/hosteko.com\/blog\/#organization"},"image":{"@id":"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt#primaryimage"},"thumbnailUrl":"https:\/\/hosteko.com\/htk-blog\/wp-content\/uploads\/2026\/01\/Tambahkan-judul1.png","keywords":["Alasan Memblok Bot Search Engine","fungsi robots.txt","Hal yang Perlu Dipahami Sebelum Menggunakan Robots.txt","Robots.txt","Struktur Dasar Robots.txt"],"articleSection":["Blog"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt#respond"]}]},{"@type":"WebPage","@id":"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt","url":"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt","name":"Cara Block Bot Search Engine Melalui Robots.txt dengan Benar dan Aman - Hosteko Blog","isPartOf":{"@id":"https:\/\/hosteko.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt#primaryimage"},"image":{"@id":"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt#primaryimage"},"thumbnailUrl":"https:\/\/hosteko.com\/htk-blog\/wp-content\/uploads\/2026\/01\/Tambahkan-judul1.png","datePublished":"2026-01-27T02:58:49+00:00","breadcrumb":{"@id":"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt#primaryimage","url":"https:\/\/hosteko.com\/htk-blog\/wp-content\/uploads\/2026\/01\/Tambahkan-judul1.png","contentUrl":"https:\/\/hosteko.com\/htk-blog\/wp-content\/uploads\/2026\/01\/Tambahkan-judul1.png","width":1366,"height":768},{"@type":"BreadcrumbList","@id":"https:\/\/hosteko.com\/blog\/cara-block-bot-search-engine-melalui-robots-txt#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/hosteko.com\/blog"},{"@type":"ListItem","position":2,"name":"Cara Block Bot Search Engine Melalui Robots.txt dengan Benar dan Aman"}]},{"@type":"WebSite","@id":"https:\/\/hosteko.com\/blog\/#website","url":"https:\/\/hosteko.com\/blog\/","name":"Hosteko Blog","description":"Berita &amp; Informasi Dunia IT","publisher":{"@id":"https:\/\/hosteko.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/hosteko.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/hosteko.com\/blog\/#organization","name":"HOSTEKO","url":"https:\/\/hosteko.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/hosteko.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/hosteko.com\/htk-blog\/wp-content\/uploads\/2019\/04\/logo-hosteko.png","contentUrl":"https:\/\/hosteko.com\/htk-blog\/wp-content\/uploads\/2019\/04\/logo-hosteko.png","width":195,"height":57,"caption":"HOSTEKO"},"image":{"@id":"https:\/\/hosteko.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/hosteko.com\/blog\/#\/schema\/person\/ffcd8071a8a3d6a862a4e1381d1c4ea0","name":"Fitri Ana","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/hosteko.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/a099f723518ab9b89aa80b7ed8cadd2d3fae127c5bd735733cf24a661e75a882?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a099f723518ab9b89aa80b7ed8cadd2d3fae127c5bd735733cf24a661e75a882?s=96&d=mm&r=g","caption":"Fitri Ana"}}]}},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/hosteko.com\/htk-blog\/wp-content\/uploads\/2026\/01\/Tambahkan-judul1.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/hosteko.com\/blog\/wp-json\/wp\/v2\/posts\/29592","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/hosteko.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hosteko.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hosteko.com\/blog\/wp-json\/wp\/v2\/users\/13"}],"replies":[{"embeddable":true,"href":"https:\/\/hosteko.com\/blog\/wp-json\/wp\/v2\/comments?post=29592"}],"version-history":[{"count":1,"href":"https:\/\/hosteko.com\/blog\/wp-json\/wp\/v2\/posts\/29592\/revisions"}],"predecessor-version":[{"id":29595,"href":"https:\/\/hosteko.com\/blog\/wp-json\/wp\/v2\/posts\/29592\/revisions\/29595"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/hosteko.com\/blog\/wp-json\/wp\/v2\/media\/29593"}],"wp:attachment":[{"href":"https:\/\/hosteko.com\/blog\/wp-json\/wp\/v2\/media?parent=29592"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hosteko.com\/blog\/wp-json\/wp\/v2\/categories?post=29592"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hosteko.com\/blog\/wp-json\/wp\/v2\/tags?post=29592"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}