对官网文档做SEO

阿里云国内75折回扣微信号：monov8

阿里云国际，腾讯云国际，低至75折。AWS 93折免费开户实名账号代冲值优惠多多微信号：monov8 飞机：@monov6

对官网文档做SEO

思路将我们现有文档喂给搜索引擎

引擎蜘蛛爬虫

爬虫 User-Agent

谷歌搜索引擎蜘蛛爬虫

google 搜索引擎蜘蛛爬虫的 UA 一般为

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)、
Googlebot/2.1 (+http://www.googlebot.com/bot.html)、
Googlebot/2.1 (+http://www.google.com/bot.html)、
Googlebot-Image/1.0

其中最后一个是 google 图片搜索蜘蛛爬虫。

百度搜索引擎蜘蛛爬虫

移动 UA: Mozilla/5.0 (Linux;u;Android 4.2.2;zh-cn;) AppleWebKit/534.46 (KHTML,likeGecko) Version/5.1 Mobile Safari/10600.6.3 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
PC UA: Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html
新增渲染 UA:
- 移动 UA Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 likeMac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)
- PC UAMozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)

搜狗搜索引擎蜘蛛爬虫

搜狗搜索引擎 UA 为

Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
图片蜘蛛Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)

360搜索引擎蜘蛛爬虫

360搜索蜘蛛爬虫的 UA 为

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36; 360Spider

神马搜索引擎蜘蛛爬虫

神马 Spider 的 user-agent 为YisouSpider

Bing 搜索引擎蜘蛛爬虫

Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

Yandex 搜索引擎蜘蛛爬虫

Yandex 是俄罗斯的搜索引擎其 UA 是 Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)

UA 汇总

搜索引擎	bot	User-Agent 详细
谷歌	googlebot	Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
		Googlebot/2.1 (+http://www.googlebot.com/bot.html)
		Googlebot/2.1 (+http://www.google.com/bot.html)
		Googlebot-Image/1.0
百度	Baiduspider	移动 UA: Mozilla/5.0 (Linux;u;Android 4.2.2;zh-cn;) AppleWebKit/534.46 (KHTML,likeGecko) Version/5.1 Mobile Safari/10600.6.3 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
		PC UA: Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html
		移动 UA Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 likeMac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)
		PC UAMozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)
搜狗	sogou web spider	Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
	sogou pic spider	图片蜘蛛Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
360	360Spider	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36; 360Spider
神马	YisouSpider	YisouSpider
Bing	bingbot	Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Yandex	YandexBot	Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)

判断 UA

测试环境

机器IP xx.xx.229.139

工作目录 /opt/www/doc.rongcloud.net/xxxxxxx

Nginx 服务配置 /usr/local/openresty/nginx/conf/vhosts/doc.rongcloud.net.conf

Nginx 配置修改

        location / {
                root /opt/www/docqa.rongcloud.net/xxxxxxx;
+                default_type text/html;
+                if ( $http_user_agent ~* "(Baiduspider)|(360Spider)|(bingbot)|(YandexBot)" ){
+                        root /opt/www/doc.rongcloud.net/xxxxxxx/docs;
+                }
                try_files $uri $uri/ /index.html;
                error_page 404 /index.html;
        }
+        location ~* .*.(js|css|png|jpg|svg)$ {
+                root /opt/www/doc.rongcloud.net/xxxxxxx;
+        }

注意部署时机器上有目录与文件名冲突的。将目录删除即可。

生成SEO文件

将 md 文件转换为 html 添加默认类型
添加头尾

验证百度SEO UA 带有 Baiduspider

curl 'https://doc.rongcloud.net/livevideoroom/Android/2.X/guides/intro' \
  -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
  -H 'Accept-Language: zh-CN,zh;q=0.9' \
  -H 'Cache-Control: no-cache' \
  -H 'Connection: keep-alive' \
  -H 'Cookie: im-token-login=0; ' \
  -H 'Pragma: no-cache' \
  -H 'Sec-Fetch-Dest: document' \
  -H 'Sec-Fetch-Mode: navigate' \
  -H 'Sec-Fetch-Site: same-origin' \
  -H 'Sec-Fetch-User: ?1' \
  -H 'Upgrade-Insecure-Requests: 1' \
  -H $'User-Agent: \'Mozilla/5.0 (Linux; Android 8.0.0; SM-G955U Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Mobile Safari/537.36 Baiduspider' \
  -H 'sec-ch-ua: ""' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: ""' \
  --compressed

验证google UA 带有 googlebot

curl 'https://doc.rongcloud.net/livevideoroom/Android/2.X/guides/intro' \
  -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
  -H 'Accept-Language: zh-CN,zh;q=0.9' \
  -H 'Cache-Control: no-cache' \
  -H 'Connection: keep-alive' \
  -H 'Cookie: im-token-login=0; trajectoryId=M_G0bm3NaS8EyOqQ379wAg; ' \
  -H 'Pragma: no-cache' \
  -H 'Sec-Fetch-Dest: document' \
  -H 'Sec-Fetch-Mode: navigate' \
  -H 'Sec-Fetch-Site: same-origin' \
  -H 'Sec-Fetch-User: ?1' \
  -H 'Upgrade-Insecure-Requests: 1' \
  -H $'User-Agent: \'Mozilla/5.0 (Linux; Android 8.0.0; SM-G955U Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Mobile Safari/537.36 GOOGLEBOT' \
  -H 'sec-ch-ua: ""' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: ""' \
  --compressed