{"id":1223,"date":"2024-08-02T11:06:04","date_gmt":"2024-08-02T03:06:04","guid":{"rendered":"http:\/\/blue.yn.cn\/?p=1223"},"modified":"2024-08-12T22:26:49","modified_gmt":"2024-08-12T14:26:49","slug":"logstash%e6%97%a5%e5%bf%97%e6%94%b6%e9%9b%86%e4%b8%8e%e5%88%86%e6%9e%90%e5%ae%9e%e8%b7%b5","status":"publish","type":"post","link":"http:\/\/blue.yn.cn\/?p=1223","title":{"rendered":"logstash\u65e5\u5fd7\u6536\u96c6\u4e0e\u5206\u6790\u5b9e\u8df5"},"content":{"rendered":"

logstash\u67b6\u6784\u4ecb\u7ecd<\/h3>\n

\u4e3a\u4ec0\u4e48\u9700\u8981logstash<\/h4>\n

\u5bf9\u4e8e\u90e8\u5206\u751f\u4ea7\u4e0a\u7684\u65e5\u5fd7\u65e0\u6cd5\u50cfnginx<\/code>\u90a3\u6837\uff0c\u53ef\u4ee5\u76f4\u63a5\u5c06\u8f93\u51fa\u7684\u65e5\u5fd7\u8f6c\u4e3ajson<\/code>\u683c\u5f0f\uff0c\u4f46\u662f\u53ef\u4ee5\u501f\u52a9logstash<\/code>\u6765\u5c06\u6211\u4eec\u7684\u975e\u7ed3\u6784\u5316\u6570\u636e<\/em>\u8f6c\u4e3a\u7ed3\u6784\u5316\u6570\u636e<\/em><\/p>\n

filebeat<\/code>-->logstash<\/code>-->es<\/code>
\nlogstash(input fileter ouput)--es<\/code><\/p>\n

\u4ec0\u4e48\u662flogstash<\/h4>\n

logstash<\/code>\u662f\u5f00\u6e90\u7684\u6570\u636e\u5904\u7406\u7ba1\u9053\uff0c\u80fd\u591f\u540c\u65f6\u4ece\u591a\u4e2a\u6e90\u91c7\u96c6\u6570\u636e\uff0c\u8f6c\u6362\u6570\u636e\uff0c\u7136\u540e\u8f93\u51fa\u6570\u636e<\/p>\n

logstash\u67b6\u6784\u4ecb\u7ecd<\/h4>\n
    \n
  • logstash<\/code>\u7684\u57fa\u7840\u67b6\u6784\u7c7b\u4f3c\u4e8epipeline<\/code>\u6d41\u6c34\u7ebf\uff0c\u8fc7\u4e0b\u6240\u793a\uff1a\n
      \n
    • input<\/code>\uff1a\u6570\u636e\u91c7\u96c6\uff08\u5e38\u7528\u63d2\u4ef6\uff1astdin\u3001file\u3001kafka\u3001filebeat\u3001http<\/code>\uff09<\/li>\n
    • filter<\/code>\uff1a\u6570\u636e\u89e3\u6790\/\u8f6c\u6362\uff08\u5e38\u7528\u63d2\u4ef6\uff1agrok\u3001date\u3001geoip\u3001mutate\u3001useragent<\/code>\uff09<\/li>\n
    • output<\/code>\uff1a\u6570\u636e\u8f93\u51fa\uff08\u5e38\u7528\u63d2\u4ef6\uff1aelasticresearch<\/code>\uff09<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n

      \u5b89\u88c5logstash<\/h4>\n
      yum install java -y\nrpm -ivh logstash-7.8.1.rpm\n\ngrep "^[a-Z]" \/etc\/logstash\/logstash.yml\nnode.name: logstash-node1\npath.data: \/var\/lib\/logstash\npipeline.workers: 4  # \u548ccpu\u6838\u5fc3\u4e00\u81f4\npipeline.batch.size: 1000\npipeline.ordered: auto\npath.logs: \/var\/log\/logstash\n\nvim \/etc\/logstash\/jvm.options\n-Xms8g  # \u662f\u5185\u5b58\u7684\u4e00\u534a\n-Xmx8g<\/code><\/pre>\n
      logstash input\u63d2\u4ef6<\/h3>\n
      input<\/code>\u63d2\u4ef6\u7528\u4e8e\u6307\u5b9a\u8f93\u5165\u6e90\uff0c\u4e00\u4e2apipeline<\/code>\u53ef\u4ee5\u6709\u591a\u4e2ainput<\/code>\u63d2\u4ef6\uff0c\u6211\u4eec\u4e3b\u8981\u56f4\u7ed5\u4e0b\u9762\u51e0\u4e2ainput<\/code>\u63d2\u4ef6\u8fdb\u884c\u4ecb\u7ecd\uff1a<\/p>\n
        \n
      • stdin<\/code><\/li>\n
      • file<\/code><\/li>\n
      • filebeat<\/code><\/li>\n
      • kafka<\/code><\/li>\n
      • http<\/code><\/li>\n<\/ul>\n
        stdin\u63d2\u4ef6<\/h4>\n
          \n
        • \u4ece\u6807\u51c6\u8f93\u5165\u8bfb\u53d6\u6570\u636e\uff0c\u4ece\u6807\u51c6\u8f93\u51fa\u4e2d\u8f93\u51fa\u5185\u5bb9<\/li>\n<\/ul>\n
          cat \/etc\/logstash\/conf.d\/stdin.logstash.conf\n\ninput {\n        stdin {\n                type => "stdin"\n                tags => "stdin_type"\n        }\n}\n\noutput {\n        stdout{\n                codec => "rubydebug"\n        }\n}<\/code><\/pre>\n
            \n
          • \u6267\u884c\u6d4b\u8bd5<\/li>\n<\/ul>\n
            \/usr\/share\/logstash\/bin\/logstash -f \/etc\/logstash\/conf.d\/stdin.logstash.conf<\/code><\/p>\n
              \n
            • \u8fd4\u56de\u7ed3\u679c<\/li>\n<\/ul>\n
              hello saturday\n{\n          "type" => "stdin",\n      "@version" => "1",\n          "host" => "logstash",\n       "message" => "hello saturday",\n    "@timestamp" => 2024-08-02T14:08:32.334Z,\n          "tags" => [\n        [0] "stdin_type"\n    ]\n}<\/code><\/pre>\n
              file\u63d2\u4ef6<\/h4>\n
                \n
              • \u4ecefile<\/code>\u6587\u4ef6\u4e2d\u8bfb\u53d6\u6570\u636e\uff0c\u4ece\u6807\u51c6\u8f93\u51fa\u4e2d\u8f93\u51fa\u5185\u5bb9<\/li>\n<\/ul>\n
                cat \/etc\/logstash\/conf.d\/file.logstash.conf\n\ninput {\n        file {\n                path => "\/var\/log\/test.log"\n                type => syslog\n                start_position => "beginning"  # \u7b2c\u4e00\u6b21\u4ece\u5934\u5f00\u59cb\u8bfb\u53d6\u6587\u4ef6\n                stat_interval => "3"  # \u5b9a\u65f6\u68c0\u67e5\u6587\u4ef6\u662f\u5426\u66f4\u65b0\uff0c\u9ed8\u8ba41s\n        }\n}\n\noutput {\n        stdout{\n                codec => "rubydebug"\n        }\n}<\/code><\/pre>\n
                  \n
                • \n
                  \u6267\u884c\u6d4b\u8bd5
                  \n\/usr\/share\/logstash\/bin\/logstash -f \/etc\/logstash\/conf.d\/file.logstash.conf<\/code><\/p>\n<\/li>\n
                • \n
                  \u8fd4\u56de\u7ed3\u679c<\/p>\n<\/li>\n<\/ul>\n
                  {\n          "host" => "logstash",\n       "message" => "Aug  3 09:10:50 logstash systemd: Started Session 86 of user root.",\n          "type" => "syslog",\n      "@version" => "1",\n    "@timestamp" => 2024-08-03T01:19:28.681Z,\n          "path" => "\/var\/log\/test.log"\n}<\/code><\/pre>\n
                  logstash filter\u63d2\u4ef6<\/h3>\n
                  \u6570\u636e\u4ece\u6e90\u4f20\u8f93\u5230\u5b58\u50a8\u7684\u8fc7\u7a0b\u4e2d\uff0clogstash<\/code>\u7684filter<\/code>\u8fc7\u6ee4\u5668\u80fd\u591f\u89e3\u6790\u5404\u4e2a\u4e8b\u4ef6\uff0c\u8bc6\u522b\u5df2\u547d\u540d\u7684\u5b57\u6bb5\u7ed3\u6784\uff0c\u5e76\u5c06\u5b83\u4eec\u8f6c\u6362\u6210\u901a\u7528\u683c\u5f0f\uff0c\u4ee5\u4fbf\u66f4\u8f7b\u677e\u3001\u66f4\u5feb\u901f\u5730\u5206\u6790\u548c\u5b9e\u73b0\u5546\u4e1a\u4ef7\u503c\uff1b<\/p>\n
                    \n
                  • \u5229\u7528grok<\/code>\u4ece\u975e\u7ed3\u6784\u5316\u6570\u636e\u4e2d\u6d3e\u751f\u51fa\u7ed3\u6784<\/li>\n
                  • \u5229\u7528geoip<\/code>\u4eceip<\/code>\u5730\u5740\u5206\u6790\u51fa\u5730\u7406\u5750\u6807<\/li>\n
                  • \u5229\u7528useragent<\/code>\u4ece\u8bf7\u6c42\u4e2d\u5206\u6790\u64cd\u4f5c\u7cfb\u7edf\u3001\u8bbe\u5907\u7c7b\u578b<\/li>\n<\/ul>\n
                    \"file\"<\/p>\n
                    grok\u63d2\u4ef6<\/h4>\n
                      \n
                    • \u6211\u4eec\u5e0c\u671b\u5c06\u5982\u4e0b\u975e\u7ed3\u6784\u5316\u7684\u6570\u636e\u89e3\u6790\u6210json<\/code>\u7ed3\u6784\u5316\u6570\u636e\u683c\u5f0f<\/li>\n<\/ul>\n
                      10.168.99.6 - - [03\/Aug\/2024:09:51:18 +0800] "POST \/wp-admin\/admin-ajax.php HTTP\/1.1" 200 58 "http:\/\/blue.yn.cn\/wp-admin\/post.php?post=1223&action=edit" "Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/126.0.0.0 Safari\/537.36 Edg\/126.0.0.0" "-"<\/code><\/p>\n
                      \u9700\u8981\u4f7f\u7528\u975e\u5e38\u590d\u6742\u7684\u6b63\u5219\u8868\u8fbe\u5f0f<\/p>\n
                        \n
                      • \n
                        grok<\/code>\u5176\u5b9e\u662f\u5e26\u6709\u540d\u5b57\u7684\u6b63\u5219\u8868\u8fbe\u5f0f\u96c6\u5408\u3002grok<\/code>\u5185\u7f6e\u4e86\u5f88\u591apattern<\/code>\u53ef\u4ee5\u76f4\u63a5\u4f7f\u7528\uff1b<\/p>\n<\/li>\n
                      • \n
                        grok<\/code>\u793a\u4f8b\uff1a\u4f7f\u7528grok pattern<\/code>\u5c06nginx<\/code>\u65e5\u5fd7\u683c\u5f0f\u8f6c\u6362\u4e3ajson<\/code>\u683c\u5f0f<\/p>\n<\/li>\n<\/ul>\n
                        cat \/etc\/logstash\/conf.d\/http.logstash.conf\n\ninput {\n        http {\n                port => 5656\n        }\n}\n\nfilter {\n        grok {\n                match => {\n                        "message" => "%{COMBINEDAPACHELOG}"\n                }\n        }\n}\n\noutput {\n        stdout{\n                codec => "rubydebug"\n        }\n}<\/code><\/pre>\n
                          \n
                        • \u4f7f\u7528insomnia post nginx<\/code>\u7684\u975e\u7ed3\u6784\u5316\u65e5\u5fd7\u6570\u636e\u5230logstash<\/code>\u76845656\u6d4b\u8bd5\u7aef\u53e3<\/li>\n<\/ul>\n
                          \"file\"<\/p>\n
                            \n
                          • \u8fd4\u56de\u7ed3\u679c\u5982\u4e0b\uff1a<\/li>\n<\/ul>\n
                            {\n    "httpversion" => "1.1",\n        "headers" => {\n              "http_host" => "192.168.99.21:5656",\n           "request_path" => "\/",\n           "http_version" => "HTTP\/1.1",\n         "content_length" => "291",\n            "http_accept" => "*\/*",\n        "http_user_agent" => "insomnia\/2021.6.0",\n         "request_method" => "POST"\n    },\n          "ident" => "-",\n        "message" => "10.168.99.6 - - [03\/Aug\/2024:09:51:18 +0800] \\"POST \/wp-admin\/admin-ajax.p                                                                                          hp HTTP\/1.1\\" 200 58 \\"http:\/\/blue.yn.cn\/wp-admin\/post.php?post=1223&action=edit\\" \\"Mozi                                                                                          lla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/126.0.0.0 Sa                                                                                          fari\/537.36 Edg\/126.0.0.0\\" \\"-\\"",\n      "timestamp" => "03\/Aug\/2024:09:51:18 +0800",\n        "request" => "\/wp-admin\/admin-ajax.php",\n       "@version" => "1",\n           "verb" => "POST",\n          "bytes" => "58",\n           "auth" => "-",\n       "response" => "200",\n     "@timestamp" => 2024-08-03T02:22:25.201Z,\n       "referrer" => "\\"http:\/\/blue.yn.cn\/wp-admin\/post.php?post=1223&action=edit\\"",\n           "host" => "10.168.99.6",\n       "clientip" => "10.168.99.6",\n          "agent" => "\\"Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, lik                                                                                          e Gecko) Chrome\/126.0.0.0 Safari\/537.36 Edg\/126.0.0.0\\""\n}<\/code><\/pre>\n
                            geoip\u63d2\u4ef6<\/h4>\n
                              \n
                            • \n
                              geoip<\/code>\u63d2\u4ef6\uff1a\u6839\u636eip<\/code>\u5730\u5740\u63d0\u4f9b\u7684\u5bf9\u5e94\u5730\u57df\u6d88\u606f\uff0c\u6bd4\u5982\u7ecf\u7eac\u5ea6\u3001\u57ce\u5e02\u540d\u7b49\uff0c\u65b9\u4fbf\u8fdb\u884c\u5730\u7406\u6570\u636e\u5206\u6790\uff1b<\/p>\n<\/li>\n
                            • \n
                              geoip<\/code>\u793a\u4f8b\uff1a\u901a\u8fc7geoip<\/code>\u63d0\u53d6nginx<\/code>\u65e5\u5fd7\u4e2dclientip<\/code>\u5b57\u6bb5\uff0c\u5e76\u83b7\u53d6\u5730\u57df\u6d88\u606f\uff1b<\/p>\n<\/li>\n<\/ul>\n
                              cat \/etc\/logstash\/conf.d\/http.logstash.conf\n\ninput {\n        http {\n                port => 5656\n        }\n}\n\nfilter {\n        grok {\n                match => {\n                        "message" => "%{COMBINEDAPACHELOG}"\n                }\n        }\n\n        geoip {\n                source => "clientip"\n        }\n}\n\noutput {\n        stdout{\n                codec => "rubydebug"\n        }\n}<\/code><\/pre>\n
                                \n
                              • geoip<\/code>\u7ed3\u679c\uff1a<\/li>\n<\/ul>\n
                                 "geoip" => {\n                    "ip" => "222.217.160.1",\n           "region_name" => "Guangxi",\n        "continent_code" => "AS",\n              "timezone" => "Asia\/Shanghai",\n         "country_code2" => "CN",\n          "country_name" => "China",\n              "latitude" => 22.8137,\n             "longitude" => 108.3146,\n           "region_code" => "GX",\n              "location" => {\n            "lon" => 108.3146,\n            "lat" => 22.8137\n        },\n         "country_code3" => "CN"\n    },<\/code><\/pre>\n
                                fields\u5b57\u6bb5<\/h4>\n
                                  \n
                                • \u7531\u4e8e\u8f93\u51fa\u5185\u5bb9\u592a\u591a\uff0c\u53ef\u4ee5\u901a\u8fc7fileds<\/code>\u9009\u9879\u9009\u62e9\u81ea\u5df1\u9700\u8981\u7684\u4fe1\u606f<\/li>\n<\/ul>\n
                                  cat \/etc\/logstash\/conf.d\/http.logstash.conf\n\ninput {\n        http {\n                port => 5656\n        }\n}\n\nfilter {\n        grok {\n                match => {\n                        "message" => "%{COMBINEDAPACHELOG}"\n                }\n        }\n\n        geoip {\n                source => "clientip"\n                fields => ["country_name","region_name"]\n        }\n}\n\noutput {\n        stdout{\n                codec => "rubydebug"\n        }\n}<\/code><\/pre>\n
                                    \n
                                  • \u8fd4\u56de\u7ed3\u679c\uff1a<\/li>\n<\/ul>\n
                                    "geoip" => {\n        "country_name" => "China",\n         "region_name" => "Guangxi"\n    },<\/code><\/pre>\n
                                    date\u63d2\u4ef6<\/h4>\n
                                      \n
                                    • \n
                                      date<\/code>\u63d2\u4ef6\uff1a\u5c06\u65e5\u671f\u5b57\u7b26\u4e32\u89e3\u6790\u4e3a\u65e5\u5fd7\u7c7b\u578b\u3002\u7136\u540e\u66ff\u6362@timestamp<\/code>\u5b57\u6bb5\u6216\u6307\u5b9a\u7684\u5176\u4ed6\u5b57\u6bb5<\/p>\n
                                        \n
                                      • match<\/code>\u7c7b\u578b\u4e3a\u6570\u7ec4\uff0c\u7528\u4e8e\u6307\u5b9a\u65e5\u671f\u5339\u914d\u7684\u683c\u5f0f\uff0c\u53ef\u4ee5\u4ee5\u6b64\u6307\u5b9a\u591a\u79cd\u65e5\u671f\u683c\u5f0f<\/li>\n
                                      • target<\/code>\u7c7b\u578b\u4e3a\u5b57\u7b26\u4e32\uff0c\u7528\u4e8e\u6307\u5b9a\u8d4b\u503c\u7684\u5b57\u6bb5\u540d\uff0c\u9ed8\u8ba4\u662f@timestamp<\/code><\/li>\n
                                      • timezone<\/code>\u7c7b\u578b\u4e3a\u5b57\u7b26\u4e32\uff0c\u7528\u4e8e\u6307\u5b9a\u65f6\u533a\u57df<\/li>\n<\/ul>\n<\/li>\n
                                      • \n
                                        date<\/code>\u793a\u4f8b\uff0c\u5c06nginx<\/code>\u8bf7\u6c42\u4e2d\u7684timestamp<\/code>\u65e5\u5fd7\u8fdb\u884c\u89e3\u6790<\/p>\n<\/li>\n<\/ul>\n
                                        cat \/etc\/logstash\/conf.d\/http.logstash.conf\n\ninput {\n        http {\n                port => 5656\n        }\n}\n\nfilter {\n        grok {\n                match => {\n                        "message" => "%{COMBINEDAPACHELOG}"\n                }\n        }\n\n        geoip {\n                source => "clientip"\n                fields => ["country_name","region_name"]\n        }\n\n        date {\n                match => ["timestamp","dd\/MMM\/yyyy:HH:mm:ss Z"]\n                target => "@timestamp"\n                timezone => "Asia\/Shanghai"\n        }\n}\n\noutput {\n        stdout{\n                codec => "rubydebug"\n        }\n}<\/code><\/pre>\n
                                          \n
                                        • \u8fd4\u56de\u7ed3\u679c<\/li>\n<\/ul>\n
                                          {\n           "auth" => "-",\n          "ident" => "-",\n     "@timestamp" => 2024-08-03T01:51:18.000Z,\n      "timestamp" => "03\/Aug\/2024:09:51:18 +0800",\n       "referrer" => "\\"http:\/\/blue.yn.cn\/wp-admin\/post.php?post=1223&action=edit\\"",\n    "httpversion" => "1.1",\n          "agent" => "\\"Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/126.0.0.0 Safari\/537.36 Edg\/126.0.0.0\\"",\n        "request" => "\/wp-admin\/admin-ajax.php",\n          "bytes" => "58",\n           "verb" => "POST",\n       "response" => "200",\n        "headers" => {\n           "http_version" => "HTTP\/1.1",\n              "http_host" => "192.168.99.21:5656",\n         "request_method" => "POST",\n            "http_accept" => "*\/*",\n         "content_length" => "293",\n        "http_user_agent" => "insomnia\/2021.6.0",\n           "request_path" => "\/"\n    },<\/code><\/pre>\n
                                          useragent\u63d2\u4ef6<\/h4>\n
                                            \n
                                          • \n
                                            useragent<\/code>\u63d2\u4ef6\uff1a\u6839\u636e\u8bf7\u6c42\u4e2d\u7684user-agent<\/code>\u5b57\u6bb5\uff0c\u89e3\u6790\u51fa\u6d4f\u89c8\u5668\u8bbe\u5907\u3001\u64cd\u4f5c\u7cfb\u7edf\u7b49\u4fe1\u606f<\/p>\n<\/li>\n
                                          • \n
                                            useragent<\/code>\u793a\u4f8b\uff1a<\/p>\n<\/li>\n<\/ul>\n
                                            cat \/etc\/logstash\/conf.d\/http.logstash.conf\n\ninput {\n        http {\n                port => 5656\n        }\n}\n\nfilter {\n        grok {\n                match => {\n                        "message" => "%{COMBINEDAPACHELOG}"\n                }\n        }\n\n        geoip {\n                source => "clientip"\n                fields => ["country_name","region_name"]\n        }\n\n        date {\n                match => ["timestamp","dd\/MMM\/yyyy:HH:mm:ss Z"]\n                target => "@timestamp"\n                timezone => "Asia\/Shanghai"\n        }\n\n        useragent {\n                source => "agent"\n                target => "useragnet"\n        }\n}\n\noutput {\n        stdout{\n                codec => "rubydebug"\n        }\n}<\/code><\/pre>\n