https://segmentfault.com/a/1190000002972420
通配符
? 匹配单个字符
* 匹配0到多个字符
kiba?a, el*search
? * 不能用作第一个字符，例如：?text *text
====================
正则
es支持部分正则功能,性能较差
name:/joh?n(ath[oa]n)/
====================
模糊搜索
quikc~ brwn~ foks~
~:在一个单词后面加上~启用模糊搜索，可以搜到一些拼写错误的单词
first~ 这种也能匹配到 frist
还可以设置编辑距离（整数），指定需要多少相似度
cromm~1 会匹配到 from 和 chrome
默认2，越大越接近搜索的原始值，设置为1基本能搜到80%拼写错误的单词
====================
逻辑操作
AND
OR
+：搜索结果中必须包含此项
-：不能含有此项
+apache -jakarta test aaa bbb：结果中必须存在apache，不能有jakarta，剩余部分尽量都匹配到
====================
分组
(jakarta OR apache) AND jakarta
====================
转义特殊字符
+ - = && || > < ! ( ) { } [ ] ^ " ~ * ?
ELK 教學 - 定期清除 Elasticsearch 資料
https://blog.johnwu.cc/article/elk-purge-elasticsearch-index.html
https://www.elastic.co/guide/en/elasticsearch/client/curator/current/configfile.html
https://www.elastic.co/guide/en/elasticsearch/client/curator/current/ex_delete_indices.html
https://anjia0532.github.io/2017/04/06/elasticsearch-delete-indices-by-date/
config.yml
# Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" client: hosts: - xxx.xxx.xxx.xxx port: 9200 url_prefix: use_ssl: False certificate: client_cert: client_key: ssl_no_validate: False http_auth: timeout: 30 #timeout: 60 master_only: False logging: loglevel: INFO logfile: logformat: default blacklist: ['elasticsearch', 'urllib3'] curator_filebeat.yml
# Remember, leave a key empty if there is no value.
all output columns with logstash
filter { if [srctype] == "etl" { #[fields][srctype] csv { columns => [ "number", "hash", "parent_hash", "nonce", "sha3_uncles", "logs_bloom", "transactions_root", "state_root", "receipts_root", "timestamp", "extra_data", "transaction_count", "gas_limit", "size", "total_difficulty", "difficulty", "miner", "block_hash", "block_number", "transaction_index", "from_address", "to_address", "value", "gas", "gas_price", "input", "address", "bytecode", "function_sighashes", "is_erc20", "is_erc721", "log_index", "transaction_hash", "data", "topics", "cumulative_gas_used", "gas_used", "contract_address", "root,status" ] separator => "," remove_field => ["message"] #autodetect_column_names => true #have problems #autogenerate_column_names => true #have problems skip_empty_columns => true skip_empty_rows => true } }
https://sueboy.blogspot.com/2018/11/elk60filebeatdocumenttype.html
filebeat.yml
- type: log paths: - /var/log/geth.log exclude_files: ['.gz$'] fields: srctype: "geth" pipleline logstah.conf
if [fields][srctype] == “geth” {
BUT fields_under_root: true - type: log paths: - /var/log/geth.log exclude_files: ['.gz$'] fields: srctype: "geth" fields_under_root: true if [srctype] == “geth” {
Oragin
geoip { source => "filebeatserverip" target => "filebeatserveripgeoip" add_field => [ "[filebeatserveripgeoip][coordinates]", "%{[filebeatserveripgeoip][longitude]}" ] add_field => [ "[filebeatserveripgeoip][coordinates]", "%{[filebeatserveripgeoip][latitude]}" ] } mutate { convert => ["[filebeatserveripgeoip][coordinates]", "float"] } Delete
add_field => [ “[filebeatserveripgeoip][coordinates]”, “%{[filebeatserveripgeoip][longitude]}” ]
add_field => [ “[filebeatserveripgeoip][coordinates]”, “%{[filebeatserveripgeoip][latitude]}” ]
convert => ["[filebeatserveripgeoip][coordinates]", “float”]
geoip { source => "filebeatserverip" target => "filebeatserveripgeoip" } mutate { } =====
{ "index_patterns": ["filebeat*", "heartbeat*"], "settings": { "number_of_shards": 1 }, "mappings": { "doc": { "properties": { "filebeatserveripgeoip.
geth log mined
INFO [12-07|13:04:44] 🔨 mined potential block number=1934700 hash=3f9161…88da7d
only month-day …….
grok { match => ["message", "%{LOGLEVEL:logType} \[%{DATA:gethmm}-%{DATA:gethdd}\|%{DATA:gethtime}\] %{GREEDYDATA:tmessage} number=(?\b\w+\b) hash=(?\b\w+...\w+\b)"] add_field => ["gethdate", "%{[gethmm]}-%{[gethdd]} %{[gethtime]}"] } ruby { code => " tstamp = event.get('@timestamp').to_i event.set('epoch',tstamp) event.set('gethdate', Time.at(tstamp).strftime('%Y')+'-'+event.get('gethdate')) " } date { match => [ "gethdate" , "YYYY-MM-dd HH:mm:ss"] target => "gethdate" timezone => "Asia/Taipei" } Recreate index
GET _cat/indices?v
GET _cat/indices?v&s=index
GET filebeat-6.5.1-2018.12.06
DELETE filebeat-6.5.1-2018.12.06
Export
index-pattern
curl http://xxx.xxx.xxx.xxx:5601/api/saved_objects/index-pattern/f1836c20-e880-11e8-8d66-7d7b4c3a5906 > index-pattern-export.json
visualization
curl http://xxx.xxx.xxx.xxx:5601/api/saved_objects/visualization/1eb85311-f901-11e8-864c-bd4880954537 > visual-export.json
Import
index-pattern
curl -v -XPOST kibana:5601/api/kibana/dashboards/import?force=true -H “kbn-xsrf:true” -H “Content-type:application/json” -d @/usr/share/config/config/index-pattern-export.json
visualization
curl -v -XPOST kibana:5601/api/kibana/dashboards/import?force=true -H “kbn-xsrf:true” -H “Content-type:application/json” -d @/usr/share/config/config/visual-export.json
PS：
visualization can be multi
{ "objects": [ {"id":"0c298010-f901-11e8-864c-bd4880954537",...}, {"id":"1eb85300-f901-11e8-864c-bd4880954537",...} ]}
