out of memory 해결 influx DB사용법 및

Quick Guide -- config file -- default error log -- connectprecision 은 시간 대 . -- commands -- queryhttps : // docs.influxdata.com/influxdb/v1.8/query_language/spec/

docs . influxdata . comFluxInfluxQLGet startedhttps : // docs.influxdata.com/influxdb/v1.8/introduction/get-started/

docs.influxdata.com를 활용해 보시고 measurement를 테이블이라고 생각해 주세요. 페널티킥은 항상 타임이에요tags, fields는 테이블 컬럼입니다. tags는 인덱스되어 있고 fields는 인덱스되어 있지 않습니다. RDB와 다른 점은 수백만 개의 테이블(measurement)을 가질 수 있으며 스키머 형태를 갖출 필요 없이 null 값을 저장하지 않습니다.

아래의 포맷으로 influx DB line protocol을 통해 influx DB에 글을 남깁니다.예시 ) QueryManagement Queryhttps : // docs.influxdata.com/influxdb/v1.8/query_language/manage-database/

DROPRETENTION POLICYPorotocolLine Protocol Syntax

ElementOptional / RequiredDescriptionType ( See data types for more information . ) MeasurementRequiredThe measurement name . InfluxDB accepts one measurement per point .StringTag setOptionalAll tag key - value pairs for the point .Tag keys and tag values are both strings . Field setRequired . Points must have at least one field . All field key - value pairs for the point . Field keys are strings . Field values can be floats , integers , strings , or Booleans . TimestampOptional . InfluxDB uses the server ' s local nanosecond timestamp in UTC if the timestamp is not included with the point .The timestamp for the data point . InfluxDB accepts one timestamp per point . Unix nanosecond timestamp . Specify alternative precisions with the InfluxDB API . Data TypeDatatypeElement ( s ) DescriptionFloatField valuesDefault numerical type . IEEE - 754 64 - bit floating - point numbers ( except NaN or + / - Inf ) . Examples : 1 , 1 . 0 , 1 . e + 78 , 1 . E + 78 . IntegerField valuesSigned 64 - bit integers ( - 9223372036854775808 to 9223372036854775807 ) . Specify an integer with a trailing i on the number . Example : 1i .StringMeasurements , tag keys , tag values , field keys , field valuesLength limit 64KB . BooleanField valuesStores TRUE or FALSE values .

TRUE write syntax : [ t , T , true , True , TRUE ] .

FALSE write syntax : [ f , F , false , False , FALSE ] TimestampTimestampsUnix nanosecond timestamp . Specify alternative precisions with the InfluxDB API . The minimum valid timestamp is - 9223372036854775806 or 1677 - 09 - 21 T 00 : 12 : 43 . 145224194Z . The maximum valid timestamp is 9223372036854775806 or 2262 - 04 - 11 T 23 : 47 : 16 . 854775806Z .ConceptsTime Series Index ( TSI ) https://docs.influxdata.com/influxdb/v 1 . 7 / concepts / time - series - index / https : // docs.influxdata.com/influxdb/v1.8/concepts/storage_engine/

InfluxDB stores measurement and tag information in an index so data can be queried quickly . Time Series Index ( TSI ) stores index data both in memory and on disk , removing RAM restrictions . This lets you store more series on a machine . TSI uses the operating system ' s page cache to pull hot data into memory , leaving cold data on disk .

To support a large number of time series , that is , a very high cardinality in the number of unique time series that the database stores , InfluxData has added the new Time Series Index ( TSI ) . InfluxData supports customers using InfluxDB with tens of millions of time series . InfluxData ' s goal , however , is to expand to hundreds of millions , and eventually billions . Using InfluxData ' s TSI storage engine , users should be able to have millions of unique time series . The goal is that the number of series should be unbounded by the amount of memory on the server hardware . Importantly , the number of series that exist in the database will have a negligible impact on database startup time . This work represents the most significant technical advancement in the database since InfluxData released the Time Series Merge Tree ( TSM ) storage engine in 2016 .

The new time series index ( TSI ) moves the index to files on disk that we memory map . This means that we let the operating system handle being the Least Recently Used ( LRU ) memory . Much like the TSM engine for raw time series data we have a write - ahead log with an in - memory structure that gets merged at query time with the memory - mapped index . Background routines run constantly to compact the index into larger and larger files to avoid having to do too many index merges at query time . Under the covers , we ' re using techniques like Robin Hood Hashing to do fast index lookups and HyperLogLog ++ to keep sketches of cardinality estimates . The latter will give us the ability to add things to the query languages like the SHOW CARDINALITY queries .

Time Structured Merge tree ( TSM ) The purpose - built data storage format for InfluxDB . TSM allows for greater compaction and higher write and read throughput than existing B + or LSM tree implementations . See Storage Engine for more .

The Time - Structured Merge Tree ( TSM ) engine that InfluxData built in 2015 and continued enhancing in 2016 was an effort to solve the problem of getting maximum throughput , compression , and query speed for raw time series data . Up unti

더 깊은 비교 자료 https://blog.timescale.com/blog/timescaledb-vs-influxdb-for-time-series-data-timescale-influx-sql-nosql-36489299877/An in - depth look into how two leading time - series databases stack up in terms of data model , query language , reliability , performance , ecosystem , operational management , and company / community support . blog . timescale . comTroubleshooting

clustering을 위해서는 enterprise로 가거나 오픈소스 cluster를 도입해야 하는데, 이도저도 아닌 single node를 사용하는 DB에서 out of memory가 나왔다. 「single node로 견딜 수 있는 정도에 대한 가이드 라인을 제시하지만, 디스크만을 메모리에 꽂아도 해결된다고는 생각되지 않는다.https://docs.influxdata.com/influxdb/v1.7/guides/hardware_sizing/

influxDB out of memory issue

continuesquery가 부하가 있을지도 모르니까 꺼봐...retention policy로 인해 삭제 시 메모리를 먹을 수 있고 retention을 못하게 늘려봐.결국에는 메모리를 많이 사용하지 않도록 데이터량을 줄이는 수밖에... 물리 데이터 공간을 800G 이상 사용하고 있는데.. 이 물리 파일을 접근할 때마다 메모리를 많이 쓰니까.

현재 상태: dropshard xx; 명령 불가 shard 파일이 500M 로 작은데 명령을 내렸을 때 influxdb 가 다운됩니다.

단서 1. 위에 tsi를 rebuild 할 때 실제로 인덱스 파일을 rm 하는 것에 착안하여 단서 2. drop shard xx; 하고 끝나기 전에 바로 취소하고 같은 명령으로 drop shard xx했을 때 shard가 삭제된 것처럼 show shard에 나타나지 않았습니다. 하지만 실제 물리 위치에는 존재. 이걸 봐도 아직 메타 데이터와 데이터 파일의 매핑이 깔끔하지 않은 것 같습니다.

해결 : 따라서 influx DB를 내리고 삭제한 shard 파일을 물리적으로 mv한 후 다시 influx DB를 올립니다. 혹시 모르니까 올라가서 drop shard를 꺼낼게요

 

I switched the index from memory to tsi 1 , as I read in the 1 . 7 documents , but it doesn ' t helpshow retention policies

https://github.com/influxdata/influxdb/issues/13318https : // github.com/influxdata/influxdb/issues/10468

Q & A ) In the mean time , if you have any more information on your setups that would be very helpful . For example :

Q . What index are you using ( TSI / inmem ) ? A . index - version = " tsi 1 "

Q . How many shards do you have ? A . Around 10000

Q . What ' s your database cardinality ? SHOW SERIES CARDINALITY . A . There are roughly 100 databases , the ' worst ' one has carnality of around 40000

Q . If you could provide profiles whilst your 1 . 7 system is using lots of heap ( the higher the heap use the better ) that would really help us . You can do curl - o profiles . tar . gz http : // localhost : 8086 / debug / pprof / all ? cpu = true It will take 30s for the request to complete . A . Getting 404 on this influx endpoint for some reason 참고 자료 Manualhttps : // docs . influxdata.com/influxdb/v1.7/administration/config/ # data - settingshttps : // docs.influxdata.com/influxdb/v1.7/administration/upgrading/https : // docs.influxdata.com/influxdb/v1.7/concepts/tsi-details/

사용사례 https://www.popit.kr/influxdb_telegraf_grafana_2/ 두 번째로 Telegraf를 이용하여 OS 및 MySQL 모니터링 데이터를 수집하여 Influx DB로 저장하는 부분에 대한 설명입니다.www . popit . krFAQhttps : // docs.influxdata.com/influxdb/v1.8/troubleshooting/frequently-asked-questions/

docs.influxdata.com 용어설명 measurement, tag-key, tag-value, field-key, field-value https://blog. naver.com/PostView.nhn? blogId=alice_k106&logNo= 221226137412&parent CategoryNo=&categoryNo= InfluxD ... blog . naver . comdesign insight & tradeoffhttps : // docs.influxdata.com/influxdb/v1.8/concepts/insights_tradeoffs/

docs . influxdata . comTimescaleDB 와 influx DB 의 비교 자료 https : // severalnines.com/database-blog/which-time-series-database-better-timescaledb-vs-influxdbIn the new time - series database world , TimescaleDB and InfluxDB are two popular options with fundamentally different architectures . One is based off a relational database , PostgreSQL , the other build as a NoSQL engine . In this blog , we ' ll give you a short description of those two , and how they stack ...severalnines . comhttps : // blog.timescale.com/blog/timescaledb-vs-influxdb-for-time-series-data-timescale-influx-sql-nosql-36489299877/An in - depth look into how two leading time - series databases stack up in terms of data model , query language , reliability , performance , ecosystem , operational management , and company / community support.blog.timescale.com가장 많이 쓰이는 time series db 종류 들 https : // medium.com/schkn/4-best-time-series-databases-to-watch-in-2019-ef1e89a72377When developing IoT , financial or industrial applications , the choice of a good time series database is most of the time a headache …medium . comhttps : // dzone.com/articles/4-best-time-series-databases-to-watch-in-2019Four time series databases that you should use in 2019 are InfluxDB , TimescaleDB , OpenTSDB , and Graphite . dzone . comhttps : // db-engines.com/en/ranking/time+series+dbmsPopularity ranking of time Series DBMS . db - engines . cominfluxDB 에 대한 자세한 리뷰 https : // www.youtube.com/watch  ? v = a00l1 GxJszM