{"id":7836,"date":"2025-02-12T21:13:31","date_gmt":"2025-02-12T20:13:31","guid":{"rendered":"https:\/\/swissmakers.ch\/?p=7836"},"modified":"2025-03-11T12:30:11","modified_gmt":"2025-03-11T11:30:11","slug":"flexibly-generate-kibana-dashboards-part-1-3","status":"publish","type":"post","link":"https:\/\/swissmakers.ch\/en\/flexibel-kibana-dashboards-generieren-teil-1-3\/","title":{"rendered":"Generate Kibana dashboards flexibly - Part 1\/3"},"content":{"rendered":"<div class=\"gb-container gb-container-1a78881e\">\n\n<h2 class=\"gb-headline gb-headline-779574ff gb-headline-text\">Motivation<\/h2>\n\n\n\n<p class=\"gb-headline gb-headline-6464b20d gb-headline-text\">It is easy and practical to create interactive dashboards in Kibana. However, you often want to have different variants of a dashboard, for example for different environments or customers. Or you want to use exactly the same positions or the same colours for dashboard elements. Then it becomes tedious to design everything interactively. In such cases, it is simpler and more efficient if dashboards can be generated automatically with the help of variables.<\/p>\n\n<\/div>\n\n<div class=\"gb-container gb-container-0af3f288\">\n\n<h2 class=\"gb-headline gb-headline-2697ef74 gb-headline-text\">Idea<\/h2>\n\n\n\n<p class=\"gb-headline gb-headline-274f4738 gb-headline-text\">templates and a simple <a href=\"https:\/\/en.wikipedia.org\/wiki\/Template_processor\" target=\"_blank\" rel=\"noreferrer noopener\">Template processor<\/a> (template engine) can be used to flexibly generate Kibana dashboards.<\/p>\n\n\n\n<p class=\"gb-headline gb-headline-82a6836d gb-headline-text\">This idea can be used analogously for many other automation solutions and is not a special idea for Kibana dashboards. Originally, template processors were mainly used to generate HTML pages.<\/p>\n\n\n\n<p class=\"gb-headline gb-headline-a7f3b9b9 gb-headline-text\">The following <strong>Components<\/strong> are necessary:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Input file(s) with variables<\/li>\n\n\n\n<li>Template as text file<\/li>\n\n\n\n<li>Template processor<\/li>\n<\/ul>\n\n\n\n<p class=\"gb-headline gb-headline-21ead8b4 gb-headline-text\">The template processor generates output file(s) from the template together with the input file(s), such as configuration files or files in one of the following formats: html, xml, json, ndjson or whatever is needed in the given context.<\/p>\n\n\n\n<p class=\"gb-headline gb-headline-43bd0a77 gb-headline-text\">The template processor must be able to perform at least simple mathematical operations (addition, multiplication, modulo), loops and conditions and not just replace or generate texts. See also <a href=\"http:\/\/www.simple-is-better.org\/template\/index.html\" target=\"_blank\" rel=\"noopener\">http:\/\/www.simple-is-better.org\/template\/index.html<\/a>.<\/p>\n\n<\/div>\n\n<div class=\"gb-container gb-container-c586702b\">\n\n<h2 class=\"gb-headline gb-headline-2a61ffd1 gb-headline-text\">Selected tools<\/h2>\n\n\n\n<p class=\"gb-headline gb-headline-35667904 gb-headline-text\"><strong>Json files<\/strong> are used for the input file with variables, but yaml files or other formats are also possible.<\/p>\n\n\n\n<p class=\"gb-headline gb-headline-1f868e75 gb-headline-text\">The Python template processor <strong><a href=\"https:\/\/pypi.org\/project\/Jinja2\/\" target=\"_blank\" rel=\"noreferrer noopener\">Jinja2<\/a> <\/strong>is used as a template processor. Python is widely used and Jinja2 is also often used, for example in Ansible.<\/p>\n\n\n\n<p class=\"gb-headline gb-headline-081282f1 gb-headline-text\">Jinja2 can be used in Python scripts. However, it is easier to use a ready-made command line tool to generate dashboards. For more complex applications, this can also be called from a shell script. Here <strong><a href=\"https:\/\/github.com\/kpfleming\/jinjanator\" target=\"_blank\" rel=\"noreferrer noopener\">jinjanator<\/a><\/strong> that is actively developed further. An alternative would be <strong><a href=\"https:\/\/github.com\/kblomqvist\/yasha\" target=\"_blank\" rel=\"noreferrer noopener\">yasha<\/a><\/strong>which works well but is somewhat older.<\/p>\n\n\n\n<p class=\"gb-headline gb-headline-bcc89a66 gb-headline-text\">There are many other alternatives for template processors, e.g. the <a href=\"https:\/\/template-toolkit.org\/about.html\" target=\"_blank\" rel=\"noreferrer noopener\">Perl Template Toolkit<\/a>. This is much more powerful than Python and jinja2, but is no longer really being developed further.<\/p>\n\n\n\n<p class=\"gb-headline gb-headline-06817f86 gb-headline-text\">For a list of template processors, see e.g: <a href=\"https:\/\/en.wikipedia.org\/wiki\/Comparison_of_web_template_engines\" target=\"_blank\" rel=\"noopener\">https:\/\/en.wikipedia.org\/wiki\/Comparison_of_web_template_engines<\/a><\/p>\n\n\n\n<p class=\"gb-headline gb-headline-30cceac2 gb-headline-text\">If Python is installed, Jinja2 and jinjanator are installed with pip:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"bash\" class=\"language-bash\">pip install Jinja2\npip install jinjanator<\/code><\/pre>\n\n\n\n<p class=\"gb-headline gb-headline-55c24ed4 gb-headline-text\"><strong>VisualStudioCode <\/strong>(VSCode) is used as an editor for the input files and templates. VSCode can be used under Windows, Linux and well also in Windows WSL2 and there is good support for VSCode for Python and json files with extensions in the <a href=\"https:\/\/marketplace.visualstudio.com\/VSCode\" target=\"_blank\" rel=\"noreferrer noopener\">VSCode-Marketplace<\/a>. <\/p>\n\n\n\n<p class=\"gb-headline gb-headline-1105a908 gb-headline-text\">To create the templates, I recommend using VSCode with the extension <strong><a href=\"https:\/\/marketplace.visualstudio.com\/items?itemName=andyyaldoo.vscode-json\" target=\"_blank\" rel=\"noreferrer noopener\">vscode-json<\/a><\/strong> together with the <a href=\"https:\/\/marketplace.visualstudio.com\/items?itemName=ms-python.python\" target=\"_blank\" rel=\"noreferrer noopener\">Python Extension from Microsoft<\/a> to use.<\/p>\n\n<\/div>\n\n<div class=\"gb-container gb-container-ec0c7785\">\n\n<h2 class=\"gb-headline gb-headline-9e13d11e gb-headline-text\">Implementation for Kibana dashboards<\/h2>\n\n\n\n<h3 class=\"gb-headline gb-headline-8d51fcea gb-headline-text\">Prerequisites<\/h3>\n\n\n\n<p class=\"gb-headline gb-headline-a954a4a7 gb-headline-text\">Like other Kibana Saved Objects, Kibana Dashboards have a representation as an ndjson file. Such ndjson files can be exported from Kibana and imported into Kibana.<\/p>\n\n\n\n<p class=\"gb-headline gb-headline-3020d3fd gb-headline-text\">Ndjson files are text files and can be generated to create flexible dashboards.<\/p>\n\n\n\n<h3 class=\"gb-headline gb-headline-afe03037 gb-headline-text\">Ndjson files with dashboard definitions<\/h3>\n\n\n\n<p class=\"gb-headline gb-headline-4488c5c7 gb-headline-text\">Ndjson stands for \"<a href=\"https:\/\/github.com\/ndjson\/ndjson-spec\" target=\"_blank\" rel=\"noreferrer noopener\">Newline delimited json<\/a>\". It is a standard for files that contain one json document per line. This is very practical for exporting several objects to a single file in Kibana or importing them from a file. However, the files have very long lines and are therefore not well suited for making interactive changes to them.<\/p>\n\n\n\n<p class=\"gb-headline gb-headline-1dc67a59 gb-headline-text\">The ndjson format for dashboards is not documented except in the Kibana code. Sometimes it takes a little imagination to understand the format. Tests and simple examples can help. Updating for new Kibana versions can therefore be time-consuming and you don't have a good overview of what is really possible when generating dashboards from programmes. <\/p>\n\n\n\n<p class=\"gb-headline gb-headline-b19f7814 gb-headline-text\">The following files in the Kibana code can provide a few tips on the ndjson format for dashboards:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/github.com\/elastic\/kibana\/blob\/5342f327ee62219c5cf12d50e424da9b91330d5e\/packages\/kbn-check-mappings-update-cli\/current_fields.json#L263\" target=\"_blank\" rel=\"noreferrer noopener\">kibana\/packages\/kbn-check-mappings-update-cli\/current_fields.json<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/github.com\/elastic\/kibana\/blob\/5342f327ee62219c5cf12d50e424da9b91330d5e\/api_docs\/dashboard.devdocs.json#L1288\" target=\"_blank\" rel=\"noreferrer noopener\">kibana\/api_docs\/dashboard.devdocs.json<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/github.com\/elastic\/kibana\/blob\/5342f327ee62219c5cf12d50e424da9b91330d5e\/src\/plugins\/dashboard\/server\/content_management\/v3\/transform_utils.ts#L49\" target=\"_blank\" rel=\"noreferrer noopener\">kibana\/src\/plugins\/dashboard\/server\/content_management\/v3\/transform_utils.ts<\/a><\/li>\n<\/ul>\n\n\n\n<h3 class=\"gb-headline gb-headline-213d13fe gb-headline-text\">Dashboard as a template<\/h3>\n\n\n\n<p class=\"gb-headline gb-headline-65db2c7a gb-headline-text\">In Kibana first <strong>Interactive dashboard<\/strong> which then serves as the basis for a template. Panels that are to be placed several times in the generated dashboard should already appear several times in the template. If possible, all elements and settings that are needed should already be included in the template so that they are already represented in the ndjson file of the dashboard.<\/p>\n\n\n\n<h3 class=\"gb-headline gb-headline-a5e7c405 gb-headline-text\">Export dashboard<\/h3>\n\n\n\n<p class=\"gb-headline gb-headline-2a341b9a gb-headline-text\">In Elastic, dashboards (and other saved objects) can be exported and saved as ndjson files under the menu item \"Management \u2192 Stack Management \u2192 Saved Objects\".<\/p>\n\n\n\n<p class=\"gb-headline gb-headline-e1dee027 gb-headline-text\">It is advisable to save both the dashboard that serves as a template and the ndjson file with the exported dashboard. After Kibana updates, the updated dashboard can then be re-exported and compared with the old, exported version. This helps to adapt templates for generating dashboards to new Kibana versions.<\/p>\n\n\n\n<h3 class=\"gb-headline gb-headline-ce7859e4 gb-headline-text\">Input file with variables<\/h3>\n\n\n\n<p class=\"gb-headline gb-headline-63ad10df gb-headline-text\">A json input file must be written with all the variables that are to appear in the generated dashboard. It is also advisable to define a title (name) for the dashboard in the input file so that the dashboards can be differentiated in the Kibana GUI. The exact format of the input file is not so important, it must only be possible to access the values in a simple way.<\/p>\n\n\n\n<p class=\"gb-headline gb-headline-8628703c gb-headline-text\">Different dashboards can then be created by customising the input files or using different input files.<\/p>\n\n\n\n<h3 class=\"gb-headline gb-headline-f24cf99f gb-headline-text\">Reshaping the ndjson file with the dashboard<\/h3>\n\n\n\n<p class=\"gb-headline gb-headline-cd0deaf2 gb-headline-text\">Ndjson files are not suitable for interactive changes in VSCode as they have long lines and are not easy to read. In addition, they are not valid json files and many json editors and extensions have difficulty dealing with them.<\/p>\n\n\n\n<p class=\"gb-headline gb-headline-bd8fd452 gb-headline-text\">Another difficulty is that the information on the dashboard panels that are to be customised is contained in array elements that actually contain json parts again. However, as it is not possible to define such nested elements in json, a prefix of <strong>Backslash<\/strong>. Here you must ensure that these backslashes are retained when reformatting, otherwise the ndjson file generated at the end can no longer be imported into Kibana.<\/p>\n\n\n\n<p class=\"gb-headline gb-headline-e1a7079e gb-headline-text\">The easiest way to do this is to format the files in VSCode before editing and then undo the formatting. The following steps serve this purpose:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Copy the ndjson file with the dashboard to a file with the extension .j2, so that the original file is retained. Changing the file extension is not really necessary, but marks the file as a file with jinja2 elements.<\/li>\n\n\n\n<li>Copy the line with the required dashboard definition and save this line in a separate, temporary file with the extension .json.<\/li>\n\n\n\n<li>Open the json file with the dashboard definition in VSCode.<\/li>\n\n\n\n<li>Convert to a more readable, formatted json file with &lt;ctrl&gt;&lt;alt&gt;&lt;b&gt; (vscode-json: <strong>Beautify<\/strong>).<\/li>\n\n\n\n<li>If necessary, insert line breaks and additional characters in the part with the dashboard panel definition so that the file can be edited more easily. This can be done by replacing the commas in the editor, inserting a line break and a character string that does not otherwise appear in the file.<\/li>\n\n\n\n<li>Edit the file and insert jinja2 elements.<\/li>\n\n\n\n<li>Delete any line breaks and character strings inserted in step 5.<\/li>\n\n\n\n<li>To be on the safe side, delete IDs so that new IDs are generated during the subsequent import and no conflicts arise. In particular, there must not be several dashboards with the same ID in one file, otherwise there will be an error during import.<\/li>\n\n\n\n<li>Reset the formatting of the file with <u> (vscode-json: <strong>Uglify<\/strong>).<\/li>\n\n\n\n<li>Copy the line with the dashboard template and replace the dashboard definition with it in the j2 file created in step 1.<\/li>\n<\/ol>\n\n\n\n<h3 class=\"gb-headline gb-headline-d6e2da7b gb-headline-text\">Template with jinja2<\/h3>\n\n\n\n<p class=\"gb-headline gb-headline-4bdd9682 gb-headline-text\">In step 6 above, a template with jinja2 variables and expressions is created interactively. The variables and expressions use the definitions of the values from the input file. The documentation for Jinja2 can be found here: <a href=\"https:\/\/jinja.palletsprojects.com\/en\/stable\/templates\" target=\"_blank\" rel=\"noopener\">https:\/\/jinja.palletsprojects.com\/en\/stable\/templates<\/a><\/p>\n\n\n\n<p class=\"gb-headline gb-headline-8b3a1718 gb-headline-text\">To document the template, jinja2 comments of the form <code>{# Comment text #}<\/code> can be inserted directly into the template file.<\/p>\n\n\n\n<h3 class=\"gb-headline gb-headline-75fbf297 gb-headline-text\">Template processor<\/h3>\n\n\n\n<p class=\"gb-headline gb-headline-5c300721 gb-headline-text\">The template processor jinja2 can now be used to create an ndjson file with the generated dashboard from the input file with the variables and the jinja2 template file. To do this&nbsp; <strong><a href=\"https:\/\/github.com\/kpfleming\/jinjanator\" target=\"_blank\" rel=\"noreferrer noopener\">jinjanate<\/a><\/strong> can be used on the command line.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"bash\" class=\"language-bash\">jinjanate template_file.j2 inputfile_vars.json -o generated_dashboard.ndjson<\/code><\/pre>\n\n\n\n<h3 class=\"gb-headline gb-headline-427b1975 gb-headline-text\">Import dashboard<\/h3>\n\n\n\n<p class=\"gb-headline gb-headline-c42143f3 gb-headline-text\">In Elastic, dashboards (and other saved objects) can be imported interactively from ndjson files under the menu item \"Management \u2192 Stack Management \u2192 Saved Objects\". The generated dashboard can now be imported into Kibana here.<\/p>\n\n\n\n<p class=\"gb-headline gb-headline-fd5771d8 gb-headline-text\">It is best to select the option \"Create new objects with random Ids\" to avoid conflicts with Ids as far as possible.<\/p>\n\n<\/div>\n\n<div class=\"gb-container gb-container-589d9d28\">\n\n<h2 class=\"gb-headline gb-headline-09e2fd62 gb-headline-text\">Would you like examples?<\/h2>\n\n\n\n<p class=\"gb-headline gb-headline-e6ab0f43 gb-headline-text\">Curious about the first application examples? You'll find them in a few weeks' time in the next part of this blog post.<\/p>\n\n\n\n<p class=\"gb-headline gb-headline-e2c548f9 gb-headline-text\">Of course, you can always contact Swissmakers for support in creating professional Kibana dashboards.<\/p>\n\n<\/div>","protected":false},"excerpt":{"rendered":"<p>Motivation It is easy and practical to design interactive dashboards in Kibana. You often want to ... <\/p>\n<p class=\"read-more-container\"><a title=\"Generate Kibana dashboards flexibly - Part 1\/3\" class=\"read-more button\" href=\"https:\/\/swissmakers.ch\/en\/flexibel-kibana-dashboards-generieren-teil-1-3\/#more-7836\" aria-label=\"Read more about Generate Kibana dashboards flexibly - Part 1\/3\">Read more<\/a><\/p>","protected":false},"author":9,"featured_media":7868,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_eb_attr":"","_kad_blocks_custom_css":"","_kad_blocks_head_custom_js":"","_kad_blocks_body_custom_js":"","_kad_blocks_footer_custom_js":"","footnotes":""},"categories":[55,69,70,71,16],"tags":[],"class_list":["post-7836","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-linux","category-automatisierung","category-elastic","category-kibana","category-monitoring","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50"],"taxonomy_info":{"category":[{"value":55,"label":"Linux"},{"value":69,"label":"Automatisierung"},{"value":70,"label":"Elastic"},{"value":71,"label":"Kibana"},{"value":16,"label":"Monitoring"}]},"featured_image_src_large":["https:\/\/swissmakers.ch\/wp-content\/uploads\/2025\/02\/Bild-Kibana-Dashboards-1200x600-1-1024x512.png",1024,512,true],"author_info":{"display_name":"Matthias Dillier","author_link":"https:\/\/swissmakers.ch\/en\/author\/matthias\/"},"comment_info":1,"category_info":[{"term_id":55,"name":"Linux","slug":"linux","term_group":0,"term_taxonomy_id":55,"taxonomy":"category","description":"","parent":0,"count":10,"filter":"raw","cat_ID":55,"category_count":10,"category_description":"","cat_name":"Linux","category_nicename":"linux","category_parent":0},{"term_id":69,"name":"Automatisierung","slug":"automatisierung","term_group":0,"term_taxonomy_id":69,"taxonomy":"category","description":"","parent":0,"count":3,"filter":"raw","cat_ID":69,"category_count":3,"category_description":"","cat_name":"Automatisierung","category_nicename":"automatisierung","category_parent":0},{"term_id":70,"name":"Elastic","slug":"elastic","term_group":0,"term_taxonomy_id":70,"taxonomy":"category","description":"","parent":16,"count":3,"filter":"raw","cat_ID":70,"category_count":3,"category_description":"","cat_name":"Elastic","category_nicename":"elastic","category_parent":16},{"term_id":71,"name":"Kibana","slug":"kibana","term_group":0,"term_taxonomy_id":71,"taxonomy":"category","description":"","parent":16,"count":3,"filter":"raw","cat_ID":71,"category_count":3,"category_description":"","cat_name":"Kibana","category_nicename":"kibana","category_parent":16},{"term_id":16,"name":"Monitoring","slug":"monitoring","term_group":0,"term_taxonomy_id":16,"taxonomy":"category","description":"","parent":0,"count":8,"filter":"raw","cat_ID":16,"category_count":8,"category_description":"","cat_name":"Monitoring","category_nicename":"monitoring","category_parent":0}],"tag_info":false,"_links":{"self":[{"href":"https:\/\/swissmakers.ch\/en\/wp-json\/wp\/v2\/posts\/7836","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/swissmakers.ch\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/swissmakers.ch\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/swissmakers.ch\/en\/wp-json\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"https:\/\/swissmakers.ch\/en\/wp-json\/wp\/v2\/comments?post=7836"}],"version-history":[{"count":14,"href":"https:\/\/swissmakers.ch\/en\/wp-json\/wp\/v2\/posts\/7836\/revisions"}],"predecessor-version":[{"id":7910,"href":"https:\/\/swissmakers.ch\/en\/wp-json\/wp\/v2\/posts\/7836\/revisions\/7910"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/swissmakers.ch\/en\/wp-json\/wp\/v2\/media\/7868"}],"wp:attachment":[{"href":"https:\/\/swissmakers.ch\/en\/wp-json\/wp\/v2\/media?parent=7836"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/swissmakers.ch\/en\/wp-json\/wp\/v2\/categories?post=7836"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/swissmakers.ch\/en\/wp-json\/wp\/v2\/tags?post=7836"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}