Motivation
It is easy and practical to create interactive dashboards in Kibana. However, you often want to have different variants of a dashboard, for example for different environments or customers. Or you want to use exactly the same positions or the same colours for dashboard elements. Then it becomes tedious to design everything interactively. In such cases, it is simpler and more efficient if dashboards can be generated automatically with the help of variables.
Idea
templates and a simple Template processor (template engine) can be used to flexibly generate Kibana dashboards.
This idea can be used analogously for many other automation solutions and is not a special idea for Kibana dashboards. Originally, template processors were mainly used to generate HTML pages.
The following Components are necessary:
- Input file(s) with variables
- Template as text file
- Template processor
The template processor generates output file(s) from the template together with the input file(s), such as configuration files or files in one of the following formats: html, xml, json, ndjson or whatever is needed in the given context.
The template processor must be able to perform at least simple mathematical operations (addition, multiplication, modulo), loops and conditions and not just replace or generate texts. See also http://www.simple-is-better.org/template/index.html.
Selected tools
Json files are used for the input file with variables, but yaml files or other formats are also possible.
The Python template processor Jinja2 is used as a template processor. Python is widely used and Jinja2 is also often used, for example in Ansible.
Jinja2 can be used in Python scripts. However, it is easier to use a ready-made command line tool to generate dashboards. For more complex applications, this can also be called from a shell script. Here jinjanator that is actively developed further. An alternative would be yashawhich works well but is somewhat older.
There are many other alternatives for template processors, e.g. the Perl Template Toolkit. This is much more powerful than Python and jinja2, but is no longer really being developed further.
For a list of template processors, see e.g: https://en.wikipedia.org/wiki/Comparison_of_web_template_engines
If Python is installed, Jinja2 and jinjanator are installed with pip:
pip install Jinja2
pip install jinjanatorVisualStudioCode (VSCode) is used as an editor for the input files and templates. VSCode can be used under Windows, Linux and well also in Windows WSL2 and there is good support for VSCode for Python and json files with extensions in the VSCode-Marketplace.
To create the templates, I recommend using VSCode with the extension vscode-json together with the Python Extension from Microsoft to use.
Implementation for Kibana dashboards
Prerequisites
Like other Kibana Saved Objects, Kibana Dashboards have a representation as an ndjson file. Such ndjson files can be exported from Kibana and imported into Kibana.
Ndjson files are text files and can be generated to create flexible dashboards.
Ndjson files with dashboard definitions
Ndjson stands for "Newline delimited json". It is a standard for files that contain one json document per line. This is very practical for exporting several objects to a single file in Kibana or importing them from a file. However, the files have very long lines and are therefore not well suited for making interactive changes to them.
The ndjson format for dashboards is not documented except in the Kibana code. Sometimes it takes a little imagination to understand the format. Tests and simple examples can help. Updating for new Kibana versions can therefore be time-consuming and you don't have a good overview of what is really possible when generating dashboards from programmes.
The following files in the Kibana code can provide a few tips on the ndjson format for dashboards:
- kibana/packages/kbn-check-mappings-update-cli/current_fields.json
- kibana/api_docs/dashboard.devdocs.json
- kibana/src/plugins/dashboard/server/content_management/v3/transform_utils.ts
Dashboard as a template
In Kibana first Interactive dashboard which then serves as the basis for a template. Panels that are to be placed several times in the generated dashboard should already appear several times in the template. If possible, all elements and settings that are needed should already be included in the template so that they are already represented in the ndjson file of the dashboard.
Export dashboard
In Elastic, dashboards (and other saved objects) can be exported and saved as ndjson files under the menu item "Management → Stack Management → Saved Objects".
It is advisable to save both the dashboard that serves as a template and the ndjson file with the exported dashboard. After Kibana updates, the updated dashboard can then be re-exported and compared with the old, exported version. This helps to adapt templates for generating dashboards to new Kibana versions.
Input file with variables
A json input file must be written with all the variables that are to appear in the generated dashboard. It is also advisable to define a title (name) for the dashboard in the input file so that the dashboards can be differentiated in the Kibana GUI. The exact format of the input file is not so important, it must only be possible to access the values in a simple way.
Different dashboards can then be created by customising the input files or using different input files.
Reshaping the ndjson file with the dashboard
Ndjson files are not suitable for interactive changes in VSCode as they have long lines and are not easy to read. In addition, they are not valid json files and many json editors and extensions have difficulty dealing with them.
Another difficulty is that the information on the dashboard panels that are to be customised is contained in array elements that actually contain json parts again. However, as it is not possible to define such nested elements in json, a prefix of Backslash. Here you must ensure that these backslashes are retained when reformatting, otherwise the ndjson file generated at the end can no longer be imported into Kibana.
The easiest way to do this is to format the files in VSCode before editing and then undo the formatting. The following steps serve this purpose:
- Copy the ndjson file with the dashboard to a file with the extension .j2, so that the original file is retained. Changing the file extension is not really necessary, but marks the file as a file with jinja2 elements.
- Copy the line with the required dashboard definition and save this line in a separate, temporary file with the extension .json.
- Open the json file with the dashboard definition in VSCode.
- Convert to a more readable, formatted json file with <ctrl><alt><b> (vscode-json: Beautify).
- If necessary, insert line breaks and additional characters in the part with the dashboard panel definition so that the file can be edited more easily. This can be done by replacing the commas in the editor, inserting a line break and a character string that does not otherwise appear in the file.
- Edit the file and insert jinja2 elements.
- Delete any line breaks and character strings inserted in step 5.
- To be on the safe side, delete IDs so that new IDs are generated during the subsequent import and no conflicts arise. In particular, there must not be several dashboards with the same ID in one file, otherwise there will be an error during import.
- Reset the formatting of the file with (vscode-json: Uglify).
- Copy the line with the dashboard template and replace the dashboard definition with it in the j2 file created in step 1.
Template with jinja2
In step 6 above, a template with jinja2 variables and expressions is created interactively. The variables and expressions use the definitions of the values from the input file. The documentation for Jinja2 can be found here: https://jinja.palletsprojects.com/en/stable/templates
To document the template, jinja2 comments of the form {# Comment text #} can be inserted directly into the template file.
Template processor
The template processor jinja2 can now be used to create an ndjson file with the generated dashboard from the input file with the variables and the jinja2 template file. To do this jinjanate can be used on the command line.
jinjanate template_file.j2 inputfile_vars.json -o generated_dashboard.ndjsonImport dashboard
In Elastic, dashboards (and other saved objects) can be imported interactively from ndjson files under the menu item "Management → Stack Management → Saved Objects". The generated dashboard can now be imported into Kibana here.
It is best to select the option "Create new objects with random Ids" to avoid conflicts with Ids as far as possible.
Would you like examples?
Curious about the first application examples? You'll find them in a week's time in the next part of this blog post.
Of course, you can always contact Swissmakers for support in creating professional Kibana dashboards.
