Hurdle 1: Unaware of common Flux Packages and Utility Functions

Sometimes the largest hurdle that people face in life is simply not being aware of existing solutions. While Flux has a wide variety of Flux Packages and Utility Functions, there are a few that you can’t really live without if you plan to write Flux. If you’re not already familiar with the common Flux Packages and Utility Functions described below, I highly recommend taking advantage of them.

Solution 1: Some important Flux tips and Flux utilities to make writing Flux easier

Here are four Flux Packages and Utility Functions you should be aware of if you plan to write any Flux:

from(bucket:"items")    
|> range(start:-1h)
|> aggregateWindow(every: 1m, fn: mean())
multByX = (x) => (column, tables=<-) =>  tables  
|> mean(column:column)
|> map(fn: (r) => ({ r with _value: r._value * x}))
from(bucket: "my-bucket")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "cpu")
|> filter(fn: (r) => r["_field"] == "usage_system")
|> limit(n: 10)
|> aggregateWindow(every: v.windowPeriod, fn: multByX(x:2.0))

Hurdle 2: Unaware of performance gains that make your query faster

When crafting a Flux script, there are certain guidelines to follow in order to optimize your query for faster performance.

Solution 2: Learning about memory optimizations and new pushdown patterns to optimize your Flux scripts

In order to provide context for the optimization guidelines, let’s first take a moment to understand how Flux works. Flux is able to query data efficiently because some functions push down the data transformation workload to storage rather than performing the transformations in memory. Combinations of functions that do this work are called pushdown patterns.

  • |> count()
  • |> sum()
  • |> first()
  • |> last()
  • |> min()
  • |> max()
  • |> mean()
  • |>group() |> count()
  • |>group() |> sum()
  • …etc.
  • |>window() |> count()
  • |>window() |> sum()
  • …etc.
  • |>aggregateWindow(fn: count)
  • |>aggregateWindow(fn: sum)
  • …etc.

Hurdle 3: Using schema mutation functions at the wrong time

Users can accidentally decrease their query performance by applying schema mutation functions at the wrong place in their Flux query.

Solution 3: Learn when to apply schema mutations

Schema mutation functions are any functions that change the columns in your Flux tables. They include functions like keep(), drop(), rename(), duplicate(), and set(). If you can, it’s best to reserve these functions for the end of your Flux script. If you’re using an aggregates or selector function in your query, try to include the schema mutation functions after applying aggregation functions to preserve any pushdown patterns that you might have.

Hurdle 4: Unsure about task writing best practices

Sometimes users are unaware of how to best take advantage of the UI to craft tasks. Additionally, some users might not be aware of how to take advantage of variables to improve the efficiency of their tasks.

Solution 4: Task writing recommendations.

If you’ve read any of my other posts, you know that I truly believe the InfluxDB UI is your best friend when it comes to learning how to write Flux. Learning how to craft tasks is no exception. After you’ve crafted your task in the Data Explorer, use the Save As button in the top right corner to convert your Flux script to a task.

option task = {name: “rateTask”, every: 1h}rate = from(bucket: “items”)
|> range(start: -task.every)
|> filter(fn: (r) => (r._measurement == “totalItems”))
|> filter(fn: (r) => (r._field == “items”))
|> group(columns: [“itemGroupName”])
|> aggregateWindow(fn: sum, every: task.every)
|> map(fn: (r) => {
return: _time: r._time, _stop: r._stop, _start: r._start, _measurement: task.name, newTag: “rate”, _value: r._value, itemGroupName: itemGroupName,
}
})
rate
|> to(bucket: “newItems”)
Copy
from(bucket: “neItems”)
|> range(start: -task.every)
|> filter(fn: (r) => (r._measurement == “rateTask”)
|> filter(fn: (r) => (r.newTag == “rate”)
option task = {name: "Downsampling CPU", every: 1m}data = from(bucket: "my-bucket")
|> range(start: -task.every)
|> filter(fn: (r) => r._measurement == "my_measurement")
data
|> mean()
|> set(key: "agg_type",value: "mean_temp")
|> to(bucket: "downsampled", org: "my-org", tagColumns: ["agg_type"])
data
|> count()
|> set(key: "agg_type",value: “count_temp")
|> to(bucket: "downsampled", org: "my-org", tagColumns: ["agg_type"])

Hurdle 5: Unsure about when to write a downsampling task

Users are sometimes unsure about when they should write a downsampling task. They might have questions like: “How do I know when I need a downsampling task?” or “How do downsampling tasks affect my cardinality?”.

Solution 5: Learn about when to write a downsampling task

The solution to this hurdle is not simple. I will try to summarize it here, but I strongly encourage you to refer to Downsample and retain data and Downsampling with InfluxDB v2.0 for detailed explanations.There are two main reasons why users write downsampling tasks. First, you need to improve the visualization of your time series data and the performance of your dashboards. Consider aggregating your data to capture overall trends in your time series data to reduce noise in your data and increase your query/visualization performance. Second, you recognize that your data has unbounded tags and you might experience runaway series cardinality. In this case consider keeping the raw data with unbounded tags for a short period of time, by utilizing a short retention policy. Then use a downsampling task to remove the problem tag and aggregate your data as you see fit. You might be able to remove said tag with the drop() function for example. You can use the operational monitoring template to identify when you have a cardinality problem.

Next steps for tacking hurdles for intermediate Flux users

I hope this post helped you tackle some of the difficult Flux questions you might have. If you still have problems writing Flux or want advice on how to optimize your Flux scripts, please ask us for help and share your story! Share your thoughts, concerns, or questions in the comments section, on our community site, or in our Slack channel. We’d love to get your feedback and help you with any problems you run into! Your good questions are the inspiration for these posts.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Anais Dotis

Anais Dotis

Developer Advocate at InfluxData