We can extend #868 to create a universal widget that can map multiple-arrays of n-dimensional data to arbitrary graphics. Need to brainstorm to get this right.

One idea: add arrays in as "groups". For a given single "group of arrays", I think one way to organize things is that this "array group" should be added in together and they all map to the same Graphic type. The ImageWidget implementation in #868 is an example of how 1 group (representing images) can be represented. I do not think it makes sense to have multiple arrays within a group that map to different graphics, because then you can't define the graphical representation in a simple way.

For example, let's say you want display videos in a few subplots, scatters in a few subplots (representing tracking/keypoints for example), and polygons. Not every subplot will have an image, scatter, and polygons. But for a given graphical-representation, the arrays can be added in as a "group" if they should be managed together (still not entirely sure what this would look like).

If multiple arrays that map to the same type of graphic are to be added to the same subplot, they should be in separate groups, for example: multi-channel imaging data.

The current index displayed would of course be the same across all groups, and all arrays.

Another idea, gives way more flexibility: Allow adding n-dimensional arrays along with a specified graphical representation to arbitrary subplots, NDImageProcessor-like objects are created under the hood and automatically managed and used to generate the sliders and get/set indices to set the currently displayed slice. Maybe a crazy idea, but the implementation might be simpler without requiring specified "groups" of arrays, and it might allow for more customized viz.

Graphics we want to map to:

• Image
• Scatter
• Line (need to think about LOD)
• Heatmap (need to think about LOD and setting visible segments)
• Polygons or image overlays

More thoughts about lines:

• mode to display the entire line with a linear selector
• mode to display a chunk of full time series or other line-data as a window centered around the current timepoint
  • this would also be useful for display (x, y) behavior tracks, with a point selector that denotes the current timepoint and the line is only drawn for a certain window of time around the current time index.
• similar ideas for heatmap
• would also be useful for displaying neural trajectories

Thoughts about scatters:

• Would also be useful to display the scatter for all timepoints (or whatever dimension), and use a point selector tool that denotes the current index in that dimension.

Thoughts on LOD and out of core rendering:

• NDProcessor should have spatial LOD levels for all graphical representations
  • supply additional data arrays that can be loaded for specific extent ranges in the subplot.
  • auto-subsample data based on the extent of the currently visible space in the subplot and the number of screen pixels used. We could pre-define a certain number of levels that are used to auto subsample the data in space.
• Mode to display the "visible window of data", dynamically slice in space (the last 2 or 3 dims). This would be one form of out of core rendering for images, but also useful for lines and scatters. It corresponds to the "mode to display a chunk of time series data as a window centered around the current timepoint" for lines.
  • This mode and the ability to auto-slice in space should be managed in the backend by the NDProcessor. The corresponding mode in the widget would use canvas pointer events to move around the full data.
  • We can display a mini-map of the full data in the corner of the subplot, useful for all graphics.
  • I don't think this mode makes sense with scatters, if you want to view them as sub-sampled densities a heatmap should be used instead.

Some more potential features:

• ability to switch between graphic representations, for example an image vs a line-stack, useful for ephys data

The constructor of this class should probably pass **kwargs to Figure since this will deal with multiple types of graphics.

Once this is in a release, we could probably keep ImageWidget for 1 release and deprecate it, remove it in the following release.