Module org.bigdataviewer.core
Package bdv.export

Class WriteSequenceToHdf5

  • public class WriteSequenceToHdf5
extends Object
    Create a hdf5 files containing image data from all views and all timepoints in a chunked, mipmaped representation.

    Every image is stored in multiple resolutions. The resolutions are described as int[] arrays defining multiple of original pixel size in every dimension. For example {1,1,1} is the original resolution, {4,4,2} is downsampled by factor 4 in X and Y and factor 2 in Z. Each resolution of the image is stored as a chunked three-dimensional array (each chunk corresponds to one cell of a CellImg when the data is loaded). The chunk sizes are defined by the subdivisions parameter which is an array of int[], one per resolution. Each int[] array describes the X,Y,Z chunk size for one resolution. For instance {32,32,8} says that the (downsampled) image is divided into 32x32x8 pixel blocks.

    For every mipmap level we have a (3D) int[] resolution array, so the full mipmap pyramid is specified by a nested int[][] array. Likewise, we have a (3D) int[] subdivions array for every mipmap level, so the full chunking of the full pyramid is specfied by a nested int[][] array.

    A data-set can be stored in a single hdf5 file or split across several hdf5 "partitions" with one master hdf5 linking into the partitions.

    Author:
    Tobias Pietzsch

    • Constructor Detail

      • WriteSequenceToHdf5

        public WriteSequenceToHdf5()

    • Method Detail

      • writeHdf5File

        public static void writeHdf5File​(mpicbg.spim.data.generic.sequence.AbstractSequenceDescription<?,​?,​?> seq,
                                 Map<Integer,​ExportMipmapInfo> perSetupMipmapInfo,
                                 boolean deflate,
                                 File hdf5File,
                                 ExportScalePyramid.LoopbackHeuristic loopbackHeuristic,
                                 ExportScalePyramid.AfterEachPlane afterEachPlane,
                                 int numCellCreatorThreads,
                                 ProgressWriter progressWriter)
        Create a hdf5 file containing image data from all views and all timepoints in a chunked, mipmaped representation.
        Parameters:
        seq - description of the sequence to be stored as hdf5. (The AbstractSequenceDescription contains the number of setups and timepoints as well as an BasicImgLoader that provides the image data, Registration information is not needed here, that will go into the accompanying xml).
        perSetupMipmapInfo - this maps from setup id to ExportMipmapInfo for that setup. The ExportMipmapInfo contains for each mipmap level, the subsampling factors and subdivision block sizes.
        deflate - whether to compress the data with the HDF5 DEFLATE filter.
        hdf5File - hdf5 file to which the image data is written.
        loopbackHeuristic - heuristic to decide whether to create each resolution level by reading pixels from the original image or by reading back a finer resolution level already written to the hdf5. may be null (in this case always use the original image).
        afterEachPlane - this is called after each "plane of chunks" is written, giving the opportunity to clear caches, etc.
        numCellCreatorThreads - The number of threads that will be instantiated to generate cell data. Must be at least 1. (In addition the cell creator threads there is one writer thread that saves the generated data to HDF5.)
        progressWriter - completion ratio and status output will be directed here.

      • writeHdf5File

        public static void writeHdf5File​(mpicbg.spim.data.generic.sequence.AbstractSequenceDescription<?,​?,​?> seq,
                                 int[][] resolutions,
                                 int[][] subdivisions,
                                 boolean deflate,
                                 File hdf5File,
                                 ExportScalePyramid.LoopbackHeuristic loopbackHeuristic,
                                 ExportScalePyramid.AfterEachPlane afterEachPlane,
                                 int numCellCreatorThreads,
                                 ProgressWriter progressWriter)
        Create a hdf5 file containing image data from all views and all timepoints in a chunked, mipmaped representation. This is the same as writeHdf5File(AbstractSequenceDescription, Map, boolean, File, LoopbackHeuristic, AfterEachPlane, int, ProgressWriter) except that only one set of supsampling factors and and subdivision blocksizes is given, which is used for all views.
        Parameters:
        seq - description of the sequence to be stored as hdf5. (The AbstractSequenceDescription contains the number of setups and timepoints as well as an BasicImgLoader that provides the image data, Registration information is not needed here, that will go into the accompanying xml).
        resolutions - this nested arrays contains per mipmap level, the subsampling factors.
        subdivisions - this nested arrays contains per mipmap level, the subdivision block sizes.
        deflate - whether to compress the data with the HDF5 DEFLATE filter.
        hdf5File - hdf5 file to which the image data is written.
        loopbackHeuristic - heuristic to decide whether to create each resolution level by reading pixels from the original image or by reading back a finer resolution level already written to the hdf5. may be null (in this case always use the original image).
        afterEachPlane - this is called after each "plane of chunks" is written, giving the opportunity to clear caches, etc.
        numCellCreatorThreads - The number of threads that will be instantiated to generate cell data. Must be at least 1. (In addition the cell creator threads there is one writer thread that saves the generated data to HDF5.)
        progressWriter - completion ratio and status output will be directed here.

      • writeHdf5PartitionLinkFile

        public static void writeHdf5PartitionLinkFile​(mpicbg.spim.data.generic.sequence.AbstractSequenceDescription<?,​?,​?> seq,
                                              Map<Integer,​ExportMipmapInfo> perSetupMipmapInfo)
        Create a hdf5 master file linking to image data from all views and all timepoints. This is the same as writeHdf5PartitionLinkFile(AbstractSequenceDescription, Map, ArrayList, File), except that the information about the partition files as well as the path of the master file to be written is obtained from the BasicImgLoader of the sequence, which must be a Hdf5ImageLoader.
        Parameters:
        seq - description of the sequence to be stored as hdf5. (The AbstractSequenceDescription contains the number of setups and timepoints as well as an BasicImgLoader that provides the image data, Registration information is not needed here, that will go into the accompanying xml).
        perSetupMipmapInfo - this maps from setup id to ExportMipmapInfo for that setup. The ExportMipmapInfo contains for each mipmap level, the subsampling factors and subdivision block sizes.

      • writeHdf5PartitionFile

        public static void writeHdf5PartitionFile​(mpicbg.spim.data.generic.sequence.AbstractSequenceDescription<?,​?,​?> seq,
                                          Map<Integer,​ExportMipmapInfo> perSetupMipmapInfo,
                                          boolean deflate,
                                          Partition partition,
                                          ExportScalePyramid.LoopbackHeuristic loopbackHeuristic,
                                          ExportScalePyramid.AfterEachPlane afterEachPlane,
                                          int numCellCreatorThreads,
                                          ProgressWriter progressWriter)
        Create a hdf5 partition file containing image data for a subset of views and timepoints in a chunked, mipmaped representation. Please note that the description of the full dataset must be given in the seq, perSetupResolutions, and perSetupSubdivisions parameters. Then only the part described by partition will be written.
        Parameters:
        seq - description of the sequence to be stored as hdf5. (The AbstractSequenceDescription contains the number of setups and timepoints as well as an BasicImgLoader that provides the image data, Registration information is not needed here, that will go into the accompanying xml).
        perSetupMipmapInfo - this maps from setup id to ExportMipmapInfo for that setup. The ExportMipmapInfo contains for each mipmap level, the subsampling factors and subdivision block sizes.
        deflate - whether to compress the data with the HDF5 DEFLATE filter.
        partition - which part of the dataset to write, and to which file.
        loopbackHeuristic - heuristic to decide whether to create each resolution level by reading pixels from the original image or by reading back a finer resolution level already written to the hdf5. may be null (in this case always use the original image).
        afterEachPlane - this is called after each "plane of chunks" is written, giving the opportunity to clear caches, etc.
        numCellCreatorThreads - The number of threads that will be instantiated to generate cell data. Must be at least 1. (In addition the cell creator threads there is one writer thread that saves the generated data to HDF5.)
        progressWriter - completion ratio and status output will be directed here.

      • writeViewToHdf5PartitionFile

        public static <T extends net.imglib2.type.numeric.RealType<T> & net.imglib2.type.NativeType<T>> void writeViewToHdf5PartitionFile​(net.imglib2.RandomAccessibleInterval<T> img,
                                                                                                                                  T type,
                                                                                                                                  int timepointIdPartition,
                                                                                                                                  int setupIdPartition,
                                                                                                                                  ExportMipmapInfo mipmapInfo,
                                                                                                                                  boolean writeMipmapInfo,
                                                                                                                                  boolean deflate,
                                                                                                                                  bdv.export.Hdf5BlockWriterThread writerQueue,
                                                                                                                                  ExecutorService executorService,
                                                                                                                                  int numThreads,
                                                                                                                                  ExportScalePyramid.LoopbackHeuristic loopbackHeuristic,
                                                                                                                                  ExportScalePyramid.AfterEachPlane afterEachPlane,
                                                                                                                                  ProgressWriter progressWriter)
        Write a single view to a hdf5 partition file, in a chunked, mipmaped representation. Note that the specified view must not already exist in the partition file!
        Parameters:
        img - the view to be written.
        type - pixel type
        timepointIdPartition - the timepoint id wrt the partition of the view to be written. The information in partition relates this to timepoint id in the full sequence.
        setupIdPartition - the setup id wrt the partition of the view to be written. The information in partition relates this to setup id in the full sequence.
        mipmapInfo - contains for each mipmap level of the setup, the subsampling factors and subdivision block sizes.
        writeMipmapInfo - whether to write mipmap description for the setup. must be done (at least) once for each setup in the partition.
        deflate - whether to compress the data with the HDF5 DEFLATE filter.
        writerQueue - block writing tasks are enqueued here.
        executorService - executor used for creating (possibly down-sampled) blocks of the view to be written.
        numThreads -
        loopbackHeuristic - heuristic to decide whether to create each resolution level by reading pixels from the original image or by reading back a finer resolution level already written to the hdf5. may be null (in this case always use the original image).
        afterEachPlane - this is called after each "plane of chunks" is written, giving the opportunity to clear caches, etc.
        progressWriter - completion ratio and status output will be directed here. may be null.