Python Modules to Process Polygons and HDF5 Fille Types

I would like to know if anyone has good suggestions on Python modules to average gridded/raster values that are contained within a polygon.

I would like to stay away from ArcPy.

I am also looking for good suggestions for post processes HDF5/netCDF files.