Python Modules to Process Polygons and HDF5 Fille Types

I would like to know if anyone has good suggestions on Python modules to average gridded/raster values that are contained within a polygon. I would like to stay away from ArcPy. I am also looking for good suggestions for post processes HDF5/netCDF files.