Some asteroids in the Asteroids dimension have outer layers formed of Dense Ice. It has a translucent blue appearance similar to regular ice in Minecraft. It will only melt when there's a very bright light source near by (e.g. glowstone not glowstone torch).
Hollow asteroids always have an outer layer formed of Dense Ice. But not all ice asteroids are hollow: a little over half of them are not hollow.
Dense Ice can be turned into ice in the compressor (same way as producing compressed iron for exemple).