Upscaling Mars - Super resolution on images from NASA's Perseverance Rover

It is fantastic we are in a technological age where we can send a rover to Mars and send back such stunning photography as has been shared by NASA in the last month. What an amazing feat NASA, JPL and all involved have achieved with the Perseverance rover mission.

Throughout this current pandemic and the lockdowns it has given me time to pursue my Artificial Intelligence Computer Vision Deep Learning research further. I have been experimenting with improving on existing super resolution techniques and attempting to create an algorithm that works well with high resolution images and video.

Super resolution is having an algorithm that effectively imagines the detail that would be present if the media was higher resolution. The super resolution resulting in the images below is carried out by a model I have trained using a deep neural network architecture that I have created for this purpose. The intention was of my research and experiments was to create high resolution versions of films where none existed, similar to the remastering into 4K which is occurring with more and more older films that were never filmed in that quality. A few weeks ago I wondered about applying my research to the images and videos being returned from Mars and this is the result.

To me, upscaling images and video from Mars seems perhaps one of the most interesting and useful applications of super resolution. It is not possible to easily upgrade the cameras on a Mars rover and I'm sure data bandwidth is a consideration with the distances involved in transmission.

The quality of these upscaled images are is a very subjective question as there is no ground truth to compare them with and many of the common academic image quality metrics have been questioned as how they relate to human perceptions of quality. With super resolution of high resolution and high quality images I believe it only needs a relatively small improvement over the standard bicubic upscaling methods to really stand out.

This is all work in progress and I hope there will be further improvements, at which point I will write a more scientific and technical article to publish with the usual comparisons. I wanted to share some of my results especially with images from Mars being so topical. This is all achieved with an innovative deep neural network architecture trained on high resolution images using an innovative loss function.

Upscaling Mars images

This is a selection of recent images from NASA's Perseverance rover where the neural network model I have trained has attempted to infer improvements upon to increase their resolution. These images are a 2x enlargement up to either 3296x2400 or 2560x1920 pixel images depending on the source image, where the source images were half that size in each dimension. These images are best viewed on a QHD or 4K monitor.

Upscaling Mars video

These are a few early experiments in enhancing 4K (2160p) video and the super resolution of 4K (2160p) video to 8K (4320p) video. To the best of my knowledge little research has gone into upscaling from and to such high resolution video using deep neural networks. This is very much a work in progress and the source video frames are a complex subject matter to perform super resolution upon.

NASA's 4K version of "Perseverance Rover’s Descent and Touchdown on Mars" with super resolution to 8K, full video enhanced with variant 4 of my super resolution model

https://youtu.be/2rfkmYVMBb8

NASA's 4K version of "Perseverance Rover’s Descent and Touchdown on Mars" with super resolution to 8K, full video enhanced with variant 2 of my super resolution model

https://youtu.be/CPGaZpjLgQM