Landsat Earth for Orbiter (LEO)

martins

Orbiter Founder
Orbiter Founder
Joined
Mar 31, 2008
Messages
2,448
Reaction score
474
Points
83
Website
orbit.medphys.ucl.ac.uk
1. Introduction

As reported earlier, the next Orbiter release will include support for planet surface terrain. If you have missed it, here is the video I uploaded a while ago introducing a first glimpse of a preliminary terrain implementation:
Pretty good already, isn't it?


11. More things to try

At the moment, woodland areas look a bit dark and don't have much contrast. Apparently, this can be improved by mixing band 4 (near infrared) into the green band 2, because NIR has good contrast for vegetation. I want to try that next.

Also, I want to try histogram-matching the scenes against a global reference (the Visible Earth Blue Marble map that is the basis of the current Orbiter Earth texture). If I can colour-match the high-resolution Landsat maps against the VE map, then I could retain the latter for the lower resolution levels without a visible break when switching to high-resolution patches. Also, mapping against a global reference would ensure that the relative colour balance of the Landsat-generated map would remain consistent across large distances without drift or similar problems.


12. And finally, a plea for help

Writing the scripts for the image processing tasks described above took me about two weeks. But now the work only starts. So far, I have generated a single 16384 x 16384 tile covering 11.25 x 11.25 degrees (the central Europe tile used as an example in this article). The processing time for this tile took about 5 hours, not including the time for downloading the Landsat scenes (the tile is composed of about 100 scenes, each a download of about 300MB). My hard disk is filling up quickly, and I won't have the time, storage capacity and internet bandwidth to do the entire Earth. So I am asking you guys, as future beneficiaries of an improved Earth map, for help. Is anybody prepared to share in the processing work? If you are interested, this is what is required:

  1. A PC with 8GB RAM, running Windows-64 or Linux-64
  2. Matlab-64 (this is the killer, I suspect. I've written all processing scripts in Matlab. They are not very complex - if somebody wants to convert them to Python or similar, you are most welcome).
  3. Matlab must understand the geotiffread and geotiffwrite commands. This implies the Mapping toolbox, I think
  4. A utility to extract files from tar.gz files. Linux users are ok, Windows users can do it for example with 7-zip
  5. GDAL
  6. About 100GB HDD space during processing the scenes
  7. An internet connection that doesn't have problems with downloading ~30GB worth of Landsat data
  8. A way to upload the resulting map tile so I can pick it up.
Also (to cover all bases), by processing the scenes and sending me the result, you agree to pass the copyright to the tile to me. This is to avoid ending up with a map that has 500 different copyrights attached to it ;)

If there are any takers who fulfill all criteria and want to help, please let me know. I can then pass you my Matlab scripts and detailed informations. The more volunteers I can gather, the quicker we could end up with a global high-resolution map!
 
Give me something that can run it in Linux without MatLab, and I can dedicate... probably 6 cores on a VM on my workstation and 4 cores on my fileserver to it, at least for a while.

Edit: Note, I also have probably about 2TB I can dedicate on the fileserver too. I can pass on FTP credentials, and I'll just need to know when it's been "picked up."
 
Last edited:
Ah, so the dream is becoming true. Thank you so much ! :hail:
 
2. Matlab-64 (this is the killer, I suspect. I've written all processing scripts in Matlab. They are not very complex - if somebody wants to convert them to Python or similar, you are most welcome).
3. Matlab must understand the geotiffread and geotiffwrite commands. This implies the Mapping toolbox, I think

I think Matlab is indeed the killer here. I have a bit of experience in Python and could take a look at converting the Matlab scripts to it. I guess this would increase the amount of potential participants quite a bit.

There is certainly enough processor capacity to spare if the tools to access it are available to our community.

Looks fantastic, BTW! :thumbup:
 
Fantastic job ! I would be very glad to participate if only I can download 30 Gb on a run without making the whole Internet in my town explode.
But I thought of something (crazy yes, but what not?): since your script can run in Linux, why not use a "processing server" that will participate in the automated task of making the tiles? That could be a bit expensive, and is really crazy I think, but I also think this will be the fastest way to get our Blue Marble mapped for the greatest simulator of all life!
 
That's an astonishing improvement over the terrain we have now in Orbiter!

I'd be available to do some processing. The only thing is: what am I supposed to download/install regarding GDAL? I'm running 64-bit Windows.

Other than that, I'm all set.

EDIT: ah dammit, MATLAB. As others mentioned below, an Octave-compatible version of the scripts would make it possible for MANY more users to contribute.
 
Last edited:
Oh. My. GOD.
G5hhP.jpg
 
This looks terrific. I am curious about the terrain formatting though. Will it be possible to define custom bodies terrain as a greyscale bitmap, or will that need to be replaced by a new method?

Also, what is the FPS impact of terrain as you go higher up over the planet? (ie more terrain in frame to be rendered)
 
Thanks for the many offers of help! I really appreciate it. Give me a few more days to fix a few things with the code (I want to try the trick with the infrared band in the green channel). I'll then upload my Matlab codes, so it will become a bit clearer what I'm talking about. Then we can take it from there.
 
Looks fantastic. I don't have MATLAB, but I'd definitely offer my CPU time to help out if there was a non-MATLAB script available.
 
Face said:
Quote:
Originally Posted by martins View Post
2. Matlab-64 (this is the killer, I suspect. I've written all processing scripts in Matlab. They are not very complex - if somebody wants to convert them to Python or similar, you are most welcome).
3. Matlab must understand the geotiffread and geotiffwrite commands. This implies the Mapping toolbox, I think
I think Matlab is indeed the killer here. I have a bit of experience in Python and could take a look at converting the Matlab scripts to it. I guess this would increase the amount of potential participants quite a bit.

There is certainly enough processor capacity to spare if the tools to access it are available to our community.

Looks fantastic, BTW!

Looks fantastic. I don't have MATLAB, but I'd definitely offer my CPU time to help out if there was a non-MATLAB script available.

As above. As a 8-9 year user of Orbiter with no contribution other than the odd bug report and presence in the on-line community, I'd appreciate the opportunity to contribute a very small something back by processing. I would be dependent on non-MATLAB scripts, but can fulfil the other requirements.
 
Well, I'd also be glad to help if I knew how to. I don't fulfill all the requirements (4GB RAM only), and I have no idea of what Matlab or GDAL is, but it wouldn't worry me to let my CPU mix data all the night (or day). I have an optical fiber connection, though.

SolarLiner, maybe we could do something together, if you live near Toulouse. I could download the raw data for you, put it on a portable storage device (I don't have that though), then you mix the data, give it back to me and I finally can upload it ? Another thing I could suggest to you is to "infiltrate" the Paul Sabatier University of Science. Here they have really nasty DL/UL rates.

If an big online storage place was required to store and share the data, I'd be ok to contribute with several Euro-bucks. :tiphat:
 
Ditch MATLAB and I'm here to help!
Windows 8.1 x64
8GB DDR3 RAM
6-core AMD CPU
700GB HDD space free
200Mb/s down, 100Mb/s up

Regarding MATLAB, I feel like GNU Octave would be the easiest to convert to. I look at Octave as a free and open MATLAB.
 
Thanks for the many offers of help! I really appreciate it. Give me a few more days to fix a few things with the code (I want to try the trick with the infrared band in the green channel). I'll then upload my Matlab codes, so it will become a bit clearer what I'm talking about. Then we can take it from there.

Octave is supposed to be MATLAB compatible. If your code runs in Octave, I'll definitely give it a go.
 
Well, I'd also be glad to help if I knew how to. I don't fulfill all the requirements (4GB RAM only), and I have no idea of what Matlab or GDAL is, but it wouldn't worry me to let my CPU mix data all the night (or day). I have an optical fiber connection, though.

SolarLiner, maybe we could do something together, if you live near Toulouse. I could download the raw data for you, put it on a portable storage device (I don't have that though), then you mix the data, give it back to me and I finally can upload it ? Another thing I could suggest to you is to "infiltrate" the Paul Sabatier University of Science. Here they have really nasty DL/UL rates.

If an big online storage place was required to store and share the data, I'd be ok to contribute with several Euro-bucks. :tiphat:

Definitely. I have a 1Tb HDD but it's not empty, still it got 200 Gb left so that should be fine for processed data. I'm all for a team work N_Molson!
You can send me a 64 Gb USB drive drive with the data on it that I'll return to you with processed data so you can upload.
 
The problem is not porting to Scilab or Octave, but finding geotiffread and geotiffwrite for these alternatives.

---------- Post added at 09:30 AM ---------- Previous post was at 06:50 AM ----------

Another thought that comes into my mind is that you can download 30-days trial of Matlab, do the job on full speed, and then forget about it.
 
My laptop should fit to the specs and is a bit idle since I mostly use my Surface RT in the evening, but no Matlab here.

Wouldn't these calculations be a good application for CUDA/OpenCL? Or even JOCL?

http://jogamp.org/jocl/www/

My machine specs:

i7-3610QM CPU
16 GB of RAM
1 TB of HDD
128 GB of SSD
Geforce 680

If storage is a problem, I also have a NAS...
 
Last edited:
The problem is not porting to Scilab or Octave, but finding geotiffread and geotiffwrite for these alternatives.

A bit of googling shows open source versions of these functions for other languages. It should be feasible to at least script them, I would think.

And GeoTIFF specifications can also be found online if it helps.
 
A few updates:
  • I decided in the end not to mix the NIR band into the green channel. I tried a 80% band 2 / 20% band 4 mix, but the result didn't look very good. Instead I'll just brighten up the overall green channel at the lower end a bit.
  • I managed to avoid all geotiff-related function calls except for generating unblended mosaics, which is only required for outputting intermediate results. It's not required for the "production code" unless you are interested in the different stages, e.g. for debugging.
  • I am looking into compiling the matlab scripts into standalone code. This would require downloading and installing the Matlab runtimes (free) on your part. For now, I'll only try Linux to see if I can get it to work at all.
 
Back
Top