Developments in Shoreline Aerial SurveyingHome

Stuart A Spray C.Eng MIEE | MD Aerial-PhotoCo Ltd

CONTENTS

ABSTRACT: Establishing the extent and proportion of live biomass and dead shell in estuary mussel beds has been carried out on foot typically during intertidal periods at low water springs (LWS). Clearly, such periods are of short duration and estimations of biomass vs shell have necessarily to be done on a sampling basis to cover sufficient ground. (fig1)

A quadrat in use for evaluating mussel quality
fig1 - A 'Quadrat' in use for evaluating Mussel quality

Given the limited time available and  patchy and non-uniform nature of the ground, there is potential for significant error. For larger beds, such foot mapping may need to extend over several tides, extending the project both in terms of time and cost. In a previous proof of concept project[10] to address these issues it was successfully demonstrated that a lightweight UAV had the potential to provide a photographic overview of an entire 3 Ha mussel bed without user intervention.

However, the technical limitations of the UAV (short flight times) and of the post processing photo-stitching techniques deployed at the time (Hugin)[11] suggested that further development was necessary to achieve an outcome which might usefully complement the data ordinarily collected by hand. Being a 2D process, no depth data was available and thus no direct assessment of biomass volume was possible.

In this new  project, using better equipment, more accurate flight data, longer flight times and significant image overlap it is intended to demonstrate the validity of 'Structure from Motion'[9] image processing techniques in real-world situations as a useful and practical means for extracting depth information and volume directly.

The Mission: To cover the planned survey area 2 missions were flown by a standard DJI Phantom 2V+ UAV equipped with the standard FC200 camera and Fisheye 5mm lens. Mission parameters are as shown in tbl 1 and as can be seen each was at different altitudes and number of tracks in order to meet area requirements and flight time limitations imposed by the platform. These parametric differences also provided the opportunity of validating the performance of the reconstructive process as evidenced by the the outputs exemplified by figs 7-9
Mission North South
Altitude(M) 50 40
Distance travelled (M) 1200 800
Track length(M) 190 120
Track spacing(M) 12.5 10
Numberof tracks 4 5
Area M2 950 600
Duration (Minutes) 15 10
Images 92 78
tbl1 - Flight Mission parameters

The UAV tracks for the 2 missions superimposed on a Google Earth image. Ordinarily the GPS parameters embedded in the image EXIF data would not be precise enough for accurate tracking so the data was acquired from a separate on-board flight data recorder.
 
Composite aerial photography mission
fig2

Two successive raw images taken along the track. Note the barrel distortion (an artifact of the fisheye lens) and low contrast due to very flat lighting and the natural characteristics of the sandy terrain. Feature detection and matching is rendered more difficult under these non-ideal conditions.
successive images showing barrel distortion
fig3
successive images showing barrel distortion
fig4

As above corrected for barrel distortion and contrast enhanced to improve feature detection and matching.
successive images showing barrel distortion
fig5
successive images showing barrel distortion
fig6


Structure from Motion: SfM[9] derives from Human's perception of 3D structures by their relative movement. This mechanism has been applied to photogrammetric 3D object reconstruction by identifying corresponding features such as corner points in and between overlapping 2-dimensional images as a camera moves through the environment.

The 3D reconstructive process implemented here is well known and originates from the work of ChangChang Wu[1] and Michal Jancosek[2] It is not the latest or fastest implementation but its binaries are freely available (although partly closed source), well documented and some useful explanatory material exists e.g. more recently[16][18][19]. Since its publication as a research paper it has been commercialised[8] with significant speed improvements.

'VisualSfM'[6] by ChangChang Wu and CMP/MVS[7] by Michal Jancosek. provide the essential programmes necessary for generating the camera poses, feature detection, sparse and detailed point clouds and various mesh reconstructions and imagery output. The feature detector is the Scale-Invariant Feature Tansform (SIFT).[4][5] published in 1999 and patented in the US by the University of British Columbia.[12] It exploits GPU multicore parallelism for feature detection, feature matching, and bundle adjustment[13] achieving processing time improvements orders of magnitude better than with a CPU alone.  

For this implementation a Dell XPS-8300 machine was used running Windows 10 equipped with 16Gb Ram, 4-core i7 3.4Ghz processor and NVIDEA GTX970 GPU (1664 cores). Even with this GPU support, typical processing times ran into hours so running without GPU support is unlikely to be a realistic option other than for the simplest of cases (few images).

Figs 7-8  show output from the first stage of the reconstructive process which are composite views of the missions showing the 170 camera positions (the coloured rectangles) superimposed over a sparse point cloud representing the ground. Note that the  positions and orientations (poses) of the cameras are entirely computed by 3D triangulation from matching features within the image content and not from attached  EXIF or GPS data. (tbl1).

The apparent curvature of the ground and camera planes is a consequence of incomplete or inaccurate lens distortion compensation. Fig9 is a ground projection of of pixel size (in effect the Ground Sampling Distance GSD) showing colour differences corresponding to the 2 missions which were flown at altitudes of 40M (left) and 50M (right).

;
Sparse point cloud and camera poses output from VisualSfM viewed from above.
3D reconstructive process initial output plan view
fig7

Sparse point cloud and camera poses output from VisualSfM viewed from the front. Note the residual curvature of ground and camera planes despite 'correction' applied to raw images.
3D reconstructive process initial output frontal view
fig8

False colour representation VisualSfM output mapping camera altitude/effective GSD.
fig9

A smoothly rendered part of the mesh. Note the colours of the points and mesh vertices are conveyed from the original image.
smooth
fig10

The same part of the mesh rendered with flat faces .  Colouring of the mesh faces is accomplished by interpolating the colours of the adjoining points.
point cloud close up rendered with flat faces
fig11

An Orthophoto derived from  the point cloud
orthophoto derived from complete mesh
fig12.

A perspective view of part of the 3D rendered mesh showing some depth features
perspective view of 3D rendered mesh
fig13

Reliably discriminating object height variations of, for example 1cm, at a camera range of 50M (1 part in 5000: a reasonable minimum requirement for this surveying application) represents a significant technical challenge given the likely sub-optimum light conditions, ground subject definition and contrast, camera stability, motion blur and lens distortions.

Process outputs from the VisualSfM and CMP/MVS achieved so far are encouraging but suggest that much further work is still required. It is anticipated that improvements in source image quality and alternative processing algorithms may yield better results.


Further application for SfM?
Estuarial Surveys carried out by the Devon and Severn Inshore Fisheries Conservation Authority (D&SIFCA)[20] monitor the distribution and quantity of crab tiles to ensure that it is in accordance with local byelaws and codes of conduct. This is carried out every 4 years and until 2016 was implemented by local volunteers during Low Water Springs inter tidal periods using hand-held GPS receivers and paper recording of Tile Field coordinates and other physical parameters.

In 2016 the D&SIFCA decided to do an aerial survey which I undertook to do some preliminary pre-contract work for. In considering some of the Photogrammetric implications of using drone imagery for estuarial surveys[21], it became evident that the manpower tradeoffs created two significant problems; the management of large numbers of images and the subsequent visual screening of of the images for evidence of tiling and the potential for feature duplication caused by image overlap.

A demonstration custom GUI application 'IMAGIS'[22] was created as a data management tool and interface to a Geographic Information System (GIS) but still needing the analysis to be carried out manually (mk1 eyeball) on an image by image basis. Given the significant survey area (about 350 Ha) and the several thousand images involved, photo-analysis demands a significant level of automation if the benefits of the Drone image acquisition are not to be squandered. Further research may yield a solution.


Crab tiles in the Exe Estuary - Images courtesy the Devon and Severn Inshore Fisheries Conservation Authority
fig14


Update
Since its original publication, this paper has been updated with some recent work on Sfm, in particular the use of 'Colmap'[23] by Johannes Schönberger.[24] Some initial results are presented here in the form of a sparse and dense reconstructions from an original set of images used for the VisualSfM project (as above) enhanced with a recently acquired set of images using a different camera, image resolution and lighting conditions. Figure 15 below shows in real time the reconstruction process with views of its sparse point cloud outputs at fig16/17 and the dense reconstruction at fig18, the image feature extraction and matching having already been carried out.

fig15 - the reconstruction process in realtime

fig16 - Sparse point cloud - plan view.
fig17 - Sparse point cloud - perspective view.

fig18 - Dense point cloud

Acknowledgements are made to ChanChang Wu for ‘VisualSFM’,  a GUI application for 3D reconstruction , Michal Jancosek for  the multi-view reconstruction software ‘CMPMVS’ , the designers of ‘Meshlab’ an open source, portable, and extensible system for the processing and editing of unstructured 3D triangular meshes and the open source 3D graphics and animation software ‘Blender’  without  all of which this project would not have been possible.


References (Hover/Click to follow):

[1] Dr Changchang Wu. Personal profile
[2] Dr Michal Jancosek:: Citations Google Scholar
[3] SIFT (Scale-invariant feature transform): Definition Wikipedia
[4] SIFT: David Lowe Citations Google Scholar
[5] SIFT: Advantages/Details University of Columbia
[6] VisualSfM: A Visual Structure from Motion System. Dr Changchang Wu
[7] CMPMVS: Multi-View Reconstruction Software. Dr Michal Jancosek
[8] 'RealityCapture': 3D reconstruction software CapturingReality.com.
[9] Structure from Motion: Definition Wikipedia
[10] Mussel bed survey: Proof of concept project Aerial-PhotoCo Ltd
[11] 'Hugin': A panoramic photo-stitcher Hugin
[12] University of British Columbia
[13] Multicore Bundle Adjustment: Processing Parallelism Dr Changchang Wu et. al.
[14] Meshlab: 3D Mesh Editor Meshlab.net
[15] Blender: 3D Modeller Blender.org
[16] Photogrammetry 3D Scanning hackaday.io
[17] Aerial Photogrammetry - some issues Aerial-PhotoCo Ltd
[18] Orthophotos using SfM (pdf)Kumpee Teeravech Asian Institute of Technology, Thailand
[19] Generating a Photogrammetric model Peter Falkingham academia.edu
[20] Exe Estuary Crab Tile Survey Exe Estuary management Partnership
[21] Aerial Photogrammetry - some issues Aerial-Photoco Ltd
[22] 'IMAGIS' - from Image to GIS Aerial-Photoco Ltd
[23] Colmap: a general-purpose SfM and MVS pipeline Johannes Schönberger et al
[24] Johannes Schönberger ETH Zurich

Useful relevant documents (Hover/Click to view/download):
  1. 2001_Sensor Fusion of Structure-From-Motion, Bathymetric 3D, and Beacon-Based Navigation Modalities.pdf
  2. 2002_Dealaunay ch5.pdf
  3. 2004_Comparing Four Methods of Correcting GPS Data.pdf
  4. 2009_patchmatch A Randomized Correspondence Algorithm for Structural Image Editing.pdf
  5. 2011_patchmatch_phd_thesis.pdf
  6. 2012_structure-from-motion-photogrammetry-a-low-cost-effective-tool-for-geoscience-applications.pdf
  7. 2013_Direct Georeferencing of Ultrahigh-Resolution UAV Imagery.pdf
  8. 2014_The rayCloud - A Vision Beyond the Point Cloud.pdf
  9. 2015 _AUTOMATED 3D ARCHITECTURE RECONSTRUCTION SIFT CMPMVS isprsarchives-XL-7-W3-1425-2015.pdf
  10. 2015_Superpixel Meshes for Fast Edge-Preserving Surface Reconstruction.pdf
  11. 2015_Integrating structure-from-motion photogrammetry with geospatial software as a novel technique for quantifying 3D ecological characteristics of coral reefs.pdf
  12. 2015_Structure from Motion (SfM) Photogrammetry.pdf
  13. 2015_Structure-from-Motion Photogrammetry Assessing Accuracy and Precision against Traditional Ground-Based Erosion Measurements.pdf
  14. 2015_The Application and Accuracy of Structure from Motion Computer Vision Models with Full-Scale Geotechnical Field Tests.pdf
  15. 2016_Assessment of Image-Based Point Cloud Products to Generate a Bare Earth Surface and Estimate Canopy Heights in a Woodland Ecosystem .pdf
  16. 2016_clarity-from-above PwC global report on the commercial applications of drone technology.pdf
  17. 2016_Wright - Robotic Augmentation of UAV Navigation in Cluttered Environments v1.pdf
  18. Anderson_2008_karen.anderson Fellowship report.pdf
  19. Anderson_2012_Croft 2012_On the use of RS techniques for monitoring spatio-temporal SOM dynamics.pdf
  20. Anderson_2013_Gaston_Frontiers_ecology_Lightweight unmanned aerial vehicles.pdf
  21. Anderson_2015_The use of Structure-from-Motion when quantifying subtle soil erosion processes from a small-scale experiment.pdf
  22. Anderson_2016_Drone-acquired structure-from-motion photogrammetry for high-precision measurements of biomass in semi-arid rangelands.pdf
  23. Horus_Guide_to_post-processing_of_the_point_cloud_EN.pdff
  24. Schoenberger_2014_STRUCTURE-FROM-MOTION FOR MAV IMAGE SEQUENCE ANALYSIS WITH PHOTOGRAMMETRIC APPLICATIONS.pdf
  25. Schonberger_2016_Structure-From-Motion_Revisited_CVPR_2016_paper.pdf
  26. Wu_2011_Towards Linear-time Incremental Structure from Motion.pdf
  27. Wu_2013_Critical Configurations For Radial Distortion Self-Calibration .pdf
Return to text Home