Caesia Software logoCaesia Software

Introducing MineSim: a virtual Pilbara mine for testing autonomous haulage AI

Today I'm sharing MineSim, a new Caesia Software R&D project. It's a working simulation of Mt Whaleback, the open-cut iron ore mine at Newman in the Pilbara, built in Unreal Engine 5.8, with a fleet of autonomous haul trucks running the full haul cycle.

MineSim is the testbed for what comes next: using a video vision-language model (VLM) to work out why an autonomous truck has stopped.

Why build it

Autonomous haul trucks stop whenever their sensors pick up something they can't rule out: a rock on the road, spillage, a light vehicle, a person, a wall of dust. Each stoppage needs someone to look at it and decide what happens next, and the truck isn't hauling while it waits.

A video VLM could take the first look. But to know whether you can trust one, you need a lot of stoppage footage where you already know the answer, and real labelled footage is hard to come by. A simulator gives you that for free: every stoppage it creates comes with its ground truth.

What MineSim does today

  • A real pit, rebuilt from open data. Terrain from Copernicus GLO-30 elevation data, ground colour from Sentinel-2 satellite imagery. The 12 m benches, berms, in-pit ramps and waste dumps are reconstructed along the survey's own contours.
  • Autonomous haul trucks. 930E-class trucks (290 t payload) with Chaos vehicle physics drive themselves along a 1.4 km haul road, climbing 100 m out of the pit at 10%.
  • The full haul cycle. Each truck queues, reverses in beside a face shovel, takes three passes of about 97 t, hauls out, reverses onto the dump and tips.
  • A fleet. Three trucks share the load bay and dump pad, queue when a bay is taken, and keep four truck lengths apart on the road.
  • A perception camera. Every truck carries a forward camera modelled on a generic automotive one, with wide-angle lens distortion, motion blur, auto exposure, sensor noise and dust on the lens. It records repeatable 30 fps clips. This is what the VLM will see.
  • Matched against real photos. The look was tuned round by round against public photos of Whaleback. The video steps through each round and finishes with photo-versus-render comparisons.

The limits: it's one pit, the benches are reconstructed rather than surveyed, and the trucks don't have LiDAR or radar yet. It's a testbed, not a digital twin of a working mine.

Built in four days

MineSim went from an empty project to what you see in the video in four days and 43 commits. Most of the build was done by AI coding agents working from a written plan, one round at a time, with every round's renders checked before the next began. It's the same AI-native approach I use on client work.

What's next

  1. Virtual LiDAR and radar on the trucks, so they stop for obstacles the way real ones do, plus a rolling video buffer.
  2. A stoppage injector: rockfall, spillage, a light vehicle, a person, dust.
  3. VLM triage: send the clip leading up to each stoppage to a video VLM and get back a structured verdict for a triage dashboard, covering what it saw and what it recommends.
  4. Scoring: run hundreds of injected stoppages across weather and time of day, and measure each verdict against what was actually injected.

I'll share results here as each step lands.

If you work on autonomous haulage, mine technology or remote operations and this is a problem you're thinking about, I'd like to hear from you. Get in touch.

Credits

More postsGet in touch