Building the field app
AR Scan: a work in progress
What this page is
This is a build diary, not a product page. AR Scan is an experimental prototype that I have tested on two Samsung phones, an S9 and an S20. There is no app to download and nothing to sign up to yet. Everything here is described as it was tested, including what did not work.
Can different people return to the same piece of habitat?
Looking at habitat boulders, I started wondering how reports and assessments could build up over time.
Researchers monitoring a shore would know which boulders they were following. But someone visiting independently, such as a citizen scientist, a student or a community volunteer, might not know which ones to look at.
I imagined researchers mapping the boulders at the beginning and giving each one a permanent identity. A visitor could then use their phone to find the intended boulder and follow a survey designed by the research team. Each visit could add to the record of that particular piece of habitat.
That question led me to explore augmented reality. Could a phone recognise the surroundings and show people which boulder they were looking at?
My first experiments were in my driveway, drawing outlines around the grass-block pavers through my phone. They gave me somewhere close to home to begin testing the idea.
This page follows that work: the experiments, the setbacks, the useful milestones and the questions still to answer. The same approach could one day apply to natural boulders and rockpools, and to any habitat modules a future project installs. For now it is a way to test whether a phone can help people return to the same patch.
Where we are now
Reviewed 5 November 2026.
Working prototype
Android module mapping, saved-location recognition experiments, file-based sharing between phones, and local photo-set capture and export.
Demonstrated in particular tests
Two phones recognising the same saved location after fresh app starts, and a 3D reconstruction made on a computer from one phone photo set. Both were tests on a paved or home test area, not at the coast.
Still being developed
Dependable repeat visits in coastal conditions, shared site maps, observation submission and review, public visit history, and possible iNaturalist links.
First priority
Helping people return to the correct patch and contribute a useful observation record.
Participation
We are looking for conversations that could shape an early supported trial. There is no general public app programme open yet.
These are separate prototypes and tests, not one connected service.
Build journal, oldest first
Retrospective build notes
The early entries below are retrospective build notes. I have reconstructed them from my chats, test records and memory. Each shows the date of the experiment and, separately, the date it was written up. Later entries will be added as they happen.
-
From an outline to a remembered place
Drawing a boundary during one camera session was only the beginning. For the monitoring idea to work, someone needed to close the app, come back later and find the same patch. I started testing saved locations and return visits. Some attempts were encouraging; others didn't work. We needed to understand the difference.
What this showed
An outline that stays visible during one session is not proof that a saved location has been recognised again.
What it didn't
That return visits work.
-
Another person's phone
The original idea depended on someone else being able to use the map. Early sharing tests showed how many things had to work between saving a location on one phone and seeing its outline on another. On 16 October a location saved on the S20 and imported into the S9 first failed to match on the S9, even though the data had arrived intact. Saving the location to the cloud on the first phone, sending the file, importing it on the second phone, recognising the place and showing the outline turned out to be five separate steps, and each is a separate place for it to go wrong.
- Save the location to the cloud
- Send the file
- Import it
- Recognise the place
- Show the outline
Diagram. The five steps between saving a location on one phone and seeing its outline on another. Each step is a separate place for it to go wrong. What this showed
Where the chain broke.
What it didn't
That sharing worked yet.
-
Recordings helped explain the failures
Making screen recordings let us compare what I was seeing with what the app was actually doing. An unsuccessful scan could then teach us something, rather than simply ending with a timeout. One thing stood out: the app could show hundreds of tracked points, over 400 in one test, beside a "Low detail" message, which confused me.
What this showed
Recordings plus logs turn a failure into a diagnosis.
What it didn't
A fix for every failure.
-
Cross-phone milestones
A single test never proved this, so here are the steps in order.
- 11 October. I saved a new location on the S20, sent it to the S9, and the S9 recognised it and let me select it. Same day, same place.
- 12 October. A recording on the S9 showed the outline appearing once I reached the outdoor stone bed. The search time includes time spent indoors, away from the target. That location was saved as a low-detail test.
- 16 October. After force-stopping and restarting the app on both phones, the S20 and S9 each recognised the same shared location. On the final request the S20 took about 3.9 seconds and the S9 about 5.1. From its first attempt, including a restart of the search, the S9 took about 26 seconds. I would not say "it recognises in five seconds".
What this showed
Two phones can resolve the same saved location after fresh starts, in this test.
What it didn't
Reliability on a different day, in other light or wetness, on a similar neighbouring spot, or at the coast.
Technical note: timings from 16 October
S20: app start 16:08:32, final request 16:08:53.8, success 16:08:57.7. S9: app start 16:08:30.6, final request 16:08:56.0, success 16:09:01.2. Times are Pacific/Auckland. Durations refer to the final request, not the time from opening the app.
-
Making mapping usable
Mapping an irregular patch and selecting one repeated panel are different jobs. I built shape templates, editable patterns, line visibility and ways to select, and checked the menus on a phone. By design the pattern on screen is a visual guide only; the saved patch boundary is separate, so changing the pattern should not change a location I've already mapped. I haven't yet checked tile placement on a real surface. A displayed grid does not measure an uneven rock surface.
What this showed
Mapping and selecting needed separate tools.
What it didn't
That a grid measures anything accurately.
-
A confusing quality message
I asked why the app showed "Low detail" even while tracking hundreds of points. The number of tracked points is not the target; what helps a phone recognise a place is visual detail. I tried different viewpoints and a torch to understand the problem: in a phone test the torch switched on when the app reported too little light. These were experiments, not fixes, and I have not shown they help on wet rock.
What this showed
The status message and the cause needed explaining.
What it didn't
A way to scan wet rock reliably.
-
A real rockpool exposed an app bug
I took the app to a real rockpool, mapped two new outlines and finished both. Neither was saved for a return visit. When the app starts it searches for old saved places, and on a new shore each unrelated search could take up to 90 seconds, which kept the "Save AR location" button switched off. The message "All finished modules saved on this phone" only meant the outlines were saved on the phone, which is easy to mistake for saved for a return visit.
The fix in version 1.27 was partly in the app's behaviour (starting a new outline now stops the unrelated searches) and partly in saying more clearly what has and hasn't been saved.
What this showed
A field visit found a problem the driveway never would.
What it didn't
That saving now works on wet rock, glare or water. That needs a new field test.
-
What if someone has five minutes?
I also tried a separate web journey for a short visit: pick a rockpool from a layout, compare a reference photo, add a photograph, then finish or explore further. It is an interaction prototype; the submission is simulated and nothing is uploaded or stored.
Building it showed me that the layout, the identifiers, the reference photographs and the eventual visit record all have to agree.
What this showed
The parts of a visit need to line up.
What it didn't
A working observation system.
-
A whole habitat photo and individual observations
The plan is for one visit to hold two kinds of record: a photograph of the whole patch, and observations of individual living things within it. Possible iNaturalist links remain a plan; nothing is connected today.
ConceptOne visit record
Whole-patch photographThe same patch each visitIndividual observationsLiving things found within itPossible iNaturalist links: planned, not connected.
Concept diagram. How the records could fit together. Not a working feature, and it shows no real submissions or species. What this shows
How the records could fit together.
What it doesn't
Any working link to iNaturalist.
-
Exploring 3D records
On 2 November I took 25 photos on the S9 of a 60 mm blue-and-orange ball sitting on weathered timber, with paving, stones and plants around it. A computer, not the phone, turned them into a textured 3D model: about four minutes for the dense step, with a rough result. The ball's colours and rough shape come through, with gaps and a few untextured triangles. The ball-and-surroundings test is not a marine habitat.
It suggests another way to revisit a recorded surface on screen. It also raises questions about completeness, scale and whether measurements can be trusted. I gave the 60 mm size myself, and the scale of the model was set from it, so it hasn't been independently checked.
What this showed
Photos to textured model is possible on this data set.
What it didn't
Accuracy, repeatability, wet or underwater suitability, or any processing inside the app.
Roadmap and current questions
We have not set dates. Each stage waits for its gate.
| Stage | Intended outcome | Gate before moving on |
|---|---|---|
| Community discovery | Learn what local people and educators want to observe repeatedly. | One useful task and a group willing to help shape a trial. |
| Early method review | Agree locations, minimum records, permissions and who reviews records, with scientific input. | A coordinator can explain what makes a usable record. |
| Complete the core workflow | Approved site map, location confirmation with a fallback, photograph, durable visit saving and review. | The right place is identified and recovery is tested; a failed recognition never creates a falsely confirmed record. |
| Supported repeat visits | A small group uses the workflow several times. | Record usefulness, participant understanding, failures and review effort are assessed. |
| Research evaluation | Assess consistency, limitations and appropriate scientific uses. | Clear findings and a prioritised improvement plan. |
| Explore support for the next stage | Look for support for the next defined stage of delivery. | Costs, responsibilities and acceptance criteria are written down. |
3D feasibility work runs alongside the main workflow unless a scientific collaborator says it is the priority.
How we will judge usefulness
Wrong-location records, unusable photographs, lost or recovered contributions, time to complete, reviewer workload and repeat participation. Download counts and total photos are not enough.
Current questions
What is the smallest useful record? Who checks it? What should a visitor do when recognition fails? What would make a scientist trust a citizen record?
Help shape an early supported trial
I'd like to hear from people who could try the instructions, point out practical barriers, help choose questions worth asking and review how good the records are. There is no general public app programme open yet, and nothing to sign up to.
Discuss the monitoring approach
If you work in marine science or monitoring and have views on recording methods, I'd like to learn from them early.
Reviewed 5 November 2026. Prototype status can change; entries are added as work continues.