Workflow: Add new observations

Import audio data

Audio data winds up in several places:  a local workstation, redundant backups of yearly directories, and the RAID-0 archive.

  • Rotate recorder SD cards (biweekly)
  • Copy recorder files to yearly directories (back up continuously for 2 years, store older years on at least two external backup drives, store at least one backup drive offsite)
  • Copy/paste files from local directory into the corresponding archive directory.
  • TIP: The goal is to have MacOS skip duplicates but find gaps in previous transfers (an alternative is to use rsync from the command line). Take care to reconnect to the archive computer when mounting the network-shared Archive directory.  Otherwise, permissions may not be right.  Being able to use the “skip” existing files option during the copy operation is the goal.  Stop the process and reconnect to the drive if only the “replace” option is presented.

Run BirdNET–Analyzer to identify calls

Rerun the analysis of the current-year directory of the bioacoustic audio archive to create a selection.table file of ID/Results for each audio file that hasn’t already been analyzed

  • TIP: set field recorder sampling rate to 48k – avoids the need to resample audio, much shorter processing time
  • TIP: use the –skip_existing_results option to narrow the analysis to new audio only
  • TIP: use the –lat and –lon options to narrow the possible-species list to the location of the recorders
  • TIP: use the –threads option to manage the load on the computer
  • TIP: use the file and folder naming conventions listed below
  • CD into BirdNET-Analyzer script directory
cd /Users/mikeoconnor/Documents/BirdNET-Analyzer\ github/BirdNET-Analyzer
  • Launch the Python virtual environment defined during installation
source venv/bin/activate
  • Run the script – update paths and GPS coordinates
python3 -m birdnet_analyzer.analyze --threads 8 --lat 44.415947 --lon -91.700659 --skip_existing_results -o /Users/mikeoconnor/Documents/BirdWorkbench/birdnet_analyzer_RUN_FILES/run_files_2025 /Volumes/Bio/Bio\ 2025/From\ Recorders
  • Shut down the virtual environment
deactivate

Combine results into a single text file – one row per ID, one combined file per year

Combine the selection.table files to regenerate a run_files_COMBINED_20nn.txt  file with all the results for the year

  • TIP: use the file and folder naming conventions listed below
  • Use the BirdNET_run-file_combinerVN.sh script
  • Edit the Directory statements at the top of the script to point to input and output paths (see Naming Conventions below)
  • Run the script from the command line
sh /Users/mikeoconnor/Documents/BirdWorkbench/BirdWorkbench_BASH_scripts/BirdNET_run-file_combinerV3.sh
  • Note: V3 skips the “params” file generated by newer versions of BirdNET-Analyzer

Convert the merged selection.tables files into an .FST file for the visualizer program.

This refreshes the data table that the R UI loads each time a new session request comes in.   A number of columns are calculated from the selection-table data and local weather data is posted for each ID.

  • TIP: use the file and folder naming conventions listed below
  • Use the rscript combine_birdnet_runs_vN.R script
  • Edit the top of the script to add yearly variables and point to input path and output paths
  • Run the script from the command line
    • e.g. 
rscript /Users/mikeoconnor/Documents/BirdWorkbench/BirdWorkbench_R_Scripts/combine_birdnet_runs.R

Restart the server

This isn’t absolutely required.  The updated .FST table should be available to new sessions as soon as it’s written during that last step, but restarting just seems like a good idea to get everything all right and tight.

Directory and file naming conventions:

bioacoustic archive: contains yearly directories of bioacoustic recordings stored one level below (“From Recordings”) to match the structure that was in place when the project started.  Yearly directories were retained to allow annual backups to smaller drives.

bio/Bio 2019/From Recorders
bio/Bio 2020/From Recorders
...
bio/Bio 2025/From Recorders

birdnet_analyzer_RUN_FILES directory: contains yearly destination directories, for the BirdNET-Analyzer script, each contains 1 selection.table.txt file per ID for the year

run_files_2020
run_files_2021
...
run_files_2025

birdnet_analyzer_run_files_COMBINED directory: contains yearly destination directories, for the file-combiner shell script.  Each contains a single yearly run_files_COMBINED_202n.txt file that is updated throughout the year

run_files_COMBINED_2020
run_files_COMBINED_2021
...
run_files_COMBINED_2025

birdnet_analyzer_run_files_combined_FST directory:  the destination directory for the R script which creates the run_files_ combined_FST.fst file.  This is the data table used by the R script that powers the UI.

run_files_combined_FST.fst