Open Source Summit + Embedded Linux Conference North America 2026

PX4-based drones are powerful, but interacting with them typically requires specialized ground control software and trained operators. This talk presents a new interaction model: controlling and querying a PX4 drone using natural language from a standard mobile phone.

The system combines a Model Context Protocol (MCP)–style interface (inspired by ROS 2 MCP implementations) to expose PX4 capabilities as structured, machine-readable commands, with OpenAI’s real-time ChatGPT API to interpret user intent. A phone call or voice interaction—handled via Twilio—becomes the primary user interface, allowing operators to issue commands such as “take off to 10 meters,” “orbit that location,” or “what’s your battery state?” and receive immediate spoken feedback.

The talk will cover:

* How PX4 commands, telemetry, and state are exposed through an MCP-like abstraction
* Real-time bidirectional communication between phone, AI model, and drone using Twilio and RealTime API
* Safety considerations, command validation, and constraints when using AI-mediated control
* Practical use cases, fly missions, object detection all using hands free control
* Lessons learned, what worked and didn't work