Edge computing is vital for IoT applications that require rapid, secure data processing, yet these applications are resource-heavy, and edge resources are often outmatched by cloud capabilities. Efficient resource utilization is crucial to fulfill application demands like latency, privacy, and cost. Static scheduling systems frequently fail to meet these needs in dynamic, hybrid IoT environments. This thesis details the design, implementation, and testing of a dynamic distributed scheduler tailored for edge environments. It orchestrates task execution across all available edge resources to meet the diverse requirements of IoT applications, including lightweight AI applications running in containers and resource-intensive DNN applications. Central to this scheduler is its ability to continuously monitor and adapt to the changing state of the IoT infrastructure, adjusting task scheduling based on real-time environmental and system conditions. The practicality of this scheduler is confirmed through a prototype that has been evaluated across various AI applications. To enhance the architecture further, various middleware layers have been evaluated to deepen our understanding and establish a foundation for future improvements. The research pioneers an efficient AI system designed to handle a variety of user requests at the edge.

This collaborative research effort extends previous studies, recognizing the contributions of fellow researchers and the wider academic community. We are thankful for the support provided by NSF grants.