• Home
  • Insight
  • Blog
  • Business
  • Entertainment
  • Health
  • Politics
  • Shop
    • Gift Shop
    • Value Shop
    • Store
    • Bargain Shop
    • Discount
  • Sports
  • Tech
  • Travel
  • USA
  • Video
  • World
    • Asia
    • Africa
    • South America
    • North America
    • Europe
    • Oceania
Wednesday, November 18, 2026
No Result
View All Result
Subscribe Now
  • Home
  • Insight
  • Blog
  • Business
  • Entertainment
  • Health
  • Politics
  • Shop
    • Gift Shop
    • Value Shop
    • Store
    • Bargain Shop
    • Discount
  • Sports
  • Tech
  • Travel
  • USA
    Mary Peltola Emerges Victorious in Alaska Senate Race, Making History for Democrats

    Mary Peltola Emerges Victorious in Alaska Senate Race, Making History for Democrats

    Stranded in Conflict: Palestinians Navigate the Tenuous Balance of War and Peace Amid Ceasefire Efforts

    Stranded in Conflict: Palestinians Navigate the Tenuous Balance of War and Peace Amid Ceasefire Efforts

    Trump Administration Revokes Pay Increase for Head Start Educators

    Trump Administration Revokes Pay Increase for Head Start Educators

    Unexpected Triumph: Gay Candidates Achieve Unforeseen Victories in Challenging Races

    Unexpected Triumph: Gay Candidates Achieve Unforeseen Victories in Challenging Races

    Microsoft Outlook Experiences Prolonged Outage Affecting Users Nationwide

    Microsoft Outlook Experiences Prolonged Outage Affecting Users Nationwide

    Gunfire Erupts in D.C.: Authorities Search for Suspect Amid Rising Concerns Over Public Safety

    Gunfire Erupts in D.C.: Authorities Search for Suspect Amid Rising Concerns Over Public Safety

    LAUSD Takes Strong Action Against Hate Speech Following Racial Slur Incidents

    LAUSD Takes Strong Action Against Hate Speech Following Racial Slur Incidents

    Beloved Former Los Angeles Times Columnist Steve Harvey Passes Away at 80

    Beloved Former Los Angeles Times Columnist Steve Harvey Passes Away at 80

    Exploring Diversity: ‘Ten Lives’ Showcases the Varied Economic Journeys of Americans

    Exploring Diversity: ‘Ten Lives’ Showcases the Varied Economic Journeys of Americans

  • Video
  • World
    • Asia
    • Africa
    • South America
    • North America
    • Europe
    • Oceania
The Insight Post
  • Home
  • Insight
  • Blog
  • Business
  • Entertainment
  • Health
  • Politics
  • Shop
    • Gift Shop
    • Value Shop
    • Store
    • Bargain Shop
    • Discount
  • Sports
  • Tech
  • Travel
  • USA
    Mary Peltola Emerges Victorious in Alaska Senate Race, Making History for Democrats

    Mary Peltola Emerges Victorious in Alaska Senate Race, Making History for Democrats

    Stranded in Conflict: Palestinians Navigate the Tenuous Balance of War and Peace Amid Ceasefire Efforts

    Stranded in Conflict: Palestinians Navigate the Tenuous Balance of War and Peace Amid Ceasefire Efforts

    Trump Administration Revokes Pay Increase for Head Start Educators

    Trump Administration Revokes Pay Increase for Head Start Educators

    Unexpected Triumph: Gay Candidates Achieve Unforeseen Victories in Challenging Races

    Unexpected Triumph: Gay Candidates Achieve Unforeseen Victories in Challenging Races

    Microsoft Outlook Experiences Prolonged Outage Affecting Users Nationwide

    Microsoft Outlook Experiences Prolonged Outage Affecting Users Nationwide

    Gunfire Erupts in D.C.: Authorities Search for Suspect Amid Rising Concerns Over Public Safety

    Gunfire Erupts in D.C.: Authorities Search for Suspect Amid Rising Concerns Over Public Safety

    LAUSD Takes Strong Action Against Hate Speech Following Racial Slur Incidents

    LAUSD Takes Strong Action Against Hate Speech Following Racial Slur Incidents

    Beloved Former Los Angeles Times Columnist Steve Harvey Passes Away at 80

    Beloved Former Los Angeles Times Columnist Steve Harvey Passes Away at 80

    Exploring Diversity: ‘Ten Lives’ Showcases the Varied Economic Journeys of Americans

    Exploring Diversity: ‘Ten Lives’ Showcases the Varied Economic Journeys of Americans

  • Video
  • World
    • Asia
    • Africa
    • South America
    • North America
    • Europe
    • Oceania
No Result
View All Result
No Result
View All Result
Home Mobile

Implementing Instant Object Detection in Live Image Streams using TFLite in Flutter

by Theinsightpost
August 25, 2023
in Mobile
0 0
0
Implementing Instant Object Detection in Live Image Streams using TFLite in Flutter

This article is written by Vadym Pinchuk

TL;DR

In Flutter’s early days in 2019, I developed a live object detection system for a major German company, despite the platform’s constraints. With release of Flutter 3.7 and advancements of TensorFlow have catalyzed the need to refine or overhaul this approach.

This article discusses the newest techniques in live-stream object detection as showcased in the flutter-tflite GitHub repository.

Intro

Back in 2019, during Flutter’s early rise, I undertook a project for a prominent German company: the challenge was to enable live object detection for ongoing operations. Despite the limitations of the Flutter framework at the time, I successfully devised a solution and detailed my findings in this article. This method provided adequate ‘live-stream’ object detection to meet the company’s needs.

However, the tech landscape evolves rapidly. With the introduction of Flutter 3.7, it became evident that refining or entirely revamping the existing logic was in order. This was especially true considering the advancements made by the TensorFlow team in the realm of object detection. This article delves into the most recent developments in live-stream object detection as incorporated into the flutter-tflite GitHub repository’s examples module.

Camera Stream Processing: Challenges and Solutions

The camera plugin effortlessly taps into hardware cameras, enabling real-time image streaming. But post-streaming, the question emerges: How to handle these data in the main process (root isolate)? Processing image frames is a resource-intensive task, especially when managed by the same process that handles the UI in Dart. Therefore, it’s ideal to perform this operation in the background.

Camera’s Output: Camera dispatches CameraImage Objects to the application. These images require meticulous processing to suit ML models, ensuring accurate object detection / classification. Furthermore, detection results should overlay seamlessly on the UI’s camera stream. Let’s break it down step by step.

Here’s a breakdown:

  • From CameraImage to usable Image: CameraImage conversion entails translating to an appropriate format based on the camera’s output. Typically, this involves byte reordering within a new matrix.
Future<image_lib.Image?> convertCameraImageToImage(CameraImage cameraImage) async {
  image_lib.Image image;

  if (cameraImage.format.group == ImageFormatGroup.yuv420) {
    image = convertYUV420ToImage(cameraImage);
  } else if (cameraImage.format.group == ImageFormatGroup.bgra8888) {
    image = convertBGRA8888ToImage(cameraImage);
  } else if (cameraImage.format.group == ImageFormatGroup.jpeg) {
    image = convertJPEGToImage(cameraImage);
  } else if (cameraImage.format.group == ImageFormatGroup.nv21) {
    image = convertNV21ToImage(cameraImage);
  } else {
    return null;
  }

  return image;
}
  • Rotation Handling: Camera plugin may offer rotated frames. Leveraging image Exif data offers an optimal solution for correct orientation.

  • Image Rescaling: For ML compatibility, images should be resized to match the model’s trained data dimensions, often a square.

    /// Resizing image for model, [mlModelInputSize, mlModelInputSize]
    final imageInput = image_lib.copyResize(
      image!,
      width: mlModelInputSize,
      height: mlModelInputSize,
    );
  • Inference Process: While object detection isn’t the most expensive operation, it still requires a significant amount of resources.
  /// Object detection main function
  List<List<Object>> _runInference(
    List<List<List<num>>> imageMatrix,
  ) {
    // Set input tensor [1, 300, 300, 3]
    final input = [imageMatrix];

    // Set output tensor
    // Locations: [1, 10, 4]
    // Classes: [1, 10],
    // Scores: [1, 10],
    // Number of detections: [1]
    final output = {
      0: [List<List<num>>.filled(10, List<num>.filled(4, 0))],
      1: [List<num>.filled(10, 0)],
      2: [List<num>.filled(10, 0)],
      3: [0.0],
    };

    _interpreter!.runForMultipleInputs([input], output);
    return output.values.toList();
  }
  • Detection Boxes Adjustment: Post-detection, these boxes should revert to their original preview dimensions, visually integrating with the image frame.
  /// Returns bounding box rectangle corresponding to the
  /// displayed image on screen
  ///
  /// This is the actual location where rectangle is rendered on
  /// the screen
  Rect get renderLocation {
    final double scaleX = ScreenParams.screenPreviewSize.width / 300;
    final double scaleY = ScreenParams.screenPreviewSize.height / 300;
    return Rect.fromLTWH(
      location.left * scaleX,
      location.top * scaleY,
      location.width * scaleX,
      location.height * scaleY,
    );
  }

Given the resource-intensiveness, the above operations ideally run in a background isolate. While it has always been feasible to shift tasks to a background isolate, Flutter 3.7 revolutionized it by enabling background access to plugins registered in root isolate. This approach, illustrated in Aaron Clarke’s article, is evident in the live-stream object detection example.

TFLite Configurations: Bridging Past and Present

Since my 2019 research, the landscape has shifted. While some plugins I relied on have been deprecated or not supported, TensorFlow’s acquisition of the flutter-tflite plugin has signaled progress towards Flutter MediaPipe.

  • Detector Setup: The detector preps the model and labels for the Interpreter’s object detection.
  static Future<Interpreter> _loadModel() async {
    final interpreterOptions = InterpreterOptions();

    // Use XNNPACK Delegate
    if (Platform.isAndroid) {
      interpreterOptions.addDelegate(XNNPackDelegate());
    }

    // Use Metal Delegate
    if (Platform.isIOS) {
      interpreterOptions.addDelegate(GpuDelegate());
    }

    return Interpreter.fromAsset(
      _modelPath,
      options: interpreterOptions..threads = 4,
    );
  }
  • Interpreter’s Role: The Interpreter analyses the input, based on the output matrix parameterized for each ML model.

Model-specific data configurations might vary. Although the Interpreter initializes with model and labels from assets in the main process/isolate, background access remains possible for image frame processing by accessing Interpreter with its address:

   _interpreter = Interpreter.fromAddress(command.args?[1] as int);

Isolates in Tandem: Collaborative Processing

//  _________________                         ________________________
//   [:root isolate]                             [:background isolate]
//  -----------------                         ------------------------
//         |                                              |
//         |<---------------(init)------------------------|
//         |----------------(init)----------------------->|
//         |<---------------(ready)---------------------->|
//         |                                              |
//         |----------------(detect)--------------------->|
//         |<---------------(busy)------------------------|
//         |<---------------(result)----------------------|
//         |                 . . .                        |
//         |----------------(detect)--------------------->|
//         |<---------------(busy)------------------------|
//         |<---------------(result)----------------------|

Image frame processing will be managed in a distinct background isolate, with access to the root Isolate’s Interpreter and plugins. As of Flutter 3.7, each background isolate can get access to the root isolate’s plugins with BackgroundIsolateBinaryMessenger.ensureInitialized(rootIsolateToken). This action configures the BinaryMessenger for background isolate communication with plugins.

To configure a background isolate, follow these straightforward steps:

  1. Spawn new isolate from the root, with the SendPort reference.
   final Isolate isolate = await Isolate.spawn(_DetectorServer._run, receivePort.sendPort);
  1. From the Background Isolate, relay an initialization message to the root isolate, including its sendPort.
    // receivePort.sendPort - used by root isolate to send commands to the service receiverPort
    sendPort.send(_Command(_Codes.init, args: [receivePort.sendPort]));
  1. From Root to Background send RootIsolateToken and _interpreter.address.
      case _Codes.init:
        _sendPort = command.args?[0] as SendPort;
        // Before using platform channels and plugins from background isolates 
        // we need to register it with its root isolate. This is achieved by
        // acquiring a [RootIsolateToken] which the background isolate uses to
        // invoke [BackgroundIsolateBinaryMessenger.ensureInitialized].
        // 
        RootIsolateToken rootIsolateToken = RootIsolateToken.instance!;
        _sendPort.send(_Command(_Codes.init, args: [
          rootIsolateToken,
          _interpreter.address,
          _labels,
        ]));
  1. Upon receipt in the background, capture the token and address. Subsequently, establish BinaryMessenger using the RootIsolateToken and access the Interpreter via the Address.
        // The [RootIsolateToken] is required for
        // [BackgroundIsolateBinaryMessenger.ensureInitialized] and must be
        // obtained on the root isolate and passed into the background isolate 
        // via a [SendPort].

        RootIsolateToken rootIsolateToken =
            command.args?[0] as RootIsolateToken;
 
        // [BackgroundIsolateBinaryMessenger.ensureInitialized] for each
        // background isolate that will use plugins. This sets up the
        // [BinaryMessenger] that the Platform Channels will communicate with 
        // on the background isolate.
 
        BackgroundIsolateBinaryMessenger.ensureInitialized(rootIsolateToken);
        _interpreter = Interpreter.fromAddress(command.args?[1] as int);
        _labels = command.args?[2] as List<String>;
        _sendPort.send(const _Command(_Codes.ready));

Configuration complete, the system stands operational and ready for live-stream frames processing. For an intricate understanding of Isolate collaborations, refer to Aaron Clarke’s article.

From here on, isolates can work together, processing camera frames in the background and showcasing results in the main UI isolate.

Summary

In the initial stages of Flutter, implementing real-time object detection was quite the challenge. Drawing from a personal experience in 2019, a rudimentary solution was developed, apt for the requirements at that point. However, advancements in Flutter 3.7 have revolutionized the approach.
This article delves into the modern methods of live-stream object detection, emphasizing the role of TensorFlow’s TFLite models and integrated features from Flutter 3.7. Discussed the intricacies of processing image frames, given the computational constraints it carries to the main isolate in Dart. With the inspiration from Aaron Clarke’s piece, highlighted the seamless integration of background isolates, especially the ability to access plugins registered in root isolates from Flutter 3.7 onwards. The article provides a step-by-step guide to setting up a background isolate, ensuring smooth and efficient object detection in real-time streams.

ShareTweetSend
Previous Post

Klaviyo files to go public in latest sign of life for IPO market

Next Post

Alabama wants to be the 1st state to execute a prisoner by making him breathe only nitrogen

Related News

Understanding the Recent Apple Outage: Causes and Implications for Users
Mobile

Understanding the Recent Apple Outage: Causes and Implications for Users

November 18, 2026
Unlocking Potential: Exploring the FDE Model in Enterprise AI Solutions
Mobile

Unlocking Potential: Exploring the FDE Model in Enterprise AI Solutions

November 14, 2026
Unlocking the Scoop: Upcoming Sale Dates, Price Highlights, and Previous Bargains
Mobile

Unlocking the Scoop: Upcoming Sale Dates, Price Highlights, and Previous Bargains

November 14, 2026
Unlocking Website Security: The Marketbyte Edge in Open Source Solutions
Mobile

Unlocking Website Security: The Marketbyte Edge in Open Source Solutions

November 13, 2026
Next Post
Alabama wants to be the 1st state to execute a prisoner by making him breathe only nitrogen

Alabama wants to be the 1st state to execute a prisoner by making him breathe only nitrogen

Discussion about this post

Subscribe To Our Newsletters

    Customer Support


    1251 Wilcrest Drive
    Houston, Texas
    77042 USA
    Call-832.795.1420
    e-mail – news@theinsightpost.com

    Subscribe To Our Newsletters

      Categories

      • Africa
      • Africa-East
      • African Sports
      • American Sports
      • Arts
      • Asia
      • Australia
      • Business
      • Business Asia
      • Business- Africa
      • Canada
      • Defense
      • Education
      • Egypt
      • Energy
      • Entertainment
      • Europe
      • European Soccer
      • Finance
      • Germany
      • Ghana
      • Health
      • Insight
      • International
      • Investing
      • Japan
      • Latest Headlines
      • Life & Living
      • Markets
      • Mobile
      • Movies
      • New Zealand
      • Nigeria
      • Politics
      • Scholarships
      • Science
      • South Africa
      • South America
      • Sports
      • Tech
      • Travel
      • Travel-Africa
      • UK
      • USA
      • Weather
      • World
      No Result
      View All Result

      Recent News

      Top 7 Retirement Threats and Effective Strategies to Navigate Them

      Top 7 Retirement Threats and Effective Strategies to Navigate Them

      November 18, 2026
      Charting the Future: Unveiling the TEFCA 2026-2027 Workplan and Innovative Monthly Metrics

      Charting the Future: Unveiling the TEFCA 2026-2027 Workplan and Innovative Monthly Metrics

      November 18, 2026
      Inevitable Buyer’s Remorse Looms Over Midterm Elections

      Inevitable Buyer’s Remorse Looms Over Midterm Elections

      November 18, 2026
      Understanding the Recent Apple Outage: Causes and Implications for Users

      Understanding the Recent Apple Outage: Causes and Implications for Users

      November 18, 2026
      • Home
      • Advertise With Us
      • About Us
      • Corporate
      • Consumer Rewards
      • Forum
      • Privacy Policy
      • Social Trends

      Theinsightpost ©2026 | All Rights Reserved. Theinsightpost is an Elnegy LLC company, registered in Texas, USA

      Welcome Back!

      Login to your account below

      Forgotten Password?

      Retrieve your password

      Please enter your username or email address to reset your password.

      Log In

      Add New Playlist

      We are using cookies to give you the best experience on our website.

      You can find out more about which cookies we are using or switch them off in .

      No Result
      View All Result
      • Home
      • Insight
      • Blog
      • Business
      • Entertainment
      • Health
      • Politics
      • Shop
        • Gift Shop
        • Value Shop
        • Store
        • Bargain Shop
        • Discount
      • Sports
      • Tech
      • Travel
      • USA
      • Video
      • World
        • Asia
        • Africa
        • South America
        • North America
        • Europe
        • Oceania

      Theinsightpost ©2026 | All Rights Reserved. Theinsightpost is an Elnegy LLC company, registered in Texas, USA

      The Insight Post
      Powered by  GDPR Cookie Compliance
      Privacy Overview

      This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.

      Strictly Necessary Cookies

      Strictly Necessary Cookie should be enabled at all times so that we can save your preferences for cookie settings.

      Cookie Policy

      More information about our Cookie Policy