Barcode, OpenCV and QML (Qt 6)

In this post, I would like to share how to build a barcode scanner (for Android and iOS) in QML using OpenCV.

As a first step, we need to build the OpenCV library or download it from here: https://github.com/niqt/ocvdecoder (the readme discusses QR code scanning, but the library can also be used for barcode scanning).

To reach the goal we have to do:

• Create the C++ class
• Create the QML Item
• Use the QML component

C++

The logic involves receiving frames from the camera and decoding them until the barcode is recognized. These frames are captured by videoSync, which is the object responsible for obtaining frames from the camera.

/*
 * This class implement a qml wrapper for the OpenCV bar code feature
 * */

class Bardecoder : public QObject
{
    Q_OBJECT
    QML_ELEMENT
    Q_PROPERTY(QObject* videoSink WRITE setVideoSink) // the video sync used to get the frames
    Q_PROPERTY(bool run READ run WRITE setRun) // start/top the decode
public:
    Bardecoder(QObject *parent = nullptr);
    void setVideoSink(QObject *videoSink);
    bool isDecoding() {return m_decoding; } // the status of the decoding - decondig in progress
    void setRun(bool run);
    bool run() {return m_run;}
public slots:
    void setFrame(const QVideoFrame &frame);


signals:
    void videoSyncChanged();
    void decoded(const QString &code);

private:
    QVideoSink *m_videoSink;
    cv::Ptr<cv::barcode::BarcodeDetector> m_bardet;
    QFuture<void> m_processThread;
    bool m_decoding;
    bool m_run;
};

The Bardecoder class is a QML_ELEMENT. It has two properties: videoSync (to set the videoSync) and run to initiate the decoding process.

The setFrame slot is invoked when videoSync captures a frame from the camera.

In the private section of the class, the key components are m_barded (the OpenCV barcode decoder) and m_processThread (the thread that executes the decoding action).

In the cpp part we have:

Bardecoder::Bardecoder(QObject *parent)
    : QObject{parent}
{
    m_bardet = makePtr<barcode::BarcodeDetector>("", ""); // create the bardetector
}

void Bardecoder::setVideoSink(QObject *videoSink)
{
    m_videoSink = qobject_cast<QVideoSink*>(videoSink);

    if (m_videoSink) {
        QObject::connect(m_videoSink, &QVideoSink::videoFrameChanged, // when a frame is received call setFrame
                         this, &Bardecoder::setFrame);
    }
}

void Bardecoder::setFrame(const QVideoFrame &frame)
{
    if(m_run && !isDecoding() && m_processThread.isFinished()) { // if start decode and is not decoding and the previous is finished => decode the frame
        m_decoding = true;
        QImage image = frame.toImage().convertToFormat(QImage::Format_RGB32).rgbSwapped(); // convert the frame to an imageso can be used from opecv

        m_processThread = QtConcurrent::run([=]() {
            if(image.isNull()) {
                m_decoding = false;
                return;
            }
            cv::Mat img(image.height(), image.width(), CV_8UC4, (void *) image.bits(), image.bytesPerLine());
            vector<Mat> points;

            vector<Point> corners;
            vector<cv::String> decode_info;
            vector<string> decode_type;
            m_bardet->detectMulti(img, corners);
            m_bardet->detectAndDecodeWithType(img, decode_info, decode_type, corners);
            if (decode_info.size() > 0) {
                auto code = QString(decode_info.at(0).c_str());
                if (code.length() > 0) // if the barcode is decoded emit the signal
                    emit decoded(QString(decode_info.at(0).c_str()));
            }
            m_decoding = false;
        });
    }
}

void Bardecoder::setRun(bool run)
{
    m_run = run;
}

The two main components are setVideoSync, which sets the slot, and setFrame. When a frame changes, setVideoSync triggers setFrame.

Within setFrame, the decoding action occurs. This process begins if run is true (where run functions as a start command). The decoding action starts only if there is no other frame currently being decoded and if the previous decoding thread has finished. Before using the OpenCV decoder functions, it’s necessary to transform the frame into an image in a specific format. Once this transformation is complete, the OpenCV functions are called as detailed in the OpenCV documentation. If the barcode is successfully decoded, a signal is emitted.

QML

Create the BarReader QML component:

import QtQuick
import Felgo
import QtQuick.Controls 2.0
import QtQuick.Layouts 1.1
import QtMultimedia
import bardecoder

/*
  This element contains the logic and the components for the Barcode reader.
  It use the camera, the CaputureSession to capture the video from the camera and it sent
  to VideoOutput to be shown. The decoder get the video frames using the videoSync from the VideoOutput.
  The signal barCaptured is emitted if the decoder found a barcode.
  */

Item {

    id: item

    // This property enable/disable the decoding action
    property alias run: decoder.run

    // This signal is emitted when the barcode is found
    signal barCaptured(string barcode)

    Camera // The camera
    {
        id:camera
        active: run
        focusMode: Camera.FocusModeAuto
    }

    CaptureSession {
        camera: camera
        videoOutput: videoOutput
    }

    VideoOutput
    {
        id: videoOutput
        anchors.fill: parent

        // The follow properties are used to scale the red capture area on the camera video
        property double captureRectStartFactorX: 0.25
        property double captureRectStartFactorY: 0.25
        property double captureRectFactorWidth: 0.5
        property double captureRectFactorHeight: 0.5

        MouseArea { // Clicking remove the autofocus
            anchors.fill: parent
            onClicked: {
                camera.customFocusPoint = Qt.point(mouseX / width,  mouseY / height);
                camera.focusMode = Camera.FocusModeManual;
            }
        }

        Rectangle { // The capture zone
            id: captureZone
            color: "red"
            opacity: 0.2
            width: parent.width * parent.captureRectFactorWidth
            height: parent.height * parent.captureRectFactorHeight
            x: parent.width * parent.captureRectStartFactorX
            y: parent.height * parent.captureRectStartFactorY

        }
    }

    Bardecoder { // The barcode decoder
        id: decoder
        videoSink: videoOutput.videoSink // in this way capture the video frames

        onDecoded: code => {
                       console.log("QR " + code)
                       run = false // stop to decode
                       barCaptured(code) // send the value
                   }
    }
}

The logic behind this process is detailed in the code comments. The only aspect not specified in these comments is that we use a pink (red with opacity)rectangle to define the area of the camera used for decoding.

Now, we can integrate it into our page in the following way:

BarReader {
        id: isbnReader
        visible: false
        run: false
        anchors.fill: parent
        onBarCaptured: barcode => {
            visible = false // hide itself
            run = false // stop to scanning
        }

    }

A working example of this project can be found here: https://github.com/niqt/felgobar. Although it uses Felgo, the same principles apply to pure Qt as well.