Proposal for LFX Term-3 at Kube-Edge

by Ronak Raj

Mentor: Zimu Zheng, Shujing Hu

Parent Issue: #230

Comprehensive Example Restoration for Kube-Edge Ianvs

Table of Content

0. Raising the issue (10M)
1. Background (40M)
2. Goals (15M)
3. Scope (15M)
4.1 Architecture and Module (10M)
4.2 Module details (10M)

Background (40M)

I have select robot/lifelong_learning_bench/ sementic-segmenation
The Structure of robot/lifelong_learning_bench/

• sam_annotation- is for data preparation (converting annotations to segmentation masks).
• semantic_segmentation- for benchmarking (running experiments and evaluating results).

sam_annotation

This folder provides a utility script, convert_ls_to_trainid.py, to prepare data for training or benchmarking. It converts annotation data from Label Studio's JSON format into segmentation label images (Train_ids), ensuring user-provided data is in the correct format for the example's models.

`python convert_ls_to_trainid.py --dataset_dir <your_dataset_dir> --ls_json_name <your_labelstudio_export.json> --config_path <your_config.json>

This will generate segmentation masks in the train_ids folder.
If we have Label Studio annotations, first use sam_annotation to convert them, then use the output in semantic_segmentation.

So , my proposal will focus on examples\robot\lifelong_learning_bench\semantic-segmentation

What is Semantic-Segmentation ?

( ref: ianvs/docs/proposals/algorithms/lifelong-learning )
Semantic segmentation is an image segmentation algorithm or a crucial computer vision task whose goal is to assign a semantic category label to each pixel in the input image, thereby segmenting the image into regions with semantic information. This process divides the image into regions with meaningful class information, enabling detailed scene understanding.
Uses : autonomous driving, medical imaging, and robotics, where precise, pixel-level classification is required.
Purpose : This segmentation provides an automated method for data annotation, making the process more efficient and accurate.

RoadBlocks

1) Module not found error -

Even after installing the third-party dependencies for ianvs., we face the issue of certain "module not found " error for this example . Here are possible modules which need to be installed externally :

ModuleNotFoundError: No module named 'torch'
ModuleNotFoundError: No module named 'transformers'
ModuleNotFoundError: No module named 'tqdm'
ModuleNotFoundError: No module named 'prettytable'
ModuleNotFoundError: No module named 'sklearn'
ModuleNotFoundError: No module named 'tensorboard'
ModuleNotFoundError: No module named 'mmcv' , "mmdet","mmengine", "openmim", "mmsegmenatation" , "numpy",'charset_normalizer'.

Solution:

Either put all this in requirements.txt and run the installation command or install individually like this :

pip install torch transformers tqdm prettytable scikit-learn tensorboard mmengine numpy charset_normalizer ```

Some of these are special case modules .For them we need to run special commands with specified versions :
Install `openmim` first:  `pip install openmim`

Then use it to install the other packages:

```bash
mim install "mmcv==2.1.0"
mim install "mmdet==3.2.0"
mim install "mmsegmentation==1.0.0"

Justification:

These modules are critical because it directly impacts the project's usability and value.

2) Hard-coded paths :

Hard coded paths are in huge quantity across this example :

Files like testenv-all.yaml , testenv-city.yaml , testenv.yaml and apart from this some deep hidden hardcoded paths are found in rfnet/RFNet/eval.py.

Solution:

Change them with relative paths .

Justification:

This wastes the time of the developers who want to run this example , because hardcoded paths are one-user limited .

3) Problem with dataset download:

There is a doubt and troubleshooting regarding the datasets too .

To download the datasets, users can use this link :

download datasets in https://www.kaggle.com/datasets/kubeedgeianvs/cloud-robotics

While the README.md gives a link that takes to a Kube-edge website , then there is another link at the very bottom ,which will take us to the kaggle page . The really astonishing thing is the datasets are as big as 15 GB , which is going to take good time to install.

Solution:

We can have a clear direct link to the kaggledataset , user will download them , a proper update in README.md is needed which will instruct users on how to unzip the dataset file and place them in the dataset folder.

justification:

Ambiguous instructions for a large dataset directly lead to user frustration, wasted bandwidth, and significant time loss, undermining the project's reliability and acting as a major deterrent for new contributors.

This is how the dataset directory looks after unzipping it .

   ├── 1280x760
│   ├── gtFine
│   │   ├── test
│   │   ├── train
│   │   └── val
│   ├── rgb
│   │   ├── test
│   │   ├── train
│   │   └── val
│   └── viz
│       ├── test
│       ├── train
│       └── val
├── 2048x1024
│   ├── gtFine
│   │   ├── test
│   │   ├── train
│   │   └── val
│   ├── rgb
│   │   ├── test
│   │   ├── train
│   │   └── val
│   └── viz
│       ├── test
│       ├── train
│       └── val
└── 640x480
    ├── gtFine
    │   ├── test
    │   ├── train
    │   └── val
    ├── json
    │   ├── test
    │   ├── train
    │   └── val
    ├── rgb
    │   ├── test
    │   ├── train
    │   └── val
    └── viz
        ├── test
        ├── train
        └── val     
        

4) train_url and test_url s' path configuration?

It is confusing to understand the paths' configuration of test_url and train_url , they lead to path errors and one thing to take care of is -
Their are many testenv.yaml file in testenv folder , and all of them use train_url and test_url as constants , due to this a user may face :

RuntimeError: benchmarkingjob runs failed, error: prepare dataset failed, error: not one of train_index/train_data/train_data_info..

Solution:

All path should be turned to relative paths , so no user feels this issue Dataset path not found . Same wise , the path of workspace and datasets should be pre-fixed , so could people don't find trouble in it.
error: not one of train_index/train_data/train_data_info.. is a dataset index file problem. To deal this , a user can turn this constants to any of three , lets say for example
train_url turns to train_index and test_url turns to test_index.

justification:

The incorrect configuration keys makes the benchmark example impossible to run and invalidates its purpose as a portable and reproducible test case for the ianvs framework.

5) Configuration of file paths

I think all the configuration paths were set when the folder "sementic_segmentaion" did not exist. We can check that there number of files where path to accuracy.py , eval.py , sam_algorithm.yaml etc has a common path issue , which misses sementic_segmentation

Solution:

Add the sementic_segmenation in the paths . There can be another solution of removing this sementic _segmation folder . But I believe we should let it be there , it differentiates and separates types of files.

Justification:

These paths connect different algorithmic files together ,like benchmarkingjob.yaml uses the testenv.yaml and rfnet_algorithm-simple.yaml , the testenv.yaml usee the accuracy.py and so on with the configured path , wrong path break the loop and connection leading to failure of example.

6) Versions mismatch

There are few modules which are version specific , and if their versions are incorrect , they will mislead their dependency packages too like:
Note: Use of python 3.8 is must , python 3.12 will not support the version of modules
we need .

wrong version of pandas
wrong version of mmcv and mmdet

"/mnt/c/Users/ronak/OneDrive/Desktop/ianvs/core/cmd/benchmarking.py", line 41, in main
 raise RuntimeError(f"benchmarkingjob runs failed, error: {err}.") from error , failed, error: MMCV==2.2.0 is used but incompatible. Please install mmcv>=2.0.0rc4, <2.2.0...

Solution:

The correct version of pandas is pandas==1.1.5 and mmcv and mmdet are such modules which on each other version . I recommend to install openmim first and use it to install mmcv==2.1.0 and mmdet==3.2.0.

Justification:

Version mismatches create an unpredictable and non-reproducible environment, causing immediate runtime failures and placing an unreasonable troubleshooting burden on users

7) Sedna : version mismatch and jsonl support

The example fails to run due to a version mismatch with the sedna library, a key dependency for ianvs .

InportError:cannot import name 'JsonlDataParse' from 'sedna.datasources'(/hello/anaconda3/envs/ianvs/lib/python3.8/site-packages/sedna/datasources/ _init_.py)
RuntimeError( RuntimeError: (paradigm=lifelonglearning) pipeline runs failed, error: 'TaskDefinitionByOrigin' object has no attribute 'get'

Solution:

The solution requires replacing the standard sedna package with a modified version that includes the necessary jsonl support. This can be done by replacing the sedna file in the Python site-packages directory with the special sedna-with-jsonl.zip file provided in the ianvs repository at examples/resources/sedna-with-jsonl.zip. This is a temporary manual fix until the custom sedna code is officially merged into a formal release

Justification:

This is a fragile and non-reproducible method that can break a user's Python environment. The issue demonstrates a lack of a stable, public dependency, making the example unreliable and preventing users from correctly running the ianvs benchmarks.

8) What is failing and where

When I run the benchmarking command , it shows this output processing in the terminal which waits for a long time and then it fails with a :

(env) itsme_ronak2262@RONAK:/mnt/c/Users/ronak/OneDrive/Desktop/ianvs$ ianvs -f examples/robot/lifelong_learning_bench/semantic-segmentation/benchmarkingjob-simple.yaml 
The cache for model files in Transformers v4.22.0 has been updated. Migrating your old cache. This is a one-time only operation. You can interrupt this and resume the migration later on by calling `transformers.utils.move_cache()`.

0it [00:00, ?it/s]
un_classes:30
Upsample layer: in = 128, skip = 64, out = 128
Upsample layer: in = 128, skip = 128, out = 128
Upsample layer: in = 128, skip = 256, out = 128
128
.
.
.
RuntimeError: testcase(id=e139c552-2c87-11ef-b834-b42e99a3b90d) runs failed, error: (paradigm=lifelonglearning) pipeline runs failed, error: [Errno 2] No such file or directory: '/home/hsj/ianvs/project/cache.pickle'

We don't know which part of the code is passed and what is failing , which makes the debugging so difficult.

Solution :

We should replace the print statements with logger.info. As there are many files where print is used , creating a logger.py is suggested and then use logger import logger and replacing print with logger.info is a good idea , just take care , logger.info only accept single arguments .

Justification:

Relying on standard print statements provides no context or level of severity for a given message, making it impossible to determine the source of a failure or debug the execution flow when an error occurs in a large and complex framework.

Goals

1. Reproduce segment anything model based on open world segmentation dataset.
2. Achieve >0.45 accuracy for open-world object segmentation (e.g., AP, mIoU).
3. Enhance Portability: Remove all hardcoded and absolute paths, ensuring the example can be run correctly on any system without manual configuration changes.
4. Correct Dependencies: Update the example's requirements and documentation to ensure all necessary modules are correctly installed, resolving ModuleNotFoundError issues and version conflicts.
5. Improve Configuration Integrity: Fix incorrect keys in the testenv.yaml file, such as train_url to train_index, to align the example's configuration with the ianvs framework's API.
6. Establish a Robust Guide: Create a clear and accurate README.md that guides new users through the entire setup and execution process from start to finish, significantly lowering the barrier to entry and improving the user experience.
7. Yes , the major contribution of this proposal would be bug fix and enhancement of documents like README.md that blocks the working of this example/

Scope

Expected User: Me , Python developers , AI enthusiasts , AI researchers ,the broader Kube-Edge and ianvs community,

Scope:

1. The scope is to fix the robot/lifelong_learning_bench/semantic-segmentation example with Comprehensive Bug Fixes, Documentation Enhancement, Functionality Restoration.
2. This example of sementic _segmentation has two aspect , first is a basic lifelong_benchmarking and second is large SAM lifelong benchmarking which enlarges the scope of this project .We will have to fix and run both the models.

Comparission with other issues:

Detailed Design

Architecture:

This project will be based on Kube0-Edge Ianvs. Then, which ianvs module(s) would the proposed revision locate?

This is the architecture of the sam_annotation :

**What is the architecture of sementic_segmentaion ? **

The sementic_segmenation has two benchmark models:
1) simple-lifelong
2) large-SAM

Due to the presence of two models there are number files and modules in this example :

1. testenv folder :
   It has 5.yamlfiles related to testenv. For simple benchmarking or general use, start with testenv.yaml. For specialized experiments (e.g., city scenes, all datasets, robot-specific data), use the corresponding YAML file ( testenv-city.yaml, testenv-all.yaml etc.). The specific YAML file to use is referenced in your benchmarkingjob.yaml configuration.

2. testalgorithms folder: It has many differnet python files like basemodel files and many .yaml files like rfnet_algorithm and sam_algorithm .

• Python files implement different model architectures and utilities.
• YAML files configure which algorithm, task allocation, and task definition to use for benchmarking.

3. benchmarkingjob.yaml files: Two benchmarkingjob files exist , one for each model ,Configuration for running the simple and sam based benchmarking job .

Overview :

1. Dataset Preparation

• Datasets are organized into standard folders (train, val, test) with ground truth and RGB images.
• Index files (train_index.txt, test_index.txt) are generated for reproducibility.

2. Configuration

• Configuration files (testenv.yaml, algorithm.yaml, benchmarkingjob.yaml) define the dataset paths, algorithm details, and benchmarking parameters.

3. Algorithm Integration

• The example integrates semantic segmentation algorithms (e.g., RFNet).
• Optionally, it supports large vision models like SAM (Segment Anything Model) for advanced segmentation and annotation.

4. Benchmarking Workflow

• Ianvs orchestrates the benchmarking process:
• Loads datasets and algorithms as specified in the configs.
• Runs training and evaluation in a lifelong learning paradigm (i.e., models learn incrementally from new data/tasks).
• Supports both basic and cloud-edge collaborative workflows.

5. Output & Evaluation

• Results (metrics like accuracy, mIoU, etc.) are output to the console and saved in a workspace directory.
• Output includes detailed benchmarking logs and performance tables.

Workflow Architechture:

[Dataset] 
   ↓
[Configuration Files]
   ↓
[Algorithm (RFNet, SAM, etc.)]
   ↓
[Ianvs Benchmarking Engine]
   ↓
[Training & Evaluation (Lifelong Learning)]
   ↓
[Results & Output]

Module details

The revised and updated modules would be the

1. testenv
2. testalgorithms/rfnet
3. benchmarkingjob-sam.yaml
4. bechamarkingjob-simple.yaml
5. requiremets.txt
6. README.md file modules.

I will be adding new, specific paths in the files to these folders /modules to make my project a formal Ianvs benchmark. Here are the key considerations for each:

testenv Module :

Purpose:

This module defines the environment for the test case.
Revisions: I will need to add new revised paths to this module to define a unique simulation based on ianvs for all the files named testenv-all.yaml , testenv.yaml , testenv-city.yaml , testenv-robot-small.yaml , testenv-robot.yaml .

Justification:

Ianvs has a Test-Environment Manager that is specifically designed to handle the CRUD (Create, Read, Update, Delete) of test environments. By providing our new environment friendly relative paths , we are using the framework as intended, not altering it.

testalgorithms folder:

The algorithm files, such as rfnet_algorithm-simple.py , rfnet_algorithm.yaml, sam_algorithm.yaml , sam_vit_algoritm.yaml , vit_algorithm.yaml , will be reviewed to correct any internal path configurations. I will also integrate a logging utility (logger.info) to replace print statements, improving the clarity and debuggability of the execution process.

benchmarkingjob files:

The benchmarkingjob-simple.yaml and benchmarkingjob-sam.yaml files will be corrected to align with the framework's expected syntax and parameters.

requiremnts.txt:

All the external modules needs to added with their correct versions.

README.md :

a proper clear guide will made which will update the make the current README.md robust.

Do we need to change the core-code of Ianvs??

The sementic_segmentation example does not change the core code, we don't need to do that because changing the core code like the command line interface(CLI) , test-environment manager , test-case controller or the report generator is a bad idea , because they are the guts of the system . Here are few points which tells why this is a bad idea:

• It breaks the system: Modifying core code can introduce bugs and make the system unstable.
• It's not sustainable: My changes would be specific to this example and would not be useful to the broader community or broader sets of examples.
• It goes against the design: Ianvs is designed to be modular. This means we can plug in new "modules" ( project's environment and algorithm) without messing with the main system.

The PR #242 is to deal with this example.

Thank You