Analyzers and initial instrumentation for pandas (Closes stefan-grafberger#6) (PR stefan-grafberger#18)

* Started backend introduction

* Started backend introduction

* Printing first 5 read_csv values working

* Alternative

* Analyzer with iterator and returning iterator

* more experimenting: now with pd concant and merge to join annoations, input and output data into one row

* Can create iterator, but is slow currently

* Start implementing iterator creation

* Fast iterator creation works, some tuple naming not correct yet

* Efficiently pass pandas rows to a test analyzer. There still are some bugs with identifying rows/annotations

* Extremely fast annotation propagation works for read_csv and dropna. Still have to introduce map for storing and deleting annotations for wir nodes as necessary. overall overhead of our complete library for adult_easy seems to be in the order of 30-60ms only

* Minor cleanup

* Added some todo comments

* Added some todo comments

* Added some todo comments

* Added some todo comments

* Added some todo comments

* Removed _x and _y leftovers in column names analyzers see after merging dfs in pandas backend

* Passing Pandas annotations without maps or similar by subclassing dataframe and series

* Update todos

* Code cleanup

* Started introducing analyzer interface

* Passing analyzers from outside to inspector and executor

* Some cleanup in pandas backend

* Introduce InspectionResult class

* renamed graph to dag in inspection result

* Test for print_first_rows_analyzer and cleanup

* Fixed bug with passing analyzers

* Improve PipelineExecutor annotation output

* Each analyzer now only sees its own annotations

* Renamed print_first_rows_analyzer to materialize_first_rows_analyzer

* Renamed print_first_rows_analyzer to materialize_first_rows_analyzer

* Update todo and minor cleanup

* Update todo

* Added test for backend annotation propagation

* Fixed bug resulting from storing state in static variables that only occured in tests

* Fix pylint duplicate codee warning

* Update readme

* Added test to see if everything works for multiple analyzers and fixed a bug discoverd by this test

* Added operatorType enum, introduced operator_context to analyzer input and added type hints

* Minor cleanup

* Use data classes for wir and dag node classes

* Introduce code_reference

* Renamed wir node attribute

* Cleaned up function_info extraction in pipeline_executor

* Move all backend-specific code into backends

* Pandas Backend now supports projections

* Very minor cleanup

* Removed leftover comment

* Update two docstrings

* Delete some leftover WIP code

Author: stefan-grafberger

README.md

@@ -36,13 +36,17 @@ Prerequisite: python >=  3.8
 Make it easy to analyze your pipeline and automatically check for common issues.
 ```python
 from mlinspect.pipeline_inspector import PipelineInspector
+from mlinspect.instrumentation.analyzers.materialize_first_rows_analyzer import MaterializeFirstRowsAnalyzer
 
 IPYNB_PATH = ...
 
-extracted_annotated_dag = PipelineInspector\
-        .on_jupyter_pipeline(IPYNB_PATH)\
-        .add_analyzer("test")\
+inspection_result = PipelineInspector \
+        .on_pipeline_from_ipynb_file(IPYNB_PATH) \
+        .add_analyzer(MaterializeFirstRowsAnalyzer(2)) \
         .execute()
+
+extracted_dag = inspection_result.dag
+analyzer_results = inspection_result.analyzer_to_annotations
 ```
 
 ## Notes

mlinspect/instrumentation/analyzers/__init__.py

@@ -0,0 +1,45 @@
+"""
+The Interface for the different instrumentation backends
+"""
+import abc
+from typing import Union, Iterable
+
+from mlinspect.instrumentation.analyzers.analyzer_input import OperatorContext, AnalyzerInputDataSource, \
+    AnalyzerInputUnaryOperator
+
+
+class Analyzer(metaclass=abc.ABCMeta):
+    """
+    The Interface for the different instrumentation backends
+    """
+
+    @property
+    @abc.abstractmethod
+    def analyzer_id(self):
+        """The id of the analyzer"""
+        raise NotImplementedError
+
+    @abc.abstractmethod
+    def visit_operator(self, operator_context: OperatorContext,
+                       row_iterator: Union[Iterable[AnalyzerInputDataSource], Iterable[AnalyzerInputUnaryOperator]])\
+            -> Iterable[any]:
+        """Visit an operator in the DAG"""
+        raise NotImplementedError
+
+    @abc.abstractmethod
+    def get_operator_annotation_after_visit(self) -> any:
+        """Get the output to be included in the DAG"""
+        raise NotImplementedError
+
+    def __eq__(self, other):
+        """Analyzers must implement equals"""
+        return (isinstance(other, self.__class__) and
+                self.analyzer_id == other.analyzer_id)
+
+    def __hash__(self):
+        """Analyzers must be hashable"""
+        return hash((self.__class__.__name__, self.analyzer_id))
+
+    def __repr__(self):
+        """Analyzers must be hashable"""
+        return "{}({})".format(self.__class__.__name__, self.analyzer_id)

mlinspect/instrumentation/analyzers/analyzer.py

@@ -0,0 +1,57 @@
+"""
+Data classes used as input for the analyzers
+"""
+import dataclasses
+from typing import Tuple
+
+from mlinspect.instrumentation.dag_node import OperatorType
+
+
+@dataclasses.dataclass(frozen=True)
+class AnalyzerInputRow:
+    """
+    A class we use to efficiently pass pandas/sklearn rows
+    """
+    values: list
+    fields: list
+
+    def get_index_of_column(self, column_name):
+        """
+        Get the values index for some column
+        """
+        if column_name in self.fields:
+            return self.fields.index(column_name)
+        return None
+
+    def get_value_by_column_index(self, index):
+        """
+        Get the value at some index
+        """
+        return self.values[index]
+
+
+@dataclasses.dataclass(frozen=True)
+class AnalyzerInputDataSource:
+    """
+    Wrapper class for the only operator without a parent: a Data Source
+    """
+    output: AnalyzerInputRow
+
+
+@dataclasses.dataclass(frozen=True)
+class AnalyzerInputUnaryOperator:
+    """
+    Wrapper class for the operators with one parent like Selections and Projections
+    """
+    input: AnalyzerInputRow
+    annotation: AnalyzerInputRow
+    output: AnalyzerInputRow
+
+
+@dataclasses.dataclass(frozen=True)
+class OperatorContext:
+    """
+    Additional context for the analyzer. Contains, most importantly, the operator type.
+    """
+    operator: OperatorType
+    function_info: Tuple[str, str]

mlinspect/instrumentation/analyzers/analyzer_input.py

@@ -0,0 +1,46 @@
+"""
+A simple example analyzer
+"""
+from typing import Union, Iterable
+
+from mlinspect.instrumentation.analyzers.analyzer_input import OperatorContext, AnalyzerInputDataSource, \
+    AnalyzerInputUnaryOperator
+from mlinspect.instrumentation.analyzers.analyzer import Analyzer
+
+
+class MaterializeFirstRowsAnalyzer(Analyzer):
+    """
+    A simple example analyzer
+    """
+
+    def __init__(self, row_count: int):
+        self.row_count = row_count
+        self._analyzer_id = self.row_count
+        self._operator_output = None
+
+    @property
+    def analyzer_id(self):
+        return self._analyzer_id
+
+    def visit_operator(self, operator_context: OperatorContext,
+                       row_iterator: Union[Iterable[AnalyzerInputDataSource], Iterable[AnalyzerInputUnaryOperator]])\
+            -> Iterable[any]:
+        """
+        Visit an operator
+        """
+        current_count = - 1
+        operator_output = []
+
+        for row in row_iterator:
+            current_count += 1
+            if current_count < self.row_count:
+                operator_output.append(row.output)
+            yield None
+
+        self._operator_output = operator_output
+
+    def get_operator_annotation_after_visit(self) -> any:
+        assert self._operator_output  # May only be called after the operator visit is finished
+        result = self._operator_output
+        self._operator_output = None
+        return result

mlinspect/instrumentation/analyzers/materialize_first_rows_analyzer.py

@@ -0,0 +1,13 @@
+"""
+Get all available backends
+"""
+from typing import List
+
+from mlinspect.instrumentation.backends.backend import Backend
+from mlinspect.instrumentation.backends.pandas_backend import PandasBackend
+from mlinspect.instrumentation.backends.sklearn_backend import SklearnBackend
+
+
+def get_all_backends() -> List[Backend]:
+    """Get the list of all currently available backends"""
+    return [PandasBackend(), SklearnBackend()]

mlinspect/instrumentation/backends/__init__.py

@@ -0,0 +1,66 @@
+"""
+The Interface for the different instrumentation backends
+"""
+import abc
+
+import networkx
+
+
+class Backend(metaclass=abc.ABCMeta):
+    """
+    The Interface for the different instrumentation backends
+    """
+
+    def __init__(self):
+        self.code_reference_to_description = {}
+        self.code_reference_analyzer_output_map = {}
+        self.analyzers = []
+
+    @property
+    @abc.abstractmethod
+    def prefix(self):
+        """The prefix of the module of the library the backend is for"""
+        raise NotImplementedError
+
+    @property
+    @abc.abstractmethod
+    def operator_map(self):
+        """The list of known operator mappings"""
+        raise NotImplementedError
+
+    @property
+    @abc.abstractmethod
+    def replacement_type_map(self):
+        """The list of used data type replacements"""
+        raise NotImplementedError
+
+    @staticmethod
+    @abc.abstractmethod
+    def preprocess_wir(wir: networkx.DiGraph) -> networkx.DiGraph:
+        """Preprocess the wir if necessary"""
+        raise NotImplementedError
+
+    @abc.abstractmethod
+    def before_call_used_value(self, function_info, subscript, call_code, value_code, value_value,
+                               code_reference):
+        """The value or module a function may be called on"""
+        # pylint: disable=too-many-arguments, unused-argument
+        raise NotImplementedError
+
+    @abc.abstractmethod
+    def before_call_used_args(self, function_info, subscript, call_code, args_code, code_reference, args_values):
+        """The arguments a function may be called with"""
+        # pylint: disable=too-many-arguments, unused-argument
+        raise NotImplementedError
+
+    @abc.abstractmethod
+    def before_call_used_kwargs(self, function_info, subscript, call_code, kwargs_code, code_reference, kwargs_values):
+        """The keyword arguments a function may be called with"""
+        # pylint: disable=too-many-arguments, unused-argument
+        raise NotImplementedError
+
+    @abc.abstractmethod
+    def after_call_used(self, function_info, subscript, call_code, return_value, code_reference):
+        """The return value of some function"""
+        # pylint: disable=too-many-arguments, unused-argument
+        raise NotImplementedError