How to vectorize my code in pandas? it is long and inefficient

Question

I have this code that I wrote and it's taking too long to run. I was advised to vectorize this operation but so far I have found only multiplication examples. Here is my code:

my_dict = {}
for i in list(df.index):
    my_dict[i] = myClass(df.loc[i, 'name'])
    my_dict[i].class_method({'col1': df.loc[i, 'col1']})
    my_dict[i].class_method({'col2': df.loc[i, 'col2']})
    ...

and so on until 'col17'. Someone reviewed my code and said to 'use the fact that df is a dataframe and not loop through and don't use the expensive .loc() operation'

The only thing I could come up with is:

my_list = ['col1', 'col2', ..., 'col17']
my_dict = {}

for i in list(df.index):
    my_dict[i] = myClass(df.loc[i, 'name'])
        for col in my_list:
            my_dict[i].class_method({col: df.loc[i, col})

but this is not really vectorizing anything... are there any secret ways around pandas vectorization that I don't know about?

When you make an operation among columns you are going to execute it in all the rows. I'm not sure what you trying to do, but you could try to instead of iterating through the rows "for i in list(df.index)" iterate only through the columns. Could you give us more details? What the "class_method" is doing? — Gustavo Alves
– Gustavo Alves, Commented Jan 28 at 14:47
I agree with @GustavoAlves comments. Please include an explanation of what you are trying to do and we can possibly provide alternate means to accomplish the task. — itprorh66
– itprorh66, Commented Jan 28 at 15:07
what you are doing here, creating an instance of myClass from some value in the "name" column and then calling some method of that instance with various other values from the same row, isn't really vectorizable, but you could at least use something like df.itertuples() instead of for i in df.index: ... in combination with .loc. — juanpa.arrivillaga
– juanpa.arrivillaga, Commented Jan 28 at 16:31

Learning is a mess · Accepted Answer · 2025-01-28 17:00:52Z

0

.loc can be expensive as it needs to look up if you are passing a slice or an iterable over keys. Converting your dataframe to a dict of dict should bring faster lookups:

my_list = ['col1', 'col2', ..., 'col17']
my_dict = {}

for row_key, row in df.T.to_dict().items():
    my_dict[row_key] = myClass(row['name'])
        for col in my_list:
            my_dict[row_key].class_method({col: row[col})

answered Jan 28 at 17:00

Learning is a mess

8,3178 gold badges42 silver badges82 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to vectorize my code in pandas? it is long and inefficient

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related