Pandas 2.0: A Game-Changer for Data Scientists? 1. Performance, Speed, and Memory-Efficiency 2. Arrow Data Types and Numpy Indices 3. Easier Handling of Missing Values 4. Copy-On-Write Optimization 5. Optional Dependencies Taking it for a spin! The Verdict: Performance, Flexibility, Interoperability! About me

-

Being built on top of numpy made it hard for pandas to handle missing values in a hassle-free, flexible way, since

As an illustration, , which isn’t ideal:

Missing Values: Conversion to drift. Snippet by Creator.

Note how points routinely changes from int64 to float64 after the introduction of a singleNone value.

, especially inside a data-centric AI paradigm.

Erroneous typesets directly impact data preparation decisions, cause incompatibilities between different chunks of information, and even when passing silently, they may compromise certain operations that output nonsensical ends in return.

For example, on the Data-Centric AI Community, we’re currenlty working on a project around synthetic data for data privacy. Considered one of the features, NOC (number of kids), has missing values and subsequently it’s routinely converted to float when the information is loaded. The, when passing the information right into a generative model as a float , we’d get output values as decimals corresponding to 2.5 — unless you’re a mathematician with 2 kids, a newborn, and a weird humorousness, having 2.5 children isn’t OK.

dtype = 'numpy_nullable', so we are able to keep our original data types (int64 on this case):

Leveraging ‘numpy_nullable’, pandas 2.0 can handle missing values without changing the unique data types. Snippet by Creator.

, but under the hood it signifies that now pandas can natively . This makes operations , since pandas doesn’t need to implement its own version for handling null values for every data type.

ASK ANA

What are your thoughts on this topic?
Let us know in the comments below.

1 COMMENT

0 0 votes
Article Rating
guest
1 Comment
Oldest
Newest Most Voted
Inline Feedbacks
View all comments

Share this article

Recent posts

1
0
Would love your thoughts, please comment.x
()
x