Kaggle Pandas Cheat Sheet

Explore and run machine learning code with Kaggle Notebooks Using data from Kernel Files. The Most Comprehensive Cheat Sheet. This one is from the pandas guys, so it makes sense that. This cheat sheet is inspired by the data wrangling cheat sheets from RStudio and pandas. Examples are based on the Kaggle Titanic data set. Created by Tom Kwong, November 2020. V0.22 rev1 Page 1 / 2 Mutation: use sort! Mutation: use select! Mutation: use dropmissing!, allowmissing!, or disallowmissing! Nrow(df) ncol(df) Number of rows and columns.

This cheat sheet is a quick reference for data wrangling with Pandas, complete with code samples.

by Karlijn Willems

By now, you’ll already know the Pandas library is one of the most preferred tools for data manipulation and analysis, and you’ll have explored the fast, flexible, and expressive Pandas data structures, maybe with the help of DataCamp’s Pandas Basics cheat sheet.

Kaggle Pandas Cheat Sheet Printable

Yet, there is still much functionality that is built into this package to explore, especially when you get hands-on with the data: you’ll need to reshape or rearrange your data, iterate over DataFrames, visualize your data, and much more. And this might be even more difficult than “just” mastering the basics.

That’s why today’s post introduces a new, more advanced Pandas cheat sheet.

It’s a quick guide through the functionalities that Pandas can offer you when you get into more advanced data wrangling with Python.

(Do you want to learn more? Start our Pandas Foundations course for free now or try out our Pandas DataFrame tutorial! )

The Pandas cheat sheet will guide you through some more advanced indexing techniques, DataFrame iteration, handling missing values or duplicate data, grouping and combining data, data functionality, and data visualization.

In short, everything that you need to complete your data manipulation with Python!

Don’t miss out on our other cheat sheets for data science that cover Matplotlib, SciPy, Numpy, and the Python basics.

Sponsored by Datadog: pythonbytes.fm/datadog

Special guest: Vicki Boykis: @vboykis

Michael #1:clize: Turn functions into command-line interfaces

• via Marcelo
• Follow up from Typer on episode 164.
• Features
  • Create command-line interfaces by creating functions and passing them to [clize.run](https://clize.readthedocs.io/en/stable/api.html#clize.run).
  • Enjoy a CLI automatically created from your functions’ parameters.
  • Bring your users familiar --help messages generated from your docstrings.
  • Reuse functionality across multiple commands using decorators.
  • Extend Clize with new parameter behavior.
• I love how this is pure Python without its own API for the default case

Vicki #2:How to cheat at Kaggle AI contests

• Kaggle is a platform, now owned by Google, that allows data scientists to find data sets, learn data science, and participate in competitions
• Many people participate in Kaggle competitions to sharpen their data science/modeling skills
• Recently, a competition that was related to analyzing pet shelter data resulted in a huge controversy
• Petfinder.my is a platform that helps people find pets to rescue in Malaysia from shelters. In 2019, they announced a collaboration with Kaggle to create a machine learning predictor algorithm of which pets (worldwide) were more likely to be adopted based on the metadata of the descriptions on the site.
• The total prize offered was $25,000
• After several months, a contestant won. He was previously a Kaggle grandmaster, and won $10k.
• A volunteer, Benjamin Minixhofer, offered to put the algorithm in production, and when he did, he found that there was a huge discrepancy between first and second place
• Technical Aspects of the controversy:
  • The data they gave asked the contestants to predict the speed at which a pet would be adopted, from 1-5, and included input features like type of animal, breed, coloration, whether the animal was vaccinated, and adoption fee
  • The initial training set had 15k animals and the teams, after a couple months, were then given 4k animals that their algorithms had not seen before as a test of how accurate they were (common machine learning best practice).
  • In a Jupyter notebook Kernel on Kaggle, Minixhofer explains how the winning team cheated
  • First, they individually scraped Petfinder.my to find the answers for the 4k test data
  • Using md5, they created a hash for each unique pet, and looked up the score for each hash from the external dataset - there were 3500 overlaps
  • Did Pandas column manipulation to get at the hidden prediction variable for every 10th pet and replaces the prediction that should have been generated by the algorithm with the actual value
  • Using mostly: obfuscated functions, Pandas, and dictionaries, as well as MD5 hashes
• Fallout:
  • He was fired from H20.ai
  • Kaggle issued an apology

Michael #3: Configuring uWSGI for Production Deployment

• We run a lot of uWSGI backed services. I’ve spoken in-depth back on Talk Python 215: The software powering Talk Python courses and podcast about this.
• This is guidance from Bloomberg Engineering’s Structured Products Applications group
• We chose uWSGI as our host because of its performance and feature set. But, while powerful, uWSGI’s defaults are driven by backward compatibility and are not ideal for new deployments.
• There is also an official Things to Know doc.
• Unbit, the developer of uWSGI, has “decided to fix all of the bad defaults (especially for the Python plugin) in the 2.1 branch.” The 2.1 branch is not released yet.
• Warning, I had trouble with die-on-term and systemctl
• Settings I’m using:

Vicki #4: Thinc:A functional take on deep learning, compatible with Tensorflow, PyTorch, and MXNet

• A deep learning library that abstracts away some TF and Pytorch boilerplate, from Explosion
• Already runs under the covers in SpaCy, an NLP library used for deep learning
• type checking, particularly helpful for Tensors: PyTorchWrapper and TensorFlowWrapper classes and the intermingling of both
• Deep support for numpy structures and semantics
• Assumes you’re going to be using stochastic gradient descent
• And operates in batches
• Also cleans up the configuration and hyperparameters
• Mainly hopes to make it easier and more flexible to do matrix manipulations, using a codebase that already existed but was not customer-facing.
• Examples and code are all available in notebooks in the GitHub repo

Michael #5: pandas-vet

• via Jacob Deppen
• A plugin for Flake8 that checks pandas code
• Starting with pandas can be daunting.
• The usual internet help sites are littered with different ways to do the same thing and some features that the pandas docs themselves discourage live on in the API.
• Makes pandas a little more friendly for newcomers by taking some opinionated stances about pandas best practices.
• The idea to create a linter was sparked by Ania Kapuścińska's talk at PyCascades 2019, 'Lint your code responsibly!'

Vicki #6: NumPy beginner documentation

• NumPy is the backbone of numerical computing in Python: Pandas (which I mentioned before), scikit-learn, Tensorflow, and Pytorch, all lean heavily if not directly depend on its core concepts, which include matrix operations through a data structure known as a NumPy array (which is different than a Python list) - ndarray
• Anne Bonner wrote up new documentation for NumPy that introduces these fundamental concepts to beginners coming to both Python and scientific computing
• Before, you went directly to the section about arrays and had to search through it find what you wanted. The new guide, which is very nice, includes a step-by-step on how arrays work, how to reshape them, and illustrated guides on basic array operations.

Extras:

Vicki

Kaggle Pandas Cheat Sheet Pdf

• I write a newsletter, Normcore Tech, about all things tech that I’m not seeing covered in the mainstream tech media. I’ve written before about machine learning, data for NLP, Elon Musk memes, and Nginx.
• There’s a free version that goes out once a week and paid subscribers get access to one more newsletter per week, but really it’s more about the idea of supporting in-depth writing about tech. vicki.substack.com

Michael:

• pip 20.0 Released - Default to doing a user install (as if --user was passed) when the main site-packages directory is not writeable and user site-packages are enabled, cache wheels built from Git requirements, and more.
• Homebrew: brew install python@3.8

Joke:

An SEO expert walks into a bar, bars, pub, public house, Irish pub, tavern, bartender, beer, liquor, wine, alcohol, spirits...