Pandas Series, DataFrames, and Operations

Here is a sales file: a region name, a date, a product, a price and a quantity. You want the average revenue per region. With plain arrays you would have to hold the whole thing in your head — region is column 0, price is column 3, quantity is column 4 — and arrays cannot even store the text and the numbers together, so you would need one array for the strings and another for the numbers, kept in the same order by hand.

1# The version nobody wants to maintain2regions = ["north", "south", "north", "east"]3prices  = np.array([10.0, 12.5, 10.0, 8.0])4qty     = np.array([3, 1, 5, 2])56totals = {}7for i, r in enumerate(regions):8    totals.setdefault(r, []).append(prices[i] * qty[i])

One reordering of any array and the whole thing silently produces wrong answers. Pandas fixes this by attaching labels to the data — column names and a row index that travel with the values through every operation — and by letting a single table hold different types per column. The same question becomes one line:

df.assign(revenue=df.price * df.qty).groupby("region")["revenue"].mean()

Series: an array that remembers what things are called

A Series is a one-dimensional array plus an index.

1import pandas as pd23s = pd.Series([88, 92, 79, 95], index=["ada", "alan", "grace", "linus"])4print(s["grace"])      # 795print(s.mean())        # 88.56print(s[s > 85])       # ada 88, alan 92, linus 95

The index is the point. When you multiply two Series, Pandas aligns them by label, not by position:

1a = pd.Series([1, 2, 3], index=["x", "y", "z"])2b = pd.Series([10, 20, 30], index=["z", "y", "x"])   # reversed order3print(a * b)4# x     305# y     406# z     30

Two arrays in that situation would have multiplied 1×10, giving nonsense. Pandas matched x with x. That automatic alignment is the feature that prevents the whole class of "my columns got out of sync" bugs — and it is also why a merge on mismatched labels produces surprising NaNs rather than an error. Alignment is always happening, whether you asked for it or not.

DataFrames

A DataFrame is a dictionary of Series that share one index. Each column has its own dtype.

1df = pd.DataFrame({2    "region":  ["north", "south", "north", "east", "south"],3    "product": ["A", "B", "A", "C", "A"],4    "price":   [10.0, 12.5, 10.0, 8.0, 10.0],5    "qty":     [3, 1, 5, 2, 4],6})

The first four things to run on any new table

1df.head(3)      # first rows -- eyeball the actual values2df.shape        # (5, 4) -- rows, columns3df.info()       # dtypes and non-null counts per column: the highest-value one4df.describe()   # count, mean, std, min, quartiles, max for numeric columns

info() earns its place because it answers two questions at once. If a column you expect to be numeric shows str (or object in pandas 2 and earlier), something in it is text — a stray "N/A", a currency symbol, a thousands separator — and every arithmetic operation on it will either fail or concatenate strings. If the non-null count is below the row count, you have missing data. Both problems are far cheaper to find now than after you have built a model.

Selecting data: the three accessors

This is where beginners lose the most time, because Pandas offers several ways to select and they behave differently.

df.loc[1:3, ["region", "price"]]   # rows labelled 1,2,3 -- THREE rowsdf.iloc[1:3, [0, 2]]               # rows at positions 1,2 -- TWO rows

That difference is deliberate. loc works on labels, and with a label-based slice there is no sensible "one past the end", so the endpoint is included. iloc works on positions and follows the ordinary Python convention. Mixing them up gives you an off-by-one that does not raise an error.

The chained assignment trap

Sooner or later you will write this, get a warning, and find that nothing changed:

df[df.region == "north"]["price"] = 11.0     # WRONG -- does nothing

The first bracket produces a new object, and the assignment writes into that temporary, which is thrown away a microsecond later. Since pandas 3.0 this is guaranteed: under the "copy-on-write" rules every filtered result behaves as a copy, so a chained assignment never updates df, and pandas warns with ChainedAssignmentError. (Older versions sometimes wrote through and sometimes did not, and warned with SettingWithCopyWarning — you will still see that name in older tutorials.)

df.loc[df.region == "north", "price"] = 11.0   # correct: one operation

Whenever you are assigning into a filtered table, put the row condition and the column name inside a single .loc[]. Two sets of brackets in a row on the left of an = is always a bug.

Filtering, sorting, and building columns

1df[(df.price > 9) & (df.qty >= 3)]         # & and |, each term bracketed2df[df.region.isin(["north", "east"])]      # cleaner than chained ORs3df[df["product"].str.startswith("A")]      # .str gives string methods on a column4df.query("price > 9 and qty >= 3")         # readable alternative56df.sort_values("price", ascending=False).head(3)7df.sort_values(["region", "price"], ascending=[True, False])

The &/| rule is the same as with arrays, and so is the reason: and demands a single true/false answer, and a column of five values cannot supply one. It raises ValueError: The truth value of a Series is ambiguous.

1df["revenue"] = df.price * df.qty                 # vectorised, no loop2df["tier"] = pd.cut(df.revenue, bins=[0, 20, 50, 999],3                    labels=["small", "medium", "large"])4df["region_code"] = df.region.map({"north": 1, "south": 2, "east": 3})56df = df.drop(columns=["region_code"])7df = df.rename(columns={"qty": "quantity"})

assign is worth knowing because it returns a new frame and therefore chains:

1summary = (df2    .assign(revenue=lambda d: d.price * d.quantity)3    .query("revenue > 15")4    .sort_values("revenue", ascending=False)5)

Chained expressions like that read top to bottom as a sequence of steps, and they never mutate the input, so if the answer looks wrong you still have the original to inspect.

GroupBy: split, apply, combine

Almost every analytical question is a group-by in disguise. "Average revenue per region", "best-selling product per month", "churn rate per plan" — same shape every time.

1df.groupby("region")["revenue"].mean()23df.groupby("region").agg(4    total_revenue=("revenue", "sum"),5    avg_price=("price", "mean"),6    orders=("revenue", "count"),7).sort_values("total_revenue", ascending=False)

Pandas does three things there: splits the rows into groups by region, applies the aggregation within each group, and combines the results into a new frame indexed by region. The named-aggregation form above is the one to prefer — it produces sensible column names instead of a multi-level header you then have to flatten.

df.groupby(["region", "product"])["revenue"].sum()      # multiple keysdf.groupby("region")["revenue"].transform("mean")       # broadcast back to rows

transform is the underrated one. Where agg gives you one row per group, transform gives you one value per original row, so you can compare each row against its own group:

df["vs_region_avg"] = df.revenue - df.groupby("region")["revenue"].transform("mean")

One behaviour catches everyone: by default, rows where the grouping key is missing are dropped entirely. If 12% of your rows have no region, your regional totals will not add up to the overall total and nothing will tell you why. Pass dropna=False when you want those rows counted.

Combining tables

Merge joins on values

1orders = pd.DataFrame({"cust_id": [1, 2, 3], "amount": [100, 250, 75]})2custs  = pd.DataFrame({"cust_id": [1, 2, 4], "name": ["Ada", "Alan", "Grace"]})34pd.merge(orders, custs, on="cust_id", how="inner")   # 2 rows: ids 1 and 25pd.merge(orders, custs, on="cust_id", how="left")    # 3 rows: id 3 gets NaN name

The failure mode that bites hardest is a merge where the key is not unique on the right. If custs accidentally contains customer 2 twice, an inner join produces two rows for every one of customer 2's orders — and your revenue total doubles for that customer with no error anywhere. Check the row count before and after every merge, or pass validate="many_to_one" and let Pandas raise if the assumption breaks.

A merge that changes your row count in a way you did not predict is a bug, even when the numbers look plausible. Assert the shape.

Concat stacks

pd.concat([jan, feb, mar], ignore_index=True)   # stack rows: same columnspd.concat([features, targets], axis=1)          # stack columns: same index

Use merge when you are matching on a key, concat when you are piling up pieces that already have the same structure. ignore_index=True matters when stacking rows: without it you keep three copies of the index values 0, 1, 2, and later .loc[0] returns three rows.

Getting data in and out

1df = pd.read_csv(2    "sales.csv",3    parse_dates=["order_date"],          # otherwise dates arrive as text4    dtype={"postcode": "string"},        # stop 01234 becoming 12345    na_values=["N/A", "-", "missing"],   # map your file's blanks to NaN6    usecols=["order_date", "region", "price", "qty"],   # read less, go faster7)89df.to_csv("clean.csv", index=False)      # index=False, or you gain a junk column10df.to_parquet("clean.parquet")           # round-trips dtypes correctly

Those read_csv arguments prevent the three most common import bugs: dates that stay as strings and cannot be sorted chronologically, identifier columns that lose their leading zeros because Pandas guessed "integer", and placeholder text like "-" that stops a whole column from being numeric.

The shape of a real analysis

1import pandas as pd23sales = pd.read_csv("sales.csv", parse_dates=["order_date"])4custs = pd.read_csv("customers.csv")56before = len(sales)7df = sales.merge(custs, on="cust_id", how="left", validate="many_to_one")8assert len(df) == before, "merge changed the row count"910report = (df11    .assign(revenue=lambda d: d.price * d.qty,12            month=lambda d: d.order_date.dt.to_period("M"))13    .query("revenue > 0")14    .groupby(["month", "segment"], dropna=False)15    .agg(revenue=("revenue", "sum"),16         orders=("revenue", "count"),17         avg_order=("revenue", "mean"))18    .round(2)19    .sort_values("revenue", ascending=False)20)2122print(report.head(10))

Three defensive choices are doing quiet work there. The validate argument and the assertion catch a duplicated customer key before it silently inflates revenue. dropna=False means customers with no segment appear as their own group rather than vanishing from the totals. And every step returns a new frame, so sales and custs are still pristine if the output looks wrong.

The habit that matters more than any single method: after each step in a chain, check .shape and look at .head(). Pandas will very rarely stop you from doing something incorrect. It will align, join and aggregate whatever you hand it, and hand back a table that looks entirely reasonable. The row count is usually the first thing to notice that it is not.