← Lab 6

1 · Look before you analyse

Lecture part 1 again, slowly, with you typing the code. Find out what is in the survey, and the two things to be careful about, before asking it anything.

From the lecture
df.shape · df.head()
how big the table is; the first five rows
df.info()
the type of each column, and whether anything is missing
df.describe()
a quick summary of every number column
df["Honeybees"].sum()
one column (a Series), then a method on it
df["Allotment"].value_counts()
how many rows have each value
(df["Honeybees"] > 5).sum()
a mask of True/False, then count the Trues
In the notebook
  • ▸the size and shape of the table
  • ▸what describe() tells you, row by row
  • ▸the lecture’s revision numbers, worked out yourself
  • ▸two odd things: flower counts that are estimates, and Weald (495 surveys) vs Camp_Site (2)
  • ▸counting with a condition
📓 01 · Look before you analyse →

About 35 minutes. Every exercise has a self-check that tells you what to fix.

2 · Pick out the rows you want

Stop looking at the whole table. Choose the columns, keep the rows that match, put them in order, and do step 1 of the lecture yourself.

From the lecture
df[["Crop", "Bumblebees"]]
choose columns: a list inside the brackets
df[df["Crop"] == "Bbean"]
keep the rows where the mask is True
(cond 1) & (cond 2)
“and”: round brackets around each condition
df["Crop"].isin([...])
match any value from a list
df.sort_values("Flowers", ascending=False)
put the rows in order, biggest first
In the notebook
  • ▸choosing columns, and the double brackets
  • ▸filtering rows, read from the inside out
  • ▸two conditions with &, and the error message to look out for
  • ▸sorting, and the busiest surveys
  • ▸step 1: who visits broad bean?
📓 02 · Pick out the rows you want →

About 35 minutes.

3 · From one crop to every crop

Steps 2 to 5 of the lecture: if the bumblebees disappeared, which crop would lose the most? Then one more check, to make the comparison fair.

From the lecture
df.groupby("Crop")["Bumblebees"].sum()
for each crop, add up the bumblebees
visits["share"] = ... / (... + ...)
a new column, calculated row by row
(visits["share"] * 100).round(1)
the share as a percentage
visits.sort_values("share", ascending=False)
which crop is top?
visits[visits["Bumblebees"] >= 20]
keep only the crops with enough data
df.groupby("Crop").size()
how many surveys of each crop
In the notebook
  • ▸groupby as split, take, combine
  • ▸the bumblebee share, and what NaN means
  • ▸why strawberry “wins” with 100%
  • ▸the trust filter, and the lecture’s answer
  • ▸per survey: two questions, two answers
📓 03 · From one crop to every crop →

About 40 minutes. Lecture 7 turns tables like these into figures.

Advanced · The real survey file

Optional, for when notebooks 1 to 3 felt comfortable. The file the researchers published: 23 columns, and the problems every real dataset has. Fewer demos, more decisions.

Part A · Cleaning: four problems
df.columns.str.strip()
a column that won’t work: a hidden space in its name
df.isna().sum()
missing plant counts. Why brambles? And should you drop them?
df.duplicated().sum()
rows that appear twice. Mistakes, or real empty surveys?
pd.to_datetime(df["Date"])
dates stored as text, and six surveys in the wrong month
Part B · Analysis: three questions
  • ▸do shared plots attract more insects? (per survey vs per flower)
  • ▸which bumblebee species visit broad bean, and what is nectar robbing?
  • ▸was 2018 a worse year for bees?
  • ▸then: a question of your own
📓 04 · Advanced: the real survey file →

About an hour; fine to spread over two sessions. Nothing in the assessed work depends on it.