Python数据分析基础

• Preparation
• Exercise 1- US - Baby Names
• • • Introduction:
    • Step 1. Import the necessary libraries
    • Step 2. Import the dataset from this [address](https://raw.githubusercontent.com/guipsamora/pandas_exercises/master/06_Stats/US_Baby_Names/US_Baby_Names_right.csv).
    • Step 3. Assign it to a variable called baby_names.
    • Step 4. See the first 10 entries
    • Step 5. Delete the column 'Unnamed: 0' and 'Id'
    • Step 6. Is there more male or female names in the dataset?
    • Step 7. Group the dataset by name and assign to names
    • Step 8. How many different names exist in the dataset?
    • Step 9. What is the name with most occurrences?
    • Step 10. How many different names have the least occurrences?
    • Step 11. What is the median name occurrence?
    • Step 12. What is the standard deviation of names?
    • Step 13. Get a summary with the mean, min, max, std and quartiles.
• Exercise 2- Wind Statistics
• • • Introduction:
    • Step 1. Import the necessary libraries
    • Step 2. Import the dataset from this [address](https://raw.githubusercontent.com/guipsamora/pandas_exercises/master/06_Stats/Wind_Stats/wind.data)
    • Step 3. Assign it to a variable called data and replace the first 3 columns by a proper datetime index.
    • Step 4. Year 2061? Do we really have data from this year? Create a function to fix it and apply it.
    • Step 5. Set the right dates as the index. Pay attention at the data type, it should be datetime64[ns].
    • Step 6. Compute how many values are missing for each location over the entire record.
    • • They should be ignored in all calculations below.
    • Step 7. Compute how many non-missing values there are in total.
    • Step 8. Calculate the mean windspeeds of the windspeeds over all the locations and all the times.
    • • A single number for the entire dataset.
    • Step 9. Create a DataFrame called loc_stats and calculate the min, max and mean windspeeds and standard deviations of the windspeeds at each location over all the days
    • • A different set of numbers for each location.
    • Step 10. Create a DataFrame called day_stats and calculate the min, max and mean windspeed and standard deviations of the windspeeds across all the locations at each day.
    • • A different set of numbers for each day.
    • Step 11. Find the average windspeed in January for each location.
    • • Treat January 1961 and January 1962 both as January.
    • Step 12. Downsample the record to a yearly frequency for each location.
    • Step 13. Downsample the record to a monthly frequency for each location.
    • Step 14. Downsample the record to a weekly frequency for each location.
    • Step 15. Calculate the min, max and mean windspeeds and standard deviations of the windspeeds across all locations for each week (assume that the first week starts on January 2 1961) for the first 52 weeks.
• Conclusion

Preparation

需要数据集可以自行网上寻找（都是公开的数据集）或私聊博主，传到csdn，你们下载要会员，就不传了。下面数据集链接下载不一定能成功。

Exercise 1- US - Baby Names

Introduction:

We are going to use a subset of US Baby Names from Kaggle.
In the file it will be names from 2004 until 2014

Step 1. Import the necessary libraries

代码如下：

import pandas as pd

Step 2. Import the dataset from this address.

Step 3. Assign it to a variable called baby_names.

代码如下：

baby_names = pd.read_csv("US_Baby_Names_right.csv")
baby_names.info()

输出结果如下：

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1016395 entries, 0 to 1016394
Data columns (total 7 columns):
Unnamed: 0    1016395 non-null int64
Id            1016395 non-null int64
Name          1016395 non-null object
Year          1016395 non-null int64
Gender        1016395 non-null object
State         1016395 non-null object
Count         1016395 non-null int64
dtypes: int64(4), object(3)
memory usage: 54.3+ MB

Step 4. See the first 10 entries

代码如下：

baby_names.head(10)

输出结果如下：

Step 5. Delete the column ‘Unnamed: 0’ and ‘Id’

代码如下：

del baby_names['Id']
# OR del baby_names['Unnamed: 0']
baby_names = baby_names.loc[:, ~baby_names.columns.str.contains('^Unnamed')]
baby_names.head()

输出结果如下：

Step 6. Is there more male or female names in the dataset?

代码如下：

# baby_names['Gender'].value_counts()
baby_names.groupby('Gender').Count.sum()

输出结果如下：

Gender
F    16380293
M    19041199
Name: Count, dtype: int64

Step 7. Group the dataset by name and assign to names

代码如下：

del baby_names["Year"]
names = baby_names.groupby("Name").sum()
names.head()
print(names.shape)
names.sort_values("Count", ascending = 0).head()
# names= baby_names.groupby('Name')
# names.head(1)

输出结果如下：

(17632, 1)

Step 8. How many different names exist in the dataset?

代码如下：

len(names)

输出结果如下：

17632

Step 9. What is the name with most occurrences?

代码如下：

# names['Count'].sum().argmax()
names.Count.idxmax()  # idxmax()获取pandas中series最大值对应的索引

输出结果如下：

'Jacob'

Step 10. How many different names have the least occurrences?

代码如下：

len(names[names.Count == names.Count.min()])

输出结果如下：

2578

Step 11. What is the median name occurrence?

代码如下：

names[names.Count == names.Count.median()]

输出结果如下：

66 rows × 1 columns

Step 12. What is the standard deviation of names?

代码如下：

names.Count.std()

输出结果如下：

11006.069467891111

Step 13. Get a summary with the mean, min, max, std and quartiles.

代码如下：

names.describe()

输出结果如下：

Exercise 2- Wind Statistics

Introduction:

The data have been modified to contain some missing values, identified by NaN.
Using pandas should make this exercise
easier, in particular for the bonus question.

You should be able to perform all of these operations without using
a for loop or other looping construct.

1. The data in ‘wind.data’ has the following format:

"""
Yr Mo Dy   RPT   VAL   ROS   KIL   SHA   BIR   DUB   CLA   MUL   CLO   BEL   MAL
61  1  1 15.04 14.96 13.17  9.29   NaN  9.87 13.67 10.25 10.83 12.58 18.50 15.04
61  1  2 14.71   NaN 10.83  6.50 12.62  7.67 11.50 10.04  9.79  9.67 17.54 13.83
61  1  3 18.50 16.88 12.33 10.13 11.17  6.17 11.25   NaN  8.50  7.67 12.75 12.71
"""

'\nYr Mo Dy   RPT   VAL   ROS   KIL   SHA   BIR   DUB   CLA   MUL   CLO   BEL   MAL\n61  1  1 15.04 14.96 13.17  9.29   NaN  9.87 13.67 10.25 10.83 12.58 18.50 15.04\n61  1  2 14.71   NaN 10.83  6.50 12.62  7.67 11.50 10.04  9.79  9.67 17.54 13.83\n61  1  3 18.50 16.88 12.33 10.13 11.17  6.17 11.25   NaN  8.50  7.67 12.75 12.71\n'

The first three columns are year, month and day. The
remaining 12 columns are average windspeeds in knots at 12
locations in Ireland on that day.

More information about the dataset go here.

Step 1. Import the necessary libraries

代码如下：

import pandas as pd
import datetime

Step 2. Import the dataset from this address

Step 3. Assign it to a variable called data and replace the first 3 columns by a proper datetime index.

代码如下：

data = pd.read_table('wind.data', sep='\s+', parse_dates = [[0, 1, 2]])
data.head()

输出结果如下：

Step 4. Year 2061? Do we really have data from this year? Create a function to fix it and apply it.

代码如下：

def fix_century(x):
    year = x.year - 100 if x.year > 1989 else x.year
    return datetime.date(year, x.month, x.day)
data['Yr_Mo_Dy'] = data['Yr_Mo_Dy'].apply(fix_century)
data.head()

输出结果如下：

Step 5. Set the right dates as the index. Pay attention at the data type, it should be datetime64[ns].

代码如下：

data["Yr_Mo_Dy"] = pd.to_datetime(data["Yr_Mo_Dy"])  # 转换为datetime64
data = data.set_index('Yr_Mo_Dy')
data.head()

输出结果如下：

Step 6. Compute how many values are missing for each location over the entire record.

They should be ignored in all calculations below.

代码如下：

data.isnull().sum()

输出结果如下：

RPT    6
VAL    3
ROS    2
KIL    5
SHA    2
BIR    0
DUB    3
CLA    2
MUL    3
CLO    1
BEL    0
MAL    4
dtype: int64

Step 7. Compute how many non-missing values there are in total.

代码如下：

data.shape[0] - data.isnull().sum()
#OR data.notnull.sum()

输出结果如下：

RPT    6568
VAL    6571
ROS    6572
KIL    6569
SHA    6572
BIR    6574
DUB    6571
CLA    6572
MUL    6571
CLO    6573
BEL    6574
MAL    6570
dtype: int64

Step 8. Calculate the mean windspeeds of the windspeeds over all the locations and all the times.

A single number for the entire dataset.

代码如下：

data.fillna(0).values.flatten().mean()  # a.flatten()就是把data降到一维，默认是按行的方向降

输出结果如下：

10.223864592840483

Step 9. Create a DataFrame called loc_stats and calculate the min, max and mean windspeeds and standard deviations of the windspeeds at each location over all the days

A different set of numbers for each location.

代码如下：

# loc_stats = data.loc[:, 'RPT':'MAL'].describe(percentiles=[])
# loc_stats
data.describe(percentiles=[])

输出结果如下：

Step 10. Create a DataFrame called day_stats and calculate the min, max and mean windspeed and standard deviations of the windspeeds across all the locations at each day.

A different set of numbers for each day.

代码如下：

day_stats = pd.DataFrame()

day_stats['min'] = data.min(axis = 1)
day_stats['max'] = data.max(axis = 1)
day_stats['mean'] = data.mean(axis = 1)
day_stats['std'] = data.std(axis = 1)

day_stats.head()

输出结果如下：

Step 11. Find the average windspeed in January for each location.

Treat January 1961 and January 1962 both as January.

代码如下：

data.loc[data.index.month == 1].mean()

输出结果如下：

RPT    14.847325
VAL    12.914560
ROS    13.299624
KIL     7.199498
SHA    11.667734
BIR     8.054839
DUB    11.819355
CLA     9.512047
MUL     9.543208
CLO    10.053566
BEL    14.550520
MAL    18.028763
dtype: float64

Step 12. Downsample the record to a yearly frequency for each location.

代码如下：

# pd.Period()创建时期数据
# pd.Period()参数：一个时间戳 + freq 参数 → freq 用于指明该 period 的长度，时间戳则说明该 period 在时间轴上的位置
# DatetimeIndex对象的数据转换为PeriodIndex
data.groupby(data.index.to_period('A')).mean() 

输出结果如下：

Step 13. Downsample the record to a monthly frequency for each location.

代码如下：

data.groupby(data.index.to_period('M')).mean().head()

输出结果如下：

Step 14. Downsample the record to a weekly frequency for each location.

代码如下：

data.groupby(data.index.to_period('W')).mean().head()

输出结果如下：

Step 15. Calculate the min, max and mean windspeeds and standard deviations of the windspeeds across all locations for each week (assume that the first week starts on January 2 1961) for the first 52 weeks.

代码如下：

# data.groupby(data.index.to_period('1961-01-02', 'W')).describe(percentiles=[]).head()
weekly = data.resample('W').agg(['min', 'max', 'mean', 'std'])  # resample()重新设置频率采样，再sh
weekly.loc[weekly.index[1:53], "RPT":"MAL"].head(10)

输出结果如下：

10 rows × 48 columns

Conclusion

今天的pandas题更新，继续刷题，加油！